Gemma: teaching a language model to paint with Optimus

Google DeepMind’s Gemma is a family of open-weight models that developers can adapt to new tasks. In this research collaboration, we explored a visual one: teaching a Gemma research checkpoint to write programs that paint.

The program mattered as much as the image. A finished render lets you see the result; its source lets you change how it was made. You could adjust the palette, move a shape, or reuse a brush operation without starting over. We wanted that kind of control to remain available after generation.

Google DeepMind and Iacon set the research direction and the criteria for judging the work. The project combined SFT with reinforcement learning. Optimus built the environment and experiment pipelines around those decisions. The central challenge became clear during training: a reward could improve without producing paintings we considered better.

Giving Gemma a medium it could practice

We used p5.brush, a JavaScript library for programmatic painting. Gemma received a visual instruction and wrote a complete sketch. A sandboxed browser executed the code and rendered an image, which the training system evaluated. The model could practice the same task repeatedly, with each attempt leaving both a program and a visible result.

Optimus implemented the renderer and the checks around it. A sketch had to run, use the supported brush interface, and produce an artifact before its visual quality could count. These checks gave the model a concrete contract: an explanation of a painting could not substitute for executable work.

That made the environment useful for both learning and review. We could inspect a disappointing image, open the code that produced it, and rerun that program. The output remained a working material that someone could edit.

Combining SFT, RLVR, and prompt optimization

We used supervised fine-tuning (SFT) alongside reinforcement learning with verifiable rewards (RLVR). They give the model different learning signals. SFT supplies target completions to learn from. Reinforcement learning lets Gemma sample its own programs and uses the results of executing them to guide weight updates. GEPA operated on another part of the system: the prompt. The diagram separates these learning signals and update paths; it does not specify the order of training stages.

Optimus · training & evaluationθ = model weights

SFT: learning from target completions

Supervised fine-tuning increases the likelihood of target tokens under a training example. For a prompt x and target completion y, the usual token-level objective is:

L_SFT(θ) = −Σ_t log π_θ(y_t | x, y_<t)

Here, θ denotes Gemma’s weights, and each prediction is conditioned on the prompt and preceding target tokens. This gives the model a direct signal about which completion to produce. It does not itself measure the quality of a rendered painting; that requires executing and evaluating the generated code.

RLVR: checking what the program actually does

Optimus implemented deterministic checks for execution and use of the supported brush API. Those checks supplied the verifiable part of the reward. A program had to satisfy the execution contract before its painting could earn visual credit. This ties learning to behavior we can test in the environment.

Composition and prompt adherence needed a separate signal. We used learned visual preference and comparisons against reference paintings for that part of the objective. The overall reward was therefore hybrid: exact program checks combined with judgments about the rendered result. RLVR describes the verifiable feedback; it is not the name of the policy-update algorithm.

GRPO: learning from a group of attempts

Group Relative Policy Optimization, or GRPO, provided the policy-update method. For the same prompt, Gemma samples a group of G programs. Optimus renders and scores the candidates so the update can use their relative performance.

GRPO compares each candidate’s reward with its group. A common stabilized form of the group-relative advantage is:

A_i = (r_i − μ_group) / (σ_group + ε)

Here μ and σ are the group’s reward mean and standard deviation; ε stabilizes the denominator. Above-average candidates have positive advantage; below-average candidates have negative advantage. If every reward is equal, this signal is zero. This is the advantage calculation, not the full training loss: GRPO uses it in a policy objective with clipping and, in the original formulation, a penalty for departing from a reference policy.

Optimus connected generation, rendering, verification, and scoring to this training loop. Human review established the reference taste; the experiment translated those judgments and executable checks into learning signals.

The score improved before the paintings did

The first objective combined several signals, including execution checks, a code-length incentive, and several judgments of visual quality. It looked like a thorough rubric. During training, though, reward increased while the paintings settled into a narrow, repetitive style. The programs also grew longer without a corresponding improvement in the images.

Optimus broke the reward into its components and examined their trajectories. Several visual judges gave correlated scores, suggesting overlap in what they rewarded. Our concern was that combining them could give one preference several votes, without adding much information about the paintings.

The code-length component contributed strongly early in training and then saturated. Meanwhile, a visual-preference signal continued to distinguish candidates but carried little weight. These patterns suggested an imbalance in the objective; they did not isolate the effect of any one component. Optimus used that diagnosis to propose a different reward design.

This changed what we worked on next. Before asking the checkpoint to train longer, we needed to improve the feedback it was learning from. Optimus used the component analysis and rendered artifacts to guide that redesign.

Comparing paintings gave the reward more resolution

An absolute aesthetic score asks a judge to place an image on a numeric scale. In the early run, those scores clustered together, making it difficult to distinguish a promising attempt from an ordinary one. We moved toward pairwise comparison: show the generated image beside a relevant reference and choose the stronger result.

For comparisons with a decisive winner, the signal has a simple interpretation: pairwise reward = comparisons won / comparisons made. This is a win rate against the chosen references under the chosen judge, not an absolute measure of artistic quality. Comparing that rate across runs requires keeping the reference pool and judging protocol fixed.

Human reviewers organized a pool of generated reference images into quality tiers. Strong examples anchored the comparisons, with additional references to broaden the range of palettes and compositions. The references came from the same programmable medium, so they illustrated what the interface could express without assuming that the learner could already reproduce it.

Optimus simplified the reward around execution and library checks, a small length check, a learned preference score, and the pairwise judge. The intent was to make differences between valid paintings more informative for the policy update.

What the redesign changed

Optimus changed how an attempt earned credit: executable checks established whether the program worked, while reference comparisons supplied relative visual feedback. The aim was to reward a stronger painting without encouraging unnecessary code. A shorter program is useful when it remains valid and preserves the quality of the render.

Changing the objective also changes how we must read the results. Raw reward values under different definitions are not directly comparable. A speedup requires a common evaluation target and a stated budget, such as generated samples or elapsed time. Visual gains likewise need a fixed evaluation outside the feedback used to optimize the model.

A smaller interface made the task clearer

We found a separate issue in the system prompt. Early versions included a long API reference, yet Gemma still invented plausible methods that did not exist. A well-formatted program could fail before there was any painting to judge.

Optimus used GEPA, a method that reflects on evaluation traces to propose and select prompt revisions. SFT and GRPO update model weights; GEPA changes the instructions supplied to the model. Optimus tested those candidate instructions against the painting evaluator. The search converged on a short prompt with an explicit allowlist of supported brush methods, replacing the large manual and its examples.

The smaller contract left more context for the visual instruction and made API errors easier to diagnose: we could check a program against a specific set of allowed operations. Reducing invalid calls was the goal of this change; the prompt's effect needs to be measured separately from changes to training and reward.

What we learned about Optimus

The useful work happened between training runs. Optimus built the renderer and verifiers, inspected the reward components, revised the objective, and tested the prompt. It could follow a problem across the environment, evaluation, and training code instead of treating the current setup as fixed.

The collaboration also depended on human judgment. Researchers established the task and reviewed the reference work; Optimus implemented experiments against that direction and brought back programs, renders, and measurements. That is how we want autonomous research to support customers: make ideas testable while keeping the research question and judgment of the work in human hands.

This project reinforced a practical lesson for us. A rising reward is a reason to inspect what the model learned. Keeping the generated code beside the image made that inspection possible, and Optimus could then change the machinery producing the feedback. The resulting environment also gives the team a way to test new reference styles, brush interfaces, and reward designs.