// Reinforcement Learning From Executed Trials
Adapt-1 learns from the actions that an application executes and measures. Trajectory learning uses measured outcomes to retain executions, select changes of structure and adapt refinement steps. With progress-guided settings, outcomes also direct acquisition edits.
Our previous article showed Adapt-1 learning continuous control sequences from many starting conditions. This post looks at trajectory learning itself. It shows how Adapt-1 acquires a command sequence from its own executions, on tasks where the learner can compare each revision with its parent.
Learning from executed experience
Trajectory learning improves the sequence of commands that executes an action. It learns from the measured outcome of each execution.
Each experiment in this post evaluates trajectory learning through the public routes of the API, with substrate planning disabled.
The trajectory learner changes only when the application reports the outcome of an execution. A proposal reads the adaptive state and leaves it unchanged. In 12 runs of the feeding-schedule task, each observed attempt raised the state version by exactly 1, and no proposal changed it. Each run made 4,000 attempts. The application reports the commands that it actually executed, with any change from a safety check. A decision_id binds each outcome to the proposal that produced it (Figure 1).
Under this rule, the work of the application and the learning of Adapt-1 use the same executions. Box (1957) proposed evolutionary operation for chemical plants. The plant runs with small planned changes to its settings, so normal production also gives information about how to improve the process. Trajectory learning applies this idea to a task in the loop. Each execution gives a result to the application and a comparison to the learner.
Sutton (2001) argued that a system can keep knowledge only to the degree that it can check that knowledge itself. The trajectory learner executes each change to check it. A revision counts as an improvement only after its execution scores higher than its parent.
Definitions
- Episode
- 1 execution of a task from its start to its score y. In these runs, 1 episode executes 1 proposal.
- Attempt k
- The episode with number k in acquisition. Numbers start at 0.
- Parent p(k)
- The retained execution that attempt k revises.
- Changed set Δk
- The commands of the parent that attempt k replaces.
- Credit ck
- ck = yk − yp(k). A 1-interval credit has |Δk| = 1.
- Retained set
- The executions that the learner keeps. It held 8 executions in every feeding-schedule run.
- Version
- A counter in the adaptive state. Each observed execution raises it by 1.
- Stage
- A change of structure (selection, deletion or ordering). It must win on nomination contexts and then on separate confirmation contexts.
- Default arm
- Acquisition with the native settings. Each attempt carries only the final score.
- Profile arm
- Acquisition with the documented profile. Attempts also carry per-step outcomes.
One score per execution
Trajectory learning is reinforcement learning from executed command sequences. Its own executions select the data that it sees, and it must act before it knows enough. The score of an execution arrives at its end. The score tells how good the execution was, and it never names a better execution (Sutton and Barto, 2018). The method that Adapt-1 uses for this problem has 4 properties:
- Credit per execution. Each execution returns 1 score. Deisenroth et al. (2013) call policy search at this level episode-based. The profile can also send 1 measured outcome per command.
- Open loop. A proposal is a complete command sequence for 1 start and 1 goal. No command in it reacts to a measurement taken during the execution.
- Revision from trial to trial. After the first attempts, most proposals revise a retained execution. Iterative learning control (ILC) works in the same setting (Arimoto et al., 1984; Bristow et al., 2006). A machine repeats 1 task from the same initial state and improves its input from the earlier trials. ILC uses the error at each time step, and our default arm used only the final score.
- Credit from measured scores. The credit of a revision is the difference between 2 measured scores. No value estimate enters it.
Sutton and Barto (2018) note a weakness of evolutionary methods in RL: they ignore the states and actions that each candidate visits during its episode. Trajectory learning keeps the executed trace and edits indexed steps of it.
An analysis of 3 ChemGym runs checked where edits start. In more than 99% of edits with a changed row, the first changed row was inside the commands that the parent executed. The runs included 2 default runs and 1 profile run. Some parents executed only 5 or 7 commands of a 50-row plan. So edit placement uses the executed length of the parent.
Measured credit is available after the first few executions. Each credit is also a pair of logged executions, and anyone with the log can check it (Figure 4).
A worked example: a feeding schedule
Our main test task is the chemostat design problem of Treloar et al. (2022). A chemostat is a culture vessel with a constant flow. The learner sets the concentrations of 2 nutrients in the inflow, Cin and C0,in, for each of 10 intervals of 2 hours. A sensor measures only the cell density.
The score is the log-determinant of the Fisher information of 3 growth parameters. We report it as D-efficiency, relative to the best design that we know. The score has many local optima: 17 local searches stopped at 14 different values.
A steady feed of 0.5 g/L of each nutrient scores 0.08%, because a steady culture hides the effect of the 2 growth constants (Figure 2). The learned design of run 3 changes the state of the culture several times. It starves the culture of 1 nutrient and then the other. Then it lets the culture grow at nearly μmax, and from about hour 11 the density falls until the end. The design scores 95.1%.
Most of this score comes from the last hours, while the density falls. The benchmark code computes the sensitivity of the density from the direct effect of each parameter. This sensitivity does not decay when the culture washes out. With the full sensitivity equations, the learned design scores 54%. The learner optimized the score that it received. All designs and comparators in this post use the same benchmark code (methods notes 6 and 7).
The trajectory learner found these states only from score differences. Select Replay attempt 74 in Figure 2. That edit lowered C0,in in interval 3 of attempt 70 from 0.241 to 0.037 g/L. The nutrient of C0,in then became scarce near hour 5, and the score rose by 8.0 points. You can also drag any bar of any design and read the credit for your change.
On the chemostat, the 6 default runs reached a mean D-efficiency of 87.1% after acquisition. Sequence stages and refinement raised the mean to 94.5%, with at most 4,395 episodes per run. The next sections follow the revisions and changes of structure that produced these results.
Credit from a controlled difference
When an execution scores higher than its parent, which changed commands caused the gain?
Sutton (1984) called this the problem of temporal credit assignment. Most RL methods estimate the answer (Figure 3). A Monte Carlo policy gradient gives the return to every step and averages over many episodes (Williams, 1992). PPO and other actor-critic methods give each step the change in a learned value (Schulman et al., 2017). In evolutionary search, such as CMA-ES or differential evolution (DE), the score goes to a random change of all parameters at once (Salimans et al., 2017).
The trajectory learner of Adapt-1 measures the answer. It copies an earlier execution, changes a small part of it and executes the result. The score difference goes to the commands that changed. On a deterministic task with 1 start, a revision of 1 interval gives the exact credit of that interval. No estimate enters it, and anyone with the log of the 2 executions can check it.
Many control and design tasks start from 1 known state and give 1 score at the end. If such a task is also deterministic, an open-loop sequence can do as well as any policy, because nothing unexpected can occur during the episode. The chemostat simulator is such a task.
| Rule | Credit goes to | Credit comes from | Cost of 1 credit | When the start or the score varies |
|---|---|---|---|---|
| Trajectory learningc = y(revision) − y(parent) | The changed commands | 2 measured scores | 1 execution | The score difference mixes the edit with that change. Table 1 gives the conditions under which the credit is exact. |
| Monte Carlo policy gradient∇J ≈ Σt ∇log π(at | st) (G − b) | Every step | 1 measured return | Many episodes | The average over episodes handles it. Each step carries the noise of all other steps, so the variance is high. |
| Actor-criticAt = rt + γV(st+1) − V(st) | Every step | A learned value V | Training of V | V handles it. The credit has a bias while V is wrong. |
| Evolutionary search∇J ≈ (1/nσ) Σi y(θ + σεi) εi | All parameters at once | Measured scores of sampled vectors | Many samples per update | The average over samples handles it. Its noise rises with the number of parameters. |
Each rule pays for its credit in a different way. Trajectory learning pays 1 new execution for each comparison with a measured parent. On a deterministic task with the same start, the difference is exact for the whole revision. If several commands change, the difference does not separate their individual effects. Table 1 distinguishes exact measurement, credit across revisions and execution cost. Figure 8 lets you change the start, the interactions and the noise.
Run 3 shows how this process develops from an empty state. Its first execution scored 18.5%. By attempt 284, its best execution scored 89.3%. Figure 4 shows its first 600 attempts, with the executions that scored lower than their parents.
Each attempt is 1 episode, and 1 episode executes 1 proposal. propose returns a full matrix of commands and a decision_id. The application executes the matrix and sends observe with the executed states, the commands and 1 outcome. After the first attempts, most proposals revise a retained execution. A revision keeps a prefix and changes the rest, or it changes specific intervals.
Attempt 74 copied attempt 70 and changed interval 3. D-efficiency went from 47.2% to 55.2%. The other 9 intervals were the same as in the parent. So the gain of 8.0 points belongs to interval 3, and no critic must estimate it.
Attempt 75 then copied attempt 74. It kept the new command at interval 3 and changed interval 7, which added 0.6 points. Attempt 76 copied attempt 75 and changed interval 1. Its score fell by 1.7 points. The sequence carries an earlier successful change into a new trial, and the new trial measures the next change. The loss also gives credit, with a negative sign.
Over the 6 runs of the default arm, exactly 1 interval changed in 1,611 of 1,656 improvements (97.3%). These improvements carried 371.6 of the 439.7 points of summed gain (84.5%).
In all 12 chemostat runs, the retained set reached 8 executions at attempt 8 and held 8 executions until attempt 4,000. These executions remain available as parents for later revisions. Frozen use returns the best retained execution. The result is an executable command sequence from the accumulated experience.
Over the 6 default runs, the first execution scored between 10.4% and 37.4%. After 4,000 attempts, the best execution scored 87.1% on average. The next sections follow how outcomes change later attempts and the structure of the sequence.
How outcomes change the next attempt
Each outcome changes what Adapt-1 proposes next. Our documentation describes how earlier executions serve as the basis for later proposals. A proposal can preserve a prefix and revise the continuation, or change selected commands. Our trace analysis shows that the effect of an outcome differs at each stage:
Default acquisition. The default arm uses the parent's executed length to place a seeded random edit. Outcomes determine which executions stay in the retained set, and so which executions can become parents. Figure 4 follows this process: attempt 74 improves on attempt 70, attempt 75 revises 74, and attempt 76 scores lower than 75.
Profile acquisition. In this arm, the progress-guided settings also use outcome history to direct edits. Edits return to parts of the plan that recently improved. Changes became smaller as the learner tried more children of the same parent in 2 of the 3 profile runs checked for edit size. The pattern varied by run, as detailed in the edit-analysis note.
Sequence stages. Selection, deletion and ordering list their candidates and measure each one. Outcomes select the candidate to keep. The stages change the retained structure through these comparisons.
Contextual refinement. The refinement traces show changes to both step size and direction after measured outcomes. Later steps can follow the last step that improved the score. The refinement section follows these changes.
Adapt-1 applies these update rules to the accumulated executions and outcomes. This experience determines which plans it retains, which changes of structure it adopts and how refinement adjusts later proposals.
When the credit arrives
Figure 5 places each 1-interval gain of 6 runs by interval and by attempt. Attempts go from the centre to the edge on a log scale. In the default arm, 39.4% of this credit came before attempt 100, and 2.9% came at attempt 1,000 or later. Run 3 gained less than 6 points in its last 3,700 attempts.
Each new execution compares a revision with an already measured parent. Early edits produce the largest gains, and later 1-interval edits refine the acquired design. All of these executions count in the budget.
The profile arm uses the documented acquisition profile, and its attempts also carry per-step outcomes. Its edits are larger. 336 of its 574 improvements changed exactly 1 interval, and they carried 27.4% of its gain. The rest came from edits of 2 to 10 intervals. Its acquisition also went further: 95.0% on average, against 87.1% for the default arm.
We read this as a match between edit size and task structure. The best designs that we know hold an inflow at a bound for many intervals. Larger edits can move such a run of intervals in 1 step. Their measured gain gives joint credit to the changed intervals. The comparison measures the combined effect of the acquisition settings and the per-step outcomes. It does not attribute the gain to a single setting.
Learning the structure of a sequence
After acquisition, 3 sequence stages edit the structure of the best execution: selection, deletion and ordering. Each stage uses a new Domain and the exported executions. The stage lists its candidates, executes each one and selects by measured outcome. It first measures the candidates on nomination contexts. Then it compares the best candidate with the current plan on separate confirmation contexts. The candidate replaces the current plan only if it also wins there.
This is sample splitting (Cox, 1975). It guards against maximization bias (Sutton and Barto, 2018, section 6.7). If 1 sample both selects the best of many noisy candidates and scores it, that score is too high.
Deletion tests the removal of each contiguous span, and ordering inserts a bridge row that changes 1 inflow before the other. Both stages changed 5 of the 6 default runs. Our executor holds the last command until the episode ends. In 4 runs, deletion removed a span that ended at interval 9, and the old interval 10 then held until the end. In all 5 runs, the bridge that won changed C0,in 1 interval before Cin. In run 1, this order gave 98.5%, and the opposite order gave 9.8%.
Selection retained the acquired plans. Over the 6 default runs, the mean went from 87.1% after acquisition to 94.2% after the stages, and to 94.5% after contextual refinement. The stages added the most to run 6, which acquisition left at 68.0%. They took it to 83.5%. Run 3 reached 95.1% in acquisition, and the stages kept its plan.
Each run used at most 4,395 episodes, and every stage candidate counts in that budget. The profile arm reached 95.0% in acquisition and 95.1% after the same stages.
On this deterministic task with 1 start, the confirmation repeated the nomination measurement, and every selected candidate passed. On noisy tasks, the gate separates real gains from lucky measurements. On the dodecane bench, each stage measurement was the mean of 8 fresh executions. Deletion tests n(n + 1)/2 spans for a plan of n commands.
Refinement adapts its next step
Contextual refinement adjusts the commands around an acquired sequence. A separate analysis of 20-interval chemostat runs follows how measured outcomes change these adjustments. Each update tests 4 candidates: the current plan, a pair of opposite steps and a probe in another direction. The radius sets the size of the proposed steps.
In all 4 runs checked, every win multiplied the radius by 1.25. After a non-win, the radius stayed the same or decreased. So the next update tests steps at a scale that depends on the earlier outcomes.
A direction analysis followed 1 run. In the reported update windows, the positive probe closely followed the last step that won. During 5 consecutive wins, the objective J rose from 19.929 to 19.984.
The proposal rule uses the measured history within the run. A successful step changes the base for later candidates and can change their direction and size. In the 10-interval study, contextual refinement raised the default mean from 94.2% after the sequence stages to 94.5%.
Test tasks
Our test tasks use continuous commands, discrete laboratory actions and token sequences (Figure 7).
Across the tasks, the task interface changes and the acquisition process stays the same. We did not tune the settings of either arm for a task. Each task acquires its experience independently.
We chose these tasks with 1 start and a score-based goal to isolate the effect of each revision (Patterson et al., 2024). A fixed start makes the score difference exact on deterministic tasks. Each episode returns 1 score. The feeding-schedule simulator is deterministic. The laboratory tasks can have noisy measurements, and many individual steps give no gain. In the sequence tests, most edits leave the score unchanged.
On these benchmarks, trajectory learning optimizes one sequence for one start. In the previous article, the same API used observations of the machine and the target to learn corrections across many starts.
All experiments use simulators and mathematical test functions. Their budgets count executions in these tests.
All tasks side by side
| Task | Commands per episode | Noise | Starts | When the score arrives | Structure | Learning question |
|---|---|---|---|---|---|---|
| Feeding scheduleChemostat, 10 intervals | 10 intervals × 2 continuous inflows | None | 1 | At the end | Many local optima. The best designs hold an inflow at a bound for many intervals. | How can revisions change a schedule with long runs of similar commands? |
| Salt extractionChemGym water-oil | Up to 50 of 41 bench actions | None: fixed random numbers | 1 | At the end. The plan does its work by step 10. | A safe zero. Most actions risk a spill, at −0.1 each. | How can a final score guide a procedure when most steps give no gain? |
| Product extractionChemGym dodecane | Up to 50 of 41 bench actions | As shipped | 1 | At the end. Most of it by step 8. | A safe zero. Most actions risk a spill, at −0.1 each. | How can the learner compare procedures when their measured scores vary? |
| Sequence AEhrlich f3 | 128 tokens from 32 | None | 1 | At the end. It rises when 1 more motif element matches. | A product over 4 motifs, with banned token pairs. | How can revisions build a sequence when most edits change no score? |
| Sequence BEhrlich f4 | 128 tokens from 32 | None | 1 | At the end, in 64 levels | A product over 8 motifs, with banned token pairs. | How far can the same process build a sequence with more required matches? |
ChemGym, a simulated chemistry benchmark, supplies the 2 laboratory tasks (Beeler et al., 2024). The learner chooses actions that move, mix and separate materials. In the water-oil task, it must extract salt from an oil mixture. In the other task, it must collect a target chemical called dodecane and remove the salt.
The score rewards the purity of the collected material and penalizes spills. Our water-oil executor fixes the random numbers at each step. The dodecane task remains noisy.
On the 2 ChemGym benches, the first executions scored 0 or less. After 2,000 episodes, the default arm reached 0.752 on water-oil and 0.559 on dodecane. These are 77.6% and 85.5% of ChemGym's hand-written plans. The profile arm also used per-step outcomes, which were mostly 0 or a spill penalty. It reached 0.108 on water-oil, and 2 of its 3 runs ended at 0, the safe score of the bench.
The sequence tasks ask the learner to build a list of 128 tokens from 32 symbols. The score increases as the sequence matches hidden patterns, called motifs. Each motif specifies symbols at fixed spacings. The task bans some adjacent token pairs.
These are the Ehrlich functions f3 and f4, which model constraints in biophysical sequence design (Stanton et al., 2024). The first uses 4 motifs of 4 elements. The second uses 8 motifs of 8 elements.
On Ehrlich f3, all 3 default runs ended at 0.75, with 15 of the 16 motif elements in place. We stopped acquisition by hand at this 15-element plateau. On f4, the median default run placed 44 of the 64 elements. In these Ehrlich runs, 23–26% of proposals changed no token, and 35–42% of attempts repeated an earlier sequence.
What a comparison measures
A comparison measures the change from 1 executed sequence to another. Table 1 separates 4 questions about that measurement. With the same start and no noise, it gives the exact effect of the whole revision with its parent. Interactions affect whether this credit holds when other commands change. Execution cost sets how many revisions the learner can test. Figure 8 shows the first 3 questions on a toy task.
| Condition | Effect on the comparison | In our runs | What Adapt-1 provides |
|---|---|---|---|
| 1. Same startThe revision starts from the state of its parent. | When starts differ, the score difference includes both the edit and the change of start. The runs here execute each proposed sequence open-loop. | Every task with a panel in Figure 7 had 1 start. In a separate run with 4 starts, each start kept its own 8 executions.† On the hidden-parameter chemostat, 1 fixed design had to serve every draw of the parameters. | Contextual refinement learns offsets from the start observation. |
| 2. No interactionsThis lets the effect of an edit hold when other commands change. | The measured credit applies to the edit in its parent sequence. Joint edits measure gains that occur only when several commands change together. | Chemostat: 97.3% of the default arm's improvements changed 1 interval, and its acquisition reached 87.1%. The profile arm made larger edits and reached 95.0%. | Edits of several steps can move a group of commands together. The deletion and ordering stages test changes of structure. |
| 3. No noiseThe same sequence from the same start gives the same score. | 1 comparison can give credit to noise. | Dodecane is noisy as shipped, and its stage measurements averaged 8 fresh executions. On the hidden-parameter chemostat, each outcome averaged 32 simulated experiments. We did not measure the effect of noise on the credit directly. | The application can average several measurements in 1 outcome. Each stage confirms its winner on separate contexts. |
| 4. Affordable executionsEach comparison costs 1 execution. | The budget sets how many edits the learner can test. When most edits change nothing, most executions return no credit. | Chemostat: 39.4% of the 1-interval credit came before attempt 100, and 2.9% at attempt 1,000 or later. Ehrlich: 23–26% of proposals changed no token, and 35–42% of attempts repeated an earlier sequence. | The progress-guided settings direct edits from outcome history. Their effect varied by task: on water-oil, the profile mean was 0.108 and the default mean was 0.752. |
Iterative learning control works under the first condition too: it resets the initial state before each trial (Bristow et al., 2006). Go-Explore uses the same strategy. It returns to states in a deterministic simulator to find trajectories. A separate phase then trains a policy for noisy conditions (Ecoffet et al., 2021). Adapt-1 uses contextual refinement to learn corrections from the start observation.
A hidden-parameter version of the chemostat tests 1 fixed design when the growth parameters change from draw to draw. The design does not observe the draw during execution. Treloar et al. (2022) built their closed-loop agent, RT3D, for this setting. Adapt-1 reached its published level in 7 of 10 runs, against a pre-registered requirement of 8.
Condition 2 concerns interactions, which one-factor-at-a-time experiments do not estimate (Czitrom, 1999). A 1-interval credit is the effect of the edit together with the other 9 intervals of its parent. Edits of several steps and the deletion and ordering stages measure the effects of joint changes (Figure 6). On a 20-interval version of the chemostat, the difference between the arms was larger. The default arm reached 82.7% and the profile arm 91.2%, with 6 runs each.
Noise is the third condition. On noisy tasks, Box repeated each set of small changes for several cycles to separate the effect from the noise (Box, 1957). An Adapt-1 application can do the same in 1 outcome, and the stages confirm each winner on separate contexts.
For the fourth condition, each comparison costs 1 execution, so the share of edits that change the score sets how much credit the budget buys. Each comparison answers a local question: is the revision better than its parent from this start?
The task interface and the learning conditions
In each run, the application defines the task and executes the proposals. Its choices affect what the learner can acquire and how much each attempt costs. We list all of the supplied information and task changes:
- Task interface. It gives the state and action dimensions, the horizon and what the score measures. We set the goal above the optimum, so no attempt can reach it. On Ehrlich, the goal was the optimum, 1.0.
- State features. We chose them.
- Executor. It maps commands to the task. On ChemGym it maps 2 numbers to 1 of 41 bench actions. On Ehrlich it maps 1 number to 1 of 32 tokens and moves a banned token to the nearest allowed token. On a short plan it holds or repeats the last command.
- 1 start per task, with the native defaults or the documented profile. We did not tune any setting for a task, and there was no transfer between tasks and no demonstration.
- 1 stop by hand. On Ehrlich f3 we ended acquisition at a plateau, after 14,301 to 14,571 attempts.
- Measurement conditions. Our water-oil executor fixed the random numbers of the bench at each step, as in PEGASUS (Ng and Jordan, 2000). On the hidden-parameter chemostat, each outcome was the mean of 32 simulated experiments, and all of them count in the budget.
Table 1 gives the conditions that apply to measured credit. In these runs, proposals execute open-loop and the learner keeps the retained executions separately for each start.
The retained executions support later revisions. Versions and decision records connect an outcome to the commands that ran. Confirmation contexts test changes before they serve. The stages spend episodes, which count in each budget.
What these runs establish
These runs show how Adapt-1 acquires executable sequences from its own trials. Acquisition starts with an empty adaptive state. Outcomes select executions to retain, and later attempts revise them. Sequence stages use measured comparisons to select changes to the length and order of the commands. The trace analysis also shows how refinement adjusts later steps from earlier outcomes. Frozen use returns the best retained execution.
Each test task used the same acquisition process (Figure 7). The chemostat records show how these sequences develop: a successful revision becomes the parent of another trial, and later stages improve the structure.
Methods notes
- Service and routes. All runs used the public Adapt-1 API. Acquisition used the trajectory routes
configure,propose,observeandstatewith theepisode_creditmechanism. The sequence stages usedconfirmed_execution, withinterventionset toselection,deletionorordering. Refinement usedcontextual_execution. Each final value comes from 1proposecall infrozenmode. - Builds. Most runs used build af6670b. The hidden-parameter runs used the production service, and its seed-1 run gave the same outcomes as the earlier build at all 1,000 attempts.
- Arms. The default arm uses the native acquisition settings, and each attempt carries only the final score. The profile arm uses the documented acquisition profile and also sends per-step outcomes. In both arms, the executed states include the score so far.
- Chemostat. Each inflow is between 0.01 and 1 g/L. D-efficiency is exp((J − J*)/3). Here J is the log-determinant of the Fisher information, and J* = 20.825 is the best local optimum that we know. Three wide searches found designs that differ in their inputs and agree within 2.1 × 10−4 in J. Relative to the best known design, the volume of the asymptotic confidence region for the 3 parameters is D−3/2 (Figure 7). Runs 1 to 6 follow the replicate order.
- Chemostat runs and simulator. Each run made 4,000 acquisition attempts, then the stages and contextual refinement (40 updates, 321 episodes), with at most 4,395 episodes in total. Figure 2 uses our own simulator of the benchmark code (Treloar et al., 2022), with the same model, parameter values, integration step and score. The model has a dilution rate of 0.5 per hour and μmax = 1 per hour. Its constants are Ks = 4.9 × 10−4 g/L and Ks0 = 6.8 × 10−5 g/L. Each experiment starts at 20 × 109 cells/L, with the nutrients of Cin and C0,in at 0 and 1 g/L. The simulator gives J* for the best known design and the logged scores of the designs of run 3.
- Chemostat score. The standard errors are the square roots of the diagonal of the inverse Fisher information, for the logarithms of the parameters. The benchmark code integrates the sensitivity of the density from the direct effect of each parameter on growth. We also computed the score with the full sensitivity equations and the same noise model. Relative to the best known design, the steady washout feed then scores 0.9%, and the learned design of run 3 scores 54%. All designs and comparators in this post use the benchmark code.
- Chemostat model. Growth follows a Monod law for each nutrient (Monod, 1949). The density carries information about Ks or Ks0 only while that nutrient is scarce, near its constant. Both constants are small, so the cells must use up a nutrient before it becomes scarce. The steady feed of 0.5 g/L leaves the standard error of Ks0 at about 100 times its value. In the learned design of run 3, the standard errors are 4.8% for μmax, 10.6% for Ks and 28.2% for Ks0. A steady feed that washes the culture out scores 33.6%, with a correlation of 0.9998 between the estimates of μmax and Ks.
- Credit statistics. An improvement is an attempt that scores higher than its parent. A 1-interval improvement changes the inflows of exactly 1 interval. Gains are in D-efficiency points. Attempt numbers start at 0, and attempt 0 of each run comes from an empty adaptive state. The version and the retained count come from the attempt logs of all 12 runs.
- Stages. Deletion measured the current plan and 54 contiguous spans. Ordering measured 2 bridge rows at each switch where both inflows change. Each stage confirmed its selected candidate against the current plan on 1 separate context. Refinement learns offsets for the whole sequence from the start observation. On the chemostat, the tested configuration used 4 candidates per update and the offsets did not use the context.
- ChemGym. WaterOilExtract-v0 and GenWurtzExtract-v2 with dodecane as the target, bench seed 0, 50-step episodes and 41 discrete actions through a 2-number decoder. The score is ChemGym's purity at the end minus 0.1 for each spill. Our water-oil executor seeds the random numbers of the bench at each step, so water-oil is deterministic. Dodecane is noisy as shipped, and its values are means of 200 fresh readings. With the stages, runs used 2,546 to 9,096 episodes, so this post reads the values at 2,000 episodes.
- Ehrlich. Instances f3 and f4 of Chen et al. (2024), seed 0: 4 motifs of 4 elements (f3) and 8 motifs of 8 elements (f4). One command selects 1 token through 32 equal bins. The state has 2 values: the position and the score of the prefix. On f3 we stopped acquisition by hand at a plateau, after 14,301 to 14,571 attempts.
- Evidence. We recomputed the performance and credit statistics from the raw attempt logs and comparator files. This applies to Figures 4, 5, 6 and 7 and Table 1. Figure 2 computes its values with the simulator of note 5. A † marks the exceptions from run reports. The additional measurements of edit placement, edit size and refinement come from our trace analysis, as detailed in the edit-analysis notes below. The published RT3D level comes from Treloar et al. (2022), and we did not rerun RT3D.
- Related methods. Go-Explore (Ecoffet et al., 2021) also keeps executed trajectories and returns to them, with no confirmation gate. Episodic control (Blundell et al., 2016) stores returns for each state. Iterative learning control (Bristow et al., 2006) refines 1 input sequence from the error at each time step. Evolution strategies adapt step size from success and failure (Auger and Hansen, 2013). Adaptive operator selection uses measured effects to choose later operators (Fialho et al., 2010).
- Execution budgets. All budgets count simulator executions. Each chemostat episode simulates 20 hours of culture growth.
- Edit placement. Our trace analysis supplies these measurements. For edit placement, the first changed row was usually within the parent's executed commands. The counts were 1,879 of 1,891 edits in
ext-t0-default-r3, 1,931 of 1,938 inoild-t0-default-r3, and 1,816 of 1,822 inext-t0-profile-r3. Each denominator counts edits with a changed row. The other edits started at the next row. - Edit size. We measured edit size as the largest absolute change to a control value. The analysis grouped the edits by the number of earlier children of the same parent and by thirds of the run. It found shrinkage in
oedpc10-t0-profile-seed1andlra10-t0-profile, but not inoed10-t0-profile-r5. The last is one of the main study's profile runs. - Refinement analysis. This analysis concerns separate 20-interval chemostat runs: default runs 1 and 2, and profile runs 1 and 3. The analysis checked direction in default run 1. The analysis records the trace paths and analysis commands. We report these measurements from that analysis. We did not recompute them from the figure data.
- Abbreviations. CMA-ES: covariance matrix adaptation evolution strategy. DE: differential evolution. PPO: proximal policy optimization (Schulman et al., 2017). RL: reinforcement learning.
References
- Arimoto, S., Kawamura, S., and Miyazaki, F. (1984). Bettering operation of robots by learning. Journal of Robotic Systems 1(2), 123–140.
- Auger, A., and Hansen, N. (2013). Linear convergence on positively homogeneous functions of a comparison based step-size adaptive randomized search: the (1+1) ES with generalized one-fifth success rule. arXiv:1310.8397.
- Beeler, C., et al. (2024). ChemGymRL: A customizable interactive framework for reinforcement learning for digital chemistry. Digital Discovery 3(4), 742–758. arXiv:2305.14177.
- Blundell, C., Uria, B., Pritzel, A., Li, Y., Ruderman, A., Leibo, J. Z., Rae, J., Wierstra, D., and Hassabis, D. (2016). Model-free episodic control. arXiv:1606.04460.
- Box, G. E. P. (1957). Evolutionary operation: A method for increasing industrial productivity. Journal of the Royal Statistical Society, Series C (Applied Statistics) 6(2), 81–101.
- Bristow, D. A., Tharayil, M., and Alleyne, A. G. (2006). A survey of iterative learning control. IEEE Control Systems Magazine 26(3), 96–114.
- Chen, A., Stanton, S. D., et al. (2024). Generalists vs. specialists: Evaluating LLMs on highly-constrained biophysical sequence optimization tasks. arXiv:2410.22296.
- Cox, D. R. (1975). A note on data-splitting for the evaluation of significance levels. Biometrika 62(2), 441–444.
- Czitrom, V. (1999). One-factor-at-a-time versus designed experiments. The American Statistician 53(2), 126–131.
- Deisenroth, M. P., Neumann, G., and Peters, J. (2013). A survey on policy search for robotics. Foundations and Trends in Robotics 2(1–2), 1–142.
- Ecoffet, A., Huizinga, J., Lehman, J., Stanley, K. O., and Clune, J. (2021). First return, then explore. Nature 590, 580–586.
- Fialho, Á., Schoenauer, M., and Sebag, M. (2010). Toward comparison-based adaptive operator selection. Proceedings of GECCO 2010, 767–774. doi:10.1145/1830483.1830619.
- Monod, J. (1949). The growth of bacterial cultures. Annual Review of Microbiology 3, 371–394.
- Ng, A. Y., and Jordan, M. I. (2000). PEGASUS: A policy search method for large MDPs and POMDPs. Proceedings of UAI 2000, 406–415.
- Patterson, A., Neumann, S., White, M., and White, A. (2024). Empirical design in reinforcement learning. Journal of Machine Learning Research 25(318), 1–63.
- Salimans, T., Ho, J., Chen, X., Sidor, S., and Sutskever, I. (2017). Evolution strategies as a scalable alternative to reinforcement learning. arXiv:1703.03864.
- Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017). Proximal policy optimization algorithms. arXiv:1707.06347.
- Stanton, S., et al. (2024). Closed-form test functions for biophysical sequence design. arXiv:2407.00236.
- Sutton, R. S. (1984). Temporal credit assignment in reinforcement learning. PhD thesis, University of Massachusetts Amherst.
- Sutton, R. S. (2001). Verification, the key to AI. Essay, incompleteideas.net.
- Sutton, R. S., and Barto, A. G. (2018). Reinforcement Learning: An Introduction, 2nd edition. MIT Press.
- Treloar, N. J., Braniff, N., Ingalls, B., and Barnes, C. P. (2022). Deep reinforcement learning for optimal experimental design in biology. PLOS Computational Biology 18(11): e1010695.
- Williams, R. J. (1992). Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning 8, 229–256.