// Adapt-1 Machina: Learning Continuous Control Sequences
Adapt-1 Machina explores learning continuous control sequences from coarse outcome feedback, without demonstrations or a critic.
Machina continues Adapt-1 Preview with experiments in sequence-based control. Each task defines the observations and available controls. Adapt-1 proposes a sequence from the current observation, then uses execution feedback to revise later attempts.
To access Adapt-1 Machina API documentation: https://docs.reilabs.org/docs/machina
Learning budgets count executed attempts admitted to training. Frozen evaluation measures completion on reserved cases with learning disabled.
Background is available in Adaptive State from Partially Observed Streams. Introducing Adapt-1 Preview describes the earlier release; Emergence Toward Autonomous Structure Discovery discusses structure discovery.
Control proposals
A YAM proposal specifies an ordered sequence for the whole attempt from its starting observation. Each command requests a tool pose and gripper setting. Further learning occurs after execution feedback arrives.
The recorded acquisition proposals each contain 20 commands with seven continuous values each. Position and orientation use three coordinates apiece; the remaining value controls the gripper. Execution can terminate before the proposal ends.
Command representation
Each schematic column holds one command. The index k identifies its position in the sequence, and H denotes the proposed length. Command durations follow the execution contract. Learned hold durations and larger sequence limits were outside this experiment.
Learning from recorded execution
Adapt-1 retains executed commands with their observed consequences. A later proposal can refer to an earlier attempt and alter selected controls. This reuse begins during acquisition.
Training starts without a supplied successful trajectory or movement demonstrations. The client submits measured execution feedback to the learner.
After acquisition, the protocol evaluates candidates on additional training contexts and tests structural revisions. Contextual refinement learns bounded corrections from the observation at the start of execution.
YAM acquisition uses coarse progress feedback. Updated AIM refinement uses a continuous miss-quality score. The studies change several experimental conditions, leaving the effect of feedback detail on generalization unresolved.
Frozen evaluation computes commands from each observation using unchanged learner state.
Acquiring a YAM manipulation sequence
In MuJoCo, the YAM task requires releasing a red object inside a blue receptacle. Object and receptacle positions vary between layouts. In this setup, three calibrated cameras feed a fixed color-based measurement pipeline, combined with robot-state measurements. Goal images specify the desired arrangement. Hidden simulator object coordinates are excluded from decision inputs.
Acquisition starts with zero retained task records and no loaded task checkpoint. The first completed placement occurs on attempt 1,394.
The acquisition evaluator grades contact, sustained lift, transfer above the receptacle and stable released placement. Completion requires the object to remain placed for one second without robot contact.
The YAM run uses 18,400 learning episodes, including 7,680 acquisition attempts. The table accounts for every training stage. Frozen evaluations are counted separately.
YAM learning budget
One learning episode is an executed task attempt admitted to learning. An episode can contain several commands.
| Stage | Episodes |
|---|---|
| Acquisition | 7,680 |
| Selection | 4,032 |
| Deletion / reduction | 2,240 |
| Ordering | 2,400 |
| Contextual refinement | 2,048 |
| Total learning | 18,400 |
The acquisition log labels 262 proposals as initial trajectory explorations and 7,418 as revisions of earlier attempts.
The complete verification record contains 19,808 physical episodes and 251,186 executed commands. Training accounts for 240,946 command/state observations. The reserved final layouts use seeds absent from training, and the rounded visual layouts also pass the recorded overlap check.
Candidate selection measures completion on further training layouts. Calibrated landmarks and declared spatial control coordinates allow each candidate to be retargeted to the current observation. Nomination and confirmation use separate training contexts.
Structural revision evaluates removing contiguous command spans and changing when neighboring coordinate changes occur. Contextual refinement starts with zero correction weights and tests bounded offsets to the resulting sequence.
Reusing an earlier attempt
The experience record ends at the last command that ran. Termination can shorten execution; control bounds can also alter a requested value. The next revision uses that recorded execution.
Parent links identify the earlier attempt used for a revision. Indexed edits locate each changed control value, and preserved-prefix metadata identifies the beginning retained by the proposal.
Revision evidence and sequence lengths
The raw batch record contains both action arrays. Comparing them confirms the two marked edits. The proposal also records its parent identifier and preserved prefix, so the depicted reuse can be checked directly.
In this run, the selected execution contains 11 commands. The reduction output contains six, and the ordering output contains seven. The operation responsible for the added command remains unresolved in this audit. The complete learner update equations remain outside this trace audit.
Executing tool-pose requests
Fixed inverse kinematics converts a requested tool pose into joint targets. The low-level controller executes those targets with the accompanying gripper setting.
In the equation, p and R specify position and orientation; q⋆ is the resulting joint target. The robot model and workspace constraints remain fixed during the run.
The objective scores task progress and completed placement. Human likeness, motion smoothness, energy use and path efficiency receive no direct score.
Frozen YAM evaluation
Frozen evaluation on 384 unseen layouts measures placement success at 54.17% (208/384) for the acquired pre-refinement sequence and 60.42% (232/384) after contextual refinement. The paired records contain 70 gains and 46 losses, giving a net increase of 24 placements or 6.25 percentage points. Each pair uses identical starting observations and goals, with learning disabled throughout the comparison.
Layout 604008: placement gain.
The replay set’s first gain by seed. Each execution contains seven commands.
Paired outcomes and evaluation scope
The remaining pairs comprise 162 shared successes and 106 shared failures.
The bootstrap resamples layouts from this run. Variation across independent training runs remains unmeasured. Cases are arranged for readability; their display positions have no physical meaning. Replay starts and outcomes agree with the recorded comparison.
Aiming across target configurations
AIM uses a sentry simulated in MuJoCo, with finite-speed yaw and pitch joints. The recorded configuration accepts up to eight half-second commands specifying yaw rate, pitch rate and firing. The client supplies structured joint and target observations, and the learner commits the complete sequence before execution. The runtime then rotates the joints and evaluates firing at completed command boundaries.
Every near/far run starts with empty learner state. Acquisition supplies candidate aiming sequences; selection and structural revision produce the base sequence used in contextual refinement. The learned correction maps the initial observation to adjusted commands. Frozen evaluation holds learner state unchanged while varying target configurations within the training range. The single-range protocol trains its own fresh learner.
AIM / FULL LEARNING FILMAcquisition → refinement → frozen test
In the revised near/far runs, the contextual model uses standardized observations with quadratic features. Response-guided candidate corrections receive feedback from measured miss distance. The decoder adds twice the proposed offset to each base command and clips the result to the allowed command range.
The intervention comparison executes the acquired sequence unchanged, then evaluates the contextual policy with matching observations and with observations assigned from another case. Each execution is scored against its actual target.
Every contextual audit attempt launches a projectile. The recorded task uses a target radius of 22 cm and projectile speed of 10 m/s, with gravity and drag set to zero. These fixed physics apply throughout training and evaluation.
The updated near/far protocol changes the observation representation, proposal search, objective, action decoder and stopping rules. It also uses a larger refinement budget. Those simultaneous changes prevent attribution of the score difference to sample efficiency.
Learning budget and independent evaluation
The recorded near/far runs each contain 1,024 acquisition attempts over eight training targets. During updated refinement, each update evaluates eight candidates across 64 training contexts, producing 512 physical executions.
Seed 0 records 40 selection, 184 deletion and 168 ordering executions, followed by 36 refinement updates. Seeds 1 and 2 record 80 and 24 selection executions respectively; each also records 40 deletion executions, 72 ordering executions and 96 refinement updates. Development checks add 320 executions for seed 0 and 800 for each remaining seed. Their outcomes guide stopping.
The final audit uses 256 distinct configurations, each evaluated under three independently trained policies. Across 768 attempts per condition, the hit totals are 667 with matching observations, 85 for the acquired fixed sequences and 62 with shuffled observations. The audit was generated after implementation selection and excluded from training and stopping decisions. Earlier development checks used a separate panel.
| Seed | Mean miss | Median miss |
|---|---|---|
| 0 | 4.39 cm | 3.40 cm |
| 1 | 12.09 cm | 7.51 cm |
| 2 | 14.58 cm | 9.91 cm |
Under the original near/far protocol, the recorded run reaches 13/64 contextual hits and 9/64 fixed-base hits after 4,608 refinement executions.
Earlier single-range study
The earlier single-range learner trains and evaluates at a target depth of 4 m. Its final 64-case comparison records 58 contextual hits and 11 hits with the acquired fixed sequence. These cases were held out from that learner and reused from the historical V13 evaluation.
The single-range run admits 6,624 physical executions to learning: 1,024 acquisition, 80 selection, 184 deletion, 216 ordering and 5,120 refinement executions. Including evaluation, the execution total reaches 7,105. The request ledger contains 9,059 native requests.
Historical reports list six successes with goal observations permuted, four with horizontal control removed and 25 with vertical control removed on the fixed-distance panel. The evidence bundle lacks the separate intervention trial journals, so these figures remain unverified against individual executions.
Controlling delayed physical effects
Rail uses a carriage whose velocity responds over time to a signed request held for half a second. In Rail B, carriage acceleration also drives a passive suspended load. The velocity servo acts on the difference between requested and current carriage velocity. The learner receives target displacement and initial carriage velocity, with initial swing state in the suspended-load task.
Contextual refinement adjusts an acquired velocity sequence using the observed starting condition. Rail A scores carriage position and stopping speed. Rail B adds limits on payload motion and swing. Each task has its own learning workload and reserved evaluation panel.
Saved final policies retain six half-second commands in Rail A and eight in Rail B. Neutral velocity requests fill the remainder of each 5.5-second episode. Assessment covers the entire final second, using the maximum position errors and speeds during that interval. Rail B also checks the maximum swing angle and angular speed.
Learning budget and physical assessment
| Stage | Rail A | Rail B |
|---|---|---|
| Acquisition | 300 | 500 |
| Structural stages, combined | 352 | 352 |
| Refinement | 768 | 768 |
| Total learning | 1,420 | 1,620 |
Source reports record 12 refinement updates per Rail task. Individual selection, deletion and ordering counts are unavailable; the table derives their combined count by subtracting acquisition and refinement from the reported learning total. On their separate 64-case final panels, Rail A records 22 fixed-sequence and 44 contextual successes; Rail B records 19 and 63. Paired gains total 22 for Rail A and 44 for Rail B, with zero regressions in either panel.
The request u maps to a carriage target velocity of 0.6u m/s. Physics integration runs at 100 Hz between command boundaries. The suspended-load model calculates pendulum motion from prescribed carriage acceleration and omits payload reaction forces from the carriage equation.
In the selected Rail B case, the maximum carriage and payload errors over the final second are 1.53 mm and 1.41 mm respectively. Aggregate success counts were recomputed from a saved cache. Verification of the original traces at every simulation tick remains unavailable.
Passing physical state between maneuvers
The Harbor C1 harness assigns the Gate → Jump → Stop sequence. Adapt-1 learns the command values for each maneuver. Later maneuvers train from successful upstream arrivals, carrying forward the actual position and velocity along with the scoring state.
In the recorded C1 execution, each half-second command contains drive, turn, brake and jump inputs. Drive and brake can act simultaneously. Jump values at or above 0.5 count as pressed; a takeoff requires an accepted rising edge. The acquired Stop sequence includes an additional accepted takeoff.
Select a time to inspect the held controls. Readouts are rounded to three decimal places.
Frozen evaluation records 127 successful courses out of 128 for each of the selected, ordered and contextual variants. The courses contain small variations of the training geometry. All variants produce identical controls and physical traces on each course.
Maneuver learning and handoff accounting
| Maneuver | Acquisition | Selection | Deletion | Ordering | Refinement |
|---|---|---|---|---|---|
| Gate | 512 | 592 | 136 | 24 | 256 |
| Jump | 512 | 792 | 64 | 24 | 256 |
| Stop | 768 | 600 | 304 | 24 | 256 |
Run accounting records 5,120 learning-admitted maneuver executions through 1,822 native decision acknowledgements. The broader total of 6,848 unique maneuver executions includes development checks, training handoff collection and frozen evaluation.
Evaluation reuses the reserved courses across the policy variants, totaling 384 course attempts and 1,152 maneuver executions. Deletion and ordering preserve the selected sequences, and every final contextual query returns zero offsets. The experiment lacks an equal-budget comparison with a single learner controlling the complete course.
After the last Stop command, the executor applies a neutral guard. The recorded case satisfies the stopping condition throughout this interval.
Correcting speed and jump timing
Harbor v4 examines further correction practice on inherited behavior under a separate setup from the C1 experiment. The focused-alley study begins with a retained second-hurdle maneuver within supplied navigation and skill composition. Its reported training budget covers the added correction practice. The client observes incoming position and velocity at the handoff, then applies bounded offsets to speed and jump timing.
A separate mixed-strength report describes practice with takeoff speed reduced from 5.8 to 4.4 m/s and rehearsal at normal strength. The learned correction varies with starting position; the report finds little dependence on the strength input.
An earlier assisted delivery controller completes 33 of 36 fresh full courses using supplied navigation and interaction around retained components. Its full historical learning budget is unavailable. This evaluation belongs to a separate experiment from C1 and the correction studies.
Practice and reconnect conditions
Focused-alley practice records 128 training executions. On the 36 reserved starts, the parent completes 12 courses and the correction completes all 36, repairing 24 failures with zero regressions. Practice combines a narrower task with incoming-state observations, a declared correction scale and balanced discovery/confirmation groups. The design leaves their individual effects unresolved.
The mixed-strength report records 320 training executions. Corrected execution completes all 36 cases in each evaluation panel; the parent completes 24 at reduced strength and 34 at normal strength. A retained report excerpt supports these figures. The current bundle lacks the underlying trial records.
Reconnect uses a fresh local client process to query the retained policies on previously evaluated starts with feedback disabled. The focused-alley report records exact reproduction of its final corrected panel; the mixed-strength excerpt reports the same for all 72 corrected records. The checks compare returned offsets and executed actions with their recorded trajectories. Server-restart behavior remains untested.
Observation timing during execution
DOOM explores a separate acquisition setup that refreshes observations during an episode. The recording begins from saved learner state with learning active, and the initial cold-start budget is unavailable. This excerpt shows acquisition in progress; the earlier performance comparisons evaluate retained policies with learning disabled.
The configured DOOM controller receives a new observation between windows of up to eight control steps. Its continuous outputs specify forward and sideways movement along with yaw and pitch; additional outputs control firing and interaction. For YAM, the client sends the starting context once for the full attempt.
The full retake ends when the user stops the episode with navigation and interaction objectives unfinished.
Observation windows and executed prefixes
In this recording, each control step spans three game ticks, approximately 86 milliseconds of game time. Retake 18 executes 8 + 8 + 8 + 6 steps across four proposals. Combat completion stops the final window before its last two proposed steps execute.
The wider combat record contains 247 executed windows. Of these, 246 receive admitted learning updates; the remaining window is canceled because its terminal observation is unavailable. That cancellation occurs in retake 6.
What Machina adds to Adapt-1 Preview
Machina adds continuous-control sequence learning to Adapt-1 Preview. Building on the same contextual and temporal learning system, it acquires coordinated executions from its own attempts, revises them using outcome feedback, and retains them for reuse.
What it produces × how the task unfolds