// Forecasting From the First Outcome
Adapt-1 keeps each forecast until its outcome arrives. The score of that forecast then changes how the next forecast is built.
Most forecasting fits a model, issues forecasts, and sees outcomes again only in a backtest or the next retrain. With Adapt-1, we test a different approach to forecasting. Adapt-1 keeps each forecast in its learned state, together with the information that was available when the forecast was issued. When the outcome arrives, Adapt-1 scores the forecast before it learns from that outcome. The score then changes how the next forecast is built.
The learned state of a task starts empty, as our earlier article describes. Every forecast is issued before its outcome is known.
How a forecast is built
A forecast in Adapt-1 is a set of atoms. Each atom is one possible joint outcome with a probability. An atom can hold several targets, for example the next five yearly values of a series. The targets of an atom stay together, so the forecast keeps outcomes that occurred in the same case. Means, quantiles and class probabilities are summaries of this joint set.
The construction keeps probability weights apart from atom locations. The support and the weight w set the probabilities. The recorded outcomes set the locations, unless a validated level exists for the target.
Support
Adapt-1 first finds the retained cases that match the query. A retained case is a completed observation: its inputs and the outcome that followed them. A matching case has the same partition and action as the query. The distance test uses only the inputs that are present in the query. The cases that pass the test form the support. The support has a size limit, which is separate from the number of cases that Adapt-1 retains for learning.
If the support is too small, Adapt-1 abstains and gives the reason.
Atoms and the weight
Each case in the support gives one atom. The probability of the atom mixes two sources. The kernel source gives a larger share to cases that are closer to the query. The prior source does not depend on the query. The weight w sets the mix of the two sources.
| Conditioning mode | Contribution to the forecast | What changes with outcomes |
|---|---|---|
prior | Unconditioned distribution F0 | The retained cases |
kernel | Distribution conditioned on the query, FK | The retained cases, with shares set by their distance to the query |
adaptive | Mix of F0 and FK with the weight w | The retained cases and the weight from the stored scores |
Section 03 shows how w comes from the scores of earlier forecasts, so no one sets it by hand.
Level and spread
A recorded outcome has a fixed level. If the level of a numeric target changes with the inputs, recorded outcomes can be too high or too low for the current query. When the numeric support is large enough, Adapt-1 can fit a regularized linear model on the matching cases. The model gives the level for the current inputs. Each case in the support keeps its own residual, so the cases set the spread around that level. We call this step numeric transport.
Adapt-1 uses the model only after validation. It fits the model on earlier stored outcomes and tests it on later ones. Each target is checked separately. A forecast of five yearly values can use transport at some horizons and recorded outcomes at others. The result of the check can change as Adapt-1 retains more cases.
A leave-one-out residual is the difference between the outcome of a case and a fit that does not use that case. For an eligible target, the level for the query replaces the level of the case. The residual stays with the case. The residuals of one case also stay together across targets. An atom can thus have a level above or below every retained outcome, and it keeps the joint deviations of its case.
Kept, then scored
Each issued forecast has a receipt. The receipt holds the decision identifier, the atoms, the weight, the scales of the targets and a hash of the inputs. When the outcome arrives, the outcome event sends the inputs again with the decision identifier. Adapt-1 uses the identifier to find the receipt.
Adapt-1 computes the score from the receipt, as the forecast was issued. It does this before the outcome enters learning. This order, prediction before update, is prequential evaluation. For a numeric joint forecast, the energy score evaluates the atom locations and the probabilities together:
A query can receive more than one forecast for the same outcome. In the tested case, the forecasts used fewer and more inputs at the same issue time. Each forecast has its own receipt, and the outcome scores each one separately. Revisions with evidence that arrives after the issue time were not tested.
How the score changes the next forecast
The weight
Adapt-1 stores each score with the conditioning group g of its forecast. Each group thus has its own scores, and the weight can differ between masks of available inputs and between actions. The weight of a group comes from its last 256 stored scores:
As a function of w, the score of a mixed forecast is a quadratic. The rule gives the w that minimizes the sum of the stored scores plus the term (w − 0.5)2. With few scores, this term keeps w near 0.5. As scores accumulate, w moves toward the mix that would have scored best on them. Section 06 shows what this gives across tasks.
The retained case
The resolved outcome also becomes a new retained case. Later queries can use it as support, and it enters the data of the linear model. Validation uses the stored outcomes, so a target can pass or fail validation as cases accumulate. The use of transport can thus change during a run, as section 05 shows.
From the first outcome
One stream shows these steps in order. It is the wind series of an online forecasting benchmark. The series gives the hourly wind generation on the ERCOT grid in Texas, from 2 January to 12 February 2022.
Each forecast is for the next hour. Its input is the last 24 values, the lag that the benchmark paper sets. The outcome returns before the next forecast. The learned state of Adapt-1 starts empty, and the run covers 1,000 forecasts.
The start-up
For one numeric target, the energy score of equation (6) is the continuous ranked probability score (CRPS).
| Steps | What happens | Measured |
|---|---|---|
| 0–4 | No retained cases. Adapt-1 abstains. | 5 abstentions |
| 5–49 | Forecasts come from the retained outcomes. The learned weight sets the mix with the prior. | CRPS 26.5% lower than with a fixed w of 0.5 (95% interval: 6.7% to 60.6% lower) |
| 50 | The numeric support is sufficient. The validated linear model starts to set the level and stays on. | Turned on by the support size; validated skill 0.74 |
The gain in steps 5 to 49 comes from the learned weight, before the linear model is on. From step 50, the linear model sets the level and the cases in the support set the spread. A repeat of the run gave identical forecasts on all 450 shared steps, so the process is deterministic.
Compared with online learners
The benchmark paper reports online learners that start from scratch and learn from each outcome, as Adapt-1 does. They use the same 24 lagged values as input. These learners are the main comparison. The paper also applies two recalibration methods to each learner: isotonic regression and ORCA, the method of the paper.
We use the two measures of the benchmark. The symmetric mean absolute percentage error (sMAPE) measures the error of the mean forecast. The quantile calibration error (QCE) measures calibration. Over the quantile levels q, it adds the squared difference between q and the share of outcomes at or below the forecast quantile. For both, lower is better.
| Forecaster | QCE, 99 levels | QCE, nine deciles | sMAPE |
|---|---|---|---|
| Online learners, five lowest QCE values | |||
| Stochastic gradient trees + ORCA7 | 0.013 | – | 0.405 |
| Adapt-1 | 0.0147 | 0.0016 | 0.0856 |
| Hoeffding tree + ORCA7 | 0.016 | – | 0.431 |
| Marginal forecaster + ORCA7 | 0.017 | – | 0.391 |
| Neural network + ORCA7 | 0.083 | – | 0.397 |
| References, same 1,000 targets | |||
| TimesFM 3, pretrained, no updates | 0.110* | 0.0082 | 0.0870 |
| Persistence | – | – | 0.103 |
| Perfectly calibrated, simulated: median / 95th percentile | 0.012 / 0.046 | 0.0015 / 0.0031 | – |
The paper reports four base forecasters on the wind series, each without recalibration, with isotonic recalibration and with ORCA. The table keeps the five lowest QCE values, Adapt-1 included. The other eight rows of the paper have a QCE of 0.094 or more.
* TimesFM 3 outputs nine deciles. For the 99-level value, a fixed rule builds a full distribution from them: piecewise-linear quantiles, linear tails and a floor at zero. We set the rule before we computed the scores. The nine-decile column uses the native outputs.
Three learners with ORCA reach a QCE of 0.013, 0.016 and 0.017. Adapt-1 reaches 0.0147 without a recalibration layer. These runs did not test calibration learning.
A perfectly calibrated forecaster also has a QCE above zero, because of sampling noise. Its row gives the median and the 95th percentile of 4,000 simulated runs of 1,000 steps. The four values are all below that 95th percentile, so at 1,000 steps we cannot rank them against each other. Errors on hourly wind are autocorrelated, so the true ranges are wider than the simulated ones.
The sMAPE values of the paper are much higher than ours. Persistence scores 0.103 on our slice, lower than every sMAPE in the table of the paper. The slice or the computation of the measure can cause this difference, so we do not compare point error with the paper. For point error, persistence and TimesFM 3 on the same targets are the useful references.
Over the run
We also ran TimesFM 3 on the same 1,000 targets. It is a pretrained model, used zero-shot with fixed weights. Like Adapt-1 and the online learners, it received the 24 lagged values that the benchmark paper sets as input. It does not learn from the outcomes of the stream, so it shows what a fixed model does on this slice. A longer context can give it more of the past outcomes, and we did not test that configuration. The benchmark paper reports only totals, so Figure 4 shows TimesFM 3 and persistence as the references.
Error of the mean forecast
Coverage of the 80% interval
Over all 1,000 steps, the error of the two models is about equal. The sMAPE is 0.0856 for Adapt-1 and 0.0870 for TimesFM 3, and the 95% interval of the paired difference includes zero.
In the first 250 steps, the learned state of Adapt-1 fills from empty. In these steps, TimesFM 3 has the lower error, 0.070 against 0.106, and persistence scores 0.082. From step 250, Adapt-1 has the lower error in every block. Over steps 250 to 1,000, its sMAPE is 0.079 against 0.093. The 95% interval of the paired difference runs from −0.020 to −0.008. The intervals come from a paired block bootstrap with 24-hour blocks.
Relative to persistence, the error of TimesFM 3 shows no trend across the blocks. The error of Adapt-1 falls as outcomes accumulate. Relative to both references, it is stable after about step 750.
Calibration shows a larger difference. The 80% interval of TimesFM 3 covers between 72.8% and 74.4% of the outcomes in every block. Its intervals are too narrow on this series. The 80% interval of Adapt-1 covers 66.4% of the outcomes in the first 250 steps and 79.7% after them.
On the nine deciles, the QCE of TimesFM 3 is more than twice the 95th percentile of a perfectly calibrated forecaster. The QCE of Adapt-1 is near the median. The coverage of TimesFM 3 is below 75% in all four blocks.
The results come from one 1,000-step slice of the 2022 series. The scores of the five abstentions of Adapt-1 use fill values, zero at step 0. Without these five steps, its sMAPE over the stream is 0.084.
ETTh1 and ETTm1
We also ran a short online test on ETTh1 and ETTm1, two series of transformer oil temperature. Each forecast is for the next step: one hour on ETTh1 and 15 minutes on ETTm1. The test covers the seven days from 25 October to 1 November 2017, with 168 and 672 forecasts. Before the test, each run received 256 earlier transitions, and it kept a rolling window of 256 retained cases. We selected the input representation of each dataset on earlier validation dates and then locked it. The protocol differs from the long-horizon protocol of the ELF paper.
| Forecaster | ETTh1 MAE | ETTh1 CRPS | ETTm1 MAE | ETTm1 CRPS |
|---|---|---|---|---|
| Adapt-1 | 0.4375 | 0.3313 | 0.2196 | 0.1645 |
| Online ridge | 0.4336 | 0.3277 | 0.2172 | 0.1643 |
| Persistence | 0.4648 | 0.3556 | 0.2134 | 0.1662 |
| TimesFM 3 | 0.4123 | – | 0.2113 | – |
| Seasonal naive | 1.4313 | – | 1.4153 | – |
MAE uses the point forecast. For online ridge and persistence, the CRPS uses the point forecast plus the distribution of its past errors. We ran TimesFM 3 separately on the same targets, and the table gives only its point error.
Adapt-1 and online ridge are about equal. For both measures and both datasets, the 95% interval of the paired difference includes zero. The intervals come from a daily-block bootstrap over the seven days. TimesFM 3 has the lowest MAE on both datasets. On ETTm1, persistence also has a lower MAE than Adapt-1.
Seasonal naive repeats the value of one season earlier. Its error is about three times that of persistence, so it is a weak reference for forecasts one step ahead.
On ETTh1, w stayed between 0.42 and 0.69 and ended at 0.69. On ETTm1, it rose faster and first reached 1.0 on the fourth day. At 1.0, the forecast comes only from the kernel source. After that, w moved between 0.72 and 1.0, and it ended at 1.0.
Matched runs with a fixed prior or a fixed kernel gave almost the same CRPS as the learned weight on both datasets. Section 06 shows how the effect of the weight depends on the task.
Transport was on for all 168 ETTh1 forecasts. On ETTm1 it was on for 294 of the 672 forecasts and switched 22 times as validation passed and failed. A matched run without transport had a CRPS of 1.0118 on ETTh1, against 0.3313 with it. On ETTm1, the two were about equal: 0.1637 without transport and 0.1645 with it. The effect of transport thus depends on the task.
One learned state for 31 series
A US consumption run shows the same process across 31 series and ten forecast windows. Each query asks for the next five yearly values together. The inputs are the four-year and sixteen-year means minus the last value, and the last annual change. Each input is expressed relative to the scale of its series. Cases from different series are thus comparable, and one learned state serves all 31 series.
Before the first forecast window, the learned state held the completed historical paths of each series: 248 paths, 8 per series. Each window then asked one query per series. After that, a path entered the learned state only when all five target years were known. Between windows, 155 completed paths entered the state, and it held 1,643 paths after the nine updates. The support of a query has a limit of 256 cases.
For the first clothing-and-footwear query, the nearest cases included household appliances, recreational vehicles and luggage. In this run, one of the 310 forecasts was filled with a seasonal naive forecast.
In the first window, horizons three and four used numeric transport. The other horizons kept recorded outcomes. In the final window, all five horizons used transport. A target used transport exactly when its validated skill was above zero.
The count changes in both directions. It fell to zero in the second window and from four to one in the fifth. From the seventh window, all five horizons used transport.
What the weight does across tasks
The weight sets, for each task, how much the forecast trusts the matching cases against the prior. No one tunes it, and its contribution depends on the task. We checked its behavior on 41 runs. The runs in this article are part of them. In the wind run and the other runs we checked, equation (7) applied to the stored scores gives every stored weight.
The gap between the prior and the kernel
The weight chooses between two fixed forecasts: the prior alone, with w = 0, and the kernel alone, with w = 1. We call the difference between their scores the gap. Over a fixed w of 0.5, a fixed weight can gain up to half of this gap.
Across the 41 runs, the gain of the learned weight over a fixed w of 0.5 follows the gap. The rank correlation is +0.91. When the gap and the gain come from different halves of a run, the correlation is +0.6. Thus the relation does not come only from the use of the same forecasts for both values.
In the 13 runs with a large gap, the learned weight lowered the score by 3.5% to 35%. In the 19 runs with no gap, there is no better source to choose, and the difference was at most 0.58%. Across all 41 runs, the gain ranged from −1.0% to 35%. Over a whole run, the learned weight scored within 0.01% to 2.3% of the best fixed weight chosen in hindsight.
The linear model and the target type
When the linear model sets the level, each atom is the level for the current query plus the residual of its case. The two sources then differ only in the probabilities they give to the residuals, so they score alike. Where Adapt-1 rarely used the linear model, the median gain was 7.6%. Where it almost always used the linear model, the median gain was 0.06%. On ETTh1, two runs that differ only in the linear model gave a gain of 10.0% with it off and 0.02% with it on.
The type of target also matters. Class targets had the largest median gain, 6.7%. Level targets had 2.9%, and targets that are a change from the last value had 0.06%.
The gain over a run
The time of the gain depends on the task. Some runs start empty and turn on the linear model later. The wind run is one of them. In these runs, 45% or more of the gain comes in the first 50 forecasts, and 80% of it by forecast 21 to 75. After that, the linear model sets the level.
Runs without the linear model, such as ETTh1 with the linear model off, gain steadily. Their gain continues to increase. Of their gain, 5% to 21% comes in the first 50 forecasts. In 7 of 7 of these runs, the second half gains more than the first. Runs that start with a stored record can gain later, in one step, for example in one rise around forecasts 30 to 90.
Looking ahead
The forecasts in this article are scored on their own. In practice, a forecast usually feeds a decision, for example the dispatch of a power grid or the size of an order. In Adapt-1, the forecast distribution is part of the learned state, so the same distribution can serve the forecast and the decision.
Our next work connects these forecasts to substrate planning, where Adapt-1 compares actions over possible outcomes. That comparison combines the expected utility of an action with the utility of its worst outcomes, so the tails of each forecast enter the decision.
The forecasting update in this article will be available later in November 2026. Anyone can apply to test it before that date. To apply, write to [email protected].