Building Social World Models with Continual Learning
Abstract
We study the social world model: a model that predicts how a population's belief about a specific, dated event changes after news arrives. We ask whether continual learning, updating the weights on the outcomes of the model's own predictions rather than training once and freezing them, is a practical way to build one. We contribute two things: (1) a 390-day backtest with monthly rolling deployment, in which the model went through twelve rounds of self-distillation fine-tuning (SDFT) on live prediction-market contracts and was scored against a price-only baseline and against prompted frontier models, including GPT-5.6; and (2) TrajOps, the runtime we built to run this in production. It exposes three interfaces, collect, infer and train; it controls what each checkpoint is allowed to see, turns later observations into training signals, and records exactly which data each update consumed.
1
What is Social World Model (SWM)?
A world model predicts what happens next; a social world model predicts what a population believes next. In Building Social World Models with Large Language Models we make this precise. A social belief is what a crowd collectively thinks about a yes-or-no question that the future will settle, such as whether the Fed cuts rates by September. The social state is how that belief has been moving recently, and an event is a piece of news with a timestamp. The social world model learns the transition between them: given the current state and a new event, it predicts where the belief goes next. We measure the state with prediction markets, mainly Polymarket and Kalshi, where a contract trading at 62¢ means the crowd is betting real money on roughly a 62% probability.
2
How to deploy SWM in the real world?
We connect the model to a live news feed and to live prediction-market order books. At each call it sees the market state up to that point and the news that has just arrived, and it predicts which way the crowd's belief will move. Here is one sample from 14:00 UTC on 13 February 2026. The market was pricing whether the S&P 500 would touch 6,000, and the contract stood at 0.525. Before the next hour's outcome was known, the model had to judge whether the last hour's news contained anything the price had not yet absorbed. If the weights stay frozen, nothing the model sees during deployment changes what it can do tomorrow; every call is answered by the same fixed model. Figure 2 walks through this one recorded decision, from what the model saw to what the market did.
- 12:30US, Russia, Ukraine Agree to New Round of Talks in Geneva Next Week
- 12:57German Chancellor Merz: U.S. Leadership Under Question, Perhaps Squandered
- 13:04Ronshine China (03301.HK): January 2026 contracted sales at RMB 217.64 million
- 13:30U.S. January Unadjusted CPI 325.252, forecast 325.408, previous 324.054.
- 13:30U.S. January Seasonally Adjusted CPI MoM 0.2%, forecast 0.30%, previous 0.30%.
- 13:30U.S. January Real Earnings MoM 0.5%, previous revised from -0.3% to -0.5%.
3
Can we train SWM in a continual learning way?
Social cognition and market logic change over time: which news matters changes, how participants react changes, and none of it is announced in advance. A model whose weights are frozen at one point has no guarantee of fitting what comes later, so for a social world model, continual learning is a requirement of long-running deployment, not a training trick. The direct way to learn from each hour of market feedback would be to fit the weights to the price change itself, but the price change is mostly noise: it says how far the market moved, not which news mattered or why that news changed what participants believed. Instead, we have the model learn the reasoning that connects a news event to a change in belief. What we store for training is the trajectory: what was visible at the moment of the call, the call itself, and what happened afterward, each piece stamped with the time it became knowable and separated from the call by a cutoff. The training method is SDFT, self-distillation fine-tuning (Shenfeld et al.): the teacher11The teacher's extra context is a hindsight demonstration: reasoning the model writes after seeing the outcome. It is never part of the student's input. sees the part of the trajectory that had arrived by the end of the training window, the student does not, and the difference between their token probabilities is the training signal. Figure 3 shows this on a row where a ceasefire headline moved the crowd: the teacher, shown the outcome, names the headline, and the student's own sentence is rescored under it.
The relevant headline is "U.S. State Department: Israel, Lebanon Agree to Ceasefire Following U.S.-Led Talks."
hindsight reasoning — the same model, written after it saw the outcome, 0.835The assumption behind this is that the reasoning from event to crowd sentiment to market move, once trained into the weights, generalises to markets the model has not seen. It depends on the cutoff holding. If any information from after the call leaks into the student's input, the model is being trained to answer a question whose answer it was shown, and the evaluation metrics will not reveal it.
4
SWM Continual Learning Performance
We ran a 7B model through this process for twelve months of recorded calls. This section reports what continual learning improved and where it did not help.
All prompted comparisons below are scored on the same 2,145 held-out rows against the same frozen price-only baseline22The price-only baseline is a linear rule fitted once on the first window and then frozen; it predicts the next hour's change from the current price alone.. If a model's answer cannot be parsed, it is scored as the baseline, which corresponds to a skill of zero33Skill is the share of the baseline's squared error a model removes: zero matches the baseline, positive is better, negative is worse..
finding 1The trained social world model beats prompted LLM baselines.
Before training, the base model scores 4.14% worse than the baseline. One window of 461 judgments brings it to +1.70%, and the fully trained model removes 2.51% of the baseline's squared error on the permanent holdout. It is the only approach shown in Figure 4 that scores above zero; the prompted frontier models shown there score below. gpt-5.6-sol actually predicts the direction of the crowd's move more accurately than our model, but its predictions are about two and a half times larger in magnitude on a target that is mostly noise, and squared error penalises that. Our margin comes from restraint: on four rows out of five the model predicts that the news changes nothing, and on the remaining rows its prediction is closer than the baseline 55% of the time.
finding 2Continual learning wins on robustness: a higher floor than the frontier models.
Repeated training does not degrade the model's reasoning: across all twelve rounds its answers stay well above the 236-token median of the training demonstrations, and the only side effect we found is that the answer format sometimes appears in unrelated prompts. The monthly breakdown matters more. In two months, w01 and w1144The 390-day run is split into twelve monthly windows, numbered w00 through w11., both gpt-5.6-sol and deepseek-v4-pro lose heavily, because both make predictions two to three times farther from the baseline than ours; in a normal month that costs them a little, and in those two months it costs them badly. Our model is positive in both of those months, and its monthly scores stay within a band about one fifth as wide as deepseek-v4-pro's.
- ours−0.033
- gpt-5.6-sol−0.062
- deepseek-v4-pro−0.140
- claude-opus-5−0.537
| Model | Partition skill range | Worst | Mean | Best |
|---|---|---|---|---|
| ours | −0.047 | 0.004 | 0.080 | |
| gpt-5.6-sol | −0.458 | −0.110 | 0.024 | |
| deepseek-v4-pro | −0.607 | −0.116 | 0.069 | |
| claude-opus-5 | < −0.700 | −0.349 | 0.150 |
finding 3Frozen checkpoints provide a control for evaluating updates.
At each round we froze a copy of the model and kept scoring it on later windows to assess how its predictions held up after training stopped. The table reports these checkpoints alongside the monthly retraining strategy and the permanent-holdout evaluation. Each result must be read with its evaluation scope: the rows cover different sets of predictions, so differences between their aggregate scores do not directly measure the benefit of additional updates. Rank correlation and bucket accuracy55The model answers in one of nine buckets for how far the next hour lands above or below the baseline; bucket accuracy is how often it picks the right one. provide additional measures of prediction quality across training rounds.
| model version | training rows | skill | scored on |
|---|---|---|---|
| base model, no training | 0 | −4.14% | all later months |
| checkpoint after month 1 (r00), then frozen | 461 | +1.70% | months w01–w11 · no decay |
| checkpoint after month 12 (r11), then frozen | 4,839 | +1.94% | months after training |
| monthly retraining, always the newest checkpoint | grows each month | +2.70% | 6,323 forward predictions |
| final model on the permanent holdout | — | +2.51% | 2,145 rows never trained on (as in Figure 4) |
5
SWM Continual Learning Infrastructure
Deploying an LLM on prediction markets is one instance of a more general problem. Any team that wants a model to keep learning from real-world data after it leaves the lab runs into the same three engineering questions.
- What is knowable at the moment of judgment? When the model makes a prediction, how do you guarantee it sees only information that was available at that time, with no future data leaking into its context?
- When does the outcome become knowable? Real-world feedback arrives in fragments and with delay, and the system has to decide exactly when to collect the result that the prediction will be graded against.
- Did the update help? The model rewrites its own weights on new data every day. How do you confirm it is improving, rather than catastrophically forgetting66Catastrophic forgetting: training on new data erases abilities the model already had. or simply memorising?
We built TrajOps to handle these three problems: a standardised MLOps engine designed for continual learning. Its interface is small. It exposes three operations, collect, inference and train, and implements a standard SDFT recipe in the underlying workflow.
The three operations cover the whole cycle. Collect retrieves the news around an event and, once it is available, the real market outcome. Inference serves the deployed model and enforces the time horizon, so a prediction can only see data from before its cutoff. Train compiles what collect gathered into training signals through the built-in SDFT recipe and updates the model weights.
The 7B model is still running on this infrastructure, making live market predictions every day. The record is published at trajops.astraculum.com.