arXiv preprint2026

Lasting Effects of Abstract Pretraining Beyond Perplexity

Adelaide University and Australian Institute for Machine LearningIdiap Research InstituteMetacognition AI
Zachary Shinnick1,*Hemanth Saratchandran1Damien Teney2Anton van den Hengel1,3
1 AIML, Adelaide University2 Idiap Research Institute3 Metacognition AI
01 · STRUCTURE
push Apush Bpop AAAB emptyAB on topA remains

Abstract warm-up

STACK · ~80M tokens
02 · LANGUAGE

The Moon reflects sunlight. Its gravity helps shape the ocean’s tides.

Natural language

FineWeb-Edu · 2.69–7.2B tokens
03 · REASONING
Which country does Peter Parker live in?
lives inlocated in PeterParker New York ANSWERUnited Statesof America

Multi-hop QA

Task-specific fine-tuning
Illustrative examples
The central finding

Reasoning starts before language.

A brief warm-up on a simple abstract task based on stack memory operations improves multi-hop reasoning, with gains concentrated on sequential composition. In controlled experiments, it also accelerates learning of longer reasoning chains. These effects persist after billions of language-training tokens without a corresponding improvement in language-modeling loss.

+3.9 F1

MuSiQue gain after fine-tuning
SmolLM-135M · 2.69B language tokens

~80M

Additional abstract tokens
~3% at 135M · ~1.1% at 360M

02 / A simple abstract task

Simulating a stack memory.

After a sequence of push and pop operations, the model predicts the remaining stack contents from top to bottom. The symbols have no natural-language meaning; the target depends on the sequence of state updates.

Last in, first out.

A pop removes the most recently added token. The remaining contents depend on the whole sequence.

Add your own values, then remove them. Switch rules to replay your operations as a stack or queue.

Operation history
    Memory state0 operations
    Empty memory

    Start with an empty memory.

    Schematic stack and queue operations

    The same operations produce different targets under STACK and QUEUE. Replacing the memory rule does not reproduce the STACK gains.

    01 / Beyond the language-modeling loss

    Beyond language-modeling loss

    After benchmark-specific fine-tuning, MuSiQue and HotpotQA improve at both model sizes. 2WikiMultiHopQA improves at 135M and is approximately unchanged at 360M. Compare QA F1 with language-pretraining loss below.

    Multi-hop question answering

    Random initializationSTACK warm-up
    Meanwhile, during pretraining

    Language loss does not
    capture the QA gains.

    At 135M, validation-loss curves converge to essentially the same value.

    Validation cross-entropy loss · lower is better
    03 / What transfers

    Gains concentrate on sequential composition.

    The improvements concentrate on sequential composition: answering one question supplies the information needed to answer the next.

    Consistent gains

    Dependent reasoning chains

    1234

    Each step uses the previous answer.

    No consistent gain

    Merging or comparing facts

    AB?

    Separate branches contribute to an answer.

    Performance by question structure
    Question-type F1 comparison. STACK gains appear in every chain-dominated category; merge and comparison categories show mixed effects.
    Figure 3 · Mean token-level F1 by question type. Annotations show relative differences from the language-only baseline. Gains are consistent across chain-dominated categories, with no consistent benefit for merge or comparison categories.

    Faster learning on controlled multi-hop queries.

    DEPO asks a model to follow one to four links through a directed cycle. After STACK and language pretraining, the evaluated four-layer models learn this task faster and reach near-perfect accuracy at all tested hop counts. The language-only baselines still struggle with four-hop queries at the end of fine-tuning.

    Controlled multi-hop reasoning · DEPO
    DEPO learning curves compare language-only baselines with STACK then language. STACK warm-up accelerates acquisition across all four hop counts.
    Figure 4 · A four-layer GPT-2-style model, with TinyStories or FineWeb-Edu pretraining. The horizontal axis measures downstream DEPO training tokens. This is a controlled synthetic task, separate from the SmolLM QA experiments.
    04 / When the data appears matters

    Initial exposure produces the largest gain.

    The same abstract task has a different effect depending on when it appears. An initial phase produces the largest gain among the tested single-phase schedules.

    One STACK phase at different insertion points.

    Mean Δ F1 across three QA benchmarks · SmolLM-135M

    Insertion point through language pretraining
    Before language+2.32 F1 points

    Initial STACK training gives the largest mean QA gain among the tested single-phase schedules.

    STACKLanguage

    Mean change in F1 across the three QA benchmarks at each tested insertion point (Figure 9). The initial +2.32 is the mean of the three QA gains reported above. STACK uses a separate optimizer; intermediate insertions pause language pretraining and its learning-rate schedule, which then resume.

    QA gains persist through language pretraining.

    At matched language-token counts, separately fine-tuned SmolLM-135M checkpoints retain higher QA F1 with initial STACK training. This advantage is present at every evaluated checkpoint, through the full 2.69B-token language-pretraining budget.

    Downstream performance throughout language pretraining
    MuSiQue, HotpotQA, and 2WikiMultiHopQA F1 across language-pretraining checkpoints. STACK stays above the random-initialization baseline in all three benchmarks.
    Figure 12 · Each point is evaluated after separately fine-tuning a language-pretraining checkpoint. The lines do not represent a single continuous QA fine-tuning run.
    Read, explore, build on
    PREPRINT · 2026

    Lasting Effects of Abstract Pretraining Beyond Perplexity

    Zachary Shinnick · Hemanth Saratchandran · Damien Teney · Anton van den Hengel

    Read the full paper

    Citation

    @article{shinnick2026lastingeffects,
      title   = {Lasting Effects of Abstract Pretraining Beyond Perplexity},
      author  = {Shinnick, Zachary and Saratchandran, Hemanth and Teney, Damien and van den Hengel, Anton},
      journal = {arXiv preprint arXiv:2609.38764},
      year    = {2026}
    }

    Research figure

    Open full-resolution image