Procedural Core
ARXIV PREPRINT2026

Procedural Core

A Compact Recurrent Initialization for Vision Transformers

Adelaide University and Australian Institute for Machine LearningBielefeld UniversityMetacognition AIIdiap Research Institute

Zachary Shinnick1 Christian Internò2 Hemanth Saratchandran1 Anton van den Hengel1,3 Damien Teney4

1 AIML, Adelaide University2 Bielefeld University3 Metacognition AI4 Idiap Research Institute

A compact, reusable initialization.

Learning a core set of weights

Non-visual procedurally generated data
( [ { } ] )
{ ( [ ] ) }
[ { ( ) } ]
( { [ ] } )
{ [ ( ) ] }
COMPACT CORE
First block
Shared middle block
Last block
~1M parameters · 3 unique blocks

One small recurrent model learns a reusable initialization for ViTs of different widths and depths. Motion is schematic.

1 million parameters → 85 million parameters

Compact core → ViT-Base

+2.2pp

ImageNet-1K top-1 accuracy

+10.6pts

ImageNet-S segmentation mAP

A reusable starting point

Warming up a ViT on non-visual procedurally generated data can improve subsequent learning, but a separate warm-up is needed for each target model (Shinnick et al., CVPR 2026). Procedural Core captures this learned structure in a compact set of recurrent weights. In our main setting, a 1-million-parameter core initializes an 85-million-parameter ViT-Base and improves final ImageNet-1K top-1 accuracy from 77.6% to 79.8%. The same set of weights initializes ViTs of different widths and depths, transferring the benefits of procedural warm-up without warming up each target model separately.

01 / THE IDEA

Recurrence learns compact weights that can be expanded

A small recurrent model first learns on non-visual procedural data. We then expand its learned weights in width and depth to initialize a larger ViT, whose layers train independently on images.

01

Learning Procedural Core

Our recurrent model has three unique layers and predicts masked closing tokens in bracket sequences, with a looped middle layer.

02

Expand the weights.

Tile and rescale matrices to match the target width. Copy the recurrent middle block across the target’s middle layers, keeping distinct first and last blocks.

03

Image training

The target layers are untied and train independently during standard image training. The core provides the initialisation.

02 / THE EVIDENCE

Performance of Procedural Core

We compare ImageNet classification accuracy across initializations, then evaluate spatial tasks with the trained backbones frozen. DINO uses the same initialisation as the supervised vision experiments. Language and code experiments use a separately trained GPT-style recurrent model matched to the target architecture.

ImageNet classification

ImageNet top-1 accuracy by initializationDefault 77.6 plus or minus 0.2; Mimetic 79.5 plus or minus 0.7; Procedural warm-up 79.4 plus or minus 0.3; Procedural Core 79.8 plus or minus 0.6 percent. Mean and standard deviation over three seeds.

03 / INSIDE THE MODEL

Reducing attention sinks

Attention sinks are tokens that attract a disproportionate share of attention. In the trained ViTs, high-norm patch tokens show this behavior. Procedural Core reduces both these outliers and the attention they receive.

PEAK ATTENTION MASS TO HIGH-NORM TOKENS
50.4%→17.7%

Default → Core. Peak across blocks on 200 ImageNet-1K validation images.

High-norm: above 3× the per-image median. Attention is averaged over heads, patch queries, and images; the class token is excluded. Figure 9(d) ↗

Input image
Golf flag example used in the token-norm experiment
Default initialization
Procedural Core
Early layersDeep layers

Golf flag · Layer 12 of 12

LowHigh norm

These heatmaps show hidden-state token norms, not attention weights. Each is independently normalized per image, model, and layer (0–1), so colors show spatial patterns rather than absolute norm differences.

The value/output pathway matters

Shuffling value/output weights before image training restores default-like attention to high-norm tokens and removes much of the transfer benefit. Shuffling query/key weights largely preserves the reduced attention to outliers and the transfer gains (Figure 10).

Read the mechanistic analysis

04 / KEEP EXPLORING

PREPRINT · 2026

Procedural Core: A Compact Recurrent Initialization for Vision Transformers

Zachary Shinnick · Christian Internò · Hemanth Saratchandran · Anton van den Hengel · Damien Teney

Read the full paper

Cite this work

@article{shinnick2026proceduralcore,
  title   = {Procedural Core: A Compact Recurrent Initialization for Vision Transformers},
  author  = {Shinnick, Zachary and Internò, Christian and Saratchandran, Hemanth and van den Hengel, Anton and Teney, Damien},
  journal = {arXiv preprint arXiv:2609.37631},
  year    = {2026}
}