Learning Procedural Core
Our recurrent model has three unique layers and predicts masked closing tokens in bracket sequences, with a looped middle layer.
A Compact Recurrent Initialization for Vision Transformers



Learning a core set of weights
One small recurrent model learns a reusable initialization for ViTs of different widths and depths. Motion is schematic.
Compact core → ViT-Base
ImageNet-1K top-1 accuracy
ImageNet-S segmentation mAP
Warming up a ViT on non-visual procedurally generated data can improve subsequent learning, but a separate warm-up is needed for each target model (Shinnick et al., CVPR 2026). Procedural Core captures this learned structure in a compact set of recurrent weights. In our main setting, a 1-million-parameter core initializes an 85-million-parameter ViT-Base and improves final ImageNet-1K top-1 accuracy from 77.6% to 79.8%. The same set of weights initializes ViTs of different widths and depths, transferring the benefits of procedural warm-up without warming up each target model separately.
01 / THE IDEA
A small recurrent model first learns on non-visual procedural data. We then expand its learned weights in width and depth to initialize a larger ViT, whose layers train independently on images.
Our recurrent model has three unique layers and predicts masked closing tokens in bracket sequences, with a looped middle layer.
Tile and rescale matrices to match the target width. Copy the recurrent middle block across the target’s middle layers, keeping distinct first and last blocks.
The target layers are untied and train independently during standard image training. The core provides the initialisation.
02 / THE EVIDENCE
We compare ImageNet classification accuracy across initializations, then evaluate spatial tasks with the trained backbones frozen. DINO uses the same initialisation as the supervised vision experiments. Language and code experiments use a separately trained GPT-style recurrent model matched to the target architecture.
mIoU · higher is better
mAP · higher is better
CorLoc · higher is better
RMSE (meters) · lower is better
Weighted k-NN accuracy (%) · higher is better
Training epoch
NATURAL LANGUAGE
Validation perplexity · lower is better
CODE
Validation perplexity · lower is better
03 / INSIDE THE MODEL
Attention sinks are tokens that attract a disproportionate share of attention. In the trained ViTs, high-norm patch tokens show this behavior. Procedural Core reduces both these outliers and the attention they receive.
Default → Core. Peak across blocks on 200 ImageNet-1K validation images.
High-norm: above 3× the per-image median. Attention is averaged over heads, patch queries, and images; the class token is excluded. Figure 9(d) ↗

Golf flag · Layer 12 of 12
These heatmaps show hidden-state token norms, not attention weights. Each is independently normalized per image, model, and layer (0–1), so colors show spatial patterns rather than absolute norm differences.
Shuffling value/output weights before image training restores default-like attention to high-norm tokens and removes much of the transfer benefit. Shuffling query/key weights largely preserves the reduced attention to outliers and the transfer gains (Figure 10).
Read the mechanistic analysis04 / KEEP EXPLORING
@article{shinnick2026proceduralcore,
title = {Procedural Core: A Compact Recurrent Initialization for Vision Transformers},
author = {Shinnick, Zachary and Internò, Christian and Saratchandran, Hemanth and van den Hengel, Anton and Teney, Damien},
journal = {arXiv preprint arXiv:2609.37631},
year = {2026}
}