Qwen2.5-0.5B: A Causal Atlas

Component Atlas: Layer-Level Ablation

Finding 1: L2 is a universal importance hub with positional specialization. High

Zero-ablating Layer 2 causes the largest KL divergence across all 12 task families (0.5–11.5 nats). Key observations:

• L2 MLP dominates — not just residual magnitude. MLP ablation confirms L2 MLP is the single most important subcomponent (max KL 11.26).
• Positional specialization: L2 routes first tokens (instruction, mean 3.34) and last tokens (prediction, mean 5.03). Operator tokens have near-zero effect (−0.09).
• L2 is NOT a uniform processing layer — it selectively processes the first and last positions, acting as an instruction-to-prediction router.

MLP-Level Ablation

Finding 2: L0 MLP and L2 MLP are the two most important MLP components. Medium

MLP ablation reveals L2 MLP has the highest effect (max KL 11.26), with L0 MLP second. This confirms L2's role is driven by its MLP subcomponent, not just residual stream magnitude.

Head-Level Ablation

Finding 3: Individual head effects are small (max KL 0.046), suggesting distributed processing. Medium

Head ablation effects are 200× smaller than layer-level effects. No single head dominates. Attention in Qwen2.5-0.5B operates through distributed head contributions rather than specialist heads.

Steering Vectors

Finding 4: L2 steering with factual direction causally boosts target token probability 3.3×. Medium

Steering L2 with a factual recall direction increases "Rome" probability from 0.064 to 0.213 for "capital of Italy". Negative steering suppresses it. However, extreme steering (s ≥ +2) causes degeneration (Chinese characters, repetition), indicating a finite steering budget.

LoRA Training Perturbation

Finding 5: Each skill concentrates in DIFFERENT layers after LoRA training. Medium

The hypothesis that training universally concentrates skills into early layers (H002) is REJECTED. Each skill family has its own concentration pattern:

• factual_recall: L3, L16, L19
• code_semantics: L1, L10, L21
• json_schema: L6, L12, L13
• copying: dispersed (no clear concentration)
• delimiter_tracking: fully absorbed (0 ablation sensitivity)

Targeted intervention must be skill-specific — there is no universal "training target" layer.

Training Dynamics

Finding 8: Core circuit (L2/L7/L9) locks in by step 10 (first 10% of training). Medium

The JSON core circuit stabilizes at step 10 and drifts <1% through step 100. Loss drops from 0.587 (step 10) to 0.062 (step 100). Secondary layers (L15, L6) continue shifting (+2.85/+2.73), suggesting a two-phase training process: rapid core circuit formation followed by secondary layer refinement.

Finding 9: Adapter norms peak at late layers (L20–L23) but ablation effects peak at early layers (L0–L2). Medium

This norm-effect separation is a key architectural finding. Training writes the largest weight changes to late layers, but the functional impact (measured by ablation) is concentrated in early layers. Effects propagate upstream — the adapter modifies late layers, but the information that matters for behavior flows through early layers.

Finding 10: Adapters can be combined with varying interference. Medium

• factual + json: synergistic (+2.35 factual, +1.17 json)
• code + json: compatible
• delimiter: destructive when stacked (−7 to −16 nats)

The delimiter adapter's extreme behavior may indicate format-specific overfitting. The clean stacking of factual + json suggests these skills occupy orthogonal subspaces.

Cross-Model Activation Transfer

Finding 12: Trained activations can partially transfer learned behavior to the base model. Medium

Cross-model patching reveals that trained model activations at specific layers can transfer learned behavior into the base model. Top transfer layers: L23 (recovery=1.000), L22 (recovery=0.966), L21 (recovery=0.947). The LoRA adapter's learned behavior is partially encoded in the activation patterns at these layers, not solely in the weight modifications.

Skill Knockout via Negative Steering

Finding 13: Negative steering can selectively suppress learned skills. Medium

For factual_recall, the best knockout was at L19 with selectivity ratio 11,654×. Negative steering at moderate strengths (−1.0 to −2.0) can suppress skill-specific tokens while preserving non-skill behavior. Higher strengths (−4.0 to −8.0) cause broader degradation. Learned skills can be selectively removed without full model retraining.

Adapter-Only Ablation: Norm vs Effect

Finding 14: Adapter norm and ablation effect are spatially separated, supporting upstream propagation. Medium

The correlation between adapter weight norm and ablation effect is 0.855, indicating a weak or negative relationship. Layers with low adapter norms but high ablation effects (upstream propagation evidence): L12. Top adapter ablation effect layers: L23 (KL=0.872), L22 (KL=0.809), L21 (KL=0.723). This supports hypothesis H6: adapter weights write to late layers but the functional effects propagate through early layers.

Layer Ablation Heatmap

The full layer ablation reveals a multi-peaked importance distribution. Layer 2 stands alone as the dominant hub (mean KL 19.11), but several secondary peaks appear:

The mean L2 ablation effect across all 12 task families is: [18.42, 21.48, 18.54, 16.50, 21.77, 17.31, 16.71, 20.68, 20.38, 19.12, 21.17, 17.23]. The highest effects occur on factual recall (21.77) and arithmetic (21.48), while the lowest is on delimiter tracking (16.50).

MLP Details

MLP ablation isolates the feedforward subcomponents within each layer. L2 MLP has the highest single-component effect (max KL 11.26), confirming that L2's dominance is driven by its MLP sublayer rather than just residual stream magnitude. L0 MLP is second. All other MLPs have substantially lower effects. This two-component dominance suggests that feedforward networks at layers 0 and 2 carry the majority of the model's representational capacity.

Head Details

Head ablation across 6 layers × 14 heads reveals that no individual head has a significant effect. The maximum single-head KL is 0.046 — roughly 200× smaller than the top layer effect (19.11) and 170× smaller than the top MLP effect (11.26). This is a strong indicator of distributed attention processing: the 14 GQA heads (with 2 KV heads) share computational load evenly rather than having specialist heads. This pattern contrasts with larger models where specific heads take on identifiable roles.

Position Specialization Summary

The model exhibits a clear positional architecture where different token positions are routed through distinct layer pathways:

This positional specialization suggests the model processes instruction tokens and prediction tokens through different pathways within the same layer stack, with operators and delimiters serving as near-transparent pass-throughs.

LoRA Rank Sweep: Precision vs Coverage

Finding 6: L0 MLP concentration peaks at r=4. Higher rank distributes rather than concentrates. Medium

Lower rank produces more localized adapters. At r=4, the adapter concentrates its effect at L0 MLP (the model's strongest component). At r=16, the effect distributes across multiple layers, diluting the L0 concentration. This has implications for efficient skill injection — r=4 may be the optimal precision/coverage tradeoff. Total adapter norm scales linearly with rank (6.14 at r=1 to 22.92 at r=16), confirming that higher rank adds proportionally more parameters without proportionally more impact.

LoRA Module Sweep: Target Module Comparison

Finding 7: o_proj is the most efficient skill injection pathway. Medium

The o_proj (output projection) writes directly to the residual stream, making it the most parameter-efficient injection point. It achieves +3.64 L0 effect with only 344K parameters (0.07% of the model). In contrast, MLP-only requires 3.3M parameters (10× more) but achieves a smaller effect (+1.92). This is because o_proj's output feeds immediately into the residual stream, while MLP changes are mediated through the feedforward computation.

Dataset Shard Ablation: Skill-Specific Patterns

Training on 5 different dataset shards and comparing the resulting component maps reveals that each skill family writes to a unique set of layers:

This is the direct rejection of hypothesis H002 ("LoRA concentrates skill into early layers"). Each skill has a unique fingerprint. Delimiter tracking is particularly notable — after training, it becomes completely absorbed into the model's baseline processing (zero ablation sensitivity), meaning the model has internalized this skill so thoroughly that removing any layer no longer affects it.

Checkpoint Timeline: Two-Phase Training

Tracking the component map across 5 checkpoints (steps 10, 25, 50, 75, 100) reveals a clear two-phase training architecture:

• Phase 1 (steps 1–10): Core circuit (L2/L7/L9) locks in. The model establishes its processing skeleton. Loss drops rapidly from 0.587 to initial levels.
• Phase 2 (steps 10–100): Secondary layers (L15, L6, and skill-specific layers) continue shifting. The model fills in task-specific details. Loss drops from 0.587 (step 10) to 0.062 (step 100).

Practical implication: Early training steps are critical for establishing the processing architecture. Short training runs (10 steps) may be sufficient for basic skill acquisition, while longer runs refine skill-specific components. This has direct implications for efficient fine-tuning budgets.

Adapter Stacking: Multi-Skill Composition

Weighted merging of adapters trained on different skills reveals which skills can coexist:

The factual + json pairing is synergistic — both skills improve when combined. This suggests they occupy orthogonal subspaces in the model's representation. The delimiter adapter is destructive when stacked, likely due to format-specific overfitting that interferes with other skills' processing patterns. Practical implication: Multi-skill models can be built by stacking compatible adapters without retraining.

Efficiency Experiments: Layer Skipping and Early Exit

Finding 15: Naive layer skipping destroys output — all layers are necessary, even "weak" ones. High (negative result)

We tested 10 layer-skip configurations, from skipping single weak layers (L15, L4, L8) to skipping 8 layers at once. Results:

Even skipping L15 alone (the weakest layer, max ablation KL of 3.37) produces KL of 9.02 and 0% top-5 overlap. The "weak" label means the layer contributes less to the KL when ablated, not that it can be safely removed. Every layer is essential for correct output.

Finding 16: Early exit at L22 (unembedding layer) does not work naively. High (negative result)

Despite L22 being the unembedding pathway (97% recovery in cross-model patching), projecting L22's hidden state through the lm_head gives:

• 0% argmax match (not a single prediction matches the full model)
• KL of 9.14 (massive divergence)
• Only L23 (full model) gives correct output

This is because each layer transforms the residual stream. L22's hidden state is the input to L23, not the final representation. The lm_head expects L23's output, and L22's output is not directly projectable to vocab.

What the Atlas DOES Tell Us About Efficiency

The efficiency experiments are negative for naive inference optimization, but the atlas reveals where efficiency gains ARE possible:

1. Training efficiency: Core circuit (L2/L7/L9) locks in by step 10 — 90% of training steps are refinement. Shorter training runs may suffice for basic skill acquisition.
2. Parameter efficiency: o_proj-only LoRA with r=4 achieves 80%+ of full-adapter effect with 344K params (0.07% of model). This is the most efficient skill injection pathway.
3. Selective skill manipulation: Negative steering at L19 selectively removes factual recall (11,654× selectivity) — no retraining needed for skill removal.
4. Adapter stacking: factual + json adapters compose cleanly — multi-skill models without retraining.
5. Targeted optimization: The atlas identifies L2 as the universal hub — optimizing L2's implementation (e.g., kernel fusion, quantization-aware training) benefits all tasks.

Negative Results

The following null and negative results are reported to avoid publication bias:

1. Full SFT OOMs on 8GB VRAM — LoRA required. Full fine-tuning may produce different internal changes.
2. Full-residual activation patching gives KL=0 everywhere — position-specific patching needed for meaningful results.
3. H002 (universal L0-L2 concentration) rejected — skill-specific patterns, not universal concentration.
4. Clean/corrupt pair v0 had tokenization misalignment — fixed in v1 with verified single-token targets.
5. Extreme steering (s ≥ +2) causes degeneration — finite steering budget; the model breaks under strong intervention.
6. L2 is NOT position-uniform — operator tokens near-zero, first+last tokens dominant.
7. H6 (upstream propagation) rejected — adapter effects are at the same layers as norms (corr=0.85), not upstream.
8. Naive layer skipping destroys output — ALL configs give 0% top-5 overlap, KL 7.9–13.8.
9. Early exit at L22 gives 0% argmax match — intermediate hidden states not directly projectable to vocab.
10. PeftModel.from_pretrained modifies base in-place — must use disable_adapter() for base behavior.

Decision Log

Key methodological decisions and their rationale:

Reproducibility

All 22 experiments are fully reproducible via the scripts in the repository: