Frontier model training methodologies
Alex Wa’s Blog · January 2026
Reading summary
This post argues that the hardest frontier-model gains come from disciplined systems engineering rather than isolated novelty. It highlights that data-mixture quality and stage-wise scheduling often matter more than one-off architecture swaps, while long-context success depends on infrastructural choices such as masking and positional strategy. In post-training, it surfaces subtle optimization pathologies like in-flight off-policy drift and reward hacking in hybrid reasoning modes. The most practical takeaway is operational: many critical failures originate in storage/dataloader behavior, so rigorous ablation design and pipeline observability are first-class research tools.
Notes in progress
The detailed notes and figures are still a draft.
Writing outline
1. Thesis
[Write the central argument in 3-5 sentences.]
[What was non-obvious or surprising?]
2. Key Technical Claims
[Claim 1 + why it matters.]
[Claim 2 + dependency or assumption.]
[Claim 3 + practical implication.]
3. Evidence and Methodology Quality
[Assess strength of evidence, ablations, and possible confounders.]
[What was well-supported?]
[What felt under-validated?]
[What additional experiment would change your confidence most?]
4. Failure Modes / Open Risks
[Failure mode 1.]
[Failure mode 2.]
[Failure mode 3.]
5. Transfer to Your Work
[How this changes your current evaluation or safety pipeline.]
[One concrete action item for next week.]
6. Questions to Revisit
[Question 1.]
[Question 2.]
[Question 3.]