Skip to content
Herman Brunborg@hermanbrunborg · Oct 28, 2026

I improved NanoGPT Speedrun from 39.9s → 6.7s (-83.2%), more than every record since 2025 combined (-81.4%)! The web…

5 tweets1 min read♥ 2Koriginal

Summary

A developer improved NanoGPT Speedrun performance from 39.9s to 6.7s (83.2% faster) using retrieval-augmented n-gram hints that provide the model with contextual tokens during training. The technique leverages idle CPU resources with optimized SIMD code to build a hint corpus in parallel with GPU training, achieving 500x speedup over Python implementations.

Summarized by ThreadOut AI from the full thread. May miss nuance — read the thread below.

  1. #1

    I improved NanoGPT Speedrun from 39.9s → 6.7s (-83.2%), more than every record since 2025 combined (-81.4%)! The web is full of rare, repeated content that the LM can’t memorize in time. I solve this with retrieval-augmented n-gram hints🧵

  2. #2

    I build a hint corpus of the tokens that follow each n-gram in the training data. The model receives these tokens as hints and learns how to use them.

  3. #3

    Scaling the hint corpus has accelerating returns. On a fixed model checkpoint, each doubling cuts the loss more than the last. I build the hint corpus in the timed run, so a faster builder means a larger corpus.

  4. #4

    The CPUs used to be idle, so I build the corpus on them, in parallel with model training on the GPUs. My build is branch-free SIMD code that fits in L2 cache and uses a custom data structure. It is 500x faster than Python and 11x faster than the best auto-researched Rust code.

  5. #5