GOTOken: An Inference Engine in QBasic

I built an inference engine for SmolLM2-135M in QBasic, the first language I learned. It runs at about one token per second. Here’s how it works and what I learned along the way.

Why BASIC?

My first line of code was a cheat. My father, a civil engineer by training, opened QBasic and loaded NIBBLES.BAS to keep me busy while he worked. It’s a snake game, for anyone who missed that particular era of computing. Somewhere in the code there’s a factor that makes the snake faster with each level. I turned it down so I could get through the later levels. After that, I started typing programs from books.

Yes, I still do have my GW Basic book in my shelf.

Decades later, I wanted to revisit QBasic and see whether I could use it to understand something I work with today: language model inference. I know the production side of it. I’ve contributed to vLLM, and at Poolside I work on Atlas, our inference engine built on top of vLLM. Work at that level is about batching, scheduling, and kernels, several layers above the arithmetic. I wanted to see the arithmetic. GOTOken runs SmolLM2-135M, a small model with the Llama architecture. It’s about 1,500 lines of BASIC, compiled with QB64, and reads config.json, model.safetensors, and tokenizer.json directly. It matches HuggingFace’s output in the comparisons below. You don’t need Python to run it; even the JSON parser is written in BASIC.

In BASIC, I had to work through the operations behind each component. Matrix multiplication was two nested FOR loops. The KV cache took me longer to understand.

I used two references throughout the project. Andrej Karpathy’s llama2.c runs a Llama model in a single C file. The BASIC implementation follows its run.c closely, with pointers replaced by integer offsets into one flat SINGLE array. Sebastian Raschka’s Build a Large Language Model (From Scratch) walks through the architecture one component at a time. I followed that approach, checking each piece before adding the next.

Building it one piece at a time

I wanted to understand how inference works, and coding agents implemented a lot of the details. We built the engine in eight steps, checking each one before moving on.

For each step, a Python script compared the BASIC output with a reference calculation or the HuggingFace model. I call these scripts the oracles below. That gave us something concrete to check, whether we were loading weights, implementing one layer, or generating tokens. The table shows what we added and how we checked it.

Table 1. The eight steps and what the oracle compared after each one

In step 3, I discovered that matching PyTorch bit for bit was the wrong thing to check. The oracle replayed my BASIC loop in fp32, with the same rounding order, and matched all 576 outputs of the query projection bit for bit. A replay using fused multiply-add matched 243 of them. PyTorch’s own matmul, through BLAS, matched 61. The implementations rounded differently, so I needed to check the size of the differences and whether they affected the generated tokens.

In step 5, the oracle printed the gap between the highest and second-highest logit for every generated token. The smallest gap was 0.19, while the largest drift from the reference was 1.7e-5. The differences were small enough that they didn’t change which token greedy decoding picked in this test.

About one token per second

Most of the arithmetic happens in this function:

' xout(d) = W(d, n) * x(n), W stored row-major at w(woff).
' run.c's matmul(xout, x, w, n, d), float* replaced by an offset.
SUB MatMul (xout() AS SINGLE, x() AS SINGLE, woff AS LONG, _
            n AS LONG, d AS LONG)
    DIM i AS LONG, j AS LONG, rowStart AS LONG, acc AS SINGLE
    FOR i = 0 TO d - 1
        rowStart = woff + i * n
        acc = 0
        FOR j = 0 TO n - 1
            acc = acc + w(rowStart + j) * x(j)
        NEXT
        xout(i) = acc
    NEXT
END SUB

Every projection in every layer, and the final multiplication against the vocabulary, goes through it. SmolLM2-135M has 134.5 million weights, and one forward pass touches nearly all of them once. That’s about 134 million multiply-adds per token. QB64 compiles to C++, and I passed no optimization flags.

In my measurements, the engine took four seconds to load and then generated about one token per second, using 1.1 GB of memory. The checkpoint is 270 MB with bf16 weights; the engine converts those to fp32 when loading them.

The KV cache was the part I understood least going in. Without it, the engine processes the whole prompt and the tokens generated so far every time it needs another token. For a five-token prompt and eight new tokens, that meant 68 forward passes and 65.3 seconds.

The key and value rows for position t depend only on tokens 0 to t. Those tokens don’t change as generation continues, so we can keep the rows and reuse them. The engine processes the prompt once, then only the new token at each step. For the same example, that brought the total down to 12 forward passes and 11.5 seconds. The output was bit-identical to the uncached run, which gave me confidence that the cache was working correctly.

There is no batching, SIMD, or quantization. The context length is a constant in the source, set to 1,024 tokens.

Loading the model and tokenizer

Reading the HuggingFace files directly from BASIC also meant dealing with a few details I’d previously left to libraries.

The checkpoint format. A safetensors file starts with an 8-byte header length, followed by a JSON header mapping tensor names to offsets, then raw bytes. So the engine needed a JSON parser. That added 250 lines of BASIC; it reads the 2 MB tokenizer.json in 70 ms. The weights are bf16, which stores the top 16 bits of an fp32 value. Converting each weight to fp32 means shifting those bits left by 16.

The rotary convention. HuggingFace applies rotary position embeddings to the pair (j, j + head_dim/2) within a head. llama2.c rotates adjacent pairs. To account for that difference, the loader reorders the rows of the query and key projections as it converts them. This lets the BASIC RoPE implementation follow run.c. Applying the same permutation to both sides preserves the dot products used for attention scores.

The tokenizer. The vocabulary in tokenizer.json is written in GPT-2’s byte-level alphabet, where a space shows up as Ġ, so the loader has to map it back to raw bytes. QB64 has no regular expressions, so the GPT-2 pre-tokenizer pattern became a hand-written scanner with Unicode letter, digit, and whitespace tables. The result matches HuggingFace on 2,060 test strings, CJK included.

Try It

docker run -it bmuskalla/gotoken

The image is 370 MB, built for amd64 and arm64, and holds only the binary and the model. Type a prompt and wait for the continuation. /temp 0 makes it greedy and deterministic. At 0.7 you’ll get different continuations; above 1.2, the output fell apart in my experiments. This is a base model, so give it some text to continue rather than expecting a chat response. The source and oracles are on GitHub.

What I learned

The KV cache and floating-point comparisons were the parts I learned the most from. I understand them better now that I’ve had to work through an implementation and check the results.

There’s plenty left to explore: batching, faster kernels, quantization. That’s where the real engineering in vLLM and Atlas lives, and it’s a different problem from the one I set out to understand here. For now, GOTOken runs SmolLM2 at about one token per second, and I had fun getting back to QBasic after all these years.