How LLMs Actually Work, for People Who Decide Things

C-suites get paid to decide where they need better information, and AI resists the instinct, because the vocabulary around it seems almost designed to stifle exploration: neural, attention, emergent, hallucination. Language like that makes the machine sound like a colleague. It is not a colleague. It is a pipeline of arithmetic, and the only honest way to prove that is to run the pipeline in front of you, which is what this essay does. To test, debug, cost, or capacity-plan a system, you first have to be able to explain it, and by the end you will be able to explain this one.

The vehicle is a pocket transformer taken from the executive session of an AI curriculum I wrote and teach: ten words of vocabulary, one layer, one attention head, 126 parameters in total. Every production model you have heard of runs the same equations, just wider and deeper. We will give it four words, "the modem needs a," and follow every computation until it answers and then decides, on its own, to stop talking. If you can multiply and add, you can check every step on paper.

Two sentences summarize everything. First, AI is a mathematical pipeline: language goes in, becomes numbers, passes through layers of learned transformations, and a probability comes out. Second, meaning becomes geometry: words become points in space, similar meanings land close together, and "how similar are these two ideas?" becomes a question about the angle between two points, computable with multiply-and-add.

Everything below is detail on those two sentences.

The model never sees words

An LLM never actually reads your text. The first thing that happens to a prompt is tokenization: the text gets carved into subword pieces drawn from a fixed vocabulary of around 100,000 entries. Common words like "the" and "and" come through whole. Rarer words get split into reusable parts, so "tokenization" shows up as token + ization. There is good trivia buried here, too: the algorithm doing the carving descends from a data-compression trick published in 1994, which means the first thing a billion-dollar model does with your question is run it through a thirty-year-old compression scheme.

A word about tokens, because these small groups of characters set the price of running AI. API rates, context windows, and generation speed are all quoted in tokens. In English, four characters is roughly one token, and 750 words lands near a thousand. The same content in another language can run two to three times the tokens, worth knowing before a multilingual customer-facing deployment. You pay for AI by the token, so a budget conversation is a token conversation.

Next, each token is traded for its row number in the vocabulary, and that list of integers is what the model actually takes in. The point that matters: the integers are arbitrary labels. Nothing makes token 4438 "closer" to 1587 than to 11. "Dog" might sit at entry 7 while "puppy" sits at 98,431. Doing arithmetic on the labels produces nonsense; "dog" plus one is not a slightly better dog, it is whatever unrelated word occupies the next shelf. That single fact explains a familiar class of AI embarrassments, like a model miscounting the r's in "strawberry": it never saw letters, only one or two opaque chunks.

Where meaning enters

The next stage fixes the arbitrary-label problem. Every token ID indexes into an enormous learned table, and what comes back is a vector: thousands of numbers that encode the token's meaning as a position in space.

The property worth pausing on: similar meanings produce similar vectors. In this space, "dog" and "puppy" point in nearly the same direction, while "dog" and "spreadsheet" point somewhere else entirely. No one programmed that in. It fell out of training the model to predict text. And once meaning has coordinates, the similarity of two meanings is just the angle between two vectors, which multiplication and addition can compute, as you are about to do.

A great deal of the industry stands on that one idea. Semantic search is a nearest-neighbor hunt through meaning-space. Retrieval-augmented generation, RAG, locates the document chunks whose vectors sit nearest your question and passes them to the model as context. A recommendation engine measures a user's vector against item vectors. Agent memory pulls up past situations whose vectors look like the present one. One geometric trick, five product names.

Meet the pocket transformer

Time to open the machine. Our model's world is ten words, and its entire stock of meaning is one small table. Each word gets four numbers. In a production model those axes are unlabeled and there are 4,096 of them; here I have nicknamed the four so the story can be read aloud: device, action, grammar, and end (as in end-of-sentence).

Read a few rows and the table starts talking. "Modem" is pure device. "Needs" is pure action-required. "A" is pure grammar, an article waiting for its noun. The period is pure end-of-sentence. "Reboot" straddles two axes, a device-flavored action, and that detail will decide the ending.

In a real model every one of these numbers is learned from data; here they are hand-set so the mechanism stays visible, and the displayed values are rounded to two decimals, so if you recompute a step you may land within a penny of what I show. Beyond this table the model owns a small position stamp, three attention lenses, and two detectors with their write-back rows, each introduced when it comes on stage. That inventory is the whole machine. Here is the route the data takes through it:

Reading the input

Step one, tokenize. "the modem needs a" becomes the ID list [1, 2, 3, 4]. Step two, embed: each ID copies its row from the table above. No arithmetic yet, just lookups.

Step three, stamp the position. Attention has no built-in sense of order; without a stamp, "modem needs a" and "a needs modem" would look identical. Our stamp is a small value added on the end axis that grows with the slot: 0 for slot one, 0.1 for slot two, 0.2 for slot three, 0.3 for slot four. One addition per token, and the sentence now knows its own order:

From here we follow the last token, "a," because the last position is the one that predicts the next word. The word "a" is an article waiting for its noun, and attention is how it asks the rest of the sentence what is going on. Every other token runs the same math in parallel.

Attention, by hand

First, why attention exists at all. Take the sentence "The trophy didn't fit in the suitcase because it was too big." Everyone reads "it" as the trophy. Swap one word, "too small," and "it" becomes the suitcase. The sentence's shape never changed; context alone decided. Attention is the machinery that lets each word weigh context like that, and here is the whole formula:

Attention(Q, K, V) = softmax(QKᵀ / √d) · V

It looks worse than it is. Q is a question, K is a set of labels, V is a set of offerings, and with our vector width of 4, the scary √d is the square root of four, which is 2. The next five steps are that formula, one piece at a time.

Step four builds the question. A lens called WQ turns a token's vector into a query. A lens is just a small grid of weights, and ours has two live channels:

Read down the first column: grammar contributes 3 to the "seeking device" channel. In plain English, grammar words go looking for devices, which is exactly what an article should do. Now push "a," whose stamped vector is [0, 0, 1, 0.30], through that column: 0×0 + 0×2.5 + 1×3 + 0.30×0 = 3.00. The second channel works out the same way. So q = [3.00, 3.00], and you can read it aloud: "a" walks in asking two questions, any devices here, and any actions needed?

Every token also runs its vector through a second lens, Wₖ, which builds its label, what it advertises to searchers. That lens simply copies the device axis into channel one and the action axis into channel two. The results:

"Modem" advertises device-ness. "Needs" advertises an unmet action. "The" and "a" advertise nothing, because the lens deleted their grammar content and there was nothing underneath.

Step five scores the question against every label, one dot product each. Against modem: 3×1.00 + 3×0 = 3.00. Against needs: 3×0 + 3×0.80 = 2.40. Against "the" and "a": zero. Then the one division in the whole machine: divide everything by √4 = 2, which keeps scores in a range the next step handles gracefully. The scaled row is 0, 1.50, 1.20, 0.

Step six is softmax, the move that turns scores into percentages, and it is three rows of arithmetic. Exponentiate each score: e⁰ = 1, e¹·⁵ = 4.48, e¹·² = 3.32, e⁰ = 1. Add them up: 9.80. Divide each by the total: 10%, 46%, 34%, 10%. That is the entire mystery of softmax, and it never changes the order of the scores, it only converts gaps into proportions.

Look at what the arithmetic just did. The article handed 80 percent of its attention to "modem" and "needs," the two words that matter, and nobody programmed that reading of the sentence. The lenses produced it.

Step seven blends what the winners offer. A third lens, Wₛ, decides what each token hands over if selected: modem offers its device signal, needs offers its action signal, the others offer nothing. Multiply each offering by its attention share and add. On the device axis: 0.46×1.00 = 0.46. On the action axis: 0.34×0.80 = 0.27. The gathered context is z = [0.46, 0.27, 0, 0], and it reads as: mostly a device, plus a need.

Step eight adds it back. The residual move, position by position: [0, 0, 1, 0.30] + [0.46, 0.27, 0, 0] = [0.46, 0.27, 1.00, 0.30]. "A" keeps its own identity, grammar still 1.00, and gains context. That keep-and-add pattern is what lets real models stack 32 layers without early information washing out. Attention is finished: the article now knows its sentence.

Two detectors interrogate the token

The feed-forward network sounds grand. It is a bank of detectors, and ours has two. Each one weighs the evidence in the token's vector, subtracts a threshold, and keeps the result only if positive (negatives become zero, a rule called ReLU). A production layer runs 14,336 of these; two are enough to watch the mechanism work.

Detector A is asking: an article, in a context of devices and needs? Detector B is asking: device plus action with no article pending, in other words, is this sentence finished? Note B's grammar weight of minus two. An open article is evidence against being done.

Run A on our token [0.46, 0.27, 1.00, 0.30]: 0.4×0.46 + 0.4×0.27 + 1.2×1.00 = 1.49. Subtract the threshold: 1.49 − 0.90 = 0.59. Positive, so A fires at 0.59. Run B: 0.46 + 0.27 − 2×1.00 = −1.27, minus 1.40 is −2.67. Negative, so B stays silent. The model knows the sentence is not done, and that knowledge is a negative number.

Step ten lets the firing detector speak. Each detector owns a write-back row, its opinion about what the token should become:

A fired at 0.59, so its message is 0.59 times its row: [1.42, 2.71, −0.94, −0.30]. B writes nothing. Add the message to the token and the result is h′ = [1.88, 2.99, 0.05, 0.00]. Read the change: action-needed surged from 0.27 to 2.99, device rose, and the article-ness was almost entirely cancelled. The token stopped being an article and became a request for a fix.

The answer

Step eleven scores every word in the vocabulary, and the elegant part is that no new weights are needed. The scorer is the embedding table from the beginning, reused backwards: how aligned is the finished token with each word's meaning? One dot product per word. For reboot: 0.6×1.88 + 0.8×2.99 = 3.52. The full shelf of raw scores, called logits:

Reboot leads because it matches both signals at once, the device context and the surging action-need. "Needs" matches only one and "modem" only the other. Step twelve is softmax again, this time across the vocabulary:

Reboot, at 45 percent, triple the runner-up. The machine has completed "the modem needs a reboot," and you checked the arithmetic that did it.

The loop, and the stop

Step thirteen: append "reboot" and run all thirteen steps again. This is how every model writes, one token per full pass, hundreds of passes per paragraph. I will give you pass two in summary, and every number is checkable the same way.

"Reboot" is now the question-asker, an action word hunting its object, and attention comes back quieter: modem 36 percent, reboot itself 24. After the blend and residual its vector is [1.11, 1.10, 0.00, 0.40]: device and action both loud, article gone. Now watch the detectors trade places. A's evidence totals 0.88 against its threshold of 0.90. Two pennies short, so it stays silent; with no article on deck, there is nothing for a fix-word to complete. B's evidence is 1.11 + 1.10 with no grammar penalty, clearing its bar to fire at 0.80, and B's write-back row points almost entirely at the end axis: 0.80 × 3.6 = 2.90 lands there. Score the vocabulary and the period wins at 77 percent.

That ending is the part worth remembering. The model completed the thought and then decided, by arithmetic, that it was finished talking. A threshold is a decision, and you just watched one flip on a margin of 0.02.

The same math at real scale

Substitute the production sizes and nothing else changes:

Our trace was perhaps ten minutes of careful arithmetic. A human doing one production token at one operation per second would need about 500 years; a GPU does it in milliseconds, mostly as matrix multiplication, which is the same multiply-and-add you just performed, in bulk. That is also the whole reason the GPU won AI: a CPU is sixteen brilliant chefs, a GPU is twenty thousand line cooks all chopping at once, and this workload is nearly all chopping. I did simplify honestly along the way: real models run 32 attention heads side by side, normalize between steps, learn their weights from data instead of taking them from me, and cache old keys and values for speed. Every one of those is a size or bookkeeping difference, not a different algorithm.

Prediction, not lookup

Now the payoff for sitting through the arithmetic. What you watched the model produce was a score for every word it knows. An LLM is a fluency-maximizing plausibility engine. Its weights encode how text tends to continue, and that tendency is the entire mechanism behind an answer that feels retrieved. Ask a big model to finish "The capital of France is" and Paris scores about 96 percent for the same reason reboot scored 45 in our toy: everything it read points that way.

For anyone signing off on a deployment, the consequence is the whole ballgame: a continuation can be perfectly fluent, score at the top, and be flat wrong, and it will sound precisely as confident as a correct answer. The machine optimizes plausibility. Truth was never part of the objective, so hallucination is native behavior, and no vendor patch is coming for it. Grounding techniques like RAG exist for exactly this reason. They swap the question "what does the model remember?" for "what do these retrieved documents say?", and the second question is one you can audit.

What this buys you

Cost stops being magic. Tokens and GPU-hours are engineering units, the way kilowatt-hours are. Context windows, latency, and per-seat pricing all come back to how many times the pipeline runs and how wide it runs. A team that understands the pipeline can work out, from first principles, what to build, what to buy, and what either should cost.

Vendor claims become testable. "Our model understands your business" unpacks to: the weights encode continuation tendencies, and the context window carries whatever gets fed in per request. The follow-ups then ask themselves: what gets retrieved, what gets logged, where does our data enter, and does it leave?

Risk conversations get concrete, too. The question of whether the machine hallucinates is settled, it does, so the useful questions are engineering ones: grounded on which sources, evaluated how, monitored by whom, with human checkpoints wherever consequences are real. Governance improves the moment it has a mechanism to talk about.

And the coming years of AI headlines will be assembled from parts you can now name. An agent is this same loop, allowed to call tools along the way. A world model runs the same recipe of tokens, embeddings, attention, and next-thing prediction over environments: video, 3-D space, physics.

A transformer is arithmetic you could do by hand, and now have, repeated until it looks like thought. A leader who has internalized that stops being an audience for AI claims and becomes a qualified judge of them, able to reason about capability and risk from first principles. That reasoning outlasts every model it will ever be applied to, which is what makes it the durable advantage.