How LLMs Actually Work (for Engineers)

You don't need the math, you need a model that predicts behavior

You wire up an LLM API, send a prompt, and get back something brilliant. Then you change one word and it falls apart. Or it invents a function that doesn't exist. Or your bill triples overnight and you have no idea why. The model feels like a slot machine, and you're the one holding the lever.

Here's the thing: you don't need transformer math or attention diagrams to stop being surprised. You need a working mental model, one that lets you reason about why the model behaves the way it does, what it costs, and where its limits are. That's what this article gives you. No gradients, no softmax derivations. Just the moving parts an engineer touches.

The one-sentence definition

A large language model is a function that, given a sequence of tokens, predicts the next token.

That's it. Everything else, chat, code generation, summarization, agents, is that single trick repeated. The model looks at everything so far and produces a probability distribution over what comes next. It picks one token, appends it, and looks again. Repeat until done. The "intelligence" is an emergent property of doing this billions of times over a vast amount of text.

Autocomplete on steroids is a genuinely good first model. The differences are scale and memory: instead of guessing one word from the last three, an LLM weighs thousands of prior tokens to guess the next one, and it does so with a sense of grammar, facts, and reasoning patterns absorbed during training.

The inference pipeline, end to end

When you send a prompt, it travels through a fixed sequence of stages before you get text back. Knowing these stages tells you exactly where cost is incurred, where text gets cut off, and which knobs change behavior.

1. 1

   Prompt

   You send raw text, a system prompt, the conversation history, and the user's latest message all concatenated together.

2. 2

   Tokenizer

   The text is split into tokens (sub-word chunks) and mapped to integer IDs. "tokenizer" might become two tokens; " the" is usually one. This is the unit everything is counted and billed in.

3. 3

   Model forward pass

   The token IDs run through the network once. The output isn't a word, it's a score for every token in the vocabulary (tens of thousands of them).

4. 4

   Probabilities

   Those scores are turned into a probability distribution: token X has a 40% chance of being next, token Y 12%, and so on.

5. 5

   Sampler

   This is where temperature and top-p live. The sampler decides how to pick from the distribution, always the top choice, or a weighted roll of the dice.

6. 6

   Detokenize

   The chosen token ID is mapped back to text and streamed to you, which is why responses appear word-by-word.

7. 7

   Append & repeat

   The new token is appended to the input and the whole loop runs again. Generating 500 tokens means 500 forward passes. That's why output is slower and pricier than input.

The three knobs you actually control

You can't change the model's weights, but you can change how the sampler reads its output and when generation stops. These three parameters cover ~90% of what you'll tune in practice.

Here's a request showing all three in context. The shape is the same across providers; only the field names and endpoint differ.

import anthropic

client = anthropic.Anthropic()  # reads ANTHROPIC_API_KEY from env

response = client.messages.create(
    model="claude-sonnet-4-5",
    max_tokens=512,        # hard cap on the OUTPUT, stops here even mid-sentence
    temperature=0.2,       # low = focused & repeatable; raise toward 1.0 for variety
    top_p=0.95,            # nucleus sampling; leave near default unless you have a reason
    system="You are a terse senior engineer. Answer in <=3 sentences.",
    messages=[
        {"role": "user", "content": "Explain what a token is, like I ship code."},
    ],
)

print(response.content[0].text)
# Tokens consumed are reported on response.usage, 
# input_tokens (your prompt) + output_tokens (what it generated).
print(response.usage)

Why context windows and tokens matter

The context window is the maximum number of tokens the model can see at once, your system prompt, the full conversation history, retrieved documents, and the answer it's generating all share one budget. Everything important about cost and reliability flows from this single constraint.

• Cost is per token, both directions. You pay for input tokens (everything you send) and output tokens (everything generated), and output is usually several times more expensive. A chat app that resends the whole history on every turn pays for that history again every single message.
• Truncation is silent and brutal. When the conversation plus the requested output exceeds the window, the oldest tokens fall off the front. The model literally forgets the start of the chat, and it won't tell you it did. Long agent loops are where this bites hardest.
• A full window can mean a cut-off answer. If input fills most of the window, there's little room left for output. The model hits the ceiling and stops mid-thought, looks like a bug, is actually arithmetic.
• Context pressure feeds hallucination. When the model can't see the relevant fact (because it was truncated, or never fit), it does what it always does: predicts a *plausible* next token. Plausible-but-wrong is exactly what a hallucination is. More context is not always better, irrelevant filler dilutes the signal too.
• Tokens are not words. A rough rule of thumb for English is ~4 characters per token, or ~0.75 words per token. Code, JSON, and non-English text tokenize less efficiently. Estimate before you assume something fits.

The practical upshot: treat the context window as a budget you actively manage, trim history, summarize old turns, retrieve only what's relevant, and measure token usage from the response rather than guessing.

Common misconceptions that trip up new builders

1. "It looks things up." It doesn't, at least not by default. A base model has no live knowledge and no database; it predicts from patterns frozen at training time. Current facts come from tools, search, or documents *you* put in the context.
2. "Same prompt, same answer." Only at temperature 0 (and even then, not always guaranteed across infrastructure). With any randomness in the sampler, identical prompts can produce different outputs by design.
3. "It understands like a person." It models statistical relationships in language extraordinarily well. That produces useful, often reasoning-like behavior, but it has no goals, beliefs, or awareness. Don't anthropomorphize your debugging.
4. "Bigger context window = better answers." A bigger window lets you fit more in; it doesn't make the model focus. Stuffing irrelevant text in raises cost and can *hurt* accuracy. Relevance beats volume.
5. "It generates the whole answer at once." It generates one token at a time, each conditioned on the last. That's why it streams, why output cost scales with length, and why early tokens shape the rest of the response.

Takeaways

Where to go next

Now that you can predict *why* the model behaves the way it does, the next steps are shaping its input, calling it cleanly, and keeping the bill sane. These build directly on this mental model.

• Prompt Engineering Fundamentals, how the tokens you put in front shape every token that comes out.
• Working with the LLM API, turning that request body into production code: streaming, retries, structured output.
• LLM Cost & Latency Optimization, managing the token budget so quality stays high and the bill stays low.
• Follow the full AI Engineer career path to go from these foundations to shipping real LLM systems.