Why the answer stops mid-sentence
The output halts as if someone pulled the plug - mid-word, mid-line of code. Four causes are candidates, and in the result they look almost identical.
July 29, 2026tokens · truncation · operations
The answer runs, gets long, gets longer - and then ends with return calcul. No
error, no message, just a text that stops. This is almost never a lapse by the
model. It's a limit something ran into, and the display usually doesn't say
which.
1. The output limit was reached
The most common case. Every call has a ceiling on the output (max_tokens, named
differently by some providers), and once it's hit, generation stops - wherever in
the sentence that happens to be. The model knows nothing about it: it emits token
after token, and at some point the server stops accepting them.
You can see it in the response's finish reason. If it says length instead
of stop, that's exactly what happened. This field is the single most important
clue, and plenty of integrations simply never read it - which is how a truncated
text gets processed onward as a complete one.
The LLM Call
One request, one response - the raw building block everything else grows from. And the key insight right at the start: the brain is not inside the agent. It's at the other end of the line.
We call a language model for the first time: one message out, one answer back - and then the line goes dead. No state, no memory, no tools. That hanging up explains almost everything that follows. And we're not calling just any model: we talk to our own - qwen3.8-27b-fast, running on our inference stack. How to run a model yourself is Block III's topic; for now we simply assume one exists somewhere.
Chat + GPT: who does what here?
The name ChatGPT gives away the division of labour this whole block is about: GPT is the model - it runs on GPUs somewhere and can do exactly one thing: text in, text out. Chat is the software around it - it remembers the conversation, gives the model a role, hands it tools. That software part is what we rebuild here, layer by layer - just not as a chat, but as an agent. The harness is the chat part. The model is rented - or, in our case, self-hosted.
One call, no subscription
Now that the video has shown us that all the magic behind "AI" and ChatGPT actually sits in the LLM, that is what we will look at first. So for now we assume we have access to a language model. How to run language models yourself is the topic of Block III (inference). How to get hold of an API is covered, among other places, in our OpenRouter key tutorial.
Let's try calling it: the simplest possible form is a bare web request - curl is enough. The endpoint address (LLM_BASE_URL, in our case a self-hosted vLLM server) and the key (LLM_API_KEY) come from the environment - that is where you put whichever provider or own server you want to talk to. In the call itself we say which model we want to talk to (model), how long the answer may get at most (max_tokens), and what should be said (messages):
curl "$LLM_BASE_URL"/chat/completions \
-H "Authorization: Bearer $LLM_API_KEY" \
-H "content-type: application/json" \
-d '{
"model": "qwen3.8-27b-fast",
"max_tokens": 200,
"messages": [
{ "role": "user", "content": "Explain in 2 sentences what an AI agent is." }
]
}'
What comes back is - again - just JSON, and far more technical than anything ChatGPT ever shows:
{
"choices": [
{
"message": { "role": "assistant", "content": "An AI agent is a program that …" },
"finish_reason": "stop"
}
],
"usage": { "prompt_tokens": 23, "completion_tokens": 61 }
}
Three fields determine everything that follows: choices[0].message is the answer. finish_reason says why the model stopped talking - stop means "finished on its own", length means "cut off". And usage counts tokens - the currency inference is measured in, even on your own hardware.
From code, the same call is a few lines of Python or JavaScript - no library required, an HTTP client is enough. In the experiments below you see the raw request and raw response of every run; further down you'll find both as ready-made reference files and as a prompt for your coding agent to build it yourself.
The pen pal with no short-term memory
The most surprising thing about the first call is what does not happen: the model remembers nothing. The second call is a request like any other - the model doesn't know there ever was a first one, even if it happened milliseconds ago. Think of a pen pal who forgets everything between two calls: every time we ring, we have to retell everything that ever happened - and the moment he has answered, we hang up.
An LLM call is nothing magical: JSON in, JSON out. No login, no session, no account balance on the other side - just one request and one response.
And the call is stateless: the next call starts from zero, the voice at the other end remembers nothing - technically it is one huge, clever random-number generator with no memory at all. Everything an agent can do later - conversation, role, tools - the harness has to bring along anew on every call. The brain is not inside the agent. It's at the other end of the line - and over the next components we build everything around it.
Why does it hit agents so often? Because the defaults come from a time when answers were short. 1024 tokens is a lot for a chat and very little for a generated file. And it shows up late: the truncated text looks correct right down to the last line.
2. The context window is full
A model has one overall budget for input and output. If the message history
occupies 190,000 tokens and the window holds 200,000, then 10,000 remain for the
answer - even if max_tokens is set far higher. The effect is the same as above,
but the cause sits elsewhere: not in the limit, but in the history.
That explains a behaviour many people observe: an agent produces decent answers early in a run and truncates them wholesale towards the end. The history simply grew. If you want the underlying reason: it's the same one that makes the tenth turn cost so much more than the first - the history rides along in full on every call.
3. A stop sequence appeared in the text
Stop sequences tell the model to halt as soon as a given string shows up. Useful
for cutting off role markers like \nUser: - and treacherous when the sequence
legitimately occurs inside the content you wanted. The classic: ``` as a stop
sequence while the answer contains a code block. The finish reason then usually
reads stop, because from the server's point of view everything went to plan -
just early.
4. The connection died
When streaming, the answer arrives in many small chunks over an open connection. Anything in between can sever it: a proxy with a time limit, a load balancer that closes idle connections, a restart. To the application this looks like an ending - the text stops, and a finish reason never arrives.
This is the only one of the four cases where something is actually broken. You
recognise it by the fact that no final event arrives: no finish_reason, no
usage figures, no closing chunk. If you only look at the collected text, you see
the same stump as in case 1.
The order in which to look
- Read the finish reason.
length→ output limit.stopon obviously unfinished text → stop sequence. None at all → severed connection. - Look at usage. If output tokens sit exactly on the limit, the case is clear. If input tokens plus the limit exceed the window, it's case 2.
- Only then touch the prompt. "Be more concise" fixes none of the four cases; it only shifts when they occur.
And for operations, the one rule that saves the most grief: a response with
finish reason length is a failure, not a result. Treat it as one - abort,
re-request, have it continue. Just don't pretend it's complete.