Grounder › Guides › Local LLM context overflow

Why your local LLM returns nothing when you feed it a web page

Published July 15, 2026 · Last updated July 2026

You scraped a page (or three), pasted it into your local model to "ground" the answer, and got… nothing. An empty response, or a hard error. That wall is context-window overflow, and it's the first thing most people hit when they try to give a local model live web data. Here's exactly what's happening - with numbers from our own test - and the fix.

What overflow actually looks like (measured)

In our benchmark we ran a local Qwen2.5-7B with an 8,192-token window and fed it the naive way: search, fetch the top result pages in full, and dump them into the prompt. The payload was wildly unpredictable - across 8 questions it averaged 3,794 tokens but swung as high as 22,759 tokens on a single question, 2.8× the 8k window.

On "who is the CEO of Anthropic?" the fetched pages came to 91,039 characters (~22,759 tokens), and the model returned:

HTTP 400: n_keep 29992 >= n_ctx 8192

The same model, given a token-capped pack, answered "Dario Amodei" from 3,454 characters.

Why it returns nothing, not a worse answer

A cloud model with a 200k window quietly absorbs a few pages. A local model can't: you run it with a small window - the one that fits your VRAM, commonly 4k-32k, not the model's theoretical maximum. When the prompt exceeds n_ctx, the runtime (llama.cpp, under LM Studio or Ollama) doesn't summarize or drop the middle for you - it truncates or errors, and you get an empty or broken response.

And it's intermittent: the same pipeline that works on a short page hard-fails on the next question when the live page happens to be huge. In our run the 7B overflowed on 1 of 8 questions and a 27B on 3 of 8 - it tracks page size, not a fixed rate. The stable truth isn't "it always overflows"; it's that raw pages routinely approach or blow a small window, and when they do, you get nothing.

The fix: token-capped evidence, not raw pages

The cure isn't a bigger window (see the FAQ) - it's sending less, but the right less: the few passages that actually answer, capped to a known token budget. In the same test, replacing the raw dump with a ~1,000-token evidence pack (it stayed in a 756-1,254 token band across all 8 questions):

Fewer tokens fixed correctness (3/8 bare → 7/8), eliminated overflows, and ran 2.6× faster (prefill scales with tokens). Full method and raw results are on the benchmark page.

Three ways to get a token-capped pack

Approach 1 - Build it yourself. Fetch each page, strip boilerplate to markdown, split it into chunks, embed and rank the chunks against your question, and keep the top few until you hit your token budget. That "rank" step is the one most first attempts skip - the full pipeline is in the how-to guide.

Approach 2 - Use a fetch that caps for you. A retrieval API can return only the relevant passages under a token limit. With Grounder's fetch, pass a query and max_tokens and you get back passages that fit:

import requests
KEY = "your_grounder_key"

page = requests.post("https://grounder.dev/v1/fetch",
    headers={"Authorization": f"Bearer {KEY}"},
    json={"url": result_url,
          "query": "who is the CEO of Anthropic",
          "max_tokens": 600},          # <- guarantees it fits an 8k model
    timeout=90).json()

evidence = "\n\n".join(p["text"] for p in page["passages"])
# `evidence` is a few hundred tokens, not a whole page.

For a question that needs several pages, deep_search does the search → fetch → rank in one call and returns a token-capped evidence pack:

pack = requests.post("https://grounder.dev/v1/deep_search",
    headers={"Authorization": f"Bearer {KEY}"},
    json={"query": "which databases use the Raft consensus algorithm",
          "max_tokens": 600}, timeout=180).json()
# pack["passages"] are cited and fit your window - answer only from them.

Then hand evidence to your local model - it stays well under the window:

import requests  # any OpenAI-compatible local server (LM Studio, Ollama)
resp = requests.post("http://localhost:1234/v1/chat/completions", json={
    "model": "qwen2.5-7b-instruct",
    "messages": [{"role": "user",
        "content": f"Answer using only the evidence.\n\nEVIDENCE:\n{evidence}\n\nQuestion: ..."}]
}).json()
print(resp["choices"][0]["message"]["content"])

Approach 3 - Raise your context window - but know the trade-off: more context costs VRAM and slows prefill, and long prompts suffer the "lost in the middle" effect (models attend worse to the middle of a very long context). A ranked, capped pack is usually cheaper and more accurate than brute force.

FAQ

Transparency note: Grounder is a commercial product we build, so we have an interest here. The numbers above are from our own benchmark (Qwen2.5-7B, 8k window, 8 questions) - a directional run, not a statistical study; the full method and the losses are on the benchmark page. The overflow problem and the token-cap fix are true regardless of which tool you use - building the fetch-and-rank step yourself is a valid path, covered in the guide.