Prompt Compression Techniques for Long Context

Prompt compression shrinks the text you send to a model so each call uses fewer input tokens while keeping the facts the task needs. You pay less per request, wait less for the first token on long prefixes, and leave room in the window for the answer. This guide covers production prompt compression for long context: extractive and summary methods, LLMLingua-style token pruning, a measurable workflow, and cost and latency tradeoffs versus caching or retrieval.

You leave with techniques, a six-step checklist, failure modes that waste spend, and mid-2026 notes for Claude Sonnet 5, Opus 5, and Fable 5, GPT-5.6 Sol, Terra, and Luna, plus Gemini 3.5 Flash and Gemini 3.1 Pro.

What prompt compression is (and who needs it)

Prompt compression is any step that reduces the token count of the prompt before the main model runs. The compressed string (or chunk list) still has to support the job: answer the question, extract the fields, cite the source. You drop padding, repeated boilerplate, low-value sentences, and tokens a small scorer marks as safe to remove. The main model then reads a shorter input.

Long context made stuffing easy. Teams paste whole PDFs, full ticket threads, and multi-file repos because the window fits. That habit raises the bill and slows prefill. Attention still spreads across a larger haystack, so quality can fall even when nothing truncates. Compression attacks the length problem at the source. Caching attacks reuse of an identical prefix. Retrieval attacks which documents enter the prompt at all. Use all three as separate levers.

Who needs this: API builders who send the same long corpus with a new question each call, RAG systems that over-retrieve, agent loops that accumulate tool dumps, and support or research bots that attach thick policy packs. Skip heavy compression for short RTF prompts, one-shot drafts under a few hundred tokens, and prompts where every character is a hard contract (raw JSON schemas, exact legal quotes you must not alter). Those cases need edit by hand, not blind pruning.

How prompt compression works in production

Treat compression as a pipeline stage with a budget. You set a target token count or a keep rate (for example keep 40% of the original). You choose a method that matches how the context is built. You run the compressor, then you call the main model on the shorter prompt. You score answer quality on a fixed eval set so you know the trade. Without that score, you only know the input got shorter.

Methods fall into three practical buckets. Structural edits you do in code: strip headers, dedupe lines, keep only query-matched spans. Summary passes: a cheap model rewrites each document into a dense note, then the main model reads the notes. Token-level compressors: a small model scores each token and drops low-information ones, which is the LLMLingua-style path. Mix buckets when traffic justifies it. Start with structural edits; they cost almost nothing and often cut 20-50% before you add another model.

Cost and latency sit in the same spreadsheet. Input tokens dominate many bills when the prompt is huge and the answer is short. Prefill time grows with prompt length, so a shorter prompt can cut time-to-first-token even if the compressor itself takes a few dozen milliseconds. The compressor must finish faster than the savings you hope for on the main call. On a local GPU that is common. On a far-away API for every tiny request, the extra hop can erase the win. Measure end-to-end wall time, not only token counts.

Extractive, summary, and structural compression

Structural compression is the first pass. Remove navigational junk, repeated email signatures, CSS from HTML scrapes, lockfile blobs, and identical paragraphs that appear twice. Normalize whitespace. Keep section titles that help the model locate facts. This step is deterministic and easy to unit test.

Extractive compression keeps original sentences. Score each sentence against the user question with embeddings or keyword overlap. Keep the top-k sentences plus a few neighbors for context. The wording stays faithful to the source, which helps cite-or-refuse flows. Summary compression asks a small, cheap model to rewrite each chunk into a short note with IDs. The main model then reads the note pack. Summaries cut more tokens than extraction on chatty docs, but they can drop rare numbers if you do not pin a checklist ("keep all amounts, dates, and IDs").

Example shape for a policy Q&A call: (1) strip HTML and signatures, (2) retrieve or select the three sections that match the ticket, (3) extract the top 12 sentences plus section headers, (4) append the ticket text and the output schema. You may never need a token pruner if that pack already sits under your budget.

Token-level compressors (LLMLingua family)

Token-level prompt compression scores tokens and drops ones that carry little information for the task. Microsoft's LLMLingua line is the open baseline teams cite in 2026. LLMLingua uses a small language model and perplexity-style scores to prune. LongLLMLingua adds question-aware ranking and reordering so the kept context sits where the main model can use it, which helps long RAG packs. LLMLingua-2 trains a BERT-class encoder as a token classifier (distilled from stronger models) and runs faster for many out-of-domain packs.

In practice you load a PromptCompressor, pass the context list and optional question, set a rate or target token count, and force-keep tokens you must not lose (newlines, "?", schema keywords). Integrations exist for common orchestration stacks. Plan for a local or sidecar GPU for the scorer. Treat the compressor as a product dependency with version pins and eval gates. Compressed text can look broken to humans; debug with side-by-side original vs compressed on failing cases, not by reading the pruned string alone.

Do not run token pruning on brittle structured tails. JSON schemas, code fences, and exact quotes belong in a force-keep zone or outside the compressor. Prune the narrative context. Leave the contract intact.

A production workflow for prompt compression

Ship compression the way you ship any latency feature: baseline first, change one stage, measure quality and timing, then widen. Teams that flip on a 5× compressor across every prompt on day one drown in silent quality loss. Budget half a day to instrument token counts and latency, then an afternoon to try structural cuts on one high-traffic path.

Pick a path where the prompt is long and repeated with a short answer: handbook Q&A, ticket triage with a policy pack, or research answers over a fixed note set. Log original tokens, compressed tokens, compressor time, main-model time-to-first-token, and a quality score on twenty to fifty labeled cases. That dashboard tells you whether to keep the stage.

If the instruction block at the front of the prompt is still a messy paragraph, tighten it before you compress the corpus. A short, clear task contract leaves more of the token budget for facts. PromptMake /text can turn a rough brief into a tight system prompt you freeze in code; soft sell only. The tool shapes the stable front. Compression then targets the long, variable middle.

Steps 1-3: Baseline, budget, and cheap cuts

1. Capture ten real production prompts with token counts and answers you trust. Score them with your current checklist (schema valid, cite present, no invented IDs).
2. Set a budget: max input tokens for this path, or a keep rate such as 0.4. Tie the budget to a cost target (dollars per 1,000 calls) so the number is not arbitrary.
3. Apply structural and extractive cuts only. Re-score the same ten cases. If quality holds and tokens drop enough, stop here for this path. Many products never need a learned compressor.

At this point you should know whether junk removal was the real problem. If tokens remain high because the docs are dense and relevant, move to summary or token-level methods with the same eval sheet.

Steps 4-6: Compress, gate, and deploy

1. Add a summary pass on a cheap model, or wire LLMLingua-2 / LongLLMLingua with a moderate rate (start near 0.5 keep, not 0.1). Force-keep schema tokens and citation markers.
2. Gate on the eval set. Ship only if quality stays within your allowed drop (for example ≤2 failed cases of 50) and end-to-end latency or cost beats the baseline. Watch compressor overhead on p95, not only the mean.
3. Deploy behind a flag with logging of keep rate, forced-token hits, and main-model usage. Version the compressor model id next to the prompt version. Re-run the gate when you change Claude Sonnet 5 for Gemini 3.5 Flash or when the corpus template changes.

After a win, extend to the next traffic class. Keep a kill switch that sends the uncompressed prompt if the compressor errors. A failed compressor should not blank the user-facing path.

Common prompt compression mistakes

Teams compress the system prompt and the few-shot demos with the same aggressive rate they use on retrieved docs. Instructions and demos are short and high value. Leave them intact. Compress the corpus.

Teams chase maximum compression ratio as a vanity metric. A 10× cut that fails cite-or-refuse checks costs more in support load than it saves on tokens. Pick the mildest cut that hits the budget.

Teams skip a holdout set and judge quality by vibes on two happy-path questions. Compression fails on rare entities, numbers, and negation. Put those cases in the gate.

Teams run a heavy compressor on every 300-token chat message. Overhead dominates. Gate compression on a minimum length (for example only when context exceeds 2,000 tokens).

Teams combine lossy compression with prompt caching without thinking about the prefix. If the compressed text changes every call, you lose cache hits. Prefer a stable compressed handbook for the session, or compress only the per-query retrieved chunks that sit after a cached prefix.

Teams paste uncompressed tool dumps from agents (full HTML pages, entire stack traces) and hope the main model "finds the point." Truncate and extract at the tool boundary. Compression starts when you design the tool output, not after the dump lands in the prompt.

Cost, latency, and model notes for 2026

As of mid-2026, frontier APIs still bill input and output on separate lines, and long prompts remain the main cost driver for RAG-style apps. Hedge exact dollar figures; they move. The pattern stays stable: fewer input tokens cut spend in a straight line when output length is fixed. Prefill latency also falls with shorter prompts on most serving stacks, which matters for interactive UI and for agents that chain many calls.

Compare three bills on the same traffic: (A) full dump, (B) compressed prompt, (C) cached stable prefix plus short question. Caching wins when the large block is identical across calls inside the TTL. Compression wins when each call brings a different long document and reuse is rare. Retrieval wins when most of the corpus is irrelevant to the question. Many production systems use retrieval to select, compression to shrink, and caching on the frozen system and tool block.

Model notes: Claude Sonnet 5 and Gemini 3.5 Flash fit high-volume compressed Q&A where latency and price matter. Claude Opus 5, Claude Fable 5, GPT-5.6 Sol, and Gemini 3.1 Pro fit harder synthesis over a compressed pack when a miss is expensive. Keep execution prompts for reasoning-class models short on process fluff: goal, constraints, format. Do not add old "think step by step" scaffolding. For the summary compressor stage, prefer a fast cheap model (Flash / Haiku-class / Luna-class) so the stage stays cheaper than the tokens you remove from the main call.

Vendor context windows in the hundreds of thousands or millions of tokens do not remove the need to compress. They raise the temptation to stuff. Lost-in-the-middle effects and prefill cost still punish oversized packs. Use the window as headroom for the answer and for a few strong chunks, not as a mandate to send the whole library.

When to compress, and where PromptMake fits

Compress when the variable context is long, relevant on most lines, and unique per request. Cache when a large prefix repeats. Retrieve when most of the library is irrelevant. Hand-edit when the prompt is already short and quality is the only issue.

Draft the stable instruction block once. Freeze it. Point compression at the documents and history, not at the contract. If you still stare at a blank page for that contract, run the rough job description through PromptMake /text, pick the model you will call in production, and paste the result into your repo as the uncached (or cached) front. Then measure token cuts on the corpus path.

Ship with a dashboard for input tokens, compressor time, and eval pass rate. Revisit keep rates when pricing or model ids change. Prompt compression is an ops habit, not a one-time script.

FAQ

What is prompt compression in plain terms?

Prompt compression means you shorten the prompt before the main model reads it. You keep the facts and rules the task needs and drop low-value text. The goal is fewer input tokens, lower cost, and often lower latency on long context. The main model still does the job; it just sees a denser pack.

How is prompt compression different from prompt caching?

Caching reuses a processed copy of an identical prefix across calls. Compression changes the text to make it shorter. Caching needs byte-stable fronts and repeated traffic inside a TTL. Compression helps when each call brings a new long document. You can cache a frozen system prompt and still compress the retrieved chunks that follow it.

When does prompt compression save the most money?

Savings show up when input tokens dominate the bill: large context, short answers, high QPS. A keep rate of 0.3-0.5 on a 10k-token pack can cut a large share of input spend if quality holds. Savings shrink if the compressor is expensive, if you only cut a few hundred tokens, or if output tokens dominate the invoice. Run the arithmetic on your own usage export.

Does prompt compression hurt answer quality?

It can, if you over-prune numbers, names, or negations. Mild structural and extractive cuts often hold quality while cutting fluff. Aggressive token pruning needs an eval gate. Keep schemas and exact quotes outside the pruner. Compare compressed vs full prompts on the same labeled set before you ship.

What is LLMLingua and when should I use it?

LLMLingua is a Microsoft open-source line of prompt compression methods (LLMLingua, LongLLMLingua, LLMLingua-2) that score and drop tokens, with question-aware options for long RAG context. Use it when cheap structural cuts are not enough and you can run a scorer model near your traffic. Skip it for short prompts and for brittle code or JSON blocks you must keep verbatim.

Should I compress or build RAG instead?

Build retrieval when the library is large and most documents are irrelevant to a given question. Compress when the selected context is still too long or too chatty. Production stacks often retrieve first, then compress the top chunks. Pasting the whole corpus into a million-token window skips both controls and burns budget.

How do I start with prompt compression this week?

Log token counts on one long-context path. Strip junk and extract query-matched sentences until you hit a budget. Score twenty labeled cases before and after. If you need a cleaner system prompt for the frozen front, draft it once in PromptMake /text on the free tier, lock it in code, and keep compression aimed at the documents. Add LLMLingua-style pruning only if the cheap cuts miss the cost target.