Ranking code by perplexity compresses context 5.6× without hurting accuracy

Coding LLMs can already complete, explain and fix code, but real projects make them read thousands of lines. Long context windows help, yet they cost time and money — and they paradoxically hurt accuracy: the model starts drowning in detail and misses hidden dependencies between files and functions. Generic text compressors throw away phrases and tokens with no regard for code structure. Similarity-based RAG catches surface matches and often misses implicit links — where a parameter actually comes from in the configuration, say.

The idea: compress without breaking meaning

The authors of LongCodeZip propose a simple but precise recipe: keep in context only what actually helps answer the current prompt. The framework is training-free and requires no fine-tuning of the model. It works in two steps. First it coarsely picks the functions and classes that most reduce the model's uncertainty about the query. Then, inside the selected functions, it carefully cuts the code into semantic blocks and packs them optimally into a given token budget. The result is compact but informative input: the structure survives, the junk is gone.

How it works under the hood

What makes a fragment useful? The researchers measure how much the presence of a chunk of code lowers perplexity on the text of the query — in effect, how much easier it becomes for the model to predict the words of the prompt once it can see that code. This is an approximation of mutual information: the bigger the drop in perplexity, the greater the fragment's contribution.

Two levels of compression

• Coarse selection. The repository is cut at function and class boundaries. Each piece is ranked by its conditional perplexity with respect to the prompt, and the top candidates are taken under a preliminary budget. The rest are replaced with short stubs so the overall structure does not break.
• Fine-grained pruning. Inside the surviving functions, lines are treated as atoms and grouped into blocks at spikes in perplexity — usually the points where the code changes topic: a new branch of logic, an important check, a call to an external API. Every file gets an adaptive budget, with the more relevant ones getting more. Block selection is formalized as a knapsack problem: value is relevance, weight is the token count, and the goal is to fit the limit while carrying off as much meaning as possible.

What the experiments showed

The tests covered three kinds of task: code completion, module summarization and questions about a repository. The models were both open (DeepSeek-Coder-6.7B, Qwen2.5-Coder-7B, Seed-Coder-8B) and closed (GPT-4o, Claude-3.7-Sonnet).

The headline result: up to 5.6× compression with no drop in quality, and sometimes a gain — the noise goes, the signal stays. On code completion, Qwen2.5-Coder-7B reached 57.55 ES and 32.40 EM at 4.3×, beating function-level RAG on a smaller budget. DeepSeek-Coder-6.7B did better on the same data than it did with no compression at all, at 5.3×. Seed-Coder-8B hit the top rate of 5.6× with the best quality of any method. On RepoQA, accuracy climbed to 87–91 at 4.5–5.3×, closed models included. Summarization stayed level with the uncompressed baseline or improved slightly at 1.7–3.5×.

The savings in time and money are real. Take Qwen2.5-Coder-7B on code completion: generation took 6.6 seconds after compression instead of 15.7 seconds without it, and the cost of input tokens fell by roughly 77%. Compression itself runs in about 2.6 seconds and uses little memory. It can be made faster still by running it on a small 0.5B model: quality barely moves and resources are saved.

What matters most in the design

The ablations show that the bulk of the benefit comes from the coarse selection by conditional perplexity. Swap it for embedding similarity and quality falls off noticeably. But the second stage earns its keep too: semantic blocks, the adaptive budget and the knapsack add percentage points of accuracy, especially under a hard token limit. Together they raise the density of useful information in every character you show the model.

Where this is useful in practice

• Completion across several files, when the model's window cannot hold the whole project.
• Answering questions inside a repository: finding the right implementation, tracing a call chain, without loading everything in sight.
• Summarizing large modules and assembling context for review.

Notably, LongCodeZip does not depend on any particular LLM and requires no training — it can sit in front of any model as a transparent filter. That is valuable where the model is closed or cannot be fine-tuned.

Limitations and what comes next

Relevance scoring rests on perplexity, which means access to logits is required. That said, the authors show that even a tiny compressor model does the job.

One more caveat: the summaries were scored by an LLM judge; bias was reduced by reversing the order of comparisons, but a human check is always worth having. The next steps look like a more precise accounting of cross-file dependencies and faster segmentation for multilingual repositories.

💾 Code