It's nice that this article goes beyond the "context too long" scapegoat diagnostic.

There is more to it than just reducing content in context. LLMs attention mechanism works by finding graphs of related words and concepts and it works best when the relevant relationships are close together in the text. Length matters but how well the information is organized matters as well.

It's useful to be familiar with the concept of dependency length. The two sentences in this image ( https://benoitessiambre.com/img/dependencylength.png ) have the same words and same meaning but sentence C is organised to minimize the distance between related words.

This can also apply to paragraphs and sections. Ordering them so that related information is recursively closer together, helps AIs think better (also helps humans, Dependency Length Minimization, DLM is an idea from psycho-linguistics).

This is one reason progressive disclosure and sub agents are important. Including unnecessary information spreads apart surrounding information and makes it more difficult for AIs to figure things out.

For eg. optimizing skills, it can help to put more important information at the bottom of a skill. Usually after a skill is loaded, the AI will start “thinking”, talking to itself to try to reason through a problem. The thinking block will have an easier time considering the text closest to it, the last part of the skill, than information farther up.

Thinking blocks can produce a lot of text (a lot of it often hidden from users) that also spreads information apart. Sometimes, when an early instruction is still relevant but there’s been a lot of thinking and discussion added to the context since, it’s worth repeating a summary the early information ( keeping in mind that too many reminders can also spread other information apart ). I think however, LLMs are tuned to overweight very early information so that they don't disregard system prompts. You might be able to exploit this by placing some important facts near the very beginning of a conversation.

If you've ever written compression algorithms where repeated blocks are referenced instead of encoded, you'll be familiar with how related information being clustered allows for better compression. It's another example of the parallels between intelligence and compression and it hints at the information theoretic aspect of DLM.

I like the idea of VictorTaelin's UniiChat [1] which summarizes every message into a single line and on top of that a binary tree of lines combined from lower levels. The context is then a view of that tree with more detail in recent messages and less detail in historical messages, plus a `zoom` tool that enables the agent to navigate the tree and read messages.

An implementation of this idea based on the pi harness: https://github.com/jonaslsaa/pi-optchat

[1]: https://gist.github.com/VictorTaelin/91837951a5ce5b38f341ec1...

> As we focus more on memory, we also need to think about strategic forgetting. Over time, overly precise memories become conflicted and messy, and nobody has yet figured out how to handle this.

> I think that’s the context problem of tomorrow.

This passage is explains why humans have imperfect memory. Seems the perfect memory we wish LLMs had impedes creativity and is maladaptive.

Strategic forgetting helps put the non-deterministic machine to best use. LLMs don't need perfect memory; that's covered by the computer itself.

I like that this justifies the continuing relevance of context engineering - a seemingly dated term - in the face of trends like loop engineering.

> The tea task for FireRed was in its weights, so it poisoned its context.

Mmm, to my mind, isn't this mixing completely different things? I thought poisoned context is stuff in the context that makes the model behave poorly. But if it's in the weights, that's just... model issues.

> When building intelligent applications, harnesses, and workflows, our job is to choose when to be deterministic (what to put in code) and when to be probabilistic (when to let the model drive).

I really like this final sentence from the presentation. When I read this phrase, I realized that it captured one of the core dynamics that I've been using to figure out when and how to apply AI to a task. I often find myself trying to figure out if it is better to A) ask the AI to do a task for me or B) ask the AI to build something that can do the task for me. A major component of that answer comes down to how deterministic or probabilistic the task is.

> For me, around 200,000 or 300,000 is where I start looking. I’ll check, see it’s at 250,000, and think: I’ve got a big job coming up. I’m going to reset and start fresh

I don't find this to be true, at least with the newer models. I also find that the models fill up to 250k very quickly (presumably mostly by looking through the codebase). Without that context I find the models perform badly. I also don't really notice that large of a change until around 700-800k context and even then it isn't as bad as it used to be. I suspect, that (especially for coding), the long horizon RLVR tasks have helped with this quite a bit.

I like the overall idea of deliberate forgetting. And I found checkpointing into artifacts (code, design, intent) to start with fresh context to work well for that purpose.

Interestingly, the artifacts include not just code but alao retroactive design and retroactive “execution plan”. I feed them to another agent to validate the approach and resume work with fresh context and update mission statement.

Unexpectedly, design doc and an execution plan are now outcomes of coding process, not inputs to it. I’m not sure how to think about this.

Anecdotally (sample size == 1), I have observed that as my context grows longer, the model seems to get dumber. I'm not an advanced user with harnesses, etc.; I just use the chat interface, as that's all I can afford, but after a few turns the model (I'm talking frontier models here from OAI and Ant) seems to just do stupider things.

So like- I considered this idea of using a fast decision model to prune before compaction, but I scrapped it because A; im not bottlenecked by speed, and B; cache invalidation dark magic

From my understanding, if we have turns [A, B, C, D, E] and we decide that turn C is useless context, then we lose our cache for the [D, E] segment. (plz correct me if im wrong, I havent tested this)

Personally, I pay no mind to session state nowadays. I exclusively use provider native hosted compaction, which if you believe the providers, is the way to go. For context management- I spend most of my effort on the things which dump output back to the model.

If anyone has any good reads on benchmarking OpenAI and Anthropic hosted/native compaction against custom stuff, I would love to read!

Until a few months ago I did it this way, but OpenAI and Anthropic built their own harnesses, and since those end up conflicting with the harness I built myself, having no setup at all actually ended up performing better. As a result, it doesn't seem necessary to set up rules or memory separately, and if I do, using skills or agents made of only logging and flow (without things like rules or guardrails) seems like the best option going forward.

Forgetting is the part I see break first. Old facts in memory don't get deleted, they get contradicted, and the model happily picks whichever one is closer in the prompt. What's worked better for me is having this really cool way of jev compaction, there was a youtube video out there somewhere that spoke about how to get compaction to be super fast, cheap and have it retain the most useful information from the context compaction.

I can't tell if this engineering or just autocomplete that is mostly helpful, sometimes.

>This is the advice I keep giving now. Someone says, “My coding agent’s doing something weird. Claude’s been nerfed.” And I ask, “How big is your context?”

That’s really true, but on the other hand, if there’s too little information, it also ends up working in strange ways. Most AGENT code anti-patterns work well at a medium amount of context, but when there’s too much or too little information, they don’t behave properly.

Anecdotally I only stuggle with this with Claude Code. Codex has been great at auto managing its context and, when combined with how I handle memory outside of the session, I haven't thought about context window size in ages when using Codex.

I thought the price reduction was because both companies implemented Deepseek's published research which demonstrate effective approaches to lowering the cost of inference while maintaining quality. Thanks to Deepseek, they did lower their costs and they passed that in to their customers.

interesting read, thanks for sharing

"Engineering"

[flagged]

[flagged]

[flagged]