Eight gigabytes still wasn't enough

In the previous post, I finally got Omnigrex to finish its first contribution to itself. One problem was still waiting for me: the agent processes kept running out of memory.

I gave them more. Then more again. Eventually, even roughly 8 GB per agent container wasn’t enough to finish reliably.

I was struggling to finish anything and starting to wonder how the earlier work had succeeded at all. Could previous work somehow affect what was happening now?

Maybe I should replace the agent runtime?

At first, I suspected the processes the agent was starting. Compiling Go code and running tests use memory too. I didn’t have enough visibility to confidently separate that from the agent runtime itself.

Then I started suspecting OpenCode. I found reports from other users about high memory usage, which made that explanation feel more plausible. I was already considering Pi Agent as a lighter alternative: fewer features, but perhaps a better fit for running agents autonomously on a server.

Normally, I would focus on getting better measurements before replacing a major component. But with agents available to help implement the replacement, starting that work was surprisingly tempting. Apparently, making a large change easier doesn’t necessarily make it the right next step.

Meanwhile, the failures did have one useful side effect. They kept exercising Omnigrex’s recovery paths, and I fixed problems I would probably have encountered later anyway. Although, if I replaced the runtime, I wasn’t sure how much of that work would carry over.

Three changed files. Thousands of recorded changes.

The investigation eventually pointed to the saved Agent Session.

In one examined session, the Developer had changed three source files. OpenCode’s session summary contained more than 7,000 file changes. Almost all of the rest were Go build-cache files and downloaded dependencies.

Those files had ended up inside the repository workspace.

Wait, what? How was that possible? I had explicitly configured Go’s environment variables to point outside the workspace. Why were those files ending up there anyway?

OpenCode recorded those files as changes, then repeatedly stored the entire accumulated list in session summaries and the event log. The result was about 1.7 GB of session data on disk. Just exporting that saved session took roughly 5 GB of memory in a separate test, without running the agent’s development work.

So my question about previous work affecting later attempts wasn’t as strange as it had sounded. Omnigrex was continuing the same Agent Sessions. Starting a fresh container didn’t give it a fresh history. The accumulated data came back with it.

Increasing the limit gave that data more room, but didn’t stop it accumulating.

The environment was part of the problem

So how could the caches end up in the workspace when I had configured them to live elsewhere? The environment variables were starting settings, not boundaries the agent couldn’t change. They also didn’t guarantee that the locations I had chosen were usable for the work.

The agent containers were deliberately restricted. Most of the filesystem was read-only, temporary storage was small, and Go’s usual cache locations weren’t all usable. The repository workspace was writable.

In an earlier attempt, I could see the agent explicitly overriding those settings for a command, redirecting Go’s caches into the workspace after hitting access restrictions. That was a clever workaround: it found somewhere writable and changed the tool configuration to use it. It hadn’t solved everything, since temporary build files still ran out of space, but it was trying to get past the environment problems rather than simply stopping. That wasn’t proof of how every affected session got there, but it showed a concrete way the environment encouraged the workaround. For the largest failing sessions, the exact source of the redirects wasn’t established.

The outcome was clear, though: files needed to build and test the code were mixed into the directory being tracked as the agent’s work. OpenCode’s repeated recording of those changes turned that into a much larger memory problem.

I had been looking at a process consuming too much memory. Understanding why meant looking at the environment I had given it, the files being created, and what survived between attempts.

A healthier graph

The fix was to provide suitable locations outside the repository workspace for tool caches and temporary files. They were disk-backed and isolated between assignments, so the agent could build and test without putting that data among its source changes.

That prevented new cache files from entering the change history through this route. It didn’t make existing inflated sessions smaller; those still needed to be retired or reset.

The later memory graph looked much healthier:

This isn’t a controlled benchmark or proof that every memory problem is gone. But I had a reproduced cause, a fix aimed at it, and a much more encouraging observation afterward. Replacing OpenCode no longer looked like the obvious answer to this particular problem.

What stayed with me was that the agent’s resourcefulness was both useful and a source of unexpected behavior. Finding somewhere writable solved the immediate problem, but putting those files in the workspace had consequences for session history and memory use. The agent had found a way around an obstacle I had put in its environment, with a side effect I hadn’t anticipated.

A lot of agent-assisted development still happens directly on developers’ machines, sometimes with broad access and few guardrails. That gives a resourceful agent plenty of options for getting things done. It also means we may not notice what it changes along the way.

That’s part of why I’m interested in running agents in isolated, controlled environments. I want them to be resourceful, but within boundaries I can understand, with enough visibility to see what they’re doing. This experience showed that setting those boundaries takes more than restricting access: the environment also needs to give agents a sensible way to build, test, and do their work.

I think projects like Omnigrex could make these controlled environments practical, without every developer having to design and debug their own setup. I’m still learning what that requires. This was one very concrete lesson.

For now, I’ll take the healthier graph. Fingers crossed for the next Work Item.

Discuss this post on Mastodon