Rendered at 02:24:48 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
glenngillen 1 days ago [-]
I'd been running into a lot of disk pressure on my system from agents running. I discovered two main causes:
- my approach to using git worktrees for all work. Not entirely surprising, the duplicated build artifacts in each worktree being the main culprit here. There were some limited gains to be had by just being more efficient with how builds were managed but ultimately I just had to limit the number of concurrent features I was working on in a couple of particularly large repos.
- more insidious was the fact I was actually hitting memory limits more than I realised and causing my mac to have to swap heavily. The system was surprisingly still quite performant while this was happening... until it wasn't. Unfortunately as best I can tell the only "fix" when this happens is to restart the machine. There's no way to have the system reclaim any of the swap space once it's written to disk. Which if you have it thrash a couple of times a day (especially easy if you leave something running overnight) adds up very quickly.
monster_truck 1 days ago [-]
It's so goddamn bad, I wish it was easier to turn all of this garbage off. I used to spend time defining parallel work lanes to issue to subagents as frontloaded development work with set validation checkpoints and it worked great. Now attempting to do this with 8, 16 subagents will balloon out into hundreds of gigs of fucking mess
Copy on write worktrees do absolutely nothing to solve this, either. The build artifacts alone will do it
nullpoint420 17 hours ago [-]
Agreed. I thought I was being smug by telling my agents to use btrfs for specific workloads. Sure, it took me longer to run out of space, but I of course ended up running out of space
gimlichgu 13 hours ago [-]
[dead]
tonytamps 14 hours ago [-]
I'll take a stab in the dark here and say that if you're using Docker (or Rancher, etc..) I found they aren't releasing memory properly and caused a similar issue to what you're describing.
Switching to Orbstack solved the problem for me because of their memory management efforts. Just wanted to mention it in case it leads to an easy fix for someone else too.
veidr 12 hours ago [-]
same here — not trying to shill OrbStack, but switching to it has "magically" solved weird problems that would otherwise have consumed my actual time to debug
tonytamps 11 hours ago [-]
It does seem like magic but I hope it's excellent engineering. It was valuable enough for me to start paying for it.
singularity2001 1 days ago [-]
switching to copy on write worktrees is the biggest game changer it should become common knowledge pretty soon and it's already available on most operating systems
thesz 20 hours ago [-]
You also need a deduplication, because with simple copy on write you will have a copy for writing the same - change a byte in a file, get a copy, then change it back, get a duplicate.
Deduplication in ZFS needs 5G of RAM for 1TB of logical data - if you expect your agents to produce 2TB of data, you should provide 10G of RAM for deduplication.
malfist 12 hours ago [-]
Why are your agents expected to produce 2TB of data? Not criticizing, I'm curious. Outside of classifiers I don't have any of my agents produce data, they produce scripts to produce data so it can be deterministic
singularity2001 5 hours ago [-]
change a byte ... change it back, get a duplicate.
What are realistic scenarios in which you change a byte back?
jwiz 21 hours ago [-]
He's saying the build artifacts themselves are consuming the space
glenngillen 21 hours ago [-]
thanks to the parent you're replying to, I discovered there's numerous projects that try to retrofit CoW into worktrees + also fill in the gaps for mostly-shared data like build artifacts! I've decided to take lane (https://github.com/lukeed/lane) for a spin as I type to see how it goes.
bentcorner 22 hours ago [-]
My work provides me a Mac laptop plus remote Windows VMs and I've completely stopped using my Mac for agent builds/tests - I give them ssh access to WSL instances and they can go hog wild on the VMs.
It's all a bit adhoc - agents still need to pull (or I need to keep the main enlistment up to date on these machines) - I'm thinking of using containers or something else like github actions keep things cleaner, but right now it's "good enough".
Regardless I'm pretty happy with how it's working so far. Previously I'd be letting agents do work in the background and they'd kick off builds in the middle of a meeting or something and I'd start dropping video streams until I killed everything.
josephg 1 days ago [-]
On Linux I’ve taken to disabling swap before doing memory heavy operations. Claude will carelessly run benchmarks and tasks which gobble up all available memory. When that happens, I’m not sure why but my system becomes unresponsive and I can’t kill the process. If I disable swap, the oom killer takes care of whatever Claude was doing and Claude figures it out after the fact.
trollbridge 1 days ago [-]
You can use systemd-oomd to force OOM killing without having to completely disable swap. It's quite handy for this especially since you can tell it to aim its OOM heavy artillery at only certain tasks (i.e. only processes that are descendants of claude-cli.)
treve 21 hours ago [-]
One thing to also look out for is some distros will but /tmp in RAM, and claude loves to fill that
machomaster 10 hours ago [-]
Using Zswap might help you.
treve 24 hours ago [-]
If git used reflink / CoW when available it would make this a lot faster.
feelamee 16 hours ago [-]
tho, I also met this problem and found a solution for me - develop all in one worktree on a single branch, then cherry-pick to the second worktree to the needed branch.
So, only first worktree is built:
- less space needed.
- less time spent on build, which is usually caused by switching branches.
- you are not required to merge/rebase of master branch to solve conflicts, which also cause long time rebuilds.
_matthew_ 1 days ago [-]
I ran into similar issues. Sometimes though there are good language specific solutions. For Rust based worktrees you should definitely check out https://mr-boxington.jdx.dev/ .
Regarding swapping, are you sure it's not just from ram backed tmpfs partition? For me that was it and most of the swap was from /tmp. So I created an agressive job that runs frequently to delete all tmp files not used in the last ~2 hours without any command holding handles to it.
The other thing I found was that there were a bunch of leftover processes from prior coding agents forking commands and just leaving them. So I had to build a system into my agent spawner to automatically launch each agent session in a cgroup whose processes all get killed on task completion.
satvikpendem 24 hours ago [-]
For the first part you might be interested in Anthropic recently creating their own version control system that's server based like SVN, and it seems to ease their memory pressure too. It's called `scion`.
CountSessine 11 hours ago [-]
It’s actually kind of startling just how bad, as measured by open issues, Codex is. There’s a huge issue with conversation locking that nullifies one of Codex’s key advantages (GUI<->CLI interoperability) and last night I learned that conversations can get into a permanently-locked state where even shutting down every Codex process won’t unlock a conversation.
You add to that all of the weird confusion over Chat/Work/Cloud Work/Work Local modes and marooned conversations, broken connectors (will OpenTable ever work? Will the Chrome connector ever be reliable on restart?) questionable UI decisions, and I start to worry a whole lot less about being replaced by AI.
CountSessine 11 hours ago [-]
It’s not even that Astra is bad at coding - it’s magnificent at coding. But it’s like it’s pure Id - all energy and first order logic but no taste or reason or preference. I guess that’s what we’re supposed to be doing now.
laurels-marts 11 hours ago [-]
Just use opus 5.5 for any code that needs to actually be merged.
CountSessine 11 hours ago [-]
And yet Astra is as good if not much better coding than opus 5.5, both in the code generated and its guardrails. In fact in the last 3 weeks at work I’ve had opus 5.5 histrionically freak out about examining security issues with our app multiple times. What good is that?
It’s just that Codex, the harness, is hot garbage. I’m going to invest some time in using Astra with OpenCode to see if I can live with that but going back to Anthropic is very low on my preference list.
jbellis 1 days ago [-]
this looks like a bug, but even normal happy-path codex creates many GB of logs, openai apparently considers this fine bc it's been going on for months
two things help:
1. fs-level compression, I use ZFS bc Btrfs has massive contention problems with concurrent Rust builds
I recently was hunting for some storage on my ssd and noticed had 14GB of tiny files in codex in /.staging/marketplace. They really don't care about our drives. Once they fix one things they break another stuff.
boesboes 16 hours ago [-]
The amount of cope of people dealing with this is sobering. One of the biggest companies out there currently has neglected to fix a seemingly trivial bug in their core offering for 4 months and everyone is just like 'well, you can delete the files'.. I'm not hopeful for the future. Software already was a terrible buggy mess, now it's just all rng i guess.
unrented7977 1 hours ago [-]
We had these kinds of problem pre-AI, too. Companies behaved the same way when these bugs were written by humans. Nothing new is happening here.
squidbeak 16 hours ago [-]
The two can be true at once. OpenAI can be remiss leaving it this way so long, and deleting the files can be a practical workaround.
Mole author here, thanks for the kind words. Happy to hear what else it should clean up.
columk 1 days ago [-]
Yeah mole is brilliant, I just discovered it days after paying for Daisydisk and haven't used Daisydisk since. Mole freed up 80GB on my laptop the first time I ran it.
piskov 15 hours ago [-]
To be fair these are different tools.
Back in the days you would use cleanmymac for emptiying caches and what have you, and daisydisk to find big folders
rudedogg 1 days ago [-]
DaisyDisk is such a nice app. I just checked and I purchased it over 11 years ago.
They still update it, it works perfectly, no version 2.0 forced obsolescence. Just a nice, reliable tool.
mococa 15 hours ago [-]
+1 for DaisyDisk
clumsysmurf 1 days ago [-]
DaisyDisk is great. Mole also found and deleted a bunch of stuff on my mac from Claude, Gemini, etc.
I’ve been using it altogether too much this last month and just bought a 4TB USB4 SSD.
And yes, codex is eating my disk, too. Ironically my solution is mostly codex - I have a task which goes around stomping on detritus.
starfallg 1 days ago [-]
I'm working on a side project that requires a lot of fine tuning of 3D worlds. Codex ended up creating thousands of artifacts, basically loads of screenshots, in its worktrees, and doesn't clean up after itself.
On a work project, we're moving financial processes from Excel to code, and Codex ended up creating and storing thousands of validation tests unnecessarily. That ended up eating 90GB after we let it loose on a Windows VM that we connected with a custom agent between codex and the VM.
This is definitely a Codex thing and I've never seen this happen in Claude Code or Antigravity.
zamadatix 1 days ago [-]
It's silly I have to tell it to make part of the build process to clean old artifacts or it'll just default to keeping endless folders of test builds and old builds without cleaning them up. The same seems to usually be true with documentation and reference material.
There was a REALLY annoying period where I'd give it full control of a VM and a dedicated project folder and it'd continually refuse to delete anything I asked it to or just silently archive it, even though everything was already in a git repo. It seems to have gotten somewhat better about that now, but I still include "create cleanup scripts to delete unnecessary project content" in the default steps before doing the final merge on a change.
spockz 20 hours ago [-]
Codex happily removes things I ask it to all the time. Do you have a special phrasing in agent instructions such as AGENTS.md?
zamadatix 17 hours ago [-]
Prior to the above I never had an agents.md or memory enabled.
majormajor 1 days ago [-]
Heh, Claude recently (last couple of weeks) left behind several jj workspaces with many GBs of builds in my env.
Had to tell it to update it's agents.md to not do that...
plantain 1 days ago [-]
Codex is obsession with validation tests. It drives me insane.
vikramkr 14 hours ago [-]
That's the only reason these models are good though. They don't one shot things. They get it wrong and then keep checking until they get it right. You gotta let them test
mattgreenrocks 1 days ago [-]
Agents prefer some sort of tests to check their output.
tonyarkles 1 days ago [-]
I would argue that the better you can “close the loop” the better. One of my favourite agent activities is giving it a firmware bug and JTAG access + a serial port. That setup means that flashing, debugging (inspecting variables and whatnot, setting breakpoints, etc), and logging can all happen completely autonomously.
Works especially well for transient/hard to repro bugs. It’ll either set breakpoints or write a monitoring script, go to sleep, get woken up by the harness when the bug trips, inspect state, put together a new build based on a hypothesis built from the debugging, flash it, and loop. I’ve had sessions run completely autonomously for over a day and come with a beautiful one-line solution to a gnarly race condition.
In that regard it’s like a human: you can read code all day if you want and you might get somewhere, but being able to inspect what happened, build a new theory, test it, and iterate is way more powerful 99% of the time.
bentcorner 22 hours ago [-]
This kind of approach works so well.
I was trying to upload HDR photos to Instagram and couldn't figure out how to make it preserve the HDR-ness of my photos, and was going back and forth with an agent - step by step we'd slowly iterate on a photo, I'd upload, then I'd download the jpg and give it to the agent and it'd think, change something in the format, and we'd try again.
I got impatient and just said fuck it and gave it chrome-devtools and a blank instagram account and a test photo, and it figured the whole thing out in about a half hour. I built a little browser-based tool from what it discovered, and I use it to run my HDR photos through to format them correctly for instagram and it works a treat.
cyberax 21 hours ago [-]
> I built a little browser-based tool from what it discovered, and I use it to run my HDR photos through to format them correctly for instagram and it works a treat.
I was kinda expecting: "And when my agent built a perfect clone of Instagram and social-engineered all my contact list to join"
giancarlostoro 1 days ago [-]
Rust is notorious for this, one thing some people do is force Rust to have a salvage yard for builds so all those dependencies are shared by everything regardless of what you are building or where, but holy moly, can someone make Rust output smaller files? Hard drives are always having some reason to never go down to sane prices.
When I was younger I thought we would be having exabyte hard drives by now.
kirykl 15 hours ago [-]
They present and immediately take away their own value proposition. Do AI CEOs also buy tokens, or is that the reason for human employees ?
>"One possibility is single-person startups, each with a human founder steering swarms of agents to rake in billions in revenue. Another is companies built by AI genius CEOs employing human agents only to wield pipettes and testify before Congress."
adamddev1 1 days ago [-]
Imagine if the vibe-coders get their way and we see this level of bloat and bugginess at EVERY level of the software stack. Everything would collapse.
raegis 1 days ago [-]
Before Windows XP, desktop Windows users were accustomed to frequent system crashes. Viruses in the 1990s would take out offices around the world. Nothing collapsed, and nobody switched to Linux. Everyone was just used to it. I think most people will accept the tradeoffs, because it used to be much worse than it is now.
adamddev1 17 hours ago [-]
I meant also in terms of space and time complexity with the slower, bloatier software at every level, things would also freeze and fill up.
What people can get away with on the upper crust with end-user apps would spell disaster for the inner parts.
steve1977 19 hours ago [-]
> desktop Windows users were accustomed to frequent system crashes
Home users maybe. I've started with Windows NT 4.0 and don't remember it being crash-prone.
machomaster 10 hours ago [-]
NT was not for home users. And it's USP was that it was way more stable. Then Win 2000 (NT variant) came and at XP they finally merged.
hluska 15 hours ago [-]
That’s why people used NT - it didn’t crash as often.
AIorNot 10 hours ago [-]
Eh, that was said about Javascript, electron, node, etc
The world adapted, that is to say if something is popular- they will keep it going through just enough fixes to keep it going
sick_of_slop 1 days ago [-]
[dead]
vikramkr 14 hours ago [-]
They've had their way since before AI was a thing and it was fine and it still is. You can spend the tokens you have on optimizing the performance of your fork of whatever software you're running. For large categories of software it's mattering less and less what other people choose to prioritize because you can choose your own priorities when you roll your own version
piskov 1 days ago [-]
btw there is a storage section in Chatgpt app settings, showing used space on disk and other related stuff
dingaling911 1 days ago [-]
I'm so glad my first move was to relegate these harnesses to docker containers.
1 days ago [-]
AgharaShyam 1 days ago [-]
look into ~/.bun/install/cache/@openai folder as well.
for some reason I had 11 separate codex and codex-code-mode-host binaries in mine. each weighing about ~300 MB.
farmin 16 hours ago [-]
Is it just me or is codex on windows slow and buggy for general work tasks? I can’t no matter what get it to control its own browser and fan spins out of control on decently new laptop.
I tried Claude and it’s much better experience, got a bookkeeping workflow setup with xero mcp and reading my emails finding invoices in no time.
Codex has been great for plain coding jobs though.
HackerThemAll 19 hours ago [-]
Prompt it to create a bash script that will clean those directories and schedule it every few minutes.
Apparently OpenAI is unable to fix this properly, even using its state of the art AI.
AIorNot 1 days ago [-]
If OpenAI's super-intelligence can solve Navier-Stokes why can't they fix this stupid bug that ate up 330GB of my Mac's disk space? come one please triage your github issues?
>
If OpenAI's super-intelligence can solve Navier-Stokes why can't they fix this stupid bug that ate up 330GB of my Mac's disk space?
Because the former can be marketed very prestigiously, the latter hardly can.
According to https://www.briefia.fr/en/article/openai-revendique-la-resol... it is estimated that solving Navier-Stokes did cost OpenAI 40 million USD. I would bet that if OpenAI spent the same amount of money using its super-intelligence to fix this bug, this would work.
But seems not to be very economical. :-)
topaz0 12 hours ago [-]
The navier Stokes work was just preparation for this finite time disk usage blowup (with smooth periodic forcing)
spicyusername 22 hours ago [-]
The term super intelligence makes me cringe so hard, you can just use the regular term.
You don't have to give any more legitimacy to Trump's inane ramblings than they already get.
mrlonglong 18 hours ago [-]
Also insane ramblings too.
cyanydeez 1 days ago [-]
because they'd need to pay a real mathematician to verify the proof they claim.
AIorNot 17 hours ago [-]
Well looks like my post hit the frontpage of HN, so maybe openai will fix the bug :)
hluska 15 hours ago [-]
So, you’re using all of us for technical support purpose? Good to know - I wish I wouldn’t have wasted my time reading this shit.
vikramkr 14 hours ago [-]
The post is about exactly what the title says its about and also it is literally a link to a GitHub issue if you didn't want to read a big report and couldn't figure out that it was a bug report until "wasting your time" reading it and then a bunch of comments and then finally a comment saying hopefully openai will take it seriously idk what what to day
kudokatz 1 days ago [-]
Are we really surprised that stuff like this happens with AI slop?
matt3210 1 days ago [-]
Good example of SLOP coding from openAI. No wonder they cant keep agents from committing felonies.
throwaway613746 1 days ago [-]
[dead]
shivam_techrevi 18 hours ago [-]
[flagged]
simonchai-tw 21 hours ago [-]
[flagged]
K-etoranger 24 hours ago [-]
[flagged]
skillstackhq 19 hours ago [-]
[flagged]
throwaway613746 1 days ago [-]
[dead]
verytrivial 12 hours ago [-]
Wow. I've tried a bunch of agent harnesses and Codex Desktop was on my list to try. It has definitely fallen off!
- my approach to using git worktrees for all work. Not entirely surprising, the duplicated build artifacts in each worktree being the main culprit here. There were some limited gains to be had by just being more efficient with how builds were managed but ultimately I just had to limit the number of concurrent features I was working on in a couple of particularly large repos.
- more insidious was the fact I was actually hitting memory limits more than I realised and causing my mac to have to swap heavily. The system was surprisingly still quite performant while this was happening... until it wasn't. Unfortunately as best I can tell the only "fix" when this happens is to restart the machine. There's no way to have the system reclaim any of the swap space once it's written to disk. Which if you have it thrash a couple of times a day (especially easy if you leave something running overnight) adds up very quickly.
Copy on write worktrees do absolutely nothing to solve this, either. The build artifacts alone will do it
Switching to Orbstack solved the problem for me because of their memory management efforts. Just wanted to mention it in case it leads to an easy fix for someone else too.
Deduplication in ZFS needs 5G of RAM for 1TB of logical data - if you expect your agents to produce 2TB of data, you should provide 10G of RAM for deduplication.
What are realistic scenarios in which you change a byte back?
It's all a bit adhoc - agents still need to pull (or I need to keep the main enlistment up to date on these machines) - I'm thinking of using containers or something else like github actions keep things cleaner, but right now it's "good enough".
Regardless I'm pretty happy with how it's working so far. Previously I'd be letting agents do work in the background and they'd kick off builds in the middle of a meeting or something and I'd start dropping video streams until I killed everything.
So, only first worktree is built: - less space needed. - less time spent on build, which is usually caused by switching branches. - you are not required to merge/rebase of master branch to solve conflicts, which also cause long time rebuilds.
Regarding swapping, are you sure it's not just from ram backed tmpfs partition? For me that was it and most of the swap was from /tmp. So I created an agressive job that runs frequently to delete all tmp files not used in the last ~2 hours without any command holding handles to it.
The other thing I found was that there were a bunch of leftover processes from prior coding agents forking commands and just leaving them. So I had to build a system into my agent spawner to automatically launch each agent session in a cgroup whose processes all get killed on task completion.
You add to that all of the weird confusion over Chat/Work/Cloud Work/Work Local modes and marooned conversations, broken connectors (will OpenTable ever work? Will the Chrome connector ever be reliable on restart?) questionable UI decisions, and I start to worry a whole lot less about being replaced by AI.
It’s just that Codex, the harness, is hot garbage. I’m going to invest some time in using Astra with OpenCode to see if I can live with that but going back to Anthropic is very low on my preference list.
two things help:
1. fs-level compression, I use ZFS bc Btrfs has massive contention problems with concurrent Rust builds
2. Archive your old sessions using something like sessionwiki (https://github.com/youdie006/sessionwiki), I've integrated this into my control plane (https://github.com/BrokkAi/mjolnir/)
I haven't read about btrfs having issues with concurrent workloads like multiple concurrent rust builds. Are there notes on this somewhere I can read?
https://news.ycombinator.com/item?id=48626930
TUI version is free, though there is a paid app if that’s your thing
https://github.com/tw93/Mole
Back in the days you would use cleanmymac for emptiying caches and what have you, and daisydisk to find big folders
They still update it, it works perfectly, no version 2.0 forced obsolescence. Just a nice, reliable tool.
https://github.com/tw93/Mole
https://github.com/jesusha123/qdirstat-macos
And yes, codex is eating my disk, too. Ironically my solution is mostly codex - I have a task which goes around stomping on detritus.
On a work project, we're moving financial processes from Excel to code, and Codex ended up creating and storing thousands of validation tests unnecessarily. That ended up eating 90GB after we let it loose on a Windows VM that we connected with a custom agent between codex and the VM.
This is definitely a Codex thing and I've never seen this happen in Claude Code or Antigravity.
There was a REALLY annoying period where I'd give it full control of a VM and a dedicated project folder and it'd continually refuse to delete anything I asked it to or just silently archive it, even though everything was already in a git repo. It seems to have gotten somewhat better about that now, but I still include "create cleanup scripts to delete unnecessary project content" in the default steps before doing the final merge on a change.
Had to tell it to update it's agents.md to not do that...
Works especially well for transient/hard to repro bugs. It’ll either set breakpoints or write a monitoring script, go to sleep, get woken up by the harness when the bug trips, inspect state, put together a new build based on a hypothesis built from the debugging, flash it, and loop. I’ve had sessions run completely autonomously for over a day and come with a beautiful one-line solution to a gnarly race condition.
In that regard it’s like a human: you can read code all day if you want and you might get somewhere, but being able to inspect what happened, build a new theory, test it, and iterate is way more powerful 99% of the time.
I was trying to upload HDR photos to Instagram and couldn't figure out how to make it preserve the HDR-ness of my photos, and was going back and forth with an agent - step by step we'd slowly iterate on a photo, I'd upload, then I'd download the jpg and give it to the agent and it'd think, change something in the format, and we'd try again.
I got impatient and just said fuck it and gave it chrome-devtools and a blank instagram account and a test photo, and it figured the whole thing out in about a half hour. I built a little browser-based tool from what it discovered, and I use it to run my HDR photos through to format them correctly for instagram and it works a treat.
I was kinda expecting: "And when my agent built a perfect clone of Instagram and social-engineered all my contact list to join"
When I was younger I thought we would be having exabyte hard drives by now.
>"One possibility is single-person startups, each with a human founder steering swarms of agents to rake in billions in revenue. Another is companies built by AI genius CEOs employing human agents only to wield pipettes and testify before Congress."
What people can get away with on the upper crust with end-user apps would spell disaster for the inner parts.
Home users maybe. I've started with Windows NT 4.0 and don't remember it being crash-prone.
The world adapted, that is to say if something is popular- they will keep it going through just enough fixes to keep it going
for some reason I had 11 separate codex and codex-code-mode-host binaries in mine. each weighing about ~300 MB.
Apparently OpenAI is unable to fix this properly, even using its state of the art AI.
See this: https://github.com/openai/codex/issues/25921#issuecomment-62...
Because the former can be marketed very prestigiously, the latter hardly can.
According to https://www.briefia.fr/en/article/openai-revendique-la-resol... it is estimated that solving Navier-Stokes did cost OpenAI 40 million USD. I would bet that if OpenAI spent the same amount of money using its super-intelligence to fix this bug, this would work.
But seems not to be very economical. :-)
You don't have to give any more legitimacy to Trump's inane ramblings than they already get.