2026-09-01

A full VPS disk on Coolify: 1905 MB per rebuild, down to 31 MB

The disk didn't fill for one reason. It filled for four, and only one of them was obvious.

My VPS went from 95% disk down to 66%. I want to write down what actually happened, because it wasn't what I expected. I went in assuming the classic Docker story: old images pile up, nobody cleans them, disk fills. That was only a quarter of it. The other three quarters were mistakes I'd made myself, and one was invisible until I went looking for it specifically.

The tell that something more than "Docker is Docker" was going on: the box filled up twice in one day and killed a client site's deploy. Coolify's own cleanup runs on a schedule, so if cleanup were the whole problem it would fail slowly, not twice in a day. That gap is what sent me looking for four separate causes instead of stopping at the first one I found.

Cause one: I was pushing twice per unit of work, into a project with no build filter

I'd set up a polish loop that pushed twice per slice of work: once for the actual change, once more as a docs-only commit to update a queue file. On its own that's just a git habit. The problem was what Coolify did with it.

Coolify has a watch_paths setting meant to tell it which paths actually warrant a rebuild. Mine was null. I assumed that meant "nothing matches, so nothing rebuilds unless I say so". It's the opposite: the path matcher includes a push by default when only exclusion patterns are configured, and an empty or null config behaves the same way. Every push rebuilt the app regardless of what changed, so the docs-only commit, which touched nothing that runs, triggered a full rebuild anyway. Provably, half of all builds in that period were inert.

Cause two: the per-rebuild cost itself was 1905 MB

Frequency was one axis. The other was cost per build, the number that matters most because it's measured, not inferred: I built the image twice in a row with no code changes and diffed df before and after. First build: 1905 MB added to disk. After the fixes below: 31 MB. Same app, same code, same commit.

A trailing chown -R was rewriting the entire app into a new layer

The Dockerfile ended with a line like this:

RUN chown -R app:app /app

That single line was the biggest contributor to the 1905 MB. I don't have an exact split across this and the two causes below, so treat "biggest contributor" as an inference from the before/after diff, not a precise measured share. What it does: rewrite every file already copied into the image into a brand new layer, because chown touches the file and Docker layers are copy-on-write. Same end permissions, but you pay for the entire tree twice. Fix: never chown after the fact, set ownership at copy time instead:

COPY --chown=app:app . /app

Same end state, no duplicate layer.

Build-only dependencies were shipped in the runtime image

The API container only ever serves frontend/dist, a folder of static built assets. But the frontend's full node_modules, 354 MB of it, was sitting in the runtime image anyway, left over from building the frontend in the same stage that ran the API. Never read at runtime, and also getting swept up in the chown layer above, so paid for twice. Fix: a proper multi-stage build, a frontend-builder stage that produces dist/, and a separate final stage that copies in only that folder. node_modules never crosses into the image that ships.

Committed screenshots and concept files were copied into the image twice

_shots/ (62 MB of screenshots) and concepts/ (52 MB) were both committed to the repo and copied into the image twice, despite neither ever being read at runtime. Fix: a .dockerignore entry for each, so they're excluded from the context Docker even sends to the daemon, not just from the final image:

_shots/
concepts/

Cause three: cleanup was correct, just too infrequent for the accrual rate

This is the one that overturned my assumption that "the cleanup must be broken". It wasn't. Coolify's per-app retention (docker_images_to_keep, default 2) was doing exactly what it was configured to do: it protects the running tag, and docker rmi refuses to remove an image still in use. The tell it was working: no image on the box was ever older than about seven hours.

The actual problem was the schedule. It ran once a day:

0 0 * * *

while the box, under the doubled build frequency from cause one plus the 1905 MB cost from cause two, was accruing roughly 9 GB an hour. A cleanup that runs once a day can't keep up with an accrual rate measured in gigabytes per hour, however correct its logic is. I moved it to run every two hours instead:

0 */2 * * *

Cause four: a zombie deployment I didn't spot until I went looking

This one wasn't a disk-space cause so much as a consequence that made everything else worse. A deployment failed outright when the disk went full mid-build, leaving an in_progress row with no build container running. Coolify serialises deployments per application, so that one stuck row queued five later pushes for hours while the previously running container sat there going stale.

I found it by checking the deployment queue directly on the Coolify container:

docker exec coolify php artisan check:deployment-queue

Force-failing the stuck row doesn't drain the queue behind it, because nothing then calls Coolify's next() to move things along. The fix is to cancel the orphaned row directly and trigger a fresh deploy yourself rather than wait for the queue to notice.

The cron job that stops it coming back

Fixing the four causes got the box back to 66%. It didn't stop the next unrelated thing from creeping the disk back up over weeks, so I run a small deterministic janitor script on my own machine that reaches the VPS over SSH every 30 minutes:

ssh your-server "docker system df"
# ...decide against thresholds, then act if needed

It's deliberately not an LLM call: deciding whether to run docker rmi and doing slope arithmetic on disk-usage trend don't need a model in the loop, and a plain script can't talk itself into a prune it shouldn't run. It acts at 70% disk used, escalates by notification at 85% after cleaning, or whenever the fill trend projects fewer than 6 hours until full, whichever comes first. That six-hour projection is the check that would have caught the original incident earlier: the box filled in hours, not days.

What it is allowed to do, and what it never does:

Per-app retention is also restricted by a pattern match on the repo name, so it only ever touches images that look like a Coolify app UUID. Infrastructure containers like postgres, traefik, or a self-hosted Supabase are structurally out of reach, not just excluded by convention. It also supports a dry run:

DRY_RUN=1 ./vps-janitor.sh

What to check on your own box

Topics

Still stuck? Send it to Emsden Studio and get a straight answer on what it is and what fixing it costs.

More notes

All notes