RedwoodSDK, Finally Unlocked from Cloudflare

The last post answered whether RedwoodSDK’s dependency on Cloudflare was architectural or just a default — it’s a default. What it didn’t answer is whether a self-hosted deployment actually holds up once it’s running for real: a persistent node, a real storage backend, state that has to survive a restart. That’s the part that turns “this technically works” into “this is deployable.”

Quick reminder if you’re starting here: celld is a self-hosted, open-source reimplementation of Cloudflare’s Workers/Durable Objects platform — Workers, Durable Objects, D1, R2, KV, Queues, Workflows, running on infrastructure you control instead of Cloudflare’s. A running celld node (or a few, sharing one bucket) is called a fleet; that’s the whole system, once it’s up. Everything below assumes that much.

It mostly held up. The one place it didn’t was the storage backend — and the way it failed is worth walking through in full, because the failure mode was silent right up until it wasn’t.

Picking a storage backend got harder mid-project

celld needs an S3-compatible bucket for its state. The obvious self-hosted choice was MinIO — it’s the store celld’s own docs mention by name. It’s also, as of this writing, in the middle of unwinding its free distribution: Community Edition stopped getting new images in October 2025, and Docker Hub removed the minio namespace entirely in September 2026. What’s left are archived, pre-cutoff images with no future updates — usable, but a dead end.

Garage looked like the natural replacement: actively maintained, built specifically for self-hosting, with a straightforward Docker image. We wired it up, pointed celld at it, and it looked fine — the container started, the S3 API responded, buckets listed correctly.

The bug wasn’t in the setup

celld refused to serve from it anyway:

Error: the bucket does not keep the storage contract, so celld cannot serve
cells safely on it: the store overwrote an object although the write was
conditional on that object being absent; the store accepts If-Match /
If-None-Match and does not enforce it, so two nodes can own one cell.

celld’s entire ownership model rests on one guarantee: a conditional write — “create this object only if it doesn’t already exist” — either succeeds exclusively or fails outright. No consensus protocol, no membership service, just that one atomic operation. Garage accepts the If-None-Match header without actually enforcing it, so two nodes racing to claim the same cell could both believe they’d won. That’s not a rough edge — it’s the one thing the whole design depends on not going wrong.

In fairness to Garage, this is a genuinely new corner of the S3 API. AWS itself only shipped atomic put-if-absent in August 2024; MinIO didn’t correctly enforce it until a September 2024 release. Garage’s own compatibility docs don’t mention conditional writes at all, in either direction — this isn’t a bug so much as a feature that hasn’t landed yet.

Verifying the replacement instead of trusting the docs

SeaweedFS documents the exact primitive by name: If-None-Match: "*" as “a primitive for distributed leader election, exclusive file creation, or idempotent uploads.” That’s specific enough to sound right, and specific enough to be worth actually checking rather than taking on faith — especially right after Garage’s docs implied nothing was wrong either.

So we stood up a real SeaweedFS instance and ran celld diagnose against it — the same conditional-write probe that had just failed against Garage:

ok bucket conditional write: create, reject-create, update, reject-stale

celld’s own correctness check, passing against a live store. Not a doc claim — an actual pass.

Two Docker problems, one of them worth remembering

Getting the storage swap running in Docker surfaced two more issues, both more interesting than “we misconfigured something.”

The first: a bucket created via one container wasn’t visible to the container actually serving traffic. The cause wasn’t networking — docker-compose’s depends_on was fine. It was that the SeaweedFS image declares its own VOLUME /data, so a second container started from that image — with no explicit volume mount — silently got its own throwaway anonymous volume. The bucket-creation step “succeeded” against storage nobody else could see. The fix wasn’t a workaround; it was removing the separate container entirely and creating the bucket from inside the same process that serves the data, over localhost, where there’s no volume ambiguity to have in the first place.

The second: a local port override meant to replace 8080 with 18080 instead ran both — Compose merges list-valued keys like ports across override files by default rather than replacing them, so celld tried to bind two ports and failed on the one already taken by something unrelated. The fix is the !override merge tag, which does what most people assume depends_on-style overrides already do.

Neither of these is specific to celld. Both are the kind of thing that looks like a flaky environment until you find the one-sentence mechanism, and then it’s obvious.

What actually shipped

The final deploy path needs nothing custom — no bespoke image, no extra tooling beyond what already exists:

docker compose up -d seaweedfs
docker compose exec seaweedfs weed shell -master=127.0.0.1:9333
# > s3.bucket.create -name celld-fleet
docker compose up -d celld

Then the actual deploy, reusing the same ghcr.io/denoland/celld image already running the service, as a throwaway container on the same network:

docker compose run --rm --entrypoint celld \
  -v $(pwd)/dist:/project \
  celld deploy /project \
  --bucket s3://celld-fleet --endpoint http://seaweedfs:8333 --region us-east-1

celld picked up the deployment without a restart — the log line says as much: awaiting_initial_deployment flips to serving traffic the moment the bucket has something in it. The first thing we proved it with was as simple as a RedwoodSDK app gets: a single Durable Object counter, incremented through a real server action, in a real browser, against a real running node. It survived reloads. It survived us coming back later and finding the count higher than we’d left it, because state that persists is state someone can poke at.

That was the recipe working. What we actually wanted to put on it needed more than a counter.

A reason to actually stress the primitive

A counter proves state persists. It doesn’t prove the thing celld is really for: coordinating many clients through one authoritative process, in real time. So the thing we’re building on this fleet now is a synchronized reaction game — a light flashes at the same instant for every visitor currently on the page, timed by a single Durable Object’s own alarm, not any client’s clock, and whoever’s click message the object receives first after that broadcast wins the round.

It’s deliberately not fakeable on a static site. Every visitor needs the same authoritative “go” moment, which means the demo can’t exist without the exact primitive it’s supposed to demonstrate — one Durable Object holding hibernatable WebSocket connections open across everyone on the page, plus its own alarm driving the whole game loop. Two browsers watching the same fleet saw the flash within single-digit milliseconds of each other in testing, and both came back with byte-identical results afterward — the server decided who won, not either client’s clock.

Automating the part we’d been doing by hand

Every deploy up to this point had been us, at a terminal, running the commands above by hand. Turning it into something that ships on every push meant a GitHub Actions workflow doing exactly the same three things a person would: build, adapt the output with recell, then reach the server and run that same celld deploy command — except now over SSH, from a CI runner that had never touched this infrastructure before.

That gap — between “we know these commands work” and “a stranger’s ephemeral runner can run them too” — is where the real remaining bugs were.

The bugs a CI runner finds that a terminal doesn’t

The first: secrets were set on a GitHub environment, but the workflow’s job never declared it. secrets.* didn’t error — it silently resolved to empty strings. The tell was in the deploy step’s own printed config: target: /apps/reaction-game, with no path in front of it at all, and an SSH step failing with “can’t connect without a private SSH key or password.” Nothing was wrong with the secrets. The job just never had access to them.

The second: the deploy user could SSH in fine, but permission denied while trying to connect to the docker API at unix:///var/run/docker.sock — the same class of problem as the local sandbox we’d developed all of this in, just on the real machine now. usermod -aG docker fixed it, and — unlike a long-lived local shell, which needs a fresh login before it notices new group membership — a brand-new SSH session on every CI run picks it up immediately. No workaround needed there, just the actual fix.

The third: DEPLOY_PATH pointed at a directory that didn’t share a name with the one the persistent platform had actually been started from. Docker Compose derives a project’s identity from the directory name it’s run in, so docker compose run under a differently-named path didn’t reuse the running platform at all — it spun up an entire second copy, new network, new volumes, and collided on the port the real SeaweedFS already held.

The fourth, once that was fixed: the deploy failed once more, Connection refused reaching SeaweedFS — because the platform simply wasn’t running yet when the workflow fired. Bringing it up first (docker compose up -d seaweedfs, create the bucket, docker compose up -d celld — the manual, occasional step from earlier, entirely separate from the automated deploy) was the last piece. Once it was, the pipeline went green end to end.

The deploy command was never part of celld

One thing about that docker compose run line is easy to misread: it looks like it’s reaching into the running server and handing it a new version. It isn’t, and the distinction is worth being precise about, because it’s the whole shape of the system.

celld the binary does two almost unrelated jobs depending on how it’s invoked. Run it with --bucket and no subcommand, and it’s a fleet node — a long-running process that takes a lease, serves Workers, hosts their Durable Objects, and keeps running until it’s killed. Run it with deploy, and it does exactly one thing: write a single object into the bucket, then exit. The container that does the deploying never takes a lease, never joins the fleet, never serves a single request. It is exactly as external to the running system as aws s3 cp would be — it just happens to speak the fleet’s own upload format correctly, because it’s built from the same source.

So the fleet — the actual long-running node or nodes, sharing one bucket, serving Workers and their Durable Objects — is the entire system. That’s all celld is, at runtime. Deploying to it isn’t handing anything to that system directly; it’s writing to the one thing the system already watches. The running node picks up the new version on its own next poll of the bucket, on its own schedule, with zero coordination from whatever wrote it there — us, a CI runner, or someone doing it by hand.

One fleet, many apps, one header

The same name field that decided where in the bucket a deployment lands (s3://celld-fleet/deploy/reaction-game/...) turns out to do double duty: it’s also the only thing celld uses to pick which deployed worker answers a given request. A fleet isn’t one app — it can hold as many deployed workers as you throw at it, and the Host header on each incoming request is matched against each one’s wrangler.jsonc name to decide who answers.

That’s easy to miss entirely, because a fleet with exactly one worker deployed doesn’t need any of this. Our nginx config passed the real public domain straight through (proxy_set_header Host $host;), and it worked — not because celld was matching that domain to anything, but because there was nothing else on the fleet to disambiguate from. One worker, any Host, same answer every time.

It stops working silently the moment a second app lands on the same fleet. The public domain was never going to match either worker’s actual name, so each site’s nginx block has to hardcode the Host it sends upstream to that specific worker’s name — proxy_set_header Host reaction-game;, not $host — with X-Forwarded-Host left as the real domain, since that’s the one the app itself needs to build correct absolute URLs. Worth setting explicitly from the first app on a fleet, not just once a second one exposes that it was ever needed.

Where this leaves it

Cloudflare was never the hard dependency — the storage layer’s correctness guarantees were, and those are exactly the kind of thing that looks fine until the one operation your whole design depends on turns out to be unenforced. Verify the primitive you’re actually relying on, not the one the docs imply you’re getting.

The gap between “this works” and “this works here” did close, in the end. The reaction game is live now, at game.softwaiz.com — the same recipe from this whole post, run for real, serving real players off a self-hosted fleet with no Cloudflare in the chain anywhere.