Limits

Quotas, fleet bounds, rate limits, and the scale-to-zero sleep mechanism.

Quotas

Fleet bounds

Every app is its own celld fleet on one host, each in its own cgroup with its own memory.max. The bounds stop one account — or one busy app — from owning the machine. A fleet that hits its cap is OOM-killed inside its own group and restarted, and the kill records a tenant-visible fleet event and fleet log line (fleet restarted: memory limit … MiB exceeded (Settings → Limits)). The cgroups need the multi-only grant (docker/compose.multi.yaml); without it fleets share the container cgroup and multi stays at 503.

Rate limits

The edge (Caddy) limits every site before a request reaches a worker; the control UI adds finer limits of its own on sign-in routes. Requests per minute, 0 disables any of them:

An app admin can override both app limits (per visitor and whole app) in the app’s Settings → Limits. The edge counts over a sliding window whose whole-event budget never exceeds the configured rate: 1800/min lets 300 requests through in any 10 seconds, and below 6/min the window widens so a small limit stays small (1/min is one request per 60 s). Retry-After carries the real remaining time of the limit that refused. Per-visitor limits group IPv6 addresses by /64. Behind a proxy, see Edge protection.

The edge also bounds each request before a worker sees it: request headers past 16 KiB answer 431, header fields whose names contain _ or . are dropped (a backend that maps -, _ and . to one name would read them as another header), and a body upload or response write that makes no progress for a minute closes the connection. A pause between writes, like a quiet event stream, does not count.

Sleep (scale to zero)

An app idle past its threshold stops costing anything; the next request brings it back transparently (a cold wake measures ~1.3 s). The platform threshold is RUNNER_SLEEP_AFTER_MIN minutes, or RUNNER_SLEEP_AFTER_H hours — minutes win when both are set. Unset, it derives from detected memory: 15 minutes at ≤ 2 GiB, 6 hours at ≤ 4 GiB, else 24 hours. 0 on either disables sleeping install-wide and no per-app override re-enables it; the longest window is 2592000 seconds (30 days) — a larger value clamps there with a boot warning. The sweep runs a quarter of the threshold apart (5 minutes on a ≤ 2 GiB host, an hour on the longer tiers), so an app parks close to its threshold.

Per-app override. Settings → Limits → Sleep after sets this app’s threshold in seconds (0 = never, valid range 60–2592000 = 30 days); empty falls back to the install default. Preview environments use their owning app’s override.

Scheduled work never auto-sleeps. An app whose last deploy uploaded a Wrangler config with triggers.crons or queues.consumers skips the sweep unless its owner set an explicit override — a sleeping scheduler would silently miss its triggers. Memory-pressure parking below still parks it; the next request wakes it.

Sweep. Every RUNNER_SLEEP_SWEEP_S seconds — unset, a quarter of the threshold, clamped 300–3600 — the runner parks apps that are deployed, desired running, awake, and request-free for their window.

Activity is the newest of: the last hour bucket with a fleet-served request in app_hour (edge answers — the :80 redirect, a gate refusal, a rate-limit 429, the starting page — never count), the app’s last wake (woke_at), its last deploy. A fresh deploy or manual start gets a full window.

A lossy log feed never parks. The access log is the activity feed, and its tail drops a whole backlog it cannot catch up on (over 8 MiB unread in one tick). Each such skip is recorded; while it falls inside an app’s sleep window, the sweep refuses to park that app on idleness — the dropped lines may have been its requests. Explicit sleeps (API, operator) and memory-pressure parking are never blocked, and a skip older than the window no longer matters.

Asleep is not stopped. desired_state stays running; sleep is its own column asleep_since, and status reads sleeping. A stopped app never sleeps or wakes; stopping an asleep app clears the flag. Going to sleep: mark asleep → rewrite Caddy sites to wake-on-demand → stop the fleet with normal SIGTERM drain.

Wake flow. Asleep sites route to the fleet port preceded by forward_auth to GET /v1/edge/wake. Caddy holds the request, body included, while the runner:

1. Resolves the app from X-Forwarded-Host (tenant slug or custom domain).
2. Clears the flag, sets woke_at.
3. Spawns the fleet immediately.
4. Waits for /.well-known/celld/health 200, bounded by RUNNER_WAKE_TIMEOUT_S (default 120).
5. Rewrites the Caddyfile back to the plain proxy.
6. Answers 200; Caddy proxies the held request as if the app never slept.

Concurrent requests share one wake (per-app lock), and at most RUNNER_WAKE_CONCURRENCY apps (default 4) cold-start at once; the rest queue within the same timeout. Only a failed wake answers 503 — and records a tenant-visible wake failed fleet event plus the wake_failed sleep counter. A deploy, rollback, or web commit to an asleep app wakes it first; TLS ask treats asleep apps as live so certificates renew. First-request cost after a quiet day: one cold fleet start (celld boot + ready gate, typically seconds).

Memory pressure

The runner samples memory use every RUNNER_PRESSURE_SWEEP_S seconds (default 30): the container’s memory charge when the container is bounded, or host used memory when it is unbounded. At or above RUNNER_PRESSURE_PCT of the detected limit (default 85; 0 disables), it parks the least-recently-active app that has been idle for RUNNER_PRESSURE_IDLE_S seconds (default 300), stopping at 10 percentage points below the trigger. A parked app wakes on its next request like any sleeping app, and an app with an in-flight deploy is never chosen. Setting RUNNER_PRESSURE_PCT=0 turns the guard off; the fleet and deploy bounds in the tables above are the first line, and this is the last one before the kernel OOM-killer.

/ready and GET /v1/admin/stats report memory.scope (container or host), current_bytes, limit_bytes and used_percent. A bounded container is measured against its own cgroup; an unbounded one is measured against the host (MemTotal - MemAvailable), because RustFS, dockerd and the system live outside the container — the host figure is what the OOM killer uses. With per-fleet cgroups, stats also lists each fleet’s own charge under fleet_memory. make usage diffs them over a window.