Storage

Object-store backends, volumes, snapshots, and the telemetry pipeline behind a Noite install.

Backend

celld qualifies Amazon S3, Cloudflare R2, Google Cloud Storage, Tigris, and Azure Blob Storage; the store must provide conditional writes, read-after-write consistency, and ranged reads. celld runs a storage-contract check at node startup — a contract violation stops startup, an ambiguous transport error warns — so a store that fails the contract stops loudly instead of corrupting state later. MinIO community edition passes celld’s storage test but is not qualified for production.

Store connection variables (S3_ENDPOINT, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, NOITE_S3_BUCKET) are documented in Environment variables.

Scoped bucket keys per store

Fleets and release commands hold keys that reach only their own prefix (fleets/{slug}/, or envs/{slug}/{label}/ for a preview environment). NOITE_CREDENTIAL_PROVIDER (auto default) decides how, per store:

• Bundled RustFS and MinIO (auto, or minio to force): keys are minted through the MinIO-compatible admin API. Nothing to configure.
• AWS S3: set NOITE_CREDENTIAL_PROVIDER=aws. The keys above must be IAM-capable (the runner creates one IAM user per app and environment, with an inline prefix policy, and mints an access key for it). Every user is stamped with this install’s tag — a short hash of the bucket and region shown as install_tag in /ready — in its name (noite-<tag>-…) and its path (/noite/<tag>/), so two installs on one AWS account mint different users and never revoke each other’s keys. At boot and every ten minutes the runner also reconciles the store against its credential rows (ListUsers, paginated): any tagged user under the install’s path with no row — a mint whose row never landed — is revoked, so a lost row can no longer leave a live key behind. When a user name collides, the runner reads it back first (GetUser) and only clears a user under its own path; anything else fails the mint loudly instead of deleting a foreign user. Without it, fleets keep the root keys as an explicit root mode. Minimal policy for the runner’s keys: user administration on the names it mints (including iam:GetUser for the collision check), plus listing users (IAM allows no narrower resource for this call), plus the data permissions the bucket layout needs:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "iam:CreateUser",
        "iam:DeleteUser",
        "iam:GetUser",
        "iam:PutUserPolicy",
        "iam:DeleteUserPolicy",
        "iam:CreateAccessKey",
        "iam:ListAccessKeys",
        "iam:DeleteAccessKey"
      ],
      "Resource": "arn:aws:iam::<account-id>:user/noite-*"
    },
    {
      "Effect": "Allow",
      "Action": ["iam:ListUsers"],
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject"],
      "Resource": "arn:aws:s3:::<bucket>/*"
    },
    {
      "Effect": "Allow",
      "Action": ["s3:ListBucket", "s3:GetBucketLocation"],
      "Resource": "arn:aws:s3:::<bucket>"
    }
  ]
}

• Cloudflare R2: R2 API tokens scope to buckets, not prefixes, so there are no long-lived prefix-scoped keys. Fleets keep the root keys as an explicit root mode; the bucket layout stays one bucket per install. If that is not acceptable for your tenants, move the install to MinIO or S3.
• Anything else: auto probes the store for the MinIO admin API once the bucket answers. If it answers, keys are scoped; if not, fleets keep the root keys as an explicit root mode, named in the boot log, /ready and make doctor.

Install tag and the orphan sweep. With scoped keys (MinIO/RustFS or AWS), every key this install mints is stamped with its install_tag: an 8-hex hash of the store endpoint, bucket and region, shown in /ready and make doctor. On AWS the tag is in the user’s name and path; on MinIO/RustFS it’s in the key’s policy name. The sweep only ever revokes keys carrying this install’s tag and no credential row.

• One bucket serves one install. Two installs with the same endpoint, bucket and region get the same tag, so they would revoke each other’s rowless keys. They would also fight over runner/state/. Give every install its own bucket.
• Spelling never re-tags. Whitespace, case, a trailing slash and the scheme’s default port (:80, :443) are normalized away.
• A real endpoint, bucket or region change does re-tag. Keys minted under the old tag then read as foreign: the sweep never revokes them, and deleting their app still does. Point the install back, or delete them at the store by hand.
• Older keys are never swept. Keys minted before tags existed (alpha.8 and earlier) carry no tag. The sweep never revokes them; deleting an app still revokes its keys. Remove genuine leftovers by hand.

Volumes

Bucket prefixes under the single NOITE_S3_BUCKET: git/ (tip bundles), fleets/ (tenant celld), envs/<slug>/<label>/ (one preview environment’s celld and storage), control/ (UI worker, its D1 and the control fleet’s telemetry), runner/state/ (runner snapshots).

Snapshots

The runner snapshots its SQLite into the bucket at most every 15 s once changes settle (and on every graceful stop), and restores it on boot when the volume has none, so the bucket alone is enough to rebuild an install: mirrors rehydrate from the tip bundles and fleets from their prefixes. docker compose down keeps volumes; docker compose down -v deletes them. With your own bucket, delete its objects at the provider.

The loss window on an ungraceful crash is about 22 s of API mutations, as long as uploads succeed. While uploads fail, every change since the last successful upload is at risk. /ready reports the snapshot status (snapshot: still the bucket owner, seconds since the last upload, last upload error), and make doctor prints it. A claim() writes runner/state/owner.json at boot and every upload re-checks it first, so a replaced runner stops uploading instead of overwriting its successor — a fence rather than a conditional write, because not every S3 store supports those. Boot with an empty volume downloads the snapshot before connecting; any error other than not-found retries and then fails the boot, because an empty database would overwrite the good snapshot. Restoring an older snapshot revokes keys minted after it: their credential rows are gone, so the next orphan sweep treats those live fleet keys as orphans and deletes them — restart fleets after a restore (or expect one revoke-and-re-mint cycle).

Telemetry

Tenant fleets run CELLD_OTEL=1 with celld’s bucket sink: Parquet under telemetry/traces/ and telemetry/logs/ in the fleet bucket, no collector service. The flush interval comes from RUNNER_OTEL_FLUSH_MS (30 s by default) and retention from RUNNER_TELEMETRY_RETENTION_DAYS (30 days), which also bounds the runner’s hourly-counter prune.

Hourly counters

Next to the Parquet ingest, the runner keeps one row per fleet per hour in metrics.sqlite: app_hour holds the request count, the 5xx error count, p50 and p95 latency, peak and mean memory, CPU time and the newest request stamp for (app, environment, hour), and app_hour_lat holds the fixed log-scale latency histogram those percentiles are derived from (one row per bucket hit, so an hour is bounded by a couple of dozen rows instead of kept samples).

Requests, errors and latency come from the Caddy access log — one JSON line per request, already tailed for the error index — so they need no Parquet read at all. Only lines a fleet’s own reverse_proxy answered count: the edge logs the proxy’s upstream next to the route’s fleet upstream, so the :80 redirect, a gate refusal, an edge rate-limit answer and the edge’s starting page never look like traffic (and never keep an idle app awake). Memory is the fleet’s cgroup charge sampled every reconcile tick (peak and mean over the hour, the _control fleet included), and CPU is the same /proc delta the hourly counters carry. The tail advances its offset only after a line’s rows commit, and every write replaces the fleet’s open hour from memory rather than incrementing it, so a retried line is counted exactly once; a restart mid-hour re-seeds from the stored row and continues from there. Retention is the same RUNNER_TELEMETRY_RETENTION_DAYS window.

The control fleet’s own telemetry

The control UI is a celld fleet too, and it runs the same telemetry environment as a tenant (RUNNER_OTEL_FLUSH_MS, RUNNER_TELEMETRY_RETENTION_DAYS) with its own bucket prefix, so celld writes under s3://<bucket>/control/telemetry/{traces,logs}/<node>/<yyyy>/<mm>/<dd>/<hh>/. It is ingested, compacted, pruned and error-grouped exactly like a tenant fleet, keyed on the reserved _control slug, and served on the same runner routes (/v1/apps/_control/metrics|spans|logs|errors) and the UI’s SSE streams.

There is deliberately no app row for _control (_control is reserved at create): nothing that iterates apps — reconcile, the Caddyfile writer, scale-to-zero, purge, backup, GET /v1/apps — ever sees the control plane. The UI shows the surfaces only to a real instance admin, never an impersonated session. Control traffic includes the dashboard’s own polling and event streams, so those numbers are real load, not a baseline to subtract.

--slug _control is the way to replay just the control fleet’s window (see below).

Frequent flushes need a compaction job — queries grow slow within hours otherwise, and celld does not compact its own files. The runner runs it hourly via host/metrics/compact.rs::compact_fleet: for the hour that just ended, per node directory, the runner folds that directory’s objects with its own Parquet writer into one zstd compacted.parquet — union by name, so an hour that spans a schema change (the docs call the schema v0-unstable) keeps both shapes and the unsigned column annotations — then the source files are deleted. Deletion happens only after head-object proves the compacted file exists, so a failed copy can never lose spans. An hour whose fold could not produce a file the ingest accepts (inputs past the 160 MiB fold budget, or an output past the 256 MiB object cap) is left raw: its sources stay readable, the ingest reads them, and later hours are not blocked. The current hour is never compacted.

Runner queries are hour-scoped: the aggregation window stops 10 s behind the flush and names only the hour directories it spans, instead of globbing the whole retention.

Ingest, re-reads and downtime

The runner ingests telemetry into metrics.sqlite (next to the runner database, never snapshotted) by decoding celld’s Parquet in-process, with no external process. Each pass processes hour-aligned chunks: the completed hours it still owes, then the partial current hour, and fills the log ring (app_log) and the error tables (app_error_*) — the only Parquet-derived tables left; requests, errors and latency come from the access log into the hourly counters. Every table it owns is written with replace semantics (app_log, app_error_*), so re-reading a window always yields the same aggregates instead of double counting. A single corrupt or foreign object is skipped with a warning instead of failing the fleet’s chunk forever.

That is what makes failures and downtime safe:

• A failed pass (an S3 or SQLite error) is distinguished from an idle one (the window listed no object). Idle completes the chunk; a real error leaves the watermark where it was, so the chunk is retried — and the retry is exact.
• One broken fleet cannot hold the others. When a pass covering several fleets fails, the runner immediately retries it per fleet: healthy fleets commit and advance, and only the failing fleet keeps its watermark. A fleet that keeps failing is retried after a short backoff, so a persistent problem does not re-fail every tick.
• After the runner is off for hours or days, the next ticks walk the gap one hour-chunk at a time, oldest first, advancing the watermark per committed chunk and bounded by RUNNER_TELEMETRY_RETENTION_DAYS. This includes fleets that have stopped, parked or crashed since: any fleet whose watermark is behind is drained through its last complete hour, so a scale-to-zero park does not strand the tail of its telemetry (it lands within the hour it stopped in). Data older than retention has been pruned by celld and cannot be recovered.
• Compaction cannot change a re-read’s counts: an hour directory that holds a compacted.parquet is read from that file alone, so the intermediate state of a compaction (compacted file written, sources not yet deleted) never counts twice, and a leftover from an interrupted source delete is cleaned up on the next compaction pass.

Replaying a window

Because the ingest replaces, a window can be replayed safely — for example to recover rows dropped by the pre-alpha.2 ingest bug (a whole pass was skipped when any running app had no requests or logs in the hour, and the watermark advanced as if there had been nothing to read).

Replaying a window re-reads already-stored Parquet through the same in-process reader, so it costs S3 reads proportional to the window and the fleet count — a handful of fleets over 30 days takes minutes; a large install on a slow store can take longer. It runs in the background and does not block the control plane.

To replay by hand — a later window, one fleet — run:

docker compose exec noite noite-runner telemetry reingest --since 7d
docker compose exec noite noite-runner telemetry reingest --slug blog --since 2026-01-01T00:00:00Z

--since takes an RFC3339 timestamp or a duration back from now (24h, 7d); the default is the whole retention window, clamped to what the bucket still holds. Without --no-wait the command blocks until the runner reports it has caught up (or --timeout seconds pass, after which it continues in the background).

To read the raw telemetry yourself, point any Parquet reader at the fleet bucket: s3://<bucket>/fleets/<slug>/telemetry/{traces,logs}/<node>/<yyyy>/<mm>/<dd>/<hh>/*.parquet holds celld’s spans and console lines (requests and latency live in the hourly counters instead, fed by the access log). A directory holding compacted.parquet must be read from that file alone.