Operations

Update Noite, back up and restore its volumes, and troubleshoot ready gates and boot failures.

Releases and channels

Installs follow releases, not main. NOITE_IMAGE (or NOITE_VERSION for the installer) picks what an update pulls:

Every release is tagged v<version> and passes the end-to-end lanes (passkey signup, invites, git push, deploys, and the hostile-tenant suite) before its image is published. Its notes are in CHANGELOG.md and on its GitHub release; each has an Operator action required section, even when it is “None”. Read it before you update.

A new release starts as a GitHub pre-release: its version tag exists, but alpha and stable don’t move. To try it on your own server first, run the installer with --pre, which installs the newest release (pre-releases included) pinned to its version, with the installer and compose.yaml from its tag:

curl -fsSL https://noite.now/run.sh | bash -s install --pre

Once the pre-release is marked as a full release on GitHub, the channels point at the same image (no rebuild): alpha always, and stable too for a final version. A plain install then puts a --pre server back on the channel without a restart, since the image is the same. Only the latest release moves the channels; the maintainers re-point them by hand to roll a channel back.

To roll back, set NOITE_IMAGE to the previous version tag. That works across releases that did not change the database schema; a release that did refuses to be downgraded (see Troubleshooting), so take a backup first.

Updating

Re-run the installer, or pull and recreate by hand:

curl -fsSL https://noite.now/run.sh | bash -s install
# or
cd /opt/noite && docker compose pull && docker compose up -d
# multi installs add the cgroup grant to every recreating command:
# docker compose -f compose.yaml -f compose.multi.yaml pull && docker compose -f compose.yaml -f compose.multi.yaml up -d

The image is disposable and state lives in the volumes and the bucket, so an update is a pull plus a recreate. The runner stops every fleet in parallel inside its stop budget (NOITE_STOP_BUDGET_MS, 25 s; Compose’s stop_grace_period is 35 s), then Caddy, then writes a final state snapshot — measured at ~0.3 s for 15 running fleets.

A new image also restarts every tenant fleet, and that is the part you feel. Measured on a 32-core host with 1, 5 and 15 sample apps: the runner spawned every fleet within 50 ms of the new container starting — it starts them before it deploys the control UI, so a release whose UI bundle changed no longer delays them (that deploy measures 0.66 s against a local store, more against a remote one; SIGTERM handling is installed before any child starts and the REST listener binds before that deploy, so an upgrade interrupted at any point still stops every fleet cleanly inside the stop budget and writes the final state snapshot) — and every app was serving again within 5.1 s of the container starting, at every fleet count (the fastest fleet answered in ~2.1 s). Fleets boot in parallel, so the window tracks the slowest fleet rather than the number of apps.

What a visitor sees in that window is a delayed response, not an error: the edge holds the request for up to 20 s (lb_try_duration) and serves it the moment the fleet is healthy. Only a fleet that takes longer than that answers the short “starting” page (503 with Retry-After: 5, which reloads itself). The installer says so and asks once before it recreates a changed image (--yes, NOITE_YES=1 or NOITE_CONFIRM=1 skips the prompt); with many tenants, batch upgrades and run them off-peak anyway.

The ingest is idempotent, so a window can be replayed by hand to recover rows an earlier ingest skipped; it is bounded by RUNNER_TELEMETRY_RETENTION_DAYS and documented under Telemetry. No replay is queued automatically on upgrade.

Release check (opt-out)

The runner watches for new releases so admins hear about them in the control UI: on by default, once a day, it makes one anonymous GET of the public release list (https://api.github.com/repos/ryuzcorp/noite/releases — a fixed URL, a fixed noite-update-check User-Agent, no auth) and shows admins a dismissible banner when a newer promoted release exists on the install’s channel (alpha hears about every promoted release, a final release only about finals; dev and SHA builds never notify). Drafts and unpromoted pre-releases never notify.

What is sent: nothing about the instance. The request carries no version, no install id, no identifiers of any kind — it reads the same public list a browser would. Failures are silent (a debug log) and keep the last result.

Turning it off: clear Check for new Noite releases on the control page (/apps/_control?t=settings) and save, or set NOITE_UPDATE_CHECK=0 in /opt/noite/.env and recreate (false and off work too; the installer accepts it as well). The variable wins over the UI toggle, and the card says so while it applies. make doctor reports the check state and the newest release seen.

Stop-budget tuning lives in Environment variables.

Backups

A consistent backup snapshots the runner database, stops the stack, tars both volumes and starts again. Downtime is the tar time.

cd /opt/noite
STAMP=$(date -u +%Y%m%dT%H%M%SZ) && mkdir -p "backups/$STAMP"
docker compose exec -T noite sh -c \
  'curl -fsS -X POST -H "Authorization: Bearer $RUNNER_TOKEN" http://127.0.0.1:8080/v1/admin/snapshot'
# A multi install recreates with the cgroup grant (every `up`/`down` below
# takes the same two flags); single installs use the bare commands as written.
docker compose stop
for vol in noite-data rustfs-data; do
  docker run --rm --entrypoint tar -v "noite_$vol:/vol:ro" -v "$PWD/backups/$STAMP:/out" \
    ghcr.io/ryuzcorp/noite:alpha -cf "/out/$vol.tar" -C /vol .
done
docker compose up -d
# multi: docker compose -f compose.yaml -f compose.multi.yaml up -d

With your own bucket, turn on versioning at the provider instead; rustfs-data then does not exist and only noite-data needs the tar.

Restore

cd /opt/noite
# A multi install downs and ups with the cgroup grant (same two flags on
# both commands); single installs use the bare commands as written.
docker compose down
# multi: docker compose -f compose.yaml -f compose.multi.yaml down
for vol in noite-data rustfs-data; do
  docker volume rm -f "noite_$vol" && docker volume create "noite_$vol"
  docker run --rm --entrypoint tar -v "noite_$vol:/vol" -v "$PWD/backups/<stamp>:/in:ro" \
    ghcr.io/ryuzcorp/noite:alpha -xf "/in/$vol.tar" -C /vol
done
docker compose up -d
# multi: docker compose -f compose.yaml -f compose.multi.yaml up -d

Compose warns that the volumes were not created by it; that is expected. Verify with docker compose exec noite curl -s http://127.0.0.1:8080/ready, then open an app.

Void email and AI

Void apps (revalidate(), sendEmail(), ai.*) reach platform services through a runner-owned noite-void-proxy script the deploy cohosts in each Void fleet beside the app. ISR revalidation works with no operator config; email and AI need this section, and the runner refuses to boot on a misconfigured one, naming the var.

Email needs a sender domain and a relay:

NOITE_VOID_EMAIL_DOMAIN=mail.example.com
NOITE_VOID_SMTP_HOST=smtp.example.com
NOITE_VOID_SMTP_PORT=587          # unset = 587/465/25 by mode
NOITE_VOID_SMTP_MODE=starttls     # or `implicit`, or `plain` (needs the allowance below)
NOITE_VOID_EMAIL_ALLOWLIST=@example.com,[email protected]

Senders are pinned to <slug>[+tag]@<domain>; every recipient must parse as one addr-spec (a malformed address fails the whole send with 400 INVALID_TO), and recipients outside the allowlist are refused per-recipient (an empty allowlist fails closed: nothing sends). @domain entries match that domain exactly — subdomains need their own entry. A send carries an idempotency key (Void sends one per call): the proxy replays the recorded response for that key for 24 hours and submits nothing — also for a partial accept, so a retry never re-sends to a recipient the relay already took — while a send where no delivery was observed accepted (failed, outcome unknown, or every recipient refused) is not recorded at all, so retrying with the same key really retries. An outcome_unknown delivery means the message data left and no reply arrived: the relay may already have queued it, so check the relay’s log before retrying, or the retry can deliver a duplicate. The SMTP login is per app, in each app’s Settings (VOID_PROXY_SMTP_USERNAME / VOID_PROXY_SMTP_PASSWORD) — never here: proxy vars are plaintext in the fleet bucket, so no shared operator password may fan out to every fleet. plain SMTP additionally needs NOITE_VOID_SMTP_ALLOW_PLAINTEXT=1, and the runner refuses plain without it.

AI needs the flag plus, for /ai/run, a model map to an OpenAI-compatible endpoint:

NOITE_VOID_AI_ENABLED=1
NOITE_VOID_AI_MODELS={"@cf/meta/llama-3-8b": {"endpoint": "https://ai.example.com/v1", "model": "llama"}}

Unmapped models answer a loud 501 (never a silent substitute), and /ai/run returns the endpoint’s native response shape untranslated. The endpoint key, when the endpoint needs one, is per app (VOID_PROXY_AI_API_KEY in Settings); /ai/provider always uses the tenant’s own provider key, never operator config. Endpoints are https except loopback sidecars (a local model server on 127.0.0.1/localhost), which additionally need NOITE_VOID_AI_ALLOW_PLAINTEXT=1 — the runner refuses any other http endpoint at boot, since the per-app key would travel unencrypted. A new app deploy picks up operator-config changes (like any deploy-time value, they apply from the next deploy).

In multi tenancy (NOITE_TENANCY=multi) the relay and every AI endpoint must be publicly reachable: the proxy runs inside the tenant fleet, and the fleet egress policy rejects fleet→loopback/private destinations. A loopback, private, link-local, CGNAT or ULA address — or an obviously local name (localhost, *.local, *.internal, a single-label name) — fails the boot naming the var, instead of answering 502/UPSTREAM_ERROR at send time. In particular NOITE_VOID_AI_ALLOW_PLAINTEXT=1 (loopback http sidecars) cannot work in multi; single-tenancy installs keep today’s behavior.

Void sandboxes

Void apps that declare a sandbox (getSandbox() in the Void runtime) run it through a runner-owned noite-void-sandbox script the deploy cohosts in each sandbox fleet beside the app — before the proxy, with the app deploying last. The controller is fleet-local: one fleet’s sandboxes are unreachable from any other fleet or from the public internet, and previews get their own controller. Each fleet admits at most NOITE_VOID_SANDBOX_MAX_CONCURRENT live sandboxes (default 4; 0 admits none); over the cap, getSandbox() throws Void’s SandboxLimitError, exactly as on Void’s own platform.

What works is Void’s sandbox surface: invoke (exec runs commands, files read and write, fetch reaches the sandbox’s ports), PTY terminal sessions (pty: true or { cols, rows }, with isPty and resize), the 30 s lease heartbeat with idle reaping, and direct port fetches refreshing the idle deadline. Abort follows Workers semantics: aborting an exec’s signal sends SIGKILL to the process, delivered into the container by celld.

Image rules follow the platform’s container rules, with two sandbox additions. A Dockerfile in the project must run nothing (RUN is refused; FROM/COPY/WORKDIR and friends are vetted and every base pinned by digest — Void’s default sandbox image already qualifies), and the image must carry the sandbox shim (/usr/local/bin/sandbox-shim): custom images FROM — or COPY --from — a shim-carrying image, and a missing shim fails container start naming that contract. A registry image named directly must be digest-pinned (@sha256: plus 64 lowercase hex); an unpinned tag fails the deploy before anything builds. instanceType is ignored in v1: every sandbox runs the install’s default. The controller speaks Void’s sandbox protocol 2 only: a sandbox app whose void major differs from the bundle’s fails the deploy with Void’s own upgrade message, before anything deploys.

Troubleshooting

• Boot stops with refusing to start: the runner database is schema version N (or the same for the control database): the data was written by a newer Noite than the image you started. Run the newer image again, or restore a backup taken before the upgrade. Downgrading across a schema change is not supported, and the refusal is what stops an old image from damaging newer data.

• Locked out after losing the passkey: see Lost passkey.

• /ready answers 503: its body names the failing gate (reconcile, bucket, isolation, control, caddy). A fresh install waits for the control bundle’s first deploy into s3://<bucket>/control (seconds; revision-gated, so a restart with the same bundle and vars skips it).

• isolation: … in multi: the container lacks a capability, nft, or the per-fleet cgroups (per-fleet cgroups unavailable … names the cgroup cause). Grant NET_ADMIN, SETUID, SETGID and CHOWN in compose, then layer the multi cgroup grant on top (docker compose -f docker/compose.yaml -f docker/compose.multi.yaml up -d), or run NOITE_TENANCY=single.

• Boot stops with runner state restore failed: the volume is empty and the bucket did not answer. The runner refuses to start with an empty database (it would overwrite the good snapshot); fix the store and it retries on restart.

• Boot stops with config invalid naming an env var: a numeric setting is malformed (a typo like RUNNER_POLL_MS=5s). The runner refuses to boot instead of silently using the default; fix the named variable to its expected form and restart. Unset (or empty) still means the default.

• A deploy fails with celld deploy does not support these config keys: a Wrangler key celld does not accept. Noite strips the keys it consumes (release, build) before deploying; anything else must go.

• A deploy fails with nothing to deploy: the tree has no Wrangler config, no index.html in dist/, build/ or out/, and no package.json main. See No config at all.

• A deploy fails with starts a Node HTTP server: package.json main calls .listen(). Noite runs Workers; see Frameworks and Node servers.

• Many 429 responses: the edge rate limits. Behind a proxy, check NOITE_TRUSTED_PROXIES; otherwise raise NOITE_EDGE_RPM or the app’s own limit (Edge protection).

• Metrics, logs or errors are missing for a period: the ingest now holds its watermark on a failed pass and catches up hour by hour, so most gaps fill themselves in within a flush interval or two. If a gap persists, replay the window by hand (idempotent, bounded by retention):

  docker compose exec noite noite-runner telemetry reingest --since 30d
  docker compose exec noite noite-runner telemetry reingest --slug blog --since 2026-01-01T00:00:00Z
  docker compose exec noite noite-runner telemetry reingest --slug _control --since 7d

  --slug _control replays the control plane’s own telemetry (the reserved pseudo app, visible to instance admins only). Its Parquet lives under s3://<bucket>/control/telemetry/ rather than fleets/.

  Add --no-wait to return immediately; the runner keeps replaying and logs telemetry replay complete. To check how far the ingest has covered a fleet, read the persisted aggregates through the runner’s own API (the container has curl and the token):

  docker compose exec noite sh -c 'curl -fsS -H "Authorization: Bearer $RUNNER_TOKEN" \
    http://127.0.0.1:8080/v1/apps'                       # id + slug per app
  docker compose exec noite sh -c 'curl -fsS -H "Authorization: Bearer $RUNNER_TOKEN" \
    "http://127.0.0.1:8080/v1/apps/<id>/metrics?hours=720"'   # hourly counters, 30 days

  A missing hour shows as a gap in that list (or in the app’s Metrics tab); compare it with the hour directories under s3://<bucket>/fleets/<slug>/telemetry/ to see whether the source data still exists. Nothing older than RUNNER_TELEMETRY_RETENTION_DAYS is recoverable: celld has already pruned it from the bucket.