State, config, and secrets
The state boundary
The immutable unit holds no durable state. State lives outside it, in a database the app connects to. Keeping a database inside a disposable unit would force a dump and restore on every update — downtime scaling with data size, and a failed restore is data loss on the ordinary path. Every immutable system that tried it abandoned it; the tell is simple — if state must be dumped, it is in the wrong place. Once state is external, revisions swap freely and there is nothing to dump. kixctl's own control plane works this way, its Postgres external to every container, which is why the appliance picks up where it left off after a self-update.
Config declared once, carried forward
The thin layer that tells an app where its state lives — plus its secrets and environment — is declared once per app, stored encrypted at rest, and injected into every revision at launch. Because it is applied when the instance is created, changing a value takes effect on the next revision rather than a running one: a config change is itself a deploy trigger, which is exactly right for an immutable model. Managing these values is covered in Inject per-app configuration and secrets.
Delivery as credential files
Injected values are delivered as files pushed into the container's credential store (/etc/credstore/<KEY>, root-only), between the instance being created and started — so they never appear in the instance's configuration. The app's service imports them through systemd's credential mechanism, and a small env-bridge exposes each as an environment variable, so an ordinary app reads them from its environment. A strict mode, where the app reads the credential directory directly and the secret never enters the environment, is available for apps that want it. Values are never baked into the image, and the job logs their key names only, never their values.
The secret chain
Secrets sit in three layers, each matched to whether the secret exists at build time or only at run time:
- kixctl's own infrastructure secrets — webhook secrets, Incus certificates, the database password, the application key — are managed with sops-nix and decrypted at system activation. They exist when the configuration is authored.
- A deployed app's secrets — a user's database URL — are encrypted at rest in kixctl's Postgres under the application key, and that key is itself a sops-nix secret, so app secrets are encrypted under a sops-guarded key.
- Delivery into the container is the credential mechanism above.
sops-nix structurally cannot reach the delivery layer: it decrypts committed ciphertext into a specific machine's configuration at build time, but a user's runtime-entered database URL does not exist at image-build time and must stay generic across every deployment. At-rest encryption of the delivered credential is the enterprise hardening, not the first cut.
The three-tier database direction
How a deployed app gets a database, cheapest to richest, built in this order:
- Bring your own — the app connects to a database you already run, through an injected connection string. kixctl stores and runs nothing extra. This is the shipped path.
- kixctl provisions a database container on your own cluster — a persistent container, the one place a storage volume is correct and the unit is deliberately not immutable — with its connection string injected into the app. It reuses the same injection path.
- Managed database — a later, opt-in, paid convenience: the same path pointed at a kixctl-operated target. It is never the default and never the only path; forcing data off your own cluster would contradict the self-hosted thesis.
The same injection machinery sits under all three.