Operations

Backups and restoration

Back up PostgreSQL, S3 output objects and lifecycle configuration, gateway TLS and host CA material, catalog public keys, identity configuration and the designated audit export volume. Preserve the export cursor with its matching files and the shared audit.cursor_key_file key across gateway instances. Protect private keys and credentials through your secret-management backup process. Keep audit checkpoints in a separate trust domain.

Test restoration in an isolated environment. Check schema compatibility, database privileges, identity login, host connection, catalog distribution and audit verification. Restore object references and objects from compatible backup points. Missing or expired objects must be reported as unavailable; a valid audit chain does not restore their contents.

Backups of hosts include /var/lib/allowops/ (private identity and catalog state) and /var/lib/allowops-exec/ (root-owned highest versions). Never clone one host's identity to another. Preserving only catalog bytes while losing highest-version state weakens rollback detection. Treat such restoration as a reviewed trust recovery, not a routine file copy.

Upgrade order

Upgrade the gateway first, then hosts using verified artifacts from the intended release. A gateway supports its own and previous major host versions (only major zero for a major-zero gateway), but rejects a host with a newer full SemVer. Do not introduce a new host version before the gateway supports it.

Hosts can be upgraded online with the agent_upgrade example action (risk: dangerous, run_as: root). It installs the selected allowops-agent version from the host's configured apt or yum repository, then restarts the agent from a transient systemd timer about 5 seconds later so the task result reaches the gateway first. List only versions not newer than the gateway in the version enum and publish a new catalog after each gateway upgrade. The action does nothing on a host already at the requested version and fails without restarting when the installed version does not match afterwards; it does not downgrade. Package signatures are checked by apt/yum according to the repository configuration. Roll out in batches with host selectors, and grant the action through a dedicated policy rule. A host upgraded beyond the gateway is refused and must be downgraded on the host.

Run the release migration with the maintenance database role. Helm uses a pre-install/pre-upgrade migration hook; deployments that manage binaries run allowops-gateway migrate separately. Preserve the runtime role's restricted audit privileges. Review migration compatibility before rolling back binaries; do not edit migration history or assume a newer schema works with an older gateway.

SIGTERM stops new work, drains in-flight work up to the configured grace period and asks hosts to reconnect. Allow sufficient container termination time. Disconnects terminate active task process groups and do not replay tasks. Cancellation and disconnect do not reverse side effects already made.

After an upgrade, verify the running image digest or behavior unique to the new code, then check authenticated login, inventory, an authorized test action, its output and audit. A reported version or commit alone does not establish that the new executable is serving requests. Host package installation enables but does not start the service; operators control the restart.

Key and token rotation

Catalog key rotation is described in publishing: distribute the new public key, publish a dual-signed package, confirm host loading, then remove the old key. Revoke compromised keys on hosts, not only at the gateway.

For service tokens, issue a replacement, update one client, validate access and revoke the old token. Separate accounts for different clients simplify revocation. Keep passwords and tokens out of process arguments and shell tracing.

The gateway TLS CA pin is distinct from the host identity CA. Renew gateway TLS certificates with the expected names and full chain. Changing the pinned TLS CA requires a coordinated operator update of every host's configured pin. Secret changes copied by Helm init containers require workload restarts. Do not replace the host identity CA as a routine rotation: in-place host CA rotation is outside the MVP. Plan a controlled re-registration if that trust must change.

Storage and retention

Use an existing S3 bucket, configure the output region/endpoint and deliver credentials through the AWS SDK credential chain. Configure encryption support: output uploads request AES256 server-side encryption. SeaweedFS needs its SSE key configured independently. Retention belongs to the bucket lifecycle policy; the gateway does not set it automatically. The intended default is one year.

SeaweedFS lifecycle configuration APIs and compressed output operations have local integration coverage. Actual delayed expiry deletion requires operator verification over the configured retention interval. Check the selected storage version and its encryption/lifecycle behavior before relying on deletion. Full output writes are asynchronous; monitor failed persistence even when jobs succeed. The output_stored flag distinguishes execution from persistence.

Monitoring and recovery

Use HTTPS GET /health/live for liveness and /health/ready for readiness. PostgreSQL connection/schema failure returns HTTP 503. Output storage failure keeps readiness at HTTP 200 with degraded: true; monitor the body and output persistence rather than interpreting HTTP 200 as proof that every dependency is healthy. The allowops-gateway health command checks status and TLS but does not fail on the degradation flag. Health endpoints do not verify user login or action authorization.

Synchronize clocks. Catalog activation allows five minutes of skew; excessive skew can break certificate or catalog validation. Monitor offline hosts, failed catalog loading, unsupported actions, output persistence, audit export lag and dependency health. Catalog expiry and token expiry require renewal before use ends. Inventory load reports reflect the last complete report, including for offline hosts; they do not prove the host's present execution environment.

Use short join-token lifetimes and bounded uses. Set authorization labels only after checking the registered host identity. Deletion invalidates certificates; a same-name replacement has a new immutable ID. No host credential is an API identity.

If publication or management returns an audit-result error after a possible commit, read the resource before retrying. If an execution response is lost, inspect jobs before submitting again. There is no general exactly-once side-effect guarantee. A gateway-instance loss can require task reconciliation without replay.

New release notice

The gateway can tell console users when a newer release exists. It is off by default, so the gateway makes no outbound request unless you enable it:

update_check:
  enabled: true
  url: https://liubing0806.github.io/allowops/version.json
  interval: 24h

url must be HTTPS without credentials and interval must be at least 1h. Point url at an internal copy of the manifest when the gateway has no internet access. The manifest is a JSON object with version (strict SemVer) and an optional published_at (RFC 3339). The documentation build generates the published manifest from the first release heading of CHANGELOG.md.

Each check sends one GET request with User-Agent: allowops-gateway/<version> and nothing else. It uses the proxy from the environment, follows at most three HTTPS redirects and reads at most 64 KiB. Gateway instances share one result in PostgreSQL, so a cluster fetches about once per interval. The gateway only informs; it never downloads or installs anything. Development builds report dev and never show a notice.

When a newer release exists, the sidebar shows an Update mark next to the gateway version that opens the GitHub release notes, and the overview lists the release under Needs attention. The system page shows whether the check is enabled, when it last ran, the last error, the number of hosts whose agent is older than the gateway, and a warning when gateway instances run different versions.