Atomic DigitalOcean deployment

Veetbot deploys to one Ubuntu Droplet at api.veetbot.com. PostgreSQL, the API, the interactive and asynchronous run workers, the maintenance worker, the credential-free execution service, the browser-profile service, Docker with gVisor, and Nginx share that host, with the optional schedule, notification, surface, and calling roles when they are enabled. CircleCI packages the tested main commit and promotes an immutable release below /opt/veetbot/releases. The same pipeline publishes the complete MkDocs site at docs.veetbot.com from a checksummed artifact tied to that release.

The production deployment scripts require the Ubuntu GNU userland: Bash, GNU coreutils (including mv -T and sha256sum), GNU tar and findutils, and util-linux flock. Local macOS deployment-script tests require the matching Homebrew tools:

brew install coreutils gnu-tar findutils util-linux

The harness maps gmv, gsha256sum, gtar, gfind, and the Homebrew util-linux flock into the production command names during both fixture creation and deployment. Privileged Nginx and systemd operations remain stubbed.

The flow adapts the useful deployment invariants from avitus/mankunku to Veetbot's Python and systemd runtime. ADR-0048 records the differences and security boundary.

Public website

The visitor-facing homepage and Google OAuth policy pages are a separate static site defined under website/:

• https://www.veetbot.com/
• https://www.veetbot.com/privacy
• https://www.veetbot.com/tos

The source is a Next.js static export with no account system, analytics, database, object storage, or Veetbot application credential. Run make website-install test-website locally. CircleCI repeats that build in a credential-free Node job and passes only the generated website/out tree to release packaging.

The application, documentation, and website artifacts receive the same release identity. The Nginx deployment validates and atomically promotes the website at /opt/veetbot/shared/website/current; configuration validation or reload failure restores both static-site pointers and the prior configuration. ADR-0084 records the boundary and rationale.

Semantic-index database prerequisite

ADR-0179 adds the rebuildable memory_vectors table and pgvector 0.8.7. The release builds veetbot-postgres:16-alpine-vector0.8.7 from deploy/postgres.Dockerfile before restarting PostgreSQL or migrating. It retains the existing PostgreSQL 16 Alpine locale environment and volume path; it does not switch an existing volume to the hosted CI image. Follow the existing backup and migration rehearsal before production delivery. Semantic retrieval remains disabled: this schema and image change does not activate a provider, backfill private memory or enable recall.

Release flow

static + contract + integration + sandbox
                    |
                    v
          package tested commit
                    |
                    v
        SSH stage + checksum verify
                    |
                    v
 lock -> uv sync -> image -> PostgreSQL -> migrate -> preflight
                    |
                    v
      current symlink + systemd restart
                    |
                    v
 release.sh: local release header + API contract probe
                    |
                    v
 CircleCI: public release header

Each release is named YYYYMMDD-HHMMSS-<7-character-commit>. The server:

1. takes /opt/veetbot/shared/deploy.lock;
2. refuses a timestamped release older than the currently active release;
3. installs a release-local .venv from uv.lock using a shared download cache, and compiles the bytecode of its dependencies and of src, because the units' read-only filesystem keeps Python from caching it at run time;
4. builds agent-core-sandbox:<release-id>;
5. checks active provider pins against the staged registry on an existing installation, then ensures the local PostgreSQL service is running;
6. reconciles owner permissions into the staged .owner-scopes.env, applies alembic upgrade head, and runs the production preflight;
7. stops both run workers, rechecks active provider pins, then switches /opt/veetbot/current, tags the sandbox image as production, and restarts the credential-free execution service and all application units;
8. requires the local readiness probe to return X-Veetbot-Release: <release-id> within 180 one-second attempts (VEETBOT_HEALTH_TIMEOUT_SECS; every unit restarts at once, and the API has needed more than a minute to bind), makes an authenticated request to the authoritative session index and model settings, and requires every process to run from the promoted directory;
9. retains the five newest valid releases; and
10. retains the two newest timestamped agent-core-sandbox and veetbot-browser-profile-service image tags and prunes build cache that no build has used for 48 hours.

The provider-pin checks enforce ADR-0119's requirement to drain runs before a provider-registry change. They include queued, running, approval-waiting and user-waiting runs across every tenant. A mismatch, malformed pin or failed probe refuses promotion; the report names only run IDs. Finish or explicitly cancel the listed runs on the current release, then retry deployment. Terminal runs and queued runs that have not selected a provider do not block a release. The second check runs with both pin-writing workers stopped, closing the race with the first check. A failure before promotion restarts the previous workers. Install the updated deploy/sudoers/veetbot-deploy contract before the first release using this guard; its two additional exact commands stop and restart veetbot-worker veetbot-async-worker.

A run already stranded by an earlier registry change fails as model_permanent_error with ProviderPinUnavailableError and asks the owner to send a new message. It never substitutes the new registry or executes the pending tool. The provider pin and approval history remain intact; a terminal failed run is not automatically retried. Terminal checkpoints discard provider continuation state even when failure precedes entry to the model loop.

Before step 3 compiled bytecode, every service start and every stdio MCP server spawn compiled its imports from source. On the idle production host, importing the CLI took 17.5 s of CPU from source and 8.4 s from bytecode, and a Gmail server's module 4.1 s and 2.2 s. The API gives in-flight requests five seconds when it is stopped, then cancels them; clients resume a cut run event stream from Last-Event-ID.

Release pruning normally runs as the deployment identity, which also owns the bytecode each release compiles. Older releases may contain service-owned Python bytecode from deployments that predate the strict read-only systemd filesystem policy. If direct removal of such a release fails, the script retries with the just-built sandbox image as uid 0 in a one-shot, network-disabled container. The container has a read-only root filesystem, only the releases directory is mounted writable, and its fixed entrypoint removes only the validated release name selected by the retention loop. This fallback uses the deployment identity's existing Docker trust boundary; application identities still receive no Docker access.

Image retention runs only after every verification above has passed. For each of the two repositories it sorts the YYYYMMDD-HHMMSS-<revision> tags newest first and removes every tag after the second, which keeps the active release's image and its predecessor, the manual rollback target. It never removes the tag just released, a tag that docker ps reports a container running, or a tag outside that pattern such as production, local, or dev. Each removal is independent: a failure is logged as a warning and the release still succeeds, so a rerun removes only what remains. The same step runs docker builder prune --all --force --filter until=48h, which discards build cache no build has touched in two days while keeping the dependency and Chromium layers that consecutive builds share warm. The step prints docker system df before and after so the CircleCI deployment log records the reclaimed space. VEETBOT_KEEP_IMAGES overrides the tag count. The deploy/browser-profile-service.Dockerfile installs dependencies and Chromium above the application source so that only the final layers change between releases; without that order each release snapshot repeats those layers and the retained tags share nothing.

A successful server release returns control to CircleCI, which polls the public TLS readiness endpoint until it reports the same release ID. Exhausting that bounded public-probe budget fails the CircleCI job after promotion; it is not a failure inside deploy/app/release.sh.

The optional Telegram and WhatsApp channels run in the separate surface role. Use the Telegram surface runbook for the bot, token file, both environments, activation, pairing, live smoke, and rollback, and the WhatsApp integration runbook for the Meta owner ceremony that adds the second channel; do not place channel credentials in the application environment, which refuses to start with one.

A pre-promotion failure removes only its staged directory. A post-promotion failure remains visible for diagnosis and manual rollback. Database migrations are never downgraded automatically.

One-time host preparation

Inventory the shared Droplet before changing it:

sudo ss -ltnp
sudo systemctl --type=service --state=running
sudo docker ps --format 'table {{.Names}}\t{{.Image}}\t{{.Ports}}'
docker --version
docker compose version
nginx -v
uv --version
df -h
free -h

Confirm port 8000 is free, choose a free loopback PostgreSQL port, and record all existing Docker containers. Reuse the existing Docker and Nginx installations; do not replace services used by other applications on the Droplet.

Create separate deployment, application, and sandbox-execution identities plus the persistent paths:

Current production host (verified 2026-09-03)

The Droplet was not provisioned with the separate veetbot-deploy identity below. CircleCI logs in as the veetbot service account: its key is in /home/veetbot/.ssh/authorized_keys, veetbot owns /opt/veetbot and /opt/veetbot/shared/deploy.lock, and the sudo contract is installed for veetbot. Because that account is also the User= of the API, worker, and maintenance units, it carries the deploy identity's Docker-group membership into every application process, which is the boundary ADR-0067 decision 1 draws to keep application services away from the Docker socket. Until the host is reconciled, read every veetbot-deploy below as veetbot when operating the current Droplet, and treat "create the veetbot-deploy account, move the CircleCI key and sudo contract to it, and remove veetbot from the docker group" as the open operations item that restores the documented boundary. The account name is not visible to CI: DEPLOY_USER in the veetbot-production context selects it.

sudo useradd --system --create-home --shell /bin/bash veetbot-deploy
sudo useradd --system --no-create-home --shell /usr/sbin/nologin veetbot
sudo useradd --system --no-create-home --shell /usr/sbin/nologin veetbot-exec
sudo usermod -g veetbot veetbot-exec
sudo usermod -aG veetbot veetbot-deploy
sudo usermod -aG docker veetbot-deploy
sudo usermod -aG docker veetbot-exec
sudo mkdir -p \
  /opt/veetbot/releases \
  /opt/veetbot/shared/docs/releases \
  /opt/veetbot/shared/website/releases \
  /opt/veetbot/shared/uv-cache \
  /etc/veetbot \
  /var/lib/veetbot/artifacts
sudo chown -R veetbot-deploy:veetbot /opt/veetbot
sudo chown -R veetbot:veetbot /var/lib/veetbot
sudo chmod 0700 /var/lib/veetbot/artifacts

veetbot-execution.service also sets DynamicUser=yes. A fully prepared host uses the static veetbot-exec account above; on an older host that does not yet have it, systemd allocates the same dedicated service identity so the atomic release can complete without temporarily running the execution service as an application user.

The veetbot-deploy account owns /opt/veetbot, including the documentation releases. Nginx and the application group receive read-only traversal through the ordinary directory modes. The API and worker identities are not members of the Docker group and their units expose no Docker socket; application services have no Docker-socket access. Only the credential-free veetbot-exec execution service receives Docker access at application runtime and owns sandbox lifecycle. The separate deploy identity retains delivery-time Docker access because the audited release contract builds and tags both images, reconciles the browser-profile service, and prunes stale release images and build cache. That trusted operator boundary is the explicit tradeoff in ADR-0067 decision 1; it is never an application systemd identity. Consequently neither an application process nor an application-launched container can modify the documentation release. On a single host, workers reach the service at /run/veetbot/execution.sock. On a multi-host deployment, move the service to the dedicated sandbox host described by ADR-0008 and replace /run/veetbot/execution.sock with an authenticated network transport while preserving the ExecutionEnvironment port.

The deploy key must log in as veetbot-deploy, so that account needs an executable shell. Restrict that key in authorized_keys to the CircleCI source and disable port, agent, and X11 forwarding. Install the committed sudo contract deploy/sudoers/veetbot-deploy as /etc/sudoers.d/veetbot-deploy with mode 0440, replacing its placeholder user deploy with this account, and validate with visudo -c -f /etc/sudoers.d/veetbot-deploy. The contract is the exact union of the sudo commands deploy/app/release.sh and deploy/nginx/deploy.sh invoke, and a repository test reconciles the scripts against it, so a script gaining a sudo command without a rule fails CI rather than the production release. That authority is effectively host administration; do not reuse the key for application clients or interactive users.

Install stable runsc if it is not already registered. Preserve the existing Docker configuration before the required daemon restart and verify all previous containers afterward:

if sudo test -f /etc/docker/daemon.json; then
  sudo cp -a /etc/docker/daemon.json /etc/docker/daemon.json.before-runsc
fi
sudo runsc install
sudo systemctl restart docker
sudo docker info --format '{{json .Runtimes}}'
sudo docker ps
docker run --rm --runtime=runsc hello-world

Copy deploy/veetbot.env.example to /etc/veetbot/veetbot.env, replace every REQUIRED_ value, select the free PostgreSQL port in both locations, and add VEETBOT_OPENAI_KEY for the shipped production astra policy. A reviewed model-policy overlay may instead retarget astra to another configured provider.

New production sessions default to astra, selecting OpenAI gpt-6-astra; flagship and fable select Anthropic claude-fable-5-1. Existing sessions retain their recorded agent version and model policy. balanced explicitly selects gpt-5.6-sol (gpt-5.6 alias), and claude-opus-5 (opus alias) remains registered for operator overlays. The new profiles use the standard prices verified on 2026-09-11 in the OpenAI model documentation and Anthropic model documentation: USD 10 input and USD 50 output per million tokens; cached input is USD 1 for Astra and USD 0.25 for Fable. Cache writes are USD 12.50 with the Anthropic adapter's default five-minute TTL. Astra retains the 272,000-token window required by ADR-0039's single-tier pricing rule; Fable uses 1,000,000 tokens, and both allow up to 128,000 output tokens.

Changing these profiles changes the registry version. Drain active and suspended runs before deploying: existing provider pins fail closed when their registry version is unavailable. Provider-assisted memory also requires evaluation evidence for the exact selected model and model-policy name; the GPT-5.6 Sol artifacts do not authorize Astra or Fable. Without matching evidence, auto uses deterministic formation and required refuses startup.

Memory formation deliberately remains on GPT-5.6 Sol while chat uses the new defaults (ADR-0093). formation.model_policy: balanced in memory/profiles.yaml independently selects its evidenced tuple, so the bundled formation@9 evidence still activates and formation@10 remains its evidenced fallback. An AGENT_CONFIG_DIR overlay may change that reference, but the replacement still needs its own exact evidence. Selection audits name both chat_model_policy and the memory model_policy. The memory evaluators' --model-policy astra continues to evaluate Astra explicitly. Memory formation remains active Milestone 21 work: audit scorer semantics and run a controlled Sol/Astra comparison before changing this temporary choice.

To enable the initial Keenable comparison, add all three provider credentials and route half of each capability to Keenable in that same root-owned file:

WEB_SEARCH_PROVIDERS=tavily:50,keenable:50
WEB_FETCH_PROVIDERS=firecrawl:50,keenable:50
TAVILY_API_KEY=<production Tavily key>
FIRECRAWL_API_KEY=<production Firecrawl key>
KEENABLE_API_KEY=<production Keenable key>

Do not put the real values in .env.example, the production template, CircleCI configuration, a commit, or a PR. The worker reads this environment when the release restarts its systemd unit. Plural selector entries must be unique positive integer percentages summing to 100. The legacy singular selectors remain valid for one-provider deployments, and both capabilities stay disabled when neither form enables them.

To enable image and video generation (ADR-0140), provision TENSORSCALE_API_KEY in the API and worker environment and grant media.generate plus artifact.write in the owner's AUTH_SCOPES. Doppler's development key cache does not update the production root-owned environment file. Permit SenseNova U1.5 and LTX-2.5 Fast (ltx2-5-fast) on the key, restart the API and workers, and open a new Chat: old sessions retain their pinned tool roster. No additional feature flag is needed for generation. To upload reference pictures, enable AGENT_ATTACHMENT_UPLOADS_ENABLED=1 and grant artifact.read for input reads. Version 1.1.0 accepts PNG/JPEG/WebP references from the same chat: up to eight for image edits (20 MiB each, 64 MiB total), or first/last frames for video (10 MiB each). Each approved call creates one PNG or MP4 attachment; video duration is bounded to 5 or 10 seconds. External provider charges are separate from language-model usage accounting. There are no automatic retries for failures or timeouts, since the provider may already have charged. Keep the credential out of sandbox and scheduler-only environments.

To enable the typed-judgment provider (ADR-0110), add its selector and key to that same root-owned file:

JUDGMENT_PROVIDER=typesafe
TYPESAFE_API_KEY=<production TypeSafe key>

The selector is the only switch: the key alone enables nothing, and the selector alone enables no consumer. Each consumer has its own default-off knob in its checked-in profile, such as proposals.judgment_matching_enabled in folders/profiles.yaml. With the selector set and the key missing, the services still start, log one judgment_credential_missing warning that names the two variables, and every judgment call fails without dialing so each consumer takes its deterministic fallback. Keep both variables out of /etc/veetbot/veetbot-schedule.env and /etc/veetbot/veetbot-notify.env: those workers refuse to start when their environment holds a provider key. The adapter dials one fixed endpoint, api.typesafe.ai on port 443, and needs no egress-policy entry.

The policy advisory layer (ADR-0111) has two switches, and they are not equivalent. AGENT_POLICY_ADVISORY_OBSERVE_ENABLED=1 in that same file consults the advisor on allowed web searches, page fetches and browser navigations and records what it would have escalated; no decision changes and policy_version does not move. Enforcing is the policy-profile value advisory.enabled: true, set in an operator overlay at policy/default.yaml under AGENT_CONFIG_DIR. That changes policy_version: a pending approval whose re-evaluation is not an allow is voided, and the bundled memory-formation, People and email-People release evidence no longer matches the running composition, so provider-assisted formation falls back to deterministic until the evidence is regenerated on the new version. Observe first, read the agent.policy.advisory.* metrics, and enforce only when no approval is pending.

The client's model settings (ADR-0119) need settings.read,settings.write. Deployment provisions both automatically and requires an authenticated HTTP 200 from GET /v1/settings/models before declaring success (ADR-0149). The routes are always mounted. A saved chat model applies to new app chats; the chat effort applies to later agent runs except typed email tasks, which retain provider-default effort. The memory choice applies to later formation.

The template's AUTH_SCOPES is the reviewed owner permission set, including optional Email, Persona, calling, media and delegation features. Every release unions that set with the host's explicit grants, sorts and deduplicates it, and writes only AUTH_SCOPES to the staged .owner-scopes.env. The API and owner workers load it after the host environment; restricted service roles never load it. Deployment does not rewrite /etc/veetbot/veetbot.env or copy its credentials. Promotion activates the file with the release; rollback uses the target's file, or the host grants for older targets without one.

New platform scopes must be classified in test_deployment_scopes.py and added to the template when owner-facing, so adding a feature cannot silently omit its deployment grants. Feature flags, provider credentials, evidence gates, policy and approvals still govern activation and use. Scopes are exact strings, with no wildcard; unknown scopes stop startup. MCP scopes continue to derive from account configuration. Removing a managed scope only from the host file does not revoke it: a deliberately restricted owner deployment needs a reviewed change to the template. When running owner CLI commands directly, source the current .owner-scopes.env after the host environment if the file exists.

The surface role's own environment carries surface.read and surface.write.

The host exports no metrics, so the service log answers whether these consumers are running. Every long-running service writes JSON lines, one object per event, with first-party events from INFO and every other logger from WARNING:

sudo journalctl -u veetbot-api -u veetbot-worker -u veetbot-async-worker \
  --since '-1 hour' -o cat | grep policy_advisory
sudo journalctl -u veetbot-maintenance --since '-1 hour' -o cat | grep folder_proposal_pass

policy_advisory_consulted appears once per consultation with tool, verdict, enforced, signals and seconds; a verdict other than abstain is what enforcing would have sent for approval. policy_advisory_abstained carries the cause: timeout, or a provider reason code such as judgment.auth_failed or judgment.payment_required, the latter meaning the vendor account has no credits. folder_proposal_pass appears once per pass: judgment_requests above zero with judgment_fallback_used false and judgment_error_class null is a working matcher, and judgment_matched counts the conversations it proposed to file. None of these lines carries a query, a URL, a title, or a probability.

Run docker compose ls before the first automated release. If an existing Veetbot PostgreSQL container was created under a Compose project name other than veetbot, set COMPOSE_PROJECT_NAME to that exact existing name. Changing it silently selects a different named volume and therefore an empty database.

sudo chown root:veetbot /etc/veetbot/veetbot.env
sudo chmod 0640 /etc/veetbot/veetbot.env

When scheduled runs are enabled, separately copy deploy/veetbot-schedule.env.example to /etc/veetbot/veetbot-schedule.env, fill its required database and identity values, and protect it with the same ownership and mode. The schedule unit never loads the shared environment because that file contains API, model-provider, web-provider, and browser-profile credentials that the scheduler must not see. If Gmail is enabled, mirror AGENT_EMAIL_ENABLED=1 into this schedule environment. For named accounts also mirror the non-secret GMAIL_ACCOUNTS_FILE path; for legacy single-account configuration leave that path empty. Do not copy any GMAIL_*_CREDENTIAL_FILE variable. The schedule loader reads the manifest's account ids to derive current Gmail scopes but never opens the referenced OAuth credential documents. The tenant-scoped database role must not have PostgreSQL BYPASSRLS authority: schedule child-table isolation depends on the forced row-level-security policy on schedules applying inside each child-policy lookup. Reserve BYPASSRLS roles for explicit administration and never use one for API, worker, or scheduler tenant access.

Provision the veetbot_schedule login independently with a generated password, then grant only the tables its materialization transaction and schema-head check use. The release runs scripts/check_schedule_database_permissions.py through the protected schedule environment before promotion and refuses a role that is administrative, inherits authority, can replicate or bypass row-level security, can switch to any other role, is missing any privilege below, or has any other effective table or column-level privilege on a table, view, materialized view or foreign table in the public schema. The check is an exact allowlist, including grants inherited from another role or PUBLIC, rather than a presence-only checklist. Its privilege vocabulary intentionally matches the deployed PostgreSQL 16 release; PostgreSQL 17's MAINTAIN privilege is not accepted while production remains pinned to version 16. The checker reads server_version_num from the connected server and fails before the privilege comparison unless that server is PostgreSQL 16.

ALTER ROLE veetbot_schedule
  NOSUPERUSER NOCREATEDB NOCREATEROLE NOINHERIT NOREPLICATION NOBYPASSRLS;
-- Do not grant veetbot_schedule membership in any other role: NOINHERIT does
-- not prevent it from acquiring that role's authority with SET ROLE.
GRANT CONNECT ON DATABASE agent TO veetbot_schedule;
GRANT USAGE ON SCHEMA public TO veetbot_schedule;
REVOKE ALL PRIVILEGES ON ALL TABLES IN SCHEMA public FROM veetbot_schedule;
GRANT SELECT ON
  agents,
  alembic_version,
  checkpoints,
  derived_event_keys,
  events,
  notification_outbox,
  process_events,
  projection_watermarks,
  runs,
  schedule_occurrences,
  schedule_revisions,
  schedules,
  session_history_items,
  sessions
TO veetbot_schedule;
GRANT INSERT ON
  checkpoints,
  derived_event_keys,
  events,
  notification_outbox,
  process_events,
  projection_watermarks,
  runs,
  schedule_occurrences,
  session_history_items,
  sessions
TO veetbot_schedule;
GRANT UPDATE ON
  projection_watermarks,
  runs,
  schedules,
  sessions
TO veetbot_schedule;
GRANT DELETE ON
  checkpoints,
  projection_watermarks,
  session_history_items
TO veetbot_schedule;

Run the revocation before reapplying the allowlist whenever this role is repaired. If the validator still reports a surplus effective privilege, remove the grant from the role membership, column grant, ownership, PUBLIC, or default-privilege source that supplies it; revoking a direct table grant cannot mask authority supplied by a different source.

The projection grants are part of checkpoint seeding, not optional reporting: without them a due schedule can retry until its misfire window expires without ever committing the session and run.

sudo chown root:veetbot /etc/veetbot/veetbot-schedule.env
sudo chmod 0640 /etc/veetbot/veetbot-schedule.env

Do not add VEETBOT_RELEASE_ID to that shared file. The release script writes it and the fixed host-native execution-service socket to /opt/veetbot/current/.release.env, which every credential-bearing application unit loads after the shared environment. This release-local wiring lets an existing host adopt the execution service without rewriting the protected shared environment before the first compatible release.

When the Telegram surface is enabled, the surface role connects as its own veetbot_surface login, never as the application login. The Telegram surface runbook gives the one command that creates the login with a generated password, applies the allowlist below, and writes it into /etc/veetbot/veetbot-surface.env; rerunning it rotates the password. Whenever AGENT_SURFACE_WORKER_ENABLED=1, the release runs scripts/check_surface_database_permissions.py through that environment before promotion and applies the same exact-allowlist rules as the schedule login above: no administrative attribute, no inherited or settable role, and no effective table or column privilege in public beyond this list.

ALTER ROLE veetbot_surface
  NOSUPERUSER NOCREATEDB NOCREATEROLE NOINHERIT NOREPLICATION NOBYPASSRLS;
-- Do not grant veetbot_surface membership in any other role.
GRANT CONNECT ON DATABASE agent TO veetbot_surface;
GRANT USAGE ON SCHEMA public TO veetbot_surface;
REVOKE ALL PRIVILEGES ON ALL TABLES IN SCHEMA public FROM veetbot_surface;
GRANT SELECT ON
  agents,
  alembic_version,
  approvals,
  checkpoints,
  delegations,
  derived_event_keys,
  devices,
  events,
  export_consent,
  notification_deliveries,
  notification_outbox,
  notification_run_receipts,
  process_events,
  projection_watermarks,
  runs,
  session_history_items,
  sessions,
  surface_inbound_receipts,
  surface_pairing_codes,
  surface_pairings,
  surface_replies,
  surface_sender_lockouts,
  surface_sessions,
  tool_invocations
TO veetbot_surface;
GRANT INSERT ON
  checkpoints,
  derived_event_keys,
  devices,
  events,
  notification_deliveries,
  process_events,
  projection_watermarks,
  runs,
  session_history_items,
  sessions,
  surface_inbound_receipts,
  surface_pairings,
  surface_sender_lockouts,
  surface_sessions
TO veetbot_surface;
GRANT UPDATE ON
  approvals,
  delegations,
  devices,
  events,
  notification_outbox,
  projection_watermarks,
  runs,
  sessions,
  surface_inbound_receipts,
  surface_pairing_codes,
  surface_pairings,
  surface_replies,
  surface_sender_lockouts,
  surface_sessions,
  tool_invocations
TO veetbot_surface;
GRANT DELETE ON
  surface_sender_lockouts
TO veetbot_surface;

The list was derived by running every surface path under a login holding only these grants and then removing each grant in turn; every one is needed. The surface integration test runs the same paths under it. Several UPDATE grants are row locks rather than rewrites: SELECT ... FOR UPDATE on events and runs serializes a message's history and run writes with People erasure, and on approvals, surface_pairing_codes, surface_replies, notification_outbox, and tool_invocations it takes the row being resolved or claimed. The history projection grants are part of submitting a message to an existing conversation, as they are for the scheduler. The role reads no memory, People, email, knowledge, browser, or schedule table. Repair it as the schedule login above: revoke, reapply, and remove any surplus the validator still reports at its source.

Browser profile service host prerequisites

Reviewed site/account definitions are optional (ADR-0175). Set BROWSER_PROFILE_VERIFICATION_FILE in the host environment to an absolute private JSON file, owned by uid 65532 with mode 0600 and no symlink. The release validates that path before replacing containers and adds deploy/docker-compose.browser-verification.yml after the production overlay. Only the profile service receives this read-only file. The service rejects oversized or invalid catalogs at startup. Empty or unset host configuration keeps the existing differential verification; the overlay must then be omitted. Catalogs contain account identities and profile bindings: keep them outside Git, use synthetic identities in fixtures, and qualify both the authenticated and empty-session page before activation. A valid saved session must pass without a prompt; an expired or wrong-account session must stop before a lease is issued. Changing a bind-mounted file or this setting requires container recreation. Preserve the same overlay when manually recreating a configured service. Rolling application units back does not replace the browser-service container; an explicit browser-image rollback must preserve a catalog supported by that image or disable the catalog deliberately before recreation.

The reviewed morning X briefing has a separate default-off application switch, BROWSER_FEED_WORKFLOW_ENABLED=1 (ADR-0176). Before enabling it in the application environment shared by all roles, deploy matching application and browser-service images and install the qualified private X account definition above. The pinned observation tool must support version 1.8.0 article extraction; older pinned chats retain ordinary execution. Confirm the existing schedule's complete instruction still matches the reviewed catalog digest and its selected profile is the qualified account. After activation, verify a real scheduled occurrence reports observed personalized-feed evidence without scroll approvals. A feed that stays empty must report insufficient evidence. Clearing the switch disables recipe selection; an already suspended recipe fails closed instead of changing its definition. No production schedule or profile is rewritten by deployment.

The browser service mounts /tmp as a 512 MiB, memory-backed filesystem with noexec,nosuid,nodev, inside the container memory limit. Playwright's Chromium places shared-memory files there. Device sign-in verifies the supplied session against a second, signed-out browser concurrently; the former 128 MiB mount filled during these loads and crashed the browser drivers. Keep the two-browser capacity when changing container settings. Every hosted browser is headed (ADR-0145): one used about 610 MiB and 181 processes and threads on a light page, and a verification's two about 1.1 GiB and 370. The service admits at most three browsers at once, and the container allows 3 GiB, a gibibyte for each, and 1,024 processes; change the two together. The limit is a ceiling, not a reservation: the host needs that much memory free beside its other services. The service still enforces one thirty-second deadline for verification, including browser startup, page loading and session capture; a page load has no shorter deadline.

A hosted browser asks the egress proxy for nothing on its own except one connection to accounts.google.com as it starts (ADR-0146). The launch switches that stop the rest are in the service's code, and the image carries a Chromium managed policy for the one request no switch reaches. The switches name Chromium features and repeat a list from Playwright. Hosted CI repeats ADR-0146's measurement in the built image on x86_64 and arm64 whenever the image, its limits, the Playwright version or the switches change, and release packaging waits for it (ADR-0152). To measure locally before pushing, run make test-browser-image.

The release script refuses to run until the browser-profile secrets exist, even while BROWSER_PROVIDER=disabled, because the production compose file always starts the hardened profile service and bind-mounts these paths read-only. Provision them once:

umask 077
sudo install -d -m 0711 /etc/veetbot/secrets

# One token, two source files: the container refuses secrets it does not own
# (uid 65532), while systemd copies the deploy-owned control credential into
# each application unit's private credential directory.
credential="$(openssl rand -hex 32)"
printf '%s\n' "$credential" | sudo tee /etc/veetbot/secrets/browser-profile-service-auth >/dev/null
printf '%s\n' "$credential" | sudo tee /etc/veetbot/secrets/browser-control-plane-credential >/dev/null
unset credential
openssl rand -hex 32 | sudo tee /etc/veetbot/secrets/browser-profile-session-secret >/dev/null

# Keyring: the service does not generate keys; its mount is read-only.
sudo install -d -m 0700 /etc/veetbot/secrets/browser-profile-keys
openssl rand -base64 32 | sudo tee /etc/veetbot/secrets/browser-profile-keys/v1.key >/dev/null
printf 'v1\n' | sudo tee /etc/veetbot/secrets/browser-profile-keys/current >/dev/null

sudo chown 65532:65532 /etc/veetbot/secrets/browser-profile-service-auth \
  /etc/veetbot/secrets/browser-profile-session-secret
sudo chown -R 65532:65532 /etc/veetbot/secrets/browser-profile-keys
sudo chown veetbot-deploy:veetbot-deploy \
  /etc/veetbot/secrets/browser-control-plane-credential
sudo chmod 0600 /etc/veetbot/secrets/browser-profile-service-auth \
  /etc/veetbot/secrets/browser-profile-session-secret \
  /etc/veetbot/secrets/browser-control-plane-credential \
  /etc/veetbot/secrets/browser-profile-keys/v1.key \
  /etc/veetbot/secrets/browser-profile-keys/current

The service's fail-closed loader rejects any secret file with group or other permission bits, a wrong owner, a symlink, or multi-line content; key files must be base64 of exactly 32 bytes named <version>.key, with current naming an existing version. The profile-service mount paths are in deploy/veetbot.env.example. The application control credential is deliberately absent from that shared environment: the checked-in application units use systemd LoadCredential, while the deployment preflight reads the protected source as veetbot-deploy.

The committed Nginx virtual hosts expect Let's Encrypt certificates at /etc/letsencrypt/live/api.veetbot.com/, /etc/letsencrypt/live/browser.veetbot.com/, and /etc/letsencrypt/live/docs.veetbot.com/. The homepage certificate covering both veetbot.com and www.veetbot.com lives under /etc/letsencrypt/live/veetbot.com/. The Nginx installer changes only Veetbot's sites-available and sites-enabled entries; it preserves other virtual hosts.

Before merging a new static-host configuration to main:

1. Confirm that docs.veetbot.com resolves to the same address as api.veetbot.com. For the public website, the apex A record must target that address and www must be a CNAME to the apex. Wait for public resolution.
2. Provision the dedicated certificate at the exact path above. Prefer Certbot's DigitalOcean DNS plugin so issuance and renewal do not depend on a temporary HTTP virtual host. Keep its API token in a root-owned 0600 credentials file outside the repository and CircleCI.
3. Run sudo nginx -t without changing the existing production site.

The deployment deliberately does not bootstrap around a missing certificate: strict Nginx validation must fail rather than publish a plaintext or misidentified documentation endpoint.

The API virtual host keeps client_max_body_size 1m for every path except the chat attachment upload (ADR-0120), whose location allows 33m and streams the body to the application unbuffered. Activating attachments needs three changes together: set AGENT_ATTACHMENT_UPLOADS_ENABLED=1, add artifact.write and knowledge.write to the owner principal's AUTH_SCOPES, and let deploy-nginx publish the location after the release. Until the Nginx job lands, an upload above 1 MiB receives Nginx's own 413.

The browser.veetbot.com virtual host keeps client_max_body_size 64k except on the device sign-in handoff (ADR-0128). That location allows 1m, streams the body to the browser-profile service unbuffered through an in-memory buffer, so it is never written to disk, allows ten seconds between body reads, and waits sixty seconds while the service verifies the session. Every TLS server block allows only TLS 1.2 and 1.3. Like every Nginx change, these reach production through deploy-nginx after the release. Once it has run, check the handoff location from any machine:

head -c 70000 /dev/zero | tr '\0' ' ' | curl -s -o /dev/null -w '%{http_code}\n' -X POST -H 'Content-Type: application/json' --data-binary @- https://browser.veetbot.com/authentication/00000000-0000-0000-0000-000000000000/handoff

The 70,000-byte body is over the virtual host's 64 KiB and within the handoff's 1 MiB, and the service refuses a missing capability before it reads any body. So 401 means the location is live, 413 means Nginx still lacks it, and 404 means the browser-profile service predates device sign-in. A small body cannot tell these apart: both locations pass it and it gets 401. Then check the TLS floor:

openssl s_client -connect browser.veetbot.com:443 -servername browser.veetbot.com -tls1_1 -cipher 'DEFAULT@SECLEVEL=0' </dev/null 2>&1 | grep -Eo 'alert protocol version|Cipher is .*'

It should print alert protocol version and Cipher is (NONE): the server refused TLS 1.1. A cipher name after Cipher is means the server accepted TLS 1.1 and the floor is missing. No output means the local openssl could not offer TLS 1.1, so the check did not run. OpenSSL 3 offers it only with the lowered security level above, and macOS's LibreSSL rejects that option, so use OpenSSL 1.1 or later, such as Homebrew's openssl@3.

When a handoff answers 409 tool.browser.provider_unavailable, the browser-profile service's container log holds one line that says where the verification stopped (ADR-0128, amendment of 2026-10-06), for example:

WARNING agent_core.browser_control_plane.sessions device verification did not finish (with_session=settle) after 30.0 s

It names the load that stopped the verification and the stage it was in, then the seconds since the loads were started; docs/plan/browser-automation.md lists the stages. A time at the thirty-second deadline means the deadline stopped the named loads, and a shorter one means the named stage failed. Match the line to a handoff by the time of the 409 in Nginx's access log. The line holds no identifier and nothing about the website or the session.

BROWSER_PROFILE_DEVICE_SIGN_IN_ENABLED (true when unset) turns device sign-in off in the browser-profile service without a release. Remote sign-in is unaffected. Compose reads it from /etc/veetbot/veetbot.env only when it creates the container, so docker restart and docker compose restart keep the old value: the container must be recreated. Run the complete block below as root on the production host, in one shell. It sets the value, takes the deployment lock, recreates the service from the current release's image with the release's own Compose invocation, and confirms the container's value. Set value=true instead to turn device sign-in back on.

set -euo pipefail
value=false
environment_file=/etc/veetbot/veetbot.env
if grep -q '^BROWSER_PROFILE_DEVICE_SIGN_IN_ENABLED=' "$environment_file"; then
  sed -i "s/^BROWSER_PROFILE_DEVICE_SIGN_IN_ENABLED=.*/BROWSER_PROFILE_DEVICE_SIGN_IN_ENABLED=$value/" \
    "$environment_file"
else
  printf 'BROWSER_PROFILE_DEVICE_SIGN_IN_ENABLED=%s\n' "$value" >>"$environment_file"
fi
exec 9>/opt/veetbot/shared/deploy.lock
flock -w 900 9
cd /opt/veetbot/current
set -a
. "$environment_file"
. ./.release.env
set +a
export BROWSER_PROFILE_SERVICE_IMAGE="veetbot-browser-profile-service:$VEETBOT_RELEASE_ID"
compose=(docker compose --env-file "$environment_file"
  --project-name "${COMPOSE_PROJECT_NAME:-veetbot}"
  -f docker-compose.yml -f deploy/docker-compose.production.yml)
if [[ -n "${BROWSER_PROFILE_VERIFICATION_FILE:-}" ]]; then
  compose+=(-f deploy/docker-compose.browser-verification.yml)
fi
"${compose[@]}" up -d --no-build --wait --wait-timeout 60 browser-profile-service
docker inspect --format '{{range .Config.Env}}{{println .}}{{end}}' \
  "$("${compose[@]}" ps -q browser-profile-service)" \
  | grep -Fx "BROWSER_PROFILE_DEVICE_SIGN_IN_ENABLED=$value"
exec 9>&-

Browser task grants (ADR-0129) are off until the owner turns them on, and only after every device the owner uses runs a client build that understands approve_for_task. Turning them on takes three changes together in /etc/veetbot/veetbot.env: add browser.grant.read and browser.grant.write to the owner principal's AUTH_SCOPES, set BROWSER_TASK_GRANT_SCOPES optionally to initial exact site scopes (one public-HTTPS origin and one path segment each, comma-separated; production initially used https://www.duolingo.com/lesson), and set BROWSER_TASK_GRANTS_ENABLED=1. Then restart the API and the workers. Both settings require BROWSER_PROVIDER=hosted.

After the ADR-0141 migration, the enabled server seeds the owner's persisted list only when no policy exists. Thereafter use Settings → Website Access → Task approval websites on any client to add or remove entries. Changes are shared across clients and workers immediately; no deployment, environment edit or restart is needed. Clearing the list persists an empty policy, so restarts never repopulate it from the seed. Removing an entry ends its active task permissions. Adding an entry enables a task offer; the owner still approves each task. Unsetting the feature flag removes both management routes and task-grant routes and offers.

CircleCI setup

Generate a Veetbot-only Ed25519 deploy key on a protected operator machine:

umask 077
ssh-keygen -t ed25519 -N '' -f veetbot-circleci -C veetbot-circleci-production
ssh-keygen -lf veetbot-circleci.pub

Install only the public key for the veetbot-deploy account, using the restricted authorized_keys options described above. Add the private key to the Veetbot CircleCI project's SSH keys and do not add the Mankunku key or any personal key to this project. The deployment jobs intentionally load the project's attached keys, so the dedicated Veetbot key must be the only one present. Store and rotate it independently from every other application on the Droplet.

Create a restricted CircleCI context named veetbot-production with:

Variable	Value
DEPLOY_HOST	SSH hostname or IP for the Droplet
DEPLOY_USER	Dedicated deploy account, veetbot-deploy once provisioned; veetbot on the current host
DEPLOY_KNOWN_HOSTS	Pinned OpenSSH known-hosts record verified out of band
DEPLOY_PORT	Optional SSH port; defaults to 22
PRODUCTION_URL	Optional public origin; defaults to https://api.veetbot.com
DOCS_URL	Optional docs origin; defaults to https://docs.veetbot.com
WEBSITE_URL	Optional website origin; defaults to https://www.veetbot.com

Obtain the public host-key record from the server or provider console and verify its fingerprint through a second trusted channel before placing it in the context. Do not treat an unverified ssh-keyscan response as identity proof.

The context does not contain the API bearer token, database password, or model provider keys. Those stay in the protected server environment. Restrict context use to the repository and protected main branch.

macOS TestFlight delivery

The apple-testflight job uses two Apple credential boundaries that are deliberately separate from veetbot-production.

First, upload an active Apple distribution signing identity and the macOS App Store provisioning profile for com.veetbot.apple to CircleCI's code-signing store, then create a signing bundle named veetbot-app-store. The profile must carry the production entitlements used by the Release target, including APNs. CircleCI installs that identity and profile into its temporary managed keychain. Do not add the installer identity to this bundle: CircleCI exposes neither the keychain password nor the source PKCS#12, so productbuild cannot receive non-interactive access to its private key.

The archive selects Apple Distribution for the app signature. For the installer package, the job discovers the single installed 3rd Party Mac Developer Installer certificate and selects its SHA-1 fingerprint explicitly for productbuild. Keep those roles separate: the App Store provisioning profile contains the app distribution certificate, not the installer certificate.

Create a second restricted context named veetbot-apple-signing with:

Variable	Value
APPLE_INSTALLER_CERTIFICATE_BASE64	Single-line base64 of a password-protected PKCS#12 containing the Mac Installer Distribution private key, leaf certificate, and Apple WWDR G3 intermediate
APPLE_INSTALLER_CERTIFICATE_PASSWORD	The strong, unique PKCS#12 password

Restrict this context to the Veetbot project and the expression (pipeline.git.branch == "main" or pipeline.git.branch == "dev") and not job.ssh.enabled and not (pipeline.config_source starts-with "api"). The job decodes the PKCS#12 under a mode-restricted temporary directory, imports it into a fresh random-password keychain, grants only that imported key the apple-tool:, apple:, and codesign: partitions, passes the exact keychain to productbuild, and deletes both files on every exit. Keep the original private key in the owner's protected credential store only if certificate recovery is required; never place the PKCS#12, password, or decoded key in the repository, a project variable, cache, workspace, artifact, or log.

The apple-signing-smoke job proves this complete signing path in the requested dev pipeline that precedes a merge to main (ADR-0107). It runs the same repository-owned archive and package script as apple-testflight, including the archived identity checks, application-signature verification, productbuild, and pkgutil verification. It receives neither the veetbot-apple-testflight context nor an App Store Connect API key, does not invoke altool, persists no artifact, and therefore cannot publish its package. It does receive veetbot-apple-signing, so access to dev and changes to the CircleCI workflow are nevertheless release-signing security boundaries. Keep dev restricted to trusted maintainers and require the smoke job to pass on the exact revision proposed for main.

ADR-0219 also selects the isolated codex/recall-production-release branch. Before its ordinary requested pipeline, the operator must retain the original context restrictions and admit only the final CodeRabbit-reviewed revision. Preserve project/group restrictions and the SSH/API-configuration denials; add and verify each replacement expression before removing its predecessor. Restore the original expression after the attempt, including failure or cancellation, and retain readbacks. The workflow branch selector alone grants no signing access. This route does not require a push to dev or permit publication from the release branch.

Second, create a restricted context named veetbot-apple-testflight with:

Variable	Value
APP_STORE_CONNECT_API_KEY_ID	Ten-character App Store Connect key identifier
APP_STORE_CONNECT_ISSUER_ID	App Store Connect issuer UUID
APP_STORE_CONNECT_API_KEY_BASE64	Base64 encoding of the downloaded .p8 private key

The non-secret Apple team identifier remains authoritative in the checked-in Xcode project. The archive uses that Release build setting; do not duplicate the identifier in the restricted context.

Encode the private key without creating another plaintext copy:

APP_STORE_CONNECT_API_KEY_ID=REPLACE_WITH_KEY_ID
base64 -i "AuthKey_${APP_STORE_CONNECT_API_KEY_ID}.p8" -o -

Paste that output as the context value. Add a CircleCI project restriction for this repository to the context and restrict it to the protected main delivery path. Treat the managed signing bundle, both Apple contexts, and any CircleCI role that can reference or replace them as publication authority. The API key needs only the App Store Connect role and resource access required to upload Veetbot and manage its signing; it grants no server, model-provider, database, or Veetbot API authority. Keep the original .p8 in the owner's protected credential store because App Store Connect does not offer it for download a second time.

Before enabling the job, confirm that CircleCI's next pipeline.number is greater than the latest accepted macOS TestFlight build number for the current marketing version. Then configure the intended TestFlight group for automatic distribution and enable automatic updates in TestFlight on each Mac. Once CI owns the build counter, do not make a manual upload with a build number ahead of that counter.

Automatic delivery

A pipeline for any branch other than main starts no hosted workflow by itself; such a branch is verified on the sidecar and, before it is proposed for main, by one pipeline requested with run_verify: true (ADR-0107, docs/plan/development-toolchain.md). main is protected: the seven verification jobs are required status checks, so a pull request cannot merge until a requested run reports them green on its head commit; the owner retains an admin override. A requested dev pipeline also runs the non-publishing Apple signing smoke described above; the isolated recall release uses the exact-revision context admission in ADR-0219. On main, a verification job other than public-site whose source tree already passed it, normally in that requested run, reports success without rerunning; a merge commit whose tree nothing verified is verified in full (ADR-0114). After all seven required verification jobs pass:

• public-site installs the locked Node dependencies, builds, tests, and lints the static export, and exposes only that output to downstream packaging;
• package-release archives the exact tested commit, builds MkDocs in strict mode, stamps the website output, and records all three artifacts' SHA-256 values;
• deploy-app stages the archive, verifies the checksum, and executes the locked server release; and
• deploy-nginx follows a successful application release, takes the same host lock, atomically promotes /opt/veetbot/shared/docs/current and /opt/veetbot/shared/website/current, and reconciles the versioned virtual hosts. If a newer pipeline has already promoted another application release, the older proxy job detects the release-identity mismatch, reports a distinct stale outcome, and exits without overwriting the newer sites or config. CircleCI skips that older job's documentation and website identity probes because the newer release remains authoritative; and
• apple-testflight follows a successful application release in a separate serial group, archives and verifies the macOS application with pipeline.number as its build number, creates the signed installer package directly with productbuild, verifies the package signature, and uploads it to App Store Connect with altool and progress logging. The upload is bound to fifteen minutes: altool retries a rejected upload part without limit while still logging, so CircleCI's no-output timeout never fires, and on 2026-09-09 one such loop held the macOS runner for over forty minutes. A healthy upload finishes in about a minute; past the bound the step terminates altool and fails with exit 124, and a rerun is safe because Apple rejects a build number it has already accepted. This keeps signing and packaging separate from Apple API delivery.

Both deployment jobs use CircleCI's shared production serial group in addition to the server lock. The release ID is created with the packaged artifact and reused by failed-job reruns. A rerun of an already active release verifies it without extracting over the live source directory. So when deploy-app fails after promotion but https://api.veetbot.com/health/ready already returns the new X-Veetbot-Release, rerun the workflow from failed: deploy-app verifies without restarting anything, and deploy-nginx then ships the documentation, website, and proxy configuration that the failure skipped.

The Nginx installer stores the prior Veetbot configuration under /etc/nginx/veetbot-backups, runs nginx -t, and reloads Nginx. Validation or reload failure restores the prior file. So does a failure while installing the candidate file or activating its symlink. deploy/app/release.sh requires the local readiness header to identify the staged release and the authenticated session-index route to respond successfully; after it returns, CircleCI polls the public readiness endpoint for that exact identity. The Nginx job then requires both https://docs.veetbot.com/release.txt and https://www.veetbot.com/release.txt to return the same identity. A public probe failure is therefore a post-promotion CircleCI failure boundary. The release verifies every application process through its /proc working directory. The execution service instead relies on its checked-in ExecStart, the completed restart, and an active main PID because DynamicUser=yes deliberately prevents the deploy identity from dereferencing that process's /proc working directory. A post-promotion failure prints bounded systemd status for each managed unit before reporting the manual rollback target.

The manual and nightly live-model workflows remain tests; they do not deploy. The TestFlight upload job does not claim that Apple's asynchronous processing or device installation has completed; verify those external states in App Store Connect after the first automated delivery.

Verification

After the first successful pipeline, verify the active revision and services. DEPLOY_USER is the account CircleCI logs in as: veetbot on the current host, veetbot-deploy once the host is reconciled:

DEPLOY_USER=veetbot
curl --fail --show-error --dump-header - --output /dev/null \
  https://api.veetbot.com/health/ready
curl --fail --show-error https://docs.veetbot.com/release.txt
curl --fail --show-error https://www.veetbot.com/release.txt
ssh "$DEPLOY_USER@api.veetbot.com" '
  set -eu
  app_release="$(readlink -f /opt/veetbot/current)"
  . "$app_release/.release.env"
  docs_release="$(readlink -f /opt/veetbot/shared/docs/current)"
  website_release="$(readlink -f /opt/veetbot/shared/website/current)"
  test "$(cat "$docs_release/release.txt")" = "$VEETBOT_RELEASE_ID"
  test "$(cat "$website_release/release.txt")" = "$VEETBOT_RELEASE_ID"
  printf "%s\n%s\n" "$docs_release" "$website_release"
'
ssh "$DEPLOY_USER@api.veetbot.com" \
  'systemctl is-active veetbot-execution veetbot-api veetbot-worker veetbot-async-worker veetbot-maintenance'

Then make an authenticated API request, submit a run, confirm the worker completes it with a real provider, and run one generated-code task through runsc. Reboot once and confirm PostgreSQL, Nginx, the execution service, and all application units return.

The SMS capture ceremony

Milestone 24 (device-channel-and-sms.md) lets the owner's iPhone act as a Veetbot SMS channel: outbound texts ride the system compose sheet, and inbound texts reach Veetbot only because the owner wires a Shortcuts personal automation to the app's "Forward Message to Veetbot" App Intent. Nothing enables this by default. The owner completes the following one-time, per-device ceremony after the app is installed and paired (apple-client.md), on the Milestone 18 bootstrap-ceremony precedent (plan/email-integration.md, "Credentials and the bootstrap ceremony"): a documented, owner-run, one-time setup step with its own verification, rather than something CI or the server can perform for them.

1. Confirm prerequisites on the owner's iPhone: iOS 17 or later — the Shortcuts "When I get a message" personal automation trigger requires it; the device already paired to Veetbot and push-verified through POST /v1/devices and the authenticated POST /v1/devices/{device_id}/test-notification route; and, in the app's connection settings, the "SMS Integration" toggle (sms-integration.enabled) turned on. That one toggle both opens the compose-sheet send path and adds device.sms.send to this device's registered capabilities — turning it off later removes the capability and the App Intent silently stops forwarding.
2. In the Shortcuts app: Automation tab → the add control → Create Personal Automation → "When I get a message" → set the automation to run immediately, not "Ask Before Running" (it must fire unattended) → add the "Forward Message to Veetbot" action → map the trigger's Sender variable to the action's Sender parameter and its Content variable to the action's Message parameter → finish and confirm the automation is enabled.
3. Verify end to end: from a second phone, send the owner's number a short, distinctive test text, then confirm capture actually happened —
4. In Shortcuts, open the automation's run history and confirm it ran without a logged failure. This is what catches iOS having silently disabled the automation before the message ever reached the app.
5. In Veetbot, confirm the standing SMS triage session shows the forwarded message (device-channel-and-sms.md, "SMS ingest," which routes each message into one standing session per device and channel): a new conversation is seeded, or the existing standing (device, sms) session continues, with content matching the test text sent rather than an empty or truncated body.

Only a forwarded message with its real sender and body confirms, on this owner's specific iOS version, the design's two open questions: that the automation actually delivers both sender and body to the App Intent, and that the App Intent's Keychain read succeeds with the app backgrounded, which an immediately-run automation leaves it. Treat the ceremony as incomplete until this step passes.

Capture is deliberately best-effort, exactly as designed: iOS can silently disable a personal automation with no notice to the owner or to Veetbot, and the App Intent swallows every forwarding failure — the integration toggled off, the device unresolved, a network error — into a silent no-op rather than surfacing an error on the owner's phone. There is no retry and no alert when a text is silently dropped; the only signal is the absence of the triage session update step 3 above confirms. Re-run step 3 after an iOS upgrade, or whenever inbound capture seems to have gone quiet, to confirm it is still live. The send side has its own fragility: invocation expiry is judged against the device's own clock, so a phone whose clock is materially fast can silently expire a still-live request before the owner ever sees it. And a result the app could not post to the server survives only in that process's memory, so a crash or force-quit between the owner's Send tap and a successful result post can re-present an already-sent message the next time the app fetches, bounded by the server's five-minute invocation expiry.

Manual rollback

Schema-head equality is not enough for versioned JSONB discriminators: once any schedule revision has a definition.cadence.kind of MONTHLY or YEARLY, a release from before Milestone 20 cannot deserialize the database and is not a valid code-only rollback target. The runbook below detects the target's cadence support and checks the stored values while all schedule writers are stopped. If it refuses the target, roll forward instead, or restore a database snapshot from before the first such revision and then roll the code back. Never run an automatic Alembic downgrade as part of rollback. Run the complete block in one shell so file descriptor 9 holds the same deployment lock from before writer quiescence through readiness verification.

set -euo pipefail
environment_file="${VEETBOT_ENV_FILE:-/etc/veetbot/veetbot.env}"
exec 9>/opt/veetbot/shared/deploy.lock
if ! flock -w 900 9; then
  echo "Could not acquire /opt/veetbot/shared/deploy.lock" >&2
  exit 1
fi

target_id=YYYYMMDD-HHMMSS-abcdef0
target="/opt/veetbot/releases/$target_id"
docs_target="/opt/veetbot/shared/docs/releases/$target_id"
website_target="/opt/veetbot/shared/website/releases/$target_id"
test -d "$target"
test -f "$target/.release.env"
test -x "$target/.venv/bin/python"
grep -Fqx "VEETBOT_RELEASE_ID=$target_id" "$target/.release.env"
test -d "$docs_target"
test -f "$docs_target/release.txt"
test "$(cat "$docs_target/release.txt")" = "$target_id"
test -d "$website_target"
test -f "$website_target/release.txt"
test "$(cat "$website_target/release.txt")" = "$target_id"
test -L /opt/veetbot/current
test -L /opt/veetbot/shared/docs/current
test -L /opt/veetbot/shared/website/current
previous_target="$(readlink -f /opt/veetbot/current)"
previous_docs_target="$(readlink -f /opt/veetbot/shared/docs/current)"
previous_website_target="$(readlink -f /opt/veetbot/shared/website/current)"
test -d "$previous_website_target"
test -f "$previous_website_target/release.txt"
test -f "$environment_file"
docker image inspect "agent-core-sandbox:$target_id" >/dev/null
previous_production_image="$(
  docker image inspect --format '{{.Id}}' agent-core-sandbox:production
)"
test -n "$previous_production_image"

managed_units=(
  veetbot-execution
  veetbot-maintenance
  veetbot-worker
  veetbot-async-worker
  veetbot-api
)
for optional_unit in veetbot-schedule veetbot-notify; do
  if sudo systemctl is-enabled --quiet "$optional_unit"; then
    managed_units+=("$optional_unit")
  fi
done

target_calendar_compatibility="$({
  cd "$target"
  "$target/.venv/bin/python" - <<'PY'
from agent_core.domain.schedules import CadenceKind

required = {"MONTHLY", "YEARLY"}
available = {kind.value for kind in CadenceKind}
print("compatible" if required <= available else "incompatible")
PY
})"
case "$target_calendar_compatibility" in
  compatible | incompatible) ;;
  *)
    echo "Could not determine the target's schedule cadence compatibility" >&2
    exit 1
    ;;
esac

app_next="/opt/veetbot/.rollback-$target_id-$$"
docs_next="/opt/veetbot/shared/docs/.rollback-$target_id-$$"
website_next="/opt/veetbot/shared/website/.rollback-$target_id-$$"
app_restore="/opt/veetbot/.rollback-restore-$$"
docs_restore="/opt/veetbot/shared/docs/.rollback-restore-$$"
website_restore="/opt/veetbot/shared/website/.rollback-restore-$$"
health_headers=""

rollback_pending=0
rollback_on_exit() {
  status=$?
  trap - EXIT
  if test "$status" -ne 0 && test "$rollback_pending" -eq 1; then
    set +e
    rm -f -- "$app_restore" "$docs_restore" "$website_restore"
    ln -s "$previous_target" "$app_restore"
    ln -s "$previous_docs_target" "$docs_restore"
    ln -s "$previous_website_target" "$website_restore"
    mv -Tf "$app_restore" /opt/veetbot/current
    mv -Tf "$docs_restore" /opt/veetbot/shared/docs/current
    mv -Tf "$website_restore" /opt/veetbot/shared/website/current
    docker tag "$previous_production_image" agent-core-sandbox:production
    sudo systemctl restart "${managed_units[@]}"
  fi
  if test -n "$health_headers"; then
    rm -f -- "$health_headers"
  fi
  rm -f -- "$app_next" "$docs_next" "$website_next" \
    "$app_restore" "$docs_restore" "$website_restore"
  exit "$status"
}
trap rollback_on_exit EXIT

rollback_pending=1
sudo systemctl stop "${managed_units[@]}"
if test "$target_calendar_compatibility" = incompatible; then
  incompatible_schedule_revisions="$({
    set -a
    # shellcheck disable=SC1090
    . "$environment_file"
    set +a
    : "${POSTGRES_USER:?POSTGRES_USER is required}"
    : "${POSTGRES_DB:?POSTGRES_DB is required}"
    timeout --signal=TERM --kill-after=5s 20s \
      docker compose \
        --env-file "$environment_file" \
        --project-directory "$previous_target" \
        --project-name "${COMPOSE_PROJECT_NAME:-veetbot}" \
        -f "$previous_target/docker-compose.yml" \
        -f "$previous_target/deploy/docker-compose.production.yml" \
        exec -T \
        --env PGCONNECT_TIMEOUT=5 \
        --env 'PGOPTIONS=-c statement_timeout=5000' \
        postgres psql \
        --no-psqlrc --tuples-only --no-align \
        --username "$POSTGRES_USER" --dbname "$POSTGRES_DB" \
        --command "SELECT count(*) FROM schedule_revisions WHERE definition #>> '{cadence,kind}' IN ('MONTHLY','YEARLY');" \
      | tr -d '[:space:]'
  })"
  case "$incompatible_schedule_revisions" in
    '' | *[!0-9]*)
      echo "Could not validate stored schedule cadence values" >&2
      exit 1
      ;;
  esac
  if test "$incompatible_schedule_revisions" -ne 0; then
    echo "Refusing a pre-Milestone-20 target: incompatible schedule revisions exist" >&2
    exit 1
  fi
fi
ln -s "$target" "$app_next"
ln -s "$docs_target" "$docs_next"
ln -s "$website_target" "$website_next"
docker tag "agent-core-sandbox:$target_id" agent-core-sandbox:production
mv -Tf "$app_next" /opt/veetbot/current
app_next=""
mv -Tf "$docs_next" /opt/veetbot/shared/docs/current
docs_next=""
mv -Tf "$website_next" /opt/veetbot/shared/website/current
website_next=""
sudo systemctl restart "${managed_units[@]}"
health_headers="$(mktemp /opt/veetbot/shared/rollback-health.XXXXXX)"
ready=0
for ((attempt = 1; attempt <= 180; attempt++)); do
  : >"$health_headers"
  if curl --fail --silent --show-error --connect-timeout 2 --max-time 5 \
    --dump-header "$health_headers" --output /dev/null \
    http://127.0.0.1:8000/health/ready \
    && awk -F ': *' -v expected="$target_id" '
      tolower($1) == "x-veetbot-release" {
        sub(/\r$/, "", $2)
        if ($2 == expected) found = 1
      }
      END { exit found ? 0 : 1 }
    ' "$health_headers"; then
    ready=1
    break
  fi
  sleep 1
done
if test "$ready" -ne 1; then
  echo "The restarted API did not report $target_id in 180 attempts" >&2
  exit 1
fi
rm -f -- "$health_headers"
health_headers=""
test "$(cat /opt/veetbot/shared/docs/current/release.txt)" = "$target_id"
test "$(cat /opt/veetbot/shared/website/current/release.txt)" = "$target_id"
for unit in "${managed_units[@]}"; do
  sudo systemctl is-active --quiet "$unit"
  pid="$(sudo systemctl show --property MainPID --value "$unit")"
  test "$pid" -gt 0
  process_cwd="$(readlink -f "/proc/$pid/cwd")"
  test "$process_cwd" = "$target"
done
rollback_pending=0
trap - EXIT

The returned X-Veetbot-Release and both public surfaces' release.txt files must equal target_id, and each unit's MainPID working directory must resolve to target. The readiness check waits up to 180 one-second attempts, the same budget as release.sh, because the API refuses connections until it has started. The matching documentation releases are preconditions, so a missing or mismatched release.txt fails before any symlink changes. Image validation and tagging also precede all three pointer switches. If either public-surface switch, service restart, or readiness check fails, the exit trap restores all three previous pointers and the previous production image tag, then restarts the previous release; deploy/app/rollback.test.sh failure-injects the second switch to verify that recovery. Image retention keeps the active release's image and its predecessor, so the newest rollback target normally still has its image; if an older release image was pruned, rebuild it from that retained source tree before switching.

Nginx backups are independent. To recover one, copy the selected file from /etc/nginx/veetbot-backups to /etc/nginx/sites-available/veetbot, run sudo nginx -t, and reload Nginx.

Accepted limitations

Milestone 15 (operational-hardening.md, ADR-0065) specifies the backup, restore rehearsal, alerting, firewall, rollback, and watchdog controls that shrink this list to the residual single failure domain; until that milestone lands, the limitations below stand.

This is still a single-server failure domain with no required cloud firewall, off-host database, backup, restore rehearsal, monitoring, load balancer, rolling deployment, or high availability. Loss of the Droplet may mean unrecoverable data loss. The API binds port 8000 only on loopback, so remote clients must use the Nginx TLS hostname. A firewall is still recommended to contain any unrelated service that is accidentally bound to a public interface.

Optional Scale VP data access

ADR-0153 is off by default. To enable it:

1. On a machine with a browser, from a checkout of the release, run the sign-in once and complete the Scale VP page it opens:

uv run python -m svp_mcp bootstrap --output-file "$HOME/.config/veetbot/svp/credential.json"

It prints how many read operations the API published to the new grant. If it fails, no file is written. 2. Move that file to the host; do not copy it. The grant has one holder, and a second copy in use ends it. Install it in a directory the service user owns:

sudo install -d -o veetbot -g veetbot -m 0700 /var/lib/veetbot/svp
sudo install -o veetbot -g veetbot -m 0600 credential.json /var/lib/veetbot/svp/credential.json

1. Set AGENT_SVP_ENABLED=1 and SVP_CREDENTIAL_FILE=/var/lib/veetbot/svp/credential.json in /etc/veetbot/veetbot.env only, then restart the API and the three workers. Their units allow writes to that directory when it exists.

New chats are then offered mcp.svp_read.list_operations, describe_operation and call_operation; scheduled runs are not. On the production roster the three are entries in the deferred tool index, reached with tool.call, and the production-roster gates include the flag (ADR-0124). A call that returns svp.credential_rejected after the grant was working means the grant has ended: repeat steps 1 and 2 with a new file.

Optional Bland calling services

Bland setup defines the private configuration, distinct database roles, separate ingress Linux user and required live verification. The release script leaves both roles disabled by default. When enabled it validates matching owner/profile bindings, installs veetbot-call.service and optionally veetbot-call-ingress.service, then starts them after promotion. That preflight runs as the deploy identity without sudo. The guide's file permissions let that identity read both calling environment files and reach, but not read, their credentials. The proxy adds only the exact /webhooks/bland route when the active release enables intake. Verify both units independently alongside the normal public release probes. Before rolling back across calling support, stop both calling units and restore the target release's environment and proxy flags; only restart roles that exist in that release. Do not downgrade the call migration to roll back code: preserve correspondence and erasure tombstones, following the normal expand-only policy.

The release tree is created with umask 027, and veetbot-call-ingress is deliberately outside the veetbot group. When ingress is enabled, the release therefore grants that user read access to the staged release with a recursive ACL (setfacl -R -P -m u:veetbot-call-ingress:rX) as the last step before promotion. It runs after the deployment check, whose uv run can reinstall the project's entry points and would leave a reinstalled agent unreadable. Calling units have no readiness probe: after the API checks, the release waits until VEETBOT_CALL_SETTLE_SECS (default 90) have passed since it restarted them, then requires each enabled calling unit to be active with a main process and no automatic restart. Otherwise the release fails after promotion, like any other unit failure. These probes use the calling rules in deploy/sudoers/veetbot-deploy, so reinstall that contract on hosts provisioned before calling support.

For ADR-0143 notification reconciliation, include notification.write in the owner client's AUTH_SCOPES and bearer-token grants. Apply the migration before starting the new API and dispatcher. The API database role needs SELECT and INSERT on notification_run_receipts; the notify and surface dispatcher roles need SELECT (the surface allowlist above includes it). Check these privileges for the deployed database roles and grant any missing ones before restarting dispatch. Apple clients from before this change still receive alerts; cleanup and viewed-result acknowledgement require the updated client.

Owner-reviewed dreaming rollout (ADR-0174)

Deploy the reviewed-dreaming build and run its normal migration to the current schema head (daily review state was introduced in ea3210a1b00c). Apply these same flags to every enabled application role:

AGENT_MEMORY_API_ENABLED=1
AGENT_MEMORY_RECONSOLIDATION_API_ENABLED=1
AGENT_MEMORY_DREAMING_REVIEW_ENABLED=1
AGENT_MEMORY_RECONSOLIDATION_ENABLED=0
AGENT_MEMORY_RECONSOLIDATION_RESIDENCY_PROVIDER=<configured-memory-provider>

The residency provider must match the resolved memory formation provider. The provider must be configured and available; ordinary privacy/attribution checks still apply to every source. Retain memory.read,memory.write in the owner's existing token scopes. The API, interactive/asynchronous workers and maintenance read /etc/veetbot/veetbot.env; schedule, notify, surface and call workers read their own /etc/veetbot/veetbot-<role>.env files. Update only the five flags in each enabled role's file, preserving its credentials and permissions; do not copy the owner's bearer or database credentials between roles. Restart the API, interactive and asynchronous run workers, maintenance, and other enabled application roles so every process uses the same reviewed-recall mode. Then open https://api.veetbot.com/dreaming and connect with the owner token. Do not place a token in a URL. No additional proxy route or external web service is needed: the existing API upstream serves the page and all requests.

The next maintenance sweep claims a daily slice; after that, the schedule is at least 24 hours from its previous claim. Check Run history for proposed, empty, blocked, failed or interrupted outcomes. No provider call is made by opening the page or approving an operation. Inspect the exact statement and supporting originals, select Approve and apply, and confirm. Applied merges can be undone here. Summaries/connections retain the existing memory browser rejection/deletion controls. The HTTPS review session lasts up to thirty days (ADR-0186); reloading reads saved server decisions without another token. Pause previews from Review settings to stop new daily claims; an already running bounded slice may finish.

An empty or blocked run can mean there are no eligible attributed memories; it does not authorize reconstructing lost source evidence or relaxing privacy. Unattended dreaming remains off and still needs the unchanged precision, answer-lift and release-evidence requirements. The review path needs no new blinded packet. To stop reviewed recall and proposals, disable the review flag and restart those same roles; retain the operation API for inspection and undo. Do not downgrade the database after review/scheduling state exists: the migration refuses to discard that history.

Forward-only cutover after legacy attribution loss (ADR-0202)

Deploy the current schema head, including migration ea3210a1b00d, before activation. With the existing protected owner operator environment loaded, run:

agent memory reconsolidations forward-only --owner TENANT/PRINCIPAL --confirm

Use the exact configured owner identity. The command refuses an active dreaming lease; retry after that bounded slice finishes. A successful retry returns the same cutover, never a later one. It retires queued legacy groups while retaining their records, excludes existing memories and source messages, and leaves the next daily opportunity and spend budgets unchanged. It restores no deleted text.

Verify that the authenticated review page shows the cutover date and that its current legacy blocker is absent. Retained blocked run receipts remain visible. The next daily run may legitimately report no change until enough suitable new evidence exists. Keep supervised review enabled and unattended application off; new proposals still require individual approval. Never remove an activated boundary or replace unknown legacy attribution with empty metadata to unblock work.

Owner-reviewed bounded recall rollout (ADR-0220)

The bounded exception uses recall-bounded-evidence@1 and recall-bounded-admission@1; legacy passing-evidence admission is unchanged. Do not retag the completed failed comparison as a passing M33 release. The owner's approval of rollout scope is not a completed human failure review.

Before enabling it, retain and byte-verify the complete original collection, failed attempts and every ledger. Obtain the actual human review of difficult answers and all interruption/uncertain-cost items, including residual-risk findings. Recompute the separate comparative criteria and retain every original quality exception. Record exact measured/target file and dependency differences, their individual dispositions, and verification receipts on the final target. The compatibility claim must cover the actual release tree and effective values, including the PostgreSQL schema, reader, formation and verified local encoder. Unknown material differences require new matching evidence, not a copied hash.

Retain the previous effective configuration and a schema-compatible application, documentation and website rollback revision using the rollback procedure above. Rehearse evidence withdrawal and expiry against the release configuration before cutover. The manifest must name the final reviewed commit, evidence hash, complete target-input manifest, configured tenant/owner, rollback revision and a fixed activation/expiry window no longer than seven days. No wildcard or descendant inheritance is permitted. Supply the two files through the existing semantic index evidence/manifest settings, with the local encoder and query preparation enabled and unattended reconsolidation disabled. Existing request and USD 2 daily indexing ceilings remain unchanged.

Complete CodeRabbit and exact-head CI before merge. Follow application/docs/ website delivery and verify public revision identities, authenticated owner recall, and sparse fallback. Expiry or observed file/source mutation withdraws the current admission handle and its query cache; restoring a file does not revive that handle. For a confirmed access/lifecycle breach or unacceptable obsolete answer, immediately disable AGENT_MEMORY_SEMANTIC_INDEX_ENABLED, restore the retained configuration and restart. If an application rollback is needed, use the compatible retained release without downgrading the database. Verify readiness and public identities after either action, retain the incident, and record the rollout outcome before any separately authorized extension.

Application reliability diagnostics and optional alerts (ADR-0205)

After deploying the reliability changes, use the existing protected owner operator environment to run agent run reliability --hours 24 --json. It reports durable run outcomes, failure groups and current queue health without selecting prompts, drafts, exception messages or tool result bodies. It does not count API requests that failed before creating a run. A missing completion rate means no observed outcomes, and completion latency includes approval waiting.

For a failed task, copy its native diagnostic reference and correlate it with diagnostic_id in the appropriate service journal. API errors retain their request ID; search journal request_id to find their diagnostic. First-party locations, bounded cause classes, error_fingerprint, run/session/tool IDs and release_id identify the failing code without recording exception text or local variables. Group recurring errors by fingerprint within a release; compare code locations across releases, since source movement can change fingerprints. Keep journal access restricted to authorized operators.

To activate the optional application alarm after review and deployment:

1. Verify the PostgreSQL schema is current and the existing notification dispatcher and approved delivery destination work. Run the read-only report.
2. Install deploy/systemd/veetbot-reliability.service and deploy/systemd/veetbot-reliability.timer in /etc/systemd/system/ as root. These are operator-managed host units; delivery does not enable them.
3. Install deploy/ops/reliability.env.example as /etc/veetbot/reliability.env, owned by root and mode 0600. Review its one-hour window, minimum ten outcomes, twenty-percent failure rate, three internal errors and 120-second ready-queue threshold. Set AGENT_RELIABILITY_MONITOR_ENABLED=1 only when ready for notifications.
4. Run sudo systemctl daemon-reload, then sudo systemctl start veetbot-reliability.service. Inspect the journal and outbox delivery. Exit 0 means no current threshold breach, 2 means a degraded observation was recorded, and 1 means the check could not complete. The unit accepts 2 as a successful measurement; timeouts remain failures.
5. Enable recurrence with sudo systemctl enable --now veetbot-reliability.timer. Disable it with sudo systemctl disable --now veetbot-reliability.timer and reset the flag to 0 before rolling back to a release without this command.

--check alone is read-only. Adding --notify requires enabled notification configuration and writes only alarm transitions and outbox records. Concurrent checks do not duplicate alerts; recovery and a later new episode each produce one transition. Quiet traffic does not clear an error-rate alarm. Threshold changes apply at the next observation; review them as operational changes.

This monitor cannot deliver when PostgreSQL, the host or the notification path is down. Retain an independent external uptime/dead-man channel and complete Milestone 15's backup/restore and host-hardening work. Deployment and activation are separate from branch implementation; neither is implied by passing tests.