deploy/solver-fleet/UPGRADE-A2.md

Reviewed upgrade to 0.2.0a2

This upgrade was activated at 21:23:06 UTC on 2026-09-09, after the operator verified all published wheel bytes and reproduced the production PCR. The exact activated PCR is 51e61c21424fabbf27f3d8a3e8a6a14af8538752c9d55238b33b00c4bda2e01392ab27131f5769defa33d39270a2f2b7. Fresh live attestation, installed package provenance, service limits and retained artifact readback were verified. Evidence is in measurements/hetzner-fleet-upgrade-a2-20260909/. Commands below describe the performed procedure, not a request to repeat the one-time upgrade.

Review the service health and limits, actual PyPI installation report, fresh signed attestation, and post-upgrade readback.

The first install attempt saw a stale PyPI index and stopped with only baseline pip in the new environment. Its log and initial staged script were preserved. After public bytes were verified, explicit prepare --resume-empty recorded that baseline and installed the released wheels successfully with pip caching disabled.

The expected measured software is exactly 0.1.0+a10323dede4413fbf295916b8ad12e3dbad7514e, with 43,768,124 iterations, seven segments, a 86,400-second epoch and verified Graviton5 hardware. The service remains nine independent full-mode solvers with CPUQuota=900%, MemoryHigh=22G, MemoryMax=24G, no swap and Nice=10. The origin stays http://127.0.0.1:29081 through the existing direct Hetzner-to-AWS tunnel; the scoped relay-archive-v2 S3 profile and all archive, receipt and checkpoint paths are preserved.

Stage reviewed files

Create a new root-owned staging directory accessible for read/execute by the service account: /opt/attested-relay-fleet-v2/upgrade-0.2.0a2/, mode 0755. Copy these source files there without replacing live deployment files:

The source runner and unit release pins now target a2. The separate a1 venv remains untouched, including its installed package snapshots.

Prepare after PyPI publication is confirmed

python3 /opt/attested-relay-fleet-v2/upgrade-0.2.0a2/upgrade-a2.py prepare

This creates a new /opt/attested-relay-fleet-v2/venv-0.2.0a2, installs both exact a2 packages from the actual public PyPI index plus boto3, and retains pip's JSON download report and installed version list. It refuses an existing a2 environment instead of modifying it. As relay-fleet-v2, it verifies installed package and native executable locations in that venv, then runs pip check. No service or active configuration changes during preparation.

If preparation fails, preserve the partially prepared environment and logs for inspection. Do not remove or silently reuse it. If inspection shows that package installation never began and only baseline pip (optionally setuptools) exists, an explicitly authorized retry may use prepare --resume-empty with a new log file. This records the inspected inventory, refuses any additional installed package or completed preparation, and bypasses pip's cache. A previous pip report also stops the retry for inspection rather than being overwritten.

Activate only after production identity is confirmed

python3 /opt/attested-relay-fleet-v2/upgrade-0.2.0a2/upgrade-a2.py activate \
  --expected-pcr0 CONFIRMED_EXACT_PRODUCTION_PCR0

Activation first validates a candidate public configuration and uses the new installed client to make a fresh challenge over the actual inner TLS connection. It verifies the Nitro signature and AWS chain, current certificate validity and signed timestamp freshness, a random nonce, exact PCR0, actual TLS SPKI, verified Graviton5 hardware, exact RandomX policy and exact software version. The signed state must be warming with no current epoch. The public proof is retained in the service account's smoke/ directory. An archive document or plain host status is not accepted as a replacement for this live challenge.

Immediately before stopping the solver, the script checks its process has no native children. It saves the prior runner, active configuration and both units in content-addressed read-only backups, then briefly stops only the idle solver. It installs the reviewed files, validates systemd configuration, starts the solver and restarts the mirror so both use the new venv. The tunnel stays running. It checks both service states and the actual solver command's venv and PCR before writing the activation proof. If validation or restart fails, it preserves the attempted files too and restores the prior files and services without deleting either environment or any state.

After activation, inspect actual systemd limits, running command arguments, idle job count and mirror scan receipt. An empty index during authenticated warm-up is healthy; it is not evidence that a production puzzle has been solved.

Focused local checks are:

python3 -m unittest discover -s deploy/solver-fleet -p test_config.py -v
python3 -m unittest discover -s deploy/solver-fleet -p test_upgrade.py -v

They cover the nine-worker bound, rejecting ten workers, exact production work and software gates, and preservation of both old and attempted configuration bytes. They do not replace the live attestation and post-restart health checks.