deploy/solver-fleet/README.md
On this page

External Hetzner solver and mirror fleet

Activated on 178.105.23.35 on 2026-09-09 after live production attestation verification. This uses a separate system account, environment, configuration and state directory; existing host services are unchanged.

The live host was upgraded to a2 at 21:23:06 UTC on 2026-09-09 after actual PyPI wheel verification and a fresh production attestation. The retained procedure is in UPGRADE-A2.md, with safe upgrade evidence. The previous a1 environment, configuration backups, artifacts and checkpoints remain preserved.

The solver unit permits nine full RandomX workers with a 900% CPU quota, 22 GiB memory pressure threshold and 24 GiB hard memory limit. Nine datasets and caches used 20.61 GiB in the concurrent benchmark. This leaves capacity for other work on the 30 GiB host. Lower scheduling priority can slow recovery under load; these limits do not establish a seven-day wall-clock guarantee. RandomX's secure JIT needs executable mappings, so the unit does not prohibit all JIT memory.

The mirror unit has a 384 MiB memory limit and a 50% CPU quota. It reads public artifacts, preserves all prior files, uses conditional creates, and verifies replicas by reading their complete bytes back. It always writes the independent Hetzner directory and can additionally write the scoped AWS archive prefix.

Activation gate

/etc/attested-relay-fleet-v2/fleet.json is intentionally absent after provisioning. Both units require that file, and the runner refuses a missing origin, zero/debug PCR0, a malformed exact 96-character PCR0, unsafe HTTP origins, or more than nine workers. fleet.pending.json is a nonfunctional template, not an active config.

When the exact production origin and independently verified production PCR0 are available, create the active configuration with public values:

{
  "origin": "https://relay.example",
  "expected_pcr0": "REPLACE_WITH_EXACT_96_LOWERCASE_HEX_PCR0",
  "max_workers": 9,
  "poll_seconds": 60,
  "insecure_local": false,
  "s3_replica": {
    "bucket": "attested-relay-archive-370686332139-us-west-2",
    "region": "us-west-2",
    "profile": "relay-archive-v2"
  }
}

The optional s3_replica accepts only bucket, region and profile names. Its prefix is fixed to artifacts/. It does not accept credential values. The mirror's SDK profile file is /etc/attested-relay-fleet-v2/aws-credentials, owned by root:relay-fleet-v2 with mode 0640, inside the 0750 root-owned configuration directory. A newly provisioned service credential should permit only s3:GetObject and s3:PutObject for that archive prefix. No existing/root AWS credentials or private SSH keys were copied by these provisioning scripts.

The solver authenticates each historical bundle to the configured PCR0 before starting computation. A stale diagnostic bundle or other invalid bundle is rejected without stopping unrelated authenticated jobs. No allow_dev or unverified-hardware override is supplied by the production runner.

Validate the configuration before enabling the units:

/opt/attested-relay-fleet-v2/venv-0.2.0a2/bin/python \
  /opt/attested-relay-fleet-v2/run-fleet.py validate
systemctl enable --now attested-relay-mirror-v2.service attested-relay-solvers-v2.service

These activation commands are separate from provisioning. An explicitly authenticated SSH tunnel may temporarily provide http://127.0.0.1:PORT with insecure_local:true; a tunnel through the operator's laptop is not independent continuous availability and must be replaced by a direct durable tunnel or public HTTPS origin before claiming an autonomous fleet.

The deployed configuration uses the direct persistent Hetzner-to-AWS tunnel at http://127.0.0.1:29081 with insecure_local:true, the production PCR0 recorded in measurements/hetzner-fleet-upgrade-a2-20260909/a2-production-warming-be6740b022714735878ffd53e80a26bc.json, and the scoped relay-archive-v2 S3 profile. The tunnel, mirror and solver units were all verified enabled and running with zero restarts. The first complete mirror scan discovered zero artifacts and reported zero failures: the authenticated production enclave was still generating its first epoch. The nine-worker solver daemon is therefore idle until authenticated bundles appear. This activation does not demonstrate a completed production puzzle or production record recovery.

The fleet was increased from seven to nine workers after concurrent benchmarks showed that seven could not keep up. Nine independent processes measured 3,907.78 hashes/s collectively, estimating one epoch per 21.78 hours against the measured 25.14-hour generation cadence (15.44% capacity margin). Individual recovery is estimated at 7.82–8.69 days on this host; the seven-day reference is the measured Graviton5 solver rate, not a guaranteed Hetzner delay. Both benchmark datasets are retained under measurements/hetzner-{seven,nine}-solver-calibration-20260909/.

Provisioning and preserved updates

provision.py installs the exact releases from public PyPI into the isolated environment, creates only the dedicated account/paths, verifies units, and reloads systemd configuration. It never starts or enables units. A repeated run refuses to overwrite differing installed files. managed-update.py updates only this fleet's inactive managed files, preserving every previous version under a content-hash backup name. No files are removed.

Actual diagnostic evidence

On 2026-09-09, the installed public Linux wheel and client strictly verified a real, nondebug Nitro diagnostic enclave through an authenticated temporary SSH tunnel. The proof mirrored and read back five artifacts, solved one eight-iteration RandomX puzzle, and decrypted two actual recorded exchanges.

The public summary is retained in evidence/hetzner-diagnostic-proof-20260909.json. Remote ciphertext, signed evidence, recovered diagnostic key, decrypted fixture records and readback receipts remain under /var/lib/attested-relay-fleet-v2/diagnostic-20260909/. This proves interoperability and independent recovery, not a production-duration delay. The diagnostic PCR0 must never be substituted for the production PCR0.

The small diagnostic-proof.py command refuses puzzles with iteration counts other than eight, keeping the one-shot test separate from the production fleet. Local configuration tests run with:

python3 -m unittest discover -s deploy/solver-fleet -v

Scoped S3 verification

evidence/hetzner-scoped-s3-proof-20260909.json records a real conditional upload of diagnostic ciphertext, complete readback, an idempotent repeat and rejection of a duplicate conditional write. The test ran as relay-fleet-v2 using only the new relay-archive-v2 profile. Its proof object remains in the archive bucket. No deletion, IAM policy change or retention setting was used. Existing versioning and a credential without delete access do not provide S3 Object Lock retention.

verify-warming.py authenticates a fresh live Nitro challenge to an exact production PCR0 and iteration count before fleet activation. It checks current certificate validity, signature, nonce, freshness, actual inner TLS peer binding and verified Graviton5 hardware, then requires the signed state to be warming. Its public proof explicitly reports application_ready:false; it sends no proxy request and does not relax the installed client's application readiness gate.