Graviton5 RandomX scaling investigation — 2026-09-11

The production slowdown reproduces on a fresh c9g.8xlarge. A substantial part comes from the way secure JIT workers interact with the kernel when they share one process. Separate worker processes improve measured enclave throughput by 1.688×. Combining that with a smaller JIT permission range gives 2.095× the original throughput, without changing RandomX or disabling write/execute protection.

These are synthetic benchmarks and an optimization prototype. The running production generator has not been replaced or restarted.

Environment and method

• Separate EC2 instance i-0eb568edd6dbc6af3, c9g.8xlarge, us-west-2; 32 physical ARM cores, 64 GiB RAM. No production keys or VPN credentials.
• Parent: Amazon Linux 2023, Linux 6.18.44-99.149.amzn2023.aarch64.
• Benchmark Nitro enclaves: 24 physical cores, 8192 MiB RAM, debug enabled. Kernel 4.14.256-209.484.amzn2.aarch64; kernel PCR1 matches production: 3b4a7e1b5f13c5a1000b3ed32ef8995ee13e9876329f9bc72650b918329ef9cf4e2e4d1e1e37375dab0ba56ba0974d03.
• Frozen production source 30feebbeea8588fb1d1aa7b5ef40c9903bec0df5. GCC 11.5, release build, ARM crypto instructions, full dataset, RandomX V2, secure JIT. Published a4 native executable also tested independently.
• Each worker is pinned to a distinct physical core. Each hash consumes its previous result. Workers have different chains; no chain is parallelized.
• The harness initializes the dataset once, then starts threads or forks processes. Dataset pages are inherited; each worker creates its own VM. Dataset initialization is outside the measured hashing interval.
• Short screens use 10,000 hashes per worker. Confirmation uses 30,000 hashes per worker in thread/process/process/thread order, without profiling. The 24-thread parent screen alone was profiled and is exploratory.

Confirmed baseline versus process workers

These are means of two trials each. Production itself has sustained roughly 6,900 hashes/s including its actual generation workload, about 18.7 hours for 465M. Projections above are hashing estimates, not full deployment timings.

The published native binary achieved 679.3 hashes/s with one enclave worker, but only 7,050.3 total with 24. The benchmark reproduces the scaling problem. Outside the enclave, the process harness achieved 16,162.9 total with 24 workers (673.5 each). The remaining enclave gap is not fully explained.

Why this helps

Secure RandomX JIT repeatedly switches generated code between writable and executable permissions. There are eight generated programs per hash, with two permission transitions per program. Threads share a process memory map and its kernel locks. Separate processes remove this shared lock bottleneck.

Evidence supporting that explanation:

• For 30,000 hashes, thread workers averaged 53.4 seconds of user CPU time and 5.2 seconds of system CPU time, despite about 100 seconds elapsed. Process workers averaged 57.2 seconds user plus 1.8 seconds system, with about 59.3 seconds elapsed. Much of the thread runtime was waiting.
• Parent profiling sampled rwsem_spin_on_owner under mprotect (2.98%). This is parent-kernel evidence, not a direct profile of the enclave kernel.
• Approximately 2.28 GiB of anonymous huge pages were present before hashing; failure to obtain transparent huge pages does not explain the result.
• Changing worker isolation substantially improves throughput inside the same enclave kernel. The host/enclave kernel versions differ, but this experiment does not isolate kernel version as the sole cause.

Smaller JIT permission range candidate

narrow-jit.patch limits full-memory VM permission changes to the main code region instead of also changing the unused superscalar code tail. Light-mode and cache JIT expand to the full region when they generate superscalar code. The destructor still makes the entire allocation writable and wipes it all. The candidate preserves secure JIT and does not change generated instructions.

The initial enclave screen achieved 8,178.4 hashes/s with 24 threads and 15,036.2 with 24 processes. Two longer process trials subsequently achieved 15,092.085 and 15,035.049 hashes/s (30,000 hashes per worker), averaging 15,063.567 hashes/s, or 627.649 per worker. That projects to 8.575 hours for 465M hashes. The final comparison also brackets these trials with the unchanged process implementation and tests the candidate with threads.

The bracketing baseline process trials averaged 12,125.890 hashes/s, so the smaller permission range adds 24.2% to process throughput. The candidate's longer threaded trial achieved 8,160.398 hashes/s (one trial). All 349 same-input final-digest comparisons passed across the benchmark records.

The native hardening fixture passed in the benchmark enclave for light and full modes. It also passed on the parent with AddressSanitizer and UndefinedBehaviorSanitizer: 16 secure-JIT/interpreter hash matches per mode, VM/scratchpad/entire JIT allocation erasure checks, and five injected permission failures per mode that aborted before returning a hash. Sanitizers do not instrument generated machine code; the differential hash tests cover its outputs. The first sanitizer launch lacked runtime libraries (exit 127); installing libasan and libubsan resolved this, with the failed log preserved.

Evidence and reproduction

summary.json contains individual results and digest comparisons. Run python3 summarize.py in this directory to recompute it from the raw logs. scaling.cpp is the instrumented harness; scaling-v1.cpp is the initial screening harness. thread-profile.txt contains the relevant parent profile. benchmark-reproduction-final.tar.gz preserves remote build/launch scripts, benchmark entrypoints, and native test commands/logs. benchmark-metadata.json records the machine, compiler, executable digests, and enclave launches.

Build against the frozen vendor source with:

cmake -S vendor/randomx -B randomx-build -DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=OFF
cmake --build randomx-build --target randomx -j 24
g++ -O3 -pthread -I vendor/randomx/src scaling.cpp randomx-build/librandomx.a -o scaling
./scaling thread 24 30000 0
./scaling process 24 30000 0

The last argument is the first CPU in the worker affinity range. The enclave exposes CPUs 0–23. The parent screening used CPUs 1–24 before reserving them for Nitro. Preserved launch scripts describe the minimal benchmark images.

Production integration boundary

The relay is multithreaded: directly forking its live runtime is not a safe implementation strategy. Shipping process workers requires an isolated worker helper or early supervisor, bounded IPC, failure handling, progress reporting, and an audit of secret lifetimes and erasure. It also needs a new reproducible enclave build and normal acceptance tests. The benchmark harness is not that production integration.

The improvement accelerates parallel generation. It does not let one solver parallelize a sequential puzzle chain or justify shortening its work count. The existing production generation remains intact while these experiments run.

The benchmark EC2 instance was verified stopped after the final trial. Its root disk vol-065aa2a90992a322a and all experiment files are retained.