Native RandomX source supplement — GPT-5.6 Sol

Date: 2026-09-09
Reviewed source: deployed commit a10323dede4413fbf295916b8ad12e3dbad7514e
Relation to the main review: this is a bounded follow-up. It does not modify or replace codex__gpt-5.6-sol.md.

Scope and result

I followed the Rust FFI into the vendored RandomX C++ implementation, including allocation, dataset/cache initialization, per-worker VMs, AArch64 JIT selection, v2 dispatch, diagnostics, and teardown. The reviewed native and Rust files have no diff from the named deployed commit.

The supplement does not change the main review's objective verdict: no implementation-level RandomX shortcut or native disclosure path was found, but early-disclosure resistance remains conditional on Nitro isolation, correct native execution, WebPKI authentication, and the timing calibration. The literal objective still cannot cover a malicious destination voluntarily giving the plaintext it necessarily receives to a colluding operator.

Reachable allocation, ownership, and concurrency

Dataset::new obtains detected CPU flags, allocates and initializes the cache, allocates and fully initializes the roughly 2.08 GiB dataset in full mode, and only then returns (crates/timelock/src/randomx.rs:49-70). The seven scoped workers are started afterward; each constructs its own VM and hashes its own secret chain (crates/timelock/src/lib.rs:326-351). Vm<'a> carries a borrow of Dataset, and std::thread::scope waits for all scoped threads, including unjoined handles during early error unwinding, before the dataset can drop. Dataset::drop releases the dataset before the cache (randomx.rs:84-91). I found no use-after-free in this composition.

The manually asserted Sync property is justified for this particular full-memory path after initialization. randomx_init_dataset writes the shared dataset before workers exist (vendor/randomx/src/randomx.cpp:192-212). Each call to randomx_create_vm allocates a distinct VM and scratchpad; it copies cacheKey, points the full VM at the shared dataset, and calls allocate (randomx.cpp:225-357). The full compiled VM reads datasetPtr->memory during execution (vendor/randomx/src/vm_compiled.cpp:45-70). Its program, register file, configuration, memory-register view, scratchpad pointer, flags, key copy, temporary hash, and JIT compiler are per-VM (vendor/randomx/src/virtual_machine.hpp:35-100). No worker mutates the dataset or cache after sharing in the reviewed path.

Allocation failure handling is bounded but mostly availability-oriented. Cache, dataset, VM, scratchpad, and JIT allocation exceptions are converted to null at the C API boundary and then fixed Rust errors where applicable (randomx.cpp:69-128,152-183,225-357; randomx.rs:52-76). The void cache/dataset initialization APIs rely on native assertions and do not give Rust a success result. With the fixed valid sizes and checked allocations I did not identify a host-controlled confidentiality trigger through that gap, but exceptional native behavior is not comprehensively fail-safe.

The target FFI types match this Linux/AArch64 build: flags are passed as c_int, size_t as usize, dataset item counts as c_ulong, and native objects as opaque pointers. Rust supplies a live input slice and exact 32-byte output buffer for the synchronous call. The build script hashes all listed vendored files, including the native files discussed here, before producing a static Release library (crates/timelock/build.rs:4-25; vendor/randomx/SHA256SUMS). The pinned source tree and measured image remain part of the attestation requirement.

Concrete reachable native bug: concurrent hard-AES self-test data race

Finding N1 — C++ data race/undefined behavior during production VM construction; no demonstrated disclosure impact.

VmBase::allocate performs a hardware-AES instruction probe by loading, transforming, and storing a process-global object:

alignas(16) volatile static rx_vec_i128 aesDummy;
// ...
rx_vec_i128 tmp = rx_load_vec_i128((const rx_vec_i128*)&aesDummy);
tmp = rx_aesenc_vec_i128(tmp, tmp);
rx_store_vec_i128((rx_vec_i128*)&aesDummy, tmp);

This is at vendor/randomx/src/virtual_machine.cpp:98-116. Production AArch64 is compiled with crypto instructions (vendor/randomx/CMakeLists.txt:152-173), randomx_get_flags selects hard AES when the CPU reports it (randomx.cpp:49-66), and the application creates seven VMs concurrently (lib.rs:327-350). The loads and stores are non-atomic; volatile does not make concurrent access valid in the C++ memory model. Therefore this execution has a real data race and undefined behavior without any attacker-controlled input.

The dummy value is unrelated to a seed or epoch key and the race occurs before the worker's first secret hash. I found no path from it to plaintext/key disclosure or faster puzzle solving. Its demonstrated impact is loss of a well-defined native execution model, with plausible crash/availability risk; broader consequences would require an exploit argument that this review does not have.

Use std::once_flag/std::call_once for the instruction probe, or create VMs serially before launching hashing workers. A focused regression test should create seven hard-AES VMs concurrently under ThreadSanitizer and verify the current code reports the race and the corrected code does not. A stress test on the production architecture can supplement this, but lack of observed crashes would not resolve the C++ data race.

V2 and secure-JIT flag propagation

Rust ORs numeric 128 and 16, then 4 in full mode (randomx.rs:56-68). In the pinned native header these are exactly RANDOMX_FLAG_V2, RANDOMX_FLAG_SECURE, and RANDOMX_FLAG_FULL_MEM (vendor/randomx/src/randomx.h:42-53). The numeric literals are brittle for a future vendor update, but the vendored hash check makes them correct for this image.

randomx_create_vm masks only the class-selection bits for its switch and passes the complete flag value into the selected VM constructor. SECURE selects the secure compiled class (randomx.cpp:225-350). The VM stores all flags, the compiled VM passes all flags to its private compiler, and secure JIT switches the mapping to writable only for generation and executable for use (virtual_machine.hpp:60-84,89-99; vm_compiled.cpp:37-60). On AArch64, the compiler tests RANDOMX_FLAG_V2 to select the v2 FE-mix and prefetch code (vendor/randomx/src/jit_compiler_a64.cpp:173-210). Program size also tests v2 (vendor/randomx/src/program.hpp:53-58). The interpreted fallback passes the flags into bytecode compilation and has explicit v2 dataset-index and FE-mix behavior (vendor/randomx/src/vm_interpreted.cpp:50-121).

This source tracing supports v2 execution rather than merely v2 labeling. The coordinating reviewer additionally reported that full_mode_upstream_v2_vector passed explicitly after the main review. I did not rerun it for this supplement.

One hardening gap remains in secure JIT error handling: setPagesRW and setPagesRX discard mprotect/VirtualProtect return values (vendor/randomx/src/virtual_memory.c:159-204). In the reviewed secure path, a failed transition normally leaves an RW or RX mapping and should fault at the following write or execute, so I did not derive an RWX disclosure path. Still, permission-transition failure is not explicitly fail-closed or reported.

Host input and diagnostic/output paths

The native hash input is DOMAIN || epoch || segment || iteration || x (crates/timelock/src/lib.rs:238-244). The dataset key, seven seeds, and epoch key come from enclave entropy; epoch, segment count, and production iterations are constrained inside the measured application. Request method, URI, headers, body, response, parent tunnel bytes, DNS responses, and solver submissions never enter the RandomX FFI. The parent can deny CPU/memory/network resources, kill the enclave, or prevent publication, which affects availability, but I found no interface by which it selects RandomX flags, pointers, dataset key, seeds, or chain inputs in an accepted measured enclave.

The production CMake does not define TRACE; randomx::trace is compile-time false unless that definition is supplied (vendor/randomx/src/common.hpp:106-110). Native allocation catches do not print exception text. At the application boundary, v2 diagnostics enqueue only bounded printable &'static str messages (crates/enclave/src/v2_diagnostics.rs:1-40), and the top-level error replaces nested external or request-derived errors with a fixed message (crates/enclave/src/v2_main.rs:117-131). I found no reachable diagnostic, native logging, or C API output that contains a secret seed, chain value, dataset key before publication, or epoch key.

This is an absence-of-path finding for the reviewed sources, not proof against arbitrary native corruption or side channels.

Secret erasure and native side-channel limits

Native destruction frees memory but does not explicitly clear it:

• VmBase::~VmBase frees the scratchpad without wiping it (virtual_machine.cpp:100-103). The VM object also contains secret-dependent registers, program/configuration, tempHash, and a copy of the dataset key (virtual_machine.hpp:69-84).
• randomx_calculate_hash leaves its local 64-byte tempHash uncleared (randomx.cpp:392-403). Registers and compiler temporaries are likewise outside a guaranteed erasure discipline.
• The per-VM AArch64 JIT mapping contains code derived from the secret-dependent RandomX program and is unmapped without wiping (jit_compiler_a64.cpp:92-125).
• Cache and dataset deallocators call the allocator directly, and the default aligned allocator ignores the size and frees without a wipe (vendor/randomx/src/dataset.hpp:80-84; vendor/randomx/src/dataset.cpp:60-69; vendor/randomx/src/allocator.cpp:37-60). Cache construction uses ARGON2_DEFAULT_FLAGS, not a clear-memory option (dataset.cpp:71-140).
• Page-backed allocations are unmapped without a preceding explicit clear (virtual_memory.c:234-242). OS zero-fill on later mapping may protect cross-process reuse, but that does not prove erasure before unmap or prevent same-process heap reuse for ordinary allocations.

The dataset key becomes public with the manifest, while scratchpad/register/JIT state is derived from unpublished segment-chain values and can include late chain state. Under the stated trusted Nitro boundary, the operator cannot directly inspect enclave RAM, and I found no service path that returns freed or uninitialized native storage. Thus this is insufficient evidence for robust erasure and resilience to a future memory-disclosure flaw, rather than a demonstrated current leak.

RandomX's primitive strength does not establish that its JIT, cache access, allocation, or CPU behavior is side-channel-free. This bounded source review did not prove resistance to cache/timing/speculation leakage, fault injection, compiler miscompilation, or exploitation of other native undefined behavior. Exploiting such effects from the parent while preserving an accepted attestation was not demonstrated here. Confidentiality therefore remains conditional on the stated Nitro/hardware isolation assumptions and on native-code safety.

Composition trust boundary: WebPKI

Outbound TLS authenticates the requested upstream name with the webpki_roots trust store (crates/enclave/src/net.rs:56-69,110-151). Trusting AWS does not itself imply that every WebPKI CA will issue certificates honestly. The threat model explicitly assumes WebPKI authentication; that is an additional condition for content confidentiality. A compromised or misissuing trusted CA could let an intermediary impersonate an upstream, receive the request plaintext, and disclose it immediately. That would be an authentication-boundary failure, not a RandomX shortcut. Separately, the genuine destination necessarily learns both request and response content and can voluntarily share them under the stated unrestricted collusion model.

The coordinating reviewer reported 18 verifier plus delayed-ACK integration tests passing after the main review. Those tests strengthen the attestation/publication state-machine evidence but do not exercise the native race, secret erasure, CA issuance, or native side channels. No tests or production calls were run for this source-only supplement.