autodidakt
Menu
← Research

Sandbox Infrastructure for Algorithm Discovery

We are building infrastructure to discover iterative Krylov subspace linear solvers that are accurate and efficient on production multi-node HPC clusters. A critical part of the algorithm discovery loop is deterministic evaluation of thousands of LLM rollouts containing solver code in C against the target solution.

Our objective is to achieve a lower total wall-clock time for solver setup and the linear solve. The time scale of the slowest solver is on the order of 100 ms, which we want to improve by at least a factor of 2×. Therefore, the verification and evaluation infrastructure has to be sensitive enough to detect a sub-millisecond time difference between rollouts. Current state-of-the-art sandbox providers did not meet our sensitivity and determinism requirements, hence we built our own custom single-host, multi-tenant sandbox infrastructure on bare-metal AWS EC2 instances.

The Problem

We started with Modal sandboxes. However, we quickly ran into the following issues:

CPU hardware variance

For the algorithm discovery task, we simulate a quarter-scale version of a real multi-node HPC cluster with 64 nodes, each with two AMD EPYC 9354 processors (Zen 4, “Genoa”), 64 cores, and one rank per core. More information is available here: Communication-efficient Linear Solver Discovery.

We calibrate our score function such that the starting reference solver is at the base 1.0× score level.

Empirically, we found that Modal provisions CPU sandboxes on different hardware platforms and can land on AMD Genoa (Zen 4), AMD Turin (Zen 5), AMD Milan (Zen 3), or even Intel Ice Lake in some cases. This throws off our calibration, as compute is 1.58× Genoa’s on Milan and 0.88× on Turin, and absolute simulated times varied 10.7–13.8% across sandboxes. Even across Genoa-only sandboxes, host speed ranged from 2.79 to 3.54 GFlops, and single-core memory bandwidth from 21.6 to 43.8 GB/s (CV 19.5%). This points to a severe case of noisy neighbors due to their high-availability service at scale.

Runtime CPU clock tick measurement

With Modal sandboxes’ default runtime, gVisor, we noticed that CLOCK_THREAD_CPUTIME_ID only advances in 10 ms increments, and even just reading the clock costs ~643 ns. This is because instead of directly measuring a thread’s CPU time, gVisor has a background ticker of 10 ms length and increments the tick counter for each thread that is running. Reading the clock is a simple system call into gVisor’s kernel that returns that counter.

Seventeen identical Modal sandbox requests land on four CPU types: seven Genoa, six Turin, two Milan, and two Ice Lake. Calibrated compute and paired solver speedups vary, even within Genoa sandboxes.
Fig. 1. The sandbox lottery: identical requests land on different CPUs, with timing variation even on the same CPU family.

Our Sandboxing Solution

The desiderata for a proper sandbox service, therefore, are:

  • The same hardware every time
  • No virtualization overhead
  • No timing noise
  • Isolation from noisy neighbors

A 4th-generation EPYC package is built from chiplets: up to eight Zen 4 cores form a core complex with a shared 32 MB L3 cache, fabricated as one die termed a Core Complex Die (CCD) [, ]. Each core in a CCD also has a private L2 cache. Up to twelve CCDs surround a central I/O die, which holds the twelve DDR5 memory controllers. The NUMA-per-socket (NPS) setting chooses how memory is presented, with NPS1 making each socket one NUMA node.

c7a.metal-48xl is a whole physical server with no hypervisor, two AMD EPYC 9R14 processors (Genoa, 96 cores each), 192 cores in all, and 384 GiB of memory [, , ]. AWS turns off simultaneous multithreading on this family, so each of the 192 virtual CPUs is a full physical core [, ].

Normally, each of those cores would be assigned to a sandbox through a container (i.e., eight sandboxes for each CCD), which is almost a necessity for high-volume, high-availability services like Modal. This multi-tenant architecture puts pressure on the shared L3 cache of a CCD and contributes to the unpredictable timing behavior described earlier.

Instead, we assign a single sandbox to each CCD, i.e., one container per eight cores. The container process then runs on the first core of the CCD with exclusive access to the L3 cache. This architecture seems wasteful, but for a batch size of N = 1 with a use case that is highly sensitive to noisy neighbors, this is necessary.

Shared sandbox hosting compared with our two-socket EPYC host. CCD 0 handles housekeeping; each remaining CCD contains one sandbox with its own cores and L3 cache. Memory traffic still shares the socket’s I/O die.
Fig. 2. One sandbox per CCD isolates its cores and L3 cache from other sandbox workloads.

Along with other important fixes, we were able to bring down the timing noise substantially, as shown below.

For 44 measurements in each environment, Modal gVisor solve times range from 116 to 293 ms with a coefficient of variation of 19.98%. Our AWS metal sandbox times range from 151 to 157 ms with a coefficient of variation of 0.70%.
Fig. 3. Timing noise for the same native solve across 44 measurements per environment.

References

  1. AMD, “4th Gen AMD EPYC Processor Architecture” (white paper).
  2. AMD, “HPC Tuning Guide for AMD EPYC 9004 Series Processors” (58002).
  3. AMD, “Amazon EC2 Instances Powered by AMD EPYC Server CPUs — Tuning Guide” (63841).
  4. AWS, “Amazon EC2 C7a Instances”.
  5. AWS, “Specifications for Amazon EC2 compute optimized instances”.
  6. AWS, “Instances built on the AWS Nitro System”.