Senior Systems Engineer, Test Frameworks & Validation Platform

 

What You'll Do:

The Systems Engineering team owns the host software stack that turns a freshly provisioned bare-metal machine into a healthy Kubernetes worker — the OS image, kernel and drivers, and the hardware and firmware beneath them. We build and operate the test framework that qualifies every server image on real physical hardware before it reaches one of the largest GPU fleets in the world, and we're extending that framework into HPC verification, Slurm-on-Kubernetes, and further down the stack.

About the role:

As a Senior Software Engineer on the Systems Engineering team, you will own and evolve our test framework — in-house, Kubernetes-native system that qualifies host images and infrastructure on real hardware. You'll harden the framework's core, broaden what it can validate, and keep CI fast and trustworthy at fleet scale, where a 0.1% failure rate is thousands of GPUs. Day-to-day, you'll build test harnesses and CI infrastructure, extend coverage into HPC and Slurm-on-Kubernetes (SUNK), and shape the abstractions other engineers use to add their own tests without reinventing the framework underneath them.

Some of what you'll work on:

• Own and extend the test framework — that pins to a real host, boots the configuration under test, runs a containerized check, validates it, and records a structured result.
• Broaden what the framework qualifies: HPC and fabric verification, Slurm-on-Kubernetes (SUNK), and further down the stack into firmware and hardware, building the abstractions that make new coverage easy to add.
• Keep the pipeline fast, hermetic, and trustworthy — boot consistency, per-host resource locking, and ruthless flaky-test elimination — so engineers trust the signal.
• Own the results and reporting path end to end: structured test results, storage, dashboards, and the triage surfaces engineers use to turn a red run into a root cause quickly.
• Explore AI-native testing — LLM-driven log triage, failure classification, and regression detection — as the fleet and test surface grow.
• Collaborate with firmware, kernel, imaging, and HPC teams to embed testing into their release process.

Who You Are:

• 3+ years of experience building test infrastructure, systems software, or platform tooling at scale, with real ownership of frameworks or automation for low-level software.
• Fluent in Python, with either proven Rust experience or a strong systems background (Go, C, C++) and a genuine appetite to build in Rust — the framework's core is Rust, and you'll live in it.
• Comfortable operating in a Kubernetes environment and reasoning about how software is built, containerized, deployed, and tested.
• Solid Linux systems background — the boot chain, kernel and drivers, low-level debugging — and comfort working close to the hardware.
• Real testing discipline: you've built automation that proves systems work, and you have strong opinions about flakiness, hermeticity, and signal.
• Clear communicator who treats the test framework as a product other engineers want to use, not a chore they route around.

Preferred:

• Rust and Kubernetes-native workflow orchestration (e.g., Argo Workflows) experience.
• HPC or large-cluster experience — InfiniBand/RoCE, GPU/accelerator validation, or performance-regression frameworks.
• Slurm or Slurm-on-Kubernetes (SUNK) experience.
• Firmware or lower-level hardware validation experience.
• Applying LLMs to test workflows — triage, failure classification, flaky-test detection.
• Contributions to open-source test frameworks, Rust crates, or systems projects.

The base salary range for this role is $153,000 to $204,000. The starting salary will be determined based on job-related knowledge, skills, experience, and market location. We strive for both market alignment and internal equity when determining compensation. In addition to base salary, our total rewards package includes a discretionary bonus, equity awards, and a comprehensive benefits program (all based on eligibility).