Software Engineer, Production Engineering

NVIDIA's Production Engineering team is looking for highly motivated Platform engineers to lead a global, dynamic, state-of-the-art Service Reliability Operations center, to provide extraordinary levels of support for our Cloud products and services.  As a key member of the Production Engineering team, you will partner with other key members of our organization including Site Reliability Engineering, Security Operations Center, DevOps teams, and other partners to help make our services capable of providing near 100% availability. On occasion that an incident occurs, you will be our front line to decrease the frequency and duration of any issue.  Working in partnership with the development community you will develop monitors, alarms, and alerts to help make the service more reliable and improve our customer experience. 
 

What you will be doing:

• Being a member of the 24/7 Production engineering team to support Production Kubernetes Services, with a focus on automation and on reducing manual tasks. Flexibility to work on split-weekend shifts.

• Perform large scale K8s administration, systems administration, and security monitoring tasks to maintain service SLAs,integrity and reliability.

• Utilize alerts, alarms, and observability tools to proactively monitor, detect, prevent, and respond to incidents. 

• Apply deep systems knowledge to analyze logs, metrics, and system behavior to troubleshoot issues, lead root cause analysis, and implement effective resolutions.

• Initiate and lead incident management calls, ensuring timely detection, escalation, and resolution of issues by engaging subject matter experts and service owners as needed to resolve complex incidents efficiently. 

What we need to see:

• 7+ years of demonstrated experience administering large-scale production Kubernetes systems  in high-availability Internet, Cloud, or Data Center environments. Strong preference for on-prem expertise.

• BS in Computer Science, Engineering, Mathematics, or equivalent experience.

• Advanced hands-on experience with Kubernetes, SLURM, and large-scale cluster management.

• Familiarity with GPU / DPU hardware and high-performance computing Cluster environments.

• Strong Linux system administration, DNS, DHCP and core linux networking (IP Tables, routing, firewalls) experience with skills to troubleshoot and maintain Services on large-scale bare-metal infrastructure.

• Experience working with CI/CD tools like Jenkins, ArgoCD.

• Experience in scripting, Programming in Python or Golang or Rust preferred, but not required.

• Strong communication and soft skills, able to present to cross-functional group members in a persuasive manner.

Ways to stand out from the crowd:

• Experience architecting, building, and deploying K8s for large-scale environments, used by thousands of people.

• Passion for innovation and experience in groundbreaking high-performance Cluster technologies.

• Ability to learn new technologies quickly.