Job description
Position overview:
We are seeking a Site Reliability Engineer to join our Operations Group. This role plays a key part in advancing scientific discovery by supporting high-performance computing (HPC) and data analysis for the organization.
Our center provides essential HPC and data systems to more than 10,000 researchers working in areas such as alternative energy, climate science, energy efficiency, environmental science, and other missions.
As a Site Reliability Engineer, you will be part of a 24⁄7 operations team that ensures our systems are accessible, reliable, secure, and available to the scientific community. You will work with a state-of-the-art data collection and monitoring system to maintain and optimize performance across complex HPC and data environments.
What You Will Do at Level 2
- Work five shifts per week monitoring a large HPC facility, including 2–3 overnight shifts (midnight–8 a.m.) per week.
- Split time between on-site and off-site shifts depending on staffing needs.
- Review and respond to alerts from computing systems, storage, networks, and other data center/facility systems by triaging or escalating to on-call staff.
- Develop solutions to improve processes, prevent recurrence of issues, and automate responses to routine service conditions.
- Identify areas for improved monitoring and automation; propose and implement solutions.
- Respond to monitoring alerts to ensure continuous 24⁄7 data collection for real-time diagnoses.
- Develop and maintain tools within the monitoring pipeline in collaboration with the Operations Team.
- Create software programs to provide alerts and notifications from HPC system APIs into the monitoring pipeline.
- Configure software and solve technical issues to ensure programs scale reliably with increasing data volume.
- Collaborate with other groups to ensure workflows are understood and maintained.
- Assign technical tasks to other monitoring team members as needed.
- Coordinate system maintenance activities and manage diagnostic and notification software during outages.
- Provide accurate documentation in ticketing systems for outages, updates, and incidents.
- Work on and resolve problems of diverse scope where analysis requires evaluation of identifiable factors.
Additional Responsibilities (For Level 3 only)
- Provide leadership in developing monitoring and alerting pipelines, documentation, and software.
- Contribute to the design and deployment of the monitoring cluster.
- Partner with other technical groups to improve monitoring experiences.
- Tackle complex problems requiring in-depth evaluation of variable factors.
- Determine methods and procedures on new assignments and may coordinate activities of other team members.
Required Qualifications (Level 2)
- Typically requires 5+ years of related experience with a Bachelor’s degree, or 3+ years with a Master’s degree, or equivalent work experience.
- Strong hands-on knowledge of Linux shell and command-line environments.
- Experience developing tools using languages such as C, C++, Perl, Java, or Python.
- Knowledge of IT infrastructure and large data communication networks supporting highly available systems.
- Ability to learn and work with data center management technologies (e.g., Kubernetes, Prometheus, alerting/monitoring tools, building management software, cooling/power systems).
- Strong communication skills and ability to collaborate across multiple technical teams.
- Experience working in a 24⁄7 operations team managing large data centers or installations.
- Knowledge of network security, ACLs, firewalls, and protocols.
- Relevant certifications in system administration or related areas.
Required Qualifications (Level 3)
- Typically requires 8+ years of related experience with a Bachelor’s degree, or 6+ years with a Master’s degree, or equivalent.
- Advanced expertise in one or more programming languages such as C, C++, Perl, Java, or Python.
- Demonstrated excellence with monitoring and automation tools.
- Experience leading technical projects.
- Strong ability to respond proactively to complex issues.
Additional Details
- Shift: Includes overnight “Owl” shifts (12 a.m. – 8 a.m.), primarily on-site.
- This is a full-time, exempt position (monthly paid).
- A background check is required. Convictions are reviewed in relation to job responsibilities and do not automatically disqualify applicants.
- This position requires substantial on-site presence, but hybrid schedules may be available depending on business needs. Candidates must reside within 150 miles of the work site.
