Sr. Engineer - Scale & Performance Engineering (Hybrid)

The Role

As a Senior Engineer for the Scale and Performance Engineering team, you will play a critical role in enhancing the scalability, performance, and reliability of HashiCorp's cloud products. With at least 6 years of experience in site reliability engineering or a related field, you will lead efforts to identify performance bottlenecks, address, and mitigate operational challenges before they impact our customers. Your expertise in load testing, performance analysis, and system hardening will ensure that our services meet the highest standards of scale and performance excellence.

Having a holistic view of enterprise and cloud systems, you will play a pivotal role in enhancing our Scale resilience and maintaining the reliability and performance of our enterprise and cloud-based products. With a focus on overall Quality you will be at the forefront of ensuring high availability and performance across HashiCorp’s offerings.

You will provide expert execution of the test plans, defining system wide strategies for product load and performance testing. You will be working on a wide variety of tools and exploring new avenues to ensure all the products meet the essential scale and performance criteria. 

Utilize top-notch troubleshooting techniques like simulating the system with Chaos to identify, organize, and advocate for novel solutions to remediate customer impact on complex interconnected systems. 

Key Responsibilities

• Implement best practices for system reliability, including proactive identification of potential failure points and the development of automated mitigations
• Design and execute comprehensive performance testing strategies to identify performance bottlenecks and scalability limits across our cloud products
• Work with the engineering teams to identify potential application and infrastructure bottlenecks and suggest changes.
• Work closely with engineering and product teams to integrate scale and performance readiness into the development lifecycle, enhancing product stability and user satisfaction.
• Build and refine tools and frameworks for automated testing, environment simulation, and incident reproduction, reducing manual effort and increasing test coverage.
• Conduct in-depth analysis of testing results, documenting findings and making actionable recommendations for system enhancements.
• Drive Systemic Improvements to the products by introducing Chaos Testing and partnering with product development teams. 
• Share your knowledge and expertise with team members, fostering a culture of learning and continuous improvement.
• Develop and implement disaster recovery and backup strategies to ensure data integrity and system resilience.

Ideal Candidate

• 6+ years of experience in performance engineering  systems engineering, reliability engineering or non functional testing roles with a focus on performance testing, load testing or system scalability.
• Strong programming skills in Python / Golang and exposure to scripting languages like javascript or shell script 
• Experience with version control systems such as Git. 
• Strong experience with performance testing tools like K6, Artillery, Vegeta, Locust etc or similar tools for deriving key performance metrics for a product
• Proven track record of leading successful performance testing and optimization initiatives in cloud and on-prem environments.
• Experience in creating and managing test environments for automated testing.
• Experience in creating CI/CD pipelines and maintaining quality gates for system testing.
• Understanding of monitoring and observability tools such as Datadog or Prometheus to develop dashboards indicating metrics that accurately reflect system performance and load break points and regressions.
• Exposure to cloud technologies ( AWS, Azure, Or GCP) and container technologies like Nomad or Kubernetes and/Or working in a Hybrid cloud environment.
• Effective communication and collaboration skills, capable of working with cross-functional teams and articulating technical concepts to diverse audiences.

Nice to Have:

• Experience with infrastructure as code ( Terraform ) or any HashiCorp products is a plus.
• Experience with Javascript development / using any test framework based on Java script is a plus.
• Experience in driving systemic improvements through Chaos engineering is a plus. #LI-Hybrid