Staff Backend Engineer - Grafana Cloud Observability (Remote, Canada EST & AST)

This is a remote opportunity and we would be interested in applicants from the Canadian Eastern and Atlantic time zones only at this time.

What is Grafana Cloud?

Grafana Cloud is our composable observability platform that integrates metrics, logs, and traces with Grafana. It allows our customers to leverage the best open source observability software – including Prometheus, Mimir, Loki, and Tempo – without the overhead of installing, maintaining and scaling their own observability stack

The Observability department is focused on enabling developers to understand the health and performance of their applications and infrastructure in any environment by providing tools to instrument their code, ingest observability data into Grafana Cloud and visualize and explore it.

In this role, you will be part of the team that builds our Cloud Observability stack that allows customers to collect and visualize metrics from various systems and applications. We build and maintain the backend of opinionated apps such as Cloud Provider O11y, Database O11y, and K8s Monitoring. This includes the dashboards, alerts, documentation, and infrastructure while working closely with other teams to ensure seamless experiences. We also strive to incorporate OSS contributions in our work by continuing to augment projects like Alloy, Prometheus, Open Telemetry, and Beyla. The Observability department provides a core building block for customers using Grafana Cloud, and contributes significantly to our user value.

As a company we are remote-first and global, we embrace people of different experiences and backgrounds to build diverse teams where every person brings a unique perspective to the software. We are looking for Engineers that are passionate about communicating with data and providing seamless experiences for our customers to join our growing team! Engineers at Grafana also have the opportunity to contribute to Open Source communities.

Description of Duties

At Grafana Labs, ICs have a dedicated career path and do not have to become managers to progress in their career. Staff Software Engineers at Grafana have a large amount of experience across multiple areas. They are able to estimate, plan, coordinate and deliver large tasks spanning multiple systems. They actively coach and mentor other team members in their team and are able to identify and resolve issues with technology and product processes.

Key Responsibilities

You will bring your software engineering expertise to develop and maintain our portfolio of opinionated observability apps such as our Cloud Provider Observability solution, working to develop, maintain and scale our infrastructure observability features in Grafana Cloud.

• Design and implement high-quality, scalable integrations for various infrastructure components, databases, and applications
• Create middleware components, SDKs, and libraries that simplify development and maintenance of observability solutions
• When necessary, represent Grafana Labs in open source forums, working groups, and events
• Work with product teams to develop features that align with wider product strategy and customer needs
• Support the technical direction and vision of the team, contributing to strategic discussions and future development of observability solutions
• Work with other departments including GTM, Product, and Support teams to deliver a holistic product experience
• Be a part of your team’s follow-the-sun on-call rotations and take ownership of the services you’re running
• Embrace our open-source culture and contribute to other projects that may not directly fall within your team’s scope

As we are remote-first and our engineering organization is largely remote, we provide guidance and meet regularly using video calls, so an independent attitude and good communication skills are a must.

Requirements

Minimum Qualifications:

• 8+ years of experience in software development, with strong proficiency in Golang.
• Experience with cloud-based software development and production systems
• Experience writing clean, maintainable, robust, and performant software
• Familiarity with observability tooling (e.g., Grafana, Prometheus, Loki, OpenTelemetry)
• Experience with delivering projects from start to finish in a self-driven manner
• Experience in a hands-on technical leadership role, setting technical direction and leading project teams
• Experience with microservices architecture and distributed systems
• Excellent problem-solving and debugging skills
• Strong mentoring and leadership skills
• Experience participating in an on-call rotation and incident management

Preferred Qualifications

• Experience designing and building observability backends for various systems and applications
• Experience contributing to or maintaining open source projects, with evidence of successful pull requests and community collaboration
• Experience deploying and maintaining workloads running on  Kubernetes
• In-depth knowledge cloud platform internals
• Strong understanding of metrics collection, visualization, and alerting concepts

What you’ll bring to the role

At Grafana Labs, we define senior roles via the scope of their business impact more than via their expertise or their specific activities. So the most important thing we are looking for in an engineer is the ability to take ownership. You will need to drive technical implementations working with the upstream community and contribute to business conversations, helping shape the long-term strategy.

• You have a passion for observability and like to share your knowledge by writing documentation and blog posts.
• You love to engage with customers and help them out.
• You have excellent communication skills.
• You have relevant open source experience, ideally in the observability domain.
• You are willing to become an active member of the OpenTelemetry and Prometheus communities.
• You’re curious and you enjoy learning new programming languages and frameworks, setting up examples, and figuring out how things work.
• You have a good understanding of typical production environments. Ideally you have been responsible for operating production services and organizing on-call.
• You actively mentor other team members, identifying areas for focus and improvement.