Senior Platform Engineer (Infrastructure)

Hybrid - 2 days per week in office (London Bridge/Tower Bridge area)

The RVU London cloud infrastructure team

We are committed to Open Source software in order to build services that help millions of customers to save money and make confident decisions. As well as helping our customers, we also give back to the community by open sourcing interesting projects that we build that might benefit others. https://www.rvu.co.uk/open-source

We’re looking for an experienced Platform/Infrastructure Engineer to join our infrastructure platform team, known internally as ‘Airship’.

Our goal as a team is to enable our development teams to deliver services quickly, reliably and securely. We do this by running multiple Kubernetes EKS and Fargate clusters in AWS, creating common tooling to aid in development tasks and running shared services such as Opensearch, Envoy, Vault and Prometheus to name a few. The team has also recently expanded its scope to simplify Data engineering in the organisation using the same techniques we used to ease creating web applications on data pipelines, leveraging Argo Workflows and Argo Events  as well as completed a migration to Github Actions.

Day to day tasks will include:

• Planning and working on our infrastructure platform: from maintenance to design systems improvements or to adopt new technologies
• Working with product engineering and data teams to design, build and improve scalability and reliability of their systems with an emphasis to provide the best DevEx
• Developing tooling to help our teams work more efficiently

Requirements

The ideal candidate will have some of the following skills:

• Extensive experience  in running Kubernetes clusters in production
• Knowledge of Golang, Helm and Terraform  (some knowledge of  Python is definitely a plus)
• Production experience in Cilium and/or eBPF and networking in general
• Extensive experience in monitoring systems and their performance
• The ability to debug large and complex systems and solving large problems that affect a wide user base in a simple way
• Experience with image vulnerability scanning and patching strategies for large systems
• Experience / Familiarity with AWS Multi Accounts system designs tools like  Crossplane and Control Tower 
• Familiarity with Argo Workflows or similar data pipeline as a service tools
• Familiarity working with a variety of Cloud Native projects
• Familiarity with Github Action 
• Familiarity with OpenTelemetry

Out team has been featured in a few conferences:

CNCF:  https://www.youtube.com/watch?v=-v9tiGTH86Q 

PlatformCon:  https://www.youtube.com/watch?v=8YNXTI8E13s  and

https://platformcon.com/talks/from-cloud-cost-management-to-finops 

We have also been featured  in the London AWS Summit 2023 for contribution to the EKS tooling community https://surajincloud.com/announcing-kubectl-eks-plugin-v0-1-0 

We also hosted and held the Terraform Hashicorp  User Group meetup in London in April. 

Examples of some projects we have worked on:

Short lived database credentials

Our running services previously relied on having long lived credentials to access data that were rarely, if ever, rotated. We wanted human and pod identity to be used to grant short-lived credentials based on policies. We used Vault to build a solution to this problem, creating tooling such as vault-creds/vault-webhook to make it as easy as possible for developers to use these credentials with their services. (Blog)

Kiam: a service that integrates AWS IAM with Kubernetes

We have a lot of existing AWS resource that have their access limited using IAM. We used Kube2IAM initially but experienced race conditions that would hand different role credentials to pods. We started work on a replacement and have worked with the community to get it used in other places.

Yggdrasil: Envoy control plane for multi-cluster load balancing

For some of our more important applications it was important to have them survive a total cluster outage. This meant we needed a way to easily route traffic to an application spread out across multiple clusters so we created Yggdrasil, a tool to configure Envoy nodes to route our traffic between clusters based on Ingress resources. (Blog)

Hermod: more confidence in the status of your deployments

It tracks deployments as they roll out and posts useful status updates into Slack. It does this by watching the Kubernetes api for namespaces and deployments with the correct annotations. When a new deployment rollout begins and completes updates are posted to the Slack API. Any errors during the deployment rollout are captured and included in the Slack message (see example below). This can be very useful to help quickly debug a failing deployment.

You can also check out our medium page to see a number of blogs on what we’ve been up to.

Our commitment to you

At RVU, we are dedicated to developing valuable, inclusive, and user-friendly products and services for all. To achieve this it’s essential that our teams reflect the diverse range of people in our community. We believe in being the change we wish to see in the world, by embracing our differences and holding ourselves accountable to being open and inclusive teammates and wider community members.

Benefits

What we’ll give back to you:

We want to give you a great work environment; contribute back to both your personal and professional development; and give you great benefits to make your time at RVU even more enjoyable. Some of these benefits include:

• Employer matching pension up to 7.5%
• Hybrid approach of in-office and remote working, and a “Work from Home” budget to help contribute towards a great work environment at home
• Excellent maternity, paternity and adoption leave policy, for those key moments in your life
• 25 days holiday (increasing to 30 days) + 2 days “My Time” per year
• Up to 30 days per year “working from anywhere”
• A healthy learning and training budget, as well as the chance to go to conferences around the world every year
• Electric vehicles scheme
• In office gym
• Free breakfast in the office daily
• Health insurance
• Access to the Calm and Peppy app for physical and mental health
• Regular events - from team socials to company-wide events with insightful external speakers, we want to make sure our colleagues continue to feel connected