Lead Test Engineer (Azure | Databricks)
Contract | Remote | UK Public Sector Project
Overview
We are seeking a Lead Test Engineer to work within a multi-disciplinary team to own the test strategy and quality engineering approach for an Azure Databricks Platform. This role involves shaping testing practices across the platform, championing data quality tooling, and building the foundations for reliable, well-tested data engineering across the programme. The Lead Test Engineer will play a key role in designing and implementing a structured test framework, embedding automated testing into the delivery pipeline, and collaborate with test and data engineers.
Who We’re Looking For
An experienced, hands-on Lead Test Engineer who is equally comfortable defining a test strategy at platform level and writing the tests themselves. You will assess the current state of testing across the data platform estate, identify gaps and risks, and design a structured, maintainable test framework that engineers can adopt consistently. You should be confident introducing test-driven development practices, building CI/CD pipelines that run tests automatically, and championing data quality tooling as part of the platform's testing approach. You will also set standards and raise the overall quality bar for the platform.
Key Responsibilities
- Define and own the test strategy for the Azure Databricks data platform.
- Design and implement a structured test framework to replace ad-hoc pytest usage, establishing consistent patterns, standards, and reusable components for the team.
- Build CI/CD pipelines (Azure DevOps) that run automated tests and surface data and pipeline issues before they reach production.
- Introduce and embed test-driven development (TDD) practices across the engineering team.
- Define and implement data quality checks and validation rules across ingestion and transformation stages, using tools such as Soda Data Quality or DQX.
- Lead, mentor and set direction for test engineers, reviewing their work and raising testing standards across the platform.
- Collaborate with data engineers, architects and platform leads to embed testing early in the development lifecycle.
- Produce clear documentation of the test strategy, framework, and standards for ongoing use by the wider engineering team.
- Track and report on test coverage, defect trends, and platform quality metrics to programme and technical leads.
Mandatory Requirements
- Proven experience defining and delivering a test strategy for a data platform, not just executing pre-defined test cases.
- Hands-on experience building test frameworks for Databricks, including writing and structuring pytest-based tests.
- Practical experience with test-driven development (TDD) and embedding it within an engineering team.
- Experience building or configuring CI/CD pipelines (Azure DevOps) to run automated tests as part of the delivery pipeline.
- Experience leading or mentoring test engineers, with the ability to set standards and raise quality practices across a team.
- Strong understanding of data pipeline testing challenges, including data quality, schema validation, and transformation correctness.
- Ensure documentation is reviewed and approved by central data platform team per acceptance criteria.
- Participate in communities of practice, ceremonies, and approval forums as required.
Desirable
- Prior experience working within UK Public Sector environments or with Government Digital Standards.
Soft Skills
- Strong stakeholder communication and cross-team collaboration.
- Analytical and solution-focused mindset.
- Able to work independently, take ownership and drive progress.
- Commitment to clean, scalable, well-documented engineering practices.
- Adaptable, proactive, and comfortable working in dynamic delivery environments.
Contract Details
- Contract Type: Freelance / Contract
- Location: Remote (UK-based candidates preferred)
- Start Date: ASAP
- Clearance: Candidates must be eligible to work with UK Government departments / BPSS.
Compensation: Competitive Salary
