Senior Data Engineer

This is a remote position.

About the Engagement

We are looking for experienced Senior Data Engineers to join a large-scale, multi-year Data Mapping and DataOps Platform modernisation programme for a major government ministry.

This strategic initiative focuses on transforming how data is managed, governed, and leveraged as a core enterprise asset.

Key Focus Areas

• Data Engineering: Designing, building, and optimising scalable data pipelines to enhance stability, traceability, and refresh frequency
  
• Data Mapping & Metadata Capture: Onboarding critical financial systems into a centralised enterprise data catalogue and supporting governance initiatives
  

Work will be delivered in an Agile/Scrum model, collaborating closely with data product managers and cross-functional teams.

Platform Overview

The platform is built on Microsoft Azure and Microsoft Fabric, with an existing in-house Finance Data Catalogue (FDC).

This role will contribute to scaling the platform into a production-grade DataOps ecosystem.

Key Responsibilities

• Design, build, optimise, and maintain scalable data pipelines
  
• Develop and manage ELT pipelines, orchestration, and automation using Python
  
• Capture and onboard metadata into enterprise data catalogues
  
• Perform data analysis to identify patterns, anomalies, and quality improvements
  
• Contribute to data product design, including data models and structures
  
• Design and implement data models for analytics and reporting
  
• Build and maintain CI/CD pipelines for automated deployment
  
• Develop automation scripts using Python, Bash, and PowerShell
  
• Develop and maintain RESTful APIs for integration and interoperability
  
• Implement data quality checks and validation frameworks
  
• Set up monitoring, alerting, and performance tracking
  
• Develop dashboards and reporting solutions using Microsoft Fabric & Power BI
  
• Collaborate with stakeholders to translate business needs into technical solutions
  
• Ensure compliance with data governance, privacy, and security standards
  
• Maintain technical documentation and communicate with both technical and business teams
  

Mandatory Requirements

Candidates must have recent (last 48 months) hands-on production experience in:

• Python (data pipelines, scripting, automation)
  
• SQL (data transformation, modelling)
  
• Microsoft Azure & Microsoft Fabric
  
• ELT Pipeline Development
  
• Git (version control & collaboration)
  
• Metadata Management & Data Catalogue onboarding
  
• Data Product Design
  
• Agile/Scrum methodologies
  
• Data Analysis & Data Quality Assessment
  

Technical Skills Required

Cloud & Platform

• Microsoft Azure
  
• Microsoft Fabric
  

Data Engineering

• Python, SQL
  
• Bash, PowerShell
  

Pipelines & DevOps

• ELT pipelines
  
• CI/CD pipelines
  

Data Modelling

• Schema design
  
• Metadata management
  
• Data product design
  

API Development

• RESTful APIs
  

Analytics & Reporting

• Power BI
  
• Microsoft Fabric Reporting
  

Version Control

• Git, GitHub
  

Methodology

• Agile / Scrum
  

 Nice to Have

• Advanced Power BI (DAX, dashboarding, modelling)
  
• Data visualization beyond standard reporting
  
• Experience with REST API integrations
  
• Exposure to:
  
• Azure Synapse Analytics
  
• Azure Data Lake
  
• Azure DevOps
  
• Experience in government/public sector or regulated environments
  
• Familiarity with data governance frameworks
  
• Experience in financial systems modernisation programs