Match score not available

Data Engineer | 2024TA1106

Remote:

Full Remote

Contract:

Full time

Experience:

Senior (5-10 years)

Work from:

United Kingdom

Offer summary

Qualifications:

7+ years of experience in Data Engineering, Strong proficiency in Python and Scala, Extensive experience with Microsoft Azure, Expertise in SQL and data pipeline tools.

Key responsabilities:

Monitor and maintain data pipelines
Provide technical support and manage source code

MINDVERSE CONSULTING SERVICES SME https://www.mindverseconsulting.com

2 - 10 Employees

See more MINDVERSE CONSULTING SERVICES offers

Job description

This is a remote position.

Our Customer is looking for Data Engineers.

Main Responsibilities

Data Pipeline Maintenance: Continuously monitor and maintain data pipelines for ingesting and transforming data using Scala and SQL on Spark. Diagnose and resolve errors and performance bottlenecks, addressing data discrepancies, ambiguities, and inconsistencies as needed.
Technical Support and Version Control: Provide technical support for data analysis while managing source code and configuration artifacts via GitHub. Deploy code artifacts through GitHub Workflows/Actions.
Technical Leadership: Offer hands-on technical guidance and leadership in developing Spark-based data processing applications using Scala, with a focus on Microsoft Azure Synapse Spark Runtime.
Pipeline Optimization: Design and enhance data pipelines to streamline processing across various stages of the Medallion architecture using Azure Synapse Pipelines.
Data Management: Oversee data ingestion processes, enforce data quality checks using tools like DQ, and manage validation and error-handling workflows.
Configuration Management: Develop and manage configuration settings using JSON-based configurations (e.g., ApplicationConfig, TableConfig) for multiple data zones.
Collaboration: Work closely with data scientists, analysts, and cross-functional teams to ensure smooth integration of data engineering efforts with marketing and business strategies.
Logging and Auditing: Implement and manage logging, auditing, and error-handling practices to maintain data processing integrity, leveraging tools like Azure Log Analytics and KQL queries where applicable.
Testing and Quality Assurance: Conduct unit testing with tools like ScalaTest and maintain rigorous data quality checks to ensure dependable processing outcomes.

Technical Requirements

7+ years of professional experience in Data Engineering
Fluent English
Strong proficiency in programming languages such as Python and Scala
Extensive experience with Microsoft Azure
Hands-on experience with data pipeline tools
Expertise in relational and non-relational databases, including SQL
Solid knowledge of cloud platforms and their data services.
Familiarity with containerization and orchestration (e.g., Docker, Kubernetes) is a plus.
Strong problem-solving skills and the ability to work in a fast-paced environment.
Excellent communication and collaboration abilities.