Cloud Data Engineer

As a recruitment company, DCG understands that every business is powered by experienced professionals. Our management style and partnership approach enable us to meet your needs and provide continuous support. Due to our ongoing growth and the large number of recruitment projects we undertake for our partners, we are currently looking for:

Cloud Data Engineer

Responsibilities:

• Implement and manage data ingestion pipelines from diverse sources such as Kafka, RDBMS (Postgres) using CDC (Change Data Capture), and file systems (CSV) following Medalion Architecture principles
• Develop and optimize data transformations using PySpark and SQL to handle data ranging from MB to GB, depending on the source
• Conduct unit testing and integration testing to ensure the accuracy and reliability of data transformations and pipelines
• Work with AWS technologies, including S3 for data storage and Docker on AWS for containerized applications
• Implement and manage infrastructure using Terraform, such as creating S3 buckets, managing Databricks Service Principals, and deploying infrastructure as code
• Deploy and manage solutions using CI/CD pipelines, particularly with CircleCI, to ensure seamless and automated deployment processes

Requirements:

• Minimum 4-5 years of a professional experience
• Proficiency in SQL and Python
• Strong experience with AWS cloud services
• Hands-on experience with DataBricks
• Knowledge of ETL Processing
• Effective communication skills in English (minimum B2 level)
• Knowledge of system design
• Understanding of Medalion Architecture

Nice to have:

• Familiarity with Kedro and Airbyte
• Knowledge of Machine Learning

Offer:

• Private medical care
• Co-financing for the sport card
• Training & learning opportunities
• Constant support of dedicated consultant
• Team-building events organized by DCG
• Employee referral program