Data Engineer

Skills:
pyspark, AWS Glue, Terraform, streaming, python, DevOps,

Data Engineer Duties & Responsibilities

Design and implement scalable data pipelines to process and transform large volumes of

data using AWS services such as Glue, Kafka, S3, EMR, Athena.

Collaborate with cross-functional teams including data scientists, analysts, and software

engineers to understand data requirements and deliver solutions that meet business

needs.

Optimize data workflows for performance, reliability, and cost-efficiency on AWS cloud

platform. Performance Tune ETL Pipelines, database queries.

Develop and maintain ETL processes to ensure data quality and integrity throughout the

data lifecycle.

Implement best practices for data security, access control, and compliance in

accordance with industry standards and regulations.

Continuously monitor and troubleshoot data pipelines to identify and resolve issues in a

timely manner.

Stay updated with the latest AWS services and technologies and evaluate their potential

for improving our data infrastructure.

Document data architecture, design decisions, and technical specifications for reference

and knowledge sharing.

Data Engineer Experience & Qualifications

Design and implement scalable data pipelines to process and transform large volumes of

data using AWS services such as Glue, Kafka, S3, EMR, Athena.

Collaborate with cross-functional teams including data scientists, analysts, and software

engineers to understand data requirements and deliver solutions that meet business

needs.

Optimize data workflows for performance, reliability, and cost-efficiency on AWS cloud

platform. Performance Tune ETL Pipelines, database queries.

Develop and maintain ETL processes to ensure data quality and integrity throughout the

data lifecycle.

Implement best practices for data security, access control, and compliance in

accordance with industry standards and regulations.

Continuously monitor and troubleshoot data pipelines to identify and resolve issues in a

timely manner.

Stay updated with the latest AWS services and technologies and evaluate their potential

for improving our data infrastructure.

Document data architecture, design decisions, and technical specifications for reference

and knowledge sharing.

Data Engineer Educational Qualifications & Skills

Bachelor or Masters degree in Computer Science Engineering, or related field.

Proven experience as a Data Engineer or similar role, with at least 5 years of experience

in building and maintaining data infrastructure on AWS cloud platform.

Strong proficiency in AWS services such as Glue, Redshift, Lambda, Athena, S3,

QuickSight etc.

Experience with big data technologies such as Hadoop, Spark, Kafka, etc.

In-depth knowledge of data architecture principles, data modelling, and database

design.

Hands-on experience with SQL, Python, and any other programming languages for data

manipulation and scripting.

Familiarity with DevOps practices and tools for CI/CD, infrastructure as code, and

automated deployment.

Knowledge of machine learning concepts and techniques for data analysis and

predictive modelling.

Excellent problem-solving skills and attention to detail, with a focus on delivering high-

quality solutions.

Ability to work effectively in a fast-paced environment with tight deadlines and changing

priorities.

Strong communication and collaboration skills, with the ability to work effectively in a

cross-functional team environment