Offer summary

Qualifications:

Expertise in ETL optimization and big data processes using Apache Spark or similar technologies., Advanced knowledge in SQL, Python, Spark or PySpark is required., Good understanding of AWS technologies and distributed systems is essential., Proficiency in JIRA, Atlassian, and Git is a must. .

Key responsibilities:

Maintain and enhance Confluent Kafka architecture and design principles.

Build and optimize data integration (ETL) pipelines and streaming applications.

Provide technical leadership and mentor junior engineers on data engineering best practices.

Develop REST-APIs for data transfers and ensure data quality and lineage.

Job description

Expertise in ETL optimization, designing, coding, and tuning big data processes using Apache Spark or similar technologies.

Experience building robust and scalable data integration (ETL) pipelines using SQL, Python, Spark or PySpark. Advanced knowledge in one of the programming language is must.

Maintain and enhance Confluent Kafka architecture, Confluent Kafka design principles, CI/CD Deployment procedures

Experience with building streaming applications with Confluent Kafka (Confluent Kafka preferred but open-source Kafka acceptable)

Development experience using Kafka producers, consumers, and streams (Confluent Kafka preferred but open-source Kafka acceptable)

Experience with building data pipelines and applications to stream and process datasets at low latencies.

Experience with real-time and scalable systems development using Apache Kafka or Confluent Kafka or Kafka Streams.

Show efficiency in handling data - tracking data lineage, ensuring data quality, and improving discover-ability of data.

Good understanding of AWS technologies (S3, AWS Glue, CDK, ECS, EMR, Redshift, Athena)

Sound knowledge of distributed systems and data architecture (lambda)- design and implement batch and stream data processing pipelines, knows how to optimize the distribution, partitioning, and MPP of high-level data structures.

Knowledge of Engineering and Operational Excellence using standard methodologies.

Experience with process improvement, workflow, bench marking and / or evaluation of business processes.

Familiarity with CI/CD process.

Work in a fast paced agile environment.

Experience providing technical leadership and mentoring other junior engineers for best practices on data engineering.

Experience in building REST-APIs for data transfers.

Background in Java and Spring framework is a plus

Proficiency in one of JIRA, Atlassian, and Git is must.

Requirements

Experience with real-time and scalable systems development using Apache Kafka or Confluent Kafka or Kafka Streams.

Experience providing technical leadership and mentoring other junior engineers for best practices on data engineering.

Experience in building REST-APIs for data transfers.