Senior Data Engineer (Kafka, Python, Elasticsearch) (CPT Remote)

Senior Data Engineer (Kafka, Python, Elasticsearch) (CPT Remote) IT - Analyst, Data Management

Cape Town - Western Cape ~ Remote

ENVIRONMENT:

AN award-winning leader in contact centre AI software seeks a passionate Data Engineer with expertise in Kafka pipelines and a thorough understanding of Elastic, looking to contribute to cutting-edge technology and make a difference in the Financial Services industry. You will design, implement, and maintain robust data pipelines; troubleshoot data pipeline and Elasticsearch issues while ensuring data infrastructure aligns with business needs. The ideal candidate will have proven experience in designing and implementing data pipelines including End-to-End Testing of analytics pipelines & managing and optimising Elasticsearch clusters, including performance tuning and scalability. You will also be proficient with Python, Scala or Java and DevOps.

DUTIES:

• Design, implement, and maintain robust data pipelines, ensuring the efficient and reliable flow of data across systems.
• Develop and maintain Elasticsearch clusters, fine-tuning them for high performance and scalability.
• Collaborate with cross-functional teams to Extract, Transform, and Load (ETL) data into Elasticsearch for advanced analytics and search capabilities.
• Troubleshoot data pipeline and Elasticsearch issues, ensuring the integrity and availability of data for analytics and reporting.
• Participate in the design and development of data models and schemas to support business requirements.
• Continuously monitor and optimise data pipeline and Elastic performance to meet growing data demands.
• Collaborate with Data Scientists and Analysts to enable efficient data access and query performance.
• Contribute to the evaluation and implementation of new technologies and tools that enhance Data Engineering capabilities.
• Demonstrate strong analytical, problem-solving, and troubleshooting skills to address data-related challenges.
• Collaborate effectively with team members and stakeholders to ensure data infrastructure aligns with business needs.
• Embody the company values of playing to win, putting people over everything, driving results, pursuing knowledge, and working together.
• Implement standards, conventions and best practices.
  
  

REQUIREMENTS:

• Proven experience in designing and implementing data pipelines.
• Experience with End-to-End Testing of analytics pipelines.
• Expertise in managing and optimising Elasticsearch clusters, including performance tuning and scalability.
• Strong proficiency with data extraction, transformation, and loading (ETL) processes.
• Familiarity with data modeling and schema design for efficient data storage and retrieval.
• Good programming and scripting skills using languages like Python, Scala, or Java.
• Knowledge of DevOps and automation practices related to Data Engineering.
  
  

As a Data Engineer with a focus on Kafka pipelines and Elastic, you will work with the following technologies:

Data Pipelines:

• Kafka / ksqlDB
• Python
• Redis
  
  

Data Storage and Analysis:

• Elasticsearch, cluster management and optimisation
• AWS S3
• PostgreSQL
  
  

DevOps:

• AWS
  
  

Advantageous

• Experience with Data Engineering in an Agile / Scrum environment.
• Familiarity with ksqlDB / Kafka or other stream processing frameworks.
• Familiarity of Data Lakes and the querying thereof.
• Experience with integrating Machine Learning models into data pipelines.
• Familiarity with other data-related technologies and tools.
  
  

ATTRIBUTES:

• Strong analytical and problem-solving abilities, with a keen attention to detail.
• Excellent communication and collaboration skills to work effectively with cross-functional teams.
• A commitment to staying up to date with the latest developments in Data Engineering and technology.
• Alignment with company values and a dedication to driving positive change through data.
  
  

Apply for this Job