AWS Data Engineer

Description

Company Overview:

Lean Tech is a rapidly expanding organization situated in Medellín, Colombia. We pride ourselves on possessing one of the most influential networks within software development and IT services for the entertainment, financial, and logistics sectors. Our corporate projections offer a multitude of opportunities for professionals to elevate their careers and experience substantial growth. Joining our team means engaging with expansive engineering teams across Latin America and the United States, contributing to cuttingedge developments in multiple industries.

Currently, we are seeking a Mid+ AWS Data Engineer to join our team. Here are the challenges that our next warrior will face and the requirements we look for:

Position Title: AWS Data Engineer

Location: Remote LATAM

What you will be doing:

This position involves translating highlevel solution designs into functional AWS data pipelines and services. You will closely collaborate with an Onshore Data Architect to implement realtime change data capture using Oracle GoldenGate and integrate it with Amazon MSK for downstream processing. Key responsibilities include developing containerized transformation microservices on AWS EKS, designing efficient NoSQL storage in Apache Cassandra, implementing graphbased pipelines for Amazon Neptune clusters, and enabling adhoc analytics via Amazon Athena on S3 data lakes. Automating deployments using Terraform or CloudFormation, and integrating CICD processes for continuous improvements is critical. This role requires seamless collaboration with crossfunctional teams to onboard new data sources, support analyticsML use cases, and maintain data quality and governance across all pipelines. Working within Lean Techs distributed team, the position presents opportunities to tackle complex data challenges using advanced AWS services.

Collaborate with the Onshore Data Architect to transform highlevel solution designs into functional AWS data pipelines and services.
Ingest realtime change data from legacy systems using Oracle GoldenGate and publish it to Amazon MSK (Kafka) for processing by downstream services.
Develop containerized microservices for data transformation on AWS EKS, ensuring code is modular and reusable.
Build and maintain batch workflows to copy data nightly into Kafka topics and S3 landing zones.
Design and optimize NoSQL storage solutions in Apache Cassandra for highthroughput event data, ensuring efficient readwrite patterns.
Implement graphbased pipelines using custom systems to populate and update Amazon Neptune clusters with periodic jobs.
Facilitate adhoc analytics by organizing data lakes on Amazon S3 and configuring Amazon Athena schemas for effective, selfservice querying.
Automate infrastructure and deployments through Terraform or CloudFormation, integrating CICD processes for microservices and InfrastructureasCode (IaC) changes.
Monitor and troubleshoot comprehensive data flows using AWS CloudWatch, Kubernetes dashboards, and Kafka tools, implementing alerts on key Service Level Agreements (SLAs).
Ensure robust data quality and governance by embedding validation checks, evolving schema strategies, and documenting all data pipelines.
Work closely with crossfunctional teams to onboard new data sources, supporting analytical and machine learning use cases.
Document processes and facilitate knowledge transfer to ensure longterm maintainability and continuous improvement within the nearshore team.

Required Skills & Experience:

3+ years of AWS data engineering experience, including proficiency with MSK, EKS, S3, Athena, and Neptune.
Advanced programming skills in Python and Java for developing transformation microservices in Kubernetes (EKS).
Experience with Oracle GoldenGate for realtime change data capture from legacy systems to Amazon MSK (Kafka).
Proficiency with NoSQL databases, specifically Apache Cassandra, for highthroughput event data management.
Comprehensive understanding of InfrastructureasCode (IaC) using Terraform and CloudFormation.
Intermediate to advanced skills in CICD methodologies, utilizing tools such as Jenkins, GitHub Actions, or AWS CodePipeline.
Handson experience with container orchestration using Docker and Kubernetes.
Advanced knowledge of Apache Kafka for data parsing, enrichment, and restructuring in data pipelines.
Practiced expertise in implementing graphbased data architectures using Amazon Neptune.
Skilled in data lake management and analytics using Amazon S3 and configuring Amazon Athena schemas.
Ability to automate monitoring and alert systems using AWS CloudWatch.
Excellent written and verbal communication skills, especially for remote collaboration with crossfunctional teams.

Good to Have:

Experience with Apache Spark for distributed data processing and analytics.
Familiarity with ElasticSearch for advanced search capabilities and data exploration.
Certification in AWS Certified Solutions Architect or AWS Certified DevOps Engineer.
Exposure to data governance frameworks and tools for maintaining data quality.
Knowledge of RESTful API development and integration.
Strong problemsolving skills and adaptability in fastpaced environments.
Experience with advanced data visualization tools, such as Tableau or Power BI.
Proven experience in agile methodologies for project management.

Soft Skills:

Strong written and verbal communication skills, essential for creating clear documentation of data pipelines and collaborating effectively using remote communication tools like Slack and Confluence.
Excellent problemsolving abilities, crucial for troubleshooting complex AWS data flows and ensuring the robustness of infrastructure deployments and microservices.
Effective teamwork and collaboration skills, demonstrated by working closely with onshore and crossfunctional teams to align on deliverables and expectations.
Adaptability and flexibility in handling dynamic project requirements, shown by integrating tooling such as CICD processes and infrastructure automation with Terraform and CloudFormation.
Leadership in managing remote collaborations by delivering proactive updates on progress and coordinating with various stakeholders to meet timelines and objectives.

Why you will love Lean Tech:

Join a powerful tech workforce and help us change the world through technology
Professional development opportunities with international customers
Collaborative work environment
Career path and mentorship programs that will lead to new levels.

Join Lean Tech and contribute to shaping the data landscape within a dynamic and growing organization. Your skills will be honed, and your contributions will play a vital role in our continued success. Lean Tech is an equal opportunity employer. We celebrate diversity and are committed to creating an inclusive environment for all employees.