Key Facts

Remote From:

Full time

Senior (5-10 years)

Portuguese

Hard Skills

Apache Spark Python (Programming Language) Apache Spark Databricks Databricks AWS Glue Data Quality Assessment Dimensional Modeling SQL (Programming Language) Version Control +10 more

Other Skills

•
Collaboration
•
Communication
•
Teamwork
•
Problem Solving

Roles & Responsibilities

Experience as a Data Engineer (Mid Level).
Strong knowledge of Databricks and Apache Spark (PySpark and Spark SQL), with proficiency in Python and advanced SQL.
Experience with cloud environments, preferably AWS (S3, Glue Catalog), and Delta Lake / Lakehouse architecture.
Familiarity with batch and/or streaming data pipelines, Git/version control, and basic CI/CD for data.

Requirements:

Develop and maintain batch and streaming data pipelines on Databricks using PySpark and Spark SQL, following the Medallion (Bronze-Silver-Gold) architecture.
Migrate legacy pipelines and jobs from IBM DataStage, Azure Data Factory, and Azure Synapse Analytics to Databricks Workflows; migrate notebooks from Databricks on Azure to AWS; manage notebooks with Databricks Repos and Git.
Implement basic data quality tests and support the dimensional modeling of the Data Warehouse (Star Schema); contribute to data governance initiatives (Unity Catalog) and CI/CD automation.
Monitor production pipelines and provide support, collaborating with DevOps and architecture teams.

Compass.uol

About Compass.uol

Compasso UOL is a Brazilian technology company owned by the UOL Group, which offers technology services, and provides state-of-the-art solutions, contributing to the digital transformation of its customers so they can become leaders in its sectors of activity. Compasso UOL is a company that cultivates people's talent and use state-of-the-art technologies such as Agile Development, Multicloud, Data&Analytics, Cyber Security, Artificial Intelligence/Machine Learning, APIs/Microservices, IoT and others, providing technology and knowledge that help customers in building digital solutions that enable the transformation and evolution of their business.

Company type: XLarge

Founded: 2018

Company size: 5001 - 10000

LinkedIn See all jobs →

Job description

JOB DESCRIPTION

RESPONSIBILITIES AND ASSIGNMENTS

Desenvolver e manter pipelines de dados batch e streaming no Databricks utilizando PySpark e Spark SQL;
Implementar pipelines seguindo arquitetura Medallion (Bronze, Silver e Gold) já definida pela arquitetura de dados;
Apoiar na migração de pipelines e jobs legados de ferramentas como IBM DataStage, Azure Data Factory e Azure Synapse Analytics para Databricks Workflows;
Auxiliar na migração de rotinas e notebooks do ambiente Databricks on Azure para AWS;
Desenvolver e versionar notebooks e códigos utilizando Databricks Repos e Git;
Implementar testes básicos e rotinas de qualidade de dados em pipelines;
Apoiar a modelagem dimensional do Data Warehouse (Star Schema) junto ao time de arquitetura;
Colaborar com o time de DevOps na automação de deploys e CI/CD;
Participar da implementação de governança de dados com Unity Catalog;
Monitorar e dar suporte a pipelines de dados em produção.

REQUIREMENTS AND QUALIFICATIONS

Experiência como Engenheiro(a) de Dados ou Data Engineer Pleno;
Conhecimento em Databricks e Apache Spark (PySpark e Spark SQL);
Experiência com Python e SQL avançado;
Vivência com ambientes em nuvem, preferencialmente AWS (S3, Glue Catalog);
Conhecimento em arquitetura Lakehouse e Delta Lake;
Experiência com pipelines de dados batch e/ou streaming;
Familiaridade com Git e versionamento de código;
Noções de modelagem dimensional (Star Schema / Data Warehouse);
Noções de CI/CD aplicadas a dados;

Diferenciais:

Conhecimento em IBM DataStage (migração ou suporte a legado);
Conhecimento em Azure Data Factory e Azure Synapse Analytics;
Experiência com Databricks em Azure;
Noções de governança de dados e catálogo (Unity Catalog ou similares).

Become a Compasser, be part of AI/R.

Compass UOL is a global firm and part of the AI Revolution Company, together transforming organizations using Artificial Intelligence, Generative AI, and other of today’s most advanced technologies.

We equip our team with proprietary and external AI-driven tools to design and build digital-native platforms, integrating cutting-edge technologies and enabling companies to innovate, transform their businesses, and drive success in their markets.

To achieve this, we attract and develop the best talent, creating opportunities that enhance people’s lives and highlight the positive impact of disruptive technologies.

We empower borderless talent and promote knowledge and opportunities in the latest market trends, driving significant personal and professional growth.

Join us and be part of the AI-driven revolution.

Ready to apply?

APPLY

Share ·