Key Facts

Remote From:

Anywhere

Full time

Senior (5-10 years)

English

Hard Skills

Other Skills

•
Communication
•
Teamwork
•
Problem Solving

Roles & Responsibilities

Proven experience in development, cluster administration, and extensive use of Databricks Notebooks
Hands-on track record migrating Data Lakes or moving workloads between environments (e.g., Legacy Hive Metastore to Unity Catalog)
Strong knowledge in implementing Unity Catalog, data lineage, and permissions management
AWS ecosystem experience with S3, IAM, EC2 and data-related networking services

Requirements:

Plan and execute migration of workloads, notebooks, and data between Databricks workspaces
Refactor and optimize Databricks Notebooks (Python/PySpark/SQL) to ensure compatibility and performance in the new environment
Implement Delta Lake architecture with focus on cost optimization and performance on AWS
Implement Unity Catalog for centralized metadata management, data lineage, and access control (ACLs)

Compass.uol

About Compass.uol

Compasso UOL is a Brazilian technology company owned by the UOL Group, which offers technology services, and provides state-of-the-art solutions, contributing to the digital transformation of its customers so they can become leaders in its sectors of activity. Compasso UOL is a company that cultivates people's talent and use state-of-the-art technologies such as Agile Development, Multicloud, Data&Analytics, Cyber Security, Artificial Intelligence/Machine Learning, APIs/Microservices, IoT and others, providing technology and knowledge that help customers in building digital solutions that enable the transformation and evolution of their business.

Company type: XLarge

Founded: 2018

Company size: 5001 - 10000

LinkedIn See all jobs →

Job description

JOB DESCRIPTION

RESPONSIBILITIES AND ASSIGNMENTS

Planejar e executar a migração de workloads, notebooks e dados entre workspaces Databricks;
Refatorar e otimizar Databricks Notebooks (Python/PySpark/SQL) para garantir compatibilidade e performance no novo ambiente;
Implementar arquitetura Delta Lake com foco em otimização de custos e performance na AWS;
Implementar o Unity Catalog para gerenciamento centralizado de metadados, linhagem de dados e controle de acesso (ACLs);
Definir políticas de governança, garantindo conformidade com LGPD/GDPR e segurança a nível de linha/coluna;
Assegurar a qualidade e integridade dos dados durante e após o processo de migração;
Criar pipelines de ingestão de dados provenientes de arquiteturas baseadas em microsserviços;
Gerenciar a extração e transformação de dados de bancos NoSQL, especificamente MongoDB, para o Data Lake;
Orquestrar fluxos de dados utilizando ferramentas nativas da AWS ou Databricks Workflows;
Disponibilizar datasets otimizados e "gold-standard" para consumo no Tableau;
Colaborar com analistas de negócios para garantir que a modelagem dos dados atenda aos requisitos de performance dos dashboards no Tableau.

REQUIREMENTS AND QUALIFICATIONS

Experiência comprovada em desenvolvimento, administração de clusters e uso extensivo de Databricks Notebooks;
Histórico prático em migrações de Data Lakes ou movimentação de workloads entre ambientes (ex: Legacy Hive Metastore para Unity Catalog);
Forte conhecimento em implementação de Unity Catalog, linhagem de dados e gestão de permissões;
Ecossistema AWS: Experiência sólida com S3, IAM, EC2 e serviços de rede relacionados a dados;
NoSQL & Microsserviços: Experiência prática com manipulação de documentos JSON e extração de dados do MongoDB;
Capacidade de integrar Databricks com Tableau (conectores, otimização de queries, publicação de fontes de dados);
Domínio de Python (PySpark);
SQL avançado.

Diferenciais :
Certificação Databricks Data Engineer (Associate ou Professional).
Experiência com Infraestrutura como Código (Terrafor

Become a Compasser, be part of AI/R.

Compass UOL is a global firm and part of the AI Revolution Company, together transforming organizations using Artificial Intelligence, Generative AI, and other of today’s most advanced technologies.

We equip our team with proprietary and external AI-driven tools to design and build digital-native platforms, integrating cutting-edge technologies and enabling companies to innovate, transform their businesses, and drive success in their markets.

To achieve this, we attract and develop the best talent, creating opportunities that enhance people’s lives and highlight the positive impact of disruptive technologies.

We empower borderless talent and promote knowledge and opportunities in the latest market trends, driving significant personal and professional growth.

Join us and be part of the AI-driven revolution.

Ready to apply?

APPLY

Share ·