Data Platform Engineer (102-08SENG-01)

• 30+ days ago

OpsBrasil

Role overview

Responsibilities

Convert Azure Data Factory estate into Databricks workflows on AWS
Migrate 2,094 ADF pipelines as reusable templates
Translate 9,859 pipeline activities, requiring mapping and rewriting as Lambda or Step Functions
Move 471 Spark dataflows onto Databricks on AWS
Rehost 4 Databricks workspaces including notebook paths and Unity Catalog rewiring
Reconcile migrated data against source systems in a regulated environment

Key facts

Remote from: Brazil
Full time
Platform Engineer
English

Hard skills

About the company

OpsBrasil

IT Services & Consulting
SME
11–50

Staff augmentation, software development team augmentation, international recruiting, virtual teams, remote tech talent

See all jobs

Your match analysis

See how your profile stacks up against this role.

We compared the job requirements to your profile to show where you're strong and where you fall short.

Job description

This role converts a large Azure Data Factory estate into Databricks workflows on AWS. The scope for year one: 2,094 ADF pipelines to migrate — built as reusable templates rather than one-by-one — 9,859 pipeline activities to translate (some map directly, others need rewriting as Lambda or Step Functions), 471 Spark dataflows to move onto Databricks on AWS, and 4 Databricks workspaces (Dev, QA, Pre-prod, Prod) to rehost, including notebook paths and Unity Catalog rewiring. This is a regulated environment, so reconciling migrated data against source systems is part of the definition of done, not an afterthought.