Role overview

Qualifications

15+ years' experience required
Strong SQL and data modeling skills
Experience with cloud platforms and distributed systems
Prior experience with data migration from various data sources

Responsibilities

Design and develop scalable ETL/ELT pipelines using Databricks
Develop batch, streaming, and CDC ingestion pipelines for near real-time data processing
Leverage Unity Catalog for secure data governance
Integrate Power BI/Tableau/Looker with Databricks using optimized connectors

Key facts

Remote from: Washington (USA)
Freelance
Consultant
English

Hard skills

Databricks Databricks Live Streaming Data Migration Power BI Tableau (Business Intelligence Software) SQL (Programming Language) Data Modeling Apache Kafka Autoload CI/CD

Other skills

Collaboration
Problem Solving

About the company

Diverse Lynx

We are a WBENC and NMSDC certified company helping our clients in their Diversity spending on Staffing or Contingent Workforce Services. Established in 2002 and headquartered out of Princeton-NJ, our 2000+ associates’ strength globally helps clients with talent across Technology, Healthcare, Life Sciences, Aerospace, Automotive, Energy, Pharmaceuticals, Retail, Telecom, Manufacturing and Engineering domains. Our presence in USA, Canada & India helps us support clients in IT, Non-IT, Healthcare, Hospital and Clinical hiring, across the globe.

Company details

Company typeLarge

Company size1001 - 5000

Links

Website LinkedIn See all jobs

Your match analysis

See how your profile stacks up against this role.

We compared the job requirements to your profile to show where you're strong and where you fall short.

Job description

Pay Rate $60.00 - $75.00

Location: Washington, DC Metro Area - Hybrid - strong preference will be given to local candidates to Washington Dc metro area . Need to attend the office in person based on the need.
Duration: 6 months
Competencies: 15+ years’ experience required
Digital : Databricks

Job description:

Design and develop scalable ETL/ELT pipelines using Databricks including Delta Lake, Auto Loader, and DLT.
Develop batch, streaming, and CDC ingestion pipelines to enable efficient and near real-time data processing.
Prior experience with data migration from various data sources like Redshift, DB2, Oracle, Mysql, PLSSql in to databricks Implement ingestion patterns using Auto Loader with checkpointing and schema evolution for structured and semi-structured data Develop and maintain reusable DBX functions for data transformation, data quality checks, and validation logic, ensuring consistent implementation across DLT and notebook-based pipelines.
Leverage Unity Catalog to establish secure data governance, enabling controlled access, lineage tracking, and management of External Locations and Volumes across workspaces.
Partner with cross-functional teams to enable secure data sharing using Delta Sharing across internal domains and external partners.
Integrate Power BI/Tableau/Looker with Databricks using optimized connectors (ODBC/JDBC) and Unity Catalog security controls Implement Medallion architecture (Bronze, Silver, Gold layers).
Develop incremental and CDC-based ingestion pipelines Design and implement real-time streaming pipelines using Kafka and Structured Streaming.
Optimize Spark jobs, SQL queries, and streaming pipelines.
Strong SQL and data modeling skills.
Experience with cloud platforms and distributed systems.
Build, package, and deploy Databricks data pipelines using Databricks Asset Bundles (DAB), with automated CI/CD processes managed through GitHub or GitLab.

Diverse Lynx LLC is an Equal Employment Opportunity employer. All qualified applicants will receive due consideration for employment without any discrimination. All applicants will be evaluated solely on the basis of their ability, competence and their proven capability to perform the functions outlined in the corresponding role. We promote and support a diverse workforce across all levels in the company.

Apply once. Then go straight to the hiring manager.

After you apply, unlock the direct contact details of the people who actually make the call. A quick follow-up makes you 5x more likely to land an interview.