Databricks Specialist / Azure Databricks Consultant

Databricks Specialist / Azure Databricks Consultant

Position Overview:

We are seeking a highly skilled Databricks Specialist with deep expertise in building and deploying enterprise-grade data engineering solutions within Azure Databricks environments. This role will focus on modern data pipeline design, CI/CD enablement, declarative deployment frameworks, and scalable lakehouse architecture.

The ideal candidate will have hands-on experience with Databricks Asset Bundles (Declarative Automation Bundles), declarative data pipelines, Azure cloud services, Terraform, and modern data platform engineering best practices.

This resource will partner closely with platform, analytics, AI, and application teams to build reliable, scalable, and production-ready data solutions.

Core Technical Skills:

· Strong hands-on experience with Azure Databricks

· Proven expertise with Databricks Asset Bundles / Declarative Automation Bundles

· Experience implementing Declarative Data Pipelines

· Strong knowledge of Delta Lake, Lakehouse architecture, and medallion design patterns

· Experience building batch and streaming ETL/ELT pipelines

· Proficiency in Python, PySpark, and SQL

· Experience with Lakeflow / Delta Live Tables

· Strong Azure cloud experience, including:

o Azure Data Lake Storage

o Azure Key Vault

o Azure networking and security fundamentals

· Experience with Terraform (Infrastructure as Code)

· Exposure to cloud architecture and modern data platform design

· Experience with CI/CD pipeline implementation for Databricks deployments

· Working knowledge of Git-based version control workflows

· Experience with Unity Catalog, data governance, and access controls

· Strong understanding of job orchestration, workflows, and scheduling

Key Responsibilities:

· Design, develop, and optimize scalable Azure Databricks data pipelines

· Implement Declarative Automation Bundles / Asset Bundles for repeatable multi-environment deployments

· Build production-grade Lakeflow / Delta Live Table pipelines

· Develop reusable PySpark notebooks, workflows, and modular code packages

· Create CI/CD deployment patterns using GitHub Actions

o Manage infrastructure deployment patterns using Terraform + Databricks Asset Bundles

· Implement best practices for:

o monitoring

o alerting

o testing

o observability

o rollback strategies

· Optimize data workloads for:

o cost

o cluster efficiency

o job runtime

o performance

· Support governance using Unity Catalog, schema management, and permissions

· Partner with architecture teams on enterprise lakehouse modernization initiatives

· Support AI/ML enablement teams with curated feature and training data pipelines

Nice to Have:

These additions make the JD stronger and help recruiting identify better talent:

· Experience with data quality frameworks

· Knowledge of FinOps and Databricks cost optimization

· Exposure to cross-cloud Databricks (AWS/GCP)

Certification / Nice to Have:

· Databricks Certified Data Engineer (Associate or better)

Ideal Background

· 4+ years in data engineering or cloud data platform roles

· 2+ years focused on Databricks implementations

· Experience supporting enterprise data modernization initiatives

· Strong understanding of DevOps and software engineering best practices for data platforms

· Comfortable working in client-facing consulting or augmentation environments