Strategic Data Leader – Databricks

Strategic Data Leader – Databricks | Large-Scale Data Platform Leadership
Remote Role
Duration: 12+ months

Job Summary:
Client is seeking a highly experienced Strategic Data Leader with deep expertise in Databricks, enterprise-scale data architecture, and large-volume data transformation. This role requires a hands-on leader who can drive data strategy, modernize data platforms, and lead large-scale data engineering initiatives across complex business environments. The ideal candidate should have strong experience handling large data volumes (“Large Data Hunger”), designing scalable cloud-native architectures, and leading cross-functional teams to deliver high-performance analytics and AI-ready data ecosystems.

Key Responsibilities:

Strategic Leadership

• Define and drive enterprise-wide data strategy, architecture roadmap, and modernization initiatives.
• Lead large-scale data platform transformation programs using Databricks, Snowflake, Azure/AWS/GCP cloud ecosystems.
• Partner with business leaders, analytics teams, and executive stakeholders to align data strategy with business goals.
• Build and scale data governance, data quality, metadata management, and data security frameworks.

Databricks Platform Ownership

• Architect and manage enterprise-wide Databricks Lakehouse Platform.
• Lead implementation of Delta Lake, Unity Catalog, ML pipelines, and real-time data processing frameworks.
• Optimize Databricks workloads for performance, scalability, and cost efficiency.
• Establish best practices for cluster management, notebook standards, CI/CD, and deployment automation.

Large Data Engineering & Architecture

• Design and implement high-volume batch and streaming data pipelines using Spark, PySpark, SQL, and Databricks workflows.
• Handle petabyte-scale structured and unstructured data from multiple enterprise sources.
• Build robust ETL/ELT frameworks supporting advanced analytics, AI/ML, and BI reporting.
• Drive migration from legacy platforms to modern lakehouse architecture.

Team Leadership

• Lead and mentor data engineers, architects, analysts, and platform teams.
• Provide architectural guidance and hands-on technical leadership.
• Establish engineering standards, delivery governance, and operational excellence.
• Drive agile delivery across multiple strategic programs.

Required Skills & Qualifications:

Technical Expertise

• 12+ years of experience in Data Engineering, Data Architecture, and Enterprise Data Strategy
• 5+ years of strong hands-on experience with Databricks
• Expert knowledge of:
  • Apache Spark / PySpark
  • Delta Lake
  • SQL / Advanced SQL Optimization
  • Data Warehousing concepts
  • Data Modeling (Dimensional + Lakehouse)
  • ETL / ELT Frameworks
  • Cloud platforms: Azure / AWS / GCP
  • CI/CD for data platforms
  • Data Governance & Security

Leadership Experience

• Proven experience leading enterprise-wide data transformation initiatives
• Experience managing large distributed technical teams
• Strong executive stakeholder management skills
• Ability to translate business strategy into technical execution

Preferred Qualifications

• Experience with Snowflake, Kafka, Airflow, dbt, Synapse, Fabric
• Exposure to AI/ML platform integration
• Experience with Data Mesh / Data Fabric architecture
• Databricks Certifications preferred
• Strong consulting or client-facing leadership experience