BI Consultant

BI Consultant

**Job Title: Databricks Tech Lead** **Location:** Raritan, NJ (Onsite) **Experience :** 10–15 years **Role Overview:** We are seeking an experienced Databricks Tech Lead with a strong background in data engineering, data transformation, ETL/ELT, and modern cloud data platforms. The successful candidate will be responsible for designing and leading scalable data solutions using Databricks, Apache Spark, Delta Lake, SQL, Python, and cloud data services. **Key Responsibilities:** - Design and architect end-to-end data engineering and data transformation solutions using Databricks. - Lead the development of scalable ETL/ELT pipelines for batch and near-real-time data processing. - Develop robust data transformation frameworks using Databricks, Apache Spark, PySpark, and SQL. - Design and implement Delta Lake / Lakehouse architectures, including data ingestion, transformation, optimization, and consumption layers. - Define architecture patterns for bronze, silver, and gold data layers and implement appropriate data modeling strategies. - Design scalable ingestion frameworks for structured and unstructured data from databases, APIs, files, cloud storage, and enterprise applications. - Develop reusable frameworks for data cleansing, transformation, enrichment, validation, and reconciliation. - Optimize Spark workloads, Databricks clusters, SQL queries, and data pipelines for performance and cost efficiency. - Implement incremental processing, CDC, SCD Type 1/2, partitioning, caching, and data optimization strategies. - Define and implement data quality, data validation, exception handling, and monitoring mechanisms. - Establish engineering standards around coding, CI/CD, version control, testing, deployment, and operational support. - Work with cloud platforms such as AWS/Azure/GCP and integrate Databricks with cloud-native data services. - Lead technical design discussions, architecture reviews, and proof-of-concepts. - Mentor and guide data engineers and provide technical leadership across projects. - Collaborate with enterprise/data architects to ensure alignment with broader data platform and integration architecture. ** Technical Skills:** - Strong hands-on experience with Databricks and Apache Spark. - Excellent knowledge of PySpark and Spark SQL. - Strong understanding of Delta Lake and Lakehouse architecture. - Experience with Databricks Workflows, Jobs, notebooks, clusters, and performance optimization. - Strong understanding of ETL and ELT architectures and design patterns. - Experience designing complex data transformation pipelines. - Strong SQL skills and experience with large-scale data processing. - Familiarity with incremental data processing, CDC, SCD Type 1/Type 2, data reconciliation, data quality and validation, error handling, and metadata-driven pipelines. - Strong understanding of Data Lake, Data Warehouse, and Lakehouse architectures. - Experience with dimensional modeling, star/snowflake schemas, and data marts. - Understanding of data partitioning, file formats, and storage optimization. - Experience with major cloud platforms (AWS, Azure, GCP) and their respective services. - Git-based development and branching strategies. - CI/CD using tools such as Azure DevOps, GitHub Actions, Jenkins, or GitLab. - Experience with automated testing and deployment of Databricks notebooks/jobs. - Infrastructure-as-Code exposure using Terraform is desirable. - Strong understanding of SDLC, Agile/Scrum, and production support processes. **Note:** The client's name must not be disclosed in job postings, candidate communications, or any external channels.