SEMANTIC & GENIE ENGINEER (DATABRICKS)

• Design and implement Unity Catalog strategies to manage data access, lineage, and governance effectively.
  
• Architect and maintain Delta Lake structures to ensure high-performance data storage and processing.
  
• Conduct comprehensive data profiling and establish rigorous validation frameworks to uphold data integrity.
  
• Optimize complex SQL queries and database performance to support real-time analytics and reporting needs.
  
• Develop and deploy custom applications using Python, Streamlit, and Databricks Apps to visualize insights and automate workflows.
  
• Engineer advanced multi-agent systems and Databricks Genie Agents to drive autonomous data operations and decision-making.
  
• Configure and monitor Databricks Metric Views to track key performance indicators and system health.
  
• Manage version control and collaborative development processes using Git for all codebases and configurations.
  
• Integrate Machine Learning Pipelines (MCP) to enhance the predictive capabilities of data solutions.
  

Requirements

• Minimum of 5 years of professional experience in data engineering or related technical fields.
  
• Proficiency in Python programming and advanced SQL for data manipulation and analysis.
  
• Deep expertise with Databricks ecosystem components, including Unity Catalog, Delta Lake, and Databricks Genie.
  
• Strong background in data quality assurance, profiling techniques, and validation methodologies.
  
• Demonstrated ability to perform query optimization and tune data pipelines for maximum efficiency.
  
• Experience building interactive dashboards and applications using Streamlit and Databricks Apps.
  
• Knowledge of multi-agent orchestration patterns and the implementation of Genie Agents/Spaces.
  
• Familiarity with Git version control systems and collaborative development practices.
  
• Understanding of Machine Learning Pipelines (MCP) and their integration into data workflows.