Cloud Engineer Azure Databricks

Description

At Sequoia Connect, we are a Talent-First Technology Ecosystem that redefines how elite professionals interact with the global digital landscape. We move beyond traditional models to act as a catalyst for the top 1% of global talent, connecting human potential with complex industrial execution. By joining our inner circle, you are not simply taking a position; you are aligning with a strategic partner dedicated to updating your "Human OS" and accelerating your growth through world-class, high-impact projects.

We are currently partnering with a global IT powerhouse that represents the connected world through innovative, customer-centric experiences. As a USD 6 billion organization and one of the top 7 IT service providers globally, our client empowers over 1,200 global customers—including several Fortune 500 companies—to "Rise™." With a massive network of 163,000+ professionals across 90 countries, they are at the absolute forefront of digital transformation, leveraging next-generation technologies such as 5G, AI, Blockchain, and Quantum Computing.

This is your chance to thrive in a workplace recognized as one of the most sustainable corporations in the world. You will join an environment that values innovation and societal impact, working on end-to-end digital transformation projects for global leaders. If you are a driven professional looking for global career opportunities and exposure to high-impact projects within an international network of expertise, this is where you belong.

We are currently searching for a Cloud Engineer Azure Databricks:

The Challenge (Responsibilities)

• Design, implement, and administer the Azure foundation that Our Client's platform runs on.
• Author and maintain reusable Terraform modules that provision the Azure data estate, including Databricks workspaces, ADLS Gen2, Data Factory, Key Vault, networking, and Azure ML.
• Own remote state, module versioning, drift detection, and the deployment pipelines across development, test, and production environments.
• Design and implement private networking and security postures (private endpoints, hub-and-spoke topology, VNet injection, NSGs, RBAC, etc.).
• Own ADLS Gen2 at the infrastructure layer, including storage account architecture, lifecycle policies, encryption, and access patterns.
• Manage Azure Data Factory platform operations, including integration runtime provisioning, linked service credential design, and deployment automation.
• Provision and operate infrastructure supporting machine learning workloads, including Azure ML workspaces, GPU capacity planning, MLflow registry integration, and Unity Catalog connectivity.
• Build and maintain the monitoring layer using Azure Monitor and Log Analytics, including alerting and operational runbooks.
• Manage FinOps and capacity planning, optimizing costs across DBU and storage spend.
• Establish platform standards, reference configurations, and multi-workspace strategies for the dedicated Databricks administration team.
• Advise on Unity Catalog metastore design, permission models, and Delta Sharing configurations.
• Oversee Identity & Access management via Entra ID integration, SCIM provisioning, and identity federation.
• Enforce compute governance, including cluster policies, instance pools, node type standards, and serverless evaluation.
• Perform cross-boundary diagnostics to investigate complex failures spanning infrastructure, Databricks configuration, or workloads.

Your Profile (Requirements)

• 6+ years in cloud infrastructure or platform engineering, with at least 4 years focused on Microsoft Azure.
• Expert-level, hands-on experience with Terraform, including authoring production modules, managing remote state, and running IaC through CI/CD.
• Demonstrable hands-on Databricks administration experience (workspace administration, Unity Catalog, cluster policies, identity federation, cost governance).
• Deep working knowledge of Azure Networking & Security (private endpoints, VNets, NSGs, Entra ID, managed identities, RBAC, Key Vault).
• Data Lake Architecture experience regarding ADLS Gen2 design and access control at enterprise scale.
• Azure Data Factory platform-side experience (integration runtimes, managed VNet, credential management).
• Strong automation and scripting skills using Python, plus PowerShell and/or Bash.
• Sufficient working knowledge of Spark & SQL to diagnose infrastructure/configuration-level performance issues.
• A track record as a senior technical partner capable of guiding stakeholders and building consensus around architectural approaches.
• High-Performance Mindset: Resilience, emotional intelligence, and a focus on agile delivery.
• Technologist DNA: A deep understanding of the difference between "coding" and "engineering."

Desired

• Experience with Databricks Asset Bundles, the Terraform Databricks provider, and workspace-as-code patterns.
• Experience leading a Unity Catalog migration or multi-workspace consolidation.
• Azure Machine Learning, MLflow, or model-serving infrastructure experience.
• Experience with Kubernetes / AKS and containerized workloads.
• Experience with Policy as Code (Azure Policy, OPA, Sentinel, or Checkov).
• Familiarity with Event-driven infrastructure (Event Hubs, Kafka, or Stream Analytics).
• Formal FinOps practice experience.
• Experience with multi-region or multi-tenant Databricks deployments.
• Relevant certifications: Databricks Certified Data Engineer Professional, Databricks platform administrator credential, Microsoft Certified: Azure Solutions Architect Expert (AZ-305), Microsoft Certified: Azure Administrator Associate (AZ-104), Microsoft Certified: DevOps Engineer Expert (AZ-400), or HashiCorp Certified: Terraform Associate.
• Familiarity with cloud-native foundations or AI coding assistants.

Languages

• Advanced Oral English: For seamless collaboration with global teams.
• Advanced Spanish.

Special Notes

• Our group defines the platform standards and reference architectures that data engineering and data science teams build within.

Work Arrangement

We value flexibility to support your lifestyle. This position is available as:

• Remote

If you meet these qualifications and are pursuing new challenges, start your application on our website to join an award-winning employer. Explore all our job openings | Sequoia Career’s Page: https://www.sequoia-connect.com/careers/

Requirements

• 6+ years in cloud infrastructure or platform engineering, with at least 4 years focused on Microsoft Azure.
• Terraform: Expert-level, hands-on. You have authored and maintained production Terraform modules, managed remote state, and run infrastructure as code through CI/CD.
• Databricks Administration: Demonstrable hands-on administration experience — account and workspace administration, Unity Catalog, cluster policies, identity federation, and cost governance.
• Technical Partnership: A track record as a senior technical resource to adjacent teams.
• Azure Networking & Security: Deep working knowledge of private endpoints, VNets and hub-spoke design, NSGs, Entra ID, managed identities, RBAC, and Key Vault.
• Data Lake Architecture: ADLS Gen2 design and access control at enterprise scale.
• Azure Data Factory: Platform-side experience with integration runtimes, managed VNet, credential management, and deployment automation.
• Automation & Scripting: Strong Python, plus PowerShell and/or Bash, for platform tooling and automation.
• Spark & SQL Literacy: Sufficient working knowledge to diagnose infrastructure and configuration-level performance issues.