Lead ML/AI Solution Engineer with Databricks

We are looking for an experienced Lead ML/AI Solution Engineer to design, build, and deploy a personalization platform that serves recommendation and ranking models through a scalable API layer. The solution consists of a Python FastAPI service exposed through AWS API Gateway and Lambda, which integrates with Databricks Model Serving endpoints. The engineer will work closely with Data Scientists to operationalize personalization models, ensure production-grade reliability, and validate performance through testing and monitoring. Responsibilities Collaborate with Data Scientists to adapt, package, and deploy personalization models (recommendation and ranking models) to Databricks Model Serving using MLflow Build and maintain a FastAPI service that invokes one or more model serving endpoints Implement production-grade engineering practices, including caching, connection pooling, request timeouts, and fail-open fallback mechanisms Configure secure authentication between services and Databricks using Service Principals Set up and maintain the AWS integration layer, including API Gateway, AWS Lambda, and networking/egress connectivity Define and manage infrastructure using Terraform and establish a simple CI/CD pipeline Implement observability and monitoring, including latency, error, and fallback metrics Conduct load and performance testing to validate system behavior under expected traffic volumes Define REST API contracts and implement contract testing to enable early integration by consuming teams Perform offline evaluation of personalization models and support online validation through A/B testing or shadow testing where feasible Present technical results and business impact to both technical and non-technical stakeholders Requirements 5+ years of experience in ML/AI engineering, building and serving recommendation or ranking models, including offline model evaluation At least 1 year of relevant leadership experience Hands-on knowledge of Collaborative Filtering, Two-Tower Architectures, and Learning-to-Rank approaches Strong proficiency in Python with production experience building services using FastAPI Understanding of asynchronous processing, connection pooling, and caching strategies Expertise in timeout handling, fail-open mechanisms, and resilient service design Experience registering, versioning, and serving ML models with MLflow and Databricks Model Serving Familiarity with Service Principal authentication, endpoint scaling, and version management Background in monitoring latency, errors, and fallback rates, including creating alerts and dashboards Skills in load-testing tools such as k6 or Locust, with the capability to validate p99 latency targets Ability to explain technical concepts and outcomes in business terms, working with engineering, data science, and business stakeholders English proficiency at an Upper-Intermediate level (B2) or higher Nice to have Proficiency in Infrastructure-as-Code using Terraform and CI/CD pipeline implementation sufficient to deploy and hand over a reproducible POC environment Experience with AWS API Gateway and Lambda integration, including understanding of cold starts and performance optimization Knowledge of VPC networking and egress connectivity to Databricks Competency in OpenAPI-first REST API design and contract testing using Pact or schema-based testing approaches Capability to design services for multi-region deployment and future scalability Familiarity with GenAI for personalization, including embeddings, vector search, LLM-based re-ranking, and personalized explanations powered by GenAI