Senior Cloud and mlops platform Engineer

About Company
Data science company with a software platform that runs secure and compliant machine learning workloads at scale. We specialize in auditable environments and in synthetic data for defense use cases where the source data cannot leave a government installation. Our first program builds physics-informed failure data for aircraft drivetrain health monitoring, working with an engineering partner that models how bearings and gears fail.

THE ROLE
You will be the initial platform engineer on a defense machine learning program in AWS GovCloud building from an internally defined architecture. You will implement the environment as code for a partner operations team to build and run, then build and operate the pipeline that generates physics-based synthetic sensor data, trains and screens models, and records every run as auditable evidence. Later phases move parts of the workload into an air-gapped government high-performance computing center, a separate operating environment with its own rules.

WHAT YOU WILL DO IN THE FIRST SIX MONTHS
▪ Write Terraform modules, configuration and runbooks that a partner team uses to build and monitor a SageMaker-based environment in GovCloud, and review the result against the definition.
▪ Wrap a partner's physics simulation engine as containerized SageMaker Processing jobs and orchestrate generation with Step Functions.
▪ Build training, tuning and inference pipelines on GPU instances within fixed monthly run budgets.
▪ Build the constraint framework: hard caps, admission control and pre-flight validation, checkpointing and resume, early stopping, and an experiment registry.
▪ Implement lineage: signed manifests linking every dataset and model to its inputs, code, container digest and configuration.
▪ Build container images in GitLab CI with offline dependencies, so they also run under Apptainer on a batch-scheduled cluster with no internet access.
▪ Write clear runbooks and contribute to monthly technical reports.

REQUIRED
▪ Five or more years in cloud or platform engineering, including two or more building machine learning infrastructure on AWS.
▪ Production Terraform, including modules written for other teams to apply.
▪ Amazon SageMaker (Processing, Training, hyperparameter tuning, endpoints) and Step Functions.
▪ Docker and CI/CD (GitLab preferred), with images pinned by digest and dependencies vendored.
▪ AWS security fundamentals: IAM least privilege, KMS customer managed keys, VPC endpoints and PrivateLink, Secrets Manager.
▪ Experience in a regulated environment: AWS GovCloud, FedRAMP High, DoD IL4/IL5, or CUI under NIST SP 800-171.
▪ Strong Python.

PREFERRED
▪ HPC experience: Slurm or PBS, Apptainer or Singularity, Lustre, offline software stacks.
▪ MLflow or a comparable experiment tracker; SageMaker Model Monitor.
▪ AWS cost controls and FinOps for GPU workloads.
▪ Multi-account AWS Organizations under Control Tower, Lake Formation, API Gateway.
▪ DoD Risk Management Framework documentation or CMMC.
▪ AWS certifications (Solutions Architect Professional, Machine Learning, Security).

WORKING ARRANGEMENT
This is a remote role, open to candidates anywhere in the United States. Company provides the hardware and a company-managed laptop.

CITIZENSHIP AND CLEARANCE
U.S. citizenship required. An active Secret clearance is a bonus; eligibility to obtain one required.

EEO:
“Mindlance is an Equal Opportunity Employer and does not discriminate in employment on the basis of – Minority/Gender/Disability/Religion/LGBTQI/Age/Veterans.”