Lead Data Software Engineer

Our team is expanding, and we're in search of a Lead Data Software Engineer to join us. This position is well-suited for an individual who enjoys engineering resilient data systems that inform key business decisions. You'll work alongside a group committed to producing clean, well-validated code and maintaining disciplined engineering practices. For someone drawn to solving data problems at scale, this role presents considerable opportunity for growth and lasting impact Responsibilities Build and structure scalable data pipelines that ensure reliable data flow across a variety of systems Develop and keep up ingestion pipelines that source data from multiple, varied origin systems Write clean, high-quality, well-tested code rooted in sound software engineering practices Work together with cross-departmental teams to understand data needs and convert them into viable technical solutions Observe data pipeline performance closely, addressing potential issues early to maintain consistent system reliability Contribute to code review processes to help sustain quality expectations and share insights across the team Investigate and correct data quality concerns to maintain dataset accuracy and consistency Play a part in continuously refining data architecture methods and engineering standards Compile documentation covering technical processes and solutions to facilitate knowledge sharing and support new hire onboarding Work inside an agile structure, engaging in sprint planning and other routine team gatherings Requirements At least 5 years of relevant practical experience in the field A minimum of one year spent in roles leading and managing development teams Demonstrated skill with PySpark and Spark SQL for developing and managing large-scale data workflows Hands-on background constructing and supervising Lakeflow pipelines and Jobs for data orchestration purposes Proficient with Auto Loader and JDBC/CDC ingestion strategies, including experience with sources like DB2 and SQL Server Strong familiarity with Delta Lake and Unity Catalog for managing data storage and governance requirements Track record maintaining data quality and carrying out source reconciliation to ensure accurate, trustworthy data Background developing unit tests using pytest to validate code reliability and correctness Working knowledge of Git and Asset Bundles to support version control and deployment management Upper-intermediate English language skills or higher (B2) Nice to have Experience participating in DataStage migration efforts Background extracting data from Dynamics 365/Dataverse systems Databricks certification held