Senior Data Engineer

• Strategy Creation: Collaborate with cross-functional teams to define the data engineering strategy aligned to business objectives, including data modeling that unifies data assets across a range of source systems used to manage the operations of our partnering hospitals.
• Pipeline Development: Define and execute processes needed to develop, test, deploy, and maintain high quality data pipelines. Oversee the end-to-end development of data pipelines from source data extraction through to production-grade analytical dataset delivery, ensuring data quality and security throughout the pipeline.
• Performance Optimization: Continuously monitor and optimize data processing performance and efficiency. Identify and address bottlenecks, optimize query performance, and improve overall system stability.
• Data Governance: Establish and enforce data quality management policies, data access controls, and data privacy standards.
• Technical Leadership: Stay abreast of the latest developments in engineering tools and best practices. Provide guidance to the team about technical challenges.
• Documentation: Maintain clear and comprehensive documentation of data pipelines, architecture, and processes to ensure knowledge sharing and team continuity.
• Third-party Management: Evaluate and manage relationships with third-party vendors and tools, making informed decisions about when to leverage external solutions.

Requirements

• 6+ years in data engineering roles with progressively increasing responsibilities.
• Deep understanding of data modeling, data architecture, and data integration best practices.
• Strong hands-on experience with Apache Spark and cloud computing technologies.
• Advanced proficiency in Python and SQL.
• Familiarity with data governance, security, and privacy principles.
• Comfort using collaboration tools such as GitHub or equivalent to manage development life cycle.
• Excellent data modeling and engineering skills, and a talent for translating business objectives into technical solutions.
• High energy, humble team player with “get it done” attitude, seeking collaboration with colleagues.
• Ability to manage multiple projects simultaneously.

• Experience engineering in Databricks strongly preferred.
• 3+ years of software engineering with python in a production environment.
• Experience with the Azure cloud ecosystem.
• Experience developing production-ready, real-time machine learning model serving pipelines.
• Comfort developing in the Apache Spark Structured Streaming paradigm.
• Experience working in the veterinary services industry, or a private equity-backed services company.
• Working knowledge of Microsoft Excel and Office 365.