Offer summary

Qualifications:

Minimum of 2 to 3 years of experience in data engineering and architecture., Strong expertise in Databricks, Snowflake, and/or Microsoft Fabric., Proficiency in SQL, Python, Spark, and distributed data processing frameworks., Knowledge of AI/ML workflows and data governance best practices..

Key responsibilities:

Design and implement scalable, cloud-based data pipelines and architectures.

Develop and optimize ETL/ELT workflows for efficient data movement.

Collaborate with clients to understand business needs and deliver tailored data solutions.

Provide training and support to enhance organizational data literacy.

Job description

This is a remote position.

Infoverity is seeking an experienced Data Engineer to join our dynamic consulting team, helping clients architect, build, and optimize modern data platforms. In this role, you will design and implement scalable data solutions, leveraging cloud technologies, specifically Databricks, to drive innovation in AI/ML, data warehousing, and data integration. As a key player at Infoverity you will collaborate with cross-functional teams, including data scientists, business analysts, and solution architects, to deliver high-impact data solutions tailored to our clients’ needs.

Key Responsibilities

Data Engineering & Architecture: Design and implement scalable, cloud-based data pipelines to ingest, transform, and store data from various sources.

Data Warehousing & Integration: Develop and optimize ETL/ELT workflows, ensuring efficient data movement between systems, specifically Databricks.

AI/ML Enablement: Work alongside data science teams to support feature engineering, model training, and ML operations (MLOps) in cloud environments.

Data Modeling & Governance: Develop data models, schemas, and best practices to ensure data integrity, consistency, and security.

Performance Optimization: Monitor, troubleshoot, and optimize query performance, ensuring scalability and efficiency.

Consulting & Client Engagement: Work directly with clients to understand business needs, recommend best practices, and deliver tailored data solutions.

Data Architecture Design: Design and implement scalable, high-performance data architectures using Databricks.

Requirement Gathering: Lead requirement-gathering sessions with clients and internal teams to understand business needs and define best practices for solutions including data workflows.

Cloud Collaboration: Collaborate with cloud platform teams to optimize data storage and retrieval in environments like AWS S3, Azure Data Lake, and Delta Lake, among others.

Workflow Optimization: Translate complex data processes, such as those in Alteryx and Tableau, into optimized Databricks workflows using PySpark and SQL.

Automation Development: Develop reusable automation scripts to streamline workflow migrations and improve operational efficiency.

Development Support: Provide hands-on development and troubleshooting support to ensure smooth implementation and optimal performance.

Governance & Best Practices: Partner with cross-functional teams to establish data governance frameworks, best practices, and standardized reporting processes.