Data Engineer II

Job Title: Contract Data Engineer (API & Database Focus)

Role Summary:

We are seeking an experienced and highly skilled Contract Data Engineer. This role is focused on building robust data pipelines, primarily involving the integration of data from various APIs and managing critical data infrastructure. The successful candidate will work hands-on with Databricks, Snowflake, and/or relational databases, ensuring data is clean, reliable, and optimized for consumption by analytics tools like Tableau. This is a project-based contract position requiring someone who can quickly integrate and deliver results.

Key Responsibilities:

• Design, develop, and maintain scalable data ingestion pipelines to extract data from various third-party and internal APIs (REST, SOAP, etc.).
• Implement efficient data transformation and loading processes (ETL/ELT) within the data platform.
• Manage and optimize data storage and schemas in Snowflake and Postgres databases.
• Utilize Databricks for data processing, transformation, and orchestration tasks.
• Ensure data quality, accuracy, and integrity throughout the data pipelines.
• Collaborate with data analysts and BI developers (particularly Tableau users) to understand data requirements and optimize data models for performance.
• Monitor data pipelines and systems for performance issues, errors, and data discrepancies, implementing necessary fixes and improvements.
• Develop and maintain technical documentation for data pipelines, processes, and data models.
• Troubleshoot data-related issues and provide timely resolutions.
• Implement best practices for data security and governance within the data platform.

Required Skills and Qualifications:

• Proven experience as a Data Engineer, with a strong focus on API integration and database management.
• Experience building data pipelines for API data ingestion, including handling authentication, error handling, and data parsing.
• Strong proficiency in SQL and experience with database design, query optimization, and performance tuning in relational databases (e.g., Postgres).
• Hands-on experience with Snowflake as a cloud data warehouse, including data loading, querying, and performance optimization.
• Experience using Databricks for data processing, ETL/ELT, and pipeline orchestration (e.g., PySpark, notebooks).
• Proficiency in a programming language commonly used for data engineering (e.g., Python).
• Experience working with cloud platforms (e.g., AWS, Azure, GCP) where Snowflake and Databricks are deployed.
• Understanding of data warehousing concepts and best practices.
• Experience supporting BI tools (like Tableau) by providing clean, structured, and performant data sources.
• Excellent problem-solving skills and ability to work independently in a fast-paced environment.
• Strong communication and collaboration skills.

Desired Skills (Nice to Have):

• Experience with specific ETL/orchestration tools (e.g., Airflow, Fivetran, dbt).
• Familiarity with containerization technologies (e.g., Docker).
• Experience in a contract or consulting role.
• Knowledge of data governance and compliance standards.
• Experience with BI tools like Tableau, including dashboard creation or assisting BI developers with visualization best practices related to data structure.