Senior Data Engineer – Integration Hub & Data Pipelines

Key Responsibilities

Data Pipeline Design & Development

• Design, build, and maintain robust ETL/ELT data pipelines for batch and streaming workloads.
• Implement data ingestion and transformation workflows using Apache Airflow, Apache NiFi, Apache Spark, and Kafka.
• Integrate data from multiple sources including REST APIs, files, relational databases, message queues, and external SaaS platforms.
• Optimize pipelines for performance, scalability, reliability, and cost efficiency.

Integration Hub & Platform Engineering

• Develop and operate a centralized data integration hub that supports multiple upstream and downstream systems.
• Build reusable, modular integration components and frameworks.
• Ensure high availability, fault tolerance, and observability of data workflows.

Data Infrastructure & Storage

• Design and manage data warehouses, data lakes, and operational data stores using PostgreSQL and related technologies.
• Implement appropriate data modeling strategies for analytical and operational use cases.
• Manage schema evolution, metadata, and versioning.
   
   

Data Quality, Security & Governance

• Implement data validation, monitoring, and reconciliation mechanisms to ensure data accuracy and completeness.
• Enforce data security best practices, access controls, and compliance with internal governance policies.
• Establish logging, alerting, and auditability across pipelines.

Automation & Operations

• Automate data workflows, deployments, and operational processes to support scale and reliability.
• Monitor pipelines proactively and troubleshoot production issues.
• Improve CI/CD practices for data engineering workflows.

Collaboration & Stakeholder Engagement

• Work closely with data scientists, analysts, backend engineers, and business stakeholders to understand data requirements.
• Translate business needs into technical data solutions.
• Provide technical guidance and best practices across teams.

• 5+ years of hands-on experience as a Data Engineer or in a similar role.
• Proven experience as an individual contributor on at least three end-to-end data engineering projects, from design to production.
• Strong hands-on experience with:
  • Apache Airflow / Dagster
  • Apache NiFi
  • Apache Spark
  • Apache Kafka
  • PostgreSQL 
• Extensive experience integrating data from APIs, files, databases, and third-party systems.
• Strong SQL skills and experience with data modeling.
• Solid programming experience in Python and/or Java/Scala.
• Experience with Linux environments and version control systems (Git).
• Strong problem-solving, debugging, and performance-tuning skills.

• Experience with cloud platforms (AWS, Azure, or GCP).
• Experience with containerization and orchestration (Docker, Kubernetes).
• Knowledge of data lake technologies and formats (Parquet, ORC, Iceberg, Delta Lake).
• Familiarity with monitoring and observability tools for data pipelines.
• Experience working in fast-paced or startup environments.