Senior Data Engineer

This is a remote position.

About the Job 
As a Senior Data Engineer at Aspire, will play a key role in designing, developing, and maintaining data workflows and pipelines, ensuring the efficient and reliable movement of data across various systems.

• Design, implement, and manage complex DAGs and SubDAGs in Apache Airflow.
  
• Utilize Operators, macros, connections, variables, and xcom for efficient task execution.
  
• Conduct unit testing to ensure the robustness of Airflow workflows.
  
• Perform file operations and data movement between HDFS and S3 (and vice versa).
  
• Manage Hive external tables, update table DDL, and handle partition operations.
  
• Execute basic Hadoop commands and navigate/traverse HDFS.
  
• Work with Hive on Tez for optimized query execution.
  
• Utilize AWS command line tools for tasks such as listing and copying files from S3.
  
• Implement and manage secrets using tools like Ansible Vault and AWS Secrets Manager.
  
• Develop and optimize data processing tasks using PySpark.
  
• Understand and leverage Livy for efficient Spark job execution.
  
• Import and export data between Microsoft SQL Server and HDFS using Sqoop.
  
• Apply dimension modeling principles, define facts and dimensions, and manage surrogate keys.
  
• Implement effective branching strategies in Git.
  
• Participate in the process of raising and reviewing Merge Requests in GitLab.
  
• Utilize SSIS for migrating data workflows, ensuring a smooth transition.
  
• Model data in Snowflake, differentiating between external and internal tables.
  
• Understand the use of views and materialized views in Snowflake.
  

• Bachelor’s degree in computer science, information technology, or a related field.
  
• Proficient in Python (both Python 2.7 and 3.6+).
  
• Experience in unit testing and test-driven development.
  
• Familiarity with Secrets Management tools and techniques.
  
• Knowledge of data modeling principles and practices.
  
• Strong understanding of Git version control and GitLab workflows.
  
• Experience with SSIS and Snowflake is a plus.
  
• Excellent problem-solving and communication skills.
  
• Awareness or knowledge of IT security best practices as defined by ISO/SOC or similar.
  

• Be part of a (Remote is here-to stay) organization. 
  
• Work and learn from great minds.
  
• Explore new opportunities to learn and grow everyday by attending technical and nontechnical training. 
  
• Get market exposure by working with international tech leaders. 
  
• Nursery reimbursement benefit. 
  
• Aspire Wellness Program.
  
• Attend virtual and onsite international tech conference.
  
• Exposure to work in an IT environment that adheres to rigorous security and compliance standards defined by ISO/SOC.