Offer summary

Qualifications:

Bachelor's degree in computer science, information technology, or a related field., Proficiency in Python programming (Python 2.7 and 3.6+)., Experience with data modeling principles and practices., Strong understanding of Git version control and experience with GitLab workflows..

Key responsibilities:

Design and develop data workflows and pipelines using Apache Airflow.

Manage data movement between HDFS, S3, and other systems, including file operations and data transfer.

Implement and optimize data processing tasks with PySpark and manage Hive tables and partitions.

Participate in code reviews, testing, and deployment processes to ensure robust data workflows.

Job description

This is a remote position.

About the Job
As a Senior Data Engineer at Aspire, will play a key role in designing, developing, and maintaining data workflows and pipelines, ensuring the efficient and reliable movement of data across various systems.

What you’ll do

Design, implement, and manage complex DAGs and SubDAGs in Apache Airflow.

Utilize Operators, macros, connections, variables, and xcom for efficient task execution.

Conduct unit testing to ensure the robustness of Airflow workflows.

Perform file operations and data movement between HDFS and S3 (and vice versa).

Manage Hive external tables, update table DDL, and handle partition operations.

Execute basic Hadoop commands and navigatetraverse HDFS.

Work with Hive on Tez for optimized query execution.

Utilize AWS command line tools for tasks such as listing and copying files from S3.

Implement and manage secrets using tools like Ansible Vault and AWS Secrets Manager.

Develop and optimize data processing tasks using PySpark.

Understand and leverage Livy for efficient Spark job execution.

Import and export data between Microsoft SQL Server and HDFS using Sqoop.

Apply dimension modeling principles, define facts and dimensions, and manage surrogate keys.

Implement effective branching strategies in Git.

Participate in the process of raising and reviewing Merge Requests in GitLab.

Utilize SSIS for migrating data workflows, ensuring a smooth transition.

Model data in Snowflake, differentiating between external and internal tables.

Understand the use of views and materialized views in Snowflake.

What you’ll need

Bachelor’s degree in computer science, information technology, or a related field.

Proficient in Python (both Python 2.7 and 3.6+).

Experience in unit testing and testdriven development.

Familiarity with Secrets Management tools and techniques.

Knowledge of data modeling principles and practices.

Strong understanding of Git version control and GitLab workflows.

Experience with SSIS and Snowflake is a plus.

Excellent problemsolving and communication skills.

Awareness or knowledge of IT security best practices as defined by ISOSOC or similar.

Why Aspire
In addition to a competitive longterm total compensation with salary and performancebased bonus, we have a reward philosophy that expands beyond this.

Be part of a (Remote is hereto stay) organization.

Work and learn from great minds.

Explore new opportunities to learn and grow everyday by attending technical and nontechnical training.

Get market exposure by working with international tech leaders.

Nursery reimbursement benefit.

Aspire Wellness Program.

Attend virtual and onsite international tech conference.

Exposure to work in an IT environment that adheres to rigorous security and compliance standards defined by ISOSOC.