BigData / Hadoop Engineer with Azure - Fulltime

Job Description

Role: BigData / Hadoop Engineer with Azure

Location: Pittsburgh or Houston - Remote

Mode: Full-time / Permanent

It's basically a Hadoop Engineer with Azure

Key Responsibilities:

• Design, develop and maintain an optimal data pipeline architecture
 using both structured data sources and big data for both on-premise and
 cloud-based environments in both streaming and real time.

• Develop and automate ETL code using scripting languages, ETL tools
 and job scheduling software to support all reporting and analytical 
data needs.

• Design and build dimensional data models to support the data warehouse initiatives.

• Assemble large, complex data sets that meet the analytical needs of the data science team.

• Assess new data sources to better understand availability and quality of data.

• Identify, design, and implement internal process improvements: 
automating manual processes, optimizing data pipeline performance, 
re-designing infrastructure for greater scalability and access to 
information.

• Participate in requirements gathering sessions to distill technical requirements from business requests.

• Collaborate with business partners to productionize, optimize, and scale enterprise analytics.

• Collaborate with data architects and modelers on data store designs and best practices


Education/Certifications:

• Bachelor’s degree in Computer Science, Engineering, Information Science, Math or related discipline

• Data engineering, data management or cloud certification is a plus

Experience/Minimum Requirements:

• Five (5)+ years’ experience in traditional and modern Big Data 
technologies (HDFS, Hadoop, Hive, Pig, Sqoop, Kafka, Apache Spark, 
hBase, Oozie, No SQL databases, PostgreSQL, GIT, Python, REST API, 
Snowflake, etc.)

• Two (2)+ years’ experience building data platforms using Azure stack (Azure Data Factory, Azure DataBricks, etc.)

• Experience with object-oriented/object function scripting languages: Python, Java, C++, Scala

• Experience extracting/querying/joining large data sets at scale

• Experience utilizing Snowflake to build data marts with the data residing in Azure storage is a plus

Other Skills/Abilities:

• Thorough understanding of relational, columnar and NoSQL database architectures and industry best practices for development

• Understanding of dimensional data modeling for designing and building data warehouses

• Excellent advanced SQL coding and performance tuning skills

• Experience with parsing data formats such as XML/JSON and leveraging external APIs

• Understanding of agile development methodologies

• Ability to work in a team-oriented, collaborative environment; good interpersonal skills

• Strong analytical and problem-solving skills; ability to weigh 
various suggested technical solutions against the original business 
needs and choose the most cost-effective solution

• Keen attention to detail and ability to access impact of design changes prior to implementation

• Self-driven, highly motivated and ability to learn quick

• Ability to effectively prioritize and execute tasks in a high-pressure environment

• Strong customer service orientation

• Ability to present and explain technical information to diverse 
types of audiences in a way that establishes rapport and gains 
understanding

• Work experience with geospatial data and spatial analytics is preferred
 

Qualifications

This is Fulltime position and looking for only USC/GC

No H1B or EAD's

Additional Information