Careers

Sr. Data Engineer

Job Category: Data Engineer
Job Type: Full Time
Job Locations: Chicago IL Knoxville TN

About the Role:

We are looking for a Senior Data Engineer with strong real-life experience in python development, including pySpark in an AWS Cloud environment. The ideal engineer will have experience with AWS Cloud on data integration with Apache Spark, EMR, Glue, Kafka, Kinesis and Lambda in S3, Redshift, RDS, and MongoDB/DynamoDB ecosystems.

Responsibilities:

  • Build the infrastructure required for optimal extraction, transformation and loading of data from a wide variety of sources Hadoop, Spark, AWS Lambda, etc.
  • Experience with AWS Cloud on data integration with Apache Spark, EMR, Glue, Kafka, Kinesis, and Lambda in S3, Redshift, RDS, MongoDB/DynamoDB ecosystems
  • Strong real-life experience in python development especially in pySpark in AWS Cloud environment.
  • Design, develop, test, deploy, maintain and improve data integration pipeline.
  • Develop pipeline objects using Apache Spark / Pyspark /Python or Scala
  • Design and develop data pipeline architectures using Hadoop, Spark and related AWS services.
  • Load and performance test data pipelines built using the above-mentioned technologies

Required Knowledge And Level Of Experience

  • At least 5 years of experience with AWS and Python programming, experience with Python frameworks (e.g. Django, Flask, Bottle)
  • Expert-level knowledge of using SQL to write complex, highly-optimized queries across large volumes of data.
  • Working experience on ETL pipeline implementation using AWS services such as Glue, Lambda, EMR, Athena, S3,SNS,Kinesis, Data-Pipelines, Pyspark
  • Working experience on ETL pipeline implementation using AWS services such as Glue, Lambda, EMR, Athena, S3,SNS,Kinesis, Data-Pipelines, Pyspark
  • Hands-on Experience using programming language Scala, python, R, or Java
  • Expert-level knowledge of using SQL to write complex, highly-optimized queries across large volumes of data.
  • Knowledge or experience in architectural best practices in building data lakes.
  • Strong problem solving and troubleshooting skills with the ability to exercise mature judgment.

Apply for this position

Allowed Type(s): .pdf, .doc, .docx