Careers

Sr. Data Engineer

Job Type: Full Time
Job Locations: Chicago IL Knoxville TN

Wavicle Data Solutions, a rapidly growing data analytics solutions consulting firm, could be the right fit for you.

We are looking for a Senior Data Engineer with strong real-life experience in python development, including pySpark in an AWS Cloud environment. The ideal engineer will have experience with AWS Cloud on data integration with Apache Spark, EMR, Glue, Kafka, Kinesis and Lambda in S3, Redshift, RDS, and MongoDB/DynamoDB ecosystems.

Role and Responsibilities

  • Experience with AWS Cloud on data integration with Apache Spark, EMR, Glue, Kafka, Kinesis, and Lambda in S3, Redshift, RDS, MongoDB/DynamoDB ecosystems
  • Strong real-life experience in python development especially in pySpark in AWS Cloud environment.
  • Design, develop, test, deploy, maintain and improve data integration pipeline.
  • Working experience on ETL pipeline implementation using AWS services such as Glue, Lambda, EMR, Athena, S3,SNS,Kinesis, Data-Pipelines, Pyspark
  • Hands-on Experience using programming language Scala, python, R, or Java
  • Develop pipeline objects using Apache Spark / Pyspark /Python or Scala
  • Design and develop data pipeline architectures using Hadoop, Spark and related AWS services.
  • Build the infrastructure required for optimal extraction, transformation and loading of data from a wide variety of sources Hadoop, Spark, AWS Lambda, etc.
  • Knowledge or experience in architectural best practices in building data lakes.
  • Load and performance test data pipelines built using the above-mentioned technologies
  • Expert-level knowledge of using SQL to write complex, highly-optimized queries across large volumes of data.
  • Strong problem solving and troubleshooting skills with the ability to exercise mature judgment.

Requirements

  • At least 5 years of experience with AWS and Python programming, experience with Python frameworks (e.g. Django, Flask, Bottle)
  • Expert-level knowledge of using SQL to write complex, highly-optimized queries across large volumes of data.
  • Working experience on ETL pipeline implementation using AWS services such as Glue, Lambda, EMR, Athena, S3,SNS,Kinesis, Data-Pipelines, Pyspark

Apply for this position

Allowed Type(s): .pdf, .doc, .docx