Python + PySpark - Junior Developer Job at Compunnel, Inc., New York, NY

ZWt2UWVza3BVWnRBK1ljZTJKV2pMV3RxU0E9PQ==
  • Compunnel, Inc.
  • New York, NY

Job Description

The Python + PySpark + AWS Consultant will be responsible for designing, developing, and optimizing data engineering solutions using Python and PySpark. The role requires hands-on experience in building scalable ETL pipelines, working with cloud platforms such as AWS or Azure, and ensuring data integrity and performance optimization. The candidate should have strong SQL skills and a deep understanding of data engineering principles. Key Responsibilities

  • Develop and maintain data processing workflows using Python and PySpark
  • Design and implement ETL pipelines for structured and unstructured data
  • Optimize Spark-based data processing for efficiency and scalability
  • Deploy and manage data solutions on AWS or Azure
  • Write and optimize SQL queries for data transformation and analysis
  • Troubleshoot and resolve performance issues in data pipelines
  • Work closely with cross-functional teams to ensure data reliability and integrity
Required Qualifications
  • 5+ years of experience in data engineering
  • Strong proficiency in Python and object-oriented programming
  • Hands-on experience with PySpark for large-scale data processing
  • Proficiency in SQL for data manipulation and query performance tuning
  • Experience with AWS or Azure for cloud-based data solutions
  • Knowledge of ETL processes and data pipeline automation
  • Experience with Hadoop is acceptable
Preferred Qualifications
  • Experience in optimizing Spark jobs for performance and cost efficiency
  • Familiarity with DevOps practices for data engineering
  • Understanding of data governance, security, and compliance best practices
#J-18808-Ljbffr Compunnel, Inc.

Job Tags

Similar Jobs

Comfort Nerds Heating Air and Home

HVAC Apprentice Job at Comfort Nerds Heating Air and Home

 ...Job Description Job Description We are looking for an apprentice for our Bay Area work. The job consists of 2-4 Days a week. Wage will be DOE. Lodging is provided. We are looking for an individual with an upbeat attitude, hardworking, and responsible. inquire for... 

Mellow Mushroom Southside

Pizza Maker Job at Mellow Mushroom Southside

 ...peppers, onions and mushrooms) Monitor food stock Use less traditional ingredients to appeal to a larger audience (for example, vegan cheese and gluten-free flour) Cook appetizers, for example: bruschetta, pretzel bites, chicken wings, garlic breads and salads... 

CyberCoders

Associate Attorney Job at CyberCoders

 ...Job Description Job Description Associate Attorney We are a nationally recognized litigation firm with offices in 16 cities and experience handling cases in all 50 states. We focus on representing corporate defendants in complex, high-stakes matters and are known... 

The People Co.

Quality Assurance Manager Job at The People Co.

 ...seeking a QA Manager to be responsible for leading the QA team members with various inspections and activities in relation to the quality of product processed throughout the facility. This talented QA Manager with have extensive HACCP management, product specifications... 

Plastipak Holdings

Yard Driver Job at Plastipak Holdings

 ...Career Growth: We prioritize promotions from within, giving you a clear path for advancement. You Will Moves trailers from yard area into assigned docks and then secures/locks trailer to assure no movement until released. Drops trailers into assigned yard areas...