Senior Data Engineer

Job Summary

We are looking for a Senior Data Engineer with strong experience designing and delivering scalable data platforms across Microsoft Azure and AWS ecosystems. The candidate will be responsible for building high-performance data pipelines, data lakes, and real-time data ingestion frameworks using PySpark and Databricks .

Key Responsibilities

Design and develop scalable data platforms on Azure and AWS.

Build and optimize high-performance ETL/ELT pipelines using PySpark and Databricks.

Develop and manage cloud-based data lakes and data warehouses .

Design real-time and batch data ingestion frameworks.

Implement data processing solutions using Apache Spark/PySpark .

Optimize data pipelines for performance, scalability, and reliability.

Work with cloud services such as Azure Data Factory, ADLS, Azure Synapse, AWS S3, Glue, and EMR .

Implement data quality, security, monitoring, and governance standards.

Collaborate with Data Architects, Engineers, and business stakeholders.

Required Skills

5+ years of experience in Data Engineering.

Strong hands-on experience with Azure and/or AWS data platforms.

Excellent PySpark and Databricks experience.

Strong knowledge of Data Lakes, ETL/ELT, and real-time data ingestion .

Experience with SQL and Python .

Knowledge of Apache Spark and distributed data processing.

Experience with cloud data services and data architecture.

Good understanding of CI/CD, Git, and Agile methodologies.

Preferred Skills

Azure Data Factory / Azure Synapse / ADLS

AWS S3 / Glue / EMR

Kafka or other real-time streaming technologies

Delta Lake

Terraform / Infrastructure as Code

Data governance and security