Data Integration Engineer

Job Summary

We seek a Senior Data Engineer to design and optimize scalable data pipelines and processing solutions using Big Data, cloud, and AI technologies. Collaborate with architects, data scientists, and engineers to deliver high-performance, production-ready data systems.

Responsibilities

Design, develop, and maintain scalable batch and real-time data pipelines using Apache Spark/PySpark, SQL, and Python to support large-scale data processing

Develop and optimize ETL/ELT pipelines to ensure efficient data ingestion and transformation

Build data ingestion frameworks integrating databases, APIs, files, and streaming platforms for reliable data flow

Utilize AWS services including S3, Glue, EMR, Redshift, Kinesis, Lambda, and DynamoDB to deploy and manage cloud-based data solutions

Develop and optimize data processing workflows using Hadoop, Hive, Spark, Kafka, Cloudera, and Databricks for high throughput and low latency

Perform SQL and Spark performance tuning to enhance processing speed and resource utilization

Design and implement data models, data warehouses, and data lake architectures to support analytics and reporting needs

Develop CI/CD pipelines and manage containerized deployments using Docker, Kubernetes, and OpenShift to streamline development and operations

Integrate Generative AI and NLP capabilities into enterprise data applications to enhance data insights and automation

Develop solutions leveraging LLM frameworks, RAG, vector databases, and AI APIs to support advanced AI-driven data applications

Collaborate with solution architects, data scientists, software engineers, and business stakeholders to deliver scalable, production-ready data solutions

Participate in system design, development, testing, deployment, and provide production support to ensure system reliability and performance

Required competencies and certifications

Bachelor’s or Master’s degree in Computer Science, Information Technology, Engineering, or related field

Minimum 8 years of experience in Data Engineering / Big Data with expertise in large-scale data processing

Proficient programming skills in Python, Java, and/or Scala

Hands-on experience with Apache Spark/PySpark, SQL, Hadoop, and Hive

Experience working with AWS cloud platform and its data services

Knowledge of data warehouses, relational databases, and data lake technologies

Experience with Kafka, Airflow, Jenkins, Git, and CI/CD practices

Preferred competencies and qualifications

Experience with Docker, Kubernetes, or OpenShift container orchestration platforms

Familiarity with Databricks, Snowflake, and Cloudera platforms

Exposure to Generative AI, LLMs, RAG, LangChain/LangGraph, and vector databases

Strong analytical, problem-solving, and communication skills

#J-18808-Ljbffr