Senior Data Engineer

'Data Pipeline Development &Operations

. Design, build, and operate scalable and reliable data pipelines on theDatabricks platform

. Develop end-to-end data workflows from ingestion through transformation toconsumption

. Implement robust error handling, monitoring, and alerting mechanisms

. Ensure data pipeline reliability, performance, and maintainability

. Optimize pipeline performance through efficient Spark job design and clusterconfiguration

. Manage and orchestrate complex data workflows using Databricks Jobs andworkflows

Legacy Code Modernization

. Refactor legacy code and data pipelines to PySpark for improved performanceand scalability

. Migrate traditional ETL processes to modern ELT patterns on Databricks

. Assess existing codebases and identify opportunities for optimization andmodernization

. Ensure backward compatibility and data integrity during migration processes

. Document refactoring approaches and create migration playbooks

. Collaborate with stakeholders to minimize disruption during code transitions

Data Engineering Excellence

. Implement data quality checks and validation frameworks

. Design and maintain Delta Lake tables with appropriate optimizationstrategies

. Develop reusable code libraries and frameworks for common data engineeringtasks

. Follow software engineering best practices including version control,testing, and CI/CD

. Participate in code reviews and provide constructive feedback to teammembers

. Troubleshoot and resolve data pipeline issues in production environments

Collaboration & Knowledge Sharing

. Work closely with data architects, analysts, and business stakeholders

. Collaborate with Infrastructure (Infra), Applications (Apps), and Cyberteams

. Share knowledge and best practices with Team NCS

. Mentor junior data engineers on PySpark and Databricks technologies

. Document technical solutions and maintain comprehensive documentation' 'EssentialTechnical Skills

. Data Engineering: Strong foundation in data engineering principles, ETL/ELTprocesses, and data pipeline design patterns

. PySpark: Proven hands-on experience developing data pipelines using PySpark,including DataFrames API, Spark SQL, and performance optimization

. Databricks Platform: Practical experience with Databricks workspace, clustermanagement, notebooks, and job orchestration

. Workspace AI Agent: Knowledge of Databricks Workspace AI Agent capabilitiesand integration

. Data Modelling: Experience implementing data models including dimensionalmodeling, data vault, or lakehouse architectures

. Delta Lake: Understanding of Delta Lake features including ACIDtransactions, schema evolution, and optimization techniques

. Python: Strong Python programming skills for data processing and automation

Additional Technical Skills

. SQL proficiency for data querying and transformation

. Experience with cloud platforms (Azure, AWS, or GCP)

. Understanding of data governance and security best practices

. Knowledge of streaming data processing (Structured Streaming)

. Familiarity with DevOps practices and CI/CD pipelines

. Experience with version control systems (Git)

. Understanding of data quality frameworks and testing methodologies

Professional Experience

. Minimum 8 years in data engineering or related roles

. At least 2-3 years of hands-on experience with Databricks platform

. Proven track record of refactoring legacy code to modern frameworks

. Experience building and maintaining production data pipelines at scale

. Background working across multiple data sources and formats

. Experience in agile development environments

Required Certifications - mandatory to haveat least one certification

. Databricks Certified Data Engineer Associate OR Databricks Certified DataEngineer Professional

Additional Certifications (Preferred)

. Databricks Certified Associate Developer for Apache Spark

. Cloud platform certifications (Azure Data Engineer Associate, AWS CertifiedData Analytics, or Google Cloud Professional Data Engineer)

. Relevant data engineering or big data certifications

Soft Skills

. Strong problem-solving and analytical thinking abilities

. Excellent communication skills to explain technical concepts clearly

. Ability to work collaboratively in cross-functional teams

. Self-motivated with strong attention to detail

. Adaptable to changing priorities and technologies

. Client-focused mindset with commitment to quality delivery'

'Minimum 8 years and above ofexperience.