Data Engineer (AWS, Databricks)

Role and Responsibilities

Operational Responsibilities

Monitor and maintain production data pipelines to ensure 99.9% uptime and optimal performance

Implement comprehensive logging, alerting, and monitoring systems using Application monitoring tools

Perform regular health checks performance, job execution times, and resource utilization to identify and resolve bottlenecks proactively

Manage incident response procedures for pipeline failures, including root cause analysis, resolution, and post-incident reviews

Establish and maintain disaster recovery procedures and backup strategies for critical data assets within the Databricks environment

Conduct regular performance tuning of Spark jobs and Databricks cluster configurations to optimize cost and execution efficiency

Maintain comprehensive documentation for operational procedures, runbooks, and troubleshooting guides

Coordinate scheduled maintenance windows and system upgrades with minimal business impact

Manage user access controls, workspace configurations, and security policies within Application environments

Requirements / Qualifications

Education & Experience

Degree in Computer Science or Computer Engineering

Minimum 5 years working experience in system operations compliance and management areas

Project hands-on experience specifically with AWS platform (primary requirement)

Project experience in cloud operations or cloud architecture

Must be cloud certified (AWS)

Core Technical Skills

Proficiency in Databricks platform, including workspace management, cluster configuration, and job orchestration

Strong expertise in Apache Spark within Databricks environment, including Spark SQL, DataFrames, and RDDs

Good in-depth understanding of data warehouse concepts, data profiling, data verification and advanced analytics techniques

Strong knowledge of monitoring, incident management, and cloud cost control

Technology Stack Experience

Databricks

AWS cloud services and architecture

IDMC (Informatica Data Management Cloud)

Tableau for data visualization

Oracle Database management

ML Ops practices within Databricks environment

STATA for statistical analysis is advantage

Amazon SageMaker integration with Databricks

DataRobot platform integration

Soft Skills & Stakeholder Management

Good interpersonal skills with the ability to work with different groups of stakeholders

Strong problem-solving skills and ability to work independently in a fast-paced environment with minimal supervision

Excellent communication skills for technical documentation and cross team collaboration

Desirable Requirements

AWS certification (Associate or Professional level) - highly preferred

Exposure to hospital information/clinical systems is an added advantage

Understanding of DevOps practices and CI/CD pipelines for Databricks based data engineering projects

#J-18808-Ljbffr