Data Engineer (Azure, Databrick)

Key Responsibilities

Data Management & Transformation

Design, develop and maintain data pipelines that extract data from business applications, databases and external systems.

Implement data quality checks and validation processes to ensure data accuracy, consistency and completeness.

Perform data cleansing, aggregation, filtering, transformation and enrichment.

Establish and maintain governance frameworks for data and algorithms used in analytics, automated decision-making and analytical applications.

Develop and maintain logical and physical data models, including data structures, relationships and constraints.

Ensure data solutions comply with security, governance and organisational best practices.

Data Platform Optimisation & Automation

Develop and maintain

CI/CD pipelines

for data deployments and cloud resource provisioning.

Optimise data pipelines and processing workflows for performance, scalability and efficiency.

Optimise data models and algorithms for data quality, security, governance, performance and scalability.

Monitor data warehouse and ETL platform capacity, including capacity planning and forecasting.

Monitor and tune data platforms, troubleshoot performance bottlenecks and implement optimisation techniques such as caching and indexing.

Monitor platform resource and credit consumption and perform regular housekeeping activities.

Support the deployment, integration and maintenance of

AI/ML solutions

within the data platform.

Collaboration & Business Enablement

Partner with the Data Lead and business users to manage and maximise the value of data as a business asset.

Work with stakeholders to understand data requirements and translate them into scalable technical solutions.

Guide business users in the creation and maintenance of reports and dashboards.

Collaborate with technology, data and business teams to improve data quality, accessibility and usability.

Any other ad-hoc duties as assigned by supervisor

Requirements

5+ years of experience

in data management or data engineering, covering areas such as data integration, data modelling, data quality and platform optimisation.

Hands-on experience designing and implementing data architectures using Azure and Databricks.

Strong programming skills in

Python, PySpark and SQL .

Strong experience with

relational and non-relational databases , including SQL and NoSQL databases.

Hands-on experience with

Azure DevOps

and

Databricks LakeFlow Jobs , including version control, CI/CD and pipeline deployment.

Experience with data catalogue and governance tools such as

Databricks Unity Catalog

and

Microsoft Purview .

Experience supporting

AI/ML model deployment, feature engineering and production inference

is an advantage.

#J-18808-Ljbffr