Data Engineer – Databricks / Lakehouse _ Contract

Data Engineer – Databricks / Lakehouse

Location: Alexandra Building

Contract Duration: 12 Months

Experience:

Minimum 6 years of relevant experience

About the Role

We are looking for an experienced

Data Engineer

to design, develop and operationalise enterprise-scale data platforms, Lakehouse solutions and data products.

The successful candidate will have strong hands-on experience in

Data Engineering, Databricks, Apache Spark, PySpark, Python and SQL , with exposure to modern Data Lake/Lakehouse architectures, cloud platforms and large-scale data ingestion and processing.

The role involves building scalable batch and streaming pipelines, developing reusable data products, implementing data quality frameworks, and supporting modern analytics, AI and GenAI use cases.

Key Responsibilities

Design, develop and operationalise enterprise

Data Lake, Lakehouse and data engineering platforms .

Develop scalable

batch, streaming, CDC and API-based data ingestion pipelines .

Build and maintain data pipelines using

Spark, PySpark, Python, SQL and related Big Data technologies .

Develop reusable foundation and business

data products , including appropriate data contracts, SLAs and data quality controls.

Implement data ingestion, transformation, reconciliation and data quality frameworks.

Work with modern Lakehouse and open table technologies such as

Delta Lake, Apache Iceberg and Apache Hudi .

Develop solutions for structured, semi-structured and unstructured data, including extraction and processing of content from different file formats.

Work with streaming and distributed data technologies such as

Kafka, Spark Streaming, Flink, Airflow, Hive, Trino or Dremio .

Design data architectures supporting

AI, NLP, RAG, vector search and GenAI/agentic applications .

Support ingestion, curation, governance and consumption of unstructured data for AI-driven analytics.

Expose data through APIs, event streams, dashboards and other enterprise consumption channels.

Perform performance tuning, troubleshooting, production support and root cause analysis.

Implement automated deployment and engineering practices using

Docker, Kubernetes/OpenShift and CI/CD pipelines .

Work closely with architecture, engineering, analytics and business teams across multiple projects.

Prepare technical documentation, deployment guides and operational runbooks.

Ensure solutions comply with engineering standards, security requirements, DevSecOps controls and software delivery practices.

Requirements

Bachelor's degree in

Computer Science, Engineering, Information Technology

or a related discipline.

Minimum

6 years of relevant experience

in Data Engineering, Big Data, Data Lake, Data Warehouse or Lakehouse implementations.

Strong hands-on experience with

Apache Spark, PySpark, Python and SQL .

Experience with one or more enterprise data platforms such as

Databricks, Snowflake, Cloudera, Azure, AWS or GCP .

Strong experience developing

ETL/ELT, data ingestion, transformation and data processing pipelines .

Experience working with Data Lake/Lakehouse architectures and technologies such as

Delta Lake, Iceberg or Hudi .

Experience with distributed and streaming technologies such as

Kafka, Spark Streaming, Flink, Hive or Airflow .

Good understanding of

data modelling, metadata management, data lineage, governance and data quality .

Experience with containerisation and DevOps technologies such as

Kubernetes, OpenShift, Docker, Jenkins, Git and CI/CD .

Strong troubleshooting, performance optimisation and root cause analysis skills.

Experience working in Agile environments and delivering enterprise-scale technology solutions.

Strong communication, collaboration and stakeholder management skills.

Good to Have

Experience developing

data products or enterprise data marketplace solutions .

Experience supporting

RAG, vector search, GenAI, NLP or AI/ML data pipelines .

Experience processing multimodal or unstructured data such as documents, images, audio and video.

Knowledge of

Scala or Java

for data engineering.

Experience with

MLflow, Spark MLlib, scikit-learn or XGBoost .

Experience developing internal engineering tools using

Python, shell scripting, Flask or React .

Exposure to

Teradata, Netezza, Greenplum or other MPP migration programmes .

Relevant certifications such as

Databricks Certified Data Engineer, Azure Data Engineer Associate, Google Professional Data Engineer, SnowPro or DAMA CDMP .

We look forward to your application!

#J-18808-Ljbffr