Data Engineer (AWS)

Key Responsibilities Architecture & Design

Design and architect the end-to-end AWS Data Lake and Lakehouse solution, including Landing Zone, Transformed Zone, and Curated/Consumption Zone layers

Define and govern data architecture standards, patterns, and best practices across the platform

Architect reusable data ingestion pipelines supporting REST APIs, JDBC databases, S3 file uploads, and SaaS connectors (e.g. Salesforce via AWS AppFlow)

Design data storage strategies including hot, warm, and cold storage tiers, encryption, and data lifecycle policies

Development & Deployment

Develop and deploy data ingestion pipelines using AWS Glue, Lambda, Step Functions, EventBridge, and API Gateway

Build and maintain data transformation workflows (batch and stream processing) using AWS Glue and Amazon Redshift

Implement orchestration, monitoring, logging, and notification frameworks for pipeline operations

Develop and maintain the AWS Glue Data Catalogue, including schema evolution tracking and metadata tagging

Security & Governance

Configure and enforce data security policies using AWS Lake Formation, IAM, and Secrets Manager

Implement granular access controls at database, table, and column levels

Ensure compliance with data classification, retention, and audit requirements

Support data quality frameworks and observability monitoring

Maintenance & Operations

Monitor platform health, performance, and pipeline reliability

Troubleshoot and resolve data pipeline failures and data quality issues

Maintain documentation for architecture decisions, pipeline configurations, and operational runbooks

Continuously optimise platform performance and cost efficiency on AWS

Requirements Essential

Minimum 3 to 5 years of experience in data engineering, data architecture, or cloud infrastructure roles

Hands-on expertise with core AWS data services: Amazon S3, AWS Glue, Amazon Redshift, AWS Lambda, Amazon Kinesis, AWS Step Functions, Amazon EventBridge, AWS AppFlow, AWS Lake Formation

Strong proficiency in SQL and at least one scripting language (Python or Scala)

Experience designing and implementing Data Lake or Lakehouse architectures

Solid understanding of data governance, data cataloguing, and metadata management

Experience with batch and streaming data processing patterns

AWS Certified Data Engineer - Associate or AWS Certified Solutions Architect certification (or equivalent)

Preferred

Experience integrating with Tableau or similar BI visualisation tools via Amazon Redshift or S3

Familiarity with MLOps frameworks and AI/ML model deployment on AWS SageMaker

Experience with Salesforce data integration using AWS AppFlow

Knowledge of Change Data Capture (CDC) and incremental data load patterns

Prior experience in a government or public sector data environment