Observability Engineer (Public Sector)

1-year contract, renewable

Government project

Hybrid work arrangement

We are looking for an Observability Engineer to help build and operate the observability capabilities of the future SSOE platform. You will help provide end-to-end visibility across MOE's technology environment, spanning on-premises infrastructure, networks, applications, cloud platforms, and hybrid environments. You will enable engineering and operations teams to understand system health, identify issues early, diagnose incidents quickly, and continuously improve service reliability.

What You Will Be Working On

As an Observability Engineer, you will establish and operate consistent observability capabilities across SSOE infrastructure and applications.

You will work across metrics, events, logs, and traces to provide a unified view of service health and performance. You will define instrumentation standards, service-level indicators and objectives, alerting strategies, dashboards, and operational health signals.

You will work closely with application, infrastructure, network, security, and platform teams to ensure observability is built into services from the outset rather than added after deployment.

Key Responsibilities

Observability Engineering

Design and operate end-to-end observability across on-premise infrastructure, networks, applications, cloud platforms, and hybrid environments

Collect, aggregate, and correlate metrics, events, logs, and traces across infrastructure and application workloads

Define and maintain observability standards that work consistently across legacy, on-premise, containerised, and cloud-native systems

Establish application and infrastructure instrumentation standards using OpenTelemetry and other appropriate technologies

Support engineering teams with instrumentation, SDK, agent, and telemetry integration

Define common conventions for service naming, metadata, tagging, correlation IDs, and telemetry enrichment

Identify observability gaps and continuously improve end-to-end visibility across SSOE services

Service Reliability & Monitoring

Define SLIs, SLOs, alerting rules, and service health indicators for critical services

Build operational dashboards covering infrastructure health, application performance, user experience, availability, capacity, and service reliability

Develop leadership-level views that provide meaningful visibility into service performance and operational trends

Design actionable alerting that enables teams to identify and respond to issues while minimising unnecessary alert noise

Establish monitoring and operational-readiness requirements for new applications, infrastructure, and platform components

Use observability data to support capacity planning, performance analysis, reliability improvements, and operational decision-making

Telemetry & Integration

Define secure telemetry collection and routing across on-premise environments, GCC, cloud platforms, and approved SaaS services

Work with infrastructure and platform teams to integrate telemetry from servers, network devices, applications, containers, databases, and managed cloud services

Design observability approaches that account for network boundaries, security zones, data residency, and connectivity constraints

Define telemetry retention, lifecycle, and cost-management requirements

Ensure logs, metrics, and traces can be correlated across distributed and hybrid systems

Incident Management & Continuous Improvement

Support operational teams during incidents by using observability data to identify symptoms, dependencies, and potential root causes

Participate in incident investigation, root-cause analysis, and post-incident reviews

Identify recurring operational issues and recommend improvements to instrumentation, alerting, architecture, or operational processes

Define appropriate SLOs and operational health indicators for observability services

Participate in operational support and on-call responsibilities for owned services

Maintain architecture documentation, operational procedures, and runbooks

What We Are Looking For

Experience

Minimum

3-5 years of experience

in observability engineering, Site Reliability Engineering (SRE), platform engineering, infrastructure engineering, or a related discipline

At least

2 years of hands-on experience

implementing or operating observability and monitoring capabilities in production environments

Demonstrated experience working with

metrics, logging, tracing, dashboards, alerting, and incident troubleshooting

Experience monitoring and supporting

production infrastructure, applications, or distributed systems

Experience working with

on-premise and/or cloud environments , with an understanding of hybrid infrastructure

Experience working with engineering or operations teams to implement instrumentation and improve service reliability

Technical Skills

Observability: Dynatrace, Elastic, Grafana, Prometheus or equivalent

Instrumentation: OpenTelemetry

Cloud: AWS-native observability and monitoring capabilities

Infrastructure: Docker, ECS, CI/CD, SHIP-HATS, Terraform / OpenTofu, Ansible

Data & Storage: Cloud-native storage and telemetry data services

Event & Telemetry: Kafka, MQ, event-driven telemetry patterns

Engineering Practices

Treats observability configuration, instrumentation, dashboards, and platform components as version-controlled engineering artefacts

Uses automation and Infrastructure as Code for repeatable and auditable deployments

Designs observability for reliability, scalability, security, and operational sustainability

Understands the difference between collecting telemetry and creating useful operational signals

Designs monitoring and alerting around service and user impact rather than individual infrastructure metrics alone

Builds observability into services from the beginning of the engineering lifecycle

Uses code review, testing, and CI/CD for observability-related changes where appropriate

Collaborates effectively across application, infrastructure, network, security, data, and platform teams

Security & Governance

Apply MOE and Government data-classification requirements

Ensure telemetry is collected, transmitted, stored, and accessed according to applicable security and data-residency requirements

Prevent sensitive information, credentials, and secrets from being unnecessarily captured in telemetry

Implement appropriate access controls for observability platforms and operational data

Maintain auditability and traceability of observability configuration and operational activities

Participate in security, architecture, and operational-readiness reviews

Nice to Have

Experience with Singapore Government platforms such as TechPass, SHIP-HATS, SEED, and GCC

Familiarity with OC/SN data-classification requirements

AWS or Azure cloud certifications

Experience implementing OpenTelemetry at scale

Experience operating observability platforms across large or distributed environments

Experience monitoring hybrid infrastructure spanning data centres and cloud environments

Familiarity with SRE practices such as error budgets, SLO management, incident response, and reliability engineering