Location: Singapore
Job Function: Technology Group
Job Type: Permanent
Req ID: 17626
Recruiter: Yvonne Loh
GIC is one of the world’s largest sovereign wealth funds. With over 2,000 employees across 11 locations around the world, we invest in more than 40 countries globally across asset classes and businesses. Working at GIC gives you exposure to an extraordinary network of the world’s industry leaders. As a leading global long-term investor, we Work at the Point of Impact for Singapore’s financial future, and the communities we invest in worldwide.
Technology Group
We experiment, design, and lead a 24×7 global business where we support core capabilities in asset management, trading, investment operations, and risk management. We deliver secure, reliable, and integrated solutions, and provide insights on new, and emerging technologies.
Infrastructure & Cybersecurity Resilience (ICR)
We design, build, and secure the technology foundations that power GIC’s global investment operations. We aim to deliver resilient, scalable, and secure infrastructure that empowers our people and businesses to perform securely, efficiently, and effectively.
What impact will you make in this role?
As a VP, Site Reliability & Observability Engineer you will be the hands-on subject matter expert (SME) who governs, designs, and engineers the enterprise observability estate across infrastructure, applications, and AI/ML systems. You will drive proactive monitoring, automation, and resilience using Datadog, Dynatrace, Arize (ML/LLM observability), and the AWS and Azure platforms - ensuring operational stability, risk control, and compliance with financial regulatory standards.
This is a deep hands-on, individual-contributor SME role - not a people-management position. You will personally instrument services, build telemetry pipelines, write automation, and engineer the platform as code. You bring an AI-first mindset, using AI and agentic-AI solutions to operate, investigate, and automate observability - and you also observe the AI itself, monitoring model and LLM behaviour in production. You will define and apply SRE strategy - SLIs, SLOs, and error budgets - and build automation that reduces MTTD and MTTR and strengthens operational resilience.
What will you do as a Site Reliability & Observability Engineer?
Observability Engineering & Governance
- Design and own the enterprise observability architecture aligned with operational resilience mandates (MAS TRM, DORA, APRA CPS 230).
- Deploy, instrument, and optimize observability platforms - Datadog, Dynatrace, Splunk - for full-stack visibility (infra, application, network, and user experience), hands-on.
- Establish telemetry governance for metrics, logs, and traces (incl. OpenTelemetry) - consistency, retention compliance, and security controls - implemented as code.
- Integrate observability with incident management, ITSM, and AIOps for predictive alerting and anomaly detection.
AI/ML & LLM Observability (Arize)
- Engineer observability for AI/ML and LLM systems using Arize - monitoring model performance, data drift, feature drift, prediction quality, bias, and LLM response quality in production.
- Build evaluation, tracing, and guardrail monitoring for LLM and agentic-AI applications (prompt/response tracing, hallucination and toxicity signals, latency and cost).
- Define SLIs/SLOs and alerting for model and LLM health, and integrate ML observability into the broader reliability and incident-management workflow.
- Partner with Data Science and ML Engineering to close the loop from production signals back into retraining and model governance.
AI-First Automation & Agentic AI Engineering
- Adopt an AI-first operating model: use AI copilots, LLMs, and agentic-AI workflows (e.g., Python, LangGraph) to operate, investigate, and automate the observability platform through code.
- Build agentic-AI investigators that correlate telemetry, enrich context, triage alerts, run guided diagnostics, and draft incident timelines and root-cause hypotheses.
- Apply AI/ML for anomaly detection, intelligent event correlation and deduplication, alert-noise reduction, and predictive alerting across infrastructure and applications.
- Engineer closed-loop, AI-assisted self-healing and auto-remediation integrated with runbooks and IaC, with explicit guardrails, blast-radius limits, and human-in-the-loop controls for high-risk actions.
- Treat telemetry pipelines, automation, and AI agents as governed products - version-controlled, tested, observable, secure, and auditable.
Reliability Engineering & SRE Strategy
- Define and apply SRE strategy for infrastructure and business-critical applications - SLIs, SLOs, error-budget policies, and reliability scorecards for key production services.
- Design and automate runbooks, alerts, self-healing actions, and auto-remediation via Python, Ansible, and Terraform.
- Reduce MTTD/MTTR, incident recurrence, and manual intervention through observability- and AI-led automation; track toil reduction explicitly.
- Conduct resilience testing, chaos engineering, and capacity validation aligned with business-continuity standards.
- Partner hands-on with Application, Infrastructure, and Cyber teams to codify reliability into the delivery lifecycle.
Cloud Observability & Platform Engineering
- Architect and manage observability for cloud-native workloads in AWS and Azure (CloudWatch, X-Ray, CloudTrail; Monitor, Log Analytics, App Insights) across compute, storage, and network layers.
- Integrate cloud observability into landing zones and CI/CD pipelines for continuous compliance with deployment controls.
- Implement infrastructure-as-code using Terraform and Ansible for consistent, auditable provisioning.
Operational Excellence & Technical Enablement
- Serve as the highest escalation point for complex incidents and outages; perform deep-dive RCA and engineer long-term preventive solutions; drive blameless post-mortems.
- Partner with Service Delivery, Cyber, and Application teams for predictive incident prevention and root-cause transparency.
- Deliver executive dashboards highlighting availability, reliability KPIs, and operational-risk indicators.
- Create reusable instrumentation, patterns, and runbooks; uplift peers on observability, SRE, and AI-assisted automation.
What makes you a successful candidate?
- Bachelor’s or Master’s degree in Computer Science, Engineering, or a related discipline.
- 12+ years in Infrastructure, Cloud, or SRE roles, with 5+ years hands-on in an SRE/observability SME capacity in financial institutions or regulated environments.
Proven hands-on expertise in:
- Observability Platforms: Datadog, Dynatrace, Splunk, ELK; OpenTelemetry instrumentation.
- AI/ML Observability: Arize or equivalent ML/LLM observability - model/LLM monitoring, drift, and evaluation.
- Automation / IaC: Terraform, Ansible, Python, and CI/CD tooling.
- AI / Agentic AI: Practical use of LLMs and agentic-AI frameworks (e.g., Python, LangGraph) to build operational agents, investigations, and auto-remediation.
- Cloud Platforms: AWS (CloudWatch, X-Ray, CloudTrail), Azure (Monitor, Log Analytics, App Insights).
- Deep understanding of SRE strategy - service-health modelling, SLIs/SLOs, error budgets, and auto-remediation design.
- Proven architectural and design skills for scalable, resilient, high-performance observability platforms.
- Strong analytical and deep-dive troubleshooting skills, with a bias toward long-term preventive, engineered solutions.
- Familiarity with financial-sector operational resilience frameworks, regulatory compliance, and incident governance.
Certifications Required:
- Datadog Certified Observability Professional / Dynatrace Certified Associate (or higher)
- Terraform / Ansible / Python certification
Highly desirable:
- AWS Certified DevOps Engineer / Azure DevOps Engineer Expert
- SRE Foundation / Practitioner (DevOps Institute)
- ITIL v4
Key Responsibility Areas
- Architecture & Design - Design and evolve highly scalable, resilient, performant observability leveraging Datadog, Dynatrace, Splunk, and Arize; define best practices and patterns as code.
- AI/ML & LLM Observability - Monitor model/LLM health with Arize - drift, quality, bias, hallucination, latency, cost - and integrate into reliability and incident workflows.
- AI-First Automation - Use AI, LLMs, and agentic-AI workflows (Python/LangGraph) to operate and automate the platform via code - investigation, correlation, predictive alerting, and guarded auto-remediation.
- Reliability (SRE) Strategy - Define SLIs/SLOs and error-budget policies; reduce MTTD/MTTR, incident recurrence, and toil through engineered automation.
- Troubleshooting & Incidents - Highest escalation point for complex incidents and outages; deep-dive RCA; drive blameless post-mortems and preventive fixes
- Performance & Capacity - Identify bottlenecks, optimize resources, and forecast capacity for continued reliability and efficiency.
- Automation & Tooling (IaC) - Build instrumentation, runbooks, and self-healing via Python/Ansible/Terraform following IaC principles - version-controlled and auditable.
- Security & Compliance - Enforce telemetry governance, retention, access controls, and audit logging; ensure compliance with MAS TRM, DORA, CPS 230, and internal policy
Work at the Point of Impact
We need to be forward-looking to attract the right people to help us become the Leading Global Long-term Investor. Join our ambitious, agile, and diverse teams - be empowered to push boundaries and pursue innovative ideas, share your views, and be heard. Be anchored on our PRIME Values: Prudence, Respect, Integrity, Merit and Excellence, which guides us in how we make our day-to-day decisions. We strive to inspire. To make an impact.
GIC is a Great Place to Work
At GIC, our offices are vibrant hubs for ideation, professional growth, and interpersonal connection. At the same time, we believe that flexibility allows us to do our best work and be our best selves. Thus, our teams come into the office four days per week to harness the benefits of in-person collaboration, but have the flexibility to choose which days they work from home and adjust this arrangement as situational needs arise.
GIC is an equal opportunity employer
As an employer, we passionately believe every individual brings with them unique diversity of thought and perspectives to meaningfully enrich perspectives of GIC teams to drive competitive performance. An inclusive environment yields exceptional contribution.
Learn more about our Technology Group here:
https://gic.careers/group/technology-group/