Software Quality Assurance Engineer – GenAI / LLM

Primary Focus:

Software Quality Assurance / Software Testing / Test Automation - GenAI, LLM & Agentic AI

Secondary Exposure:

Solution Analysis / Technology Solution Design / Enterprise Integration

Domain / Project:

Global Markets, Capital Markets Banking Technology & Market Risk Technology

Role Overview

We are looking for a

Senior GenAI Quality Engineer / Solution Analyst

to design, analyse, test and validate

production-grade Generative AI (GenAI), Large Language Model (LLM), RAG and Agentic AI applications

within a complex enterprise environment.

This is

not a traditional manual QA or software testing role .

The role combines:

Software Quality Engineering

GenAI / LLM Testing & Evaluation

Agentic AI / AI Agent Testing

UI & API Testing

Test Automation

Solution Analysis

Enterprise Integration Testing

Observability & Troubleshooting

You will work across

discovery, solution design, development, testing and release , translating business requirements into clear application behaviours and validating end-to-end application quality across user interfaces, APIs, data flows, LLMs, RAG components, AI agents and enterprise integrations.

Key Responsibilities

GenAI / LLM Quality Engineering

Define and execute

end-to-end quality engineering and test strategies

covering:

Web / UI workflows

REST APIs

Backend services

Enterprise integrations

GenAI applications

LLM workflows

RAG pipelines

Agentic AI / AI Agent interfaces

Perform

GenAI / LLM testing and evaluation

covering:

Response quality

Task completion

Grounding

Faithfulness

Relevance

Consistency

Citation accuracy

Hallucination risk

Safe failure behaviour

Test non-deterministic / probabilistic AI systems using:

Evaluation datasets

Repeat testing

Quality thresholds

Acceptance criteria

Regression evaluation

Validate

RAG / Retrieval-Augmented Generation

solutions, including retrieval quality, grounding and response accuracy.

Agentic AI / AI Agent Testing

Test end-to-end

Agentic AI and AI Agent workflows , including:

Multi-turn conversations

Context handling

Agent planning

Tool selection

Tool calling / function calling

Tool inputs and outputs

State transitions

Memory and state

Human-in-the-loop approvals

Handoffs

Retries

Timeouts

Fallback behaviour

Error recovery

Termination conditions

Partial failures

Validate that AI agents behave correctly across both successful and failure scenarios.

Software & API Quality Engineering

Perform:

Functional Testing

Integration Testing

API Testing

Regression Testing

Exploratory Testing

Negative Testing

Resilience Testing

Basic Performance Testing

End-to-End Testing

Design comprehensive

REST API tests

covering:

API contracts

Authentication

Authorisation

Input validation

Error handling

Idempotency

Rate limits

Downstream system failures

Test web application behaviour across browsers and realistic end-user journeys, including:

Loading states

Interrupted sessions

Error messages

Feedback capture

Accessibility fundamentals

Test Automation

Develop and maintain

risk-based test automation

that reduces:

Regression testing time

Manual testing effort

Release cycle time

Production risk

Use automation frameworks and tools such as:

Playwright

Cypress

Selenium

pytest

REST Assured

Postman

Equivalent UI / API automation frameworks

Apply pragmatic automation principles by prioritising stable, high-value and frequently executed test scenarios.

GenAI Evaluation & AI Safety Testing

Validate LLM and GenAI applications for:

Grounded responses

Hallucinations

Retrieval quality

Citation accuracy

Prompt behaviour

Prompt injection

Unsupported requests

Restricted content handling

Safe failure behaviour

Adversarial scenarios

Support

AI evaluation / LLM evaluation

using appropriate evaluation datasets, quality metrics and repeatable evaluation approaches.

Exposure to

AI Red Teaming / Adversarial Testing

would be advantageous.

Observability & Troubleshooting

Use application and GenAI observability to identify the source of defects across:

Application

LLM / Model

RAG / Retrieval

Data

API / Integration

Platform

Analyse:

Logs

Distributed traces

API requests / responses

Payloads

Network calls

Database records

Agent execution traces

Exposure to observability and LLM evaluation tools such as:

Langfuse

LangSmith

OpenTelemetry

Elastic / Elasticsearch

Splunk

is advantageous.

Solution Analysis & Design

The role also acts as a hands‑on

Solution Analyst

for GenAI applications.

Responsibilities include:

Partner with product owners, business users, architects, engineers and GenAI specialists during discovery and solution design.

Analyse proposed GenAI use cases and determine whether the requirement should use:

Conventional application logic

Deterministic business rules

Search / retrieval

RAG

Workflow automation

Agentic AI

Human approval

Translate business requirements into:

Functional requirements

End-to-end solution flows

User journeys

Acceptance criteria

Interface behaviour

Decision rules

Non-functional requirements

Map interactions across:

User Interfaces

APIs

LLMs / Models

Prompts

RAG / Retrieval components

Enterprise data sources

AI Agent tools

Downstream enterprise systems

Analyse solution design trade-offs involving:

Quality

Complexity

Cost

Latency

Security

Data access

Maintainability

Operational risk

Identify missing controls, integration assumptions, ownership gaps, failure scenarios and operational risks before development begins.

Support the design of:

Human-in-the-loop approval

Fallback flows

Escalation

Exception handling

Solution Documentation

Produce practical technical and functional artefacts including:

Process Flows

Sequence Diagrams

Context Diagrams

Interface Specifications

Decision Tables

User Stories

Acceptance Criteria

Test Scenarios

Traceability Documentation

Maintain traceability across:

Business Requirement → Solution Design → Implementation → Test / Evaluation Scenario → Release Evidence

Release Quality & Governance

Create and maintain:

Test scenarios

Test datasets

Reusable regression scenarios

Test evidence

Defect reports

Quality metrics

Release quality reports

Provide evidence-based release recommendations identifying:

Known defects

Known limitations

Residual risks

Quality concerns

Areas requiring production monitoring

Core Requirements

Experience

5-8 years of experience

in Software Quality Engineering, Test Engineering, Test Automation, SDET or similar hands‑on software testing roles.

Strong experience testing complex enterprise applications.

Strong experience testing:

Web applications

REST APIs

Backend services

Enterprise integrations

Test Automation / Programming

Hands‑on experience with one or more of:

Playwright

Cypress

Selenium

pytest

REST Assured

Postman

Equivalent automation frameworks

Working programming knowledge of:

Python

Java

JavaScript

TypeScript

Candidates should be capable of developing, reviewing and troubleshooting test automation.

Software Engineering / DevOps

Experience with:

Git

Pull Requests

CI/CD

Automated Testing

Test Reporting

Defect Management

Experience validating distributed systems including:

Asynchronous Processing

Queues

Batch Processing

APIs

Downstream Dependencies

Enterprise Integrations

GenAI / LLM Requirements

Practical understanding of:

Generative AI / GenAI

Large Language Models / LLM

LLM Evaluation

LLM Testing

Retrieval-Augmented Generation / RAG

RAG Evaluation

Agentic AI

AI Agents

Multi-Agent Workflows

Prompts / Prompt Engineering

Context Windows

Embeddings

Tool Calling

Agent Memory & State

LLM Observability

Candidates should understand how GenAI applications differ from conventional deterministic software and how to validate probabilistic AI behaviour.

Security & Risk Testing

Understanding of software and GenAI security fundamentals including:

Access Control

Authentication / Authorisation

Sensitive Data Handling

Input Validation

Auditability

Prompt Injection

AI Safety Testing

Adversarial Testing

Nice to Have

Experience with:

Banking / Financial Services

Regulated enterprise environments

Contract Testing

Service Virtualisation

Synthetic Monitoring

Performance Testing

AI Red Teaming

Accessibility Testing / WCAG

Kubernetes

OpenShift

AWS

Containerised Application Deployment

Key Domain / Technical Skills

1. Software Quality Engineering, API Testing & Test Automation

2. GenAI / LLM Evaluation, RAG & Agentic AI Testing

3. Solution Analysis, Observability & Enterprise Integration

Key Search Keywords

GenAI Quality Engineer,AI Quality Engineer,LLM Quality Engineer,Generative AI Testing,GenAI Testing,LLM Testing,LLM Evaluation,AI Evaluation,Agentic AI Testing,AI Agent Testing,RAG Testing,RAG Evaluation,Retrieval-Augmented Generation,Software Quality Engineering,Quality Engineering,Software QA,Test Automation,SDET,Automation Testing,API Testing,REST API Testing,UI Testing,Integration Testing,Regression Testing,End-to-End Testing,Playwright,Cypress,Selenium,pytest,REST Assured,Postman,Python,Java,JavaScript,TypeScript,CI/CD,Git,Prompt Testing,Prompt Injection,Hallucination Testing,Grounding,Faithfulness,AI Safety Testing,Adversarial Testing,AI Red Teaming,Langfuse,LangSmith,OpenTelemetry,Elastic,Splunk,Observability,Distributed Systems,Kubernetes,OpenShift,Solution Analysis

#J-18808-Ljbffr