Job Description:

As a LLM / AI Quality Engineer, lead the end-to-end evaluation of AI applications

- LLM features, RAG systems, and multi-agent workflows

- To ensure they meet business outcomes, safety requirements, and platform standards. Own test design, execution, and reporting across offline, pre-prod, and in-prod stages, integrating with CI/CD and working closely with product, data, and platform teams.

1) AI/LLM Evaluation & Test Design - Define evaluation strategies (golden sets, adversarial suites, regressions), pass/fail gates, and SLOs for quality, safety, latency, and cost. - Establish rubric-based human reviews (usefulness, faithfulness, safety, clarity) and calibrate annotators. - Instrument LLM-as-judge where appropriate with calibration and spot checks.

2) RAG, Retrieval, & Grounding - Measure retrieval precision/recall, MRR/nDCG, and answer faithfulness to sources; detect hallucination and citation errors. - Test chunking, prompt templates, filters, and policy chains; monitor stale/poisoned content.

3) Agentic & Tool-Use Scenarios - Validate multi-step plans, tool selection, error recovery, retries, and idempotency for functions with side effects. - Contract-test JSON schemas and structured outputs across services.

4) Non-Functional, Performance & Cost - Run token-aware load/soak tests (context length, temperature, batching); track p50/p95/p99, throughput, timeouts, cache hit rate, and cost per successful task. - Recommend optimizations (prompt/policy changes, retrieval tweaks, caching).

5) Security, Privacy & Safety - Red-team for prompt injection, data exfiltration, indirect injections via retrieved content; validate guardrails pre/post inference. - Enforce PII controls, data-residency, and compliance checks; align with organizational security testing practices.

6) Observability & CI/CD Integration - Implement prompt/dataset/version lineage and trace-based evals; automate in CI (pre-merge golden tests, nightly adversarials) with canary/A-B in prod and rollback criteria. - Produce clear, decision-ready reports with risk assessments and release recommendations.

7) Project Delivery & Collaboration - Analyze requirements, enhance test plans with additional cases, prepare environments (including cloud), execute tests per plan, and drive defect resolution. - Provide regular status updates; manage test activities to schedule; support SIT/UAT and production readiness.

8) Performance, API & Platform Testing (Carry-over) - Execute API, performance, and load testing for microservices/web services that underpin AI features; integrate automated testing into CI/CD.

9) Team & Standards - Adopt and improve test standards/methodology; share practices, train teams, participate in peer reviews, and pursue self-directed learning.

Qualifications

The ideal candidate should possess:

- 3+ years in software testing/QA with strong test methodology and tooling; hands-on API testing and performance testing.

- Programming familiarity (e.g., Python/TypeScript) and experience with CI/CD and version control.

- Cloud basics (AWS/Azure/GCP) and microservices fundamentals.

- Degree/Diploma in CS/IT or equivalent.

- Preferred (AI/ML Focus)

- Understanding of ML concepts and MLOps; experience with model validation and monitoring in production.

- Experience with AI-specific security testing and vulnerability assessment.

- Familiarity with evaluation/observability tools (any of): LangSmith, Weights & Biases, RAGAS, TruLens, Promptfoo, DeepEval, Guardrails/LlamaGuard, Presidio; plus OpenTelemetry-style LLM traces.

- Practical exposure to Azure OpenAI/Bedrock/Vertex and model gateways; quota & token accounting know-how.

Tooling & Automation

- Modern automation frameworks (e.g., Playwright, Cypress, Selenium), API test tools (Postman/REST Assured), performance tools (k6/JMeter), and CI/CD integration.

- Data evaluation pipelines for RAG (embedding validation, filtering, drift detection).

- Traits

- Outcome-oriented, high standards; strong communication and collaboration; customer-focused; proficient in written and spoken English.

- Telco Context (Nice-to-Have)

- Experience testing copilots/agents for BSS/OSS, NOC analytics, and enterprise care; ability to tie eval KPIs to CSAT, AHT, FCR, MTTR.

Additional Information

- Lead high-impact Data & AI advisory programs for major enterprises and public sector clients.

- Shape enterprise strategies and governance frameworks that drive real transformation.

- Work with a talented, multidisciplinary team in a collaborative environment.

- Competitive compensation and strong professional development support.

Thanks,
Saghana Sithara

Business Manager, Recruitment| Quess Selection & Services, Singapore

EA License Number: 23C2060 Registration ID is R1550224

Disclaimer: The company is committed to ensuring the privacy and security of your information. By submitting this form, you consent to the collection, processing, and retention of the information you provide. The data collected (which may include your contact details, educational background, work experience and skills) will be used solely for the purpose of evaluating your qualifications for the position you're applying for. Your data will be stored securely and retained for the duration necessary to fulfil our hiring process. If you are not selected for the position, your data will be kept on file for a limited period in case future opportunities arise. You have the right to access, correct, or delete your data at any time by contacting us at Quess Singapore | A Leading Staffing Services Provider in Singapore (quesscorp.sg)

“This is in partnership with the Employment and Employability Institute Pte Ltd (“e2i”).

e2i is the empowering network for workers and employers seeking employment and employability solutions. e2i serves as a bridge between workers and employers, connecting with workers to offer job security through job-matching, career guidance and skills upgrading services, and partnering employers to address their manpower needs through recruitment, training, and job redesign solutions. e2i is a tripartite initiative of the National Trades Union Congress set up to support nation-wide manpower and skills upgrading initiatives. By applying for this role, you consent to Quesscorp Singapore’s PDPA and e2i’s PDPA.”