Hiring.Camp

AI Test Engineer - VLabs

Vialto

·

Sep 7, 2026

Location
Bengaluru, India
Workplace
Onsite
Type
Full-time
Department
Engineering
Source
Workday

Description

About Vialto Labs (VLabs) 

Vialto Labs (VLabs) is responsible for redesigning how work is delivered in the tax and immigration service lines, as well as driving operational efficiency across Vialto’s functional areas using AI. The team builds and deploys novel AI-enabled solutions that directly improve productivity and increase delivery quality for our clients. VLabs is accountable for rapidly turning innovative experiments into production-ready deliverables at scale and embedding them into day-to-day operations. This team focuses on the highest-impact workflows, creating standardized, repeatable capabilities that can be deployed globally. Operating with a mandate for speed and measurable outcomes, VLabs works alongside service line, product, and platform leaders. 

About the Role 

AI Test Engineering is a hands-on role within VLabs Quality Engineering, responsible for validating the performance, reliability, and integrity of AI-enabled solutions in production environments. This role operates at the intersection of AI engineering and quality assurance, ensuring that outputs from LLMs, OCR pipelines, document classification models, and agentic workflows perform as expected at scale and meet defined business performance thresholds.  Working closely with the Programme Test Manager and partnering with engineering, product, and delivery teams, this role translates AI testing strategy into executable frameworks, evaluation pipelines, and reusable assets embedded into the delivery lifecycle. 

Success requires independent execution, strong technical depth, and the ability to proactively identify risks, patterns, and performance gaps while enabling rapid, production-grade deployment of AI capabilities. 

Key Responsibilities 

AI Evaluation & Test Design 

  • Translate AI testing strategy into executable test scenarios across LLM outputs, document classification, extraction accuracy, agent workflows, and edge cases 

  • Design adversarial and boundary test inputs to expose hallucination, misclassification, and failure modes 

  • Validate AI outputs for structure, consistency, accuracy, and production readiness against defined performance thresholds 

Evaluation Engineering & Automation 

  • Build reusable Python-based evaluation frameworks, including output validation, hallucination detection, and scoring mechanisms 

  • Develop parameterized test scripts reusable across features, models, and releases 

  • Implement AI-as-Judge frameworks, including prompt design, scoring logic, and calibration of evaluation reliability 

  • Embed evaluation frameworks into CI/CD pipelines to support continuous testing and deployment 

Drift Detection & Quality Monitoring 

  • Design and operate drift detection frameworks using fixed baseline datasets and scheduled re-evaluation 

  • Establish thresholds to distinguish acceptable variation from performance degradation 

  • Enable release gating by identifying regressions prior to production deployment 

Ground Truth & Data Quality 

  • Build and maintain ground truth datasets in partnership with subject matter experts 

  • Define standards for classification, extraction accuracy, and acceptable output characteristics 

  • Continuously update datasets to reflect evolving business requirements and use cases 

Workflow & Integration Testing 

  • Test end-to-end agentic workflows, validating data integrity, error propagation, and fallback behavior 

  • Perform API-level testing of AI pipeline endpoints using Python and Postman/Newman 

  • Validate data persistence and integrity across system layers using SQL 

  • Partner with engineering teams to ensure testability, observability, and system reliability 

Standardization & Scaling 

  • Define and scale standardized AI evaluation patterns and reusable quality frameworks across VLabs 

  • Contribute to enterprise AI quality standards and reference architectures 

Governance & Responsible AI 

  • Ensure adherence to Responsible AI, data privacy, and governance requirements 

  • Support auditability, traceability, and transparency of AI outputs and evaluation processes 

Stakeholder Enablement 

  • Translate evaluation results into actionable insights for engineering, product, and business stakeholders 

  • Support decision-making on model readiness, release risk, and performance trade-offs 

  • Proactively identify risks, patterns, and systemic issues and escalate appropriately 

Qualifications & Experience 

Professional Experience 

  • 7+ years in software testing, including 2–3 years focused on AI/ML-enabled systems in production environments 

  • Proven experience designing and executing AI evaluation frameworks and quality strategies 

  • Strong track record building ground truth datasets, drift detection systems, and scalable evaluation pipelines 

  • Experience testing multi-step agentic workflows and AI-driven automation systems 

  • Experience operating in fast-paced, iterative delivery environments 

  • Background in regulated or compliance-driven environments preferred 

Technical Expertise 

  • Advanced Python programming for evaluation frameworks, batch processing, and data analysis 

  • Experience with LLM evaluation tools such as deepeval, RAGAS, promptfoo, or similar 

  • Strong capabilities in: 

  • AI output validation, hallucination detection, and grounding checks 

  • Drift detection frameworks and statistical evaluation methods 

  • OCR, VLM, and document AI testing (classification, extraction, edge cases) 

  • API testing using Python (requests/httpx) and Postman/Newman 

  • SQL for data validation and pipeline integrity checks 

  • Familiarity with LangChain, LlamaIndex, or similar frameworks 

  • Experience with cloud AI platforms such as Azure AI Foundry or AWS Bedrock preferred 

Operating Capabilities 

  • Ability to operate independently in fast-moving, ambiguous environments 

  • Strong analytical mindset with attention to detail and quality rigor 

  • Ability to balance speed and rigor in AI evaluation and delivery cycles 

  • Proactive communicator who identifies risks and drives resolution 

  • Ability to translate technical findings into business-relevant insights 

Education 

  • Bachelor’s degree required

  • Advanced degree in Computer Science, Data Science, or related field preferred 

Additional Information:

  • This job is based in our Bangalore office with the possibility of hybrid work mode

  • We are an equal opportunity employer that does not discriminate on the basis of any legally protected status

  • Please note, AI is used as part of the application process

Skills

PythonAWSAzureCI/CDSQLData ScienceCompliance

Similar Jobs

30

AI Test Engineer

AlgoShack

5d ago

AI Test Engineer

Fortinet·Santa Clara, California

1y+ ago

Sr. Electrical Test Engineer, AI Satellites (Starmind)

Spacex·Bastrop, Texas·Onsite

1w ago

Optical Test Engineer, AI Satellites (Starmind)

Spacex·Bastrop, Texas

1w ago

Applied AI Engineer – Test & Evaluation (T&E)

Canvasinc·Tennessee, US·Onsite

2w ago

Senior Test Engineer (AI-Augmented & Spec-Driven Testing)

KMS Technology·Ho Chi Minh City, Vietnam·Hybrid

3w ago

Electrical Test Engineer, AI Satellites (Starmind)

Spacex·Bastrop, Texas

1mo ago

Staff Hardware Test Engineer (AI Device/Module ATE)

Renesas Electronics·Bayan Lepas, Penang

1mo ago

Automation Test Engineer - AI

Accenture·London, England

1mo ago

Senior AI Engineer, Test Automation (m/w/d)

Carrier·Germany·Onsite

1mo ago

Integration & Test Engineer, AI Satellites (Starmind)

Spacex·Bastrop, Texas

3mo ago

Senior AI Test Engineer (Remote)

TestPros·Sterling, Virginia·Remote

3mo ago

Staff Software Development Test Engineer - AI Evaluation

Tekion·Bengaluru, Karnataka·Onsite

4mo ago

Senior Software Development Test Engineer - AI

Tekion·Bengaluru, Karnataka·Onsite

4mo ago

Networking Solution Test Engineer - AI IB and Ethernet Cluster Debugging

Nvidia·Shanghai, China

Senior Software Test Development Engineer - AI Infra, SDET

Nvidia·Shanghai, China

2d ago

Senior Software Test Development Engineer - AI Infra, SDET

Nvidia·Shanghai, China

2d ago

Test Automation Engineer (AI-Driven Testing)

Rydoo·Kristiansand, Agder·Hybrid

3d ago

Test Automation Engineer (AI-Driven Testing)

Rydoo·Antwerp, Ohio·Hybrid

3d ago

Test Automation Engineer (AI-Driven Testing)

Rydoo·Warsaw, Mazovia·Hybrid

3d ago

Test Engineer III - AI/ML Server Platforms (401 Shift)

Jabil·US

4d ago

Test Engineer III - AI/ML Server Platforms (401 Shift)

Jabil·US

1w ago

AI - Software Development Engineer in Test (AI - SDET)

Web·Buenos Aires, Buenos Aires F.D. +1·Remote

1w ago

Staff System Test Automation Engineer, AI

Sonatus·Detroit, Michigan·Remote

2w ago

Systems Test Engineer, HPC & AI Systems

Hewlett Packard Enterprise (HP)·Bengaluru, Karnataka·Hybrid

2w ago

Systems Test Engineer, HPC & AI Systems

Hewlett Packard Enterprise (HP)·Bengaluru, Karnataka·Hybrid

2w ago

Systems Test Engineer, HPC & AI Systems

Hewlett Packard Enterprise (HP)·Bengaluru, Karnataka·Hybrid

2w ago

Software Development Engineer in Test - AI

Web·Mumbai, Maharashtra·Remote

2w ago

College Intern – Hardware Test Automation & AI Engineer

Hp·SGP25 - Singapore - Depot 2

3w ago

College Intern – Hardware Test Automation & AI Engineer

HP·Singapore

3w ago