Hiring.Camp

AI Test Engineer

AlgoShack

·

5 days ago

Department
Engineering
Source
Keka

Description

AI Test Engineer:

The role in one line: A hands-on engineer who proves our AI features are accurate, safe and stable before and after every change, and who can build that testing setup alone.
What this person will actually do
    • Build automated accuracy checks for the AI
    • Make those checks a gate in the CI pipeline,]
    • Build and maintain test datasets
    • Test the whole pipeline end to end:
    • Watch quality in production using LLM tracing tools (Arize Phoenix, LangSmith) to spot drift, rising failures and cost or latency jumps.
    • Run safety and robustness testing: prompt injection inside documents, hallucinated fields, PHI leaking into logs, and inconsistent answers to the same input.
    • Produce evidence for a regulated product. Test reports need to stand up in a medical-device-style audit (IEC 62304, HIPAA).
Must-have skills (screen hard on these)
    1. Strong Python test automation: pytest, API testing, CI pipelines (GitLab).
    2. LLM evaluation hands-on: has built eval sets and scoring, not just “used ChatGPT.” Ask them to name the metrics they used and how they set pass/fail thresholds.
    3. Experience with at least one eval or tracing tool: LangSmith, Arize Phoenix, DeepEval, Ragas, promptfoo, or similar.
    4. Understands non-deterministic testing: how to test something that gives slightly different answers each run (repeat runs, tolerances, LLM-as-judge and its limits).
    5. Document or vision AI exposure: OCR or extraction from PDFs and images is a big plus for us.
    6. Solo-capable: has set up a test approach from scratch, not only executed someone else’s.
Good-to-have
    • LangChain / LangGraph. Useful, but not essential: our AI Doc calls Gemini directly through Google’s SDK. Don’t reject strong candidates for lacking LangGraph.
    • Google Cloud / Vertex AI, Django APIs, Playwright.
    • Healthcare or regulated-domain testing, and handling of PHI and de-identified data.
    • Performance and cost testing of LLM calls (latency, tokens).

Skills

PythonDjangoGitLabPlaywrightHIPAALLM

Similar Jobs

30

AI Test Engineer

Fortinet·Santa Clara, California

1y+ ago

Sr. Electrical Test Engineer, AI Satellites (Starmind)

Spacex·Bastrop, Texas·Onsite

1w ago

Optical Test Engineer, AI Satellites (Starmind)

Spacex·Bastrop, Texas

1w ago

Applied AI Engineer – Test & Evaluation (T&E)

Canvasinc·Tennessee, US·Onsite

2w ago

Senior Test Engineer (AI-Augmented & Spec-Driven Testing)

KMS Technology·Ho Chi Minh City, Vietnam·Hybrid

3w ago

Electrical Test Engineer, AI Satellites (Starmind)

Spacex·Bastrop, Texas

1mo ago

Staff Hardware Test Engineer (AI Device/Module ATE)

Renesas Electronics·Bayan Lepas, Penang

1mo ago

Automation Test Engineer - AI

Accenture·London, England

1mo ago

Senior AI Engineer, Test Automation (m/w/d)

Carrier·Germany·Onsite

1mo ago

AI Test Engineer - VLabs

Vialto·Bengaluru, Karnataka·Onsite

2mo ago

Integration & Test Engineer, AI Satellites (Starmind)

Spacex·Bastrop, Texas

3mo ago

Senior AI Test Engineer (Remote)

TestPros·Sterling, Virginia·Remote

3mo ago

Staff Software Development Test Engineer - AI Evaluation

Tekion·Bengaluru, Karnataka·Onsite

4mo ago

Senior Software Development Test Engineer - AI

Tekion·Bengaluru, Karnataka·Onsite

4mo ago

Networking Solution Test Engineer - AI IB and Ethernet Cluster Debugging

Nvidia·Shanghai, China

Senior Software Test Development Engineer - AI Infra, SDET

Nvidia·Shanghai, China

2d ago

Senior Software Test Development Engineer - AI Infra, SDET

Nvidia·Shanghai, China

2d ago

Test Automation Engineer (AI-Driven Testing)

Rydoo·Kristiansand, Agder·Hybrid

3d ago

Test Automation Engineer (AI-Driven Testing)

Rydoo·Antwerp, Ohio·Hybrid

3d ago

Test Automation Engineer (AI-Driven Testing)

Rydoo·Warsaw, Mazovia·Hybrid

3d ago

Test Engineer III - AI/ML Server Platforms (401 Shift)

Jabil·US

4d ago

Test Engineer III - AI/ML Server Platforms (401 Shift)

Jabil·US

1w ago

AI - Software Development Engineer in Test (AI - SDET)

Web·Buenos Aires, Buenos Aires F.D. +1·Remote

1w ago

Staff System Test Automation Engineer, AI

Sonatus·Detroit, Michigan·Remote

2w ago

Systems Test Engineer, HPC & AI Systems

Hewlett Packard Enterprise (HP)·Bengaluru, Karnataka·Hybrid

2w ago

Systems Test Engineer, HPC & AI Systems

Hewlett Packard Enterprise (HP)·Bengaluru, Karnataka·Hybrid

2w ago

Systems Test Engineer, HPC & AI Systems

Hewlett Packard Enterprise (HP)·Bengaluru, Karnataka·Hybrid

2w ago

Software Development Engineer in Test - AI

Web·Mumbai, Maharashtra·Remote

2w ago

College Intern – Hardware Test Automation & AI Engineer

Hp·SGP25 - Singapore - Depot 2

3w ago

College Intern – Hardware Test Automation & AI Engineer

HP·Singapore

3w ago