Hiring.Camp

AI Test Engineer

AlgoShack

·

Today

Department
Engineering
Source
Keka

Description

AI Test Engineer:

The role in one line: A hands-on engineer who proves our AI features are accurate, safe and stable before and after every change, and who can build that testing setup alone.
What this person will actually do
    • Build automated accuracy checks for the AI
    • Make those checks a gate in the CI pipeline,]
    • Build and maintain test datasets
    • Test the whole pipeline end to end:
    • Watch quality in production using LLM tracing tools (Arize Phoenix, LangSmith) to spot drift, rising failures and cost or latency jumps.
    • Run safety and robustness testing: prompt injection inside documents, hallucinated fields, PHI leaking into logs, and inconsistent answers to the same input.
    • Produce evidence for a regulated product. Test reports need to stand up in a medical-device-style audit (IEC 62304, HIPAA).
Must-have skills (screen hard on these)
    1. Strong Python test automation: pytest, API testing, CI pipelines (GitLab).
    2. LLM evaluation hands-on: has built eval sets and scoring, not just “used ChatGPT.” Ask them to name the metrics they used and how they set pass/fail thresholds.
    3. Experience with at least one eval or tracing tool: LangSmith, Arize Phoenix, DeepEval, Ragas, promptfoo, or similar.
    4. Understands non-deterministic testing: how to test something that gives slightly different answers each run (repeat runs, tolerances, LLM-as-judge and its limits).
    5. Document or vision AI exposure: OCR or extraction from PDFs and images is a big plus for us.
    6. Solo-capable: has set up a test approach from scratch, not only executed someone else’s.
Good-to-have
    • LangChain / LangGraph. Useful, but not essential: our AI Doc calls Gemini directly through Google’s SDK. Don’t reject strong candidates for lacking LangGraph.
    • Google Cloud / Vertex AI, Django APIs, Playwright.
    • Healthcare or regulated-domain testing, and handling of PHI and de-identified data.
    • Performance and cost testing of LLM calls (latency, tokens).

Skills

PythonDjangoGitLabPlaywrightHIPAALLM

Similar Jobs

30

AI Test Engineer

Fortinet·Santa Clara, California

1y+ ago

Sr. Electrical Test Engineer, AI Satellites (Starmind)

Spacex·Bastrop, Texas·Onsite

5d ago

Optical Test Engineer, AI Satellites (Starmind)

Spacex·Bastrop, Texas

1w ago

Applied AI Engineer – Test & Evaluation (T&E)

Canvasinc·Tennessee, US·Onsite

1w ago

Senior Test Engineer (AI-Augmented & Spec-Driven Testing)

KMS Technology·Ho Chi Minh City, Vietnam·Hybrid

2w ago

Mobile Application Test Engineer – AI-Enabled Testing

Anblicks·Hyderabad, Telangana

1mo ago

Electrical Test Engineer, AI Satellites (Starmind)

Spacex·Bastrop, Texas

1mo ago

Staff Hardware Test Engineer (AI Device/Module ATE)

Renesas Electronics·Bayan Lepas, Penang

1mo ago

Senior AI Engineer, Test Automation (m/w/d)

Carrier·Germany·Onsite

1mo ago

AI Test Engineer - VLabs

Vialto·Bengaluru, Karnataka·Onsite

2mo ago

Integration & Test Engineer, AI Satellites (Starmind)

Spacex·Bastrop, Texas

2mo ago

Senior AI Test Engineer (Remote)

TestPros·Sterling, Virginia·Remote

3mo ago

Staff Software Development Test Engineer - AI Evaluation

Tekion·Bengaluru, Karnataka·Onsite

4mo ago

Senior Software Development Test Engineer - AI

Tekion·Bengaluru, Karnataka·Onsite

4mo ago

Networking Solution Test Engineer - AI IB and Ethernet Cluster Debugging

Nvidia·Shanghai, China

Test Engineer III - AI/ML Server Platforms (401 Shift)

Jabil·US

5d ago

AI - Software Development Engineer in Test (AI - SDET)

Web·Buenos Aires, Buenos Aires F.D. +1·Remote

6d ago

Staff System Test Automation Engineer, AI

Sonatus·Detroit, Michigan·Remote

1w ago

Systems Test Engineer, HPC & AI Systems

Hewlett Packard Enterprise (HP)·Bengaluru, Karnataka·Hybrid

1w ago

Systems Test Engineer, HPC & AI Systems

Hewlett Packard Enterprise (HP)·Bengaluru, Karnataka·Hybrid

1w ago

Systems Test Engineer, HPC & AI Systems

Hewlett Packard Enterprise (HP)·Bengaluru, Karnataka·Hybrid

1w ago

Software Development Engineer in Test - AI

Web·Mumbai, Maharashtra·Remote

2w ago

College Intern – Hardware Test Automation & AI Engineer

Hp·SGP25 - Singapore - Depot 2

2w ago

College Intern – Hardware Test Automation & AI Engineer

HP·Singapore

2w ago

College Intern – Hardware Test Automation & AI Engineer

Hp·SGP25 - Singapore - Depot 2

2w ago

Software Engineer, Test Infrastructure (AI DevEx)

Siftstack·Marina del Rey, California +1·Hybrid

2w ago

Compliance Test Engineer; HPC-AI Products

Hewlett Packard Enterprise (HP)·Chippewa Falls, Wisconsin·Onsite

3w ago

Compliance Test Engineer; HPC-AI Products

Hewlett Packard Enterprise (HP)·Chippewa Falls, Wisconsin·Onsite

3w ago

Compliance Test Engineer; HPC-AI Products

Hewlett Packard Enterprise (HP)·Chippewa Falls, Wisconsin·Onsite

3w ago

Sr. Software Development Engineer in Test, AI/ML

Sony Interactive Entertainment·Waterloo, Ontario·Hybrid

3w ago