AI Test Engineer
AlgoShack
·Today
- Department
- Engineering
- Source
- Keka
Description
- Build automated accuracy checks for the AI
- Make those checks a gate in the CI pipeline,]
- Build and maintain test datasets
- Test the whole pipeline end to end:
- Watch quality in production using LLM tracing tools (Arize Phoenix, LangSmith) to spot drift, rising failures and cost or latency jumps.
- Run safety and robustness testing: prompt injection inside documents, hallucinated fields, PHI leaking into logs, and inconsistent answers to the same input.
- Produce evidence for a regulated product. Test reports need to stand up in a medical-device-style audit (IEC 62304, HIPAA).
- Strong Python test automation: pytest, API testing, CI pipelines (GitLab).
- LLM evaluation hands-on: has built eval sets and scoring, not just “used ChatGPT.” Ask them to name the metrics they used and how they set pass/fail thresholds.
- Experience with at least one eval or tracing tool: LangSmith, Arize Phoenix, DeepEval, Ragas, promptfoo, or similar.
- Understands non-deterministic testing: how to test something that gives slightly different answers each run (repeat runs, tolerances, LLM-as-judge and its limits).
- Document or vision AI exposure: OCR or extraction from PDFs and images is a big plus for us.
- Solo-capable: has set up a test approach from scratch, not only executed someone else’s.
- LangChain / LangGraph. Useful, but not essential: our AI Doc calls Gemini directly through Google’s SDK. Don’t reject strong candidates for lacking LangGraph.
- Google Cloud / Vertex AI, Django APIs, Playwright.
- Healthcare or regulated-domain testing, and handling of PHI and de-identified data.
- Performance and cost testing of LLM calls (latency, tokens).