- Location
- Bengaluru
- Type
- Full-time
- Department
- Engineering
- Seniority
- Senior
- Experience
- 7+ years
- Source
- Breezy HR
Description
Job Position: Senior AI Quality Assurance (QA) Engineer
Experience: 7+ Years
Location: Bengaluru
Required Skills
- 7+ years of experience in Software Quality Assurance, Test Automation, or AI Quality Engineering.
- Hands-on experience evaluating LLM-powered applications, conversational AI systems, NL2SQL solutions, or similar AI workflows.
- Strong understanding of AI benchmarking methodologies and experience creating benchmark (golden) datasets for regression testing.
- Experience with AI evaluation and observability platforms such as Lang Smith, MLflow, Arize Phoenix, or equivalent.
- Strong understanding of AI evaluation metrics, including Precision, Recall, Precision@K, Recall@K, F1 Score, Exact Match (EM), Mean Reciprocal Rank (MRR), Execution Accuracy, latency (P50/P95/P99), token usage, and cost analysis.
- Strong proficiency in Python with hands-on experience building automation using Pytest, including unit testing, integration testing, API testing, exception handling, mocking, fixtures, and parameterized testing.
- Experience testing REST APIs and distributed backend services.
- Strong SQL skills with the ability to validate generated queries, execution results, schema alignment, and business logic.
- Strong analytical and debugging skills with the ability to perform root cause analysis across complex AI systems.
Skills
PythonSQLREST