Sr. GenAI Engineer (3-6 month contract)

Department: Engineering

Employment Type: Full Time

Location: US

Reporting To: Pete Martin

Compensation: $129.00 - $150.00 / hour

Description

What You’ll Do

• Test and evaluate GenAI applications, AI agents, and LLM-powered workflows for quality, reliability, accuracy, safety, and task effectiveness.
• Develop and execute repeatable GenAI evaluation methodologies, test datasets, benchmark suites, regression tests, and automated evaluation pipelines.
• Use DeepEval and related evaluation approaches to measure GenAI output quality and identify regressions across prompts, workflows, application changes, and model versions.
• Optimize GenAI solution architecture by decomposing complex prompts and workflows into appropriate subtasks, tools, and specialized agents.
• Build and improve multi-agent workflows using orchestration frameworks such as LangGraph and LangChain.
• Optimize prompts, system instructions, context management, agent interactions, and model selection / configurations using evaluation results (including latency and model performance) and production feedback (including user feedback).
• Evaluate new foundation models and model versions to determine quality, behavior, risks, performance, (groundedness, completeness, bias, etc) and suitability for specific use cases.
• Design and implement monitoring capabilities for GenAI systems, including quality metrics, operational telemetry, audit trails, alerts, and performance reporting.
• Help drive AI governance activities for selected initiatives, including model and application documentation, risk assessments, evaluation evidence, model versioning, registry management, experiment tracking, and change traceability.
• Collaborate across technical, governance, security, compliance, and business teams to improve GenAI solutions and support responsible production use.
  
  

• GenAI testing and quality evaluation
• GenAI evaluation methodology and automated evaluation
• Prompt, context, and application architecture optimization
• Multi-agent architecture and orchestration
• Foundation model and model-version testing
• AI governance and model lifecycle management
• GenAI monitoring, observability, and traceability
• Traditional machine learning testing and validation
• Analytical problem-solving and cross-functional collaboration

What’s Required

• 10+ years of overall professional experience in data science, machine learning, AI engineering, or a related technical field.
• 2+ years of hands-on experience developing, testing, evaluating, or optimizing Generative AI systems.
• Demonstrated experience testing GenAI applications and establishing systematic methods for assessing LLM output quality.
• Hands-on experience with DeepEval, LangGraph, LangChain, Python, and modern AI and machine learning libraries.
• Experience designing or optimizing multi-agent systems, agent orchestration patterns, or complex LLM workflows.
• Strong hands-on experience with Databricks for data, AI, or machine learning workloads, including model versioning, model registries, and experiment tracking.
• Practical experience supporting AI governance, model lifecycle controls, documentation, traceability, and auditability.
• Prior experience on traditional machine learning projects as an ML engineer, data scientist, or similar contributor, with a strong understanding of model testing and validation methods.
• Ability to communicate technical findings, evaluation results, risks, and recommendations to technical and nontechnical stakeholders.
• Ability to maintain at least four hours of overlap with U.S. Eastern Time (EST/EDT) each workday.
• Bachelor's or advanced degree in computer science, data science, engineering, mathematics, statistics, physics, computational science, or another scientific, quantitative, or computational discipline.
  
  

• Experience with Databricks Mosaic AI.
• Experience working with PHI / PII data inside highly regulated domain like healthcare.
• Experience testing or optimizing multi-agent GenAI systems in production or production-like environments.
• Experience working with embedding models, semantic retrieval, vector search, or Retrieval-Augmented Generation architectures.
• Experience applying AI governance or Responsible AI requirements in a regulated enterprise environment.

Compensation & Benefits