Software Engineer - AI Evaulation & Automation

 OVERVIEW
We are seeking an experienced Software Engineer – AI Evaluation & Automation with a strong background in software engineering, developer tooling, and automation to support the client's efforts to build and scale the tooling used to measure how well AI-powered software development tools actually perform. The successful candidate will develop evaluation harnesses, automate benchmark runs, and help ensure that results are reproducible and hold up to scrutiny.

This role is ideal for a proactive, detail-oriented engineer who cares as much about getting the measurement right as about automating it, and who is comfortable working across engineering and data teams in a fast-paced AI environment.

Location: Heredia, CR (Hybrid)
Contract Length: 6+ Months (Extension or FTE Conversion Possible)
Hours: Standard Business Hours, Local Time Zone
Experience Level: Intermediate (3–5 years)

RESPONSIBILITIES

• Build and integrate evaluation harnesses and automation for software development use cases, including turning real engineering artifacts like merged pull requests into repeatable benchmark tasks.
• Build versioned, repeatable processes to evaluate AI tools, models, and harnesses, with reproducible run environments (pinned dependencies, containerized runs, isolated worktrees) so results stay comparable over time.
• Validate and calibrate evaluation approaches against human judgment, so scores are consistent and correct rather than just repeatable.
• Support execution-based benchmarking across quality, productivity, and efficiency measures, including cost and latency.
• Analyze results across repeated runs, looking at variance, failure patterns, and cost per outcome, and identify ways to make workflows more reliable and more automated.
• Work with engineering and data teams to improve the tooling, and document how the evaluations work and what they found for both technical and leadership audiences.

 

QUALIFICATIONS

• 3–5 years of software engineering experience, with real experience building automation, developer tooling, or test and validation systems.
• Proficient in Python, and comfortable in at least one of Java, JavaScript, or a similar language.
• Solid working knowledge of Git, including how branches, history, and working trees behave, and of containerization with Docker.
• Experience with APIs, development environments, CI/CD pipelines, and typical engineering workflows.
• Some familiarity with how AI, LLM, or agent evaluation works and where it goes wrong, such as why a judge can be consistent but still wrong, why a single run can mislead, and how benchmark contamination happens.
• Able to troubleshoot technical problems, think clearly about whether a measurement is valid, and analyze results carefully.

 

Preferred

• Hands-on work with AI-powered coding tools and agentic applications, such as Claude Code, Devin, or OpenCode.
• Experience designing benchmarks or evaluations for software systems, especially execution-based grading that verifies against tests.
• Familiarity with LLM-as-judge or agent-as-judge approaches, and how to check them against human raters.
• Experience with build-system-aware test selection, such as Bazel or mapping changed files to the tests that cover them.
• Experience building reproducible test environments and managing versioned evaluation datasets.
• Comfortable writing up methodology and results for engineering leadership.

Education

• Post-secondary degree in Computer Science, Software Engineering, or a related discipline (or equivalent hands-on experience).

Notes for Applicants
Resumes must clearly demonstrate hands-on software engineering experience, including specific examples of automation, tooling, or testing/validation systems built, along with relevant languages and technologies used.