Hiring.Camp

AI Evaluation Rubric Reviewer

Mercor

·

Today

Salary
$40 – $75/hr
Location
US
Workplace
Remote
Type
Contract
Closing date
Today
Source
CareersPage

Description

The work

We are testing AI agents on realistic shopping and delivery tasks. You'll review AI-drafted tasks (prompt, grading rubric) along with model attempts, and decide whether each failure is fair:

- Was the information available?

- Does the rubric accurately grade what the prompt asked for?

- Would a valid answer have been marked wrong?

- Is the rubric structure correct per the guidelines?

Fix the criteria that aren't fair, flag the tasks that can't be saved, and write a short note on each call.

Must-haves

- Substantial experience writing or reviewing rubrics for LLM evals. Please list the projects in your application.

- Can read a long agent transcript and find where it went wrong.

- Precise writing, and you check the data instead of trusting the reference answer.

Skills

LLM

Similar Jobs

3

AI Evaluation Rubric Reviewer

Mercor·New York City, New York·Remote

Today

AI Evaluation Rubric Reviewer

Mercor·New York City, New York·Remote

Today

AI Evaluation Rubric Reviewer

Mercor·New York City, New York·Remote

Today