You’ll build reinforcement learning environments for shopping agents. Our environments are based on real shopper questions. Each one includes the tasks, the scoring rubrics and the automated graders that check whether an agent gave a correct, safe and useful answer.
What you’ll do
- Turn real shopper questions into tasks with clear, checkable requirements.
- Design rubrics that break an answer into separate checks, such as whether the product exists, meets each requirement and is safe for the shopper.
- Build automated graders, including LLM-based judges, and check their scores against human reviewers.
- Find and close ways an agent could game the scoring.
- Build the pipelines that generate, version and run environments at scale.
What we’re looking for
- Experience with LLM evaluation, reward design, benchmarks or AI training data.
- Strong Python skills and production engineering habits.
- Experience checking automated scores against human judgment.
- Clear written communication.
- Full-time engineering experience or experience at an AI training data company is a plus.
How to apply
Email a link to your GitHub or a project you built, plus one sentence about the most interesting evaluation or environment you’ve built.
To apply, email harsh@thankly.ai with the role in the subject line.
Apply for this role