Scale AI / Engineering case study
Building the pipelines behind LLM evaluation.
Retrieval, candidate filtering, and multi-stage reranking for annotation inputs, alongside high-throughput response benchmarking.
The work, at a glance
At Scale AI, I worked on the Gen AI, LLM Evals team. My projects covered the inputs used in annotation and the systems used to score model responses, with a focus on reducing annotation errors.
Input preparation
A retrieve-and-rerank RAG pipeline that automated LLM inputs for data annotation.
Response evaluation
A benchmark system and multi-stage reranking using LLM-as-a-judge and adjudication.
- Model responses scored daily
- 8K+
- Reduction in annotation errors
- 60%
01 / Context
Evaluation quality starts with the inputs.
My work addressed two parts of the evaluation workflow: preparing useful inputs for annotation and scoring the responses produced by models. Candidate selection and response measurement both affect the quality of the resulting evaluation data.
The projects brought together retrieval-augmented generation (RAG), candidate filtering, reranking, and benchmark infrastructure.
02 / Pipeline design
Retrieve, filter, and rerank.
I developed a retrieve-and-rerank RAG pipeline to automate LLM inputs for data annotation. Candidate filtering was part of that preparation, narrowing the material passed through the pipeline.
I also worked on multi-stage reranking using LLM-as-a-judge and adjudication. These stages supported the goal of reducing annotation errors.
Prepare inputs. Evaluate responses.
03 / Evaluation
Make response scoring a repeatable workflow.
I built a high-throughput benchmark system that scored more than 8,000 model responses per day. The reported outcome of the evaluation work was a 60% reduction in annotation errors.
The two workstreams addressed different parts of the same problem: preparing candidate inputs and assessing the model outputs used in annotation. Treating evaluation as a pipeline made those responsibilities explicit.