Evals & benchmarks

50 vendors.

• Anthromind — Human evaluation, RLHF preference data and post-training datasets with a scalable-oversight focus.
• Appen — Publicly listed veteran of the data industry offering training data, annotation and model evaluation with a large global crowd.
• Arena (LMArena) — Crowd-voted public leaderboard for language, image and code models, with commercial evaluation services.
• Bugcrowd — Crowdsourced security platform (bug bounties, pen testing) that also applies its researcher network to AI security testing.
• Centific — Provides multilingual datasets, secure data operations and responsible-AI services to model labs and enterprises.
• Chakra — Research and design lab producing expert-authored task sets for training and benchmarks for evaluation.
• Collinear — Simulation lab producing training data and RL environments so agents can be tested in realistic scenarios.
• Datacurve — Produces custom data for long-horizon reasoning, software engineering and data-science tasks for frontier labs.
• Good Start Labs — Game-based simulations for measuring and improving model reasoning, tool use and enterprise skills.
• Gray Swan — AI security firm: adversarial testing and runtime protection for LLM applications.
• Halluminate — Builds computer-use and tool-use RL environments modeled on banking, private-equity and consulting work.
• Handshake AI — The campus-jobs network runs a paid fellowship that recruits experts and students to produce AI training and evaluation work.
• iMerit — Expert-led annotation and fine-tuning services backed by its own workflow tooling.
• Incalmo — Autonomous-cybersecurity company building cyber environments for training and testing models.
• Innodata — Listed data-engineering firm providing training data, annotation, evaluation and red-teaming.
• Invisible Technologies — Mixes an expert workforce with orchestration software; supplies post-training data to labs and production AI workflows to enterprises.
• Lightwheel — Makes simulation-ready 3D assets, synthetic and egocentric human data, and evaluation platforms for embodied AI.
• LXT — Collects, annotates and evaluates audio, speech, text, image and video data across a very wide range of language locales.
• Mecka — Captures and structures real-world human activity into datasets and evaluations for robotics teams.
• Mercor — Recruits domain experts at scale and turns their work into training data, benchmarks and agent-training tasks for labs and enterprises.
• micro1 — Describes itself as a data lab: expert human data, real-world training environments and contextual agent evaluations.
• OpenTrain AI — Marketplace for hiring vetted AI trainers and labelers for RLHF, evaluation and annotation.
• Pareto AI — Expert human-data provider focused on feedback signals that improve frontier models.
• Patronus AI — Research lab building training data and simulated environments for frontier AI, evolving from an evaluation-tools origin.
• Perle AI — Routes work from a large international pool of credentialed professionals into AI training and evaluation projects.
• Plato — Simulated web worlds for training and evaluating browser agents.
• Polymath — Simulation environments for training and evaluating long-horizon agents.
• Prime Intellect — Open stack spanning compute, training, inference, sandboxes and a hub of RL environments.
• Prolific — Self-serve platform for recruiting verified participants and experts to give human feedback and evaluation data.
• Quesma — Analytics for coding-agent cost and outcomes; also publishes coding-agent benchmarks.
• Revelo — Latin American engineering-talent company that now sells code-focused expert data, RL environments and evals to AI labs.
• Runloop — Secure code sandboxes and evaluation tooling for building and refining AI coding agents.
• Sama — Annotation company covering computer vision and generative-AI data with human-in-the-loop quality control.
• Scale AI — Long-running supplier of labeled data, human feedback and model evaluations to frontier labs, government agencies and large enterprises.
• Snorkel AI — Builds specialized datasets, benchmarks and evaluation environments aimed at high-stakes domains for frontier models and agents.
• SuperAnnotate — Data infrastructure platform paired with managed expert workforces for annotation, evaluation and training-data projects.
• Surge AI — Human-data company focused on high-skill annotators and RLHF data for frontier model training.
• Symbal — Trains models and deploys them for enterprises while supplying post-training data to frontier labs.
• Synthetic Sciences — AI research lab building infrastructure and evaluations for scientific AI.
• TaskUs — Outsourcing company whose AI services cover data labeling, RLHF and safety/red-teaming.
• Taste Labs — Builds infrastructure for bringing brand and creative IP into AI systems, including design-preference data.
• TELUS Digital — Digital-services arm of TELUS offering AI data collection, annotation and evaluation at scale.
• Toloka — Data partner for agentic, coding and safety work, blending human experts with tooling to generate training and evaluation data.
• Trajectory Labs — Public-benefit corporation offering safety evaluations, red teaming and RL environments to frontier labs.
• TrustScale (formerly e2f) — Long-time language-data provider, now branded TrustScale, that also markets a hallucination-verification product.
• Turing — Draws on a large engineering talent network to produce coding and reasoning data and help enterprises deploy agents.
• Vals AI — Independent testing of models on real-world work, publishing benchmarks across coding, finance, law and more.
• Veris AI — Builds simulated copies of a company's systems, data, APIs and users so agents can be tested and trained before production.
• Vetto — Post-training partner covering RL environments, rubrics and verifiers, RLHF, SFT and expert human evaluation.
• XDOF — Robotics company collecting teleoperation data and motion capabilities for autonomous systems.