How Hugo Audited Human Preference Data for Demographic Bias

At a Glance

A client preparing to train on human preference data needed to know one thing before committing: was the signal real, or an artifact of who happened to be rating? Hugo audited the preference judgments produced by 215 specialists across five demographic variables. The finding: preference held steady across every group. Demographics shaped how decisively specialists committed, not which model they preferred. The signal reflected the work, not the pool.

Results:

• 215 specialists in the audited pool
• 5 demographic variables analyzed: gender, age, STEM background, education, experience
• 2 to 3 points, the maximum preference difference across any demographic group
• 94.7% Instruction Following tie rate on Slides (over 80% on Websites), the evaluation dimension, not any demographic factor, was what actually drove variation

The Client

The client operates at the scale where AI-generated outputs reach millions of people. To improve those outputs, they rely on human preference data: specialists comparing two model responses and judging which is better, with those judgments feeding back as a training signal. The quality of that signal determines the quality of the next model, which makes one question unavoidable before scaling it.

The Challenge

Human preference data carries a hidden risk. If the people producing the judgments lean a particular way, the model learns that lean, not the quality of the work. A signal skewed by who happened to be in the pool trains a worse model while looking, on the surface, like progress.

The client needed to know whether their preference signal was trustworthy before scaling it into Phase 2. Specifically: 

• Were preference outcomes being driven by the actual quality of the model outputs, or by the demographic makeup of the specialist pool? 
• Would the signal generalize, or was it an artifact of this particular group? 
• If bias existed, where was it, and how large?

These are not questions a quality score answers. A team can hit its agreement targets and still be encoding a systematic tilt. The only way to know was to audit the judgments against the people who made them.

The Hugo Approach

Hugo ran a demographic audit across the full body of work: 6,523 audited jobs produced by 215 specialists, analyzed against five variables (gender, age, STEM background, education, and experience).

The audit was built to separate two things that are easy to confuse:

• Preference, meaning which model a specialist judged better.
• Decisiveness, meaning how often a specialist committed to a call rather than landing on a tie.

For each demographic variable, Hugo compared the preference distribution (Model A / Tie / Model B) against the pool average, and separately tracked tie rates as a measure of decisiveness. Crucially, the audit also compared demographic effects against a non-demographic factor: the evaluation dimension itself (Visual Design, Functionality, Instruction Following, Overall Preference). This gave a baseline for what a large effect actually looks like in the data, so any demographic effect could be judged against a real yardstick rather than in isolation.

The specialist pool varied meaningfully across all five variables, which is what made the test worth running. A homogeneous pool would have made a null result meaningless. This one did not.

Key Results

• Preference held steady across every demographic group: No variable shifted which model was preferred by more than two to three points. Gender is representative: neither men nor women leaned systematically toward either model. Age, STEM background, education, and experience told the same story.
• The evaluation dimension drove far more variation than any demographic factor: Instruction Following tie rates reached 94.7% on Slides and over 80% on Websites: the two models rarely differed on whether they followed the prompt. No demographic variable came close to that kind of range. What was being evaluated mattered enormously; who evaluated it barely moved the outcome.
• Where demographics did show up, it was in confidence, not taste: Male specialists were consistently more decisive, tying less often across dimensions. Master’s-degree holders showed the same pattern. In every case, the direction of preference stayed nearly identical. Some groups simply committed more readily than others.

Key Learnings

The headline result is a null result, and that is exactly what made it valuable. The client’s preference signal was not skewed by the pool. It reflected the quality of the outputs, which is what a training signal is supposed to do.

Two learnings sit underneath that.

• Demographics explain confidence, not preference: The only reliable pattern was decisiveness. Some groups tied less often, but they leaned the same direction as everyone else. This is a lever rather than a problem: if sharper differentiation is a goal, specialist selection can be tuned for decisiveness without distorting what the signal actually says.
• Judge effects against a real baseline, not in isolation: A 2 to 3 point demographic difference sounds like something until it sits next to the dimension effect, where tie rates ran from the low double digits up past 94%. Comparing demographic variation against a known large factor is what turned an ambiguous set of small numbers into a clear conclusion.

What Phase 2 Could Optimize For

Phase 2 will test new evaluation dimensions, specifically factuality and hallucination, since the audit showed Instruction Following added little differentiating signal and the higher-value differences lay elsewhere. 

Beyond that, because the signal was clean, the audit surfaced a few additional levers the client could pull:

• Test factuality and hallucination. With Instruction Following rarely separating the models, factuality is the natural candidate for a higher-value dimension in Phase 2.
• Pool composition. The pool can be adjusted toward heavier representation of particular groups, without concern that doing so would tilt the signal.
• Aesthetic latitude. In dimensions like Visual Design, where both models are often technically sound and the call comes down to taste, the rubric can be relaxed to license more personal aesthetic judgment, or specialist selection can lean toward more decisive profiles. Either lever sharpens differentiation, and the audit shows neither would distort which model is actually preferred.

Conclusion

Before you scale a preference signal, you should know what is in it. This audit answered that: the signal was driven by the work, not the workforce. That is the difference between training on quality and training on the accident of who showed up. For a client about to commit real resources to Phase 2, knowing the foundation is sound is worth as much as any single result.