How Hugo Scaled Audio Sentiment Annotation Across Six Workflows
TL;DR
1.5M evaluations across six audio workflows at 94.97% accuracy against a 90% target, with AHT 31.4% faster than goal. 130 raters calibrated for tone, accent and emotional ambiguity, on audio alone.
Sentiment is easy to hear and hard to label. A speaker can sound flat while meaning something sharp, warm while masking frustration, or ambiguous in a way that two careful listeners will read two different ways. Strip out the face, the gesture and the context, leave only the waveform, and the judgment gets harder still.
A technology organization building conversational and voice AI systems, the kind of frontier and reasoning lab workload Hugo staffs for, engaged Hugo to annotate and evaluate audio for sentiment, emotion and conversation quality. The programme delivered 1,506,081 evaluations across six workflows at 94.97% accuracy against a 90% target, with average handling time 31.4% faster than target.
Six Workflows, One Signal
Every queue on the programme worked from audio alone. What changed between them was the unit of judgment: a transcript, an intent, a turn boundary, an error category. Turn boundaries and speaker identification are speech and audio diarization work, and half the programme rested on getting them right before any sentiment call was made.
- GenAI spontaneous speech transcription. Correcting first-pass ASR output to capture fillers, false starts, non-verbal sounds, pauses, intonation and crosstalk, not just the words.
- Conversation intent. Classifying a full user and model exchange into L1 and L2 intent categories, with a pass or fail on conversation quality.
- User turn labeling. Segmenting user speech into turns with start and end timestamps, and deciding whether each turn warranted a model response.
- Segmented speech transcription. Correcting pre-populated ASR at the level of the individual turn, with timestamps that had to hold.
- Fluidity. Speaker identification, device-directed speech, interruptions, noisy segments and turn completion across the whole conversation.
- Error categories. Tagging model failures: looping, topic hijacking, glitches, unnatural speech, latency, hallucinations, refusals, emotional mismatch and audio artifacts.
Why It Was Hard
Emotional cues are layered, not stated
Sentiment is rarely explicit. Raters decoded tone, pitch variation, rhythm, pacing, stress and hesitation, and then had to separate genuine neutrality from sarcasm delivered flat. Speakers regularly expressed mixed emotions, or drifted from one register to another inside a single clip.
Audio quality varies and ambiguity follows it
Clips differed in background noise, microphone quality, echo, reverb, speaker clarity and overlapping speech. Poor audio does not just make transcription harder. It widens the range of defensible emotional readings, which shows up later as rater disagreement.
Accent and culture change what a cue means
Speaking styles carry emotion differently across regions and linguistic backgrounds. A rater working from one set of norms will over-read some cues and under-read others. Neutrality here is a trained skill, not a default.
No visual context
Multimodal tasks let an annotator resolve an ambiguous tone by looking at a face. Audio-only work removes that fallback entirely, raising cognitive load and the error margin along with it.
How Hugo Approached It
Upfront preparation
Sentiment evaluation SOPs set out the emotional classification hierarchy and drew the working line between sentiment, emotion and tone. Calibration packs carried boundary examples, the cases that actually cause drift: “slightly annoyed” against “neutral but tense”, emotion blends, and cultural edge cases. Separate audio quality guidelines covered how to treat noise, muffled speech and unclear pronunciation, so raters were not inventing a policy per clip.
Team structure matched to the work
- Raters trained specifically in acoustic nuance and linguistic sensitivity, at a billable headcount of 130.
- QA leads and quality engineers specialising in affective annotation consistency.
- Subject matter experts for rare or genuinely ambiguous emotion cases.
- Programme and delivery management running RCA cycles, escalation handling and SOP updates.
A six-part annotation framework
The framework borrows its structure from Hugo’s wider perception and annotation practice, narrowed to what a waveform can actually support.
- Acoustic signal analysis. Clarity, noise level, intensity, pitch variation, stress and emphasis.
- Sentiment classification. Mapping to positive, negative, neutral or mixed against the rubric, using tone, pace and intonation.
- Emotion identification. Where required, separating anger, sadness, joy, frustration, calmness and confusion.
- Confidence scoring. Raters rated their own certainty, which is what makes noisy and sparse clips visible in the data rather than silently averaged in.
- Safety and integrity filters. Harmful or sensitive speech, self-harm cues, violence, hate content and distress indicators, applying the same standards Hugo uses on red teaming and alignment work.
- Error tagging and reasoning. Written rationale on ambiguous cues, cultural interpretation issues and audio interference, feeding model misclassification patterns back to the client.
A fixed evaluation sequence
Each item followed the same three steps. Raters listened to the full audio first, to understand the conversation before judging any part of it. They then segmented user turns with accurate start and end timestamps, including pauses inside a single thought. Only then did they evaluate each turn against the quality attributes: side speech, background noise, engagement, multiple speakers and model interruption.
Results
- 1,506,081 evaluations across six workflows.
- 94.97% quality against a 90% target, 4.97 points clear.
- Average handling time 576 seconds against an 840-second target, 31.4% faster.
- 130 billable raters, running since March 2024 and ongoing.
Quality held above target across every workflow despite uneven complexity between them. Error Categories was the hardest queue by a distance, asking raters to distinguish conversational, speech, audio, hallucination and behavioural failures in context-sensitive and emotion-sensitive exchanges. It stayed above 90% on the same guideline discipline and calibration cadence as the rest of the programme.
What We Learned
Emotional ambiguity is the primary failure mode
Not noise, not tooling. The clips that produced low-confidence labels and rater variance were the ones where the emotional cue was genuinely weak, particularly at the boundary between neutral and low-level negative.
Audio quality is a quality variable, not a nuisance
Background interference produced measurable errors in tone judgment and in telling annoyance apart from flat neutral delivery. Treating audio condition as an annotated attribute, rather than something to work around, made those errors visible.
Cultural neutrality has to be taught
Accent and speaking style shift how emotion reads. Without explicit training and calibration on it, raters over-interpret some cues and under-interpret others, and the bias is systematic rather than random.
Calibration moves the number
Repeated calibration cut rater disagreement, reduced misreading of hesitation and stress, and pulled raters off over-reliance on the words themselves. On subjective work it is the single highest-leverage intervention available.
Short clips manufacture ambiguity
Many sentiment cues need contextual buildup to be legible. Short-duration clips strip that away and produce divergent judgments from raters who are each reading correctly. That is a sampling decision, and it belongs upstream of the annotation.
Why It Mattered
The work fed emotion-aware conversational agents and sharper acoustic sentiment detection, identified the specific failure modes in affective interpretation rather than reporting an aggregate error rate, and raised training data quality for foundational speech systems. The confidence scores and written reasoning did something a pass rate cannot: they told the client’s research teams where the model was wrong and why a human found the same case hard.
Where This Sits in Hugo’s Evaluation Stack
Audio sentiment work sits at the intersection of acoustic perception, linguistic analysis and safety judgment. It is part of Omni Evals, Hugo’s human evaluation practice for frontier models.
Oluwajuwonlo Akinnusi
As an Associate Project Manager at Hugo, Oluwajuwonlo Akinnusi has spent 6 years honing enterprise data annotation delivery. In his featured case study, he took evaluations to production scale while holding 94.45% audit precision and 52.6% audit coverage across 1,464 annotations, proof that speed and rigor aren’t opposites. Oluwajuwonlo sums up his approach in one line: accuracy and trust come first, and disciplined process is how you get there.
Build your Dream Team
Ask about our 30 day free trial. Grow faster with Hugo!
