Revisiting Automatic Voice Similarity Assessment: From Equal Error Rate to Embedding Geometry

Meta description: Speaker verification models used as voice similarity proxies align with human perception more through training objectives and embedding geometry than EER.

Tags: speaker verification, voice similarity, embedding geometry, AAM-Softmax, speech generation

Featured image: Abstract waveform grid and low-dimensional vector space, soft blue and violet tones, no people, no logos, no text

AAM-Softmax Reduces Voice Alignment as d_eff Tracks −0.95 Correlation

Speaker verification (SV) models have become common automated stand-ins for human judgments of voice similarity in speech generation tasks. A systematic analysis challenges that habit: it shows that how an SV model is trained, not its headline verification accuracy, tracks human perceptual alignment more closely. The finding matters because SV scores are often used as convenient proxies when listening tests are costly, and a proxy that looks accurate can still disagree with what people hear.

Background: Why SV Models Were Used as Human Judges

SV systems compare voices by turning recordings into embeddings, vector representations that summarize speaker characteristics. Equal error rate (EER) measures verification accuracy by finding the operating point where false acceptances and false rejections are equal. As EER improved, the field commonly assumed that better verification meant better capture of speaker nuances. That assumption helped spread the use of SV models as automated proxies for human voice similarity in speech generation tasks.

The assumption also reflected practical pressure. Human evaluation of synthetic or converted speech is expensive, slow, and subjective. A model that can score similarity automatically is attractive for development loops, benchmarking, and large-scale testing. The question is whether accuracy in deciding "same speaker or not" is the same thing as producing a similarity score that matches human perception.

Development Process: Training Objectives, Not EER, Move the Metric

The analysis establishes a human perceptual alignment metric and compares it with conventional performance measures. It finds that perceptual alignment is governed far more by the learning objective than by how well the model performs on EER. Standard margin-based classification losses, such as AAM-Softmax, yield substantially lower perceptual alignment than prototypical metric losses.

Margin-based losses focus on separating classes with clear decision boundaries, while prototypical metric losses organize embeddings around representative points for speakers or classes. The comparison directly challenges the community's implicit assumption that verification accuracy and human-perceptual similarity improve together. EER itself fails to track human judgment, so two models with similar verification scores can support very different judgments about voice similarity.

Latest Progress: Geometry Explains the Mismatch

The study traces this divergence to embedding geometry, the shape and spread of speaker vectors in high-dimensional space. A model's effective dimensionality, written d_eff, tracks perceptual alignment with a −0.95 rank correlation. This strong relationship suggests that human-perceptual alignment becomes more predictable when embeddings occupy a smaller effective dimensional structure.

Classification losses tend to favor dimensional spread, while human voice perception appears to rely on a lower-dimensional structure. That mismatch explains why a model with strong verification accuracy can still produce similarity scores that feel misaligned with listeners. The geometric account turns a training-choice observation into a measurable property of the embedding space.

Current Stage: Bottleneck Evidence and What to Watch

Imposing a dimensionality bottleneck compresses d_eff and raises perceptual alignment. The reported metric, ρ_align, moves from 0.08 to 0.74. That result establishes a principled geometric criterion for evaluating voice similarity: look beyond EER and examine how many meaningful dimensions a model uses to represent speaker identity.

The research places the field at a point where voice similarity proxies can be judged by their geometry as well as their accuracy. What to watch next is whether dimensionality-based diagnostics become routine in speech generation evaluation, and whether further studies confirm the link between d_eff, ρ_align, and human perception across different loss functions and datasets.