Duplex-MPE: A Benchmark for Multi-Party Interaction in Full-Duplex Conversations
Duplex-MPE benchmarks voice assistants in 2,000 audio-only multi-party chats, testing addressee, response timing, answers, silence, and stopping.
Key Takeaways
- Duplex-MPE evaluates voice assistants in full-duplex multi-party conversations, focusing on when to speak, answer, stay silent, or stop.
- It uses 2,000 paired audio-only scenarios with three or four human speakers and an assistant, contrasting explicit and implicit addressing.
- Models must infer who is speaking and whether a response is appropriate without transcripts or supplied turn boundaries.
- The benchmark separates evaluation into response initiation, answer correctness, appropriate silence, and stopping after human resolution.
- The framework is documented on the project page, with data and code planned for release before December.
Meta description: Duplex-MPE benchmark evaluates whether voice assistants speak, stay silent, answer, or stop in multi-party conversations using audio-only scenarios.
Tags: voice assistants, speech benchmarks, Duplex-MPE, multi-party conversation, AI research
Featured image: Abstract technology visual of overlapping soundwaves and speech-turn markers, with no real people, no company logos, and no text.
Background
When multiple people talk, a voice assistant faces a harder question than transcription: whether to respond at all. A reply delivered at the wrong moment can interrupt, confuse, or derail the group, while silence can be the safer answer.
Development Process
The Duplex-MPE benchmark uses 2,000 paired scenarios for continuous multi-party conversations. Each scenario includes three or four human speakers and an assistant, contrasting explicit and implicit addressing. Models receive audio without transcripts or supplied turn boundaries, forcing systems to infer who is speaking, whether the assistant is being spoken to, and when a response is appropriate. Separate metrics evaluate response initiation, answer correctness, remaining silent, and stopping when a human resolves the request.
Latest Progress
The project page, https://step-out.github.io/Duplex-MPE-Page/, documents the benchmark, while data and code are being prepared for release before December. The current stage is evaluation design: the framework is defined and documented, but the full release materials are still pending.
Comments
No comments yet. Be the first to comment.