Every parameter was set before the corpus was read.
- Corpus
- MultiWOZ 2.2, train + dev + test: 10,437 human–human dialogues, 143,044 turns. Text dialogues, so the engine is not voice-specific.
- Go / no-go
- Set in advance: at least 400 candidate moments, or stop. It came back 5,153, so the run went ahead.
- Model
all-MiniLM-L6-v2sentence embeddings of the user’s request, not the system’s refusal.- Threshold
- Cosine similarity 0.58, locked before the corpus was read. Not swept, not tuned.
- Same-gap rule
- Two moments are the same gap if one single knowledge-base entry would have answered both.
- Clusters
- 680 raw micro-clusters, grouped into 62 named gaps. Every merge, and every cluster left out with its reason, is listed in the files below.
- Thresholds
- Set in advance: at least 8 gaps, 15 recurrences in the largest, 40% coverage. Came back 24 substantive gaps, 1,065, and 49.5%.
- Coverage
- 49.5% of the 5,153 moments (2,550) land in one of the 24 substantive gaps; 63.9% counting all 62 named gaps.
- ENTRY / TOOL
- ENTRY: one knowledge-base entry would answer it. TOOL: no entry can; it needs a live lookup or an action. 1,339 moments (41%) are ENTRY and 1,956 (59%) are TOOL.
What the run found.
ENTRY one knowledge-base entry would answer it. TOOL no entry can; it needs a live lookup or an action. The dataset is set in Cambridge, UK, which is why gaps read like “outside the east”. Restaurant and train rows are domains in the dataset, not target customers.
| # | What the user couldn’t get | Times | Domains | Fix | |
|---|---|---|---|---|---|
| 1 | Booking slot unavailable (room or table) | 1,065 | hotel, restaurant | TOOL | |
| 2 | Hotel search: no match for the requested combination | 711 | hotel | TOOL | |
| 3 | Entertainment venue in the centre | 106 | attraction | ENTRY | |
| 4 | Multiple-sports venue outside the east | 102 | attraction | ENTRY | |
| 5 | Train: no service at the requested time / route | 97 | train | TOOL | |
| 6 | Restaurant search: common cuisine, no match for price / area | 78 | restaurant | TOOL | |
| 7 | Entrance fee / admission price not in the data | 74 | attraction | ENTRY | |
| 8 | College outside the centre / west | 38 | attraction | ENTRY | |
| 9 | Theatre / cinema in an area with none | 36 | attraction | ENTRY | |
| 10 | African restaurant in the wrong area / price | 34 | restaurant | ENTRY | |
| 11 | Swimming pool in an area with none | 33 | attraction | ENTRY | |
| 12 | Concert hall outside the centre | 27 | attraction | ENTRY | |
| 13 | Architecture outside the centre | 26 | attraction | ENTRY | |
| 14 | Nightclub outside the centre / south | 25 | attraction | ENTRY | |
| 15 | Vietnamese restaurant in the wrong area / price | 14 | restaurant | ENTRY | |
| 16 | Lebanese restaurant in the wrong area / price | 12 | restaurant | ENTRY | |
| 17 | Japanese restaurant in the wrong area / price | 11 | restaurant | ENTRY | |
| 18 | Turkish restaurant in the wrong area / price | 11 | restaurant | ENTRY | |
| 19 | Thai restaurant in the wrong area / price | 10 | restaurant | ENTRY | |
| 20 | Boating in an area with none | 10 | attraction | ENTRY | |
| 21 | Park in the west | 10 | attraction | ENTRY | |
| 22 | Korean restaurant in the wrong area / price | 9 | restaurant | ENTRY | |
| 23 | Seafood restaurant in the wrong area / price | 6 | restaurant | ENTRY | |
| 24 | Buy / reserve attraction tickets | 5 | attraction, train | TOOL | |
| · | Excluded: 38 “No [cuisine] restaurant” rows, a dataset artifact | 745 | restaurant |
Moments in all 62 named gaps, by how they can be fixed
Most of these misses can’t be fixed by a reply. 59% need a live lookup or an action, such as checking room availability, which no knowledge entry can supply.
That is why the product classifies instead of just counting: the owner is only asked questions a one-line reply can answer, and the rest are reported as integration work. By number of gaps the split is 57 to 5 in favour of entries; by how often they happen it is the reverse, and how often is what customers feel.
Within the 24 substantive gaps alone (2,550 moments), 23% (594) are entry-fixable. The two figures cover different sets of gaps, which is why they differ.
What this does not show
MultiWOZ is a research corpus with a bounded database; its misses are mostly database misses, not production knowledge-base gaps. 38 of the 62 named gaps are absent-cuisine rows from an ontology that lists cuisines with no venues. This demonstrates that the engine works on data it has never seen. It does not demonstrate demand.