# Method — Unknown Ops gap engine on MultiWOZ 2.2

**Public data only.** No client data, client counts, client gap labels or results from any other engagement were used, referenced or compared against. The pipeline (`run.py`, `step0.py`) was written fresh for this demonstration.

## Corpus
MultiWOZ 2.2 — github.com/budzianowski/multiwoz, `data/MultiWOZ_2.2`, train + dev + test: 10,437 human–human dialogues, 143,044 turns. The public MultiWOZ venue database (`db/`) was used only to decide ENTRY vs TOOL.

## Step 0 — candidate population (go / no-go set before counting: E ≥ 400)
B (NoOffer / NoBook acts) 4,397 · C (lexical, no such act) 130 · D (user question with no schema slot — heuristic) 684 · **E = 5,153 → GO**. See `step0_report.md`. D's precision is low (most are in-schema questions the annotators left untagged); E without D is 4,527, so the go decision does not depend on it.

## Clustering — parameters fixed before any output was seen
- Model: `sentence-transformers/all-MiniLM-L6-v2`, normalised embeddings
- Text embedded: the **user's request** (for B/C, the user turn immediately before the refusal)
- Agglomerative clustering, average linkage, cosine; similarity threshold **0.58**, locked, not swept
- Only micro-clusters with ≥ 5 members go to the naming pass; smaller ones are the long tail

**Raw micro-clusters: 680** (180 with ≥ 5 members, covering 4,320 moments; 304 singletons). **Grouped: 62 named gaps.**

## The locked same-gap rule
> Two moments are the same gap if one single knowledge-base entry would have answered both.

For TOOL-class moments, where no entry can answer, this is read as "one fix (one tool) would have served both". That reading is disclosed because it is what allows the largest gap to merge hotel and restaurant booking failures.

**ENTRY vs TOOL** (decided against the public venue database, not against the transcripts): ENTRY if the requested thing is absent from the database, exists in ≤ 3 venues (one entry can list them), or is an attraction type confined to known areas; TOOL if the answer depends on live availability or a multi-attribute search.

## Every merge
All in `merged_raw_clusters.json` — each named gap lists its raw cluster ids and the reason. Reviewed clusters that were **not** named, with the reason:
- **mixed — the locked rule cannot unify the members (different facts would answer them):** 14 clusters, 429 moments
- **context-dependent fragment — the request only makes sense with earlier turns (an area, a price, a day, a party size):** 29 clusters, 352 moments
- **not a gap — in-schema request the annotators left untagged (step-0 D noise: reference numbers, addresses, star rating, cost, travel time, 'help me find a place'):** 14 clusters, 244 moments

## Thresholds (set in the brief before the data was touched)
| Check | Result | Threshold | |
|---|---|---|---|
| Named gap clusters | 62 | ≥ 8 | met |
| Occurrences in largest cluster | 1065 | ≥ 15 | met |
| Share of candidates in a named cluster | 63.9% | ≥ 40% | met |

No persistence metric: MultiWOZ dialogues carry no meaningful timestamps, so none is reported.

## What this demonstrates — and what it does not

**What this demonstrates:** the engine ingests a corpus it has never seen, identifies the moments where a user asked for something the system could not satisfy, and resolves many phrasings into a small number of named, ranked, fix-classified gaps — using a pipeline whose model, threshold and grouping rule were locked before the corpus was read.

**What it does not demonstrate:** that MultiWOZ has a knowledge-base problem. It is a research corpus with a bounded database, and its misses are mostly database misses. The wizards had a database and mostly could answer; the dataset was built so tasks complete. Nothing here is drawn from, or compared with, any production deployment.

## Known weaknesses, stated plainly
- The naming / merging pass is a human judgment applied with the locked rule; it is fully listed so it can be checked.
- The largest gap merges hotel and restaurant booking failures under the TOOL reading of the rule — the loosest merge in the table.
- D-sourced moments inside named gaps may be untagged ordinary questions rather than failures.
- Per brief, the worked cluster is the second-largest gap; at 711 members it is a TOOL gap (multi-attribute hotel search), not a single-fact entry.