Generated reply · draft, in Spanish
Highlighted: passages flagged by the judge or by me. Hover, tap or tab to them to see why.
A hotel receives hundreds of reviews, and one badly written sentence can admit a fault nobody has verified. This system interprets the review, drafts the reply, checks it and decides whether it is published on its own or goes to a person.
I spent years in hospitality handling guest complaints, and I know how delicate it is to do well.
Start with the guided tour: 9 cases, from a clean success to the failure the judge misses.
Highlighted: passages flagged by the judge or by me. Hover, tap or tab to them to see why.
A classifier decides whether the review is negative, neutral or positive, and how confident it is. It is mRoBERTa, a model trained for Spanish that I fine-tuned on real reviews, averaged over three seeds. On 11,414 reviews it never saw in training, it reaches a macro F1 of 0.781: a score from 0 to 1 that weighs all three classes equally. The neutral class, the hardest, stays at 0.549.
A language model writes well, but it tends to make things up. To limit that, it gets the hotel's full fact sheet and, through RAG, the policies and approved replies closest to this review. RAG (Retrieval-Augmented Generation) here means finding the relevant documents and handing them to the model before it writes. For policies, it combines search by keywords (BM25) and by meaning (bge-m3 embeddings).
Qwen3.6 27B, an LLM running locally, works out what the guest is talking about and drafts the reply. Two filters then check it: fixed rules (services missing from the fact sheet, compensation, people's names) and a second model, Gemma 4 12B, which I call the judge. It comes from another family so it does not share the generator's biases: it answers yes or no to 8 criteria and quotes the sentence behind each failure.
The gate makes the final decision. Negative and neutral reviews, doubtful ones (confidence below 0.85), traps and any accident or legal risk go to a person. Of the 54 reviews in the test, 8 pass every check. Of the 24 real 1 to 3★ reviews, none was published without review. It is not auto-publishing: it prepares drafts, at a median of 48 s per review on one PC.
This project began as an earlier version that appeared to work, but the result was very poor, so I audited it: 50.8% of the final exam (the test set) also appeared, word for word, in the training data. The model already knew the answers. This is called data leakage (reproduction). It also picked the model with that same exam and had never seen a 4★ review. I rebuilt it from scratch: duplicates removed before splitting, and a separate validation set.
Almost every reply opens the same way, so I tried more varied versions of the prompt, the model's instructions. Comparing them blind, without knowing which was which, I preferred the current one on 15 reviews and the newer one on 10, a difference that could be chance (p = 0.42). I kept the current one, and the replies are still repetitive. I will try again when I have more reviews, so as not to fit the prompt to these same cases and overfit.
I compared the judge with my own blind labels on 24 replies. It failed 7 and I failed 4, all of them among its 7: it missed none of mine, with 3 false alarms. The sample is small, and reading other replies I found implicit admissions of fault that it lets through (p08 and rag-carteles). So anything delicate always has to go to a person.
Every figure on this page comes from a file in the repository. Here they all are, with their sample size and source. I built the project with Claude Code: what I decided and what was done with the tool.
Each figure links to its file. Intervals on proportions are Wilson intervals; those on F1 come from each report's bootstrap. Calibration used rubric v2, and production uses v2g.
| Set | n | Metric | Value | 95% CI | Source |
|---|---|---|---|---|---|
| AHR, never used for training | 11,414 | F1 macro | 0.781 | 0.770 to 0.791 | phase3b_realista.json |
| AHR | 11,414 | F1 of the neutral class | 0.549 | — | phase3b_realista.json |
| AHR, confidence ≥ 0.85 | 8,487 | Error | 264 of 8,487 (3.1%) | 2.8% to 3.5% | phase3b_realista.json |
| Clean COAH test | 269 | F1 macro | 0.797 | 0.732 to 0.853 | phase3b_realista.json |
| COAH, confidence ≥ 0.85 | 191 | Error | 2 of 191 (1.0%) | 0.3% to 3.7% | phase3b_realista.json |
| SFU, another website | 50 | Correct | 39 of 50 (78%) | 64.8% to 87.2% | phase3b_realista.json |
| Calibration, rubric v2 | 24 | Judge precision | 4 of 7 (57%) | 25.0% to 84.2% | phase6_calibracion.md |
| Calibration, rubric v2 | 24 | Judge recall | 4 of 4 (100%) | 51.0% to 100.0% | phase6_calibracion.md |
| Calibration, rubric v2 | 24 | Kappa on “does not admit fault” | 0.70 | — | phase6_calibracion.md |
| Seeds, rubric v2 | 10 | The person's failures the judge detects | 1 of 3 (33%) | 6.1% to 79.2% | phase6_semillas.json |
| Demo queue | 54 | Pass every check | 8 of 54 (15%) | 7.7% to 26.6% | pipeline_cola.json |
| Real 1 to 3★ reviews | 24 | Published without review (by design) | 0 of 24 (0.0%) | 0.0% to 13.8% | pipeline_cola.json |
| Blind comparison, v6 and v9 | 25 | Prefers v6, excluding 3 ties (p = 0.42) | 15 of 25 (60%) | 40.7% to 76.6% | phase9_ciegas.md |
| Blind comparison, v6 | 30 | Publishable as it stands | 8 of 30 (27%) | 14.2% to 44.4% | phase9_ciegas.md |
| Latency on one PC | 15 | Median time to verdict | 48.2 s (p90 52.2 s) | — | phase9_latencia.md |
Only 6 cases have my verdict, with the criteria written before generating, and 1 fails: the matrix shows which cases they are, it does not measure the judge. Each ID opens the case in the viewer.
They measure different things and barely overlap, so the system has no single “n”.
| Set | Size | Origin | Use |
|---|---|---|---|
| Clean COAH test | 269 | Real, TripAdvisor | Measure the classifier |
| AHR | 11,414 | Real, TripAdvisor, 2021 | Measure the classifier on unseen reviews |
| SFU | 50 | Real, ciao.es | Measure it on another website |
| Demo queue | 54 | 14 synthetic and 40 real from AHR | Full system and gate |
| Blind comparison | 30, 28 rated | Written by me | Choose between prompt versions |
| Calibration | 24 | Replies from 3 prompt versions | Judge against a person |
| Seeds | 10 | Replies from the first versions | Real generator failures |
| This page's selector | 15 | 6 written by me and 9 illustrative | Show cases; not a sample |
ollama_gpu.py: the check that prevents running on the CPU without warning, added after that incident.