Every week, someone on this team sits down and reads conversations our agents have held with real people. It is not glamorous. It does not scale, in the sense the industry likes that word. And it has done more to improve the product this year than anything else we tried.
It is late December, which is the season for looking back. Here is what a year of that habit taught us — no customer data, just the patterns that repeat until you can no longer pretend not to see them.
Why read, rather than just measure
Automated tests measure the measurable: did the agent reply, how fast, did anything fall over. They do not measure the thing that matters — whether that reply was the one a good salesperson would have given that person at that moment.
We have found no shortcut to knowing that. You have to read. And reading is uncomfortable, because you meet your own mistakes in cold print — in the agent's handwriting, but with your signature underneath: the ambiguous instruction you wrote, the case you failed to anticipate. No dashboard embarrasses you like that. Which is precisely why it works.
People write when nobody is in
We suspected this when we built the product — we wrote about it at the time — but a year of reading turns it from a hunch into something you can no longer argue with: the interesting conversations happen in the evening, at the weekend, over a lunch break. Not as the exception. As the rule.
The enquiry typed at eleven on a Tuesday night is not an impatient one; it is a person who has finally found a spare half hour to sort their own life out. It deserved a proper conversation, not "our office hours are Monday to Friday".
"I don't know" has to be granted in writing
This was the lesson that cost us the most. A language model wants to help; it is its nature. Ask it something it does not know, without giving it a way out, and it will fill the gap with something that merely sounds reasonable. Telling it not to make things up is not enough. You have to give it explicit permission not to know, and a dignified sentence to say it with.
Then came the surprise: when the agent says "I don't know that one — let me check with the team", the conversation does not go cold. It gets warmer. People trust the party that admits a limit more than the one that answers everything with the same even confidence. It took us a while to accept this because it runs against instinct: surely a salesperson who hesitates sells less. It is the other way round. Honesty, read across hundreds of conversations, sells better than the appearance of knowing.
For the same reason, our agents introduce themselves as AI from the first message. Article 50 of the EU AI Act requires it — but at this point we would do it without the regulation.
Guarantees belong in code, not in the instructions
You can tell an agent "never invent prices" and it will obey almost every time. Almost. For prices, stock and products, "almost every time" is not a standard; it is a complaint with a date yet to be assigned.
Which is why the real guarantees do not live in the instructions. They live in code: deterministic hard floors that make the agent unable to state a price or a product that is not in the data, however hard the conversation pushes. We learned the difference between asking and guaranteeing by reading — watching a well-meaning agent drift towards the edge — and we fixed it where these things get fixed: somewhere the model cannot disobey, because it does not get a vote.
Qualifying is a conversation, not an interview
The same five questions, asked two different ways, produce opposite results. Chained like a form — field, field, field — people tire and leave: nobody wants to fill in paperwork over WhatsApp. Woven the way a genuinely interested person would ask them — one at a time, picking up what was just said before asking for the next thing — people answer, and often volunteer more than they were asked.
The information your sales team needs does not change. What changes is whether obtaining it feels like a conversation or like admin. It took many weeks of reading before we stopped optimising the script and started optimising the rhythm.
Handover is a feature, not a failure
A confession from the start of the year: deep down, we treated every handover to a human as a small defeat for the product. Twelve months of reading cured us. A handover done well — at the right moment, with the full context of the conversation, so the person never has to repeat themselves — is among the most valuable things the system does.
Because the customer does not want to talk to an AI, or to a human. They want their problem sorted without telling the story twice. An agent that knows when to step aside, and steps aside well, earns more trust than one determined to go the distance alone.
We will keep reading
None of the above came off a metrics dashboard. It came from reading, week after week, what the agent says when nobody on the team is watching — which is exactly when it matters.
Next year the method stays the same. If you want to see how all of this shows up in the product, it is laid out here. If not, take the one lesson that cost us most to learn: an AI that sells needs the right to say "I don't know" — granted in writing.