Whether we’re seeking customer support, looking for recommendations, or simply asking a quick question, AI chatbots are designed to give us the answers we’re looking for. But there’s more going on beneath the surface. Every time we chat with them, they are learning from us to improve their understanding and responses. And the type of language we use, whether formal or informal, directly affects the quality of their answers, according to new research.
In general, people naturally adapt their conversation style to the person they are speaking with. So, in a bank, for example, when talking to a loan officer, we might use more formal language and complete sentences. But in a relaxed setting like a bar with friends, we might use abbreviations and more casual phrasing. Fulei Zhang and Zhou Yu at Amazon wanted to see if this same conversation shift happens when people talk to chatbots versus other humans, and if so, how it affects a chatbot’s performance.
The researchers compared thousands of messages people sent to human agents with those sent to AI chatbots, focusing on features like grammar, vocabulary and politeness. They found that people were 14.5% more polite and formal and 5.3% more grammatically fluent when chatting with humans than when talking with AI, based on analysis by the Claude 3.5 Sonnet model.
Next, they trained an AI model called Mistral 7B on about 13,000 real chats between people, then tested how well it understood more than 1,300 messages people had sent to chatbots. To broaden the AI’s exposure, they also created blunt and polite rewrites of those messages to simulate different communication styles.
It turns out that chatbots trained on a diverse mix of message styles, including real and fake messages, were 2.9% better at understanding user intent than AI trained solely on original human conversations. The researchers also tried to improve Mistral AI’s understanding by rewriting informal messages at the last minute to be more formal, but this led to a drop in understanding by almost 2%.
So the best way to make chatbots smarter is to train them on a range of communication styles, as the researchers state in their paper published on the arXiv preprint server. “Training-time exposure to diverse linguistic variation is more effective than inference-time normalization. Models must learn to interpret diverse communication styles during training, rather than rely on brittle post-hoc transformations that risk semantic distortion.”
Findings like these could be key to improving the quality of chatbot responses in real-world settings.
Written for you by our author Paul Arnold, edited by Lisa Lock, and fact-checked and reviewed by Robert Egan—this article is the result of careful human work. We rely on readers like you to keep independent science journalism alive.
If this reporting matters to you,
please consider a donation (especially monthly).
You’ll get an ad-free account as a thank-you.
More information:
Fulei Zhang et al, Mind the Gap: Linguistic Divergence and Adaptation Strategies in Human-LLM Assistant vs. Human-Human Interactions, arXiv (2025). DOI: 10.48550/arxiv.2510.02645
arXiv
© 2025 Science X Network
Citation:
The way we talk to chatbots affects their accuracy, new research reveals (2025, October 16)
retrieved 16 October 2025
from https://techxplore.com/news/2025-10-chatbots-affects-accuracy-reveals.html
This document is subject to copyright. Apart from any fair dealing for the purpose of private study or research, no
part may be reproduced without the written permission. The content is provided for information purposes only.
