Assessing the Impact of Content Quality on AI Assistant Response Feedback
Portfolio note: All identifying details and proprietary information have been changed or omitted. This abstract summarizes a longer case study that is available by request.
Abstract
This case study examines whether the quality of retrieved enterprise knowledge content influences user feedback on responses generated by a Retrieval-Augmented Generation (RAG) system. Focusing on AI assistant responses that retrieved internal wiki documentation, the study analyzes whether measurable content characteristics, such as freshness, readability, sentence length, ownership, and source curation, are associated with higher positive feedback rates.
Using feedback events from RAG-enabled AI responses, the analysis combines two-proportion tests and logistic regression to identify document-level signals that reliably predict positive user feedback. The results show a statistically significant relationship between higher-quality retrieved documentation and better perceived response quality. In particular, stronger page-health scores, shorter sentence lengths, and content drawn from highly curated documentation areas were associated with higher positive feedback.
The findings suggest that documentation quality is not merely a human usability concern; it is also a meaningful input to AI system performance in enterprise RAG environments. While the analysis is observational and subject to selection effects from the retrieval and ranking system, it provides practical evidence that improving the freshness, clarity, and maintainability of internal documentation can contribute to better AI-assisted knowledge experiences.
To request the full version (sanitized) of this case study, contact me.