Retrieval Augmented Generation: An Evaluation of RAG-based Chatbot for Customer Support

Abstract

The rapid advancements in generative artificial intelligence and large language models have revolutionized the field of natural language processing, giving rise to sophisticated frameworks such as Retrieval Augmentation Generation (RAG). The RAG framework combines the strengths of information retrieval and text generation, enabling the development of highly capable chatbots. As chatbots become increasingly popular in various domains, including customer support, their ability to deliver accurate and relevant responses is crucial. The evaluation of these chatbots is equally important to ensure their reliability and effectiveness. This thesis evaluates the performance of a Retrieval Augmentation Generation (RAG)-based chatbot for customer support developed at a software company. This study focus specifically on the evaluation of the retriever module. The study investigates the impact of keyword generation/extraction, the effects of different prompting techniques, and additional parameters influencing system performance. A dataset for evaluation was created as a part of this study, encompassing a wide range of customer queries related to four different software products offered by the company.