INNOVATIVE APPLICATIONS OF RAG-ENHANCED SMALL LLM FOR CLOSED-DOMAIN Q&A

초록

Efficiently integrating Large Language Models (LLMs) into business operations requires addressing hallucination issues, incorporating proprietary internal datasets, and ensuring economic viability. Key strategies to tackle this challenge involve implementing a Retrieval-Augmented Generation (RAG) methodology, fine-tuning open-source small LLMs using proprietary internal datasets, and establishing on-premises GPU infrastructure. In this study, we validate the performance of this methodology through practical applications. Our selected use case involves responding to queries related to operational knowledge, including regulations, guidelines, and manuals. To achieve this, we deploy a Machine Reading Comprehension (MRC)-based RAG system. Additionally, the Llama 2 model undergoes fine-tuning on both internal and external datasets to enhance Korean language understanding and acquire domain-specific knowledge. The system’s performance was evaluated based on the accuracy achieved using two datasets: a set of 200 Q&A pairs prepared by task managers and a dataset comprising 150 Q&A pairs derived from training exam questions and answers for new employees. The evaluations yielded accuracy scores of 92.7% and 79.3%, respectively. © 2025, Int. J. Innov. Comput. Inf. Control. All rights reserved.

키워드