Paper #227
LoRA Fine-Tuning and Retrieval-Augmented Baseline Analysis for a Topical-Chat Conversational LLM
Presenting author: Mohammed Musthafa
Session: I02
Abstract
This paper investigates lightweight adaptation of a conversational large language model for topical open-domain dialogue. Using the Topical-Chat dataset, conversations were converted into supervised next-response prediction examples, where the model learns to generate the next assistant reply from preceding dialogue history. A Qwen2.5-1.5B-Instruct model was adapted using Low-Rank Adaptation (LoRA) under Kaggle GPU constraints. In addition, a local retrieval-augmented generation (RAG) baseline was implemented using wiki-backed Topical-Chat factual sections and BM25 retrieval. Fine-tuning experiments across progressively larger training subsets show that validation loss decreased from 3.1550 in a 300-sample pilot run to 2.6430 in the strongest reported run, with perplexity decreasing to 14.0557. The RAG baseline provided explicit evidence retrieval for factual prompts, while LoRA fine-tuning improved dialogue modeling. At this stage ,the system is suitable for supervised prototyping rather than direct public deployment.