报告题目：Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs

作者：Oded Ovadia *†, Menachem Brief†, Moshik Mishaeli, and Oren Elisha 

单位：Microsoft, Israel

报告人：李婕

报告时间：2024年1月18日

报告地点：博学楼621会议室

报告内容摘要：Large language models (LLMs) encapsulate a vast amount of factual information within their pre-trained weights, as evidenced by their ability to answer diverse questions across different domains. However, this knowledge is inherently limited, relying heavily on the characteristics of the training data. Consequently, using external datasets to incorporate new information or refine the capabilities of LLMs on previously seen information poses a significant challenge. In this study, we compare two common approaches: fine-tuning and retrieval-augmented generation (RAG). We evaluate both approaches on a variety of knowledge-intensive tasks across different topics. Our findings reveal that while fine-tuning offers some improvement, RAG consistently outperforms it, both for existing knowledge encountered during training and entirely new knowledge. Moreover, we find that LLMs struggle to learn new factual information through fine-tuning, and that exposing them to numerous variations of the same fact during training could alleviate this problem.