A team of researchers from Peking University have developed a large language model, Lawyer LLaMA, specifically designed for the legal domain. The model aims to overcome challenges that existing models face in understanding and applying legal knowledge to address practical issues. By fine-tuning the model with legal domain data and integrating a retrieval module to generate more reliable responses, the team has significantly improved the model’s performance in the legal domain.

Large Language Models (LLMs) have exhibited remarkable performance across various general tasks, but they often struggle to perform well in specific domains such as law or medicine. This is due to their limited exposure to domain-specific resources and the vast difference in strategy required to analyze and solve domain-specific tasks. The research focuses on addressing the challenges faced by LLMs in the legal domain.

The paper presents a comparison between LLaMA pre-trained on a general corpus (BELLE) and Lawyer LLaMA designed for the legal domain. It highlights that LLMs pre-trained in the general domain lack domain-specific knowledge and struggle to solve legal problems, even when provided with necessary legal knowledge.

An LLM capable of handling legal tasks should have clear objectives: 1. Convey accurate and precise meaning without ambiguity; 2. Understand and distinguish legal terminology; and 3. Be able to analyze practical cases. Developing a model that can effectively fulfill these requirements requires a well-thought-out approach.

The researchers laid out a four-step process in developing Lawyer LLaMA, ensuring the model could correctly apply legal knowledge and solve practical legal problems:

  1. Injecting legal knowledge: Continual pretraining with legal domain content, collecting raw text in the field.
  2. Knowledge distillation: Transferring the learned legal knowledge to the model for better understanding.
  3. Model fine-tuning: Fine-tuning the model on legal datasets to improve domain-specific performance.
  4. Designing specific tasks: Creating tasks to test the model’s performance in the legal domain.

Additionally, the researchers trained the model using real queries and generated answers with ChatGPT to teach the model to solve domain-specific tasks with appropriate knowledge.

Integrating a Retrieval Module

To improve accuracy and faithfulness, the team introduced an information retrieval module for LLaMA. The module retrieves relevant law articles based on user queries and context and uses them as evidence to generate more reliable responses.

Enhancing Performance with Multilingual General Corpus

The LLaMA model is pretrained in two stages - a multilingual general corpus and a Chinese legal corpus. The multilingual general corpus includes both Chinese and English articles, improving the model’s ability to represent and comprehend Chinese. The Chinese legal corpus allows the model to leverage domain-specific knowledge to handle legal tasks effectively.

Improving AI Performance in Specific Domains

The development of Lawyer LLaMA demonstrates the potential for enhancing LLMs in specific domains to generate more accurate and reliable responses. This research serves as an excellent example for future development within other domain-specific areas, such as medicine, finance, and more. By following a similar approach - designing models with targeted datasets, focused continual pretraining, and delivering external evidence through retrieval modules - large language models can be adapted to address specialized tasks more effectively, fostering their development and utility in a wide range of practical applications.

Original Paper