A recent research article proposes Inference-Time Policy Adapters (IPA), a resource-efficient method for tailoring large language models (LLMs) without the need for fine-tuning. The IPA framework can improve performance in tasks like text generation, dialogue safety control, and reducing toxicity without relying on costly fine-tuning processes. The authors of this research are affiliated with the Allen Institute for Artificial Intelligence, University of Washington, and University of Southern California.

Background and Limitations in Tailoring Large Language Models

Language models have shown promise in tackling various natural language understanding tasks, but their practical adoption has been hindered by their limitation in ensuring desired objectives through prompting alone. Fine-tuning language models through supervised learning or reinforcement learning can help tailor their behavior, but it can be expensive and inaccessible to the broader community.

Existing inference-time algorithms can customize language models without accessing their parameters, but they are often less effective than fine-tuning. In response to these limitations, the authors propose Inference-Time Policy Adapters (IPA) as an efficient and lightweight method to tailor language models without fine-tuning their parameters.

Introducing Inference-Time Policy Adapters (IPA)

IPA combines a base language model’s output distribution with a smaller adapter policy and optimizes it using reinforcement learning, working efficiently even for large language models. To tailor a model, IPA trains a lightweight policy adapter to optimize an arbitrary user objective. As a result, IPA consistently improves language model performance on tasks like text generation and dialogue safety control without requiring resource-intensive model fine-tuning.

During the process, the base policy’s parameters are kept frozen, and only the adapter policy’s parameters are updated. IPA can easily integrate with any reinforcement learning algorithm, making it versatile and adaptable. At inference time, the next-token distribution is obtained from the tailored policy and used with a standard decoding algorithm.

Performance Evaluation and Applications

IPA was evaluated on a variety of tasks and consistently outperformed off-the-shelf language models, like GPT-3, and sometimes yielded even better results than expensive fine-tuned versions of GPT-3:

  • Toxicity reduction: IPA was applied to reduce the toxicity of autogenerated text using the RE-ALTOXICITYPROMPTS benchmark. It significantly outperformed all previous baselines for controlling GPT-3.

  • Lexically constrained generation: Using the CommonGen dataset, IPA improved constraint coverage and generation quality at a fraction of the cost of fine-tuning GPT-3.

  • Open-ended generation: IPA was used to enhance the fluency, coherency, and human-like qualities of machine-generated content using the XSum dataset.

  • Dialogue safety control: IPA was applied to Blenderbot models to generate safe and coherent responses to potentially unsafe user utterances, outperforming other off-the-shelf dialogue models.

  • Knowledge-grounded dialogue: IPA improved the faithfulness of dialogue responses without sacrificing quality, demonstrating potential to improve the reliability and trustworthiness of NLP systems.

Conclusion and Future Implications

Inference-Time Policy Adapters (IPA) offer a lightweight and efficient method to tailor large language models without the need for fine-tuning. By consistently outperforming competitive baselines across different text generation tasks, IPA demonstrates the potential to significantly improve AI capabilities in real-life applications.

For the artificial intelligence community, the key takeaway is that IPA inherits the generalizability of the reinforcement learning approach and incorporates the flexibility of inference-time techniques. By customizing base policy models without sacrificing their attributes and complementing model scaling with resource efficiency, IPA could pave the way for new algorithmic innovations that benefit the broader community and improve AI capabilities across many applications.

Original Paper