Instruction tuning is a crucial step in LLM alignment, i.e., shaping the habits of enormous language fashions (LLMs) to raised align with the supposed goal. It includes fine-tuning a pre-trained LLM on a different set of directions, every paired with a desired output. This course of allows the mannequin to generalize throughout varied duties and codecs, finally enhancing its efficiency in understanding and responding to person directions. In essence, instruction tuning empowers LLMs to observe directions extra successfully, thereby making them extra helpful and dependable instruments for a variety of functions. Current progress in instruction tuning highlights the crucial position of high-quality knowledge in enhancing LLMs’ instruction-following capabilities. Nonetheless, buying such knowledge by way of human annotation stays cost-prohibitive and troublesome to scale, hindering additional progress.
Alternatively, latest work explores synthesizing instruction–response pairs for LLM alignment by prompting fashions with instance knowledge and iteratively refining the outcomes. Whereas these strategies are efficient at producing different directions for LLM alignment broadly, real-world functions typically prioritize tailoring the LLM to particular downstream duties akin to particular person enterprise functions or private assistant brokers, which regularly contain completely different instruction distributions. This want for task-specific alignment brings us to a core query for knowledge synthesis: how can we tailor artificial knowledge to align LLMs for various instruction-following duties?
In “CodecLM: Aligning Language Fashions with Tailor-made Artificial Knowledge”, offered at NAACL 2024, we current a novel framework, CodecLM, that systematically generates tailor-made high-quality knowledge to align LLMs for particular downstream duties. Impressed by the rules of the encode-decode course of, we leverage a robust LLM (i.e., an LLM that has robust instruction-following functionality for knowledge synthesis, akin to Gemini Professional or text-unicorn) as a codec, to encode seed directions from our goal process into instruction metadata (key phrases that seize the use case of the instruction, and the abilities required for an LLM to reply to the instruction). We then decode the metadata into tailor-made artificial directions. Within the decoding course of, we suggest two complementary methods, Self-Rubrics and Contrastive Filtering, to boost artificial knowledge high quality. Self-Rubrics leverages the robust LLM to generate rubrics and actions to make artificial instruction tougher. Contrastive Filtering additional selects the directions to which the goal LLM (the LLM to be aligned) fails to reply properly. CodecLM achieves state-of-the-art efficiency on open-domain instruction-following benchmarks with varied LLMs, demonstrating its effectiveness in LLM alignment for diverse instruction distributions.
