File size: 10,159 Bytes
8858287 b2f5e9c c2f9de6 8858287 b2f5e9c 3087655 b2f5e9c b3045b5 8858287 3087655 b3045b5 8858287 b3045b5 b2f5e9c b3045b5 8858287 b3045b5 8858287 914ac8f b3045b5 914ac8f b3045b5 914ac8f b3045b5 914ac8f |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 |
---
base_model:
- deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B
datasets:
- yolay/RAIF-ComplexInstruction-DeepSeek
library_name: transformers
metrics:
- accuracy
pipeline_tag: text-generation
license: apache-2.0
language: en
---
# Incentivizing Reasoning for Advanced Instruction-Following of Large Language Models
This model is the official implementation of the paper [Incentivizing Reasoning for Advanced Instruction-Following of Large Language Models](https://huggingface.co/papers/2506.01413).
You can find the official code and more details on the [GitHub repository](https://github.com/yuleiqin/RAIF).
## Abstract
Existing large language models (LLMs) face challenges of following complex instructions, especially when multiple constraints are present and organized in paralleling, chaining, and branching structures. One intuitive solution, namely chain-of-thought (CoT), is expected to universally improve capabilities of LLMs. However, we find that the vanilla CoT exerts a negative impact on performance due to its superficial reasoning pattern of simply paraphrasing the instructions. It fails to peel back the compositions of constraints for identifying their relationship across hierarchies of types and dimensions. To this end, we propose RAIF, a systematic method to boost LLMs in dealing with complex instructions via incentivizing reasoning for test-time compute scaling. First, we stem from the decomposition of complex instructions under existing taxonomies and propose a reproducible data acquisition method. Second, we exploit reinforcement learning (RL) with verifiable rule-centric reward signals to cultivate reasoning specifically for instruction following. We address the shallow, non-essential nature of reasoning under complex instructions via sample-wise contrast for superior CoT enforcement. We also exploit behavior cloning of experts to facilitate steady distribution shift from fast-thinking LLMs to skillful reasoners. Extensive evaluations on seven comprehensive benchmarks confirm the validity of the proposed method, where a 1.5B LLM achieves 11.74% gains with performance comparable to a 8B LLM. Evaluation on OOD constraints also confirms the generalizability of our RAIF.
## Usage
This model can be loaded and used directly with the Hugging Face `transformers` library.
```python
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch
model_name = "yolay/RAIF-DeepSeek-Qwen-1.5B"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
model_name,
torch_dtype=torch.bfloat16,
device_map="auto"
)
# Example for instruction following
messages = [
{"role": "user", "content": "Explain the concept of quantum entanglement in simple terms, using an analogy from everyday life."}
]
# Apply chat template for proper input formatting
input_ids = tokenizer.apply_chat_template(
messages,
add_generation_prompt=True,
return_tensors="pt"
).to(model.device)
# Generate response using parameters from generation_config.json
outputs = model.generate(
input_ids,
max_new_tokens=256,
do_sample=True,
temperature=0.6,
top_p=0.9
)
generated_text = tokenizer.decode(outputs[0][input_ids.shape[-1]:], skip_special_tokens=True)
print(generated_text)
```
## Model Details
The model DeepSeek-Qwen2.5-1.5B is our optimized model for its advanced instruction-following capabilities under complex instructions. It corresponds to the **DeepSeek-Qwen1.5B (Ours)** in the Table 1.
**Table 1** Performance on seven instruction benchmarks. Best/2nd best are marked **bold**/<u>underlined</u>.
| Model | Method | IFEval | CELLO | CF Bench | Complex Bench | FB Bench | Follow Bench | Info Bench | Avg. |
|------------------------|----------|--------|-------|----------|--------------|----------|--------------|------------|--------------|
| Qwen2.5-1.5B-Instruct | I/O | 45.28 | 71.00 | 36.00 | 50.97 | 39.81 | 40.00 | 71.24 | 50.61 |
| Qwen2.5-1.5B-Instruct | CoT | 28.65 | 59.30 | 22.00 | 32.94 | 37.31 | 29.28 | 62.22 | 38.81 (-11.79%) |
| Qwen2.5-1.5B-Instruct | SDC | 41.95 | 66.10 | 30.00 | 41.70 | 36.52 | 37.39 | 67.55 | 45.89 (-4.71%) |
| Qwen2.5-1.5B-Instruct | SFT | 65.61 | 71.20 | 48.00 | 57.46 | 42.75 | 56.47 | 76.22 | 59.67 (+9.06%) |
| Qwen2.5-1.5B-Instruct | Ours | 44.91 | 73.50 | 53.66 | 63.92 | 58.67 | 59.82 | 81.95 | 62.35 (+11.74%) |
| DeepSeek-Qwen1.5B | I/O† | 36.04 | 62.50 | 27.99 | 39.89 | 34.51 | 20.29 | 52.00 | 39.03 |
| DeepSeek-Qwen1.5B | SFT | 45.29 | 63.20 | 25.33 | 35.53 | 37.59 | 22.18 | 51.96 | 40.15 (+1.12%) |
| DeepSeek-Qwen1.5B | Ours | 57.67 | 69.00 | 40.00 | 44.38 | 37.78 | 37.79 | 60.48 | 49.58 (+10.54%) |
| DeepScaleR-1.5B | I/O† | 41.77 | 65.00 | 30.00 | 40.70 | 40.24 | 26.01 | 60.31 | 43.43 |
| DeepScaleR-1.5B | SFT | 48.24 | 62.90 | 28.00 | 36.68 | 35.72 | 26.50 | 54.22 | 41.75 (-1.67%) |
| DeepScaleR-1.5B | Ours | 55.63 | 67.30 | 39.33 | 43.23 | 37.81 | 36.80 | 60.08 | 48.60 (+5.17%) |
| Qwen2.5-7B-Instruct | I/O | 72.82 | 76.50 | 64.33 | 74.47 | 59.29 | 75.03 | <u>85.60</u> | <u>72.58</u> |
| Qwen2.5-7B-Instruct | CoT | 69.50 | 75.20 | 61.66 | 72.00 | 42.65 | 74.86 | 82.13 | 68.28 (-4.29%) |
| Qwen2.5-7B-Instruct | SDC | 60.44 | 72.60 | **65.66**| <u>76.53</u> | <u>60.07</u> | **76.09** | **86.88** | 71.18 (-1.39%) |
| Qwen2.5-7B-Instruct | SFT | 72.45 | <u>77.50</u> | 63.33 | 74.23 | 58.76 | 75.92 | 84.31 | 72.36 (-0.21%) |
| Qwen2.5-7B-Instruct | Ours | 70.06 | **79.20** | <u>65.00</u> | **77.40** | **64.45** | 75.32 | 82.67 | **73.44** (+0.85%) |
| Ministral-8B-Instruct | I/O | 59.51 | 76.20 | 62.33 | 70.03 | 54.54 | 73.49 | 84.00 | 68.58 |
| Ministral-8B-Instruct | CoT | 48.79 | 61.90 | 49.66 | 61.31 | 39.17 | 61.75 | 79.73 | 57.47 (-11.11%) |
| Ministral-8B-Instruct | SDC | 58.59 | 63.60 | 56.99 | 68.32 | 48.06 | 69.37 | 84.08 | 64.14 (-4.43%) |
| Ministral-8B-Instruct | SFT | 68.57 | 66.30 | 48.66 | 67.20 | 37.26 | 54.37 | 76.62 | 59.85 (-8.72%) |
| Ministral-8B-Instruct | Ours | 72.64 | 72.6 | 59.33 | 70.45 | 54.35 | <u>76.08</u> | 75.33 | 68.68 (+0.10%) |
| DeepSeek-Qwen7B | I/O† | 60.81 | 72.39 | 57.99 | 66.86 | 59.59 | 62.80 | 79.64 | 65.73 |
| DeepSeek-Qwen7B | SFT | 67.09 | 69.10 | 58.66 | 58.42 | 55.60 | 65.96 | 79.15 | 64.85 (-0.88%) |
| DeepSeek-Qwen7B | Ours | 71.35 | 71.40 | 58.67 | 62.04 | 59.65 | 59.38 | 82.00 | 66.35 (+0.62%) |
---
**Table 2** Performance on ComplexBench (Qwen2.5-7B-Instruct). Best/2nd best are marked **bold**/<u>underlined</u>. OD, SC, CNFR, FC, and SR stand for Oracle Decomposition, Self-Consistency, Conifer, FollowComplex, and Self-Refine.
| Category | ND | I/O | OD | SC | CNFR | FC | SR | Ours |
|------------------|------|--------|--------|--------|--------|--------|--------|---------|
| And | 1 | __85.85__ | 84.27 | 84.03 | 75.10 | 84.77 | 85.66 | **86.57** |
| **Chain** | | | | | | | | |
| | 1 | 72.18| __74.68__ | 73.54 | 60.95 | 66.27 | **75.25** | 73.96 |
| | 2 | 70.56| 72.70 | 69.63 | 64.43 | 70.66 | __73.07__ | **76.88** |
| *Avg.* | - | 70.96 | 73.18 | 70.57 | 63.59 | 69.60 | __73.59__ | **76.18** |
| **Selection** | | | | | | | | |
| | 1 | **77.25** | __76.61__ | 72.08 | 60.52 | 71.67 | 69.61 | 73.39 |
| | 2 | 65.61| __71.83__ | 68.23 | 53.25 | 61.96 | 64.34 | **72.92** |
| | 3 | __63.39__ | **68.45** | 56.13 | 46.04 | 51.70 | 58.67 | 60.75 |
| *Avg.* | - | 65.67 | **70.49** | 65.83 | 51.92 | 60.92 | 62.69 | __69.16__ |
| **Selection & Chain** | | | | | | | | |
| | 2 | __65.64__ | **65.94** | 60.81 | 47.33 | 61.07 | 52.01 | 61.06 |
| | 3 | 59.70| **65.77** | 64.08 | 48.53 | 57.65 | 60.41 | __65.00__ |
| *Avg.* | - | 62.68 | **65.85** | 62.44 | 47.93 | 59.36 | 56.20 | __63.03__ |
| **Overall** | - | 74.47 | __76.26__ | 73.76 | 63.51 | 71.97 | 74.00 | **77.40** |
---
## Acknowledgement🫡
In this project, we follow the SimpleRL and the OpenRLHF framework to prepare the codebase. We acknowledge their great work for open-sourcing the implementations of reinforcement learning algorithms.
* [[SimpleRL](https://github.com/hkust-nlp/simpleRL-reason/)]
* [[OpenRLHF](https://github.com/OpenRLHF/OpenRLHF)]
We also would like to express gratitude to the research community that organize the existing benchmarks for validating the LLMs of solving complex instructions.
## License🪪
Please refer to `License_RAIF` for the license of this project.
## Citation🎓
If you find this work useful, please consider the following citation:
```
@article{qin2025incentivizingreasoningadvancedinstructionfollowing,
title={Incentivizing Reasoning for Advanced Instruction-Following of Large Language Models},
author={Yulei Qin and Gang Li and Zongyi Li and Zihan Xu and Yuchen Shi and Zhekai Lin and Xiao Cui and Ke Li and Xing Sun},
year={2025},
eprint={2506.01413},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2506.01413}
}
``` |