QRWKV in, Qwerky out

README.md

@@ -7,13 +7,13 @@ library_name: transformers
 
 ![image/png](https://cdn-uploads.huggingface.co/production/uploads/633e85093a17ab61de8d9073/dM-i7n313mUnY-fbmElVM.png)
 
-- Try out the model on [![Featherless](https://img.shields.io/badge/featherless--ai%2FQwerky--72B-Dummy?style=flat&label=Featherless&color=facc15)](https://featherless.ai/models/featherless-ai/Qwerky-72B)
+- Try out the model on [![Featherless](https://img.shields.io/badge/featherless--ai%2FQRWKV--72B-Dummy?style=flat&label=Featherless&color=facc15)](https://featherless.ai/models/featherless-ai/QRWKV-72B)
 - Model details from our blog post here! [![Substack](https://img.shields.io/badge/Substack-Dummy?style=flat&color=facc15)](https://substack.recursal.ai/p/qwerky-72b-and-32b-training-large)
 - This model was presented in [RADLADS: Rapid Attention Distillation to Linear Attention Decoders at Scale](https://huggingface.co/papers/2505.03005).
 
-Benchmarks is as follows for both Qwerky-QwQ-32B and Qwerky-72B models:
+Benchmarks is as follows for both QRWKV-QwQ-32B and QRWKV-72B models:
 
-| Tasks | Metric | Qwerky-QwQ-32B | Qwen/QwQ-32B | Qwerky-72B | Qwen2.5-72B-Instruct |
+| Tasks | Metric | QRWKV-QwQ-32B | Qwen/QwQ-32B | QRWKV-72B | Qwen2.5-72B-Instruct |
 |:---:|:---:|:---:|:---:|:---:|:---:|
 | arc_challenge | acc_norm | **0.5640** | 0.5563 | **0.6382** | 0.6323 |
 | arc_easy | acc_norm | 0.7837 | **0.7866** | **0.8443** | 0.8329 |
@@ -33,7 +33,7 @@ Since this model is not on transformers at the moment you will have to enable re
 ```py
 # ...
 
-model = AutoModelForCausalLM.from_pretrained("featherless-ai/Qwerky-72B", trust_remote_code=True)
+model = AutoModelForCausalLM.from_pretrained("featherless-ai/QRWKV-72B", trust_remote_code=True)
 
 # ...
 ```
@@ -43,7 +43,7 @@ Other than enabling remote code, you may run the model like a regular model with
 ```py
 from transformers import AutoModelForCausalLM, AutoTokenizer
 
-model_name = "featherless-ai/Qwerky-72B"
+model_name = "featherless-ai/QRWKV-72B"
 
 model = AutoModelForCausalLM.from_pretrained(
     model_name,
@@ -79,7 +79,7 @@ response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
 
 Linear models offer a promising approach to significantly reduce computational costs at scale, particularly for large context lengths. Enabling a >1000x improvement in inference costs, enabling o1 inference time thinking and wider AI accessibility.
 
-As demonstrated with our Qwerky-72B-Preview and prior models such as QRWKV6-32B Instruct Preview, we have successfully converted Qwen 2.5 72B into a RWKV variant without requiring a pretrain on the base model or retraining the model from scratch. Enabling us to test and validate the more efficient RWKV Linear attention with a much smaller budget. Since our preview, we have continued to refine our technique and managed to improve the model over the preview model iteration.
+As demonstrated with our QRWKV-72B-Preview and prior models such as QRWKV6-32B Instruct Preview, we have successfully converted Qwen 2.5 72B into a RWKV variant without requiring a pretrain on the base model or retraining the model from scratch. Enabling us to test and validate the more efficient RWKV Linear attention with a much smaller budget. Since our preview, we have continued to refine our technique and managed to improve the model over the preview model iteration.
 
 As with our previous models, the model's inherent knowledge and dataset training are inherited from its "parent" model. Consequently, unlike previous RWKV models trained on over 100+ languages, the QRWKV model is limited to approximately 30 languages supported by the Qwen line of models.