Push model using huggingface_hub.

README.md

@@ -1,41 +1,9 @@
 ---
-license: cc-by-4.0
-language:
-- en
+tags:
+- model_hub_mixin
+- pytorch_model_hub_mixin
 ---
 
- # ARC-Encoder models
-
- This page houses `ARC8-Encoder_multi` from three different versions of pretrained ARC-Encoders. Architectures and methods to train them are described in the paper *ARC-Encoder: learning compressed text representations for large language models* available [here](https://github.com/kyutai-labs/ARC-Encoder/blob/main/ARC_Encoder_preprint.pdf). A code to reproduce the pretraining, further fine-tune the encoders or even evaluate them on dowstream tasks is available at [ARC-Encoder repository](https://github.com/kyutai-labs/ARC-Encoder/tree/main).
-
- ## Models Details 
-
- All the encoders released here are trained on web crawl filtered using [Dactory](https://github.com/kyutai-labs/dactory) based on a [Llama3.2-3B](https://github.com/meta-llama/llama-cookbook) base backbone. It consists in two ARC-Encoder specifically trained for one decoder and one for two decoders in the same time:
-- `ARC8-Encoder_Llama`, trained on 2.6B tokens on [Llama3.1-8B](https://github.com/meta-llama/llama-cookbook) base specifically with a pooling factor of 8.    
-- `ARC8-Encoder_Mistral`, trained on 2.6B tokens on [Mistral-7B](https://github.com/mistralai/mistral-finetune?tab=readme-ov-file) base specifically with a pooling factor of 8.    
-- `ARC8-Encoder_multi`, trained by sampling among the two decoders with a pooling factor of 8. 
-
- ### Uses
-
- As described in the [paper](https://github.com/kyutai-labs/ARC-Encoder/blob/main/ARC_Encoder_preprint.pdf), the pretrained ARC-Encoders can be fine-tuned to perform various downstream tasks.
-You can also adapt an ARC-Encoder to a new pooling factor (PF) by fine-tuning it on the desired PF.
-For optimal results, we recommend fine-tuning toward a lower PF than the one used during pretraining.
-To reproduce the results presented in the paper, you can use our released fine-tuning dataset, [ARC_finetuning](https://huggingface.co/datasets/kyutai/ARC_finetuning).
-
- ### Licensing
-
- ARC-Encoders are licensed under the CC-BY 4.0 license.
- 
-Terms of use: As the released models are pretrained from Llama3.2 3B backbone, ARC-Encoders are subject to the Llama Terms of Use found at [Llama license](https://www.llama.com/license/).
-
- ## Citations
-
- If you use one of these models, please cite:
-
- ```bibtex
-@techreport{pilchen2025arc_encoder,
-  title={ARC-Encoder: learning compressed text representations for large language models},
-  author={Pilchen, Hippolyte and Grave, Edouard and P{\'e}rez, Patrick},
-  year={2025}
-}
-```
+This model has been pushed to the Hub using the [PytorchModelHubMixin](https://huggingface.co/docs/huggingface_hub/package_reference/mixins#huggingface_hub.PyTorchModelHubMixin) integration:
+- Library: [More Information Needed]
+- Docs: [More Information Needed]

config.json

@@ -1,43 +1,7 @@
 {
-  "bridge_args": {
-    "bridge_type": "multi_module",
-    "hidden_dim": 2048,
-    "in_dim": 3072,
-    "out_dim": 4096
-  },
   "embedder": null,
-  "embedder_args": {
-    "causal_embedder": false,
-    "compress_rates": [
-      -8
-    ],
-    "cont_tok": true,
-    "memory_tokens": 0,
-    "n_truncated_layers": 2,
-    "pooling_module": {
-      "pool_type": "mean_pooled_queries",
-      "where": "before"
-    },
-    "rec_tok": true,
-    "train_embedding_mtx": true,
-    "trained_layers": 27
-  },
+  "embedder_args": null,
   "empty_init": 2,
   "llms": [],
-  "model_args": {
-    "_sliding_window": null,
-    "dim": 3072,
-    "head_dim": 128,
-    "hidden_dim": 8192,
-    "max_batch_size": 1,
-    "model_type": "transformer",
-    "n_heads": 24,
-    "n_kv_heads": 8,
-    "n_layers": 28,
-    "non_parametric_norm": false,
-    "norm_eps": "1e-05",
-    "rope_theta": 500000.0,
-    "sliding_window": null,
-    "vocab_size": 128256
-  }
+  "model_args": null
 }

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1e8fbebeb7d71d087c70d74c7835fe82abdbe0327fb0f5d7fa3ef825ee82322c
+size 12163149040