update html
Browse files
architectures/incoder.txt
CHANGED
|
@@ -1,13 +1,19 @@
|
|
| 1 |
[InCoder](https://huggingface.co/facebook/incoder-6B) uses a decoder-only Transformer with Causal Masking objective, to train a left-to-right language model to fill in masked token segments, with a context length of 2048.
|
|
|
|
| 2 |
|
| 3 |
|Model | # parameters |
|
| 4 |
| - | - |
|
| 5 |
| Decoder |1.3B |
|
| 6 |
| Decoder |6.7B |
|
| 7 |
|
|
|
|
| 8 |
|
| 9 |
[Causal Masking objective](https://arxiv.org/abs/2201.07520) is a hybrid approach of Causal and Masked language models, "it combines the benefit of per-token generation with optional bi-directionality specifically tailored to prompting".
|
| 10 |
-
During the training of InCoder, spans of code were randomly masked and moved to the end of each file, which allows for bidirectional context. Figure
|
|
|
|
|
|
|
|
|
|
|
|
|
| 11 |
|
| 12 |
So in addition to program synthesis (via left-to-right generation), InCoder can also perform editing (via infilling). The model gives promising results in some zero-shot code infilling tasks such as type prediction, variable re-naming and comment generation.
|
| 13 |
|
|
|
|
| 1 |
[InCoder](https://huggingface.co/facebook/incoder-6B) uses a decoder-only Transformer with Causal Masking objective, to train a left-to-right language model to fill in masked token segments, with a context length of 2048.
|
| 2 |
+
<div align="center">
|
| 3 |
|
| 4 |
|Model | # parameters |
|
| 5 |
| - | - |
|
| 6 |
| Decoder |1.3B |
|
| 7 |
| Decoder |6.7B |
|
| 8 |
|
| 9 |
+
</div>
|
| 10 |
|
| 11 |
[Causal Masking objective](https://arxiv.org/abs/2201.07520) is a hybrid approach of Causal and Masked language models, "it combines the benefit of per-token generation with optional bi-directionality specifically tailored to prompting".
|
| 12 |
+
During the training of InCoder, spans of code were randomly masked and moved to the end of each file, which allows for bidirectional context. Figure below from InCoder [paper](https://arxiv.org/pdf/2204.05999.pdf) illustrates the training process.
|
| 13 |
+
|
| 14 |
+
<p align="center">
|
| 15 |
+
<img src="https://huggingface.co/datasets/loubnabnl/repo-images/raw/main/incoder.png" alt="drawing" width="220"/>
|
| 16 |
+
</p>
|
| 17 |
|
| 18 |
So in addition to program synthesis (via left-to-right generation), InCoder can also perform editing (via infilling). The model gives promising results in some zero-shot code infilling tasks such as type prediction, variable re-naming and comment generation.
|
| 19 |
|