Trl Lora Without Regret Smollm3
🚀
Display tracking information
Lukas PRO
sirluk
AI & ML interests
None yet
Recent Activity
updated
a Space
about 2 months ago
sirluk/trl-lora-without-regret-smollm3
published
a Space
about 2 months ago
sirluk/trl-lora-without-regret-smollm3
commented on
their
article
4 months ago
Efficient LLM Pretraining: Packed Sequences and Masked Attention
Organizations
updated
a
Space
about 2 months ago
published
a
Space
about 2 months ago
commented on
Efficient LLM Pretraining: Packed Sequences and Masked Attention
4 months ago
Can you maybe share a full script to reproduce this error?
upvoted
an
article
7 months ago
Article
xLSTM-based time series model TiRex significantly outperforms competing models in forecasting accuracy
•
12
commented on
Efficient LLM Pretraining: Packed Sequences and Masked Attention
7 months ago
Yes, this sounds like the model just learns to not attend to earlier tokens when an eos token comes after. With a packed sequence mask you can enforce this explicitly by just masking out previous tokens.
commented on
Efficient LLM Pretraining: Packed Sequences and Masked Attention
8 months ago
Hey @shantanuagarwal , glad you enjoyed the article! Even though I havent tried it out myself you should be able to leverage pytorch flexattention api for this. Have a look at the tutorial here https://pytorch.org/blog/flexattention/. Section "Document Masking/Jagged Sequences" talks about these packed sequence masks.
upvoted
a
paper
about 1 year ago
authored
a
paper
about 1 year ago
upvoted
2
papers
about 1 year ago
published
an
article
about 1 year ago
Article
Efficient LLM Pretraining: Packed Sequences and Masked Attention
•
64
apply_chat_template method not working correctly for llama 3 tokenizer
3
#35 opened over 1 year ago
by
sirluk
liked
a
dataset
over 1 year ago
published
an
article
almost 2 years ago
Article
Multilabel Classification using Mistral-7B on a single GPU with quantization and LoRA
•
26