This technique is not a better lesson pilled at all. Waste of time when the model will just learn to do this anyways.
Kian Kyars PRO
kyars
·
AI & ML interests
None yet
Recent Activity
commented on
an
article
about 1 month ago
Efficient LLM Pretraining: Packed Sequences and Masked Attention
commented on
a paper
3 months ago
Treasure Hunt: Real-time Targeting of the Long Tail using Training-Time
Markers
commented on
a paper
3 months ago
Treasure Hunt: Real-time Targeting of the Long Tail using Training-Time
Markers
Organizations
commented on
Efficient LLM Pretraining: Packed Sequences and Masked Attention
about 1 month ago
commented
2
papers
3 months ago
updated
a
Space
3 months ago
liked
2
models
3 months ago
awesome resource
#1 opened 3 months ago
by
kyars
commented on
SmolLM3: smol, multilingual, long-context reasoner
4 months ago
thanks!
commented on
SmolLM3: smol, multilingual, long-context reasoner
4 months ago
makes sense
commented on
SmolLM3: smol, multilingual, long-context reasoner
4 months ago
For the reasoning mid-training we used the ChatML template (so no system prompt), for SFT we use SmolLM3's final chat template.
ChatML does have a system prompt, can you please elaborate?
commented on
SmolLM3: smol, multilingual, long-context reasoner
4 months ago
is it not possible to do on-policy distillation?
commented on
SmolLM3: smol, multilingual, long-context reasoner
4 months ago
nice thoughts
commented on
SmolLM3: smol, multilingual, long-context reasoner
4 months ago
tied embeddings were just used because llama uses it or was there an ablation?
upvoted
an
article
4 months ago
Article
SmolLM3: smol, multilingual, long-context reasoner
•
721
commented on
SmolLM3: smol, multilingual, long-context reasoner
4 months ago
Let's do this with Muon.
TP Question
2
#113 opened 5 months ago
by
kyars
published
a
Space
5 months ago
updated
a
Space
5 months ago
published
a
Space
5 months ago
commented
a
paper
6 months ago