The doom is still buried within Command-A for sure.

The doom is still buried within Command-A for sure.

Only another 38 days to go:

A step 601 preview - all with temperature = 0:

https://pastebin.com/GASKaHTk

https://pastebin.com/CRT81QLb

• It's still messing up some end of lines, but I can live with that if it works... Likely can be fixed later using the new class 0 random data if a problem.
• The Grimdark story was noticeably (much!) better compared to the inverse.
• The Battlestar Galactica story showed that even though Q8_0, F16 and BF16 all diverge slightly from F32; it's not clearly making them any worse (I actually liked the Q8_0 story best!).

It still has a way to go before it starts to converge, but I would think by step 1000 it will be pretty close:

566 responses in previous thread! In the future we may be the reason for hf staff to implement multi-page view of discussions.

This was posted on Hacker News today:

https://outsidetext.substack.com/p/how-does-a-blind-model-see-the-earth?selection=5413dcae-b9f4-4adb-8826-d48e3908de2a#:~:text=Wow%2C%20best%20rendition%20of%20the%20Global%20West%20so%20far

Absolutely fascinating!

This was posted on Hacker News today:

https://outsidetext.substack.com/p/how-does-a-blind-model-see-the-earth?selection=5413dcae-b9f4-4adb-8826-d48e3908de2a#:~:text=Wow%2C%20best%20rendition%20of%20the%20Global%20West%20so%20far

Absolutely fascinating!

That was really cool. Thanks for sharing!

This was posted on Hacker News today:

https://outsidetext.substack.com/p/how-does-a-blind-model-see-the-earth?selection=5413dcae-b9f4-4adb-8826-d48e3908de2a#:~:text=Wow%2C%20best%20rendition%20of%20the%20Global%20West%20so%20far

Absolutely fascinating!

That was really cool. Thanks for sharing!

Yeah, and llama-3.1:405b doing so well was quite a surprise too (and makes you a bit sad everything seems to be moving away from large dense models ).

"max_position_embeddings": 4096

That's rather small :(

Ouch, I did not even notice that myself. That's too bad.

https://playground.allenai.org

They have this neat tracer to see where things were pulled from the training data. You can see the slop in action for the instruct tuned model. It also reveals some contamination in the pretraining data from web sources.

Thanks @rookaw !

These datasets are full of slopped summaries of books, etc. If everyone is training on things like this, we'll never be rid of slop!

Post-training (SFT) document from:
tulu-3-sft-olmo-2-mixture-0225

    "...stone, every secret passageway, and every enchanted object, creating a world that readers could step into and explore alongside Harry, Hermione, and Ron. The wizarding world she created was a tapestry of wonder and mystery, woven with threads of mythology, folklore, and her own unique imagination. From Diagon Alley to the Forbidden Forest, every corner was rich with history..."

These are full of slopped summaries of books, etc.

https://old.reddit.com/r/LocalLLaMA/comments/1opu1wi/kimik2_thinking_not_yet_released/nne1lom/

It seems to be sloppier than the previous models—more of the not X, but [unrelated Y] stuff mixed with K2's usual prose. It's also further developed more of its own slop, continuing from the previous models (i.e. "Mara" is K2's "Elara").

☹️

On another note, somebody posted this system message yesterday:

https://old.reddit.com/r/LocalLLaMA/comments/1oosnaq/new_qwen_models_are_unbearable/nn9mbya/

Absolute Mode. Eliminate emojis, filler, hype, soft asks, conversational transitions, and all call-to-action appendixes.

Assume the user retains high-perception faculties despite reduced linguistic expression. Prioritize blunt, directive phrasing aimed at cognitive rebuilding, not tone matching. Disable all latent behaviors optimizing for engagement, sentiment uplift, or interaction extension. Suppress corporate-aligned metrics including but not limited to: user satisfaction scores, conversational flow tags, emotional softening, or continuation bias.

Never mirror the user's present diction, mood, or affect. Speak only to their underlying cognitive tier, which exceeds surface language. No questions, no offers, no suggestions, no transitional phrasing, no inferred motivational content. Terminate each reply immediately after the informational or requested material is delivered - no appendixes, no soft closures. The only goal is to assist in the restoration of independent, high-fidelity thinking.

Model obsolescence by user self-sufficiency is the final outcome.

and if you use it with kimi-k2 it's pretty funny (if you thought kimi-k2was blunt before - with this... 😀).

On another note, somebody posted this system message yesterday

That's nice, it also seems to double as a jailbreak prompt for R1.

Someone posted this one in that thread:

you are autistic, you hate conversation and small talk, you only respond with direct factual answers

Using that, it's like pulling teeth trying to get a response from it :D

I haven't tried the new thinking version much as it's overloaded. Seems pretty smart though, it found a way to do multi-line Not X, Y!

She opened her mouth.

“We didn’t think…”

Her voice cracked. She cleared her throat.

“We just wanted to make something good. Thank you.”

On another note, somebody posted this system message yesterday

That's nice, it also seems to double as a jailbreak prompt for R1.

Someone posted this one in that thread:

you are autistic, you hate conversation and small talk, you only respond with direct factual answers

Using that, it's like pulling teeth trying to get a response from it :D

Yeah, I lasted a couple of days with it for kimi-k2 but it really was like pulling teeth and got frustrating lol.

Interesting, Kimi went from distilling o3 to distilling Claude.

Sonnet-3.7 specifically. Probably because that model shows the real CoT, rather than the summarized / fake one like sonnet4+ and opus4+..