Update README.md

README.md

@@ -26,6 +26,7 @@ Notes:
 <br>\- Reminder, this isn't a native 32K model. It has it's issues, but it's coherent and working well.
 
 Sanity Check // Needle in a Haystack Results:
+<br>\- This is not as complex as RULER or NIAN, but it's a basic evaluator. Some improper train examples had Haystack scores ranging from Red to Orange for most of the extended contexts.
 ![Results](https://huggingface.co/Sao10K/L3-8B-Stheno-v3.3-32K/resolve/main/haystack.png)
 
 Wandb Run:
@@ -39,6 +40,7 @@ Relevant Axolotl Configurations:
 <br>\- 2M Rope Theta had the best loss results during training compared to other values.
 <br>\- Leaving it at 500K rope wasn't that much worse, but 4M and 8M Theta made the grad_norm values worsen even if loss drops fast.
 <br>\- Mixing in Pretraining Data was a PITA. Made it a lot worse with formatting. -> Tried at low value mixes, eg. <20% and lower.
+<br>\- Pretraining / Noise made it worse at Haystack too? It wasn't all Green, Mainly Oranges.
 <br>\- Improper / Bad Rope Theta shows in Grad_Norm exploding to thousands. It'll drop to low values alright, but it's a scary fast drop even with gradient clipping.
 
 ```