Update README.md

README.md

@@ -15,12 +15,16 @@ tags:
 
 An encoder-decoder (T5 architecture) pretrained with [nanoT5](https://github.com/pszemraj/nanoT5/tree/flan-dataset):
 
-- tokenizer: custom llama2 with 48k vocab (from [vocab scaling laws](https://hf.co/collections/sail/scaling-laws-with-vocabulary-6699e0cbd77a8b2870859bfe))
+- tokenizer: sentencepiece BPE w/ byte fallback, 48k vocab (from [vocab scaling laws](https://hf.co/collections/sail/scaling-laws-with-vocabulary-6699e0cbd77a8b2870859bfe))
 - data: `fineweb-edu-dedup` split of `HuggingFaceTB/smollm-corpus`
 - context length: 1024 ctx
 
 ## details
 
+Detailed info, including training logs, configs, and checkpoints can be found under `checkpoints/` in this repo.
+
+<details>
+<summary><strong>Expand hyperparameter overview</strong></summary>
 
 1. Model:
    - Dropout rate: 0.0
@@ -46,6 +50,7 @@ An encoder-decoder (T5 architecture) pretrained with [nanoT5](https://github.com
 4. Hardware:
    - Device: RTX 4080
    - Precision: bfloat16, tf32
+</details>
 
 ## plots
 
@@ -55,6 +60,9 @@ training loss
 ![loss](./checkpoints/loss_over_steps.png)
 
 
+<details>
+<summary><strong>Expand grad and weights L2 norm plots</strong></summary>
+
 grad norm 
 
 ![grad](./checkpoints/grad_l2_over_steps.png)
@@ -65,5 +73,6 @@ weights norm
 
 ![weights](./checkpoints/weights_l2_over_steps.png)
 
+</details>
 
 ---