How to use CPU Offload for this model? I keep getting OOM
#4
by
crystech
- opened
I tried using --cpu-offload-gb 100GB but still it is causing OOM.
using 8 x RTX 5090 barely fit hence i am thinking to use some cpu offload to try it out but unless i set context length below 12000 i would always OOM.
am i doing something wrong as it doesn't seem to offload to cpu ram