(Un)officially support 1x96GB?
#4
by deleted - opened
Can't wait to try this, thanks.
Worth noting: M2.5 actually runs pretty well at UD-IQ2_M, in 96GB. A bit limited in context, but it works. That brings it within reach of non-commercial enthusiasts with 4x3090, or 3x32GB.
I notice that your recommended configs are for 4x96GB and 8x96GB, so you have the right hardware, and could "unofficially support" 1x96GB too :)
This is really just me begging you to keep (at least the lower end of) your models at 200-220B, even if you add multimodality in future. It's a great level of complexity, and just within reach for a lot of us, but I suppose it could help you get more attention too, if you unofficially support systems that more people have.
deleted changed discussion title from Officially support 1x96GB? to (Un)officially support 1x96GB?