K_G_3.50 quant outperforming the K_G_4.00 quant I wonder if the 4 bit didn't quantize correctly?
Not sure why but the 4 bit hallucinates way more often than the 3.5 bit, gives strangely worded responses and has worse code quality. Using Identical settings so I am wondering if the 4 bit quantized incorrectly.
Hmm, there shouldn't be any problems with it, but this model is known for issues with Q4_K tensors in llama-cpp specifically on CUDA model loading so there might be more of those NaN issues than that one tensor Unsloth pointed out.
I ran all my internal testing on Vulkan so I didn't catch any of this though in my tests.
Hmm, there shouldn't be any problems with it, but this model is known for issues with Q4_K tensors in llama-cpp specifically on CUDA model loading so there might be more of those NaN issues than that one tensor Unsloth pointed out.
I ran all my internal testing on Vulkan so I didn't catch any of this though in my tests.
im also running vulkan it doesn't make any logical sense to me because higher bpw typically also means higher quality but I use the model daily and 3.5 bit is defiantly better. Really have no clue why.
im also running vulkan it doesn't make any logical sense to me because higher bpw typically also means higher quality but I use the model daily and 3.5 bit is defiantly better. Really have no clue why.
Interesting, yeah I am not sure why that would be. I overall didn't like this model for my personal use cases so I stopped using it after making the quants. Even at Q8 it was giving me rough results with lots of mistakes compared to Qwen3.5 397B
im also running vulkan it doesn't make any logical sense to me because higher bpw typically also means higher quality but I use the model daily and 3.5 bit is defiantly better. Really have no clue why.
Interesting, yeah I am not sure why that would be. I overall didn't like this model for my personal use cases so I stopped using it after making the quants. Even at Q8 it was giving me rough results with lots of mistakes compared to Qwen3.5 397B
Huh interesting for me the 3.5 bit as been incredibly close to sonnet 4.5 and has been getting a lot of things sonnet 4.6 gets wrong correct. Although claude has been nerfed lately. Feels like its 90% of what sonnet 4.5 was.