Infinite thinking (after </thinking> tag with updated quants
Hi!
I'm on llama.cpp 080b16199, silicon mac, GLM-4.7-Flash-UD-Q6_K_XL.gguf (after fix) running command:
./llama-server -c 65536 -fa on --jinja -m ~/Downloads/GLM-4.7-Flash-REAP-23B-A3B-UD-Q8_K_XL.gguf --repeat-penalty 1.0 --temp 1.0 --top-p 0.95 --min-p 0.01
And also with -fa off. And asking "Write a minimalistic and generic object pool class in JS" in llama.cpp web UI.
Each time model generates relatively long thinking trace for this question and then it continues to refine and re-think it's answer instead of short and targeted answer. It thinks after thinking.
Compared to MLX quants within LM studio I can conclude, that it's a huge difference. MLX models think fast and compact and then produce minimal helpful answer.
Best regards, let me know if I can help to debug and find the root cause, I can run some tests if required, and thank's for your work!
Try temp 0.7 and top p 1.0. those are the recommended settings for tool calling
Try temp 0.7 and top p 1.0. those are the recommended settings for tool calling
Thank you for suggestion. As far as I can see, the issue persists with these sampling parameters as well.
Here is an example:
Some refinement is injected right inside the code block within answer section. Thinking "counter" is extended even after the code, without any actual model thoughts.
Having the same issue sparadically!
Try to add either:
--chat-template-kwargs '{"enable_thinking": true}'
or
--chat-template-kwargs '{"enable_thinking": false}'
To get normal thinking or instruct model respectively.
Without specifying any of two, model produces intermixed thinking after closing think tag.
For me it's not an issue anymore, cause workaround with enable_thinking makes model work just fine. But I leave the issue open for unsloth to decide if there are any fixes required within chat template.
Best wishes.