Yes, and also waiting for the next iteration of Gemma. Muse or Qwen are optimized for coding, while IMO Gemma is still better for non-coding tasks.
I am working on a project where we have to classify customer calls into more than 10 categories. As the client wants everything locally I tried a few local LLMs. Gemma turned out to be the best model for this task. The classification accuracy is impressive, and the client is happy that I am using an American model.
You can partially tell by the tokeniser; which gives you some hint into the training corpus mix.
</div> is four Gemma4 tokens, but one Qwen3.6 token.
{"deleted":true,"id":49248506,"parent":49243395,"time":1786390046,"type":"comment"}
Looks like we have a /r/localllama dweller here.
Where do you find this information for each model?
When you look on HuggingFace.co at the files of a model, for each model you will see a file "tokenizer.json".
In that file you can see all tokens and their corresponding numeric codes.
The tokenizers are included in the open s̶o̶u̶r̶c̶e̶ weights releases; you wouldn’t be able to use the weights without the corresponding encoder/decoder, in fact.