Hacker News new | past | comments | ask | show | jobs | submit
Where do you find this information for each model?
When you look on HuggingFace.co at the files of a model, for each model you will see a file "tokenizer.json".

In that file you can see all tokens and their corresponding numeric codes.

The tokenizers are included in the open s̶o̶u̶r̶c̶e̶ weights releases; you wouldn’t be able to use the weights without the corresponding encoder/decoder, in fact.