Hacker News new | past | comments | ask | show | jobs | submit
bfung | on: They’re made out of weights
And you know what the tokenizer is made of?

Weights.

jrahmy | parent
A tokenizer is a deterministic string-matching program, it's not made out of weights in the same sense as a neural network itself.
davrosthedalek | root | parent
But it could be. It's just less efficient.