Hacker News new | past | comments | ask | show | jobs | submit
jrahmy | on: They’re made out of weights
A tokenizer is a deterministic string-matching program, it's not made out of weights in the same sense as a neural network itself.
davrosthedalek | parent
But it could be. It's just less efficient.