Anthropic spent a lot more time editing their pre-training data with higher-quality material. For example, they used actual books for pre-training, because books are generally run through human editors for higher quality. (this is from the copyright lawsuits)
[ ... ]
Meanwhile its obvious that companies like Google would use trash data like email and chat logs or Internet forums for pre-training. Just complete trash data.
Google has been scanning every book they can get their hands on for 20 years or more. They have a bunch of patents on non-destructive book scanning gear. Surely they are using that data to train their models.won't help if they also include trash internet text in the training set as well.
it's kinda known that the gemini models write better english which goes against the "trash data" remarks.
Maybe not. If they’re trained on real world emails and Reddit comments, they’ll write more like “real” people, but they’ll be dumber.
loading story #50556771
Have you ever tried the Gemini models? They are absolutely atrocious. Like, hilariously bad. One time they kept asking me for "Harriet! Where is Harriet?!" and another time it responded in chinese.
They were not trained on good data.
And higher-quality source material results in higher-quality emergent behavior, everything from thinking to tool use.
Seriously people, read their copyright court case. It literally states why Anthropic focused on books and higher quality material.
Google has to learn to not use trash internet content if they want to catch up.
(note that I haven't tried Neon yet so maybe they learned their lesson and fixed it. All my experience is from 3 months on their Ultra plan and previous models from earlier this year)
Surely they are destroying scanned books for training
loading story #50555364
loading story #50561448
loading story #50554793