Hacker News new | past | comments | ask | show | jobs | submit
peterlk | on: Sebastian Raschka's from-scratch video course on building reasoning models
I think this is an outcome of playing with systems that we don’t actually understand very well on a deep level. We have found recipes that work, and have used graduate descent (so many PhD theses) to understand and optimize these systems, but we’ve built a model of language that also encodes reasoning, and those things are extraordinarily complex. Providing proper mathematical rigor for their explanation requires, as you say, the level of commitment equivalent to a math degree. Also, because language is so complicated and subtle, you need highly precise statistical models of it, and that just takes a shitload of data; hence the scale. You can train toy models at home, but they will suck because you actually need massive amounts of data to capture the nuance that language requires. And that’s to say nothing of multimodal models.
billiam | parent | next
I think the level of interaction on forums like HN is a great example of the still-yawning gulf between LLMs and humans when it comes to encoding reasoning into language. Even with the self-selection that comes with posting here, think of how many times it is almost impossible to tell if posters are shit-posting, serious, angry, sock puppets, or just weird.
giardini | parent
You say "...because you actually need massive amounts of data to capture the nuance that language requires..." but humans don't require that! So why do LLMs?

How many books did you read in your education? Far fewer that what LLMs read, no doubt. Of course we "have a life" and so we interact with hundreds of people, which is also a form of learning.

But in any case there's are fundamental differences here between these two systems (LLMs and humans) that we have not yet discerned clearly.

jodrellblank | root | parent | next
Far fewer than an LLM but then I “know” far less than an LLM. Sam Altman talked about a PhD level expert on any topic; if we expect a SOTA model to answer any question on any topic in a library, presumably that requires a library worth of books go into it?

Say I ask it who are the most beloved Indian cricketers and why, it might not be correct but it will have the right vocabulary - cricket terms and gameplay I’ve not heard of, player names and styles I don’t recognise, things critics and commentators like and dislike about game strategies. And it is like that for almost any topic.

If they hallucinate a Python function, it will look like a plausible function e.g. “hn.create_user_account()”. They don’t hallucinate like

    # create HN user account
    # requires eye of newt and frog leg
    “Oh great spirit of the internet I request and command you make a new account…”
It’s a terrific amount of facts to be able to write even plausibly about tons of topics.
loading story #50254991
loading story #50254168
loading story #50252215
ordersofmag | root | parent
I think it's more accurate to say that the fundamental differences (between LLM's and humans) are abundantly clear and what we haven't really got a handle on are the similarities. Why is it that an LLM's can so convincingly mimic some things that humans do? Of course the fact that we don't really understand how humans do it in the first place is a real barrier. Perhaps looking at the intersection will shed some light on how humans work.