Hacker News new | past | comments | ask | show | jobs | submit
nater5000 | on: Show HN: Needle2: 14MB agentic LLM for phones, wearables, smart home and robots
This is cool. I definitely think the "micro" sized LLM space is underappreciated, so it's always good to see work like this. I foresee a paradigm in some contexts where you have a hierarchy of LLMs, with more competent models actively training smaller models to solve specific tasks very efficiently, and something like this could be the smallest layer in that stack.

With that being said, the web demo is not particularly impressive. It really doesn't like anything I throw at it. I'm fine with accepting that fine-tuning is the solution to this, but I wonder if there's anything to gain from a bigger model? I know it's completely counter to the whole point of this, but a 14MB binary using 28MB of RAM seems unnecessarily small and pretty arbitrary.

Like, what does a 28MB binary get you? Or a 140MB binary? Or a 1.4MB binary? I'm guessing the choice of 14MB came from minimizing the size as much as possible while meeting certain requirements/performance expectations, but even a Pi 5 has plenty more room to spare. Curious if there's a good explanation for this (which I may have missed in my skim of the post).

HenryNdubuaku | parent | next
So, its not a general language model, focused on tool call strictly for tiny edge-devices. There are solutions everywhere for high-capacity devices, Needle is for sub-$200 devices.
anon373839 | root | parent | next
It seems to me that the model struggles to have enough general intelligence, knowledge, or reasoning capacity for arbitrary prompted tool calling. At this size, not surprising.

I am VERY interested in seeing how it could perform with some fine-tuning for a specific family of tools/tasks. That would be a great addition to the demo.

electroglyph | root | parent
i would assume a model this size would require finetuning tbh. even functiongemma recommends that.
fwipsy | root | parent
14mb? More like sub-$20 devices.
TomatoCo | root | parent
Most pi pico's come with 16mb of flash. I wonder what kind of performance that can eek out.
SequoiaHope | root | parent
Well running from QSPI flash (even the internal memory versions use SPI internally) so any inference would be very slow streaming from that compared to RAM. The featured article says: “With a peak session RAM around 28MB, Needle runs on newer microcontrollers like ESP32-S3.” So I don’t see this doing anything useful on a Pico. The Pico 2 (RP2350) for example has 520k of RAM.
Rohansi | root | parent
An ESP32 has the same amount of SRAM as the Pi Pico. You can hook up PSRAM to the Pi Pico just like ESP32 to get 16MB more RAM.
loading story #49256283
loading story #49253739