Hacker News new | past | comments | ask | show | jobs | submit
skull8888888 | on: #41453832
Hey there, apologies for the late reply.

> Could you clarify why would some need to analyze LLMs out of all the things?

When you want to understand trends of the output of your Agent / RAG on scale, without looking manually at each trace, you need to another LLM to process the output. For instance, you want to understand what is the most common topic discussed with your agent. You can prompt another LLM to extract this info, Laminar will host everything, and turn this data into metrics.

> Why do we need to evaluate LLMs?

You right, devs who want to evaluate output of the LLM apps, truly care about the quality or some other metric. For this kind of cases evals are invaluable. Good example would be, AI drive-through agents or AI voice agents for mortgages (use cases we've seen on Laminar)

Oras | parent
Topic modelling and classifications are real problems in LLM observability and evaluation, glad to see a platform doing this.

I see that you have chained prompts, does that mean I can define agents and functions inside the platform without having it in the code?

skull8888888 | root | parent
Yes! Our pipeline builder is pretty versatile. You can define conditional routing, parallel branches, and cycles. Right now we support LLM node and util nodes (json extractor). If you can defined your logic purely from those nodes (and in majority of cases you will be), then great, you can host everything on Laminar! You follow this guide (https://docs.lmnr.ai/tutorials/control-flow-with-LLM) it's bit outdated by gives you a good idea on how to create and run pipelines.