Hacker News new | past | comments | ask | show | jobs | submit
kentm | on: SQLite Critical CVEs or LLM Slop?
Yes but the key here is doing proper risk assessment. "What is the consequence if the LLM gets this wrong?" "How do we verify the output?" "What are the legal ramifications for using the LLM in this way?" "Who is responsible when the LLM fails?" "Whats the expected accuracy here?" etc.

In the current AI mania, there's a lot of due diligence simply being ignored. Plenty of "Well humans make mistakes too!" going on here on HN too.

ivan_gammel | parent
The due diligence not being done is people putting cats in microwaves. It‘s not the dangerous part. The real danger is risk assessments coming to wrong conclusions, because it is still terra incognita. Talented engineers were in this situation before, doing mistakes with cars, airplanes, buildings etc.
kentm | root | parent
No, I'm sorry but I think thats a cop out. The fact that LLM are stochastic and can give incorrect answers is not particularly difficult to comprehend, and the risks that fall out of that are reasonably understandable. The issue is entirely down to bad choices by the people driving LLMs, because they are engaging with what they wish LLMs do instead of what they actually do.
ivan_gammel | root | parent
So you say that it is easy to understand what LLMs are, yet the issue is that people don‘t understand it. Either you put too many people in the bucket labelled „idiots“ or you are blowing that problem out of proportion, at the same time seeing only the tip of the iceberg.

Risks are not binary. Most people are not idiots. They do understand that LLMs aren‘t thinking humanly or 100% logically. You are talking about some sort of faith, I think this is more about trust, which is built on observations. Statistically, the outcomes of LLM work may hit the goal quite a lot. More of that in the beginning of the journey than in the end. For many reasonable people that becomes a trap, where minor, acceptable deviations accumulate into a fireball. Still, saying that LLMs are failure with inevitable fireball at the end and giving up is a trivial and stupid solution. Staying within razor-thin distance from that fireball and managing to deliver a working solution is what everyone tries at the moment. This is how it always happened and always worked. It‘s the art of engineering.