Real-world data collection is a big missing component at this stage. An obvious one is journalism where an AI might be able to write the most eloquent article in the world, but it can't get out on the street to collect the information. But it also applies to other areas, like if you ask an AGI to solve climate change, it'll need accurate data to come up with an accurate plan.
Of course it's also yet another case where the AI takes over the creative part and leaves us with the mundane part...
ASI will be able to design factories that can produce robots it also designed that it can then use as a remote sensor and manipulator network.
until there are someone crazy enough that put those robot access to LLM network that can execute and visualize real world, we fine
I remember someone sharing their bank account details and a new Twitter account with ChatGPT 3.5 just a few days after it was launched.
People are already talking about doing this. Some people (e/acc types esp.) are at least rhetorically ok with AI replacing humanity.