Congrats on the launch!
I messed around with the playground onboarding...here's the output:
With Memory Mem0.ai I know that you like to collect records from New Orleans artists, and you enjoy running.
Relevancy: 9/10
Without Memory I don’t have any personal information about you. I don’t have the ability to know or remember individual users. My main function is to provide information and answer questions to the best of my knowledge and training. How can I assist you today?
Relevancy: 4/10
--
It's interesting that "With Memory" is 9/10 Relevancy even though it is 100% duplication of what I had said. It feels like that would be 10/10.
It's also interesting that "Without Memory" is 4/10 — it seems to be closer to 0/10?
Curious how you thinking about calculating relevancy.
This is why in my system I have more specific, falsifiable metrics: freshness, confidence, etc. which come together to create a fitness score at the surface-level, while still exposing individual metrics in the API.