I tested with
import needle
@needle.tool
def add(a: int, b: int):
"Add two numbers."
return a + b
agent = needle.Needle(tools=[add])
print(agent.run("calculate 1 + 1?")["reasoning"])
conclusion: completly useless
Try the following tool description: "Calculate the sum of two numbers. Use for any arithmetic or math question." instead of "Add two numbers." Let me know how it goes, thanks!
It does better with clearer tool description, but we are taking note of these complaints for future improvements.
works better that way, thanks :)
but still struggle when changing the quesion:
import needle
@needle.tool
def add(a: int, b: int):
"Calculate the sum of two numbers. Use for any arithmetic or math question."
return a + b
agent = needle.Needle(tools=[add])
print(agent.run("what is 5 + 7?")["reasoning"])
Ah, another failure point on our end! So a simple "5 + 7" and "add 5 and 7" works. But to handle ambiguity, the python package ships pipelines to synthesize augmentations and fine-tune on your samples for robustness. Just run "needle playground" and use the UI. Apologies.
Thanks, we shall improve this for the next release.