The chart at the top is somewhat confusing. It says, “lower is better” but the y-axis is inverted! So visually higher in the chart is better but lower in terms of # value.

Claude seems to forget what you tell it in very long work sessions (things that take weeks to develop), no matter how many times you tell it which part is extra important. I dont use goal (I guess I should), but presumably it makes it actually remember the most important instruction. I believe this here is about shorter sessions where the issue doesn't crop up as much.

/goal has replaced plan mode for me. This is the pattern I use for 95% of my AI work now:

1. Read X feature of Y and tell me when you fully understand it (if there's any detail missing in the summary, repeat until the context is primed)

2. What time is it?

3. /goal Spend X minutes from $time writing a technical design doc on $feature. There must not be any vague language or ambiguity in the document. Read carry_forward_requirements.md and testing_best_practices.md and explicitly incorporate them into the document you write. The document should be executable for a contextless implementer when done and include specific code and document references and changes needed. Spend the full X minutes working on and reviewing this document - do not quit early and wait

Even just spending 10 minutes forcing GPT to write a design doc results in much more robust plans than plan mode, in my experience, and saves time I would spend iterating on the initial plan mode draft anyway.

What is /goal?

Both are entirely useless on complex problem, because they have a bias in training data they can only partially detect in their own output. The answers are getting worse and worse as you dive deeper into the topic you are working on. I thought I could sharpen one of the documents I worked on using Opus 4.8 and GPT-5.5 together. Fable 5 and GPT-5.6 completely destroyed it. Not only it is not human readable anymore, but also doesn't make sense.

Results seem mostly noise to me. One eval per model, in a large problem space (i.e. a problem which requires many attempts to solve well).

If you're curious what the actual optimal Paris cost is, I suggest formulating the problem as an integer linear program and submitting it to Gurobi on NEOS [0]. Gurobi is arguably the strongest commercial ILP solver; big companies pay big dollars to use it to optimise schedules, industrial processes, etc. I'm not sure it could solve this problem to optimality in the 8 hours NEOS provides you, but it might -- KIRO has some similarities to the Vehicle Routing Problem, variants of which are very commercially important. In any case, Gurobi is a monster, and even if you don't get an exact solution, it will give you a lower bound (which may not be tight, but it's nonetheless interesting).

[0] https://neos-server.org/neos/

Just use OpenEvolve for such problems.

A deepdive on the /goal effect on a problem literally made for this.

I love that we have this on one hand and me cleaning up catastrophic CSS made by Sol on the other. Then again, maybe CSS is the ultimate benchmark.

Very interesting, will surely help my future projects

...is this not a Travelling Salesman Problem?

[dead]

[dead]

[dead]