- What is it?
- What does it do?
- Why does it do that?
- What is the API?
- What does it return?
- What are some examples of proper, real world usage (that don't involve foo/bar but instead, real world inputs/outputs I'd likely see)?
But then I realized that a lot of what makes a set of tests good documentation is comments, and those rot, maybe worse than dedicated documentation.
Keeping documentation up to date is a hard problem that I haven't yet seen solved in my career.
My favorite example is Stripe. They've never skimped on docs and you can tell they've made it a core competency requirement for their team.
The one lesson I have learned over my career: Don't work in teams (or for managers) that rely on discipline to get things done. Every time I've encountered them, they've been using it as an excuse to avoid better processes.
Sure, some counterexamples exist. Chances are, those counterexamples aren't where a given reader of your comment is working.
At its core, a good test will take an example and do something with it to demonstrate an outcome.
That's exactly what how to docs do - often with the exact same examples.
Logically, they should be the same thing.
You just need a (non turing complete) language that is dual use - it generates docs and runs tests.
For example:
https://github.com/crdoconnor/strictyaml/blob/master/hitch/s...
And:
https://hitchdev.com/strictyaml/using/alpha/scalar/email-and...
You're right, it is a matter of culture and discipline. It's much harder to maintain a consistent and legible theory of a software component than it is to wing it with your 1-2 other teammates. Naming things is hard, especially when the names and their meanings eventually change.
In the documentation, you can include code examples that, if written a certain way, not only looks good when rendered but can also be tested for their form and documented outputs as well. While this doesn't help with the descriptive text of documentation, at least it can flag you when the documented examples are no longer valid... which can in turn capture your attention enough to check out the descriptive elements of that same area of documentation.
This isn't to say these documentation tests are intended to replace regular unit tests: these documentation tests are really just testing what is easily testable to validate the documentation, the code examples.
Something can be better than nothing and I think that's true here.
Including screenshots, which a lot of tech writing teams raise as a maintenance burden: https://simonwillison.net/2022/Oct/14/automating-screenshots...
Then there are tools like Doc Detective to inline tests in the docs, making them dependent on each other; if documented steps stop working, the test derived from them fails: https://doc-detective.com/
Rust doctests. They unite documentation and unit test. Basically documentation that's never so out of sync its assert fail.
This could all easily fit in the top-level comments of a main() function or the help text of a CLI app.
- What is the API?
This could be gleaned from the code, either by reading it or by generating automatic documentation from it.
- What does it return?
This is commonly documented in function code.
- What are some examples of proper, real world usage (that don't involve foo/bar but instead, real world inputs/outputs I'd likely see)?
This is typically in comments or help text if it's a CLI app.
And what should be obvious or it’s still too complex.