Hacker News new | past | comments | ask | show | jobs | submit
MathMonkeyMan | on: Unit tests as documentation
I was going to say that unit tests have the benefit of breaking when the truth changes.

But then I realized that a lot of what makes a set of tests good documentation is comments, and those rot, maybe worse than dedicated documentation.

Keeping documentation up to date is a hard problem that I haven't yet seen solved in my career.

rglover | parent | next
The only fix for that is discipline. You can't automate away quality. The best people/teams understand that and make good docs a feature requirement, not an afterthought.

My favorite example is Stripe. They've never skimped on docs and you can tell they've made it a core competency requirement for their team.

hitchdev | root | parent | next
I dont think it is about discipline. Discipline is required if you're duplicating tedious work, not for creativity.

At its core, a good test will take an example and do something with it to demonstrate an outcome.

That's exactly what how to docs do - often with the exact same examples.

Logically, they should be the same thing.

You just need a (non turing complete) language that is dual use - it generates docs and runs tests.

For example:

https://github.com/crdoconnor/strictyaml/blob/master/hitch/s...

And:

https://hitchdev.com/strictyaml/using/alpha/scalar/email-and...

invaderzirp | root | parent
No, you just need to both understand how your system works and then clearly write down what it's doing and why. If projects like Postgres and SQLite and musl libc and the Linux kernel can all do it, I think the CRUD app authors can do it, too. But it's not magic, and another tool won't solve it (source: I've seen a hundred of these tools on scores of teams, and they don't help when people have no clue what's happening and then they don't write anything down).
MathMonkeyMan | root | parent
I wonder if there's some conservation law for "concerted mental effort." As if by spending time and energy on the often exasperating task of keeping documentation relevant, you reduce the time and energy required to comprehend the system.

You're right, it is a matter of culture and discipline. It's much harder to maintain a consistent and legible theory of a software component than it is to wing it with your 1-2 other teammates. Naming things is hard, especially when the names and their meanings eventually change.

sbuttgereit | parent | next
Elixir's documentation (ExDoc) & unit testing framework (ExUnit) doesn't solve this problem but provides a facility to ease it a bit.

In the documentation, you can include code examples that, if written a certain way, not only looks good when rendered but can also be tested for their form and documented outputs as well. While this doesn't help with the descriptive text of documentation, at least it can flag you when the documented examples are no longer valid... which can in turn capture your attention enough to check out the descriptive elements of that same area of documentation.

This isn't to say these documentation tests are intended to replace regular unit tests: these documentation tests are really just testing what is easily testable to validate the documentation, the code examples.

Something can be better than nothing and I think that's true here.

starkparker | parent | next
Not that this solves the hard problem, but there's a simonw post for that: https://simonwillison.net/2018/Jul/28/documentation-unit-tes...

Including screenshots, which a lot of tech writing teams raise as a maintenance burden: https://simonwillison.net/2022/Oct/14/automating-screenshots...

Then there are tools like Doc Detective to inline tests in the docs, making them dependent on each other; if documented steps stop working, the test derived from them fails: https://doc-detective.com/

Ygg2 | parent
> Keeping documentation up to date is a hard problem that I haven't yet seen solved in my career.

Rust doctests. They unite documentation and unit test. Basically documentation that's never so out of sync its assert fail.