Yea since writing this I think it has more to do with the regulator circuit. I plan to do a small rewrite and change the title to something like "When 3.3V isn't actually 3.3V" to more accurately reflect the situation. A decoupling cap would probably still help, but there were some mistakes made on the regulator circuit.

Yeah. Decoupling capacitors are for smaller ripples than that.

There might be a resonnance point on that regulator, or maybe a maximum capacitance that was violated on the feedback.

There are a TON of ways to screw up your PDN on a PCB. It's nominally a master's degree level subject.

Some small switching regulators go into a low power mode when the output current goes below a threshold. The frequency drops to some "hovering just above zero" level. I've had to artificially load a power supply, to get it to be stable, e.g., with a shunt resistor. Naturally, that's inefficient, so it goes onto the TODO list to improve the design.

1.5 Vpp ripple measured on a 40 MHz scope - when the waveform is 50 MHz according to him...

decoupling is a real issue, but I think you are right in this case.