In auto, none of the buttons is pressed (and preferable a small label that says auto is shown). Pressing dark/light will force that mode, enable that button and remove the auto label.
In dark/light mode, you can press the other button to enable the other state. Or you can press the current button to disable it, and switch to auto. This last one is the confusing-at-first interaction, but makes sense if you think about it (it's like saying: I want dark / I want light / I don't care)
In my opinion, the existing UI widget that solves this problem best is a plain old group of radio buttons (auto, light, dark). Two checkboxes that interact with each other and also a sometimes-hidden label is something new for users to learn and also more complex to implement and maintain.