Interesting. Far from a domain expert in materials science, but I've worked professionally in this area. What's your method for identifying valid "novel" compounds? Certainly, anything actually novel has been included in the models' training set already, unless you're doing a CASP-like coordinated blind test...right?
As an aside, the "Fable lies and cheats" section made me laugh -- have encountered this same failure mode, albeit for much simpler models. Polymerization (I realize this is not exactly the same thing) is an unbounded 3D playing field for constraint escape. You can try to put an additional constraint on polymerization, but then it will just make minor variations on the monomers...
So novel is defined as just anything that has not been synthesized as a thin film before. These are usually structures (phases or polymorphs) of materials that have not been reported in the literature.
We did find there are ways we can prevent this particular reward hacking, just consistently surprised by how these models can find tricks to technically achieve the set goal