Even to just get into "very large nuke" territory, you'd require impactors in the million ton range (200m diameter stony asteroid). LEO payload for Saturn-V/Starship is in the 100 ton range, for comparison, so you'd need about 10000 of those (~10km/s of delta-v is barely enough to fish anything out of the asteroid belt, see [1]).
If your hypothetical rogue government/terrorist org adversary can get control of the equivalent of 10000 Saturn-V, then them pursuing some harebrained asteroid redirection plot afterwards is the least of your problems.
I'm not bad-faithing or strawmaning anything here-- my point is simply that any space mining program that is plausibly imaginable within the next centuries (which is gonna limit things to <10000 tons of payload in my view) is a non-concern from a military utility perspective (that would be roughly a chelyabinsk meteor at the very most, for reference, and depending on trajectory and structure might not even hit the ground).
[1]: https://planet4589.org/jcm/pubs/sci/papers/2018/Taylor18.pdf