Even to just get into "very large nuke" territory, you'd require impactors in the million ton range (200m diameter stony asteroid). LEO payload for Saturn-V/Starship is in the 100 ton range, for comparison, so you'd need about 10000 of those (~10km/s of delta-v is barely enough to fish anything out of the asteroid belt, see [1]).
If your hypothetical rogue government/terrorist org adversary can get control of the equivalent of 10000 Saturn-V, then them pursuing some harebrained asteroid redirection plot afterwards is the least of your problems.
I'm not bad-faithing or strawmaning anything here-- my point is simply that any space mining program that is plausibly imaginable within the next centuries (which is gonna limit things to <10000 tons of payload in my view) is a non-concern from a military utility perspective (that would be roughly a chelyabinsk meteor at the very most, for reference, and depending on trajectory and structure might not even hit the ground).
[1]: https://planet4589.org/jcm/pubs/sci/papers/2018/Taylor18.pdf
Clearly not, otherwise there would have been many more mass extinctions.
In any case, shifting that by a couple of orders of magnitude doesn't save the argument.
Meanwhile, consider what a relatively minor event caused https://en.wikipedia.org/wiki/Year_Without_a_Summer . I don't know the exact size needed for an equivalent meteor impact, but it's a lot smaller than the Chicxulub impactor and we still don't want to get anywhere close to it. Even something like the Chelyabinsk meteor, which injured over a thousand people, would be a half-decent terror weapon.