Many more mass extinctions, you say? https://en.wikipedia.org/wiki/List_of_extinction_events These are still just the highlights.

Meanwhile, consider what a relatively minor event caused https://en.wikipedia.org/wiki/Year_Without_a_Summer . I don't know the exact size needed for an equivalent meteor impact, but it's a lot smaller than the Chicxulub impactor and we still don't want to get anywhere close to it. Even something like the Chelyabinsk meteor, which injured over a thousand people, would be a half-decent terror weapon.