Here's the study on turnout. And basically comes to the conclusion extremists motivate the opposing party base more than their own.

Here are a couple of journal articles.

https://www.cambridge.org/core/journals/american-political-s...

https://academic.oup.com/poq/advance-article-abstract/doi/10...

Note: There is small minority that show that this is effect shrinking with time. My personal belief for why this is happening is basically voters are judging individual politicians more by the moderation/extremeness of the party's positions and less by the politicians personal beliefs.