From The Atlantic, How American Politics Went Insane [1] by Jonathan Rauch
Moreover, recent research by the political scientists Jamie L. Carson and Jason M. Roberts finds that party leaders of yore did a better job of encouraging qualified mainstream candidates to challenge incumbents. “In congressional districts across the country, party leaders were able to carefully select candidates who would contribute to the collective good of the ticket,” Carson and Roberts write in their 2013 book, Ambition, Competition, and Electoral Reform: The Politics of Congressional Elections Across Time. “This led to a plentiful supply of quality candidates willing to enter races, since the potential costs of running and losing were largely underwritten by the party organization.” The switch to direct primaries, in which contenders generally self-recruit and succeed or fail on their own account, has produced more oddball and extreme challengers and thereby made general elections less competitive. “A series of reforms that were intended to create more open and less ‘insider’ dominated elections actually produced more entrenched politicians,” Carson and Roberts write. The paradoxical result is that members of Congress today are simultaneously less responsive to mainstream interests and harder to dislodge.
Was the switch to direct public nomination a net benefit or drawback? The answer to that question is subjective. But one effect is not in doubt: Institutionalists have less power than ever before to protect loyalists who play well with other politicians, or who take a tough congressional vote for the team, or who dare to cross single-issue voters and interests; and they have little capacity to fend off insurgents who owe nothing to anybody. Walled safely inside their gerrymandered districts, incumbents are insulated from general-election challenges that might pull them toward the political center, but they are perpetually vulnerable to primary challenges from extremists who pull them toward the fringes. Everyone worries about being the next Eric Cantor, the Republican House majority leader who, in a shocking upset, lost to an unknown Tea Partier in his 2014 primary. Legislators are scared of voting for anything that might increase the odds of a primary challenge, which is one reason it is so hard to raise the debt limit or pass a budget.