I hope Devin doesn’t mind if I quote a lot of his post, read it all:
Our first method is to determine whether an institution is a good filter of the truth is to look at the feedback loop. I trust that the linebacker coach for the New England Patriots knows a heck of a lot more about football than me, because the NFL has a feedback loop where coaches who lose games get fired. This feedback loop does not exist in academia.
Academics in the social science to do not get fired or demoted if they get things wrong. They do not get additional grad students if they are right. The grad school and peer review process reward one thing – conforming to the current intellectual fashions. For example read here, here, and here. This is the selection process for a priesthood, not for truth. As the history of organized religion shows, when the selection process of an institution is based on ideological self-selection, the views of that institution can diverge an arbitrary amount from reality.
The second method to judge an institution or person is to look at predictions. I fired the New York Times years ago because it kept making horrible predictions. Now my blog roll is filled with people considered "cranks" by the mainstream yet they consistently make better predictions than the NY Times.
For one example of many, compare these comments threads (here and here) written in late 2006 by one Mencius Moldbug to Ben Bernanke’s report to Congress made at a similar time. Ben Bernanke is as filtered and respectable as it gets – he was a tenured Princeton professor, appointed by a president from the right, approved by Paul Krugman, and reappointed by a president from the left. And yet he (and the minions who actually wrote the report) got it totally wrong. Meanwhile Moldbug, a pseudonymous software engineer with a large collection of old John Birch Society literature, got the analysis dead on.
And it’s not just that Moldbug made a good predictions and the Fed made a bad prediction. The economy is like the weather – it’s a chaotic system in which it’s possible to make a perfect analysis but still get the prediction wrong. The real problem is that the Fed misses the storm clouds altogether, and in fact, misidentifies storm clouds as balmy weather (hint to the Fed: if credit is expanding at 9.5%, ie much faster than income is rising, that’s a sign of alarm, not health).
And for all the bad analysis and bad predictions, has Bernanke been sacked? No. Has anyone at the Fed responsible for writing that report been sacked? No. We see bad predictions and no feedback loop.
The final way to judge academia is to pick an issue and check the source material yourself. Read everything you can find – primary sources, secondary sources, victorian era sources, new sources. Look for recommendations from the far left, center, universities, far-right, and the Sith. Actually read the studies that people cite and examine the assumptions and the math. Compile your own stats, do your own calculations, and build your own toy models.
This process is extremely laborious. It’s not for the policy dilettante. But its one of the only true ways to find out if academia is properly filtering information and finding truth. You may find out that it is – you find the far out sources are indeed cranks, they are making up facts, using poor logic, making bad predictions etc. But you may find out the opposite. You find a an entire world of brilliant, perceptive sources that academia has excluded for reasons of fashion and politics. You may find that academia is not a filter for quickly finding the truth, but a priesthood that produces a black and white, cartoon version of events.
Over the past years I have subjected myself to the intellectual detoxification treatment. The results have been quite disturbing. A few years ago I was an optimistic, generally progressive, academia believing Obama voter. But as I began examining issues more closely I found the mainstream academic view to be far adrift from reality. Part of the reason I issue this challenge is so that I can get a second opinion on whether I’m crazy or everyone else is.
Go read his list of topics. They are perhaps the most interesting part.
If one looks carefully enough, one sees cracks in the facade of accepted wisdom. As you see more, the cracks turn into canyons.
I’d suggesting adding a question on the causes and consequences of the Civil War and perhaps one on the "abolition" of slavery.