How to Spot a Data Charlatan

I love this article. It gives a very straightforward view and critique of data management, data analysis, and data science. It should be required reading for anyone who is trying to gain a realistic understanding of the state-of-the-art vs the state-of-the-practices in this area. We have far too many data charlatans who are pulling the wool over the eyes of managers, policy makers, auditors, and the general public.

If you observe data charlatans in the wild, you’ll notice that they love to spin fancy stories to “explain” observed data. The more academic-sounding, the better. Nevermind that these stories only (over)fit the data in hindsight.

My concern is that I don’t believe that the answer is as easy as this article states:

To protect yourself against charlatans, all you have to do is make sure you keep some test data out of reach of their prying eyes, then treat everything else as analytics (don’t take it seriously). When you’re faced with a theory you’re in danger of buying into, use it to call the shot, and then open your secret test data to see if the theory is nonsense. It’s as easy as that!

What do you think?