In the era of big data and super-fast information capturing and processing systems, it is easy to imagine that we have all the information that lead to actionable insights, that we need to make good decisions. However, according to David Hand, Emeritus Professor of Mathematics and Senior Research Investigator at Imperial College London, the data we have are never complete. Just as much of the universe is composed of dark matter, invisible to us but nonetheless present, the universe of information is full of dark data that we overlook at our peril. In his new book “Dark Data: Why What You Don’t Know Matters” Professor David Hand takes us on a fascinating and enlightening journey into the world of the data we don’t see.

As in his book “Brief Answers to the Big Questions” Stephen Hawking notes “No matter how powerful a computer you have, if you put lousy data in you will get lousy predictions out”, it is essential to understand anomalies and imperfections that a dataset may have. These imperfections may lead to incorrect and misleading insights.

The book “Dark Data: Why What You Don’t Know Matters” explores the many ways in which we can be blind to missing data and how that can lead us to conclusions and actions that are mistaken, dangerous, or even disastrous. Full of real-life examples, from the Challenger shuttle explosion to complex financial frauds, the book outlines a practical taxonomy of the types of dark data that exist and the situations in which they can arise, and informs the readers how to recognize and control dark data. Professor David Hand guides us not only to be alert to the problems presented by the things we don’t know, but also shows how dark data can be used to our advantage, leading to greater understanding and better decisions. Data is essential for decision making; the books shows us all how to reduce the risk of making bad decisions.

Complement this conversation with equally fascinating discussion on Artificial Intelligence with professor Bart Selman “Artificial Intelligence: Fascinating Opportunities and Emerging Challenges” and professor Toby Walsh “2062: The World That AI Made”.