Recently I read a book, Big Data: A Revolution That Will Transform How We Live, Work and Think and find it really informative. I would recommend it to any one curious about big data and its impact. The book does not assume your technology background.
Below are some of the insights that the book provides:
Big data is one of the consequences of a change that is taking place now; the authors describes it as datafication – a concept which refers to taking information about all things under the sun – including ones we never used to think of as information at all, such as a person’s location, the vibration of an engine, or the stress on a bridge – and transforming it into data format to make it quantified. This allows us to use information in new ways, such as in predictive analysis: detecting that an engine is prone to a break-down based on the heat or vibration that it produces. As a result, we can unlock the implicit, latent value of the information.
Big data ascendancy represents three shifts in the way we analyze information that transform how we understand and organize society.
- The ability to analyze vast amount of data (sometime all available data) about a topic rather than be forced to settle for smaller sets.
Since nineteenth century, society has dependency on using samples when faced with large numbers. Yet the need for sampling is an artifact of a period of information scarcity, a product of the natural constraints on interacting with information in an analog era. Before the prevalence of high-performance digital technologies, we didn’t recognize sampling as artificial fetters- we usually just took it for granted. Using all the data lets us see details we never could when we were limited to smaller quantities. Big data gives us an especially clear view of the granular: subcategories and submarkets that samples can’t access.
- Embrace data’s real-world messiness rather than privilege exactitude.
Often big data is messy, varies in quality, and is distributed among countless servers around the world. With big data, we’ll often be satisfied with a sense of general direction rather than knowing a phenomenon down to the inch, the penny, the atom. We don’t give up on exactitude entirely; we only give up devotion to it. What we lose in accuracy at the micro level we gain insight at the macro level.
- Respect for correlations rather than a continuing quest for elusive causality.
This is a move away from the age-old search for causality. As humans we have been conditioned for look for causes, even though searching for causality is often difficult and may lead down the wrong path. In a big data world, by contrast, we won’t have to be fixated on causality; instead we can discover patterns and correlations may not tell us precisely why something is happening but they alert us that is happening.
The effect on individuals may be the biggest shock of all. Specific area expertise matters less in a world where probability and correlation are paramount. In the movie Moneyball, baseball scouts were upstaged by statisticians when gut instinct gave way to sophisticated analytics. Similarly, subject-mater specialists will not go away, but they will have to contend with the big-data analysis says. This will force an adjustment to traditional ideas of management, decision-making, human resources, and education.