Impact Lab


Subscribe Now to Our Free Email Newsletter
October 30th, 2020 at 10:09 am

The ‘failure’ of big data

big-data-k90h84d

In a May 2011 special research report, Big data: The next frontier for innovation, competition, and productivity, the management consulting firm McKinsey put forth the case that “Big data will become a key basis of competition, underpinning new waves of productivity growth, innovation, and consumer surplus.” The McKinsey report went on to note that, “The amount of data in our world has been exploding. Leaders in every sector will have to grapple with the implications of big data, not just a few data-oriented managers. The increasing volume and detail of information captured by enterprises, the rise of multimedia, social media, and the Internet of Things will fuel exponential growth in data for the foreseeable future.”

General usage of the term “Big Data” can be traced to the McKinsey report and similar reports from IBM that ensued around this time. The McKinsey report was prescient in its observations that “Leaders in every sector will have to grapple with the implications of big data, not just a few data-oriented managers.” In retrospect, this was the key insight. From this point forward, interest in data would no longer be limited to the purview of “a few data-oriented managers,” but rather would become the purview of “leaders in every sector.” The McKinsey report went on to describe the advent of the era of Big Data as heralding “new waves of productivity growth, innovation, and consumer surplus.” The report contained one important caveat however, noting that these advances were all predicated “as long as the right policies and enablers are in place.”

IL-Header-Communicating-with-the-Future

519EF7CC-FCB5-403D-8326-5D119F863A87

The advent of Big Data was met by an enthusiastic embrace by the venture capital and investment communities as well. In November 2011, Accel Partner announced the launch of a “$100M fund to invest in disruptive Big Data companies.” Accel’s Ping Li noted, “this Big Data wave will usher in a new era of multi-billion software companies that solve existing problems differently or new problems previously unthinkable.” Accel also announced the launch of a Big Data conference to “network and drive deeper discussion on technology trends in the space.” In June 2013, Accel launched a 2nd $100M fund, with Accel Partner Jake Flomenberg commenting, “Over the past few years, we’ve focused a tremendous amount of attention on what people like to call the ‘three Vs’ of big data: variety, volume, and velocity. We now believe there is a fourth V, which is end user value, and that hasn’t been addressed to the same extent.” Accel’s Ping Li added, “We are seeing an accelerated rate of innovation in big data, with the newest generation of entrepreneurs re-imagining ways to extract the most value out of big data and fundamentally change the way we work and process information.”

2021 will represent a decade since Big Data came into prominence, and based on industry findings, the promise remains largely unfulfilled. Consider the findings of NewVantage Partners 2020 Big Data and Executive Survey, the 8th annual survey of Fortune 1000 senior C-executive decision-makers with responsibility and oversight for data initiatives:

  • Only 26.8% of firms reported having forged a data-culture
  • Only 37.8% of firms reported that they were data-driven
  • Only 45.1 of firms reported that were now competing on data and analytics.

The on the ground reality for most organizations has been that the adoption of Big Data initiatives and the establishment of Big Data practices has not happened overnight. In practice, data-driven transformation has been shown to be a journey that unfolds over multiple years, not without missteps and failures over time, and is a process that is complex and requires an alignment of business, technology, and organizational perspectives and practices. Operating as a data-driven business requires an organizational change in mindset, thinking and approach.

Looking back over the course of the past decade, the term “Big Data” was initially embraced as heralding a revolution in how organizations would leverage vast amounts of data, extracted and successfully integrated from a proliferation of existing and emerging data sources (“the three V’s”), fueled by greater computing power than had ever been available, to enable a succession of newly data-driven firms to compete on data and analytics to achieve business dominance. Today, we have more data than ever, greater computing power than ever, and a next generation of data management, cataloging, extraction, analysis, and reporting tools and technology. Yet, the connection of data investments to business insights and successful business outcomes remains an elusive ambition for most.

In retrospect, and with the benefit of hindsight, the Big Data movement failed to affect the kind of revolutionary data-driven business and technology transformation as envisioned a decade ago by McKinsey, Accel Partners and others. Where the Big Data moment did succeed however was in raising consciousness of the vast transformational power of data—what McKinsey foresaw as the elevation of data awareness from “a few data-oriented managers” to “leaders in every sector.”

Looking forward from where we have come from to where we stand today, there are many reasons to celebrate the data achievements of the past decade. The glass is indeed half full. Data literacy has grown. If being data-driven remains an elusive aspiration for most organizations, firms are at a minimum, inching closer to that outcome. Public and business awareness of the power of data has increased. We have grown more data-conscious in large measure due to the aspirational promise of Big Data and the powerful mental image that this concept and term captured and conveyed. In this sense, the Big Data movement created the foundation upon which the evolutionary data-driven business and technology transformations that are occurring around us today exist.

At a consequential moment in our history, when the veracity and integrity of data, analytics, and science are being called into question by those who denigrate and attack expertise and knowledge, it is important to recognize the contributions of the Big Data movement, to acknowledge the value of data, and appreciate the pioneers, leaders, managers and analysts who persevere each day with the goal of making data accessible, usable and insightful.

Rome was not built in a day. The journey continues.

Via Forbes.com

 

IL-Header-Communicating-with-the-Future

Comments are closed.

Are certificates more valuable than college degrees?