Cyber Defense Review

Big Data is Dead, Long Live Big Data

By Dr. David Thomson | March 10, 2016

The Gartner Hype Cycle, which assigns emerging technologies into 5 regions: Innovation Trigger, Peak of Inflated Expectations, Trough of Disillusionment, Slope of Enlightenment and Plateau of Productivity. In 2014, Big Data was at the edge of the Peak of Inflated Expectations, where the hype has already generated an enormous amount of goodwill through amazing success stories, and on a descent towards the Trough of Disillusionment, where the rate of new successes relative to the Peak creates a depressed sense of its novelty.

Big Data fell off the chart in 2015.

So is Big Data dead? Not so says Gartner analyst Nick Heudecker, whose blog post is entitled “Big Data Isn’t Obsolete. It’s Normal’’ [0b]. An in-depth look of how Big Data fell off Gartner’s list is available for purchase (or via a Gartner account) [0c], but for those whose pocketbooks or (organizations’ pocketbooks) don’t allow, multiple avenues have indicated that the Internet of Things have emerged to take over Big Data’s hype [1-3].

I would posit a slightly different interpretation: as Big Data has normalized, considering Big Data as its own topic has been eschewed in exchange for considering its various theories (data mining, machine learning, natural language processing, etc.) and resulting technologies (Internet of Things, autonomous vehicles, etc.) as their own entities. And, in my opinion, rightly so.

Big Data may be off the hype cycle, but it maintains healthy funding:

The Defense Advanced Research Projects Agency (DARPA) has increased their spending on programs with a Big Data component from $200M in FY14 to $242M in FY15 to $243M in FY16, with 4 programs receiving more than $20M each:

  1. XData ($39M): Develop tools to analyze large data sets with noisy or incomplete data.
  2. Wireless Network Defense ($35M): Develop new technologies and protocols for robust control of wireless networks, focusing particularly on securing current and next-generation products.
  3. PlanX ($25M): A program to develop technologies to operate on the cyber battlefront in a similar way that the military has developed tools and technologies to support kinetic warfare.
  4. Multifunctional Materials and Structures ($23M): Create design methodology for synthetic materials, and build materials/systems that demonstrate multi-functionality value to DoD.

Focusing only on projects which are specified as a core Big Data Program shows DARPA budgeting numbers of $97M/$118M/$165M over FY14/FY15/FY16, respectively, despite removing PlanX from the budgeting; see [4].

Moreover, the National Science Foundation [5] has solicitations for full proposals in the related areas of:

  1. Critical Techniques, Technologies and Methodologies for Advancing Foundations and Applications of Big Data Sciences and Engineering (BIGDATA).
  2. Campus Cyberinfrastructure – Data, Networking, and Innovation Program (CC*DNI).
  3. Computational and Data Enabled Science and Engineering (CDS&E).
  4. Exploiting Parallelism and Scalability (XPS).
  5. Secure and Trusted Cyberspace (SaTC).

These sources of funding are being leveraged across the academic, government and private sectors.

The prevalence of so-called Open Data (open-source data often resulting from government studies) is allowing researchers across multiple disciplines to test hypotheses on relevant data sets and put new theory, tools and techniques or algorithms to the test. The website is the US government’s own open data repository, providing open access to nearly 200,000 datasets from across the various government departments.

All of which indicates that data is not dead, but due to ubiquity and maturity, it is becoming normal. Long Live Big Data.




[0]: A. Brantly and J. Frigo, Big Data and Cybersecurity, Available (Online), accessed 03/2016, (2015).

[0b]: N. Heudecker, Big data isn’t obsolete. It’s normal, Available (Online), accessed 03/2016, (2015).

[0c]: N. Heudecker, et al, The demise of Big Data, its lessons and the state of things to come, Gartner Technical Report, Available (Online), (2015) 6 pages.

[1]: A. White, Internet of Things will dwarf Big Data, Available (Online), accessed 03/2016, (2015).

[2]: G. Press, It’s official: the Internet of Things takes over Big Data as the most hyped technology, Forbes, Available (Online), accessed 03/2016, (2014).

[3]: G. B. Peddibhotla, Gartner 2015 Hype Cycle: Big Data is out, Machine Learning is in, KDNuggest, Available (Online), accessed 03/2016, (2015).

[4]: J. Lutton, DARPA is spending big on Big Data, FCW, Available (Online), accessed 03/2016, (2015).

[5]: Critical Techniques, Technologies and Methodologies for Advancing Foundations and Applications of Big Data Sciences and Engineering (BIGDATA), National Science Foundation proposal solicitation, Available (Online), accessed 03/2016, (2015).