Dark data is that which is acquired through various operational sources sometimes nefarious, but not used in any manner to derive insights or for decision making. In an industrial context, dark data can include information gathered by sensors and telematics. The ability of an organisation to collect data can exceed the rate at which it can analyse it, and in some cases the organisation may not even be aware that the data is being collected. According to IBM, "about 90 percent of data generated by most sensors and A-to-D conversions on the market never gets utilised".
Reasons for retaining
Organizations retain dark data for a multitude of reasons. Often it is stored for regulatory compliance and record keeping. Some organizations believe that dark data could be useful to them in the future, once they have acquired better analytic and BI technology to process the information. Because storage is inexpensive, storing data is easy. However, storing and securing the data usually entails greater expenses (or even risk) than the potential return profit. Data-informed attributes the retention of data to being "culturally accustomed to saving everything, in the belief that all data has some value".
A lot of dark data is unstructured, which means that the information is in formats that may be difficult to categorise, be read by the computer and thus analysed. Often the reason that business do not analyse their dark data is because of the amount of resources it would take and the difficulty of having that data analysed. According to ComputerWeekly.com, 60% of organisations believe that their "BI (business intelligence) reporting capability" is "inadequate" and 65% say that they have "somewhat disorganised content management approaches".
Many companies in the IT sector are looking at creating "cognitive computer systems" that are able to analyse unstructured dark data. The IBM Watson is considered to be a future system that would be able to analyse this unstructured data and be able to produce meaningful results that will utilise a lot of dark data that it is either practically impossible or very difficult to process at present. In terms of current systems, IBM have advertised the IBM Spark as a system that "can extract insight from that information almost immediately. This enables businesses to build data rich products and services that use that information to transform the customer experience." Furthermore, they also give an even broader definition of dark data, one that also includes data that is not currently processed by computing systems but could be, for example in law.
Useful data may become dark data after it becomes irrelevant, as it is not processed fast enough. This is called "perishable insights" in "live flowing data". For example, if the geolocation of a customer is known to a business, the business can make offer based on the location, however if this data is not processed immediately, it may be irrelevant in the future. According to IBM, about 60 percent of data loses its value immediately. Not analysing data immediately and letting it go 'dark' can lead to significant losses for an organisation in terms of not identifying fraud, for example, fast enough and then only addressing the issue when it is too late.
According to the New York Times, 90% of energy used by data centres is wasted. If data was not stored, energy costs could be saved. Furthermore, there are costs associated with the underutilisation of information and thus missed opportunities. Another cost is associated with keeping the data secure and other related IT costs. According to Datamation, "the storage environments of EMEA organizations consist of 54 percent dark data, 32 percent Redundant, Obsolete and Trivial data and 14 percent business-critical data. By 2020, this can add up to $891 billion in storage and management costs that can otherwise be avoided."
The continuous storage of dark data can put an organisation at risk, especially if this data is sensitive. In the case of a breach, this can result in serious repercussions. These can be financial, legal and can seriously hurt an organisation's reputation. For example, a breach of private records of customers could result in the stealing of sensitive information, which could result in identity theft. Another example could be the breach of the company's own sensitive information, for example relating to research and development. These risks can be mitigated by assessing and auditing whether this data is useful to the organisation, employing strong encryption and security and finally, if it is determined to be discarded, then it should be discarded in a way that it becomes unretrievable.
It is generally considered that as more advanced computing systems for analysis of data are built, the higher the value of dark data will be. It has been noted that "data and analytics will be the foundation of the modern industrial revolution". Of course, this includes data that is currently considered "dark data" since there are not enough resources to process it. All this data that is being collected can be used in the future to bring maximum productivity and an ability for organisations to meet consumers' demand. Furthermore, many organisations do not realise the value of dark data right now, for example in healthcare and education organisations deal with large amounts of data that could create a significant "potential to service students and patients in the manner in which the consumer and financial services pursue their target population".
- "Dark Data". Gartner.
- Ed Tittel (24 September 2014). "The Dangers of Dark Data and How to Minimize Your Exposure". CIO.
- Teradata. "TeradataVoice: Factories Of The Future: The Value Of Dark Data". Forbes.
- "The API Briefing: the Challenge of Government’s Dark Data". digitalgov.gov.
- "Digging up dark data: What puts IBM at the forefront of insight economy | #IBMinsight". SiliconANGLE. Retrieved 2015-11-03.
- "Dark Data: What is it and Why Should I Care?". www.r1soft.com. Retrieved 2015-11-02.
- "Don’t Be Spooked by Dark Data". Data Informed. Retrieved 2015-11-02.
- "Dark data could halt big data’s path to success". ComputerWeekly. Retrieved 2015-11-03.
- "IBM Cognitive Colloquium Spotlights Uncovering Dark Data - InformationWeek". InformationWeek. Retrieved 2015-11-03.
- "Digging up dark data: What puts IBM at the forefront of insight economy | #IBMinsight". SiliconANGLE. Retrieved 2015-11-04.
- Gutierrez, Daniel. "Deriving Value from Data Before It Goes Dark". insideBIGDATA. Retrieved 2015-11-04.
- Glanz, James (2012-09-22). "Data Centers Waste Vast Amounts of Energy, Belying Industry Image". The New York Times. ISSN 0362-4331. Retrieved 2015-11-02.
- "Enterprises are Hoarding 'Dark' Data: Veritas - Datamation". www.datamation.com. Retrieved 2015-11-04.
- "The Dangers of Dark Data and How to Minimize Your Exposure". CIO. Retrieved 2015-11-02.
- "Leveraging Dark Data: Q&A with Melissa McCormack - Predictive Analytics Times - predictive analytics & big data news". Predictive Analytics Times. Retrieved 2015-11-04.