β Must-know
Further detail
Data were initially collected for specific purposes, such as national statistical agencies analyzing unemployment and GDP, while even small companies now collect individualized data for analysis.
Eric Schmidt stated in 2010 that as much information was being created every two days as had been created between the dawn of civilization and 2003, and that the pace was increasing.
Twitter and Facebook generate about 15 terabytes of data every day, equivalent to approximately 60 standard computer hard drives.
Google has seen 30 trillion URLs, examines 20 billion URLs a day, and answers 100 billion search queries each month. - Sullivan, Danny, Google : 100 Billion Searches per Month, Search to Integrate Gmail, Launching Enhanced Search App for iOS., 2012
Picture data accumulating from a small stream into an unprecedented mountain
β Must-know
π Structured and unstructured data can be analysed separately, but unstructured data generally complement standard structured data when analysed jointly.
Further detail
Examples of traditional structured data include:
Tweets, Facebook posts, and music are examples of unstructured data.
Nyman et al. (2021), in News and narratives in financial systems: Exploiting big data for systemic risk assessment, use news and narratives in financial systems as a big-data application. - News and narratives in financial systems : Exploiting big data for systemic risk assessment
Picault and Renault (2017), in Words are not all created equal: A new measure of ECB communication, analyse the content of ECB press conferences. - Words are not all created equal : A new measure of ECB communication
Structured tables fit formatted repositories; variety also includes multiple data types
β Must-know
π The faster arrival of data allows economists to move from forecasting to nowcasting, answering some questions in real time rather than months, quarters, or years later.
Further detail
During the 1980s and 1990s, economists mostly relied on yearly or quarterly data, such as quarterly GDP and unemployment rates.
Facebook uploads more than 900 million photos every day, and more than 50 million tweets are posted every day.
More than 3 million emails are sent each second, and more than 70 purchases are made each second on Amazon.
DβAmuri and Marcucci (2017) use Google searches to predict changes in US unemployment. - DβAmuri, F., and J. Marcucci., The predictive power of Google searches in forecasting US unemployment
Glaeser, Kim, and Luca (2017) use Yelp data to measure local economic activity and nowcast local business patterns. - Glaeser, E. L., H. Kim, and M. Luca, Nowcasting the Local Economy : Using Yelp Data to Measure Economic Activity
β Must-know
Further detail
β Must-know
New technologies such as social media and connected watches have increased the volume of available data, especially unstructured data.
Big Data results from the combination of more available data, multiple data types, increased storage capacity, lower storage costs, and more efficient analysis.
Further detail
Data storage capacity has grown, allowing more data to be stored than before.
Data storage costs have decreased as storage capacity has grown.
Mooreβs law is an empirical law stating that computers become smaller, faster, cheaper, and more efficient over time.
More data and new data types, plus cheaper storage and faster analysis β Big Data.
β Must-know
π New data are not necessarily Big Data: a single central-bank speech or social-media content can be a new unstructured data source without having Big Data scale.
Further detail
Berg et al. (2019), in βOn the rise of FinTechs: Credit scoring using digital footprints,β analyse 10 digital-footprint variables to predict consumer loan default, making these new data rather than Big Data. - On the rise of FinTechs : Credit scoring using digital footprints
Social-media data analysis is almost always associated with Big Data because it involves large volumes of unstructured tweets or posts collected daily.
New describes the data source or type; big describes scale and volume.
β Must-know
Further detail
Banking uses Big Data to predict default probability or fraudulent transactions.
Finance uses Big Data to predict risk premia.
Healthcare uses Big Data to identify the global spread of viruses.
Manufacturing uses Big Data to improve supply-chain management.
Media uses Big Data for real-time reporting.
Sport uses Big Data to predict winners and losers.
β Must-know
π Tall and Fat Data pose analysis challenges, whereas Large Data primarily poses storage and data-pooling challenges.
Further detail
π Machine-learning techniques were developed for datasets with few columns and a very large number of rows, and may perform better or worse than econometric techniques depending on the case.
Tall means many rows and few columns; Fat means many columns and few rows.
β Must-know
π Econometric models fail to handle Fat Data, whereas machine-learning techniques allow this type of data to be handled.
Further detail
Social-media data can be Fat Data because each individual is an observation and the many available connections can each correspond to a variable.
A dataset with many variables relative to observations is generally described as the curse of dimensionality.
Many variables relative to observations β overfitting.
β Must-know
Further detail
Analyzing billions of Google search queries is impossible with standard tools such as Excel spreadsheets.
Solutions identified for handling Large Data are:
Big Data Configurations
| Configuration | Data shape | Main challenge |
|---|---|---|
| Tall Data | Many observations relative to variables | Analysis |
| Fat Data | Many variables relative to observations | Analysis and overfitting |
| Large Data | Exceeds traditional storage and software capacity | Storage and data pooling |
Take a 11-question practice quiz on Big Data Characteristics and Sources, with every answer explained.
1. Which description best captures big data and its potential business value?
2. A team cannot find one short definition of big data, but experts broadly agree on its main characteristics. What conclusion fits this situation?
Memorize the key concepts of Big Data Characteristics and Sources with 10 flashcards.
According to SAS, what does big data describe?
Large, hard-to-manage volumes of structured and unstructured data that grow at increasing rates and can be analyzed to improve decisions and strategic business moves.
According to Investopedia, what are the three commonly cited characteristics of big data?
Volume, velocity, and variety.
What does Big Data Volume measure?
The amount of data generated, which has grown exponentially to unprecedented levels.
Upload your class notes and Revizly writes a study guide, a quiz and flashcards in about a minute.
Study guide maker