Study guide: Big Data Characteristics and Sources

Course Outline

  1. Big Data Definition and Three V’s
  2. Big Data Volume
  3. Big Data Variety
  4. Big Data Velocity and Nowcasting
  5. Other Big Data Characteristics
  6. Technological Origins of Big Data
  7. New Data Versus Big Data
  8. Cross-Sector Uses of Big Data
  9. Three Big Data Configurations
  10. Tall Data and Statistical Methods
  11. Fat Data and Overfitting
  12. Large Data and IT Challenges

1. Big Data Definition and Three V’s

Key Concepts & Definitions

  • Big data: SAS - Big data describes large, hard-to-manage volumes of structured and unstructured data that grow at increasing rates and can be analyzed to improve decisions and strategic business moves.

β˜… Must-know

  • The three commonly cited characteristics of big data are volume, velocity, and variety. - Investopedia

Further detail

  • There is no clear, simple definition of big data, but there is consensus around its main characteristics, known as the three V’s.

2. Big Data Volume

Key Concepts & Definitions

  • Volume: The amount of data generated, which has grown exponentially to unprecedented levels.

Essential Points

  • Data were initially collected for specific purposes, such as national statistical agencies analyzing unemployment and GDP, while even small companies now collect individualized data for analysis.

  • Eric Schmidt stated in 2010 that as much information was being created every two days as had been created between the dawn of civilization and 2003, and that the pace was increasing.

  • Twitter and Facebook generate about 15 terabytes of data every day, equivalent to approximately 60 standard computer hard drives.

  • Google has seen 30 trillion URLs, examines 20 billion URLs a day, and answers 100 billion search queries each month. - Sullivan, Danny, Google : 100 Billion Searches per Month, Search to Integrate Gmail, Launching Enhanced Search App for iOS., 2012

  • Companies moved from terabytes of data to zettabytes of data. - Techtarget
  • The volume of data was projected to multiply by more than five between 2018 and 2025. - David Reinsel – John Gantz – John Rydning, The Digitization of the World From Edge to Core, November 2018

Memory Hook

Picture data accumulating from a small stream into an unprecedented mountain

3. Big Data Variety

Key Concepts & Definitions

  • Variety: The availability of multiple types of data, including text, photos, and videos.
  • Unstructured data: Not arranged in a formatted repository and include images, text, and videos.
  • Structured data: Organized into a formatted repository, such as a database or spreadsheet.

β˜… Must-know

  • Unstructured information that was previously unused can be used to create new datasets, including from central-bank speeches and satellite images.

πŸ“Œ Structured and unstructured data can be analysed separately, but unstructured data generally complement standard structured data when analysed jointly.

Further detail

  • Examples of traditional structured data include:

    • balance sheet data
    • prices of goods and assets
    • macroeconomic series
  • Tweets, Facebook posts, and music are examples of unstructured data.

  • Nyman et al. (2021), in News and narratives in financial systems: Exploiting big data for systemic risk assessment, use news and narratives in financial systems as a big-data application. - News and narratives in financial systems : Exploiting big data for systemic risk assessment

  • Picault and Renault (2017), in Words are not all created equal: A new measure of ECB communication, analyse the content of ECB press conferences. - Words are not all created equal : A new measure of ECB communication

Memory Hook

Structured tables fit formatted repositories; variety also includes multiple data types

4. Big Data Velocity and Nowcasting

Key Concepts & Definitions

  • Velocity: Measures how fast data are coming in.

β˜… Must-know

  • Big data are generally observed daily, and some are recorded intraday, such as minute-by-minute asset prices.

πŸ“Œ The faster arrival of data allows economists to move from forecasting to nowcasting, answering some questions in real time rather than months, quarters, or years later.

Further detail

  • During the 1980s and 1990s, economists mostly relied on yearly or quarterly data, such as quarterly GDP and unemployment rates.

  • Facebook uploads more than 900 million photos every day, and more than 50 million tweets are posted every day.

  • More than 3 million emails are sent each second, and more than 70 purchases are made each second on Amazon.

  • D’Amuri and Marcucci (2017) use Google searches to predict changes in US unemployment. - D’Amuri, F., and J. Marcucci., The predictive power of Google searches in forecasting US unemployment

  • Glaeser, Kim, and Luca (2017) use Yelp data to measure local economic activity and nowcast local business patterns. - Glaeser, E. L., H. Kim, and M. Luca, Nowcasting the Local Economy : Using Yelp Data to Measure Economic Activity

5. Other Big Data Characteristics

Key Concepts & Definitions

  • Veracity: Concerns data quality, including whether data are clean, accurate, and useful.
  • Value: Concerns whether data can be analysed and exploited quickly enough to generate business profits.

β˜… Must-know

  • The three core characteristics of Big Data are described as the 3 V’s, and Big Data is more than simply a large quantity of data.

Further detail

  • Veracity and value are sometimes included among Big Data characteristics, but they are not core characteristics.

6. Technological Origins of Big Data

β˜… Must-know

  • New technologies such as social media and connected watches have increased the volume of available data, especially unstructured data.

  • Big Data results from the combination of more available data, multiple data types, increased storage capacity, lower storage costs, and more efficient analysis.

Further detail

  • Data storage capacity has grown, allowing more data to be stored than before.

  • Data storage costs have decreased as storage capacity has grown.

  • Moore’s law is an empirical law stating that computers become smaller, faster, cheaper, and more efficient over time.

Memory Hook

More data and new data types, plus cheaper storage and faster analysis β†’ Big Data.

7. New Data Versus Big Data

β˜… Must-know

πŸ“Œ New data are not necessarily Big Data: a single central-bank speech or social-media content can be a new unstructured data source without having Big Data scale.

Further detail

  • Berg et al. (2019), in β€œOn the rise of FinTechs: Credit scoring using digital footprints,” analyse 10 digital-footprint variables to predict consumer loan default, making these new data rather than Big Data. - On the rise of FinTechs : Credit scoring using digital footprints

  • Social-media data analysis is almost always associated with Big Data because it involves large volumes of unstructured tweets or posts collected daily.

Memory Hook

New describes the data source or type; big describes scale and volume.

8. Cross-Sector Uses of Big Data

β˜… Must-know

  • Big Data is used in these sectors:
    • banking
    • finance
    • healthcare
    • manufacturing
    • media
    • sport

Further detail

  • Banking uses Big Data to predict default probability or fraudulent transactions.

  • Finance uses Big Data to predict risk premia.

  • Healthcare uses Big Data to identify the global spread of viruses.

  • Manufacturing uses Big Data to improve supply-chain management.

  • Media uses Big Data for real-time reporting.

  • Sport uses Big Data to predict winners and losers.

9. Three Big Data Configurations

β˜… Must-know

  • Varian (2014) distinguishes three configurations of Big Data: Tall Data, Fat Data, and Large Data.

πŸ“Œ Tall and Fat Data pose analysis challenges, whereas Large Data primarily poses storage and data-pooling challenges.

Further detail

  • Economists are particularly interested in using Big Data and related techniques to solve problems and improve analysis, while computer scientists are particularly interested in handling Big Data for business.

10. Tall Data and Statistical Methods

Key Concepts & Definitions

  • Tall Data: A configuration in which the number of observations, N, is very large relative to the number of variables, p.
  • Machine learning: A field of study that gives a computer the ability to learn without being explicitly programmed, as defined by Samuel (1959).
  • Econometrics: The application of statistical methods to economic data.

Essential Points

  • In Tall Data, the large number of observations can be analyzed with usual statistical tools, which present no particular statistical issue in this setting.

πŸ“Œ Machine-learning techniques were developed for datasets with few columns and a very large number of rows, and may perform better or worse than econometric techniques depending on the case.

  • Big Data has strengthened the bridge between economists and computer scientists, and economists need machine-learning skills to work with Big Data.

Memory Hook

Tall means many rows and few columns; Fat means many columns and few rows.

11. Fat Data and Overfitting

Key Concepts & Definitions

  • Overfitting: A situation in which a model's accuracy breaks down on a sample other than the one on which it was estimated, called the in-sample.

β˜… Must-know

πŸ“Œ Econometric models fail to handle Fat Data, whereas machine-learning techniques allow this type of data to be handled.

Further detail

  • Social-media data can be Fat Data because each individual is an observation and the many available connections can each correspond to a variable.

  • A dataset with many variables relative to observations is generally described as the curse of dimensionality.

Memory Hook

Many variables relative to observations β†’ overfitting.

12. Large Data and IT Challenges

Key Concepts & Definitions

  • Large Data: Data that requires more storage capacity than traditional applications and software can provide for loading and use.

β˜… Must-know

  • Large Data creates two main IT challenges: storing data because of its large volume, and pooling data because of its large variety.

Further detail

  • Analyzing billions of Google search queries is impossible with standard tools such as Excel spreadsheets.

  • Solutions identified for handling Large Data are:

    • Hadoop Distributed File System (HDFS)
    • MapReduce
    • NoSQL

Synthesis Tables

Big Data Configurations

ConfigurationData shapeMain challenge
Tall DataMany observations relative to variablesAnalysis
Fat DataMany variables relative to observationsAnalysis and overfitting
Large DataExceeds traditional storage and software capacityStorage and data pooling

Test yourself

Take a 11-question practice quiz on Big Data Characteristics and Sources, with every answer explained.

1. Which description best captures big data and its potential business value?

2. A team cannot find one short definition of big data, but experts broadly agree on its main characteristics. What conclusion fits this situation?

Take the quiz β†’

Study with flashcards

Memorize the key concepts of Big Data Characteristics and Sources with 10 flashcards.

According to SAS, what does big data describe?

Large, hard-to-manage volumes of structured and unstructured data that grow at increasing rates and can be analyzed to improve decisions and strategic business moves.

According to Investopedia, what are the three commonly cited characteristics of big data?

Volume, velocity, and variety.

What does Big Data Volume measure?

The amount of data generated, which has grown exponentially to unprecedented levels.

See the flashcards β†’

Related classes

Make your own study guides

Upload your class notes and Revizly writes a study guide, a quiz and flashcards in about a minute.

Study guide maker