Course outline
Module 1
Module 1: Introduction to Big Data
Introduction to Big Data
Knowledge check
Q1. Which of the following best describes 'Big Data'?
- A. Data that is stored only in relational databases
- B. Datasets so large or complex that traditional data processing tools cannot handle them efficiently ✓
- C. Any spreadsheet containing more than 1,000 rows
- D. Data that has been encrypted for security purposes
Why: Big Data refers to datasets whose volume, velocity, or variety exceeds the capacity of traditional data processing and management tools.
Q2. Which of the following is NOT one of the commonly cited 'V's of Big Data?
- A. Volume
- B. Velocity
- C. Veracity
- D. Visualization ✓
Why: The core V's are Volume, Velocity, Variety, Veracity, and sometimes Value. Visualization is an analysis technique, not a defining characteristic of Big Data.
Q3. Which term refers to the speed at which data is generated and processed?
- A. Volume
- B. Variety
- C. Velocity ✓
- D. Veracity
Why: Velocity describes how fast data is produced, collected, and must be processed, such as streaming sensor or social media data.
Q4. Which of the following is an example of unstructured data?
- A. A table of customer orders in a SQL database
- B. A CSV file of monthly sales figures
- C. A collection of customer emails and social media posts ✓
- D. A spreadsheet with fixed columns for name and age
Why: Unstructured data has no predefined data model or schema. Free-form text such as emails and social media posts is a classic example.
Q5. Which open-source framework is widely used for distributed storage and processing of Big Data across clusters of computers?
- A. Microsoft Excel
- B. Apache Hadoop ✓
- C. Adobe Photoshop
- D. Google Chrome
Why: Apache Hadoop is an open-source framework that enables distributed storage (HDFS) and parallel processing (MapReduce) of large datasets across commodity hardware clusters.
Module 2
Module 2: Data Storage and Distributed Systems
Data Storage and Distributed Systems
Knowledge check
Q1. What is the primary purpose of the Hadoop Distributed File System (HDFS)?
- A. To process transactions in real time with low latency
- B. To store very large files reliably across many commodity machines
- C. To provide a graphical interface for database administration
- D. To encrypt data stored on a single server
Q2. In HDFS, what is the default replication factor for data blocks?
Q3. Which HDFS component is responsible for storing the actual data blocks?
- A. NameNode
- B. DataNode
- C. JobTracker
- D. ResourceManager
Q4. Why are large block sizes (such as 128 MB) used in HDFS instead of small blocks typical of traditional file systems?
- A. To make files easier to read manually
- B. To reduce metadata overhead and minimize seek time relative to data transfer
- C. To allow each block to be stored in a different language format
- D. To ensure each block fits exactly in RAM
Q5. What does it mean for a distributed storage system to be 'fault tolerant'?
- A. It never experiences hardware failures
- B. It continues to operate correctly even when some nodes fail
- C. It automatically corrects errors in user-written code
- D. It prevents users from making mistakes when entering data
Module 3
Module 3: Data Ingestion and Processing
Data Ingestion and Processing
Knowledge check
Q1. What is the primary purpose of data ingestion in a big data pipeline?
- A. To permanently delete outdated records from storage
- B. To collect and move data from various sources into a storage or processing system
- C. To create visual dashboards for business users
- D. To train machine learning models on historical data
Why: Data ingestion is the process of collecting data from diverse sources (databases, logs, sensors, APIs) and moving it into a system where it can be stored and processed.
Q2. Which term describes data that arrives continuously and is processed as individual records or small windows as it comes in?
- A. Batch processing
- B. Archival storage
- C. Stream processing
- D. Data warehousing
Why: Stream processing handles data in near real time as it is generated, unlike batch processing, which handles accumulated data at scheduled intervals.
Q3. In big data systems, what does the term 'schema-on-read' mean?
- A. The schema must be fully defined before any data is written
- B. Data is stored in raw form and structure is applied only when the data is queried
- C. Schemas are read from a separate configuration file at startup
- D. Only relational databases can enforce a schema
Why: Schema-on-read lets raw data be stored without a predefined structure, with the schema interpreted at query time. This is common in data lakes.
Q4. Which tool is widely used for distributed, fault-tolerant ingestion of high-volume streaming data?
- A. Microsoft Excel
- B. Apache Kafka
- C. Adobe Photoshop
- D. Git
Why: Apache Kafka is a distributed event streaming platform designed to ingest and transport large volumes of data reliably and with fault tolerance.
Q5. What is the main difference between ETL and ELT?
- A. ETL uses cloud storage while ELT uses on-premises storage
- B. ETL transforms data before loading it into the target, while ELT loads raw data first and transforms it inside the target system
- C. ETL is only for streaming data while ELT is only for batch data
- D. ETL deletes source data after extraction while ELT keeps it
Why: In ETL, transformation happens before loading. In ELT, raw data is loaded first and transformed later using the target system's compute power, which suits modern data warehouses and lakes.
Module 4
Module 4: Batch vs. Stream Processing
Batch vs. Stream Processing
Knowledge check
Q1. Which processing approach is best described as collecting data over a period of time and processing it all at once?
- A. Stream processing
- B. Batch processing ✓
- C. Real-time processing
- D. Interactive processing
Why: Batch processing collects a bounded set of data over time and processes it together as a single job, typically on a schedule.
Q2. Which of the following is a typical use case for stream processing?
- A. Monthly payroll calculation
- B. Detecting fraudulent credit card transactions as they occur ✓
- C. Generating an annual financial report
- D. Archiving historical log files
Why: Stream processing handles continuous data and reacts to events with low latency, making it ideal for real-time fraud detection.
Q3. What is a key characteristic of batch processing compared to stream processing?
- A. It provides results with millisecond latency
- B. It processes unbounded, continuous data
- C. It typically has higher latency but can handle very large volumes efficiently ✓
- D. It cannot process data stored in distributed file systems
Why: Batch processing trades latency for throughput: results come later, but large volumes of stored data can be processed efficiently.
Q4. In stream processing, what does the term 'event time' refer to?
- A. The time at which the processing system receives the event
- B. The time at which the event actually occurred at its source ✓
- C. The time when the batch job was scheduled
- D. The total time taken to process a dataset
Why: Event time is the timestamp when an event originally happened, which is distinct from processing time when the system handles it; this distinction matters for out-of-order data.
Q5. Which framework is commonly associated with both batch and stream processing through a unified engine?
- A. Apache Spark ✓
- B. Microsoft Excel
- C. MySQL
- D. Apache Tomcat
Why: Apache Spark provides batch processing through its core engine and stream processing through Structured Streaming, allowing both models to share APIs.
Module 5
Module 5: Big Data Analytics Tools
Big Data Analytics Tools
Knowledge check
Q1. Which tool is primarily used for distributed storage of large datasets across clusters of commodity hardware?
- A. Apache Kafka
- B. Hadoop Distributed File System (HDFS) ✓
- C. Apache Tableau
- D. MySQL
Why: HDFS is designed to store very large files across many machines in a cluster, providing fault tolerance through data replication. Kafka is a messaging/streaming platform, Tableau is a visualization tool, and MySQL is a traditional single-node relational database.
Q2. What is the main programming model used by Hadoop MapReduce for processing data?
- A. Sequential processing in a single thread
- B. Splitting work into map and reduce phases that run in parallel ✓
- C. Real-time event streaming only
- D. Graphical drag-and-drop pipelines
Why: MapReduce divides a job into a map phase, which processes input data in parallel as key-value pairs, and a reduce phase, which aggregates the intermediate results. This parallelism is what allows it to scale across large clusters.
Q3. Which Apache tool is best known for fast, in-memory data processing and supports batch, streaming, SQL, and machine learning workloads?
- A. Apache Spark ✓
- B. Apache Pig
- C. Apache Flume
- D. Apache ZooKeeper
Why: Apache Spark performs computations in memory, which makes it much faster than disk-based MapReduce for many workloads. It includes libraries for SQL (Spark SQL), streaming, machine learning (MLlib), and graph processing. Pig is a scripting layer over Hadoop, Flume collects log data, and ZooKeeper coordinates distributed services.
Q4. What is the role of Apache Hive in the Big Data ecosystem?
- A. It provides a SQL-like query language (HiveQL) for querying data stored in Hadoop ✓
- B. It collects streaming log data from web servers
- C. It manages cluster resource allocation
- D. It serves as a NoSQL key-value database
Why: Hive lets users write queries in HiveQL, a SQL-like language, which are translated into jobs that run on Hadoop. This makes large datasets accessible to analysts familiar with SQL, without writing low-level MapReduce code.
Q5. Which tool is commonly used to ingest and move large volumes of streaming data into systems like HDFS?
- A. Apache Flume ✓
- B. Apache Mahout
- C. Apache Ambari
- D. Apache Solr
Why: Apache Flume is designed to collect, aggregate, and move large amounts of streaming data, such as log files, into Hadoop storage. Mahout is a machine learning library, Ambari is for provisioning and monitoring clusters, and Solr is a search platform.
Module 6
Module 6: Data Quality and Governance
Data Quality and Governance
Knowledge check
Q1. Which data quality dimension refers to whether data values are accurate and correctly represent the real-world entity they describe?
- A. Completeness
- B. Accuracy ✓
- C. Timeliness
- D. Uniqueness
Why: Accuracy measures how closely data values match the real-world objects or events they represent. Completeness concerns missing values, timeliness concerns how up to date data is, and uniqueness concerns duplicate records.
Q2. What is the primary purpose of data governance in a big data environment?
- A. To maximize the speed of data ingestion
- B. To compress data and reduce storage costs
- C. To define policies, roles, and standards that ensure data is managed consistently, securely, and responsibly ✓
- D. To replace all manual data entry with automated sensors
Why: Data governance establishes the framework of policies, ownership, standards, and accountability that keeps data trustworthy, secure, and compliant throughout its lifecycle. Speed, storage, and automation are separate concerns.
Q3. Which of the following is an example of a data completeness problem?
- A. A customer's phone number is recorded in the wrong format
- B. The same customer appears twice in the database
- C. A customer's email address field is empty for thousands of records ✓
- D. A customer's record was last updated five years ago
Why: Completeness problems occur when required values are missing. An empty email field across many records is a missing-value issue. Wrong format relates to validity, duplicates relate to uniqueness, and outdated records relate to timeliness.
Q4. What does the term 'data lineage' describe?
- A. The physical location of servers storing the data
- B. The number of users who have accessed a dataset
- C. The record of where data originated and how it has been transformed as it moves through systems ✓
- D. The encryption algorithm used to protect the data
Why: Data lineage traces the origin, movement, and transformations of data across systems. It helps teams understand trust, debug errors, and meet audit requirements.
Q5. Why is data validation important before data is loaded into an analytics system?
- A. It increases the file size so storage is used more efficiently
- B. It ensures data meets defined rules and formats, preventing errors from propagating into analyses and decisions ✓
- C. It removes the need for any future data governance policies
- D. It guarantees that the data will never need to be updated
Why: Validation checks that incoming data conforms to expected rules, types, and ranges. Catching problems early prevents bad data from corrupting reports, models, and business decisions downstream.
Module 7
Module 7: Big Data Applications and Case Studies
Big Data Applications and Case Studies
Knowledge check
Q1. Which of the following is the most common real-world application of big data in retail?
- A. Predicting weather patterns for agricultural planning
- B. Analyzing customer purchase history to generate personalized product recommendations ✓
- C. Tracking satellite images of ocean currents
- D. Monitoring seismic activity in earthquake zones
Why: Retailers use large volumes of transaction and browsing data to recommend products, which is a classic big data application. The other options describe domains that may use big data but are not specific to retail customer analytics.
Q2. What does the term 'velocity' refer to in the context of big data?
- A. The physical size of data storage devices
- B. The speed at which data is generated, collected, and processed ✓
- C. The variety of file formats in a dataset
- D. The accuracy of data values
Why: Velocity is one of the core V's of big data and describes how quickly data arrives and must be processed, such as streaming sensor readings or social media posts.
Q3. A hospital uses patient records, lab results, and wearable device data to predict which patients are at risk of readmission. This is an example of which big data benefit?
- A. Predictive analytics in healthcare ✓
- B. Data encryption for network security
- C. Manual record keeping
- D. Hardware cost reduction
Why: Combining diverse data sources to forecast future outcomes, such as patient readmission risk, is predictive analytics. It helps clinicians intervene earlier and improve patient care.
Q4. Which big data technology is designed to process large datasets across clusters of computers using a map and reduce approach?
- A. Microsoft Excel
- B. Apache Hadoop ✓
- C. Adobe Photoshop
- D. Google Chrome
Why: Apache Hadoop uses the MapReduce programming model to distribute storage and computation across many machines, making it suitable for very large datasets. The other options are general-purpose tools not built for distributed big data processing.
Q5. Ride-sharing companies analyze real-time GPS and demand data to adjust prices during peak hours. What is this practice commonly called?
- A. Dynamic pricing ✓
- B. Static budgeting
- C. Data archiving
- D. Manual inventory control
Why: Dynamic pricing adjusts prices in response to changing supply and demand signals, which big data analysis of location and demand makes possible in real time.