Has some organizational properties (tags/markers) but no strict schema.
Examples: JSON files, XML documents, email (headers + body), CSV files, HTML pages.
🎵 Unstructured Data
No predefined model or schema. Cannot be stored in rows/columns directly.
Examples: Images, videos, audio files, social media posts, PDFs, sensor logs.
📈 Data Volume Distribution (Approximate Global Data)
Structured 20%
Semi 10%
Unstructured 70% of all enterprise data
Key Insight: Over 80% of the world's data is unstructured, which is why Big Data tools like Hadoop were designed — traditional RDBMS cannot handle it.
Topic 02
Structured vs Unstructured Data
A detailed comparison of the two dominant data types.
Parameter
Structured Data
Unstructured Data
Format
Fixed schema, rows & columns
No fixed format
Storage
RDBMS (MySQL, Oracle)
Data Lakes, NoSQL, Hadoop HDFS
Query Language
SQL
MapReduce, Hive, Pig, Spark
Scalability
Limited (vertical scaling)
Highly scalable (horizontal)
Analysis
Easy — reports, dashboards
Complex — NLP, CV, ML needed
Examples
Bank DB, HR System, ERP
Tweets, CCTV footage, emails
Volume
~20% of enterprise data
~80% of enterprise data
Semi-Structured Deep Dive
Semi-structured data acts as a bridge. For example, a JSON document from Twitter's API has fields (structured aspect) but the tweet text itself is unstructured text. NoSQL databases like MongoDB handle such data natively.
Real-World Use Case
E-commerce: Order table = Structured; product reviews = Unstructured
Healthcare: Patient ID/DOB = Structured; doctor notes = Unstructured
Big Data refers to datasets that are too large, fast, or complex to be processed using traditional database tools. It involves the capture, storage, analysis and visualization of massive data sets.
Doug Laney (2001) — Gartner
First coined the 3 Vs model: Volume, Velocity, Variety — the foundational definition of Big Data that is still used today.
Evolution Timeline
1990s
Data Warehousing Era — enterprises begin storing historical data for reporting.
2001
3 Vs coined by Doug Laney at Gartner META Group.
2004
Google publishes MapReduce paper — foundation for parallel processing of large data.
2006
Apache Hadoop launched (Yahoo!) — open-source big data framework for distributed storage and processing.
2010s
Cloud Big Data era — AWS, Azure, GCP offer managed big data services. Spark, Kafka, NoSQL rise.
Topic 04
Characteristics of Big Data — The 5 V's
The defining properties that distinguish Big Data from traditional data.
Volume
The sheer scale of data generated. Big Data is measured in Petabytes, Exabytes, Zettabytes.
Example: Facebook generates ~4 petabytes of data per day. Google processes ~100 petabytes/day with MapReduce.
Velocity
The speed at which data is generated, collected and processed. Real-time vs batch processing.
Example: Twitter produces ~6,000 tweets/second. Stock market feeds process millions of ticks per second.
Variety
Data comes in multiple formats — structured tables, JSON, videos, images, audio, logs.
Example: A hospital has patient records (structured), MRI scans (images), doctor dictations (audio), and lab PDFs.
Veracity
The quality and trustworthiness of data. Data can be uncertain, noisy or inconsistent.
Example: Social media data has fake profiles, duplicate entries, missing fields. Sensor data may have errors due to hardware faults.
Value
The usefulness of the data — raw data alone means nothing without deriving insights.
Example: Netflix analyzing user watch history to recommend content → increases viewer engagement → $1B+ savings in churn.
Topic 05
Challenges with Big Data
Key obstacles organizations face when dealing with massive datasets.
🗄️ Storage
Storing petabytes of data cost-effectively. Traditional RDBMS can't scale horizontally.
Variability — data meaning changes over time Visualization — complex data needs charts/graphs Viscosity — resistance to data flow in pipelines Virality — speed at which data spreads across networks
Topic 07
Traditional BI vs Big Data Analytics
Understanding how Big Data analytics differs from traditional Business Intelligence.
Dimension
Traditional BI
Big Data Analytics
Data Type
Structured only
Structured + Semi + Unstructured
Data Volume
GB to low TB
Petabytes to Zettabytes
Processing
Batch, nightly ETL
Real-time & batch (Spark, Flink)
Storage
RDBMS, Data Warehouse
HDFS, NoSQL, Cloud Data Lakes
Analysis Type
Descriptive (what happened)
Descriptive + Predictive + Prescriptive
Latency
Hours to days
Milliseconds to seconds
Technology
SQL, Cognos, Business Objects
Hadoop, Spark, Kafka, Hive
Users
Business analysts, managers
Data scientists, ML engineers
Cost
High licensing (SAP, Oracle)
Open-source; cloud pay-as-you-go
Example: A retail bank using traditional BI generates monthly sales reports (descriptive). With Big Data Analytics, the same bank can detect fraudulent transactions in real-time and predict which customers are likely to churn — then act on it immediately.
Topic 08
Data Warehouse vs Hadoop Environment
Two paradigms for large-scale data storage and analysis.
🏛️ Data Warehouse
Centralized repository for structured, historical data
Best for: large-scale ML, log analysis, clickstream data
Raw Data
→
HDFS
→
MapReduce
→
Insights
Hadoop Core Components:HDFS — Distributed StorageYARN — Resource ManagementMapReduce — Processing EngineHive — SQL on HadoopHBase — NoSQL on HDFSPig — Data Flow LanguageSpark — In-memory Engine
Topic 09
Classification of Analytics
The four levels of analytics — from description to prescription.
1️⃣ Descriptive
"What happened?" Summarizes historical data.
Example: Monthly sales report showing revenue by region.
SQLExcelTableau
2️⃣ Diagnostic
"Why did it happen?" Root cause analysis.
Example: Sales dropped in Q3 — drill-down shows product recall in July caused it.
OLAPDrill-down
3️⃣ Predictive
"What will happen?" Forecasting with ML models.
Example: Credit scoring model predicts customer loan default probability.
MLR/Python
4️⃣ Prescriptive
"What should we do?" Recommends optimal actions.
Example: Dynamic pricing engine adjusts Uber fare in real-time based on supply/demand.
AIOptimization
Maturity Ladder: Organizations typically start with Descriptive analytics and progressively adopt Predictive and Prescriptive as they build their data infrastructure and talent.
Topic 09b
Big Data Analytics — Challenges & Importance
Why analytics is hard, and why it matters more than ever.
⚠️ Challenges in Big Data Analytics
Data heterogeneity — combining structured + unstructured data is hard
Scalability — models trained on terabytes need distributed computing
Real-time analysis — low-latency processing for time-critical decisions
Data governance — who owns the data, lineage, compliance
Interpretability — complex ML models are black boxes
Retail: Recommendation engines increase sales by 20-35%
Government: Smart city traffic optimization reduces congestion by 30%
Manufacturing: Predictive maintenance reduces downtime by 45%
Agriculture: Crop yield prediction using satellite + weather data
Gartner's Hype Cycle Insight: Big Data analytics has moved past the "Peak of Inflated Expectations" into the "Plateau of Productivity" — meaning real ROI is now measurable in mature deployments across industries.
Topic 10
Data Science & The Data Scientist
The interdisciplinary field at the heart of Big Data analytics.
🔬 What is Data Science?
Data Science is an interdisciplinary field combining statistics, computer science, and domain expertise to extract actionable insights from data using scientific methods, algorithms, and systems.
Data Science Process (CRISP-DM)
Business Understanding
Define the problem
Data Collection
Gather relevant data
Data Preparation
Clean, transform, EDA
Modelling
Build ML/DL models
Evaluation & Deployment
Validate & go live
👩💻 Who is a Data Scientist?
A Data Scientist is someone who collects, analyzes, and interprets large datasets using statistics, programming, and ML to solve complex business problems.
HBR (2012): Harvard Business Review called Data Scientist "the sexiest job of the 21st century." The average US salary for a Data Scientist is $130,000/year (BLS 2024).
Topic 11
Key Terminologies in Big Data
Essential vocabulary every Big Data practitioner must know.
🗺️ MapReduce
A programming model for processing large datasets in parallel across a cluster. Map phase filters/sorts; Reduce phase aggregates results.
Example: Counting word frequency in 1TB of web pages.
📁 HDFS
Hadoop Distributed File System — stores files across multiple machines with replication (default 3× copies) for fault tolerance.
Example: A 1GB file split into 128MB blocks stored across 8 DataNodes.
⚡ Apache Spark
In-memory distributed computing engine — 100× faster than MapReduce. Supports batch, streaming, ML (MLlib), graph processing.
Example: Real-time clickstream analytics for e-commerce.
🗄️ NoSQL Database
Non-relational DBs designed for horizontal scaling. Types: Document (MongoDB), Key-Value (Redis), Column (Cassandra), Graph (Neo4j).
Example: Instagram uses Cassandra to store billions of photos metadata.
🌊 Data Lake
A centralized repository storing raw data in its native format until needed — unlike data warehouse which stores processed data.
Example: Netflix's AWS S3 data lake stores raw streaming logs, then processes on demand.
🔄 ETL vs ELT
ETL: Extract → Transform → Load (traditional). ELT: Extract → Load → Transform (modern Big Data approach — transform in the data lake).
Example: Snowflake uses ELT; data loaded raw, transformed with SQL later.
Topic 12
BASE — Consistency Model for Big Data
The alternative to ACID for distributed, scalable NoSQL systems.
B
Basically Available
The system guarantees availability — it responds to every request, even if some nodes are down. It may return stale or partial data rather than failing.
Example: Amazon's shopping cart is always available, even if one data center is down.
S
Soft State
The state of the system may change over time, even without input. Data is not guaranteed to be consistent at every moment — it is "soft" or flexible.
Example: Facebook "likes" count may differ by seconds across data centers as updates propagate.
E
Eventual Consistency
Given enough time without new updates, all replicas of data will converge to the same value. Consistency is eventual, not immediate.
Example: DNS updates propagate globally — all servers will eventually show the same IP, though it takes time.
ACID vs BASE ACID (traditional RDBMS): Atomicity, Consistency, Isolation, Durability — strict, safe, but not scalable. BASE (NoSQL/Big Data): Trades strong consistency for availability and scalability — suitable for web-scale systems.
Systems Using BASE
Apache Cassandra — tunable consistency levels
Amazon DynamoDB — eventually consistent reads
CouchDB — multi-master replication
Riak — distributed key-value store
Topic 13
More Big Data Terminologies
Important concepts and terms used in the Big Data ecosystem.
📡 CAP Theorem
A distributed system can only guarantee 2 of 3: Consistency, Availability, Partition Tolerance. BASE systems prioritize A+P.
🌊 Apache Kafka
Distributed message streaming platform. Producers publish events, consumers read in real-time. Used for log aggregation, stream processing.
Example: LinkedIn processes 1 trillion events/day using Kafka.
🐝 Apache Hive
Data warehouse infrastructure on Hadoop. Provides SQL-like querying (HiveQL) on HDFS data. Best for batch analytics.
🐷 Apache Pig
High-level scripting platform (Pig Latin language) for analyzing large datasets on Hadoop. Converts scripts to MapReduce jobs.
🏪 Data Mart
A subset of a data warehouse focused on a specific business function (e.g., sales data mart, HR data mart). Smaller and faster.
📊 OLAP
Online Analytical Processing — multi-dimensional data analysis. Supports drill-down, roll-up, slice, dice operations. Enables complex BI queries.
Topic 14
Top Analytics Tools
The leading platforms and frameworks in the Big Data analytics ecosystem.
🐘
Apache Hadoop
Distributed storage (HDFS) & batch processing (MapReduce). The foundation of Big Data.
Everything you need to remember from Unit 1 at a glance.
Data Classification
Structured → RDBMS, SQL
Semi-structured → JSON, XML
Unstructured → 80% of data
Big Data 5 V's
Volume — Scale of data
Velocity — Speed of data
Variety — Types of data
Veracity — Quality of data
Value — Worth of data
Hadoop Ecosystem
HDFS — Distributed storage
MapReduce — Processing
YARN — Resource management
Hive/Pig/HBase — Tools
Analytics Types
Descriptive — What happened
Diagnostic — Why it happened
Predictive — What will happen
Prescriptive — What to do
BASE Model
Basically Available
Soft State
Eventual Consistency
Used in NoSQL systems
BI vs Big Data
BI → structured, batch, SQL
Big Data → all types, real-time
Hadoop replaces ETL pipelines
Open-source vs proprietary
"Data is the new oil — but only if you can refine it." Big Data gives organizations the tools to refine raw data into competitive advantage.
🎓
End of Unit 1
You've covered the complete foundation of Big Data and Analytics. You're now ready for Unit 2 — Hadoop Architecture, HDFS internals, and MapReduce programming.
18 Slides Covered12 Major Topics30+ Real Examples10+ Visualizations