Hadoop Overview and Terminology

 Trying to list the components of Hadoop as an overview:

1. HDFS (Hadoop Distributed File system)

• Data Storage
• Distribute the storage of Big Data across clusters of computers
• Makes all clusters look like 1 file system
• Maintains copies of data for backup and also to be used when a computer in a clusters fails.

2. YARN (Yet another resource negotiator)

• Data Processing
• Manages the resources on the computer cluster
• Decides which nodes are free/available to be run etc

3. MapReduce

• Process Data
• Programming model that allows one to process data
• Consists of mappers and reducers
• Mappers transform data in parallel
• Converts raw data into key value pairs
• Same key can appear multiple times in a dataset
• Shuffle and sort to sort and group the data by keys
• Reducers aggregate the data
• Process each keys values

4. PIG

• SQL Style syntax
• Programming API to retrieve data

5. HIVE

• Like a SQL Database
• Can run SQL queries on the database (makes the distributed data system look like a SQL DB)
• Hadoop is not a relational DB and HIVE makes it look like one

6. Apache Ambari

• Gives view of your cluster
• Resources being used by your systems
• Execute HIVE Queries, import DB into HIVE
• Execute PIG Queries

7. SPARK

• Allows to run queries on the data
• Create spark queries using Python/Java/Scala
• Quickly and efficiently process data in the cluster

8. HBASE

• NoSQL DB
• Columnar Data store
• Fast DB meant for large transactions

9. Apache Storm

• Process streaming data

10. OOZIE

• Schedule jobs on your cluster

11. ZOOKEEPER

• Coordinating resources
• Tracks which nodes are functioning
• Applications rely on zookeeper to maintain reliable and consistent performance across clusters

12. SQOOP

• Data Ingestion
• Connector between relational DB and HDFS

13. FLUME

• Data Ingestion
• Transport Web Logs to your cluster

14. KAFKA

• Data Ingestion
• Collect data from a cluster or web servers and broadcast to HADOOP cluster.

Probability (Simple/Conditional)

Probability is how likely is it that something will occur.

All probabilities are numbers between 0 and 1. A probability of 1 indicates 100% chance that something will occur. Probability of 0 implies 0% chance that something will occur.

A probability of getting a heads when we flip a coin once is 50% or 0.5

A probability value between 0 and 0.5 implies its most likely the event may not occur.

A probability value of 0.5 implies the event has an equal chance to occur or non-occur.

A probability value between 0.5 and 1 implies its most likely the event may occur.

Probability of an event occurring is expressed as

P(event) = (All possible outcomes that meet the criteria)/(All possible outcome)

Probability of a coin being flipped and we getting a tails is:

P(Tails) = 1/2 = 0.5

since in this case there are 2 possible outcomes (Heads and Tails)

Probability of flipping a dice and getting the number 5 is

P(5) = 1/6 = 0.16 (or 16%)

Since there can be 6 possible outcomes when we flip a coin (from 1...6)

What I described above is Theoretical Probability.

We also have Experimental Probability, which basically is probability when we conduct tests and record them.

e.g we flipped a dice 7 times and got the number five 4 times.

Whats the experimental probability?

P(5) = 4/7 = 0.57 (57%)

As per Law of Large numbers, the more experiments we conduct, we shall see that the Experimental probability value will get closer and closer to the Theoretical probability value.

The Addition Rule:

Definition:

The addition rule states the probability of two events is the sum of the probability that either will happen minus the probability that both will happen.

e.g.

What is the probability of getting a "Heads" if we flip a coin 2 times?

When we flip a coin, the options we have is:

Heads - Heads

Heads - Tails

Tails - Heads

Tails - Tails

So by the definition of the addition rule:

P(Heads) = P(Getting heads in the first flip) + P(Getting heads in the second flip) - P(Both flips we get heads)

P(Heads) = (2/4) + (2/4) - (1/4) = 3/4 = 75%