Apache Spark a brief overview

Apache Spark a PHD resarch project at UCBerkley came into limelight when it broke the record of sorting petabyte of data recently.
Various Sorting Records.
Databrick



What is Spark ?

Apache Spark™ is a fast cluster computing framework and general engine for large-scale data processing.

Spark Goals:
Generality : Diverse workloads, operators, jobsizes.
Latency:Low Latency
Fault Tolerance


Spark supports Hadoop, Amazon S3, Cassandra, cluster management tools like YARN and Mesos. Spark does more In memory data processing as compared to DiskBased processing as in Hadoop.

Spark stack comes bundled with tools like Spark SQL, MLlib, Spark Streaming and GraphX.

1. SparkSQL: Unified access to structured data , provides compatibility with Apace Hive and Standard Connectivity to tools like JDBC and ODBC.
2. Spark Streaming:  For Scalable fault tolerant Streaming applications and spark can run in both batch and interactive mode.
3. MLlib:Scalable Machine Learning library.
4. GraphX:Large Scale Graph Processing Framework.

Learn More about Spark


Spark vs Hadoop

Spark is 100X times faster then Hadoop.
The speed can be attributed to the fact that Spark keeps the intermediate data cached in local JVM.  Hadoop on the other hand in the name of Fault Tolerance write the intermediate data on to the disk and disk is expensive.
                                                                                                               *Image from Spark

Spark doesnt replace anything in Hadoop Ecosystem rather it offers a readable, testable way to write programs freeing us from the painful Map Reduce jobs. MR model is unsuitable for Iterative algorithms. MR jobs are pain to program too. Although there are tools to reduce our efforts in writing MR jobs like Hive, Cascading etc but internally they call MR jobs thus not improving the performance.

Spark programming model
The main abstraction for computation in Spark is Resilient(Can start automatically) Distributed Datasets.

What is RDD ? 
RDD(Spark Paper) :

RDD's are fault-tolerant, parallel data structures that let users explicitly persist intermediate results in memory, control their partitioning to optimize data placement, and manipulate them using a rich set of operators.

Spark transformations applies same data operation to many data items. This results in better Fault tolerance as only the lineage of transformation is logged rather then the actual data. Also if some of RDD's are lost it has enough information to how it is derived from other  RDD's.
RDD can be created through Transformations(map,filter,join). Spark creates Direct Acyclic Graph of these transformation . Once the RDD's are defined through transformations , actions can be applied on them.
Actions are application that returns a value(count,collect and save). In Spark RDD's can be stored in disk by calling persist.
Since it is just an RDD it can be queried via SQL Interface, ML algos etc.

In short
User Program => 
Create Spark Context
sc = new SparkContest
Create Distributed Datasets called RDD's 
Perform Operations.

Inside spark context act as client and master per application.

Block tracker => What is in memory what is on Disk ?
Shuffle => Shuffle operation like Groupby

Scheduler talk through cluster manager talks to a worker.
Contains Block Manager for Block Mgmt.
Recieves task that run in thread pools.
Task can talk to HDFS.

Probabilistically Bounded Staleness for Practical Partial Quorums : Summery

I recently gave a presentation on paper by PeterBailis and his friends at University of California Berkley
http://vldb.org/pvldb/vol5/p776_peterbailis_vldb2012.pdf

Unlike other papers does not propose a consistency model rather the paper talks about a probabilistic model to predict the staleness of data . They introduce term called Probabilistic bounded staleness to provide bounds on staleness of data wrt both version and clock time. I found the paper pretty interesting and I feel that because of the move more and more towards eventual consistency the question "How Eventual and How Consistent " makes more sense as it will help the users to choose the best available read and write quorum configuration for there system. Or in words it provides a lens for analyzing, improving, and predicting the behavior of existing, widely deployed systems.

Quorum System :  A quorum system is a collection of sets where all pairs of sets intersect.

A (strict) quorum System Q* over a universe U is a set system over U such that for every
    Q1, Q2 Є Q, Q1 П Q0 ≠ Ф Each Q Є Q is called a quorum.
N – No of nodes that store Replica
W – No of Replicas acknowledge the receipt of the update before the update completes
R – the number of replicas that are contacted when a data object is accessed through a read operation
For Strong  Consistency : R+W  > N (Read ,Write set Overlap)
For Eventual Consistency :  R+W =< N

* Probabilistic Quorum Systems : Dahlia Malkhi

Probabilistic Quorum Systems
Relax  Quorum Intersection Property
Hold with High Probability 1 – ε (ε small Number)
Scaling no of replicas increase probability

A quorum system obeys PBS k-staleness consistency with probability 1 –psk where psk is the probability of non-intersection with one of the last k independent quorums.

 N = 3, R = W = 1
Probability of returning a version
Within 2 versions is 0.5
Within 3 versions is 0.703
Within 5 versions > 0.868
Within 10 versions > 0.98

t-visibility is the probability that a read operation, starting t seconds after a write commits will observe the latest value of data.
t- inconsistency window.


Assumption :  reads occur instantaneously
and writes commit immediately after W replicas there is no delay acknowledging the write to the coordinating node.

A quorum system obeys PBS <k-t> -staleness consistency if, with probability 1- pskt, at least one value in any read quorum will be within k versions of the latest committed version when the read begins, provided the read begins t units of time after the previous k versions commit.



 


* http://vldb.org/pvldb/vol5/p776_peterbailis_vldb2012.pdf