Setting up spark on a multicluster Machine

1. Spark work on a master - slave model. 
2. Master Machine and Slave Machines required. You can have master and slave on same machine too. 
3. For testing purpose you can generate the cluster on same machine , approach similar 
4. In each machine check if Java is installed
   
   JAVA install/update 
5. If you are using a new machine, ec2 it wont come preinstalled with Java.  
6. You can set up Java following below steps
7.  First update package index :
   1. sudo apt-get update
   2. check version java -version 
   3. If Java not found →  Install JRE 
      1. sudo apt-get install default-jre
      2. sudo apt-get install default-jdk
8. check Java-version 
9. This might install Java 1.6 
10. sudo apt-get install openjdk-7-jre
11. sudo apt-get isntall openjdk-7-jdk
12. Issue : Java still points to version 6. 
13. Follow below steps to update 
    1. update java-alternatives -l
    2. sudo update-java-alternatives s java1.7.0-openjdk-amd64 (your path to the newer) 
    
    Instance Setup 
14. Login to Dev or EC2 instances 
15. If Running on Ec2 Do the following:
    1. Create an account, generate private  and public key pairs download the keys to one of the machine. You can get those steps online on how to generate key pairs. 
    2. Create two system variables for AWS_ACCESS_KEY and AWS_SECRET_KEY
    3. Go to spark directory on your local machine  and run the ec2 script from ./sbin/ec2 file.
    4. ./spark-ec2 --key-pair=awskey --identity-file=awskey.pem --region=us-west-1 --zone=us-west-1a launch my-spark-cluster
    5. This will create two Large instances cluster. 
    
    SetUp Spark 
16. Download Spark → Select Hadoop latest  Version and follow the link  
17. For dev box : you can download using : 
1. wget spark link 
2. unzip using tar -xzf folder for spark
18. Do this in all machines you want to run spark on .
19. Set up env variable SPARK_HOME to point folder just before the bin directory, in normal case it will be the unzipped folder
20. Save it using source ./bashrc
21. Decide on which machine you want your master and slave
    
    
    Master Machine 
22. Login to Master Machine, follow below steps
23. edit hosts file for master
    1. vi /etc/hosts
    2. Provide the link between IP and name 
    3. IP Machine1
       IP Machine 2 
        
24. Next few steps are changes in spark code
    1. slaves template is located in the conf directory.
    2. there is a file called slave template, same file can be used or a new file called you can create from that buy copying the content to slaves
    3. Paste the names of all slaves in this file 
    1. slave 1
    2. slave 2
    3. slave 3
25. Go to Spark Sbin Directory 
    1. cd ./sbin/
    2. run ./sbin/start-master.sh
26. This will start the master on port 8080, access it via IP address or Machine Name
27. For current case go to any web browser and type :  http://machinename:8080
28. This was required because you need to get the spark master address. When you go to the link it will show the url at the top spark://machinename:7077 
29. Copy it 
30. Go to /conf/spark-defaults.conf.template
31. Add the masters url in the file 
32. spark.master spark://machinename:7077 (it can also start with IP or a bigger name like machinename:7077
33. Go to /sbin/spark-env.sh
34. Add the following lines to it 
35. export SPARK_WORKER_MEMORY=1g  You can specify how much memory you wanna give to worker 
    export SPARK_WORKER_INSTANCES=2 → No of workers
    export SPARK_MASTER_IP=spark-2 → Name of Spark master
    Slave
36.  Go to slave Machine /sbin/start-slave.sh
37. Run JPS in both slaves and master

Note : You need a password less access between master and  copy masters pub key defined in : id_rsa.pub move to authorized_key in slave and save. 

Very Useful links

http://blog.cloudera.com/blog/2015/03/how-to-tune-your-apache-spark-jobs-part-1/

http://blog.cloudera.com/blog/2015/05/working-with-apache-spark-or-how-i-learned-to-stop-worrying-and-love-the-shuffle/

https://www.codementor.io/spark/tutorial/spark-python-rdd-basics

Install Hadoop steps - Both single and Multiple machine...

Hadoop Installation for Windows..

Steps :

A. Prequisites

1. Install Putty. http://www.chiark.greenend.org.uk/~sgtatham/putty/

2.Get Ec2 Instance.

3.Go to EC2 instance and check the public DNS name.

4.Get Private key(.pem)

B.Convert .pem to .ppk

1. Start PuTTYgen.

2.Under Type of key to generate, select SSH-2 RSA.

3.Click Load. By default, PuTTYgen displays only files with the extension .ppk. To locate your .pem file, select the option to display files of all types. Generate PPK from Pem file you obtained from ec2.

 4a. Click Load if already saved Config

 

4b.Click SSh to browse PPK key.

5.Install JAVA (Hadoop is written in Java)

sudo apt-get install openjdk-7-jdk

6.Install OpenSSH for SSH

sudo apt-get install openssh-server

7.Create Hadoop user. So that you can run your program

sudo addgroup hadoop // creating Hadoop Group

sudo adduser --ingroup hadoop hduser // adding user to the group

sudo adduser hduser sudo

8. Generating Keys

$ ssh-keygen -t rsa -P ''

...

Your identification has been saved in /home/hduser/.ssh/id_rsa.

...

$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

$ ssh localhost

9.Disabling IPV6

net.ipv6.conf.all.disable_ipv6 = 1

net.ipv6.conf.default.disable_ipv6 = 1

net.ipv6.conf.lo.disable_ipv6 = 1

9. Installing Hadoop

Cd /usr/local/ 

sudo wget http://archive.apache.org/dist/hadoop/core/hadoop-0.20.203.0/hadoop-0.20.203.0rc1.tar.gz

//Unzip the tar

sudo tar xzf hadoop-0.20.203.0rc1.tar.gz

//move 

sudo mv hadoop-0.20.203.0rc1 hadoop

//set ownership   

sudo chown -R hduser:hadoop hadoop

10.Update .bashrc with the proper path

cd ~

add java_home

    

export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64

export HADOOP_HOME=/usr/local/hadoop

11.cd /usr/local/hadoop/etc/hadoop

vi hadoop-env.sh

JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64

12.Create tmp for hdfs 

$ sudo mkdir -p /app/hadoop/tmp

$ sudo chown hduser:hadoop /app/hadoop/tmp

# ...and if you want to tighten up security, chmod from 755 to 750...

$ sudo chmod 750 /app/hadoop/tmp

13. Configure Hadoop - edi core-site.xml

cd /usr/local/hadoop/etc/hadoop

or 
cd vi /usr/local/hadoop/cong/core-site.xml

vi core-site.xml

For version <2 

<property>

  <name>hadoop.tmp.dir</name>

  <value>/app/hadoop/tmp</value>

  <description>A base for other temporary directories.</description>

</property>

<property>

  <name>fs.default.name</name>

  <value>hdfs://localhost:54310</value>

  <description>The name of the default file system.  A URI whose

  scheme and authority determine the FileSystem implementation.  The

  uri's scheme determines the config property (fs.SCHEME.impl) naming

  the FileSystem implementation class.  The uri's authority is used to

  determine the host, port, etc. for a filesystem.</description>

</property> 

For version >2

 Bcz of Yarn

<property>

   <name>fs.default.name</name>

   <value>hdfs://localhost:9000</value>

</property

13. update for version < 2

mapred-site.xml

<property>

  <name>mapred.job.tracker</name>

  <value>localhost:54311</value>

  <description>The host and port that the MapReduce job tracker runs

  at.  If "local", then jobs are run in-process as a single map

  and reduce task.

  </description>

</property> 

for version >2

<property>

   <name>yarn.nodemanager.aux-services</name>

   <value>mapreduce_shuffle</value>

</property>

<property>

   <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>

   <value>org.apache.hadoop.mapred.ShuffleHandler</value>

</property>

14. Set Replication Factor 

file conf/hdfs-site.xml

<property>

  <name>dfs.replication</name>

  <value>1</value>

  <description>Default block replication.

  The actual number of replications can be specified when the file is created.

  The default is used if replication is not specified in create time.

  </description>

</property>

15. Format nAmenode

hadoop namenode -format

16.Start-all.sh

depecated for v >2 but will work Invokes both dfs and mapred-site.xml

 ----- Till here single node ---

For multinode.. 

17.Everything is working fine..

18.Create few more instances of EC2 clusters in same way

Update vi etc/hosts from home directory

add IP address and name which you will use

ex...

10.23.45.67 master10.11.11.11 slave1

10.12.12.12 slave2

19.Copy pub key of master to the list of authorized key of client 

vi /home/hduser/.ssh/id_rsa.pub

add the content to 

vi /home/hduser/.ssh/id_rsa/authorized_keys of slave1

20.you can ssh to both master and slave

21.once you type ssh slave you will be taken to slave machine 

exit it from maser

22.On master edit conf/master 

master

23. On master edit conf/slaves  

add the following

master

slave1 

slave2

23. Update core-site.xml ALL MACHINE

 <property>

  <name>fs.default.name</name>

  <value>hdfs://IPADDRESS OF MASTER HERE:54310</value>

  <description>The name of the default file system.  A URI whose

  scheme and authority determine the FileSystem implementation.  The

  uri's scheme determines the config property (fs.SCHEME.impl) naming

  the FileSystem implementation class.  The uri's authority is used to

  determine the host, port, etc. for a filesystem.</description>

</property>

23. update mapred-site.xml for all machine

<property>

  <name>mapred.job.tracker</name>

  <value>masterIP ADDRESS:54311</value>

  <description>The host and port that the MapReduce job tracker runs

  at.  If "local", then jobs are run in-process as a single map

  and reduce task.

  </description>

</property> 

24.set  replication update hdfs-site.xml

<property>

  <name>dfs.replication</name>

  <value>2</value>

  <description>Default block replication.

  The actual number of replications can be specified when the file is created.

  The default is used if replication is not specified in create time.

  </description>

</property> 

25.format name node 

26.start-all.sh

This is what i got from reading from various blogs.. I have installed Hadoop many times 

and feel like I should write the steps  

Its the mix of both single and multinode cluster

I used http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-multi-node-cluster/#java-io-ioexception-incompatible-namespaceids

but modified in many ways..