Apache spark in docker container

I was participating in an IndustryHack last month and as a proof of concept wanted to use machine learning and predictive analysis of sensor data.

Basically the need of the hour was to have Apache Spark up and running in minimal configuration and be able to use predictive analysis with limited data and see the data correlation of different events. The sensor data was provided by the company who was hosting the hackathon.

I thought that it would be great if we can start Apache Spark in docker container and play with it during the hackathon as we don't need a full fledged cluster to do initial learning and linear regression based predictions of the data.

The docker file that we used for starting a container with Apache Spark is as follows,

 # Ubuntu Dockerfile
 #
 # https://github.com/dockerfile/ubuntu
 #
 FROM ubuntu

 # File Author / Maintainer
 MAINTAINER Ganesh Vasudevan

 RUN apt update && \
     apt install -y wget vim less ssh default-jre && \
     sed -i 's/PermitRootLogin prohibit-password/PermitRootLogin yes/' /etc/ssh/sshd_config && \
     cd home && \
     wget http://d3kbcqa49mib13.cloudfront.net/spark-2.0.2-bin-hadoop2.7.tgz && \
     tar xvzf spark-2.0.2-bin-hadoop2.7.tgz

 EXPOSE 22

 # Define default command.
 ENTRYPOINT service ssh restart && bash

This basically downloads the latest ubuntu, installs the packages and fetches the spark 2.0.2 and untars the same in /home/.

To get the docker up and running,

1. copy the file and execute docker build -t
2. Once the image is built successfully docker run -i -t

This will start an interactive session, if you need ssh to the ubuntu instance then set a new password (passwd to set password) and then can ssh from your terminal. Once ssh is working you can stop the docker and start without -i flag.

Cant test the apache spark by running some examples which are shipped with the latest Spark.

cd /home/spark-2.0.2-bin-hadoop2.7
./bin/run-example SparkPi

This will execute the Pi calculation and provide the result. The examples are located in
examples/src/main/scala/org/apache/spark/examples/

With this above setup we were able to do initial studies and data analysis within few hours.

Once you land in Helsinki airport, what next?

Finland is a land of mobile phones, well said, if u are traveling to Helsinki then don't look for pay phones here. You cant find those pay phones anywhere. I heard that there are some payphones in the airport but never managed to find one. So how do u contact friends waiting for you, or how do you call some one and tell that u reached safe?

If you are planning to live in here in Finland for some time then its mandatory to have a phone connection. Pre paid connections are cheap and instant connection and you dont need to submit any proof or other papers to get a connection, kewl. What next? Where to buy them?

Pre paid SIM card can be bought from R-kioski. DNA, Saunalahti, and elisa are the main service providers and the SIM card could cost between 5 to 10 euros. Have a look at the Helsini-Vanta aiport web page. [ https://www.finavia.fi/en/helsinki-airport/ ] You must be arriving thru the arrival gate 2A or 2B. There are 3 R-Kiosks in the Helsinki International airport. You can search for the shop in the map (There is a search tab, click on services A-O, you should be able to find the R-Kiosk there). You are free to go to the departure hall and there are no security gate in between Arrival and departure terminal. So the nearest R-Kiosk could be the next one to the Gate 2A. Its on the 1st floor (Or second floor?? there is some confusion between the floor counting between indian and europe style. The floor cont starts from 1 in finland.), take the stairs and on your left hand side you can locate the R-Kiosk.
Once you are in the shop,
1. Ask for a pre paid SIM card. They should be explaining the different possibility. I am not sure which one is the best or cheap. But buy one SIM card there. It should come with some talk time as well, but you could buy a 10 Euro top up card.
2. If you are planning to travel here in public transport then buy a travel card as well. Travel card costs 9 euros. You could charge the travel card with some money say 20 Euros. You can do all this R-Kioski. Just say i need a travel card and want to load 20 Euros to the card. This could cost you 20 Euros. You can read more about the Travel cards here [ http://www.ytv.fi/ENG/transport/tickets_and_fares/frontpage.htm ]. The same travel card can be used for tram, train, metro, ferry and bus transport in Helsinki, Espoo and vanta.

ISD calls are costly with the prepaid mobile connections. So keep your ISD calls sweet and short. Will write different post about making ISD calls which are bit cheaper than this.

Now you are all set to find the nearest exit and catch a cab and go to your hotel or apartment.

Have a nice stay in Finland! :)