Giraph Machine Learning Project - Setting up

Giraph machine learning project, is a relatively new large scale machine learning project at incubation stage under Apache. It is the only open source implementation I am aware of Google's Pregel (BSP = Bulk Synchronous Parallel) framework.

I got the following instructions, from my colleague and friend Aapo Kyrola:

1. INSTALL HADOOP: Must be version 0.20.203 or later.
- This is simple, just download and extract.

2. Set HADOOP_HOME variable to point to the hadoop directory.

3. Set Hadoop configuration (under HADOOP_HOME/config) according
to what is explained here.

* NOTE: set the hdfs directory appropriately: core-site.xml, property hadoop.tmp.dir

3.5 Start Hadoop:
bin/start-all.sh

4. Install zookeeper
- just download and extract

5. Configure conf/zoo.cfg properly. (Just copy the sample config and change to sensible parameters).
- set clientPort=22181

6. Start up zookeeper:
bin/zkServer.sh start

7. Install and build Giraph as explained in the end of this website:
http://incubator.apache.org/giraph/

8. In HADOOP_HOME, run PageRank:

bin/hadoop jar ../../GraphLab/giraph/giraph/trunk/target/giraph-0.70-jar-with-dependencies.jar org.apache.giraph.benchmark.PageRankBenchmark -e 100 -s 5 -V 10000 -w 1 -v

If everything went OK you will get:

11/09/19 18:23:20 INFO mapred.JobClient:   Giraph Timers
11/09/19 18:23:20 INFO mapred.JobClient:     Total (milliseconds)=260128
11/09/19 18:23:20 INFO mapred.JobClient:     Superstep 3 (milliseconds)=54578
11/09/19 18:23:20 INFO mapred.JobClient:     Setup (milliseconds)=2771
11/09/19 18:23:20 INFO mapred.JobClient:     Shutdown (milliseconds)=92
11/09/19 18:23:20 INFO mapred.JobClient:     Vertex input superstep (milliseconds)=2386
11/09/19 18:23:20 INFO mapred.JobClient:     Superstep 0 (milliseconds)=8059
11/09/19 18:23:20 INFO mapred.JobClient:     Superstep 4 (milliseconds)=70263
11/09/19 18:23:20 INFO mapred.JobClient:     Superstep 5 (milliseconds)=1879
11/09/19 18:23:20 INFO mapred.JobClient:     Superstep 2 (milliseconds)=66531
11/09/19 18:23:20 INFO mapred.JobClient:     Superstep 1 (milliseconds)=53564
11/09/19 18:23:20 INFO mapred.JobClient:   Giraph Stats
11/09/19 18:23:20 INFO mapred.JobClient:     Aggregate edges=1000000
11/09/19 18:23:20 INFO mapred.JobClient:     Superstep=6
11/09/19 18:23:20 INFO mapred.JobClient:     Current workers=1
11/09/19 18:23:20 INFO mapred.JobClient:     Current master task partition=0
11/09/19 18:23:20 INFO mapred.JobClient:     Sent messages=0
11/09/19 18:23:20 INFO mapred.JobClient:     Aggregate finished vertices=10000
11/09/19 18:23:20 INFO mapred.JobClient:     Aggregate vertices=10000

Anyway Aapo has a great Nordic sense of humor. This is what he sent me later:
For your convenience, I have pasted the documentation of Giraph to this email.

-- Begin --
-- End --


Additionally, a quick start document is available here:
https://github.com/aching/Giraph/wiki/Quick-Start-Guide

Giraph Machine Learning Project

I heard from Joey Gonzalez, about this project: http://incubator.apache.org/giraph/project-info.html :


"It is an open-source framework for large-scale graph computation that is community maintained (Apache incubator status), with contributors from LinkedIn and Yahoo!.
It runs on very large (4000 node) cloud systems, and does so efficiently (in memory using low-latency communication and fault-tolerant snapshots)."


"Giraph was recently featured at the HotCloud conference and has been going on a lecture circuit around the valley (...).  Unlike Pegasus Giraph is relatively well designed with in memory storage, HadoopRPC messaging, infrequent snapshots, and map-only execution.   Basically with the exception of startup and occasional snapshots, Giraph runs entirely in memory and communicates
over sockets."

It definitely sounds like an interesting project. Although no Giraph will beat our labardor! (GraphLab.. :-)