GraphChi based new partitioning method wins the best paper at SASO 2013!

Just had a great visit in KTH University in Sweden. I learned there on a very interesting work about a new algorithm for graph partitioning from Fatemeh Rahimian who won the best paper award at SASO 2013. The paper uses a simulated annealing based local search to improve the partitioning. 

I got the following clarification from Fatemeh:

Please find attached the two papers that we have, one is for edge-cut partitioning (JabeJa) and the other is for vertex-cut partitioning (JabeJa-vc), which is inspired by the first algorithm, as its name suggest.
Our algorithm, JabeJa, can be executed with different data distribution models: (i) in a completely distributed environment (like a p2p network, where each peer is actually a graph node), we call this model one-host-one node model; or (ii) in an environment where a machine can host a part of the graph, we call this model the one-host-multiple-nodes.

The implementation of JabeJa on GraphChi consists of the following files:
1. JabeJa.java: the main algorithm of JabeJa.
2. JabeJaWeighted.java: this is JabeJa for weighted graphs.
3. MessageRelay.java: this is the file that implements the "mail" API, i.e., "send" and "get".
4. PartitionAnalysis.java: this file writes the final partitions into different output files.

We are working on adding this code contribution to GraphChi Java code, in the meantime anyone who is interested in welcome to email Fatemeh directly. 

Another interesting fact is that a new Data Intensive Computing course at KTH is teaching about GraphLab, among other systems. The slides are available for everyone on the web.

3rd GraphLab Conference is getting closer!!

GraphLab conference attracts the most interesting emerging data science projects. Join us on Monday July 21, 2014 at the Nikko Hotel in SF.

We will have oral talks from GraphLab, Spark, Datapad (a startup from the creator of python pandas), Trifacta ( a startup from the creator of d3.js), Cloudera, Microsoft, Google, Pivotal, Adobe, Lab41, CMU and Pandora. 

We can roughly divide the presenters to several domains: graph analytics (graphlab, pregel, petuum, grappa, stinger, grafos.ml, parameter server etc.), graph databases, graph visualization, python data science tools, and applications on graphs. 

Graph Databases is an emerging field. Graph databases are used to store and query the graph and are optimized for high performance on data which has a graph structure. We will have demos from all the influential graph databases out there: Neo Technology (Neo4j), Aurelius (Titan), Franz, Objectivity (InfiniteGraph), Sparsity Technologies, which are all the leading graph databases companies. 

Visualization helps data scientists deep dive into their data. In terms of visualization, we will have presentations for Trifacta, Cambridge Intelligence, Graphistry (viz using gpus), Linkorious (a startup from the creators of Gephi open source), Ayasdi,  Tom Sawyer Software, Plot.ly

In terms of python/ data science we will have presenters from Skytree, bigML, Zipfian Academy (python training), Continuum Analytics, iPython, Domino Data Labs, Dataiku.

We have a very interesting presence of academic projects.  Some examples are Petuum (CMU) a new system by Prof. Eric Xing, Parameter Server (CMU) a mega scale framework for cluster implementation of ML methods by Prof. Alex Smola. Grappa (UW) by mark Oskin from UW, a super fast graph analytic framework. Stinger - a streaming graph system from Georgia Tech.

Graphs are everywhere! We assembled the most interesting use cases for graphs in industry. For example, Senzari, a company based in Florida is creating the largest music graph - with 100 billion facts related to music! Ravel law is using graphs obtain by supreme court rules to deduce interesting and useful facts about law. Lumiata is compiling a healthcare graph for medical science based graph analytics. Crosswise is using graphs for security and entity disambiguation purposes.

GraphLab conference started with 300 attendees on 2012, grown to 600 attendees in 2013, and we expect 900 data scientists in 2014. Secure your place today!

A limited special offer of 20% discount: Dannysblog. This offer will expire in a couple of days!

100 interesting datasets

I got this link from my colleague Zach Nation: 100 interesting datasets.

O'Reilly visit in Israel

A couple of weeks ago I enjoyed hosting O'Reilly guys (and girl) in Israel. Here is a venturebeat article which covers this visit. The participants were Ben Lorica (chief scientist), Roger Magoulas (director of research) and Gina Blabler (VP conferences).

O'Reilly team came to examine the Israeli high tech scene. We are interested in promoting an Israeli Strata conference in Israel and already assembled a few companies who would like to support this effort. Anyone interested in helping promote this is welcome to connect.

Polychart: front end visualization of mysql data

I got this from my colleague Eric Wolfe. Another interesting attempt to make data science easier: Polychart has a UI which allows drag and drop plotting of mysql data tables. The drawback is that you basically need to know what you are looking for - but still an interesting attempt to make data drilling easier. 

Pacific Northwest National Lab uses GraphLab for its Cyber Security Project

Just got a nice visualization from PNNL researcher Sutanay Choundhury. It uses GraphLab spectral clustering to cluster different nodes in a network. And here is a paragraph about the meaning of the graphs:
The input graph was partitioned using the spectral clustering implementation in GraphLab. The size of the nodes in the rendered graph is determined by a cost function. In this visualization, the cost function simply returns the degree of the node, although one may envision using centrality or other non-graph theoretic metric of importance. We used a threshold k (20 in this case), to display at most k nodes with highest costs in a cluster. The goal of the visualization was to provide a capability to see the distribution of "important" nodes across the clusters.

The above effort is a part from a larger project. Pacific Northwest National Lab, USA has a cyber security related project which uses GraphLab, among other tools. The project is called M&Ms4Graphs: multi-scale, multi-dimensional graph analytics for cyber security. Anyone who is interested in learning more about it, is welcome to attend our 3rd GraphLab conference, where Sutanay will give a demo.

2nd Large Scale Recommender Systems Workshop

We just announced our 2nd Large Scale Recommender System Workshop (LSRS 2014), to take place jointly with RecSys 2014. The workshop organizing committee is Tao Ye from Pandra, Qiang Yan from Taobao and your humble servant.

Last year, our workshop attracted the largest audience in RecSys.

Anyway who is working on large scale recommender systems is encouraged to submit an abstract or slides which describe their work. Submission date is July 21, 2014.