Mahout - SVD matrix factorization - formatting input matrix

Converting Input Format into Mahout's SVD Distributed Matrix Factorization Solver

Purpose

The code below, converts a matrix from csv format:

<from row>,<to col>,<value>\n

Into Mahout's SVD solver format.

For example, 
The 3x3 matrix:
0    1.0 2.1
3.0  4.0 5.0
-5.0 6.2 0


Will be given as input in a csv file as:
1,0,3.0
2,0,-5.0
0,1,1.0
1,1,4.0
2,1,6.2
0,2,2.1
1,2,5.0

NOTE: I ASSUME THE MATRIX IS SORTED BY THE COLUMNS ORDER
This code is based on code by Danny Leshem, ContextIn.

Command line arguments:

args[0] - path to csv input file
args[1] - cardinality of the matrix (number of columns)
args[2] - path the resulting Mahout's SVD input file

Method:
The code below, goes over the csv file, and for each matrix column, creates a SequentialAccessSparseVector which contains all the non-zero row entries for this column.
Then it appends the column vector to file.

Compilation:
Copy the java code below into an java file named Convert2SVD.java
Add to your IDE project path both Mahout and Hadoop jars. Alternatively, a command line option for compilation is given below.

import java.io.BufferedReader;
import java.io.FileReader;
import java.util.StringTokenizer;

import org.apache.mahout.math.SequentialAccessSparseVector;
import org.apache.mahout.math.Vector;
import org.apache.mahout.math.VectorWritable;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.SequenceFile;
import org.apache.hadoop.io.SequenceFile.CompressionType;

/**
 * Code for converting CSV format to Mahout's SVD format
 * @author Danny Bickson, CMU
 * Note: I ASSUME THE CSV FILE IS SORTED BY THE COLUMN (NAMELY THE SECOND FIELD).
 *
 */

public class Convert2SVD {


        public static int Cardinality;

        /**
         * 
         * @param args[0] - input csv file
         * @param args[1] - cardinality (length of vector)
         * @param args[2] - output file for svd
         */
        public static void main(String[] args){

try {
        Cardinality = Integer.parseInt(args[1]);
        final Configuration conf = new Configuration();
        final FileSystem fs = FileSystem.get(conf);
        final SequenceFile.Writer writer = SequenceFile.createWriter(fs, conf, new Path(args[2]), IntWritable.class, VectorWritable.class, CompressionType.BLOCK);

          final IntWritable key = new IntWritable();
          final VectorWritable value = new VectorWritable();

   
           String thisLine;
        
           BufferedReader br = new BufferedReader(new FileReader(args[0]));
           Vector vector = null;
           int from = -1,to  =-1;
           int last_to = -1;
           float val = 0;
           int total = 0;
           int nnz = 0;
           int e = 0;
           int max_to =0;
           int max_from = 0;

           while ((thisLine = br.readLine()) != null) { // while loop begins here
            
                 StringTokenizer st = new StringTokenizer(thisLine, ",");
                 while(st.hasMoreTokens()) {
                     from = Integer.parseInt(st.nextToken())-1; //convert from 1 based to zero based
                     to = Integer.parseInt(st.nextToken())-1; //convert from 1 based to zero basd
                     val = Float.parseFloat(st.nextToken());
                     if (max_from < from) max_from = from;
                     if (max_to < to) max_to = to;
                     if (from < 0 || to < 0 || from > Cardinality || val == 0.0)
                         throw new NumberFormatException("wrong data" + from + " to: " + to + " val: " + val);
                 }
              
                 //we are working on an existing column, set non-zero rows in it
                 if (last_to != to && last_to != -1){
                     value.set(vector);
                     
                     writer.append(key, value); //write the older vector
                     e+= vector.getNumNondefaultElements();
                 }
                 //a new column is observed, open a new vector for it
                 if (last_to != to){
                     vector = new SequentialAccessSparseVector(Cardinality); 
                     key.set(to); // open a new vector
                     total++;
                 }

                 vector.set(from, val);
                 nnz++;

                 if (nnz % 1000000 == 0){
                   System.out.println("Col" + total + " nnz: " + nnz);
                 }
                 last_to = to;

          } // end while 

           value.set(vector);
           writer.append(key,value);//write last row
           e+= vector.getNumNondefaultElements();
           total++;
           
           writer.close();
           System.out.println("Wrote a total of " + total + " cols " + " nnz: " + nnz);
           if (e != nnz)
                System.err.println("Bug:missing edges! we only got" + e);
          
           System.out.println("Highest column: " + max_to + " highest row: " + max_from );
        } catch(Exception ex){
                ex.printStackTrace();
        }
    }
}

A second option to compile this file is create a Makefile, with the following in it:

all:
        javac -cp /mnt/bigbrofs/usr7/bickson/hadoop-0.20.2/lib/core-3.1.1.jar:/mnt/bigbrofs/usr7/bickson/mahout-0.4/taste-web/target/mahout-taste-webapp-0.5-SNAPSHOT/WEB-INF/lib/mahout-core-0.5-SNAPSHOT.jar:/mnt/bigbrofs/usr7/bickson/mahout-0.4/taste-web/target/mahout-taste-webapp-0.5-SNAPSHOT/WEB-INF/lib/mahout-math-0.5-SNAPSHOT.jar:/mnt/bigbrofs/usr7/bickson/hadoop-0.20.2/lib/commons-cli-1.2.jar:/mnt/bigbrofs/usr7/bickson/hadoop-0.20.2/hadoop-0.20.2-core.jar *.java

Note that you will have the change location of the jars to point to where your jars are stored.

Example for running this conversion for netflix data:

java -cp .:/mnt/bigbrofs/usr7/bickson/hadoop-0.20.2/lib/core-3.1.1.jar:/mnt/bigbrofs/usr7/bickson/mahout-0.4/taste-web/target/mahout-taste-webapp-0.5-SNAPSHOT/WEB-INF/lib/mahout-core-0.5-SNAPSHOT.jar:/mnt/bigbrofs/usr7/bickson/mahout-0.4/taste-web/target/mahout-taste-webapp-0.5-SNAPSHOT/WEB-INF/lib/mahout-math-0.5-SNAPSHOT.jar:/mnt/bigbrofs/usr7/bickson/hadoop-0.20.2/lib/commons-cli-1.2.jar:/mnt/bigbrofs/usr7/bickson/hadoop-0.20.2/hadoop-0.20.2-core.jar:/mnt/bigbrofs/usr7/bickson/hadoop-0.20.2/lib/commons-logging-1.0.4.jar:/mnt/bigbrofs/usr7/bickson/hadoop-0.20.2/lib/commons-logging-api-1.0.4.jar Convert2SVD ../../netflixe.csv 17770 netflixe.seq

Aug 23, 2011 1:16:06 PM org.apache.hadoop.util.NativeCodeLoader
WARNING: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Aug 23, 2011 1:16:06 PM org.apache.hadoop.io.compress.CodecPool getCompressor
INFO: Got brand-new compressor
Wrote a total of 241 rows, nnz: 1000000
Wrote a total of 381 rows, nnz: 2000000

Wrote a total of 571 rows, nnz: 3000000

Wrote a total of 789 rows, nnz: 4000000

Wrote a total of 1046 rows, nnz: 5000000

Wrote a total of 1216 rows, nnz: 6000000

Wrote a total of 1441 rows, nnz: 7000000

...

NOTE: You may want also to checkout GraphLab's collaborative filtering library: here. GraphLab has a 100% compatible SVD solver to Mahout, with performance gains up to x50 times faster. I have created Java code to convert Mahout sequence files into Graphlab's format and back. Email me and I will send you the code.