I am often contacted from different researchers (both universities and companies) who are trying to benchmark and compared different machine learning frameworks. I am trying to introduce them to each other, since basically everyone is compiling the same benchmark tests, and it will be a good idea to create uniform measures and practices for comparing systems. One such example is Intel Labs report I wrote about a couple of months ago in my blog.
A few days ago I got from my collaborator Aapo Kyrola a related paper: Li, Kevin; Gibson, Charles; Ho, David; Zhou, Qi; Kim, Jason; Buhisi, Omar; Brown, Donald E.; Gerber, Matthew, "Assessment of machine learning algorithms in cloud computing frameworks", Systems and Information Engineering Design Symposium (SIEDS), 2013 IEEE, pp.98,103, 26-26 April 2013. IEEExplore
The above paper performs some comparison tests on Amazon EC2, using the same hardware, and similar algorithms and datasets. And here is the bottom line:
Hopefully the construction in the paper is detailed enough so people will be able to reproduce it.