Best Paper Award at KDD 2016 awarded to Matroid

Leading a group of industry and academic researchers, a KDD 2016 Best Paper Award (runner-up) was awarded to Matroid, for a paper describing the matrix computations and optimization primitives inside Apache Spark. The collaboration involved many leading companies and universities: Databricks, HP Labs, Twitter, UC Berkeley, MIT, and Stanford. As a thank you to the Apache Software Foundation for their sheperding the Apache Spark project, Matroid donated the prize money from this award to the Apache Software Foundation.

The paper describes the matrix computations available in the cluster programming framework, Apache Spark. Out of the box, Spark provides abstractions and implementations for distributed matrices and optimization routines using these matrices. When translating single-node algorithms to run on a distributed cluster, we observe that often a simple idea is enough: separating matrix operations from vector operations and shipping the matrix operations to be ran on the cluster, while keeping vector operations local to the driver.

In the case of the Singular Value Decomposition, by taking this idea to an extreme, we are able to exploit the computational power of a cluster, while running code written decades ago for a single core. Another example is our Spark port of the popular TFOCS optimization package, originally built for MATLAB, which allows for solving Linear programs as well as a variety of other convex programs. We conclude with a comprehensive set of benchmarks for hardware accelerated matrix computations from the JVM, which is interesting in its own right, as many cluster programming frameworks use the JVM. The contributions described in this paper are already merged into Apache Spark and available on Spark installations by default, and commercially supported by a slew of companies which provide further services.

To hear more about this, attend Matroid’s company conference: The Scaled Machine Learning Conference, or see our KDD talk Wednesday at 10am during the plenary session.