ACM SIGMOD proposed for the first time to test the code of submitted papers in 2008. The first experimental repeatability and reproducibility efforts continued until 2012 and are summarized in this web page.
Subsequently, starting from the accepted papers of ACM SIGMOD 2015, the ACM SIGMOD Reproducibility track started reproducing papers and presenting the results along with the awards of the "Most Reproducible Papers" in the subsequent conference. The reproducibility information is now available in the ACM Digital Library in the form of reproducibility badges and the award winning papers are maintained as part of the main SIGMOD awards.
Chair: Stratos Idreos, Harvard University
Advisory Committee
Dennis Shasha, New York University, USA
Juliana Freire, New York University, USA
Committee
Subarna Chatterjee, Harvard University
Siqiang Luo, Harvard University
Sanket Purandare, Harvard University
Chair: Stratos Idreos, Harvard University
Advisory Committee
Dennis Shasha, New York University, USA
Juliana Freire, New York University, USA
Committee
Boston University, USA: Manos Athanassoulis, Subhadeep Sarkar
UMass Dartmouth, USA: David Koop
UMass Amherst, USA: Sainyam Galhotra
TU Dresden, Germany: Wolfgang Lehner, Thomas Kissinger
ETH Zurich, Switzerland: Lefteris Sidirourgos
Sungkyunkwan University, Korea: Sang Won Lee
EPF Lausanne, Switzerland: Anastasia Ailamaki, Sioulas Panagiotis
Harvard University, USA: Stratos Idreos, Wilson Qin, Abdul Wasay
Chair: Stratos Idreos, Harvard University
Advisory Committee
Dennis Shasha, New York University, USA
Juliana Freire, New York University, USA
Committee
University of Buffalo, USA: Oliver Kenedy
UMass Dartmouth, USA: David Koop
TU Dresden, Germany: Wolfgang Lehner, Thomas Kissinger
TU Dortmund, Germany: Jens Teubner
University of Glasgow, UK: Peter Triantafillou
Harvard University, USA: Stratos Idreos, Manos Athanassoulis, Michael S. Kester
EPF Lausanne, Switzerland: Anastasia Ailamaki
Oxford, UK: Dan Olteanu, Haozhe Zhang
Sungkyunkwan University, Korea: Sang Won Lee
ETH Zurich, Switzerland: Lefteris Sidirourgos
NYU Abu Dhabi, UAE: Azza Abouzied
Chair: Stratos Idreos, Harvard University
Advisory Committee
Dennis Shasha, New York University, USA
Juliana Freire, New York University, USA
Philippe Bonnet, IT University of Copenhagen, Denmark
Awards Committee
Dennis Shasha, New York University, USA
Committee
University of Buffalo, USA: Oliver Kenedy, Ying Yang, Gokhan Kul
Columbia University, USA: Ken Ross, Orestis Polychroniou
TU Dortmund, Germany: Jens Teubner, Henning Funke
Univeristy of Glasgow, UK: Peter Triantafillou, George Sfakianakis
Harvard University, USA: Stratos Idreos, Manos Athanassoulis, Michael S. Kester
HP Labs, USA: Hideaki Kimura
HP Labs, USA: Alkis Simitsis
Imperial College, UK: Thomas Ηeinis, Pooyan Jamshidi
Logicblox, USA: Ryan Johnson, Tianzheng Wang (University of Toronto)
UMass Dartmouth, USA: David Koop
National University of Singapore, Singapore: Roland Yap
New York University, USA: Juliana Freire, Fernando Seabra Chirigati, Tuan-Anh Hoang-Vu
NYU Abu Dhabi, UAE: Azza Abouzied
Ohio State University, USA: Spyros Blanas, Feilong Liu
Oxford, UK: Dan Olteanu, Milos Nikolic
University of Trento, Italy: Kostas Zoumpatianos
Chair: Stratos Idreos, Harvard University
Advisory Committee
Dennis Shasha, New York University, USA
Juliana Freire, New York University, USA
Philippe Bonnet, IT University of Copenhagen, Denmark
Committee
University of Buffalo, USA: Oliver Kenedy, Ying Yang
Columbia University, USA: Ken Ross, Orestis Polychroniou
TU Dortmund, Germany: Jens Teubner, Thomas Lindemann, Michael Kußmann
Univeristy of Glasgow, UK: Peter Triantafillou, George Sfakianakis
Harvard University, USA: Stratos Idreos, Manos Athanassoulis, Michael S. Kester
HP Labs, USA: Alkis Simitsis
Imperial College, UK: Thomas Ηeinis
INRIA, France: Ioana Manolescu, Stamatis Zampetakis
TU Munchen, Germany: Thomas Νeumann
National University of Singapore, Singapore: Roland Yap
New York University, USA: Juliana Freire, Fernando Seabra Chirigati, Tuan-Anh Hoang-Vu
NYU Abu Dhabi, UAE: Azza Abouzied
Ohio State University, USA: Spyros Blanas, Feilong Liu
Oxford, UK: Dan Olteanu, Yu Tang
UC San Diego, USA: Yannis Papakonstantinou
University of Toronto, Canada: Ryan Johnson, Tianzheng Wang
University of Trento, Italy: Kostas Zoumpatianos
The goal of establishing reproducibility is to ensure your SIGMOD 2012 research paper stands as reliable work that can be referenced by future research. The premise is that experimental papers will be most useful when their results have been tested and generalized by objective third parties.
SIGMOD 2011 offers authors an experimental repeatability and workability evaluation of their accepted papers. The repeatability & workability process tests that the experiments published in SIGMOD 2011 can be reproduced (repeatability) and possibly extended by modifying some aspects of the experiment design (workability). Authors participate on a voluntary basis, but authors benefit as well: (i) mention on the repeatability website, (ii) the ability to run their software on other sites, (iii) often, far better documentation for new members of research teams.
The repeatability and workability evaluation in conjunction with SIGMOD 2010 continues along the lines of the 2009 edition, with some improvements related to the procedure. On a voluntary basis, authors of accepted SIGMOD 2010 papers can provide their code/binaries, experimental setups and data to be tested for (i)repeatability of the experiments described in the accepted papers, and (ii) workability in the sense of running different/more experiments with different/more parameters than shown in the respective papers.
Given the quite positive experiences with and feedback about the SIGMOD 2008 repeatability initiative, SIGMOD 2009 continued the repeatability initiative in a slightly modified and extended version. A report on this effort has been published in ACM SIGMOD Record, 38(3):40-43, September 2009.
SIGMOD 2008 was the first database conference that proposed testing the code associated to conference submissions against the data sets used by the authors, to test the repeatability of the experiments presented in the submitted papers. A detailed report on this initiative has been published in ACM SIGMOD Record, 37(1):39-45, March 2008.