2008-2012: ACM SIGMOD proposed for the first time to test the code of submitted papers in 2008. The first experimental repeatability and reproducibility efforts continued until 2012 and are summarized in this web page.
Since 2015: Subsequently, starting from the accepted papers of ACM SIGMOD 2015, the ACM SIGMOD Reproducibility track started reproducing papers and presenting the results along with the awards of the "Most Reproducible Papers" in the subsequent conference. The reproducibility information is now available in the ACM Digital Library in the form of reproducibility badges and the award winning papers are maintained as part of the main SIGMOD awards.
Tutorials on guidelines and lessons learned: Previous SIGMOD Reproducibility pioneers gave tutorials on best practices including an ICDE 2008 tutorial and an EDBT 2009 tutorial by Ioana Manolescu and Stefan Manegold, as well as a SIGMOD 2012 tutorial by Juliana Freire, Philippe Bonnet, and Dennis Shasha. In June 2022, an ACM SIGMOD Record article summarized the state of affairs, future plans, and long-term goals with respect to availability and reproducibility efforts in the data management community.
Chairs
Advisory Committee
Artifact Committee
Web Chair
Chairs
Manos Athanassoulis, Boston University, USA
Holger Pirk, Imperial College London, UK
Advisory Committee
Juliana Freire, New York University, USA
Stratos Idreos, Harvard University, USA
Dennis Shasha, New York University, USA
Reproducibility Committee
Ziawasch Abedjan, Leibniz Universität Hannover, Germany
Angelos Christos Anadiotis, Ecole Polytechnique & IPP, France and EPFL, Switzerland
Raja Appuswamy, Eurecom, France
Spyros Blanas, The Ohio State University, USA
Matthias Boehm, Graz University of Technology, Austria
Dmytro Bogatov, Boston University, USA
Philippe Bonnet, ITU Copenhagen, Denmark
Paris Carbone, RISE Research Institutes of Sweden, Sweden
Philippe Cudre-Mauroux, University of Fribourg, Switzerland
Maria Daltayanni, University of San Francisco, USA
Jens Dittrich, Saarland University, Germany
Herodotos Herodotou, Cyprus University of Technology, Cyprus
Yannis Ioannidis, University of Athens & Athena Research and Innovation Center, Greece
Vasiliki Kalavri, Boston University, USA
Asterios Katsifodimos, TU Delft, Netherlands
Wolfgang Lehner, TU Dresden, Germany
Milos Nikolic, University of Edinburgh, UK
John Paparrizos, University of Chicago, USA
Ilia Petrov, Reutlingen University, Germany
Kostas Stefanidis, Tampere University, Finland
Dimitrios Tsoumakos, National Technical University of Athens, Greece
Tianzheng Wang, Simon Fraser University, Canada
Yingjun Wu, Singularity Data
Dorde Zivanovic, University of Oxford, UK
Availability Committee
Sonia Horchidan, KTH Royal Institute of Technology, Sweden
Tianxun Hu, Simon Fraser University, Canada
Kaisong Huang, Simon Fraser University, Canada
Andy Huynh, Boston University, USA
Fotios Kounelis, Imperial College London, UK
Alexander Krause, TU Dresden, Germany
Chunwei Liu, University of Chicago, USA
Baotong Lu, Chinese University of Hong Kong, Hong Kong
Edson Lucas Filho, Universidade Federal do Paraná, Brazil
Madhulika Mohanty, Inria Saclay, France
Hubert Mohr-Daurat, Imperial College London, UK
Ju Hyoung Mun, Boston University, USA
Thorsten Papenbrock, Philipps University of Marburg, Germany
Aneesh Raman, Boston University, USA
Georgios Theodorakis, Imperial College London, UK
Georgia Troullinou, FORTH-ICS, Greece
Prajna Upadhyay, INRIA Saclay, France
Bo Zhao, Imperial College London, UK
Zichen Zhu, Boston University, USA
Web Chair
Subhadeep Sarkar, Boston University, USA
Chair: Manos Athanassoulis, Boston University, USA
Advisory Committee
Juliana Freire, New York University, USA
Stratos Idreos, Harvard University, USA
Dennis Shasha, New York University, USA
Reproducibility Committee
Angelos-Christos Anadiotis, Ecole Polytechnique, France
Raja Appuswamy, Eurecom, France
Joy Arulraj, Georgia Institute of Technology, USA
Dmytro Bogatov, Boston University, USA
Renata Borovica-Gajic, University of Melbourne, Australia
Shimin Chen, Institute of Computing Technology, Chinese Academy of Sciences, China
Raul Castro Fernandez, University of Chicago, USA
Thomas Heinis, Imperial College, UK
Asterios Katsifodimos, Delft University of Technology, Netherlands
Andreas Kipf, Massachusetts Institute of Technology, USA
Wolfgang Lehner, TU Dresden, Germany
John Paparrizos, University of Chicago, USA
Ilia Petrov, Reutlingen University, Germany
Mirek Riedewald, Northeastern University, USA
Yingjun Wu, Amazon, USA
Dong Xie, Penn State University, USA
Huanchen Zhang, Snowflake, USA
Kostas Zoumpatianos, Snowflake, USA
Availability Committee
Zichen Zhu, Boston University, USA
Chair: Stratos Idreos, Harvard University
Advisory Committee
Dennis Shasha, New York University, USA
Juliana Freire, New York University, USA
Committee
Subarna Chatterjee, Harvard University
Siqiang Luo, Harvard University
Sanket Purandare, Harvard University
Chair: Stratos Idreos, Harvard University
Advisory Committee
Dennis Shasha, New York University, USA
Juliana Freire, New York University, USA
Committee
Boston University, USA: Manos Athanassoulis, Subhadeep Sarkar
UMass Dartmouth, USA: David Koop
UMass Amherst, USA: Sainyam Galhotra
TU Dresden, Germany: Wolfgang Lehner, Thomas Kissinger
ETH Zurich, Switzerland: Lefteris Sidirourgos
Sungkyunkwan University, Korea: Sang Won Lee
EPF Lausanne, Switzerland: Anastasia Ailamaki, Sioulas Panagiotis
Harvard University, USA: Stratos Idreos, Wilson Qin, Abdul Wasay
Chair: Stratos Idreos, Harvard University
Advisory Committee
Dennis Shasha, New York University, USA
Juliana Freire, New York University, USA
Committee
University of Buffalo, USA: Oliver Kenedy
UMass Dartmouth, USA: David Koop
TU Dresden, Germany: Wolfgang Lehner, Thomas Kissinger
TU Dortmund, Germany: Jens Teubner
University of Glasgow, UK: Peter Triantafillou
Harvard University, USA: Stratos Idreos, Manos Athanassoulis, Michael S. Kester
EPF Lausanne, Switzerland: Anastasia Ailamaki
Oxford, UK: Dan Olteanu, Haozhe Zhang
Sungkyunkwan University, Korea: Sang Won Lee
ETH Zurich, Switzerland: Lefteris Sidirourgos
NYU Abu Dhabi, UAE: Azza Abouzied
Chair: Stratos Idreos, Harvard University
Advisory Committee
Dennis Shasha, New York University, USA
Juliana Freire, New York University, USA
Philippe Bonnet, IT University of Copenhagen, Denmark
Awards Committee
Dennis Shasha, New York University, USA
Committee
University of Buffalo, USA: Oliver Kenedy, Ying Yang, Gokhan Kul
Columbia University, USA: Ken Ross, Orestis Polychroniou
TU Dortmund, Germany: Jens Teubner, Henning Funke
Univeristy of Glasgow, UK: Peter Triantafillou, George Sfakianakis
Harvard University, USA: Stratos Idreos, Manos Athanassoulis, Michael S. Kester
HP Labs, USA: Hideaki Kimura
HP Labs, USA: Alkis Simitsis
Imperial College, UK: Thomas Ηeinis, Pooyan Jamshidi
Logicblox, USA: Ryan Johnson, Tianzheng Wang (University of Toronto)
UMass Dartmouth, USA: David Koop
National University of Singapore, Singapore: Roland Yap
New York University, USA: Juliana Freire, Fernando Seabra Chirigati, Tuan-Anh Hoang-Vu
NYU Abu Dhabi, UAE: Azza Abouzied
Ohio State University, USA: Spyros Blanas, Feilong Liu
Oxford, UK: Dan Olteanu, Milos Nikolic
University of Trento, Italy: Kostas Zoumpatianos
Chair: Stratos Idreos, Harvard University
Advisory Committee
Dennis Shasha, New York University, USA
Juliana Freire, New York University, USA
Philippe Bonnet, IT University of Copenhagen, Denmark
Committee
University of Buffalo, USA: Oliver Kenedy, Ying Yang
Columbia University, USA: Ken Ross, Orestis Polychroniou
TU Dortmund, Germany: Jens Teubner, Thomas Lindemann, Michael Kußmann
Univeristy of Glasgow, UK: Peter Triantafillou, George Sfakianakis
Harvard University, USA: Stratos Idreos, Manos Athanassoulis, Michael S. Kester
HP Labs, USA: Alkis Simitsis
Imperial College, UK: Thomas Ηeinis
INRIA, France: Ioana Manolescu, Stamatis Zampetakis
TU Munchen, Germany: Thomas Νeumann
National University of Singapore, Singapore: Roland Yap
New York University, USA: Juliana Freire, Fernando Seabra Chirigati, Tuan-Anh Hoang-Vu
NYU Abu Dhabi, UAE: Azza Abouzied
Ohio State University, USA: Spyros Blanas, Feilong Liu
Oxford, UK: Dan Olteanu, Yu Tang
UC San Diego, USA: Yannis Papakonstantinou
University of Toronto, Canada: Ryan Johnson, Tianzheng Wang
University of Trento, Italy: Kostas Zoumpatianos
The goal of establishing reproducibility is to ensure your SIGMOD 2012 research paper stands as reliable work that can be referenced by future research. The premise is that experimental papers will be most useful when their results have been tested and generalized by objective third parties.
SIGMOD 2011 offers authors an experimental repeatability and workability evaluation of their accepted papers. The repeatability & workability process tests that the experiments published in SIGMOD 2011 can be reproduced (repeatability) and possibly extended by modifying some aspects of the experiment design (workability). Authors participate on a voluntary basis, but authors benefit as well: (i) mention on the repeatability website, (ii) the ability to run their software on other sites, (iii) often, far better documentation for new members of research teams.
The repeatability and workability evaluation in conjunction with SIGMOD 2010 continues along the lines of the 2009 edition, with some improvements related to the procedure. On a voluntary basis, authors of accepted SIGMOD 2010 papers can provide their code/binaries, experimental setups and data to be tested for (i)repeatability of the experiments described in the accepted papers, and (ii) workability in the sense of running different/more experiments with different/more parameters than shown in the respective papers.
Given the quite positive experiences with and feedback about the SIGMOD 2008 repeatability initiative, SIGMOD 2009 continued the repeatability initiative in a slightly modified and extended version. A report on this effort has been published in ACM SIGMOD Record, 38(3):40-43, September 2009.
SIGMOD 2008 was the first database conference that proposed testing the code associated to conference submissions against the data sets used by the authors, to test the repeatability of the experiments presented in the submitted papers. A detailed report on this initiative has been published in ACM SIGMOD Record, 37(1):39-45, March 2008.