circle-symbol
  • Home
  • Awards
  • Process
  • Guidelines
  • Committee
  • History
  • Reports
  • Reproduced Papers

Past Reproducibility Efforts

2008-2012: ACM SIGMOD proposed for the first time to test the code of submitted papers in 2008. The first experimental repeatability and reproducibility efforts continued until 2012 and are summarized in this web page.

Since 2015: Subsequently, starting from the accepted papers of ACM SIGMOD 2015, the ACM SIGMOD Reproducibility track started reproducing papers and presenting the results along with the awards of the "Most Reproducible Papers" in the subsequent conference. The reproducibility information is now available in the ACM Digital Library in the form of reproducibility badges and the award winning papers are maintained as part of the main SIGMOD awards.

Tutorials on guidelines and lessons learned: Previous SIGMOD Reproducibility pioneers gave tutorials on best practices including an ICDE 2008 tutorial and an EDBT 2009 tutorial by Ioana Manolescu and Stefan Manegold, as well as a SIGMOD 2012 tutorial by Juliana Freire, Philippe Bonnet, and Dennis Shasha.


SIGMOD 2022 Availability & Reproducibility Committee

Reproducing papers presented at ACM SIGMOD 2021

Chairs

Manos Athanassoulis, Boston University, USA

Holger Pirk, Imperial College London, UK

Advisory Committee

Juliana Freire, New York University, USA

Stratos Idreos, Harvard University, USA

Dennis Shasha, New York University, USA

Reproducibility Committee

Ziawasch Abedjan, Leibniz Universität Hannover, Germany

Angelos Christos Anadiotis, Ecole Polytechnique & IPP, France and EPFL, Switzerland

Raja Appuswamy, Eurecom, France

Spyros Blanas, The Ohio State University, USA

Matthias Boehm, Graz University of Technology, Austria

Dmytro Bogatov, Boston University, USA

Philippe Bonnet, ITU Copenhagen, Denmark

Paris Carbone, RISE Research Institutes of Sweden, Sweden

Philippe Cudre-Mauroux, University of Fribourg, Switzerland

Maria Daltayanni, University of San Francisco, USA

Jens Dittrich, Saarland University, Germany

Herodotos Herodotou, Cyprus University of Technology, Cyprus

Yannis Ioannidis, University of Athens & Athena Research and Innovation Center, Greece

Vasiliki Kalavri, Boston University, USA

Asterios Katsifodimos, TU Delft, Netherlands

Wolfgang Lehner, TU Dresden, Germany

Milos Nikolic, University of Edinburgh, UK

John Paparrizos, University of Chicago, USA

Ilia Petrov, Reutlingen University, Germany

Kostas Stefanidis, Tampere University, Finland

Dimitrios Tsoumakos, National Technical University of Athens, Greece

Tianzheng Wang, Simon Fraser University, Canada

Yingjun Wu, Singularity Data

Dorde Zivanovic, University of Oxford, UK

Availability Committee

Sonia Horchidan, KTH Royal Institute of Technology, Sweden

Tianxun Hu, Simon Fraser University, Canada

Kaisong Huang, Simon Fraser University, Canada

Andy Huynh, Boston University, USA

Fotios Kounelis, Imperial College London, UK

Alexander Krause, TU Dresden, Germany

Chunwei Liu, University of Chicago, USA

Baotong Lu, Chinese University of Hong Kong, Hong Kong

Edson Lucas Filho, Universidade Federal do Paraná, Brazil

Madhulika Mohanty, Inria Saclay, France

Hubert Mohr-Daurat, Imperial College London, UK

Ju Hyoung Mun, Boston University, USA

Thorsten Papenbrock, Philipps University of Marburg, Germany

Aneesh Raman, Boston University, USA

Georgios Theodorakis, Imperial College London, UK

Georgia Troullinou, FORTH-ICS, Greece

Prajna Upadhyay, INRIA Saclay, France

Bo Zhao, Imperial College London, UK

Zichen Zhu, Boston University, USA

Web Chair

Subhadeep Sarkar, Boston University, USA


SIGMOD 2021 Availability & Reproducibility Committee

Reproducing papers presented at ACM SIGMOD 2020

Chair: Manos Athanassoulis, Boston University, USA

Advisory Committee

Juliana Freire, New York University, USA

Stratos Idreos, Harvard University, USA

Dennis Shasha, New York University, USA

Reproducibility Committee

Angelos-Christos Anadiotis, Ecole Polytechnique, France

Raja Appuswamy, Eurecom, France

Joy Arulraj, Georgia Institute of Technology, USA

Dmytro Bogatov, Boston University, USA

Renata Borovica-Gajic, University of Melbourne, Australia

Shimin Chen, Institute of Computing Technology, Chinese Academy of Sciences, China

Raul Castro Fernandez, University of Chicago, USA

Thomas Heinis, Imperial College, UK

Asterios Katsifodimos, Delft University of Technology, Netherlands

Andreas Kipf, Massachusetts Institute of Technology, USA

Wolfgang Lehner, TU Dresden, Germany

John Paparrizos, University of Chicago, USA

Ilia Petrov, Reutlingen University, Germany

Mirek Riedewald, Northeastern University, USA

Yingjun Wu, Amazon, USA

Dong Xie, Penn State University, USA

Huanchen Zhang, Snowflake, USA

Kostas Zoumpatianos, Snowflake, USA

Availability Committee

Zichen Zhu, Boston University, USA


SIGMOD 2020 Reproducibility Committee

Reproducing papers presented at ACM SIGMOD 2019

Chair: Stratos Idreos, Harvard University

Advisory Committee

Dennis Shasha, New York University, USA

Juliana Freire, New York University, USA

Committee

Subarna Chatterjee, Harvard University

Siqiang Luo, Harvard University

Sanket Purandare, Harvard University


SIGMOD 2019 Reproducibility Committee

Reproducing papers presented at ACM SIGMOD 2018

Chair: Stratos Idreos, Harvard University

Advisory Committee

Dennis Shasha, New York University, USA

Juliana Freire, New York University, USA

Committee

Boston University, USA: Manos Athanassoulis, Subhadeep Sarkar

UMass Dartmouth, USA: David Koop

UMass Amherst, USA: Sainyam Galhotra

TU Dresden, Germany: Wolfgang Lehner, Thomas Kissinger

ETH Zurich, Switzerland: Lefteris Sidirourgos

Sungkyunkwan University, Korea: Sang Won Lee

EPF Lausanne, Switzerland: Anastasia Ailamaki, Sioulas Panagiotis

Harvard University, USA: Stratos Idreos, Wilson Qin, Abdul Wasay


SIGMOD 2018 Reproducibility Committee

Reproducing papers presented at ACM SIGMOD 2017

Chair: Stratos Idreos, Harvard University

Advisory Committee

Dennis Shasha, New York University, USA

Juliana Freire, New York University, USA

Committee

University of Buffalo, USA: Oliver Kenedy

UMass Dartmouth, USA: David Koop

TU Dresden, Germany: Wolfgang Lehner, Thomas Kissinger

TU Dortmund, Germany: Jens Teubner

University of Glasgow, UK: Peter Triantafillou

Harvard University, USA: Stratos Idreos, Manos Athanassoulis, Michael S. Kester

EPF Lausanne, Switzerland: Anastasia Ailamaki

Oxford, UK: Dan Olteanu, Haozhe Zhang

Sungkyunkwan University, Korea: Sang Won Lee

ETH Zurich, Switzerland: Lefteris Sidirourgos

NYU Abu Dhabi, UAE: Azza Abouzied


SIGMOD 2017 Reproducibility Committee

Reproducing papers presented at ACM SIGMOD 2016

Chair: Stratos Idreos, Harvard University

Advisory Committee

Dennis Shasha, New York University, USA

Juliana Freire, New York University, USA

Philippe Bonnet, IT University of Copenhagen, Denmark

Awards Committee

Dennis Shasha, New York University, USA

Committee

University of Buffalo, USA: Oliver Kenedy, Ying Yang, Gokhan Kul

Columbia University, USA: Ken Ross, Orestis Polychroniou

TU Dortmund, Germany: Jens Teubner, Henning Funke

Univeristy of Glasgow, UK: Peter Triantafillou, George Sfakianakis

Harvard University, USA: Stratos Idreos, Manos Athanassoulis, Michael S. Kester

HP Labs, USA: Hideaki Kimura

HP Labs, USA: Alkis Simitsis

Imperial College, UK: Thomas Ηeinis, Pooyan Jamshidi

Logicblox, USA: Ryan Johnson, Tianzheng Wang (University of Toronto)

UMass Dartmouth, USA: David Koop

National University of Singapore, Singapore: Roland Yap

New York University, USA: Juliana Freire, Fernando Seabra Chirigati, Tuan-Anh Hoang-Vu

NYU Abu Dhabi, UAE: Azza Abouzied

Ohio State University, USA: Spyros Blanas, Feilong Liu

Oxford, UK: Dan Olteanu, Milos Nikolic

University of Trento, Italy: Kostas Zoumpatianos


SIGMOD 2016 Reproducibility Committee

Reproducing papers presented at ACM SIGMOD 2015

Chair: Stratos Idreos, Harvard University

Advisory Committee

Dennis Shasha, New York University, USA

Juliana Freire, New York University, USA

Philippe Bonnet, IT University of Copenhagen, Denmark

Committee

University of Buffalo, USA: Oliver Kenedy, Ying Yang

Columbia University, USA: Ken Ross, Orestis Polychroniou

TU Dortmund, Germany: Jens Teubner, Thomas Lindemann, Michael Kußmann

Univeristy of Glasgow, UK: Peter Triantafillou, George Sfakianakis

Harvard University, USA: Stratos Idreos, Manos Athanassoulis, Michael S. Kester

HP Labs, USA: Alkis Simitsis

Imperial College, UK: Thomas Ηeinis

INRIA, France: Ioana Manolescu, Stamatis Zampetakis

TU Munchen, Germany: Thomas Νeumann

National University of Singapore, Singapore: Roland Yap

New York University, USA: Juliana Freire, Fernando Seabra Chirigati, Tuan-Anh Hoang-Vu

NYU Abu Dhabi, UAE: Azza Abouzied

Ohio State University, USA: Spyros Blanas, Feilong Liu

Oxford, UK: Dan Olteanu, Yu Tang

UC San Diego, USA: Yannis Papakonstantinou

University of Toronto, Canada: Ryan Johnson, Tianzheng Wang

University of Trento, Italy: Kostas Zoumpatianos

ACM SIGMOD 2012 Reproducibility

The goal of establishing reproducibility is to ensure your SIGMOD 2012 research paper stands as reliable work that can be referenced by future research. The premise is that experimental papers will be most useful when their results have been tested and generalized by objective third parties.

Click for more details.

ACM SIGMOD 2011 Reproducibility

SIGMOD 2011 offers authors an experimental repeatability and workability evaluation of their accepted papers. The repeatability & workability process tests that the experiments published in SIGMOD 2011 can be reproduced (repeatability) and possibly extended by modifying some aspects of the experiment design (workability). Authors participate on a voluntary basis, but authors benefit as well: (i) mention on the repeatability website, (ii) the ability to run their software on other sites, (iii) often, far better documentation for new members of research teams.

Click for more details.

ACM SIGMOD 2010 Reproducibility

The repeatability and workability evaluation in conjunction with SIGMOD 2010 continues along the lines of the 2009 edition, with some improvements related to the procedure. On a voluntary basis, authors of accepted SIGMOD 2010 papers can provide their code/binaries, experimental setups and data to be tested for (i)repeatability of the experiments described in the accepted papers, and (ii) workability in the sense of running different/more experiments with different/more parameters than shown in the respective papers.

Click for more details.

ACM SIGMOD 2009 Reproducibility

Given the quite positive experiences with and feedback about the SIGMOD 2008 repeatability initiative, SIGMOD 2009 continued the repeatability initiative in a slightly modified and extended version. A report on this effort has been published in ACM SIGMOD Record, 38(3):40-43, September 2009.

Click for more details.

ACM SIGMOD 2008 Reproducibility

SIGMOD 2008 was the first database conference that proposed testing the code associated to conference submissions against the data sets used by the authors, to test the repeatability of the experiments presented in the submitted papers. A detailed report on this initiative has been published in ACM SIGMOD Record, 37(1):39-45, March 2008.

Click for more details.





Copyright © ACM SIGMOD. All Rights Reserved.