2008-2012: ACM SIGMOD proposed for the first time to test the code of submitted papers in 2008. The first experimental repeatability and reproducibility efforts continued until 2012 and are summarized below in this web page.
Since 2015: Subsequently, starting from the accepted papers of ACM SIGMOD 2015, the ACM SIGMOD Reproducibility track started reproducing papers and presenting the results along with the awards of the "Best Artifacts" (earlier termed "Most Reproducible Papers" and "Most Comprehensively Reproduced Papers") in the subsequent conference. The reproducibility information is now available in the ACM Digital Library in the form of reproducibility badges and the award-winning papers are maintained as part of the main SIGMOD awards.
Tutorials on guidelines and lessons learned: Previous SIGMOD Reproducibility pioneers gave tutorials on best practices, including an ICDE 2008 tutorial and an EDBT 2009 tutorial by Ioana Manolescu and Stefan Manegold, as well as a SIGMOD 2012 tutorial by Juliana Freire, Philippe Bonnet, and Dennis Shasha. In June 2022, an ACM SIGMOD Record article summarized the state of affairs, future plans, and long-term goals with respect to availability and reproducibility efforts in the data management community.
Availability and Reproducibility Committees: Below, we maintain the detailed information of all the availability and reproducibility committees since 2015.
Chairs [email chairs]
Manos Athanassoulis, Boston University, USA
Holger Pirk, Imperial College London, UK
Advisory Committee
Juliana Freire, New York University, USA
Stratos Idreos, Harvard University, USA
Dennis Shasha, New York University, USA
Availability and Reproducibility Committee (ARC)
Matthew Butrovich, Carnegie Mellon University, USA
Supawit Chockchowwat, University of Illinois at Urbana-Champaign, USA
Binyang Dai, Hong Kong University of Science and Technology, Hong Kong
Kyle Deeds, University of Washington, USA
Yangshen Deng, Southern University of Science and Technology, China
Xiaojun Dong, University of California, Riverside, USA
Osnat Drien, Bar Ilan University, Israel
Yixiang Fang, The Chinese University of Hong Kong, Shenzhen, China
Saeed Fathollahzadeh, Graz University of Technology, Austria
Luca Gagliardelli, University of Modena & Reggio Emilia, Italy
Sainyam Galhotra, University of Chicago, USA
Chang Ge, University of Minnesota, USA
Immanuel Haffner, Saarland University, Saarland Informatics Campus, Germany
Xiaolin Han, University of Hong Kong, Hong Kong
Denis Hirn, Universität Tübingen, Germany
Sonia Horchidan, KTH Royal Institute of Technology, Sweden
Jiafeng Hu, Google
Tianxun Hu, Simon Fraser University, Canada
Thomas Hütter, University of Salzburg, Austria
Andy Huynh, Boston University, USA
Jalal Khalil, St. Cloud State University, USA
Aamod Khatiwada, Northeastern University, USA
Abdelouahab Khelifati, University of Fribourg, Switzerland
Daniel Kocher, University of Salzburg, Austria
Fotios Kounelis, Imperial College London, UK
Alexander Krause, TU Dresden, Germany
Huan Li, Zhejiang University, China
Keming Li, Southern University of Science and Technology, China
Xiaodong Li, The University of Hong Kong, Hong Kong
Wenqing Lin, Tencent
Chunwei Liu, Massachusetts Institute of Technology, USA
Haotian Liu, Southern University of Science and Technology, China
Siqiang Luo, Nanyang Technological University, Singapore
Chenhao Ma, The Chinese University of Hong Kong, Shenzhen, China
Pingchuan Ma, Hong Kong University of Science and Technology, Hong Kong
Kajetan Maliszewski, TU Berlin, Germany
Hunter Mccoy, University of Utah, USA
Xupeng Miao, Carnegie Mellon University, USA
Songsong Mo, Nanyang Technological University, Singapore
Madhulika Mohanty, Inria Saclay, France
Hubert Mohr-Daurat, Imperial College London, UK
Sepideh Nikookar, New Jersey Institute of Technology, USA
Reham Omar, Concordia University, Canada
Hongchao Qin, Beijing Institute of Technology, China
Aunn Raza, EPFL, Switzerland
Douglas Rumbaugh, Penn State University, USA
Majid Saeedan, University of California, Riverside, USA
Seyed Mahmoud Sajjadi Mohammadabadi, University of Nevada, Reno, USA
Sebastian Schmidl, Hasso Plattner Institute, University of Potsdam, Germany
Nima Shahbazi, University of Illinois at Chicago, USA
Zheqi Shen, University of California, Riverside, USA
Nikhil Sheoran, University of Illinois at Urbana Champaign, USA
Jonas Spenger, KTH Royal Institute of Technology, Sweden
Hongshi Tan, National University of Singapore, Singapore
Bo Tang, Southern University of Science and Technology, China
Georgios Theodorakis, Imperial College London, UK
Olamide Timothy, Tawose University of Nevada, Reno, USA
Dennis Treder, Tschechlov Universität Stuttgart, Germany
Prajna Upadhyay, INRIA Saclay, France
Guangjing Wang, Michigan State University, USA
Junchang Wang, Nanjing University of Posts and Telecommunications, China
Letong Wang, University of California, Riverside, USA
Sibo Wang, The Chinese University of Hong Kong, Hong Kong
Yunhai Wang, Shandong University, China
Zheng Wang, Nanyang Technological University, Singapore
Zhaomin Wu, National University of Singapore, Singapore
Helen Xu, Lawrence Berkeley National Laboratory, USA
Zhengxin You, Southern University of Science and Technology, China
Kaiqiang Yu, Nanyang Technological University, Singapore
Bo Zhao, Imperial College London, UK
Zichen Zhu, Boston University, USA
Web Chair
Subhadeep Sarkar, Boston University, USA
Chairs
Manos Athanassoulis, Boston University, USA
Holger Pirk, Imperial College London, UK
Advisory Committee
Juliana Freire, New York University, USA
Stratos Idreos, Harvard University, USA
Dennis Shasha, New York University, USA
Awards Committee
Dennis Shasha (chair), New York University, USA
Asterios Katsifodimos, Technische Universiteit Delft, Netherlands
Torsten Grust, Universität Tübingen, Germany
Juan Lastra-Díaz, Universidad Nacional de Educación a Distancia, Spain
Availability Committee
Lennart Behme, Technische Universität Berlin, Germany
Matthew Butrovich, Carnegie Mellon University, USA
Yixiang Fang, The Chinese University of Hong Kong, Shenzhen, Hong Kong
Luca Gagliardelli, University of Modena and Reggio Emilia, Italy
Sainyam Galhotra, University of Chicago, USA
Chang Ge, University of Minnesota, USA
Denis Hirn, Universität Tübingen, Germany
Sonia Horchidan, KTH Royal Institute of Technology, Sweden
Tianxun Hu, Simon Fraser University, Canada
Rana Hussein, University of Fribourg, Switzerland
Abdelouahab Khelifati, University of Fribourg, Switzerland
Fotios Kounelis, Imperial College London, UK
Alexander Krause, TU Dresden, Germany
Xiaodong Li, The University of Hong Kong, Hong Kong
Chunwei Liu, University of Chicago, USA
Baotong Lu, Chinese University of Hong Kong, Hong Kong
Edson Lucas Filho, Universidade Federal do Paraná, Brazil
Chenhao Ma, The University of Hong Kong, Hong Kong
Kajetan Maliszewski, TU Berlin, Germany
Songsong Mo, Nanyang Technological University, Singapore
Madhulika Mohanty, Inria Saclay, France
Hubert Mohr, Daurat Imperial College London, UK
Ju Hyoung Mun, Boston University, USA
Hamish Nicholson, EPFL, Switzerland
Sepideh Nikookar, New Jersey Institute of Technology, USA
Aunn Raza, EPFL, Switzerland
Douglas Rumbaugh, Penn State University, USA
Sebastian Schmidl, Hasso Plattner Institute & University of Potsdam, Germany
Nima Shahbazi, University of Illinois at Chicago, USA
Jonas Spenger, KTH Royal Institute of Technology, Sweden
Georgios Theodorakis, Imperial College London, UK
Prajna Upadhyay, INRIA Saclay, France
Ziyun Wei, Cornell University, USA
Brian Wheatman, Johns Hopkins University, USA
Helen Xu, Lawrence Berkeley National Laboratories, USA
Zichen Zhu, Boston University, USA
Reproducibility Committee
Matthias Boehm, Technische Universität Berlin, Germany
Dmytro Bogatov, Amazon, USA
Paris Carbone, RISE Research Institutes of Sweden, Sweden
Maria Daltayanni, University of San Francisco, USA
Niv Dayan, University of Toronto, Canada
Bailu Ding, Microsoft Research, USA
Jens Dittrich, Saarland University, Germany
Yannis Foufoulas, University of Athens, Greece
Torsten Grust, Universität Tübingen, Germany
Haridimos Kondylakis, FORTH-ICS, Greece
Alberto Lerner, University of Fribourg, Switzerland
Fatemeh Nargesian, University of Rochester, USA
Thomas Neumann, Technische Universität Munich, Germany
Milos Nikolic, University of Edinburgh, UK
Prashant Pandey, University of Utah, USA
John Paparrizos, The Ohio State University, USA
Yuxin Tang, Rice University, USA
Dimitrios Tsoumakos, National Technical University of Athens, Greece
Tianzheng Wang, Simon Fraser University, Canada
Matthias Weidlich, Humboldt-Universität zu Berlin, Germany
Dong Xie, Penn State University, USA
Dorde Zivanovic, University of Oxford, UK
Web Chair
Subhadeep Sarkar, Boston University, USA
Chairs
Manos Athanassoulis, Boston University, USA
Holger Pirk, Imperial College London, UK
Advisory Committee
Juliana Freire, New York University, USA
Stratos Idreos, Harvard University, USA
Dennis Shasha, New York University, USA
Reproducibility Committee
Ziawasch Abedjan, Leibniz Universität Hannover, Germany
Angelos Christos Anadiotis, Ecole Polytechnique & IPP, France and EPFL, Switzerland
Raja Appuswamy, Eurecom, France
Spyros Blanas, The Ohio State University, USA
Matthias Boehm, Graz University of Technology, Austria
Dmytro Bogatov, Boston University, USA
Philippe Bonnet, ITU Copenhagen, Denmark
Paris Carbone, RISE Research Institutes of Sweden, Sweden
Philippe Cudre-Mauroux, University of Fribourg, Switzerland
Maria Daltayanni, University of San Francisco, USA
Jens Dittrich, Saarland University, Germany
Herodotos Herodotou, Cyprus University of Technology, Cyprus
Yannis Ioannidis, University of Athens & Athena Research and Innovation Center, Greece
Vasiliki Kalavri, Boston University, USA
Asterios Katsifodimos, TU Delft, Netherlands
Wolfgang Lehner, TU Dresden, Germany
Milos Nikolic, University of Edinburgh, UK
John Paparrizos, University of Chicago, USA
Ilia Petrov, Reutlingen University, Germany
Kostas Stefanidis, Tampere University, Finland
Dimitrios Tsoumakos, National Technical University of Athens, Greece
Tianzheng Wang, Simon Fraser University, Canada
Yingjun Wu, Singularity Data
Dorde Zivanovic, University of Oxford, UK
Availability Committee
Sonia Horchidan, KTH Royal Institute of Technology, Sweden
Tianxun Hu, Simon Fraser University, Canada
Kaisong Huang, Simon Fraser University, Canada
Andy Huynh, Boston University, USA
Fotios Kounelis, Imperial College London, UK
Alexander Krause, TU Dresden, Germany
Chunwei Liu, University of Chicago, USA
Baotong Lu, Chinese University of Hong Kong, Hong Kong
Edson Lucas Filho, Universidade Federal do Paraná, Brazil
Madhulika Mohanty, Inria Saclay, France
Hubert Mohr-Daurat, Imperial College London, UK
Ju Hyoung Mun, Boston University, USA
Thorsten Papenbrock, Philipps University of Marburg, Germany
Aneesh Raman, Boston University, USA
Georgios Theodorakis, Imperial College London, UK
Georgia Troullinou, FORTH-ICS, Greece
Prajna Upadhyay, INRIA Saclay, France
Bo Zhao, Imperial College London, UK
Zichen Zhu, Boston University, USA
Web Chair
Subhadeep Sarkar, Boston University, USA
Chair
Manos Athanassoulis, Boston University, USAAdvisory Committee
Juliana Freire, New York University, USA
Stratos Idreos, Harvard University, USA
Dennis Shasha, New York University, USA
Reproducibility Committee
Angelos-Christos Anadiotis, Ecole Polytechnique, France
Raja Appuswamy, Eurecom, France
Joy Arulraj, Georgia Institute of Technology, USA
Dmytro Bogatov, Boston University, USA
Renata Borovica-Gajic, University of Melbourne, Australia
Shimin Chen, Institute of Computing Technology, Chinese Academy of Sciences, China
Raul Castro Fernandez, University of Chicago, USA
Thomas Heinis, Imperial College, UK
Asterios Katsifodimos, Delft University of Technology, Netherlands
Andreas Kipf, Massachusetts Institute of Technology, USA
Wolfgang Lehner, TU Dresden, Germany
John Paparrizos, University of Chicago, USA
Ilia Petrov, Reutlingen University, Germany
Mirek Riedewald, Northeastern University, USA
Yingjun Wu, Amazon, USA
Dong Xie, Penn State University, USA
Huanchen Zhang, Snowflake, USA
Kostas Zoumpatianos, Snowflake, USA
Availability Committee
Zichen Zhu, Boston University, USA
Chair: Stratos Idreos, Harvard University
Advisory Committee
Dennis Shasha, New York University, USA
Juliana Freire, New York University, USA
Committee
Subarna Chatterjee, Harvard University
Siqiang Luo, Harvard University
Sanket Purandare, Harvard University
Chair: Stratos Idreos, Harvard University
Advisory Committee
Dennis Shasha, New York University, USA
Juliana Freire, New York University, USA
Committee
Boston University, USA: Manos Athanassoulis, Subhadeep Sarkar
UMass Dartmouth, USA: David Koop
UMass Amherst, USA: Sainyam Galhotra
TU Dresden, Germany: Wolfgang Lehner, Thomas Kissinger
ETH Zurich, Switzerland: Lefteris Sidirourgos
Sungkyunkwan University, Korea: Sang Won Lee
EPF Lausanne, Switzerland: Anastasia Ailamaki, Sioulas Panagiotis
Harvard University, USA: Stratos Idreos, Wilson Qin, Abdul Wasay
Chair: Stratos Idreos, Harvard University
Advisory Committee
Dennis Shasha, New York University, USA
Juliana Freire, New York University, USA
Committee
University of Buffalo, USA: Oliver Kenedy
UMass Dartmouth, USA: David Koop
TU Dresden, Germany: Wolfgang Lehner, Thomas Kissinger
TU Dortmund, Germany: Jens Teubner
University of Glasgow, UK: Peter Triantafillou
Harvard University, USA: Stratos Idreos, Manos Athanassoulis, Michael S. Kester
EPF Lausanne, Switzerland: Anastasia Ailamaki
Oxford, UK: Dan Olteanu, Haozhe Zhang
Sungkyunkwan University, Korea: Sang Won Lee
ETH Zurich, Switzerland: Lefteris Sidirourgos
NYU Abu Dhabi, UAE: Azza Abouzied
Chair: Stratos Idreos, Harvard University
Advisory Committee
Dennis Shasha, New York University, USA
Juliana Freire, New York University, USA
Philippe Bonnet, IT University of Copenhagen, Denmark
Awards Committee
Dennis Shasha, New York University, USA
Committee
University of Buffalo, USA: Oliver Kenedy, Ying Yang, Gokhan Kul
Columbia University, USA: Ken Ross, Orestis Polychroniou
TU Dortmund, Germany: Jens Teubner, Henning Funke
Univeristy of Glasgow, UK: Peter Triantafillou, George Sfakianakis
Harvard University, USA: Stratos Idreos, Manos Athanassoulis, Michael S. Kester
HP Labs, USA: Hideaki Kimura
HP Labs, USA: Alkis Simitsis
Imperial College, UK: Thomas Ηeinis, Pooyan Jamshidi
Logicblox, USA: Ryan Johnson, Tianzheng Wang (University of Toronto)
UMass Dartmouth, USA: David Koop
National University of Singapore, Singapore: Roland Yap
New York University, USA: Juliana Freire, Fernando Seabra Chirigati, Tuan-Anh Hoang-Vu
NYU Abu Dhabi, UAE: Azza Abouzied
Ohio State University, USA: Spyros Blanas, Feilong Liu
Oxford, UK: Dan Olteanu, Milos Nikolic
University of Trento, Italy: Kostas Zoumpatianos
Chair: Stratos Idreos, Harvard University
Advisory Committee
Dennis Shasha, New York University, USA
Juliana Freire, New York University, USA
Philippe Bonnet, IT University of Copenhagen, Denmark
Committee
University of Buffalo, USA: Oliver Kenedy, Ying Yang
Columbia University, USA: Ken Ross, Orestis Polychroniou
TU Dortmund, Germany: Jens Teubner, Thomas Lindemann, Michael Kußmann
Univeristy of Glasgow, UK: Peter Triantafillou, George Sfakianakis
Harvard University, USA: Stratos Idreos, Manos Athanassoulis, Michael S. Kester
HP Labs, USA: Alkis Simitsis
Imperial College, UK: Thomas Ηeinis
INRIA, France: Ioana Manolescu, Stamatis Zampetakis
TU Munchen, Germany: Thomas Νeumann
National University of Singapore, Singapore: Roland Yap
New York University, USA: Juliana Freire, Fernando Seabra Chirigati, Tuan-Anh Hoang-Vu
NYU Abu Dhabi, UAE: Azza Abouzied
Ohio State University, USA: Spyros Blanas, Feilong Liu
Oxford, UK: Dan Olteanu, Yu Tang
UC San Diego, USA: Yannis Papakonstantinou
University of Toronto, Canada: Ryan Johnson, Tianzheng Wang
University of Trento, Italy: Kostas Zoumpatianos
The goal of establishing reproducibility is to ensure your SIGMOD 2012 research paper stands as reliable work that can be referenced by future research. The premise is that experimental papers will be most useful when their results have been tested and generalized by objective third parties.
SIGMOD 2011 offers authors an experimental repeatability and workability evaluation of their accepted papers. The repeatability & workability process tests that the experiments published in SIGMOD 2011 can be reproduced (repeatability) and possibly extended by modifying some aspects of the experiment design (workability). Authors participate on a voluntary basis, but authors benefit as well: (i) mention on the repeatability website, (ii) the ability to run their software on other sites, (iii) often, far better documentation for new members of research teams.
The repeatability and workability evaluation in conjunction with SIGMOD 2010 continues along the lines of the 2009 edition, with some improvements related to the procedure. On a voluntary basis, authors of accepted SIGMOD 2010 papers can provide their code/binaries, experimental setups and data to be tested for (i)repeatability of the experiments described in the accepted papers, and (ii) workability in the sense of running different/more experiments with different/more parameters than shown in the respective papers.
Given the quite positive experiences with and feedback about the SIGMOD 2008 repeatability initiative, SIGMOD 2009 continued the repeatability initiative in a slightly modified and extended version. A report on this effort has been published in ACM SIGMOD Record, 38(3):40-43, September 2009.
SIGMOD 2008 was the first database conference that proposed testing the code associated to conference submissions against the data sets used by the authors, to test the repeatability of the experiments presented in the submitted papers. A detailed report on this initiative has been published in ACM SIGMOD Record, 37(1):39-45, March 2008.