Authors of papers published at SIGMOD 2022 (or earlier instances of the conference) who want to submit an availability or reproducibility package please reach out to the ACM SIGMOD ARC 2023 Chairs.
Quick guides for authors, availability reviewers, and reproducibility reviewers.
SIGMOD Availability & Reproducibility has three goals:
In short, the goal is to assist in building a culture where sharing results, code, and scripts of database research is the norm rather than an exception. The challenge is to do this efficiently, which means building technical expertise on how to do better research via creating repeatable and shareable research. The SIGMOD Availability & Reproducibility Committee is here to help you with this.
The SIGMOD Availability & Reproducibility effort works in coordination with the PVLDB Reproducibility to encourage the database community to develop a culture of sharing and cross-validation.
On the high-level, the workflow of the SIGMOD Availability and Reproducibility process is shown below. For more details keep scrolling to the Process and Guidelines.
You will be making it easy for other researchers to compare with your work, to adopt and extend your research. This instantly means more recognition directly visible through ACM badges for your work and higher impact.
Taking part in the SIGMOD Availability & Reproducibility process allows you to (i) host your data, scripts and code in the ACM digital library as well to make them available to a broad audience, which will award the ACM Artifacts Available and/or the ACM Artifacts Evaluated - Reusable labels, and (ii) take the ACM Results Reproduced label once your results are independently reproduced. All three badges are embedded in your pdf in the ACM Digital Library.
Successful papers will be advertised at DBworld and the list of award winners are maintained in the main SIGMOD website. In addition, the official ACM Digital Library maintains all reproduced SIGMOD papers and all SIGMOD papers with available artifacts.
The awards are selected by the Reproducibility Awards Committee, chaired by Dennis Shasha. The committee is formed after all submissions are received so that there are no conflicts. Decisions are made based on scores that reviewers assign to each paper for all factors described above.
At first, making research shareable seems like an extra overhead for authors. You just had your paper accepted in a major conference; why should you spend more time on it? The answer is to have more impact!
If you ask any experienced researcher in academia or in industry, they will tell you that they in fact already follow the reproducibility principles on a daily basis! Not as an afterthought, but as a way of doing good research.
Maintaining easily reproducible experiments, simply makes working on hard problems much easier by being able to repeat your analysis for different data sets, different hardware, different parameters, etc. Like other leading system designers, you will save significant amounts of time because you will minimize the set up and tuning effort for your experiments. In addition, such practices will help bring new students up to speed after a project has lain dormant for a few months.
Ideally availability of artifacts reproducibility should be close to zero effort.
Each submission should contain: (1) A prototype system provided as a white box (source, configuration files, build environment) or a black-box system fully specified. (2) Input Data: Either the process to generate the input data should be made available, or when the data is not generated, the actual data itself or a link to the data should be provided. (3) The set of experiments (system configuration and initialization, scripts, workload, measurement protocol) used to produce the raw experimental data. (4) For full reproducibility submissions, the scripts needed to transform the raw data into the graphs included in the paper. By providing the artifacts you are awarded the "Artifacts Availabile" badge. By providing artifacts that are of a quality that significantly exceeds minimal functionality and are clearly documented, well-structured, and facilitate reuse you are awarded the "Artifacts Evaluated - Reusable" badge.
The central results and claims of the paper should be supported by the submitted experiments, meaning we can recreate result data and graphs that demonstrate similar behavior with that shown in the paper. Typically when the results are about response times, the exact numbers will depend on the underlying hardware. We do not expect to get identical results with the paper unless it happens that we get access to identical hardware. Instead, what we expect to see is that the overall behavior matches the conclusions drawn in the paper, e.g., that a given algorithm is significantly faster than another one, or that a given parameter affects negatively or positively the behavior of a system. By having all core results reproduced by an independent reproducibility reviewer you are awarded the "Results Reproduced" badge.
The artifacts of each paper are reviewed by one artifacts reviewer to ensure the quality of the submission. The role of the artifact availability reviewer is to compile, deploy, and test the artifacts and communicate directly with the authors for any issues face during this process. The ultimate goal is to resolve any issues so that all artifacts are made publicly available.
After the initial availability review phase, each paper is reviewed by one database group. The process happens in communication with the reviewers so that authors and reviewers can iron out any lingering technical issues. The end result is a short report which describes the result of the process. For successful papers the report is maintained in the Reproducibility Reports page.
Every case is slightly different. Sometimes the Availability & Reproducibility committee can simply rerun software (e.g., rerun some existing benchmark). At other times, obtaining raw data may require special hardware (e.g., sensors in the arctic). In the latter case, the committee will not be able to reproduce the acquisition of raw data, but then you can provide the committee with a protocol, including detailed procedures for system set-up, experiment set-up, and measurements.
Whenever raw data acquisition can be produced, the following information should be provided.
Authors should explicitly specify the OS and tools that should be installed as the environment. Such specification should include dependencies with specific hardware features (e.g., 25 GB of RAM are needed) or dependencies within the environment (e.g., the compiler that should be used must be run with a specific version of the OS).
System setup is one of the most challenging aspects when repeating experiments. System setup will be easier to conduct if it is automatic rather than manual. Authors should test that the system they distribute can actually be installed in a new environment. The documentation should detail every step in system setup:
The above tasks should be achieved by executing a set o scripts provided by the authors that will download needed components (systems, libraries), initialize the environment, check that software and hardware is compatible, and deploy the system.
The committee strongly suggests using one of the following tools to streamline the process of reproducibility. These tools can be used to capture the environment, the input files, the expected output files, and the required libraries in a container-like suite. This will help both the authors and the evaluators to seamlessly rerun experiments under specific environments and settings. If using all these tools proves to be difficult for a particular paper, the committee will work with the authors to find the proper solution based on the specifics of the paper and the environment needed. Below is a list of the tools recommended by the SIGMOD Reproducibility Committee.
Given a system, the authors should provide the complete set of experiments to reproduce the paper's results. Typically, each experiment will consist of the following parts.
The authors should document (i) how to perform the setup, running and clean-up phases, and (ii) how to check that these phases complete as they should. The authors should document the expected effect of the setup phase (e.g., a cold file cache is enforced) and the different steps of the running phase, e.g., by documenting the combination of command line options used to run a given experiment script.
Experiments should be automatic, e.g., via a script that takes a range of values for each experiment parameter as arguments, rather than manual, e.g., via a script that must be edited so that a constant takes the value of a given experiment parameter.
For each graph in the paper, the authors should describe how the graph is obtained from the experimental measurements. The submission should contain the scripts (or spreadsheets) that are used to generate the graphs. We strongly encourage authors to provide scripts for all their graphs. The authors are free to choose from their favorite plotting tool using a tool such as Gnuplot, Matlab, Matplotlib, R, or Octave.
At a minimum the authors should provide a complete set of scripts to install the system, produce the data, run experiments and produce the resulting graphs along with a detailed Readme file that describes the process step by step so it can be easily reproduced by a reviewer.
The ideal reproducibility submission consists of a master script that:
... to produce a new PDF for the paper that contains the new graphs. It is possible!
Chairs [email chairs]
Manos Athanassoulis, Boston University, USA
Holger Pirk, Imperial College London, UK
Advisory Committee
Juliana Freire, New York University, USA
Stratos Idreos, Harvard University, USA
Dennis Shasha, New York University, USA
Availability Committee
Lennart Behme, Technische Universität Berlin, Germany
Matthew Butrovich, Carnegie Mellon University, USA
Yixiang Fang, The Chinese University of Hong Kong, Shenzhen, Hong Kong
Luca Gagliardelli, University of Modena and Reggio Emilia, Italy
Sainyam Galhotra, University of Chicago, USA
Chang Ge, University of Minnesota, USA
Denis Hirn, Universität Tübingen, Germany
Sonia Horchidan, KTH Royal Institute of Technology, Sweden
Tianxun Hu, Simon Fraser University, Canada
Rana Hussein, University of Fribourg, Switzerland
Abdelouahab Khelifati, University of Fribourg, Switzerland
Fotios Kounelis, Imperial College London, UK
Alexander Krause, TU Dresden, Germany
Xiaodong Li, The University of Hong Kong, Hong Kong
Chunwei Liu, University of Chicago, USA
Baotong Lu, Chinese University of Hong Kong, Hong Kong
Edson Lucas Filho, Universidade Federal do Paraná, Brazil
Chenhao Ma, The University of Hong Kong, Hong Kong
Kajetan Maliszewski, TU Berlin, Germany
Songsong Mo, Nanyang Technological University, Singapore
Madhulika Mohanty, Inria Saclay, France
Hubert Mohr, Daurat Imperial College London, UK
Ju Hyoung Mun, Boston University, USA
Hamish Nicholson, EPFL, Switzerland
Sepideh Nikookar, New Jersey Institute of Technology, USA
Aunn Raza, EPFL, Switzerland
Douglas Rumbaugh, Penn State University, USA
Sebastian Schmidl, Hasso Plattner Institute & University of Potsdam, Germany
Nima Shahbazi, University of Illinois at Chicago, USA
Jonas Spenger, KTH Royal Institute of Technology, Sweden
Georgios Theodorakis, Imperial College London, UK
Prajna Upadhyay, INRIA Saclay, France
Ziyun Wei, Cornell University, USA
Brian Wheatman, Johns Hopkins University, USA
Helen Xu, Lawrence Berkeley National Laboratories, USA
Zichen Zhu, Boston University, USA
Reproducibility Committee
Matthias Boehm, Technische Universität Berlin, Germany
Dmytro Bogatov, Amazon, USA
Paris Carbone, RISE Research Institutes of Sweden, Sweden
Maria Daltayanni, University of San Francisco, USA
Niv Dayan, University of Toronto, Canada
Bailu Ding, Microsoft Research, USA
Jens Dittrich, Saarland University, Germany
Yannis Foufoulas, University of Athens, Greece
Torsten Grust, Universität Tübingen, Germany
Haridimos Kondylakis, FORTH-ICS, Greece
Alberto Lerner, University of Fribourg, Switzerland
Fatemeh Nargesian, University of Rochester, USA
Thomas Neumann, Technische Universität Munich, Germany
Milos Nikolic, University of Edinburgh, UK
Prashant Pandey, University of Utah, USA
John Paparrizos, The Ohio State University, USA
Yuxin Tang, Rice University, USA
Dimitrios Tsoumakos, National Technical University of Athens, Greece
Tianzheng Wang, Simon Fraser University, Canada
Matthias Weidlich, Humboldt-Universität zu Berlin, Germany
Dong Xie, Penn State University, USA
Dorde Zivanovic, University of Oxford, UK
Web Chair
Subhadeep Sarkar, Boston University, USA