A scientific paper consists of a constellation of artifacts beyond the document itself: software, data sets, scripts, hardware, evaluation data and documentation, raw survey results, mechanized test suites, benchmarks, and so on. Often, the quality of these artifacts is as important as that of the document itself. Based on the growing success of the Availability & Reproducibility Initiative (ARI) of previous SIGMOD conferences, we will run again this year an optional artifact evaluation process. All papers presented at SIGMOD 2024 -- accepted for publication PACMMOD Vol. 1(3), Vol.1(4), Vol.2(1), and Vol.2(3) -- are encouraged to participate in the artifact evaluation process, as well as papers from the SIGMOD 2024 Industry Track.
The submission process should follow these guidelines so that an artifact associated with a paper is considered for its availability and functionality, along with the reproducibility of the paper’s key results and claims. Please see this quick guide that summarizes the key requirements and guidelines of your submission.
The artifact evaluation has two phases: a single-anonymous phase of reviewing the overall quality of the artifact and a zero-anonymous phase of reproducing the results, during which reviewers are invited to collaborate with authors. At the end of the process, for every successfully reproduced paper the reviewers (and optionally all or some of the authors) are co-authoring a reproducibility report to document the process, the core reproduced results, and any success stories, i.e., cases that during the reproducibility review the artifacts quality was improved.
All accepted SIGMOD research and industry papers are encouraged to participate in artifact evaluation.
Submitting the artifacts associated with your accepted SIGMOD paper is a two-step process.
The ARC recommends that you create a single web page at a stable URL that contains your artifact package. The ARC may contact you with questions about your artifacts as needed.
We are looking for members of the Availability and Reproducibility Committee (ARC), who will contribute to the SIGMOD 2024 Availability and Reproducibility review process by evaluating submitted artifacts. ARC membership is especially suitable for researchers early in their career, such as PhD students. Even as a first-year PhD student, you are welcome to join the ARC, provided you are working in a topic area covered by SIGMOD (broadly data management). You can be located anywhere in the world as all committee discussions will happen online.
As an ARC member, you will not only help promote the reproducibility of experimental results in systems research, but also get to familiarize yourself with research papers just accepted for publication at SIGMOD 2024 and explore their artifacts. For a given artifact, you may be asked to evaluate its public availability, functionality, and/or ability to reproduce the results from the paper. You will be able to discuss with other ARC members and interact with the authors as necessary, for instance if you are unable to get the artifact to work as expected. Finally, you will provide a review for the artifact to give constructive feedback to its authors, discuss the artifact with fellow reviewers, and help award the paper artifact evaluation badges. For all successfully reproduced artifact, you will co-author a reproducibility report with your co-reviewers and optionally the authors of the paper to document the process, the core reproduced results, and any success stories, i.e., cases that during the reproducibility review the artifacts quality was improved.
We expect that each member will evaluate 2-3 artifacts. The duration of evaluating different artifacts may vary depending on its computational cost (to be checked during the "Sanity-Check" period). ARC members are expected to allocate time to choose the artifacts they want to review, to read the chosen papers, to evaluate and review the corresponding artifacts, and to be available for online discussion until artifact notification deadline. Please ensure that you have sufficient time and availability for the ARC during the evaluation period September 10 to November 10 2024. Please also ensure you will be able to carry out the evaluation independently, without sharing artifacts or related information with others and limiting all the discussions to within the ARC.We expect that evaluations can be done on your own computer (any moderately recent desktop or laptop computer will do). In other cases and to the extent possible, authors will arrange their artifacts so as to run in community research testbeds or will provide remote access to their systems (e.g., via SSH). Please also see this quick guide for reviewers.
If you are interested in taking part in the ARC, please complete this online self-nomination form.
Deadline: August 24, 2024, Anywhere on Earth
You can contact the chairs for any questions.
Chairs [email chairs]
Manos Athanassoulis, Boston University, USA
Holger Pirk, Imperial College London, UK
Natacha Crooks, UC Berkeley, USA
Advisory Committee
Juliana Freire, New York University, USA
Stratos Idreos, Harvard University, USA
Dennis Shasha, New York University, USA
Availability and Reproducibility Committee (ARC)
Akshata Kishore Moharir, Microsoft
Aleksander Vedvik, Independent researcher
Alexander Krause, TU Dresden
Alexandros Zeakis, Athena Research Center
Amedeo Pachera, Lyon 1 University
Anwesha Saha, Boston University
Ayushi Singh, Netflix
Bogdan Bindea, Technical University of Cluj-Napoca
Carlos Enrique Muñiz Cuza, Technische Universität Berlin
Chenhao Ma, The Chinese University of Hong Kong, Shenzhen
Chuqing Gao, Purdue University
Daniel Kocher, University of Salzburg
David Chu, UC Berkeley
Denis Hirn, Universität Tübingen
Dohyun Park, University of Illinois Urbana-Champaign
Donghyun Sohn, Northwestern University
Douglas Rumbaugh, Penn State University
Gabriele Oliaro, Carnegie Mellon University
Gansen Hu, Shanghai Jiaotong University
George Katsogiannis, Université Grenoble Alpes & Athena Research Center
Georgiy Lebedev, EPFL
Giorgio Vinciguerra, Università di Pisa
Guang Yang, Imperial College London
Guangjing Wang, University of South Florida
Guozhang Sun, Northeastern University
Haralampos Gavriilidis, Technische Universität Berlin
Hein Meling, University of Stavanger
Hengfeng Wei, Nanjing University
Hongshi Tan, National University of Singapore
Huan Li, Zhejiang University
Hubert Mohr-Daurat, Imperial College London
Jiani Yang, Student of Zhejiang University
Jingzhi Fang, The Hong Kong University of Science and Technology
Junchang Wang, Nanjing University of Posts and Telecommunications
Kaiqiang Yu, Nanyang Technological University
Konstantinos Kanellis, University of Wisconsin-Madison
Kyle Deeds, University of Washington
Kyoseung Koo, Seoul National University
Long Gu, TU-Darmstadt
Luca Zecchini, University of Modena and Reggio Emilia
Lukas Schwerdtfeger, Bifold / DIMA at TU-Berlin
Max Norfolk, Penn State University
Maximilian Hüttner, TU Darmstadt
Mingfei Cheng, Singapore Management University
Minguk Choi, Dankook University
Mo Sha, Alibaba Cloud
Muhammad Farhan, Australian National University
Niccolo Meneghetti, University of Michigan-Dearborn
Nick Glaze, Microsoft
Olga Ovcharenko, ETH Zürich
Prajna Upadhyay, BITS Pilani Hyderabad
Qiaolin Yu, Cornell University
Rohan Puri, Samsung
Saeed Fathollahzadeh, Concordia University
Sami Hadouaj, University of Michigan- Dearborn
Sebastian Baunsgaard, Technische Universität Berlin
Sheng Yao, The Hong Kong University of Science and Technology
Shubham Vashisth, PhD Candidate at McGill University, Montreal, Canada
Supawit Chockchowwat, University of Illinois Urbana-Champaign
Suyang Zhong, National University of Singapore
Sven Helmer, University of Zurich
Tapan Srivastava, The University of Chicago
Ted Shaowang, The University of Chicago
Van-Hoang Le, The University of Newcastle, Australia
Varun Jana, University of Pennsylvania
Viktor Sanca, EPFL
Wan Shen Lim, Carnegie Mellon University
William Zhang, Carnegie Mellon University
Xiang Lian, Kent State University
Xiaodong Li, The University of Hong Kong
Xiaohan Yang, Apple
Xiaojun Dong, University of California, Riverside
Xupeng Li, Columbia University
Yangshen Deng, Southern University of Science and Technology
Yesdaulet Izenov, Nazarbayev University
Yicong Huang, University of California, Irvine
Yihao Hu, Duke University
Yihao Liu, Tsinghua University
Yin Lou, Ant Group
Yingli Zhou, The Chinese University of Hong Kong, Shenzhen
Yinzhao Yan, The Hong Kong University of Science and Technology
Yuanhui Luo, Renmin University of China
Yue Gong, The University of Chicago
Yutong Ye, East China Normal University
Yuxuan Zhu, University of Illinois Urbana Champaign
Zeyan Liu, University of Louisville
Zheng Wang, Nanyang Technological University
Zheqi Shen, UC Riverside
Zhiru Zhu, The University of Chicago
Ziwei Jin, Eindhoven University of Technology
Zixuan Yi, University of Pennsylvania
Ziyun Wei, Bytedance