Welcome to ACSAC 41!Please see available submission sites below

Accepting papers, posters, case studies, panels, wips, and workshops

ACSAC 2026 Artifact Evaluation Reviewer GuideArtifact Evaluation Reviewer Guide

This evaluator guide is adapted from prior security and systems artifact evaluation processes, including the ACSAC 2025 guide and the IEEE S&P 2026 artifact evaluation process. ACSAC 2026 uses the following artifact badges: Available, Functional, and Reproduced. These definitions align with the artifact packaging and evaluation model described in the ACSAC 2026 Call for Paper Artifacts.

If you have general questions, please contact the artifact evaluation chairs . If you have a question about a specific artifact, see below for instructions on asking the authors.

Your Goal as an Artifact Evaluation Reviewer

The goal of artifact evaluation is to help science by ensuring that published papers are accompanied by high-quality artifacts (e.g., software, hardware, datasets, etc.) that can be reused and extended by others.

Your main goal is to read the paper and judge how well the artifact matches the expectations the paper sets. We expect artifacts to be (i) consistent with the paper, (ii) as complete as possible, (iii) documented well, and (iv) easy to reuse for further research.

Keep in mind that artifact evaluation is a cooperative process. Even if artifacts initially do not meet the requirements for badges, it should be your goal to enable the authors to revise the artifact so that it qualifies for badges. Hence, feedback should be actionable and interactive. Artifacts should only “miss” badges if there was not enough time to reasonably address the evaluators’ concerns, or if the authors were unresponsive or unreasonable. Note that authors will be able to update their artifacts with no restrictions in response to your comments (e.g., to fix bugs). Authors can submit artifact/documentation updates through external repositories rather than HotCRP.

We also ask you to actively engage not only with the authors, but your fellow reviewers. Peer review heavily relies on you being active in facilitating the exchange of perspectives with other reviewers and the authors.

The papers under evaluation have already been accepted by the technical program committee, or are under minor revision, so you do not need to evaluate their scientific soundness. However, if you believe you have found a technical flaw in a paper anyway, contact the artifact evaluation chairs.

Confidentiality: Keep in mind that all artifacts, reviews, and discussions are confidential. Some artifacts may even contain embargoed material (e.g., exploits) due to vulnerability disclosure. As such, you should access papers and reviews only for the purpose of discussing the submissions within the ACSAC peer review process. You should treat all submissions and reviews as confidential, and not share them with external parties. Hence, we trust you will treat all the AE material as confidential and also delete the artifact after the evaluation has finished (although most authors will publish the artifact upon publication of the paper).

Please also refer to the IEEE CompSoc Reviewer Guidelines and contact the chairs in case of further questions.

Timeline

Artifacts registration deadline: September 14 - 11:59pm AoE
Artifacts submission deadline: September 16 - 11:59pm AoEArtifacts submission deadline: September 12 — 11:59pm AoE
Reviewer bidding deadline: [AEC CHAIRS TO CONFIRM]Reviewer bidding deadline: September 14 — 11:59pm AoE
Artifact assignments released: [AEC CHAIRS TO CONFIRM]Artifact assignments released: September 15
Interactive artifact evaluation period: September 17–October 21
Kick-the-tires period: September 17–23Kick-the-tires deadline: September 22 — 11:59pm AoE
Mid-evaluation feedback deadline: [AEC CHAIRS TO CONFIRM]Mid-evaluation feedback deadline: October 6 — 11:59pm AoE
Final review deadline: [AEC CHAIRS TO CONFIRM]Final review deadline: October 19 — 11:59pm AoE
Final reviewer discussion period: [AEC CHAIRS TO CONFIRM]Final reviewer discussion period: October 20–22
Artifact evaluation decision: October 20
Final packaged artifacts and permanent repository links due: October 21Artifact evaluation decision: October 23
Final papers with AE badge labels due: refer to the camera-ready deadlineFinal papers with AE badge labels due: refer to camera-ready deadline

The bidding deadline allows the chairs to distribute artifacts in a way that balances evaluator expertise, conflicts of interest, infrastructure constraints, and reviewer load. When bidding, pay close attention to each artifact’s stated hardware, software, public infrastructure, and runtime requirements.

After artifacts are assigned, reviewers will have a kick-the-tires period to determine whether they can access the artifact, understand the documentation, run a minimal working example, and develop a realistic evaluation plan. Reviewers should provide authors with a brief summary of blocking issues or missing information by the kick-the-tires deadline.

During the subsequent review and author-discussion period, evaluators and authors actively interact to evaluate and improve the artifacts. Reviewers should communicate with authors through HotCRP to request clarifications, documentation improvements, packaging changes, or fixes to blocking issues. Mid-evaluation feedback gives authors a clear view of where the artifact stands while there is still time to address problems.

Final reviews are due before badge decisions are made. After final reviews are submitted, reviewers should use the final discussion period to converge on badge recommendations with their fellow evaluators and the AEC chairs. The final deadline for agreeing on badges is strict.

Communicating with Authors

Artifact evaluation is single-blind, meaning authors do not and must not know who you are so that you can be honest and unbiased in your assessment. To enable this, all communication between authors and reviewers must be done through HotCRP, not by other means such as email.

Please make sure that in your HotCRP profile, under “Preferences”, the “Send mail” box for “Reviews and comments on authored or reviewed submissions” is checked, so that you are notified of comments on your assigned artifacts from authors and fellow reviewers.

To add a comment on HotCRP, at the bottom of the artifact page, click on the “Add comment” button to show a form, type your comment, and select the right visibility for your comment. Discussion with authors must be “Author discussion”, while discussion with evaluators must be “Reviewer discussion”. For chairs-only comments, you can use “Administrators only”. Leave the second option to “about: submission”.

You can notify a fellow evaluator with an @-mention in a HotCRP comment, as on many other platforms. Type @ and let HotCRP autocomplete the name you want. You can also use the same @-mention mechanism to tag the AEC chairs and bring an issue to their attention. For general chair contact, use ae-chairs@acsac.org.

Use “Reviewer discussion” comments to synchronize with your fellow evaluators and ensure the same issue was not raised by another review before.

Authors submit their initial version of the artifact, artifact documentation, and any other information needed to evaluate the artifact. You should carefully read these documents and make recommendations to the authors to improve the documentation or the artifact. Any errors you find or missing information should be documented and communicated as early as possible to the authors. They will then update the artifact or documentation. The badge decision is made based on the last submitted version of the artifact package and documentation and should be independent of how many problems you ran into or changes that were needed on the path there.The badge decision is made based on the last submitted version of the artifact and appendix and should be independent of how many problems you ran into or changes that were needed on the path there.

Evaluation setup

2026 Update: Evaluators are expected to actively map each artifact’s stated resource requirements to an appropriate public infrastructure or chair-approved alternative during the kick-the-tires phase; authors should provide the necessary metadata and recommendations, but the evaluator is responsible for verifying that the proposed evaluation environment is feasible.

ACSAC 2026 expects artifacts requesting Functional or Reproduced badges to be evaluated on public research infrastructure whenever feasible. Authors should have indicated their recommended infrastructure, dependencies, expected runtime, and resource requirements in their artifact submission and metadata.toml file.

Public research infrastructure may include:

SPHERE
Chameleon
CloudLab
FABRIC
Google Colab
another appropriate public platform selected by the authors

When feasible, at least one evaluator should exercise the artifact on the public infrastructure recommended by the authors or on another appropriate public infrastructure. SPHERE is preferred when it is compatible with the artifact’s needs, but ACSAC does not require every artifact to use SPHERE.

Some artifacts may not be feasible to evaluate on public infrastructure. Examples include artifacts that require special hardware, special geolocation, licensing constraints, privacy restrictions, live services, or security-sensitive environments. In these cases, authors should explain the constraint and provide an alternative evaluation path, such as Docker or VM-based evaluation, anonymous SSH access to a special machine, access to a live service, or another chair-approved mechanism.

If reviewers must access author-provided infrastructure directly, they should preserve evaluator anonymity. Do not reveal your identity, institutional IP address, personal SSH key, or other identifying information unless explicitly cleared by the AEC chairs. When in doubt, contact the chairs before accessing the system.

If an author-provided website, service, or infrastructure uses analytics, access logging, tracking scripts, or another mechanism that could reveal an evaluator’s identity, pause before accessing it and contact the AEC chairs through HotCRP.

Some public infrastructures may have account-creation, eligibility, allocation, or security policies. If you are unable to access a required infrastructure, or if an assignment creates an infrastructure-access concern, contact the AEC chairs immediately.

Reviewers are discouraged from evaluating artifacts only on their personal desktops or laptops when public infrastructure or a reusable packaged environment is available. The goal is not merely to confirm that the artifact works once, but to assess whether the artifact can be reused by researchers outside the authors’ own environment.

Use of LLM Coding Agents

Reviewers may use large language models or other AI-based tools to assist with artifact evaluation, including troubleshooting setup issues, fixing minor errors to reproduce, interpreting scripts, and reproducing reported results.

However, use of such tools is permitted only as assistance, not as a substitute for reviewer judgment. Reviewers must verify all claims and conclusions produced by an AI tool before relying on them in an evaluation. The reviewer remains fully responsible for the accuracy, fairness, confidentiality, and completeness of their review when awarding badges to works.

In particular, reviewers must be able to explain and defend any statements in their review if asked by the artifact evaluation committee. Reviews should not include claims based solely on unverified AI output. Reviewers must also ensure that their use of AI tools does not violate confidentiality obligations or disclose non-public artifacts on sensitive works, such as undisclosed bugs in publicly used software.

Bidding phase

Once artifacts are submitted, you need to bid for the artifacts you want to review. You can enter your preferences by the bidding deadline by logging into HotCRP and clicking on “Review preferences”. You can use -20 to 20 as the range to rank the artifacts by preference and -100 to declare a conflict of interest (contact the AE chairs if unsure). When bidding, also pay attention to the hardware/software requirements of the artifact. Bid positively for at least 7 artifacts.

Note: We will try to match artifacts to your preferences, but if you don’t bid for enough (7) artifacts by the deadline, you may be assigned less-than-ideal artifact(s) for your profile.

Reviewing artifacts

Each artifact will be assigned to at least two AEC members. Evaluators should exercise the artifact independently, coordinate through HotCRP reviewer discussion, and discuss final badge recommendations when they disagree; one evaluator’s successful execution does not substitute for the other’s assessment.

The initial “kick the tires” period

Once you have been assigned artifacts, the first step is the initial “kick the tires” period. The goal of this period is to quickly determine whether the artifact can be evaluated as submitted and to map the artifact’s stated requirements onto an appropriate public research infrastructure or approved alternative environment.

During this phase, evaluators are responsible for creating an initial resource-to-infrastructure mapping for each assigned artifact. The authors’ documentation and metadata.toml file should be treated as the starting point, but evaluators should verify whether the proposed infrastructure is sufficient, practical, and available for evaluation.

In particular, make sure to do the following:

Confirm that you can access the paper, artifact package, repository, documentation, metadata.toml, datasets, models, container images, virtual machines, live services, or other required materials.
Identify the badges requested by the authors: Available, Functional, and/or Reproduced.
Extract the artifact’s stated resource requirements, including operating system, CPU architecture, GPU needs, memory, storage, network access, runtime, software dependencies, special hardware, geolocation constraints, account requirements, and security or privacy constraints.
Map those requirements to an evaluation environment. When feasible, this should be a public research infrastructure platform such as SPHERE, Chameleon, CloudLab, Google Colab, FABRIC, or another appropriate public platform.
Determine whether the authors’ recommended public infrastructure is suitable. If it is suitable, document which infrastructure will be used and what resources are needed. If it is not suitable, identify a better public-infrastructure option or explain why public infrastructure is not viable.
If public infrastructure is not viable, identify the appropriate alternative evaluation path, such as a Docker container, virtual machine, live service, anonymous SSH access to special hardware, or another chair-approved mechanism.
Identify any infrastructure-access issues that require chair intervention, such as account creation, eligibility constraints, GPU/node allocation, special hardware access, anonymized access to author-provided systems, or privacy/security concerns.
Run a minimal working example or equivalent basic task, if one is provided, to confirm that the artifact can be exercised in the selected environment.
Check whether the artifact clearly maps artifact components to paper claims, figures, tables, or experiments, especially for artifacts requesting the Reproduced badge.
Develop a concrete evaluation plan for the full review period, including which claims will be evaluated, where they will be run, what resources are required, and whether the proposed experiments can be completed within the expected evaluation time.
Share blocking issues, missing information, infrastructure concerns, and initial feedback with the authors through HotCRP as early as possible.

The kick-the-tires review should not be a full final evaluation. Its purpose is to ensure that each artifact has a feasible evaluation path, that required resources are mapped to public infrastructure or an approved alternative, and that authors have time to fix blocking issues before the main evaluation period.

At the end of kick-the-tires, each evaluator must record a short assessment in HotCRP identifying the selected public research infrastructure or approved alternative; compute, memory, storage, GPU or specialized hardware, network, software and operating-system requirements; expected runtime; whether metadata.toml and the documentation were sufficient to provision the environment; whether a minimal working example or equivalent basic operation succeeded; any blocking infrastructure or packaging issues; any chair intervention required; and the claims or experiments planned for the main review period. The kick-the-tires checkpoint is required even when no blocking issues are found.

Review Period

For each artifact you are assigned to, you will produce one review explaining which badges you believe should be awarded and why or why not. To facilitate engagement, a draft review showing the current state of the process will be made available to the authors in the middle of the review period.

You will work with the authors to produce your review, as this is a cooperative process. Authors are a resource you can use, exclusively through HotCRP, if you have trouble with an artifact or if you need more details about specific portions of an artifact.

Use the ACSAC 2026 badge definitions in this guide when evaluating artifacts. These definitions are aligned with the IEEE S&P 2026 artifact evaluation process. The IEEE Xplore badge page is supplemental background only.First, read the description of IEEE Xplorer badges . Then, read the ACSAC 2026 badge definitions: Available, Functional, and Reproduced. The review form and the badge checklists below provide a structured way to track the requirements for each badge.

For Available, determine whether the artifact will be made permanently and publicly retrievable through an appropriate archival repository.
For Functional, determine whether the artifact conforms to the expectations set by the paper for functionality, usability, and relevance. This includes assessing documentation, completeness, and exercisability.
For Reproduced, determine whether you can use the submitted artifact to obtain results that support the main claims of the paper. The goal is not exactit-for-bit reproduction, but independent results within a reasonable tolerance that validate the paper’s main claims. Scaled-down experiments are acceptable when the authors clearly and convincingly explain why they are meaningful.

You are not expected to understand every line of code, but you should be confident that the artifact overall matches the paper’s description and supports the requested badges.

Most of your time should be spent auditing artifacts, not debugging them. It is the authors’ responsibility to make their artifacts work, not yours. You do not need to spend hours trying to debug and fix complex issues; if you encounter a non-trivial error, first ask your fellow evaluators if they encountered it too, or if they know how to fix it, then ask the authors to fix it. If you run into issues such as missing dependencies, try to quickly work around them, such as finding the right package containing the dependency for your operating system and letting the authors know they have to fix their documentation.

When an artifact does not rehost cleanly on public infrastructure, communicate the problem promptly and work with the authors to identify what must change. Evaluators may diagnose packaging or infrastructure issues, but authors remain responsible for fixing substantive artifact problems.

It is acceptable to deny badges if artifacts require unreasonable effort,It is acceptable to deny badges if artifacts require unreasonable effort , especially if such effort could be avoided through automation. For instance, if reproducing a claim requires 50 points of data, and the artifact requires you to manually edit 5 config files then run 4 commands on 3 machines for each data point, you do not need to actually perform hundreds of manual steps; instead, ask the authors to automate this, or even write a script yourself if you have the time that you can then share with the authors.

Once you are finished evaluating an artifact, fill in the review form and submit it at your earliest convenience. Your review must explain in detail why the artifact should or should not get each of the badges that the authors requested. You can also include additional suggestions for the authors to improve their artifacts if you have any. Note that you can edit your review as many times as you like since reviews only become visible to the authors when final decisions are announced.

Remember that the artifact evaluation process is cooperative, not adversarial. Give authors a chance to fix issues by discussing through HotCRP comments before deciding that their artifact should not get a badge. In other words, help the authors improve their artifacts and reach badge status in the allocated time, whenever possible. However, if authors are being unresponsive or unreasonable, feel free to post a comment stating a badge cannot be awarded unless the authors take the specified steps in time by the deadline.

HotCRP allows you to rate your fellow evaluators’ reviews. If you think a review is well done, don’t hesitate to add a positive vote! If you think a review could use improvement, you can leave a negative vote and optionally a reviewer discussion comment explaining your thoughts.

When to contact the AEC chairs

Contact the AEC chairs promptly if:

you cannot access the artifact, infrastructure, or required resources;
the arti infrastructure, live services, or unusual access arrangements;
the artifact appears to contain sensitive, embargoed, or potentially harmful material not described by the authors;
author-provided access could reveal your identity;
an artifact’s public release plan seems inconsistent with the requested Available badge;
the artifact requires more than one day of evaluation and no scaled-down experiment is provided;
authors are unresponsive to blocking issues;
reviewers disagree about badge recommendations;
you identify a possible technical flaw in the accepted paper rather than merely an artifact-packaging issue.

Do not attempt to resolve anonymity, infrastructure-access, legal, ethical, or safety concerns on your own.

Badge Checklists

Below are guidelines for the three badges that can be awarded during the artifact evaluation process. The HotCRP review form will also ask reviewers to assess these categories.

Cumulative badge policy

ACSAC 2026 artifact badges are cumulative. An artifact must satisfy the requirements for Available before it can receive Functional, and must satisfy the requirements for both Available and Functional before it can receive Reproduced.

Reviewers should still independently evaluate and report whether the technical criteria for each requested badge have been satisfied. The final badges awarded, however, follow the cumulative hierarchy above.

Artifact Available Badge Checklist

The Available badge is about permanent public retrieval. To recommend this badge, check whether:

A mutable repository such as GitHub or GitLab may be used during evaluation, and reviewers may make a provisional recommendation before archival packaging is complete. Final permanent storage is a condition of receiving Available; a commitment alone is not sufficient for the final award.

The authors have provided, or committed to provide by the final artifact deadline, a permanent public repository link.
The final repository is suitable for long-term archival access, preferably with a DOI.
Acceptable repositories may include Zenodo, FigShare, Dryad, Software Heritage, institutional repositories, or other appropriate archival repositories.
The permanent repository contains the artifact files themselves, not merely a pointer to GitHub, a project webpage, or a personal website.
Any parts of the evaluated artifact that will not be publicly released are clearly identified and justified.
The public artifact remains meaningful in isolation. If the public version contains substantially less information than the evaluated artifact, contact the AEC chairs.

GitHub, GitLab, and project webpages may be useful for development, dissemination, and future updates, but they are not sufficient by themselves for the Available badge unless paired with an appropriate permanent archival repository.

The AEC chairs must verify the final permanent repository before badges are awarded. Because the badges are cumulative, failure to satisfy the final Available requirements also prevents Functional or Reproduced from being awarded.

Artifact Functitional

badge is about whether the artifact can be exercised and reused in a way that matches the paper. To recommend this badge, check whether the artifact satisfies the following criteria.

Documentation:

The artifact includes a clear README or equivalent documentation.
The documentation explains the artifact’s purpose, structure, dependencies, supported environments, and expected outputs.
The documentation describes how to install, configure, run, and interpret the artifact.
The artifact includes a minimal working example or equivalent basic test when appropriate.
The artifact explains expected runtime and resource requirements.
The artifact includes the metadata.toml file or equivalent metadata requested by the AEC.

Completeness:

The artifact includes the key components described in the paper.
The artifact uses terminology that is consistent with the paper.
The artifact clearly maps components to the paper’s claims, figures, tables, or experiments.
The artifact does not depend on missing private files, undocumented services, unpublished configuration, or obsolete code/data.
If proprietary or restricted components are required, the authors document how evaluators can access them or provide meaningful proxies.

Exercisability:

The artifact can be installed, configured, and run on a machine other than the authors’ machines.
The artifact avoids hardcoded paths, usernames, IP addresses, machine identifiers, or local-environment assumptions.
The artifact can run on the recommended public infrastructure, container, VM, live service, or chair-approved special environment.
Included scripts and software execute successfully enough to demonstrate the artifact’s claimed functionality.
Included data can be accessed and manipulated as described.
Containers or VMs include, when feasible, the Dockerfile, build script, or setup instructions needed to recreate the environment.

Results Reproduced Badge Checklist

The Reproduced badge is about whether evaluators can obtain results that support the paper’s main claims using the submitted artifact. To recommend this badge, check whether:

The authors identify the key claims, figures, tables, or experiments that the artifact is intended to reproduce.
The artifact includes scripts, workflows, or instructions for producing the relevant results.
The artifact states expected outputs and explains how to compare evaluator-generated results with the paper’s results.
The reproduced results support the main claims of the paper within a reasonable tolerance.
Deviations from the paper’s exact numbers are explained or fall within expected nondeterminism, scaling, randomness, hardware variation, dataset differences, or other stated constraints.
Scaled-down experiments, if used, are clearly justified and still meaningfully support the paper’s analyses.
The runtime for proposed evaluation tasks is reasonable, generally no more than one day unless the AEC chairs approve an exception.

The Reproduced badge does not require reproducing every number in the paper exacllows evaluators to independently obtain results that validate the paper’s main claims.