Introduction

This is the artifact package for our paper submitted to FSE 2026. This package includes:

• A dataset demonstrating the varied fault-revealing capabilities of different program state observation strategies observed from test code.
• The source code of our artifact.
• A PDF document detailing the experimental process and implementation.
• Supplemental 3D images for visualizing state-observation strategies across multiple dimensions for each subject studied.

Table of Contents

• Dataset
• Source Code and Documentation
• Supplemental Images

Dataset

The dataset is provided under the dataset directory. For each human-written test suit, we report the fault-revealing capabilities across varied program state observing strategies.

Each column in the CSV files captures fault-revealing capabilities on non-crashing mutants under specific configurations:

• The "baseline" column reports the capabilities of the assertions already present in the test suite.
• The "trace" column reports the proportion of non-crashing mutants detected by changes in test execution logic.
• Additional columns represent specific configurations based on: (1) state extraction depth, (2) extraction approach, (3) observation scope, and (4) trace inclusion.

Column headings are formatted as "depth-approach-scope_trace." For example, "2-getter_static_getField_no_trace" represents a configuration with:

1. A state extraction depth of 2.
2. Using getters to extract program states.
3. Observing static fields and field dereferences.
4. Excluding changes in execution trace from the fault-revealing capability calculation.

In total, fault-revealing capabilities for each subject are reported across 620 configurations.

Source Code and Documentation

The source code for our experiments is included in the source_code directory. The experiments comprise several subprojects, each documented briefly in its respective README file. Detailed implementation information, the overall experimental workflow, and decision points are documented in "Documentation.pdf."

Supplemental Images

The images directory contains large 3D plots that provide an overview of the data analyzed in our study.

Each image illustrates the fault-revealing capabilities of various state-observation strategies for a given test suite:

• The first independent variable, the state extraction approach—either getter-based or field-based—is represented by stars and triangles.
• The second variable, the state extraction scope, is shown on the x-axis, covering aspects like test-specific trace changes, static fields, variable sameness/equality, field dereferences, local variables, method return values, and combinations thereof.
• The third variable, state extraction depth, is plotted on the y-axis, ranging from 1 to 5.
• The z-axis shows the fault-revealing power of each state-observation configuration, which corresponds to assertion sets in the test code based on scope, depth, and extraction approach.

The fault-revealing capabilities of the test suite’s original assertions are indicated by a yellow plane. Our paper includes additional analysis of specific slices of these graphs.