Skip to content

Experiment Archive Specification v1

Status: Draft
Applies to: ABMForge alpha releases before the first beta
Purpose: Define the minimum directory, metadata, dataset, validation, and compatibility contract for ABMForge experiment archives.

1. Scope

An ABMForge experiment archive is a directory-based research artifact.

It is intended to make simulation outputs:

  • inspectable;
  • validated;
  • summarizable;
  • queryable;
  • citable;
  • easier to rerun or audit.

This specification covers the archive structure and metadata contract. It does not yet guarantee full independent reconstruction of a run without preserving source code, inputs, and the execution environment.

2. Design Goals

A valid ABMForge archive should:

  1. preserve the scenario or experiment configuration;
  2. preserve run-level metadata;
  3. preserve structured simulation records;
  4. preserve the dataset schema;
  5. preserve reproducibility metadata;
  6. preserve failure information;
  7. support validation;
  8. support summarization and reporting;
  9. allow future migration;
  10. remain human-inspectable.

3. Archive Layout

A v1 archive should use the following logical layout:

archive/
├── manifest.json
├── dataset_schema.json
├── registry.json
├── run_index.json
├── configs/
│   ├── scenario.yaml
│   └── experiment.yaml
├── data/
│   ├── runs.json
│   ├── model_records.jsonl
│   ├── agent_records.jsonl
│   ├── event_records.jsonl
│   ├── lifecycle_records.jsonl
│   └── errors.jsonl
├── reports/
│   └── run_summary.json
├── logs/
├── snapshots/
└── artifacts/

Implementations may omit files that do not apply to a particular archive type, but the absence of optional files must not break validation.

Machine-readable implementation contract

The Python implementation exposes the current public-alpha storage contract via abmforge.experiment.archive.archive_v1_contract().

The contract includes:

  • archive_format;
  • supported_archive_formats;
  • required_directories;
  • optional_directories;
  • required_top_level_files;
  • legacy_optional_top_level_files;
  • json_dataset_files;
  • parquet_dataset_files.

Current required directories are:

configs/
data/
reports/
logs/
snapshots/

The artifacts/ directory is optional in the current alpha contract. It may be used for additional analysis or review artifacts, but validators must not require it for minimum archive validity.

Current required top-level files are:

manifest.json
dataset_schema.json

run_index.json and registry.json are part of the v1 design, but remain legacy-optional during alpha. New archive writers should create run_index.json; readers and validators should continue to tolerate older archives without it.

The JSON and Parquet storage backends use the same logical dataset table names: runs, model_records, agent_records, event_records, lifecycle_records, and errors.

4. Required Top-Level Files

manifest.json

Required.

The manifest records reproducibility and integrity metadata.

At minimum, it should include:

  • archive format version;
  • ABMForge version;
  • Python version;
  • platform information;
  • creation timestamp;
  • storage format;
  • dataset table counts;
  • dataset table hashes where available;
  • scenario or experiment configuration hash where available.

dataset_schema.json

Required.

The dataset schema records the Dataset Schema version used by the archive.

It should define the expected structure of the standard tables:

  • runs;
  • model_records;
  • agent_records;
  • event_records;
  • lifecycle_records;
  • errors.

run_index.json

Required for new v1 archives.

Older alpha archives may omit it, but new archive writers should create it.

The run index is a compact discovery layer for archived runs. It should not replace data/runs.json; instead, it provides fast access to run identifiers, parameters, seeds, statuses, execution identity, and output locations.

For recovery, new entries should preserve at least:

  • model class name;
  • model module;
  • model qualified name;
  • recovery identity schema version;
  • integrity-checked execution fingerprint;
  • model-source fingerprint kind and SHA-256;
  • scenario name;
  • seed;
  • requested or completed step count;
  • canonical run parameters;
  • completion status.

Recovery must fail closed when execution identity is incomplete. New entries use run-identity-v2 and execution-fingerprint-v3. Older entries without a supported identity version, a valid fingerprint, model-source hash, model module, qualified name, or requested step count remain readable, but they must not silently suppress a newly planned run. A modified fingerprint or disagreement between the fingerprint and outer run metadata has the same fail-closed result. Programmatic stop_when callbacks are not recoverable matches until a stable persisted callback identity is defined.

The V3 fingerprint covers explicitly declared input-file checksums but does not automatically discover undeclared inputs. Dependency state, recorder configuration, plugins, or arbitrary callback identity. See Safe Experiment Recovery for scope and migration rules.

registry.json

Optional in early alpha archives.

When present, the registry may describe archive components, storage backends, table locations, report files, and artifact references.

5. configs/

The configs/ directory stores the user-facing configuration files that produced the archive.

Expected files:

  • configs/scenario.yaml for single-scenario runs;
  • configs/experiment.yaml for multi-run experiments.

At least one configuration file should be preserved when the archive was created through the CLI.

6. data/

The data/ directory stores machine-readable simulation outputs.

Standard JSON/JSONL files:

File Table Purpose
runs.json runs Run-level metadata, parameters, seed, status, timing, and stop reason
model_records.jsonl model_records Model-level time series observations
agent_records.jsonl agent_records Agent-level observations
event_records.jsonl event_records Event queue or event execution records
lifecycle_records.jsonl lifecycle_records Agent creation, removal, and lifecycle events
errors.jsonl errors Failure records and exception metadata

Parquet-backed archives may use equivalent .parquet files. The logical table names remain the same.

7. reports/

The reports/ directory stores derived summaries.

Reports are not the source of truth. They should be reproducible from the archive metadata and dataset tables.

Examples:

  • run_summary.json;
  • experiment_summary.json;
  • validation_report.json;
  • human_report.md.

Archive writers should finalize the manifest after required run or experiment summaries and compatibility outputs have been written. Commands that add later derived reports should register only the generated files without replacing checksum records for unrelated existing artifacts.

Experiment execution summary

Multi-run archives may include reports/experiment_summary.json.

New experiment writers record:

Field Meaning
run_count_expected Number of runs planned by the experiment configuration
run_count_executed Number of runs actually attempted
unexecuted_count Planned runs not reached
execution_status Overall execution coverage and failure state
result_summary Counts and statuses from the persisted ExperimentResult

Valid execution_status values produced by the current writer are:

  • completed: all planned runs executed successfully;
  • completed_with_failures: all planned runs executed, with one or more failures;
  • partial: fail-fast execution stopped before the complete plan was attempted.

A partial archive must include the failed run that stopped execution, all earlier persisted runs, and their error records. Runs that were never attempted must not be fabricated as run records.

8. logs/

The logs/ directory is reserved for execution logs.

Logs are optional. They should not be required for dataset validation.

9. snapshots/

The snapshots/ directory is reserved for model snapshots and replay artifacts.

Snapshot support is evolving. A v1 archive may contain snapshots, but the presence of snapshots does not imply full deterministic replay unless a replay contract explicitly says so.

10. artifacts/

The artifacts/ directory stores additional user or analysis artifacts.

Examples:

  • generated figures;
  • derived tables;
  • ODD documents;
  • analysis notebooks;
  • domain-specific outputs.

Artifact files should be referenced from the manifest or registry when they are part of the declared research artifact.

11. Run Status Semantics

A run should have one of the following statuses:

  • completed;
  • failed;
  • stopped;
  • running;
  • unknown.

New archive writers should avoid leaving a run permanently marked as running after a process exits. If an archive is resumed or repaired, unfinished runs should be represented explicitly.

12. Failure Records

Failures are first-class research data.

Derived analytical summaries should use only runs whose status is exactly completed. Failed, stopped, running, and unknown runs should remain inspectable through run-status and failure outputs, but their partial metrics should not be mixed into completed-run summaries or rankings.

A failed run should preserve:

  • run identifier;
  • scenario or experiment name;
  • seed;
  • parameters;
  • exception type where safe;
  • error message where safe;
  • step or time where failure occurred if available.

Failure records should be present even when the experiment is configured to continue after errors.

13. Validation Requirements

Archive validation should check at least:

  1. required directories exist;
  2. manifest.json exists;
  3. dataset_schema.json exists;
  4. data/ exists and is non-empty;
  5. required logical tables can be read;
  6. record counts match manifest metadata where available;
  7. hashes match manifest metadata where available;
  8. scenario or experiment configuration exists when expected;
  9. run index is well formed for new v1 archives;
  10. failure records are readable.

Validation should fail loudly on corrupted or unreadable core files.

14. Compatibility Rules

During alpha development, archive format changes may occur.

However:

  • Dataset Schema v1 changes should avoid silent breaking changes;
  • breaking changes should be documented;
  • migration paths should be provided for non-compatible archive changes;
  • readers should detect unsupported archive versions clearly.

15. Non-Goals for v1

A v1 archive does not, by itself, guarantee:

  • full source-code preservation;
  • full input-data preservation;
  • environment reconstruction;
  • Docker or Conda environment recreation;
  • bitwise identical results across platforms;
  • complete deterministic replay.

Those guarantees require additional reproducibility tiers and artifacts.

A user should be able to run:

abmforge validate outputs/archive
abmforge summarize outputs/archive
abmforge report outputs/archive

against both single-run and multi-run archives.

17. Future Extensions

Future archive versions may add:

  • archive_schema.json;
  • manifest JSON Schema validation;
  • source bundle references;
  • input artifact hashes;
  • lockfile or SBOM references;
  • streaming Parquet metadata;
  • deterministic replay metadata;
  • migration metadata;
  • signed checksums.