External configurations and writers

External configuration paths

--config accepts any readable JavaScript Object Notation file path supplied by the caller. No code searches for a repository-owned default. Configuration inheritance through extends is resolved relative to the child file, so a private repository can keep a self-contained configuration tree.

mta-dataset \
  --config /private/project/configs/campaign.json \
  --output /private/project/generated/run-001

The package hashes the fully resolved configuration in the manifest and writes that effective value to effective_configuration.json. CSV mode also writes simulation_research.json for local inspection of canonical master records, Provider missingness, budgets, and outcomes.

Select output tables

# Path table only
mta-dataset --config ./campaign.json --output ./output --table path

# Path and performance tables
mta-dataset \
  --config ./campaign.json \
  --output ./output \
  --table path \
  --table performance

The complete bundle is still generated and validated before selection. This prevents a path-only export from bypassing cross-table integrity checks.

External writer protocol

A writer class, instance, or zero-argument factory can live in another installed package:

from pathlib import Path

from simulations.baseline.mta_dataset import DatasetBundle


class PrivateDatabaseWriter:
    """Write selected simulation tables to a caller-owned database."""

    def write(
        self,
        bundle: DatasetBundle,
        output_directory: Path,
        table_names: tuple[str, ...],
    ) -> list[Path]:
        """Persist selected tables and return local artifacts, if any."""

        selected_rows = bundle.as_table_mapping(table_names)
        for table_name, rows in selected_rows.items():
            write_rows_to_private_database(table_name, rows)
        return []


def create_writer() -> PrivateDatabaseWriter:
    """Create the private project's configured database adapter."""

    return PrivateDatabaseWriter()

Run it with:

mta-dataset \
  --config ./campaign.json \
  --output ./run-metadata \
  --storage none \
  --table path \
  --writer private_package.database_writer:create_writer

output_directory remains required because validation and manifest metadata are local even when rows go to a remote database. An external writer returns local artifact paths that should be hashed in the manifest, or an empty list when it creates no local file.

Trusted code only

Importing an external writer executes Python code. Supply only a module controlled by your organization.

Programmatic use

from pathlib import Path

from simulations.baseline.mta_dataset import run_pipeline
from private_package.database_writer import PrivateDatabaseWriter


manifest = run_pipeline(
    configuration_path=Path("campaign.json"),
    output_directory=Path("run-metadata"),
    storage_mode="none",
    table_names=("path",),
    writers=(PrivateDatabaseWriter(),),
)

Direct PostgreSQL research mode

Install the optional driver and supply the connection through the environment so credentials do not appear in shell history:

python -m pip install './ZheyuanWu[postgresql]'
export MTA_SIM_DATABASE_URL='postgresql://...'
mta-dataset \
  --config examples/research-100k-postgresql.json \
  --output generated/research-database-metadata \
  --storage postgresql \
  --postgres-batch-size 1000 \
  --reset-database

This mode sends the generated bundle directly to PostgreSQL with parameterized batched inserts. It does not create a large intermediate CSV or research JSON. --reset-database is the only operation that drops tables, and only the simulator-owned table set is targeted. Starting the simulator without that flag never destroys populated tables.

Compatibility guarantee

Adapters should depend only on DatasetBundle.as_table_mapping(), stable table names, and the writer protocol. They should not import simulation internals. This keeps private integration code isolated from future changes to user-behavior generation.