Dataset generation flow

This page explains the pipeline without assuming any repository-owned production configuration. The baseline simulator and regional extension share the same data and export contracts.

Overall cycle

flowchart LR
    A["Caller-owned JSON configuration"] --> B["Resolve inheritance and validate parameters"]
    B --> C["Generate complete in-memory bundle"]
    C --> D["Validate schemas and cross-table relationships"]
    D --> E["Select tables requested by caller"]
    E --> F["Built-in CSV, SQLite, or PostgreSQL writer"]
    E --> G["Trusted external writer"]
    D --> H["Validation report"]
    F --> I["Deterministic manifest"]
    G --> I
    H --> I

Table selection happens after complete validation. A caller may export only the path table, but the pipeline still checks that every path touchpoint matches the generated performance catalog and that the hidden truth table is internally coherent.

Step 1: load a caller-owned configuration

configuration.py reads the path passed through --config. There is no built-in production default. If a file contains extends, the parent path is resolved relative to that file, which allows a private project to keep reusable configuration layers outside this repository.

The loader checks dates, probabilities, positive scale factors, unique identifiers, billing rules, and path-to-touchpoint references before simulation begins.

Step 2: create daily performance rows

For each date, marketplace, and configured Touchpoint opportunity, simulation.py applies weekday behavior, trend, marketplace scaling, and random variation. It emits a native IMPRESSION and may then emit a native CLICK. Cost follows Cost Per Click or Cost Per Mille billing independently of event identity, so interaction rows can be summed without double counting.

The generated Amazon Ads-like table is observational diagnostic data. Its platform-reported purchases are not causal truth.

Step 3: create ordered path rows

For each date, marketplace, and configured path scenario, the simulator draws an audience and estimates a purchase probability from the baseline log odds, touchpoint effects, adjacent-touchpoint synergy, and optional behavior adjustments.

If a path contains ordered touchpoints $T_1,T_2,\dots ,T_n$, its unbounded purchase score is:

$$\begin{array}{rl}\text{path log odds}=& \text{baseline conversion log odds}\\ & +\sum _{i=1}^n\text{touchpoint effect}(T_i)\\ & +\text{adjacent synergy contribution}\\ & +\text{behavior adjustment}\\ & +\text{daily conversion noise}\end{array}$$

The score is converted to a probability with the logistic function documented in Mathematical Notation.

Step 4: create simulation ground truth

The simulator removes each touchpoint from the path mechanism and compares expected conversion probability before and after removal. Positive removal differences are normalized into credit shares.

This is a known answer inside the fictional world, not measured real-world causality. It exists to test whether an attribution method can recover the mechanism that generated the synthetic outcomes.

Step 5: validate the complete bundle

validation.py checks column contracts, types, ranges, structural nulls, normalized-key shape, path-to-performance matching, outcome constraints, and ground-truth credit totals. A failed bundle produces a validation report and no model table export.

Step 6: select and write tables

The default writes all tables. --table path writes only amc_path_report; repeating --table selects more tables. Built-in writers support Comma-Separated Values (CSV), SQLite, both, or direct PostgreSQL. The latter uses bounded batches and writes canonical research records without an intermediate CSV. --storage none --writer package.module:attribute delegates data output to trusted caller code.

See External Configurations and Writers for the adapter contract.

What each emitted file means

amc_path_report.csv

Primary path-based training input. It contains aggregated ordered paths and observed outcomes. A private project may choose to export only this table.

amazon_ads_daily_touchpoint_performance.csv

Optional training, feature, diagnostic, or reporting input. It describes delivery and platform-reported response by normalized touchpoint. Do not add its purchases to path-table purchases.

simulation_ground_truth.csv

Evaluation-only answer key. Never train on it as a feature.

mta_simulation.sqlite3

The same selected logical tables written by the built-in SQLite adapter. Its purpose is to prove that storage can change without changing simulation.

validation_report.json

Machine-readable result of validating the complete in-memory bundle, including tables that the caller chose not to export.

dataset_manifest.json

Reproducibility metadata: resolved-configuration hash, seed, generator version, generated row counts, selected export tables, storage mode, and hashes for local artifacts.

effective_configuration.json

The resolved simulation, Provider, Product, Campaign, Touchpoint, and budget experiment configuration used by the run.

simulation_research.json

CSV mode's local sidecar for domain entities, latent/observed Touchpoints, budget versus actual spend, delivery, ordinary outcomes, evaluation-only organic/incremental truth, and lineage. It is intentionally not another CSV schema. PostgreSQL mode writes the same research semantics directly to relational tables and does not create this large local sidecar.

Training boundary

Train path-based attribution from amc_path_report, optionally joined to carefully defined performance features. Hold simulation_ground_truth out until evaluation. This mirrors an ordinary supervised evaluation principle: an answer key cannot be both a training feature and an honest test target.

The complete stable schema is in Dataset Contract.