Dataset generation flow

This page explains the whole generation cycle before the individual parameters. It covers both the peer-compatible baseline and the optional regional extension. Specialized symbols are defined in Mathematical Notation.

Inputs become a reproducible run

The generator starts with a JavaScript Object Notation configuration file. Relative inheritance is resolved first, touchpoint overrides are applied second, and the final resolved configuration is validated and hashed. The combination of that resolved configuration and the seed completely determines the generated rows.

1. Resolve and validate configuration

2. Simulate each date and marketplace

3. Validate schemas and relationships

4. Write Comma-Separated Values or SQLite

5. Hash files and write manifest

This is a cycle rather than a one-time export: edit a configuration, generate, validate, compare the result with expectations, and then either approve the version or revise the assumptions.

Daily traffic cycle

For each reporting date, the baseline combines weekly_traffic_multipliers with daily_traffic_trend:

$$\begin{array}{rl}\text{seasonal multiplier}& =\text{weekly multiplier for this weekday}\\ & \times (1+\text{daily traffic trend}\times \text{day number})\end{array}$$

The same daily cycle influences performance-table impressions and path-table users. That creates a consistent busy-day or quiet-day pattern across the two observational tables without making their conversion counts additive.

Performance-table generation

For every date, marketplace, and configured touchpoint, expected advertising volume is scaled before random positive variation:

$$\begin{array}{rl}\text{expected impressions}& =\text{base impressions}\times \text{marketplace volume multiplier}\\ & \times \text{seasonal multiplier}\end{array}$$$$\begin{array}{rl}\text{volume noise}& \sim \mathrm{Normal}(0,\text{performance volume noise standard deviation})\\ \text{impressions}& =max(0,\mathrm{round}(\text{expected impressions}\times \exp (\text{volume noise})))\end{array}$$

Clicks and Amazon Ads diagnostic purchases are sampled as counts:

$$\text{clicks}\sim \mathrm{Binomial}(\text{impressions},\text{click-through rate})$$$$\text{diagnostic purchases}\sim \mathrm{Binomial}(\text{clicks},\text{platform conversion rate})$$

See Touchpoint Parameters for the first definitions of the rates and billing values.

Path-table generation

Each configured path scenario supplies an ordered touchpoint sequence and a neutral user count. Path users follow the same market and seasonal scale as impressions:

$$\begin{array}{rl}\text{path users}& =max(1,\mathrm{round}(\text{base users}\times \text{marketplace volume multiplier}\\ & \times \text{seasonal multiplier}\times \exp (\text{audience noise})))\end{array}$$

The conversion response begins with baseline_conversion_log_odds, then adds active conversion_log_odds_effect values, adjacent adjacent_synergy_log_odds, and daily noise:

$$\begin{array}{rl}\text{conversion log odds}& =\text{baseline conversion log odds}+\text{daily noise}\\ & +\sum _{\text{active touchpoint}}\text{touchpoint log-odds effect}\\ & +\sum _{\text{active adjacent pair}}\text{adjacent synergy log odds}\end{array}$$$$\text{conversion probability}=\mathrm{logistic}(\text{conversion log odds})$$$$\text{converted users}\sim \mathrm{Binomial}(\text{path users},\text{conversion probability})$$

The logistic response keeps probability between zero and one. A configured effect of 0.91 means the conversion odds are multiplied by $\exp (0.91)$, not that conversion probability rises by 91 percentage points.

Evaluation truth and output

The simulator knows its own response function, so it enumerates every ordering of positions in a short path and calculates exact Shapley marginal contributions. Those values are written only to simulation_ground_truth; they are evaluation labels and never model-input features.

After row generation:

• validation checks exact column order, non-negative values, path-to-touchpoint matching, conversion bounds, and credit conservation;
• a Comma-Separated Values writer is the default;
• a SQLite writer demonstrates future Structured Query Language migration without changing the generator;
• the manifest records row counts, configuration hash, file hashes, seed, generator family, generator version, repository, and authors.

Generated files and their modeling roles

One successful run produces a small dataset package. The files have different responsibilities and must not all be passed to a training program.

amc_path_report.csv — primary model input

This is the synthetic Amazon Marketing Cloud path-outcome table. Each row contains a reporting window, marketplace, advertiser, ordered path, audience count, converted-user count, purchase count, and revenue.

Use for training: Yes. It supplies the ordered journey and the outcome quantities that an attribution method must allocate. For an aggregated conversion-prediction method, users is the number of trials and converted_users is the number of observed successes.

Important limit: In the finalized processing plan, an actual Amazon Marketing Cloud path query supplies these outcomes. In this repository, the values are produced by the simulator. They are synthetic outcome targets, not Amazon observations and not proof of causal credit.

amazon_ads_daily_touchpoint_performance.csv — model input and diagnostic context

This is the synthetic daily performance table. It describes each standardized touchpoint's dimensions, currency, impressions, clicks, cost, platform-diagnostic purchases, sales, and units sold.

Use for training: Yes, when the method needs touchpoint attributes, exposure, engagement, or cost context. Join a path element to normalizedTouchpoint; use marketplace, account, date, and currency when the modeling grain requires them.

Do not use incorrectly: purchases and sales in this file are Amazon Ads-style diagnostic quantities. Never add them to purchase_count or revenue from amc_path_report.csv. They can support reconciliation, features, and reporting, but they do not replace the path outcomes.

simulation_ground_truth.csv — evaluation only

This file records the known marginal contributions and credit shares calculated from the simulator's response function.

Use for training: No. It must remain hidden from feature construction, parameter fitting, preprocessing choices, and model selection performed on training data.

Use after training: Compare a fitted attribution method's credit allocation with credit_share or causal_increment on a held-out synthetic dataset. This measures recovery of the simulator's mechanism; it does not establish real-world causality.

validation_report.json — quality gate

This machine-readable report states whether the generated rows passed schema, range, key-matching, and conservation checks.

Use for training: No. A pipeline should read it before training and stop if valid is false or error_count is nonzero.

dataset_manifest.json — provenance and reproducibility

This manifest identifies the generator family and version, repository URL, authors ZheyuanWu and GPT5.6-Sol, storage mode, seed, resolved-configuration hash, row counts, and hashes of the generated data and validation files.

Repository: Trance-0/MTA-SIM-dataset

Use for training: No. Store it with experiment metadata so a result can be traced to the exact dataset package. manifest_version: 2 identifies the metadata layout that includes repository and author fields.

mta_simulation.sqlite3 — alternative storage, not another dataset

This file appears only when --storage sqlite or --storage both is selected. It contains the same logical path, performance, and ground-truth tables as the Comma-Separated Values files.

Use for training: Use the path and performance tables exactly as described above. Keep the ground-truth table evaluation-only. Do not combine SQLite rows with the equivalent Comma-Separated Values rows, because that would duplicate the dataset.

Fit with the finalized processing specification

The governing contract is MTA数据处理方案_简明版.md. The simulator targets its two final result tables; it does not claim to reproduce Amazon's upstream systems.

Exact result-table schemas

The specification requires nine Amazon Marketing Cloud path columns and sixteen Amazon Ads daily performance columns. schemas.py fixes their names and order, while validation rejects a missing, extra, or reordered column.

Four-segment touchpoint language

The specification defines:

$$\begin{array}{rl}\text{normalized touchpoint}& =\text{ad product}:\text{ad type or inventory type}\\ & :\text{placement}:\text{creative type}\end{array}$$

The simulator constructs exactly four colon-separated segments and checks that every element inside an Amazon Marketing Cloud path matches a performance-table normalizedTouchpoint.

Structural null values

The specification says that fields which do not apply to an advertising product are structural nulls, not missing measurements. The simulator retains null in raw adType, creativeType, inventoryType, or placement fields where required. It substitutes UNSPECIFIED only while constructing the normalized key.

Outcome and diagnostic separation

The specification treats Amazon Marketing Cloud purchase_count and revenue as path outcomes. Amazon Ads purchases and sales are diagnostic measures. The generator creates them through separate sampling steps, and project rules prohibit summing them.

Numeric and conservation checks

The specification requires non-negative counts and monetary values, converted_users no greater than users, four-segment keys, matched path elements, and attribution-weight conservation. validation_report.json covers those generated-table checks, while the synthetic Shapley credit shares must sum to one for each positive-effect path observation.

Storage migration

The current workflow writes Comma-Separated Values files. The SQLite adapter writes the same schemas without changing row generation, demonstrating the intended migration boundary for a future Structured Query Language database.

What the simulator does not implement from the specification

The processing specification also describes an upstream production data pipeline. Those responsibilities are outside the current synthetic-result generator:

• landing raw Amazon Marketing Cloud, Amazon Ads, and sixteen Amazon Marketing Stream feeds without rewriting them;
• detecting and removing a Chinese field-description row from an incoming Ads file;
• Amazon Marketing Stream message idempotency, replay protection, and dataset routing;
• account-time-zone conversion from hourly events to local reportDate;
• separate traffic and conversion pre-aggregation before joining at their true common grain;
• slowly changing dimension type 2 history and as-of joins for campaign-management data;
• reject tables and production source-to-target audit records;
• training First, Last, Linear, Position-based, Time-decay, Markov, or other attribution estimators;
• producing final credit_purchases, credit_revenue, attributed return on advertising spend, or attributed cost per acquisition.

These are not silently simulated or claimed as complete. The current package starts at the two finalized result-table shapes so the team can develop and test attribution logic before real upstream feeds are available.

Where the two versions differ

The pipeline and formulas above are shared. Only the marketplace behavior adjustment changes:

• Baseline version 1.0.0 uses traffic_multiplier directly, adds no regional conversion shift, and uses the global daily-noise spread.
• Regional version 0.1.0 additionally uses internet reach, target-audience density, an economic-willingness proxy, and the Gini coefficient.

This separation keeps the peer-compatible benchmark stable while letting regional assumptions evolve independently.