airbyte_ops_mcp.cli.cloud
CLI commands for Airbyte Cloud operations.
Commands:
airbyte-ops cloud connector get-version-info - Get connector version info airbyte-ops cloud connector set-version-override - Set connector version override airbyte-ops cloud connector clear-version-override - Clear connector version override airbyte-ops cloud connector regression-test - Run regression tests (single-version or comparison) airbyte-ops cloud connector fetch-connection-config - Fetch connection config to local file airbyte-ops cloud organization search - Search organizations by name/email airbyte-ops cloud organization info - Get organization info airbyte-ops cloud workspace search - Search workspaces by name/slug or email domain airbyte-ops cloud workspace info - Get workspace info
CLI reference
The commands below are regenerated by poe docs-generate via cyclopts's
programmatic docs API; see docs/generate_cli.py.
airbyte-ops cloud COMMAND
Airbyte Cloud operations.
Commands:
connection: Connection operations in Airbyte Cloud.connector: Deployed connector operations in Airbyte Cloud.db: Database operations for Airbyte Cloud Prod DB Replica.logs: GCP Cloud Logging operations for Airbyte Cloud.organization: Organization operations in Airbyte Cloud.workspace: Workspace operations in Airbyte Cloud.
airbyte-ops cloud connector
Deployed connector operations in Airbyte Cloud.
airbyte-ops cloud connector eu-runner
EU Cloud Batch regression runner.
Commands:
build-job-spec: Write a Batch v1 Job JSON file.render-summary: Print the GitHub step-summary Markdown.write-verdict: Write a masked metadata-only verdict JSON.
airbyte-ops cloud connector eu-runner build-job-spec
airbyte-ops cloud connector eu-runner build-job-spec JOB-ID SOURCE-ARCHIVE-URI RUN-PREFIX-URI CLI-ARGS-JSON GITHUB-RUN-ID CONNECTOR MAX-RUN-DURATION-SECONDS OUT [ARGS]
Write a Batch v1 Job JSON file.
Parameters:
JOB-ID, --job-id: Cloud Batch job ID. [required]SOURCE-ARCHIVE-URI, --source-archive-uri: GCS source archive URI. [required]RUN-PREFIX-URI, --run-prefix-uri: GCS run-prefix URI. [required]CLI-ARGS-JSON, --cli-args-json: JSON array of CLI arguments. [required]GITHUB-RUN-ID, --github-run-id: GitHub Actions run ID. [required]CONNECTOR, --connector: Connector name for the Batch label. [required]MAX-RUN-DURATION-SECONDS, --max-run-duration-seconds: Batch deadline in seconds. [required]OUT, --out: Write the Batch Job JSON here. [required]SEED-STATE-STAGED, --seed-state-staged, --no-seed-state-staged: A seed state was staged as state.json under the run prefix; the VM then stops when it cannot download it. [default: False]TARGET-IMAGE, --target-image: The image the workflow built and staged as target-image.tar under the run prefix; the VM loads it and stops unless the archive holds that image alone. Empty when nothing was staged. [default: ""]MACHINE-TYPE, --machine-type: [default: e2-standard-2]CPU-MILLI, --cpu-milli: [default: 2000]MEMORY-MIB, --memory-mib: [default: 6144]DISK-GB, --disk-gb: [default: 30]
airbyte-ops cloud connector eu-runner render-summary
airbyte-ops cloud connector eu-runner render-summary BATCH-URL LOGS-URL REPORT-URL VERDICT
Print the GitHub step-summary Markdown.
Parameters:
BATCH-URL, --batch-url: Cloud Batch job URL. [required]LOGS-URL, --logs-url: Cloud Logging URL. [required]REPORT-URL, --report-url: Report object URL. [required]VERDICT, --verdict: Metadata verdict JSON. [required]
airbyte-ops cloud connector eu-runner write-verdict
airbyte-ops cloud connector eu-runner write-verdict OUTPUT-DIR OUT
Write a masked metadata-only verdict JSON.
Parameters:
OUTPUT-DIR, --output-dir: Run output directory. [required]OUT, --out: Write the metadata-only verdict here. [required]
airbyte-ops cloud connector get-version-info
airbyte-ops cloud connector get-version-info WORKSPACE-ID CONNECTOR-ID CONNECTOR-TYPE
Get the current version information for a deployed connector.
Parameters:
WORKSPACE-ID, --workspace-id: The Airbyte Cloud workspace ID. [required]CONNECTOR-ID, --connector-id: The ID of the deployed connector (source or destination). [required]CONNECTOR-TYPE, --connector-type: The type of connector. [required] [choices: source, destination]
airbyte-ops cloud connector set-version-override
airbyte-ops cloud connector set-version-override VERSION REASON ISSUE-URL APPROVAL-COMMENT-URL [ARGS]
Set a version override for a deployed connector.
Requires admin authentication via AIRBYTE_INTERNAL_ADMIN_FLAG and
AIRBYTE_INTERNAL_ADMIN_USER environment variables.
The --override-level flag selects the scope at which the pin is applied:
actor(default): pins a single deployed connector instance. Requires--workspace-id,--connector-id, and--connector-type.workspace: pins ALL instances of a connector type within a workspace. Requires--workspace-idand--actor-definition-id.organization: pins ALL instances of a connector type across an organization. Requires--organization-idand--actor-definition-id.
The customer_tier_filter gates the operation: the call fails if the
actual tier of the target organization does not match. Use ALL to
proceed regardless of tier (a warning is shown for sensitive tiers).
Parameters:
VERSION, --version: The semver version string to pin to (e.g., '2.1.5-preview.abc1234'). [required]REASON, --reason: Explanation for the override (min 10 characters). [required]ISSUE-URL, --issue-url: GitHub issue URL providing context for this operation. [required]APPROVAL-COMMENT-URL, --approval-comment-url: Slack approval record URL where admin authorized this deployment. [required]OVERRIDE-LEVEL, --override-level: Scope at which to apply the version override: 'actor' (single deployed connector instance), 'workspace' (all instances of a connector type within a workspace), or 'organization' (all instances across an organization). Defaults to 'actor'. [choices: actor, workspace, organization] [default: actor]WORKSPACE-ID, --workspace-id: The Airbyte Cloud workspace ID. Required for 'actor' and 'workspace' override levels.ORGANIZATION-ID, --organization-id: The Airbyte Cloud organization ID. Required for 'organization' override level.CONNECTOR-ID, --connector-id: The ID of the deployed connector (source or destination). Required for 'actor' override level.CONNECTOR-TYPE, --connector-type: The type of connector. Required for 'actor' override level. [choices: source, destination]ACTOR-DEFINITION-ID, --actor-definition-id: The connector definition UUID. Required for 'workspace' and 'organization' override levels.AI-AGENT-SESSION-URL, --ai-agent-session-url: URL to AI agent session driving this operation (for auditability).REASON-URL, --reason-url: Optional URL with more context (e.g., issue link).CUSTOMER-TIER-FILTER, --customer-tier-filter: Tier filter: 'TIER_0', 'TIER_1', 'TIER_2', 'UNKNOWN', or 'ALL'. The operation will be rejected if the actual customer tier does not match. Defaults to 'TIER_2' (non-sensitive customers). [choices: TIER_0, TIER_1, TIER_2, UNKNOWN, ALL] [default: TIER_2]
airbyte-ops cloud connector clear-version-override
airbyte-ops cloud connector clear-version-override ISSUE-URL APPROVAL-COMMENT-URL [ARGS]
Clear a version override from a deployed connector.
Requires admin authentication via AIRBYTE_INTERNAL_ADMIN_FLAG and
AIRBYTE_INTERNAL_ADMIN_USER environment variables.
The --override-level flag selects the scope at which the pin is removed:
actor(default): clears a single deployed connector instance pin. Requires--workspace-id,--connector-id, and--connector-type.workspace: clears the workspace-level pin for a connector type. Requires--workspace-idand--actor-definition-id.organization: clears the organization-level pin for a connector type. Requires--organization-idand--actor-definition-id.
Parameters:
ISSUE-URL, --issue-url: GitHub issue URL providing context for this operation. [required]APPROVAL-COMMENT-URL, --approval-comment-url: Slack approval record URL where admin authorized this deployment. [required]OVERRIDE-LEVEL, --override-level: Scope at which to clear the version override: 'actor', 'workspace', or 'organization'. Defaults to 'actor'. [choices: actor, workspace, organization] [default: actor]WORKSPACE-ID, --workspace-id: The Airbyte Cloud workspace ID. Required for 'actor' and 'workspace' override levels.ORGANIZATION-ID, --organization-id: The Airbyte Cloud organization ID. Required for 'organization' override level.CONNECTOR-ID, --connector-id: The ID of the deployed connector (source or destination). Required for 'actor' override level.CONNECTOR-TYPE, --connector-type: The type of connector. Required for 'actor' override level. [choices: source, destination]ACTOR-DEFINITION-ID, --actor-definition-id: The connector definition UUID. Required for 'workspace' and 'organization' override levels.AI-AGENT-SESSION-URL, --ai-agent-session-url: URL to AI agent session driving this operation (for auditability).CUSTOMER-TIER-FILTER, --customer-tier-filter: Tier filter: 'TIER_0', 'TIER_1', 'TIER_2', 'UNKNOWN', or 'ALL'. The operation will be rejected if the actual customer tier does not match. Defaults to 'TIER_2' (non-sensitive customers). [choices: TIER_0, TIER_1, TIER_2, UNKNOWN, ALL] [default: TIER_2]
airbyte-ops cloud connector build-target-image
airbyte-ops cloud connector build-target-image CONNECTOR-NAME [ARGS]
Build the image regression-test --connector-name would build, and nothing else.
Writes the image to the image GitHub output. CI builds here, in a job
that holds no credentials, because a Java connector's build runs the pull
request's own Gradle on the host. The job that holds the credentials loads
the image and passes it to regression-test as --test-image.
Parameters:
CONNECTOR-NAME, --connector-name: Connector to build from source (e.g., 'source-pokeapi'). [required]REPO-ROOT, --repo-root: Path to the airbyte repo root, when it cannot be auto-detected.
airbyte-ops cloud connector regression-test
airbyte-ops cloud connector regression-test [ARGS]
Run regression tests on connectors.
This command supports two modes:
Comparison mode (skip_compare=False, default): Runs the specified Airbyte protocol command against both the target (new) and control (baseline) connector versions, then compares the results. This helps identify regressions between versions.
Single-version mode (skip_compare=True): Runs the specified Airbyte protocol command against a single connector and validates the output. No comparison is performed.
Results are written to the output directory and to GitHub Actions outputs if running in CI.
You can provide the test image in three ways:
- --test-image: Use a pre-built image from Docker registry
- --connector-name: Build the image locally from source code
- --connection-id: Auto-detect from an Airbyte Cloud connection
You can provide config/catalog either via file paths OR via a connection_id that fetches them from Airbyte Cloud.
Parameters:
SKIP-COMPARE, --skip-compare, --no-skip-compare: If True, skip comparison and run single-version tests only. If False (default), run comparison tests (target vs control). [default: False]SKIP-RECORD-COMPARISON, --skip-record-comparison, --no-skip-record-comparison: If True, skip the record-level comparison for read commands (counts, PK presence, PK integrity, field values) and gate the verdict on command outcomes only. Escape hatch for sources whose data legitimately changes between the two live runs (rolling windows, feeds that age out records) until HTTP request caching makes strict comparison reliable. [default: False]TEST-IMAGE, --test-image: Test connector image with tag (e.g., airbyte/source-github:1.0.0). When provided with connector_name, this image takes precedence over building from source. In comparison mode, it's compared against control_image.CONTROL-IMAGE, --control-image: Control connector image (baseline version) with tag (e.g., airbyte/source-github:1.0.0). Ignored ifskip_compare=True.CONNECTOR-NAME, --connector-name: Connector name to build image from source (e.g., 'source-pokeapi'). If provided without test_image, builds the image locally with tag 'dev'. When both are provided, test_image takes precedence and no build occurs. For comparison tests (default), this builds the target image. For single-version tests (skip_compare=True), this builds the test image.REPO-ROOT, --repo-root: Path to the airbyte repo root. Required if connector_name is provided and the repo cannot be auto-detected.COMMAND, --command: The Airbyte command to run. [choices: spec, check, discover, read] [default: check]CONNECTION-ID, --connection-id: Airbyte Cloud connection ID to fetch config/catalog from. Mutually exclusive with config-path/catalog-path. If provided, test_image/control_image can be auto-detected.CONFIG-PATH, --config-path: Path to the connector config JSON file.CATALOG-PATH, --catalog-path: Path to the configured catalog JSON file (required for read).STATE-PATH, --state-path: Path to the state JSON file (optional for read).OUTPUT-DIR, --output-dir: Directory to store test artifacts. [default: /tmp/regression_test_artifacts]ENABLE-HTTP-METRICS, --enable-http-metrics, --no-enable-http-metrics: Capture HTTP traffic via mitmproxy. Comparison mode always captures, with or without replay -- the per-version request counts come from the capture -- so this flag adds nothing there, and--skip-compareruns a single version without a proxy. Requires mitmdump to be installed; without it, and for an image with no certifi bundle to trust the proxy with (a Java or Kotlin connector), the run proceeds without metrics. [default: False]ENABLE-HTTP-REPLAY, --enable-http-replay, --no-http-replay: Replay the control run's recorded HTTP responses for the target run, so both versions see identical upstream data. Defaults toTruein comparison mode; has no effect with--skip-compare, which has no control run to replay from. Pass --no-http-replay to run both versions against the live API, which is the fallback whenever replay cannot be delivered anyway; the traffic is still captured, so the report keeps its per-version request counts.HTTP-REPLAY-MAX-BODY, --http-replay-max-body: Response bodies larger than this are streamed rather than recorded, and therefore re-fetched live instead of replayed. Raise it for a connector with large pages, at the cost of replay memory. Accepts a mitmproxy size such as '1m' or '10m' -- never a bare number, which means bytes. Unlike the other --http-replay-* options this one governs recording, so it applies to every comparison, replay or not. [default: 1m]HTTP-REPLAY-MAX-DUMP-SIZE-MB, --http-replay-max-dump-size-mb: Skip replay and run the target live when the filtered replay corpus is larger than this, and discard a recorded dump this large once the run is done with it. mitmproxy holds the whole corpus in memory for the entire run, and a dump grows with everything the connector downloads, so this is what keeps a big connector from exhausting the CI runner's memory and disk. Dumps are never uploaded as artifacts, so on a local run -- where the dump is the only thing that can diagnose a coverage shortfall offline -- raise it to keep the dump. [default: 256]HTTP-REPLAY-REUSE, --http-replay-reuse, --no-http-replay-reuse: Serve a recorded response more than once. By default each recorded flow is consumed on first match, which preserves recorded ordering and lets memory fall as the run proceeds; reuse keeps the whole corpus resident for the entire run. [default: False]HTTP-REPLAY-IGNORE-PARAMS, --http-replay-ignore-params: Comma-separated query parameters to exclude from the replay match key, for connectors that put a timestamp or nonce in the URL.STRICT-REPLAY, --strict-replay, --no-strict-replay: Kill requests that match nothing in the replay corpus instead of forwarding them upstream. Diagnostic only: it turns unmatched requests into connector errors, which is how you count them. mitmproxy stops intervening once its flowmap is empty, so combine this with --http-replay-reuse to cover the whole run rather than only the part before the corpus is consumed. [default: False]SELECTED-STREAMS, --selected-streams: Comma-separated list of stream names to include in the read. Only these streams will be included in the configured catalog. This is useful to limit data volume by testing only specific streams.DISCOVER-MISSING-STREAMS, --discover-missing-streams, --no-discover-missing-streams: Read selected streams the configured catalog does not have, such as a stream the PR adds: READ discovers them on the version under test and adds them to the catalog. A stream the control version also discovers is read by both versions and compared; one it does not discover has no control side and is read by the target only. Needs --selected-streams. A selected stream the target does not discover fails the run before READ, as a setup error. [default: False]INJECTED-SYNC-MODE, --injected-sync-mode: Sync mode for streams added by --discover-missing-streams. Default: full refresh where the stream supports it.incrementaluses the stream's default cursor and falls back to full refresh, with a note, for a stream that cannot be read incrementally. [choices: full_refresh, incremental]ENABLE-DEBUG-LOGS, --enable-debug-logs, --no-enable-debug-logs: Enable debug-level logging for regression test output. Also passed asLOG_LEVEL=DEBUGto the connector Docker container. Switched off on a masked run for now:--output-mode masked, or a--connection-idrun that leaves the mode unset. [default: False]EXTRA-ENV, --extra-env, --empty-extra-env: Additional KEY=VALUE environment variables for the connector containers. In comparison mode, applies to both target and control. Values are visible in the debug log of thedocker runcommand; do not use for secrets.LOG_LEVEL=DEBUGorTRACEhere counts as debug logs and is dropped on a masked run.WITH-STATE, --with-state, --no-state: Fetch and pass the connection's current state to the read command, producing a warm read instead of a cold read. Defaults toTruewhen--connection-idis provided,Falseotherwise. Has no effect unless the command isread. Ignored when--state-pathis explicitly provided.READ-TIMEOUT, --read-timeout: The read deadline in seconds. A read still running at its deadline is stopped, keeps what it had written, and is reported as stopped early. In comparison mode it is the budget for the wholereadcommand, from its start to its report: the two versions read one after the other and share it, ten minutes (or a tenth of the budget, if less) is kept for the comparison and the report, the control may use up to 60% of the read time left when it starts, and the target gets the rest; a version with no time left does not start. In single-version mode it covers the command the same way, with the same reserve, and the one read gets the rest. Only applies toread. In CI set it below the step's own timeout, so the comparison and the report still run on the partial output instead of the step being killed with nothing to show. [default: 14400]MAX-RECORDS, --max-records: Stop each side of the read when a RECORD message arrives past this many; the record past the cap is not kept, and a read of no more than this many is complete. Bounds a read of a large connection; a read the cap stops is a smoke test of the records read before the cap, not a full regression, and the report says so on each side that stopped early. Has no effect unless the command isread.OUTPUT-MODE, --output-mode: How the connector output is written to disk.maskedmasks every record value as a type-and-length token in the write pass -- primary keys are masked and a cursor stays readable only when its declared type is an integer or a date; DEBUG, TRACE, ERROR and FATAL log messages, stack traces, error internals and every stderr line keep only their length, while INFO and WARN log messages, a TRACE error's message and the CONNECTION_STATUS message stay readable.rawkeeps the output as the connector emitted it. Unset ismaskedon a--connection-idrun, which reads a customer's data, andrawotherwise; passrawto keep a connection run's output as emitted, ormaskedto mask a GSM or local run. Debug logs are off on a masked run for now:--enable-debug-logs, andLOG_LEVEL=DEBUGorTRACEin--extra-env, are dropped with a note and the run goes on. [choices: masked, raw]MASK-KEY-PATH, --mask-key-path: A file holding the run's masking key as 64 hex characters. Every command of one run (spec, check, discover, read) is given the same path, so their tokens pair within the run and with no other run's; without it the CLI makes a fresh key for this process, so a run by hand pairs its own two sides and nothing else. The key is never printed or written to any output. Ignored on a raw run.REPRO-CASE-DIR, --repro-case-dir: Reproduction case directory (seeregression-case-setup). Its catalog.json, state.json and expectations.json replace --catalog-path and --state-path, HTTP replay is turned off, and a read passes only when the control violates the case's expectations and the target satisfies them (see --repro-goal). Writes the verdict toresult.jsonin the output directory. Requires--command read, --config-path, --sandbox-scenario and comparison mode; run the command undersandbox exec. Runs raw:--output-mode maskedis refused.REPRO-GOAL, --repro-goal: What a case read must show.fixed(default): the control violates the case and the target satisfies it.reproduced: both violate it -- the case gate's check, with the unmodified source of the affected release as the target. Requires --repro-case-dir. [choices: fixed, reproduced]SANDBOX-SCENARIO, --sandbox-scenario: SaaS Sandbox scenario UUID the case was set up in. For a read, its baseline snapshot is restored immediately before each side runs, so control and target start from identical data and failure rules.
airbyte-ops cloud connector regression-case-setup
airbyte-ops cloud connector regression-case-setup CASE-DIR [ARGS]
Set up a reproduction case in the SaaS Sandbox.
Applies the case's sandbox/dataset.json together with its optional
sandbox/behavior.json (failure rules), snapshots the result as the
baseline that regression-test --repro-case-dir restores before each side
of a read, and renders config.template.json with the scenario's
synthetic credential. The sandbox connection comes from SAAS_API_URL,
SAAS_API_KEY, SAAS_WORKSPACE_ID and XDG_STATE_HOME.
Writes scenario_id, config_path and the resolved case_dir to the
GitHub step outputs.
Parameters:
CASE-DIR, --case-dir: Reproduction case directory to set up. [required]CONFIG-OUTPUT, --config-output: Where to write the rendered connector config. Defaults to a private temporary directory outside the artifacts tree.
airbyte-ops cloud connector sandbox-support
airbyte-ops cloud connector sandbox-support CONNECTOR-NAME [ARGS]
Check statically whether a SaaS Sandbox twin covers a connector's needs.
The first gate before a reproduction case: provider host match, the HTTP
operations, query parameters and page sizes the streams use, and optional
dataset needs, against the sandbox's offline discovery data (sandbox ...
--offline; needs the sandbox CLI, but no server). Prints a JSON report
whose result is supported, unsupported or unknown. unknown means
the requests could not be read statically; the case lane's twin-gap gate
still decides during the run. With --host-only, only the provider match is
decided (the oncall shadow gate's "has a twin" question).
Parameters:
CONNECTOR-NAME, --connector-name: Connector to check (e.g. 'source-slack'). [required]STREAMS, --streams: Comma-separated streams the fix touches; their parent streams are included. Defaults to every stream in the manifest.DATASET-NEEDS, --dataset-needs: Comma-separatedentityorentity.fieldthe reproduction needs in the twin's dataset (e.g. 'messages.thread_ts').CONNECTOR-DIR, --connector-dir: Local connector directory. Without it, the files are read from airbytehq/airbyte at --ref.REF, --ref: airbytehq/airbyte ref to read the connector from. [default: master]HOST-ONLY, --host-only, --no-host-only: Only decide whether a twin serves the connector's provider: one serving an allowed host and, when the manifest's request hosts can be read, one of those (token endpoints don't count). No operation is checked, sooperationsis empty. Cannot be combined with --streams or --dataset-needs. [default: False]
airbyte-ops cloud connector regression-case-gate
airbyte-ops cloud connector regression-case-gate [ARGS]
Check whether a reproduction-case PR may merge without human review.
Prints a JSON report whose status is n/a (the PR touches no case),
pass or fail, and exits nonzero on fail. Checks that the PR only adds
one new case under regression_cases/<connector>/<case>/, as regular files
with known names; that the case loads with an origin.json naming a
published airbytehq/airbyte release whose source is at affected_ref, a
commit on master; that the sandbox validates sandbox/ offline; and that
the static support check says supported. The reproduction itself is a
case run with goal reproduced. Writes connector, case,
affected_image, affected_ref, problems (a JSON list) and status to
the GitHub step outputs.
Parameters:
BASE-REF, --base-ref: The branch the PR merges into, e.g.origin/main. Required unless --case-dir is given.HEAD-REF, --head-ref: The PR's head commit; the case is read at this commit. [default: HEAD]REPO-DIR, --repo-dir: airbyte-ops-mcp clone holding both --base-ref and --head-ref. Its working tree is not read. [default: .]CASE-DIR, --case-dir: Check this case directory on disk instead of a PR: no scope check, and the connector is the name of its parent directory.REQUIRE-CASE, --require-case, --no-require-case: Fail, instead of answeringn/a, when the PR adds no case. [default: False]CONNECTOR-DIR, --connector-dir: Local connector directory at the case'saffected_ref. Without it, the files are read from airbytehq/airbyte ataffected_ref.JSON-OUT, --json-out: Also write the JSON report to this file.SANDBOX-REF, --sandbox-ref: The saas-sandbox commit the case runs use. Defaults to the pin in.github/saas-sandbox-refof the current directory (the code checkout); the case'sorigin.jsonsandbox_refmust equal it.
airbyte-ops cloud connector regression-case-validate-decide
airbyte-ops cloud connector regression-case-validate-decide PR NEEDS GATE-REPORT [ARGS]
Decide what sandbox-case-validate.yml does after the gate and the runs.
Re-reads the PR through gh (set GH_REPO and GH_TOKEN) and prints a
JSON decision: action (fail, report or merge), the commit status
state and description, and on fail the PR comment, also written to
--comment-out. Writes action, state, description and validated_sha
to the GitHub step outputs.
Parameters:
PR, --pr: The case PR's number. [required]NEEDS, --needs: File holding the workflow'stoJSON(needs). [required]GATE-REPORT, --gate-report: The gate's--json-outreport; may be missing. [required]MERGE-ENABLED, --merge-enabled, --no-merge-enabled: The merge switch (SANDBOX_CASE_AUTOMERGE_ENABLED). [default: False]RUN-URL, --run-url: This run's URL, for the comment. [default: ""]REPO, --repo:owner/nameof the repository; defaults toGH_REPO.COMMENT-OUT, --comment-out: Where to write the failure comment, when there is one. [default: comment.md]
airbyte-ops cloud connector regression-case-merge
airbyte-ops cloud connector regression-case-merge PR SHA CONNECTOR CASE [ARGS]
Merge a validated case PR at its validated head, waiting a bounded time.
Polls the PR through gh (set GH_TOKEN to a token that may merge) until
GitHub reports it mergeable, then squash-merges it through the REST merge
endpoint pinned to sha, with a fixed commit title. Never enables
auto-merge. Prints the outcome and writes outcome, merge_state, state
and description (the final commit status) to the GitHub step outputs.
Exits nonzero unless the PR merged at sha or simply did not become
mergeable in time.
Parameters:
PR, --pr: The case PR's number. [required]SHA, --sha: The validated head commit. [required]CONNECTOR, --connector: The case's connector, for the title. [required]CASE, --case: The case's name, for the title. [required]REPO, --repo:owner/nameof the repository; defaults toGH_REPO.RUN-URL, --run-url: The validating run, for the commit message. [default: ""]TIMEOUT-MINUTES, --timeout-minutes: How long to wait for GitHub to report the PR mergeable. [default: 10]
airbyte-ops cloud connector consolidate-regression-reports
airbyte-ops cloud connector consolidate-regression-reports ARTIFACTS-DIR [ARGS]
Fold one run's per-command regression reports into a single page.
The regression test runs the CLI once per Airbyte command, so each command writes its own report into its own artifact. This produces the run-level view -- a verdict table across commands plus every report inlined -- so a reviewer downloads one file instead of four.
Never fails the run: a command whose report is missing or unreadable is skipped, because this is a reporting convenience and the per-command artifacts remain the source of truth.
Parameters:
ARTIFACTS-DIR, --artifacts-dir: Directory holding one subdirectory per command, each with thereport.htmlthatregression-test --output-dirwrote. [required]OUTPUT, --output: Where to write the consolidated report. Defaults toreport.htmlin--artifacts-dir.
airbyte-ops cloud connector fetch-connection-config
airbyte-ops cloud connector fetch-connection-config CONNECTION-ID [ARGS]
Fetch connection configuration from Airbyte Cloud to a local file.
This command retrieves the source configuration for a given connection ID
and writes it to a JSON file. When --output-path is omitted the file is
written to the platform temp directory to avoid accidentally committing
secrets to a git repository.
Requires authentication via AIRBYTE_CLOUD_CLIENT_ID and AIRBYTE_CLOUD_CLIENT_SECRET environment variables.
When --with-secrets is specified, the command fetches unmasked secrets from the internal database using the connection-retriever. This additionally requires:
- An OC issue URL for audit logging (--oc-issue-url)
- GCP credentials via
GCP_PROD_DB_ACCESS_CREDENTIALSenv var orgcloud auth application-default login - Cloud SQL Python Connector access to the Prod DB replica.
Parameters:
CONNECTION-ID, --connection-id: The UUID of the Airbyte Cloud connection. [required]OUTPUT-PATH, --output-path: Path to output file or directory. If directory, writes connection--config.json inside it. Default: platform temp directory (e.g. /tmp/connection- -config.json) WITH-SECRETS, --with-secrets, --no-secrets: If set, fetches unmasked secrets from the internal database. Requires GCP_PROD_DB_ACCESS_CREDENTIALS env var orgcloud auth application-default login. Must be used with --oc-issue-url. [default: False]OC-ISSUE-URL, --oc-issue-url: OC issue URL for audit logging. Required when using --with-secrets.
airbyte-ops cloud connector rollout
Connector rollout operations.
Commands:
autopilot: AutoPilot rollout orchestration commands.list: List connector rollouts from the production database.
airbyte-ops cloud connector rollout autopilot
AutoPilot rollout orchestration commands.
airbyte-ops cloud connector rollout autopilot auto-start
airbyte-ops cloud connector rollout autopilot auto-start [ARGS]
Start INITIALIZED rollouts that have autopilot auto-start enabled.
Queries the prod DB for rollouts in initialized state, checks the
connector's autopilotConfig.autoStart gate, and starts eligible
rollouts via the Cloud Config API.
Parameters:
CONNECTOR, --connector: Filter to a specific connector by canonical name (e.g., 'source-github').DRY-RUN, --dry-run, --no-dry-run: Preview actions without executing them. [default: False]
airbyte-ops cloud connector rollout autopilot auto-advance
airbyte-ops cloud connector rollout autopilot auto-advance [ARGS]
Advance IN_PROGRESS rollouts within their current tier.
Advances in_progress rollouts that haven't reached their target
percentage based on the configured strategy pacing
(fast/slow/default).
Also recovers workflow_started rollouts: a rollout confirmed to have
zero eligible actors is finalized as succeeded (0-of-0 eligible is
complete, not stuck — promoting the RC to GA) instead of being left
wedged; one with a positive estimate is re-driven; an unavailable
estimate is skipped for a later retry.
Parameters:
CONNECTOR, --connector: Filter to a specific connector by canonical name (e.g., 'source-github').DRY-RUN, --dry-run, --no-dry-run: Preview actions without executing them. [default: False]
airbyte-ops cloud connector rollout autopilot auto-promote
airbyte-ops cloud connector rollout autopilot auto-promote [ARGS]
Promote rollouts when at 100% of current tier.
Checks the autopilotConfig.autoPromoteStages gate, then scans the tier
order forward for the next tier that has eligible actors and starts a
rollout there, skipping empty intermediate tiers. If no later customer
tier has actors (or the current tier is ALL), the current tier is
finalized as succeeded (GA promotion).
Parameters:
CONNECTOR, --connector: Filter to a specific connector by canonical name (e.g., 'source-github').DRY-RUN, --dry-run, --no-dry-run: Preview actions without executing them. [default: False]
airbyte-ops cloud connector rollout autopilot auto-triage-failed
airbyte-ops cloud connector rollout autopilot auto-triage-failed [ARGS]
Triage failed rollouts: log all failures, check unpin eligibility.
Finds rollouts in errored or paused state. Logs every failure
unconditionally. Checks safe-to-downgrade eligibility per
unsafeDowngrades. Actor-level unpinning is not yet implemented.
Parameters:
CONNECTOR, --connector: Filter to a specific connector by canonical name (e.g., 'source-github').DRY-RUN, --dry-run, --no-dry-run: Preview actions without executing them. [default: False]
airbyte-ops cloud connector rollout autopilot auto-close
airbyte-ops cloud connector rollout autopilot auto-close [ARGS]
Close obsolete rollouts so only a connector's active candidate remains.
Closes a rollout when a newer RC supersedes it, when its RC is already the
registry GA default, or when its RC is not the highest advertised release
candidate for the connector. Uses retain_pins_on_cancellation=True so
pinned actors are left undisturbed.
Parameters:
CONNECTOR, --connector: Filter to a specific connector by canonical name (e.g., 'source-github').DRY-RUN, --dry-run, --no-dry-run: Preview actions without executing them. [default: False]
airbyte-ops cloud connector rollout autopilot auto-rollback-failed
airbyte-ops cloud connector rollout autopilot auto-rollback-failed [ARGS]
Full rollback/cancel of failed rollouts when safe to downgrade.
Finds rollouts in errored state. Checks the unsafeDowngrades gate.
If the version is safe to downgrade, calls finalize_connector_rollout
with state failed_rolled_back to cancel the entire rollout.
Parameters:
CONNECTOR, --connector: Filter to a specific connector by canonical name (e.g., 'source-github').DRY-RUN, --dry-run, --no-dry-run: Preview actions without executing them. [default: False]
airbyte-ops cloud connector rollout list
airbyte-ops cloud connector rollout list [OPTIONS]
List connector rollouts from the production database.
Parameters:
--include-inactive, --no-include-inactive: Include terminal (canceled/succeeded/errored) rollouts. Requires --limit. [default: False]--limit: Maximum number of rollouts to return. Required when --include-inactive is set. [default: 0]
airbyte-ops cloud db
Database operations for Airbyte Cloud Prod DB Replica.
airbyte-ops cloud db start-proxy
airbyte-ops cloud db start-proxy [ARGS]
Start the Cloud SQL Proxy for database access.
This command starts the Cloud SQL Auth Proxy to enable connections to the Airbyte Cloud Prod DB Replica. The proxy is required for database query tools.
By default, runs as a daemon (background process). Use --no-daemon to run in foreground mode where you can see logs and stop with Ctrl+C.
Credentials are read from the GCP_PROD_DB_ACCESS_CREDENTIALS environment variable, which should contain the service account JSON credentials.
After starting the proxy, set these environment variables to use database tools: export USE_CLOUD_SQL_PROXY=1 export DB_PORT={port}
Parameters:
PORT, --port: Port for the Cloud SQL Proxy to listen on. [default: 15432]DAEMON, --daemon, --no-daemon: Run as daemon in background (default). Use --no-daemon for foreground. [default: True]
airbyte-ops cloud db stop-proxy
airbyte-ops cloud db stop-proxy
Stop the Cloud SQL Proxy daemon.
This command stops a Cloud SQL Proxy that was started with 'start-proxy'. It reads the PID from the PID file and sends a SIGTERM signal to stop the process.
airbyte-ops cloud connection
Connection operations in Airbyte Cloud.
airbyte-ops cloud connection state
Connection state operations in Airbyte Cloud.
Commands:
get: Get the current state for an Airbyte Cloud connection.reset: Reset a configured stream's state so the next sync full-refreshes it.set: Set the state for an Airbyte Cloud connection.
airbyte-ops cloud connection state get
airbyte-ops cloud connection state get CONNECTION-ID [ARGS]
Get the current state for an Airbyte Cloud connection.
Parameters:
CONNECTION-ID, --connection-id: The connection ID (UUID) to fetch state for. [required]STREAM, --stream: Optional stream name to filter state for a single stream.NAMESPACE, --namespace: Optional stream namespace to narrow the stream filter.OUTPUT, --output: Path to write the state JSON output to a file.
airbyte-ops cloud connection state set
airbyte-ops cloud connection state set CONNECTION-ID [ARGS]
Set the state for an Airbyte Cloud connection.
State JSON can be provided as a positional argument, via --input
Uses the safe variant that prevents updates while a sync is running. When --stream is provided, only that stream's state is updated within the existing connection state.
Parameters:
CONNECTION-ID, --connection-id: The connection ID (UUID) to update state for. [required]STATE-JSON, --state-json: The connection state as a JSON string. When --stream is used, this is just the stream's state blob (e.g., '{"cursor": "2024-01-01"}'). Otherwise, must include 'stateType', 'connectionId', and the appropriate state field. Can also be provided via --input or STDIN.STREAM, --stream: Optional stream name to update state for a single stream only.NAMESPACE, --namespace: Optional stream namespace to identify the stream.INPUT, --input: Path to a JSON file containing the state to set.
airbyte-ops cloud connection state reset
airbyte-ops cloud connection state reset CONNECTION-ID STREAM [ARGS]
Reset a configured stream's state so the next sync full-refreshes it.
Uses the safe variant that prevents updates while a sync is running.
Returns a restorable previous_state_backup in raw Config API format.
Parameters:
CONNECTION-ID, --connection-id: The connection ID (UUID) to update state for. [required]STREAM, --stream: The configured stream name whose state should be reset. [required]NAMESPACE, --namespace: Optional stream namespace to identify the stream.
airbyte-ops cloud connection catalog
Connection catalog operations in Airbyte Cloud.
Commands:
get: Get the configured catalog for an Airbyte Cloud connection.set: Set the configured catalog for an Airbyte Cloud connection.
airbyte-ops cloud connection catalog get
airbyte-ops cloud connection catalog get CONNECTION-ID [ARGS]
Get the configured catalog for an Airbyte Cloud connection.
Parameters:
CONNECTION-ID, --connection-id: The connection ID (UUID) to fetch catalog for. [required]OUTPUT, --output: Path to write the catalog JSON output to a file.
airbyte-ops cloud connection catalog set
airbyte-ops cloud connection catalog set CONNECTION-ID [ARGS]
Set the configured catalog for an Airbyte Cloud connection.
Catalog JSON can be provided as a positional argument, via --input
WARNING: This replaces the entire configured catalog.
Parameters:
CONNECTION-ID, --connection-id: The connection ID (UUID) to update catalog for. [required]CATALOG-JSON, --catalog-json: The configured catalog as a JSON string. Can also be provided via --input or STDIN.INPUT, --input: Path to a JSON file containing the catalog to set.
airbyte-ops cloud connection get-region
airbyte-ops cloud connection get-region CONNECTION-ID
Get a connection's dataplane group and derived residency.
Reads only the dataplane group, not connection or actor secrets. Requires
GCP_PROD_DB_ACCESS_CREDENTIALS or Application Default Credentials. On any
lookup failure it exits non-zero and writes no GitHub output.
Parameters:
CONNECTION-ID, --connection-id: The connection ID (UUID) to check. [required]
airbyte-ops cloud logs
GCP Cloud Logging operations for Airbyte Cloud.
airbyte-ops cloud logs lookup-cloud-backend-error
airbyte-ops cloud logs lookup-cloud-backend-error ERROR-ID [ARGS]
Look up error details from GCP Cloud Logging by error ID.
When an Airbyte Cloud API returns an error response with only an error ID (e.g., {"errorId": "3173452e-8f22-4286-a1ec-b0f16c1e078a"}), this command fetches the full stack trace and error details from GCP Cloud Logging.
Requires GCP credentials with Logs Viewer role on the target project. Set up credentials with: gcloud auth application-default login
Parameters:
ERROR-ID, --error-id: The error ID (UUID) to search for. This is typically returned in API error responses as {'errorId': '...'} [required]LOOKBACK-DAYS, --lookback-days: Number of days to look back in logs. [default: 7]MIN-SEVERITY-FILTER, --min-severity-filter: Optional minimum severity level to filter logs. [choices: debug, info, notice, warning, error, critical, alert, emergency]RAW, --raw, --no-raw: Output raw JSON instead of formatted text. [default: False]
airbyte-ops cloud organization
Organization operations in Airbyte Cloud.
airbyte-ops cloud organization search
airbyte-ops cloud organization search NAME-CONTAINS [ARGS]
Search organizations by name or email substring.
Parameters:
NAME-CONTAINS, --name-contains: Case-insensitive substring to match against organization name or email. [required]LIMIT, --limit: Maximum number of results to return. [default: 20]
airbyte-ops cloud organization info
airbyte-ops cloud organization info ORGANIZATION-ID
Get information about an organization (stub — not yet implemented).
Parameters:
ORGANIZATION-ID, --organization-id: The organization UUID. [required]
airbyte-ops cloud workspace
Workspace operations in Airbyte Cloud.
airbyte-ops cloud workspace search
airbyte-ops cloud workspace search [ARGS]
Search workspaces by name substring or email domain.
Parameters:
NAME-CONTAINS, --name-contains: Case-insensitive substring to match against workspace name or slug.EMAIL-DOMAIN, --email-domain: Email domain to search for (e.g. 'motherduck.com').LIMIT, --limit: Maximum number of results to return. [default: 100]
airbyte-ops cloud workspace info
airbyte-ops cloud workspace info WORKSPACE-ID
Get information about a workspace.
Parameters:
WORKSPACE-ID, --workspace-id: The workspace UUID. [required]
1# Copyright (c) 2025 Airbyte, Inc., all rights reserved. 2"""CLI commands for Airbyte Cloud operations. 3 4Commands: 5 airbyte-ops cloud connector get-version-info - Get connector version info 6 airbyte-ops cloud connector set-version-override - Set connector version override 7 airbyte-ops cloud connector clear-version-override - Clear connector version override 8 airbyte-ops cloud connector regression-test - Run regression tests (single-version or comparison) 9 airbyte-ops cloud connector fetch-connection-config - Fetch connection config to local file 10 airbyte-ops cloud organization search - Search organizations by name/email 11 airbyte-ops cloud organization info - Get organization info 12 airbyte-ops cloud workspace search - Search workspaces by name/slug or email domain 13 airbyte-ops cloud workspace info - Get workspace info 14 15## CLI reference 16 17The commands below are regenerated by `poe docs-generate` via cyclopts's 18programmatic docs API; see `docs/generate_cli.py`. 19 20.. include:: ../../../docs/generated/cli/cloud.md 21 :start-line: 2 22""" 23 24from __future__ import annotations 25 26# Hide Python-level members from the pdoc page for this module; the rendered 27# docs for this CLI group come entirely from the grafted `.. include::` in 28# the module docstring above. 29__all__: list[str] = [] 30 31import atexit 32import json 33import logging 34import os 35import secrets 36import shutil 37import signal 38import socket 39import subprocess 40import sys 41import tempfile 42import time 43import uuid 44from collections.abc import Callable 45from contextlib import suppress 46from pathlib import Path 47from typing import Annotated, Any, Literal, NoReturn 48 49import google.auth.exceptions 50import requests 51import sqlalchemy.exc 52import yaml 53from airbyte.cloud.workspaces import CloudWorkspace 54from airbyte.exceptions import PyAirbyteInputError 55from airbyte_cdk.models.connector_metadata import MetadataFile 56from airbyte_cdk.utils.connector_paths import find_connector_root_from_name 57from airbyte_cdk.utils.docker import build_connector_image, verify_docker_installation 58from airbyte_protocol.models import AirbyteCatalog, ConfiguredAirbyteCatalog 59from cyclopts import Parameter 60from fastmcp_extensions.cli import ( 61 exit_with_error, 62 print_error, 63 print_json, 64 print_success, 65 print_warning, 66) 67from google.api_core import exceptions as google_api_exceptions 68 69from airbyte_ops_mcp.cli._base import App, app 70from airbyte_ops_mcp.cloud_admin.auth import CloudAuthError 71from airbyte_ops_mcp.cloud_admin.connection_config import fetch_connection_config 72from airbyte_ops_mcp.cloud_admin.connection_state import ( 73 reset_stream_state as reset_cloud_stream_state, 74) 75from airbyte_ops_mcp.cloud_admin.registry_lookup import ( 76 resolve_definition_id_to_canonical_info, 77) 78from airbyte_ops_mcp.cloud_admin.version_overrides import ( 79 ResolvedCloudAuth, 80 VersionOverrideTarget, 81 get_connector_version_info, 82) 83from airbyte_ops_mcp.cloud_admin.version_overrides import ( 84 set_version_override as set_cloud_version_override, 85) 86from airbyte_ops_mcp.connection_config_retriever import ( 87 ConnectionNotFoundError, 88 get_connection_dataplane_group, 89 get_connection_residency, 90 residency_for_dataplane_group, 91) 92from airbyte_ops_mcp.connection_config_retriever.residency import ( 93 EU_REGION_EU_ONLY_MESSAGE, 94 EU_REGION_REQUIRED_MESSAGE, 95 is_running_in_eu_region, 96) 97from airbyte_ops_mcp.connector_secrets import ( 98 DEFAULT_GCP_PROJECT_ID, 99 ConnectorSecretsUnavailableError, 100) 101from airbyte_ops_mcp.constants import ( 102 CLOUD_SQL_INSTANCE, 103 CLOUD_SQL_PROXY_PID_FILE, 104 DEFAULT_CLOUD_SQL_PROXY_PORT, 105 ENV_GCP_PROD_DB_ACCESS_CREDENTIALS, 106) 107from airbyte_ops_mcp.gcp_logs import GCPSeverity, fetch_error_logs 108from airbyte_ops_mcp.github_api import resolve_default_github_token 109from airbyte_ops_mcp.prod_db_access.db_engine import ( 110 CloudRunProdDbAccessNotConfiguredError, 111 ProdDbCloudSqlIamError, 112 ProdDbSecretAccessError, 113) 114from airbyte_ops_mcp.prod_db_access.queries import ( 115 query_workspace_info, 116 query_workspaces_by_email_domain, 117) 118from airbyte_ops_mcp.prod_db_access.queries import ( 119 search_organizations as _search_organizations, 120) 121from airbyte_ops_mcp.prod_db_access.queries import ( 122 search_workspaces as _search_workspaces, 123) 124from airbyte_ops_mcp.regression_tests.case_gate import ( 125 NAME_PATTERN, 126 SANDBOX_PIN_FILE, 127 gate_step_outputs, 128 read_sandbox_pin, 129 run_case_dir_gate, 130 run_case_gate, 131) 132from airbyte_ops_mcp.regression_tests.case_validate import ( 133 decide_validation, 134 gh_merge, 135 gh_read_pr, 136 runs_from_needs, 137 wait_and_merge, 138) 139from airbyte_ops_mcp.regression_tests.cdk_secrets import ( 140 get_integration_test_config_from_secrets, 141) 142from airbyte_ops_mcp.regression_tests.ci_output import ( 143 ComparisonOutcome, 144 command_result_succeeded, 145 config_error_streams, 146 could_not_test, 147 describe_could_not_test, 148 describe_setup_error, 149 evaluate_comparison_outcome, 150 generate_regression_report, 151 generate_single_version_report, 152 is_github_actions, 153 run_readable_fields, 154 stopped_early_sides, 155 target_errors_only_from, 156 undefined_streams, 157 write_github_output, 158 write_github_outputs, 159 write_github_summary, 160 write_json_output, 161) 162from airbyte_ops_mcp.regression_tests.config_overrides import ( 163 InjectedCatalogs, 164 InjectedSyncMode, 165 StreamsNotDiscoveredError, 166 build_injected_catalogs, 167 filter_configured_catalog_file, 168) 169from airbyte_ops_mcp.regression_tests.connection_fetcher import ( 170 enrich_catalog_schemas_from_cloud, 171 fetch_connection_data, 172 fetch_connection_state, 173 save_connection_data_to_files, 174 streams_missing_schemas, 175) 176from airbyte_ops_mcp.regression_tests.connection_secret_retriever import ( 177 SecretRetrievalError, 178 enrich_catalog_schemas_from_platform_db, 179 enrich_config_with_secrets, 180 should_use_secret_retriever, 181) 182from airbyte_ops_mcp.regression_tests.connector_runner import ( 183 DEFAULT_TIMEOUT_SECONDS, 184 ConnectorRunner, 185 ensure_image_available, 186 proxy_trust_gap, 187) 188from airbyte_ops_mcp.regression_tests.eu_runner.cli import eu_runner_app 189from airbyte_ops_mcp.regression_tests.http_metrics import ( 190 DEFAULT_MAX_REPLAY_DUMP_SIZE_MB, 191 DEFAULT_REPLAY_EXTRA, 192 DEFAULT_STREAM_LARGE_BODIES, 193 MitmproxyManager, 194 ReplayOptions, 195 discard_oversized_dump, 196 parse_connector_cache_metrics, 197 parse_http_dump, 198) 199from airbyte_ops_mcp.regression_tests.masking import ( 200 MaskPolicy, 201 mask_input_state, 202) 203from airbyte_ops_mcp.regression_tests.models import ( 204 Command, 205 ComparableOutputs, 206 ConnectorUnderTest, 207 ExecutionInputs, 208 OutputMode, 209 StoppedEarly, 210 TargetOrControl, 211 duplicate_stream_names, 212 get_primary_keys_per_stream, 213 split_records_per_stream, 214) 215from airbyte_ops_mcp.regression_tests.normalization import stream_label, to_stream_id 216from airbyte_ops_mcp.regression_tests.read_budget import ReadBudget 217from airbyte_ops_mcp.regression_tests.regression import ( 218 ComparisonResult, 219 RecordComparisonSummary, 220 compare_catalog_schemas, 221 compare_final_states, 222 compare_specs, 223 run_record_comparisons_from_files, 224) 225from airbyte_ops_mcp.regression_tests.regression.comparators import ( 226 _read_records_file, 227 _scan_pk_integrity, 228) 229from airbyte_ops_mcp.regression_tests.report import ( 230 HTML_REPORT_FILENAME, 231 CheckResult, 232 DiffBlock, 233 DiffLine, 234 build_comparison_report_model, 235 build_diff_block, 236 build_json_block, 237 build_single_version_report_model, 238 consolidate_reports, 239 enrich_report_model, 240 failed_checks, 241 render_inline_summary, 242 report_section, 243 write_html_report, 244) 245from airbyte_ops_mcp.regression_tests.repro_case import ( 246 CASE_RESULT_FILE, 247 SHA_PATTERN, 248 ReproCase, 249 ReproductionVerdict, 250 ReproGoal, 251 SandboxCli, 252 SandboxError, 253 SandboxSide, 254 case_result, 255 check_twin_gaps, 256 evaluate_reproduction, 257 evaluate_side, 258 restore_side, 259 set_up_scenario, 260) 261from airbyte_ops_mcp.regression_tests.sandbox_support import ( 262 run_sandbox_support_check, 263) 264from airbyte_ops_mcp.regression_tests.seed_state import ( 265 seed_state_error, 266 seeded_streams, 267) 268from airbyte_ops_mcp.regression_tests.validation.catalog_validators import ( 269 validate_cursors_exist_in_schema, 270 validate_primary_keys_exist_in_schema, 271) 272from airbyte_ops_mcp.regression_tests.validation.record_validators import ( 273 check_incremental_cursor, 274 check_record_schema, 275 declared_types, 276) 277from airbyte_ops_mcp.telemetry import track_regression_test 278from airbyte_ops_mcp.tier_cache import resolve_workspace 279 280# Path to connectors directory within the airbyte repo 281CONNECTORS_SUBDIR = Path("airbyte-integrations") / "connectors" 282 283# Create the cloud sub-app 284cloud_app = App(name="cloud", help="Airbyte Cloud operations.") 285app.command(cloud_app) 286 287# Create the connector sub-app under cloud 288connector_app = App( 289 name="connector", help="Deployed connector operations in Airbyte Cloud." 290) 291cloud_app.command(connector_app) 292connector_app.command(eu_runner_app) 293 294# Create the db sub-app under cloud 295db_app = App(name="db", help="Database operations for Airbyte Cloud Prod DB Replica.") 296cloud_app.command(db_app) 297 298# Create the connection sub-app under cloud 299connection_app = App(name="connection", help="Connection operations in Airbyte Cloud.") 300cloud_app.command(connection_app) 301 302# Create the state sub-app under connection 303state_app = App(name="state", help="Connection state operations in Airbyte Cloud.") 304connection_app.command(state_app) 305 306# Create the catalog sub-app under connection 307catalog_app = App( 308 name="catalog", help="Connection catalog operations in Airbyte Cloud." 309) 310connection_app.command(catalog_app) 311 312# Create the logs sub-app under cloud 313logs_app = App(name="logs", help="GCP Cloud Logging operations for Airbyte Cloud.") 314cloud_app.command(logs_app) 315 316# Create the organization sub-app under cloud 317organization_app = App( 318 name="organization", help="Organization operations in Airbyte Cloud." 319) 320cloud_app.command(organization_app) 321 322# Create the workspace sub-app under cloud 323workspace_app = App(name="workspace", help="Workspace operations in Airbyte Cloud.") 324cloud_app.command(workspace_app) 325 326 327@db_app.command(name="start-proxy") 328def start_proxy( 329 port: Annotated[ 330 int, 331 Parameter(help="Port for the Cloud SQL Proxy to listen on."), 332 ] = DEFAULT_CLOUD_SQL_PROXY_PORT, 333 daemon: Annotated[ 334 bool, 335 Parameter( 336 help="Run as daemon in background (default). Use --no-daemon for foreground." 337 ), 338 ] = True, 339) -> None: 340 """Start the Cloud SQL Proxy for database access. 341 342 This command starts the Cloud SQL Auth Proxy to enable connections to the 343 Airbyte Cloud Prod DB Replica. The proxy is required for database query tools. 344 345 By default, runs as a daemon (background process). Use --no-daemon to run in 346 foreground mode where you can see logs and stop with Ctrl+C. 347 348 Credentials are read from the GCP_PROD_DB_ACCESS_CREDENTIALS environment variable, 349 which should contain the service account JSON credentials. 350 351 After starting the proxy, set these environment variables to use database tools: 352 export USE_CLOUD_SQL_PROXY=1 353 export DB_PORT={port} 354 355 Example: 356 airbyte-ops cloud db start-proxy 357 airbyte-ops cloud db start-proxy --port 15432 358 airbyte-ops cloud db start-proxy --no-daemon 359 """ 360 # Check if proxy is already running on the requested port (idempotency) 361 try: 362 with socket.create_connection(("127.0.0.1", port), timeout=0.5): 363 # Something is already listening on this port 364 pid_file = Path(CLOUD_SQL_PROXY_PID_FILE) 365 pid_info = "" 366 if pid_file.exists(): 367 pid_info = f" (PID: {pid_file.read_text().strip()})" 368 print_success( 369 f"Cloud SQL Proxy is already running on port {port}{pid_info}" 370 ) 371 print_success("") 372 print_success("To use database tools, set these environment variables:") 373 print_success(" export USE_CLOUD_SQL_PROXY=1") 374 print_success(f" export DB_PORT={port}") 375 return 376 except (OSError, TimeoutError, ConnectionRefusedError): 377 pass # Port not in use, proceed with starting proxy 378 379 # Check if cloud-sql-proxy is installed 380 proxy_path = shutil.which("cloud-sql-proxy") 381 if not proxy_path: 382 exit_with_error( 383 "cloud-sql-proxy not found in PATH. " 384 "Install it from: https://cloud.google.com/sql/docs/mysql/sql-proxy" 385 ) 386 387 # Get credentials from environment 388 creds_json = os.getenv(ENV_GCP_PROD_DB_ACCESS_CREDENTIALS) 389 if not creds_json: 390 exit_with_error( 391 f"{ENV_GCP_PROD_DB_ACCESS_CREDENTIALS} environment variable is not set. " 392 "This should contain the GCP service account JSON credentials." 393 ) 394 395 # Build the command using --json-credentials to avoid writing to disk 396 cmd = [ 397 proxy_path, 398 CLOUD_SQL_INSTANCE, 399 f"--port={port}", 400 f"--json-credentials={creds_json}", 401 ] 402 403 print_success(f"Starting Cloud SQL Proxy on port {port}...") 404 print_success(f"Instance: {CLOUD_SQL_INSTANCE}") 405 print_success("") 406 print_success("To use database tools, set these environment variables:") 407 print_success(" export USE_CLOUD_SQL_PROXY=1") 408 print_success(f" export DB_PORT={port}") 409 print_success("") 410 411 if daemon: 412 # Run in background (daemon mode) with log file for diagnostics 413 log_file_path = Path("/tmp/airbyte-cloud-sql-proxy.log") 414 log_file = log_file_path.open("ab") 415 process = subprocess.Popen( 416 cmd, 417 stdout=subprocess.DEVNULL, 418 stderr=log_file, 419 start_new_session=True, 420 ) 421 422 # Brief wait to verify the process started successfully 423 time.sleep(0.5) 424 if process.poll() is not None: 425 # Process exited immediately - read any error output 426 log_file.close() 427 error_output = "" 428 if log_file_path.exists(): 429 error_output = log_file_path.read_text()[-1000:] # Last 1000 chars 430 exit_with_error( 431 f"Cloud SQL Proxy failed to start (exit code: {process.returncode}).\n" 432 f"Check logs at {log_file_path}\n" 433 f"Recent output: {error_output}" 434 ) 435 436 # Write PID to file for stop-proxy command 437 pid_file = Path(CLOUD_SQL_PROXY_PID_FILE) 438 pid_file.write_text(str(process.pid)) 439 print_success(f"Cloud SQL Proxy started as daemon (PID: {process.pid})") 440 print_success(f"Logs: {log_file_path}") 441 print_success("To stop: airbyte-ops cloud db stop-proxy") 442 else: 443 # Run in foreground - replace current process 444 # Signals (Ctrl+C) will be handled directly by the cloud-sql-proxy process 445 print_success("Running in foreground. Press Ctrl+C to stop the proxy.") 446 print_success("") 447 os.execv(proxy_path, cmd) 448 449 450@db_app.command(name="stop-proxy") 451def stop_proxy() -> None: 452 """Stop the Cloud SQL Proxy daemon. 453 454 This command stops a Cloud SQL Proxy that was started with 'start-proxy'. 455 It reads the PID from the PID file and sends a SIGTERM signal to stop the process. 456 457 Example: 458 airbyte-ops cloud db stop-proxy 459 """ 460 pid_file = Path(CLOUD_SQL_PROXY_PID_FILE) 461 462 if not pid_file.exists(): 463 exit_with_error( 464 f"PID file not found at {CLOUD_SQL_PROXY_PID_FILE}. " 465 "No Cloud SQL Proxy daemon appears to be running." 466 ) 467 468 pid_str = pid_file.read_text().strip() 469 if not pid_str.isdigit(): 470 pid_file.unlink() 471 exit_with_error(f"Invalid PID in {CLOUD_SQL_PROXY_PID_FILE}: {pid_str}") 472 473 pid = int(pid_str) 474 475 # Check if process is still running 476 try: 477 os.kill(pid, 0) # Signal 0 just checks if process exists 478 except ProcessLookupError: 479 pid_file.unlink() 480 print_success( 481 f"Cloud SQL Proxy (PID: {pid}) is not running. Cleaned up PID file." 482 ) 483 return 484 except PermissionError: 485 exit_with_error(f"Permission denied to check process {pid}.") 486 487 # Send SIGTERM to stop the process 488 try: 489 os.kill(pid, signal.SIGTERM) 490 print_success(f"Sent SIGTERM to Cloud SQL Proxy (PID: {pid}).") 491 except ProcessLookupError: 492 print_success(f"Cloud SQL Proxy (PID: {pid}) already stopped.") 493 except PermissionError: 494 exit_with_error(f"Permission denied to stop process {pid}.") 495 496 # Clean up PID file 497 pid_file.unlink(missing_ok=True) 498 print_success("Cloud SQL Proxy stopped.") 499 500 501@connector_app.command(name="get-version-info") 502def get_version_info( 503 workspace_id: Annotated[ 504 str, 505 Parameter(help="The Airbyte Cloud workspace ID."), 506 ], 507 connector_id: Annotated[ 508 str, 509 Parameter(help="The ID of the deployed connector (source or destination)."), 510 ], 511 connector_type: Annotated[ 512 Literal["source", "destination"], 513 Parameter(help="The type of connector."), 514 ], 515) -> None: 516 """Get the current version information for a deployed connector.""" 517 result = get_connector_version_info( 518 auth=_resolve_cli_cloud_auth(), 519 workspace_id=workspace_id, 520 actor_id=connector_id, 521 actor_type=connector_type, 522 ) 523 print_json(result.model_dump()) 524 525 526_OverrideLevel = Literal["actor", "workspace", "organization"] 527_TierFilter = Literal["TIER_0", "TIER_1", "TIER_2", "UNKNOWN", "ALL"] 528 529 530def _resolve_cli_cloud_auth() -> ResolvedCloudAuth: 531 """Resolve Airbyte Cloud auth from environment variables for CLI use. 532 533 Mirrors the priority order the MCP server uses: 534 535 1. `AIRBYTE_CLOUD_BEARER_TOKEN` 536 2. `AIRBYTE_CLOUD_CLIENT_ID` + `AIRBYTE_CLOUD_CLIENT_SECRET` 537 538 Raises a CLI error via `exit_with_error` if no usable credentials are present. 539 """ 540 bearer_token = os.environ.get("AIRBYTE_CLOUD_BEARER_TOKEN") 541 if bearer_token: 542 return ResolvedCloudAuth(bearer_token=bearer_token) 543 544 client_id = os.environ.get("AIRBYTE_CLOUD_CLIENT_ID") 545 client_secret = os.environ.get("AIRBYTE_CLOUD_CLIENT_SECRET") 546 if client_id and client_secret: 547 return ResolvedCloudAuth(client_id=client_id, client_secret=client_secret) 548 549 exit_with_error( 550 "Missing Airbyte Cloud credentials. Set AIRBYTE_CLOUD_BEARER_TOKEN, or " 551 "both AIRBYTE_CLOUD_CLIENT_ID and AIRBYTE_CLOUD_CLIENT_SECRET." 552 ) 553 554 555def _validate_override_scope_args( 556 *, 557 override_level: _OverrideLevel, 558 workspace_id: str | None, 559 organization_id: str | None, 560 connector_id: str | None, 561 connector_type: Literal["source", "destination"] | None, 562 actor_definition_id: str | None, 563) -> None: 564 """Validate scope-specific arguments for `set/clear-version-override`. 565 566 Calls `exit_with_error` (which terminates the process) if the supplied 567 arguments are inconsistent with the requested `override_level`. 568 """ 569 if override_level == "actor": 570 missing = [ 571 name 572 for name, value in ( 573 ("--workspace-id", workspace_id), 574 ("--connector-id", connector_id), 575 ("--connector-type", connector_type), 576 ) 577 if not value 578 ] 579 if missing: 580 exit_with_error( 581 "Override level 'actor' requires " + ", ".join(missing) + "." 582 ) 583 if actor_definition_id is not None: 584 exit_with_error( 585 "Override level 'actor' must not be combined with " 586 "--actor-definition-id." 587 ) 588 if organization_id is not None: 589 exit_with_error( 590 "Override level 'actor' must not be combined with --organization-id." 591 ) 592 return 593 594 if override_level == "workspace": 595 missing = [ 596 name 597 for name, value in ( 598 ("--workspace-id", workspace_id), 599 ("--actor-definition-id", actor_definition_id), 600 ) 601 if not value 602 ] 603 if missing: 604 exit_with_error( 605 "Override level 'workspace' requires " + ", ".join(missing) + "." 606 ) 607 if connector_id is not None or connector_type is not None: 608 exit_with_error( 609 "Override level 'workspace' must not be combined with " 610 "--connector-id or --connector-type." 611 ) 612 if organization_id is not None: 613 exit_with_error( 614 "Override level 'workspace' must not be combined with " 615 "--organization-id." 616 ) 617 return 618 619 # override_level == "organization" 620 missing = [ 621 name 622 for name, value in ( 623 ("--organization-id", organization_id), 624 ("--actor-definition-id", actor_definition_id), 625 ) 626 if not value 627 ] 628 if missing: 629 exit_with_error( 630 "Override level 'organization' requires " + ", ".join(missing) + "." 631 ) 632 if connector_id is not None or connector_type is not None: 633 exit_with_error( 634 "Override level 'organization' must not be combined with " 635 "--connector-id or --connector-type." 636 ) 637 if workspace_id is not None: 638 exit_with_error( 639 "Override level 'organization' must not be combined with --workspace-id." 640 ) 641 642 643def _dispatch_version_override( 644 *, 645 override_level: _OverrideLevel, 646 workspace_id: str | None, 647 organization_id: str | None, 648 connector_id: str | None, 649 connector_type: Literal["source", "destination"] | None, 650 actor_definition_id: str | None, 651 version: str | None, 652 unset: bool, 653 reason: str | None, 654 reason_url: str | None, 655 issue_url: str, 656 approval_comment_url: str, 657 ai_agent_session_url: str | None, 658 customer_tier_filter: _TierFilter, 659) -> None: 660 """Route a version-override request through the normalized core helper. 661 662 Resolves `--actor-definition-id` to its canonical name and connector type 663 via the cloud registry for `workspace`/`organization` scopes. Prints the 664 operation result as JSON. 665 """ 666 _validate_override_scope_args( 667 override_level=override_level, 668 workspace_id=workspace_id, 669 organization_id=organization_id, 670 connector_id=connector_id, 671 connector_type=connector_type, 672 actor_definition_id=actor_definition_id, 673 ) 674 675 auth = _resolve_cli_cloud_auth() 676 677 if override_level == "actor": 678 assert workspace_id is not None 679 assert connector_id is not None 680 assert connector_type is not None 681 assert organization_id is None 682 try: 683 ws_resolution = resolve_workspace(workspace_id) 684 if not ws_resolution.organization_id: 685 exit_with_error("Could not resolve organization for workspace.") 686 result = set_cloud_version_override( 687 auth=auth, 688 target=VersionOverrideTarget( 689 scope="actor", 690 organization_id=ws_resolution.organization_id, 691 workspace_id=workspace_id, 692 actor_id=connector_id, 693 connector_type=connector_type, 694 ), 695 approval_comment_url=approval_comment_url, 696 version=version, 697 unset=unset, 698 override_reason=reason, 699 override_reason_reference_url=reason_url, 700 issue_url=issue_url, 701 ai_agent_session_url=ai_agent_session_url, 702 customer_tier_filter=customer_tier_filter, 703 ) 704 except (PyAirbyteInputError, CloudAuthError) as e: 705 exit_with_error(str(e)) 706 _print_override_result(result.success, result.message) 707 print_json(result.model_dump()) 708 return 709 710 try: 711 assert actor_definition_id is not None 712 canonical_name, resolved_connector_type = ( 713 resolve_definition_id_to_canonical_info(actor_definition_id) 714 ) 715 except (PyAirbyteInputError, requests.RequestException) as e: 716 exit_with_error(f"Failed to resolve --actor-definition-id: {e}") 717 typed_connector_type: Literal["source", "destination"] = ( 718 "source" if resolved_connector_type == "source" else "destination" 719 ) 720 721 if override_level == "workspace": 722 assert workspace_id is not None 723 try: 724 ws_resolution = resolve_workspace(workspace_id) 725 if not ws_resolution.organization_id: 726 exit_with_error("Could not resolve organization for workspace.") 727 target = VersionOverrideTarget( 728 scope="workspace", 729 organization_id=ws_resolution.organization_id, 730 workspace_id=workspace_id, 731 connector_name=canonical_name, 732 connector_type=typed_connector_type, 733 ) 734 except PyAirbyteInputError as e: 735 exit_with_error(str(e)) 736 else: 737 assert organization_id is not None 738 target = VersionOverrideTarget( 739 scope="organization", 740 organization_id=organization_id, 741 connector_name=canonical_name, 742 connector_type=typed_connector_type, 743 ) 744 745 try: 746 result = set_cloud_version_override( 747 auth=auth, 748 target=target, 749 approval_comment_url=approval_comment_url, 750 version=version, 751 unset=unset, 752 override_reason=reason, 753 override_reason_reference_url=reason_url, 754 issue_url=issue_url, 755 ai_agent_session_url=ai_agent_session_url, 756 customer_tier_filter=customer_tier_filter, 757 ) 758 except (PyAirbyteInputError, CloudAuthError) as e: 759 exit_with_error(str(e)) 760 _print_override_result(result.success, result.message) 761 print_json(result.model_dump()) 762 763 764def _print_override_result(success: bool, message: str) -> None: 765 """Emit an override-operation status message via the shared CLI helpers.""" 766 if success: 767 print_success(message) 768 else: 769 print_error(message) 770 771 772@connector_app.command(name="set-version-override") 773def set_version_override( 774 version: Annotated[ 775 str, 776 Parameter( 777 help="The semver version string to pin to (e.g., '2.1.5-preview.abc1234')." 778 ), 779 ], 780 reason: Annotated[ 781 str, 782 Parameter(help="Explanation for the override (min 10 characters)."), 783 ], 784 issue_url: Annotated[ 785 str, 786 Parameter(help="GitHub issue URL providing context for this operation."), 787 ], 788 approval_comment_url: Annotated[ 789 str, 790 Parameter( 791 help="Slack approval record URL where admin authorized this deployment." 792 ), 793 ], 794 override_level: Annotated[ 795 _OverrideLevel, 796 Parameter( 797 help=( 798 "Scope at which to apply the version override: 'actor' (single " 799 "deployed connector instance), 'workspace' (all instances of a " 800 "connector type within a workspace), or 'organization' (all " 801 "instances across an organization). Defaults to 'actor'." 802 ), 803 ), 804 ] = "actor", 805 workspace_id: Annotated[ 806 str | None, 807 Parameter( 808 help=( 809 "The Airbyte Cloud workspace ID. Required for 'actor' and " 810 "'workspace' override levels." 811 ) 812 ), 813 ] = None, 814 organization_id: Annotated[ 815 str | None, 816 Parameter( 817 help=( 818 "The Airbyte Cloud organization ID. Required for " 819 "'organization' override level." 820 ) 821 ), 822 ] = None, 823 connector_id: Annotated[ 824 str | None, 825 Parameter( 826 help=( 827 "The ID of the deployed connector (source or destination). " 828 "Required for 'actor' override level." 829 ) 830 ), 831 ] = None, 832 connector_type: Annotated[ 833 Literal["source", "destination"] | None, 834 Parameter(help="The type of connector. Required for 'actor' override level."), 835 ] = None, 836 actor_definition_id: Annotated[ 837 str | None, 838 Parameter( 839 help=( 840 "The connector definition UUID. Required for 'workspace' and " 841 "'organization' override levels." 842 ) 843 ), 844 ] = None, 845 ai_agent_session_url: Annotated[ 846 str | None, 847 Parameter( 848 help="URL to AI agent session driving this operation (for auditability)." 849 ), 850 ] = None, 851 reason_url: Annotated[ 852 str | None, 853 Parameter(help="Optional URL with more context (e.g., issue link)."), 854 ] = None, 855 customer_tier_filter: Annotated[ 856 _TierFilter, 857 Parameter( 858 help=( 859 "Tier filter: 'TIER_0', 'TIER_1', 'TIER_2', 'UNKNOWN', or 'ALL'. " 860 "The operation will be rejected if the actual customer tier does not match. " 861 "Defaults to 'TIER_2' (non-sensitive customers)." 862 ), 863 ), 864 ] = "TIER_2", 865) -> None: 866 """Set a version override for a deployed connector. 867 868 Requires admin authentication via `AIRBYTE_INTERNAL_ADMIN_FLAG` and 869 `AIRBYTE_INTERNAL_ADMIN_USER` environment variables. 870 871 The `--override-level` flag selects the scope at which the pin is applied: 872 873 - `actor` (default): pins a single deployed connector instance. Requires 874 `--workspace-id`, `--connector-id`, and `--connector-type`. 875 - `workspace`: pins ALL instances of a connector type within a workspace. 876 Requires `--workspace-id` and `--actor-definition-id`. 877 - `organization`: pins ALL instances of a connector type across an 878 organization. Requires `--organization-id` and `--actor-definition-id`. 879 880 The `customer_tier_filter` gates the operation: the call fails if the 881 actual tier of the target organization does not match. Use `ALL` to 882 proceed regardless of tier (a warning is shown for sensitive tiers). 883 """ 884 _dispatch_version_override( 885 override_level=override_level, 886 workspace_id=workspace_id, 887 organization_id=organization_id, 888 connector_id=connector_id, 889 connector_type=connector_type, 890 actor_definition_id=actor_definition_id, 891 version=version, 892 unset=False, 893 reason=reason, 894 reason_url=reason_url, 895 issue_url=issue_url, 896 approval_comment_url=approval_comment_url, 897 ai_agent_session_url=ai_agent_session_url, 898 customer_tier_filter=customer_tier_filter, 899 ) 900 901 902@connector_app.command(name="clear-version-override") 903def clear_version_override( 904 issue_url: Annotated[ 905 str, 906 Parameter(help="GitHub issue URL providing context for this operation."), 907 ], 908 approval_comment_url: Annotated[ 909 str, 910 Parameter( 911 help="Slack approval record URL where admin authorized this deployment." 912 ), 913 ], 914 override_level: Annotated[ 915 _OverrideLevel, 916 Parameter( 917 help=( 918 "Scope at which to clear the version override: 'actor', " 919 "'workspace', or 'organization'. Defaults to 'actor'." 920 ), 921 ), 922 ] = "actor", 923 workspace_id: Annotated[ 924 str | None, 925 Parameter( 926 help=( 927 "The Airbyte Cloud workspace ID. Required for 'actor' and " 928 "'workspace' override levels." 929 ) 930 ), 931 ] = None, 932 organization_id: Annotated[ 933 str | None, 934 Parameter( 935 help=( 936 "The Airbyte Cloud organization ID. Required for " 937 "'organization' override level." 938 ) 939 ), 940 ] = None, 941 connector_id: Annotated[ 942 str | None, 943 Parameter( 944 help=( 945 "The ID of the deployed connector (source or destination). " 946 "Required for 'actor' override level." 947 ) 948 ), 949 ] = None, 950 connector_type: Annotated[ 951 Literal["source", "destination"] | None, 952 Parameter(help="The type of connector. Required for 'actor' override level."), 953 ] = None, 954 actor_definition_id: Annotated[ 955 str | None, 956 Parameter( 957 help=( 958 "The connector definition UUID. Required for 'workspace' and " 959 "'organization' override levels." 960 ) 961 ), 962 ] = None, 963 ai_agent_session_url: Annotated[ 964 str | None, 965 Parameter( 966 help="URL to AI agent session driving this operation (for auditability)." 967 ), 968 ] = None, 969 customer_tier_filter: Annotated[ 970 _TierFilter, 971 Parameter( 972 help=( 973 "Tier filter: 'TIER_0', 'TIER_1', 'TIER_2', 'UNKNOWN', or 'ALL'. " 974 "The operation will be rejected if the actual customer tier does not match. " 975 "Defaults to 'TIER_2' (non-sensitive customers)." 976 ), 977 ), 978 ] = "TIER_2", 979) -> None: 980 """Clear a version override from a deployed connector. 981 982 Requires admin authentication via `AIRBYTE_INTERNAL_ADMIN_FLAG` and 983 `AIRBYTE_INTERNAL_ADMIN_USER` environment variables. 984 985 The `--override-level` flag selects the scope at which the pin is removed: 986 987 - `actor` (default): clears a single deployed connector instance pin. 988 Requires `--workspace-id`, `--connector-id`, and `--connector-type`. 989 - `workspace`: clears the workspace-level pin for a connector type. 990 Requires `--workspace-id` and `--actor-definition-id`. 991 - `organization`: clears the organization-level pin for a connector type. 992 Requires `--organization-id` and `--actor-definition-id`. 993 """ 994 _dispatch_version_override( 995 override_level=override_level, 996 workspace_id=workspace_id, 997 organization_id=organization_id, 998 connector_id=connector_id, 999 connector_type=connector_type, 1000 actor_definition_id=actor_definition_id, 1001 version=None, 1002 unset=True, 1003 reason=None, 1004 reason_url=None, 1005 issue_url=issue_url, 1006 approval_comment_url=approval_comment_url, 1007 ai_agent_session_url=ai_agent_session_url, 1008 customer_tier_filter=customer_tier_filter, 1009 ) 1010 1011 1012def _load_json_file(file_path: Path) -> dict | None: 1013 """Load a JSON file and return its contents. 1014 1015 Returns None if the file doesn't exist or contains invalid JSON. 1016 """ 1017 if not file_path.exists(): 1018 return None 1019 try: 1020 return json.loads(file_path.read_text()) 1021 except json.JSONDecodeError as e: 1022 print_error(f"Failed to parse JSON in file: {file_path}\nError: {e}") 1023 return None 1024 1025 1026def _parse_extra_env(pairs: list[str] | None) -> dict[str, str]: 1027 """Parse repeatable KEY=VALUE arguments for connector container env.""" 1028 parsed: dict[str, str] = {} 1029 for item in pairs or []: 1030 if "=" not in item: 1031 exit_with_error(f"Invalid --extra-env value (expected KEY=VALUE): {item}") 1032 key, value = item.split("=", 1) 1033 key = key.strip() 1034 if not key: 1035 exit_with_error(f"Invalid --extra-env value (expected KEY=VALUE): {item}") 1036 parsed[key] = value 1037 return parsed 1038 1039 1040_DEBUG_LOG_LEVELS = frozenset({"DEBUG", "TRACE"}) 1041 1042 1043def _private_dir(prefix: str) -> Path: 1044 """A temp directory for the connector's secrets, removed when the CLI exits. 1045 1046 `mkdtemp` makes it 0700. On a runner the VM dies with the job; on a laptop 1047 nothing else would remove it, and it holds the config and the raw state. 1048 `atexit` runs on a normal exit and on Ctrl-C, not on SIGTERM or `kill -9`, 1049 so a run killed that way still leaves the directory behind. 1050 """ 1051 path = Path(tempfile.mkdtemp(prefix=prefix)) 1052 atexit.register(shutil.rmtree, path, ignore_errors=True) 1053 return path 1054 1055 1056def _without_debug_logs( 1057 *, enable_debug_logs: bool, extra_env: dict[str, str] 1058) -> tuple[dict[str, str], list[str]]: 1059 """The container env without a debug log level, and the settings dropped. 1060 1061 Debug logs are off on masked runs for now, to keep masking simple. A 1062 `LOG_LEVEL` passed through `--extra-env` reaches the connector exactly as 1063 `--enable-debug-logs` does, so a DEBUG or TRACE one is dropped too. 1064 """ 1065 dropped = ["--enable-debug-logs"] if enable_debug_logs else [] 1066 level = extra_env.get("LOG_LEVEL") 1067 if level is not None and level.strip().upper() in _DEBUG_LOG_LEVELS: 1068 dropped.append(f"LOG_LEVEL={level} from --extra-env") 1069 extra_env = { 1070 key: value for key, value in extra_env.items() if key != "LOG_LEVEL" 1071 } 1072 return extra_env, dropped 1073 1074 1075# The masking key's length: HMAC-SHA256's block is 64 bytes, and 32 random 1076# bytes is the full strength of the hash it feeds. 1077MASK_KEY_BYTES = 32 1078 1079 1080def _resolve_mask_key(mask_key_path: str | None) -> bytes: 1081 """The key every token of this run is hashed under. 1082 1083 From the file when given: the workflow makes one key per run and hands 1084 every command the path, so the four commands' tokens pair and the key 1085 itself goes through no argument the runner would print. Without it a 1086 fresh key for this process, so a run by hand pairs its own two sides and 1087 nothing else. The key is returned, never printed; the errors name the 1088 file, not what it held. 1089 """ 1090 if mask_key_path is None: 1091 return secrets.token_bytes(MASK_KEY_BYTES) 1092 try: 1093 text = Path(mask_key_path).read_text().strip() 1094 except OSError as exc: 1095 error = f"--mask-key-path cannot be read: {exc}" 1096 else: 1097 try: 1098 key = bytes.fromhex(text) 1099 except ValueError: 1100 key = b"" 1101 if len(key) == MASK_KEY_BYTES: 1102 return key 1103 error = ( 1104 f"--mask-key-path must hold the run's masking key as " 1105 f"{2 * MASK_KEY_BYTES} hex characters: {mask_key_path}" 1106 ) 1107 write_github_output("success", False) 1108 write_github_output("error", error) 1109 exit_with_error(error) 1110 1111 1112def _mask_policy( 1113 catalog: Any, output_mode: OutputMode, mask_key: bytes | None 1114) -> MaskPolicy | None: 1115 """The policy a masked run masks `catalog`'s streams under; `None` on a raw run. 1116 1117 A masked run without a key is a programming error, not a raw run: 1118 refusing it keeps a missing key from writing output under the wrong 1119 mode. 1120 """ 1121 if output_mode != "masked": 1122 return None 1123 if mask_key is None: 1124 raise ValueError("A masked run needs its masking key") 1125 return MaskPolicy.from_configured_catalog(catalog, key=mask_key) 1126 1127 1128def _run_connector_command( 1129 connector_image: str, 1130 command: Command, 1131 output_dir: Path, 1132 target_or_control: TargetOrControl, 1133 config_path: Path | None = None, 1134 catalog_path: Path | None = None, 1135 state_path: Path | None = None, 1136 proxy_url: str | None = None, 1137 ca_cert_path: Path | None = None, 1138 enable_debug_logs: bool = False, 1139 timeout_seconds: int | None = None, 1140 max_records: int | None = None, 1141 output_mode: OutputMode = "raw", 1142 mask_key: bytes | None = None, 1143 mask_catalog_path: Path | None = None, 1144 extra_env: dict[str, str] | None = None, 1145) -> tuple[dict, ComparableOutputs]: 1146 """Run a connector command and return its results and its declared objects. 1147 1148 The dict is a flattened view of `ExecutionResult`: counts, exit status and 1149 artifact paths, which is all the reports need. What a comparison needs 1150 instead -- the spec, the discovered catalog -- travels beside it as 1151 `ComparableOutputs` rather than growing a dict that several consumers read. 1152 The `ExecutionResult` itself is still dropped here: it holds every parsed 1153 message, and a comparison keeps both sides alive at once. 1154 1155 Args: 1156 connector_image: Full connector image name with tag. 1157 command: The Airbyte command to run. 1158 output_dir: Directory to store output files. 1159 target_or_control: Whether this is target or control version. 1160 config_path: Path to connector config JSON file. 1161 catalog_path: Path to configured catalog JSON file. 1162 state_path: Path to state JSON file. 1163 proxy_url: Optional HTTP proxy URL for traffic capture. 1164 ca_cert_path: Path to the proxy's CA certificate, so the container trusts 1165 it. Without it every HTTPS request through the proxy fails TLS 1166 verification and the capture comes back near-empty. 1167 enable_debug_logs: If `True`, pass `LOG_LEVEL=DEBUG` to the connector container. 1168 timeout_seconds: How long the container may run before the harness stops 1169 it and reports the run as stopped early; the runner's default when 1170 `None`. 1171 max_records: Stop the container after this many RECORD messages and 1172 report the run as stopped early; no cap when `None`. 1173 output_mode: `masked` masks every stdout line as it is written, 1174 under a policy built from the configured catalog that masks primary 1175 keys and keeps a cursor readable only when its declared type is an 1176 integer or a date; `raw` writes the output as emitted. 1177 mask_key: The run's masking key, which every token is hashed under; 1178 both sides of a comparison are given the same one. Required on a 1179 masked run, unused on a raw one. 1180 mask_catalog_path: The catalog the mask policy is built from, when it 1181 is not `catalog_path`'s: both sides of a comparison must mask under 1182 one policy, so their tokens pair. 1183 extra_env: Additional environment variables for the connector container. 1184 1185 Returns: 1186 The flattened results, and the objects the connector declared. A run 1187 the harness stopped carries `stopped_early` -- its reason, the records 1188 kept and the seconds it ran -- and `None` there otherwise. 1189 """ 1190 connector = ConnectorUnderTest.from_image_name(connector_image, target_or_control) 1191 1192 config = _load_json_file(config_path) if config_path else None 1193 state = _load_json_file(state_path) if state_path else None 1194 1195 configured_catalog = None 1196 if catalog_path and catalog_path.exists(): 1197 catalog_json = catalog_path.read_text() 1198 configured_catalog = ConfiguredAirbyteCatalog.parse_raw(catalog_json) # ty: ignore[deprecated] 1199 1200 mask_catalog = configured_catalog 1201 if mask_catalog_path is not None and output_mode == "masked": 1202 mask_catalog = ConfiguredAirbyteCatalog.parse_raw(mask_catalog_path.read_text()) # ty: ignore[deprecated] 1203 1204 # Pass log level as an environment variable to the connector container 1205 container_env: dict[str, str] = {} 1206 if enable_debug_logs: 1207 container_env["LOG_LEVEL"] = "DEBUG" 1208 container_env.update(extra_env or {}) 1209 1210 execution_inputs = ExecutionInputs( 1211 connector_under_test=connector, 1212 command=command, 1213 output_dir=output_dir, 1214 config=config, 1215 configured_catalog=configured_catalog, 1216 state=state, 1217 environment_variables=container_env or None, 1218 ) 1219 1220 mask_policy = _mask_policy(mask_catalog, output_mode, mask_key) 1221 runner = ConnectorRunner( 1222 execution_inputs, 1223 timeout_seconds=( 1224 DEFAULT_TIMEOUT_SECONDS if timeout_seconds is None else timeout_seconds 1225 ), 1226 proxy_url=proxy_url, 1227 ca_cert_path=ca_cert_path, 1228 max_records=max_records, 1229 mask_policy=mask_policy, 1230 ) 1231 result = runner.run() 1232 1233 result.save_artifacts(output_dir) 1234 1235 result_dict: dict[str, object] = { 1236 "connector": connector_image, 1237 "command": command.value, 1238 "success": result.success, 1239 "exit_code": result.exit_code, 1240 "stdout_file": str(result.stdout_file_path), 1241 "stderr_file": str(result.stderr_file_path), 1242 "message_counts": { 1243 k.value: v for k, v in result.get_message_count_per_type().items() 1244 }, 1245 "record_counts_per_stream": result.get_record_count_per_stream(), 1246 "stopped_early": ( 1247 result.stopped_early.to_dict() if result.stopped_early else None 1248 ), 1249 "output_mode": output_mode, 1250 } 1251 if mask_policy is not None and command.needs_catalog(): 1252 # Per stream, what the mask left readable and why, for the report. Only 1253 # a command that reads records has a schema-enriched catalog and output 1254 # to keep readable; anywhere else every stream reads `schema missing`. 1255 result_dict["readable_fields"] = { 1256 name: info.to_dict() for name, info in mask_policy.streams.items() 1257 } 1258 1259 # Under `http_metrics` because that is what they are, but set here rather 1260 # than in `_run_with_optional_http_metrics`: the connector's own cache is 1261 # measurable with or without the proxy, so gating it on the proxy would 1262 # hide the one metric that needs none. The wrapper merges its proxy 1263 # counters into whatever is already here. Read off the summary that 1264 # `save_artifacts` collected above, not off a second walk of stdout: a 1265 # `read`'s stdout is the dataset. 1266 connector_cache = parse_connector_cache_metrics(result.summary.analytics) 1267 if connector_cache is not None: 1268 result_dict["http_metrics"] = { 1269 "connector_request_count": connector_cache.request_count, 1270 "connector_cache_hits_count": connector_cache.cache_hits_count, 1271 "connector_cache_hit_ratio": connector_cache.cache_hit_ratio, 1272 } 1273 1274 # For CHECK, include the parsed connectionStatus from the Airbyte protocol. 1275 # The CDK exits 0 regardless of check outcome; the real pass/fail signal 1276 # lives in the CONNECTION_STATUS message (status=SUCCEEDED or FAILED). 1277 if command == Command.CHECK: 1278 conn_status = result.get_connection_status() 1279 if conn_status is not None: 1280 result_dict["connection_status"] = conn_status.status.value 1281 result_dict["connection_status_message"] = conn_status.message or "" 1282 else: 1283 result_dict["connection_status"] = None 1284 result_dict["connection_status_message"] = "" 1285 1286 return result_dict, ComparableOutputs.from_execution_result(result) 1287 1288 1289def _read_not_started( 1290 connector_image: str, 1291 command: Command, 1292 output_dir: Path, 1293 output_mode: OutputMode, 1294) -> tuple[dict, ComparableOutputs]: 1295 """The result of a side whose deadline passed before its turn came. 1296 1297 No container runs: a read given no time would only push the comparison and 1298 the report past the step's own timeout. The side reads like any other run 1299 stopped at its deadline -- a clean exit, `stopped_early` with reason 1300 `timeout` and no records -- so the verdict is incomplete, never a pass, 1301 and the report is still written. Its output files exist, empty but for a 1302 note in stderr, so every reader of a run's files finds them, and it 1303 carries the run's `output_mode`, so the report still says how the run's 1304 output was written. 1305 """ 1306 output_dir.mkdir(parents=True, exist_ok=True) 1307 stdout_path = output_dir / "stdout.txt" 1308 stderr_path = output_dir / "stderr.txt" 1309 stdout_path.write_text("") 1310 note = ( 1311 f"{connector_image} was not started: nothing was left of --read-timeout " 1312 "for its read" 1313 ) 1314 stderr_path.write_text(f"{note}\n") 1315 print_warning(note) 1316 return ( 1317 { 1318 "connector": connector_image, 1319 "command": command.value, 1320 "success": True, 1321 "exit_code": 0, 1322 "stdout_file": str(stdout_path), 1323 "stderr_file": str(stderr_path), 1324 "message_counts": {}, 1325 "record_counts_per_stream": {}, 1326 "stopped_early": StoppedEarly( 1327 reason="timeout", records_seen=0, elapsed_s=0.0 1328 ).to_dict(), 1329 "output_mode": output_mode, 1330 }, 1331 ComparableOutputs(), 1332 ) 1333 1334 1335def _record_read_share(result: dict, share_s: int | None, budget_s: int) -> None: 1336 """Name the share of `--read-timeout` a side stopped at, for its report line. 1337 1338 Only on a deadline stop under the budget: a capped side stopped for 1339 another reason, and a side that finished needs no excuse. `share_s` 0 is a 1340 side that never started. A copy replaces the entry rather than adding to 1341 it, since the dict may be shared with whoever produced it. 1342 """ 1343 stopped = result.get("stopped_early") 1344 if share_s is None or not stopped or stopped.get("reason") != "timeout": 1345 return 1346 result["stopped_early"] = {**stopped, "share_s": share_s, "budget_s": budget_s} 1347 1348 1349def _annotate_run_verdict(command: str, result: dict[str, Any]) -> dict[str, Any]: 1350 """Copy a run result with `success` as the verdict, exit status kept separately. 1351 1352 `ExecutionResult.success` is the container exit status, which for `check` says 1353 nothing about whether the connector could connect -- the CDK exits 0 even when a 1354 check fails. Any payload read as the verdict must therefore not expose the exit 1355 status under `success`, at any nesting level. The raw value stays available as 1356 `exited_cleanly`; for every command but `check` the two are equal -- except on 1357 a side the harness stopped early, including one it never started for lack of 1358 time: that side read part of what it would have, so its `success` is false, 1359 as the run's is, and `stopped_early` says why. 1360 1361 Returns a copy: callers pass the unannotated originals to the report generator, 1362 which reads `success` as the exit status. 1363 """ 1364 return { 1365 **result, 1366 "success": ( 1367 command_result_succeeded(command, result) 1368 and not result.get("stopped_early") 1369 ), 1370 "exited_cleanly": result["success"], 1371 } 1372 1373 1374# The lists in the run's JSON payload that hold record values: missing and 1375# duplicated primary keys, and the field diffs of paired records. Named alike, 1376# so a new example list is covered without a change here. 1377_EXAMPLE_LIST_SUFFIX = "_examples" 1378# What a raw run's printed payload says in place of a stream's state diff. 1379_STATE_DIFF_OMITTED = ( 1380 "Omitted from the job log on a raw run; both versions' state messages are " 1381 "in the saved artifacts" 1382) 1383 1384 1385def _without_examples(payload: dict[str, Any]) -> dict[str, Any]: 1386 """A copy of the payload with every example list emptied, keys kept, and 1387 each stream's final-state diff replaced by a `diff_omitted` note. 1388 1389 The job log outlives the artifacts and is not masked, so on a raw run the 1390 printed payload carries the shape and the counts of what was found and 1391 none of the record values; `examples_omitted` says why the lists are 1392 empty. The state diff is the one diff that holds customer values rather 1393 than types: a moved cursor puts both sides' values, and any partition 1394 value, in its `values_changed` entry, where catalog and spec diffs hold 1395 schema. The `regression_report` output and the artifacts keep the full 1396 payload: the artifact is where the detail belongs. 1397 """ 1398 1399 def strip(value: Any, key: str = "") -> Any: 1400 if isinstance(value, dict): 1401 return {name: strip(item, name) for name, item in value.items()} 1402 if isinstance(value, list): 1403 if key.endswith(_EXAMPLE_LIST_SUFFIX): 1404 return [] 1405 return [strip(item) for item in value] 1406 return value 1407 1408 stripped = strip(payload) 1409 final_state = stripped.get("comparison", {}).get("final_state") 1410 if isinstance(final_state, dict): 1411 for entry in final_state.get("streams", {}).values(): 1412 if isinstance(entry, dict) and "diff" in entry: 1413 del entry["diff"] 1414 entry["diff_omitted"] = _STATE_DIFF_OMITTED 1415 return {**stripped, "examples_omitted": True} 1416 1417 1418def _printable_payload( 1419 payload: dict[str, Any], output_mode: OutputMode 1420) -> dict[str, Any]: 1421 """The payload as `print_json` may show it: whole on a masked run, where 1422 every example is a token, and without its examples on a raw run.""" 1423 return payload if output_mode == "masked" else _without_examples(payload) 1424 1425 1426# The checks the output comparisons contribute, named for the row a reviewer 1427# reads in the report and in the step summary. 1428_SPEC_CHECK = "Spec compatibility" 1429_CATALOG_SCHEMA_CHECK = "Catalog schema" 1430_FINAL_STATE_CHECK = "Final state per stream" 1431 1432# How many individual changes a check's one-line summary names before it says 1433# how many more there are. The full list is in the report and in the run's JSON 1434# payload, and the comparators order their findings worst-first, so the ones 1435# named here are the ones worth naming. 1436_MAX_LISTED_CHANGES = 3 1437 1438# How many changes the JSON payload lists per comparison. Same reason as 1439# `_MAX_DIFF_PAYLOAD_CHARS`: the payload is written to `GITHUB_OUTPUT`, and a 1440# reordered `oneOf` on a wide spec produces hundreds of sentences. 1441_MAX_PAYLOAD_CHANGES = 100 1442 1443# How many per-stream entries it lists. A wide source has thousands of streams, 1444# and one line each is enough to exceed the same output budget without a single 1445# diff being attached. 1446_MAX_PAYLOAD_STREAMS = 100 1447 1448# How much of a comparison's detail the run's JSON payload carries. That payload 1449# is written to `GITHUB_OUTPUT`, which is size-limited, and a wide connector's 1450# catalog diff has no natural bound -- so the payload keeps as many stream diffs 1451# as fit and says how many it dropped. The catalogs themselves are saved 1452# artifacts, which is where an unbounded diff belongs. 1453_MAX_DIFF_PAYLOAD_CHARS = 50_000 1454 1455 1456def _compare_outputs( 1457 *, 1458 command: str, 1459 outcome: ComparisonOutcome, 1460 control_outputs: ComparableOutputs, 1461 target_outputs: ComparableOutputs, 1462 partial: bool = False, 1463 no_control_streams: tuple[str, ...] = (), 1464 could_not_test: bool = False, 1465 deferred_diff_blocks: list[Callable[[], tuple[DiffBlock, ...]]] | None = None, 1466) -> tuple[tuple[CheckResult, ...], tuple[DiffBlock, ...], dict[str, Any]]: 1467 """Compare what the two versions produced, for the commands that produce it. 1468 1469 This is the content check for `spec`, `discover` and `read`: without it their 1470 verdict is the exit code, and a connector that drops a config field, changes 1471 a stream's schema or corrupts the state the next sync resumes from exits 0 1472 all the way to a release. `check` reports its own status, so it is the one 1473 command with nothing to diff here. 1474 1475 Only runs when both commands succeeded. A failed run has nothing to compare, 1476 and reporting "no spec to compare" beside the command failure that caused it 1477 would state the same problem twice, in weaker terms the second time. 1478 1479 A comparator that raises is left to propagate. The comparators normalise what 1480 they are given, so the only ways here are a programming error in a future 1481 caller or a schema nested past the recursion limit -- and the workflow 1482 already has the right vocabulary for both: a step that dies is an 1483 `internal_failure`, reported as infrastructure rather than as a regression, 1484 with the connector's artifacts uploaded either way. Catching it here would 1485 only let a tooling bug read as "the new version broke its spec". 1486 1487 Args: 1488 command: The Airbyte command that was run, as the CLI names it. This is 1489 always `"read"` for a read, including one given a state file: the 1490 `READ` -> `READ_WITH_STATE` upgrade only changes the `Command` the 1491 container is run with. 1492 outcome: The already-evaluated command outcome. 1493 control_outputs: What the control version produced. 1494 target_outputs: What the target version produced. 1495 partial: Whether a side was stopped early. Only the final-state check 1496 is affected: the states of a stopped read are checkpoints from 1497 wherever each side got to, so the check is reported rather than 1498 folded. 1499 1500 Returns: 1501 The checks to fold into the verdict, the diff blocks that show what they 1502 found, and the comparison detail for the run's JSON payload -- all empty 1503 when there was nothing to compare. 1504 """ 1505 if not outcome.success and not could_not_test: 1506 return (), (), {} 1507 1508 if command == "spec": 1509 return _spec_comparison( 1510 control_outputs.spec, target_outputs.spec, deferred_diff_blocks 1511 ) 1512 1513 if command == "discover": 1514 return _catalog_comparison( 1515 control_outputs.catalog, target_outputs.catalog, deferred_diff_blocks 1516 ) 1517 1518 if command == "read": 1519 return _final_state_comparison( 1520 control_outputs.final_states, 1521 target_outputs.final_states, 1522 partial, 1523 no_control_streams, 1524 deferred_diff_blocks, 1525 ) 1526 1527 return (), (), {} 1528 1529 1530def _spec_comparison( 1531 control_spec: Any, 1532 target_spec: Any, 1533 deferred_diff_blocks: list[Callable[[], tuple[DiffBlock, ...]]] | None = None, 1534) -> tuple[tuple[CheckResult, ...], tuple[DiffBlock, ...], dict[str, Any]]: 1535 """Run the spec comparison and shape it for every surface.""" 1536 result = compare_specs(control_spec, target_spec) 1537 1538 check = CheckResult( 1539 name=_SPEC_CHECK, 1540 passed=result.passed, 1541 summary=_change_summary( 1542 result.message, 1543 # A passing spec check still says what was added: an additive change 1544 # is the thing a reviewer most wants to see confirmed. 1545 result.errors if result.errors else result.warnings, 1546 ), 1547 details=tuple(result.errors), 1548 ) 1549 # The incompatible changes are the check's `details`, which the report renders 1550 # under the check row and the step summary lists -- a block would repeat them. 1551 # The compatible ones have nowhere else to go, and a reviewer asking "what 1552 # else moved?" is asking about exactly those. 1553 blocks = _report_diff_blocks( 1554 lambda: tuple( 1555 block 1556 for block in ( 1557 _change_block("Spec — compatible changes", result.warnings, "add"), 1558 ) 1559 if block is not None 1560 ), 1561 deferred_diff_blocks, 1562 ) 1563 1564 return (check,), blocks, {"spec": _spec_detail(result)} 1565 1566 1567def _catalog_comparison( 1568 control_catalog: Any, 1569 target_catalog: Any, 1570 deferred_diff_blocks: list[Callable[[], tuple[DiffBlock, ...]]] | None = None, 1571) -> tuple[tuple[CheckResult, ...], tuple[DiffBlock, ...], dict[str, Any]]: 1572 """Run the discovered-catalog comparison and shape it for every surface.""" 1573 result = compare_catalog_schemas(control_catalog, target_catalog) 1574 1575 check = CheckResult( 1576 name=_CATALOG_SCHEMA_CHECK, 1577 passed=result.passed, 1578 # A catalog that only grew -- new streams, new fields, nothing removed 1579 # or retyped -- is reported and does not gate the release. Connectors 1580 # add streams constantly, and a check that goes red on every one of them 1581 # is a check people learn to click past. 1582 severity="diagnostic" if result.additive_only else "strict", 1583 summary=_change_summary(result.message, result.errors), 1584 details=tuple(result.errors), 1585 ) 1586 1587 # The findings themselves are the check's `details`, which the report renders 1588 # under the check row: a stream that is missing or new has no schema diff, so 1589 # that list is the only place it appears. 1590 blocks = _report_diff_blocks( 1591 lambda: _stream_diff_blocks( 1592 result, 1593 block_title="Schema diff", 1594 omission_title="Schema diffs omitted for size", 1595 artifact_hint=( 1596 "Both discovered catalogs are in the artifact, under " 1597 "airbyte_messages/catalog.jsonl." 1598 ), 1599 ), 1600 deferred_diff_blocks, 1601 ) 1602 1603 return (check,), blocks, {"catalog_schema": _catalog_schema_detail(result)} 1604 1605 1606def _final_state_comparison( 1607 control_states: dict[tuple[str | None, str], Any] | None, 1608 target_states: dict[tuple[str | None, str], Any] | None, 1609 partial: bool = False, 1610 no_control_streams: tuple[str, ...] = (), 1611 deferred_diff_blocks: list[Callable[[], tuple[DiffBlock, ...]]] | None = None, 1612) -> tuple[tuple[CheckResult, ...], tuple[DiffBlock, ...], dict[str, Any]]: 1613 """Run the final-state comparison and shape it for every surface.""" 1614 excluded = set(no_control_streams) 1615 compared_control = ( 1616 {key: value for key, value in control_states.items() if key[1] not in excluded} 1617 if control_states is not None 1618 else None 1619 ) 1620 compared_target = ( 1621 {key: value for key, value in target_states.items() if key[1] not in excluded} 1622 if target_states is not None 1623 else None 1624 ) 1625 result = compare_final_states(compared_control, compared_target) 1626 1627 # Structural findings first, then the value-only ones: the check lists 1628 # everything it found, and the summary quotes the front of that list. 1629 findings = [*result.errors, *result.warnings] 1630 1631 summary = _change_summary(result.message, findings) 1632 not_compared = sorted(no_control_streams) 1633 if not_compared: 1634 summary += ( 1635 f"; {len(not_compared)} stream(s) read by the target only were not " 1636 f"compared: {', '.join(not_compared)}" 1637 ) 1638 passed = result.passed 1639 if partial and not result.inconclusive: 1640 # Under a stop neither side emitted a *final* state: what each left 1641 # behind is the checkpoint it had reached, and the two were cut at 1642 # different points -- so a state present on one side only, a cursor 1643 # that differs, or two that agree, says where each stopped, not what 1644 # changed. Never a pass, for the same reason the count row is not. 1645 passed = False 1646 summary += ( 1647 " (inconclusive on a stopped-early read: these are the checkpoints " 1648 "each side had reached, not final states)" 1649 ) 1650 1651 check = CheckResult( 1652 name=_FINAL_STATE_CHECK, 1653 passed=passed, 1654 # Three ways a non-passing state check does not gate. A cursor that only 1655 # advanced: until HTTP replay lands, the two versions call the live API 1656 # in separate runs and a timestamp cursor moves between them on its own. 1657 # A read with no state on either side, where there was nothing to 1658 # compare rather than something wrong. And a read that was stopped 1659 # early, whose states are not final. 1660 severity=( 1661 "diagnostic" 1662 if partial or result.value_only or result.inconclusive 1663 else "strict" 1664 ), 1665 summary=summary, 1666 details=tuple(findings), 1667 ) 1668 1669 blocks = _report_diff_blocks( 1670 lambda: _stream_diff_blocks( 1671 result, 1672 block_title="Final state diff", 1673 omission_title="Final state diffs omitted for size", 1674 artifact_hint=( 1675 "Both versions' state messages are in the artifact, under " 1676 "airbyte_messages/state.jsonl." 1677 ), 1678 ), 1679 deferred_diff_blocks, 1680 ) 1681 1682 detail = _final_state_detail(result) 1683 detail["not_compared"] = not_compared 1684 return (check,), blocks, {"final_state": detail} 1685 1686 1687def _report_diff_blocks( 1688 build: Callable[[], tuple[DiffBlock, ...]], 1689 deferred: list[Callable[[], tuple[DiffBlock, ...]]] | None, 1690) -> tuple[DiffBlock, ...]: 1691 """Defer display-only diffs while keeping comparison checks and JSON eager.""" 1692 if deferred is not None: 1693 deferred.append(build) 1694 return () 1695 return build() 1696 1697 1698def _stream_diff_blocks( 1699 result: ComparisonResult, 1700 *, 1701 block_title: str, 1702 omission_title: str, 1703 artifact_hint: str, 1704) -> tuple[DiffBlock, ...]: 1705 """One block per changed stream, within the report's diff budget. 1706 1707 In comparison order -- only a *changed* stream has a diff, so there is no 1708 severity left to sort by here -- and within the same budget the payload uses: 1709 a report that inlines every schema of a 200-stream source is not a report 1710 anyone reads. What the budget drops is named rather than skipped, since a 1711 block that is simply absent reads as "there was nothing to see". 1712 """ 1713 blocks: list[DiffBlock] = [] 1714 budget = _MAX_DIFF_PAYLOAD_CHARS 1715 dropped: list[str] = [] 1716 1717 for name in result.failed_streams: 1718 stream = result.stream_results[name] 1719 if not stream.schema_diff: 1720 continue 1721 1722 payload = json.dumps(stream.schema_diff, indent=2, sort_keys=True) 1723 if len(payload) > budget: 1724 dropped.append(name) 1725 continue 1726 1727 budget -= len(payload) 1728 blocks.append(build_json_block(f"{block_title} — {name}", payload)) 1729 1730 if dropped: 1731 blocks.append( 1732 build_diff_block( 1733 omission_title, 1734 [ 1735 DiffLine( 1736 kind="meta", 1737 text=( 1738 "These streams changed, and their diffs were too " 1739 f"large for this report. {artifact_hint}" 1740 ), 1741 ), 1742 *(DiffLine(kind="del", text=name) for name in dropped), 1743 ], 1744 language="text", 1745 ) 1746 ) 1747 1748 return tuple(blocks) 1749 1750 1751def _change_block(title: str, changes: list[str], kind: str) -> DiffBlock | None: 1752 """One diff block listing a comparison's findings, or `None` if there are none.""" 1753 if not changes: 1754 return None 1755 1756 return build_diff_block( 1757 title, 1758 [DiffLine(kind=kind, text=change) for change in changes], 1759 language="text", 1760 ) 1761 1762 1763def _change_summary(message: str, changes: list[str]) -> str: 1764 """One line: what the comparison concluded, and the first few reasons.""" 1765 if not changes: 1766 return message 1767 1768 listed = "; ".join(changes[:_MAX_LISTED_CHANGES]) 1769 remaining = len(changes) - _MAX_LISTED_CHANGES 1770 if remaining > 0: 1771 listed = f"{listed}; and {remaining} more" 1772 1773 return f"{message}: {listed}" 1774 1775 1776def _bounded_changes(changes: list[str]) -> list[str]: 1777 """A change list the JSON payload can carry, saying what it left out.""" 1778 if len(changes) <= _MAX_PAYLOAD_CHANGES: 1779 return changes 1780 1781 dropped = len(changes) - _MAX_PAYLOAD_CHANGES 1782 1783 return [ 1784 *changes[:_MAX_PAYLOAD_CHANGES], 1785 f"... and {dropped} more, in the report and the console output", 1786 ] 1787 1788 1789def _spec_detail(result: ComparisonResult) -> dict[str, Any]: 1790 """The spec comparison, for the run's JSON payload.""" 1791 return { 1792 "passed": result.passed, 1793 "message": result.message, 1794 "breaking_changes": _bounded_changes(result.errors), 1795 "compatible_changes": _bounded_changes(result.warnings), 1796 } 1797 1798 1799def _catalog_schema_detail(result: ComparisonResult) -> dict[str, Any]: 1800 """The catalog comparison, for the run's JSON payload.""" 1801 streams, omitted, dropped = _bounded_stream_diffs( 1802 result, 1803 omission_note="Too large for this payload; diff the catalogs", 1804 artifact_hint="Both discovered catalogs are in the saved artifacts.", 1805 ) 1806 1807 detail: dict[str, Any] = { 1808 "passed": result.passed, 1809 # Why a `passed: false` did not fail the run: everything found was an 1810 # addition, which is reported as a warning rather than a regression. 1811 "additive_only": result.additive_only, 1812 "message": result.message, 1813 "changed_streams": _bounded_changes(result.failed_streams), 1814 "streams": streams, 1815 } 1816 if omitted: 1817 detail["streams_with_omitted_diffs"] = omitted 1818 if dropped: 1819 detail["streams_omitted"] = dropped 1820 1821 return detail 1822 1823 1824def _final_state_detail(result: ComparisonResult) -> dict[str, Any]: 1825 """The final-state comparison, for the run's JSON payload.""" 1826 streams, omitted, dropped = _bounded_stream_diffs( 1827 result, 1828 omission_note="Too large for this payload; diff the state messages", 1829 artifact_hint="Both versions' state messages are in the saved artifacts.", 1830 ) 1831 1832 detail: dict[str, Any] = { 1833 "passed": result.passed, 1834 # The two reasons a `passed: false` did not fail the run, kept apart so 1835 # the payload never says "every state that moved moved in value only" 1836 # about a read where nothing moved because nothing was compared. 1837 "value_only": result.value_only, 1838 "inconclusive": result.inconclusive, 1839 "message": result.message, 1840 "changed_streams": _bounded_changes(result.failed_streams), 1841 "streams": streams, 1842 } 1843 if omitted: 1844 detail["streams_with_omitted_diffs"] = omitted 1845 if dropped: 1846 detail["streams_omitted"] = dropped 1847 1848 return detail 1849 1850 1851def _bounded_stream_diffs( 1852 result: ComparisonResult, 1853 *, 1854 omission_note: str, 1855 artifact_hint: str, 1856) -> tuple[dict[str, Any], list[str], int]: 1857 """A comparison's per-stream verdicts and diffs, sized for the JSON payload. 1858 1859 Every stream that changed is named with its verdict -- so the payload says 1860 what moved, not just that something did -- while the diffs themselves are 1861 kept only while they fit in `_MAX_DIFF_PAYLOAD_CHARS`, and the number of 1862 entries is capped in turn: a thousand-stream source would otherwise put a 1863 thousand of them into `GITHUB_OUTPUT`. Both bounds say what they dropped, 1864 on the console and in the payload: a bound nobody mentions reads as "there 1865 was nothing more to see". 1866 """ 1867 streams: dict[str, Any] = {} 1868 omitted: list[str] = [] 1869 budget = _MAX_DIFF_PAYLOAD_CHARS 1870 1871 # Non-passing first, so the cap falls on the streams with nothing to say. 1872 ordered = sorted(result.stream_results.items(), key=lambda item: item[1].passed) 1873 kept, dropped_entries = ( 1874 ordered[:_MAX_PAYLOAD_STREAMS], 1875 ordered[_MAX_PAYLOAD_STREAMS:], 1876 ) 1877 1878 for name, stream in kept: 1879 entry: dict[str, Any] = {"passed": stream.passed, "message": stream.message} 1880 streams[name] = entry 1881 1882 if not stream.schema_diff: 1883 continue 1884 1885 rendered = len(json.dumps(stream.schema_diff, separators=(",", ":"))) 1886 if rendered > budget: 1887 omitted.append(name) 1888 entry["diff_omitted"] = omission_note 1889 continue 1890 1891 budget -= rendered 1892 entry["diff"] = stream.schema_diff 1893 1894 if omitted: 1895 print_warning( 1896 f"Comparison detail for {len(omitted)} streams was too large for the " 1897 f"run's JSON output ({', '.join(omitted)}); their diffs are omitted " 1898 f"there. {artifact_hint}" 1899 ) 1900 1901 if dropped_entries: 1902 print_warning( 1903 f"The run's JSON output lists {_MAX_PAYLOAD_STREAMS} of " 1904 f"{len(ordered)} streams, changed ones first. The report carries " 1905 f"them all. {artifact_hint}" 1906 ) 1907 1908 return streams, omitted, len(dropped_entries) 1909 1910 1911def _build_connector_image_from_source( 1912 connector_name: str, 1913 repo_root: Path | None = None, 1914 tag: str = "dev", 1915 *, 1916 prefer_repo_root: bool = False, 1917) -> str | None: 1918 """Build a connector image from source code. 1919 1920 Args: 1921 connector_name: Name of the connector (e.g., 'source-pokeapi'). 1922 repo_root: Optional path to the airbyte repo root. If not provided, 1923 will attempt to auto-detect from current directory. 1924 tag: Tag to apply to the built image (default: 'dev'). 1925 prefer_repo_root: Use `repo_root` even when auto-discovery would find 1926 a connector directory. Reproduction cases set it: auto-discovery 1927 searches from the working directory and can pick up a different 1928 airbyte checkout on the machine, silently building the wrong source. 1929 1930 Returns: 1931 The full image name with tag if successful, None if build fails. 1932 """ 1933 if not verify_docker_installation(): 1934 print_error("Docker is not installed or not running") 1935 return None 1936 1937 connector_directory: Path | None = None 1938 if not (prefer_repo_root and repo_root): 1939 with suppress(FileNotFoundError): 1940 connector_directory = find_connector_root_from_name(connector_name) 1941 if connector_directory is None: 1942 if repo_root: 1943 connector_directory = repo_root / CONNECTORS_SUBDIR / connector_name 1944 if not connector_directory.exists(): 1945 print_error(f"Connector directory not found: {connector_directory}") 1946 return None 1947 else: 1948 print_error( 1949 f"Could not find connector '{connector_name}'. " 1950 "Try providing --repo-root to specify the airbyte repo location." 1951 ) 1952 return None 1953 metadata_file_path = connector_directory / "metadata.yaml" 1954 if not metadata_file_path.exists(): 1955 print_error(f"metadata.yaml not found at {metadata_file_path}") 1956 return None 1957 1958 metadata = MetadataFile.from_file(metadata_file_path) 1959 print_success(f"Building image for connector: {connector_name}") 1960 1961 built_image = build_connector_image( 1962 connector_name=connector_name, 1963 connector_directory=connector_directory, 1964 metadata=metadata, 1965 tag=tag, 1966 no_verify=False, 1967 ) 1968 print_success(f"Successfully built image: {built_image}") 1969 return built_image 1970 1971 1972def _fetch_control_image_from_metadata(connector_name: str) -> str | None: 1973 """Fetch the current released connector image from metadata.yaml on main branch. 1974 1975 This fetches the connector's metadata.yaml from the airbyte monorepo's master branch 1976 and extracts the dockerRepository and dockerImageTag to construct the control image. 1977 1978 Args: 1979 connector_name: The connector name (e.g., 'source-github'). 1980 1981 Returns: 1982 The full connector image with tag (e.g., 'airbyte/source-github:1.0.0'), 1983 or None if the metadata could not be fetched or parsed. 1984 """ 1985 metadata_url = ( 1986 f"https://raw.githubusercontent.com/airbytehq/airbyte/master/" 1987 f"airbyte-integrations/connectors/{connector_name}/metadata.yaml" 1988 ) 1989 response = requests.get(metadata_url, timeout=30) 1990 if not response.ok: 1991 print_error( 1992 f"Failed to fetch metadata for {connector_name}: " 1993 f"HTTP {response.status_code} from {metadata_url}" 1994 ) 1995 return None 1996 1997 metadata = yaml.safe_load(response.text) 1998 if not isinstance(metadata, dict): 1999 print_error(f"Invalid metadata format for {connector_name}: expected dict") 2000 return None 2001 2002 data = metadata.get("data", {}) 2003 docker_repository = data.get("dockerRepository") 2004 docker_image_tag = data.get("dockerImageTag") 2005 2006 if not docker_repository or not docker_image_tag: 2007 print_error( 2008 f"Could not find dockerRepository/dockerImageTag in metadata for {connector_name}" 2009 ) 2010 return None 2011 2012 return f"{docker_repository}:{docker_image_tag}" 2013 2014 2015def _existing_http_metrics(result: dict[str, Any]) -> dict[str, object]: 2016 """The HTTP metrics a run already reported, for a payload that adds to them. 2017 2018 `_run_connector_command` sets the connector's own cache counters, which need 2019 no proxy; the proxy counters are added here afterwards. Overwriting rather 2020 than merging would silently drop the connector's numbers on every 2021 comparison run -- the runs most likely to be read. 2022 """ 2023 return dict(result.get("http_metrics") or {}) 2024 2025 2026def _run_with_optional_http_metrics( 2027 connector_image: str, 2028 command: Command, 2029 output_dir: Path, 2030 target_or_control: TargetOrControl, 2031 enable_http_metrics: bool, 2032 config_path: Path | None, 2033 catalog_path: Path | None, 2034 state_path: Path | None, 2035 enable_debug_logs: bool = False, 2036 extra_env: dict[str, str] | None = None, 2037 replay_expected: bool = False, 2038 replay_from: Path | None = None, 2039 replay_options: ReplayOptions | None = None, 2040 stream_large_bodies: str = DEFAULT_STREAM_LARGE_BODIES, 2041 timeout_seconds: int | None = None, 2042 max_records: int | None = None, 2043 output_mode: OutputMode = "raw", 2044 mask_key: bytes | None = None, 2045 mask_catalog_path: Path | None = None, 2046) -> tuple[dict, ComparableOutputs]: 2047 """Run a connector command with optional HTTP metrics capture. 2048 2049 When enable_http_metrics is True, starts mitmproxy to capture HTTP traffic. 2050 If mitmproxy fails to start, or the image cannot trust its CA (see 2051 `proxy_trust_gap`), falls back to running without metrics. 2052 2053 Args: 2054 connector_image: Full connector image name with tag. 2055 command: The Airbyte command to run. 2056 output_dir: Directory to store output files. 2057 target_or_control: Whether this is target or control version. 2058 enable_http_metrics: Whether to capture HTTP metrics via mitmproxy. 2059 config_path: Path to connector config JSON file. 2060 catalog_path: Path to configured catalog JSON file. 2061 state_path: Path to state JSON file. 2062 enable_debug_logs: If `True`, pass `LOG_LEVEL=DEBUG` to the connector container. 2063 extra_env: Additional environment variables for the connector container. 2064 replay_expected: Whether this run was *meant* to replay. Separate from 2065 `replay_from`, which is only non-None once a recording actually 2066 exists -- when the proxy is unavailable there is no recording to 2067 point at, and that is exactly the case that has to be reported. 2068 replay_from: A previous run's HTTP dump to serve responses from, so this 2069 run sees the same upstream data the recorded one did. Requires 2070 `enable_http_metrics`; ignored without it. 2071 replay_options: Replay tuning; defaults apply when omitted. 2072 stream_large_bodies: Body-size threshold above which mitmproxy streams 2073 rather than records a body. 2074 timeout_seconds: Passed through to `_run_connector_command`. At 0 the 2075 deadline has already passed, and neither the proxy nor the 2076 container starts (see `_read_not_started`). 2077 max_records: Passed through to `_run_connector_command`. 2078 output_mode: Passed through to `_run_connector_command`. 2079 mask_key: Passed through to `_run_connector_command`. 2080 mask_catalog_path: Passed through to `_run_connector_command`. 2081 2082 Returns: 2083 What `_run_connector_command` returns, with `http_metrics` added to the 2084 results dict when they were captured -- plus `http_dump_file`, the dump 2085 this run recorded, which is what a subsequent run replays from. 2086 """ 2087 if timeout_seconds is not None and timeout_seconds <= 0: 2088 return _read_not_started(connector_image, command, output_dir, output_mode) 2089 2090 # An image the proxy's CA cannot be handed to runs direct, as with capture 2091 # off: through the proxy each of its HTTPS calls would fail verification, 2092 # which reads as a connector bug. Its requests then read as not captured. 2093 capture_gap = proxy_trust_gap(connector_image) if enable_http_metrics else None 2094 if capture_gap: 2095 print_warning(f"Running without HTTP capture: {capture_gap}") 2096 if not enable_http_metrics or capture_gap: 2097 result, comparable = _run_connector_command( 2098 connector_image=connector_image, 2099 command=command, 2100 output_dir=output_dir, 2101 target_or_control=target_or_control, 2102 config_path=config_path, 2103 catalog_path=catalog_path, 2104 state_path=state_path, 2105 enable_debug_logs=enable_debug_logs, 2106 timeout_seconds=timeout_seconds, 2107 max_records=max_records, 2108 output_mode=output_mode, 2109 mask_key=mask_key, 2110 mask_catalog_path=mask_catalog_path, 2111 extra_env=extra_env, 2112 ) 2113 if capture_gap: 2114 skipped = {"capture_skipped_reason": capture_gap} 2115 if replay_expected: 2116 skipped["replay_skipped_reason"] = ( 2117 f"{capture_gap}, so nothing could be replayed" 2118 ) 2119 result["http_metrics"] = {**_existing_http_metrics(result), **skipped} 2120 return result, comparable 2121 2122 # The manager rather than `MitmproxyManager.start`: the reason a startup 2123 # failed and the fact that a shutdown had to be killed are only reachable 2124 # through it -- there is no session to carry the first, and the second is 2125 # not known until the session is gone. 2126 manager = MitmproxyManager( 2127 output_dir, 2128 replay_from=replay_from, 2129 replay_options=replay_options, 2130 stream_large_bodies=stream_large_bodies, 2131 ) 2132 with manager.running() as session: 2133 if session is None: 2134 startup_failure = ( 2135 manager.startup_failure_reason or "mitmproxy is unavailable" 2136 ) 2137 print_error( 2138 f"Mitmproxy unavailable ({startup_failure}), " 2139 "running without HTTP metrics" 2140 ) 2141 result, comparable = _run_connector_command( 2142 connector_image=connector_image, 2143 command=command, 2144 output_dir=output_dir, 2145 target_or_control=target_or_control, 2146 config_path=config_path, 2147 catalog_path=catalog_path, 2148 state_path=state_path, 2149 enable_debug_logs=enable_debug_logs, 2150 timeout_seconds=timeout_seconds, 2151 max_records=max_records, 2152 output_mode=output_mode, 2153 mask_key=mask_key, 2154 mask_catalog_path=mask_catalog_path, 2155 extra_env=extra_env, 2156 ) 2157 if replay_expected: 2158 # The one skip path that produces no session, and so reported 2159 # nothing at all: no metrics, no reason, an ordinary green 2160 # comparison against live data. Keyed on `replay_expected` and 2161 # not on `replay_from`: when mitmdump is missing the *control* 2162 # run records no dump either, so there is nothing to point at 2163 # and the condition that looks natural here is never true in 2164 # the failure it defends against. 2165 # 2166 # The precise reason, not a fixed "mitmproxy is unavailable": 2167 # `wait_for_proxy_listener` distinguishes a proxy that died on 2168 # startup and a corpus that took longer than its ceiling to 2169 # load, and those point at a code fix where the hardcoded 2170 # string tells a reader to install mitmproxy. 2171 result["http_metrics"] = { 2172 **_existing_http_metrics(result), 2173 "replay_skipped_reason": ( 2174 f"{startup_failure}, so nothing could be replayed" 2175 ), 2176 } 2177 return result, comparable 2178 2179 if session.replay_source: 2180 print_success( 2181 f"Started mitmproxy on {session.proxy_url} " 2182 f"(replaying {session.replay_source}, " 2183 f"{session.unreplayable_flow_count} flow(s) not replayable)" 2184 ) 2185 else: 2186 print_success(f"Started mitmproxy on {session.proxy_url}") 2187 if session.replay_skipped_reason: 2188 print_warning( 2189 f"HTTP replay unavailable ({session.replay_skipped_reason}); " 2190 "this run hits the live API, so it cannot be compared against " 2191 "identical upstream data" 2192 ) 2193 2194 result, comparable = _run_connector_command( 2195 connector_image=connector_image, 2196 command=command, 2197 output_dir=output_dir, 2198 target_or_control=target_or_control, 2199 config_path=config_path, 2200 catalog_path=catalog_path, 2201 state_path=state_path, 2202 proxy_url=session.proxy_url, 2203 ca_cert_path=session.ca_cert_path, 2204 enable_debug_logs=enable_debug_logs, 2205 timeout_seconds=timeout_seconds, 2206 max_records=max_records, 2207 output_mode=output_mode, 2208 mask_key=mask_key, 2209 mask_catalog_path=mask_catalog_path, 2210 extra_env=extra_env, 2211 ) 2212 2213 # The path, not the metrics: this is what the next run replays from. 2214 result["http_dump_file"] = str(session.dump_file_path) 2215 2216 # Everything the metrics are built from, carried out of the block. The 2217 # dump itself cannot be read here: mitmproxy's `save` addon writes each 2218 # flow through a block-buffered file and only flushes and closes it in 2219 # `done()`, holding in-flight flows until shutdown -- so parsing inside 2220 # the `with` misses the buffered tail, and `live_flow_count` is the 2221 # feature's acceptance criterion, not a rough count. 2222 dump_file_path = session.dump_file_path 2223 replay_source = session.replay_source 2224 replay_skipped_reason = session.replay_skipped_reason 2225 unreplayable_flow_count = session.unreplayable_flow_count 2226 2227 http_metrics = parse_http_dump(dump_file_path, replay_source) 2228 metrics_payload: dict[str, object] = { 2229 # Whatever the connector reported about its own cache, which 2230 # `_run_connector_command` set and this must not drop: the two are 2231 # different metrics of different things, and the report shows both. 2232 **_existing_http_metrics(result), 2233 "flow_count": http_metrics.flow_count, 2234 "duplicate_flow_count": http_metrics.duplicate_flow_count, 2235 # On a replaying target run, `live_flow_count` is the acceptance 2236 # criterion for replay -- so it belongs in the report rather than 2237 # only in the raw dump. `replay_hit_ratio` is what the legacy report 2238 # called its cache-hit ratio: the share mitmproxy served from the 2239 # recording, a property of the harness rather than of the connector. 2240 "replayed_flow_count": http_metrics.replayed_flow_count, 2241 "live_flow_count": http_metrics.live_flow_count, 2242 "replay_hit_ratio": http_metrics.replay_hit_ratio, 2243 } 2244 if replay_from is not None: 2245 # Keyed on replay having been *asked* for, not on a corpus having 2246 # survived: when every recorded flow is unreplayable there is no 2247 # corpus, and that is exactly the run whose dropped count matters. 2248 # Poor replay coverage is usually this, and the fix is a bigger 2249 # `--http-replay-max-body`. 2250 metrics_payload["unreplayable_flow_count"] = unreplayable_flow_count 2251 if replay_source: 2252 metrics_payload["replay_source"] = str(replay_source) 2253 # Only on a replaying run, where a live request is a request the corpus 2254 # could not answer and so the one thing that explains a coverage 2255 # shortfall. On a recording run every request is live, so the same list 2256 # would be a request log, which is not a surface to widen for no 2257 # diagnostic gain. Query-parameter values are masked either way -- these 2258 # URLs reach the step summary, the workflow log and `report.html`, and 2259 # plenty of APIs put credentials in the query string; see `redact_url`. 2260 live_urls = http_metrics.top_live_urls() 2261 if live_urls: 2262 metrics_payload["live_urls"] = live_urls 2263 metrics_payload["live_unique_url_count"] = len(http_metrics.live_url_counts) 2264 if manager.metrics_incomplete_reason: 2265 # Read after the block, not inside it: a killed shutdown is only known 2266 # once `_stop` has run. Recorded because every count below is then a 2267 # lower bound -- and short in the direction that makes a run look closer 2268 # to the replay acceptance criterion than it was. 2269 metrics_payload["metrics_incomplete"] = manager.metrics_incomplete_reason 2270 skipped_reason = replay_skipped_reason 2271 if replay_expected and replay_source is None and not skipped_reason: 2272 # The proxy came up but there was no recording to replay -- the 2273 # control run captured nothing. Same outcome as any other skip, so 2274 # it has to read the same way rather than as a clean replayed run. 2275 skipped_reason = "the control run recorded no HTTP dump to replay from" 2276 if skipped_reason: 2277 metrics_payload["replay_skipped_reason"] = skipped_reason 2278 result["http_metrics"] = metrics_payload 2279 2280 print_success( 2281 f"Captured {http_metrics.flow_count} HTTP flows " 2282 f"({http_metrics.duplicate_flow_count} duplicates)" 2283 ) 2284 if replay_source: 2285 print_success( 2286 f"Replayed {http_metrics.replayed_flow_count} of " 2287 f"{http_metrics.flow_count} request(s) " 2288 f"({http_metrics.replay_hit_ratio}); " 2289 f"{http_metrics.live_flow_count} went live" 2290 ) 2291 return result, comparable 2292 2293 2294def _discard_oversized_http_dumps(*results: dict, max_bytes: int) -> None: 2295 """Drop recorded dumps too large to keep, and say so in the report. 2296 2297 "Too large to keep" is about the runner's disk, not the artifact upload: 2298 dumps are excluded from every upload step regardless of size. 2299 2300 Deleting is not silent: the size that was discarded lands in the run's 2301 `http_metrics`, so a reader looking for the dump learns why it is not there 2302 instead of finding a dangling path. 2303 """ 2304 for result in results: 2305 dump_file = result.get("http_dump_file") 2306 if not dump_file: 2307 continue 2308 2309 discarded = discard_oversized_dump(Path(dump_file), max_bytes) 2310 if discarded is None: 2311 continue 2312 2313 result.pop("http_dump_file", None) 2314 metrics = result.get("http_metrics") 2315 if isinstance(metrics, dict): 2316 metrics["discarded_dump_mb"] = round(discarded / 1024 / 1024) 2317 print_warning( 2318 f"Discarded {dump_file}: {discarded / 1024 / 1024:.0f} MB exceeds the " 2319 "retention cap, and a dump this large is what fills the runner's disk. " 2320 "Dumps are never uploaded as artifacts; raise " 2321 "--http-replay-max-dump-size-mb to keep it for a local run" 2322 ) 2323 2324 2325def _compare_read_records( 2326 target_result: dict, 2327 control_result: dict, 2328 catalog_file: Path | None, 2329 target_record_files: dict[str, Path] | None = None, 2330) -> tuple[RecordComparisonSummary | None, str | None]: 2331 """Run record-level comparisons between control and target read outputs. 2332 2333 Uses primary keys from the configured catalog (source-defined primary 2334 keys come from the discovered catalog) to compare record counts, PK 2335 presence, PK integrity, and field values. Streams without a primary key 2336 fall back to a directional record count comparison. 2337 2338 Each run's stdout is first split into per-stream record files (a single 2339 streaming pass), and the comparison then takes one stream at a time. 2340 The per-stream files land next to each run's stdout and double as 2341 debugging artifacts. Within one stream the records stay on disk: each 2342 check reads them back from the file, and the field comparison keeps 2343 only a position and a fingerprint per record, loading a PK's records 2344 back only where the two versions' fingerprints differ. Peak memory is 2345 the largest stream's PK indexes -- a few hundred bytes per record -- 2346 not its records: a synthetic PayPal-shaped stream of 40k records 2347 (~127 MB a side) added ~35 MB to the process, where holding both sides 2348 had added ~2.8 GB. Start here when debugging an OOM. 2349 2350 Returns: 2351 `(summary, skip_reason)`. The summary is None when the comparison 2352 cannot run at all -- then `skip_reason` says why, so the caller can 2353 surface a diagnostic check instead of letting the run read as a pass 2354 that compared records. An unexpected error during the comparison 2355 itself returns a FAILED summary instead of raising: by this point 2356 both (expensive, live-API) runs have completed, and a comparison bug 2357 must fail the verdict, not destroy the run's outputs and reports. 2358 """ 2359 if catalog_file is None or not catalog_file.exists(): 2360 reason = "no configured catalog available" 2361 print_warning(f"Skipping record-level comparison: {reason}") 2362 return None, reason 2363 2364 try: 2365 configured_catalog = ConfiguredAirbyteCatalog.model_validate_json( 2366 catalog_file.read_text() 2367 ) 2368 except Exception as exc: 2369 reason = f"configured catalog could not be parsed ({exc})" 2370 print_warning(f"Skipping record-level comparison: {reason}") 2371 return None, reason 2372 2373 try: 2374 primary_keys = get_primary_keys_per_stream(configured_catalog) 2375 # A stream name that repeats across namespaces merges two streams in 2376 # this name-keyed pipeline; those names degraded to count-only above, 2377 # and the reader must hear why from the summary, not just stdout. 2378 collision_warnings = [ 2379 f"Stream name {name!r} appears in more than one namespace; the " 2380 "record comparison keys streams by name, so its records are " 2381 "compared by count only" 2382 for name in duplicate_stream_names(configured_catalog) 2383 ] 2384 control_record_files = split_records_per_stream( 2385 Path(control_result["stdout_file"]) 2386 ) 2387 target_record_files = target_record_files or split_records_per_stream( 2388 Path(target_result["stdout_file"]) 2389 ) 2390 2391 summary = run_record_comparisons_from_files( 2392 control_record_files=control_record_files, 2393 target_record_files=target_record_files, 2394 primary_keys_per_stream=primary_keys, 2395 # The two outputs end at different points when a side was 2396 # stopped, so counts and PK presence are reported, not folded. 2397 partial=bool( 2398 stopped_early_sides( 2399 ("control", control_result), ("target", target_result) 2400 ) 2401 ), 2402 ) 2403 summary.warnings[:0] = collision_warnings 2404 except Exception as exc: 2405 summary = RecordComparisonSummary( 2406 passed=False, 2407 errored=True, 2408 count_comparison=ComparisonResult( 2409 passed=False, message="Record comparison errored" 2410 ), 2411 errors=[f"Record comparison errored: {exc!r}"], 2412 ) 2413 2414 if summary.passed and summary.partial: 2415 print_success( 2416 "Record comparison passed on a stopped-early read (PK integrity and " 2417 "field values on shared PKs; counts and PK presence are diagnostic)" 2418 ) 2419 elif summary.passed: 2420 print_success("Record comparison passed (counts, PKs, field values)") 2421 else: 2422 print_error("Record comparison failed:") 2423 for error in summary.errors[:20]: 2424 print_error(f" - {error}") 2425 for warning in summary.warnings[:20]: 2426 print_warning(warning) 2427 2428 return summary, None 2429 2430 2431@connector_app.command(name="build-target-image") 2432def build_target_image( 2433 connector_name: Annotated[ 2434 str, 2435 Parameter(help="Connector to build from source (e.g., 'source-pokeapi')."), 2436 ], 2437 repo_root: Annotated[ 2438 str | None, 2439 Parameter( 2440 help="Path to the airbyte repo root, when it cannot be auto-detected." 2441 ), 2442 ] = None, 2443) -> None: 2444 """Build the image `regression-test --connector-name` would build, and nothing else. 2445 2446 Writes the image to the `image` GitHub output. CI builds here, in a job 2447 that holds no credentials, because a Java connector's build runs the pull 2448 request's own Gradle on the host. The job that holds the credentials loads 2449 the image and passes it to `regression-test` as `--test-image`. 2450 """ 2451 built_image = _build_connector_image_from_source( 2452 connector_name=connector_name, 2453 repo_root=Path(repo_root) if repo_root else None, 2454 tag="dev", 2455 ) 2456 if not built_image: 2457 exit_with_error(f"Failed to build image for {connector_name}") 2458 write_github_output("image", built_image) 2459 2460 2461def _iter_record_data(path: Path): 2462 with path.open() as records_file: 2463 for line in records_file: 2464 message = json.loads(line) 2465 record = message.get("record") 2466 data = record.get("data") if isinstance(record, dict) else None 2467 if isinstance(data, dict): 2468 yield data 2469 2470 2471def _record_count(path: Path) -> int: 2472 return sum(1 for _ in _iter_record_data(path)) 2473 2474 2475def _new_stream_record_checks( 2476 *, 2477 stream_names: tuple[str, ...], 2478 record_files: dict[str, Path], 2479 catalog_file: Path | None, 2480 target_outputs: ComparableOutputs, 2481 record_comparison: RecordComparisonSummary | None, 2482 masked: bool, 2483 mask_policy: MaskPolicy | None, 2484 stopped: bool, 2485) -> tuple[tuple[CheckResult, ...], dict[str, dict[str, Any]], bool]: 2486 configured_catalog = ( 2487 ConfiguredAirbyteCatalog.model_validate_json(catalog_file.read_text()) 2488 if catalog_file is not None and catalog_file.exists() 2489 else None 2490 ) 2491 configured = { 2492 item.stream.name: item 2493 for item in (configured_catalog.streams if configured_catalog else []) 2494 } 2495 primary_keys = ( 2496 get_primary_keys_per_stream(configured_catalog) if configured_catalog else {} 2497 ) 2498 stream_results: dict[str, dict[str, Any]] = {} 2499 schema_summaries: list[str] = [] 2500 pk_summaries: list[str] = [] 2501 cursor_summaries: list[str] = [] 2502 schema_failed = False 2503 pk_failed = False 2504 cursor_failed = False 2505 cursor_diagnostic = False 2506 schema_diagnostic = False 2507 regression_found = False 2508 has_pk = False 2509 has_incremental = False 2510 2511 for name in stream_names: 2512 path = record_files.get(name) 2513 if path is None: 2514 continue 2515 count = _record_count(path) 2516 if not count: 2517 continue 2518 item = configured.get(name) 2519 stream = item.stream if item else None 2520 result: dict[str, Any] = { 2521 "records": count, 2522 "partial": stopped, 2523 "schema_mismatched_records": None, 2524 "undeclared_field_records": None, 2525 "pk_null": None, 2526 "pk_duplicates": None, 2527 "null_cursor_records": None, 2528 "state_has_cursor_value": None, 2529 "state_below_max": None, 2530 "check_failed": False, 2531 } 2532 stream_results[name] = result 2533 try: 2534 if stream is None: 2535 raise ValueError("configured stream missing") 2536 schema_result = check_record_schema( 2537 _iter_record_data(path), stream.json_schema or {}, masked=masked 2538 ) 2539 result["schema_mismatched_records"] = schema_result.mismatched_records 2540 result["undeclared_field_records"] = schema_result.undeclared_field_records 2541 if schema_result.mismatched_records: 2542 result["check_failed"] = True 2543 schema_failed = True 2544 regression_found = True 2545 elif schema_result.undeclared_field_records: 2546 schema_diagnostic = True 2547 if schema_result.undeclared_field_records: 2548 schema_summaries.append( 2549 f"{name}: {schema_result.undeclared_field_records} record(s) " 2550 "have undeclared field(s)" 2551 ) 2552 properties = (stream.json_schema or {}).get("properties", {}) 2553 for field_name, mismatch_count in sorted( 2554 schema_result.mismatches_by_field.items() 2555 ): 2556 observed_types = schema_result.observed_types_by_field.get(field_name) 2557 observed = ", ".join(observed_types) if observed_types else "unknown" 2558 declared = declared_types(properties.get(field_name)) 2559 expected = ( 2560 "unconstrained" if declared is None else "|".join(sorted(declared)) 2561 ) 2562 schema_summaries.append( 2563 f"{name}.{field_name}: {mismatch_count} record(s), observed " 2564 f"{observed}, declared {expected}" 2565 ) 2566 if ( 2567 stopped 2568 and not schema_result.mismatched_records 2569 and not schema_result.undeclared_field_records 2570 ): 2571 schema_summaries.append( 2572 f"{name}: declared top-level types match on observed records" 2573 ) 2574 except Exception as exc: 2575 schema_failed = True 2576 result["check_failed"] = True 2577 schema_summaries.append(f"{name}: errored ({type(exc).__name__})") 2578 2579 pk_paths = primary_keys.get(name) 2580 if pk_paths: 2581 has_pk = True 2582 try: 2583 for pk_path in pk_paths: 2584 catalog_stream = stream.model_copy( # ty: ignore[unresolved-attribute] 2585 update={"source_defined_primary_key": [pk_path]} 2586 ) 2587 validation = validate_primary_keys_exist_in_schema( 2588 AirbyteCatalog(streams=[catalog_stream]) 2589 ) 2590 if not validation.passed: 2591 pk_failed = True 2592 result["check_failed"] = True 2593 regression_found = True 2594 pk_summaries.append( 2595 f"{name}: primary key {'.'.join(pk_path)} missing from schema" 2596 ) 2597 if record_comparison is not None: 2598 integrity = ( 2599 record_comparison.target_pk_uniqueness.stream_results.get(name) # ty: ignore[unresolved-attribute] 2600 ) 2601 presence = record_comparison.pk_presence.stream_results.get(name) # ty: ignore[unresolved-attribute] 2602 null_count = integrity.records_missing_pk if integrity else 0 2603 duplicate_count = integrity.duplicate_pk_count if integrity else 0 2604 presence_detail = ( 2605 "presence pass" 2606 if presence is None or presence.passed 2607 else "presence failed" 2608 ) 2609 else: 2610 integrity = _scan_pk_integrity(_read_records_file(path), pk_paths) 2611 null_count = integrity.records_missing_pk 2612 duplicate_count = len(integrity.duplicates) 2613 presence_detail = "presence not compared" 2614 result["pk_null"] = null_count 2615 result["pk_duplicates"] = duplicate_count 2616 pk_failed |= bool(null_count or duplicate_count) 2617 result["check_failed"] |= bool(null_count or duplicate_count) 2618 regression_found |= bool(null_count or duplicate_count) 2619 control_note = ( 2620 " (stream not read by control)" 2621 if record_comparison is not None and duplicate_count 2622 else "" 2623 ) 2624 pk_summaries.append( 2625 f"{name}: {null_count} null/missing PK record(s), " 2626 f"{duplicate_count} duplicate PK group(s){control_note}, " 2627 f"{presence_detail}" 2628 ) 2629 except Exception as exc: 2630 pk_failed = True 2631 result["check_failed"] = True 2632 pk_summaries.append(f"{name}: errored ({type(exc).__name__})") 2633 2634 cursor_path = ( 2635 list(item.cursor_field or []) 2636 if item and item.cursor_field 2637 else list(stream.default_cursor_field or []) 2638 if stream and stream.source_defined_cursor 2639 else [] 2640 ) 2641 sync_mode = ( 2642 item.sync_mode.value 2643 if item is not None and hasattr(item.sync_mode, "value") 2644 else item.sync_mode 2645 if item is not None 2646 else None 2647 ) 2648 incremental = bool(sync_mode == "incremental" and cursor_path) 2649 if incremental: 2650 has_incremental = True 2651 try: 2652 validation = validate_cursors_exist_in_schema( 2653 AirbyteCatalog(streams=[stream]), # ty: ignore[invalid-argument-type] 2654 cursor_paths_by_stream={name: cursor_path}, 2655 ) 2656 if not validation.passed: 2657 cursor_failed = True 2658 result["check_failed"] = True 2659 regression_found = True 2660 cursor_summaries.append( 2661 f"{name}: cursor {'.'.join(cursor_path)} missing from schema" 2662 ) 2663 state = (target_outputs.final_states or {}).get( 2664 (stream.namespace, stream.name) # ty: ignore[unresolved-attribute] 2665 ) 2666 cursor_readable = ( 2667 mask_policy.cursor_readable(name) 2668 if masked and mask_policy is not None 2669 else not masked 2670 ) 2671 cursor_result = check_incremental_cursor( 2672 _iter_record_data(path), 2673 cursor_path, 2674 state, 2675 cursor_readable=cursor_readable, 2676 stopped=stopped, 2677 ) 2678 result["null_cursor_records"] = cursor_result.null_cursor_records 2679 result["state_has_cursor_value"] = cursor_result.state_has_cursor_value 2680 result["state_below_max"] = cursor_result.state_below_max 2681 cursor_failed |= cursor_result.strict_failed 2682 result["check_failed"] |= cursor_result.strict_failed 2683 regression_found |= cursor_result.strict_failed 2684 diagnostics = [] 2685 if cursor_result.null_cursor_records: 2686 diagnostics.append( 2687 f"{cursor_result.null_cursor_records} null/missing cursor record(s)" 2688 ) 2689 if cursor_result.state_below_max: 2690 diagnostics.append("state below maximum record cursor") 2691 if cursor_result.cursor_key_not_found: 2692 diagnostics.append("cursor key not found in state") 2693 if diagnostics: 2694 cursor_diagnostic = True 2695 cursor_summaries.append( 2696 f"{name}: {'; '.join(diagnostics) if diagnostics else 'cursor checks passed'}" 2697 ) 2698 except Exception as exc: 2699 cursor_failed = True 2700 result["check_failed"] = True 2701 cursor_summaries.append(f"{name}: errored ({type(exc).__name__})") 2702 2703 checks: list[CheckResult] = [] 2704 if stream_results: 2705 checks.append( 2706 CheckResult( 2707 name="Record schema (new streams)", 2708 passed=not (schema_failed or schema_diagnostic), 2709 severity="diagnostic" 2710 if schema_diagnostic and not schema_failed 2711 else "strict", 2712 summary="; ".join(schema_summaries) 2713 or ( 2714 f"{len(stream_results)} new stream(s): declared top-level types match" 2715 ), 2716 ) 2717 ) 2718 if has_pk: 2719 checks.append( 2720 CheckResult( 2721 name="Primary key (new streams)", 2722 passed=not pk_failed, 2723 summary="; ".join(pk_summaries), 2724 ) 2725 ) 2726 if has_incremental: 2727 checks.append( 2728 CheckResult( 2729 name="Incremental cursor (new streams)", 2730 passed=not (cursor_failed or cursor_diagnostic), 2731 severity="diagnostic" 2732 if cursor_diagnostic and not cursor_failed 2733 else "strict", 2734 summary="; ".join(cursor_summaries), 2735 ) 2736 ) 2737 return tuple(checks), stream_results, regression_found 2738 2739 2740@connector_app.command(name="regression-test") 2741def regression_test( 2742 skip_compare: Annotated[ 2743 bool, 2744 Parameter( 2745 help="If True, skip comparison and run single-version tests only. " 2746 "If False (default), run comparison tests (target vs control)." 2747 ), 2748 ] = False, 2749 skip_record_comparison: Annotated[ 2750 bool, 2751 Parameter( 2752 help="If True, skip the record-level comparison for read commands " 2753 "(counts, PK presence, PK integrity, field values) and gate the " 2754 "verdict on command outcomes only. Escape hatch for sources whose " 2755 "data legitimately changes between the two live runs (rolling " 2756 "windows, feeds that age out records) until HTTP request caching " 2757 "makes strict comparison reliable." 2758 ), 2759 ] = False, 2760 test_image: Annotated[ 2761 str | None, 2762 Parameter( 2763 help="Test connector image with tag (e.g., airbyte/source-github:1.0.0). " 2764 "When provided with connector_name, this image takes precedence over " 2765 "building from source. In comparison mode, it's compared against control_image." 2766 ), 2767 ] = None, 2768 control_image: Annotated[ 2769 str | None, 2770 Parameter( 2771 help="Control connector image (baseline version) with tag (e.g., airbyte/source-github:1.0.0). " 2772 "Ignored if `skip_compare=True`." 2773 ), 2774 ] = None, 2775 connector_name: Annotated[ 2776 str | None, 2777 Parameter( 2778 help="Connector name to build image from source (e.g., 'source-pokeapi'). " 2779 "If provided without test_image, builds the image locally with tag 'dev'. " 2780 "When both are provided, test_image takes precedence and no build occurs. " 2781 "For comparison tests (default), this builds the target image. " 2782 "For single-version tests (skip_compare=True), this builds the test image." 2783 ), 2784 ] = None, 2785 repo_root: Annotated[ 2786 str | None, 2787 Parameter( 2788 help="Path to the airbyte repo root. Required if connector_name is provided " 2789 "and the repo cannot be auto-detected." 2790 ), 2791 ] = None, 2792 command: Annotated[ 2793 Literal["spec", "check", "discover", "read"], 2794 Parameter(help="The Airbyte command to run."), 2795 ] = "check", 2796 connection_id: Annotated[ 2797 str | None, 2798 Parameter( 2799 help="Airbyte Cloud connection ID to fetch config/catalog from. " 2800 "Mutually exclusive with config-path/catalog-path. " 2801 "If provided, test_image/control_image can be auto-detected." 2802 ), 2803 ] = None, 2804 config_path: Annotated[ 2805 str | None, 2806 Parameter(help="Path to the connector config JSON file."), 2807 ] = None, 2808 catalog_path: Annotated[ 2809 str | None, 2810 Parameter(help="Path to the configured catalog JSON file (required for read)."), 2811 ] = None, 2812 state_path: Annotated[ 2813 str | None, 2814 Parameter(help="Path to the state JSON file (optional for read)."), 2815 ] = None, 2816 output_dir: Annotated[ 2817 str, 2818 Parameter(help="Directory to store test artifacts."), 2819 ] = "/tmp/regression_test_artifacts", 2820 enable_http_metrics: Annotated[ 2821 bool, 2822 Parameter( 2823 help="Capture HTTP traffic via mitmproxy. Comparison mode always " 2824 "captures, with or without replay -- the per-version request counts " 2825 "come from the capture -- so this flag adds nothing there, and " 2826 "`--skip-compare` runs a single version without a proxy. Requires " 2827 "mitmdump to be installed; without it, and for an image with no " 2828 "certifi bundle to trust the proxy with (a Java or Kotlin " 2829 "connector), the run proceeds without metrics." 2830 ), 2831 ] = False, 2832 enable_http_replay: Annotated[ 2833 bool | None, 2834 Parameter( 2835 negative="--no-http-replay", 2836 help="Replay the control run's recorded HTTP responses for the target " 2837 "run, so both versions see identical upstream data. Defaults to " 2838 "`True` in comparison mode; has no effect with `--skip-compare`, " 2839 "which has no control run to replay from. Pass --no-http-replay to " 2840 "run both versions against the live API, which is the fallback " 2841 "whenever replay cannot be delivered anyway; the traffic is still " 2842 "captured, so the report keeps its per-version request counts.", 2843 ), 2844 ] = None, 2845 http_replay_max_body: Annotated[ 2846 str, 2847 Parameter( 2848 help="Response bodies larger than this are streamed rather than " 2849 "recorded, and therefore re-fetched live instead of replayed. Raise it " 2850 "for a connector with large pages, at the cost of replay memory. " 2851 "Accepts a mitmproxy size such as '1m' or '10m' -- never a bare number, " 2852 "which means bytes. Unlike the other --http-replay-* options this one " 2853 "governs recording, so it applies to every comparison, replay or not." 2854 ), 2855 ] = DEFAULT_STREAM_LARGE_BODIES, 2856 http_replay_max_dump_size_mb: Annotated[ 2857 int, 2858 Parameter( 2859 help="Skip replay and run the target live when the filtered replay " 2860 "corpus is larger than this, and discard a recorded dump this large " 2861 "once the run is done with it. mitmproxy holds the whole corpus in " 2862 "memory for the entire run, and a dump grows with everything the " 2863 "connector downloads, so this is what keeps a big connector from " 2864 "exhausting the CI runner's memory and disk. Dumps are never " 2865 "uploaded as artifacts, so on a local run -- where the dump is the " 2866 "only thing that can diagnose a coverage shortfall offline -- raise " 2867 "it to keep the dump." 2868 ), 2869 ] = DEFAULT_MAX_REPLAY_DUMP_SIZE_MB, 2870 http_replay_reuse: Annotated[ 2871 bool, 2872 Parameter( 2873 help="Serve a recorded response more than once. By default each " 2874 "recorded flow is consumed on first match, which preserves recorded " 2875 "ordering and lets memory fall as the run proceeds; reuse keeps the " 2876 "whole corpus resident for the entire run." 2877 ), 2878 ] = False, 2879 http_replay_ignore_params: Annotated[ 2880 str | None, 2881 Parameter( 2882 help="Comma-separated query parameters to exclude from the replay match " 2883 "key, for connectors that put a timestamp or nonce in the URL." 2884 ), 2885 ] = None, 2886 strict_replay: Annotated[ 2887 bool, 2888 Parameter( 2889 help="Kill requests that match nothing in the replay corpus instead of " 2890 "forwarding them upstream. Diagnostic only: it turns unmatched requests " 2891 "into connector errors, which is how you count them. mitmproxy stops " 2892 "intervening once its flowmap is empty, so combine this with " 2893 "--http-replay-reuse to cover the whole run rather than only the part " 2894 "before the corpus is consumed." 2895 ), 2896 ] = False, 2897 selected_streams: Annotated[ 2898 str | None, 2899 Parameter( 2900 help="Comma-separated list of stream names to include in the read. " 2901 "Only these streams will be included in the configured catalog. " 2902 "This is useful to limit data volume by testing only specific streams." 2903 ), 2904 ] = None, 2905 discover_missing_streams: Annotated[ 2906 bool, 2907 Parameter( 2908 help="Read selected streams the configured catalog does not have, such " 2909 "as a stream the PR adds: READ discovers them on the version under test " 2910 "and adds them to the catalog. A stream the control version also " 2911 "discovers is read by both versions and compared; one it does not " 2912 "discover has no control side and is read by the target only. Needs " 2913 "--selected-streams. A selected stream the target does not discover " 2914 "fails the run before READ, as a setup error." 2915 ), 2916 ] = False, 2917 injected_sync_mode: Annotated[ 2918 InjectedSyncMode | None, 2919 Parameter( 2920 help="Sync mode for streams added by --discover-missing-streams. " 2921 "Default: full refresh where the stream supports it. `incremental` " 2922 "uses the stream's default cursor and falls back to full refresh, " 2923 "with a note, for a stream that cannot be read incrementally." 2924 ), 2925 ] = None, 2926 enable_debug_logs: Annotated[ 2927 bool, 2928 Parameter( 2929 help="Enable debug-level logging for regression test output. " 2930 "Also passed as `LOG_LEVEL=DEBUG` to the connector Docker container. " 2931 "Switched off on a masked run for now: `--output-mode masked`, or a " 2932 "`--connection-id` run that leaves the mode unset." 2933 ), 2934 ] = False, 2935 extra_env: Annotated[ 2936 list[str] | None, 2937 Parameter( 2938 help=( 2939 "Additional KEY=VALUE environment variables for the connector " 2940 "containers. In comparison mode, applies to both target and " 2941 "control. Values are visible in the debug log of the `docker run` " 2942 "command; do not use for secrets. `LOG_LEVEL=DEBUG` or `TRACE` here " 2943 "counts as debug logs and is dropped on a masked run." 2944 ) 2945 ), 2946 ] = None, 2947 with_state: Annotated[ 2948 bool | None, 2949 Parameter( 2950 negative="--no-state", 2951 help="Fetch and pass the connection's current state to the read command, " 2952 "producing a warm read instead of a cold read. Defaults to `True` when " 2953 "`--connection-id` is provided, `False` otherwise. Has no effect unless " 2954 "the command is `read`. Ignored when `--state-path` is explicitly provided.", 2955 ), 2956 ] = None, 2957 read_timeout: Annotated[ 2958 int, 2959 Parameter( 2960 help="The read deadline in seconds. A read still running at its " 2961 "deadline is stopped, keeps what it had written, and is reported as " 2962 "stopped early. In comparison mode it is the budget for the whole " 2963 "`read` command, from its start to its report: the two versions read " 2964 "one after the other and share it, ten minutes (or a tenth of the " 2965 "budget, if less) is kept for the comparison and the report, the " 2966 "control may use up to 60% of the read time left when it starts, and " 2967 "the target gets the rest; a version with no time left does not " 2968 "start. In single-version mode it covers the command the same way, " 2969 "with the same reserve, and the one read gets the rest. Only " 2970 "applies to `read`. In CI set it below the step's own " 2971 "timeout, so the comparison and the report still run on the partial " 2972 "output instead of the step being killed with nothing to show." 2973 ), 2974 ] = DEFAULT_TIMEOUT_SECONDS, 2975 max_records: Annotated[ 2976 int | None, 2977 Parameter( 2978 help="Stop each side of the read when a RECORD message arrives past " 2979 "this many; the record past the cap is not kept, and a read of no " 2980 "more than this many is complete. Bounds a read of a large " 2981 "connection; a read the cap stops is a smoke test of the records " 2982 "read before the cap, not a full regression, and the report says so " 2983 "on each side that stopped early. Has no effect unless the command " 2984 "is `read`." 2985 ), 2986 ] = None, 2987 output_mode: Annotated[ 2988 Literal["masked", "raw"] | None, 2989 Parameter( 2990 help="How the connector output is written to disk. `masked` " 2991 "masks every record value as a type-and-length token in the write " 2992 "pass -- primary keys are masked and a cursor stays readable only " 2993 "when its declared type is an integer or a date; DEBUG, TRACE, ERROR " 2994 "and FATAL log messages, stack traces, error internals and every " 2995 "stderr line keep only their length, while INFO and WARN log messages, " 2996 "a TRACE error's message and the CONNECTION_STATUS message stay " 2997 "readable. `raw` keeps the output as the connector emitted it. " 2998 "Unset is `masked` on a `--connection-id` run, which reads a " 2999 "customer's data, and `raw` otherwise; pass `raw` to keep a " 3000 "connection run's output as emitted, or `masked` to mask a GSM or " 3001 "local run. Debug logs are off on a masked run for now: " 3002 "`--enable-debug-logs`, and `LOG_LEVEL=DEBUG` or `TRACE` in " 3003 "`--extra-env`, are dropped with a note and the run goes on." 3004 ), 3005 ] = None, 3006 mask_key_path: Annotated[ 3007 str | None, 3008 Parameter( 3009 help="A file holding the run's masking key as 64 hex characters. " 3010 "Every command of one run (spec, check, discover, read) is given " 3011 "the same path, so their tokens pair within the run and with no " 3012 "other run's; without it the CLI makes a fresh key for this " 3013 "process, so a run by hand pairs its own two sides and nothing " 3014 "else. The key is never printed or written to any output. Ignored " 3015 "on a raw run." 3016 ), 3017 ] = None, 3018 repro_case_dir: Annotated[ 3019 str | None, 3020 Parameter( 3021 help="Reproduction case directory (see `regression-case-setup`). Its " 3022 "catalog.json, state.json and expectations.json replace --catalog-path " 3023 "and --state-path, HTTP replay is turned off, and a read passes only " 3024 "when the control violates the case's expectations and the target " 3025 "satisfies them (see --repro-goal). Writes the verdict to " 3026 "`result.json` in the output directory. Requires `--command read`, " 3027 "--config-path, --sandbox-scenario and comparison mode; run the " 3028 "command under `sandbox exec`. Runs raw: `--output-mode masked` is " 3029 "refused." 3030 ), 3031 ] = None, 3032 repro_goal: Annotated[ 3033 Literal["fixed", "reproduced"] | None, 3034 Parameter( 3035 help="What a case read must show. `fixed` (default): the control " 3036 "violates the case and the target satisfies it. `reproduced`: both " 3037 "violate it -- the case gate's check, with the unmodified source of " 3038 "the affected release as the target. Requires --repro-case-dir." 3039 ), 3040 ] = None, 3041 sandbox_scenario: Annotated[ 3042 str | None, 3043 Parameter( 3044 help="SaaS Sandbox scenario UUID the case was set up in. For a read, " 3045 "its baseline snapshot is restored immediately before each side runs, " 3046 "so control and target start from identical data and failure rules." 3047 ), 3048 ] = None, 3049) -> None: 3050 """Run regression tests on connectors. 3051 3052 This command supports two modes: 3053 3054 Comparison mode (skip_compare=False, default): 3055 Runs the specified Airbyte protocol command against both the target (new) 3056 and control (baseline) connector versions, then compares the results. 3057 This helps identify regressions between versions. 3058 3059 Single-version mode (skip_compare=True): 3060 Runs the specified Airbyte protocol command against a single connector 3061 and validates the output. No comparison is performed. 3062 3063 Results are written to the output directory and to GitHub Actions outputs 3064 if running in CI. 3065 3066 You can provide the test image in three ways: 3067 1. --test-image: Use a pre-built image from Docker registry 3068 2. --connector-name: Build the image locally from source code 3069 3. --connection-id: Auto-detect from an Airbyte Cloud connection 3070 3071 You can provide config/catalog either via file paths OR via a connection_id 3072 that fetches them from Airbyte Cloud. 3073 """ 3074 # Before anything else: `--read-timeout` covers the whole command, so the 3075 # setup below spends it as surely as the reads do. 3076 read_budget = ReadBudget.start(read_timeout) 3077 parsed_extra_env = _parse_extra_env(extra_env) 3078 3079 output_path = Path(output_dir) 3080 output_path.mkdir(parents=True, exist_ok=True) 3081 3082 cmd = Command(command) 3083 3084 if read_timeout <= 0: 3085 write_github_output("success", False) 3086 write_github_output( 3087 "error", "--read-timeout must be a positive number of seconds" 3088 ) 3089 exit_with_error("--read-timeout must be a positive number of seconds") 3090 if max_records is not None and max_records < 1: 3091 write_github_output("success", False) 3092 write_github_output("error", "--max-records must be at least 1") 3093 exit_with_error("--max-records must be at least 1") 3094 if max_records is not None and cmd is not Command.READ: 3095 print_warning("--max-records only applies to `read`; ignoring it") 3096 max_records = None 3097 if max_records is not None: 3098 print_success( 3099 f"Capping each read at {max_records} record(s): a read that goes " 3100 "past them is stopped there, and its comparison covers only the " 3101 "records read before the cap" 3102 ) 3103 connection_residency: str | None = None 3104 if connection_id: 3105 try: 3106 connection_residency = get_connection_residency(connection_id) 3107 except _RESIDENCY_LOOKUP_ERRORS as exc: 3108 message = _describe_residency_lookup_error(connection_id, exc) 3109 write_github_output("success", False) 3110 write_github_output("error", message) 3111 exit_with_error(message) 3112 if connection_residency == "eu" and output_mode == "raw": 3113 message = ( 3114 "EU-resident connections always run masked; output_mode=raw is refused." 3115 ) 3116 write_github_output("success", False) 3117 write_github_output("error", message) 3118 exit_with_error(message) 3119 if connection_residency == "eu": 3120 _refuse_eu_connection_outside_eu_region(connection_id, connection_residency) 3121 output_mode = "masked" 3122 else: 3123 _refuse_non_eu_connection_in_eu_region(connection_id) 3124 if output_mode is None: 3125 # A connection reads a customer's data, so unset is masked there and 3126 # `raw` is the opt-in; a GSM or local config is the team's own. Kept 3127 # apart from an explicit value until here so the EU run path can 3128 # resolve an unset mode by residency. 3129 output_mode = "masked" if connection_id else "raw" 3130 if output_mode == "masked": 3131 print_success( 3132 "Output mode: masked (the default for a connection run); pass " 3133 "--output-mode raw to keep the output as emitted" 3134 ) 3135 if output_mode == "masked": 3136 parsed_extra_env, dropped = _without_debug_logs( 3137 enable_debug_logs=enable_debug_logs, extra_env=parsed_extra_env 3138 ) 3139 if dropped: 3140 enable_debug_logs = False 3141 print_warning( 3142 "Debug logs are off on masked runs; ignoring " + " and ".join(dropped) 3143 ) 3144 print_success( 3145 "Masking connector output: record values become tokens; primary keys " 3146 "are masked; only integer and date cursors stay readable" 3147 ) 3148 # The one key every policy of this invocation masks under, resolved before 3149 # anything is fetched so a bad key file fails the run in seconds. A raw 3150 # run has none, and does not read the file it may have been given. 3151 mask_key = _resolve_mask_key(mask_key_path) if output_mode == "masked" else None 3152 3153 # Configure debug logging for the regression test harness when requested 3154 if enable_debug_logs: 3155 logging.basicConfig( 3156 level=logging.DEBUG, 3157 format="%(asctime)s - %(name)s - %(levelname)s - %(message)s", 3158 force=True, 3159 ) 3160 3161 repro_case: ReproCase | None = None 3162 sandbox: SandboxCli | None = None 3163 if repro_case_dir or sandbox_scenario or repro_goal: 3164 case_error = _repro_case_argument_error( 3165 command=cmd, 3166 repro_case_dir=repro_case_dir, 3167 sandbox_scenario=sandbox_scenario, 3168 skip_compare=skip_compare, 3169 connection_id=connection_id, 3170 output_mode=output_mode, 3171 config_path=config_path, 3172 catalog_path=catalog_path, 3173 state_path=state_path, 3174 selected_streams=selected_streams, 3175 ) 3176 if case_error is None and repro_case_dir: 3177 try: 3178 repro_case = ReproCase.load(Path(repro_case_dir)) 3179 sandbox = SandboxCli() 3180 except (OSError, ValueError, SandboxError) as exc: 3181 case_error = f"Invalid reproduction case: {exc}" 3182 if case_error is None and repro_case is not None: 3183 case_error = _repro_case_command_error(cmd, repro_case.expectations.command) 3184 if case_error is not None: 3185 write_github_output("success", False) 3186 write_github_output("error", case_error) 3187 exit_with_error(case_error) 3188 assert repro_case is not None 3189 if repro_case.expectations.command == "read": 3190 catalog_path = str(repro_case.catalog_path) 3191 state_path = str(repro_case.state_path) if repro_case.state_path else None 3192 # The sandbox is the controlled upstream; replaying one side's traffic 3193 # to the other would bypass the per-side baseline restore. 3194 if enable_http_replay or enable_http_metrics: 3195 print_warning("HTTP replay and metrics are off for reproduction cases") 3196 enable_http_replay = False 3197 enable_http_metrics = False 3198 3199 # On by default, but only where it can work: single-version mode has no 3200 # control run to replay from. Asking for it there explicitly is a mistake 3201 # worth naming; the default landing there is not, so it is not a warning. 3202 if skip_compare: 3203 if enable_http_replay: 3204 print_warning("--enable-http-replay needs a control run; ignoring it") 3205 replay_enabled = False 3206 else: 3207 replay_enabled = True if enable_http_replay is None else enable_http_replay 3208 # Every comparison captures, replay or not: the per-version request 3209 # counts come from the capture, and a `--no-http-replay` run is the one 3210 # where they matter most. Replay stays a separate decision; a 3211 # reproduction case keeps the capture off (see above), and an image 3212 # that cannot trust the proxy runs without it. 3213 if repro_case is None: 3214 enable_http_metrics = True 3215 3216 if not replay_enabled and ( 3217 http_replay_reuse or strict_replay or http_replay_ignore_params 3218 ): 3219 # Otherwise someone debugging poor coverage with --strict-replay alone 3220 # concludes strict mode found nothing wrong. `--http-replay-max-body` 3221 # and `--http-replay-max-dump-size-mb` are deliberately not in this 3222 # list: the first governs recording and the second decides whether a 3223 # recorded dump is kept, and every comparison records. 3224 print_warning( 3225 "Replay tuning flags do nothing with --no-http-replay; ignoring them" 3226 ) 3227 3228 replay_options = ReplayOptions( 3229 reuse=http_replay_reuse, 3230 extra="kill" if strict_replay else DEFAULT_REPLAY_EXTRA, 3231 ignore_params=tuple( 3232 param.strip() 3233 for param in (http_replay_ignore_params or "").split(",") 3234 if param.strip() 3235 ), 3236 max_dump_bytes=http_replay_max_dump_size_mb * 1024 * 1024, 3237 ) 3238 3239 config_file: Path | None = None 3240 catalog_file: Path | None = None 3241 state_file = Path(state_path) if state_path else None 3242 connection_image: str | None = None 3243 3244 selected_stream_names: set[str] = ( 3245 {s.strip() for s in selected_streams.split(",") if s.strip()} 3246 if selected_streams 3247 else set() 3248 ) 3249 3250 # Resolve the test image (used in both single-version and comparison modes) 3251 resolved_test_image: str | None = test_image 3252 resolved_control_image: str | None = control_image 3253 3254 # Validate conflicting parameters 3255 # Single-version mode: reject comparison-specific parameters 3256 if skip_compare and control_image: 3257 write_github_output("success", False) 3258 write_github_output( 3259 "error", "Cannot specify control_image with skip_compare=True" 3260 ) 3261 exit_with_error( 3262 "Cannot specify --control-image with --skip-compare. " 3263 "Control image is only used in comparison mode." 3264 ) 3265 3266 built_test_image_locally = False 3267 if connector_name and not resolved_test_image: 3268 repo_root_path = Path(repo_root) if repo_root else None 3269 built_image = _build_connector_image_from_source( 3270 connector_name=connector_name, 3271 repo_root=repo_root_path, 3272 tag="dev", 3273 prefer_repo_root=repro_case is not None, 3274 ) 3275 if not built_image: 3276 write_github_output("success", False) 3277 write_github_output("error", f"Failed to build image for {connector_name}") 3278 exit_with_error(f"Failed to build image for {connector_name}") 3279 resolved_test_image = built_image 3280 built_test_image_locally = True 3281 3282 if connection_id: 3283 if config_path or catalog_path: 3284 write_github_output("success", False) 3285 write_github_output( 3286 "error", "Cannot specify both connection_id and file paths" 3287 ) 3288 exit_with_error( 3289 "Cannot specify both connection_id and config_path/catalog_path" 3290 ) 3291 3292 print_success(f"Fetching config/catalog from connection: {connection_id}") 3293 connection_data = fetch_connection_data(connection_id) 3294 3295 # Check if we should retrieve unmasked secrets 3296 if should_use_secret_retriever(): 3297 print_success( 3298 "USE_CONNECTION_SECRET_RETRIEVER enabled - enriching config with unmasked secrets..." 3299 ) 3300 try: 3301 connection_data = enrich_config_with_secrets( 3302 connection_data, 3303 retrieval_reason="Regression test with USE_CONNECTION_SECRET_RETRIEVER=true", 3304 ) 3305 print_success("Successfully retrieved unmasked secrets from database") 3306 except SecretRetrievalError as e: 3307 write_github_output("success", False) 3308 write_github_output("error", str(e)) 3309 exit_with_error( 3310 f"{e}\n\n" 3311 f"This connection cannot be used for regression testing. " 3312 f"Please use a connection from a non-EU workspace, or use GSM-based " 3313 f"integration test credentials instead (by omitting --connection-id)." 3314 ) 3315 3316 # Fill in stream schemas: the minimal catalog from the public API has 3317 # empty json_schema values, which silently yields zero records for 3318 # connectors that derive request fields from the catalog schema. 3319 # Only the read command consumes the catalog, so spec/check/discover 3320 # skip the extra Cloud API round trip. 3321 enrichment_state: list[dict[str, Any]] | None = None 3322 enrichment_state_fetched = False 3323 if cmd.needs_catalog(): 3324 merged_count = 0 3325 try: 3326 enrichment = enrich_catalog_schemas_from_cloud(connection_data) 3327 connection_data = enrichment.connection_data 3328 merged_count = enrichment.merged_count 3329 enrichment_state = enrichment.state 3330 enrichment_state_fetched = True 3331 except Exception as exc: 3332 print_error(f"Failed to fetch stream schemas from Cloud API: {exc}") 3333 missing_schemas = streams_missing_schemas(connection_data.catalog) 3334 if selected_stream_names: 3335 missing_schemas = [ 3336 s for s in missing_schemas if s in selected_stream_names 3337 ] 3338 if missing_schemas and should_use_secret_retriever(): 3339 try: 3340 connection_data, db_merged_count = ( 3341 enrich_catalog_schemas_from_platform_db( 3342 connection_data, 3343 retrieval_reason="Regression test catalog schema enrichment", 3344 ) 3345 ) 3346 merged_count += db_merged_count 3347 except Exception as exc: 3348 print_error(f"Failed to merge stored stream schemas: {exc}") 3349 missing_schemas = streams_missing_schemas(connection_data.catalog) 3350 if selected_stream_names: 3351 missing_schemas = [ 3352 s for s in missing_schemas if s in selected_stream_names 3353 ] 3354 total_streams = len(connection_data.catalog.get("streams", [])) 3355 checked_streams = ( 3356 f"{len(missing_schemas)} of {len(selected_stream_names)} " 3357 "selected stream(s)" 3358 if selected_stream_names 3359 else f"{len(missing_schemas)} of {total_streams} stream(s)" 3360 ) 3361 if not total_streams: 3362 error_msg = ( 3363 "No configured streams found in the connection catalog. " 3364 "A read with zero streams would produce zero records on " 3365 "both versions, so failing instead of producing a false pass." 3366 ) 3367 write_github_output("success", False) 3368 write_github_output("error", error_msg) 3369 exit_with_error(error_msg) 3370 if missing_schemas: 3371 error_msg = ( 3372 f"Stream schemas missing for {checked_streams}: " 3373 f"{', '.join(missing_schemas)}. " 3374 "Connectors that derive request fields from the catalog " 3375 "schema would silently return zero records, so failing " 3376 "instead of producing a false pass." 3377 ) 3378 write_github_output("success", False) 3379 write_github_output("error", error_msg) 3380 exit_with_error(error_msg) 3381 print_success( 3382 f"Stream schemas ready: merged {merged_count} of {total_streams} " 3383 "stream(s)" 3384 + ( 3385 f"; checked {len(selected_stream_names)} selected stream(s)" 3386 if selected_stream_names 3387 else "" 3388 ) 3389 ) 3390 3391 # Resolve with_state default: True when connection_id is provided 3392 resolved_with_state = with_state if with_state is not None else True 3393 3394 if not (resolved_with_state and not state_file and command == "read"): 3395 connection_data.state = None 3396 elif enrichment_state_fetched: 3397 connection_data.state = enrichment_state 3398 if connection_data.state: 3399 print_success( 3400 f"Using state fetched during schema enrichment " 3401 f"({len(connection_data.state)} stream(s))" 3402 ) 3403 else: 3404 print_success("No state available for this connection (initial sync)") 3405 else: 3406 print_success("Fetching connection state for warm read...") 3407 try: 3408 connection_data.state = fetch_connection_state(connection_data) 3409 if connection_data.state: 3410 print_success( 3411 f"Fetched state with {len(connection_data.state)} stream(s)" 3412 ) 3413 else: 3414 connection_data.state = None 3415 print_success( 3416 "No state available for this connection (initial sync)" 3417 ) 3418 except Exception as exc: 3419 print_error(f"Failed to fetch connection state: {exc}") 3420 print_success("Falling back to cold read (no state)") 3421 3422 # The output directory is uploaded whole as the run's artifact, so the 3423 # config -- the one object that carries the secrets -- is written to a 3424 # private directory outside it, as the GSM path below already does. A 3425 # masked run keeps the raw state there too: the artifact gets a copy 3426 # masked under the same rule as the records. 3427 config_file, catalog_file, state_file_from_conn = save_connection_data_to_files( 3428 connection_data, 3429 output_path / "connection_data", 3430 config_dir=_private_dir("connection-config-"), 3431 mask_policy=_mask_policy(connection_data.catalog, output_mode, mask_key), 3432 ) 3433 3434 if state_file_from_conn and not state_file: 3435 state_file = state_file_from_conn 3436 3437 print_success( 3438 f"Fetched config for source: {connection_data.source_name} " 3439 f"with {len(connection_data.stream_names)} streams" 3440 ) 3441 3442 connection_image = connection_data.connector_image 3443 3444 # Auto-detect test/control image from connection if not provided 3445 if not resolved_test_image and connection_data.connector_image: 3446 resolved_test_image = connection_data.connector_image 3447 print_success(f"Auto-detected test image: {resolved_test_image}") 3448 3449 if ( 3450 not skip_compare 3451 and not resolved_control_image 3452 and connection_data.connector_image 3453 ): 3454 resolved_control_image = connection_data.connector_image 3455 print_success(f"Auto-detected control image: {resolved_control_image}") 3456 elif config_path: 3457 config_file = Path(config_path) 3458 catalog_file = Path(catalog_path) if catalog_path else None 3459 elif connector_name: 3460 # Prefer the default config secret, falling back to other GSM secrets. 3461 print_success( 3462 f"No connection_id or config_path provided. " 3463 f"Attempting to fetch integration test config from GSM for {connector_name}..." 3464 ) 3465 gsm_unavailable_message = None 3466 try: 3467 gsm_config = get_integration_test_config_from_secrets(connector_name) 3468 except ConnectorSecretsUnavailableError as exc: 3469 gsm_unavailable_message = ( 3470 f"{exc} Or pass a connection_id to test against a customer connection." 3471 ) 3472 if cmd != Command.SPEC: 3473 write_github_output("success", False) 3474 write_github_output("error", gsm_unavailable_message) 3475 exit_with_error(gsm_unavailable_message) 3476 print_error(gsm_unavailable_message) 3477 gsm_config = None 3478 3479 if gsm_config: 3480 # Write config to a temp file (not in output_path to avoid artifact upload) 3481 gsm_config_dir = Path( 3482 tempfile.mkdtemp(prefix=f"gsm-config-{connector_name}-") 3483 ) 3484 gsm_config_dir.chmod(0o700) 3485 gsm_config_file = gsm_config_dir / "config.json" 3486 gsm_config_file.write_text(json.dumps(gsm_config.config, indent=2)) 3487 gsm_config_file.chmod(0o600) 3488 config_file = gsm_config_file 3489 # Use catalog_path if provided (e.g., generated from discover output) 3490 catalog_file = Path(catalog_path) if catalog_path else None 3491 print_success( 3492 f"Fetched integration test config from GSM secret " 3493 f"{gsm_config.secret_name} for {connector_name}" 3494 ) 3495 else: 3496 if gsm_unavailable_message is None: 3497 msg = ( 3498 f"No usable GSM integration-test secret was found for " 3499 f"{connector_name} in {DEFAULT_GCP_PROJECT_ID}. " 3500 f"The secret may be missing, have no enabled version, or contain " 3501 f"an empty placeholder (every value blank or null). Pass a " 3502 f"connection_id to test against a customer connection, or have " 3503 f"the connector's integration-test secret added or refreshed." 3504 ) 3505 if cmd != Command.SPEC: 3506 write_github_output("success", False) 3507 write_github_output("error", msg) 3508 exit_with_error(msg) 3509 print_error(msg) 3510 config_file = None 3511 # Use catalog_path if provided (e.g., generated from discover output) 3512 catalog_file = Path(catalog_path) if catalog_path else None 3513 else: 3514 config_file = None 3515 catalog_file = Path(catalog_path) if catalog_path else None 3516 3517 # Auto-detect control_image from metadata.yaml if connector_name is provided (comparison mode only) 3518 if not skip_compare and not resolved_control_image and connector_name: 3519 resolved_control_image = _fetch_control_image_from_metadata(connector_name) 3520 if resolved_control_image: 3521 print_success( 3522 f"Auto-detected control image from metadata.yaml: {resolved_control_image}" 3523 ) 3524 3525 # Validate that we have the required images 3526 if not resolved_test_image: 3527 write_github_output("success", False) 3528 write_github_output("error", "No test image specified") 3529 exit_with_error( 3530 "You must provide one of the following: a test_image, a connector_name " 3531 "to build the image from source, or a connection_id to auto-detect the image." 3532 ) 3533 3534 if not skip_compare and not resolved_control_image: 3535 write_github_output("success", False) 3536 write_github_output("error", "No control image specified") 3537 exit_with_error( 3538 "You must provide one of the following: a control_image, a connection_id " 3539 "for a connection that has an associated connector image, or a connector_name " 3540 "to auto-detect the control image from the airbyte repo's metadata.yaml." 3541 ) 3542 3543 # Pull images if they weren't just built locally 3544 if not built_test_image_locally and not ensure_image_available(resolved_test_image): 3545 write_github_output("success", False) 3546 write_github_output("error", f"Failed to pull image: {resolved_test_image}") 3547 exit_with_error(f"Failed to pull test image: {resolved_test_image}") 3548 3549 if ( 3550 not skip_compare 3551 and resolved_control_image 3552 and not ensure_image_available(resolved_control_image) 3553 ): 3554 write_github_output("success", False) 3555 write_github_output("error", f"Failed to pull image: {resolved_control_image}") 3556 exit_with_error( 3557 f"Failed to pull control connector image: {resolved_control_image}" 3558 ) 3559 3560 if discover_missing_streams and not selected_stream_names: 3561 error_msg = ( 3562 "--discover-missing-streams needs --selected-streams naming the " 3563 "streams to add to the catalog." 3564 ) 3565 write_github_output("success", False) 3566 write_github_output("error", error_msg) 3567 exit_with_error(error_msg) 3568 3569 # Streams the connection does not sync yet, typically one the PR adds, are 3570 # configured from DISCOVER. The two versions can then get different 3571 # catalogs: a stream with no control side goes to the target only. 3572 control_catalog_file = catalog_file 3573 injection: InjectedCatalogs | None = None 3574 if discover_missing_streams and cmd.needs_catalog(): 3575 injected = _inject_discovered_streams( 3576 catalog_file=catalog_file, 3577 selected_stream_names=selected_stream_names, 3578 test_image=resolved_test_image, 3579 reference_image=( 3580 resolved_control_image 3581 if not skip_compare 3582 else connection_image 3583 or ( 3584 _fetch_control_image_from_metadata(connector_name) 3585 if connector_name 3586 else None 3587 ) 3588 ), 3589 skip_compare=skip_compare, 3590 config_file=config_file, 3591 output_path=output_path, 3592 enable_debug_logs=enable_debug_logs, 3593 extra_env=parsed_extra_env, 3594 sync_mode=injected_sync_mode, 3595 output_mode=output_mode, 3596 mask_key=mask_key, 3597 read_budget=read_budget, 3598 ) 3599 if injected is not None: 3600 catalog_file, control_catalog_file, injection = injected 3601 3602 # Apply selected_streams filter to catalog if requested. Commands that read 3603 # no catalog skip it under --discover-missing-streams: the connection's 3604 # catalog does not have the streams being added, so the filter would fail 3605 # SPEC, CHECK and DISCOVER on a stream only READ will configure. 3606 if ( 3607 selected_stream_names 3608 and catalog_file 3609 and injection is None 3610 and not (discover_missing_streams and not cmd.needs_catalog()) 3611 ): 3612 print_success( 3613 f"Filtering catalog to {len(selected_stream_names)} selected streams: " 3614 f"{', '.join(sorted(selected_stream_names))}" 3615 ) 3616 filter_configured_catalog_file(catalog_file, selected_stream_names) 3617 3618 # A seed the catalog cannot apply would run as a cold read that the log 3619 # and the report still called seeded. A reproduction case brings its own 3620 # catalog and state, judged when the case is loaded. 3621 if cmd == Command.READ and state_path and repro_case is None: 3622 seed_error, unseeded_streams = _judge_seed_file(Path(state_path), catalog_file) 3623 if seed_error: 3624 write_github_output("success", False) 3625 write_github_output("error", seed_error) 3626 exit_with_error(seed_error) 3627 for stream_name in unseeded_streams: 3628 print_warning( 3629 f"The configured catalog does not sync '{stream_name}' " 3630 "incrementally, so the connector will ignore its seeded state" 3631 ) 3632 3633 # Upgrade READ → READ_WITH_STATE when a state file is available 3634 state_seeded = cmd == Command.READ and bool(state_path) 3635 if cmd == Command.READ and state_file: 3636 cmd = Command.READ_WITH_STATE 3637 if state_seeded: 3638 print_success( 3639 f"Using seeded state from {state_file} — read-with-state starts " 3640 "from its cursors, not the connection's" 3641 ) 3642 else: 3643 print_success("State available — using read-with-state (warm read)") 3644 3645 # Only a read runs long enough for the deadline to matter, and only a read 3646 # has partial output worth keeping when it does. 3647 read_timeout_seconds = ( 3648 read_timeout if cmd in (Command.READ, Command.READ_WITH_STATE) else None 3649 ) 3650 3651 # The report embeds the connection's input state; in a masked run it is 3652 # masked under the catalog the run uses and under the run's key, so its 3653 # tokens are the ones the records carry. A seeded state shows its outline 3654 # only (`_seed_outline`). 3655 report_mask_policy = _mask_policy( 3656 _load_json_file(catalog_file) if catalog_file else None, output_mode, mask_key 3657 ) 3658 3659 # Track telemetry for the regression test 3660 # Extract version from image tag (e.g., "airbyte/source-github:1.0.0" -> "1.0.0") 3661 target_version = ( 3662 resolved_test_image.rsplit(":", 1)[-1] 3663 if ":" in resolved_test_image 3664 else "unknown" 3665 ) 3666 control_version = None 3667 if resolved_control_image and ":" in resolved_control_image: 3668 control_version = resolved_control_image.rsplit(":", 1)[-1] 3669 3670 # Get tester identity from environment (GitHub Actions sets GITHUB_ACTOR) 3671 tester = os.getenv("GITHUB_ACTOR") or os.getenv("USER") 3672 3673 track_regression_test( 3674 user_id=tester, 3675 connector_image=resolved_test_image, 3676 command=command, 3677 target_version=target_version, 3678 control_version=control_version, 3679 additional_properties={ 3680 "connection_id": connection_id, 3681 "skip_compare": skip_compare, 3682 "with_state": cmd == Command.READ_WITH_STATE, 3683 "state_seeded": state_seeded, 3684 "max_records": max_records, 3685 "output_mode": output_mode, 3686 }, 3687 ) 3688 3689 # Execute the appropriate mode 3690 if skip_compare: 3691 # Single-version mode: run only the connector image. There is no control 3692 # to compare its spec or catalog against, so the declared objects go 3693 # unread here. Its read keeps the same reserve as a comparison's, so 3694 # the report is written before the step's own timeout. 3695 single_timeout_s = ( 3696 read_budget.read_time_left_s() if read_timeout_seconds is not None else None 3697 ) 3698 if single_timeout_s is not None and single_timeout_s <= 0: 3699 result, single_connector_outputs = _read_not_started( 3700 resolved_test_image, cmd, output_path, output_mode 3701 ) 3702 else: 3703 result, single_connector_outputs = _run_connector_command( 3704 connector_image=resolved_test_image, 3705 command=cmd, 3706 output_dir=output_path, 3707 target_or_control=TargetOrControl.TARGET, 3708 config_path=config_file, 3709 catalog_path=catalog_file, 3710 state_path=state_file, 3711 enable_debug_logs=enable_debug_logs, 3712 timeout_seconds=single_timeout_s, 3713 output_mode=output_mode, 3714 mask_key=mask_key, 3715 extra_env=parsed_extra_env, 3716 max_records=max_records, 3717 ) 3718 _record_read_share(result, single_timeout_s, read_timeout) 3719 3720 configured_streams = _configured_stream_names(cmd, catalog_file) 3721 report_model = build_single_version_report_model( 3722 connector_image=resolved_test_image, 3723 command=command, 3724 result=result, 3725 configured_streams=configured_streams, 3726 target_only_streams=( 3727 tuple(sorted((*injection.new_streams, *injection.kept_target_only))) 3728 if injection is not None 3729 else () 3730 ), 3731 stream_statuses=single_connector_outputs.stream_statuses, 3732 stream_errors=single_connector_outputs.stream_errors, 3733 include_report_sections=False, 3734 ) 3735 3736 # Same per-version rule as comparison mode: for CHECK the exit code alone 3737 # is not the verdict, connectionStatus is. 3738 annotated_result = _annotate_run_verdict(command, result) 3739 incomplete = report_model.incomplete 3740 single_success = report_model.passed 3741 annotated_result.update(report_model.verdict_flags()) 3742 3743 print_json(_printable_payload(annotated_result, output_mode)) 3744 3745 single_outputs: dict[str, object] = { 3746 **report_model.verdict_flags(github=True), 3747 "connector": resolved_test_image, 3748 "command": command, 3749 "exit_code": result["exit_code"], 3750 "output_mode": output_mode, 3751 } 3752 if command == "check": 3753 # Named to match the comparison-mode output the workflow summary reads. 3754 single_outputs["target_connection_status"] = result.get("connection_status") 3755 3756 write_github_outputs(single_outputs) 3757 3758 write_json_output( 3759 "regression_report", 3760 { 3761 **report_model.verdict_flags(), 3762 "command": command, 3763 "output_mode": output_mode, 3764 "target": _annotate_run_verdict(command, result), 3765 "check_results": report_model.check_results(), 3766 "partial_streams": list(report_model.partial_streams), 3767 }, 3768 ) 3769 3770 connection_objects: tuple[DiffBlock, ...] = () 3771 with report_section("connection objects"): 3772 connection_objects = _collect_connection_objects( 3773 catalog_file, 3774 state_file, 3775 output_path / "airbyte_messages", 3776 state_seeded=state_seeded, 3777 mask_policy=report_mask_policy, 3778 ) 3779 report_model = enrich_report_model( 3780 report_model, 3781 target_result=result, 3782 configured_streams=configured_streams, 3783 connection_objects=connection_objects, 3784 ) 3785 3786 # Generate report.md with detailed metrics 3787 with report_section("markdown report"): 3788 report_path = generate_single_version_report( 3789 connector_image=resolved_test_image, 3790 command=command, 3791 result=result, 3792 output_dir=output_path, 3793 report_model=report_model, 3794 ) 3795 print_success(f"Generated report: {report_path}") 3796 3797 # The detailed surface is report.html in the uploaded artifact; the step 3798 # summary gets the concise block instead of the whole report. Both are 3799 # built from the unannotated result: the model derives the verdict itself. 3800 with report_section("HTML report"): 3801 html_report_path = write_html_report(report_model, output_path) 3802 print_success(f"Generated HTML report: {html_report_path}") 3803 3804 # Write the concise summary to GITHUB_STEP_SUMMARY (if env var exists) 3805 with report_section("GitHub step summary"): 3806 write_github_summary(render_inline_summary(report_model)) 3807 3808 if incomplete: 3809 print_error( 3810 f"Single-version regression test incomplete for " 3811 f"{resolved_test_image}: the read was stopped early." 3812 ) 3813 elif single_success: 3814 print_success( 3815 f"Single-version regression test passed for {resolved_test_image}" 3816 ) 3817 elif command == "check" and result["success"]: 3818 print_error( 3819 f"Single-version regression test failed for {resolved_test_image}: " 3820 f"connectionStatus {result.get('connection_status') or 'missing'} " 3821 "(the connector itself exited 0)." 3822 ) 3823 else: 3824 print_error( 3825 f"Single-version regression test failed for {resolved_test_image}" 3826 ) 3827 else: 3828 # Comparison mode: run both control and target images. The control 3829 # version runs FIRST on purpose: the record comparison is 3830 # directional (extra records in the target are allowed, missing 3831 # ones fail), so data created upstream between the two live runs 3832 # must land in the target output — where it is tolerated — rather 3833 # than in the control output, where it would read as records 3834 # missing from the target and fail the comparison. Accepted side 3835 # effect (applies to every command, not just read): a hanging or 3836 # timing-out control run means the target never runs, so such a 3837 # run says nothing about the version under test. 3838 # 3839 # HTTP replay needs the same order for its own reason: the target 3840 # replays the dump the control run recorded, so that dump has to 3841 # exist before the target starts. 3842 target_output = output_path / "target" 3843 control_output = output_path / "control" 3844 3845 # The control reads `control_catalog_file`: the configured catalog, or 3846 # under --discover-missing-streams one without the target-only streams 3847 # (set above). The NOT EXERCISED rule below judges the difference. 3848 3849 # A read's two sides share `--read-timeout` with the comparison and 3850 # the report, each taking its share as it starts (see `ReadBudget`): 3851 # given the whole budget each, two long reads back to back outlast the 3852 # step, which then dies with no report. Other commands keep the 3853 # runner's default. 3854 control_sandbox = _restore_sandbox_baseline(sandbox, sandbox_scenario) 3855 control_timeout_s = ( 3856 read_budget.control_share_s() if read_timeout_seconds is not None else None 3857 ) 3858 # Use the target catalog so changed shared-stream cursor or primary-key 3859 # declarations cannot change record tokens; catalog comparison shows them. 3860 control_result, control_outputs = _run_with_optional_http_metrics( 3861 connector_image=resolved_control_image, # ty: ignore[invalid-argument-type] 3862 command=cmd, 3863 output_dir=control_output, 3864 target_or_control=TargetOrControl.CONTROL, 3865 enable_http_metrics=enable_http_metrics, 3866 config_path=config_file, 3867 catalog_path=control_catalog_file, 3868 state_path=state_file, 3869 enable_debug_logs=enable_debug_logs, 3870 extra_env=parsed_extra_env, 3871 stream_large_bodies=http_replay_max_body, 3872 timeout_seconds=control_timeout_s, 3873 max_records=max_records, 3874 output_mode=output_mode, 3875 mask_key=mask_key, 3876 mask_catalog_path=catalog_file, 3877 ) 3878 _record_read_share(control_result, control_timeout_s, read_timeout) 3879 control_sandbox = _check_sandbox_twin_gaps( 3880 sandbox, sandbox_scenario, control_sandbox, repro_case 3881 ) 3882 3883 replay_from: Path | None = None 3884 if replay_enabled: 3885 control_dump = control_result.get("http_dump_file") 3886 if control_dump: 3887 replay_from = Path(control_dump) 3888 else: 3889 print_warning( 3890 "Control run recorded no HTTP dump; the target will run live" 3891 ) 3892 3893 target_sandbox = _restore_sandbox_baseline(sandbox, sandbox_scenario) 3894 target_timeout_s = ( 3895 read_budget.target_share_s() if read_timeout_seconds is not None else None 3896 ) 3897 target_result, target_outputs = _run_with_optional_http_metrics( 3898 connector_image=resolved_test_image, 3899 command=cmd, 3900 output_dir=target_output, 3901 target_or_control=TargetOrControl.TARGET, 3902 enable_http_metrics=enable_http_metrics, 3903 config_path=config_file, 3904 catalog_path=catalog_file, 3905 state_path=state_file, 3906 enable_debug_logs=enable_debug_logs, 3907 extra_env=parsed_extra_env, 3908 replay_expected=replay_enabled, 3909 replay_from=replay_from, 3910 replay_options=replay_options, 3911 stream_large_bodies=http_replay_max_body, 3912 timeout_seconds=target_timeout_s, 3913 max_records=max_records, 3914 output_mode=output_mode, 3915 mask_key=mask_key, 3916 ) 3917 _record_read_share(target_result, target_timeout_s, read_timeout) 3918 target_sandbox = _check_sandbox_twin_gaps( 3919 sandbox, sandbox_scenario, target_sandbox, repro_case 3920 ) 3921 3922 # Both dumps have served their purpose by now -- metrics parsed, replay 3923 # corpus built -- so anything too large to be a usable artifact goes, 3924 # rather than being uploaded from a runner with ~14 GB of disk. 3925 _discard_oversized_http_dumps( 3926 control_result, 3927 target_result, 3928 max_bytes=http_replay_max_dump_size_mb * 1024 * 1024, 3929 ) 3930 3931 outcome = evaluate_comparison_outcome(command, target_result, control_result) 3932 # A side the harness stopped makes the run incomplete: the comparison 3933 # sees part of what each version would have emitted, so it can still 3934 # find a regression but can never clear one. 3935 incomplete = bool( 3936 stopped_early_sides(("control", control_result), ("target", target_result)) 3937 ) 3938 # A stream only the target read and read nothing from proves nothing 3939 # about the change: never a pass, whatever the compared streams say. 3940 target_only_streams = _target_only_stream_names( 3941 cmd, catalog_file, control_catalog_file 3942 ) 3943 setup_error_streams = undefined_streams( 3944 target_only_streams, 3945 target_outputs.stream_statuses or {}, 3946 target_result.get("record_counts_per_stream", {}), 3947 target_outputs.stream_errors or {}, 3948 ) 3949 config_errors = config_error_streams( 3950 target_only_streams, target_outputs.stream_errors or {} 3951 ) 3952 setup_error = bool(setup_error_streams) 3953 could_not_test_result = could_not_test( 3954 target_only_streams, target_outputs.stream_errors or {} 3955 ) 3956 setup_error_clears_regression = setup_error and target_errors_only_from( 3957 target_only_streams, target_outputs.stream_errors or {} 3958 ) 3959 3960 # A reproduction case replaces the record and final-state comparisons 3961 # for a read: each side is held to the case's expectations instead of 3962 # to the other side. 3963 reproduction: ReproductionVerdict | None = None 3964 if repro_case is not None and sandbox is not None: 3965 reproduction = evaluate_reproduction( 3966 evaluate_side( 3967 control_result, 3968 repro_case, 3969 sandbox_side=control_sandbox, 3970 final_states=control_outputs.final_states, 3971 ), 3972 evaluate_side( 3973 target_result, 3974 repro_case, 3975 sandbox_side=target_sandbox, 3976 final_states=target_outputs.final_states, 3977 ), 3978 ReproGoal(repro_goal or ReproGoal.FIXED.value), 3979 ) 3980 3981 # Record-level comparison for READ commands (counts, PK presence, 3982 # PK integrity, field values). Only meaningful when both runs 3983 # succeeded; a comparison failure is a regression and fails the 3984 # verdict even though both containers exited cleanly. `outcome` 3985 # itself stays the pure exit-status/connectionStatus layer — the 3986 # folded verdict lives in `success`/`regression_detected` below. 3987 record_comparison: RecordComparisonSummary | None = None 3988 target_record_files: dict[str, Path] | None = None 3989 new_record_checks: tuple[CheckResult, ...] = () 3990 record_checks_payload: dict[str, dict[str, Any]] = {} 3991 record_checks_regression = False 3992 if ( 3993 cmd in (Command.READ, Command.READ_WITH_STATE) 3994 and bool(target_result.get("success")) 3995 and reproduction is None 3996 and not skip_record_comparison 3997 and target_only_streams 3998 ): 3999 try: 4000 target_record_files = split_records_per_stream( 4001 Path(target_result["stdout_file"]) 4002 ) 4003 except Exception as exc: 4004 record_checks_regression = False 4005 new_record_checks = ( 4006 CheckResult( 4007 name="Record schema (new streams)", 4008 passed=False, 4009 summary=f"errored ({type(exc).__name__})", 4010 ), 4011 ) 4012 # A skipped comparison must be visible on the report surfaces, not 4013 # only on stdout: without this row the run reads as a pass that 4014 # compared records. Diagnostic, because the skip is not a regression. 4015 record_comparison_skipped: CheckResult | None = None 4016 if ( 4017 cmd in (Command.READ, Command.READ_WITH_STATE) 4018 and outcome.both_succeeded 4019 and not skip_record_comparison 4020 and reproduction is None 4021 ): 4022 record_comparison, skip_reason = _compare_read_records( 4023 target_result=target_result, 4024 control_result=control_result, 4025 catalog_file=catalog_file, 4026 target_record_files=target_record_files, 4027 ) 4028 if record_comparison is None and skip_reason is not None: 4029 record_comparison_skipped = CheckResult( 4030 name="Record comparison", 4031 passed=False, 4032 severity="diagnostic", 4033 summary=f"Skipped — {skip_reason}; records were not compared", 4034 ) 4035 elif ( 4036 skip_record_comparison 4037 and cmd in (Command.READ, Command.READ_WITH_STATE) 4038 and reproduction is None 4039 ): 4040 print_warning( 4041 "Record-level comparison skipped (--skip-record-comparison); " 4042 "the verdict gates on command outcomes only." 4043 ) 4044 record_comparison_skipped = CheckResult( 4045 name="Record comparison", 4046 passed=False, 4047 severity="diagnostic", 4048 summary="Skipped (--skip-record-comparison); records were not compared", 4049 ) 4050 if target_record_files is not None: 4051 try: 4052 ( 4053 new_record_checks, 4054 record_checks_payload, 4055 record_checks_regression, 4056 ) = _new_stream_record_checks( 4057 stream_names=target_only_streams, 4058 record_files=target_record_files, 4059 catalog_file=catalog_file, 4060 target_outputs=target_outputs, 4061 record_comparison=record_comparison, 4062 masked=output_mode == "masked", 4063 mask_policy=report_mask_policy, 4064 stopped=bool(stopped_early_sides(("target", target_result))), 4065 ) 4066 except Exception as exc: 4067 record_checks_regression = False 4068 new_record_checks = ( 4069 CheckResult( 4070 name="Record schema (new streams)", 4071 passed=False, 4072 summary=f"errored ({type(exc).__name__})", 4073 ), 4074 ) 4075 4076 # Spec/catalog-schema comparison of what the two versions declared 4077 # about themselves; a failed check here is a regression as much as 4078 # a dropped record. 4079 comparison_checks: tuple[CheckResult, ...] = () 4080 comparison_diff_blocks: tuple[DiffBlock, ...] = () 4081 deferred_diff_blocks: list[Callable[[], tuple[DiffBlock, ...]]] = [] 4082 comparison_detail: dict[str, Any] = {} 4083 if reproduction is None: 4084 ( 4085 comparison_checks, 4086 comparison_diff_blocks, 4087 comparison_detail, 4088 ) = _compare_outputs( 4089 command=command, 4090 outcome=outcome, 4091 control_outputs=control_outputs, 4092 target_outputs=target_outputs, 4093 partial=incomplete, 4094 no_control_streams=target_only_streams, 4095 could_not_test=could_not_test_result, 4096 deferred_diff_blocks=deferred_diff_blocks, 4097 ) 4098 failed_comparisons = failed_checks(comparison_checks) 4099 extra_checks = ( 4100 *new_record_checks, 4101 *comparison_checks, 4102 *([record_comparison_skipped] if record_comparison_skipped else []), 4103 ) 4104 report_target_result = { 4105 **target_result, 4106 "setup_error": setup_error, 4107 "setup_error_clears_regression": setup_error_clears_regression, 4108 "record_checks_regression": record_checks_regression, 4109 "could_not_test": could_not_test_result, 4110 "setup_error_detail": describe_setup_error(setup_error_streams), 4111 "could_not_test_detail": describe_could_not_test( 4112 config_errors, target_outputs.stream_errors or {} 4113 ), 4114 "excluded_not_exercised_streams": ( 4115 *setup_error_streams, 4116 *config_errors, 4117 ), 4118 } 4119 configured_streams = _configured_stream_names(cmd, catalog_file) 4120 report_model = build_comparison_report_model( 4121 target_image=resolved_test_image, 4122 control_image=resolved_control_image, # ty: ignore[invalid-argument-type] 4123 command=command, 4124 target_result=report_target_result, 4125 control_result=control_result, 4126 outcome=outcome, 4127 record_comparison=record_comparison, 4128 extra_checks=extra_checks, 4129 configured_streams=configured_streams, 4130 include_report_sections=False, 4131 reproduction=reproduction, 4132 target_only_streams=target_only_streams, 4133 failed_new_streams=tuple( 4134 name 4135 for name, checks in record_checks_payload.items() 4136 if checks.get("check_failed") 4137 ), 4138 ) 4139 4140 # Annotate copies of each side, never the originals: `generate_regression_report` 4141 # below reads `result["success"]` as the exit status, so handing it a dict whose 4142 # `success` is the verdict would corrupt the report's per-version wording. 4143 combined_result = { 4144 "target": _annotate_run_verdict(command, target_result), 4145 "control": _annotate_run_verdict(command, control_result), 4146 "both_succeeded": outcome.both_succeeded, 4147 **report_model.verdict_flags(), 4148 "output_mode": output_mode, 4149 "check_results": report_model.check_results(), 4150 } 4151 if record_checks_payload: 4152 combined_result["record_checks"] = record_checks_payload 4153 readable_fields = run_readable_fields(target_result, control_result) 4154 if readable_fields is not None: 4155 combined_result["readable_fields"] = readable_fields 4156 if record_comparison is not None: 4157 combined_result["record_comparison"] = record_comparison.to_dict() 4158 if comparison_detail: 4159 combined_result["comparison"] = comparison_detail 4160 if reproduction is not None: 4161 combined_result["reproduction"] = reproduction.to_dict() 4162 4163 print_json(_printable_payload(combined_result, output_mode)) 4164 4165 gh_outputs: dict[str, object] = { 4166 **report_model.verdict_flags(github=True), 4167 "target_image": resolved_test_image, 4168 "control_image": resolved_control_image, 4169 "command": command, 4170 "target_exit_code": target_result["exit_code"], 4171 "control_exit_code": control_result["exit_code"], 4172 "output_mode": output_mode, 4173 } 4174 4175 if command == "check": 4176 gh_outputs["target_connection_status"] = target_result.get( 4177 "connection_status" 4178 ) 4179 gh_outputs["control_connection_status"] = control_result.get( 4180 "connection_status" 4181 ) 4182 4183 if record_comparison is not None: 4184 # The record comparison's own verdict: on an incomplete run it says 4185 # the strict checks passed on what was read, so it can be true 4186 # while `success` is false. 4187 gh_outputs["record_comparison_passed"] = record_comparison.passed 4188 4189 if reproduction is not None: 4190 gh_outputs["reproduction_verdict"] = reproduction.label 4191 4192 write_github_outputs(gh_outputs) 4193 4194 write_json_output( 4195 "regression_report", 4196 { 4197 **combined_result, 4198 "partial_streams": list(report_model.partial_streams), 4199 }, 4200 ) 4201 4202 if reproduction is not None and repro_case is not None: 4203 result_path = output_path / CASE_RESULT_FILE 4204 result_path.write_text( 4205 json.dumps( 4206 case_result( 4207 reproduction, 4208 repro_case, 4209 control_image=resolved_control_image or "", 4210 target_image=resolved_test_image, 4211 ), 4212 indent=2, 4213 ) 4214 + "\n" 4215 ) 4216 print_success(f"Wrote reproduction result: {result_path}") 4217 4218 connection_objects = () 4219 with report_section("connection objects"): 4220 connection_objects = _collect_connection_objects( 4221 catalog_file, 4222 state_file, 4223 target_output / "airbyte_messages", 4224 state_seeded=state_seeded, 4225 mask_policy=report_mask_policy, 4226 ) 4227 for build_blocks in deferred_diff_blocks: 4228 with report_section("comparison diff blocks"): 4229 comparison_diff_blocks += build_blocks() 4230 report_model = enrich_report_model( 4231 report_model, 4232 target_result=report_target_result, 4233 control_result=control_result, 4234 configured_streams=configured_streams, 4235 record_comparison=record_comparison, 4236 extra_diff_blocks=comparison_diff_blocks, 4237 connection_objects=connection_objects, 4238 ) 4239 4240 with report_section("markdown report"): 4241 report_path = generate_regression_report( 4242 report_model=report_model, 4243 target_image=resolved_test_image, 4244 control_image=resolved_control_image, # ty: ignore[invalid-argument-type] 4245 command=command, 4246 target_result=report_target_result, 4247 control_result=control_result, 4248 output_dir=output_path, 4249 record_comparison=( 4250 record_comparison.to_dict() 4251 if record_comparison is not None 4252 else None 4253 ), 4254 # Without these the markdown report would announce "no regression" 4255 # over the same run the HTML report and the CLI call a failure -- 4256 # and, for a warning, would show no finding at all where the other 4257 # two surfaces show one. 4258 comparison_findings=[ 4259 (check.status, f"{check.name}: {check.summary}") 4260 for check in (*comparison_checks, *new_record_checks) 4261 if check.status in {"fail", "warn"} 4262 ], 4263 inconclusive_findings=[ 4264 f"{check.name}: {check.summary}" 4265 for check in new_record_checks 4266 if check.status == "fail" and not record_checks_regression 4267 ], 4268 reproduction=reproduction, 4269 target_only_streams=target_only_streams, 4270 ) 4271 print_success(f"Generated regression report: {report_path}") 4272 4273 with report_section("HTML report"): 4274 html_report_path = write_html_report(report_model, output_path) 4275 print_success(f"Generated regression HTML report: {html_report_path}") 4276 4277 # Write the concise summary to GITHUB_STEP_SUMMARY (if env var exists) 4278 with report_section("GitHub step summary"): 4279 write_github_summary(render_inline_summary(report_model)) 4280 4281 if ( 4282 report_model.incomplete 4283 or report_model.setup_error 4284 or report_model.could_not_test 4285 or report_model.not_exercised 4286 ): 4287 print_error(f"{report_model.verdict_label} — {report_model.verdict_detail}") 4288 elif reproduction is not None: 4289 report_line = ( 4290 f"Reproduction case {repro_case.name if repro_case else ''}: " 4291 f"{reproduction.label} -- {reproduction.detail}" 4292 ) 4293 if report_model.passed: 4294 print_success(report_line) 4295 else: 4296 print_error(report_line) 4297 for finding in reproduction.details: 4298 print_error(f" - {finding}") 4299 elif failed_comparisons: 4300 # Only reachable when both commands ran cleanly -- a comparison is 4301 # skipped otherwise -- so this says what the exit codes could not. 4302 print_error( 4303 f"Regression detected between {resolved_test_image} and " 4304 f"{resolved_control_image}: " 4305 + "; ".join( 4306 f"{check.name.lower()} — {check.summary}" 4307 for check in failed_comparisons 4308 ) 4309 ) 4310 elif report_model.outcome and report_model.outcome.check_improvement: 4311 print_success( 4312 f"Regression test passed for {resolved_test_image} vs " 4313 f"{resolved_control_image}: target connectionStatus SUCCEEDED " 4314 "while control FAILED (improvement)." 4315 ) 4316 elif report_model.comparison_errored: 4317 print_error( 4318 f"Record comparison errored for {resolved_test_image} vs " 4319 f"{resolved_control_image}. Test failed/inconclusive; this " 4320 "is a tooling failure, not evidence of a regression." 4321 ) 4322 elif report_model.strict_failed: 4323 print_error( 4324 f"Regression detected between {resolved_test_image} and " 4325 f"{resolved_control_image}: both versions ran, but the " 4326 "record comparison failed." 4327 ) 4328 elif report_model.outcome and report_model.outcome.both_succeeded: 4329 print_success( 4330 f"Regression test passed for {resolved_test_image} vs {resolved_control_image}" 4331 ) 4332 elif report_model.regression_detected: 4333 print_error( 4334 f"Regression detected between {resolved_test_image} and {resolved_control_image}" 4335 ) 4336 elif report_model.both_failed: 4337 print_error( 4338 f"Both versions failed for {resolved_test_image} vs {resolved_control_image}. " 4339 "Test failed/inconclusive; cannot rule out a regression." 4340 ) 4341 elif report_model.outcome and report_model.outcome.control_failed_only: 4342 print_error( 4343 f"Control {resolved_control_image} failed while target " 4344 f"{resolved_test_image} succeeded. Test failed/inconclusive; " 4345 "nothing to compare the target against." 4346 ) 4347 else: 4348 print_error( 4349 f"Regression test failed for {resolved_test_image} vs " 4350 f"{resolved_control_image}: {report_model.verdict_label} — " 4351 f"{report_model.verdict_detail}" 4352 ) 4353 4354 4355def _repro_case_argument_error( 4356 *, 4357 command: Command, 4358 repro_case_dir: str | None, 4359 sandbox_scenario: str | None, 4360 skip_compare: bool, 4361 connection_id: str | None, 4362 output_mode: OutputMode, 4363 config_path: str | None, 4364 catalog_path: str | None, 4365 state_path: str | None, 4366 selected_streams: str | None, 4367) -> str | None: 4368 """Why the reproduction-case options cannot be used as given, if they cannot.""" 4369 if not (repro_case_dir and sandbox_scenario): 4370 return ( 4371 "--repro-case-dir and --sandbox-scenario must be given together, " 4372 "and --repro-goal needs them" 4373 ) 4374 if command not in (Command.READ, Command.READ_WITH_STATE, Command.CHECK): 4375 # Only a read or a CHECK is judged against a case. 4376 return "Reproduction cases run only `--command read` or `--command check`" 4377 if skip_compare: 4378 return "Reproduction cases need comparison mode; drop --skip-compare" 4379 if connection_id: 4380 return "Reproduction cases cannot use --connection-id" 4381 if output_mode == "masked": 4382 # The case is judged on raw values: masked records would read as 4383 # missing, and a masked error as not the expected one. 4384 return ( 4385 "Reproduction cases run raw: the sandbox holds no customer data and " 4386 "the case's expectations are raw values; drop --output-mode masked" 4387 ) 4388 if not config_path: 4389 return "Reproduction cases need --config-path (from regression-case-setup)" 4390 if catalog_path or state_path: 4391 return "The reproduction case supplies the catalog and state" 4392 if selected_streams: 4393 # The filter rewrites the catalog file in place, i.e. the committed case. 4394 return "The reproduction case's catalog selects its own streams" 4395 return None 4396 4397 4398def _repro_case_command_error(command: Command, case_command: str) -> str | None: 4399 """Why `--command` is not the command the case is judged by, if it is not.""" 4400 runs = "check" if command is Command.CHECK else "read" 4401 if runs != case_command: 4402 return f"This case runs only `--command {case_command}`" 4403 return None 4404 4405 4406def _restore_sandbox_baseline( 4407 sandbox: SandboxCli | None, scenario_id: str | None 4408) -> SandboxSide: 4409 """Restore the case's baseline before one side of a case read, else a no-op.""" 4410 if sandbox is None or not scenario_id: 4411 return SandboxSide() 4412 side = restore_side(sandbox, scenario_id) 4413 if side.restore_error: 4414 print_error(f"Could not restore the sandbox baseline: {side.restore_error}") 4415 return side 4416 4417 4418def _check_sandbox_twin_gaps( 4419 sandbox: SandboxCli | None, 4420 scenario_id: str | None, 4421 side: SandboxSide, 4422 repro_case: ReproCase | None = None, 4423) -> SandboxSide: 4424 """Gate one side on the twin supporting every request it made.""" 4425 if sandbox is None or not scenario_id: 4426 return side 4427 side = check_twin_gaps( 4428 sandbox, 4429 scenario_id, 4430 side, 4431 repro_case.injected_failures if repro_case else (), 4432 ) 4433 for gap in side.twin_gaps: 4434 print_error(f"Twin gap: {gap}") 4435 if side.gap_check_error: 4436 print_error(f"Twin-gap check failed: {side.gap_check_error}") 4437 return side 4438 4439 4440@connector_app.command(name="regression-case-setup") 4441def regression_case_setup( 4442 case_dir: Annotated[ 4443 str, 4444 Parameter(help="Reproduction case directory to set up."), 4445 ], 4446 config_output: Annotated[ 4447 str | None, 4448 Parameter( 4449 help="Where to write the rendered connector config. Defaults to a " 4450 "private temporary directory outside the artifacts tree." 4451 ), 4452 ] = None, 4453) -> None: 4454 """Set up a reproduction case in the SaaS Sandbox. 4455 4456 Applies the case's sandbox/dataset.json together with its optional 4457 sandbox/behavior.json (failure rules), snapshots the result as the 4458 `baseline` that `regression-test --repro-case-dir` restores before each side 4459 of a read, and renders config.template.json with the scenario's 4460 synthetic credential. The sandbox connection comes from `SAAS_API_URL`, 4461 `SAAS_API_KEY`, `SAAS_WORKSPACE_ID` and `XDG_STATE_HOME`. 4462 4463 Writes `scenario_id`, `config_path` and the resolved `case_dir` to the 4464 GitHub step outputs. 4465 """ 4466 try: 4467 case = ReproCase.load(Path(case_dir)) 4468 sandbox = SandboxCli() 4469 name = f"{case.name}-{os.getenv('GITHUB_RUN_ID') or uuid.uuid4().hex[:8]}" 4470 destination = ( 4471 Path(config_output) 4472 if config_output 4473 else Path(tempfile.mkdtemp(prefix=f"repro-case-{case.name}-")) 4474 / "config.json" 4475 ) 4476 scenario_id, config_file, credential = set_up_scenario( 4477 case, sandbox, scenario_name=name, config_destination=destination 4478 ) 4479 except (OSError, ValueError, SandboxError) as exc: 4480 exit_with_error(f"Could not set up reproduction case {case_dir}: {exc}") 4481 4482 if is_github_actions(): 4483 # The workflow log must never show the credential, even a synthetic one. 4484 print(f"::add-mask::{credential}") 4485 outputs = { 4486 "scenario_id": scenario_id, 4487 "config_path": str(config_file), 4488 "case_dir": str(case.directory), 4489 # From a fixed set, so case content cannot inject an output line. 4490 "command": case.expectations.command, 4491 } 4492 write_github_outputs(outputs) 4493 print_json(outputs) 4494 4495 4496@connector_app.command(name="sandbox-support") 4497def sandbox_support( 4498 connector_name: Annotated[ 4499 str, 4500 Parameter(help="Connector to check (e.g. 'source-slack')."), 4501 ], 4502 streams: Annotated[ 4503 str | None, 4504 Parameter( 4505 help="Comma-separated streams the fix touches; their parent streams " 4506 "are included. Defaults to every stream in the manifest." 4507 ), 4508 ] = None, 4509 dataset_needs: Annotated[ 4510 str | None, 4511 Parameter( 4512 help="Comma-separated `entity` or `entity.field` the reproduction " 4513 "needs in the twin's dataset (e.g. 'messages.thread_ts')." 4514 ), 4515 ] = None, 4516 connector_dir: Annotated[ 4517 str | None, 4518 Parameter( 4519 help="Local connector directory. Without it, the files are read from " 4520 "airbytehq/airbyte at --ref." 4521 ), 4522 ] = None, 4523 ref: Annotated[ 4524 str, 4525 Parameter(help="airbytehq/airbyte ref to read the connector from."), 4526 ] = "master", 4527 host_only: Annotated[ 4528 bool, 4529 Parameter( 4530 help="Only decide whether a twin serves the connector's provider: " 4531 "one serving an allowed host and, when the manifest's request hosts " 4532 "can be read, one of those (token endpoints don't count). No " 4533 "operation is checked, so `operations` is empty. Cannot be combined " 4534 "with --streams or --dataset-needs." 4535 ), 4536 ] = False, 4537) -> None: 4538 """Check statically whether a SaaS Sandbox twin covers a connector's needs. 4539 4540 The first gate before a reproduction case: provider host match, the HTTP 4541 operations, query parameters and page sizes the streams use, and optional 4542 dataset needs, against the sandbox's offline discovery data (`sandbox ... 4543 --offline`; needs the `sandbox` CLI, but no server). Prints a JSON report 4544 whose `result` is `supported`, `unsupported` or `unknown`. `unknown` means 4545 the requests could not be read statically; the case lane's twin-gap gate 4546 still decides during the run. With --host-only, only the provider match is 4547 decided (the oncall shadow gate's "has a twin" question). 4548 """ 4549 # Checked on presence, so `--streams ""` is refused too. 4550 if host_only and (streams is not None or dataset_needs is not None): 4551 exit_with_error( 4552 "--host-only checks no stream or dataset need; drop --streams and " 4553 "--dataset-needs" 4554 ) 4555 print_json( 4556 run_sandbox_support_check( 4557 connector_name=connector_name, 4558 streams=_split_csv(streams), 4559 dataset_needs=_split_csv(dataset_needs), 4560 connector_dir=Path(connector_dir) if connector_dir else None, 4561 ref=ref, 4562 host_only=host_only, 4563 ) 4564 ) 4565 4566 4567@connector_app.command(name="regression-case-gate") 4568def regression_case_gate( 4569 base_ref: Annotated[ 4570 str | None, 4571 Parameter( 4572 help="The branch the PR merges into, e.g. `origin/main`. Required " 4573 "unless --case-dir is given." 4574 ), 4575 ] = None, 4576 head_ref: Annotated[ 4577 str, 4578 Parameter(help="The PR's head commit; the case is read at this commit."), 4579 ] = "HEAD", 4580 repo_dir: Annotated[ 4581 str, 4582 Parameter( 4583 help="airbyte-ops-mcp clone holding both --base-ref and --head-ref. " 4584 "Its working tree is not read." 4585 ), 4586 ] = ".", 4587 case_dir: Annotated[ 4588 str | None, 4589 Parameter( 4590 help="Check this case directory on disk instead of a PR: no scope " 4591 "check, and the connector is the name of its parent directory." 4592 ), 4593 ] = None, 4594 require_case: Annotated[ 4595 bool, 4596 Parameter(help="Fail, instead of answering `n/a`, when the PR adds no case."), 4597 ] = False, 4598 connector_dir: Annotated[ 4599 str | None, 4600 Parameter( 4601 help="Local connector directory at the case's `affected_ref`. Without " 4602 "it, the files are read from airbytehq/airbyte at `affected_ref`." 4603 ), 4604 ] = None, 4605 json_out: Annotated[ 4606 str | None, 4607 Parameter(help="Also write the JSON report to this file."), 4608 ] = None, 4609 sandbox_ref: Annotated[ 4610 str | None, 4611 Parameter( 4612 help="The saas-sandbox commit the case runs use. Defaults to the pin " 4613 "in `.github/saas-sandbox-ref` of the current directory (the code " 4614 "checkout); the case's `origin.json` `sandbox_ref` must equal it." 4615 ), 4616 ] = None, 4617) -> None: 4618 """Check whether a reproduction-case PR may merge without human review. 4619 4620 Prints a JSON report whose `status` is `n/a` (the PR touches no case), 4621 `pass` or `fail`, and exits nonzero on `fail`. Checks that the PR only adds 4622 one new case under `regression_cases/<connector>/<case>/`, as regular files 4623 with known names; that the case loads with an `origin.json` naming a 4624 published airbytehq/airbyte release whose source is at `affected_ref`, a 4625 commit on `master`; that the sandbox validates `sandbox/` offline; and that 4626 the static support check says `supported`. The reproduction itself is a 4627 case run with goal `reproduced`. Writes `connector`, `case`, 4628 `affected_image`, `affected_ref`, `problems` (a JSON list) and `status` to 4629 the GitHub step outputs. 4630 """ 4631 if sandbox_ref is None: 4632 try: 4633 sandbox_ref = read_sandbox_pin(Path(SANDBOX_PIN_FILE)) 4634 except ValueError as exc: 4635 exit_with_error(f"{exc}; run from the code checkout or pass --sandbox-ref") 4636 elif not SHA_PATTERN.fullmatch(sandbox_ref): 4637 exit_with_error("--sandbox-ref must be a full 40-character commit SHA") 4638 token = resolve_default_github_token(allow_none=True) 4639 connector_path = Path(connector_dir) if connector_dir else None 4640 if case_dir is not None: 4641 report = run_case_dir_gate( 4642 Path(case_dir), 4643 sandbox_ref=sandbox_ref, 4644 github_token=token, 4645 connector_dir=connector_path, 4646 ) 4647 elif base_ref is None: 4648 exit_with_error("Pass --base-ref, or --case-dir to check a case directory.") 4649 else: 4650 try: 4651 report = run_case_gate( 4652 Path(repo_dir), 4653 base_ref, 4654 head_ref, 4655 sandbox_ref=sandbox_ref, 4656 require_case=require_case, 4657 github_token=token, 4658 connector_dir=connector_path, 4659 ) 4660 except subprocess.CalledProcessError as exc: 4661 exit_with_error( 4662 f"Could not compare {base_ref}...{head_ref}: {exc.stderr.strip()}" 4663 ) 4664 if json_out: 4665 Path(json_out).write_text(json.dumps(report.to_dict(), indent=2)) 4666 write_github_outputs(gate_step_outputs(report)) 4667 print_json(report.to_dict()) 4668 if report.status == "fail": 4669 exit_with_error("Case gate failed: " + "; ".join(report.problems)) 4670 4671 4672@connector_app.command(name="regression-case-validate-decide") 4673def regression_case_validate_decide( 4674 pr: Annotated[int, Parameter(help="The case PR's number.")], 4675 needs: Annotated[ 4676 str, 4677 Parameter(help="File holding the workflow's `toJSON(needs)`."), 4678 ], 4679 gate_report: Annotated[ 4680 str, 4681 Parameter(help="The gate's `--json-out` report; may be missing."), 4682 ], 4683 merge_enabled: Annotated[ 4684 bool, 4685 Parameter(help="The merge switch (`SANDBOX_CASE_AUTOMERGE_ENABLED`)."), 4686 ] = False, 4687 run_url: Annotated[str, Parameter(help="This run's URL, for the comment.")] = "", 4688 repo: Annotated[ 4689 str | None, 4690 Parameter(help="`owner/name` of the repository; defaults to `GH_REPO`."), 4691 ] = None, 4692 comment_out: Annotated[ 4693 str, 4694 Parameter(help="Where to write the failure comment, when there is one."), 4695 ] = "comment.md", 4696) -> None: 4697 """Decide what `sandbox-case-validate.yml` does after the gate and the runs. 4698 4699 Re-reads the PR through `gh` (set `GH_REPO` and `GH_TOKEN`) and prints a 4700 JSON decision: `action` (`fail`, `report` or `merge`), the commit status 4701 `state` and `description`, and on `fail` the PR comment, also written to 4702 --comment-out. Writes `action`, `state`, `description` and `validated_sha` 4703 to the GitHub step outputs. 4704 """ 4705 needs_data = json.loads(Path(needs).read_text()) 4706 gate_outputs = (needs_data.get("gate") or {}).get("outputs") or {} 4707 validated_sha = str(gate_outputs.get("head_sha") or "") 4708 # It is written back to the step outputs, so it must be a plain SHA. 4709 if not SHA_PATTERN.fullmatch(validated_sha): 4710 validated_sha = "" 4711 report_path = Path(gate_report) 4712 gate = json.loads(report_path.read_text()) if report_path.is_file() else None 4713 try: 4714 pr_now = gh_read_pr(repo or os.environ.get("GH_REPO", ""), pr) 4715 except (RuntimeError, ValueError, subprocess.SubprocessError): 4716 pr_now = None 4717 decision = decide_validation( 4718 gate=gate, 4719 gate_refusal=gate_outputs.get("refusal") or None, 4720 runs=runs_from_needs(needs_data), 4721 pr=pr_now, 4722 validated_sha=validated_sha, 4723 merge_enabled=merge_enabled, 4724 run_url=run_url, 4725 ) 4726 if decision.comment is not None: 4727 Path(comment_out).write_text(decision.comment) 4728 write_github_outputs( 4729 { 4730 "action": decision.action, 4731 "state": decision.state, 4732 "description": decision.description, 4733 "validated_sha": validated_sha, 4734 } 4735 ) 4736 print_json(decision.to_dict()) 4737 4738 4739@connector_app.command(name="regression-case-merge") 4740def regression_case_merge( 4741 pr: Annotated[int, Parameter(help="The case PR's number.")], 4742 sha: Annotated[str, Parameter(help="The validated head commit.")], 4743 connector: Annotated[str, Parameter(help="The case's connector, for the title.")], 4744 case: Annotated[str, Parameter(help="The case's name, for the title.")], 4745 repo: Annotated[ 4746 str | None, 4747 Parameter(help="`owner/name` of the repository; defaults to `GH_REPO`."), 4748 ] = None, 4749 run_url: Annotated[ 4750 str, Parameter(help="The validating run, for the commit message.") 4751 ] = "", 4752 timeout_minutes: Annotated[ 4753 float, 4754 Parameter(help="How long to wait for GitHub to report the PR mergeable."), 4755 ] = 10, 4756) -> None: 4757 """Merge a validated case PR at its validated head, waiting a bounded time. 4758 4759 Polls the PR through `gh` (set `GH_TOKEN` to a token that may merge) until 4760 GitHub reports it mergeable, then squash-merges it through the REST merge 4761 endpoint pinned to `sha`, with a fixed commit title. Never enables 4762 auto-merge. Prints the outcome and writes `outcome`, `merge_state`, `state` 4763 and `description` (the final commit status) to the GitHub step outputs. 4764 Exits nonzero unless the PR merged at `sha` or simply did not become 4765 mergeable in time. 4766 """ 4767 repo = repo or os.environ.get("GH_REPO") 4768 if not repo: 4769 exit_with_error("Pass --repo or set GH_REPO.") 4770 if not ( 4771 SHA_PATTERN.fullmatch(sha) 4772 and NAME_PATTERN.fullmatch(connector) 4773 and NAME_PATTERN.fullmatch(case) 4774 ): 4775 exit_with_error("--sha must be a full SHA; --connector and --case plain names.") 4776 title = f"test(regression-cases): add {connector} {case} reproduction case (#{pr})" 4777 message = ( 4778 f"Validated by {run_url}" if run_url else "Validated by sandbox-case-validate" 4779 ) 4780 outcome = wait_and_merge( 4781 sha, 4782 timeout_seconds=timeout_minutes * 60, 4783 read_pr=lambda: gh_read_pr(repo, pr), 4784 merge=lambda: gh_merge(repo, pr, sha, title, message), 4785 ) 4786 result = outcome.to_dict() 4787 write_github_outputs( 4788 { 4789 "outcome": outcome.result, 4790 "merge_state": outcome.merge_state or "", 4791 "state": result["state"], 4792 "description": result["description"], 4793 } 4794 ) 4795 print_json(result) 4796 if not outcome.succeeded: 4797 exit_with_error(f"Case PR #{pr} was not merged: {outcome.result}") 4798 4799 4800def _split_csv(value: str | None) -> list[str]: 4801 return [item.strip() for item in (value or "").split(",") if item.strip()] 4802 4803 4804def _stop_with_setup_error(message: str) -> NoReturn: 4805 """Report a failed setup verdict but exit 0 to avoid infrastructure failures.""" 4806 write_github_output("success", False) 4807 write_github_output("setup_error", True) 4808 write_github_output("error", message) 4809 write_github_summary( 4810 f"## `READ` Test Results\n\n**Result:** SETUP ERROR — {message}" 4811 ) 4812 print_error(message) 4813 raise SystemExit(0) 4814 4815 4816def _injection_discover_timeout_message( 4817 image: str, timeout_s: int, read_budget: ReadBudget 4818) -> str: 4819 if timeout_s <= 0: 4820 return ( 4821 f"Setup error: DISCOVER of {image} for --discover-missing-streams had " 4822 "no read time left to run in " 4823 f"(--read-timeout {read_budget.budget_s} s). Raise read_timeout_minutes, " 4824 "or check why DISCOVER hangs." 4825 ) 4826 return ( 4827 f"Setup error: DISCOVER of {image} for --discover-missing-streams did not " 4828 f"finish within the {timeout_s} s left of the read budget " 4829 f"(--read-timeout {read_budget.budget_s} s). Raise read_timeout_minutes, " 4830 "or check why DISCOVER hangs." 4831 ) 4832 4833 4834def _discovered_catalog( 4835 connector_image: str, 4836 target_or_control: TargetOrControl, 4837 output_dir: Path, 4838 config_file: Path | None, 4839 enable_debug_logs: bool, 4840 extra_env: dict[str, str] | None, 4841 output_mode: OutputMode, 4842 mask_key: bytes | None, 4843 timeout_seconds: int, 4844) -> tuple[dict[str, Any] | None, dict[str, Any] | None]: 4845 """Run a budgeted DISCOVER and return its catalog and stopped-early result.""" 4846 result, outputs = _run_connector_command( 4847 connector_image=connector_image, 4848 command=Command.DISCOVER, 4849 output_dir=output_dir, 4850 target_or_control=target_or_control, 4851 timeout_seconds=timeout_seconds, 4852 config_path=config_file, 4853 enable_debug_logs=enable_debug_logs, 4854 output_mode=output_mode, 4855 mask_key=mask_key, 4856 extra_env=extra_env, 4857 ) 4858 if outputs.catalog is None: 4859 return None, result.get("stopped_early") 4860 return ( 4861 outputs.catalog.model_dump(mode="json", exclude_none=True), 4862 result.get("stopped_early"), 4863 ) 4864 4865 4866def _inject_discovered_streams( 4867 *, 4868 catalog_file: Path | None, 4869 selected_stream_names: set[str], 4870 test_image: str, 4871 reference_image: str | None, 4872 skip_compare: bool, 4873 config_file: Path | None, 4874 output_path: Path, 4875 enable_debug_logs: bool, 4876 extra_env: dict[str, str] | None, 4877 sync_mode: InjectedSyncMode | None, 4878 output_mode: OutputMode, 4879 mask_key: bytes | None, 4880 read_budget: ReadBudget, 4881) -> tuple[Path, Path, InjectedCatalogs] | None: 4882 """Add the selected streams the configured catalog lacks, from DISCOVER. 4883 4884 The target's DISCOVER configures the streams. The reference version's 4885 DISCOVER -- the control image, or in single-version mode the released 4886 image -- says whether each one is new: a stream it also discovers is only 4887 unselected on the connection, so both versions read it using their own 4888 discovered entries. Without the reference catalog the novelty is unknown 4889 and the stream is read by the target only, as if new, and reported as 4890 unknown. Each DISCOVER gets the read time left when it starts. 4891 4892 A selected stream already in the catalog that the reference version does 4893 not discover is kept from the control, as a new stream would be. That is 4894 what a run without a connection needs: its catalog comes from the target's 4895 DISCOVER, so the PR's new streams are in it already. 4896 4897 Returns: 4898 The target catalog path, the control catalog path and the 4899 classification, or `None` when neither catalog would change. 4900 """ 4901 configured = ( 4902 json.loads(catalog_file.read_text()) 4903 if catalog_file is not None and catalog_file.exists() 4904 else None 4905 ) 4906 configured_names = { 4907 stream.get("stream", stream).get("name", "") 4908 for stream in (configured or {}).get("streams", []) 4909 } 4910 missing = selected_stream_names - configured_names 4911 4912 discover_dir = output_path / "discover_for_injection" 4913 target_timeout_s = read_budget.setup_share_s() 4914 if target_timeout_s <= 0: 4915 _stop_with_setup_error( 4916 _injection_discover_timeout_message( 4917 test_image, target_timeout_s, read_budget 4918 ) 4919 ) 4920 target_catalog, target_stopped_early = _discovered_catalog( 4921 test_image, 4922 TargetOrControl.TARGET, 4923 discover_dir / "target", 4924 config_file, 4925 enable_debug_logs, 4926 extra_env, 4927 output_mode, 4928 mask_key, 4929 target_timeout_s, 4930 ) 4931 if target_stopped_early: 4932 _stop_with_setup_error( 4933 _injection_discover_timeout_message( 4934 test_image, target_timeout_s, read_budget 4935 ) 4936 ) 4937 if target_catalog is None and not missing: 4938 print_warning( 4939 f"{test_image} returned no catalog from DISCOVER; the selected streams " 4940 "are read as configured" 4941 ) 4942 return None 4943 if target_catalog is None: 4944 error_msg = ( 4945 f"Setup error: {test_image} returned no catalog from DISCOVER, so the selected " 4946 "streams missing from the configured catalog cannot be added." 4947 ) 4948 _stop_with_setup_error(error_msg) 4949 4950 reference_catalog: dict[str, Any] | None = None 4951 if reference_image and reference_image != test_image: 4952 if skip_compare and not ensure_image_available(reference_image): 4953 print_warning( 4954 f"Could not pull {reference_image}; whether the added streams are " 4955 "new is unknown, so they are read by the target only" 4956 ) 4957 else: 4958 reference_timeout_s = read_budget.setup_share_s() 4959 if reference_timeout_s <= 0: 4960 _stop_with_setup_error( 4961 _injection_discover_timeout_message( 4962 reference_image, reference_timeout_s, read_budget 4963 ) 4964 ) 4965 reference_catalog, reference_stopped_early = _discovered_catalog( 4966 reference_image, 4967 TargetOrControl.CONTROL, 4968 discover_dir / "control", 4969 config_file, 4970 enable_debug_logs, 4971 extra_env, 4972 output_mode, 4973 mask_key, 4974 reference_timeout_s, 4975 ) 4976 if reference_stopped_early: 4977 _stop_with_setup_error( 4978 _injection_discover_timeout_message( 4979 reference_image, reference_timeout_s, read_budget 4980 ) 4981 ) 4982 if reference_catalog is None: 4983 print_warning( 4984 f"{reference_image} returned no catalog from DISCOVER; whether " 4985 "the added streams are new is unknown, so they are read by the " 4986 "target only" 4987 ) 4988 else: 4989 print_warning( 4990 "No control version to compare the discovered streams with; whether " 4991 "the added streams are new is unknown, so they are read by the target " 4992 "only" 4993 ) 4994 4995 try: 4996 injection = build_injected_catalogs( 4997 configured, 4998 selected_stream_names, 4999 target_catalog, 5000 reference_catalog, 5001 sync_mode=sync_mode, 5002 ) 5003 except StreamsNotDiscoveredError as exc: 5004 _stop_with_setup_error(f"Setup error: {exc}") 5005 if not injection.changes_anything: 5006 return None 5007 5008 if not skip_compare and not injection.control_catalog["streams"]: 5009 error_msg = ( 5010 "Every selected stream has no control side: " 5011 f"{', '.join(injection.target_only_streams)}. The control version would " 5012 "read nothing, so add one stream the connection already syncs to " 5013 "--selected-streams, or use --skip-compare for a single-version run." 5014 ) 5015 _stop_with_setup_error(f"Setup error: {error_msg}") 5016 5017 catalog_dir = output_path / "connection_data" 5018 catalog_dir.mkdir(parents=True, exist_ok=True) 5019 target_catalog_file = ( 5020 catalog_file if catalog_file is not None else catalog_dir / "catalog.json" 5021 ) 5022 control_catalog_file = target_catalog_file.with_name("control_catalog.json") 5023 target_catalog_file.write_text(json.dumps(injection.target_catalog, indent=2)) 5024 control_catalog_file.write_text(json.dumps(injection.control_catalog, indent=2)) 5025 # The image novelty was decided against: the control, a pinned connection 5026 # version or metadata.yaml, so a reader never has to guess which. 5027 summary = { 5028 **injection.to_dict(), 5029 "reference_image": reference_image if reference_catalog is not None else None, 5030 } 5031 (output_path / "stream_injection.json").write_text(json.dumps(summary, indent=2)) 5032 write_github_output("stream_injection", json.dumps(summary)) 5033 5034 for label, names in ( 5035 (f"new, not in {reference_image}, target only", injection.new_streams), 5036 (f"already in {reference_image}, read by both", injection.shared_streams), 5037 ("novelty unknown, target only", injection.unknown_streams), 5038 ): 5039 if names: 5040 print_success(f"Added stream(s) ({label}): {', '.join(names)}") 5041 if injection.kept_target_only: 5042 print_success( 5043 f"Kept from the control (not in {reference_image}, target only): " 5044 f"{', '.join(injection.kept_target_only)}" 5045 ) 5046 for note in injection.notes: 5047 print_warning(note) 5048 return target_catalog_file, control_catalog_file, injection 5049 5050 5051def _configured_stream_names( 5052 command: Command, catalog_file: Path | None 5053) -> tuple[str, ...]: 5054 """Every stream the run was configured to read. 5055 5056 The report's per-stream tables are keyed on emitted records, so a stream 5057 that returned nothing has no entry and would be missing from the report 5058 entirely. Reading the configured catalog is what lets it be shown as zero. 5059 5060 Only for commands that read records. `--connection-id` fetches a catalog for 5061 every command, and handing it to `spec`, `check` or `discover` would give 5062 them an all-zero stream table and a coverage warning saying no stream 5063 returned anything -- true, meaningless, and enough noise to train a reviewer 5064 to ignore the warning on the one command where it means something. 5065 """ 5066 if command not in (Command.READ, Command.READ_WITH_STATE): 5067 return () 5068 5069 if catalog_file is None or not catalog_file.exists(): 5070 return () 5071 5072 try: 5073 catalog = ConfiguredAirbyteCatalog.parse_raw(catalog_file.read_text()) # ty: ignore[deprecated] 5074 except Exception as exc: 5075 print_warning(f"Could not read the configured catalog for the report: {exc}") 5076 return () 5077 5078 return tuple( 5079 sorted( 5080 configured.stream.name 5081 for configured in catalog.streams 5082 if configured.stream 5083 ) 5084 ) 5085 5086 5087def _configured_streams_by_id(catalog: Any) -> dict[tuple[str | None, str], dict]: 5088 """A configured catalog's stream entries, keyed by `(namespace, name)`. 5089 5090 Keyed as `to_stream_id` keys a stream, so `users` in two namespaces is two 5091 entries. An entry without a stream name is skipped. 5092 """ 5093 streams = catalog.get("streams", []) if isinstance(catalog, dict) else [] 5094 return { 5095 to_stream_id( 5096 stream["stream"].get("namespace"), stream["stream"]["name"] 5097 ): stream 5098 for stream in streams 5099 if isinstance(stream, dict) 5100 and isinstance(stream.get("stream"), dict) 5101 and isinstance(stream["stream"].get("name"), str) 5102 } 5103 5104 5105def _target_only_stream_names( 5106 command: Command, target_catalog: Path | None, control_catalog: Path | None 5107) -> tuple[str, ...]: 5108 """The streams configured for the target and not for the control. 5109 5110 One of these that returns no records makes a comparison read NOT 5111 EXERCISED (see `not_exercised_streams`). Both sides read the same catalog 5112 today, so this is empty and the rule is dormant; it fires once a run 5113 configures a catalog per side, with the streams a connector PR adds on 5114 the target's only. 5115 """ 5116 if control_catalog == target_catalog: 5117 return () 5118 5119 control = set(_configured_stream_names(command, control_catalog)) 5120 5121 return tuple( 5122 stream 5123 for stream in _configured_stream_names(command, target_catalog) 5124 if stream not in control 5125 ) 5126 5127 5128def _judge_seed_file( 5129 state_file: Path, catalog_file: Path | None 5130) -> tuple[str | None, list[str]]: 5131 """Why `--state-path` cannot seed this read, and the seeded streams it will not reach. 5132 5133 A connector ignores the state of a stream it syncs as full_refresh, so a 5134 seed is refused when none of the streams it names is one the configured 5135 catalog syncs incrementally -- the catalog generated from DISCOVER when 5136 there is no connection puts every stream on full_refresh, and 5137 `--selected-streams` can drop a seeded stream -- since the read would be 5138 cold while the log and the report called it seeded. A seeded stream the 5139 catalog reads in full is returned for a warning; the rest still seed. 5140 Streams match on namespace and name, as the connector matches a state to 5141 its stream: a seed for `public.users` is not taken by an incremental 5142 `reporting.users`. A seed in another shape (a GLOBAL state, say) names no 5143 stream the harness can match, and is held only to the coarse rule that 5144 some stream syncs incrementally. Without a catalog there is nothing to 5145 judge against, and the read fails on its own. 5146 5147 Returns: 5148 The error, or `None`; and the seeded streams the catalog reads in full, 5149 as `namespace.name` labels. 5150 """ 5151 try: 5152 text = state_file.read_text() 5153 state = json.loads(text) 5154 except (OSError, ValueError) as exc: 5155 return f"--state-path cannot be read as JSON: {exc}", [] 5156 if state == []: 5157 return ( 5158 "--state-path holds an empty list, which seeds nothing; drop it for " 5159 "a cold read", 5160 [], 5161 ) 5162 if catalog_file is None: 5163 return None, [] 5164 try: 5165 catalog = json.loads(catalog_file.read_text()) 5166 except (OSError, ValueError) as exc: 5167 return f"--catalog-path cannot be read as JSON: {exc}", [] 5168 incremental = { 5169 stream_id 5170 for stream_id, stream in _configured_streams_by_id(catalog).items() 5171 if stream.get("sync_mode") == "incremental" 5172 } 5173 fix = ( 5174 "A catalog generated from DISCOVER puts every stream on full_refresh; " 5175 "give --connection-id so the catalog is the connection's own, or a " 5176 "--catalog-path that syncs the seeded streams incrementally" 5177 ) 5178 if seed_state_error(text) is not None: 5179 if incremental: 5180 return None, [] 5181 return ( 5182 "--state-path cannot seed this read: no stream of the configured " 5183 "catalog syncs incrementally, so the connector would ignore the " 5184 f"state and the run would still report it as seeded. {fix}", 5185 [], 5186 ) 5187 seeded = [to_stream_id(*stream) for stream in seeded_streams(text)] 5188 if not set(seeded) & incremental: 5189 named = ", ".join(stream_label(*stream) for stream in seeded) 5190 synced = ( 5191 ", ".join(sorted(stream_label(*stream) for stream in incremental)) 5192 or "no stream" 5193 ) 5194 return ( 5195 f"--state-path cannot seed this read: it names {named}, and the " 5196 f"configured catalog syncs {synced} incrementally, so the connector " 5197 "would ignore the state and the run would still report it as " 5198 f"seeded. {fix}", 5199 [], 5200 ) 5201 return None, [ 5202 stream_label(*stream) for stream in seeded if stream not in incremental 5203 ] 5204 5205 5206def _seed_outline(seed_text: str, catalog_file: Path | None) -> str: 5207 """What the report shows of a seeded state: the streams it names, no value. 5208 5209 A seed is customer state, and the report is an uploaded artifact, so the 5210 `State (seeded)` block lists each seeded stream's namespace and name with 5211 the sync mode and cursor field the configured catalog gives it; a stream 5212 the catalog does not configure has neither. A seed that is not a list of 5213 STREAM messages names no stream to list, and gives an empty outline, so 5214 the block is left off. 5215 """ 5216 if seed_state_error(seed_text) is not None: 5217 return "" 5218 try: 5219 catalog = json.loads(catalog_file.read_text()) if catalog_file else None 5220 except (OSError, ValueError): 5221 catalog = None 5222 configured = _configured_streams_by_id(catalog) 5223 outline = [] 5224 for namespace, name in seeded_streams(seed_text): 5225 stream = configured.get(to_stream_id(namespace, name), {}) 5226 cursor_field = stream.get("cursor_field") or stream.get("stream", {}).get( 5227 "default_cursor_field" 5228 ) 5229 outline.append( 5230 { 5231 "namespace": namespace, 5232 "name": name, 5233 "sync_mode": stream.get("sync_mode"), 5234 "cursor_field": cursor_field or [], 5235 } 5236 ) 5237 return json.dumps(outline) 5238 5239 5240def _collect_connection_objects( 5241 catalog_file: Path | None, 5242 state_file: Path | None, 5243 messages_dir: Path | None, 5244 *, 5245 state_seeded: bool = False, 5246 mask_policy: MaskPolicy | None = None, 5247) -> tuple[DiffBlock, ...]: 5248 """Gather the objects the run was given or discovered, for the report. 5249 5250 The connector config is deliberately absent: it is the one object that 5251 carries secrets, and the report is an artifact anyone with repo access can 5252 download. A seeded state is customer state the caller brought, so its 5253 values are left out too: only the streams it names are shown (see 5254 `_seed_outline`). 5255 5256 Args: 5257 catalog_file: The configured catalog handed to the run. 5258 state_file: The state handed to the run, for a warm read. 5259 messages_dir: The target side's `airbyte_messages` directory, which 5260 holds the catalog the connector discovered. 5261 state_seeded: Whether the caller chose the state (`--state-path`) 5262 rather than the run fetching the connection's own. The label says 5263 so, since a reader otherwise assumes the connection's cursors. 5264 mask_policy: The masked run's policy, under which the connection's 5265 own state is masked before it goes on the page (a seeded state 5266 shows its outline only, in either mode); the file itself is the 5267 raw one the connector read. A state that does not parse is left 5268 off the page rather than embedded raw. `None` in a raw run. 5269 5270 Returns: 5271 One JSON block per object that exists, in the order a reader wants them. 5272 """ 5273 sources: list[tuple[str, Path | None, bool]] = [ 5274 ("Configured catalog (input)", catalog_file, False), 5275 ("State (seeded)" if state_seeded else "State (input)", state_file, True), 5276 ( 5277 "Discovered catalog (from the target)", 5278 (messages_dir / "catalog.jsonl") if messages_dir else None, 5279 False, 5280 ), 5281 ] 5282 5283 blocks: list[DiffBlock] = [] 5284 for title, path, is_state in sources: 5285 if path is None or not path.is_file(): 5286 continue 5287 5288 try: 5289 payload = path.read_text() 5290 except OSError: 5291 continue 5292 5293 if is_state and state_seeded: 5294 payload = _seed_outline(payload, catalog_file) 5295 elif is_state and mask_policy is not None: 5296 try: 5297 state = json.loads(payload) 5298 except ValueError: 5299 continue 5300 payload = json.dumps(mask_input_state(state, mask_policy), indent=2) 5301 5302 if payload.strip(): 5303 blocks.append(build_json_block(title, payload)) 5304 5305 return tuple(blocks) 5306 5307 5308# The order commands are presented in, matching how a sync runs them. Anything 5309# else found under the artifacts directory follows, sorted. 5310_REPORT_COMMAND_ORDER = ("spec", "check", "discover", "read") 5311 5312 5313@connector_app.command(name="consolidate-regression-reports") 5314def consolidate_regression_reports( 5315 artifacts_dir: Annotated[ 5316 str, 5317 Parameter( 5318 help="Directory holding one subdirectory per command, each with the " 5319 "`report.html` that `regression-test --output-dir` wrote." 5320 ), 5321 ], 5322 output_path: Annotated[ 5323 str | None, 5324 Parameter( 5325 name=["--output"], 5326 help="Where to write the consolidated report. " 5327 "Defaults to `report.html` in `--artifacts-dir`.", 5328 ), 5329 ] = None, 5330) -> None: 5331 """Fold one run's per-command regression reports into a single page. 5332 5333 The regression test runs the CLI once per Airbyte command, so each command 5334 writes its own report into its own artifact. This produces the run-level 5335 view -- a verdict table across commands plus every report inlined -- so a 5336 reviewer downloads one file instead of four. 5337 5338 Never fails the run: a command whose report is missing or unreadable is 5339 skipped, because this is a reporting convenience and the per-command 5340 artifacts remain the source of truth. 5341 """ 5342 artifacts_path = Path(artifacts_dir) 5343 ordered = [artifacts_path / command for command in _REPORT_COMMAND_ORDER] 5344 extra = sorted( 5345 path 5346 for path in artifacts_path.glob("*") 5347 if path.is_dir() and path not in ordered 5348 ) 5349 report_paths = [ 5350 directory / HTML_REPORT_FILENAME for directory in [*ordered, *extra] 5351 ] 5352 5353 consolidated = consolidate_reports(report_paths) 5354 if consolidated is None: 5355 print_warning( 5356 f"No per-command reports found under {artifacts_path}; " 5357 "nothing to consolidate." 5358 ) 5359 return 5360 5361 destination = Path(output_path) if output_path else artifacts_path / "report.html" 5362 destination.parent.mkdir(parents=True, exist_ok=True) 5363 destination.write_text(consolidated) 5364 print_success(f"Consolidated regression report: {destination}") 5365 5366 5367@connector_app.command(name="fetch-connection-config") 5368def fetch_connection_config_cmd( 5369 connection_id: Annotated[ 5370 str, 5371 Parameter(help="The UUID of the Airbyte Cloud connection."), 5372 ], 5373 output_path: Annotated[ 5374 str | None, 5375 Parameter( 5376 help="Path to output file or directory. " 5377 "If directory, writes connection-<id>-config.json inside it. " 5378 "Default: platform temp directory (e.g. /tmp/connection-<id>-config.json)" 5379 ), 5380 ] = None, 5381 with_secrets: Annotated[ 5382 bool, 5383 Parameter( 5384 name="--with-secrets", 5385 negative="--no-secrets", 5386 help="If set, fetches unmasked secrets from the internal database. " 5387 "Requires GCP_PROD_DB_ACCESS_CREDENTIALS env var or `gcloud auth application-default login`. " 5388 "Must be used with --oc-issue-url.", 5389 ), 5390 ] = False, 5391 oc_issue_url: Annotated[ 5392 str | None, 5393 Parameter( 5394 help="OC issue URL for audit logging. Required when using --with-secrets." 5395 ), 5396 ] = None, 5397) -> None: 5398 """Fetch connection configuration from Airbyte Cloud to a local file. 5399 5400 This command retrieves the source configuration for a given connection ID 5401 and writes it to a JSON file. When `--output-path` is omitted the file is 5402 written to the platform temp directory to avoid accidentally committing 5403 secrets to a git repository. 5404 5405 Requires authentication via AIRBYTE_CLOUD_CLIENT_ID and 5406 AIRBYTE_CLOUD_CLIENT_SECRET environment variables. 5407 5408 When --with-secrets is specified, the command fetches unmasked secrets from 5409 the internal database using the connection-retriever. This additionally requires: 5410 - An OC issue URL for audit logging (--oc-issue-url) 5411 - GCP credentials via `GCP_PROD_DB_ACCESS_CREDENTIALS` env var or `gcloud auth application-default login` 5412 - Cloud SQL Python Connector access to the Prod DB replica. 5413 """ 5414 path = Path(output_path) if output_path else None 5415 result = fetch_connection_config( 5416 connection_id=connection_id, 5417 output_path=path, 5418 with_secrets=with_secrets, 5419 oc_issue_url=oc_issue_url, 5420 ) 5421 if result.success: 5422 print_success(result.message) 5423 else: 5424 print_error(result.message) 5425 print_json(result.model_dump()) 5426 5427 5428def _read_json_input( 5429 json_str: str | None, 5430 input_file: str | None, 5431) -> dict: 5432 """Read JSON from a positional arg, --input file, or STDIN (in that priority).""" 5433 if json_str is not None: 5434 return json.loads(json_str) 5435 if input_file is not None: 5436 return json.loads(Path(input_file).read_text()) 5437 if not sys.stdin.isatty(): 5438 return json.loads(sys.stdin.read()) 5439 exit_with_error( 5440 "No JSON input provided. Pass it as a positional argument, " 5441 "via --input <file>, or pipe to STDIN." 5442 ) 5443 raise SystemExit(1) # unreachable, but satisfies type checker 5444 5445 5446# Every way the residency lookup can fail before it has an answer. Callers turn 5447# these into one line and exit, so a lookup failure never routes a connection. 5448_RESIDENCY_LOOKUP_ERRORS: tuple[type[Exception], ...] = ( 5449 ConnectionNotFoundError, 5450 ValueError, 5451 google.auth.exceptions.GoogleAuthError, 5452 ProdDbSecretAccessError, 5453 ProdDbCloudSqlIamError, 5454 CloudRunProdDbAccessNotConfiguredError, 5455 sqlalchemy.exc.SQLAlchemyError, 5456 google_api_exceptions.GoogleAPIError, 5457 OSError, 5458) 5459 5460 5461def _describe_residency_lookup_error(connection_id: str, exc: Exception) -> str: 5462 """Describe a failed residency lookup in one line, without a traceback.""" 5463 if isinstance( 5464 exc, 5465 ( 5466 google.auth.exceptions.DefaultCredentialsError, 5467 google.auth.exceptions.RefreshError, 5468 ), 5469 ): 5470 return ( 5471 "No usable GCP credentials for the Prod DB replica " 5472 f"({type(exc).__name__}). Set {ENV_GCP_PROD_DB_ACCESS_CREDENTIALS} " 5473 "or configure Application Default Credentials." 5474 ) 5475 if isinstance( 5476 exc, 5477 ( 5478 ConnectionNotFoundError, 5479 ValueError, 5480 ProdDbSecretAccessError, 5481 ProdDbCloudSqlIamError, 5482 ), 5483 ): 5484 return str(exc) 5485 return ( 5486 "Could not reach the Prod DB replica to read the dataplane group of " 5487 f"connection {connection_id} ({type(exc).__name__})." 5488 ) 5489 5490 5491def _refuse_eu_connection_outside_eu_region(connection_id: str, residency: str) -> None: 5492 """Exit before any fetch if an EU connection is used outside the EU region. 5493 5494 This check runs before the public API returns config or catalog, keeping 5495 them off a US runner. 5496 """ 5497 if residency == "eu" and not is_running_in_eu_region(): 5498 message = ( 5499 f"{EU_REGION_REQUIRED_MESSAGE}; connection ID {connection_id} was " 5500 "refused before its config or catalog was fetched." 5501 ) 5502 write_github_output("success", False) 5503 write_github_output("error", message) 5504 exit_with_error( 5505 f"{message}\n\n" 5506 "Use a connection from a non-EU workspace, or use GSM-based " 5507 "integration test credentials instead (by omitting --connection-id)." 5508 ) 5509 5510 5511def _refuse_non_eu_connection_in_eu_region(connection_id: str) -> None: 5512 """Exit before any fetch if a non-EU connection is used in the EU region. 5513 5514 The mirror of `_refuse_eu_connection_outside_eu_region`, so each runtime reads 5515 only connections in the data region their customer chose. 5516 """ 5517 if is_running_in_eu_region(): 5518 message = ( 5519 f"{EU_REGION_EU_ONLY_MESSAGE}; connection ID {connection_id} was " 5520 "refused before its config or catalog was fetched." 5521 ) 5522 write_github_output("success", False) 5523 write_github_output("error", message) 5524 exit_with_error(message) 5525 5526 5527@connection_app.command(name="get-region") 5528def get_connection_region( 5529 connection_id: Annotated[ 5530 str, 5531 Parameter(help="The connection ID (UUID) to check."), 5532 ], 5533) -> None: 5534 """Get a connection's dataplane group and derived residency. 5535 5536 Reads only the dataplane group, not connection or actor secrets. Requires 5537 `GCP_PROD_DB_ACCESS_CREDENTIALS` or Application Default Credentials. On any 5538 lookup failure it exits non-zero and writes no GitHub output. 5539 """ 5540 try: 5541 dataplane_group = get_connection_dataplane_group(connection_id) 5542 except _RESIDENCY_LOOKUP_ERRORS as exc: 5543 exit_with_error(_describe_residency_lookup_error(connection_id, exc)) 5544 5545 residency = residency_for_dataplane_group(dataplane_group) 5546 write_github_output("dataplane_group", dataplane_group) 5547 write_github_output("residency", residency) 5548 print_json( 5549 { 5550 "connection_id": connection_id, 5551 "dataplane_group": dataplane_group, 5552 "residency": residency, 5553 } 5554 ) 5555 5556 5557@state_app.command(name="get") 5558def get_connection_state( 5559 connection_id: Annotated[ 5560 str, 5561 Parameter(help="The connection ID (UUID) to fetch state for."), 5562 ], 5563 stream_name: Annotated[ 5564 str | None, 5565 Parameter( 5566 help="Optional stream name to filter state for a single stream.", 5567 name="--stream", 5568 ), 5569 ] = None, 5570 stream_namespace: Annotated[ 5571 str | None, 5572 Parameter( 5573 help="Optional stream namespace to narrow the stream filter.", 5574 name="--namespace", 5575 ), 5576 ] = None, 5577 output: Annotated[ 5578 str | None, 5579 Parameter( 5580 help="Path to write the state JSON output to a file.", 5581 name="--output", 5582 ), 5583 ] = None, 5584) -> None: 5585 """Get the current state for an Airbyte Cloud connection.""" 5586 workspace = CloudWorkspace.from_env() 5587 conn = workspace.get_connection(connection_id) 5588 5589 if stream_name is not None: 5590 stream_state = conn.get_stream_state( 5591 stream_name=stream_name, 5592 stream_namespace=stream_namespace, 5593 ) 5594 result = { 5595 "connection_id": connection_id, 5596 "stream_name": stream_name, 5597 "stream_namespace": stream_namespace, 5598 "stream_state": stream_state, 5599 } 5600 else: 5601 result = conn.dump_raw_state() 5602 5603 json_output = json.dumps(result, indent=2, default=str) 5604 if output is not None: 5605 Path(output).write_text(json_output + "\n") 5606 print(f"State written to {output}", file=sys.stderr) 5607 else: 5608 print(json_output) 5609 5610 5611@state_app.command(name="set") 5612def set_connection_state( 5613 connection_id: Annotated[ 5614 str, 5615 Parameter(help="The connection ID (UUID) to update state for."), 5616 ], 5617 state_json: Annotated[ 5618 str | None, 5619 Parameter( 5620 help="The connection state as a JSON string. When --stream is used, " 5621 'this is just the stream\'s state blob (e.g., \'{"cursor": "2024-01-01"}\').' 5622 " Otherwise, must include 'stateType', 'connectionId', and the appropriate " 5623 "state field. Can also be provided via --input or STDIN." 5624 ), 5625 ] = None, 5626 stream_name: Annotated[ 5627 str | None, 5628 Parameter( 5629 help="Optional stream name to update state for a single stream only.", 5630 name="--stream", 5631 ), 5632 ] = None, 5633 stream_namespace: Annotated[ 5634 str | None, 5635 Parameter( 5636 help="Optional stream namespace to identify the stream.", 5637 name="--namespace", 5638 ), 5639 ] = None, 5640 input_file: Annotated[ 5641 str | None, 5642 Parameter( 5643 help="Path to a JSON file containing the state to set.", 5644 name="--input", 5645 ), 5646 ] = None, 5647) -> None: 5648 """Set the state for an Airbyte Cloud connection. 5649 5650 State JSON can be provided as a positional argument, via --input <file>, 5651 or piped through STDIN. 5652 5653 Uses the safe variant that prevents updates while a sync is running. 5654 When --stream is provided, only that stream's state is updated within 5655 the existing connection state. 5656 """ 5657 parsed_state = _read_json_input(state_json, input_file) 5658 workspace = CloudWorkspace.from_env() 5659 conn = workspace.get_connection(connection_id) 5660 5661 if stream_name is not None: 5662 conn.set_stream_state( 5663 stream_name=stream_name, 5664 state_blob_dict=parsed_state, 5665 stream_namespace=stream_namespace, 5666 ) 5667 else: 5668 conn.import_raw_state(parsed_state) 5669 5670 result = conn.dump_raw_state() 5671 json_output = json.dumps(result, indent=2, default=str) 5672 print(json_output) 5673 5674 5675@state_app.command(name="reset") 5676def reset_stream_state( 5677 connection_id: Annotated[ 5678 str, 5679 Parameter(help="The connection ID (UUID) to update state for."), 5680 ], 5681 stream_name: Annotated[ 5682 str, 5683 Parameter( 5684 help="The configured stream name whose state should be reset.", 5685 name="--stream", 5686 ), 5687 ], 5688 stream_namespace: Annotated[ 5689 str | None, 5690 Parameter( 5691 help="Optional stream namespace to identify the stream.", 5692 name="--namespace", 5693 ), 5694 ] = None, 5695) -> None: 5696 """Reset a configured stream's state so the next sync full-refreshes it. 5697 5698 Uses the safe variant that prevents updates while a sync is running. 5699 Returns a restorable `previous_state_backup` in raw Config API format. 5700 """ 5701 workspace = CloudWorkspace.from_env() 5702 conn = workspace.get_connection(connection_id) 5703 5704 try: 5705 result = reset_cloud_stream_state( 5706 conn, 5707 stream_name=stream_name, 5708 stream_namespace=stream_namespace, 5709 ) 5710 except PyAirbyteInputError as e: 5711 exit_with_error(str(e)) 5712 5713 print(json.dumps(result.model_dump(), indent=2, default=str)) 5714 5715 5716@catalog_app.command(name="get") 5717def get_connection_catalog( 5718 connection_id: Annotated[ 5719 str, 5720 Parameter(help="The connection ID (UUID) to fetch catalog for."), 5721 ], 5722 output: Annotated[ 5723 str | None, 5724 Parameter( 5725 help="Path to write the catalog JSON output to a file.", 5726 name="--output", 5727 ), 5728 ] = None, 5729) -> None: 5730 """Get the configured catalog for an Airbyte Cloud connection.""" 5731 workspace = CloudWorkspace.from_env() 5732 conn = workspace.get_connection(connection_id) 5733 result = conn.dump_raw_catalog() 5734 if result is None: 5735 exit_with_error("No configured catalog found for this connection.") 5736 raise SystemExit(1) # unreachable, but satisfies type checker 5737 5738 json_output = json.dumps(result, indent=2, default=str) 5739 if output is not None: 5740 Path(output).write_text(json_output + "\n") 5741 print(f"Catalog written to {output}", file=sys.stderr) 5742 else: 5743 print(json_output) 5744 5745 5746@catalog_app.command(name="set") 5747def set_connection_catalog( 5748 connection_id: Annotated[ 5749 str, 5750 Parameter(help="The connection ID (UUID) to update catalog for."), 5751 ], 5752 catalog_json: Annotated[ 5753 str | None, 5754 Parameter( 5755 help="The configured catalog as a JSON string. " 5756 "Can also be provided via --input or STDIN." 5757 ), 5758 ] = None, 5759 input_file: Annotated[ 5760 str | None, 5761 Parameter( 5762 help="Path to a JSON file containing the catalog to set.", 5763 name="--input", 5764 ), 5765 ] = None, 5766) -> None: 5767 """Set the configured catalog for an Airbyte Cloud connection. 5768 5769 Catalog JSON can be provided as a positional argument, via --input <file>, 5770 or piped through STDIN. 5771 5772 WARNING: This replaces the entire configured catalog. 5773 """ 5774 parsed_catalog = _read_json_input(catalog_json, input_file) 5775 workspace = CloudWorkspace.from_env() 5776 conn = workspace.get_connection(connection_id) 5777 conn.import_raw_catalog(parsed_catalog) 5778 5779 result = conn.dump_raw_catalog() 5780 if result is None: 5781 exit_with_error("Failed to retrieve catalog after import.") 5782 raise SystemExit(1) # unreachable, but satisfies type checker 5783 5784 json_output = json.dumps(result, indent=2, default=str) 5785 print(json_output) 5786 5787 5788@logs_app.command(name="lookup-cloud-backend-error") 5789def lookup_cloud_backend_error( 5790 error_id: Annotated[ 5791 str, 5792 Parameter( 5793 help=( 5794 "The error ID (UUID) to search for. This is typically returned " 5795 "in API error responses as {'errorId': '...'}" 5796 ) 5797 ), 5798 ], 5799 lookback_days: Annotated[ 5800 int, 5801 Parameter(help="Number of days to look back in logs."), 5802 ] = 7, 5803 min_severity_filter: Annotated[ 5804 GCPSeverity | None, 5805 Parameter( 5806 help="Optional minimum severity level to filter logs.", 5807 ), 5808 ] = None, 5809 raw: Annotated[ 5810 bool, 5811 Parameter(help="Output raw JSON instead of formatted text."), 5812 ] = False, 5813) -> None: 5814 """Look up error details from GCP Cloud Logging by error ID. 5815 5816 When an Airbyte Cloud API returns an error response with only an error ID 5817 (e.g., {"errorId": "3173452e-8f22-4286-a1ec-b0f16c1e078a"}), this command 5818 fetches the full stack trace and error details from GCP Cloud Logging. 5819 5820 Requires GCP credentials with Logs Viewer role on the target project. 5821 Set up credentials with: gcloud auth application-default login 5822 """ 5823 print(f"Searching for error ID: {error_id}", file=sys.stderr) 5824 print(f"Lookback days: {lookback_days}", file=sys.stderr) 5825 if min_severity_filter: 5826 print(f"Severity filter: {min_severity_filter}", file=sys.stderr) 5827 print(file=sys.stderr) 5828 5829 result = fetch_error_logs( 5830 error_id=error_id, 5831 lookback_days=lookback_days, 5832 min_severity_filter=min_severity_filter, 5833 ) 5834 5835 if raw: 5836 print_json(result.model_dump()) 5837 return 5838 5839 print(f"Found {result.total_entries_found} log entries", file=sys.stderr) 5840 print(file=sys.stderr) 5841 5842 if result.payloads: 5843 for i, payload in enumerate(result.payloads): 5844 print(f"=== Log Group {i + 1} ===") 5845 print(f"Timestamp: {payload.timestamp}") 5846 print(f"Severity: {payload.severity}") 5847 if payload.resource.labels.pod_name: 5848 print(f"Pod: {payload.resource.labels.pod_name}") 5849 print(f"Lines: {payload.num_log_lines}") 5850 print() 5851 print(payload.message) 5852 print() 5853 elif result.entries: 5854 print("No grouped payloads, showing raw entries:", file=sys.stderr) 5855 for entry in result.entries: 5856 print(f"[{entry.timestamp}] {entry.severity}: {entry.payload}") 5857 else: 5858 print_error("No log entries found for this error ID.") 5859 5860 5861# ============================================================================= 5862# Organization commands 5863# ============================================================================= 5864 5865 5866@organization_app.command(name="search") 5867def organization_search( 5868 name_contains: Annotated[ 5869 str, 5870 Parameter( 5871 name="--name-contains", 5872 help="Case-insensitive substring to match against organization name or email.", 5873 ), 5874 ], 5875 limit: Annotated[ 5876 int, 5877 Parameter( 5878 name="--limit", 5879 help="Maximum number of results to return.", 5880 ), 5881 ] = 20, 5882) -> None: 5883 """Search organizations by name or email substring.""" 5884 rows = _search_organizations(name_contains=name_contains, limit=limit) 5885 if not rows: 5886 print_error(f"No organizations found matching '{name_contains}'.") 5887 return 5888 5889 print_json([dict(row) for row in rows]) 5890 5891 5892@organization_app.command(name="info") 5893def organization_info( 5894 organization_id: Annotated[ 5895 str, 5896 Parameter(help="The organization UUID."), 5897 ], 5898) -> None: 5899 """Get information about an organization (stub — not yet implemented).""" 5900 exit_with_error( 5901 "organization info is not yet implemented. " 5902 "Use 'airbyte-ops cloud organization search' to find organizations by name." 5903 ) 5904 5905 5906# ============================================================================= 5907# Workspace commands 5908# ============================================================================= 5909 5910 5911@workspace_app.command(name="search") 5912def workspace_search( 5913 name_contains: Annotated[ 5914 str | None, 5915 Parameter( 5916 name="--name-contains", 5917 help="Case-insensitive substring to match against workspace name or slug.", 5918 ), 5919 ] = None, 5920 email_domain: Annotated[ 5921 str | None, 5922 Parameter( 5923 name="--email-domain", 5924 help="Email domain to search for (e.g. 'motherduck.com').", 5925 ), 5926 ] = None, 5927 limit: Annotated[ 5928 int, 5929 Parameter( 5930 name="--limit", 5931 help="Maximum number of results to return.", 5932 ), 5933 ] = 100, 5934) -> None: 5935 """Search workspaces by name substring or email domain.""" 5936 if not name_contains and not email_domain: 5937 exit_with_error( 5938 "At least one of --name-contains or --email-domain must be provided." 5939 ) 5940 5941 if name_contains: 5942 rows = _search_workspaces(name_contains=name_contains, limit=limit) 5943 else: 5944 rows = query_workspaces_by_email_domain( 5945 email_domain=email_domain.lstrip("@"), # ty: ignore[unresolved-attribute] 5946 limit=limit, 5947 ) 5948 5949 if not rows: 5950 search_term = name_contains or email_domain 5951 print_error(f"No workspaces found matching '{search_term}'.") 5952 return 5953 5954 print_json([dict(row) for row in rows]) 5955 5956 5957@workspace_app.command(name="info") 5958def workspace_info_cmd( 5959 workspace_id: Annotated[ 5960 str, 5961 Parameter(help="The workspace UUID."), 5962 ], 5963) -> None: 5964 """Get information about a workspace.""" 5965 result = query_workspace_info(workspace_id=workspace_id) 5966 if not result: 5967 print_error(f"No workspace found with ID '{workspace_id}'.") 5968 return 5969 5970 print_json(dict(result))