airbyte_ops_mcp.cli.registry
CLI commands for connector registry operations.
This module provides CLI wrappers for registry operations. The core logic
lives in the airbyte_ops_mcp.registry capability module.
Command groups:
airbyte-ops registry connector list - List all connectors in registry airbyte-ops registry connector-version list - List versions for a connector airbyte-ops registry connector-version next - Compute next version tag (prerelease/RC) airbyte-ops registry connector-version yank - Mark a connector version as yanked airbyte-ops registry connector-version unyank - Remove yank marker from a connector version airbyte-ops registry connector-version metadata get - Read connector metadata from GCS airbyte-ops registry connector-version artifacts generate - Generate version artifacts locally via docker airbyte-ops registry connector-version artifacts publish - Publish version artifacts to GCS airbyte-ops registry store mirror --local|--gcs-bucket|--s3-bucket - Mirror entire registry for testing airbyte-ops registry store compile --store coral:dev|coral:prod - Compile registry indexes and sync latest/ dirs airbyte-ops registry marketing-stubs sync --store coral:prod - Sync connector_stubs.json to GCS airbyte-ops registry marketing-stubs check --store coral:prod - Compare local file with GCS
CLI reference
The commands below are regenerated by poe docs-generate via cyclopts's
programmatic docs API; see docs/generate_cli.py.
airbyte-ops registry COMMAND
Connector registry operations (GCS metadata service).
Commands:
connector: Connector listing operations.connector-version: Connector version operations (list, yank, unyank, artifacts).marketing-stubs: Marketing connector stubs GCS operations (whole-file sync).progressive-rollout: Progressive rollout lifecycle operations.store: Whole-registry store operations (mirror, compile).
airbyte-ops registry connector
Connector listing operations.
airbyte-ops registry connector list
airbyte-ops registry connector list [OPTIONS] STORE
List connectors in the registry.
When filters are applied, reads the compiled cloud_registry.json index
for fast lookups. Without filters, falls back to scanning individual
metadata blobs (captures all connectors including OSS-only).
Requires GCS_CREDENTIALS environment variable to be set.
Parameters:
STORE, --store: Store target (e.g. 'coral:dev', 'coral:prod'). [required]--certified-only, --no-certified-only: Include only certified connectors. [default: False]--support-level: Exact support level to match (e.g.certified,community,archived). [choices: archived, community, certified]--min-support-level: Minimum support level (inclusive). Levels from lowest to highest:archived,community,certified. [choices: archived, community, certified]--connector-type: Filter by connector type:sourceordestination. [choices: source, destination]--language: Filter by implementation language (e.g.python,java,manifest-only). [choices: python, java, low-code, manifest-only]--format: Output format: 'json' for JSON array, 'text' for newline-separated. [choices: json, text] [default: json]
airbyte-ops registry connector-version
Connector version operations (list, yank, unyank, artifacts).
airbyte-ops registry connector-version metadata
Connector metadata inspection.
Commands:
get: Read a connector version's metadata from the registry.
airbyte-ops registry connector-version metadata get
airbyte-ops registry connector-version metadata get [OPTIONS] NAME STORE
Read a connector version's metadata from the registry.
Returns the full metadata.yaml content for a connector at the specified version.
Requires GCS_CREDENTIALS environment variable to be set.
Parameters:
NAME, --name: Connector name (e.g., 'source-faker', 'destination-postgres'). [required]STORE, --store: Store target (e.g. 'coral:dev', 'coral:prod'). [required]--version: Version to read (e.g., 'latest', '1.2.3'). [default: latest]--format: Output format: 'json' for JSON, 'raw' for YAML. [choices: json, raw] [default: json]
airbyte-ops registry connector-version artifacts
Version artifact generation and publishing.
Commands:
generate: Generate version artifacts for a connector locally.publish: Publish version artifacts to GCS using fsspec rsync.
airbyte-ops registry connector-version artifacts generate
airbyte-ops registry connector-version artifacts generate METADATA-FILE DOCKER-IMAGE [ARGS]
Generate version artifacts for a connector locally.
Runs the connector's docker image with DEPLOYMENT_MODE=cloud and DEPLOYMENT_MODE=oss to obtain both spec variants, then generates the registry entries (cloud.json, oss.json) by applying registryOverrides from the metadata.
The generated metadata.yaml is enriched with git commit info, SBOM URL,
and (when applicable) components SHA before writing. Validation is run
after generation by default; pass --no-validate to skip.
This is a local-only operation -- no files are uploaded to GCS.
Use artifacts publish to upload generated artifacts to GCS.
Parameters:
METADATA-FILE, --metadata-file: Path to the connector's metadata.yaml file. [required]DOCKER-IMAGE, --docker-image: Docker image to run spec against (e.g., 'airbyte/source-faker:6.2.38'). [required]OUTPUT-DIR, --output-dir: Directory to write artifacts to. If not specified, a temp directory is created.REPO-ROOT, --repo-root: Root of the Airbyte repo checkout (for resolving doc.md). If not specified, inferred by walking up from metadata-file.DRY-RUN, --dry-run, --no-dry-run: Show what would be generated without running docker or writing files. [default: False]WITH-VALIDATE, --with-validate, --no-validate: Run metadata validators after generation (default: enabled). Use --no-validate to skip. [default: True]WITH-SBOM, --with-sbom, --no-sbom: Generate spdx.json (SBOM) for connectors (default: enabled). Use --no-sbom to skip. [default: True]WITH-DEPENDENCY-DUMP, --with-dependency-dump, --no-dependency-dump: Generate dependencies.json for Python connectors (default: enabled). Use --no-dependency-dump to skip. [default: True]
airbyte-ops registry connector-version artifacts publish
airbyte-ops registry connector-version artifacts publish [OPTIONS] NAME VERSION ARTIFACTS-DIR STORE
Publish version artifacts to GCS using fsspec rsync.
Uploads locally generated artifacts (from artifacts generate) to the
versioned path in GCS. By default, metadata is validated before upload;
pass --no-validate to skip.
Uses --store to select the destination store and environment:
coral:dev→ coral dev bucket at rootcoral:prod→ coral prod bucket at rootcoral:dev/aj-test100→ coral dev bucket underaj-test100/prefix
Requires GCS_CREDENTIALS environment variable to be set.
Parameters:
NAME, --name: Connector name (e.g., 'source-faker'). [required]VERSION, --version: Version to publish artifacts for (e.g., '1.2.3'). [required]ARTIFACTS-DIR, --artifacts-dir: Directory containing generated artifacts to publish (from 'artifacts generate'). [required]STORE, --store: Store target (e.g. 'coral:dev', 'coral:prod', 'coral:dev/prefix'). [required]--dry-run, --no-dry-run: Show what would be published without writing to GCS. [default: False]--with-validate, --no-validate: Validate metadata before uploading (default: enabled). Use --no-validate to skip. [default: True]
airbyte-ops registry connector-version next
airbyte-ops registry connector-version next NAME SHA [ARGS]
Compute the next version tag for a connector.
Outputs the version tag to stdout for easy capture in shell scripts. This is the single source of truth for pre-release version format.
The command fetches the connector's metadata.yaml from GitHub at the given SHA to determine the base version. It also compares against the master branch and prints a warning to stderr if no version bump is detected.
If --base-version is provided, it is used directly instead of fetching from GitHub.
Parameters:
NAME, --name: Connector name (e.g., 'source-github'). [required]SHA, --sha: Git commit SHA (full or at least 7 characters). [required]BASE-VERSION, --base-version: Base version override. If not provided, fetched from metadata.yaml at the given SHA.
airbyte-ops registry connector-version list
airbyte-ops registry connector-version list [OPTIONS] NAME STORE
List all versions of a connector in the registry.
Scans the registry bucket to find all versions of a specific connector.
Requires GCS_CREDENTIALS environment variable to be set.
Parameters:
NAME, --name: Connector name (e.g., 'source-faker'). [required]STORE, --store: Store target (e.g. 'coral:dev', 'coral:prod'). [required]--format: Output format: 'json' for JSON array, 'text' for newline-separated. [choices: json, text] [default: json]
airbyte-ops registry connector-version yank
airbyte-ops registry connector-version yank [OPTIONS] NAME VERSION STORE
Mark a connector version as yanked.
Writes a version-yank.yml marker file to the version's directory in GCS. Yanked versions are excluded when determining the latest version of a connector.
Requires GCS_CREDENTIALS environment variable to be set.
Parameters:
NAME, --name: Connector name (e.g., 'source-faker'). [required]VERSION, --version: Version to yank (e.g., '1.2.3'). [required]STORE, --store: Store target (e.g. 'coral:dev', 'coral:prod'). [required]--reason: Reason for yanking this version. [default: ""]--approval-url: Approval evidence URL to record in the yank marker. [default: ""]--dry-run, --no-dry-run: Show what would be done without making changes. [default: False]
airbyte-ops registry connector-version unyank
airbyte-ops registry connector-version unyank [OPTIONS] NAME VERSION STORE
Rename the active yank marker to a dated audit marker.
Moves version-yank.yml to version-unyanked-yyyymmdd.yml, making the
version eligible again when determining the latest version.
Requires GCS_CREDENTIALS environment variable to be set.
Parameters:
NAME, --name: Connector name (e.g., 'source-faker'). [required]VERSION, --version: Version to unyank (e.g., '1.2.3'). [required]STORE, --store: Store target (e.g. 'coral:dev', 'coral:prod'). [required]--dry-run, --no-dry-run: Show what would be done without making changes. [default: False]
airbyte-ops registry store
Whole-registry store operations (mirror, compile).
airbyte-ops registry store mirror
airbyte-ops registry store mirror [ARGS]
Create a mirror of the connector registry from the source store.
Reads all connector metadata from the source store and copies it to the specified output target. Supports local filesystem, GCS, and S3 as output targets via fsspec.
Output targets are mutually exclusive: specify exactly one of --local, --gcs-bucket, or --s3-bucket.
The production bucket is categorically disallowed as an output target.
To clean up legacy artifacts (e.g. disabled strict-encrypt connectors)
after mirroring, run compile with --with-legacy-migration v1::
airbyte-ops registry store compile --store coral:dev/my-prefix \
--with-legacy-migration v1
Parameters:
LOCAL, --local: Write output to a local directory. Mutually exclusive with --gcs-bucket and --s3-bucket. [default: False]GCS-BUCKET, --gcs-bucket: Write output to a GCS bucket. Must not be the prod bucket. Mutually exclusive with --local and --s3-bucket.S3-BUCKET, --s3-bucket: Write output to an S3 bucket. Mutually exclusive with --local and --gcs-bucket.OUTPUT-PATH-ROOT, --output-path-root: Root path/prefix for the output. For --local, this is a directory path (defaults to a new temp dir if omitted). For --gcs-bucket/--s3-bucket, this prefix is prepended to all blob paths.DRY-RUN, --dry-run, --no-dry-run: Show what would be rebuilt without writing any files. [default: False]SOURCE-STORE, --source-store: Source store to read from (e.g. 'coral:prod', 'coral:dev'). [default: coral:prod]CONNECTOR-NAME, --connector-name, --empty-connector-name: Only rebuild these connectors (by name). Can be specified multiple times, e.g. --connector-name source-faker --connector-name destination-bigquery.
airbyte-ops registry store compile
airbyte-ops registry store compile [OPTIONS] STORE
Compile the registry: sync latest/ dirs, write global and per-connector indexes.
Scans all version directories in the target store, determines the latest GA
semver per connector (excluding yanked and pre-release versions), ensures
each latest/ directory matches the computed latest, and writes:
registries/v0/cloud_registry.json-- global cloud registry indexregistries/v0/oss_registry.json-- global OSS registry indexmetadata/airbyte/<connector>/versions.json-- per-connector version index
With --with-secrets-mask, also regenerates:
registries/v0/specs_secrets_mask.yaml-- properties marked as secrets
With --with-legacy-migration=v1, deletes cloud.json / oss.json
files for connectors whose registryOverrides.cloud.enabled or
registryOverrides.oss.enabled is false.
By default, injects connector quality metrics from the latest analytics
JSONL export into generated.metrics. Use --no-metrics for offline
scenarios.
With --force, resyncs all latest/ directories even if the version marker
matches the computed latest version.
Uses efficient glob patterns for scanning (no file downloads during discovery).
Requires GCS_CREDENTIALS environment variable to be set.
Parameters:
STORE, --store: Store target (e.g. 'coral:dev', 'coral:prod', 'coral:dev/prefix'). [required]--connector-name, --empty-connector-name: Only compile these connectors (can be repeated).--dry-run, --no-dry-run: Show what would be done without writing. [default: False]--with-secrets-mask, --no-with-secrets-mask: Also regenerate specs_secrets_mask.yaml by scanning all connector specs for airbyte_secret properties. [default: False]--with-legacy-migration: Run a one-time legacy migration step during compile. Currently supported: 'v1' — delete cloud.json / oss.json files for connectors whose registryOverrides.cloud.enabled or registryOverrides.oss.enabled is false. This cleans up artifacts produced by the legacy pipeline that did not respect the enabled flag.--with-metrics, --no-metrics: Inject latest connector quality metrics from the analytics JSONL export into generated.metrics. [default: True]--force, --no-force: Force resync of latest/ directories even if version markers are current. Useful when metadata changes without a version bump. [default: False]
airbyte-ops registry store delete-dev-latest
airbyte-ops registry store delete-dev-latest [OPTIONS] STORE
Delete all latest/ directories from a dev registry store.
Discovers every connector that has a latest/ directory and
deletes each one in parallel using a thread pool.
This is useful before a full re-compile to prove that latest/ directories can be correctly regenerated from versioned data.
Only dev stores are allowed (store must begin with 'coral:dev').
Requires GCS_CREDENTIALS environment variable to be set.
Parameters:
STORE, --store: Store target (must begin with 'coral:dev'). [required]--connector-name, --empty-connector-name: Only delete latest/ for these connectors (can be repeated).--dry-run, --no-dry-run: Show what would be done without deleting. [default: False]
airbyte-ops registry store compare
airbyte-ops registry store compare [OPTIONS] STORE REFERENCE-STORE
Compare a store against a reference store and report differences.
Evaluates the --store target against --reference-store and reports
per-connector artifact diffs and global index diffs.
Requires GCS_CREDENTIALS environment variable to be set.
Parameters:
STORE, --store: Store target being evaluated (e.g. 'coral:dev/20260306-mirror-compile'). [required]REFERENCE-STORE, --reference-store: Known-good reference store to compare against. [required]--connector-name, --empty-connector-name: Only compare these connectors (can be repeated).--with-artifacts, --no-artifacts: Compare per-connector artifact files (metadata.yaml, cloud.json, oss.json, spec.json). [default: True]--with-indexes, --no-indexes: Compare global registry index files (cloud_registry.json, oss_registry.json). [default: True]
airbyte-ops registry progressive-rollout
Progressive rollout lifecycle operations.
airbyte-ops registry progressive-rollout status
airbyte-ops registry progressive-rollout status [OPTIONS] NAME
Get progressive rollout status for a connector.
Parameters:
NAME, --name: Connector technical name (e.g., source-github). [required]--repo-path: Path to the Airbyte monorepo. Defaults to current directory. [default: /home/runner/work/airbyte-ops-mcp/airbyte-ops-mcp]--active-only, --with-terminal: Only return active non-terminal rollouts. [default: True]--limit: Maximum number of rollout records to return. [default: 100]
airbyte-ops registry progressive-rollout finalize-marker
airbyte-ops registry progressive-rollout finalize-marker [OPTIONS] NAME STORE OUTCOME
Rename an active progressive-rollout.yml marker to an audit marker.
Parameters:
NAME, --name: Connector technical name (e.g., source-github). [required]STORE, --store: Store target (e.g. 'coral:dev', 'coral:prod'). [required]OUTCOME, --outcome: Marker outcome used in the dated audit filename. [required] [choices: promoted, aborted]--version: Version to finalize. If omitted, exactly one active marker must exist.--dry-run, --no-dry-run: Show what would be done without making changes. [default: False]
airbyte-ops registry marketing-stubs
Marketing connector stubs GCS operations (whole-file sync).
airbyte-ops registry marketing-stubs check
airbyte-ops registry marketing-stubs check [OPTIONS] STORE
Compare local connector_stubs.json with the version in GCS.
This command reads the entire local connector_stubs.json file and compares it with the version currently published in GCS.
Exit codes:
0: Local file matches GCS (check passed) 1: Differences found (check failed)
Output:
STDOUT: JSON representation of the comparison result STDERR: Informational messages and comparison details
Parameters:
STORE, --store: Store target (e.g. 'coral:dev', 'coral:prod'). [required]--repo-root: Path to the airbyte-enterprise repository root. Defaults to current directory. [default: /home/runner/work/airbyte-ops-mcp/airbyte-ops-mcp]
airbyte-ops registry marketing-stubs sync
airbyte-ops registry marketing-stubs sync [OPTIONS] STORE
Sync local connector_stubs.json to GCS.
This command uploads the entire local connector_stubs.json file to GCS, replacing the existing file. Use this after merging changes to master in the airbyte-enterprise repository.
Exit codes:
0: Sync successful (or dry-run completed) 1: Error (file not found, validation failed, etc.)
Output:
STDOUT: JSON representation of the sync result STDERR: Informational messages and status updates
Parameters:
STORE, --store: Store target (e.g. 'coral:dev', 'coral:prod'). [required]--repo-root: Path to the airbyte-enterprise repository root. Defaults to current directory. [default: /home/runner/work/airbyte-ops-mcp/airbyte-ops-mcp]--dry-run, --no-dry-run: Show what would be uploaded without making changes. [default: False]
1# Copyright (c) 2025 Airbyte, Inc., all rights reserved. 2"""CLI commands for connector registry operations. 3 4This module provides CLI wrappers for registry operations. The core logic 5lives in the `airbyte_ops_mcp.registry` capability module. 6 7Command groups: 8 airbyte-ops registry connector list - List all connectors in registry 9 airbyte-ops registry connector-version list - List versions for a connector 10 airbyte-ops registry connector-version next - Compute next version tag (prerelease/RC) 11 airbyte-ops registry connector-version yank - Mark a connector version as yanked 12 airbyte-ops registry connector-version unyank - Remove yank marker from a connector version 13 airbyte-ops registry connector-version metadata get - Read connector metadata from GCS 14 airbyte-ops registry connector-version artifacts generate - Generate version artifacts locally via docker 15 airbyte-ops registry connector-version artifacts publish - Publish version artifacts to GCS 16 airbyte-ops registry store mirror --local|--gcs-bucket|--s3-bucket - Mirror entire registry for testing 17 airbyte-ops registry store compile --store coral:dev|coral:prod - Compile registry indexes and sync latest/ dirs 18 airbyte-ops registry marketing-stubs sync --store coral:prod - Sync connector_stubs.json to GCS 19 airbyte-ops registry marketing-stubs check --store coral:prod - Compare local file with GCS 20 21## CLI reference 22 23The commands below are regenerated by `poe docs-generate` via cyclopts's 24programmatic docs API; see `docs/generate_cli.py`. 25 26.. include:: ../../../docs/generated/cli/registry.md 27 :start-line: 2 28""" 29 30from __future__ import annotations 31 32# Hide Python-level members from the pdoc page for this module; the rendered 33# docs for this CLI group come entirely from the grafted `.. include::` in 34# the module docstring above. 35__all__: list[str] = [] 36 37import sys 38from pathlib import Path 39from typing import Annotated, Literal 40 41import yaml 42from cyclopts import Parameter 43 44from airbyte_ops_mcp.cli._base import App, app 45from airbyte_ops_mcp.cli._shared import ( 46 error_console, 47 exit_with_error, 48 print_error, 49 print_json, 50 print_success, 51) 52from airbyte_ops_mcp.github_api import ( 53 get_file_contents_at_ref, 54 resolve_default_github_token, 55) 56from airbyte_ops_mcp.mcp.prerelease import ( 57 compute_prerelease_docker_image_tag, 58) 59from airbyte_ops_mcp.registry import ( 60 resolve_registry_store, 61) 62from airbyte_ops_mcp.registry._enums import ( 63 ConnectorLanguage, 64 ConnectorType, 65 SupportLevel, 66) 67from airbyte_ops_mcp.registry.compare import compare_stores 68from airbyte_ops_mcp.registry.connector_stubs import ( 69 CONNECTOR_STUBS_FILE, 70) 71from airbyte_ops_mcp.registry.generate import generate_version_artifacts 72from airbyte_ops_mcp.registry.progressive_rollout_status import ( 73 get_connector_rollout_status, 74) 75from airbyte_ops_mcp.registry.registry_store_base import ( 76 Registry, 77 get_registry, 78) 79 80# Create the registry sub-app 81registry_app = App( 82 name="registry", help="Connector registry operations (GCS metadata service)." 83) 84app.command(registry_app) 85 86# Create the connector sub-app under registry 87connector_app = App(name="connector", help="Connector listing operations.") 88registry_app.command(connector_app) 89 90# Create the connector-version sub-app under registry 91connector_version_app = App( 92 name="connector-version", 93 help="Connector version operations (list, yank, unyank, artifacts).", 94) 95registry_app.command(connector_version_app) 96 97# Create the metadata sub-app under connector-version 98metadata_app = App( 99 name="metadata", 100 help="Connector metadata inspection.", 101) 102connector_version_app.command(metadata_app) 103 104# Create the artifacts sub-app under connector-version 105artifacts_app = App( 106 name="artifacts", 107 help="Version artifact generation and publishing.", 108) 109connector_version_app.command(artifacts_app) 110 111# Create the store sub-app under registry (whole-registry operations) 112store_app = App( 113 name="store", 114 help="Whole-registry store operations (mirror, compile).", 115) 116registry_app.command(store_app) 117 118# Create the progressive-rollout sub-app under registry 119progressive_rollout_app = App( 120 name="progressive-rollout", 121 help="Progressive rollout lifecycle operations.", 122) 123registry_app.command(progressive_rollout_app) 124 125# Create the marketing-stubs sub-app under registry (for whole-file GCS operations) 126marketing_stubs_app = App( 127 name="marketing-stubs", 128 help="Marketing connector stubs GCS operations (whole-file sync).", 129) 130registry_app.command(marketing_stubs_app) 131 132 133AIRBYTE_REPO_OWNER = "airbytehq" 134AIRBYTE_ENTERPRISE_REPO_NAME = "airbyte-enterprise" 135AIRBYTE_REPO_NAME = "airbyte" 136CONNECTOR_PATH_PREFIX = "airbyte-integrations/connectors" 137 138 139def _resolve_store(store: str) -> Registry: 140 """Resolve `--store` to a `Registry` instance. 141 142 Wraps `resolve_registry_store` and `get_registry`, converting 143 `ValueError` into a user-friendly CLI error via `exit_with_error`. 144 """ 145 try: 146 resolved = resolve_registry_store(store=store) 147 except ValueError as e: 148 exit_with_error(str(e)) 149 raise # unreachable; satisfies type checker 150 return get_registry(resolved) 151 152 153def _get_connector_version_from_github( 154 connector_name: str, 155 ref: str, 156 token: str | None = None, 157) -> str | None: 158 """Fetch connector version from metadata.yaml via GitHub API. 159 160 Args: 161 connector_name: Connector name (e.g., "source-github") 162 ref: Git ref (commit SHA, branch name, or tag) 163 token: GitHub API token (optional for public repos) 164 165 Returns: 166 Version string from metadata.yaml, or None if not found. 167 """ 168 path = f"{CONNECTOR_PATH_PREFIX}/{connector_name}/metadata.yaml" 169 contents = get_file_contents_at_ref( 170 owner=AIRBYTE_REPO_OWNER, 171 repo=AIRBYTE_REPO_NAME, 172 path=path, 173 ref=ref, 174 token=token, 175 ) 176 if contents is None: 177 return None 178 179 metadata = yaml.safe_load(contents) 180 return metadata.get("data", {}).get("dockerImageTag") 181 182 183@connector_version_app.command(name="next") 184def compute_next_version( 185 name: Annotated[ 186 str, 187 Parameter(help="Connector name (e.g., 'source-github')."), 188 ], 189 sha: Annotated[ 190 str, 191 Parameter(help="Git commit SHA (full or at least 7 characters)."), 192 ], 193 base_version: Annotated[ 194 str | None, 195 Parameter( 196 help="Base version override. If not provided, fetched from metadata.yaml at the given SHA." 197 ), 198 ] = None, 199) -> None: 200 """Compute the next version tag for a connector. 201 202 Outputs the version tag to stdout for easy capture in shell scripts. 203 This is the single source of truth for pre-release version format. 204 205 The command fetches the connector's metadata.yaml from GitHub at the given SHA 206 to determine the base version. It also compares against the master branch and 207 prints a warning to stderr if no version bump is detected. 208 209 If --base-version is provided, it is used directly instead of fetching from GitHub. 210 211 Examples: 212 airbyte-ops registry connector-version next --name source-github --sha abcdef1234567 213 # Output: 1.2.3-preview.abcdef1 214 215 airbyte-ops registry connector-version next --name source-github --sha abcdef1234567 --base-version 1.2.3 216 # Output: 1.2.3-preview.abcdef1 (uses provided version, skips GitHub API) 217 """ 218 # Try to get a GitHub token (optional, but helps avoid rate limiting) 219 # Token resolution may fail if no token is configured, which is fine for public repos 220 token: str | None = None 221 token = resolve_default_github_token(allow_none=True) 222 223 # Determine base version 224 version: str 225 if base_version: 226 version = base_version 227 else: 228 # Fetch version from metadata.yaml at the given SHA 229 fetched_version = _get_connector_version_from_github(name, sha, token) 230 if fetched_version is None: 231 print( 232 f"Error: Could not fetch metadata.yaml for {name} at ref {sha}", 233 file=sys.stderr, 234 ) 235 sys.exit(1) 236 version = fetched_version 237 238 # Compare with master branch version and warn if no bump detected 239 master_version = _get_connector_version_from_github(name, "master", token) 240 if master_version and master_version == version: 241 print( 242 f"Warning: No version bump detected for {name}. " 243 f"Version {version} matches master branch.", 244 file=sys.stderr, 245 ) 246 247 # Compute and output the prerelease tag 248 tag = compute_prerelease_docker_image_tag(version, sha) 249 print(tag) 250 251 252@progressive_rollout_app.command(name="status") 253def progressive_rollout_status( 254 name: Annotated[ 255 str, 256 Parameter(help="Connector technical name (e.g., source-github)."), 257 ], 258 *, 259 repo_path: Annotated[ 260 Path, 261 Parameter(help="Path to the Airbyte monorepo. Defaults to current directory."), 262 ] = Path.cwd(), 263 active_only: Annotated[ 264 bool, 265 Parameter( 266 help="Only return active non-terminal rollouts.", 267 negative="--with-terminal", 268 ), 269 ] = True, 270 limit: Annotated[ 271 int, 272 Parameter(help="Maximum number of rollout records to return."), 273 ] = 100, 274) -> None: 275 """Get progressive rollout status for a connector.""" 276 if not repo_path.exists(): 277 exit_with_error(f"Repository path not found: {repo_path}") 278 try: 279 result = get_connector_rollout_status( 280 repo_path=repo_path, 281 connector_name=name, 282 active_only=active_only, 283 limit=limit, 284 ) 285 except (FileNotFoundError, ValueError) as e: 286 exit_with_error(str(e)) 287 288 print_json(result.model_dump()) 289 290 291@progressive_rollout_app.command(name="finalize-marker") 292def progressive_rollout_finalize_marker( 293 name: Annotated[ 294 str, 295 Parameter(help="Connector technical name (e.g., source-github)."), 296 ], 297 store: Annotated[ 298 str, 299 Parameter(help="Store target (e.g. 'coral:dev', 'coral:prod')."), 300 ], 301 outcome: Annotated[ 302 Literal["promoted", "aborted"], 303 Parameter(help="Marker outcome used in the dated audit filename."), 304 ], 305 *, 306 version: Annotated[ 307 str | None, 308 Parameter( 309 help="Version to finalize. If omitted, exactly one active marker must exist." 310 ), 311 ] = None, 312 dry_run: Annotated[ 313 bool, 314 Parameter(help="Show what would be done without making changes."), 315 ] = False, 316) -> None: 317 """Rename an active `progressive-rollout.yml` marker to an audit marker.""" 318 registry = _resolve_store(store) 319 result = registry.finalize_progressive_rollout_marker( 320 connector_name=name, 321 outcome=outcome, 322 version=version, 323 dry_run=dry_run, 324 ) 325 print_json(result.to_dict()) 326 if result.success: 327 print_success(result.message) 328 else: 329 exit_with_error(result.message, code=1) 330 331 332# ============================================================================= 333# REGISTRY I/O - READ COMMANDS 334# ============================================================================= 335 336 337@metadata_app.command(name="get") 338def get_connector_version_metadata_cmd( 339 name: Annotated[ 340 str, 341 Parameter( 342 help="Connector name (e.g., 'source-faker', 'destination-postgres')." 343 ), 344 ], 345 store: Annotated[ 346 str, 347 Parameter( 348 help="Store target (e.g. 'coral:dev', 'coral:prod').", 349 ), 350 ], 351 *, 352 version: Annotated[ 353 str, 354 Parameter(help="Version to read (e.g., 'latest', '1.2.3')."), 355 ] = "latest", 356 format: Annotated[ 357 Literal["json", "raw"], 358 Parameter(help="Output format: 'json' for JSON, 'raw' for YAML."), 359 ] = "json", 360) -> None: 361 """Read a connector version's metadata from the registry. 362 363 Returns the full metadata.yaml content for a connector at the specified version. 364 365 Requires GCS_CREDENTIALS environment variable to be set. 366 367 Examples: 368 airbyte-ops registry connector-version metadata get --name source-faker --store coral:dev 369 airbyte-ops registry connector-version metadata get --name source-faker --store coral:dev --version 6.2.38 370 airbyte-ops registry connector-version metadata get --name source-faker --store coral:prod 371 """ 372 registry = _resolve_store(store) 373 374 try: 375 metadata = registry.get_connector_metadata( 376 connector_name=name, 377 version=version, 378 ) 379 except FileNotFoundError as e: 380 exit_with_error(str(e), code=1) 381 except Exception as e: 382 exit_with_error(f"Error reading metadata: {e}", code=1) 383 384 if format == "json": 385 print_json(metadata) 386 else: 387 print(yaml.dump(metadata, default_flow_style=False)) 388 389 390@connector_app.command(name="list") 391def list_connectors_cmd( 392 store: Annotated[ 393 str, 394 Parameter( 395 help="Store target (e.g. 'coral:dev', 'coral:prod').", 396 ), 397 ], 398 *, 399 certified_only: Annotated[ 400 bool, 401 Parameter(help="Include only certified connectors."), 402 ] = False, 403 support_level: Annotated[ 404 SupportLevel | None, 405 Parameter( 406 help=( 407 "Exact support level to match " 408 "(e.g. `certified`, `community`, `archived`)." 409 ) 410 ), 411 ] = None, 412 min_support_level: Annotated[ 413 SupportLevel | None, 414 Parameter( 415 help=( 416 "Minimum support level (inclusive). " 417 "Levels from lowest to highest: `archived`, `community`, `certified`." 418 ) 419 ), 420 ] = None, 421 connector_type: Annotated[ 422 ConnectorType | None, 423 Parameter(help="Filter by connector type: `source` or `destination`."), 424 ] = None, 425 language: Annotated[ 426 ConnectorLanguage | None, 427 Parameter( 428 help=( 429 "Filter by implementation language " 430 "(e.g. `python`, `java`, `manifest-only`)." 431 ) 432 ), 433 ] = None, 434 format: Annotated[ 435 Literal["json", "text"], 436 Parameter( 437 help="Output format: 'json' for JSON array, 'text' for newline-separated." 438 ), 439 ] = "json", 440) -> None: 441 """List connectors in the registry. 442 443 When filters are applied, reads the compiled `cloud_registry.json` index 444 for fast lookups. Without filters, falls back to scanning individual 445 metadata blobs (captures all connectors including OSS-only). 446 447 Requires GCS_CREDENTIALS environment variable to be set. 448 """ 449 registry = _resolve_store(store) 450 451 # `--certified-only` is sugar for `--support-level certified`. 452 effective_support_level = support_level 453 if certified_only: 454 if support_level and support_level != SupportLevel.CERTIFIED: 455 exit_with_error( 456 "`--certified-only` conflicts with `--support-level " 457 f"{support_level}`. Use one or the other.", 458 code=1, 459 ) 460 effective_support_level = SupportLevel.CERTIFIED 461 462 try: 463 connectors = registry.list_connectors( 464 support_level=effective_support_level, 465 min_support_level=min_support_level, 466 connector_type=connector_type, 467 language=language, 468 ) 469 except Exception as e: 470 exit_with_error(f"Error listing connectors: {e}", code=1) 471 472 if format == "json": 473 print_json({"connectors": connectors, "count": len(connectors)}) 474 else: 475 for connector in connectors: 476 print(connector) 477 478 479@connector_version_app.command(name="list") 480def list_connector_versions_cmd( 481 name: Annotated[ 482 str, 483 Parameter(help="Connector name (e.g., 'source-faker')."), 484 ], 485 store: Annotated[ 486 str, 487 Parameter( 488 help="Store target (e.g. 'coral:dev', 'coral:prod').", 489 ), 490 ], 491 *, 492 format: Annotated[ 493 Literal["json", "text"], 494 Parameter( 495 help="Output format: 'json' for JSON array, 'text' for newline-separated." 496 ), 497 ] = "json", 498) -> None: 499 """List all versions of a connector in the registry. 500 501 Scans the registry bucket to find all versions of a specific connector. 502 503 Requires GCS_CREDENTIALS environment variable to be set. 504 """ 505 registry = _resolve_store(store) 506 507 try: 508 versions = registry.list_connector_versions( 509 connector_name=name, 510 ) 511 except Exception as e: 512 exit_with_error(f"Error listing versions: {e}", code=1) 513 514 if format == "json": 515 print_json({"connector": name, "versions": versions, "count": len(versions)}) 516 else: 517 for v in versions: 518 print(v) 519 520 521@marketing_stubs_app.command(name="check") 522def marketing_stubs_check( 523 store: Annotated[ 524 str, 525 Parameter( 526 help="Store target (e.g. 'coral:dev', 'coral:prod').", 527 ), 528 ], 529 *, 530 repo_root: Annotated[ 531 Path, 532 Parameter( 533 help="Path to the airbyte-enterprise repository root. Defaults to current directory." 534 ), 535 ] = Path.cwd(), 536) -> None: 537 """Compare local connector_stubs.json with the version in GCS. 538 539 This command reads the entire local connector_stubs.json file and compares it 540 with the version currently published in GCS. 541 542 Exit codes: 543 0: Local file matches GCS (check passed) 544 1: Differences found (check failed) 545 546 Output: 547 STDOUT: JSON representation of the comparison result 548 STDERR: Informational messages and comparison details 549 550 Example: 551 airbyte-ops registry marketing-stubs check --store coral:prod --repo-root /path/to/airbyte-enterprise 552 airbyte-ops registry marketing-stubs check --store coral:dev 553 """ 554 registry = _resolve_store(store) 555 556 try: 557 result = registry.marketing_stubs_check(repo_root=repo_root) 558 except FileNotFoundError as e: 559 exit_with_error(str(e)) 560 except ValueError as e: 561 exit_with_error(str(e)) 562 563 error_console.print( 564 f"Comparing local {CONNECTOR_STUBS_FILE} with {result.get('bucket', '')}/{result.get('path', '')}" 565 ) 566 567 differences = result.get("differences", []) 568 if differences: 569 error_console.print( 570 f"[yellow]Warning:[/yellow] {len(differences)} difference(s) found:" 571 ) 572 for diff in differences: 573 error_console.print(f" {diff['id']}: {diff['status']}") 574 print_json(result) 575 sys.exit(1) 576 577 error_console.print( 578 f"[green]Local file is in sync with GCS ({result.get('local_count', 0)} stubs)[/green]" 579 ) 580 print_json(result) 581 582 583@marketing_stubs_app.command(name="sync") 584def marketing_stubs_sync( 585 store: Annotated[ 586 str, 587 Parameter( 588 help="Store target (e.g. 'coral:dev', 'coral:prod').", 589 ), 590 ], 591 *, 592 repo_root: Annotated[ 593 Path, 594 Parameter( 595 help="Path to the airbyte-enterprise repository root. Defaults to current directory." 596 ), 597 ] = Path.cwd(), 598 dry_run: Annotated[ 599 bool, 600 Parameter(help="Show what would be uploaded without making changes."), 601 ] = False, 602) -> None: 603 """Sync local connector_stubs.json to GCS. 604 605 This command uploads the entire local connector_stubs.json file to GCS, 606 replacing the existing file. Use this after merging changes to master 607 in the airbyte-enterprise repository. 608 609 Exit codes: 610 0: Sync successful (or dry-run completed) 611 1: Error (file not found, validation failed, etc.) 612 613 Output: 614 STDOUT: JSON representation of the sync result 615 STDERR: Informational messages and status updates 616 617 Example: 618 airbyte-ops registry marketing-stubs sync --store coral:prod --repo-root /path/to/airbyte-enterprise 619 airbyte-ops registry marketing-stubs sync --store coral:dev 620 airbyte-ops registry marketing-stubs sync --store coral:dev --dry-run 621 """ 622 registry = _resolve_store(store) 623 624 try: 625 result = registry.marketing_stubs_sync( 626 repo_root=repo_root, 627 dry_run=dry_run, 628 ) 629 except FileNotFoundError as e: 630 exit_with_error(str(e)) 631 except ValueError as e: 632 exit_with_error(str(e)) 633 634 bucket_name = result.get("bucket", "") 635 path = result.get("path", "") 636 stub_count = result.get("stub_count", 0) 637 638 if dry_run: 639 error_console.print( 640 f"[DRY RUN] Would upload {stub_count} stubs to {bucket_name}/{path}" 641 ) 642 else: 643 error_console.print( 644 f"[green]Synced {stub_count} stubs to {bucket_name}/{path}[/green]" 645 ) 646 print_json(result) 647 648 649# ============================================================================= 650# REGISTRY REBUILD COMMANDS 651# ============================================================================= 652 653 654@store_app.command(name="mirror") 655def mirror_cmd( 656 local: Annotated[ 657 bool, 658 Parameter( 659 help="Write output to a local directory. Mutually exclusive with --gcs-bucket and --s3-bucket.", 660 negative="", 661 ), 662 ] = False, 663 gcs_bucket: Annotated[ 664 str | None, 665 Parameter( 666 help="Write output to a GCS bucket. Must not be the prod bucket. " 667 "Mutually exclusive with --local and --s3-bucket.", 668 ), 669 ] = None, 670 s3_bucket: Annotated[ 671 str | None, 672 Parameter( 673 help="Write output to an S3 bucket. " 674 "Mutually exclusive with --local and --gcs-bucket.", 675 ), 676 ] = None, 677 output_path_root: Annotated[ 678 str | None, 679 Parameter( 680 help="Root path/prefix for the output. For --local, this is a directory path " 681 "(defaults to a new temp dir if omitted). For --gcs-bucket/--s3-bucket, " 682 "this prefix is prepended to all blob paths.", 683 ), 684 ] = None, 685 dry_run: Annotated[ 686 bool, 687 Parameter(help="Show what would be rebuilt without writing any files."), 688 ] = False, 689 source_store: Annotated[ 690 str, 691 Parameter( 692 help="Source store to read from (e.g. 'coral:prod', 'coral:dev').", 693 ), 694 ] = "coral:prod", 695 connector_name: Annotated[ 696 tuple[str, ...] | None, 697 Parameter( 698 help="Only rebuild these connectors (by name). " 699 "Can be specified multiple times, e.g. " 700 "--connector-name source-faker --connector-name destination-bigquery.", 701 ), 702 ] = None, 703) -> None: 704 """Create a mirror of the connector registry from the source store. 705 706 Reads all connector metadata from the source store and copies it 707 to the specified output target. Supports local filesystem, GCS, and S3 708 as output targets via fsspec. 709 710 Output targets are mutually exclusive: specify exactly one of 711 --local, --gcs-bucket, or --s3-bucket. 712 713 The production bucket is categorically disallowed as an output target. 714 715 To clean up legacy artifacts (e.g. disabled strict-encrypt connectors) 716 after mirroring, run compile with `--with-legacy-migration v1`:: 717 718 airbyte-ops registry store compile --store coral:dev/my-prefix \\ 719 --with-legacy-migration v1 720 721 Examples: 722 airbyte-ops registry store mirror --local 723 airbyte-ops registry store mirror --local --source-store coral:dev 724 airbyte-ops registry store mirror --gcs-bucket dev-airbyte-cloud-connector-metadata-service-2 \\ 725 --output-path-root test-run-123 726 airbyte-ops registry store mirror --s3-bucket my-test-bucket --dry-run 727 """ 728 # Validate mutually exclusive output targets 729 targets = [local, gcs_bucket is not None, s3_bucket is not None] 730 if sum(targets) != 1: 731 exit_with_error( 732 "Specify exactly one output target: --local, --gcs-bucket, or --s3-bucket." 733 ) 734 735 if local: 736 output_mode = "local" 737 elif gcs_bucket is not None: 738 output_mode = "gcs" 739 else: 740 output_mode = "s3" 741 742 registry = _resolve_store(source_store) 743 744 result = registry.mirror( 745 output_mode=output_mode, 746 output_path_root=output_path_root, 747 gcs_bucket=gcs_bucket, 748 s3_bucket=s3_bucket, 749 dry_run=dry_run, 750 connector_name=list(connector_name) if connector_name else None, 751 ) 752 753 print_json( 754 { 755 "status": result.status, 756 "source_bucket": result.source_bucket, 757 "output_mode": result.output_mode, 758 "output_root": result.output_root, 759 "connectors_processed": result.connectors_processed, 760 "blobs_copied": result.blobs_copied, 761 "blobs_skipped": result.blobs_skipped, 762 "error_count": len(result.errors), 763 "dry_run": result.dry_run, 764 } 765 ) 766 767 if result.errors: 768 for err in result.errors[:10]: 769 print_error(err) 770 if len(result.errors) > 10: 771 print_error(f"... and {len(result.errors) - 10} more errors") 772 773 if result.status == "success": 774 print_success(result.summary()) 775 elif result.status == "dry-run": 776 print_success(f"[DRY RUN] {result.summary()}") 777 else: 778 error_console.print(f"[yellow]{result.summary()}[/yellow]") 779 780 781# ============================================================================= 782# VERSION YANK COMMANDS 783# ============================================================================= 784 785 786@connector_version_app.command(name="yank") 787def yank_cmd( 788 name: Annotated[ 789 str, 790 Parameter(help="Connector name (e.g., 'source-faker')."), 791 ], 792 version: Annotated[ 793 str, 794 Parameter(help="Version to yank (e.g., '1.2.3')."), 795 ], 796 store: Annotated[ 797 str, 798 Parameter( 799 help="Store target (e.g. 'coral:dev', 'coral:prod').", 800 ), 801 ], 802 *, 803 reason: Annotated[ 804 str, 805 Parameter(help="Reason for yanking this version."), 806 ] = "", 807 approval_url: Annotated[ 808 str, 809 Parameter(help="Approval evidence URL to record in the yank marker."), 810 ] = "", 811 dry_run: Annotated[ 812 bool, 813 Parameter(help="Show what would be done without making changes."), 814 ] = False, 815) -> None: 816 """Mark a connector version as yanked. 817 818 Writes a version-yank.yml marker file to the version's directory in GCS. 819 Yanked versions are excluded when determining the latest version of a 820 connector. 821 822 Requires GCS_CREDENTIALS environment variable to be set. 823 824 Examples: 825 airbyte-ops registry connector-version yank --name source-faker --version 1.2.3 --store coral:dev 826 airbyte-ops registry connector-version yank --name source-faker --version 1.2.3 --store coral:dev --reason "Critical bug" 827 airbyte-ops registry connector-version yank --name source-faker --version 1.2.3 --store coral:prod 828 """ 829 registry = _resolve_store(store) 830 831 result = registry.yank( 832 connector_name=name, 833 version=version, 834 reason=reason, 835 approval_url=approval_url, 836 dry_run=dry_run, 837 ) 838 839 print_json(result.to_dict()) 840 841 if result.success: 842 print_success(result.message) 843 if not dry_run: 844 error_console.print( 845 "\n[yellow]Note:[/yellow] The registry indexes are now stale. " 846 "To update them, run:\n\n" 847 f" airbyte-ops registry store compile --store {store}\n\n" 848 "Or wait for the next scheduled compile operation." 849 ) 850 else: 851 exit_with_error(result.message, code=1) 852 853 854@connector_version_app.command(name="unyank") 855def unyank_cmd( 856 name: Annotated[ 857 str, 858 Parameter(help="Connector name (e.g., 'source-faker')."), 859 ], 860 version: Annotated[ 861 str, 862 Parameter(help="Version to unyank (e.g., '1.2.3')."), 863 ], 864 store: Annotated[ 865 str, 866 Parameter( 867 help="Store target (e.g. 'coral:dev', 'coral:prod').", 868 ), 869 ], 870 *, 871 dry_run: Annotated[ 872 bool, 873 Parameter(help="Show what would be done without making changes."), 874 ] = False, 875) -> None: 876 """Rename the active yank marker to a dated audit marker. 877 878 Moves `version-yank.yml` to `version-unyanked-yyyymmdd.yml`, making the 879 version eligible again when determining the latest version. 880 881 Requires GCS_CREDENTIALS environment variable to be set. 882 883 Examples: 884 airbyte-ops registry connector-version unyank --name source-faker --version 1.2.3 --store coral:dev 885 airbyte-ops registry connector-version unyank --name source-faker --version 1.2.3 --store coral:prod 886 """ 887 registry = _resolve_store(store) 888 889 result = registry.unyank( 890 connector_name=name, 891 version=version, 892 dry_run=dry_run, 893 ) 894 895 print_json(result.to_dict()) 896 897 if result.success: 898 print_success(result.message) 899 if not dry_run: 900 error_console.print( 901 "\n[yellow]Note:[/yellow] The registry indexes are now stale. " 902 "To update them, run:\n\n" 903 f" airbyte-ops registry store compile --store {store}\n\n" 904 "Or wait for the next scheduled compile operation." 905 ) 906 else: 907 exit_with_error(result.message, code=1) 908 909 910# ============================================================================= 911# ARTIFACT GENERATION COMMANDS 912# ============================================================================= 913 914 915@artifacts_app.command(name="generate") 916def generate_version_artifacts_cmd( 917 metadata_file: Annotated[ 918 Path, 919 Parameter(help="Path to the connector's metadata.yaml file."), 920 ], 921 docker_image: Annotated[ 922 str, 923 Parameter( 924 help="Docker image to run spec against (e.g., 'airbyte/source-faker:6.2.38')." 925 ), 926 ], 927 output_dir: Annotated[ 928 Path | None, 929 Parameter( 930 help="Directory to write artifacts to. If not specified, a temp directory is created." 931 ), 932 ] = None, 933 repo_root: Annotated[ 934 Path | None, 935 Parameter( 936 help=( 937 "Root of the Airbyte repo checkout (for resolving doc.md). " 938 "If not specified, inferred by walking up from metadata-file." 939 ), 940 ), 941 ] = None, 942 dry_run: Annotated[ 943 bool, 944 Parameter( 945 help="Show what would be generated without running docker or writing files." 946 ), 947 ] = False, 948 with_validate: Annotated[ 949 bool, 950 Parameter( 951 help=( 952 "Run metadata validators after generation (default: enabled). " 953 "Use --no-validate to skip." 954 ), 955 negative="--no-validate", 956 ), 957 ] = True, 958 with_sbom: Annotated[ 959 bool, 960 Parameter( 961 help=( 962 "Generate spdx.json (SBOM) for connectors " 963 "(default: enabled). Use --no-sbom to skip." 964 ), 965 negative="--no-sbom", 966 ), 967 ] = True, 968 with_dependency_dump: Annotated[ 969 bool, 970 Parameter( 971 help=( 972 "Generate dependencies.json for Python connectors " 973 "(default: enabled). Use --no-dependency-dump to skip." 974 ), 975 negative="--no-dependency-dump", 976 ), 977 ] = True, 978) -> None: 979 """Generate version artifacts for a connector locally. 980 981 Runs the connector's docker image with DEPLOYMENT_MODE=cloud and 982 DEPLOYMENT_MODE=oss to obtain both spec variants, then generates 983 the registry entries (cloud.json, oss.json) by applying 984 registryOverrides from the metadata. 985 986 The generated metadata.yaml is enriched with git commit info, SBOM URL, 987 and (when applicable) components SHA before writing. Validation is run 988 after generation by default; pass `--no-validate` to skip. 989 990 This is a local-only operation -- no files are uploaded to GCS. 991 Use `artifacts publish` to upload generated artifacts to GCS. 992 993 Examples: 994 airbyte-ops registry connector-version artifacts generate \\ 995 --metadata-file path/to/metadata.yaml \\ 996 --docker-image airbyte/source-faker:6.2.38 997 998 airbyte-ops registry connector-version artifacts generate \\ 999 --metadata-file path/to/metadata.yaml \\ 1000 --docker-image airbyte/source-faker:6.2.38 \\ 1001 --output-dir ./artifacts --with-validate 1002 """ 1003 result = generate_version_artifacts( 1004 metadata_file=metadata_file, 1005 docker_image=docker_image, 1006 output_dir=output_dir, 1007 repo_root=repo_root, 1008 dry_run=dry_run, 1009 with_validate=with_validate, 1010 with_dependency_dump=with_dependency_dump, 1011 with_sbom=with_sbom, 1012 ) 1013 1014 print_json(result.to_dict()) 1015 1016 if result.success: 1017 print_success( 1018 f"Generated {len(result.artifacts_written)} artifacts to {result.output_dir}" 1019 ) 1020 else: 1021 all_errors = result.errors + result.validation_errors 1022 exit_with_error( 1023 f"Generation completed with {len(all_errors)} error(s): " 1024 + "; ".join(all_errors), 1025 code=1, 1026 ) 1027 1028 1029@artifacts_app.command(name="publish") 1030def publish_version_artifacts_cmd( 1031 name: Annotated[ 1032 str, 1033 Parameter(help="Connector name (e.g., 'source-faker')."), 1034 ], 1035 version: Annotated[ 1036 str, 1037 Parameter(help="Version to publish artifacts for (e.g., '1.2.3')."), 1038 ], 1039 artifacts_dir: Annotated[ 1040 Path, 1041 Parameter( 1042 help="Directory containing generated artifacts to publish (from 'artifacts generate')." 1043 ), 1044 ], 1045 store: Annotated[ 1046 str, 1047 Parameter( 1048 help="Store target (e.g. 'coral:dev', 'coral:prod', 'coral:dev/prefix').", 1049 ), 1050 ], 1051 *, 1052 dry_run: Annotated[ 1053 bool, 1054 Parameter(help="Show what would be published without writing to GCS."), 1055 ] = False, 1056 with_validate: Annotated[ 1057 bool, 1058 Parameter( 1059 help=( 1060 "Validate metadata before uploading (default: enabled). " 1061 "Use --no-validate to skip." 1062 ), 1063 negative="--no-validate", 1064 ), 1065 ] = True, 1066) -> None: 1067 """Publish version artifacts to GCS using fsspec rsync. 1068 1069 Uploads locally generated artifacts (from `artifacts generate`) to the 1070 versioned path in GCS. By default, metadata is validated before upload; 1071 pass `--no-validate` to skip. 1072 1073 Uses `--store` to select the destination store and environment: 1074 1075 * `coral:dev` → coral dev bucket at root 1076 * `coral:prod` → coral prod bucket at root 1077 * `coral:dev/aj-test100` → coral dev bucket under `aj-test100/` prefix 1078 1079 Requires GCS_CREDENTIALS environment variable to be set. 1080 1081 Examples: 1082 airbyte-ops registry connector-version artifacts publish \\ 1083 --name source-faker --version 6.2.38 \\ 1084 --artifacts-dir ./artifacts --store coral:dev --with-validate 1085 1086 airbyte-ops registry connector-version artifacts publish \\ 1087 --name source-faker --version 6.2.38 \\ 1088 --artifacts-dir ./artifacts --store coral:prod 1089 1090 airbyte-ops registry connector-version artifacts publish \\ 1091 --name source-faker --version 6.2.38 \\ 1092 --artifacts-dir ./artifacts --store coral:dev/aj-test100 1093 """ 1094 registry = _resolve_store(store) 1095 1096 result = registry.publish_version_artifacts( 1097 connector_name=name, 1098 version=version, 1099 artifacts_dir=artifacts_dir, 1100 dry_run=dry_run, 1101 with_validate=with_validate, 1102 ) 1103 1104 print_json( 1105 { 1106 "status": result.status, 1107 "connector_name": result.connector_name, 1108 "version": result.version, 1109 "target": result.target, 1110 "gcs_destination": result.gcs_destination, 1111 "files_uploaded": result.files_uploaded, 1112 "errors": result.errors, 1113 "validation_errors": result.validation_errors, 1114 "dry_run": result.dry_run, 1115 } 1116 ) 1117 1118 if result.success: 1119 print_success( 1120 f"Published {len(result.files_uploaded)} artifacts for " 1121 f"{result.connector_name}@{result.version} → {result.gcs_destination}" 1122 ) 1123 else: 1124 all_errors = result.errors + result.validation_errors 1125 exit_with_error( 1126 f"Publish completed with {len(all_errors)} error(s): " 1127 + "; ".join(all_errors), 1128 code=1, 1129 ) 1130 1131 1132# ============================================================================= 1133# COMPILE COMMAND 1134# ============================================================================= 1135 1136 1137@store_app.command(name="compile") 1138def compile_cmd( 1139 store: Annotated[ 1140 str, 1141 Parameter( 1142 help="Store target (e.g. 'coral:dev', 'coral:prod', 'coral:dev/prefix').", 1143 ), 1144 ], 1145 *, 1146 connector_name: Annotated[ 1147 tuple[str, ...] | None, 1148 Parameter( 1149 help="Only compile these connectors (can be repeated).", 1150 ), 1151 ] = None, 1152 dry_run: Annotated[ 1153 bool, 1154 Parameter(help="Show what would be done without writing."), 1155 ] = False, 1156 with_secrets_mask: Annotated[ 1157 bool, 1158 Parameter( 1159 help=( 1160 "Also regenerate specs_secrets_mask.yaml by scanning all " 1161 "connector specs for airbyte_secret properties." 1162 ), 1163 ), 1164 ] = False, 1165 with_legacy_migration: Annotated[ 1166 str | None, 1167 Parameter( 1168 help=( 1169 "Run a one-time legacy migration step during compile. " 1170 "Currently supported: 'v1' — delete cloud.json / oss.json " 1171 "files for connectors whose registryOverrides.cloud.enabled " 1172 "or registryOverrides.oss.enabled is false. This cleans up " 1173 "artifacts produced by the legacy pipeline that did not " 1174 "respect the enabled flag." 1175 ), 1176 ), 1177 ] = None, 1178 with_metrics: Annotated[ 1179 bool, 1180 Parameter( 1181 help=( 1182 "Inject latest connector quality metrics from the analytics " 1183 "JSONL export into generated.metrics." 1184 ), 1185 negative="--no-metrics", 1186 ), 1187 ] = True, 1188 force: Annotated[ 1189 bool, 1190 Parameter( 1191 help=( 1192 "Force resync of latest/ directories even if version markers are current. " 1193 "Useful when metadata changes without a version bump." 1194 ), 1195 ), 1196 ] = False, 1197) -> None: 1198 """Compile the registry: sync latest/ dirs, write global and per-connector indexes. 1199 1200 Scans all version directories in the target store, determines the latest GA 1201 semver per connector (excluding yanked and pre-release versions), ensures 1202 each `latest/` directory matches the computed latest, and writes: 1203 1204 * `registries/v0/cloud_registry.json` -- global cloud registry index 1205 * `registries/v0/oss_registry.json` -- global OSS registry index 1206 * `metadata/airbyte/<connector>/versions.json` -- per-connector version index 1207 1208 With `--with-secrets-mask`, also regenerates: 1209 1210 * `registries/v0/specs_secrets_mask.yaml` -- properties marked as secrets 1211 1212 With `--with-legacy-migration=v1`, deletes `cloud.json` / `oss.json` 1213 files for connectors whose `registryOverrides.cloud.enabled` or 1214 `registryOverrides.oss.enabled` is `false`. 1215 1216 By default, injects connector quality metrics from the latest analytics 1217 JSONL export into `generated.metrics`. Use `--no-metrics` for offline 1218 scenarios. 1219 1220 With `--force`, resyncs all latest/ directories even if the version marker 1221 matches the computed latest version. 1222 1223 Uses efficient glob patterns for scanning (no file downloads during discovery). 1224 1225 Requires GCS_CREDENTIALS environment variable to be set. 1226 1227 Examples: 1228 airbyte-ops registry store compile --store coral:dev --dry-run 1229 1230 airbyte-ops registry store compile --store coral:dev/aj-test100 \\ 1231 --connector-name source-faker --connector-name destination-bigquery 1232 1233 airbyte-ops registry store compile --store coral:prod --with-secrets-mask 1234 1235 airbyte-ops registry store compile --store coral:dev \\ 1236 --with-legacy-migration v1 1237 """ 1238 registry = _resolve_store(store) 1239 1240 result = registry.compile( 1241 connector_name=list(connector_name) if connector_name else None, 1242 dry_run=dry_run, 1243 with_secrets_mask=with_secrets_mask, 1244 with_legacy_migration=with_legacy_migration, 1245 with_metrics=with_metrics, 1246 force=force, 1247 ) 1248 1249 print_json( 1250 { 1251 "status": result.status, 1252 "target": result.target, 1253 "connectors_scanned": result.connectors_scanned, 1254 "versions_found": result.versions_found, 1255 "yanked_versions": result.yanked_versions, 1256 "latest_updated": result.latest_updated, 1257 "latest_already_current": result.latest_already_current, 1258 "cloud_registry_entries": result.cloud_registry_entries, 1259 "oss_registry_entries": result.oss_registry_entries, 1260 "composite_registry_entries": result.composite_registry_entries, 1261 "metrics_connector_count": result.metrics_connector_count, 1262 "metrics_registry_entries": result.metrics_registry_entries, 1263 "metrics_source": result.metrics_source, 1264 "metrics_error": result.metrics_error, 1265 "version_indexes_written": result.version_indexes_written, 1266 "specs_secrets_mask_properties": result.specs_secrets_mask_properties, 1267 "errors": result.errors, 1268 "dry_run": result.dry_run, 1269 } 1270 ) 1271 1272 if result.status == "success" or result.status == "dry-run": 1273 print_success(result.summary()) 1274 else: 1275 exit_with_error( 1276 f"Compile completed with {len(result.errors)} error(s): " 1277 + "; ".join(result.errors), 1278 code=1, 1279 ) 1280 1281 1282# ============================================================================= 1283# DELETE-DEV-LATEST COMMAND 1284# ============================================================================= 1285 1286 1287@store_app.command(name="delete-dev-latest") 1288def delete_dev_latest_cmd( 1289 store: Annotated[ 1290 str, 1291 Parameter( 1292 help="Store target (must begin with 'coral:dev').", 1293 ), 1294 ], 1295 *, 1296 connector_name: Annotated[ 1297 tuple[str, ...] | None, 1298 Parameter( 1299 help="Only delete latest/ for these connectors (can be repeated).", 1300 ), 1301 ] = None, 1302 dry_run: Annotated[ 1303 bool, 1304 Parameter(help="Show what would be done without deleting."), 1305 ] = False, 1306) -> None: 1307 """Delete all latest/ directories from a dev registry store. 1308 1309 Discovers every connector that has a `latest/` directory and 1310 deletes each one in parallel using a thread pool. 1311 1312 This is useful before a full re-compile to prove that latest/ 1313 directories can be correctly regenerated from versioned data. 1314 1315 Only dev stores are allowed (store must begin with 'coral:dev'). 1316 1317 Requires GCS_CREDENTIALS environment variable to be set. 1318 1319 Examples: 1320 airbyte-ops registry store delete-dev-latest --store coral:dev --dry-run 1321 1322 airbyte-ops registry store delete-dev-latest --store coral:dev/aj-test100 1323 1324 airbyte-ops registry store delete-dev-latest --store coral:dev \\ 1325 --connector-name source-faker --connector-name destination-bigquery 1326 """ 1327 if not store.startswith("coral:dev"): 1328 exit_with_error( 1329 "delete-dev-latest only supports dev stores " 1330 f"(store must begin with 'coral:dev', got '{store}').", 1331 code=1, 1332 ) 1333 1334 registry = _resolve_store(store) 1335 1336 result = registry.delete_dev_latest( 1337 connector_name=list(connector_name) if connector_name else None, 1338 dry_run=dry_run, 1339 ) 1340 1341 print_json( 1342 { 1343 "status": result.status, 1344 "target": result.target, 1345 "connectors_found": result.connectors_found, 1346 "latest_dirs_deleted": result.latest_dirs_deleted, 1347 "errors": result.errors, 1348 "dry_run": result.dry_run, 1349 } 1350 ) 1351 1352 if result.status in ("success", "dry-run"): 1353 print_success(result.summary()) 1354 else: 1355 exit_with_error( 1356 f"Delete completed with {len(result.errors)} error(s): " 1357 + "; ".join(result.errors[:5]), 1358 code=1, 1359 ) 1360 1361 1362# ============================================================================= 1363# STORE COMPARE COMMAND 1364# ============================================================================= 1365 1366 1367@store_app.command(name="compare") 1368def compare_cmd( 1369 store: Annotated[ 1370 str, 1371 Parameter( 1372 help="Store target being evaluated (e.g. 'coral:dev/20260306-mirror-compile').", 1373 ), 1374 ], 1375 reference_store: Annotated[ 1376 str, 1377 Parameter( 1378 help="Known-good reference store to compare against.", 1379 ), 1380 ], 1381 *, 1382 connector_name: Annotated[ 1383 tuple[str, ...] | None, 1384 Parameter( 1385 help="Only compare these connectors (can be repeated).", 1386 ), 1387 ] = None, 1388 with_artifacts: Annotated[ 1389 bool, 1390 Parameter( 1391 help="Compare per-connector artifact files " 1392 "(metadata.yaml, cloud.json, oss.json, spec.json).", 1393 negative="--no-artifacts", 1394 ), 1395 ] = True, 1396 with_indexes: Annotated[ 1397 bool, 1398 Parameter( 1399 help="Compare global registry index files " 1400 "(cloud_registry.json, oss_registry.json).", 1401 negative="--no-indexes", 1402 ), 1403 ] = True, 1404) -> None: 1405 """Compare a store against a reference store and report differences. 1406 1407 Evaluates the `--store` target against `--reference-store` and reports 1408 per-connector artifact diffs and global index diffs. 1409 1410 Requires GCS_CREDENTIALS environment variable to be set. 1411 1412 Examples: 1413 airbyte-ops registry store compare --store coral:dev/20260306-mirror \\ 1414 --reference-store coral:prod 1415 1416 airbyte-ops registry store compare --store coral:dev/my-test \\ 1417 --connector-name source-faker --no-indexes 1418 1419 airbyte-ops registry store compare --store coral:dev/my-test \\ 1420 --no-artifacts 1421 """ 1422 store_target = _resolve_store(store) 1423 ref_target = _resolve_store(reference_store) 1424 1425 store_prefix = f"{store_target.prefix}/" if store_target.prefix else "" 1426 ref_prefix = f"{ref_target.prefix}/" if ref_target.prefix else "" 1427 1428 result = compare_stores( 1429 store_bucket=store_target.bucket_name, 1430 store_prefix=store_prefix, 1431 reference_bucket=ref_target.bucket_name, 1432 reference_prefix=ref_prefix, 1433 connector_name=list(connector_name) if connector_name else None, 1434 with_artifacts=with_artifacts, 1435 with_indexes=with_indexes, 1436 ) 1437 1438 print_json(result.to_dict()) 1439 1440 if result.status == "match": 1441 print_success(result.summary()) 1442 elif result.status == "differences-found": 1443 error_console.print(f"[yellow]{result.summary()}[/yellow]") 1444 1445 # Print a concise per-connector diff summary 1446 for diff in result.connector_diffs: 1447 if diff.status in ("only_in_store", "only_in_reference"): 1448 error_console.print(f" {diff.connector}: {diff.status}") 1449 else: 1450 for ad in diff.artifact_diffs: 1451 error_console.print( 1452 f" {diff.connector}/{ad.file}: {ad.status}" 1453 + (f" ({ad.details})" if ad.details else "") 1454 ) 1455 1456 for idx_diff in result.index_diffs: 1457 if idx_diff.status != "match": 1458 error_console.print( 1459 f" [index] {idx_diff.file}: {idx_diff.status}" 1460 + ( 1461 f" (store={idx_diff.entry_count_store}," 1462 f" ref={idx_diff.entry_count_reference})" 1463 if idx_diff.entry_count_store or idx_diff.entry_count_reference 1464 else "" 1465 ) 1466 ) 1467 1468 sys.exit(1) 1469 else: 1470 exit_with_error( 1471 f"Compare completed with {len(result.errors)} error(s): " 1472 + "; ".join(result.errors[:5]), 1473 code=1, 1474 )