airbyte.cloud
PyAirbyte classes and methods for interacting with the Airbyte Cloud API.
You can use this module to interact with Airbyte Cloud, OSS, and Enterprise.
Self-managed Airbyte instances
For self-managed Airbyte instances, set api_root to the Public API root for your
deployment. For the default self-managed route, that usually ends in /api/public/v1.
PyAirbyte uses the Public API for workspace and organization discovery.
Some Cloud module methods also call the Config API, including methods such as
CloudConnection.dump_raw_catalog(), which reads the configured catalog directly
from Airbyte. For documented self-managed deployments where the Public API root ends in
/api/public/v1, PyAirbyte infers the Config API root by replacing that suffix with
/api/v1.
If your deployment uses custom ingress or a nonstandard reverse proxy, pass
config_api_root explicitly or set the AIRBYTE_CLOUD_CONFIG_API_URL environment
variable.
from airbyte import cloud
workspace = cloud.CloudWorkspace(
workspace_id="...",
client_id="...",
client_secret="...",
api_root="https://airbyte.example.com/api/public/v1",
config_api_root="https://airbyte.example.com/api/v1",
)
connection = workspace.get_connection(connection_id="...")
raw_catalog = connection.dump_raw_catalog()
Examples
Basic Sync Example:
import airbyte as ab
from airbyte import cloud
# Initialize an Airbyte Cloud workspace object
workspace = cloud.CloudWorkspace(
workspace_id="123",
api_key=ab.get_secret("AIRBYTE_CLOUD_API_KEY"),
)
# Run a sync job on Airbyte Cloud
connection = workspace.get_connection(connection_id="456")
sync_result = connection.run_sync()
print(sync_result.get_job_status())
Example Read From Cloud Destination:
If your destination is supported, you can read records directly from the
SyncResult object. Currently this is supported in Snowflake and BigQuery only.
# Assuming we've already created a `connection` object...
# Get the latest job result and print the stream names
sync_result = connection.get_sync_result()
print(sync_result.stream_names)
# Get a dataset from the sync result
dataset: CachedDataset = sync_result.get_dataset("users")
# Get a SQLAlchemy table to use in SQL queries...
users_table = dataset.to_sql_table()
print(f"Table name: {users_table.name}")
# Or iterate over the dataset directly
for record in dataset:
print(record)
1# Copyright (c) 2024 Airbyte, Inc., all rights reserved. 2"""PyAirbyte classes and methods for interacting with the Airbyte Cloud API. 3 4You can use this module to interact with Airbyte Cloud, OSS, and Enterprise. 5 6## Self-managed Airbyte instances 7 8For self-managed Airbyte instances, set `api_root` to the Public API root for your 9deployment. For the default self-managed route, that usually ends in `/api/public/v1`. 10PyAirbyte uses the Public API for workspace and organization discovery. 11 12Some Cloud module methods also call the Config API, including methods such as 13`CloudConnection.dump_raw_catalog()`, which reads the configured catalog directly 14from Airbyte. For documented self-managed deployments where the Public API root ends in 15`/api/public/v1`, PyAirbyte infers the Config API root by replacing that suffix with 16`/api/v1`. 17 18If your deployment uses custom ingress or a nonstandard reverse proxy, pass 19`config_api_root` explicitly or set the `AIRBYTE_CLOUD_CONFIG_API_URL` environment 20variable. 21 22```python 23from airbyte import cloud 24 25workspace = cloud.CloudWorkspace( 26 workspace_id="...", 27 client_id="...", 28 client_secret="...", 29 api_root="https://airbyte.example.com/api/public/v1", 30 config_api_root="https://airbyte.example.com/api/v1", 31) 32 33connection = workspace.get_connection(connection_id="...") 34raw_catalog = connection.dump_raw_catalog() 35``` 36 37## Examples 38 39### Basic Sync Example: 40 41```python 42import airbyte as ab 43from airbyte import cloud 44 45# Initialize an Airbyte Cloud workspace object 46workspace = cloud.CloudWorkspace( 47 workspace_id="123", 48 api_key=ab.get_secret("AIRBYTE_CLOUD_API_KEY"), 49) 50 51# Run a sync job on Airbyte Cloud 52connection = workspace.get_connection(connection_id="456") 53sync_result = connection.run_sync() 54print(sync_result.get_job_status()) 55``` 56 57### Example Read From Cloud Destination: 58 59If your destination is supported, you can read records directly from the 60`SyncResult` object. Currently this is supported in Snowflake and BigQuery only. 61 62 63```python 64# Assuming we've already created a `connection` object... 65 66# Get the latest job result and print the stream names 67sync_result = connection.get_sync_result() 68print(sync_result.stream_names) 69 70# Get a dataset from the sync result 71dataset: CachedDataset = sync_result.get_dataset("users") 72 73# Get a SQLAlchemy table to use in SQL queries... 74users_table = dataset.to_sql_table() 75print(f"Table name: {users_table.name}") 76 77# Or iterate over the dataset directly 78for record in dataset: 79 print(record) 80``` 81""" 82 83from __future__ import annotations 84 85from typing import TYPE_CHECKING 86 87from airbyte.cloud.client import CloudClient 88from airbyte.cloud.client_config import CloudClientConfig 89from airbyte.cloud.connections import CloudConnection 90from airbyte.cloud.models import ( 91 CloudDefaultContextInfo, 92 CloudWorkspaceInfo, 93 JobStatusEnum, 94 JobTypeEnum, 95 WorkspacePrivilegeScope, 96) 97from airbyte.cloud.organizations import CloudOrganization 98from airbyte.cloud.sync_results import SyncResult 99from airbyte.cloud.workspaces import CloudWorkspace 100 101 102# Submodules imported here for documentation reasons: https://github.com/mitmproxy/pdoc/issues/757 103if TYPE_CHECKING: 104 # ruff: noqa: TC004 105 from airbyte.cloud import ( 106 client, 107 client_config, 108 connections, 109 constants, 110 organizations, 111 sync_results, 112 workspaces, 113 ) 114 115 116__all__ = [ 117 # Submodules 118 "workspaces", 119 "client", 120 "organizations", 121 "connections", 122 "constants", 123 "client_config", 124 "sync_results", 125 # Classes 126 "CloudClient", 127 "CloudOrganization", 128 "CloudWorkspace", 129 "CloudConnection", 130 "CloudClientConfig", 131 "CloudDefaultContextInfo", 132 "CloudWorkspaceInfo", 133 "SyncResult", 134 # Enums 135 "JobStatusEnum", 136 "JobTypeEnum", 137 "WorkspacePrivilegeScope", 138]
112@dataclass(init=False, kw_only=True) 113class CloudClient: 114 """Authenticated client for Airbyte Cloud and self-managed Airbyte APIs.""" 115 116 _credentials: _AirbyteCredentials 117 _membership_organization_ids: tuple[str, ...] | None 118 _user_permissions: tuple[dict[str, Any], ...] | None 119 _direct_workspace_infos: dict[str, CloudWorkspaceInfo | None] 120 _workspace_organizations: dict[str, CloudOrganizationInfo | None] 121 _validated_direct_workspace_result: tuple[list[CloudWorkspaceInfo], int] | None 122 _authenticated_user_info: dict[str, Any] | None = field(repr=False) 123 _authenticated_user_id: str | None = field(repr=False) 124 _authenticated_bearer_token: SecretString | None 125 126 def __init__( 127 self, 128 *, 129 client_id: str | SecretString | None = None, 130 client_secret: str | SecretString | None = None, 131 bearer_token: str | SecretString | None = None, 132 public_api_root: str | None = None, 133 config_api_root: str | None = None, 134 workspace_id: str | None = None, 135 organization_id: str | None = None, 136 ) -> None: 137 """Initialize a `CloudClient` from explicit auth values.""" 138 self._credentials = _AirbyteCredentials.from_auth( 139 client_id=client_id, 140 client_secret=client_secret, 141 bearer_token=bearer_token, 142 public_api_root=public_api_root, 143 config_api_root=config_api_root, 144 workspace_id=workspace_id, 145 organization_id=organization_id, 146 env_vars=False, 147 ) 148 self._membership_organization_ids = None 149 self._user_permissions = None 150 self._direct_workspace_infos = {} 151 self._workspace_organizations = {} 152 self._validated_direct_workspace_result = None 153 self._authenticated_user_info = None 154 self._authenticated_user_id = None 155 self._authenticated_bearer_token = None 156 157 @property 158 def client_id(self) -> SecretString | None: 159 """OAuth client ID used for authentication.""" 160 return self._credentials.client_id 161 162 @property 163 def client_secret(self) -> SecretString | None: 164 """OAuth client secret used for authentication.""" 165 return self._credentials.client_secret 166 167 @property 168 def bearer_token(self) -> SecretString | None: 169 """Bearer token used for authentication.""" 170 return self._credentials.bearer_token 171 172 @property 173 def public_api_root(self) -> str: 174 """Airbyte Public API root.""" 175 return self._credentials.public_api_root 176 177 @property 178 def config_api_root(self) -> str | None: 179 """Airbyte Config API root.""" 180 return self._credentials.config_api_root 181 182 @property 183 def organization_id(self) -> str | None: 184 """Default organization ID for organization-scoped operations.""" 185 return self._credentials.organization_id 186 187 @property 188 def default_workspace_id(self) -> str | None: 189 """Default workspace ID for workspace-scoped operations.""" 190 return self._credentials.workspace_id 191 192 @classmethod 193 def from_auth( 194 cls, 195 *, 196 env_vars: bool = False, 197 organization_id: str | None = None, 198 client_id: str | SecretString | None = None, 199 client_secret: str | SecretString | None = None, 200 bearer_token: str | SecretString | None = None, 201 public_api_root: str | None = None, 202 config_api_root: str | None = None, 203 ) -> CloudClient: 204 """Create a client from explicit inputs and optionally environment variables. 205 206 When `env_vars` is True, environment variables are checked as a fallback 207 after any explicitly provided values. 208 """ 209 credentials = _AirbyteCredentials.from_auth( 210 organization_id=organization_id, 211 client_id=client_id, 212 client_secret=client_secret, 213 bearer_token=bearer_token, 214 public_api_root=public_api_root, 215 config_api_root=config_api_root, 216 env_vars=env_vars, 217 ) 218 return cls._from_credentials(credentials) 219 220 @classmethod 221 def _from_credentials(cls, credentials: _AirbyteCredentials) -> CloudClient: 222 """Create a client from resolved Cloud credentials.""" 223 return cls( 224 client_id=credentials.client_id, 225 client_secret=credentials.client_secret, 226 bearer_token=credentials.bearer_token, 227 public_api_root=credentials.public_api_root, 228 config_api_root=credentials.config_api_root, 229 workspace_id=credentials.workspace_id, 230 organization_id=credentials.organization_id, 231 ) 232 233 def get_workspace(self, workspace_id: str | None = None) -> CloudWorkspace: 234 """Create a `CloudWorkspace` using this client's credentials. 235 236 See the module docstring for how the workspace is resolved. 237 """ 238 resolved_workspace_id = workspace_id or self.resolve_default_workspace_id() 239 if not resolved_workspace_id: 240 raise exc.PyAirbyteInputError( 241 message="Workspace ID is required.", 242 guidance=( 243 "No workspace was configured, and no default workspace could be resolved " 244 "for the authenticated user. Provide a workspace ID, or call " 245 "`get_default_cloud_context` to discover your workspaces and organizations." 246 ), 247 ) 248 249 credentials = self._credentials.with_workspace_id(resolved_workspace_id) 250 return CloudWorkspace( 251 workspace_id=credentials.workspace_id, 252 client_id=credentials.client_id, 253 client_secret=credentials.client_secret, 254 bearer_token=credentials.bearer_token, 255 api_root=credentials.public_api_root, 256 config_api_root=credentials.config_api_root, 257 ) 258 259 def create_workspace( 260 self, 261 *, 262 name: str, 263 organization_id: str | None = None, 264 region_id: str | None = None, 265 ) -> CloudWorkspaceInfo: 266 """Create an Airbyte workspace.""" 267 resolved_organization_id = organization_id or self.organization_id 268 workspace = api_util.create_workspace( 269 name=name, 270 organization_id=resolved_organization_id, 271 region_id=region_id, 272 api_root=self.public_api_root, 273 client_id=self.client_id, 274 client_secret=self.client_secret, 275 bearer_token=self.bearer_token, 276 ) 277 return CloudWorkspaceInfo.from_api_response(workspace) 278 279 def rename_workspace( 280 self, 281 workspace_id: str, 282 *, 283 name: str, 284 ) -> CloudWorkspaceInfo: 285 """Rename an Airbyte workspace.""" 286 workspace = api_util.rename_workspace( 287 workspace_id=workspace_id, 288 name=name, 289 api_root=self.public_api_root, 290 client_id=self.client_id, 291 client_secret=self.client_secret, 292 bearer_token=self.bearer_token, 293 ) 294 return CloudWorkspaceInfo.from_api_response(workspace) 295 296 def permanently_delete_workspace( 297 self, 298 workspace_id: str, 299 *, 300 workspace_name: str | None = None, 301 safe_mode: bool = True, 302 ) -> None: 303 """Permanently delete an Airbyte workspace if it has no connections. 304 305 When `safe_mode` is enabled, the workspace name must contain `delete-me` 306 or `deleteme`. This also checks for existing connections before deleting 307 and raises `AirbyteWorkspaceNotEmptyError` if the workspace is not empty. 308 """ 309 api_util.permanently_delete_workspace( 310 workspace_id=workspace_id, 311 workspace_name=workspace_name, 312 api_root=self.public_api_root, 313 client_id=self.client_id, 314 client_secret=self.client_secret, 315 bearer_token=self.bearer_token, 316 safe_mode=safe_mode, 317 ) 318 319 @overload 320 def list_workspaces( 321 self, 322 name: str | None = None, 323 *, 324 organization_id: None = None, 325 organization_name: str | None = None, 326 workspace_id: str | None = None, 327 name_contains: str | None = None, 328 name_filter: Callable[[str], bool] | None = None, 329 limit: int | None = None, 330 privilege_scope: WorkspacePrivilegeScope = WorkspacePrivilegeScope.MEMBER_OF, 331 all_organizations: bool = False, 332 ) -> list[CloudWorkspaceInfo]: 333 raise NotImplementedError 334 335 @overload 336 def list_workspaces( 337 self, 338 name: str | None = None, 339 *, 340 organization_id: str, 341 organization_name: str | None = None, 342 workspace_id: str | None = None, 343 name_contains: str | None = None, 344 name_filter: Callable[[str], bool] | None = None, 345 limit: int | None = None, 346 privilege_scope: WorkspacePrivilegeScope = WorkspacePrivilegeScope.MEMBER_OF, 347 all_organizations: bool = False, 348 ) -> list[CloudWorkspaceInfo]: 349 raise NotImplementedError 350 351 def list_workspaces( # noqa: PLR0911, PLR0913 352 self, 353 name: str | None = None, 354 *, 355 organization_id: str | None = None, 356 organization_name: str | None = None, 357 workspace_id: str | None = None, 358 name_contains: str | None = None, 359 name_filter: Callable[[str], bool] | None = None, 360 limit: int | None = None, 361 privilege_scope: WorkspacePrivilegeScope = WorkspacePrivilegeScope.MEMBER_OF, 362 all_organizations: bool = False, 363 ) -> list[CloudWorkspaceInfo]: 364 """List workspaces available to this client. 365 366 `privilege_scope` controls whether this lists direct member workspaces, 367 organization workspaces, or instance-wide workspaces. The deprecated 368 `all_organizations` alias maps to `WorkspacePrivilegeScope.ANY`. 369 """ 370 if limit is not None and limit <= 0: 371 raise exc.PyAirbyteInputError(message="`limit` must be greater than 0.") 372 if organization_id is not None and organization_name is not None: 373 raise exc.PyAirbyteInputError( 374 message="Provide either organization ID or organization name." 375 ) 376 has_explicit_organization = organization_id is not None or organization_name is not None 377 has_explicit_workspace = workspace_id is not None 378 379 if all_organizations: 380 if privilege_scope is not WorkspacePrivilegeScope.MEMBER_OF: 381 raise exc.PyAirbyteInputError( 382 message="all_organizations cannot be combined with privilege_scope." 383 ) 384 warnings.warn( 385 "`all_organizations` is deprecated; use `privilege_scope` instead.", 386 DeprecationWarning, 387 stacklevel=2, 388 ) 389 privilege_scope = WorkspacePrivilegeScope.ANY 390 if name_contains is not None and name_filter is not None: 391 raise exc.PyAirbyteInputError( 392 message="You can provide name_contains or name_filter, but not both." 393 ) 394 if name is not None and name_contains is not None: 395 raise exc.PyAirbyteInputError( 396 message="You can provide name or name_contains, but not both." 397 ) 398 if has_explicit_organization or has_explicit_workspace: 399 resolved_organization_id = self._resolve_workspace_organization_id( 400 organization_id=organization_id, 401 organization_name=organization_name, 402 workspace_id=workspace_id, 403 ) 404 if resolved_organization_id is None: 405 return [] 406 return self._list_workspaces_in_organizations( 407 (resolved_organization_id,), 408 name=name, 409 name_contains=name_contains, 410 name_filter=name_filter, 411 limit=limit, 412 ) 413 414 if privilege_scope is WorkspacePrivilegeScope.MEMBER_OF: 415 return self._list_member_workspaces( 416 name=name, 417 name_contains=name_contains, 418 name_filter=name_filter, 419 limit=limit, 420 ) 421 422 if privilege_scope is WorkspacePrivilegeScope.INSTANCE_ADMIN: 423 if not self._is_instance_admin(): 424 raise exc.PyAirbyteInputError( 425 message="privilege_scope=instance_admin requires the instance_admin permission." 426 ) 427 return self._list_unscoped_workspaces( 428 name=name, 429 name_contains=name_contains, 430 name_filter=name_filter, 431 limit=limit, 432 ) 433 434 if privilege_scope is WorkspacePrivilegeScope.ANY and self._is_instance_admin(): 435 return self._list_unscoped_workspaces( 436 name=name, 437 name_contains=name_contains, 438 name_filter=name_filter, 439 limit=limit, 440 ) 441 442 if privilege_scope in { 443 WorkspacePrivilegeScope.ORGANIZATION_ADMIN, 444 WorkspacePrivilegeScope.ANY, 445 }: 446 organization_ids = self._get_membership_organization_ids() 447 if not organization_ids: 448 return [] 449 return self._list_workspaces_in_organizations( 450 organization_ids, 451 name=name, 452 name_contains=name_contains, 453 name_filter=name_filter, 454 limit=limit, 455 ) 456 457 raise exc.PyAirbyteInputError(message="Unsupported workspace privilege scope.") 458 459 def _list_member_workspaces( 460 self, 461 *, 462 name: str | None = None, 463 name_contains: str | None = None, 464 name_filter: Callable[[str], bool] | None = None, 465 limit: int | None = None, 466 ) -> list[CloudWorkspaceInfo]: 467 """List workspaces granted directly to the authenticated user.""" 468 workspaces, unvalidated_count = self._validate_direct_workspaces() 469 name_substring = name_contains.casefold() if name_contains is not None else None 470 filtered_workspaces: list[CloudWorkspaceInfo] = [] 471 472 def accepts(workspace: CloudWorkspaceInfo) -> bool: 473 if name is not None and workspace.name != name: 474 return False 475 if name_substring is not None and name_substring not in workspace.name.casefold(): 476 return False 477 return name_filter is None or name_filter(workspace.name) 478 479 for workspace in workspaces: 480 if accepts(workspace): 481 filtered_workspaces.append(workspace) 482 if limit is not None and len(filtered_workspaces) == limit: 483 break 484 if unvalidated_count > 0 and (limit is None or len(filtered_workspaces) < limit): 485 for workspace_id in self._get_direct_workspace_ids()[MAX_WORKSPACES_TO_VALIDATE:]: 486 workspace = self._get_direct_workspace_info(workspace_id) 487 if workspace is None or not accepts(workspace): 488 continue 489 filtered_workspaces.append(workspace) 490 if limit is not None and len(filtered_workspaces) == limit: 491 break 492 return filtered_workspaces 493 494 def _list_unscoped_workspaces( 495 self, 496 *, 497 name: str | None, 498 name_contains: str | None, 499 name_filter: Callable[[str], bool] | None, 500 limit: int | None, 501 ) -> list[CloudWorkspaceInfo]: 502 """List workspaces across the instance.""" 503 if name_contains is not None: 504 name_substring = name_contains.casefold() 505 506 def matches_name(workspace_name: str) -> bool: 507 return name_substring in workspace_name.casefold() 508 509 name_filter = matches_name 510 name = None 511 workspaces = api_util.list_workspaces( 512 workspace_id="", 513 api_root=self.public_api_root, 514 client_id=self.client_id, 515 client_secret=self.client_secret, 516 bearer_token=self.bearer_token, 517 name_filter=name_filter, 518 name=name, 519 limit=limit, 520 ) 521 return [CloudWorkspaceInfo.from_api_response(workspace) for workspace in workspaces] 522 523 def _list_workspaces_in_organizations( 524 self, 525 organization_ids: tuple[str, ...], 526 *, 527 name: str | None, 528 name_contains: str | None, 529 name_filter: Callable[[str], bool] | None, 530 limit: int | None, 531 ) -> list[CloudWorkspaceInfo]: 532 """List and combine workspaces from one or more organizations.""" 533 workspace_infos: list[CloudWorkspaceInfo] = [] 534 for organization_id in organization_ids: 535 remaining_limit = None if limit is None else limit - len(workspace_infos) 536 if remaining_limit == 0: 537 break 538 workspaces = api_util.list_workspaces_in_organization( 539 organization_id=organization_id, 540 api_root=self.public_api_root, 541 config_api_root=self.config_api_root, 542 client_id=self.client_id, 543 client_secret=self.client_secret, 544 bearer_token=self._get_config_api_bearer_token(), 545 name_contains=name_contains or name, 546 limit=None if name is not None or name_filter is not None else remaining_limit, 547 ) 548 organization_workspaces = [ 549 CloudWorkspaceInfo.from_mapping(workspace) for workspace in workspaces 550 ] 551 if name is not None: 552 organization_workspaces = [ 553 workspace for workspace in organization_workspaces if workspace.name == name 554 ] 555 if name_filter is not None: 556 organization_workspaces = [ 557 workspace 558 for workspace in organization_workspaces 559 if name_filter(workspace.name) 560 ] 561 workspace_infos.extend(organization_workspaces) 562 if limit is not None and len(workspace_infos) >= limit: 563 break 564 return workspace_infos[:limit] if limit is not None else workspace_infos 565 566 def _resolve_workspace_organization_id( 567 self, 568 *, 569 organization_id: str | None, 570 organization_name: str | None, 571 workspace_id: str | None, 572 ) -> str | None: 573 """Resolve the organization for a workspace listing.""" 574 if organization_id is not None or organization_name is not None: 575 if organization_id is not None: 576 return organization_id 577 # Do not use explicit name lookup to infer a default organization. 578 return self.get_organization(organization_name=organization_name).organization_id 579 580 if workspace_id is not None: 581 return self._get_workspace_parent_organization_id(workspace_id) 582 583 return self._resolve_ambient_organization_id() 584 585 def _resolve_ambient_organization_id(self) -> str | None: 586 """Resolve an organization from configured client context or memberships.""" 587 if self.organization_id is not None: 588 return self.organization_id 589 if self.default_workspace_id is not None: 590 try: 591 return self._get_workspace_parent_organization_id(self.default_workspace_id) 592 except (exc.AirbyteError, exc.PyAirbyteInputError): 593 pass 594 user_default_workspace_id = self._get_user_default_workspace_id() 595 if user_default_workspace_id: 596 try: 597 return self._get_workspace_parent_organization_id(user_default_workspace_id) 598 except (exc.AirbyteError, exc.PyAirbyteInputError): 599 pass 600 601 try: 602 organization_ids = self._get_membership_organization_ids() 603 except (exc.AirbyteError, exc.PyAirbyteInputError): 604 return None 605 if len(organization_ids) > 1: 606 self._raise_ambiguous_organization_error(organization_ids) 607 return organization_ids[0] if organization_ids else None 608 609 def _get_config_api_bearer_token(self) -> SecretString | None: 610 """Get and cache a bearer token for Config API requests.""" 611 if self._authenticated_bearer_token is not None: 612 return self._authenticated_bearer_token 613 if self.bearer_token is not None: 614 self._authenticated_bearer_token = self.bearer_token 615 elif self.client_id is not None and self.client_secret is not None: 616 self._authenticated_bearer_token = api_util.get_bearer_token( 617 client_id=self.client_id, 618 client_secret=self.client_secret, 619 api_root=self.public_api_root, 620 ) 621 return self._authenticated_bearer_token 622 623 def _get_workspace_parent_organization_id(self, workspace_id: str) -> str: 624 """Resolve a workspace's parent organization ID.""" 625 organization = api_util.get_workspace_organization_info( 626 workspace_id=workspace_id, 627 api_root=self.public_api_root, 628 config_api_root=self.config_api_root, 629 client_id=self.client_id, 630 client_secret=self.client_secret, 631 bearer_token=self._get_config_api_bearer_token(), 632 ) 633 resolved_organization_id = organization.get("organizationId") 634 if isinstance(resolved_organization_id, str) and resolved_organization_id: 635 return resolved_organization_id 636 raise exc.PyAirbyteInputError( 637 message="The workspace response did not include an organization ID.", 638 context={"workspace_id": workspace_id, "response": organization}, 639 ) 640 641 def get_workspace_parent_organization_id(self, workspace_id: str) -> str | None: 642 """Return the parent organization ID of a workspace, or `None` if it cannot be resolved.""" 643 try: 644 return self._get_workspace_parent_organization_id(workspace_id) 645 except (exc.AirbyteError, exc.PyAirbyteInputError): 646 return None 647 648 def _get_authenticated_user_info(self) -> dict[str, Any]: 649 """Get and cache the Airbyte user record for the current credentials.""" 650 if self._authenticated_user_info is not None: 651 return self._authenticated_user_info 652 653 bearer_token = self._get_config_api_bearer_token() 654 if bearer_token is None: 655 raise exc.PyAirbyteInputError( 656 message="No authentication credentials provided.", 657 guidance="Provide either client credentials or a bearer token.", 658 ) 659 auth_user_id = api_util.get_user_id_from_bearer_token(bearer_token) 660 self._authenticated_user_info = api_util.get_user_by_auth_id( 661 auth_user_id, 662 api_root=self.public_api_root, 663 config_api_root=self.config_api_root, 664 client_id=self.client_id, 665 client_secret=self.client_secret, 666 bearer_token=bearer_token, 667 ) 668 return self._authenticated_user_info 669 670 def _get_authenticated_user_id(self) -> str: 671 """Get and cache the Airbyte user ID for the current credentials.""" 672 if self._authenticated_user_id is not None: 673 return self._authenticated_user_id 674 675 user = self._get_authenticated_user_info() 676 user_id = user.get("userId") 677 if not isinstance(user_id, str) or not user_id: 678 raise exc.PyAirbyteInputError( 679 message="The Airbyte user response did not include a user ID.", 680 context={"response": user}, 681 ) 682 self._authenticated_user_id = user_id 683 return self._authenticated_user_id 684 685 def _get_user_default_workspace_id(self) -> str | None: 686 """Get the authenticated user's default workspace ID, when available.""" 687 try: 688 default_workspace_id = self._get_authenticated_user_info().get("defaultWorkspaceId") 689 except (exc.AirbyteError, exc.PyAirbyteInputError): 690 return None 691 return ( 692 default_workspace_id 693 if isinstance(default_workspace_id, str) and default_workspace_id 694 else None 695 ) 696 697 def resolve_default_workspace_id(self) -> str | None: 698 """Resolve the configured or authenticated user's default workspace ID.""" 699 configured_workspace_id = self.default_workspace_id 700 if configured_workspace_id: 701 return configured_workspace_id 702 user_default_workspace_id = self._get_user_default_workspace_id() 703 if user_default_workspace_id: 704 return user_default_workspace_id 705 try: 706 live_workspaces, unvalidated_count = self._validate_direct_workspaces() 707 except (AirbyteError, exc.PyAirbyteInputError): 708 return None 709 return ( 710 live_workspaces[0].workspace_id 711 if unvalidated_count == 0 and len(live_workspaces) == 1 712 else None 713 ) 714 715 def _get_user_permissions(self) -> tuple[dict[str, Any], ...]: 716 """Get and cache permissions for the authenticated user.""" 717 if self._user_permissions is None: 718 self._user_permissions = tuple( 719 permission 720 for permission in api_util.list_permissions_for_user( 721 self._get_authenticated_user_id(), 722 api_root=self.public_api_root, 723 config_api_root=self.config_api_root, 724 client_id=self.client_id, 725 client_secret=self.client_secret, 726 bearer_token=self._get_config_api_bearer_token(), 727 ) 728 if isinstance(permission, dict) 729 ) 730 return self._user_permissions 731 732 def _get_membership_organization_ids(self) -> tuple[str, ...]: 733 """Get and cache organization IDs from the caller's permissions.""" 734 if self._membership_organization_ids is not None: 735 return self._membership_organization_ids 736 737 permissions = self._get_user_permissions() 738 organization_ids: list[str] = [] 739 for permission in permissions: 740 permission_organization_id = permission.get("organizationId") 741 if ( 742 isinstance(permission_organization_id, str) 743 and permission_organization_id 744 and permission_organization_id not in organization_ids 745 ): 746 organization_ids.append(permission_organization_id) 747 self._membership_organization_ids = tuple(organization_ids) 748 return self._membership_organization_ids 749 750 def _get_direct_workspace_ids(self) -> tuple[str, ...]: 751 """Get unique workspace IDs from the caller's direct permissions.""" 752 workspace_ids: list[str] = [] 753 for permission in self._get_user_permissions(): 754 workspace_id = permission.get("workspaceId") 755 if isinstance(workspace_id, str) and workspace_id and workspace_id not in workspace_ids: 756 workspace_ids.append(workspace_id) 757 return tuple(workspace_ids) 758 759 def _get_direct_workspace_info(self, workspace_id: str) -> CloudWorkspaceInfo | None: 760 """Fetch a directly granted workspace, or `None` if the grant is stale (404).""" 761 if workspace_id in self._direct_workspace_infos: 762 return self._direct_workspace_infos[workspace_id] 763 try: 764 workspace = api_util.get_workspace( 765 workspace_id=workspace_id, 766 api_root=self.public_api_root, 767 client_id=self.client_id, 768 client_secret=self.client_secret, 769 bearer_token=self.bearer_token, 770 ) 771 except exc.AirbyteMissingResourceError: 772 self._direct_workspace_infos[workspace_id] = None 773 return None 774 workspace_info = CloudWorkspaceInfo.from_api_response(workspace) 775 self._direct_workspace_infos[workspace_id] = workspace_info 776 return workspace_info 777 778 def _get_workspace_organization(self, workspace_id: str) -> CloudOrganizationInfo | None: 779 """Fetch and cache organization info for a workspace.""" 780 if workspace_id in self._workspace_organizations: 781 return self._workspace_organizations[workspace_id] 782 try: 783 organization = api_util.get_workspace_organization_info( 784 workspace_id=workspace_id, 785 api_root=self.public_api_root, 786 config_api_root=self.config_api_root, 787 client_id=self.client_id, 788 client_secret=self.client_secret, 789 bearer_token=self._get_config_api_bearer_token(), 790 ) 791 except (AirbyteError, NotImplementedError): 792 # The workspace is readable via the public API but its organization is not 793 # (e.g. the caller lacks org-level read, or no Config API root can be derived 794 # from a custom public API root). Keep the live workspace and leave the 795 # organization unknown. 796 self._workspace_organizations[workspace_id] = None 797 return None 798 organization_id = organization.get("organizationId") 799 if not isinstance(organization_id, str) or not organization_id: 800 self._workspace_organizations[workspace_id] = None 801 return None 802 organization_info = CloudOrganizationInfo( 803 organization_id=organization_id, 804 organization_name=( 805 organization.get("organizationName") 806 if isinstance(organization.get("organizationName"), str) 807 else None 808 ), 809 ) 810 self._workspace_organizations[workspace_id] = organization_info 811 return organization_info 812 813 def _validate_direct_workspaces(self) -> tuple[list[CloudWorkspaceInfo], int]: 814 """Validate direct workspace grants once within the configured cap.""" 815 if self._validated_direct_workspace_result is not None: 816 return self._validated_direct_workspace_result 817 workspace_ids = self._get_direct_workspace_ids() 818 live_workspaces: list[CloudWorkspaceInfo] = [] 819 for workspace_id in workspace_ids[:MAX_WORKSPACES_TO_VALIDATE]: 820 workspace = self._get_direct_workspace_info(workspace_id) 821 if workspace is None: 822 continue 823 organization = self._get_workspace_organization(workspace_id) 824 if organization is not None: 825 workspace = workspace.model_copy( 826 update={ 827 "organization_id": organization.organization_id, 828 "organization_name": organization.organization_name, 829 } 830 ) 831 live_workspaces.append(workspace) 832 result = ( 833 live_workspaces, 834 max(0, len(workspace_ids) - MAX_WORKSPACES_TO_VALIDATE), 835 ) 836 self._validated_direct_workspace_result = result 837 return result 838 839 def _is_instance_admin(self) -> bool: 840 """Return whether the caller has an instance-admin permission.""" 841 return any( 842 permission.get("permissionType") == "instance_admin" 843 for permission in self._get_user_permissions() 844 ) 845 846 def get_default_context_for_user(self) -> CloudDefaultContextInfo: 847 """Describe the authenticated user's explicit Cloud affinities.""" 848 user_id: str | None = None 849 user_name: str | None = None 850 user_email: str | None = None 851 try: 852 user = self._get_authenticated_user_info() 853 except (AirbyteError, exc.PyAirbyteInputError): 854 pass 855 else: 856 user_id = user.get("userId") if isinstance(user.get("userId"), str) else None 857 user_name = user.get("name") if isinstance(user.get("name"), str) else None 858 user_email = user.get("email") if isinstance(user.get("email"), str) else None 859 860 default_workspace_id = self.resolve_default_workspace_id() 861 try: 862 permissions = self._get_user_permissions() 863 except (AirbyteError, exc.PyAirbyteInputError): 864 permissions = () 865 membership_organization_ids = () 866 member_workspaces = [] 867 member_organizations_truncated = False 868 unvalidated_workspace_count = 0 869 else: 870 membership_organization_ids = self._get_membership_organization_ids() 871 member_organizations_truncated = ( 872 len(membership_organization_ids) > MAX_ORGANIZATION_CANDIDATES 873 ) 874 try: 875 member_workspaces, unvalidated_workspace_count = self._validate_direct_workspaces() 876 except (AirbyteError, exc.PyAirbyteInputError): 877 member_workspaces = [] 878 unvalidated_workspace_count = 0 879 member_workspaces = member_workspaces[:] 880 881 member_organizations = [ 882 CloudOrganizationInfo.model_validate(candidate) 883 for candidate in self._get_organization_candidates( 884 membership_organization_ids[:MAX_ORGANIZATION_CANDIDATES] 885 ) 886 ] 887 default_workspace_info: CloudWorkspaceInfo | None = None 888 default_workspace_organization: CloudOrganizationInfo | None = None 889 if default_workspace_id is not None: 890 try: 891 default_workspace_info = self._get_direct_workspace_info(default_workspace_id) 892 except (AirbyteError, exc.PyAirbyteInputError): 893 default_workspace_info = None 894 if default_workspace_info is not None: 895 default_workspace_organization = self._get_workspace_organization( 896 default_workspace_id 897 ) 898 discovery_hints: list[str] = [] 899 if any(permission.get("permissionType") == "instance_admin" for permission in permissions): 900 discovery_hints.append( 901 "Instance-admin access may include every organization and workspace in the " 902 "instance. Use list_cloud_organizations(name_contains=...) or " 903 "list_cloud_workspaces(organization_id=...) to discover others." 904 ) 905 if membership_organization_ids: 906 discovery_hints.append( 907 "Organization membership grants access to every workspace in those " 908 "organizations. Use list_cloud_workspaces(organization_id=<id>) to " 909 "discover workspaces." 910 ) 911 return CloudDefaultContextInfo( 912 user_id=user_id, 913 user_name=user_name, 914 user_email=user_email, 915 default_workspace_id=default_workspace_id, 916 default_workspace_name=( 917 default_workspace_info.name if default_workspace_info else None 918 ), 919 default_workspace_verified=default_workspace_info is not None, 920 default_organization_id=( 921 default_workspace_organization.organization_id 922 if default_workspace_organization is not None 923 else None 924 ), 925 default_organization_name=( 926 default_workspace_organization.organization_name 927 if default_workspace_organization is not None 928 else None 929 ), 930 configured_workspace_id=self.default_workspace_id, 931 configured_organization_id=self.organization_id, 932 member_organizations=member_organizations, 933 member_workspaces=member_workspaces, 934 member_organizations_truncated=member_organizations_truncated, 935 member_workspaces_truncated=unvalidated_workspace_count > 0, 936 unvalidated_workspace_count=unvalidated_workspace_count, 937 discovery_hints=discovery_hints, 938 ) 939 940 def _get_organization_candidates( 941 self, 942 organization_ids: tuple[str, ...], 943 ) -> list[dict[str, str | None]]: 944 """Get names for membership-derived organization candidates.""" 945 candidates: list[dict[str, str | None]] = [] 946 for organization_id in organization_ids: 947 organization_name = None 948 try: 949 organization_info = api_util.get_organization_info( 950 organization_id=organization_id, 951 api_root=self.public_api_root, 952 config_api_root=self.config_api_root, 953 client_id=self.client_id, 954 client_secret=self.client_secret, 955 bearer_token=self._get_config_api_bearer_token(), 956 ) 957 except AirbyteError: 958 pass 959 else: 960 candidate_name = organization_info.get("organizationName") 961 if isinstance(candidate_name, str): 962 organization_name = candidate_name 963 candidates.append( 964 { 965 "organization_id": organization_id, 966 "organization_name": organization_name, 967 } 968 ) 969 return candidates 970 971 def _raise_ambiguous_organization_error( 972 self, 973 organization_ids: tuple[str, ...], 974 ) -> NoReturn: 975 """Raise an error enumerating the caller's candidate organizations.""" 976 candidates = self._get_organization_candidates( 977 organization_ids[:MAX_ORGANIZATION_CANDIDATES] 978 ) 979 candidate_details = ", ".join( 980 f"{candidate['organization_id']} " 981 f"({candidate['organization_name'] or 'name unavailable'})" 982 for candidate in candidates 983 ) 984 raise exc.PyAirbyteInputError( 985 message=( 986 "Multiple organization memberships were found for these credentials. Retry " 987 "with one of these " 988 "organization IDs " 989 f"(showing {len(candidates)} of {len(organization_ids)}): {candidate_details}. " 990 "Call `get_default_cloud_context` to see your memberships." 991 ), 992 context={ 993 "organization_ids": list(organization_ids), 994 "organization_candidates": candidates, 995 "total_candidates": len(organization_ids), 996 }, 997 ) 998 999 def list_organizations( 1000 self, 1001 *, 1002 name_contains: str | None = None, 1003 limit: int | None = None, 1004 ) -> list[CloudOrganization]: 1005 """List organizations available to this client. 1006 1007 See the module docstring for how organization search and limits are resolved. 1008 """ 1009 if limit is not None and limit <= 0: 1010 raise exc.PyAirbyteInputError(message="`limit` must be greater than 0.") 1011 1012 if name_contains is not None or limit is not None: 1013 try: 1014 return self._list_organizations_by_user_id( 1015 name_contains=name_contains, 1016 limit=limit, 1017 ) 1018 except AirbyteError: 1019 pass 1020 1021 organizations = self._fetch_organizations() 1022 if name_contains is not None: 1023 name_substring = name_contains.casefold() 1024 organizations = [ 1025 organization 1026 for organization in organizations 1027 if name_substring in (organization.organization_name or "").casefold() 1028 ] 1029 return organizations if limit is None else organizations[:limit] 1030 1031 def _list_organizations_by_user_id( 1032 self, 1033 *, 1034 name_contains: str | None = None, 1035 limit: int | None = None, 1036 ) -> list[CloudOrganization]: 1037 """List organizations via the Config API, with server-side search and paging.""" 1038 user_id = self._get_authenticated_user_id() 1039 return [ 1040 self._organization_from_mapping(organization) 1041 for organization in api_util.list_organizations_for_user_id( 1042 user_id=user_id, 1043 api_root=self.public_api_root, 1044 config_api_root=self.config_api_root, 1045 client_id=self.client_id, 1046 client_secret=self.client_secret, 1047 bearer_token=self._get_config_api_bearer_token(), 1048 name_contains=name_contains, 1049 limit=limit, 1050 ) 1051 ] 1052 1053 def _organization_from_mapping( 1054 self, 1055 organization: Mapping[str, Any], 1056 ) -> CloudOrganization: 1057 """Build a `CloudOrganization` from a Config API organization mapping.""" 1058 return CloudOrganization( 1059 organization_id=organization["organizationId"], 1060 organization_name=organization.get("organizationName"), 1061 email=organization.get("email"), 1062 client_id=self.client_id, 1063 client_secret=self.client_secret, 1064 bearer_token=self.bearer_token, 1065 public_api_root=self.public_api_root, 1066 config_api_root=self.config_api_root, 1067 ) 1068 1069 def _fetch_organizations(self) -> list[CloudOrganization]: 1070 """Fetch all organizations available to this client.""" 1071 return [ 1072 CloudOrganization( 1073 organization_id=organization.organization_id, 1074 organization_name=organization.organization_name, 1075 email=organization.email, 1076 client_id=self.client_id, 1077 client_secret=self.client_secret, 1078 bearer_token=self.bearer_token, 1079 public_api_root=self.public_api_root, 1080 config_api_root=self.config_api_root, 1081 ) 1082 for organization in api_util.list_organizations_for_user( 1083 api_root=self.public_api_root, 1084 client_id=self.client_id, 1085 client_secret=self.client_secret, 1086 bearer_token=self.bearer_token, 1087 ) 1088 ] 1089 1090 def _resolve_default_organization_id(self) -> str | None: 1091 """Resolve the organization to use when no organization argument is given.""" 1092 return self._resolve_ambient_organization_id() 1093 1094 def _get_organization_by_id(self, organization_id: str) -> CloudOrganization | None: 1095 """Look up a single organization via the Config API, if available.""" 1096 try: 1097 organization_info = api_util.get_organization_info( 1098 organization_id=organization_id, 1099 api_root=self.public_api_root, 1100 config_api_root=self.config_api_root, 1101 client_id=self.client_id, 1102 client_secret=self.client_secret, 1103 bearer_token=self._get_config_api_bearer_token(), 1104 ) 1105 except AirbyteError: 1106 return None 1107 if not isinstance(organization_info.get("organizationId"), str): 1108 return None 1109 return self._organization_from_mapping(organization_info) 1110 1111 def _search_organizations_by_name( 1112 self, 1113 organization_name: str | None, 1114 ) -> list[CloudOrganization]: 1115 """Get organizations whose names contain `organization_name`, if available.""" 1116 if organization_name is not None: 1117 try: 1118 return self._list_organizations_by_user_id(name_contains=organization_name) 1119 except AirbyteError: 1120 pass 1121 return self._fetch_organizations() 1122 1123 def get_organization( 1124 self, 1125 organization_id: str | None = None, 1126 *, 1127 organization_name: str | None = None, 1128 ) -> CloudOrganization: 1129 """Resolve an organization by ID or exact name. 1130 1131 See the module docstring for how the organization is resolved when no 1132 argument is given. 1133 """ 1134 resolved_organization_id = organization_id 1135 if resolved_organization_id and organization_name: 1136 raise exc.PyAirbyteInputError( 1137 message="Provide either organization ID or organization name." 1138 ) 1139 if resolved_organization_id is None and organization_name is None: 1140 resolved_organization_id = self._resolve_default_organization_id() 1141 if not resolved_organization_id and not organization_name: 1142 raise exc.PyAirbyteInputError( 1143 message="Organization ID or organization name is required.", 1144 guidance=( 1145 "Provide an organization ID or name, or call `get_default_cloud_context` " 1146 "to discover your organizations." 1147 ), 1148 ) 1149 1150 if resolved_organization_id: 1151 organization = self._get_organization_by_id(resolved_organization_id) 1152 if organization is not None: 1153 return organization 1154 matching_organizations = [ 1155 candidate 1156 for candidate in self._fetch_organizations() 1157 if candidate.organization_id == resolved_organization_id 1158 ] 1159 else: 1160 matching_organizations = [ 1161 candidate 1162 for candidate in self._search_organizations_by_name(organization_name) 1163 if candidate.organization_name == organization_name 1164 ] 1165 1166 if not matching_organizations: 1167 raise AirbyteMissingResourceError( 1168 resource_type="organization", 1169 resource_name_or_id=resolved_organization_id or organization_name, 1170 ) 1171 if len(matching_organizations) > 1: 1172 total_matches = len(matching_organizations) 1173 shown_matches = matching_organizations[:10] 1174 match_details = ", ".join( 1175 f"{organization.organization_id} ({organization.email or 'email unavailable'})" 1176 for organization in shown_matches 1177 ) 1178 raise exc.PyAirbyteInputError( 1179 message=( 1180 "Organization name matches multiple organizations. Provide an " 1181 f"organization ID to disambiguate. Matching organizations " 1182 f"(showing {len(shown_matches)} of {total_matches}): {match_details}" 1183 ), 1184 context={ 1185 "organization_name": organization_name, 1186 "matching_organizations": [ 1187 { 1188 "organization_id": organization.organization_id, 1189 "email": organization.email, 1190 } 1191 for organization in shown_matches 1192 ], 1193 "total_matches": total_matches, 1194 }, 1195 ) 1196 1197 return matching_organizations[0]
Authenticated client for Airbyte Cloud and self-managed Airbyte APIs.
126 def __init__( 127 self, 128 *, 129 client_id: str | SecretString | None = None, 130 client_secret: str | SecretString | None = None, 131 bearer_token: str | SecretString | None = None, 132 public_api_root: str | None = None, 133 config_api_root: str | None = None, 134 workspace_id: str | None = None, 135 organization_id: str | None = None, 136 ) -> None: 137 """Initialize a `CloudClient` from explicit auth values.""" 138 self._credentials = _AirbyteCredentials.from_auth( 139 client_id=client_id, 140 client_secret=client_secret, 141 bearer_token=bearer_token, 142 public_api_root=public_api_root, 143 config_api_root=config_api_root, 144 workspace_id=workspace_id, 145 organization_id=organization_id, 146 env_vars=False, 147 ) 148 self._membership_organization_ids = None 149 self._user_permissions = None 150 self._direct_workspace_infos = {} 151 self._workspace_organizations = {} 152 self._validated_direct_workspace_result = None 153 self._authenticated_user_info = None 154 self._authenticated_user_id = None 155 self._authenticated_bearer_token = None
Initialize a CloudClient from explicit auth values.
157 @property 158 def client_id(self) -> SecretString | None: 159 """OAuth client ID used for authentication.""" 160 return self._credentials.client_id
OAuth client ID used for authentication.
162 @property 163 def client_secret(self) -> SecretString | None: 164 """OAuth client secret used for authentication.""" 165 return self._credentials.client_secret
OAuth client secret used for authentication.
167 @property 168 def bearer_token(self) -> SecretString | None: 169 """Bearer token used for authentication.""" 170 return self._credentials.bearer_token
Bearer token used for authentication.
172 @property 173 def public_api_root(self) -> str: 174 """Airbyte Public API root.""" 175 return self._credentials.public_api_root
Airbyte Public API root.
177 @property 178 def config_api_root(self) -> str | None: 179 """Airbyte Config API root.""" 180 return self._credentials.config_api_root
Airbyte Config API root.
182 @property 183 def organization_id(self) -> str | None: 184 """Default organization ID for organization-scoped operations.""" 185 return self._credentials.organization_id
Default organization ID for organization-scoped operations.
187 @property 188 def default_workspace_id(self) -> str | None: 189 """Default workspace ID for workspace-scoped operations.""" 190 return self._credentials.workspace_id
Default workspace ID for workspace-scoped operations.
192 @classmethod 193 def from_auth( 194 cls, 195 *, 196 env_vars: bool = False, 197 organization_id: str | None = None, 198 client_id: str | SecretString | None = None, 199 client_secret: str | SecretString | None = None, 200 bearer_token: str | SecretString | None = None, 201 public_api_root: str | None = None, 202 config_api_root: str | None = None, 203 ) -> CloudClient: 204 """Create a client from explicit inputs and optionally environment variables. 205 206 When `env_vars` is True, environment variables are checked as a fallback 207 after any explicitly provided values. 208 """ 209 credentials = _AirbyteCredentials.from_auth( 210 organization_id=organization_id, 211 client_id=client_id, 212 client_secret=client_secret, 213 bearer_token=bearer_token, 214 public_api_root=public_api_root, 215 config_api_root=config_api_root, 216 env_vars=env_vars, 217 ) 218 return cls._from_credentials(credentials)
Create a client from explicit inputs and optionally environment variables.
When env_vars is True, environment variables are checked as a fallback
after any explicitly provided values.
233 def get_workspace(self, workspace_id: str | None = None) -> CloudWorkspace: 234 """Create a `CloudWorkspace` using this client's credentials. 235 236 See the module docstring for how the workspace is resolved. 237 """ 238 resolved_workspace_id = workspace_id or self.resolve_default_workspace_id() 239 if not resolved_workspace_id: 240 raise exc.PyAirbyteInputError( 241 message="Workspace ID is required.", 242 guidance=( 243 "No workspace was configured, and no default workspace could be resolved " 244 "for the authenticated user. Provide a workspace ID, or call " 245 "`get_default_cloud_context` to discover your workspaces and organizations." 246 ), 247 ) 248 249 credentials = self._credentials.with_workspace_id(resolved_workspace_id) 250 return CloudWorkspace( 251 workspace_id=credentials.workspace_id, 252 client_id=credentials.client_id, 253 client_secret=credentials.client_secret, 254 bearer_token=credentials.bearer_token, 255 api_root=credentials.public_api_root, 256 config_api_root=credentials.config_api_root, 257 )
Create a CloudWorkspace using this client's credentials.
See the module docstring for how the workspace is resolved.
259 def create_workspace( 260 self, 261 *, 262 name: str, 263 organization_id: str | None = None, 264 region_id: str | None = None, 265 ) -> CloudWorkspaceInfo: 266 """Create an Airbyte workspace.""" 267 resolved_organization_id = organization_id or self.organization_id 268 workspace = api_util.create_workspace( 269 name=name, 270 organization_id=resolved_organization_id, 271 region_id=region_id, 272 api_root=self.public_api_root, 273 client_id=self.client_id, 274 client_secret=self.client_secret, 275 bearer_token=self.bearer_token, 276 ) 277 return CloudWorkspaceInfo.from_api_response(workspace)
Create an Airbyte workspace.
279 def rename_workspace( 280 self, 281 workspace_id: str, 282 *, 283 name: str, 284 ) -> CloudWorkspaceInfo: 285 """Rename an Airbyte workspace.""" 286 workspace = api_util.rename_workspace( 287 workspace_id=workspace_id, 288 name=name, 289 api_root=self.public_api_root, 290 client_id=self.client_id, 291 client_secret=self.client_secret, 292 bearer_token=self.bearer_token, 293 ) 294 return CloudWorkspaceInfo.from_api_response(workspace)
Rename an Airbyte workspace.
296 def permanently_delete_workspace( 297 self, 298 workspace_id: str, 299 *, 300 workspace_name: str | None = None, 301 safe_mode: bool = True, 302 ) -> None: 303 """Permanently delete an Airbyte workspace if it has no connections. 304 305 When `safe_mode` is enabled, the workspace name must contain `delete-me` 306 or `deleteme`. This also checks for existing connections before deleting 307 and raises `AirbyteWorkspaceNotEmptyError` if the workspace is not empty. 308 """ 309 api_util.permanently_delete_workspace( 310 workspace_id=workspace_id, 311 workspace_name=workspace_name, 312 api_root=self.public_api_root, 313 client_id=self.client_id, 314 client_secret=self.client_secret, 315 bearer_token=self.bearer_token, 316 safe_mode=safe_mode, 317 )
Permanently delete an Airbyte workspace if it has no connections.
When safe_mode is enabled, the workspace name must contain delete-me
or deleteme. This also checks for existing connections before deleting
and raises AirbyteWorkspaceNotEmptyError if the workspace is not empty.
351 def list_workspaces( # noqa: PLR0911, PLR0913 352 self, 353 name: str | None = None, 354 *, 355 organization_id: str | None = None, 356 organization_name: str | None = None, 357 workspace_id: str | None = None, 358 name_contains: str | None = None, 359 name_filter: Callable[[str], bool] | None = None, 360 limit: int | None = None, 361 privilege_scope: WorkspacePrivilegeScope = WorkspacePrivilegeScope.MEMBER_OF, 362 all_organizations: bool = False, 363 ) -> list[CloudWorkspaceInfo]: 364 """List workspaces available to this client. 365 366 `privilege_scope` controls whether this lists direct member workspaces, 367 organization workspaces, or instance-wide workspaces. The deprecated 368 `all_organizations` alias maps to `WorkspacePrivilegeScope.ANY`. 369 """ 370 if limit is not None and limit <= 0: 371 raise exc.PyAirbyteInputError(message="`limit` must be greater than 0.") 372 if organization_id is not None and organization_name is not None: 373 raise exc.PyAirbyteInputError( 374 message="Provide either organization ID or organization name." 375 ) 376 has_explicit_organization = organization_id is not None or organization_name is not None 377 has_explicit_workspace = workspace_id is not None 378 379 if all_organizations: 380 if privilege_scope is not WorkspacePrivilegeScope.MEMBER_OF: 381 raise exc.PyAirbyteInputError( 382 message="all_organizations cannot be combined with privilege_scope." 383 ) 384 warnings.warn( 385 "`all_organizations` is deprecated; use `privilege_scope` instead.", 386 DeprecationWarning, 387 stacklevel=2, 388 ) 389 privilege_scope = WorkspacePrivilegeScope.ANY 390 if name_contains is not None and name_filter is not None: 391 raise exc.PyAirbyteInputError( 392 message="You can provide name_contains or name_filter, but not both." 393 ) 394 if name is not None and name_contains is not None: 395 raise exc.PyAirbyteInputError( 396 message="You can provide name or name_contains, but not both." 397 ) 398 if has_explicit_organization or has_explicit_workspace: 399 resolved_organization_id = self._resolve_workspace_organization_id( 400 organization_id=organization_id, 401 organization_name=organization_name, 402 workspace_id=workspace_id, 403 ) 404 if resolved_organization_id is None: 405 return [] 406 return self._list_workspaces_in_organizations( 407 (resolved_organization_id,), 408 name=name, 409 name_contains=name_contains, 410 name_filter=name_filter, 411 limit=limit, 412 ) 413 414 if privilege_scope is WorkspacePrivilegeScope.MEMBER_OF: 415 return self._list_member_workspaces( 416 name=name, 417 name_contains=name_contains, 418 name_filter=name_filter, 419 limit=limit, 420 ) 421 422 if privilege_scope is WorkspacePrivilegeScope.INSTANCE_ADMIN: 423 if not self._is_instance_admin(): 424 raise exc.PyAirbyteInputError( 425 message="privilege_scope=instance_admin requires the instance_admin permission." 426 ) 427 return self._list_unscoped_workspaces( 428 name=name, 429 name_contains=name_contains, 430 name_filter=name_filter, 431 limit=limit, 432 ) 433 434 if privilege_scope is WorkspacePrivilegeScope.ANY and self._is_instance_admin(): 435 return self._list_unscoped_workspaces( 436 name=name, 437 name_contains=name_contains, 438 name_filter=name_filter, 439 limit=limit, 440 ) 441 442 if privilege_scope in { 443 WorkspacePrivilegeScope.ORGANIZATION_ADMIN, 444 WorkspacePrivilegeScope.ANY, 445 }: 446 organization_ids = self._get_membership_organization_ids() 447 if not organization_ids: 448 return [] 449 return self._list_workspaces_in_organizations( 450 organization_ids, 451 name=name, 452 name_contains=name_contains, 453 name_filter=name_filter, 454 limit=limit, 455 ) 456 457 raise exc.PyAirbyteInputError(message="Unsupported workspace privilege scope.")
List workspaces available to this client.
privilege_scope controls whether this lists direct member workspaces,
organization workspaces, or instance-wide workspaces. The deprecated
all_organizations alias maps to WorkspacePrivilegeScope.ANY.
641 def get_workspace_parent_organization_id(self, workspace_id: str) -> str | None: 642 """Return the parent organization ID of a workspace, or `None` if it cannot be resolved.""" 643 try: 644 return self._get_workspace_parent_organization_id(workspace_id) 645 except (exc.AirbyteError, exc.PyAirbyteInputError): 646 return None
Return the parent organization ID of a workspace, or None if it cannot be resolved.
697 def resolve_default_workspace_id(self) -> str | None: 698 """Resolve the configured or authenticated user's default workspace ID.""" 699 configured_workspace_id = self.default_workspace_id 700 if configured_workspace_id: 701 return configured_workspace_id 702 user_default_workspace_id = self._get_user_default_workspace_id() 703 if user_default_workspace_id: 704 return user_default_workspace_id 705 try: 706 live_workspaces, unvalidated_count = self._validate_direct_workspaces() 707 except (AirbyteError, exc.PyAirbyteInputError): 708 return None 709 return ( 710 live_workspaces[0].workspace_id 711 if unvalidated_count == 0 and len(live_workspaces) == 1 712 else None 713 )
Resolve the configured or authenticated user's default workspace ID.
846 def get_default_context_for_user(self) -> CloudDefaultContextInfo: 847 """Describe the authenticated user's explicit Cloud affinities.""" 848 user_id: str | None = None 849 user_name: str | None = None 850 user_email: str | None = None 851 try: 852 user = self._get_authenticated_user_info() 853 except (AirbyteError, exc.PyAirbyteInputError): 854 pass 855 else: 856 user_id = user.get("userId") if isinstance(user.get("userId"), str) else None 857 user_name = user.get("name") if isinstance(user.get("name"), str) else None 858 user_email = user.get("email") if isinstance(user.get("email"), str) else None 859 860 default_workspace_id = self.resolve_default_workspace_id() 861 try: 862 permissions = self._get_user_permissions() 863 except (AirbyteError, exc.PyAirbyteInputError): 864 permissions = () 865 membership_organization_ids = () 866 member_workspaces = [] 867 member_organizations_truncated = False 868 unvalidated_workspace_count = 0 869 else: 870 membership_organization_ids = self._get_membership_organization_ids() 871 member_organizations_truncated = ( 872 len(membership_organization_ids) > MAX_ORGANIZATION_CANDIDATES 873 ) 874 try: 875 member_workspaces, unvalidated_workspace_count = self._validate_direct_workspaces() 876 except (AirbyteError, exc.PyAirbyteInputError): 877 member_workspaces = [] 878 unvalidated_workspace_count = 0 879 member_workspaces = member_workspaces[:] 880 881 member_organizations = [ 882 CloudOrganizationInfo.model_validate(candidate) 883 for candidate in self._get_organization_candidates( 884 membership_organization_ids[:MAX_ORGANIZATION_CANDIDATES] 885 ) 886 ] 887 default_workspace_info: CloudWorkspaceInfo | None = None 888 default_workspace_organization: CloudOrganizationInfo | None = None 889 if default_workspace_id is not None: 890 try: 891 default_workspace_info = self._get_direct_workspace_info(default_workspace_id) 892 except (AirbyteError, exc.PyAirbyteInputError): 893 default_workspace_info = None 894 if default_workspace_info is not None: 895 default_workspace_organization = self._get_workspace_organization( 896 default_workspace_id 897 ) 898 discovery_hints: list[str] = [] 899 if any(permission.get("permissionType") == "instance_admin" for permission in permissions): 900 discovery_hints.append( 901 "Instance-admin access may include every organization and workspace in the " 902 "instance. Use list_cloud_organizations(name_contains=...) or " 903 "list_cloud_workspaces(organization_id=...) to discover others." 904 ) 905 if membership_organization_ids: 906 discovery_hints.append( 907 "Organization membership grants access to every workspace in those " 908 "organizations. Use list_cloud_workspaces(organization_id=<id>) to " 909 "discover workspaces." 910 ) 911 return CloudDefaultContextInfo( 912 user_id=user_id, 913 user_name=user_name, 914 user_email=user_email, 915 default_workspace_id=default_workspace_id, 916 default_workspace_name=( 917 default_workspace_info.name if default_workspace_info else None 918 ), 919 default_workspace_verified=default_workspace_info is not None, 920 default_organization_id=( 921 default_workspace_organization.organization_id 922 if default_workspace_organization is not None 923 else None 924 ), 925 default_organization_name=( 926 default_workspace_organization.organization_name 927 if default_workspace_organization is not None 928 else None 929 ), 930 configured_workspace_id=self.default_workspace_id, 931 configured_organization_id=self.organization_id, 932 member_organizations=member_organizations, 933 member_workspaces=member_workspaces, 934 member_organizations_truncated=member_organizations_truncated, 935 member_workspaces_truncated=unvalidated_workspace_count > 0, 936 unvalidated_workspace_count=unvalidated_workspace_count, 937 discovery_hints=discovery_hints, 938 )
Describe the authenticated user's explicit Cloud affinities.
999 def list_organizations( 1000 self, 1001 *, 1002 name_contains: str | None = None, 1003 limit: int | None = None, 1004 ) -> list[CloudOrganization]: 1005 """List organizations available to this client. 1006 1007 See the module docstring for how organization search and limits are resolved. 1008 """ 1009 if limit is not None and limit <= 0: 1010 raise exc.PyAirbyteInputError(message="`limit` must be greater than 0.") 1011 1012 if name_contains is not None or limit is not None: 1013 try: 1014 return self._list_organizations_by_user_id( 1015 name_contains=name_contains, 1016 limit=limit, 1017 ) 1018 except AirbyteError: 1019 pass 1020 1021 organizations = self._fetch_organizations() 1022 if name_contains is not None: 1023 name_substring = name_contains.casefold() 1024 organizations = [ 1025 organization 1026 for organization in organizations 1027 if name_substring in (organization.organization_name or "").casefold() 1028 ] 1029 return organizations if limit is None else organizations[:limit]
List organizations available to this client.
See the module docstring for how organization search and limits are resolved.
1123 def get_organization( 1124 self, 1125 organization_id: str | None = None, 1126 *, 1127 organization_name: str | None = None, 1128 ) -> CloudOrganization: 1129 """Resolve an organization by ID or exact name. 1130 1131 See the module docstring for how the organization is resolved when no 1132 argument is given. 1133 """ 1134 resolved_organization_id = organization_id 1135 if resolved_organization_id and organization_name: 1136 raise exc.PyAirbyteInputError( 1137 message="Provide either organization ID or organization name." 1138 ) 1139 if resolved_organization_id is None and organization_name is None: 1140 resolved_organization_id = self._resolve_default_organization_id() 1141 if not resolved_organization_id and not organization_name: 1142 raise exc.PyAirbyteInputError( 1143 message="Organization ID or organization name is required.", 1144 guidance=( 1145 "Provide an organization ID or name, or call `get_default_cloud_context` " 1146 "to discover your organizations." 1147 ), 1148 ) 1149 1150 if resolved_organization_id: 1151 organization = self._get_organization_by_id(resolved_organization_id) 1152 if organization is not None: 1153 return organization 1154 matching_organizations = [ 1155 candidate 1156 for candidate in self._fetch_organizations() 1157 if candidate.organization_id == resolved_organization_id 1158 ] 1159 else: 1160 matching_organizations = [ 1161 candidate 1162 for candidate in self._search_organizations_by_name(organization_name) 1163 if candidate.organization_name == organization_name 1164 ] 1165 1166 if not matching_organizations: 1167 raise AirbyteMissingResourceError( 1168 resource_type="organization", 1169 resource_name_or_id=resolved_organization_id or organization_name, 1170 ) 1171 if len(matching_organizations) > 1: 1172 total_matches = len(matching_organizations) 1173 shown_matches = matching_organizations[:10] 1174 match_details = ", ".join( 1175 f"{organization.organization_id} ({organization.email or 'email unavailable'})" 1176 for organization in shown_matches 1177 ) 1178 raise exc.PyAirbyteInputError( 1179 message=( 1180 "Organization name matches multiple organizations. Provide an " 1181 f"organization ID to disambiguate. Matching organizations " 1182 f"(showing {len(shown_matches)} of {total_matches}): {match_details}" 1183 ), 1184 context={ 1185 "organization_name": organization_name, 1186 "matching_organizations": [ 1187 { 1188 "organization_id": organization.organization_id, 1189 "email": organization.email, 1190 } 1191 for organization in shown_matches 1192 ], 1193 "total_matches": total_matches, 1194 }, 1195 ) 1196 1197 return matching_organizations[0]
Resolve an organization by ID or exact name.
See the module docstring for how the organization is resolved when no argument is given.
22class CloudOrganization: 23 """Information about an organization in Airbyte Cloud. 24 25 This class provides lazy loading of organization attributes including billing status. 26 It is typically created via `CloudWorkspace.get_organization()`. 27 """ 28 29 def __init__( 30 self, 31 organization_id: str, 32 organization_name: str | None = None, 33 email: str | None = None, 34 *, 35 client_id: str | SecretString | None = None, 36 client_secret: str | SecretString | None = None, 37 bearer_token: str | SecretString | None = None, 38 public_api_root: str | None = None, 39 config_api_root: str | None = None, 40 ) -> None: 41 """Initialize a `CloudOrganization`.""" 42 self.organization_id = organization_id 43 """The organization ID.""" 44 45 self._organization_name = organization_name 46 """Display name of the organization.""" 47 48 self._email = email 49 """Email associated with the organization.""" 50 51 self._credentials = _AirbyteCredentials( 52 client_id=SecretString(client_id) if client_id else None, 53 client_secret=SecretString(client_secret) if client_secret else None, 54 bearer_token=SecretString(bearer_token) if bearer_token else None, 55 public_api_root=public_api_root or api_util.CLOUD_API_ROOT, 56 config_api_root=config_api_root, 57 organization_id=organization_id, 58 ) 59 self._organization_info: dict[str, Any] | None = None 60 self._organization_info_fetch_failed: bool = False 61 62 def _fetch_organization_info(self, *, force_refresh: bool = False) -> dict[str, Any]: 63 """Fetch and cache organization info including billing status.""" 64 if force_refresh: 65 self._organization_info_fetch_failed = False 66 67 if self._organization_info_fetch_failed and self._organization_info is None: 68 return {} 69 70 if not force_refresh and self._organization_info is not None: 71 return self._organization_info 72 73 try: 74 self._organization_info = api_util.get_organization_info( 75 organization_id=self.organization_id, 76 api_root=self._credentials.public_api_root, 77 config_api_root=self._credentials.config_api_root, 78 client_id=self._credentials.client_id, 79 client_secret=self._credentials.client_secret, 80 bearer_token=self._credentials.bearer_token, 81 ) 82 except Exception as ex: 83 logger.debug("Failed to fetch organization info.", exc_info=ex) 84 if self._organization_info is None: 85 self._organization_info_fetch_failed = True 86 return self._organization_info or {} 87 else: 88 return self._organization_info 89 90 @property 91 def organization_name(self) -> str | None: 92 """Display name of the organization.""" 93 if self._organization_name is not None: 94 return self._organization_name 95 info = self._fetch_organization_info() 96 return info.get("organizationName") 97 98 @property 99 def email(self) -> str | None: 100 """Email associated with the organization.""" 101 if self._email is not None: 102 return self._email 103 info = self._fetch_organization_info() 104 return info.get("email") 105 106 def get_billing_status(self) -> CloudOrganizationBillingInfo: 107 """Fetch billing status for the organization or raise on failure.""" 108 try: 109 info = api_util.get_organization_info( 110 organization_id=self.organization_id, 111 api_root=self._credentials.public_api_root, 112 config_api_root=self._credentials.config_api_root, 113 client_id=self._credentials.client_id, 114 client_secret=self._credentials.client_secret, 115 bearer_token=self._credentials.bearer_token, 116 ) 117 except (requests.RequestException, ValueError) as ex: 118 raise AirbyteError( 119 message="Failed to retrieve organization billing information.", 120 context={"organization_id": self.organization_id}, 121 ) from ex 122 billing = info.get("billing") 123 if not isinstance(billing, dict): 124 raise AirbyteError( 125 message="Organization info did not include billing details.", 126 context={"organization_id": self.organization_id}, 127 ) 128 payment_status = billing.get("paymentStatus") 129 subscription_status = billing.get("subscriptionStatus") 130 return CloudOrganizationBillingInfo( 131 payment_status=payment_status if isinstance(payment_status, str) else None, 132 subscription_status=( 133 subscription_status if isinstance(subscription_status, str) else None 134 ), 135 is_account_locked=api_util.is_account_locked(payment_status, subscription_status), 136 ) 137 138 @property 139 def payment_status(self) -> str | None: 140 """Payment status of the organization.""" 141 info = self._fetch_organization_info() 142 return (info.get("billing") or {}).get("paymentStatus") 143 144 @property 145 def subscription_status(self) -> str | None: 146 """Subscription status of the organization.""" 147 info = self._fetch_organization_info() 148 return (info.get("billing") or {}).get("subscriptionStatus") 149 150 @property 151 def is_account_locked(self) -> bool: 152 """Whether the account is locked due to billing issues.""" 153 return api_util.is_account_locked(self.payment_status, self.subscription_status)
Information about an organization in Airbyte Cloud.
This class provides lazy loading of organization attributes including billing status.
It is typically created via CloudWorkspace.get_organization().
29 def __init__( 30 self, 31 organization_id: str, 32 organization_name: str | None = None, 33 email: str | None = None, 34 *, 35 client_id: str | SecretString | None = None, 36 client_secret: str | SecretString | None = None, 37 bearer_token: str | SecretString | None = None, 38 public_api_root: str | None = None, 39 config_api_root: str | None = None, 40 ) -> None: 41 """Initialize a `CloudOrganization`.""" 42 self.organization_id = organization_id 43 """The organization ID.""" 44 45 self._organization_name = organization_name 46 """Display name of the organization.""" 47 48 self._email = email 49 """Email associated with the organization.""" 50 51 self._credentials = _AirbyteCredentials( 52 client_id=SecretString(client_id) if client_id else None, 53 client_secret=SecretString(client_secret) if client_secret else None, 54 bearer_token=SecretString(bearer_token) if bearer_token else None, 55 public_api_root=public_api_root or api_util.CLOUD_API_ROOT, 56 config_api_root=config_api_root, 57 organization_id=organization_id, 58 ) 59 self._organization_info: dict[str, Any] | None = None 60 self._organization_info_fetch_failed: bool = False
Initialize a CloudOrganization.
90 @property 91 def organization_name(self) -> str | None: 92 """Display name of the organization.""" 93 if self._organization_name is not None: 94 return self._organization_name 95 info = self._fetch_organization_info() 96 return info.get("organizationName")
Display name of the organization.
98 @property 99 def email(self) -> str | None: 100 """Email associated with the organization.""" 101 if self._email is not None: 102 return self._email 103 info = self._fetch_organization_info() 104 return info.get("email")
Email associated with the organization.
106 def get_billing_status(self) -> CloudOrganizationBillingInfo: 107 """Fetch billing status for the organization or raise on failure.""" 108 try: 109 info = api_util.get_organization_info( 110 organization_id=self.organization_id, 111 api_root=self._credentials.public_api_root, 112 config_api_root=self._credentials.config_api_root, 113 client_id=self._credentials.client_id, 114 client_secret=self._credentials.client_secret, 115 bearer_token=self._credentials.bearer_token, 116 ) 117 except (requests.RequestException, ValueError) as ex: 118 raise AirbyteError( 119 message="Failed to retrieve organization billing information.", 120 context={"organization_id": self.organization_id}, 121 ) from ex 122 billing = info.get("billing") 123 if not isinstance(billing, dict): 124 raise AirbyteError( 125 message="Organization info did not include billing details.", 126 context={"organization_id": self.organization_id}, 127 ) 128 payment_status = billing.get("paymentStatus") 129 subscription_status = billing.get("subscriptionStatus") 130 return CloudOrganizationBillingInfo( 131 payment_status=payment_status if isinstance(payment_status, str) else None, 132 subscription_status=( 133 subscription_status if isinstance(subscription_status, str) else None 134 ), 135 is_account_locked=api_util.is_account_locked(payment_status, subscription_status), 136 )
Fetch billing status for the organization or raise on failure.
138 @property 139 def payment_status(self) -> str | None: 140 """Payment status of the organization.""" 141 info = self._fetch_organization_info() 142 return (info.get("billing") or {}).get("paymentStatus")
Payment status of the organization.
70@dataclass(init=False, kw_only=True) # noqa: PLR0904 # Core cloud API facade. 71class CloudWorkspace: 72 """A remote workspace on the Airbyte Cloud. 73 74 By overriding `api_root`, you can use this class to interact with self-managed Airbyte 75 instances, both OSS and Enterprise. 76 77 Two authentication methods are supported (mutually exclusive): 78 1. OAuth2 client credentials (client_id + client_secret) 79 2. Bearer token authentication 80 81 Example with client credentials: 82 ```python 83 workspace = CloudWorkspace( 84 workspace_id="...", 85 client_id="...", 86 client_secret="...", 87 ) 88 ``` 89 90 Example with bearer token: 91 ```python 92 workspace = CloudWorkspace( 93 workspace_id="...", 94 bearer_token="...", 95 ) 96 ``` 97 """ 98 99 workspace_id: str 100 client_id: SecretString | None 101 client_secret: SecretString | None 102 api_root: str 103 config_api_root: str | None 104 """The Config API root URL.""" 105 bearer_token: SecretString | None 106 107 # Internal credentials objects (set in __init__, excluded from repr) 108 _credentials: _AirbyteCredentials = field(init=False, repr=False) 109 _client_config: CloudClientConfig = field(init=False, repr=False) 110 111 def __init__( 112 self, 113 *, 114 workspace_id: str | None = None, 115 client_id: str | SecretString | None = None, 116 client_secret: str | SecretString | None = None, 117 api_root: str | None = None, 118 config_api_root: str | None = None, 119 bearer_token: str | SecretString | None = None, 120 ) -> None: 121 """Validate and initialize credentials.""" 122 env_vars = not (client_id or client_secret or bearer_token) 123 credentials = _AirbyteCredentials.from_auth( 124 workspace_id=workspace_id, 125 client_id=client_id, 126 client_secret=client_secret, 127 bearer_token=bearer_token, 128 public_api_root=api_root, 129 config_api_root=config_api_root, 130 env_vars=env_vars, 131 ) 132 if not credentials.workspace_id: 133 raise exc.PyAirbyteInputError( 134 message="Workspace ID is required.", 135 guidance=( 136 "Provide a workspace ID, or call `get_default_cloud_context` to discover " 137 "available workspaces." 138 ), 139 ) 140 141 self._credentials = credentials 142 self.workspace_id = credentials.workspace_id or "" 143 self.client_id = credentials.client_id 144 self.client_secret = credentials.client_secret 145 self.bearer_token = credentials.bearer_token 146 self.api_root = credentials.public_api_root 147 self.config_api_root = credentials.config_api_root 148 149 # Create internal CloudClientConfig object (validates mutual exclusivity) 150 self._client_config = CloudClientConfig( 151 client_id=self.client_id, 152 client_secret=self.client_secret, 153 bearer_token=self.bearer_token, 154 api_root=self.api_root, 155 config_api_root=self.config_api_root, 156 ) 157 158 @classmethod 159 def from_env( 160 cls, 161 workspace_id: str | None = None, 162 *, 163 api_root: str | None = None, 164 config_api_root: str | None = None, 165 ) -> CloudWorkspace: 166 """Create a CloudWorkspace using credentials from environment variables. 167 168 This factory method resolves credentials from environment variables, 169 providing a convenient way to create a workspace without explicitly 170 passing credentials. 171 172 Two authentication methods are supported (mutually exclusive): 173 1. Bearer token (checked first) 174 2. OAuth2 client credentials (fallback) 175 176 Environment variables used: 177 - `AIRBYTE_CLOUD_BEARER_TOKEN`: Bearer token (alternative to client credentials). 178 - `AIRBYTE_CLOUD_CLIENT_ID`: OAuth client ID (for client credentials flow). 179 - `AIRBYTE_CLOUD_CLIENT_SECRET`: OAuth client secret (for client credentials flow). 180 - `AIRBYTE_CLOUD_WORKSPACE_ID`: The workspace ID (if not passed as argument). 181 - `AIRBYTE_CLOUD_API_URL`: Optional. The API root URL (defaults to Airbyte Cloud). 182 - `AIRBYTE_CLOUD_CONFIG_API_URL`: Optional. The Config API root URL. 183 184 Args: 185 workspace_id: The workspace ID. If not provided, will be resolved from 186 the `AIRBYTE_CLOUD_WORKSPACE_ID` environment variable. 187 api_root: The API root URL. If not provided, will be resolved from 188 the `AIRBYTE_CLOUD_API_URL` environment variable, or default to 189 the Airbyte Cloud API. 190 config_api_root: The Config API root URL. If not provided, will be resolved 191 from the `AIRBYTE_CLOUD_CONFIG_API_URL` environment variable. 192 193 Returns: 194 A CloudWorkspace instance configured with credentials from the environment. 195 196 Raises: 197 PyAirbyteInputError: If required credentials are not found in 198 the environment or are incomplete. 199 200 Example: 201 ```python 202 # With workspace_id from environment 203 workspace = CloudWorkspace.from_env() 204 205 # With explicit workspace_id 206 workspace = CloudWorkspace.from_env(workspace_id="your-workspace-id") 207 ``` 208 """ 209 return cls( 210 workspace_id=workspace_id, 211 api_root=api_root, 212 config_api_root=config_api_root, 213 ) 214 215 @property 216 def workspace_url(self) -> str | None: 217 """The web URL of the workspace.""" 218 return f"{get_web_url_root(self.api_root)}/workspaces/{self.workspace_id}" 219 220 @cached_property 221 def _organization_info(self) -> dict[str, Any]: 222 """Fetch and cache organization info for this workspace. 223 224 Uses the Config API endpoint for an efficient O(1) lookup. 225 This is an internal method; use get_organization() for public access. 226 """ 227 return api_util.get_workspace_organization_info( 228 workspace_id=self.workspace_id, 229 api_root=self.api_root, 230 config_api_root=self.config_api_root, 231 client_id=self.client_id, 232 client_secret=self.client_secret, 233 bearer_token=self.bearer_token, 234 ) 235 236 @overload 237 def get_organization(self) -> CloudOrganization: ... 238 239 @overload 240 def get_organization( 241 self, 242 *, 243 raise_on_error: Literal[True], 244 ) -> CloudOrganization: ... 245 246 @overload 247 def get_organization( 248 self, 249 *, 250 raise_on_error: Literal[False], 251 ) -> CloudOrganization | None: ... 252 253 def get_organization( 254 self, 255 *, 256 raise_on_error: bool = True, 257 ) -> CloudOrganization | None: 258 """Get the organization this workspace belongs to. 259 260 Fetching organization info requires ORGANIZATION_READER permissions on the organization, 261 which may not be available with workspace-scoped credentials. 262 263 Args: 264 raise_on_error: If True (default), raises AirbyteError on permission or API errors. 265 If False, returns None instead of raising. 266 267 Returns: 268 CloudOrganization object with organization_id and organization_name, 269 or None if raise_on_error=False and an error occurred. 270 271 Raises: 272 AirbyteError: If raise_on_error=True and the organization info cannot be fetched 273 (e.g., due to insufficient permissions or missing data). 274 """ 275 try: 276 info = self._organization_info 277 except (AirbyteError, NotImplementedError): 278 if raise_on_error: 279 raise 280 return None 281 282 organization_id = info.get("organizationId") 283 organization_name = info.get("organizationName") 284 285 # Validate that both organization_id and organization_name are non-null and non-empty 286 if not organization_id or not organization_name: 287 if raise_on_error: 288 raise AirbyteError( 289 message="Organization info is incomplete.", 290 context={ 291 "organization_id": organization_id, 292 "organization_name": organization_name, 293 }, 294 ) 295 return None 296 297 organization_credentials = self._credentials.with_organization_id(organization_id) 298 return CloudOrganization( 299 organization_id=organization_id, 300 organization_name=organization_name, 301 client_id=organization_credentials.client_id, 302 client_secret=organization_credentials.client_secret, 303 bearer_token=organization_credentials.bearer_token, 304 public_api_root=organization_credentials.public_api_root, 305 config_api_root=organization_credentials.config_api_root, 306 ) 307 308 # Test connection and creds 309 310 def connect(self) -> None: 311 """Check that the workspace is reachable and raise an exception otherwise. 312 313 Note: It is not necessary to call this method before calling other operations. It 314 serves primarily as a simple check to ensure that the workspace is reachable 315 and credentials are correct. 316 """ 317 _ = api_util.get_workspace( 318 api_root=self.api_root, 319 workspace_id=self.workspace_id, 320 client_id=self.client_id, 321 client_secret=self.client_secret, 322 bearer_token=self.bearer_token, 323 ) 324 print(f"Successfully connected to workspace: {self.workspace_url}") 325 326 # Get sources, destinations, and connections 327 328 def get_connection( 329 self, 330 connection_id: str, 331 ) -> CloudConnection: 332 """Get a connection by ID. 333 334 This method does not fetch data from the API. It returns a `CloudConnection` object, 335 which will be loaded lazily as needed. 336 """ 337 return CloudConnection( 338 workspace=self, 339 connection_id=connection_id, 340 ) 341 342 def get_source( 343 self, 344 source_id: str, 345 ) -> CloudSource: 346 """Get a source by ID. 347 348 This method does not fetch data from the API. It returns a `CloudSource` object, 349 which will be loaded lazily as needed. 350 """ 351 return CloudSource( 352 workspace=self, 353 connector_id=source_id, 354 ) 355 356 def get_destination( 357 self, 358 destination_id: str, 359 ) -> CloudDestination: 360 """Get a destination by ID. 361 362 This method does not fetch data from the API. It returns a `CloudDestination` object, 363 which will be loaded lazily as needed. 364 """ 365 return CloudDestination( 366 workspace=self, 367 connector_id=destination_id, 368 ) 369 370 # Deploy sources and destinations 371 372 def deploy_source( 373 self, 374 name: str, 375 source: Source, 376 *, 377 unique: bool = True, 378 random_name_suffix: bool = False, 379 ) -> CloudSource: 380 """Deploy a source to the workspace. 381 382 Returns the newly deployed source. 383 384 Args: 385 name: The name to use when deploying. 386 source: The source object to deploy. 387 unique: Whether to require a unique name. If `True`, duplicate names 388 are not allowed. Defaults to `True`. 389 random_name_suffix: Whether to append a random suffix to the name. 390 """ 391 source_config_dict = source._hydrated_config.copy() # noqa: SLF001 (non-public API) 392 source_config_dict["sourceType"] = source.name.replace("source-", "") 393 394 if random_name_suffix: 395 name += f" (ID: {text_util.generate_random_suffix()})" 396 397 if unique: 398 existing = self.list_sources(name=name) 399 if existing: 400 raise exc.AirbyteDuplicateResourcesError( 401 resource_type="source", 402 resource_name=name, 403 ) 404 405 deployed_source = api_util.create_source( 406 name=name, 407 api_root=self.api_root, 408 workspace_id=self.workspace_id, 409 config=source_config_dict, 410 client_id=self.client_id, 411 client_secret=self.client_secret, 412 bearer_token=self.bearer_token, 413 ) 414 return CloudSource( 415 workspace=self, 416 connector_id=deployed_source.source_id, 417 ) 418 419 def deploy_destination( 420 self, 421 name: str, 422 destination: Destination | dict[str, Any], 423 *, 424 unique: bool = True, 425 random_name_suffix: bool = False, 426 ) -> CloudDestination: 427 """Deploy a destination to the workspace. 428 429 Returns the newly deployed destination ID. 430 431 Args: 432 name: The name to use when deploying. 433 destination: The destination to deploy. Can be a local Airbyte `Destination` object or a 434 dictionary of configuration values. 435 unique: Whether to require a unique name. If `True`, duplicate names 436 are not allowed. Defaults to `True`. 437 random_name_suffix: Whether to append a random suffix to the name. 438 """ 439 if isinstance(destination, Destination): 440 destination_conf_dict = destination._hydrated_config.copy() # noqa: SLF001 (non-public API) 441 destination_conf_dict["destinationType"] = destination.name.replace("destination-", "") 442 # raise ValueError(destination_conf_dict) 443 else: 444 destination_conf_dict = destination.copy() 445 if "destinationType" not in destination_conf_dict: 446 raise exc.PyAirbyteInputError( 447 message="Missing `destinationType` in configuration dictionary.", 448 ) 449 450 if random_name_suffix: 451 name += f" (ID: {text_util.generate_random_suffix()})" 452 453 if unique: 454 existing = self.list_destinations(name=name) 455 if existing: 456 raise exc.AirbyteDuplicateResourcesError( 457 resource_type="destination", 458 resource_name=name, 459 ) 460 461 deployed_destination = api_util.create_destination( 462 name=name, 463 api_root=self.api_root, 464 workspace_id=self.workspace_id, 465 config=destination_conf_dict, # Wants a dataclass but accepts dict 466 client_id=self.client_id, 467 client_secret=self.client_secret, 468 bearer_token=self.bearer_token, 469 ) 470 return CloudDestination( 471 workspace=self, 472 connector_id=deployed_destination.destination_id, 473 ) 474 475 def permanently_delete_source( 476 self, 477 source: str | CloudSource, 478 *, 479 safe_mode: bool = True, 480 ) -> None: 481 """Delete a source from the workspace. 482 483 You can pass either the source ID `str` or a deployed `Source` object. 484 485 Args: 486 source: The source ID or CloudSource object to delete 487 safe_mode: If True, requires the source name to contain "delete-me" or "deleteme" 488 (case insensitive) to prevent accidental deletion. Defaults to True. 489 """ 490 if not isinstance(source, (str, CloudSource)): 491 raise exc.PyAirbyteInputError( 492 message="Invalid source type.", 493 input_value=type(source).__name__, 494 ) 495 496 api_util.delete_source( 497 source_id=source.connector_id if isinstance(source, CloudSource) else source, 498 source_name=source.name if isinstance(source, CloudSource) else None, 499 api_root=self.api_root, 500 client_id=self.client_id, 501 client_secret=self.client_secret, 502 bearer_token=self.bearer_token, 503 safe_mode=safe_mode, 504 ) 505 506 # Deploy and delete destinations 507 508 def permanently_delete_destination( 509 self, 510 destination: str | CloudDestination, 511 *, 512 safe_mode: bool = True, 513 ) -> None: 514 """Delete a deployed destination from the workspace. 515 516 You can pass either the `Cache` class or the deployed destination ID as a `str`. 517 518 Args: 519 destination: The destination ID or CloudDestination object to delete 520 safe_mode: If True, requires the destination name to contain "delete-me" or "deleteme" 521 (case insensitive) to prevent accidental deletion. Defaults to True. 522 """ 523 if not isinstance(destination, (str, CloudDestination)): 524 raise exc.PyAirbyteInputError( 525 message="Invalid destination type.", 526 input_value=type(destination).__name__, 527 ) 528 529 api_util.delete_destination( 530 destination_id=( 531 destination if isinstance(destination, str) else destination.destination_id 532 ), 533 destination_name=( 534 destination.name if isinstance(destination, CloudDestination) else None 535 ), 536 api_root=self.api_root, 537 client_id=self.client_id, 538 client_secret=self.client_secret, 539 bearer_token=self.bearer_token, 540 safe_mode=safe_mode, 541 ) 542 543 # Deploy and delete connections 544 545 def deploy_connection( 546 self, 547 connection_name: str, 548 *, 549 source: CloudSource | str, 550 selected_streams: list[str], 551 destination: CloudDestination | str, 552 table_prefix: str | None = None, 553 ) -> CloudConnection: 554 """Create a new connection between an already deployed source and destination. 555 556 Returns the newly deployed connection object. 557 558 Args: 559 connection_name: The name of the connection. 560 source: The deployed source. You can pass a source ID or a CloudSource object. 561 destination: The deployed destination. You can pass a destination ID or a 562 CloudDestination object. 563 table_prefix: Optional. The table prefix to use when syncing to the destination. 564 selected_streams: The selected stream names to sync within the connection. 565 """ 566 if not selected_streams: 567 raise exc.PyAirbyteInputError( 568 guidance="You must provide `selected_streams` when creating a connection." 569 ) 570 571 source_id: str = source if isinstance(source, str) else source.connector_id 572 destination_id: str = ( 573 destination if isinstance(destination, str) else destination.connector_id 574 ) 575 576 deployed_connection = api_util.create_connection( 577 name=connection_name, 578 source_id=source_id, 579 destination_id=destination_id, 580 api_root=self.api_root, 581 workspace_id=self.workspace_id, 582 selected_stream_names=selected_streams, 583 prefix=table_prefix or "", 584 client_id=self.client_id, 585 client_secret=self.client_secret, 586 bearer_token=self.bearer_token, 587 ) 588 589 return CloudConnection( 590 workspace=self, 591 connection_id=deployed_connection.connection_id, 592 source=deployed_connection.source_id, 593 destination=deployed_connection.destination_id, 594 ) 595 596 def permanently_delete_connection( 597 self, 598 connection: str | CloudConnection, 599 *, 600 cascade_delete_source: bool = False, 601 cascade_delete_destination: bool = False, 602 safe_mode: bool = True, 603 ) -> None: 604 """Delete a deployed connection from the workspace. 605 606 Args: 607 connection: The connection ID or CloudConnection object to delete 608 cascade_delete_source: If True, also delete the source after deleting the connection 609 cascade_delete_destination: If True, also delete the destination after deleting 610 the connection 611 safe_mode: If True, requires the connection name to contain "delete-me" or "deleteme" 612 (case insensitive) to prevent accidental deletion. Defaults to True. Also applies 613 to cascade deletes. 614 """ 615 if connection is None: 616 raise ValueError("No connection ID provided.") 617 618 if isinstance(connection, str): 619 connection = CloudConnection( 620 workspace=self, 621 connection_id=connection, 622 ) 623 624 api_util.delete_connection( 625 connection_id=connection.connection_id, 626 connection_name=connection.name, 627 api_root=self.api_root, 628 workspace_id=self.workspace_id, 629 client_id=self.client_id, 630 client_secret=self.client_secret, 631 bearer_token=self.bearer_token, 632 safe_mode=safe_mode, 633 ) 634 635 if cascade_delete_source: 636 self.permanently_delete_source( 637 source=connection.source_id, 638 safe_mode=safe_mode, 639 ) 640 if cascade_delete_destination: 641 self.permanently_delete_destination( 642 destination=connection.destination_id, 643 safe_mode=safe_mode, 644 ) 645 646 # List workspaces, sources, destinations, and connections 647 648 def list_workspaces( 649 self, 650 name: str | None = None, 651 *, 652 name_filter: Callable | None = None, 653 limit: int | None = None, 654 ) -> list[CloudWorkspaceInfo]: 655 """List workspaces available to the current credentials, with an optional limit.""" 656 return [ 657 CloudWorkspaceInfo.from_api_response(workspace) 658 for workspace in api_util.list_workspaces( 659 workspace_id="", 660 api_root=self.api_root, 661 name=name, 662 name_filter=name_filter, 663 client_id=self.client_id, 664 client_secret=self.client_secret, 665 bearer_token=self.bearer_token, 666 limit=limit, 667 ) 668 ] 669 670 def rename( 671 self, 672 name: str, 673 ) -> CloudWorkspace: 674 """Rename this workspace.""" 675 api_util.rename_workspace( 676 workspace_id=self.workspace_id, 677 name=name, 678 api_root=self.api_root, 679 client_id=self.client_id, 680 client_secret=self.client_secret, 681 bearer_token=self.bearer_token, 682 ) 683 return self 684 685 def permanently_delete( 686 self, 687 *, 688 workspace_name: str | None = None, 689 safe_mode: bool = True, 690 ) -> None: 691 """Permanently delete this workspace if it has no connections. 692 693 When `safe_mode` is enabled, the workspace name must contain `delete-me` 694 or `deleteme`. This also checks for existing connections before deleting 695 and raises `AirbyteWorkspaceNotEmptyError` if the workspace is not empty. 696 """ 697 api_util.permanently_delete_workspace( 698 workspace_id=self.workspace_id, 699 workspace_name=workspace_name, 700 api_root=self.api_root, 701 client_id=self.client_id, 702 client_secret=self.client_secret, 703 bearer_token=self.bearer_token, 704 safe_mode=safe_mode, 705 ) 706 707 def list_connections( 708 self, 709 name: str | None = None, 710 *, 711 name_filter: Callable | None = None, 712 limit: int | None = None, 713 ) -> list[CloudConnection]: 714 """List connections by name in the workspace, with an optional limit.""" 715 connections = api_util.list_connections( 716 api_root=self.api_root, 717 workspace_id=self.workspace_id, 718 name=name, 719 name_filter=name_filter, 720 limit=limit, 721 client_id=self.client_id, 722 client_secret=self.client_secret, 723 bearer_token=self.bearer_token, 724 ) 725 return [ 726 CloudConnection._from_connection_response( # noqa: SLF001 (non-public API) 727 workspace=self, 728 connection_response=connection, 729 ) 730 for connection in connections 731 ] 732 733 def list_sources( 734 self, 735 name: str | None = None, 736 *, 737 name_filter: Callable | None = None, 738 limit: int | None = None, 739 ) -> list[CloudSource]: 740 """List all sources in the workspace, with an optional limit.""" 741 sources = api_util.list_sources( 742 api_root=self.api_root, 743 workspace_id=self.workspace_id, 744 name=name, 745 name_filter=name_filter, 746 limit=limit, 747 client_id=self.client_id, 748 client_secret=self.client_secret, 749 bearer_token=self.bearer_token, 750 ) 751 return [ 752 CloudSource._from_source_response( # noqa: SLF001 (non-public API) 753 workspace=self, 754 source_response=source, 755 ) 756 for source in sources 757 ] 758 759 def list_destinations( 760 self, 761 name: str | None = None, 762 *, 763 name_filter: Callable | None = None, 764 limit: int | None = None, 765 ) -> list[CloudDestination]: 766 """List all destinations in the workspace, with an optional limit.""" 767 destinations = api_util.list_destinations( 768 api_root=self.api_root, 769 workspace_id=self.workspace_id, 770 name=name, 771 name_filter=name_filter, 772 limit=limit, 773 client_id=self.client_id, 774 client_secret=self.client_secret, 775 bearer_token=self.bearer_token, 776 ) 777 return [ 778 CloudDestination._from_destination_response( # noqa: SLF001 (non-public API) 779 workspace=self, 780 destination_response=destination, 781 ) 782 for destination in destinations 783 ] 784 785 def publish_custom_source_definition( 786 self, 787 name: str, 788 *, 789 manifest_yaml: dict[str, Any] | Path | str | None = None, 790 docker_image: str | None = None, 791 docker_tag: str | None = None, 792 unique: bool = True, 793 pre_validate: bool = True, 794 testing_values: dict[str, Any] | None = None, 795 ) -> CustomCloudSourceDefinition: 796 """Publish a custom source connector definition. 797 798 You must specify EITHER manifest_yaml (for YAML connectors) OR both docker_image 799 and docker_tag (for Docker connectors), but not both. 800 801 Args: 802 name: Display name for the connector definition 803 manifest_yaml: Low-code CDK manifest (dict, Path to YAML file, or YAML string) 804 docker_image: Docker repository (e.g., 'airbyte/source-custom') 805 docker_tag: Docker image tag (e.g., '1.0.0') 806 unique: Whether to enforce name uniqueness 807 pre_validate: Whether to validate manifest client-side (YAML only) 808 testing_values: Optional configuration values to use for testing in the 809 Connector Builder UI. If provided, these values are stored as the complete 810 testing values object for the connector builder project (replaces any existing 811 values), allowing immediate test read operations. 812 813 Returns: 814 CustomCloudSourceDefinition object representing the created definition 815 816 Raises: 817 PyAirbyteInputError: If both or neither of manifest_yaml and docker_image provided 818 AirbyteDuplicateResourcesError: If unique=True and name already exists 819 """ 820 is_yaml = manifest_yaml is not None 821 is_docker = docker_image is not None 822 823 if is_yaml == is_docker: 824 raise exc.PyAirbyteInputError( 825 message=( 826 "Must specify EITHER manifest_yaml (for YAML connectors) OR " 827 "docker_image + docker_tag (for Docker connectors), but not both" 828 ), 829 context={ 830 "manifest_yaml_provided": is_yaml, 831 "docker_image_provided": is_docker, 832 }, 833 ) 834 835 if is_docker and docker_tag is None: 836 raise exc.PyAirbyteInputError( 837 message="docker_tag is required when docker_image is specified", 838 context={"docker_image": docker_image}, 839 ) 840 841 if unique: 842 existing = self.list_custom_source_definitions( 843 definition_type="yaml" if is_yaml else "docker", 844 ) 845 if any(d.name == name for d in existing): 846 raise exc.AirbyteDuplicateResourcesError( 847 resource_type="custom_source_definition", 848 resource_name=name, 849 ) 850 851 if is_yaml: 852 manifest_dict: dict[str, Any] 853 if isinstance(manifest_yaml, Path): 854 manifest_dict = yaml.safe_load(manifest_yaml.read_text()) 855 elif isinstance(manifest_yaml, str): 856 manifest_dict = yaml.safe_load(manifest_yaml) 857 elif manifest_yaml is not None: 858 manifest_dict = manifest_yaml 859 else: 860 raise exc.PyAirbyteInputError( 861 message="manifest_yaml is required for YAML connectors", 862 context={"name": name}, 863 ) 864 865 if pre_validate: 866 api_util.validate_yaml_manifest(manifest_dict, raise_on_error=True) 867 868 result = api_util.create_custom_yaml_source_definition( 869 name=name, 870 workspace_id=self.workspace_id, 871 manifest=manifest_dict, 872 api_root=self.api_root, 873 client_id=self.client_id, 874 client_secret=self.client_secret, 875 bearer_token=self.bearer_token, 876 ) 877 custom_definition = CustomCloudSourceDefinition._from_yaml_response( # noqa: SLF001 878 self, result 879 ) 880 881 # Set testing values if provided 882 if testing_values is not None: 883 custom_definition.set_testing_values(testing_values) 884 885 return custom_definition 886 887 raise NotImplementedError( 888 "Docker custom source definitions are not yet supported. " 889 "Only YAML manifest-based custom sources are currently available." 890 ) 891 892 def list_custom_source_definitions( 893 self, 894 *, 895 definition_type: Literal["yaml", "docker"], 896 ) -> list[CustomCloudSourceDefinition]: 897 """List custom source connector definitions. 898 899 Args: 900 definition_type: Connector type to list ("yaml" or "docker"). Required. 901 902 Returns: 903 List of CustomCloudSourceDefinition objects matching the specified type 904 """ 905 if definition_type == "yaml": 906 yaml_definitions = api_util.list_custom_yaml_source_definitions( 907 workspace_id=self.workspace_id, 908 api_root=self.api_root, 909 client_id=self.client_id, 910 client_secret=self.client_secret, 911 bearer_token=self.bearer_token, 912 ) 913 return [ 914 CustomCloudSourceDefinition._from_yaml_response(self, d) # noqa: SLF001 915 for d in yaml_definitions 916 ] 917 918 raise NotImplementedError( 919 "Docker custom source definitions are not yet supported. " 920 "Only YAML manifest-based custom sources are currently available." 921 ) 922 923 def get_custom_source_definition( 924 self, 925 definition_id: str, 926 *, 927 definition_type: Literal["yaml", "docker"], 928 ) -> CustomCloudSourceDefinition: 929 """Get a specific custom source definition by ID. 930 931 Args: 932 definition_id: The definition ID 933 definition_type: Connector type ("yaml" or "docker"). Required. 934 935 Returns: 936 CustomCloudSourceDefinition object 937 """ 938 if definition_type == "yaml": 939 result = api_util.get_custom_yaml_source_definition( 940 workspace_id=self.workspace_id, 941 definition_id=definition_id, 942 api_root=self.api_root, 943 client_id=self.client_id, 944 client_secret=self.client_secret, 945 bearer_token=self.bearer_token, 946 ) 947 return CustomCloudSourceDefinition._from_yaml_response(self, result) # noqa: SLF001 948 949 raise NotImplementedError( 950 "Docker custom source definitions are not yet supported. " 951 "Only YAML manifest-based custom sources are currently available." 952 )
A remote workspace on the Airbyte Cloud.
By overriding api_root, you can use this class to interact with self-managed Airbyte
instances, both OSS and Enterprise.
Two authentication methods are supported (mutually exclusive):
- OAuth2 client credentials (client_id + client_secret)
- Bearer token authentication
Example with client credentials:
workspace = CloudWorkspace( workspace_id="...", client_id="...", client_secret="...", )
Example with bearer token:
workspace = CloudWorkspace( workspace_id="...", bearer_token="...", )
111 def __init__( 112 self, 113 *, 114 workspace_id: str | None = None, 115 client_id: str | SecretString | None = None, 116 client_secret: str | SecretString | None = None, 117 api_root: str | None = None, 118 config_api_root: str | None = None, 119 bearer_token: str | SecretString | None = None, 120 ) -> None: 121 """Validate and initialize credentials.""" 122 env_vars = not (client_id or client_secret or bearer_token) 123 credentials = _AirbyteCredentials.from_auth( 124 workspace_id=workspace_id, 125 client_id=client_id, 126 client_secret=client_secret, 127 bearer_token=bearer_token, 128 public_api_root=api_root, 129 config_api_root=config_api_root, 130 env_vars=env_vars, 131 ) 132 if not credentials.workspace_id: 133 raise exc.PyAirbyteInputError( 134 message="Workspace ID is required.", 135 guidance=( 136 "Provide a workspace ID, or call `get_default_cloud_context` to discover " 137 "available workspaces." 138 ), 139 ) 140 141 self._credentials = credentials 142 self.workspace_id = credentials.workspace_id or "" 143 self.client_id = credentials.client_id 144 self.client_secret = credentials.client_secret 145 self.bearer_token = credentials.bearer_token 146 self.api_root = credentials.public_api_root 147 self.config_api_root = credentials.config_api_root 148 149 # Create internal CloudClientConfig object (validates mutual exclusivity) 150 self._client_config = CloudClientConfig( 151 client_id=self.client_id, 152 client_secret=self.client_secret, 153 bearer_token=self.bearer_token, 154 api_root=self.api_root, 155 config_api_root=self.config_api_root, 156 )
Validate and initialize credentials.
158 @classmethod 159 def from_env( 160 cls, 161 workspace_id: str | None = None, 162 *, 163 api_root: str | None = None, 164 config_api_root: str | None = None, 165 ) -> CloudWorkspace: 166 """Create a CloudWorkspace using credentials from environment variables. 167 168 This factory method resolves credentials from environment variables, 169 providing a convenient way to create a workspace without explicitly 170 passing credentials. 171 172 Two authentication methods are supported (mutually exclusive): 173 1. Bearer token (checked first) 174 2. OAuth2 client credentials (fallback) 175 176 Environment variables used: 177 - `AIRBYTE_CLOUD_BEARER_TOKEN`: Bearer token (alternative to client credentials). 178 - `AIRBYTE_CLOUD_CLIENT_ID`: OAuth client ID (for client credentials flow). 179 - `AIRBYTE_CLOUD_CLIENT_SECRET`: OAuth client secret (for client credentials flow). 180 - `AIRBYTE_CLOUD_WORKSPACE_ID`: The workspace ID (if not passed as argument). 181 - `AIRBYTE_CLOUD_API_URL`: Optional. The API root URL (defaults to Airbyte Cloud). 182 - `AIRBYTE_CLOUD_CONFIG_API_URL`: Optional. The Config API root URL. 183 184 Args: 185 workspace_id: The workspace ID. If not provided, will be resolved from 186 the `AIRBYTE_CLOUD_WORKSPACE_ID` environment variable. 187 api_root: The API root URL. If not provided, will be resolved from 188 the `AIRBYTE_CLOUD_API_URL` environment variable, or default to 189 the Airbyte Cloud API. 190 config_api_root: The Config API root URL. If not provided, will be resolved 191 from the `AIRBYTE_CLOUD_CONFIG_API_URL` environment variable. 192 193 Returns: 194 A CloudWorkspace instance configured with credentials from the environment. 195 196 Raises: 197 PyAirbyteInputError: If required credentials are not found in 198 the environment or are incomplete. 199 200 Example: 201 ```python 202 # With workspace_id from environment 203 workspace = CloudWorkspace.from_env() 204 205 # With explicit workspace_id 206 workspace = CloudWorkspace.from_env(workspace_id="your-workspace-id") 207 ``` 208 """ 209 return cls( 210 workspace_id=workspace_id, 211 api_root=api_root, 212 config_api_root=config_api_root, 213 )
Create a CloudWorkspace using credentials from environment variables.
This factory method resolves credentials from environment variables, providing a convenient way to create a workspace without explicitly passing credentials.
Two authentication methods are supported (mutually exclusive):
- Bearer token (checked first)
- OAuth2 client credentials (fallback)
Environment variables used:
AIRBYTE_CLOUD_BEARER_TOKEN: Bearer token (alternative to client credentials).AIRBYTE_CLOUD_CLIENT_ID: OAuth client ID (for client credentials flow).AIRBYTE_CLOUD_CLIENT_SECRET: OAuth client secret (for client credentials flow).AIRBYTE_CLOUD_WORKSPACE_ID: The workspace ID (if not passed as argument).AIRBYTE_CLOUD_API_URL: Optional. The API root URL (defaults to Airbyte Cloud).AIRBYTE_CLOUD_CONFIG_API_URL: Optional. The Config API root URL.
Arguments:
- workspace_id: The workspace ID. If not provided, will be resolved from
the
AIRBYTE_CLOUD_WORKSPACE_IDenvironment variable. - api_root: The API root URL. If not provided, will be resolved from
the
AIRBYTE_CLOUD_API_URLenvironment variable, or default to the Airbyte Cloud API. - config_api_root: The Config API root URL. If not provided, will be resolved
from the
AIRBYTE_CLOUD_CONFIG_API_URLenvironment variable.
Returns:
A CloudWorkspace instance configured with credentials from the environment.
Raises:
- PyAirbyteInputError: If required credentials are not found in the environment or are incomplete.
Example:
# With workspace_id from environment workspace = CloudWorkspace.from_env() # With explicit workspace_id workspace = CloudWorkspace.from_env(workspace_id="your-workspace-id")
215 @property 216 def workspace_url(self) -> str | None: 217 """The web URL of the workspace.""" 218 return f"{get_web_url_root(self.api_root)}/workspaces/{self.workspace_id}"
The web URL of the workspace.
253 def get_organization( 254 self, 255 *, 256 raise_on_error: bool = True, 257 ) -> CloudOrganization | None: 258 """Get the organization this workspace belongs to. 259 260 Fetching organization info requires ORGANIZATION_READER permissions on the organization, 261 which may not be available with workspace-scoped credentials. 262 263 Args: 264 raise_on_error: If True (default), raises AirbyteError on permission or API errors. 265 If False, returns None instead of raising. 266 267 Returns: 268 CloudOrganization object with organization_id and organization_name, 269 or None if raise_on_error=False and an error occurred. 270 271 Raises: 272 AirbyteError: If raise_on_error=True and the organization info cannot be fetched 273 (e.g., due to insufficient permissions or missing data). 274 """ 275 try: 276 info = self._organization_info 277 except (AirbyteError, NotImplementedError): 278 if raise_on_error: 279 raise 280 return None 281 282 organization_id = info.get("organizationId") 283 organization_name = info.get("organizationName") 284 285 # Validate that both organization_id and organization_name are non-null and non-empty 286 if not organization_id or not organization_name: 287 if raise_on_error: 288 raise AirbyteError( 289 message="Organization info is incomplete.", 290 context={ 291 "organization_id": organization_id, 292 "organization_name": organization_name, 293 }, 294 ) 295 return None 296 297 organization_credentials = self._credentials.with_organization_id(organization_id) 298 return CloudOrganization( 299 organization_id=organization_id, 300 organization_name=organization_name, 301 client_id=organization_credentials.client_id, 302 client_secret=organization_credentials.client_secret, 303 bearer_token=organization_credentials.bearer_token, 304 public_api_root=organization_credentials.public_api_root, 305 config_api_root=organization_credentials.config_api_root, 306 )
Get the organization this workspace belongs to.
Fetching organization info requires ORGANIZATION_READER permissions on the organization, which may not be available with workspace-scoped credentials.
Arguments:
- raise_on_error: If True (default), raises AirbyteError on permission or API errors. If False, returns None instead of raising.
Returns:
CloudOrganization object with organization_id and organization_name, or None if raise_on_error=False and an error occurred.
Raises:
- AirbyteError: If raise_on_error=True and the organization info cannot be fetched (e.g., due to insufficient permissions or missing data).
310 def connect(self) -> None: 311 """Check that the workspace is reachable and raise an exception otherwise. 312 313 Note: It is not necessary to call this method before calling other operations. It 314 serves primarily as a simple check to ensure that the workspace is reachable 315 and credentials are correct. 316 """ 317 _ = api_util.get_workspace( 318 api_root=self.api_root, 319 workspace_id=self.workspace_id, 320 client_id=self.client_id, 321 client_secret=self.client_secret, 322 bearer_token=self.bearer_token, 323 ) 324 print(f"Successfully connected to workspace: {self.workspace_url}")
Check that the workspace is reachable and raise an exception otherwise.
Note: It is not necessary to call this method before calling other operations. It serves primarily as a simple check to ensure that the workspace is reachable and credentials are correct.
328 def get_connection( 329 self, 330 connection_id: str, 331 ) -> CloudConnection: 332 """Get a connection by ID. 333 334 This method does not fetch data from the API. It returns a `CloudConnection` object, 335 which will be loaded lazily as needed. 336 """ 337 return CloudConnection( 338 workspace=self, 339 connection_id=connection_id, 340 )
Get a connection by ID.
This method does not fetch data from the API. It returns a CloudConnection object,
which will be loaded lazily as needed.
342 def get_source( 343 self, 344 source_id: str, 345 ) -> CloudSource: 346 """Get a source by ID. 347 348 This method does not fetch data from the API. It returns a `CloudSource` object, 349 which will be loaded lazily as needed. 350 """ 351 return CloudSource( 352 workspace=self, 353 connector_id=source_id, 354 )
Get a source by ID.
This method does not fetch data from the API. It returns a CloudSource object,
which will be loaded lazily as needed.
356 def get_destination( 357 self, 358 destination_id: str, 359 ) -> CloudDestination: 360 """Get a destination by ID. 361 362 This method does not fetch data from the API. It returns a `CloudDestination` object, 363 which will be loaded lazily as needed. 364 """ 365 return CloudDestination( 366 workspace=self, 367 connector_id=destination_id, 368 )
Get a destination by ID.
This method does not fetch data from the API. It returns a CloudDestination object,
which will be loaded lazily as needed.
372 def deploy_source( 373 self, 374 name: str, 375 source: Source, 376 *, 377 unique: bool = True, 378 random_name_suffix: bool = False, 379 ) -> CloudSource: 380 """Deploy a source to the workspace. 381 382 Returns the newly deployed source. 383 384 Args: 385 name: The name to use when deploying. 386 source: The source object to deploy. 387 unique: Whether to require a unique name. If `True`, duplicate names 388 are not allowed. Defaults to `True`. 389 random_name_suffix: Whether to append a random suffix to the name. 390 """ 391 source_config_dict = source._hydrated_config.copy() # noqa: SLF001 (non-public API) 392 source_config_dict["sourceType"] = source.name.replace("source-", "") 393 394 if random_name_suffix: 395 name += f" (ID: {text_util.generate_random_suffix()})" 396 397 if unique: 398 existing = self.list_sources(name=name) 399 if existing: 400 raise exc.AirbyteDuplicateResourcesError( 401 resource_type="source", 402 resource_name=name, 403 ) 404 405 deployed_source = api_util.create_source( 406 name=name, 407 api_root=self.api_root, 408 workspace_id=self.workspace_id, 409 config=source_config_dict, 410 client_id=self.client_id, 411 client_secret=self.client_secret, 412 bearer_token=self.bearer_token, 413 ) 414 return CloudSource( 415 workspace=self, 416 connector_id=deployed_source.source_id, 417 )
Deploy a source to the workspace.
Returns the newly deployed source.
Arguments:
- name: The name to use when deploying.
- source: The source object to deploy.
- unique: Whether to require a unique name. If
True, duplicate names are not allowed. Defaults toTrue. - random_name_suffix: Whether to append a random suffix to the name.
419 def deploy_destination( 420 self, 421 name: str, 422 destination: Destination | dict[str, Any], 423 *, 424 unique: bool = True, 425 random_name_suffix: bool = False, 426 ) -> CloudDestination: 427 """Deploy a destination to the workspace. 428 429 Returns the newly deployed destination ID. 430 431 Args: 432 name: The name to use when deploying. 433 destination: The destination to deploy. Can be a local Airbyte `Destination` object or a 434 dictionary of configuration values. 435 unique: Whether to require a unique name. If `True`, duplicate names 436 are not allowed. Defaults to `True`. 437 random_name_suffix: Whether to append a random suffix to the name. 438 """ 439 if isinstance(destination, Destination): 440 destination_conf_dict = destination._hydrated_config.copy() # noqa: SLF001 (non-public API) 441 destination_conf_dict["destinationType"] = destination.name.replace("destination-", "") 442 # raise ValueError(destination_conf_dict) 443 else: 444 destination_conf_dict = destination.copy() 445 if "destinationType" not in destination_conf_dict: 446 raise exc.PyAirbyteInputError( 447 message="Missing `destinationType` in configuration dictionary.", 448 ) 449 450 if random_name_suffix: 451 name += f" (ID: {text_util.generate_random_suffix()})" 452 453 if unique: 454 existing = self.list_destinations(name=name) 455 if existing: 456 raise exc.AirbyteDuplicateResourcesError( 457 resource_type="destination", 458 resource_name=name, 459 ) 460 461 deployed_destination = api_util.create_destination( 462 name=name, 463 api_root=self.api_root, 464 workspace_id=self.workspace_id, 465 config=destination_conf_dict, # Wants a dataclass but accepts dict 466 client_id=self.client_id, 467 client_secret=self.client_secret, 468 bearer_token=self.bearer_token, 469 ) 470 return CloudDestination( 471 workspace=self, 472 connector_id=deployed_destination.destination_id, 473 )
Deploy a destination to the workspace.
Returns the newly deployed destination ID.
Arguments:
- name: The name to use when deploying.
- destination: The destination to deploy. Can be a local Airbyte
Destinationobject or a dictionary of configuration values. - unique: Whether to require a unique name. If
True, duplicate names are not allowed. Defaults toTrue. - random_name_suffix: Whether to append a random suffix to the name.
475 def permanently_delete_source( 476 self, 477 source: str | CloudSource, 478 *, 479 safe_mode: bool = True, 480 ) -> None: 481 """Delete a source from the workspace. 482 483 You can pass either the source ID `str` or a deployed `Source` object. 484 485 Args: 486 source: The source ID or CloudSource object to delete 487 safe_mode: If True, requires the source name to contain "delete-me" or "deleteme" 488 (case insensitive) to prevent accidental deletion. Defaults to True. 489 """ 490 if not isinstance(source, (str, CloudSource)): 491 raise exc.PyAirbyteInputError( 492 message="Invalid source type.", 493 input_value=type(source).__name__, 494 ) 495 496 api_util.delete_source( 497 source_id=source.connector_id if isinstance(source, CloudSource) else source, 498 source_name=source.name if isinstance(source, CloudSource) else None, 499 api_root=self.api_root, 500 client_id=self.client_id, 501 client_secret=self.client_secret, 502 bearer_token=self.bearer_token, 503 safe_mode=safe_mode, 504 )
Delete a source from the workspace.
You can pass either the source ID str or a deployed Source object.
Arguments:
- source: The source ID or CloudSource object to delete
- safe_mode: If True, requires the source name to contain "delete-me" or "deleteme" (case insensitive) to prevent accidental deletion. Defaults to True.
508 def permanently_delete_destination( 509 self, 510 destination: str | CloudDestination, 511 *, 512 safe_mode: bool = True, 513 ) -> None: 514 """Delete a deployed destination from the workspace. 515 516 You can pass either the `Cache` class or the deployed destination ID as a `str`. 517 518 Args: 519 destination: The destination ID or CloudDestination object to delete 520 safe_mode: If True, requires the destination name to contain "delete-me" or "deleteme" 521 (case insensitive) to prevent accidental deletion. Defaults to True. 522 """ 523 if not isinstance(destination, (str, CloudDestination)): 524 raise exc.PyAirbyteInputError( 525 message="Invalid destination type.", 526 input_value=type(destination).__name__, 527 ) 528 529 api_util.delete_destination( 530 destination_id=( 531 destination if isinstance(destination, str) else destination.destination_id 532 ), 533 destination_name=( 534 destination.name if isinstance(destination, CloudDestination) else None 535 ), 536 api_root=self.api_root, 537 client_id=self.client_id, 538 client_secret=self.client_secret, 539 bearer_token=self.bearer_token, 540 safe_mode=safe_mode, 541 )
Delete a deployed destination from the workspace.
You can pass either the Cache class or the deployed destination ID as a str.
Arguments:
- destination: The destination ID or CloudDestination object to delete
- safe_mode: If True, requires the destination name to contain "delete-me" or "deleteme" (case insensitive) to prevent accidental deletion. Defaults to True.
545 def deploy_connection( 546 self, 547 connection_name: str, 548 *, 549 source: CloudSource | str, 550 selected_streams: list[str], 551 destination: CloudDestination | str, 552 table_prefix: str | None = None, 553 ) -> CloudConnection: 554 """Create a new connection between an already deployed source and destination. 555 556 Returns the newly deployed connection object. 557 558 Args: 559 connection_name: The name of the connection. 560 source: The deployed source. You can pass a source ID or a CloudSource object. 561 destination: The deployed destination. You can pass a destination ID or a 562 CloudDestination object. 563 table_prefix: Optional. The table prefix to use when syncing to the destination. 564 selected_streams: The selected stream names to sync within the connection. 565 """ 566 if not selected_streams: 567 raise exc.PyAirbyteInputError( 568 guidance="You must provide `selected_streams` when creating a connection." 569 ) 570 571 source_id: str = source if isinstance(source, str) else source.connector_id 572 destination_id: str = ( 573 destination if isinstance(destination, str) else destination.connector_id 574 ) 575 576 deployed_connection = api_util.create_connection( 577 name=connection_name, 578 source_id=source_id, 579 destination_id=destination_id, 580 api_root=self.api_root, 581 workspace_id=self.workspace_id, 582 selected_stream_names=selected_streams, 583 prefix=table_prefix or "", 584 client_id=self.client_id, 585 client_secret=self.client_secret, 586 bearer_token=self.bearer_token, 587 ) 588 589 return CloudConnection( 590 workspace=self, 591 connection_id=deployed_connection.connection_id, 592 source=deployed_connection.source_id, 593 destination=deployed_connection.destination_id, 594 )
Create a new connection between an already deployed source and destination.
Returns the newly deployed connection object.
Arguments:
- connection_name: The name of the connection.
- source: The deployed source. You can pass a source ID or a CloudSource object.
- destination: The deployed destination. You can pass a destination ID or a CloudDestination object.
- table_prefix: Optional. The table prefix to use when syncing to the destination.
- selected_streams: The selected stream names to sync within the connection.
596 def permanently_delete_connection( 597 self, 598 connection: str | CloudConnection, 599 *, 600 cascade_delete_source: bool = False, 601 cascade_delete_destination: bool = False, 602 safe_mode: bool = True, 603 ) -> None: 604 """Delete a deployed connection from the workspace. 605 606 Args: 607 connection: The connection ID or CloudConnection object to delete 608 cascade_delete_source: If True, also delete the source after deleting the connection 609 cascade_delete_destination: If True, also delete the destination after deleting 610 the connection 611 safe_mode: If True, requires the connection name to contain "delete-me" or "deleteme" 612 (case insensitive) to prevent accidental deletion. Defaults to True. Also applies 613 to cascade deletes. 614 """ 615 if connection is None: 616 raise ValueError("No connection ID provided.") 617 618 if isinstance(connection, str): 619 connection = CloudConnection( 620 workspace=self, 621 connection_id=connection, 622 ) 623 624 api_util.delete_connection( 625 connection_id=connection.connection_id, 626 connection_name=connection.name, 627 api_root=self.api_root, 628 workspace_id=self.workspace_id, 629 client_id=self.client_id, 630 client_secret=self.client_secret, 631 bearer_token=self.bearer_token, 632 safe_mode=safe_mode, 633 ) 634 635 if cascade_delete_source: 636 self.permanently_delete_source( 637 source=connection.source_id, 638 safe_mode=safe_mode, 639 ) 640 if cascade_delete_destination: 641 self.permanently_delete_destination( 642 destination=connection.destination_id, 643 safe_mode=safe_mode, 644 )
Delete a deployed connection from the workspace.
Arguments:
- connection: The connection ID or CloudConnection object to delete
- cascade_delete_source: If True, also delete the source after deleting the connection
- cascade_delete_destination: If True, also delete the destination after deleting the connection
- safe_mode: If True, requires the connection name to contain "delete-me" or "deleteme" (case insensitive) to prevent accidental deletion. Defaults to True. Also applies to cascade deletes.
648 def list_workspaces( 649 self, 650 name: str | None = None, 651 *, 652 name_filter: Callable | None = None, 653 limit: int | None = None, 654 ) -> list[CloudWorkspaceInfo]: 655 """List workspaces available to the current credentials, with an optional limit.""" 656 return [ 657 CloudWorkspaceInfo.from_api_response(workspace) 658 for workspace in api_util.list_workspaces( 659 workspace_id="", 660 api_root=self.api_root, 661 name=name, 662 name_filter=name_filter, 663 client_id=self.client_id, 664 client_secret=self.client_secret, 665 bearer_token=self.bearer_token, 666 limit=limit, 667 ) 668 ]
List workspaces available to the current credentials, with an optional limit.
670 def rename( 671 self, 672 name: str, 673 ) -> CloudWorkspace: 674 """Rename this workspace.""" 675 api_util.rename_workspace( 676 workspace_id=self.workspace_id, 677 name=name, 678 api_root=self.api_root, 679 client_id=self.client_id, 680 client_secret=self.client_secret, 681 bearer_token=self.bearer_token, 682 ) 683 return self
Rename this workspace.
685 def permanently_delete( 686 self, 687 *, 688 workspace_name: str | None = None, 689 safe_mode: bool = True, 690 ) -> None: 691 """Permanently delete this workspace if it has no connections. 692 693 When `safe_mode` is enabled, the workspace name must contain `delete-me` 694 or `deleteme`. This also checks for existing connections before deleting 695 and raises `AirbyteWorkspaceNotEmptyError` if the workspace is not empty. 696 """ 697 api_util.permanently_delete_workspace( 698 workspace_id=self.workspace_id, 699 workspace_name=workspace_name, 700 api_root=self.api_root, 701 client_id=self.client_id, 702 client_secret=self.client_secret, 703 bearer_token=self.bearer_token, 704 safe_mode=safe_mode, 705 )
Permanently delete this workspace if it has no connections.
When safe_mode is enabled, the workspace name must contain delete-me
or deleteme. This also checks for existing connections before deleting
and raises AirbyteWorkspaceNotEmptyError if the workspace is not empty.
707 def list_connections( 708 self, 709 name: str | None = None, 710 *, 711 name_filter: Callable | None = None, 712 limit: int | None = None, 713 ) -> list[CloudConnection]: 714 """List connections by name in the workspace, with an optional limit.""" 715 connections = api_util.list_connections( 716 api_root=self.api_root, 717 workspace_id=self.workspace_id, 718 name=name, 719 name_filter=name_filter, 720 limit=limit, 721 client_id=self.client_id, 722 client_secret=self.client_secret, 723 bearer_token=self.bearer_token, 724 ) 725 return [ 726 CloudConnection._from_connection_response( # noqa: SLF001 (non-public API) 727 workspace=self, 728 connection_response=connection, 729 ) 730 for connection in connections 731 ]
List connections by name in the workspace, with an optional limit.
733 def list_sources( 734 self, 735 name: str | None = None, 736 *, 737 name_filter: Callable | None = None, 738 limit: int | None = None, 739 ) -> list[CloudSource]: 740 """List all sources in the workspace, with an optional limit.""" 741 sources = api_util.list_sources( 742 api_root=self.api_root, 743 workspace_id=self.workspace_id, 744 name=name, 745 name_filter=name_filter, 746 limit=limit, 747 client_id=self.client_id, 748 client_secret=self.client_secret, 749 bearer_token=self.bearer_token, 750 ) 751 return [ 752 CloudSource._from_source_response( # noqa: SLF001 (non-public API) 753 workspace=self, 754 source_response=source, 755 ) 756 for source in sources 757 ]
List all sources in the workspace, with an optional limit.
759 def list_destinations( 760 self, 761 name: str | None = None, 762 *, 763 name_filter: Callable | None = None, 764 limit: int | None = None, 765 ) -> list[CloudDestination]: 766 """List all destinations in the workspace, with an optional limit.""" 767 destinations = api_util.list_destinations( 768 api_root=self.api_root, 769 workspace_id=self.workspace_id, 770 name=name, 771 name_filter=name_filter, 772 limit=limit, 773 client_id=self.client_id, 774 client_secret=self.client_secret, 775 bearer_token=self.bearer_token, 776 ) 777 return [ 778 CloudDestination._from_destination_response( # noqa: SLF001 (non-public API) 779 workspace=self, 780 destination_response=destination, 781 ) 782 for destination in destinations 783 ]
List all destinations in the workspace, with an optional limit.
785 def publish_custom_source_definition( 786 self, 787 name: str, 788 *, 789 manifest_yaml: dict[str, Any] | Path | str | None = None, 790 docker_image: str | None = None, 791 docker_tag: str | None = None, 792 unique: bool = True, 793 pre_validate: bool = True, 794 testing_values: dict[str, Any] | None = None, 795 ) -> CustomCloudSourceDefinition: 796 """Publish a custom source connector definition. 797 798 You must specify EITHER manifest_yaml (for YAML connectors) OR both docker_image 799 and docker_tag (for Docker connectors), but not both. 800 801 Args: 802 name: Display name for the connector definition 803 manifest_yaml: Low-code CDK manifest (dict, Path to YAML file, or YAML string) 804 docker_image: Docker repository (e.g., 'airbyte/source-custom') 805 docker_tag: Docker image tag (e.g., '1.0.0') 806 unique: Whether to enforce name uniqueness 807 pre_validate: Whether to validate manifest client-side (YAML only) 808 testing_values: Optional configuration values to use for testing in the 809 Connector Builder UI. If provided, these values are stored as the complete 810 testing values object for the connector builder project (replaces any existing 811 values), allowing immediate test read operations. 812 813 Returns: 814 CustomCloudSourceDefinition object representing the created definition 815 816 Raises: 817 PyAirbyteInputError: If both or neither of manifest_yaml and docker_image provided 818 AirbyteDuplicateResourcesError: If unique=True and name already exists 819 """ 820 is_yaml = manifest_yaml is not None 821 is_docker = docker_image is not None 822 823 if is_yaml == is_docker: 824 raise exc.PyAirbyteInputError( 825 message=( 826 "Must specify EITHER manifest_yaml (for YAML connectors) OR " 827 "docker_image + docker_tag (for Docker connectors), but not both" 828 ), 829 context={ 830 "manifest_yaml_provided": is_yaml, 831 "docker_image_provided": is_docker, 832 }, 833 ) 834 835 if is_docker and docker_tag is None: 836 raise exc.PyAirbyteInputError( 837 message="docker_tag is required when docker_image is specified", 838 context={"docker_image": docker_image}, 839 ) 840 841 if unique: 842 existing = self.list_custom_source_definitions( 843 definition_type="yaml" if is_yaml else "docker", 844 ) 845 if any(d.name == name for d in existing): 846 raise exc.AirbyteDuplicateResourcesError( 847 resource_type="custom_source_definition", 848 resource_name=name, 849 ) 850 851 if is_yaml: 852 manifest_dict: dict[str, Any] 853 if isinstance(manifest_yaml, Path): 854 manifest_dict = yaml.safe_load(manifest_yaml.read_text()) 855 elif isinstance(manifest_yaml, str): 856 manifest_dict = yaml.safe_load(manifest_yaml) 857 elif manifest_yaml is not None: 858 manifest_dict = manifest_yaml 859 else: 860 raise exc.PyAirbyteInputError( 861 message="manifest_yaml is required for YAML connectors", 862 context={"name": name}, 863 ) 864 865 if pre_validate: 866 api_util.validate_yaml_manifest(manifest_dict, raise_on_error=True) 867 868 result = api_util.create_custom_yaml_source_definition( 869 name=name, 870 workspace_id=self.workspace_id, 871 manifest=manifest_dict, 872 api_root=self.api_root, 873 client_id=self.client_id, 874 client_secret=self.client_secret, 875 bearer_token=self.bearer_token, 876 ) 877 custom_definition = CustomCloudSourceDefinition._from_yaml_response( # noqa: SLF001 878 self, result 879 ) 880 881 # Set testing values if provided 882 if testing_values is not None: 883 custom_definition.set_testing_values(testing_values) 884 885 return custom_definition 886 887 raise NotImplementedError( 888 "Docker custom source definitions are not yet supported. " 889 "Only YAML manifest-based custom sources are currently available." 890 )
Publish a custom source connector definition.
You must specify EITHER manifest_yaml (for YAML connectors) OR both docker_image and docker_tag (for Docker connectors), but not both.
Arguments:
- name: Display name for the connector definition
- manifest_yaml: Low-code CDK manifest (dict, Path to YAML file, or YAML string)
- docker_image: Docker repository (e.g., 'airbyte/source-custom')
- docker_tag: Docker image tag (e.g., '1.0.0')
- unique: Whether to enforce name uniqueness
- pre_validate: Whether to validate manifest client-side (YAML only)
- testing_values: Optional configuration values to use for testing in the Connector Builder UI. If provided, these values are stored as the complete testing values object for the connector builder project (replaces any existing values), allowing immediate test read operations.
Returns:
CustomCloudSourceDefinition object representing the created definition
Raises:
- PyAirbyteInputError: If both or neither of manifest_yaml and docker_image provided
- AirbyteDuplicateResourcesError: If unique=True and name already exists
892 def list_custom_source_definitions( 893 self, 894 *, 895 definition_type: Literal["yaml", "docker"], 896 ) -> list[CustomCloudSourceDefinition]: 897 """List custom source connector definitions. 898 899 Args: 900 definition_type: Connector type to list ("yaml" or "docker"). Required. 901 902 Returns: 903 List of CustomCloudSourceDefinition objects matching the specified type 904 """ 905 if definition_type == "yaml": 906 yaml_definitions = api_util.list_custom_yaml_source_definitions( 907 workspace_id=self.workspace_id, 908 api_root=self.api_root, 909 client_id=self.client_id, 910 client_secret=self.client_secret, 911 bearer_token=self.bearer_token, 912 ) 913 return [ 914 CustomCloudSourceDefinition._from_yaml_response(self, d) # noqa: SLF001 915 for d in yaml_definitions 916 ] 917 918 raise NotImplementedError( 919 "Docker custom source definitions are not yet supported. " 920 "Only YAML manifest-based custom sources are currently available." 921 )
List custom source connector definitions.
Arguments:
- definition_type: Connector type to list ("yaml" or "docker"). Required.
Returns:
List of CustomCloudSourceDefinition objects matching the specified type
923 def get_custom_source_definition( 924 self, 925 definition_id: str, 926 *, 927 definition_type: Literal["yaml", "docker"], 928 ) -> CustomCloudSourceDefinition: 929 """Get a specific custom source definition by ID. 930 931 Args: 932 definition_id: The definition ID 933 definition_type: Connector type ("yaml" or "docker"). Required. 934 935 Returns: 936 CustomCloudSourceDefinition object 937 """ 938 if definition_type == "yaml": 939 result = api_util.get_custom_yaml_source_definition( 940 workspace_id=self.workspace_id, 941 definition_id=definition_id, 942 api_root=self.api_root, 943 client_id=self.client_id, 944 client_secret=self.client_secret, 945 bearer_token=self.bearer_token, 946 ) 947 return CustomCloudSourceDefinition._from_yaml_response(self, result) # noqa: SLF001 948 949 raise NotImplementedError( 950 "Docker custom source definitions are not yet supported. " 951 "Only YAML manifest-based custom sources are currently available." 952 )
Get a specific custom source definition by ID.
Arguments:
- definition_id: The definition ID
- definition_type: Connector type ("yaml" or "docker"). Required.
Returns:
CustomCloudSourceDefinition object
45class CloudConnection: # noqa: PLR0904 # Too many public methods 46 """A connection is an extract-load (EL) pairing of a source and destination in Airbyte Cloud. 47 48 You can use a connection object to run sync jobs, retrieve logs, and manage the connection. 49 """ 50 51 def __init__( 52 self, 53 workspace: CloudWorkspace, 54 connection_id: str, 55 source: str | None = None, 56 destination: str | None = None, 57 ) -> None: 58 """It is not recommended to create a `CloudConnection` object directly. 59 60 Instead, use `CloudWorkspace.get_connection()` to create a connection object. 61 """ 62 self.connection_id = connection_id 63 """The ID of the connection.""" 64 65 self.workspace = workspace 66 """The workspace that the connection belongs to.""" 67 68 self._source_id = source 69 """The ID of the source.""" 70 71 self._destination_id = destination 72 """The ID of the destination.""" 73 74 self._connection_info: CloudConnectionInfo | None = None 75 """The connection info object. (Cached.)""" 76 77 self._cloud_source_object: CloudSource | None = None 78 """The source object. (Cached.)""" 79 80 self._cloud_destination_object: CloudDestination | None = None 81 """The destination object. (Cached.)""" 82 83 def _fetch_connection_info( 84 self, 85 *, 86 force_refresh: bool = False, 87 verify: bool = True, 88 ) -> CloudConnectionInfo: 89 """Fetch and cache connection info from the API. 90 91 By default, this method will only fetch from the API if connection info is not 92 already cached. It also verifies that the connection belongs to the expected 93 workspace unless verification is explicitly disabled. 94 95 Args: 96 force_refresh: If True, always fetch from the API even if cached. 97 If False (default), only fetch if not already cached. 98 verify: If True (default), verify that the connection is valid (e.g., that 99 the workspace_id matches this object's workspace). Raises an error if 100 validation fails. 101 102 Returns: 103 Information about the connection from the API. 104 105 Raises: 106 AirbyteWorkspaceMismatchError: If verify is True and the connection's 107 workspace_id doesn't match the expected workspace. 108 AirbyteMissingResourceError: If the connection doesn't exist. 109 """ 110 if not force_refresh and self._connection_info is not None: 111 # Use cached info, but still verify if requested 112 if verify: 113 self._verify_workspace_match(self._connection_info) 114 return self._connection_info 115 116 # Fetch from API 117 connection_info = api_util.get_connection( 118 workspace_id=self.workspace.workspace_id, 119 connection_id=self.connection_id, 120 api_root=self.workspace.api_root, 121 client_id=self.workspace.client_id, 122 client_secret=self.workspace.client_secret, 123 bearer_token=self.workspace.bearer_token, 124 ) 125 result = CloudConnectionInfo.from_api_response(connection_info) 126 127 self._connection_info = result 128 129 # Verify if requested 130 if verify: 131 self._verify_workspace_match(result) 132 133 return result 134 135 def _verify_workspace_match(self, connection_info: CloudConnectionInfo) -> None: 136 """Verify that the connection belongs to the expected workspace. 137 138 Raises: 139 AirbyteWorkspaceMismatchError: If the workspace IDs don't match. 140 """ 141 if connection_info.workspace_id != self.workspace.workspace_id: 142 raise AirbyteWorkspaceMismatchError( 143 resource_type="connection", 144 resource_id=self.connection_id, 145 workspace=self.workspace, 146 expected_workspace_id=self.workspace.workspace_id, 147 actual_workspace_id=connection_info.workspace_id, 148 message=( 149 f"Connection '{self.connection_id}' belongs to workspace " 150 f"'{connection_info.workspace_id}', not '{self.workspace.workspace_id}'." 151 ), 152 ) 153 154 def check_is_valid(self) -> bool: 155 """Check if this connection exists and belongs to the expected workspace. 156 157 This method fetches connection info from the API (if not already cached) and 158 verifies that the connection's workspace_id matches the workspace associated 159 with this CloudConnection object. 160 161 Returns: 162 True if the connection exists and belongs to the expected workspace. 163 164 Raises: 165 AirbyteWorkspaceMismatchError: If the connection belongs to a different workspace. 166 AirbyteMissingResourceError: If the connection doesn't exist. 167 """ 168 self._fetch_connection_info(force_refresh=False, verify=True) 169 return True 170 171 @classmethod 172 def _from_connection_response( 173 cls, 174 workspace: CloudWorkspace, 175 connection_response: _ConnectionResponseLike, 176 ) -> CloudConnection: 177 """Create a CloudConnection from an API connection response.""" 178 connection_info = CloudConnectionInfo.from_api_response(connection_response) 179 result = cls( 180 workspace=workspace, 181 connection_id=connection_info.connection_id, 182 source=connection_info.source_id, 183 destination=connection_info.destination_id, 184 ) 185 result._connection_info = connection_info # noqa: SLF001 # Accessing Non-Public API 186 return result 187 188 # Properties 189 190 @property 191 def name(self) -> str | None: 192 """Get the display name of the connection, if available. 193 194 E.g. "My Postgres to Snowflake", not the connection ID. 195 """ 196 if not self._connection_info: 197 self._connection_info = self._fetch_connection_info() 198 199 return self._connection_info.name 200 201 @property 202 def source_id(self) -> str: 203 """The ID of the source.""" 204 if not self._source_id: 205 if not self._connection_info: 206 self._connection_info = self._fetch_connection_info() 207 208 self._source_id = self._connection_info.source_id 209 210 return self._source_id 211 212 @property 213 def source(self) -> CloudSource: 214 """Get the source object.""" 215 if self._cloud_source_object: 216 return self._cloud_source_object 217 218 self._cloud_source_object = CloudSource( 219 workspace=self.workspace, 220 connector_id=self.source_id, 221 ) 222 return self._cloud_source_object 223 224 @property 225 def destination_id(self) -> str: 226 """The ID of the destination.""" 227 if not self._destination_id: 228 if not self._connection_info: 229 self._connection_info = self._fetch_connection_info() 230 231 self._destination_id = self._connection_info.destination_id 232 233 return self._destination_id 234 235 @property 236 def destination(self) -> CloudDestination: 237 """Get the destination object.""" 238 if self._cloud_destination_object: 239 return self._cloud_destination_object 240 241 self._cloud_destination_object = CloudDestination( 242 workspace=self.workspace, 243 connector_id=self.destination_id, 244 ) 245 return self._cloud_destination_object 246 247 @property 248 def stream_names(self) -> list[str]: 249 """The stream names.""" 250 if not self._connection_info: 251 self._connection_info = self._fetch_connection_info() 252 253 return [stream.name for stream in self._connection_info.configurations.streams or []] 254 255 @property 256 def table_prefix(self) -> str: 257 """The table prefix.""" 258 if not self._connection_info: 259 self._connection_info = self._fetch_connection_info() 260 261 return self._connection_info.prefix or "" 262 263 @property 264 def connection_url(self) -> str | None: 265 """The web URL to the connection.""" 266 return f"{self.workspace.workspace_url}/connections/{self.connection_id}" 267 268 @property 269 def job_history_url(self) -> str | None: 270 """The URL to the job history for the connection.""" 271 return f"{self.connection_url}/timeline" 272 273 # Run Sync 274 275 def run_sync( 276 self, 277 *, 278 wait: bool = True, 279 wait_timeout: int = 300, 280 ) -> SyncResult: 281 """Run a sync.""" 282 connection_response = api_util.run_connection( 283 connection_id=self.connection_id, 284 api_root=self.workspace.api_root, 285 workspace_id=self.workspace.workspace_id, 286 client_id=self.workspace.client_id, 287 client_secret=self.workspace.client_secret, 288 bearer_token=self.workspace.bearer_token, 289 ) 290 sync_result = SyncResult( 291 workspace=self.workspace, 292 connection=self, 293 job_id=connection_response.job_id, 294 ) 295 296 if wait: 297 sync_result.wait_for_completion( 298 wait_timeout=wait_timeout, 299 raise_failure=True, 300 raise_timeout=True, 301 ) 302 303 return sync_result 304 305 def _get_latest_cancellable_sync_job_id(self) -> int: 306 """Get the latest cancellable sync job ID.""" 307 sync_results = self.get_previous_sync_logs( 308 limit=1, 309 job_type=JobTypeEnum.SYNC, 310 ) 311 sync_result = sync_results[0] if sync_results else None 312 if sync_result is None: 313 raise PyAirbyteInputError( 314 message="No sync jobs found for this connection.", 315 ) 316 if sync_result.is_job_complete(): 317 raise PyAirbyteInputError( 318 message=( 319 f"The latest sync job is already finished with status " 320 f"'{sync_result.get_job_status().value}'. " 321 "Pass an explicit job_id to target a different job." 322 ), 323 ) 324 return sync_result.job_id 325 326 def _validated_cancellable_job_id(self, job_id: int) -> int: 327 """Validate an explicit cancellable job ID.""" 328 job_info = api_util.get_job_info( 329 job_id=job_id, 330 api_root=self.workspace.api_root, 331 client_id=self.workspace.client_id, 332 client_secret=self.workspace.client_secret, 333 bearer_token=self.workspace.bearer_token, 334 ) 335 if job_info.connection_id != self.connection_id: 336 raise PyAirbyteInputError( 337 message=( 338 f"Job {job_id} belongs to connection '{job_info.connection_id}', " 339 f"not '{self.connection_id}'." 340 ), 341 ) 342 job_status = CloudJobInfo.from_api_response(job_info).status 343 if job_status in FINAL_STATUSES: 344 raise PyAirbyteInputError( 345 message=f"Job {job_id} is already finished with status " f"'{job_status.value}'.", 346 ) 347 return job_id 348 349 def cancel_sync(self, job_id: int | None = None) -> SyncResult: 350 """Cancel a running sync job. 351 352 Defaults to the connection's most recent sync job. Other job types must be 353 targeted with an explicit `job_id`. 354 """ 355 target_job_id: int = ( 356 self._get_latest_cancellable_sync_job_id() 357 if job_id is None 358 else self._validated_cancellable_job_id(job_id) 359 ) 360 361 job_response = api_util.cancel_job( 362 job_id=target_job_id, 363 api_root=self.workspace.api_root, 364 client_id=self.workspace.client_id, 365 client_secret=self.workspace.client_secret, 366 bearer_token=self.workspace.bearer_token, 367 ) 368 return SyncResult( 369 workspace=self.workspace, 370 connection=self, 371 job_id=job_response.job_id, 372 _latest_job_info=CloudJobInfo.from_api_response(job_response), 373 ) 374 375 def __repr__(self) -> str: 376 """String representation of the connection.""" 377 return ( 378 f"CloudConnection(connection_id={self.connection_id}, source_id={self.source_id}, " 379 f"destination_id={self.destination_id}, connection_url={self.connection_url})" 380 ) 381 382 # Logs 383 384 def get_previous_sync_logs( 385 self, 386 *, 387 limit: int = 20, 388 offset: int | None = None, 389 from_tail: bool = True, 390 job_type: str | JobTypeEnum | None = None, 391 ) -> list[SyncResult]: 392 """Get previous sync jobs for a connection with pagination support. 393 394 Returns SyncResult objects containing job metadata (job_id, status, bytes_synced, 395 rows_synced, start_time). Full log text can be fetched lazily via 396 `SyncResult.get_full_log_text()`. 397 398 Args: 399 limit: Maximum number of jobs to return. Defaults to 20. 400 offset: Number of jobs to skip from the beginning. Defaults to None (0). 401 from_tail: If True, returns jobs ordered newest-first (createdAt DESC). 402 If False, returns jobs ordered oldest-first (createdAt ASC). 403 Defaults to True. 404 job_type: Filter by job type (e.g., `sync`, `refresh`). 405 If not specified, defaults to sync and reset jobs only (API default behavior). 406 407 Returns: 408 A list of SyncResult objects representing the sync jobs. 409 """ 410 order_by = ( 411 api_util.JOB_ORDER_BY_CREATED_AT_DESC 412 if from_tail 413 else api_util.JOB_ORDER_BY_CREATED_AT_ASC 414 ) 415 sync_logs = api_util.get_job_logs( 416 connection_id=self.connection_id, 417 api_root=self.workspace.api_root, 418 workspace_id=self.workspace.workspace_id, 419 limit=limit, 420 offset=offset, 421 order_by=order_by, 422 job_type=job_type, 423 client_id=self.workspace.client_id, 424 client_secret=self.workspace.client_secret, 425 bearer_token=self.workspace.bearer_token, 426 ) 427 return [ 428 SyncResult( 429 workspace=self.workspace, 430 connection=self, 431 job_id=sync_log.job_id, 432 _latest_job_info=CloudJobInfo.from_api_response(sync_log), 433 ) 434 for sync_log in sync_logs 435 ] 436 437 def get_sync_result( 438 self, 439 job_id: int | None = None, 440 ) -> SyncResult | None: 441 """Get the sync result for the connection. 442 443 If `job_id` is not provided, the most recent sync job will be used. 444 445 Returns `None` if job_id is omitted and no previous jobs are found. 446 """ 447 if job_id is None: 448 # Get the most recent sync job 449 results = self.get_previous_sync_logs( 450 limit=1, 451 ) 452 if results: 453 return results[0] 454 455 return None 456 457 # Get the sync job by ID (lazy loaded) 458 return SyncResult( 459 workspace=self.workspace, 460 connection=self, 461 job_id=job_id, 462 ) 463 464 # Artifacts 465 466 @deprecated("Use 'dump_raw_state()' instead.") 467 def get_state_artifacts(self) -> list[dict[str, Any]] | None: 468 """Deprecated. Use `dump_raw_state()` instead.""" 469 state_response = api_util.get_connection_state( 470 connection_id=self.connection_id, 471 api_root=self.workspace.api_root, 472 client_id=self.workspace.client_id, 473 client_secret=self.workspace.client_secret, 474 bearer_token=self.workspace.bearer_token, 475 config_api_root=self.workspace.config_api_root, 476 ) 477 if state_response.get("stateType") == "not_set": 478 return None 479 return state_response.get("streamState", []) 480 481 @overload 482 def dump_raw_state(self, *, normalize: Literal[True] = True) -> list[dict[str, Any]]: ... 483 484 @overload 485 def dump_raw_state(self, *, normalize: Literal[False]) -> dict[str, Any]: ... 486 487 def dump_raw_state( 488 self, 489 *, 490 normalize: bool = True, 491 ) -> dict[str, Any] | list[dict[str, Any]]: 492 """Dump the state for this connection. 493 494 By default, returns a list of Airbyte protocol `AirbyteStateMessage` dicts 495 with snake_case keys, suitable for passing to a connector's `--state` flag. 496 497 When `normalize` is `False`, returns the raw Config API dict (camelCase keys, 498 includes `stateType` and `connectionId`). This raw format can be passed 499 directly to `import_raw_state()` for backup/restore workflows. 500 501 Args: 502 normalize: If `True` (default), convert to Airbyte protocol format. 503 If `False`, return the raw Config API response. 504 505 Returns: 506 Normalized: list of protocol-format state message dicts (empty list if 507 no state). Raw: the full Config API state dict. 508 """ 509 raw = api_util.get_connection_state( 510 connection_id=self.connection_id, 511 api_root=self.workspace.api_root, 512 client_id=self.workspace.client_id, 513 client_secret=self.workspace.client_secret, 514 bearer_token=self.workspace.bearer_token, 515 config_api_root=self.workspace.config_api_root, 516 ) 517 if normalize: 518 return _normalize_state_to_protocol(raw) 519 return raw 520 521 def import_raw_state( 522 self, 523 connection_state: dict[str, Any] | list[dict[str, Any]], 524 ) -> dict[str, Any]: 525 """Import (restore) the full state for this connection. 526 527 > ⚠️ **WARNING:** Modifying the state directly is not recommended and 528 > could result in broken connections, and/or incorrect sync behavior. 529 530 Replaces the entire connection state with the provided state blob. 531 Uses the safe variant that prevents updates while a sync is running (HTTP 423). 532 533 This is the counterpart to `dump_raw_state()` for backup/restore workflows. 534 The `connectionId` in the blob is always overridden with this connection's 535 ID, making state blobs portable across connections. 536 537 Accepts either format: 538 539 - **Config API format** (dict with `stateType`): passed through directly. 540 - **Airbyte protocol format** (list of `AirbyteStateMessage` dicts): automatically 541 converted to Config API format before sending. 542 543 Args: 544 connection_state: Connection state in either Config API or Airbyte protocol format. 545 546 Returns: 547 The updated connection state as a dictionary. 548 549 Raises: 550 AirbyteConnectionSyncActiveError: If a sync is currently running on this 551 connection (HTTP 423). Wait for the sync to complete before retrying. 552 """ 553 api_state: dict[str, Any] 554 if isinstance(connection_state, list): 555 if not _is_protocol_state_format(connection_state): 556 msg = ( 557 "Expected connection_state list to contain Airbyte protocol state " 558 "message dicts (each with a top-level `type` of STREAM, GLOBAL, " 559 "or LEGACY). Got a list that does not match protocol format." 560 ) 561 raise ValueError(msg) 562 api_state = _denormalize_protocol_state_to_api( 563 protocol_messages=connection_state, 564 connection_id=self.connection_id, 565 ) 566 elif isinstance(connection_state, dict): 567 if _is_protocol_state_format(connection_state): 568 api_state = _denormalize_protocol_state_to_api( 569 protocol_messages=[connection_state], 570 connection_id=self.connection_id, 571 ) 572 else: 573 api_state = connection_state 574 else: 575 msg = f"Expected a dict or list, got {type(connection_state)}" 576 raise TypeError(msg) 577 578 return api_util.replace_connection_state( 579 connection_id=self.connection_id, 580 connection_state_dict=api_state, 581 api_root=self.workspace.api_root, 582 client_id=self.workspace.client_id, 583 client_secret=self.workspace.client_secret, 584 bearer_token=self.workspace.bearer_token, 585 config_api_root=self.workspace.config_api_root, 586 ) 587 588 def get_stream_state( 589 self, 590 stream_name: str, 591 stream_namespace: str | None = None, 592 ) -> dict[str, Any] | None: 593 """Get the state blob for a single stream within this connection. 594 595 Returns just the stream's state dictionary (e.g., {"cursor": "2024-01-01"}), 596 not the full connection state envelope. 597 598 This is compatible with `stream`-type state and stream-level entries 599 within a `global`-type state. It is not compatible with `legacy` state. 600 To get or set the entire connection-level state artifact, use 601 `dump_raw_state` and `import_raw_state` instead. 602 603 Args: 604 stream_name: The name of the stream to get state for. 605 stream_namespace: The source-side stream namespace. This refers to the 606 namespace from the source (e.g., database schema), not any destination 607 namespace override set in connection advanced settings. 608 609 Returns: 610 The stream's state blob as a dictionary, or None if the stream is not found. 611 """ 612 state_data = self.dump_raw_state(normalize=False) 613 result = ConnectionStateResponse(**state_data) 614 615 streams = _get_stream_list(result) 616 matching = [s for s in streams if _match_stream(s, stream_name, stream_namespace)] 617 618 if not matching: 619 available = [s.stream_descriptor.name for s in streams] 620 logger.warning( 621 "Stream '%s' not found in connection state for connection '%s'. " 622 "Available streams: %s", 623 stream_name, 624 self.connection_id, 625 available, 626 ) 627 return None 628 629 return matching[0].stream_state 630 631 def set_stream_state( 632 self, 633 stream_name: str, 634 state_blob_dict: dict[str, Any], 635 stream_namespace: str | None = None, 636 ) -> None: 637 """Set the state for a single stream within this connection. 638 639 Fetches the current full state, replaces only the specified stream's state, 640 then sends the full updated state back to the API. If the stream does not 641 exist in the current state, it is appended. 642 643 This is compatible with `stream`-type state and stream-level entries 644 within a `global`-type state. It is not compatible with `legacy` state. 645 To get or set the entire connection-level state artifact, use 646 `dump_raw_state` and `import_raw_state` instead. 647 648 Uses the safe variant that prevents updates while a sync is running (HTTP 423). 649 650 Args: 651 stream_name: The name of the stream to update state for. 652 state_blob_dict: The state blob dict for this stream (e.g., {"cursor": "2024-01-01"}). 653 stream_namespace: The source-side stream namespace. This refers to the 654 namespace from the source (e.g., database schema), not any destination 655 namespace override set in connection advanced settings. 656 657 Raises: 658 PyAirbyteInputError: If the connection state type is not supported for 659 stream-level operations (not_set, legacy). 660 AirbyteConnectionSyncActiveError: If a sync is currently running on this 661 connection (HTTP 423). Wait for the sync to complete before retrying. 662 """ 663 state_data = self.dump_raw_state(normalize=False) 664 current = ConnectionStateResponse(**state_data) 665 666 if current.state_type == "not_set": 667 raise PyAirbyteInputError( 668 message="Cannot set stream state: connection has no existing state.", 669 context={"connection_id": self.connection_id}, 670 ) 671 672 if current.state_type == "legacy": 673 raise PyAirbyteInputError( 674 message="Cannot set stream state on a legacy-type connection state.", 675 context={"connection_id": self.connection_id}, 676 ) 677 678 new_stream_entry = { 679 "streamDescriptor": { 680 "name": stream_name, 681 **( 682 { 683 "namespace": stream_namespace, 684 } 685 if stream_namespace 686 else {} 687 ), 688 }, 689 "streamState": state_blob_dict, 690 } 691 692 raw_streams: list[dict[str, Any]] 693 if current.state_type == "stream": 694 raw_streams = state_data.get("streamState", []) 695 elif current.state_type == "global": 696 raw_streams = state_data.get("globalState", {}).get("streamStates", []) 697 else: 698 raw_streams = [] 699 700 streams = _get_stream_list(current) 701 found = False 702 updated_streams_raw: list[dict[str, Any]] = [] 703 for raw_s, parsed_s in zip(raw_streams, streams, strict=False): 704 if _match_stream(parsed_s, stream_name, stream_namespace): 705 updated_streams_raw.append(new_stream_entry) 706 found = True 707 else: 708 updated_streams_raw.append(raw_s) 709 710 if not found: 711 updated_streams_raw.append(new_stream_entry) 712 713 full_state: dict[str, Any] = { 714 **state_data, 715 } 716 717 if current.state_type == "stream": 718 full_state["streamState"] = updated_streams_raw 719 elif current.state_type == "global": 720 original_global = state_data.get("globalState", {}) 721 full_state["globalState"] = { 722 **original_global, 723 "streamStates": updated_streams_raw, 724 } 725 726 self.import_raw_state(full_state) 727 728 @deprecated("Use 'dump_raw_catalog()' instead.") 729 def get_catalog_artifact(self) -> dict[str, Any] | None: 730 """Get the configured catalog for this connection. 731 732 Returns the full configured catalog (syncCatalog) for this connection, 733 including stream schemas, sync modes, cursor fields, and primary keys. 734 735 Uses the Config API endpoint: POST /v1/web_backend/connections/get 736 737 Returns: 738 Dictionary containing the configured catalog, or `None` if not found. 739 """ 740 return self.dump_raw_catalog() 741 742 def dump_raw_catalog( 743 self, 744 *, 745 normalize: bool = True, 746 ) -> dict[str, Any] | None: 747 """Dump the configured catalog for this connection. 748 749 By default, returns the catalog in Airbyte protocol format 750 (`ConfiguredAirbyteCatalog` with snake_case keys), suitable for passing 751 to a connector's `--catalog` flag. 752 753 When `normalize` is `False`, returns the raw `syncCatalog` dict from the 754 Config API (camelCase keys, nested `config` block). This raw format can be 755 passed directly to `import_raw_catalog()` for backup/restore workflows. 756 757 Args: 758 normalize: If `True` (default), convert to Airbyte protocol format. 759 If `False`, return the raw Config API catalog. 760 761 Returns: 762 The configured catalog dict, or `None` if not found. 763 """ 764 connection_response = api_util.get_connection_catalog( 765 connection_id=self.connection_id, 766 api_root=self.workspace.api_root, 767 client_id=self.workspace.client_id, 768 client_secret=self.workspace.client_secret, 769 bearer_token=self.workspace.bearer_token, 770 config_api_root=self.workspace.config_api_root, 771 ) 772 raw = connection_response.get("syncCatalog") 773 if raw is None: 774 return None 775 if normalize: 776 return _normalize_catalog_to_protocol(raw) 777 return raw 778 779 def import_raw_catalog(self, catalog: dict[str, Any]) -> None: 780 """Replace the configured catalog for this connection. 781 782 > ⚠️ **WARNING:** Modifying the catalog directly is not recommended and 783 > could result in broken connections, and/or incorrect sync behavior. 784 785 Accepts a configured catalog dict and replaces the connection's entire 786 catalog with it. All other connection settings remain unchanged. 787 788 Accepts either format: 789 790 - **Config API format** (`syncCatalog` with camelCase keys and nested `config`): 791 passed through directly. 792 - **Airbyte protocol format** (`ConfiguredAirbyteCatalog` with snake_case keys): 793 automatically converted to Config API format before sending. 794 795 Args: 796 catalog: The configured catalog dict in either format. 797 """ 798 if _is_protocol_catalog_format(catalog): 799 catalog = _denormalize_catalog_to_api(catalog) 800 801 api_util.replace_connection_catalog( 802 connection_id=self.connection_id, 803 configured_catalog_dict=catalog, 804 api_root=self.workspace.api_root, 805 client_id=self.workspace.client_id, 806 client_secret=self.workspace.client_secret, 807 bearer_token=self.workspace.bearer_token, 808 config_api_root=self.workspace.config_api_root, 809 ) 810 811 def rename(self, name: str) -> CloudConnection: 812 """Rename the connection. 813 814 Args: 815 name: New name for the connection 816 817 Returns: 818 Updated CloudConnection object with refreshed info 819 """ 820 updated_response = api_util.patch_connection( 821 connection_id=self.connection_id, 822 api_root=self.workspace.api_root, 823 client_id=self.workspace.client_id, 824 client_secret=self.workspace.client_secret, 825 bearer_token=self.workspace.bearer_token, 826 name=name, 827 ) 828 self._connection_info = CloudConnectionInfo.from_api_response(updated_response) 829 return self 830 831 def set_table_prefix(self, prefix: str) -> CloudConnection: 832 """Set the table prefix for the connection. 833 834 Args: 835 prefix: New table prefix to use when syncing to the destination 836 837 Returns: 838 Updated CloudConnection object with refreshed info 839 """ 840 updated_response = api_util.patch_connection( 841 connection_id=self.connection_id, 842 api_root=self.workspace.api_root, 843 client_id=self.workspace.client_id, 844 client_secret=self.workspace.client_secret, 845 bearer_token=self.workspace.bearer_token, 846 prefix=prefix, 847 ) 848 self._connection_info = CloudConnectionInfo.from_api_response(updated_response) 849 return self 850 851 def set_selected_streams(self, stream_names: list[str]) -> CloudConnection: 852 """Set the selected streams for the connection. 853 854 This is a destructive operation that can break existing connections if the 855 stream selection is changed incorrectly. Use with caution. 856 857 Args: 858 stream_names: List of stream names to sync 859 860 Returns: 861 Updated CloudConnection object with refreshed info 862 """ 863 configurations = api_util.build_stream_configurations(stream_names) 864 865 updated_response = api_util.patch_connection( 866 connection_id=self.connection_id, 867 api_root=self.workspace.api_root, 868 client_id=self.workspace.client_id, 869 client_secret=self.workspace.client_secret, 870 bearer_token=self.workspace.bearer_token, 871 configurations=configurations, 872 ) 873 self._connection_info = CloudConnectionInfo.from_api_response(updated_response) 874 return self 875 876 # Enable/Disable 877 878 @property 879 def enabled(self) -> bool: 880 """Get the current enabled status of the connection. 881 882 This property always fetches fresh data from the API to ensure accuracy, 883 as another process or user may have toggled the setting. 884 885 Returns: 886 True if the connection status is 'active', False otherwise. 887 """ 888 connection_info = self._fetch_connection_info(force_refresh=True) 889 return connection_info.status == "active" 890 891 @enabled.setter 892 def enabled(self, value: bool) -> None: 893 """Set the enabled status of the connection. 894 895 Args: 896 value: True to enable (set status to 'active'), False to disable 897 (set status to 'inactive'). 898 """ 899 self.set_enabled(enabled=value) 900 901 def set_enabled( 902 self, 903 *, 904 enabled: bool, 905 ignore_noop: bool = True, 906 ) -> None: 907 """Set the enabled status of the connection. 908 909 Args: 910 enabled: True to enable (set status to 'active'), False to disable 911 (set status to 'inactive'). 912 ignore_noop: If True (default), silently return if the connection is already 913 in the requested state. If False, raise ValueError when the requested 914 state matches the current state. 915 916 Raises: 917 ValueError: If ignore_noop is False and the connection is already in the 918 requested state. 919 """ 920 # Always fetch fresh data to check current status 921 connection_info = self._fetch_connection_info(force_refresh=True) 922 current_status = connection_info.status 923 desired_status = "active" if enabled else "inactive" 924 925 if current_status == desired_status: 926 if ignore_noop: 927 return 928 raise ValueError( 929 f"Connection is already {'enabled' if enabled else 'disabled'}. " 930 f"Current status: {current_status}" 931 ) 932 933 updated_response = api_util.patch_connection( 934 connection_id=self.connection_id, 935 api_root=self.workspace.api_root, 936 client_id=self.workspace.client_id, 937 client_secret=self.workspace.client_secret, 938 bearer_token=self.workspace.bearer_token, 939 status=desired_status, 940 ) 941 self._connection_info = CloudConnectionInfo.from_api_response(updated_response) 942 943 # Scheduling 944 945 def set_schedule( 946 self, 947 cron_expression: str, 948 ) -> None: 949 """Set a cron schedule for the connection. 950 951 Args: 952 cron_expression: A cron expression defining when syncs should run. 953 954 Examples: 955 - "0 0 * * *" # Daily at midnight UTC 956 - "0 */6 * * *" # Every 6 hours 957 - "0 0 * * 0" # Weekly on Sunday at midnight UTC 958 """ 959 updated_response = api_util.patch_connection( 960 connection_id=self.connection_id, 961 api_root=self.workspace.api_root, 962 client_id=self.workspace.client_id, 963 client_secret=self.workspace.client_secret, 964 bearer_token=self.workspace.bearer_token, 965 schedule=api_util.build_connection_schedule( 966 schedule_type="cron", 967 cron_expression=cron_expression, 968 ), 969 ) 970 self._connection_info = CloudConnectionInfo.from_api_response(updated_response) 971 972 def set_manual_schedule(self) -> None: 973 """Set the connection to manual scheduling. 974 975 Disables automatic syncs. Syncs will only run when manually triggered. 976 """ 977 updated_response = api_util.patch_connection( 978 connection_id=self.connection_id, 979 api_root=self.workspace.api_root, 980 client_id=self.workspace.client_id, 981 client_secret=self.workspace.client_secret, 982 bearer_token=self.workspace.bearer_token, 983 schedule=api_util.build_connection_schedule(schedule_type="manual"), 984 ) 985 self._connection_info = CloudConnectionInfo.from_api_response(updated_response) 986 987 # Deletions 988 989 def permanently_delete( 990 self, 991 *, 992 cascade_delete_source: bool = False, 993 cascade_delete_destination: bool = False, 994 ) -> None: 995 """Delete the connection. 996 997 Args: 998 cascade_delete_source: Whether to also delete the source. 999 cascade_delete_destination: Whether to also delete the destination. 1000 """ 1001 self.workspace.permanently_delete_connection(self) 1002 1003 if cascade_delete_source: 1004 self.workspace.permanently_delete_source(self.source_id) 1005 1006 if cascade_delete_destination: 1007 self.workspace.permanently_delete_destination(self.destination_id)
A connection is an extract-load (EL) pairing of a source and destination in Airbyte Cloud.
You can use a connection object to run sync jobs, retrieve logs, and manage the connection.
51 def __init__( 52 self, 53 workspace: CloudWorkspace, 54 connection_id: str, 55 source: str | None = None, 56 destination: str | None = None, 57 ) -> None: 58 """It is not recommended to create a `CloudConnection` object directly. 59 60 Instead, use `CloudWorkspace.get_connection()` to create a connection object. 61 """ 62 self.connection_id = connection_id 63 """The ID of the connection.""" 64 65 self.workspace = workspace 66 """The workspace that the connection belongs to.""" 67 68 self._source_id = source 69 """The ID of the source.""" 70 71 self._destination_id = destination 72 """The ID of the destination.""" 73 74 self._connection_info: CloudConnectionInfo | None = None 75 """The connection info object. (Cached.)""" 76 77 self._cloud_source_object: CloudSource | None = None 78 """The source object. (Cached.)""" 79 80 self._cloud_destination_object: CloudDestination | None = None 81 """The destination object. (Cached.)"""
It is not recommended to create a CloudConnection object directly.
Instead, use CloudWorkspace.get_connection() to create a connection object.
154 def check_is_valid(self) -> bool: 155 """Check if this connection exists and belongs to the expected workspace. 156 157 This method fetches connection info from the API (if not already cached) and 158 verifies that the connection's workspace_id matches the workspace associated 159 with this CloudConnection object. 160 161 Returns: 162 True if the connection exists and belongs to the expected workspace. 163 164 Raises: 165 AirbyteWorkspaceMismatchError: If the connection belongs to a different workspace. 166 AirbyteMissingResourceError: If the connection doesn't exist. 167 """ 168 self._fetch_connection_info(force_refresh=False, verify=True) 169 return True
Check if this connection exists and belongs to the expected workspace.
This method fetches connection info from the API (if not already cached) and verifies that the connection's workspace_id matches the workspace associated with this CloudConnection object.
Returns:
True if the connection exists and belongs to the expected workspace.
Raises:
- AirbyteWorkspaceMismatchError: If the connection belongs to a different workspace.
- AirbyteMissingResourceError: If the connection doesn't exist.
190 @property 191 def name(self) -> str | None: 192 """Get the display name of the connection, if available. 193 194 E.g. "My Postgres to Snowflake", not the connection ID. 195 """ 196 if not self._connection_info: 197 self._connection_info = self._fetch_connection_info() 198 199 return self._connection_info.name
Get the display name of the connection, if available.
E.g. "My Postgres to Snowflake", not the connection ID.
201 @property 202 def source_id(self) -> str: 203 """The ID of the source.""" 204 if not self._source_id: 205 if not self._connection_info: 206 self._connection_info = self._fetch_connection_info() 207 208 self._source_id = self._connection_info.source_id 209 210 return self._source_id
The ID of the source.
212 @property 213 def source(self) -> CloudSource: 214 """Get the source object.""" 215 if self._cloud_source_object: 216 return self._cloud_source_object 217 218 self._cloud_source_object = CloudSource( 219 workspace=self.workspace, 220 connector_id=self.source_id, 221 ) 222 return self._cloud_source_object
Get the source object.
224 @property 225 def destination_id(self) -> str: 226 """The ID of the destination.""" 227 if not self._destination_id: 228 if not self._connection_info: 229 self._connection_info = self._fetch_connection_info() 230 231 self._destination_id = self._connection_info.destination_id 232 233 return self._destination_id
The ID of the destination.
235 @property 236 def destination(self) -> CloudDestination: 237 """Get the destination object.""" 238 if self._cloud_destination_object: 239 return self._cloud_destination_object 240 241 self._cloud_destination_object = CloudDestination( 242 workspace=self.workspace, 243 connector_id=self.destination_id, 244 ) 245 return self._cloud_destination_object
Get the destination object.
247 @property 248 def stream_names(self) -> list[str]: 249 """The stream names.""" 250 if not self._connection_info: 251 self._connection_info = self._fetch_connection_info() 252 253 return [stream.name for stream in self._connection_info.configurations.streams or []]
The stream names.
255 @property 256 def table_prefix(self) -> str: 257 """The table prefix.""" 258 if not self._connection_info: 259 self._connection_info = self._fetch_connection_info() 260 261 return self._connection_info.prefix or ""
The table prefix.
263 @property 264 def connection_url(self) -> str | None: 265 """The web URL to the connection.""" 266 return f"{self.workspace.workspace_url}/connections/{self.connection_id}"
The web URL to the connection.
268 @property 269 def job_history_url(self) -> str | None: 270 """The URL to the job history for the connection.""" 271 return f"{self.connection_url}/timeline"
The URL to the job history for the connection.
275 def run_sync( 276 self, 277 *, 278 wait: bool = True, 279 wait_timeout: int = 300, 280 ) -> SyncResult: 281 """Run a sync.""" 282 connection_response = api_util.run_connection( 283 connection_id=self.connection_id, 284 api_root=self.workspace.api_root, 285 workspace_id=self.workspace.workspace_id, 286 client_id=self.workspace.client_id, 287 client_secret=self.workspace.client_secret, 288 bearer_token=self.workspace.bearer_token, 289 ) 290 sync_result = SyncResult( 291 workspace=self.workspace, 292 connection=self, 293 job_id=connection_response.job_id, 294 ) 295 296 if wait: 297 sync_result.wait_for_completion( 298 wait_timeout=wait_timeout, 299 raise_failure=True, 300 raise_timeout=True, 301 ) 302 303 return sync_result
Run a sync.
349 def cancel_sync(self, job_id: int | None = None) -> SyncResult: 350 """Cancel a running sync job. 351 352 Defaults to the connection's most recent sync job. Other job types must be 353 targeted with an explicit `job_id`. 354 """ 355 target_job_id: int = ( 356 self._get_latest_cancellable_sync_job_id() 357 if job_id is None 358 else self._validated_cancellable_job_id(job_id) 359 ) 360 361 job_response = api_util.cancel_job( 362 job_id=target_job_id, 363 api_root=self.workspace.api_root, 364 client_id=self.workspace.client_id, 365 client_secret=self.workspace.client_secret, 366 bearer_token=self.workspace.bearer_token, 367 ) 368 return SyncResult( 369 workspace=self.workspace, 370 connection=self, 371 job_id=job_response.job_id, 372 _latest_job_info=CloudJobInfo.from_api_response(job_response), 373 )
Cancel a running sync job.
Defaults to the connection's most recent sync job. Other job types must be
targeted with an explicit job_id.
384 def get_previous_sync_logs( 385 self, 386 *, 387 limit: int = 20, 388 offset: int | None = None, 389 from_tail: bool = True, 390 job_type: str | JobTypeEnum | None = None, 391 ) -> list[SyncResult]: 392 """Get previous sync jobs for a connection with pagination support. 393 394 Returns SyncResult objects containing job metadata (job_id, status, bytes_synced, 395 rows_synced, start_time). Full log text can be fetched lazily via 396 `SyncResult.get_full_log_text()`. 397 398 Args: 399 limit: Maximum number of jobs to return. Defaults to 20. 400 offset: Number of jobs to skip from the beginning. Defaults to None (0). 401 from_tail: If True, returns jobs ordered newest-first (createdAt DESC). 402 If False, returns jobs ordered oldest-first (createdAt ASC). 403 Defaults to True. 404 job_type: Filter by job type (e.g., `sync`, `refresh`). 405 If not specified, defaults to sync and reset jobs only (API default behavior). 406 407 Returns: 408 A list of SyncResult objects representing the sync jobs. 409 """ 410 order_by = ( 411 api_util.JOB_ORDER_BY_CREATED_AT_DESC 412 if from_tail 413 else api_util.JOB_ORDER_BY_CREATED_AT_ASC 414 ) 415 sync_logs = api_util.get_job_logs( 416 connection_id=self.connection_id, 417 api_root=self.workspace.api_root, 418 workspace_id=self.workspace.workspace_id, 419 limit=limit, 420 offset=offset, 421 order_by=order_by, 422 job_type=job_type, 423 client_id=self.workspace.client_id, 424 client_secret=self.workspace.client_secret, 425 bearer_token=self.workspace.bearer_token, 426 ) 427 return [ 428 SyncResult( 429 workspace=self.workspace, 430 connection=self, 431 job_id=sync_log.job_id, 432 _latest_job_info=CloudJobInfo.from_api_response(sync_log), 433 ) 434 for sync_log in sync_logs 435 ]
Get previous sync jobs for a connection with pagination support.
Returns SyncResult objects containing job metadata (job_id, status, bytes_synced,
rows_synced, start_time). Full log text can be fetched lazily via
SyncResult.get_full_log_text().
Arguments:
- limit: Maximum number of jobs to return. Defaults to 20.
- offset: Number of jobs to skip from the beginning. Defaults to None (0).
- from_tail: If True, returns jobs ordered newest-first (createdAt DESC). If False, returns jobs ordered oldest-first (createdAt ASC). Defaults to True.
- job_type: Filter by job type (e.g.,
sync,refresh). If not specified, defaults to sync and reset jobs only (API default behavior).
Returns:
A list of SyncResult objects representing the sync jobs.
437 def get_sync_result( 438 self, 439 job_id: int | None = None, 440 ) -> SyncResult | None: 441 """Get the sync result for the connection. 442 443 If `job_id` is not provided, the most recent sync job will be used. 444 445 Returns `None` if job_id is omitted and no previous jobs are found. 446 """ 447 if job_id is None: 448 # Get the most recent sync job 449 results = self.get_previous_sync_logs( 450 limit=1, 451 ) 452 if results: 453 return results[0] 454 455 return None 456 457 # Get the sync job by ID (lazy loaded) 458 return SyncResult( 459 workspace=self.workspace, 460 connection=self, 461 job_id=job_id, 462 )
Get the sync result for the connection.
If job_id is not provided, the most recent sync job will be used.
Returns None if job_id is omitted and no previous jobs are found.
466 @deprecated("Use 'dump_raw_state()' instead.") 467 def get_state_artifacts(self) -> list[dict[str, Any]] | None: 468 """Deprecated. Use `dump_raw_state()` instead.""" 469 state_response = api_util.get_connection_state( 470 connection_id=self.connection_id, 471 api_root=self.workspace.api_root, 472 client_id=self.workspace.client_id, 473 client_secret=self.workspace.client_secret, 474 bearer_token=self.workspace.bearer_token, 475 config_api_root=self.workspace.config_api_root, 476 ) 477 if state_response.get("stateType") == "not_set": 478 return None 479 return state_response.get("streamState", [])
Deprecated. Use dump_raw_state() instead.
487 def dump_raw_state( 488 self, 489 *, 490 normalize: bool = True, 491 ) -> dict[str, Any] | list[dict[str, Any]]: 492 """Dump the state for this connection. 493 494 By default, returns a list of Airbyte protocol `AirbyteStateMessage` dicts 495 with snake_case keys, suitable for passing to a connector's `--state` flag. 496 497 When `normalize` is `False`, returns the raw Config API dict (camelCase keys, 498 includes `stateType` and `connectionId`). This raw format can be passed 499 directly to `import_raw_state()` for backup/restore workflows. 500 501 Args: 502 normalize: If `True` (default), convert to Airbyte protocol format. 503 If `False`, return the raw Config API response. 504 505 Returns: 506 Normalized: list of protocol-format state message dicts (empty list if 507 no state). Raw: the full Config API state dict. 508 """ 509 raw = api_util.get_connection_state( 510 connection_id=self.connection_id, 511 api_root=self.workspace.api_root, 512 client_id=self.workspace.client_id, 513 client_secret=self.workspace.client_secret, 514 bearer_token=self.workspace.bearer_token, 515 config_api_root=self.workspace.config_api_root, 516 ) 517 if normalize: 518 return _normalize_state_to_protocol(raw) 519 return raw
Dump the state for this connection.
By default, returns a list of Airbyte protocol AirbyteStateMessage dicts
with snake_case keys, suitable for passing to a connector's --state flag.
When normalize is False, returns the raw Config API dict (camelCase keys,
includes stateType and connectionId). This raw format can be passed
directly to import_raw_state() for backup/restore workflows.
Arguments:
- normalize: If
True(default), convert to Airbyte protocol format. IfFalse, return the raw Config API response.
Returns:
Normalized: list of protocol-format state message dicts (empty list if no state). Raw: the full Config API state dict.
521 def import_raw_state( 522 self, 523 connection_state: dict[str, Any] | list[dict[str, Any]], 524 ) -> dict[str, Any]: 525 """Import (restore) the full state for this connection. 526 527 > ⚠️ **WARNING:** Modifying the state directly is not recommended and 528 > could result in broken connections, and/or incorrect sync behavior. 529 530 Replaces the entire connection state with the provided state blob. 531 Uses the safe variant that prevents updates while a sync is running (HTTP 423). 532 533 This is the counterpart to `dump_raw_state()` for backup/restore workflows. 534 The `connectionId` in the blob is always overridden with this connection's 535 ID, making state blobs portable across connections. 536 537 Accepts either format: 538 539 - **Config API format** (dict with `stateType`): passed through directly. 540 - **Airbyte protocol format** (list of `AirbyteStateMessage` dicts): automatically 541 converted to Config API format before sending. 542 543 Args: 544 connection_state: Connection state in either Config API or Airbyte protocol format. 545 546 Returns: 547 The updated connection state as a dictionary. 548 549 Raises: 550 AirbyteConnectionSyncActiveError: If a sync is currently running on this 551 connection (HTTP 423). Wait for the sync to complete before retrying. 552 """ 553 api_state: dict[str, Any] 554 if isinstance(connection_state, list): 555 if not _is_protocol_state_format(connection_state): 556 msg = ( 557 "Expected connection_state list to contain Airbyte protocol state " 558 "message dicts (each with a top-level `type` of STREAM, GLOBAL, " 559 "or LEGACY). Got a list that does not match protocol format." 560 ) 561 raise ValueError(msg) 562 api_state = _denormalize_protocol_state_to_api( 563 protocol_messages=connection_state, 564 connection_id=self.connection_id, 565 ) 566 elif isinstance(connection_state, dict): 567 if _is_protocol_state_format(connection_state): 568 api_state = _denormalize_protocol_state_to_api( 569 protocol_messages=[connection_state], 570 connection_id=self.connection_id, 571 ) 572 else: 573 api_state = connection_state 574 else: 575 msg = f"Expected a dict or list, got {type(connection_state)}" 576 raise TypeError(msg) 577 578 return api_util.replace_connection_state( 579 connection_id=self.connection_id, 580 connection_state_dict=api_state, 581 api_root=self.workspace.api_root, 582 client_id=self.workspace.client_id, 583 client_secret=self.workspace.client_secret, 584 bearer_token=self.workspace.bearer_token, 585 config_api_root=self.workspace.config_api_root, 586 )
Import (restore) the full state for this connection.
⚠️ WARNING: Modifying the state directly is not recommended and could result in broken connections, and/or incorrect sync behavior.
Replaces the entire connection state with the provided state blob. Uses the safe variant that prevents updates while a sync is running (HTTP 423).
This is the counterpart to dump_raw_state() for backup/restore workflows.
The connectionId in the blob is always overridden with this connection's
ID, making state blobs portable across connections.
Accepts either format:
- Config API format (dict with
stateType): passed through directly. - Airbyte protocol format (list of
AirbyteStateMessagedicts): automatically converted to Config API format before sending.
Arguments:
- connection_state: Connection state in either Config API or Airbyte protocol format.
Returns:
The updated connection state as a dictionary.
Raises:
- AirbyteConnectionSyncActiveError: If a sync is currently running on this connection (HTTP 423). Wait for the sync to complete before retrying.
588 def get_stream_state( 589 self, 590 stream_name: str, 591 stream_namespace: str | None = None, 592 ) -> dict[str, Any] | None: 593 """Get the state blob for a single stream within this connection. 594 595 Returns just the stream's state dictionary (e.g., {"cursor": "2024-01-01"}), 596 not the full connection state envelope. 597 598 This is compatible with `stream`-type state and stream-level entries 599 within a `global`-type state. It is not compatible with `legacy` state. 600 To get or set the entire connection-level state artifact, use 601 `dump_raw_state` and `import_raw_state` instead. 602 603 Args: 604 stream_name: The name of the stream to get state for. 605 stream_namespace: The source-side stream namespace. This refers to the 606 namespace from the source (e.g., database schema), not any destination 607 namespace override set in connection advanced settings. 608 609 Returns: 610 The stream's state blob as a dictionary, or None if the stream is not found. 611 """ 612 state_data = self.dump_raw_state(normalize=False) 613 result = ConnectionStateResponse(**state_data) 614 615 streams = _get_stream_list(result) 616 matching = [s for s in streams if _match_stream(s, stream_name, stream_namespace)] 617 618 if not matching: 619 available = [s.stream_descriptor.name for s in streams] 620 logger.warning( 621 "Stream '%s' not found in connection state for connection '%s'. " 622 "Available streams: %s", 623 stream_name, 624 self.connection_id, 625 available, 626 ) 627 return None 628 629 return matching[0].stream_state
Get the state blob for a single stream within this connection.
Returns just the stream's state dictionary (e.g., {"cursor": "2024-01-01"}), not the full connection state envelope.
This is compatible with stream-type state and stream-level entries
within a global-type state. It is not compatible with legacy state.
To get or set the entire connection-level state artifact, use
dump_raw_state and import_raw_state instead.
Arguments:
- stream_name: The name of the stream to get state for.
- stream_namespace: The source-side stream namespace. This refers to the namespace from the source (e.g., database schema), not any destination namespace override set in connection advanced settings.
Returns:
The stream's state blob as a dictionary, or None if the stream is not found.
631 def set_stream_state( 632 self, 633 stream_name: str, 634 state_blob_dict: dict[str, Any], 635 stream_namespace: str | None = None, 636 ) -> None: 637 """Set the state for a single stream within this connection. 638 639 Fetches the current full state, replaces only the specified stream's state, 640 then sends the full updated state back to the API. If the stream does not 641 exist in the current state, it is appended. 642 643 This is compatible with `stream`-type state and stream-level entries 644 within a `global`-type state. It is not compatible with `legacy` state. 645 To get or set the entire connection-level state artifact, use 646 `dump_raw_state` and `import_raw_state` instead. 647 648 Uses the safe variant that prevents updates while a sync is running (HTTP 423). 649 650 Args: 651 stream_name: The name of the stream to update state for. 652 state_blob_dict: The state blob dict for this stream (e.g., {"cursor": "2024-01-01"}). 653 stream_namespace: The source-side stream namespace. This refers to the 654 namespace from the source (e.g., database schema), not any destination 655 namespace override set in connection advanced settings. 656 657 Raises: 658 PyAirbyteInputError: If the connection state type is not supported for 659 stream-level operations (not_set, legacy). 660 AirbyteConnectionSyncActiveError: If a sync is currently running on this 661 connection (HTTP 423). Wait for the sync to complete before retrying. 662 """ 663 state_data = self.dump_raw_state(normalize=False) 664 current = ConnectionStateResponse(**state_data) 665 666 if current.state_type == "not_set": 667 raise PyAirbyteInputError( 668 message="Cannot set stream state: connection has no existing state.", 669 context={"connection_id": self.connection_id}, 670 ) 671 672 if current.state_type == "legacy": 673 raise PyAirbyteInputError( 674 message="Cannot set stream state on a legacy-type connection state.", 675 context={"connection_id": self.connection_id}, 676 ) 677 678 new_stream_entry = { 679 "streamDescriptor": { 680 "name": stream_name, 681 **( 682 { 683 "namespace": stream_namespace, 684 } 685 if stream_namespace 686 else {} 687 ), 688 }, 689 "streamState": state_blob_dict, 690 } 691 692 raw_streams: list[dict[str, Any]] 693 if current.state_type == "stream": 694 raw_streams = state_data.get("streamState", []) 695 elif current.state_type == "global": 696 raw_streams = state_data.get("globalState", {}).get("streamStates", []) 697 else: 698 raw_streams = [] 699 700 streams = _get_stream_list(current) 701 found = False 702 updated_streams_raw: list[dict[str, Any]] = [] 703 for raw_s, parsed_s in zip(raw_streams, streams, strict=False): 704 if _match_stream(parsed_s, stream_name, stream_namespace): 705 updated_streams_raw.append(new_stream_entry) 706 found = True 707 else: 708 updated_streams_raw.append(raw_s) 709 710 if not found: 711 updated_streams_raw.append(new_stream_entry) 712 713 full_state: dict[str, Any] = { 714 **state_data, 715 } 716 717 if current.state_type == "stream": 718 full_state["streamState"] = updated_streams_raw 719 elif current.state_type == "global": 720 original_global = state_data.get("globalState", {}) 721 full_state["globalState"] = { 722 **original_global, 723 "streamStates": updated_streams_raw, 724 } 725 726 self.import_raw_state(full_state)
Set the state for a single stream within this connection.
Fetches the current full state, replaces only the specified stream's state, then sends the full updated state back to the API. If the stream does not exist in the current state, it is appended.
This is compatible with stream-type state and stream-level entries
within a global-type state. It is not compatible with legacy state.
To get or set the entire connection-level state artifact, use
dump_raw_state and import_raw_state instead.
Uses the safe variant that prevents updates while a sync is running (HTTP 423).
Arguments:
- stream_name: The name of the stream to update state for.
- state_blob_dict: The state blob dict for this stream (e.g., {"cursor": "2024-01-01"}).
- stream_namespace: The source-side stream namespace. This refers to the namespace from the source (e.g., database schema), not any destination namespace override set in connection advanced settings.
Raises:
- PyAirbyteInputError: If the connection state type is not supported for stream-level operations (not_set, legacy).
- AirbyteConnectionSyncActiveError: If a sync is currently running on this connection (HTTP 423). Wait for the sync to complete before retrying.
728 @deprecated("Use 'dump_raw_catalog()' instead.") 729 def get_catalog_artifact(self) -> dict[str, Any] | None: 730 """Get the configured catalog for this connection. 731 732 Returns the full configured catalog (syncCatalog) for this connection, 733 including stream schemas, sync modes, cursor fields, and primary keys. 734 735 Uses the Config API endpoint: POST /v1/web_backend/connections/get 736 737 Returns: 738 Dictionary containing the configured catalog, or `None` if not found. 739 """ 740 return self.dump_raw_catalog()
Get the configured catalog for this connection.
Returns the full configured catalog (syncCatalog) for this connection, including stream schemas, sync modes, cursor fields, and primary keys.
Uses the Config API endpoint: POST /v1/web_backend/connections/get
Returns:
Dictionary containing the configured catalog, or
Noneif not found.
742 def dump_raw_catalog( 743 self, 744 *, 745 normalize: bool = True, 746 ) -> dict[str, Any] | None: 747 """Dump the configured catalog for this connection. 748 749 By default, returns the catalog in Airbyte protocol format 750 (`ConfiguredAirbyteCatalog` with snake_case keys), suitable for passing 751 to a connector's `--catalog` flag. 752 753 When `normalize` is `False`, returns the raw `syncCatalog` dict from the 754 Config API (camelCase keys, nested `config` block). This raw format can be 755 passed directly to `import_raw_catalog()` for backup/restore workflows. 756 757 Args: 758 normalize: If `True` (default), convert to Airbyte protocol format. 759 If `False`, return the raw Config API catalog. 760 761 Returns: 762 The configured catalog dict, or `None` if not found. 763 """ 764 connection_response = api_util.get_connection_catalog( 765 connection_id=self.connection_id, 766 api_root=self.workspace.api_root, 767 client_id=self.workspace.client_id, 768 client_secret=self.workspace.client_secret, 769 bearer_token=self.workspace.bearer_token, 770 config_api_root=self.workspace.config_api_root, 771 ) 772 raw = connection_response.get("syncCatalog") 773 if raw is None: 774 return None 775 if normalize: 776 return _normalize_catalog_to_protocol(raw) 777 return raw
Dump the configured catalog for this connection.
By default, returns the catalog in Airbyte protocol format
(ConfiguredAirbyteCatalog with snake_case keys), suitable for passing
to a connector's --catalog flag.
When normalize is False, returns the raw syncCatalog dict from the
Config API (camelCase keys, nested config block). This raw format can be
passed directly to import_raw_catalog() for backup/restore workflows.
Arguments:
- normalize: If
True(default), convert to Airbyte protocol format. IfFalse, return the raw Config API catalog.
Returns:
The configured catalog dict, or
Noneif not found.
779 def import_raw_catalog(self, catalog: dict[str, Any]) -> None: 780 """Replace the configured catalog for this connection. 781 782 > ⚠️ **WARNING:** Modifying the catalog directly is not recommended and 783 > could result in broken connections, and/or incorrect sync behavior. 784 785 Accepts a configured catalog dict and replaces the connection's entire 786 catalog with it. All other connection settings remain unchanged. 787 788 Accepts either format: 789 790 - **Config API format** (`syncCatalog` with camelCase keys and nested `config`): 791 passed through directly. 792 - **Airbyte protocol format** (`ConfiguredAirbyteCatalog` with snake_case keys): 793 automatically converted to Config API format before sending. 794 795 Args: 796 catalog: The configured catalog dict in either format. 797 """ 798 if _is_protocol_catalog_format(catalog): 799 catalog = _denormalize_catalog_to_api(catalog) 800 801 api_util.replace_connection_catalog( 802 connection_id=self.connection_id, 803 configured_catalog_dict=catalog, 804 api_root=self.workspace.api_root, 805 client_id=self.workspace.client_id, 806 client_secret=self.workspace.client_secret, 807 bearer_token=self.workspace.bearer_token, 808 config_api_root=self.workspace.config_api_root, 809 )
Replace the configured catalog for this connection.
⚠️ WARNING: Modifying the catalog directly is not recommended and could result in broken connections, and/or incorrect sync behavior.
Accepts a configured catalog dict and replaces the connection's entire catalog with it. All other connection settings remain unchanged.
Accepts either format:
- Config API format (
syncCatalogwith camelCase keys and nestedconfig): passed through directly. - Airbyte protocol format (
ConfiguredAirbyteCatalogwith snake_case keys): automatically converted to Config API format before sending.
Arguments:
- catalog: The configured catalog dict in either format.
811 def rename(self, name: str) -> CloudConnection: 812 """Rename the connection. 813 814 Args: 815 name: New name for the connection 816 817 Returns: 818 Updated CloudConnection object with refreshed info 819 """ 820 updated_response = api_util.patch_connection( 821 connection_id=self.connection_id, 822 api_root=self.workspace.api_root, 823 client_id=self.workspace.client_id, 824 client_secret=self.workspace.client_secret, 825 bearer_token=self.workspace.bearer_token, 826 name=name, 827 ) 828 self._connection_info = CloudConnectionInfo.from_api_response(updated_response) 829 return self
Rename the connection.
Arguments:
- name: New name for the connection
Returns:
Updated CloudConnection object with refreshed info
831 def set_table_prefix(self, prefix: str) -> CloudConnection: 832 """Set the table prefix for the connection. 833 834 Args: 835 prefix: New table prefix to use when syncing to the destination 836 837 Returns: 838 Updated CloudConnection object with refreshed info 839 """ 840 updated_response = api_util.patch_connection( 841 connection_id=self.connection_id, 842 api_root=self.workspace.api_root, 843 client_id=self.workspace.client_id, 844 client_secret=self.workspace.client_secret, 845 bearer_token=self.workspace.bearer_token, 846 prefix=prefix, 847 ) 848 self._connection_info = CloudConnectionInfo.from_api_response(updated_response) 849 return self
Set the table prefix for the connection.
Arguments:
- prefix: New table prefix to use when syncing to the destination
Returns:
Updated CloudConnection object with refreshed info
851 def set_selected_streams(self, stream_names: list[str]) -> CloudConnection: 852 """Set the selected streams for the connection. 853 854 This is a destructive operation that can break existing connections if the 855 stream selection is changed incorrectly. Use with caution. 856 857 Args: 858 stream_names: List of stream names to sync 859 860 Returns: 861 Updated CloudConnection object with refreshed info 862 """ 863 configurations = api_util.build_stream_configurations(stream_names) 864 865 updated_response = api_util.patch_connection( 866 connection_id=self.connection_id, 867 api_root=self.workspace.api_root, 868 client_id=self.workspace.client_id, 869 client_secret=self.workspace.client_secret, 870 bearer_token=self.workspace.bearer_token, 871 configurations=configurations, 872 ) 873 self._connection_info = CloudConnectionInfo.from_api_response(updated_response) 874 return self
Set the selected streams for the connection.
This is a destructive operation that can break existing connections if the stream selection is changed incorrectly. Use with caution.
Arguments:
- stream_names: List of stream names to sync
Returns:
Updated CloudConnection object with refreshed info
878 @property 879 def enabled(self) -> bool: 880 """Get the current enabled status of the connection. 881 882 This property always fetches fresh data from the API to ensure accuracy, 883 as another process or user may have toggled the setting. 884 885 Returns: 886 True if the connection status is 'active', False otherwise. 887 """ 888 connection_info = self._fetch_connection_info(force_refresh=True) 889 return connection_info.status == "active"
Get the current enabled status of the connection.
This property always fetches fresh data from the API to ensure accuracy, as another process or user may have toggled the setting.
Returns:
True if the connection status is 'active', False otherwise.
901 def set_enabled( 902 self, 903 *, 904 enabled: bool, 905 ignore_noop: bool = True, 906 ) -> None: 907 """Set the enabled status of the connection. 908 909 Args: 910 enabled: True to enable (set status to 'active'), False to disable 911 (set status to 'inactive'). 912 ignore_noop: If True (default), silently return if the connection is already 913 in the requested state. If False, raise ValueError when the requested 914 state matches the current state. 915 916 Raises: 917 ValueError: If ignore_noop is False and the connection is already in the 918 requested state. 919 """ 920 # Always fetch fresh data to check current status 921 connection_info = self._fetch_connection_info(force_refresh=True) 922 current_status = connection_info.status 923 desired_status = "active" if enabled else "inactive" 924 925 if current_status == desired_status: 926 if ignore_noop: 927 return 928 raise ValueError( 929 f"Connection is already {'enabled' if enabled else 'disabled'}. " 930 f"Current status: {current_status}" 931 ) 932 933 updated_response = api_util.patch_connection( 934 connection_id=self.connection_id, 935 api_root=self.workspace.api_root, 936 client_id=self.workspace.client_id, 937 client_secret=self.workspace.client_secret, 938 bearer_token=self.workspace.bearer_token, 939 status=desired_status, 940 ) 941 self._connection_info = CloudConnectionInfo.from_api_response(updated_response)
Set the enabled status of the connection.
Arguments:
- enabled: True to enable (set status to 'active'), False to disable (set status to 'inactive').
- ignore_noop: If True (default), silently return if the connection is already in the requested state. If False, raise ValueError when the requested state matches the current state.
Raises:
- ValueError: If ignore_noop is False and the connection is already in the requested state.
945 def set_schedule( 946 self, 947 cron_expression: str, 948 ) -> None: 949 """Set a cron schedule for the connection. 950 951 Args: 952 cron_expression: A cron expression defining when syncs should run. 953 954 Examples: 955 - "0 0 * * *" # Daily at midnight UTC 956 - "0 */6 * * *" # Every 6 hours 957 - "0 0 * * 0" # Weekly on Sunday at midnight UTC 958 """ 959 updated_response = api_util.patch_connection( 960 connection_id=self.connection_id, 961 api_root=self.workspace.api_root, 962 client_id=self.workspace.client_id, 963 client_secret=self.workspace.client_secret, 964 bearer_token=self.workspace.bearer_token, 965 schedule=api_util.build_connection_schedule( 966 schedule_type="cron", 967 cron_expression=cron_expression, 968 ), 969 ) 970 self._connection_info = CloudConnectionInfo.from_api_response(updated_response)
Set a cron schedule for the connection.
Arguments:
- cron_expression: A cron expression defining when syncs should run.
Examples:
- "0 0 * * *" # Daily at midnight UTC
- "0 */6 * * *" # Every 6 hours
- "0 0 * * 0" # Weekly on Sunday at midnight UTC
972 def set_manual_schedule(self) -> None: 973 """Set the connection to manual scheduling. 974 975 Disables automatic syncs. Syncs will only run when manually triggered. 976 """ 977 updated_response = api_util.patch_connection( 978 connection_id=self.connection_id, 979 api_root=self.workspace.api_root, 980 client_id=self.workspace.client_id, 981 client_secret=self.workspace.client_secret, 982 bearer_token=self.workspace.bearer_token, 983 schedule=api_util.build_connection_schedule(schedule_type="manual"), 984 ) 985 self._connection_info = CloudConnectionInfo.from_api_response(updated_response)
Set the connection to manual scheduling.
Disables automatic syncs. Syncs will only run when manually triggered.
989 def permanently_delete( 990 self, 991 *, 992 cascade_delete_source: bool = False, 993 cascade_delete_destination: bool = False, 994 ) -> None: 995 """Delete the connection. 996 997 Args: 998 cascade_delete_source: Whether to also delete the source. 999 cascade_delete_destination: Whether to also delete the destination. 1000 """ 1001 self.workspace.permanently_delete_connection(self) 1002 1003 if cascade_delete_source: 1004 self.workspace.permanently_delete_source(self.source_id) 1005 1006 if cascade_delete_destination: 1007 self.workspace.permanently_delete_destination(self.destination_id)
Delete the connection.
Arguments:
- cascade_delete_source: Whether to also delete the source.
- cascade_delete_destination: Whether to also delete the destination.
59@dataclass 60class CloudClientConfig: 61 """Client configuration for Airbyte Cloud API. 62 63 This class encapsulates the authentication and API configuration needed to connect 64 to Airbyte Cloud, OSS, or Enterprise instances. It supports two mutually 65 exclusive authentication methods: 66 67 1. OAuth2 client credentials flow (client_id + client_secret) 68 2. Bearer token authentication 69 70 Exactly one authentication method must be provided. Providing both or neither 71 will raise a validation error. 72 73 Attributes: 74 client_id: OAuth2 client ID for client credentials flow. 75 client_secret: OAuth2 client secret for client credentials flow. 76 bearer_token: Pre-generated bearer token for direct authentication. 77 api_root: The API root URL. Defaults to Airbyte Cloud API. 78 config_api_root: The Config API root URL. 79 """ 80 81 client_id: SecretString | None = None 82 """OAuth2 client ID for client credentials authentication.""" 83 84 client_secret: SecretString | None = None 85 """OAuth2 client secret for client credentials authentication.""" 86 87 bearer_token: SecretString | None = None 88 """Bearer token for direct authentication (alternative to client credentials).""" 89 90 api_root: str = api_util.CLOUD_API_ROOT 91 """The API root URL. Defaults to Airbyte Cloud API.""" 92 93 config_api_root: str | None = None 94 """The Config API root URL.""" 95 96 def __post_init__(self) -> None: 97 """Validate credentials and ensure secrets are properly wrapped.""" 98 # Wrap secrets in SecretString if they aren't already 99 if self.client_id is not None: 100 self.client_id = SecretString(self.client_id) 101 if self.client_secret is not None: 102 self.client_secret = SecretString(self.client_secret) 103 if self.bearer_token is not None: 104 self.bearer_token = SecretString(self.bearer_token) 105 106 # Validate mutual exclusivity 107 has_client_credentials = self.client_id is not None or self.client_secret is not None 108 has_bearer_token = self.bearer_token is not None 109 110 if has_client_credentials and has_bearer_token: 111 raise PyAirbyteInputError( 112 message="Cannot use both client credentials and bearer token authentication.", 113 guidance=( 114 "Provide either client_id and client_secret together, " 115 "or bearer_token alone, but not both." 116 ), 117 ) 118 119 if has_client_credentials and (self.client_id is None or self.client_secret is None): 120 # If using client credentials, both must be provided 121 raise PyAirbyteInputError( 122 message="Incomplete client credentials.", 123 guidance=( 124 "When using client credentials authentication, " 125 "both client_id and client_secret must be provided." 126 ), 127 ) 128 129 if not has_client_credentials and not has_bearer_token: 130 raise PyAirbyteInputError( 131 message="No authentication credentials provided.", 132 guidance=( 133 "Provide either client_id and client_secret together for OAuth2 " 134 "client credentials flow, or bearer_token for direct authentication." 135 ), 136 ) 137 138 @property 139 def uses_bearer_token(self) -> bool: 140 """Return True if using bearer token authentication.""" 141 return self.bearer_token is not None 142 143 @property 144 def uses_client_credentials(self) -> bool: 145 """Return True if using client credentials authentication.""" 146 return self.client_id is not None and self.client_secret is not None 147 148 @classmethod 149 def from_env( 150 cls, 151 *, 152 api_root: str | None = None, 153 config_api_root: str | None = None, 154 ) -> CloudClientConfig: 155 """Create CloudClientConfig from environment variables. 156 157 This factory method resolves credentials from environment variables, 158 providing a convenient way to create credentials without explicitly 159 passing secrets. 160 161 Environment variables used: 162 - `AIRBYTE_CLOUD_CLIENT_ID`: OAuth client ID (for client credentials flow). 163 - `AIRBYTE_CLOUD_CLIENT_SECRET`: OAuth client secret (for client credentials flow). 164 - `AIRBYTE_CLOUD_BEARER_TOKEN`: Bearer token (alternative to client credentials). 165 - `AIRBYTE_CLOUD_API_URL`: Optional. The API root URL (defaults to Airbyte Cloud). 166 - `AIRBYTE_CLOUD_CONFIG_API_URL`: Optional. The Config API root URL. 167 168 The method will first check for a bearer token. If not found, it will 169 attempt to use client credentials. 170 171 Args: 172 api_root: The API root URL. If not provided, will be resolved from 173 the `AIRBYTE_CLOUD_API_URL` environment variable, or default to 174 the Airbyte Cloud API. 175 config_api_root: The Config API root URL. If not provided, will be resolved 176 from the `AIRBYTE_CLOUD_CONFIG_API_URL` environment variable. 177 178 Returns: 179 A CloudClientConfig instance configured with credentials from the environment. 180 181 Raises: 182 PyAirbyteSecretNotFoundError: If required credentials are not found in 183 the environment. 184 """ 185 resolved_api_root = resolve_cloud_api_url(api_root) 186 resolved_config_api_root = resolve_cloud_config_api_url(config_api_root) 187 188 # Try bearer token first 189 bearer_token = resolve_cloud_bearer_token() 190 if bearer_token: 191 return cls( 192 bearer_token=bearer_token, 193 api_root=resolved_api_root, 194 config_api_root=resolved_config_api_root, 195 ) 196 197 # Fall back to client credentials 198 return cls( 199 client_id=resolve_cloud_client_id(), 200 client_secret=resolve_cloud_client_secret(), 201 api_root=resolved_api_root, 202 config_api_root=resolved_config_api_root, 203 )
Client configuration for Airbyte Cloud API.
This class encapsulates the authentication and API configuration needed to connect to Airbyte Cloud, OSS, or Enterprise instances. It supports two mutually exclusive authentication methods:
- OAuth2 client credentials flow (client_id + client_secret)
- Bearer token authentication
Exactly one authentication method must be provided. Providing both or neither will raise a validation error.
Attributes:
- client_id: OAuth2 client ID for client credentials flow.
- client_secret: OAuth2 client secret for client credentials flow.
- bearer_token: Pre-generated bearer token for direct authentication.
- api_root: The API root URL. Defaults to Airbyte Cloud API.
- config_api_root: The Config API root URL.
138 @property 139 def uses_bearer_token(self) -> bool: 140 """Return True if using bearer token authentication.""" 141 return self.bearer_token is not None
Return True if using bearer token authentication.
143 @property 144 def uses_client_credentials(self) -> bool: 145 """Return True if using client credentials authentication.""" 146 return self.client_id is not None and self.client_secret is not None
Return True if using client credentials authentication.
148 @classmethod 149 def from_env( 150 cls, 151 *, 152 api_root: str | None = None, 153 config_api_root: str | None = None, 154 ) -> CloudClientConfig: 155 """Create CloudClientConfig from environment variables. 156 157 This factory method resolves credentials from environment variables, 158 providing a convenient way to create credentials without explicitly 159 passing secrets. 160 161 Environment variables used: 162 - `AIRBYTE_CLOUD_CLIENT_ID`: OAuth client ID (for client credentials flow). 163 - `AIRBYTE_CLOUD_CLIENT_SECRET`: OAuth client secret (for client credentials flow). 164 - `AIRBYTE_CLOUD_BEARER_TOKEN`: Bearer token (alternative to client credentials). 165 - `AIRBYTE_CLOUD_API_URL`: Optional. The API root URL (defaults to Airbyte Cloud). 166 - `AIRBYTE_CLOUD_CONFIG_API_URL`: Optional. The Config API root URL. 167 168 The method will first check for a bearer token. If not found, it will 169 attempt to use client credentials. 170 171 Args: 172 api_root: The API root URL. If not provided, will be resolved from 173 the `AIRBYTE_CLOUD_API_URL` environment variable, or default to 174 the Airbyte Cloud API. 175 config_api_root: The Config API root URL. If not provided, will be resolved 176 from the `AIRBYTE_CLOUD_CONFIG_API_URL` environment variable. 177 178 Returns: 179 A CloudClientConfig instance configured with credentials from the environment. 180 181 Raises: 182 PyAirbyteSecretNotFoundError: If required credentials are not found in 183 the environment. 184 """ 185 resolved_api_root = resolve_cloud_api_url(api_root) 186 resolved_config_api_root = resolve_cloud_config_api_url(config_api_root) 187 188 # Try bearer token first 189 bearer_token = resolve_cloud_bearer_token() 190 if bearer_token: 191 return cls( 192 bearer_token=bearer_token, 193 api_root=resolved_api_root, 194 config_api_root=resolved_config_api_root, 195 ) 196 197 # Fall back to client credentials 198 return cls( 199 client_id=resolve_cloud_client_id(), 200 client_secret=resolve_cloud_client_secret(), 201 api_root=resolved_api_root, 202 config_api_root=resolved_config_api_root, 203 )
Create CloudClientConfig from environment variables.
This factory method resolves credentials from environment variables, providing a convenient way to create credentials without explicitly passing secrets.
Environment variables used:
AIRBYTE_CLOUD_CLIENT_ID: OAuth client ID (for client credentials flow).AIRBYTE_CLOUD_CLIENT_SECRET: OAuth client secret (for client credentials flow).AIRBYTE_CLOUD_BEARER_TOKEN: Bearer token (alternative to client credentials).AIRBYTE_CLOUD_API_URL: Optional. The API root URL (defaults to Airbyte Cloud).AIRBYTE_CLOUD_CONFIG_API_URL: Optional. The Config API root URL.
The method will first check for a bearer token. If not found, it will attempt to use client credentials.
Arguments:
- api_root: The API root URL. If not provided, will be resolved from
the
AIRBYTE_CLOUD_API_URLenvironment variable, or default to the Airbyte Cloud API. - config_api_root: The Config API root URL. If not provided, will be resolved
from the
AIRBYTE_CLOUD_CONFIG_API_URLenvironment variable.
Returns:
A CloudClientConfig instance configured with credentials from the environment.
Raises:
- PyAirbyteSecretNotFoundError: If required credentials are not found in the environment.
160class CloudDefaultContextInfo(BaseModel): 161 """Explicit organization and workspace affinities for the authenticated user.""" 162 163 user_id: str | None 164 """The Airbyte user ID, if available.""" 165 166 user_name: str | None 167 """The authenticated user's name, if available.""" 168 169 user_email: str | None 170 """The authenticated user's email, if available.""" 171 172 default_workspace_id: str | None 173 """The resolved default workspace ID, if available.""" 174 175 default_workspace_name: str | None 176 """The resolved default workspace name, if available.""" 177 178 default_workspace_verified: bool 179 """Whether the resolved default workspace was verified as accessible.""" 180 181 unvalidated_workspace_count: int = 0 182 """Number of direct workspace grants not validated due to the validation cap.""" 183 184 default_organization_id: str | None 185 """The organization containing the resolved default workspace, if available.""" 186 187 default_organization_name: str | None 188 """The name of the organization containing the resolved default workspace, if available.""" 189 190 configured_workspace_id: str | None 191 """The explicitly configured workspace ID, if available.""" 192 193 configured_organization_id: str | None 194 """The configured organization ID, if available.""" 195 196 member_organizations: list[CloudOrganizationInfo] 197 """Organizations identified by explicit organization membership grants.""" 198 199 member_workspaces: list[CloudWorkspaceInfo] 200 """Workspaces identified by explicit workspace membership grants.""" 201 202 member_organizations_truncated: bool 203 """True if organization memberships beyond the returned list were omitted.""" 204 205 member_workspaces_truncated: bool 206 """True if workspace memberships beyond the returned list were omitted.""" 207 208 discovery_hints: list[str] 209 """Hints for discovering additional organizations or workspaces."""
Explicit organization and workspace affinities for the authenticated user.
The resolved default workspace ID, if available.
The resolved default workspace name, if available.
Whether the resolved default workspace was verified as accessible.
Number of direct workspace grants not validated due to the validation cap.
The organization containing the resolved default workspace, if available.
The name of the organization containing the resolved default workspace, if available.
The explicitly configured workspace ID, if available.
The configured organization ID, if available.
Organizations identified by explicit organization membership grants.
True if organization memberships beyond the returned list were omitted.
102class CloudWorkspaceInfo(BaseModel): 103 """Information about an Airbyte workspace.""" 104 105 model_config = ConfigDict(populate_by_name=True) 106 107 workspace_id: str = Field(alias="workspaceId") 108 """The workspace ID.""" 109 110 name: str 111 """The workspace name.""" 112 113 data_residency: str | None = Field(default=None, alias="dataResidency") 114 """The data residency setting for the workspace, if available.""" 115 116 organization_id: str | None = Field(default=None, alias="organizationId") 117 """The organization ID for the workspace, if available.""" 118 119 organization_name: str | None = Field(default=None, alias="organizationName") 120 """The organization name for the workspace, if available.""" 121 122 notifications: dict[str, object | None] | list[dict[str, object | None]] = Field( 123 default_factory=dict 124 ) 125 """Workspace notification settings.""" 126 127 @classmethod 128 def from_api_response(cls, workspace: _WorkspaceResponseLike) -> CloudWorkspaceInfo: 129 """Create a public model from an internal API workspace response.""" 130 return cls( 131 workspace_id=workspace.workspace_id, 132 name=workspace.name, 133 data_residency=workspace.data_residency, 134 organization_id=getattr(workspace, "organization_id", None), 135 notifications=_notifications_to_dict(workspace.notifications), 136 ) 137 138 @classmethod 139 def from_mapping(cls, workspace: Mapping[str, object]) -> CloudWorkspaceInfo: 140 """Create a public model from a workspace mapping.""" 141 return cls.model_validate(workspace) 142 143 def to_dict(self) -> dict[str, object]: 144 """Return a JSON-serializable dictionary.""" 145 return self.model_dump(mode="json")
Information about an Airbyte workspace.
Workspace notification settings.
127 @classmethod 128 def from_api_response(cls, workspace: _WorkspaceResponseLike) -> CloudWorkspaceInfo: 129 """Create a public model from an internal API workspace response.""" 130 return cls( 131 workspace_id=workspace.workspace_id, 132 name=workspace.name, 133 data_residency=workspace.data_residency, 134 organization_id=getattr(workspace, "organization_id", None), 135 notifications=_notifications_to_dict(workspace.notifications), 136 )
Create a public model from an internal API workspace response.
218@dataclass 219class SyncResult: 220 """The result of a sync operation. 221 222 **This class is not meant to be instantiated directly.** Instead, obtain a `SyncResult` by 223 interacting with the `.CloudWorkspace` and `.CloudConnection` objects. 224 """ 225 226 workspace: CloudWorkspace 227 connection: CloudConnection 228 job_id: int 229 table_name_prefix: str = "" 230 table_name_suffix: str = "" 231 _latest_job_info: CloudJobInfo | None = None 232 _connection_response: CloudConnectionInfo | None = None 233 _cache: CacheBase | None = None 234 _job_with_attempts_info: dict[str, Any] | None = None 235 236 @property 237 def job_url(self) -> str: 238 """Return the URL of the sync job. 239 240 Note: This currently returns the connection's job history URL, as there is no direct URL 241 to a specific job in the Airbyte Cloud web app. 242 243 TODO: Implement a direct job logs URL on top of the event-id of the specific attempt number. 244 E.g. {self.connection.job_history_url}?eventId={event-guid}&openLogs=true 245 """ 246 return f"{self.connection.job_history_url}" 247 248 def _get_connection_info(self, *, force_refresh: bool = False) -> CloudConnectionInfo: 249 """Return connection info for the sync job.""" 250 if self._connection_response and not force_refresh: 251 return self._connection_response 252 253 self._connection_response = CloudConnectionInfo.from_api_response( 254 api_util.get_connection( 255 workspace_id=self.workspace.workspace_id, 256 api_root=self.workspace.api_root, 257 connection_id=self.connection.connection_id, 258 client_id=self.workspace.client_id, 259 client_secret=self.workspace.client_secret, 260 bearer_token=self.workspace.bearer_token, 261 ) 262 ) 263 return self._connection_response 264 265 def _get_destination_configuration(self, *, force_refresh: bool = False) -> dict[str, Any]: 266 """Return the destination configuration for the sync job.""" 267 connection_info = self._get_connection_info(force_refresh=force_refresh) 268 destination_response = api_util.get_destination( 269 destination_id=connection_info.destination_id, 270 api_root=self.workspace.api_root, 271 client_id=self.workspace.client_id, 272 client_secret=self.workspace.client_secret, 273 bearer_token=self.workspace.bearer_token, 274 ) 275 return asdict(destination_response.configuration) 276 277 def is_job_complete(self) -> bool: 278 """Check if the sync job is complete.""" 279 return self.get_job_status() in FINAL_STATUSES 280 281 def get_job_status(self) -> JobStatusEnum: 282 """Check if the sync job is still running.""" 283 return self._fetch_latest_job_info().status 284 285 def _fetch_latest_job_info(self) -> CloudJobInfo: 286 """Return the job info for the sync job.""" 287 if self._latest_job_info and self._latest_job_info.status in FINAL_STATUSES: 288 return self._latest_job_info 289 290 self._latest_job_info = CloudJobInfo.from_api_response( 291 api_util.get_job_info( 292 job_id=self.job_id, 293 api_root=self.workspace.api_root, 294 client_id=self.workspace.client_id, 295 client_secret=self.workspace.client_secret, 296 bearer_token=self.workspace.bearer_token, 297 ) 298 ) 299 return self._latest_job_info 300 301 @property 302 def bytes_synced(self) -> int: 303 """Return the number of records processed.""" 304 return self._fetch_latest_job_info().bytes_synced or 0 305 306 @property 307 def records_synced(self) -> int: 308 """Return the number of records processed.""" 309 return self._fetch_latest_job_info().rows_synced or 0 310 311 @property 312 def start_time(self) -> datetime: 313 """Return the start time of the sync job in UTC.""" 314 try: 315 return ab_datetime_parse(self._fetch_latest_job_info().start_time) 316 except (ValueError, TypeError) as e: 317 if "Invalid isoformat string" in str(e): 318 job_info_raw = api_util._make_config_api_request( # noqa: SLF001 319 api_root=self.workspace.api_root, 320 config_api_root=self.workspace.config_api_root, 321 path="/jobs/get", 322 json={"id": self.job_id}, 323 client_id=self.workspace.client_id, 324 client_secret=self.workspace.client_secret, 325 bearer_token=self.workspace.bearer_token, 326 ) 327 raw_start_time = job_info_raw.get("startTime") 328 if raw_start_time: 329 return ab_datetime_parse(raw_start_time) 330 raise 331 332 def _fetch_job_with_attempts(self) -> dict[str, Any]: 333 """Fetch job info with attempts from Config API using lazy loading pattern.""" 334 if self._job_with_attempts_info is not None: 335 return self._job_with_attempts_info 336 337 self._job_with_attempts_info = api_util._make_config_api_request( # noqa: SLF001 # Config API helper 338 api_root=self.workspace.api_root, 339 config_api_root=self.workspace.config_api_root, 340 path="/jobs/get", 341 json={ 342 "id": self.job_id, 343 }, 344 client_id=self.workspace.client_id, 345 client_secret=self.workspace.client_secret, 346 bearer_token=self.workspace.bearer_token, 347 ) 348 return self._job_with_attempts_info 349 350 def get_attempts(self) -> list[SyncAttempt]: 351 """Return a list of attempts for this sync job.""" 352 job_with_attempts = self._fetch_job_with_attempts() 353 attempts_data = job_with_attempts.get("attempts", []) 354 355 return [ 356 SyncAttempt( 357 workspace=self.workspace, 358 connection=self.connection, 359 job_id=self.job_id, 360 attempt_number=i, 361 _attempt_data=attempt_data, 362 ) 363 for i, attempt_data in enumerate(attempts_data, start=0) 364 ] 365 366 def raise_failure_status( 367 self, 368 *, 369 refresh_status: bool = False, 370 ) -> None: 371 """Raise an exception if the sync job failed. 372 373 By default, this method will use the latest status available. If you want to refresh the 374 status before checking for failure, set `refresh_status=True`. If the job has failed, this 375 method will raise a `AirbyteConnectionSyncError`. 376 377 Otherwise, do nothing. 378 """ 379 if not refresh_status and self._latest_job_info: 380 latest_status = self._latest_job_info.status 381 else: 382 latest_status = self.get_job_status() 383 384 if latest_status in FAILED_STATUSES: 385 raise AirbyteConnectionSyncError( 386 workspace=self.workspace, 387 connection_id=self.connection.connection_id, 388 job_id=self.job_id, 389 job_status=self.get_job_status(), 390 ) 391 392 def wait_for_completion( 393 self, 394 *, 395 wait_timeout: int = DEFAULT_SYNC_TIMEOUT_SECONDS, 396 raise_timeout: bool = True, 397 raise_failure: bool = False, 398 ) -> JobStatusEnum: 399 """Wait for a job to finish running.""" 400 start_time = time.time() 401 while True: 402 latest_status = self.get_job_status() 403 if latest_status in FINAL_STATUSES: 404 if raise_failure: 405 # No-op if the job succeeded or is still running: 406 self.raise_failure_status() 407 408 return latest_status 409 410 if time.time() - start_time > wait_timeout: 411 if raise_timeout: 412 raise AirbyteConnectionSyncTimeoutError( 413 workspace=self.workspace, 414 connection_id=self.connection.connection_id, 415 job_id=self.job_id, 416 job_status=latest_status, 417 timeout=wait_timeout, 418 ) 419 420 return latest_status # This will be a non-final status 421 422 time.sleep(api_util.JOB_WAIT_INTERVAL_SECS) 423 424 def get_sql_cache(self) -> CacheBase: 425 """Return a SQL Cache object for working with the data in a SQL-based destination's.""" 426 if self._cache: 427 return self._cache 428 429 destination_configuration = self._get_destination_configuration() 430 self._cache = destination_to_cache(destination_configuration=destination_configuration) 431 return self._cache 432 433 def get_sql_engine(self) -> sqlalchemy.engine.Engine: 434 """Return a SQL Engine for querying a SQL-based destination.""" 435 return self.get_sql_cache().get_sql_engine() 436 437 def get_sql_table_name(self, stream_name: str) -> str: 438 """Return the SQL table name of the named stream.""" 439 return self.get_sql_cache().processor.get_sql_table_name(stream_name=stream_name) 440 441 def get_sql_table( 442 self, 443 stream_name: str, 444 ) -> sqlalchemy.Table: 445 """Return a SQLAlchemy table object for the named stream.""" 446 return self.get_sql_cache().processor.get_sql_table(stream_name) 447 448 def get_dataset(self, stream_name: str) -> CachedDataset: 449 """Retrieve an `airbyte.datasets.CachedDataset` object for a given stream name. 450 451 This can be used to read and analyze the data in a SQL-based destination. 452 453 TODO: In a future iteration, we can consider providing stream configuration information 454 (catalog information) to the `CachedDataset` object via the "Get stream properties" 455 API: https://reference.airbyte.com/reference/getstreamproperties 456 """ 457 return CachedDataset( 458 self.get_sql_cache(), 459 stream_name=stream_name, 460 stream_configuration=False, # Don't look for stream configuration in cache. 461 ) 462 463 def get_sql_database_name(self) -> str: 464 """Return the SQL database name.""" 465 cache = self.get_sql_cache() 466 return cache.get_database_name() 467 468 def get_sql_schema_name(self) -> str: 469 """Return the SQL schema name.""" 470 cache = self.get_sql_cache() 471 return cache.schema_name 472 473 @property 474 def stream_names(self) -> list[str]: 475 """Return the set of stream names.""" 476 return self.connection.stream_names 477 478 @final 479 @property 480 def streams( 481 self, 482 ) -> _SyncResultStreams: # pyrefly: ignore[unknown-name] 483 """Return a mapping of stream names to `airbyte.CachedDataset` objects. 484 485 This is a convenience wrapper around the `stream_names` 486 property and `get_dataset()` method. 487 """ 488 return self._SyncResultStreams(self) 489 490 class _SyncResultStreams(Mapping[str, CachedDataset]): 491 """A mapping of stream names to cached datasets.""" 492 493 def __init__( 494 self, 495 parent: SyncResult, 496 /, 497 ) -> None: 498 self.parent: SyncResult = parent 499 500 def __getitem__(self, key: str) -> CachedDataset: 501 return self.parent.get_dataset(stream_name=key) 502 503 def __iter__(self) -> Iterator[str]: 504 return iter(self.parent.stream_names) 505 506 def __len__(self) -> int: 507 return len(self.parent.stream_names)
The result of a sync operation.
This class is not meant to be instantiated directly. Instead, obtain a SyncResult by
interacting with the .CloudWorkspace and .CloudConnection objects.
236 @property 237 def job_url(self) -> str: 238 """Return the URL of the sync job. 239 240 Note: This currently returns the connection's job history URL, as there is no direct URL 241 to a specific job in the Airbyte Cloud web app. 242 243 TODO: Implement a direct job logs URL on top of the event-id of the specific attempt number. 244 E.g. {self.connection.job_history_url}?eventId={event-guid}&openLogs=true 245 """ 246 return f"{self.connection.job_history_url}"
Return the URL of the sync job.
Note: This currently returns the connection's job history URL, as there is no direct URL to a specific job in the Airbyte Cloud web app.
TODO: Implement a direct job logs URL on top of the event-id of the specific attempt number. E.g. {self.connection.job_history_url}?eventId={event-guid}&openLogs=true
277 def is_job_complete(self) -> bool: 278 """Check if the sync job is complete.""" 279 return self.get_job_status() in FINAL_STATUSES
Check if the sync job is complete.
281 def get_job_status(self) -> JobStatusEnum: 282 """Check if the sync job is still running.""" 283 return self._fetch_latest_job_info().status
Check if the sync job is still running.
301 @property 302 def bytes_synced(self) -> int: 303 """Return the number of records processed.""" 304 return self._fetch_latest_job_info().bytes_synced or 0
Return the number of records processed.
306 @property 307 def records_synced(self) -> int: 308 """Return the number of records processed.""" 309 return self._fetch_latest_job_info().rows_synced or 0
Return the number of records processed.
311 @property 312 def start_time(self) -> datetime: 313 """Return the start time of the sync job in UTC.""" 314 try: 315 return ab_datetime_parse(self._fetch_latest_job_info().start_time) 316 except (ValueError, TypeError) as e: 317 if "Invalid isoformat string" in str(e): 318 job_info_raw = api_util._make_config_api_request( # noqa: SLF001 319 api_root=self.workspace.api_root, 320 config_api_root=self.workspace.config_api_root, 321 path="/jobs/get", 322 json={"id": self.job_id}, 323 client_id=self.workspace.client_id, 324 client_secret=self.workspace.client_secret, 325 bearer_token=self.workspace.bearer_token, 326 ) 327 raw_start_time = job_info_raw.get("startTime") 328 if raw_start_time: 329 return ab_datetime_parse(raw_start_time) 330 raise
Return the start time of the sync job in UTC.
350 def get_attempts(self) -> list[SyncAttempt]: 351 """Return a list of attempts for this sync job.""" 352 job_with_attempts = self._fetch_job_with_attempts() 353 attempts_data = job_with_attempts.get("attempts", []) 354 355 return [ 356 SyncAttempt( 357 workspace=self.workspace, 358 connection=self.connection, 359 job_id=self.job_id, 360 attempt_number=i, 361 _attempt_data=attempt_data, 362 ) 363 for i, attempt_data in enumerate(attempts_data, start=0) 364 ]
Return a list of attempts for this sync job.
366 def raise_failure_status( 367 self, 368 *, 369 refresh_status: bool = False, 370 ) -> None: 371 """Raise an exception if the sync job failed. 372 373 By default, this method will use the latest status available. If you want to refresh the 374 status before checking for failure, set `refresh_status=True`. If the job has failed, this 375 method will raise a `AirbyteConnectionSyncError`. 376 377 Otherwise, do nothing. 378 """ 379 if not refresh_status and self._latest_job_info: 380 latest_status = self._latest_job_info.status 381 else: 382 latest_status = self.get_job_status() 383 384 if latest_status in FAILED_STATUSES: 385 raise AirbyteConnectionSyncError( 386 workspace=self.workspace, 387 connection_id=self.connection.connection_id, 388 job_id=self.job_id, 389 job_status=self.get_job_status(), 390 )
Raise an exception if the sync job failed.
By default, this method will use the latest status available. If you want to refresh the
status before checking for failure, set refresh_status=True. If the job has failed, this
method will raise a AirbyteConnectionSyncError.
Otherwise, do nothing.
392 def wait_for_completion( 393 self, 394 *, 395 wait_timeout: int = DEFAULT_SYNC_TIMEOUT_SECONDS, 396 raise_timeout: bool = True, 397 raise_failure: bool = False, 398 ) -> JobStatusEnum: 399 """Wait for a job to finish running.""" 400 start_time = time.time() 401 while True: 402 latest_status = self.get_job_status() 403 if latest_status in FINAL_STATUSES: 404 if raise_failure: 405 # No-op if the job succeeded or is still running: 406 self.raise_failure_status() 407 408 return latest_status 409 410 if time.time() - start_time > wait_timeout: 411 if raise_timeout: 412 raise AirbyteConnectionSyncTimeoutError( 413 workspace=self.workspace, 414 connection_id=self.connection.connection_id, 415 job_id=self.job_id, 416 job_status=latest_status, 417 timeout=wait_timeout, 418 ) 419 420 return latest_status # This will be a non-final status 421 422 time.sleep(api_util.JOB_WAIT_INTERVAL_SECS)
Wait for a job to finish running.
424 def get_sql_cache(self) -> CacheBase: 425 """Return a SQL Cache object for working with the data in a SQL-based destination's.""" 426 if self._cache: 427 return self._cache 428 429 destination_configuration = self._get_destination_configuration() 430 self._cache = destination_to_cache(destination_configuration=destination_configuration) 431 return self._cache
Return a SQL Cache object for working with the data in a SQL-based destination's.
433 def get_sql_engine(self) -> sqlalchemy.engine.Engine: 434 """Return a SQL Engine for querying a SQL-based destination.""" 435 return self.get_sql_cache().get_sql_engine()
Return a SQL Engine for querying a SQL-based destination.
437 def get_sql_table_name(self, stream_name: str) -> str: 438 """Return the SQL table name of the named stream.""" 439 return self.get_sql_cache().processor.get_sql_table_name(stream_name=stream_name)
Return the SQL table name of the named stream.
441 def get_sql_table( 442 self, 443 stream_name: str, 444 ) -> sqlalchemy.Table: 445 """Return a SQLAlchemy table object for the named stream.""" 446 return self.get_sql_cache().processor.get_sql_table(stream_name)
Return a SQLAlchemy table object for the named stream.
448 def get_dataset(self, stream_name: str) -> CachedDataset: 449 """Retrieve an `airbyte.datasets.CachedDataset` object for a given stream name. 450 451 This can be used to read and analyze the data in a SQL-based destination. 452 453 TODO: In a future iteration, we can consider providing stream configuration information 454 (catalog information) to the `CachedDataset` object via the "Get stream properties" 455 API: https://reference.airbyte.com/reference/getstreamproperties 456 """ 457 return CachedDataset( 458 self.get_sql_cache(), 459 stream_name=stream_name, 460 stream_configuration=False, # Don't look for stream configuration in cache. 461 )
Retrieve an airbyte.datasets.CachedDataset object for a given stream name.
This can be used to read and analyze the data in a SQL-based destination.
TODO: In a future iteration, we can consider providing stream configuration information
(catalog information) to the CachedDataset object via the "Get stream properties"
API: https://reference.airbyte.com/reference/getstreamproperties
463 def get_sql_database_name(self) -> str: 464 """Return the SQL database name.""" 465 cache = self.get_sql_cache() 466 return cache.get_database_name()
Return the SQL database name.
468 def get_sql_schema_name(self) -> str: 469 """Return the SQL schema name.""" 470 cache = self.get_sql_cache() 471 return cache.schema_name
Return the SQL schema name.
473 @property 474 def stream_names(self) -> list[str]: 475 """Return the set of stream names.""" 476 return self.connection.stream_names
Return the set of stream names.
478 @final 479 @property 480 def streams( 481 self, 482 ) -> _SyncResultStreams: # pyrefly: ignore[unknown-name] 483 """Return a mapping of stream names to `airbyte.CachedDataset` objects. 484 485 This is a convenience wrapper around the `stream_names` 486 property and `get_dataset()` method. 487 """ 488 return self._SyncResultStreams(self)
Return a mapping of stream names to airbyte.CachedDataset objects.
This is a convenience wrapper around the stream_names
property and get_dataset() method.
60class JobStatusEnum(str, Enum): 61 """Status values for an Airbyte Cloud job.""" 62 63 PENDING = "pending" 64 RUNNING = "running" 65 INCOMPLETE = "incomplete" 66 FAILED = "failed" 67 SUCCEEDED = "succeeded" 68 CANCELLED = "cancelled"
Status values for an Airbyte Cloud job.
71class JobTypeEnum(str, Enum): 72 """Job type values for Airbyte Cloud jobs.""" 73 74 SYNC = "sync" 75 RESET = "reset" 76 REFRESH = "refresh" 77 CLEAR = "clear"
Job type values for Airbyte Cloud jobs.
80class WorkspacePrivilegeScope(str, Enum): 81 """How broadly `list_workspaces` searches for workspaces.""" 82 83 MEMBER_OF = "member_of" 84 ORGANIZATION_ADMIN = "organization_admin" 85 INSTANCE_ADMIN = "instance_admin" 86 ANY = "any"
How broadly list_workspaces searches for workspaces.