airbyte.experimental
Experimental features which may change.
NOTE: The following "experimental" features are now "stable" and can be accessed directly from the
airbyte.get_source()
method:
- Docker sources, using the
docker_image
argument.- Yaml sources, using the
source_manifest
argument.
About Experimental Features
Experimental features may change without notice between minor versions of PyAirbyte. Although rare, they may also be entirely removed or refactored in future versions of PyAirbyte. Experimental features may also be less stable than other features, and may not be as well-tested.
You can help improve this product by reporting issues and providing feedback for improvements in our GitHub issue tracker.
1# Copyright (c) 2024 Airbyte, Inc., all rights reserved. 2"""Experimental features which may change. 3 4> **NOTE:** 5> The following "experimental" features are now "stable" and can be accessed directly from the 6`airbyte.get_source()` method: 7> - Docker sources, using the `docker_image` argument. 8> - Yaml sources, using the `source_manifest` argument. 9 10## About Experimental Features 11 12Experimental features may change without notice between minor versions of PyAirbyte. Although rare, 13they may also be entirely removed or refactored in future versions of PyAirbyte. Experimental 14features may also be less stable than other features, and may not be as well-tested. 15 16You can help improve this product by reporting issues and providing feedback for improvements in our 17[GitHub issue tracker](https://github.com/airbytehq/pyairbyte/issues). 18""" 19 20from __future__ import annotations 21 22from airbyte.sources.util import get_source 23 24 25__all__ = [ 26 "get_source", 27]
48def get_source( # noqa: PLR0913 # Too many arguments 49 name: str, 50 config: dict[str, Any] | None = None, 51 *, 52 config_change_callback: ConfigChangeCallback | None = None, 53 streams: str | list[str] | None = None, 54 version: str | None = None, 55 use_python: bool | Path | str | None = None, 56 pip_url: str | None = None, 57 local_executable: Path | str | None = None, 58 docker_image: bool | str | None = None, 59 use_host_network: bool = False, 60 source_manifest: bool | dict | Path | str | None = None, 61 install_if_missing: bool = True, 62 install_root: Path | None = None, 63) -> Source: 64 """Get a connector by name and version. 65 66 If an explicit install or execution method is requested (e.g. `local_executable`, 67 `docker_image`, `pip_url`, `source_manifest`), the connector will be executed using this method. 68 69 Otherwise, an appropriate method will be selected based on the available connector metadata: 70 1. If the connector is registered and has a YAML source manifest is available, the YAML manifest 71 will be downloaded and used to to execute the connector. 72 2. Else, if the connector is registered and has a PyPI package, it will be installed via pip. 73 3. Else, if the connector is registered and has a Docker image, and if Docker is available, it 74 will be executed using Docker. 75 76 Args: 77 name: connector name 78 config: connector config - if not provided, you need to set it later via the set_config 79 method. 80 config_change_callback: callback function to be called when the connector config changes. 81 streams: list of stream names to select for reading. If set to "*", all streams will be 82 selected. If not provided, you can set it later via the `select_streams()` or 83 `select_all_streams()` method. 84 version: connector version - if not provided, the currently installed version will be used. 85 If no version is installed, the latest available version will be used. The version can 86 also be set to "latest" to force the use of the latest available version. 87 use_python: (Optional.) Python interpreter specification: 88 - True: Use current Python interpreter. (Inferred if `pip_url` is set.) 89 - False: Use Docker instead. 90 - Path: Use interpreter at this path. 91 - str: Use specific Python version. E.g. "3.11" or "3.11.10". If the version is not yet 92 installed, it will be installed by uv. (This generally adds less than 3 seconds 93 to install times.) 94 pip_url: connector pip URL - if not provided, the pip url will be inferred from the 95 connector name. 96 local_executable: If set, the connector will be assumed to already be installed and will be 97 executed using this path or executable name. Otherwise, the connector will be installed 98 automatically in a virtual environment. 99 docker_image: If set, the connector will be executed using Docker. You can specify `True` 100 to use the default image for the connector, or you can specify a custom image name. 101 If `version` is specified and your image name does not already contain a tag 102 (e.g. `my-image:latest`), the version will be appended as a tag (e.g. `my-image:0.1.0`). 103 use_host_network: If set, along with docker_image, the connector will be executed using 104 the host network. This is useful for connectors that need to access resources on 105 the host machine, such as a local database. This parameter is ignored when 106 `docker_image` is not set. 107 source_manifest: If set, the connector will be executed based on a declarative YAML 108 source definition. This input can be `True` to attempt to auto-download a YAML spec, 109 `dict` to accept a Python dictionary as the manifest, `Path` to pull a manifest from 110 the local file system, or `str` to pull the definition from a web URL. 111 install_if_missing: Whether to install the connector if it is not available locally. This 112 parameter is ignored when `local_executable` or `source_manifest` are set. 113 install_root: (Optional.) The root directory where the virtual environment will be 114 created. If not provided, the current working directory will be used. 115 """ 116 return Source( 117 name=name, 118 config=config, 119 config_change_callback=config_change_callback, 120 streams=streams, 121 executor=get_connector_executor( 122 name=name, 123 version=version, 124 use_python=use_python, 125 pip_url=pip_url, 126 local_executable=local_executable, 127 docker_image=docker_image, 128 use_host_network=use_host_network, 129 source_manifest=source_manifest, 130 install_if_missing=install_if_missing, 131 install_root=install_root, 132 ), 133 )
Get a connector by name and version.
If an explicit install or execution method is requested (e.g. local_executable
,
docker_image
, pip_url
, source_manifest
), the connector will be executed using this method.
Otherwise, an appropriate method will be selected based on the available connector metadata:
- If the connector is registered and has a YAML source manifest is available, the YAML manifest will be downloaded and used to to execute the connector.
- Else, if the connector is registered and has a PyPI package, it will be installed via pip.
- Else, if the connector is registered and has a Docker image, and if Docker is available, it will be executed using Docker.
Arguments:
- name: connector name
- config: connector config - if not provided, you need to set it later via the set_config method.
- config_change_callback: callback function to be called when the connector config changes.
- streams: list of stream names to select for reading. If set to "*", all streams will be
selected. If not provided, you can set it later via the
select_streams()
orselect_all_streams()
method. - version: connector version - if not provided, the currently installed version will be used. If no version is installed, the latest available version will be used. The version can also be set to "latest" to force the use of the latest available version.
- use_python: (Optional.) Python interpreter specification:
- True: Use current Python interpreter. (Inferred if
pip_url
is set.) - False: Use Docker instead.
- Path: Use interpreter at this path.
- str: Use specific Python version. E.g. "3.11" or "3.11.10". If the version is not yet installed, it will be installed by uv. (This generally adds less than 3 seconds to install times.)
- True: Use current Python interpreter. (Inferred if
- pip_url: connector pip URL - if not provided, the pip url will be inferred from the connector name.
- local_executable: If set, the connector will be assumed to already be installed and will be executed using this path or executable name. Otherwise, the connector will be installed automatically in a virtual environment.
- docker_image: If set, the connector will be executed using Docker. You can specify
True
to use the default image for the connector, or you can specify a custom image name. Ifversion
is specified and your image name does not already contain a tag (e.g.my-image:latest
), the version will be appended as a tag (e.g.my-image:0.1.0
). - use_host_network: If set, along with docker_image, the connector will be executed using
the host network. This is useful for connectors that need to access resources on
the host machine, such as a local database. This parameter is ignored when
docker_image
is not set. - source_manifest: If set, the connector will be executed based on a declarative YAML
source definition. This input can be
True
to attempt to auto-download a YAML spec,dict
to accept a Python dictionary as the manifest,Path
to pull a manifest from the local file system, orstr
to pull the definition from a web URL. - install_if_missing: Whether to install the connector if it is not available locally. This
parameter is ignored when
local_executable
orsource_manifest
are set. - install_root: (Optional.) The root directory where the virtual environment will be created. If not provided, the current working directory will be used.