airbyte_cdk

Welcome to the Airbyte Python CDK!

The Airbyte Python CDK is a Python library that provides a set of tools to help you build connectors for the Airbyte platform.

Building Source Connectors

To build a source connector, you will want to refer to the following classes and modules:

Building Destination Connectors

To build a destination connector, you will want to refer to the following classes and modules:

Working with Airbyte Protocol Models

The Airbyte CDK provides a set of classes that help you work with the Airbyte protocol models:

Using the CLI (airbyte_cdk.cli)

The Airbyte CDK provides two command-line interfaces (CLIs) for interacting with the framework.

  • airbyte-cdk: This is the main CLI for the Airbyte CDK. It provides commands for building and testing connectors, as well as other utilities. See the airbyte_cdk.cli.airbyte_cdk module for more details.
  • source-declarative-manifest: This command allows you to run declarative manifests directly. See the airbyte_cdk.cli.source_declarative_manifest module for more details.

API Reference


  1# Copyright (c) 2021 Airbyte, Inc., all rights reserved.
  2"""
  3# Welcome to the Airbyte Python CDK!
  4
  5The Airbyte Python CDK is a Python library that provides a set of tools to help you build
  6connectors for the Airbyte platform.
  7
  8## Building Source Connectors
  9
 10To build a source connector, you will want to refer to
 11the following classes and modules:
 12
 13- `airbyte_cdk.sources`
 14- `airbyte_cdk.sources.concurrent_source`
 15- `airbyte_cdk.sources.config`
 16- `airbyte_cdk.sources.file_based`
 17- `airbyte_cdk.sources.streams`
 18
 19## Building Destination Connectors
 20
 21To build a destination connector, you will want to refer to
 22the following classes and modules:
 23
 24- `airbyte_cdk.destinations`
 25- `airbyte_cdk.destinations.Destination`
 26- `airbyte_cdk.destinations.vector_db_based`
 27
 28## Working with Airbyte Protocol Models
 29
 30The Airbyte CDK provides a set of classes that help you work with the Airbyte protocol models:
 31
 32- `airbyte_cdk.models.airbyte_protocol`
 33- `airbyte_cdk.models.airbyte_protocol_serializers`
 34
 35## Using the CLI (`airbyte_cdk.cli`)
 36
 37The Airbyte CDK provides two command-line interfaces (CLIs) for interacting with the framework.
 38
 39- `airbyte-cdk`: This is the main CLI for the Airbyte CDK. It provides commands for building
 40  and testing connectors, as well as other utilities. See the `airbyte_cdk.cli.airbyte_cdk` module
 41  for more details.
 42- `source-declarative-manifest`: This command allows you to run declarative manifests directly.
 43  See the `airbyte_cdk.cli.source_declarative_manifest` module for more details.
 44
 45---
 46
 47API Reference
 48
 49---
 50
 51"""
 52
 53# Warning: The below imports are not stable and will cause circular
 54# dependencies if auto-sorted with isort. Please keep them in the same order.
 55# TODO: Submodules should import from lower-level modules, rather than importing from here.
 56# Imports should also be placed in `if TYPE_CHECKING` blocks if they are only used as type
 57# hints - again, to avoid circular dependencies.
 58# Once those issues are resolved, the below can be sorted with isort.
 59import dunamai as _dunamai
 60
 61from .config_observation import (
 62    create_connector_config_control_message,
 63    emit_configuration_as_airbyte_control_message,
 64)
 65from .connector import BaseConnector, Connector
 66from .destinations import Destination
 67from .entrypoint import AirbyteEntrypoint, launch
 68from .legacy.sources.declarative.declarative_stream import DeclarativeStream
 69from .legacy.sources.declarative.incremental import DatetimeBasedCursor
 70from .logger import AirbyteLogFormatter, init_logger
 71from .models import (
 72    AdvancedAuth,
 73    AirbyteConnectionStatus,
 74    AirbyteLogMessage,
 75    AirbyteMessage,
 76    AirbyteRecordMessage,
 77    AirbyteStream,
 78    ConfiguredAirbyteCatalog,
 79    ConfiguredAirbyteStream,
 80    ConnectorSpecification,
 81    DestinationSyncMode,
 82    FailureType,
 83    Level,
 84    OAuthConfigSpecification,
 85    OrchestratorType,
 86    Status,
 87    SyncMode,
 88    Type,
 89)
 90from .sources import AbstractSource, Source
 91from .sources.concurrent_source.concurrent_source import ConcurrentSource
 92from .sources.concurrent_source.concurrent_source_adapter import ConcurrentSourceAdapter
 93from .sources.config import BaseConfig
 94from .sources.connector_state_manager import ConnectorStateManager
 95from .sources.declarative.auth import DeclarativeOauth2Authenticator
 96from .sources.declarative.auth.declarative_authenticator import DeclarativeAuthenticator, NoAuth
 97from .sources.declarative.auth.oauth import DeclarativeSingleUseRefreshTokenOauth2Authenticator
 98from .sources.declarative.auth.token import (
 99    ApiKeyAuthenticator,
100    BasicHttpAuthenticator,
101    BearerAuthenticator,
102)
103from .sources.declarative.datetime.min_max_datetime import MinMaxDatetime
104from .sources.declarative.decoders import Decoder, JsonDecoder
105from .sources.declarative.exceptions import ReadException
106from .sources.declarative.extractors import (
107    CombinedExtractor,
108    CombineMode,
109    DpathExtractor,
110    RecordSelector,
111)
112from .sources.declarative.extractors.record_extractor import RecordExtractor
113from .sources.declarative.extractors.record_filter import RecordFilter
114from .sources.declarative.interpolation import InterpolatedBoolean, InterpolatedString
115from .sources.declarative.migrations.legacy_to_per_partition_state_migration import (
116    LegacyToPerPartitionStateMigration,
117)
118from .sources.declarative.partition_routers import (
119    CartesianProductStreamSlicer,
120    SinglePartitionRouter,
121    SubstreamPartitionRouter,
122    UnionPartitionRouter,
123)
124from .sources.declarative.partition_routers.substream_partition_router import ParentStreamConfig
125from .sources.declarative.requesters import HttpRequester, Requester
126from .sources.declarative.requesters.error_handlers import BackoffStrategy
127from .sources.declarative.requesters.paginators import DefaultPaginator, PaginationStrategy
128from .sources.declarative.requesters.paginators.strategies import (
129    CursorPaginationStrategy,
130    OffsetIncrement,
131    PageIncrement,
132    StopConditionPaginationStrategyDecorator,
133)
134from .sources.declarative.requesters.request_option import RequestOption, RequestOptionType
135from .sources.declarative.requesters.request_options.default_request_options_provider import (
136    DefaultRequestOptionsProvider,
137)
138from .sources.declarative.requesters.request_options.interpolated_request_input_provider import (
139    InterpolatedRequestInputProvider,
140)
141from .sources.declarative.requesters.requester import HttpMethod
142from .sources.declarative.retrievers import SimpleRetriever
143from .sources.declarative.schema import JsonFileSchemaLoader
144from .sources.declarative.transformations.add_fields import AddedFieldDefinition, AddFields
145from .sources.declarative.transformations.transformation import RecordTransformation
146from .sources.declarative.types import FieldPointer
147from .sources.declarative.yaml_declarative_source import YamlDeclarativeSource
148from .sources.message import InMemoryMessageRepository, MessageRepository
149from .sources.source import TState
150from .sources.streams.availability_strategy import AvailabilityStrategy
151from .sources.streams.call_rate import (
152    AbstractAPIBudget,
153    CachedLimiterSession,
154    HttpAPIBudget,
155    HttpRequestMatcher,
156    LimiterSession,
157    MovingWindowCallRatePolicy,
158    Rate,
159)
160from .sources.streams.checkpoint import Cursor as LegacyCursor
161from .sources.streams.checkpoint import ResumableFullRefreshCursor
162from .sources.streams.concurrent.adapters import StreamFacade
163from .sources.streams.concurrent.cursor import (
164    ConcurrentCursor,
165    Cursor,
166    CursorField,
167    FinalStateCursor,
168)
169from .sources.streams.concurrent.state_converters.datetime_stream_state_converter import (
170    EpochValueConcurrentStreamStateConverter,
171    IsoMillisConcurrentStreamStateConverter,
172)
173from .sources.streams.core import IncrementalMixin, Stream, package_name_from_class
174from .sources.streams.http import HttpStream, HttpSubStream
175from .sources.streams.http.availability_strategy import HttpAvailabilityStrategy
176from .sources.streams.http.exceptions import (
177    BaseBackoffException,
178    DefaultBackoffException,
179    UserDefinedBackoffException,
180)
181from .sources.streams.http.rate_limiting import default_backoff_handler
182from .sources.streams.http.requests_native_auth import (
183    Oauth2Authenticator,
184    SingleUseRefreshTokenOauth2Authenticator,
185    TokenAuthenticator,
186)
187from .sources.streams.http.requests_native_auth.abstract_token import AbstractHeaderAuthenticator
188from .sources.types import Config, Record, StreamSlice
189from .sources.utils import casing
190from .sources.utils.schema_helpers import (
191    InternalConfig,
192    ResourceSchemaLoader,
193    check_config_against_spec_or_exit,
194    expand_refs,
195    split_config,
196)
197from .sources.utils.transform import TransformConfig, TypeTransformer
198from .utils import AirbyteTracedException, is_cloud_environment
199from .utils.constants import ENV_REQUEST_CACHE_PATH
200from .utils.event_timing import create_timer
201from .utils.oneof_option_config import OneOfOptionConfig
202from .utils.spec_schema_transformations import resolve_refs
203from .utils.stream_status_utils import as_airbyte_message
204
205__all__ = [
206    # Availability strategy
207    "AvailabilityStrategy",
208    "HttpAvailabilityStrategy",
209    # Checkpoint
210    "LegacyCursor",
211    "ResumableFullRefreshCursor",
212    # Concurrent
213    "ConcurrentCursor",
214    "ConcurrentSource",
215    "ConcurrentSourceAdapter",
216    "Cursor",
217    "CursorField",
218    "DEFAULT_CONCURRENCY",
219    "EpochValueConcurrentStreamStateConverter",
220    "FinalStateCursor",
221    "IsoMillisConcurrentStreamStateConverter",
222    "StreamFacade",
223    # Config observation
224    "create_connector_config_control_message",
225    "emit_configuration_as_airbyte_control_message",
226    # Connector
227    "AbstractSource",
228    "BaseConfig",
229    "BaseConnector",
230    "Connector",
231    "Destination",
232    "Source",
233    "TState",
234    # Declarative
235    "AddFields",
236    "AddedFieldDefinition",
237    "ApiKeyAuthenticator",
238    "BackoffStrategy",
239    "BasicHttpAuthenticator",
240    "BearerAuthenticator",
241    "CartesianProductStreamSlicer",
242    "CombineMode",
243    "CombinedExtractor",
244    "CursorPaginationStrategy",
245    "DatetimeBasedCursor",
246    "DeclarativeAuthenticator",
247    "DeclarativeOauth2Authenticator",
248    "DeclarativeSingleUseRefreshTokenOauth2Authenticator",
249    "DeclarativeStream",
250    "Decoder",
251    "DefaultPaginator",
252    "DefaultRequestOptionsProvider",
253    "DpathExtractor",
254    "FieldPointer",
255    "HttpMethod",
256    "HttpRequester",
257    "InterpolatedBoolean",
258    "InterpolatedRequestInputProvider",
259    "InterpolatedString",
260    "JsonDecoder",
261    "JsonFileSchemaLoader",
262    "LegacyToPerPartitionStateMigration",
263    "MinMaxDatetime",
264    "NoAuth",
265    "OffsetIncrement",
266    "PageIncrement",
267    "PaginationStrategy",
268    "ParentStreamConfig",
269    "ReadException",
270    "RecordExtractor",
271    "RecordFilter",
272    "RecordSelector",
273    "RecordTransformation",
274    "RequestOption",
275    "RequestOptionType",
276    "Requester",
277    "ResponseStatus",
278    "SimpleRetriever",
279    "SinglePartitionRouter",
280    "StopConditionPaginationStrategyDecorator",
281    "StreamSlice",
282    "SubstreamPartitionRouter",
283    "UnionPartitionRouter",
284    "YamlDeclarativeSource",
285    # Entrypoint
286    "launch",
287    "AirbyteEntrypoint",
288    # HTTP
289    "AbstractAPIBudget",
290    "AbstractHeaderAuthenticator",
291    "BaseBackoffException",
292    "CachedLimiterSession",
293    "DefaultBackoffException",
294    "default_backoff_handler",
295    "HttpAPIBudget",
296    "HttpAuthenticator",
297    "HttpRequestMatcher",
298    "HttpStream",
299    "HttpSubStream",
300    "LimiterSession",
301    "MovingWindowCallRatePolicy",
302    "MultipleTokenAuthenticator",
303    "Oauth2Authenticator",
304    "Rate",
305    "SingleUseRefreshTokenOauth2Authenticator",
306    "TokenAuthenticator",
307    "UserDefinedBackoffException",
308    # Logger
309    "AirbyteLogFormatter",
310    "init_logger",
311    # Protocol classes
312    "AirbyteStream",
313    "AirbyteConnectionStatus",
314    "AirbyteMessage",
315    "ConfiguredAirbyteCatalog",
316    "Status",
317    "Type",
318    "OrchestratorType",
319    "ConfiguredAirbyteStream",
320    "DestinationSyncMode",
321    "SyncMode",
322    "FailureType",
323    "AdvancedAuth",
324    "AirbyteLogMessage",
325    "OAuthConfigSpecification",
326    "ConnectorSpecification",
327    "Level",
328    "AirbyteRecordMessage",
329    # Repository
330    "InMemoryMessageRepository",
331    "MessageRepository",
332    # State management
333    "ConnectorStateManager",
334    # Stream
335    "IncrementalMixin",
336    "Stream",
337    "StreamData",
338    "package_name_from_class",
339    # Utils
340    "AirbyteTracedException",
341    "is_cloud_environment",
342    "casing",
343    "InternalConfig",
344    "ResourceSchemaLoader",
345    "check_config_against_spec_or_exit",
346    "split_config",
347    "TransformConfig",
348    "TypeTransformer",
349    "ENV_REQUEST_CACHE_PATH",
350    "create_timer",
351    "OneOfOptionConfig",
352    "resolve_refs",
353    "as_airbyte_message",
354    # Types
355    "Config",
356    "Record",
357    "Source",
358    "StreamSlice",
359]
360
361__version__: str
362"""Version generated by poetry dynamic versioning during publish.
363
364When running in development, dunamai will calculate a new prerelease version
365from existing git release tag info.
366"""
367
368try:
369    __version__ = _dunamai.get_version(
370        "airbyte-cdk",
371        third_choice=_dunamai.Version.from_any_vcs,
372        fallback=_dunamai.Version("0.0.0+dev"),
373    ).serialize()
374except:
375    __version__ = "0.0.0+dev"
class AvailabilityStrategy(abc.ABC):
19class AvailabilityStrategy(ABC):
20    """
21    Abstract base class for checking stream availability.
22    """
23
24    @abstractmethod
25    def check_availability(
26        self, stream: Stream, logger: logging.Logger, source: Optional["Source"] = None
27    ) -> Tuple[bool, Optional[str]]:
28        """
29        Checks stream availability.
30
31        :param stream: stream
32        :param logger: source logger
33        :param source: (optional) source
34        :return: A tuple of (boolean, str). If boolean is true, then the stream
35          is available, and no str is required. Otherwise, the stream is unavailable
36          for some reason and the str should describe what went wrong and how to
37          resolve the unavailability, if possible.
38        """
39
40    @staticmethod
41    def get_first_stream_slice(stream: Stream) -> Optional[Mapping[str, Any]]:
42        """
43        Gets the first stream_slice from a given stream's stream_slices.
44        :param stream: stream
45        :raises StopIteration: if there is no first slice to return (the stream_slices generator is empty)
46        :return: first stream slice from 'stream_slices' generator (`None` is a valid stream slice)
47        """
48        # We wrap the return output of stream_slices() because some implementations return types that are iterable,
49        # but not iterators such as lists or tuples
50        slices = iter(
51            stream.stream_slices(
52                cursor_field=stream.cursor_field,  # type: ignore[arg-type]
53                sync_mode=SyncMode.full_refresh,
54            )
55        )
56        return next(slices)
57
58    @staticmethod
59    def get_first_record_for_slice(
60        stream: Stream, stream_slice: Optional[Mapping[str, Any]]
61    ) -> StreamData:
62        """
63        Gets the first record for a stream_slice of a stream.
64
65        :param stream: stream instance from which to read records
66        :param stream_slice: stream_slice parameters for slicing the stream
67        :raises StopIteration: if there is no first record to return (the read_records generator is empty)
68        :return: StreamData containing the first record in the slice
69        """
70        # Store the original value of exit_on_rate_limit
71        original_exit_on_rate_limit = stream.exit_on_rate_limit
72
73        try:
74            # Ensure exit_on_rate_limit is safely set to True if possible
75            stream.exit_on_rate_limit = True
76
77            # We wrap the return output of read_records() because some implementations return types that are iterable,
78            # but not iterators such as lists or tuples
79            records_for_slice = iter(
80                stream.read_records(sync_mode=SyncMode.full_refresh, stream_slice=stream_slice)
81            )
82
83            return next(records_for_slice)
84        finally:
85            # Restore the original exit_on_rate_limit value
86            stream.exit_on_rate_limit = original_exit_on_rate_limit

Abstract base class for checking stream availability.

@abstractmethod
def check_availability( self, stream: Stream, logger: logging.Logger, source: Optional[Source] = None) -> Tuple[bool, Optional[str]]:
24    @abstractmethod
25    def check_availability(
26        self, stream: Stream, logger: logging.Logger, source: Optional["Source"] = None
27    ) -> Tuple[bool, Optional[str]]:
28        """
29        Checks stream availability.
30
31        :param stream: stream
32        :param logger: source logger
33        :param source: (optional) source
34        :return: A tuple of (boolean, str). If boolean is true, then the stream
35          is available, and no str is required. Otherwise, the stream is unavailable
36          for some reason and the str should describe what went wrong and how to
37          resolve the unavailability, if possible.
38        """

Checks stream availability.

Parameters
  • stream: stream
  • logger: source logger
  • source: (optional) source
Returns

A tuple of (boolean, str). If boolean is true, then the stream is available, and no str is required. Otherwise, the stream is unavailable for some reason and the str should describe what went wrong and how to resolve the unavailability, if possible.

@staticmethod
def get_first_stream_slice( stream: Stream) -> Optional[Mapping[str, Any]]:
40    @staticmethod
41    def get_first_stream_slice(stream: Stream) -> Optional[Mapping[str, Any]]:
42        """
43        Gets the first stream_slice from a given stream's stream_slices.
44        :param stream: stream
45        :raises StopIteration: if there is no first slice to return (the stream_slices generator is empty)
46        :return: first stream slice from 'stream_slices' generator (`None` is a valid stream slice)
47        """
48        # We wrap the return output of stream_slices() because some implementations return types that are iterable,
49        # but not iterators such as lists or tuples
50        slices = iter(
51            stream.stream_slices(
52                cursor_field=stream.cursor_field,  # type: ignore[arg-type]
53                sync_mode=SyncMode.full_refresh,
54            )
55        )
56        return next(slices)

Gets the first stream_slice from a given stream's stream_slices.

Parameters
  • stream: stream
Raises
  • StopIteration: if there is no first slice to return (the stream_slices generator is empty)
Returns

first stream slice from 'stream_slices' generator (None is a valid stream slice)

@staticmethod
def get_first_record_for_slice( stream: Stream, stream_slice: Optional[Mapping[str, Any]]) -> Union[Mapping[str, Any], AirbyteMessage]:
58    @staticmethod
59    def get_first_record_for_slice(
60        stream: Stream, stream_slice: Optional[Mapping[str, Any]]
61    ) -> StreamData:
62        """
63        Gets the first record for a stream_slice of a stream.
64
65        :param stream: stream instance from which to read records
66        :param stream_slice: stream_slice parameters for slicing the stream
67        :raises StopIteration: if there is no first record to return (the read_records generator is empty)
68        :return: StreamData containing the first record in the slice
69        """
70        # Store the original value of exit_on_rate_limit
71        original_exit_on_rate_limit = stream.exit_on_rate_limit
72
73        try:
74            # Ensure exit_on_rate_limit is safely set to True if possible
75            stream.exit_on_rate_limit = True
76
77            # We wrap the return output of read_records() because some implementations return types that are iterable,
78            # but not iterators such as lists or tuples
79            records_for_slice = iter(
80                stream.read_records(sync_mode=SyncMode.full_refresh, stream_slice=stream_slice)
81            )
82
83            return next(records_for_slice)
84        finally:
85            # Restore the original exit_on_rate_limit value
86            stream.exit_on_rate_limit = original_exit_on_rate_limit

Gets the first record for a stream_slice of a stream.

Parameters
  • stream: stream instance from which to read records
  • stream_slice: stream_slice parameters for slicing the stream
Raises
  • StopIteration: if there is no first record to return (the read_records generator is empty)
Returns

StreamData containing the first record in the slice

class HttpAvailabilityStrategy(airbyte_cdk.AvailabilityStrategy):
18class HttpAvailabilityStrategy(AvailabilityStrategy):
19    def check_availability(
20        self, stream: Stream, logger: logging.Logger, source: Optional["Source"] = None
21    ) -> Tuple[bool, Optional[str]]:
22        """
23        Check stream availability by attempting to read the first record of the
24        stream.
25
26        :param stream: stream
27        :param logger: source logger
28        :param source: (optional) source
29        :return: A tuple of (boolean, str). If boolean is true, then the stream
30          is available, and no str is required. Otherwise, the stream is unavailable
31          for some reason and the str should describe what went wrong and how to
32          resolve the unavailability, if possible.
33        """
34        reason: Optional[str]
35        try:
36            # Some streams need a stream slice to read records (e.g. if they have a SubstreamPartitionRouter)
37            # Streams that don't need a stream slice will return `None` as their first stream slice.
38            stream_slice = self.get_first_stream_slice(stream)
39        except StopIteration:
40            # If stream_slices has no `next()` item (Note - this is different from stream_slices returning [None]!)
41            # This can happen when a substream's `stream_slices` method does a `for record in parent_records: yield <something>`
42            # without accounting for the case in which the parent stream is empty.
43            reason = f"Cannot attempt to connect to stream {stream.name} - no stream slices were found, likely because the parent stream is empty."
44            return False, reason
45        except AirbyteTracedException as error:
46            return False, error.message
47
48        try:
49            self.get_first_record_for_slice(stream, stream_slice)
50            return True, None
51        except StopIteration:
52            logger.info(f"Successfully connected to stream {stream.name}, but got 0 records.")
53            return True, None
54        except AirbyteTracedException as error:
55            return False, error.message

Abstract base class for checking stream availability.

def check_availability( self, stream: Stream, logger: logging.Logger, source: Optional[Source] = None) -> Tuple[bool, Optional[str]]:
19    def check_availability(
20        self, stream: Stream, logger: logging.Logger, source: Optional["Source"] = None
21    ) -> Tuple[bool, Optional[str]]:
22        """
23        Check stream availability by attempting to read the first record of the
24        stream.
25
26        :param stream: stream
27        :param logger: source logger
28        :param source: (optional) source
29        :return: A tuple of (boolean, str). If boolean is true, then the stream
30          is available, and no str is required. Otherwise, the stream is unavailable
31          for some reason and the str should describe what went wrong and how to
32          resolve the unavailability, if possible.
33        """
34        reason: Optional[str]
35        try:
36            # Some streams need a stream slice to read records (e.g. if they have a SubstreamPartitionRouter)
37            # Streams that don't need a stream slice will return `None` as their first stream slice.
38            stream_slice = self.get_first_stream_slice(stream)
39        except StopIteration:
40            # If stream_slices has no `next()` item (Note - this is different from stream_slices returning [None]!)
41            # This can happen when a substream's `stream_slices` method does a `for record in parent_records: yield <something>`
42            # without accounting for the case in which the parent stream is empty.
43            reason = f"Cannot attempt to connect to stream {stream.name} - no stream slices were found, likely because the parent stream is empty."
44            return False, reason
45        except AirbyteTracedException as error:
46            return False, error.message
47
48        try:
49            self.get_first_record_for_slice(stream, stream_slice)
50            return True, None
51        except StopIteration:
52            logger.info(f"Successfully connected to stream {stream.name}, but got 0 records.")
53            return True, None
54        except AirbyteTracedException as error:
55            return False, error.message

Check stream availability by attempting to read the first record of the stream.

Parameters
  • stream: stream
  • logger: source logger
  • source: (optional) source
Returns

A tuple of (boolean, str). If boolean is true, then the stream is available, and no str is required. Otherwise, the stream is unavailable for some reason and the str should describe what went wrong and how to resolve the unavailability, if possible.

@dataclass
class ResumableFullRefreshCursor(airbyte_cdk.sources.streams.checkpoint.cursor.Cursor):
11@dataclass
12class ResumableFullRefreshCursor(Cursor):
13    """
14    Cursor that allows for the checkpointing of sync progress according to a synthetic cursor based on the pagination state
15    of the stream. Resumable full refresh syncs are only intended to retain state in between sync attempts of the same job
16    with the platform responsible for removing said state.
17    """
18
19    def __init__(self) -> None:
20        self._cursor: StreamState = {}
21
22    def get_stream_state(self) -> StreamState:
23        return self._cursor
24
25    def set_initial_state(self, stream_state: StreamState) -> None:
26        self._cursor = stream_state
27
28    def observe(self, stream_slice: StreamSlice, record: Record) -> None:
29        """
30        Resumable full refresh manages state using a page number so it does not need to update state by observing incoming records.
31        """
32        pass
33
34    def close_slice(self, stream_slice: StreamSlice, *args: Any) -> None:
35        self._cursor = stream_slice.cursor_slice
36
37    def should_be_synced(self, record: Record) -> bool:
38        """
39        Unlike date-based cursors which filter out records outside slice boundaries, resumable full refresh records exist within pages
40        that don't have filterable bounds. We should always return them.
41        """
42        return True
43
44    def select_state(self, stream_slice: Optional[StreamSlice] = None) -> Optional[StreamState]:
45        # A top-level RFR cursor only manages the state of a single partition
46        return self._cursor

Cursor that allows for the checkpointing of sync progress according to a synthetic cursor based on the pagination state of the stream. Resumable full refresh syncs are only intended to retain state in between sync attempts of the same job with the platform responsible for removing said state.

def get_stream_state(self) -> Mapping[str, Any]:
22    def get_stream_state(self) -> StreamState:
23        return self._cursor

Returns the current stream state. We would like to restrict it's usage since it does expose internal of state. As of 2023-06-14, it is used for two things:

  • Interpolation of the requests
  • Transformation of records
  • Saving the state

For the first case, we are probably stuck with exposing the stream state. For the second, we can probably expose a method that allows for emitting the state to the platform.

def set_initial_state(self, stream_state: Mapping[str, Any]) -> None:
25    def set_initial_state(self, stream_state: StreamState) -> None:
26        self._cursor = stream_state

Cursors are not initialized with their state. As state is needed in order to function properly, this method should be called before calling anything else

Parameters
  • stream_state: The state of the stream as returned by get_stream_state
def observe( self, stream_slice: StreamSlice, record: Record) -> None:
28    def observe(self, stream_slice: StreamSlice, record: Record) -> None:
29        """
30        Resumable full refresh manages state using a page number so it does not need to update state by observing incoming records.
31        """
32        pass

Resumable full refresh manages state using a page number so it does not need to update state by observing incoming records.

def close_slice( self, stream_slice: StreamSlice, *args: Any) -> None:
34    def close_slice(self, stream_slice: StreamSlice, *args: Any) -> None:
35        self._cursor = stream_slice.cursor_slice

Update state based on the stream slice. Note that stream_slice.cursor_slice and most_recent_record.associated_slice are expected to be the same but we make it explicit here that stream_slice should be leveraged to update the state. We do not pass in the latest record, since cursor instances should maintain the relevant internal state on their own.

Parameters
  • stream_slice: slice to close
def should_be_synced(self, record: Record) -> bool:
37    def should_be_synced(self, record: Record) -> bool:
38        """
39        Unlike date-based cursors which filter out records outside slice boundaries, resumable full refresh records exist within pages
40        that don't have filterable bounds. We should always return them.
41        """
42        return True

Unlike date-based cursors which filter out records outside slice boundaries, resumable full refresh records exist within pages that don't have filterable bounds. We should always return them.

def select_state( self, stream_slice: Optional[StreamSlice] = None) -> Optional[Mapping[str, Any]]:
44    def select_state(self, stream_slice: Optional[StreamSlice] = None) -> Optional[StreamState]:
45        # A top-level RFR cursor only manages the state of a single partition
46        return self._cursor

Get the state value of a specific stream_slice. For incremental or resumable full refresh cursors which only manage state in a single dimension this is the entire state object. For per-partition cursors used by substreams, this returns the state of a specific parent delineated by the incoming slice's partition object.

class ConcurrentCursor(airbyte_cdk.Cursor):
179class ConcurrentCursor(Cursor):
180    _START_BOUNDARY = 0
181    _END_BOUNDARY = 1
182
183    def copy_without_state(self) -> "ConcurrentCursor":
184        return self.__class__(
185            stream_name=self._stream_name,
186            stream_namespace=self._stream_namespace,
187            stream_state={},
188            message_repository=NoopMessageRepository(),
189            connector_state_manager=ConnectorStateManager(),
190            connector_state_converter=self._connector_state_converter,
191            cursor_field=self._cursor_field,
192            slice_boundary_fields=self._slice_boundary_fields,
193            start=self._start,
194            end_provider=self._end_provider,
195            lookback_window=self._lookback_window,
196            slice_range=self._slice_range,
197            cursor_granularity=self._cursor_granularity,
198            clamping_strategy=self._clamping_strategy,
199            is_compare_strictly=self._is_compare_strictly,
200        )
201
202    def __init__(
203        self,
204        stream_name: str,
205        stream_namespace: Optional[str],
206        stream_state: Any,
207        message_repository: MessageRepository,
208        connector_state_manager: ConnectorStateManager,
209        connector_state_converter: AbstractStreamStateConverter,
210        cursor_field: CursorField,
211        slice_boundary_fields: Optional[Tuple[str, str]],
212        start: Optional[CursorValueType],
213        end_provider: Callable[[], CursorValueType],
214        lookback_window: Optional[GapType] = None,
215        slice_range: Optional[GapType] = None,
216        cursor_granularity: Optional[GapType] = None,
217        clamping_strategy: ClampingStrategy = NoClamping(),
218        is_compare_strictly: bool = False,
219    ) -> None:
220        self._stream_name = stream_name
221        self._stream_namespace = stream_namespace
222        self._message_repository = message_repository
223        self._connector_state_converter = connector_state_converter
224        self._connector_state_manager = connector_state_manager
225        self._cursor_field = cursor_field
226        # To see some example where the slice boundaries might not be defined, check https://github.com/airbytehq/airbyte/blob/1ce84d6396e446e1ac2377362446e3fb94509461/airbyte-integrations/connectors/source-stripe/source_stripe/streams.py#L363-L379
227        self._slice_boundary_fields = slice_boundary_fields
228        self._start = start
229        self._end_provider = end_provider
230        self.start, self._concurrent_state = self._get_concurrent_state(stream_state)
231        self._lookback_window = lookback_window
232        self._slice_range = slice_range
233        self._most_recent_cursor_value_per_partition: MutableMapping[
234            Union[StreamSlice, Mapping[str, Any], None], Any
235        ] = {}
236        self._has_closed_at_least_one_slice = False
237        self._cursor_granularity = cursor_granularity
238        # Flag to track if the logger has been triggered (per stream)
239        self._should_be_synced_logger_triggered = False
240        self._clamping_strategy = clamping_strategy
241        self._is_compare_strictly = is_compare_strictly
242        self._is_ascending_order = True
243
244        # A lock is required when closing a partition because updating the cursor's concurrent_state is
245        # not thread safe. When multiple partitions are being closed by the cursor at the same time, it is
246        # possible for one partition to update concurrent_state after a second partition has already read
247        # the previous state. This can lead to the second partition overwriting the previous one's state.
248        self._lock = threading.Lock()
249
250    @property
251    def state(self) -> MutableMapping[str, Any]:
252        return self._connector_state_converter.convert_to_state_message(
253            self.cursor_field, self._concurrent_state
254        )
255
256    @property
257    def cursor_field(self) -> CursorField:
258        return self._cursor_field
259
260    @property
261    def _slice_boundary_fields_wrapper(self) -> Tuple[str, str]:
262        return (
263            self._slice_boundary_fields
264            if self._slice_boundary_fields
265            else (
266                self._connector_state_converter.START_KEY,
267                self._connector_state_converter.END_KEY,
268            )
269        )
270
271    def _get_concurrent_state(
272        self, state: MutableMapping[str, Any]
273    ) -> Tuple[CursorValueType, MutableMapping[str, Any]]:
274        if self._connector_state_converter.is_state_message_compatible(state):
275            partitioned_state = self._connector_state_converter.deserialize(state)
276            slices_from_partitioned_state = partitioned_state.get("slices", [])
277
278            value_from_partitioned_state = None
279            if slices_from_partitioned_state:
280                # We assume here that the slices have been already merged
281                first_slice = slices_from_partitioned_state[0]
282                value_from_partitioned_state = (
283                    first_slice[self._connector_state_converter.MOST_RECENT_RECORD_KEY]
284                    if self._connector_state_converter.MOST_RECENT_RECORD_KEY in first_slice
285                    else first_slice[self._connector_state_converter.END_KEY]
286                )
287            return (
288                value_from_partitioned_state
289                or self._start
290                or self._connector_state_converter.zero_value,
291                partitioned_state,
292            )
293        return self._connector_state_converter.convert_from_sequential_state(
294            self._cursor_field, state, self._start
295        )
296
297    def observe(self, record: Record) -> None:
298        # Because observe writes to the most_recent_cursor_value_per_partition mapping,
299        # it is not thread-safe. However, this shouldn't lead to concurrency issues because
300        # observe() is only invoked by PartitionReader.process_partition(). Since the map is
301        # broken down according to partition, concurrent threads processing only read/write
302        # from different keys which avoids any conflicts.
303        #
304        # If we were to add thread safety, we should implement a lock per-partition
305        # which is instantiated during stream_slices()
306        most_recent_cursor_value = self._most_recent_cursor_value_per_partition.get(
307            record.associated_slice
308        )
309        try:
310            cursor_value = self._extract_cursor_value(record)
311
312            if most_recent_cursor_value is None or most_recent_cursor_value < cursor_value:
313                self._most_recent_cursor_value_per_partition[record.associated_slice] = cursor_value
314            elif most_recent_cursor_value > cursor_value:
315                self._is_ascending_order = False
316        except ValueError:
317            self._log_for_record_without_cursor_value()
318
319    def _extract_cursor_value(self, record: Record) -> Any:
320        return self._connector_state_converter.parse_value(self._cursor_field.extract_value(record))
321
322    def close_partition(self, partition: Partition) -> None:
323        with self._lock:
324            slice_count_before = len(self._concurrent_state.get("slices", []))
325            self._add_slice_to_state(partition)
326            if slice_count_before < len(
327                self._concurrent_state["slices"]
328            ):  # only emit if at least one slice has been processed
329                self._merge_partitions()
330                self._emit_state_message()
331        self._has_closed_at_least_one_slice = True
332
333    def _add_slice_to_state(self, partition: Partition) -> None:
334        most_recent_cursor_value = self._most_recent_cursor_value_per_partition.get(
335            partition.to_slice()
336        )
337
338        if self._slice_boundary_fields:
339            if "slices" not in self._concurrent_state:
340                raise RuntimeError(
341                    f"The state for stream {self._stream_name} should have at least one slice to delineate the sync start time, but no slices are present. This is unexpected. Please contact Support."
342                )
343            self._concurrent_state["slices"].append(
344                {
345                    self._connector_state_converter.START_KEY: self._extract_from_slice(
346                        partition, self._slice_boundary_fields[self._START_BOUNDARY]
347                    ),
348                    self._connector_state_converter.END_KEY: self._extract_from_slice(
349                        partition, self._slice_boundary_fields[self._END_BOUNDARY]
350                    ),
351                    self._connector_state_converter.MOST_RECENT_RECORD_KEY: most_recent_cursor_value,
352                }
353            )
354        elif most_recent_cursor_value:
355            if self._has_closed_at_least_one_slice:
356                # If we track state value using records cursor field, we can only do that if there is one partition. This is because we save
357                # the state every time we close a partition. We assume that if there are multiple slices, they need to be providing
358                # boundaries. There are cases where partitions could not have boundaries:
359                # * The cursor should be per-partition
360                # * The stream state is actually the parent stream state
361                # There might be other cases not listed above. Those are not supported today hence the stream should not use this cursor for
362                # state management. For the specific user that was affected with this issue, we need to:
363                # * Fix state tracking (which is currently broken)
364                # * Make the new version available
365                # * (Probably) ask the user to reset the stream to avoid data loss
366                raise ValueError(
367                    "Given that slice_boundary_fields is not defined and that per-partition state is not supported, only one slice is "
368                    "expected. Please contact the Airbyte team."
369                )
370
371            self._concurrent_state["slices"].append(
372                {
373                    self._connector_state_converter.START_KEY: self.start,
374                    self._connector_state_converter.END_KEY: most_recent_cursor_value,
375                    self._connector_state_converter.MOST_RECENT_RECORD_KEY: most_recent_cursor_value,
376                }
377            )
378
379    def _emit_state_message(self) -> None:
380        self._connector_state_manager.update_state_for_stream(
381            self._stream_name,
382            self._stream_namespace,
383            self.state,
384        )
385        state_message = self._connector_state_manager.create_state_message(
386            self._stream_name, self._stream_namespace
387        )
388        self._message_repository.emit_message(state_message)
389
390    def _merge_partitions(self) -> None:
391        self._concurrent_state["slices"] = self._connector_state_converter.merge_intervals(
392            self._concurrent_state["slices"]
393        )
394
395    def _extract_from_slice(self, partition: Partition, key: str) -> CursorValueType:
396        try:
397            _slice = partition.to_slice()
398            if not _slice:
399                raise KeyError(f"Could not find key `{key}` in empty slice")
400            return self._connector_state_converter.parse_value(_slice[key])  # type: ignore  # we expect the devs to specify a key that would return a CursorValueType
401        except KeyError as exception:
402            raise KeyError(
403                f"Partition is expected to have key `{key}` but could not be found"
404            ) from exception
405
406    def ensure_at_least_one_state_emitted(self) -> None:
407        """
408        The platform expect to have at least one state message on successful syncs. Hence, whatever happens, we expect this method to be
409        called.
410        """
411        self._emit_state_message()
412
413    def stream_slices(self) -> Iterable[StreamSlice]:
414        """
415        Generating slices based on a few parameters:
416        * lookback_window: Buffer to remove from END_KEY of the highest slice
417        * slice_range: Max difference between two slices. If the difference between two slices is greater, multiple slices will be created
418        * start: `_split_per_slice_range` will clip any value to `self._start which means that:
419          * if upper is less than self._start, no slices will be generated
420          * if lower is less than self._start, self._start will be used as the lower boundary (lookback_window will not be considered in that case)
421
422        Note that the slices will overlap at their boundaries. We therefore expect to have at least the lower or the upper boundary to be
423        inclusive in the API that is queried.
424        """
425        self._merge_partitions()
426
427        if self._start is not None and self._is_start_before_first_slice():
428            yield from self._split_per_slice_range(
429                self._start,
430                self._concurrent_state["slices"][0][self._connector_state_converter.START_KEY],
431                False,
432            )
433
434        if len(self._concurrent_state["slices"]) == 1:
435            yield from self._split_per_slice_range(
436                self._calculate_lower_boundary_of_last_slice(
437                    self._concurrent_state["slices"][0][self._connector_state_converter.END_KEY]
438                ),
439                self._end_provider(),
440                True,
441            )
442        elif len(self._concurrent_state["slices"]) > 1:
443            for i in range(len(self._concurrent_state["slices"]) - 1):
444                if self._cursor_granularity:
445                    yield from self._split_per_slice_range(
446                        self._concurrent_state["slices"][i][self._connector_state_converter.END_KEY]
447                        + self._cursor_granularity,
448                        self._concurrent_state["slices"][i + 1][
449                            self._connector_state_converter.START_KEY
450                        ],
451                        False,
452                    )
453                else:
454                    yield from self._split_per_slice_range(
455                        self._concurrent_state["slices"][i][
456                            self._connector_state_converter.END_KEY
457                        ],
458                        self._concurrent_state["slices"][i + 1][
459                            self._connector_state_converter.START_KEY
460                        ],
461                        False,
462                    )
463            yield from self._split_per_slice_range(
464                self._calculate_lower_boundary_of_last_slice(
465                    self._concurrent_state["slices"][-1][self._connector_state_converter.END_KEY]
466                ),
467                self._end_provider(),
468                True,
469            )
470        else:
471            raise ValueError("Expected at least one slice")
472
473    def _is_start_before_first_slice(self) -> bool:
474        return (
475            self._start is not None
476            and self._start
477            < self._concurrent_state["slices"][0][self._connector_state_converter.START_KEY]
478        )
479
480    def _calculate_lower_boundary_of_last_slice(
481        self, lower_boundary: CursorValueType
482    ) -> CursorValueType:
483        if self._lookback_window:
484            return lower_boundary - self._lookback_window
485        return lower_boundary
486
487    def _split_per_slice_range(
488        self, lower: CursorValueType, upper: CursorValueType, upper_is_end: bool
489    ) -> Iterable[StreamSlice]:
490        if lower > upper or (lower == upper and not self._cursor_granularity):
491            return
492
493        if self._start and upper < self._start:
494            return
495
496        lower = max(lower, self._start) if self._start else lower
497        # Clamp first, else a step shorter than the clamping period collapses the first window
498        lower = self._clamping_strategy.clamp(lower)
499        # With a granularity the last window includes upper, so lower == upper is one unit to read,
500        # unless the API rejects start == end. In a gap, the next slice already reads upper.
501        if lower > upper or (lower == upper and (self._is_compare_strictly or not upper_is_end)):
502            return
503        if not self._slice_range or self._evaluate_upper_safely(lower, self._slice_range) >= upper:
504            # With a granularity upper is the last unit to read: clamping it would read past the end
505            clamped_upper = (
506                upper
507                if upper_is_end and self._cursor_granularity
508                else self._clamping_strategy.clamp(upper)
509            )
510            start_value, end_value = (
511                (lower, clamped_upper - self._cursor_granularity)
512                if self._cursor_granularity and not upper_is_end
513                else (lower, clamped_upper)
514            )
515            yield StreamSlice(
516                partition={},
517                cursor_slice={
518                    self._slice_boundary_fields_wrapper[
519                        self._START_BOUNDARY
520                    ]: self._connector_state_converter.output_format(start_value),
521                    self._slice_boundary_fields_wrapper[
522                        self._END_BOUNDARY
523                    ]: self._connector_state_converter.output_format(end_value),
524                },
525            )
526        else:
527            stop_processing = False
528            current_lower_boundary = lower
529            while not stop_processing:
530                current_upper_boundary = min(
531                    self._evaluate_upper_safely(current_lower_boundary, self._slice_range), upper
532                )
533                clamped_upper = (
534                    self._clamping_strategy.clamp(current_upper_boundary)
535                    if current_upper_boundary != upper
536                    else current_upper_boundary
537                )
538                # A boundary clamped onto upper makes this the last window, else upper is dropped
539                has_reached_upper_boundary = (
540                    current_upper_boundary >= upper or clamped_upper == upper
541                )
542                clamped_lower = self._clamping_strategy.clamp(current_lower_boundary)
543                if clamped_lower >= clamped_upper:
544                    # clamping collapsed both values which means that it is time to stop processing
545                    # FIXME should this be replace by proper end_provider
546                    break
547                start_value, end_value = (
548                    (clamped_lower, clamped_upper - self._cursor_granularity)
549                    if self._cursor_granularity
550                    and (not upper_is_end or not has_reached_upper_boundary)
551                    else (clamped_lower, clamped_upper)
552                )
553                yield StreamSlice(
554                    partition={},
555                    cursor_slice={
556                        self._slice_boundary_fields_wrapper[
557                            self._START_BOUNDARY
558                        ]: self._connector_state_converter.output_format(start_value),
559                        self._slice_boundary_fields_wrapper[
560                            self._END_BOUNDARY
561                        ]: self._connector_state_converter.output_format(end_value),
562                    },
563                )
564                current_lower_boundary = clamped_upper
565                if has_reached_upper_boundary:
566                    stop_processing = True
567
568    def _evaluate_upper_safely(self, lower: CursorValueType, step: GapType) -> CursorValueType:
569        """
570        Given that we set the default step at datetime.timedelta.max, we will generate an OverflowError when evaluating the next start_date
571        This method assumes that users would never enter a step that would generate an overflow. Given that would be the case, the code
572        would have broken anyway.
573        """
574        try:
575            return lower + step
576        except OverflowError:
577            return self._end_provider()
578
579    def should_be_synced(self, record: Record) -> bool:
580        """
581        Determines if a record should be synced based on its cursor value.
582        :param record: The record to evaluate
583
584        :return: True if the record's cursor value falls within the sync boundaries
585        """
586        try:
587            record_cursor_value: CursorValueType = self._extract_cursor_value(record)
588        except ValueError:
589            self._log_for_record_without_cursor_value()
590            return True
591        return self._lowest_cursor_value_to_sync() <= record_cursor_value <= self._end_provider()
592
593    def _lowest_cursor_value_to_sync(self) -> CursorValueType:
594        # Keep the records the lookback window re-reads, never below the configured start. If the state is below the
595        # configured start, keep records from the state rather than raising the floor to the configured start.
596        if not self._lookback_window:
597            return self.start
598        try:
599            lower = self.start - self._lookback_window
600        except OverflowError:
601            return self.start
602        return max(lower, min(self.start, self._start)) if self._start else lower
603
604    def _log_for_record_without_cursor_value(self) -> None:
605        if not self._should_be_synced_logger_triggered:
606            LOGGER.warning(
607                f"Could not find cursor field `{self.cursor_field.cursor_field_key}` in record for stream {self._stream_name}. The incremental sync will assume it needs to be synced"
608            )
609            self._should_be_synced_logger_triggered = True
610
611    def reduce_slice_range(self, stream_slice: StreamSlice) -> StreamSlice:
612        # In theory, we might be more flexible here meaning that it doesn't need to be in ascending order but it just
613        # needs to be ordered. For now though, we will only support ascending order.
614        if not self._is_ascending_order:
615            LOGGER.warning(
616                "Attempting to reduce slice while records are not returned in incremental order might lead to missing records"
617            )
618
619        if stream_slice in self._most_recent_cursor_value_per_partition:
620            return StreamSlice(
621                partition=stream_slice.partition,
622                cursor_slice={
623                    self._slice_boundary_fields_wrapper[
624                        self._START_BOUNDARY
625                    ]: self._connector_state_converter.output_format(
626                        self._most_recent_cursor_value_per_partition[stream_slice]
627                    ),
628                    self._slice_boundary_fields_wrapper[
629                        self._END_BOUNDARY
630                    ]: stream_slice.cursor_slice[
631                        self._slice_boundary_fields_wrapper[self._END_BOUNDARY]
632                    ],
633                },
634                extra_fields=stream_slice.extra_fields,
635            )
636        else:
637            return stream_slice
638
639    def split_request_window(
640        self, stream_slice: StreamSlice, min_split_window: Optional[datetime.timedelta] = None
641    ) -> Optional[List[StreamSlice]]:
642        """
643        Split `stream_slice` in half along this cursor's own boundary fields, granularity, and output format.
644
645        Returns `None` when the slice cannot be split any further: either it spans at most one
646        `cursor_granularity` unit already (the floor), it is already at or below `min_split_window` (a
647        connector-configured floor independent of and typically looser than `cursor_granularity`), the
648        boundaries are reversed, or no `cursor_granularity` was configured at all (there is then no way to know
649        the smallest addressable unit, or how to keep two children from overlapping at their shared edge).
650        Raises `AirbyteTracedException` instead if a boundary is missing or unparseable: that is a bug in how
651        the slice was produced, not the API rejecting a window, so it must not be reported as one. This is the
652        implementation `SimpleRetriever.request_window_splitter` binds to (see
653        `airbyte_cdk.sources.declarative.retrievers.simple_retriever`) that reuses the same
654        parsing/formatting/comparison operations `_split_per_slice_range` already relies on, so it works for
655        any `CursorValueType`/`GapType` pair this cursor was built with (datetime/timedelta, or int/int),
656        without a second, duplicate implementation of that logic living outside the cursor.
657        """
658        if not self._cursor_granularity:
659            return None
660
661        start_field, end_field = self._slice_boundary_fields_wrapper
662        try:
663            start_value = self._connector_state_converter.parse_value(
664                stream_slice.cursor_slice[start_field]
665            )
666            end_value = self._connector_state_converter.parse_value(
667                stream_slice.cursor_slice[end_field]
668            )
669        except (KeyError, ValueError, TypeError) as exception:
670            # A missing or unparseable boundary means this slice was produced (or passed in) wrong - a bug in
671            # slice generation, not something the API did. Raising here, rather than returning `None` (which
672            # `SimpleRetriever` reports as "the API kept rejecting..."), keeps that distinction visible instead
673            # of blaming the API for our own state.
674            raise AirbyteTracedException(
675                internal_message=(
676                    f"Cannot split {stream_slice}: boundary field {start_field!r} or {end_field!r} is missing "
677                    f"or not parseable: {exception}"
678                ),
679                message=(
680                    f"Stream {self._stream_name} produced a request window with a missing or invalid "
681                    f"{start_field!r}/{end_field!r} boundary and could not split it further."
682                ),
683                failure_type=FailureType.system_error,
684            ) from exception
685
686        if start_value >= end_value:
687            # Reversed or degenerate boundaries: nothing safe to split.
688            return None
689
690        span = end_value - start_value  # type: ignore[operator]  # concretely a GapType (timedelta/int) at runtime
691        if min_split_window is not None and span <= min_split_window:  # type: ignore[operator]
692            return None
693
694        total_units = span // self._cursor_granularity + 1  # type: ignore[operator]  # inclusive count of granularity units
695        if total_units <= 1:
696            # Already at the minimum granularity and still rejected; nothing left to split.
697            return None
698
699        half_units = total_units // 2
700        first_end = (
701            start_value + (half_units * self._cursor_granularity) - self._cursor_granularity  # type: ignore[operator]
702        )
703        second_start = first_end + self._cursor_granularity
704
705        if not (start_value <= first_end < second_start <= end_value):
706            # No-progress guard, independent of the granularity check above: refuse to return children that
707            # are not strictly ordered and non-degenerate, rather than risk an identical or overlapping split.
708            return None
709
710        return [
711            StreamSlice(
712                partition=stream_slice.partition,
713                cursor_slice={
714                    start_field: self._connector_state_converter.output_format(start_value),
715                    end_field: self._connector_state_converter.output_format(first_end),
716                },
717                extra_fields=stream_slice.extra_fields,
718            ),
719            StreamSlice(
720                partition=stream_slice.partition,
721                cursor_slice={
722                    start_field: self._connector_state_converter.output_format(second_start),
723                    end_field: self._connector_state_converter.output_format(end_value),
724                },
725                extra_fields=stream_slice.extra_fields,
726            ),
727        ]
728
729    def get_cursor_datetime_from_state(
730        self, stream_state: Mapping[str, Any]
731    ) -> datetime.datetime | None:
732        """Extract and parse the cursor datetime from the given stream state.
733
734        For concurrent cursors, the state can be in two formats:
735        1. Sequential/legacy format: {cursor_field: cursor_value}
736        2. Concurrent format: {state_type: "date-range", slices: [...]}
737
738        Returns the cursor datetime if present and parseable, otherwise returns None.
739        """
740        # Check if state is in concurrent format (need to convert to dict for type compatibility)
741        mutable_state: MutableMapping[str, Any] = dict(stream_state)
742        if self._connector_state_converter.is_state_message_compatible(mutable_state):
743            slices = stream_state.get("slices", [])
744            if not slices:
745                return None
746            # Get the most recent cursor value from the first slice (after merging)
747            first_slice = slices[0]
748            cursor_value = first_slice.get(
749                self._connector_state_converter.MOST_RECENT_RECORD_KEY
750            ) or first_slice.get(self._connector_state_converter.END_KEY)
751            if not cursor_value:
752                return None
753            try:
754                parsed_value = self._connector_state_converter.parse_value(cursor_value)
755                if isinstance(parsed_value, datetime.datetime):
756                    return parsed_value
757                return None
758            except (ValueError, TypeError):
759                return None
760
761        # Sequential/legacy format: {cursor_field: cursor_value}
762        cursor_value = stream_state.get(self._cursor_field.cursor_field_key)
763        if not cursor_value:
764            return None
765        try:
766            parsed_value = self._connector_state_converter.parse_value(cursor_value)
767            if isinstance(parsed_value, datetime.datetime):
768                return parsed_value
769            return None
770        except (ValueError, TypeError):
771            return None

Slices the stream into chunks that can be fetched independently. Slices enable state checkpointing and data retrieval parallelization.

ConcurrentCursor( stream_name: str, stream_namespace: Optional[str], stream_state: Any, message_repository: MessageRepository, connector_state_manager: ConnectorStateManager, connector_state_converter: airbyte_cdk.sources.streams.concurrent.state_converters.abstract_stream_state_converter.AbstractStreamStateConverter, cursor_field: CursorField, slice_boundary_fields: Optional[Tuple[str, str]], start: Optional[airbyte_cdk.sources.streams.concurrent.cursor_types.CursorValueType], end_provider: Callable[[], airbyte_cdk.sources.streams.concurrent.cursor_types.CursorValueType], lookback_window: Optional[airbyte_cdk.sources.streams.concurrent.cursor_types.GapType] = None, slice_range: Optional[airbyte_cdk.sources.streams.concurrent.cursor_types.GapType] = None, cursor_granularity: Optional[airbyte_cdk.sources.streams.concurrent.cursor_types.GapType] = None, clamping_strategy: airbyte_cdk.sources.streams.concurrent.clamping.ClampingStrategy = <airbyte_cdk.sources.streams.concurrent.clamping.NoClamping object>, is_compare_strictly: bool = False)
202    def __init__(
203        self,
204        stream_name: str,
205        stream_namespace: Optional[str],
206        stream_state: Any,
207        message_repository: MessageRepository,
208        connector_state_manager: ConnectorStateManager,
209        connector_state_converter: AbstractStreamStateConverter,
210        cursor_field: CursorField,
211        slice_boundary_fields: Optional[Tuple[str, str]],
212        start: Optional[CursorValueType],
213        end_provider: Callable[[], CursorValueType],
214        lookback_window: Optional[GapType] = None,
215        slice_range: Optional[GapType] = None,
216        cursor_granularity: Optional[GapType] = None,
217        clamping_strategy: ClampingStrategy = NoClamping(),
218        is_compare_strictly: bool = False,
219    ) -> None:
220        self._stream_name = stream_name
221        self._stream_namespace = stream_namespace
222        self._message_repository = message_repository
223        self._connector_state_converter = connector_state_converter
224        self._connector_state_manager = connector_state_manager
225        self._cursor_field = cursor_field
226        # To see some example where the slice boundaries might not be defined, check https://github.com/airbytehq/airbyte/blob/1ce84d6396e446e1ac2377362446e3fb94509461/airbyte-integrations/connectors/source-stripe/source_stripe/streams.py#L363-L379
227        self._slice_boundary_fields = slice_boundary_fields
228        self._start = start
229        self._end_provider = end_provider
230        self.start, self._concurrent_state = self._get_concurrent_state(stream_state)
231        self._lookback_window = lookback_window
232        self._slice_range = slice_range
233        self._most_recent_cursor_value_per_partition: MutableMapping[
234            Union[StreamSlice, Mapping[str, Any], None], Any
235        ] = {}
236        self._has_closed_at_least_one_slice = False
237        self._cursor_granularity = cursor_granularity
238        # Flag to track if the logger has been triggered (per stream)
239        self._should_be_synced_logger_triggered = False
240        self._clamping_strategy = clamping_strategy
241        self._is_compare_strictly = is_compare_strictly
242        self._is_ascending_order = True
243
244        # A lock is required when closing a partition because updating the cursor's concurrent_state is
245        # not thread safe. When multiple partitions are being closed by the cursor at the same time, it is
246        # possible for one partition to update concurrent_state after a second partition has already read
247        # the previous state. This can lead to the second partition overwriting the previous one's state.
248        self._lock = threading.Lock()
def copy_without_state(self) -> ConcurrentCursor:
183    def copy_without_state(self) -> "ConcurrentCursor":
184        return self.__class__(
185            stream_name=self._stream_name,
186            stream_namespace=self._stream_namespace,
187            stream_state={},
188            message_repository=NoopMessageRepository(),
189            connector_state_manager=ConnectorStateManager(),
190            connector_state_converter=self._connector_state_converter,
191            cursor_field=self._cursor_field,
192            slice_boundary_fields=self._slice_boundary_fields,
193            start=self._start,
194            end_provider=self._end_provider,
195            lookback_window=self._lookback_window,
196            slice_range=self._slice_range,
197            cursor_granularity=self._cursor_granularity,
198            clamping_strategy=self._clamping_strategy,
199            is_compare_strictly=self._is_compare_strictly,
200        )
state: MutableMapping[str, Any]
250    @property
251    def state(self) -> MutableMapping[str, Any]:
252        return self._connector_state_converter.convert_to_state_message(
253            self.cursor_field, self._concurrent_state
254        )
cursor_field: CursorField
256    @property
257    def cursor_field(self) -> CursorField:
258        return self._cursor_field
def observe(self, record: Record) -> None:
297    def observe(self, record: Record) -> None:
298        # Because observe writes to the most_recent_cursor_value_per_partition mapping,
299        # it is not thread-safe. However, this shouldn't lead to concurrency issues because
300        # observe() is only invoked by PartitionReader.process_partition(). Since the map is
301        # broken down according to partition, concurrent threads processing only read/write
302        # from different keys which avoids any conflicts.
303        #
304        # If we were to add thread safety, we should implement a lock per-partition
305        # which is instantiated during stream_slices()
306        most_recent_cursor_value = self._most_recent_cursor_value_per_partition.get(
307            record.associated_slice
308        )
309        try:
310            cursor_value = self._extract_cursor_value(record)
311
312            if most_recent_cursor_value is None or most_recent_cursor_value < cursor_value:
313                self._most_recent_cursor_value_per_partition[record.associated_slice] = cursor_value
314            elif most_recent_cursor_value > cursor_value:
315                self._is_ascending_order = False
316        except ValueError:
317            self._log_for_record_without_cursor_value()

Indicate to the cursor that the record has been emitted

def close_partition( self, partition: airbyte_cdk.sources.streams.concurrent.partitions.partition.Partition) -> None:
322    def close_partition(self, partition: Partition) -> None:
323        with self._lock:
324            slice_count_before = len(self._concurrent_state.get("slices", []))
325            self._add_slice_to_state(partition)
326            if slice_count_before < len(
327                self._concurrent_state["slices"]
328            ):  # only emit if at least one slice has been processed
329                self._merge_partitions()
330                self._emit_state_message()
331        self._has_closed_at_least_one_slice = True

Indicate to the cursor that the partition has been successfully processed

def ensure_at_least_one_state_emitted(self) -> None:
406    def ensure_at_least_one_state_emitted(self) -> None:
407        """
408        The platform expect to have at least one state message on successful syncs. Hence, whatever happens, we expect this method to be
409        called.
410        """
411        self._emit_state_message()

The platform expect to have at least one state message on successful syncs. Hence, whatever happens, we expect this method to be called.

def stream_slices(self) -> Iterable[StreamSlice]:
413    def stream_slices(self) -> Iterable[StreamSlice]:
414        """
415        Generating slices based on a few parameters:
416        * lookback_window: Buffer to remove from END_KEY of the highest slice
417        * slice_range: Max difference between two slices. If the difference between two slices is greater, multiple slices will be created
418        * start: `_split_per_slice_range` will clip any value to `self._start which means that:
419          * if upper is less than self._start, no slices will be generated
420          * if lower is less than self._start, self._start will be used as the lower boundary (lookback_window will not be considered in that case)
421
422        Note that the slices will overlap at their boundaries. We therefore expect to have at least the lower or the upper boundary to be
423        inclusive in the API that is queried.
424        """
425        self._merge_partitions()
426
427        if self._start is not None and self._is_start_before_first_slice():
428            yield from self._split_per_slice_range(
429                self._start,
430                self._concurrent_state["slices"][0][self._connector_state_converter.START_KEY],
431                False,
432            )
433
434        if len(self._concurrent_state["slices"]) == 1:
435            yield from self._split_per_slice_range(
436                self._calculate_lower_boundary_of_last_slice(
437                    self._concurrent_state["slices"][0][self._connector_state_converter.END_KEY]
438                ),
439                self._end_provider(),
440                True,
441            )
442        elif len(self._concurrent_state["slices"]) > 1:
443            for i in range(len(self._concurrent_state["slices"]) - 1):
444                if self._cursor_granularity:
445                    yield from self._split_per_slice_range(
446                        self._concurrent_state["slices"][i][self._connector_state_converter.END_KEY]
447                        + self._cursor_granularity,
448                        self._concurrent_state["slices"][i + 1][
449                            self._connector_state_converter.START_KEY
450                        ],
451                        False,
452                    )
453                else:
454                    yield from self._split_per_slice_range(
455                        self._concurrent_state["slices"][i][
456                            self._connector_state_converter.END_KEY
457                        ],
458                        self._concurrent_state["slices"][i + 1][
459                            self._connector_state_converter.START_KEY
460                        ],
461                        False,
462                    )
463            yield from self._split_per_slice_range(
464                self._calculate_lower_boundary_of_last_slice(
465                    self._concurrent_state["slices"][-1][self._connector_state_converter.END_KEY]
466                ),
467                self._end_provider(),
468                True,
469            )
470        else:
471            raise ValueError("Expected at least one slice")

Generating slices based on a few parameters:

  • lookback_window: Buffer to remove from END_KEY of the highest slice
  • slice_range: Max difference between two slices. If the difference between two slices is greater, multiple slices will be created
  • start: _split_per_slice_range will clip any value to `self._start which means that:
    • if upper is less than self._start, no slices will be generated
    • if lower is less than self._start, self._start will be used as the lower boundary (lookback_window will not be considered in that case)

Note that the slices will overlap at their boundaries. We therefore expect to have at least the lower or the upper boundary to be inclusive in the API that is queried.

def should_be_synced(self, record: Record) -> bool:
579    def should_be_synced(self, record: Record) -> bool:
580        """
581        Determines if a record should be synced based on its cursor value.
582        :param record: The record to evaluate
583
584        :return: True if the record's cursor value falls within the sync boundaries
585        """
586        try:
587            record_cursor_value: CursorValueType = self._extract_cursor_value(record)
588        except ValueError:
589            self._log_for_record_without_cursor_value()
590            return True
591        return self._lowest_cursor_value_to_sync() <= record_cursor_value <= self._end_provider()

Determines if a record should be synced based on its cursor value.

Parameters
  • record: The record to evaluate
Returns

True if the record's cursor value falls within the sync boundaries

def reduce_slice_range( self, stream_slice: StreamSlice) -> StreamSlice:
611    def reduce_slice_range(self, stream_slice: StreamSlice) -> StreamSlice:
612        # In theory, we might be more flexible here meaning that it doesn't need to be in ascending order but it just
613        # needs to be ordered. For now though, we will only support ascending order.
614        if not self._is_ascending_order:
615            LOGGER.warning(
616                "Attempting to reduce slice while records are not returned in incremental order might lead to missing records"
617            )
618
619        if stream_slice in self._most_recent_cursor_value_per_partition:
620            return StreamSlice(
621                partition=stream_slice.partition,
622                cursor_slice={
623                    self._slice_boundary_fields_wrapper[
624                        self._START_BOUNDARY
625                    ]: self._connector_state_converter.output_format(
626                        self._most_recent_cursor_value_per_partition[stream_slice]
627                    ),
628                    self._slice_boundary_fields_wrapper[
629                        self._END_BOUNDARY
630                    ]: stream_slice.cursor_slice[
631                        self._slice_boundary_fields_wrapper[self._END_BOUNDARY]
632                    ],
633                },
634                extra_fields=stream_slice.extra_fields,
635            )
636        else:
637            return stream_slice
def split_request_window( self, stream_slice: StreamSlice, min_split_window: Optional[datetime.timedelta] = None) -> Optional[List[StreamSlice]]:
639    def split_request_window(
640        self, stream_slice: StreamSlice, min_split_window: Optional[datetime.timedelta] = None
641    ) -> Optional[List[StreamSlice]]:
642        """
643        Split `stream_slice` in half along this cursor's own boundary fields, granularity, and output format.
644
645        Returns `None` when the slice cannot be split any further: either it spans at most one
646        `cursor_granularity` unit already (the floor), it is already at or below `min_split_window` (a
647        connector-configured floor independent of and typically looser than `cursor_granularity`), the
648        boundaries are reversed, or no `cursor_granularity` was configured at all (there is then no way to know
649        the smallest addressable unit, or how to keep two children from overlapping at their shared edge).
650        Raises `AirbyteTracedException` instead if a boundary is missing or unparseable: that is a bug in how
651        the slice was produced, not the API rejecting a window, so it must not be reported as one. This is the
652        implementation `SimpleRetriever.request_window_splitter` binds to (see
653        `airbyte_cdk.sources.declarative.retrievers.simple_retriever`) that reuses the same
654        parsing/formatting/comparison operations `_split_per_slice_range` already relies on, so it works for
655        any `CursorValueType`/`GapType` pair this cursor was built with (datetime/timedelta, or int/int),
656        without a second, duplicate implementation of that logic living outside the cursor.
657        """
658        if not self._cursor_granularity:
659            return None
660
661        start_field, end_field = self._slice_boundary_fields_wrapper
662        try:
663            start_value = self._connector_state_converter.parse_value(
664                stream_slice.cursor_slice[start_field]
665            )
666            end_value = self._connector_state_converter.parse_value(
667                stream_slice.cursor_slice[end_field]
668            )
669        except (KeyError, ValueError, TypeError) as exception:
670            # A missing or unparseable boundary means this slice was produced (or passed in) wrong - a bug in
671            # slice generation, not something the API did. Raising here, rather than returning `None` (which
672            # `SimpleRetriever` reports as "the API kept rejecting..."), keeps that distinction visible instead
673            # of blaming the API for our own state.
674            raise AirbyteTracedException(
675                internal_message=(
676                    f"Cannot split {stream_slice}: boundary field {start_field!r} or {end_field!r} is missing "
677                    f"or not parseable: {exception}"
678                ),
679                message=(
680                    f"Stream {self._stream_name} produced a request window with a missing or invalid "
681                    f"{start_field!r}/{end_field!r} boundary and could not split it further."
682                ),
683                failure_type=FailureType.system_error,
684            ) from exception
685
686        if start_value >= end_value:
687            # Reversed or degenerate boundaries: nothing safe to split.
688            return None
689
690        span = end_value - start_value  # type: ignore[operator]  # concretely a GapType (timedelta/int) at runtime
691        if min_split_window is not None and span <= min_split_window:  # type: ignore[operator]
692            return None
693
694        total_units = span // self._cursor_granularity + 1  # type: ignore[operator]  # inclusive count of granularity units
695        if total_units <= 1:
696            # Already at the minimum granularity and still rejected; nothing left to split.
697            return None
698
699        half_units = total_units // 2
700        first_end = (
701            start_value + (half_units * self._cursor_granularity) - self._cursor_granularity  # type: ignore[operator]
702        )
703        second_start = first_end + self._cursor_granularity
704
705        if not (start_value <= first_end < second_start <= end_value):
706            # No-progress guard, independent of the granularity check above: refuse to return children that
707            # are not strictly ordered and non-degenerate, rather than risk an identical or overlapping split.
708            return None
709
710        return [
711            StreamSlice(
712                partition=stream_slice.partition,
713                cursor_slice={
714                    start_field: self._connector_state_converter.output_format(start_value),
715                    end_field: self._connector_state_converter.output_format(first_end),
716                },
717                extra_fields=stream_slice.extra_fields,
718            ),
719            StreamSlice(
720                partition=stream_slice.partition,
721                cursor_slice={
722                    start_field: self._connector_state_converter.output_format(second_start),
723                    end_field: self._connector_state_converter.output_format(end_value),
724                },
725                extra_fields=stream_slice.extra_fields,
726            ),
727        ]

Split stream_slice in half along this cursor's own boundary fields, granularity, and output format.

Returns None when the slice cannot be split any further: either it spans at most one cursor_granularity unit already (the floor), it is already at or below min_split_window (a connector-configured floor independent of and typically looser than cursor_granularity), the boundaries are reversed, or no cursor_granularity was configured at all (there is then no way to know the smallest addressable unit, or how to keep two children from overlapping at their shared edge). Raises AirbyteTracedException instead if a boundary is missing or unparseable: that is a bug in how the slice was produced, not the API rejecting a window, so it must not be reported as one. This is the implementation SimpleRetriever.request_window_splitter binds to (see airbyte_cdk.sources.declarative.retrievers.simple_retriever) that reuses the same parsing/formatting/comparison operations _split_per_slice_range already relies on, so it works for any CursorValueType/GapType pair this cursor was built with (datetime/timedelta, or int/int), without a second, duplicate implementation of that logic living outside the cursor.

def get_cursor_datetime_from_state(self, stream_state: Mapping[str, Any]) -> datetime.datetime | None:
729    def get_cursor_datetime_from_state(
730        self, stream_state: Mapping[str, Any]
731    ) -> datetime.datetime | None:
732        """Extract and parse the cursor datetime from the given stream state.
733
734        For concurrent cursors, the state can be in two formats:
735        1. Sequential/legacy format: {cursor_field: cursor_value}
736        2. Concurrent format: {state_type: "date-range", slices: [...]}
737
738        Returns the cursor datetime if present and parseable, otherwise returns None.
739        """
740        # Check if state is in concurrent format (need to convert to dict for type compatibility)
741        mutable_state: MutableMapping[str, Any] = dict(stream_state)
742        if self._connector_state_converter.is_state_message_compatible(mutable_state):
743            slices = stream_state.get("slices", [])
744            if not slices:
745                return None
746            # Get the most recent cursor value from the first slice (after merging)
747            first_slice = slices[0]
748            cursor_value = first_slice.get(
749                self._connector_state_converter.MOST_RECENT_RECORD_KEY
750            ) or first_slice.get(self._connector_state_converter.END_KEY)
751            if not cursor_value:
752                return None
753            try:
754                parsed_value = self._connector_state_converter.parse_value(cursor_value)
755                if isinstance(parsed_value, datetime.datetime):
756                    return parsed_value
757                return None
758            except (ValueError, TypeError):
759                return None
760
761        # Sequential/legacy format: {cursor_field: cursor_value}
762        cursor_value = stream_state.get(self._cursor_field.cursor_field_key)
763        if not cursor_value:
764            return None
765        try:
766            parsed_value = self._connector_state_converter.parse_value(cursor_value)
767            if isinstance(parsed_value, datetime.datetime):
768                return parsed_value
769            return None
770        except (ValueError, TypeError):
771            return None

Extract and parse the cursor datetime from the given stream state.

For concurrent cursors, the state can be in two formats:

  1. Sequential/legacy format: {cursor_field: cursor_value}
  2. Concurrent format: {state_type: "date-range", slices: [...]}

Returns the cursor datetime if present and parseable, otherwise returns None.

class ConcurrentSource:
 31class ConcurrentSource:
 32    """
 33    A Source that reads data from multiple AbstractStreams concurrently.
 34    It does so by submitting partition generation, and partition read tasks to a thread pool.
 35    The tasks asynchronously add their output to a shared queue.
 36    The read is done when all partitions for all streams w ere generated and read.
 37    """
 38
 39    DEFAULT_TIMEOUT_SECONDS = 900  # Deprecated: default of the ignored `timeout_seconds`.
 40
 41    @staticmethod
 42    def create(
 43        num_workers: int,
 44        initial_number_of_partitions_to_generate: int,
 45        logger: logging.Logger,
 46        slice_logger: SliceLogger,
 47        message_repository: MessageRepository,
 48        queue: Optional[Queue[QueueItem]] = None,
 49        timeout_seconds: int = DEFAULT_TIMEOUT_SECONDS,
 50    ) -> "ConcurrentSource":
 51        if initial_number_of_partitions_to_generate < 1:
 52            raise ValueError(
 53                f"initial_number_of_partitions_to_generate must be >= 1, got {initial_number_of_partitions_to_generate}"
 54            )
 55        is_single_threaded = initial_number_of_partitions_to_generate == 1 and num_workers == 1
 56        too_many_generator = (
 57            not is_single_threaded and initial_number_of_partitions_to_generate >= num_workers
 58        )
 59        assert not too_many_generator, (
 60            "It is required to have more workers than threads generating partitions"
 61        )
 62        threadpool = ThreadPoolManager(
 63            concurrent.futures.ThreadPoolExecutor(
 64                max_workers=num_workers, thread_name_prefix="workerpool"
 65            ),
 66            logger,
 67        )
 68        return ConcurrentSource(
 69            threadpool=threadpool,
 70            logger=logger,
 71            slice_logger=slice_logger,
 72            queue=queue,
 73            message_repository=message_repository,
 74            initial_number_partitions_to_generate=initial_number_of_partitions_to_generate,
 75            timeout_seconds=timeout_seconds,
 76        )
 77
 78    def __init__(
 79        self,
 80        threadpool: ThreadPoolManager,
 81        logger: logging.Logger,
 82        slice_logger: SliceLogger = DebugSliceLogger(),
 83        queue: Optional[Queue[QueueItem]] = None,
 84        message_repository: MessageRepository = InMemoryMessageRepository(),
 85        initial_number_partitions_to_generate: int = 1,
 86        timeout_seconds: int = DEFAULT_TIMEOUT_SECONDS,
 87    ) -> None:
 88        """
 89        :param threadpool: The threadpool to submit tasks to
 90        :param logger: The logger to log to
 91        :param slice_logger: The slice logger used to create messages on new slices
 92        :param message_repository: The repository to emit messages to
 93        :param initial_number_partitions_to_generate: The initial number of concurrent partition generation tasks. Limiting this number ensures will limit the latency of the first records emitted. While the latency is not critical, emitting the records early allows the platform and the destination to process them as early as possible.
 94        :param timeout_seconds: Deprecated and ignored. Kept so existing callers do not break.
 95        """
 96        self._threadpool = threadpool
 97        self._logger = logger
 98        self._slice_logger = slice_logger
 99        self._message_repository = message_repository
100        self._initial_number_partitions_to_generate = initial_number_partitions_to_generate
101
102        # We set a maxsize to for the main thread to process record items when the queue size grows. This assumes that there are less
103        # threads generating partitions that than are max number of workers. If it weren't the case, we could have threads only generating
104        # partitions which would fill the queue. This number is arbitrarily set to 10_000 but will probably need to be changed given more
105        # information and might even need to be configurable depending on the source
106        self._queue = queue or Queue(maxsize=10_000)
107
108    def read(
109        self,
110        streams: List[AbstractStream],
111    ) -> Iterator[AirbyteMessage]:
112        self._logger.info("Starting syncing")
113        concurrent_stream_processor = ConcurrentReadProcessor(
114            streams,
115            PartitionEnqueuer(self._queue, self._threadpool),
116            self._threadpool,
117            self._logger,
118            self._slice_logger,
119            self._message_repository,
120            PartitionReader(
121                self._queue,
122                PartitionLogger(self._slice_logger, self._logger, self._message_repository),
123            ),
124            max_concurrent_partition_generators=self._initial_number_partitions_to_generate,
125        )
126
127        # Enqueue initial partition generation tasks
128        yield from self._submit_initial_partition_generators(concurrent_stream_processor)
129
130        # Read from the queue until all partitions were generated and read
131        yield from self._consume_from_queue(
132            self._queue,
133            concurrent_stream_processor,
134        )
135        self._threadpool.check_for_errors_and_shutdown()
136        self._logger.info("Finished syncing")
137
138    def _submit_initial_partition_generators(
139        self, concurrent_stream_processor: ConcurrentReadProcessor
140    ) -> Iterable[AirbyteMessage]:
141        for _ in range(self._initial_number_partitions_to_generate):
142            status_message = concurrent_stream_processor.start_next_partition_generator()
143            if status_message:
144                yield status_message
145
146    def _consume_from_queue(
147        self,
148        queue: Queue[QueueItem],
149        concurrent_stream_processor: ConcurrentReadProcessor,
150    ) -> Iterable[AirbyteMessage]:
151        while airbyte_message_or_record_or_exception := queue.get():
152            yield from self._handle_item(
153                airbyte_message_or_record_or_exception,
154                concurrent_stream_processor,
155            )
156            # In the event that a partition raises an exception, anything remaining in
157            # the queue will be missed because is_done() can raise an exception and exit
158            # out of this loop before remaining items are consumed
159            if queue.empty() and concurrent_stream_processor.is_done():
160                # all partitions were generated and processed. we're done here
161                break
162
163    def _handle_item(
164        self,
165        queue_item: QueueItem,
166        concurrent_stream_processor: ConcurrentReadProcessor,
167    ) -> Iterable[AirbyteMessage]:
168        # handle queue item and call the appropriate handler depending on the type of the queue item
169        if isinstance(queue_item, StreamThreadException):
170            yield from concurrent_stream_processor.on_exception(queue_item)
171        elif isinstance(queue_item, PartitionGenerationCompletedSentinel):
172            yield from concurrent_stream_processor.on_partition_generation_completed(queue_item)
173        elif isinstance(queue_item, Partition):
174            concurrent_stream_processor.on_partition(queue_item)
175        elif isinstance(queue_item, PartitionCompleteSentinel):
176            yield from concurrent_stream_processor.on_partition_complete_sentinel(queue_item)
177        elif isinstance(queue_item, Record):
178            yield from concurrent_stream_processor.on_record(queue_item)
179        elif isinstance(queue_item, AirbyteMessage):
180            yield queue_item
181        else:
182            raise ValueError(f"Unknown queue item type: {type(queue_item)}")

A Source that reads data from multiple AbstractStreams concurrently. It does so by submitting partition generation, and partition read tasks to a thread pool. The tasks asynchronously add their output to a shared queue. The read is done when all partitions for all streams w ere generated and read.

 78    def __init__(
 79        self,
 80        threadpool: ThreadPoolManager,
 81        logger: logging.Logger,
 82        slice_logger: SliceLogger = DebugSliceLogger(),
 83        queue: Optional[Queue[QueueItem]] = None,
 84        message_repository: MessageRepository = InMemoryMessageRepository(),
 85        initial_number_partitions_to_generate: int = 1,
 86        timeout_seconds: int = DEFAULT_TIMEOUT_SECONDS,
 87    ) -> None:
 88        """
 89        :param threadpool: The threadpool to submit tasks to
 90        :param logger: The logger to log to
 91        :param slice_logger: The slice logger used to create messages on new slices
 92        :param message_repository: The repository to emit messages to
 93        :param initial_number_partitions_to_generate: The initial number of concurrent partition generation tasks. Limiting this number ensures will limit the latency of the first records emitted. While the latency is not critical, emitting the records early allows the platform and the destination to process them as early as possible.
 94        :param timeout_seconds: Deprecated and ignored. Kept so existing callers do not break.
 95        """
 96        self._threadpool = threadpool
 97        self._logger = logger
 98        self._slice_logger = slice_logger
 99        self._message_repository = message_repository
100        self._initial_number_partitions_to_generate = initial_number_partitions_to_generate
101
102        # We set a maxsize to for the main thread to process record items when the queue size grows. This assumes that there are less
103        # threads generating partitions that than are max number of workers. If it weren't the case, we could have threads only generating
104        # partitions which would fill the queue. This number is arbitrarily set to 10_000 but will probably need to be changed given more
105        # information and might even need to be configurable depending on the source
106        self._queue = queue or Queue(maxsize=10_000)
Parameters
  • threadpool: The threadpool to submit tasks to
  • logger: The logger to log to
  • slice_logger: The slice logger used to create messages on new slices
  • message_repository: The repository to emit messages to
  • initial_number_partitions_to_generate: The initial number of concurrent partition generation tasks. Limiting this number ensures will limit the latency of the first records emitted. While the latency is not critical, emitting the records early allows the platform and the destination to process them as early as possible.
  • timeout_seconds: Deprecated and ignored. Kept so existing callers do not break.
DEFAULT_TIMEOUT_SECONDS = 900
@staticmethod
def create( num_workers: int, initial_number_of_partitions_to_generate: int, logger: logging.Logger, slice_logger: airbyte_cdk.sources.utils.slice_logger.SliceLogger, message_repository: MessageRepository, queue: Optional[queue.Queue[Union[Record, airbyte_cdk.sources.streams.concurrent.partitions.partition.Partition, airbyte_cdk.sources.streams.concurrent.partitions.types.PartitionCompleteSentinel, airbyte_cdk.sources.concurrent_source.partition_generation_completed_sentinel.PartitionGenerationCompletedSentinel, Exception, AirbyteMessage]]] = None, timeout_seconds: int = 900) -> ConcurrentSource:
41    @staticmethod
42    def create(
43        num_workers: int,
44        initial_number_of_partitions_to_generate: int,
45        logger: logging.Logger,
46        slice_logger: SliceLogger,
47        message_repository: MessageRepository,
48        queue: Optional[Queue[QueueItem]] = None,
49        timeout_seconds: int = DEFAULT_TIMEOUT_SECONDS,
50    ) -> "ConcurrentSource":
51        if initial_number_of_partitions_to_generate < 1:
52            raise ValueError(
53                f"initial_number_of_partitions_to_generate must be >= 1, got {initial_number_of_partitions_to_generate}"
54            )
55        is_single_threaded = initial_number_of_partitions_to_generate == 1 and num_workers == 1
56        too_many_generator = (
57            not is_single_threaded and initial_number_of_partitions_to_generate >= num_workers
58        )
59        assert not too_many_generator, (
60            "It is required to have more workers than threads generating partitions"
61        )
62        threadpool = ThreadPoolManager(
63            concurrent.futures.ThreadPoolExecutor(
64                max_workers=num_workers, thread_name_prefix="workerpool"
65            ),
66            logger,
67        )
68        return ConcurrentSource(
69            threadpool=threadpool,
70            logger=logger,
71            slice_logger=slice_logger,
72            queue=queue,
73            message_repository=message_repository,
74            initial_number_partitions_to_generate=initial_number_of_partitions_to_generate,
75            timeout_seconds=timeout_seconds,
76        )
def read( self, streams: List[airbyte_cdk.sources.streams.concurrent.abstract_stream.AbstractStream]) -> Iterator[AirbyteMessage]:
108    def read(
109        self,
110        streams: List[AbstractStream],
111    ) -> Iterator[AirbyteMessage]:
112        self._logger.info("Starting syncing")
113        concurrent_stream_processor = ConcurrentReadProcessor(
114            streams,
115            PartitionEnqueuer(self._queue, self._threadpool),
116            self._threadpool,
117            self._logger,
118            self._slice_logger,
119            self._message_repository,
120            PartitionReader(
121                self._queue,
122                PartitionLogger(self._slice_logger, self._logger, self._message_repository),
123            ),
124            max_concurrent_partition_generators=self._initial_number_partitions_to_generate,
125        )
126
127        # Enqueue initial partition generation tasks
128        yield from self._submit_initial_partition_generators(concurrent_stream_processor)
129
130        # Read from the queue until all partitions were generated and read
131        yield from self._consume_from_queue(
132            self._queue,
133            concurrent_stream_processor,
134        )
135        self._threadpool.check_for_errors_and_shutdown()
136        self._logger.info("Finished syncing")
 34class ConcurrentSourceAdapter(AbstractSource, ABC):
 35    def __init__(self, concurrent_source: ConcurrentSource, **kwargs: Any) -> None:
 36        """
 37        ConcurrentSourceAdapter is a Source that wraps a concurrent source and exposes it as a regular source.
 38
 39        The source's streams are still defined through the streams() method.
 40        Streams wrapped in a StreamFacade will be processed concurrently.
 41        Other streams will be processed sequentially as a later step.
 42        """
 43        self._concurrent_source = concurrent_source
 44        super().__init__(**kwargs)
 45
 46    def read(
 47        self,
 48        logger: logging.Logger,
 49        config: Mapping[str, Any],
 50        catalog: ConfiguredAirbyteCatalog,
 51        state: Optional[List[AirbyteStateMessage]] = None,
 52    ) -> Iterator[AirbyteMessage]:
 53        abstract_streams = self._select_abstract_streams(config, catalog)
 54        concurrent_stream_names = {stream.name for stream in abstract_streams}
 55        configured_catalog_for_regular_streams = ConfiguredAirbyteCatalog(
 56            streams=[
 57                stream
 58                for stream in catalog.streams
 59                if stream.stream.name not in concurrent_stream_names
 60            ]
 61        )
 62        if abstract_streams:
 63            yield from self._concurrent_source.read(abstract_streams)
 64        if configured_catalog_for_regular_streams.streams:
 65            yield from super().read(logger, config, configured_catalog_for_regular_streams, state)
 66
 67    def _select_abstract_streams(
 68        self, config: Mapping[str, Any], configured_catalog: ConfiguredAirbyteCatalog
 69    ) -> List[AbstractStream]:
 70        """
 71        Selects streams that can be processed concurrently and returns their abstract representations.
 72        """
 73        all_streams = self.streams(config)
 74        stream_name_to_instance: Mapping[str, Stream] = {s.name: s for s in all_streams}
 75        abstract_streams: List[AbstractStream] = []
 76        for configured_stream in configured_catalog.streams:
 77            stream_instance = stream_name_to_instance.get(configured_stream.stream.name)
 78            if not stream_instance:
 79                continue
 80
 81            if isinstance(stream_instance, AbstractStreamFacade):
 82                abstract_streams.append(stream_instance.get_underlying_stream())
 83        return abstract_streams
 84
 85    def convert_to_concurrent_stream(
 86        self,
 87        logger: logging.Logger,
 88        stream: Stream,
 89        state_manager: ConnectorStateManager,
 90        cursor: Optional[Cursor] = None,
 91    ) -> Stream:
 92        """
 93        Prepares a stream for concurrent processing by initializing or assigning a cursor,
 94        managing the stream's state, and returning an updated Stream instance.
 95        """
 96        state: MutableMapping[str, Any] = {}
 97
 98        if cursor:
 99            state = state_manager.get_stream_state(stream.name, stream.namespace)
100
101            stream.cursor = cursor  # type: ignore[assignment]  # cursor is of type ConcurrentCursor, which inherits from Cursor
102            if hasattr(stream, "parent"):
103                stream.parent.cursor = cursor
104        else:
105            cursor = FinalStateCursor(
106                stream_name=stream.name,
107                stream_namespace=stream.namespace,
108                message_repository=self.message_repository,  # type: ignore[arg-type]  # _default_message_repository will be returned in the worst case
109            )
110        return StreamFacade.create_from_stream(stream, self, logger, state, cursor)
111
112    def initialize_cursor(
113        self,
114        stream: Stream,
115        state_manager: ConnectorStateManager,
116        converter: AbstractStreamStateConverter,
117        slice_boundary_fields: Optional[Tuple[str, str]],
118        start: Optional[CursorValueType],
119        end_provider: Callable[[], CursorValueType],
120        lookback_window: Optional[GapType] = None,
121        slice_range: Optional[GapType] = None,
122    ) -> Optional[ConcurrentCursor]:
123        lookback_window = lookback_window or timedelta(seconds=DEFAULT_LOOKBACK_SECONDS)
124
125        cursor_field_name = stream.cursor_field
126
127        if cursor_field_name:
128            if not isinstance(cursor_field_name, str):
129                raise ValueError(
130                    f"Cursor field type must be a string, but received {type(cursor_field_name).__name__}."
131                )
132
133            return ConcurrentCursor(
134                stream.name,
135                stream.namespace,
136                state_manager.get_stream_state(stream.name, stream.namespace),
137                self.message_repository,  # type: ignore[arg-type]  # _default_message_repository will be returned in the worst case
138                state_manager,
139                converter,
140                CursorField(cursor_field_name),
141                slice_boundary_fields,
142                start,
143                end_provider,
144                lookback_window,
145                slice_range,
146            )
147
148        return None

Abstract base class for an Airbyte Source. Consumers should implement any abstract methods in this class to create an Airbyte Specification compliant Source.

ConcurrentSourceAdapter( concurrent_source: ConcurrentSource, **kwargs: Any)
35    def __init__(self, concurrent_source: ConcurrentSource, **kwargs: Any) -> None:
36        """
37        ConcurrentSourceAdapter is a Source that wraps a concurrent source and exposes it as a regular source.
38
39        The source's streams are still defined through the streams() method.
40        Streams wrapped in a StreamFacade will be processed concurrently.
41        Other streams will be processed sequentially as a later step.
42        """
43        self._concurrent_source = concurrent_source
44        super().__init__(**kwargs)

ConcurrentSourceAdapter is a Source that wraps a concurrent source and exposes it as a regular source.

The source's streams are still defined through the streams() method. Streams wrapped in a StreamFacade will be processed concurrently. Other streams will be processed sequentially as a later step.

def read( self, logger: logging.Logger, config: Mapping[str, Any], catalog: airbyte_protocol_dataclasses.models.airbyte_protocol.ConfiguredAirbyteCatalog, state: Optional[List[airbyte_cdk.models.airbyte_protocol.AirbyteStateMessage]] = None) -> Iterator[AirbyteMessage]:
46    def read(
47        self,
48        logger: logging.Logger,
49        config: Mapping[str, Any],
50        catalog: ConfiguredAirbyteCatalog,
51        state: Optional[List[AirbyteStateMessage]] = None,
52    ) -> Iterator[AirbyteMessage]:
53        abstract_streams = self._select_abstract_streams(config, catalog)
54        concurrent_stream_names = {stream.name for stream in abstract_streams}
55        configured_catalog_for_regular_streams = ConfiguredAirbyteCatalog(
56            streams=[
57                stream
58                for stream in catalog.streams
59                if stream.stream.name not in concurrent_stream_names
60            ]
61        )
62        if abstract_streams:
63            yield from self._concurrent_source.read(abstract_streams)
64        if configured_catalog_for_regular_streams.streams:
65            yield from super().read(logger, config, configured_catalog_for_regular_streams, state)

Implements the Read operation from the Airbyte Specification. See https://docs.airbyte.com/understanding-airbyte/airbyte-protocol/.

def convert_to_concurrent_stream( self, logger: logging.Logger, stream: Stream, state_manager: ConnectorStateManager, cursor: Optional[Cursor] = None) -> Stream:
 85    def convert_to_concurrent_stream(
 86        self,
 87        logger: logging.Logger,
 88        stream: Stream,
 89        state_manager: ConnectorStateManager,
 90        cursor: Optional[Cursor] = None,
 91    ) -> Stream:
 92        """
 93        Prepares a stream for concurrent processing by initializing or assigning a cursor,
 94        managing the stream's state, and returning an updated Stream instance.
 95        """
 96        state: MutableMapping[str, Any] = {}
 97
 98        if cursor:
 99            state = state_manager.get_stream_state(stream.name, stream.namespace)
100
101            stream.cursor = cursor  # type: ignore[assignment]  # cursor is of type ConcurrentCursor, which inherits from Cursor
102            if hasattr(stream, "parent"):
103                stream.parent.cursor = cursor
104        else:
105            cursor = FinalStateCursor(
106                stream_name=stream.name,
107                stream_namespace=stream.namespace,
108                message_repository=self.message_repository,  # type: ignore[arg-type]  # _default_message_repository will be returned in the worst case
109            )
110        return StreamFacade.create_from_stream(stream, self, logger, state, cursor)

Prepares a stream for concurrent processing by initializing or assigning a cursor, managing the stream's state, and returning an updated Stream instance.

112    def initialize_cursor(
113        self,
114        stream: Stream,
115        state_manager: ConnectorStateManager,
116        converter: AbstractStreamStateConverter,
117        slice_boundary_fields: Optional[Tuple[str, str]],
118        start: Optional[CursorValueType],
119        end_provider: Callable[[], CursorValueType],
120        lookback_window: Optional[GapType] = None,
121        slice_range: Optional[GapType] = None,
122    ) -> Optional[ConcurrentCursor]:
123        lookback_window = lookback_window or timedelta(seconds=DEFAULT_LOOKBACK_SECONDS)
124
125        cursor_field_name = stream.cursor_field
126
127        if cursor_field_name:
128            if not isinstance(cursor_field_name, str):
129                raise ValueError(
130                    f"Cursor field type must be a string, but received {type(cursor_field_name).__name__}."
131                )
132
133            return ConcurrentCursor(
134                stream.name,
135                stream.namespace,
136                state_manager.get_stream_state(stream.name, stream.namespace),
137                self.message_repository,  # type: ignore[arg-type]  # _default_message_repository will be returned in the worst case
138                state_manager,
139                converter,
140                CursorField(cursor_field_name),
141                slice_boundary_fields,
142                start,
143                end_provider,
144                lookback_window,
145                slice_range,
146            )
147
148        return None
 58class Cursor(StreamSlicer, ABC):
 59    @property
 60    @abstractmethod
 61    def state(self) -> MutableMapping[str, Any]: ...
 62
 63    @abstractmethod
 64    def observe(self, record: Record) -> None:
 65        """
 66        Indicate to the cursor that the record has been emitted
 67        """
 68        raise NotImplementedError()
 69
 70    @abstractmethod
 71    def close_partition(self, partition: Partition) -> None:
 72        """
 73        Indicate to the cursor that the partition has been successfully processed
 74        """
 75        raise NotImplementedError()
 76
 77    @abstractmethod
 78    def ensure_at_least_one_state_emitted(self) -> None:
 79        """
 80        State messages are emitted when a partition is closed. However, the platform expects at least one state to be emitted per sync per
 81        stream. Hence, if no partitions are generated, this method needs to be called.
 82        """
 83        raise NotImplementedError()
 84
 85    @abstractmethod
 86    def should_be_synced(self, record: Record) -> bool:
 87        pass
 88
 89    def stream_slices(self) -> Iterable[StreamSlice]:
 90        """
 91        Default placeholder implementation of generate_slices.
 92        Subclasses can override this method to provide actual behavior.
 93        """
 94        yield StreamSlice(partition={}, cursor_slice={})
 95
 96    def get_cursor_datetime_from_state(
 97        self, stream_state: Mapping[str, Any]
 98    ) -> datetime.datetime | None:
 99        """Extract and parse the cursor datetime from the given stream state.
100
101        This method is used by StateDelegatingStream to validate cursor age against
102        an API's data retention period. Subclasses should implement this method to
103        extract the cursor value from their specific state structure and parse it
104        into a datetime object.
105
106        Returns None if the cursor cannot be extracted or parsed, which will cause
107        StateDelegatingStream to fall back to full refresh (safe default).
108
109        Raises NotImplementedError by default - subclasses must implement this method
110        if they want to support cursor age validation with api_retention_period.
111        """
112        raise NotImplementedError(
113            f"{self.__class__.__name__} does not implement get_cursor_datetime_from_state. "
114            f"Cursor age validation with api_retention_period is not supported for this cursor type."
115        )

Slices the stream into chunks that can be fetched independently. Slices enable state checkpointing and data retrieval parallelization.

state: MutableMapping[str, Any]
59    @property
60    @abstractmethod
61    def state(self) -> MutableMapping[str, Any]: ...
@abstractmethod
def observe(self, record: Record) -> None:
63    @abstractmethod
64    def observe(self, record: Record) -> None:
65        """
66        Indicate to the cursor that the record has been emitted
67        """
68        raise NotImplementedError()

Indicate to the cursor that the record has been emitted

@abstractmethod
def close_partition( self, partition: airbyte_cdk.sources.streams.concurrent.partitions.partition.Partition) -> None:
70    @abstractmethod
71    def close_partition(self, partition: Partition) -> None:
72        """
73        Indicate to the cursor that the partition has been successfully processed
74        """
75        raise NotImplementedError()

Indicate to the cursor that the partition has been successfully processed

@abstractmethod
def ensure_at_least_one_state_emitted(self) -> None:
77    @abstractmethod
78    def ensure_at_least_one_state_emitted(self) -> None:
79        """
80        State messages are emitted when a partition is closed. However, the platform expects at least one state to be emitted per sync per
81        stream. Hence, if no partitions are generated, this method needs to be called.
82        """
83        raise NotImplementedError()

State messages are emitted when a partition is closed. However, the platform expects at least one state to be emitted per sync per stream. Hence, if no partitions are generated, this method needs to be called.

@abstractmethod
def should_be_synced(self, record: Record) -> bool:
85    @abstractmethod
86    def should_be_synced(self, record: Record) -> bool:
87        pass
def stream_slices(self) -> Iterable[StreamSlice]:
89    def stream_slices(self) -> Iterable[StreamSlice]:
90        """
91        Default placeholder implementation of generate_slices.
92        Subclasses can override this method to provide actual behavior.
93        """
94        yield StreamSlice(partition={}, cursor_slice={})

Default placeholder implementation of generate_slices. Subclasses can override this method to provide actual behavior.

def get_cursor_datetime_from_state(self, stream_state: Mapping[str, Any]) -> datetime.datetime | None:
 96    def get_cursor_datetime_from_state(
 97        self, stream_state: Mapping[str, Any]
 98    ) -> datetime.datetime | None:
 99        """Extract and parse the cursor datetime from the given stream state.
100
101        This method is used by StateDelegatingStream to validate cursor age against
102        an API's data retention period. Subclasses should implement this method to
103        extract the cursor value from their specific state structure and parse it
104        into a datetime object.
105
106        Returns None if the cursor cannot be extracted or parsed, which will cause
107        StateDelegatingStream to fall back to full refresh (safe default).
108
109        Raises NotImplementedError by default - subclasses must implement this method
110        if they want to support cursor age validation with api_retention_period.
111        """
112        raise NotImplementedError(
113            f"{self.__class__.__name__} does not implement get_cursor_datetime_from_state. "
114            f"Cursor age validation with api_retention_period is not supported for this cursor type."
115        )

Extract and parse the cursor datetime from the given stream state.

This method is used by StateDelegatingStream to validate cursor age against an API's data retention period. Subclasses should implement this method to extract the cursor value from their specific state structure and parse it into a datetime object.

Returns None if the cursor cannot be extracted or parsed, which will cause StateDelegatingStream to fall back to full refresh (safe default).

Raises NotImplementedError by default - subclasses must implement this method if they want to support cursor age validation with api_retention_period.

class CursorField:
44class CursorField:
45    def __init__(
46        self, cursor_field_key: str, supports_catalog_defined_cursor_field: bool = False
47    ) -> None:
48        self.cursor_field_key = cursor_field_key
49        self.supports_catalog_defined_cursor_field = supports_catalog_defined_cursor_field
50
51    def extract_value(self, record: Record) -> Any:
52        cursor_value = record.data.get(self.cursor_field_key)
53        if cursor_value is None:
54            raise ValueError(f"Could not find cursor field {self.cursor_field_key} in record")
55        return cursor_value  # type: ignore  # we assume that the value the path points at is a comparable
CursorField( cursor_field_key: str, supports_catalog_defined_cursor_field: bool = False)
45    def __init__(
46        self, cursor_field_key: str, supports_catalog_defined_cursor_field: bool = False
47    ) -> None:
48        self.cursor_field_key = cursor_field_key
49        self.supports_catalog_defined_cursor_field = supports_catalog_defined_cursor_field
cursor_field_key
supports_catalog_defined_cursor_field
def extract_value(self, record: Record) -> Any:
51    def extract_value(self, record: Record) -> Any:
52        cursor_value = record.data.get(self.cursor_field_key)
53        if cursor_value is None:
54            raise ValueError(f"Could not find cursor field {self.cursor_field_key} in record")
55        return cursor_value  # type: ignore  # we assume that the value the path points at is a comparable
DEFAULT_CONCURRENCY
115class EpochValueConcurrentStreamStateConverter(DateTimeStreamStateConverter):
116    """
117    e.g.
118    { "created": 1617030403 }
119    =>
120    {
121        "state_type": "date-range",
122        "metadata": { … },
123        "slices": [
124            {starts: 0, end: 1617030403, finished_processing: true}
125        ]
126    }
127    """
128
129    _zero_value = 0
130
131    def increment(self, timestamp: datetime) -> datetime:
132        return timestamp + timedelta(seconds=1)
133
134    def output_format(self, timestamp: datetime) -> int:
135        return int(timestamp.timestamp())
136
137    def parse_timestamp(self, timestamp: int) -> datetime:
138        dt_object = AirbyteDateTime.fromtimestamp(timestamp, timezone.utc)
139        if not isinstance(dt_object, AirbyteDateTime):
140            raise ValueError(
141                f"AirbyteDateTime object was expected but got {type(dt_object)} from AirbyteDateTime.fromtimestamp({timestamp})"
142            )
143        return dt_object

e.g. { "created": 1617030403 } => { "state_type": "date-range", "metadata": { … }, "slices": [ {starts: 0, end: 1617030403, finished_processing: true} ] }

def increment(self, timestamp: datetime.datetime) -> datetime.datetime:
131    def increment(self, timestamp: datetime) -> datetime:
132        return timestamp + timedelta(seconds=1)

Increment a timestamp by a single unit.

def output_format(self, timestamp: datetime.datetime) -> int:
134    def output_format(self, timestamp: datetime) -> int:
135        return int(timestamp.timestamp())

Convert the cursor value type to a JSON valid type.

def parse_timestamp(self, timestamp: int) -> datetime.datetime:
137    def parse_timestamp(self, timestamp: int) -> datetime:
138        dt_object = AirbyteDateTime.fromtimestamp(timestamp, timezone.utc)
139        if not isinstance(dt_object, AirbyteDateTime):
140            raise ValueError(
141                f"AirbyteDateTime object was expected but got {type(dt_object)} from AirbyteDateTime.fromtimestamp({timestamp})"
142            )
143        return dt_object
class FinalStateCursor(airbyte_cdk.Cursor):
118class FinalStateCursor(Cursor):
119    """Cursor that is used to guarantee at least one state message is emitted for a concurrent stream."""
120
121    def __init__(
122        self,
123        stream_name: str,
124        stream_namespace: Optional[str],
125        message_repository: MessageRepository,
126    ) -> None:
127        self._stream_name = stream_name
128        self._stream_namespace = stream_namespace
129        self._message_repository = message_repository
130        # Normally the connector state manager operates at the source-level. However, we only need it to write the sentinel
131        # state message rather than manage overall source state. This is also only temporary as we move to the resumable
132        # full refresh world where every stream uses a FileBasedConcurrentCursor with incremental state.
133        self._connector_state_manager = ConnectorStateManager()
134        self._has_closed_at_least_one_slice = False
135
136    @property
137    def state(self) -> MutableMapping[str, Any]:
138        return {NO_CURSOR_STATE_KEY: True}
139
140    def observe(self, record: Record) -> None:
141        pass
142
143    def close_partition(self, partition: Partition) -> None:
144        pass
145
146    def ensure_at_least_one_state_emitted(self) -> None:
147        """
148        Used primarily for full refresh syncs that do not have a valid cursor value to emit at the end of a sync
149        """
150
151        self._connector_state_manager.update_state_for_stream(
152            self._stream_name, self._stream_namespace, self.state
153        )
154        state_message = self._connector_state_manager.create_state_message(
155            self._stream_name, self._stream_namespace
156        )
157        self._message_repository.emit_message(state_message)
158
159    def should_be_synced(self, record: Record) -> bool:
160        return True
161
162    def get_cursor_datetime_from_state(
163        self, stream_state: Mapping[str, Any]
164    ) -> datetime.datetime | None:
165        """Return now() if state indicates a completed full refresh, else None.
166
167        When the state has NO_CURSOR_STATE_KEY: True, it means the previous sync was a
168        completed full refresh. Returning now() indicates the cursor is "current" and
169        within any retention period, so we should use incremental sync.
170
171        For any other state format, return None to indicate this cursor cannot parse it,
172        allowing the incremental cursor to handle the state instead.
173        """
174        if stream_state.get(NO_CURSOR_STATE_KEY):
175            return datetime.datetime.now(datetime.timezone.utc)
176        return None

Cursor that is used to guarantee at least one state message is emitted for a concurrent stream.

FinalStateCursor( stream_name: str, stream_namespace: Optional[str], message_repository: MessageRepository)
121    def __init__(
122        self,
123        stream_name: str,
124        stream_namespace: Optional[str],
125        message_repository: MessageRepository,
126    ) -> None:
127        self._stream_name = stream_name
128        self._stream_namespace = stream_namespace
129        self._message_repository = message_repository
130        # Normally the connector state manager operates at the source-level. However, we only need it to write the sentinel
131        # state message rather than manage overall source state. This is also only temporary as we move to the resumable
132        # full refresh world where every stream uses a FileBasedConcurrentCursor with incremental state.
133        self._connector_state_manager = ConnectorStateManager()
134        self._has_closed_at_least_one_slice = False
state: MutableMapping[str, Any]
136    @property
137    def state(self) -> MutableMapping[str, Any]:
138        return {NO_CURSOR_STATE_KEY: True}
def observe(self, record: Record) -> None:
140    def observe(self, record: Record) -> None:
141        pass

Indicate to the cursor that the record has been emitted

def close_partition( self, partition: airbyte_cdk.sources.streams.concurrent.partitions.partition.Partition) -> None:
143    def close_partition(self, partition: Partition) -> None:
144        pass

Indicate to the cursor that the partition has been successfully processed

def ensure_at_least_one_state_emitted(self) -> None:
146    def ensure_at_least_one_state_emitted(self) -> None:
147        """
148        Used primarily for full refresh syncs that do not have a valid cursor value to emit at the end of a sync
149        """
150
151        self._connector_state_manager.update_state_for_stream(
152            self._stream_name, self._stream_namespace, self.state
153        )
154        state_message = self._connector_state_manager.create_state_message(
155            self._stream_name, self._stream_namespace
156        )
157        self._message_repository.emit_message(state_message)

Used primarily for full refresh syncs that do not have a valid cursor value to emit at the end of a sync

def should_be_synced(self, record: Record) -> bool:
159    def should_be_synced(self, record: Record) -> bool:
160        return True
def get_cursor_datetime_from_state(self, stream_state: Mapping[str, Any]) -> datetime.datetime | None:
162    def get_cursor_datetime_from_state(
163        self, stream_state: Mapping[str, Any]
164    ) -> datetime.datetime | None:
165        """Return now() if state indicates a completed full refresh, else None.
166
167        When the state has NO_CURSOR_STATE_KEY: True, it means the previous sync was a
168        completed full refresh. Returning now() indicates the cursor is "current" and
169        within any retention period, so we should use incremental sync.
170
171        For any other state format, return None to indicate this cursor cannot parse it,
172        allowing the incremental cursor to handle the state instead.
173        """
174        if stream_state.get(NO_CURSOR_STATE_KEY):
175            return datetime.datetime.now(datetime.timezone.utc)
176        return None

Return now() if state indicates a completed full refresh, else None.

When the state has NO_CURSOR_STATE_KEY: True, it means the previous sync was a completed full refresh. Returning now() indicates the cursor is "current" and within any retention period, so we should use incremental sync.

For any other state format, return None to indicate this cursor cannot parse it, allowing the incremental cursor to handle the state instead.

Inherited Members
Cursor
stream_slices
146class IsoMillisConcurrentStreamStateConverter(DateTimeStreamStateConverter):
147    """
148    e.g.
149    { "created": "2021-01-18T21:18:20.000Z" }
150    =>
151    {
152        "state_type": "date-range",
153        "metadata": { … },
154        "slices": [
155            {starts: "2020-01-18T21:18:20.000Z", end: "2021-01-18T21:18:20.000Z", finished_processing: true}
156        ]
157    }
158    """
159
160    _zero_value = "0001-01-01T00:00:00.000Z"
161
162    def __init__(
163        self, is_sequential_state: bool = True, cursor_granularity: Optional[timedelta] = None
164    ):
165        super().__init__(is_sequential_state=is_sequential_state)
166        self._cursor_granularity = cursor_granularity or timedelta(milliseconds=1)
167
168    def increment(self, timestamp: datetime) -> datetime:
169        return timestamp + self._cursor_granularity
170
171    def output_format(self, timestamp: datetime) -> str:
172        """Format datetime with milliseconds always included.
173
174        Args:
175            timestamp: The datetime to format.
176
177        Returns:
178            str: ISO8601/RFC3339 formatted string with milliseconds.
179        """
180        dt = AirbyteDateTime.from_datetime(timestamp)
181        # Always include milliseconds, even if zero
182        millis = dt.microsecond // 1000 if dt.microsecond else 0
183        return f"{dt.year:04d}-{dt.month:02d}-{dt.day:02d}T{dt.hour:02d}:{dt.minute:02d}:{dt.second:02d}.{millis:03d}Z"
184
185    def parse_timestamp(self, timestamp: str) -> datetime:
186        dt_object = ab_datetime_parse(timestamp)
187        if not isinstance(dt_object, AirbyteDateTime):
188            raise ValueError(
189                f"AirbyteDateTime object was expected but got {type(dt_object)} from parse({timestamp})"
190            )
191        return dt_object

e.g. { "created": "2021-01-18T21:18:20.000Z" } => { "state_type": "date-range", "metadata": { … }, "slices": [ {starts: "2020-01-18T21:18:20.000Z", end: "2021-01-18T21:18:20.000Z", finished_processing: true} ] }

IsoMillisConcurrentStreamStateConverter( is_sequential_state: bool = True, cursor_granularity: Optional[datetime.timedelta] = None)
162    def __init__(
163        self, is_sequential_state: bool = True, cursor_granularity: Optional[timedelta] = None
164    ):
165        super().__init__(is_sequential_state=is_sequential_state)
166        self._cursor_granularity = cursor_granularity or timedelta(milliseconds=1)
def increment(self, timestamp: datetime.datetime) -> datetime.datetime:
168    def increment(self, timestamp: datetime) -> datetime:
169        return timestamp + self._cursor_granularity

Increment a timestamp by a single unit.

def output_format(self, timestamp: datetime.datetime) -> str:
171    def output_format(self, timestamp: datetime) -> str:
172        """Format datetime with milliseconds always included.
173
174        Args:
175            timestamp: The datetime to format.
176
177        Returns:
178            str: ISO8601/RFC3339 formatted string with milliseconds.
179        """
180        dt = AirbyteDateTime.from_datetime(timestamp)
181        # Always include milliseconds, even if zero
182        millis = dt.microsecond // 1000 if dt.microsecond else 0
183        return f"{dt.year:04d}-{dt.month:02d}-{dt.day:02d}T{dt.hour:02d}:{dt.minute:02d}:{dt.second:02d}.{millis:03d}Z"

Format datetime with milliseconds always included.

Arguments:
  • timestamp: The datetime to format.
Returns:

str: ISO8601/RFC3339 formatted string with milliseconds.

def parse_timestamp(self, timestamp: str) -> datetime.datetime:
185    def parse_timestamp(self, timestamp: str) -> datetime:
186        dt_object = ab_datetime_parse(timestamp)
187        if not isinstance(dt_object, AirbyteDateTime):
188            raise ValueError(
189                f"AirbyteDateTime object was expected but got {type(dt_object)} from parse({timestamp})"
190            )
191        return dt_object
@deprecated('This class is experimental. Use at your own risk.', category=ExperimentalClassWarning)
class StreamFacade(airbyte_cdk.sources.streams.concurrent.abstract_stream_facade.AbstractStreamFacade[airbyte_cdk.sources.streams.concurrent.default_stream.DefaultStream], airbyte_cdk.Stream):
 49@deprecated(
 50    "This class is experimental. Use at your own risk.",
 51    category=ExperimentalClassWarning,
 52)
 53class StreamFacade(AbstractStreamFacade[DefaultStream], Stream):
 54    """
 55    The StreamFacade is a Stream that wraps an AbstractStream and exposes it as a Stream.
 56
 57    All methods either delegate to the wrapped AbstractStream or provide a default implementation.
 58    The default implementations define restrictions imposed on Streams migrated to the new interface. For instance, only source-defined cursors are supported.
 59    """
 60
 61    @classmethod
 62    def create_from_stream(
 63        cls,
 64        stream: Stream,
 65        source: AbstractSource,
 66        logger: logging.Logger,
 67        state: Optional[MutableMapping[str, Any]],
 68        cursor: Cursor,
 69    ) -> Stream:
 70        """
 71        Create a ConcurrentStream from a Stream object.
 72        :param source: The source
 73        :param stream: The stream
 74        :param max_workers: The maximum number of worker thread to use
 75        :return:
 76        """
 77        pk = get_primary_key_from_stream(stream.primary_key)
 78        cursor_field = get_cursor_field_from_stream(stream)
 79
 80        if not source.message_repository:
 81            raise ValueError(
 82                "A message repository is required to emit non-record messages. Please set the message repository on the source."
 83            )
 84
 85        message_repository = source.message_repository
 86        return StreamFacade(
 87            DefaultStream(
 88                partition_generator=StreamPartitionGenerator(
 89                    stream,
 90                    message_repository,
 91                    SyncMode.full_refresh
 92                    if isinstance(cursor, FinalStateCursor)
 93                    else SyncMode.incremental,
 94                    [cursor_field] if cursor_field is not None else None,
 95                    state,
 96                ),
 97                name=stream.name,
 98                namespace=stream.namespace,
 99                json_schema=stream.get_json_schema(),
100                primary_key=pk,
101                cursor_field=CursorField(cursor_field_key=cursor_field) if cursor_field else None,
102                logger=logger,
103                cursor=cursor,
104            ),
105            stream,
106            cursor,
107            slice_logger=source._slice_logger,
108            logger=logger,
109        )
110
111    @property
112    def state(self) -> MutableMapping[str, Any]:
113        raise NotImplementedError(
114            "This should not be called as part of the Concurrent CDK code. Please report the problem to Airbyte"
115        )
116
117    @state.setter
118    def state(self, value: Mapping[str, Any]) -> None:
119        if "state" in dir(self._legacy_stream):
120            self._legacy_stream.state = value  # type: ignore  # validating `state` is attribute of stream using `if` above
121
122    def __init__(
123        self,
124        stream: DefaultStream,
125        legacy_stream: Stream,
126        cursor: Cursor,
127        slice_logger: SliceLogger,
128        logger: logging.Logger,
129    ):
130        """
131        :param stream: The underlying AbstractStream
132        """
133        self._abstract_stream = stream
134        self._legacy_stream = legacy_stream
135        self._cursor = cursor
136        self._slice_logger = slice_logger
137        self._logger = logger
138
139    def read(
140        self,
141        configured_stream: ConfiguredAirbyteStream,
142        logger: logging.Logger,
143        slice_logger: SliceLogger,
144        stream_state: MutableMapping[str, Any],
145        state_manager: ConnectorStateManager,
146        internal_config: InternalConfig,
147    ) -> Iterable[StreamData]:
148        yield from self._read_records()
149
150    def read_records(
151        self,
152        sync_mode: SyncMode,
153        cursor_field: Optional[List[str]] = None,
154        stream_slice: Optional[Mapping[str, Any]] = None,
155        stream_state: Optional[Mapping[str, Any]] = None,
156    ) -> Iterable[StreamData]:
157        try:
158            yield from self._read_records()
159        except Exception as exc:
160            if hasattr(self._cursor, "state"):
161                state = str(self._cursor.state)
162            else:
163                # This shouldn't happen if the ConcurrentCursor was used
164                state = "unknown; no state attribute was available on the cursor"
165            yield AirbyteMessage(
166                type=Type.LOG,
167                log=AirbyteLogMessage(
168                    level=Level.ERROR, message=f"Cursor State at time of exception: {state}"
169                ),
170            )
171            raise exc
172
173    def _read_records(self) -> Iterable[StreamData]:
174        for partition in self._abstract_stream.generate_partitions():
175            if self._slice_logger.should_log_slice_message(self._logger):
176                yield self._slice_logger.create_slice_log_message(partition.to_slice())
177            for record in partition.read():
178                yield record.data
179
180    @property
181    def name(self) -> str:
182        return self._abstract_stream.name
183
184    @property
185    def primary_key(self) -> Optional[Union[str, List[str], List[List[str]]]]:
186        # This method is not expected to be called directly. It is only implemented for backward compatibility with the old interface
187        return self.as_airbyte_stream().source_defined_primary_key  # type: ignore # source_defined_primary_key is known to be an Optional[List[List[str]]]
188
189    @property
190    def cursor_field(self) -> Union[str, List[str]]:
191        if self._abstract_stream.cursor_field is None:
192            return []
193        else:
194            return self._abstract_stream.cursor_field
195
196    @property
197    def cursor(self) -> Optional[Cursor]:  # type: ignore[override] # StreamFaced expects to use only airbyte_cdk.sources.streams.concurrent.cursor.Cursor
198        return self._cursor
199
200    @property
201    def block_simultaneous_read(self) -> str:
202        """Returns the blocking group name from the underlying stream"""
203        return self._abstract_stream.block_simultaneous_read
204
205    # FIXME the lru_cache seems to be mostly there because of typing issue
206    @lru_cache(maxsize=None)
207    def get_json_schema(self) -> Mapping[str, Any]:
208        return self._abstract_stream.get_json_schema()
209
210    @property
211    def supports_incremental(self) -> bool:
212        return self._legacy_stream.supports_incremental
213
214    def as_airbyte_stream(self) -> AirbyteStream:
215        return self._abstract_stream.as_airbyte_stream()
216
217    def log_stream_sync_configuration(self) -> None:
218        self._abstract_stream.log_stream_sync_configuration()
219
220    def get_underlying_stream(self) -> DefaultStream:
221        return self._abstract_stream

The StreamFacade is a Stream that wraps an AbstractStream and exposes it as a Stream.

All methods either delegate to the wrapped AbstractStream or provide a default implementation. The default implementations define restrictions imposed on Streams migrated to the new interface. For instance, only source-defined cursors are supported.

StreamFacade( stream: airbyte_cdk.sources.streams.concurrent.default_stream.DefaultStream, legacy_stream: Stream, cursor: Cursor, slice_logger: airbyte_cdk.sources.utils.slice_logger.SliceLogger, logger: logging.Logger)
122    def __init__(
123        self,
124        stream: DefaultStream,
125        legacy_stream: Stream,
126        cursor: Cursor,
127        slice_logger: SliceLogger,
128        logger: logging.Logger,
129    ):
130        """
131        :param stream: The underlying AbstractStream
132        """
133        self._abstract_stream = stream
134        self._legacy_stream = legacy_stream
135        self._cursor = cursor
136        self._slice_logger = slice_logger
137        self._logger = logger
Parameters
  • stream: The underlying AbstractStream
@classmethod
def create_from_stream( cls, stream: Stream, source: AbstractSource, logger: logging.Logger, state: Optional[MutableMapping[str, Any]], cursor: Cursor) -> Stream:
 61    @classmethod
 62    def create_from_stream(
 63        cls,
 64        stream: Stream,
 65        source: AbstractSource,
 66        logger: logging.Logger,
 67        state: Optional[MutableMapping[str, Any]],
 68        cursor: Cursor,
 69    ) -> Stream:
 70        """
 71        Create a ConcurrentStream from a Stream object.
 72        :param source: The source
 73        :param stream: The stream
 74        :param max_workers: The maximum number of worker thread to use
 75        :return:
 76        """
 77        pk = get_primary_key_from_stream(stream.primary_key)
 78        cursor_field = get_cursor_field_from_stream(stream)
 79
 80        if not source.message_repository:
 81            raise ValueError(
 82                "A message repository is required to emit non-record messages. Please set the message repository on the source."
 83            )
 84
 85        message_repository = source.message_repository
 86        return StreamFacade(
 87            DefaultStream(
 88                partition_generator=StreamPartitionGenerator(
 89                    stream,
 90                    message_repository,
 91                    SyncMode.full_refresh
 92                    if isinstance(cursor, FinalStateCursor)
 93                    else SyncMode.incremental,
 94                    [cursor_field] if cursor_field is not None else None,
 95                    state,
 96                ),
 97                name=stream.name,
 98                namespace=stream.namespace,
 99                json_schema=stream.get_json_schema(),
100                primary_key=pk,
101                cursor_field=CursorField(cursor_field_key=cursor_field) if cursor_field else None,
102                logger=logger,
103                cursor=cursor,
104            ),
105            stream,
106            cursor,
107            slice_logger=source._slice_logger,
108            logger=logger,
109        )

Create a ConcurrentStream from a Stream object.

Parameters
  • source: The source
  • stream: The stream
  • max_workers: The maximum number of worker thread to use
Returns
state: MutableMapping[str, Any]
111    @property
112    def state(self) -> MutableMapping[str, Any]:
113        raise NotImplementedError(
114            "This should not be called as part of the Concurrent CDK code. Please report the problem to Airbyte"
115        )
block_simultaneous_read: str
200    @property
201    def block_simultaneous_read(self) -> str:
202        """Returns the blocking group name from the underlying stream"""
203        return self._abstract_stream.block_simultaneous_read

Returns the blocking group name from the underlying stream

def get_underlying_stream( self) -> airbyte_cdk.sources.streams.concurrent.default_stream.DefaultStream:
220    def get_underlying_stream(self) -> DefaultStream:
221        return self._abstract_stream

Return the underlying stream facade object.

def create_connector_config_control_message( config: MutableMapping[str, Any]) -> AirbyteMessage:
106def create_connector_config_control_message(config: MutableMapping[str, Any]) -> AirbyteMessage:
107    control_message = AirbyteControlMessage(
108        type=OrchestratorType.CONNECTOR_CONFIG,
109        emitted_at=time.time() * 1000,
110        connectorConfig=AirbyteControlConnectorConfigMessage(config=config),
111    )
112    return AirbyteMessage(type=Type.CONTROL, control=control_message)
def emit_configuration_as_airbyte_control_message(config: MutableMapping[str, Any]) -> None:
 90def emit_configuration_as_airbyte_control_message(config: MutableMapping[str, Any]) -> None:
 91    """Print the connector config as a CONNECTOR_CONFIG control message directly to stdout.
 92
 93    Single-use OAuth refresh token rotation relies on this direct emission: it reaches the
 94    platform for every command (including `check` and `discover`) and does not depend on a
 95    message repository being drained. The payload and its newline are written in one `write()`
 96    call so a concurrent writer to the shared stdout buffer cannot split the JSON line.
 97    """
 98    airbyte_message = create_connector_config_control_message(config)
 99    print(
100        f"{orjson.dumps(AirbyteMessageSerializer.dump(airbyte_message)).decode()}\n",
101        end="",
102        flush=True,
103    )

Print the connector config as a CONNECTOR_CONFIG control message directly to stdout.

Single-use OAuth refresh token rotation relies on this direct emission: it reaches the platform for every command (including check and discover) and does not depend on a message repository being drained. The payload and its newline are written in one write() call so a concurrent writer to the shared stdout buffer cannot split the JSON line.

 53class AbstractSource(Source, ABC):
 54    """
 55    Abstract base class for an Airbyte Source. Consumers should implement any abstract methods
 56    in this class to create an Airbyte Specification compliant Source.
 57    """
 58
 59    @abstractmethod
 60    def check_connection(
 61        self, logger: logging.Logger, config: Mapping[str, Any]
 62    ) -> Tuple[bool, Optional[Any]]:
 63        """
 64        :param logger: source logger
 65        :param config: The user-provided configuration as specified by the source's spec.
 66          This usually contains information required to check connection e.g. tokens, secrets and keys etc.
 67        :return: A tuple of (boolean, error). If boolean is true, then the connection check is successful
 68          and we can connect to the underlying data source using the provided configuration.
 69          Otherwise, the input config cannot be used to connect to the underlying data source,
 70          and the "error" object should describe what went wrong.
 71          The error object will be cast to string to display the problem to the user.
 72        """
 73
 74    @abstractmethod
 75    def streams(self, config: Mapping[str, Any]) -> List[Stream]:
 76        """
 77        :param config: The user-provided configuration as specified by the source's spec.
 78        Any stream construction related operation should happen here.
 79        :return: A list of the streams in this source connector.
 80        """
 81
 82    # Stream name to instance map for applying output object transformation
 83    _stream_to_instance_map: Dict[str, Stream] = {}
 84    _slice_logger: SliceLogger = DebugSliceLogger()
 85
 86    def discover(self, logger: logging.Logger, config: Mapping[str, Any]) -> AirbyteCatalog:
 87        """Implements the Discover operation from the Airbyte Specification.
 88        See https://docs.airbyte.com/understanding-airbyte/airbyte-protocol/#discover.
 89        """
 90        streams = [stream.as_airbyte_stream() for stream in self.streams(config=config)]
 91        return AirbyteCatalog(streams=streams)
 92
 93    def check(self, logger: logging.Logger, config: Mapping[str, Any]) -> AirbyteConnectionStatus:
 94        """Implements the Check Connection operation from the Airbyte Specification.
 95        See https://docs.airbyte.com/understanding-airbyte/airbyte-protocol/#check.
 96        """
 97        check_succeeded, error = self.check_connection(logger, config)
 98        if not check_succeeded:
 99            return AirbyteConnectionStatus(status=Status.FAILED, message=repr(error))
100        return AirbyteConnectionStatus(status=Status.SUCCEEDED)
101
102    def read(
103        self,
104        logger: logging.Logger,
105        config: Mapping[str, Any],
106        catalog: ConfiguredAirbyteCatalog,
107        state: Optional[List[AirbyteStateMessage]] = None,
108    ) -> Iterator[AirbyteMessage]:
109        """Implements the Read operation from the Airbyte Specification. See https://docs.airbyte.com/understanding-airbyte/airbyte-protocol/."""
110        logger.info(f"Starting syncing {self.name}")
111        config, internal_config = split_config(config)
112        # TODO assert all streams exist in the connector
113        # get the streams once in case the connector needs to make any queries to generate them
114        stream_instances = {s.name: s for s in self.streams(config)}
115        state_manager = ConnectorStateManager(state=state)
116        self._stream_to_instance_map = stream_instances
117
118        stream_name_to_exception: MutableMapping[str, AirbyteTracedException] = {}
119
120        with create_timer(self.name) as timer:
121            for configured_stream in catalog.streams:
122                stream_instance = stream_instances.get(configured_stream.stream.name)
123                is_stream_exist = bool(stream_instance)
124                try:
125                    # Used direct reference to `stream_instance` instead of `is_stream_exist` to avoid mypy type checking errors
126                    if not stream_instance:
127                        if not self.raise_exception_on_missing_stream:
128                            yield stream_status_as_airbyte_message(
129                                configured_stream.stream, AirbyteStreamStatus.INCOMPLETE
130                            )
131                            continue
132
133                        error_message = (
134                            f"The stream '{configured_stream.stream.name}' in your connection configuration was not found in the source. "
135                            f"Refresh the schema in your replication settings and remove this stream from future sync attempts."
136                        )
137
138                        # Use configured_stream as stream_instance to support references in error handling.
139                        stream_instance = configured_stream.stream
140
141                        raise AirbyteTracedException(
142                            message="A stream listed in your configuration was not found in the source. Please check the logs for more "
143                            "details.",
144                            internal_message=error_message,
145                            failure_type=FailureType.config_error,
146                        )
147
148                    timer.start_event(f"Syncing stream {configured_stream.stream.name}")
149                    logger.info(f"Marking stream {configured_stream.stream.name} as STARTED")
150                    yield stream_status_as_airbyte_message(
151                        configured_stream.stream, AirbyteStreamStatus.STARTED
152                    )
153                    yield from self._read_stream(
154                        logger=logger,
155                        stream_instance=stream_instance,
156                        configured_stream=configured_stream,
157                        state_manager=state_manager,
158                        internal_config=internal_config,
159                    )
160                    logger.info(f"Marking stream {configured_stream.stream.name} as STOPPED")
161                    yield stream_status_as_airbyte_message(
162                        configured_stream.stream, AirbyteStreamStatus.COMPLETE
163                    )
164
165                except Exception as e:
166                    yield from self._emit_queued_messages()
167                    logger.exception(
168                        f"Encountered an exception while reading stream {configured_stream.stream.name}"
169                    )
170                    logger.info(f"Marking stream {configured_stream.stream.name} as STOPPED")
171                    yield stream_status_as_airbyte_message(
172                        configured_stream.stream, AirbyteStreamStatus.INCOMPLETE
173                    )
174
175                    stream_descriptor = StreamDescriptor(name=configured_stream.stream.name)
176
177                    if isinstance(e, AirbyteTracedException):
178                        traced_exception = e
179                        info_message = f"Stopping sync on error from stream {configured_stream.stream.name} because {self.name} does not support continuing syncs on error."
180                    else:
181                        traced_exception = self._serialize_exception(
182                            stream_descriptor, e, stream_instance=stream_instance
183                        )
184                        info_message = f"{self.name} does not support continuing syncs on error from stream {configured_stream.stream.name}"
185
186                    yield traced_exception.as_sanitized_airbyte_message(
187                        stream_descriptor=stream_descriptor
188                    )
189                    stream_name_to_exception[stream_instance.name] = traced_exception  # type: ignore # use configured_stream if stream_instance is None
190                    if self.stop_sync_on_stream_failure:
191                        logger.info(info_message)
192                        break
193                finally:
194                    # Finish read event only if the stream instance exists;
195                    # otherwise, there's no need as it never started
196                    if is_stream_exist:
197                        timer.finish_event()
198                        logger.info(f"Finished syncing {configured_stream.stream.name}")
199                        logger.info(timer.report())
200
201        if len(stream_name_to_exception) > 0:
202            error_message = generate_failed_streams_error_message(
203                {key: [value] for key, value in stream_name_to_exception.items()}
204            )
205            logger.info(error_message)
206            # We still raise at least one exception when a stream raises an exception because the platform currently relies
207            # on a non-zero exit code to determine if a sync attempt has failed. We also raise the exception as a config_error
208            # type because this combined error isn't actionable, but rather the previously emitted individual errors.
209            raise AirbyteTracedException(
210                message=error_message, failure_type=FailureType.config_error
211            )
212        logger.info(f"Finished syncing {self.name}")
213
214    @staticmethod
215    def _serialize_exception(
216        stream_descriptor: StreamDescriptor, e: Exception, stream_instance: Optional[Stream] = None
217    ) -> AirbyteTracedException:
218        display_message = stream_instance.get_error_display_message(e) if stream_instance else None
219        if display_message:
220            return AirbyteTracedException.from_exception(
221                e, message=display_message, stream_descriptor=stream_descriptor
222            )
223        return AirbyteTracedException.from_exception(e, stream_descriptor=stream_descriptor)
224
225    @property
226    def raise_exception_on_missing_stream(self) -> bool:
227        return False
228
229    def _read_stream(
230        self,
231        logger: logging.Logger,
232        stream_instance: Stream,
233        configured_stream: ConfiguredAirbyteStream,
234        state_manager: ConnectorStateManager,
235        internal_config: InternalConfig,
236    ) -> Iterator[AirbyteMessage]:
237        if internal_config.page_size and isinstance(stream_instance, HttpStream):
238            logger.info(
239                f"Setting page size for {stream_instance.name} to {internal_config.page_size}"
240            )
241            stream_instance.page_size = internal_config.page_size
242        logger.debug(
243            f"Syncing configured stream: {configured_stream.stream.name}",
244            extra={
245                "sync_mode": configured_stream.sync_mode,
246                "primary_key": configured_stream.primary_key,
247                "cursor_field": configured_stream.cursor_field,
248            },
249        )
250        stream_instance.log_stream_sync_configuration()
251
252        stream_name = configured_stream.stream.name
253        stream_state = state_manager.get_stream_state(stream_name, stream_instance.namespace)
254
255        # This is a hack. Existing full refresh streams that are converted into resumable full refresh need to discard
256        # the state because the terminal state for a full refresh sync is not compatible with substream resumable full
257        # refresh state. This is only required when running live traffic regression testing since the platform normally
258        # handles whether to pass state
259        if stream_state == {"__ab_no_cursor_state_message": True}:
260            stream_state = {}
261
262        if "state" in dir(stream_instance):
263            stream_instance.state = stream_state  # type: ignore # we check that state in the dir(stream_instance)
264            logger.info(f"Setting state of {self.name} stream to {stream_state}")
265
266        record_iterator = stream_instance.read(
267            configured_stream,
268            logger,
269            self._slice_logger,
270            stream_state,
271            state_manager,
272            internal_config,
273        )
274
275        record_counter = 0
276        logger.info(f"Syncing stream: {stream_name} ")
277        for record_data_or_message in record_iterator:
278            record = self._get_message(record_data_or_message, stream_instance)
279            if record.type == MessageType.RECORD:
280                record_counter += 1
281                if record_counter == 1:
282                    logger.info(f"Marking stream {stream_name} as RUNNING")
283                    # If we just read the first record of the stream, emit the transition to the RUNNING state
284                    yield stream_status_as_airbyte_message(
285                        configured_stream.stream, AirbyteStreamStatus.RUNNING
286                    )
287            yield from self._emit_queued_messages()
288            yield record
289
290        logger.info(f"Read {record_counter} records from {stream_name} stream")
291
292    def _emit_queued_messages(self) -> Iterable[AirbyteMessage]:
293        if self.message_repository:
294            yield from self.message_repository.consume_queue()
295        return
296
297    def _get_message(
298        self, record_data_or_message: Union[StreamData, AirbyteMessage], stream: Stream
299    ) -> AirbyteMessage:
300        """
301        Converts the input to an AirbyteMessage if it is a StreamData. Returns the input as is if it is already an AirbyteMessage
302        """
303        match record_data_or_message:
304            case AirbyteMessage():
305                return record_data_or_message
306            case _:
307                return stream_data_to_airbyte_message(
308                    stream.name,
309                    record_data_or_message,
310                    stream.transformer,
311                    stream.get_json_schema(),
312                )
313
314    @property
315    def message_repository(self) -> Union[None, MessageRepository]:
316        return _default_message_repository
317
318    @property
319    def stop_sync_on_stream_failure(self) -> bool:
320        """
321        WARNING: This function is in-development which means it is subject to change. Use at your own risk.
322
323        By default, when a source encounters an exception while syncing a stream, it will emit an error trace message and then
324        continue syncing the next stream. This can be overwritten on a per-source basis so that the source will stop the sync
325        on the first error seen and emit a single error trace message for that stream.
326        """
327        return False

Abstract base class for an Airbyte Source. Consumers should implement any abstract methods in this class to create an Airbyte Specification compliant Source.

@abstractmethod
def check_connection( self, logger: logging.Logger, config: Mapping[str, Any]) -> Tuple[bool, Optional[Any]]:
59    @abstractmethod
60    def check_connection(
61        self, logger: logging.Logger, config: Mapping[str, Any]
62    ) -> Tuple[bool, Optional[Any]]:
63        """
64        :param logger: source logger
65        :param config: The user-provided configuration as specified by the source's spec.
66          This usually contains information required to check connection e.g. tokens, secrets and keys etc.
67        :return: A tuple of (boolean, error). If boolean is true, then the connection check is successful
68          and we can connect to the underlying data source using the provided configuration.
69          Otherwise, the input config cannot be used to connect to the underlying data source,
70          and the "error" object should describe what went wrong.
71          The error object will be cast to string to display the problem to the user.
72        """
Parameters
  • logger: source logger
  • config: The user-provided configuration as specified by the source's spec. This usually contains information required to check connection e.g. tokens, secrets and keys etc.
Returns

A tuple of (boolean, error). If boolean is true, then the connection check is successful and we can connect to the underlying data source using the provided configuration. Otherwise, the input config cannot be used to connect to the underlying data source, and the "error" object should describe what went wrong. The error object will be cast to string to display the problem to the user.

@abstractmethod
def streams( self, config: Mapping[str, Any]) -> List[Stream]:
74    @abstractmethod
75    def streams(self, config: Mapping[str, Any]) -> List[Stream]:
76        """
77        :param config: The user-provided configuration as specified by the source's spec.
78        Any stream construction related operation should happen here.
79        :return: A list of the streams in this source connector.
80        """
Parameters
  • config: The user-provided configuration as specified by the source's spec. Any stream construction related operation should happen here.
Returns

A list of the streams in this source connector.

def discover( self, logger: logging.Logger, config: Mapping[str, Any]) -> airbyte_protocol_dataclasses.models.airbyte_protocol.AirbyteCatalog:
86    def discover(self, logger: logging.Logger, config: Mapping[str, Any]) -> AirbyteCatalog:
87        """Implements the Discover operation from the Airbyte Specification.
88        See https://docs.airbyte.com/understanding-airbyte/airbyte-protocol/#discover.
89        """
90        streams = [stream.as_airbyte_stream() for stream in self.streams(config=config)]
91        return AirbyteCatalog(streams=streams)

Implements the Discover operation from the Airbyte Specification. See https://docs.airbyte.com/understanding-airbyte/airbyte-protocol/#discover.

def check( self, logger: logging.Logger, config: Mapping[str, Any]) -> airbyte_protocol_dataclasses.models.airbyte_protocol.AirbyteConnectionStatus:
 93    def check(self, logger: logging.Logger, config: Mapping[str, Any]) -> AirbyteConnectionStatus:
 94        """Implements the Check Connection operation from the Airbyte Specification.
 95        See https://docs.airbyte.com/understanding-airbyte/airbyte-protocol/#check.
 96        """
 97        check_succeeded, error = self.check_connection(logger, config)
 98        if not check_succeeded:
 99            return AirbyteConnectionStatus(status=Status.FAILED, message=repr(error))
100        return AirbyteConnectionStatus(status=Status.SUCCEEDED)

Implements the Check Connection operation from the Airbyte Specification. See https://docs.airbyte.com/understanding-airbyte/airbyte-protocol/#check.

def read( self, logger: logging.Logger, config: Mapping[str, Any], catalog: airbyte_protocol_dataclasses.models.airbyte_protocol.ConfiguredAirbyteCatalog, state: Optional[List[airbyte_cdk.models.airbyte_protocol.AirbyteStateMessage]] = None) -> Iterator[AirbyteMessage]:
102    def read(
103        self,
104        logger: logging.Logger,
105        config: Mapping[str, Any],
106        catalog: ConfiguredAirbyteCatalog,
107        state: Optional[List[AirbyteStateMessage]] = None,
108    ) -> Iterator[AirbyteMessage]:
109        """Implements the Read operation from the Airbyte Specification. See https://docs.airbyte.com/understanding-airbyte/airbyte-protocol/."""
110        logger.info(f"Starting syncing {self.name}")
111        config, internal_config = split_config(config)
112        # TODO assert all streams exist in the connector
113        # get the streams once in case the connector needs to make any queries to generate them
114        stream_instances = {s.name: s for s in self.streams(config)}
115        state_manager = ConnectorStateManager(state=state)
116        self._stream_to_instance_map = stream_instances
117
118        stream_name_to_exception: MutableMapping[str, AirbyteTracedException] = {}
119
120        with create_timer(self.name) as timer:
121            for configured_stream in catalog.streams:
122                stream_instance = stream_instances.get(configured_stream.stream.name)
123                is_stream_exist = bool(stream_instance)
124                try:
125                    # Used direct reference to `stream_instance` instead of `is_stream_exist` to avoid mypy type checking errors
126                    if not stream_instance:
127                        if not self.raise_exception_on_missing_stream:
128                            yield stream_status_as_airbyte_message(
129                                configured_stream.stream, AirbyteStreamStatus.INCOMPLETE
130                            )
131                            continue
132
133                        error_message = (
134                            f"The stream '{configured_stream.stream.name}' in your connection configuration was not found in the source. "
135                            f"Refresh the schema in your replication settings and remove this stream from future sync attempts."
136                        )
137
138                        # Use configured_stream as stream_instance to support references in error handling.
139                        stream_instance = configured_stream.stream
140
141                        raise AirbyteTracedException(
142                            message="A stream listed in your configuration was not found in the source. Please check the logs for more "
143                            "details.",
144                            internal_message=error_message,
145                            failure_type=FailureType.config_error,
146                        )
147
148                    timer.start_event(f"Syncing stream {configured_stream.stream.name}")
149                    logger.info(f"Marking stream {configured_stream.stream.name} as STARTED")
150                    yield stream_status_as_airbyte_message(
151                        configured_stream.stream, AirbyteStreamStatus.STARTED
152                    )
153                    yield from self._read_stream(
154                        logger=logger,
155                        stream_instance=stream_instance,
156                        configured_stream=configured_stream,
157                        state_manager=state_manager,
158                        internal_config=internal_config,
159                    )
160                    logger.info(f"Marking stream {configured_stream.stream.name} as STOPPED")
161                    yield stream_status_as_airbyte_message(
162                        configured_stream.stream, AirbyteStreamStatus.COMPLETE
163                    )
164
165                except Exception as e:
166                    yield from self._emit_queued_messages()
167                    logger.exception(
168                        f"Encountered an exception while reading stream {configured_stream.stream.name}"
169                    )
170                    logger.info(f"Marking stream {configured_stream.stream.name} as STOPPED")
171                    yield stream_status_as_airbyte_message(
172                        configured_stream.stream, AirbyteStreamStatus.INCOMPLETE
173                    )
174
175                    stream_descriptor = StreamDescriptor(name=configured_stream.stream.name)
176
177                    if isinstance(e, AirbyteTracedException):
178                        traced_exception = e
179                        info_message = f"Stopping sync on error from stream {configured_stream.stream.name} because {self.name} does not support continuing syncs on error."
180                    else:
181                        traced_exception = self._serialize_exception(
182                            stream_descriptor, e, stream_instance=stream_instance
183                        )
184                        info_message = f"{self.name} does not support continuing syncs on error from stream {configured_stream.stream.name}"
185
186                    yield traced_exception.as_sanitized_airbyte_message(
187                        stream_descriptor=stream_descriptor
188                    )
189                    stream_name_to_exception[stream_instance.name] = traced_exception  # type: ignore # use configured_stream if stream_instance is None
190                    if self.stop_sync_on_stream_failure:
191                        logger.info(info_message)
192                        break
193                finally:
194                    # Finish read event only if the stream instance exists;
195                    # otherwise, there's no need as it never started
196                    if is_stream_exist:
197                        timer.finish_event()
198                        logger.info(f"Finished syncing {configured_stream.stream.name}")
199                        logger.info(timer.report())
200
201        if len(stream_name_to_exception) > 0:
202            error_message = generate_failed_streams_error_message(
203                {key: [value] for key, value in stream_name_to_exception.items()}
204            )
205            logger.info(error_message)
206            # We still raise at least one exception when a stream raises an exception because the platform currently relies
207            # on a non-zero exit code to determine if a sync attempt has failed. We also raise the exception as a config_error
208            # type because this combined error isn't actionable, but rather the previously emitted individual errors.
209            raise AirbyteTracedException(
210                message=error_message, failure_type=FailureType.config_error
211            )
212        logger.info(f"Finished syncing {self.name}")

Implements the Read operation from the Airbyte Specification. See https://docs.airbyte.com/understanding-airbyte/airbyte-protocol/.

raise_exception_on_missing_stream: bool
225    @property
226    def raise_exception_on_missing_stream(self) -> bool:
227        return False
message_repository: Optional[MessageRepository]
314    @property
315    def message_repository(self) -> Union[None, MessageRepository]:
316        return _default_message_repository
stop_sync_on_stream_failure: bool
318    @property
319    def stop_sync_on_stream_failure(self) -> bool:
320        """
321        WARNING: This function is in-development which means it is subject to change. Use at your own risk.
322
323        By default, when a source encounters an exception while syncing a stream, it will emit an error trace message and then
324        continue syncing the next stream. This can be overwritten on a per-source basis so that the source will stop the sync
325        on the first error seen and emit a single error trace message for that stream.
326        """
327        return False

WARNING: This function is in-development which means it is subject to change. Use at your own risk.

By default, when a source encounters an exception while syncing a stream, it will emit an error trace message and then continue syncing the next stream. This can be overwritten on a per-source basis so that the source will stop the sync on the first error seen and emit a single error trace message for that stream.

class BaseConfig(pydantic.v1.main.BaseModel):
13class BaseConfig(BaseModel):
14    """Base class for connector spec, adds the following behaviour:
15
16    - resolve $ref and replace it with definition
17    - replace all occurrences of anyOf with oneOf
18    - drop description
19    """
20
21    @classmethod
22    def schema(cls, *args: Any, **kwargs: Any) -> Dict[str, Any]:
23        """We're overriding the schema classmethod to enable some post-processing"""
24        schema = super().schema(*args, **kwargs)
25        rename_key(schema, old_key="anyOf", new_key="oneOf")  # UI supports only oneOf
26        expand_refs(schema)
27        schema.pop("description", None)  # description added from the docstring
28        return schema

Base class for connector spec, adds the following behaviour:

  • resolve $ref and replace it with definition
  • replace all occurrences of anyOf with oneOf
  • drop description
@classmethod
def schema(cls, *args: Any, **kwargs: Any) -> Dict[str, Any]:
21    @classmethod
22    def schema(cls, *args: Any, **kwargs: Any) -> Dict[str, Any]:
23        """We're overriding the schema classmethod to enable some post-processing"""
24        schema = super().schema(*args, **kwargs)
25        rename_key(schema, old_key="anyOf", new_key="oneOf")  # UI supports only oneOf
26        expand_refs(schema)
27        schema.pop("description", None)  # description added from the docstring
28        return schema

We're overriding the schema classmethod to enable some post-processing

class BaseConnector(abc.ABC, typing.Generic[~TConfig]):
 34class BaseConnector(ABC, Generic[TConfig]):
 35    # configure whether the `check_config_against_spec_or_exit()` needs to be called
 36    check_config_against_spec: bool = True
 37
 38    @abstractmethod
 39    def configure(self, config: Mapping[str, Any], temp_dir: str) -> TConfig:
 40        """
 41        Persist config in temporary directory to run the Source job
 42        """
 43
 44    @staticmethod
 45    def read_config(config_path: str) -> MutableMapping[str, Any]:
 46        config = BaseConnector._read_json_file(config_path)
 47        if isinstance(config, MutableMapping):
 48            return config
 49        else:
 50            raise ValueError(
 51                f"The content of {config_path} is not an object and therefore is not a valid config. Please ensure the file represent a config."
 52            )
 53
 54    @staticmethod
 55    def _read_json_file(file_path: str) -> Any:
 56        with open(file_path, "r") as file:
 57            contents = file.read()
 58
 59        try:
 60            return json.loads(contents)
 61        except json.JSONDecodeError as error:
 62            raise ValueError(
 63                f"Could not read json file {file_path}: {error}. Please ensure that it is a valid JSON."
 64            )
 65
 66    @staticmethod
 67    def write_config(config: TConfig, config_path: str) -> None:
 68        with open(config_path, "w") as fh:
 69            fh.write(json.dumps(config))
 70
 71    def spec(self, logger: logging.Logger) -> ConnectorSpecification:
 72        """
 73        Returns the spec for this integration. The spec is a JSON-Schema object describing the required configurations (e.g: username and password)
 74        required to run this integration. By default, this will be loaded from a "spec.yaml" or a "spec.json" in the package root.
 75        """
 76
 77        package = self.__class__.__module__.split(".")[0]
 78
 79        yaml_spec = load_optional_package_file(package, "spec.yaml")
 80        json_spec = load_optional_package_file(package, "spec.json")
 81
 82        if yaml_spec and json_spec:
 83            raise RuntimeError(
 84                "Found multiple spec files in the package. Only one of spec.yaml or spec.json should be provided."
 85            )
 86
 87        if yaml_spec:
 88            spec_obj = yaml.load(yaml_spec, Loader=yaml.SafeLoader)
 89        elif json_spec:
 90            try:
 91                spec_obj = json.loads(json_spec)
 92            except json.JSONDecodeError as error:
 93                raise ValueError(
 94                    f"Could not read json spec file: {error}. Please ensure that it is a valid JSON."
 95                )
 96        else:
 97            raise FileNotFoundError("Unable to find spec.yaml or spec.json in the package.")
 98
 99        return ConnectorSpecificationSerializer.load(spec_obj)
100
101    @abstractmethod
102    def check(self, logger: logging.Logger, config: TConfig) -> AirbyteConnectionStatus:
103        """
104        Tests if the input configuration can be used to successfully connect to the integration e.g: if a provided Stripe API token can be used to connect
105        to the Stripe API.
106        """

Helper class that provides a standard way to create an ABC using inheritance.

check_config_against_spec: bool = True
@abstractmethod
def configure(self, config: Mapping[str, Any], temp_dir: str) -> ~TConfig:
38    @abstractmethod
39    def configure(self, config: Mapping[str, Any], temp_dir: str) -> TConfig:
40        """
41        Persist config in temporary directory to run the Source job
42        """

Persist config in temporary directory to run the Source job

@staticmethod
def read_config(config_path: str) -> MutableMapping[str, Any]:
44    @staticmethod
45    def read_config(config_path: str) -> MutableMapping[str, Any]:
46        config = BaseConnector._read_json_file(config_path)
47        if isinstance(config, MutableMapping):
48            return config
49        else:
50            raise ValueError(
51                f"The content of {config_path} is not an object and therefore is not a valid config. Please ensure the file represent a config."
52            )
@staticmethod
def write_config(config: ~TConfig, config_path: str) -> None:
66    @staticmethod
67    def write_config(config: TConfig, config_path: str) -> None:
68        with open(config_path, "w") as fh:
69            fh.write(json.dumps(config))
def spec( self, logger: logging.Logger) -> ConnectorSpecification:
71    def spec(self, logger: logging.Logger) -> ConnectorSpecification:
72        """
73        Returns the spec for this integration. The spec is a JSON-Schema object describing the required configurations (e.g: username and password)
74        required to run this integration. By default, this will be loaded from a "spec.yaml" or a "spec.json" in the package root.
75        """
76
77        package = self.__class__.__module__.split(".")[0]
78
79        yaml_spec = load_optional_package_file(package, "spec.yaml")
80        json_spec = load_optional_package_file(package, "spec.json")
81
82        if yaml_spec and json_spec:
83            raise RuntimeError(
84                "Found multiple spec files in the package. Only one of spec.yaml or spec.json should be provided."
85            )
86
87        if yaml_spec:
88            spec_obj = yaml.load(yaml_spec, Loader=yaml.SafeLoader)
89        elif json_spec:
90            try:
91                spec_obj = json.loads(json_spec)
92            except json.JSONDecodeError as error:
93                raise ValueError(
94                    f"Could not read json spec file: {error}. Please ensure that it is a valid JSON."
95                )
96        else:
97            raise FileNotFoundError("Unable to find spec.yaml or spec.json in the package.")
98
99        return ConnectorSpecificationSerializer.load(spec_obj)

Returns the spec for this integration. The spec is a JSON-Schema object describing the required configurations (e.g: username and password) required to run this integration. By default, this will be loaded from a "spec.yaml" or a "spec.json" in the package root.

@abstractmethod
def check( self, logger: logging.Logger, config: ~TConfig) -> airbyte_protocol_dataclasses.models.airbyte_protocol.AirbyteConnectionStatus:
101    @abstractmethod
102    def check(self, logger: logging.Logger, config: TConfig) -> AirbyteConnectionStatus:
103        """
104        Tests if the input configuration can be used to successfully connect to the integration e.g: if a provided Stripe API token can be used to connect
105        to the Stripe API.
106        """

Tests if the input configuration can be used to successfully connect to the integration e.g: if a provided Stripe API token can be used to connect to the Stripe API.

124class Connector(DefaultConnectorMixin, BaseConnector[Mapping[str, Any]], ABC): ...

Helper class that provides a standard way to create an ABC using inheritance.

 31class Destination(Connector, ABC):
 32    VALID_CMDS = {"spec", "check", "write"}
 33
 34    @abstractmethod
 35    def write(
 36        self,
 37        config: Mapping[str, Any],
 38        configured_catalog: ConfiguredAirbyteCatalog,
 39        input_messages: Iterable[AirbyteMessage],
 40    ) -> Iterable[AirbyteMessage]:
 41        """Implement to define how the connector writes data to the destination"""
 42
 43    def _run_check(self, config: Mapping[str, Any]) -> AirbyteMessage:
 44        check_result = self.check(logger, config)
 45        return AirbyteMessage(type=Type.CONNECTION_STATUS, connectionStatus=check_result)
 46
 47    def _parse_input_stream(self, input_stream: io.TextIOWrapper) -> Iterable[AirbyteMessage]:
 48        """Reads from stdin, converting to Airbyte messages"""
 49        for line in input_stream:
 50            try:
 51                yield AirbyteMessageSerializer.load(orjson.loads(line))
 52            except orjson.JSONDecodeError:
 53                logger.info(
 54                    f"ignoring input which can't be deserialized as Airbyte Message: {line}"
 55                )
 56
 57    def _run_write(
 58        self,
 59        config: Mapping[str, Any],
 60        configured_catalog_path: str,
 61        input_stream: io.TextIOWrapper,
 62    ) -> Iterable[AirbyteMessage]:
 63        catalog = ConfiguredAirbyteCatalogSerializer.load(
 64            orjson.loads(open(configured_catalog_path).read())
 65        )
 66        input_messages = self._parse_input_stream(input_stream)
 67        logger.info("Begin writing to the destination...")
 68        yield from self.write(
 69            config=config, configured_catalog=catalog, input_messages=input_messages
 70        )
 71        logger.info("Writing complete.")
 72
 73    def parse_args(self, args: List[str]) -> argparse.Namespace:
 74        """
 75        :param args: commandline arguments
 76        :return:
 77        """
 78
 79        parent_parser = argparse.ArgumentParser(add_help=False)
 80        main_parser = argparse.ArgumentParser()
 81        subparsers = main_parser.add_subparsers(title="commands", dest="command")
 82
 83        # spec
 84        subparsers.add_parser(
 85            "spec", help="outputs the json configuration specification", parents=[parent_parser]
 86        )
 87
 88        # check
 89        check_parser = subparsers.add_parser(
 90            "check", help="checks the config can be used to connect", parents=[parent_parser]
 91        )
 92        required_check_parser = check_parser.add_argument_group("required named arguments")
 93        required_check_parser.add_argument(
 94            "--config", type=str, required=True, help="path to the json configuration file"
 95        )
 96
 97        # write
 98        write_parser = subparsers.add_parser(
 99            "write", help="Writes data to the destination", parents=[parent_parser]
100        )
101        write_required = write_parser.add_argument_group("required named arguments")
102        write_required.add_argument(
103            "--config", type=str, required=True, help="path to the JSON configuration file"
104        )
105        write_required.add_argument(
106            "--catalog", type=str, required=True, help="path to the configured catalog JSON file"
107        )
108
109        parsed_args = main_parser.parse_args(args)
110        cmd = parsed_args.command
111        if not cmd:
112            raise Exception("No command entered. ")
113        elif cmd not in ["spec", "check", "write"]:
114            # This is technically dead code since parse_args() would fail if this was the case
115            # But it's non-obvious enough to warrant placing it here anyways
116            raise Exception(f"Unknown command entered: {cmd}")
117
118        return parsed_args
119
120    def run_cmd(self, parsed_args: argparse.Namespace) -> Iterable[AirbyteMessage]:
121        cmd = parsed_args.command
122        if cmd not in self.VALID_CMDS:
123            raise Exception(f"Unrecognized command: {cmd}")
124
125        spec = self.spec(logger)
126        if cmd == "spec":
127            yield AirbyteMessage(type=Type.SPEC, spec=spec)
128            return
129        config = self.read_config(config_path=parsed_args.config)
130        # Register the config's secrets with the log filter before anything can print them,
131        # as the source entrypoint does. A config validation error quotes config values.
132        update_secrets(get_secrets(spec.connectionSpecification, config))
133        if self.check_config_against_spec or cmd == "check":
134            try:
135                check_config_against_spec_or_exit(config, spec)
136            except AirbyteTracedException as traced_exc:
137                connection_status = traced_exc.as_connection_status_message()
138                if connection_status and cmd == "check":
139                    yield connection_status
140                    return
141                raise traced_exc
142
143        if cmd == "check":
144            yield self._run_check(config=config)
145        elif cmd == "write":
146            # Wrap in UTF-8 to override any other input encodings
147            wrapped_stdin = io.TextIOWrapper(sys.stdin.buffer, encoding="utf-8")
148            yield from self._run_write(
149                config=config,
150                configured_catalog_path=parsed_args.catalog,
151                input_stream=wrapped_stdin,
152            )
153
154    def run(self, args: List[str]) -> None:
155        init_uncaught_exception_handler(logger)
156        parsed_args = self.parse_args(args)
157        output_messages = self.run_cmd(parsed_args)
158        for message in output_messages:
159            print(orjson.dumps(AirbyteMessageSerializer.dump(message)).decode())

Helper class that provides a standard way to create an ABC using inheritance.

VALID_CMDS = {'check', 'write', 'spec'}
@abstractmethod
def write( self, config: Mapping[str, Any], configured_catalog: airbyte_protocol_dataclasses.models.airbyte_protocol.ConfiguredAirbyteCatalog, input_messages: Iterable[AirbyteMessage]) -> Iterable[AirbyteMessage]:
34    @abstractmethod
35    def write(
36        self,
37        config: Mapping[str, Any],
38        configured_catalog: ConfiguredAirbyteCatalog,
39        input_messages: Iterable[AirbyteMessage],
40    ) -> Iterable[AirbyteMessage]:
41        """Implement to define how the connector writes data to the destination"""

Implement to define how the connector writes data to the destination

def parse_args(self, args: List[str]) -> argparse.Namespace:
 73    def parse_args(self, args: List[str]) -> argparse.Namespace:
 74        """
 75        :param args: commandline arguments
 76        :return:
 77        """
 78
 79        parent_parser = argparse.ArgumentParser(add_help=False)
 80        main_parser = argparse.ArgumentParser()
 81        subparsers = main_parser.add_subparsers(title="commands", dest="command")
 82
 83        # spec
 84        subparsers.add_parser(
 85            "spec", help="outputs the json configuration specification", parents=[parent_parser]
 86        )
 87
 88        # check
 89        check_parser = subparsers.add_parser(
 90            "check", help="checks the config can be used to connect", parents=[parent_parser]
 91        )
 92        required_check_parser = check_parser.add_argument_group("required named arguments")
 93        required_check_parser.add_argument(
 94            "--config", type=str, required=True, help="path to the json configuration file"
 95        )
 96
 97        # write
 98        write_parser = subparsers.add_parser(
 99            "write", help="Writes data to the destination", parents=[parent_parser]
100        )
101        write_required = write_parser.add_argument_group("required named arguments")
102        write_required.add_argument(
103            "--config", type=str, required=True, help="path to the JSON configuration file"
104        )
105        write_required.add_argument(
106            "--catalog", type=str, required=True, help="path to the configured catalog JSON file"
107        )
108
109        parsed_args = main_parser.parse_args(args)
110        cmd = parsed_args.command
111        if not cmd:
112            raise Exception("No command entered. ")
113        elif cmd not in ["spec", "check", "write"]:
114            # This is technically dead code since parse_args() would fail if this was the case
115            # But it's non-obvious enough to warrant placing it here anyways
116            raise Exception(f"Unknown command entered: {cmd}")
117
118        return parsed_args
Parameters
  • args: commandline arguments
Returns
def run_cmd( self, parsed_args: argparse.Namespace) -> Iterable[AirbyteMessage]:
120    def run_cmd(self, parsed_args: argparse.Namespace) -> Iterable[AirbyteMessage]:
121        cmd = parsed_args.command
122        if cmd not in self.VALID_CMDS:
123            raise Exception(f"Unrecognized command: {cmd}")
124
125        spec = self.spec(logger)
126        if cmd == "spec":
127            yield AirbyteMessage(type=Type.SPEC, spec=spec)
128            return
129        config = self.read_config(config_path=parsed_args.config)
130        # Register the config's secrets with the log filter before anything can print them,
131        # as the source entrypoint does. A config validation error quotes config values.
132        update_secrets(get_secrets(spec.connectionSpecification, config))
133        if self.check_config_against_spec or cmd == "check":
134            try:
135                check_config_against_spec_or_exit(config, spec)
136            except AirbyteTracedException as traced_exc:
137                connection_status = traced_exc.as_connection_status_message()
138                if connection_status and cmd == "check":
139                    yield connection_status
140                    return
141                raise traced_exc
142
143        if cmd == "check":
144            yield self._run_check(config=config)
145        elif cmd == "write":
146            # Wrap in UTF-8 to override any other input encodings
147            wrapped_stdin = io.TextIOWrapper(sys.stdin.buffer, encoding="utf-8")
148            yield from self._run_write(
149                config=config,
150                configured_catalog_path=parsed_args.catalog,
151                input_stream=wrapped_stdin,
152            )
def run(self, args: List[str]) -> None:
154    def run(self, args: List[str]) -> None:
155        init_uncaught_exception_handler(logger)
156        parsed_args = self.parse_args(args)
157        output_messages = self.run_cmd(parsed_args)
158        for message in output_messages:
159            print(orjson.dumps(AirbyteMessageSerializer.dump(message)).decode())
56class Source(
57    DefaultConnectorMixin,
58    BaseSource[Mapping[str, Any], List[AirbyteStateMessage], ConfiguredAirbyteCatalog],
59    ABC,
60):
61    # can be overridden to change an input state.
62    @classmethod
63    def read_state(cls, state_path: str) -> List[AirbyteStateMessage]:
64        """
65        Retrieves the input state of a sync by reading from the specified JSON file. Incoming state can be deserialized into either
66        a JSON object for legacy state input or as a list of AirbyteStateMessages for the per-stream state format. Regardless of the
67        incoming input type, it will always be transformed and output as a list of AirbyteStateMessage(s).
68        :param state_path: The filepath to where the stream states are located
69        :return: The complete stream state based on the connector's previous sync
70        """
71        parsed_state_messages = []
72        if state_path:
73            state_obj = BaseConnector._read_json_file(state_path)
74            if state_obj:
75                for state in state_obj:  # type: ignore  # `isinstance(state_obj, List)` ensures that this is a list
76                    parsed_message = AirbyteStateMessageSerializer.load(state)
77                    if (
78                        not parsed_message.stream
79                        and not parsed_message.data
80                        and not parsed_message.global_
81                    ):
82                        raise ValueError(
83                            "AirbyteStateMessage should contain either a stream, global, or state field"
84                        )
85                    parsed_state_messages.append(parsed_message)
86        return parsed_state_messages
87
88    # can be overridden to change an input catalog
89    @classmethod
90    def read_catalog(cls, catalog_path: str) -> ConfiguredAirbyteCatalog:
91        return ConfiguredAirbyteCatalogSerializer.load(cls._read_json_file(catalog_path))
92
93    @property
94    def name(self) -> str:
95        """Source name"""
96        return self.__class__.__name__

Helper class that provides a standard way to create an ABC using inheritance.

@classmethod
def read_state( cls, state_path: str) -> List[airbyte_cdk.models.airbyte_protocol.AirbyteStateMessage]:
62    @classmethod
63    def read_state(cls, state_path: str) -> List[AirbyteStateMessage]:
64        """
65        Retrieves the input state of a sync by reading from the specified JSON file. Incoming state can be deserialized into either
66        a JSON object for legacy state input or as a list of AirbyteStateMessages for the per-stream state format. Regardless of the
67        incoming input type, it will always be transformed and output as a list of AirbyteStateMessage(s).
68        :param state_path: The filepath to where the stream states are located
69        :return: The complete stream state based on the connector's previous sync
70        """
71        parsed_state_messages = []
72        if state_path:
73            state_obj = BaseConnector._read_json_file(state_path)
74            if state_obj:
75                for state in state_obj:  # type: ignore  # `isinstance(state_obj, List)` ensures that this is a list
76                    parsed_message = AirbyteStateMessageSerializer.load(state)
77                    if (
78                        not parsed_message.stream
79                        and not parsed_message.data
80                        and not parsed_message.global_
81                    ):
82                        raise ValueError(
83                            "AirbyteStateMessage should contain either a stream, global, or state field"
84                        )
85                    parsed_state_messages.append(parsed_message)
86        return parsed_state_messages

Retrieves the input state of a sync by reading from the specified JSON file. Incoming state can be deserialized into either a JSON object for legacy state input or as a list of AirbyteStateMessages for the per-stream state format. Regardless of the incoming input type, it will always be transformed and output as a list of AirbyteStateMessage(s).

Parameters
  • state_path: The filepath to where the stream states are located
Returns

The complete stream state based on the connector's previous sync

@classmethod
def read_catalog( cls, catalog_path: str) -> airbyte_protocol_dataclasses.models.airbyte_protocol.ConfiguredAirbyteCatalog:
89    @classmethod
90    def read_catalog(cls, catalog_path: str) -> ConfiguredAirbyteCatalog:
91        return ConfiguredAirbyteCatalogSerializer.load(cls._read_json_file(catalog_path))
name: str
93    @property
94    def name(self) -> str:
95        """Source name"""
96        return self.__class__.__name__

Source name

@dataclass
class AddFields(airbyte_cdk.RecordTransformation):
 37@dataclass
 38class AddFields(RecordTransformation):
 39    """
 40    Transformation which adds field to an output record. The path of the added field can be nested. Adding nested fields will create all
 41    necessary parent objects (like mkdir -p). Adding fields to an array will extend the array to that index (filling intermediate
 42    indices with null values). So if you add a field at index 5 to the array ["value"], it will become ["value", null, null, null, null,
 43    "new_value"].
 44
 45
 46    This transformation has access to the following contextual values:
 47        record: the record about to be output by the connector
 48        config: the input configuration provided to a connector
 49        stream_state: the current state of the stream
 50        stream_slice: the current stream slice being read
 51
 52
 53
 54    Examples of instantiating this transformation via YAML:
 55    - type: AddFields
 56      fields:
 57        # hardcoded constant
 58        - path: ["path"]
 59          value: "static_value"
 60
 61        # nested path
 62        - path: ["path", "to", "field"]
 63          value: "static"
 64
 65        # from config
 66        - path: ["shop_id"]
 67          value: "{{ config.shop_id }}"
 68
 69        # from stream_interval
 70        - path: ["date"]
 71          value: "{{ stream_interval.start_date }}"
 72
 73        # from record
 74        - path: ["unnested_value"]
 75          value: {{ record.nested.field }}
 76
 77        # from stream_slice
 78        - path: ["start_date"]
 79          value: {{ stream_slice.start_date }}
 80
 81        # by supplying any valid Jinja template directive or expression https://jinja.palletsprojects.com/en/3.1.x/templates/#
 82        - path: ["two_times_two"]
 83          value: {{ 2 * 2 }}
 84
 85    Attributes:
 86        fields (List[AddedFieldDefinition]): A list of transformations (path and corresponding value) that will be added to the record
 87    """
 88
 89    fields: List[AddedFieldDefinition]
 90    parameters: InitVar[Mapping[str, Any]]
 91    condition: str = ""
 92    _parsed_fields: List[ParsedAddFieldDefinition] = field(
 93        init=False, repr=False, default_factory=list
 94    )
 95
 96    def __post_init__(self, parameters: Mapping[str, Any]) -> None:
 97        self._filter_interpolator = InterpolatedBoolean(
 98            condition=self.condition, parameters=parameters
 99        )
100
101        for add_field in self.fields:
102            if len(add_field.path) < 1:
103                raise ValueError(
104                    f"Expected a non-zero-length path for the AddFields transformation {add_field}"
105                )
106
107            if not isinstance(add_field.value, InterpolatedString):
108                if not isinstance(add_field.value, str):
109                    raise f"Expected a string value for the AddFields transformation: {add_field}"
110                else:
111                    self._parsed_fields.append(
112                        ParsedAddFieldDefinition(
113                            add_field.path,
114                            InterpolatedString.create(add_field.value, parameters=parameters),
115                            value_type=add_field.value_type,
116                            parameters=parameters,
117                        )
118                    )
119            else:
120                self._parsed_fields.append(
121                    ParsedAddFieldDefinition(
122                        add_field.path,
123                        add_field.value,
124                        value_type=add_field.value_type,
125                        parameters={},
126                    )
127                )
128
129    def transform(
130        self,
131        record: Dict[str, Any],
132        config: Optional[Config] = None,
133        stream_state: Optional[StreamState] = None,
134        stream_slice: Optional[StreamSlice] = None,
135    ) -> None:
136        if config is None:
137            config = {}
138        kwargs = {"record": record, "stream_slice": stream_slice}
139        for parsed_field in self._parsed_fields:
140            valid_types = (parsed_field.value_type,) if parsed_field.value_type else None
141            value = parsed_field.value.eval(config, valid_types=valid_types, **kwargs)
142            is_empty_condition = not self.condition
143            if is_empty_condition or self._filter_interpolator.eval(
144                config, value=value, path=parsed_field.path, **kwargs
145            ):
146                dpath.new(record, parsed_field.path, value)
147
148    def __eq__(self, other: Any) -> bool:
149        return bool(self.__dict__ == other.__dict__)

Transformation which adds field to an output record. The path of the added field can be nested. Adding nested fields will create all necessary parent objects (like mkdir -p). Adding fields to an array will extend the array to that index (filling intermediate indices with null values). So if you add a field at index 5 to the array ["value"], it will become ["value", null, null, null, null, "new_value"].

This transformation has access to the following contextual values:

record: the record about to be output by the connector config: the input configuration provided to a connector stream_state: the current state of the stream stream_slice: the current stream slice being read

Examples of instantiating this transformation via YAML:

  • type: AddFields fields: # hardcoded constant
    • path: ["path"] value: "static_value"
# nested path
- path: ["path", "to", "field"]
  value: "static"

# from config
- path: ["shop_id"]
  value: "{{ config.shop_id }}"

# from stream_interval
- path: ["date"]
  value: "{{ stream_interval.start_date }}"

# from record
- path: ["unnested_value"]
  value: {{ record.nested.field }}

# from stream_slice
- path: ["start_date"]
  value: {{ stream_slice.start_date }}

# by supplying any valid Jinja template directive or expression https://jinja.palletsprojects.com/en/3.1.x/templates/#
- path: ["two_times_two"]
  value: {{ 2 * 2 }}
Attributes:
  • fields (List[AddedFieldDefinition]): A list of transformations (path and corresponding value) that will be added to the record
AddFields( fields: List[AddedFieldDefinition], parameters: dataclasses.InitVar[typing.Mapping[str, typing.Any]], condition: str = '')
fields: List[AddedFieldDefinition]
parameters: dataclasses.InitVar[typing.Mapping[str, typing.Any]]
condition: str = ''
def transform( self, record: Dict[str, Any], config: Optional[Mapping[str, Any]] = None, stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[StreamSlice] = None) -> None:
129    def transform(
130        self,
131        record: Dict[str, Any],
132        config: Optional[Config] = None,
133        stream_state: Optional[StreamState] = None,
134        stream_slice: Optional[StreamSlice] = None,
135    ) -> None:
136        if config is None:
137            config = {}
138        kwargs = {"record": record, "stream_slice": stream_slice}
139        for parsed_field in self._parsed_fields:
140            valid_types = (parsed_field.value_type,) if parsed_field.value_type else None
141            value = parsed_field.value.eval(config, valid_types=valid_types, **kwargs)
142            is_empty_condition = not self.condition
143            if is_empty_condition or self._filter_interpolator.eval(
144                config, value=value, path=parsed_field.path, **kwargs
145            ):
146                dpath.new(record, parsed_field.path, value)

Transform a record by adding, deleting, or mutating fields directly from the record reference passed in argument.

Parameters
  • record: The input record to be transformed
  • config: The user-provided configuration as specified by the source's spec
  • stream_state: The stream state
  • stream_slice: The stream slice
Returns

The transformed record

@dataclass(frozen=True)
class AddedFieldDefinition:
17@dataclass(frozen=True)
18class AddedFieldDefinition:
19    """Defines the field to add on a record"""
20
21    path: FieldPointer
22    value: Union[InterpolatedString, str]
23    value_type: Optional[Type[Any]]
24    parameters: InitVar[Mapping[str, Any]]

Defines the field to add on a record

AddedFieldDefinition( path: List[str], value: Union[InterpolatedString, str], value_type: Optional[Type[Any]], parameters: dataclasses.InitVar[typing.Mapping[str, typing.Any]])
path: List[str]
value: Union[InterpolatedString, str]
value_type: Optional[Type[Any]]
parameters: dataclasses.InitVar[typing.Mapping[str, typing.Any]]
@dataclass
class ApiKeyAuthenticator(airbyte_cdk.DeclarativeAuthenticator):
24@dataclass
25class ApiKeyAuthenticator(DeclarativeAuthenticator):
26    """
27    ApiKeyAuth sets a request header on the HTTP requests sent.
28
29    The header is of the form:
30    `"<header>": "<token>"`
31
32    For example,
33    `ApiKeyAuthenticator("Authorization", "Bearer hello")`
34    will result in the following header set on the HTTP request
35    `"Authorization": "Bearer hello"`
36
37    Attributes:
38        request_option (RequestOption): request option how to inject the token into the request
39        token_provider (TokenProvider): Provider of the token
40        config (Config): The user-provided configuration as specified by the source's spec
41        parameters (Mapping[str, Any]): Additional runtime parameters to be used for string interpolation
42    """
43
44    request_option: RequestOption
45    token_provider: TokenProvider
46    config: Config
47    parameters: InitVar[Mapping[str, Any]]
48
49    @property
50    def auth_header(self) -> str:
51        options = self._get_request_options(RequestOptionType.header)
52        return next(iter(options.keys()), "")
53
54    @property
55    def token(self) -> str:
56        return self.token_provider.get_token()
57
58    def _get_request_options(self, option_type: RequestOptionType) -> Mapping[str, Any]:
59        options: MutableMapping[str, Any] = {}
60        if self.request_option.inject_into == option_type:
61            self.request_option.inject_into_request(options, self.token, self.config)
62        return options
63
64    def get_request_params(self) -> Mapping[str, Any]:
65        return self._get_request_options(RequestOptionType.request_parameter)
66
67    def get_request_body_data(self) -> Union[Mapping[str, Any], str]:
68        return self._get_request_options(RequestOptionType.body_data)
69
70    def get_request_body_json(self) -> Mapping[str, Any]:
71        return self._get_request_options(RequestOptionType.body_json)

ApiKeyAuth sets a request header on the HTTP requests sent.

The header is of the form: "<header>": "<token>"

For example, ApiKeyAuthenticator("Authorization", "Bearer hello") will result in the following header set on the HTTP request "Authorization": "Bearer hello"

Attributes:
  • request_option (RequestOption): request option how to inject the token into the request
  • token_provider (TokenProvider): Provider of the token
  • config (Config): The user-provided configuration as specified by the source's spec
  • parameters (Mapping[str, Any]): Additional runtime parameters to be used for string interpolation
ApiKeyAuthenticator( request_option: RequestOption, token_provider: airbyte_cdk.sources.declarative.auth.token_provider.TokenProvider, config: Mapping[str, Any], parameters: dataclasses.InitVar[typing.Mapping[str, typing.Any]])
request_option: RequestOption
config: Mapping[str, Any]
parameters: dataclasses.InitVar[typing.Mapping[str, typing.Any]]
auth_header: str
49    @property
50    def auth_header(self) -> str:
51        options = self._get_request_options(RequestOptionType.header)
52        return next(iter(options.keys()), "")

HTTP header to set on the requests

token: str
54    @property
55    def token(self) -> str:
56        return self.token_provider.get_token()

The header value to set on outgoing HTTP requests

def get_request_params(self) -> Mapping[str, Any]:
64    def get_request_params(self) -> Mapping[str, Any]:
65        return self._get_request_options(RequestOptionType.request_parameter)

HTTP request parameter to add to the requests

def get_request_body_data(self) -> Union[Mapping[str, Any], str]:
67    def get_request_body_data(self) -> Union[Mapping[str, Any], str]:
68        return self._get_request_options(RequestOptionType.body_data)

Form-encoded body data to set on the requests

def get_request_body_json(self) -> Mapping[str, Any]:
70    def get_request_body_json(self) -> Mapping[str, Any]:
71        return self._get_request_options(RequestOptionType.body_json)

JSON-encoded body data to set on the requests

class BackoffStrategy(abc.ABC):
12class BackoffStrategy(ABC):
13    @abstractmethod
14    def backoff_time(
15        self,
16        response_or_exception: Optional[Union[requests.Response, requests.RequestException]],
17        attempt_count: int,
18    ) -> Optional[float]:
19        """
20        Override this method to dynamically determine backoff time e.g: by reading the X-Retry-After header.
21
22        Not called for every retryable response. `HttpClient` skips the strategies entirely when a
23        rate-limited response can be retried on another credential -- the authenticator says so via
24        `TokenRotatingAuthenticator.has_alternative_token` -- because the wait computed here is
25        derived from the credential that was rejected, and the retry will not use it. Implementations
26        must therefore not rely on being called for side effects such as counting attempts or
27        emitting metrics.
28
29        :param response_or_exception: The response or exception that caused the backoff.
30        :param attempt_count: The number of attempts already performed for this request.
31        :return how long to backoff in seconds. The return value may be a floating point number for subsecond precision. Returning None defers backoff
32        to the default backoff behavior (e.g using an exponential algorithm).
33        """
34        pass

Helper class that provides a standard way to create an ABC using inheritance.

@abstractmethod
def backoff_time( self, response_or_exception: Union[requests.models.Response, requests.exceptions.RequestException, NoneType], attempt_count: int) -> Optional[float]:
13    @abstractmethod
14    def backoff_time(
15        self,
16        response_or_exception: Optional[Union[requests.Response, requests.RequestException]],
17        attempt_count: int,
18    ) -> Optional[float]:
19        """
20        Override this method to dynamically determine backoff time e.g: by reading the X-Retry-After header.
21
22        Not called for every retryable response. `HttpClient` skips the strategies entirely when a
23        rate-limited response can be retried on another credential -- the authenticator says so via
24        `TokenRotatingAuthenticator.has_alternative_token` -- because the wait computed here is
25        derived from the credential that was rejected, and the retry will not use it. Implementations
26        must therefore not rely on being called for side effects such as counting attempts or
27        emitting metrics.
28
29        :param response_or_exception: The response or exception that caused the backoff.
30        :param attempt_count: The number of attempts already performed for this request.
31        :return how long to backoff in seconds. The return value may be a floating point number for subsecond precision. Returning None defers backoff
32        to the default backoff behavior (e.g using an exponential algorithm).
33        """
34        pass

Override this method to dynamically determine backoff time e.g: by reading the X-Retry-After header.

Not called for every retryable response. HttpClient skips the strategies entirely when a rate-limited response can be retried on another credential -- the authenticator says so via TokenRotatingAuthenticator.has_alternative_token -- because the wait computed here is derived from the credential that was rejected, and the retry will not use it. Implementations must therefore not rely on being called for side effects such as counting attempts or emitting metrics.

Parameters
  • response_or_exception: The response or exception that caused the backoff.
  • attempt_count: The number of attempts already performed for this request. :return how long to backoff in seconds. The return value may be a floating point number for subsecond precision. Returning None defers backoff to the default backoff behavior (e.g using an exponential algorithm).
@dataclass
class BasicHttpAuthenticator(airbyte_cdk.DeclarativeAuthenticator):
101@dataclass
102class BasicHttpAuthenticator(DeclarativeAuthenticator):
103    """
104    Builds auth based off the basic authentication scheme as defined by RFC 7617, which transmits credentials as USER ID/password pairs, encoded using base64
105    https://developer.mozilla.org/en-US/docs/Web/HTTP/Authentication#basic_authentication_scheme
106
107    The header is of the form
108    `"Authorization": "Basic <encoded_credentials>"`
109
110    Attributes:
111        username (Union[InterpolatedString, str]): The username
112        config (Config): The user-provided configuration as specified by the source's spec
113        password (Union[InterpolatedString, str]): The password
114        parameters (Mapping[str, Any]): Additional runtime parameters to be used for string interpolation
115    """
116
117    username: Union[InterpolatedString, str]
118    config: Config
119    parameters: InitVar[Mapping[str, Any]]
120    password: Union[InterpolatedString, str] = ""
121
122    def __post_init__(self, parameters: Mapping[str, Any]) -> None:
123        self._username = InterpolatedString.create(self.username, parameters=parameters)
124        self._password = InterpolatedString.create(self.password, parameters=parameters)
125
126    @property
127    def auth_header(self) -> str:
128        return "Authorization"
129
130    @property
131    def token(self) -> str:
132        auth_string = (
133            f"{self._username.eval(self.config)}:{self._password.eval(self.config)}".encode("utf8")
134        )
135        b64_encoded = base64.b64encode(auth_string).decode("utf8")
136        return f"Basic {b64_encoded}"

Builds auth based off the basic authentication scheme as defined by RFC 7617, which transmits credentials as USER ID/password pairs, encoded using base64 https://developer.mozilla.org/en-US/docs/Web/HTTP/Authentication#basic_authentication_scheme

The header is of the form "Authorization": "Basic <encoded_credentials>"

Attributes:
  • username (Union[InterpolatedString, str]): The username
  • config (Config): The user-provided configuration as specified by the source's spec
  • password (Union[InterpolatedString, str]): The password
  • parameters (Mapping[str, Any]): Additional runtime parameters to be used for string interpolation
BasicHttpAuthenticator( username: Union[InterpolatedString, str], config: Mapping[str, Any], parameters: dataclasses.InitVar[typing.Mapping[str, typing.Any]], password: Union[InterpolatedString, str] = '')
username: Union[InterpolatedString, str]
config: Mapping[str, Any]
parameters: dataclasses.InitVar[typing.Mapping[str, typing.Any]]
password: Union[InterpolatedString, str] = ''
auth_header: str
126    @property
127    def auth_header(self) -> str:
128        return "Authorization"

HTTP header to set on the requests

token: str
130    @property
131    def token(self) -> str:
132        auth_string = (
133            f"{self._username.eval(self.config)}:{self._password.eval(self.config)}".encode("utf8")
134        )
135        b64_encoded = base64.b64encode(auth_string).decode("utf8")
136        return f"Basic {b64_encoded}"

The header value to set on outgoing HTTP requests

@dataclass
class BearerAuthenticator(airbyte_cdk.DeclarativeAuthenticator):
74@dataclass
75class BearerAuthenticator(DeclarativeAuthenticator):
76    """
77    Authenticator that sets the Authorization header on the HTTP requests sent.
78
79    The header is of the form:
80    `"Authorization": "Bearer <token>"`
81
82    Attributes:
83        token_provider (TokenProvider): Provider of the token
84        config (Config): The user-provided configuration as specified by the source's spec
85        parameters (Mapping[str, Any]): Additional runtime parameters to be used for string interpolation
86    """
87
88    token_provider: TokenProvider
89    config: Config
90    parameters: InitVar[Mapping[str, Any]]
91
92    @property
93    def auth_header(self) -> str:
94        return "Authorization"
95
96    @property
97    def token(self) -> str:
98        return f"Bearer {self.token_provider.get_token()}"

Authenticator that sets the Authorization header on the HTTP requests sent.

The header is of the form: "Authorization": "Bearer <token>"

Attributes:
  • token_provider (TokenProvider): Provider of the token
  • config (Config): The user-provided configuration as specified by the source's spec
  • parameters (Mapping[str, Any]): Additional runtime parameters to be used for string interpolation
BearerAuthenticator( token_provider: airbyte_cdk.sources.declarative.auth.token_provider.TokenProvider, config: Mapping[str, Any], parameters: dataclasses.InitVar[typing.Mapping[str, typing.Any]])
config: Mapping[str, Any]
parameters: dataclasses.InitVar[typing.Mapping[str, typing.Any]]
auth_header: str
92    @property
93    def auth_header(self) -> str:
94        return "Authorization"

HTTP header to set on the requests

token: str
96    @property
97    def token(self) -> str:
98        return f"Bearer {self.token_provider.get_token()}"

The header value to set on outgoing HTTP requests

@dataclass
class CartesianProductStreamSlicer(airbyte_cdk.sources.declarative.partition_routers.partition_router.PartitionRouter):
 40@dataclass
 41class CartesianProductStreamSlicer(PartitionRouter):
 42    """
 43    Stream slicers that iterates over the cartesian product of input stream slicers
 44    Given 2 stream slicers with the following slices:
 45    A: [{"i": 0}, {"i": 1}, {"i": 2}]
 46    B: [{"s": "hello"}, {"s": "world"}]
 47    the resulting stream slices are
 48    [
 49        {"i": 0, "s": "hello"},
 50        {"i": 0, "s": "world"},
 51        {"i": 1, "s": "hello"},
 52        {"i": 1, "s": "world"},
 53        {"i": 2, "s": "hello"},
 54        {"i": 2, "s": "world"},
 55    ]
 56
 57    Attributes:
 58        stream_slicers (List[PartitionRouter]): Underlying stream slicers. The RequestOptions (e.g: Request headers, parameters, etc..) returned by this slicer are the combination of the RequestOptions of its input slicers. If there are conflicts e.g: two slicers define the same header or request param, the conflict is resolved by taking the value from the first slicer, where ordering is determined by the order in which slicers were input to this composite slicer.
 59    """
 60
 61    stream_slicers: List[PartitionRouter]
 62    parameters: InitVar[Mapping[str, Any]]
 63
 64    def __post_init__(self, parameters: Mapping[str, Any]) -> None:
 65        check_for_substream_in_slicers(self.stream_slicers, self.logger.warning)
 66
 67    def get_request_params(
 68        self,
 69        *,
 70        stream_state: Optional[StreamState] = None,
 71        stream_slice: Optional[StreamSlice] = None,
 72        next_page_token: Optional[Mapping[str, Any]] = None,
 73    ) -> Mapping[str, Any]:
 74        return dict(
 75            ChainMap(
 76                *[  # type: ignore # ChainMap expects a MutableMapping[Never, Never] for reasons
 77                    s.get_request_params(
 78                        stream_state=stream_state,
 79                        stream_slice=stream_slice,
 80                        next_page_token=next_page_token,
 81                    )
 82                    for s in self.stream_slicers
 83                ]
 84            )
 85        )
 86
 87    def get_request_headers(
 88        self,
 89        *,
 90        stream_state: Optional[StreamState] = None,
 91        stream_slice: Optional[StreamSlice] = None,
 92        next_page_token: Optional[Mapping[str, Any]] = None,
 93    ) -> Mapping[str, Any]:
 94        return dict(
 95            ChainMap(
 96                *[  # type: ignore # ChainMap expects a MutableMapping[Never, Never] for reasons
 97                    s.get_request_headers(
 98                        stream_state=stream_state,
 99                        stream_slice=stream_slice,
100                        next_page_token=next_page_token,
101                    )
102                    for s in self.stream_slicers
103                ]
104            )
105        )
106
107    def get_request_body_data(
108        self,
109        *,
110        stream_state: Optional[StreamState] = None,
111        stream_slice: Optional[StreamSlice] = None,
112        next_page_token: Optional[Mapping[str, Any]] = None,
113    ) -> Mapping[str, Any]:
114        return dict(
115            ChainMap(
116                *[  # type: ignore # ChainMap expects a MutableMapping[Never, Never] for reasons
117                    s.get_request_body_data(
118                        stream_state=stream_state,
119                        stream_slice=stream_slice,
120                        next_page_token=next_page_token,
121                    )
122                    for s in self.stream_slicers
123                ]
124            )
125        )
126
127    def get_request_body_json(
128        self,
129        *,
130        stream_state: Optional[StreamState] = None,
131        stream_slice: Optional[StreamSlice] = None,
132        next_page_token: Optional[Mapping[str, Any]] = None,
133    ) -> Mapping[str, Any]:
134        return dict(
135            ChainMap(
136                *[  # type: ignore # ChainMap expects a MutableMapping[Never, Never] for reasons
137                    s.get_request_body_json(
138                        stream_state=stream_state,
139                        stream_slice=stream_slice,
140                        next_page_token=next_page_token,
141                    )
142                    for s in self.stream_slicers
143                ]
144            )
145        )
146
147    def stream_slices(self) -> Iterable[StreamSlice]:
148        sub_slices = (s.stream_slices() for s in self.stream_slicers)
149        product = itertools.product(*sub_slices)
150        for stream_slice_tuple in product:
151            partition = dict(ChainMap(*[s.partition for s in stream_slice_tuple]))  # type: ignore # ChainMap expects a MutableMapping[Never, Never] for reasons
152            cursor_slices = [s.cursor_slice for s in stream_slice_tuple if s.cursor_slice]
153            extra_fields = dict(ChainMap(*[s.extra_fields for s in stream_slice_tuple]))  # type: ignore # ChainMap expects a MutableMapping[Never, Never] for reasons
154            if len(cursor_slices) > 1:
155                raise ValueError(
156                    f"There should only be a single cursor slice. Found {cursor_slices}"
157                )
158            if cursor_slices:
159                cursor_slice = cursor_slices[0]
160            else:
161                cursor_slice = {}
162            yield StreamSlice(
163                partition=partition, cursor_slice=cursor_slice, extra_fields=extra_fields
164            )
165
166    def get_stream_state(self) -> Optional[Mapping[str, StreamState]]:
167        """
168        Parent stream states are not supported for cartesian product stream slicer
169        """
170        pass
171
172    @property
173    def logger(self) -> logging.Logger:
174        return logging.getLogger("airbyte.CartesianProductStreamSlicer")

Stream slicers that iterates over the cartesian product of input stream slicers Given 2 stream slicers with the following slices: A: [{"i": 0}, {"i": 1}, {"i": 2}] B: [{"s": "hello"}, {"s": "world"}] the resulting stream slices are [ {"i": 0, "s": "hello"}, {"i": 0, "s": "world"}, {"i": 1, "s": "hello"}, {"i": 1, "s": "world"}, {"i": 2, "s": "hello"}, {"i": 2, "s": "world"}, ]

Attributes:
  • stream_slicers (List[PartitionRouter]): Underlying stream slicers. The RequestOptions (e.g: Request headers, parameters, etc..) returned by this slicer are the combination of the RequestOptions of its input slicers. If there are conflicts e.g: two slicers define the same header or request param, the conflict is resolved by taking the value from the first slicer, where ordering is determined by the order in which slicers were input to this composite slicer.
CartesianProductStreamSlicer( stream_slicers: List[airbyte_cdk.sources.declarative.partition_routers.PartitionRouter], parameters: dataclasses.InitVar[typing.Mapping[str, typing.Any]])
parameters: dataclasses.InitVar[typing.Mapping[str, typing.Any]]
def get_request_params( self, *, stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[StreamSlice] = None, next_page_token: Optional[Mapping[str, Any]] = None) -> Mapping[str, Any]:
67    def get_request_params(
68        self,
69        *,
70        stream_state: Optional[StreamState] = None,
71        stream_slice: Optional[StreamSlice] = None,
72        next_page_token: Optional[Mapping[str, Any]] = None,
73    ) -> Mapping[str, Any]:
74        return dict(
75            ChainMap(
76                *[  # type: ignore # ChainMap expects a MutableMapping[Never, Never] for reasons
77                    s.get_request_params(
78                        stream_state=stream_state,
79                        stream_slice=stream_slice,
80                        next_page_token=next_page_token,
81                    )
82                    for s in self.stream_slicers
83                ]
84            )
85        )

Specifies the query parameters that should be set on an outgoing HTTP request given the inputs.

E.g: you might want to define query parameters for paging if next_page_token is not None.

def get_request_headers( self, *, stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[StreamSlice] = None, next_page_token: Optional[Mapping[str, Any]] = None) -> Mapping[str, Any]:
 87    def get_request_headers(
 88        self,
 89        *,
 90        stream_state: Optional[StreamState] = None,
 91        stream_slice: Optional[StreamSlice] = None,
 92        next_page_token: Optional[Mapping[str, Any]] = None,
 93    ) -> Mapping[str, Any]:
 94        return dict(
 95            ChainMap(
 96                *[  # type: ignore # ChainMap expects a MutableMapping[Never, Never] for reasons
 97                    s.get_request_headers(
 98                        stream_state=stream_state,
 99                        stream_slice=stream_slice,
100                        next_page_token=next_page_token,
101                    )
102                    for s in self.stream_slicers
103                ]
104            )
105        )

Return any non-auth headers. Authentication headers will overwrite any overlapping headers returned from this method.

def get_request_body_data( self, *, stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[StreamSlice] = None, next_page_token: Optional[Mapping[str, Any]] = None) -> Mapping[str, Any]:
107    def get_request_body_data(
108        self,
109        *,
110        stream_state: Optional[StreamState] = None,
111        stream_slice: Optional[StreamSlice] = None,
112        next_page_token: Optional[Mapping[str, Any]] = None,
113    ) -> Mapping[str, Any]:
114        return dict(
115            ChainMap(
116                *[  # type: ignore # ChainMap expects a MutableMapping[Never, Never] for reasons
117                    s.get_request_body_data(
118                        stream_state=stream_state,
119                        stream_slice=stream_slice,
120                        next_page_token=next_page_token,
121                    )
122                    for s in self.stream_slicers
123                ]
124            )
125        )

Specifies how to populate the body of the request with a non-JSON payload.

If returns a ready text that it will be sent as is. If returns a dict that it will be converted to a urlencoded form. E.g. {"key1": "value1", "key2": "value2"} => "key1=value1&key2=value2"

At the same time only one of the 'request_body_data' and 'request_body_json' functions can be overridden.

def get_request_body_json( self, *, stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[StreamSlice] = None, next_page_token: Optional[Mapping[str, Any]] = None) -> Mapping[str, Any]:
127    def get_request_body_json(
128        self,
129        *,
130        stream_state: Optional[StreamState] = None,
131        stream_slice: Optional[StreamSlice] = None,
132        next_page_token: Optional[Mapping[str, Any]] = None,
133    ) -> Mapping[str, Any]:
134        return dict(
135            ChainMap(
136                *[  # type: ignore # ChainMap expects a MutableMapping[Never, Never] for reasons
137                    s.get_request_body_json(
138                        stream_state=stream_state,
139                        stream_slice=stream_slice,
140                        next_page_token=next_page_token,
141                    )
142                    for s in self.stream_slicers
143                ]
144            )
145        )

Specifies how to populate the body of the request with a JSON payload.

At the same time only one of the 'request_body_data' and 'request_body_json' functions can be overridden.

def stream_slices(self) -> Iterable[StreamSlice]:
147    def stream_slices(self) -> Iterable[StreamSlice]:
148        sub_slices = (s.stream_slices() for s in self.stream_slicers)
149        product = itertools.product(*sub_slices)
150        for stream_slice_tuple in product:
151            partition = dict(ChainMap(*[s.partition for s in stream_slice_tuple]))  # type: ignore # ChainMap expects a MutableMapping[Never, Never] for reasons
152            cursor_slices = [s.cursor_slice for s in stream_slice_tuple if s.cursor_slice]
153            extra_fields = dict(ChainMap(*[s.extra_fields for s in stream_slice_tuple]))  # type: ignore # ChainMap expects a MutableMapping[Never, Never] for reasons
154            if len(cursor_slices) > 1:
155                raise ValueError(
156                    f"There should only be a single cursor slice. Found {cursor_slices}"
157                )
158            if cursor_slices:
159                cursor_slice = cursor_slices[0]
160            else:
161                cursor_slice = {}
162            yield StreamSlice(
163                partition=partition, cursor_slice=cursor_slice, extra_fields=extra_fields
164            )

Defines stream slices

Returns

An iterable of stream slices

def get_stream_state(self) -> Optional[Mapping[str, Mapping[str, Any]]]:
166    def get_stream_state(self) -> Optional[Mapping[str, StreamState]]:
167        """
168        Parent stream states are not supported for cartesian product stream slicer
169        """
170        pass

Parent stream states are not supported for cartesian product stream slicer

logger: logging.Logger
172    @property
173    def logger(self) -> logging.Logger:
174        return logging.getLogger("airbyte.CartesianProductStreamSlicer")
class CombineMode(enum.Enum):
19class CombineMode(Enum):
20    """
21    How a `CombinedExtractor` combines the output of its sub-extractors.
22    """
23
24    union = "union"
25    first_match = "first_match"

How a CombinedExtractor combines the output of its sub-extractors.

union = <CombineMode.union: 'union'>
first_match = <CombineMode.first_match: 'first_match'>
@dataclass
class CombinedExtractor(airbyte_cdk.RecordExtractor):
 46@dataclass
 47class CombinedExtractor(RecordExtractor):
 48    """Combines the output of several record extractors into a single stream of records.
 49
 50    A single `DpathExtractor` can only describe one path into the response. Several connectors need
 51    more than that and reach for a custom `components.py` today, which keeps them out of the
 52    Connector Builder. This component covers the two shapes those connectors implement:
 53
 54    - `union` (default) — yield every record of every sub-extractor, in sub-extractor order. Use it
 55      when one response carries records under several paths, for example a GraphQL document that
 56      returns `issues.nodes` and `pullRequests.nodes` side by side.
 57    - `first_match` — yield the records of the first sub-extractor that produces at least one
 58      record, and skip the remaining sub-extractors. If no sub-extractor produces a record, nothing
 59      is yielded. Use it when an API answers with one of several alternative shapes. The winning
 60      sub-extractor is only peeked at, never restarted: the peeked record is chained back in front
 61      of the remaining ones, so large responses are still streamed lazily and no sub-extractor is
 62      ever materialized into a list.
 63
 64    Examples of instantiating this component:
 65    ```
 66      extractor:
 67        type: CombinedExtractor
 68        mode: union
 69        extractors:
 70          - type: DpathExtractor
 71            field_path: ["data", "repository", "issues", "nodes"]
 72          - type: DpathExtractor
 73            field_path: ["data", "repository", "pullRequests", "nodes"]
 74    ```
 75
 76    ```
 77      extractor:
 78        type: CombinedExtractor
 79        mode: first_match
 80        extractors:
 81          - type: DpathExtractor
 82            field_path: ["data", "boards", "*", "items_page", "items", "*"]
 83          - type: DpathExtractor
 84            field_path: ["data", "next_items_page", "items"]
 85    ```
 86
 87    ## Streaming decoders are rejected
 88
 89    Every sub-extractor is handed the same `requests.Response`, so the response body must be
 90    readable more than once. That holds for the decoders that buffer the whole body, which is the
 91    common case: `JsonDecoder` and any `CompositeRawDecoder` built with `stream_response=False`
 92    read `response.content`, `XmlDecoder` reads `response.text` and `ZipfileDecoder` reads
 93    `response.content` — `requests` caches all of those, so every sub-extractor sees the full body.
 94
 95    It does NOT hold for streaming decoders, i.e. a `CompositeRawDecoder` with
 96    `stream_response=True`, which is what `CsvDecoder`, `JsonlDecoder`, `JsonItemsDecoder` and
 97    `GzipDecoder` resolve to outside the Connector Builder, nor for `IterableDecoder`. Those read
 98    `response.raw` and close it afterwards, so the first sub-extractor drains the stream and the
 99    later ones read a dead body. That loss is SILENT rather than loud: `requests` puts a
100    `urllib3.HTTPResponse` in `response.raw`, and reading a closed `urllib3.HTTPResponse` returns
101    an empty body instead of raising, so `union` emits only the first sub-extractor's records and
102    `first_match` emits nothing at all when the first path misses.
103
104    Because of that, `ModelToComponentFactory.create_combined_extractor` refuses to build this
105    component over a streaming decoder and raises a configuration error, naming the decoder in the
106    internal message. The Connector Builder forces those same decoders to `stream_response=False`,
107    so the rejection is applied there too rather than letting a manifest test-read correctly and
108    lose records once published.
109
110    ## Empty records
111
112    `skip_empty_records` drops falsy records — `None`, `{}`, `[]`, `""` — before any of the modes
113    sees them. It is off by default, so a sub-extractor's output is passed through verbatim.
114
115    Turn it on when an API can answer with nulls in the middle of a record list. A GraphQL API
116    that returns a partial response puts `null` in the `data` array for the fields it failed to
117    resolve and describes the failure in a sibling `errors` field; without this flag those nulls
118    reach the record stream and fail schema validation downstream.
119
120    Under `first_match` the flag also changes which sub-extractor wins: a sub-extractor whose
121    records are all empty no longer counts as a match, so the next sub-extractor is tried. That is
122    the behaviour of `source-monday`'s `MondayIncrementalItemsExtractor`, where a page of nulls
123    must fall through to the pagination path rather than lock in the primary path.
124
125    The dropped records still count toward the size of the page, see "Record-counting paginators"
126    below.
127
128    ## Cost
129
130    Each sub-extractor decodes the response independently — `CompositeRawDecoder` re-parses the
131    cached body on every `decode()` call — so a response is parsed once per sub-extractor.
132    Combining three extractors over a large response costs roughly three times the decode time of
133    a single one.
134
135    ## Record-counting paginators
136
137    The count `OffsetIncrement` and `PageIncrement` obtain is the combined count, not the API's
138    page size:
139
140    - `union` returns the SUM over all sub-extractors. An `OffsetIncrement` advances the offset by
141      that sum, so the next page starts past the records that were never read — two sub-extractors
142      returning two records each with `page_size: 2` request offsets 0, 4, 8 instead of 0, 2, 4 and
143      silently drop two thirds of the records. `ModelToComponentFactory.create_offset_increment`
144      therefore rejects a `union` `CombinedExtractor` (nested ones included) with a configuration
145      error. A `PageIncrement` only compares the count against `page_size` to decide whether to
146      stop, so it loses nothing, but it issues one extra request whenever the summed count of the
147      last page happens to equal `page_size`.
148    - `first_match` returns the count of the winning sub-extractor, which does not inflate the
149      count, so it works with either paginator as long as the path the count comes from is the one
150      the API paginates over.
151
152    Empty records dropped by `skip_empty_records` are part of that count. The API returned them as
153    slots of the page, so a full page holding a null must still read as full: counting only the
154    surviving records would make both paginators stop at the first page with a null, and would
155    make an `OffsetIncrement` re-request records it already read. The factory sets
156    `count_dropped_empty_records` on the extractor of a `RecordSelector`, which makes it yield a
157    placeholder for every empty record it drops; the `RecordSelector` counts the placeholders and
158    discards them. The winner under `first_match` is still chosen by the surviving records alone,
159    so the placeholders never change which records are emitted.
160
161    Attributes:
162        extractors (List[RecordExtractor]): The sub-extractors to combine. At least one is required.
163        mode (CombineMode): How the sub-extractor outputs are combined. Defaults to `union`.
164        skip_empty_records (bool): Whether falsy records are dropped before they are combined.
165            Defaults to `False`.
166        count_dropped_empty_records (bool): Set by the factory on the extractor of a
167            `RecordSelector`, which discards the placeholders: yield a placeholder in place of
168            every dropped empty record so it counts toward the page size. Defaults to `False`.
169    """
170
171    extractors: List[RecordExtractor]
172    parameters: InitVar[Mapping[str, Any]]
173    mode: CombineMode = CombineMode.union
174    skip_empty_records: bool = False
175    count_dropped_empty_records: bool = False
176
177    def __post_init__(self, parameters: Mapping[str, Any]) -> None:
178        if not self.extractors:
179            raise ValueError(
180                "CombinedExtractor requires at least one extractor in its `extractors` field."
181            )
182        if not isinstance(self.mode, CombineMode):
183            self.mode = CombineMode(self.mode)
184
185    def extract_records(self, response: requests.Response) -> Iterable[Mapping[str, Any]]:
186        if self.mode == CombineMode.first_match:
187            yield from self._extract_first_match(response)
188        else:
189            yield from self._extract_union(response)
190
191    def _extract_union(self, response: requests.Response) -> Iterable[Mapping[str, Any]]:
192        for index in range(len(self.extractors)):
193            yield from self._records_of(index, response)
194
195    def _extract_first_match(self, response: requests.Response) -> Iterable[Mapping[str, Any]]:
196        for index in range(len(self.extractors)):
197            records: Iterator[Mapping[str, Any]] = iter(self._records_of(index, response))
198            # Placeholders for dropped empty records do not make a sub-extractor win, but they are
199            # kept so the winner's count still includes the ones that came before its first record.
200            dropped_placeholders: List[Mapping[str, Any]] = []
201            first_record: Union[Mapping[str, Any], _NoRecord] = _NO_RECORD
202            for record in records:
203                if isinstance(record, _DroppedEmptyRecord):
204                    dropped_placeholders.append(record)
205                    continue
206                first_record = record
207                break
208            if isinstance(first_record, _NoRecord):
209                continue
210            # Chain the peeked record back instead of restarting the extractor: the rest of the
211            # records stay lazy and the response is never read twice.
212            yield from chain(dropped_placeholders, [first_record], records)
213            return
214
215    def _records_of(
216        self, extractor_index: int, response: requests.Response
217    ) -> Iterable[Mapping[str, Any]]:
218        records = self.extractors[extractor_index].extract_records(response)
219        if not self.skip_empty_records:
220            return records
221        return self._without_empty_records(records, extractor_index)
222
223    def _without_empty_records(
224        self, records: Iterable[Mapping[str, Any]], extractor_index: int
225    ) -> Iterator[Mapping[str, Any]]:
226        dropped = 0
227        for record in records:
228            if not record:
229                dropped += 1
230                if self.count_dropped_empty_records:
231                    yield _DROPPED_EMPTY_RECORD  # type: ignore[misc]  # only ever counted, never emitted
232                continue
233            yield record
234        if dropped:
235            logger.warning(
236                "CombinedExtractor dropped %s empty record(s) yielded by sub-extractor %s (%s) "
237                "because `skip_empty_records` is enabled. An API that returns empty records "
238                "alongside real ones usually reports the reason in an error field of the "
239                "response body.",
240                dropped,
241                extractor_index,
242                type(self.extractors[extractor_index]).__name__,
243            )

Combines the output of several record extractors into a single stream of records.

A single DpathExtractor can only describe one path into the response. Several connectors need more than that and reach for a custom components.py today, which keeps them out of the Connector Builder. This component covers the two shapes those connectors implement:

  • union (default) — yield every record of every sub-extractor, in sub-extractor order. Use it when one response carries records under several paths, for example a GraphQL document that returns issues.nodes and pullRequests.nodes side by side.
  • first_match — yield the records of the first sub-extractor that produces at least one record, and skip the remaining sub-extractors. If no sub-extractor produces a record, nothing is yielded. Use it when an API answers with one of several alternative shapes. The winning sub-extractor is only peeked at, never restarted: the peeked record is chained back in front of the remaining ones, so large responses are still streamed lazily and no sub-extractor is ever materialized into a list.

Examples of instantiating this component:

  extractor:
    type: CombinedExtractor
    mode: union
    extractors:
      - type: DpathExtractor
        field_path: ["data", "repository", "issues", "nodes"]
      - type: DpathExtractor
        field_path: ["data", "repository", "pullRequests", "nodes"]
  extractor:
    type: CombinedExtractor
    mode: first_match
    extractors:
      - type: DpathExtractor
        field_path: ["data", "boards", "*", "items_page", "items", "*"]
      - type: DpathExtractor
        field_path: ["data", "next_items_page", "items"]

Streaming decoders are rejected

Every sub-extractor is handed the same requests.Response, so the response body must be readable more than once. That holds for the decoders that buffer the whole body, which is the common case: JsonDecoder and any CompositeRawDecoder built with stream_response=False read response.content, XmlDecoder reads response.text and ZipfileDecoder reads response.content — requests caches all of those, so every sub-extractor sees the full body.

It does NOT hold for streaming decoders, i.e. a CompositeRawDecoder with stream_response=True, which is what CsvDecoder, JsonlDecoder, JsonItemsDecoder and GzipDecoder resolve to outside the Connector Builder, nor for IterableDecoder. Those read response.raw and close it afterwards, so the first sub-extractor drains the stream and the later ones read a dead body. That loss is SILENT rather than loud: requests puts a urllib3.HTTPResponse in response.raw, and reading a closed urllib3.HTTPResponse returns an empty body instead of raising, so union emits only the first sub-extractor's records and first_match emits nothing at all when the first path misses.

Because of that, ModelToComponentFactory.create_combined_extractor refuses to build this component over a streaming decoder and raises a configuration error, naming the decoder in the internal message. The Connector Builder forces those same decoders to stream_response=False, so the rejection is applied there too rather than letting a manifest test-read correctly and lose records once published.

Empty records

skip_empty_records drops falsy records — None, {}, [], "" — before any of the modes sees them. It is off by default, so a sub-extractor's output is passed through verbatim.

Turn it on when an API can answer with nulls in the middle of a record list. A GraphQL API that returns a partial response puts null in the data array for the fields it failed to resolve and describes the failure in a sibling errors field; without this flag those nulls reach the record stream and fail schema validation downstream.

Under first_match the flag also changes which sub-extractor wins: a sub-extractor whose records are all empty no longer counts as a match, so the next sub-extractor is tried. That is the behaviour of source-monday's MondayIncrementalItemsExtractor, where a page of nulls must fall through to the pagination path rather than lock in the primary path.

The dropped records still count toward the size of the page, see "Record-counting paginators" below.

Cost

Each sub-extractor decodes the response independently — CompositeRawDecoder re-parses the cached body on every decode() call — so a response is parsed once per sub-extractor. Combining three extractors over a large response costs roughly three times the decode time of a single one.

Record-counting paginators

The count OffsetIncrement and PageIncrement obtain is the combined count, not the API's page size:

  • union returns the SUM over all sub-extractors. An OffsetIncrement advances the offset by that sum, so the next page starts past the records that were never read — two sub-extractors returning two records each with page_size: 2 request offsets 0, 4, 8 instead of 0, 2, 4 and silently drop two thirds of the records. ModelToComponentFactory.create_offset_increment therefore rejects a union CombinedExtractor (nested ones included) with a configuration error. A PageIncrement only compares the count against page_size to decide whether to stop, so it loses nothing, but it issues one extra request whenever the summed count of the last page happens to equal page_size.
  • first_match returns the count of the winning sub-extractor, which does not inflate the count, so it works with either paginator as long as the path the count comes from is the one the API paginates over.

Empty records dropped by skip_empty_records are part of that count. The API returned them as slots of the page, so a full page holding a null must still read as full: counting only the surviving records would make both paginators stop at the first page with a null, and would make an OffsetIncrement re-request records it already read. The factory sets count_dropped_empty_records on the extractor of a RecordSelector, which makes it yield a placeholder for every empty record it drops; the RecordSelector counts the placeholders and discards them. The winner under first_match is still chosen by the surviving records alone, so the placeholders never change which records are emitted.

Attributes:
  • extractors (List[RecordExtractor]): The sub-extractors to combine. At least one is required.
  • mode (CombineMode): How the sub-extractor outputs are combined. Defaults to union.
  • skip_empty_records (bool): Whether falsy records are dropped before they are combined. Defaults to False.
  • count_dropped_empty_records (bool): Set by the factory on the extractor of a RecordSelector, which discards the placeholders: yield a placeholder in place of every dropped empty record so it counts toward the page size. Defaults to False.
CombinedExtractor( extractors: List[RecordExtractor], parameters: dataclasses.InitVar[typing.Mapping[str, typing.Any]], mode: CombineMode = <CombineMode.union: 'union'>, skip_empty_records: bool = False, count_dropped_empty_records: bool = False)
extractors: List[RecordExtractor]
parameters: dataclasses.InitVar[typing.Mapping[str, typing.Any]]
mode: CombineMode = <CombineMode.union: 'union'>
skip_empty_records: bool = False
count_dropped_empty_records: bool = False
def extract_records(self, response: requests.models.Response) -> Iterable[Mapping[str, Any]]:
185    def extract_records(self, response: requests.Response) -> Iterable[Mapping[str, Any]]:
186        if self.mode == CombineMode.first_match:
187            yield from self._extract_first_match(response)
188        else:
189            yield from self._extract_union(response)

Selects records from the response

Parameters
  • response: The response to extract the records from
Returns

List of Records extracted from the response

@dataclass
class CursorPaginationStrategy(airbyte_cdk.PaginationStrategy):
 28@dataclass
 29class CursorPaginationStrategy(PaginationStrategy):
 30    """
 31    Pagination strategy that evaluates an interpolated string to define the next page token.
 32
 33    Attributes:
 34        page_size (Optional[Union[str, int]]): the number of records to request
 35        cursor_value (Union[InterpolatedString, str]): template string evaluating to the cursor value
 36        config (Config): connection config
 37        stop_condition (Optional[InterpolatedBoolean]): template string evaluating when to stop paginating
 38        decoder (Decoder): decoder to decode the response
 39    """
 40
 41    cursor_value: Union[InterpolatedString, str]
 42    config: Config
 43    parameters: InitVar[Mapping[str, Any]]
 44    page_size: Optional[Union[str, int]] = None
 45    stop_condition: Optional[Union[InterpolatedBoolean, str]] = None
 46    decoder: Decoder = field(
 47        default_factory=lambda: PaginationDecoderDecorator(decoder=JsonDecoder(parameters={}))
 48    )
 49
 50    def __post_init__(self, parameters: Mapping[str, Any]) -> None:
 51        if isinstance(self.cursor_value, str):
 52            self._cursor_value = InterpolatedString.create(self.cursor_value, parameters=parameters)
 53        else:
 54            self._cursor_value = self.cursor_value
 55        if isinstance(self.stop_condition, str):
 56            self._stop_condition: Optional[InterpolatedBoolean] = InterpolatedBoolean(
 57                condition=self.stop_condition, parameters=parameters
 58            )
 59        else:
 60            self._stop_condition = self.stop_condition
 61
 62        if isinstance(self.page_size, int) or (self.page_size is None):
 63            self._page_size = self.page_size
 64        else:
 65            page_size = InterpolatedString(self.page_size, parameters=parameters).eval(self.config)
 66            if not isinstance(page_size, int):
 67                raise Exception(f"{page_size} is of type {type(page_size)}. Expected {int}")
 68            self._page_size = page_size
 69
 70        if self._page_size is None:
 71            self._reject_unbound_page_size_variable()
 72
 73    def _reject_unbound_page_size_variable(self) -> None:
 74        """
 75        Fail construction when an expression reads `page_size` while the strategy declares none.
 76
 77        The variable holds the page size that was actually requested, so it is the one safe thing to compare
 78        `last_page_size` against while a `page_size_reduction` is shrinking the page. Without a declared
 79        `page_size` there is nothing to bind it to, and an unbound comparison does not raise: Jinja fails, the
 80        interpolation falls back to the raw template string, and a non-empty string is truthy. A
 81        `stop_condition` written that way stops after the first page and drops the rest of the partition
 82        without failing, which is why this is a construction error rather than a warning.
 83        """
 84        for field_name, template in (
 85            ("stop_condition", self.stop_condition),
 86            ("cursor_value", self.cursor_value),
 87        ):
 88            expression = template.string if isinstance(template, InterpolatedString) else template
 89            if isinstance(template, InterpolatedBoolean):
 90                expression = template.condition
 91            if not isinstance(expression, str) or not references_requested_page_size(expression):
 92                continue
 93            raise ValueError(
 94                f"The `{field_name}` {expression!r} reads the `{REQUESTED_PAGE_SIZE_VARIABLE}` interpolation "
 95                f"variable, but the CursorPagination strategy it belongs to declares no `page_size`, so there "
 96                f"is nothing to bind it to. The comparison would not fail either - it renders as the template "
 97                f"string itself, which is truthy - so the pagination would end after the first page and the "
 98                f"rest of the partition would be dropped silently. Declare `page_size` on the pagination "
 99                f"strategy, or compare against a value the manifest defines, such as "
100                f"`config['page_size']`."
101            )
102
103    @property
104    def initial_token(self) -> Optional[Any]:
105        """
106        CursorPaginationStrategy does not have an initial value because the next cursor is typically included
107        in the response of the first request. For Resumable Full Refresh streams that checkpoint the page
108        cursor, the next cursor should be read from the state or stream slice object.
109        """
110        return None
111
112    def next_page_token(
113        self,
114        response: requests.Response,
115        last_page_size: int,
116        last_record: Optional[Record],
117        last_page_token_value: Optional[Any] = None,
118        page_size_override: Optional[int] = None,
119        stream_slice: Optional[StreamSlice] = None,
120    ) -> Optional[Any]:
121        # The next page is a cursor read from the response, so `page_size_override` does not change how the token
122        # is computed. It is still exposed to the interpolation context as `page_size` because a `stop_condition`
123        # comparing `last_page_size` to a hardcoded page size would read a full reduced page as a short page and
124        # end the pagination early, silently dropping the rest of the partition. Writing the condition as
125        # `{{ last_page_size < page_size }}` keeps it correct while a reduction is in effect.
126        requested_page_size = (
127            page_size_override if page_size_override is not None else self._page_size
128        )
129        decoded_response = next(self.decoder.decode(response))
130        # The default way that link is presented in requests.Response is a string of various links (last, next, etc). This
131        # is not indexable or useful for parsing the cursor, so we replace it with the link dictionary from response.links
132        headers: Dict[str, Any] = dict(response.headers)
133        headers["link"] = response.links
134        if self._stop_condition:
135            should_stop = self._stop_condition.eval(
136                self.config,
137                response=decoded_response,
138                headers=headers,
139                last_record=last_record,
140                last_page_size=last_page_size,
141                page_size=requested_page_size,
142                stream_slice=stream_slice,
143            )
144            if should_stop:
145                return None
146        token = self._cursor_value.eval(
147            config=self.config,
148            response=decoded_response,
149            headers=headers,
150            last_record=last_record,
151            last_page_size=last_page_size,
152            page_size=requested_page_size,
153            stream_slice=stream_slice,
154        )
155        return token if token else None
156
157    def get_page_size(self) -> Optional[int]:
158        return self._page_size

Pagination strategy that evaluates an interpolated string to define the next page token.

Attributes:
  • page_size (Optional[Union[str, int]]): the number of records to request
  • cursor_value (Union[InterpolatedString, str]): template string evaluating to the cursor value
  • config (Config): connection config
  • stop_condition (Optional[InterpolatedBoolean]): template string evaluating when to stop paginating
  • decoder (Decoder): decoder to decode the response
CursorPaginationStrategy( cursor_value: Union[InterpolatedString, str], config: Mapping[str, Any], parameters: dataclasses.InitVar[typing.Mapping[str, typing.Any]], page_size: Union[str, int, NoneType] = None, stop_condition: Union[InterpolatedBoolean, str, NoneType] = None, decoder: Decoder = <factory>)
cursor_value: Union[InterpolatedString, str]
config: Mapping[str, Any]
parameters: dataclasses.InitVar[typing.Mapping[str, typing.Any]]
page_size: Union[str, int, NoneType] = None
stop_condition: Union[InterpolatedBoolean, str, NoneType] = None
decoder: Decoder
initial_token: Optional[Any]
103    @property
104    def initial_token(self) -> Optional[Any]:
105        """
106        CursorPaginationStrategy does not have an initial value because the next cursor is typically included
107        in the response of the first request. For Resumable Full Refresh streams that checkpoint the page
108        cursor, the next cursor should be read from the state or stream slice object.
109        """
110        return None

CursorPaginationStrategy does not have an initial value because the next cursor is typically included in the response of the first request. For Resumable Full Refresh streams that checkpoint the page cursor, the next cursor should be read from the state or stream slice object.

def next_page_token( self, response: requests.models.Response, last_page_size: int, last_record: Optional[Record], last_page_token_value: Optional[Any] = None, page_size_override: Optional[int] = None, stream_slice: Optional[StreamSlice] = None) -> Optional[Any]:
112    def next_page_token(
113        self,
114        response: requests.Response,
115        last_page_size: int,
116        last_record: Optional[Record],
117        last_page_token_value: Optional[Any] = None,
118        page_size_override: Optional[int] = None,
119        stream_slice: Optional[StreamSlice] = None,
120    ) -> Optional[Any]:
121        # The next page is a cursor read from the response, so `page_size_override` does not change how the token
122        # is computed. It is still exposed to the interpolation context as `page_size` because a `stop_condition`
123        # comparing `last_page_size` to a hardcoded page size would read a full reduced page as a short page and
124        # end the pagination early, silently dropping the rest of the partition. Writing the condition as
125        # `{{ last_page_size < page_size }}` keeps it correct while a reduction is in effect.
126        requested_page_size = (
127            page_size_override if page_size_override is not None else self._page_size
128        )
129        decoded_response = next(self.decoder.decode(response))
130        # The default way that link is presented in requests.Response is a string of various links (last, next, etc). This
131        # is not indexable or useful for parsing the cursor, so we replace it with the link dictionary from response.links
132        headers: Dict[str, Any] = dict(response.headers)
133        headers["link"] = response.links
134        if self._stop_condition:
135            should_stop = self._stop_condition.eval(
136                self.config,
137                response=decoded_response,
138                headers=headers,
139                last_record=last_record,
140                last_page_size=last_page_size,
141                page_size=requested_page_size,
142                stream_slice=stream_slice,
143            )
144            if should_stop:
145                return None
146        token = self._cursor_value.eval(
147            config=self.config,
148            response=decoded_response,
149            headers=headers,
150            last_record=last_record,
151            last_page_size=last_page_size,
152            page_size=requested_page_size,
153            stream_slice=stream_slice,
154        )
155        return token if token else None
Parameters
  • response: response to process
  • last_page_size: the number of records read from the response
  • last_record: the last record extracted from the response
  • last_page_token_value: The current value of the page token made on the last request
  • page_size_override: the page size that was actually requested, when it differs from the configured one because of a REDUCE_PAGE_SIZE response action. Strategies that use their configured page size as a stop condition must honor this value, else they end the pagination early and skip records.
  • stream_slice: the slice the page was read for. Only passed to strategies that declare it, so existing strategies defined outside of the CDK keep working unchanged.
Returns

next page token. Returns None if there are no more pages to fetch

def get_page_size(self) -> Optional[int]:
157    def get_page_size(self) -> Optional[int]:
158        return self._page_size
Returns

page size: The number of records to fetch in a page. Returns None if unspecified

 28@dataclass
 29class DatetimeBasedCursor(DeclarativeCursor):
 30    """
 31    Slices the stream over a datetime range and create a state with format {<cursor_field>: <datetime> }
 32
 33    Given a start time, end time, a step function, and an optional lookback window,
 34    the stream slicer will partition the date range from start time - lookback window to end time.
 35
 36    The step function is defined as a string of the form ISO8601 duration
 37
 38    The timestamp format accepts the same format codes as datetime.strfptime, which are
 39    all the format codes required by the 1989 C standard.
 40    Full list of accepted format codes: https://man7.org/linux/man-pages/man3/strftime.3.html
 41
 42    Attributes:
 43        start_datetime (Union[MinMaxDatetime, str]): the datetime that determines the earliest record that should be synced
 44        end_datetime (Optional[Union[MinMaxDatetime, str]]): the datetime that determines the last record that should be synced
 45        cursor_field (Union[InterpolatedString, str]): record's cursor field
 46        datetime_format (str): format of the datetime
 47        step (Optional[str]): size of the timewindow (ISO8601 duration)
 48        cursor_granularity (Optional[str]): smallest increment the datetime_format has (ISO 8601 duration) that will be used to ensure that the start of a slice does not overlap with the end of the previous one
 49        config (Config): connection config
 50        start_time_option (Optional[RequestOption]): request option for start time
 51        end_time_option (Optional[RequestOption]): request option for end time
 52        partition_field_start (Optional[str]): partition start time field
 53        partition_field_end (Optional[str]): stream slice end time field
 54        lookback_window (Optional[InterpolatedString]): how many days before start_datetime to read data for (ISO8601 duration)
 55    """
 56
 57    start_datetime: Union[MinMaxDatetime, str]
 58    cursor_field: Union[InterpolatedString, str]
 59    datetime_format: str
 60    config: Config
 61    parameters: InitVar[Mapping[str, Any]]
 62    _highest_observed_cursor_field_value: Optional[str] = field(
 63        repr=False, default=None
 64    )  # tracks the latest observed datetime, which may not be safe to emit in the case of out-of-order records
 65    _cursor: Optional[str] = field(
 66        repr=False, default=None
 67    )  # tracks the latest observed datetime that is appropriate to emit as stream state
 68    end_datetime: Optional[Union[MinMaxDatetime, str]] = None
 69    step: Optional[Union[InterpolatedString, str]] = None
 70    cursor_granularity: Optional[str] = None
 71    start_time_option: Optional[RequestOption] = None
 72    end_time_option: Optional[RequestOption] = None
 73    partition_field_start: Optional[str] = None
 74    partition_field_end: Optional[str] = None
 75    lookback_window: Optional[Union[InterpolatedString, str]] = None
 76    message_repository: Optional[MessageRepository] = None
 77    is_compare_strictly: Optional[bool] = False
 78    cursor_datetime_formats: List[str] = field(default_factory=lambda: [])
 79
 80    def __post_init__(self, parameters: Mapping[str, Any]) -> None:
 81        if (self.step and not self.cursor_granularity) or (
 82            not self.step and self.cursor_granularity
 83        ):
 84            raise ValueError(
 85                f"If step is defined, cursor_granularity should be as well and vice-versa. "
 86                f"Right now, step is `{self.step}` and cursor_granularity is `{self.cursor_granularity}`"
 87            )
 88        self._start_datetime = MinMaxDatetime.create(self.start_datetime, parameters)
 89        self._end_datetime = (
 90            None if not self.end_datetime else MinMaxDatetime.create(self.end_datetime, parameters)
 91        )
 92
 93        self._timezone = datetime.timezone.utc
 94        self._interpolation = JinjaInterpolation()
 95
 96        self._step = (
 97            self._parse_timedelta(
 98                InterpolatedString.create(self.step, parameters=parameters).eval(self.config)
 99            )
100            if self.step
101            else datetime.timedelta.max
102        )
103        self._cursor_granularity = self._parse_timedelta(self.cursor_granularity)
104        self.cursor_field = InterpolatedString.create(self.cursor_field, parameters=parameters)
105        self._lookback_window = (
106            InterpolatedString.create(self.lookback_window, parameters=parameters)
107            if self.lookback_window
108            else None
109        )
110        self._partition_field_start = InterpolatedString.create(
111            self.partition_field_start or "start_time", parameters=parameters
112        )
113        self._partition_field_end = InterpolatedString.create(
114            self.partition_field_end or "end_time", parameters=parameters
115        )
116        self._parser = DatetimeParser()
117
118        # If datetime format is not specified then start/end datetime should inherit it from the stream slicer
119        if not self._start_datetime.datetime_format:
120            self._start_datetime.datetime_format = self.datetime_format
121        if self._end_datetime and not self._end_datetime.datetime_format:
122            self._end_datetime.datetime_format = self.datetime_format
123
124        if not self.cursor_datetime_formats:
125            self.cursor_datetime_formats = [self.datetime_format]
126
127        _validate_component_request_option_paths(
128            self.config, self.start_time_option, self.end_time_option
129        )
130
131    def get_stream_state(self) -> StreamState:
132        return {self.cursor_field.eval(self.config): self._cursor} if self._cursor else {}  # type: ignore  # cursor_field is converted to an InterpolatedString in __post_init__
133
134    def set_initial_state(self, stream_state: StreamState) -> None:
135        """
136        Cursors are not initialized with their state. As state is needed in order to function properly, this method should be called
137        before calling anything else
138
139        :param stream_state: The state of the stream as returned by get_stream_state
140        """
141        self._cursor = (
142            stream_state.get(self.cursor_field.eval(self.config)) if stream_state else None  # type: ignore [union-attr]
143        )
144
145    def observe(self, stream_slice: StreamSlice, record: Record) -> None:
146        """
147        Register a record with the cursor; the cursor instance can then use it to manage the state of the in-progress stream read.
148
149        :param stream_slice: The current slice, which may or may not contain the most recently observed record
150        :param record: the most recently-read record, which the cursor can use to update the stream state. Outwardly-visible changes to the
151          stream state may need to be deferred depending on whether the source reliably orders records by the cursor field.
152        """
153        record_cursor_value = record.get(self.cursor_field.eval(self.config))  # type: ignore  # cursor_field is converted to an InterpolatedString in __post_init__
154        # if the current record has no cursor value, we cannot meaningfully update the state based on it, so there is nothing more to do
155        if not record_cursor_value:
156            return
157
158        start_field = self._partition_field_start.eval(self.config)
159        end_field = self._partition_field_end.eval(self.config)
160        is_highest_observed_cursor_value = (
161            not self._highest_observed_cursor_field_value
162            or self.parse_date(record_cursor_value)
163            > self.parse_date(self._highest_observed_cursor_field_value)
164        )
165        if (
166            self._is_within_daterange_boundaries(
167                record,
168                stream_slice.get(start_field),  # type: ignore [arg-type]
169                stream_slice.get(end_field),  # type: ignore [arg-type]
170            )
171            and is_highest_observed_cursor_value
172        ):
173            self._highest_observed_cursor_field_value = record_cursor_value
174
175    def close_slice(self, stream_slice: StreamSlice, *args: Any) -> None:
176        if stream_slice.partition:
177            raise ValueError(
178                f"Stream slice {stream_slice} should not have a partition. Got {stream_slice.partition}."
179            )
180        cursor_value_str_by_cursor_value_datetime = dict(
181            map(
182                # we need to ensure the cursor value is preserved as is in the state else the CATs might complain of something like
183                # 2023-01-04T17:30:19.000Z' <= '2023-01-04T17:30:19.000000Z'
184                lambda datetime_str: (self.parse_date(datetime_str), datetime_str),  # type: ignore # because of the filter on the next line, this will only be called with a str
185                filter(
186                    lambda item: item, [self._cursor, self._highest_observed_cursor_field_value]
187                ),
188            )
189        )
190        self._cursor = (
191            cursor_value_str_by_cursor_value_datetime[
192                max(cursor_value_str_by_cursor_value_datetime.keys())
193            ]
194            if cursor_value_str_by_cursor_value_datetime
195            else None
196        )
197
198    def stream_slices(self) -> Iterable[StreamSlice]:
199        """
200        Partition the daterange into slices of size = step.
201
202        The start of the window is the minimum datetime between start_datetime - lookback_window and the stream_state's datetime
203        The end of the window is the minimum datetime between the start of the window and end_datetime.
204
205        :return:
206        """
207        end_datetime = self.select_best_end_datetime()
208        start_datetime = self._calculate_earliest_possible_value(self.select_best_end_datetime())
209        return self._partition_daterange(start_datetime, end_datetime, self._step)
210
211    def select_state(self, stream_slice: Optional[StreamSlice] = None) -> Optional[StreamState]:
212        # Datetime based cursors operate over slices made up of datetime ranges. Stream state is based on the progress
213        # through each slice and does not belong to a specific slice. We just return stream state as it is.
214        return self.get_stream_state()
215
216    def _calculate_earliest_possible_value(
217        self, end_datetime: datetime.datetime
218    ) -> datetime.datetime:
219        lookback_delta = self._parse_timedelta(
220            self._lookback_window.eval(self.config) if self._lookback_window else "P0D"
221        )
222        earliest_possible_start_datetime = min(
223            self._start_datetime.get_datetime(self.config), end_datetime
224        )
225        try:
226            cursor_datetime = (
227                self._calculate_cursor_datetime_from_state(self.get_stream_state()) - lookback_delta
228            )
229        except OverflowError:
230            # cursor_datetime defers to the minimum date if it does not exist in the state. Trying to subtract
231            # a timedelta from the minimum datetime results in an OverflowError
232            cursor_datetime = self._calculate_cursor_datetime_from_state(self.get_stream_state())
233        return max(earliest_possible_start_datetime, cursor_datetime)
234
235    def select_best_end_datetime(self) -> datetime.datetime:
236        """
237        Returns the optimal end datetime.
238        This method compares the current datetime with a pre-configured end datetime
239        and returns the earlier of the two. If no pre-configured end datetime is set,
240        the current datetime is returned.
241
242        :return datetime.datetime: The best end datetime, which is either the current datetime or the pre-configured end datetime, whichever is earlier.
243        """
244        now = datetime.datetime.now(tz=self._timezone)
245        if not self._end_datetime:
246            return now
247        return min(self._end_datetime.get_datetime(self.config), now)
248
249    def _calculate_cursor_datetime_from_state(
250        self, stream_state: Mapping[str, Any]
251    ) -> datetime.datetime:
252        if self.cursor_field.eval(self.config, stream_state=stream_state) in stream_state:  # type: ignore  # cursor_field is converted to an InterpolatedString in __post_init__
253            return self.parse_date(stream_state[self.cursor_field.eval(self.config)])  # type: ignore  # cursor_field is converted to an InterpolatedString in __post_init__
254        return datetime.datetime.min.replace(tzinfo=datetime.timezone.utc)
255
256    def _format_datetime(self, dt: datetime.datetime) -> str:
257        return self._parser.format(dt, self.datetime_format)
258
259    def _partition_daterange(
260        self,
261        start: datetime.datetime,
262        end: datetime.datetime,
263        step: Union[datetime.timedelta, Duration],
264    ) -> List[StreamSlice]:
265        start_field = self._partition_field_start.eval(self.config)
266        end_field = self._partition_field_end.eval(self.config)
267        dates = []
268
269        while self._is_within_date_range(start, end):
270            next_start = self._evaluate_next_start_date_safely(start, step)
271            end_date = self._get_date(next_start - self._cursor_granularity, end, min)
272            dates.append(
273                StreamSlice(
274                    partition={},
275                    cursor_slice={
276                        start_field: self._format_datetime(start),
277                        end_field: self._format_datetime(end_date),
278                    },
279                )
280            )
281            start = next_start
282        return dates
283
284    def _is_within_date_range(self, start: datetime.datetime, end: datetime.datetime) -> bool:
285        if self.is_compare_strictly:
286            return start < end
287        return start <= end
288
289    def _evaluate_next_start_date_safely(
290        self, start: datetime.datetime, step: datetime.timedelta
291    ) -> datetime.datetime:
292        """
293        Given that we set the default step at datetime.timedelta.max, we will generate an OverflowError when evaluating the next start_date
294        This method assumes that users would never enter a step that would generate an overflow. Given that would be the case, the code
295        would have broken anyway.
296        """
297        try:
298            return start + step
299        except OverflowError:
300            return datetime.datetime.max.replace(tzinfo=datetime.timezone.utc)
301
302    def _get_date(
303        self,
304        cursor_value: datetime.datetime,
305        default_date: datetime.datetime,
306        comparator: Callable[[datetime.datetime, datetime.datetime], datetime.datetime],
307    ) -> datetime.datetime:
308        cursor_date = cursor_value or default_date
309        return comparator(cursor_date, default_date)
310
311    def parse_date(self, date: str) -> datetime.datetime:
312        for datetime_format in self.cursor_datetime_formats + [self.datetime_format]:
313            try:
314                return self._parser.parse(date, datetime_format)
315            except ValueError:
316                pass
317        raise ValueError(f"No format in {self.cursor_datetime_formats} matching {date}")
318
319    @classmethod
320    def _parse_timedelta(cls, time_str: Optional[str]) -> Union[datetime.timedelta, Duration]:
321        """
322        :return Parses an ISO 8601 durations into datetime.timedelta or Duration objects.
323        """
324        if not time_str:
325            return datetime.timedelta(0)
326        return parse_duration(time_str)
327
328    def get_request_params(
329        self,
330        *,
331        stream_state: Optional[StreamState] = None,
332        stream_slice: Optional[StreamSlice] = None,
333        next_page_token: Optional[Mapping[str, Any]] = None,
334    ) -> Mapping[str, Any]:
335        return self._get_request_options(RequestOptionType.request_parameter, stream_slice)
336
337    def get_request_headers(
338        self,
339        *,
340        stream_state: Optional[StreamState] = None,
341        stream_slice: Optional[StreamSlice] = None,
342        next_page_token: Optional[Mapping[str, Any]] = None,
343    ) -> Mapping[str, Any]:
344        return self._get_request_options(RequestOptionType.header, stream_slice)
345
346    def get_request_body_data(
347        self,
348        *,
349        stream_state: Optional[StreamState] = None,
350        stream_slice: Optional[StreamSlice] = None,
351        next_page_token: Optional[Mapping[str, Any]] = None,
352    ) -> Mapping[str, Any]:
353        return self._get_request_options(RequestOptionType.body_data, stream_slice)
354
355    def get_request_body_json(
356        self,
357        *,
358        stream_state: Optional[StreamState] = None,
359        stream_slice: Optional[StreamSlice] = None,
360        next_page_token: Optional[Mapping[str, Any]] = None,
361    ) -> Mapping[str, Any]:
362        return self._get_request_options(RequestOptionType.body_json, stream_slice)
363
364    def request_kwargs(self) -> Mapping[str, Any]:
365        # Never update kwargs
366        return {}
367
368    def _get_request_options(
369        self, option_type: RequestOptionType, stream_slice: Optional[StreamSlice]
370    ) -> Mapping[str, Any]:
371        options: MutableMapping[str, Any] = {}
372        if not stream_slice:
373            return options
374
375        if self.start_time_option and self.start_time_option.inject_into == option_type:
376            start_time_value = stream_slice.get(self._partition_field_start.eval(self.config))
377            self.start_time_option.inject_into_request(options, start_time_value, self.config)
378
379        if self.end_time_option and self.end_time_option.inject_into == option_type:
380            end_time_value = stream_slice.get(self._partition_field_end.eval(self.config))
381            self.end_time_option.inject_into_request(options, end_time_value, self.config)
382
383        return options
384
385    def should_be_synced(self, record: Record) -> bool:
386        cursor_field = self.cursor_field.eval(self.config)  # type: ignore  # cursor_field is converted to an InterpolatedString in __post_init__
387        record_cursor_value = record.get(cursor_field)
388        if not record_cursor_value:
389            self._send_log(
390                Level.WARN,
391                f"Could not find cursor field `{cursor_field}` in record. The incremental sync will assume it needs to be synced",
392            )
393            return True
394        latest_possible_cursor_value = self.select_best_end_datetime()
395        earliest_possible_cursor_value = self._calculate_earliest_possible_value(
396            latest_possible_cursor_value
397        )
398        return self._is_within_daterange_boundaries(
399            record, earliest_possible_cursor_value, latest_possible_cursor_value
400        )
401
402    def _is_within_daterange_boundaries(
403        self,
404        record: Record,
405        start_datetime_boundary: Union[datetime.datetime, str],
406        end_datetime_boundary: Union[datetime.datetime, str],
407    ) -> bool:
408        cursor_field = self.cursor_field.eval(self.config)  # type: ignore  # cursor_field is converted to an InterpolatedString in __post_init__
409        record_cursor_value = record.get(cursor_field)
410        if not record_cursor_value:
411            self._send_log(
412                Level.WARN,
413                f"Could not find cursor field `{cursor_field}` in record. The record will not be considered when emitting sync state",
414            )
415            return False
416        if isinstance(start_datetime_boundary, str):
417            start_datetime_boundary = self.parse_date(start_datetime_boundary)
418        if isinstance(end_datetime_boundary, str):
419            end_datetime_boundary = self.parse_date(end_datetime_boundary)
420        return (
421            start_datetime_boundary <= self.parse_date(record_cursor_value) <= end_datetime_boundary
422        )
423
424    def _send_log(self, level: Level, message: str) -> None:
425        if self.message_repository:
426            self.message_repository.emit_message(
427                AirbyteMessage(
428                    type=Type.LOG,
429                    log=AirbyteLogMessage(level=level, message=message),
430                )
431            )
432
433    def set_runtime_lookback_window(self, lookback_window_in_seconds: int) -> None:
434        """
435        Updates the lookback window based on a given number of seconds if the new duration
436        is greater than the currently configured lookback window.
437
438        :param lookback_window_in_seconds: The lookback duration in seconds to potentially update to.
439        """
440        runtime_lookback_window = duration_isoformat(timedelta(seconds=lookback_window_in_seconds))
441        config_lookback = parse_duration(
442            self._lookback_window.eval(self.config) if self._lookback_window else "P0D"
443        )
444
445        # Check if the new runtime lookback window is greater than the current config lookback
446        if parse_duration(runtime_lookback_window) > config_lookback:
447            self._lookback_window = InterpolatedString.create(
448                runtime_lookback_window, parameters={}
449            )

Slices the stream over a datetime range and create a state with format {: }

Given a start time, end time, a step function, and an optional lookback window, the stream slicer will partition the date range from start time - lookback window to end time.

The step function is defined as a string of the form ISO8601 duration

The timestamp format accepts the same format codes as datetime.strfptime, which are all the format codes required by the 1989 C standard. Full list of accepted format codes: https://man7.org/linux/man-pages/man3/strftime.3.html

Attributes:
  • start_datetime (Union[MinMaxDatetime, str]): the datetime that determines the earliest record that should be synced
  • end_datetime (Optional[Union[MinMaxDatetime, str]]): the datetime that determines the last record that should be synced
  • cursor_field (Union[InterpolatedString, str]): record's cursor field
  • datetime_format (str): format of the datetime
  • step (Optional[str]): size of the timewindow (ISO8601 duration)
  • cursor_granularity (Optional[str]): smallest increment the datetime_format has (ISO 8601 duration) that will be used to ensure that the start of a slice does not overlap with the end of the previous one
  • config (Config): connection config
  • start_time_option (Optional[RequestOption]): request option for start time
  • end_time_option (Optional[RequestOption]): request option for end time
  • partition_field_start (Optional[str]): partition start time field
  • partition_field_end (Optional[str]): stream slice end time field
  • lookback_window (Optional[InterpolatedString]): how many days before start_datetime to read data for (ISO8601 duration)
DatetimeBasedCursor( start_datetime: Union[MinMaxDatetime, str], cursor_field: Union[InterpolatedString, str], datetime_format: str, config: Mapping[str, Any], parameters: dataclasses.InitVar[typing.Mapping[str, typing.Any]], _highest_observed_cursor_field_value: Optional[str] = None, _cursor: Optional[str] = None, end_datetime: Union[MinMaxDatetime, str, NoneType] = None, step: Union[InterpolatedString, str, NoneType] = None, cursor_granularity: Optional[str] = None, start_time_option: Optional[RequestOption] = None, end_time_option: Optional[RequestOption] = None, partition_field_start: Optional[str] = None, partition_field_end: Optional[str] = None, lookback_window: Union[InterpolatedString, str, NoneType] = None, message_repository: Optional[MessageRepository] = None, is_compare_strictly: Optional[bool] = False, cursor_datetime_formats: List[str] = <factory>)
start_datetime: Union[MinMaxDatetime, str]
cursor_field: Union[InterpolatedString, str]
datetime_format: str
config: Mapping[str, Any]
parameters: dataclasses.InitVar[typing.Mapping[str, typing.Any]]
end_datetime: Union[MinMaxDatetime, str, NoneType] = None
step: Union[InterpolatedString, str, NoneType] = None
cursor_granularity: Optional[str] = None
start_time_option: Optional[RequestOption] = None
end_time_option: Optional[RequestOption] = None
partition_field_start: Optional[str] = None
partition_field_end: Optional[str] = None
lookback_window: Union[InterpolatedString, str, NoneType] = None
message_repository: Optional[MessageRepository] = None
is_compare_strictly: Optional[bool] = False
cursor_datetime_formats: List[str]
def get_stream_state(self) -> Mapping[str, Any]:
131    def get_stream_state(self) -> StreamState:
132        return {self.cursor_field.eval(self.config): self._cursor} if self._cursor else {}  # type: ignore  # cursor_field is converted to an InterpolatedString in __post_init__

Returns the current stream state. We would like to restrict it's usage since it does expose internal of state. As of 2023-06-14, it is used for two things:

  • Interpolation of the requests
  • Transformation of records
  • Saving the state

For the first case, we are probably stuck with exposing the stream state. For the second, we can probably expose a method that allows for emitting the state to the platform.

def set_initial_state(self, stream_state: Mapping[str, Any]) -> None:
134    def set_initial_state(self, stream_state: StreamState) -> None:
135        """
136        Cursors are not initialized with their state. As state is needed in order to function properly, this method should be called
137        before calling anything else
138
139        :param stream_state: The state of the stream as returned by get_stream_state
140        """
141        self._cursor = (
142            stream_state.get(self.cursor_field.eval(self.config)) if stream_state else None  # type: ignore [union-attr]
143        )

Cursors are not initialized with their state. As state is needed in order to function properly, this method should be called before calling anything else

Parameters
  • stream_state: The state of the stream as returned by get_stream_state
def observe( self, stream_slice: StreamSlice, record: Record) -> None:
145    def observe(self, stream_slice: StreamSlice, record: Record) -> None:
146        """
147        Register a record with the cursor; the cursor instance can then use it to manage the state of the in-progress stream read.
148
149        :param stream_slice: The current slice, which may or may not contain the most recently observed record
150        :param record: the most recently-read record, which the cursor can use to update the stream state. Outwardly-visible changes to the
151          stream state may need to be deferred depending on whether the source reliably orders records by the cursor field.
152        """
153        record_cursor_value = record.get(self.cursor_field.eval(self.config))  # type: ignore  # cursor_field is converted to an InterpolatedString in __post_init__
154        # if the current record has no cursor value, we cannot meaningfully update the state based on it, so there is nothing more to do
155        if not record_cursor_value:
156            return
157
158        start_field = self._partition_field_start.eval(self.config)
159        end_field = self._partition_field_end.eval(self.config)
160        is_highest_observed_cursor_value = (
161            not self._highest_observed_cursor_field_value
162            or self.parse_date(record_cursor_value)
163            > self.parse_date(self._highest_observed_cursor_field_value)
164        )
165        if (
166            self._is_within_daterange_boundaries(
167                record,
168                stream_slice.get(start_field),  # type: ignore [arg-type]
169                stream_slice.get(end_field),  # type: ignore [arg-type]
170            )
171            and is_highest_observed_cursor_value
172        ):
173            self._highest_observed_cursor_field_value = record_cursor_value

Register a record with the cursor; the cursor instance can then use it to manage the state of the in-progress stream read.

Parameters
  • stream_slice: The current slice, which may or may not contain the most recently observed record
  • record: the most recently-read record, which the cursor can use to update the stream state. Outwardly-visible changes to the stream state may need to be deferred depending on whether the source reliably orders records by the cursor field.
def close_slice( self, stream_slice: StreamSlice, *args: Any) -> None:
175    def close_slice(self, stream_slice: StreamSlice, *args: Any) -> None:
176        if stream_slice.partition:
177            raise ValueError(
178                f"Stream slice {stream_slice} should not have a partition. Got {stream_slice.partition}."
179            )
180        cursor_value_str_by_cursor_value_datetime = dict(
181            map(
182                # we need to ensure the cursor value is preserved as is in the state else the CATs might complain of something like
183                # 2023-01-04T17:30:19.000Z' <= '2023-01-04T17:30:19.000000Z'
184                lambda datetime_str: (self.parse_date(datetime_str), datetime_str),  # type: ignore # because of the filter on the next line, this will only be called with a str
185                filter(
186                    lambda item: item, [self._cursor, self._highest_observed_cursor_field_value]
187                ),
188            )
189        )
190        self._cursor = (
191            cursor_value_str_by_cursor_value_datetime[
192                max(cursor_value_str_by_cursor_value_datetime.keys())
193            ]
194            if cursor_value_str_by_cursor_value_datetime
195            else None
196        )

Update state based on the stream slice. Note that stream_slice.cursor_slice and most_recent_record.associated_slice are expected to be the same but we make it explicit here that stream_slice should be leveraged to update the state. We do not pass in the latest record, since cursor instances should maintain the relevant internal state on their own.

Parameters
  • stream_slice: slice to close
def stream_slices(self) -> Iterable[StreamSlice]:
198    def stream_slices(self) -> Iterable[StreamSlice]:
199        """
200        Partition the daterange into slices of size = step.
201
202        The start of the window is the minimum datetime between start_datetime - lookback_window and the stream_state's datetime
203        The end of the window is the minimum datetime between the start of the window and end_datetime.
204
205        :return:
206        """
207        end_datetime = self.select_best_end_datetime()
208        start_datetime = self._calculate_earliest_possible_value(self.select_best_end_datetime())
209        return self._partition_daterange(start_datetime, end_datetime, self._step)

Partition the daterange into slices of size = step.

The start of the window is the minimum datetime between start_datetime - lookback_window and the stream_state's datetime The end of the window is the minimum datetime between the start of the window and end_datetime.

Returns
def select_state( self, stream_slice: Optional[StreamSlice] = None) -> Optional[Mapping[str, Any]]:
211    def select_state(self, stream_slice: Optional[StreamSlice] = None) -> Optional[StreamState]:
212        # Datetime based cursors operate over slices made up of datetime ranges. Stream state is based on the progress
213        # through each slice and does not belong to a specific slice. We just return stream state as it is.
214        return self.get_stream_state()

Get the state value of a specific stream_slice. For incremental or resumable full refresh cursors which only manage state in a single dimension this is the entire state object. For per-partition cursors used by substreams, this returns the state of a specific parent delineated by the incoming slice's partition object.

def select_best_end_datetime(self) -> datetime.datetime:
235    def select_best_end_datetime(self) -> datetime.datetime:
236        """
237        Returns the optimal end datetime.
238        This method compares the current datetime with a pre-configured end datetime
239        and returns the earlier of the two. If no pre-configured end datetime is set,
240        the current datetime is returned.
241
242        :return datetime.datetime: The best end datetime, which is either the current datetime or the pre-configured end datetime, whichever is earlier.
243        """
244        now = datetime.datetime.now(tz=self._timezone)
245        if not self._end_datetime:
246            return now
247        return min(self._end_datetime.get_datetime(self.config), now)

Returns the optimal end datetime. This method compares the current datetime with a pre-configured end datetime and returns the earlier of the two. If no pre-configured end datetime is set, the current datetime is returned.

Returns

The best end datetime, which is either the current datetime or the pre-configured end datetime, whichever is earlier.

def parse_date(self, date: str) -> datetime.datetime:
311    def parse_date(self, date: str) -> datetime.datetime:
312        for datetime_format in self.cursor_datetime_formats + [self.datetime_format]:
313            try:
314                return self._parser.parse(date, datetime_format)
315            except ValueError:
316                pass
317        raise ValueError(f"No format in {self.cursor_datetime_formats} matching {date}")
def get_request_params( self, *, stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[StreamSlice] = None, next_page_token: Optional[Mapping[str, Any]] = None) -> Mapping[str, Any]:
328    def get_request_params(
329        self,
330        *,
331        stream_state: Optional[StreamState] = None,
332        stream_slice: Optional[StreamSlice] = None,
333        next_page_token: Optional[Mapping[str, Any]] = None,
334    ) -> Mapping[str, Any]:
335        return self._get_request_options(RequestOptionType.request_parameter, stream_slice)

Specifies the query parameters that should be set on an outgoing HTTP request given the inputs.

E.g: you might want to define query parameters for paging if next_page_token is not None.

def get_request_headers( self, *, stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[StreamSlice] = None, next_page_token: Optional[Mapping[str, Any]] = None) -> Mapping[str, Any]:
337    def get_request_headers(
338        self,
339        *,
340        stream_state: Optional[StreamState] = None,
341        stream_slice: Optional[StreamSlice] = None,
342        next_page_token: Optional[Mapping[str, Any]] = None,
343    ) -> Mapping[str, Any]:
344        return self._get_request_options(RequestOptionType.header, stream_slice)

Return any non-auth headers. Authentication headers will overwrite any overlapping headers returned from this method.

def get_request_body_data( self, *, stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[StreamSlice] = None, next_page_token: Optional[Mapping[str, Any]] = None) -> Mapping[str, Any]:
346    def get_request_body_data(
347        self,
348        *,
349        stream_state: Optional[StreamState] = None,
350        stream_slice: Optional[StreamSlice] = None,
351        next_page_token: Optional[Mapping[str, Any]] = None,
352    ) -> Mapping[str, Any]:
353        return self._get_request_options(RequestOptionType.body_data, stream_slice)

Specifies how to populate the body of the request with a non-JSON payload.

If returns a ready text that it will be sent as is. If returns a dict that it will be converted to a urlencoded form. E.g. {"key1": "value1", "key2": "value2"} => "key1=value1&key2=value2"

At the same time only one of the 'request_body_data' and 'request_body_json' functions can be overridden.

def get_request_body_json( self, *, stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[StreamSlice] = None, next_page_token: Optional[Mapping[str, Any]] = None) -> Mapping[str, Any]:
355    def get_request_body_json(
356        self,
357        *,
358        stream_state: Optional[StreamState] = None,
359        stream_slice: Optional[StreamSlice] = None,
360        next_page_token: Optional[Mapping[str, Any]] = None,
361    ) -> Mapping[str, Any]:
362        return self._get_request_options(RequestOptionType.body_json, stream_slice)

Specifies how to populate the body of the request with a JSON payload.

At the same time only one of the 'request_body_data' and 'request_body_json' functions can be overridden.

def request_kwargs(self) -> Mapping[str, Any]:
364    def request_kwargs(self) -> Mapping[str, Any]:
365        # Never update kwargs
366        return {}
def should_be_synced(self, record: Record) -> bool:
385    def should_be_synced(self, record: Record) -> bool:
386        cursor_field = self.cursor_field.eval(self.config)  # type: ignore  # cursor_field is converted to an InterpolatedString in __post_init__
387        record_cursor_value = record.get(cursor_field)
388        if not record_cursor_value:
389            self._send_log(
390                Level.WARN,
391                f"Could not find cursor field `{cursor_field}` in record. The incremental sync will assume it needs to be synced",
392            )
393            return True
394        latest_possible_cursor_value = self.select_best_end_datetime()
395        earliest_possible_cursor_value = self._calculate_earliest_possible_value(
396            latest_possible_cursor_value
397        )
398        return self._is_within_daterange_boundaries(
399            record, earliest_possible_cursor_value, latest_possible_cursor_value
400        )

Evaluating if a record should be synced allows for filtering and stop condition on pagination

def set_runtime_lookback_window(self, lookback_window_in_seconds: int) -> None:
433    def set_runtime_lookback_window(self, lookback_window_in_seconds: int) -> None:
434        """
435        Updates the lookback window based on a given number of seconds if the new duration
436        is greater than the currently configured lookback window.
437
438        :param lookback_window_in_seconds: The lookback duration in seconds to potentially update to.
439        """
440        runtime_lookback_window = duration_isoformat(timedelta(seconds=lookback_window_in_seconds))
441        config_lookback = parse_duration(
442            self._lookback_window.eval(self.config) if self._lookback_window else "P0D"
443        )
444
445        # Check if the new runtime lookback window is greater than the current config lookback
446        if parse_duration(runtime_lookback_window) > config_lookback:
447            self._lookback_window = InterpolatedString.create(
448                runtime_lookback_window, parameters={}
449            )

Updates the lookback window based on a given number of seconds if the new duration is greater than the currently configured lookback window.

Parameters
  • lookback_window_in_seconds: The lookback duration in seconds to potentially update to.
@dataclass
class DeclarativeAuthenticator(airbyte_cdk.AbstractHeaderAuthenticator):
14@dataclass
15class DeclarativeAuthenticator(AbstractHeaderAuthenticator):
16    """
17    Interface used to associate which authenticators can be used as part of the declarative framework
18    """
19
20    def get_request_params(self) -> Mapping[str, Any]:
21        """HTTP request parameter to add to the requests"""
22        return {}
23
24    def get_request_body_data(self) -> Union[Mapping[str, Any], str]:
25        """Form-encoded body data to set on the requests"""
26        return {}
27
28    def get_request_body_json(self) -> Mapping[str, Any]:
29        """JSON-encoded body data to set on the requests"""
30        return {}

Interface used to associate which authenticators can be used as part of the declarative framework

def get_request_params(self) -> Mapping[str, Any]:
20    def get_request_params(self) -> Mapping[str, Any]:
21        """HTTP request parameter to add to the requests"""
22        return {}

HTTP request parameter to add to the requests

def get_request_body_data(self) -> Union[Mapping[str, Any], str]:
24    def get_request_body_data(self) -> Union[Mapping[str, Any], str]:
25        """Form-encoded body data to set on the requests"""
26        return {}

Form-encoded body data to set on the requests

def get_request_body_json(self) -> Mapping[str, Any]:
28    def get_request_body_json(self) -> Mapping[str, Any]:
29        """JSON-encoded body data to set on the requests"""
30        return {}

JSON-encoded body data to set on the requests

 28@dataclass
 29class DeclarativeOauth2Authenticator(AbstractOauth2Authenticator, DeclarativeAuthenticator):
 30    """
 31    Generates OAuth2.0 access tokens from an OAuth2.0 refresh token and client credentials based on
 32    a declarative connector configuration file. Credentials can be defined explicitly or via interpolation
 33    at runtime. The generated access token is attached to each request via the Authorization header.
 34
 35    Attributes:
 36        token_refresh_endpoint (Union[InterpolatedString, str]): The endpoint to refresh the access token
 37        client_id (Union[InterpolatedString, str]): The client id
 38        client_secret (Union[InterpolatedString, str]): Client secret (can be empty for APIs that support this)
 39        refresh_token (Union[InterpolatedString, str]): The token used to refresh the access token
 40        access_token_name (Union[InterpolatedString, str]): THe field to extract access token from in the response
 41        expires_in_name (Union[InterpolatedString, str]): The field to extract expires_in from in the response
 42        config (Mapping[str, Any]): The user-provided configuration as specified by the source's spec
 43        scopes (Optional[List[str]]): The scopes to request
 44        token_expiry_date (Optional[Union[InterpolatedString, str]]): The access token expiration date
 45        token_expiry_date_format str: format of the datetime; provide it if expires_in is returned in datetime instead of seconds
 46        token_expiry_is_time_of_expiration bool: set True it if expires_in is returned as time of expiration instead of the number seconds until expiration
 47        refresh_request_body (Optional[Mapping[str, Any]]): The request body to send in the refresh request
 48        refresh_request_headers (Optional[Mapping[str, Any]]): The request headers to send in the refresh request
 49        send_refresh_request_as_query_params (bool): When True, the standard refresh args (`grant_type`, `refresh_token`, client credentials when not in an `Authorization` header, scopes, plus any `refresh_request_body` extras) are sent on the URL query string instead of in the request body, and the body is emitted empty. Use this for OAuth providers like Gong that document their refresh endpoint with refresh args on the URL query string. Defaults to False.
 50        grant_type: The grant_type to request for access_token. If set to refresh_token, the refresh_token parameter has to be provided
 51        message_repository (MessageRepository): the message repository used to emit logs on HTTP requests
 52        refresh_token_error_status_codes (Tuple[int, ...]): Status codes to identify refresh token errors in response
 53        refresh_token_error_key (str): Key to identify refresh token error in response
 54        refresh_token_error_values (Tuple[str, ...]): List of values to check for exception during token refresh process
 55    """
 56
 57    config: Mapping[str, Any]
 58    parameters: InitVar[Mapping[str, Any]]
 59    client_id: Optional[Union[InterpolatedString, str]] = None
 60    client_secret: Optional[Union[InterpolatedString, str]] = None
 61    token_refresh_endpoint: Optional[Union[InterpolatedString, str]] = None
 62    refresh_token: Optional[Union[InterpolatedString, str]] = None
 63    scopes: Optional[List[str]] = None
 64    token_expiry_date: Optional[Union[InterpolatedString, str]] = None
 65    _token_expiry_date: Optional[AirbyteDateTime] = field(init=False, repr=False, default=None)
 66    token_expiry_date_format: Optional[str] = None
 67    token_expiry_is_time_of_expiration: bool = False
 68    access_token_name: Union[InterpolatedString, str] = "access_token"
 69    access_token_value: Optional[Union[InterpolatedString, str]] = None
 70    client_id_name: Union[InterpolatedString, str] = "client_id"
 71    client_secret_name: Union[InterpolatedString, str] = "client_secret"
 72    expires_in_name: Union[InterpolatedString, str] = "expires_in"
 73    refresh_token_name: Union[InterpolatedString, str] = "refresh_token"
 74    refresh_request_body: Optional[Mapping[str, Any]] = None
 75    refresh_request_headers: Optional[Mapping[str, Any]] = None
 76    send_refresh_request_as_query_params: bool = False
 77    grant_type_name: Union[InterpolatedString, str] = "grant_type"
 78    grant_type: Union[InterpolatedString, str] = "refresh_token"
 79    message_repository: MessageRepository = NoopMessageRepository()
 80    profile_assertion: Optional[DeclarativeAuthenticator] = None
 81    use_profile_assertion: Optional[Union[InterpolatedBoolean, str, bool]] = False
 82    refresh_token_error_status_codes: Tuple[int, ...] = ()
 83    refresh_token_error_key: str = ""
 84    refresh_token_error_values: Tuple[str, ...] = ()
 85
 86    def __post_init__(self, parameters: Mapping[str, Any]) -> None:
 87        super().__init__(
 88            refresh_token_error_status_codes=self.refresh_token_error_status_codes,
 89            refresh_token_error_key=self.refresh_token_error_key,
 90            refresh_token_error_values=self.refresh_token_error_values,
 91        )
 92        if self.token_refresh_endpoint is not None:
 93            self._token_refresh_endpoint: Optional[InterpolatedString] = InterpolatedString.create(
 94                self.token_refresh_endpoint, parameters=parameters
 95            )
 96        else:
 97            self._token_refresh_endpoint = None
 98        self._client_id_name = InterpolatedString.create(self.client_id_name, parameters=parameters)
 99        self._client_id = (
100            InterpolatedString.create(self.client_id, parameters=parameters)
101            if self.client_id
102            else self.client_id
103        )
104        self._client_secret_name = InterpolatedString.create(
105            self.client_secret_name, parameters=parameters
106        )
107        self._client_secret = (
108            InterpolatedString.create(self.client_secret, parameters=parameters)
109            if self.client_secret
110            else self.client_secret
111        )
112        self._refresh_token_name = InterpolatedString.create(
113            self.refresh_token_name, parameters=parameters
114        )
115        if self.refresh_token is not None:
116            self._refresh_token: Optional[InterpolatedString] = InterpolatedString.create(
117                self.refresh_token, parameters=parameters
118            )
119        else:
120            self._refresh_token = None
121        self.access_token_name = InterpolatedString.create(
122            self.access_token_name, parameters=parameters
123        )
124        self.expires_in_name = InterpolatedString.create(
125            self.expires_in_name, parameters=parameters
126        )
127        self.grant_type_name = InterpolatedString.create(
128            self.grant_type_name, parameters=parameters
129        )
130        self.grant_type = InterpolatedString.create(
131            "urn:ietf:params:oauth:grant-type:jwt-bearer"
132            if self.use_profile_assertion
133            else self.grant_type,
134            parameters=parameters,
135        )
136        self._refresh_request_body = InterpolatedMapping(
137            self.refresh_request_body or {}, parameters=parameters
138        )
139        self._refresh_request_headers = InterpolatedMapping(
140            self.refresh_request_headers or {}, parameters=parameters
141        )
142        self._send_refresh_request_as_query_params = self.send_refresh_request_as_query_params
143        try:
144            if (
145                isinstance(self.token_expiry_date, (int, str))
146                and str(self.token_expiry_date).isdigit()
147            ):
148                self._token_expiry_date = ab_datetime_parse(self.token_expiry_date)
149            else:
150                self._token_expiry_date = (
151                    ab_datetime_parse(
152                        InterpolatedString.create(
153                            self.token_expiry_date, parameters=parameters
154                        ).eval(self.config)
155                    )
156                    if self.token_expiry_date
157                    else ab_datetime_now() - timedelta(days=1)
158                )
159        except ValueError as e:
160            raise ValueError(f"Invalid token expiry date format: {e}")
161        self.use_profile_assertion = (
162            InterpolatedBoolean(self.use_profile_assertion, parameters=parameters)
163            if isinstance(self.use_profile_assertion, str)
164            else self.use_profile_assertion
165        )
166        self.assertion_name = "assertion"
167
168        if self.access_token_value is not None:
169            self._access_token_value = InterpolatedString.create(
170                self.access_token_value, parameters=parameters
171            ).eval(self.config)
172        else:
173            self._access_token_value = None
174
175        self._access_token: Optional[str] = (
176            self._access_token_value if self.access_token_value else None
177        )
178
179        if not self.use_profile_assertion and any(
180            client_creds is None for client_creds in [self.client_id, self.client_secret]
181        ):
182            raise ValueError(
183                "OAuthAuthenticator configuration error: Both 'client_id' and 'client_secret' are required for the "
184                "basic OAuth flow."
185            )
186        if self.profile_assertion is None and self.use_profile_assertion:
187            raise ValueError(
188                "OAuthAuthenticator configuration error: 'profile_assertion' is required when using the profile assertion flow."
189            )
190        if self.get_grant_type() == "refresh_token" and self._refresh_token is None:
191            raise ValueError(
192                "OAuthAuthenticator configuration error: A 'refresh_token' is required when the 'grant_type' is set to 'refresh_token'."
193            )
194
195    def get_token_refresh_endpoint(self) -> Optional[str]:
196        if self._token_refresh_endpoint is not None:
197            refresh_token_endpoint: str = self._token_refresh_endpoint.eval(self.config)
198            if not refresh_token_endpoint:
199                raise ValueError(
200                    "OAuthAuthenticator was unable to evaluate token_refresh_endpoint parameter"
201                )
202            return refresh_token_endpoint
203        return None
204
205    def get_client_id_name(self) -> str:
206        return self._client_id_name.eval(self.config)  # type: ignore # eval returns a string in this context
207
208    def get_client_id(self) -> str:
209        client_id = self._client_id.eval(self.config) if self._client_id else self._client_id
210        if not client_id:
211            raise ValueError("OAuthAuthenticator was unable to evaluate client_id parameter")
212        return client_id  # type: ignore # value will be returned as a string, or an error will be raised
213
214    def get_client_secret_name(self) -> str:
215        return self._client_secret_name.eval(self.config)  # type: ignore # eval returns a string in this context
216
217    def get_client_secret(self) -> str:
218        client_secret = (
219            self._client_secret.eval(self.config) if self._client_secret else self._client_secret
220        )
221        if not client_secret:
222            # We've seen some APIs allowing empty client_secret so we will only log here
223            logger.warning(
224                "OAuthAuthenticator was unable to evaluate client_secret parameter hence it'll be empty"
225            )
226        return client_secret  # type: ignore # value will be returned as a string, which might be empty
227
228    def get_refresh_token_name(self) -> str:
229        return self._refresh_token_name.eval(self.config)  # type: ignore # eval returns a string in this context
230
231    def get_refresh_token(self) -> Optional[str]:
232        return None if self._refresh_token is None else str(self._refresh_token.eval(self.config))
233
234    def get_scopes(self) -> List[str]:
235        return self.scopes or []
236
237    def get_access_token_name(self) -> str:
238        return self.access_token_name.eval(self.config)  # type: ignore # eval returns a string in this context
239
240    def get_expires_in_name(self) -> str:
241        return self.expires_in_name.eval(self.config)  # type: ignore # eval returns a string in this context
242
243    def get_grant_type_name(self) -> str:
244        return self.grant_type_name.eval(self.config)  # type: ignore # eval returns a string in this context
245
246    def get_grant_type(self) -> str:
247        return self.grant_type.eval(self.config)  # type: ignore # eval returns a string in this context
248
249    def get_refresh_request_body(self) -> Mapping[str, Any]:
250        return self._refresh_request_body.eval(self.config)
251
252    def get_refresh_request_headers(self) -> Mapping[str, Any]:
253        return self._refresh_request_headers.eval(self.config)
254
255    def should_send_refresh_request_as_query_params(self) -> bool:
256        return self._send_refresh_request_as_query_params
257
258    def get_token_expiry_date(self) -> AirbyteDateTime:
259        if not self._has_access_token_been_initialized():
260            return AirbyteDateTime.from_datetime(datetime.min)
261        return self._token_expiry_date  # type: ignore # _token_expiry_date is an AirbyteDateTime. It is never None despite what mypy thinks
262
263    def _has_access_token_been_initialized(self) -> bool:
264        return self._access_token is not None
265
266    def set_token_expiry_date(self, value: AirbyteDateTime) -> None:
267        # Overrides of `refresh_access_token` written before 6.45.5 return the raw `expires_in`.
268        if not isinstance(value, datetime):
269            with suppress(ValueError, OverflowError):
270                value = self._parse_token_expiration_date(value)
271        self._token_expiry_date = value
272
273    def get_assertion_name(self) -> str:
274        return self.assertion_name
275
276    def get_assertion(self) -> str:
277        if self.profile_assertion is None:
278            raise ValueError("profile_assertion is not set")
279        return self.profile_assertion.token
280
281    def build_refresh_request_body(self) -> Mapping[str, Any]:
282        """
283        Returns the request body to set on the refresh request
284
285        Override to define additional parameters
286        """
287        if self.use_profile_assertion:
288            return {
289                self.get_grant_type_name(): self.get_grant_type(),
290                self.get_assertion_name(): self.get_assertion(),
291            }
292        return super().build_refresh_request_body()
293
294    @property
295    def access_token(self) -> str:
296        if self._access_token is None:
297            raise ValueError("access_token is not set")
298        return self._access_token
299
300    @access_token.setter
301    def access_token(self, value: str) -> None:
302        self._access_token = value
303
304    @property
305    def _message_repository(self) -> MessageRepository:
306        """
307        Overriding AbstractOauth2Authenticator._message_repository to allow for HTTP request logs
308        """
309        return self.message_repository

Generates OAuth2.0 access tokens from an OAuth2.0 refresh token and client credentials based on a declarative connector configuration file. Credentials can be defined explicitly or via interpolation at runtime. The generated access token is attached to each request via the Authorization header.

Attributes:
  • token_refresh_endpoint (Union[InterpolatedString, str]): The endpoint to refresh the access token
  • client_id (Union[InterpolatedString, str]): The client id
  • client_secret (Union[InterpolatedString, str]): Client secret (can be empty for APIs that support this)
  • refresh_token (Union[InterpolatedString, str]): The token used to refresh the access token
  • access_token_name (Union[InterpolatedString, str]): THe field to extract access token from in the response
  • expires_in_name (Union[InterpolatedString, str]): The field to extract expires_in from in the response
  • config (Mapping[str, Any]): The user-provided configuration as specified by the source's spec
  • scopes (Optional[List[str]]): The scopes to request
  • token_expiry_date (Optional[Union[InterpolatedString, str]]): The access token expiration date
  • token_expiry_date_format str: format of the datetime; provide it if expires_in is returned in datetime instead of seconds
  • token_expiry_is_time_of_expiration bool: set True it if expires_in is returned as time of expiration instead of the number seconds until expiration
  • refresh_request_body (Optional[Mapping[str, Any]]): The request body to send in the refresh request
  • refresh_request_headers (Optional[Mapping[str, Any]]): The request headers to send in the refresh request
  • send_refresh_request_as_query_params (bool): When True, the standard refresh args (grant_type, refresh_token, client credentials when not in an Authorization header, scopes, plus any refresh_request_body extras) are sent on the URL query string instead of in the request body, and the body is emitted empty. Use this for OAuth providers like Gong that document their refresh endpoint with refresh args on the URL query string. Defaults to False.
  • grant_type: The grant_type to request for access_token. If set to refresh_token, the refresh_token parameter has to be provided
  • message_repository (MessageRepository): the message repository used to emit logs on HTTP requests
  • refresh_token_error_status_codes (Tuple[int, ...]): Status codes to identify refresh token errors in response
  • refresh_token_error_key (str): Key to identify refresh token error in response
  • refresh_token_error_values (Tuple[str, ...]): List of values to check for exception during token refresh process
DeclarativeOauth2Authenticator( config: Mapping[str, Any], parameters: dataclasses.InitVar[typing.Mapping[str, typing.Any]], client_id: Union[InterpolatedString, str, NoneType] = None, client_secret: Union[InterpolatedString, str, NoneType] = None, token_refresh_endpoint: Union[InterpolatedString, str, NoneType] = None, refresh_token: Union[InterpolatedString, str, NoneType] = None, scopes: Optional[List[str]] = None, token_expiry_date: Union[InterpolatedString, str, NoneType] = None, token_expiry_date_format: Optional[str] = None, token_expiry_is_time_of_expiration: bool = False, access_token_name: Union[InterpolatedString, str] = 'access_token', access_token_value: Union[InterpolatedString, str, NoneType] = None, client_id_name: Union[InterpolatedString, str] = 'client_id', client_secret_name: Union[InterpolatedString, str] = 'client_secret', expires_in_name: Union[InterpolatedString, str] = 'expires_in', refresh_token_name: Union[InterpolatedString, str] = 'refresh_token', refresh_request_body: Optional[Mapping[str, Any]] = None, refresh_request_headers: Optional[Mapping[str, Any]] = None, send_refresh_request_as_query_params: bool = False, grant_type_name: Union[InterpolatedString, str] = 'grant_type', grant_type: Union[InterpolatedString, str] = 'refresh_token', message_repository: MessageRepository = <airbyte_cdk.sources.message.NoopMessageRepository object>, profile_assertion: Optional[DeclarativeAuthenticator] = None, use_profile_assertion: Union[InterpolatedBoolean, str, bool, NoneType] = False, refresh_token_error_status_codes: Tuple[int, ...] = (), refresh_token_error_key: str = '', refresh_token_error_values: Tuple[str, ...] = ())

If all of refresh_token_error_status_codes, refresh_token_error_key, and refresh_token_error_values are set, then http errors with such params will be wrapped in AirbyteTracedException.

config: Mapping[str, Any]
parameters: dataclasses.InitVar[typing.Mapping[str, typing.Any]]
client_id: Union[InterpolatedString, str, NoneType] = None
client_secret: Union[InterpolatedString, str, NoneType] = None
token_refresh_endpoint: Union[InterpolatedString, str, NoneType] = None
refresh_token: Union[InterpolatedString, str, NoneType] = None
scopes: Optional[List[str]] = None
token_expiry_date: Union[InterpolatedString, str, NoneType] = None
token_expiry_date_format: Optional[str] = None

Format of the datetime; exists it if expires_in is returned as the expiration datetime instead of seconds until it expires

token_expiry_is_time_of_expiration: bool = False

Indicates that the Token Expiry returns the date until which the token will be valid, not the amount of time it will be valid.

access_token_name: Union[InterpolatedString, str] = 'access_token'
access_token_value: Union[InterpolatedString, str, NoneType] = None
client_id_name: Union[InterpolatedString, str] = 'client_id'
client_secret_name: Union[InterpolatedString, str] = 'client_secret'
expires_in_name: Union[InterpolatedString, str] = 'expires_in'
refresh_token_name: Union[InterpolatedString, str] = 'refresh_token'
refresh_request_body: Optional[Mapping[str, Any]] = None
refresh_request_headers: Optional[Mapping[str, Any]] = None
send_refresh_request_as_query_params: bool = False
grant_type_name: Union[InterpolatedString, str] = 'grant_type'
grant_type: Union[InterpolatedString, str] = 'refresh_token'
profile_assertion: Optional[DeclarativeAuthenticator] = None
use_profile_assertion: Union[InterpolatedBoolean, str, bool, NoneType] = False
refresh_token_error_status_codes: Tuple[int, ...] = ()
refresh_token_error_key: str = ''
refresh_token_error_values: Tuple[str, ...] = ()
def get_token_refresh_endpoint(self) -> Optional[str]:
195    def get_token_refresh_endpoint(self) -> Optional[str]:
196        if self._token_refresh_endpoint is not None:
197            refresh_token_endpoint: str = self._token_refresh_endpoint.eval(self.config)
198            if not refresh_token_endpoint:
199                raise ValueError(
200                    "OAuthAuthenticator was unable to evaluate token_refresh_endpoint parameter"
201                )
202            return refresh_token_endpoint
203        return None

Returns the endpoint to refresh the access token

def get_client_id_name(self) -> str:
205    def get_client_id_name(self) -> str:
206        return self._client_id_name.eval(self.config)  # type: ignore # eval returns a string in this context

The client id name to authenticate

def get_client_id(self) -> str:
208    def get_client_id(self) -> str:
209        client_id = self._client_id.eval(self.config) if self._client_id else self._client_id
210        if not client_id:
211            raise ValueError("OAuthAuthenticator was unable to evaluate client_id parameter")
212        return client_id  # type: ignore # value will be returned as a string, or an error will be raised

The client id to authenticate

def get_client_secret_name(self) -> str:
214    def get_client_secret_name(self) -> str:
215        return self._client_secret_name.eval(self.config)  # type: ignore # eval returns a string in this context

The client secret name to authenticate

def get_client_secret(self) -> str:
217    def get_client_secret(self) -> str:
218        client_secret = (
219            self._client_secret.eval(self.config) if self._client_secret else self._client_secret
220        )
221        if not client_secret:
222            # We've seen some APIs allowing empty client_secret so we will only log here
223            logger.warning(
224                "OAuthAuthenticator was unable to evaluate client_secret parameter hence it'll be empty"
225            )
226        return client_secret  # type: ignore # value will be returned as a string, which might be empty

The client secret to authenticate

def get_refresh_token_name(self) -> str:
228    def get_refresh_token_name(self) -> str:
229        return self._refresh_token_name.eval(self.config)  # type: ignore # eval returns a string in this context

The refresh token name to authenticate

def get_refresh_token(self) -> Optional[str]:
231    def get_refresh_token(self) -> Optional[str]:
232        return None if self._refresh_token is None else str(self._refresh_token.eval(self.config))

The token used to refresh the access token when it expires

def get_scopes(self) -> List[str]:
234    def get_scopes(self) -> List[str]:
235        return self.scopes or []

List of requested scopes

def get_access_token_name(self) -> str:
237    def get_access_token_name(self) -> str:
238        return self.access_token_name.eval(self.config)  # type: ignore # eval returns a string in this context

Field to extract access token from in the response

def get_expires_in_name(self) -> str:
240    def get_expires_in_name(self) -> str:
241        return self.expires_in_name.eval(self.config)  # type: ignore # eval returns a string in this context

Returns the expires_in field name

def get_grant_type_name(self) -> str:
243    def get_grant_type_name(self) -> str:
244        return self.grant_type_name.eval(self.config)  # type: ignore # eval returns a string in this context

Returns grant_type specified name for requesting access_token

def get_grant_type(self) -> str:
246    def get_grant_type(self) -> str:
247        return self.grant_type.eval(self.config)  # type: ignore # eval returns a string in this context

Returns grant_type specified for requesting access_token

def get_refresh_request_body(self) -> Mapping[str, Any]:
249    def get_refresh_request_body(self) -> Mapping[str, Any]:
250        return self._refresh_request_body.eval(self.config)

Returns the request body to set on the refresh request

def get_refresh_request_headers(self) -> Mapping[str, Any]:
252    def get_refresh_request_headers(self) -> Mapping[str, Any]:
253        return self._refresh_request_headers.eval(self.config)

Returns the request headers to set on the refresh request

def should_send_refresh_request_as_query_params(self) -> bool:
255    def should_send_refresh_request_as_query_params(self) -> bool:
256        return self._send_refresh_request_as_query_params

Returns True if the standard refresh args should be sent on the URL query string instead of in the request body.

Defaults to False so existing authenticators retain their previous behavior (params in body, no query params on the refresh URL). Subclasses can override this to opt into the URL-query-string shape required by OAuth providers like Gong.

def get_token_expiry_date(self) -> airbyte_cdk.utils.datetime_helpers.AirbyteDateTime:
258    def get_token_expiry_date(self) -> AirbyteDateTime:
259        if not self._has_access_token_been_initialized():
260            return AirbyteDateTime.from_datetime(datetime.min)
261        return self._token_expiry_date  # type: ignore # _token_expiry_date is an AirbyteDateTime. It is never None despite what mypy thinks

Expiration date of the access token

def set_token_expiry_date(self, value: airbyte_cdk.utils.datetime_helpers.AirbyteDateTime) -> None:
266    def set_token_expiry_date(self, value: AirbyteDateTime) -> None:
267        # Overrides of `refresh_access_token` written before 6.45.5 return the raw `expires_in`.
268        if not isinstance(value, datetime):
269            with suppress(ValueError, OverflowError):
270                value = self._parse_token_expiration_date(value)
271        self._token_expiry_date = value

Setter for access token expiration date

def get_assertion_name(self) -> str:
273    def get_assertion_name(self) -> str:
274        return self.assertion_name
def get_assertion(self) -> str:
276    def get_assertion(self) -> str:
277        if self.profile_assertion is None:
278            raise ValueError("profile_assertion is not set")
279        return self.profile_assertion.token
def build_refresh_request_body(self) -> Mapping[str, Any]:
281    def build_refresh_request_body(self) -> Mapping[str, Any]:
282        """
283        Returns the request body to set on the refresh request
284
285        Override to define additional parameters
286        """
287        if self.use_profile_assertion:
288            return {
289                self.get_grant_type_name(): self.get_grant_type(),
290                self.get_assertion_name(): self.get_assertion(),
291            }
292        return super().build_refresh_request_body()

Returns the request body to set on the refresh request

Override to define additional parameters

access_token: str
294    @property
295    def access_token(self) -> str:
296        if self._access_token is None:
297            raise ValueError("access_token is not set")
298        return self._access_token

Returns the access token

@dataclass
class DeclarativeSingleUseRefreshTokenOauth2Authenticator(airbyte_cdk.SingleUseRefreshTokenOauth2Authenticator, airbyte_cdk.DeclarativeAuthenticator):
312@dataclass
313class DeclarativeSingleUseRefreshTokenOauth2Authenticator(
314    SingleUseRefreshTokenOauth2Authenticator, DeclarativeAuthenticator
315):
316    """
317    Declarative version of SingleUseRefreshTokenOauth2Authenticator which can be used in declarative connectors.
318    """
319
320    def __init__(self, *args: Any, **kwargs: Any) -> None:
321        super().__init__(*args, **kwargs)

Declarative version of SingleUseRefreshTokenOauth2Authenticator which can be used in declarative connectors.

DeclarativeSingleUseRefreshTokenOauth2Authenticator(*args: Any, **kwargs: Any)
320    def __init__(self, *args: Any, **kwargs: Any) -> None:
321        super().__init__(*args, **kwargs)
Arguments:
  • connector_config (Mapping[str, Any]): The full connector configuration
  • token_refresh_endpoint (str): Full URL to the token refresh endpoint
  • scopes (List[str], optional): List of OAuth scopes to pass in the refresh token request body. Defaults to None.
  • access_token_name (str, optional): Name of the access token field, used to parse the refresh token response. Defaults to "access_token".
  • expires_in_name (str, optional): Name of the name of the field that characterizes when the current access token will expire, used to parse the refresh token response. Defaults to "expires_in".
  • refresh_token_name (str, optional): Name of the name of the refresh token field, used to parse the refresh token response. Defaults to "refresh_token".
  • refresh_request_body (Mapping[str, Any], optional): Custom key value pair that will be added to the refresh token request body. Defaults to None.
  • refresh_request_headers (Mapping[str, Any], optional): Custom key value pair that will be added to the refresh token request headers. Defaults to None.
  • send_refresh_request_as_query_params (bool, optional): When True, the standard refresh args (grant_type, refresh_token, client credentials when not in an Authorization header, scopes, plus any refresh_request_body extras) are sent on the URL query string and the request body is emitted empty. Use this for OAuth providers like Gong that document their refresh endpoint with refresh args on the URL query string. Defaults to False.
  • grant_type (str, optional): OAuth grant type. Defaults to "refresh_token".
  • client_id (Optional[str]): The client id to authenticate. If not specified, defaults to credentials.client_id in the config object.
  • client_secret (Optional[str]): The client secret to authenticate. If not specified, defaults to credentials.client_secret in the config object.
  • access_token_config_path (Sequence[str]): Dpath to the access_token field in the connector configuration. Defaults to ("credentials", "access_token").
  • refresh_token_config_path (Sequence[str]): Dpath to the refresh_token field in the connector configuration. Defaults to ("credentials", "refresh_token").
  • token_expiry_date_config_path (Sequence[str]): Dpath to the token_expiry_date field in the connector configuration. Defaults to ("credentials", "token_expiry_date").
  • token_expiry_date_format (Optional[str]): Date format of the token expiry date field (set by expires_in_name). If specified the token expiry date is interpreted as time of expiration, as with token_expiry_is_time_of_expiration, otherwise as number of seconds until expiration.
  • token_expiry_is_time_of_expiration bool: set True it if expires_in is returned as time of expiration instead of the number seconds until expiration
  • message_repository (MessageRepository): The message repository used to emit logs on HTTP requests and control message on config update.
  • emit_control_message_to_message_repository (bool): Whether the message repository additionally receives the CONNECTOR_CONFIG message. This is only for in-process consumers such as the Connector Builder, because messages in the repository are otherwise printed to stdout by the entrypoint and would duplicate the direct stdout emission. Defaults to False.
@deprecated('DeclarativeStream has been deprecated in favor of the concurrent DefaultStream')
@dataclass
class DeclarativeStream(airbyte_cdk.Stream):
 33@deprecated("DeclarativeStream has been deprecated in favor of the concurrent DefaultStream")
 34@dataclass
 35class DeclarativeStream(Stream):
 36    """
 37    DeclarativeStream is a Stream that delegates most of its logic to its schema_load and retriever
 38
 39    Attributes:
 40        name (str): stream name
 41        primary_key (Optional[Union[str, List[str], List[List[str]]]]): the primary key of the stream
 42        schema_loader (SchemaLoader): The schema loader
 43        retriever (Retriever): The retriever
 44        config (Config): The user-provided configuration as specified by the source's spec
 45        stream_cursor_field (Optional[Union[InterpolatedString, str]]): The cursor field
 46        stream. Transformations are applied in the order in which they are defined.
 47    """
 48
 49    retriever: Retriever
 50    config: Config
 51    parameters: InitVar[Mapping[str, Any]]
 52    name: str
 53    primary_key: Optional[Union[str, List[str], List[List[str]]]]
 54    state_migrations: List[StateMigration] = field(repr=True, default_factory=list)
 55    schema_loader: Optional[SchemaLoader] = None
 56    _name: str = field(init=False, repr=False, default="")
 57    _primary_key: str = field(init=False, repr=False, default="")
 58    stream_cursor_field: Optional[Union[InterpolatedString, str]] = None
 59
 60    def __post_init__(self, parameters: Mapping[str, Any]) -> None:
 61        self._stream_cursor_field = (
 62            InterpolatedString.create(self.stream_cursor_field, parameters=parameters)
 63            if isinstance(self.stream_cursor_field, str)
 64            else self.stream_cursor_field
 65        )
 66        self._schema_loader = (
 67            self.schema_loader
 68            if self.schema_loader
 69            else DefaultSchemaLoader(config=self.config, parameters=parameters)
 70        )
 71
 72    @property  # type: ignore
 73    def primary_key(self) -> Optional[Union[str, List[str], List[List[str]]]]:
 74        return self._primary_key
 75
 76    @primary_key.setter
 77    def primary_key(self, value: str) -> None:
 78        if not isinstance(value, property):
 79            self._primary_key = value
 80
 81    @property
 82    def exit_on_rate_limit(self) -> bool:
 83        if isinstance(self.retriever, AsyncRetriever):
 84            return self.retriever.exit_on_rate_limit
 85
 86        return self.retriever.requester.exit_on_rate_limit  # type: ignore # abstract Retriever class has not requester attribute
 87
 88    @exit_on_rate_limit.setter
 89    def exit_on_rate_limit(self, value: bool) -> None:
 90        if isinstance(self.retriever, AsyncRetriever):
 91            self.retriever.exit_on_rate_limit = value
 92        else:
 93            self.retriever.requester.exit_on_rate_limit = value  # type: ignore[attr-defined]
 94
 95    @property  # type: ignore
 96    def name(self) -> str:
 97        """
 98        :return: Stream name. By default this is the implementing class name, but it can be overridden as needed.
 99        """
100        return self._name
101
102    @name.setter
103    def name(self, value: str) -> None:
104        if not isinstance(value, property):
105            self._name = value
106
107    @property
108    def state(self) -> MutableMapping[str, Any]:
109        return self.retriever.state  # type: ignore
110
111    @state.setter
112    def state(self, value: MutableMapping[str, Any]) -> None:
113        """State setter, accept state serialized by state getter."""
114        state: Mapping[str, Any] = value
115        if self.state_migrations:
116            for migration in self.state_migrations:
117                if migration.should_migrate(state):
118                    state = migration.migrate(state)
119        self.retriever.state = state
120
121    def get_updated_state(
122        self, current_stream_state: MutableMapping[str, Any], latest_record: Mapping[str, Any]
123    ) -> MutableMapping[str, Any]:
124        return self.state
125
126    @property
127    def cursor_field(self) -> Union[str, List[str]]:
128        """
129        Override to return the default cursor field used by this stream e.g: an API entity might always use created_at as the cursor field.
130        :return: The name of the field used as a cursor. If the cursor is nested, return an array consisting of the path to the cursor.
131        """
132        cursor = self._stream_cursor_field.eval(self.config)  # type: ignore # _stream_cursor_field is always cast to interpolated string
133        return cursor if cursor else []
134
135    @property
136    def is_resumable(self) -> bool:
137        # Declarative sources always implement state getter/setter, but whether it supports checkpointing is based on
138        # if the retriever has a cursor defined.
139        return self.retriever.cursor is not None if hasattr(self.retriever, "cursor") else False
140
141    def read_records(
142        self,
143        sync_mode: SyncMode,
144        cursor_field: Optional[List[str]] = None,
145        stream_slice: Optional[Mapping[str, Any]] = None,
146        stream_state: Optional[Mapping[str, Any]] = None,
147    ) -> Iterable[Mapping[str, Any]]:
148        """
149        :param: stream_state We knowingly avoid using stream_state as we want cursors to manage their own state.
150        """
151        if stream_slice is None or (
152            not isinstance(stream_slice, StreamSlice) and stream_slice == {}
153        ):
154            # As the parameter is Optional, many would just call `read_records(sync_mode)` during testing without specifying the field
155            # As part of the declarative model without custom components, this should never happen as the CDK would wire up a
156            # SinglePartitionRouter that would create this StreamSlice properly
157            # As part of the declarative model with custom components, a user that would return a `None` slice would now have the default
158            # empty slice which seems to make sense.
159            stream_slice = StreamSlice(partition={}, cursor_slice={})
160        if not isinstance(stream_slice, StreamSlice):
161            raise ValueError(
162                f"DeclarativeStream does not support stream_slices that are not StreamSlice. Got {stream_slice}"
163            )
164        yield from self.retriever.read_records(self.get_json_schema(), stream_slice)  # type: ignore # records are of the correct type
165
166    def get_json_schema(self) -> Mapping[str, Any]:  # type: ignore
167        """
168        :return: A dict of the JSON schema representing this stream.
169
170        The default implementation of this method looks for a JSONSchema file with the same name as this stream's "name" property.
171        Override as needed.
172        """
173        return self._schema_loader.get_json_schema()
174
175    def stream_slices(
176        self,
177        *,
178        sync_mode: SyncMode,
179        cursor_field: Optional[List[str]] = None,
180        stream_state: Optional[Mapping[str, Any]] = None,
181    ) -> Iterable[Optional[StreamSlice]]:
182        """
183        Override to define the slices for this stream. See the stream slicing section of the docs for more information.
184
185        :param sync_mode:
186        :param cursor_field:
187        :param stream_state: we knowingly avoid using stream_state as we want cursors to manage their own state
188        :return:
189        """
190        return self.retriever.stream_slices()
191
192    @property
193    def state_checkpoint_interval(self) -> Optional[int]:
194        """
195        We explicitly disable checkpointing here. There are a couple reasons for that and not all are documented here but:
196        * In the case where records are not ordered, the granularity of what is ordered is the slice. Therefore, we will only update the
197            cursor value once at the end of every slice.
198        * Updating the state once every record would generate issues for data feed stop conditions or semi-incremental syncs where the
199            important state is the one at the beginning of the slice
200        """
201        return None
202
203    def get_cursor(self) -> Optional[Cursor]:
204        return None
205
206    def _get_checkpoint_reader(
207        self,
208        logger: logging.Logger,
209        cursor_field: Optional[List[str]],
210        sync_mode: SyncMode,
211        stream_state: MutableMapping[str, Any],
212    ) -> CheckpointReader:
213        """
214        This method is overridden to prevent issues with stream slice classification for incremental streams that have parent streams.
215
216        The classification logic, when used with `itertools.tee`, creates a copy of the stream slices. When `stream_slices` is called
217        the second time, the parent records generated during the classification phase are lost. This occurs because `itertools.tee`
218        only buffers the results, meaning the logic in `simple_retriever` that observes and updates the cursor isn't executed again.
219
220        By overriding this method, we ensure that the stream slices are processed correctly and parent records are not lost,
221        allowing the cursor to function as expected.
222        """
223        mappings_or_slices = self.stream_slices(
224            cursor_field=cursor_field,
225            sync_mode=sync_mode,  # todo: change this interface to no longer rely on sync_mode for behavior
226            stream_state=stream_state,
227        )
228
229        cursor = self.get_cursor()
230        checkpoint_mode = self._checkpoint_mode
231
232        if isinstance(
233            cursor, (GlobalSubstreamCursor, PerPartitionCursor, PerPartitionWithGlobalCursor)
234        ):
235            self.has_multiple_slices = True
236            return CursorBasedCheckpointReader(
237                stream_slices=mappings_or_slices,
238                cursor=cursor,
239                read_state_from_cursor=checkpoint_mode == CheckpointMode.RESUMABLE_FULL_REFRESH,
240            )
241
242        return super()._get_checkpoint_reader(logger, cursor_field, sync_mode, stream_state)

DeclarativeStream is a Stream that delegates most of its logic to its schema_load and retriever

Attributes:
  • name (str): stream name
  • primary_key (Optional[Union[str, List[str], List[List[str]]]]): the primary key of the stream
  • schema_loader (SchemaLoader): The schema loader
  • retriever (Retriever): The retriever
  • config (Config): The user-provided configuration as specified by the source's spec
  • stream_cursor_field (Optional[Union[InterpolatedString, str]]): The cursor field
  • stream. Transformations are applied in the order in which they are defined.
DeclarativeStream( retriever: airbyte_cdk.sources.declarative.retrievers.Retriever, config: Mapping[str, Any], parameters: dataclasses.InitVar[typing.Mapping[str, typing.Any]], name: str = <property object>, primary_key: Union[str, List[str], List[List[str]], NoneType] = <property object>, state_migrations: List[airbyte_cdk.sources.declarative.migrations.state_migration.StateMigration] = <factory>, schema_loader: Optional[airbyte_cdk.sources.declarative.schema.SchemaLoader] = None, stream_cursor_field: Union[InterpolatedString, str, NoneType] = None)
config: Mapping[str, Any]
parameters: dataclasses.InitVar[typing.Mapping[str, typing.Any]]
name: str
 95    @property  # type: ignore
 96    def name(self) -> str:
 97        """
 98        :return: Stream name. By default this is the implementing class name, but it can be overridden as needed.
 99        """
100        return self._name
Returns

Stream name. By default this is the implementing class name, but it can be overridden as needed.

primary_key: Union[str, List[str], List[List[str]], NoneType]
72    @property  # type: ignore
73    def primary_key(self) -> Optional[Union[str, List[str], List[List[str]]]]:
74        return self._primary_key
Returns

string if single primary key, list of strings if composite primary key, list of list of strings if composite primary key consisting of nested fields. If the stream has no primary keys, return None.

schema_loader: Optional[airbyte_cdk.sources.declarative.schema.SchemaLoader] = None
stream_cursor_field: Union[InterpolatedString, str, NoneType] = None
exit_on_rate_limit: bool
81    @property
82    def exit_on_rate_limit(self) -> bool:
83        if isinstance(self.retriever, AsyncRetriever):
84            return self.retriever.exit_on_rate_limit
85
86        return self.retriever.requester.exit_on_rate_limit  # type: ignore # abstract Retriever class has not requester attribute

Exit on rate limit getter, should return bool value. False if the stream will retry endlessly when rate limited.

state: MutableMapping[str, Any]
107    @property
108    def state(self) -> MutableMapping[str, Any]:
109        return self.retriever.state  # type: ignore

State setter, accept state serialized by state getter.

def get_updated_state( self, current_stream_state: MutableMapping[str, Any], latest_record: Mapping[str, Any]) -> MutableMapping[str, Any]:
121    def get_updated_state(
122        self, current_stream_state: MutableMapping[str, Any], latest_record: Mapping[str, Any]
123    ) -> MutableMapping[str, Any]:
124        return self.state

DEPRECATED. Please use explicit state property instead, see IncrementalMixin docs.

Override to extract state from the latest record. Needed to implement incremental sync.

Inspects the latest record extracted from the data source and the current state object and return an updated state object.

For example: if the state object is based on created_at timestamp, and the current state is {'created_at': 10}, and the latest_record is {'name': 'octavia', 'created_at': 20 } then this method would return {'created_at': 20} to indicate state should be updated to this object.

Parameters
  • current_stream_state: The stream's current state object
  • latest_record: The latest record extracted from the stream
Returns

An updated state object

cursor_field: Union[str, List[str]]
126    @property
127    def cursor_field(self) -> Union[str, List[str]]:
128        """
129        Override to return the default cursor field used by this stream e.g: an API entity might always use created_at as the cursor field.
130        :return: The name of the field used as a cursor. If the cursor is nested, return an array consisting of the path to the cursor.
131        """
132        cursor = self._stream_cursor_field.eval(self.config)  # type: ignore # _stream_cursor_field is always cast to interpolated string
133        return cursor if cursor else []

Override to return the default cursor field used by this stream e.g: an API entity might always use created_at as the cursor field.

Returns

The name of the field used as a cursor. If the cursor is nested, return an array consisting of the path to the cursor.

is_resumable: bool
135    @property
136    def is_resumable(self) -> bool:
137        # Declarative sources always implement state getter/setter, but whether it supports checkpointing is based on
138        # if the retriever has a cursor defined.
139        return self.retriever.cursor is not None if hasattr(self.retriever, "cursor") else False
Returns

True if this stream allows the checkpointing of sync progress and can resume from it on subsequent attempts. This differs from supports_incremental because certain kinds of streams like those supporting resumable full refresh can checkpoint progress in between attempts for improved fault tolerance. However, they will start from the beginning on the next sync job.

def read_records( self, sync_mode: airbyte_protocol_dataclasses.models.airbyte_protocol.SyncMode, cursor_field: Optional[List[str]] = None, stream_slice: Optional[Mapping[str, Any]] = None, stream_state: Optional[Mapping[str, Any]] = None) -> Iterable[Mapping[str, Any]]:
141    def read_records(
142        self,
143        sync_mode: SyncMode,
144        cursor_field: Optional[List[str]] = None,
145        stream_slice: Optional[Mapping[str, Any]] = None,
146        stream_state: Optional[Mapping[str, Any]] = None,
147    ) -> Iterable[Mapping[str, Any]]:
148        """
149        :param: stream_state We knowingly avoid using stream_state as we want cursors to manage their own state.
150        """
151        if stream_slice is None or (
152            not isinstance(stream_slice, StreamSlice) and stream_slice == {}
153        ):
154            # As the parameter is Optional, many would just call `read_records(sync_mode)` during testing without specifying the field
155            # As part of the declarative model without custom components, this should never happen as the CDK would wire up a
156            # SinglePartitionRouter that would create this StreamSlice properly
157            # As part of the declarative model with custom components, a user that would return a `None` slice would now have the default
158            # empty slice which seems to make sense.
159            stream_slice = StreamSlice(partition={}, cursor_slice={})
160        if not isinstance(stream_slice, StreamSlice):
161            raise ValueError(
162                f"DeclarativeStream does not support stream_slices that are not StreamSlice. Got {stream_slice}"
163            )
164        yield from self.retriever.read_records(self.get_json_schema(), stream_slice)  # type: ignore # records are of the correct type
Parameters
  • stream_state We knowingly avoid using stream_state as we want cursors to manage their own state.
def get_json_schema(self) -> Mapping[str, Any]:
166    def get_json_schema(self) -> Mapping[str, Any]:  # type: ignore
167        """
168        :return: A dict of the JSON schema representing this stream.
169
170        The default implementation of this method looks for a JSONSchema file with the same name as this stream's "name" property.
171        Override as needed.
172        """
173        return self._schema_loader.get_json_schema()
Returns

A dict of the JSON schema representing this stream.

The default implementation of this method looks for a JSONSchema file with the same name as this stream's "name" property. Override as needed.

def stream_slices( self, *, sync_mode: airbyte_protocol_dataclasses.models.airbyte_protocol.SyncMode, cursor_field: Optional[List[str]] = None, stream_state: Optional[Mapping[str, Any]] = None) -> Iterable[Optional[StreamSlice]]:
175    def stream_slices(
176        self,
177        *,
178        sync_mode: SyncMode,
179        cursor_field: Optional[List[str]] = None,
180        stream_state: Optional[Mapping[str, Any]] = None,
181    ) -> Iterable[Optional[StreamSlice]]:
182        """
183        Override to define the slices for this stream. See the stream slicing section of the docs for more information.
184
185        :param sync_mode:
186        :param cursor_field:
187        :param stream_state: we knowingly avoid using stream_state as we want cursors to manage their own state
188        :return:
189        """
190        return self.retriever.stream_slices()

Override to define the slices for this stream. See the stream slicing section of the docs for more information.

Parameters
  • sync_mode:
  • cursor_field:
  • stream_state: we knowingly avoid using stream_state as we want cursors to manage their own state
Returns
state_checkpoint_interval: Optional[int]
192    @property
193    def state_checkpoint_interval(self) -> Optional[int]:
194        """
195        We explicitly disable checkpointing here. There are a couple reasons for that and not all are documented here but:
196        * In the case where records are not ordered, the granularity of what is ordered is the slice. Therefore, we will only update the
197            cursor value once at the end of every slice.
198        * Updating the state once every record would generate issues for data feed stop conditions or semi-incremental syncs where the
199            important state is the one at the beginning of the slice
200        """
201        return None

We explicitly disable checkpointing here. There are a couple reasons for that and not all are documented here but:

  • In the case where records are not ordered, the granularity of what is ordered is the slice. Therefore, we will only update the cursor value once at the end of every slice.
  • Updating the state once every record would generate issues for data feed stop conditions or semi-incremental syncs where the important state is the one at the beginning of the slice
def get_cursor(self) -> Optional[airbyte_cdk.sources.streams.checkpoint.Cursor]:
203    def get_cursor(self) -> Optional[Cursor]:
204        return None

A Cursor is an interface that a stream can implement to manage how its internal state is read and updated while reading records. Historically, Python connectors had no concept of a cursor to manage state. Python streams need to define a cursor implementation and override this method to manage state through a Cursor.

@dataclass
class Decoder:
15@dataclass
16class Decoder:
17    """
18    Decoder strategy to transform a requests.Response into a Mapping[str, Any]
19    """
20
21    @abstractmethod
22    def is_stream_response(self) -> bool:
23        """
24        Set to True if you'd like to use stream=True option in http requester
25        """
26
27    @abstractmethod
28    def decode(self, response: requests.Response) -> DECODER_OUTPUT_TYPE:
29        """
30        Decodes a requests.Response into a Mapping[str, Any] or an array
31        :param response: the response to decode
32        :return: Generator of Mapping describing the response
33        """

Decoder strategy to transform a requests.Response into a Mapping[str, Any]

@abstractmethod
def is_stream_response(self) -> bool:
21    @abstractmethod
22    def is_stream_response(self) -> bool:
23        """
24        Set to True if you'd like to use stream=True option in http requester
25        """

Set to True if you'd like to use stream=True option in http requester

@abstractmethod
def decode( self, response: requests.models.Response) -> Generator[MutableMapping[str, Any], NoneType, NoneType]:
27    @abstractmethod
28    def decode(self, response: requests.Response) -> DECODER_OUTPUT_TYPE:
29        """
30        Decodes a requests.Response into a Mapping[str, Any] or an array
31        :param response: the response to decode
32        :return: Generator of Mapping describing the response
33        """

Decodes a requests.Response into a Mapping[str, Any] or an array

Parameters
  • response: the response to decode
Returns

Generator of Mapping describing the response

@dataclass
class DefaultPaginator(airbyte_cdk.sources.declarative.requesters.paginators.paginator.Paginator):
 37@dataclass
 38class DefaultPaginator(Paginator):
 39    """
 40    Default paginator to request pages of results with a fixed size until the pagination strategy no longer returns a next_page_token
 41
 42    Examples:
 43        1.
 44        * fetches up to 10 records at a time by setting the "limit" request param to 10
 45        * updates the request path with  "{{ response._metadata.next }}"
 46        ```
 47          paginator:
 48            type: "DefaultPaginator"
 49            page_size_option:
 50              type: RequestOption
 51              inject_into: request_parameter
 52              field_name: limit
 53            page_token_option:
 54              type: RequestPath
 55              path: "location"
 56            pagination_strategy:
 57              type: "CursorPagination"
 58              cursor_value: "{{ response._metadata.next }}"
 59              page_size: 10
 60        ```
 61
 62        2.
 63        * fetches up to 5 records at a time by setting the "page_size" header to 5
 64        * increments a record counter and set the request parameter "offset" to the value of the counter
 65        ```
 66          paginator:
 67            type: "DefaultPaginator"
 68            page_size_option:
 69              type: RequestOption
 70              inject_into: header
 71              field_name: page_size
 72            pagination_strategy:
 73              type: "OffsetIncrement"
 74              page_size: 5
 75            page_token_option:
 76              option_type: "request_parameter"
 77              field_name: "offset"
 78        ```
 79
 80        3.
 81        * fetches up to 5 records at a time by setting the "page_size" request param to 5
 82        * increments a page counter and set the request parameter "page" to the value of the counter
 83        ```
 84          paginator:
 85            type: "DefaultPaginator"
 86            page_size_option:
 87              type: RequestOption
 88              inject_into: request_parameter
 89              field_name: page_size
 90            pagination_strategy:
 91              type: "PageIncrement"
 92              page_size: 5
 93            page_token_option:
 94              type: RequestOption
 95              option_type: "request_parameter"
 96              field_name: "page"
 97        ```
 98    Attributes:
 99        page_size_option (Optional[RequestOption]): the request option to set the page size. Cannot be injected in the path.
100        page_token_option (Optional[RequestPath, RequestOption]): the request option to set the page token
101        pagination_strategy (PaginationStrategy): Strategy defining how to get the next page token
102        config (Config): connection config
103        url_base (Union[InterpolatedString, str]): endpoint's base url
104        decoder (Decoder): decoder to decode the response
105    """
106
107    pagination_strategy: PaginationStrategy
108    config: Config
109    url_base: Union[InterpolatedString, str]
110    parameters: InitVar[Mapping[str, Any]]
111    decoder: Decoder = field(
112        default_factory=lambda: PaginationDecoderDecorator(decoder=JsonDecoder(parameters={}))
113    )
114    page_size_option: Optional[RequestOption] = None
115    page_token_option: Optional[Union[RequestPath, RequestOption]] = None
116
117    def __post_init__(self, parameters: Mapping[str, Any]) -> None:
118        if self.page_size_option and not self.pagination_strategy.get_page_size():
119            raise ValueError(
120                "page_size_option cannot be set if the pagination strategy does not have a page_size"
121            )
122        if isinstance(self.url_base, str):
123            self.url_base = InterpolatedString(string=self.url_base, parameters=parameters)
124
125        if self.page_token_option and not isinstance(self.page_token_option, RequestPath):
126            _validate_component_request_option_paths(
127                self.config,
128                self.page_size_option,
129                self.page_token_option,
130            )
131
132    def get_initial_token(self) -> Optional[Any]:
133        """
134        Return the page token that should be used for the first request of a stream
135
136        WARNING: get_initial_token() should not be used by streams that use RFR that perform checkpointing
137        of state using page numbers. Because paginators are stateless
138        """
139        return self.pagination_strategy.initial_token
140
141    def get_page_size(self) -> Optional[int]:
142        return self.pagination_strategy.get_page_size()
143
144    def next_page_token(
145        self,
146        response: requests.Response,
147        last_page_size: int,
148        last_record: Optional[Record],
149        last_page_token_value: Optional[Any] = None,
150        page_size_override: Optional[int] = None,
151        stream_slice: Optional[StreamSlice] = None,
152    ) -> Optional[Mapping[str, Any]]:
153        next_page_token = self.pagination_strategy.next_page_token(
154            response=response,
155            last_page_size=last_page_size,
156            last_record=last_record,
157            last_page_token_value=last_page_token_value,
158            **page_size_override_kwargs(page_size_override),
159            **stream_slice_kwargs(self.pagination_strategy.next_page_token, stream_slice),
160        )
161        if next_page_token:
162            return {"next_page_token": next_page_token}
163        else:
164            return None
165
166    def path(
167        self,
168        next_page_token: Optional[Mapping[str, Any]],
169        stream_state: Optional[Mapping[str, Any]] = None,
170        stream_slice: Optional[StreamSlice] = None,
171    ) -> Optional[str]:
172        token = next_page_token.get("next_page_token") if next_page_token else None
173        if token and self.page_token_option and isinstance(self.page_token_option, RequestPath):
174            return str(token)
175        else:
176            return None
177
178    def get_request_params(
179        self,
180        *,
181        stream_state: Optional[StreamState] = None,
182        stream_slice: Optional[StreamSlice] = None,
183        next_page_token: Optional[Mapping[str, Any]] = None,
184        page_size_override: Optional[int] = None,
185    ) -> MutableMapping[str, Any]:
186        return self._get_request_options(
187            RequestOptionType.request_parameter, next_page_token, page_size_override
188        )
189
190    def get_request_headers(
191        self,
192        *,
193        stream_state: Optional[StreamState] = None,
194        stream_slice: Optional[StreamSlice] = None,
195        next_page_token: Optional[Mapping[str, Any]] = None,
196        page_size_override: Optional[int] = None,
197    ) -> Mapping[str, str]:
198        return self._get_request_options(
199            RequestOptionType.header, next_page_token, page_size_override
200        )
201
202    def get_request_body_data(
203        self,
204        *,
205        stream_state: Optional[StreamState] = None,
206        stream_slice: Optional[StreamSlice] = None,
207        next_page_token: Optional[Mapping[str, Any]] = None,
208        page_size_override: Optional[int] = None,
209    ) -> Mapping[str, Any]:
210        return self._get_request_options(
211            RequestOptionType.body_data, next_page_token, page_size_override
212        )
213
214    def get_request_body_json(
215        self,
216        *,
217        stream_state: Optional[StreamState] = None,
218        stream_slice: Optional[StreamSlice] = None,
219        next_page_token: Optional[Mapping[str, Any]] = None,
220        page_size_override: Optional[int] = None,
221    ) -> Mapping[str, Any]:
222        return self._get_request_options(
223            RequestOptionType.body_json, next_page_token, page_size_override
224        )
225
226    def _get_request_options(
227        self,
228        option_type: RequestOptionType,
229        next_page_token: Optional[Mapping[str, Any]],
230        page_size_override: Optional[int] = None,
231    ) -> MutableMapping[str, Any]:
232        options: MutableMapping[str, Any] = {}
233
234        token = next_page_token.get("next_page_token") if next_page_token else None
235        if (
236            self.page_token_option
237            and token is not None
238            and isinstance(self.page_token_option, RequestOption)
239            and self.page_token_option.inject_into == option_type
240        ):
241            self.page_token_option.inject_into_request(options, token, self.config)
242
243        if self.page_size_option and self.page_size_option.inject_into == option_type:
244            page_size = (
245                page_size_override
246                if page_size_override is not None
247                else self.pagination_strategy.get_page_size()
248            )
249            if page_size:
250                self.page_size_option.inject_into_request(options, page_size, self.config)
251
252        return options

Default paginator to request pages of results with a fixed size until the pagination strategy no longer returns a next_page_token

Examples:

1.

  • fetches up to 10 records at a time by setting the "limit" request param to 10
  • updates the request path with "{{ response._metadata.next }}"
  paginator:
    type: "DefaultPaginator"
    page_size_option:
      type: RequestOption
      inject_into: request_parameter
      field_name: limit
    page_token_option:
      type: RequestPath
      path: "location"
    pagination_strategy:
      type: "CursorPagination"
      cursor_value: "{{ response._metadata.next }}"
      page_size: 10

2.

  • fetches up to 5 records at a time by setting the "page_size" header to 5
  • increments a record counter and set the request parameter "offset" to the value of the counter
  paginator:
    type: "DefaultPaginator"
    page_size_option:
      type: RequestOption
      inject_into: header
      field_name: page_size
    pagination_strategy:
      type: "OffsetIncrement"
      page_size: 5
    page_token_option:
      option_type: "request_parameter"
      field_name: "offset"

3.

  • fetches up to 5 records at a time by setting the "page_size" request param to 5
  • increments a page counter and set the request parameter "page" to the value of the counter
  paginator:
    type: "DefaultPaginator"
    page_size_option:
      type: RequestOption
      inject_into: request_parameter
      field_name: page_size
    pagination_strategy:
      type: "PageIncrement"
      page_size: 5
    page_token_option:
      type: RequestOption
      option_type: "request_parameter"
      field_name: "page"
Attributes:
  • page_size_option (Optional[RequestOption]): the request option to set the page size. Cannot be injected in the path.
  • page_token_option (Optional[RequestPath, RequestOption]): the request option to set the page token
  • pagination_strategy (PaginationStrategy): Strategy defining how to get the next page token
  • config (Config): connection config
  • url_base (Union[InterpolatedString, str]): endpoint's base url
  • decoder (Decoder): decoder to decode the response
DefaultPaginator( pagination_strategy: PaginationStrategy, config: Mapping[str, Any], url_base: Union[InterpolatedString, str], parameters: dataclasses.InitVar[typing.Mapping[str, typing.Any]], decoder: Decoder = <factory>, page_size_option: Optional[RequestOption] = None, page_token_option: Union[airbyte_cdk.sources.declarative.requesters.request_path.RequestPath, RequestOption, NoneType] = None)
pagination_strategy: PaginationStrategy
config: Mapping[str, Any]
url_base: Union[InterpolatedString, str]
parameters: dataclasses.InitVar[typing.Mapping[str, typing.Any]]
decoder: Decoder
page_size_option: Optional[RequestOption] = None
def get_initial_token(self) -> Optional[Any]:
132    def get_initial_token(self) -> Optional[Any]:
133        """
134        Return the page token that should be used for the first request of a stream
135
136        WARNING: get_initial_token() should not be used by streams that use RFR that perform checkpointing
137        of state using page numbers. Because paginators are stateless
138        """
139        return self.pagination_strategy.initial_token

Return the page token that should be used for the first request of a stream

WARNING: get_initial_token() should not be used by streams that use RFR that perform checkpointing of state using page numbers. Because paginators are stateless

def get_page_size(self) -> Optional[int]:
141    def get_page_size(self) -> Optional[int]:
142        return self.pagination_strategy.get_page_size()

Evaluated against the config alone. A pagination strategy whose page_size template references the response evaluates it with the response in context inside next_page_token, so the two can disagree - pre-existing, and only observable for a page_size that is not a constant.

Returns

the number of records this paginator asks for per page, or None if it does not define one

def next_page_token( self, response: requests.models.Response, last_page_size: int, last_record: Optional[Record], last_page_token_value: Optional[Any] = None, page_size_override: Optional[int] = None, stream_slice: Optional[StreamSlice] = None) -> Optional[Mapping[str, Any]]:
144    def next_page_token(
145        self,
146        response: requests.Response,
147        last_page_size: int,
148        last_record: Optional[Record],
149        last_page_token_value: Optional[Any] = None,
150        page_size_override: Optional[int] = None,
151        stream_slice: Optional[StreamSlice] = None,
152    ) -> Optional[Mapping[str, Any]]:
153        next_page_token = self.pagination_strategy.next_page_token(
154            response=response,
155            last_page_size=last_page_size,
156            last_record=last_record,
157            last_page_token_value=last_page_token_value,
158            **page_size_override_kwargs(page_size_override),
159            **stream_slice_kwargs(self.pagination_strategy.next_page_token, stream_slice),
160        )
161        if next_page_token:
162            return {"next_page_token": next_page_token}
163        else:
164            return None

Returns the next_page_token to use to fetch the next page of records.

Parameters
  • response: the response to process
  • last_page_size: the number of records read from the response
  • last_record: the last record extracted from the response
  • last_page_token_value: The current value of the page token made on the last request
  • page_size_override: the page size that was actually requested, when it differs from the configured one because of a REDUCE_PAGE_SIZE response action
  • stream_slice: the slice the page was read for, so that a stop condition can compare the page against the slice's own window or partition
Returns

A mapping {"next_page_token": } for the next page from the input response object. Returning None means there are no more pages to read in this response.

def path( self, next_page_token: Optional[Mapping[str, Any]], stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[StreamSlice] = None) -> Optional[str]:
166    def path(
167        self,
168        next_page_token: Optional[Mapping[str, Any]],
169        stream_state: Optional[Mapping[str, Any]] = None,
170        stream_slice: Optional[StreamSlice] = None,
171    ) -> Optional[str]:
172        token = next_page_token.get("next_page_token") if next_page_token else None
173        if token and self.page_token_option and isinstance(self.page_token_option, RequestPath):
174            return str(token)
175        else:
176            return None

Returns the URL path to hit to fetch the next page of records

e.g: if you wanted to hit https://myapi.com/v1/some_entity then this will return "some_entity"

Returns

path to hit to fetch the next request. Returning None means the path is not defined by the next_page_token

def get_request_params( self, *, stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[StreamSlice] = None, next_page_token: Optional[Mapping[str, Any]] = None, page_size_override: Optional[int] = None) -> MutableMapping[str, Any]:
178    def get_request_params(
179        self,
180        *,
181        stream_state: Optional[StreamState] = None,
182        stream_slice: Optional[StreamSlice] = None,
183        next_page_token: Optional[Mapping[str, Any]] = None,
184        page_size_override: Optional[int] = None,
185    ) -> MutableMapping[str, Any]:
186        return self._get_request_options(
187            RequestOptionType.request_parameter, next_page_token, page_size_override
188        )

Specifies the query parameters that should be set on an outgoing HTTP request given the inputs.

E.g: you might want to define query parameters for paging if next_page_token is not None.

def get_request_headers( self, *, stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[StreamSlice] = None, next_page_token: Optional[Mapping[str, Any]] = None, page_size_override: Optional[int] = None) -> Mapping[str, str]:
190    def get_request_headers(
191        self,
192        *,
193        stream_state: Optional[StreamState] = None,
194        stream_slice: Optional[StreamSlice] = None,
195        next_page_token: Optional[Mapping[str, Any]] = None,
196        page_size_override: Optional[int] = None,
197    ) -> Mapping[str, str]:
198        return self._get_request_options(
199            RequestOptionType.header, next_page_token, page_size_override
200        )

Return any non-auth headers. Authentication headers will overwrite any overlapping headers returned from this method.

def get_request_body_data( self, *, stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[StreamSlice] = None, next_page_token: Optional[Mapping[str, Any]] = None, page_size_override: Optional[int] = None) -> Mapping[str, Any]:
202    def get_request_body_data(
203        self,
204        *,
205        stream_state: Optional[StreamState] = None,
206        stream_slice: Optional[StreamSlice] = None,
207        next_page_token: Optional[Mapping[str, Any]] = None,
208        page_size_override: Optional[int] = None,
209    ) -> Mapping[str, Any]:
210        return self._get_request_options(
211            RequestOptionType.body_data, next_page_token, page_size_override
212        )

Specifies how to populate the body of the request with a non-JSON payload.

If returns a ready text that it will be sent as is. If returns a dict that it will be converted to a urlencoded form. E.g. {"key1": "value1", "key2": "value2"} => "key1=value1&key2=value2"

At the same time only one of the 'request_body_data' and 'request_body_json' functions can be overridden.

def get_request_body_json( self, *, stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[StreamSlice] = None, next_page_token: Optional[Mapping[str, Any]] = None, page_size_override: Optional[int] = None) -> Mapping[str, Any]:
214    def get_request_body_json(
215        self,
216        *,
217        stream_state: Optional[StreamState] = None,
218        stream_slice: Optional[StreamSlice] = None,
219        next_page_token: Optional[Mapping[str, Any]] = None,
220        page_size_override: Optional[int] = None,
221    ) -> Mapping[str, Any]:
222        return self._get_request_options(
223            RequestOptionType.body_json, next_page_token, page_size_override
224        )

Specifies how to populate the body of the request with a JSON payload.

At the same time only one of the 'request_body_data' and 'request_body_json' functions can be overridden.

15@dataclass
16class DefaultRequestOptionsProvider(RequestOptionsProvider):
17    """
18    Request options provider that extracts fields from the stream_slice and injects them into the respective location in the
19    outbound request being made
20    """
21
22    parameters: InitVar[Mapping[str, Any]]
23
24    def __post_init__(self, parameters: Mapping[str, Any]) -> None:
25        pass
26
27    def get_request_params(
28        self,
29        *,
30        stream_state: Optional[StreamState] = None,
31        stream_slice: Optional[StreamSlice] = None,
32        next_page_token: Optional[Mapping[str, Any]] = None,
33    ) -> Mapping[str, Any]:
34        return {}
35
36    def get_request_headers(
37        self,
38        *,
39        stream_state: Optional[StreamState] = None,
40        stream_slice: Optional[StreamSlice] = None,
41        next_page_token: Optional[Mapping[str, Any]] = None,
42    ) -> Mapping[str, Any]:
43        return {}
44
45    def get_request_body_data(
46        self,
47        *,
48        stream_state: Optional[StreamState] = None,
49        stream_slice: Optional[StreamSlice] = None,
50        next_page_token: Optional[Mapping[str, Any]] = None,
51    ) -> Union[Mapping[str, Any], str]:
52        return {}
53
54    def get_request_body_json(
55        self,
56        *,
57        stream_state: Optional[StreamState] = None,
58        stream_slice: Optional[StreamSlice] = None,
59        next_page_token: Optional[Mapping[str, Any]] = None,
60    ) -> Mapping[str, Any]:
61        return {}

Request options provider that extracts fields from the stream_slice and injects them into the respective location in the outbound request being made

DefaultRequestOptionsProvider(parameters: dataclasses.InitVar[typing.Mapping[str, typing.Any]])
parameters: dataclasses.InitVar[typing.Mapping[str, typing.Any]]
def get_request_params( self, *, stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[StreamSlice] = None, next_page_token: Optional[Mapping[str, Any]] = None) -> Mapping[str, Any]:
27    def get_request_params(
28        self,
29        *,
30        stream_state: Optional[StreamState] = None,
31        stream_slice: Optional[StreamSlice] = None,
32        next_page_token: Optional[Mapping[str, Any]] = None,
33    ) -> Mapping[str, Any]:
34        return {}

Specifies the query parameters that should be set on an outgoing HTTP request given the inputs.

E.g: you might want to define query parameters for paging if next_page_token is not None.

def get_request_headers( self, *, stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[StreamSlice] = None, next_page_token: Optional[Mapping[str, Any]] = None) -> Mapping[str, Any]:
36    def get_request_headers(
37        self,
38        *,
39        stream_state: Optional[StreamState] = None,
40        stream_slice: Optional[StreamSlice] = None,
41        next_page_token: Optional[Mapping[str, Any]] = None,
42    ) -> Mapping[str, Any]:
43        return {}

Return any non-auth headers. Authentication headers will overwrite any overlapping headers returned from this method.

def get_request_body_data( self, *, stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[StreamSlice] = None, next_page_token: Optional[Mapping[str, Any]] = None) -> Union[Mapping[str, Any], str]:
45    def get_request_body_data(
46        self,
47        *,
48        stream_state: Optional[StreamState] = None,
49        stream_slice: Optional[StreamSlice] = None,
50        next_page_token: Optional[Mapping[str, Any]] = None,
51    ) -> Union[Mapping[str, Any], str]:
52        return {}

Specifies how to populate the body of the request with a non-JSON payload.

If returns a ready text that it will be sent as is. If returns a dict that it will be converted to a urlencoded form. E.g. {"key1": "value1", "key2": "value2"} => "key1=value1&key2=value2"

At the same time only one of the 'request_body_data' and 'request_body_json' functions can be overridden.

def get_request_body_json( self, *, stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[StreamSlice] = None, next_page_token: Optional[Mapping[str, Any]] = None) -> Mapping[str, Any]:
54    def get_request_body_json(
55        self,
56        *,
57        stream_state: Optional[StreamState] = None,
58        stream_slice: Optional[StreamSlice] = None,
59        next_page_token: Optional[Mapping[str, Any]] = None,
60    ) -> Mapping[str, Any]:
61        return {}

Specifies how to populate the body of the request with a JSON payload.

At the same time only one of the 'request_body_data' and 'request_body_json' functions can be overridden.

@dataclass
class DpathExtractor(airbyte_cdk.RecordExtractor):
 19@dataclass
 20class DpathExtractor(RecordExtractor):
 21    """
 22    Record extractor that searches a decoded response over a path defined as an array of fields.
 23
 24    If the field path points to an array, that array is returned.
 25    If the field path points to an object, that object is returned wrapped as an array.
 26    If the field path points to an empty object, an empty array is returned.
 27    If the field path points to a non-existing path, an empty array is returned.
 28
 29    Optionally, records can be expanded by providing a RecordExpander component.
 30    When record_expander is configured, each extracted record is passed through the
 31    expander which extracts items from nested array fields and emits each item as a
 32    separate record.
 33
 34    Examples of instantiating this transform:
 35    ```
 36      extractor:
 37        type: DpathExtractor
 38        field_path:
 39          - "root"
 40          - "data"
 41    ```
 42
 43    ```
 44      extractor:
 45        type: DpathExtractor
 46        field_path:
 47          - "root"
 48          - "{{ parameters['field'] }}"
 49    ```
 50
 51    ```
 52      extractor:
 53        type: DpathExtractor
 54        field_path: []
 55    ```
 56
 57    ```
 58      extractor:
 59        type: DpathExtractor
 60        field_path:
 61          - "data"
 62          - "object"
 63        record_expander:
 64          type: RecordExpander
 65          expand_records_from_field:
 66            - "lines"
 67            - "data"
 68          remain_original_record: true
 69    ```
 70
 71    Attributes:
 72        field_path (Union[InterpolatedString, str]): Path to the field that should be extracted
 73        config (Config): The user-provided configuration as specified by the source's spec
 74        decoder (Decoder): The decoder responsible to transfom the response in a Mapping
 75        record_expander (Optional[RecordExpander]): Optional component to expand records by extracting items from nested array fields
 76    """
 77
 78    field_path: List[Union[InterpolatedString, str]]
 79    config: Config
 80    parameters: InitVar[Mapping[str, Any]]
 81    decoder: Decoder = field(default_factory=lambda: JsonDecoder(parameters={}))
 82    record_expander: Optional[RecordExpander] = None
 83
 84    def __post_init__(self, parameters: Mapping[str, Any]) -> None:
 85        self._field_path = [
 86            InterpolatedString.create(path, parameters=parameters) for path in self.field_path
 87        ]
 88        for path_index in range(len(self.field_path)):
 89            if isinstance(self.field_path[path_index], str):
 90                self._field_path[path_index] = InterpolatedString.create(
 91                    self.field_path[path_index], parameters=parameters
 92                )
 93
 94    def extract_records(self, response: requests.Response) -> Iterable[MutableMapping[Any, Any]]:
 95        for body in self.decoder.decode(response):
 96            if len(self._field_path) == 0:
 97                extracted = body
 98            else:
 99                path = [path.eval(self.config) for path in self._field_path]
100                if "*" in path:
101                    extracted = dpath.values(body, path)
102                else:
103                    extracted = dpath.get(body, path, default=[])  # type: ignore # extracted will be a MutableMapping, given input data structure
104            if isinstance(extracted, list):
105                if not self.record_expander:
106                    yield from extracted
107                else:
108                    for record in extracted:
109                        yield from self.record_expander.expand_record(record)
110            elif extracted:
111                if self.record_expander:
112                    yield from self.record_expander.expand_record(extracted)
113                else:
114                    yield extracted
115            else:
116                yield from []

Record extractor that searches a decoded response over a path defined as an array of fields.

If the field path points to an array, that array is returned. If the field path points to an object, that object is returned wrapped as an array. If the field path points to an empty object, an empty array is returned. If the field path points to a non-existing path, an empty array is returned.

Optionally, records can be expanded by providing a RecordExpander component. When record_expander is configured, each extracted record is passed through the expander which extracts items from nested array fields and emits each item as a separate record.

Examples of instantiating this transform:

  extractor:
    type: DpathExtractor
    field_path:
      - "root"
      - "data"
  extractor:
    type: DpathExtractor
    field_path:
      - "root"
      - "{{ parameters['field'] }}"
  extractor:
    type: DpathExtractor
    field_path: []
  extractor:
    type: DpathExtractor
    field_path:
      - "data"
      - "object"
    record_expander:
      type: RecordExpander
      expand_records_from_field:
        - "lines"
        - "data"
      remain_original_record: true
Attributes:
  • field_path (Union[InterpolatedString, str]): Path to the field that should be extracted
  • config (Config): The user-provided configuration as specified by the source's spec
  • decoder (Decoder): The decoder responsible to transfom the response in a Mapping
  • record_expander (Optional[RecordExpander]): Optional component to expand records by extracting items from nested array fields
DpathExtractor( field_path: List[Union[InterpolatedString, str]], config: Mapping[str, Any], parameters: dataclasses.InitVar[typing.Mapping[str, typing.Any]], decoder: Decoder = <factory>, record_expander: Optional[airbyte_cdk.sources.declarative.expanders.RecordExpander] = None)
field_path: List[Union[InterpolatedString, str]]
config: Mapping[str, Any]
parameters: dataclasses.InitVar[typing.Mapping[str, typing.Any]]
decoder: Decoder
def extract_records( self, response: requests.models.Response) -> Iterable[MutableMapping[Any, Any]]:
 94    def extract_records(self, response: requests.Response) -> Iterable[MutableMapping[Any, Any]]:
 95        for body in self.decoder.decode(response):
 96            if len(self._field_path) == 0:
 97                extracted = body
 98            else:
 99                path = [path.eval(self.config) for path in self._field_path]
100                if "*" in path:
101                    extracted = dpath.values(body, path)
102                else:
103                    extracted = dpath.get(body, path, default=[])  # type: ignore # extracted will be a MutableMapping, given input data structure
104            if isinstance(extracted, list):
105                if not self.record_expander:
106                    yield from extracted
107                else:
108                    for record in extracted:
109                        yield from self.record_expander.expand_record(record)
110            elif extracted:
111                if self.record_expander:
112                    yield from self.record_expander.expand_record(extracted)
113                else:
114                    yield extracted
115            else:
116                yield from []

Selects records from the response

Parameters
  • response: The response to extract the records from
Returns

List of Records extracted from the response

FieldPointer = typing.List[str]
class HttpMethod(enum.Enum):
19class HttpMethod(Enum):
20    """
21    Http Method to use when submitting an outgoing HTTP request
22    """
23
24    DELETE = "DELETE"
25    GET = "GET"
26    PATCH = "PATCH"
27    POST = "POST"

Http Method to use when submitting an outgoing HTTP request

DELETE = <HttpMethod.DELETE: 'DELETE'>
GET = <HttpMethod.GET: 'GET'>
PATCH = <HttpMethod.PATCH: 'PATCH'>
POST = <HttpMethod.POST: 'POST'>
@dataclass
class HttpRequester(airbyte_cdk.Requester):
 37@dataclass
 38class HttpRequester(Requester):
 39    """
 40    Default implementation of a Requester
 41
 42    Attributes:
 43        name (str): Name of the stream. Only used for request/response caching
 44        url_base (Union[InterpolatedString, str]): Base url to send requests to
 45        path (Union[InterpolatedString, str]): Path to send requests to
 46        http_method (Union[str, HttpMethod]): HTTP method to use when sending requests
 47        request_options_provider (Optional[InterpolatedRequestOptionsProvider]): request option provider defining the options to set on outgoing requests
 48        authenticator (DeclarativeAuthenticator): Authenticator defining how to authenticate to the source
 49        error_handler (Optional[ErrorHandler]): Error handler defining how to detect and handle errors
 50        backoff_strategies (Optional[List[BackoffStrategy]]): List of backoff strategies to use when retrying requests
 51        config (Config): The user-provided configuration as specified by the source's spec
 52        use_cache (bool): Indicates that data should be cached for this stream
 53    """
 54
 55    name: str
 56    config: Config
 57    parameters: InitVar[Mapping[str, Any]]
 58
 59    url: Optional[Union[InterpolatedString, str]] = None
 60    url_base: Optional[Union[InterpolatedString, str]] = None
 61    path: Optional[Union[InterpolatedString, str]] = None
 62    authenticator: Optional[DeclarativeAuthenticator] = None
 63    http_method: Union[str, HttpMethod] = HttpMethod.GET
 64    request_options_provider: Optional[InterpolatedRequestOptionsProvider] = None
 65    error_handler: Optional[ErrorHandler] = None
 66    api_budget: Optional[APIBudget] = None
 67    disable_retries: bool = False
 68    message_repository: MessageRepository = NoopMessageRepository()
 69    use_cache: bool = False
 70    connect_timeout_in_seconds: Optional[float] = None
 71    read_timeout_in_seconds: Optional[float] = None
 72    use_tcp_keepalive: bool = False
 73    _exit_on_rate_limit: bool = False
 74    stream_response: bool = False
 75    decoder: Decoder = field(default_factory=lambda: JsonDecoder(parameters={}))
 76
 77    def __post_init__(self, parameters: Mapping[str, Any]) -> None:
 78        self._url = InterpolatedString.create(
 79            self.url if self.url else EmptyString, parameters=parameters
 80        )
 81        # deprecated
 82        self._url_base = InterpolatedString.create(
 83            self.url_base if self.url_base else EmptyString, parameters=parameters
 84        )
 85        # deprecated
 86        self._path = InterpolatedString.create(
 87            self.path if self.path else EmptyString, parameters=parameters
 88        )
 89        if self.request_options_provider is None:
 90            self._request_options_provider = InterpolatedRequestOptionsProvider(
 91                config=self.config, parameters=parameters
 92            )
 93        elif isinstance(self.request_options_provider, dict):
 94            self._request_options_provider = InterpolatedRequestOptionsProvider(
 95                config=self.config, **self.request_options_provider
 96            )
 97        else:
 98            self._request_options_provider = self.request_options_provider
 99        self._authenticator = self.authenticator or NoAuth(parameters=parameters)
100        self._http_method = (
101            HttpMethod[self.http_method] if isinstance(self.http_method, str) else self.http_method
102        )
103        self.error_handler = self.error_handler
104        self._parameters = parameters
105
106        if self.error_handler is not None and hasattr(self.error_handler, "backoff_strategies"):
107            backoff_strategies = self.error_handler.backoff_strategies  # type: ignore
108        else:
109            backoff_strategies = None
110
111        self._http_client = HttpClient(
112            name=self.name,
113            logger=self.logger,
114            error_handler=self.error_handler,
115            api_budget=self.api_budget,
116            authenticator=self._authenticator,
117            use_cache=self.use_cache,
118            backoff_strategy=backoff_strategies,
119            disable_retries=self.disable_retries,
120            message_repository=self.message_repository,
121            request_timeout=(
122                (self.connect_timeout_in_seconds, self.read_timeout_in_seconds)
123                if self.connect_timeout_in_seconds is not None
124                or self.read_timeout_in_seconds is not None
125                else None
126            ),
127            use_tcp_keepalive=self.use_tcp_keepalive,
128        )
129
130    @property
131    def exit_on_rate_limit(self) -> bool:
132        return self._exit_on_rate_limit
133
134    @exit_on_rate_limit.setter
135    def exit_on_rate_limit(self, value: bool) -> None:
136        self._exit_on_rate_limit = value
137
138    def get_authenticator(self) -> DeclarativeAuthenticator:
139        return self._authenticator
140
141    def get_url(
142        self,
143        *,
144        stream_state: Optional[StreamState] = None,
145        stream_slice: Optional[StreamSlice] = None,
146        next_page_token: Optional[Mapping[str, Any]] = None,
147    ) -> str:
148        interpolation_context = get_interpolation_context(
149            stream_state=stream_state,
150            stream_slice=stream_slice,
151            next_page_token=next_page_token,
152        )
153
154        return str(self._url.eval(self.config, **interpolation_context))
155
156    def _get_url(
157        self,
158        *,
159        path: Optional[str] = None,
160        stream_state: Optional[StreamState] = None,
161        stream_slice: Optional[StreamSlice] = None,
162        next_page_token: Optional[Mapping[str, Any]] = None,
163    ) -> str:
164        url = self.get_url(
165            stream_state=stream_state,
166            stream_slice=stream_slice,
167            next_page_token=next_page_token,
168        )
169
170        url_base = self.get_url_base(
171            stream_state=stream_state,
172            stream_slice=stream_slice,
173            next_page_token=next_page_token,
174        )
175
176        path = path or self.get_path(
177            stream_state=stream_state,
178            stream_slice=stream_slice,
179            next_page_token=next_page_token,
180        )
181
182        full_url = (
183            self._join_url(url_base, path)
184            if url_base
185            else self._join_url(url, path)
186            if path
187            else url
188        )
189
190        return full_url
191
192    def get_url_base(
193        self,
194        *,
195        stream_state: Optional[StreamState] = None,
196        stream_slice: Optional[StreamSlice] = None,
197        next_page_token: Optional[Mapping[str, Any]] = None,
198    ) -> str:
199        interpolation_context = get_interpolation_context(
200            stream_state=stream_state,
201            stream_slice=stream_slice,
202            next_page_token=next_page_token,
203        )
204        return str(self._url_base.eval(self.config, **interpolation_context))
205
206    def get_path(
207        self,
208        *,
209        stream_state: Optional[StreamState] = None,
210        stream_slice: Optional[StreamSlice] = None,
211        next_page_token: Optional[Mapping[str, Any]] = None,
212    ) -> str:
213        interpolation_context = get_interpolation_context(
214            stream_state=stream_state,
215            stream_slice=stream_slice,
216            next_page_token=next_page_token,
217        )
218        path = str(self._path.eval(self.config, **interpolation_context))
219        return path.lstrip("/")
220
221    def get_method(self) -> HttpMethod:
222        return self._http_method
223
224    def get_request_params(
225        self,
226        *,
227        stream_state: Optional[StreamState] = None,
228        stream_slice: Optional[StreamSlice] = None,
229        next_page_token: Optional[Mapping[str, Any]] = None,
230    ) -> MutableMapping[str, Any]:
231        return self._request_options_provider.get_request_params(
232            stream_state=stream_state,
233            stream_slice=stream_slice,
234            next_page_token=next_page_token,
235        )
236
237    def get_request_headers(
238        self,
239        *,
240        stream_state: Optional[StreamState] = None,
241        stream_slice: Optional[StreamSlice] = None,
242        next_page_token: Optional[Mapping[str, Any]] = None,
243    ) -> Mapping[str, Any]:
244        return self._request_options_provider.get_request_headers(
245            stream_state=stream_state,
246            stream_slice=stream_slice,
247            next_page_token=next_page_token,
248        )
249
250    # fixing request options provider types has a lot of dependencies
251    def get_request_body_data(  # type: ignore
252        self,
253        *,
254        stream_state: Optional[StreamState] = None,
255        stream_slice: Optional[StreamSlice] = None,
256        next_page_token: Optional[Mapping[str, Any]] = None,
257    ) -> Union[Mapping[str, Any], str]:
258        return (
259            self._request_options_provider.get_request_body_data(
260                stream_state=stream_state,
261                stream_slice=stream_slice,
262                next_page_token=next_page_token,
263            )
264            or {}
265        )
266
267    # fixing request options provider types has a lot of dependencies
268    def get_request_body_json(  # type: ignore
269        self,
270        *,
271        stream_state: Optional[StreamState] = None,
272        stream_slice: Optional[StreamSlice] = None,
273        next_page_token: Optional[Mapping[str, Any]] = None,
274    ) -> Optional[Mapping[str, Any]]:
275        return self._request_options_provider.get_request_body_json(
276            stream_state=stream_state,
277            stream_slice=stream_slice,
278            next_page_token=next_page_token,
279        )
280
281    @property
282    def logger(self) -> logging.Logger:
283        return logging.getLogger(f"airbyte.HttpRequester.{self.name}")
284
285    def _get_request_options(
286        self,
287        stream_state: Optional[StreamState],
288        stream_slice: Optional[StreamSlice],
289        next_page_token: Optional[Mapping[str, Any]],
290        requester_method: Callable[..., Optional[Union[Mapping[str, Any], str]]],
291        auth_options_method: Callable[..., Optional[Union[Mapping[str, Any], str]]],
292        extra_options: Optional[Union[Mapping[str, Any], str]] = None,
293    ) -> Union[Mapping[str, Any], str]:
294        """
295        Get the request_option from the requester, the authenticator and extra_options passed in.
296        Raise a ValueError if there's a key collision
297        Returned merged mapping otherwise
298        """
299
300        is_body_json = requester_method.__name__ == "get_request_body_json"
301
302        return combine_mappings(
303            [
304                requester_method(
305                    stream_state=stream_state,
306                    stream_slice=stream_slice,
307                    next_page_token=next_page_token,
308                ),
309                auth_options_method(),
310                extra_options,
311            ],
312            allow_same_value_merge=is_body_json,
313        )
314
315    def _request_headers(
316        self,
317        stream_state: Optional[StreamState] = None,
318        stream_slice: Optional[StreamSlice] = None,
319        next_page_token: Optional[Mapping[str, Any]] = None,
320        extra_headers: Optional[Mapping[str, Any]] = None,
321    ) -> Mapping[str, Any]:
322        """
323        Specifies request headers.
324        Authentication headers will overwrite any overlapping headers returned from this method.
325        """
326        headers = self._get_request_options(
327            stream_state,
328            stream_slice,
329            next_page_token,
330            self.get_request_headers,
331            self.get_authenticator().get_auth_header,
332            extra_headers,
333        )
334        if isinstance(headers, str):
335            raise ValueError("Request headers cannot be a string")
336        return {str(k): str(v) for k, v in headers.items()}
337
338    def _request_params(
339        self,
340        stream_state: Optional[StreamState],
341        stream_slice: Optional[StreamSlice],
342        next_page_token: Optional[Mapping[str, Any]],
343        extra_params: Optional[Mapping[str, Any]] = None,
344    ) -> Mapping[str, Any]:
345        """
346        Specifies the query parameters that should be set on an outgoing HTTP request given the inputs.
347
348        E.g: you might want to define query parameters for paging if next_page_token is not None.
349        """
350        options = self._get_request_options(
351            stream_state,
352            stream_slice,
353            next_page_token,
354            self.get_request_params,
355            self.get_authenticator().get_request_params,
356            extra_params,
357        )
358        if isinstance(options, str):
359            raise ValueError("Request params cannot be a string")
360
361        for k, v in options.items():
362            if isinstance(v, (dict,)):
363                raise ValueError(
364                    f"Invalid value for `{k}` parameter. The values of request params cannot be an object."
365                )
366
367        return options
368
369    def _request_body_data(
370        self,
371        stream_state: Optional[StreamState],
372        stream_slice: Optional[StreamSlice],
373        next_page_token: Optional[Mapping[str, Any]],
374        extra_body_data: Optional[Union[Mapping[str, Any], str]] = None,
375    ) -> Optional[Union[Mapping[str, Any], str]]:
376        """
377        Specifies how to populate the body of the request with a non-JSON payload.
378
379        If returns a ready text that it will be sent as is.
380        If returns a dict that it will be converted to a urlencoded form.
381        E.g. {"key1": "value1", "key2": "value2"} => "key1=value1&key2=value2"
382
383        At the same time only one of the 'request_body_data' and 'request_body_json' functions can be overridden.
384        """
385        # Warning: use self.state instead of the stream_state passed as argument!
386        return self._get_request_options(
387            stream_state,
388            stream_slice,
389            next_page_token,
390            self.get_request_body_data,
391            self.get_authenticator().get_request_body_data,
392            extra_body_data,
393        )
394
395    def _request_body_json(
396        self,
397        stream_state: Optional[StreamState],
398        stream_slice: Optional[StreamSlice],
399        next_page_token: Optional[Mapping[str, Any]],
400        extra_body_json: Optional[Mapping[str, Any]] = None,
401    ) -> Optional[Mapping[str, Any]]:
402        """
403        Specifies how to populate the body of the request with a JSON payload.
404
405        At the same time only one of the 'request_body_data' and 'request_body_json' functions can be overridden.
406        """
407        # Warning: use self.state instead of the stream_state passed as argument!
408        options = self._get_request_options(
409            stream_state,
410            stream_slice,
411            next_page_token,
412            self.get_request_body_json,
413            self.get_authenticator().get_request_body_json,
414            extra_body_json,
415        )
416        if isinstance(options, str):
417            raise ValueError("Request body json cannot be a string")
418        return options
419
420    @classmethod
421    def _join_url(cls, url_base: str, path: Optional[str] = None) -> str:
422        """
423        Joins a base URL with a given path and returns the resulting URL with any trailing slash removed.
424
425        This method ensures that there are no duplicate slashes when concatenating the base URL and the path,
426        which is useful when the full URL is provided from an interpolation context.
427
428        Args:
429            url_base (str): The base URL to which the path will be appended.
430            path (Optional[str]): The path to join with the base URL.
431
432        Returns:
433            str: The resulting joined URL.
434
435        Note:
436            Related issue: https://github.com/airbytehq/airbyte-internal-issues/issues/11869
437            - If the path is an empty string or None, the method returns the base URL with any trailing slash removed.
438
439        Example:
440            1) _join_url("https://example.com/api/", "endpoint") >> 'https://example.com/api/endpoint'
441            2) _join_url("https://example.com/api", "/endpoint") >> 'https://example.com/api/endpoint'
442            3) _join_url("https://example.com/api/", "") >> 'https://example.com/api/'
443            4) _join_url("https://example.com/api", None) >> 'https://example.com/api'
444        """
445
446        # return a full-url if provided directly from interpolation context
447        if path == EmptyString or path is None:
448            return url_base
449        else:
450            # since we didn't provide a full-url, the url_base might not have a trailing slash
451            # so we join the url_base and path correctly
452            if not url_base.endswith("/"):
453                url_base += "/"
454
455        return urljoin(url_base, path)
456
457    def send_request(
458        self,
459        stream_state: Optional[StreamState] = None,
460        stream_slice: Optional[StreamSlice] = None,
461        next_page_token: Optional[Mapping[str, Any]] = None,
462        path: Optional[str] = None,
463        request_headers: Optional[Mapping[str, Any]] = None,
464        request_params: Optional[Mapping[str, Any]] = None,
465        request_body_data: Optional[Union[Mapping[str, Any], str]] = None,
466        request_body_json: Optional[Mapping[str, Any]] = None,
467        log_formatter: Optional[Callable[[requests.Response], Any]] = None,
468    ) -> Optional[requests.Response]:
469        request, response = self._http_client.send_request(
470            http_method=self.get_method().value,
471            url=self._get_url(
472                path=path,
473                stream_state=stream_state,
474                stream_slice=stream_slice,
475                next_page_token=next_page_token,
476            ),
477            request_kwargs={"stream": self.stream_response},
478            headers=self._request_headers(
479                stream_state, stream_slice, next_page_token, request_headers
480            ),
481            params=self._request_params(
482                stream_state, stream_slice, next_page_token, request_params
483            ),
484            json=self._request_body_json(
485                stream_state, stream_slice, next_page_token, request_body_json
486            ),
487            data=self._request_body_data(
488                stream_state, stream_slice, next_page_token, request_body_data
489            ),
490            dedupe_query_params=True,
491            log_formatter=log_formatter,
492            exit_on_rate_limit=self._exit_on_rate_limit,
493        )
494
495        return response

Default implementation of a Requester

Attributes:
  • name (str): Name of the stream. Only used for request/response caching
  • url_base (Union[InterpolatedString, str]): Base url to send requests to
  • path (Union[InterpolatedString, str]): Path to send requests to
  • http_method (Union[str, HttpMethod]): HTTP method to use when sending requests
  • request_options_provider (Optional[InterpolatedRequestOptionsProvider]): request option provider defining the options to set on outgoing requests
  • authenticator (DeclarativeAuthenticator): Authenticator defining how to authenticate to the source
  • error_handler (Optional[ErrorHandler]): Error handler defining how to detect and handle errors
  • backoff_strategies (Optional[List[BackoffStrategy]]): List of backoff strategies to use when retrying requests
  • config (Config): The user-provided configuration as specified by the source's spec
  • use_cache (bool): Indicates that data should be cached for this stream
HttpRequester( name: str, config: Mapping[str, Any], parameters: dataclasses.InitVar[typing.Mapping[str, typing.Any]], url: Union[InterpolatedString, str, NoneType] = None, url_base: Union[InterpolatedString, str, NoneType] = None, path: Union[InterpolatedString, str, NoneType] = None, authenticator: Optional[DeclarativeAuthenticator] = None, http_method: Union[str, HttpMethod] = <HttpMethod.GET: 'GET'>, request_options_provider: Optional[airbyte_cdk.sources.declarative.requesters.request_options.InterpolatedRequestOptionsProvider] = None, error_handler: Optional[airbyte_cdk.sources.streams.http.error_handlers.ErrorHandler] = None, api_budget: Optional[airbyte_cdk.sources.streams.call_rate.APIBudget] = None, disable_retries: bool = False, message_repository: MessageRepository = <airbyte_cdk.sources.message.NoopMessageRepository object>, use_cache: bool = False, connect_timeout_in_seconds: Optional[float] = None, read_timeout_in_seconds: Optional[float] = None, use_tcp_keepalive: bool = False, _exit_on_rate_limit: bool = False, stream_response: bool = False, decoder: Decoder = <factory>)
name: str
config: Mapping[str, Any]
parameters: dataclasses.InitVar[typing.Mapping[str, typing.Any]]
url: Union[InterpolatedString, str, NoneType] = None
url_base: Union[InterpolatedString, str, NoneType] = None
path: Union[InterpolatedString, str, NoneType] = None
authenticator: Optional[DeclarativeAuthenticator] = None
http_method: Union[str, HttpMethod] = <HttpMethod.GET: 'GET'>
disable_retries: bool = False
use_cache: bool = False
connect_timeout_in_seconds: Optional[float] = None
read_timeout_in_seconds: Optional[float] = None
use_tcp_keepalive: bool = False
stream_response: bool = False
decoder: Decoder
exit_on_rate_limit: bool
130    @property
131    def exit_on_rate_limit(self) -> bool:
132        return self._exit_on_rate_limit
def get_authenticator( self) -> DeclarativeAuthenticator:
138    def get_authenticator(self) -> DeclarativeAuthenticator:
139        return self._authenticator

Specifies the authenticator to use when submitting requests

def get_url( self, *, stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[StreamSlice] = None, next_page_token: Optional[Mapping[str, Any]] = None) -> str:
141    def get_url(
142        self,
143        *,
144        stream_state: Optional[StreamState] = None,
145        stream_slice: Optional[StreamSlice] = None,
146        next_page_token: Optional[Mapping[str, Any]] = None,
147    ) -> str:
148        interpolation_context = get_interpolation_context(
149            stream_state=stream_state,
150            stream_slice=stream_slice,
151            next_page_token=next_page_token,
152        )
153
154        return str(self._url.eval(self.config, **interpolation_context))
Returns

URL base for the API endpoint e.g: if you wanted to hit https://myapi.com/v1/some_entity then this should return "https://myapi.com/v1/"

def get_url_base( self, *, stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[StreamSlice] = None, next_page_token: Optional[Mapping[str, Any]] = None) -> str:
192    def get_url_base(
193        self,
194        *,
195        stream_state: Optional[StreamState] = None,
196        stream_slice: Optional[StreamSlice] = None,
197        next_page_token: Optional[Mapping[str, Any]] = None,
198    ) -> str:
199        interpolation_context = get_interpolation_context(
200            stream_state=stream_state,
201            stream_slice=stream_slice,
202            next_page_token=next_page_token,
203        )
204        return str(self._url_base.eval(self.config, **interpolation_context))
Returns

URL base for the API endpoint e.g: if you wanted to hit https://myapi.com/v1/some_entity then this should return "https://myapi.com/v1/"

def get_path( self, *, stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[StreamSlice] = None, next_page_token: Optional[Mapping[str, Any]] = None) -> str:
206    def get_path(
207        self,
208        *,
209        stream_state: Optional[StreamState] = None,
210        stream_slice: Optional[StreamSlice] = None,
211        next_page_token: Optional[Mapping[str, Any]] = None,
212    ) -> str:
213        interpolation_context = get_interpolation_context(
214            stream_state=stream_state,
215            stream_slice=stream_slice,
216            next_page_token=next_page_token,
217        )
218        path = str(self._path.eval(self.config, **interpolation_context))
219        return path.lstrip("/")

Returns the URL path for the API endpoint e.g: if you wanted to hit https://myapi.com/v1/some_entity then this should return "some_entity"

def get_method(self) -> HttpMethod:
221    def get_method(self) -> HttpMethod:
222        return self._http_method

Specifies the HTTP method to use

def get_request_params( self, *, stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[StreamSlice] = None, next_page_token: Optional[Mapping[str, Any]] = None) -> MutableMapping[str, Any]:
224    def get_request_params(
225        self,
226        *,
227        stream_state: Optional[StreamState] = None,
228        stream_slice: Optional[StreamSlice] = None,
229        next_page_token: Optional[Mapping[str, Any]] = None,
230    ) -> MutableMapping[str, Any]:
231        return self._request_options_provider.get_request_params(
232            stream_state=stream_state,
233            stream_slice=stream_slice,
234            next_page_token=next_page_token,
235        )

Specifies the query parameters that should be set on an outgoing HTTP request given the inputs.

E.g: you might want to define query parameters for paging if next_page_token is not None.

def get_request_headers( self, *, stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[StreamSlice] = None, next_page_token: Optional[Mapping[str, Any]] = None) -> Mapping[str, Any]:
237    def get_request_headers(
238        self,
239        *,
240        stream_state: Optional[StreamState] = None,
241        stream_slice: Optional[StreamSlice] = None,
242        next_page_token: Optional[Mapping[str, Any]] = None,
243    ) -> Mapping[str, Any]:
244        return self._request_options_provider.get_request_headers(
245            stream_state=stream_state,
246            stream_slice=stream_slice,
247            next_page_token=next_page_token,
248        )

Return any non-auth headers. Authentication headers will overwrite any overlapping headers returned from this method.

def get_request_body_data( self, *, stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[StreamSlice] = None, next_page_token: Optional[Mapping[str, Any]] = None) -> Union[Mapping[str, Any], str]:
251    def get_request_body_data(  # type: ignore
252        self,
253        *,
254        stream_state: Optional[StreamState] = None,
255        stream_slice: Optional[StreamSlice] = None,
256        next_page_token: Optional[Mapping[str, Any]] = None,
257    ) -> Union[Mapping[str, Any], str]:
258        return (
259            self._request_options_provider.get_request_body_data(
260                stream_state=stream_state,
261                stream_slice=stream_slice,
262                next_page_token=next_page_token,
263            )
264            or {}
265        )

Specifies how to populate the body of the request with a non-JSON payload.

If returns a ready text that it will be sent as is. If returns a dict that it will be converted to a urlencoded form. E.g. {"key1": "value1", "key2": "value2"} => "key1=value1&key2=value2"

At the same time only one of the 'request_body_data' and 'request_body_json' functions can be overridden.

def get_request_body_json( self, *, stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[StreamSlice] = None, next_page_token: Optional[Mapping[str, Any]] = None) -> Optional[Mapping[str, Any]]:
268    def get_request_body_json(  # type: ignore
269        self,
270        *,
271        stream_state: Optional[StreamState] = None,
272        stream_slice: Optional[StreamSlice] = None,
273        next_page_token: Optional[Mapping[str, Any]] = None,
274    ) -> Optional[Mapping[str, Any]]:
275        return self._request_options_provider.get_request_body_json(
276            stream_state=stream_state,
277            stream_slice=stream_slice,
278            next_page_token=next_page_token,
279        )

Specifies how to populate the body of the request with a JSON payload.

At the same time only one of the 'request_body_data' and 'request_body_json' functions can be overridden.

logger: logging.Logger
281    @property
282    def logger(self) -> logging.Logger:
283        return logging.getLogger(f"airbyte.HttpRequester.{self.name}")
def send_request( self, stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[StreamSlice] = None, next_page_token: Optional[Mapping[str, Any]] = None, path: Optional[str] = None, request_headers: Optional[Mapping[str, Any]] = None, request_params: Optional[Mapping[str, Any]] = None, request_body_data: Union[Mapping[str, Any], str, NoneType] = None, request_body_json: Optional[Mapping[str, Any]] = None, log_formatter: Optional[Callable[[requests.models.Response], Any]] = None) -> Optional[requests.models.Response]:
457    def send_request(
458        self,
459        stream_state: Optional[StreamState] = None,
460        stream_slice: Optional[StreamSlice] = None,
461        next_page_token: Optional[Mapping[str, Any]] = None,
462        path: Optional[str] = None,
463        request_headers: Optional[Mapping[str, Any]] = None,
464        request_params: Optional[Mapping[str, Any]] = None,
465        request_body_data: Optional[Union[Mapping[str, Any], str]] = None,
466        request_body_json: Optional[Mapping[str, Any]] = None,
467        log_formatter: Optional[Callable[[requests.Response], Any]] = None,
468    ) -> Optional[requests.Response]:
469        request, response = self._http_client.send_request(
470            http_method=self.get_method().value,
471            url=self._get_url(
472                path=path,
473                stream_state=stream_state,
474                stream_slice=stream_slice,
475                next_page_token=next_page_token,
476            ),
477            request_kwargs={"stream": self.stream_response},
478            headers=self._request_headers(
479                stream_state, stream_slice, next_page_token, request_headers
480            ),
481            params=self._request_params(
482                stream_state, stream_slice, next_page_token, request_params
483            ),
484            json=self._request_body_json(
485                stream_state, stream_slice, next_page_token, request_body_json
486            ),
487            data=self._request_body_data(
488                stream_state, stream_slice, next_page_token, request_body_data
489            ),
490            dedupe_query_params=True,
491            log_formatter=log_formatter,
492            exit_on_rate_limit=self._exit_on_rate_limit,
493        )
494
495        return response

Sends a request and returns the response. Might return no response if the error handler chooses to ignore the response or throw an exception in case of an error. If path is set, the path configured on the requester itself is ignored. If header, params and body are set, they are merged with the ones configured on the requester itself.

If a log formatter is provided, it's used to log the performed request and response. If it's not provided, no logging is performed.

@dataclass
class InterpolatedBoolean:
30@dataclass
31class InterpolatedBoolean:
32    f"""
33    Wrapper around a string to be evaluated to a boolean value.
34    The string will be evaluated as False if it interpolates to a value in {FALSE_VALUES}
35
36    Attributes:
37        condition (str): The string representing the condition to evaluate to a boolean
38    """
39    condition: str
40    parameters: InitVar[Mapping[str, Any]]
41
42    def __post_init__(self, parameters: Mapping[str, Any]) -> None:
43        self._default = "False"
44        self._interpolation = JinjaInterpolation()
45        self._parameters = parameters
46
47    def eval(self, config: Config, **additional_parameters: Any) -> bool:
48        """
49        Interpolates the predicate condition string using the config and other optional arguments passed as parameter.
50
51        :param config: The user-provided configuration as specified by the source's spec
52        :param additional_parameters: Optional parameters used for interpolation
53        :return: The interpolated string
54        """
55        if isinstance(self.condition, bool):
56            return self.condition
57        else:
58            evaluated = self._interpolation.eval(
59                self.condition,
60                config,
61                self._default,
62                parameters=self._parameters,
63                **additional_parameters,
64            )
65            if evaluated in FALSE_VALUES:
66                return False
67            # The presence of a value is generally regarded as truthy, so we treat it as such
68            return True
InterpolatedBoolean( condition: str, parameters: dataclasses.InitVar[typing.Mapping[str, typing.Any]])
condition: str
parameters: dataclasses.InitVar[typing.Mapping[str, typing.Any]]
def eval(self, config: Mapping[str, Any], **additional_parameters: Any) -> bool:
47    def eval(self, config: Config, **additional_parameters: Any) -> bool:
48        """
49        Interpolates the predicate condition string using the config and other optional arguments passed as parameter.
50
51        :param config: The user-provided configuration as specified by the source's spec
52        :param additional_parameters: Optional parameters used for interpolation
53        :return: The interpolated string
54        """
55        if isinstance(self.condition, bool):
56            return self.condition
57        else:
58            evaluated = self._interpolation.eval(
59                self.condition,
60                config,
61                self._default,
62                parameters=self._parameters,
63                **additional_parameters,
64            )
65            if evaluated in FALSE_VALUES:
66                return False
67            # The presence of a value is generally regarded as truthy, so we treat it as such
68            return True

Interpolates the predicate condition string using the config and other optional arguments passed as parameter.

Parameters
  • config: The user-provided configuration as specified by the source's spec
  • additional_parameters: Optional parameters used for interpolation
Returns

The interpolated string

@dataclass
class InterpolatedRequestInputProvider:
15@dataclass
16class InterpolatedRequestInputProvider:
17    """
18    Helper class that generically performs string interpolation on the provided dictionary or string input
19    """
20
21    parameters: InitVar[Mapping[str, Any]]
22    request_inputs: Optional[Union[str, Mapping[str, str]]] = field(default=None)
23    config: Config = field(default_factory=dict)
24    _interpolator: Optional[Union[InterpolatedString, InterpolatedMapping]] = field(
25        init=False, repr=False, default=None
26    )
27    _request_inputs: Optional[Union[str, Mapping[str, str]]] = field(
28        init=False, repr=False, default=None
29    )
30
31    def __post_init__(self, parameters: Mapping[str, Any]) -> None:
32        self._request_inputs = self.request_inputs or {}
33        if isinstance(self._request_inputs, str):
34            self._interpolator = InterpolatedString(
35                self._request_inputs, default="", parameters=parameters
36            )
37        else:
38            self._interpolator = InterpolatedMapping(self._request_inputs, parameters=parameters)
39
40    def eval_request_inputs(
41        self,
42        stream_slice: Optional[StreamSlice] = None,
43        next_page_token: Optional[Mapping[str, Any]] = None,
44        valid_key_types: Optional[Tuple[Type[Any]]] = None,
45        valid_value_types: Optional[Tuple[Type[Any], ...]] = None,
46    ) -> Mapping[str, Any]:
47        """
48        Returns the request inputs to set on an outgoing HTTP request
49
50        :param stream_slice: The stream slice
51        :param next_page_token: The pagination token
52        :param valid_key_types: A tuple of types that the interpolator should allow
53        :param valid_value_types: A tuple of types that the interpolator should allow
54        :return: The request inputs to set on an outgoing HTTP request
55        """
56        kwargs = get_interpolation_context(
57            stream_slice=stream_slice,
58            next_page_token=next_page_token,
59        )
60        interpolated_value = self._interpolator.eval(  # type: ignore # self._interpolator is always initialized with a value and will not be None
61            self.config,
62            valid_key_types=valid_key_types,
63            valid_value_types=valid_value_types,
64            **kwargs,
65        )
66
67        if isinstance(interpolated_value, dict):
68            non_null_tokens = {k: v for k, v in interpolated_value.items() if v is not None}
69            return non_null_tokens
70        return interpolated_value  # type: ignore[no-any-return]

Helper class that generically performs string interpolation on the provided dictionary or string input

InterpolatedRequestInputProvider( parameters: dataclasses.InitVar[typing.Mapping[str, typing.Any]], request_inputs: Union[str, Mapping[str, str], NoneType] = None, config: Mapping[str, Any] = <factory>)
parameters: dataclasses.InitVar[typing.Mapping[str, typing.Any]]
request_inputs: Union[str, Mapping[str, str], NoneType] = None
config: Mapping[str, Any]
def eval_request_inputs( self, stream_slice: Optional[StreamSlice] = None, next_page_token: Optional[Mapping[str, Any]] = None, valid_key_types: Optional[Tuple[Type[Any]]] = None, valid_value_types: Optional[Tuple[Type[Any], ...]] = None) -> Mapping[str, Any]:
40    def eval_request_inputs(
41        self,
42        stream_slice: Optional[StreamSlice] = None,
43        next_page_token: Optional[Mapping[str, Any]] = None,
44        valid_key_types: Optional[Tuple[Type[Any]]] = None,
45        valid_value_types: Optional[Tuple[Type[Any], ...]] = None,
46    ) -> Mapping[str, Any]:
47        """
48        Returns the request inputs to set on an outgoing HTTP request
49
50        :param stream_slice: The stream slice
51        :param next_page_token: The pagination token
52        :param valid_key_types: A tuple of types that the interpolator should allow
53        :param valid_value_types: A tuple of types that the interpolator should allow
54        :return: The request inputs to set on an outgoing HTTP request
55        """
56        kwargs = get_interpolation_context(
57            stream_slice=stream_slice,
58            next_page_token=next_page_token,
59        )
60        interpolated_value = self._interpolator.eval(  # type: ignore # self._interpolator is always initialized with a value and will not be None
61            self.config,
62            valid_key_types=valid_key_types,
63            valid_value_types=valid_value_types,
64            **kwargs,
65        )
66
67        if isinstance(interpolated_value, dict):
68            non_null_tokens = {k: v for k, v in interpolated_value.items() if v is not None}
69            return non_null_tokens
70        return interpolated_value  # type: ignore[no-any-return]

Returns the request inputs to set on an outgoing HTTP request

Parameters
  • stream_slice: The stream slice
  • next_page_token: The pagination token
  • valid_key_types: A tuple of types that the interpolator should allow
  • valid_value_types: A tuple of types that the interpolator should allow
Returns

The request inputs to set on an outgoing HTTP request

@dataclass
class InterpolatedString:
13@dataclass
14class InterpolatedString:
15    """
16    Wrapper around a raw string to be interpolated with the Jinja2 templating engine
17
18    Attributes:
19        string (str): The string to evalute
20        default (Optional[str]): The default value to return if the evaluation returns an empty string
21        parameters (Mapping[str, Any]): Additional runtime parameters to be used for string interpolation
22    """
23
24    string: str
25    parameters: InitVar[Mapping[str, Any]]
26    default: Optional[str] = None
27
28    def __post_init__(self, parameters: Mapping[str, Any]) -> None:
29        self.default = self.default or self.string
30        self._interpolation = JinjaInterpolation()
31        self._parameters = parameters
32        # indicates whether passed string is just a plain string, not Jinja template
33        # This allows for optimization, but we do not know it yet at this stage
34        self._is_plain_string = None
35
36    def eval(self, config: Config, **kwargs: Any) -> Any:
37        """
38        Interpolates the input string using the config and other optional arguments passed as parameter.
39
40        :param config: The user-provided configuration as specified by the source's spec
41        :param kwargs: Optional parameters used for interpolation
42        :return: The interpolated string
43        """
44        if self._is_plain_string:
45            return self.string
46        if self._is_plain_string is None:
47            # Let's check whether output from evaluation is the same as input.
48            # This indicates occurrence of a plain string, not a template and we can skip Jinja in subsequent runs.
49            evaluated = self._interpolation.eval(
50                self.string, config, self.default, parameters=self._parameters, **kwargs
51            )
52            self._is_plain_string = self.string == evaluated
53            return evaluated
54        return self._interpolation.eval(
55            self.string, config, self.default, parameters=self._parameters, **kwargs
56        )
57
58    def __eq__(self, other: Any) -> bool:
59        if not isinstance(other, InterpolatedString):
60            return False
61        return self.string == other.string and self.default == other.default
62
63    @classmethod
64    def create(
65        cls,
66        string_or_interpolated: Union["InterpolatedString", str],
67        *,
68        parameters: Mapping[str, Any],
69    ) -> "InterpolatedString":
70        """
71        Helper function to obtain an InterpolatedString from either a raw string or an InterpolatedString.
72
73        :param string_or_interpolated: either a raw string or an InterpolatedString.
74        :param parameters: parameters propagated from parent component
75        :return: InterpolatedString representing the input string.
76        """
77        if isinstance(string_or_interpolated, str):
78            return InterpolatedString(string=string_or_interpolated, parameters=parameters)
79        else:
80            return string_or_interpolated

Wrapper around a raw string to be interpolated with the Jinja2 templating engine

Attributes:
  • string (str): The string to evalute
  • default (Optional[str]): The default value to return if the evaluation returns an empty string
  • parameters (Mapping[str, Any]): Additional runtime parameters to be used for string interpolation
InterpolatedString( string: str, parameters: dataclasses.InitVar[typing.Mapping[str, typing.Any]], default: Optional[str] = None)
string: str
parameters: dataclasses.InitVar[typing.Mapping[str, typing.Any]]
default: Optional[str] = None
def eval(self, config: Mapping[str, Any], **kwargs: Any) -> Any:
36    def eval(self, config: Config, **kwargs: Any) -> Any:
37        """
38        Interpolates the input string using the config and other optional arguments passed as parameter.
39
40        :param config: The user-provided configuration as specified by the source's spec
41        :param kwargs: Optional parameters used for interpolation
42        :return: The interpolated string
43        """
44        if self._is_plain_string:
45            return self.string
46        if self._is_plain_string is None:
47            # Let's check whether output from evaluation is the same as input.
48            # This indicates occurrence of a plain string, not a template and we can skip Jinja in subsequent runs.
49            evaluated = self._interpolation.eval(
50                self.string, config, self.default, parameters=self._parameters, **kwargs
51            )
52            self._is_plain_string = self.string == evaluated
53            return evaluated
54        return self._interpolation.eval(
55            self.string, config, self.default, parameters=self._parameters, **kwargs
56        )

Interpolates the input string using the config and other optional arguments passed as parameter.

Parameters
  • config: The user-provided configuration as specified by the source's spec
  • kwargs: Optional parameters used for interpolation
Returns

The interpolated string

@classmethod
def create( cls, string_or_interpolated: Union[InterpolatedString, str], *, parameters: Mapping[str, Any]) -> InterpolatedString:
63    @classmethod
64    def create(
65        cls,
66        string_or_interpolated: Union["InterpolatedString", str],
67        *,
68        parameters: Mapping[str, Any],
69    ) -> "InterpolatedString":
70        """
71        Helper function to obtain an InterpolatedString from either a raw string or an InterpolatedString.
72
73        :param string_or_interpolated: either a raw string or an InterpolatedString.
74        :param parameters: parameters propagated from parent component
75        :return: InterpolatedString representing the input string.
76        """
77        if isinstance(string_or_interpolated, str):
78            return InterpolatedString(string=string_or_interpolated, parameters=parameters)
79        else:
80            return string_or_interpolated

Helper function to obtain an InterpolatedString from either a raw string or an InterpolatedString.

Parameters
  • string_or_interpolated: either a raw string or an InterpolatedString.
  • parameters: parameters propagated from parent component
Returns

InterpolatedString representing the input string.

class JsonDecoder(airbyte_cdk.Decoder):
20class JsonDecoder(Decoder):
21    """
22    Decoder strategy that returns the json-encoded content of a response, if any.
23
24    Usually, we would try to instantiate the equivalent `CompositeRawDecoder(parser=JsonParser(), stream_response=False)` but there were specific historical behaviors related to the JsonDecoder that we didn't know if we could remove like the fallback on {} in case of errors.
25    """
26
27    def __init__(self, parameters: Mapping[str, Any]):
28        self._decoder = CompositeRawDecoder(parser=JsonParser(), stream_response=False)
29
30    def is_stream_response(self) -> bool:
31        return self._decoder.is_stream_response()
32
33    def decode(
34        self, response: requests.Response
35    ) -> Generator[MutableMapping[str, Any], None, None]:
36        """
37        Given the response is an empty string or an emtpy list, the function will return a generator with an empty mapping.
38        """
39        has_yielded = False
40        try:
41            for element in self._decoder.decode(response):
42                yield element
43                has_yielded = True
44        except Exception:
45            yield {}
46
47        if not has_yielded:
48            yield {}

Decoder strategy that returns the json-encoded content of a response, if any.

Usually, we would try to instantiate the equivalent CompositeRawDecoder(parser=JsonParser(), stream_response=False) but there were specific historical behaviors related to the JsonDecoder that we didn't know if we could remove like the fallback on {} in case of errors.

JsonDecoder(parameters: Mapping[str, Any])
27    def __init__(self, parameters: Mapping[str, Any]):
28        self._decoder = CompositeRawDecoder(parser=JsonParser(), stream_response=False)
def is_stream_response(self) -> bool:
30    def is_stream_response(self) -> bool:
31        return self._decoder.is_stream_response()

Set to True if you'd like to use stream=True option in http requester

def decode( self, response: requests.models.Response) -> Generator[MutableMapping[str, Any], NoneType, NoneType]:
33    def decode(
34        self, response: requests.Response
35    ) -> Generator[MutableMapping[str, Any], None, None]:
36        """
37        Given the response is an empty string or an emtpy list, the function will return a generator with an empty mapping.
38        """
39        has_yielded = False
40        try:
41            for element in self._decoder.decode(response):
42                yield element
43                has_yielded = True
44        except Exception:
45            yield {}
46
47        if not has_yielded:
48            yield {}

Given the response is an empty string or an emtpy list, the function will return a generator with an empty mapping.

33@dataclass
34class JsonFileSchemaLoader(ResourceSchemaLoader, SchemaLoader):
35    """
36    Loads the schema from a json file
37
38    Attributes:
39        file_path (Union[InterpolatedString, str]): The path to the json file describing the schema
40        name (str): The stream's name
41        config (Config): The user-provided configuration as specified by the source's spec
42        parameters (Mapping[str, Any]): Additional arguments to pass to the string interpolation if needed
43    """
44
45    config: Config
46    parameters: InitVar[Mapping[str, Any]]
47    file_path: Union[InterpolatedString, str] = field(default="")
48
49    def __post_init__(self, parameters: Mapping[str, Any]) -> None:
50        if not self.file_path:
51            self.file_path = _default_file_path()
52        self.file_path = InterpolatedString.create(self.file_path, parameters=parameters)
53
54    def get_json_schema(self) -> Mapping[str, Any]:
55        # todo: It is worth revisiting if we can replace file_path with just file_name if every schema is in the /schemas directory
56        # this would require that we find a creative solution to store or retrieve source_name in here since the files are mounted there
57        json_schema_path = self._get_json_filepath()
58        resource, schema_path = self.extract_resource_and_schema_path(json_schema_path)
59        raw_json_file = pkgutil.get_data(resource, schema_path)
60
61        if not raw_json_file:
62            raise IOError(f"Cannot find file {json_schema_path}")
63        try:
64            raw_schema = json.loads(raw_json_file)
65        except ValueError as err:
66            raise RuntimeError(f"Invalid JSON file format for file {json_schema_path}") from err
67        self.package_name = resource
68        return self._resolve_schema_references(raw_schema)
69
70    def _get_json_filepath(self) -> Any:
71        return self.file_path.eval(self.config)  # type: ignore # file_path is always cast to an interpolated string
72
73    @staticmethod
74    def extract_resource_and_schema_path(json_schema_path: str) -> Tuple[str, str]:
75        """
76        When the connector is running on a docker container, package_data is accessible from the resource (source_<name>), so we extract
77        the resource from the first part of the schema path and the remaining path is used to find the schema file. This is a slight
78        hack to identify the source name while we are in the airbyte_cdk module.
79        :param json_schema_path: The path to the schema JSON file
80        :return: Tuple of the resource name and the path to the schema file
81        """
82        split_path = json_schema_path.split("/")
83
84        if split_path[0] == "" or split_path[0] == ".":
85            split_path = split_path[1:]
86
87        if len(split_path) == 0:
88            return "", ""
89
90        if len(split_path) == 1:
91            return "", split_path[0]
92
93        return split_path[0], "/".join(split_path[1:])

Loads the schema from a json file

Attributes:
  • file_path (Union[InterpolatedString, str]): The path to the json file describing the schema
  • name (str): The stream's name
  • config (Config): The user-provided configuration as specified by the source's spec
  • parameters (Mapping[str, Any]): Additional arguments to pass to the string interpolation if needed
JsonFileSchemaLoader( config: Mapping[str, Any], parameters: dataclasses.InitVar[typing.Mapping[str, typing.Any]], file_path: Union[InterpolatedString, str] = '')
config: Mapping[str, Any]
parameters: dataclasses.InitVar[typing.Mapping[str, typing.Any]]
file_path: Union[InterpolatedString, str] = ''
def get_json_schema(self) -> Mapping[str, Any]:
54    def get_json_schema(self) -> Mapping[str, Any]:
55        # todo: It is worth revisiting if we can replace file_path with just file_name if every schema is in the /schemas directory
56        # this would require that we find a creative solution to store or retrieve source_name in here since the files are mounted there
57        json_schema_path = self._get_json_filepath()
58        resource, schema_path = self.extract_resource_and_schema_path(json_schema_path)
59        raw_json_file = pkgutil.get_data(resource, schema_path)
60
61        if not raw_json_file:
62            raise IOError(f"Cannot find file {json_schema_path}")
63        try:
64            raw_schema = json.loads(raw_json_file)
65        except ValueError as err:
66            raise RuntimeError(f"Invalid JSON file format for file {json_schema_path}") from err
67        self.package_name = resource
68        return self._resolve_schema_references(raw_schema)

Returns a mapping describing the stream's schema

@staticmethod
def extract_resource_and_schema_path(json_schema_path: str) -> Tuple[str, str]:
73    @staticmethod
74    def extract_resource_and_schema_path(json_schema_path: str) -> Tuple[str, str]:
75        """
76        When the connector is running on a docker container, package_data is accessible from the resource (source_<name>), so we extract
77        the resource from the first part of the schema path and the remaining path is used to find the schema file. This is a slight
78        hack to identify the source name while we are in the airbyte_cdk module.
79        :param json_schema_path: The path to the schema JSON file
80        :return: Tuple of the resource name and the path to the schema file
81        """
82        split_path = json_schema_path.split("/")
83
84        if split_path[0] == "" or split_path[0] == ".":
85            split_path = split_path[1:]
86
87        if len(split_path) == 0:
88            return "", ""
89
90        if len(split_path) == 1:
91            return "", split_path[0]
92
93        return split_path[0], "/".join(split_path[1:])

When the connector is running on a docker container, package_data is accessible from the resource (source_), so we extract the resource from the first part of the schema path and the remaining path is used to find the schema file. This is a slight hack to identify the source name while we are in the airbyte_cdk module.

Parameters
  • json_schema_path: The path to the schema JSON file
Returns

Tuple of the resource name and the path to the schema file

class LegacyToPerPartitionStateMigration(airbyte_cdk.sources.declarative.migrations.state_migration.StateMigration):
 21class LegacyToPerPartitionStateMigration(StateMigration):
 22    """
 23    Transforms the input state for per-partitioned streams from the legacy format to the low-code format.
 24    The cursor field and partition ID fields are automatically extracted from the stream's DatetimebasedCursor and SubstreamPartitionRouter.
 25
 26    Example input state:
 27    {
 28    "13506132": {
 29      "last_changed": "2022-12-27T08:34:39+00:00"
 30    }
 31    Example output state:
 32    {
 33      "partition": {"id": "13506132"},
 34      "cursor": {"last_changed": "2022-12-27T08:34:39+00:00"}
 35    }
 36    """
 37
 38    def __init__(
 39        self,
 40        partition_router: Union[
 41            SubstreamPartitionRouter, UnionPartitionRouter, CustomPartitionRouter
 42        ],
 43        cursor: DatetimeBasedCursor,
 44        config: Mapping[str, Any],
 45        parameters: Mapping[str, Any],
 46    ):
 47        self._partition_router = partition_router
 48        self._cursor = cursor
 49        self._config = config
 50        self._parameters = parameters
 51        self._partition_key_field = InterpolatedString.create(
 52            self._get_partition_field(partition_router), parameters=self._parameters
 53        ).eval(self._config)
 54        self._cursor_field = InterpolatedString.create(
 55            self._cursor.cursor_field, parameters=self._parameters
 56        ).eval(self._config)
 57
 58    def _get_partition_field(
 59        self,
 60        partition_router: Union[
 61            SubstreamPartitionRouter, UnionPartitionRouter, CustomPartitionRouter
 62        ],
 63    ) -> str:
 64        if isinstance(partition_router, UnionPartitionRouter):
 65            return partition_router.partition_field
 66
 67        parent_stream_config = partition_router.parent_stream_configs[0]  # type: ignore # custom partition routers are expected to expose parent_stream_configs
 68
 69        # Retrieve the partition field with a condition, as properties are returned as a dictionary for custom components.
 70        partition_field = (
 71            parent_stream_config.partition_field
 72            if isinstance(parent_stream_config, ParentStreamConfig)
 73            else parent_stream_config.get("partition_field")  # type: ignore # See above comment on why parent_stream_config might be a dict
 74        )
 75
 76        return partition_field
 77
 78    def should_migrate(self, stream_state: Mapping[str, Any]) -> bool:
 79        if _is_already_migrated(stream_state):
 80            return False
 81
 82        # UnionPartitionRouter has no parent_stream_configs; its partitions are already
 83        # normalized to a single partition field so the parent stream check does not apply.
 84        if not isinstance(self._partition_router, UnionPartitionRouter):
 85            # There is exactly one parent stream
 86            number_of_parent_streams = len(self._partition_router.parent_stream_configs)  # type: ignore # custom partition will introduce this attribute if needed
 87            if number_of_parent_streams != 1:
 88                # There should be exactly one parent stream
 89                return False
 90        """
 91        The expected state format is
 92        "<parent_key_id>" : {
 93          "<cursor_field>" : "<cursor_value>"
 94        }
 95        """
 96        if not stream_state:
 97            return False
 98        for key, value in stream_state.items():
 99            # it is expected the internal value to be a dictionary according to docstring
100            if not isinstance(value, dict):
101                return False
102            keys = list(value.keys())
103            if len(keys) != 1:
104                # The input partitioned state should only have one key
105                return False
106            if keys[0] != self._cursor_field:
107                # Unexpected key. Found {keys[0]}. Expected {self._cursor.cursor_field}
108                return False
109
110        return True
111
112    def migrate(self, stream_state: Mapping[str, Any]) -> Mapping[str, Any]:
113        states = [
114            {"partition": {self._partition_key_field: key}, "cursor": value}
115            for key, value in stream_state.items()
116        ]
117        return {"states": states}

Transforms the input state for per-partitioned streams from the legacy format to the low-code format. The cursor field and partition ID fields are automatically extracted from the stream's DatetimebasedCursor and SubstreamPartitionRouter.

Example input state: { "13506132": { "last_changed": "2022-12-27T08:34:39+00:00" } Example output state: { "partition": {"id": "13506132"}, "cursor": {"last_changed": "2022-12-27T08:34:39+00:00"} }

38    def __init__(
39        self,
40        partition_router: Union[
41            SubstreamPartitionRouter, UnionPartitionRouter, CustomPartitionRouter
42        ],
43        cursor: DatetimeBasedCursor,
44        config: Mapping[str, Any],
45        parameters: Mapping[str, Any],
46    ):
47        self._partition_router = partition_router
48        self._cursor = cursor
49        self._config = config
50        self._parameters = parameters
51        self._partition_key_field = InterpolatedString.create(
52            self._get_partition_field(partition_router), parameters=self._parameters
53        ).eval(self._config)
54        self._cursor_field = InterpolatedString.create(
55            self._cursor.cursor_field, parameters=self._parameters
56        ).eval(self._config)
def should_migrate(self, stream_state: Mapping[str, Any]) -> bool:
 78    def should_migrate(self, stream_state: Mapping[str, Any]) -> bool:
 79        if _is_already_migrated(stream_state):
 80            return False
 81
 82        # UnionPartitionRouter has no parent_stream_configs; its partitions are already
 83        # normalized to a single partition field so the parent stream check does not apply.
 84        if not isinstance(self._partition_router, UnionPartitionRouter):
 85            # There is exactly one parent stream
 86            number_of_parent_streams = len(self._partition_router.parent_stream_configs)  # type: ignore # custom partition will introduce this attribute if needed
 87            if number_of_parent_streams != 1:
 88                # There should be exactly one parent stream
 89                return False
 90        """
 91        The expected state format is
 92        "<parent_key_id>" : {
 93          "<cursor_field>" : "<cursor_value>"
 94        }
 95        """
 96        if not stream_state:
 97            return False
 98        for key, value in stream_state.items():
 99            # it is expected the internal value to be a dictionary according to docstring
100            if not isinstance(value, dict):
101                return False
102            keys = list(value.keys())
103            if len(keys) != 1:
104                # The input partitioned state should only have one key
105                return False
106            if keys[0] != self._cursor_field:
107                # Unexpected key. Found {keys[0]}. Expected {self._cursor.cursor_field}
108                return False
109
110        return True

Check if the stream_state should be migrated

Parameters
  • stream_state: The stream_state to potentially migrate
Returns

true if the state is of the expected format and should be migrated. False otherwise.

def migrate(self, stream_state: Mapping[str, Any]) -> Mapping[str, Any]:
112    def migrate(self, stream_state: Mapping[str, Any]) -> Mapping[str, Any]:
113        states = [
114            {"partition": {self._partition_key_field: key}, "cursor": value}
115            for key, value in stream_state.items()
116        ]
117        return {"states": states}

Migrate the stream_state. Assumes should_migrate(stream_state) returned True.

Parameters
  • stream_state: The stream_state to migrate
Returns

The migrated stream_state

@dataclass
class MinMaxDatetime:
 14@dataclass
 15class MinMaxDatetime:
 16    """
 17    Compares the provided date against optional minimum or maximum times. If date is earlier than
 18    min_date, then min_date is returned. If date is greater than max_date, then max_date is returned.
 19    If neither, the input date is returned.
 20
 21    The timestamp format accepts the same format codes as datetime.strfptime, which are
 22    all the format codes required by the 1989 C standard.
 23    Full list of accepted format codes: https://man7.org/linux/man-pages/man3/strftime.3.html
 24
 25    Attributes:
 26        datetime (Union[InterpolatedString, str]): InterpolatedString or string representing the datetime in the format specified by `datetime_format`
 27        datetime_format (str): Format of the datetime passed as argument
 28        min_datetime (Union[InterpolatedString, str]): Represents the minimum allowed datetime value.
 29        max_datetime (Union[InterpolatedString, str]): Represents the maximum allowed datetime value.
 30    """
 31
 32    datetime: Union[InterpolatedString, str]
 33    parameters: InitVar[Mapping[str, Any]]
 34    # datetime_format is a unique case where we inherit it from the parent if it is not specified before using the default value
 35    # which is why we need dedicated getter/setter methods and private dataclass field
 36    datetime_format: str
 37    _datetime_format: str = field(init=False, repr=False, default="")
 38    min_datetime: Union[InterpolatedString, str] = ""
 39    max_datetime: Union[InterpolatedString, str] = ""
 40
 41    def __post_init__(self, parameters: Mapping[str, Any]) -> None:
 42        self.datetime = InterpolatedString.create(self.datetime, parameters=parameters or {})
 43        self._parser = DatetimeParser()
 44        self.min_datetime = (
 45            InterpolatedString.create(self.min_datetime, parameters=parameters)  # type: ignore [assignment]  #  expression has type "InterpolatedString | None", variable has type "InterpolatedString | str"
 46            if self.min_datetime
 47            else None
 48        )  # type: ignore
 49        self.max_datetime = (
 50            InterpolatedString.create(self.max_datetime, parameters=parameters)  # type: ignore [assignment]  #  expression has type "InterpolatedString | None", variable has type "InterpolatedString | str"
 51            if self.max_datetime
 52            else None
 53        )  # type: ignore
 54
 55    def get_datetime(
 56        self, config: Mapping[str, Any], **additional_parameters: Mapping[str, Any]
 57    ) -> dt.datetime:
 58        """
 59        Evaluates and returns the datetime
 60        :param config: The user-provided configuration as specified by the source's spec
 61        :param additional_parameters: Additional arguments to be passed to the strings for interpolation
 62        :return: The evaluated datetime
 63        """
 64        # We apply a default datetime format here instead of at instantiation, so it can be set by the parent first
 65        datetime_format = self._datetime_format
 66        if not datetime_format:
 67            datetime_format = "%Y-%m-%dT%H:%M:%S.%f%z"
 68
 69        time = self._parser.parse(
 70            str(
 71                self.datetime.eval(  # type: ignore[union-attr] # str has no attribute "eval"
 72                    config,
 73                    **additional_parameters,
 74                )
 75            ),
 76            datetime_format,
 77        )  # type: ignore # datetime is always cast to an interpolated string
 78
 79        if self.min_datetime:
 80            min_time = str(self.min_datetime.eval(config, **additional_parameters))  # type: ignore # min_datetime is always cast to an interpolated string
 81            if min_time:
 82                min_datetime = self._parser.parse(min_time, datetime_format)  # type: ignore # min_datetime is always cast to an interpolated string
 83                time = max(time, min_datetime)
 84        if self.max_datetime:
 85            max_time = str(self.max_datetime.eval(config, **additional_parameters))  # type: ignore # max_datetime is always cast to an interpolated string
 86            if max_time:
 87                max_datetime = self._parser.parse(max_time, datetime_format)
 88                time = min(time, max_datetime)
 89        return time
 90
 91    @property  # type: ignore # properties don't play well with dataclasses...
 92    def datetime_format(self) -> str:
 93        """The format of the string representing the datetime"""
 94        return self._datetime_format
 95
 96    @datetime_format.setter
 97    def datetime_format(self, value: str) -> None:
 98        """Setter for the datetime format"""
 99        # Covers the case where datetime_format is not provided in the constructor, which causes the property object
100        # to be set which we need to avoid doing
101        if not isinstance(value, property):
102            self._datetime_format = value
103
104    @classmethod
105    def create(
106        cls,
107        interpolated_string_or_min_max_datetime: Union[InterpolatedString, str, "MinMaxDatetime"],
108        parameters: Optional[Mapping[str, Any]] = None,
109    ) -> "MinMaxDatetime":
110        if parameters is None:
111            parameters = {}
112        if isinstance(interpolated_string_or_min_max_datetime, InterpolatedString) or isinstance(
113            interpolated_string_or_min_max_datetime, str
114        ):
115            return MinMaxDatetime(  # type: ignore [call-arg]
116                datetime=interpolated_string_or_min_max_datetime, parameters=parameters
117            )
118        else:
119            return interpolated_string_or_min_max_datetime

Compares the provided date against optional minimum or maximum times. If date is earlier than min_date, then min_date is returned. If date is greater than max_date, then max_date is returned. If neither, the input date is returned.

The timestamp format accepts the same format codes as datetime.strfptime, which are all the format codes required by the 1989 C standard. Full list of accepted format codes: https://man7.org/linux/man-pages/man3/strftime.3.html

Attributes:
  • datetime (Union[InterpolatedString, str]): InterpolatedString or string representing the datetime in the format specified by datetime_format
  • datetime_format (str): Format of the datetime passed as argument
  • min_datetime (Union[InterpolatedString, str]): Represents the minimum allowed datetime value.
  • max_datetime (Union[InterpolatedString, str]): Represents the maximum allowed datetime value.
MinMaxDatetime( datetime: Union[InterpolatedString, str], parameters: dataclasses.InitVar[typing.Mapping[str, typing.Any]], datetime_format: str = <property object>, min_datetime: Union[InterpolatedString, str] = '', max_datetime: Union[InterpolatedString, str] = '')
datetime: Union[InterpolatedString, str]
parameters: dataclasses.InitVar[typing.Mapping[str, typing.Any]]
datetime_format: str
91    @property  # type: ignore # properties don't play well with dataclasses...
92    def datetime_format(self) -> str:
93        """The format of the string representing the datetime"""
94        return self._datetime_format

The format of the string representing the datetime

min_datetime: Union[InterpolatedString, str] = ''
max_datetime: Union[InterpolatedString, str] = ''
def get_datetime( self, config: Mapping[str, Any], **additional_parameters: Mapping[str, Any]) -> datetime.datetime:
55    def get_datetime(
56        self, config: Mapping[str, Any], **additional_parameters: Mapping[str, Any]
57    ) -> dt.datetime:
58        """
59        Evaluates and returns the datetime
60        :param config: The user-provided configuration as specified by the source's spec
61        :param additional_parameters: Additional arguments to be passed to the strings for interpolation
62        :return: The evaluated datetime
63        """
64        # We apply a default datetime format here instead of at instantiation, so it can be set by the parent first
65        datetime_format = self._datetime_format
66        if not datetime_format:
67            datetime_format = "%Y-%m-%dT%H:%M:%S.%f%z"
68
69        time = self._parser.parse(
70            str(
71                self.datetime.eval(  # type: ignore[union-attr] # str has no attribute "eval"
72                    config,
73                    **additional_parameters,
74                )
75            ),
76            datetime_format,
77        )  # type: ignore # datetime is always cast to an interpolated string
78
79        if self.min_datetime:
80            min_time = str(self.min_datetime.eval(config, **additional_parameters))  # type: ignore # min_datetime is always cast to an interpolated string
81            if min_time:
82                min_datetime = self._parser.parse(min_time, datetime_format)  # type: ignore # min_datetime is always cast to an interpolated string
83                time = max(time, min_datetime)
84        if self.max_datetime:
85            max_time = str(self.max_datetime.eval(config, **additional_parameters))  # type: ignore # max_datetime is always cast to an interpolated string
86            if max_time:
87                max_datetime = self._parser.parse(max_time, datetime_format)
88                time = min(time, max_datetime)
89        return time

Evaluates and returns the datetime

Parameters
  • config: The user-provided configuration as specified by the source's spec
  • additional_parameters: Additional arguments to be passed to the strings for interpolation
Returns

The evaluated datetime

@classmethod
def create( cls, interpolated_string_or_min_max_datetime: Union[InterpolatedString, str, MinMaxDatetime], parameters: Optional[Mapping[str, Any]] = None) -> MinMaxDatetime:
104    @classmethod
105    def create(
106        cls,
107        interpolated_string_or_min_max_datetime: Union[InterpolatedString, str, "MinMaxDatetime"],
108        parameters: Optional[Mapping[str, Any]] = None,
109    ) -> "MinMaxDatetime":
110        if parameters is None:
111            parameters = {}
112        if isinstance(interpolated_string_or_min_max_datetime, InterpolatedString) or isinstance(
113            interpolated_string_or_min_max_datetime, str
114        ):
115            return MinMaxDatetime(  # type: ignore [call-arg]
116                datetime=interpolated_string_or_min_max_datetime, parameters=parameters
117            )
118        else:
119            return interpolated_string_or_min_max_datetime
@dataclass
class NoAuth(airbyte_cdk.DeclarativeAuthenticator):
33@dataclass
34class NoAuth(DeclarativeAuthenticator):
35    parameters: InitVar[Mapping[str, Any]]
36
37    @property
38    def auth_header(self) -> str:
39        return ""
40
41    @property
42    def token(self) -> str:
43        return ""
NoAuth(parameters: dataclasses.InitVar[typing.Mapping[str, typing.Any]])
parameters: dataclasses.InitVar[typing.Mapping[str, typing.Any]]
auth_header: str
37    @property
38    def auth_header(self) -> str:
39        return ""

HTTP header to set on the requests

token: str
41    @property
42    def token(self) -> str:
43        return ""

The header value to set on outgoing HTTP requests

@dataclass
class OffsetIncrement(airbyte_cdk.PaginationStrategy):
 25@dataclass
 26class OffsetIncrement(PaginationStrategy):
 27    """
 28    Pagination strategy that returns the number of records reads so far and returns it as the next page token
 29    Examples:
 30        # page_size to be a constant integer value
 31        pagination_strategy:
 32          type: OffsetIncrement
 33          page_size: 2
 34
 35        # page_size to be a constant string value
 36        pagination_strategy:
 37          type: OffsetIncrement
 38          page_size: "2"
 39
 40        # page_size to be an interpolated string value
 41        pagination_strategy:
 42          type: OffsetIncrement
 43          page_size: "{{ parameters['items_per_page'] }}"
 44
 45    Attributes:
 46        page_size (InterpolatedString): the number of records to request
 47        extractor (Optional[RecordExtractor]): counts the records of a page by extracting them
 48            again. Leave it unset to use the count of the `RecordSelector` that read the page
 49    """
 50
 51    config: Config
 52    page_size: Optional[Union[str, int]]
 53    parameters: InitVar[Mapping[str, Any]]
 54    extractor: Optional[RecordExtractor] = None
 55    decoder: Decoder = field(
 56        default_factory=lambda: PaginationDecoderDecorator(decoder=JsonDecoder(parameters={}))
 57    )
 58    inject_on_first_request: bool = False
 59
 60    def __post_init__(self, parameters: Mapping[str, Any]) -> None:
 61        page_size = str(self.page_size) if isinstance(self.page_size, int) else self.page_size
 62        if page_size:
 63            self._page_size: Optional[InterpolatedString] = InterpolatedString(
 64                page_size, parameters=parameters
 65            )
 66        else:
 67            self._page_size = None
 68
 69    @property
 70    def initial_token(self) -> Optional[Any]:
 71        if self.inject_on_first_request:
 72            return 0
 73        return None
 74
 75    def next_page_token(
 76        self,
 77        response: requests.Response,
 78        last_page_size: int,
 79        last_record: Optional[Record],
 80        last_page_token_value: Optional[Any] = None,
 81        page_size_override: Optional[int] = None,
 82        stream_slice: Optional[StreamSlice] = None,
 83    ) -> Optional[Any]:
 84        # The records the API returned, not the ones emitted: a record filter must neither end the
 85        # pagination nor shorten the offset.
 86        if self.extractor:
 87            last_page_size = len(list(self.extractor.extract_records(response=response)))
 88        else:
 89            last_page_size = extracted_record_count(response, default=last_page_size)
 90
 91        # Stop paginating when there are fewer records than the page size or the current page has no records.
 92        # The comparison uses the page size that was actually requested: after a `REDUCE_PAGE_SIZE` reduction, a
 93        # full page is smaller than the configured page size and comparing against the latter would end the
 94        # pagination early and skip records.
 95        if page_size_override is not None:
 96            requested_page_size: Optional[int] = page_size_override
 97        elif isinstance(self.page_size, int):
 98            # A constant needs no response, which would otherwise be decoded a second time.
 99            requested_page_size = self.page_size
100        elif self._page_size:
101            requested_page_size = self._page_size.eval(
102                self.config, response=next(self.decoder.decode(response))
103            )
104        else:
105            requested_page_size = None
106        if (requested_page_size and last_page_size < requested_page_size) or last_page_size == 0:
107            return None
108        elif last_page_token_value is None:
109            # If the OffsetIncrement strategy does not inject on the first request, the incoming last_page_token_value
110            # will be None. For this case, we assume that None was the first page and progress to the next offset
111            return 0 + last_page_size
112        elif not isinstance(last_page_token_value, int):
113            raise ValueError(
114                f"Last page token value {last_page_token_value} for OffsetIncrement pagination strategy was not an integer"
115            )
116        else:
117            return last_page_token_value + last_page_size
118
119    def get_page_size(self) -> Optional[int]:
120        if self._page_size:
121            page_size = self._page_size.eval(self.config)
122            if not isinstance(page_size, int):
123                raise Exception(f"{page_size} is of type {type(page_size)}. Expected {int}")
124            return page_size
125        else:
126            return None

Pagination strategy that returns the number of records reads so far and returns it as the next page token

Examples:

page_size to be a constant integer value

pagination_strategy: type: OffsetIncrement page_size: 2

page_size to be a constant string value

pagination_strategy: type: OffsetIncrement page_size: "2"

page_size to be an interpolated string value

pagination_strategy: type: OffsetIncrement page_size: "{{ parameters['items_per_page'] }}"

Attributes:
  • page_size (InterpolatedString): the number of records to request
  • extractor (Optional[RecordExtractor]): counts the records of a page by extracting them again. Leave it unset to use the count of the RecordSelector that read the page
OffsetIncrement( config: Mapping[str, Any], page_size: Union[str, int, NoneType], parameters: dataclasses.InitVar[typing.Mapping[str, typing.Any]], extractor: Optional[RecordExtractor] = None, decoder: Decoder = <factory>, inject_on_first_request: bool = False)
config: Mapping[str, Any]
page_size: Union[str, int, NoneType]
parameters: dataclasses.InitVar[typing.Mapping[str, typing.Any]]
extractor: Optional[RecordExtractor] = None
decoder: Decoder
inject_on_first_request: bool = False
initial_token: Optional[Any]
69    @property
70    def initial_token(self) -> Optional[Any]:
71        if self.inject_on_first_request:
72            return 0
73        return None

Return the initial value of the token

def next_page_token( self, response: requests.models.Response, last_page_size: int, last_record: Optional[Record], last_page_token_value: Optional[Any] = None, page_size_override: Optional[int] = None, stream_slice: Optional[StreamSlice] = None) -> Optional[Any]:
 75    def next_page_token(
 76        self,
 77        response: requests.Response,
 78        last_page_size: int,
 79        last_record: Optional[Record],
 80        last_page_token_value: Optional[Any] = None,
 81        page_size_override: Optional[int] = None,
 82        stream_slice: Optional[StreamSlice] = None,
 83    ) -> Optional[Any]:
 84        # The records the API returned, not the ones emitted: a record filter must neither end the
 85        # pagination nor shorten the offset.
 86        if self.extractor:
 87            last_page_size = len(list(self.extractor.extract_records(response=response)))
 88        else:
 89            last_page_size = extracted_record_count(response, default=last_page_size)
 90
 91        # Stop paginating when there are fewer records than the page size or the current page has no records.
 92        # The comparison uses the page size that was actually requested: after a `REDUCE_PAGE_SIZE` reduction, a
 93        # full page is smaller than the configured page size and comparing against the latter would end the
 94        # pagination early and skip records.
 95        if page_size_override is not None:
 96            requested_page_size: Optional[int] = page_size_override
 97        elif isinstance(self.page_size, int):
 98            # A constant needs no response, which would otherwise be decoded a second time.
 99            requested_page_size = self.page_size
100        elif self._page_size:
101            requested_page_size = self._page_size.eval(
102                self.config, response=next(self.decoder.decode(response))
103            )
104        else:
105            requested_page_size = None
106        if (requested_page_size and last_page_size < requested_page_size) or last_page_size == 0:
107            return None
108        elif last_page_token_value is None:
109            # If the OffsetIncrement strategy does not inject on the first request, the incoming last_page_token_value
110            # will be None. For this case, we assume that None was the first page and progress to the next offset
111            return 0 + last_page_size
112        elif not isinstance(last_page_token_value, int):
113            raise ValueError(
114                f"Last page token value {last_page_token_value} for OffsetIncrement pagination strategy was not an integer"
115            )
116        else:
117            return last_page_token_value + last_page_size
Parameters
  • response: response to process
  • last_page_size: the number of records read from the response
  • last_record: the last record extracted from the response
  • last_page_token_value: The current value of the page token made on the last request
  • page_size_override: the page size that was actually requested, when it differs from the configured one because of a REDUCE_PAGE_SIZE response action. Strategies that use their configured page size as a stop condition must honor this value, else they end the pagination early and skip records.
  • stream_slice: the slice the page was read for. Only passed to strategies that declare it, so existing strategies defined outside of the CDK keep working unchanged.
Returns

next page token. Returns None if there are no more pages to fetch

def get_page_size(self) -> Optional[int]:
119    def get_page_size(self) -> Optional[int]:
120        if self._page_size:
121            page_size = self._page_size.eval(self.config)
122            if not isinstance(page_size, int):
123                raise Exception(f"{page_size} is of type {type(page_size)}. Expected {int}")
124            return page_size
125        else:
126            return None
Returns

page size: The number of records to fetch in a page. Returns None if unspecified

@dataclass
class PageIncrement(airbyte_cdk.PaginationStrategy):
 22@dataclass
 23class PageIncrement(PaginationStrategy):
 24    """
 25    Pagination strategy that returns the number of pages reads so far and returns it as the next page token
 26
 27    Attributes:
 28        page_size (int): the number of records to request
 29        start_from_page (int): number of the initial page
 30        extractor (Optional[RecordExtractor]): counts the records of a page by extracting them
 31            again. Leave it unset to use the count of the `RecordSelector` that read the page
 32    """
 33
 34    config: Config
 35    page_size: Optional[Union[str, int]]
 36    parameters: InitVar[Mapping[str, Any]]
 37    start_from_page: int = 0
 38    inject_on_first_request: bool = False
 39    extractor: Optional[RecordExtractor] = None
 40
 41    def __post_init__(self, parameters: Mapping[str, Any]) -> None:
 42        if isinstance(self.page_size, int) or (self.page_size is None):
 43            self._page_size = self.page_size
 44        else:
 45            page_size = InterpolatedString(self.page_size, parameters=parameters).eval(self.config)
 46            if not isinstance(page_size, int):
 47                raise Exception(f"{page_size} is of type {type(page_size)}. Expected {int}")
 48            self._page_size = page_size
 49
 50    @property
 51    def initial_token(self) -> Optional[Any]:
 52        if self.inject_on_first_request:
 53            return self.start_from_page
 54        return None
 55
 56    def next_page_token(
 57        self,
 58        response: requests.Response,
 59        last_page_size: int,
 60        last_record: Optional[Record],
 61        last_page_token_value: Optional[Any],
 62        page_size_override: Optional[int] = None,
 63        stream_slice: Optional[StreamSlice] = None,
 64    ) -> Optional[Any]:
 65        if page_size_override is not None:
 66            # Reachable only when the factory is bypassed: a manifest naming PageIncrement, and a
 67            # CustomPaginationStrategy that subclasses it, are both rejected at config time. An unrecognized
 68            # ValueError here would be reported as a generic system error even though the message describes a
 69            # configuration mistake. `next_page_token` runs after the page's records have been emitted, so an
 70            # out-of-tree caller gets this mid-stream rather than at startup - still better than silently
 71            # ignoring the override and skipping records.
 72            raise AirbyteTracedException(
 73                internal_message="PageIncrement received a page_size_override",
 74                message="PageIncrement does not support reducing the page size while paginating: pages are "
 75                "addressed as page number * page size, so a smaller page size shifts every following page "
 76                "boundary and would skip records. Use OffsetIncrement or CursorPagination instead.",
 77                failure_type=FailureType.config_error,
 78            )
 79
 80        # The record count is dependent on the records returned from the response which may not always
 81        # align with the size of pages emitted. For example, a record filter can reduce the number of
 82        # records observed below the page size even though the API returned a full page.
 83        if self.extractor:
 84            last_page_size = len(list(self.extractor.extract_records(response=response)))
 85        else:
 86            last_page_size = extracted_record_count(response, default=last_page_size)
 87
 88        # Stop paginating when there are fewer records than the page size or the current page has no records
 89        if (self._page_size and last_page_size < self._page_size) or last_page_size == 0:
 90            return None
 91        elif last_page_token_value is None:
 92            # If the PageIncrement strategy does not inject on the first request, the incoming last_page_token_value
 93            # may be None. When this is the case, we assume we've already requested the first page specified by
 94            # start_from_page and must now get the next page
 95            return self.start_from_page + 1
 96        elif not isinstance(last_page_token_value, int):
 97            raise ValueError(
 98                f"Last page token value {last_page_token_value} for PageIncrement pagination strategy was not an integer"
 99            )
100        else:
101            return last_page_token_value + 1
102
103    def get_page_size(self) -> Optional[int]:
104        return self._page_size

Pagination strategy that returns the number of pages reads so far and returns it as the next page token

Attributes:
  • page_size (int): the number of records to request
  • start_from_page (int): number of the initial page
  • extractor (Optional[RecordExtractor]): counts the records of a page by extracting them again. Leave it unset to use the count of the RecordSelector that read the page
PageIncrement( config: Mapping[str, Any], page_size: Union[str, int, NoneType], parameters: dataclasses.InitVar[typing.Mapping[str, typing.Any]], start_from_page: int = 0, inject_on_first_request: bool = False, extractor: Optional[RecordExtractor] = None)
config: Mapping[str, Any]
page_size: Union[str, int, NoneType]
parameters: dataclasses.InitVar[typing.Mapping[str, typing.Any]]
start_from_page: int = 0
inject_on_first_request: bool = False
extractor: Optional[RecordExtractor] = None
initial_token: Optional[Any]
50    @property
51    def initial_token(self) -> Optional[Any]:
52        if self.inject_on_first_request:
53            return self.start_from_page
54        return None

Return the initial value of the token

def next_page_token( self, response: requests.models.Response, last_page_size: int, last_record: Optional[Record], last_page_token_value: Optional[Any], page_size_override: Optional[int] = None, stream_slice: Optional[StreamSlice] = None) -> Optional[Any]:
 56    def next_page_token(
 57        self,
 58        response: requests.Response,
 59        last_page_size: int,
 60        last_record: Optional[Record],
 61        last_page_token_value: Optional[Any],
 62        page_size_override: Optional[int] = None,
 63        stream_slice: Optional[StreamSlice] = None,
 64    ) -> Optional[Any]:
 65        if page_size_override is not None:
 66            # Reachable only when the factory is bypassed: a manifest naming PageIncrement, and a
 67            # CustomPaginationStrategy that subclasses it, are both rejected at config time. An unrecognized
 68            # ValueError here would be reported as a generic system error even though the message describes a
 69            # configuration mistake. `next_page_token` runs after the page's records have been emitted, so an
 70            # out-of-tree caller gets this mid-stream rather than at startup - still better than silently
 71            # ignoring the override and skipping records.
 72            raise AirbyteTracedException(
 73                internal_message="PageIncrement received a page_size_override",
 74                message="PageIncrement does not support reducing the page size while paginating: pages are "
 75                "addressed as page number * page size, so a smaller page size shifts every following page "
 76                "boundary and would skip records. Use OffsetIncrement or CursorPagination instead.",
 77                failure_type=FailureType.config_error,
 78            )
 79
 80        # The record count is dependent on the records returned from the response which may not always
 81        # align with the size of pages emitted. For example, a record filter can reduce the number of
 82        # records observed below the page size even though the API returned a full page.
 83        if self.extractor:
 84            last_page_size = len(list(self.extractor.extract_records(response=response)))
 85        else:
 86            last_page_size = extracted_record_count(response, default=last_page_size)
 87
 88        # Stop paginating when there are fewer records than the page size or the current page has no records
 89        if (self._page_size and last_page_size < self._page_size) or last_page_size == 0:
 90            return None
 91        elif last_page_token_value is None:
 92            # If the PageIncrement strategy does not inject on the first request, the incoming last_page_token_value
 93            # may be None. When this is the case, we assume we've already requested the first page specified by
 94            # start_from_page and must now get the next page
 95            return self.start_from_page + 1
 96        elif not isinstance(last_page_token_value, int):
 97            raise ValueError(
 98                f"Last page token value {last_page_token_value} for PageIncrement pagination strategy was not an integer"
 99            )
100        else:
101            return last_page_token_value + 1
Parameters
  • response: response to process
  • last_page_size: the number of records read from the response
  • last_record: the last record extracted from the response
  • last_page_token_value: The current value of the page token made on the last request
  • page_size_override: the page size that was actually requested, when it differs from the configured one because of a REDUCE_PAGE_SIZE response action. Strategies that use their configured page size as a stop condition must honor this value, else they end the pagination early and skip records.
  • stream_slice: the slice the page was read for. Only passed to strategies that declare it, so existing strategies defined outside of the CDK keep working unchanged.
Returns

next page token. Returns None if there are no more pages to fetch

def get_page_size(self) -> Optional[int]:
103    def get_page_size(self) -> Optional[int]:
104        return self._page_size
Returns

page size: The number of records to fetch in a page. Returns None if unspecified

@dataclass
class PaginationStrategy:
15@dataclass
16class PaginationStrategy:
17    """
18    Defines how to get the next page token
19
20    One instance paginates every partition of the stream, several at once, so `next_page_token` must derive the
21    next token from its arguments, such as `last_page_token_value`, rather than from state kept on the instance.
22    """
23
24    @property
25    @abstractmethod
26    def initial_token(self) -> Optional[Any]:
27        """
28        Return the initial value of the token
29        """
30
31    @abstractmethod
32    def next_page_token(
33        self,
34        response: requests.Response,
35        last_page_size: int,
36        last_record: Optional[Record],
37        last_page_token_value: Optional[Any],
38        page_size_override: Optional[int] = None,
39        stream_slice: Optional[StreamSlice] = None,
40    ) -> Optional[Any]:
41        """
42        :param response: response to process
43        :param last_page_size: the number of records read from the response
44        :param last_record: the last record extracted from the response
45        :param last_page_token_value: The current value of the page token made on the last request
46        :param page_size_override: the page size that was actually requested, when it differs from the configured
47            one because of a `REDUCE_PAGE_SIZE` response action. Strategies that use their configured page size as
48            a stop condition must honor this value, else they end the pagination early and skip records.
49        :param stream_slice: the slice the page was read for. Only passed to strategies that declare it, so existing
50            strategies defined outside of the CDK keep working unchanged.
51        :return: next page token. Returns None if there are no more pages to fetch
52        """
53        pass
54
55    @abstractmethod
56    def get_page_size(self) -> Optional[int]:
57        """
58        :return: page size: The number of records to fetch in a page. Returns None if unspecified
59        """

Defines how to get the next page token

One instance paginates every partition of the stream, several at once, so next_page_token must derive the next token from its arguments, such as last_page_token_value, rather than from state kept on the instance.

initial_token: Optional[Any]
24    @property
25    @abstractmethod
26    def initial_token(self) -> Optional[Any]:
27        """
28        Return the initial value of the token
29        """

Return the initial value of the token

@abstractmethod
def next_page_token( self, response: requests.models.Response, last_page_size: int, last_record: Optional[Record], last_page_token_value: Optional[Any], page_size_override: Optional[int] = None, stream_slice: Optional[StreamSlice] = None) -> Optional[Any]:
31    @abstractmethod
32    def next_page_token(
33        self,
34        response: requests.Response,
35        last_page_size: int,
36        last_record: Optional[Record],
37        last_page_token_value: Optional[Any],
38        page_size_override: Optional[int] = None,
39        stream_slice: Optional[StreamSlice] = None,
40    ) -> Optional[Any]:
41        """
42        :param response: response to process
43        :param last_page_size: the number of records read from the response
44        :param last_record: the last record extracted from the response
45        :param last_page_token_value: The current value of the page token made on the last request
46        :param page_size_override: the page size that was actually requested, when it differs from the configured
47            one because of a `REDUCE_PAGE_SIZE` response action. Strategies that use their configured page size as
48            a stop condition must honor this value, else they end the pagination early and skip records.
49        :param stream_slice: the slice the page was read for. Only passed to strategies that declare it, so existing
50            strategies defined outside of the CDK keep working unchanged.
51        :return: next page token. Returns None if there are no more pages to fetch
52        """
53        pass
Parameters
  • response: response to process
  • last_page_size: the number of records read from the response
  • last_record: the last record extracted from the response
  • last_page_token_value: The current value of the page token made on the last request
  • page_size_override: the page size that was actually requested, when it differs from the configured one because of a REDUCE_PAGE_SIZE response action. Strategies that use their configured page size as a stop condition must honor this value, else they end the pagination early and skip records.
  • stream_slice: the slice the page was read for. Only passed to strategies that declare it, so existing strategies defined outside of the CDK keep working unchanged.
Returns

next page token. Returns None if there are no more pages to fetch

@abstractmethod
def get_page_size(self) -> Optional[int]:
55    @abstractmethod
56    def get_page_size(self) -> Optional[int]:
57        """
58        :return: page size: The number of records to fetch in a page. Returns None if unspecified
59        """
Returns

page size: The number of records to fetch in a page. Returns None if unspecified

@dataclass
class ParentStreamConfig:
 59@dataclass
 60class ParentStreamConfig:
 61    """
 62    Describes how to create a stream slice from a parent stream
 63
 64    stream: The stream to read records from
 65    parent_key: The key of the parent stream's records that will be the stream slice key
 66    partition_field: The partition key
 67    extra_fields: Additional field paths to include in the stream slice
 68    request_option: How to inject the slice value on an outgoing HTTP request
 69    incremental_dependency (bool): Indicates if the parent stream should be read incrementally.
 70    """
 71
 72    stream: "AbstractStream"
 73    parent_key: Union[InterpolatedString, str]
 74    partition_field: Union[InterpolatedString, str]
 75    config: Config
 76    parameters: InitVar[Mapping[str, Any]]
 77    extra_fields: Optional[Union[List[List[str]], List[List[InterpolatedString]]]] = (
 78        None  # List of field paths (arrays of strings)
 79    )
 80    request_option: Optional[RequestOption] = None
 81    incremental_dependency: bool = False
 82    lazy_read_pointer: Optional[List[Union[InterpolatedString, str]]] = None
 83
 84    def __post_init__(self, parameters: Mapping[str, Any]) -> None:
 85        self.parent_key = InterpolatedString.create(self.parent_key, parameters=parameters)
 86        self.partition_field = InterpolatedString.create(
 87            self.partition_field, parameters=parameters
 88        )
 89        if self.extra_fields:
 90            # Create InterpolatedString for each field path in extra_keys
 91            self.extra_fields = [
 92                [InterpolatedString.create(path, parameters=parameters) for path in key_path]
 93                for key_path in self.extra_fields
 94            ]
 95
 96        self.lazy_read_pointer = (
 97            [
 98                InterpolatedString.create(path, parameters=parameters)
 99                if isinstance(path, str)
100                else path
101                for path in self.lazy_read_pointer
102            ]
103            if self.lazy_read_pointer
104            else None
105        )

Describes how to create a stream slice from a parent stream

stream: The stream to read records from parent_key: The key of the parent stream's records that will be the stream slice key partition_field: The partition key extra_fields: Additional field paths to include in the stream slice request_option: How to inject the slice value on an outgoing HTTP request incremental_dependency (bool): Indicates if the parent stream should be read incrementally.

ParentStreamConfig( stream: airbyte_cdk.sources.streams.concurrent.abstract_stream.AbstractStream, parent_key: Union[InterpolatedString, str], partition_field: Union[InterpolatedString, str], config: Mapping[str, Any], parameters: dataclasses.InitVar[typing.Mapping[str, typing.Any]], extra_fields: Union[List[List[str]], List[List[InterpolatedString]], NoneType] = None, request_option: Optional[RequestOption] = None, incremental_dependency: bool = False, lazy_read_pointer: Optional[List[Union[InterpolatedString, str]]] = None)
parent_key: Union[InterpolatedString, str]
partition_field: Union[InterpolatedString, str]
config: Mapping[str, Any]
parameters: dataclasses.InitVar[typing.Mapping[str, typing.Any]]
extra_fields: Union[List[List[str]], List[List[InterpolatedString]], NoneType] = None
request_option: Optional[RequestOption] = None
incremental_dependency: bool = False
lazy_read_pointer: Optional[List[Union[InterpolatedString, str]]] = None
class ReadException(builtins.Exception):
 7class ReadException(Exception):
 8    """
 9    Raise when there is an error reading data from an API Source
10    """

Raise when there is an error reading data from an API Source

@dataclass
class RecordExtractor:
12@dataclass
13class RecordExtractor:
14    """
15    Responsible for translating an HTTP response into a list of records by extracting records from the response.
16    """
17
18    @abstractmethod
19    def extract_records(
20        self,
21        response: requests.Response,
22    ) -> Iterable[Mapping[str, Any]]:
23        """
24        Selects records from the response
25        :param response: The response to extract the records from
26        :return: List of Records extracted from the response
27        """
28        pass

Responsible for translating an HTTP response into a list of records by extracting records from the response.

@abstractmethod
def extract_records(self, response: requests.models.Response) -> Iterable[Mapping[str, Any]]:
18    @abstractmethod
19    def extract_records(
20        self,
21        response: requests.Response,
22    ) -> Iterable[Mapping[str, Any]]:
23        """
24        Selects records from the response
25        :param response: The response to extract the records from
26        :return: List of Records extracted from the response
27        """
28        pass

Selects records from the response

Parameters
  • response: The response to extract the records from
Returns

List of Records extracted from the response

@dataclass
class RecordFilter:
13@dataclass
14class RecordFilter:
15    """
16    Filter applied on a list of Records
17
18    config (Config): The user-provided configuration as specified by the source's spec
19    condition (str): The string representing the predicate to filter a record. Records will be removed if evaluated to False
20    """
21
22    parameters: InitVar[Mapping[str, Any]]
23    config: Config
24    condition: str = ""
25
26    def __post_init__(self, parameters: Mapping[str, Any]) -> None:
27        self._filter_interpolator = InterpolatedBoolean(
28            condition=self.condition, parameters=parameters
29        )
30
31    def filter_records(
32        self,
33        records: Iterable[Mapping[str, Any]],
34        stream_state: StreamState,
35        stream_slice: Optional[StreamSlice] = None,
36        next_page_token: Optional[Mapping[str, Any]] = None,
37    ) -> Iterable[Mapping[str, Any]]:
38        kwargs = {
39            "stream_state": stream_state,
40            "stream_slice": stream_slice,
41            "next_page_token": next_page_token,
42            "stream_slice.extra_fields": stream_slice.extra_fields if stream_slice else {},
43        }
44        for record in records:
45            if self._filter_interpolator.eval(self.config, record=record, **kwargs):
46                yield record

Filter applied on a list of Records

config (Config): The user-provided configuration as specified by the source's spec condition (str): The string representing the predicate to filter a record. Records will be removed if evaluated to False

RecordFilter( parameters: dataclasses.InitVar[typing.Mapping[str, typing.Any]], config: Mapping[str, Any], condition: str = '')
parameters: dataclasses.InitVar[typing.Mapping[str, typing.Any]]
config: Mapping[str, Any]
condition: str = ''
def filter_records( self, records: Iterable[Mapping[str, Any]], stream_state: Mapping[str, Any], stream_slice: Optional[StreamSlice] = None, next_page_token: Optional[Mapping[str, Any]] = None) -> Iterable[Mapping[str, Any]]:
31    def filter_records(
32        self,
33        records: Iterable[Mapping[str, Any]],
34        stream_state: StreamState,
35        stream_slice: Optional[StreamSlice] = None,
36        next_page_token: Optional[Mapping[str, Any]] = None,
37    ) -> Iterable[Mapping[str, Any]]:
38        kwargs = {
39            "stream_state": stream_state,
40            "stream_slice": stream_slice,
41            "next_page_token": next_page_token,
42            "stream_slice.extra_fields": stream_slice.extra_fields if stream_slice else {},
43        }
44        for record in records:
45            if self._filter_interpolator.eval(self.config, record=record, **kwargs):
46                yield record
@dataclass
class RecordSelector(airbyte_cdk.sources.declarative.extractors.http_selector.HttpSelector):
 39@dataclass
 40class RecordSelector(HttpSelector):
 41    """
 42    Responsible for translating an HTTP response into a list of records by extracting records from the response and optionally filtering
 43    records based on a heuristic.
 44
 45    Attributes:
 46        extractor (RecordExtractor): The record extractor responsible for extracting records from a response
 47        schema_normalization (TypeTransformer): The record normalizer responsible for casting record values to stream schema types
 48        record_filter (RecordFilter): The record filter responsible for filtering extracted records
 49        transformations (List[RecordTransformation]): The transformations to be done on the records
 50    """
 51
 52    extractor: RecordExtractor
 53    config: Config
 54    parameters: InitVar[Mapping[str, Any]]
 55    schema_normalization: Union[TypeTransformer, DeclarativeTypeTransformer]
 56    name: str
 57    _name: Union[InterpolatedString, str] = field(init=False, repr=False, default="")
 58    record_filter: Optional[RecordFilter] = None
 59    transformations: List[RecordTransformation] = field(default_factory=lambda: [])
 60    transform_before_filtering: bool = False
 61    file_uploader: Optional[DefaultFileUploader] = None
 62
 63    def __post_init__(self, parameters: Mapping[str, Any]) -> None:
 64        self._parameters = parameters
 65        self._name = (
 66            InterpolatedString(self._name, parameters=parameters)
 67            if isinstance(self._name, str)
 68            else self._name
 69        )
 70
 71    @property  # type: ignore
 72    def name(self) -> str:
 73        """
 74        :return: Stream name
 75        """
 76        return (
 77            str(self._name.eval(self.config))
 78            if isinstance(self._name, InterpolatedString)
 79            else self._name
 80        )
 81
 82    @name.setter
 83    def name(self, value: str) -> None:
 84        if not isinstance(value, property):
 85            self._name = value
 86
 87    def select_records(
 88        self,
 89        response: requests.Response,
 90        stream_state: StreamState,
 91        records_schema: Mapping[str, Any],
 92        stream_slice: Optional[StreamSlice] = None,
 93        next_page_token: Optional[Mapping[str, Any]] = None,
 94    ) -> Iterable[Record]:
 95        """
 96        Selects records from the response
 97        :param response: The response to select the records from
 98        :param stream_state: The stream state
 99        :param records_schema: json schema of records to return
100        :param stream_slice: The stream slice
101        :param next_page_token: The paginator token
102        :return: List of Records selected from the response
103        """
104        all_data: Iterable[Mapping[str, Any]] = self._extract_and_count(response)
105        yield from self.filter_and_transform(
106            all_data, stream_state, records_schema, stream_slice, next_page_token
107        )
108        # A custom record filter can stop reading before the end of the page, which would leave the
109        # page uncounted.
110        for _ in all_data:
111            pass
112
113    def _extract_and_count(self, response: requests.Response) -> Iterable[Mapping[str, Any]]:
114        count = 0
115        for record in self.extractor.extract_records(response):
116            count += 1
117            if not isinstance(record, _DroppedEmptyRecord):
118                yield record
119        # Kept on the response, not on the selector: a stream has one retriever, and so one
120        # selector, reading all of its partitions concurrently.
121        setattr(response, _EXTRACTED_RECORD_COUNT, count)
122
123    def filter_and_transform(
124        self,
125        all_data: Iterable[Mapping[str, Any]],
126        stream_state: StreamState,
127        records_schema: Mapping[str, Any],
128        stream_slice: Optional[StreamSlice] = None,
129        next_page_token: Optional[Mapping[str, Any]] = None,
130    ) -> Iterable[Record]:
131        """
132        There is an issue with the selector as of 2024-08-30: it does technology-agnostic processing like filtering, transformation and
133        normalization with an API that is technology-specific (as requests.Response is only for HTTP communication using the requests
134        library).
135
136        Until we decide to move this logic away from the selector, we made this method public so that users like AsyncJobRetriever could
137        share the logic of doing transformations on a set of records.
138        """
139        if self.transform_before_filtering:
140            transformed_data = self._transform(all_data, stream_state, stream_slice)
141            transformed_filtered_data = self._filter(
142                transformed_data, stream_state, stream_slice, next_page_token
143            )
144        else:
145            filtered_data = self._filter(all_data, stream_state, stream_slice, next_page_token)
146            transformed_filtered_data = self._transform(filtered_data, stream_state, stream_slice)
147        normalized_data = self._normalize_by_schema(
148            transformed_filtered_data, schema=records_schema
149        )
150        for data in normalized_data:
151            record = Record(data=data, stream_name=self.name, associated_slice=stream_slice)
152            if self.file_uploader:
153                self.file_uploader.upload(record)
154            yield record
155
156    def _normalize_by_schema(
157        self, records: Iterable[Mapping[str, Any]], schema: Optional[Mapping[str, Any]]
158    ) -> Iterable[Mapping[str, Any]]:
159        if schema:
160            # record has type Mapping[str, Any], but dict[str, Any] expected
161            for record in records:
162                normalized_record = dict(record)
163                self.schema_normalization.transform(normalized_record, schema)
164                yield normalized_record
165        else:
166            yield from records
167
168    def _filter(
169        self,
170        records: Iterable[Mapping[str, Any]],
171        stream_state: StreamState,
172        stream_slice: Optional[StreamSlice],
173        next_page_token: Optional[Mapping[str, Any]],
174    ) -> Iterable[Mapping[str, Any]]:
175        if self.record_filter:
176            yield from self.record_filter.filter_records(
177                records,
178                stream_state=stream_state,
179                stream_slice=stream_slice,
180                next_page_token=next_page_token,
181            )
182        else:
183            yield from records
184
185    def _transform(
186        self,
187        records: Iterable[Mapping[str, Any]],
188        stream_state: StreamState,
189        stream_slice: Optional[StreamSlice] = None,
190    ) -> Iterable[Mapping[str, Any]]:
191        for record in records:
192            for transformation in self.transformations:
193                transformation.transform(
194                    record,  # type: ignore  # record has type Mapping[str, Any], but Dict[str, Any] expected
195                    config=self.config,
196                    stream_state=stream_state,
197                    stream_slice=stream_slice,
198                )
199            yield record

Responsible for translating an HTTP response into a list of records by extracting records from the response and optionally filtering records based on a heuristic.

Attributes:
  • extractor (RecordExtractor): The record extractor responsible for extracting records from a response
  • schema_normalization (TypeTransformer): The record normalizer responsible for casting record values to stream schema types
  • record_filter (RecordFilter): The record filter responsible for filtering extracted records
  • transformations (List[RecordTransformation]): The transformations to be done on the records
RecordSelector( extractor: RecordExtractor, config: Mapping[str, Any], parameters: dataclasses.InitVar[typing.Mapping[str, typing.Any]], schema_normalization: Union[TypeTransformer, airbyte_cdk.sources.declarative.extractors.TypeTransformer], name: str = <property object>, record_filter: Optional[RecordFilter] = None, transformations: List[RecordTransformation] = <factory>, transform_before_filtering: bool = False, file_uploader: Optional[airbyte_cdk.sources.declarative.retrievers.file_uploader.DefaultFileUploader] = None)
extractor: RecordExtractor
config: Mapping[str, Any]
parameters: dataclasses.InitVar[typing.Mapping[str, typing.Any]]
name: str
71    @property  # type: ignore
72    def name(self) -> str:
73        """
74        :return: Stream name
75        """
76        return (
77            str(self._name.eval(self.config))
78            if isinstance(self._name, InterpolatedString)
79            else self._name
80        )
Returns

Stream name

record_filter: Optional[RecordFilter] = None
transformations: List[RecordTransformation]
transform_before_filtering: bool = False
def select_records( self, response: requests.models.Response, stream_state: Mapping[str, Any], records_schema: Mapping[str, Any], stream_slice: Optional[StreamSlice] = None, next_page_token: Optional[Mapping[str, Any]] = None) -> Iterable[Record]:
 87    def select_records(
 88        self,
 89        response: requests.Response,
 90        stream_state: StreamState,
 91        records_schema: Mapping[str, Any],
 92        stream_slice: Optional[StreamSlice] = None,
 93        next_page_token: Optional[Mapping[str, Any]] = None,
 94    ) -> Iterable[Record]:
 95        """
 96        Selects records from the response
 97        :param response: The response to select the records from
 98        :param stream_state: The stream state
 99        :param records_schema: json schema of records to return
100        :param stream_slice: The stream slice
101        :param next_page_token: The paginator token
102        :return: List of Records selected from the response
103        """
104        all_data: Iterable[Mapping[str, Any]] = self._extract_and_count(response)
105        yield from self.filter_and_transform(
106            all_data, stream_state, records_schema, stream_slice, next_page_token
107        )
108        # A custom record filter can stop reading before the end of the page, which would leave the
109        # page uncounted.
110        for _ in all_data:
111            pass

Selects records from the response

Parameters
  • response: The response to select the records from
  • stream_state: The stream state
  • records_schema: json schema of records to return
  • stream_slice: The stream slice
  • next_page_token: The paginator token
Returns

List of Records selected from the response

def filter_and_transform( self, all_data: Iterable[Mapping[str, Any]], stream_state: Mapping[str, Any], records_schema: Mapping[str, Any], stream_slice: Optional[StreamSlice] = None, next_page_token: Optional[Mapping[str, Any]] = None) -> Iterable[Record]:
123    def filter_and_transform(
124        self,
125        all_data: Iterable[Mapping[str, Any]],
126        stream_state: StreamState,
127        records_schema: Mapping[str, Any],
128        stream_slice: Optional[StreamSlice] = None,
129        next_page_token: Optional[Mapping[str, Any]] = None,
130    ) -> Iterable[Record]:
131        """
132        There is an issue with the selector as of 2024-08-30: it does technology-agnostic processing like filtering, transformation and
133        normalization with an API that is technology-specific (as requests.Response is only for HTTP communication using the requests
134        library).
135
136        Until we decide to move this logic away from the selector, we made this method public so that users like AsyncJobRetriever could
137        share the logic of doing transformations on a set of records.
138        """
139        if self.transform_before_filtering:
140            transformed_data = self._transform(all_data, stream_state, stream_slice)
141            transformed_filtered_data = self._filter(
142                transformed_data, stream_state, stream_slice, next_page_token
143            )
144        else:
145            filtered_data = self._filter(all_data, stream_state, stream_slice, next_page_token)
146            transformed_filtered_data = self._transform(filtered_data, stream_state, stream_slice)
147        normalized_data = self._normalize_by_schema(
148            transformed_filtered_data, schema=records_schema
149        )
150        for data in normalized_data:
151            record = Record(data=data, stream_name=self.name, associated_slice=stream_slice)
152            if self.file_uploader:
153                self.file_uploader.upload(record)
154            yield record

There is an issue with the selector as of 2024-08-30: it does technology-agnostic processing like filtering, transformation and normalization with an API that is technology-specific (as requests.Response is only for HTTP communication using the requests library).

Until we decide to move this logic away from the selector, we made this method public so that users like AsyncJobRetriever could share the logic of doing transformations on a set of records.

@dataclass
class RecordTransformation:
13@dataclass
14class RecordTransformation:
15    """
16    Implementations of this class define transformations that can be applied to records of a stream.
17    """
18
19    @abstractmethod
20    def transform(
21        self,
22        record: Dict[str, Any],
23        config: Optional[Config] = None,
24        stream_state: Optional[StreamState] = None,
25        stream_slice: Optional[StreamSlice] = None,
26    ) -> None:
27        """
28        Transform a record by adding, deleting, or mutating fields directly from the record reference passed in argument.
29
30        :param record: The input record to be transformed
31        :param config: The user-provided configuration as specified by the source's spec
32        :param stream_state: The stream state
33        :param stream_slice: The stream slice
34        :return: The transformed record
35        """
36
37    def __eq__(self, other: object) -> bool:
38        return other.__dict__ == self.__dict__

Implementations of this class define transformations that can be applied to records of a stream.

@abstractmethod
def transform( self, record: Dict[str, Any], config: Optional[Mapping[str, Any]] = None, stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[StreamSlice] = None) -> None:
19    @abstractmethod
20    def transform(
21        self,
22        record: Dict[str, Any],
23        config: Optional[Config] = None,
24        stream_state: Optional[StreamState] = None,
25        stream_slice: Optional[StreamSlice] = None,
26    ) -> None:
27        """
28        Transform a record by adding, deleting, or mutating fields directly from the record reference passed in argument.
29
30        :param record: The input record to be transformed
31        :param config: The user-provided configuration as specified by the source's spec
32        :param stream_state: The stream state
33        :param stream_slice: The stream slice
34        :return: The transformed record
35        """

Transform a record by adding, deleting, or mutating fields directly from the record reference passed in argument.

Parameters
  • record: The input record to be transformed
  • config: The user-provided configuration as specified by the source's spec
  • stream_state: The stream state
  • stream_slice: The stream slice
Returns

The transformed record

@dataclass
class RequestOption:
 25@dataclass
 26class RequestOption:
 27    """
 28    Describes an option to set on a request
 29
 30    Attributes:
 31        field_name (str): Describes the name of the parameter to inject. Mutually exclusive with field_path.
 32        field_path (list(str)): Describes the path to a nested field as a list of field names.
 33          Only valid for body_json injection type, and mutually exclusive with field_name.
 34        inject_into (RequestOptionType): Describes where in the HTTP request to inject the parameter
 35    """
 36
 37    inject_into: RequestOptionType
 38    parameters: InitVar[Mapping[str, Any]]
 39    field_name: Optional[Union[InterpolatedString, str]] = None
 40    field_path: Optional[List[Union[InterpolatedString, str]]] = None
 41
 42    def __post_init__(self, parameters: Mapping[str, Any]) -> None:
 43        # Validate inputs. We should expect either field_name or field_path, but not both
 44        if self.field_name is None and self.field_path is None:
 45            raise ValueError("RequestOption requires either a field_name or field_path")
 46
 47        if self.field_name is not None and self.field_path is not None:
 48            raise ValueError(
 49                "Only one of field_name or field_path can be provided to RequestOption"
 50            )
 51
 52        # Nested field injection is only supported for body JSON injection
 53        if self.field_path is not None and self.inject_into != RequestOptionType.body_json:
 54            raise ValueError(
 55                "Nested field injection is only supported for body JSON injection. Please use a top-level field_name for other injection types."
 56            )
 57
 58        # Convert field_name and field_path into InterpolatedString objects if they are strings
 59        if self.field_name is not None:
 60            self.field_name = InterpolatedString.create(self.field_name, parameters=parameters)
 61        elif self.field_path is not None:
 62            self.field_path = [
 63                InterpolatedString.create(segment, parameters=parameters)
 64                for segment in self.field_path
 65            ]
 66
 67    @property
 68    def _is_field_path(self) -> bool:
 69        """Returns whether this option is a field path (ie, a nested field)"""
 70        return self.field_path is not None
 71
 72    def inject_into_request(
 73        self,
 74        target: MutableMapping[str, Any],
 75        value: Any,
 76        config: Config,
 77    ) -> None:
 78        """
 79        Inject a request option value into a target request structure using either field_name or field_path.
 80        For non-body-json injection, only top-level field names are supported.
 81        For body-json injection, both field names and nested field paths are supported.
 82
 83        Args:
 84            target: The request structure to inject the value into
 85            value: The value to inject
 86            config: The config object to use for interpolation
 87        """
 88        if self._is_field_path:
 89            if self.inject_into != RequestOptionType.body_json:
 90                raise ValueError(
 91                    "Nested field injection is only supported for body JSON injection. Please use a top-level field_name for other injection types."
 92                )
 93
 94            assert self.field_path is not None  # for type checker
 95            current = target
 96            # Convert path segments into strings, evaluating any interpolated segments
 97            # Example: ["data", "{{ config[user_type] }}", "id"] -> ["data", "admin", "id"]
 98            *path_parts, final_key = [
 99                str(
100                    segment.eval(config=config)
101                    if isinstance(segment, InterpolatedString)
102                    else segment
103                )
104                for segment in self.field_path
105            ]
106
107            # Build a nested dictionary structure and set the final value at the deepest level
108            for part in path_parts:
109                current = current.setdefault(part, {})
110            current[final_key] = value
111        else:
112            # For non-nested fields, evaluate the field name if it's an interpolated string
113            key = (
114                self.field_name.eval(config=config)
115                if isinstance(self.field_name, InterpolatedString)
116                else self.field_name
117            )
118            target[str(key)] = value

Describes an option to set on a request

Attributes:
  • field_name (str): Describes the name of the parameter to inject. Mutually exclusive with field_path.
  • field_path (list(str)): Describes the path to a nested field as a list of field names. Only valid for body_json injection type, and mutually exclusive with field_name.
  • inject_into (RequestOptionType): Describes where in the HTTP request to inject the parameter
RequestOption( inject_into: RequestOptionType, parameters: dataclasses.InitVar[typing.Mapping[str, typing.Any]], field_name: Union[InterpolatedString, str, NoneType] = None, field_path: Optional[List[Union[InterpolatedString, str]]] = None)
inject_into: RequestOptionType
parameters: dataclasses.InitVar[typing.Mapping[str, typing.Any]]
field_name: Union[InterpolatedString, str, NoneType] = None
field_path: Optional[List[Union[InterpolatedString, str]]] = None
def inject_into_request( self, target: MutableMapping[str, Any], value: Any, config: Mapping[str, Any]) -> None:
 72    def inject_into_request(
 73        self,
 74        target: MutableMapping[str, Any],
 75        value: Any,
 76        config: Config,
 77    ) -> None:
 78        """
 79        Inject a request option value into a target request structure using either field_name or field_path.
 80        For non-body-json injection, only top-level field names are supported.
 81        For body-json injection, both field names and nested field paths are supported.
 82
 83        Args:
 84            target: The request structure to inject the value into
 85            value: The value to inject
 86            config: The config object to use for interpolation
 87        """
 88        if self._is_field_path:
 89            if self.inject_into != RequestOptionType.body_json:
 90                raise ValueError(
 91                    "Nested field injection is only supported for body JSON injection. Please use a top-level field_name for other injection types."
 92                )
 93
 94            assert self.field_path is not None  # for type checker
 95            current = target
 96            # Convert path segments into strings, evaluating any interpolated segments
 97            # Example: ["data", "{{ config[user_type] }}", "id"] -> ["data", "admin", "id"]
 98            *path_parts, final_key = [
 99                str(
100                    segment.eval(config=config)
101                    if isinstance(segment, InterpolatedString)
102                    else segment
103                )
104                for segment in self.field_path
105            ]
106
107            # Build a nested dictionary structure and set the final value at the deepest level
108            for part in path_parts:
109                current = current.setdefault(part, {})
110            current[final_key] = value
111        else:
112            # For non-nested fields, evaluate the field name if it's an interpolated string
113            key = (
114                self.field_name.eval(config=config)
115                if isinstance(self.field_name, InterpolatedString)
116                else self.field_name
117            )
118            target[str(key)] = value

Inject a request option value into a target request structure using either field_name or field_path. For non-body-json injection, only top-level field names are supported. For body-json injection, both field names and nested field paths are supported.

Arguments:
  • target: The request structure to inject the value into
  • value: The value to inject
  • config: The config object to use for interpolation
class RequestOptionType(enum.Enum):
14class RequestOptionType(Enum):
15    """
16    Describes where to set a value on a request
17    """
18
19    request_parameter = "request_parameter"
20    header = "header"
21    body_data = "body_data"
22    body_json = "body_json"

Describes where to set a value on a request

request_parameter = <RequestOptionType.request_parameter: 'request_parameter'>
header = <RequestOptionType.header: 'header'>
body_data = <RequestOptionType.body_data: 'body_data'>
body_json = <RequestOptionType.body_json: 'body_json'>
 30class Requester(RequestOptionsProvider):
 31    @abstractmethod
 32    def get_authenticator(self) -> DeclarativeAuthenticator:
 33        """
 34        Specifies the authenticator to use when submitting requests
 35        """
 36        pass
 37
 38    @abstractmethod
 39    def get_url(
 40        self,
 41        *,
 42        stream_state: Optional[StreamState],
 43        stream_slice: Optional[StreamSlice],
 44        next_page_token: Optional[Mapping[str, Any]],
 45    ) -> str:
 46        """
 47        :return: URL base for the  API endpoint e.g: if you wanted to hit https://myapi.com/v1/some_entity then this should return "https://myapi.com/v1/"
 48        """
 49
 50    @abstractmethod
 51    def get_url_base(
 52        self,
 53        *,
 54        stream_state: Optional[StreamState],
 55        stream_slice: Optional[StreamSlice],
 56        next_page_token: Optional[Mapping[str, Any]],
 57    ) -> str:
 58        """
 59        :return: URL base for the  API endpoint e.g: if you wanted to hit https://myapi.com/v1/some_entity then this should return "https://myapi.com/v1/"
 60        """
 61
 62    @abstractmethod
 63    def get_path(
 64        self,
 65        *,
 66        stream_state: Optional[StreamState],
 67        stream_slice: Optional[StreamSlice],
 68        next_page_token: Optional[Mapping[str, Any]],
 69    ) -> str:
 70        """
 71        Returns the URL path for the API endpoint e.g: if you wanted to hit https://myapi.com/v1/some_entity then this should return "some_entity"
 72        """
 73
 74    @abstractmethod
 75    def get_method(self) -> HttpMethod:
 76        """
 77        Specifies the HTTP method to use
 78        """
 79
 80    @abstractmethod
 81    def get_request_params(
 82        self,
 83        *,
 84        stream_state: Optional[StreamState] = None,
 85        stream_slice: Optional[StreamSlice] = None,
 86        next_page_token: Optional[Mapping[str, Any]] = None,
 87    ) -> MutableMapping[str, Any]:
 88        """
 89        Specifies the query parameters that should be set on an outgoing HTTP request given the inputs.
 90
 91        E.g: you might want to define query parameters for paging if next_page_token is not None.
 92        """
 93
 94    @abstractmethod
 95    def get_request_headers(
 96        self,
 97        *,
 98        stream_state: Optional[StreamState] = None,
 99        stream_slice: Optional[StreamSlice] = None,
100        next_page_token: Optional[Mapping[str, Any]] = None,
101    ) -> Mapping[str, Any]:
102        """
103        Return any non-auth headers. Authentication headers will overwrite any overlapping headers returned from this method.
104        """
105
106    @abstractmethod
107    def get_request_body_data(
108        self,
109        *,
110        stream_state: Optional[StreamState] = None,
111        stream_slice: Optional[StreamSlice] = None,
112        next_page_token: Optional[Mapping[str, Any]] = None,
113    ) -> Union[Mapping[str, Any], str]:
114        """
115        Specifies how to populate the body of the request with a non-JSON payload.
116
117        If returns a ready text that it will be sent as is.
118        If returns a dict that it will be converted to a urlencoded form.
119        E.g. {"key1": "value1", "key2": "value2"} => "key1=value1&key2=value2"
120
121        At the same time only one of the 'request_body_data' and 'request_body_json' functions can be overridden.
122        """
123
124    @abstractmethod
125    def get_request_body_json(
126        self,
127        *,
128        stream_state: Optional[StreamState] = None,
129        stream_slice: Optional[StreamSlice] = None,
130        next_page_token: Optional[Mapping[str, Any]] = None,
131    ) -> Mapping[str, Any]:
132        """
133        Specifies how to populate the body of the request with a JSON payload.
134
135        At the same time only one of the 'request_body_data' and 'request_body_json' functions can be overridden.
136        """
137
138    @abstractmethod
139    def send_request(
140        self,
141        stream_state: Optional[StreamState] = None,
142        stream_slice: Optional[StreamSlice] = None,
143        next_page_token: Optional[Mapping[str, Any]] = None,
144        path: Optional[str] = None,
145        request_headers: Optional[Mapping[str, Any]] = None,
146        request_params: Optional[Mapping[str, Any]] = None,
147        request_body_data: Optional[Union[Mapping[str, Any], str]] = None,
148        request_body_json: Optional[Mapping[str, Any]] = None,
149        log_formatter: Optional[Callable[[requests.Response], Any]] = None,
150    ) -> Optional[requests.Response]:
151        """
152        Sends a request and returns the response. Might return no response if the error handler chooses to ignore the response or throw an exception in case of an error.
153        If path is set, the path configured on the requester itself is ignored.
154        If header, params and body are set, they are merged with the ones configured on the requester itself.
155
156        If a log formatter is provided, it's used to log the performed request and response. If it's not provided, no logging is performed.
157        """

Defines the request options to set on an outgoing HTTP request

Options can be passed by

  • request parameter
  • request headers
  • body data
  • json content
@abstractmethod
def get_authenticator( self) -> DeclarativeAuthenticator:
31    @abstractmethod
32    def get_authenticator(self) -> DeclarativeAuthenticator:
33        """
34        Specifies the authenticator to use when submitting requests
35        """
36        pass

Specifies the authenticator to use when submitting requests

@abstractmethod
def get_url( self, *, stream_state: Optional[Mapping[str, Any]], stream_slice: Optional[StreamSlice], next_page_token: Optional[Mapping[str, Any]]) -> str:
38    @abstractmethod
39    def get_url(
40        self,
41        *,
42        stream_state: Optional[StreamState],
43        stream_slice: Optional[StreamSlice],
44        next_page_token: Optional[Mapping[str, Any]],
45    ) -> str:
46        """
47        :return: URL base for the  API endpoint e.g: if you wanted to hit https://myapi.com/v1/some_entity then this should return "https://myapi.com/v1/"
48        """
Returns

URL base for the API endpoint e.g: if you wanted to hit https://myapi.com/v1/some_entity then this should return "https://myapi.com/v1/"

@abstractmethod
def get_url_base( self, *, stream_state: Optional[Mapping[str, Any]], stream_slice: Optional[StreamSlice], next_page_token: Optional[Mapping[str, Any]]) -> str:
50    @abstractmethod
51    def get_url_base(
52        self,
53        *,
54        stream_state: Optional[StreamState],
55        stream_slice: Optional[StreamSlice],
56        next_page_token: Optional[Mapping[str, Any]],
57    ) -> str:
58        """
59        :return: URL base for the  API endpoint e.g: if you wanted to hit https://myapi.com/v1/some_entity then this should return "https://myapi.com/v1/"
60        """
Returns

URL base for the API endpoint e.g: if you wanted to hit https://myapi.com/v1/some_entity then this should return "https://myapi.com/v1/"

@abstractmethod
def get_path( self, *, stream_state: Optional[Mapping[str, Any]], stream_slice: Optional[StreamSlice], next_page_token: Optional[Mapping[str, Any]]) -> str:
62    @abstractmethod
63    def get_path(
64        self,
65        *,
66        stream_state: Optional[StreamState],
67        stream_slice: Optional[StreamSlice],
68        next_page_token: Optional[Mapping[str, Any]],
69    ) -> str:
70        """
71        Returns the URL path for the API endpoint e.g: if you wanted to hit https://myapi.com/v1/some_entity then this should return "some_entity"
72        """

Returns the URL path for the API endpoint e.g: if you wanted to hit https://myapi.com/v1/some_entity then this should return "some_entity"

@abstractmethod
def get_method(self) -> HttpMethod:
74    @abstractmethod
75    def get_method(self) -> HttpMethod:
76        """
77        Specifies the HTTP method to use
78        """

Specifies the HTTP method to use

@abstractmethod
def get_request_params( self, *, stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[StreamSlice] = None, next_page_token: Optional[Mapping[str, Any]] = None) -> MutableMapping[str, Any]:
80    @abstractmethod
81    def get_request_params(
82        self,
83        *,
84        stream_state: Optional[StreamState] = None,
85        stream_slice: Optional[StreamSlice] = None,
86        next_page_token: Optional[Mapping[str, Any]] = None,
87    ) -> MutableMapping[str, Any]:
88        """
89        Specifies the query parameters that should be set on an outgoing HTTP request given the inputs.
90
91        E.g: you might want to define query parameters for paging if next_page_token is not None.
92        """

Specifies the query parameters that should be set on an outgoing HTTP request given the inputs.

E.g: you might want to define query parameters for paging if next_page_token is not None.

@abstractmethod
def get_request_headers( self, *, stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[StreamSlice] = None, next_page_token: Optional[Mapping[str, Any]] = None) -> Mapping[str, Any]:
 94    @abstractmethod
 95    def get_request_headers(
 96        self,
 97        *,
 98        stream_state: Optional[StreamState] = None,
 99        stream_slice: Optional[StreamSlice] = None,
100        next_page_token: Optional[Mapping[str, Any]] = None,
101    ) -> Mapping[str, Any]:
102        """
103        Return any non-auth headers. Authentication headers will overwrite any overlapping headers returned from this method.
104        """

Return any non-auth headers. Authentication headers will overwrite any overlapping headers returned from this method.

@abstractmethod
def get_request_body_data( self, *, stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[StreamSlice] = None, next_page_token: Optional[Mapping[str, Any]] = None) -> Union[Mapping[str, Any], str]:
106    @abstractmethod
107    def get_request_body_data(
108        self,
109        *,
110        stream_state: Optional[StreamState] = None,
111        stream_slice: Optional[StreamSlice] = None,
112        next_page_token: Optional[Mapping[str, Any]] = None,
113    ) -> Union[Mapping[str, Any], str]:
114        """
115        Specifies how to populate the body of the request with a non-JSON payload.
116
117        If returns a ready text that it will be sent as is.
118        If returns a dict that it will be converted to a urlencoded form.
119        E.g. {"key1": "value1", "key2": "value2"} => "key1=value1&key2=value2"
120
121        At the same time only one of the 'request_body_data' and 'request_body_json' functions can be overridden.
122        """

Specifies how to populate the body of the request with a non-JSON payload.

If returns a ready text that it will be sent as is. If returns a dict that it will be converted to a urlencoded form. E.g. {"key1": "value1", "key2": "value2"} => "key1=value1&key2=value2"

At the same time only one of the 'request_body_data' and 'request_body_json' functions can be overridden.

@abstractmethod
def get_request_body_json( self, *, stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[StreamSlice] = None, next_page_token: Optional[Mapping[str, Any]] = None) -> Mapping[str, Any]:
124    @abstractmethod
125    def get_request_body_json(
126        self,
127        *,
128        stream_state: Optional[StreamState] = None,
129        stream_slice: Optional[StreamSlice] = None,
130        next_page_token: Optional[Mapping[str, Any]] = None,
131    ) -> Mapping[str, Any]:
132        """
133        Specifies how to populate the body of the request with a JSON payload.
134
135        At the same time only one of the 'request_body_data' and 'request_body_json' functions can be overridden.
136        """

Specifies how to populate the body of the request with a JSON payload.

At the same time only one of the 'request_body_data' and 'request_body_json' functions can be overridden.

@abstractmethod
def send_request( self, stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[StreamSlice] = None, next_page_token: Optional[Mapping[str, Any]] = None, path: Optional[str] = None, request_headers: Optional[Mapping[str, Any]] = None, request_params: Optional[Mapping[str, Any]] = None, request_body_data: Union[Mapping[str, Any], str, NoneType] = None, request_body_json: Optional[Mapping[str, Any]] = None, log_formatter: Optional[Callable[[requests.models.Response], Any]] = None) -> Optional[requests.models.Response]:
138    @abstractmethod
139    def send_request(
140        self,
141        stream_state: Optional[StreamState] = None,
142        stream_slice: Optional[StreamSlice] = None,
143        next_page_token: Optional[Mapping[str, Any]] = None,
144        path: Optional[str] = None,
145        request_headers: Optional[Mapping[str, Any]] = None,
146        request_params: Optional[Mapping[str, Any]] = None,
147        request_body_data: Optional[Union[Mapping[str, Any], str]] = None,
148        request_body_json: Optional[Mapping[str, Any]] = None,
149        log_formatter: Optional[Callable[[requests.Response], Any]] = None,
150    ) -> Optional[requests.Response]:
151        """
152        Sends a request and returns the response. Might return no response if the error handler chooses to ignore the response or throw an exception in case of an error.
153        If path is set, the path configured on the requester itself is ignored.
154        If header, params and body are set, they are merged with the ones configured on the requester itself.
155
156        If a log formatter is provided, it's used to log the performed request and response. If it's not provided, no logging is performed.
157        """

Sends a request and returns the response. Might return no response if the error handler chooses to ignore the response or throw an exception in case of an error. If path is set, the path configured on the requester itself is ignored. If header, params and body are set, they are merged with the ones configured on the requester itself.

If a log formatter is provided, it's used to log the performed request and response. If it's not provided, no logging is performed.

ResponseStatus
@dataclass
class SimpleRetriever(airbyte_cdk.sources.declarative.retrievers.retriever.Retriever):
 84@dataclass
 85class SimpleRetriever(Retriever):
 86    """
 87    Retrieves records by synchronously sending requests to fetch records.
 88
 89    The retriever acts as an orchestrator between the requester, the record selector, the paginator, and the stream slicer.
 90
 91    For each stream slice, submit requests until there are no more pages of records to fetch.
 92
 93    This retriever currently inherits from HttpStream to reuse the request submission and pagination machinery.
 94    As a result, some of the parameters passed to some methods are unused.
 95    The two will be decoupled in a future release.
 96
 97    Attributes:
 98        stream_name (str): The stream's name
 99        stream_primary_key (Optional[Union[str, List[str], List[List[str]]]]): The stream's primary key
100        requester (Requester): The HTTP requester
101        record_selector (HttpSelector): The record selector
102        paginator (Optional[Paginator]): The paginator
103        stream_slicer (Optional[StreamSlicer]): The stream slicer
104        parameters (Mapping[str, Any]): Additional runtime parameters to be used for string interpolation
105        post_pagination_filter (Optional[ClientSideIncrementalRecordFilterDecorator]): Set for data feed streams only.
106            Records the cursor considers already synced are dropped once pagination has observed them
107        page_size_reduction (Optional[PageSizeReduction]): How much to shrink the page size when an error handler
108            resolves to `ResponseAction.REDUCE_PAGE_SIZE`. `None` disables page size reduction entirely.
109            It is immutable configuration; the page size in effect lives in a `PageSizeReducer` that
110            `_read_pages` creates per call, so the retriever and its paginator - both shared by every
111            partition of the stream, read concurrently - stay stateless. When `page_size_reduction` is
112            `None` no reducer is created and `_read_pages` keeps its previous behaviour
113        request_window_splitting (Optional[RequestWindowSplitting]): Policy applied when an error handler
114            resolves to `ResponseAction.SPLIT_REQUEST_WINDOW`, or when `RequestWindowSplitRequiredException`
115            is raised directly by custom code. `None` disables request-window splitting entirely: `read_records`
116            converts the exception into `RequestWindowSplitNotSupportedException` instead of attempting a
117            split it cannot perform.
118        request_window_splitter: Bound method - normally the stream's own cursor's `split_request_window` -
119            asked to replace a failing `StreamSlice` with smaller children. `None` has the same effect as
120            `request_window_splitting` being `None`.
121    """
122
123    requester: Requester
124    record_selector: HttpSelector
125    config: Config
126    parameters: InitVar[Mapping[str, Any]]
127    name: str
128    _name: Union[InterpolatedString, str] = field(init=False, repr=False, default="")
129    primary_key: Optional[Union[str, List[str], List[List[str]]]]
130    _primary_key: str = field(init=False, repr=False, default="")
131    paginator: Optional[Paginator] = None
132    stream_slicer: StreamSlicer = field(
133        default_factory=lambda: SinglePartitionRouter(parameters={})
134    )
135    request_option_provider: RequestOptionsProvider = field(
136        default_factory=lambda: DefaultRequestOptionsProvider(parameters={})
137    )
138    ignore_stream_slicer_parameters_on_paginated_requests: bool = False
139    additional_query_properties: Optional[QueryProperties] = None
140    log_formatter: Optional[Callable[[requests.Response], Any]] = None
141    pagination_tracker_factory: Callable[[], PaginationTracker] = field(
142        default_factory=lambda: lambda: PaginationTracker()
143    )
144    post_pagination_filter: Optional[ClientSideIncrementalRecordFilterDecorator] = None
145    page_size_reduction: Optional[PageSizeReduction] = None
146    request_window_splitting: Optional[RequestWindowSplitting] = None
147    request_window_splitter: Optional[
148        Callable[[StreamSlice, Optional[datetime.timedelta]], Optional[List[StreamSlice]]]
149    ] = None
150
151    def __post_init__(self, parameters: Mapping[str, Any]) -> None:
152        self._paginator = self.paginator or NoPagination(parameters=parameters)
153        self._parameters = parameters
154        self._name = (
155            InterpolatedString(self._name, parameters=parameters)
156            if isinstance(self._name, str)
157            else self._name
158        )
159
160    @property  # type: ignore
161    def name(self) -> str:
162        """
163        :return: Stream name
164        """
165        return (
166            str(self._name.eval(self.config))
167            if isinstance(self._name, InterpolatedString)
168            else self._name
169        )
170
171    @name.setter
172    def name(self, value: str) -> None:
173        if not isinstance(value, property):
174            self._name = value
175
176    def _get_mapping(
177        self, method: Callable[..., Optional[Union[Mapping[str, Any], str]]], **kwargs: Any
178    ) -> Tuple[Union[Mapping[str, Any], str], Set[str]]:
179        """
180        Get mapping from the provided method, and get the keys of the mapping.
181        If the method returns a string, it will return the string and an empty set.
182        If the method returns a dict, it will return the dict and its keys.
183        """
184        mapping = method(**kwargs) or {}
185        keys = set(mapping.keys()) if not isinstance(mapping, str) else set()
186        return mapping, keys
187
188    def _get_request_options(
189        self,
190        stream_slice: Optional[StreamSlice],
191        next_page_token: Optional[Mapping[str, Any]],
192        paginator_method: Callable[..., Optional[Union[Mapping[str, Any], str]]],
193        stream_slicer_method: Callable[..., Optional[Union[Mapping[str, Any], str]]],
194        page_size_override: Optional[int] = None,
195    ) -> Union[Mapping[str, Any], str]:
196        """
197        Get the request_option from the paginator and the stream slicer.
198        Raise a ValueError if there's a key collision
199        Returned merged mapping otherwise
200        """
201        is_body_json = paginator_method.__name__ == "get_request_body_json"
202
203        mappings = [
204            paginator_method(
205                stream_slice=stream_slice,
206                next_page_token=next_page_token,
207                **page_size_override_kwargs(page_size_override),
208            ),
209        ]
210        if not next_page_token or not self.ignore_stream_slicer_parameters_on_paginated_requests:
211            mappings.append(
212                stream_slicer_method(
213                    stream_slice=stream_slice,
214                    next_page_token=next_page_token,
215                )
216            )
217        return combine_mappings(mappings, allow_same_value_merge=is_body_json)
218
219    def _request_headers(
220        self,
221        stream_slice: Optional[StreamSlice] = None,
222        next_page_token: Optional[Mapping[str, Any]] = None,
223        page_size_override: Optional[int] = None,
224    ) -> Mapping[str, Any]:
225        """
226        Specifies request headers.
227        Authentication headers will overwrite any overlapping headers returned from this method.
228        """
229        headers = self._get_request_options(
230            stream_slice,
231            next_page_token,
232            self._paginator.get_request_headers,
233            self.request_option_provider.get_request_headers,
234            **page_size_override_kwargs(page_size_override),
235        )
236        if isinstance(headers, str):
237            raise ValueError("Request headers cannot be a string")
238        return {str(k): str(v) for k, v in headers.items()}
239
240    def _request_params(
241        self,
242        stream_slice: Optional[StreamSlice] = None,
243        next_page_token: Optional[Mapping[str, Any]] = None,
244        page_size_override: Optional[int] = None,
245    ) -> Mapping[str, Any]:
246        """
247        Specifies the query parameters that should be set on an outgoing HTTP request given the inputs.
248
249        E.g: you might want to define query parameters for paging if next_page_token is not None.
250        """
251        params = self._get_request_options(
252            stream_slice,
253            next_page_token,
254            self._paginator.get_request_params,
255            self.request_option_provider.get_request_params,
256            **page_size_override_kwargs(page_size_override),
257        )
258        if isinstance(params, str):
259            raise ValueError("Request params cannot be a string")
260        return params
261
262    def _request_body_data(
263        self,
264        stream_slice: Optional[StreamSlice] = None,
265        next_page_token: Optional[Mapping[str, Any]] = None,
266        page_size_override: Optional[int] = None,
267    ) -> Union[Mapping[str, Any], str]:
268        """
269        Specifies how to populate the body of the request with a non-JSON payload.
270
271        If returns a ready text that it will be sent as is.
272        If returns a dict that it will be converted to a urlencoded form.
273        E.g. {"key1": "value1", "key2": "value2"} => "key1=value1&key2=value2"
274
275        At the same time only one of the 'request_body_data' and 'request_body_json' functions can be overridden.
276        """
277        return self._get_request_options(
278            stream_slice,
279            next_page_token,
280            self._paginator.get_request_body_data,
281            self.request_option_provider.get_request_body_data,
282            **page_size_override_kwargs(page_size_override),
283        )
284
285    def _request_body_json(
286        self,
287        stream_slice: Optional[StreamSlice] = None,
288        next_page_token: Optional[Mapping[str, Any]] = None,
289        page_size_override: Optional[int] = None,
290    ) -> Optional[Mapping[str, Any]]:
291        """
292        Specifies how to populate the body of the request with a JSON payload.
293
294        At the same time only one of the 'request_body_data' and 'request_body_json' functions can be overridden.
295        """
296        body_json = self._get_request_options(
297            stream_slice,
298            next_page_token,
299            self._paginator.get_request_body_json,
300            self.request_option_provider.get_request_body_json,
301            **page_size_override_kwargs(page_size_override),
302        )
303        if isinstance(body_json, str):
304            raise ValueError("Request body json cannot be a string")
305        return body_json
306
307    def _paginator_path(
308        self,
309        next_page_token: Optional[Mapping[str, Any]] = None,
310        stream_slice: Optional[StreamSlice] = None,
311    ) -> Optional[str]:
312        """
313        If the paginator points to a path, follow it, else return nothing so the requester is used.
314        :param next_page_token:
315        :return:
316        """
317        return self._paginator.path(
318            next_page_token=next_page_token,
319            stream_state={},  # stream_state as an interpolation context is deprecated
320            stream_slice=stream_slice,
321        )
322
323    def _parse_response(
324        self,
325        response: Optional[requests.Response],
326        records_schema: Mapping[str, Any],
327        stream_slice: Optional[StreamSlice] = None,
328        next_page_token: Optional[Mapping[str, Any]] = None,
329    ) -> Iterable[Record]:
330        if not response:
331            yield from []
332        else:
333            yield from self.record_selector.select_records(
334                response=response,
335                stream_state={},  # stream_state as an interpolation context is deprecated
336                records_schema=records_schema,
337                stream_slice=stream_slice,
338                next_page_token=next_page_token,
339            )
340
341    @property  # type: ignore
342    def primary_key(self) -> Optional[Union[str, List[str], List[List[str]]]]:
343        """The stream's primary key"""
344        return self._primary_key
345
346    @primary_key.setter
347    def primary_key(self, value: str) -> None:
348        if not isinstance(value, property):
349            self._primary_key = value
350
351    def _next_page_token(
352        self,
353        response: requests.Response,
354        last_page_size: int,
355        last_record: Optional[Record],
356        last_page_token_value: Optional[Any],
357        page_size_override: Optional[int] = None,
358        stream_slice: Optional[StreamSlice] = None,
359    ) -> Optional[Mapping[str, Any]]:
360        """
361        Specifies a pagination strategy.
362
363        The value returned from this method is passed to most other methods in this class. Use it to form a request e.g: set headers or query params.
364
365        :return: The token for the next page from the input response object. Returning None means there are no more pages to read in this response.
366        """
367        return self._paginator.next_page_token(
368            response=response,
369            last_page_size=last_page_size,
370            last_record=last_record,
371            last_page_token_value=last_page_token_value,
372            **page_size_override_kwargs(page_size_override),
373            **stream_slice_kwargs(self._paginator.next_page_token, stream_slice),
374        )
375
376    def _fetch_next_page(
377        self,
378        stream_slice: StreamSlice,
379        next_page_token: Optional[Mapping[str, Any]] = None,
380        page_size_override: Optional[int] = None,
381    ) -> Optional[requests.Response]:
382        return self.requester.send_request(
383            path=self._paginator_path(
384                next_page_token=next_page_token,
385                stream_slice=stream_slice,
386            ),
387            stream_state={},  # stream_state as an interpolation context is deprecated
388            stream_slice=stream_slice,
389            next_page_token=next_page_token,
390            request_headers=self._request_headers(
391                stream_slice=stream_slice,
392                next_page_token=next_page_token,
393                **page_size_override_kwargs(page_size_override),
394            ),
395            request_params=self._request_params(
396                stream_slice=stream_slice,
397                next_page_token=next_page_token,
398                **page_size_override_kwargs(page_size_override),
399            ),
400            request_body_data=self._request_body_data(
401                stream_slice=stream_slice,
402                next_page_token=next_page_token,
403                **page_size_override_kwargs(page_size_override),
404            ),
405            request_body_json=self._request_body_json(
406                stream_slice=stream_slice,
407                next_page_token=next_page_token,
408                **page_size_override_kwargs(page_size_override),
409            ),
410            log_formatter=self.log_formatter,
411        )
412
413    # This logic is similar to _read_pages in the HttpStream class. When making changes here, consider making changes there as well.
414    def _read_pages(
415        self,
416        records_generator_fn: Callable[[Optional[requests.Response]], Iterable[Record]],
417        stream_slice: StreamSlice,
418    ) -> Iterable[Record]:
419        original_stream_slice = stream_slice
420        pagination_tracker = self.pagination_tracker_factory()
421        page_size_reducer = (
422            PageSizeReducer(
423                self.page_size_reduction,
424                self._paginator.get_page_size(),
425                stream_name=self.name,
426            )
427            if self.page_size_reduction
428            else None
429        )
430        reset_pagination = False
431        reduce_page_size = False
432        next_page_token = self._get_initial_next_page_token()
433        while True:
434            page_size_override = page_size_reducer.page_size_override if page_size_reducer else None
435            merged_records: MutableMapping[str, Any] = defaultdict(dict)
436            last_page_size = 0
437            last_record: Optional[Record] = None
438
439            response = None
440            try:
441                if self.additional_query_properties:
442                    for (
443                        properties
444                    ) in self.additional_query_properties.get_request_property_chunks():
445                        stream_slice = StreamSlice(
446                            partition=stream_slice.partition or {},
447                            cursor_slice=stream_slice.cursor_slice or {},
448                            extra_fields={"query_properties": properties},
449                        )
450                        response = self._fetch_next_page(
451                            stream_slice,
452                            next_page_token,
453                            **page_size_override_kwargs(page_size_override),
454                        )
455
456                        for current_record in records_generator_fn(response):
457                            if self.additional_query_properties.property_chunking:
458                                merge_key = self.additional_query_properties.property_chunking.get_merge_key(
459                                    current_record
460                                )
461                                if merge_key:
462                                    _deep_merge(merged_records[merge_key], current_record)
463                                else:
464                                    # We should still emit records even if the record did not have a merge key
465                                    pagination_tracker.observe(current_record)
466                                    last_page_size += 1
467                                    last_record = current_record
468                                    yield current_record
469                            else:
470                                pagination_tracker.observe(current_record)
471                                last_page_size += 1
472                                last_record = current_record
473                                yield current_record
474
475                    for merged_record in merged_records.values():
476                        record = Record(
477                            data=merged_record, stream_name=self.name, associated_slice=stream_slice
478                        )
479                        pagination_tracker.observe(record)
480                        last_page_size += 1
481                        last_record = record
482                        yield record
483                else:
484                    response = self._fetch_next_page(
485                        stream_slice,
486                        next_page_token,
487                        **page_size_override_kwargs(page_size_override),
488                    )
489                    for current_record in records_generator_fn(response):
490                        pagination_tracker.observe(current_record)
491                        last_page_size += 1
492                        last_record = current_record
493                        yield current_record
494            except PaginationResetRequiredException:
495                reset_pagination = True
496            except PageSizeReductionRequiredException as exception:
497                if page_size_reducer is None:
498                    # The action can be attached to a requester we cannot validate at config time, such as one
499                    # built by a custom error handler or a CustomRequester. Re-raise as the misconfiguration it
500                    # is: the exception being handled is the neutral "the API asked for a smaller page" signal.
501                    raise PageSizeReductionNotSupportedException(stream_name=self.name)
502                if last_page_size:
503                    # The reduction is safe only because it re-issues a page whose records were not emitted.
504                    # Every in-CDK way of reaching this raises from `_fetch_next_page`, before the record loop,
505                    # and the factory rejects the manifest constructs that would not, but a custom extractor,
506                    # filter or transformation can issue its own request from inside the record generator.
507                    # Re-issuing the page then duplicates the records already yielded, so fail instead.
508                    raise AirbyteTracedException(
509                        internal_message=f"Stream {self.name} requested a page size reduction after {last_page_size} records of the page had already been emitted",
510                        message=f"Stream {self.name} asked for a smaller page size in the middle of a page. The page cannot be requested again without duplicating the records already read from it. Move the REDUCE_PAGE_SIZE action to the error handler of the stream's main requester.",
511                        failure_type=FailureType.config_error,
512                    )
513                # Raises once the page size cannot be reduced any further and the retries allowed at that
514                # floor are spent, which is what stops the loop when the API keeps failing.
515                page_size_reducer.reduce(exception.error_message)
516                reduce_page_size = True
517            else:
518                if page_size_reducer:
519                    page_size_reducer.on_successful_page()
520                if not response:
521                    break
522
523            if reduce_page_size:
524                # Retry the very same page: neither the token nor the slice change, only the page size does -
525                # and not even that once the reducer is at its floor and only waiting is left.
526                reduce_page_size = False
527                continue
528
529            if reset_pagination or pagination_tracker.has_reached_limit():
530                next_page_token = self._get_initial_next_page_token()
531                previous_slice = stream_slice
532                stream_slice = pagination_tracker.reduce_slice_range_if_possible(
533                    stream_slice, original_stream_slice
534                )
535                LOGGER.info(
536                    f"Hitting PaginationReset event. StreamSlice used will go from {previous_slice} to {stream_slice}"
537                )
538                reset_pagination = False
539            else:
540                last_page_token_value = (
541                    next_page_token.get("next_page_token") if next_page_token else None
542                )
543                next_page_token = self._next_page_token(
544                    response=response,  # type:ignore # we are breaking from the loop on the try/else if there are no response so this should be fine
545                    last_page_size=last_page_size,
546                    last_record=last_record,
547                    last_page_token_value=last_page_token_value,
548                    **page_size_override_kwargs(page_size_override),
549                    **stream_slice_kwargs(self._next_page_token, stream_slice),
550                )
551                if not next_page_token:
552                    break
553
554        # Always return an empty generator just in case no records were ever yielded
555        yield from []
556
557    def _get_initial_next_page_token(self) -> Optional[Mapping[str, Any]]:
558        initial_token = self._paginator.get_initial_token()
559        next_page_token = {"next_page_token": initial_token} if initial_token is not None else None
560        return next_page_token
561
562    def read_records(
563        self,
564        records_schema: Mapping[str, Any],
565        stream_slice: Optional[StreamSlice] = None,
566    ) -> Iterable[StreamData]:
567        """
568        Fetch a stream's records from an HTTP API source
569
570        :param records_schema: json schema to describe record
571        :param stream_slice: The stream slice to read data for
572        :return: The records read from the API source
573        """
574        _slice = stream_slice or StreamSlice(partition={}, cursor_slice={})  # None-check
575        yield from self._read_records_or_split_request_window(
576            records_schema, _slice, _slice, depth=0
577        )
578
579    def _read_records_or_split_request_window(
580        self,
581        records_schema: Mapping[str, Any],
582        stream_slice: StreamSlice,
583        original_slice: StreamSlice,
584        depth: int,
585    ) -> Iterable[StreamData]:
586        """
587        Read `stream_slice` to completion, replacing it with smaller children and recursing into each of them in
588        turn when a `RequestWindowSplitRequiredException` is raised while reading it.
589
590        Each recursive call re-enters this method - and, through it, `_read_pages` - from scratch for the child
591        slice it is given, so every child gets its own paginator token, `PaginationTracker`, and
592        `PageSizeReducer` for free: nothing here needs to reset that state explicitly. Because the whole
593        recursion lives inside the single generator `DeclarativePartition.read()` consumes,
594        `PartitionReader.process_partition()` only calls `cursor.close_partition()` once, after this generator is
595        fully exhausted - so a failure anywhere in the recursion (a child that cannot be read, or a window that
596        cannot be split any further) propagates out without ever checkpointing the original partition.
597
598        `original_slice` is threaded through the recursion unchanged so every record yielded, however deep the
599        recursion went to produce it, is re-stamped with the partition's own slice rather than the child slice it
600        was actually read against - see `_reassociate_with_original_slice`. `depth` bounds the recursion
601        independently of `request_window_splitter`'s own no-progress guard: it is enforced here so a
602        misbehaving custom cursor cannot recurse indefinitely regardless of what that guard does or does not
603        catch.
604        """
605        record_generator = partial(
606            self._parse_records,
607            stream_slice=stream_slice,
608            records_schema=records_schema,
609        )
610
611        emitted_count = 0
612        try:
613            records: Iterable[Mapping[str, Any]] = self._read_pages(record_generator, stream_slice)
614            if self.post_pagination_filter:
615                # A data feed paginates until it reaches a record older than the cursor, so the page that triggers the stop
616                # condition still holds already-synced records. Those are filtered here rather than in the record selector
617                # so that the paginator keeps seeing the whole page: the stop condition is evaluated on the last record of
618                # the page, which is precisely one of the records being dropped. Two consequences of filtering this late:
619                # the pagination tracker observes the dropped records, and a `file_uploader` on the record selector has
620                # already uploaded their files by the time they are dropped.
621                records = self.post_pagination_filter.filter_records(
622                    records,
623                    # the filter is only used for its cursor comparison, which does not read the stream state
624                    stream_state={},
625                    stream_slice=stream_slice,
626                )
627            for record in records:
628                emitted_count += 1
629                yield self._reassociate_with_original_slice(record, original_slice)
630        except RequestWindowSplitRequiredException as exception:
631            if self.request_window_splitting is None or self.request_window_splitter is None:
632                raise RequestWindowSplitNotSupportedException(stream_name=self.name) from exception
633
634            if emitted_count:
635                # Splitting and re-reading the window re-emits these records: a failed partition is never
636                # checkpointed, so the next attempt would re-emit them anyway. Always replaying makes forward
637                # progress instead of retrying the same oversized window forever; primary-key dedup downstream
638                # handles the duplicates either way.
639                LOGGER.warning(
640                    f"Stream {self.name} already emitted {emitted_count} record(s) from {stream_slice} before "
641                    f"it was rejected; splitting and re-reading the window may re-emit them."
642                )
643
644            rejection = f": {exception.error_message}" if exception.error_message else ""
645            if depth >= _MAX_REQUEST_WINDOW_SPLIT_DEPTH:
646                # Logged separately from the exception below so it stays visible even if the trace message's
647                # internal_message isn't surfaced by whatever catches it.
648                LOGGER.warning(
649                    f"Stream {self.name} hit the maximum request window split depth "
650                    f"({_MAX_REQUEST_WINDOW_SPLIT_DEPTH}) while splitting {original_slice}."
651                )
652                raise AirbyteTracedException(
653                    internal_message=f"Stream {self.name} exceeded the maximum request window split depth of {_MAX_REQUEST_WINDOW_SPLIT_DEPTH} while splitting {original_slice}; the API still rejected {stream_slice}{rejection}",
654                    # `transient_error`, so the only remediation is the connector's own, if it defined one.
655                    message=self._with_request_window_failure_message(
656                        f"Stream {self.name} could not split its request window to a size the API accepts "
657                        f"within {_MAX_REQUEST_WINDOW_SPLIT_DEPTH} splits."
658                    ),
659                    failure_type=FailureType.transient_error,
660                ) from exception
661
662            children = self.request_window_splitter(
663                stream_slice, self.request_window_splitting.min_split_window
664            )
665            if children is None:
666                min_split_window_note = (
667                    f", its configured `min_split_window` ({self.request_window_splitting.min_split_window})"
668                    if self.request_window_splitting.min_split_window
669                    else ""
670                )
671                raise AirbyteTracedException(
672                    internal_message=f"Stream {self.name} could not split its request window {stream_slice} any further; the API rejected it{rejection}",
673                    # `transient_error` regardless of how the triggering response was classified: exhaustion is
674                    # never the user's fault, matching `PageSizeReducer`'s equivalent exhaustion branch.
675                    message=self._with_request_window_failure_message(
676                        f"The API kept rejecting stream {self.name}'s request window even at the smallest window "
677                        f"its cursor allows{min_split_window_note}, or splitting the window further would not "
678                        f"make progress."
679                    ),
680                    failure_type=FailureType.transient_error,
681                ) from exception
682
683            LOGGER.info(
684                f"Stream {self.name}: the API rejected request window {stream_slice} (split depth {depth}); "
685                f"reducing it to {children} and reading each in turn."
686            )
687            for child in children:
688                yield from self._read_records_or_split_request_window(
689                    records_schema, child, original_slice, depth + 1
690                )
691
692    def _with_request_window_failure_message(self, message: str) -> str:
693        """
694        :return: the message followed by `request_window_splitting`'s configured `failure_message`, when it
695            defined one - mirroring `PageSizeReducer._with_failure_message`
696        """
697        assert self.request_window_splitting is not None
698        failure_message = (self.request_window_splitting.failure_message or "").strip()
699        if not failure_message:
700            return message
701        return f"{message} {failure_message}"
702
703    @staticmethod
704    def _reassociate_with_original_slice(
705        record: StreamData, original_slice: StreamSlice
706    ) -> StreamData:
707        """
708        Records read while recursing into a split child window are parsed against that child's `StreamSlice`,
709        so `RecordSelector` stamps them with the child as `associated_slice`. `DeclarativePartition.read()`
710        passes an already-built `Record` through unchanged rather than re-wrapping it, so left uncorrected the
711        child slice - not the partition's own slice - is what `ConcurrentCursor.observe()` would key its
712        per-partition bookkeeping by. `close_partition()` always looks that bookkeeping up by the partition's own
713        slice, so it would never find it, silently losing the precise most-recently-observed cursor value for a
714        split partition (recoverable in practice today only because callers of that lookup fall back to the
715        slice's end boundary when it is missing). Re-stamping every record with `original_slice` here restores
716        the same tracking a non-split read already gets, rather than relying on that fallback.
717        """
718        if isinstance(record, Record) and record.associated_slice is not original_slice:
719            return Record(
720                data=record.data,
721                stream_name=record.stream_name,
722                associated_slice=original_slice,
723                file_reference=record.file_reference,
724            )
725        return record
726
727    def _parse_records(
728        self,
729        response: Optional[requests.Response],
730        records_schema: Mapping[str, Any],
731        stream_slice: Optional[StreamSlice],
732    ) -> Iterable[Record]:
733        yield from self._parse_response(
734            response,
735            stream_slice=stream_slice,
736            records_schema=records_schema,
737        )
738
739    def must_deduplicate_query_params(self) -> bool:
740        return True
741
742    @staticmethod
743    def _to_partition_key(to_serialize: Any) -> str:
744        # separators have changed in Python 3.4. To avoid being impacted by further change, we explicitly specify our own value
745        return json.dumps(to_serialize, indent=None, separators=(",", ":"), sort_keys=True)

Retrieves records by synchronously sending requests to fetch records.

The retriever acts as an orchestrator between the requester, the record selector, the paginator, and the stream slicer.

For each stream slice, submit requests until there are no more pages of records to fetch.

This retriever currently inherits from HttpStream to reuse the request submission and pagination machinery. As a result, some of the parameters passed to some methods are unused. The two will be decoupled in a future release.

Attributes:
  • stream_name (str): The stream's name
  • stream_primary_key (Optional[Union[str, List[str], List[List[str]]]]): The stream's primary key
  • requester (Requester): The HTTP requester
  • record_selector (HttpSelector): The record selector
  • paginator (Optional[Paginator]): The paginator
  • stream_slicer (Optional[StreamSlicer]): The stream slicer
  • parameters (Mapping[str, Any]): Additional runtime parameters to be used for string interpolation
  • post_pagination_filter (Optional[ClientSideIncrementalRecordFilterDecorator]): Set for data feed streams only. Records the cursor considers already synced are dropped once pagination has observed them
  • page_size_reduction (Optional[PageSizeReduction]): How much to shrink the page size when an error handler resolves to ResponseAction.REDUCE_PAGE_SIZE. None disables page size reduction entirely. It is immutable configuration; the page size in effect lives in a PageSizeReducer that _read_pages creates per call, so the retriever and its paginator - both shared by every partition of the stream, read concurrently - stay stateless. When page_size_reduction is None no reducer is created and _read_pages keeps its previous behaviour
  • request_window_splitting (Optional[RequestWindowSplitting]): Policy applied when an error handler resolves to ResponseAction.SPLIT_REQUEST_WINDOW, or when RequestWindowSplitRequiredException is raised directly by custom code. None disables request-window splitting entirely: read_records converts the exception into RequestWindowSplitNotSupportedException instead of attempting a split it cannot perform.
  • request_window_splitter: Bound method - normally the stream's own cursor's split_request_window - asked to replace a failing StreamSlice with smaller children. None has the same effect as request_window_splitting being None.
SimpleRetriever( requester: Requester, record_selector: airbyte_cdk.sources.declarative.extractors.HttpSelector, config: Mapping[str, Any], parameters: dataclasses.InitVar[typing.Mapping[str, typing.Any]], name: str = <property object>, primary_key: Union[str, List[str], List[List[str]], NoneType] = <property object>, paginator: Optional[airbyte_cdk.sources.declarative.requesters.paginators.Paginator] = None, stream_slicer: airbyte_cdk.sources.declarative.stream_slicers.StreamSlicer = <factory>, request_option_provider: airbyte_cdk.sources.declarative.requesters.request_options.RequestOptionsProvider = <factory>, ignore_stream_slicer_parameters_on_paginated_requests: bool = False, additional_query_properties: Optional[airbyte_cdk.sources.declarative.requesters.query_properties.QueryProperties] = None, log_formatter: Optional[Callable[[requests.models.Response], Any]] = None, pagination_tracker_factory: Callable[[], airbyte_cdk.sources.declarative.retrievers.pagination_tracker.PaginationTracker] = <factory>, post_pagination_filter: Optional[airbyte_cdk.sources.declarative.extractors.record_filter.ClientSideIncrementalRecordFilterDecorator] = None, page_size_reduction: Optional[airbyte_cdk.sources.declarative.retrievers.page_size_reducer.PageSizeReduction] = None, request_window_splitting: Optional[airbyte_cdk.sources.declarative.retrievers.request_window_splitting.RequestWindowSplitting] = None, request_window_splitter: Optional[Callable[[StreamSlice, Optional[datetime.timedelta]], Optional[List[StreamSlice]]]] = None)
requester: Requester
config: Mapping[str, Any]
parameters: dataclasses.InitVar[typing.Mapping[str, typing.Any]]
name: str
160    @property  # type: ignore
161    def name(self) -> str:
162        """
163        :return: Stream name
164        """
165        return (
166            str(self._name.eval(self.config))
167            if isinstance(self._name, InterpolatedString)
168            else self._name
169        )
Returns

Stream name

primary_key: Union[str, List[str], List[List[str]], NoneType]
341    @property  # type: ignore
342    def primary_key(self) -> Optional[Union[str, List[str], List[List[str]]]]:
343        """The stream's primary key"""
344        return self._primary_key

The stream's primary key

ignore_stream_slicer_parameters_on_paginated_requests: bool = False
log_formatter: Optional[Callable[[requests.models.Response], Any]] = None
request_window_splitter: Optional[Callable[[StreamSlice, Optional[datetime.timedelta]], Optional[List[StreamSlice]]]] = None
def read_records( self, records_schema: Mapping[str, Any], stream_slice: Optional[StreamSlice] = None) -> Iterable[Union[Mapping[str, Any], AirbyteMessage]]:
562    def read_records(
563        self,
564        records_schema: Mapping[str, Any],
565        stream_slice: Optional[StreamSlice] = None,
566    ) -> Iterable[StreamData]:
567        """
568        Fetch a stream's records from an HTTP API source
569
570        :param records_schema: json schema to describe record
571        :param stream_slice: The stream slice to read data for
572        :return: The records read from the API source
573        """
574        _slice = stream_slice or StreamSlice(partition={}, cursor_slice={})  # None-check
575        yield from self._read_records_or_split_request_window(
576            records_schema, _slice, _slice, depth=0
577        )

Fetch a stream's records from an HTTP API source

Parameters
  • records_schema: json schema to describe record
  • stream_slice: The stream slice to read data for
Returns

The records read from the API source

def must_deduplicate_query_params(self) -> bool:
739    def must_deduplicate_query_params(self) -> bool:
740        return True
@dataclass
class SinglePartitionRouter(airbyte_cdk.sources.declarative.partition_routers.partition_router.PartitionRouter):
13@dataclass
14class SinglePartitionRouter(PartitionRouter):
15    """Partition router returning only a stream slice"""
16
17    parameters: InitVar[Mapping[str, Any]]
18
19    def get_request_params(
20        self,
21        stream_state: Optional[StreamState] = None,
22        stream_slice: Optional[StreamSlice] = None,
23        next_page_token: Optional[Mapping[str, Any]] = None,
24    ) -> Mapping[str, Any]:
25        return {}
26
27    def get_request_headers(
28        self,
29        stream_state: Optional[StreamState] = None,
30        stream_slice: Optional[StreamSlice] = None,
31        next_page_token: Optional[Mapping[str, Any]] = None,
32    ) -> Mapping[str, Any]:
33        return {}
34
35    def get_request_body_data(
36        self,
37        stream_state: Optional[StreamState] = None,
38        stream_slice: Optional[StreamSlice] = None,
39        next_page_token: Optional[Mapping[str, Any]] = None,
40    ) -> Mapping[str, Any]:
41        return {}
42
43    def get_request_body_json(
44        self,
45        stream_state: Optional[StreamState] = None,
46        stream_slice: Optional[StreamSlice] = None,
47        next_page_token: Optional[Mapping[str, Any]] = None,
48    ) -> Mapping[str, Any]:
49        return {}
50
51    def stream_slices(self) -> Iterable[StreamSlice]:
52        yield StreamSlice(partition={}, cursor_slice={})
53
54    def get_stream_state(self) -> Optional[Mapping[str, StreamState]]:
55        """
56        SinglePartitionRouter doesn't have parent streams
57        """
58        pass

Partition router returning only a stream slice

SinglePartitionRouter(parameters: dataclasses.InitVar[typing.Mapping[str, typing.Any]])
parameters: dataclasses.InitVar[typing.Mapping[str, typing.Any]]
def get_request_params( self, stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[StreamSlice] = None, next_page_token: Optional[Mapping[str, Any]] = None) -> Mapping[str, Any]:
19    def get_request_params(
20        self,
21        stream_state: Optional[StreamState] = None,
22        stream_slice: Optional[StreamSlice] = None,
23        next_page_token: Optional[Mapping[str, Any]] = None,
24    ) -> Mapping[str, Any]:
25        return {}

Specifies the query parameters that should be set on an outgoing HTTP request given the inputs.

E.g: you might want to define query parameters for paging if next_page_token is not None.

def get_request_headers( self, stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[StreamSlice] = None, next_page_token: Optional[Mapping[str, Any]] = None) -> Mapping[str, Any]:
27    def get_request_headers(
28        self,
29        stream_state: Optional[StreamState] = None,
30        stream_slice: Optional[StreamSlice] = None,
31        next_page_token: Optional[Mapping[str, Any]] = None,
32    ) -> Mapping[str, Any]:
33        return {}

Return any non-auth headers. Authentication headers will overwrite any overlapping headers returned from this method.

def get_request_body_data( self, stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[StreamSlice] = None, next_page_token: Optional[Mapping[str, Any]] = None) -> Mapping[str, Any]:
35    def get_request_body_data(
36        self,
37        stream_state: Optional[StreamState] = None,
38        stream_slice: Optional[StreamSlice] = None,
39        next_page_token: Optional[Mapping[str, Any]] = None,
40    ) -> Mapping[str, Any]:
41        return {}

Specifies how to populate the body of the request with a non-JSON payload.

If returns a ready text that it will be sent as is. If returns a dict that it will be converted to a urlencoded form. E.g. {"key1": "value1", "key2": "value2"} => "key1=value1&key2=value2"

At the same time only one of the 'request_body_data' and 'request_body_json' functions can be overridden.

def get_request_body_json( self, stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[StreamSlice] = None, next_page_token: Optional[Mapping[str, Any]] = None) -> Mapping[str, Any]:
43    def get_request_body_json(
44        self,
45        stream_state: Optional[StreamState] = None,
46        stream_slice: Optional[StreamSlice] = None,
47        next_page_token: Optional[Mapping[str, Any]] = None,
48    ) -> Mapping[str, Any]:
49        return {}

Specifies how to populate the body of the request with a JSON payload.

At the same time only one of the 'request_body_data' and 'request_body_json' functions can be overridden.

def stream_slices(self) -> Iterable[StreamSlice]:
51    def stream_slices(self) -> Iterable[StreamSlice]:
52        yield StreamSlice(partition={}, cursor_slice={})

Defines stream slices

Returns

An iterable of stream slices

def get_stream_state(self) -> Optional[Mapping[str, Mapping[str, Any]]]:
54    def get_stream_state(self) -> Optional[Mapping[str, StreamState]]:
55        """
56        SinglePartitionRouter doesn't have parent streams
57        """
58        pass

SinglePartitionRouter doesn't have parent streams

class StopConditionPaginationStrategyDecorator(airbyte_cdk.PaginationStrategy):
44class StopConditionPaginationStrategyDecorator(PaginationStrategy):
45    def __init__(self, _delegate: PaginationStrategy, stop_condition: PaginationStopCondition):
46        self._delegate = _delegate
47        self._stop_condition = stop_condition
48
49    def next_page_token(
50        self,
51        response: requests.Response,
52        last_page_size: int,
53        last_record: Optional[Record],
54        last_page_token_value: Optional[Any] = None,
55        page_size_override: Optional[int] = None,
56        stream_slice: Optional[StreamSlice] = None,
57    ) -> Optional[Any]:
58        # We evaluate in reverse order because the assumption is that most of the APIs using data feed structure
59        # will return records in descending order. In terms of performance/memory, we return the records lazily
60        if last_record and self._stop_condition.is_met(last_record):
61            return None
62        return self._delegate.next_page_token(
63            response,
64            last_page_size,
65            last_record,
66            last_page_token_value,
67            **page_size_override_kwargs(page_size_override),
68            **stream_slice_kwargs(self._delegate.next_page_token, stream_slice),
69        )
70
71    def get_page_size(self) -> Optional[int]:
72        return self._delegate.get_page_size()
73
74    @property
75    def initial_token(self) -> Optional[Any]:
76        return self._delegate.initial_token

Defines how to get the next page token

One instance paginates every partition of the stream, several at once, so next_page_token must derive the next token from its arguments, such as last_page_token_value, rather than from state kept on the instance.

StopConditionPaginationStrategyDecorator( _delegate: PaginationStrategy, stop_condition: airbyte_cdk.sources.declarative.requesters.paginators.strategies.stop_condition.PaginationStopCondition)
45    def __init__(self, _delegate: PaginationStrategy, stop_condition: PaginationStopCondition):
46        self._delegate = _delegate
47        self._stop_condition = stop_condition
def next_page_token( self, response: requests.models.Response, last_page_size: int, last_record: Optional[Record], last_page_token_value: Optional[Any] = None, page_size_override: Optional[int] = None, stream_slice: Optional[StreamSlice] = None) -> Optional[Any]:
49    def next_page_token(
50        self,
51        response: requests.Response,
52        last_page_size: int,
53        last_record: Optional[Record],
54        last_page_token_value: Optional[Any] = None,
55        page_size_override: Optional[int] = None,
56        stream_slice: Optional[StreamSlice] = None,
57    ) -> Optional[Any]:
58        # We evaluate in reverse order because the assumption is that most of the APIs using data feed structure
59        # will return records in descending order. In terms of performance/memory, we return the records lazily
60        if last_record and self._stop_condition.is_met(last_record):
61            return None
62        return self._delegate.next_page_token(
63            response,
64            last_page_size,
65            last_record,
66            last_page_token_value,
67            **page_size_override_kwargs(page_size_override),
68            **stream_slice_kwargs(self._delegate.next_page_token, stream_slice),
69        )
Parameters
  • response: response to process
  • last_page_size: the number of records read from the response
  • last_record: the last record extracted from the response
  • last_page_token_value: The current value of the page token made on the last request
  • page_size_override: the page size that was actually requested, when it differs from the configured one because of a REDUCE_PAGE_SIZE response action. Strategies that use their configured page size as a stop condition must honor this value, else they end the pagination early and skip records.
  • stream_slice: the slice the page was read for. Only passed to strategies that declare it, so existing strategies defined outside of the CDK keep working unchanged.
Returns

next page token. Returns None if there are no more pages to fetch

def get_page_size(self) -> Optional[int]:
71    def get_page_size(self) -> Optional[int]:
72        return self._delegate.get_page_size()
Returns

page size: The number of records to fetch in a page. Returns None if unspecified

initial_token: Optional[Any]
74    @property
75    def initial_token(self) -> Optional[Any]:
76        return self._delegate.initial_token

Return the initial value of the token

class StreamSlice(typing.Mapping[str, typing.Any]):
 76class StreamSlice(Mapping[str, Any]):
 77    def __init__(
 78        self,
 79        *,
 80        partition: Mapping[str, Any],
 81        cursor_slice: Mapping[str, Any],
 82        extra_fields: Optional[Mapping[str, Any]] = None,
 83    ) -> None:
 84        """
 85        :param partition: The partition keys representing a unique partition in the stream.
 86        :param cursor_slice: The incremental cursor slice keys, such as dates or pagination tokens.
 87        :param extra_fields: Additional fields that should not be part of the partition but passed along, such as metadata from the parent stream.
 88        """
 89        self._partition = partition
 90        self._cursor_slice = cursor_slice
 91        self._extra_fields = extra_fields or {}
 92
 93        # Ensure that partition keys do not overlap with cursor slice keys
 94        if partition.keys() & cursor_slice.keys():
 95            raise ValueError("Keys for partition and incremental sync cursor should not overlap")
 96
 97        self._stream_slice = dict(partition) | dict(cursor_slice)
 98
 99    @property
100    def partition(self) -> Mapping[str, Any]:
101        """Returns the partition portion of the stream slice."""
102        p = self._partition
103        while isinstance(p, StreamSlice):
104            p = p.partition
105        return p
106
107    @property
108    def cursor_slice(self) -> Mapping[str, Any]:
109        """Returns the cursor slice portion of the stream slice."""
110        c = self._cursor_slice
111        while isinstance(c, StreamSlice):
112            c = c.cursor_slice
113        return c
114
115    @property
116    def extra_fields(self) -> Mapping[str, Any]:
117        """Returns the extra fields that are not part of the partition."""
118        return self._extra_fields
119
120    def __repr__(self) -> str:
121        return repr(self._stream_slice)
122
123    def __setitem__(self, key: str, value: Any) -> None:
124        raise ValueError("StreamSlice is immutable")
125
126    def __getitem__(self, key: str) -> Any:
127        return self._stream_slice[key]
128
129    def __len__(self) -> int:
130        return len(self._stream_slice)
131
132    def __iter__(self) -> Iterator[str]:
133        return iter(self._stream_slice)
134
135    def __contains__(self, item: Any) -> bool:
136        return item in self._stream_slice
137
138    def keys(self) -> KeysView[str]:
139        return self._stream_slice.keys()
140
141    def items(self) -> ItemsView[str, Any]:
142        return self._stream_slice.items()
143
144    def values(self) -> ValuesView[Any]:
145        return self._stream_slice.values()
146
147    def get(self, key: str, default: Any = None) -> Optional[Any]:
148        return self._stream_slice.get(key, default)
149
150    def __eq__(self, other: Any) -> bool:
151        if isinstance(other, dict):
152            return self._stream_slice == other
153        if isinstance(other, StreamSlice):
154            # noinspection PyProtectedMember
155            return self._partition == other._partition and self._cursor_slice == other._cursor_slice
156        return False
157
158    def __ne__(self, other: Any) -> bool:
159        return not self.__eq__(other)
160
161    def __json_serializable__(self) -> Any:
162        return self._stream_slice
163
164    def __hash__(self) -> int:
165        return SliceHasher.hash(
166            stream_slice=self._stream_slice
167        )  # no need to provide stream_name here as this is used for slicing the cursor
168
169    def __bool__(self) -> bool:
170        return bool(self._stream_slice) or bool(self._extra_fields)

A Mapping is a generic container for associating key/value pairs.

This class provides concrete generic implementations of all methods except for __getitem__, __iter__, and __len__.

StreamSlice( *, partition: Mapping[str, Any], cursor_slice: Mapping[str, Any], extra_fields: Optional[Mapping[str, Any]] = None)
77    def __init__(
78        self,
79        *,
80        partition: Mapping[str, Any],
81        cursor_slice: Mapping[str, Any],
82        extra_fields: Optional[Mapping[str, Any]] = None,
83    ) -> None:
84        """
85        :param partition: The partition keys representing a unique partition in the stream.
86        :param cursor_slice: The incremental cursor slice keys, such as dates or pagination tokens.
87        :param extra_fields: Additional fields that should not be part of the partition but passed along, such as metadata from the parent stream.
88        """
89        self._partition = partition
90        self._cursor_slice = cursor_slice
91        self._extra_fields = extra_fields or {}
92
93        # Ensure that partition keys do not overlap with cursor slice keys
94        if partition.keys() & cursor_slice.keys():
95            raise ValueError("Keys for partition and incremental sync cursor should not overlap")
96
97        self._stream_slice = dict(partition) | dict(cursor_slice)
Parameters
  • partition: The partition keys representing a unique partition in the stream.
  • cursor_slice: The incremental cursor slice keys, such as dates or pagination tokens.
  • extra_fields: Additional fields that should not be part of the partition but passed along, such as metadata from the parent stream.
partition: Mapping[str, Any]
 99    @property
100    def partition(self) -> Mapping[str, Any]:
101        """Returns the partition portion of the stream slice."""
102        p = self._partition
103        while isinstance(p, StreamSlice):
104            p = p.partition
105        return p

Returns the partition portion of the stream slice.

cursor_slice: Mapping[str, Any]
107    @property
108    def cursor_slice(self) -> Mapping[str, Any]:
109        """Returns the cursor slice portion of the stream slice."""
110        c = self._cursor_slice
111        while isinstance(c, StreamSlice):
112            c = c.cursor_slice
113        return c

Returns the cursor slice portion of the stream slice.

extra_fields: Mapping[str, Any]
115    @property
116    def extra_fields(self) -> Mapping[str, Any]:
117        """Returns the extra fields that are not part of the partition."""
118        return self._extra_fields

Returns the extra fields that are not part of the partition.

def keys(self) -> KeysView[str]:
138    def keys(self) -> KeysView[str]:
139        return self._stream_slice.keys()

D.keys() -> a set-like object providing a view on D's keys

def items(self) -> ItemsView[str, Any]:
141    def items(self) -> ItemsView[str, Any]:
142        return self._stream_slice.items()

D.items() -> a set-like object providing a view on D's items

def values(self) -> ValuesView[Any]:
144    def values(self) -> ValuesView[Any]:
145        return self._stream_slice.values()

D.values() -> an object providing a view on D's values

def get(self, key: str, default: Any = None) -> Optional[Any]:
147    def get(self, key: str, default: Any = None) -> Optional[Any]:
148        return self._stream_slice.get(key, default)

D.get(k[,d]) -> D[k] if k in D, else d. d defaults to None.

@dataclass
class SubstreamPartitionRouter(airbyte_cdk.sources.declarative.partition_routers.partition_router.PartitionRouter):
108@dataclass
109class SubstreamPartitionRouter(PartitionRouter):
110    """
111    Partition router that iterates over the parent's stream records and emits slices
112    Will populate the state with `partition_field` and `parent_slice` so they can be accessed by other components
113
114    Attributes:
115        parent_stream_configs (List[ParentStreamConfig]): parent streams to iterate over and their config
116    """
117
118    parent_stream_configs: List[ParentStreamConfig]
119    config: Config
120    parameters: InitVar[Mapping[str, Any]]
121
122    def __post_init__(self, parameters: Mapping[str, Any]) -> None:
123        if not self.parent_stream_configs:
124            raise ValueError("SubstreamPartitionRouter needs at least 1 parent stream")
125        self._parameters = parameters
126
127    def get_request_params(
128        self,
129        stream_state: Optional[StreamState] = None,
130        stream_slice: Optional[StreamSlice] = None,
131        next_page_token: Optional[Mapping[str, Any]] = None,
132    ) -> Mapping[str, Any]:
133        # Pass the stream_slice from the argument, not the cursor because the cursor is updated after processing the response
134        return self._get_request_option(RequestOptionType.request_parameter, stream_slice)
135
136    def get_request_headers(
137        self,
138        stream_state: Optional[StreamState] = None,
139        stream_slice: Optional[StreamSlice] = None,
140        next_page_token: Optional[Mapping[str, Any]] = None,
141    ) -> Mapping[str, Any]:
142        # Pass the stream_slice from the argument, not the cursor because the cursor is updated after processing the response
143        return self._get_request_option(RequestOptionType.header, stream_slice)
144
145    def get_request_body_data(
146        self,
147        stream_state: Optional[StreamState] = None,
148        stream_slice: Optional[StreamSlice] = None,
149        next_page_token: Optional[Mapping[str, Any]] = None,
150    ) -> Mapping[str, Any]:
151        # Pass the stream_slice from the argument, not the cursor because the cursor is updated after processing the response
152        return self._get_request_option(RequestOptionType.body_data, stream_slice)
153
154    def get_request_body_json(
155        self,
156        stream_state: Optional[StreamState] = None,
157        stream_slice: Optional[StreamSlice] = None,
158        next_page_token: Optional[Mapping[str, Any]] = None,
159    ) -> Mapping[str, Any]:
160        # Pass the stream_slice from the argument, not the cursor because the cursor is updated after processing the response
161        return self._get_request_option(RequestOptionType.body_json, stream_slice)
162
163    def _get_request_option(
164        self, option_type: RequestOptionType, stream_slice: Optional[StreamSlice]
165    ) -> Mapping[str, Any]:
166        params: MutableMapping[str, Any] = {}
167        if stream_slice:
168            for parent_config in self.parent_stream_configs:
169                if (
170                    parent_config.request_option
171                    and parent_config.request_option.inject_into == option_type
172                ):
173                    key = parent_config.partition_field.eval(self.config)  # type: ignore # partition_field is always casted to an interpolated string
174                    value = stream_slice.get(key)
175                    if value:
176                        parent_config.request_option.inject_into_request(params, value, self.config)
177        return params
178
179    def stream_slices(self) -> Iterable[StreamSlice]:
180        """
181        Iterate over each parent stream's record and create a StreamSlice for each record.
182
183        For each stream, iterate over its stream_slices.
184        For each stream slice, iterate over each record.
185        yield a stream slice for each such records.
186
187        If a parent slice contains no record, emit a slice with parent_record=None.
188
189        The template string can interpolate the following values:
190        - parent_stream_slice: mapping representing the parent's stream slice
191        - parent_record: mapping representing the parent record
192        - parent_stream_name: string representing the parent stream name
193        """
194        if not self.parent_stream_configs:
195            yield from []
196        else:
197            for parent_stream_config in self.parent_stream_configs:
198                parent_stream = parent_stream_config.stream
199                parent_field = parent_stream_config.parent_key.eval(self.config)  # type: ignore # parent_key is always casted to an interpolated string
200                partition_field = parent_stream_config.partition_field.eval(self.config)  # type: ignore # partition_field is always casted to an interpolated string
201                extra_fields = None
202                if parent_stream_config.extra_fields:
203                    extra_fields = [
204                        [field_path_part.eval(self.config) for field_path_part in field_path]  # type: ignore [union-attr]
205                        for field_path in parent_stream_config.extra_fields
206                    ]
207
208                for partition, is_last_slice in iterate_with_last_flag(
209                    parent_stream.generate_partitions()
210                ):
211                    if partition is None:
212                        break
213                    for parent_record, is_last_record_in_slice in iterate_with_last_flag(
214                        partition.read()
215                    ):
216                        emit_slice = parent_record is not None
217                        if parent_record is not None:
218                            # In the previous CDK implementation, state management was done internally by the stream.
219                            # However, this could cause issues when doing availability check for example as the availability
220                            # check would progress the state so state management was moved outside of the read method.
221                            # Hence, we need to call the cursor here.
222                            # Note that we call observe and close_partition before emitting the associated record as the
223                            # ConcurrentPerPartitionCursor will associate a record with the state of the stream after the
224                            # record was consumed.
225                            parent_stream.cursor.observe(parent_record)
226                            parent_partition = (
227                                parent_record.associated_slice.partition
228                                if parent_record.associated_slice
229                                else {}
230                            )
231                            record_data = parent_record.data
232
233                            partition_value = dpath.get(
234                                record_data,  # type: ignore [arg-type]
235                                parent_field,
236                                default=None,
237                            )
238                            if partition_value is None:
239                                self.logger.debug(
240                                    f"Skipping {parent_stream.name} record with missing or null parent key '{parent_field}'"
241                                )
242                                emit_slice = False
243
244                            if emit_slice:
245                                # Add extra fields
246                                extracted_extra_fields = self._extract_extra_fields(
247                                    record_data, extra_fields
248                                )
249
250                                if parent_stream_config.lazy_read_pointer:
251                                    extracted_extra_fields = {
252                                        "child_response": self._extract_child_response(
253                                            record_data,
254                                            parent_stream_config.lazy_read_pointer,  # type: ignore[arg-type]  # lazy_read_pointer type handeled in __post_init__ of parent_stream_config
255                                        ),
256                                        **extracted_extra_fields,
257                                    }
258
259                        if is_last_record_in_slice:
260                            parent_stream.cursor.close_partition(partition)
261                            if is_last_slice:
262                                parent_stream.cursor.ensure_at_least_one_state_emitted()
263
264                        if emit_slice:
265                            yield StreamSlice(
266                                partition={
267                                    partition_field: partition_value,
268                                    "parent_slice": parent_partition or {},
269                                },
270                                cursor_slice={},
271                                extra_fields=extracted_extra_fields,
272                            )
273
274                yield from []
275
276    def _extract_child_response(
277        self, parent_record: Mapping[str, Any] | AirbyteMessage, pointer: List[InterpolatedString]
278    ) -> requests.Response:
279        """Extract child records from a parent record based on lazy pointers."""
280
281        def _create_response(data: MutableMapping[str, Any]) -> SafeResponse:
282            """Create a SafeResponse with the given data."""
283            response = SafeResponse()
284            response.content = json.dumps(data).encode("utf-8")
285            response.status_code = 200
286            return response
287
288        path = [path.eval(self.config) for path in pointer]
289        return _create_response(dpath.get(parent_record, path, default=[]))  # type: ignore # argunet will be a MutableMapping, given input data structure
290
291    def _extract_extra_fields(
292        self,
293        parent_record: Mapping[str, Any] | AirbyteMessage,
294        extra_fields: Optional[List[List[str]]] = None,
295    ) -> Mapping[str, Any]:
296        """
297        Extracts additional fields specified by their paths from the parent record.
298
299        Args:
300            parent_record (Mapping[str, Any]): The record from the parent stream to extract fields from.
301            extra_fields (Optional[List[List[str]]]): A list of field paths (as lists of strings) to extract from the parent record.
302
303        Returns:
304            Mapping[str, Any]: A dictionary containing the extracted fields.
305                               The keys are the joined field paths, and the values are the corresponding extracted values.
306        """
307        extracted_extra_fields = {}
308        if extra_fields:
309            for extra_field_path in extra_fields:
310                try:
311                    extra_field_value = dpath.get(
312                        parent_record,  # type: ignore [arg-type]
313                        extra_field_path,
314                    )
315                    self.logger.debug(
316                        f"Extracted extra_field_path: {extra_field_path} with value: {extra_field_value}"
317                    )
318                except KeyError:
319                    self.logger.debug(f"Failed to extract extra_field_path: {extra_field_path}")
320                    extra_field_value = None
321                extracted_extra_fields[".".join(extra_field_path)] = extra_field_value
322        return extracted_extra_fields
323
324    def get_stream_state(self) -> Optional[Mapping[str, StreamState]]:
325        """
326        Get the state of the parent streams.
327
328        Returns:
329            StreamState: The current state of the parent streams.
330
331        Example of state format:
332        {
333            "parent_stream_name1": {
334                "last_updated": "2023-05-27T00:00:00Z"
335            },
336            "parent_stream_name2": {
337                "last_updated": "2023-05-27T00:00:00Z"
338            }
339        }
340        """
341        parent_state = {}
342        for parent_config in self.parent_stream_configs:
343            if parent_config.incremental_dependency:
344                parent_state[parent_config.stream.name] = copy.deepcopy(
345                    parent_config.stream.cursor.state
346                )
347        return parent_state
348
349    @property
350    def logger(self) -> logging.Logger:
351        return logging.getLogger("airbyte.SubstreamPartitionRouter")

Partition router that iterates over the parent's stream records and emits slices Will populate the state with partition_field and parent_slice so they can be accessed by other components

Attributes:
  • parent_stream_configs (List[ParentStreamConfig]): parent streams to iterate over and their config
SubstreamPartitionRouter( parent_stream_configs: List[ParentStreamConfig], config: Mapping[str, Any], parameters: dataclasses.InitVar[typing.Mapping[str, typing.Any]])
parent_stream_configs: List[ParentStreamConfig]
config: Mapping[str, Any]
parameters: dataclasses.InitVar[typing.Mapping[str, typing.Any]]
def get_request_params( self, stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[StreamSlice] = None, next_page_token: Optional[Mapping[str, Any]] = None) -> Mapping[str, Any]:
127    def get_request_params(
128        self,
129        stream_state: Optional[StreamState] = None,
130        stream_slice: Optional[StreamSlice] = None,
131        next_page_token: Optional[Mapping[str, Any]] = None,
132    ) -> Mapping[str, Any]:
133        # Pass the stream_slice from the argument, not the cursor because the cursor is updated after processing the response
134        return self._get_request_option(RequestOptionType.request_parameter, stream_slice)

Specifies the query parameters that should be set on an outgoing HTTP request given the inputs.

E.g: you might want to define query parameters for paging if next_page_token is not None.

def get_request_headers( self, stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[StreamSlice] = None, next_page_token: Optional[Mapping[str, Any]] = None) -> Mapping[str, Any]:
136    def get_request_headers(
137        self,
138        stream_state: Optional[StreamState] = None,
139        stream_slice: Optional[StreamSlice] = None,
140        next_page_token: Optional[Mapping[str, Any]] = None,
141    ) -> Mapping[str, Any]:
142        # Pass the stream_slice from the argument, not the cursor because the cursor is updated after processing the response
143        return self._get_request_option(RequestOptionType.header, stream_slice)

Return any non-auth headers. Authentication headers will overwrite any overlapping headers returned from this method.

def get_request_body_data( self, stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[StreamSlice] = None, next_page_token: Optional[Mapping[str, Any]] = None) -> Mapping[str, Any]:
145    def get_request_body_data(
146        self,
147        stream_state: Optional[StreamState] = None,
148        stream_slice: Optional[StreamSlice] = None,
149        next_page_token: Optional[Mapping[str, Any]] = None,
150    ) -> Mapping[str, Any]:
151        # Pass the stream_slice from the argument, not the cursor because the cursor is updated after processing the response
152        return self._get_request_option(RequestOptionType.body_data, stream_slice)

Specifies how to populate the body of the request with a non-JSON payload.

If returns a ready text that it will be sent as is. If returns a dict that it will be converted to a urlencoded form. E.g. {"key1": "value1", "key2": "value2"} => "key1=value1&key2=value2"

At the same time only one of the 'request_body_data' and 'request_body_json' functions can be overridden.

def get_request_body_json( self, stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[StreamSlice] = None, next_page_token: Optional[Mapping[str, Any]] = None) -> Mapping[str, Any]:
154    def get_request_body_json(
155        self,
156        stream_state: Optional[StreamState] = None,
157        stream_slice: Optional[StreamSlice] = None,
158        next_page_token: Optional[Mapping[str, Any]] = None,
159    ) -> Mapping[str, Any]:
160        # Pass the stream_slice from the argument, not the cursor because the cursor is updated after processing the response
161        return self._get_request_option(RequestOptionType.body_json, stream_slice)

Specifies how to populate the body of the request with a JSON payload.

At the same time only one of the 'request_body_data' and 'request_body_json' functions can be overridden.

def stream_slices(self) -> Iterable[StreamSlice]:
179    def stream_slices(self) -> Iterable[StreamSlice]:
180        """
181        Iterate over each parent stream's record and create a StreamSlice for each record.
182
183        For each stream, iterate over its stream_slices.
184        For each stream slice, iterate over each record.
185        yield a stream slice for each such records.
186
187        If a parent slice contains no record, emit a slice with parent_record=None.
188
189        The template string can interpolate the following values:
190        - parent_stream_slice: mapping representing the parent's stream slice
191        - parent_record: mapping representing the parent record
192        - parent_stream_name: string representing the parent stream name
193        """
194        if not self.parent_stream_configs:
195            yield from []
196        else:
197            for parent_stream_config in self.parent_stream_configs:
198                parent_stream = parent_stream_config.stream
199                parent_field = parent_stream_config.parent_key.eval(self.config)  # type: ignore # parent_key is always casted to an interpolated string
200                partition_field = parent_stream_config.partition_field.eval(self.config)  # type: ignore # partition_field is always casted to an interpolated string
201                extra_fields = None
202                if parent_stream_config.extra_fields:
203                    extra_fields = [
204                        [field_path_part.eval(self.config) for field_path_part in field_path]  # type: ignore [union-attr]
205                        for field_path in parent_stream_config.extra_fields
206                    ]
207
208                for partition, is_last_slice in iterate_with_last_flag(
209                    parent_stream.generate_partitions()
210                ):
211                    if partition is None:
212                        break
213                    for parent_record, is_last_record_in_slice in iterate_with_last_flag(
214                        partition.read()
215                    ):
216                        emit_slice = parent_record is not None
217                        if parent_record is not None:
218                            # In the previous CDK implementation, state management was done internally by the stream.
219                            # However, this could cause issues when doing availability check for example as the availability
220                            # check would progress the state so state management was moved outside of the read method.
221                            # Hence, we need to call the cursor here.
222                            # Note that we call observe and close_partition before emitting the associated record as the
223                            # ConcurrentPerPartitionCursor will associate a record with the state of the stream after the
224                            # record was consumed.
225                            parent_stream.cursor.observe(parent_record)
226                            parent_partition = (
227                                parent_record.associated_slice.partition
228                                if parent_record.associated_slice
229                                else {}
230                            )
231                            record_data = parent_record.data
232
233                            partition_value = dpath.get(
234                                record_data,  # type: ignore [arg-type]
235                                parent_field,
236                                default=None,
237                            )
238                            if partition_value is None:
239                                self.logger.debug(
240                                    f"Skipping {parent_stream.name} record with missing or null parent key '{parent_field}'"
241                                )
242                                emit_slice = False
243
244                            if emit_slice:
245                                # Add extra fields
246                                extracted_extra_fields = self._extract_extra_fields(
247                                    record_data, extra_fields
248                                )
249
250                                if parent_stream_config.lazy_read_pointer:
251                                    extracted_extra_fields = {
252                                        "child_response": self._extract_child_response(
253                                            record_data,
254                                            parent_stream_config.lazy_read_pointer,  # type: ignore[arg-type]  # lazy_read_pointer type handeled in __post_init__ of parent_stream_config
255                                        ),
256                                        **extracted_extra_fields,
257                                    }
258
259                        if is_last_record_in_slice:
260                            parent_stream.cursor.close_partition(partition)
261                            if is_last_slice:
262                                parent_stream.cursor.ensure_at_least_one_state_emitted()
263
264                        if emit_slice:
265                            yield StreamSlice(
266                                partition={
267                                    partition_field: partition_value,
268                                    "parent_slice": parent_partition or {},
269                                },
270                                cursor_slice={},
271                                extra_fields=extracted_extra_fields,
272                            )
273
274                yield from []

Iterate over each parent stream's record and create a StreamSlice for each record.

For each stream, iterate over its stream_slices. For each stream slice, iterate over each record. yield a stream slice for each such records.

If a parent slice contains no record, emit a slice with parent_record=None.

The template string can interpolate the following values:

  • parent_stream_slice: mapping representing the parent's stream slice
  • parent_record: mapping representing the parent record
  • parent_stream_name: string representing the parent stream name
def get_stream_state(self) -> Optional[Mapping[str, Mapping[str, Any]]]:
324    def get_stream_state(self) -> Optional[Mapping[str, StreamState]]:
325        """
326        Get the state of the parent streams.
327
328        Returns:
329            StreamState: The current state of the parent streams.
330
331        Example of state format:
332        {
333            "parent_stream_name1": {
334                "last_updated": "2023-05-27T00:00:00Z"
335            },
336            "parent_stream_name2": {
337                "last_updated": "2023-05-27T00:00:00Z"
338            }
339        }
340        """
341        parent_state = {}
342        for parent_config in self.parent_stream_configs:
343            if parent_config.incremental_dependency:
344                parent_state[parent_config.stream.name] = copy.deepcopy(
345                    parent_config.stream.cursor.state
346                )
347        return parent_state

Get the state of the parent streams.

Returns:

StreamState: The current state of the parent streams.

Example of state format: { "parent_stream_name1": { "last_updated": "2023-05-27T00:00:00Z" }, "parent_stream_name2": { "last_updated": "2023-05-27T00:00:00Z" } }

logger: logging.Logger
349    @property
350    def logger(self) -> logging.Logger:
351        return logging.getLogger("airbyte.SubstreamPartitionRouter")
 16@dataclass
 17class UnionPartitionRouter(PartitionRouter):
 18    """
 19    A partition router that yields the deduplicated union of its child partition routers' slices.
 20
 21    Each emitted slice's partition is normalized to exactly `{partition_field: value}`. Any other
 22    partition keys coming from a child router (e.g. a SubstreamPartitionRouter's `parent_slice`)
 23    are moved into `extra_fields` so that per-partition state keys stay stable across children.
 24
 25    Given two child routers producing:
 26    A: [{"repository": "org/a"}, {"repository": "org/b"}]
 27    B: [{"repository": "org/b", "parent_slice": {...}}, {"repository": "org/c", "parent_slice": {...}}]
 28    the union yields:
 29    [{"repository": "org/a"}, {"repository": "org/b"}, {"repository": "org/c"}]
 30
 31    Partition values must be hashable — deduplication holds every distinct value in a set
 32    for the duration of slice generation, so memory grows linearly with the number of
 33    unique partitions. Note also that a union's partition count is the (deduplicated) sum
 34    of its children's, which counts against `ConcurrentPerPartitionCursor`'s per-partition
 35    state limit.
 36
 37    Attributes:
 38        partition_routers (List[PartitionRouter]): The child partition routers to union.
 39        partition_field (str): The single partition key all child slices are normalized to.
 40    """
 41
 42    partition_routers: List[PartitionRouter]
 43    partition_field: str
 44    parameters: InitVar[Mapping[str, Any]]
 45
 46    def __post_init__(self, parameters: Mapping[str, Any]) -> None:
 47        self._parameters = parameters
 48
 49    def stream_slices(self) -> Iterable[StreamSlice]:
 50        """
 51        Iterate over the child partition routers in order, yielding each partition value once.
 52
 53        The first occurrence of a partition value wins; later duplicates from any child are skipped.
 54        """
 55        # The concurrent framework consumes stream_slices() lazily, so this set stays live
 56        # (and grows with each distinct partition value) for essentially the whole sync.
 57        seen: set[Any] = set()
 58        for router in self.partition_routers:
 59            for stream_slice in router.stream_slices():
 60                if self.partition_field not in stream_slice.partition:
 61                    raise ValueError(
 62                        f"UnionPartitionRouter expects all child partition routers to emit the "
 63                        f"partition field '{self.partition_field}'. Got {stream_slice.partition}"
 64                    )
 65                value = stream_slice.partition[self.partition_field]
 66                try:
 67                    if value in seen:
 68                        continue
 69                    seen.add(value)
 70                except TypeError as exception:
 71                    raise ValueError(
 72                        f"UnionPartitionRouter can only deduplicate hashable partition values. "
 73                        f"Got unhashable value {value!r} for partition field '{self.partition_field}'."
 74                    ) from exception
 75                carried_over_fields = {
 76                    key: field_value
 77                    for key, field_value in stream_slice.partition.items()
 78                    if key != self.partition_field
 79                }
 80                yield StreamSlice(
 81                    partition={self.partition_field: value},
 82                    cursor_slice={},
 83                    extra_fields={
 84                        **(dict(stream_slice.extra_fields) if stream_slice.extra_fields else {}),
 85                        **carried_over_fields,
 86                    },
 87                )
 88
 89    def get_request_params(
 90        self,
 91        *,
 92        stream_state: Optional[StreamState] = None,
 93        stream_slice: Optional[StreamSlice] = None,
 94        next_page_token: Optional[Mapping[str, Any]] = None,
 95    ) -> Mapping[str, Any]:
 96        return {}
 97
 98    def get_request_headers(
 99        self,
100        *,
101        stream_state: Optional[StreamState] = None,
102        stream_slice: Optional[StreamSlice] = None,
103        next_page_token: Optional[Mapping[str, Any]] = None,
104    ) -> Mapping[str, Any]:
105        return {}
106
107    def get_request_body_data(
108        self,
109        *,
110        stream_state: Optional[StreamState] = None,
111        stream_slice: Optional[StreamSlice] = None,
112        next_page_token: Optional[Mapping[str, Any]] = None,
113    ) -> Mapping[str, Any]:
114        return {}
115
116    def get_request_body_json(
117        self,
118        *,
119        stream_state: Optional[StreamState] = None,
120        stream_slice: Optional[StreamSlice] = None,
121        next_page_token: Optional[Mapping[str, Any]] = None,
122    ) -> Mapping[str, Any]:
123        return {}
124
125    def get_stream_state(self) -> Optional[Mapping[str, StreamState]]:
126        """
127        Merge the parent stream states of all child partition routers.
128
129        States are merged with a shallow dict update, which assumes no two child routers
130        reference a parent stream with the same name. If they do, the last child's state
131        for that parent wins and a warning is logged.
132        """
133        merged_state: dict[str, StreamState] = {}
134        for router in self.partition_routers:
135            child_state = router.get_stream_state()
136            if child_state:
137                colliding_keys = merged_state.keys() & child_state.keys()
138                if colliding_keys:
139                    logger.warning(
140                        f"UnionPartitionRouter child routers reference the same parent stream(s) "
141                        f"{sorted(colliding_keys)}; the last child's state for each colliding parent "
142                        f"stream overwrites the previous one."
143                    )
144                merged_state.update(child_state)
145        return merged_state

A partition router that yields the deduplicated union of its child partition routers' slices.

Each emitted slice's partition is normalized to exactly {partition_field: value}. Any other partition keys coming from a child router (e.g. a SubstreamPartitionRouter's parent_slice) are moved into extra_fields so that per-partition state keys stay stable across children.

Given two child routers producing: A: [{"repository": "org/a"}, {"repository": "org/b"}] B: [{"repository": "org/b", "parent_slice": {...}}, {"repository": "org/c", "parent_slice": {...}}] the union yields: [{"repository": "org/a"}, {"repository": "org/b"}, {"repository": "org/c"}]

Partition values must be hashable — deduplication holds every distinct value in a set for the duration of slice generation, so memory grows linearly with the number of unique partitions. Note also that a union's partition count is the (deduplicated) sum of its children's, which counts against ConcurrentPerPartitionCursor's per-partition state limit.

Attributes:
  • partition_routers (List[PartitionRouter]): The child partition routers to union.
  • partition_field (str): The single partition key all child slices are normalized to.
UnionPartitionRouter( partition_routers: List[airbyte_cdk.sources.declarative.partition_routers.PartitionRouter], partition_field: str, parameters: dataclasses.InitVar[typing.Mapping[str, typing.Any]])
partition_field: str
parameters: dataclasses.InitVar[typing.Mapping[str, typing.Any]]
def stream_slices(self) -> Iterable[StreamSlice]:
49    def stream_slices(self) -> Iterable[StreamSlice]:
50        """
51        Iterate over the child partition routers in order, yielding each partition value once.
52
53        The first occurrence of a partition value wins; later duplicates from any child are skipped.
54        """
55        # The concurrent framework consumes stream_slices() lazily, so this set stays live
56        # (and grows with each distinct partition value) for essentially the whole sync.
57        seen: set[Any] = set()
58        for router in self.partition_routers:
59            for stream_slice in router.stream_slices():
60                if self.partition_field not in stream_slice.partition:
61                    raise ValueError(
62                        f"UnionPartitionRouter expects all child partition routers to emit the "
63                        f"partition field '{self.partition_field}'. Got {stream_slice.partition}"
64                    )
65                value = stream_slice.partition[self.partition_field]
66                try:
67                    if value in seen:
68                        continue
69                    seen.add(value)
70                except TypeError as exception:
71                    raise ValueError(
72                        f"UnionPartitionRouter can only deduplicate hashable partition values. "
73                        f"Got unhashable value {value!r} for partition field '{self.partition_field}'."
74                    ) from exception
75                carried_over_fields = {
76                    key: field_value
77                    for key, field_value in stream_slice.partition.items()
78                    if key != self.partition_field
79                }
80                yield StreamSlice(
81                    partition={self.partition_field: value},
82                    cursor_slice={},
83                    extra_fields={
84                        **(dict(stream_slice.extra_fields) if stream_slice.extra_fields else {}),
85                        **carried_over_fields,
86                    },
87                )

Iterate over the child partition routers in order, yielding each partition value once.

The first occurrence of a partition value wins; later duplicates from any child are skipped.

def get_request_params( self, *, stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[StreamSlice] = None, next_page_token: Optional[Mapping[str, Any]] = None) -> Mapping[str, Any]:
89    def get_request_params(
90        self,
91        *,
92        stream_state: Optional[StreamState] = None,
93        stream_slice: Optional[StreamSlice] = None,
94        next_page_token: Optional[Mapping[str, Any]] = None,
95    ) -> Mapping[str, Any]:
96        return {}

Specifies the query parameters that should be set on an outgoing HTTP request given the inputs.

E.g: you might want to define query parameters for paging if next_page_token is not None.

def get_request_headers( self, *, stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[StreamSlice] = None, next_page_token: Optional[Mapping[str, Any]] = None) -> Mapping[str, Any]:
 98    def get_request_headers(
 99        self,
100        *,
101        stream_state: Optional[StreamState] = None,
102        stream_slice: Optional[StreamSlice] = None,
103        next_page_token: Optional[Mapping[str, Any]] = None,
104    ) -> Mapping[str, Any]:
105        return {}

Return any non-auth headers. Authentication headers will overwrite any overlapping headers returned from this method.

def get_request_body_data( self, *, stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[StreamSlice] = None, next_page_token: Optional[Mapping[str, Any]] = None) -> Mapping[str, Any]:
107    def get_request_body_data(
108        self,
109        *,
110        stream_state: Optional[StreamState] = None,
111        stream_slice: Optional[StreamSlice] = None,
112        next_page_token: Optional[Mapping[str, Any]] = None,
113    ) -> Mapping[str, Any]:
114        return {}

Specifies how to populate the body of the request with a non-JSON payload.

If returns a ready text that it will be sent as is. If returns a dict that it will be converted to a urlencoded form. E.g. {"key1": "value1", "key2": "value2"} => "key1=value1&key2=value2"

At the same time only one of the 'request_body_data' and 'request_body_json' functions can be overridden.

def get_request_body_json( self, *, stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[StreamSlice] = None, next_page_token: Optional[Mapping[str, Any]] = None) -> Mapping[str, Any]:
116    def get_request_body_json(
117        self,
118        *,
119        stream_state: Optional[StreamState] = None,
120        stream_slice: Optional[StreamSlice] = None,
121        next_page_token: Optional[Mapping[str, Any]] = None,
122    ) -> Mapping[str, Any]:
123        return {}

Specifies how to populate the body of the request with a JSON payload.

At the same time only one of the 'request_body_data' and 'request_body_json' functions can be overridden.

def get_stream_state(self) -> Optional[Mapping[str, Mapping[str, Any]]]:
125    def get_stream_state(self) -> Optional[Mapping[str, StreamState]]:
126        """
127        Merge the parent stream states of all child partition routers.
128
129        States are merged with a shallow dict update, which assumes no two child routers
130        reference a parent stream with the same name. If they do, the last child's state
131        for that parent wins and a warning is logged.
132        """
133        merged_state: dict[str, StreamState] = {}
134        for router in self.partition_routers:
135            child_state = router.get_stream_state()
136            if child_state:
137                colliding_keys = merged_state.keys() & child_state.keys()
138                if colliding_keys:
139                    logger.warning(
140                        f"UnionPartitionRouter child routers reference the same parent stream(s) "
141                        f"{sorted(colliding_keys)}; the last child's state for each colliding parent "
142                        f"stream overwrites the previous one."
143                    )
144                merged_state.update(child_state)
145        return merged_state

Merge the parent stream states of all child partition routers.

States are merged with a shallow dict update, which assumes no two child routers reference a parent stream with the same name. If they do, the last child's state for that parent wins and a warning is logged.

18class YamlDeclarativeSource(ConcurrentDeclarativeSource):
19    """Declarative source defined by a yaml file"""
20
21    def __init__(
22        self,
23        path_to_yaml: str,
24        debug: bool = False,
25        catalog: Optional[ConfiguredAirbyteCatalog] = None,
26        config: Optional[Mapping[str, Any]] = None,
27        state: Optional[List[AirbyteStateMessage]] = None,
28        config_path: Optional[str] = None,
29    ) -> None:
30        """
31        :param path_to_yaml: Path to the yaml file describing the source
32        """
33        self._path_to_yaml = path_to_yaml
34        source_config = self._read_and_parse_yaml_file(path_to_yaml)
35
36        super().__init__(
37            catalog=catalog or ConfiguredAirbyteCatalog(streams=[]),
38            config=config or {},
39            state=state or [],
40            source_config=source_config,
41            config_path=config_path,
42        )
43
44    def _read_and_parse_yaml_file(self, path_to_yaml_file: str) -> ConnectionDefinition:
45        try:
46            # For testing purposes, we want to allow to just pass a file
47            with open(path_to_yaml_file, "r") as f:
48                return yaml.safe_load(f)  # type: ignore  # we assume the yaml represents a ConnectionDefinition
49        except FileNotFoundError:
50            # Running inside the container, the working directory during an operation is not structured the same as the static files
51            package = self.__class__.__module__.split(".")[0]
52
53            yaml_config = pkgutil.get_data(package, path_to_yaml_file)
54            if yaml_config:
55                decoded_yaml = yaml_config.decode()
56                return self._parse(decoded_yaml)
57            return {}
58
59    def _emit_manifest_debug_message(self, extra_args: dict[str, Any]) -> None:
60        extra_args["path_to_yaml"] = self._path_to_yaml
61
62    @staticmethod
63    def _parse(connection_definition_str: str) -> ConnectionDefinition:
64        """
65        Parses a yaml file into a manifest. Component references still exist in the manifest which will be
66        resolved during the creating of the DeclarativeSource.
67        :param connection_definition_str: yaml string to parse
68        :return: The ConnectionDefinition parsed from connection_definition_str
69        """
70        return yaml.safe_load(connection_definition_str)  # type: ignore # yaml.safe_load doesn't return a type but know it is a Mapping

Declarative source defined by a yaml file

YamlDeclarativeSource( path_to_yaml: str, debug: bool = False, catalog: Optional[airbyte_protocol_dataclasses.models.airbyte_protocol.ConfiguredAirbyteCatalog] = None, config: Optional[Mapping[str, Any]] = None, state: Optional[List[airbyte_cdk.models.airbyte_protocol.AirbyteStateMessage]] = None, config_path: Optional[str] = None)
21    def __init__(
22        self,
23        path_to_yaml: str,
24        debug: bool = False,
25        catalog: Optional[ConfiguredAirbyteCatalog] = None,
26        config: Optional[Mapping[str, Any]] = None,
27        state: Optional[List[AirbyteStateMessage]] = None,
28        config_path: Optional[str] = None,
29    ) -> None:
30        """
31        :param path_to_yaml: Path to the yaml file describing the source
32        """
33        self._path_to_yaml = path_to_yaml
34        source_config = self._read_and_parse_yaml_file(path_to_yaml)
35
36        super().__init__(
37            catalog=catalog or ConfiguredAirbyteCatalog(streams=[]),
38            config=config or {},
39            state=state or [],
40            source_config=source_config,
41            config_path=config_path,
42        )
Parameters
  • path_to_yaml: Path to the yaml file describing the source
def launch(source: Source, args: List[str]) -> None:
389def launch(source: Source, args: List[str]) -> None:
390    source_entrypoint = AirbyteEntrypoint(source)
391    parsed_args = source_entrypoint.parse_args(args)
392    # temporarily removes the PrintBuffer because we're seeing weird print behavior for concurrent syncs
393    # Refer to: https://github.com/airbytehq/oncall/issues/6235
394    with PRINT_BUFFER:
395        for message in source_entrypoint.run(parsed_args):
396            # simply printing is creating issues for concurrent CDK as Python uses different two instructions to print: one for the message and
397            # the other for the break line. Adding `\n` to the message ensure that both are printed at the same time
398            print(f"{message}\n", end="")
class AirbyteEntrypoint:
 55class AirbyteEntrypoint(object):
 56    def __init__(self, source: Source):
 57        init_uncaught_exception_handler(logger)
 58
 59        # Deployment mode is read when instantiating the entrypoint because it is the common path shared by syncs and connector builder test requests
 60        if is_cloud_environment():
 61            _init_internal_request_filter()
 62
 63        self.source = source
 64        self.logger = logging.getLogger(f"airbyte.{getattr(source, 'name', '')}")
 65        self._memory_monitor = MemoryMonitor()
 66
 67    @staticmethod
 68    def parse_args(args: List[str]) -> argparse.Namespace:
 69        # set up parent parsers
 70        parent_parser = argparse.ArgumentParser(add_help=False)
 71        parent_parser.add_argument(
 72            "--debug", action="store_true", help="enables detailed debug logs related to the sync"
 73        )
 74        main_parser = argparse.ArgumentParser()
 75        subparsers = main_parser.add_subparsers(title="commands", dest="command")
 76
 77        # spec
 78        subparsers.add_parser(
 79            "spec", help="outputs the json configuration specification", parents=[parent_parser]
 80        )
 81
 82        # check
 83        check_parser = subparsers.add_parser(
 84            "check", help="checks the config can be used to connect", parents=[parent_parser]
 85        )
 86        required_check_parser = check_parser.add_argument_group("required named arguments")
 87        required_check_parser.add_argument(
 88            "--config", type=str, required=True, help="path to the json configuration file"
 89        )
 90        check_parser.add_argument(
 91            "--manifest-path",
 92            type=str,
 93            required=False,
 94            help="path to the YAML manifest file to inject into the config",
 95        )
 96        check_parser.add_argument(
 97            "--components-path",
 98            type=str,
 99            required=False,
100            help="path to the custom components file, if it exists",
101        )
102
103        # discover
104        discover_parser = subparsers.add_parser(
105            "discover",
106            help="outputs a catalog describing the source's schema",
107            parents=[parent_parser],
108        )
109        required_discover_parser = discover_parser.add_argument_group("required named arguments")
110        required_discover_parser.add_argument(
111            "--config", type=str, required=True, help="path to the json configuration file"
112        )
113        discover_parser.add_argument(
114            "--manifest-path",
115            type=str,
116            required=False,
117            help="path to the YAML manifest file to inject into the config",
118        )
119        discover_parser.add_argument(
120            "--components-path",
121            type=str,
122            required=False,
123            help="path to the custom components file, if it exists",
124        )
125
126        # read
127        read_parser = subparsers.add_parser(
128            "read", help="reads the source and outputs messages to STDOUT", parents=[parent_parser]
129        )
130
131        read_parser.add_argument(
132            "--state", type=str, required=False, help="path to the json-encoded state file"
133        )
134        required_read_parser = read_parser.add_argument_group("required named arguments")
135        required_read_parser.add_argument(
136            "--config", type=str, required=True, help="path to the json configuration file"
137        )
138        required_read_parser.add_argument(
139            "--catalog",
140            type=str,
141            required=True,
142            help="path to the catalog used to determine which data to read",
143        )
144        read_parser.add_argument(
145            "--manifest-path",
146            type=str,
147            required=False,
148            help="path to the YAML manifest file to inject into the config",
149        )
150        read_parser.add_argument(
151            "--components-path",
152            type=str,
153            required=False,
154            help="path to the custom components file, if it exists",
155        )
156
157        return main_parser.parse_args(args)
158
159    def run(self, parsed_args: argparse.Namespace) -> Iterable[str]:
160        cmd = parsed_args.command
161        if not cmd:
162            raise Exception("No command passed")
163
164        if (hasattr(parsed_args, "debug") and parsed_args.debug) or is_platform_debug_log_enabled():
165            self.logger.setLevel(logging.DEBUG)
166            logger.setLevel(logging.DEBUG)
167            self.logger.debug("Debug logs enabled")
168        else:
169            self.logger.setLevel(logging.INFO)
170
171        source_spec: ConnectorSpecification = self.source.spec(self.logger)
172        try:
173            with tempfile.TemporaryDirectory(
174                # Cleanup can fail on Windows due to file locks. Ignore if so,
175                # rather than failing the whole process.
176                ignore_cleanup_errors=True,
177            ) as temp_dir:
178                os.environ[ENV_REQUEST_CACHE_PATH] = (
179                    temp_dir  # set this as default directory for request_cache to store *.sqlite files
180                )
181                if cmd == "spec":
182                    message = AirbyteMessage(type=Type.SPEC, spec=source_spec)
183                    yield from [
184                        self.airbyte_message_to_string(queued_message)
185                        for queued_message in self._emit_queued_messages(self.source)
186                    ]
187                    yield self.airbyte_message_to_string(message)
188                else:
189                    raw_config = self.source.read_config(parsed_args.config)
190                    config = self.source.configure(raw_config, temp_dir)
191
192                    yield from [
193                        self.airbyte_message_to_string(queued_message)
194                        for queued_message in self._emit_queued_messages(self.source)
195                    ]
196                    if cmd == "check":
197                        yield from map(
198                            AirbyteEntrypoint.airbyte_message_to_string,
199                            self.check(source_spec, config),
200                        )
201                    elif cmd == "discover":
202                        yield from map(
203                            AirbyteEntrypoint.airbyte_message_to_string,
204                            self.discover(source_spec, config),
205                        )
206                    elif cmd == "read":
207                        config_catalog = self.source.read_catalog(parsed_args.catalog)
208                        state = self.source.read_state(parsed_args.state)
209
210                        yield from map(
211                            AirbyteEntrypoint.airbyte_message_to_string,
212                            self.read(source_spec, config, config_catalog, state),
213                        )
214                    else:
215                        raise Exception("Unexpected command " + cmd)
216        finally:
217            yield from [
218                self.airbyte_message_to_string(queued_message)
219                for queued_message in self._emit_queued_messages(self.source)
220            ]
221
222    def check(
223        self, source_spec: ConnectorSpecification, config: TConfig
224    ) -> Iterable[AirbyteMessage]:
225        self.set_up_secret_filter(config, source_spec.connectionSpecification)
226        try:
227            self.validate_connection(source_spec, config)
228        except AirbyteTracedException as traced_exc:
229            connection_status = traced_exc.as_connection_status_message()
230            # The platform uses the exit code to surface unexpected failures so we raise the exception if the failure type not a config error
231            # If the failure is not exceptional, we'll emit a failed connection status message and return
232            if traced_exc.failure_type != FailureType.config_error:
233                raise traced_exc
234            if connection_status:
235                yield from self._emit_queued_messages(self.source)
236                yield connection_status
237                return
238
239        try:
240            check_result = self.source.check(self.logger, config)
241        except AirbyteTracedException as traced_exc:
242            yield traced_exc.as_sanitized_airbyte_message()
243            # The platform uses the exit code to surface unexpected failures so we raise the exception if the failure type not a config error
244            # If the failure is not exceptional, we'll emit a failed connection status message and return
245            if traced_exc.failure_type != FailureType.config_error:
246                raise traced_exc
247            else:
248                connection_status = traced_exc.as_connection_status_message()
249                if connection_status:
250                    yield connection_status
251                return
252        if check_result.status == Status.SUCCEEDED:
253            self.logger.info("Check succeeded")
254        else:
255            self.logger.error("Check failed")
256
257        yield from self._emit_queued_messages(self.source)
258        yield AirbyteMessage(type=Type.CONNECTION_STATUS, connectionStatus=check_result)
259
260    def discover(
261        self, source_spec: ConnectorSpecification, config: TConfig
262    ) -> Iterable[AirbyteMessage]:
263        self.set_up_secret_filter(config, source_spec.connectionSpecification)
264        if self.source.check_config_against_spec:
265            self.validate_connection(source_spec, config)
266        catalog = self.source.discover(self.logger, config)
267
268        yield from self._emit_queued_messages(self.source)
269        yield AirbyteMessage(type=Type.CATALOG, catalog=catalog)
270
271    def read(
272        self, source_spec: ConnectorSpecification, config: TConfig, catalog: Any, state: list[Any]
273    ) -> Iterable[AirbyteMessage]:
274        self.set_up_secret_filter(config, source_spec.connectionSpecification)
275        if self.source.check_config_against_spec:
276            self.validate_connection(source_spec, config)
277
278        # The Airbyte protocol dictates that counts be expressed as float/double to better protect against integer overflows
279        stream_message_counter: DefaultDict[HashableStreamDescriptor, float] = defaultdict(float)
280        for message in self.source.read(self.logger, config, catalog, state):
281            yield self.handle_record_counts(message, stream_message_counter)
282            try:
283                self._memory_monitor.check_memory_usage()
284            except AirbyteTracedException:
285                # Flush queued messages (state checkpoints, logs) before propagating
286                # the memory fail-fast exception, so the platform receives the last
287                # committed state for the next sync.
288                for queued_message in self._emit_queued_messages(self.source):
289                    yield self.handle_record_counts(queued_message, stream_message_counter)
290                raise
291
292        # Flush queued messages after normal completion of the read loop.
293        for message in self._emit_queued_messages(self.source):
294            yield self.handle_record_counts(message, stream_message_counter)
295
296    @staticmethod
297    def handle_record_counts(
298        message: AirbyteMessage, stream_message_count: DefaultDict[HashableStreamDescriptor, float]
299    ) -> AirbyteMessage:
300        match message.type:
301            case Type.RECORD:
302                if message.record is None:
303                    raise ValueError("Record message must have a record attribute")
304
305                stream_message_count[
306                    HashableStreamDescriptor(
307                        name=message.record.stream,  # type: ignore[union-attr] # record has `stream`
308                        namespace=message.record.namespace,  # type: ignore[union-attr] # record has `namespace`
309                    )
310                ] += 1.0
311            case Type.STATE:
312                if message.state is None:
313                    raise ValueError("State message must have a state attribute")
314
315                stream_descriptor = message_utils.get_stream_descriptor(message)
316
317                # Set record count from the counter onto the state message
318                message.state.sourceStats = message.state.sourceStats or AirbyteStateStats()  # type: ignore[union-attr] # state has `sourceStats`
319                message.state.sourceStats.recordCount = stream_message_count.get(  # type: ignore[union-attr] # state has `sourceStats`
320                    stream_descriptor, 0.0
321                )
322
323                # Reset the counter
324                stream_message_count[stream_descriptor] = 0.0
325        return message
326
327    @staticmethod
328    def validate_connection(source_spec: ConnectorSpecification, config: TConfig) -> None:
329        # Remove internal flags from config before validating so
330        # jsonschema's additionalProperties flag won't fail the validation
331        connector_config, _ = split_config(config)
332        check_config_against_spec_or_exit(connector_config, source_spec)
333
334    @staticmethod
335    def set_up_secret_filter(config: TConfig, connection_specification: Mapping[str, Any]) -> None:
336        # Now that we have the config, we can use it to get a list of ai airbyte_secrets
337        # that we should filter in logging to avoid leaking secrets
338        config_secrets = get_secrets(connection_specification, config)
339        update_secrets(config_secrets)
340
341    @staticmethod
342    def airbyte_message_to_string(airbyte_message: AirbyteMessage) -> str:
343        global _HAS_LOGGED_FOR_SERIALIZATION_ERROR
344        serialized_message = AirbyteMessageSerializer.dump(airbyte_message)
345        try:
346            return orjson.dumps(serialized_message).decode()
347        except Exception as exception:
348            if not _HAS_LOGGED_FOR_SERIALIZATION_ERROR:
349                logger.warning(
350                    f"There was an error during the serialization of an AirbyteMessage: `{exception}`. This might impact the sync performances."
351                )
352                _HAS_LOGGED_FOR_SERIALIZATION_ERROR = True
353            try:
354                return json.dumps(serialized_message)
355            except Exception as json_exception:
356                raise AirbyteTracedException(
357                    internal_message=f"Failed to serialize AirbyteMessage to JSON: `{json_exception}`",
358                    failure_type=FailureType.system_error,
359                    message="A record returned from the API failed to be serialized to JSON.",
360                ) from json_exception
361
362    @classmethod
363    def extract_state(cls, args: List[str]) -> Optional[Any]:
364        parsed_args = cls.parse_args(args)
365        if hasattr(parsed_args, "state"):
366            return parsed_args.state
367        return None
368
369    @classmethod
370    def extract_catalog(cls, args: List[str]) -> Optional[Any]:
371        parsed_args = cls.parse_args(args)
372        if hasattr(parsed_args, "catalog"):
373            return parsed_args.catalog
374        return None
375
376    @classmethod
377    def extract_config(cls, args: List[str]) -> Optional[Any]:
378        parsed_args = cls.parse_args(args)
379        if hasattr(parsed_args, "config"):
380            return parsed_args.config
381        return None
382
383    def _emit_queued_messages(self, source: Source) -> Iterable[AirbyteMessage]:
384        if hasattr(source, "message_repository") and source.message_repository:
385            yield from source.message_repository.consume_queue()
386        return
AirbyteEntrypoint(source: Source)
56    def __init__(self, source: Source):
57        init_uncaught_exception_handler(logger)
58
59        # Deployment mode is read when instantiating the entrypoint because it is the common path shared by syncs and connector builder test requests
60        if is_cloud_environment():
61            _init_internal_request_filter()
62
63        self.source = source
64        self.logger = logging.getLogger(f"airbyte.{getattr(source, 'name', '')}")
65        self._memory_monitor = MemoryMonitor()
source
logger
@staticmethod
def parse_args(args: List[str]) -> argparse.Namespace:
 67    @staticmethod
 68    def parse_args(args: List[str]) -> argparse.Namespace:
 69        # set up parent parsers
 70        parent_parser = argparse.ArgumentParser(add_help=False)
 71        parent_parser.add_argument(
 72            "--debug", action="store_true", help="enables detailed debug logs related to the sync"
 73        )
 74        main_parser = argparse.ArgumentParser()
 75        subparsers = main_parser.add_subparsers(title="commands", dest="command")
 76
 77        # spec
 78        subparsers.add_parser(
 79            "spec", help="outputs the json configuration specification", parents=[parent_parser]
 80        )
 81
 82        # check
 83        check_parser = subparsers.add_parser(
 84            "check", help="checks the config can be used to connect", parents=[parent_parser]
 85        )
 86        required_check_parser = check_parser.add_argument_group("required named arguments")
 87        required_check_parser.add_argument(
 88            "--config", type=str, required=True, help="path to the json configuration file"
 89        )
 90        check_parser.add_argument(
 91            "--manifest-path",
 92            type=str,
 93            required=False,
 94            help="path to the YAML manifest file to inject into the config",
 95        )
 96        check_parser.add_argument(
 97            "--components-path",
 98            type=str,
 99            required=False,
100            help="path to the custom components file, if it exists",
101        )
102
103        # discover
104        discover_parser = subparsers.add_parser(
105            "discover",
106            help="outputs a catalog describing the source's schema",
107            parents=[parent_parser],
108        )
109        required_discover_parser = discover_parser.add_argument_group("required named arguments")
110        required_discover_parser.add_argument(
111            "--config", type=str, required=True, help="path to the json configuration file"
112        )
113        discover_parser.add_argument(
114            "--manifest-path",
115            type=str,
116            required=False,
117            help="path to the YAML manifest file to inject into the config",
118        )
119        discover_parser.add_argument(
120            "--components-path",
121            type=str,
122            required=False,
123            help="path to the custom components file, if it exists",
124        )
125
126        # read
127        read_parser = subparsers.add_parser(
128            "read", help="reads the source and outputs messages to STDOUT", parents=[parent_parser]
129        )
130
131        read_parser.add_argument(
132            "--state", type=str, required=False, help="path to the json-encoded state file"
133        )
134        required_read_parser = read_parser.add_argument_group("required named arguments")
135        required_read_parser.add_argument(
136            "--config", type=str, required=True, help="path to the json configuration file"
137        )
138        required_read_parser.add_argument(
139            "--catalog",
140            type=str,
141            required=True,
142            help="path to the catalog used to determine which data to read",
143        )
144        read_parser.add_argument(
145            "--manifest-path",
146            type=str,
147            required=False,
148            help="path to the YAML manifest file to inject into the config",
149        )
150        read_parser.add_argument(
151            "--components-path",
152            type=str,
153            required=False,
154            help="path to the custom components file, if it exists",
155        )
156
157        return main_parser.parse_args(args)
def run(self, parsed_args: argparse.Namespace) -> Iterable[str]:
159    def run(self, parsed_args: argparse.Namespace) -> Iterable[str]:
160        cmd = parsed_args.command
161        if not cmd:
162            raise Exception("No command passed")
163
164        if (hasattr(parsed_args, "debug") and parsed_args.debug) or is_platform_debug_log_enabled():
165            self.logger.setLevel(logging.DEBUG)
166            logger.setLevel(logging.DEBUG)
167            self.logger.debug("Debug logs enabled")
168        else:
169            self.logger.setLevel(logging.INFO)
170
171        source_spec: ConnectorSpecification = self.source.spec(self.logger)
172        try:
173            with tempfile.TemporaryDirectory(
174                # Cleanup can fail on Windows due to file locks. Ignore if so,
175                # rather than failing the whole process.
176                ignore_cleanup_errors=True,
177            ) as temp_dir:
178                os.environ[ENV_REQUEST_CACHE_PATH] = (
179                    temp_dir  # set this as default directory for request_cache to store *.sqlite files
180                )
181                if cmd == "spec":
182                    message = AirbyteMessage(type=Type.SPEC, spec=source_spec)
183                    yield from [
184                        self.airbyte_message_to_string(queued_message)
185                        for queued_message in self._emit_queued_messages(self.source)
186                    ]
187                    yield self.airbyte_message_to_string(message)
188                else:
189                    raw_config = self.source.read_config(parsed_args.config)
190                    config = self.source.configure(raw_config, temp_dir)
191
192                    yield from [
193                        self.airbyte_message_to_string(queued_message)
194                        for queued_message in self._emit_queued_messages(self.source)
195                    ]
196                    if cmd == "check":
197                        yield from map(
198                            AirbyteEntrypoint.airbyte_message_to_string,
199                            self.check(source_spec, config),
200                        )
201                    elif cmd == "discover":
202                        yield from map(
203                            AirbyteEntrypoint.airbyte_message_to_string,
204                            self.discover(source_spec, config),
205                        )
206                    elif cmd == "read":
207                        config_catalog = self.source.read_catalog(parsed_args.catalog)
208                        state = self.source.read_state(parsed_args.state)
209
210                        yield from map(
211                            AirbyteEntrypoint.airbyte_message_to_string,
212                            self.read(source_spec, config, config_catalog, state),
213                        )
214                    else:
215                        raise Exception("Unexpected command " + cmd)
216        finally:
217            yield from [
218                self.airbyte_message_to_string(queued_message)
219                for queued_message in self._emit_queued_messages(self.source)
220            ]
def check( self, source_spec: ConnectorSpecification, config: ~TConfig) -> Iterable[AirbyteMessage]:
222    def check(
223        self, source_spec: ConnectorSpecification, config: TConfig
224    ) -> Iterable[AirbyteMessage]:
225        self.set_up_secret_filter(config, source_spec.connectionSpecification)
226        try:
227            self.validate_connection(source_spec, config)
228        except AirbyteTracedException as traced_exc:
229            connection_status = traced_exc.as_connection_status_message()
230            # The platform uses the exit code to surface unexpected failures so we raise the exception if the failure type not a config error
231            # If the failure is not exceptional, we'll emit a failed connection status message and return
232            if traced_exc.failure_type != FailureType.config_error:
233                raise traced_exc
234            if connection_status:
235                yield from self._emit_queued_messages(self.source)
236                yield connection_status
237                return
238
239        try:
240            check_result = self.source.check(self.logger, config)
241        except AirbyteTracedException as traced_exc:
242            yield traced_exc.as_sanitized_airbyte_message()
243            # The platform uses the exit code to surface unexpected failures so we raise the exception if the failure type not a config error
244            # If the failure is not exceptional, we'll emit a failed connection status message and return
245            if traced_exc.failure_type != FailureType.config_error:
246                raise traced_exc
247            else:
248                connection_status = traced_exc.as_connection_status_message()
249                if connection_status:
250                    yield connection_status
251                return
252        if check_result.status == Status.SUCCEEDED:
253            self.logger.info("Check succeeded")
254        else:
255            self.logger.error("Check failed")
256
257        yield from self._emit_queued_messages(self.source)
258        yield AirbyteMessage(type=Type.CONNECTION_STATUS, connectionStatus=check_result)
def discover( self, source_spec: ConnectorSpecification, config: ~TConfig) -> Iterable[AirbyteMessage]:
260    def discover(
261        self, source_spec: ConnectorSpecification, config: TConfig
262    ) -> Iterable[AirbyteMessage]:
263        self.set_up_secret_filter(config, source_spec.connectionSpecification)
264        if self.source.check_config_against_spec:
265            self.validate_connection(source_spec, config)
266        catalog = self.source.discover(self.logger, config)
267
268        yield from self._emit_queued_messages(self.source)
269        yield AirbyteMessage(type=Type.CATALOG, catalog=catalog)
def read( self, source_spec: ConnectorSpecification, config: ~TConfig, catalog: Any, state: list[typing.Any]) -> Iterable[AirbyteMessage]:
271    def read(
272        self, source_spec: ConnectorSpecification, config: TConfig, catalog: Any, state: list[Any]
273    ) -> Iterable[AirbyteMessage]:
274        self.set_up_secret_filter(config, source_spec.connectionSpecification)
275        if self.source.check_config_against_spec:
276            self.validate_connection(source_spec, config)
277
278        # The Airbyte protocol dictates that counts be expressed as float/double to better protect against integer overflows
279        stream_message_counter: DefaultDict[HashableStreamDescriptor, float] = defaultdict(float)
280        for message in self.source.read(self.logger, config, catalog, state):
281            yield self.handle_record_counts(message, stream_message_counter)
282            try:
283                self._memory_monitor.check_memory_usage()
284            except AirbyteTracedException:
285                # Flush queued messages (state checkpoints, logs) before propagating
286                # the memory fail-fast exception, so the platform receives the last
287                # committed state for the next sync.
288                for queued_message in self._emit_queued_messages(self.source):
289                    yield self.handle_record_counts(queued_message, stream_message_counter)
290                raise
291
292        # Flush queued messages after normal completion of the read loop.
293        for message in self._emit_queued_messages(self.source):
294            yield self.handle_record_counts(message, stream_message_counter)
@staticmethod
def handle_record_counts( message: AirbyteMessage, stream_message_count: DefaultDict[airbyte_cdk.sources.connector_state_manager.HashableStreamDescriptor, float]) -> AirbyteMessage:
296    @staticmethod
297    def handle_record_counts(
298        message: AirbyteMessage, stream_message_count: DefaultDict[HashableStreamDescriptor, float]
299    ) -> AirbyteMessage:
300        match message.type:
301            case Type.RECORD:
302                if message.record is None:
303                    raise ValueError("Record message must have a record attribute")
304
305                stream_message_count[
306                    HashableStreamDescriptor(
307                        name=message.record.stream,  # type: ignore[union-attr] # record has `stream`
308                        namespace=message.record.namespace,  # type: ignore[union-attr] # record has `namespace`
309                    )
310                ] += 1.0
311            case Type.STATE:
312                if message.state is None:
313                    raise ValueError("State message must have a state attribute")
314
315                stream_descriptor = message_utils.get_stream_descriptor(message)
316
317                # Set record count from the counter onto the state message
318                message.state.sourceStats = message.state.sourceStats or AirbyteStateStats()  # type: ignore[union-attr] # state has `sourceStats`
319                message.state.sourceStats.recordCount = stream_message_count.get(  # type: ignore[union-attr] # state has `sourceStats`
320                    stream_descriptor, 0.0
321                )
322
323                # Reset the counter
324                stream_message_count[stream_descriptor] = 0.0
325        return message
@staticmethod
def validate_connection( source_spec: ConnectorSpecification, config: ~TConfig) -> None:
327    @staticmethod
328    def validate_connection(source_spec: ConnectorSpecification, config: TConfig) -> None:
329        # Remove internal flags from config before validating so
330        # jsonschema's additionalProperties flag won't fail the validation
331        connector_config, _ = split_config(config)
332        check_config_against_spec_or_exit(connector_config, source_spec)
@staticmethod
def set_up_secret_filter(config: ~TConfig, connection_specification: Mapping[str, Any]) -> None:
334    @staticmethod
335    def set_up_secret_filter(config: TConfig, connection_specification: Mapping[str, Any]) -> None:
336        # Now that we have the config, we can use it to get a list of ai airbyte_secrets
337        # that we should filter in logging to avoid leaking secrets
338        config_secrets = get_secrets(connection_specification, config)
339        update_secrets(config_secrets)
@staticmethod
def airbyte_message_to_string( airbyte_message: AirbyteMessage) -> str:
341    @staticmethod
342    def airbyte_message_to_string(airbyte_message: AirbyteMessage) -> str:
343        global _HAS_LOGGED_FOR_SERIALIZATION_ERROR
344        serialized_message = AirbyteMessageSerializer.dump(airbyte_message)
345        try:
346            return orjson.dumps(serialized_message).decode()
347        except Exception as exception:
348            if not _HAS_LOGGED_FOR_SERIALIZATION_ERROR:
349                logger.warning(
350                    f"There was an error during the serialization of an AirbyteMessage: `{exception}`. This might impact the sync performances."
351                )
352                _HAS_LOGGED_FOR_SERIALIZATION_ERROR = True
353            try:
354                return json.dumps(serialized_message)
355            except Exception as json_exception:
356                raise AirbyteTracedException(
357                    internal_message=f"Failed to serialize AirbyteMessage to JSON: `{json_exception}`",
358                    failure_type=FailureType.system_error,
359                    message="A record returned from the API failed to be serialized to JSON.",
360                ) from json_exception
@classmethod
def extract_state(cls, args: List[str]) -> Optional[Any]:
362    @classmethod
363    def extract_state(cls, args: List[str]) -> Optional[Any]:
364        parsed_args = cls.parse_args(args)
365        if hasattr(parsed_args, "state"):
366            return parsed_args.state
367        return None
@classmethod
def extract_catalog(cls, args: List[str]) -> Optional[Any]:
369    @classmethod
370    def extract_catalog(cls, args: List[str]) -> Optional[Any]:
371        parsed_args = cls.parse_args(args)
372        if hasattr(parsed_args, "catalog"):
373            return parsed_args.catalog
374        return None
@classmethod
def extract_config(cls, args: List[str]) -> Optional[Any]:
376    @classmethod
377    def extract_config(cls, args: List[str]) -> Optional[Any]:
378        parsed_args = cls.parse_args(args)
379        if hasattr(parsed_args, "config"):
380            return parsed_args.config
381        return None
class AbstractAPIBudget(abc.ABC):
528class AbstractAPIBudget(abc.ABC):
529    """Interface to some API where a client allowed to have N calls per T interval.
530
531    Important: APIBudget is not doing any API calls, the end user code is responsible to call this interface
532        to respect call rate limitation of the API.
533
534    It supports multiple policies applied to different group of requests. To distinct these groups we use RequestMatchers.
535    Individual policy represented by MovingWindowCallRatePolicy and currently supports only moving window strategy.
536    """
537
538    @abc.abstractmethod
539    def acquire_call(
540        self, request: Any, block: bool = True, timeout: Optional[float] = None
541    ) -> None:
542        """Try to get a call from budget, will block by default
543
544        :param request:
545        :param block: when true (default) will block the current thread until call credit is available
546        :param timeout: if set will limit maximum time in block, otherwise will wait until credit is available
547        :raises: CallRateLimitHit - when no credits left and if timeout was set the waiting time exceed the timeout
548        """
549
550    @abc.abstractmethod
551    def get_matching_policy(self, request: Any) -> Optional[AbstractCallRatePolicy]:
552        """Find matching call rate policy for specific request"""
553
554    @abc.abstractmethod
555    def update_from_response(self, request: Any, response: Any) -> None:
556        """Update budget information based on response from API
557
558        :param request: the initial request that triggered this response
559        :param response: response from the API
560        """

Interface to some API where a client allowed to have N calls per T interval.

Important: APIBudget is not doing any API calls, the end user code is responsible to call this interface to respect call rate limitation of the API.

It supports multiple policies applied to different group of requests. To distinct these groups we use RequestMatchers. Individual policy represented by MovingWindowCallRatePolicy and currently supports only moving window strategy.

@abc.abstractmethod
def acquire_call( self, request: Any, block: bool = True, timeout: Optional[float] = None) -> None:
538    @abc.abstractmethod
539    def acquire_call(
540        self, request: Any, block: bool = True, timeout: Optional[float] = None
541    ) -> None:
542        """Try to get a call from budget, will block by default
543
544        :param request:
545        :param block: when true (default) will block the current thread until call credit is available
546        :param timeout: if set will limit maximum time in block, otherwise will wait until credit is available
547        :raises: CallRateLimitHit - when no credits left and if timeout was set the waiting time exceed the timeout
548        """

Try to get a call from budget, will block by default

Parameters
  • request:
  • block: when true (default) will block the current thread until call credit is available
  • timeout: if set will limit maximum time in block, otherwise will wait until credit is available
Raises
  • CallRateLimitHit - when no credits left and if timeout was set the waiting time exceed the timeout
@abc.abstractmethod
def get_matching_policy( self, request: Any) -> Optional[airbyte_cdk.sources.streams.call_rate.AbstractCallRatePolicy]:
550    @abc.abstractmethod
551    def get_matching_policy(self, request: Any) -> Optional[AbstractCallRatePolicy]:
552        """Find matching call rate policy for specific request"""

Find matching call rate policy for specific request

@abc.abstractmethod
def update_from_response(self, request: Any, response: Any) -> None:
554    @abc.abstractmethod
555    def update_from_response(self, request: Any, response: Any) -> None:
556        """Update budget information based on response from API
557
558        :param request: the initial request that triggered this response
559        :param response: response from the API
560        """

Update budget information based on response from API

Parameters
  • request: the initial request that triggered this response
  • response: response from the API
class AbstractHeaderAuthenticator(requests.auth.AuthBase):
13class AbstractHeaderAuthenticator(AuthBase):
14    """Abstract class for an header-based authenticators that add a header to outgoing HTTP requests."""
15
16    def __call__(self, request: requests.PreparedRequest) -> Any:
17        """Attach the HTTP headers required to authenticate on the HTTP request"""
18        request.headers.update(self.get_auth_header())
19        return request
20
21    def get_auth_header(self) -> Mapping[str, Any]:
22        """The header to set on outgoing HTTP requests"""
23        if self.auth_header:
24            return {self.auth_header: self.token}
25        return {}
26
27    @property
28    @abstractmethod
29    def auth_header(self) -> str:
30        """HTTP header to set on the requests"""
31
32    @property
33    @abstractmethod
34    def token(self) -> str:
35        """The header value to set on outgoing HTTP requests"""

Abstract class for an header-based authenticators that add a header to outgoing HTTP requests.

def get_auth_header(self) -> Mapping[str, Any]:
21    def get_auth_header(self) -> Mapping[str, Any]:
22        """The header to set on outgoing HTTP requests"""
23        if self.auth_header:
24            return {self.auth_header: self.token}
25        return {}

The header to set on outgoing HTTP requests

auth_header: str
27    @property
28    @abstractmethod
29    def auth_header(self) -> str:
30        """HTTP header to set on the requests"""

HTTP header to set on the requests

token: str
32    @property
33    @abstractmethod
34    def token(self) -> str:
35        """The header value to set on outgoing HTTP requests"""

The header value to set on outgoing HTTP requests

class BaseBackoffException(requests.exceptions.HTTPError):
14class BaseBackoffException(requests.exceptions.HTTPError):
15    def __init__(
16        self,
17        request: requests.PreparedRequest,
18        response: Optional[Union[requests.Response, Exception]],
19        error_message: str = "",
20        failure_type: Optional[FailureType] = None,
21    ):
22        self.failure_type = failure_type
23        if isinstance(response, requests.Response):
24            error_message = (
25                error_message
26                or f"Request URL: {request.url}, Response Code: {response.status_code}, Response Text: {response.text}"
27            )
28            super().__init__(error_message, request=request, response=response)
29        else:
30            error_message = error_message or f"Request URL: {request.url}, Exception: {response}"
31            super().__init__(error_message, request=request, response=None)

An HTTP error occurred.

BaseBackoffException( request: requests.models.PreparedRequest, response: Union[requests.models.Response, Exception, NoneType], error_message: str = '', failure_type: Optional[airbyte_protocol_dataclasses.models.airbyte_protocol.FailureType] = None)
15    def __init__(
16        self,
17        request: requests.PreparedRequest,
18        response: Optional[Union[requests.Response, Exception]],
19        error_message: str = "",
20        failure_type: Optional[FailureType] = None,
21    ):
22        self.failure_type = failure_type
23        if isinstance(response, requests.Response):
24            error_message = (
25                error_message
26                or f"Request URL: {request.url}, Response Code: {response.status_code}, Response Text: {response.text}"
27            )
28            super().__init__(error_message, request=request, response=response)
29        else:
30            error_message = error_message or f"Request URL: {request.url}, Exception: {response}"
31            super().__init__(error_message, request=request, response=None)

Initialize RequestException with request and response objects.

failure_type
class CachedLimiterSession(requests_cache.session.CacheMixin, airbyte_cdk.sources.streams.call_rate.LimiterMixin, requests.sessions.Session):
754class CachedLimiterSession(requests_cache.CacheMixin, LimiterMixin, requests.Session):
755    """Session class with caching and rate-limiting behavior."""

Session class with caching and rate-limiting behavior.

class DefaultBackoffException(airbyte_cdk.BaseBackoffException):
67class DefaultBackoffException(BaseBackoffException):
68    pass

An HTTP error occurred.

def default_backoff_handler( max_tries: Optional[int], factor: float, max_time: Optional[int] = None, **kwargs: Any) -> Callable[[Callable[[requests.models.PreparedRequest, Mapping[str, Any]], requests.models.Response]], Callable[[requests.models.PreparedRequest, Mapping[str, Any]], requests.models.Response]]:
34def default_backoff_handler(
35    max_tries: Optional[int], factor: float, max_time: Optional[int] = None, **kwargs: Any
36) -> Callable[[SendRequestCallableType], SendRequestCallableType]:
37    def log_retry_attempt(details: Mapping[str, Any]) -> None:
38        _, exc, _ = sys.exc_info()
39        if isinstance(exc, RequestException) and exc.response:
40            logger.info(
41                f"Status code: {exc.response.status_code!r}, Response Content: {exc.response.content!r}"
42            )
43        logger.info(
44            f"Caught retryable error '{str(exc)}' after {details['tries']} tries. Waiting {details['wait']} seconds then retrying..."
45        )
46
47    def should_give_up(exc: Exception) -> bool:
48        # If a non-rate-limiting related 4XX error makes it this far, it means it was unexpected and probably consistent, so we shouldn't back off
49        if isinstance(exc, RequestException):
50            if exc.response is not None:
51                give_up: bool = (
52                    exc.response is not None
53                    and exc.response.status_code != codes.too_many_requests
54                    and 400 <= exc.response.status_code < 500
55                )
56                if give_up:
57                    logger.info(f"Giving up for returned HTTP status: {exc.response.status_code!r}")
58                return give_up
59        # Only RequestExceptions are retryable, so if we get here, it's not retryable
60        return False
61
62    return backoff.on_exception(  # type: ignore # Decorator function returns a function with a different signature than the input function, so mypy can't infer the type of the returned function
63        backoff.expo,
64        TRANSIENT_EXCEPTIONS,
65        jitter=None,
66        on_backoff=log_retry_attempt,
67        giveup=should_give_up,
68        max_tries=max_tries,
69        max_time=max_time,
70        factor=factor,
71        **kwargs,
72    )
class HttpAPIBudget(airbyte_cdk.sources.streams.call_rate.APIBudget):
681class HttpAPIBudget(APIBudget):
682    """Implementation of AbstractAPIBudget for HTTP"""
683
684    def __init__(
685        self,
686        ratelimit_reset_header: str = "ratelimit-reset",
687        ratelimit_remaining_header: str = "ratelimit-remaining",
688        status_codes_for_ratelimit_hit: list[int] = [429],
689        **kwargs: Any,
690    ):
691        """Constructor
692
693        :param ratelimit_reset_header: name of the header that has a timestamp of the next reset of call budget
694        :param ratelimit_remaining_header: name of the header that has the number of calls left
695        :param status_codes_for_ratelimit_hit: list of HTTP status codes that signal about rate limit being hit
696        """
697        self._ratelimit_reset_header = ratelimit_reset_header
698        self._ratelimit_remaining_header = ratelimit_remaining_header
699        self._status_codes_for_ratelimit_hit = status_codes_for_ratelimit_hit
700        super().__init__(**kwargs)
701
702    def update_from_response(self, request: Any, response: Any) -> None:
703        policy = self.get_matching_policy(request)
704        if not policy:
705            return
706
707        if isinstance(response, requests.Response):
708            available_calls = self.get_calls_left_from_response(response)
709            reset_ts = self.get_reset_ts_from_response(response)
710            policy.update(available_calls=available_calls, call_reset_ts=reset_ts)
711
712    def get_reset_ts_from_response(
713        self, response: requests.Response
714    ) -> Optional[datetime.datetime]:
715        if response.headers.get(self._ratelimit_reset_header):
716            return datetime.datetime.fromtimestamp(
717                int(response.headers[self._ratelimit_reset_header])
718            )
719        return None
720
721    def get_calls_left_from_response(self, response: requests.Response) -> Optional[int]:
722        if response.headers.get(self._ratelimit_remaining_header):
723            return int(response.headers[self._ratelimit_remaining_header])
724
725        if response.status_code in self._status_codes_for_ratelimit_hit:
726            return 0
727
728        return None

Implementation of AbstractAPIBudget for HTTP

HttpAPIBudget( ratelimit_reset_header: str = 'ratelimit-reset', ratelimit_remaining_header: str = 'ratelimit-remaining', status_codes_for_ratelimit_hit: list[int] = [429], **kwargs: Any)
684    def __init__(
685        self,
686        ratelimit_reset_header: str = "ratelimit-reset",
687        ratelimit_remaining_header: str = "ratelimit-remaining",
688        status_codes_for_ratelimit_hit: list[int] = [429],
689        **kwargs: Any,
690    ):
691        """Constructor
692
693        :param ratelimit_reset_header: name of the header that has a timestamp of the next reset of call budget
694        :param ratelimit_remaining_header: name of the header that has the number of calls left
695        :param status_codes_for_ratelimit_hit: list of HTTP status codes that signal about rate limit being hit
696        """
697        self._ratelimit_reset_header = ratelimit_reset_header
698        self._ratelimit_remaining_header = ratelimit_remaining_header
699        self._status_codes_for_ratelimit_hit = status_codes_for_ratelimit_hit
700        super().__init__(**kwargs)

Constructor

Parameters
  • ratelimit_reset_header: name of the header that has a timestamp of the next reset of call budget
  • ratelimit_remaining_header: name of the header that has the number of calls left
  • status_codes_for_ratelimit_hit: list of HTTP status codes that signal about rate limit being hit
def update_from_response(self, request: Any, response: Any) -> None:
702    def update_from_response(self, request: Any, response: Any) -> None:
703        policy = self.get_matching_policy(request)
704        if not policy:
705            return
706
707        if isinstance(response, requests.Response):
708            available_calls = self.get_calls_left_from_response(response)
709            reset_ts = self.get_reset_ts_from_response(response)
710            policy.update(available_calls=available_calls, call_reset_ts=reset_ts)

Update budget information based on the API response.

Parameters
  • request: the initial request that triggered this response
  • response: response from the API
def get_reset_ts_from_response(self, response: requests.models.Response) -> Optional[datetime.datetime]:
712    def get_reset_ts_from_response(
713        self, response: requests.Response
714    ) -> Optional[datetime.datetime]:
715        if response.headers.get(self._ratelimit_reset_header):
716            return datetime.datetime.fromtimestamp(
717                int(response.headers[self._ratelimit_reset_header])
718            )
719        return None
def get_calls_left_from_response(self, response: requests.models.Response) -> Optional[int]:
721    def get_calls_left_from_response(self, response: requests.Response) -> Optional[int]:
722        if response.headers.get(self._ratelimit_remaining_header):
723            return int(response.headers[self._ratelimit_remaining_header])
724
725        if response.status_code in self._status_codes_for_ratelimit_hit:
726            return 0
727
728        return None
HttpAuthenticator
class HttpRequestMatcher(airbyte_cdk.sources.streams.call_rate.RequestMatcher):
103class HttpRequestMatcher(RequestMatcher):
104    """Simple implementation of RequestMatcher for HTTP requests using HttpRequestRegexMatcher under the hood."""
105
106    def __init__(
107        self,
108        method: Optional[str] = None,
109        url: Optional[str] = None,
110        params: Optional[Mapping[str, Any]] = None,
111        headers: Optional[Mapping[str, Any]] = None,
112    ):
113        """Constructor
114
115        :param method: HTTP method (e.g., "GET", "POST").
116        :param url: Full URL to match.
117        :param params: Dictionary of query parameters to match.
118        :param headers: Dictionary of headers to match.
119        """
120        # Parse the URL to extract the base and path
121        if url:
122            parsed_url = parse.urlsplit(url)
123            url_base = f"{parsed_url.scheme}://{parsed_url.netloc}"
124            url_path = parsed_url.path if parsed_url.path != "/" else None
125        else:
126            url_base = None
127            url_path = None
128
129        # Use HttpRequestRegexMatcher under the hood
130        self._regex_matcher = HttpRequestRegexMatcher(
131            method=method,
132            url_base=url_base,
133            url_path_pattern=re.escape(url_path) if url_path else None,
134            params=params,
135            headers=headers,
136        )
137
138    def __call__(self, request: Any) -> bool:
139        """
140        :param request: A requests.Request or requests.PreparedRequest instance.
141        :return: True if the request matches all provided criteria; False otherwise.
142        """
143        return self._regex_matcher(request)
144
145    def __str__(self) -> str:
146        return (
147            f"HttpRequestMatcher(method={self._regex_matcher._method}, "
148            f"url={self._regex_matcher._url_base}{self._regex_matcher._url_path_pattern.pattern if self._regex_matcher._url_path_pattern else ''}, "
149            f"params={self._regex_matcher._params}, headers={self._regex_matcher._headers})"
150        )

Simple implementation of RequestMatcher for HTTP requests using HttpRequestRegexMatcher under the hood.

HttpRequestMatcher( method: Optional[str] = None, url: Optional[str] = None, params: Optional[Mapping[str, Any]] = None, headers: Optional[Mapping[str, Any]] = None)
106    def __init__(
107        self,
108        method: Optional[str] = None,
109        url: Optional[str] = None,
110        params: Optional[Mapping[str, Any]] = None,
111        headers: Optional[Mapping[str, Any]] = None,
112    ):
113        """Constructor
114
115        :param method: HTTP method (e.g., "GET", "POST").
116        :param url: Full URL to match.
117        :param params: Dictionary of query parameters to match.
118        :param headers: Dictionary of headers to match.
119        """
120        # Parse the URL to extract the base and path
121        if url:
122            parsed_url = parse.urlsplit(url)
123            url_base = f"{parsed_url.scheme}://{parsed_url.netloc}"
124            url_path = parsed_url.path if parsed_url.path != "/" else None
125        else:
126            url_base = None
127            url_path = None
128
129        # Use HttpRequestRegexMatcher under the hood
130        self._regex_matcher = HttpRequestRegexMatcher(
131            method=method,
132            url_base=url_base,
133            url_path_pattern=re.escape(url_path) if url_path else None,
134            params=params,
135            headers=headers,
136        )

Constructor

Parameters
  • method: HTTP method (e.g., "GET", "POST").
  • url: Full URL to match.
  • params: Dictionary of query parameters to match.
  • headers: Dictionary of headers to match.
class HttpStream(airbyte_cdk.Stream, airbyte_cdk.sources.streams.core.CheckpointMixin, abc.ABC):
 45class HttpStream(Stream, CheckpointMixin, ABC):
 46    """
 47    Base abstract class for an Airbyte Stream using the HTTP protocol. Basic building block for users building an Airbyte source for a HTTP API.
 48    """
 49
 50    source_defined_cursor = True  # Most HTTP streams use a source defined cursor (i.e: the user can't configure it like on a SQL table)
 51    page_size: Optional[int] = (
 52        None  # Use this variable to define page size for API http requests with pagination support
 53    )
 54
 55    def __init__(
 56        self, authenticator: Optional[AuthBase] = None, api_budget: Optional[APIBudget] = None
 57    ):
 58        self._exit_on_rate_limit: bool = False
 59        self._http_client = HttpClient(
 60            name=self.name,
 61            logger=self.logger,
 62            error_handler=self.get_error_handler(),
 63            api_budget=api_budget or APIBudget(policies=[]),
 64            authenticator=authenticator,
 65            use_cache=self.use_cache,
 66            backoff_strategy=self.get_backoff_strategy(),
 67            message_repository=InMemoryMessageRepository(),
 68        )
 69
 70        # There are three conditions that dictate if RFR should automatically be applied to a stream
 71        # 1. Streams that explicitly initialize their own cursor should defer to it and not automatically apply RFR
 72        # 2. Streams with at least one cursor_field are incremental and thus a superior sync to RFR.
 73        # 3. Streams overriding read_records() do not guarantee that they will call the parent implementation which can perform
 74        #    per-page checkpointing so RFR is only supported if a stream use the default `HttpStream.read_records()` method
 75        if (
 76            not self.cursor
 77            and len(self.cursor_field) == 0
 78            and type(self).read_records is HttpStream.read_records
 79        ):
 80            self.cursor = ResumableFullRefreshCursor()
 81
 82    @property
 83    def exit_on_rate_limit(self) -> bool:
 84        """
 85        :return: False if the stream will retry endlessly when rate limited
 86        """
 87        return self._exit_on_rate_limit
 88
 89    @exit_on_rate_limit.setter
 90    def exit_on_rate_limit(self, value: bool) -> None:
 91        self._exit_on_rate_limit = value
 92
 93    @property
 94    def cache_filename(self) -> str:
 95        """
 96        Override if needed. Return the name of cache file
 97        Note that if the environment variable REQUEST_CACHE_PATH is not set, the cache will be in-memory only.
 98        """
 99        return f"{self.name}.sqlite"
100
101    @property
102    def use_cache(self) -> bool:
103        """
104        Override if needed. If True, all records will be cached.
105        Note that if the environment variable REQUEST_CACHE_PATH is not set, the cache will be in-memory only.
106        """
107        return False
108
109    @property
110    @abstractmethod
111    def url_base(self) -> str:
112        """
113        :return: URL base for the  API endpoint e.g: if you wanted to hit https://myapi.com/v1/some_entity then this should return "https://myapi.com/v1/"
114        """
115
116    @property
117    def http_method(self) -> str:
118        """
119        Override if needed. See get_request_data/get_request_json if using POST/PUT/PATCH.
120        """
121        return "GET"
122
123    @property
124    @deprecated(
125        "Deprecated as of CDK version 3.0.0. "
126        "You should set error_handler explicitly in HttpStream.get_error_handler() instead."
127    )
128    def raise_on_http_errors(self) -> bool:
129        """
130        Override if needed. If set to False, allows opting-out of raising HTTP code exception.
131        """
132        return True
133
134    @property
135    @deprecated(
136        "Deprecated as of CDK version 3.0.0. "
137        "You should set backoff_strategies explicitly in HttpStream.get_backoff_strategy() instead."
138    )
139    def max_retries(self) -> Union[int, None]:
140        """
141        Override if needed. Specifies maximum amount of retries for backoff policy. Return None for no limit.
142        """
143        return 5
144
145    @property
146    @deprecated(
147        "Deprecated as of CDK version 3.0.0. "
148        "You should set backoff_strategies explicitly in HttpStream.get_backoff_strategy() instead."
149    )
150    def max_time(self) -> Union[int, None]:
151        """
152        Override if needed. Specifies maximum total waiting time (in seconds) for backoff policy. Return None for no limit.
153        """
154        return 60 * 10
155
156    @property
157    @deprecated(
158        "Deprecated as of CDK version 3.0.0. "
159        "You should set backoff_strategies explicitly in HttpStream.get_backoff_strategy() instead."
160    )
161    def retry_factor(self) -> float:
162        """
163        Override if needed. Specifies factor for backoff policy.
164        """
165        return 5
166
167    @abstractmethod
168    def next_page_token(self, response: requests.Response) -> Optional[Mapping[str, Any]]:
169        """
170        Override this method to define a pagination strategy.
171
172        The value returned from this method is passed to most other methods in this class. Use it to form a request e.g: set headers or query params.
173
174        :return: The token for the next page from the input response object. Returning None means there are no more pages to read in this response.
175        """
176
177    @abstractmethod
178    def path(
179        self,
180        *,
181        stream_state: Optional[Mapping[str, Any]] = None,
182        stream_slice: Optional[Mapping[str, Any]] = None,
183        next_page_token: Optional[Mapping[str, Any]] = None,
184    ) -> str:
185        """
186        Returns the URL path for the API endpoint e.g: if you wanted to hit https://myapi.com/v1/some_entity then this should return "some_entity"
187        """
188
189    def request_params(
190        self,
191        stream_state: Optional[Mapping[str, Any]],
192        stream_slice: Optional[Mapping[str, Any]] = None,
193        next_page_token: Optional[Mapping[str, Any]] = None,
194    ) -> MutableMapping[str, Any]:
195        """
196        Override this method to define the query parameters that should be set on an outgoing HTTP request given the inputs.
197
198        E.g: you might want to define query parameters for paging if next_page_token is not None.
199        """
200        return {}
201
202    def request_headers(
203        self,
204        stream_state: Optional[Mapping[str, Any]],
205        stream_slice: Optional[Mapping[str, Any]] = None,
206        next_page_token: Optional[Mapping[str, Any]] = None,
207    ) -> Mapping[str, Any]:
208        """
209        Override to return any non-auth headers. Authentication headers will overwrite any overlapping headers returned from this method.
210        """
211        return {}
212
213    def request_body_data(
214        self,
215        stream_state: Optional[Mapping[str, Any]],
216        stream_slice: Optional[Mapping[str, Any]] = None,
217        next_page_token: Optional[Mapping[str, Any]] = None,
218    ) -> Optional[Union[Mapping[str, Any], str]]:
219        """
220        Override when creating POST/PUT/PATCH requests to populate the body of the request with a non-JSON payload.
221
222        If returns a ready text that it will be sent as is.
223        If returns a dict that it will be converted to a urlencoded form.
224        E.g. {"key1": "value1", "key2": "value2"} => "key1=value1&key2=value2"
225
226        At the same time only one of the 'request_body_data' and 'request_body_json' functions can be overridden.
227        """
228        return None
229
230    def request_body_json(
231        self,
232        stream_state: Optional[Mapping[str, Any]],
233        stream_slice: Optional[Mapping[str, Any]] = None,
234        next_page_token: Optional[Mapping[str, Any]] = None,
235    ) -> Optional[Mapping[str, Any]]:
236        """
237        Override when creating POST/PUT/PATCH requests to populate the body of the request with a JSON payload.
238
239        At the same time only one of the 'request_body_data' and 'request_body_json' functions can be overridden.
240        """
241        return None
242
243    def request_kwargs(
244        self,
245        stream_state: Optional[Mapping[str, Any]],
246        stream_slice: Optional[Mapping[str, Any]] = None,
247        next_page_token: Optional[Mapping[str, Any]] = None,
248    ) -> Mapping[str, Any]:
249        """
250        Override to return a mapping of keyword arguments to be used when creating the HTTP request.
251        Any option listed in https://docs.python-requests.org/en/latest/api/#requests.adapters.BaseAdapter.send for can be returned from
252        this method. Note that these options do not conflict with request-level options such as headers, request params, etc..
253        """
254        return {}
255
256    @abstractmethod
257    def parse_response(
258        self,
259        response: requests.Response,
260        *,
261        stream_state: Mapping[str, Any],
262        stream_slice: Optional[Mapping[str, Any]] = None,
263        next_page_token: Optional[Mapping[str, Any]] = None,
264    ) -> Iterable[Mapping[str, Any]]:
265        """
266        Parses the raw response object into a list of records.
267        By default, this returns an iterable containing the input. Override to parse differently.
268        :param response:
269        :param stream_state:
270        :param stream_slice:
271        :param next_page_token:
272        :return: An iterable containing the parsed response
273        """
274
275    def get_backoff_strategy(self) -> Optional[Union[BackoffStrategy, List[BackoffStrategy]]]:
276        """
277        Used to initialize Adapter to avoid breaking changes.
278        If Stream has a `backoff_time` method implementation, we know this stream uses old (pre-HTTPClient) backoff handlers and thus an adapter is needed.
279
280        Override to provide custom BackoffStrategy
281        :return Optional[BackoffStrategy]:
282        """
283        if hasattr(self, "backoff_time"):
284            return HttpStreamAdapterBackoffStrategy(self)
285        else:
286            return None
287
288    def get_error_handler(self) -> Optional[ErrorHandler]:
289        """
290        Used to initialize Adapter to avoid breaking changes.
291        If Stream has a `should_retry` method implementation, we know this stream uses old (pre-HTTPClient) error handlers and thus an adapter is needed.
292
293        Override to provide custom ErrorHandler
294        :return Optional[ErrorHandler]:
295        """
296        if hasattr(self, "should_retry"):
297            error_handler = HttpStreamAdapterHttpStatusErrorHandler(
298                stream=self,
299                logger=logging.getLogger(),
300                max_retries=self.max_retries,
301                max_time=timedelta(seconds=self.max_time or 0),
302            )
303            return error_handler
304        else:
305            return None
306
307    @classmethod
308    def _join_url(cls, url_base: str, path: str) -> str:
309        return urljoin(url_base, path)
310
311    @classmethod
312    def parse_response_error_message(cls, response: requests.Response) -> Optional[str]:
313        """
314        Parses the raw response object from a failed request into a user-friendly error message.
315        By default, this method tries to grab the error message from JSON responses by following common API patterns. Override to parse differently.
316
317        :param response:
318        :return: A user-friendly message that indicates the cause of the error
319        """
320
321        # default logic to grab error from common fields
322        def _try_get_error(value: Optional[JsonType]) -> Optional[str]:
323            if isinstance(value, str):
324                return value
325            elif isinstance(value, list):
326                errors_in_value = [_try_get_error(v) for v in value]
327                return ", ".join(v for v in errors_in_value if v is not None)
328            elif isinstance(value, dict):
329                new_value = (
330                    value.get("message")
331                    or value.get("messages")
332                    or value.get("error")
333                    or value.get("errors")
334                    or value.get("failures")
335                    or value.get("failure")
336                    or value.get("detail")
337                )
338                return _try_get_error(new_value)
339            return None
340
341        try:
342            body = response.json()
343            return _try_get_error(body)
344        except requests.exceptions.JSONDecodeError:
345            return None
346
347    def get_error_display_message(self, exception: BaseException) -> Optional[str]:
348        """
349        Retrieves the user-friendly display message that corresponds to an exception.
350        This will be called when encountering an exception while reading records from the stream, and used to build the AirbyteTraceMessage.
351
352        The default implementation of this method only handles HTTPErrors by passing the response to self.parse_response_error_message().
353        The method should be overriden as needed to handle any additional exception types.
354
355        :param exception: The exception that was raised
356        :return: A user-friendly message that indicates the cause of the error
357        """
358        if isinstance(exception, requests.HTTPError) and exception.response is not None:
359            return self.parse_response_error_message(exception.response)
360        return None
361
362    def read_records(
363        self,
364        sync_mode: SyncMode,
365        cursor_field: Optional[List[str]] = None,
366        stream_slice: Optional[Mapping[str, Any]] = None,
367        stream_state: Optional[Mapping[str, Any]] = None,
368    ) -> Iterable[StreamData]:
369        # A cursor_field indicates this is an incremental stream which offers better checkpointing than RFR enabled via the cursor
370        if self.cursor_field or not isinstance(self.get_cursor(), ResumableFullRefreshCursor):
371            yield from self._read_pages(
372                lambda req, res, state, _slice: self.parse_response(
373                    res, stream_slice=_slice, stream_state=state
374                ),
375                stream_slice,
376                stream_state,
377            )
378        else:
379            yield from self._read_single_page(
380                lambda req, res, state, _slice: self.parse_response(
381                    res, stream_slice=_slice, stream_state=state
382                ),
383                stream_slice,
384                stream_state,
385            )
386
387    @property
388    def state(self) -> MutableMapping[str, Any]:
389        cursor = self.get_cursor()
390        if cursor:
391            return cursor.get_stream_state()  # type: ignore
392        return self._state
393
394    @state.setter
395    def state(self, value: MutableMapping[str, Any]) -> None:
396        cursor = self.get_cursor()
397        if cursor:
398            cursor.set_initial_state(value)
399        self._state = value
400
401    def get_cursor(self) -> Optional[Cursor]:
402        # I don't love that this is semi-stateful but not sure what else to do. We don't know exactly what type of cursor to
403        # instantiate when creating the class. We can make a few assumptions like if there is a cursor_field which implies
404        # incremental, but we don't know until runtime if this is a substream. Ideally, a stream should explicitly define
405        # its cursor, but because we're trying to automatically apply RFR we're stuck with this logic where we replace the
406        # cursor at runtime once we detect this is a substream based on self.has_multiple_slices being reassigned
407        if self.has_multiple_slices and isinstance(self.cursor, ResumableFullRefreshCursor):
408            self.cursor = SubstreamResumableFullRefreshCursor()
409            return self.cursor
410        else:
411            return self.cursor
412
413    def _read_pages(
414        self,
415        records_generator_fn: Callable[
416            [
417                requests.PreparedRequest,
418                requests.Response,
419                Mapping[str, Any],
420                Optional[Mapping[str, Any]],
421            ],
422            Iterable[StreamData],
423        ],
424        stream_slice: Optional[Mapping[str, Any]] = None,
425        stream_state: Optional[Mapping[str, Any]] = None,
426    ) -> Iterable[StreamData]:
427        stream_state = stream_state or {}
428        pagination_complete = False
429        next_page_token = None
430        while not pagination_complete:
431            request, response = self._fetch_next_page(stream_slice, stream_state, next_page_token)
432            yield from records_generator_fn(request, response, stream_state, stream_slice)
433
434            next_page_token = self.next_page_token(response)
435            if not next_page_token:
436                pagination_complete = True
437
438        cursor = self.get_cursor()
439        if cursor and isinstance(cursor, SubstreamResumableFullRefreshCursor):
440            partition, _, _ = self._extract_slice_fields(stream_slice=stream_slice)
441            # Substreams checkpoint state by marking an entire parent partition as completed so that on the subsequent attempt
442            # after a failure, completed parents are skipped and the sync can make progress
443            cursor.close_slice(StreamSlice(cursor_slice={}, partition=partition))
444
445        # Always return an empty generator just in case no records were ever yielded
446        yield from []
447
448    def _read_single_page(
449        self,
450        records_generator_fn: Callable[
451            [
452                requests.PreparedRequest,
453                requests.Response,
454                Mapping[str, Any],
455                Optional[Mapping[str, Any]],
456            ],
457            Iterable[StreamData],
458        ],
459        stream_slice: Optional[Mapping[str, Any]] = None,
460        stream_state: Optional[Mapping[str, Any]] = None,
461    ) -> Iterable[StreamData]:
462        partition, cursor_slice, remaining_slice = self._extract_slice_fields(
463            stream_slice=stream_slice
464        )
465        stream_state = stream_state or {}
466        next_page_token = cursor_slice or None
467
468        request, response = self._fetch_next_page(remaining_slice, stream_state, next_page_token)
469        yield from records_generator_fn(request, response, stream_state, remaining_slice)
470
471        next_page_token = self.next_page_token(response) or {
472            "__ab_full_refresh_sync_complete": True
473        }
474
475        cursor = self.get_cursor()
476        if cursor:
477            cursor.close_slice(StreamSlice(cursor_slice=next_page_token, partition=partition))
478
479        # Always return an empty generator just in case no records were ever yielded
480        yield from []
481
482    @staticmethod
483    def _extract_slice_fields(
484        stream_slice: Optional[Mapping[str, Any]],
485    ) -> tuple[Mapping[str, Any], Mapping[str, Any], Mapping[str, Any]]:
486        if not stream_slice:
487            return {}, {}, {}
488
489        if isinstance(stream_slice, StreamSlice):
490            partition = stream_slice.partition
491            cursor_slice = stream_slice.cursor_slice
492            remaining = {k: v for k, v in stream_slice.items()}
493        else:
494            # RFR streams that implement stream_slices() to generate stream slices in the legacy mapping format are converted into a
495            # structured stream slice mapping by the LegacyCursorBasedCheckpointReader. The structured mapping object has separate
496            # fields for the partition and cursor_slice value
497            partition = stream_slice.get("partition", {})
498            cursor_slice = stream_slice.get("cursor_slice", {})
499            remaining = {
500                key: val
501                for key, val in stream_slice.items()
502                if key != "partition" and key != "cursor_slice"
503            }
504        return partition, cursor_slice, remaining
505
506    def _fetch_next_page(
507        self,
508        stream_slice: Optional[Mapping[str, Any]] = None,
509        stream_state: Optional[Mapping[str, Any]] = None,
510        next_page_token: Optional[Mapping[str, Any]] = None,
511    ) -> Tuple[requests.PreparedRequest, requests.Response]:
512        request, response = self._http_client.send_request(
513            http_method=self.http_method,
514            url=self._join_url(
515                self.url_base,
516                self.path(
517                    stream_state=stream_state,
518                    stream_slice=stream_slice,
519                    next_page_token=next_page_token,
520                ),
521            ),
522            request_kwargs=self.request_kwargs(
523                stream_state=stream_state,
524                stream_slice=stream_slice,
525                next_page_token=next_page_token,
526            ),
527            headers=self.request_headers(
528                stream_state=stream_state,
529                stream_slice=stream_slice,
530                next_page_token=next_page_token,
531            ),
532            params=self.request_params(
533                stream_state=stream_state,
534                stream_slice=stream_slice,
535                next_page_token=next_page_token,
536            ),
537            json=self.request_body_json(
538                stream_state=stream_state,
539                stream_slice=stream_slice,
540                next_page_token=next_page_token,
541            ),
542            data=self.request_body_data(
543                stream_state=stream_state,
544                stream_slice=stream_slice,
545                next_page_token=next_page_token,
546            ),
547            dedupe_query_params=True,
548            log_formatter=self.get_log_formatter(),
549            exit_on_rate_limit=self.exit_on_rate_limit,
550        )
551
552        return request, response
553
554    def get_log_formatter(self) -> Optional[Callable[[requests.Response], Any]]:
555        """
556
557        :return Optional[Callable[[requests.Response], Any]]: Function that will be used in logging inside HttpClient
558        """
559        return None

Base abstract class for an Airbyte Stream using the HTTP protocol. Basic building block for users building an Airbyte source for a HTTP API.

source_defined_cursor = True

Return False if the cursor can be configured by the user.

page_size: Optional[int] = None
exit_on_rate_limit: bool
82    @property
83    def exit_on_rate_limit(self) -> bool:
84        """
85        :return: False if the stream will retry endlessly when rate limited
86        """
87        return self._exit_on_rate_limit
Returns

False if the stream will retry endlessly when rate limited

cache_filename: str
93    @property
94    def cache_filename(self) -> str:
95        """
96        Override if needed. Return the name of cache file
97        Note that if the environment variable REQUEST_CACHE_PATH is not set, the cache will be in-memory only.
98        """
99        return f"{self.name}.sqlite"

Override if needed. Return the name of cache file Note that if the environment variable REQUEST_CACHE_PATH is not set, the cache will be in-memory only.

use_cache: bool
101    @property
102    def use_cache(self) -> bool:
103        """
104        Override if needed. If True, all records will be cached.
105        Note that if the environment variable REQUEST_CACHE_PATH is not set, the cache will be in-memory only.
106        """
107        return False

Override if needed. If True, all records will be cached. Note that if the environment variable REQUEST_CACHE_PATH is not set, the cache will be in-memory only.

url_base: str
109    @property
110    @abstractmethod
111    def url_base(self) -> str:
112        """
113        :return: URL base for the  API endpoint e.g: if you wanted to hit https://myapi.com/v1/some_entity then this should return "https://myapi.com/v1/"
114        """
Returns

URL base for the API endpoint e.g: if you wanted to hit https://myapi.com/v1/some_entity then this should return "https://myapi.com/v1/"

http_method: str
116    @property
117    def http_method(self) -> str:
118        """
119        Override if needed. See get_request_data/get_request_json if using POST/PUT/PATCH.
120        """
121        return "GET"

Override if needed. See get_request_data/get_request_json if using POST/PUT/PATCH.

raise_on_http_errors: bool
123    @property
124    @deprecated(
125        "Deprecated as of CDK version 3.0.0. "
126        "You should set error_handler explicitly in HttpStream.get_error_handler() instead."
127    )
128    def raise_on_http_errors(self) -> bool:
129        """
130        Override if needed. If set to False, allows opting-out of raising HTTP code exception.
131        """
132        return True

Override if needed. If set to False, allows opting-out of raising HTTP code exception.

max_retries: Optional[int]
134    @property
135    @deprecated(
136        "Deprecated as of CDK version 3.0.0. "
137        "You should set backoff_strategies explicitly in HttpStream.get_backoff_strategy() instead."
138    )
139    def max_retries(self) -> Union[int, None]:
140        """
141        Override if needed. Specifies maximum amount of retries for backoff policy. Return None for no limit.
142        """
143        return 5

Override if needed. Specifies maximum amount of retries for backoff policy. Return None for no limit.

max_time: Optional[int]
145    @property
146    @deprecated(
147        "Deprecated as of CDK version 3.0.0. "
148        "You should set backoff_strategies explicitly in HttpStream.get_backoff_strategy() instead."
149    )
150    def max_time(self) -> Union[int, None]:
151        """
152        Override if needed. Specifies maximum total waiting time (in seconds) for backoff policy. Return None for no limit.
153        """
154        return 60 * 10

Override if needed. Specifies maximum total waiting time (in seconds) for backoff policy. Return None for no limit.

retry_factor: float
156    @property
157    @deprecated(
158        "Deprecated as of CDK version 3.0.0. "
159        "You should set backoff_strategies explicitly in HttpStream.get_backoff_strategy() instead."
160    )
161    def retry_factor(self) -> float:
162        """
163        Override if needed. Specifies factor for backoff policy.
164        """
165        return 5

Override if needed. Specifies factor for backoff policy.

@abstractmethod
def next_page_token(self, response: requests.models.Response) -> Optional[Mapping[str, Any]]:
167    @abstractmethod
168    def next_page_token(self, response: requests.Response) -> Optional[Mapping[str, Any]]:
169        """
170        Override this method to define a pagination strategy.
171
172        The value returned from this method is passed to most other methods in this class. Use it to form a request e.g: set headers or query params.
173
174        :return: The token for the next page from the input response object. Returning None means there are no more pages to read in this response.
175        """

Override this method to define a pagination strategy.

The value returned from this method is passed to most other methods in this class. Use it to form a request e.g: set headers or query params.

Returns

The token for the next page from the input response object. Returning None means there are no more pages to read in this response.

@abstractmethod
def path( self, *, stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[Mapping[str, Any]] = None, next_page_token: Optional[Mapping[str, Any]] = None) -> str:
177    @abstractmethod
178    def path(
179        self,
180        *,
181        stream_state: Optional[Mapping[str, Any]] = None,
182        stream_slice: Optional[Mapping[str, Any]] = None,
183        next_page_token: Optional[Mapping[str, Any]] = None,
184    ) -> str:
185        """
186        Returns the URL path for the API endpoint e.g: if you wanted to hit https://myapi.com/v1/some_entity then this should return "some_entity"
187        """

Returns the URL path for the API endpoint e.g: if you wanted to hit https://myapi.com/v1/some_entity then this should return "some_entity"

def request_params( self, stream_state: Optional[Mapping[str, Any]], stream_slice: Optional[Mapping[str, Any]] = None, next_page_token: Optional[Mapping[str, Any]] = None) -> MutableMapping[str, Any]:
189    def request_params(
190        self,
191        stream_state: Optional[Mapping[str, Any]],
192        stream_slice: Optional[Mapping[str, Any]] = None,
193        next_page_token: Optional[Mapping[str, Any]] = None,
194    ) -> MutableMapping[str, Any]:
195        """
196        Override this method to define the query parameters that should be set on an outgoing HTTP request given the inputs.
197
198        E.g: you might want to define query parameters for paging if next_page_token is not None.
199        """
200        return {}

Override this method to define the query parameters that should be set on an outgoing HTTP request given the inputs.

E.g: you might want to define query parameters for paging if next_page_token is not None.

def request_headers( self, stream_state: Optional[Mapping[str, Any]], stream_slice: Optional[Mapping[str, Any]] = None, next_page_token: Optional[Mapping[str, Any]] = None) -> Mapping[str, Any]:
202    def request_headers(
203        self,
204        stream_state: Optional[Mapping[str, Any]],
205        stream_slice: Optional[Mapping[str, Any]] = None,
206        next_page_token: Optional[Mapping[str, Any]] = None,
207    ) -> Mapping[str, Any]:
208        """
209        Override to return any non-auth headers. Authentication headers will overwrite any overlapping headers returned from this method.
210        """
211        return {}

Override to return any non-auth headers. Authentication headers will overwrite any overlapping headers returned from this method.

def request_body_data( self, stream_state: Optional[Mapping[str, Any]], stream_slice: Optional[Mapping[str, Any]] = None, next_page_token: Optional[Mapping[str, Any]] = None) -> Union[str, Mapping[str, Any], NoneType]:
213    def request_body_data(
214        self,
215        stream_state: Optional[Mapping[str, Any]],
216        stream_slice: Optional[Mapping[str, Any]] = None,
217        next_page_token: Optional[Mapping[str, Any]] = None,
218    ) -> Optional[Union[Mapping[str, Any], str]]:
219        """
220        Override when creating POST/PUT/PATCH requests to populate the body of the request with a non-JSON payload.
221
222        If returns a ready text that it will be sent as is.
223        If returns a dict that it will be converted to a urlencoded form.
224        E.g. {"key1": "value1", "key2": "value2"} => "key1=value1&key2=value2"
225
226        At the same time only one of the 'request_body_data' and 'request_body_json' functions can be overridden.
227        """
228        return None

Override when creating POST/PUT/PATCH requests to populate the body of the request with a non-JSON payload.

If returns a ready text that it will be sent as is. If returns a dict that it will be converted to a urlencoded form. E.g. {"key1": "value1", "key2": "value2"} => "key1=value1&key2=value2"

At the same time only one of the 'request_body_data' and 'request_body_json' functions can be overridden.

def request_body_json( self, stream_state: Optional[Mapping[str, Any]], stream_slice: Optional[Mapping[str, Any]] = None, next_page_token: Optional[Mapping[str, Any]] = None) -> Optional[Mapping[str, Any]]:
230    def request_body_json(
231        self,
232        stream_state: Optional[Mapping[str, Any]],
233        stream_slice: Optional[Mapping[str, Any]] = None,
234        next_page_token: Optional[Mapping[str, Any]] = None,
235    ) -> Optional[Mapping[str, Any]]:
236        """
237        Override when creating POST/PUT/PATCH requests to populate the body of the request with a JSON payload.
238
239        At the same time only one of the 'request_body_data' and 'request_body_json' functions can be overridden.
240        """
241        return None

Override when creating POST/PUT/PATCH requests to populate the body of the request with a JSON payload.

At the same time only one of the 'request_body_data' and 'request_body_json' functions can be overridden.

def request_kwargs( self, stream_state: Optional[Mapping[str, Any]], stream_slice: Optional[Mapping[str, Any]] = None, next_page_token: Optional[Mapping[str, Any]] = None) -> Mapping[str, Any]:
243    def request_kwargs(
244        self,
245        stream_state: Optional[Mapping[str, Any]],
246        stream_slice: Optional[Mapping[str, Any]] = None,
247        next_page_token: Optional[Mapping[str, Any]] = None,
248    ) -> Mapping[str, Any]:
249        """
250        Override to return a mapping of keyword arguments to be used when creating the HTTP request.
251        Any option listed in https://docs.python-requests.org/en/latest/api/#requests.adapters.BaseAdapter.send for can be returned from
252        this method. Note that these options do not conflict with request-level options such as headers, request params, etc..
253        """
254        return {}

Override to return a mapping of keyword arguments to be used when creating the HTTP request. Any option listed in https://docs.python-requests.org/en/latest/api/#requests.adapters.BaseAdapter.send for can be returned from this method. Note that these options do not conflict with request-level options such as headers, request params, etc..

@abstractmethod
def parse_response( self, response: requests.models.Response, *, stream_state: Mapping[str, Any], stream_slice: Optional[Mapping[str, Any]] = None, next_page_token: Optional[Mapping[str, Any]] = None) -> Iterable[Mapping[str, Any]]:
256    @abstractmethod
257    def parse_response(
258        self,
259        response: requests.Response,
260        *,
261        stream_state: Mapping[str, Any],
262        stream_slice: Optional[Mapping[str, Any]] = None,
263        next_page_token: Optional[Mapping[str, Any]] = None,
264    ) -> Iterable[Mapping[str, Any]]:
265        """
266        Parses the raw response object into a list of records.
267        By default, this returns an iterable containing the input. Override to parse differently.
268        :param response:
269        :param stream_state:
270        :param stream_slice:
271        :param next_page_token:
272        :return: An iterable containing the parsed response
273        """

Parses the raw response object into a list of records. By default, this returns an iterable containing the input. Override to parse differently.

Parameters
  • response:
  • stream_state:
  • stream_slice:
  • next_page_token:
Returns

An iterable containing the parsed response

def get_backoff_strategy( self) -> Union[BackoffStrategy, List[BackoffStrategy], NoneType]:
275    def get_backoff_strategy(self) -> Optional[Union[BackoffStrategy, List[BackoffStrategy]]]:
276        """
277        Used to initialize Adapter to avoid breaking changes.
278        If Stream has a `backoff_time` method implementation, we know this stream uses old (pre-HTTPClient) backoff handlers and thus an adapter is needed.
279
280        Override to provide custom BackoffStrategy
281        :return Optional[BackoffStrategy]:
282        """
283        if hasattr(self, "backoff_time"):
284            return HttpStreamAdapterBackoffStrategy(self)
285        else:
286            return None

Used to initialize Adapter to avoid breaking changes. If Stream has a backoff_time method implementation, we know this stream uses old (pre-HTTPClient) backoff handlers and thus an adapter is needed.

Override to provide custom BackoffStrategy

Returns
def get_error_handler( self) -> Optional[airbyte_cdk.sources.streams.http.error_handlers.ErrorHandler]:
288    def get_error_handler(self) -> Optional[ErrorHandler]:
289        """
290        Used to initialize Adapter to avoid breaking changes.
291        If Stream has a `should_retry` method implementation, we know this stream uses old (pre-HTTPClient) error handlers and thus an adapter is needed.
292
293        Override to provide custom ErrorHandler
294        :return Optional[ErrorHandler]:
295        """
296        if hasattr(self, "should_retry"):
297            error_handler = HttpStreamAdapterHttpStatusErrorHandler(
298                stream=self,
299                logger=logging.getLogger(),
300                max_retries=self.max_retries,
301                max_time=timedelta(seconds=self.max_time or 0),
302            )
303            return error_handler
304        else:
305            return None

Used to initialize Adapter to avoid breaking changes. If Stream has a should_retry method implementation, we know this stream uses old (pre-HTTPClient) error handlers and thus an adapter is needed.

Override to provide custom ErrorHandler

Returns
@classmethod
def parse_response_error_message(cls, response: requests.models.Response) -> Optional[str]:
311    @classmethod
312    def parse_response_error_message(cls, response: requests.Response) -> Optional[str]:
313        """
314        Parses the raw response object from a failed request into a user-friendly error message.
315        By default, this method tries to grab the error message from JSON responses by following common API patterns. Override to parse differently.
316
317        :param response:
318        :return: A user-friendly message that indicates the cause of the error
319        """
320
321        # default logic to grab error from common fields
322        def _try_get_error(value: Optional[JsonType]) -> Optional[str]:
323            if isinstance(value, str):
324                return value
325            elif isinstance(value, list):
326                errors_in_value = [_try_get_error(v) for v in value]
327                return ", ".join(v for v in errors_in_value if v is not None)
328            elif isinstance(value, dict):
329                new_value = (
330                    value.get("message")
331                    or value.get("messages")
332                    or value.get("error")
333                    or value.get("errors")
334                    or value.get("failures")
335                    or value.get("failure")
336                    or value.get("detail")
337                )
338                return _try_get_error(new_value)
339            return None
340
341        try:
342            body = response.json()
343            return _try_get_error(body)
344        except requests.exceptions.JSONDecodeError:
345            return None

Parses the raw response object from a failed request into a user-friendly error message. By default, this method tries to grab the error message from JSON responses by following common API patterns. Override to parse differently.

Parameters
  • response:
Returns

A user-friendly message that indicates the cause of the error

def get_error_display_message(self, exception: BaseException) -> Optional[str]:
347    def get_error_display_message(self, exception: BaseException) -> Optional[str]:
348        """
349        Retrieves the user-friendly display message that corresponds to an exception.
350        This will be called when encountering an exception while reading records from the stream, and used to build the AirbyteTraceMessage.
351
352        The default implementation of this method only handles HTTPErrors by passing the response to self.parse_response_error_message().
353        The method should be overriden as needed to handle any additional exception types.
354
355        :param exception: The exception that was raised
356        :return: A user-friendly message that indicates the cause of the error
357        """
358        if isinstance(exception, requests.HTTPError) and exception.response is not None:
359            return self.parse_response_error_message(exception.response)
360        return None

Retrieves the user-friendly display message that corresponds to an exception. This will be called when encountering an exception while reading records from the stream, and used to build the AirbyteTraceMessage.

The default implementation of this method only handles HTTPErrors by passing the response to self.parse_response_error_message(). The method should be overriden as needed to handle any additional exception types.

Parameters
  • exception: The exception that was raised
Returns

A user-friendly message that indicates the cause of the error

def read_records( self, sync_mode: airbyte_protocol_dataclasses.models.airbyte_protocol.SyncMode, cursor_field: Optional[List[str]] = None, stream_slice: Optional[Mapping[str, Any]] = None, stream_state: Optional[Mapping[str, Any]] = None) -> Iterable[Union[Mapping[str, Any], AirbyteMessage]]:
362    def read_records(
363        self,
364        sync_mode: SyncMode,
365        cursor_field: Optional[List[str]] = None,
366        stream_slice: Optional[Mapping[str, Any]] = None,
367        stream_state: Optional[Mapping[str, Any]] = None,
368    ) -> Iterable[StreamData]:
369        # A cursor_field indicates this is an incremental stream which offers better checkpointing than RFR enabled via the cursor
370        if self.cursor_field or not isinstance(self.get_cursor(), ResumableFullRefreshCursor):
371            yield from self._read_pages(
372                lambda req, res, state, _slice: self.parse_response(
373                    res, stream_slice=_slice, stream_state=state
374                ),
375                stream_slice,
376                stream_state,
377            )
378        else:
379            yield from self._read_single_page(
380                lambda req, res, state, _slice: self.parse_response(
381                    res, stream_slice=_slice, stream_state=state
382                ),
383                stream_slice,
384                stream_state,
385            )

This method should be overridden by subclasses to read records based on the inputs

state: MutableMapping[str, Any]
387    @property
388    def state(self) -> MutableMapping[str, Any]:
389        cursor = self.get_cursor()
390        if cursor:
391            return cursor.get_stream_state()  # type: ignore
392        return self._state

State getter, should return state in form that can serialized to a string and send to the output as a STATE AirbyteMessage.

A good example of a state is a cursor_value: { self.cursor_field: "cursor_value" }

State should try to be as small as possible but at the same time descriptive enough to restore syncing process from the point where it stopped.

def get_cursor(self) -> Optional[airbyte_cdk.sources.streams.checkpoint.Cursor]:
401    def get_cursor(self) -> Optional[Cursor]:
402        # I don't love that this is semi-stateful but not sure what else to do. We don't know exactly what type of cursor to
403        # instantiate when creating the class. We can make a few assumptions like if there is a cursor_field which implies
404        # incremental, but we don't know until runtime if this is a substream. Ideally, a stream should explicitly define
405        # its cursor, but because we're trying to automatically apply RFR we're stuck with this logic where we replace the
406        # cursor at runtime once we detect this is a substream based on self.has_multiple_slices being reassigned
407        if self.has_multiple_slices and isinstance(self.cursor, ResumableFullRefreshCursor):
408            self.cursor = SubstreamResumableFullRefreshCursor()
409            return self.cursor
410        else:
411            return self.cursor

A Cursor is an interface that a stream can implement to manage how its internal state is read and updated while reading records. Historically, Python connectors had no concept of a cursor to manage state. Python streams need to define a cursor implementation and override this method to manage state through a Cursor.

def get_log_formatter(self) -> Optional[Callable[[requests.models.Response], Any]]:
554    def get_log_formatter(self) -> Optional[Callable[[requests.Response], Any]]:
555        """
556
557        :return Optional[Callable[[requests.Response], Any]]: Function that will be used in logging inside HttpClient
558        """
559        return None
Returns

Function that will be used in logging inside HttpClient

class HttpSubStream(airbyte_cdk.HttpStream, abc.ABC):
562class HttpSubStream(HttpStream, ABC):
563    def __init__(self, parent: HttpStream, **kwargs: Any):
564        """
565        :param parent: should be the instance of HttpStream class
566        """
567        super().__init__(**kwargs)
568        self.parent = parent
569        self.has_multiple_slices = (
570            True  # Substreams are based on parent records which implies there are multiple slices
571        )
572
573        # There are three conditions that dictate if RFR should automatically be applied to a stream
574        # 1. Streams that explicitly initialize their own cursor should defer to it and not automatically apply RFR
575        # 2. Streams with at least one cursor_field are incremental and thus a superior sync to RFR.
576        # 3. Streams overriding read_records() do not guarantee that they will call the parent implementation which can perform
577        #    per-page checkpointing so RFR is only supported if a stream use the default `HttpStream.read_records()` method
578        if (
579            not self.cursor
580            and len(self.cursor_field) == 0
581            and type(self).read_records is HttpStream.read_records
582        ):
583            self.cursor = SubstreamResumableFullRefreshCursor()
584
585    def stream_slices(
586        self,
587        sync_mode: SyncMode,
588        cursor_field: Optional[List[str]] = None,
589        stream_state: Optional[Mapping[str, Any]] = None,
590    ) -> Iterable[Optional[Mapping[str, Any]]]:
591        # read_stateless() assumes the parent is not concurrent. This is currently okay since the concurrent CDK does
592        # not support either substreams or RFR, but something that needs to be considered once we do
593        for parent_record in self.parent.read_only_records(stream_state):
594            # Skip non-records (eg AirbyteLogMessage)
595            if isinstance(parent_record, AirbyteMessage):
596                if parent_record.type == MessageType.RECORD:
597                    parent_record = parent_record.record.data  # type: ignore [assignment, union-attr]  # Incorrect type for assignment
598                else:
599                    continue
600            elif isinstance(parent_record, Record):
601                parent_record = parent_record.data
602            yield {"parent": parent_record}

Base abstract class for an Airbyte Stream using the HTTP protocol. Basic building block for users building an Airbyte source for a HTTP API.

HttpSubStream( parent: HttpStream, **kwargs: Any)
563    def __init__(self, parent: HttpStream, **kwargs: Any):
564        """
565        :param parent: should be the instance of HttpStream class
566        """
567        super().__init__(**kwargs)
568        self.parent = parent
569        self.has_multiple_slices = (
570            True  # Substreams are based on parent records which implies there are multiple slices
571        )
572
573        # There are three conditions that dictate if RFR should automatically be applied to a stream
574        # 1. Streams that explicitly initialize their own cursor should defer to it and not automatically apply RFR
575        # 2. Streams with at least one cursor_field are incremental and thus a superior sync to RFR.
576        # 3. Streams overriding read_records() do not guarantee that they will call the parent implementation which can perform
577        #    per-page checkpointing so RFR is only supported if a stream use the default `HttpStream.read_records()` method
578        if (
579            not self.cursor
580            and len(self.cursor_field) == 0
581            and type(self).read_records is HttpStream.read_records
582        ):
583            self.cursor = SubstreamResumableFullRefreshCursor()
Parameters
  • parent: should be the instance of HttpStream class
parent
has_multiple_slices = False
def stream_slices( self, sync_mode: airbyte_protocol_dataclasses.models.airbyte_protocol.SyncMode, cursor_field: Optional[List[str]] = None, stream_state: Optional[Mapping[str, Any]] = None) -> Iterable[Optional[Mapping[str, Any]]]:
585    def stream_slices(
586        self,
587        sync_mode: SyncMode,
588        cursor_field: Optional[List[str]] = None,
589        stream_state: Optional[Mapping[str, Any]] = None,
590    ) -> Iterable[Optional[Mapping[str, Any]]]:
591        # read_stateless() assumes the parent is not concurrent. This is currently okay since the concurrent CDK does
592        # not support either substreams or RFR, but something that needs to be considered once we do
593        for parent_record in self.parent.read_only_records(stream_state):
594            # Skip non-records (eg AirbyteLogMessage)
595            if isinstance(parent_record, AirbyteMessage):
596                if parent_record.type == MessageType.RECORD:
597                    parent_record = parent_record.record.data  # type: ignore [assignment, union-attr]  # Incorrect type for assignment
598                else:
599                    continue
600            elif isinstance(parent_record, Record):
601                parent_record = parent_record.data
602            yield {"parent": parent_record}

Override to define the slices for this stream. See the stream slicing section of the docs for more information.

Parameters
  • sync_mode:
  • cursor_field:
  • stream_state:
Returns
class LimiterSession(airbyte_cdk.sources.streams.call_rate.LimiterMixin, requests.sessions.Session):
750class LimiterSession(LimiterMixin, requests.Session):
751    """Session that adds rate-limiting behavior to requests."""

Session that adds rate-limiting behavior to requests.

class MovingWindowCallRatePolicy(airbyte_cdk.sources.streams.call_rate.BaseCallRatePolicy):
425class MovingWindowCallRatePolicy(BaseCallRatePolicy):
426    """
427    Policy to control requests rate implemented on top of PyRateLimiter lib.
428    The main difference between this policy and FixedWindowCallRatePolicy is that the rate-limiting window
429    is moving along requests that we made, and there is no moment when we reset an available number of calls.
430    This strategy requires saving of timestamps of all requests within a window.
431    """
432
433    def __init__(self, rates: list[Rate], matchers: list[RequestMatcher]):
434        """Constructor
435
436        :param rates: list of rates, the order is important and must be ascending
437        :param matchers:
438        """
439        if not rates:
440            raise ValueError("The list of rates can not be empty")
441        pyrate_rates = [
442            PyRateRate(limit=rate.limit, interval=int(rate.interval.total_seconds() * 1000))
443            for rate in rates
444        ]
445        self._bucket = InMemoryBucket(pyrate_rates)
446        # Limiter will create the background task that clears old requests in the bucket
447        self._limiter = Limiter(self._bucket)
448        super().__init__(matchers=matchers)
449
450    def try_acquire(self, request: Any, weight: int) -> None:
451        if not self.matches(request):
452            raise ValueError("Request does not match the policy")
453        lowest_limit = min(rate.limit for rate in self._bucket.rates)
454        if weight > lowest_limit:
455            raise ValueError(
456                f"Weight can not exceed the lowest configured rate limit ({lowest_limit})"
457            )
458
459        try:
460            self._limiter.try_acquire(request, weight=weight)
461        except BucketFullException as exc:
462            item = self._limiter.bucket_factory.wrap_item(request, weight)
463            assert isinstance(item, RateItem)
464
465            with self._limiter.lock:
466                time_to_wait = self._bucket.waiting(item)
467                assert isinstance(time_to_wait, int)
468
469                raise CallRateLimitHit(
470                    error=str(exc.meta_info["error"]),
471                    item=request,
472                    weight=int(exc.meta_info["weight"]),
473                    rate=str(exc.meta_info["rate"]),
474                    time_to_wait=timedelta(milliseconds=time_to_wait),
475                )
476
477    def update(
478        self, available_calls: Optional[int], call_reset_ts: Optional[datetime.datetime]
479    ) -> None:
480        """Adjust call bucket to reflect the state of the API server
481
482        :param available_calls:
483        :param call_reset_ts:
484        :return:
485        """
486        if (
487            available_calls is not None and call_reset_ts is None
488        ):  # we do our best to sync buckets with API
489            if available_calls == 0:
490                with self._limiter.lock:
491                    now: int = TimeClock().now()  # type: ignore[no-untyped-call]
492                    items_to_add = self._get_available_calls(now)
493                    if items_to_add > 0:
494                        added = self._bucket.put(
495                            RateItem(name="dummy", timestamp=now, weight=items_to_add)
496                        )
497                        if not added:
498                            logger.warning(
499                                "Failed to sync rate limit bucket with API: could not add %s items",
500                                items_to_add,
501                            )
502        # TODO: add support if needed, it might be that it is not possible to make a good solution for this case
503        # if available_calls is not None and call_reset_ts is not None:
504        #     ts = call_reset_ts.timestamp()
505
506    def _get_available_calls(self, now: int) -> int:
507        """Return the number of calls still allowed by the strictest configured rate at the given timestamp (ms)."""
508        return min(
509            rate.limit
510            - sum(1 for item in self._bucket.items if item.timestamp >= now - rate.interval)
511            for rate in self._bucket.rates
512        )
513
514    def __str__(self) -> str:
515        """Return a human-friendly description of the moving window rate policy for logging purposes."""
516        rates_info = ", ".join(
517            f"{rate.limit} per {timedelta(milliseconds=rate.interval)}"
518            for rate in self._bucket.rates
519        )
520        current_bucket_count = self._bucket.count()
521        matcher_str = ", ".join(f"{matcher}" for matcher in self._matchers)
522        return (
523            f"MovingWindowCallRatePolicy(rates=[{rates_info}], current_bucket_count={current_bucket_count}, "
524            f"matchers=[{matcher_str}])"
525        )

Policy to control requests rate implemented on top of PyRateLimiter lib. The main difference between this policy and FixedWindowCallRatePolicy is that the rate-limiting window is moving along requests that we made, and there is no moment when we reset an available number of calls. This strategy requires saving of timestamps of all requests within a window.

MovingWindowCallRatePolicy( rates: list[Rate], matchers: list[airbyte_cdk.sources.streams.call_rate.RequestMatcher])
433    def __init__(self, rates: list[Rate], matchers: list[RequestMatcher]):
434        """Constructor
435
436        :param rates: list of rates, the order is important and must be ascending
437        :param matchers:
438        """
439        if not rates:
440            raise ValueError("The list of rates can not be empty")
441        pyrate_rates = [
442            PyRateRate(limit=rate.limit, interval=int(rate.interval.total_seconds() * 1000))
443            for rate in rates
444        ]
445        self._bucket = InMemoryBucket(pyrate_rates)
446        # Limiter will create the background task that clears old requests in the bucket
447        self._limiter = Limiter(self._bucket)
448        super().__init__(matchers=matchers)

Constructor

Parameters
  • rates: list of rates, the order is important and must be ascending
  • matchers:
def try_acquire(self, request: Any, weight: int) -> None:
450    def try_acquire(self, request: Any, weight: int) -> None:
451        if not self.matches(request):
452            raise ValueError("Request does not match the policy")
453        lowest_limit = min(rate.limit for rate in self._bucket.rates)
454        if weight > lowest_limit:
455            raise ValueError(
456                f"Weight can not exceed the lowest configured rate limit ({lowest_limit})"
457            )
458
459        try:
460            self._limiter.try_acquire(request, weight=weight)
461        except BucketFullException as exc:
462            item = self._limiter.bucket_factory.wrap_item(request, weight)
463            assert isinstance(item, RateItem)
464
465            with self._limiter.lock:
466                time_to_wait = self._bucket.waiting(item)
467                assert isinstance(time_to_wait, int)
468
469                raise CallRateLimitHit(
470                    error=str(exc.meta_info["error"]),
471                    item=request,
472                    weight=int(exc.meta_info["weight"]),
473                    rate=str(exc.meta_info["rate"]),
474                    time_to_wait=timedelta(milliseconds=time_to_wait),
475                )

Try to acquire request

Parameters
  • request: a request object representing a single call to API
  • weight: number of requests to deduct from credit
Returns
def update( self, available_calls: Optional[int], call_reset_ts: Optional[datetime.datetime]) -> None:
477    def update(
478        self, available_calls: Optional[int], call_reset_ts: Optional[datetime.datetime]
479    ) -> None:
480        """Adjust call bucket to reflect the state of the API server
481
482        :param available_calls:
483        :param call_reset_ts:
484        :return:
485        """
486        if (
487            available_calls is not None and call_reset_ts is None
488        ):  # we do our best to sync buckets with API
489            if available_calls == 0:
490                with self._limiter.lock:
491                    now: int = TimeClock().now()  # type: ignore[no-untyped-call]
492                    items_to_add = self._get_available_calls(now)
493                    if items_to_add > 0:
494                        added = self._bucket.put(
495                            RateItem(name="dummy", timestamp=now, weight=items_to_add)
496                        )
497                        if not added:
498                            logger.warning(
499                                "Failed to sync rate limit bucket with API: could not add %s items",
500                                items_to_add,
501                            )
502        # TODO: add support if needed, it might be that it is not possible to make a good solution for this case
503        # if available_calls is not None and call_reset_ts is not None:
504        #     ts = call_reset_ts.timestamp()

Adjust call bucket to reflect the state of the API server

Parameters
  • available_calls:
  • call_reset_ts:
Returns
MultipleTokenAuthenticator
 27class Oauth2Authenticator(AbstractOauth2Authenticator):
 28    """
 29    Generates OAuth2.0 access tokens from an OAuth2.0 refresh token and client credentials.
 30    The generated access token is attached to each request via the Authorization header.
 31    If a connector_config is provided any mutation of it's value in the scope of this class will emit AirbyteControlConnectorConfigMessage.
 32    """
 33
 34    def __init__(
 35        self,
 36        token_refresh_endpoint: str,
 37        client_id: str,
 38        client_secret: str,
 39        refresh_token: str,
 40        client_id_name: str = "client_id",
 41        client_secret_name: str = "client_secret",
 42        refresh_token_name: str = "refresh_token",
 43        scopes: List[str] | None = None,
 44        token_expiry_date: AirbyteDateTime | None = None,
 45        token_expiry_date_format: str | None = None,
 46        access_token_name: str = "access_token",
 47        expires_in_name: str = "expires_in",
 48        refresh_request_body: Mapping[str, Any] | None = None,
 49        refresh_request_headers: Mapping[str, Any] | None = None,
 50        send_refresh_request_as_query_params: bool = False,
 51        grant_type_name: str = "grant_type",
 52        grant_type: str = "refresh_token",
 53        token_expiry_is_time_of_expiration: bool = False,
 54        refresh_token_error_status_codes: Tuple[int, ...] = (),
 55        refresh_token_error_key: str = "",
 56        refresh_token_error_values: Tuple[str, ...] = (),
 57    ) -> None:
 58        self._token_refresh_endpoint = token_refresh_endpoint
 59        self._client_secret_name = client_secret_name
 60        self._client_secret = client_secret
 61        self._client_id_name = client_id_name
 62        self._client_id = client_id
 63        self._refresh_token_name = refresh_token_name
 64        self._refresh_token = refresh_token
 65        self._scopes = scopes
 66        self._access_token_name = access_token_name
 67        self._expires_in_name = expires_in_name
 68        self._refresh_request_body = refresh_request_body
 69        self._refresh_request_headers = refresh_request_headers
 70        self._send_refresh_request_as_query_params = send_refresh_request_as_query_params
 71        self._grant_type_name = grant_type_name
 72        self._grant_type = grant_type
 73
 74        self._token_expiry_date = token_expiry_date or (ab_datetime_now() - timedelta(days=1))
 75        self._token_expiry_date_format = token_expiry_date_format
 76        self._token_expiry_is_time_of_expiration = token_expiry_is_time_of_expiration
 77        self._access_token = None
 78        super().__init__(
 79            refresh_token_error_status_codes, refresh_token_error_key, refresh_token_error_values
 80        )
 81
 82    def get_token_refresh_endpoint(self) -> str:
 83        return self._token_refresh_endpoint
 84
 85    def get_client_id_name(self) -> str:
 86        return self._client_id_name
 87
 88    def get_client_id(self) -> str:
 89        return self._client_id
 90
 91    def get_client_secret_name(self) -> str:
 92        return self._client_secret_name
 93
 94    def get_client_secret(self) -> str:
 95        return self._client_secret
 96
 97    def get_refresh_token_name(self) -> str:
 98        return self._refresh_token_name
 99
100    def get_refresh_token(self) -> str:
101        return self._refresh_token
102
103    def get_access_token_name(self) -> str:
104        return self._access_token_name
105
106    def get_scopes(self) -> list[str]:
107        return self._scopes  # type: ignore[return-value]
108
109    def get_expires_in_name(self) -> str:
110        return self._expires_in_name
111
112    def get_refresh_request_body(self) -> Mapping[str, Any]:
113        return self._refresh_request_body  # type: ignore[return-value]
114
115    def get_refresh_request_headers(self) -> Mapping[str, Any]:
116        return self._refresh_request_headers  # type: ignore[return-value]
117
118    def should_send_refresh_request_as_query_params(self) -> bool:
119        return self._send_refresh_request_as_query_params
120
121    def get_grant_type_name(self) -> str:
122        return self._grant_type_name
123
124    def get_grant_type(self) -> str:
125        return self._grant_type
126
127    def get_token_expiry_date(self) -> AirbyteDateTime:
128        return self._token_expiry_date
129
130    def set_token_expiry_date(self, value: AirbyteDateTime) -> None:
131        # Overrides of `refresh_access_token` written before 6.45.5 return the raw `expires_in`.
132        if not isinstance(value, datetime):
133            with suppress(ValueError, OverflowError):
134                value = self._parse_token_expiration_date(value)
135        self._token_expiry_date = value
136
137    @property
138    def token_expiry_is_time_of_expiration(self) -> bool:
139        return self._token_expiry_is_time_of_expiration
140
141    @property
142    def token_expiry_date_format(self) -> Optional[str]:
143        return self._token_expiry_date_format
144
145    @property
146    def access_token(self) -> str:
147        return self._access_token  # type: ignore[return-value]
148
149    @access_token.setter
150    def access_token(self, value: str) -> None:
151        self._access_token = value  # type: ignore[assignment]  # Incorrect type for assignment

Generates OAuth2.0 access tokens from an OAuth2.0 refresh token and client credentials. The generated access token is attached to each request via the Authorization header. If a connector_config is provided any mutation of it's value in the scope of this class will emit AirbyteControlConnectorConfigMessage.

Oauth2Authenticator( token_refresh_endpoint: str, client_id: str, client_secret: str, refresh_token: str, client_id_name: str = 'client_id', client_secret_name: str = 'client_secret', refresh_token_name: str = 'refresh_token', scopes: Optional[List[str]] = None, token_expiry_date: airbyte_cdk.utils.datetime_helpers.AirbyteDateTime | None = None, token_expiry_date_format: str | None = None, access_token_name: str = 'access_token', expires_in_name: str = 'expires_in', refresh_request_body: Optional[Mapping[str, Any]] = None, refresh_request_headers: Optional[Mapping[str, Any]] = None, send_refresh_request_as_query_params: bool = False, grant_type_name: str = 'grant_type', grant_type: str = 'refresh_token', token_expiry_is_time_of_expiration: bool = False, refresh_token_error_status_codes: Tuple[int, ...] = (), refresh_token_error_key: str = '', refresh_token_error_values: Tuple[str, ...] = ())
34    def __init__(
35        self,
36        token_refresh_endpoint: str,
37        client_id: str,
38        client_secret: str,
39        refresh_token: str,
40        client_id_name: str = "client_id",
41        client_secret_name: str = "client_secret",
42        refresh_token_name: str = "refresh_token",
43        scopes: List[str] | None = None,
44        token_expiry_date: AirbyteDateTime | None = None,
45        token_expiry_date_format: str | None = None,
46        access_token_name: str = "access_token",
47        expires_in_name: str = "expires_in",
48        refresh_request_body: Mapping[str, Any] | None = None,
49        refresh_request_headers: Mapping[str, Any] | None = None,
50        send_refresh_request_as_query_params: bool = False,
51        grant_type_name: str = "grant_type",
52        grant_type: str = "refresh_token",
53        token_expiry_is_time_of_expiration: bool = False,
54        refresh_token_error_status_codes: Tuple[int, ...] = (),
55        refresh_token_error_key: str = "",
56        refresh_token_error_values: Tuple[str, ...] = (),
57    ) -> None:
58        self._token_refresh_endpoint = token_refresh_endpoint
59        self._client_secret_name = client_secret_name
60        self._client_secret = client_secret
61        self._client_id_name = client_id_name
62        self._client_id = client_id
63        self._refresh_token_name = refresh_token_name
64        self._refresh_token = refresh_token
65        self._scopes = scopes
66        self._access_token_name = access_token_name
67        self._expires_in_name = expires_in_name
68        self._refresh_request_body = refresh_request_body
69        self._refresh_request_headers = refresh_request_headers
70        self._send_refresh_request_as_query_params = send_refresh_request_as_query_params
71        self._grant_type_name = grant_type_name
72        self._grant_type = grant_type
73
74        self._token_expiry_date = token_expiry_date or (ab_datetime_now() - timedelta(days=1))
75        self._token_expiry_date_format = token_expiry_date_format
76        self._token_expiry_is_time_of_expiration = token_expiry_is_time_of_expiration
77        self._access_token = None
78        super().__init__(
79            refresh_token_error_status_codes, refresh_token_error_key, refresh_token_error_values
80        )

If all of refresh_token_error_status_codes, refresh_token_error_key, and refresh_token_error_values are set, then http errors with such params will be wrapped in AirbyteTracedException.

def get_token_refresh_endpoint(self) -> str:
82    def get_token_refresh_endpoint(self) -> str:
83        return self._token_refresh_endpoint

Returns the endpoint to refresh the access token

def get_client_id_name(self) -> str:
85    def get_client_id_name(self) -> str:
86        return self._client_id_name

The client id name to authenticate

def get_client_id(self) -> str:
88    def get_client_id(self) -> str:
89        return self._client_id

The client id to authenticate

def get_client_secret_name(self) -> str:
91    def get_client_secret_name(self) -> str:
92        return self._client_secret_name

The client secret name to authenticate

def get_client_secret(self) -> str:
94    def get_client_secret(self) -> str:
95        return self._client_secret

The client secret to authenticate

def get_refresh_token_name(self) -> str:
97    def get_refresh_token_name(self) -> str:
98        return self._refresh_token_name

The refresh token name to authenticate

def get_refresh_token(self) -> str:
100    def get_refresh_token(self) -> str:
101        return self._refresh_token

The token used to refresh the access token when it expires

def get_access_token_name(self) -> str:
103    def get_access_token_name(self) -> str:
104        return self._access_token_name

Field to extract access token from in the response

def get_scopes(self) -> list[str]:
106    def get_scopes(self) -> list[str]:
107        return self._scopes  # type: ignore[return-value]

List of requested scopes

def get_expires_in_name(self) -> str:
109    def get_expires_in_name(self) -> str:
110        return self._expires_in_name

Returns the expires_in field name

def get_refresh_request_body(self) -> Mapping[str, Any]:
112    def get_refresh_request_body(self) -> Mapping[str, Any]:
113        return self._refresh_request_body  # type: ignore[return-value]

Returns the request body to set on the refresh request

def get_refresh_request_headers(self) -> Mapping[str, Any]:
115    def get_refresh_request_headers(self) -> Mapping[str, Any]:
116        return self._refresh_request_headers  # type: ignore[return-value]

Returns the request headers to set on the refresh request

def should_send_refresh_request_as_query_params(self) -> bool:
118    def should_send_refresh_request_as_query_params(self) -> bool:
119        return self._send_refresh_request_as_query_params

Returns True if the standard refresh args should be sent on the URL query string instead of in the request body.

Defaults to False so existing authenticators retain their previous behavior (params in body, no query params on the refresh URL). Subclasses can override this to opt into the URL-query-string shape required by OAuth providers like Gong.

def get_grant_type_name(self) -> str:
121    def get_grant_type_name(self) -> str:
122        return self._grant_type_name

Returns grant_type specified name for requesting access_token

def get_grant_type(self) -> str:
124    def get_grant_type(self) -> str:
125        return self._grant_type

Returns grant_type specified for requesting access_token

def get_token_expiry_date(self) -> airbyte_cdk.utils.datetime_helpers.AirbyteDateTime:
127    def get_token_expiry_date(self) -> AirbyteDateTime:
128        return self._token_expiry_date

Expiration date of the access token

def set_token_expiry_date(self, value: airbyte_cdk.utils.datetime_helpers.AirbyteDateTime) -> None:
130    def set_token_expiry_date(self, value: AirbyteDateTime) -> None:
131        # Overrides of `refresh_access_token` written before 6.45.5 return the raw `expires_in`.
132        if not isinstance(value, datetime):
133            with suppress(ValueError, OverflowError):
134                value = self._parse_token_expiration_date(value)
135        self._token_expiry_date = value

Setter for access token expiration date

token_expiry_is_time_of_expiration: bool
137    @property
138    def token_expiry_is_time_of_expiration(self) -> bool:
139        return self._token_expiry_is_time_of_expiration

Indicates that the Token Expiry returns the date until which the token will be valid, not the amount of time it will be valid.

token_expiry_date_format: Optional[str]
141    @property
142    def token_expiry_date_format(self) -> Optional[str]:
143        return self._token_expiry_date_format

Format of the datetime; exists it if expires_in is returned as the expiration datetime instead of seconds until it expires

access_token: str
145    @property
146    def access_token(self) -> str:
147        return self._access_token  # type: ignore[return-value]

Returns the access token

@dataclasses.dataclass
class Rate:
33@dataclasses.dataclass
34class Rate:
35    """Call rate limit"""
36
37    limit: int
38    interval: timedelta

Call rate limit

Rate(limit: int, interval: datetime.timedelta)
limit: int
interval: datetime.timedelta
class SingleUseRefreshTokenOauth2Authenticator(airbyte_cdk.Oauth2Authenticator):
154class SingleUseRefreshTokenOauth2Authenticator(Oauth2Authenticator):
155    """
156    Authenticator that should be used for API implementing single use refresh tokens:
157    when refreshing access token some API returns a new refresh token that needs to used in the next refresh flow.
158    This authenticator updates the configuration with new refresh token by emitting Airbyte control message from an observed mutation.
159    By default, this authenticator expects a connector config with a "credentials" field with the following nested fields: client_id,
160    client_secret, refresh_token. This behavior can be changed by defining custom config path (using dpath paths) in client_id_config_path,
161    client_secret_config_path, refresh_token_config_path constructor arguments.
162    """
163
164    def __init__(
165        self,
166        connector_config: Mapping[str, Any],
167        token_refresh_endpoint: str,
168        scopes: List[str] | None = None,
169        access_token_name: str = "access_token",
170        expires_in_name: str = "expires_in",
171        refresh_token_name: str = "refresh_token",
172        refresh_request_body: Mapping[str, Any] | None = None,
173        refresh_request_headers: Mapping[str, Any] | None = None,
174        send_refresh_request_as_query_params: bool = False,
175        grant_type_name: str = "grant_type",
176        grant_type: str = "refresh_token",
177        client_id_name: str = "client_id",
178        client_id: Optional[str] = None,
179        client_secret_name: str = "client_secret",
180        client_secret: Optional[str] = None,
181        access_token_config_path: Sequence[str] = ("credentials", "access_token"),
182        refresh_token_config_path: Sequence[str] = ("credentials", "refresh_token"),
183        token_expiry_date_config_path: Sequence[str] = ("credentials", "token_expiry_date"),
184        token_expiry_date_format: Optional[str] = None,
185        message_repository: MessageRepository = NoopMessageRepository(),
186        token_expiry_is_time_of_expiration: bool = False,
187        refresh_token_error_status_codes: Tuple[int, ...] = (),
188        refresh_token_error_key: str = "",
189        refresh_token_error_values: Tuple[str, ...] = (),
190        emit_control_message_to_message_repository: bool = False,
191    ) -> None:
192        """
193        Args:
194            connector_config (Mapping[str, Any]): The full connector configuration
195            token_refresh_endpoint (str): Full URL to the token refresh endpoint
196            scopes (List[str], optional): List of OAuth scopes to pass in the refresh token request body. Defaults to None.
197            access_token_name (str, optional): Name of the access token field, used to parse the refresh token response. Defaults to "access_token".
198            expires_in_name (str, optional): Name of the name of the field that characterizes when the current access token will expire, used to parse the refresh token response. Defaults to "expires_in".
199            refresh_token_name (str, optional): Name of the name of the refresh token field, used to parse the refresh token response. Defaults to "refresh_token".
200            refresh_request_body (Mapping[str, Any], optional): Custom key value pair that will be added to the refresh token request body. Defaults to None.
201            refresh_request_headers (Mapping[str, Any], optional): Custom key value pair that will be added to the refresh token request headers. Defaults to None.
202            send_refresh_request_as_query_params (bool, optional): When True, the standard refresh args (`grant_type`, `refresh_token`, client credentials when not in an `Authorization` header, scopes, plus any `refresh_request_body` extras) are sent on the URL query string and the request body is emitted empty. Use this for OAuth providers like Gong that document their refresh endpoint with refresh args on the URL query string. Defaults to False.
203            grant_type (str, optional): OAuth grant type. Defaults to "refresh_token".
204            client_id (Optional[str]): The client id to authenticate. If not specified, defaults to credentials.client_id in the config object.
205            client_secret (Optional[str]): The client secret to authenticate. If not specified, defaults to credentials.client_secret in the config object.
206            access_token_config_path (Sequence[str]): Dpath to the access_token field in the connector configuration. Defaults to ("credentials", "access_token").
207            refresh_token_config_path (Sequence[str]): Dpath to the refresh_token field in the connector configuration. Defaults to ("credentials", "refresh_token").
208            token_expiry_date_config_path (Sequence[str]): Dpath to the token_expiry_date field in the connector configuration. Defaults to ("credentials", "token_expiry_date").
209            token_expiry_date_format (Optional[str]): Date format of the token expiry date field (set by expires_in_name). If specified the token expiry date is interpreted as time of expiration, as with token_expiry_is_time_of_expiration, otherwise as number of seconds until expiration.
210            token_expiry_is_time_of_expiration bool: set True it if expires_in is returned as time of expiration instead of the number seconds until expiration
211            message_repository (MessageRepository): The message repository used to emit logs on HTTP requests and control message on config update.
212            emit_control_message_to_message_repository (bool): Whether the message repository additionally receives the CONNECTOR_CONFIG message. This is only for in-process consumers such as the Connector Builder, because messages in the repository are otherwise printed to stdout by the entrypoint and would duplicate the direct stdout emission. Defaults to False.
213        """
214        self._connector_config = connector_config
215        self._client_id: str = self._get_config_value_by_path(
216            ("credentials", "client_id"), client_id
217        )
218        self._client_secret: str = self._get_config_value_by_path(
219            ("credentials", "client_secret"), client_secret
220        )
221        self._client_id_name = client_id_name
222        self._client_secret_name = client_secret_name
223        self._access_token_config_path = access_token_config_path
224        self._refresh_token_config_path = refresh_token_config_path
225        self._token_expiry_date_config_path = token_expiry_date_config_path
226        self._token_expiry_date_format = token_expiry_date_format
227        self._refresh_token_name = refresh_token_name
228        self._grant_type_name = grant_type_name
229        self._connector_config = connector_config
230        self.__message_repository = message_repository
231        self._emit_control_message_to_message_repository = (
232            emit_control_message_to_message_repository
233        )
234        super().__init__(
235            token_refresh_endpoint=token_refresh_endpoint,
236            client_id_name=self._client_id_name,
237            client_id=self._client_id,
238            client_secret_name=self._client_secret_name,
239            client_secret=self._client_secret,
240            refresh_token=self.get_refresh_token(),
241            refresh_token_name=self._refresh_token_name,
242            scopes=scopes,
243            token_expiry_date=self.get_token_expiry_date(),
244            access_token_name=access_token_name,
245            expires_in_name=expires_in_name,
246            refresh_request_body=refresh_request_body,
247            refresh_request_headers=refresh_request_headers,
248            send_refresh_request_as_query_params=send_refresh_request_as_query_params,
249            grant_type_name=self._grant_type_name,
250            grant_type=grant_type,
251            token_expiry_date_format=token_expiry_date_format,
252            token_expiry_is_time_of_expiration=token_expiry_is_time_of_expiration
253            or bool(token_expiry_date_format),
254            refresh_token_error_status_codes=refresh_token_error_status_codes,
255            refresh_token_error_key=refresh_token_error_key,
256            refresh_token_error_values=refresh_token_error_values,
257        )
258
259    @property
260    def access_token(self) -> str:
261        """
262        Retrieve the access token from the configuration.
263
264        Returns:
265            str: The access token.
266        """
267        return self._get_config_value_by_path(self._access_token_config_path)  # type: ignore[return-value]
268
269    @access_token.setter
270    def access_token(self, new_access_token: str) -> None:
271        """
272        Sets a new access token.
273
274        Args:
275            new_access_token (str): The new access token to be set.
276        """
277        self._set_config_value_by_path(self._access_token_config_path, new_access_token)
278
279    def get_refresh_token(self) -> str:
280        """
281        Retrieve the refresh token from the configuration.
282
283        This method fetches the refresh token using the configuration path specified
284        by `_refresh_token_config_path`.
285
286        Returns:
287            str: The refresh token as a string.
288        """
289        return self._get_config_value_by_path(self._refresh_token_config_path)  # type: ignore[return-value]
290
291    def set_refresh_token(self, new_refresh_token: str) -> None:
292        """
293        Updates the refresh token in the configuration.
294
295        Args:
296            new_refresh_token (str): The new refresh token to be set.
297        """
298        self._set_config_value_by_path(self._refresh_token_config_path, new_refresh_token)
299
300    def get_token_expiry_date(self) -> AirbyteDateTime:
301        """
302        Retrieves the token expiry date from the configuration.
303
304        This method fetches the token expiry date from the configuration using the specified path.
305        If the expiry date is an empty string, it returns the current date and time minus one day.
306        Otherwise, it parses the expiry date string into an AirbyteDateTime object.
307
308        Returns:
309            AirbyteDateTime: The parsed or calculated token expiry date.
310
311        Raises:
312            TypeError: If the result is not an instance of AirbyteDateTime.
313        """
314        expiry_date = self._get_config_value_by_path(self._token_expiry_date_config_path)
315        result = (
316            ab_datetime_now() - timedelta(days=1)
317            if expiry_date == ""
318            else ab_datetime_parse(str(expiry_date))
319        )
320        if isinstance(result, AirbyteDateTime):
321            return result
322        raise TypeError("Invalid datetime conversion")
323
324    def set_token_expiry_date(self, new_token_expiry_date: AirbyteDateTime) -> None:  # type: ignore[override]
325        """
326        Sets the token expiry date in the configuration.
327
328        Args:
329            new_token_expiry_date (AirbyteDateTime): The new expiry date for the token.
330        """
331        if not (
332            isinstance(new_token_expiry_date, datetime) or self.token_expiry_is_time_of_expiration
333        ):
334            # Overrides of `refresh_access_token` written before 6.45.5 return the raw `expires_in`
335            # in seconds. Other values are stored as-is and parsed on read.
336            with suppress(ValueError, OverflowError):
337                new_token_expiry_date = self._parse_token_expiration_date(new_token_expiry_date)
338        self._set_config_value_by_path(
339            self._token_expiry_date_config_path, str(new_token_expiry_date)
340        )
341
342    def token_has_expired(self) -> bool:
343        """Returns True if the token is expired"""
344        return ab_datetime_now() > self.get_token_expiry_date()
345
346    def get_access_token(self) -> str:
347        """Retrieve new access and refresh token if the access token has expired.
348
349        This method uses double-checked locking to ensure thread-safe token refresh.
350        This is especially critical for single-use refresh tokens where concurrent
351        refresh attempts would cause failures as the refresh token is invalidated
352        after first use.
353
354        The new refresh token is persisted with the set_refresh_token function.
355
356        Returns:
357            str: The current access_token, updated if it was previously expired.
358        """
359        if self.token_has_expired():
360            with self._token_refresh_lock:
361                # Double-check after acquiring lock - another thread may have already refreshed
362                if self.token_has_expired():
363                    self.refresh_and_set_access_token()
364        return self.access_token
365
366    def refresh_and_set_access_token(self) -> None:
367        """Force refresh the access token and update internal state.
368
369        For single-use refresh tokens, this also persists the new refresh token
370        and emits a control message to update the connector config. If the
371        response omits a refresh token, the existing one is preserved.
372        """
373        token_before_waiting = self._current_access_token_or_none()
374        with self._token_refresh_lock:
375            if self._current_access_token_or_none() != token_before_waiting:
376                return
377            new_access_token, access_token_expires_in, new_refresh_token = (
378                self.refresh_access_token()
379            )
380            self.access_token = new_access_token
381            if new_refresh_token is not None:
382                self.set_refresh_token(new_refresh_token)
383            self.set_token_expiry_date(access_token_expires_in)
384            self._emit_control_message()
385
386    def refresh_access_token(self) -> Tuple[str, AirbyteDateTime, Optional[str]]:  # type: ignore[override]
387        """
388        Refreshes the access token by making a handled request and extracting the necessary token information.
389
390        Returns:
391            A tuple of (access_token, token_expiry_date, refresh_token). The refresh token
392            is `None` when the OAuth provider omits it from the response.
393        """
394        response_json = self._make_handled_request()
395        return (
396            self._extract_access_token(response_json),
397            self._extract_token_expiry_date(response_json),
398            self._extract_refresh_token(response_json),
399        )
400
401    def _set_config_value_by_path(self, config_path: Union[str, Sequence[str]], value: Any) -> None:
402        """
403        Set a value in the connector configuration at the specified path.
404
405        Args:
406            config_path (Union[str, Sequence[str]]): The path within the configuration where the value should be set.
407                This can be a string representing a single key or a sequence of strings representing a nested path.
408            value (Any): The value to set at the specified path in the configuration.
409
410        Returns:
411            None
412        """
413        dpath.new(self._connector_config, config_path, value)  # type: ignore[arg-type]
414
415    def _get_config_value_by_path(
416        self, config_path: Union[str, Sequence[str]], default: Optional[str] = None
417    ) -> str | Any:
418        """
419        Retrieve a value from the connector configuration using a specified path.
420
421        Args:
422            config_path (Union[str, Sequence[str]]): The path to the desired configuration value. This can be a string or a sequence of strings.
423            default (Optional[str], optional): The default value to return if the specified path does not exist in the configuration. Defaults to None.
424
425        Returns:
426            Any: The value from the configuration at the specified path, or the default value if the path does not exist.
427        """
428        return dpath.get(
429            self._connector_config,  # type: ignore[arg-type]
430            config_path,
431            default=default if default is not None else "",
432        )
433
434    def _emit_control_message(self) -> None:
435        """Emit the updated connector config as a CONNECTOR_CONFIG control message.
436
437        The message is always printed to stdout, which is the delivery the platform relies on: it
438        is immediate and independent of whether the source exposes a message repository and of
439        when that repository is drained (for `ConcurrentDeclarativeSource` the internal repository
440        is only drained during `read`, after partitions are processed). The message repository is
441        only used in addition when explicitly requested, for in-process consumers such as the
442        Connector Builder; repository messages are otherwise printed by the entrypoint too and
443        would duplicate the stdout emission.
444        """
445        emit_configuration_as_airbyte_control_message(self._connector_config)  # type: ignore[arg-type]
446        if self._emit_control_message_to_message_repository and not isinstance(
447            self._message_repository, NoopMessageRepository
448        ):
449            self._message_repository.emit_message(
450                create_connector_config_control_message(self._connector_config)  # type: ignore[arg-type]
451            )
452
453    @property
454    def _message_repository(self) -> MessageRepository:
455        """
456        Overriding AbstractOauth2Authenticator._message_repository to allow for HTTP request logs
457        """
458        return self.__message_repository

Authenticator that should be used for API implementing single use refresh tokens: when refreshing access token some API returns a new refresh token that needs to used in the next refresh flow. This authenticator updates the configuration with new refresh token by emitting Airbyte control message from an observed mutation. By default, this authenticator expects a connector config with a "credentials" field with the following nested fields: client_id, client_secret, refresh_token. This behavior can be changed by defining custom config path (using dpath paths) in client_id_config_path, client_secret_config_path, refresh_token_config_path constructor arguments.

SingleUseRefreshTokenOauth2Authenticator( connector_config: Mapping[str, Any], token_refresh_endpoint: str, scopes: Optional[List[str]] = None, access_token_name: str = 'access_token', expires_in_name: str = 'expires_in', refresh_token_name: str = 'refresh_token', refresh_request_body: Optional[Mapping[str, Any]] = None, refresh_request_headers: Optional[Mapping[str, Any]] = None, send_refresh_request_as_query_params: bool = False, grant_type_name: str = 'grant_type', grant_type: str = 'refresh_token', client_id_name: str = 'client_id', client_id: Optional[str] = None, client_secret_name: str = 'client_secret', client_secret: Optional[str] = None, access_token_config_path: Sequence[str] = ('credentials', 'access_token'), refresh_token_config_path: Sequence[str] = ('credentials', 'refresh_token'), token_expiry_date_config_path: Sequence[str] = ('credentials', 'token_expiry_date'), token_expiry_date_format: Optional[str] = None, message_repository: MessageRepository = <airbyte_cdk.sources.message.NoopMessageRepository object>, token_expiry_is_time_of_expiration: bool = False, refresh_token_error_status_codes: Tuple[int, ...] = (), refresh_token_error_key: str = '', refresh_token_error_values: Tuple[str, ...] = (), emit_control_message_to_message_repository: bool = False)
164    def __init__(
165        self,
166        connector_config: Mapping[str, Any],
167        token_refresh_endpoint: str,
168        scopes: List[str] | None = None,
169        access_token_name: str = "access_token",
170        expires_in_name: str = "expires_in",
171        refresh_token_name: str = "refresh_token",
172        refresh_request_body: Mapping[str, Any] | None = None,
173        refresh_request_headers: Mapping[str, Any] | None = None,
174        send_refresh_request_as_query_params: bool = False,
175        grant_type_name: str = "grant_type",
176        grant_type: str = "refresh_token",
177        client_id_name: str = "client_id",
178        client_id: Optional[str] = None,
179        client_secret_name: str = "client_secret",
180        client_secret: Optional[str] = None,
181        access_token_config_path: Sequence[str] = ("credentials", "access_token"),
182        refresh_token_config_path: Sequence[str] = ("credentials", "refresh_token"),
183        token_expiry_date_config_path: Sequence[str] = ("credentials", "token_expiry_date"),
184        token_expiry_date_format: Optional[str] = None,
185        message_repository: MessageRepository = NoopMessageRepository(),
186        token_expiry_is_time_of_expiration: bool = False,
187        refresh_token_error_status_codes: Tuple[int, ...] = (),
188        refresh_token_error_key: str = "",
189        refresh_token_error_values: Tuple[str, ...] = (),
190        emit_control_message_to_message_repository: bool = False,
191    ) -> None:
192        """
193        Args:
194            connector_config (Mapping[str, Any]): The full connector configuration
195            token_refresh_endpoint (str): Full URL to the token refresh endpoint
196            scopes (List[str], optional): List of OAuth scopes to pass in the refresh token request body. Defaults to None.
197            access_token_name (str, optional): Name of the access token field, used to parse the refresh token response. Defaults to "access_token".
198            expires_in_name (str, optional): Name of the name of the field that characterizes when the current access token will expire, used to parse the refresh token response. Defaults to "expires_in".
199            refresh_token_name (str, optional): Name of the name of the refresh token field, used to parse the refresh token response. Defaults to "refresh_token".
200            refresh_request_body (Mapping[str, Any], optional): Custom key value pair that will be added to the refresh token request body. Defaults to None.
201            refresh_request_headers (Mapping[str, Any], optional): Custom key value pair that will be added to the refresh token request headers. Defaults to None.
202            send_refresh_request_as_query_params (bool, optional): When True, the standard refresh args (`grant_type`, `refresh_token`, client credentials when not in an `Authorization` header, scopes, plus any `refresh_request_body` extras) are sent on the URL query string and the request body is emitted empty. Use this for OAuth providers like Gong that document their refresh endpoint with refresh args on the URL query string. Defaults to False.
203            grant_type (str, optional): OAuth grant type. Defaults to "refresh_token".
204            client_id (Optional[str]): The client id to authenticate. If not specified, defaults to credentials.client_id in the config object.
205            client_secret (Optional[str]): The client secret to authenticate. If not specified, defaults to credentials.client_secret in the config object.
206            access_token_config_path (Sequence[str]): Dpath to the access_token field in the connector configuration. Defaults to ("credentials", "access_token").
207            refresh_token_config_path (Sequence[str]): Dpath to the refresh_token field in the connector configuration. Defaults to ("credentials", "refresh_token").
208            token_expiry_date_config_path (Sequence[str]): Dpath to the token_expiry_date field in the connector configuration. Defaults to ("credentials", "token_expiry_date").
209            token_expiry_date_format (Optional[str]): Date format of the token expiry date field (set by expires_in_name). If specified the token expiry date is interpreted as time of expiration, as with token_expiry_is_time_of_expiration, otherwise as number of seconds until expiration.
210            token_expiry_is_time_of_expiration bool: set True it if expires_in is returned as time of expiration instead of the number seconds until expiration
211            message_repository (MessageRepository): The message repository used to emit logs on HTTP requests and control message on config update.
212            emit_control_message_to_message_repository (bool): Whether the message repository additionally receives the CONNECTOR_CONFIG message. This is only for in-process consumers such as the Connector Builder, because messages in the repository are otherwise printed to stdout by the entrypoint and would duplicate the direct stdout emission. Defaults to False.
213        """
214        self._connector_config = connector_config
215        self._client_id: str = self._get_config_value_by_path(
216            ("credentials", "client_id"), client_id
217        )
218        self._client_secret: str = self._get_config_value_by_path(
219            ("credentials", "client_secret"), client_secret
220        )
221        self._client_id_name = client_id_name
222        self._client_secret_name = client_secret_name
223        self._access_token_config_path = access_token_config_path
224        self._refresh_token_config_path = refresh_token_config_path
225        self._token_expiry_date_config_path = token_expiry_date_config_path
226        self._token_expiry_date_format = token_expiry_date_format
227        self._refresh_token_name = refresh_token_name
228        self._grant_type_name = grant_type_name
229        self._connector_config = connector_config
230        self.__message_repository = message_repository
231        self._emit_control_message_to_message_repository = (
232            emit_control_message_to_message_repository
233        )
234        super().__init__(
235            token_refresh_endpoint=token_refresh_endpoint,
236            client_id_name=self._client_id_name,
237            client_id=self._client_id,
238            client_secret_name=self._client_secret_name,
239            client_secret=self._client_secret,
240            refresh_token=self.get_refresh_token(),
241            refresh_token_name=self._refresh_token_name,
242            scopes=scopes,
243            token_expiry_date=self.get_token_expiry_date(),
244            access_token_name=access_token_name,
245            expires_in_name=expires_in_name,
246            refresh_request_body=refresh_request_body,
247            refresh_request_headers=refresh_request_headers,
248            send_refresh_request_as_query_params=send_refresh_request_as_query_params,
249            grant_type_name=self._grant_type_name,
250            grant_type=grant_type,
251            token_expiry_date_format=token_expiry_date_format,
252            token_expiry_is_time_of_expiration=token_expiry_is_time_of_expiration
253            or bool(token_expiry_date_format),
254            refresh_token_error_status_codes=refresh_token_error_status_codes,
255            refresh_token_error_key=refresh_token_error_key,
256            refresh_token_error_values=refresh_token_error_values,
257        )
Arguments:
  • connector_config (Mapping[str, Any]): The full connector configuration
  • token_refresh_endpoint (str): Full URL to the token refresh endpoint
  • scopes (List[str], optional): List of OAuth scopes to pass in the refresh token request body. Defaults to None.
  • access_token_name (str, optional): Name of the access token field, used to parse the refresh token response. Defaults to "access_token".
  • expires_in_name (str, optional): Name of the name of the field that characterizes when the current access token will expire, used to parse the refresh token response. Defaults to "expires_in".
  • refresh_token_name (str, optional): Name of the name of the refresh token field, used to parse the refresh token response. Defaults to "refresh_token".
  • refresh_request_body (Mapping[str, Any], optional): Custom key value pair that will be added to the refresh token request body. Defaults to None.
  • refresh_request_headers (Mapping[str, Any], optional): Custom key value pair that will be added to the refresh token request headers. Defaults to None.
  • send_refresh_request_as_query_params (bool, optional): When True, the standard refresh args (grant_type, refresh_token, client credentials when not in an Authorization header, scopes, plus any refresh_request_body extras) are sent on the URL query string and the request body is emitted empty. Use this for OAuth providers like Gong that document their refresh endpoint with refresh args on the URL query string. Defaults to False.
  • grant_type (str, optional): OAuth grant type. Defaults to "refresh_token".
  • client_id (Optional[str]): The client id to authenticate. If not specified, defaults to credentials.client_id in the config object.
  • client_secret (Optional[str]): The client secret to authenticate. If not specified, defaults to credentials.client_secret in the config object.
  • access_token_config_path (Sequence[str]): Dpath to the access_token field in the connector configuration. Defaults to ("credentials", "access_token").
  • refresh_token_config_path (Sequence[str]): Dpath to the refresh_token field in the connector configuration. Defaults to ("credentials", "refresh_token").
  • token_expiry_date_config_path (Sequence[str]): Dpath to the token_expiry_date field in the connector configuration. Defaults to ("credentials", "token_expiry_date").
  • token_expiry_date_format (Optional[str]): Date format of the token expiry date field (set by expires_in_name). If specified the token expiry date is interpreted as time of expiration, as with token_expiry_is_time_of_expiration, otherwise as number of seconds until expiration.
  • token_expiry_is_time_of_expiration bool: set True it if expires_in is returned as time of expiration instead of the number seconds until expiration
  • message_repository (MessageRepository): The message repository used to emit logs on HTTP requests and control message on config update.
  • emit_control_message_to_message_repository (bool): Whether the message repository additionally receives the CONNECTOR_CONFIG message. This is only for in-process consumers such as the Connector Builder, because messages in the repository are otherwise printed to stdout by the entrypoint and would duplicate the direct stdout emission. Defaults to False.
access_token: str
259    @property
260    def access_token(self) -> str:
261        """
262        Retrieve the access token from the configuration.
263
264        Returns:
265            str: The access token.
266        """
267        return self._get_config_value_by_path(self._access_token_config_path)  # type: ignore[return-value]

Retrieve the access token from the configuration.

Returns:

str: The access token.

def get_refresh_token(self) -> str:
279    def get_refresh_token(self) -> str:
280        """
281        Retrieve the refresh token from the configuration.
282
283        This method fetches the refresh token using the configuration path specified
284        by `_refresh_token_config_path`.
285
286        Returns:
287            str: The refresh token as a string.
288        """
289        return self._get_config_value_by_path(self._refresh_token_config_path)  # type: ignore[return-value]

Retrieve the refresh token from the configuration.

This method fetches the refresh token using the configuration path specified by _refresh_token_config_path.

Returns:

str: The refresh token as a string.

def set_refresh_token(self, new_refresh_token: str) -> None:
291    def set_refresh_token(self, new_refresh_token: str) -> None:
292        """
293        Updates the refresh token in the configuration.
294
295        Args:
296            new_refresh_token (str): The new refresh token to be set.
297        """
298        self._set_config_value_by_path(self._refresh_token_config_path, new_refresh_token)

Updates the refresh token in the configuration.

Arguments:
  • new_refresh_token (str): The new refresh token to be set.
def get_token_expiry_date(self) -> airbyte_cdk.utils.datetime_helpers.AirbyteDateTime:
300    def get_token_expiry_date(self) -> AirbyteDateTime:
301        """
302        Retrieves the token expiry date from the configuration.
303
304        This method fetches the token expiry date from the configuration using the specified path.
305        If the expiry date is an empty string, it returns the current date and time minus one day.
306        Otherwise, it parses the expiry date string into an AirbyteDateTime object.
307
308        Returns:
309            AirbyteDateTime: The parsed or calculated token expiry date.
310
311        Raises:
312            TypeError: If the result is not an instance of AirbyteDateTime.
313        """
314        expiry_date = self._get_config_value_by_path(self._token_expiry_date_config_path)
315        result = (
316            ab_datetime_now() - timedelta(days=1)
317            if expiry_date == ""
318            else ab_datetime_parse(str(expiry_date))
319        )
320        if isinstance(result, AirbyteDateTime):
321            return result
322        raise TypeError("Invalid datetime conversion")

Retrieves the token expiry date from the configuration.

This method fetches the token expiry date from the configuration using the specified path. If the expiry date is an empty string, it returns the current date and time minus one day. Otherwise, it parses the expiry date string into an AirbyteDateTime object.

Returns:

AirbyteDateTime: The parsed or calculated token expiry date.

Raises:
  • TypeError: If the result is not an instance of AirbyteDateTime.
def set_token_expiry_date( self, new_token_expiry_date: airbyte_cdk.utils.datetime_helpers.AirbyteDateTime) -> None:
324    def set_token_expiry_date(self, new_token_expiry_date: AirbyteDateTime) -> None:  # type: ignore[override]
325        """
326        Sets the token expiry date in the configuration.
327
328        Args:
329            new_token_expiry_date (AirbyteDateTime): The new expiry date for the token.
330        """
331        if not (
332            isinstance(new_token_expiry_date, datetime) or self.token_expiry_is_time_of_expiration
333        ):
334            # Overrides of `refresh_access_token` written before 6.45.5 return the raw `expires_in`
335            # in seconds. Other values are stored as-is and parsed on read.
336            with suppress(ValueError, OverflowError):
337                new_token_expiry_date = self._parse_token_expiration_date(new_token_expiry_date)
338        self._set_config_value_by_path(
339            self._token_expiry_date_config_path, str(new_token_expiry_date)
340        )

Sets the token expiry date in the configuration.

Arguments:
  • new_token_expiry_date (AirbyteDateTime): The new expiry date for the token.
def token_has_expired(self) -> bool:
342    def token_has_expired(self) -> bool:
343        """Returns True if the token is expired"""
344        return ab_datetime_now() > self.get_token_expiry_date()

Returns True if the token is expired

def get_access_token(self) -> str:
346    def get_access_token(self) -> str:
347        """Retrieve new access and refresh token if the access token has expired.
348
349        This method uses double-checked locking to ensure thread-safe token refresh.
350        This is especially critical for single-use refresh tokens where concurrent
351        refresh attempts would cause failures as the refresh token is invalidated
352        after first use.
353
354        The new refresh token is persisted with the set_refresh_token function.
355
356        Returns:
357            str: The current access_token, updated if it was previously expired.
358        """
359        if self.token_has_expired():
360            with self._token_refresh_lock:
361                # Double-check after acquiring lock - another thread may have already refreshed
362                if self.token_has_expired():
363                    self.refresh_and_set_access_token()
364        return self.access_token

Retrieve new access and refresh token if the access token has expired.

This method uses double-checked locking to ensure thread-safe token refresh. This is especially critical for single-use refresh tokens where concurrent refresh attempts would cause failures as the refresh token is invalidated after first use.

The new refresh token is persisted with the set_refresh_token function.

Returns:

str: The current access_token, updated if it was previously expired.

def refresh_and_set_access_token(self) -> None:
366    def refresh_and_set_access_token(self) -> None:
367        """Force refresh the access token and update internal state.
368
369        For single-use refresh tokens, this also persists the new refresh token
370        and emits a control message to update the connector config. If the
371        response omits a refresh token, the existing one is preserved.
372        """
373        token_before_waiting = self._current_access_token_or_none()
374        with self._token_refresh_lock:
375            if self._current_access_token_or_none() != token_before_waiting:
376                return
377            new_access_token, access_token_expires_in, new_refresh_token = (
378                self.refresh_access_token()
379            )
380            self.access_token = new_access_token
381            if new_refresh_token is not None:
382                self.set_refresh_token(new_refresh_token)
383            self.set_token_expiry_date(access_token_expires_in)
384            self._emit_control_message()

Force refresh the access token and update internal state.

For single-use refresh tokens, this also persists the new refresh token and emits a control message to update the connector config. If the response omits a refresh token, the existing one is preserved.

def refresh_access_token( self) -> Tuple[str, airbyte_cdk.utils.datetime_helpers.AirbyteDateTime, Optional[str]]:
386    def refresh_access_token(self) -> Tuple[str, AirbyteDateTime, Optional[str]]:  # type: ignore[override]
387        """
388        Refreshes the access token by making a handled request and extracting the necessary token information.
389
390        Returns:
391            A tuple of (access_token, token_expiry_date, refresh_token). The refresh token
392            is `None` when the OAuth provider omits it from the response.
393        """
394        response_json = self._make_handled_request()
395        return (
396            self._extract_access_token(response_json),
397            self._extract_token_expiry_date(response_json),
398            self._extract_refresh_token(response_json),
399        )

Refreshes the access token by making a handled request and extracting the necessary token information.

Returns:

A tuple of (access_token, token_expiry_date, refresh_token). The refresh token is None when the OAuth provider omits it from the response.

class TokenAuthenticator(airbyte_cdk.AbstractHeaderAuthenticator):
39class TokenAuthenticator(AbstractHeaderAuthenticator):
40    """
41    Builds auth header, based on the token provided.
42    The token is attached to each request via the `auth_header` header.
43    """
44
45    @property
46    def auth_header(self) -> str:
47        return self._auth_header
48
49    @property
50    def token(self) -> str:
51        return f"{self._auth_method} {self._token}"
52
53    def __init__(self, token: str, auth_method: str = "Bearer", auth_header: str = "Authorization"):
54        self._auth_header = auth_header
55        self._auth_method = auth_method
56        self._token = token

Builds auth header, based on the token provided. The token is attached to each request via the auth_header header.

TokenAuthenticator( token: str, auth_method: str = 'Bearer', auth_header: str = 'Authorization')
53    def __init__(self, token: str, auth_method: str = "Bearer", auth_header: str = "Authorization"):
54        self._auth_header = auth_header
55        self._auth_method = auth_method
56        self._token = token
auth_header: str
45    @property
46    def auth_header(self) -> str:
47        return self._auth_header

HTTP header to set on the requests

token: str
49    @property
50    def token(self) -> str:
51        return f"{self._auth_method} {self._token}"

The header value to set on outgoing HTTP requests

class UserDefinedBackoffException(airbyte_cdk.BaseBackoffException):
40class UserDefinedBackoffException(BaseBackoffException):
41    """
42    An exception that exposes how long it attempted to backoff
43    """
44
45    def __init__(
46        self,
47        backoff: Union[int, float],
48        request: requests.PreparedRequest,
49        response: Optional[Union[requests.Response, Exception]],
50        error_message: str = "",
51        failure_type: Optional[FailureType] = None,
52    ):
53        """
54        :param backoff: how long to backoff in seconds
55        :param request: the request that triggered this backoff exception
56        :param response: the response that triggered the backoff exception
57        """
58        self.backoff = backoff
59        super().__init__(
60            request=request,
61            response=response,
62            error_message=error_message,
63            failure_type=failure_type,
64        )

An exception that exposes how long it attempted to backoff

UserDefinedBackoffException( backoff: Union[int, float], request: requests.models.PreparedRequest, response: Union[requests.models.Response, Exception, NoneType], error_message: str = '', failure_type: Optional[airbyte_protocol_dataclasses.models.airbyte_protocol.FailureType] = None)
45    def __init__(
46        self,
47        backoff: Union[int, float],
48        request: requests.PreparedRequest,
49        response: Optional[Union[requests.Response, Exception]],
50        error_message: str = "",
51        failure_type: Optional[FailureType] = None,
52    ):
53        """
54        :param backoff: how long to backoff in seconds
55        :param request: the request that triggered this backoff exception
56        :param response: the response that triggered the backoff exception
57        """
58        self.backoff = backoff
59        super().__init__(
60            request=request,
61            response=response,
62            error_message=error_message,
63            failure_type=failure_type,
64        )
Parameters
  • backoff: how long to backoff in seconds
  • request: the request that triggered this backoff exception
  • response: the response that triggered the backoff exception
backoff
class AirbyteLogFormatter(logging.Formatter):
 64class AirbyteLogFormatter(logging.Formatter):
 65    """Output log records using AirbyteMessage"""
 66
 67    # Transforming Python log levels to Airbyte protocol log levels
 68    level_mapping = {
 69        logging.FATAL: Level.FATAL,
 70        logging.ERROR: Level.ERROR,
 71        logging.WARNING: Level.WARN,
 72        logging.INFO: Level.INFO,
 73        logging.DEBUG: Level.DEBUG,
 74    }
 75
 76    def format(self, record: logging.LogRecord) -> str:
 77        """Return a JSON representation of the log message"""
 78        airbyte_level = self.level_mapping.get(record.levelno, "INFO")
 79        if airbyte_level == Level.DEBUG:
 80            extras = self.extract_extra_args_from_record(record)
 81            if is_platform_debug_log_enabled():
 82                # We have a different behavior between debug logs enabled through `--debug` argument and debug logs
 83                # enabled through environment variable. The reason is that for platform logs, we need to have these
 84                # printed as AirbyteMessage which is not the case with the current previous implementation.
 85                # Why not migrate both to AirbyteMessages then? AirbyteMessages do not support having structured logs.
 86                # which means that the DX would be degraded compared to the current solution (devs will need to identify
 87                # the `log.message` field and figure out where in this field is the response while the current solution
 88                # have a specific field that is structured for extras.
 89                message = f"{filter_secrets(record.getMessage())} ///\nExtra logs: {filter_secrets(json.dumps(extras))}"
 90                log_message = AirbyteMessage(
 91                    type=Type.LOG, log=AirbyteLogMessage(level=airbyte_level, message=message)
 92                )
 93                return orjson.dumps(AirbyteMessageSerializer.dump(log_message)).decode()
 94            else:
 95                debug_dict = {"type": "DEBUG", "message": record.getMessage(), "data": extras}
 96                return filter_secrets(json.dumps(debug_dict))
 97        else:
 98            message = super().format(record)
 99            message = filter_secrets(message)
100            log_message = AirbyteMessage(
101                type=Type.LOG, log=AirbyteLogMessage(level=airbyte_level, message=message)
102            )
103            return orjson.dumps(AirbyteMessageSerializer.dump(log_message)).decode()
104
105    @staticmethod
106    def extract_extra_args_from_record(record: logging.LogRecord) -> Mapping[str, Any]:
107        """
108        The python logger conflates default args with extra args. We use an empty log record and set operations
109        to isolate fields passed to the log record via extra by the developer.
110        """
111        default_attrs = logging.LogRecord("", 0, "", 0, None, None, None).__dict__.keys()
112        extra_keys = set(record.__dict__.keys()) - default_attrs
113        return {k: str(getattr(record, k)) for k in extra_keys if hasattr(record, k)}

Output log records using AirbyteMessage

level_mapping = {50: <Level.FATAL: 'FATAL'>, 40: <Level.ERROR: 'ERROR'>, 30: <Level.WARN: 'WARN'>, 20: <Level.INFO: 'INFO'>, 10: <Level.DEBUG: 'DEBUG'>}
def format(self, record: logging.LogRecord) -> str:
 76    def format(self, record: logging.LogRecord) -> str:
 77        """Return a JSON representation of the log message"""
 78        airbyte_level = self.level_mapping.get(record.levelno, "INFO")
 79        if airbyte_level == Level.DEBUG:
 80            extras = self.extract_extra_args_from_record(record)
 81            if is_platform_debug_log_enabled():
 82                # We have a different behavior between debug logs enabled through `--debug` argument and debug logs
 83                # enabled through environment variable. The reason is that for platform logs, we need to have these
 84                # printed as AirbyteMessage which is not the case with the current previous implementation.
 85                # Why not migrate both to AirbyteMessages then? AirbyteMessages do not support having structured logs.
 86                # which means that the DX would be degraded compared to the current solution (devs will need to identify
 87                # the `log.message` field and figure out where in this field is the response while the current solution
 88                # have a specific field that is structured for extras.
 89                message = f"{filter_secrets(record.getMessage())} ///\nExtra logs: {filter_secrets(json.dumps(extras))}"
 90                log_message = AirbyteMessage(
 91                    type=Type.LOG, log=AirbyteLogMessage(level=airbyte_level, message=message)
 92                )
 93                return orjson.dumps(AirbyteMessageSerializer.dump(log_message)).decode()
 94            else:
 95                debug_dict = {"type": "DEBUG", "message": record.getMessage(), "data": extras}
 96                return filter_secrets(json.dumps(debug_dict))
 97        else:
 98            message = super().format(record)
 99            message = filter_secrets(message)
100            log_message = AirbyteMessage(
101                type=Type.LOG, log=AirbyteLogMessage(level=airbyte_level, message=message)
102            )
103            return orjson.dumps(AirbyteMessageSerializer.dump(log_message)).decode()

Return a JSON representation of the log message

@staticmethod
def extract_extra_args_from_record(record: logging.LogRecord) -> Mapping[str, Any]:
105    @staticmethod
106    def extract_extra_args_from_record(record: logging.LogRecord) -> Mapping[str, Any]:
107        """
108        The python logger conflates default args with extra args. We use an empty log record and set operations
109        to isolate fields passed to the log record via extra by the developer.
110        """
111        default_attrs = logging.LogRecord("", 0, "", 0, None, None, None).__dict__.keys()
112        extra_keys = set(record.__dict__.keys()) - default_attrs
113        return {k: str(getattr(record, k)) for k in extra_keys if hasattr(record, k)}

The python logger conflates default args with extra args. We use an empty log record and set operations to isolate fields passed to the log record via extra by the developer.

def init_logger(name: Optional[str] = None) -> logging.Logger:
48def init_logger(name: Optional[str] = None) -> logging.Logger:
49    """Initial set up of logger"""
50    logger = logging.getLogger(name)
51    logger.setLevel(logging.INFO)
52    logging.config.dictConfig(LOGGING_CONFIG)
53    return logger

Initial set up of logger

@dataclass
class AirbyteStream:
300@dataclass
301class AirbyteStream:
302    name: str
303    json_schema: Dict[str, Any]
304    supported_sync_modes: List[SyncMode]
305    source_defined_cursor: Optional[bool] = None
306    default_cursor_field: Optional[List[str]] = None
307    source_defined_primary_key: Optional[List[List[str]]] = None
308    namespace: Optional[str] = None
309    is_resumable: Optional[bool] = None
310    is_file_based: Optional[bool] = None
AirbyteStream( name: str, json_schema: Dict[str, Any], supported_sync_modes: List[airbyte_protocol_dataclasses.models.airbyte_protocol.SyncMode], source_defined_cursor: Optional[bool] = None, default_cursor_field: Optional[List[str]] = None, source_defined_primary_key: Optional[List[List[str]]] = None, namespace: Optional[str] = None, is_resumable: Optional[bool] = None, is_file_based: Optional[bool] = None)
name: str
json_schema: Dict[str, Any]
supported_sync_modes: List[airbyte_protocol_dataclasses.models.airbyte_protocol.SyncMode]
source_defined_cursor: Optional[bool] = None
default_cursor_field: Optional[List[str]] = None
source_defined_primary_key: Optional[List[List[str]]] = None
namespace: Optional[str] = None
is_resumable: Optional[bool] = None
is_file_based: Optional[bool] = None
@dataclass
class AirbyteConnectionStatus:
180@dataclass
181class AirbyteConnectionStatus:
182    status: Status
183    message: Optional[str] = None
AirbyteConnectionStatus( status: airbyte_protocol_dataclasses.models.airbyte_protocol.Status, message: Optional[str] = None)
status: airbyte_protocol_dataclasses.models.airbyte_protocol.Status
message: Optional[str] = None
@dataclass
class AirbyteMessage:
140@dataclass
141class AirbyteMessage:
142    type: Type  # type: ignore [name-defined]
143    log: Optional[AirbyteLogMessage] = None  # type: ignore [name-defined]
144    spec: Optional[ConnectorSpecification] = None
145    connectionStatus: Optional[AirbyteConnectionStatus] = None  # type: ignore [name-defined]
146    catalog: Optional[AirbyteCatalog] = None  # type: ignore [name-defined]
147    record: Optional[AirbyteRecordMessage] = None  # type: ignore [name-defined]
148    state: Optional[AirbyteStateMessage] = None
149    trace: Optional[AirbyteTraceMessage] = None  # type: ignore [name-defined]
150    control: Optional[AirbyteControlMessage] = None  # type: ignore [name-defined]
AirbyteMessage( type: airbyte_protocol_dataclasses.models.airbyte_protocol.Type, log: Optional[airbyte_protocol_dataclasses.models.airbyte_protocol.AirbyteLogMessage] = None, spec: Optional[ConnectorSpecification] = None, connectionStatus: Optional[airbyte_protocol_dataclasses.models.airbyte_protocol.AirbyteConnectionStatus] = None, catalog: Optional[airbyte_protocol_dataclasses.models.airbyte_protocol.AirbyteCatalog] = None, record: Optional[airbyte_protocol_dataclasses.models.airbyte_protocol.AirbyteRecordMessage] = None, state: Optional[airbyte_cdk.models.airbyte_protocol.AirbyteStateMessage] = None, trace: Optional[airbyte_protocol_dataclasses.models.airbyte_protocol.AirbyteTraceMessage] = None, control: Optional[airbyte_protocol_dataclasses.models.airbyte_protocol.AirbyteControlMessage] = None)
type: airbyte_protocol_dataclasses.models.airbyte_protocol.Type
log: Optional[airbyte_protocol_dataclasses.models.airbyte_protocol.AirbyteLogMessage] = None
spec: Optional[ConnectorSpecification] = None
connectionStatus: Optional[airbyte_protocol_dataclasses.models.airbyte_protocol.AirbyteConnectionStatus] = None
catalog: Optional[airbyte_protocol_dataclasses.models.airbyte_protocol.AirbyteCatalog] = None
record: Optional[airbyte_protocol_dataclasses.models.airbyte_protocol.AirbyteRecordMessage] = None
trace: Optional[airbyte_protocol_dataclasses.models.airbyte_protocol.AirbyteTraceMessage] = None
control: Optional[airbyte_protocol_dataclasses.models.airbyte_protocol.AirbyteControlMessage] = None
@dataclass
class ConfiguredAirbyteCatalog:
379@dataclass
380class ConfiguredAirbyteCatalog:
381    streams: List[ConfiguredAirbyteStream]
ConfiguredAirbyteCatalog( streams: List[airbyte_protocol_dataclasses.models.airbyte_protocol.ConfiguredAirbyteStream])
streams: List[airbyte_protocol_dataclasses.models.airbyte_protocol.ConfiguredAirbyteStream]
class Status(enum.Enum):
175class Status(Enum):
176    SUCCEEDED = 'SUCCEEDED'
177    FAILED = 'FAILED'
SUCCEEDED = <Status.SUCCEEDED: 'SUCCEEDED'>
FAILED = <Status.FAILED: 'FAILED'>
class Type(enum.Enum):
12class Type(Enum):
13    RECORD = 'RECORD'
14    STATE = 'STATE'
15    LOG = 'LOG'
16    SPEC = 'SPEC'
17    CONNECTION_STATUS = 'CONNECTION_STATUS'
18    CATALOG = 'CATALOG'
19    TRACE = 'TRACE'
20    CONTROL = 'CONTROL'
21    DESTINATION_CATALOG = 'DESTINATION_CATALOG'
RECORD = <Type.RECORD: 'RECORD'>
STATE = <Type.STATE: 'STATE'>
LOG = <Type.LOG: 'LOG'>
SPEC = <Type.SPEC: 'SPEC'>
CONNECTION_STATUS = <Type.CONNECTION_STATUS: 'CONNECTION_STATUS'>
CATALOG = <Type.CATALOG: 'CATALOG'>
TRACE = <Type.TRACE: 'TRACE'>
CONTROL = <Type.CONTROL: 'CONTROL'>
DESTINATION_CATALOG = <Type.DESTINATION_CATALOG: 'DESTINATION_CATALOG'>
class OrchestratorType(enum.Enum):
166class OrchestratorType(Enum):
167    CONNECTOR_CONFIG = 'CONNECTOR_CONFIG'
CONNECTOR_CONFIG = <OrchestratorType.CONNECTOR_CONFIG: 'CONNECTOR_CONFIG'>
@dataclass
class ConfiguredAirbyteStream:
313@dataclass
314class ConfiguredAirbyteStream:
315    stream: AirbyteStream
316    sync_mode: SyncMode
317    destination_sync_mode: DestinationSyncMode
318    cursor_field: Optional[List[str]] = None
319    destination_object_name: Optional[str] = None
320    primary_key: Optional[List[List[str]]] = None
321    generation_id: Optional[int] = None
322    minimum_generation_id: Optional[int] = None
323    sync_id: Optional[int] = None
324    include_files: Optional[bool] = None
ConfiguredAirbyteStream( stream: airbyte_protocol_dataclasses.models.airbyte_protocol.AirbyteStream, sync_mode: airbyte_protocol_dataclasses.models.airbyte_protocol.SyncMode, destination_sync_mode: airbyte_protocol_dataclasses.models.airbyte_protocol.DestinationSyncMode, cursor_field: Optional[List[str]] = None, destination_object_name: Optional[str] = None, primary_key: Optional[List[List[str]]] = None, generation_id: Optional[int] = None, minimum_generation_id: Optional[int] = None, sync_id: Optional[int] = None, include_files: Optional[bool] = None)
stream: airbyte_protocol_dataclasses.models.airbyte_protocol.AirbyteStream
sync_mode: airbyte_protocol_dataclasses.models.airbyte_protocol.SyncMode
destination_sync_mode: airbyte_protocol_dataclasses.models.airbyte_protocol.DestinationSyncMode
cursor_field: Optional[List[str]] = None
destination_object_name: Optional[str] = None
primary_key: Optional[List[List[str]]] = None
generation_id: Optional[int] = None
minimum_generation_id: Optional[int] = None
sync_id: Optional[int] = None
include_files: Optional[bool] = None
class DestinationSyncMode(enum.Enum):
191class DestinationSyncMode(Enum):
192    append = 'append'
193    overwrite = 'overwrite'
194    append_dedup = 'append_dedup'
195    update = 'update'
196    soft_delete = 'soft_delete'
append = <DestinationSyncMode.append: 'append'>
overwrite = <DestinationSyncMode.overwrite: 'overwrite'>
append_dedup = <DestinationSyncMode.append_dedup: 'append_dedup'>
update = <DestinationSyncMode.update: 'update'>
soft_delete = <DestinationSyncMode.soft_delete: 'soft_delete'>
class SyncMode(enum.Enum):
186class SyncMode(Enum):
187    full_refresh = 'full_refresh'
188    incremental = 'incremental'
full_refresh = <SyncMode.full_refresh: 'full_refresh'>
incremental = <SyncMode.incremental: 'incremental'>
class FailureType(enum.Enum):
102class FailureType(Enum):
103    system_error = 'system_error'
104    config_error = 'config_error'
105    transient_error = 'transient_error'
system_error = <FailureType.system_error: 'system_error'>
config_error = <FailureType.config_error: 'config_error'>
transient_error = <FailureType.transient_error: 'transient_error'>
@dataclass
class AdvancedAuth:
118@dataclass
119class AdvancedAuth:
120    auth_flow_type: Optional[AuthFlowType] = None  # type: ignore [name-defined]
121    predicate_key: Optional[List[str]] = None
122    predicate_value: Optional[str] = None
123    oauth_config_specification: Optional[OAuthConfigSpecification] = None
AdvancedAuth( auth_flow_type: Optional[airbyte_protocol_dataclasses.models.airbyte_protocol.AuthFlowType] = None, predicate_key: Optional[List[str]] = None, predicate_value: Optional[str] = None, oauth_config_specification: Optional[OAuthConfigSpecification] = None)
auth_flow_type: Optional[airbyte_protocol_dataclasses.models.airbyte_protocol.AuthFlowType] = None
predicate_key: Optional[List[str]] = None
predicate_value: Optional[str] = None
oauth_config_specification: Optional[OAuthConfigSpecification] = None
@dataclass
class AirbyteLogMessage:
88@dataclass
89class AirbyteLogMessage:
90    level: Level
91    message: str
92    stack_trace: Optional[str] = None
AirbyteLogMessage( level: airbyte_protocol_dataclasses.models.airbyte_protocol.Level, message: str, stack_trace: Optional[str] = None)
level: airbyte_protocol_dataclasses.models.airbyte_protocol.Level
message: str
stack_trace: Optional[str] = None
@dataclass
class OAuthConfigSpecification:
109@dataclass
110class OAuthConfigSpecification:
111    oauth_user_input_from_connector_config_specification: Optional[Dict[str, Any]] = None
112    oauth_connector_input_specification: Optional[OauthConnectorInputSpecification] = None
113    complete_oauth_output_specification: Optional[Dict[str, Any]] = None
114    complete_oauth_server_input_specification: Optional[Dict[str, Any]] = None
115    complete_oauth_server_output_specification: Optional[Dict[str, Any]] = None
OAuthConfigSpecification( oauth_user_input_from_connector_config_specification: Optional[Dict[str, Any]] = None, oauth_connector_input_specification: Optional[airbyte_cdk.models.airbyte_protocol.OauthConnectorInputSpecification] = None, complete_oauth_output_specification: Optional[Dict[str, Any]] = None, complete_oauth_server_input_specification: Optional[Dict[str, Any]] = None, complete_oauth_server_output_specification: Optional[Dict[str, Any]] = None)
oauth_user_input_from_connector_config_specification: Optional[Dict[str, Any]] = None
oauth_connector_input_specification: Optional[airbyte_cdk.models.airbyte_protocol.OauthConnectorInputSpecification] = None
complete_oauth_output_specification: Optional[Dict[str, Any]] = None
complete_oauth_server_input_specification: Optional[Dict[str, Any]] = None
complete_oauth_server_output_specification: Optional[Dict[str, Any]] = None
@dataclass
class ConnectorSpecification:
126@dataclass
127class ConnectorSpecification:
128    connectionSpecification: Dict[str, Any]
129    documentationUrl: Optional[str] = None
130    changelogUrl: Optional[str] = None
131    supportsIncremental: Optional[bool] = None
132    supportsNormalization: Optional[bool] = False
133    supportsDBT: Optional[bool] = False
134    supported_destination_sync_modes: Optional[List[DestinationSyncMode]] = None  # type: ignore [name-defined]
135    authSpecification: Optional[AuthSpecification] = None  # type: ignore [name-defined]
136    advanced_auth: Optional[AdvancedAuth] = None
137    protocol_version: Optional[str] = None
ConnectorSpecification( connectionSpecification: Dict[str, Any], documentationUrl: Optional[str] = None, changelogUrl: Optional[str] = None, supportsIncremental: Optional[bool] = None, supportsNormalization: Optional[bool] = False, supportsDBT: Optional[bool] = False, supported_destination_sync_modes: Optional[List[airbyte_protocol_dataclasses.models.airbyte_protocol.DestinationSyncMode]] = None, authSpecification: Optional[airbyte_protocol_dataclasses.models.airbyte_protocol.AuthSpecification] = None, advanced_auth: Optional[AdvancedAuth] = None, protocol_version: Optional[str] = None)
connectionSpecification: Dict[str, Any]
documentationUrl: Optional[str] = None
changelogUrl: Optional[str] = None
supportsIncremental: Optional[bool] = None
supportsNormalization: Optional[bool] = False
supportsDBT: Optional[bool] = False
supported_destination_sync_modes: Optional[List[airbyte_protocol_dataclasses.models.airbyte_protocol.DestinationSyncMode]] = None
authSpecification: Optional[airbyte_protocol_dataclasses.models.airbyte_protocol.AuthSpecification] = None
advanced_auth: Optional[AdvancedAuth] = None
protocol_version: Optional[str] = None
class Level(enum.Enum):
79class Level(Enum):
80    FATAL = 'FATAL'
81    ERROR = 'ERROR'
82    WARN = 'WARN'
83    INFO = 'INFO'
84    DEBUG = 'DEBUG'
85    TRACE = 'TRACE'
FATAL = <Level.FATAL: 'FATAL'>
ERROR = <Level.ERROR: 'ERROR'>
WARN = <Level.WARN: 'WARN'>
INFO = <Level.INFO: 'INFO'>
DEBUG = <Level.DEBUG: 'DEBUG'>
TRACE = <Level.TRACE: 'TRACE'>
@dataclass
class AirbyteRecordMessage:
354@dataclass
355class AirbyteRecordMessage:
356    stream: str
357    data: Dict[str, Any]
358    emitted_at: int
359    namespace: Optional[str] = None
360    meta: Optional[AirbyteRecordMessageMeta] = None
361    file_reference: Optional[AirbyteRecordMessageFileReference] = None
AirbyteRecordMessage( stream: str, data: Dict[str, Any], emitted_at: int, namespace: Optional[str] = None, meta: Optional[airbyte_protocol_dataclasses.models.airbyte_protocol.AirbyteRecordMessageMeta] = None, file_reference: Optional[airbyte_protocol_dataclasses.models.airbyte_protocol.AirbyteRecordMessageFileReference] = None)
stream: str
data: Dict[str, Any]
emitted_at: int
namespace: Optional[str] = None
meta: Optional[airbyte_protocol_dataclasses.models.airbyte_protocol.AirbyteRecordMessageMeta] = None
file_reference: Optional[airbyte_protocol_dataclasses.models.airbyte_protocol.AirbyteRecordMessageFileReference] = None
class InMemoryMessageRepository(airbyte_cdk.MessageRepository):
75class InMemoryMessageRepository(MessageRepository):
76    def __init__(self, log_level: Level = Level.INFO) -> None:
77        self._message_queue: Deque[AirbyteMessage] = deque()
78        self._log_level = log_level
79
80    def emit_message(self, message: AirbyteMessage) -> None:
81        self._message_queue.append(message)
82
83    def log_message(self, level: Level, message_provider: Callable[[], LogMessage]) -> None:
84        if _is_severe_enough(self._log_level, level):
85            self.emit_message(
86                AirbyteMessage(
87                    type=Type.LOG,
88                    log=AirbyteLogMessage(
89                        level=level, message=filter_secrets(json.dumps(message_provider()))
90                    ),
91                )
92            )
93
94    def consume_queue(self) -> Iterable[AirbyteMessage]:
95        while self._message_queue:
96            yield self._message_queue.popleft()

Helper class that provides a standard way to create an ABC using inheritance.

InMemoryMessageRepository( log_level: airbyte_protocol_dataclasses.models.airbyte_protocol.Level = <Level.INFO: 'INFO'>)
76    def __init__(self, log_level: Level = Level.INFO) -> None:
77        self._message_queue: Deque[AirbyteMessage] = deque()
78        self._log_level = log_level
def emit_message( self, message: AirbyteMessage) -> None:
80    def emit_message(self, message: AirbyteMessage) -> None:
81        self._message_queue.append(message)
def log_message( self, level: airbyte_protocol_dataclasses.models.airbyte_protocol.Level, message_provider: Callable[[], dict[str, Union[dict[str, Union[dict[str, ForwardRef('JsonType')], list[ForwardRef('JsonType')], str, int, float, bool, NoneType]], list[Union[dict[str, ForwardRef('JsonType')], list[ForwardRef('JsonType')], str, int, float, bool, NoneType]], str, int, float, bool, NoneType]]]) -> None:
83    def log_message(self, level: Level, message_provider: Callable[[], LogMessage]) -> None:
84        if _is_severe_enough(self._log_level, level):
85            self.emit_message(
86                AirbyteMessage(
87                    type=Type.LOG,
88                    log=AirbyteLogMessage(
89                        level=level, message=filter_secrets(json.dumps(message_provider()))
90                    ),
91                )
92            )

Computing messages can be resource consuming. This method is specialized for logging because we want to allow for lazy evaluation if the log level is less severe than what is configured

def consume_queue(self) -> Iterable[AirbyteMessage]:
94    def consume_queue(self) -> Iterable[AirbyteMessage]:
95        while self._message_queue:
96            yield self._message_queue.popleft()
class MessageRepository(abc.ABC):
46class MessageRepository(ABC):
47    @abstractmethod
48    def emit_message(self, message: AirbyteMessage) -> None:
49        raise NotImplementedError()
50
51    @abstractmethod
52    def log_message(self, level: Level, message_provider: Callable[[], LogMessage]) -> None:
53        """
54        Computing messages can be resource consuming. This method is specialized for logging because we want to allow for lazy evaluation if
55        the log level is less severe than what is configured
56        """
57        raise NotImplementedError()
58
59    @abstractmethod
60    def consume_queue(self) -> Iterable[AirbyteMessage]:
61        raise NotImplementedError()

Helper class that provides a standard way to create an ABC using inheritance.

@abstractmethod
def emit_message( self, message: AirbyteMessage) -> None:
47    @abstractmethod
48    def emit_message(self, message: AirbyteMessage) -> None:
49        raise NotImplementedError()
@abstractmethod
def log_message( self, level: airbyte_protocol_dataclasses.models.airbyte_protocol.Level, message_provider: Callable[[], dict[str, Union[dict[str, Union[dict[str, ForwardRef('JsonType')], list[ForwardRef('JsonType')], str, int, float, bool, NoneType]], list[Union[dict[str, ForwardRef('JsonType')], list[ForwardRef('JsonType')], str, int, float, bool, NoneType]], str, int, float, bool, NoneType]]]) -> None:
51    @abstractmethod
52    def log_message(self, level: Level, message_provider: Callable[[], LogMessage]) -> None:
53        """
54        Computing messages can be resource consuming. This method is specialized for logging because we want to allow for lazy evaluation if
55        the log level is less severe than what is configured
56        """
57        raise NotImplementedError()

Computing messages can be resource consuming. This method is specialized for logging because we want to allow for lazy evaluation if the log level is less severe than what is configured

@abstractmethod
def consume_queue(self) -> Iterable[AirbyteMessage]:
59    @abstractmethod
60    def consume_queue(self) -> Iterable[AirbyteMessage]:
61        raise NotImplementedError()
class ConnectorStateManager:
 33class ConnectorStateManager:
 34    """
 35    ConnectorStateManager consolidates the various forms of a stream's incoming state message (STREAM / GLOBAL) under a common
 36    interface. It also provides methods to extract and update state
 37    """
 38
 39    def __init__(self, state: Optional[List[AirbyteStateMessage]] = None):
 40        shared_state, per_stream_states = self._extract_from_state_message(state)
 41
 42        # We explicitly throw an error if we receive a GLOBAL state message that contains a shared_state because API sources are
 43        # designed to checkpoint state independently of one another. API sources should never be emitting a state message where
 44        # shared_state is populated. Rather than define how to handle shared_state without a clear use case, we're opting to throw an
 45        # error instead and if/when we find one, we will then implement processing of the shared_state value.
 46        if shared_state:
 47            raise ValueError(
 48                "Received a GLOBAL AirbyteStateMessage that contains a shared_state. This library only ever generates per-STREAM "
 49                "STATE messages so this was not generated by this connector. This must be an orchestrator or platform error. GLOBAL "
 50                "state messages with shared_state will not be processed correctly. "
 51            )
 52        self.per_stream_states = per_stream_states
 53
 54    def get_stream_state(
 55        self, stream_name: str, namespace: Optional[str]
 56    ) -> MutableMapping[str, Any]:
 57        """
 58        Retrieves the state of a given stream based on its descriptor (name + namespace).
 59        :param stream_name: Name of the stream being fetched
 60        :param namespace: Namespace of the stream being fetched
 61        :return: The per-stream state for a stream
 62        """
 63        stream_state: AirbyteStateBlob | None = self.per_stream_states.get(
 64            HashableStreamDescriptor(name=stream_name, namespace=namespace)
 65        )
 66        if stream_state:
 67            return copy.deepcopy({k: v for k, v in stream_state.__dict__.items()})
 68        return {}
 69
 70    def update_state_for_stream(
 71        self, stream_name: str, namespace: Optional[str], value: Mapping[str, Any]
 72    ) -> None:
 73        """
 74        Overwrites the state blob of a specific stream based on the provided stream name and optional namespace
 75        :param stream_name: The name of the stream whose state is being updated
 76        :param namespace: The namespace of the stream if it exists
 77        :param value: A stream state mapping that is being updated for a stream
 78        """
 79        stream_descriptor = HashableStreamDescriptor(name=stream_name, namespace=namespace)
 80        self.per_stream_states[stream_descriptor] = AirbyteStateBlob(value)
 81
 82    def create_state_message(self, stream_name: str, namespace: Optional[str]) -> AirbyteMessage:
 83        """
 84        Generates an AirbyteMessage using the current per-stream state of a specified stream
 85        :param stream_name: The name of the stream for the message that is being created
 86        :param namespace: The namespace of the stream for the message that is being created
 87        :return: The Airbyte state message to be emitted by the connector during a sync
 88        """
 89        hashable_descriptor = HashableStreamDescriptor(name=stream_name, namespace=namespace)
 90        stream_state = self.per_stream_states.get(hashable_descriptor) or AirbyteStateBlob()
 91
 92        return AirbyteMessage(
 93            type=MessageType.STATE,
 94            state=AirbyteStateMessage(
 95                type=AirbyteStateType.STREAM,
 96                stream=AirbyteStreamState(
 97                    stream_descriptor=StreamDescriptor(name=stream_name, namespace=namespace),
 98                    stream_state=stream_state,
 99                ),
100            ),
101        )
102
103    @classmethod
104    def _extract_from_state_message(
105        cls,
106        state: Optional[List[AirbyteStateMessage]],
107    ) -> Tuple[
108        Optional[AirbyteStateBlob],
109        MutableMapping[HashableStreamDescriptor, Optional[AirbyteStateBlob]],
110    ]:
111        """
112        Takes an incoming list of state messages or a global state message and extracts state attributes according to
113        type which can then be assigned to the new state manager being instantiated
114        :param state: The incoming state input
115        :return: A tuple of shared state and per stream state assembled from the incoming state list
116        """
117        if state is None:
118            return None, {}
119
120        is_global = cls._is_global_state(state)
121
122        if is_global:
123            # We already validate that this is a global state message, not None:
124            global_state = cast(AirbyteGlobalState, state[0].global_)
125            # global_state has shared_state, also not None:
126            shared_state: AirbyteStateBlob = cast(
127                AirbyteStateBlob, copy.deepcopy(global_state.shared_state, {})
128            )
129            streams = {
130                HashableStreamDescriptor(
131                    name=per_stream_state.stream_descriptor.name,
132                    namespace=per_stream_state.stream_descriptor.namespace,
133                ): per_stream_state.stream_state
134                for per_stream_state in global_state.stream_states  # type: ignore[union-attr] # global_state has shared_state
135            }
136            return shared_state, streams
137        else:
138            streams = {
139                HashableStreamDescriptor(
140                    name=per_stream_state.stream.stream_descriptor.name,  # type: ignore[union-attr] # stream has stream_descriptor
141                    namespace=per_stream_state.stream.stream_descriptor.namespace,  # type: ignore[union-attr] # stream has stream_descriptor
142                ): per_stream_state.stream.stream_state  # type: ignore[union-attr] # stream has stream_state
143                for per_stream_state in state
144                if per_stream_state.type == AirbyteStateType.STREAM
145                and hasattr(per_stream_state, "stream")  # type: ignore # state is always a list of AirbyteStateMessage if is_per_stream is True
146            }
147            return None, streams
148
149    @staticmethod
150    def _is_global_state(state: Union[List[AirbyteStateMessage], MutableMapping[str, Any]]) -> bool:
151        return (
152            isinstance(state, List)
153            and len(state) == 1
154            and isinstance(state[0], AirbyteStateMessage)
155            and state[0].type == AirbyteStateType.GLOBAL
156        )
157
158    @staticmethod
159    def _is_per_stream_state(
160        state: Union[List[AirbyteStateMessage], MutableMapping[str, Any]],
161    ) -> bool:
162        return isinstance(state, List)

ConnectorStateManager consolidates the various forms of a stream's incoming state message (STREAM / GLOBAL) under a common interface. It also provides methods to extract and update state

ConnectorStateManager( state: Optional[List[airbyte_cdk.models.airbyte_protocol.AirbyteStateMessage]] = None)
39    def __init__(self, state: Optional[List[AirbyteStateMessage]] = None):
40        shared_state, per_stream_states = self._extract_from_state_message(state)
41
42        # We explicitly throw an error if we receive a GLOBAL state message that contains a shared_state because API sources are
43        # designed to checkpoint state independently of one another. API sources should never be emitting a state message where
44        # shared_state is populated. Rather than define how to handle shared_state without a clear use case, we're opting to throw an
45        # error instead and if/when we find one, we will then implement processing of the shared_state value.
46        if shared_state:
47            raise ValueError(
48                "Received a GLOBAL AirbyteStateMessage that contains a shared_state. This library only ever generates per-STREAM "
49                "STATE messages so this was not generated by this connector. This must be an orchestrator or platform error. GLOBAL "
50                "state messages with shared_state will not be processed correctly. "
51            )
52        self.per_stream_states = per_stream_states
per_stream_states
def get_stream_state( self, stream_name: str, namespace: Optional[str]) -> MutableMapping[str, Any]:
54    def get_stream_state(
55        self, stream_name: str, namespace: Optional[str]
56    ) -> MutableMapping[str, Any]:
57        """
58        Retrieves the state of a given stream based on its descriptor (name + namespace).
59        :param stream_name: Name of the stream being fetched
60        :param namespace: Namespace of the stream being fetched
61        :return: The per-stream state for a stream
62        """
63        stream_state: AirbyteStateBlob | None = self.per_stream_states.get(
64            HashableStreamDescriptor(name=stream_name, namespace=namespace)
65        )
66        if stream_state:
67            return copy.deepcopy({k: v for k, v in stream_state.__dict__.items()})
68        return {}

Retrieves the state of a given stream based on its descriptor (name + namespace).

Parameters
  • stream_name: Name of the stream being fetched
  • namespace: Namespace of the stream being fetched
Returns

The per-stream state for a stream

def update_state_for_stream( self, stream_name: str, namespace: Optional[str], value: Mapping[str, Any]) -> None:
70    def update_state_for_stream(
71        self, stream_name: str, namespace: Optional[str], value: Mapping[str, Any]
72    ) -> None:
73        """
74        Overwrites the state blob of a specific stream based on the provided stream name and optional namespace
75        :param stream_name: The name of the stream whose state is being updated
76        :param namespace: The namespace of the stream if it exists
77        :param value: A stream state mapping that is being updated for a stream
78        """
79        stream_descriptor = HashableStreamDescriptor(name=stream_name, namespace=namespace)
80        self.per_stream_states[stream_descriptor] = AirbyteStateBlob(value)

Overwrites the state blob of a specific stream based on the provided stream name and optional namespace

Parameters
  • stream_name: The name of the stream whose state is being updated
  • namespace: The namespace of the stream if it exists
  • value: A stream state mapping that is being updated for a stream
def create_state_message( self, stream_name: str, namespace: Optional[str]) -> AirbyteMessage:
 82    def create_state_message(self, stream_name: str, namespace: Optional[str]) -> AirbyteMessage:
 83        """
 84        Generates an AirbyteMessage using the current per-stream state of a specified stream
 85        :param stream_name: The name of the stream for the message that is being created
 86        :param namespace: The namespace of the stream for the message that is being created
 87        :return: The Airbyte state message to be emitted by the connector during a sync
 88        """
 89        hashable_descriptor = HashableStreamDescriptor(name=stream_name, namespace=namespace)
 90        stream_state = self.per_stream_states.get(hashable_descriptor) or AirbyteStateBlob()
 91
 92        return AirbyteMessage(
 93            type=MessageType.STATE,
 94            state=AirbyteStateMessage(
 95                type=AirbyteStateType.STREAM,
 96                stream=AirbyteStreamState(
 97                    stream_descriptor=StreamDescriptor(name=stream_name, namespace=namespace),
 98                    stream_state=stream_state,
 99                ),
100            ),
101        )

Generates an AirbyteMessage using the current per-stream state of a specified stream

Parameters
  • stream_name: The name of the stream for the message that is being created
  • namespace: The namespace of the stream for the message that is being created
Returns

The Airbyte state message to be emitted by the connector during a sync

@deprecated('Deprecated as of CDK version 0.87.0. Deprecated in favor of the `CheckpointMixin` which offers similar functionality.')
class IncrementalMixin(airbyte_cdk.sources.streams.core.CheckpointMixin, abc.ABC):
 95@deprecated(
 96    "Deprecated as of CDK version 0.87.0. "
 97    "Deprecated in favor of the `CheckpointMixin` which offers similar functionality."
 98)
 99class IncrementalMixin(CheckpointMixin, ABC):
100    """Mixin to make stream incremental.
101
102    class IncrementalStream(Stream, IncrementalMixin):
103        @property
104        def state(self):
105            return self._state
106
107        @state.setter
108        def state(self, value):
109            self._state[self.cursor_field] = value[self.cursor_field]
110    """

Mixin to make stream incremental.

class IncrementalStream(Stream, IncrementalMixin): @property def state(self): return self._state

@state.setter
def state(self, value):
    self._state[self.cursor_field] = value[self.cursor_field]
class Stream(abc.ABC):
119class Stream(ABC):
120    """
121    Base abstract class for an Airbyte Stream. Makes no assumption of the Stream's underlying transport protocol.
122    """
123
124    _configured_json_schema: Optional[Dict[str, Any]] = None
125    _exit_on_rate_limit: bool = False
126
127    # Use self.logger in subclasses to log any messages
128    @property
129    def logger(self) -> logging.Logger:
130        return logging.getLogger(f"airbyte.streams.{self.name}")
131
132    # TypeTransformer object to perform output data transformation
133    transformer: TypeTransformer = TypeTransformer(TransformConfig.NoTransform)
134
135    cursor: Optional[Cursor] = None
136
137    has_multiple_slices = False
138
139    @cached_property
140    def name(self) -> str:
141        """
142        :return: Stream name. By default this is the implementing class name, but it can be overridden as needed.
143        """
144        return casing.camel_to_snake(self.__class__.__name__)
145
146    def get_error_display_message(self, exception: BaseException) -> Optional[str]:
147        """
148        Retrieves the user-friendly display message that corresponds to an exception.
149        This will be called when encountering an exception while reading records from the stream, and used to build the AirbyteTraceMessage.
150
151        The default implementation of this method does not return user-friendly messages for any exception type, but it should be overriden as needed.
152
153        :param exception: The exception that was raised
154        :return: A user-friendly message that indicates the cause of the error
155        """
156        return None
157
158    def read(  # type: ignore  # ignoring typing for ConnectorStateManager because of circular dependencies
159        self,
160        configured_stream: ConfiguredAirbyteStream,
161        logger: logging.Logger,
162        slice_logger: SliceLogger,
163        stream_state: MutableMapping[str, Any],
164        state_manager,
165        internal_config: InternalConfig,
166    ) -> Iterable[StreamData]:
167        sync_mode = configured_stream.sync_mode
168        cursor_field = configured_stream.cursor_field
169        self.configured_json_schema = configured_stream.stream.json_schema
170
171        # WARNING: When performing a read() that uses incoming stream state, we MUST use the self.state that is defined as
172        # opposed to the incoming stream_state value. Because some connectors like ones using the file-based CDK modify
173        # state before setting the value on the Stream attribute, the most up-to-date state is derived from Stream.state
174        # instead of the stream_state parameter. This does not apply to legacy connectors using get_updated_state().
175        try:
176            stream_state = self.state  # type: ignore # we know the field might not exist...
177        except AttributeError:
178            pass
179
180        should_checkpoint = bool(state_manager)
181        checkpoint_reader = self._get_checkpoint_reader(
182            logger=logger, cursor_field=cursor_field, sync_mode=sync_mode, stream_state=stream_state
183        )
184
185        next_slice = checkpoint_reader.next()
186        record_counter = 0
187        stream_state_tracker = copy.deepcopy(stream_state)
188        while next_slice is not None:
189            if slice_logger.should_log_slice_message(logger):
190                yield slice_logger.create_slice_log_message(next_slice)
191            records = self.read_records(
192                sync_mode=sync_mode,  # todo: change this interface to no longer rely on sync_mode for behavior
193                stream_slice=next_slice,
194                stream_state=stream_state,
195                cursor_field=cursor_field or None,
196            )
197            for record_data_or_message in records:
198                yield record_data_or_message
199                if isinstance(record_data_or_message, Mapping) or (
200                    hasattr(record_data_or_message, "type")
201                    and record_data_or_message.type == MessageType.RECORD
202                ):
203                    record_data = (
204                        record_data_or_message
205                        if isinstance(record_data_or_message, Mapping)
206                        else record_data_or_message.record
207                    )
208
209                    # Thanks I hate it. RFR fundamentally doesn't fit with the concept of the legacy Stream.get_updated_state()
210                    # method because RFR streams rely on pagination as a cursor. Stream.get_updated_state() was designed to make
211                    # the CDK manage state using specifically the last seen record. don't @ brian.lai
212                    #
213                    # Also, because the legacy incremental state case decouples observing incoming records from emitting state, it
214                    # requires that we separate CheckpointReader.observe() and CheckpointReader.get_checkpoint() which could
215                    # otherwise be combined.
216                    if self.cursor_field:
217                        # Some connectors have streams that implement get_updated_state(), but do not define a cursor_field. This
218                        # should be fixed on the stream implementation, but we should also protect against this in the CDK as well
219                        stream_state_tracker = self.get_updated_state(
220                            stream_state_tracker,
221                            record_data,  # type: ignore [arg-type]
222                        )
223                        self._observe_state(checkpoint_reader, stream_state_tracker)
224                    record_counter += 1
225
226                    checkpoint_interval = self.state_checkpoint_interval
227                    if (
228                        should_checkpoint
229                        and checkpoint_interval
230                        and record_counter % checkpoint_interval == 0
231                    ):
232                        checkpoint = checkpoint_reader.get_checkpoint()
233                        if checkpoint:
234                            airbyte_state_message = self._checkpoint_state(
235                                checkpoint, state_manager=state_manager
236                            )
237                            yield airbyte_state_message
238
239                    if internal_config.is_limit_reached(record_counter):
240                        break
241            self._observe_state(checkpoint_reader)
242            checkpoint_state = checkpoint_reader.get_checkpoint()
243            if should_checkpoint and checkpoint_state is not None:
244                airbyte_state_message = self._checkpoint_state(
245                    checkpoint_state, state_manager=state_manager
246                )
247                yield airbyte_state_message
248
249            next_slice = checkpoint_reader.next()
250
251        checkpoint = checkpoint_reader.get_checkpoint()
252        if should_checkpoint and checkpoint is not None:
253            airbyte_state_message = self._checkpoint_state(checkpoint, state_manager=state_manager)
254            yield airbyte_state_message
255
256    def read_only_records(self, state: Optional[Mapping[str, Any]] = None) -> Iterable[StreamData]:
257        """
258        Helper method that performs a read on a stream with an optional state and emits records. If the parent stream supports
259        incremental, this operation does not update the stream's internal state (if it uses the modern state setter/getter)
260        or emit state messages.
261        """
262
263        configured_stream = ConfiguredAirbyteStream(
264            stream=AirbyteStream(
265                name=self.name,
266                json_schema={},
267                supported_sync_modes=[SyncMode.full_refresh, SyncMode.incremental],
268            ),
269            sync_mode=SyncMode.incremental if state else SyncMode.full_refresh,
270            destination_sync_mode=DestinationSyncMode.append,
271        )
272
273        yield from self.read(
274            configured_stream=configured_stream,
275            logger=self.logger,
276            slice_logger=DebugSliceLogger(),
277            stream_state=dict(state)
278            if state
279            else {},  # read() expects MutableMapping instead of Mapping which is used more often
280            state_manager=None,
281            internal_config=InternalConfig(),  # type: ignore [call-arg]
282        )
283
284    @abstractmethod
285    def read_records(
286        self,
287        sync_mode: SyncMode,
288        cursor_field: Optional[List[str]] = None,
289        stream_slice: Optional[Mapping[str, Any]] = None,
290        stream_state: Optional[Mapping[str, Any]] = None,
291    ) -> Iterable[StreamData]:
292        """
293        This method should be overridden by subclasses to read records based on the inputs
294        """
295
296    @lru_cache(maxsize=None)
297    def get_json_schema(self) -> Mapping[str, Any]:
298        """
299        :return: A dict of the JSON schema representing this stream.
300
301        The default implementation of this method looks for a JSONSchema file with the same name as this stream's "name" property.
302        Override as needed.
303        """
304        # TODO show an example of using pydantic to define the JSON schema, or reading an OpenAPI spec
305        return ResourceSchemaLoader(package_name_from_class(self.__class__)).get_schema(self.name)
306
307    def as_airbyte_stream(self) -> AirbyteStream:
308        stream = AirbyteStream(
309            name=self.name,
310            json_schema=dict(self.get_json_schema()),
311            supported_sync_modes=[SyncMode.full_refresh],
312            is_resumable=self.is_resumable,
313        )
314
315        if self.namespace:
316            stream.namespace = self.namespace
317
318        # If we can offer incremental we always should. RFR is always less reliable than incremental which uses a real cursor value
319        if self.supports_incremental:
320            stream.source_defined_cursor = self.source_defined_cursor
321            stream.supported_sync_modes.append(SyncMode.incremental)
322            stream.default_cursor_field = self._wrapped_cursor_field()
323
324        keys = Stream._wrapped_primary_key(self.primary_key)
325        if keys and len(keys) > 0:
326            stream.source_defined_primary_key = keys
327
328        return stream
329
330    @property
331    def supports_incremental(self) -> bool:
332        """
333        :return: True if this stream supports incrementally reading data
334        """
335        return len(self._wrapped_cursor_field()) > 0
336
337    @property
338    def is_resumable(self) -> bool:
339        """
340        :return: True if this stream allows the checkpointing of sync progress and can resume from it on subsequent attempts.
341        This differs from supports_incremental because certain kinds of streams like those supporting resumable full refresh
342        can checkpoint progress in between attempts for improved fault tolerance. However, they will start from the beginning
343        on the next sync job.
344        """
345        if self.supports_incremental:
346            return True
347        if self.has_multiple_slices:
348            # We temporarily gate substream to not support RFR because puts a pretty high burden on connector developers
349            # to structure stream state in a very specific way. We also can't check for issubclass(HttpSubStream) because
350            # not all substreams implement the interface and it would be a circular dependency so we use parent as a surrogate
351            return False
352        elif hasattr(type(self), "state") and getattr(type(self), "state").fset is not None:
353            # Modern case where a stream manages state using getter/setter
354            return True
355        else:
356            # Legacy case where the CDK manages state via the get_updated_state() method. This is determined by checking if
357            # the stream's get_updated_state() differs from the Stream class and therefore has been overridden
358            return type(self).get_updated_state != Stream.get_updated_state
359
360    def _wrapped_cursor_field(self) -> List[str]:
361        return [self.cursor_field] if isinstance(self.cursor_field, str) else self.cursor_field
362
363    @property
364    def cursor_field(self) -> Union[str, List[str]]:
365        """
366        Override to return the default cursor field used by this stream e.g: an API entity might always use created_at as the cursor field.
367        :return: The name of the field used as a cursor. If the cursor is nested, return an array consisting of the path to the cursor.
368        """
369        return []
370
371    @property
372    def namespace(self) -> Optional[str]:
373        """
374        Override to return the namespace of this stream, e.g. the Postgres schema which this stream will emit records for.
375        :return: A string containing the name of the namespace.
376        """
377        return None
378
379    @property
380    def source_defined_cursor(self) -> bool:
381        """
382        Return False if the cursor can be configured by the user.
383        """
384        return True
385
386    @property
387    def exit_on_rate_limit(self) -> bool:
388        """Exit on rate limit getter, should return bool value. False if the stream will retry endlessly when rate limited."""
389        return self._exit_on_rate_limit
390
391    @exit_on_rate_limit.setter
392    def exit_on_rate_limit(self, value: bool) -> None:
393        """Exit on rate limit setter, accept bool value."""
394        self._exit_on_rate_limit = value
395
396    @property
397    @abstractmethod
398    def primary_key(self) -> Optional[Union[str, List[str], List[List[str]]]]:
399        """
400        :return: string if single primary key, list of strings if composite primary key, list of list of strings if composite primary key consisting of nested fields.
401          If the stream has no primary keys, return None.
402        """
403
404    def stream_slices(
405        self,
406        *,
407        sync_mode: SyncMode,
408        cursor_field: Optional[List[str]] = None,
409        stream_state: Optional[Mapping[str, Any]] = None,
410    ) -> Iterable[Optional[Mapping[str, Any]]]:
411        """
412        Override to define the slices for this stream. See the stream slicing section of the docs for more information.
413
414        :param sync_mode:
415        :param cursor_field:
416        :param stream_state:
417        :return:
418        """
419        yield StreamSlice(partition={}, cursor_slice={})
420
421    @property
422    def state_checkpoint_interval(self) -> Optional[int]:
423        """
424        Decides how often to checkpoint state (i.e: emit a STATE message). E.g: if this returns a value of 100, then state is persisted after reading
425        100 records, then 200, 300, etc.. A good default value is 1000 although your mileage may vary depending on the underlying data source.
426
427        Checkpointing a stream avoids re-reading records in the case a sync is failed or cancelled.
428
429        return None if state should not be checkpointed e.g: because records returned from the underlying data source are not returned in
430        ascending order with respect to the cursor field. This can happen if the source does not support reading records in ascending order of
431        created_at date (or whatever the cursor is). In those cases, state must only be saved once the full stream has been read.
432        """
433        return None
434
435    # Commented-out to avoid any runtime penalty, since this is used in a hot per-record codepath.
436    # To be evaluated for re-introduction here: https://github.com/airbytehq/airbyte-python-cdk/issues/116
437    # @deprecated(
438    #     "Deprecated method `get_updated_state` as of CDK version 0.1.49. "
439    #     "Please use explicit state property instead, see `IncrementalMixin` docs."
440    # )
441    def get_updated_state(
442        self, current_stream_state: MutableMapping[str, Any], latest_record: Mapping[str, Any]
443    ) -> MutableMapping[str, Any]:
444        """DEPRECATED. Please use explicit state property instead, see `IncrementalMixin` docs.
445
446        Override to extract state from the latest record. Needed to implement incremental sync.
447
448        Inspects the latest record extracted from the data source and the current state object and return an updated state object.
449
450        For example: if the state object is based on created_at timestamp, and the current state is {'created_at': 10}, and the latest_record is
451        {'name': 'octavia', 'created_at': 20 } then this method would return {'created_at': 20} to indicate state should be updated to this object.
452
453        :param current_stream_state: The stream's current state object
454        :param latest_record: The latest record extracted from the stream
455        :return: An updated state object
456        """
457        return {}
458
459    def get_cursor(self) -> Optional[Cursor]:
460        """
461        A Cursor is an interface that a stream can implement to manage how its internal state is read and updated while
462        reading records. Historically, Python connectors had no concept of a cursor to manage state. Python streams need
463        to define a cursor implementation and override this method to manage state through a Cursor.
464        """
465        return self.cursor
466
467    def _get_checkpoint_reader(
468        self,
469        logger: logging.Logger,
470        cursor_field: Optional[List[str]],
471        sync_mode: SyncMode,
472        stream_state: MutableMapping[str, Any],
473    ) -> CheckpointReader:
474        mappings_or_slices = self.stream_slices(
475            cursor_field=cursor_field,
476            sync_mode=sync_mode,  # todo: change this interface to no longer rely on sync_mode for behavior
477            stream_state=stream_state,
478        )
479
480        # Because of poor foresight, we wrote the default Stream.stream_slices() method to return [None] which is confusing and
481        # has now normalized this behavior for connector developers. Now some connectors return [None]. This is objectively
482        # misleading and a more ideal interface is [{}] to indicate we still want to iterate over one slice, but with no
483        # specific slice values. None is bad, and now I feel bad that I have to write this hack.
484        if mappings_or_slices == [None]:
485            mappings_or_slices = [{}]
486
487        slices_iterable_copy, iterable_for_detecting_format = itertools.tee(mappings_or_slices, 2)
488        stream_classification = self._classify_stream(
489            mappings_or_slices=iterable_for_detecting_format
490        )
491
492        # Streams that override has_multiple_slices are explicitly indicating that they will iterate over
493        # multiple partitions. Inspecting slices to automatically apply the correct cursor is only needed as
494        # a backup. So if this value was already assigned to True by the stream, we don't need to reassign it
495        self.has_multiple_slices = (
496            self.has_multiple_slices or stream_classification.has_multiple_slices
497        )
498
499        cursor = self.get_cursor()
500        if cursor:
501            cursor.set_initial_state(stream_state=stream_state)
502
503        checkpoint_mode = self._checkpoint_mode
504
505        if cursor and stream_classification.is_legacy_format:
506            return LegacyCursorBasedCheckpointReader(
507                stream_slices=slices_iterable_copy, cursor=cursor, read_state_from_cursor=True
508            )
509        elif cursor:
510            return CursorBasedCheckpointReader(
511                stream_slices=slices_iterable_copy,
512                cursor=cursor,
513                read_state_from_cursor=checkpoint_mode == CheckpointMode.RESUMABLE_FULL_REFRESH,
514            )
515        elif checkpoint_mode == CheckpointMode.RESUMABLE_FULL_REFRESH:
516            # Resumable full refresh readers rely on the stream state dynamically being updated during pagination and does
517            # not iterate over a static set of slices.
518            return ResumableFullRefreshCheckpointReader(stream_state=stream_state)
519        elif checkpoint_mode == CheckpointMode.INCREMENTAL:
520            return IncrementalCheckpointReader(
521                stream_slices=slices_iterable_copy, stream_state=stream_state
522            )
523        else:
524            return FullRefreshCheckpointReader(stream_slices=slices_iterable_copy)
525
526    @property
527    def _checkpoint_mode(self) -> CheckpointMode:
528        if self.is_resumable and len(self._wrapped_cursor_field()) > 0:
529            return CheckpointMode.INCREMENTAL
530        elif self.is_resumable:
531            return CheckpointMode.RESUMABLE_FULL_REFRESH
532        else:
533            return CheckpointMode.FULL_REFRESH
534
535    @staticmethod
536    def _classify_stream(
537        mappings_or_slices: Iterator[Optional[Union[Mapping[str, Any], StreamSlice]]],
538    ) -> StreamClassification:
539        """
540        This is a bit of a crazy solution, but also the only way we can detect certain attributes about the stream since Python
541        streams do not follow consistent implementation patterns. We care about the following two attributes:
542        - is_substream: Helps to incrementally release changes since substreams w/ parents are much more complicated. Also
543          helps de-risk the release of changes that might impact all connectors
544        - uses_legacy_slice_format: Since the checkpoint reader must manage a complex state object, we opted to have it always
545          use the structured StreamSlice object. However, this requires backwards compatibility with Python sources that only
546          support the legacy mapping object
547
548        Both attributes can eventually be deprecated once stream's define this method deleted once substreams have been implemented and
549        legacy connectors all adhere to the StreamSlice object.
550        """
551        if not mappings_or_slices:
552            raise ValueError("A stream should always have at least one slice")
553        try:
554            next_slice = next(mappings_or_slices)
555            if isinstance(next_slice, StreamSlice) and next_slice == StreamSlice(
556                partition={}, cursor_slice={}
557            ):
558                is_legacy_format = False
559                slice_has_value = False
560            elif next_slice == {}:
561                is_legacy_format = True
562                slice_has_value = False
563            elif isinstance(next_slice, StreamSlice):
564                is_legacy_format = False
565                slice_has_value = True
566            else:
567                is_legacy_format = True
568                slice_has_value = True
569        except StopIteration:
570            # If the stream has no slices, the format ultimately does not matter since no data will get synced. This is technically
571            # a valid case because it is up to the stream to define its slicing behavior
572            return StreamClassification(is_legacy_format=False, has_multiple_slices=False)
573
574        if slice_has_value:
575            # If the first slice contained a partition value from the result of stream_slices(), this is a substream that might
576            # have multiple parent records to iterate over
577            return StreamClassification(
578                is_legacy_format=is_legacy_format, has_multiple_slices=slice_has_value
579            )
580
581        try:
582            # If stream_slices() returns multiple slices, this is also a substream that can potentially generate empty slices
583            next(mappings_or_slices)
584            return StreamClassification(is_legacy_format=is_legacy_format, has_multiple_slices=True)
585        except StopIteration:
586            # If the result of stream_slices() only returns a single empty stream slice, then we know this is a regular stream
587            return StreamClassification(
588                is_legacy_format=is_legacy_format, has_multiple_slices=False
589            )
590
591    def log_stream_sync_configuration(self) -> None:
592        """
593        Logs the configuration of this stream.
594        """
595        self.logger.debug(
596            f"Syncing stream instance: {self.name}",
597            extra={
598                "primary_key": self.primary_key,
599                "cursor_field": self.cursor_field,
600            },
601        )
602
603    @staticmethod
604    def _wrapped_primary_key(
605        keys: Optional[Union[str, List[str], List[List[str]]]],
606    ) -> Optional[List[List[str]]]:
607        """
608        :return: wrap the primary_key property in a list of list of strings required by the Airbyte Stream object.
609        """
610        if not keys:
611            return None
612
613        if isinstance(keys, str):
614            return [[keys]]
615        elif isinstance(keys, list):
616            wrapped_keys = []
617            for component in keys:
618                if isinstance(component, str):
619                    wrapped_keys.append([component])
620                elif isinstance(component, list):
621                    wrapped_keys.append(component)
622                else:
623                    raise ValueError(f"Element must be either list or str. Got: {type(component)}")
624            return wrapped_keys
625        else:
626            raise ValueError(f"Element must be either list or str. Got: {type(keys)}")
627
628    def _observe_state(
629        self, checkpoint_reader: CheckpointReader, stream_state: Optional[Mapping[str, Any]] = None
630    ) -> None:
631        """
632        Convenience method that attempts to read the Stream's state using the recommended way of connector's managing their
633        own state via state setter/getter. But if we get back an AttributeError, then the legacy Stream.get_updated_state()
634        method is used as a fallback method.
635        """
636
637        # This is an inversion of the original logic that used to try state getter/setters first. As part of the work to
638        # automatically apply resumable full refresh to all streams, all HttpStream classes implement default state
639        # getter/setter methods, we should default to only using the incoming stream_state parameter value is {} which
640        # indicates the stream does not override the default get_updated_state() implementation. When the default method
641        # is not overridden, then the stream defers to self.state getter
642        if stream_state:
643            checkpoint_reader.observe(stream_state)
644        elif type(self).get_updated_state == Stream.get_updated_state:
645            # We only default to the state getter/setter if the stream does not use the legacy get_updated_state() method
646            try:
647                new_state = self.state  # type: ignore # This will always exist on HttpStreams, but may not for Stream
648                if new_state:
649                    checkpoint_reader.observe(new_state)
650            except AttributeError:
651                pass
652
653    def _checkpoint_state(  # type: ignore  # ignoring typing for ConnectorStateManager because of circular dependencies
654        self,
655        stream_state: Mapping[str, Any],
656        state_manager,
657    ) -> AirbyteMessage:
658        # todo: This can be consolidated into one ConnectorStateManager.update_and_create_state_message() method, but I want
659        #  to reduce changes right now and this would span concurrent as well
660        state_manager.update_state_for_stream(self.name, self.namespace, stream_state)
661        return state_manager.create_state_message(self.name, self.namespace)  # type: ignore [no-any-return]
662
663    @property
664    def configured_json_schema(self) -> Optional[Dict[str, Any]]:
665        """
666        This property is set from the read method.
667
668        :return Optional[Dict]: JSON schema from configured catalog if provided, otherwise None.
669        """
670        return self._configured_json_schema
671
672    @configured_json_schema.setter
673    def configured_json_schema(self, json_schema: Dict[str, Any]) -> None:
674        self._configured_json_schema = self._filter_schema_invalid_properties(json_schema)
675
676    def _filter_schema_invalid_properties(
677        self, configured_catalog_json_schema: Dict[str, Any]
678    ) -> Dict[str, Any]:
679        """
680        Filters the properties in json_schema that are not present in the stream schema.
681        Configured Schemas can have very old fields, so we need to housekeeping ourselves.
682        """
683        configured_schema: Any = configured_catalog_json_schema.get("properties", {})
684        stream_schema_properties: Any = self.get_json_schema().get("properties", {})
685
686        configured_keys = configured_schema.keys()
687        stream_keys = stream_schema_properties.keys()
688        invalid_properties = configured_keys - stream_keys
689        if not invalid_properties:
690            return configured_catalog_json_schema
691
692        self.logger.warning(
693            f"Stream {self.name}: the following fields are deprecated and cannot be synced. {invalid_properties}. Refresh the connection's source schema to resolve this warning."
694        )
695
696        valid_configured_schema_properties_keys = stream_keys & configured_keys
697        valid_configured_schema_properties = {}
698
699        for configured_schema_property in valid_configured_schema_properties_keys:
700            valid_configured_schema_properties[configured_schema_property] = (
701                stream_schema_properties[configured_schema_property]
702            )
703
704        return {**configured_catalog_json_schema, "properties": valid_configured_schema_properties}

Base abstract class for an Airbyte Stream. Makes no assumption of the Stream's underlying transport protocol.

logger: logging.Logger
128    @property
129    def logger(self) -> logging.Logger:
130        return logging.getLogger(f"airbyte.streams.{self.name}")
transformer: TypeTransformer = <TypeTransformer object>
has_multiple_slices = False
name: str
139    @cached_property
140    def name(self) -> str:
141        """
142        :return: Stream name. By default this is the implementing class name, but it can be overridden as needed.
143        """
144        return casing.camel_to_snake(self.__class__.__name__)
Returns

Stream name. By default this is the implementing class name, but it can be overridden as needed.

def get_error_display_message(self, exception: BaseException) -> Optional[str]:
146    def get_error_display_message(self, exception: BaseException) -> Optional[str]:
147        """
148        Retrieves the user-friendly display message that corresponds to an exception.
149        This will be called when encountering an exception while reading records from the stream, and used to build the AirbyteTraceMessage.
150
151        The default implementation of this method does not return user-friendly messages for any exception type, but it should be overriden as needed.
152
153        :param exception: The exception that was raised
154        :return: A user-friendly message that indicates the cause of the error
155        """
156        return None

Retrieves the user-friendly display message that corresponds to an exception. This will be called when encountering an exception while reading records from the stream, and used to build the AirbyteTraceMessage.

The default implementation of this method does not return user-friendly messages for any exception type, but it should be overriden as needed.

Parameters
  • exception: The exception that was raised
Returns

A user-friendly message that indicates the cause of the error

def read( self, configured_stream: airbyte_protocol_dataclasses.models.airbyte_protocol.ConfiguredAirbyteStream, logger: logging.Logger, slice_logger: airbyte_cdk.sources.utils.slice_logger.SliceLogger, stream_state: MutableMapping[str, Any], state_manager, internal_config: InternalConfig) -> Iterable[Union[Mapping[str, Any], AirbyteMessage]]:
158    def read(  # type: ignore  # ignoring typing for ConnectorStateManager because of circular dependencies
159        self,
160        configured_stream: ConfiguredAirbyteStream,
161        logger: logging.Logger,
162        slice_logger: SliceLogger,
163        stream_state: MutableMapping[str, Any],
164        state_manager,
165        internal_config: InternalConfig,
166    ) -> Iterable[StreamData]:
167        sync_mode = configured_stream.sync_mode
168        cursor_field = configured_stream.cursor_field
169        self.configured_json_schema = configured_stream.stream.json_schema
170
171        # WARNING: When performing a read() that uses incoming stream state, we MUST use the self.state that is defined as
172        # opposed to the incoming stream_state value. Because some connectors like ones using the file-based CDK modify
173        # state before setting the value on the Stream attribute, the most up-to-date state is derived from Stream.state
174        # instead of the stream_state parameter. This does not apply to legacy connectors using get_updated_state().
175        try:
176            stream_state = self.state  # type: ignore # we know the field might not exist...
177        except AttributeError:
178            pass
179
180        should_checkpoint = bool(state_manager)
181        checkpoint_reader = self._get_checkpoint_reader(
182            logger=logger, cursor_field=cursor_field, sync_mode=sync_mode, stream_state=stream_state
183        )
184
185        next_slice = checkpoint_reader.next()
186        record_counter = 0
187        stream_state_tracker = copy.deepcopy(stream_state)
188        while next_slice is not None:
189            if slice_logger.should_log_slice_message(logger):
190                yield slice_logger.create_slice_log_message(next_slice)
191            records = self.read_records(
192                sync_mode=sync_mode,  # todo: change this interface to no longer rely on sync_mode for behavior
193                stream_slice=next_slice,
194                stream_state=stream_state,
195                cursor_field=cursor_field or None,
196            )
197            for record_data_or_message in records:
198                yield record_data_or_message
199                if isinstance(record_data_or_message, Mapping) or (
200                    hasattr(record_data_or_message, "type")
201                    and record_data_or_message.type == MessageType.RECORD
202                ):
203                    record_data = (
204                        record_data_or_message
205                        if isinstance(record_data_or_message, Mapping)
206                        else record_data_or_message.record
207                    )
208
209                    # Thanks I hate it. RFR fundamentally doesn't fit with the concept of the legacy Stream.get_updated_state()
210                    # method because RFR streams rely on pagination as a cursor. Stream.get_updated_state() was designed to make
211                    # the CDK manage state using specifically the last seen record. don't @ brian.lai
212                    #
213                    # Also, because the legacy incremental state case decouples observing incoming records from emitting state, it
214                    # requires that we separate CheckpointReader.observe() and CheckpointReader.get_checkpoint() which could
215                    # otherwise be combined.
216                    if self.cursor_field:
217                        # Some connectors have streams that implement get_updated_state(), but do not define a cursor_field. This
218                        # should be fixed on the stream implementation, but we should also protect against this in the CDK as well
219                        stream_state_tracker = self.get_updated_state(
220                            stream_state_tracker,
221                            record_data,  # type: ignore [arg-type]
222                        )
223                        self._observe_state(checkpoint_reader, stream_state_tracker)
224                    record_counter += 1
225
226                    checkpoint_interval = self.state_checkpoint_interval
227                    if (
228                        should_checkpoint
229                        and checkpoint_interval
230                        and record_counter % checkpoint_interval == 0
231                    ):
232                        checkpoint = checkpoint_reader.get_checkpoint()
233                        if checkpoint:
234                            airbyte_state_message = self._checkpoint_state(
235                                checkpoint, state_manager=state_manager
236                            )
237                            yield airbyte_state_message
238
239                    if internal_config.is_limit_reached(record_counter):
240                        break
241            self._observe_state(checkpoint_reader)
242            checkpoint_state = checkpoint_reader.get_checkpoint()
243            if should_checkpoint and checkpoint_state is not None:
244                airbyte_state_message = self._checkpoint_state(
245                    checkpoint_state, state_manager=state_manager
246                )
247                yield airbyte_state_message
248
249            next_slice = checkpoint_reader.next()
250
251        checkpoint = checkpoint_reader.get_checkpoint()
252        if should_checkpoint and checkpoint is not None:
253            airbyte_state_message = self._checkpoint_state(checkpoint, state_manager=state_manager)
254            yield airbyte_state_message
def read_only_records( self, state: Optional[Mapping[str, Any]] = None) -> Iterable[Union[Mapping[str, Any], AirbyteMessage]]:
256    def read_only_records(self, state: Optional[Mapping[str, Any]] = None) -> Iterable[StreamData]:
257        """
258        Helper method that performs a read on a stream with an optional state and emits records. If the parent stream supports
259        incremental, this operation does not update the stream's internal state (if it uses the modern state setter/getter)
260        or emit state messages.
261        """
262
263        configured_stream = ConfiguredAirbyteStream(
264            stream=AirbyteStream(
265                name=self.name,
266                json_schema={},
267                supported_sync_modes=[SyncMode.full_refresh, SyncMode.incremental],
268            ),
269            sync_mode=SyncMode.incremental if state else SyncMode.full_refresh,
270            destination_sync_mode=DestinationSyncMode.append,
271        )
272
273        yield from self.read(
274            configured_stream=configured_stream,
275            logger=self.logger,
276            slice_logger=DebugSliceLogger(),
277            stream_state=dict(state)
278            if state
279            else {},  # read() expects MutableMapping instead of Mapping which is used more often
280            state_manager=None,
281            internal_config=InternalConfig(),  # type: ignore [call-arg]
282        )

Helper method that performs a read on a stream with an optional state and emits records. If the parent stream supports incremental, this operation does not update the stream's internal state (if it uses the modern state setter/getter) or emit state messages.

@abstractmethod
def read_records( self, sync_mode: airbyte_protocol_dataclasses.models.airbyte_protocol.SyncMode, cursor_field: Optional[List[str]] = None, stream_slice: Optional[Mapping[str, Any]] = None, stream_state: Optional[Mapping[str, Any]] = None) -> Iterable[Union[Mapping[str, Any], AirbyteMessage]]:
284    @abstractmethod
285    def read_records(
286        self,
287        sync_mode: SyncMode,
288        cursor_field: Optional[List[str]] = None,
289        stream_slice: Optional[Mapping[str, Any]] = None,
290        stream_state: Optional[Mapping[str, Any]] = None,
291    ) -> Iterable[StreamData]:
292        """
293        This method should be overridden by subclasses to read records based on the inputs
294        """

This method should be overridden by subclasses to read records based on the inputs

@lru_cache(maxsize=None)
def get_json_schema(self) -> Mapping[str, Any]:
296    @lru_cache(maxsize=None)
297    def get_json_schema(self) -> Mapping[str, Any]:
298        """
299        :return: A dict of the JSON schema representing this stream.
300
301        The default implementation of this method looks for a JSONSchema file with the same name as this stream's "name" property.
302        Override as needed.
303        """
304        # TODO show an example of using pydantic to define the JSON schema, or reading an OpenAPI spec
305        return ResourceSchemaLoader(package_name_from_class(self.__class__)).get_schema(self.name)
Returns

A dict of the JSON schema representing this stream.

The default implementation of this method looks for a JSONSchema file with the same name as this stream's "name" property. Override as needed.

def as_airbyte_stream( self) -> airbyte_protocol_dataclasses.models.airbyte_protocol.AirbyteStream:
307    def as_airbyte_stream(self) -> AirbyteStream:
308        stream = AirbyteStream(
309            name=self.name,
310            json_schema=dict(self.get_json_schema()),
311            supported_sync_modes=[SyncMode.full_refresh],
312            is_resumable=self.is_resumable,
313        )
314
315        if self.namespace:
316            stream.namespace = self.namespace
317
318        # If we can offer incremental we always should. RFR is always less reliable than incremental which uses a real cursor value
319        if self.supports_incremental:
320            stream.source_defined_cursor = self.source_defined_cursor
321            stream.supported_sync_modes.append(SyncMode.incremental)
322            stream.default_cursor_field = self._wrapped_cursor_field()
323
324        keys = Stream._wrapped_primary_key(self.primary_key)
325        if keys and len(keys) > 0:
326            stream.source_defined_primary_key = keys
327
328        return stream
supports_incremental: bool
330    @property
331    def supports_incremental(self) -> bool:
332        """
333        :return: True if this stream supports incrementally reading data
334        """
335        return len(self._wrapped_cursor_field()) > 0
Returns

True if this stream supports incrementally reading data

is_resumable: bool
337    @property
338    def is_resumable(self) -> bool:
339        """
340        :return: True if this stream allows the checkpointing of sync progress and can resume from it on subsequent attempts.
341        This differs from supports_incremental because certain kinds of streams like those supporting resumable full refresh
342        can checkpoint progress in between attempts for improved fault tolerance. However, they will start from the beginning
343        on the next sync job.
344        """
345        if self.supports_incremental:
346            return True
347        if self.has_multiple_slices:
348            # We temporarily gate substream to not support RFR because puts a pretty high burden on connector developers
349            # to structure stream state in a very specific way. We also can't check for issubclass(HttpSubStream) because
350            # not all substreams implement the interface and it would be a circular dependency so we use parent as a surrogate
351            return False
352        elif hasattr(type(self), "state") and getattr(type(self), "state").fset is not None:
353            # Modern case where a stream manages state using getter/setter
354            return True
355        else:
356            # Legacy case where the CDK manages state via the get_updated_state() method. This is determined by checking if
357            # the stream's get_updated_state() differs from the Stream class and therefore has been overridden
358            return type(self).get_updated_state != Stream.get_updated_state
Returns

True if this stream allows the checkpointing of sync progress and can resume from it on subsequent attempts. This differs from supports_incremental because certain kinds of streams like those supporting resumable full refresh can checkpoint progress in between attempts for improved fault tolerance. However, they will start from the beginning on the next sync job.

cursor_field: Union[str, List[str]]
363    @property
364    def cursor_field(self) -> Union[str, List[str]]:
365        """
366        Override to return the default cursor field used by this stream e.g: an API entity might always use created_at as the cursor field.
367        :return: The name of the field used as a cursor. If the cursor is nested, return an array consisting of the path to the cursor.
368        """
369        return []

Override to return the default cursor field used by this stream e.g: an API entity might always use created_at as the cursor field.

Returns

The name of the field used as a cursor. If the cursor is nested, return an array consisting of the path to the cursor.

namespace: Optional[str]
371    @property
372    def namespace(self) -> Optional[str]:
373        """
374        Override to return the namespace of this stream, e.g. the Postgres schema which this stream will emit records for.
375        :return: A string containing the name of the namespace.
376        """
377        return None

Override to return the namespace of this stream, e.g. the Postgres schema which this stream will emit records for.

Returns

A string containing the name of the namespace.

source_defined_cursor: bool
379    @property
380    def source_defined_cursor(self) -> bool:
381        """
382        Return False if the cursor can be configured by the user.
383        """
384        return True

Return False if the cursor can be configured by the user.

exit_on_rate_limit: bool
386    @property
387    def exit_on_rate_limit(self) -> bool:
388        """Exit on rate limit getter, should return bool value. False if the stream will retry endlessly when rate limited."""
389        return self._exit_on_rate_limit

Exit on rate limit getter, should return bool value. False if the stream will retry endlessly when rate limited.

primary_key: Union[str, List[str], List[List[str]], NoneType]
396    @property
397    @abstractmethod
398    def primary_key(self) -> Optional[Union[str, List[str], List[List[str]]]]:
399        """
400        :return: string if single primary key, list of strings if composite primary key, list of list of strings if composite primary key consisting of nested fields.
401          If the stream has no primary keys, return None.
402        """
Returns

string if single primary key, list of strings if composite primary key, list of list of strings if composite primary key consisting of nested fields. If the stream has no primary keys, return None.

def stream_slices( self, *, sync_mode: airbyte_protocol_dataclasses.models.airbyte_protocol.SyncMode, cursor_field: Optional[List[str]] = None, stream_state: Optional[Mapping[str, Any]] = None) -> Iterable[Optional[Mapping[str, Any]]]:
404    def stream_slices(
405        self,
406        *,
407        sync_mode: SyncMode,
408        cursor_field: Optional[List[str]] = None,
409        stream_state: Optional[Mapping[str, Any]] = None,
410    ) -> Iterable[Optional[Mapping[str, Any]]]:
411        """
412        Override to define the slices for this stream. See the stream slicing section of the docs for more information.
413
414        :param sync_mode:
415        :param cursor_field:
416        :param stream_state:
417        :return:
418        """
419        yield StreamSlice(partition={}, cursor_slice={})

Override to define the slices for this stream. See the stream slicing section of the docs for more information.

Parameters
  • sync_mode:
  • cursor_field:
  • stream_state:
Returns
state_checkpoint_interval: Optional[int]
421    @property
422    def state_checkpoint_interval(self) -> Optional[int]:
423        """
424        Decides how often to checkpoint state (i.e: emit a STATE message). E.g: if this returns a value of 100, then state is persisted after reading
425        100 records, then 200, 300, etc.. A good default value is 1000 although your mileage may vary depending on the underlying data source.
426
427        Checkpointing a stream avoids re-reading records in the case a sync is failed or cancelled.
428
429        return None if state should not be checkpointed e.g: because records returned from the underlying data source are not returned in
430        ascending order with respect to the cursor field. This can happen if the source does not support reading records in ascending order of
431        created_at date (or whatever the cursor is). In those cases, state must only be saved once the full stream has been read.
432        """
433        return None

Decides how often to checkpoint state (i.e: emit a STATE message). E.g: if this returns a value of 100, then state is persisted after reading 100 records, then 200, 300, etc.. A good default value is 1000 although your mileage may vary depending on the underlying data source.

Checkpointing a stream avoids re-reading records in the case a sync is failed or cancelled.

return None if state should not be checkpointed e.g: because records returned from the underlying data source are not returned in ascending order with respect to the cursor field. This can happen if the source does not support reading records in ascending order of created_at date (or whatever the cursor is). In those cases, state must only be saved once the full stream has been read.

def get_updated_state( self, current_stream_state: MutableMapping[str, Any], latest_record: Mapping[str, Any]) -> MutableMapping[str, Any]:
441    def get_updated_state(
442        self, current_stream_state: MutableMapping[str, Any], latest_record: Mapping[str, Any]
443    ) -> MutableMapping[str, Any]:
444        """DEPRECATED. Please use explicit state property instead, see `IncrementalMixin` docs.
445
446        Override to extract state from the latest record. Needed to implement incremental sync.
447
448        Inspects the latest record extracted from the data source and the current state object and return an updated state object.
449
450        For example: if the state object is based on created_at timestamp, and the current state is {'created_at': 10}, and the latest_record is
451        {'name': 'octavia', 'created_at': 20 } then this method would return {'created_at': 20} to indicate state should be updated to this object.
452
453        :param current_stream_state: The stream's current state object
454        :param latest_record: The latest record extracted from the stream
455        :return: An updated state object
456        """
457        return {}

DEPRECATED. Please use explicit state property instead, see IncrementalMixin docs.

Override to extract state from the latest record. Needed to implement incremental sync.

Inspects the latest record extracted from the data source and the current state object and return an updated state object.

For example: if the state object is based on created_at timestamp, and the current state is {'created_at': 10}, and the latest_record is {'name': 'octavia', 'created_at': 20 } then this method would return {'created_at': 20} to indicate state should be updated to this object.

Parameters
  • current_stream_state: The stream's current state object
  • latest_record: The latest record extracted from the stream
Returns

An updated state object

def get_cursor(self) -> Optional[airbyte_cdk.sources.streams.checkpoint.Cursor]:
459    def get_cursor(self) -> Optional[Cursor]:
460        """
461        A Cursor is an interface that a stream can implement to manage how its internal state is read and updated while
462        reading records. Historically, Python connectors had no concept of a cursor to manage state. Python streams need
463        to define a cursor implementation and override this method to manage state through a Cursor.
464        """
465        return self.cursor

A Cursor is an interface that a stream can implement to manage how its internal state is read and updated while reading records. Historically, Python connectors had no concept of a cursor to manage state. Python streams need to define a cursor implementation and override this method to manage state through a Cursor.

def log_stream_sync_configuration(self) -> None:
591    def log_stream_sync_configuration(self) -> None:
592        """
593        Logs the configuration of this stream.
594        """
595        self.logger.debug(
596            f"Syncing stream instance: {self.name}",
597            extra={
598                "primary_key": self.primary_key,
599                "cursor_field": self.cursor_field,
600            },
601        )

Logs the configuration of this stream.

configured_json_schema: Optional[Dict[str, Any]]
663    @property
664    def configured_json_schema(self) -> Optional[Dict[str, Any]]:
665        """
666        This property is set from the read method.
667
668        :return Optional[Dict]: JSON schema from configured catalog if provided, otherwise None.
669        """
670        return self._configured_json_schema

This property is set from the read method.

Returns

JSON schema from configured catalog if provided, otherwise None.

StreamData
def package_name_from_class(cls: object) -> str:
52def package_name_from_class(cls: object) -> str:
53    """Find the package name given a class name"""
54    module = inspect.getmodule(cls)
55    if module is not None:
56        return module.__name__.split(".")[0]
57    else:
58        raise ValueError(f"Could not find package name for class {cls}")

Find the package name given a class name

class AirbyteTracedException(builtins.Exception):
 26class AirbyteTracedException(Exception):
 27    """
 28    An exception that should be emitted as an AirbyteTraceMessage
 29    """
 30
 31    def __init__(
 32        self,
 33        internal_message: Optional[str] = None,
 34        message: Optional[str] = None,
 35        failure_type: FailureType = FailureType.system_error,
 36        exception: Optional[BaseException] = None,
 37        stream_descriptor: Optional[StreamDescriptor] = None,
 38    ):
 39        """
 40        :param internal_message: the internal error that caused the failure
 41        :param message: a user-friendly message that indicates the cause of the error
 42        :param failure_type: the type of error
 43        :param exception: the exception that caused the error, from which the stack trace should be retrieved
 44        :param stream_descriptor: describe the stream from which the exception comes from
 45        """
 46        self.internal_message = internal_message
 47        self.message = message
 48        self.failure_type = failure_type
 49        self._exception = exception
 50        self._stream_descriptor = stream_descriptor
 51        super().__init__(internal_message)
 52
 53    def __str__(self) -> str:
 54        """Return the user-facing message, falling back to internal_message."""
 55        if self.message is not None:
 56            return self.message
 57        if self.internal_message is not None:
 58            return self.internal_message
 59        return ""
 60
 61    def as_airbyte_message(
 62        self, stream_descriptor: Optional[StreamDescriptor] = None
 63    ) -> AirbyteMessage:
 64        """
 65        Builds an AirbyteTraceMessage from the exception
 66
 67        :param stream_descriptor is deprecated, please use the stream_description in `__init__ or `from_exception`. If many
 68          stream_descriptors are defined, the one from `as_airbyte_message` will be discarded.
 69        """
 70        now_millis = time.time_ns() // 1_000_000
 71
 72        trace_exc = self._exception or self
 73        stack_trace_str = "".join(traceback.TracebackException.from_exception(trace_exc).format())
 74
 75        trace_message = AirbyteTraceMessage(
 76            type=TraceType.ERROR,
 77            emitted_at=now_millis,
 78            error=AirbyteErrorTraceMessage(
 79                message=self.message
 80                or "Something went wrong in the connector. See the logs for more details.",
 81                internal_message=self.internal_message,
 82                failure_type=self.failure_type,
 83                stack_trace=stack_trace_str,
 84                stream_descriptor=self._stream_descriptor
 85                if self._stream_descriptor is not None
 86                else stream_descriptor,
 87            ),
 88        )
 89
 90        return AirbyteMessage(type=MessageType.TRACE, trace=trace_message)
 91
 92    def as_connection_status_message(self) -> Optional[AirbyteMessage]:
 93        """Build a failed CONNECTION_STATUS from a config error, with secrets masked.
 94
 95        The message can quote config values, as a config validation error does, so it goes
 96        through the same secret filter as logs and traces.
 97        """
 98        if self.failure_type == FailureType.config_error:
 99            return AirbyteMessage(
100                type=MessageType.CONNECTION_STATUS,
101                connectionStatus=AirbyteConnectionStatus(
102                    status=Status.FAILED,
103                    message=filter_secrets(self.message) if self.message else self.message,
104                ),
105            )
106        return None
107
108    def emit_message(self) -> None:
109        """
110        Prints the exception as an AirbyteTraceMessage.
111        Note that this will be called automatically on uncaught exceptions when using the airbyte_cdk entrypoint.
112        """
113        message = orjson.dumps(AirbyteMessageSerializer.dump(self.as_airbyte_message())).decode()
114        filtered_message = filter_secrets(message)
115        print(filtered_message)
116
117    @classmethod
118    def from_exception(
119        cls,
120        exc: BaseException,
121        stream_descriptor: Optional[StreamDescriptor] = None,
122        *args: Any,
123        **kwargs: Any,
124    ) -> "AirbyteTracedException":
125        """
126        Helper to create an AirbyteTracedException from an existing exception
127        :param exc: the exception that caused the error
128        :param stream_descriptor: describe the stream from which the exception comes from
129        """
130        if isinstance(exc, AirbyteTracedException):
131            internal_message = exc.internal_message
132            # Preserve the original user-facing message if the caller didn't provide one
133            if "message" not in kwargs:
134                kwargs["message"] = exc.message
135        else:
136            internal_message = str(exc)
137        return cls(
138            internal_message=internal_message,
139            exception=exc,
140            stream_descriptor=stream_descriptor,
141            *args,
142            **kwargs,
143        )  # type: ignore  # ignoring because of args and kwargs
144
145    def as_sanitized_airbyte_message(
146        self, stream_descriptor: Optional[StreamDescriptor] = None
147    ) -> AirbyteMessage:
148        """
149        Builds an AirbyteTraceMessage from the exception and sanitizes any secrets from the message body
150
151        :param stream_descriptor is deprecated, please use the stream_description in `__init__ or `from_exception`. If many
152          stream_descriptors are defined, the one from `as_sanitized_airbyte_message` will be discarded.
153        """
154        error_message = self.as_airbyte_message(stream_descriptor=stream_descriptor)
155        if error_message.trace.error.message:  # type: ignore[union-attr] # AirbyteMessage with MessageType.TRACE has AirbyteTraceMessage
156            error_message.trace.error.message = filter_secrets(  # type: ignore[union-attr]
157                error_message.trace.error.message,  # type: ignore[union-attr]
158            )
159        if error_message.trace.error.internal_message:  # type: ignore[union-attr] # AirbyteMessage with MessageType.TRACE has AirbyteTraceMessage
160            error_message.trace.error.internal_message = filter_secrets(  # type: ignore[union-attr] # AirbyteMessage with MessageType.TRACE has AirbyteTraceMessage
161                error_message.trace.error.internal_message  # type: ignore[union-attr] # AirbyteMessage with MessageType.TRACE has AirbyteTraceMessage
162            )
163        if error_message.trace.error.stack_trace:  # type: ignore[union-attr] # AirbyteMessage with MessageType.TRACE has AirbyteTraceMessage
164            error_message.trace.error.stack_trace = filter_secrets(  # type: ignore[union-attr] # AirbyteMessage with MessageType.TRACE has AirbyteTraceMessage
165                error_message.trace.error.stack_trace  # type: ignore[union-attr] # AirbyteMessage with MessageType.TRACE has AirbyteTraceMessage
166            )
167        return error_message

An exception that should be emitted as an AirbyteTraceMessage

AirbyteTracedException( internal_message: Optional[str] = None, message: Optional[str] = None, failure_type: airbyte_protocol_dataclasses.models.airbyte_protocol.FailureType = <FailureType.system_error: 'system_error'>, exception: Optional[BaseException] = None, stream_descriptor: Optional[airbyte_protocol_dataclasses.models.airbyte_protocol.StreamDescriptor] = None)
31    def __init__(
32        self,
33        internal_message: Optional[str] = None,
34        message: Optional[str] = None,
35        failure_type: FailureType = FailureType.system_error,
36        exception: Optional[BaseException] = None,
37        stream_descriptor: Optional[StreamDescriptor] = None,
38    ):
39        """
40        :param internal_message: the internal error that caused the failure
41        :param message: a user-friendly message that indicates the cause of the error
42        :param failure_type: the type of error
43        :param exception: the exception that caused the error, from which the stack trace should be retrieved
44        :param stream_descriptor: describe the stream from which the exception comes from
45        """
46        self.internal_message = internal_message
47        self.message = message
48        self.failure_type = failure_type
49        self._exception = exception
50        self._stream_descriptor = stream_descriptor
51        super().__init__(internal_message)
Parameters
  • internal_message: the internal error that caused the failure
  • message: a user-friendly message that indicates the cause of the error
  • failure_type: the type of error
  • exception: the exception that caused the error, from which the stack trace should be retrieved
  • stream_descriptor: describe the stream from which the exception comes from
internal_message
message
failure_type
def as_airbyte_message( self, stream_descriptor: Optional[airbyte_protocol_dataclasses.models.airbyte_protocol.StreamDescriptor] = None) -> AirbyteMessage:
61    def as_airbyte_message(
62        self, stream_descriptor: Optional[StreamDescriptor] = None
63    ) -> AirbyteMessage:
64        """
65        Builds an AirbyteTraceMessage from the exception
66
67        :param stream_descriptor is deprecated, please use the stream_description in `__init__ or `from_exception`. If many
68          stream_descriptors are defined, the one from `as_airbyte_message` will be discarded.
69        """
70        now_millis = time.time_ns() // 1_000_000
71
72        trace_exc = self._exception or self
73        stack_trace_str = "".join(traceback.TracebackException.from_exception(trace_exc).format())
74
75        trace_message = AirbyteTraceMessage(
76            type=TraceType.ERROR,
77            emitted_at=now_millis,
78            error=AirbyteErrorTraceMessage(
79                message=self.message
80                or "Something went wrong in the connector. See the logs for more details.",
81                internal_message=self.internal_message,
82                failure_type=self.failure_type,
83                stack_trace=stack_trace_str,
84                stream_descriptor=self._stream_descriptor
85                if self._stream_descriptor is not None
86                else stream_descriptor,
87            ),
88        )
89
90        return AirbyteMessage(type=MessageType.TRACE, trace=trace_message)

Builds an AirbyteTraceMessage from the exception

:param stream_descriptor is deprecated, please use the stream_description in __init__ orfrom_exception. If many stream_descriptors are defined, the one fromas_airbyte_message` will be discarded.

def as_connection_status_message(self) -> Optional[AirbyteMessage]:
 92    def as_connection_status_message(self) -> Optional[AirbyteMessage]:
 93        """Build a failed CONNECTION_STATUS from a config error, with secrets masked.
 94
 95        The message can quote config values, as a config validation error does, so it goes
 96        through the same secret filter as logs and traces.
 97        """
 98        if self.failure_type == FailureType.config_error:
 99            return AirbyteMessage(
100                type=MessageType.CONNECTION_STATUS,
101                connectionStatus=AirbyteConnectionStatus(
102                    status=Status.FAILED,
103                    message=filter_secrets(self.message) if self.message else self.message,
104                ),
105            )
106        return None

Build a failed CONNECTION_STATUS from a config error, with secrets masked.

The message can quote config values, as a config validation error does, so it goes through the same secret filter as logs and traces.

def emit_message(self) -> None:
108    def emit_message(self) -> None:
109        """
110        Prints the exception as an AirbyteTraceMessage.
111        Note that this will be called automatically on uncaught exceptions when using the airbyte_cdk entrypoint.
112        """
113        message = orjson.dumps(AirbyteMessageSerializer.dump(self.as_airbyte_message())).decode()
114        filtered_message = filter_secrets(message)
115        print(filtered_message)

Prints the exception as an AirbyteTraceMessage. Note that this will be called automatically on uncaught exceptions when using the airbyte_cdk entrypoint.

@classmethod
def from_exception( cls, exc: BaseException, stream_descriptor: Optional[airbyte_protocol_dataclasses.models.airbyte_protocol.StreamDescriptor] = None, *args: Any, **kwargs: Any) -> AirbyteTracedException:
117    @classmethod
118    def from_exception(
119        cls,
120        exc: BaseException,
121        stream_descriptor: Optional[StreamDescriptor] = None,
122        *args: Any,
123        **kwargs: Any,
124    ) -> "AirbyteTracedException":
125        """
126        Helper to create an AirbyteTracedException from an existing exception
127        :param exc: the exception that caused the error
128        :param stream_descriptor: describe the stream from which the exception comes from
129        """
130        if isinstance(exc, AirbyteTracedException):
131            internal_message = exc.internal_message
132            # Preserve the original user-facing message if the caller didn't provide one
133            if "message" not in kwargs:
134                kwargs["message"] = exc.message
135        else:
136            internal_message = str(exc)
137        return cls(
138            internal_message=internal_message,
139            exception=exc,
140            stream_descriptor=stream_descriptor,
141            *args,
142            **kwargs,
143        )  # type: ignore  # ignoring because of args and kwargs

Helper to create an AirbyteTracedException from an existing exception

Parameters
  • exc: the exception that caused the error
  • stream_descriptor: describe the stream from which the exception comes from
def as_sanitized_airbyte_message( self, stream_descriptor: Optional[airbyte_protocol_dataclasses.models.airbyte_protocol.StreamDescriptor] = None) -> AirbyteMessage:
145    def as_sanitized_airbyte_message(
146        self, stream_descriptor: Optional[StreamDescriptor] = None
147    ) -> AirbyteMessage:
148        """
149        Builds an AirbyteTraceMessage from the exception and sanitizes any secrets from the message body
150
151        :param stream_descriptor is deprecated, please use the stream_description in `__init__ or `from_exception`. If many
152          stream_descriptors are defined, the one from `as_sanitized_airbyte_message` will be discarded.
153        """
154        error_message = self.as_airbyte_message(stream_descriptor=stream_descriptor)
155        if error_message.trace.error.message:  # type: ignore[union-attr] # AirbyteMessage with MessageType.TRACE has AirbyteTraceMessage
156            error_message.trace.error.message = filter_secrets(  # type: ignore[union-attr]
157                error_message.trace.error.message,  # type: ignore[union-attr]
158            )
159        if error_message.trace.error.internal_message:  # type: ignore[union-attr] # AirbyteMessage with MessageType.TRACE has AirbyteTraceMessage
160            error_message.trace.error.internal_message = filter_secrets(  # type: ignore[union-attr] # AirbyteMessage with MessageType.TRACE has AirbyteTraceMessage
161                error_message.trace.error.internal_message  # type: ignore[union-attr] # AirbyteMessage with MessageType.TRACE has AirbyteTraceMessage
162            )
163        if error_message.trace.error.stack_trace:  # type: ignore[union-attr] # AirbyteMessage with MessageType.TRACE has AirbyteTraceMessage
164            error_message.trace.error.stack_trace = filter_secrets(  # type: ignore[union-attr] # AirbyteMessage with MessageType.TRACE has AirbyteTraceMessage
165                error_message.trace.error.stack_trace  # type: ignore[union-attr] # AirbyteMessage with MessageType.TRACE has AirbyteTraceMessage
166            )
167        return error_message

Builds an AirbyteTraceMessage from the exception and sanitizes any secrets from the message body

:param stream_descriptor is deprecated, please use the stream_description in __init__ orfrom_exception. If many stream_descriptors are defined, the one fromas_sanitized_airbyte_message` will be discarded.

def is_cloud_environment() -> bool:
11def is_cloud_environment() -> bool:
12    """
13    Returns True if the connector is running in a cloud environment, False otherwise.
14
15    The function checks the value of the DEPLOYMENT_MODE environment variable which is set by the platform.
16    This function can be used to determine whether stricter security measures should be applied.
17    """
18    deployment_mode = os.environ.get("DEPLOYMENT_MODE", "")
19    return deployment_mode.casefold() == CLOUD_DEPLOYMENT_MODE

Returns True if the connector is running in a cloud environment, False otherwise.

The function checks the value of the DEPLOYMENT_MODE environment variable which is set by the platform. This function can be used to determine whether stricter security measures should be applied.

class InternalConfig(pydantic.v1.main.BaseModel):
210class InternalConfig(BaseModel):
211    KEYWORDS: ClassVar[set[str]] = {"_limit", "_page_size"}
212    limit: int = Field(None, alias="_limit")
213    page_size: int = Field(None, alias="_page_size")
214
215    def dict(self, *args: Any, **kwargs: Any) -> dict[str, Any]:
216        kwargs["by_alias"] = True
217        kwargs["exclude_unset"] = True
218        return super().dict(*args, **kwargs)
219
220    def is_limit_reached(self, records_counter: int) -> bool:
221        """
222        Check if record count reached limit set by internal config.
223        :param records_counter - number of records already red
224        :return True if limit reached, False otherwise
225        """
226        if self.limit:
227            if records_counter >= self.limit:
228                return True
229        return False
KEYWORDS: ClassVar[set[str]] = {'_limit', '_page_size'}
limit: int
page_size: int
def dict(self, *args: Any, **kwargs: Any) -> dict[str, typing.Any]:
215    def dict(self, *args: Any, **kwargs: Any) -> dict[str, Any]:
216        kwargs["by_alias"] = True
217        kwargs["exclude_unset"] = True
218        return super().dict(*args, **kwargs)

Generate a dictionary representation of the model, optionally specifying which fields to include or exclude.

def is_limit_reached(self, records_counter: int) -> bool:
220    def is_limit_reached(self, records_counter: int) -> bool:
221        """
222        Check if record count reached limit set by internal config.
223        :param records_counter - number of records already red
224        :return True if limit reached, False otherwise
225        """
226        if self.limit:
227            if records_counter >= self.limit:
228                return True
229        return False

Check if record count reached limit set by internal config. :param records_counter - number of records already red :return True if limit reached, False otherwise

class ResourceSchemaLoader:
136class ResourceSchemaLoader:
137    """JSONSchema loader from package resources"""
138
139    def __init__(self, package_name: str):
140        self.package_name = package_name
141
142    def get_schema(self, name: str) -> dict[str, Any]:
143        """
144        This method retrieves a JSON schema from the schemas/ folder.
145
146
147        The expected file structure is to have all top-level schemas (corresponding to streams) in the "schemas/" folder, with any shared $refs
148        living inside the "schemas/shared/" folder. For example:
149
150        schemas/shared/<shared_definition>.json
151        schemas/<name>.json # contains a $ref to shared_definition
152        schemas/<name2>.json # contains a $ref to shared_definition
153        """
154
155        schema_filename = f"schemas/{name}.json"
156        raw_file = pkgutil.get_data(self.package_name, schema_filename)
157        if not raw_file:
158            raise IOError(f"Cannot find file {schema_filename}")
159        try:
160            raw_schema = json.loads(raw_file)
161        except ValueError as err:
162            raise RuntimeError(f"Invalid JSON file format for file {schema_filename}") from err
163
164        return self._resolve_schema_references(raw_schema)
165
166    def _resolve_schema_references(self, raw_schema: dict[str, Any]) -> dict[str, Any]:
167        """
168        Resolve links to external references and move it to local "definitions" map.
169
170        :param raw_schema jsonschema to lookup for external links.
171        :return JSON serializable object with references without external dependencies.
172        """
173
174        package = importlib.import_module(self.package_name)
175        if package.__file__:
176            base = os.path.dirname(package.__file__) + "/"
177        else:
178            raise ValueError(f"Package {package} does not have a valid __file__ field")
179        resolved = jsonref.JsonRef.replace_refs(
180            raw_schema, loader=JsonFileLoader(base, "schemas/shared"), base_uri=base
181        )
182        resolved = resolve_ref_links(resolved)
183        if isinstance(resolved, dict):
184            return resolved
185        else:
186            raise ValueError(f"Expected resolved to be a dict. Got {resolved}")

JSONSchema loader from package resources

ResourceSchemaLoader(package_name: str)
139    def __init__(self, package_name: str):
140        self.package_name = package_name
package_name
def get_schema(self, name: str) -> dict[str, typing.Any]:
142    def get_schema(self, name: str) -> dict[str, Any]:
143        """
144        This method retrieves a JSON schema from the schemas/ folder.
145
146
147        The expected file structure is to have all top-level schemas (corresponding to streams) in the "schemas/" folder, with any shared $refs
148        living inside the "schemas/shared/" folder. For example:
149
150        schemas/shared/<shared_definition>.json
151        schemas/<name>.json # contains a $ref to shared_definition
152        schemas/<name2>.json # contains a $ref to shared_definition
153        """
154
155        schema_filename = f"schemas/{name}.json"
156        raw_file = pkgutil.get_data(self.package_name, schema_filename)
157        if not raw_file:
158            raise IOError(f"Cannot find file {schema_filename}")
159        try:
160            raw_schema = json.loads(raw_file)
161        except ValueError as err:
162            raise RuntimeError(f"Invalid JSON file format for file {schema_filename}") from err
163
164        return self._resolve_schema_references(raw_schema)

This method retrieves a JSON schema from the schemas/ folder.

The expected file structure is to have all top-level schemas (corresponding to streams) in the "schemas/" folder, with any shared $refs living inside the "schemas/shared/" folder. For example:

schemas/shared/.json schemas/.json # contains a $ref to shared_definition schemas/.json # contains a $ref to shared_definition

def check_config_against_spec_or_exit( config: Mapping[str, Any], spec: ConnectorSpecification) -> None:
189def check_config_against_spec_or_exit(
190    config: Mapping[str, Any], spec: ConnectorSpecification
191) -> None:
192    """
193    Check config object against spec. In case of spec is invalid, throws
194    an exception with validation error description.
195
196    :param config - config loaded from file specified over command line
197    :param spec - spec object generated by connector
198    """
199    spec_schema = spec.connectionSpecification
200    try:
201        validate(instance=config, schema=spec_schema)
202    except ValidationError as validation_error:
203        raise AirbyteTracedException(
204            message="Config validation error: " + validation_error.message,
205            internal_message=validation_error.message,
206            failure_type=FailureType.config_error,
207        ) from None  # required to prevent logging config secrets from the ValidationError's stacktrace

Check config object against spec. In case of spec is invalid, throws an exception with validation error description.

:param config - config loaded from file specified over command line :param spec - spec object generated by connector

def split_config( config: Mapping[str, Any]) -> Tuple[dict[str, Any], InternalConfig]:
232def split_config(config: Mapping[str, Any]) -> Tuple[dict[str, Any], InternalConfig]:
233    """
234    Break config map object into 2 instances: first is a dict with user defined
235    configuration and second is internal config that contains private keys for
236    acceptance test configuration.
237
238    :param
239     config - Dict object that has been loaded from config file.
240
241    :return tuple of user defined config dict with filtered out internal
242    parameters and connector acceptance test internal config object.
243    """
244    main_config = {}
245    internal_config = {}
246    for k, v in config.items():
247        if k in InternalConfig.KEYWORDS:
248            internal_config[k] = v
249        else:
250            main_config[k] = v
251    return main_config, InternalConfig.parse_obj(internal_config)

Break config map object into 2 instances: first is a dict with user defined configuration and second is internal config that contains private keys for acceptance test configuration.

:param config - Dict object that has been loaded from config file.

:return tuple of user defined config dict with filtered out internal parameters and connector acceptance test internal config object.

class TransformConfig(enum.Flag):
49class TransformConfig(Flag):
50    """
51    TypeTransformer class config. Configs can be combined using bitwise or operator e.g.
52        ```
53        TransformConfig.DefaultSchemaNormalization | TransformConfig.CustomSchemaNormalization
54        ```
55    """
56
57    # No action taken, default behavior. Cannot be combined with any other options.
58    NoTransform = auto()
59    # Applies default type casting with default_convert method which converts
60    # values by applying simple type casting to specified jsonschema type.
61    DefaultSchemaNormalization = auto()
62    # Allow registering custom type transformation callback. Can be combined
63    # with DefaultSchemaNormalization. In this case default type casting would
64    # be applied before custom one.
65    CustomSchemaNormalization = auto()

TypeTransformer class config. Configs can be combined using bitwise or operator e.g.

TransformConfig.DefaultSchemaNormalization | TransformConfig.CustomSchemaNormalization
NoTransform = <TransformConfig.NoTransform: 1>
DefaultSchemaNormalization = <TransformConfig.DefaultSchemaNormalization: 2>
CustomSchemaNormalization = <TransformConfig.CustomSchemaNormalization: 4>
class TypeTransformer:
 68class TypeTransformer:
 69    """
 70    Class for transforming object before output.
 71    """
 72
 73    _custom_normalizer: Optional[Callable[[Any, Dict[str, Any]], Any]] = None
 74
 75    def __init__(self, config: TransformConfig):
 76        """
 77        Initialize TypeTransformer instance.
 78        :param config Transform config that would be applied to object
 79        """
 80        if TransformConfig.NoTransform in config and config != TransformConfig.NoTransform:
 81            raise Exception("NoTransform option cannot be combined with other flags.")
 82        self._config = config
 83        all_validators = {
 84            key: self.__get_normalizer(key, orig_validator)
 85            for key, orig_validator in Draft7Validator.VALIDATORS.items()
 86            # Do not validate field we do not transform for maximum performance.
 87            if key in ["type", "array", "$ref", "properties", "items"]
 88        }
 89        self._normalizer = validators.create(
 90            meta_schema=Draft7Validator.META_SCHEMA, validators=all_validators
 91        )
 92
 93    def registerCustomTransform(
 94        self, normalization_callback: Callable[[Any, dict[str, Any]], Any]
 95    ) -> Callable[[Any, dict[str, Any]], Any]:
 96        """
 97        Register custom normalization callback.
 98        :param normalization_callback function to be used for value
 99        normalization. Takes original value and part type schema. Should return
100        normalized value. See docs/connector-development/cdk-python/schemas.md
101        for details.
102        :return Same callback, this is useful for using registerCustomTransform function as decorator.
103        """
104        if TransformConfig.CustomSchemaNormalization not in self._config:
105            raise Exception(
106                "Please set TransformConfig.CustomSchemaNormalization config before registering custom normalizer"
107            )
108        self._custom_normalizer = normalization_callback
109        return normalization_callback
110
111    def __normalize(self, original_item: Any, subschema: Dict[str, Any]) -> Any:
112        """
113        Applies different transform function to object's field according to config.
114        :param original_item original value of field.
115        :param subschema part of the jsonschema containing field type/format data.
116        :return Final field value.
117        """
118        if TransformConfig.DefaultSchemaNormalization in self._config:
119            original_item = self.default_convert(original_item, subschema)
120
121        if self._custom_normalizer:
122            original_item = self._custom_normalizer(original_item, subschema)
123        return original_item
124
125    @staticmethod
126    def default_convert(original_item: Any, subschema: Dict[str, Any]) -> Any:
127        """
128        Default transform function that is used when TransformConfig.DefaultSchemaNormalization flag set.
129        :param original_item original value of field.
130        :param subschema part of the jsonschema containing field type/format data.
131        :return transformed field value.
132        """
133        target_type = subschema.get("type", [])
134        if original_item is None and "null" in target_type:
135            return None
136        if isinstance(target_type, list):
137            # jsonschema type could either be a single string or array of type
138            # strings. In case if there is some disambigous and more than one
139            # type (except null) do not do any conversion and return original
140            # value. If type array has one type and null i.e. {"type":
141            # ["integer", "null"]}, convert value to specified type.
142            target_type = [t for t in target_type if t != "null"]
143            if len(target_type) != 1:
144                return original_item
145            target_type = target_type[0]
146        try:
147            if target_type == "string":
148                return str(original_item)
149            elif target_type == "number":
150                return float(original_item)
151            elif target_type == "integer":
152                return int(original_item)
153            elif target_type == "boolean":
154                if isinstance(original_item, str):
155                    return _strtobool(original_item) == 1
156                return bool(original_item)
157            elif target_type == "array":
158                item_types = set(subschema.get("items", {}).get("type", set()))
159                if (
160                    item_types.issubset(json_to_python_simple)
161                    and type(original_item) in json_to_python_simple.values()
162                ):
163                    return [original_item]
164        except (ValueError, TypeError):
165            return original_item
166        return original_item
167
168    def __get_normalizer(
169        self,
170        schema_key: str,
171        original_validator: Callable,  # type: ignore[type-arg]
172    ) -> Callable[[Any, Any, Any, dict[str, Any]], Generator[Any, Any, None]]:
173        """
174        Traverse through object fields using native jsonschema validator and apply normalization function.
175        :param schema_key related json schema key that currently being validated/normalized.
176        :original_validator: native jsonschema validator callback.
177        """
178
179        def normalizator(
180            validator_instance: Validator,
181            property_value: Any,
182            instance: Any,
183            schema: Dict[str, Any],
184        ) -> Generator[Any, Any, None]:
185            """
186            Jsonschema validator callable it uses for validating instance. We
187            override default Draft7Validator to perform value transformation
188            before validation take place. We do not take any action except
189            logging warn if object does not conform to json schema, just using
190            jsonschema algorithm to traverse through object fields.
191            Look
192            https://python-jsonschema.readthedocs.io/en/stable/creating/?highlight=validators.create#jsonschema.validators.create
193            validators parameter for detailed description.
194            :
195            """
196            # Transform object and array values before running json schema type checking for each element.
197            # Recursively normalize every value of the "instance" sub-object,
198            # if "instance" is an incorrect type - skip recursive normalization of "instance"
199            if schema_key == "properties" and isinstance(instance, dict):
200                for k, subschema in property_value.items():
201                    if k in instance:
202                        instance[k] = self.__normalize(instance[k], subschema)
203            # Recursively normalize every item of the "instance" sub-array,
204            # if "instance" is an incorrect type - skip recursive normalization of "instance"
205            elif schema_key == "items" and isinstance(instance, list):
206                for index, item in enumerate(instance):
207                    instance[index] = self.__normalize(item, property_value)
208
209            # Running native jsonschema traverse algorithm after field normalization is done.
210            yield from original_validator(
211                validator_instance,
212                property_value,
213                instance,
214                schema,
215            )
216
217        return normalizator
218
219    def transform(
220        self,
221        record: Dict[str, Any],
222        schema: Mapping[str, Any],
223    ) -> None:
224        """
225        Normalize and validate according to config.
226        :param record: record instance for normalization/transformation. All modification are done by modifying existent object.
227        :param schema: object's jsonschema for normalization.
228        """
229        if TransformConfig.NoTransform in self._config:
230            return
231        normalizer = self._normalizer(schema)
232        for e in normalizer.iter_errors(record):
233            """
234            just calling normalizer.validate() would throw an exception on
235            first validation occurrences and stop processing rest of schema.
236            """
237            logger.warning(self.get_error_message(e))
238
239    def get_error_message(self, e: ValidationError) -> str:
240        """
241        Construct a sanitized error message from a ValidationError instance.
242        """
243        field_path = ".".join(map(str, e.path))
244        type_structure = self._get_type_structure(e.instance)
245
246        return f"Failed to transform value from type '{type_structure}' to type '{e.validator_value}' at path: '{field_path}'"
247
248    def _get_type_structure(self, input_data: Any, current_depth: int = 0) -> Any:
249        """
250        Get the structure of a given input data for use in error message construction.
251        """
252        # Handle null values
253        if input_data is None:
254            return "null"
255
256        # Avoid recursing too deep
257        if current_depth >= MAX_NESTING_DEPTH:
258            return "object" if isinstance(input_data, dict) else python_to_json[type(input_data)]
259
260        if isinstance(input_data, dict):
261            return {
262                key: self._get_type_structure(field_value, current_depth + 1)
263                for key, field_value in input_data.items()
264            }
265
266        else:
267            return python_to_json[type(input_data)]

Class for transforming object before output.

TypeTransformer(config: TransformConfig)
75    def __init__(self, config: TransformConfig):
76        """
77        Initialize TypeTransformer instance.
78        :param config Transform config that would be applied to object
79        """
80        if TransformConfig.NoTransform in config and config != TransformConfig.NoTransform:
81            raise Exception("NoTransform option cannot be combined with other flags.")
82        self._config = config
83        all_validators = {
84            key: self.__get_normalizer(key, orig_validator)
85            for key, orig_validator in Draft7Validator.VALIDATORS.items()
86            # Do not validate field we do not transform for maximum performance.
87            if key in ["type", "array", "$ref", "properties", "items"]
88        }
89        self._normalizer = validators.create(
90            meta_schema=Draft7Validator.META_SCHEMA, validators=all_validators
91        )

Initialize TypeTransformer instance. :param config Transform config that would be applied to object

def registerCustomTransform( self, normalization_callback: Callable[[Any, dict[str, Any]], Any]) -> Callable[[Any, dict[str, Any]], Any]:
 93    def registerCustomTransform(
 94        self, normalization_callback: Callable[[Any, dict[str, Any]], Any]
 95    ) -> Callable[[Any, dict[str, Any]], Any]:
 96        """
 97        Register custom normalization callback.
 98        :param normalization_callback function to be used for value
 99        normalization. Takes original value and part type schema. Should return
100        normalized value. See docs/connector-development/cdk-python/schemas.md
101        for details.
102        :return Same callback, this is useful for using registerCustomTransform function as decorator.
103        """
104        if TransformConfig.CustomSchemaNormalization not in self._config:
105            raise Exception(
106                "Please set TransformConfig.CustomSchemaNormalization config before registering custom normalizer"
107            )
108        self._custom_normalizer = normalization_callback
109        return normalization_callback

Register custom normalization callback. :param normalization_callback function to be used for value normalization. Takes original value and part type schema. Should return normalized value. See docs/connector-development/cdk-python/schemas.md for details. :return Same callback, this is useful for using registerCustomTransform function as decorator.

@staticmethod
def default_convert(original_item: Any, subschema: Dict[str, Any]) -> Any:
125    @staticmethod
126    def default_convert(original_item: Any, subschema: Dict[str, Any]) -> Any:
127        """
128        Default transform function that is used when TransformConfig.DefaultSchemaNormalization flag set.
129        :param original_item original value of field.
130        :param subschema part of the jsonschema containing field type/format data.
131        :return transformed field value.
132        """
133        target_type = subschema.get("type", [])
134        if original_item is None and "null" in target_type:
135            return None
136        if isinstance(target_type, list):
137            # jsonschema type could either be a single string or array of type
138            # strings. In case if there is some disambigous and more than one
139            # type (except null) do not do any conversion and return original
140            # value. If type array has one type and null i.e. {"type":
141            # ["integer", "null"]}, convert value to specified type.
142            target_type = [t for t in target_type if t != "null"]
143            if len(target_type) != 1:
144                return original_item
145            target_type = target_type[0]
146        try:
147            if target_type == "string":
148                return str(original_item)
149            elif target_type == "number":
150                return float(original_item)
151            elif target_type == "integer":
152                return int(original_item)
153            elif target_type == "boolean":
154                if isinstance(original_item, str):
155                    return _strtobool(original_item) == 1
156                return bool(original_item)
157            elif target_type == "array":
158                item_types = set(subschema.get("items", {}).get("type", set()))
159                if (
160                    item_types.issubset(json_to_python_simple)
161                    and type(original_item) in json_to_python_simple.values()
162                ):
163                    return [original_item]
164        except (ValueError, TypeError):
165            return original_item
166        return original_item

Default transform function that is used when TransformConfig.DefaultSchemaNormalization flag set. :param original_item original value of field. :param subschema part of the jsonschema containing field type/format data. :return transformed field value.

def transform(self, record: Dict[str, Any], schema: Mapping[str, Any]) -> None:
219    def transform(
220        self,
221        record: Dict[str, Any],
222        schema: Mapping[str, Any],
223    ) -> None:
224        """
225        Normalize and validate according to config.
226        :param record: record instance for normalization/transformation. All modification are done by modifying existent object.
227        :param schema: object's jsonschema for normalization.
228        """
229        if TransformConfig.NoTransform in self._config:
230            return
231        normalizer = self._normalizer(schema)
232        for e in normalizer.iter_errors(record):
233            """
234            just calling normalizer.validate() would throw an exception on
235            first validation occurrences and stop processing rest of schema.
236            """
237            logger.warning(self.get_error_message(e))

Normalize and validate according to config.

Parameters
  • record: record instance for normalization/transformation. All modification are done by modifying existent object.
  • schema: object's jsonschema for normalization.
def get_error_message(self, e: jsonschema.exceptions.ValidationError) -> str:
239    def get_error_message(self, e: ValidationError) -> str:
240        """
241        Construct a sanitized error message from a ValidationError instance.
242        """
243        field_path = ".".join(map(str, e.path))
244        type_structure = self._get_type_structure(e.instance)
245
246        return f"Failed to transform value from type '{type_structure}' to type '{e.validator_value}' at path: '{field_path}'"

Construct a sanitized error message from a ValidationError instance.

ENV_REQUEST_CACHE_PATH = 'REQUEST_CACHE_PATH'
@contextmanager
def create_timer( name: str) -> Generator[airbyte_cdk.utils.event_timing.EventTimer, Any, NoneType]:
80@contextmanager
81def create_timer(name: str) -> Generator[EventTimer, Any, None]:
82    """
83    Creates a new EventTimer as a context manager to improve code readability.
84    """
85    a_timer = EventTimer(name)
86    yield a_timer

Creates a new EventTimer as a context manager to improve code readability.

class OneOfOptionConfig:
 9class OneOfOptionConfig:
10    """
11    Base class to configure a Pydantic model that's used as a oneOf option in a parent model in a way that's compatible with all Airbyte consumers.
12
13    Inherit from this class in the nested Config class in a model and set title and description (these show up in the UI) and discriminator (this is making sure it's marked as required in the schema).
14
15    Usage:
16
17        ```python
18        class OptionModel(BaseModel):
19            mode: Literal["option_a"] = Field("option_a", const=True)
20            option_a_field: str = Field(...)
21
22            class Config(OneOfOptionConfig):
23                title = "Option A"
24                description = "Option A description"
25                discriminator = "mode"
26        ```
27    """
28
29    @staticmethod
30    def schema_extra(schema: Dict[str, Any], model: Any) -> None:
31        if hasattr(model.Config, "description"):
32            schema["description"] = model.Config.description
33        if hasattr(model.Config, "discriminator"):
34            schema.setdefault("required", []).append(model.Config.discriminator)

Base class to configure a Pydantic model that's used as a oneOf option in a parent model in a way that's compatible with all Airbyte consumers.

Inherit from this class in the nested Config class in a model and set title and description (these show up in the UI) and discriminator (this is making sure it's marked as required in the schema).

Usage:
class OptionModel(BaseModel):
    mode: Literal["option_a"] = Field("option_a", const=True)
    option_a_field: str = Field(...)

    class Config(OneOfOptionConfig):
        title = "Option A"
        description = "Option A description"
        discriminator = "mode"
@staticmethod
def schema_extra(schema: Dict[str, Any], model: Any) -> None:
29    @staticmethod
30    def schema_extra(schema: Dict[str, Any], model: Any) -> None:
31        if hasattr(model.Config, "description"):
32            schema["description"] = model.Config.description
33        if hasattr(model.Config, "discriminator"):
34            schema.setdefault("required", []).append(model.Config.discriminator)
def resolve_refs(schema: dict[str, typing.Any]) -> dict[str, typing.Any]:
14def resolve_refs(schema: dict[str, Any]) -> dict[str, Any]:
15    """
16    For spec schemas generated using Pydantic models, the resulting JSON schema can contain refs between object
17    relationships.
18    """
19    resource = Resource.from_contents(schema, default_specification=DRAFT7)
20    registry = Registry().with_resource("", resource)
21    resolver = registry.resolver()
22    str_schema = json.dumps(schema)
23    for ref_block in re.findall(r'{"\$ref": "#\/definitions\/.+?(?="})"}', str_schema):
24        ref = json.loads(ref_block)["$ref"]
25        resolved = resolver.lookup(ref).contents
26        str_schema = str_schema.replace(ref_block, json.dumps(resolved))
27    pyschema: dict[str, Any] = json.loads(str_schema)
28    del pyschema["definitions"]
29    return pyschema

For spec schemas generated using Pydantic models, the resulting JSON schema can contain refs between object relationships.

def as_airbyte_message( stream: Union[airbyte_protocol_dataclasses.models.airbyte_protocol.AirbyteStream, airbyte_protocol_dataclasses.models.airbyte_protocol.StreamDescriptor], current_status: airbyte_protocol_dataclasses.models.airbyte_protocol.AirbyteStreamStatus, reasons: Optional[List[airbyte_protocol_dataclasses.models.airbyte_protocol.AirbyteStreamStatusReason]] = None) -> AirbyteMessage:
23def as_airbyte_message(
24    stream: Union[AirbyteStream, StreamDescriptor],
25    current_status: AirbyteStreamStatus,
26    reasons: Optional[List[AirbyteStreamStatusReason]] = None,
27) -> AirbyteMessage:
28    """
29    Builds an AirbyteStreamStatusTraceMessage for the provided stream
30    """
31
32    now_millis = datetime.now().timestamp() * 1000.0
33
34    trace_message = AirbyteTraceMessage(
35        type=TraceType.STREAM_STATUS,
36        emitted_at=now_millis,
37        stream_status=AirbyteStreamStatusTraceMessage(
38            stream_descriptor=StreamDescriptor(name=stream.name, namespace=stream.namespace),
39            status=current_status,
40            reasons=reasons,
41        ),
42    )
43
44    return AirbyteMessage(type=MessageType.TRACE, trace=trace_message)

Builds an AirbyteStreamStatusTraceMessage for the provided stream

Config = typing.Mapping[str, typing.Any]
class Record(typing.Mapping[str, typing.Any]):
22class Record(Mapping[str, Any]):
23    def __init__(
24        self,
25        data: Mapping[str, Any],
26        stream_name: str,
27        associated_slice: Optional[StreamSlice] = None,
28        file_reference: Optional[AirbyteRecordMessageFileReference] = None,
29    ):
30        self._data = data
31        self._associated_slice = associated_slice
32        self.stream_name = stream_name
33        self._file_reference = file_reference
34
35    @property
36    def data(self) -> Mapping[str, Any]:
37        return self._data
38
39    @property
40    def associated_slice(self) -> Optional[StreamSlice]:
41        return self._associated_slice
42
43    @property
44    def file_reference(self) -> AirbyteRecordMessageFileReference:
45        return self._file_reference
46
47    @file_reference.setter
48    def file_reference(self, value: AirbyteRecordMessageFileReference) -> None:
49        self._file_reference = value
50
51    def __repr__(self) -> str:
52        return repr(self._data)
53
54    def __getitem__(self, key: str) -> Any:
55        return self._data[key]
56
57    def __len__(self) -> int:
58        return len(self._data)
59
60    def __iter__(self) -> Any:
61        return iter(self._data)
62
63    def __contains__(self, item: object) -> bool:
64        return item in self._data
65
66    def __eq__(self, other: object) -> bool:
67        if isinstance(other, Record):
68            # noinspection PyProtectedMember
69            return self._data == other._data
70        return False
71
72    def __ne__(self, other: object) -> bool:
73        return not self.__eq__(other)

A Mapping is a generic container for associating key/value pairs.

This class provides concrete generic implementations of all methods except for __getitem__, __iter__, and __len__.

Record( data: Mapping[str, Any], stream_name: str, associated_slice: Optional[StreamSlice] = None, file_reference: Optional[airbyte_protocol_dataclasses.models.airbyte_protocol.AirbyteRecordMessageFileReference] = None)
23    def __init__(
24        self,
25        data: Mapping[str, Any],
26        stream_name: str,
27        associated_slice: Optional[StreamSlice] = None,
28        file_reference: Optional[AirbyteRecordMessageFileReference] = None,
29    ):
30        self._data = data
31        self._associated_slice = associated_slice
32        self.stream_name = stream_name
33        self._file_reference = file_reference
stream_name
data: Mapping[str, Any]
35    @property
36    def data(self) -> Mapping[str, Any]:
37        return self._data
associated_slice: Optional[StreamSlice]
39    @property
40    def associated_slice(self) -> Optional[StreamSlice]:
41        return self._associated_slice
file_reference: airbyte_protocol_dataclasses.models.airbyte_protocol.AirbyteRecordMessageFileReference
43    @property
44    def file_reference(self) -> AirbyteRecordMessageFileReference:
45        return self._file_reference