airbyte_cdk
Welcome to the Airbyte Python CDK!
The Airbyte Python CDK is a Python library that provides a set of tools to help you build connectors for the Airbyte platform.
Building Source Connectors
To build a source connector, you will want to refer to the following classes and modules:
airbyte_cdk.sourcesairbyte_cdk.sources.concurrent_sourceairbyte_cdk.sources.configairbyte_cdk.sources.file_basedairbyte_cdk.sources.streams
Building Destination Connectors
To build a destination connector, you will want to refer to the following classes and modules:
airbyte_cdk.destinationsairbyte_cdk.destinations.Destinationairbyte_cdk.destinations.vector_db_based
Working with Airbyte Protocol Models
The Airbyte CDK provides a set of classes that help you work with the Airbyte protocol models:
Using the CLI (airbyte_cdk.cli)
The Airbyte CDK provides two command-line interfaces (CLIs) for interacting with the framework.
airbyte-cdk: This is the main CLI for the Airbyte CDK. It provides commands for building and testing connectors, as well as other utilities. See theairbyte_cdk.cli.airbyte_cdkmodule for more details.source-declarative-manifest: This command allows you to run declarative manifests directly. See theairbyte_cdk.cli.source_declarative_manifestmodule for more details.
API Reference
1# Copyright (c) 2021 Airbyte, Inc., all rights reserved. 2""" 3# Welcome to the Airbyte Python CDK! 4 5The Airbyte Python CDK is a Python library that provides a set of tools to help you build 6connectors for the Airbyte platform. 7 8## Building Source Connectors 9 10To build a source connector, you will want to refer to 11the following classes and modules: 12 13- `airbyte_cdk.sources` 14- `airbyte_cdk.sources.concurrent_source` 15- `airbyte_cdk.sources.config` 16- `airbyte_cdk.sources.file_based` 17- `airbyte_cdk.sources.streams` 18 19## Building Destination Connectors 20 21To build a destination connector, you will want to refer to 22the following classes and modules: 23 24- `airbyte_cdk.destinations` 25- `airbyte_cdk.destinations.Destination` 26- `airbyte_cdk.destinations.vector_db_based` 27 28## Working with Airbyte Protocol Models 29 30The Airbyte CDK provides a set of classes that help you work with the Airbyte protocol models: 31 32- `airbyte_cdk.models.airbyte_protocol` 33- `airbyte_cdk.models.airbyte_protocol_serializers` 34 35## Using the CLI (`airbyte_cdk.cli`) 36 37The Airbyte CDK provides two command-line interfaces (CLIs) for interacting with the framework. 38 39- `airbyte-cdk`: This is the main CLI for the Airbyte CDK. It provides commands for building 40 and testing connectors, as well as other utilities. See the `airbyte_cdk.cli.airbyte_cdk` module 41 for more details. 42- `source-declarative-manifest`: This command allows you to run declarative manifests directly. 43 See the `airbyte_cdk.cli.source_declarative_manifest` module for more details. 44 45--- 46 47API Reference 48 49--- 50 51""" 52 53# Warning: The below imports are not stable and will cause circular 54# dependencies if auto-sorted with isort. Please keep them in the same order. 55# TODO: Submodules should import from lower-level modules, rather than importing from here. 56# Imports should also be placed in `if TYPE_CHECKING` blocks if they are only used as type 57# hints - again, to avoid circular dependencies. 58# Once those issues are resolved, the below can be sorted with isort. 59import dunamai as _dunamai 60 61from .config_observation import ( 62 create_connector_config_control_message, 63 emit_configuration_as_airbyte_control_message, 64) 65from .connector import BaseConnector, Connector 66from .destinations import Destination 67from .entrypoint import AirbyteEntrypoint, launch 68from .legacy.sources.declarative.declarative_stream import DeclarativeStream 69from .legacy.sources.declarative.incremental import DatetimeBasedCursor 70from .logger import AirbyteLogFormatter, init_logger 71from .models import ( 72 AdvancedAuth, 73 AirbyteConnectionStatus, 74 AirbyteLogMessage, 75 AirbyteMessage, 76 AirbyteRecordMessage, 77 AirbyteStream, 78 ConfiguredAirbyteCatalog, 79 ConfiguredAirbyteStream, 80 ConnectorSpecification, 81 DestinationSyncMode, 82 FailureType, 83 Level, 84 OAuthConfigSpecification, 85 OrchestratorType, 86 Status, 87 SyncMode, 88 Type, 89) 90from .sources import AbstractSource, Source 91from .sources.concurrent_source.concurrent_source import ConcurrentSource 92from .sources.concurrent_source.concurrent_source_adapter import ConcurrentSourceAdapter 93from .sources.config import BaseConfig 94from .sources.connector_state_manager import ConnectorStateManager 95from .sources.declarative.auth import DeclarativeOauth2Authenticator 96from .sources.declarative.auth.declarative_authenticator import DeclarativeAuthenticator, NoAuth 97from .sources.declarative.auth.oauth import DeclarativeSingleUseRefreshTokenOauth2Authenticator 98from .sources.declarative.auth.token import ( 99 ApiKeyAuthenticator, 100 BasicHttpAuthenticator, 101 BearerAuthenticator, 102) 103from .sources.declarative.datetime.min_max_datetime import MinMaxDatetime 104from .sources.declarative.decoders import Decoder, JsonDecoder 105from .sources.declarative.exceptions import ReadException 106from .sources.declarative.extractors import ( 107 CombinedExtractor, 108 CombineMode, 109 DpathExtractor, 110 RecordSelector, 111) 112from .sources.declarative.extractors.record_extractor import RecordExtractor 113from .sources.declarative.extractors.record_filter import RecordFilter 114from .sources.declarative.interpolation import InterpolatedBoolean, InterpolatedString 115from .sources.declarative.migrations.legacy_to_per_partition_state_migration import ( 116 LegacyToPerPartitionStateMigration, 117) 118from .sources.declarative.partition_routers import ( 119 CartesianProductStreamSlicer, 120 SinglePartitionRouter, 121 SubstreamPartitionRouter, 122 UnionPartitionRouter, 123) 124from .sources.declarative.partition_routers.substream_partition_router import ParentStreamConfig 125from .sources.declarative.requesters import HttpRequester, Requester 126from .sources.declarative.requesters.error_handlers import BackoffStrategy 127from .sources.declarative.requesters.paginators import DefaultPaginator, PaginationStrategy 128from .sources.declarative.requesters.paginators.strategies import ( 129 CursorPaginationStrategy, 130 OffsetIncrement, 131 PageIncrement, 132 StopConditionPaginationStrategyDecorator, 133) 134from .sources.declarative.requesters.request_option import RequestOption, RequestOptionType 135from .sources.declarative.requesters.request_options.default_request_options_provider import ( 136 DefaultRequestOptionsProvider, 137) 138from .sources.declarative.requesters.request_options.interpolated_request_input_provider import ( 139 InterpolatedRequestInputProvider, 140) 141from .sources.declarative.requesters.requester import HttpMethod 142from .sources.declarative.retrievers import SimpleRetriever 143from .sources.declarative.schema import JsonFileSchemaLoader 144from .sources.declarative.transformations.add_fields import AddedFieldDefinition, AddFields 145from .sources.declarative.transformations.transformation import RecordTransformation 146from .sources.declarative.types import FieldPointer 147from .sources.declarative.yaml_declarative_source import YamlDeclarativeSource 148from .sources.message import InMemoryMessageRepository, MessageRepository 149from .sources.source import TState 150from .sources.streams.availability_strategy import AvailabilityStrategy 151from .sources.streams.call_rate import ( 152 AbstractAPIBudget, 153 CachedLimiterSession, 154 HttpAPIBudget, 155 HttpRequestMatcher, 156 LimiterSession, 157 MovingWindowCallRatePolicy, 158 Rate, 159) 160from .sources.streams.checkpoint import Cursor as LegacyCursor 161from .sources.streams.checkpoint import ResumableFullRefreshCursor 162from .sources.streams.concurrent.adapters import StreamFacade 163from .sources.streams.concurrent.cursor import ( 164 ConcurrentCursor, 165 Cursor, 166 CursorField, 167 FinalStateCursor, 168) 169from .sources.streams.concurrent.state_converters.datetime_stream_state_converter import ( 170 EpochValueConcurrentStreamStateConverter, 171 IsoMillisConcurrentStreamStateConverter, 172) 173from .sources.streams.core import IncrementalMixin, Stream, package_name_from_class 174from .sources.streams.http import HttpStream, HttpSubStream 175from .sources.streams.http.availability_strategy import HttpAvailabilityStrategy 176from .sources.streams.http.exceptions import ( 177 BaseBackoffException, 178 DefaultBackoffException, 179 UserDefinedBackoffException, 180) 181from .sources.streams.http.rate_limiting import default_backoff_handler 182from .sources.streams.http.requests_native_auth import ( 183 Oauth2Authenticator, 184 SingleUseRefreshTokenOauth2Authenticator, 185 TokenAuthenticator, 186) 187from .sources.streams.http.requests_native_auth.abstract_token import AbstractHeaderAuthenticator 188from .sources.types import Config, Record, StreamSlice 189from .sources.utils import casing 190from .sources.utils.schema_helpers import ( 191 InternalConfig, 192 ResourceSchemaLoader, 193 check_config_against_spec_or_exit, 194 expand_refs, 195 split_config, 196) 197from .sources.utils.transform import TransformConfig, TypeTransformer 198from .utils import AirbyteTracedException, is_cloud_environment 199from .utils.constants import ENV_REQUEST_CACHE_PATH 200from .utils.event_timing import create_timer 201from .utils.oneof_option_config import OneOfOptionConfig 202from .utils.spec_schema_transformations import resolve_refs 203from .utils.stream_status_utils import as_airbyte_message 204 205__all__ = [ 206 # Availability strategy 207 "AvailabilityStrategy", 208 "HttpAvailabilityStrategy", 209 # Checkpoint 210 "LegacyCursor", 211 "ResumableFullRefreshCursor", 212 # Concurrent 213 "ConcurrentCursor", 214 "ConcurrentSource", 215 "ConcurrentSourceAdapter", 216 "Cursor", 217 "CursorField", 218 "DEFAULT_CONCURRENCY", 219 "EpochValueConcurrentStreamStateConverter", 220 "FinalStateCursor", 221 "IsoMillisConcurrentStreamStateConverter", 222 "StreamFacade", 223 # Config observation 224 "create_connector_config_control_message", 225 "emit_configuration_as_airbyte_control_message", 226 # Connector 227 "AbstractSource", 228 "BaseConfig", 229 "BaseConnector", 230 "Connector", 231 "Destination", 232 "Source", 233 "TState", 234 # Declarative 235 "AddFields", 236 "AddedFieldDefinition", 237 "ApiKeyAuthenticator", 238 "BackoffStrategy", 239 "BasicHttpAuthenticator", 240 "BearerAuthenticator", 241 "CartesianProductStreamSlicer", 242 "CombineMode", 243 "CombinedExtractor", 244 "CursorPaginationStrategy", 245 "DatetimeBasedCursor", 246 "DeclarativeAuthenticator", 247 "DeclarativeOauth2Authenticator", 248 "DeclarativeSingleUseRefreshTokenOauth2Authenticator", 249 "DeclarativeStream", 250 "Decoder", 251 "DefaultPaginator", 252 "DefaultRequestOptionsProvider", 253 "DpathExtractor", 254 "FieldPointer", 255 "HttpMethod", 256 "HttpRequester", 257 "InterpolatedBoolean", 258 "InterpolatedRequestInputProvider", 259 "InterpolatedString", 260 "JsonDecoder", 261 "JsonFileSchemaLoader", 262 "LegacyToPerPartitionStateMigration", 263 "MinMaxDatetime", 264 "NoAuth", 265 "OffsetIncrement", 266 "PageIncrement", 267 "PaginationStrategy", 268 "ParentStreamConfig", 269 "ReadException", 270 "RecordExtractor", 271 "RecordFilter", 272 "RecordSelector", 273 "RecordTransformation", 274 "RequestOption", 275 "RequestOptionType", 276 "Requester", 277 "ResponseStatus", 278 "SimpleRetriever", 279 "SinglePartitionRouter", 280 "StopConditionPaginationStrategyDecorator", 281 "StreamSlice", 282 "SubstreamPartitionRouter", 283 "UnionPartitionRouter", 284 "YamlDeclarativeSource", 285 # Entrypoint 286 "launch", 287 "AirbyteEntrypoint", 288 # HTTP 289 "AbstractAPIBudget", 290 "AbstractHeaderAuthenticator", 291 "BaseBackoffException", 292 "CachedLimiterSession", 293 "DefaultBackoffException", 294 "default_backoff_handler", 295 "HttpAPIBudget", 296 "HttpAuthenticator", 297 "HttpRequestMatcher", 298 "HttpStream", 299 "HttpSubStream", 300 "LimiterSession", 301 "MovingWindowCallRatePolicy", 302 "MultipleTokenAuthenticator", 303 "Oauth2Authenticator", 304 "Rate", 305 "SingleUseRefreshTokenOauth2Authenticator", 306 "TokenAuthenticator", 307 "UserDefinedBackoffException", 308 # Logger 309 "AirbyteLogFormatter", 310 "init_logger", 311 # Protocol classes 312 "AirbyteStream", 313 "AirbyteConnectionStatus", 314 "AirbyteMessage", 315 "ConfiguredAirbyteCatalog", 316 "Status", 317 "Type", 318 "OrchestratorType", 319 "ConfiguredAirbyteStream", 320 "DestinationSyncMode", 321 "SyncMode", 322 "FailureType", 323 "AdvancedAuth", 324 "AirbyteLogMessage", 325 "OAuthConfigSpecification", 326 "ConnectorSpecification", 327 "Level", 328 "AirbyteRecordMessage", 329 # Repository 330 "InMemoryMessageRepository", 331 "MessageRepository", 332 # State management 333 "ConnectorStateManager", 334 # Stream 335 "IncrementalMixin", 336 "Stream", 337 "StreamData", 338 "package_name_from_class", 339 # Utils 340 "AirbyteTracedException", 341 "is_cloud_environment", 342 "casing", 343 "InternalConfig", 344 "ResourceSchemaLoader", 345 "check_config_against_spec_or_exit", 346 "split_config", 347 "TransformConfig", 348 "TypeTransformer", 349 "ENV_REQUEST_CACHE_PATH", 350 "create_timer", 351 "OneOfOptionConfig", 352 "resolve_refs", 353 "as_airbyte_message", 354 # Types 355 "Config", 356 "Record", 357 "Source", 358 "StreamSlice", 359] 360 361__version__: str 362"""Version generated by poetry dynamic versioning during publish. 363 364When running in development, dunamai will calculate a new prerelease version 365from existing git release tag info. 366""" 367 368try: 369 __version__ = _dunamai.get_version( 370 "airbyte-cdk", 371 third_choice=_dunamai.Version.from_any_vcs, 372 fallback=_dunamai.Version("0.0.0+dev"), 373 ).serialize() 374except: 375 __version__ = "0.0.0+dev"
19class AvailabilityStrategy(ABC): 20 """ 21 Abstract base class for checking stream availability. 22 """ 23 24 @abstractmethod 25 def check_availability( 26 self, stream: Stream, logger: logging.Logger, source: Optional["Source"] = None 27 ) -> Tuple[bool, Optional[str]]: 28 """ 29 Checks stream availability. 30 31 :param stream: stream 32 :param logger: source logger 33 :param source: (optional) source 34 :return: A tuple of (boolean, str). If boolean is true, then the stream 35 is available, and no str is required. Otherwise, the stream is unavailable 36 for some reason and the str should describe what went wrong and how to 37 resolve the unavailability, if possible. 38 """ 39 40 @staticmethod 41 def get_first_stream_slice(stream: Stream) -> Optional[Mapping[str, Any]]: 42 """ 43 Gets the first stream_slice from a given stream's stream_slices. 44 :param stream: stream 45 :raises StopIteration: if there is no first slice to return (the stream_slices generator is empty) 46 :return: first stream slice from 'stream_slices' generator (`None` is a valid stream slice) 47 """ 48 # We wrap the return output of stream_slices() because some implementations return types that are iterable, 49 # but not iterators such as lists or tuples 50 slices = iter( 51 stream.stream_slices( 52 cursor_field=stream.cursor_field, # type: ignore[arg-type] 53 sync_mode=SyncMode.full_refresh, 54 ) 55 ) 56 return next(slices) 57 58 @staticmethod 59 def get_first_record_for_slice( 60 stream: Stream, stream_slice: Optional[Mapping[str, Any]] 61 ) -> StreamData: 62 """ 63 Gets the first record for a stream_slice of a stream. 64 65 :param stream: stream instance from which to read records 66 :param stream_slice: stream_slice parameters for slicing the stream 67 :raises StopIteration: if there is no first record to return (the read_records generator is empty) 68 :return: StreamData containing the first record in the slice 69 """ 70 # Store the original value of exit_on_rate_limit 71 original_exit_on_rate_limit = stream.exit_on_rate_limit 72 73 try: 74 # Ensure exit_on_rate_limit is safely set to True if possible 75 stream.exit_on_rate_limit = True 76 77 # We wrap the return output of read_records() because some implementations return types that are iterable, 78 # but not iterators such as lists or tuples 79 records_for_slice = iter( 80 stream.read_records(sync_mode=SyncMode.full_refresh, stream_slice=stream_slice) 81 ) 82 83 return next(records_for_slice) 84 finally: 85 # Restore the original exit_on_rate_limit value 86 stream.exit_on_rate_limit = original_exit_on_rate_limit
Abstract base class for checking stream availability.
24 @abstractmethod 25 def check_availability( 26 self, stream: Stream, logger: logging.Logger, source: Optional["Source"] = None 27 ) -> Tuple[bool, Optional[str]]: 28 """ 29 Checks stream availability. 30 31 :param stream: stream 32 :param logger: source logger 33 :param source: (optional) source 34 :return: A tuple of (boolean, str). If boolean is true, then the stream 35 is available, and no str is required. Otherwise, the stream is unavailable 36 for some reason and the str should describe what went wrong and how to 37 resolve the unavailability, if possible. 38 """
Checks stream availability.
Parameters
- stream: stream
- logger: source logger
- source: (optional) source
Returns
A tuple of (boolean, str). If boolean is true, then the stream is available, and no str is required. Otherwise, the stream is unavailable for some reason and the str should describe what went wrong and how to resolve the unavailability, if possible.
40 @staticmethod 41 def get_first_stream_slice(stream: Stream) -> Optional[Mapping[str, Any]]: 42 """ 43 Gets the first stream_slice from a given stream's stream_slices. 44 :param stream: stream 45 :raises StopIteration: if there is no first slice to return (the stream_slices generator is empty) 46 :return: first stream slice from 'stream_slices' generator (`None` is a valid stream slice) 47 """ 48 # We wrap the return output of stream_slices() because some implementations return types that are iterable, 49 # but not iterators such as lists or tuples 50 slices = iter( 51 stream.stream_slices( 52 cursor_field=stream.cursor_field, # type: ignore[arg-type] 53 sync_mode=SyncMode.full_refresh, 54 ) 55 ) 56 return next(slices)
Gets the first stream_slice from a given stream's stream_slices.
Parameters
- stream: stream
Raises
- StopIteration: if there is no first slice to return (the stream_slices generator is empty)
Returns
first stream slice from 'stream_slices' generator (
Noneis a valid stream slice)
58 @staticmethod 59 def get_first_record_for_slice( 60 stream: Stream, stream_slice: Optional[Mapping[str, Any]] 61 ) -> StreamData: 62 """ 63 Gets the first record for a stream_slice of a stream. 64 65 :param stream: stream instance from which to read records 66 :param stream_slice: stream_slice parameters for slicing the stream 67 :raises StopIteration: if there is no first record to return (the read_records generator is empty) 68 :return: StreamData containing the first record in the slice 69 """ 70 # Store the original value of exit_on_rate_limit 71 original_exit_on_rate_limit = stream.exit_on_rate_limit 72 73 try: 74 # Ensure exit_on_rate_limit is safely set to True if possible 75 stream.exit_on_rate_limit = True 76 77 # We wrap the return output of read_records() because some implementations return types that are iterable, 78 # but not iterators such as lists or tuples 79 records_for_slice = iter( 80 stream.read_records(sync_mode=SyncMode.full_refresh, stream_slice=stream_slice) 81 ) 82 83 return next(records_for_slice) 84 finally: 85 # Restore the original exit_on_rate_limit value 86 stream.exit_on_rate_limit = original_exit_on_rate_limit
Gets the first record for a stream_slice of a stream.
Parameters
- stream: stream instance from which to read records
- stream_slice: stream_slice parameters for slicing the stream
Raises
- StopIteration: if there is no first record to return (the read_records generator is empty)
Returns
StreamData containing the first record in the slice
18class HttpAvailabilityStrategy(AvailabilityStrategy): 19 def check_availability( 20 self, stream: Stream, logger: logging.Logger, source: Optional["Source"] = None 21 ) -> Tuple[bool, Optional[str]]: 22 """ 23 Check stream availability by attempting to read the first record of the 24 stream. 25 26 :param stream: stream 27 :param logger: source logger 28 :param source: (optional) source 29 :return: A tuple of (boolean, str). If boolean is true, then the stream 30 is available, and no str is required. Otherwise, the stream is unavailable 31 for some reason and the str should describe what went wrong and how to 32 resolve the unavailability, if possible. 33 """ 34 reason: Optional[str] 35 try: 36 # Some streams need a stream slice to read records (e.g. if they have a SubstreamPartitionRouter) 37 # Streams that don't need a stream slice will return `None` as their first stream slice. 38 stream_slice = self.get_first_stream_slice(stream) 39 except StopIteration: 40 # If stream_slices has no `next()` item (Note - this is different from stream_slices returning [None]!) 41 # This can happen when a substream's `stream_slices` method does a `for record in parent_records: yield <something>` 42 # without accounting for the case in which the parent stream is empty. 43 reason = f"Cannot attempt to connect to stream {stream.name} - no stream slices were found, likely because the parent stream is empty." 44 return False, reason 45 except AirbyteTracedException as error: 46 return False, error.message 47 48 try: 49 self.get_first_record_for_slice(stream, stream_slice) 50 return True, None 51 except StopIteration: 52 logger.info(f"Successfully connected to stream {stream.name}, but got 0 records.") 53 return True, None 54 except AirbyteTracedException as error: 55 return False, error.message
Abstract base class for checking stream availability.
19 def check_availability( 20 self, stream: Stream, logger: logging.Logger, source: Optional["Source"] = None 21 ) -> Tuple[bool, Optional[str]]: 22 """ 23 Check stream availability by attempting to read the first record of the 24 stream. 25 26 :param stream: stream 27 :param logger: source logger 28 :param source: (optional) source 29 :return: A tuple of (boolean, str). If boolean is true, then the stream 30 is available, and no str is required. Otherwise, the stream is unavailable 31 for some reason and the str should describe what went wrong and how to 32 resolve the unavailability, if possible. 33 """ 34 reason: Optional[str] 35 try: 36 # Some streams need a stream slice to read records (e.g. if they have a SubstreamPartitionRouter) 37 # Streams that don't need a stream slice will return `None` as their first stream slice. 38 stream_slice = self.get_first_stream_slice(stream) 39 except StopIteration: 40 # If stream_slices has no `next()` item (Note - this is different from stream_slices returning [None]!) 41 # This can happen when a substream's `stream_slices` method does a `for record in parent_records: yield <something>` 42 # without accounting for the case in which the parent stream is empty. 43 reason = f"Cannot attempt to connect to stream {stream.name} - no stream slices were found, likely because the parent stream is empty." 44 return False, reason 45 except AirbyteTracedException as error: 46 return False, error.message 47 48 try: 49 self.get_first_record_for_slice(stream, stream_slice) 50 return True, None 51 except StopIteration: 52 logger.info(f"Successfully connected to stream {stream.name}, but got 0 records.") 53 return True, None 54 except AirbyteTracedException as error: 55 return False, error.message
Check stream availability by attempting to read the first record of the stream.
Parameters
- stream: stream
- logger: source logger
- source: (optional) source
Returns
A tuple of (boolean, str). If boolean is true, then the stream is available, and no str is required. Otherwise, the stream is unavailable for some reason and the str should describe what went wrong and how to resolve the unavailability, if possible.
Inherited Members
11@dataclass 12class ResumableFullRefreshCursor(Cursor): 13 """ 14 Cursor that allows for the checkpointing of sync progress according to a synthetic cursor based on the pagination state 15 of the stream. Resumable full refresh syncs are only intended to retain state in between sync attempts of the same job 16 with the platform responsible for removing said state. 17 """ 18 19 def __init__(self) -> None: 20 self._cursor: StreamState = {} 21 22 def get_stream_state(self) -> StreamState: 23 return self._cursor 24 25 def set_initial_state(self, stream_state: StreamState) -> None: 26 self._cursor = stream_state 27 28 def observe(self, stream_slice: StreamSlice, record: Record) -> None: 29 """ 30 Resumable full refresh manages state using a page number so it does not need to update state by observing incoming records. 31 """ 32 pass 33 34 def close_slice(self, stream_slice: StreamSlice, *args: Any) -> None: 35 self._cursor = stream_slice.cursor_slice 36 37 def should_be_synced(self, record: Record) -> bool: 38 """ 39 Unlike date-based cursors which filter out records outside slice boundaries, resumable full refresh records exist within pages 40 that don't have filterable bounds. We should always return them. 41 """ 42 return True 43 44 def select_state(self, stream_slice: Optional[StreamSlice] = None) -> Optional[StreamState]: 45 # A top-level RFR cursor only manages the state of a single partition 46 return self._cursor
Cursor that allows for the checkpointing of sync progress according to a synthetic cursor based on the pagination state of the stream. Resumable full refresh syncs are only intended to retain state in between sync attempts of the same job with the platform responsible for removing said state.
Returns the current stream state. We would like to restrict it's usage since it does expose internal of state. As of 2023-06-14, it is used for two things:
- Interpolation of the requests
- Transformation of records
- Saving the state
For the first case, we are probably stuck with exposing the stream state. For the second, we can probably expose a method that allows for emitting the state to the platform.
Cursors are not initialized with their state. As state is needed in order to function properly, this method should be called before calling anything else
Parameters
- stream_state: The state of the stream as returned by get_stream_state
28 def observe(self, stream_slice: StreamSlice, record: Record) -> None: 29 """ 30 Resumable full refresh manages state using a page number so it does not need to update state by observing incoming records. 31 """ 32 pass
Resumable full refresh manages state using a page number so it does not need to update state by observing incoming records.
34 def close_slice(self, stream_slice: StreamSlice, *args: Any) -> None: 35 self._cursor = stream_slice.cursor_slice
Update state based on the stream slice. Note that stream_slice.cursor_slice and most_recent_record.associated_slice are expected
to be the same but we make it explicit here that stream_slice should be leveraged to update the state. We do not pass in the
latest record, since cursor instances should maintain the relevant internal state on their own.
Parameters
- stream_slice: slice to close
37 def should_be_synced(self, record: Record) -> bool: 38 """ 39 Unlike date-based cursors which filter out records outside slice boundaries, resumable full refresh records exist within pages 40 that don't have filterable bounds. We should always return them. 41 """ 42 return True
Unlike date-based cursors which filter out records outside slice boundaries, resumable full refresh records exist within pages that don't have filterable bounds. We should always return them.
44 def select_state(self, stream_slice: Optional[StreamSlice] = None) -> Optional[StreamState]: 45 # A top-level RFR cursor only manages the state of a single partition 46 return self._cursor
Get the state value of a specific stream_slice. For incremental or resumable full refresh cursors which only manage state in a single dimension this is the entire state object. For per-partition cursors used by substreams, this returns the state of a specific parent delineated by the incoming slice's partition object.
179class ConcurrentCursor(Cursor): 180 _START_BOUNDARY = 0 181 _END_BOUNDARY = 1 182 183 def copy_without_state(self) -> "ConcurrentCursor": 184 return self.__class__( 185 stream_name=self._stream_name, 186 stream_namespace=self._stream_namespace, 187 stream_state={}, 188 message_repository=NoopMessageRepository(), 189 connector_state_manager=ConnectorStateManager(), 190 connector_state_converter=self._connector_state_converter, 191 cursor_field=self._cursor_field, 192 slice_boundary_fields=self._slice_boundary_fields, 193 start=self._start, 194 end_provider=self._end_provider, 195 lookback_window=self._lookback_window, 196 slice_range=self._slice_range, 197 cursor_granularity=self._cursor_granularity, 198 clamping_strategy=self._clamping_strategy, 199 is_compare_strictly=self._is_compare_strictly, 200 ) 201 202 def __init__( 203 self, 204 stream_name: str, 205 stream_namespace: Optional[str], 206 stream_state: Any, 207 message_repository: MessageRepository, 208 connector_state_manager: ConnectorStateManager, 209 connector_state_converter: AbstractStreamStateConverter, 210 cursor_field: CursorField, 211 slice_boundary_fields: Optional[Tuple[str, str]], 212 start: Optional[CursorValueType], 213 end_provider: Callable[[], CursorValueType], 214 lookback_window: Optional[GapType] = None, 215 slice_range: Optional[GapType] = None, 216 cursor_granularity: Optional[GapType] = None, 217 clamping_strategy: ClampingStrategy = NoClamping(), 218 is_compare_strictly: bool = False, 219 ) -> None: 220 self._stream_name = stream_name 221 self._stream_namespace = stream_namespace 222 self._message_repository = message_repository 223 self._connector_state_converter = connector_state_converter 224 self._connector_state_manager = connector_state_manager 225 self._cursor_field = cursor_field 226 # To see some example where the slice boundaries might not be defined, check https://github.com/airbytehq/airbyte/blob/1ce84d6396e446e1ac2377362446e3fb94509461/airbyte-integrations/connectors/source-stripe/source_stripe/streams.py#L363-L379 227 self._slice_boundary_fields = slice_boundary_fields 228 self._start = start 229 self._end_provider = end_provider 230 self.start, self._concurrent_state = self._get_concurrent_state(stream_state) 231 self._lookback_window = lookback_window 232 self._slice_range = slice_range 233 self._most_recent_cursor_value_per_partition: MutableMapping[ 234 Union[StreamSlice, Mapping[str, Any], None], Any 235 ] = {} 236 self._has_closed_at_least_one_slice = False 237 self._cursor_granularity = cursor_granularity 238 # Flag to track if the logger has been triggered (per stream) 239 self._should_be_synced_logger_triggered = False 240 self._clamping_strategy = clamping_strategy 241 self._is_compare_strictly = is_compare_strictly 242 self._is_ascending_order = True 243 244 # A lock is required when closing a partition because updating the cursor's concurrent_state is 245 # not thread safe. When multiple partitions are being closed by the cursor at the same time, it is 246 # possible for one partition to update concurrent_state after a second partition has already read 247 # the previous state. This can lead to the second partition overwriting the previous one's state. 248 self._lock = threading.Lock() 249 250 @property 251 def state(self) -> MutableMapping[str, Any]: 252 return self._connector_state_converter.convert_to_state_message( 253 self.cursor_field, self._concurrent_state 254 ) 255 256 @property 257 def cursor_field(self) -> CursorField: 258 return self._cursor_field 259 260 @property 261 def _slice_boundary_fields_wrapper(self) -> Tuple[str, str]: 262 return ( 263 self._slice_boundary_fields 264 if self._slice_boundary_fields 265 else ( 266 self._connector_state_converter.START_KEY, 267 self._connector_state_converter.END_KEY, 268 ) 269 ) 270 271 def _get_concurrent_state( 272 self, state: MutableMapping[str, Any] 273 ) -> Tuple[CursorValueType, MutableMapping[str, Any]]: 274 if self._connector_state_converter.is_state_message_compatible(state): 275 partitioned_state = self._connector_state_converter.deserialize(state) 276 slices_from_partitioned_state = partitioned_state.get("slices", []) 277 278 value_from_partitioned_state = None 279 if slices_from_partitioned_state: 280 # We assume here that the slices have been already merged 281 first_slice = slices_from_partitioned_state[0] 282 value_from_partitioned_state = ( 283 first_slice[self._connector_state_converter.MOST_RECENT_RECORD_KEY] 284 if self._connector_state_converter.MOST_RECENT_RECORD_KEY in first_slice 285 else first_slice[self._connector_state_converter.END_KEY] 286 ) 287 return ( 288 value_from_partitioned_state 289 or self._start 290 or self._connector_state_converter.zero_value, 291 partitioned_state, 292 ) 293 return self._connector_state_converter.convert_from_sequential_state( 294 self._cursor_field, state, self._start 295 ) 296 297 def observe(self, record: Record) -> None: 298 # Because observe writes to the most_recent_cursor_value_per_partition mapping, 299 # it is not thread-safe. However, this shouldn't lead to concurrency issues because 300 # observe() is only invoked by PartitionReader.process_partition(). Since the map is 301 # broken down according to partition, concurrent threads processing only read/write 302 # from different keys which avoids any conflicts. 303 # 304 # If we were to add thread safety, we should implement a lock per-partition 305 # which is instantiated during stream_slices() 306 most_recent_cursor_value = self._most_recent_cursor_value_per_partition.get( 307 record.associated_slice 308 ) 309 try: 310 cursor_value = self._extract_cursor_value(record) 311 312 if most_recent_cursor_value is None or most_recent_cursor_value < cursor_value: 313 self._most_recent_cursor_value_per_partition[record.associated_slice] = cursor_value 314 elif most_recent_cursor_value > cursor_value: 315 self._is_ascending_order = False 316 except ValueError: 317 self._log_for_record_without_cursor_value() 318 319 def _extract_cursor_value(self, record: Record) -> Any: 320 return self._connector_state_converter.parse_value(self._cursor_field.extract_value(record)) 321 322 def close_partition(self, partition: Partition) -> None: 323 with self._lock: 324 slice_count_before = len(self._concurrent_state.get("slices", [])) 325 self._add_slice_to_state(partition) 326 if slice_count_before < len( 327 self._concurrent_state["slices"] 328 ): # only emit if at least one slice has been processed 329 self._merge_partitions() 330 self._emit_state_message() 331 self._has_closed_at_least_one_slice = True 332 333 def _add_slice_to_state(self, partition: Partition) -> None: 334 most_recent_cursor_value = self._most_recent_cursor_value_per_partition.get( 335 partition.to_slice() 336 ) 337 338 if self._slice_boundary_fields: 339 if "slices" not in self._concurrent_state: 340 raise RuntimeError( 341 f"The state for stream {self._stream_name} should have at least one slice to delineate the sync start time, but no slices are present. This is unexpected. Please contact Support." 342 ) 343 self._concurrent_state["slices"].append( 344 { 345 self._connector_state_converter.START_KEY: self._extract_from_slice( 346 partition, self._slice_boundary_fields[self._START_BOUNDARY] 347 ), 348 self._connector_state_converter.END_KEY: self._extract_from_slice( 349 partition, self._slice_boundary_fields[self._END_BOUNDARY] 350 ), 351 self._connector_state_converter.MOST_RECENT_RECORD_KEY: most_recent_cursor_value, 352 } 353 ) 354 elif most_recent_cursor_value: 355 if self._has_closed_at_least_one_slice: 356 # If we track state value using records cursor field, we can only do that if there is one partition. This is because we save 357 # the state every time we close a partition. We assume that if there are multiple slices, they need to be providing 358 # boundaries. There are cases where partitions could not have boundaries: 359 # * The cursor should be per-partition 360 # * The stream state is actually the parent stream state 361 # There might be other cases not listed above. Those are not supported today hence the stream should not use this cursor for 362 # state management. For the specific user that was affected with this issue, we need to: 363 # * Fix state tracking (which is currently broken) 364 # * Make the new version available 365 # * (Probably) ask the user to reset the stream to avoid data loss 366 raise ValueError( 367 "Given that slice_boundary_fields is not defined and that per-partition state is not supported, only one slice is " 368 "expected. Please contact the Airbyte team." 369 ) 370 371 self._concurrent_state["slices"].append( 372 { 373 self._connector_state_converter.START_KEY: self.start, 374 self._connector_state_converter.END_KEY: most_recent_cursor_value, 375 self._connector_state_converter.MOST_RECENT_RECORD_KEY: most_recent_cursor_value, 376 } 377 ) 378 379 def _emit_state_message(self) -> None: 380 self._connector_state_manager.update_state_for_stream( 381 self._stream_name, 382 self._stream_namespace, 383 self.state, 384 ) 385 state_message = self._connector_state_manager.create_state_message( 386 self._stream_name, self._stream_namespace 387 ) 388 self._message_repository.emit_message(state_message) 389 390 def _merge_partitions(self) -> None: 391 self._concurrent_state["slices"] = self._connector_state_converter.merge_intervals( 392 self._concurrent_state["slices"] 393 ) 394 395 def _extract_from_slice(self, partition: Partition, key: str) -> CursorValueType: 396 try: 397 _slice = partition.to_slice() 398 if not _slice: 399 raise KeyError(f"Could not find key `{key}` in empty slice") 400 return self._connector_state_converter.parse_value(_slice[key]) # type: ignore # we expect the devs to specify a key that would return a CursorValueType 401 except KeyError as exception: 402 raise KeyError( 403 f"Partition is expected to have key `{key}` but could not be found" 404 ) from exception 405 406 def ensure_at_least_one_state_emitted(self) -> None: 407 """ 408 The platform expect to have at least one state message on successful syncs. Hence, whatever happens, we expect this method to be 409 called. 410 """ 411 self._emit_state_message() 412 413 def stream_slices(self) -> Iterable[StreamSlice]: 414 """ 415 Generating slices based on a few parameters: 416 * lookback_window: Buffer to remove from END_KEY of the highest slice 417 * slice_range: Max difference between two slices. If the difference between two slices is greater, multiple slices will be created 418 * start: `_split_per_slice_range` will clip any value to `self._start which means that: 419 * if upper is less than self._start, no slices will be generated 420 * if lower is less than self._start, self._start will be used as the lower boundary (lookback_window will not be considered in that case) 421 422 Note that the slices will overlap at their boundaries. We therefore expect to have at least the lower or the upper boundary to be 423 inclusive in the API that is queried. 424 """ 425 self._merge_partitions() 426 427 if self._start is not None and self._is_start_before_first_slice(): 428 yield from self._split_per_slice_range( 429 self._start, 430 self._concurrent_state["slices"][0][self._connector_state_converter.START_KEY], 431 False, 432 ) 433 434 if len(self._concurrent_state["slices"]) == 1: 435 yield from self._split_per_slice_range( 436 self._calculate_lower_boundary_of_last_slice( 437 self._concurrent_state["slices"][0][self._connector_state_converter.END_KEY] 438 ), 439 self._end_provider(), 440 True, 441 ) 442 elif len(self._concurrent_state["slices"]) > 1: 443 for i in range(len(self._concurrent_state["slices"]) - 1): 444 if self._cursor_granularity: 445 yield from self._split_per_slice_range( 446 self._concurrent_state["slices"][i][self._connector_state_converter.END_KEY] 447 + self._cursor_granularity, 448 self._concurrent_state["slices"][i + 1][ 449 self._connector_state_converter.START_KEY 450 ], 451 False, 452 ) 453 else: 454 yield from self._split_per_slice_range( 455 self._concurrent_state["slices"][i][ 456 self._connector_state_converter.END_KEY 457 ], 458 self._concurrent_state["slices"][i + 1][ 459 self._connector_state_converter.START_KEY 460 ], 461 False, 462 ) 463 yield from self._split_per_slice_range( 464 self._calculate_lower_boundary_of_last_slice( 465 self._concurrent_state["slices"][-1][self._connector_state_converter.END_KEY] 466 ), 467 self._end_provider(), 468 True, 469 ) 470 else: 471 raise ValueError("Expected at least one slice") 472 473 def _is_start_before_first_slice(self) -> bool: 474 return ( 475 self._start is not None 476 and self._start 477 < self._concurrent_state["slices"][0][self._connector_state_converter.START_KEY] 478 ) 479 480 def _calculate_lower_boundary_of_last_slice( 481 self, lower_boundary: CursorValueType 482 ) -> CursorValueType: 483 if self._lookback_window: 484 return lower_boundary - self._lookback_window 485 return lower_boundary 486 487 def _split_per_slice_range( 488 self, lower: CursorValueType, upper: CursorValueType, upper_is_end: bool 489 ) -> Iterable[StreamSlice]: 490 if lower > upper or (lower == upper and not self._cursor_granularity): 491 return 492 493 if self._start and upper < self._start: 494 return 495 496 lower = max(lower, self._start) if self._start else lower 497 # Clamp first, else a step shorter than the clamping period collapses the first window 498 lower = self._clamping_strategy.clamp(lower) 499 # With a granularity the last window includes upper, so lower == upper is one unit to read, 500 # unless the API rejects start == end. In a gap, the next slice already reads upper. 501 if lower > upper or (lower == upper and (self._is_compare_strictly or not upper_is_end)): 502 return 503 if not self._slice_range or self._evaluate_upper_safely(lower, self._slice_range) >= upper: 504 # With a granularity upper is the last unit to read: clamping it would read past the end 505 clamped_upper = ( 506 upper 507 if upper_is_end and self._cursor_granularity 508 else self._clamping_strategy.clamp(upper) 509 ) 510 start_value, end_value = ( 511 (lower, clamped_upper - self._cursor_granularity) 512 if self._cursor_granularity and not upper_is_end 513 else (lower, clamped_upper) 514 ) 515 yield StreamSlice( 516 partition={}, 517 cursor_slice={ 518 self._slice_boundary_fields_wrapper[ 519 self._START_BOUNDARY 520 ]: self._connector_state_converter.output_format(start_value), 521 self._slice_boundary_fields_wrapper[ 522 self._END_BOUNDARY 523 ]: self._connector_state_converter.output_format(end_value), 524 }, 525 ) 526 else: 527 stop_processing = False 528 current_lower_boundary = lower 529 while not stop_processing: 530 current_upper_boundary = min( 531 self._evaluate_upper_safely(current_lower_boundary, self._slice_range), upper 532 ) 533 clamped_upper = ( 534 self._clamping_strategy.clamp(current_upper_boundary) 535 if current_upper_boundary != upper 536 else current_upper_boundary 537 ) 538 # A boundary clamped onto upper makes this the last window, else upper is dropped 539 has_reached_upper_boundary = ( 540 current_upper_boundary >= upper or clamped_upper == upper 541 ) 542 clamped_lower = self._clamping_strategy.clamp(current_lower_boundary) 543 if clamped_lower >= clamped_upper: 544 # clamping collapsed both values which means that it is time to stop processing 545 # FIXME should this be replace by proper end_provider 546 break 547 start_value, end_value = ( 548 (clamped_lower, clamped_upper - self._cursor_granularity) 549 if self._cursor_granularity 550 and (not upper_is_end or not has_reached_upper_boundary) 551 else (clamped_lower, clamped_upper) 552 ) 553 yield StreamSlice( 554 partition={}, 555 cursor_slice={ 556 self._slice_boundary_fields_wrapper[ 557 self._START_BOUNDARY 558 ]: self._connector_state_converter.output_format(start_value), 559 self._slice_boundary_fields_wrapper[ 560 self._END_BOUNDARY 561 ]: self._connector_state_converter.output_format(end_value), 562 }, 563 ) 564 current_lower_boundary = clamped_upper 565 if has_reached_upper_boundary: 566 stop_processing = True 567 568 def _evaluate_upper_safely(self, lower: CursorValueType, step: GapType) -> CursorValueType: 569 """ 570 Given that we set the default step at datetime.timedelta.max, we will generate an OverflowError when evaluating the next start_date 571 This method assumes that users would never enter a step that would generate an overflow. Given that would be the case, the code 572 would have broken anyway. 573 """ 574 try: 575 return lower + step 576 except OverflowError: 577 return self._end_provider() 578 579 def should_be_synced(self, record: Record) -> bool: 580 """ 581 Determines if a record should be synced based on its cursor value. 582 :param record: The record to evaluate 583 584 :return: True if the record's cursor value falls within the sync boundaries 585 """ 586 try: 587 record_cursor_value: CursorValueType = self._extract_cursor_value(record) 588 except ValueError: 589 self._log_for_record_without_cursor_value() 590 return True 591 return self._lowest_cursor_value_to_sync() <= record_cursor_value <= self._end_provider() 592 593 def _lowest_cursor_value_to_sync(self) -> CursorValueType: 594 # Keep the records the lookback window re-reads, never below the configured start. If the state is below the 595 # configured start, keep records from the state rather than raising the floor to the configured start. 596 if not self._lookback_window: 597 return self.start 598 try: 599 lower = self.start - self._lookback_window 600 except OverflowError: 601 return self.start 602 return max(lower, min(self.start, self._start)) if self._start else lower 603 604 def _log_for_record_without_cursor_value(self) -> None: 605 if not self._should_be_synced_logger_triggered: 606 LOGGER.warning( 607 f"Could not find cursor field `{self.cursor_field.cursor_field_key}` in record for stream {self._stream_name}. The incremental sync will assume it needs to be synced" 608 ) 609 self._should_be_synced_logger_triggered = True 610 611 def reduce_slice_range(self, stream_slice: StreamSlice) -> StreamSlice: 612 # In theory, we might be more flexible here meaning that it doesn't need to be in ascending order but it just 613 # needs to be ordered. For now though, we will only support ascending order. 614 if not self._is_ascending_order: 615 LOGGER.warning( 616 "Attempting to reduce slice while records are not returned in incremental order might lead to missing records" 617 ) 618 619 if stream_slice in self._most_recent_cursor_value_per_partition: 620 return StreamSlice( 621 partition=stream_slice.partition, 622 cursor_slice={ 623 self._slice_boundary_fields_wrapper[ 624 self._START_BOUNDARY 625 ]: self._connector_state_converter.output_format( 626 self._most_recent_cursor_value_per_partition[stream_slice] 627 ), 628 self._slice_boundary_fields_wrapper[ 629 self._END_BOUNDARY 630 ]: stream_slice.cursor_slice[ 631 self._slice_boundary_fields_wrapper[self._END_BOUNDARY] 632 ], 633 }, 634 extra_fields=stream_slice.extra_fields, 635 ) 636 else: 637 return stream_slice 638 639 def split_request_window( 640 self, stream_slice: StreamSlice, min_split_window: Optional[datetime.timedelta] = None 641 ) -> Optional[List[StreamSlice]]: 642 """ 643 Split `stream_slice` in half along this cursor's own boundary fields, granularity, and output format. 644 645 Returns `None` when the slice cannot be split any further: either it spans at most one 646 `cursor_granularity` unit already (the floor), it is already at or below `min_split_window` (a 647 connector-configured floor independent of and typically looser than `cursor_granularity`), the 648 boundaries are reversed, or no `cursor_granularity` was configured at all (there is then no way to know 649 the smallest addressable unit, or how to keep two children from overlapping at their shared edge). 650 Raises `AirbyteTracedException` instead if a boundary is missing or unparseable: that is a bug in how 651 the slice was produced, not the API rejecting a window, so it must not be reported as one. This is the 652 implementation `SimpleRetriever.request_window_splitter` binds to (see 653 `airbyte_cdk.sources.declarative.retrievers.simple_retriever`) that reuses the same 654 parsing/formatting/comparison operations `_split_per_slice_range` already relies on, so it works for 655 any `CursorValueType`/`GapType` pair this cursor was built with (datetime/timedelta, or int/int), 656 without a second, duplicate implementation of that logic living outside the cursor. 657 """ 658 if not self._cursor_granularity: 659 return None 660 661 start_field, end_field = self._slice_boundary_fields_wrapper 662 try: 663 start_value = self._connector_state_converter.parse_value( 664 stream_slice.cursor_slice[start_field] 665 ) 666 end_value = self._connector_state_converter.parse_value( 667 stream_slice.cursor_slice[end_field] 668 ) 669 except (KeyError, ValueError, TypeError) as exception: 670 # A missing or unparseable boundary means this slice was produced (or passed in) wrong - a bug in 671 # slice generation, not something the API did. Raising here, rather than returning `None` (which 672 # `SimpleRetriever` reports as "the API kept rejecting..."), keeps that distinction visible instead 673 # of blaming the API for our own state. 674 raise AirbyteTracedException( 675 internal_message=( 676 f"Cannot split {stream_slice}: boundary field {start_field!r} or {end_field!r} is missing " 677 f"or not parseable: {exception}" 678 ), 679 message=( 680 f"Stream {self._stream_name} produced a request window with a missing or invalid " 681 f"{start_field!r}/{end_field!r} boundary and could not split it further." 682 ), 683 failure_type=FailureType.system_error, 684 ) from exception 685 686 if start_value >= end_value: 687 # Reversed or degenerate boundaries: nothing safe to split. 688 return None 689 690 span = end_value - start_value # type: ignore[operator] # concretely a GapType (timedelta/int) at runtime 691 if min_split_window is not None and span <= min_split_window: # type: ignore[operator] 692 return None 693 694 total_units = span // self._cursor_granularity + 1 # type: ignore[operator] # inclusive count of granularity units 695 if total_units <= 1: 696 # Already at the minimum granularity and still rejected; nothing left to split. 697 return None 698 699 half_units = total_units // 2 700 first_end = ( 701 start_value + (half_units * self._cursor_granularity) - self._cursor_granularity # type: ignore[operator] 702 ) 703 second_start = first_end + self._cursor_granularity 704 705 if not (start_value <= first_end < second_start <= end_value): 706 # No-progress guard, independent of the granularity check above: refuse to return children that 707 # are not strictly ordered and non-degenerate, rather than risk an identical or overlapping split. 708 return None 709 710 return [ 711 StreamSlice( 712 partition=stream_slice.partition, 713 cursor_slice={ 714 start_field: self._connector_state_converter.output_format(start_value), 715 end_field: self._connector_state_converter.output_format(first_end), 716 }, 717 extra_fields=stream_slice.extra_fields, 718 ), 719 StreamSlice( 720 partition=stream_slice.partition, 721 cursor_slice={ 722 start_field: self._connector_state_converter.output_format(second_start), 723 end_field: self._connector_state_converter.output_format(end_value), 724 }, 725 extra_fields=stream_slice.extra_fields, 726 ), 727 ] 728 729 def get_cursor_datetime_from_state( 730 self, stream_state: Mapping[str, Any] 731 ) -> datetime.datetime | None: 732 """Extract and parse the cursor datetime from the given stream state. 733 734 For concurrent cursors, the state can be in two formats: 735 1. Sequential/legacy format: {cursor_field: cursor_value} 736 2. Concurrent format: {state_type: "date-range", slices: [...]} 737 738 Returns the cursor datetime if present and parseable, otherwise returns None. 739 """ 740 # Check if state is in concurrent format (need to convert to dict for type compatibility) 741 mutable_state: MutableMapping[str, Any] = dict(stream_state) 742 if self._connector_state_converter.is_state_message_compatible(mutable_state): 743 slices = stream_state.get("slices", []) 744 if not slices: 745 return None 746 # Get the most recent cursor value from the first slice (after merging) 747 first_slice = slices[0] 748 cursor_value = first_slice.get( 749 self._connector_state_converter.MOST_RECENT_RECORD_KEY 750 ) or first_slice.get(self._connector_state_converter.END_KEY) 751 if not cursor_value: 752 return None 753 try: 754 parsed_value = self._connector_state_converter.parse_value(cursor_value) 755 if isinstance(parsed_value, datetime.datetime): 756 return parsed_value 757 return None 758 except (ValueError, TypeError): 759 return None 760 761 # Sequential/legacy format: {cursor_field: cursor_value} 762 cursor_value = stream_state.get(self._cursor_field.cursor_field_key) 763 if not cursor_value: 764 return None 765 try: 766 parsed_value = self._connector_state_converter.parse_value(cursor_value) 767 if isinstance(parsed_value, datetime.datetime): 768 return parsed_value 769 return None 770 except (ValueError, TypeError): 771 return None
Slices the stream into chunks that can be fetched independently. Slices enable state checkpointing and data retrieval parallelization.
202 def __init__( 203 self, 204 stream_name: str, 205 stream_namespace: Optional[str], 206 stream_state: Any, 207 message_repository: MessageRepository, 208 connector_state_manager: ConnectorStateManager, 209 connector_state_converter: AbstractStreamStateConverter, 210 cursor_field: CursorField, 211 slice_boundary_fields: Optional[Tuple[str, str]], 212 start: Optional[CursorValueType], 213 end_provider: Callable[[], CursorValueType], 214 lookback_window: Optional[GapType] = None, 215 slice_range: Optional[GapType] = None, 216 cursor_granularity: Optional[GapType] = None, 217 clamping_strategy: ClampingStrategy = NoClamping(), 218 is_compare_strictly: bool = False, 219 ) -> None: 220 self._stream_name = stream_name 221 self._stream_namespace = stream_namespace 222 self._message_repository = message_repository 223 self._connector_state_converter = connector_state_converter 224 self._connector_state_manager = connector_state_manager 225 self._cursor_field = cursor_field 226 # To see some example where the slice boundaries might not be defined, check https://github.com/airbytehq/airbyte/blob/1ce84d6396e446e1ac2377362446e3fb94509461/airbyte-integrations/connectors/source-stripe/source_stripe/streams.py#L363-L379 227 self._slice_boundary_fields = slice_boundary_fields 228 self._start = start 229 self._end_provider = end_provider 230 self.start, self._concurrent_state = self._get_concurrent_state(stream_state) 231 self._lookback_window = lookback_window 232 self._slice_range = slice_range 233 self._most_recent_cursor_value_per_partition: MutableMapping[ 234 Union[StreamSlice, Mapping[str, Any], None], Any 235 ] = {} 236 self._has_closed_at_least_one_slice = False 237 self._cursor_granularity = cursor_granularity 238 # Flag to track if the logger has been triggered (per stream) 239 self._should_be_synced_logger_triggered = False 240 self._clamping_strategy = clamping_strategy 241 self._is_compare_strictly = is_compare_strictly 242 self._is_ascending_order = True 243 244 # A lock is required when closing a partition because updating the cursor's concurrent_state is 245 # not thread safe. When multiple partitions are being closed by the cursor at the same time, it is 246 # possible for one partition to update concurrent_state after a second partition has already read 247 # the previous state. This can lead to the second partition overwriting the previous one's state. 248 self._lock = threading.Lock()
183 def copy_without_state(self) -> "ConcurrentCursor": 184 return self.__class__( 185 stream_name=self._stream_name, 186 stream_namespace=self._stream_namespace, 187 stream_state={}, 188 message_repository=NoopMessageRepository(), 189 connector_state_manager=ConnectorStateManager(), 190 connector_state_converter=self._connector_state_converter, 191 cursor_field=self._cursor_field, 192 slice_boundary_fields=self._slice_boundary_fields, 193 start=self._start, 194 end_provider=self._end_provider, 195 lookback_window=self._lookback_window, 196 slice_range=self._slice_range, 197 cursor_granularity=self._cursor_granularity, 198 clamping_strategy=self._clamping_strategy, 199 is_compare_strictly=self._is_compare_strictly, 200 )
297 def observe(self, record: Record) -> None: 298 # Because observe writes to the most_recent_cursor_value_per_partition mapping, 299 # it is not thread-safe. However, this shouldn't lead to concurrency issues because 300 # observe() is only invoked by PartitionReader.process_partition(). Since the map is 301 # broken down according to partition, concurrent threads processing only read/write 302 # from different keys which avoids any conflicts. 303 # 304 # If we were to add thread safety, we should implement a lock per-partition 305 # which is instantiated during stream_slices() 306 most_recent_cursor_value = self._most_recent_cursor_value_per_partition.get( 307 record.associated_slice 308 ) 309 try: 310 cursor_value = self._extract_cursor_value(record) 311 312 if most_recent_cursor_value is None or most_recent_cursor_value < cursor_value: 313 self._most_recent_cursor_value_per_partition[record.associated_slice] = cursor_value 314 elif most_recent_cursor_value > cursor_value: 315 self._is_ascending_order = False 316 except ValueError: 317 self._log_for_record_without_cursor_value()
Indicate to the cursor that the record has been emitted
322 def close_partition(self, partition: Partition) -> None: 323 with self._lock: 324 slice_count_before = len(self._concurrent_state.get("slices", [])) 325 self._add_slice_to_state(partition) 326 if slice_count_before < len( 327 self._concurrent_state["slices"] 328 ): # only emit if at least one slice has been processed 329 self._merge_partitions() 330 self._emit_state_message() 331 self._has_closed_at_least_one_slice = True
Indicate to the cursor that the partition has been successfully processed
406 def ensure_at_least_one_state_emitted(self) -> None: 407 """ 408 The platform expect to have at least one state message on successful syncs. Hence, whatever happens, we expect this method to be 409 called. 410 """ 411 self._emit_state_message()
The platform expect to have at least one state message on successful syncs. Hence, whatever happens, we expect this method to be called.
413 def stream_slices(self) -> Iterable[StreamSlice]: 414 """ 415 Generating slices based on a few parameters: 416 * lookback_window: Buffer to remove from END_KEY of the highest slice 417 * slice_range: Max difference between two slices. If the difference between two slices is greater, multiple slices will be created 418 * start: `_split_per_slice_range` will clip any value to `self._start which means that: 419 * if upper is less than self._start, no slices will be generated 420 * if lower is less than self._start, self._start will be used as the lower boundary (lookback_window will not be considered in that case) 421 422 Note that the slices will overlap at their boundaries. We therefore expect to have at least the lower or the upper boundary to be 423 inclusive in the API that is queried. 424 """ 425 self._merge_partitions() 426 427 if self._start is not None and self._is_start_before_first_slice(): 428 yield from self._split_per_slice_range( 429 self._start, 430 self._concurrent_state["slices"][0][self._connector_state_converter.START_KEY], 431 False, 432 ) 433 434 if len(self._concurrent_state["slices"]) == 1: 435 yield from self._split_per_slice_range( 436 self._calculate_lower_boundary_of_last_slice( 437 self._concurrent_state["slices"][0][self._connector_state_converter.END_KEY] 438 ), 439 self._end_provider(), 440 True, 441 ) 442 elif len(self._concurrent_state["slices"]) > 1: 443 for i in range(len(self._concurrent_state["slices"]) - 1): 444 if self._cursor_granularity: 445 yield from self._split_per_slice_range( 446 self._concurrent_state["slices"][i][self._connector_state_converter.END_KEY] 447 + self._cursor_granularity, 448 self._concurrent_state["slices"][i + 1][ 449 self._connector_state_converter.START_KEY 450 ], 451 False, 452 ) 453 else: 454 yield from self._split_per_slice_range( 455 self._concurrent_state["slices"][i][ 456 self._connector_state_converter.END_KEY 457 ], 458 self._concurrent_state["slices"][i + 1][ 459 self._connector_state_converter.START_KEY 460 ], 461 False, 462 ) 463 yield from self._split_per_slice_range( 464 self._calculate_lower_boundary_of_last_slice( 465 self._concurrent_state["slices"][-1][self._connector_state_converter.END_KEY] 466 ), 467 self._end_provider(), 468 True, 469 ) 470 else: 471 raise ValueError("Expected at least one slice")
Generating slices based on a few parameters:
- lookback_window: Buffer to remove from END_KEY of the highest slice
- slice_range: Max difference between two slices. If the difference between two slices is greater, multiple slices will be created
- start:
_split_per_slice_rangewill clip any value to `self._start which means that:- if upper is less than self._start, no slices will be generated
- if lower is less than self._start, self._start will be used as the lower boundary (lookback_window will not be considered in that case)
Note that the slices will overlap at their boundaries. We therefore expect to have at least the lower or the upper boundary to be inclusive in the API that is queried.
579 def should_be_synced(self, record: Record) -> bool: 580 """ 581 Determines if a record should be synced based on its cursor value. 582 :param record: The record to evaluate 583 584 :return: True if the record's cursor value falls within the sync boundaries 585 """ 586 try: 587 record_cursor_value: CursorValueType = self._extract_cursor_value(record) 588 except ValueError: 589 self._log_for_record_without_cursor_value() 590 return True 591 return self._lowest_cursor_value_to_sync() <= record_cursor_value <= self._end_provider()
Determines if a record should be synced based on its cursor value.
Parameters
- record: The record to evaluate
Returns
True if the record's cursor value falls within the sync boundaries
611 def reduce_slice_range(self, stream_slice: StreamSlice) -> StreamSlice: 612 # In theory, we might be more flexible here meaning that it doesn't need to be in ascending order but it just 613 # needs to be ordered. For now though, we will only support ascending order. 614 if not self._is_ascending_order: 615 LOGGER.warning( 616 "Attempting to reduce slice while records are not returned in incremental order might lead to missing records" 617 ) 618 619 if stream_slice in self._most_recent_cursor_value_per_partition: 620 return StreamSlice( 621 partition=stream_slice.partition, 622 cursor_slice={ 623 self._slice_boundary_fields_wrapper[ 624 self._START_BOUNDARY 625 ]: self._connector_state_converter.output_format( 626 self._most_recent_cursor_value_per_partition[stream_slice] 627 ), 628 self._slice_boundary_fields_wrapper[ 629 self._END_BOUNDARY 630 ]: stream_slice.cursor_slice[ 631 self._slice_boundary_fields_wrapper[self._END_BOUNDARY] 632 ], 633 }, 634 extra_fields=stream_slice.extra_fields, 635 ) 636 else: 637 return stream_slice
639 def split_request_window( 640 self, stream_slice: StreamSlice, min_split_window: Optional[datetime.timedelta] = None 641 ) -> Optional[List[StreamSlice]]: 642 """ 643 Split `stream_slice` in half along this cursor's own boundary fields, granularity, and output format. 644 645 Returns `None` when the slice cannot be split any further: either it spans at most one 646 `cursor_granularity` unit already (the floor), it is already at or below `min_split_window` (a 647 connector-configured floor independent of and typically looser than `cursor_granularity`), the 648 boundaries are reversed, or no `cursor_granularity` was configured at all (there is then no way to know 649 the smallest addressable unit, or how to keep two children from overlapping at their shared edge). 650 Raises `AirbyteTracedException` instead if a boundary is missing or unparseable: that is a bug in how 651 the slice was produced, not the API rejecting a window, so it must not be reported as one. This is the 652 implementation `SimpleRetriever.request_window_splitter` binds to (see 653 `airbyte_cdk.sources.declarative.retrievers.simple_retriever`) that reuses the same 654 parsing/formatting/comparison operations `_split_per_slice_range` already relies on, so it works for 655 any `CursorValueType`/`GapType` pair this cursor was built with (datetime/timedelta, or int/int), 656 without a second, duplicate implementation of that logic living outside the cursor. 657 """ 658 if not self._cursor_granularity: 659 return None 660 661 start_field, end_field = self._slice_boundary_fields_wrapper 662 try: 663 start_value = self._connector_state_converter.parse_value( 664 stream_slice.cursor_slice[start_field] 665 ) 666 end_value = self._connector_state_converter.parse_value( 667 stream_slice.cursor_slice[end_field] 668 ) 669 except (KeyError, ValueError, TypeError) as exception: 670 # A missing or unparseable boundary means this slice was produced (or passed in) wrong - a bug in 671 # slice generation, not something the API did. Raising here, rather than returning `None` (which 672 # `SimpleRetriever` reports as "the API kept rejecting..."), keeps that distinction visible instead 673 # of blaming the API for our own state. 674 raise AirbyteTracedException( 675 internal_message=( 676 f"Cannot split {stream_slice}: boundary field {start_field!r} or {end_field!r} is missing " 677 f"or not parseable: {exception}" 678 ), 679 message=( 680 f"Stream {self._stream_name} produced a request window with a missing or invalid " 681 f"{start_field!r}/{end_field!r} boundary and could not split it further." 682 ), 683 failure_type=FailureType.system_error, 684 ) from exception 685 686 if start_value >= end_value: 687 # Reversed or degenerate boundaries: nothing safe to split. 688 return None 689 690 span = end_value - start_value # type: ignore[operator] # concretely a GapType (timedelta/int) at runtime 691 if min_split_window is not None and span <= min_split_window: # type: ignore[operator] 692 return None 693 694 total_units = span // self._cursor_granularity + 1 # type: ignore[operator] # inclusive count of granularity units 695 if total_units <= 1: 696 # Already at the minimum granularity and still rejected; nothing left to split. 697 return None 698 699 half_units = total_units // 2 700 first_end = ( 701 start_value + (half_units * self._cursor_granularity) - self._cursor_granularity # type: ignore[operator] 702 ) 703 second_start = first_end + self._cursor_granularity 704 705 if not (start_value <= first_end < second_start <= end_value): 706 # No-progress guard, independent of the granularity check above: refuse to return children that 707 # are not strictly ordered and non-degenerate, rather than risk an identical or overlapping split. 708 return None 709 710 return [ 711 StreamSlice( 712 partition=stream_slice.partition, 713 cursor_slice={ 714 start_field: self._connector_state_converter.output_format(start_value), 715 end_field: self._connector_state_converter.output_format(first_end), 716 }, 717 extra_fields=stream_slice.extra_fields, 718 ), 719 StreamSlice( 720 partition=stream_slice.partition, 721 cursor_slice={ 722 start_field: self._connector_state_converter.output_format(second_start), 723 end_field: self._connector_state_converter.output_format(end_value), 724 }, 725 extra_fields=stream_slice.extra_fields, 726 ), 727 ]
Split stream_slice in half along this cursor's own boundary fields, granularity, and output format.
Returns None when the slice cannot be split any further: either it spans at most one
cursor_granularity unit already (the floor), it is already at or below min_split_window (a
connector-configured floor independent of and typically looser than cursor_granularity), the
boundaries are reversed, or no cursor_granularity was configured at all (there is then no way to know
the smallest addressable unit, or how to keep two children from overlapping at their shared edge).
Raises AirbyteTracedException instead if a boundary is missing or unparseable: that is a bug in how
the slice was produced, not the API rejecting a window, so it must not be reported as one. This is the
implementation SimpleRetriever.request_window_splitter binds to (see
airbyte_cdk.sources.declarative.retrievers.simple_retriever) that reuses the same
parsing/formatting/comparison operations _split_per_slice_range already relies on, so it works for
any CursorValueType/GapType pair this cursor was built with (datetime/timedelta, or int/int),
without a second, duplicate implementation of that logic living outside the cursor.
729 def get_cursor_datetime_from_state( 730 self, stream_state: Mapping[str, Any] 731 ) -> datetime.datetime | None: 732 """Extract and parse the cursor datetime from the given stream state. 733 734 For concurrent cursors, the state can be in two formats: 735 1. Sequential/legacy format: {cursor_field: cursor_value} 736 2. Concurrent format: {state_type: "date-range", slices: [...]} 737 738 Returns the cursor datetime if present and parseable, otherwise returns None. 739 """ 740 # Check if state is in concurrent format (need to convert to dict for type compatibility) 741 mutable_state: MutableMapping[str, Any] = dict(stream_state) 742 if self._connector_state_converter.is_state_message_compatible(mutable_state): 743 slices = stream_state.get("slices", []) 744 if not slices: 745 return None 746 # Get the most recent cursor value from the first slice (after merging) 747 first_slice = slices[0] 748 cursor_value = first_slice.get( 749 self._connector_state_converter.MOST_RECENT_RECORD_KEY 750 ) or first_slice.get(self._connector_state_converter.END_KEY) 751 if not cursor_value: 752 return None 753 try: 754 parsed_value = self._connector_state_converter.parse_value(cursor_value) 755 if isinstance(parsed_value, datetime.datetime): 756 return parsed_value 757 return None 758 except (ValueError, TypeError): 759 return None 760 761 # Sequential/legacy format: {cursor_field: cursor_value} 762 cursor_value = stream_state.get(self._cursor_field.cursor_field_key) 763 if not cursor_value: 764 return None 765 try: 766 parsed_value = self._connector_state_converter.parse_value(cursor_value) 767 if isinstance(parsed_value, datetime.datetime): 768 return parsed_value 769 return None 770 except (ValueError, TypeError): 771 return None
Extract and parse the cursor datetime from the given stream state.
For concurrent cursors, the state can be in two formats:
- Sequential/legacy format: {cursor_field: cursor_value}
- Concurrent format: {state_type: "date-range", slices: [...]}
Returns the cursor datetime if present and parseable, otherwise returns None.
31class ConcurrentSource: 32 """ 33 A Source that reads data from multiple AbstractStreams concurrently. 34 It does so by submitting partition generation, and partition read tasks to a thread pool. 35 The tasks asynchronously add their output to a shared queue. 36 The read is done when all partitions for all streams w ere generated and read. 37 """ 38 39 DEFAULT_TIMEOUT_SECONDS = 900 # Deprecated: default of the ignored `timeout_seconds`. 40 41 @staticmethod 42 def create( 43 num_workers: int, 44 initial_number_of_partitions_to_generate: int, 45 logger: logging.Logger, 46 slice_logger: SliceLogger, 47 message_repository: MessageRepository, 48 queue: Optional[Queue[QueueItem]] = None, 49 timeout_seconds: int = DEFAULT_TIMEOUT_SECONDS, 50 ) -> "ConcurrentSource": 51 if initial_number_of_partitions_to_generate < 1: 52 raise ValueError( 53 f"initial_number_of_partitions_to_generate must be >= 1, got {initial_number_of_partitions_to_generate}" 54 ) 55 is_single_threaded = initial_number_of_partitions_to_generate == 1 and num_workers == 1 56 too_many_generator = ( 57 not is_single_threaded and initial_number_of_partitions_to_generate >= num_workers 58 ) 59 assert not too_many_generator, ( 60 "It is required to have more workers than threads generating partitions" 61 ) 62 threadpool = ThreadPoolManager( 63 concurrent.futures.ThreadPoolExecutor( 64 max_workers=num_workers, thread_name_prefix="workerpool" 65 ), 66 logger, 67 ) 68 return ConcurrentSource( 69 threadpool=threadpool, 70 logger=logger, 71 slice_logger=slice_logger, 72 queue=queue, 73 message_repository=message_repository, 74 initial_number_partitions_to_generate=initial_number_of_partitions_to_generate, 75 timeout_seconds=timeout_seconds, 76 ) 77 78 def __init__( 79 self, 80 threadpool: ThreadPoolManager, 81 logger: logging.Logger, 82 slice_logger: SliceLogger = DebugSliceLogger(), 83 queue: Optional[Queue[QueueItem]] = None, 84 message_repository: MessageRepository = InMemoryMessageRepository(), 85 initial_number_partitions_to_generate: int = 1, 86 timeout_seconds: int = DEFAULT_TIMEOUT_SECONDS, 87 ) -> None: 88 """ 89 :param threadpool: The threadpool to submit tasks to 90 :param logger: The logger to log to 91 :param slice_logger: The slice logger used to create messages on new slices 92 :param message_repository: The repository to emit messages to 93 :param initial_number_partitions_to_generate: The initial number of concurrent partition generation tasks. Limiting this number ensures will limit the latency of the first records emitted. While the latency is not critical, emitting the records early allows the platform and the destination to process them as early as possible. 94 :param timeout_seconds: Deprecated and ignored. Kept so existing callers do not break. 95 """ 96 self._threadpool = threadpool 97 self._logger = logger 98 self._slice_logger = slice_logger 99 self._message_repository = message_repository 100 self._initial_number_partitions_to_generate = initial_number_partitions_to_generate 101 102 # We set a maxsize to for the main thread to process record items when the queue size grows. This assumes that there are less 103 # threads generating partitions that than are max number of workers. If it weren't the case, we could have threads only generating 104 # partitions which would fill the queue. This number is arbitrarily set to 10_000 but will probably need to be changed given more 105 # information and might even need to be configurable depending on the source 106 self._queue = queue or Queue(maxsize=10_000) 107 108 def read( 109 self, 110 streams: List[AbstractStream], 111 ) -> Iterator[AirbyteMessage]: 112 self._logger.info("Starting syncing") 113 concurrent_stream_processor = ConcurrentReadProcessor( 114 streams, 115 PartitionEnqueuer(self._queue, self._threadpool), 116 self._threadpool, 117 self._logger, 118 self._slice_logger, 119 self._message_repository, 120 PartitionReader( 121 self._queue, 122 PartitionLogger(self._slice_logger, self._logger, self._message_repository), 123 ), 124 max_concurrent_partition_generators=self._initial_number_partitions_to_generate, 125 ) 126 127 # Enqueue initial partition generation tasks 128 yield from self._submit_initial_partition_generators(concurrent_stream_processor) 129 130 # Read from the queue until all partitions were generated and read 131 yield from self._consume_from_queue( 132 self._queue, 133 concurrent_stream_processor, 134 ) 135 self._threadpool.check_for_errors_and_shutdown() 136 self._logger.info("Finished syncing") 137 138 def _submit_initial_partition_generators( 139 self, concurrent_stream_processor: ConcurrentReadProcessor 140 ) -> Iterable[AirbyteMessage]: 141 for _ in range(self._initial_number_partitions_to_generate): 142 status_message = concurrent_stream_processor.start_next_partition_generator() 143 if status_message: 144 yield status_message 145 146 def _consume_from_queue( 147 self, 148 queue: Queue[QueueItem], 149 concurrent_stream_processor: ConcurrentReadProcessor, 150 ) -> Iterable[AirbyteMessage]: 151 while airbyte_message_or_record_or_exception := queue.get(): 152 yield from self._handle_item( 153 airbyte_message_or_record_or_exception, 154 concurrent_stream_processor, 155 ) 156 # In the event that a partition raises an exception, anything remaining in 157 # the queue will be missed because is_done() can raise an exception and exit 158 # out of this loop before remaining items are consumed 159 if queue.empty() and concurrent_stream_processor.is_done(): 160 # all partitions were generated and processed. we're done here 161 break 162 163 def _handle_item( 164 self, 165 queue_item: QueueItem, 166 concurrent_stream_processor: ConcurrentReadProcessor, 167 ) -> Iterable[AirbyteMessage]: 168 # handle queue item and call the appropriate handler depending on the type of the queue item 169 if isinstance(queue_item, StreamThreadException): 170 yield from concurrent_stream_processor.on_exception(queue_item) 171 elif isinstance(queue_item, PartitionGenerationCompletedSentinel): 172 yield from concurrent_stream_processor.on_partition_generation_completed(queue_item) 173 elif isinstance(queue_item, Partition): 174 concurrent_stream_processor.on_partition(queue_item) 175 elif isinstance(queue_item, PartitionCompleteSentinel): 176 yield from concurrent_stream_processor.on_partition_complete_sentinel(queue_item) 177 elif isinstance(queue_item, Record): 178 yield from concurrent_stream_processor.on_record(queue_item) 179 elif isinstance(queue_item, AirbyteMessage): 180 yield queue_item 181 else: 182 raise ValueError(f"Unknown queue item type: {type(queue_item)}")
A Source that reads data from multiple AbstractStreams concurrently. It does so by submitting partition generation, and partition read tasks to a thread pool. The tasks asynchronously add their output to a shared queue. The read is done when all partitions for all streams w ere generated and read.
78 def __init__( 79 self, 80 threadpool: ThreadPoolManager, 81 logger: logging.Logger, 82 slice_logger: SliceLogger = DebugSliceLogger(), 83 queue: Optional[Queue[QueueItem]] = None, 84 message_repository: MessageRepository = InMemoryMessageRepository(), 85 initial_number_partitions_to_generate: int = 1, 86 timeout_seconds: int = DEFAULT_TIMEOUT_SECONDS, 87 ) -> None: 88 """ 89 :param threadpool: The threadpool to submit tasks to 90 :param logger: The logger to log to 91 :param slice_logger: The slice logger used to create messages on new slices 92 :param message_repository: The repository to emit messages to 93 :param initial_number_partitions_to_generate: The initial number of concurrent partition generation tasks. Limiting this number ensures will limit the latency of the first records emitted. While the latency is not critical, emitting the records early allows the platform and the destination to process them as early as possible. 94 :param timeout_seconds: Deprecated and ignored. Kept so existing callers do not break. 95 """ 96 self._threadpool = threadpool 97 self._logger = logger 98 self._slice_logger = slice_logger 99 self._message_repository = message_repository 100 self._initial_number_partitions_to_generate = initial_number_partitions_to_generate 101 102 # We set a maxsize to for the main thread to process record items when the queue size grows. This assumes that there are less 103 # threads generating partitions that than are max number of workers. If it weren't the case, we could have threads only generating 104 # partitions which would fill the queue. This number is arbitrarily set to 10_000 but will probably need to be changed given more 105 # information and might even need to be configurable depending on the source 106 self._queue = queue or Queue(maxsize=10_000)
Parameters
- threadpool: The threadpool to submit tasks to
- logger: The logger to log to
- slice_logger: The slice logger used to create messages on new slices
- message_repository: The repository to emit messages to
- initial_number_partitions_to_generate: The initial number of concurrent partition generation tasks. Limiting this number ensures will limit the latency of the first records emitted. While the latency is not critical, emitting the records early allows the platform and the destination to process them as early as possible.
- timeout_seconds: Deprecated and ignored. Kept so existing callers do not break.
41 @staticmethod 42 def create( 43 num_workers: int, 44 initial_number_of_partitions_to_generate: int, 45 logger: logging.Logger, 46 slice_logger: SliceLogger, 47 message_repository: MessageRepository, 48 queue: Optional[Queue[QueueItem]] = None, 49 timeout_seconds: int = DEFAULT_TIMEOUT_SECONDS, 50 ) -> "ConcurrentSource": 51 if initial_number_of_partitions_to_generate < 1: 52 raise ValueError( 53 f"initial_number_of_partitions_to_generate must be >= 1, got {initial_number_of_partitions_to_generate}" 54 ) 55 is_single_threaded = initial_number_of_partitions_to_generate == 1 and num_workers == 1 56 too_many_generator = ( 57 not is_single_threaded and initial_number_of_partitions_to_generate >= num_workers 58 ) 59 assert not too_many_generator, ( 60 "It is required to have more workers than threads generating partitions" 61 ) 62 threadpool = ThreadPoolManager( 63 concurrent.futures.ThreadPoolExecutor( 64 max_workers=num_workers, thread_name_prefix="workerpool" 65 ), 66 logger, 67 ) 68 return ConcurrentSource( 69 threadpool=threadpool, 70 logger=logger, 71 slice_logger=slice_logger, 72 queue=queue, 73 message_repository=message_repository, 74 initial_number_partitions_to_generate=initial_number_of_partitions_to_generate, 75 timeout_seconds=timeout_seconds, 76 )
108 def read( 109 self, 110 streams: List[AbstractStream], 111 ) -> Iterator[AirbyteMessage]: 112 self._logger.info("Starting syncing") 113 concurrent_stream_processor = ConcurrentReadProcessor( 114 streams, 115 PartitionEnqueuer(self._queue, self._threadpool), 116 self._threadpool, 117 self._logger, 118 self._slice_logger, 119 self._message_repository, 120 PartitionReader( 121 self._queue, 122 PartitionLogger(self._slice_logger, self._logger, self._message_repository), 123 ), 124 max_concurrent_partition_generators=self._initial_number_partitions_to_generate, 125 ) 126 127 # Enqueue initial partition generation tasks 128 yield from self._submit_initial_partition_generators(concurrent_stream_processor) 129 130 # Read from the queue until all partitions were generated and read 131 yield from self._consume_from_queue( 132 self._queue, 133 concurrent_stream_processor, 134 ) 135 self._threadpool.check_for_errors_and_shutdown() 136 self._logger.info("Finished syncing")
34class ConcurrentSourceAdapter(AbstractSource, ABC): 35 def __init__(self, concurrent_source: ConcurrentSource, **kwargs: Any) -> None: 36 """ 37 ConcurrentSourceAdapter is a Source that wraps a concurrent source and exposes it as a regular source. 38 39 The source's streams are still defined through the streams() method. 40 Streams wrapped in a StreamFacade will be processed concurrently. 41 Other streams will be processed sequentially as a later step. 42 """ 43 self._concurrent_source = concurrent_source 44 super().__init__(**kwargs) 45 46 def read( 47 self, 48 logger: logging.Logger, 49 config: Mapping[str, Any], 50 catalog: ConfiguredAirbyteCatalog, 51 state: Optional[List[AirbyteStateMessage]] = None, 52 ) -> Iterator[AirbyteMessage]: 53 abstract_streams = self._select_abstract_streams(config, catalog) 54 concurrent_stream_names = {stream.name for stream in abstract_streams} 55 configured_catalog_for_regular_streams = ConfiguredAirbyteCatalog( 56 streams=[ 57 stream 58 for stream in catalog.streams 59 if stream.stream.name not in concurrent_stream_names 60 ] 61 ) 62 if abstract_streams: 63 yield from self._concurrent_source.read(abstract_streams) 64 if configured_catalog_for_regular_streams.streams: 65 yield from super().read(logger, config, configured_catalog_for_regular_streams, state) 66 67 def _select_abstract_streams( 68 self, config: Mapping[str, Any], configured_catalog: ConfiguredAirbyteCatalog 69 ) -> List[AbstractStream]: 70 """ 71 Selects streams that can be processed concurrently and returns their abstract representations. 72 """ 73 all_streams = self.streams(config) 74 stream_name_to_instance: Mapping[str, Stream] = {s.name: s for s in all_streams} 75 abstract_streams: List[AbstractStream] = [] 76 for configured_stream in configured_catalog.streams: 77 stream_instance = stream_name_to_instance.get(configured_stream.stream.name) 78 if not stream_instance: 79 continue 80 81 if isinstance(stream_instance, AbstractStreamFacade): 82 abstract_streams.append(stream_instance.get_underlying_stream()) 83 return abstract_streams 84 85 def convert_to_concurrent_stream( 86 self, 87 logger: logging.Logger, 88 stream: Stream, 89 state_manager: ConnectorStateManager, 90 cursor: Optional[Cursor] = None, 91 ) -> Stream: 92 """ 93 Prepares a stream for concurrent processing by initializing or assigning a cursor, 94 managing the stream's state, and returning an updated Stream instance. 95 """ 96 state: MutableMapping[str, Any] = {} 97 98 if cursor: 99 state = state_manager.get_stream_state(stream.name, stream.namespace) 100 101 stream.cursor = cursor # type: ignore[assignment] # cursor is of type ConcurrentCursor, which inherits from Cursor 102 if hasattr(stream, "parent"): 103 stream.parent.cursor = cursor 104 else: 105 cursor = FinalStateCursor( 106 stream_name=stream.name, 107 stream_namespace=stream.namespace, 108 message_repository=self.message_repository, # type: ignore[arg-type] # _default_message_repository will be returned in the worst case 109 ) 110 return StreamFacade.create_from_stream(stream, self, logger, state, cursor) 111 112 def initialize_cursor( 113 self, 114 stream: Stream, 115 state_manager: ConnectorStateManager, 116 converter: AbstractStreamStateConverter, 117 slice_boundary_fields: Optional[Tuple[str, str]], 118 start: Optional[CursorValueType], 119 end_provider: Callable[[], CursorValueType], 120 lookback_window: Optional[GapType] = None, 121 slice_range: Optional[GapType] = None, 122 ) -> Optional[ConcurrentCursor]: 123 lookback_window = lookback_window or timedelta(seconds=DEFAULT_LOOKBACK_SECONDS) 124 125 cursor_field_name = stream.cursor_field 126 127 if cursor_field_name: 128 if not isinstance(cursor_field_name, str): 129 raise ValueError( 130 f"Cursor field type must be a string, but received {type(cursor_field_name).__name__}." 131 ) 132 133 return ConcurrentCursor( 134 stream.name, 135 stream.namespace, 136 state_manager.get_stream_state(stream.name, stream.namespace), 137 self.message_repository, # type: ignore[arg-type] # _default_message_repository will be returned in the worst case 138 state_manager, 139 converter, 140 CursorField(cursor_field_name), 141 slice_boundary_fields, 142 start, 143 end_provider, 144 lookback_window, 145 slice_range, 146 ) 147 148 return None
Abstract base class for an Airbyte Source. Consumers should implement any abstract methods in this class to create an Airbyte Specification compliant Source.
35 def __init__(self, concurrent_source: ConcurrentSource, **kwargs: Any) -> None: 36 """ 37 ConcurrentSourceAdapter is a Source that wraps a concurrent source and exposes it as a regular source. 38 39 The source's streams are still defined through the streams() method. 40 Streams wrapped in a StreamFacade will be processed concurrently. 41 Other streams will be processed sequentially as a later step. 42 """ 43 self._concurrent_source = concurrent_source 44 super().__init__(**kwargs)
ConcurrentSourceAdapter is a Source that wraps a concurrent source and exposes it as a regular source.
The source's streams are still defined through the streams() method. Streams wrapped in a StreamFacade will be processed concurrently. Other streams will be processed sequentially as a later step.
46 def read( 47 self, 48 logger: logging.Logger, 49 config: Mapping[str, Any], 50 catalog: ConfiguredAirbyteCatalog, 51 state: Optional[List[AirbyteStateMessage]] = None, 52 ) -> Iterator[AirbyteMessage]: 53 abstract_streams = self._select_abstract_streams(config, catalog) 54 concurrent_stream_names = {stream.name for stream in abstract_streams} 55 configured_catalog_for_regular_streams = ConfiguredAirbyteCatalog( 56 streams=[ 57 stream 58 for stream in catalog.streams 59 if stream.stream.name not in concurrent_stream_names 60 ] 61 ) 62 if abstract_streams: 63 yield from self._concurrent_source.read(abstract_streams) 64 if configured_catalog_for_regular_streams.streams: 65 yield from super().read(logger, config, configured_catalog_for_regular_streams, state)
Implements the Read operation from the Airbyte Specification. See https://docs.airbyte.com/understanding-airbyte/airbyte-protocol/.
85 def convert_to_concurrent_stream( 86 self, 87 logger: logging.Logger, 88 stream: Stream, 89 state_manager: ConnectorStateManager, 90 cursor: Optional[Cursor] = None, 91 ) -> Stream: 92 """ 93 Prepares a stream for concurrent processing by initializing or assigning a cursor, 94 managing the stream's state, and returning an updated Stream instance. 95 """ 96 state: MutableMapping[str, Any] = {} 97 98 if cursor: 99 state = state_manager.get_stream_state(stream.name, stream.namespace) 100 101 stream.cursor = cursor # type: ignore[assignment] # cursor is of type ConcurrentCursor, which inherits from Cursor 102 if hasattr(stream, "parent"): 103 stream.parent.cursor = cursor 104 else: 105 cursor = FinalStateCursor( 106 stream_name=stream.name, 107 stream_namespace=stream.namespace, 108 message_repository=self.message_repository, # type: ignore[arg-type] # _default_message_repository will be returned in the worst case 109 ) 110 return StreamFacade.create_from_stream(stream, self, logger, state, cursor)
Prepares a stream for concurrent processing by initializing or assigning a cursor, managing the stream's state, and returning an updated Stream instance.
112 def initialize_cursor( 113 self, 114 stream: Stream, 115 state_manager: ConnectorStateManager, 116 converter: AbstractStreamStateConverter, 117 slice_boundary_fields: Optional[Tuple[str, str]], 118 start: Optional[CursorValueType], 119 end_provider: Callable[[], CursorValueType], 120 lookback_window: Optional[GapType] = None, 121 slice_range: Optional[GapType] = None, 122 ) -> Optional[ConcurrentCursor]: 123 lookback_window = lookback_window or timedelta(seconds=DEFAULT_LOOKBACK_SECONDS) 124 125 cursor_field_name = stream.cursor_field 126 127 if cursor_field_name: 128 if not isinstance(cursor_field_name, str): 129 raise ValueError( 130 f"Cursor field type must be a string, but received {type(cursor_field_name).__name__}." 131 ) 132 133 return ConcurrentCursor( 134 stream.name, 135 stream.namespace, 136 state_manager.get_stream_state(stream.name, stream.namespace), 137 self.message_repository, # type: ignore[arg-type] # _default_message_repository will be returned in the worst case 138 state_manager, 139 converter, 140 CursorField(cursor_field_name), 141 slice_boundary_fields, 142 start, 143 end_provider, 144 lookback_window, 145 slice_range, 146 ) 147 148 return None
Inherited Members
58class Cursor(StreamSlicer, ABC): 59 @property 60 @abstractmethod 61 def state(self) -> MutableMapping[str, Any]: ... 62 63 @abstractmethod 64 def observe(self, record: Record) -> None: 65 """ 66 Indicate to the cursor that the record has been emitted 67 """ 68 raise NotImplementedError() 69 70 @abstractmethod 71 def close_partition(self, partition: Partition) -> None: 72 """ 73 Indicate to the cursor that the partition has been successfully processed 74 """ 75 raise NotImplementedError() 76 77 @abstractmethod 78 def ensure_at_least_one_state_emitted(self) -> None: 79 """ 80 State messages are emitted when a partition is closed. However, the platform expects at least one state to be emitted per sync per 81 stream. Hence, if no partitions are generated, this method needs to be called. 82 """ 83 raise NotImplementedError() 84 85 @abstractmethod 86 def should_be_synced(self, record: Record) -> bool: 87 pass 88 89 def stream_slices(self) -> Iterable[StreamSlice]: 90 """ 91 Default placeholder implementation of generate_slices. 92 Subclasses can override this method to provide actual behavior. 93 """ 94 yield StreamSlice(partition={}, cursor_slice={}) 95 96 def get_cursor_datetime_from_state( 97 self, stream_state: Mapping[str, Any] 98 ) -> datetime.datetime | None: 99 """Extract and parse the cursor datetime from the given stream state. 100 101 This method is used by StateDelegatingStream to validate cursor age against 102 an API's data retention period. Subclasses should implement this method to 103 extract the cursor value from their specific state structure and parse it 104 into a datetime object. 105 106 Returns None if the cursor cannot be extracted or parsed, which will cause 107 StateDelegatingStream to fall back to full refresh (safe default). 108 109 Raises NotImplementedError by default - subclasses must implement this method 110 if they want to support cursor age validation with api_retention_period. 111 """ 112 raise NotImplementedError( 113 f"{self.__class__.__name__} does not implement get_cursor_datetime_from_state. " 114 f"Cursor age validation with api_retention_period is not supported for this cursor type." 115 )
Slices the stream into chunks that can be fetched independently. Slices enable state checkpointing and data retrieval parallelization.
63 @abstractmethod 64 def observe(self, record: Record) -> None: 65 """ 66 Indicate to the cursor that the record has been emitted 67 """ 68 raise NotImplementedError()
Indicate to the cursor that the record has been emitted
70 @abstractmethod 71 def close_partition(self, partition: Partition) -> None: 72 """ 73 Indicate to the cursor that the partition has been successfully processed 74 """ 75 raise NotImplementedError()
Indicate to the cursor that the partition has been successfully processed
77 @abstractmethod 78 def ensure_at_least_one_state_emitted(self) -> None: 79 """ 80 State messages are emitted when a partition is closed. However, the platform expects at least one state to be emitted per sync per 81 stream. Hence, if no partitions are generated, this method needs to be called. 82 """ 83 raise NotImplementedError()
State messages are emitted when a partition is closed. However, the platform expects at least one state to be emitted per sync per stream. Hence, if no partitions are generated, this method needs to be called.
89 def stream_slices(self) -> Iterable[StreamSlice]: 90 """ 91 Default placeholder implementation of generate_slices. 92 Subclasses can override this method to provide actual behavior. 93 """ 94 yield StreamSlice(partition={}, cursor_slice={})
Default placeholder implementation of generate_slices. Subclasses can override this method to provide actual behavior.
96 def get_cursor_datetime_from_state( 97 self, stream_state: Mapping[str, Any] 98 ) -> datetime.datetime | None: 99 """Extract and parse the cursor datetime from the given stream state. 100 101 This method is used by StateDelegatingStream to validate cursor age against 102 an API's data retention period. Subclasses should implement this method to 103 extract the cursor value from their specific state structure and parse it 104 into a datetime object. 105 106 Returns None if the cursor cannot be extracted or parsed, which will cause 107 StateDelegatingStream to fall back to full refresh (safe default). 108 109 Raises NotImplementedError by default - subclasses must implement this method 110 if they want to support cursor age validation with api_retention_period. 111 """ 112 raise NotImplementedError( 113 f"{self.__class__.__name__} does not implement get_cursor_datetime_from_state. " 114 f"Cursor age validation with api_retention_period is not supported for this cursor type." 115 )
Extract and parse the cursor datetime from the given stream state.
This method is used by StateDelegatingStream to validate cursor age against an API's data retention period. Subclasses should implement this method to extract the cursor value from their specific state structure and parse it into a datetime object.
Returns None if the cursor cannot be extracted or parsed, which will cause StateDelegatingStream to fall back to full refresh (safe default).
Raises NotImplementedError by default - subclasses must implement this method if they want to support cursor age validation with api_retention_period.
44class CursorField: 45 def __init__( 46 self, cursor_field_key: str, supports_catalog_defined_cursor_field: bool = False 47 ) -> None: 48 self.cursor_field_key = cursor_field_key 49 self.supports_catalog_defined_cursor_field = supports_catalog_defined_cursor_field 50 51 def extract_value(self, record: Record) -> Any: 52 cursor_value = record.data.get(self.cursor_field_key) 53 if cursor_value is None: 54 raise ValueError(f"Could not find cursor field {self.cursor_field_key} in record") 55 return cursor_value # type: ignore # we assume that the value the path points at is a comparable
51 def extract_value(self, record: Record) -> Any: 52 cursor_value = record.data.get(self.cursor_field_key) 53 if cursor_value is None: 54 raise ValueError(f"Could not find cursor field {self.cursor_field_key} in record") 55 return cursor_value # type: ignore # we assume that the value the path points at is a comparable
115class EpochValueConcurrentStreamStateConverter(DateTimeStreamStateConverter): 116 """ 117 e.g. 118 { "created": 1617030403 } 119 => 120 { 121 "state_type": "date-range", 122 "metadata": { … }, 123 "slices": [ 124 {starts: 0, end: 1617030403, finished_processing: true} 125 ] 126 } 127 """ 128 129 _zero_value = 0 130 131 def increment(self, timestamp: datetime) -> datetime: 132 return timestamp + timedelta(seconds=1) 133 134 def output_format(self, timestamp: datetime) -> int: 135 return int(timestamp.timestamp()) 136 137 def parse_timestamp(self, timestamp: int) -> datetime: 138 dt_object = AirbyteDateTime.fromtimestamp(timestamp, timezone.utc) 139 if not isinstance(dt_object, AirbyteDateTime): 140 raise ValueError( 141 f"AirbyteDateTime object was expected but got {type(dt_object)} from AirbyteDateTime.fromtimestamp({timestamp})" 142 ) 143 return dt_object
e.g. { "created": 1617030403 } => { "state_type": "date-range", "metadata": { … }, "slices": [ {starts: 0, end: 1617030403, finished_processing: true} ] }
131 def increment(self, timestamp: datetime) -> datetime: 132 return timestamp + timedelta(seconds=1)
Increment a timestamp by a single unit.
Convert the cursor value type to a JSON valid type.
137 def parse_timestamp(self, timestamp: int) -> datetime: 138 dt_object = AirbyteDateTime.fromtimestamp(timestamp, timezone.utc) 139 if not isinstance(dt_object, AirbyteDateTime): 140 raise ValueError( 141 f"AirbyteDateTime object was expected but got {type(dt_object)} from AirbyteDateTime.fromtimestamp({timestamp})" 142 ) 143 return dt_object
Inherited Members
118class FinalStateCursor(Cursor): 119 """Cursor that is used to guarantee at least one state message is emitted for a concurrent stream.""" 120 121 def __init__( 122 self, 123 stream_name: str, 124 stream_namespace: Optional[str], 125 message_repository: MessageRepository, 126 ) -> None: 127 self._stream_name = stream_name 128 self._stream_namespace = stream_namespace 129 self._message_repository = message_repository 130 # Normally the connector state manager operates at the source-level. However, we only need it to write the sentinel 131 # state message rather than manage overall source state. This is also only temporary as we move to the resumable 132 # full refresh world where every stream uses a FileBasedConcurrentCursor with incremental state. 133 self._connector_state_manager = ConnectorStateManager() 134 self._has_closed_at_least_one_slice = False 135 136 @property 137 def state(self) -> MutableMapping[str, Any]: 138 return {NO_CURSOR_STATE_KEY: True} 139 140 def observe(self, record: Record) -> None: 141 pass 142 143 def close_partition(self, partition: Partition) -> None: 144 pass 145 146 def ensure_at_least_one_state_emitted(self) -> None: 147 """ 148 Used primarily for full refresh syncs that do not have a valid cursor value to emit at the end of a sync 149 """ 150 151 self._connector_state_manager.update_state_for_stream( 152 self._stream_name, self._stream_namespace, self.state 153 ) 154 state_message = self._connector_state_manager.create_state_message( 155 self._stream_name, self._stream_namespace 156 ) 157 self._message_repository.emit_message(state_message) 158 159 def should_be_synced(self, record: Record) -> bool: 160 return True 161 162 def get_cursor_datetime_from_state( 163 self, stream_state: Mapping[str, Any] 164 ) -> datetime.datetime | None: 165 """Return now() if state indicates a completed full refresh, else None. 166 167 When the state has NO_CURSOR_STATE_KEY: True, it means the previous sync was a 168 completed full refresh. Returning now() indicates the cursor is "current" and 169 within any retention period, so we should use incremental sync. 170 171 For any other state format, return None to indicate this cursor cannot parse it, 172 allowing the incremental cursor to handle the state instead. 173 """ 174 if stream_state.get(NO_CURSOR_STATE_KEY): 175 return datetime.datetime.now(datetime.timezone.utc) 176 return None
Cursor that is used to guarantee at least one state message is emitted for a concurrent stream.
121 def __init__( 122 self, 123 stream_name: str, 124 stream_namespace: Optional[str], 125 message_repository: MessageRepository, 126 ) -> None: 127 self._stream_name = stream_name 128 self._stream_namespace = stream_namespace 129 self._message_repository = message_repository 130 # Normally the connector state manager operates at the source-level. However, we only need it to write the sentinel 131 # state message rather than manage overall source state. This is also only temporary as we move to the resumable 132 # full refresh world where every stream uses a FileBasedConcurrentCursor with incremental state. 133 self._connector_state_manager = ConnectorStateManager() 134 self._has_closed_at_least_one_slice = False
Indicate to the cursor that the partition has been successfully processed
146 def ensure_at_least_one_state_emitted(self) -> None: 147 """ 148 Used primarily for full refresh syncs that do not have a valid cursor value to emit at the end of a sync 149 """ 150 151 self._connector_state_manager.update_state_for_stream( 152 self._stream_name, self._stream_namespace, self.state 153 ) 154 state_message = self._connector_state_manager.create_state_message( 155 self._stream_name, self._stream_namespace 156 ) 157 self._message_repository.emit_message(state_message)
Used primarily for full refresh syncs that do not have a valid cursor value to emit at the end of a sync
162 def get_cursor_datetime_from_state( 163 self, stream_state: Mapping[str, Any] 164 ) -> datetime.datetime | None: 165 """Return now() if state indicates a completed full refresh, else None. 166 167 When the state has NO_CURSOR_STATE_KEY: True, it means the previous sync was a 168 completed full refresh. Returning now() indicates the cursor is "current" and 169 within any retention period, so we should use incremental sync. 170 171 For any other state format, return None to indicate this cursor cannot parse it, 172 allowing the incremental cursor to handle the state instead. 173 """ 174 if stream_state.get(NO_CURSOR_STATE_KEY): 175 return datetime.datetime.now(datetime.timezone.utc) 176 return None
Return now() if state indicates a completed full refresh, else None.
When the state has NO_CURSOR_STATE_KEY: True, it means the previous sync was a completed full refresh. Returning now() indicates the cursor is "current" and within any retention period, so we should use incremental sync.
For any other state format, return None to indicate this cursor cannot parse it, allowing the incremental cursor to handle the state instead.
Inherited Members
146class IsoMillisConcurrentStreamStateConverter(DateTimeStreamStateConverter): 147 """ 148 e.g. 149 { "created": "2021-01-18T21:18:20.000Z" } 150 => 151 { 152 "state_type": "date-range", 153 "metadata": { … }, 154 "slices": [ 155 {starts: "2020-01-18T21:18:20.000Z", end: "2021-01-18T21:18:20.000Z", finished_processing: true} 156 ] 157 } 158 """ 159 160 _zero_value = "0001-01-01T00:00:00.000Z" 161 162 def __init__( 163 self, is_sequential_state: bool = True, cursor_granularity: Optional[timedelta] = None 164 ): 165 super().__init__(is_sequential_state=is_sequential_state) 166 self._cursor_granularity = cursor_granularity or timedelta(milliseconds=1) 167 168 def increment(self, timestamp: datetime) -> datetime: 169 return timestamp + self._cursor_granularity 170 171 def output_format(self, timestamp: datetime) -> str: 172 """Format datetime with milliseconds always included. 173 174 Args: 175 timestamp: The datetime to format. 176 177 Returns: 178 str: ISO8601/RFC3339 formatted string with milliseconds. 179 """ 180 dt = AirbyteDateTime.from_datetime(timestamp) 181 # Always include milliseconds, even if zero 182 millis = dt.microsecond // 1000 if dt.microsecond else 0 183 return f"{dt.year:04d}-{dt.month:02d}-{dt.day:02d}T{dt.hour:02d}:{dt.minute:02d}:{dt.second:02d}.{millis:03d}Z" 184 185 def parse_timestamp(self, timestamp: str) -> datetime: 186 dt_object = ab_datetime_parse(timestamp) 187 if not isinstance(dt_object, AirbyteDateTime): 188 raise ValueError( 189 f"AirbyteDateTime object was expected but got {type(dt_object)} from parse({timestamp})" 190 ) 191 return dt_object
e.g. { "created": "2021-01-18T21:18:20.000Z" } => { "state_type": "date-range", "metadata": { … }, "slices": [ {starts: "2020-01-18T21:18:20.000Z", end: "2021-01-18T21:18:20.000Z", finished_processing: true} ] }
168 def increment(self, timestamp: datetime) -> datetime: 169 return timestamp + self._cursor_granularity
Increment a timestamp by a single unit.
171 def output_format(self, timestamp: datetime) -> str: 172 """Format datetime with milliseconds always included. 173 174 Args: 175 timestamp: The datetime to format. 176 177 Returns: 178 str: ISO8601/RFC3339 formatted string with milliseconds. 179 """ 180 dt = AirbyteDateTime.from_datetime(timestamp) 181 # Always include milliseconds, even if zero 182 millis = dt.microsecond // 1000 if dt.microsecond else 0 183 return f"{dt.year:04d}-{dt.month:02d}-{dt.day:02d}T{dt.hour:02d}:{dt.minute:02d}:{dt.second:02d}.{millis:03d}Z"
Format datetime with milliseconds always included.
Arguments:
- timestamp: The datetime to format.
Returns:
str: ISO8601/RFC3339 formatted string with milliseconds.
Inherited Members
49@deprecated( 50 "This class is experimental. Use at your own risk.", 51 category=ExperimentalClassWarning, 52) 53class StreamFacade(AbstractStreamFacade[DefaultStream], Stream): 54 """ 55 The StreamFacade is a Stream that wraps an AbstractStream and exposes it as a Stream. 56 57 All methods either delegate to the wrapped AbstractStream or provide a default implementation. 58 The default implementations define restrictions imposed on Streams migrated to the new interface. For instance, only source-defined cursors are supported. 59 """ 60 61 @classmethod 62 def create_from_stream( 63 cls, 64 stream: Stream, 65 source: AbstractSource, 66 logger: logging.Logger, 67 state: Optional[MutableMapping[str, Any]], 68 cursor: Cursor, 69 ) -> Stream: 70 """ 71 Create a ConcurrentStream from a Stream object. 72 :param source: The source 73 :param stream: The stream 74 :param max_workers: The maximum number of worker thread to use 75 :return: 76 """ 77 pk = get_primary_key_from_stream(stream.primary_key) 78 cursor_field = get_cursor_field_from_stream(stream) 79 80 if not source.message_repository: 81 raise ValueError( 82 "A message repository is required to emit non-record messages. Please set the message repository on the source." 83 ) 84 85 message_repository = source.message_repository 86 return StreamFacade( 87 DefaultStream( 88 partition_generator=StreamPartitionGenerator( 89 stream, 90 message_repository, 91 SyncMode.full_refresh 92 if isinstance(cursor, FinalStateCursor) 93 else SyncMode.incremental, 94 [cursor_field] if cursor_field is not None else None, 95 state, 96 ), 97 name=stream.name, 98 namespace=stream.namespace, 99 json_schema=stream.get_json_schema(), 100 primary_key=pk, 101 cursor_field=CursorField(cursor_field_key=cursor_field) if cursor_field else None, 102 logger=logger, 103 cursor=cursor, 104 ), 105 stream, 106 cursor, 107 slice_logger=source._slice_logger, 108 logger=logger, 109 ) 110 111 @property 112 def state(self) -> MutableMapping[str, Any]: 113 raise NotImplementedError( 114 "This should not be called as part of the Concurrent CDK code. Please report the problem to Airbyte" 115 ) 116 117 @state.setter 118 def state(self, value: Mapping[str, Any]) -> None: 119 if "state" in dir(self._legacy_stream): 120 self._legacy_stream.state = value # type: ignore # validating `state` is attribute of stream using `if` above 121 122 def __init__( 123 self, 124 stream: DefaultStream, 125 legacy_stream: Stream, 126 cursor: Cursor, 127 slice_logger: SliceLogger, 128 logger: logging.Logger, 129 ): 130 """ 131 :param stream: The underlying AbstractStream 132 """ 133 self._abstract_stream = stream 134 self._legacy_stream = legacy_stream 135 self._cursor = cursor 136 self._slice_logger = slice_logger 137 self._logger = logger 138 139 def read( 140 self, 141 configured_stream: ConfiguredAirbyteStream, 142 logger: logging.Logger, 143 slice_logger: SliceLogger, 144 stream_state: MutableMapping[str, Any], 145 state_manager: ConnectorStateManager, 146 internal_config: InternalConfig, 147 ) -> Iterable[StreamData]: 148 yield from self._read_records() 149 150 def read_records( 151 self, 152 sync_mode: SyncMode, 153 cursor_field: Optional[List[str]] = None, 154 stream_slice: Optional[Mapping[str, Any]] = None, 155 stream_state: Optional[Mapping[str, Any]] = None, 156 ) -> Iterable[StreamData]: 157 try: 158 yield from self._read_records() 159 except Exception as exc: 160 if hasattr(self._cursor, "state"): 161 state = str(self._cursor.state) 162 else: 163 # This shouldn't happen if the ConcurrentCursor was used 164 state = "unknown; no state attribute was available on the cursor" 165 yield AirbyteMessage( 166 type=Type.LOG, 167 log=AirbyteLogMessage( 168 level=Level.ERROR, message=f"Cursor State at time of exception: {state}" 169 ), 170 ) 171 raise exc 172 173 def _read_records(self) -> Iterable[StreamData]: 174 for partition in self._abstract_stream.generate_partitions(): 175 if self._slice_logger.should_log_slice_message(self._logger): 176 yield self._slice_logger.create_slice_log_message(partition.to_slice()) 177 for record in partition.read(): 178 yield record.data 179 180 @property 181 def name(self) -> str: 182 return self._abstract_stream.name 183 184 @property 185 def primary_key(self) -> Optional[Union[str, List[str], List[List[str]]]]: 186 # This method is not expected to be called directly. It is only implemented for backward compatibility with the old interface 187 return self.as_airbyte_stream().source_defined_primary_key # type: ignore # source_defined_primary_key is known to be an Optional[List[List[str]]] 188 189 @property 190 def cursor_field(self) -> Union[str, List[str]]: 191 if self._abstract_stream.cursor_field is None: 192 return [] 193 else: 194 return self._abstract_stream.cursor_field 195 196 @property 197 def cursor(self) -> Optional[Cursor]: # type: ignore[override] # StreamFaced expects to use only airbyte_cdk.sources.streams.concurrent.cursor.Cursor 198 return self._cursor 199 200 @property 201 def block_simultaneous_read(self) -> str: 202 """Returns the blocking group name from the underlying stream""" 203 return self._abstract_stream.block_simultaneous_read 204 205 # FIXME the lru_cache seems to be mostly there because of typing issue 206 @lru_cache(maxsize=None) 207 def get_json_schema(self) -> Mapping[str, Any]: 208 return self._abstract_stream.get_json_schema() 209 210 @property 211 def supports_incremental(self) -> bool: 212 return self._legacy_stream.supports_incremental 213 214 def as_airbyte_stream(self) -> AirbyteStream: 215 return self._abstract_stream.as_airbyte_stream() 216 217 def log_stream_sync_configuration(self) -> None: 218 self._abstract_stream.log_stream_sync_configuration() 219 220 def get_underlying_stream(self) -> DefaultStream: 221 return self._abstract_stream
The StreamFacade is a Stream that wraps an AbstractStream and exposes it as a Stream.
All methods either delegate to the wrapped AbstractStream or provide a default implementation. The default implementations define restrictions imposed on Streams migrated to the new interface. For instance, only source-defined cursors are supported.
122 def __init__( 123 self, 124 stream: DefaultStream, 125 legacy_stream: Stream, 126 cursor: Cursor, 127 slice_logger: SliceLogger, 128 logger: logging.Logger, 129 ): 130 """ 131 :param stream: The underlying AbstractStream 132 """ 133 self._abstract_stream = stream 134 self._legacy_stream = legacy_stream 135 self._cursor = cursor 136 self._slice_logger = slice_logger 137 self._logger = logger
Parameters
- stream: The underlying AbstractStream
61 @classmethod 62 def create_from_stream( 63 cls, 64 stream: Stream, 65 source: AbstractSource, 66 logger: logging.Logger, 67 state: Optional[MutableMapping[str, Any]], 68 cursor: Cursor, 69 ) -> Stream: 70 """ 71 Create a ConcurrentStream from a Stream object. 72 :param source: The source 73 :param stream: The stream 74 :param max_workers: The maximum number of worker thread to use 75 :return: 76 """ 77 pk = get_primary_key_from_stream(stream.primary_key) 78 cursor_field = get_cursor_field_from_stream(stream) 79 80 if not source.message_repository: 81 raise ValueError( 82 "A message repository is required to emit non-record messages. Please set the message repository on the source." 83 ) 84 85 message_repository = source.message_repository 86 return StreamFacade( 87 DefaultStream( 88 partition_generator=StreamPartitionGenerator( 89 stream, 90 message_repository, 91 SyncMode.full_refresh 92 if isinstance(cursor, FinalStateCursor) 93 else SyncMode.incremental, 94 [cursor_field] if cursor_field is not None else None, 95 state, 96 ), 97 name=stream.name, 98 namespace=stream.namespace, 99 json_schema=stream.get_json_schema(), 100 primary_key=pk, 101 cursor_field=CursorField(cursor_field_key=cursor_field) if cursor_field else None, 102 logger=logger, 103 cursor=cursor, 104 ), 105 stream, 106 cursor, 107 slice_logger=source._slice_logger, 108 logger=logger, 109 )
Create a ConcurrentStream from a Stream object.
Parameters
- source: The source
- stream: The stream
- max_workers: The maximum number of worker thread to use
Returns
200 @property 201 def block_simultaneous_read(self) -> str: 202 """Returns the blocking group name from the underlying stream""" 203 return self._abstract_stream.block_simultaneous_read
Returns the blocking group name from the underlying stream
Return the underlying stream facade object.
Inherited Members
- Stream
- logger
- transformer
- cursor
- has_multiple_slices
- name
- get_error_display_message
- read
- read_only_records
- read_records
- get_json_schema
- as_airbyte_stream
- supports_incremental
- is_resumable
- cursor_field
- namespace
- source_defined_cursor
- exit_on_rate_limit
- primary_key
- stream_slices
- state_checkpoint_interval
- get_updated_state
- get_cursor
- log_stream_sync_configuration
- configured_json_schema
106def create_connector_config_control_message(config: MutableMapping[str, Any]) -> AirbyteMessage: 107 control_message = AirbyteControlMessage( 108 type=OrchestratorType.CONNECTOR_CONFIG, 109 emitted_at=time.time() * 1000, 110 connectorConfig=AirbyteControlConnectorConfigMessage(config=config), 111 ) 112 return AirbyteMessage(type=Type.CONTROL, control=control_message)
90def emit_configuration_as_airbyte_control_message(config: MutableMapping[str, Any]) -> None: 91 """Print the connector config as a CONNECTOR_CONFIG control message directly to stdout. 92 93 Single-use OAuth refresh token rotation relies on this direct emission: it reaches the 94 platform for every command (including `check` and `discover`) and does not depend on a 95 message repository being drained. The payload and its newline are written in one `write()` 96 call so a concurrent writer to the shared stdout buffer cannot split the JSON line. 97 """ 98 airbyte_message = create_connector_config_control_message(config) 99 print( 100 f"{orjson.dumps(AirbyteMessageSerializer.dump(airbyte_message)).decode()}\n", 101 end="", 102 flush=True, 103 )
Print the connector config as a CONNECTOR_CONFIG control message directly to stdout.
Single-use OAuth refresh token rotation relies on this direct emission: it reaches the
platform for every command (including check and discover) and does not depend on a
message repository being drained. The payload and its newline are written in one write()
call so a concurrent writer to the shared stdout buffer cannot split the JSON line.
54class AbstractSource(Source, ABC): 55 """ 56 Abstract base class for an Airbyte Source. Consumers should implement any abstract methods 57 in this class to create an Airbyte Specification compliant Source. 58 """ 59 60 @abstractmethod 61 def check_connection( 62 self, logger: logging.Logger, config: Mapping[str, Any] 63 ) -> Tuple[bool, Optional[Any]]: 64 """ 65 :param logger: source logger 66 :param config: The user-provided configuration as specified by the source's spec. 67 This usually contains information required to check connection e.g. tokens, secrets and keys etc. 68 :return: A tuple of (boolean, error). If boolean is true, then the connection check is successful 69 and we can connect to the underlying data source using the provided configuration. 70 Otherwise, the input config cannot be used to connect to the underlying data source, 71 and the "error" object should describe what went wrong. 72 The error object will be cast to string to display the problem to the user. 73 """ 74 75 @abstractmethod 76 def streams(self, config: Mapping[str, Any]) -> List[Stream]: 77 """ 78 :param config: The user-provided configuration as specified by the source's spec. 79 Any stream construction related operation should happen here. 80 :return: A list of the streams in this source connector. 81 """ 82 83 # Stream name to instance map for applying output object transformation 84 _stream_to_instance_map: Dict[str, Stream] = {} 85 _slice_logger: SliceLogger = DebugSliceLogger() 86 87 def discover(self, logger: logging.Logger, config: Mapping[str, Any]) -> AirbyteCatalog: 88 """Implements the Discover operation from the Airbyte Specification. 89 See https://docs.airbyte.com/understanding-airbyte/airbyte-protocol/#discover. 90 """ 91 streams = [stream.as_airbyte_stream() for stream in self.streams(config=config)] 92 return AirbyteCatalog(streams=streams) 93 94 def check(self, logger: logging.Logger, config: Mapping[str, Any]) -> AirbyteConnectionStatus: 95 """Implements the Check Connection operation from the Airbyte Specification. 96 See https://docs.airbyte.com/understanding-airbyte/airbyte-protocol/#check. 97 """ 98 check_succeeded, error = self.check_connection(logger, config) 99 if not check_succeeded: 100 return AirbyteConnectionStatus(status=Status.FAILED, message=repr(error)) 101 return AirbyteConnectionStatus(status=Status.SUCCEEDED) 102 103 def check_section_connection( 104 self, logger: logging.Logger, section: str, config: Mapping[str, Any] 105 ) -> Tuple[bool, Optional[Any]]: 106 """ 107 `check_connection` for one section of the stepped setup form: `config` only holds the fields of `section` 108 and of the sections before it. Returns (True, None) when they are valid, (False, error) otherwise. 109 110 Override it in a source whose specification declares sections. 111 """ 112 return False, CHECK_SECTION_NOT_IMPLEMENTED.format(section=section) 113 114 def check_section( 115 self, logger: logging.Logger, section: str, config: Mapping[str, Any] 116 ) -> AirbyteConnectionStatus: 117 """Same as `check`, for one section: wraps `check_section_connection` into a connection status.""" 118 check_succeeded, error = self.check_section_connection(logger, section, config) 119 if not check_succeeded: 120 return AirbyteConnectionStatus(status=Status.FAILED, message=repr(error)) 121 return AirbyteConnectionStatus(status=Status.SUCCEEDED) 122 123 def read( 124 self, 125 logger: logging.Logger, 126 config: Mapping[str, Any], 127 catalog: ConfiguredAirbyteCatalog, 128 state: Optional[List[AirbyteStateMessage]] = None, 129 ) -> Iterator[AirbyteMessage]: 130 """Implements the Read operation from the Airbyte Specification. See https://docs.airbyte.com/understanding-airbyte/airbyte-protocol/.""" 131 logger.info(f"Starting syncing {self.name}") 132 config, internal_config = split_config(config) 133 # TODO assert all streams exist in the connector 134 # get the streams once in case the connector needs to make any queries to generate them 135 stream_instances = {s.name: s for s in self.streams(config)} 136 state_manager = ConnectorStateManager(state=state) 137 self._stream_to_instance_map = stream_instances 138 139 stream_name_to_exception: MutableMapping[str, AirbyteTracedException] = {} 140 141 with create_timer(self.name) as timer: 142 for configured_stream in catalog.streams: 143 stream_instance = stream_instances.get(configured_stream.stream.name) 144 is_stream_exist = bool(stream_instance) 145 try: 146 # Used direct reference to `stream_instance` instead of `is_stream_exist` to avoid mypy type checking errors 147 if not stream_instance: 148 if not self.raise_exception_on_missing_stream: 149 yield stream_status_as_airbyte_message( 150 configured_stream.stream, AirbyteStreamStatus.INCOMPLETE 151 ) 152 continue 153 154 error_message = ( 155 f"The stream '{configured_stream.stream.name}' in your connection configuration was not found in the source. " 156 f"Refresh the schema in your replication settings and remove this stream from future sync attempts." 157 ) 158 159 # Use configured_stream as stream_instance to support references in error handling. 160 stream_instance = configured_stream.stream 161 162 raise AirbyteTracedException( 163 message="A stream listed in your configuration was not found in the source. Please check the logs for more " 164 "details.", 165 internal_message=error_message, 166 failure_type=FailureType.config_error, 167 ) 168 169 timer.start_event(f"Syncing stream {configured_stream.stream.name}") 170 logger.info(f"Marking stream {configured_stream.stream.name} as STARTED") 171 yield stream_status_as_airbyte_message( 172 configured_stream.stream, AirbyteStreamStatus.STARTED 173 ) 174 yield from self._read_stream( 175 logger=logger, 176 stream_instance=stream_instance, 177 configured_stream=configured_stream, 178 state_manager=state_manager, 179 internal_config=internal_config, 180 ) 181 logger.info(f"Marking stream {configured_stream.stream.name} as STOPPED") 182 yield stream_status_as_airbyte_message( 183 configured_stream.stream, AirbyteStreamStatus.COMPLETE 184 ) 185 186 except Exception as e: 187 yield from self._emit_queued_messages() 188 logger.exception( 189 f"Encountered an exception while reading stream {configured_stream.stream.name}" 190 ) 191 logger.info(f"Marking stream {configured_stream.stream.name} as STOPPED") 192 yield stream_status_as_airbyte_message( 193 configured_stream.stream, AirbyteStreamStatus.INCOMPLETE 194 ) 195 196 stream_descriptor = StreamDescriptor(name=configured_stream.stream.name) 197 198 if isinstance(e, AirbyteTracedException): 199 traced_exception = e 200 info_message = f"Stopping sync on error from stream {configured_stream.stream.name} because {self.name} does not support continuing syncs on error." 201 else: 202 traced_exception = self._serialize_exception( 203 stream_descriptor, e, stream_instance=stream_instance 204 ) 205 info_message = f"{self.name} does not support continuing syncs on error from stream {configured_stream.stream.name}" 206 207 yield traced_exception.as_sanitized_airbyte_message( 208 stream_descriptor=stream_descriptor 209 ) 210 stream_name_to_exception[stream_instance.name] = traced_exception # type: ignore # use configured_stream if stream_instance is None 211 if self.stop_sync_on_stream_failure: 212 logger.info(info_message) 213 break 214 finally: 215 # Finish read event only if the stream instance exists; 216 # otherwise, there's no need as it never started 217 if is_stream_exist: 218 timer.finish_event() 219 logger.info(f"Finished syncing {configured_stream.stream.name}") 220 logger.info(timer.report()) 221 222 if len(stream_name_to_exception) > 0: 223 error_message = generate_failed_streams_error_message( 224 {key: [value] for key, value in stream_name_to_exception.items()} 225 ) 226 logger.info(error_message) 227 # We still raise at least one exception when a stream raises an exception because the platform currently relies 228 # on a non-zero exit code to determine if a sync attempt has failed. We also raise the exception as a config_error 229 # type because this combined error isn't actionable, but rather the previously emitted individual errors. 230 raise AirbyteTracedException( 231 message=error_message, failure_type=FailureType.config_error 232 ) 233 logger.info(f"Finished syncing {self.name}") 234 235 @staticmethod 236 def _serialize_exception( 237 stream_descriptor: StreamDescriptor, e: Exception, stream_instance: Optional[Stream] = None 238 ) -> AirbyteTracedException: 239 display_message = stream_instance.get_error_display_message(e) if stream_instance else None 240 if display_message: 241 return AirbyteTracedException.from_exception( 242 e, message=display_message, stream_descriptor=stream_descriptor 243 ) 244 return AirbyteTracedException.from_exception(e, stream_descriptor=stream_descriptor) 245 246 @property 247 def raise_exception_on_missing_stream(self) -> bool: 248 return False 249 250 def _read_stream( 251 self, 252 logger: logging.Logger, 253 stream_instance: Stream, 254 configured_stream: ConfiguredAirbyteStream, 255 state_manager: ConnectorStateManager, 256 internal_config: InternalConfig, 257 ) -> Iterator[AirbyteMessage]: 258 if internal_config.page_size and isinstance(stream_instance, HttpStream): 259 logger.info( 260 f"Setting page size for {stream_instance.name} to {internal_config.page_size}" 261 ) 262 stream_instance.page_size = internal_config.page_size 263 logger.debug( 264 f"Syncing configured stream: {configured_stream.stream.name}", 265 extra={ 266 "sync_mode": configured_stream.sync_mode, 267 "primary_key": configured_stream.primary_key, 268 "cursor_field": configured_stream.cursor_field, 269 }, 270 ) 271 stream_instance.log_stream_sync_configuration() 272 273 stream_name = configured_stream.stream.name 274 stream_state = state_manager.get_stream_state(stream_name, stream_instance.namespace) 275 276 # This is a hack. Existing full refresh streams that are converted into resumable full refresh need to discard 277 # the state because the terminal state for a full refresh sync is not compatible with substream resumable full 278 # refresh state. This is only required when running live traffic regression testing since the platform normally 279 # handles whether to pass state 280 if stream_state == {"__ab_no_cursor_state_message": True}: 281 stream_state = {} 282 283 if "state" in dir(stream_instance): 284 stream_instance.state = stream_state # type: ignore # we check that state in the dir(stream_instance) 285 logger.info(f"Setting state of {self.name} stream to {stream_state}") 286 287 record_iterator = stream_instance.read( 288 configured_stream, 289 logger, 290 self._slice_logger, 291 stream_state, 292 state_manager, 293 internal_config, 294 ) 295 296 record_counter = 0 297 logger.info(f"Syncing stream: {stream_name} ") 298 for record_data_or_message in record_iterator: 299 record = self._get_message(record_data_or_message, stream_instance) 300 if record.type == MessageType.RECORD: 301 record_counter += 1 302 if record_counter == 1: 303 logger.info(f"Marking stream {stream_name} as RUNNING") 304 # If we just read the first record of the stream, emit the transition to the RUNNING state 305 yield stream_status_as_airbyte_message( 306 configured_stream.stream, AirbyteStreamStatus.RUNNING 307 ) 308 yield from self._emit_queued_messages() 309 yield record 310 311 logger.info(f"Read {record_counter} records from {stream_name} stream") 312 313 def _emit_queued_messages(self) -> Iterable[AirbyteMessage]: 314 if self.message_repository: 315 yield from self.message_repository.consume_queue() 316 return 317 318 def _get_message( 319 self, record_data_or_message: Union[StreamData, AirbyteMessage], stream: Stream 320 ) -> AirbyteMessage: 321 """ 322 Converts the input to an AirbyteMessage if it is a StreamData. Returns the input as is if it is already an AirbyteMessage 323 """ 324 match record_data_or_message: 325 case AirbyteMessage(): 326 return record_data_or_message 327 case _: 328 return stream_data_to_airbyte_message( 329 stream.name, 330 record_data_or_message, 331 stream.transformer, 332 stream.get_json_schema(), 333 ) 334 335 @property 336 def message_repository(self) -> Union[None, MessageRepository]: 337 return _default_message_repository 338 339 @property 340 def stop_sync_on_stream_failure(self) -> bool: 341 """ 342 WARNING: This function is in-development which means it is subject to change. Use at your own risk. 343 344 By default, when a source encounters an exception while syncing a stream, it will emit an error trace message and then 345 continue syncing the next stream. This can be overwritten on a per-source basis so that the source will stop the sync 346 on the first error seen and emit a single error trace message for that stream. 347 """ 348 return False
Abstract base class for an Airbyte Source. Consumers should implement any abstract methods in this class to create an Airbyte Specification compliant Source.
60 @abstractmethod 61 def check_connection( 62 self, logger: logging.Logger, config: Mapping[str, Any] 63 ) -> Tuple[bool, Optional[Any]]: 64 """ 65 :param logger: source logger 66 :param config: The user-provided configuration as specified by the source's spec. 67 This usually contains information required to check connection e.g. tokens, secrets and keys etc. 68 :return: A tuple of (boolean, error). If boolean is true, then the connection check is successful 69 and we can connect to the underlying data source using the provided configuration. 70 Otherwise, the input config cannot be used to connect to the underlying data source, 71 and the "error" object should describe what went wrong. 72 The error object will be cast to string to display the problem to the user. 73 """
Parameters
- logger: source logger
- config: The user-provided configuration as specified by the source's spec. This usually contains information required to check connection e.g. tokens, secrets and keys etc.
Returns
A tuple of (boolean, error). If boolean is true, then the connection check is successful and we can connect to the underlying data source using the provided configuration. Otherwise, the input config cannot be used to connect to the underlying data source, and the "error" object should describe what went wrong. The error object will be cast to string to display the problem to the user.
75 @abstractmethod 76 def streams(self, config: Mapping[str, Any]) -> List[Stream]: 77 """ 78 :param config: The user-provided configuration as specified by the source's spec. 79 Any stream construction related operation should happen here. 80 :return: A list of the streams in this source connector. 81 """
Parameters
- config: The user-provided configuration as specified by the source's spec. Any stream construction related operation should happen here.
Returns
A list of the streams in this source connector.
87 def discover(self, logger: logging.Logger, config: Mapping[str, Any]) -> AirbyteCatalog: 88 """Implements the Discover operation from the Airbyte Specification. 89 See https://docs.airbyte.com/understanding-airbyte/airbyte-protocol/#discover. 90 """ 91 streams = [stream.as_airbyte_stream() for stream in self.streams(config=config)] 92 return AirbyteCatalog(streams=streams)
Implements the Discover operation from the Airbyte Specification. See https://docs.airbyte.com/understanding-airbyte/airbyte-protocol/#discover.
94 def check(self, logger: logging.Logger, config: Mapping[str, Any]) -> AirbyteConnectionStatus: 95 """Implements the Check Connection operation from the Airbyte Specification. 96 See https://docs.airbyte.com/understanding-airbyte/airbyte-protocol/#check. 97 """ 98 check_succeeded, error = self.check_connection(logger, config) 99 if not check_succeeded: 100 return AirbyteConnectionStatus(status=Status.FAILED, message=repr(error)) 101 return AirbyteConnectionStatus(status=Status.SUCCEEDED)
Implements the Check Connection operation from the Airbyte Specification. See https://docs.airbyte.com/understanding-airbyte/airbyte-protocol/#check.
103 def check_section_connection( 104 self, logger: logging.Logger, section: str, config: Mapping[str, Any] 105 ) -> Tuple[bool, Optional[Any]]: 106 """ 107 `check_connection` for one section of the stepped setup form: `config` only holds the fields of `section` 108 and of the sections before it. Returns (True, None) when they are valid, (False, error) otherwise. 109 110 Override it in a source whose specification declares sections. 111 """ 112 return False, CHECK_SECTION_NOT_IMPLEMENTED.format(section=section)
check_connection for one section of the stepped setup form: config only holds the fields of section
and of the sections before it. Returns (True, None) when they are valid, (False, error) otherwise.
Override it in a source whose specification declares sections.
114 def check_section( 115 self, logger: logging.Logger, section: str, config: Mapping[str, Any] 116 ) -> AirbyteConnectionStatus: 117 """Same as `check`, for one section: wraps `check_section_connection` into a connection status.""" 118 check_succeeded, error = self.check_section_connection(logger, section, config) 119 if not check_succeeded: 120 return AirbyteConnectionStatus(status=Status.FAILED, message=repr(error)) 121 return AirbyteConnectionStatus(status=Status.SUCCEEDED)
Same as check, for one section: wraps check_section_connection into a connection status.
123 def read( 124 self, 125 logger: logging.Logger, 126 config: Mapping[str, Any], 127 catalog: ConfiguredAirbyteCatalog, 128 state: Optional[List[AirbyteStateMessage]] = None, 129 ) -> Iterator[AirbyteMessage]: 130 """Implements the Read operation from the Airbyte Specification. See https://docs.airbyte.com/understanding-airbyte/airbyte-protocol/.""" 131 logger.info(f"Starting syncing {self.name}") 132 config, internal_config = split_config(config) 133 # TODO assert all streams exist in the connector 134 # get the streams once in case the connector needs to make any queries to generate them 135 stream_instances = {s.name: s for s in self.streams(config)} 136 state_manager = ConnectorStateManager(state=state) 137 self._stream_to_instance_map = stream_instances 138 139 stream_name_to_exception: MutableMapping[str, AirbyteTracedException] = {} 140 141 with create_timer(self.name) as timer: 142 for configured_stream in catalog.streams: 143 stream_instance = stream_instances.get(configured_stream.stream.name) 144 is_stream_exist = bool(stream_instance) 145 try: 146 # Used direct reference to `stream_instance` instead of `is_stream_exist` to avoid mypy type checking errors 147 if not stream_instance: 148 if not self.raise_exception_on_missing_stream: 149 yield stream_status_as_airbyte_message( 150 configured_stream.stream, AirbyteStreamStatus.INCOMPLETE 151 ) 152 continue 153 154 error_message = ( 155 f"The stream '{configured_stream.stream.name}' in your connection configuration was not found in the source. " 156 f"Refresh the schema in your replication settings and remove this stream from future sync attempts." 157 ) 158 159 # Use configured_stream as stream_instance to support references in error handling. 160 stream_instance = configured_stream.stream 161 162 raise AirbyteTracedException( 163 message="A stream listed in your configuration was not found in the source. Please check the logs for more " 164 "details.", 165 internal_message=error_message, 166 failure_type=FailureType.config_error, 167 ) 168 169 timer.start_event(f"Syncing stream {configured_stream.stream.name}") 170 logger.info(f"Marking stream {configured_stream.stream.name} as STARTED") 171 yield stream_status_as_airbyte_message( 172 configured_stream.stream, AirbyteStreamStatus.STARTED 173 ) 174 yield from self._read_stream( 175 logger=logger, 176 stream_instance=stream_instance, 177 configured_stream=configured_stream, 178 state_manager=state_manager, 179 internal_config=internal_config, 180 ) 181 logger.info(f"Marking stream {configured_stream.stream.name} as STOPPED") 182 yield stream_status_as_airbyte_message( 183 configured_stream.stream, AirbyteStreamStatus.COMPLETE 184 ) 185 186 except Exception as e: 187 yield from self._emit_queued_messages() 188 logger.exception( 189 f"Encountered an exception while reading stream {configured_stream.stream.name}" 190 ) 191 logger.info(f"Marking stream {configured_stream.stream.name} as STOPPED") 192 yield stream_status_as_airbyte_message( 193 configured_stream.stream, AirbyteStreamStatus.INCOMPLETE 194 ) 195 196 stream_descriptor = StreamDescriptor(name=configured_stream.stream.name) 197 198 if isinstance(e, AirbyteTracedException): 199 traced_exception = e 200 info_message = f"Stopping sync on error from stream {configured_stream.stream.name} because {self.name} does not support continuing syncs on error." 201 else: 202 traced_exception = self._serialize_exception( 203 stream_descriptor, e, stream_instance=stream_instance 204 ) 205 info_message = f"{self.name} does not support continuing syncs on error from stream {configured_stream.stream.name}" 206 207 yield traced_exception.as_sanitized_airbyte_message( 208 stream_descriptor=stream_descriptor 209 ) 210 stream_name_to_exception[stream_instance.name] = traced_exception # type: ignore # use configured_stream if stream_instance is None 211 if self.stop_sync_on_stream_failure: 212 logger.info(info_message) 213 break 214 finally: 215 # Finish read event only if the stream instance exists; 216 # otherwise, there's no need as it never started 217 if is_stream_exist: 218 timer.finish_event() 219 logger.info(f"Finished syncing {configured_stream.stream.name}") 220 logger.info(timer.report()) 221 222 if len(stream_name_to_exception) > 0: 223 error_message = generate_failed_streams_error_message( 224 {key: [value] for key, value in stream_name_to_exception.items()} 225 ) 226 logger.info(error_message) 227 # We still raise at least one exception when a stream raises an exception because the platform currently relies 228 # on a non-zero exit code to determine if a sync attempt has failed. We also raise the exception as a config_error 229 # type because this combined error isn't actionable, but rather the previously emitted individual errors. 230 raise AirbyteTracedException( 231 message=error_message, failure_type=FailureType.config_error 232 ) 233 logger.info(f"Finished syncing {self.name}")
Implements the Read operation from the Airbyte Specification. See https://docs.airbyte.com/understanding-airbyte/airbyte-protocol/.
339 @property 340 def stop_sync_on_stream_failure(self) -> bool: 341 """ 342 WARNING: This function is in-development which means it is subject to change. Use at your own risk. 343 344 By default, when a source encounters an exception while syncing a stream, it will emit an error trace message and then 345 continue syncing the next stream. This can be overwritten on a per-source basis so that the source will stop the sync 346 on the first error seen and emit a single error trace message for that stream. 347 """ 348 return False
WARNING: This function is in-development which means it is subject to change. Use at your own risk.
By default, when a source encounters an exception while syncing a stream, it will emit an error trace message and then continue syncing the next stream. This can be overwritten on a per-source basis so that the source will stop the sync on the first error seen and emit a single error trace message for that stream.
13class BaseConfig(BaseModel): 14 """Base class for connector spec, adds the following behaviour: 15 16 - resolve $ref and replace it with definition 17 - replace all occurrences of anyOf with oneOf 18 - drop description 19 """ 20 21 @classmethod 22 def schema(cls, *args: Any, **kwargs: Any) -> Dict[str, Any]: 23 """We're overriding the schema classmethod to enable some post-processing""" 24 schema = super().schema(*args, **kwargs) 25 rename_key(schema, old_key="anyOf", new_key="oneOf") # UI supports only oneOf 26 expand_refs(schema) 27 schema.pop("description", None) # description added from the docstring 28 return schema
Base class for connector spec, adds the following behaviour:
- resolve $ref and replace it with definition
- replace all occurrences of anyOf with oneOf
- drop description
21 @classmethod 22 def schema(cls, *args: Any, **kwargs: Any) -> Dict[str, Any]: 23 """We're overriding the schema classmethod to enable some post-processing""" 24 schema = super().schema(*args, **kwargs) 25 rename_key(schema, old_key="anyOf", new_key="oneOf") # UI supports only oneOf 26 expand_refs(schema) 27 schema.pop("description", None) # description added from the docstring 28 return schema
We're overriding the schema classmethod to enable some post-processing
40class BaseConnector(ABC, Generic[TConfig]): 41 # configure whether the `check_config_against_spec_or_exit()` needs to be called 42 check_config_against_spec: bool = True 43 44 @abstractmethod 45 def configure(self, config: Mapping[str, Any], temp_dir: str) -> TConfig: 46 """ 47 Persist config in temporary directory to run the Source job 48 """ 49 50 @staticmethod 51 def read_config(config_path: str) -> MutableMapping[str, Any]: 52 config = BaseConnector._read_json_file(config_path) 53 if isinstance(config, MutableMapping): 54 return config 55 else: 56 raise ValueError( 57 f"The content of {config_path} is not an object and therefore is not a valid config. Please ensure the file represent a config." 58 ) 59 60 @staticmethod 61 def _read_json_file(file_path: str) -> Any: 62 with open(file_path, "r") as file: 63 contents = file.read() 64 65 try: 66 return json.loads(contents) 67 except json.JSONDecodeError as error: 68 raise ValueError( 69 f"Could not read json file {file_path}: {error}. Please ensure that it is a valid JSON." 70 ) 71 72 @staticmethod 73 def write_config(config: TConfig, config_path: str) -> None: 74 with open(config_path, "w") as fh: 75 fh.write(json.dumps(config)) 76 77 def spec(self, logger: logging.Logger) -> ConnectorSpecification: 78 """ 79 Returns the spec for this integration. The spec is a JSON-Schema object describing the required configurations (e.g: username and password) 80 required to run this integration. By default, this will be loaded from a "spec.yaml" or a "spec.json" in the package root. 81 """ 82 83 package = self.__class__.__module__.split(".")[0] 84 85 yaml_spec = load_optional_package_file(package, "spec.yaml") 86 json_spec = load_optional_package_file(package, "spec.json") 87 88 if yaml_spec and json_spec: 89 raise RuntimeError( 90 "Found multiple spec files in the package. Only one of spec.yaml or spec.json should be provided." 91 ) 92 93 if yaml_spec: 94 spec_obj = yaml.load(yaml_spec, Loader=yaml.SafeLoader) 95 elif json_spec: 96 try: 97 spec_obj = json.loads(json_spec) 98 except json.JSONDecodeError as error: 99 raise ValueError( 100 f"Could not read json spec file: {error}. Please ensure that it is a valid JSON." 101 ) 102 else: 103 raise FileNotFoundError("Unable to find spec.yaml or spec.json in the package.") 104 105 return ConnectorSpecificationSerializer.load(spec_obj) 106 107 @abstractmethod 108 def check(self, logger: logging.Logger, config: TConfig) -> AirbyteConnectionStatus: 109 """ 110 Tests if the input configuration can be used to successfully connect to the integration e.g: if a provided Stripe API token can be used to connect 111 to the Stripe API. 112 """ 113 114 def check_section( 115 self, logger: logging.Logger, section: str, config: TConfig 116 ) -> AirbyteConnectionStatus: 117 """ 118 Checks the configuration up to `section` only, for the stepped setup form 119 (see `airbyte_cdk.sources.utils.check_section`). The entrypoint calls this instead of `check` when the config 120 carries a `check-section` key naming a declared section other than the last one; `config` then only holds 121 the fields of that section and of the sections before it. 122 123 Override it in a connector whose specification declares sections. 124 """ 125 return AirbyteConnectionStatus( 126 status=Status.FAILED, message=CHECK_SECTION_NOT_IMPLEMENTED.format(section=section) 127 )
Helper class that provides a standard way to create an ABC using inheritance.
44 @abstractmethod 45 def configure(self, config: Mapping[str, Any], temp_dir: str) -> TConfig: 46 """ 47 Persist config in temporary directory to run the Source job 48 """
Persist config in temporary directory to run the Source job
50 @staticmethod 51 def read_config(config_path: str) -> MutableMapping[str, Any]: 52 config = BaseConnector._read_json_file(config_path) 53 if isinstance(config, MutableMapping): 54 return config 55 else: 56 raise ValueError( 57 f"The content of {config_path} is not an object and therefore is not a valid config. Please ensure the file represent a config." 58 )
77 def spec(self, logger: logging.Logger) -> ConnectorSpecification: 78 """ 79 Returns the spec for this integration. The spec is a JSON-Schema object describing the required configurations (e.g: username and password) 80 required to run this integration. By default, this will be loaded from a "spec.yaml" or a "spec.json" in the package root. 81 """ 82 83 package = self.__class__.__module__.split(".")[0] 84 85 yaml_spec = load_optional_package_file(package, "spec.yaml") 86 json_spec = load_optional_package_file(package, "spec.json") 87 88 if yaml_spec and json_spec: 89 raise RuntimeError( 90 "Found multiple spec files in the package. Only one of spec.yaml or spec.json should be provided." 91 ) 92 93 if yaml_spec: 94 spec_obj = yaml.load(yaml_spec, Loader=yaml.SafeLoader) 95 elif json_spec: 96 try: 97 spec_obj = json.loads(json_spec) 98 except json.JSONDecodeError as error: 99 raise ValueError( 100 f"Could not read json spec file: {error}. Please ensure that it is a valid JSON." 101 ) 102 else: 103 raise FileNotFoundError("Unable to find spec.yaml or spec.json in the package.") 104 105 return ConnectorSpecificationSerializer.load(spec_obj)
Returns the spec for this integration. The spec is a JSON-Schema object describing the required configurations (e.g: username and password) required to run this integration. By default, this will be loaded from a "spec.yaml" or a "spec.json" in the package root.
107 @abstractmethod 108 def check(self, logger: logging.Logger, config: TConfig) -> AirbyteConnectionStatus: 109 """ 110 Tests if the input configuration can be used to successfully connect to the integration e.g: if a provided Stripe API token can be used to connect 111 to the Stripe API. 112 """
Tests if the input configuration can be used to successfully connect to the integration e.g: if a provided Stripe API token can be used to connect to the Stripe API.
114 def check_section( 115 self, logger: logging.Logger, section: str, config: TConfig 116 ) -> AirbyteConnectionStatus: 117 """ 118 Checks the configuration up to `section` only, for the stepped setup form 119 (see `airbyte_cdk.sources.utils.check_section`). The entrypoint calls this instead of `check` when the config 120 carries a `check-section` key naming a declared section other than the last one; `config` then only holds 121 the fields of that section and of the sections before it. 122 123 Override it in a connector whose specification declares sections. 124 """ 125 return AirbyteConnectionStatus( 126 status=Status.FAILED, message=CHECK_SECTION_NOT_IMPLEMENTED.format(section=section) 127 )
Checks the configuration up to section only, for the stepped setup form
(see airbyte_cdk.sources.utils.check_section). The entrypoint calls this instead of check when the config
carries a check-section key naming a declared section other than the last one; config then only holds
the fields of that section and of the sections before it.
Override it in a connector whose specification declares sections.
145class Connector(DefaultConnectorMixin, BaseConnector[Mapping[str, Any]], ABC): ...
Helper class that provides a standard way to create an ABC using inheritance.
31class Destination(Connector, ABC): 32 VALID_CMDS = {"spec", "check", "write"} 33 34 @abstractmethod 35 def write( 36 self, 37 config: Mapping[str, Any], 38 configured_catalog: ConfiguredAirbyteCatalog, 39 input_messages: Iterable[AirbyteMessage], 40 ) -> Iterable[AirbyteMessage]: 41 """Implement to define how the connector writes data to the destination""" 42 43 def _run_check(self, config: Mapping[str, Any]) -> AirbyteMessage: 44 check_result = self.check(logger, config) 45 return AirbyteMessage(type=Type.CONNECTION_STATUS, connectionStatus=check_result) 46 47 def _parse_input_stream(self, input_stream: io.TextIOWrapper) -> Iterable[AirbyteMessage]: 48 """Reads from stdin, converting to Airbyte messages""" 49 for line in input_stream: 50 try: 51 yield AirbyteMessageSerializer.load(orjson.loads(line)) 52 except orjson.JSONDecodeError: 53 logger.info( 54 f"ignoring input which can't be deserialized as Airbyte Message: {line}" 55 ) 56 57 def _run_write( 58 self, 59 config: Mapping[str, Any], 60 configured_catalog_path: str, 61 input_stream: io.TextIOWrapper, 62 ) -> Iterable[AirbyteMessage]: 63 catalog = ConfiguredAirbyteCatalogSerializer.load( 64 orjson.loads(open(configured_catalog_path).read()) 65 ) 66 input_messages = self._parse_input_stream(input_stream) 67 logger.info("Begin writing to the destination...") 68 yield from self.write( 69 config=config, configured_catalog=catalog, input_messages=input_messages 70 ) 71 logger.info("Writing complete.") 72 73 def parse_args(self, args: List[str]) -> argparse.Namespace: 74 """ 75 :param args: commandline arguments 76 :return: 77 """ 78 79 parent_parser = argparse.ArgumentParser(add_help=False) 80 main_parser = argparse.ArgumentParser() 81 subparsers = main_parser.add_subparsers(title="commands", dest="command") 82 83 # spec 84 subparsers.add_parser( 85 "spec", help="outputs the json configuration specification", parents=[parent_parser] 86 ) 87 88 # check 89 check_parser = subparsers.add_parser( 90 "check", help="checks the config can be used to connect", parents=[parent_parser] 91 ) 92 required_check_parser = check_parser.add_argument_group("required named arguments") 93 required_check_parser.add_argument( 94 "--config", type=str, required=True, help="path to the json configuration file" 95 ) 96 97 # write 98 write_parser = subparsers.add_parser( 99 "write", help="Writes data to the destination", parents=[parent_parser] 100 ) 101 write_required = write_parser.add_argument_group("required named arguments") 102 write_required.add_argument( 103 "--config", type=str, required=True, help="path to the JSON configuration file" 104 ) 105 write_required.add_argument( 106 "--catalog", type=str, required=True, help="path to the configured catalog JSON file" 107 ) 108 109 parsed_args = main_parser.parse_args(args) 110 cmd = parsed_args.command 111 if not cmd: 112 raise Exception("No command entered. ") 113 elif cmd not in ["spec", "check", "write"]: 114 # This is technically dead code since parse_args() would fail if this was the case 115 # But it's non-obvious enough to warrant placing it here anyways 116 raise Exception(f"Unknown command entered: {cmd}") 117 118 return parsed_args 119 120 def run_cmd(self, parsed_args: argparse.Namespace) -> Iterable[AirbyteMessage]: 121 cmd = parsed_args.command 122 if cmd not in self.VALID_CMDS: 123 raise Exception(f"Unrecognized command: {cmd}") 124 125 spec = self.spec(logger) 126 if cmd == "spec": 127 yield AirbyteMessage(type=Type.SPEC, spec=spec) 128 return 129 config = self.read_config(config_path=parsed_args.config) 130 # Register the config's secrets with the log filter before anything can print them, 131 # as the source entrypoint does. A config validation error quotes config values. 132 update_secrets(get_secrets(spec.connectionSpecification, config)) 133 if self.check_config_against_spec or cmd == "check": 134 try: 135 check_config_against_spec_or_exit(config, spec) 136 except AirbyteTracedException as traced_exc: 137 connection_status = traced_exc.as_connection_status_message() 138 if connection_status and cmd == "check": 139 yield connection_status 140 return 141 raise traced_exc 142 143 if cmd == "check": 144 yield self._run_check(config=config) 145 elif cmd == "write": 146 # Wrap in UTF-8 to override any other input encodings 147 wrapped_stdin = io.TextIOWrapper(sys.stdin.buffer, encoding="utf-8") 148 yield from self._run_write( 149 config=config, 150 configured_catalog_path=parsed_args.catalog, 151 input_stream=wrapped_stdin, 152 ) 153 154 def run(self, args: List[str]) -> None: 155 init_uncaught_exception_handler(logger) 156 parsed_args = self.parse_args(args) 157 output_messages = self.run_cmd(parsed_args) 158 for message in output_messages: 159 print(orjson.dumps(AirbyteMessageSerializer.dump(message)).decode())
Helper class that provides a standard way to create an ABC using inheritance.
34 @abstractmethod 35 def write( 36 self, 37 config: Mapping[str, Any], 38 configured_catalog: ConfiguredAirbyteCatalog, 39 input_messages: Iterable[AirbyteMessage], 40 ) -> Iterable[AirbyteMessage]: 41 """Implement to define how the connector writes data to the destination"""
Implement to define how the connector writes data to the destination
73 def parse_args(self, args: List[str]) -> argparse.Namespace: 74 """ 75 :param args: commandline arguments 76 :return: 77 """ 78 79 parent_parser = argparse.ArgumentParser(add_help=False) 80 main_parser = argparse.ArgumentParser() 81 subparsers = main_parser.add_subparsers(title="commands", dest="command") 82 83 # spec 84 subparsers.add_parser( 85 "spec", help="outputs the json configuration specification", parents=[parent_parser] 86 ) 87 88 # check 89 check_parser = subparsers.add_parser( 90 "check", help="checks the config can be used to connect", parents=[parent_parser] 91 ) 92 required_check_parser = check_parser.add_argument_group("required named arguments") 93 required_check_parser.add_argument( 94 "--config", type=str, required=True, help="path to the json configuration file" 95 ) 96 97 # write 98 write_parser = subparsers.add_parser( 99 "write", help="Writes data to the destination", parents=[parent_parser] 100 ) 101 write_required = write_parser.add_argument_group("required named arguments") 102 write_required.add_argument( 103 "--config", type=str, required=True, help="path to the JSON configuration file" 104 ) 105 write_required.add_argument( 106 "--catalog", type=str, required=True, help="path to the configured catalog JSON file" 107 ) 108 109 parsed_args = main_parser.parse_args(args) 110 cmd = parsed_args.command 111 if not cmd: 112 raise Exception("No command entered. ") 113 elif cmd not in ["spec", "check", "write"]: 114 # This is technically dead code since parse_args() would fail if this was the case 115 # But it's non-obvious enough to warrant placing it here anyways 116 raise Exception(f"Unknown command entered: {cmd}") 117 118 return parsed_args
Parameters
- args: commandline arguments
Returns
120 def run_cmd(self, parsed_args: argparse.Namespace) -> Iterable[AirbyteMessage]: 121 cmd = parsed_args.command 122 if cmd not in self.VALID_CMDS: 123 raise Exception(f"Unrecognized command: {cmd}") 124 125 spec = self.spec(logger) 126 if cmd == "spec": 127 yield AirbyteMessage(type=Type.SPEC, spec=spec) 128 return 129 config = self.read_config(config_path=parsed_args.config) 130 # Register the config's secrets with the log filter before anything can print them, 131 # as the source entrypoint does. A config validation error quotes config values. 132 update_secrets(get_secrets(spec.connectionSpecification, config)) 133 if self.check_config_against_spec or cmd == "check": 134 try: 135 check_config_against_spec_or_exit(config, spec) 136 except AirbyteTracedException as traced_exc: 137 connection_status = traced_exc.as_connection_status_message() 138 if connection_status and cmd == "check": 139 yield connection_status 140 return 141 raise traced_exc 142 143 if cmd == "check": 144 yield self._run_check(config=config) 145 elif cmd == "write": 146 # Wrap in UTF-8 to override any other input encodings 147 wrapped_stdin = io.TextIOWrapper(sys.stdin.buffer, encoding="utf-8") 148 yield from self._run_write( 149 config=config, 150 configured_catalog_path=parsed_args.catalog, 151 input_stream=wrapped_stdin, 152 )
56class Source( 57 DefaultConnectorMixin, 58 BaseSource[Mapping[str, Any], List[AirbyteStateMessage], ConfiguredAirbyteCatalog], 59 ABC, 60): 61 # can be overridden to change an input state. 62 @classmethod 63 def read_state(cls, state_path: str) -> List[AirbyteStateMessage]: 64 """ 65 Retrieves the input state of a sync by reading from the specified JSON file. Incoming state can be deserialized into either 66 a JSON object for legacy state input or as a list of AirbyteStateMessages for the per-stream state format. Regardless of the 67 incoming input type, it will always be transformed and output as a list of AirbyteStateMessage(s). 68 :param state_path: The filepath to where the stream states are located 69 :return: The complete stream state based on the connector's previous sync 70 """ 71 parsed_state_messages = [] 72 if state_path: 73 state_obj = BaseConnector._read_json_file(state_path) 74 if state_obj: 75 for state in state_obj: # type: ignore # `isinstance(state_obj, List)` ensures that this is a list 76 parsed_message = AirbyteStateMessageSerializer.load(state) 77 if ( 78 not parsed_message.stream 79 and not parsed_message.data 80 and not parsed_message.global_ 81 ): 82 raise ValueError( 83 "AirbyteStateMessage should contain either a stream, global, or state field" 84 ) 85 parsed_state_messages.append(parsed_message) 86 return parsed_state_messages 87 88 # can be overridden to change an input catalog 89 @classmethod 90 def read_catalog(cls, catalog_path: str) -> ConfiguredAirbyteCatalog: 91 return ConfiguredAirbyteCatalogSerializer.load(cls._read_json_file(catalog_path)) 92 93 @property 94 def name(self) -> str: 95 """Source name""" 96 return self.__class__.__name__
Helper class that provides a standard way to create an ABC using inheritance.
62 @classmethod 63 def read_state(cls, state_path: str) -> List[AirbyteStateMessage]: 64 """ 65 Retrieves the input state of a sync by reading from the specified JSON file. Incoming state can be deserialized into either 66 a JSON object for legacy state input or as a list of AirbyteStateMessages for the per-stream state format. Regardless of the 67 incoming input type, it will always be transformed and output as a list of AirbyteStateMessage(s). 68 :param state_path: The filepath to where the stream states are located 69 :return: The complete stream state based on the connector's previous sync 70 """ 71 parsed_state_messages = [] 72 if state_path: 73 state_obj = BaseConnector._read_json_file(state_path) 74 if state_obj: 75 for state in state_obj: # type: ignore # `isinstance(state_obj, List)` ensures that this is a list 76 parsed_message = AirbyteStateMessageSerializer.load(state) 77 if ( 78 not parsed_message.stream 79 and not parsed_message.data 80 and not parsed_message.global_ 81 ): 82 raise ValueError( 83 "AirbyteStateMessage should contain either a stream, global, or state field" 84 ) 85 parsed_state_messages.append(parsed_message) 86 return parsed_state_messages
Retrieves the input state of a sync by reading from the specified JSON file. Incoming state can be deserialized into either a JSON object for legacy state input or as a list of AirbyteStateMessages for the per-stream state format. Regardless of the incoming input type, it will always be transformed and output as a list of AirbyteStateMessage(s).
Parameters
- state_path: The filepath to where the stream states are located
Returns
The complete stream state based on the connector's previous sync
37@dataclass 38class AddFields(RecordTransformation): 39 """ 40 Transformation which adds field to an output record. The path of the added field can be nested. Adding nested fields will create all 41 necessary parent objects (like mkdir -p). Adding fields to an array will extend the array to that index (filling intermediate 42 indices with null values). So if you add a field at index 5 to the array ["value"], it will become ["value", null, null, null, null, 43 "new_value"]. 44 45 46 This transformation has access to the following contextual values: 47 record: the record about to be output by the connector 48 config: the input configuration provided to a connector 49 stream_state: the current state of the stream 50 stream_slice: the current stream slice being read 51 52 53 54 Examples of instantiating this transformation via YAML: 55 - type: AddFields 56 fields: 57 # hardcoded constant 58 - path: ["path"] 59 value: "static_value" 60 61 # nested path 62 - path: ["path", "to", "field"] 63 value: "static" 64 65 # from config 66 - path: ["shop_id"] 67 value: "{{ config.shop_id }}" 68 69 # from stream_interval 70 - path: ["date"] 71 value: "{{ stream_interval.start_date }}" 72 73 # from record 74 - path: ["unnested_value"] 75 value: {{ record.nested.field }} 76 77 # from stream_slice 78 - path: ["start_date"] 79 value: {{ stream_slice.start_date }} 80 81 # by supplying any valid Jinja template directive or expression https://jinja.palletsprojects.com/en/3.1.x/templates/# 82 - path: ["two_times_two"] 83 value: {{ 2 * 2 }} 84 85 Attributes: 86 fields (List[AddedFieldDefinition]): A list of transformations (path and corresponding value) that will be added to the record 87 """ 88 89 fields: List[AddedFieldDefinition] 90 parameters: InitVar[Mapping[str, Any]] 91 condition: str = "" 92 _parsed_fields: List[ParsedAddFieldDefinition] = field( 93 init=False, repr=False, default_factory=list 94 ) 95 96 def __post_init__(self, parameters: Mapping[str, Any]) -> None: 97 self._filter_interpolator = InterpolatedBoolean( 98 condition=self.condition, parameters=parameters 99 ) 100 101 for add_field in self.fields: 102 if len(add_field.path) < 1: 103 raise ValueError( 104 f"Expected a non-zero-length path for the AddFields transformation {add_field}" 105 ) 106 107 if not isinstance(add_field.value, InterpolatedString): 108 if not isinstance(add_field.value, str): 109 raise f"Expected a string value for the AddFields transformation: {add_field}" 110 else: 111 self._parsed_fields.append( 112 ParsedAddFieldDefinition( 113 add_field.path, 114 InterpolatedString.create(add_field.value, parameters=parameters), 115 value_type=add_field.value_type, 116 parameters=parameters, 117 ) 118 ) 119 else: 120 self._parsed_fields.append( 121 ParsedAddFieldDefinition( 122 add_field.path, 123 add_field.value, 124 value_type=add_field.value_type, 125 parameters={}, 126 ) 127 ) 128 129 def transform( 130 self, 131 record: Dict[str, Any], 132 config: Optional[Config] = None, 133 stream_state: Optional[StreamState] = None, 134 stream_slice: Optional[StreamSlice] = None, 135 ) -> None: 136 if config is None: 137 config = {} 138 kwargs = {"record": record, "stream_slice": stream_slice} 139 for parsed_field in self._parsed_fields: 140 valid_types = (parsed_field.value_type,) if parsed_field.value_type else None 141 value = parsed_field.value.eval(config, valid_types=valid_types, **kwargs) 142 is_empty_condition = not self.condition 143 if is_empty_condition or self._filter_interpolator.eval( 144 config, value=value, path=parsed_field.path, **kwargs 145 ): 146 dpath.new(record, parsed_field.path, value) 147 148 def __eq__(self, other: Any) -> bool: 149 return bool(self.__dict__ == other.__dict__)
Transformation which adds field to an output record. The path of the added field can be nested. Adding nested fields will create all necessary parent objects (like mkdir -p). Adding fields to an array will extend the array to that index (filling intermediate indices with null values). So if you add a field at index 5 to the array ["value"], it will become ["value", null, null, null, null, "new_value"].
This transformation has access to the following contextual values:
record: the record about to be output by the connector config: the input configuration provided to a connector stream_state: the current state of the stream stream_slice: the current stream slice being read
Examples of instantiating this transformation via YAML:
- type: AddFields
fields:
# hardcoded constant
- path: ["path"] value: "static_value"
# nested path
- path: ["path", "to", "field"]
value: "static"
# from config
- path: ["shop_id"]
value: "{{ config.shop_id }}"
# from stream_interval
- path: ["date"]
value: "{{ stream_interval.start_date }}"
# from record
- path: ["unnested_value"]
value: {{ record.nested.field }}
# from stream_slice
- path: ["start_date"]
value: {{ stream_slice.start_date }}
# by supplying any valid Jinja template directive or expression https://jinja.palletsprojects.com/en/3.1.x/templates/#
- path: ["two_times_two"]
value: {{ 2 * 2 }}
Attributes:
- fields (List[AddedFieldDefinition]): A list of transformations (path and corresponding value) that will be added to the record
129 def transform( 130 self, 131 record: Dict[str, Any], 132 config: Optional[Config] = None, 133 stream_state: Optional[StreamState] = None, 134 stream_slice: Optional[StreamSlice] = None, 135 ) -> None: 136 if config is None: 137 config = {} 138 kwargs = {"record": record, "stream_slice": stream_slice} 139 for parsed_field in self._parsed_fields: 140 valid_types = (parsed_field.value_type,) if parsed_field.value_type else None 141 value = parsed_field.value.eval(config, valid_types=valid_types, **kwargs) 142 is_empty_condition = not self.condition 143 if is_empty_condition or self._filter_interpolator.eval( 144 config, value=value, path=parsed_field.path, **kwargs 145 ): 146 dpath.new(record, parsed_field.path, value)
Transform a record by adding, deleting, or mutating fields directly from the record reference passed in argument.
Parameters
- record: The input record to be transformed
- config: The user-provided configuration as specified by the source's spec
- stream_state: The stream state
- stream_slice: The stream slice
Returns
The transformed record
17@dataclass(frozen=True) 18class AddedFieldDefinition: 19 """Defines the field to add on a record""" 20 21 path: FieldPointer 22 value: Union[InterpolatedString, str] 23 value_type: Optional[Type[Any]] 24 parameters: InitVar[Mapping[str, Any]]
Defines the field to add on a record
24@dataclass 25class ApiKeyAuthenticator(DeclarativeAuthenticator): 26 """ 27 ApiKeyAuth sets a request header on the HTTP requests sent. 28 29 The header is of the form: 30 `"<header>": "<token>"` 31 32 For example, 33 `ApiKeyAuthenticator("Authorization", "Bearer hello")` 34 will result in the following header set on the HTTP request 35 `"Authorization": "Bearer hello"` 36 37 Attributes: 38 request_option (RequestOption): request option how to inject the token into the request 39 token_provider (TokenProvider): Provider of the token 40 config (Config): The user-provided configuration as specified by the source's spec 41 parameters (Mapping[str, Any]): Additional runtime parameters to be used for string interpolation 42 """ 43 44 request_option: RequestOption 45 token_provider: TokenProvider 46 config: Config 47 parameters: InitVar[Mapping[str, Any]] 48 49 @property 50 def auth_header(self) -> str: 51 options = self._get_request_options(RequestOptionType.header) 52 return next(iter(options.keys()), "") 53 54 @property 55 def token(self) -> str: 56 return self.token_provider.get_token() 57 58 def _get_request_options(self, option_type: RequestOptionType) -> Mapping[str, Any]: 59 options: MutableMapping[str, Any] = {} 60 if self.request_option.inject_into == option_type: 61 self.request_option.inject_into_request(options, self.token, self.config) 62 return options 63 64 def get_request_params(self) -> Mapping[str, Any]: 65 return self._get_request_options(RequestOptionType.request_parameter) 66 67 def get_request_body_data(self) -> Union[Mapping[str, Any], str]: 68 return self._get_request_options(RequestOptionType.body_data) 69 70 def get_request_body_json(self) -> Mapping[str, Any]: 71 return self._get_request_options(RequestOptionType.body_json)
ApiKeyAuth sets a request header on the HTTP requests sent.
The header is of the form:
"<header>": "<token>"
For example,
ApiKeyAuthenticator("Authorization", "Bearer hello")
will result in the following header set on the HTTP request
"Authorization": "Bearer hello"
Attributes:
- request_option (RequestOption): request option how to inject the token into the request
- token_provider (TokenProvider): Provider of the token
- config (Config): The user-provided configuration as specified by the source's spec
- parameters (Mapping[str, Any]): Additional runtime parameters to be used for string interpolation
49 @property 50 def auth_header(self) -> str: 51 options = self._get_request_options(RequestOptionType.header) 52 return next(iter(options.keys()), "")
HTTP header to set on the requests
64 def get_request_params(self) -> Mapping[str, Any]: 65 return self._get_request_options(RequestOptionType.request_parameter)
HTTP request parameter to add to the requests
67 def get_request_body_data(self) -> Union[Mapping[str, Any], str]: 68 return self._get_request_options(RequestOptionType.body_data)
Form-encoded body data to set on the requests
70 def get_request_body_json(self) -> Mapping[str, Any]: 71 return self._get_request_options(RequestOptionType.body_json)
JSON-encoded body data to set on the requests
Inherited Members
12class BackoffStrategy(ABC): 13 @abstractmethod 14 def backoff_time( 15 self, 16 response_or_exception: Optional[Union[requests.Response, requests.RequestException]], 17 attempt_count: int, 18 ) -> Optional[float]: 19 """ 20 Override this method to dynamically determine backoff time e.g: by reading the X-Retry-After header. 21 22 Not called for every retryable response. `HttpClient` skips the strategies entirely when a 23 rate-limited response can be retried on another credential -- the authenticator says so via 24 `TokenRotatingAuthenticator.has_alternative_token` -- because the wait computed here is 25 derived from the credential that was rejected, and the retry will not use it. Implementations 26 must therefore not rely on being called for side effects such as counting attempts or 27 emitting metrics. 28 29 :param response_or_exception: The response or exception that caused the backoff. 30 :param attempt_count: The number of attempts already performed for this request. 31 :return how long to backoff in seconds. The return value may be a floating point number for subsecond precision. Returning None defers backoff 32 to the default backoff behavior (e.g using an exponential algorithm). 33 """ 34 pass
Helper class that provides a standard way to create an ABC using inheritance.
13 @abstractmethod 14 def backoff_time( 15 self, 16 response_or_exception: Optional[Union[requests.Response, requests.RequestException]], 17 attempt_count: int, 18 ) -> Optional[float]: 19 """ 20 Override this method to dynamically determine backoff time e.g: by reading the X-Retry-After header. 21 22 Not called for every retryable response. `HttpClient` skips the strategies entirely when a 23 rate-limited response can be retried on another credential -- the authenticator says so via 24 `TokenRotatingAuthenticator.has_alternative_token` -- because the wait computed here is 25 derived from the credential that was rejected, and the retry will not use it. Implementations 26 must therefore not rely on being called for side effects such as counting attempts or 27 emitting metrics. 28 29 :param response_or_exception: The response or exception that caused the backoff. 30 :param attempt_count: The number of attempts already performed for this request. 31 :return how long to backoff in seconds. The return value may be a floating point number for subsecond precision. Returning None defers backoff 32 to the default backoff behavior (e.g using an exponential algorithm). 33 """ 34 pass
Override this method to dynamically determine backoff time e.g: by reading the X-Retry-After header.
Not called for every retryable response. HttpClient skips the strategies entirely when a
rate-limited response can be retried on another credential -- the authenticator says so via
TokenRotatingAuthenticator.has_alternative_token -- because the wait computed here is
derived from the credential that was rejected, and the retry will not use it. Implementations
must therefore not rely on being called for side effects such as counting attempts or
emitting metrics.
Parameters
- response_or_exception: The response or exception that caused the backoff.
- attempt_count: The number of attempts already performed for this request. :return how long to backoff in seconds. The return value may be a floating point number for subsecond precision. Returning None defers backoff to the default backoff behavior (e.g using an exponential algorithm).
101@dataclass 102class BasicHttpAuthenticator(DeclarativeAuthenticator): 103 """ 104 Builds auth based off the basic authentication scheme as defined by RFC 7617, which transmits credentials as USER ID/password pairs, encoded using base64 105 https://developer.mozilla.org/en-US/docs/Web/HTTP/Authentication#basic_authentication_scheme 106 107 The header is of the form 108 `"Authorization": "Basic <encoded_credentials>"` 109 110 Attributes: 111 username (Union[InterpolatedString, str]): The username 112 config (Config): The user-provided configuration as specified by the source's spec 113 password (Union[InterpolatedString, str]): The password 114 parameters (Mapping[str, Any]): Additional runtime parameters to be used for string interpolation 115 """ 116 117 username: Union[InterpolatedString, str] 118 config: Config 119 parameters: InitVar[Mapping[str, Any]] 120 password: Union[InterpolatedString, str] = "" 121 122 def __post_init__(self, parameters: Mapping[str, Any]) -> None: 123 self._username = InterpolatedString.create(self.username, parameters=parameters) 124 self._password = InterpolatedString.create(self.password, parameters=parameters) 125 126 @property 127 def auth_header(self) -> str: 128 return "Authorization" 129 130 @property 131 def token(self) -> str: 132 auth_string = ( 133 f"{self._username.eval(self.config)}:{self._password.eval(self.config)}".encode("utf8") 134 ) 135 b64_encoded = base64.b64encode(auth_string).decode("utf8") 136 return f"Basic {b64_encoded}"
Builds auth based off the basic authentication scheme as defined by RFC 7617, which transmits credentials as USER ID/password pairs, encoded using base64 https://developer.mozilla.org/en-US/docs/Web/HTTP/Authentication#basic_authentication_scheme
The header is of the form
"Authorization": "Basic <encoded_credentials>"
Attributes:
- username (Union[InterpolatedString, str]): The username
- config (Config): The user-provided configuration as specified by the source's spec
- password (Union[InterpolatedString, str]): The password
- parameters (Mapping[str, Any]): Additional runtime parameters to be used for string interpolation
130 @property 131 def token(self) -> str: 132 auth_string = ( 133 f"{self._username.eval(self.config)}:{self._password.eval(self.config)}".encode("utf8") 134 ) 135 b64_encoded = base64.b64encode(auth_string).decode("utf8") 136 return f"Basic {b64_encoded}"
The header value to set on outgoing HTTP requests
74@dataclass 75class BearerAuthenticator(DeclarativeAuthenticator): 76 """ 77 Authenticator that sets the Authorization header on the HTTP requests sent. 78 79 The header is of the form: 80 `"Authorization": "Bearer <token>"` 81 82 Attributes: 83 token_provider (TokenProvider): Provider of the token 84 config (Config): The user-provided configuration as specified by the source's spec 85 parameters (Mapping[str, Any]): Additional runtime parameters to be used for string interpolation 86 """ 87 88 token_provider: TokenProvider 89 config: Config 90 parameters: InitVar[Mapping[str, Any]] 91 92 @property 93 def auth_header(self) -> str: 94 return "Authorization" 95 96 @property 97 def token(self) -> str: 98 return f"Bearer {self.token_provider.get_token()}"
Authenticator that sets the Authorization header on the HTTP requests sent.
The header is of the form:
"Authorization": "Bearer <token>"
Attributes:
- token_provider (TokenProvider): Provider of the token
- config (Config): The user-provided configuration as specified by the source's spec
- parameters (Mapping[str, Any]): Additional runtime parameters to be used for string interpolation
40@dataclass 41class CartesianProductStreamSlicer(PartitionRouter): 42 """ 43 Stream slicers that iterates over the cartesian product of input stream slicers 44 Given 2 stream slicers with the following slices: 45 A: [{"i": 0}, {"i": 1}, {"i": 2}] 46 B: [{"s": "hello"}, {"s": "world"}] 47 the resulting stream slices are 48 [ 49 {"i": 0, "s": "hello"}, 50 {"i": 0, "s": "world"}, 51 {"i": 1, "s": "hello"}, 52 {"i": 1, "s": "world"}, 53 {"i": 2, "s": "hello"}, 54 {"i": 2, "s": "world"}, 55 ] 56 57 Attributes: 58 stream_slicers (List[PartitionRouter]): Underlying stream slicers. The RequestOptions (e.g: Request headers, parameters, etc..) returned by this slicer are the combination of the RequestOptions of its input slicers. If there are conflicts e.g: two slicers define the same header or request param, the conflict is resolved by taking the value from the first slicer, where ordering is determined by the order in which slicers were input to this composite slicer. 59 """ 60 61 stream_slicers: List[PartitionRouter] 62 parameters: InitVar[Mapping[str, Any]] 63 64 def __post_init__(self, parameters: Mapping[str, Any]) -> None: 65 check_for_substream_in_slicers(self.stream_slicers, self.logger.warning) 66 67 def get_request_params( 68 self, 69 *, 70 stream_state: Optional[StreamState] = None, 71 stream_slice: Optional[StreamSlice] = None, 72 next_page_token: Optional[Mapping[str, Any]] = None, 73 ) -> Mapping[str, Any]: 74 return dict( 75 ChainMap( 76 *[ # type: ignore # ChainMap expects a MutableMapping[Never, Never] for reasons 77 s.get_request_params( 78 stream_state=stream_state, 79 stream_slice=stream_slice, 80 next_page_token=next_page_token, 81 ) 82 for s in self.stream_slicers 83 ] 84 ) 85 ) 86 87 def get_request_headers( 88 self, 89 *, 90 stream_state: Optional[StreamState] = None, 91 stream_slice: Optional[StreamSlice] = None, 92 next_page_token: Optional[Mapping[str, Any]] = None, 93 ) -> Mapping[str, Any]: 94 return dict( 95 ChainMap( 96 *[ # type: ignore # ChainMap expects a MutableMapping[Never, Never] for reasons 97 s.get_request_headers( 98 stream_state=stream_state, 99 stream_slice=stream_slice, 100 next_page_token=next_page_token, 101 ) 102 for s in self.stream_slicers 103 ] 104 ) 105 ) 106 107 def get_request_body_data( 108 self, 109 *, 110 stream_state: Optional[StreamState] = None, 111 stream_slice: Optional[StreamSlice] = None, 112 next_page_token: Optional[Mapping[str, Any]] = None, 113 ) -> Mapping[str, Any]: 114 return dict( 115 ChainMap( 116 *[ # type: ignore # ChainMap expects a MutableMapping[Never, Never] for reasons 117 s.get_request_body_data( 118 stream_state=stream_state, 119 stream_slice=stream_slice, 120 next_page_token=next_page_token, 121 ) 122 for s in self.stream_slicers 123 ] 124 ) 125 ) 126 127 def get_request_body_json( 128 self, 129 *, 130 stream_state: Optional[StreamState] = None, 131 stream_slice: Optional[StreamSlice] = None, 132 next_page_token: Optional[Mapping[str, Any]] = None, 133 ) -> Mapping[str, Any]: 134 return dict( 135 ChainMap( 136 *[ # type: ignore # ChainMap expects a MutableMapping[Never, Never] for reasons 137 s.get_request_body_json( 138 stream_state=stream_state, 139 stream_slice=stream_slice, 140 next_page_token=next_page_token, 141 ) 142 for s in self.stream_slicers 143 ] 144 ) 145 ) 146 147 def stream_slices(self) -> Iterable[StreamSlice]: 148 sub_slices = (s.stream_slices() for s in self.stream_slicers) 149 product = itertools.product(*sub_slices) 150 for stream_slice_tuple in product: 151 partition = dict(ChainMap(*[s.partition for s in stream_slice_tuple])) # type: ignore # ChainMap expects a MutableMapping[Never, Never] for reasons 152 cursor_slices = [s.cursor_slice for s in stream_slice_tuple if s.cursor_slice] 153 extra_fields = dict(ChainMap(*[s.extra_fields for s in stream_slice_tuple])) # type: ignore # ChainMap expects a MutableMapping[Never, Never] for reasons 154 if len(cursor_slices) > 1: 155 raise ValueError( 156 f"There should only be a single cursor slice. Found {cursor_slices}" 157 ) 158 if cursor_slices: 159 cursor_slice = cursor_slices[0] 160 else: 161 cursor_slice = {} 162 yield StreamSlice( 163 partition=partition, cursor_slice=cursor_slice, extra_fields=extra_fields 164 ) 165 166 def get_stream_state(self) -> Optional[Mapping[str, StreamState]]: 167 """ 168 Parent stream states are not supported for cartesian product stream slicer 169 """ 170 pass 171 172 @property 173 def logger(self) -> logging.Logger: 174 return logging.getLogger("airbyte.CartesianProductStreamSlicer")
Stream slicers that iterates over the cartesian product of input stream slicers Given 2 stream slicers with the following slices: A: [{"i": 0}, {"i": 1}, {"i": 2}] B: [{"s": "hello"}, {"s": "world"}] the resulting stream slices are [ {"i": 0, "s": "hello"}, {"i": 0, "s": "world"}, {"i": 1, "s": "hello"}, {"i": 1, "s": "world"}, {"i": 2, "s": "hello"}, {"i": 2, "s": "world"}, ]
Attributes:
- stream_slicers (List[PartitionRouter]): Underlying stream slicers. The RequestOptions (e.g: Request headers, parameters, etc..) returned by this slicer are the combination of the RequestOptions of its input slicers. If there are conflicts e.g: two slicers define the same header or request param, the conflict is resolved by taking the value from the first slicer, where ordering is determined by the order in which slicers were input to this composite slicer.
67 def get_request_params( 68 self, 69 *, 70 stream_state: Optional[StreamState] = None, 71 stream_slice: Optional[StreamSlice] = None, 72 next_page_token: Optional[Mapping[str, Any]] = None, 73 ) -> Mapping[str, Any]: 74 return dict( 75 ChainMap( 76 *[ # type: ignore # ChainMap expects a MutableMapping[Never, Never] for reasons 77 s.get_request_params( 78 stream_state=stream_state, 79 stream_slice=stream_slice, 80 next_page_token=next_page_token, 81 ) 82 for s in self.stream_slicers 83 ] 84 ) 85 )
Specifies the query parameters that should be set on an outgoing HTTP request given the inputs.
E.g: you might want to define query parameters for paging if next_page_token is not None.
87 def get_request_headers( 88 self, 89 *, 90 stream_state: Optional[StreamState] = None, 91 stream_slice: Optional[StreamSlice] = None, 92 next_page_token: Optional[Mapping[str, Any]] = None, 93 ) -> Mapping[str, Any]: 94 return dict( 95 ChainMap( 96 *[ # type: ignore # ChainMap expects a MutableMapping[Never, Never] for reasons 97 s.get_request_headers( 98 stream_state=stream_state, 99 stream_slice=stream_slice, 100 next_page_token=next_page_token, 101 ) 102 for s in self.stream_slicers 103 ] 104 ) 105 )
Return any non-auth headers. Authentication headers will overwrite any overlapping headers returned from this method.
107 def get_request_body_data( 108 self, 109 *, 110 stream_state: Optional[StreamState] = None, 111 stream_slice: Optional[StreamSlice] = None, 112 next_page_token: Optional[Mapping[str, Any]] = None, 113 ) -> Mapping[str, Any]: 114 return dict( 115 ChainMap( 116 *[ # type: ignore # ChainMap expects a MutableMapping[Never, Never] for reasons 117 s.get_request_body_data( 118 stream_state=stream_state, 119 stream_slice=stream_slice, 120 next_page_token=next_page_token, 121 ) 122 for s in self.stream_slicers 123 ] 124 ) 125 )
Specifies how to populate the body of the request with a non-JSON payload.
If returns a ready text that it will be sent as is. If returns a dict that it will be converted to a urlencoded form. E.g. {"key1": "value1", "key2": "value2"} => "key1=value1&key2=value2"
At the same time only one of the 'request_body_data' and 'request_body_json' functions can be overridden.
127 def get_request_body_json( 128 self, 129 *, 130 stream_state: Optional[StreamState] = None, 131 stream_slice: Optional[StreamSlice] = None, 132 next_page_token: Optional[Mapping[str, Any]] = None, 133 ) -> Mapping[str, Any]: 134 return dict( 135 ChainMap( 136 *[ # type: ignore # ChainMap expects a MutableMapping[Never, Never] for reasons 137 s.get_request_body_json( 138 stream_state=stream_state, 139 stream_slice=stream_slice, 140 next_page_token=next_page_token, 141 ) 142 for s in self.stream_slicers 143 ] 144 ) 145 )
Specifies how to populate the body of the request with a JSON payload.
At the same time only one of the 'request_body_data' and 'request_body_json' functions can be overridden.
147 def stream_slices(self) -> Iterable[StreamSlice]: 148 sub_slices = (s.stream_slices() for s in self.stream_slicers) 149 product = itertools.product(*sub_slices) 150 for stream_slice_tuple in product: 151 partition = dict(ChainMap(*[s.partition for s in stream_slice_tuple])) # type: ignore # ChainMap expects a MutableMapping[Never, Never] for reasons 152 cursor_slices = [s.cursor_slice for s in stream_slice_tuple if s.cursor_slice] 153 extra_fields = dict(ChainMap(*[s.extra_fields for s in stream_slice_tuple])) # type: ignore # ChainMap expects a MutableMapping[Never, Never] for reasons 154 if len(cursor_slices) > 1: 155 raise ValueError( 156 f"There should only be a single cursor slice. Found {cursor_slices}" 157 ) 158 if cursor_slices: 159 cursor_slice = cursor_slices[0] 160 else: 161 cursor_slice = {} 162 yield StreamSlice( 163 partition=partition, cursor_slice=cursor_slice, extra_fields=extra_fields 164 )
Defines stream slices
Returns
An iterable of stream slices
166 def get_stream_state(self) -> Optional[Mapping[str, StreamState]]: 167 """ 168 Parent stream states are not supported for cartesian product stream slicer 169 """ 170 pass
Parent stream states are not supported for cartesian product stream slicer
19class CombineMode(Enum): 20 """ 21 How a `CombinedExtractor` combines the output of its sub-extractors. 22 """ 23 24 union = "union" 25 first_match = "first_match"
How a CombinedExtractor combines the output of its sub-extractors.
46@dataclass 47class CombinedExtractor(RecordExtractor): 48 """Combines the output of several record extractors into a single stream of records. 49 50 A single `DpathExtractor` can only describe one path into the response. Several connectors need 51 more than that and reach for a custom `components.py` today, which keeps them out of the 52 Connector Builder. This component covers the two shapes those connectors implement: 53 54 - `union` (default) — yield every record of every sub-extractor, in sub-extractor order. Use it 55 when one response carries records under several paths, for example a GraphQL document that 56 returns `issues.nodes` and `pullRequests.nodes` side by side. 57 - `first_match` — yield the records of the first sub-extractor that produces at least one 58 record, and skip the remaining sub-extractors. If no sub-extractor produces a record, nothing 59 is yielded. Use it when an API answers with one of several alternative shapes. The winning 60 sub-extractor is only peeked at, never restarted: the peeked record is chained back in front 61 of the remaining ones, so large responses are still streamed lazily and no sub-extractor is 62 ever materialized into a list. 63 64 Examples of instantiating this component: 65 ``` 66 extractor: 67 type: CombinedExtractor 68 mode: union 69 extractors: 70 - type: DpathExtractor 71 field_path: ["data", "repository", "issues", "nodes"] 72 - type: DpathExtractor 73 field_path: ["data", "repository", "pullRequests", "nodes"] 74 ``` 75 76 ``` 77 extractor: 78 type: CombinedExtractor 79 mode: first_match 80 extractors: 81 - type: DpathExtractor 82 field_path: ["data", "boards", "*", "items_page", "items", "*"] 83 - type: DpathExtractor 84 field_path: ["data", "next_items_page", "items"] 85 ``` 86 87 ## Streaming decoders are rejected 88 89 Every sub-extractor is handed the same `requests.Response`, so the response body must be 90 readable more than once. That holds for the decoders that buffer the whole body, which is the 91 common case: `JsonDecoder` and any `CompositeRawDecoder` built with `stream_response=False` 92 read `response.content`, `XmlDecoder` reads `response.text` and `ZipfileDecoder` reads 93 `response.content` — `requests` caches all of those, so every sub-extractor sees the full body. 94 95 It does NOT hold for streaming decoders, i.e. a `CompositeRawDecoder` with 96 `stream_response=True`, which is what `CsvDecoder`, `JsonlDecoder`, `JsonItemsDecoder` and 97 `GzipDecoder` resolve to outside the Connector Builder, nor for `IterableDecoder`. Those read 98 `response.raw` and close it afterwards, so the first sub-extractor drains the stream and the 99 later ones read a dead body. That loss is SILENT rather than loud: `requests` puts a 100 `urllib3.HTTPResponse` in `response.raw`, and reading a closed `urllib3.HTTPResponse` returns 101 an empty body instead of raising, so `union` emits only the first sub-extractor's records and 102 `first_match` emits nothing at all when the first path misses. 103 104 Because of that, `ModelToComponentFactory.create_combined_extractor` refuses to build this 105 component over a streaming decoder and raises a configuration error, naming the decoder in the 106 internal message. The Connector Builder forces those same decoders to `stream_response=False`, 107 so the rejection is applied there too rather than letting a manifest test-read correctly and 108 lose records once published. 109 110 ## Empty records 111 112 `skip_empty_records` drops falsy records — `None`, `{}`, `[]`, `""` — before any of the modes 113 sees them. It is off by default, so a sub-extractor's output is passed through verbatim. 114 115 Turn it on when an API can answer with nulls in the middle of a record list. A GraphQL API 116 that returns a partial response puts `null` in the `data` array for the fields it failed to 117 resolve and describes the failure in a sibling `errors` field; without this flag those nulls 118 reach the record stream and fail schema validation downstream. 119 120 Under `first_match` the flag also changes which sub-extractor wins: a sub-extractor whose 121 records are all empty no longer counts as a match, so the next sub-extractor is tried. That is 122 the behaviour of `source-monday`'s `MondayIncrementalItemsExtractor`, where a page of nulls 123 must fall through to the pagination path rather than lock in the primary path. 124 125 The dropped records still count toward the size of the page, see "Record-counting paginators" 126 below. 127 128 ## Cost 129 130 Each sub-extractor decodes the response independently — `CompositeRawDecoder` re-parses the 131 cached body on every `decode()` call — so a response is parsed once per sub-extractor. 132 Combining three extractors over a large response costs roughly three times the decode time of 133 a single one. 134 135 ## Record-counting paginators 136 137 The count `OffsetIncrement` and `PageIncrement` obtain is the combined count, not the API's 138 page size: 139 140 - `union` returns the SUM over all sub-extractors. An `OffsetIncrement` advances the offset by 141 that sum, so the next page starts past the records that were never read — two sub-extractors 142 returning two records each with `page_size: 2` request offsets 0, 4, 8 instead of 0, 2, 4 and 143 silently drop two thirds of the records. `ModelToComponentFactory.create_offset_increment` 144 therefore rejects a `union` `CombinedExtractor` (nested ones included) with a configuration 145 error. A `PageIncrement` only compares the count against `page_size` to decide whether to 146 stop, so it loses nothing, but it issues one extra request whenever the summed count of the 147 last page happens to equal `page_size`. 148 - `first_match` returns the count of the winning sub-extractor, which does not inflate the 149 count, so it works with either paginator as long as the path the count comes from is the one 150 the API paginates over. 151 152 Empty records dropped by `skip_empty_records` are part of that count. The API returned them as 153 slots of the page, so a full page holding a null must still read as full: counting only the 154 surviving records would make both paginators stop at the first page with a null, and would 155 make an `OffsetIncrement` re-request records it already read. The factory sets 156 `count_dropped_empty_records` on the extractor of a `RecordSelector`, which makes it yield a 157 placeholder for every empty record it drops; the `RecordSelector` counts the placeholders and 158 discards them. The winner under `first_match` is still chosen by the surviving records alone, 159 so the placeholders never change which records are emitted. 160 161 Attributes: 162 extractors (List[RecordExtractor]): The sub-extractors to combine. At least one is required. 163 mode (CombineMode): How the sub-extractor outputs are combined. Defaults to `union`. 164 skip_empty_records (bool): Whether falsy records are dropped before they are combined. 165 Defaults to `False`. 166 count_dropped_empty_records (bool): Set by the factory on the extractor of a 167 `RecordSelector`, which discards the placeholders: yield a placeholder in place of 168 every dropped empty record so it counts toward the page size. Defaults to `False`. 169 """ 170 171 extractors: List[RecordExtractor] 172 parameters: InitVar[Mapping[str, Any]] 173 mode: CombineMode = CombineMode.union 174 skip_empty_records: bool = False 175 count_dropped_empty_records: bool = False 176 177 def __post_init__(self, parameters: Mapping[str, Any]) -> None: 178 if not self.extractors: 179 raise ValueError( 180 "CombinedExtractor requires at least one extractor in its `extractors` field." 181 ) 182 if not isinstance(self.mode, CombineMode): 183 self.mode = CombineMode(self.mode) 184 185 def extract_records(self, response: requests.Response) -> Iterable[Mapping[str, Any]]: 186 if self.mode == CombineMode.first_match: 187 yield from self._extract_first_match(response) 188 else: 189 yield from self._extract_union(response) 190 191 def _extract_union(self, response: requests.Response) -> Iterable[Mapping[str, Any]]: 192 for index in range(len(self.extractors)): 193 yield from self._records_of(index, response) 194 195 def _extract_first_match(self, response: requests.Response) -> Iterable[Mapping[str, Any]]: 196 for index in range(len(self.extractors)): 197 records: Iterator[Mapping[str, Any]] = iter(self._records_of(index, response)) 198 # Placeholders for dropped empty records do not make a sub-extractor win, but they are 199 # kept so the winner's count still includes the ones that came before its first record. 200 dropped_placeholders: List[Mapping[str, Any]] = [] 201 first_record: Union[Mapping[str, Any], _NoRecord] = _NO_RECORD 202 for record in records: 203 if isinstance(record, _DroppedEmptyRecord): 204 dropped_placeholders.append(record) 205 continue 206 first_record = record 207 break 208 if isinstance(first_record, _NoRecord): 209 continue 210 # Chain the peeked record back instead of restarting the extractor: the rest of the 211 # records stay lazy and the response is never read twice. 212 yield from chain(dropped_placeholders, [first_record], records) 213 return 214 215 def _records_of( 216 self, extractor_index: int, response: requests.Response 217 ) -> Iterable[Mapping[str, Any]]: 218 records = self.extractors[extractor_index].extract_records(response) 219 if not self.skip_empty_records: 220 return records 221 return self._without_empty_records(records, extractor_index) 222 223 def _without_empty_records( 224 self, records: Iterable[Mapping[str, Any]], extractor_index: int 225 ) -> Iterator[Mapping[str, Any]]: 226 dropped = 0 227 for record in records: 228 if not record: 229 dropped += 1 230 if self.count_dropped_empty_records: 231 yield _DROPPED_EMPTY_RECORD # type: ignore[misc] # only ever counted, never emitted 232 continue 233 yield record 234 if dropped: 235 logger.warning( 236 "CombinedExtractor dropped %s empty record(s) yielded by sub-extractor %s (%s) " 237 "because `skip_empty_records` is enabled. An API that returns empty records " 238 "alongside real ones usually reports the reason in an error field of the " 239 "response body.", 240 dropped, 241 extractor_index, 242 type(self.extractors[extractor_index]).__name__, 243 )
Combines the output of several record extractors into a single stream of records.
A single DpathExtractor can only describe one path into the response. Several connectors need
more than that and reach for a custom components.py today, which keeps them out of the
Connector Builder. This component covers the two shapes those connectors implement:
union(default) — yield every record of every sub-extractor, in sub-extractor order. Use it when one response carries records under several paths, for example a GraphQL document that returnsissues.nodesandpullRequests.nodesside by side.first_match— yield the records of the first sub-extractor that produces at least one record, and skip the remaining sub-extractors. If no sub-extractor produces a record, nothing is yielded. Use it when an API answers with one of several alternative shapes. The winning sub-extractor is only peeked at, never restarted: the peeked record is chained back in front of the remaining ones, so large responses are still streamed lazily and no sub-extractor is ever materialized into a list.
Examples of instantiating this component:
extractor:
type: CombinedExtractor
mode: union
extractors:
- type: DpathExtractor
field_path: ["data", "repository", "issues", "nodes"]
- type: DpathExtractor
field_path: ["data", "repository", "pullRequests", "nodes"]
extractor:
type: CombinedExtractor
mode: first_match
extractors:
- type: DpathExtractor
field_path: ["data", "boards", "*", "items_page", "items", "*"]
- type: DpathExtractor
field_path: ["data", "next_items_page", "items"]
Streaming decoders are rejected
Every sub-extractor is handed the same requests.Response, so the response body must be
readable more than once. That holds for the decoders that buffer the whole body, which is the
common case: JsonDecoder and any CompositeRawDecoder built with stream_response=False
read response.content, XmlDecoder reads response.text and ZipfileDecoder reads
response.content — requests caches all of those, so every sub-extractor sees the full body.
It does NOT hold for streaming decoders, i.e. a CompositeRawDecoder with
stream_response=True, which is what CsvDecoder, JsonlDecoder, JsonItemsDecoder and
GzipDecoder resolve to outside the Connector Builder, nor for IterableDecoder. Those read
response.raw and close it afterwards, so the first sub-extractor drains the stream and the
later ones read a dead body. That loss is SILENT rather than loud: requests puts a
urllib3.HTTPResponse in response.raw, and reading a closed urllib3.HTTPResponse returns
an empty body instead of raising, so union emits only the first sub-extractor's records and
first_match emits nothing at all when the first path misses.
Because of that, ModelToComponentFactory.create_combined_extractor refuses to build this
component over a streaming decoder and raises a configuration error, naming the decoder in the
internal message. The Connector Builder forces those same decoders to stream_response=False,
so the rejection is applied there too rather than letting a manifest test-read correctly and
lose records once published.
Empty records
skip_empty_records drops falsy records — None, {}, [], "" — before any of the modes
sees them. It is off by default, so a sub-extractor's output is passed through verbatim.
Turn it on when an API can answer with nulls in the middle of a record list. A GraphQL API
that returns a partial response puts null in the data array for the fields it failed to
resolve and describes the failure in a sibling errors field; without this flag those nulls
reach the record stream and fail schema validation downstream.
Under first_match the flag also changes which sub-extractor wins: a sub-extractor whose
records are all empty no longer counts as a match, so the next sub-extractor is tried. That is
the behaviour of source-monday's MondayIncrementalItemsExtractor, where a page of nulls
must fall through to the pagination path rather than lock in the primary path.
The dropped records still count toward the size of the page, see "Record-counting paginators" below.
Cost
Each sub-extractor decodes the response independently — CompositeRawDecoder re-parses the
cached body on every decode() call — so a response is parsed once per sub-extractor.
Combining three extractors over a large response costs roughly three times the decode time of
a single one.
Record-counting paginators
The count OffsetIncrement and PageIncrement obtain is the combined count, not the API's
page size:
unionreturns the SUM over all sub-extractors. AnOffsetIncrementadvances the offset by that sum, so the next page starts past the records that were never read — two sub-extractors returning two records each withpage_size: 2request offsets 0, 4, 8 instead of 0, 2, 4 and silently drop two thirds of the records.ModelToComponentFactory.create_offset_incrementtherefore rejects aunionCombinedExtractor(nested ones included) with a configuration error. APageIncrementonly compares the count againstpage_sizeto decide whether to stop, so it loses nothing, but it issues one extra request whenever the summed count of the last page happens to equalpage_size.first_matchreturns the count of the winning sub-extractor, which does not inflate the count, so it works with either paginator as long as the path the count comes from is the one the API paginates over.
Empty records dropped by skip_empty_records are part of that count. The API returned them as
slots of the page, so a full page holding a null must still read as full: counting only the
surviving records would make both paginators stop at the first page with a null, and would
make an OffsetIncrement re-request records it already read. The factory sets
count_dropped_empty_records on the extractor of a RecordSelector, which makes it yield a
placeholder for every empty record it drops; the RecordSelector counts the placeholders and
discards them. The winner under first_match is still chosen by the surviving records alone,
so the placeholders never change which records are emitted.
Attributes:
- extractors (List[RecordExtractor]): The sub-extractors to combine. At least one is required.
- mode (CombineMode): How the sub-extractor outputs are combined. Defaults to
union. - skip_empty_records (bool): Whether falsy records are dropped before they are combined.
Defaults to
False. - count_dropped_empty_records (bool): Set by the factory on the extractor of a
RecordSelector, which discards the placeholders: yield a placeholder in place of every dropped empty record so it counts toward the page size. Defaults toFalse.
185 def extract_records(self, response: requests.Response) -> Iterable[Mapping[str, Any]]: 186 if self.mode == CombineMode.first_match: 187 yield from self._extract_first_match(response) 188 else: 189 yield from self._extract_union(response)
Selects records from the response
Parameters
- response: The response to extract the records from
Returns
List of Records extracted from the response
28@dataclass 29class CursorPaginationStrategy(PaginationStrategy): 30 """ 31 Pagination strategy that evaluates an interpolated string to define the next page token. 32 33 Attributes: 34 page_size (Optional[Union[str, int]]): the number of records to request 35 cursor_value (Union[InterpolatedString, str]): template string evaluating to the cursor value 36 config (Config): connection config 37 stop_condition (Optional[InterpolatedBoolean]): template string evaluating when to stop paginating 38 decoder (Decoder): decoder to decode the response 39 """ 40 41 cursor_value: Union[InterpolatedString, str] 42 config: Config 43 parameters: InitVar[Mapping[str, Any]] 44 page_size: Optional[Union[str, int]] = None 45 stop_condition: Optional[Union[InterpolatedBoolean, str]] = None 46 decoder: Decoder = field( 47 default_factory=lambda: PaginationDecoderDecorator(decoder=JsonDecoder(parameters={})) 48 ) 49 50 def __post_init__(self, parameters: Mapping[str, Any]) -> None: 51 if isinstance(self.cursor_value, str): 52 self._cursor_value = InterpolatedString.create(self.cursor_value, parameters=parameters) 53 else: 54 self._cursor_value = self.cursor_value 55 if isinstance(self.stop_condition, str): 56 self._stop_condition: Optional[InterpolatedBoolean] = InterpolatedBoolean( 57 condition=self.stop_condition, parameters=parameters 58 ) 59 else: 60 self._stop_condition = self.stop_condition 61 62 if isinstance(self.page_size, int) or (self.page_size is None): 63 self._page_size = self.page_size 64 else: 65 page_size = InterpolatedString(self.page_size, parameters=parameters).eval(self.config) 66 if not isinstance(page_size, int): 67 raise Exception(f"{page_size} is of type {type(page_size)}. Expected {int}") 68 self._page_size = page_size 69 70 if self._page_size is None: 71 self._reject_unbound_page_size_variable() 72 73 def _reject_unbound_page_size_variable(self) -> None: 74 """ 75 Fail construction when an expression reads `page_size` while the strategy declares none. 76 77 The variable holds the page size that was actually requested, so it is the one safe thing to compare 78 `last_page_size` against while a `page_size_reduction` is shrinking the page. Without a declared 79 `page_size` there is nothing to bind it to, and an unbound comparison does not raise: Jinja fails, the 80 interpolation falls back to the raw template string, and a non-empty string is truthy. A 81 `stop_condition` written that way stops after the first page and drops the rest of the partition 82 without failing, which is why this is a construction error rather than a warning. 83 """ 84 for field_name, template in ( 85 ("stop_condition", self.stop_condition), 86 ("cursor_value", self.cursor_value), 87 ): 88 expression = template.string if isinstance(template, InterpolatedString) else template 89 if isinstance(template, InterpolatedBoolean): 90 expression = template.condition 91 if not isinstance(expression, str) or not references_requested_page_size(expression): 92 continue 93 raise ValueError( 94 f"The `{field_name}` {expression!r} reads the `{REQUESTED_PAGE_SIZE_VARIABLE}` interpolation " 95 f"variable, but the CursorPagination strategy it belongs to declares no `page_size`, so there " 96 f"is nothing to bind it to. The comparison would not fail either - it renders as the template " 97 f"string itself, which is truthy - so the pagination would end after the first page and the " 98 f"rest of the partition would be dropped silently. Declare `page_size` on the pagination " 99 f"strategy, or compare against a value the manifest defines, such as " 100 f"`config['page_size']`." 101 ) 102 103 @property 104 def initial_token(self) -> Optional[Any]: 105 """ 106 CursorPaginationStrategy does not have an initial value because the next cursor is typically included 107 in the response of the first request. For Resumable Full Refresh streams that checkpoint the page 108 cursor, the next cursor should be read from the state or stream slice object. 109 """ 110 return None 111 112 def next_page_token( 113 self, 114 response: requests.Response, 115 last_page_size: int, 116 last_record: Optional[Record], 117 last_page_token_value: Optional[Any] = None, 118 page_size_override: Optional[int] = None, 119 stream_slice: Optional[StreamSlice] = None, 120 ) -> Optional[Any]: 121 # The next page is a cursor read from the response, so `page_size_override` does not change how the token 122 # is computed. It is still exposed to the interpolation context as `page_size` because a `stop_condition` 123 # comparing `last_page_size` to a hardcoded page size would read a full reduced page as a short page and 124 # end the pagination early, silently dropping the rest of the partition. Writing the condition as 125 # `{{ last_page_size < page_size }}` keeps it correct while a reduction is in effect. 126 requested_page_size = ( 127 page_size_override if page_size_override is not None else self._page_size 128 ) 129 decoded_response = next(self.decoder.decode(response)) 130 # The default way that link is presented in requests.Response is a string of various links (last, next, etc). This 131 # is not indexable or useful for parsing the cursor, so we replace it with the link dictionary from response.links 132 headers: Dict[str, Any] = dict(response.headers) 133 headers["link"] = response.links 134 if self._stop_condition: 135 should_stop = self._stop_condition.eval( 136 self.config, 137 response=decoded_response, 138 headers=headers, 139 last_record=last_record, 140 last_page_size=last_page_size, 141 page_size=requested_page_size, 142 stream_slice=stream_slice, 143 ) 144 if should_stop: 145 return None 146 token = self._cursor_value.eval( 147 config=self.config, 148 response=decoded_response, 149 headers=headers, 150 last_record=last_record, 151 last_page_size=last_page_size, 152 page_size=requested_page_size, 153 stream_slice=stream_slice, 154 ) 155 return token if token else None 156 157 def get_page_size(self) -> Optional[int]: 158 return self._page_size
Pagination strategy that evaluates an interpolated string to define the next page token.
Attributes:
- page_size (Optional[Union[str, int]]): the number of records to request
- cursor_value (Union[InterpolatedString, str]): template string evaluating to the cursor value
- config (Config): connection config
- stop_condition (Optional[InterpolatedBoolean]): template string evaluating when to stop paginating
- decoder (Decoder): decoder to decode the response
103 @property 104 def initial_token(self) -> Optional[Any]: 105 """ 106 CursorPaginationStrategy does not have an initial value because the next cursor is typically included 107 in the response of the first request. For Resumable Full Refresh streams that checkpoint the page 108 cursor, the next cursor should be read from the state or stream slice object. 109 """ 110 return None
CursorPaginationStrategy does not have an initial value because the next cursor is typically included in the response of the first request. For Resumable Full Refresh streams that checkpoint the page cursor, the next cursor should be read from the state or stream slice object.
112 def next_page_token( 113 self, 114 response: requests.Response, 115 last_page_size: int, 116 last_record: Optional[Record], 117 last_page_token_value: Optional[Any] = None, 118 page_size_override: Optional[int] = None, 119 stream_slice: Optional[StreamSlice] = None, 120 ) -> Optional[Any]: 121 # The next page is a cursor read from the response, so `page_size_override` does not change how the token 122 # is computed. It is still exposed to the interpolation context as `page_size` because a `stop_condition` 123 # comparing `last_page_size` to a hardcoded page size would read a full reduced page as a short page and 124 # end the pagination early, silently dropping the rest of the partition. Writing the condition as 125 # `{{ last_page_size < page_size }}` keeps it correct while a reduction is in effect. 126 requested_page_size = ( 127 page_size_override if page_size_override is not None else self._page_size 128 ) 129 decoded_response = next(self.decoder.decode(response)) 130 # The default way that link is presented in requests.Response is a string of various links (last, next, etc). This 131 # is not indexable or useful for parsing the cursor, so we replace it with the link dictionary from response.links 132 headers: Dict[str, Any] = dict(response.headers) 133 headers["link"] = response.links 134 if self._stop_condition: 135 should_stop = self._stop_condition.eval( 136 self.config, 137 response=decoded_response, 138 headers=headers, 139 last_record=last_record, 140 last_page_size=last_page_size, 141 page_size=requested_page_size, 142 stream_slice=stream_slice, 143 ) 144 if should_stop: 145 return None 146 token = self._cursor_value.eval( 147 config=self.config, 148 response=decoded_response, 149 headers=headers, 150 last_record=last_record, 151 last_page_size=last_page_size, 152 page_size=requested_page_size, 153 stream_slice=stream_slice, 154 ) 155 return token if token else None
Parameters
- response: response to process
- last_page_size: the number of records read from the response
- last_record: the last record extracted from the response
- last_page_token_value: The current value of the page token made on the last request
- page_size_override: the page size that was actually requested, when it differs from the configured
one because of a
REDUCE_PAGE_SIZEresponse action. Strategies that use their configured page size as a stop condition must honor this value, else they end the pagination early and skip records. - stream_slice: the slice the page was read for. Only passed to strategies that declare it, so existing strategies defined outside of the CDK keep working unchanged.
Returns
next page token. Returns None if there are no more pages to fetch
28@dataclass 29class DatetimeBasedCursor(DeclarativeCursor): 30 """ 31 Slices the stream over a datetime range and create a state with format {<cursor_field>: <datetime> } 32 33 Given a start time, end time, a step function, and an optional lookback window, 34 the stream slicer will partition the date range from start time - lookback window to end time. 35 36 The step function is defined as a string of the form ISO8601 duration 37 38 The timestamp format accepts the same format codes as datetime.strfptime, which are 39 all the format codes required by the 1989 C standard. 40 Full list of accepted format codes: https://man7.org/linux/man-pages/man3/strftime.3.html 41 42 Attributes: 43 start_datetime (Union[MinMaxDatetime, str]): the datetime that determines the earliest record that should be synced 44 end_datetime (Optional[Union[MinMaxDatetime, str]]): the datetime that determines the last record that should be synced 45 cursor_field (Union[InterpolatedString, str]): record's cursor field 46 datetime_format (str): format of the datetime 47 step (Optional[str]): size of the timewindow (ISO8601 duration) 48 cursor_granularity (Optional[str]): smallest increment the datetime_format has (ISO 8601 duration) that will be used to ensure that the start of a slice does not overlap with the end of the previous one 49 config (Config): connection config 50 start_time_option (Optional[RequestOption]): request option for start time 51 end_time_option (Optional[RequestOption]): request option for end time 52 partition_field_start (Optional[str]): partition start time field 53 partition_field_end (Optional[str]): stream slice end time field 54 lookback_window (Optional[InterpolatedString]): how many days before start_datetime to read data for (ISO8601 duration) 55 """ 56 57 start_datetime: Union[MinMaxDatetime, str] 58 cursor_field: Union[InterpolatedString, str] 59 datetime_format: str 60 config: Config 61 parameters: InitVar[Mapping[str, Any]] 62 _highest_observed_cursor_field_value: Optional[str] = field( 63 repr=False, default=None 64 ) # tracks the latest observed datetime, which may not be safe to emit in the case of out-of-order records 65 _cursor: Optional[str] = field( 66 repr=False, default=None 67 ) # tracks the latest observed datetime that is appropriate to emit as stream state 68 end_datetime: Optional[Union[MinMaxDatetime, str]] = None 69 step: Optional[Union[InterpolatedString, str]] = None 70 cursor_granularity: Optional[str] = None 71 start_time_option: Optional[RequestOption] = None 72 end_time_option: Optional[RequestOption] = None 73 partition_field_start: Optional[str] = None 74 partition_field_end: Optional[str] = None 75 lookback_window: Optional[Union[InterpolatedString, str]] = None 76 message_repository: Optional[MessageRepository] = None 77 is_compare_strictly: Optional[bool] = False 78 cursor_datetime_formats: List[str] = field(default_factory=lambda: []) 79 80 def __post_init__(self, parameters: Mapping[str, Any]) -> None: 81 if (self.step and not self.cursor_granularity) or ( 82 not self.step and self.cursor_granularity 83 ): 84 raise ValueError( 85 f"If step is defined, cursor_granularity should be as well and vice-versa. " 86 f"Right now, step is `{self.step}` and cursor_granularity is `{self.cursor_granularity}`" 87 ) 88 self._start_datetime = MinMaxDatetime.create(self.start_datetime, parameters) 89 self._end_datetime = ( 90 None if not self.end_datetime else MinMaxDatetime.create(self.end_datetime, parameters) 91 ) 92 93 self._timezone = datetime.timezone.utc 94 self._interpolation = JinjaInterpolation() 95 96 self._step = ( 97 self._parse_timedelta( 98 InterpolatedString.create(self.step, parameters=parameters).eval(self.config) 99 ) 100 if self.step 101 else datetime.timedelta.max 102 ) 103 self._cursor_granularity = self._parse_timedelta(self.cursor_granularity) 104 self.cursor_field = InterpolatedString.create(self.cursor_field, parameters=parameters) 105 self._lookback_window = ( 106 InterpolatedString.create(self.lookback_window, parameters=parameters) 107 if self.lookback_window 108 else None 109 ) 110 self._partition_field_start = InterpolatedString.create( 111 self.partition_field_start or "start_time", parameters=parameters 112 ) 113 self._partition_field_end = InterpolatedString.create( 114 self.partition_field_end or "end_time", parameters=parameters 115 ) 116 self._parser = DatetimeParser() 117 118 # If datetime format is not specified then start/end datetime should inherit it from the stream slicer 119 if not self._start_datetime.datetime_format: 120 self._start_datetime.datetime_format = self.datetime_format 121 if self._end_datetime and not self._end_datetime.datetime_format: 122 self._end_datetime.datetime_format = self.datetime_format 123 124 if not self.cursor_datetime_formats: 125 self.cursor_datetime_formats = [self.datetime_format] 126 127 _validate_component_request_option_paths( 128 self.config, self.start_time_option, self.end_time_option 129 ) 130 131 def get_stream_state(self) -> StreamState: 132 return {self.cursor_field.eval(self.config): self._cursor} if self._cursor else {} # type: ignore # cursor_field is converted to an InterpolatedString in __post_init__ 133 134 def set_initial_state(self, stream_state: StreamState) -> None: 135 """ 136 Cursors are not initialized with their state. As state is needed in order to function properly, this method should be called 137 before calling anything else 138 139 :param stream_state: The state of the stream as returned by get_stream_state 140 """ 141 self._cursor = ( 142 stream_state.get(self.cursor_field.eval(self.config)) if stream_state else None # type: ignore [union-attr] 143 ) 144 145 def observe(self, stream_slice: StreamSlice, record: Record) -> None: 146 """ 147 Register a record with the cursor; the cursor instance can then use it to manage the state of the in-progress stream read. 148 149 :param stream_slice: The current slice, which may or may not contain the most recently observed record 150 :param record: the most recently-read record, which the cursor can use to update the stream state. Outwardly-visible changes to the 151 stream state may need to be deferred depending on whether the source reliably orders records by the cursor field. 152 """ 153 record_cursor_value = record.get(self.cursor_field.eval(self.config)) # type: ignore # cursor_field is converted to an InterpolatedString in __post_init__ 154 # if the current record has no cursor value, we cannot meaningfully update the state based on it, so there is nothing more to do 155 if not record_cursor_value: 156 return 157 158 start_field = self._partition_field_start.eval(self.config) 159 end_field = self._partition_field_end.eval(self.config) 160 is_highest_observed_cursor_value = ( 161 not self._highest_observed_cursor_field_value 162 or self.parse_date(record_cursor_value) 163 > self.parse_date(self._highest_observed_cursor_field_value) 164 ) 165 if ( 166 self._is_within_daterange_boundaries( 167 record, 168 stream_slice.get(start_field), # type: ignore [arg-type] 169 stream_slice.get(end_field), # type: ignore [arg-type] 170 ) 171 and is_highest_observed_cursor_value 172 ): 173 self._highest_observed_cursor_field_value = record_cursor_value 174 175 def close_slice(self, stream_slice: StreamSlice, *args: Any) -> None: 176 if stream_slice.partition: 177 raise ValueError( 178 f"Stream slice {stream_slice} should not have a partition. Got {stream_slice.partition}." 179 ) 180 cursor_value_str_by_cursor_value_datetime = dict( 181 map( 182 # we need to ensure the cursor value is preserved as is in the state else the CATs might complain of something like 183 # 2023-01-04T17:30:19.000Z' <= '2023-01-04T17:30:19.000000Z' 184 lambda datetime_str: (self.parse_date(datetime_str), datetime_str), # type: ignore # because of the filter on the next line, this will only be called with a str 185 filter( 186 lambda item: item, [self._cursor, self._highest_observed_cursor_field_value] 187 ), 188 ) 189 ) 190 self._cursor = ( 191 cursor_value_str_by_cursor_value_datetime[ 192 max(cursor_value_str_by_cursor_value_datetime.keys()) 193 ] 194 if cursor_value_str_by_cursor_value_datetime 195 else None 196 ) 197 198 def stream_slices(self) -> Iterable[StreamSlice]: 199 """ 200 Partition the daterange into slices of size = step. 201 202 The start of the window is the minimum datetime between start_datetime - lookback_window and the stream_state's datetime 203 The end of the window is the minimum datetime between the start of the window and end_datetime. 204 205 :return: 206 """ 207 end_datetime = self.select_best_end_datetime() 208 start_datetime = self._calculate_earliest_possible_value(self.select_best_end_datetime()) 209 return self._partition_daterange(start_datetime, end_datetime, self._step) 210 211 def select_state(self, stream_slice: Optional[StreamSlice] = None) -> Optional[StreamState]: 212 # Datetime based cursors operate over slices made up of datetime ranges. Stream state is based on the progress 213 # through each slice and does not belong to a specific slice. We just return stream state as it is. 214 return self.get_stream_state() 215 216 def _calculate_earliest_possible_value( 217 self, end_datetime: datetime.datetime 218 ) -> datetime.datetime: 219 lookback_delta = self._parse_timedelta( 220 self._lookback_window.eval(self.config) if self._lookback_window else "P0D" 221 ) 222 earliest_possible_start_datetime = min( 223 self._start_datetime.get_datetime(self.config), end_datetime 224 ) 225 try: 226 cursor_datetime = ( 227 self._calculate_cursor_datetime_from_state(self.get_stream_state()) - lookback_delta 228 ) 229 except OverflowError: 230 # cursor_datetime defers to the minimum date if it does not exist in the state. Trying to subtract 231 # a timedelta from the minimum datetime results in an OverflowError 232 cursor_datetime = self._calculate_cursor_datetime_from_state(self.get_stream_state()) 233 return max(earliest_possible_start_datetime, cursor_datetime) 234 235 def select_best_end_datetime(self) -> datetime.datetime: 236 """ 237 Returns the optimal end datetime. 238 This method compares the current datetime with a pre-configured end datetime 239 and returns the earlier of the two. If no pre-configured end datetime is set, 240 the current datetime is returned. 241 242 :return datetime.datetime: The best end datetime, which is either the current datetime or the pre-configured end datetime, whichever is earlier. 243 """ 244 now = datetime.datetime.now(tz=self._timezone) 245 if not self._end_datetime: 246 return now 247 return min(self._end_datetime.get_datetime(self.config), now) 248 249 def _calculate_cursor_datetime_from_state( 250 self, stream_state: Mapping[str, Any] 251 ) -> datetime.datetime: 252 if self.cursor_field.eval(self.config, stream_state=stream_state) in stream_state: # type: ignore # cursor_field is converted to an InterpolatedString in __post_init__ 253 return self.parse_date(stream_state[self.cursor_field.eval(self.config)]) # type: ignore # cursor_field is converted to an InterpolatedString in __post_init__ 254 return datetime.datetime.min.replace(tzinfo=datetime.timezone.utc) 255 256 def _format_datetime(self, dt: datetime.datetime) -> str: 257 return self._parser.format(dt, self.datetime_format) 258 259 def _partition_daterange( 260 self, 261 start: datetime.datetime, 262 end: datetime.datetime, 263 step: Union[datetime.timedelta, Duration], 264 ) -> List[StreamSlice]: 265 start_field = self._partition_field_start.eval(self.config) 266 end_field = self._partition_field_end.eval(self.config) 267 dates = [] 268 269 while self._is_within_date_range(start, end): 270 next_start = self._evaluate_next_start_date_safely(start, step) 271 end_date = self._get_date(next_start - self._cursor_granularity, end, min) 272 dates.append( 273 StreamSlice( 274 partition={}, 275 cursor_slice={ 276 start_field: self._format_datetime(start), 277 end_field: self._format_datetime(end_date), 278 }, 279 ) 280 ) 281 start = next_start 282 return dates 283 284 def _is_within_date_range(self, start: datetime.datetime, end: datetime.datetime) -> bool: 285 if self.is_compare_strictly: 286 return start < end 287 return start <= end 288 289 def _evaluate_next_start_date_safely( 290 self, start: datetime.datetime, step: datetime.timedelta 291 ) -> datetime.datetime: 292 """ 293 Given that we set the default step at datetime.timedelta.max, we will generate an OverflowError when evaluating the next start_date 294 This method assumes that users would never enter a step that would generate an overflow. Given that would be the case, the code 295 would have broken anyway. 296 """ 297 try: 298 return start + step 299 except OverflowError: 300 return datetime.datetime.max.replace(tzinfo=datetime.timezone.utc) 301 302 def _get_date( 303 self, 304 cursor_value: datetime.datetime, 305 default_date: datetime.datetime, 306 comparator: Callable[[datetime.datetime, datetime.datetime], datetime.datetime], 307 ) -> datetime.datetime: 308 cursor_date = cursor_value or default_date 309 return comparator(cursor_date, default_date) 310 311 def parse_date(self, date: str) -> datetime.datetime: 312 for datetime_format in self.cursor_datetime_formats + [self.datetime_format]: 313 try: 314 return self._parser.parse(date, datetime_format) 315 except ValueError: 316 pass 317 raise ValueError(f"No format in {self.cursor_datetime_formats} matching {date}") 318 319 @classmethod 320 def _parse_timedelta(cls, time_str: Optional[str]) -> Union[datetime.timedelta, Duration]: 321 """ 322 :return Parses an ISO 8601 durations into datetime.timedelta or Duration objects. 323 """ 324 if not time_str: 325 return datetime.timedelta(0) 326 return parse_duration(time_str) 327 328 def get_request_params( 329 self, 330 *, 331 stream_state: Optional[StreamState] = None, 332 stream_slice: Optional[StreamSlice] = None, 333 next_page_token: Optional[Mapping[str, Any]] = None, 334 ) -> Mapping[str, Any]: 335 return self._get_request_options(RequestOptionType.request_parameter, stream_slice) 336 337 def get_request_headers( 338 self, 339 *, 340 stream_state: Optional[StreamState] = None, 341 stream_slice: Optional[StreamSlice] = None, 342 next_page_token: Optional[Mapping[str, Any]] = None, 343 ) -> Mapping[str, Any]: 344 return self._get_request_options(RequestOptionType.header, stream_slice) 345 346 def get_request_body_data( 347 self, 348 *, 349 stream_state: Optional[StreamState] = None, 350 stream_slice: Optional[StreamSlice] = None, 351 next_page_token: Optional[Mapping[str, Any]] = None, 352 ) -> Mapping[str, Any]: 353 return self._get_request_options(RequestOptionType.body_data, stream_slice) 354 355 def get_request_body_json( 356 self, 357 *, 358 stream_state: Optional[StreamState] = None, 359 stream_slice: Optional[StreamSlice] = None, 360 next_page_token: Optional[Mapping[str, Any]] = None, 361 ) -> Mapping[str, Any]: 362 return self._get_request_options(RequestOptionType.body_json, stream_slice) 363 364 def request_kwargs(self) -> Mapping[str, Any]: 365 # Never update kwargs 366 return {} 367 368 def _get_request_options( 369 self, option_type: RequestOptionType, stream_slice: Optional[StreamSlice] 370 ) -> Mapping[str, Any]: 371 options: MutableMapping[str, Any] = {} 372 if not stream_slice: 373 return options 374 375 if self.start_time_option and self.start_time_option.inject_into == option_type: 376 start_time_value = stream_slice.get(self._partition_field_start.eval(self.config)) 377 self.start_time_option.inject_into_request(options, start_time_value, self.config) 378 379 if self.end_time_option and self.end_time_option.inject_into == option_type: 380 end_time_value = stream_slice.get(self._partition_field_end.eval(self.config)) 381 self.end_time_option.inject_into_request(options, end_time_value, self.config) 382 383 return options 384 385 def should_be_synced(self, record: Record) -> bool: 386 cursor_field = self.cursor_field.eval(self.config) # type: ignore # cursor_field is converted to an InterpolatedString in __post_init__ 387 record_cursor_value = record.get(cursor_field) 388 if not record_cursor_value: 389 self._send_log( 390 Level.WARN, 391 f"Could not find cursor field `{cursor_field}` in record. The incremental sync will assume it needs to be synced", 392 ) 393 return True 394 latest_possible_cursor_value = self.select_best_end_datetime() 395 earliest_possible_cursor_value = self._calculate_earliest_possible_value( 396 latest_possible_cursor_value 397 ) 398 return self._is_within_daterange_boundaries( 399 record, earliest_possible_cursor_value, latest_possible_cursor_value 400 ) 401 402 def _is_within_daterange_boundaries( 403 self, 404 record: Record, 405 start_datetime_boundary: Union[datetime.datetime, str], 406 end_datetime_boundary: Union[datetime.datetime, str], 407 ) -> bool: 408 cursor_field = self.cursor_field.eval(self.config) # type: ignore # cursor_field is converted to an InterpolatedString in __post_init__ 409 record_cursor_value = record.get(cursor_field) 410 if not record_cursor_value: 411 self._send_log( 412 Level.WARN, 413 f"Could not find cursor field `{cursor_field}` in record. The record will not be considered when emitting sync state", 414 ) 415 return False 416 if isinstance(start_datetime_boundary, str): 417 start_datetime_boundary = self.parse_date(start_datetime_boundary) 418 if isinstance(end_datetime_boundary, str): 419 end_datetime_boundary = self.parse_date(end_datetime_boundary) 420 return ( 421 start_datetime_boundary <= self.parse_date(record_cursor_value) <= end_datetime_boundary 422 ) 423 424 def _send_log(self, level: Level, message: str) -> None: 425 if self.message_repository: 426 self.message_repository.emit_message( 427 AirbyteMessage( 428 type=Type.LOG, 429 log=AirbyteLogMessage(level=level, message=message), 430 ) 431 ) 432 433 def set_runtime_lookback_window(self, lookback_window_in_seconds: int) -> None: 434 """ 435 Updates the lookback window based on a given number of seconds if the new duration 436 is greater than the currently configured lookback window. 437 438 :param lookback_window_in_seconds: The lookback duration in seconds to potentially update to. 439 """ 440 runtime_lookback_window = duration_isoformat(timedelta(seconds=lookback_window_in_seconds)) 441 config_lookback = parse_duration( 442 self._lookback_window.eval(self.config) if self._lookback_window else "P0D" 443 ) 444 445 # Check if the new runtime lookback window is greater than the current config lookback 446 if parse_duration(runtime_lookback_window) > config_lookback: 447 self._lookback_window = InterpolatedString.create( 448 runtime_lookback_window, parameters={} 449 )
Slices the stream over a datetime range and create a state with format {
Given a start time, end time, a step function, and an optional lookback window, the stream slicer will partition the date range from start time - lookback window to end time.
The step function is defined as a string of the form ISO8601 duration
The timestamp format accepts the same format codes as datetime.strfptime, which are all the format codes required by the 1989 C standard. Full list of accepted format codes: https://man7.org/linux/man-pages/man3/strftime.3.html
Attributes:
- start_datetime (Union[MinMaxDatetime, str]): the datetime that determines the earliest record that should be synced
- end_datetime (Optional[Union[MinMaxDatetime, str]]): the datetime that determines the last record that should be synced
- cursor_field (Union[InterpolatedString, str]): record's cursor field
- datetime_format (str): format of the datetime
- step (Optional[str]): size of the timewindow (ISO8601 duration)
- cursor_granularity (Optional[str]): smallest increment the datetime_format has (ISO 8601 duration) that will be used to ensure that the start of a slice does not overlap with the end of the previous one
- config (Config): connection config
- start_time_option (Optional[RequestOption]): request option for start time
- end_time_option (Optional[RequestOption]): request option for end time
- partition_field_start (Optional[str]): partition start time field
- partition_field_end (Optional[str]): stream slice end time field
- lookback_window (Optional[InterpolatedString]): how many days before start_datetime to read data for (ISO8601 duration)
131 def get_stream_state(self) -> StreamState: 132 return {self.cursor_field.eval(self.config): self._cursor} if self._cursor else {} # type: ignore # cursor_field is converted to an InterpolatedString in __post_init__
Returns the current stream state. We would like to restrict it's usage since it does expose internal of state. As of 2023-06-14, it is used for two things:
- Interpolation of the requests
- Transformation of records
- Saving the state
For the first case, we are probably stuck with exposing the stream state. For the second, we can probably expose a method that allows for emitting the state to the platform.
134 def set_initial_state(self, stream_state: StreamState) -> None: 135 """ 136 Cursors are not initialized with their state. As state is needed in order to function properly, this method should be called 137 before calling anything else 138 139 :param stream_state: The state of the stream as returned by get_stream_state 140 """ 141 self._cursor = ( 142 stream_state.get(self.cursor_field.eval(self.config)) if stream_state else None # type: ignore [union-attr] 143 )
Cursors are not initialized with their state. As state is needed in order to function properly, this method should be called before calling anything else
Parameters
- stream_state: The state of the stream as returned by get_stream_state
145 def observe(self, stream_slice: StreamSlice, record: Record) -> None: 146 """ 147 Register a record with the cursor; the cursor instance can then use it to manage the state of the in-progress stream read. 148 149 :param stream_slice: The current slice, which may or may not contain the most recently observed record 150 :param record: the most recently-read record, which the cursor can use to update the stream state. Outwardly-visible changes to the 151 stream state may need to be deferred depending on whether the source reliably orders records by the cursor field. 152 """ 153 record_cursor_value = record.get(self.cursor_field.eval(self.config)) # type: ignore # cursor_field is converted to an InterpolatedString in __post_init__ 154 # if the current record has no cursor value, we cannot meaningfully update the state based on it, so there is nothing more to do 155 if not record_cursor_value: 156 return 157 158 start_field = self._partition_field_start.eval(self.config) 159 end_field = self._partition_field_end.eval(self.config) 160 is_highest_observed_cursor_value = ( 161 not self._highest_observed_cursor_field_value 162 or self.parse_date(record_cursor_value) 163 > self.parse_date(self._highest_observed_cursor_field_value) 164 ) 165 if ( 166 self._is_within_daterange_boundaries( 167 record, 168 stream_slice.get(start_field), # type: ignore [arg-type] 169 stream_slice.get(end_field), # type: ignore [arg-type] 170 ) 171 and is_highest_observed_cursor_value 172 ): 173 self._highest_observed_cursor_field_value = record_cursor_value
Register a record with the cursor; the cursor instance can then use it to manage the state of the in-progress stream read.
Parameters
- stream_slice: The current slice, which may or may not contain the most recently observed record
- record: the most recently-read record, which the cursor can use to update the stream state. Outwardly-visible changes to the stream state may need to be deferred depending on whether the source reliably orders records by the cursor field.
175 def close_slice(self, stream_slice: StreamSlice, *args: Any) -> None: 176 if stream_slice.partition: 177 raise ValueError( 178 f"Stream slice {stream_slice} should not have a partition. Got {stream_slice.partition}." 179 ) 180 cursor_value_str_by_cursor_value_datetime = dict( 181 map( 182 # we need to ensure the cursor value is preserved as is in the state else the CATs might complain of something like 183 # 2023-01-04T17:30:19.000Z' <= '2023-01-04T17:30:19.000000Z' 184 lambda datetime_str: (self.parse_date(datetime_str), datetime_str), # type: ignore # because of the filter on the next line, this will only be called with a str 185 filter( 186 lambda item: item, [self._cursor, self._highest_observed_cursor_field_value] 187 ), 188 ) 189 ) 190 self._cursor = ( 191 cursor_value_str_by_cursor_value_datetime[ 192 max(cursor_value_str_by_cursor_value_datetime.keys()) 193 ] 194 if cursor_value_str_by_cursor_value_datetime 195 else None 196 )
Update state based on the stream slice. Note that stream_slice.cursor_slice and most_recent_record.associated_slice are expected
to be the same but we make it explicit here that stream_slice should be leveraged to update the state. We do not pass in the
latest record, since cursor instances should maintain the relevant internal state on their own.
Parameters
- stream_slice: slice to close
198 def stream_slices(self) -> Iterable[StreamSlice]: 199 """ 200 Partition the daterange into slices of size = step. 201 202 The start of the window is the minimum datetime between start_datetime - lookback_window and the stream_state's datetime 203 The end of the window is the minimum datetime between the start of the window and end_datetime. 204 205 :return: 206 """ 207 end_datetime = self.select_best_end_datetime() 208 start_datetime = self._calculate_earliest_possible_value(self.select_best_end_datetime()) 209 return self._partition_daterange(start_datetime, end_datetime, self._step)
Partition the daterange into slices of size = step.
The start of the window is the minimum datetime between start_datetime - lookback_window and the stream_state's datetime The end of the window is the minimum datetime between the start of the window and end_datetime.
Returns
211 def select_state(self, stream_slice: Optional[StreamSlice] = None) -> Optional[StreamState]: 212 # Datetime based cursors operate over slices made up of datetime ranges. Stream state is based on the progress 213 # through each slice and does not belong to a specific slice. We just return stream state as it is. 214 return self.get_stream_state()
Get the state value of a specific stream_slice. For incremental or resumable full refresh cursors which only manage state in a single dimension this is the entire state object. For per-partition cursors used by substreams, this returns the state of a specific parent delineated by the incoming slice's partition object.
235 def select_best_end_datetime(self) -> datetime.datetime: 236 """ 237 Returns the optimal end datetime. 238 This method compares the current datetime with a pre-configured end datetime 239 and returns the earlier of the two. If no pre-configured end datetime is set, 240 the current datetime is returned. 241 242 :return datetime.datetime: The best end datetime, which is either the current datetime or the pre-configured end datetime, whichever is earlier. 243 """ 244 now = datetime.datetime.now(tz=self._timezone) 245 if not self._end_datetime: 246 return now 247 return min(self._end_datetime.get_datetime(self.config), now)
Returns the optimal end datetime. This method compares the current datetime with a pre-configured end datetime and returns the earlier of the two. If no pre-configured end datetime is set, the current datetime is returned.
Returns
The best end datetime, which is either the current datetime or the pre-configured end datetime, whichever is earlier.
311 def parse_date(self, date: str) -> datetime.datetime: 312 for datetime_format in self.cursor_datetime_formats + [self.datetime_format]: 313 try: 314 return self._parser.parse(date, datetime_format) 315 except ValueError: 316 pass 317 raise ValueError(f"No format in {self.cursor_datetime_formats} matching {date}")
328 def get_request_params( 329 self, 330 *, 331 stream_state: Optional[StreamState] = None, 332 stream_slice: Optional[StreamSlice] = None, 333 next_page_token: Optional[Mapping[str, Any]] = None, 334 ) -> Mapping[str, Any]: 335 return self._get_request_options(RequestOptionType.request_parameter, stream_slice)
Specifies the query parameters that should be set on an outgoing HTTP request given the inputs.
E.g: you might want to define query parameters for paging if next_page_token is not None.
337 def get_request_headers( 338 self, 339 *, 340 stream_state: Optional[StreamState] = None, 341 stream_slice: Optional[StreamSlice] = None, 342 next_page_token: Optional[Mapping[str, Any]] = None, 343 ) -> Mapping[str, Any]: 344 return self._get_request_options(RequestOptionType.header, stream_slice)
Return any non-auth headers. Authentication headers will overwrite any overlapping headers returned from this method.
346 def get_request_body_data( 347 self, 348 *, 349 stream_state: Optional[StreamState] = None, 350 stream_slice: Optional[StreamSlice] = None, 351 next_page_token: Optional[Mapping[str, Any]] = None, 352 ) -> Mapping[str, Any]: 353 return self._get_request_options(RequestOptionType.body_data, stream_slice)
Specifies how to populate the body of the request with a non-JSON payload.
If returns a ready text that it will be sent as is. If returns a dict that it will be converted to a urlencoded form. E.g. {"key1": "value1", "key2": "value2"} => "key1=value1&key2=value2"
At the same time only one of the 'request_body_data' and 'request_body_json' functions can be overridden.
355 def get_request_body_json( 356 self, 357 *, 358 stream_state: Optional[StreamState] = None, 359 stream_slice: Optional[StreamSlice] = None, 360 next_page_token: Optional[Mapping[str, Any]] = None, 361 ) -> Mapping[str, Any]: 362 return self._get_request_options(RequestOptionType.body_json, stream_slice)
Specifies how to populate the body of the request with a JSON payload.
At the same time only one of the 'request_body_data' and 'request_body_json' functions can be overridden.
385 def should_be_synced(self, record: Record) -> bool: 386 cursor_field = self.cursor_field.eval(self.config) # type: ignore # cursor_field is converted to an InterpolatedString in __post_init__ 387 record_cursor_value = record.get(cursor_field) 388 if not record_cursor_value: 389 self._send_log( 390 Level.WARN, 391 f"Could not find cursor field `{cursor_field}` in record. The incremental sync will assume it needs to be synced", 392 ) 393 return True 394 latest_possible_cursor_value = self.select_best_end_datetime() 395 earliest_possible_cursor_value = self._calculate_earliest_possible_value( 396 latest_possible_cursor_value 397 ) 398 return self._is_within_daterange_boundaries( 399 record, earliest_possible_cursor_value, latest_possible_cursor_value 400 )
Evaluating if a record should be synced allows for filtering and stop condition on pagination
433 def set_runtime_lookback_window(self, lookback_window_in_seconds: int) -> None: 434 """ 435 Updates the lookback window based on a given number of seconds if the new duration 436 is greater than the currently configured lookback window. 437 438 :param lookback_window_in_seconds: The lookback duration in seconds to potentially update to. 439 """ 440 runtime_lookback_window = duration_isoformat(timedelta(seconds=lookback_window_in_seconds)) 441 config_lookback = parse_duration( 442 self._lookback_window.eval(self.config) if self._lookback_window else "P0D" 443 ) 444 445 # Check if the new runtime lookback window is greater than the current config lookback 446 if parse_duration(runtime_lookback_window) > config_lookback: 447 self._lookback_window = InterpolatedString.create( 448 runtime_lookback_window, parameters={} 449 )
Updates the lookback window based on a given number of seconds if the new duration is greater than the currently configured lookback window.
Parameters
- lookback_window_in_seconds: The lookback duration in seconds to potentially update to.
14@dataclass 15class DeclarativeAuthenticator(AbstractHeaderAuthenticator): 16 """ 17 Interface used to associate which authenticators can be used as part of the declarative framework 18 """ 19 20 def get_request_params(self) -> Mapping[str, Any]: 21 """HTTP request parameter to add to the requests""" 22 return {} 23 24 def get_request_body_data(self) -> Union[Mapping[str, Any], str]: 25 """Form-encoded body data to set on the requests""" 26 return {} 27 28 def get_request_body_json(self) -> Mapping[str, Any]: 29 """JSON-encoded body data to set on the requests""" 30 return {}
Interface used to associate which authenticators can be used as part of the declarative framework
20 def get_request_params(self) -> Mapping[str, Any]: 21 """HTTP request parameter to add to the requests""" 22 return {}
HTTP request parameter to add to the requests
24 def get_request_body_data(self) -> Union[Mapping[str, Any], str]: 25 """Form-encoded body data to set on the requests""" 26 return {}
Form-encoded body data to set on the requests
28 def get_request_body_json(self) -> Mapping[str, Any]: 29 """JSON-encoded body data to set on the requests""" 30 return {}
JSON-encoded body data to set on the requests
Inherited Members
28@dataclass 29class DeclarativeOauth2Authenticator(AbstractOauth2Authenticator, DeclarativeAuthenticator): 30 """ 31 Generates OAuth2.0 access tokens from an OAuth2.0 refresh token and client credentials based on 32 a declarative connector configuration file. Credentials can be defined explicitly or via interpolation 33 at runtime. The generated access token is attached to each request via the Authorization header. 34 35 Attributes: 36 token_refresh_endpoint (Union[InterpolatedString, str]): The endpoint to refresh the access token 37 client_id (Union[InterpolatedString, str]): The client id 38 client_secret (Union[InterpolatedString, str]): Client secret (can be empty for APIs that support this) 39 refresh_token (Union[InterpolatedString, str]): The token used to refresh the access token 40 access_token_name (Union[InterpolatedString, str]): THe field to extract access token from in the response 41 expires_in_name (Union[InterpolatedString, str]): The field to extract expires_in from in the response 42 config (Mapping[str, Any]): The user-provided configuration as specified by the source's spec 43 scopes (Optional[List[str]]): The scopes to request 44 token_expiry_date (Optional[Union[InterpolatedString, str]]): The access token expiration date 45 token_expiry_date_format str: format of the datetime; provide it if expires_in is returned in datetime instead of seconds 46 token_expiry_is_time_of_expiration bool: set True it if expires_in is returned as time of expiration instead of the number seconds until expiration 47 refresh_request_body (Optional[Mapping[str, Any]]): The request body to send in the refresh request 48 refresh_request_headers (Optional[Mapping[str, Any]]): The request headers to send in the refresh request 49 send_refresh_request_as_query_params (bool): When True, the standard refresh args (`grant_type`, `refresh_token`, client credentials when not in an `Authorization` header, scopes, plus any `refresh_request_body` extras) are sent on the URL query string instead of in the request body, and the body is emitted empty. Use this for OAuth providers like Gong that document their refresh endpoint with refresh args on the URL query string. Defaults to False. 50 grant_type: The grant_type to request for access_token. If set to refresh_token, the refresh_token parameter has to be provided 51 message_repository (MessageRepository): the message repository used to emit logs on HTTP requests 52 refresh_token_error_status_codes (Tuple[int, ...]): Status codes to identify refresh token errors in response 53 refresh_token_error_key (str): Key to identify refresh token error in response 54 refresh_token_error_values (Tuple[str, ...]): List of values to check for exception during token refresh process 55 refresh_token_error_messages (Optional[Mapping[str, str]]): Fixed user-facing messages for refresh token errors 56 """ 57 58 config: Mapping[str, Any] 59 parameters: InitVar[Mapping[str, Any]] 60 client_id: Optional[Union[InterpolatedString, str]] = None 61 client_secret: Optional[Union[InterpolatedString, str]] = None 62 token_refresh_endpoint: Optional[Union[InterpolatedString, str]] = None 63 refresh_token: Optional[Union[InterpolatedString, str]] = None 64 scopes: Optional[List[str]] = None 65 token_expiry_date: Optional[Union[InterpolatedString, str]] = None 66 _token_expiry_date: Optional[AirbyteDateTime] = field(init=False, repr=False, default=None) 67 token_expiry_date_format: Optional[str] = None 68 token_expiry_is_time_of_expiration: bool = False 69 access_token_name: Union[InterpolatedString, str] = "access_token" 70 access_token_value: Optional[Union[InterpolatedString, str]] = None 71 client_id_name: Union[InterpolatedString, str] = "client_id" 72 client_secret_name: Union[InterpolatedString, str] = "client_secret" 73 expires_in_name: Union[InterpolatedString, str] = "expires_in" 74 refresh_token_name: Union[InterpolatedString, str] = "refresh_token" 75 refresh_request_body: Optional[Mapping[str, Any]] = None 76 refresh_request_headers: Optional[Mapping[str, Any]] = None 77 send_refresh_request_as_query_params: bool = False 78 grant_type_name: Union[InterpolatedString, str] = "grant_type" 79 grant_type: Union[InterpolatedString, str] = "refresh_token" 80 message_repository: MessageRepository = NoopMessageRepository() 81 profile_assertion: Optional[DeclarativeAuthenticator] = None 82 use_profile_assertion: Optional[Union[InterpolatedBoolean, str, bool]] = False 83 refresh_token_error_status_codes: Tuple[int, ...] = () 84 refresh_token_error_key: str = "" 85 refresh_token_error_values: Tuple[str, ...] = () 86 refresh_token_error_messages: Optional[Mapping[str, str]] = None 87 88 def __post_init__(self, parameters: Mapping[str, Any]) -> None: 89 super().__init__( 90 refresh_token_error_status_codes=self.refresh_token_error_status_codes, 91 refresh_token_error_key=self.refresh_token_error_key, 92 refresh_token_error_values=self.refresh_token_error_values, 93 refresh_token_error_messages=self.refresh_token_error_messages, 94 ) 95 if self.token_refresh_endpoint is not None: 96 self._token_refresh_endpoint: Optional[InterpolatedString] = InterpolatedString.create( 97 self.token_refresh_endpoint, parameters=parameters 98 ) 99 else: 100 self._token_refresh_endpoint = None 101 self._client_id_name = InterpolatedString.create(self.client_id_name, parameters=parameters) 102 self._client_id = ( 103 InterpolatedString.create(self.client_id, parameters=parameters) 104 if self.client_id 105 else self.client_id 106 ) 107 self._client_secret_name = InterpolatedString.create( 108 self.client_secret_name, parameters=parameters 109 ) 110 self._client_secret = ( 111 InterpolatedString.create(self.client_secret, parameters=parameters) 112 if self.client_secret 113 else self.client_secret 114 ) 115 self._refresh_token_name = InterpolatedString.create( 116 self.refresh_token_name, parameters=parameters 117 ) 118 if self.refresh_token is not None: 119 self._refresh_token: Optional[InterpolatedString] = InterpolatedString.create( 120 self.refresh_token, parameters=parameters 121 ) 122 else: 123 self._refresh_token = None 124 self.access_token_name = InterpolatedString.create( 125 self.access_token_name, parameters=parameters 126 ) 127 self.expires_in_name = InterpolatedString.create( 128 self.expires_in_name, parameters=parameters 129 ) 130 self.grant_type_name = InterpolatedString.create( 131 self.grant_type_name, parameters=parameters 132 ) 133 self.grant_type = InterpolatedString.create( 134 "urn:ietf:params:oauth:grant-type:jwt-bearer" 135 if self.use_profile_assertion 136 else self.grant_type, 137 parameters=parameters, 138 ) 139 self._refresh_request_body = InterpolatedMapping( 140 self.refresh_request_body or {}, parameters=parameters 141 ) 142 self._refresh_request_headers = InterpolatedMapping( 143 self.refresh_request_headers or {}, parameters=parameters 144 ) 145 self._send_refresh_request_as_query_params = self.send_refresh_request_as_query_params 146 try: 147 if ( 148 isinstance(self.token_expiry_date, (int, str)) 149 and str(self.token_expiry_date).isdigit() 150 ): 151 self._token_expiry_date = ab_datetime_parse(self.token_expiry_date) 152 else: 153 self._token_expiry_date = ( 154 ab_datetime_parse( 155 InterpolatedString.create( 156 self.token_expiry_date, parameters=parameters 157 ).eval(self.config) 158 ) 159 if self.token_expiry_date 160 else ab_datetime_now() - timedelta(days=1) 161 ) 162 except ValueError as e: 163 raise ValueError(f"Invalid token expiry date format: {e}") 164 self.use_profile_assertion = ( 165 InterpolatedBoolean(self.use_profile_assertion, parameters=parameters) 166 if isinstance(self.use_profile_assertion, str) 167 else self.use_profile_assertion 168 ) 169 self.assertion_name = "assertion" 170 171 if self.access_token_value is not None: 172 self._access_token_value = InterpolatedString.create( 173 self.access_token_value, parameters=parameters 174 ).eval(self.config) 175 else: 176 self._access_token_value = None 177 178 self._access_token: Optional[str] = ( 179 self._access_token_value if self.access_token_value else None 180 ) 181 182 if not self.use_profile_assertion and any( 183 client_creds is None for client_creds in [self.client_id, self.client_secret] 184 ): 185 raise ValueError( 186 "OAuthAuthenticator configuration error: Both 'client_id' and 'client_secret' are required for the " 187 "basic OAuth flow." 188 ) 189 if self.profile_assertion is None and self.use_profile_assertion: 190 raise ValueError( 191 "OAuthAuthenticator configuration error: 'profile_assertion' is required when using the profile assertion flow." 192 ) 193 if self.get_grant_type() == "refresh_token" and self._refresh_token is None: 194 raise ValueError( 195 "OAuthAuthenticator configuration error: A 'refresh_token' is required when the 'grant_type' is set to 'refresh_token'." 196 ) 197 198 def get_token_refresh_endpoint(self) -> Optional[str]: 199 if self._token_refresh_endpoint is not None: 200 refresh_token_endpoint: str = self._token_refresh_endpoint.eval(self.config) 201 if not refresh_token_endpoint: 202 raise ValueError( 203 "OAuthAuthenticator was unable to evaluate token_refresh_endpoint parameter" 204 ) 205 return refresh_token_endpoint 206 return None 207 208 def get_client_id_name(self) -> str: 209 return self._client_id_name.eval(self.config) # type: ignore # eval returns a string in this context 210 211 def get_client_id(self) -> str: 212 client_id = self._client_id.eval(self.config) if self._client_id else self._client_id 213 if not client_id: 214 raise ValueError("OAuthAuthenticator was unable to evaluate client_id parameter") 215 return client_id # type: ignore # value will be returned as a string, or an error will be raised 216 217 def get_client_secret_name(self) -> str: 218 return self._client_secret_name.eval(self.config) # type: ignore # eval returns a string in this context 219 220 def get_client_secret(self) -> str: 221 client_secret = ( 222 self._client_secret.eval(self.config) if self._client_secret else self._client_secret 223 ) 224 if not client_secret: 225 # We've seen some APIs allowing empty client_secret so we will only log here 226 logger.warning( 227 "OAuthAuthenticator was unable to evaluate client_secret parameter hence it'll be empty" 228 ) 229 return client_secret # type: ignore # value will be returned as a string, which might be empty 230 231 def get_refresh_token_name(self) -> str: 232 return self._refresh_token_name.eval(self.config) # type: ignore # eval returns a string in this context 233 234 def get_refresh_token(self) -> Optional[str]: 235 return None if self._refresh_token is None else str(self._refresh_token.eval(self.config)) 236 237 def get_scopes(self) -> List[str]: 238 return self.scopes or [] 239 240 def get_access_token_name(self) -> str: 241 return self.access_token_name.eval(self.config) # type: ignore # eval returns a string in this context 242 243 def get_expires_in_name(self) -> str: 244 return self.expires_in_name.eval(self.config) # type: ignore # eval returns a string in this context 245 246 def get_grant_type_name(self) -> str: 247 return self.grant_type_name.eval(self.config) # type: ignore # eval returns a string in this context 248 249 def get_grant_type(self) -> str: 250 return self.grant_type.eval(self.config) # type: ignore # eval returns a string in this context 251 252 def get_refresh_request_body(self) -> Mapping[str, Any]: 253 return self._refresh_request_body.eval(self.config) 254 255 def get_refresh_request_headers(self) -> Mapping[str, Any]: 256 return self._refresh_request_headers.eval(self.config) 257 258 def should_send_refresh_request_as_query_params(self) -> bool: 259 return self._send_refresh_request_as_query_params 260 261 def get_token_expiry_date(self) -> AirbyteDateTime: 262 if not self._has_access_token_been_initialized(): 263 return AirbyteDateTime.from_datetime(datetime.min) 264 return self._token_expiry_date # type: ignore # _token_expiry_date is an AirbyteDateTime. It is never None despite what mypy thinks 265 266 def _has_access_token_been_initialized(self) -> bool: 267 return self._access_token is not None 268 269 def set_token_expiry_date(self, value: AirbyteDateTime) -> None: 270 # Overrides of `refresh_access_token` written before 6.45.5 return the raw `expires_in`. 271 if not isinstance(value, datetime): 272 with suppress(ValueError, OverflowError): 273 value = self._parse_token_expiration_date(value) 274 self._token_expiry_date = value 275 276 def get_assertion_name(self) -> str: 277 return self.assertion_name 278 279 def get_assertion(self) -> str: 280 if self.profile_assertion is None: 281 raise ValueError("profile_assertion is not set") 282 return self.profile_assertion.token 283 284 def build_refresh_request_body(self) -> Mapping[str, Any]: 285 """ 286 Returns the request body to set on the refresh request 287 288 Override to define additional parameters 289 """ 290 if self.use_profile_assertion: 291 return { 292 self.get_grant_type_name(): self.get_grant_type(), 293 self.get_assertion_name(): self.get_assertion(), 294 } 295 return super().build_refresh_request_body() 296 297 @property 298 def access_token(self) -> str: 299 if self._access_token is None: 300 raise ValueError("access_token is not set") 301 return self._access_token 302 303 @access_token.setter 304 def access_token(self, value: str) -> None: 305 self._access_token = value 306 307 @property 308 def _message_repository(self) -> MessageRepository: 309 """ 310 Overriding AbstractOauth2Authenticator._message_repository to allow for HTTP request logs 311 """ 312 return self.message_repository
Generates OAuth2.0 access tokens from an OAuth2.0 refresh token and client credentials based on a declarative connector configuration file. Credentials can be defined explicitly or via interpolation at runtime. The generated access token is attached to each request via the Authorization header.
Attributes:
- token_refresh_endpoint (Union[InterpolatedString, str]): The endpoint to refresh the access token
- client_id (Union[InterpolatedString, str]): The client id
- client_secret (Union[InterpolatedString, str]): Client secret (can be empty for APIs that support this)
- refresh_token (Union[InterpolatedString, str]): The token used to refresh the access token
- access_token_name (Union[InterpolatedString, str]): THe field to extract access token from in the response
- expires_in_name (Union[InterpolatedString, str]): The field to extract expires_in from in the response
- config (Mapping[str, Any]): The user-provided configuration as specified by the source's spec
- scopes (Optional[List[str]]): The scopes to request
- token_expiry_date (Optional[Union[InterpolatedString, str]]): The access token expiration date
- token_expiry_date_format str: format of the datetime; provide it if expires_in is returned in datetime instead of seconds
- token_expiry_is_time_of_expiration bool: set True it if expires_in is returned as time of expiration instead of the number seconds until expiration
- refresh_request_body (Optional[Mapping[str, Any]]): The request body to send in the refresh request
- refresh_request_headers (Optional[Mapping[str, Any]]): The request headers to send in the refresh request
- send_refresh_request_as_query_params (bool): When True, the standard refresh args (
grant_type,refresh_token, client credentials when not in anAuthorizationheader, scopes, plus anyrefresh_request_bodyextras) are sent on the URL query string instead of in the request body, and the body is emitted empty. Use this for OAuth providers like Gong that document their refresh endpoint with refresh args on the URL query string. Defaults to False. - grant_type: The grant_type to request for access_token. If set to refresh_token, the refresh_token parameter has to be provided
- message_repository (MessageRepository): the message repository used to emit logs on HTTP requests
- refresh_token_error_status_codes (Tuple[int, ...]): Status codes to identify refresh token errors in response
- refresh_token_error_key (str): Key to identify refresh token error in response
- refresh_token_error_values (Tuple[str, ...]): List of values to check for exception during token refresh process
- refresh_token_error_messages (Optional[Mapping[str, str]]): Fixed user-facing messages for refresh token errors
If all of refresh_token_error_status_codes, refresh_token_error_key, and refresh_token_error_values are set, then http errors with such params will be wrapped in AirbyteTracedException. refresh_token_error_messages maps fixed provider error strings to fixed user-facing messages.
Format of the datetime; exists it if expires_in is returned as the expiration datetime instead of seconds until it expires
Indicates that the Token Expiry returns the date until which the token will be valid, not the amount of time it will be valid.
198 def get_token_refresh_endpoint(self) -> Optional[str]: 199 if self._token_refresh_endpoint is not None: 200 refresh_token_endpoint: str = self._token_refresh_endpoint.eval(self.config) 201 if not refresh_token_endpoint: 202 raise ValueError( 203 "OAuthAuthenticator was unable to evaluate token_refresh_endpoint parameter" 204 ) 205 return refresh_token_endpoint 206 return None
Returns the endpoint to refresh the access token
208 def get_client_id_name(self) -> str: 209 return self._client_id_name.eval(self.config) # type: ignore # eval returns a string in this context
The client id name to authenticate
211 def get_client_id(self) -> str: 212 client_id = self._client_id.eval(self.config) if self._client_id else self._client_id 213 if not client_id: 214 raise ValueError("OAuthAuthenticator was unable to evaluate client_id parameter") 215 return client_id # type: ignore # value will be returned as a string, or an error will be raised
The client id to authenticate
217 def get_client_secret_name(self) -> str: 218 return self._client_secret_name.eval(self.config) # type: ignore # eval returns a string in this context
The client secret name to authenticate
220 def get_client_secret(self) -> str: 221 client_secret = ( 222 self._client_secret.eval(self.config) if self._client_secret else self._client_secret 223 ) 224 if not client_secret: 225 # We've seen some APIs allowing empty client_secret so we will only log here 226 logger.warning( 227 "OAuthAuthenticator was unable to evaluate client_secret parameter hence it'll be empty" 228 ) 229 return client_secret # type: ignore # value will be returned as a string, which might be empty
The client secret to authenticate
231 def get_refresh_token_name(self) -> str: 232 return self._refresh_token_name.eval(self.config) # type: ignore # eval returns a string in this context
The refresh token name to authenticate
234 def get_refresh_token(self) -> Optional[str]: 235 return None if self._refresh_token is None else str(self._refresh_token.eval(self.config))
The token used to refresh the access token when it expires
240 def get_access_token_name(self) -> str: 241 return self.access_token_name.eval(self.config) # type: ignore # eval returns a string in this context
Field to extract access token from in the response
243 def get_expires_in_name(self) -> str: 244 return self.expires_in_name.eval(self.config) # type: ignore # eval returns a string in this context
Returns the expires_in field name
246 def get_grant_type_name(self) -> str: 247 return self.grant_type_name.eval(self.config) # type: ignore # eval returns a string in this context
Returns grant_type specified name for requesting access_token
249 def get_grant_type(self) -> str: 250 return self.grant_type.eval(self.config) # type: ignore # eval returns a string in this context
Returns grant_type specified for requesting access_token
252 def get_refresh_request_body(self) -> Mapping[str, Any]: 253 return self._refresh_request_body.eval(self.config)
Returns the request body to set on the refresh request
255 def get_refresh_request_headers(self) -> Mapping[str, Any]: 256 return self._refresh_request_headers.eval(self.config)
Returns the request headers to set on the refresh request
258 def should_send_refresh_request_as_query_params(self) -> bool: 259 return self._send_refresh_request_as_query_params
Returns True if the standard refresh args should be sent on the URL
query string instead of in the request body.
Defaults to False so existing authenticators retain their previous
behavior (params in body, no query params on the refresh URL). Subclasses
can override this to opt into the URL-query-string shape required by OAuth
providers like Gong.
261 def get_token_expiry_date(self) -> AirbyteDateTime: 262 if not self._has_access_token_been_initialized(): 263 return AirbyteDateTime.from_datetime(datetime.min) 264 return self._token_expiry_date # type: ignore # _token_expiry_date is an AirbyteDateTime. It is never None despite what mypy thinks
Expiration date of the access token
269 def set_token_expiry_date(self, value: AirbyteDateTime) -> None: 270 # Overrides of `refresh_access_token` written before 6.45.5 return the raw `expires_in`. 271 if not isinstance(value, datetime): 272 with suppress(ValueError, OverflowError): 273 value = self._parse_token_expiration_date(value) 274 self._token_expiry_date = value
Setter for access token expiration date
284 def build_refresh_request_body(self) -> Mapping[str, Any]: 285 """ 286 Returns the request body to set on the refresh request 287 288 Override to define additional parameters 289 """ 290 if self.use_profile_assertion: 291 return { 292 self.get_grant_type_name(): self.get_grant_type(), 293 self.get_assertion_name(): self.get_assertion(), 294 } 295 return super().build_refresh_request_body()
Returns the request body to set on the refresh request
Override to define additional parameters
297 @property 298 def access_token(self) -> str: 299 if self._access_token is None: 300 raise ValueError("access_token is not set") 301 return self._access_token
Returns the access token
Inherited Members
315@dataclass 316class DeclarativeSingleUseRefreshTokenOauth2Authenticator( 317 SingleUseRefreshTokenOauth2Authenticator, DeclarativeAuthenticator 318): 319 """ 320 Declarative version of SingleUseRefreshTokenOauth2Authenticator which can be used in declarative connectors. 321 """ 322 323 def __init__(self, *args: Any, **kwargs: Any) -> None: 324 super().__init__(*args, **kwargs)
Declarative version of SingleUseRefreshTokenOauth2Authenticator which can be used in declarative connectors.
Arguments:
- connector_config (Mapping[str, Any]): The full connector configuration
- token_refresh_endpoint (str): Full URL to the token refresh endpoint
- scopes (List[str], optional): List of OAuth scopes to pass in the refresh token request body. Defaults to None.
- access_token_name (str, optional): Name of the access token field, used to parse the refresh token response. Defaults to "access_token".
- expires_in_name (str, optional): Name of the name of the field that characterizes when the current access token will expire, used to parse the refresh token response. Defaults to "expires_in".
- refresh_token_name (str, optional): Name of the name of the refresh token field, used to parse the refresh token response. Defaults to "refresh_token".
- refresh_request_body (Mapping[str, Any], optional): Custom key value pair that will be added to the refresh token request body. Defaults to None.
- refresh_request_headers (Mapping[str, Any], optional): Custom key value pair that will be added to the refresh token request headers. Defaults to None.
- send_refresh_request_as_query_params (bool, optional): When True, the standard refresh args (
grant_type,refresh_token, client credentials when not in anAuthorizationheader, scopes, plus anyrefresh_request_bodyextras) are sent on the URL query string and the request body is emitted empty. Use this for OAuth providers like Gong that document their refresh endpoint with refresh args on the URL query string. Defaults to False. - grant_type (str, optional): OAuth grant type. Defaults to "refresh_token".
- client_id (Optional[str]): The client id to authenticate. If not specified, defaults to credentials.client_id in the config object.
- client_secret (Optional[str]): The client secret to authenticate. If not specified, defaults to credentials.client_secret in the config object.
- access_token_config_path (Sequence[str]): Dpath to the access_token field in the connector configuration. Defaults to ("credentials", "access_token").
- refresh_token_config_path (Sequence[str]): Dpath to the refresh_token field in the connector configuration. Defaults to ("credentials", "refresh_token").
- token_expiry_date_config_path (Sequence[str]): Dpath to the token_expiry_date field in the connector configuration. Defaults to ("credentials", "token_expiry_date").
- token_expiry_date_format (Optional[str]): Date format of the token expiry date field (set by expires_in_name). If specified the token expiry date is interpreted as time of expiration, as with token_expiry_is_time_of_expiration, otherwise as number of seconds until expiration.
- token_expiry_is_time_of_expiration bool: set True it if expires_in is returned as time of expiration instead of the number seconds until expiration
- message_repository (MessageRepository): The message repository used to emit logs on HTTP requests and control message on config update.
- emit_control_message_to_message_repository (bool): Whether the message repository additionally receives the CONNECTOR_CONFIG message. This is only for in-process consumers such as the Connector Builder, because messages in the repository are otherwise printed to stdout by the entrypoint and would duplicate the direct stdout emission. Defaults to False.
- refresh_token_error_messages (Optional[Mapping[str, str]]): Fixed user-facing messages for refresh token errors
Inherited Members
- SingleUseRefreshTokenOauth2Authenticator
- access_token
- get_refresh_token
- set_refresh_token
- get_token_expiry_date
- set_token_expiry_date
- token_has_expired
- get_access_token
- refresh_and_set_access_token
- refresh_access_token
- Oauth2Authenticator
- get_token_refresh_endpoint
- get_client_id_name
- get_client_id
- get_client_secret_name
- get_client_secret
- get_refresh_token_name
- get_access_token_name
- get_scopes
- get_expires_in_name
- get_refresh_request_body
- get_refresh_request_headers
- should_send_refresh_request_as_query_params
- get_grant_type_name
- get_grant_type
- token_expiry_is_time_of_expiration
- token_expiry_date_format
33@deprecated("DeclarativeStream has been deprecated in favor of the concurrent DefaultStream") 34@dataclass 35class DeclarativeStream(Stream): 36 """ 37 DeclarativeStream is a Stream that delegates most of its logic to its schema_load and retriever 38 39 Attributes: 40 name (str): stream name 41 primary_key (Optional[Union[str, List[str], List[List[str]]]]): the primary key of the stream 42 schema_loader (SchemaLoader): The schema loader 43 retriever (Retriever): The retriever 44 config (Config): The user-provided configuration as specified by the source's spec 45 stream_cursor_field (Optional[Union[InterpolatedString, str]]): The cursor field 46 stream. Transformations are applied in the order in which they are defined. 47 """ 48 49 retriever: Retriever 50 config: Config 51 parameters: InitVar[Mapping[str, Any]] 52 name: str 53 primary_key: Optional[Union[str, List[str], List[List[str]]]] 54 state_migrations: List[StateMigration] = field(repr=True, default_factory=list) 55 schema_loader: Optional[SchemaLoader] = None 56 _name: str = field(init=False, repr=False, default="") 57 _primary_key: str = field(init=False, repr=False, default="") 58 stream_cursor_field: Optional[Union[InterpolatedString, str]] = None 59 60 def __post_init__(self, parameters: Mapping[str, Any]) -> None: 61 self._stream_cursor_field = ( 62 InterpolatedString.create(self.stream_cursor_field, parameters=parameters) 63 if isinstance(self.stream_cursor_field, str) 64 else self.stream_cursor_field 65 ) 66 self._schema_loader = ( 67 self.schema_loader 68 if self.schema_loader 69 else DefaultSchemaLoader(config=self.config, parameters=parameters) 70 ) 71 72 @property # type: ignore 73 def primary_key(self) -> Optional[Union[str, List[str], List[List[str]]]]: 74 return self._primary_key 75 76 @primary_key.setter 77 def primary_key(self, value: str) -> None: 78 if not isinstance(value, property): 79 self._primary_key = value 80 81 @property 82 def exit_on_rate_limit(self) -> bool: 83 if isinstance(self.retriever, AsyncRetriever): 84 return self.retriever.exit_on_rate_limit 85 86 return self.retriever.requester.exit_on_rate_limit # type: ignore # abstract Retriever class has not requester attribute 87 88 @exit_on_rate_limit.setter 89 def exit_on_rate_limit(self, value: bool) -> None: 90 if isinstance(self.retriever, AsyncRetriever): 91 self.retriever.exit_on_rate_limit = value 92 else: 93 self.retriever.requester.exit_on_rate_limit = value # type: ignore[attr-defined] 94 95 @property # type: ignore 96 def name(self) -> str: 97 """ 98 :return: Stream name. By default this is the implementing class name, but it can be overridden as needed. 99 """ 100 return self._name 101 102 @name.setter 103 def name(self, value: str) -> None: 104 if not isinstance(value, property): 105 self._name = value 106 107 @property 108 def state(self) -> MutableMapping[str, Any]: 109 return self.retriever.state # type: ignore 110 111 @state.setter 112 def state(self, value: MutableMapping[str, Any]) -> None: 113 """State setter, accept state serialized by state getter.""" 114 state: Mapping[str, Any] = value 115 if self.state_migrations: 116 for migration in self.state_migrations: 117 if migration.should_migrate(state): 118 state = migration.migrate(state) 119 self.retriever.state = state 120 121 def get_updated_state( 122 self, current_stream_state: MutableMapping[str, Any], latest_record: Mapping[str, Any] 123 ) -> MutableMapping[str, Any]: 124 return self.state 125 126 @property 127 def cursor_field(self) -> Union[str, List[str]]: 128 """ 129 Override to return the default cursor field used by this stream e.g: an API entity might always use created_at as the cursor field. 130 :return: The name of the field used as a cursor. If the cursor is nested, return an array consisting of the path to the cursor. 131 """ 132 cursor = self._stream_cursor_field.eval(self.config) # type: ignore # _stream_cursor_field is always cast to interpolated string 133 return cursor if cursor else [] 134 135 @property 136 def is_resumable(self) -> bool: 137 # Declarative sources always implement state getter/setter, but whether it supports checkpointing is based on 138 # if the retriever has a cursor defined. 139 return self.retriever.cursor is not None if hasattr(self.retriever, "cursor") else False 140 141 def read_records( 142 self, 143 sync_mode: SyncMode, 144 cursor_field: Optional[List[str]] = None, 145 stream_slice: Optional[Mapping[str, Any]] = None, 146 stream_state: Optional[Mapping[str, Any]] = None, 147 ) -> Iterable[Mapping[str, Any]]: 148 """ 149 :param: stream_state We knowingly avoid using stream_state as we want cursors to manage their own state. 150 """ 151 if stream_slice is None or ( 152 not isinstance(stream_slice, StreamSlice) and stream_slice == {} 153 ): 154 # As the parameter is Optional, many would just call `read_records(sync_mode)` during testing without specifying the field 155 # As part of the declarative model without custom components, this should never happen as the CDK would wire up a 156 # SinglePartitionRouter that would create this StreamSlice properly 157 # As part of the declarative model with custom components, a user that would return a `None` slice would now have the default 158 # empty slice which seems to make sense. 159 stream_slice = StreamSlice(partition={}, cursor_slice={}) 160 if not isinstance(stream_slice, StreamSlice): 161 raise ValueError( 162 f"DeclarativeStream does not support stream_slices that are not StreamSlice. Got {stream_slice}" 163 ) 164 yield from self.retriever.read_records(self.get_json_schema(), stream_slice) # type: ignore # records are of the correct type 165 166 def get_json_schema(self) -> Mapping[str, Any]: # type: ignore 167 """ 168 :return: A dict of the JSON schema representing this stream. 169 170 The default implementation of this method looks for a JSONSchema file with the same name as this stream's "name" property. 171 Override as needed. 172 """ 173 return self._schema_loader.get_json_schema() 174 175 def stream_slices( 176 self, 177 *, 178 sync_mode: SyncMode, 179 cursor_field: Optional[List[str]] = None, 180 stream_state: Optional[Mapping[str, Any]] = None, 181 ) -> Iterable[Optional[StreamSlice]]: 182 """ 183 Override to define the slices for this stream. See the stream slicing section of the docs for more information. 184 185 :param sync_mode: 186 :param cursor_field: 187 :param stream_state: we knowingly avoid using stream_state as we want cursors to manage their own state 188 :return: 189 """ 190 return self.retriever.stream_slices() 191 192 @property 193 def state_checkpoint_interval(self) -> Optional[int]: 194 """ 195 We explicitly disable checkpointing here. There are a couple reasons for that and not all are documented here but: 196 * In the case where records are not ordered, the granularity of what is ordered is the slice. Therefore, we will only update the 197 cursor value once at the end of every slice. 198 * Updating the state once every record would generate issues for data feed stop conditions or semi-incremental syncs where the 199 important state is the one at the beginning of the slice 200 """ 201 return None 202 203 def get_cursor(self) -> Optional[Cursor]: 204 return None 205 206 def _get_checkpoint_reader( 207 self, 208 logger: logging.Logger, 209 cursor_field: Optional[List[str]], 210 sync_mode: SyncMode, 211 stream_state: MutableMapping[str, Any], 212 ) -> CheckpointReader: 213 """ 214 This method is overridden to prevent issues with stream slice classification for incremental streams that have parent streams. 215 216 The classification logic, when used with `itertools.tee`, creates a copy of the stream slices. When `stream_slices` is called 217 the second time, the parent records generated during the classification phase are lost. This occurs because `itertools.tee` 218 only buffers the results, meaning the logic in `simple_retriever` that observes and updates the cursor isn't executed again. 219 220 By overriding this method, we ensure that the stream slices are processed correctly and parent records are not lost, 221 allowing the cursor to function as expected. 222 """ 223 mappings_or_slices = self.stream_slices( 224 cursor_field=cursor_field, 225 sync_mode=sync_mode, # todo: change this interface to no longer rely on sync_mode for behavior 226 stream_state=stream_state, 227 ) 228 229 cursor = self.get_cursor() 230 checkpoint_mode = self._checkpoint_mode 231 232 if isinstance( 233 cursor, (GlobalSubstreamCursor, PerPartitionCursor, PerPartitionWithGlobalCursor) 234 ): 235 self.has_multiple_slices = True 236 return CursorBasedCheckpointReader( 237 stream_slices=mappings_or_slices, 238 cursor=cursor, 239 read_state_from_cursor=checkpoint_mode == CheckpointMode.RESUMABLE_FULL_REFRESH, 240 ) 241 242 return super()._get_checkpoint_reader(logger, cursor_field, sync_mode, stream_state)
DeclarativeStream is a Stream that delegates most of its logic to its schema_load and retriever
Attributes:
- name (str): stream name
- primary_key (Optional[Union[str, List[str], List[List[str]]]]): the primary key of the stream
- schema_loader (SchemaLoader): The schema loader
- retriever (Retriever): The retriever
- config (Config): The user-provided configuration as specified by the source's spec
- stream_cursor_field (Optional[Union[InterpolatedString, str]]): The cursor field
- stream. Transformations are applied in the order in which they are defined.
95 @property # type: ignore 96 def name(self) -> str: 97 """ 98 :return: Stream name. By default this is the implementing class name, but it can be overridden as needed. 99 """ 100 return self._name
Returns
Stream name. By default this is the implementing class name, but it can be overridden as needed.
72 @property # type: ignore 73 def primary_key(self) -> Optional[Union[str, List[str], List[List[str]]]]: 74 return self._primary_key
Returns
string if single primary key, list of strings if composite primary key, list of list of strings if composite primary key consisting of nested fields. If the stream has no primary keys, return None.
81 @property 82 def exit_on_rate_limit(self) -> bool: 83 if isinstance(self.retriever, AsyncRetriever): 84 return self.retriever.exit_on_rate_limit 85 86 return self.retriever.requester.exit_on_rate_limit # type: ignore # abstract Retriever class has not requester attribute
Exit on rate limit getter, should return bool value. False if the stream will retry endlessly when rate limited.
107 @property 108 def state(self) -> MutableMapping[str, Any]: 109 return self.retriever.state # type: ignore
State setter, accept state serialized by state getter.
121 def get_updated_state( 122 self, current_stream_state: MutableMapping[str, Any], latest_record: Mapping[str, Any] 123 ) -> MutableMapping[str, Any]: 124 return self.state
DEPRECATED. Please use explicit state property instead, see IncrementalMixin docs.
Override to extract state from the latest record. Needed to implement incremental sync.
Inspects the latest record extracted from the data source and the current state object and return an updated state object.
For example: if the state object is based on created_at timestamp, and the current state is {'created_at': 10}, and the latest_record is {'name': 'octavia', 'created_at': 20 } then this method would return {'created_at': 20} to indicate state should be updated to this object.
Parameters
- current_stream_state: The stream's current state object
- latest_record: The latest record extracted from the stream
Returns
An updated state object
126 @property 127 def cursor_field(self) -> Union[str, List[str]]: 128 """ 129 Override to return the default cursor field used by this stream e.g: an API entity might always use created_at as the cursor field. 130 :return: The name of the field used as a cursor. If the cursor is nested, return an array consisting of the path to the cursor. 131 """ 132 cursor = self._stream_cursor_field.eval(self.config) # type: ignore # _stream_cursor_field is always cast to interpolated string 133 return cursor if cursor else []
Override to return the default cursor field used by this stream e.g: an API entity might always use created_at as the cursor field.
Returns
The name of the field used as a cursor. If the cursor is nested, return an array consisting of the path to the cursor.
135 @property 136 def is_resumable(self) -> bool: 137 # Declarative sources always implement state getter/setter, but whether it supports checkpointing is based on 138 # if the retriever has a cursor defined. 139 return self.retriever.cursor is not None if hasattr(self.retriever, "cursor") else False
Returns
True if this stream allows the checkpointing of sync progress and can resume from it on subsequent attempts. This differs from supports_incremental because certain kinds of streams like those supporting resumable full refresh can checkpoint progress in between attempts for improved fault tolerance. However, they will start from the beginning on the next sync job.
141 def read_records( 142 self, 143 sync_mode: SyncMode, 144 cursor_field: Optional[List[str]] = None, 145 stream_slice: Optional[Mapping[str, Any]] = None, 146 stream_state: Optional[Mapping[str, Any]] = None, 147 ) -> Iterable[Mapping[str, Any]]: 148 """ 149 :param: stream_state We knowingly avoid using stream_state as we want cursors to manage their own state. 150 """ 151 if stream_slice is None or ( 152 not isinstance(stream_slice, StreamSlice) and stream_slice == {} 153 ): 154 # As the parameter is Optional, many would just call `read_records(sync_mode)` during testing without specifying the field 155 # As part of the declarative model without custom components, this should never happen as the CDK would wire up a 156 # SinglePartitionRouter that would create this StreamSlice properly 157 # As part of the declarative model with custom components, a user that would return a `None` slice would now have the default 158 # empty slice which seems to make sense. 159 stream_slice = StreamSlice(partition={}, cursor_slice={}) 160 if not isinstance(stream_slice, StreamSlice): 161 raise ValueError( 162 f"DeclarativeStream does not support stream_slices that are not StreamSlice. Got {stream_slice}" 163 ) 164 yield from self.retriever.read_records(self.get_json_schema(), stream_slice) # type: ignore # records are of the correct type
Parameters
- stream_state We knowingly avoid using stream_state as we want cursors to manage their own state.
166 def get_json_schema(self) -> Mapping[str, Any]: # type: ignore 167 """ 168 :return: A dict of the JSON schema representing this stream. 169 170 The default implementation of this method looks for a JSONSchema file with the same name as this stream's "name" property. 171 Override as needed. 172 """ 173 return self._schema_loader.get_json_schema()
Returns
A dict of the JSON schema representing this stream.
The default implementation of this method looks for a JSONSchema file with the same name as this stream's "name" property. Override as needed.
175 def stream_slices( 176 self, 177 *, 178 sync_mode: SyncMode, 179 cursor_field: Optional[List[str]] = None, 180 stream_state: Optional[Mapping[str, Any]] = None, 181 ) -> Iterable[Optional[StreamSlice]]: 182 """ 183 Override to define the slices for this stream. See the stream slicing section of the docs for more information. 184 185 :param sync_mode: 186 :param cursor_field: 187 :param stream_state: we knowingly avoid using stream_state as we want cursors to manage their own state 188 :return: 189 """ 190 return self.retriever.stream_slices()
Override to define the slices for this stream. See the stream slicing section of the docs for more information.
Parameters
- sync_mode:
- cursor_field:
- stream_state: we knowingly avoid using stream_state as we want cursors to manage their own state
Returns
192 @property 193 def state_checkpoint_interval(self) -> Optional[int]: 194 """ 195 We explicitly disable checkpointing here. There are a couple reasons for that and not all are documented here but: 196 * In the case where records are not ordered, the granularity of what is ordered is the slice. Therefore, we will only update the 197 cursor value once at the end of every slice. 198 * Updating the state once every record would generate issues for data feed stop conditions or semi-incremental syncs where the 199 important state is the one at the beginning of the slice 200 """ 201 return None
We explicitly disable checkpointing here. There are a couple reasons for that and not all are documented here but:
- In the case where records are not ordered, the granularity of what is ordered is the slice. Therefore, we will only update the cursor value once at the end of every slice.
- Updating the state once every record would generate issues for data feed stop conditions or semi-incremental syncs where the important state is the one at the beginning of the slice
A Cursor is an interface that a stream can implement to manage how its internal state is read and updated while reading records. Historically, Python connectors had no concept of a cursor to manage state. Python streams need to define a cursor implementation and override this method to manage state through a Cursor.
15@dataclass 16class Decoder: 17 """ 18 Decoder strategy to transform a requests.Response into a Mapping[str, Any] 19 """ 20 21 @abstractmethod 22 def is_stream_response(self) -> bool: 23 """ 24 Set to True if you'd like to use stream=True option in http requester 25 """ 26 27 @abstractmethod 28 def decode(self, response: requests.Response) -> DECODER_OUTPUT_TYPE: 29 """ 30 Decodes a requests.Response into a Mapping[str, Any] or an array 31 :param response: the response to decode 32 :return: Generator of Mapping describing the response 33 """
Decoder strategy to transform a requests.Response into a Mapping[str, Any]
21 @abstractmethod 22 def is_stream_response(self) -> bool: 23 """ 24 Set to True if you'd like to use stream=True option in http requester 25 """
Set to True if you'd like to use stream=True option in http requester
27 @abstractmethod 28 def decode(self, response: requests.Response) -> DECODER_OUTPUT_TYPE: 29 """ 30 Decodes a requests.Response into a Mapping[str, Any] or an array 31 :param response: the response to decode 32 :return: Generator of Mapping describing the response 33 """
Decodes a requests.Response into a Mapping[str, Any] or an array
Parameters
- response: the response to decode
Returns
Generator of Mapping describing the response
37@dataclass 38class DefaultPaginator(Paginator): 39 """ 40 Default paginator to request pages of results with a fixed size until the pagination strategy no longer returns a next_page_token 41 42 Examples: 43 1. 44 * fetches up to 10 records at a time by setting the "limit" request param to 10 45 * updates the request path with "{{ response._metadata.next }}" 46 ``` 47 paginator: 48 type: "DefaultPaginator" 49 page_size_option: 50 type: RequestOption 51 inject_into: request_parameter 52 field_name: limit 53 page_token_option: 54 type: RequestPath 55 path: "location" 56 pagination_strategy: 57 type: "CursorPagination" 58 cursor_value: "{{ response._metadata.next }}" 59 page_size: 10 60 ``` 61 62 2. 63 * fetches up to 5 records at a time by setting the "page_size" header to 5 64 * increments a record counter and set the request parameter "offset" to the value of the counter 65 ``` 66 paginator: 67 type: "DefaultPaginator" 68 page_size_option: 69 type: RequestOption 70 inject_into: header 71 field_name: page_size 72 pagination_strategy: 73 type: "OffsetIncrement" 74 page_size: 5 75 page_token_option: 76 option_type: "request_parameter" 77 field_name: "offset" 78 ``` 79 80 3. 81 * fetches up to 5 records at a time by setting the "page_size" request param to 5 82 * increments a page counter and set the request parameter "page" to the value of the counter 83 ``` 84 paginator: 85 type: "DefaultPaginator" 86 page_size_option: 87 type: RequestOption 88 inject_into: request_parameter 89 field_name: page_size 90 pagination_strategy: 91 type: "PageIncrement" 92 page_size: 5 93 page_token_option: 94 type: RequestOption 95 option_type: "request_parameter" 96 field_name: "page" 97 ``` 98 Attributes: 99 page_size_option (Optional[RequestOption]): the request option to set the page size. Cannot be injected in the path. 100 page_token_option (Optional[RequestPath, RequestOption]): the request option to set the page token 101 pagination_strategy (PaginationStrategy): Strategy defining how to get the next page token 102 config (Config): connection config 103 url_base (Union[InterpolatedString, str]): endpoint's base url 104 decoder (Decoder): decoder to decode the response 105 """ 106 107 pagination_strategy: PaginationStrategy 108 config: Config 109 url_base: Union[InterpolatedString, str] 110 parameters: InitVar[Mapping[str, Any]] 111 decoder: Decoder = field( 112 default_factory=lambda: PaginationDecoderDecorator(decoder=JsonDecoder(parameters={})) 113 ) 114 page_size_option: Optional[RequestOption] = None 115 page_token_option: Optional[Union[RequestPath, RequestOption]] = None 116 117 def __post_init__(self, parameters: Mapping[str, Any]) -> None: 118 if self.page_size_option and not self.pagination_strategy.get_page_size(): 119 raise ValueError( 120 "page_size_option cannot be set if the pagination strategy does not have a page_size" 121 ) 122 if isinstance(self.url_base, str): 123 self.url_base = InterpolatedString(string=self.url_base, parameters=parameters) 124 125 if self.page_token_option and not isinstance(self.page_token_option, RequestPath): 126 _validate_component_request_option_paths( 127 self.config, 128 self.page_size_option, 129 self.page_token_option, 130 ) 131 132 def get_initial_token(self) -> Optional[Any]: 133 """ 134 Return the page token that should be used for the first request of a stream 135 136 WARNING: get_initial_token() should not be used by streams that use RFR that perform checkpointing 137 of state using page numbers. Because paginators are stateless 138 """ 139 return self.pagination_strategy.initial_token 140 141 def get_page_size(self) -> Optional[int]: 142 return self.pagination_strategy.get_page_size() 143 144 def next_page_token( 145 self, 146 response: requests.Response, 147 last_page_size: int, 148 last_record: Optional[Record], 149 last_page_token_value: Optional[Any] = None, 150 page_size_override: Optional[int] = None, 151 stream_slice: Optional[StreamSlice] = None, 152 ) -> Optional[Mapping[str, Any]]: 153 next_page_token = self.pagination_strategy.next_page_token( 154 response=response, 155 last_page_size=last_page_size, 156 last_record=last_record, 157 last_page_token_value=last_page_token_value, 158 **page_size_override_kwargs(page_size_override), 159 **stream_slice_kwargs(self.pagination_strategy.next_page_token, stream_slice), 160 ) 161 if next_page_token: 162 return {"next_page_token": next_page_token} 163 else: 164 return None 165 166 def path( 167 self, 168 next_page_token: Optional[Mapping[str, Any]], 169 stream_state: Optional[Mapping[str, Any]] = None, 170 stream_slice: Optional[StreamSlice] = None, 171 ) -> Optional[str]: 172 token = next_page_token.get("next_page_token") if next_page_token else None 173 if token and self.page_token_option and isinstance(self.page_token_option, RequestPath): 174 return str(token) 175 else: 176 return None 177 178 def get_request_params( 179 self, 180 *, 181 stream_state: Optional[StreamState] = None, 182 stream_slice: Optional[StreamSlice] = None, 183 next_page_token: Optional[Mapping[str, Any]] = None, 184 page_size_override: Optional[int] = None, 185 ) -> MutableMapping[str, Any]: 186 return self._get_request_options( 187 RequestOptionType.request_parameter, next_page_token, page_size_override 188 ) 189 190 def get_request_headers( 191 self, 192 *, 193 stream_state: Optional[StreamState] = None, 194 stream_slice: Optional[StreamSlice] = None, 195 next_page_token: Optional[Mapping[str, Any]] = None, 196 page_size_override: Optional[int] = None, 197 ) -> Mapping[str, str]: 198 return self._get_request_options( 199 RequestOptionType.header, next_page_token, page_size_override 200 ) 201 202 def get_request_body_data( 203 self, 204 *, 205 stream_state: Optional[StreamState] = None, 206 stream_slice: Optional[StreamSlice] = None, 207 next_page_token: Optional[Mapping[str, Any]] = None, 208 page_size_override: Optional[int] = None, 209 ) -> Mapping[str, Any]: 210 return self._get_request_options( 211 RequestOptionType.body_data, next_page_token, page_size_override 212 ) 213 214 def get_request_body_json( 215 self, 216 *, 217 stream_state: Optional[StreamState] = None, 218 stream_slice: Optional[StreamSlice] = None, 219 next_page_token: Optional[Mapping[str, Any]] = None, 220 page_size_override: Optional[int] = None, 221 ) -> Mapping[str, Any]: 222 return self._get_request_options( 223 RequestOptionType.body_json, next_page_token, page_size_override 224 ) 225 226 def _get_request_options( 227 self, 228 option_type: RequestOptionType, 229 next_page_token: Optional[Mapping[str, Any]], 230 page_size_override: Optional[int] = None, 231 ) -> MutableMapping[str, Any]: 232 options: MutableMapping[str, Any] = {} 233 234 token = next_page_token.get("next_page_token") if next_page_token else None 235 if ( 236 self.page_token_option 237 and token is not None 238 and isinstance(self.page_token_option, RequestOption) 239 and self.page_token_option.inject_into == option_type 240 ): 241 self.page_token_option.inject_into_request(options, token, self.config) 242 243 if self.page_size_option and self.page_size_option.inject_into == option_type: 244 page_size = ( 245 page_size_override 246 if page_size_override is not None 247 else self.pagination_strategy.get_page_size() 248 ) 249 if page_size: 250 self.page_size_option.inject_into_request(options, page_size, self.config) 251 252 return options
Default paginator to request pages of results with a fixed size until the pagination strategy no longer returns a next_page_token
Examples:
1.
- fetches up to 10 records at a time by setting the "limit" request param to 10
- updates the request path with "{{ response._metadata.next }}"
paginator: type: "DefaultPaginator" page_size_option: type: RequestOption inject_into: request_parameter field_name: limit page_token_option: type: RequestPath path: "location" pagination_strategy: type: "CursorPagination" cursor_value: "{{ response._metadata.next }}" page_size: 102.
- fetches up to 5 records at a time by setting the "page_size" header to 5
- increments a record counter and set the request parameter "offset" to the value of the counter
paginator: type: "DefaultPaginator" page_size_option: type: RequestOption inject_into: header field_name: page_size pagination_strategy: type: "OffsetIncrement" page_size: 5 page_token_option: option_type: "request_parameter" field_name: "offset"3.
- fetches up to 5 records at a time by setting the "page_size" request param to 5
- increments a page counter and set the request parameter "page" to the value of the counter
paginator: type: "DefaultPaginator" page_size_option: type: RequestOption inject_into: request_parameter field_name: page_size pagination_strategy: type: "PageIncrement" page_size: 5 page_token_option: type: RequestOption option_type: "request_parameter" field_name: "page"
Attributes:
- page_size_option (Optional[RequestOption]): the request option to set the page size. Cannot be injected in the path.
- page_token_option (Optional[RequestPath, RequestOption]): the request option to set the page token
- pagination_strategy (PaginationStrategy): Strategy defining how to get the next page token
- config (Config): connection config
- url_base (Union[InterpolatedString, str]): endpoint's base url
- decoder (Decoder): decoder to decode the response
132 def get_initial_token(self) -> Optional[Any]: 133 """ 134 Return the page token that should be used for the first request of a stream 135 136 WARNING: get_initial_token() should not be used by streams that use RFR that perform checkpointing 137 of state using page numbers. Because paginators are stateless 138 """ 139 return self.pagination_strategy.initial_token
Return the page token that should be used for the first request of a stream
WARNING: get_initial_token() should not be used by streams that use RFR that perform checkpointing of state using page numbers. Because paginators are stateless
Evaluated against the config alone. A pagination strategy whose page_size template references the
response evaluates it with the response in context inside next_page_token, so the two can disagree -
pre-existing, and only observable for a page_size that is not a constant.
Returns
the number of records this paginator asks for per page, or None if it does not define one
144 def next_page_token( 145 self, 146 response: requests.Response, 147 last_page_size: int, 148 last_record: Optional[Record], 149 last_page_token_value: Optional[Any] = None, 150 page_size_override: Optional[int] = None, 151 stream_slice: Optional[StreamSlice] = None, 152 ) -> Optional[Mapping[str, Any]]: 153 next_page_token = self.pagination_strategy.next_page_token( 154 response=response, 155 last_page_size=last_page_size, 156 last_record=last_record, 157 last_page_token_value=last_page_token_value, 158 **page_size_override_kwargs(page_size_override), 159 **stream_slice_kwargs(self.pagination_strategy.next_page_token, stream_slice), 160 ) 161 if next_page_token: 162 return {"next_page_token": next_page_token} 163 else: 164 return None
Returns the next_page_token to use to fetch the next page of records.
Parameters
- response: the response to process
- last_page_size: the number of records read from the response
- last_record: the last record extracted from the response
- last_page_token_value: The current value of the page token made on the last request
- page_size_override: the page size that was actually requested, when it differs from the configured
one because of a
REDUCE_PAGE_SIZEresponse action - stream_slice: the slice the page was read for, so that a stop condition can compare the page against the slice's own window or partition
Returns
A mapping {"next_page_token":
} for the next page from the input response object. Returning None means there are no more pages to read in this response.
166 def path( 167 self, 168 next_page_token: Optional[Mapping[str, Any]], 169 stream_state: Optional[Mapping[str, Any]] = None, 170 stream_slice: Optional[StreamSlice] = None, 171 ) -> Optional[str]: 172 token = next_page_token.get("next_page_token") if next_page_token else None 173 if token and self.page_token_option and isinstance(self.page_token_option, RequestPath): 174 return str(token) 175 else: 176 return None
Returns the URL path to hit to fetch the next page of records
e.g: if you wanted to hit https://myapi.com/v1/some_entity then this will return "some_entity"
Returns
path to hit to fetch the next request. Returning None means the path is not defined by the next_page_token
178 def get_request_params( 179 self, 180 *, 181 stream_state: Optional[StreamState] = None, 182 stream_slice: Optional[StreamSlice] = None, 183 next_page_token: Optional[Mapping[str, Any]] = None, 184 page_size_override: Optional[int] = None, 185 ) -> MutableMapping[str, Any]: 186 return self._get_request_options( 187 RequestOptionType.request_parameter, next_page_token, page_size_override 188 )
Specifies the query parameters that should be set on an outgoing HTTP request given the inputs.
E.g: you might want to define query parameters for paging if next_page_token is not None.
190 def get_request_headers( 191 self, 192 *, 193 stream_state: Optional[StreamState] = None, 194 stream_slice: Optional[StreamSlice] = None, 195 next_page_token: Optional[Mapping[str, Any]] = None, 196 page_size_override: Optional[int] = None, 197 ) -> Mapping[str, str]: 198 return self._get_request_options( 199 RequestOptionType.header, next_page_token, page_size_override 200 )
Return any non-auth headers. Authentication headers will overwrite any overlapping headers returned from this method.
202 def get_request_body_data( 203 self, 204 *, 205 stream_state: Optional[StreamState] = None, 206 stream_slice: Optional[StreamSlice] = None, 207 next_page_token: Optional[Mapping[str, Any]] = None, 208 page_size_override: Optional[int] = None, 209 ) -> Mapping[str, Any]: 210 return self._get_request_options( 211 RequestOptionType.body_data, next_page_token, page_size_override 212 )
Specifies how to populate the body of the request with a non-JSON payload.
If returns a ready text that it will be sent as is. If returns a dict that it will be converted to a urlencoded form. E.g. {"key1": "value1", "key2": "value2"} => "key1=value1&key2=value2"
At the same time only one of the 'request_body_data' and 'request_body_json' functions can be overridden.
214 def get_request_body_json( 215 self, 216 *, 217 stream_state: Optional[StreamState] = None, 218 stream_slice: Optional[StreamSlice] = None, 219 next_page_token: Optional[Mapping[str, Any]] = None, 220 page_size_override: Optional[int] = None, 221 ) -> Mapping[str, Any]: 222 return self._get_request_options( 223 RequestOptionType.body_json, next_page_token, page_size_override 224 )
Specifies how to populate the body of the request with a JSON payload.
At the same time only one of the 'request_body_data' and 'request_body_json' functions can be overridden.
15@dataclass 16class DefaultRequestOptionsProvider(RequestOptionsProvider): 17 """ 18 Request options provider that extracts fields from the stream_slice and injects them into the respective location in the 19 outbound request being made 20 """ 21 22 parameters: InitVar[Mapping[str, Any]] 23 24 def __post_init__(self, parameters: Mapping[str, Any]) -> None: 25 pass 26 27 def get_request_params( 28 self, 29 *, 30 stream_state: Optional[StreamState] = None, 31 stream_slice: Optional[StreamSlice] = None, 32 next_page_token: Optional[Mapping[str, Any]] = None, 33 ) -> Mapping[str, Any]: 34 return {} 35 36 def get_request_headers( 37 self, 38 *, 39 stream_state: Optional[StreamState] = None, 40 stream_slice: Optional[StreamSlice] = None, 41 next_page_token: Optional[Mapping[str, Any]] = None, 42 ) -> Mapping[str, Any]: 43 return {} 44 45 def get_request_body_data( 46 self, 47 *, 48 stream_state: Optional[StreamState] = None, 49 stream_slice: Optional[StreamSlice] = None, 50 next_page_token: Optional[Mapping[str, Any]] = None, 51 ) -> Union[Mapping[str, Any], str]: 52 return {} 53 54 def get_request_body_json( 55 self, 56 *, 57 stream_state: Optional[StreamState] = None, 58 stream_slice: Optional[StreamSlice] = None, 59 next_page_token: Optional[Mapping[str, Any]] = None, 60 ) -> Mapping[str, Any]: 61 return {}
Request options provider that extracts fields from the stream_slice and injects them into the respective location in the outbound request being made
27 def get_request_params( 28 self, 29 *, 30 stream_state: Optional[StreamState] = None, 31 stream_slice: Optional[StreamSlice] = None, 32 next_page_token: Optional[Mapping[str, Any]] = None, 33 ) -> Mapping[str, Any]: 34 return {}
Specifies the query parameters that should be set on an outgoing HTTP request given the inputs.
E.g: you might want to define query parameters for paging if next_page_token is not None.
36 def get_request_headers( 37 self, 38 *, 39 stream_state: Optional[StreamState] = None, 40 stream_slice: Optional[StreamSlice] = None, 41 next_page_token: Optional[Mapping[str, Any]] = None, 42 ) -> Mapping[str, Any]: 43 return {}
Return any non-auth headers. Authentication headers will overwrite any overlapping headers returned from this method.
45 def get_request_body_data( 46 self, 47 *, 48 stream_state: Optional[StreamState] = None, 49 stream_slice: Optional[StreamSlice] = None, 50 next_page_token: Optional[Mapping[str, Any]] = None, 51 ) -> Union[Mapping[str, Any], str]: 52 return {}
Specifies how to populate the body of the request with a non-JSON payload.
If returns a ready text that it will be sent as is. If returns a dict that it will be converted to a urlencoded form. E.g. {"key1": "value1", "key2": "value2"} => "key1=value1&key2=value2"
At the same time only one of the 'request_body_data' and 'request_body_json' functions can be overridden.
54 def get_request_body_json( 55 self, 56 *, 57 stream_state: Optional[StreamState] = None, 58 stream_slice: Optional[StreamSlice] = None, 59 next_page_token: Optional[Mapping[str, Any]] = None, 60 ) -> Mapping[str, Any]: 61 return {}
Specifies how to populate the body of the request with a JSON payload.
At the same time only one of the 'request_body_data' and 'request_body_json' functions can be overridden.
19@dataclass 20class DpathExtractor(RecordExtractor): 21 """ 22 Record extractor that searches a decoded response over a path defined as an array of fields. 23 24 If the field path points to an array, that array is returned. 25 If the field path points to an object, that object is returned wrapped as an array. 26 If the field path points to an empty object, an empty array is returned. 27 If the field path points to a non-existing path, an empty array is returned. 28 29 Optionally, records can be expanded by providing a RecordExpander component. 30 When record_expander is configured, each extracted record is passed through the 31 expander which extracts items from nested array fields and emits each item as a 32 separate record. 33 34 Examples of instantiating this transform: 35 ``` 36 extractor: 37 type: DpathExtractor 38 field_path: 39 - "root" 40 - "data" 41 ``` 42 43 ``` 44 extractor: 45 type: DpathExtractor 46 field_path: 47 - "root" 48 - "{{ parameters['field'] }}" 49 ``` 50 51 ``` 52 extractor: 53 type: DpathExtractor 54 field_path: [] 55 ``` 56 57 ``` 58 extractor: 59 type: DpathExtractor 60 field_path: 61 - "data" 62 - "object" 63 record_expander: 64 type: RecordExpander 65 expand_records_from_field: 66 - "lines" 67 - "data" 68 remain_original_record: true 69 ``` 70 71 Attributes: 72 field_path (Union[InterpolatedString, str]): Path to the field that should be extracted 73 config (Config): The user-provided configuration as specified by the source's spec 74 decoder (Decoder): The decoder responsible to transfom the response in a Mapping 75 record_expander (Optional[RecordExpander]): Optional component to expand records by extracting items from nested array fields 76 """ 77 78 field_path: List[Union[InterpolatedString, str]] 79 config: Config 80 parameters: InitVar[Mapping[str, Any]] 81 decoder: Decoder = field(default_factory=lambda: JsonDecoder(parameters={})) 82 record_expander: Optional[RecordExpander] = None 83 84 def __post_init__(self, parameters: Mapping[str, Any]) -> None: 85 self._field_path = [ 86 InterpolatedString.create(path, parameters=parameters) for path in self.field_path 87 ] 88 for path_index in range(len(self.field_path)): 89 if isinstance(self.field_path[path_index], str): 90 self._field_path[path_index] = InterpolatedString.create( 91 self.field_path[path_index], parameters=parameters 92 ) 93 94 def extract_records(self, response: requests.Response) -> Iterable[MutableMapping[Any, Any]]: 95 for body in self.decoder.decode(response): 96 if len(self._field_path) == 0: 97 extracted = body 98 else: 99 path = [path.eval(self.config) for path in self._field_path] 100 if "*" in path: 101 extracted = dpath.values(body, path) 102 else: 103 extracted = dpath.get(body, path, default=[]) # type: ignore # extracted will be a MutableMapping, given input data structure 104 if isinstance(extracted, list): 105 if not self.record_expander: 106 yield from extracted 107 else: 108 for record in extracted: 109 yield from self.record_expander.expand_record(record) 110 elif extracted: 111 if self.record_expander: 112 yield from self.record_expander.expand_record(extracted) 113 else: 114 yield extracted 115 else: 116 yield from []
Record extractor that searches a decoded response over a path defined as an array of fields.
If the field path points to an array, that array is returned. If the field path points to an object, that object is returned wrapped as an array. If the field path points to an empty object, an empty array is returned. If the field path points to a non-existing path, an empty array is returned.
Optionally, records can be expanded by providing a RecordExpander component. When record_expander is configured, each extracted record is passed through the expander which extracts items from nested array fields and emits each item as a separate record.
Examples of instantiating this transform:
extractor:
type: DpathExtractor
field_path:
- "root"
- "data"
extractor:
type: DpathExtractor
field_path:
- "root"
- "{{ parameters['field'] }}"
extractor:
type: DpathExtractor
field_path: []
extractor:
type: DpathExtractor
field_path:
- "data"
- "object"
record_expander:
type: RecordExpander
expand_records_from_field:
- "lines"
- "data"
remain_original_record: true
Attributes:
- field_path (Union[InterpolatedString, str]): Path to the field that should be extracted
- config (Config): The user-provided configuration as specified by the source's spec
- decoder (Decoder): The decoder responsible to transfom the response in a Mapping
- record_expander (Optional[RecordExpander]): Optional component to expand records by extracting items from nested array fields
94 def extract_records(self, response: requests.Response) -> Iterable[MutableMapping[Any, Any]]: 95 for body in self.decoder.decode(response): 96 if len(self._field_path) == 0: 97 extracted = body 98 else: 99 path = [path.eval(self.config) for path in self._field_path] 100 if "*" in path: 101 extracted = dpath.values(body, path) 102 else: 103 extracted = dpath.get(body, path, default=[]) # type: ignore # extracted will be a MutableMapping, given input data structure 104 if isinstance(extracted, list): 105 if not self.record_expander: 106 yield from extracted 107 else: 108 for record in extracted: 109 yield from self.record_expander.expand_record(record) 110 elif extracted: 111 if self.record_expander: 112 yield from self.record_expander.expand_record(extracted) 113 else: 114 yield extracted 115 else: 116 yield from []
Selects records from the response
Parameters
- response: The response to extract the records from
Returns
List of Records extracted from the response
19class HttpMethod(Enum): 20 """ 21 Http Method to use when submitting an outgoing HTTP request 22 """ 23 24 DELETE = "DELETE" 25 GET = "GET" 26 PATCH = "PATCH" 27 POST = "POST"
Http Method to use when submitting an outgoing HTTP request
37@dataclass 38class HttpRequester(Requester): 39 """ 40 Default implementation of a Requester 41 42 Attributes: 43 name (str): Name of the stream. Only used for request/response caching 44 url_base (Union[InterpolatedString, str]): Base url to send requests to 45 path (Union[InterpolatedString, str]): Path to send requests to 46 http_method (Union[str, HttpMethod]): HTTP method to use when sending requests 47 request_options_provider (Optional[InterpolatedRequestOptionsProvider]): request option provider defining the options to set on outgoing requests 48 authenticator (DeclarativeAuthenticator): Authenticator defining how to authenticate to the source 49 error_handler (Optional[ErrorHandler]): Error handler defining how to detect and handle errors 50 backoff_strategies (Optional[List[BackoffStrategy]]): List of backoff strategies to use when retrying requests 51 config (Config): The user-provided configuration as specified by the source's spec 52 use_cache (bool): Indicates that data should be cached for this stream 53 """ 54 55 name: str 56 config: Config 57 parameters: InitVar[Mapping[str, Any]] 58 59 url: Optional[Union[InterpolatedString, str]] = None 60 url_base: Optional[Union[InterpolatedString, str]] = None 61 path: Optional[Union[InterpolatedString, str]] = None 62 authenticator: Optional[DeclarativeAuthenticator] = None 63 http_method: Union[str, HttpMethod] = HttpMethod.GET 64 request_options_provider: Optional[InterpolatedRequestOptionsProvider] = None 65 error_handler: Optional[ErrorHandler] = None 66 api_budget: Optional[APIBudget] = None 67 disable_retries: bool = False 68 message_repository: MessageRepository = NoopMessageRepository() 69 use_cache: bool = False 70 connect_timeout_in_seconds: Optional[float] = None 71 read_timeout_in_seconds: Optional[float] = None 72 use_tcp_keepalive: bool = False 73 _exit_on_rate_limit: bool = False 74 stream_response: bool = False 75 decoder: Decoder = field(default_factory=lambda: JsonDecoder(parameters={})) 76 77 def __post_init__(self, parameters: Mapping[str, Any]) -> None: 78 self._url = InterpolatedString.create( 79 self.url if self.url else EmptyString, parameters=parameters 80 ) 81 # deprecated 82 self._url_base = InterpolatedString.create( 83 self.url_base if self.url_base else EmptyString, parameters=parameters 84 ) 85 # deprecated 86 self._path = InterpolatedString.create( 87 self.path if self.path else EmptyString, parameters=parameters 88 ) 89 if self.request_options_provider is None: 90 self._request_options_provider = InterpolatedRequestOptionsProvider( 91 config=self.config, parameters=parameters 92 ) 93 elif isinstance(self.request_options_provider, dict): 94 self._request_options_provider = InterpolatedRequestOptionsProvider( 95 config=self.config, **self.request_options_provider 96 ) 97 else: 98 self._request_options_provider = self.request_options_provider 99 self._authenticator = self.authenticator or NoAuth(parameters=parameters) 100 self._http_method = ( 101 HttpMethod[self.http_method] if isinstance(self.http_method, str) else self.http_method 102 ) 103 self.error_handler = self.error_handler 104 self._parameters = parameters 105 106 if self.error_handler is not None and hasattr(self.error_handler, "backoff_strategies"): 107 backoff_strategies = self.error_handler.backoff_strategies # type: ignore 108 else: 109 backoff_strategies = None 110 111 self._http_client = HttpClient( 112 name=self.name, 113 logger=self.logger, 114 error_handler=self.error_handler, 115 api_budget=self.api_budget, 116 authenticator=self._authenticator, 117 use_cache=self.use_cache, 118 backoff_strategy=backoff_strategies, 119 disable_retries=self.disable_retries, 120 message_repository=self.message_repository, 121 request_timeout=( 122 (self.connect_timeout_in_seconds, self.read_timeout_in_seconds) 123 if self.connect_timeout_in_seconds is not None 124 or self.read_timeout_in_seconds is not None 125 else None 126 ), 127 use_tcp_keepalive=self.use_tcp_keepalive, 128 ) 129 130 @property 131 def exit_on_rate_limit(self) -> bool: 132 return self._exit_on_rate_limit 133 134 @exit_on_rate_limit.setter 135 def exit_on_rate_limit(self, value: bool) -> None: 136 self._exit_on_rate_limit = value 137 138 def get_authenticator(self) -> DeclarativeAuthenticator: 139 return self._authenticator 140 141 def get_url( 142 self, 143 *, 144 stream_state: Optional[StreamState] = None, 145 stream_slice: Optional[StreamSlice] = None, 146 next_page_token: Optional[Mapping[str, Any]] = None, 147 ) -> str: 148 interpolation_context = get_interpolation_context( 149 stream_state=stream_state, 150 stream_slice=stream_slice, 151 next_page_token=next_page_token, 152 ) 153 154 return str(self._url.eval(self.config, **interpolation_context)) 155 156 def _get_url( 157 self, 158 *, 159 path: Optional[str] = None, 160 stream_state: Optional[StreamState] = None, 161 stream_slice: Optional[StreamSlice] = None, 162 next_page_token: Optional[Mapping[str, Any]] = None, 163 ) -> str: 164 url = self.get_url( 165 stream_state=stream_state, 166 stream_slice=stream_slice, 167 next_page_token=next_page_token, 168 ) 169 170 url_base = self.get_url_base( 171 stream_state=stream_state, 172 stream_slice=stream_slice, 173 next_page_token=next_page_token, 174 ) 175 176 path = path or self.get_path( 177 stream_state=stream_state, 178 stream_slice=stream_slice, 179 next_page_token=next_page_token, 180 ) 181 182 full_url = ( 183 self._join_url(url_base, path) 184 if url_base 185 else self._join_url(url, path) 186 if path 187 else url 188 ) 189 190 return full_url 191 192 def get_url_base( 193 self, 194 *, 195 stream_state: Optional[StreamState] = None, 196 stream_slice: Optional[StreamSlice] = None, 197 next_page_token: Optional[Mapping[str, Any]] = None, 198 ) -> str: 199 interpolation_context = get_interpolation_context( 200 stream_state=stream_state, 201 stream_slice=stream_slice, 202 next_page_token=next_page_token, 203 ) 204 return str(self._url_base.eval(self.config, **interpolation_context)) 205 206 def get_path( 207 self, 208 *, 209 stream_state: Optional[StreamState] = None, 210 stream_slice: Optional[StreamSlice] = None, 211 next_page_token: Optional[Mapping[str, Any]] = None, 212 ) -> str: 213 interpolation_context = get_interpolation_context( 214 stream_state=stream_state, 215 stream_slice=stream_slice, 216 next_page_token=next_page_token, 217 ) 218 path = str(self._path.eval(self.config, **interpolation_context)) 219 return path.lstrip("/") 220 221 def get_method(self) -> HttpMethod: 222 return self._http_method 223 224 def get_request_params( 225 self, 226 *, 227 stream_state: Optional[StreamState] = None, 228 stream_slice: Optional[StreamSlice] = None, 229 next_page_token: Optional[Mapping[str, Any]] = None, 230 ) -> MutableMapping[str, Any]: 231 return self._request_options_provider.get_request_params( 232 stream_state=stream_state, 233 stream_slice=stream_slice, 234 next_page_token=next_page_token, 235 ) 236 237 def get_request_headers( 238 self, 239 *, 240 stream_state: Optional[StreamState] = None, 241 stream_slice: Optional[StreamSlice] = None, 242 next_page_token: Optional[Mapping[str, Any]] = None, 243 ) -> Mapping[str, Any]: 244 return self._request_options_provider.get_request_headers( 245 stream_state=stream_state, 246 stream_slice=stream_slice, 247 next_page_token=next_page_token, 248 ) 249 250 # fixing request options provider types has a lot of dependencies 251 def get_request_body_data( # type: ignore 252 self, 253 *, 254 stream_state: Optional[StreamState] = None, 255 stream_slice: Optional[StreamSlice] = None, 256 next_page_token: Optional[Mapping[str, Any]] = None, 257 ) -> Union[Mapping[str, Any], str]: 258 return ( 259 self._request_options_provider.get_request_body_data( 260 stream_state=stream_state, 261 stream_slice=stream_slice, 262 next_page_token=next_page_token, 263 ) 264 or {} 265 ) 266 267 # fixing request options provider types has a lot of dependencies 268 def get_request_body_json( # type: ignore 269 self, 270 *, 271 stream_state: Optional[StreamState] = None, 272 stream_slice: Optional[StreamSlice] = None, 273 next_page_token: Optional[Mapping[str, Any]] = None, 274 ) -> Optional[Mapping[str, Any]]: 275 return self._request_options_provider.get_request_body_json( 276 stream_state=stream_state, 277 stream_slice=stream_slice, 278 next_page_token=next_page_token, 279 ) 280 281 @property 282 def logger(self) -> logging.Logger: 283 return logging.getLogger(f"airbyte.HttpRequester.{self.name}") 284 285 def _get_request_options( 286 self, 287 stream_state: Optional[StreamState], 288 stream_slice: Optional[StreamSlice], 289 next_page_token: Optional[Mapping[str, Any]], 290 requester_method: Callable[..., Optional[Union[Mapping[str, Any], str]]], 291 auth_options_method: Callable[..., Optional[Union[Mapping[str, Any], str]]], 292 extra_options: Optional[Union[Mapping[str, Any], str]] = None, 293 ) -> Union[Mapping[str, Any], str]: 294 """ 295 Get the request_option from the requester, the authenticator and extra_options passed in. 296 Raise a ValueError if there's a key collision 297 Returned merged mapping otherwise 298 """ 299 300 is_body_json = requester_method.__name__ == "get_request_body_json" 301 302 return combine_mappings( 303 [ 304 requester_method( 305 stream_state=stream_state, 306 stream_slice=stream_slice, 307 next_page_token=next_page_token, 308 ), 309 auth_options_method(), 310 extra_options, 311 ], 312 allow_same_value_merge=is_body_json, 313 ) 314 315 def _request_headers( 316 self, 317 stream_state: Optional[StreamState] = None, 318 stream_slice: Optional[StreamSlice] = None, 319 next_page_token: Optional[Mapping[str, Any]] = None, 320 extra_headers: Optional[Mapping[str, Any]] = None, 321 ) -> Mapping[str, Any]: 322 """ 323 Specifies request headers. 324 Authentication headers will overwrite any overlapping headers returned from this method. 325 """ 326 headers = self._get_request_options( 327 stream_state, 328 stream_slice, 329 next_page_token, 330 self.get_request_headers, 331 self.get_authenticator().get_auth_header, 332 extra_headers, 333 ) 334 if isinstance(headers, str): 335 raise ValueError("Request headers cannot be a string") 336 return {str(k): str(v) for k, v in headers.items()} 337 338 def _request_params( 339 self, 340 stream_state: Optional[StreamState], 341 stream_slice: Optional[StreamSlice], 342 next_page_token: Optional[Mapping[str, Any]], 343 extra_params: Optional[Mapping[str, Any]] = None, 344 ) -> Mapping[str, Any]: 345 """ 346 Specifies the query parameters that should be set on an outgoing HTTP request given the inputs. 347 348 E.g: you might want to define query parameters for paging if next_page_token is not None. 349 """ 350 options = self._get_request_options( 351 stream_state, 352 stream_slice, 353 next_page_token, 354 self.get_request_params, 355 self.get_authenticator().get_request_params, 356 extra_params, 357 ) 358 if isinstance(options, str): 359 raise ValueError("Request params cannot be a string") 360 361 for k, v in options.items(): 362 if isinstance(v, (dict,)): 363 raise ValueError( 364 f"Invalid value for `{k}` parameter. The values of request params cannot be an object." 365 ) 366 367 return options 368 369 def _request_body_data( 370 self, 371 stream_state: Optional[StreamState], 372 stream_slice: Optional[StreamSlice], 373 next_page_token: Optional[Mapping[str, Any]], 374 extra_body_data: Optional[Union[Mapping[str, Any], str]] = None, 375 ) -> Optional[Union[Mapping[str, Any], str]]: 376 """ 377 Specifies how to populate the body of the request with a non-JSON payload. 378 379 If returns a ready text that it will be sent as is. 380 If returns a dict that it will be converted to a urlencoded form. 381 E.g. {"key1": "value1", "key2": "value2"} => "key1=value1&key2=value2" 382 383 At the same time only one of the 'request_body_data' and 'request_body_json' functions can be overridden. 384 """ 385 # Warning: use self.state instead of the stream_state passed as argument! 386 return self._get_request_options( 387 stream_state, 388 stream_slice, 389 next_page_token, 390 self.get_request_body_data, 391 self.get_authenticator().get_request_body_data, 392 extra_body_data, 393 ) 394 395 def _request_body_json( 396 self, 397 stream_state: Optional[StreamState], 398 stream_slice: Optional[StreamSlice], 399 next_page_token: Optional[Mapping[str, Any]], 400 extra_body_json: Optional[Mapping[str, Any]] = None, 401 ) -> Optional[Mapping[str, Any]]: 402 """ 403 Specifies how to populate the body of the request with a JSON payload. 404 405 At the same time only one of the 'request_body_data' and 'request_body_json' functions can be overridden. 406 """ 407 # Warning: use self.state instead of the stream_state passed as argument! 408 options = self._get_request_options( 409 stream_state, 410 stream_slice, 411 next_page_token, 412 self.get_request_body_json, 413 self.get_authenticator().get_request_body_json, 414 extra_body_json, 415 ) 416 if isinstance(options, str): 417 raise ValueError("Request body json cannot be a string") 418 return options 419 420 @classmethod 421 def _join_url(cls, url_base: str, path: Optional[str] = None) -> str: 422 """ 423 Joins a base URL with a given path and returns the resulting URL with any trailing slash removed. 424 425 This method ensures that there are no duplicate slashes when concatenating the base URL and the path, 426 which is useful when the full URL is provided from an interpolation context. 427 428 Args: 429 url_base (str): The base URL to which the path will be appended. 430 path (Optional[str]): The path to join with the base URL. 431 432 Returns: 433 str: The resulting joined URL. 434 435 Note: 436 Related issue: https://github.com/airbytehq/airbyte-internal-issues/issues/11869 437 - If the path is an empty string or None, the method returns the base URL with any trailing slash removed. 438 439 Example: 440 1) _join_url("https://example.com/api/", "endpoint") >> 'https://example.com/api/endpoint' 441 2) _join_url("https://example.com/api", "/endpoint") >> 'https://example.com/api/endpoint' 442 3) _join_url("https://example.com/api/", "") >> 'https://example.com/api/' 443 4) _join_url("https://example.com/api", None) >> 'https://example.com/api' 444 """ 445 446 # return a full-url if provided directly from interpolation context 447 if path == EmptyString or path is None: 448 return url_base 449 else: 450 # since we didn't provide a full-url, the url_base might not have a trailing slash 451 # so we join the url_base and path correctly 452 if not url_base.endswith("/"): 453 url_base += "/" 454 455 return urljoin(url_base, path) 456 457 def send_request( 458 self, 459 stream_state: Optional[StreamState] = None, 460 stream_slice: Optional[StreamSlice] = None, 461 next_page_token: Optional[Mapping[str, Any]] = None, 462 path: Optional[str] = None, 463 request_headers: Optional[Mapping[str, Any]] = None, 464 request_params: Optional[Mapping[str, Any]] = None, 465 request_body_data: Optional[Union[Mapping[str, Any], str]] = None, 466 request_body_json: Optional[Mapping[str, Any]] = None, 467 log_formatter: Optional[Callable[[requests.Response], Any]] = None, 468 ) -> Optional[requests.Response]: 469 request, response = self._http_client.send_request( 470 http_method=self.get_method().value, 471 url=self._get_url( 472 path=path, 473 stream_state=stream_state, 474 stream_slice=stream_slice, 475 next_page_token=next_page_token, 476 ), 477 request_kwargs={"stream": self.stream_response}, 478 headers=self._request_headers( 479 stream_state, stream_slice, next_page_token, request_headers 480 ), 481 params=self._request_params( 482 stream_state, stream_slice, next_page_token, request_params 483 ), 484 json=self._request_body_json( 485 stream_state, stream_slice, next_page_token, request_body_json 486 ), 487 data=self._request_body_data( 488 stream_state, stream_slice, next_page_token, request_body_data 489 ), 490 dedupe_query_params=True, 491 log_formatter=log_formatter, 492 exit_on_rate_limit=self._exit_on_rate_limit, 493 ) 494 495 return response
Default implementation of a Requester
Attributes:
- name (str): Name of the stream. Only used for request/response caching
- url_base (Union[InterpolatedString, str]): Base url to send requests to
- path (Union[InterpolatedString, str]): Path to send requests to
- http_method (Union[str, HttpMethod]): HTTP method to use when sending requests
- request_options_provider (Optional[InterpolatedRequestOptionsProvider]): request option provider defining the options to set on outgoing requests
- authenticator (DeclarativeAuthenticator): Authenticator defining how to authenticate to the source
- error_handler (Optional[ErrorHandler]): Error handler defining how to detect and handle errors
- backoff_strategies (Optional[List[BackoffStrategy]]): List of backoff strategies to use when retrying requests
- config (Config): The user-provided configuration as specified by the source's spec
- use_cache (bool): Indicates that data should be cached for this stream
141 def get_url( 142 self, 143 *, 144 stream_state: Optional[StreamState] = None, 145 stream_slice: Optional[StreamSlice] = None, 146 next_page_token: Optional[Mapping[str, Any]] = None, 147 ) -> str: 148 interpolation_context = get_interpolation_context( 149 stream_state=stream_state, 150 stream_slice=stream_slice, 151 next_page_token=next_page_token, 152 ) 153 154 return str(self._url.eval(self.config, **interpolation_context))
Returns
URL base for the API endpoint e.g: if you wanted to hit https://myapi.com/v1/some_entity then this should return "https://myapi.com/v1/"
192 def get_url_base( 193 self, 194 *, 195 stream_state: Optional[StreamState] = None, 196 stream_slice: Optional[StreamSlice] = None, 197 next_page_token: Optional[Mapping[str, Any]] = None, 198 ) -> str: 199 interpolation_context = get_interpolation_context( 200 stream_state=stream_state, 201 stream_slice=stream_slice, 202 next_page_token=next_page_token, 203 ) 204 return str(self._url_base.eval(self.config, **interpolation_context))
Returns
URL base for the API endpoint e.g: if you wanted to hit https://myapi.com/v1/some_entity then this should return "https://myapi.com/v1/"
206 def get_path( 207 self, 208 *, 209 stream_state: Optional[StreamState] = None, 210 stream_slice: Optional[StreamSlice] = None, 211 next_page_token: Optional[Mapping[str, Any]] = None, 212 ) -> str: 213 interpolation_context = get_interpolation_context( 214 stream_state=stream_state, 215 stream_slice=stream_slice, 216 next_page_token=next_page_token, 217 ) 218 path = str(self._path.eval(self.config, **interpolation_context)) 219 return path.lstrip("/")
Returns the URL path for the API endpoint e.g: if you wanted to hit https://myapi.com/v1/some_entity then this should return "some_entity"
224 def get_request_params( 225 self, 226 *, 227 stream_state: Optional[StreamState] = None, 228 stream_slice: Optional[StreamSlice] = None, 229 next_page_token: Optional[Mapping[str, Any]] = None, 230 ) -> MutableMapping[str, Any]: 231 return self._request_options_provider.get_request_params( 232 stream_state=stream_state, 233 stream_slice=stream_slice, 234 next_page_token=next_page_token, 235 )
Specifies the query parameters that should be set on an outgoing HTTP request given the inputs.
E.g: you might want to define query parameters for paging if next_page_token is not None.
237 def get_request_headers( 238 self, 239 *, 240 stream_state: Optional[StreamState] = None, 241 stream_slice: Optional[StreamSlice] = None, 242 next_page_token: Optional[Mapping[str, Any]] = None, 243 ) -> Mapping[str, Any]: 244 return self._request_options_provider.get_request_headers( 245 stream_state=stream_state, 246 stream_slice=stream_slice, 247 next_page_token=next_page_token, 248 )
Return any non-auth headers. Authentication headers will overwrite any overlapping headers returned from this method.
251 def get_request_body_data( # type: ignore 252 self, 253 *, 254 stream_state: Optional[StreamState] = None, 255 stream_slice: Optional[StreamSlice] = None, 256 next_page_token: Optional[Mapping[str, Any]] = None, 257 ) -> Union[Mapping[str, Any], str]: 258 return ( 259 self._request_options_provider.get_request_body_data( 260 stream_state=stream_state, 261 stream_slice=stream_slice, 262 next_page_token=next_page_token, 263 ) 264 or {} 265 )
Specifies how to populate the body of the request with a non-JSON payload.
If returns a ready text that it will be sent as is. If returns a dict that it will be converted to a urlencoded form. E.g. {"key1": "value1", "key2": "value2"} => "key1=value1&key2=value2"
At the same time only one of the 'request_body_data' and 'request_body_json' functions can be overridden.
268 def get_request_body_json( # type: ignore 269 self, 270 *, 271 stream_state: Optional[StreamState] = None, 272 stream_slice: Optional[StreamSlice] = None, 273 next_page_token: Optional[Mapping[str, Any]] = None, 274 ) -> Optional[Mapping[str, Any]]: 275 return self._request_options_provider.get_request_body_json( 276 stream_state=stream_state, 277 stream_slice=stream_slice, 278 next_page_token=next_page_token, 279 )
Specifies how to populate the body of the request with a JSON payload.
At the same time only one of the 'request_body_data' and 'request_body_json' functions can be overridden.
457 def send_request( 458 self, 459 stream_state: Optional[StreamState] = None, 460 stream_slice: Optional[StreamSlice] = None, 461 next_page_token: Optional[Mapping[str, Any]] = None, 462 path: Optional[str] = None, 463 request_headers: Optional[Mapping[str, Any]] = None, 464 request_params: Optional[Mapping[str, Any]] = None, 465 request_body_data: Optional[Union[Mapping[str, Any], str]] = None, 466 request_body_json: Optional[Mapping[str, Any]] = None, 467 log_formatter: Optional[Callable[[requests.Response], Any]] = None, 468 ) -> Optional[requests.Response]: 469 request, response = self._http_client.send_request( 470 http_method=self.get_method().value, 471 url=self._get_url( 472 path=path, 473 stream_state=stream_state, 474 stream_slice=stream_slice, 475 next_page_token=next_page_token, 476 ), 477 request_kwargs={"stream": self.stream_response}, 478 headers=self._request_headers( 479 stream_state, stream_slice, next_page_token, request_headers 480 ), 481 params=self._request_params( 482 stream_state, stream_slice, next_page_token, request_params 483 ), 484 json=self._request_body_json( 485 stream_state, stream_slice, next_page_token, request_body_json 486 ), 487 data=self._request_body_data( 488 stream_state, stream_slice, next_page_token, request_body_data 489 ), 490 dedupe_query_params=True, 491 log_formatter=log_formatter, 492 exit_on_rate_limit=self._exit_on_rate_limit, 493 ) 494 495 return response
Sends a request and returns the response. Might return no response if the error handler chooses to ignore the response or throw an exception in case of an error. If path is set, the path configured on the requester itself is ignored. If header, params and body are set, they are merged with the ones configured on the requester itself.
If a log formatter is provided, it's used to log the performed request and response. If it's not provided, no logging is performed.
30@dataclass 31class InterpolatedBoolean: 32 f""" 33 Wrapper around a string to be evaluated to a boolean value. 34 The string will be evaluated as False if it interpolates to a value in {FALSE_VALUES} 35 36 Attributes: 37 condition (str): The string representing the condition to evaluate to a boolean 38 """ 39 condition: str 40 parameters: InitVar[Mapping[str, Any]] 41 42 def __post_init__(self, parameters: Mapping[str, Any]) -> None: 43 self._default = "False" 44 self._interpolation = JinjaInterpolation() 45 self._parameters = parameters 46 47 def eval(self, config: Config, **additional_parameters: Any) -> bool: 48 """ 49 Interpolates the predicate condition string using the config and other optional arguments passed as parameter. 50 51 :param config: The user-provided configuration as specified by the source's spec 52 :param additional_parameters: Optional parameters used for interpolation 53 :return: The interpolated string 54 """ 55 if isinstance(self.condition, bool): 56 return self.condition 57 else: 58 evaluated = self._interpolation.eval( 59 self.condition, 60 config, 61 self._default, 62 parameters=self._parameters, 63 **additional_parameters, 64 ) 65 if evaluated in FALSE_VALUES: 66 return False 67 # The presence of a value is generally regarded as truthy, so we treat it as such 68 return True
47 def eval(self, config: Config, **additional_parameters: Any) -> bool: 48 """ 49 Interpolates the predicate condition string using the config and other optional arguments passed as parameter. 50 51 :param config: The user-provided configuration as specified by the source's spec 52 :param additional_parameters: Optional parameters used for interpolation 53 :return: The interpolated string 54 """ 55 if isinstance(self.condition, bool): 56 return self.condition 57 else: 58 evaluated = self._interpolation.eval( 59 self.condition, 60 config, 61 self._default, 62 parameters=self._parameters, 63 **additional_parameters, 64 ) 65 if evaluated in FALSE_VALUES: 66 return False 67 # The presence of a value is generally regarded as truthy, so we treat it as such 68 return True
Interpolates the predicate condition string using the config and other optional arguments passed as parameter.
Parameters
- config: The user-provided configuration as specified by the source's spec
- additional_parameters: Optional parameters used for interpolation
Returns
The interpolated string
15@dataclass 16class InterpolatedRequestInputProvider: 17 """ 18 Helper class that generically performs string interpolation on the provided dictionary or string input 19 """ 20 21 parameters: InitVar[Mapping[str, Any]] 22 request_inputs: Optional[Union[str, Mapping[str, str]]] = field(default=None) 23 config: Config = field(default_factory=dict) 24 _interpolator: Optional[Union[InterpolatedString, InterpolatedMapping]] = field( 25 init=False, repr=False, default=None 26 ) 27 _request_inputs: Optional[Union[str, Mapping[str, str]]] = field( 28 init=False, repr=False, default=None 29 ) 30 31 def __post_init__(self, parameters: Mapping[str, Any]) -> None: 32 self._request_inputs = self.request_inputs or {} 33 if isinstance(self._request_inputs, str): 34 self._interpolator = InterpolatedString( 35 self._request_inputs, default="", parameters=parameters 36 ) 37 else: 38 self._interpolator = InterpolatedMapping(self._request_inputs, parameters=parameters) 39 40 def eval_request_inputs( 41 self, 42 stream_slice: Optional[StreamSlice] = None, 43 next_page_token: Optional[Mapping[str, Any]] = None, 44 valid_key_types: Optional[Tuple[Type[Any]]] = None, 45 valid_value_types: Optional[Tuple[Type[Any], ...]] = None, 46 ) -> Mapping[str, Any]: 47 """ 48 Returns the request inputs to set on an outgoing HTTP request 49 50 :param stream_slice: The stream slice 51 :param next_page_token: The pagination token 52 :param valid_key_types: A tuple of types that the interpolator should allow 53 :param valid_value_types: A tuple of types that the interpolator should allow 54 :return: The request inputs to set on an outgoing HTTP request 55 """ 56 kwargs = get_interpolation_context( 57 stream_slice=stream_slice, 58 next_page_token=next_page_token, 59 ) 60 interpolated_value = self._interpolator.eval( # type: ignore # self._interpolator is always initialized with a value and will not be None 61 self.config, 62 valid_key_types=valid_key_types, 63 valid_value_types=valid_value_types, 64 **kwargs, 65 ) 66 67 if isinstance(interpolated_value, dict): 68 non_null_tokens = {k: v for k, v in interpolated_value.items() if v is not None} 69 return non_null_tokens 70 return interpolated_value # type: ignore[no-any-return]
Helper class that generically performs string interpolation on the provided dictionary or string input
40 def eval_request_inputs( 41 self, 42 stream_slice: Optional[StreamSlice] = None, 43 next_page_token: Optional[Mapping[str, Any]] = None, 44 valid_key_types: Optional[Tuple[Type[Any]]] = None, 45 valid_value_types: Optional[Tuple[Type[Any], ...]] = None, 46 ) -> Mapping[str, Any]: 47 """ 48 Returns the request inputs to set on an outgoing HTTP request 49 50 :param stream_slice: The stream slice 51 :param next_page_token: The pagination token 52 :param valid_key_types: A tuple of types that the interpolator should allow 53 :param valid_value_types: A tuple of types that the interpolator should allow 54 :return: The request inputs to set on an outgoing HTTP request 55 """ 56 kwargs = get_interpolation_context( 57 stream_slice=stream_slice, 58 next_page_token=next_page_token, 59 ) 60 interpolated_value = self._interpolator.eval( # type: ignore # self._interpolator is always initialized with a value and will not be None 61 self.config, 62 valid_key_types=valid_key_types, 63 valid_value_types=valid_value_types, 64 **kwargs, 65 ) 66 67 if isinstance(interpolated_value, dict): 68 non_null_tokens = {k: v for k, v in interpolated_value.items() if v is not None} 69 return non_null_tokens 70 return interpolated_value # type: ignore[no-any-return]
Returns the request inputs to set on an outgoing HTTP request
Parameters
- stream_slice: The stream slice
- next_page_token: The pagination token
- valid_key_types: A tuple of types that the interpolator should allow
- valid_value_types: A tuple of types that the interpolator should allow
Returns
The request inputs to set on an outgoing HTTP request
13@dataclass 14class InterpolatedString: 15 """ 16 Wrapper around a raw string to be interpolated with the Jinja2 templating engine 17 18 Attributes: 19 string (str): The string to evalute 20 default (Optional[str]): The default value to return if the evaluation returns an empty string 21 parameters (Mapping[str, Any]): Additional runtime parameters to be used for string interpolation 22 """ 23 24 string: str 25 parameters: InitVar[Mapping[str, Any]] 26 default: Optional[str] = None 27 28 def __post_init__(self, parameters: Mapping[str, Any]) -> None: 29 self.default = self.default or self.string 30 self._interpolation = JinjaInterpolation() 31 self._parameters = parameters 32 # indicates whether passed string is just a plain string, not Jinja template 33 # This allows for optimization, but we do not know it yet at this stage 34 self._is_plain_string = None 35 36 def eval(self, config: Config, **kwargs: Any) -> Any: 37 """ 38 Interpolates the input string using the config and other optional arguments passed as parameter. 39 40 :param config: The user-provided configuration as specified by the source's spec 41 :param kwargs: Optional parameters used for interpolation 42 :return: The interpolated string 43 """ 44 if self._is_plain_string: 45 return self.string 46 if self._is_plain_string is None: 47 # Let's check whether output from evaluation is the same as input. 48 # This indicates occurrence of a plain string, not a template and we can skip Jinja in subsequent runs. 49 evaluated = self._interpolation.eval( 50 self.string, config, self.default, parameters=self._parameters, **kwargs 51 ) 52 self._is_plain_string = self.string == evaluated 53 return evaluated 54 return self._interpolation.eval( 55 self.string, config, self.default, parameters=self._parameters, **kwargs 56 ) 57 58 def __eq__(self, other: Any) -> bool: 59 if not isinstance(other, InterpolatedString): 60 return False 61 return self.string == other.string and self.default == other.default 62 63 @classmethod 64 def create( 65 cls, 66 string_or_interpolated: Union["InterpolatedString", str], 67 *, 68 parameters: Mapping[str, Any], 69 ) -> "InterpolatedString": 70 """ 71 Helper function to obtain an InterpolatedString from either a raw string or an InterpolatedString. 72 73 :param string_or_interpolated: either a raw string or an InterpolatedString. 74 :param parameters: parameters propagated from parent component 75 :return: InterpolatedString representing the input string. 76 """ 77 if isinstance(string_or_interpolated, str): 78 return InterpolatedString(string=string_or_interpolated, parameters=parameters) 79 else: 80 return string_or_interpolated
Wrapper around a raw string to be interpolated with the Jinja2 templating engine
Attributes:
- string (str): The string to evalute
- default (Optional[str]): The default value to return if the evaluation returns an empty string
- parameters (Mapping[str, Any]): Additional runtime parameters to be used for string interpolation
36 def eval(self, config: Config, **kwargs: Any) -> Any: 37 """ 38 Interpolates the input string using the config and other optional arguments passed as parameter. 39 40 :param config: The user-provided configuration as specified by the source's spec 41 :param kwargs: Optional parameters used for interpolation 42 :return: The interpolated string 43 """ 44 if self._is_plain_string: 45 return self.string 46 if self._is_plain_string is None: 47 # Let's check whether output from evaluation is the same as input. 48 # This indicates occurrence of a plain string, not a template and we can skip Jinja in subsequent runs. 49 evaluated = self._interpolation.eval( 50 self.string, config, self.default, parameters=self._parameters, **kwargs 51 ) 52 self._is_plain_string = self.string == evaluated 53 return evaluated 54 return self._interpolation.eval( 55 self.string, config, self.default, parameters=self._parameters, **kwargs 56 )
Interpolates the input string using the config and other optional arguments passed as parameter.
Parameters
- config: The user-provided configuration as specified by the source's spec
- kwargs: Optional parameters used for interpolation
Returns
The interpolated string
63 @classmethod 64 def create( 65 cls, 66 string_or_interpolated: Union["InterpolatedString", str], 67 *, 68 parameters: Mapping[str, Any], 69 ) -> "InterpolatedString": 70 """ 71 Helper function to obtain an InterpolatedString from either a raw string or an InterpolatedString. 72 73 :param string_or_interpolated: either a raw string or an InterpolatedString. 74 :param parameters: parameters propagated from parent component 75 :return: InterpolatedString representing the input string. 76 """ 77 if isinstance(string_or_interpolated, str): 78 return InterpolatedString(string=string_or_interpolated, parameters=parameters) 79 else: 80 return string_or_interpolated
Helper function to obtain an InterpolatedString from either a raw string or an InterpolatedString.
Parameters
- string_or_interpolated: either a raw string or an InterpolatedString.
- parameters: parameters propagated from parent component
Returns
InterpolatedString representing the input string.
20class JsonDecoder(Decoder): 21 """ 22 Decoder strategy that returns the json-encoded content of a response, if any. 23 24 Usually, we would try to instantiate the equivalent `CompositeRawDecoder(parser=JsonParser(), stream_response=False)` but there were specific historical behaviors related to the JsonDecoder that we didn't know if we could remove like the fallback on {} in case of errors. 25 """ 26 27 def __init__(self, parameters: Mapping[str, Any]): 28 self._decoder = CompositeRawDecoder(parser=JsonParser(), stream_response=False) 29 30 def is_stream_response(self) -> bool: 31 return self._decoder.is_stream_response() 32 33 def decode( 34 self, response: requests.Response 35 ) -> Generator[MutableMapping[str, Any], None, None]: 36 """ 37 Given the response is an empty string or an emtpy list, the function will return a generator with an empty mapping. 38 """ 39 has_yielded = False 40 try: 41 for element in self._decoder.decode(response): 42 yield element 43 has_yielded = True 44 except Exception: 45 yield {} 46 47 if not has_yielded: 48 yield {}
Decoder strategy that returns the json-encoded content of a response, if any.
Usually, we would try to instantiate the equivalent CompositeRawDecoder(parser=JsonParser(), stream_response=False) but there were specific historical behaviors related to the JsonDecoder that we didn't know if we could remove like the fallback on {} in case of errors.
Set to True if you'd like to use stream=True option in http requester
33 def decode( 34 self, response: requests.Response 35 ) -> Generator[MutableMapping[str, Any], None, None]: 36 """ 37 Given the response is an empty string or an emtpy list, the function will return a generator with an empty mapping. 38 """ 39 has_yielded = False 40 try: 41 for element in self._decoder.decode(response): 42 yield element 43 has_yielded = True 44 except Exception: 45 yield {} 46 47 if not has_yielded: 48 yield {}
Given the response is an empty string or an emtpy list, the function will return a generator with an empty mapping.
33@dataclass 34class JsonFileSchemaLoader(ResourceSchemaLoader, SchemaLoader): 35 """ 36 Loads the schema from a json file 37 38 Attributes: 39 file_path (Union[InterpolatedString, str]): The path to the json file describing the schema 40 name (str): The stream's name 41 config (Config): The user-provided configuration as specified by the source's spec 42 parameters (Mapping[str, Any]): Additional arguments to pass to the string interpolation if needed 43 """ 44 45 config: Config 46 parameters: InitVar[Mapping[str, Any]] 47 file_path: Union[InterpolatedString, str] = field(default="") 48 49 def __post_init__(self, parameters: Mapping[str, Any]) -> None: 50 if not self.file_path: 51 self.file_path = _default_file_path() 52 self.file_path = InterpolatedString.create(self.file_path, parameters=parameters) 53 54 def get_json_schema(self) -> Mapping[str, Any]: 55 # todo: It is worth revisiting if we can replace file_path with just file_name if every schema is in the /schemas directory 56 # this would require that we find a creative solution to store or retrieve source_name in here since the files are mounted there 57 json_schema_path = self._get_json_filepath() 58 resource, schema_path = self.extract_resource_and_schema_path(json_schema_path) 59 raw_json_file = pkgutil.get_data(resource, schema_path) 60 61 if not raw_json_file: 62 raise IOError(f"Cannot find file {json_schema_path}") 63 try: 64 raw_schema = json.loads(raw_json_file) 65 except ValueError as err: 66 raise RuntimeError(f"Invalid JSON file format for file {json_schema_path}") from err 67 self.package_name = resource 68 return self._resolve_schema_references(raw_schema) 69 70 def _get_json_filepath(self) -> Any: 71 return self.file_path.eval(self.config) # type: ignore # file_path is always cast to an interpolated string 72 73 @staticmethod 74 def extract_resource_and_schema_path(json_schema_path: str) -> Tuple[str, str]: 75 """ 76 When the connector is running on a docker container, package_data is accessible from the resource (source_<name>), so we extract 77 the resource from the first part of the schema path and the remaining path is used to find the schema file. This is a slight 78 hack to identify the source name while we are in the airbyte_cdk module. 79 :param json_schema_path: The path to the schema JSON file 80 :return: Tuple of the resource name and the path to the schema file 81 """ 82 split_path = json_schema_path.split("/") 83 84 if split_path[0] == "" or split_path[0] == ".": 85 split_path = split_path[1:] 86 87 if len(split_path) == 0: 88 return "", "" 89 90 if len(split_path) == 1: 91 return "", split_path[0] 92 93 return split_path[0], "/".join(split_path[1:])
Loads the schema from a json file
Attributes:
- file_path (Union[InterpolatedString, str]): The path to the json file describing the schema
- name (str): The stream's name
- config (Config): The user-provided configuration as specified by the source's spec
- parameters (Mapping[str, Any]): Additional arguments to pass to the string interpolation if needed
54 def get_json_schema(self) -> Mapping[str, Any]: 55 # todo: It is worth revisiting if we can replace file_path with just file_name if every schema is in the /schemas directory 56 # this would require that we find a creative solution to store or retrieve source_name in here since the files are mounted there 57 json_schema_path = self._get_json_filepath() 58 resource, schema_path = self.extract_resource_and_schema_path(json_schema_path) 59 raw_json_file = pkgutil.get_data(resource, schema_path) 60 61 if not raw_json_file: 62 raise IOError(f"Cannot find file {json_schema_path}") 63 try: 64 raw_schema = json.loads(raw_json_file) 65 except ValueError as err: 66 raise RuntimeError(f"Invalid JSON file format for file {json_schema_path}") from err 67 self.package_name = resource 68 return self._resolve_schema_references(raw_schema)
Returns a mapping describing the stream's schema
73 @staticmethod 74 def extract_resource_and_schema_path(json_schema_path: str) -> Tuple[str, str]: 75 """ 76 When the connector is running on a docker container, package_data is accessible from the resource (source_<name>), so we extract 77 the resource from the first part of the schema path and the remaining path is used to find the schema file. This is a slight 78 hack to identify the source name while we are in the airbyte_cdk module. 79 :param json_schema_path: The path to the schema JSON file 80 :return: Tuple of the resource name and the path to the schema file 81 """ 82 split_path = json_schema_path.split("/") 83 84 if split_path[0] == "" or split_path[0] == ".": 85 split_path = split_path[1:] 86 87 if len(split_path) == 0: 88 return "", "" 89 90 if len(split_path) == 1: 91 return "", split_path[0] 92 93 return split_path[0], "/".join(split_path[1:])
When the connector is running on a docker container, package_data is accessible from the resource (source_
Parameters
- json_schema_path: The path to the schema JSON file
Returns
Tuple of the resource name and the path to the schema file
Inherited Members
21class LegacyToPerPartitionStateMigration(StateMigration): 22 """ 23 Transforms the input state for per-partitioned streams from the legacy format to the low-code format. 24 The cursor field and partition ID fields are automatically extracted from the stream's DatetimebasedCursor and SubstreamPartitionRouter. 25 26 Example input state: 27 { 28 "13506132": { 29 "last_changed": "2022-12-27T08:34:39+00:00" 30 } 31 Example output state: 32 { 33 "partition": {"id": "13506132"}, 34 "cursor": {"last_changed": "2022-12-27T08:34:39+00:00"} 35 } 36 """ 37 38 def __init__( 39 self, 40 partition_router: Union[ 41 SubstreamPartitionRouter, UnionPartitionRouter, CustomPartitionRouter 42 ], 43 cursor: DatetimeBasedCursor, 44 config: Mapping[str, Any], 45 parameters: Mapping[str, Any], 46 ): 47 self._partition_router = partition_router 48 self._cursor = cursor 49 self._config = config 50 self._parameters = parameters 51 self._partition_key_field = InterpolatedString.create( 52 self._get_partition_field(partition_router), parameters=self._parameters 53 ).eval(self._config) 54 self._cursor_field = InterpolatedString.create( 55 self._cursor.cursor_field, parameters=self._parameters 56 ).eval(self._config) 57 58 def _get_partition_field( 59 self, 60 partition_router: Union[ 61 SubstreamPartitionRouter, UnionPartitionRouter, CustomPartitionRouter 62 ], 63 ) -> str: 64 if isinstance(partition_router, UnionPartitionRouter): 65 return partition_router.partition_field 66 67 parent_stream_config = partition_router.parent_stream_configs[0] # type: ignore # custom partition routers are expected to expose parent_stream_configs 68 69 # Retrieve the partition field with a condition, as properties are returned as a dictionary for custom components. 70 partition_field = ( 71 parent_stream_config.partition_field 72 if isinstance(parent_stream_config, ParentStreamConfig) 73 else parent_stream_config.get("partition_field") # type: ignore # See above comment on why parent_stream_config might be a dict 74 ) 75 76 return partition_field 77 78 def should_migrate(self, stream_state: Mapping[str, Any]) -> bool: 79 if _is_already_migrated(stream_state): 80 return False 81 82 # UnionPartitionRouter has no parent_stream_configs; its partitions are already 83 # normalized to a single partition field so the parent stream check does not apply. 84 if not isinstance(self._partition_router, UnionPartitionRouter): 85 # There is exactly one parent stream 86 number_of_parent_streams = len(self._partition_router.parent_stream_configs) # type: ignore # custom partition will introduce this attribute if needed 87 if number_of_parent_streams != 1: 88 # There should be exactly one parent stream 89 return False 90 """ 91 The expected state format is 92 "<parent_key_id>" : { 93 "<cursor_field>" : "<cursor_value>" 94 } 95 """ 96 if not stream_state: 97 return False 98 for key, value in stream_state.items(): 99 # it is expected the internal value to be a dictionary according to docstring 100 if not isinstance(value, dict): 101 return False 102 keys = list(value.keys()) 103 if len(keys) != 1: 104 # The input partitioned state should only have one key 105 return False 106 if keys[0] != self._cursor_field: 107 # Unexpected key. Found {keys[0]}. Expected {self._cursor.cursor_field} 108 return False 109 110 return True 111 112 def migrate(self, stream_state: Mapping[str, Any]) -> Mapping[str, Any]: 113 states = [ 114 {"partition": {self._partition_key_field: key}, "cursor": value} 115 for key, value in stream_state.items() 116 ] 117 return {"states": states}
Transforms the input state for per-partitioned streams from the legacy format to the low-code format. The cursor field and partition ID fields are automatically extracted from the stream's DatetimebasedCursor and SubstreamPartitionRouter.
Example input state: { "13506132": { "last_changed": "2022-12-27T08:34:39+00:00" } Example output state: { "partition": {"id": "13506132"}, "cursor": {"last_changed": "2022-12-27T08:34:39+00:00"} }
38 def __init__( 39 self, 40 partition_router: Union[ 41 SubstreamPartitionRouter, UnionPartitionRouter, CustomPartitionRouter 42 ], 43 cursor: DatetimeBasedCursor, 44 config: Mapping[str, Any], 45 parameters: Mapping[str, Any], 46 ): 47 self._partition_router = partition_router 48 self._cursor = cursor 49 self._config = config 50 self._parameters = parameters 51 self._partition_key_field = InterpolatedString.create( 52 self._get_partition_field(partition_router), parameters=self._parameters 53 ).eval(self._config) 54 self._cursor_field = InterpolatedString.create( 55 self._cursor.cursor_field, parameters=self._parameters 56 ).eval(self._config)
78 def should_migrate(self, stream_state: Mapping[str, Any]) -> bool: 79 if _is_already_migrated(stream_state): 80 return False 81 82 # UnionPartitionRouter has no parent_stream_configs; its partitions are already 83 # normalized to a single partition field so the parent stream check does not apply. 84 if not isinstance(self._partition_router, UnionPartitionRouter): 85 # There is exactly one parent stream 86 number_of_parent_streams = len(self._partition_router.parent_stream_configs) # type: ignore # custom partition will introduce this attribute if needed 87 if number_of_parent_streams != 1: 88 # There should be exactly one parent stream 89 return False 90 """ 91 The expected state format is 92 "<parent_key_id>" : { 93 "<cursor_field>" : "<cursor_value>" 94 } 95 """ 96 if not stream_state: 97 return False 98 for key, value in stream_state.items(): 99 # it is expected the internal value to be a dictionary according to docstring 100 if not isinstance(value, dict): 101 return False 102 keys = list(value.keys()) 103 if len(keys) != 1: 104 # The input partitioned state should only have one key 105 return False 106 if keys[0] != self._cursor_field: 107 # Unexpected key. Found {keys[0]}. Expected {self._cursor.cursor_field} 108 return False 109 110 return True
Check if the stream_state should be migrated
Parameters
- stream_state: The stream_state to potentially migrate
Returns
true if the state is of the expected format and should be migrated. False otherwise.
112 def migrate(self, stream_state: Mapping[str, Any]) -> Mapping[str, Any]: 113 states = [ 114 {"partition": {self._partition_key_field: key}, "cursor": value} 115 for key, value in stream_state.items() 116 ] 117 return {"states": states}
Migrate the stream_state. Assumes should_migrate(stream_state) returned True.
Parameters
- stream_state: The stream_state to migrate
Returns
The migrated stream_state
14@dataclass 15class MinMaxDatetime: 16 """ 17 Compares the provided date against optional minimum or maximum times. If date is earlier than 18 min_date, then min_date is returned. If date is greater than max_date, then max_date is returned. 19 If neither, the input date is returned. 20 21 The timestamp format accepts the same format codes as datetime.strfptime, which are 22 all the format codes required by the 1989 C standard. 23 Full list of accepted format codes: https://man7.org/linux/man-pages/man3/strftime.3.html 24 25 Attributes: 26 datetime (Union[InterpolatedString, str]): InterpolatedString or string representing the datetime in the format specified by `datetime_format` 27 datetime_format (str): Format of the datetime passed as argument 28 min_datetime (Union[InterpolatedString, str]): Represents the minimum allowed datetime value. 29 max_datetime (Union[InterpolatedString, str]): Represents the maximum allowed datetime value. 30 """ 31 32 datetime: Union[InterpolatedString, str] 33 parameters: InitVar[Mapping[str, Any]] 34 # datetime_format is a unique case where we inherit it from the parent if it is not specified before using the default value 35 # which is why we need dedicated getter/setter methods and private dataclass field 36 datetime_format: str 37 _datetime_format: str = field(init=False, repr=False, default="") 38 min_datetime: Union[InterpolatedString, str] = "" 39 max_datetime: Union[InterpolatedString, str] = "" 40 41 def __post_init__(self, parameters: Mapping[str, Any]) -> None: 42 self.datetime = InterpolatedString.create(self.datetime, parameters=parameters or {}) 43 self._parser = DatetimeParser() 44 self.min_datetime = ( 45 InterpolatedString.create(self.min_datetime, parameters=parameters) # type: ignore [assignment] # expression has type "InterpolatedString | None", variable has type "InterpolatedString | str" 46 if self.min_datetime 47 else None 48 ) # type: ignore 49 self.max_datetime = ( 50 InterpolatedString.create(self.max_datetime, parameters=parameters) # type: ignore [assignment] # expression has type "InterpolatedString | None", variable has type "InterpolatedString | str" 51 if self.max_datetime 52 else None 53 ) # type: ignore 54 55 def get_datetime( 56 self, config: Mapping[str, Any], **additional_parameters: Mapping[str, Any] 57 ) -> dt.datetime: 58 """ 59 Evaluates and returns the datetime 60 :param config: The user-provided configuration as specified by the source's spec 61 :param additional_parameters: Additional arguments to be passed to the strings for interpolation 62 :return: The evaluated datetime 63 """ 64 # We apply a default datetime format here instead of at instantiation, so it can be set by the parent first 65 datetime_format = self._datetime_format 66 if not datetime_format: 67 datetime_format = "%Y-%m-%dT%H:%M:%S.%f%z" 68 69 time = self._parser.parse( 70 str( 71 self.datetime.eval( # type: ignore[union-attr] # str has no attribute "eval" 72 config, 73 **additional_parameters, 74 ) 75 ), 76 datetime_format, 77 ) # type: ignore # datetime is always cast to an interpolated string 78 79 if self.min_datetime: 80 min_time = str(self.min_datetime.eval(config, **additional_parameters)) # type: ignore # min_datetime is always cast to an interpolated string 81 if min_time: 82 min_datetime = self._parser.parse(min_time, datetime_format) # type: ignore # min_datetime is always cast to an interpolated string 83 time = max(time, min_datetime) 84 if self.max_datetime: 85 max_time = str(self.max_datetime.eval(config, **additional_parameters)) # type: ignore # max_datetime is always cast to an interpolated string 86 if max_time: 87 max_datetime = self._parser.parse(max_time, datetime_format) 88 time = min(time, max_datetime) 89 return time 90 91 @property # type: ignore # properties don't play well with dataclasses... 92 def datetime_format(self) -> str: 93 """The format of the string representing the datetime""" 94 return self._datetime_format 95 96 @datetime_format.setter 97 def datetime_format(self, value: str) -> None: 98 """Setter for the datetime format""" 99 # Covers the case where datetime_format is not provided in the constructor, which causes the property object 100 # to be set which we need to avoid doing 101 if not isinstance(value, property): 102 self._datetime_format = value 103 104 @classmethod 105 def create( 106 cls, 107 interpolated_string_or_min_max_datetime: Union[InterpolatedString, str, "MinMaxDatetime"], 108 parameters: Optional[Mapping[str, Any]] = None, 109 ) -> "MinMaxDatetime": 110 if parameters is None: 111 parameters = {} 112 if isinstance(interpolated_string_or_min_max_datetime, InterpolatedString) or isinstance( 113 interpolated_string_or_min_max_datetime, str 114 ): 115 return MinMaxDatetime( # type: ignore [call-arg] 116 datetime=interpolated_string_or_min_max_datetime, parameters=parameters 117 ) 118 else: 119 return interpolated_string_or_min_max_datetime
Compares the provided date against optional minimum or maximum times. If date is earlier than min_date, then min_date is returned. If date is greater than max_date, then max_date is returned. If neither, the input date is returned.
The timestamp format accepts the same format codes as datetime.strfptime, which are all the format codes required by the 1989 C standard. Full list of accepted format codes: https://man7.org/linux/man-pages/man3/strftime.3.html
Attributes:
- datetime (Union[InterpolatedString, str]): InterpolatedString or string representing the datetime in the format specified by
datetime_format - datetime_format (str): Format of the datetime passed as argument
- min_datetime (Union[InterpolatedString, str]): Represents the minimum allowed datetime value.
- max_datetime (Union[InterpolatedString, str]): Represents the maximum allowed datetime value.
91 @property # type: ignore # properties don't play well with dataclasses... 92 def datetime_format(self) -> str: 93 """The format of the string representing the datetime""" 94 return self._datetime_format
The format of the string representing the datetime
55 def get_datetime( 56 self, config: Mapping[str, Any], **additional_parameters: Mapping[str, Any] 57 ) -> dt.datetime: 58 """ 59 Evaluates and returns the datetime 60 :param config: The user-provided configuration as specified by the source's spec 61 :param additional_parameters: Additional arguments to be passed to the strings for interpolation 62 :return: The evaluated datetime 63 """ 64 # We apply a default datetime format here instead of at instantiation, so it can be set by the parent first 65 datetime_format = self._datetime_format 66 if not datetime_format: 67 datetime_format = "%Y-%m-%dT%H:%M:%S.%f%z" 68 69 time = self._parser.parse( 70 str( 71 self.datetime.eval( # type: ignore[union-attr] # str has no attribute "eval" 72 config, 73 **additional_parameters, 74 ) 75 ), 76 datetime_format, 77 ) # type: ignore # datetime is always cast to an interpolated string 78 79 if self.min_datetime: 80 min_time = str(self.min_datetime.eval(config, **additional_parameters)) # type: ignore # min_datetime is always cast to an interpolated string 81 if min_time: 82 min_datetime = self._parser.parse(min_time, datetime_format) # type: ignore # min_datetime is always cast to an interpolated string 83 time = max(time, min_datetime) 84 if self.max_datetime: 85 max_time = str(self.max_datetime.eval(config, **additional_parameters)) # type: ignore # max_datetime is always cast to an interpolated string 86 if max_time: 87 max_datetime = self._parser.parse(max_time, datetime_format) 88 time = min(time, max_datetime) 89 return time
Evaluates and returns the datetime
Parameters
- config: The user-provided configuration as specified by the source's spec
- additional_parameters: Additional arguments to be passed to the strings for interpolation
Returns
The evaluated datetime
104 @classmethod 105 def create( 106 cls, 107 interpolated_string_or_min_max_datetime: Union[InterpolatedString, str, "MinMaxDatetime"], 108 parameters: Optional[Mapping[str, Any]] = None, 109 ) -> "MinMaxDatetime": 110 if parameters is None: 111 parameters = {} 112 if isinstance(interpolated_string_or_min_max_datetime, InterpolatedString) or isinstance( 113 interpolated_string_or_min_max_datetime, str 114 ): 115 return MinMaxDatetime( # type: ignore [call-arg] 116 datetime=interpolated_string_or_min_max_datetime, parameters=parameters 117 ) 118 else: 119 return interpolated_string_or_min_max_datetime
33@dataclass 34class NoAuth(DeclarativeAuthenticator): 35 parameters: InitVar[Mapping[str, Any]] 36 37 @property 38 def auth_header(self) -> str: 39 return "" 40 41 @property 42 def token(self) -> str: 43 return ""
25@dataclass 26class OffsetIncrement(PaginationStrategy): 27 """ 28 Pagination strategy that returns the number of records reads so far and returns it as the next page token 29 Examples: 30 # page_size to be a constant integer value 31 pagination_strategy: 32 type: OffsetIncrement 33 page_size: 2 34 35 # page_size to be a constant string value 36 pagination_strategy: 37 type: OffsetIncrement 38 page_size: "2" 39 40 # page_size to be an interpolated string value 41 pagination_strategy: 42 type: OffsetIncrement 43 page_size: "{{ parameters['items_per_page'] }}" 44 45 Attributes: 46 page_size (InterpolatedString): the number of records to request 47 extractor (Optional[RecordExtractor]): counts the records of a page by extracting them 48 again. Leave it unset to use the count of the `RecordSelector` that read the page 49 """ 50 51 config: Config 52 page_size: Optional[Union[str, int]] 53 parameters: InitVar[Mapping[str, Any]] 54 extractor: Optional[RecordExtractor] = None 55 decoder: Decoder = field( 56 default_factory=lambda: PaginationDecoderDecorator(decoder=JsonDecoder(parameters={})) 57 ) 58 inject_on_first_request: bool = False 59 60 def __post_init__(self, parameters: Mapping[str, Any]) -> None: 61 page_size = str(self.page_size) if isinstance(self.page_size, int) else self.page_size 62 if page_size: 63 self._page_size: Optional[InterpolatedString] = InterpolatedString( 64 page_size, parameters=parameters 65 ) 66 else: 67 self._page_size = None 68 69 @property 70 def initial_token(self) -> Optional[Any]: 71 if self.inject_on_first_request: 72 return 0 73 return None 74 75 def next_page_token( 76 self, 77 response: requests.Response, 78 last_page_size: int, 79 last_record: Optional[Record], 80 last_page_token_value: Optional[Any] = None, 81 page_size_override: Optional[int] = None, 82 stream_slice: Optional[StreamSlice] = None, 83 ) -> Optional[Any]: 84 # The records the API returned, not the ones emitted: a record filter must neither end the 85 # pagination nor shorten the offset. 86 if self.extractor: 87 last_page_size = len(list(self.extractor.extract_records(response=response))) 88 else: 89 last_page_size = extracted_record_count(response, default=last_page_size) 90 91 # Stop paginating when there are fewer records than the page size or the current page has no records. 92 # The comparison uses the page size that was actually requested: after a `REDUCE_PAGE_SIZE` reduction, a 93 # full page is smaller than the configured page size and comparing against the latter would end the 94 # pagination early and skip records. 95 if page_size_override is not None: 96 requested_page_size: Optional[int] = page_size_override 97 elif isinstance(self.page_size, int): 98 # A constant needs no response, which would otherwise be decoded a second time. 99 requested_page_size = self.page_size 100 elif self._page_size: 101 requested_page_size = self._page_size.eval( 102 self.config, response=next(self.decoder.decode(response)) 103 ) 104 else: 105 requested_page_size = None 106 if (requested_page_size and last_page_size < requested_page_size) or last_page_size == 0: 107 return None 108 elif last_page_token_value is None: 109 # If the OffsetIncrement strategy does not inject on the first request, the incoming last_page_token_value 110 # will be None. For this case, we assume that None was the first page and progress to the next offset 111 return 0 + last_page_size 112 elif not isinstance(last_page_token_value, int): 113 raise ValueError( 114 f"Last page token value {last_page_token_value} for OffsetIncrement pagination strategy was not an integer" 115 ) 116 else: 117 return last_page_token_value + last_page_size 118 119 def get_page_size(self) -> Optional[int]: 120 if self._page_size: 121 page_size = self._page_size.eval(self.config) 122 if not isinstance(page_size, int): 123 raise Exception(f"{page_size} is of type {type(page_size)}. Expected {int}") 124 return page_size 125 else: 126 return None
Pagination strategy that returns the number of records reads so far and returns it as the next page token
Examples:
page_size to be a constant integer value
pagination_strategy: type: OffsetIncrement page_size: 2
page_size to be a constant string value
pagination_strategy: type: OffsetIncrement page_size: "2"
page_size to be an interpolated string value
pagination_strategy: type: OffsetIncrement page_size: "{{ parameters['items_per_page'] }}"
Attributes:
- page_size (InterpolatedString): the number of records to request
- extractor (Optional[RecordExtractor]): counts the records of a page by extracting them
again. Leave it unset to use the count of the
RecordSelectorthat read the page
69 @property 70 def initial_token(self) -> Optional[Any]: 71 if self.inject_on_first_request: 72 return 0 73 return None
Return the initial value of the token
75 def next_page_token( 76 self, 77 response: requests.Response, 78 last_page_size: int, 79 last_record: Optional[Record], 80 last_page_token_value: Optional[Any] = None, 81 page_size_override: Optional[int] = None, 82 stream_slice: Optional[StreamSlice] = None, 83 ) -> Optional[Any]: 84 # The records the API returned, not the ones emitted: a record filter must neither end the 85 # pagination nor shorten the offset. 86 if self.extractor: 87 last_page_size = len(list(self.extractor.extract_records(response=response))) 88 else: 89 last_page_size = extracted_record_count(response, default=last_page_size) 90 91 # Stop paginating when there are fewer records than the page size or the current page has no records. 92 # The comparison uses the page size that was actually requested: after a `REDUCE_PAGE_SIZE` reduction, a 93 # full page is smaller than the configured page size and comparing against the latter would end the 94 # pagination early and skip records. 95 if page_size_override is not None: 96 requested_page_size: Optional[int] = page_size_override 97 elif isinstance(self.page_size, int): 98 # A constant needs no response, which would otherwise be decoded a second time. 99 requested_page_size = self.page_size 100 elif self._page_size: 101 requested_page_size = self._page_size.eval( 102 self.config, response=next(self.decoder.decode(response)) 103 ) 104 else: 105 requested_page_size = None 106 if (requested_page_size and last_page_size < requested_page_size) or last_page_size == 0: 107 return None 108 elif last_page_token_value is None: 109 # If the OffsetIncrement strategy does not inject on the first request, the incoming last_page_token_value 110 # will be None. For this case, we assume that None was the first page and progress to the next offset 111 return 0 + last_page_size 112 elif not isinstance(last_page_token_value, int): 113 raise ValueError( 114 f"Last page token value {last_page_token_value} for OffsetIncrement pagination strategy was not an integer" 115 ) 116 else: 117 return last_page_token_value + last_page_size
Parameters
- response: response to process
- last_page_size: the number of records read from the response
- last_record: the last record extracted from the response
- last_page_token_value: The current value of the page token made on the last request
- page_size_override: the page size that was actually requested, when it differs from the configured
one because of a
REDUCE_PAGE_SIZEresponse action. Strategies that use their configured page size as a stop condition must honor this value, else they end the pagination early and skip records. - stream_slice: the slice the page was read for. Only passed to strategies that declare it, so existing strategies defined outside of the CDK keep working unchanged.
Returns
next page token. Returns None if there are no more pages to fetch
119 def get_page_size(self) -> Optional[int]: 120 if self._page_size: 121 page_size = self._page_size.eval(self.config) 122 if not isinstance(page_size, int): 123 raise Exception(f"{page_size} is of type {type(page_size)}. Expected {int}") 124 return page_size 125 else: 126 return None
Returns
page size: The number of records to fetch in a page. Returns None if unspecified
22@dataclass 23class PageIncrement(PaginationStrategy): 24 """ 25 Pagination strategy that returns the number of pages reads so far and returns it as the next page token 26 27 Attributes: 28 page_size (int): the number of records to request 29 start_from_page (int): number of the initial page 30 extractor (Optional[RecordExtractor]): counts the records of a page by extracting them 31 again. Leave it unset to use the count of the `RecordSelector` that read the page 32 """ 33 34 config: Config 35 page_size: Optional[Union[str, int]] 36 parameters: InitVar[Mapping[str, Any]] 37 start_from_page: int = 0 38 inject_on_first_request: bool = False 39 extractor: Optional[RecordExtractor] = None 40 41 def __post_init__(self, parameters: Mapping[str, Any]) -> None: 42 if isinstance(self.page_size, int) or (self.page_size is None): 43 self._page_size = self.page_size 44 else: 45 page_size = InterpolatedString(self.page_size, parameters=parameters).eval(self.config) 46 if not isinstance(page_size, int): 47 raise Exception(f"{page_size} is of type {type(page_size)}. Expected {int}") 48 self._page_size = page_size 49 50 @property 51 def initial_token(self) -> Optional[Any]: 52 if self.inject_on_first_request: 53 return self.start_from_page 54 return None 55 56 def next_page_token( 57 self, 58 response: requests.Response, 59 last_page_size: int, 60 last_record: Optional[Record], 61 last_page_token_value: Optional[Any], 62 page_size_override: Optional[int] = None, 63 stream_slice: Optional[StreamSlice] = None, 64 ) -> Optional[Any]: 65 if page_size_override is not None: 66 # Reachable only when the factory is bypassed: a manifest naming PageIncrement, and a 67 # CustomPaginationStrategy that subclasses it, are both rejected at config time. An unrecognized 68 # ValueError here would be reported as a generic system error even though the message describes a 69 # configuration mistake. `next_page_token` runs after the page's records have been emitted, so an 70 # out-of-tree caller gets this mid-stream rather than at startup - still better than silently 71 # ignoring the override and skipping records. 72 raise AirbyteTracedException( 73 internal_message="PageIncrement received a page_size_override", 74 message="PageIncrement does not support reducing the page size while paginating: pages are " 75 "addressed as page number * page size, so a smaller page size shifts every following page " 76 "boundary and would skip records. Use OffsetIncrement or CursorPagination instead.", 77 failure_type=FailureType.config_error, 78 ) 79 80 # The record count is dependent on the records returned from the response which may not always 81 # align with the size of pages emitted. For example, a record filter can reduce the number of 82 # records observed below the page size even though the API returned a full page. 83 if self.extractor: 84 last_page_size = len(list(self.extractor.extract_records(response=response))) 85 else: 86 last_page_size = extracted_record_count(response, default=last_page_size) 87 88 # Stop paginating when there are fewer records than the page size or the current page has no records 89 if (self._page_size and last_page_size < self._page_size) or last_page_size == 0: 90 return None 91 elif last_page_token_value is None: 92 # If the PageIncrement strategy does not inject on the first request, the incoming last_page_token_value 93 # may be None. When this is the case, we assume we've already requested the first page specified by 94 # start_from_page and must now get the next page 95 return self.start_from_page + 1 96 elif not isinstance(last_page_token_value, int): 97 raise ValueError( 98 f"Last page token value {last_page_token_value} for PageIncrement pagination strategy was not an integer" 99 ) 100 else: 101 return last_page_token_value + 1 102 103 def get_page_size(self) -> Optional[int]: 104 return self._page_size
Pagination strategy that returns the number of pages reads so far and returns it as the next page token
Attributes:
- page_size (int): the number of records to request
- start_from_page (int): number of the initial page
- extractor (Optional[RecordExtractor]): counts the records of a page by extracting them
again. Leave it unset to use the count of the
RecordSelectorthat read the page
50 @property 51 def initial_token(self) -> Optional[Any]: 52 if self.inject_on_first_request: 53 return self.start_from_page 54 return None
Return the initial value of the token
56 def next_page_token( 57 self, 58 response: requests.Response, 59 last_page_size: int, 60 last_record: Optional[Record], 61 last_page_token_value: Optional[Any], 62 page_size_override: Optional[int] = None, 63 stream_slice: Optional[StreamSlice] = None, 64 ) -> Optional[Any]: 65 if page_size_override is not None: 66 # Reachable only when the factory is bypassed: a manifest naming PageIncrement, and a 67 # CustomPaginationStrategy that subclasses it, are both rejected at config time. An unrecognized 68 # ValueError here would be reported as a generic system error even though the message describes a 69 # configuration mistake. `next_page_token` runs after the page's records have been emitted, so an 70 # out-of-tree caller gets this mid-stream rather than at startup - still better than silently 71 # ignoring the override and skipping records. 72 raise AirbyteTracedException( 73 internal_message="PageIncrement received a page_size_override", 74 message="PageIncrement does not support reducing the page size while paginating: pages are " 75 "addressed as page number * page size, so a smaller page size shifts every following page " 76 "boundary and would skip records. Use OffsetIncrement or CursorPagination instead.", 77 failure_type=FailureType.config_error, 78 ) 79 80 # The record count is dependent on the records returned from the response which may not always 81 # align with the size of pages emitted. For example, a record filter can reduce the number of 82 # records observed below the page size even though the API returned a full page. 83 if self.extractor: 84 last_page_size = len(list(self.extractor.extract_records(response=response))) 85 else: 86 last_page_size = extracted_record_count(response, default=last_page_size) 87 88 # Stop paginating when there are fewer records than the page size or the current page has no records 89 if (self._page_size and last_page_size < self._page_size) or last_page_size == 0: 90 return None 91 elif last_page_token_value is None: 92 # If the PageIncrement strategy does not inject on the first request, the incoming last_page_token_value 93 # may be None. When this is the case, we assume we've already requested the first page specified by 94 # start_from_page and must now get the next page 95 return self.start_from_page + 1 96 elif not isinstance(last_page_token_value, int): 97 raise ValueError( 98 f"Last page token value {last_page_token_value} for PageIncrement pagination strategy was not an integer" 99 ) 100 else: 101 return last_page_token_value + 1
Parameters
- response: response to process
- last_page_size: the number of records read from the response
- last_record: the last record extracted from the response
- last_page_token_value: The current value of the page token made on the last request
- page_size_override: the page size that was actually requested, when it differs from the configured
one because of a
REDUCE_PAGE_SIZEresponse action. Strategies that use their configured page size as a stop condition must honor this value, else they end the pagination early and skip records. - stream_slice: the slice the page was read for. Only passed to strategies that declare it, so existing strategies defined outside of the CDK keep working unchanged.
Returns
next page token. Returns None if there are no more pages to fetch
15@dataclass 16class PaginationStrategy: 17 """ 18 Defines how to get the next page token 19 20 One instance paginates every partition of the stream, several at once, so `next_page_token` must derive the 21 next token from its arguments, such as `last_page_token_value`, rather than from state kept on the instance. 22 """ 23 24 @property 25 @abstractmethod 26 def initial_token(self) -> Optional[Any]: 27 """ 28 Return the initial value of the token 29 """ 30 31 @abstractmethod 32 def next_page_token( 33 self, 34 response: requests.Response, 35 last_page_size: int, 36 last_record: Optional[Record], 37 last_page_token_value: Optional[Any], 38 page_size_override: Optional[int] = None, 39 stream_slice: Optional[StreamSlice] = None, 40 ) -> Optional[Any]: 41 """ 42 :param response: response to process 43 :param last_page_size: the number of records read from the response 44 :param last_record: the last record extracted from the response 45 :param last_page_token_value: The current value of the page token made on the last request 46 :param page_size_override: the page size that was actually requested, when it differs from the configured 47 one because of a `REDUCE_PAGE_SIZE` response action. Strategies that use their configured page size as 48 a stop condition must honor this value, else they end the pagination early and skip records. 49 :param stream_slice: the slice the page was read for. Only passed to strategies that declare it, so existing 50 strategies defined outside of the CDK keep working unchanged. 51 :return: next page token. Returns None if there are no more pages to fetch 52 """ 53 pass 54 55 @abstractmethod 56 def get_page_size(self) -> Optional[int]: 57 """ 58 :return: page size: The number of records to fetch in a page. Returns None if unspecified 59 """
Defines how to get the next page token
One instance paginates every partition of the stream, several at once, so next_page_token must derive the
next token from its arguments, such as last_page_token_value, rather than from state kept on the instance.
24 @property 25 @abstractmethod 26 def initial_token(self) -> Optional[Any]: 27 """ 28 Return the initial value of the token 29 """
Return the initial value of the token
31 @abstractmethod 32 def next_page_token( 33 self, 34 response: requests.Response, 35 last_page_size: int, 36 last_record: Optional[Record], 37 last_page_token_value: Optional[Any], 38 page_size_override: Optional[int] = None, 39 stream_slice: Optional[StreamSlice] = None, 40 ) -> Optional[Any]: 41 """ 42 :param response: response to process 43 :param last_page_size: the number of records read from the response 44 :param last_record: the last record extracted from the response 45 :param last_page_token_value: The current value of the page token made on the last request 46 :param page_size_override: the page size that was actually requested, when it differs from the configured 47 one because of a `REDUCE_PAGE_SIZE` response action. Strategies that use their configured page size as 48 a stop condition must honor this value, else they end the pagination early and skip records. 49 :param stream_slice: the slice the page was read for. Only passed to strategies that declare it, so existing 50 strategies defined outside of the CDK keep working unchanged. 51 :return: next page token. Returns None if there are no more pages to fetch 52 """ 53 pass
Parameters
- response: response to process
- last_page_size: the number of records read from the response
- last_record: the last record extracted from the response
- last_page_token_value: The current value of the page token made on the last request
- page_size_override: the page size that was actually requested, when it differs from the configured
one because of a
REDUCE_PAGE_SIZEresponse action. Strategies that use their configured page size as a stop condition must honor this value, else they end the pagination early and skip records. - stream_slice: the slice the page was read for. Only passed to strategies that declare it, so existing strategies defined outside of the CDK keep working unchanged.
Returns
next page token. Returns None if there are no more pages to fetch
55 @abstractmethod 56 def get_page_size(self) -> Optional[int]: 57 """ 58 :return: page size: The number of records to fetch in a page. Returns None if unspecified 59 """
Returns
page size: The number of records to fetch in a page. Returns None if unspecified
59@dataclass 60class ParentStreamConfig: 61 """ 62 Describes how to create a stream slice from a parent stream 63 64 stream: The stream to read records from 65 parent_key: The key of the parent stream's records that will be the stream slice key 66 partition_field: The partition key 67 extra_fields: Additional field paths to include in the stream slice 68 request_option: How to inject the slice value on an outgoing HTTP request 69 incremental_dependency (bool): Indicates if the parent stream should be read incrementally. 70 """ 71 72 stream: "AbstractStream" 73 parent_key: Union[InterpolatedString, str] 74 partition_field: Union[InterpolatedString, str] 75 config: Config 76 parameters: InitVar[Mapping[str, Any]] 77 extra_fields: Optional[Union[List[List[str]], List[List[InterpolatedString]]]] = ( 78 None # List of field paths (arrays of strings) 79 ) 80 request_option: Optional[RequestOption] = None 81 incremental_dependency: bool = False 82 lazy_read_pointer: Optional[List[Union[InterpolatedString, str]]] = None 83 84 def __post_init__(self, parameters: Mapping[str, Any]) -> None: 85 self.parent_key = InterpolatedString.create(self.parent_key, parameters=parameters) 86 self.partition_field = InterpolatedString.create( 87 self.partition_field, parameters=parameters 88 ) 89 if self.extra_fields: 90 # Create InterpolatedString for each field path in extra_keys 91 self.extra_fields = [ 92 [InterpolatedString.create(path, parameters=parameters) for path in key_path] 93 for key_path in self.extra_fields 94 ] 95 96 self.lazy_read_pointer = ( 97 [ 98 InterpolatedString.create(path, parameters=parameters) 99 if isinstance(path, str) 100 else path 101 for path in self.lazy_read_pointer 102 ] 103 if self.lazy_read_pointer 104 else None 105 )
Describes how to create a stream slice from a parent stream
stream: The stream to read records from parent_key: The key of the parent stream's records that will be the stream slice key partition_field: The partition key extra_fields: Additional field paths to include in the stream slice request_option: How to inject the slice value on an outgoing HTTP request incremental_dependency (bool): Indicates if the parent stream should be read incrementally.
7class ReadException(Exception): 8 """ 9 Raise when there is an error reading data from an API Source 10 """
Raise when there is an error reading data from an API Source
12@dataclass 13class RecordExtractor: 14 """ 15 Responsible for translating an HTTP response into a list of records by extracting records from the response. 16 """ 17 18 @abstractmethod 19 def extract_records( 20 self, 21 response: requests.Response, 22 ) -> Iterable[Mapping[str, Any]]: 23 """ 24 Selects records from the response 25 :param response: The response to extract the records from 26 :return: List of Records extracted from the response 27 """ 28 pass
Responsible for translating an HTTP response into a list of records by extracting records from the response.
18 @abstractmethod 19 def extract_records( 20 self, 21 response: requests.Response, 22 ) -> Iterable[Mapping[str, Any]]: 23 """ 24 Selects records from the response 25 :param response: The response to extract the records from 26 :return: List of Records extracted from the response 27 """ 28 pass
Selects records from the response
Parameters
- response: The response to extract the records from
Returns
List of Records extracted from the response
13@dataclass 14class RecordFilter: 15 """ 16 Filter applied on a list of Records 17 18 config (Config): The user-provided configuration as specified by the source's spec 19 condition (str): The string representing the predicate to filter a record. Records will be removed if evaluated to False 20 """ 21 22 parameters: InitVar[Mapping[str, Any]] 23 config: Config 24 condition: str = "" 25 26 def __post_init__(self, parameters: Mapping[str, Any]) -> None: 27 self._filter_interpolator = InterpolatedBoolean( 28 condition=self.condition, parameters=parameters 29 ) 30 31 def filter_records( 32 self, 33 records: Iterable[Mapping[str, Any]], 34 stream_state: StreamState, 35 stream_slice: Optional[StreamSlice] = None, 36 next_page_token: Optional[Mapping[str, Any]] = None, 37 ) -> Iterable[Mapping[str, Any]]: 38 kwargs = { 39 "stream_state": stream_state, 40 "stream_slice": stream_slice, 41 "next_page_token": next_page_token, 42 "stream_slice.extra_fields": stream_slice.extra_fields if stream_slice else {}, 43 } 44 for record in records: 45 if self._filter_interpolator.eval(self.config, record=record, **kwargs): 46 yield record
Filter applied on a list of Records
config (Config): The user-provided configuration as specified by the source's spec condition (str): The string representing the predicate to filter a record. Records will be removed if evaluated to False
31 def filter_records( 32 self, 33 records: Iterable[Mapping[str, Any]], 34 stream_state: StreamState, 35 stream_slice: Optional[StreamSlice] = None, 36 next_page_token: Optional[Mapping[str, Any]] = None, 37 ) -> Iterable[Mapping[str, Any]]: 38 kwargs = { 39 "stream_state": stream_state, 40 "stream_slice": stream_slice, 41 "next_page_token": next_page_token, 42 "stream_slice.extra_fields": stream_slice.extra_fields if stream_slice else {}, 43 } 44 for record in records: 45 if self._filter_interpolator.eval(self.config, record=record, **kwargs): 46 yield record
39@dataclass 40class RecordSelector(HttpSelector): 41 """ 42 Responsible for translating an HTTP response into a list of records by extracting records from the response and optionally filtering 43 records based on a heuristic. 44 45 Attributes: 46 extractor (RecordExtractor): The record extractor responsible for extracting records from a response 47 schema_normalization (TypeTransformer): The record normalizer responsible for casting record values to stream schema types 48 record_filter (RecordFilter): The record filter responsible for filtering extracted records 49 transformations (List[RecordTransformation]): The transformations to be done on the records 50 """ 51 52 extractor: RecordExtractor 53 config: Config 54 parameters: InitVar[Mapping[str, Any]] 55 schema_normalization: Union[TypeTransformer, DeclarativeTypeTransformer] 56 name: str 57 _name: Union[InterpolatedString, str] = field(init=False, repr=False, default="") 58 record_filter: Optional[RecordFilter] = None 59 transformations: List[RecordTransformation] = field(default_factory=lambda: []) 60 transform_before_filtering: bool = False 61 file_uploader: Optional[DefaultFileUploader] = None 62 63 def __post_init__(self, parameters: Mapping[str, Any]) -> None: 64 self._parameters = parameters 65 self._name = ( 66 InterpolatedString(self._name, parameters=parameters) 67 if isinstance(self._name, str) 68 else self._name 69 ) 70 71 @property # type: ignore 72 def name(self) -> str: 73 """ 74 :return: Stream name 75 """ 76 return ( 77 str(self._name.eval(self.config)) 78 if isinstance(self._name, InterpolatedString) 79 else self._name 80 ) 81 82 @name.setter 83 def name(self, value: str) -> None: 84 if not isinstance(value, property): 85 self._name = value 86 87 def select_records( 88 self, 89 response: requests.Response, 90 stream_state: StreamState, 91 records_schema: Mapping[str, Any], 92 stream_slice: Optional[StreamSlice] = None, 93 next_page_token: Optional[Mapping[str, Any]] = None, 94 ) -> Iterable[Record]: 95 """ 96 Selects records from the response 97 :param response: The response to select the records from 98 :param stream_state: The stream state 99 :param records_schema: json schema of records to return 100 :param stream_slice: The stream slice 101 :param next_page_token: The paginator token 102 :return: List of Records selected from the response 103 """ 104 all_data: Iterable[Mapping[str, Any]] = self._extract_and_count(response) 105 yield from self.filter_and_transform( 106 all_data, stream_state, records_schema, stream_slice, next_page_token 107 ) 108 # A custom record filter can stop reading before the end of the page, which would leave the 109 # page uncounted. 110 for _ in all_data: 111 pass 112 113 def _extract_and_count(self, response: requests.Response) -> Iterable[Mapping[str, Any]]: 114 count = 0 115 for record in self.extractor.extract_records(response): 116 count += 1 117 if not isinstance(record, _DroppedEmptyRecord): 118 yield record 119 # Kept on the response, not on the selector: a stream has one retriever, and so one 120 # selector, reading all of its partitions concurrently. 121 setattr(response, _EXTRACTED_RECORD_COUNT, count) 122 123 def filter_and_transform( 124 self, 125 all_data: Iterable[Mapping[str, Any]], 126 stream_state: StreamState, 127 records_schema: Mapping[str, Any], 128 stream_slice: Optional[StreamSlice] = None, 129 next_page_token: Optional[Mapping[str, Any]] = None, 130 ) -> Iterable[Record]: 131 """ 132 There is an issue with the selector as of 2024-08-30: it does technology-agnostic processing like filtering, transformation and 133 normalization with an API that is technology-specific (as requests.Response is only for HTTP communication using the requests 134 library). 135 136 Until we decide to move this logic away from the selector, we made this method public so that users like AsyncJobRetriever could 137 share the logic of doing transformations on a set of records. 138 """ 139 if self.transform_before_filtering: 140 transformed_data = self._transform(all_data, stream_state, stream_slice) 141 transformed_filtered_data = self._filter( 142 transformed_data, stream_state, stream_slice, next_page_token 143 ) 144 else: 145 filtered_data = self._filter(all_data, stream_state, stream_slice, next_page_token) 146 transformed_filtered_data = self._transform(filtered_data, stream_state, stream_slice) 147 normalized_data = self._normalize_by_schema( 148 transformed_filtered_data, schema=records_schema 149 ) 150 for data in normalized_data: 151 record = Record(data=data, stream_name=self.name, associated_slice=stream_slice) 152 if self.file_uploader: 153 self.file_uploader.upload(record) 154 yield record 155 156 def _normalize_by_schema( 157 self, records: Iterable[Mapping[str, Any]], schema: Optional[Mapping[str, Any]] 158 ) -> Iterable[Mapping[str, Any]]: 159 if schema: 160 # record has type Mapping[str, Any], but dict[str, Any] expected 161 for record in records: 162 normalized_record = dict(record) 163 self.schema_normalization.transform(normalized_record, schema) 164 yield normalized_record 165 else: 166 yield from records 167 168 def _filter( 169 self, 170 records: Iterable[Mapping[str, Any]], 171 stream_state: StreamState, 172 stream_slice: Optional[StreamSlice], 173 next_page_token: Optional[Mapping[str, Any]], 174 ) -> Iterable[Mapping[str, Any]]: 175 if self.record_filter: 176 yield from self.record_filter.filter_records( 177 records, 178 stream_state=stream_state, 179 stream_slice=stream_slice, 180 next_page_token=next_page_token, 181 ) 182 else: 183 yield from records 184 185 def _transform( 186 self, 187 records: Iterable[Mapping[str, Any]], 188 stream_state: StreamState, 189 stream_slice: Optional[StreamSlice] = None, 190 ) -> Iterable[Mapping[str, Any]]: 191 for record in records: 192 for transformation in self.transformations: 193 transformation.transform( 194 record, # type: ignore # record has type Mapping[str, Any], but Dict[str, Any] expected 195 config=self.config, 196 stream_state=stream_state, 197 stream_slice=stream_slice, 198 ) 199 yield record
Responsible for translating an HTTP response into a list of records by extracting records from the response and optionally filtering records based on a heuristic.
Attributes:
- extractor (RecordExtractor): The record extractor responsible for extracting records from a response
- schema_normalization (TypeTransformer): The record normalizer responsible for casting record values to stream schema types
- record_filter (RecordFilter): The record filter responsible for filtering extracted records
- transformations (List[RecordTransformation]): The transformations to be done on the records
71 @property # type: ignore 72 def name(self) -> str: 73 """ 74 :return: Stream name 75 """ 76 return ( 77 str(self._name.eval(self.config)) 78 if isinstance(self._name, InterpolatedString) 79 else self._name 80 )
Returns
Stream name
87 def select_records( 88 self, 89 response: requests.Response, 90 stream_state: StreamState, 91 records_schema: Mapping[str, Any], 92 stream_slice: Optional[StreamSlice] = None, 93 next_page_token: Optional[Mapping[str, Any]] = None, 94 ) -> Iterable[Record]: 95 """ 96 Selects records from the response 97 :param response: The response to select the records from 98 :param stream_state: The stream state 99 :param records_schema: json schema of records to return 100 :param stream_slice: The stream slice 101 :param next_page_token: The paginator token 102 :return: List of Records selected from the response 103 """ 104 all_data: Iterable[Mapping[str, Any]] = self._extract_and_count(response) 105 yield from self.filter_and_transform( 106 all_data, stream_state, records_schema, stream_slice, next_page_token 107 ) 108 # A custom record filter can stop reading before the end of the page, which would leave the 109 # page uncounted. 110 for _ in all_data: 111 pass
Selects records from the response
Parameters
- response: The response to select the records from
- stream_state: The stream state
- records_schema: json schema of records to return
- stream_slice: The stream slice
- next_page_token: The paginator token
Returns
List of Records selected from the response
123 def filter_and_transform( 124 self, 125 all_data: Iterable[Mapping[str, Any]], 126 stream_state: StreamState, 127 records_schema: Mapping[str, Any], 128 stream_slice: Optional[StreamSlice] = None, 129 next_page_token: Optional[Mapping[str, Any]] = None, 130 ) -> Iterable[Record]: 131 """ 132 There is an issue with the selector as of 2024-08-30: it does technology-agnostic processing like filtering, transformation and 133 normalization with an API that is technology-specific (as requests.Response is only for HTTP communication using the requests 134 library). 135 136 Until we decide to move this logic away from the selector, we made this method public so that users like AsyncJobRetriever could 137 share the logic of doing transformations on a set of records. 138 """ 139 if self.transform_before_filtering: 140 transformed_data = self._transform(all_data, stream_state, stream_slice) 141 transformed_filtered_data = self._filter( 142 transformed_data, stream_state, stream_slice, next_page_token 143 ) 144 else: 145 filtered_data = self._filter(all_data, stream_state, stream_slice, next_page_token) 146 transformed_filtered_data = self._transform(filtered_data, stream_state, stream_slice) 147 normalized_data = self._normalize_by_schema( 148 transformed_filtered_data, schema=records_schema 149 ) 150 for data in normalized_data: 151 record = Record(data=data, stream_name=self.name, associated_slice=stream_slice) 152 if self.file_uploader: 153 self.file_uploader.upload(record) 154 yield record
There is an issue with the selector as of 2024-08-30: it does technology-agnostic processing like filtering, transformation and normalization with an API that is technology-specific (as requests.Response is only for HTTP communication using the requests library).
Until we decide to move this logic away from the selector, we made this method public so that users like AsyncJobRetriever could share the logic of doing transformations on a set of records.
13@dataclass 14class RecordTransformation: 15 """ 16 Implementations of this class define transformations that can be applied to records of a stream. 17 """ 18 19 @abstractmethod 20 def transform( 21 self, 22 record: Dict[str, Any], 23 config: Optional[Config] = None, 24 stream_state: Optional[StreamState] = None, 25 stream_slice: Optional[StreamSlice] = None, 26 ) -> None: 27 """ 28 Transform a record by adding, deleting, or mutating fields directly from the record reference passed in argument. 29 30 :param record: The input record to be transformed 31 :param config: The user-provided configuration as specified by the source's spec 32 :param stream_state: The stream state 33 :param stream_slice: The stream slice 34 :return: The transformed record 35 """ 36 37 def __eq__(self, other: object) -> bool: 38 return other.__dict__ == self.__dict__
Implementations of this class define transformations that can be applied to records of a stream.
19 @abstractmethod 20 def transform( 21 self, 22 record: Dict[str, Any], 23 config: Optional[Config] = None, 24 stream_state: Optional[StreamState] = None, 25 stream_slice: Optional[StreamSlice] = None, 26 ) -> None: 27 """ 28 Transform a record by adding, deleting, or mutating fields directly from the record reference passed in argument. 29 30 :param record: The input record to be transformed 31 :param config: The user-provided configuration as specified by the source's spec 32 :param stream_state: The stream state 33 :param stream_slice: The stream slice 34 :return: The transformed record 35 """
Transform a record by adding, deleting, or mutating fields directly from the record reference passed in argument.
Parameters
- record: The input record to be transformed
- config: The user-provided configuration as specified by the source's spec
- stream_state: The stream state
- stream_slice: The stream slice
Returns
The transformed record
25@dataclass 26class RequestOption: 27 """ 28 Describes an option to set on a request 29 30 Attributes: 31 field_name (str): Describes the name of the parameter to inject. Mutually exclusive with field_path. 32 field_path (list(str)): Describes the path to a nested field as a list of field names. 33 Only valid for body_json injection type, and mutually exclusive with field_name. 34 inject_into (RequestOptionType): Describes where in the HTTP request to inject the parameter 35 """ 36 37 inject_into: RequestOptionType 38 parameters: InitVar[Mapping[str, Any]] 39 field_name: Optional[Union[InterpolatedString, str]] = None 40 field_path: Optional[List[Union[InterpolatedString, str]]] = None 41 42 def __post_init__(self, parameters: Mapping[str, Any]) -> None: 43 # Validate inputs. We should expect either field_name or field_path, but not both 44 if self.field_name is None and self.field_path is None: 45 raise ValueError("RequestOption requires either a field_name or field_path") 46 47 if self.field_name is not None and self.field_path is not None: 48 raise ValueError( 49 "Only one of field_name or field_path can be provided to RequestOption" 50 ) 51 52 # Nested field injection is only supported for body JSON injection 53 if self.field_path is not None and self.inject_into != RequestOptionType.body_json: 54 raise ValueError( 55 "Nested field injection is only supported for body JSON injection. Please use a top-level field_name for other injection types." 56 ) 57 58 # Convert field_name and field_path into InterpolatedString objects if they are strings 59 if self.field_name is not None: 60 self.field_name = InterpolatedString.create(self.field_name, parameters=parameters) 61 elif self.field_path is not None: 62 self.field_path = [ 63 InterpolatedString.create(segment, parameters=parameters) 64 for segment in self.field_path 65 ] 66 67 @property 68 def _is_field_path(self) -> bool: 69 """Returns whether this option is a field path (ie, a nested field)""" 70 return self.field_path is not None 71 72 def inject_into_request( 73 self, 74 target: MutableMapping[str, Any], 75 value: Any, 76 config: Config, 77 ) -> None: 78 """ 79 Inject a request option value into a target request structure using either field_name or field_path. 80 For non-body-json injection, only top-level field names are supported. 81 For body-json injection, both field names and nested field paths are supported. 82 83 Args: 84 target: The request structure to inject the value into 85 value: The value to inject 86 config: The config object to use for interpolation 87 """ 88 if self._is_field_path: 89 if self.inject_into != RequestOptionType.body_json: 90 raise ValueError( 91 "Nested field injection is only supported for body JSON injection. Please use a top-level field_name for other injection types." 92 ) 93 94 assert self.field_path is not None # for type checker 95 current = target 96 # Convert path segments into strings, evaluating any interpolated segments 97 # Example: ["data", "{{ config[user_type] }}", "id"] -> ["data", "admin", "id"] 98 *path_parts, final_key = [ 99 str( 100 segment.eval(config=config) 101 if isinstance(segment, InterpolatedString) 102 else segment 103 ) 104 for segment in self.field_path 105 ] 106 107 # Build a nested dictionary structure and set the final value at the deepest level 108 for part in path_parts: 109 current = current.setdefault(part, {}) 110 current[final_key] = value 111 else: 112 # For non-nested fields, evaluate the field name if it's an interpolated string 113 key = ( 114 self.field_name.eval(config=config) 115 if isinstance(self.field_name, InterpolatedString) 116 else self.field_name 117 ) 118 target[str(key)] = value
Describes an option to set on a request
Attributes:
- field_name (str): Describes the name of the parameter to inject. Mutually exclusive with field_path.
- field_path (list(str)): Describes the path to a nested field as a list of field names. Only valid for body_json injection type, and mutually exclusive with field_name.
- inject_into (RequestOptionType): Describes where in the HTTP request to inject the parameter
72 def inject_into_request( 73 self, 74 target: MutableMapping[str, Any], 75 value: Any, 76 config: Config, 77 ) -> None: 78 """ 79 Inject a request option value into a target request structure using either field_name or field_path. 80 For non-body-json injection, only top-level field names are supported. 81 For body-json injection, both field names and nested field paths are supported. 82 83 Args: 84 target: The request structure to inject the value into 85 value: The value to inject 86 config: The config object to use for interpolation 87 """ 88 if self._is_field_path: 89 if self.inject_into != RequestOptionType.body_json: 90 raise ValueError( 91 "Nested field injection is only supported for body JSON injection. Please use a top-level field_name for other injection types." 92 ) 93 94 assert self.field_path is not None # for type checker 95 current = target 96 # Convert path segments into strings, evaluating any interpolated segments 97 # Example: ["data", "{{ config[user_type] }}", "id"] -> ["data", "admin", "id"] 98 *path_parts, final_key = [ 99 str( 100 segment.eval(config=config) 101 if isinstance(segment, InterpolatedString) 102 else segment 103 ) 104 for segment in self.field_path 105 ] 106 107 # Build a nested dictionary structure and set the final value at the deepest level 108 for part in path_parts: 109 current = current.setdefault(part, {}) 110 current[final_key] = value 111 else: 112 # For non-nested fields, evaluate the field name if it's an interpolated string 113 key = ( 114 self.field_name.eval(config=config) 115 if isinstance(self.field_name, InterpolatedString) 116 else self.field_name 117 ) 118 target[str(key)] = value
Inject a request option value into a target request structure using either field_name or field_path. For non-body-json injection, only top-level field names are supported. For body-json injection, both field names and nested field paths are supported.
Arguments:
- target: The request structure to inject the value into
- value: The value to inject
- config: The config object to use for interpolation
14class RequestOptionType(Enum): 15 """ 16 Describes where to set a value on a request 17 """ 18 19 request_parameter = "request_parameter" 20 header = "header" 21 body_data = "body_data" 22 body_json = "body_json"
Describes where to set a value on a request
30class Requester(RequestOptionsProvider): 31 @abstractmethod 32 def get_authenticator(self) -> DeclarativeAuthenticator: 33 """ 34 Specifies the authenticator to use when submitting requests 35 """ 36 pass 37 38 @abstractmethod 39 def get_url( 40 self, 41 *, 42 stream_state: Optional[StreamState], 43 stream_slice: Optional[StreamSlice], 44 next_page_token: Optional[Mapping[str, Any]], 45 ) -> str: 46 """ 47 :return: URL base for the API endpoint e.g: if you wanted to hit https://myapi.com/v1/some_entity then this should return "https://myapi.com/v1/" 48 """ 49 50 @abstractmethod 51 def get_url_base( 52 self, 53 *, 54 stream_state: Optional[StreamState], 55 stream_slice: Optional[StreamSlice], 56 next_page_token: Optional[Mapping[str, Any]], 57 ) -> str: 58 """ 59 :return: URL base for the API endpoint e.g: if you wanted to hit https://myapi.com/v1/some_entity then this should return "https://myapi.com/v1/" 60 """ 61 62 @abstractmethod 63 def get_path( 64 self, 65 *, 66 stream_state: Optional[StreamState], 67 stream_slice: Optional[StreamSlice], 68 next_page_token: Optional[Mapping[str, Any]], 69 ) -> str: 70 """ 71 Returns the URL path for the API endpoint e.g: if you wanted to hit https://myapi.com/v1/some_entity then this should return "some_entity" 72 """ 73 74 @abstractmethod 75 def get_method(self) -> HttpMethod: 76 """ 77 Specifies the HTTP method to use 78 """ 79 80 @abstractmethod 81 def get_request_params( 82 self, 83 *, 84 stream_state: Optional[StreamState] = None, 85 stream_slice: Optional[StreamSlice] = None, 86 next_page_token: Optional[Mapping[str, Any]] = None, 87 ) -> MutableMapping[str, Any]: 88 """ 89 Specifies the query parameters that should be set on an outgoing HTTP request given the inputs. 90 91 E.g: you might want to define query parameters for paging if next_page_token is not None. 92 """ 93 94 @abstractmethod 95 def get_request_headers( 96 self, 97 *, 98 stream_state: Optional[StreamState] = None, 99 stream_slice: Optional[StreamSlice] = None, 100 next_page_token: Optional[Mapping[str, Any]] = None, 101 ) -> Mapping[str, Any]: 102 """ 103 Return any non-auth headers. Authentication headers will overwrite any overlapping headers returned from this method. 104 """ 105 106 @abstractmethod 107 def get_request_body_data( 108 self, 109 *, 110 stream_state: Optional[StreamState] = None, 111 stream_slice: Optional[StreamSlice] = None, 112 next_page_token: Optional[Mapping[str, Any]] = None, 113 ) -> Union[Mapping[str, Any], str]: 114 """ 115 Specifies how to populate the body of the request with a non-JSON payload. 116 117 If returns a ready text that it will be sent as is. 118 If returns a dict that it will be converted to a urlencoded form. 119 E.g. {"key1": "value1", "key2": "value2"} => "key1=value1&key2=value2" 120 121 At the same time only one of the 'request_body_data' and 'request_body_json' functions can be overridden. 122 """ 123 124 @abstractmethod 125 def get_request_body_json( 126 self, 127 *, 128 stream_state: Optional[StreamState] = None, 129 stream_slice: Optional[StreamSlice] = None, 130 next_page_token: Optional[Mapping[str, Any]] = None, 131 ) -> Mapping[str, Any]: 132 """ 133 Specifies how to populate the body of the request with a JSON payload. 134 135 At the same time only one of the 'request_body_data' and 'request_body_json' functions can be overridden. 136 """ 137 138 @abstractmethod 139 def send_request( 140 self, 141 stream_state: Optional[StreamState] = None, 142 stream_slice: Optional[StreamSlice] = None, 143 next_page_token: Optional[Mapping[str, Any]] = None, 144 path: Optional[str] = None, 145 request_headers: Optional[Mapping[str, Any]] = None, 146 request_params: Optional[Mapping[str, Any]] = None, 147 request_body_data: Optional[Union[Mapping[str, Any], str]] = None, 148 request_body_json: Optional[Mapping[str, Any]] = None, 149 log_formatter: Optional[Callable[[requests.Response], Any]] = None, 150 ) -> Optional[requests.Response]: 151 """ 152 Sends a request and returns the response. Might return no response if the error handler chooses to ignore the response or throw an exception in case of an error. 153 If path is set, the path configured on the requester itself is ignored. 154 If header, params and body are set, they are merged with the ones configured on the requester itself. 155 156 If a log formatter is provided, it's used to log the performed request and response. If it's not provided, no logging is performed. 157 """
Defines the request options to set on an outgoing HTTP request
Options can be passed by
- request parameter
- request headers
- body data
- json content
31 @abstractmethod 32 def get_authenticator(self) -> DeclarativeAuthenticator: 33 """ 34 Specifies the authenticator to use when submitting requests 35 """ 36 pass
Specifies the authenticator to use when submitting requests
38 @abstractmethod 39 def get_url( 40 self, 41 *, 42 stream_state: Optional[StreamState], 43 stream_slice: Optional[StreamSlice], 44 next_page_token: Optional[Mapping[str, Any]], 45 ) -> str: 46 """ 47 :return: URL base for the API endpoint e.g: if you wanted to hit https://myapi.com/v1/some_entity then this should return "https://myapi.com/v1/" 48 """
Returns
URL base for the API endpoint e.g: if you wanted to hit https://myapi.com/v1/some_entity then this should return "https://myapi.com/v1/"
50 @abstractmethod 51 def get_url_base( 52 self, 53 *, 54 stream_state: Optional[StreamState], 55 stream_slice: Optional[StreamSlice], 56 next_page_token: Optional[Mapping[str, Any]], 57 ) -> str: 58 """ 59 :return: URL base for the API endpoint e.g: if you wanted to hit https://myapi.com/v1/some_entity then this should return "https://myapi.com/v1/" 60 """
Returns
URL base for the API endpoint e.g: if you wanted to hit https://myapi.com/v1/some_entity then this should return "https://myapi.com/v1/"
62 @abstractmethod 63 def get_path( 64 self, 65 *, 66 stream_state: Optional[StreamState], 67 stream_slice: Optional[StreamSlice], 68 next_page_token: Optional[Mapping[str, Any]], 69 ) -> str: 70 """ 71 Returns the URL path for the API endpoint e.g: if you wanted to hit https://myapi.com/v1/some_entity then this should return "some_entity" 72 """
Returns the URL path for the API endpoint e.g: if you wanted to hit https://myapi.com/v1/some_entity then this should return "some_entity"
74 @abstractmethod 75 def get_method(self) -> HttpMethod: 76 """ 77 Specifies the HTTP method to use 78 """
Specifies the HTTP method to use
80 @abstractmethod 81 def get_request_params( 82 self, 83 *, 84 stream_state: Optional[StreamState] = None, 85 stream_slice: Optional[StreamSlice] = None, 86 next_page_token: Optional[Mapping[str, Any]] = None, 87 ) -> MutableMapping[str, Any]: 88 """ 89 Specifies the query parameters that should be set on an outgoing HTTP request given the inputs. 90 91 E.g: you might want to define query parameters for paging if next_page_token is not None. 92 """
Specifies the query parameters that should be set on an outgoing HTTP request given the inputs.
E.g: you might want to define query parameters for paging if next_page_token is not None.
94 @abstractmethod 95 def get_request_headers( 96 self, 97 *, 98 stream_state: Optional[StreamState] = None, 99 stream_slice: Optional[StreamSlice] = None, 100 next_page_token: Optional[Mapping[str, Any]] = None, 101 ) -> Mapping[str, Any]: 102 """ 103 Return any non-auth headers. Authentication headers will overwrite any overlapping headers returned from this method. 104 """
Return any non-auth headers. Authentication headers will overwrite any overlapping headers returned from this method.
106 @abstractmethod 107 def get_request_body_data( 108 self, 109 *, 110 stream_state: Optional[StreamState] = None, 111 stream_slice: Optional[StreamSlice] = None, 112 next_page_token: Optional[Mapping[str, Any]] = None, 113 ) -> Union[Mapping[str, Any], str]: 114 """ 115 Specifies how to populate the body of the request with a non-JSON payload. 116 117 If returns a ready text that it will be sent as is. 118 If returns a dict that it will be converted to a urlencoded form. 119 E.g. {"key1": "value1", "key2": "value2"} => "key1=value1&key2=value2" 120 121 At the same time only one of the 'request_body_data' and 'request_body_json' functions can be overridden. 122 """
Specifies how to populate the body of the request with a non-JSON payload.
If returns a ready text that it will be sent as is. If returns a dict that it will be converted to a urlencoded form. E.g. {"key1": "value1", "key2": "value2"} => "key1=value1&key2=value2"
At the same time only one of the 'request_body_data' and 'request_body_json' functions can be overridden.
124 @abstractmethod 125 def get_request_body_json( 126 self, 127 *, 128 stream_state: Optional[StreamState] = None, 129 stream_slice: Optional[StreamSlice] = None, 130 next_page_token: Optional[Mapping[str, Any]] = None, 131 ) -> Mapping[str, Any]: 132 """ 133 Specifies how to populate the body of the request with a JSON payload. 134 135 At the same time only one of the 'request_body_data' and 'request_body_json' functions can be overridden. 136 """
Specifies how to populate the body of the request with a JSON payload.
At the same time only one of the 'request_body_data' and 'request_body_json' functions can be overridden.
138 @abstractmethod 139 def send_request( 140 self, 141 stream_state: Optional[StreamState] = None, 142 stream_slice: Optional[StreamSlice] = None, 143 next_page_token: Optional[Mapping[str, Any]] = None, 144 path: Optional[str] = None, 145 request_headers: Optional[Mapping[str, Any]] = None, 146 request_params: Optional[Mapping[str, Any]] = None, 147 request_body_data: Optional[Union[Mapping[str, Any], str]] = None, 148 request_body_json: Optional[Mapping[str, Any]] = None, 149 log_formatter: Optional[Callable[[requests.Response], Any]] = None, 150 ) -> Optional[requests.Response]: 151 """ 152 Sends a request and returns the response. Might return no response if the error handler chooses to ignore the response or throw an exception in case of an error. 153 If path is set, the path configured on the requester itself is ignored. 154 If header, params and body are set, they are merged with the ones configured on the requester itself. 155 156 If a log formatter is provided, it's used to log the performed request and response. If it's not provided, no logging is performed. 157 """
Sends a request and returns the response. Might return no response if the error handler chooses to ignore the response or throw an exception in case of an error. If path is set, the path configured on the requester itself is ignored. If header, params and body are set, they are merged with the ones configured on the requester itself.
If a log formatter is provided, it's used to log the performed request and response. If it's not provided, no logging is performed.
93@dataclass 94class SimpleRetriever(Retriever): 95 """ 96 Retrieves records by synchronously sending requests to fetch records. 97 98 The retriever acts as an orchestrator between the requester, the record selector, the paginator, and the stream slicer. 99 100 For each stream slice, submit requests until there are no more pages of records to fetch. 101 102 This retriever currently inherits from HttpStream to reuse the request submission and pagination machinery. 103 As a result, some of the parameters passed to some methods are unused. 104 The two will be decoupled in a future release. 105 106 Attributes: 107 stream_name (str): The stream's name 108 stream_primary_key (Optional[Union[str, List[str], List[List[str]]]]): The stream's primary key 109 requester (Requester): The HTTP requester 110 record_selector (HttpSelector): The record selector 111 paginator (Optional[Paginator]): The paginator 112 stream_slicer (Optional[StreamSlicer]): The stream slicer 113 parameters (Mapping[str, Any]): Additional runtime parameters to be used for string interpolation 114 post_pagination_filter (Optional[ClientSideIncrementalRecordFilterDecorator]): Set for data feed streams only. 115 Records the cursor considers already synced are dropped once pagination has observed them 116 page_size_reduction (Optional[PageSizeReduction]): How much to shrink the page size when an error handler 117 resolves to `ResponseAction.REDUCE_PAGE_SIZE`. `None` disables page size reduction entirely. 118 It is immutable configuration; the page size in effect lives in a `PageSizeReducer` that 119 `_read_pages` creates per call, so the retriever and its paginator - both shared by every 120 partition of the stream, read concurrently - stay stateless. When `page_size_reduction` is 121 `None` no reducer is created and `_read_pages` keeps its previous behaviour 122 request_window_splitting (Optional[RequestWindowSplitting]): Policy applied when an error handler 123 resolves to `ResponseAction.SPLIT_REQUEST_WINDOW`, or when `RequestWindowSplitRequiredException` 124 is raised directly by custom code. `None` disables request-window splitting entirely: `read_records` 125 converts the exception into `RequestWindowSplitNotSupportedException` instead of attempting a 126 split it cannot perform. 127 request_window_splitter: Bound method - normally the stream's own cursor's `split_request_window` - 128 asked to replace a failing `StreamSlice` with smaller children. `None` has the same effect as 129 `request_window_splitting` being `None`. 130 page_observer (Optional[Callable[[PageCompleted], None]]): Observes completed response pages; set 131 only on streams using `SyncTokenCursor` and `None` otherwise. 132 """ 133 134 requester: Requester 135 record_selector: HttpSelector 136 config: Config 137 parameters: InitVar[Mapping[str, Any]] 138 name: str 139 _name: Union[InterpolatedString, str] = field(init=False, repr=False, default="") 140 primary_key: Optional[Union[str, List[str], List[List[str]]]] 141 _primary_key: str = field(init=False, repr=False, default="") 142 paginator: Optional[Paginator] = None 143 stream_slicer: StreamSlicer = field( 144 default_factory=lambda: SinglePartitionRouter(parameters={}) 145 ) 146 request_option_provider: RequestOptionsProvider = field( 147 default_factory=lambda: DefaultRequestOptionsProvider(parameters={}) 148 ) 149 ignore_stream_slicer_parameters_on_paginated_requests: bool = False 150 additional_query_properties: Optional[QueryProperties] = None 151 log_formatter: Optional[Callable[[requests.Response], Any]] = None 152 pagination_tracker_factory: Callable[[], PaginationTracker] = field( 153 default_factory=lambda: lambda: PaginationTracker() 154 ) 155 post_pagination_filter: Optional[ClientSideIncrementalRecordFilterDecorator] = None 156 page_size_reduction: Optional[PageSizeReduction] = None 157 request_window_splitting: Optional[RequestWindowSplitting] = None 158 request_window_splitter: Optional[ 159 Callable[[StreamSlice, Optional[datetime.timedelta]], Optional[List[StreamSlice]]] 160 ] = None 161 page_observer: Optional[Callable[["PageCompleted"], None]] = None 162 163 def __post_init__(self, parameters: Mapping[str, Any]) -> None: 164 self._paginator = self.paginator or NoPagination(parameters=parameters) 165 self._parameters = parameters 166 self._name = ( 167 InterpolatedString(self._name, parameters=parameters) 168 if isinstance(self._name, str) 169 else self._name 170 ) 171 172 @property # type: ignore 173 def name(self) -> str: 174 """ 175 :return: Stream name 176 """ 177 return ( 178 str(self._name.eval(self.config)) 179 if isinstance(self._name, InterpolatedString) 180 else self._name 181 ) 182 183 @name.setter 184 def name(self, value: str) -> None: 185 if not isinstance(value, property): 186 self._name = value 187 188 def _get_mapping( 189 self, method: Callable[..., Optional[Union[Mapping[str, Any], str]]], **kwargs: Any 190 ) -> Tuple[Union[Mapping[str, Any], str], Set[str]]: 191 """ 192 Get mapping from the provided method, and get the keys of the mapping. 193 If the method returns a string, it will return the string and an empty set. 194 If the method returns a dict, it will return the dict and its keys. 195 """ 196 mapping = method(**kwargs) or {} 197 keys = set(mapping.keys()) if not isinstance(mapping, str) else set() 198 return mapping, keys 199 200 def _get_request_options( 201 self, 202 stream_slice: Optional[StreamSlice], 203 next_page_token: Optional[Mapping[str, Any]], 204 paginator_method: Callable[..., Optional[Union[Mapping[str, Any], str]]], 205 stream_slicer_method: Callable[..., Optional[Union[Mapping[str, Any], str]]], 206 page_size_override: Optional[int] = None, 207 ) -> Union[Mapping[str, Any], str]: 208 """ 209 Get the request_option from the paginator and the stream slicer. 210 Raise a ValueError if there's a key collision 211 Returned merged mapping otherwise 212 """ 213 is_body_json = paginator_method.__name__ == "get_request_body_json" 214 215 mappings = [ 216 paginator_method( 217 stream_slice=stream_slice, 218 next_page_token=next_page_token, 219 **page_size_override_kwargs(page_size_override), 220 ), 221 ] 222 if not next_page_token or not self.ignore_stream_slicer_parameters_on_paginated_requests: 223 mappings.append( 224 stream_slicer_method( 225 stream_slice=stream_slice, 226 next_page_token=next_page_token, 227 ) 228 ) 229 return combine_mappings(mappings, allow_same_value_merge=is_body_json) 230 231 def _request_headers( 232 self, 233 stream_slice: Optional[StreamSlice] = None, 234 next_page_token: Optional[Mapping[str, Any]] = None, 235 page_size_override: Optional[int] = None, 236 ) -> Mapping[str, Any]: 237 """ 238 Specifies request headers. 239 Authentication headers will overwrite any overlapping headers returned from this method. 240 """ 241 headers = self._get_request_options( 242 stream_slice, 243 next_page_token, 244 self._paginator.get_request_headers, 245 self.request_option_provider.get_request_headers, 246 **page_size_override_kwargs(page_size_override), 247 ) 248 if isinstance(headers, str): 249 raise ValueError("Request headers cannot be a string") 250 return {str(k): str(v) for k, v in headers.items()} 251 252 def _request_params( 253 self, 254 stream_slice: Optional[StreamSlice] = None, 255 next_page_token: Optional[Mapping[str, Any]] = None, 256 page_size_override: Optional[int] = None, 257 ) -> Mapping[str, Any]: 258 """ 259 Specifies the query parameters that should be set on an outgoing HTTP request given the inputs. 260 261 E.g: you might want to define query parameters for paging if next_page_token is not None. 262 """ 263 params = self._get_request_options( 264 stream_slice, 265 next_page_token, 266 self._paginator.get_request_params, 267 self.request_option_provider.get_request_params, 268 **page_size_override_kwargs(page_size_override), 269 ) 270 if isinstance(params, str): 271 raise ValueError("Request params cannot be a string") 272 return params 273 274 def _request_body_data( 275 self, 276 stream_slice: Optional[StreamSlice] = None, 277 next_page_token: Optional[Mapping[str, Any]] = None, 278 page_size_override: Optional[int] = None, 279 ) -> Union[Mapping[str, Any], str]: 280 """ 281 Specifies how to populate the body of the request with a non-JSON payload. 282 283 If returns a ready text that it will be sent as is. 284 If returns a dict that it will be converted to a urlencoded form. 285 E.g. {"key1": "value1", "key2": "value2"} => "key1=value1&key2=value2" 286 287 At the same time only one of the 'request_body_data' and 'request_body_json' functions can be overridden. 288 """ 289 return self._get_request_options( 290 stream_slice, 291 next_page_token, 292 self._paginator.get_request_body_data, 293 self.request_option_provider.get_request_body_data, 294 **page_size_override_kwargs(page_size_override), 295 ) 296 297 def _request_body_json( 298 self, 299 stream_slice: Optional[StreamSlice] = None, 300 next_page_token: Optional[Mapping[str, Any]] = None, 301 page_size_override: Optional[int] = None, 302 ) -> Optional[Mapping[str, Any]]: 303 """ 304 Specifies how to populate the body of the request with a JSON payload. 305 306 At the same time only one of the 'request_body_data' and 'request_body_json' functions can be overridden. 307 """ 308 body_json = self._get_request_options( 309 stream_slice, 310 next_page_token, 311 self._paginator.get_request_body_json, 312 self.request_option_provider.get_request_body_json, 313 **page_size_override_kwargs(page_size_override), 314 ) 315 if isinstance(body_json, str): 316 raise ValueError("Request body json cannot be a string") 317 return body_json 318 319 def _paginator_path( 320 self, 321 next_page_token: Optional[Mapping[str, Any]] = None, 322 stream_slice: Optional[StreamSlice] = None, 323 ) -> Optional[str]: 324 """ 325 If the paginator points to a path, follow it, else return nothing so the requester is used. 326 :param next_page_token: 327 :return: 328 """ 329 return self._paginator.path( 330 next_page_token=next_page_token, 331 stream_state={}, # stream_state as an interpolation context is deprecated 332 stream_slice=stream_slice, 333 ) 334 335 def _parse_response( 336 self, 337 response: Optional[requests.Response], 338 records_schema: Mapping[str, Any], 339 stream_slice: Optional[StreamSlice] = None, 340 next_page_token: Optional[Mapping[str, Any]] = None, 341 ) -> Iterable[Record]: 342 if not response: 343 yield from [] 344 else: 345 yield from self.record_selector.select_records( 346 response=response, 347 stream_state={}, # stream_state as an interpolation context is deprecated 348 records_schema=records_schema, 349 stream_slice=stream_slice, 350 next_page_token=next_page_token, 351 ) 352 353 @property # type: ignore 354 def primary_key(self) -> Optional[Union[str, List[str], List[List[str]]]]: 355 """The stream's primary key""" 356 return self._primary_key 357 358 @primary_key.setter 359 def primary_key(self, value: str) -> None: 360 if not isinstance(value, property): 361 self._primary_key = value 362 363 def _next_page_token( 364 self, 365 response: requests.Response, 366 last_page_size: int, 367 last_record: Optional[Record], 368 last_page_token_value: Optional[Any], 369 page_size_override: Optional[int] = None, 370 stream_slice: Optional[StreamSlice] = None, 371 ) -> Optional[Mapping[str, Any]]: 372 """ 373 Specifies a pagination strategy. 374 375 The value returned from this method is passed to most other methods in this class. Use it to form a request e.g: set headers or query params. 376 377 :return: The token for the next page from the input response object. Returning None means there are no more pages to read in this response. 378 """ 379 return self._paginator.next_page_token( 380 response=response, 381 last_page_size=last_page_size, 382 last_record=last_record, 383 last_page_token_value=last_page_token_value, 384 **page_size_override_kwargs(page_size_override), 385 **stream_slice_kwargs(self._paginator.next_page_token, stream_slice), 386 ) 387 388 def _fetch_next_page( 389 self, 390 stream_slice: StreamSlice, 391 next_page_token: Optional[Mapping[str, Any]] = None, 392 page_size_override: Optional[int] = None, 393 ) -> Optional[requests.Response]: 394 return self.requester.send_request( 395 path=self._paginator_path( 396 next_page_token=next_page_token, 397 stream_slice=stream_slice, 398 ), 399 stream_state={}, # stream_state as an interpolation context is deprecated 400 stream_slice=stream_slice, 401 next_page_token=next_page_token, 402 request_headers=self._request_headers( 403 stream_slice=stream_slice, 404 next_page_token=next_page_token, 405 **page_size_override_kwargs(page_size_override), 406 ), 407 request_params=self._request_params( 408 stream_slice=stream_slice, 409 next_page_token=next_page_token, 410 **page_size_override_kwargs(page_size_override), 411 ), 412 request_body_data=self._request_body_data( 413 stream_slice=stream_slice, 414 next_page_token=next_page_token, 415 **page_size_override_kwargs(page_size_override), 416 ), 417 request_body_json=self._request_body_json( 418 stream_slice=stream_slice, 419 next_page_token=next_page_token, 420 **page_size_override_kwargs(page_size_override), 421 ), 422 log_formatter=self.log_formatter, 423 ) 424 425 # This logic is similar to _read_pages in the HttpStream class. When making changes here, consider making changes there as well. 426 def _read_pages( 427 self, 428 records_generator_fn: Callable[[Optional[requests.Response]], Iterable[Record]], 429 stream_slice: StreamSlice, 430 ) -> Iterable[Record]: 431 original_stream_slice = stream_slice 432 pagination_tracker = self.pagination_tracker_factory() 433 page_size_reducer = ( 434 PageSizeReducer( 435 self.page_size_reduction, 436 self._paginator.get_page_size(), 437 stream_name=self.name, 438 ) 439 if self.page_size_reduction 440 else None 441 ) 442 reset_pagination = False 443 reduce_page_size = False 444 next_page_token = self._get_initial_next_page_token() 445 while True: 446 page_size_override = page_size_reducer.page_size_override if page_size_reducer else None 447 merged_records: MutableMapping[str, Any] = defaultdict(dict) 448 last_page_size = 0 449 last_record: Optional[Record] = None 450 451 response = None 452 try: 453 if self.additional_query_properties: 454 for ( 455 properties 456 ) in self.additional_query_properties.get_request_property_chunks(): 457 stream_slice = StreamSlice( 458 partition=stream_slice.partition or {}, 459 cursor_slice=stream_slice.cursor_slice or {}, 460 extra_fields={"query_properties": properties}, 461 ) 462 response = self._fetch_next_page( 463 stream_slice, 464 next_page_token, 465 **page_size_override_kwargs(page_size_override), 466 ) 467 468 for current_record in records_generator_fn(response): 469 if self.additional_query_properties.property_chunking: 470 merge_key = self.additional_query_properties.property_chunking.get_merge_key( 471 current_record 472 ) 473 if merge_key: 474 _deep_merge(merged_records[merge_key], current_record) 475 else: 476 # We should still emit records even if the record did not have a merge key 477 pagination_tracker.observe(current_record) 478 last_page_size += 1 479 last_record = current_record 480 yield current_record 481 else: 482 pagination_tracker.observe(current_record) 483 last_page_size += 1 484 last_record = current_record 485 yield current_record 486 487 for merged_record in merged_records.values(): 488 record = Record( 489 data=merged_record, stream_name=self.name, associated_slice=stream_slice 490 ) 491 pagination_tracker.observe(record) 492 last_page_size += 1 493 last_record = record 494 yield record 495 else: 496 response = self._fetch_next_page( 497 stream_slice, 498 next_page_token, 499 **page_size_override_kwargs(page_size_override), 500 ) 501 for current_record in records_generator_fn(response): 502 pagination_tracker.observe(current_record) 503 last_page_size += 1 504 last_record = current_record 505 yield current_record 506 except PaginationResetRequiredException: 507 reset_pagination = True 508 except PageSizeReductionRequiredException as exception: 509 if page_size_reducer is None: 510 # The action can be attached to a requester we cannot validate at config time, such as one 511 # built by a custom error handler or a CustomRequester. Re-raise as the misconfiguration it 512 # is: the exception being handled is the neutral "the API asked for a smaller page" signal. 513 raise PageSizeReductionNotSupportedException(stream_name=self.name) 514 if last_page_size: 515 # The reduction is safe only because it re-issues a page whose records were not emitted. 516 # Every in-CDK way of reaching this raises from `_fetch_next_page`, before the record loop, 517 # and the factory rejects the manifest constructs that would not, but a custom extractor, 518 # filter or transformation can issue its own request from inside the record generator. 519 # Re-issuing the page then duplicates the records already yielded, so fail instead. 520 raise AirbyteTracedException( 521 internal_message=f"Stream {self.name} requested a page size reduction after {last_page_size} records of the page had already been emitted", 522 message=f"Stream {self.name} asked for a smaller page size in the middle of a page. The page cannot be requested again without duplicating the records already read from it. Move the REDUCE_PAGE_SIZE action to the error handler of the stream's main requester.", 523 failure_type=FailureType.config_error, 524 ) 525 # Raises once the page size cannot be reduced any further and the retries allowed at that 526 # floor are spent, which is what stops the loop when the API keeps failing. 527 page_size_reducer.reduce(exception.error_message) 528 reduce_page_size = True 529 else: 530 if page_size_reducer: 531 page_size_reducer.on_successful_page() 532 if not response: 533 break 534 535 if reduce_page_size: 536 # Retry the very same page: neither the token nor the slice change, only the page size does - 537 # and not even that once the reducer is at its floor and only waiting is left. 538 reduce_page_size = False 539 continue 540 541 if reset_pagination or pagination_tracker.has_reached_limit(): 542 next_page_token = self._get_initial_next_page_token() 543 previous_slice = stream_slice 544 stream_slice = pagination_tracker.reduce_slice_range_if_possible( 545 stream_slice, original_stream_slice 546 ) 547 LOGGER.info( 548 f"Hitting PaginationReset event. StreamSlice used will go from {previous_slice} to {stream_slice}" 549 ) 550 reset_pagination = False 551 else: 552 last_page_token_value = ( 553 next_page_token.get("next_page_token") if next_page_token else None 554 ) 555 next_page_token = self._next_page_token( 556 response=response, # type:ignore # we are breaking from the loop on the try/else if there are no response so this should be fine 557 last_page_size=last_page_size, 558 last_record=last_record, 559 last_page_token_value=last_page_token_value, 560 **page_size_override_kwargs(page_size_override), 561 **stream_slice_kwargs(self._next_page_token, stream_slice), 562 ) 563 if response is not None and self.page_observer: 564 self.page_observer( 565 PageCompleted( 566 stream_slice=stream_slice, 567 response=response, 568 pagination_exhausted=not next_page_token, 569 ) 570 ) 571 if not next_page_token: 572 break 573 574 # Always return an empty generator just in case no records were ever yielded 575 yield from [] 576 577 def _get_initial_next_page_token(self) -> Optional[Mapping[str, Any]]: 578 initial_token = self._paginator.get_initial_token() 579 next_page_token = {"next_page_token": initial_token} if initial_token is not None else None 580 return next_page_token 581 582 def read_records( 583 self, 584 records_schema: Mapping[str, Any], 585 stream_slice: Optional[StreamSlice] = None, 586 ) -> Iterable[StreamData]: 587 """ 588 Fetch a stream's records from an HTTP API source 589 590 :param records_schema: json schema to describe record 591 :param stream_slice: The stream slice to read data for 592 :return: The records read from the API source 593 """ 594 _slice = stream_slice or StreamSlice(partition={}, cursor_slice={}) # None-check 595 yield from self._read_records_or_split_request_window( 596 records_schema, _slice, _slice, depth=0 597 ) 598 599 def _read_records_or_split_request_window( 600 self, 601 records_schema: Mapping[str, Any], 602 stream_slice: StreamSlice, 603 original_slice: StreamSlice, 604 depth: int, 605 ) -> Iterable[StreamData]: 606 """ 607 Read `stream_slice` to completion, replacing it with smaller children and recursing into each of them in 608 turn when a `RequestWindowSplitRequiredException` is raised while reading it. 609 610 Each recursive call re-enters this method - and, through it, `_read_pages` - from scratch for the child 611 slice it is given, so every child gets its own paginator token, `PaginationTracker`, and 612 `PageSizeReducer` for free: nothing here needs to reset that state explicitly. Because the whole 613 recursion lives inside the single generator `DeclarativePartition.read()` consumes, 614 `PartitionReader.process_partition()` only calls `cursor.close_partition()` once, after this generator is 615 fully exhausted - so a failure anywhere in the recursion (a child that cannot be read, or a window that 616 cannot be split any further) propagates out without ever checkpointing the original partition. 617 618 `original_slice` is threaded through the recursion unchanged so every record yielded, however deep the 619 recursion went to produce it, is re-stamped with the partition's own slice rather than the child slice it 620 was actually read against - see `_reassociate_with_original_slice`. `depth` bounds the recursion 621 independently of `request_window_splitter`'s own no-progress guard: it is enforced here so a 622 misbehaving custom cursor cannot recurse indefinitely regardless of what that guard does or does not 623 catch. 624 """ 625 record_generator = partial( 626 self._parse_records, 627 stream_slice=stream_slice, 628 records_schema=records_schema, 629 ) 630 631 emitted_count = 0 632 try: 633 records: Iterable[Mapping[str, Any]] = self._read_pages(record_generator, stream_slice) 634 if self.post_pagination_filter: 635 # A data feed paginates until it reaches a record older than the cursor, so the page that triggers the stop 636 # condition still holds already-synced records. Those are filtered here rather than in the record selector 637 # so that the paginator keeps seeing the whole page: the stop condition is evaluated on the last record of 638 # the page, which is precisely one of the records being dropped. Two consequences of filtering this late: 639 # the pagination tracker observes the dropped records, and a `file_uploader` on the record selector has 640 # already uploaded their files by the time they are dropped. 641 records = self.post_pagination_filter.filter_records( 642 records, 643 # the filter is only used for its cursor comparison, which does not read the stream state 644 stream_state={}, 645 stream_slice=stream_slice, 646 ) 647 for record in records: 648 emitted_count += 1 649 yield self._reassociate_with_original_slice(record, original_slice) 650 except RequestWindowSplitRequiredException as exception: 651 if self.request_window_splitting is None or self.request_window_splitter is None: 652 raise RequestWindowSplitNotSupportedException(stream_name=self.name) from exception 653 654 if emitted_count: 655 # Splitting and re-reading the window re-emits these records: a failed partition is never 656 # checkpointed, so the next attempt would re-emit them anyway. Always replaying makes forward 657 # progress instead of retrying the same oversized window forever; primary-key dedup downstream 658 # handles the duplicates either way. 659 LOGGER.warning( 660 f"Stream {self.name} already emitted {emitted_count} record(s) from {stream_slice} before " 661 f"it was rejected; splitting and re-reading the window may re-emit them." 662 ) 663 664 rejection = f": {exception.error_message}" if exception.error_message else "" 665 if depth >= _MAX_REQUEST_WINDOW_SPLIT_DEPTH: 666 # Logged separately from the exception below so it stays visible even if the trace message's 667 # internal_message isn't surfaced by whatever catches it. 668 LOGGER.warning( 669 f"Stream {self.name} hit the maximum request window split depth " 670 f"({_MAX_REQUEST_WINDOW_SPLIT_DEPTH}) while splitting {original_slice}." 671 ) 672 raise AirbyteTracedException( 673 internal_message=f"Stream {self.name} exceeded the maximum request window split depth of {_MAX_REQUEST_WINDOW_SPLIT_DEPTH} while splitting {original_slice}; the API still rejected {stream_slice}{rejection}", 674 # `transient_error`, so the only remediation is the connector's own, if it defined one. 675 message=self._with_request_window_failure_message( 676 f"Stream {self.name} could not split its request window to a size the API accepts " 677 f"within {_MAX_REQUEST_WINDOW_SPLIT_DEPTH} splits." 678 ), 679 failure_type=FailureType.transient_error, 680 ) from exception 681 682 children = self.request_window_splitter( 683 stream_slice, self.request_window_splitting.min_split_window 684 ) 685 if children is None: 686 min_split_window_note = ( 687 f", its configured `min_split_window` ({self.request_window_splitting.min_split_window})" 688 if self.request_window_splitting.min_split_window 689 else "" 690 ) 691 raise AirbyteTracedException( 692 internal_message=f"Stream {self.name} could not split its request window {stream_slice} any further; the API rejected it{rejection}", 693 # `transient_error` regardless of how the triggering response was classified: exhaustion is 694 # never the user's fault, matching `PageSizeReducer`'s equivalent exhaustion branch. 695 message=self._with_request_window_failure_message( 696 f"The API kept rejecting stream {self.name}'s request window even at the smallest window " 697 f"its cursor allows{min_split_window_note}, or splitting the window further would not " 698 f"make progress." 699 ), 700 failure_type=FailureType.transient_error, 701 ) from exception 702 703 LOGGER.info( 704 f"Stream {self.name}: the API rejected request window {stream_slice} (split depth {depth}); " 705 f"reducing it to {children} and reading each in turn." 706 ) 707 for child in children: 708 yield from self._read_records_or_split_request_window( 709 records_schema, child, original_slice, depth + 1 710 ) 711 712 def _with_request_window_failure_message(self, message: str) -> str: 713 """ 714 :return: the message followed by `request_window_splitting`'s configured `failure_message`, when it 715 defined one - mirroring `PageSizeReducer._with_failure_message` 716 """ 717 assert self.request_window_splitting is not None 718 failure_message = (self.request_window_splitting.failure_message or "").strip() 719 if not failure_message: 720 return message 721 return f"{message} {failure_message}" 722 723 @staticmethod 724 def _reassociate_with_original_slice( 725 record: StreamData, original_slice: StreamSlice 726 ) -> StreamData: 727 """ 728 Records read while recursing into a split child window are parsed against that child's `StreamSlice`, 729 so `RecordSelector` stamps them with the child as `associated_slice`. `DeclarativePartition.read()` 730 passes an already-built `Record` through unchanged rather than re-wrapping it, so left uncorrected the 731 child slice - not the partition's own slice - is what `ConcurrentCursor.observe()` would key its 732 per-partition bookkeeping by. `close_partition()` always looks that bookkeeping up by the partition's own 733 slice, so it would never find it, silently losing the precise most-recently-observed cursor value for a 734 split partition (recoverable in practice today only because callers of that lookup fall back to the 735 slice's end boundary when it is missing). Re-stamping every record with `original_slice` here restores 736 the same tracking a non-split read already gets, rather than relying on that fallback. 737 """ 738 if isinstance(record, Record) and record.associated_slice is not original_slice: 739 return Record( 740 data=record.data, 741 stream_name=record.stream_name, 742 associated_slice=original_slice, 743 file_reference=record.file_reference, 744 ) 745 return record 746 747 def _parse_records( 748 self, 749 response: Optional[requests.Response], 750 records_schema: Mapping[str, Any], 751 stream_slice: Optional[StreamSlice], 752 ) -> Iterable[Record]: 753 yield from self._parse_response( 754 response, 755 stream_slice=stream_slice, 756 records_schema=records_schema, 757 ) 758 759 def must_deduplicate_query_params(self) -> bool: 760 return True 761 762 @staticmethod 763 def _to_partition_key(to_serialize: Any) -> str: 764 # separators have changed in Python 3.4. To avoid being impacted by further change, we explicitly specify our own value 765 return json.dumps(to_serialize, indent=None, separators=(",", ":"), sort_keys=True)
Retrieves records by synchronously sending requests to fetch records.
The retriever acts as an orchestrator between the requester, the record selector, the paginator, and the stream slicer.
For each stream slice, submit requests until there are no more pages of records to fetch.
This retriever currently inherits from HttpStream to reuse the request submission and pagination machinery. As a result, some of the parameters passed to some methods are unused. The two will be decoupled in a future release.
Attributes:
- stream_name (str): The stream's name
- stream_primary_key (Optional[Union[str, List[str], List[List[str]]]]): The stream's primary key
- requester (Requester): The HTTP requester
- record_selector (HttpSelector): The record selector
- paginator (Optional[Paginator]): The paginator
- stream_slicer (Optional[StreamSlicer]): The stream slicer
- parameters (Mapping[str, Any]): Additional runtime parameters to be used for string interpolation
- post_pagination_filter (Optional[ClientSideIncrementalRecordFilterDecorator]): Set for data feed streams only. Records the cursor considers already synced are dropped once pagination has observed them
- page_size_reduction (Optional[PageSizeReduction]): How much to shrink the page size when an error handler
resolves to
ResponseAction.REDUCE_PAGE_SIZE.Nonedisables page size reduction entirely. It is immutable configuration; the page size in effect lives in aPageSizeReducerthat_read_pagescreates per call, so the retriever and its paginator - both shared by every partition of the stream, read concurrently - stay stateless. Whenpage_size_reductionisNoneno reducer is created and_read_pageskeeps its previous behaviour - request_window_splitting (Optional[RequestWindowSplitting]): Policy applied when an error handler
resolves to
ResponseAction.SPLIT_REQUEST_WINDOW, or whenRequestWindowSplitRequiredExceptionis raised directly by custom code.Nonedisables request-window splitting entirely:read_recordsconverts the exception intoRequestWindowSplitNotSupportedExceptioninstead of attempting a split it cannot perform. - request_window_splitter: Bound method - normally the stream's own cursor's
split_request_window- asked to replace a failingStreamSlicewith smaller children.Nonehas the same effect asrequest_window_splittingbeingNone. - page_observer (Optional[Callable[[PageCompleted], None]]): Observes completed response pages; set
only on streams using
SyncTokenCursorandNoneotherwise.
172 @property # type: ignore 173 def name(self) -> str: 174 """ 175 :return: Stream name 176 """ 177 return ( 178 str(self._name.eval(self.config)) 179 if isinstance(self._name, InterpolatedString) 180 else self._name 181 )
Returns
Stream name
353 @property # type: ignore 354 def primary_key(self) -> Optional[Union[str, List[str], List[List[str]]]]: 355 """The stream's primary key""" 356 return self._primary_key
The stream's primary key
582 def read_records( 583 self, 584 records_schema: Mapping[str, Any], 585 stream_slice: Optional[StreamSlice] = None, 586 ) -> Iterable[StreamData]: 587 """ 588 Fetch a stream's records from an HTTP API source 589 590 :param records_schema: json schema to describe record 591 :param stream_slice: The stream slice to read data for 592 :return: The records read from the API source 593 """ 594 _slice = stream_slice or StreamSlice(partition={}, cursor_slice={}) # None-check 595 yield from self._read_records_or_split_request_window( 596 records_schema, _slice, _slice, depth=0 597 )
Fetch a stream's records from an HTTP API source
Parameters
- records_schema: json schema to describe record
- stream_slice: The stream slice to read data for
Returns
The records read from the API source
Inherited Members
13@dataclass 14class SinglePartitionRouter(PartitionRouter): 15 """Partition router returning only a stream slice""" 16 17 parameters: InitVar[Mapping[str, Any]] 18 19 def get_request_params( 20 self, 21 stream_state: Optional[StreamState] = None, 22 stream_slice: Optional[StreamSlice] = None, 23 next_page_token: Optional[Mapping[str, Any]] = None, 24 ) -> Mapping[str, Any]: 25 return {} 26 27 def get_request_headers( 28 self, 29 stream_state: Optional[StreamState] = None, 30 stream_slice: Optional[StreamSlice] = None, 31 next_page_token: Optional[Mapping[str, Any]] = None, 32 ) -> Mapping[str, Any]: 33 return {} 34 35 def get_request_body_data( 36 self, 37 stream_state: Optional[StreamState] = None, 38 stream_slice: Optional[StreamSlice] = None, 39 next_page_token: Optional[Mapping[str, Any]] = None, 40 ) -> Mapping[str, Any]: 41 return {} 42 43 def get_request_body_json( 44 self, 45 stream_state: Optional[StreamState] = None, 46 stream_slice: Optional[StreamSlice] = None, 47 next_page_token: Optional[Mapping[str, Any]] = None, 48 ) -> Mapping[str, Any]: 49 return {} 50 51 def stream_slices(self) -> Iterable[StreamSlice]: 52 yield StreamSlice(partition={}, cursor_slice={}) 53 54 def get_stream_state(self) -> Optional[Mapping[str, StreamState]]: 55 """ 56 SinglePartitionRouter doesn't have parent streams 57 """ 58 pass
Partition router returning only a stream slice
19 def get_request_params( 20 self, 21 stream_state: Optional[StreamState] = None, 22 stream_slice: Optional[StreamSlice] = None, 23 next_page_token: Optional[Mapping[str, Any]] = None, 24 ) -> Mapping[str, Any]: 25 return {}
Specifies the query parameters that should be set on an outgoing HTTP request given the inputs.
E.g: you might want to define query parameters for paging if next_page_token is not None.
27 def get_request_headers( 28 self, 29 stream_state: Optional[StreamState] = None, 30 stream_slice: Optional[StreamSlice] = None, 31 next_page_token: Optional[Mapping[str, Any]] = None, 32 ) -> Mapping[str, Any]: 33 return {}
Return any non-auth headers. Authentication headers will overwrite any overlapping headers returned from this method.
35 def get_request_body_data( 36 self, 37 stream_state: Optional[StreamState] = None, 38 stream_slice: Optional[StreamSlice] = None, 39 next_page_token: Optional[Mapping[str, Any]] = None, 40 ) -> Mapping[str, Any]: 41 return {}
Specifies how to populate the body of the request with a non-JSON payload.
If returns a ready text that it will be sent as is. If returns a dict that it will be converted to a urlencoded form. E.g. {"key1": "value1", "key2": "value2"} => "key1=value1&key2=value2"
At the same time only one of the 'request_body_data' and 'request_body_json' functions can be overridden.
43 def get_request_body_json( 44 self, 45 stream_state: Optional[StreamState] = None, 46 stream_slice: Optional[StreamSlice] = None, 47 next_page_token: Optional[Mapping[str, Any]] = None, 48 ) -> Mapping[str, Any]: 49 return {}
Specifies how to populate the body of the request with a JSON payload.
At the same time only one of the 'request_body_data' and 'request_body_json' functions can be overridden.
44class StopConditionPaginationStrategyDecorator(PaginationStrategy): 45 def __init__(self, _delegate: PaginationStrategy, stop_condition: PaginationStopCondition): 46 self._delegate = _delegate 47 self._stop_condition = stop_condition 48 49 def next_page_token( 50 self, 51 response: requests.Response, 52 last_page_size: int, 53 last_record: Optional[Record], 54 last_page_token_value: Optional[Any] = None, 55 page_size_override: Optional[int] = None, 56 stream_slice: Optional[StreamSlice] = None, 57 ) -> Optional[Any]: 58 # We evaluate in reverse order because the assumption is that most of the APIs using data feed structure 59 # will return records in descending order. In terms of performance/memory, we return the records lazily 60 if last_record and self._stop_condition.is_met(last_record): 61 return None 62 return self._delegate.next_page_token( 63 response, 64 last_page_size, 65 last_record, 66 last_page_token_value, 67 **page_size_override_kwargs(page_size_override), 68 **stream_slice_kwargs(self._delegate.next_page_token, stream_slice), 69 ) 70 71 def get_page_size(self) -> Optional[int]: 72 return self._delegate.get_page_size() 73 74 @property 75 def initial_token(self) -> Optional[Any]: 76 return self._delegate.initial_token
Defines how to get the next page token
One instance paginates every partition of the stream, several at once, so next_page_token must derive the
next token from its arguments, such as last_page_token_value, rather than from state kept on the instance.
49 def next_page_token( 50 self, 51 response: requests.Response, 52 last_page_size: int, 53 last_record: Optional[Record], 54 last_page_token_value: Optional[Any] = None, 55 page_size_override: Optional[int] = None, 56 stream_slice: Optional[StreamSlice] = None, 57 ) -> Optional[Any]: 58 # We evaluate in reverse order because the assumption is that most of the APIs using data feed structure 59 # will return records in descending order. In terms of performance/memory, we return the records lazily 60 if last_record and self._stop_condition.is_met(last_record): 61 return None 62 return self._delegate.next_page_token( 63 response, 64 last_page_size, 65 last_record, 66 last_page_token_value, 67 **page_size_override_kwargs(page_size_override), 68 **stream_slice_kwargs(self._delegate.next_page_token, stream_slice), 69 )
Parameters
- response: response to process
- last_page_size: the number of records read from the response
- last_record: the last record extracted from the response
- last_page_token_value: The current value of the page token made on the last request
- page_size_override: the page size that was actually requested, when it differs from the configured
one because of a
REDUCE_PAGE_SIZEresponse action. Strategies that use their configured page size as a stop condition must honor this value, else they end the pagination early and skip records. - stream_slice: the slice the page was read for. Only passed to strategies that declare it, so existing strategies defined outside of the CDK keep working unchanged.
Returns
next page token. Returns None if there are no more pages to fetch
76class StreamSlice(Mapping[str, Any]): 77 def __init__( 78 self, 79 *, 80 partition: Mapping[str, Any], 81 cursor_slice: Mapping[str, Any], 82 extra_fields: Optional[Mapping[str, Any]] = None, 83 ) -> None: 84 """ 85 :param partition: The partition keys representing a unique partition in the stream. 86 :param cursor_slice: The incremental cursor slice keys, such as dates or pagination tokens. 87 :param extra_fields: Additional fields that should not be part of the partition but passed along, such as metadata from the parent stream. 88 """ 89 self._partition = partition 90 self._cursor_slice = cursor_slice 91 self._extra_fields = extra_fields or {} 92 93 # Ensure that partition keys do not overlap with cursor slice keys 94 if partition.keys() & cursor_slice.keys(): 95 raise ValueError("Keys for partition and incremental sync cursor should not overlap") 96 97 self._stream_slice = dict(partition) | dict(cursor_slice) 98 99 @property 100 def partition(self) -> Mapping[str, Any]: 101 """Returns the partition portion of the stream slice.""" 102 p = self._partition 103 while isinstance(p, StreamSlice): 104 p = p.partition 105 return p 106 107 @property 108 def cursor_slice(self) -> Mapping[str, Any]: 109 """Returns the cursor slice portion of the stream slice.""" 110 c = self._cursor_slice 111 while isinstance(c, StreamSlice): 112 c = c.cursor_slice 113 return c 114 115 @property 116 def extra_fields(self) -> Mapping[str, Any]: 117 """Returns the extra fields that are not part of the partition.""" 118 return self._extra_fields 119 120 def __repr__(self) -> str: 121 return repr(self._stream_slice) 122 123 def __setitem__(self, key: str, value: Any) -> None: 124 raise ValueError("StreamSlice is immutable") 125 126 def __getitem__(self, key: str) -> Any: 127 return self._stream_slice[key] 128 129 def __len__(self) -> int: 130 return len(self._stream_slice) 131 132 def __iter__(self) -> Iterator[str]: 133 return iter(self._stream_slice) 134 135 def __contains__(self, item: Any) -> bool: 136 return item in self._stream_slice 137 138 def keys(self) -> KeysView[str]: 139 return self._stream_slice.keys() 140 141 def items(self) -> ItemsView[str, Any]: 142 return self._stream_slice.items() 143 144 def values(self) -> ValuesView[Any]: 145 return self._stream_slice.values() 146 147 def get(self, key: str, default: Any = None) -> Optional[Any]: 148 return self._stream_slice.get(key, default) 149 150 def __eq__(self, other: Any) -> bool: 151 if isinstance(other, dict): 152 return self._stream_slice == other 153 if isinstance(other, StreamSlice): 154 # noinspection PyProtectedMember 155 return self._partition == other._partition and self._cursor_slice == other._cursor_slice 156 return False 157 158 def __ne__(self, other: Any) -> bool: 159 return not self.__eq__(other) 160 161 def __json_serializable__(self) -> Any: 162 return self._stream_slice 163 164 def __hash__(self) -> int: 165 return SliceHasher.hash( 166 stream_slice=self._stream_slice 167 ) # no need to provide stream_name here as this is used for slicing the cursor 168 169 def __bool__(self) -> bool: 170 return bool(self._stream_slice) or bool(self._extra_fields)
A Mapping is a generic container for associating key/value pairs.
This class provides concrete generic implementations of all methods except for __getitem__, __iter__, and __len__.
77 def __init__( 78 self, 79 *, 80 partition: Mapping[str, Any], 81 cursor_slice: Mapping[str, Any], 82 extra_fields: Optional[Mapping[str, Any]] = None, 83 ) -> None: 84 """ 85 :param partition: The partition keys representing a unique partition in the stream. 86 :param cursor_slice: The incremental cursor slice keys, such as dates or pagination tokens. 87 :param extra_fields: Additional fields that should not be part of the partition but passed along, such as metadata from the parent stream. 88 """ 89 self._partition = partition 90 self._cursor_slice = cursor_slice 91 self._extra_fields = extra_fields or {} 92 93 # Ensure that partition keys do not overlap with cursor slice keys 94 if partition.keys() & cursor_slice.keys(): 95 raise ValueError("Keys for partition and incremental sync cursor should not overlap") 96 97 self._stream_slice = dict(partition) | dict(cursor_slice)
Parameters
- partition: The partition keys representing a unique partition in the stream.
- cursor_slice: The incremental cursor slice keys, such as dates or pagination tokens.
- extra_fields: Additional fields that should not be part of the partition but passed along, such as metadata from the parent stream.
99 @property 100 def partition(self) -> Mapping[str, Any]: 101 """Returns the partition portion of the stream slice.""" 102 p = self._partition 103 while isinstance(p, StreamSlice): 104 p = p.partition 105 return p
Returns the partition portion of the stream slice.
107 @property 108 def cursor_slice(self) -> Mapping[str, Any]: 109 """Returns the cursor slice portion of the stream slice.""" 110 c = self._cursor_slice 111 while isinstance(c, StreamSlice): 112 c = c.cursor_slice 113 return c
Returns the cursor slice portion of the stream slice.
115 @property 116 def extra_fields(self) -> Mapping[str, Any]: 117 """Returns the extra fields that are not part of the partition.""" 118 return self._extra_fields
Returns the extra fields that are not part of the partition.
D.items() -> a set-like object providing a view on D's items
108@dataclass 109class SubstreamPartitionRouter(PartitionRouter): 110 """ 111 Partition router that iterates over the parent's stream records and emits slices 112 Will populate the state with `partition_field` and `parent_slice` so they can be accessed by other components 113 114 Attributes: 115 parent_stream_configs (List[ParentStreamConfig]): parent streams to iterate over and their config 116 """ 117 118 parent_stream_configs: List[ParentStreamConfig] 119 config: Config 120 parameters: InitVar[Mapping[str, Any]] 121 122 def __post_init__(self, parameters: Mapping[str, Any]) -> None: 123 if not self.parent_stream_configs: 124 raise ValueError("SubstreamPartitionRouter needs at least 1 parent stream") 125 self._parameters = parameters 126 127 def get_request_params( 128 self, 129 stream_state: Optional[StreamState] = None, 130 stream_slice: Optional[StreamSlice] = None, 131 next_page_token: Optional[Mapping[str, Any]] = None, 132 ) -> Mapping[str, Any]: 133 # Pass the stream_slice from the argument, not the cursor because the cursor is updated after processing the response 134 return self._get_request_option(RequestOptionType.request_parameter, stream_slice) 135 136 def get_request_headers( 137 self, 138 stream_state: Optional[StreamState] = None, 139 stream_slice: Optional[StreamSlice] = None, 140 next_page_token: Optional[Mapping[str, Any]] = None, 141 ) -> Mapping[str, Any]: 142 # Pass the stream_slice from the argument, not the cursor because the cursor is updated after processing the response 143 return self._get_request_option(RequestOptionType.header, stream_slice) 144 145 def get_request_body_data( 146 self, 147 stream_state: Optional[StreamState] = None, 148 stream_slice: Optional[StreamSlice] = None, 149 next_page_token: Optional[Mapping[str, Any]] = None, 150 ) -> Mapping[str, Any]: 151 # Pass the stream_slice from the argument, not the cursor because the cursor is updated after processing the response 152 return self._get_request_option(RequestOptionType.body_data, stream_slice) 153 154 def get_request_body_json( 155 self, 156 stream_state: Optional[StreamState] = None, 157 stream_slice: Optional[StreamSlice] = None, 158 next_page_token: Optional[Mapping[str, Any]] = None, 159 ) -> Mapping[str, Any]: 160 # Pass the stream_slice from the argument, not the cursor because the cursor is updated after processing the response 161 return self._get_request_option(RequestOptionType.body_json, stream_slice) 162 163 def _get_request_option( 164 self, option_type: RequestOptionType, stream_slice: Optional[StreamSlice] 165 ) -> Mapping[str, Any]: 166 params: MutableMapping[str, Any] = {} 167 if stream_slice: 168 for parent_config in self.parent_stream_configs: 169 if ( 170 parent_config.request_option 171 and parent_config.request_option.inject_into == option_type 172 ): 173 key = parent_config.partition_field.eval(self.config) # type: ignore # partition_field is always casted to an interpolated string 174 value = stream_slice.get(key) 175 if value: 176 parent_config.request_option.inject_into_request(params, value, self.config) 177 return params 178 179 def stream_slices(self) -> Iterable[StreamSlice]: 180 """ 181 Iterate over each parent stream's record and create a StreamSlice for each record. 182 183 For each stream, iterate over its stream_slices. 184 For each stream slice, iterate over each record. 185 yield a stream slice for each such records. 186 187 If a parent slice contains no record, emit a slice with parent_record=None. 188 189 The template string can interpolate the following values: 190 - parent_stream_slice: mapping representing the parent's stream slice 191 - parent_record: mapping representing the parent record 192 - parent_stream_name: string representing the parent stream name 193 """ 194 if not self.parent_stream_configs: 195 yield from [] 196 else: 197 for parent_stream_config in self.parent_stream_configs: 198 parent_stream = parent_stream_config.stream 199 parent_field = parent_stream_config.parent_key.eval(self.config) # type: ignore # parent_key is always casted to an interpolated string 200 partition_field = parent_stream_config.partition_field.eval(self.config) # type: ignore # partition_field is always casted to an interpolated string 201 extra_fields = None 202 if parent_stream_config.extra_fields: 203 extra_fields = [ 204 [field_path_part.eval(self.config) for field_path_part in field_path] # type: ignore [union-attr] 205 for field_path in parent_stream_config.extra_fields 206 ] 207 208 for partition, is_last_slice in iterate_with_last_flag( 209 parent_stream.generate_partitions() 210 ): 211 if partition is None: 212 break 213 for parent_record, is_last_record_in_slice in iterate_with_last_flag( 214 partition.read() 215 ): 216 emit_slice = parent_record is not None 217 if parent_record is not None: 218 # In the previous CDK implementation, state management was done internally by the stream. 219 # However, this could cause issues when doing availability check for example as the availability 220 # check would progress the state so state management was moved outside of the read method. 221 # Hence, we need to call the cursor here. 222 # Note that we call observe and close_partition before emitting the associated record as the 223 # ConcurrentPerPartitionCursor will associate a record with the state of the stream after the 224 # record was consumed. 225 parent_stream.cursor.observe(parent_record) 226 parent_partition = ( 227 parent_record.associated_slice.partition 228 if parent_record.associated_slice 229 else {} 230 ) 231 record_data = parent_record.data 232 233 partition_value = dpath.get( 234 record_data, # type: ignore [arg-type] 235 parent_field, 236 default=None, 237 ) 238 if partition_value is None: 239 self.logger.debug( 240 f"Skipping {parent_stream.name} record with missing or null parent key '{parent_field}'" 241 ) 242 emit_slice = False 243 244 if emit_slice: 245 # Add extra fields 246 extracted_extra_fields = self._extract_extra_fields( 247 record_data, extra_fields 248 ) 249 250 if parent_stream_config.lazy_read_pointer: 251 extracted_extra_fields = { 252 "child_response": self._extract_child_response( 253 record_data, 254 parent_stream_config.lazy_read_pointer, # type: ignore[arg-type] # lazy_read_pointer type handeled in __post_init__ of parent_stream_config 255 ), 256 **extracted_extra_fields, 257 } 258 259 if is_last_record_in_slice: 260 parent_stream.cursor.close_partition(partition) 261 if is_last_slice: 262 parent_stream.cursor.ensure_at_least_one_state_emitted() 263 264 if emit_slice: 265 yield StreamSlice( 266 partition={ 267 partition_field: partition_value, 268 "parent_slice": parent_partition or {}, 269 }, 270 cursor_slice={}, 271 extra_fields=extracted_extra_fields, 272 ) 273 274 yield from [] 275 276 def _extract_child_response( 277 self, parent_record: Mapping[str, Any] | AirbyteMessage, pointer: List[InterpolatedString] 278 ) -> requests.Response: 279 """Extract child records from a parent record based on lazy pointers.""" 280 281 def _create_response(data: MutableMapping[str, Any]) -> SafeResponse: 282 """Create a SafeResponse with the given data.""" 283 response = SafeResponse() 284 response.content = json.dumps(data).encode("utf-8") 285 response.status_code = 200 286 return response 287 288 path = [path.eval(self.config) for path in pointer] 289 return _create_response(dpath.get(parent_record, path, default=[])) # type: ignore # argunet will be a MutableMapping, given input data structure 290 291 def _extract_extra_fields( 292 self, 293 parent_record: Mapping[str, Any] | AirbyteMessage, 294 extra_fields: Optional[List[List[str]]] = None, 295 ) -> Mapping[str, Any]: 296 """ 297 Extracts additional fields specified by their paths from the parent record. 298 299 Args: 300 parent_record (Mapping[str, Any]): The record from the parent stream to extract fields from. 301 extra_fields (Optional[List[List[str]]]): A list of field paths (as lists of strings) to extract from the parent record. 302 303 Returns: 304 Mapping[str, Any]: A dictionary containing the extracted fields. 305 The keys are the joined field paths, and the values are the corresponding extracted values. 306 """ 307 extracted_extra_fields = {} 308 if extra_fields: 309 for extra_field_path in extra_fields: 310 try: 311 extra_field_value = dpath.get( 312 parent_record, # type: ignore [arg-type] 313 extra_field_path, 314 ) 315 self.logger.debug( 316 f"Extracted extra_field_path: {extra_field_path} with value: {extra_field_value}" 317 ) 318 except KeyError: 319 self.logger.debug(f"Failed to extract extra_field_path: {extra_field_path}") 320 extra_field_value = None 321 extracted_extra_fields[".".join(extra_field_path)] = extra_field_value 322 return extracted_extra_fields 323 324 def get_stream_state(self) -> Optional[Mapping[str, StreamState]]: 325 """ 326 Get the state of the parent streams. 327 328 Returns: 329 StreamState: The current state of the parent streams. 330 331 Example of state format: 332 { 333 "parent_stream_name1": { 334 "last_updated": "2023-05-27T00:00:00Z" 335 }, 336 "parent_stream_name2": { 337 "last_updated": "2023-05-27T00:00:00Z" 338 } 339 } 340 """ 341 parent_state = {} 342 for parent_config in self.parent_stream_configs: 343 if parent_config.incremental_dependency: 344 parent_state[parent_config.stream.name] = copy.deepcopy( 345 parent_config.stream.cursor.state 346 ) 347 return parent_state 348 349 @property 350 def logger(self) -> logging.Logger: 351 return logging.getLogger("airbyte.SubstreamPartitionRouter")
Partition router that iterates over the parent's stream records and emits slices
Will populate the state with partition_field and parent_slice so they can be accessed by other components
Attributes:
- parent_stream_configs (List[ParentStreamConfig]): parent streams to iterate over and their config
127 def get_request_params( 128 self, 129 stream_state: Optional[StreamState] = None, 130 stream_slice: Optional[StreamSlice] = None, 131 next_page_token: Optional[Mapping[str, Any]] = None, 132 ) -> Mapping[str, Any]: 133 # Pass the stream_slice from the argument, not the cursor because the cursor is updated after processing the response 134 return self._get_request_option(RequestOptionType.request_parameter, stream_slice)
Specifies the query parameters that should be set on an outgoing HTTP request given the inputs.
E.g: you might want to define query parameters for paging if next_page_token is not None.
136 def get_request_headers( 137 self, 138 stream_state: Optional[StreamState] = None, 139 stream_slice: Optional[StreamSlice] = None, 140 next_page_token: Optional[Mapping[str, Any]] = None, 141 ) -> Mapping[str, Any]: 142 # Pass the stream_slice from the argument, not the cursor because the cursor is updated after processing the response 143 return self._get_request_option(RequestOptionType.header, stream_slice)
Return any non-auth headers. Authentication headers will overwrite any overlapping headers returned from this method.
145 def get_request_body_data( 146 self, 147 stream_state: Optional[StreamState] = None, 148 stream_slice: Optional[StreamSlice] = None, 149 next_page_token: Optional[Mapping[str, Any]] = None, 150 ) -> Mapping[str, Any]: 151 # Pass the stream_slice from the argument, not the cursor because the cursor is updated after processing the response 152 return self._get_request_option(RequestOptionType.body_data, stream_slice)
Specifies how to populate the body of the request with a non-JSON payload.
If returns a ready text that it will be sent as is. If returns a dict that it will be converted to a urlencoded form. E.g. {"key1": "value1", "key2": "value2"} => "key1=value1&key2=value2"
At the same time only one of the 'request_body_data' and 'request_body_json' functions can be overridden.
154 def get_request_body_json( 155 self, 156 stream_state: Optional[StreamState] = None, 157 stream_slice: Optional[StreamSlice] = None, 158 next_page_token: Optional[Mapping[str, Any]] = None, 159 ) -> Mapping[str, Any]: 160 # Pass the stream_slice from the argument, not the cursor because the cursor is updated after processing the response 161 return self._get_request_option(RequestOptionType.body_json, stream_slice)
Specifies how to populate the body of the request with a JSON payload.
At the same time only one of the 'request_body_data' and 'request_body_json' functions can be overridden.
179 def stream_slices(self) -> Iterable[StreamSlice]: 180 """ 181 Iterate over each parent stream's record and create a StreamSlice for each record. 182 183 For each stream, iterate over its stream_slices. 184 For each stream slice, iterate over each record. 185 yield a stream slice for each such records. 186 187 If a parent slice contains no record, emit a slice with parent_record=None. 188 189 The template string can interpolate the following values: 190 - parent_stream_slice: mapping representing the parent's stream slice 191 - parent_record: mapping representing the parent record 192 - parent_stream_name: string representing the parent stream name 193 """ 194 if not self.parent_stream_configs: 195 yield from [] 196 else: 197 for parent_stream_config in self.parent_stream_configs: 198 parent_stream = parent_stream_config.stream 199 parent_field = parent_stream_config.parent_key.eval(self.config) # type: ignore # parent_key is always casted to an interpolated string 200 partition_field = parent_stream_config.partition_field.eval(self.config) # type: ignore # partition_field is always casted to an interpolated string 201 extra_fields = None 202 if parent_stream_config.extra_fields: 203 extra_fields = [ 204 [field_path_part.eval(self.config) for field_path_part in field_path] # type: ignore [union-attr] 205 for field_path in parent_stream_config.extra_fields 206 ] 207 208 for partition, is_last_slice in iterate_with_last_flag( 209 parent_stream.generate_partitions() 210 ): 211 if partition is None: 212 break 213 for parent_record, is_last_record_in_slice in iterate_with_last_flag( 214 partition.read() 215 ): 216 emit_slice = parent_record is not None 217 if parent_record is not None: 218 # In the previous CDK implementation, state management was done internally by the stream. 219 # However, this could cause issues when doing availability check for example as the availability 220 # check would progress the state so state management was moved outside of the read method. 221 # Hence, we need to call the cursor here. 222 # Note that we call observe and close_partition before emitting the associated record as the 223 # ConcurrentPerPartitionCursor will associate a record with the state of the stream after the 224 # record was consumed. 225 parent_stream.cursor.observe(parent_record) 226 parent_partition = ( 227 parent_record.associated_slice.partition 228 if parent_record.associated_slice 229 else {} 230 ) 231 record_data = parent_record.data 232 233 partition_value = dpath.get( 234 record_data, # type: ignore [arg-type] 235 parent_field, 236 default=None, 237 ) 238 if partition_value is None: 239 self.logger.debug( 240 f"Skipping {parent_stream.name} record with missing or null parent key '{parent_field}'" 241 ) 242 emit_slice = False 243 244 if emit_slice: 245 # Add extra fields 246 extracted_extra_fields = self._extract_extra_fields( 247 record_data, extra_fields 248 ) 249 250 if parent_stream_config.lazy_read_pointer: 251 extracted_extra_fields = { 252 "child_response": self._extract_child_response( 253 record_data, 254 parent_stream_config.lazy_read_pointer, # type: ignore[arg-type] # lazy_read_pointer type handeled in __post_init__ of parent_stream_config 255 ), 256 **extracted_extra_fields, 257 } 258 259 if is_last_record_in_slice: 260 parent_stream.cursor.close_partition(partition) 261 if is_last_slice: 262 parent_stream.cursor.ensure_at_least_one_state_emitted() 263 264 if emit_slice: 265 yield StreamSlice( 266 partition={ 267 partition_field: partition_value, 268 "parent_slice": parent_partition or {}, 269 }, 270 cursor_slice={}, 271 extra_fields=extracted_extra_fields, 272 ) 273 274 yield from []
Iterate over each parent stream's record and create a StreamSlice for each record.
For each stream, iterate over its stream_slices. For each stream slice, iterate over each record. yield a stream slice for each such records.
If a parent slice contains no record, emit a slice with parent_record=None.
The template string can interpolate the following values:
- parent_stream_slice: mapping representing the parent's stream slice
- parent_record: mapping representing the parent record
- parent_stream_name: string representing the parent stream name
324 def get_stream_state(self) -> Optional[Mapping[str, StreamState]]: 325 """ 326 Get the state of the parent streams. 327 328 Returns: 329 StreamState: The current state of the parent streams. 330 331 Example of state format: 332 { 333 "parent_stream_name1": { 334 "last_updated": "2023-05-27T00:00:00Z" 335 }, 336 "parent_stream_name2": { 337 "last_updated": "2023-05-27T00:00:00Z" 338 } 339 } 340 """ 341 parent_state = {} 342 for parent_config in self.parent_stream_configs: 343 if parent_config.incremental_dependency: 344 parent_state[parent_config.stream.name] = copy.deepcopy( 345 parent_config.stream.cursor.state 346 ) 347 return parent_state
Get the state of the parent streams.
Returns:
StreamState: The current state of the parent streams.
Example of state format: { "parent_stream_name1": { "last_updated": "2023-05-27T00:00:00Z" }, "parent_stream_name2": { "last_updated": "2023-05-27T00:00:00Z" } }
16@dataclass 17class UnionPartitionRouter(PartitionRouter): 18 """ 19 A partition router that yields the deduplicated union of its child partition routers' slices. 20 21 Each emitted slice's partition is normalized to exactly `{partition_field: value}`. Any other 22 partition keys coming from a child router (e.g. a SubstreamPartitionRouter's `parent_slice`) 23 are moved into `extra_fields` so that per-partition state keys stay stable across children. 24 25 Given two child routers producing: 26 A: [{"repository": "org/a"}, {"repository": "org/b"}] 27 B: [{"repository": "org/b", "parent_slice": {...}}, {"repository": "org/c", "parent_slice": {...}}] 28 the union yields: 29 [{"repository": "org/a"}, {"repository": "org/b"}, {"repository": "org/c"}] 30 31 Partition values must be hashable — deduplication holds every distinct value in a set 32 for the duration of slice generation, so memory grows linearly with the number of 33 unique partitions. Note also that a union's partition count is the (deduplicated) sum 34 of its children's, which counts against `ConcurrentPerPartitionCursor`'s per-partition 35 state limit. 36 37 Attributes: 38 partition_routers (List[PartitionRouter]): The child partition routers to union. 39 partition_field (str): The single partition key all child slices are normalized to. 40 """ 41 42 partition_routers: List[PartitionRouter] 43 partition_field: str 44 parameters: InitVar[Mapping[str, Any]] 45 46 def __post_init__(self, parameters: Mapping[str, Any]) -> None: 47 self._parameters = parameters 48 49 def stream_slices(self) -> Iterable[StreamSlice]: 50 """ 51 Iterate over the child partition routers in order, yielding each partition value once. 52 53 The first occurrence of a partition value wins; later duplicates from any child are skipped. 54 """ 55 # The concurrent framework consumes stream_slices() lazily, so this set stays live 56 # (and grows with each distinct partition value) for essentially the whole sync. 57 seen: set[Any] = set() 58 for router in self.partition_routers: 59 for stream_slice in router.stream_slices(): 60 if self.partition_field not in stream_slice.partition: 61 raise ValueError( 62 f"UnionPartitionRouter expects all child partition routers to emit the " 63 f"partition field '{self.partition_field}'. Got {stream_slice.partition}" 64 ) 65 value = stream_slice.partition[self.partition_field] 66 try: 67 if value in seen: 68 continue 69 seen.add(value) 70 except TypeError as exception: 71 raise ValueError( 72 f"UnionPartitionRouter can only deduplicate hashable partition values. " 73 f"Got unhashable value {value!r} for partition field '{self.partition_field}'." 74 ) from exception 75 carried_over_fields = { 76 key: field_value 77 for key, field_value in stream_slice.partition.items() 78 if key != self.partition_field 79 } 80 yield StreamSlice( 81 partition={self.partition_field: value}, 82 cursor_slice={}, 83 extra_fields={ 84 **(dict(stream_slice.extra_fields) if stream_slice.extra_fields else {}), 85 **carried_over_fields, 86 }, 87 ) 88 89 def get_request_params( 90 self, 91 *, 92 stream_state: Optional[StreamState] = None, 93 stream_slice: Optional[StreamSlice] = None, 94 next_page_token: Optional[Mapping[str, Any]] = None, 95 ) -> Mapping[str, Any]: 96 return {} 97 98 def get_request_headers( 99 self, 100 *, 101 stream_state: Optional[StreamState] = None, 102 stream_slice: Optional[StreamSlice] = None, 103 next_page_token: Optional[Mapping[str, Any]] = None, 104 ) -> Mapping[str, Any]: 105 return {} 106 107 def get_request_body_data( 108 self, 109 *, 110 stream_state: Optional[StreamState] = None, 111 stream_slice: Optional[StreamSlice] = None, 112 next_page_token: Optional[Mapping[str, Any]] = None, 113 ) -> Mapping[str, Any]: 114 return {} 115 116 def get_request_body_json( 117 self, 118 *, 119 stream_state: Optional[StreamState] = None, 120 stream_slice: Optional[StreamSlice] = None, 121 next_page_token: Optional[Mapping[str, Any]] = None, 122 ) -> Mapping[str, Any]: 123 return {} 124 125 def get_stream_state(self) -> Optional[Mapping[str, StreamState]]: 126 """ 127 Merge the parent stream states of all child partition routers. 128 129 States are merged with a shallow dict update, which assumes no two child routers 130 reference a parent stream with the same name. If they do, the last child's state 131 for that parent wins and a warning is logged. 132 """ 133 merged_state: dict[str, StreamState] = {} 134 for router in self.partition_routers: 135 child_state = router.get_stream_state() 136 if child_state: 137 colliding_keys = merged_state.keys() & child_state.keys() 138 if colliding_keys: 139 logger.warning( 140 f"UnionPartitionRouter child routers reference the same parent stream(s) " 141 f"{sorted(colliding_keys)}; the last child's state for each colliding parent " 142 f"stream overwrites the previous one." 143 ) 144 merged_state.update(child_state) 145 return merged_state
A partition router that yields the deduplicated union of its child partition routers' slices.
Each emitted slice's partition is normalized to exactly {partition_field: value}. Any other
partition keys coming from a child router (e.g. a SubstreamPartitionRouter's parent_slice)
are moved into extra_fields so that per-partition state keys stay stable across children.
Given two child routers producing: A: [{"repository": "org/a"}, {"repository": "org/b"}] B: [{"repository": "org/b", "parent_slice": {...}}, {"repository": "org/c", "parent_slice": {...}}] the union yields: [{"repository": "org/a"}, {"repository": "org/b"}, {"repository": "org/c"}]
Partition values must be hashable — deduplication holds every distinct value in a set
for the duration of slice generation, so memory grows linearly with the number of
unique partitions. Note also that a union's partition count is the (deduplicated) sum
of its children's, which counts against ConcurrentPerPartitionCursor's per-partition
state limit.
Attributes:
- partition_routers (List[PartitionRouter]): The child partition routers to union.
- partition_field (str): The single partition key all child slices are normalized to.
49 def stream_slices(self) -> Iterable[StreamSlice]: 50 """ 51 Iterate over the child partition routers in order, yielding each partition value once. 52 53 The first occurrence of a partition value wins; later duplicates from any child are skipped. 54 """ 55 # The concurrent framework consumes stream_slices() lazily, so this set stays live 56 # (and grows with each distinct partition value) for essentially the whole sync. 57 seen: set[Any] = set() 58 for router in self.partition_routers: 59 for stream_slice in router.stream_slices(): 60 if self.partition_field not in stream_slice.partition: 61 raise ValueError( 62 f"UnionPartitionRouter expects all child partition routers to emit the " 63 f"partition field '{self.partition_field}'. Got {stream_slice.partition}" 64 ) 65 value = stream_slice.partition[self.partition_field] 66 try: 67 if value in seen: 68 continue 69 seen.add(value) 70 except TypeError as exception: 71 raise ValueError( 72 f"UnionPartitionRouter can only deduplicate hashable partition values. " 73 f"Got unhashable value {value!r} for partition field '{self.partition_field}'." 74 ) from exception 75 carried_over_fields = { 76 key: field_value 77 for key, field_value in stream_slice.partition.items() 78 if key != self.partition_field 79 } 80 yield StreamSlice( 81 partition={self.partition_field: value}, 82 cursor_slice={}, 83 extra_fields={ 84 **(dict(stream_slice.extra_fields) if stream_slice.extra_fields else {}), 85 **carried_over_fields, 86 }, 87 )
Iterate over the child partition routers in order, yielding each partition value once.
The first occurrence of a partition value wins; later duplicates from any child are skipped.
89 def get_request_params( 90 self, 91 *, 92 stream_state: Optional[StreamState] = None, 93 stream_slice: Optional[StreamSlice] = None, 94 next_page_token: Optional[Mapping[str, Any]] = None, 95 ) -> Mapping[str, Any]: 96 return {}
Specifies the query parameters that should be set on an outgoing HTTP request given the inputs.
E.g: you might want to define query parameters for paging if next_page_token is not None.
98 def get_request_headers( 99 self, 100 *, 101 stream_state: Optional[StreamState] = None, 102 stream_slice: Optional[StreamSlice] = None, 103 next_page_token: Optional[Mapping[str, Any]] = None, 104 ) -> Mapping[str, Any]: 105 return {}
Return any non-auth headers. Authentication headers will overwrite any overlapping headers returned from this method.
107 def get_request_body_data( 108 self, 109 *, 110 stream_state: Optional[StreamState] = None, 111 stream_slice: Optional[StreamSlice] = None, 112 next_page_token: Optional[Mapping[str, Any]] = None, 113 ) -> Mapping[str, Any]: 114 return {}
Specifies how to populate the body of the request with a non-JSON payload.
If returns a ready text that it will be sent as is. If returns a dict that it will be converted to a urlencoded form. E.g. {"key1": "value1", "key2": "value2"} => "key1=value1&key2=value2"
At the same time only one of the 'request_body_data' and 'request_body_json' functions can be overridden.
116 def get_request_body_json( 117 self, 118 *, 119 stream_state: Optional[StreamState] = None, 120 stream_slice: Optional[StreamSlice] = None, 121 next_page_token: Optional[Mapping[str, Any]] = None, 122 ) -> Mapping[str, Any]: 123 return {}
Specifies how to populate the body of the request with a JSON payload.
At the same time only one of the 'request_body_data' and 'request_body_json' functions can be overridden.
125 def get_stream_state(self) -> Optional[Mapping[str, StreamState]]: 126 """ 127 Merge the parent stream states of all child partition routers. 128 129 States are merged with a shallow dict update, which assumes no two child routers 130 reference a parent stream with the same name. If they do, the last child's state 131 for that parent wins and a warning is logged. 132 """ 133 merged_state: dict[str, StreamState] = {} 134 for router in self.partition_routers: 135 child_state = router.get_stream_state() 136 if child_state: 137 colliding_keys = merged_state.keys() & child_state.keys() 138 if colliding_keys: 139 logger.warning( 140 f"UnionPartitionRouter child routers reference the same parent stream(s) " 141 f"{sorted(colliding_keys)}; the last child's state for each colliding parent " 142 f"stream overwrites the previous one." 143 ) 144 merged_state.update(child_state) 145 return merged_state
Merge the parent stream states of all child partition routers.
States are merged with a shallow dict update, which assumes no two child routers reference a parent stream with the same name. If they do, the last child's state for that parent wins and a warning is logged.
17class YamlDeclarativeSource(ConcurrentDeclarativeSource): 18 """Declarative source defined by a yaml file""" 19 20 def __init__( 21 self, 22 path_to_yaml: str, 23 debug: bool = False, 24 catalog: Optional[ConfiguredAirbyteCatalog] = None, 25 config: Optional[Mapping[str, Any]] = None, 26 state: Optional[List[AirbyteStateMessage]] = None, 27 config_path: Optional[str] = None, 28 ) -> None: 29 """ 30 :param path_to_yaml: Path to the yaml file describing the source 31 """ 32 self._path_to_yaml = path_to_yaml 33 source_config = self._read_and_parse_yaml_file(path_to_yaml) 34 35 super().__init__( 36 catalog=catalog or ConfiguredAirbyteCatalog(streams=[]), 37 config=config or {}, 38 state=state or [], 39 source_config=source_config, 40 config_path=config_path, 41 ) 42 43 def _read_and_parse_yaml_file(self, path_to_yaml_file: str) -> ConnectionDefinition: 44 try: 45 # For testing purposes, we want to allow to just pass a file 46 with open(path_to_yaml_file, "r") as f: 47 return safe_load_yaml(f) # type: ignore # we assume the yaml represents a ConnectionDefinition 48 except FileNotFoundError: 49 # Running inside the container, the working directory during an operation is not structured the same as the static files 50 package = self.__class__.__module__.split(".")[0] 51 52 yaml_config = pkgutil.get_data(package, path_to_yaml_file) 53 if yaml_config: 54 decoded_yaml = yaml_config.decode() 55 return self._parse(decoded_yaml) 56 return {} 57 58 def _emit_manifest_debug_message(self, extra_args: dict[str, Any]) -> None: 59 extra_args["path_to_yaml"] = self._path_to_yaml 60 61 @staticmethod 62 def _parse(connection_definition_str: str) -> ConnectionDefinition: 63 """ 64 Parses a yaml file into a manifest. Component references still exist in the manifest which will be 65 resolved during the creating of the DeclarativeSource. 66 :param connection_definition_str: yaml string to parse 67 :return: The ConnectionDefinition parsed from connection_definition_str 68 """ 69 return safe_load_yaml(connection_definition_str) # type: ignore # safe_load_yaml doesn't return a type but know it is a Mapping
Declarative source defined by a yaml file
20 def __init__( 21 self, 22 path_to_yaml: str, 23 debug: bool = False, 24 catalog: Optional[ConfiguredAirbyteCatalog] = None, 25 config: Optional[Mapping[str, Any]] = None, 26 state: Optional[List[AirbyteStateMessage]] = None, 27 config_path: Optional[str] = None, 28 ) -> None: 29 """ 30 :param path_to_yaml: Path to the yaml file describing the source 31 """ 32 self._path_to_yaml = path_to_yaml 33 source_config = self._read_and_parse_yaml_file(path_to_yaml) 34 35 super().__init__( 36 catalog=catalog or ConfiguredAirbyteCatalog(streams=[]), 37 config=config or {}, 38 state=state or [], 39 source_config=source_config, 40 config_path=config_path, 41 )
Parameters
- path_to_yaml: Path to the yaml file describing the source
Inherited Members
419def launch(source: Source, args: List[str]) -> None: 420 source_entrypoint = AirbyteEntrypoint(source) 421 parsed_args = source_entrypoint.parse_args(args) 422 # temporarily removes the PrintBuffer because we're seeing weird print behavior for concurrent syncs 423 # Refer to: https://github.com/airbytehq/oncall/issues/6235 424 with PRINT_BUFFER: 425 for message in source_entrypoint.run(parsed_args): 426 # simply printing is creating issues for concurrent CDK as Python uses different two instructions to print: one for the message and 427 # the other for the break line. Adding `\n` to the message ensure that both are printed at the same time 428 print(f"{message}\n", end="")
61class AirbyteEntrypoint(object): 62 def __init__(self, source: Source): 63 init_uncaught_exception_handler(logger) 64 65 # Deployment mode is read when instantiating the entrypoint because it is the common path shared by syncs and connector builder test requests 66 if is_cloud_environment(): 67 _init_internal_request_filter() 68 69 self.source = source 70 self.logger = logging.getLogger(f"airbyte.{getattr(source, 'name', '')}") 71 self._memory_monitor = MemoryMonitor() 72 73 @staticmethod 74 def parse_args(args: List[str]) -> argparse.Namespace: 75 # set up parent parsers 76 parent_parser = argparse.ArgumentParser(add_help=False) 77 parent_parser.add_argument( 78 "--debug", action="store_true", help="enables detailed debug logs related to the sync" 79 ) 80 main_parser = argparse.ArgumentParser() 81 subparsers = main_parser.add_subparsers(title="commands", dest="command") 82 83 # spec 84 subparsers.add_parser( 85 "spec", help="outputs the json configuration specification", parents=[parent_parser] 86 ) 87 88 # check 89 check_parser = subparsers.add_parser( 90 "check", help="checks the config can be used to connect", parents=[parent_parser] 91 ) 92 required_check_parser = check_parser.add_argument_group("required named arguments") 93 required_check_parser.add_argument( 94 "--config", type=str, required=True, help="path to the json configuration file" 95 ) 96 check_parser.add_argument( 97 "--manifest-path", 98 type=str, 99 required=False, 100 help="path to the YAML manifest file to inject into the config", 101 ) 102 check_parser.add_argument( 103 "--components-path", 104 type=str, 105 required=False, 106 help="path to the custom components file, if it exists", 107 ) 108 109 # discover 110 discover_parser = subparsers.add_parser( 111 "discover", 112 help="outputs a catalog describing the source's schema", 113 parents=[parent_parser], 114 ) 115 required_discover_parser = discover_parser.add_argument_group("required named arguments") 116 required_discover_parser.add_argument( 117 "--config", type=str, required=True, help="path to the json configuration file" 118 ) 119 discover_parser.add_argument( 120 "--manifest-path", 121 type=str, 122 required=False, 123 help="path to the YAML manifest file to inject into the config", 124 ) 125 discover_parser.add_argument( 126 "--components-path", 127 type=str, 128 required=False, 129 help="path to the custom components file, if it exists", 130 ) 131 132 # read 133 read_parser = subparsers.add_parser( 134 "read", help="reads the source and outputs messages to STDOUT", parents=[parent_parser] 135 ) 136 137 read_parser.add_argument( 138 "--state", type=str, required=False, help="path to the json-encoded state file" 139 ) 140 required_read_parser = read_parser.add_argument_group("required named arguments") 141 required_read_parser.add_argument( 142 "--config", type=str, required=True, help="path to the json configuration file" 143 ) 144 required_read_parser.add_argument( 145 "--catalog", 146 type=str, 147 required=True, 148 help="path to the catalog used to determine which data to read", 149 ) 150 read_parser.add_argument( 151 "--manifest-path", 152 type=str, 153 required=False, 154 help="path to the YAML manifest file to inject into the config", 155 ) 156 read_parser.add_argument( 157 "--components-path", 158 type=str, 159 required=False, 160 help="path to the custom components file, if it exists", 161 ) 162 163 return main_parser.parse_args(args) 164 165 def run(self, parsed_args: argparse.Namespace) -> Iterable[str]: 166 cmd = parsed_args.command 167 if not cmd: 168 raise Exception("No command passed") 169 170 if (hasattr(parsed_args, "debug") and parsed_args.debug) or is_platform_debug_log_enabled(): 171 self.logger.setLevel(logging.DEBUG) 172 logger.setLevel(logging.DEBUG) 173 self.logger.debug("Debug logs enabled") 174 else: 175 self.logger.setLevel(logging.INFO) 176 177 source_spec: ConnectorSpecification = self.source.spec(self.logger) 178 try: 179 with tempfile.TemporaryDirectory( 180 # Cleanup can fail on Windows due to file locks. Ignore if so, 181 # rather than failing the whole process. 182 ignore_cleanup_errors=True, 183 ) as temp_dir: 184 os.environ[ENV_REQUEST_CACHE_PATH] = ( 185 temp_dir # set this as default directory for request_cache to store *.sqlite files 186 ) 187 if cmd == "spec": 188 message = AirbyteMessage(type=Type.SPEC, spec=source_spec) 189 yield from [ 190 self.airbyte_message_to_string(queued_message) 191 for queued_message in self._emit_queued_messages(self.source) 192 ] 193 yield self.airbyte_message_to_string(message) 194 else: 195 raw_config = self.source.read_config(parsed_args.config) 196 config = self.source.configure(raw_config, temp_dir) 197 198 yield from [ 199 self.airbyte_message_to_string(queued_message) 200 for queued_message in self._emit_queued_messages(self.source) 201 ] 202 if cmd == "check": 203 yield from map( 204 AirbyteEntrypoint.airbyte_message_to_string, 205 self.check(source_spec, config), 206 ) 207 elif cmd == "discover": 208 yield from map( 209 AirbyteEntrypoint.airbyte_message_to_string, 210 self.discover(source_spec, config), 211 ) 212 elif cmd == "read": 213 config_catalog = self.source.read_catalog(parsed_args.catalog) 214 state = self.source.read_state(parsed_args.state) 215 216 yield from map( 217 AirbyteEntrypoint.airbyte_message_to_string, 218 self.read(source_spec, config, config_catalog, state), 219 ) 220 else: 221 raise Exception("Unexpected command " + cmd) 222 finally: 223 yield from [ 224 self.airbyte_message_to_string(queued_message) 225 for queued_message in self._emit_queued_messages(self.source) 226 ] 227 228 def check( 229 self, source_spec: ConnectorSpecification, config: TConfig 230 ) -> Iterable[AirbyteMessage]: 231 self.set_up_secret_filter(config, source_spec.connectionSpecification) 232 try: 233 # A section check (`check-section` key) only carries the fields collected up to the section, so it is 234 # validated against the specification narrowed to that section; see `airbyte_cdk.sources.utils.check_section`. 235 section = requested_section(config, source_spec) 236 if section is None: 237 self.validate_connection(source_spec, config) 238 else: 239 self.validate_connection( 240 section_spec(source_spec, section), strip_check_section(config) 241 ) 242 except AirbyteTracedException as traced_exc: 243 connection_status = traced_exc.as_connection_status_message() 244 # The platform uses the exit code to surface unexpected failures so we raise the exception if the failure type not a config error 245 # If the failure is not exceptional, we'll emit a failed connection status message and return 246 if traced_exc.failure_type != FailureType.config_error: 247 raise traced_exc 248 if connection_status: 249 yield from self._emit_queued_messages(self.source) 250 yield connection_status 251 return 252 253 try: 254 check_result = self._check(source_spec, section, config) 255 except AirbyteTracedException as traced_exc: 256 yield traced_exc.as_sanitized_airbyte_message() 257 # The platform uses the exit code to surface unexpected failures so we raise the exception if the failure type not a config error 258 # If the failure is not exceptional, we'll emit a failed connection status message and return 259 if traced_exc.failure_type != FailureType.config_error: 260 raise traced_exc 261 else: 262 connection_status = traced_exc.as_connection_status_message() 263 if connection_status: 264 yield connection_status 265 return 266 if check_result.status == Status.SUCCEEDED: 267 self.logger.info("Check succeeded") 268 else: 269 self.logger.error("Check failed") 270 271 yield from self._emit_queued_messages(self.source) 272 yield AirbyteMessage(type=Type.CONNECTION_STATUS, connectionStatus=check_result) 273 274 def _check( 275 self, source_spec: ConnectorSpecification, section: Optional[str], config: TConfig 276 ) -> AirbyteConnectionStatus: 277 """ 278 The full check, or, for a section check, the full check when the section is the last one declared (the 279 configuration is complete by then) and the connector's `check_section` otherwise. 280 """ 281 if section is None: 282 return self.source.check(self.logger, config) 283 section_config = strip_check_section(config) 284 if is_last_section(source_spec, section): 285 self.logger.info(f"Section '{section}' is the last one, running the full config check.") 286 return self.source.check(self.logger, section_config) 287 self.logger.info(f"Checking configuration section '{section}'.") 288 return self.source.check_section(self.logger, section, section_config) 289 290 def discover( 291 self, source_spec: ConnectorSpecification, config: TConfig 292 ) -> Iterable[AirbyteMessage]: 293 self.set_up_secret_filter(config, source_spec.connectionSpecification) 294 if self.source.check_config_against_spec: 295 self.validate_connection(source_spec, config) 296 catalog = self.source.discover(self.logger, config) 297 298 yield from self._emit_queued_messages(self.source) 299 yield AirbyteMessage(type=Type.CATALOG, catalog=catalog) 300 301 def read( 302 self, source_spec: ConnectorSpecification, config: TConfig, catalog: Any, state: list[Any] 303 ) -> Iterable[AirbyteMessage]: 304 self.set_up_secret_filter(config, source_spec.connectionSpecification) 305 if self.source.check_config_against_spec: 306 self.validate_connection(source_spec, config) 307 308 # The Airbyte protocol dictates that counts be expressed as float/double to better protect against integer overflows 309 stream_message_counter: DefaultDict[HashableStreamDescriptor, float] = defaultdict(float) 310 for message in self.source.read(self.logger, config, catalog, state): 311 yield self.handle_record_counts(message, stream_message_counter) 312 try: 313 self._memory_monitor.check_memory_usage() 314 except AirbyteTracedException: 315 # Flush queued messages (state checkpoints, logs) before propagating 316 # the memory fail-fast exception, so the platform receives the last 317 # committed state for the next sync. 318 for queued_message in self._emit_queued_messages(self.source): 319 yield self.handle_record_counts(queued_message, stream_message_counter) 320 raise 321 322 # Flush queued messages after normal completion of the read loop. 323 for message in self._emit_queued_messages(self.source): 324 yield self.handle_record_counts(message, stream_message_counter) 325 326 @staticmethod 327 def handle_record_counts( 328 message: AirbyteMessage, stream_message_count: DefaultDict[HashableStreamDescriptor, float] 329 ) -> AirbyteMessage: 330 match message.type: 331 case Type.RECORD: 332 if message.record is None: 333 raise ValueError("Record message must have a record attribute") 334 335 stream_message_count[ 336 HashableStreamDescriptor( 337 name=message.record.stream, # type: ignore[union-attr] # record has `stream` 338 namespace=message.record.namespace, # type: ignore[union-attr] # record has `namespace` 339 ) 340 ] += 1.0 341 case Type.STATE: 342 if message.state is None: 343 raise ValueError("State message must have a state attribute") 344 345 stream_descriptor = message_utils.get_stream_descriptor(message) 346 347 # Set record count from the counter onto the state message 348 message.state.sourceStats = message.state.sourceStats or AirbyteStateStats() # type: ignore[union-attr] # state has `sourceStats` 349 message.state.sourceStats.recordCount = stream_message_count.get( # type: ignore[union-attr] # state has `sourceStats` 350 stream_descriptor, 0.0 351 ) 352 353 # Reset the counter 354 stream_message_count[stream_descriptor] = 0.0 355 return message 356 357 @staticmethod 358 def validate_connection(source_spec: ConnectorSpecification, config: TConfig) -> None: 359 # Remove internal flags from config before validating so 360 # jsonschema's additionalProperties flag won't fail the validation 361 connector_config, _ = split_config(config) 362 check_config_against_spec_or_exit(connector_config, source_spec) 363 364 @staticmethod 365 def set_up_secret_filter(config: TConfig, connection_specification: Mapping[str, Any]) -> None: 366 # Now that we have the config, we can use it to get a list of ai airbyte_secrets 367 # that we should filter in logging to avoid leaking secrets 368 config_secrets = get_secrets(connection_specification, config) 369 update_secrets(config_secrets) 370 371 @staticmethod 372 def airbyte_message_to_string(airbyte_message: AirbyteMessage) -> str: 373 global _HAS_LOGGED_FOR_SERIALIZATION_ERROR 374 serialized_message = AirbyteMessageSerializer.dump(airbyte_message) 375 try: 376 return orjson.dumps(serialized_message).decode() 377 except Exception as exception: 378 if not _HAS_LOGGED_FOR_SERIALIZATION_ERROR: 379 logger.warning( 380 f"There was an error during the serialization of an AirbyteMessage: `{exception}`. This might impact the sync performances." 381 ) 382 _HAS_LOGGED_FOR_SERIALIZATION_ERROR = True 383 try: 384 return json.dumps(serialized_message) 385 except Exception as json_exception: 386 raise AirbyteTracedException( 387 internal_message=f"Failed to serialize AirbyteMessage to JSON: `{json_exception}`", 388 failure_type=FailureType.system_error, 389 message="A record returned from the API failed to be serialized to JSON.", 390 ) from json_exception 391 392 @classmethod 393 def extract_state(cls, args: List[str]) -> Optional[Any]: 394 parsed_args = cls.parse_args(args) 395 if hasattr(parsed_args, "state"): 396 return parsed_args.state 397 return None 398 399 @classmethod 400 def extract_catalog(cls, args: List[str]) -> Optional[Any]: 401 parsed_args = cls.parse_args(args) 402 if hasattr(parsed_args, "catalog"): 403 return parsed_args.catalog 404 return None 405 406 @classmethod 407 def extract_config(cls, args: List[str]) -> Optional[Any]: 408 parsed_args = cls.parse_args(args) 409 if hasattr(parsed_args, "config"): 410 return parsed_args.config 411 return None 412 413 def _emit_queued_messages(self, source: Source) -> Iterable[AirbyteMessage]: 414 if hasattr(source, "message_repository") and source.message_repository: 415 yield from source.message_repository.consume_queue() 416 return
62 def __init__(self, source: Source): 63 init_uncaught_exception_handler(logger) 64 65 # Deployment mode is read when instantiating the entrypoint because it is the common path shared by syncs and connector builder test requests 66 if is_cloud_environment(): 67 _init_internal_request_filter() 68 69 self.source = source 70 self.logger = logging.getLogger(f"airbyte.{getattr(source, 'name', '')}") 71 self._memory_monitor = MemoryMonitor()
73 @staticmethod 74 def parse_args(args: List[str]) -> argparse.Namespace: 75 # set up parent parsers 76 parent_parser = argparse.ArgumentParser(add_help=False) 77 parent_parser.add_argument( 78 "--debug", action="store_true", help="enables detailed debug logs related to the sync" 79 ) 80 main_parser = argparse.ArgumentParser() 81 subparsers = main_parser.add_subparsers(title="commands", dest="command") 82 83 # spec 84 subparsers.add_parser( 85 "spec", help="outputs the json configuration specification", parents=[parent_parser] 86 ) 87 88 # check 89 check_parser = subparsers.add_parser( 90 "check", help="checks the config can be used to connect", parents=[parent_parser] 91 ) 92 required_check_parser = check_parser.add_argument_group("required named arguments") 93 required_check_parser.add_argument( 94 "--config", type=str, required=True, help="path to the json configuration file" 95 ) 96 check_parser.add_argument( 97 "--manifest-path", 98 type=str, 99 required=False, 100 help="path to the YAML manifest file to inject into the config", 101 ) 102 check_parser.add_argument( 103 "--components-path", 104 type=str, 105 required=False, 106 help="path to the custom components file, if it exists", 107 ) 108 109 # discover 110 discover_parser = subparsers.add_parser( 111 "discover", 112 help="outputs a catalog describing the source's schema", 113 parents=[parent_parser], 114 ) 115 required_discover_parser = discover_parser.add_argument_group("required named arguments") 116 required_discover_parser.add_argument( 117 "--config", type=str, required=True, help="path to the json configuration file" 118 ) 119 discover_parser.add_argument( 120 "--manifest-path", 121 type=str, 122 required=False, 123 help="path to the YAML manifest file to inject into the config", 124 ) 125 discover_parser.add_argument( 126 "--components-path", 127 type=str, 128 required=False, 129 help="path to the custom components file, if it exists", 130 ) 131 132 # read 133 read_parser = subparsers.add_parser( 134 "read", help="reads the source and outputs messages to STDOUT", parents=[parent_parser] 135 ) 136 137 read_parser.add_argument( 138 "--state", type=str, required=False, help="path to the json-encoded state file" 139 ) 140 required_read_parser = read_parser.add_argument_group("required named arguments") 141 required_read_parser.add_argument( 142 "--config", type=str, required=True, help="path to the json configuration file" 143 ) 144 required_read_parser.add_argument( 145 "--catalog", 146 type=str, 147 required=True, 148 help="path to the catalog used to determine which data to read", 149 ) 150 read_parser.add_argument( 151 "--manifest-path", 152 type=str, 153 required=False, 154 help="path to the YAML manifest file to inject into the config", 155 ) 156 read_parser.add_argument( 157 "--components-path", 158 type=str, 159 required=False, 160 help="path to the custom components file, if it exists", 161 ) 162 163 return main_parser.parse_args(args)
165 def run(self, parsed_args: argparse.Namespace) -> Iterable[str]: 166 cmd = parsed_args.command 167 if not cmd: 168 raise Exception("No command passed") 169 170 if (hasattr(parsed_args, "debug") and parsed_args.debug) or is_platform_debug_log_enabled(): 171 self.logger.setLevel(logging.DEBUG) 172 logger.setLevel(logging.DEBUG) 173 self.logger.debug("Debug logs enabled") 174 else: 175 self.logger.setLevel(logging.INFO) 176 177 source_spec: ConnectorSpecification = self.source.spec(self.logger) 178 try: 179 with tempfile.TemporaryDirectory( 180 # Cleanup can fail on Windows due to file locks. Ignore if so, 181 # rather than failing the whole process. 182 ignore_cleanup_errors=True, 183 ) as temp_dir: 184 os.environ[ENV_REQUEST_CACHE_PATH] = ( 185 temp_dir # set this as default directory for request_cache to store *.sqlite files 186 ) 187 if cmd == "spec": 188 message = AirbyteMessage(type=Type.SPEC, spec=source_spec) 189 yield from [ 190 self.airbyte_message_to_string(queued_message) 191 for queued_message in self._emit_queued_messages(self.source) 192 ] 193 yield self.airbyte_message_to_string(message) 194 else: 195 raw_config = self.source.read_config(parsed_args.config) 196 config = self.source.configure(raw_config, temp_dir) 197 198 yield from [ 199 self.airbyte_message_to_string(queued_message) 200 for queued_message in self._emit_queued_messages(self.source) 201 ] 202 if cmd == "check": 203 yield from map( 204 AirbyteEntrypoint.airbyte_message_to_string, 205 self.check(source_spec, config), 206 ) 207 elif cmd == "discover": 208 yield from map( 209 AirbyteEntrypoint.airbyte_message_to_string, 210 self.discover(source_spec, config), 211 ) 212 elif cmd == "read": 213 config_catalog = self.source.read_catalog(parsed_args.catalog) 214 state = self.source.read_state(parsed_args.state) 215 216 yield from map( 217 AirbyteEntrypoint.airbyte_message_to_string, 218 self.read(source_spec, config, config_catalog, state), 219 ) 220 else: 221 raise Exception("Unexpected command " + cmd) 222 finally: 223 yield from [ 224 self.airbyte_message_to_string(queued_message) 225 for queued_message in self._emit_queued_messages(self.source) 226 ]
228 def check( 229 self, source_spec: ConnectorSpecification, config: TConfig 230 ) -> Iterable[AirbyteMessage]: 231 self.set_up_secret_filter(config, source_spec.connectionSpecification) 232 try: 233 # A section check (`check-section` key) only carries the fields collected up to the section, so it is 234 # validated against the specification narrowed to that section; see `airbyte_cdk.sources.utils.check_section`. 235 section = requested_section(config, source_spec) 236 if section is None: 237 self.validate_connection(source_spec, config) 238 else: 239 self.validate_connection( 240 section_spec(source_spec, section), strip_check_section(config) 241 ) 242 except AirbyteTracedException as traced_exc: 243 connection_status = traced_exc.as_connection_status_message() 244 # The platform uses the exit code to surface unexpected failures so we raise the exception if the failure type not a config error 245 # If the failure is not exceptional, we'll emit a failed connection status message and return 246 if traced_exc.failure_type != FailureType.config_error: 247 raise traced_exc 248 if connection_status: 249 yield from self._emit_queued_messages(self.source) 250 yield connection_status 251 return 252 253 try: 254 check_result = self._check(source_spec, section, config) 255 except AirbyteTracedException as traced_exc: 256 yield traced_exc.as_sanitized_airbyte_message() 257 # The platform uses the exit code to surface unexpected failures so we raise the exception if the failure type not a config error 258 # If the failure is not exceptional, we'll emit a failed connection status message and return 259 if traced_exc.failure_type != FailureType.config_error: 260 raise traced_exc 261 else: 262 connection_status = traced_exc.as_connection_status_message() 263 if connection_status: 264 yield connection_status 265 return 266 if check_result.status == Status.SUCCEEDED: 267 self.logger.info("Check succeeded") 268 else: 269 self.logger.error("Check failed") 270 271 yield from self._emit_queued_messages(self.source) 272 yield AirbyteMessage(type=Type.CONNECTION_STATUS, connectionStatus=check_result)
290 def discover( 291 self, source_spec: ConnectorSpecification, config: TConfig 292 ) -> Iterable[AirbyteMessage]: 293 self.set_up_secret_filter(config, source_spec.connectionSpecification) 294 if self.source.check_config_against_spec: 295 self.validate_connection(source_spec, config) 296 catalog = self.source.discover(self.logger, config) 297 298 yield from self._emit_queued_messages(self.source) 299 yield AirbyteMessage(type=Type.CATALOG, catalog=catalog)
301 def read( 302 self, source_spec: ConnectorSpecification, config: TConfig, catalog: Any, state: list[Any] 303 ) -> Iterable[AirbyteMessage]: 304 self.set_up_secret_filter(config, source_spec.connectionSpecification) 305 if self.source.check_config_against_spec: 306 self.validate_connection(source_spec, config) 307 308 # The Airbyte protocol dictates that counts be expressed as float/double to better protect against integer overflows 309 stream_message_counter: DefaultDict[HashableStreamDescriptor, float] = defaultdict(float) 310 for message in self.source.read(self.logger, config, catalog, state): 311 yield self.handle_record_counts(message, stream_message_counter) 312 try: 313 self._memory_monitor.check_memory_usage() 314 except AirbyteTracedException: 315 # Flush queued messages (state checkpoints, logs) before propagating 316 # the memory fail-fast exception, so the platform receives the last 317 # committed state for the next sync. 318 for queued_message in self._emit_queued_messages(self.source): 319 yield self.handle_record_counts(queued_message, stream_message_counter) 320 raise 321 322 # Flush queued messages after normal completion of the read loop. 323 for message in self._emit_queued_messages(self.source): 324 yield self.handle_record_counts(message, stream_message_counter)
326 @staticmethod 327 def handle_record_counts( 328 message: AirbyteMessage, stream_message_count: DefaultDict[HashableStreamDescriptor, float] 329 ) -> AirbyteMessage: 330 match message.type: 331 case Type.RECORD: 332 if message.record is None: 333 raise ValueError("Record message must have a record attribute") 334 335 stream_message_count[ 336 HashableStreamDescriptor( 337 name=message.record.stream, # type: ignore[union-attr] # record has `stream` 338 namespace=message.record.namespace, # type: ignore[union-attr] # record has `namespace` 339 ) 340 ] += 1.0 341 case Type.STATE: 342 if message.state is None: 343 raise ValueError("State message must have a state attribute") 344 345 stream_descriptor = message_utils.get_stream_descriptor(message) 346 347 # Set record count from the counter onto the state message 348 message.state.sourceStats = message.state.sourceStats or AirbyteStateStats() # type: ignore[union-attr] # state has `sourceStats` 349 message.state.sourceStats.recordCount = stream_message_count.get( # type: ignore[union-attr] # state has `sourceStats` 350 stream_descriptor, 0.0 351 ) 352 353 # Reset the counter 354 stream_message_count[stream_descriptor] = 0.0 355 return message
357 @staticmethod 358 def validate_connection(source_spec: ConnectorSpecification, config: TConfig) -> None: 359 # Remove internal flags from config before validating so 360 # jsonschema's additionalProperties flag won't fail the validation 361 connector_config, _ = split_config(config) 362 check_config_against_spec_or_exit(connector_config, source_spec)
364 @staticmethod 365 def set_up_secret_filter(config: TConfig, connection_specification: Mapping[str, Any]) -> None: 366 # Now that we have the config, we can use it to get a list of ai airbyte_secrets 367 # that we should filter in logging to avoid leaking secrets 368 config_secrets = get_secrets(connection_specification, config) 369 update_secrets(config_secrets)
371 @staticmethod 372 def airbyte_message_to_string(airbyte_message: AirbyteMessage) -> str: 373 global _HAS_LOGGED_FOR_SERIALIZATION_ERROR 374 serialized_message = AirbyteMessageSerializer.dump(airbyte_message) 375 try: 376 return orjson.dumps(serialized_message).decode() 377 except Exception as exception: 378 if not _HAS_LOGGED_FOR_SERIALIZATION_ERROR: 379 logger.warning( 380 f"There was an error during the serialization of an AirbyteMessage: `{exception}`. This might impact the sync performances." 381 ) 382 _HAS_LOGGED_FOR_SERIALIZATION_ERROR = True 383 try: 384 return json.dumps(serialized_message) 385 except Exception as json_exception: 386 raise AirbyteTracedException( 387 internal_message=f"Failed to serialize AirbyteMessage to JSON: `{json_exception}`", 388 failure_type=FailureType.system_error, 389 message="A record returned from the API failed to be serialized to JSON.", 390 ) from json_exception
528class AbstractAPIBudget(abc.ABC): 529 """Interface to some API where a client allowed to have N calls per T interval. 530 531 Important: APIBudget is not doing any API calls, the end user code is responsible to call this interface 532 to respect call rate limitation of the API. 533 534 It supports multiple policies applied to different group of requests. To distinct these groups we use RequestMatchers. 535 Individual policy represented by MovingWindowCallRatePolicy and currently supports only moving window strategy. 536 """ 537 538 @abc.abstractmethod 539 def acquire_call( 540 self, request: Any, block: bool = True, timeout: Optional[float] = None 541 ) -> None: 542 """Try to get a call from budget, will block by default 543 544 :param request: 545 :param block: when true (default) will block the current thread until call credit is available 546 :param timeout: if set will limit maximum time in block, otherwise will wait until credit is available 547 :raises: CallRateLimitHit - when no credits left and if timeout was set the waiting time exceed the timeout 548 """ 549 550 @abc.abstractmethod 551 def get_matching_policy(self, request: Any) -> Optional[AbstractCallRatePolicy]: 552 """Find matching call rate policy for specific request""" 553 554 @abc.abstractmethod 555 def update_from_response(self, request: Any, response: Any) -> None: 556 """Update budget information based on response from API 557 558 :param request: the initial request that triggered this response 559 :param response: response from the API 560 """
Interface to some API where a client allowed to have N calls per T interval.
Important: APIBudget is not doing any API calls, the end user code is responsible to call this interface to respect call rate limitation of the API.
It supports multiple policies applied to different group of requests. To distinct these groups we use RequestMatchers. Individual policy represented by MovingWindowCallRatePolicy and currently supports only moving window strategy.
538 @abc.abstractmethod 539 def acquire_call( 540 self, request: Any, block: bool = True, timeout: Optional[float] = None 541 ) -> None: 542 """Try to get a call from budget, will block by default 543 544 :param request: 545 :param block: when true (default) will block the current thread until call credit is available 546 :param timeout: if set will limit maximum time in block, otherwise will wait until credit is available 547 :raises: CallRateLimitHit - when no credits left and if timeout was set the waiting time exceed the timeout 548 """
Try to get a call from budget, will block by default
Parameters
- request:
- block: when true (default) will block the current thread until call credit is available
- timeout: if set will limit maximum time in block, otherwise will wait until credit is available
Raises
- CallRateLimitHit - when no credits left and if timeout was set the waiting time exceed the timeout
550 @abc.abstractmethod 551 def get_matching_policy(self, request: Any) -> Optional[AbstractCallRatePolicy]: 552 """Find matching call rate policy for specific request"""
Find matching call rate policy for specific request
554 @abc.abstractmethod 555 def update_from_response(self, request: Any, response: Any) -> None: 556 """Update budget information based on response from API 557 558 :param request: the initial request that triggered this response 559 :param response: response from the API 560 """
Update budget information based on response from API
Parameters
- request: the initial request that triggered this response
- response: response from the API
13class AbstractHeaderAuthenticator(AuthBase): 14 """Abstract class for an header-based authenticators that add a header to outgoing HTTP requests.""" 15 16 def __call__(self, request: requests.PreparedRequest) -> Any: 17 """Attach the HTTP headers required to authenticate on the HTTP request""" 18 request.headers.update(self.get_auth_header()) 19 return request 20 21 def get_auth_header(self) -> Mapping[str, Any]: 22 """The header to set on outgoing HTTP requests""" 23 if self.auth_header: 24 return {self.auth_header: self.token} 25 return {} 26 27 @property 28 @abstractmethod 29 def auth_header(self) -> str: 30 """HTTP header to set on the requests""" 31 32 @property 33 @abstractmethod 34 def token(self) -> str: 35 """The header value to set on outgoing HTTP requests"""
Abstract class for an header-based authenticators that add a header to outgoing HTTP requests.
21 def get_auth_header(self) -> Mapping[str, Any]: 22 """The header to set on outgoing HTTP requests""" 23 if self.auth_header: 24 return {self.auth_header: self.token} 25 return {}
The header to set on outgoing HTTP requests
14class BaseBackoffException(requests.exceptions.HTTPError): 15 def __init__( 16 self, 17 request: requests.PreparedRequest, 18 response: Optional[Union[requests.Response, Exception]], 19 error_message: str = "", 20 failure_type: Optional[FailureType] = None, 21 ): 22 self.failure_type = failure_type 23 if isinstance(response, requests.Response): 24 error_message = ( 25 error_message 26 or f"Request URL: {request.url}, Response Code: {response.status_code}, Response Text: {response.text}" 27 ) 28 super().__init__(error_message, request=request, response=response) 29 else: 30 error_message = error_message or f"Request URL: {request.url}, Exception: {response}" 31 super().__init__(error_message, request=request, response=None)
An HTTP error occurred.
15 def __init__( 16 self, 17 request: requests.PreparedRequest, 18 response: Optional[Union[requests.Response, Exception]], 19 error_message: str = "", 20 failure_type: Optional[FailureType] = None, 21 ): 22 self.failure_type = failure_type 23 if isinstance(response, requests.Response): 24 error_message = ( 25 error_message 26 or f"Request URL: {request.url}, Response Code: {response.status_code}, Response Text: {response.text}" 27 ) 28 super().__init__(error_message, request=request, response=response) 29 else: 30 error_message = error_message or f"Request URL: {request.url}, Exception: {response}" 31 super().__init__(error_message, request=request, response=None)
754class CachedLimiterSession(requests_cache.CacheMixin, LimiterMixin, requests.Session): 755 """Session class with caching and rate-limiting behavior."""
Session class with caching and rate-limiting behavior.
An HTTP error occurred.
Inherited Members
34def default_backoff_handler( 35 max_tries: Optional[int], factor: float, max_time: Optional[int] = None, **kwargs: Any 36) -> Callable[[SendRequestCallableType], SendRequestCallableType]: 37 def log_retry_attempt(details: Mapping[str, Any]) -> None: 38 _, exc, _ = sys.exc_info() 39 if isinstance(exc, RequestException) and exc.response: 40 logger.info( 41 f"Status code: {exc.response.status_code!r}, Response Content: {exc.response.content!r}" 42 ) 43 logger.info( 44 f"Caught retryable error '{str(exc)}' after {details['tries']} tries. Waiting {details['wait']} seconds then retrying..." 45 ) 46 47 def should_give_up(exc: Exception) -> bool: 48 # If a non-rate-limiting related 4XX error makes it this far, it means it was unexpected and probably consistent, so we shouldn't back off 49 if isinstance(exc, RequestException): 50 if exc.response is not None: 51 give_up: bool = ( 52 exc.response is not None 53 and exc.response.status_code != codes.too_many_requests 54 and 400 <= exc.response.status_code < 500 55 ) 56 if give_up: 57 logger.info(f"Giving up for returned HTTP status: {exc.response.status_code!r}") 58 return give_up 59 # Only RequestExceptions are retryable, so if we get here, it's not retryable 60 return False 61 62 return backoff.on_exception( # type: ignore # Decorator function returns a function with a different signature than the input function, so mypy can't infer the type of the returned function 63 backoff.expo, 64 TRANSIENT_EXCEPTIONS, 65 jitter=None, 66 on_backoff=log_retry_attempt, 67 giveup=should_give_up, 68 max_tries=max_tries, 69 max_time=max_time, 70 factor=factor, 71 **kwargs, 72 )
681class HttpAPIBudget(APIBudget): 682 """Implementation of AbstractAPIBudget for HTTP""" 683 684 def __init__( 685 self, 686 ratelimit_reset_header: str = "ratelimit-reset", 687 ratelimit_remaining_header: str = "ratelimit-remaining", 688 status_codes_for_ratelimit_hit: list[int] = [429], 689 **kwargs: Any, 690 ): 691 """Constructor 692 693 :param ratelimit_reset_header: name of the header that has a timestamp of the next reset of call budget 694 :param ratelimit_remaining_header: name of the header that has the number of calls left 695 :param status_codes_for_ratelimit_hit: list of HTTP status codes that signal about rate limit being hit 696 """ 697 self._ratelimit_reset_header = ratelimit_reset_header 698 self._ratelimit_remaining_header = ratelimit_remaining_header 699 self._status_codes_for_ratelimit_hit = status_codes_for_ratelimit_hit 700 super().__init__(**kwargs) 701 702 def update_from_response(self, request: Any, response: Any) -> None: 703 policy = self.get_matching_policy(request) 704 if not policy: 705 return 706 707 if isinstance(response, requests.Response): 708 available_calls = self.get_calls_left_from_response(response) 709 reset_ts = self.get_reset_ts_from_response(response) 710 policy.update(available_calls=available_calls, call_reset_ts=reset_ts) 711 712 def get_reset_ts_from_response( 713 self, response: requests.Response 714 ) -> Optional[datetime.datetime]: 715 if response.headers.get(self._ratelimit_reset_header): 716 return datetime.datetime.fromtimestamp( 717 int(response.headers[self._ratelimit_reset_header]) 718 ) 719 return None 720 721 def get_calls_left_from_response(self, response: requests.Response) -> Optional[int]: 722 if response.headers.get(self._ratelimit_remaining_header): 723 return int(response.headers[self._ratelimit_remaining_header]) 724 725 if response.status_code in self._status_codes_for_ratelimit_hit: 726 return 0 727 728 return None
Implementation of AbstractAPIBudget for HTTP
684 def __init__( 685 self, 686 ratelimit_reset_header: str = "ratelimit-reset", 687 ratelimit_remaining_header: str = "ratelimit-remaining", 688 status_codes_for_ratelimit_hit: list[int] = [429], 689 **kwargs: Any, 690 ): 691 """Constructor 692 693 :param ratelimit_reset_header: name of the header that has a timestamp of the next reset of call budget 694 :param ratelimit_remaining_header: name of the header that has the number of calls left 695 :param status_codes_for_ratelimit_hit: list of HTTP status codes that signal about rate limit being hit 696 """ 697 self._ratelimit_reset_header = ratelimit_reset_header 698 self._ratelimit_remaining_header = ratelimit_remaining_header 699 self._status_codes_for_ratelimit_hit = status_codes_for_ratelimit_hit 700 super().__init__(**kwargs)
Constructor
Parameters
- ratelimit_reset_header: name of the header that has a timestamp of the next reset of call budget
- ratelimit_remaining_header: name of the header that has the number of calls left
- status_codes_for_ratelimit_hit: list of HTTP status codes that signal about rate limit being hit
702 def update_from_response(self, request: Any, response: Any) -> None: 703 policy = self.get_matching_policy(request) 704 if not policy: 705 return 706 707 if isinstance(response, requests.Response): 708 available_calls = self.get_calls_left_from_response(response) 709 reset_ts = self.get_reset_ts_from_response(response) 710 policy.update(available_calls=available_calls, call_reset_ts=reset_ts)
Update budget information based on the API response.
Parameters
- request: the initial request that triggered this response
- response: response from the API
721 def get_calls_left_from_response(self, response: requests.Response) -> Optional[int]: 722 if response.headers.get(self._ratelimit_remaining_header): 723 return int(response.headers[self._ratelimit_remaining_header]) 724 725 if response.status_code in self._status_codes_for_ratelimit_hit: 726 return 0 727 728 return None
Inherited Members
103class HttpRequestMatcher(RequestMatcher): 104 """Simple implementation of RequestMatcher for HTTP requests using HttpRequestRegexMatcher under the hood.""" 105 106 def __init__( 107 self, 108 method: Optional[str] = None, 109 url: Optional[str] = None, 110 params: Optional[Mapping[str, Any]] = None, 111 headers: Optional[Mapping[str, Any]] = None, 112 ): 113 """Constructor 114 115 :param method: HTTP method (e.g., "GET", "POST"). 116 :param url: Full URL to match. 117 :param params: Dictionary of query parameters to match. 118 :param headers: Dictionary of headers to match. 119 """ 120 # Parse the URL to extract the base and path 121 if url: 122 parsed_url = parse.urlsplit(url) 123 url_base = f"{parsed_url.scheme}://{parsed_url.netloc}" 124 url_path = parsed_url.path if parsed_url.path != "/" else None 125 else: 126 url_base = None 127 url_path = None 128 129 # Use HttpRequestRegexMatcher under the hood 130 self._regex_matcher = HttpRequestRegexMatcher( 131 method=method, 132 url_base=url_base, 133 url_path_pattern=re.escape(url_path) if url_path else None, 134 params=params, 135 headers=headers, 136 ) 137 138 def __call__(self, request: Any) -> bool: 139 """ 140 :param request: A requests.Request or requests.PreparedRequest instance. 141 :return: True if the request matches all provided criteria; False otherwise. 142 """ 143 return self._regex_matcher(request) 144 145 def __str__(self) -> str: 146 return ( 147 f"HttpRequestMatcher(method={self._regex_matcher._method}, " 148 f"url={self._regex_matcher._url_base}{self._regex_matcher._url_path_pattern.pattern if self._regex_matcher._url_path_pattern else ''}, " 149 f"params={self._regex_matcher._params}, headers={self._regex_matcher._headers})" 150 )
Simple implementation of RequestMatcher for HTTP requests using HttpRequestRegexMatcher under the hood.
106 def __init__( 107 self, 108 method: Optional[str] = None, 109 url: Optional[str] = None, 110 params: Optional[Mapping[str, Any]] = None, 111 headers: Optional[Mapping[str, Any]] = None, 112 ): 113 """Constructor 114 115 :param method: HTTP method (e.g., "GET", "POST"). 116 :param url: Full URL to match. 117 :param params: Dictionary of query parameters to match. 118 :param headers: Dictionary of headers to match. 119 """ 120 # Parse the URL to extract the base and path 121 if url: 122 parsed_url = parse.urlsplit(url) 123 url_base = f"{parsed_url.scheme}://{parsed_url.netloc}" 124 url_path = parsed_url.path if parsed_url.path != "/" else None 125 else: 126 url_base = None 127 url_path = None 128 129 # Use HttpRequestRegexMatcher under the hood 130 self._regex_matcher = HttpRequestRegexMatcher( 131 method=method, 132 url_base=url_base, 133 url_path_pattern=re.escape(url_path) if url_path else None, 134 params=params, 135 headers=headers, 136 )
Constructor
Parameters
- method: HTTP method (e.g., "GET", "POST").
- url: Full URL to match.
- params: Dictionary of query parameters to match.
- headers: Dictionary of headers to match.
45class HttpStream(Stream, CheckpointMixin, ABC): 46 """ 47 Base abstract class for an Airbyte Stream using the HTTP protocol. Basic building block for users building an Airbyte source for a HTTP API. 48 """ 49 50 source_defined_cursor = True # Most HTTP streams use a source defined cursor (i.e: the user can't configure it like on a SQL table) 51 page_size: Optional[int] = ( 52 None # Use this variable to define page size for API http requests with pagination support 53 ) 54 55 def __init__( 56 self, authenticator: Optional[AuthBase] = None, api_budget: Optional[APIBudget] = None 57 ): 58 self._exit_on_rate_limit: bool = False 59 self._http_client = HttpClient( 60 name=self.name, 61 logger=self.logger, 62 error_handler=self.get_error_handler(), 63 api_budget=api_budget or APIBudget(policies=[]), 64 authenticator=authenticator, 65 use_cache=self.use_cache, 66 backoff_strategy=self.get_backoff_strategy(), 67 message_repository=InMemoryMessageRepository(), 68 request_timeout=self.request_timeout, 69 use_tcp_keepalive=self.use_tcp_keepalive, 70 ) 71 72 # There are three conditions that dictate if RFR should automatically be applied to a stream 73 # 1. Streams that explicitly initialize their own cursor should defer to it and not automatically apply RFR 74 # 2. Streams with at least one cursor_field are incremental and thus a superior sync to RFR. 75 # 3. Streams overriding read_records() do not guarantee that they will call the parent implementation which can perform 76 # per-page checkpointing so RFR is only supported if a stream use the default `HttpStream.read_records()` method 77 if ( 78 not self.cursor 79 and len(self.cursor_field) == 0 80 and type(self).read_records is HttpStream.read_records 81 ): 82 self.cursor = ResumableFullRefreshCursor() 83 84 @property 85 def exit_on_rate_limit(self) -> bool: 86 """ 87 :return: False if the stream will retry endlessly when rate limited 88 """ 89 return self._exit_on_rate_limit 90 91 @exit_on_rate_limit.setter 92 def exit_on_rate_limit(self, value: bool) -> None: 93 self._exit_on_rate_limit = value 94 95 @property 96 def cache_filename(self) -> str: 97 """ 98 Override if needed. Return the name of cache file 99 Note that if the environment variable REQUEST_CACHE_PATH is not set, the cache will be in-memory only. 100 """ 101 return f"{self.name}.sqlite" 102 103 @property 104 def use_cache(self) -> bool: 105 """ 106 Override if needed. If True, all records will be cached. 107 Note that if the environment variable REQUEST_CACHE_PATH is not set, the cache will be in-memory only. 108 """ 109 return False 110 111 @property 112 def use_tcp_keepalive(self) -> bool: 113 """ 114 Override if needed. If True, the stream's HTTP session enables TCP keepalive (60 s idle, 10 s interval, 115 6 probes), so a dropped connection fails in about two minutes instead of blocking a read forever. 116 """ 117 return False 118 119 @property 120 def request_timeout(self) -> Optional[Union[float, Tuple[Optional[float], Optional[float]]]]: 121 """ 122 Override if needed. Timeout in seconds for every request this stream sends, as a number or a 123 (connect, read) pair. A "timeout" returned by request_kwargs() takes precedence. None means no timeout. 124 """ 125 return None 126 127 @property 128 @abstractmethod 129 def url_base(self) -> str: 130 """ 131 :return: URL base for the API endpoint e.g: if you wanted to hit https://myapi.com/v1/some_entity then this should return "https://myapi.com/v1/" 132 """ 133 134 @property 135 def http_method(self) -> str: 136 """ 137 Override if needed. See get_request_data/get_request_json if using POST/PUT/PATCH. 138 """ 139 return "GET" 140 141 @property 142 @deprecated( 143 "Deprecated as of CDK version 3.0.0. " 144 "You should set error_handler explicitly in HttpStream.get_error_handler() instead." 145 ) 146 def raise_on_http_errors(self) -> bool: 147 """ 148 Override if needed. If set to False, allows opting-out of raising HTTP code exception. 149 """ 150 return True 151 152 @property 153 @deprecated( 154 "Deprecated as of CDK version 3.0.0. " 155 "You should set backoff_strategies explicitly in HttpStream.get_backoff_strategy() instead." 156 ) 157 def max_retries(self) -> Union[int, None]: 158 """ 159 Override if needed. Specifies maximum amount of retries for backoff policy. Return None for no limit. 160 """ 161 return 5 162 163 @property 164 @deprecated( 165 "Deprecated as of CDK version 3.0.0. " 166 "You should set backoff_strategies explicitly in HttpStream.get_backoff_strategy() instead." 167 ) 168 def max_time(self) -> Union[int, None]: 169 """ 170 Override if needed. Specifies maximum total waiting time (in seconds) for backoff policy. Return None for no limit. 171 """ 172 return 60 * 10 173 174 @property 175 @deprecated( 176 "Deprecated as of CDK version 3.0.0. " 177 "You should set backoff_strategies explicitly in HttpStream.get_backoff_strategy() instead." 178 ) 179 def retry_factor(self) -> float: 180 """ 181 Override if needed. Specifies factor for backoff policy. 182 """ 183 return 5 184 185 @abstractmethod 186 def next_page_token(self, response: requests.Response) -> Optional[Mapping[str, Any]]: 187 """ 188 Override this method to define a pagination strategy. 189 190 The value returned from this method is passed to most other methods in this class. Use it to form a request e.g: set headers or query params. 191 192 :return: The token for the next page from the input response object. Returning None means there are no more pages to read in this response. 193 """ 194 195 @abstractmethod 196 def path( 197 self, 198 *, 199 stream_state: Optional[Mapping[str, Any]] = None, 200 stream_slice: Optional[Mapping[str, Any]] = None, 201 next_page_token: Optional[Mapping[str, Any]] = None, 202 ) -> str: 203 """ 204 Returns the URL path for the API endpoint e.g: if you wanted to hit https://myapi.com/v1/some_entity then this should return "some_entity" 205 """ 206 207 def request_params( 208 self, 209 stream_state: Optional[Mapping[str, Any]], 210 stream_slice: Optional[Mapping[str, Any]] = None, 211 next_page_token: Optional[Mapping[str, Any]] = None, 212 ) -> MutableMapping[str, Any]: 213 """ 214 Override this method to define the query parameters that should be set on an outgoing HTTP request given the inputs. 215 216 E.g: you might want to define query parameters for paging if next_page_token is not None. 217 """ 218 return {} 219 220 def request_headers( 221 self, 222 stream_state: Optional[Mapping[str, Any]], 223 stream_slice: Optional[Mapping[str, Any]] = None, 224 next_page_token: Optional[Mapping[str, Any]] = None, 225 ) -> Mapping[str, Any]: 226 """ 227 Override to return any non-auth headers. Authentication headers will overwrite any overlapping headers returned from this method. 228 """ 229 return {} 230 231 def request_body_data( 232 self, 233 stream_state: Optional[Mapping[str, Any]], 234 stream_slice: Optional[Mapping[str, Any]] = None, 235 next_page_token: Optional[Mapping[str, Any]] = None, 236 ) -> Optional[Union[Mapping[str, Any], str]]: 237 """ 238 Override when creating POST/PUT/PATCH requests to populate the body of the request with a non-JSON payload. 239 240 If returns a ready text that it will be sent as is. 241 If returns a dict that it will be converted to a urlencoded form. 242 E.g. {"key1": "value1", "key2": "value2"} => "key1=value1&key2=value2" 243 244 At the same time only one of the 'request_body_data' and 'request_body_json' functions can be overridden. 245 """ 246 return None 247 248 def request_body_json( 249 self, 250 stream_state: Optional[Mapping[str, Any]], 251 stream_slice: Optional[Mapping[str, Any]] = None, 252 next_page_token: Optional[Mapping[str, Any]] = None, 253 ) -> Optional[Mapping[str, Any]]: 254 """ 255 Override when creating POST/PUT/PATCH requests to populate the body of the request with a JSON payload. 256 257 At the same time only one of the 'request_body_data' and 'request_body_json' functions can be overridden. 258 """ 259 return None 260 261 def request_kwargs( 262 self, 263 stream_state: Optional[Mapping[str, Any]], 264 stream_slice: Optional[Mapping[str, Any]] = None, 265 next_page_token: Optional[Mapping[str, Any]] = None, 266 ) -> Mapping[str, Any]: 267 """ 268 Override to return a mapping of keyword arguments to be used when creating the HTTP request. 269 Any option listed in https://docs.python-requests.org/en/latest/api/#requests.adapters.BaseAdapter.send for can be returned from 270 this method. Note that these options do not conflict with request-level options such as headers, request params, etc.. 271 """ 272 return {} 273 274 @abstractmethod 275 def parse_response( 276 self, 277 response: requests.Response, 278 *, 279 stream_state: Mapping[str, Any], 280 stream_slice: Optional[Mapping[str, Any]] = None, 281 next_page_token: Optional[Mapping[str, Any]] = None, 282 ) -> Iterable[Mapping[str, Any]]: 283 """ 284 Parses the raw response object into a list of records. 285 By default, this returns an iterable containing the input. Override to parse differently. 286 :param response: 287 :param stream_state: 288 :param stream_slice: 289 :param next_page_token: 290 :return: An iterable containing the parsed response 291 """ 292 293 def get_backoff_strategy(self) -> Optional[Union[BackoffStrategy, List[BackoffStrategy]]]: 294 """ 295 Used to initialize Adapter to avoid breaking changes. 296 If Stream has a `backoff_time` method implementation, we know this stream uses old (pre-HTTPClient) backoff handlers and thus an adapter is needed. 297 298 Override to provide custom BackoffStrategy 299 :return Optional[BackoffStrategy]: 300 """ 301 if hasattr(self, "backoff_time"): 302 return HttpStreamAdapterBackoffStrategy(self) 303 else: 304 return None 305 306 def get_error_handler(self) -> Optional[ErrorHandler]: 307 """ 308 Used to initialize Adapter to avoid breaking changes. 309 If Stream has a `should_retry` method implementation, we know this stream uses old (pre-HTTPClient) error handlers and thus an adapter is needed. 310 311 Override to provide custom ErrorHandler 312 :return Optional[ErrorHandler]: 313 """ 314 if hasattr(self, "should_retry"): 315 error_handler = HttpStreamAdapterHttpStatusErrorHandler( 316 stream=self, 317 logger=logging.getLogger(), 318 max_retries=self.max_retries, 319 max_time=timedelta(seconds=self.max_time or 0), 320 ) 321 return error_handler 322 else: 323 return None 324 325 @classmethod 326 def _join_url(cls, url_base: str, path: str) -> str: 327 return urljoin(url_base, path) 328 329 @classmethod 330 def parse_response_error_message(cls, response: requests.Response) -> Optional[str]: 331 """ 332 Parses the raw response object from a failed request into a user-friendly error message. 333 By default, this method tries to grab the error message from JSON responses by following common API patterns. Override to parse differently. 334 335 :param response: 336 :return: A user-friendly message that indicates the cause of the error 337 """ 338 339 # default logic to grab error from common fields 340 def _try_get_error(value: Optional[JsonType]) -> Optional[str]: 341 if isinstance(value, str): 342 return value 343 elif isinstance(value, list): 344 errors_in_value = [_try_get_error(v) for v in value] 345 return ", ".join(v for v in errors_in_value if v is not None) 346 elif isinstance(value, dict): 347 new_value = ( 348 value.get("message") 349 or value.get("messages") 350 or value.get("error") 351 or value.get("errors") 352 or value.get("failures") 353 or value.get("failure") 354 or value.get("detail") 355 ) 356 return _try_get_error(new_value) 357 return None 358 359 try: 360 body = response.json() 361 return _try_get_error(body) 362 except requests.exceptions.JSONDecodeError: 363 return None 364 365 def get_error_display_message(self, exception: BaseException) -> Optional[str]: 366 """ 367 Retrieves the user-friendly display message that corresponds to an exception. 368 This will be called when encountering an exception while reading records from the stream, and used to build the AirbyteTraceMessage. 369 370 The default implementation of this method only handles HTTPErrors by passing the response to self.parse_response_error_message(). 371 The method should be overriden as needed to handle any additional exception types. 372 373 :param exception: The exception that was raised 374 :return: A user-friendly message that indicates the cause of the error 375 """ 376 if isinstance(exception, requests.HTTPError) and exception.response is not None: 377 return self.parse_response_error_message(exception.response) 378 return None 379 380 def read_records( 381 self, 382 sync_mode: SyncMode, 383 cursor_field: Optional[List[str]] = None, 384 stream_slice: Optional[Mapping[str, Any]] = None, 385 stream_state: Optional[Mapping[str, Any]] = None, 386 ) -> Iterable[StreamData]: 387 # A cursor_field indicates this is an incremental stream which offers better checkpointing than RFR enabled via the cursor 388 if self.cursor_field or not isinstance(self.get_cursor(), ResumableFullRefreshCursor): 389 yield from self._read_pages( 390 lambda req, res, state, _slice: self.parse_response( 391 res, stream_slice=_slice, stream_state=state 392 ), 393 stream_slice, 394 stream_state, 395 ) 396 else: 397 yield from self._read_single_page( 398 lambda req, res, state, _slice: self.parse_response( 399 res, stream_slice=_slice, stream_state=state 400 ), 401 stream_slice, 402 stream_state, 403 ) 404 405 @property 406 def state(self) -> MutableMapping[str, Any]: 407 cursor = self.get_cursor() 408 if cursor: 409 return cursor.get_stream_state() # type: ignore 410 return self._state 411 412 @state.setter 413 def state(self, value: MutableMapping[str, Any]) -> None: 414 cursor = self.get_cursor() 415 if cursor: 416 cursor.set_initial_state(value) 417 self._state = value 418 419 def get_cursor(self) -> Optional[Cursor]: 420 # I don't love that this is semi-stateful but not sure what else to do. We don't know exactly what type of cursor to 421 # instantiate when creating the class. We can make a few assumptions like if there is a cursor_field which implies 422 # incremental, but we don't know until runtime if this is a substream. Ideally, a stream should explicitly define 423 # its cursor, but because we're trying to automatically apply RFR we're stuck with this logic where we replace the 424 # cursor at runtime once we detect this is a substream based on self.has_multiple_slices being reassigned 425 if self.has_multiple_slices and isinstance(self.cursor, ResumableFullRefreshCursor): 426 self.cursor = SubstreamResumableFullRefreshCursor() 427 return self.cursor 428 else: 429 return self.cursor 430 431 def _read_pages( 432 self, 433 records_generator_fn: Callable[ 434 [ 435 requests.PreparedRequest, 436 requests.Response, 437 Mapping[str, Any], 438 Optional[Mapping[str, Any]], 439 ], 440 Iterable[StreamData], 441 ], 442 stream_slice: Optional[Mapping[str, Any]] = None, 443 stream_state: Optional[Mapping[str, Any]] = None, 444 ) -> Iterable[StreamData]: 445 stream_state = stream_state or {} 446 pagination_complete = False 447 next_page_token = None 448 while not pagination_complete: 449 request, response = self._fetch_next_page(stream_slice, stream_state, next_page_token) 450 yield from records_generator_fn(request, response, stream_state, stream_slice) 451 452 next_page_token = self.next_page_token(response) 453 if not next_page_token: 454 pagination_complete = True 455 456 cursor = self.get_cursor() 457 if cursor and isinstance(cursor, SubstreamResumableFullRefreshCursor): 458 partition, _, _ = self._extract_slice_fields(stream_slice=stream_slice) 459 # Substreams checkpoint state by marking an entire parent partition as completed so that on the subsequent attempt 460 # after a failure, completed parents are skipped and the sync can make progress 461 cursor.close_slice(StreamSlice(cursor_slice={}, partition=partition)) 462 463 # Always return an empty generator just in case no records were ever yielded 464 yield from [] 465 466 def _read_single_page( 467 self, 468 records_generator_fn: Callable[ 469 [ 470 requests.PreparedRequest, 471 requests.Response, 472 Mapping[str, Any], 473 Optional[Mapping[str, Any]], 474 ], 475 Iterable[StreamData], 476 ], 477 stream_slice: Optional[Mapping[str, Any]] = None, 478 stream_state: Optional[Mapping[str, Any]] = None, 479 ) -> Iterable[StreamData]: 480 partition, cursor_slice, remaining_slice = self._extract_slice_fields( 481 stream_slice=stream_slice 482 ) 483 stream_state = stream_state or {} 484 next_page_token = cursor_slice or None 485 486 request, response = self._fetch_next_page(remaining_slice, stream_state, next_page_token) 487 yield from records_generator_fn(request, response, stream_state, remaining_slice) 488 489 next_page_token = self.next_page_token(response) or { 490 "__ab_full_refresh_sync_complete": True 491 } 492 493 cursor = self.get_cursor() 494 if cursor: 495 cursor.close_slice(StreamSlice(cursor_slice=next_page_token, partition=partition)) 496 497 # Always return an empty generator just in case no records were ever yielded 498 yield from [] 499 500 @staticmethod 501 def _extract_slice_fields( 502 stream_slice: Optional[Mapping[str, Any]], 503 ) -> tuple[Mapping[str, Any], Mapping[str, Any], Mapping[str, Any]]: 504 if not stream_slice: 505 return {}, {}, {} 506 507 if isinstance(stream_slice, StreamSlice): 508 partition = stream_slice.partition 509 cursor_slice = stream_slice.cursor_slice 510 remaining = {k: v for k, v in stream_slice.items()} 511 else: 512 # RFR streams that implement stream_slices() to generate stream slices in the legacy mapping format are converted into a 513 # structured stream slice mapping by the LegacyCursorBasedCheckpointReader. The structured mapping object has separate 514 # fields for the partition and cursor_slice value 515 partition = stream_slice.get("partition", {}) 516 cursor_slice = stream_slice.get("cursor_slice", {}) 517 remaining = { 518 key: val 519 for key, val in stream_slice.items() 520 if key != "partition" and key != "cursor_slice" 521 } 522 return partition, cursor_slice, remaining 523 524 def _fetch_next_page( 525 self, 526 stream_slice: Optional[Mapping[str, Any]] = None, 527 stream_state: Optional[Mapping[str, Any]] = None, 528 next_page_token: Optional[Mapping[str, Any]] = None, 529 ) -> Tuple[requests.PreparedRequest, requests.Response]: 530 request, response = self._http_client.send_request( 531 http_method=self.http_method, 532 url=self._join_url( 533 self.url_base, 534 self.path( 535 stream_state=stream_state, 536 stream_slice=stream_slice, 537 next_page_token=next_page_token, 538 ), 539 ), 540 request_kwargs=self.request_kwargs( 541 stream_state=stream_state, 542 stream_slice=stream_slice, 543 next_page_token=next_page_token, 544 ), 545 headers=self.request_headers( 546 stream_state=stream_state, 547 stream_slice=stream_slice, 548 next_page_token=next_page_token, 549 ), 550 params=self.request_params( 551 stream_state=stream_state, 552 stream_slice=stream_slice, 553 next_page_token=next_page_token, 554 ), 555 json=self.request_body_json( 556 stream_state=stream_state, 557 stream_slice=stream_slice, 558 next_page_token=next_page_token, 559 ), 560 data=self.request_body_data( 561 stream_state=stream_state, 562 stream_slice=stream_slice, 563 next_page_token=next_page_token, 564 ), 565 dedupe_query_params=True, 566 log_formatter=self.get_log_formatter(), 567 exit_on_rate_limit=self.exit_on_rate_limit, 568 ) 569 570 return request, response 571 572 def get_log_formatter(self) -> Optional[Callable[[requests.Response], Any]]: 573 """ 574 575 :return Optional[Callable[[requests.Response], Any]]: Function that will be used in logging inside HttpClient 576 """ 577 return None
Base abstract class for an Airbyte Stream using the HTTP protocol. Basic building block for users building an Airbyte source for a HTTP API.
84 @property 85 def exit_on_rate_limit(self) -> bool: 86 """ 87 :return: False if the stream will retry endlessly when rate limited 88 """ 89 return self._exit_on_rate_limit
Returns
False if the stream will retry endlessly when rate limited
95 @property 96 def cache_filename(self) -> str: 97 """ 98 Override if needed. Return the name of cache file 99 Note that if the environment variable REQUEST_CACHE_PATH is not set, the cache will be in-memory only. 100 """ 101 return f"{self.name}.sqlite"
Override if needed. Return the name of cache file Note that if the environment variable REQUEST_CACHE_PATH is not set, the cache will be in-memory only.
103 @property 104 def use_cache(self) -> bool: 105 """ 106 Override if needed. If True, all records will be cached. 107 Note that if the environment variable REQUEST_CACHE_PATH is not set, the cache will be in-memory only. 108 """ 109 return False
Override if needed. If True, all records will be cached. Note that if the environment variable REQUEST_CACHE_PATH is not set, the cache will be in-memory only.
111 @property 112 def use_tcp_keepalive(self) -> bool: 113 """ 114 Override if needed. If True, the stream's HTTP session enables TCP keepalive (60 s idle, 10 s interval, 115 6 probes), so a dropped connection fails in about two minutes instead of blocking a read forever. 116 """ 117 return False
Override if needed. If True, the stream's HTTP session enables TCP keepalive (60 s idle, 10 s interval, 6 probes), so a dropped connection fails in about two minutes instead of blocking a read forever.
119 @property 120 def request_timeout(self) -> Optional[Union[float, Tuple[Optional[float], Optional[float]]]]: 121 """ 122 Override if needed. Timeout in seconds for every request this stream sends, as a number or a 123 (connect, read) pair. A "timeout" returned by request_kwargs() takes precedence. None means no timeout. 124 """ 125 return None
Override if needed. Timeout in seconds for every request this stream sends, as a number or a (connect, read) pair. A "timeout" returned by request_kwargs() takes precedence. None means no timeout.
127 @property 128 @abstractmethod 129 def url_base(self) -> str: 130 """ 131 :return: URL base for the API endpoint e.g: if you wanted to hit https://myapi.com/v1/some_entity then this should return "https://myapi.com/v1/" 132 """
Returns
URL base for the API endpoint e.g: if you wanted to hit https://myapi.com/v1/some_entity then this should return "https://myapi.com/v1/"
134 @property 135 def http_method(self) -> str: 136 """ 137 Override if needed. See get_request_data/get_request_json if using POST/PUT/PATCH. 138 """ 139 return "GET"
Override if needed. See get_request_data/get_request_json if using POST/PUT/PATCH.
141 @property 142 @deprecated( 143 "Deprecated as of CDK version 3.0.0. " 144 "You should set error_handler explicitly in HttpStream.get_error_handler() instead." 145 ) 146 def raise_on_http_errors(self) -> bool: 147 """ 148 Override if needed. If set to False, allows opting-out of raising HTTP code exception. 149 """ 150 return True
Override if needed. If set to False, allows opting-out of raising HTTP code exception.
152 @property 153 @deprecated( 154 "Deprecated as of CDK version 3.0.0. " 155 "You should set backoff_strategies explicitly in HttpStream.get_backoff_strategy() instead." 156 ) 157 def max_retries(self) -> Union[int, None]: 158 """ 159 Override if needed. Specifies maximum amount of retries for backoff policy. Return None for no limit. 160 """ 161 return 5
Override if needed. Specifies maximum amount of retries for backoff policy. Return None for no limit.
163 @property 164 @deprecated( 165 "Deprecated as of CDK version 3.0.0. " 166 "You should set backoff_strategies explicitly in HttpStream.get_backoff_strategy() instead." 167 ) 168 def max_time(self) -> Union[int, None]: 169 """ 170 Override if needed. Specifies maximum total waiting time (in seconds) for backoff policy. Return None for no limit. 171 """ 172 return 60 * 10
Override if needed. Specifies maximum total waiting time (in seconds) for backoff policy. Return None for no limit.
174 @property 175 @deprecated( 176 "Deprecated as of CDK version 3.0.0. " 177 "You should set backoff_strategies explicitly in HttpStream.get_backoff_strategy() instead." 178 ) 179 def retry_factor(self) -> float: 180 """ 181 Override if needed. Specifies factor for backoff policy. 182 """ 183 return 5
Override if needed. Specifies factor for backoff policy.
185 @abstractmethod 186 def next_page_token(self, response: requests.Response) -> Optional[Mapping[str, Any]]: 187 """ 188 Override this method to define a pagination strategy. 189 190 The value returned from this method is passed to most other methods in this class. Use it to form a request e.g: set headers or query params. 191 192 :return: The token for the next page from the input response object. Returning None means there are no more pages to read in this response. 193 """
Override this method to define a pagination strategy.
The value returned from this method is passed to most other methods in this class. Use it to form a request e.g: set headers or query params.
Returns
The token for the next page from the input response object. Returning None means there are no more pages to read in this response.
195 @abstractmethod 196 def path( 197 self, 198 *, 199 stream_state: Optional[Mapping[str, Any]] = None, 200 stream_slice: Optional[Mapping[str, Any]] = None, 201 next_page_token: Optional[Mapping[str, Any]] = None, 202 ) -> str: 203 """ 204 Returns the URL path for the API endpoint e.g: if you wanted to hit https://myapi.com/v1/some_entity then this should return "some_entity" 205 """
Returns the URL path for the API endpoint e.g: if you wanted to hit https://myapi.com/v1/some_entity then this should return "some_entity"
207 def request_params( 208 self, 209 stream_state: Optional[Mapping[str, Any]], 210 stream_slice: Optional[Mapping[str, Any]] = None, 211 next_page_token: Optional[Mapping[str, Any]] = None, 212 ) -> MutableMapping[str, Any]: 213 """ 214 Override this method to define the query parameters that should be set on an outgoing HTTP request given the inputs. 215 216 E.g: you might want to define query parameters for paging if next_page_token is not None. 217 """ 218 return {}
Override this method to define the query parameters that should be set on an outgoing HTTP request given the inputs.
E.g: you might want to define query parameters for paging if next_page_token is not None.
220 def request_headers( 221 self, 222 stream_state: Optional[Mapping[str, Any]], 223 stream_slice: Optional[Mapping[str, Any]] = None, 224 next_page_token: Optional[Mapping[str, Any]] = None, 225 ) -> Mapping[str, Any]: 226 """ 227 Override to return any non-auth headers. Authentication headers will overwrite any overlapping headers returned from this method. 228 """ 229 return {}
Override to return any non-auth headers. Authentication headers will overwrite any overlapping headers returned from this method.
231 def request_body_data( 232 self, 233 stream_state: Optional[Mapping[str, Any]], 234 stream_slice: Optional[Mapping[str, Any]] = None, 235 next_page_token: Optional[Mapping[str, Any]] = None, 236 ) -> Optional[Union[Mapping[str, Any], str]]: 237 """ 238 Override when creating POST/PUT/PATCH requests to populate the body of the request with a non-JSON payload. 239 240 If returns a ready text that it will be sent as is. 241 If returns a dict that it will be converted to a urlencoded form. 242 E.g. {"key1": "value1", "key2": "value2"} => "key1=value1&key2=value2" 243 244 At the same time only one of the 'request_body_data' and 'request_body_json' functions can be overridden. 245 """ 246 return None
Override when creating POST/PUT/PATCH requests to populate the body of the request with a non-JSON payload.
If returns a ready text that it will be sent as is. If returns a dict that it will be converted to a urlencoded form. E.g. {"key1": "value1", "key2": "value2"} => "key1=value1&key2=value2"
At the same time only one of the 'request_body_data' and 'request_body_json' functions can be overridden.
248 def request_body_json( 249 self, 250 stream_state: Optional[Mapping[str, Any]], 251 stream_slice: Optional[Mapping[str, Any]] = None, 252 next_page_token: Optional[Mapping[str, Any]] = None, 253 ) -> Optional[Mapping[str, Any]]: 254 """ 255 Override when creating POST/PUT/PATCH requests to populate the body of the request with a JSON payload. 256 257 At the same time only one of the 'request_body_data' and 'request_body_json' functions can be overridden. 258 """ 259 return None
Override when creating POST/PUT/PATCH requests to populate the body of the request with a JSON payload.
At the same time only one of the 'request_body_data' and 'request_body_json' functions can be overridden.
261 def request_kwargs( 262 self, 263 stream_state: Optional[Mapping[str, Any]], 264 stream_slice: Optional[Mapping[str, Any]] = None, 265 next_page_token: Optional[Mapping[str, Any]] = None, 266 ) -> Mapping[str, Any]: 267 """ 268 Override to return a mapping of keyword arguments to be used when creating the HTTP request. 269 Any option listed in https://docs.python-requests.org/en/latest/api/#requests.adapters.BaseAdapter.send for can be returned from 270 this method. Note that these options do not conflict with request-level options such as headers, request params, etc.. 271 """ 272 return {}
Override to return a mapping of keyword arguments to be used when creating the HTTP request. Any option listed in https://docs.python-requests.org/en/latest/api/#requests.adapters.BaseAdapter.send for can be returned from this method. Note that these options do not conflict with request-level options such as headers, request params, etc..
274 @abstractmethod 275 def parse_response( 276 self, 277 response: requests.Response, 278 *, 279 stream_state: Mapping[str, Any], 280 stream_slice: Optional[Mapping[str, Any]] = None, 281 next_page_token: Optional[Mapping[str, Any]] = None, 282 ) -> Iterable[Mapping[str, Any]]: 283 """ 284 Parses the raw response object into a list of records. 285 By default, this returns an iterable containing the input. Override to parse differently. 286 :param response: 287 :param stream_state: 288 :param stream_slice: 289 :param next_page_token: 290 :return: An iterable containing the parsed response 291 """
Parses the raw response object into a list of records. By default, this returns an iterable containing the input. Override to parse differently.
Parameters
- response:
- stream_state:
- stream_slice:
- next_page_token:
Returns
An iterable containing the parsed response
293 def get_backoff_strategy(self) -> Optional[Union[BackoffStrategy, List[BackoffStrategy]]]: 294 """ 295 Used to initialize Adapter to avoid breaking changes. 296 If Stream has a `backoff_time` method implementation, we know this stream uses old (pre-HTTPClient) backoff handlers and thus an adapter is needed. 297 298 Override to provide custom BackoffStrategy 299 :return Optional[BackoffStrategy]: 300 """ 301 if hasattr(self, "backoff_time"): 302 return HttpStreamAdapterBackoffStrategy(self) 303 else: 304 return None
Used to initialize Adapter to avoid breaking changes.
If Stream has a backoff_time method implementation, we know this stream uses old (pre-HTTPClient) backoff handlers and thus an adapter is needed.
Override to provide custom BackoffStrategy
Returns
306 def get_error_handler(self) -> Optional[ErrorHandler]: 307 """ 308 Used to initialize Adapter to avoid breaking changes. 309 If Stream has a `should_retry` method implementation, we know this stream uses old (pre-HTTPClient) error handlers and thus an adapter is needed. 310 311 Override to provide custom ErrorHandler 312 :return Optional[ErrorHandler]: 313 """ 314 if hasattr(self, "should_retry"): 315 error_handler = HttpStreamAdapterHttpStatusErrorHandler( 316 stream=self, 317 logger=logging.getLogger(), 318 max_retries=self.max_retries, 319 max_time=timedelta(seconds=self.max_time or 0), 320 ) 321 return error_handler 322 else: 323 return None
Used to initialize Adapter to avoid breaking changes.
If Stream has a should_retry method implementation, we know this stream uses old (pre-HTTPClient) error handlers and thus an adapter is needed.
Override to provide custom ErrorHandler
Returns
329 @classmethod 330 def parse_response_error_message(cls, response: requests.Response) -> Optional[str]: 331 """ 332 Parses the raw response object from a failed request into a user-friendly error message. 333 By default, this method tries to grab the error message from JSON responses by following common API patterns. Override to parse differently. 334 335 :param response: 336 :return: A user-friendly message that indicates the cause of the error 337 """ 338 339 # default logic to grab error from common fields 340 def _try_get_error(value: Optional[JsonType]) -> Optional[str]: 341 if isinstance(value, str): 342 return value 343 elif isinstance(value, list): 344 errors_in_value = [_try_get_error(v) for v in value] 345 return ", ".join(v for v in errors_in_value if v is not None) 346 elif isinstance(value, dict): 347 new_value = ( 348 value.get("message") 349 or value.get("messages") 350 or value.get("error") 351 or value.get("errors") 352 or value.get("failures") 353 or value.get("failure") 354 or value.get("detail") 355 ) 356 return _try_get_error(new_value) 357 return None 358 359 try: 360 body = response.json() 361 return _try_get_error(body) 362 except requests.exceptions.JSONDecodeError: 363 return None
Parses the raw response object from a failed request into a user-friendly error message. By default, this method tries to grab the error message from JSON responses by following common API patterns. Override to parse differently.
Parameters
- response:
Returns
A user-friendly message that indicates the cause of the error
365 def get_error_display_message(self, exception: BaseException) -> Optional[str]: 366 """ 367 Retrieves the user-friendly display message that corresponds to an exception. 368 This will be called when encountering an exception while reading records from the stream, and used to build the AirbyteTraceMessage. 369 370 The default implementation of this method only handles HTTPErrors by passing the response to self.parse_response_error_message(). 371 The method should be overriden as needed to handle any additional exception types. 372 373 :param exception: The exception that was raised 374 :return: A user-friendly message that indicates the cause of the error 375 """ 376 if isinstance(exception, requests.HTTPError) and exception.response is not None: 377 return self.parse_response_error_message(exception.response) 378 return None
Retrieves the user-friendly display message that corresponds to an exception. This will be called when encountering an exception while reading records from the stream, and used to build the AirbyteTraceMessage.
The default implementation of this method only handles HTTPErrors by passing the response to self.parse_response_error_message(). The method should be overriden as needed to handle any additional exception types.
Parameters
- exception: The exception that was raised
Returns
A user-friendly message that indicates the cause of the error
380 def read_records( 381 self, 382 sync_mode: SyncMode, 383 cursor_field: Optional[List[str]] = None, 384 stream_slice: Optional[Mapping[str, Any]] = None, 385 stream_state: Optional[Mapping[str, Any]] = None, 386 ) -> Iterable[StreamData]: 387 # A cursor_field indicates this is an incremental stream which offers better checkpointing than RFR enabled via the cursor 388 if self.cursor_field or not isinstance(self.get_cursor(), ResumableFullRefreshCursor): 389 yield from self._read_pages( 390 lambda req, res, state, _slice: self.parse_response( 391 res, stream_slice=_slice, stream_state=state 392 ), 393 stream_slice, 394 stream_state, 395 ) 396 else: 397 yield from self._read_single_page( 398 lambda req, res, state, _slice: self.parse_response( 399 res, stream_slice=_slice, stream_state=state 400 ), 401 stream_slice, 402 stream_state, 403 )
This method should be overridden by subclasses to read records based on the inputs
405 @property 406 def state(self) -> MutableMapping[str, Any]: 407 cursor = self.get_cursor() 408 if cursor: 409 return cursor.get_stream_state() # type: ignore 410 return self._state
State getter, should return state in form that can serialized to a string and send to the output as a STATE AirbyteMessage.
A good example of a state is a cursor_value: { self.cursor_field: "cursor_value" }
State should try to be as small as possible but at the same time descriptive enough to restore syncing process from the point where it stopped.
419 def get_cursor(self) -> Optional[Cursor]: 420 # I don't love that this is semi-stateful but not sure what else to do. We don't know exactly what type of cursor to 421 # instantiate when creating the class. We can make a few assumptions like if there is a cursor_field which implies 422 # incremental, but we don't know until runtime if this is a substream. Ideally, a stream should explicitly define 423 # its cursor, but because we're trying to automatically apply RFR we're stuck with this logic where we replace the 424 # cursor at runtime once we detect this is a substream based on self.has_multiple_slices being reassigned 425 if self.has_multiple_slices and isinstance(self.cursor, ResumableFullRefreshCursor): 426 self.cursor = SubstreamResumableFullRefreshCursor() 427 return self.cursor 428 else: 429 return self.cursor
A Cursor is an interface that a stream can implement to manage how its internal state is read and updated while reading records. Historically, Python connectors had no concept of a cursor to manage state. Python streams need to define a cursor implementation and override this method to manage state through a Cursor.
572 def get_log_formatter(self) -> Optional[Callable[[requests.Response], Any]]: 573 """ 574 575 :return Optional[Callable[[requests.Response], Any]]: Function that will be used in logging inside HttpClient 576 """ 577 return None
Returns
Function that will be used in logging inside HttpClient
Inherited Members
580class HttpSubStream(HttpStream, ABC): 581 def __init__(self, parent: HttpStream, **kwargs: Any): 582 """ 583 :param parent: should be the instance of HttpStream class 584 """ 585 super().__init__(**kwargs) 586 self.parent = parent 587 self.has_multiple_slices = ( 588 True # Substreams are based on parent records which implies there are multiple slices 589 ) 590 591 # There are three conditions that dictate if RFR should automatically be applied to a stream 592 # 1. Streams that explicitly initialize their own cursor should defer to it and not automatically apply RFR 593 # 2. Streams with at least one cursor_field are incremental and thus a superior sync to RFR. 594 # 3. Streams overriding read_records() do not guarantee that they will call the parent implementation which can perform 595 # per-page checkpointing so RFR is only supported if a stream use the default `HttpStream.read_records()` method 596 if ( 597 not self.cursor 598 and len(self.cursor_field) == 0 599 and type(self).read_records is HttpStream.read_records 600 ): 601 self.cursor = SubstreamResumableFullRefreshCursor() 602 603 def stream_slices( 604 self, 605 sync_mode: SyncMode, 606 cursor_field: Optional[List[str]] = None, 607 stream_state: Optional[Mapping[str, Any]] = None, 608 ) -> Iterable[Optional[Mapping[str, Any]]]: 609 # read_stateless() assumes the parent is not concurrent. This is currently okay since the concurrent CDK does 610 # not support either substreams or RFR, but something that needs to be considered once we do 611 for parent_record in self.parent.read_only_records(stream_state): 612 # Skip non-records (eg AirbyteLogMessage) 613 if isinstance(parent_record, AirbyteMessage): 614 if parent_record.type == MessageType.RECORD: 615 parent_record = parent_record.record.data # type: ignore [assignment, union-attr] # Incorrect type for assignment 616 else: 617 continue 618 elif isinstance(parent_record, Record): 619 parent_record = parent_record.data 620 yield {"parent": parent_record}
Base abstract class for an Airbyte Stream using the HTTP protocol. Basic building block for users building an Airbyte source for a HTTP API.
581 def __init__(self, parent: HttpStream, **kwargs: Any): 582 """ 583 :param parent: should be the instance of HttpStream class 584 """ 585 super().__init__(**kwargs) 586 self.parent = parent 587 self.has_multiple_slices = ( 588 True # Substreams are based on parent records which implies there are multiple slices 589 ) 590 591 # There are three conditions that dictate if RFR should automatically be applied to a stream 592 # 1. Streams that explicitly initialize their own cursor should defer to it and not automatically apply RFR 593 # 2. Streams with at least one cursor_field are incremental and thus a superior sync to RFR. 594 # 3. Streams overriding read_records() do not guarantee that they will call the parent implementation which can perform 595 # per-page checkpointing so RFR is only supported if a stream use the default `HttpStream.read_records()` method 596 if ( 597 not self.cursor 598 and len(self.cursor_field) == 0 599 and type(self).read_records is HttpStream.read_records 600 ): 601 self.cursor = SubstreamResumableFullRefreshCursor()
Parameters
- parent: should be the instance of HttpStream class
603 def stream_slices( 604 self, 605 sync_mode: SyncMode, 606 cursor_field: Optional[List[str]] = None, 607 stream_state: Optional[Mapping[str, Any]] = None, 608 ) -> Iterable[Optional[Mapping[str, Any]]]: 609 # read_stateless() assumes the parent is not concurrent. This is currently okay since the concurrent CDK does 610 # not support either substreams or RFR, but something that needs to be considered once we do 611 for parent_record in self.parent.read_only_records(stream_state): 612 # Skip non-records (eg AirbyteLogMessage) 613 if isinstance(parent_record, AirbyteMessage): 614 if parent_record.type == MessageType.RECORD: 615 parent_record = parent_record.record.data # type: ignore [assignment, union-attr] # Incorrect type for assignment 616 else: 617 continue 618 elif isinstance(parent_record, Record): 619 parent_record = parent_record.data 620 yield {"parent": parent_record}
Override to define the slices for this stream. See the stream slicing section of the docs for more information.
Parameters
- sync_mode:
- cursor_field:
- stream_state:
Returns
Inherited Members
- HttpStream
- source_defined_cursor
- page_size
- exit_on_rate_limit
- cache_filename
- use_cache
- use_tcp_keepalive
- request_timeout
- url_base
- http_method
- raise_on_http_errors
- max_retries
- max_time
- retry_factor
- next_page_token
- path
- request_params
- request_headers
- request_body_data
- request_body_json
- request_kwargs
- parse_response
- get_backoff_strategy
- get_error_handler
- parse_response_error_message
- get_error_display_message
- read_records
- state
- get_cursor
- get_log_formatter
750class LimiterSession(LimiterMixin, requests.Session): 751 """Session that adds rate-limiting behavior to requests."""
Session that adds rate-limiting behavior to requests.
Inherited Members
425class MovingWindowCallRatePolicy(BaseCallRatePolicy): 426 """ 427 Policy to control requests rate implemented on top of PyRateLimiter lib. 428 The main difference between this policy and FixedWindowCallRatePolicy is that the rate-limiting window 429 is moving along requests that we made, and there is no moment when we reset an available number of calls. 430 This strategy requires saving of timestamps of all requests within a window. 431 """ 432 433 def __init__(self, rates: list[Rate], matchers: list[RequestMatcher]): 434 """Constructor 435 436 :param rates: list of rates, the order is important and must be ascending 437 :param matchers: 438 """ 439 if not rates: 440 raise ValueError("The list of rates can not be empty") 441 pyrate_rates = [ 442 PyRateRate(limit=rate.limit, interval=int(rate.interval.total_seconds() * 1000)) 443 for rate in rates 444 ] 445 self._bucket = InMemoryBucket(pyrate_rates) 446 # Limiter will create the background task that clears old requests in the bucket 447 self._limiter = Limiter(self._bucket) 448 super().__init__(matchers=matchers) 449 450 def try_acquire(self, request: Any, weight: int) -> None: 451 if not self.matches(request): 452 raise ValueError("Request does not match the policy") 453 lowest_limit = min(rate.limit for rate in self._bucket.rates) 454 if weight > lowest_limit: 455 raise ValueError( 456 f"Weight can not exceed the lowest configured rate limit ({lowest_limit})" 457 ) 458 459 try: 460 self._limiter.try_acquire(request, weight=weight) 461 except BucketFullException as exc: 462 item = self._limiter.bucket_factory.wrap_item(request, weight) 463 assert isinstance(item, RateItem) 464 465 with self._limiter.lock: 466 time_to_wait = self._bucket.waiting(item) 467 assert isinstance(time_to_wait, int) 468 469 raise CallRateLimitHit( 470 error=str(exc.meta_info["error"]), 471 item=request, 472 weight=int(exc.meta_info["weight"]), 473 rate=str(exc.meta_info["rate"]), 474 time_to_wait=timedelta(milliseconds=time_to_wait), 475 ) 476 477 def update( 478 self, available_calls: Optional[int], call_reset_ts: Optional[datetime.datetime] 479 ) -> None: 480 """Adjust call bucket to reflect the state of the API server 481 482 :param available_calls: 483 :param call_reset_ts: 484 :return: 485 """ 486 if ( 487 available_calls is not None and call_reset_ts is None 488 ): # we do our best to sync buckets with API 489 if available_calls == 0: 490 with self._limiter.lock: 491 now: int = TimeClock().now() # type: ignore[no-untyped-call] 492 items_to_add = self._get_available_calls(now) 493 if items_to_add > 0: 494 added = self._bucket.put( 495 RateItem(name="dummy", timestamp=now, weight=items_to_add) 496 ) 497 if not added: 498 logger.warning( 499 "Failed to sync rate limit bucket with API: could not add %s items", 500 items_to_add, 501 ) 502 # TODO: add support if needed, it might be that it is not possible to make a good solution for this case 503 # if available_calls is not None and call_reset_ts is not None: 504 # ts = call_reset_ts.timestamp() 505 506 def _get_available_calls(self, now: int) -> int: 507 """Return the number of calls still allowed by the strictest configured rate at the given timestamp (ms).""" 508 return min( 509 rate.limit 510 - sum(1 for item in self._bucket.items if item.timestamp >= now - rate.interval) 511 for rate in self._bucket.rates 512 ) 513 514 def __str__(self) -> str: 515 """Return a human-friendly description of the moving window rate policy for logging purposes.""" 516 rates_info = ", ".join( 517 f"{rate.limit} per {timedelta(milliseconds=rate.interval)}" 518 for rate in self._bucket.rates 519 ) 520 current_bucket_count = self._bucket.count() 521 matcher_str = ", ".join(f"{matcher}" for matcher in self._matchers) 522 return ( 523 f"MovingWindowCallRatePolicy(rates=[{rates_info}], current_bucket_count={current_bucket_count}, " 524 f"matchers=[{matcher_str}])" 525 )
Policy to control requests rate implemented on top of PyRateLimiter lib. The main difference between this policy and FixedWindowCallRatePolicy is that the rate-limiting window is moving along requests that we made, and there is no moment when we reset an available number of calls. This strategy requires saving of timestamps of all requests within a window.
433 def __init__(self, rates: list[Rate], matchers: list[RequestMatcher]): 434 """Constructor 435 436 :param rates: list of rates, the order is important and must be ascending 437 :param matchers: 438 """ 439 if not rates: 440 raise ValueError("The list of rates can not be empty") 441 pyrate_rates = [ 442 PyRateRate(limit=rate.limit, interval=int(rate.interval.total_seconds() * 1000)) 443 for rate in rates 444 ] 445 self._bucket = InMemoryBucket(pyrate_rates) 446 # Limiter will create the background task that clears old requests in the bucket 447 self._limiter = Limiter(self._bucket) 448 super().__init__(matchers=matchers)
Constructor
Parameters
- rates: list of rates, the order is important and must be ascending
- matchers:
450 def try_acquire(self, request: Any, weight: int) -> None: 451 if not self.matches(request): 452 raise ValueError("Request does not match the policy") 453 lowest_limit = min(rate.limit for rate in self._bucket.rates) 454 if weight > lowest_limit: 455 raise ValueError( 456 f"Weight can not exceed the lowest configured rate limit ({lowest_limit})" 457 ) 458 459 try: 460 self._limiter.try_acquire(request, weight=weight) 461 except BucketFullException as exc: 462 item = self._limiter.bucket_factory.wrap_item(request, weight) 463 assert isinstance(item, RateItem) 464 465 with self._limiter.lock: 466 time_to_wait = self._bucket.waiting(item) 467 assert isinstance(time_to_wait, int) 468 469 raise CallRateLimitHit( 470 error=str(exc.meta_info["error"]), 471 item=request, 472 weight=int(exc.meta_info["weight"]), 473 rate=str(exc.meta_info["rate"]), 474 time_to_wait=timedelta(milliseconds=time_to_wait), 475 )
Try to acquire request
Parameters
- request: a request object representing a single call to API
- weight: number of requests to deduct from credit
Returns
477 def update( 478 self, available_calls: Optional[int], call_reset_ts: Optional[datetime.datetime] 479 ) -> None: 480 """Adjust call bucket to reflect the state of the API server 481 482 :param available_calls: 483 :param call_reset_ts: 484 :return: 485 """ 486 if ( 487 available_calls is not None and call_reset_ts is None 488 ): # we do our best to sync buckets with API 489 if available_calls == 0: 490 with self._limiter.lock: 491 now: int = TimeClock().now() # type: ignore[no-untyped-call] 492 items_to_add = self._get_available_calls(now) 493 if items_to_add > 0: 494 added = self._bucket.put( 495 RateItem(name="dummy", timestamp=now, weight=items_to_add) 496 ) 497 if not added: 498 logger.warning( 499 "Failed to sync rate limit bucket with API: could not add %s items", 500 items_to_add, 501 ) 502 # TODO: add support if needed, it might be that it is not possible to make a good solution for this case 503 # if available_calls is not None and call_reset_ts is not None: 504 # ts = call_reset_ts.timestamp()
Adjust call bucket to reflect the state of the API server
Parameters
- available_calls:
- call_reset_ts:
Returns
Inherited Members
27class Oauth2Authenticator(AbstractOauth2Authenticator): 28 """ 29 Generates OAuth2.0 access tokens from an OAuth2.0 refresh token and client credentials. 30 The generated access token is attached to each request via the Authorization header. 31 If a connector_config is provided any mutation of it's value in the scope of this class will emit AirbyteControlConnectorConfigMessage. 32 `refresh_token_error_messages` provides fixed user-facing messages for configured refresh token errors. 33 """ 34 35 def __init__( 36 self, 37 token_refresh_endpoint: str, 38 client_id: str, 39 client_secret: str, 40 refresh_token: str, 41 client_id_name: str = "client_id", 42 client_secret_name: str = "client_secret", 43 refresh_token_name: str = "refresh_token", 44 scopes: List[str] | None = None, 45 token_expiry_date: AirbyteDateTime | None = None, 46 token_expiry_date_format: str | None = None, 47 access_token_name: str = "access_token", 48 expires_in_name: str = "expires_in", 49 refresh_request_body: Mapping[str, Any] | None = None, 50 refresh_request_headers: Mapping[str, Any] | None = None, 51 send_refresh_request_as_query_params: bool = False, 52 grant_type_name: str = "grant_type", 53 grant_type: str = "refresh_token", 54 token_expiry_is_time_of_expiration: bool = False, 55 refresh_token_error_status_codes: Tuple[int, ...] = (), 56 refresh_token_error_key: str = "", 57 refresh_token_error_values: Tuple[str, ...] = (), 58 refresh_token_error_messages: Optional[Mapping[str, str]] = None, 59 ) -> None: 60 self._token_refresh_endpoint = token_refresh_endpoint 61 self._client_secret_name = client_secret_name 62 self._client_secret = client_secret 63 self._client_id_name = client_id_name 64 self._client_id = client_id 65 self._refresh_token_name = refresh_token_name 66 self._refresh_token = refresh_token 67 self._scopes = scopes 68 self._access_token_name = access_token_name 69 self._expires_in_name = expires_in_name 70 self._refresh_request_body = refresh_request_body 71 self._refresh_request_headers = refresh_request_headers 72 self._send_refresh_request_as_query_params = send_refresh_request_as_query_params 73 self._grant_type_name = grant_type_name 74 self._grant_type = grant_type 75 76 self._token_expiry_date = token_expiry_date or (ab_datetime_now() - timedelta(days=1)) 77 self._token_expiry_date_format = token_expiry_date_format 78 self._token_expiry_is_time_of_expiration = token_expiry_is_time_of_expiration 79 self._access_token = None 80 super().__init__( 81 refresh_token_error_status_codes, 82 refresh_token_error_key, 83 refresh_token_error_values, 84 refresh_token_error_messages=refresh_token_error_messages, 85 ) 86 87 def get_token_refresh_endpoint(self) -> str: 88 return self._token_refresh_endpoint 89 90 def get_client_id_name(self) -> str: 91 return self._client_id_name 92 93 def get_client_id(self) -> str: 94 return self._client_id 95 96 def get_client_secret_name(self) -> str: 97 return self._client_secret_name 98 99 def get_client_secret(self) -> str: 100 return self._client_secret 101 102 def get_refresh_token_name(self) -> str: 103 return self._refresh_token_name 104 105 def get_refresh_token(self) -> str: 106 return self._refresh_token 107 108 def get_access_token_name(self) -> str: 109 return self._access_token_name 110 111 def get_scopes(self) -> list[str]: 112 return self._scopes # type: ignore[return-value] 113 114 def get_expires_in_name(self) -> str: 115 return self._expires_in_name 116 117 def get_refresh_request_body(self) -> Mapping[str, Any]: 118 return self._refresh_request_body # type: ignore[return-value] 119 120 def get_refresh_request_headers(self) -> Mapping[str, Any]: 121 return self._refresh_request_headers # type: ignore[return-value] 122 123 def should_send_refresh_request_as_query_params(self) -> bool: 124 return self._send_refresh_request_as_query_params 125 126 def get_grant_type_name(self) -> str: 127 return self._grant_type_name 128 129 def get_grant_type(self) -> str: 130 return self._grant_type 131 132 def get_token_expiry_date(self) -> AirbyteDateTime: 133 return self._token_expiry_date 134 135 def set_token_expiry_date(self, value: AirbyteDateTime) -> None: 136 # Overrides of `refresh_access_token` written before 6.45.5 return the raw `expires_in`. 137 if not isinstance(value, datetime): 138 with suppress(ValueError, OverflowError): 139 value = self._parse_token_expiration_date(value) 140 self._token_expiry_date = value 141 142 @property 143 def token_expiry_is_time_of_expiration(self) -> bool: 144 return self._token_expiry_is_time_of_expiration 145 146 @property 147 def token_expiry_date_format(self) -> Optional[str]: 148 return self._token_expiry_date_format 149 150 @property 151 def access_token(self) -> str: 152 return self._access_token # type: ignore[return-value] 153 154 @access_token.setter 155 def access_token(self, value: str) -> None: 156 self._access_token = value # type: ignore[assignment] # Incorrect type for assignment
Generates OAuth2.0 access tokens from an OAuth2.0 refresh token and client credentials.
The generated access token is attached to each request via the Authorization header.
If a connector_config is provided any mutation of it's value in the scope of this class will emit AirbyteControlConnectorConfigMessage.
refresh_token_error_messages provides fixed user-facing messages for configured refresh token errors.
35 def __init__( 36 self, 37 token_refresh_endpoint: str, 38 client_id: str, 39 client_secret: str, 40 refresh_token: str, 41 client_id_name: str = "client_id", 42 client_secret_name: str = "client_secret", 43 refresh_token_name: str = "refresh_token", 44 scopes: List[str] | None = None, 45 token_expiry_date: AirbyteDateTime | None = None, 46 token_expiry_date_format: str | None = None, 47 access_token_name: str = "access_token", 48 expires_in_name: str = "expires_in", 49 refresh_request_body: Mapping[str, Any] | None = None, 50 refresh_request_headers: Mapping[str, Any] | None = None, 51 send_refresh_request_as_query_params: bool = False, 52 grant_type_name: str = "grant_type", 53 grant_type: str = "refresh_token", 54 token_expiry_is_time_of_expiration: bool = False, 55 refresh_token_error_status_codes: Tuple[int, ...] = (), 56 refresh_token_error_key: str = "", 57 refresh_token_error_values: Tuple[str, ...] = (), 58 refresh_token_error_messages: Optional[Mapping[str, str]] = None, 59 ) -> None: 60 self._token_refresh_endpoint = token_refresh_endpoint 61 self._client_secret_name = client_secret_name 62 self._client_secret = client_secret 63 self._client_id_name = client_id_name 64 self._client_id = client_id 65 self._refresh_token_name = refresh_token_name 66 self._refresh_token = refresh_token 67 self._scopes = scopes 68 self._access_token_name = access_token_name 69 self._expires_in_name = expires_in_name 70 self._refresh_request_body = refresh_request_body 71 self._refresh_request_headers = refresh_request_headers 72 self._send_refresh_request_as_query_params = send_refresh_request_as_query_params 73 self._grant_type_name = grant_type_name 74 self._grant_type = grant_type 75 76 self._token_expiry_date = token_expiry_date or (ab_datetime_now() - timedelta(days=1)) 77 self._token_expiry_date_format = token_expiry_date_format 78 self._token_expiry_is_time_of_expiration = token_expiry_is_time_of_expiration 79 self._access_token = None 80 super().__init__( 81 refresh_token_error_status_codes, 82 refresh_token_error_key, 83 refresh_token_error_values, 84 refresh_token_error_messages=refresh_token_error_messages, 85 )
If all of refresh_token_error_status_codes, refresh_token_error_key, and refresh_token_error_values are set, then http errors with such params will be wrapped in AirbyteTracedException. refresh_token_error_messages maps fixed provider error strings to fixed user-facing messages.
117 def get_refresh_request_body(self) -> Mapping[str, Any]: 118 return self._refresh_request_body # type: ignore[return-value]
Returns the request body to set on the refresh request
120 def get_refresh_request_headers(self) -> Mapping[str, Any]: 121 return self._refresh_request_headers # type: ignore[return-value]
Returns the request headers to set on the refresh request
123 def should_send_refresh_request_as_query_params(self) -> bool: 124 return self._send_refresh_request_as_query_params
Returns True if the standard refresh args should be sent on the URL
query string instead of in the request body.
Defaults to False so existing authenticators retain their previous
behavior (params in body, no query params on the refresh URL). Subclasses
can override this to opt into the URL-query-string shape required by OAuth
providers like Gong.
135 def set_token_expiry_date(self, value: AirbyteDateTime) -> None: 136 # Overrides of `refresh_access_token` written before 6.45.5 return the raw `expires_in`. 137 if not isinstance(value, datetime): 138 with suppress(ValueError, OverflowError): 139 value = self._parse_token_expiration_date(value) 140 self._token_expiry_date = value
Setter for access token expiration date
142 @property 143 def token_expiry_is_time_of_expiration(self) -> bool: 144 return self._token_expiry_is_time_of_expiration
Indicates that the Token Expiry returns the date until which the token will be valid, not the amount of time it will be valid.
146 @property 147 def token_expiry_date_format(self) -> Optional[str]: 148 return self._token_expiry_date_format
Format of the datetime; exists it if expires_in is returned as the expiration datetime instead of seconds until it expires
150 @property 151 def access_token(self) -> str: 152 return self._access_token # type: ignore[return-value]
Returns the access token
Inherited Members
33@dataclasses.dataclass 34class Rate: 35 """Call rate limit""" 36 37 limit: int 38 interval: timedelta
Call rate limit
159class SingleUseRefreshTokenOauth2Authenticator(Oauth2Authenticator): 160 """ 161 Authenticator that should be used for API implementing single use refresh tokens: 162 when refreshing access token some API returns a new refresh token that needs to used in the next refresh flow. 163 This authenticator updates the configuration with new refresh token by emitting Airbyte control message from an observed mutation. 164 By default, this authenticator expects a connector config with a "credentials" field with the following nested fields: client_id, 165 client_secret, refresh_token. This behavior can be changed by defining custom config path (using dpath paths) in client_id_config_path, 166 client_secret_config_path, refresh_token_config_path constructor arguments. 167 `refresh_token_error_messages` provides fixed user-facing messages for configured refresh token errors. 168 """ 169 170 def __init__( 171 self, 172 connector_config: Mapping[str, Any], 173 token_refresh_endpoint: str, 174 scopes: List[str] | None = None, 175 access_token_name: str = "access_token", 176 expires_in_name: str = "expires_in", 177 refresh_token_name: str = "refresh_token", 178 refresh_request_body: Mapping[str, Any] | None = None, 179 refresh_request_headers: Mapping[str, Any] | None = None, 180 send_refresh_request_as_query_params: bool = False, 181 grant_type_name: str = "grant_type", 182 grant_type: str = "refresh_token", 183 client_id_name: str = "client_id", 184 client_id: Optional[str] = None, 185 client_secret_name: str = "client_secret", 186 client_secret: Optional[str] = None, 187 access_token_config_path: Sequence[str] = ("credentials", "access_token"), 188 refresh_token_config_path: Sequence[str] = ("credentials", "refresh_token"), 189 token_expiry_date_config_path: Sequence[str] = ("credentials", "token_expiry_date"), 190 token_expiry_date_format: Optional[str] = None, 191 message_repository: MessageRepository = NoopMessageRepository(), 192 token_expiry_is_time_of_expiration: bool = False, 193 refresh_token_error_status_codes: Tuple[int, ...] = (), 194 refresh_token_error_key: str = "", 195 refresh_token_error_values: Tuple[str, ...] = (), 196 emit_control_message_to_message_repository: bool = False, 197 refresh_token_error_messages: Optional[Mapping[str, str]] = None, 198 ) -> None: 199 """ 200 Args: 201 connector_config (Mapping[str, Any]): The full connector configuration 202 token_refresh_endpoint (str): Full URL to the token refresh endpoint 203 scopes (List[str], optional): List of OAuth scopes to pass in the refresh token request body. Defaults to None. 204 access_token_name (str, optional): Name of the access token field, used to parse the refresh token response. Defaults to "access_token". 205 expires_in_name (str, optional): Name of the name of the field that characterizes when the current access token will expire, used to parse the refresh token response. Defaults to "expires_in". 206 refresh_token_name (str, optional): Name of the name of the refresh token field, used to parse the refresh token response. Defaults to "refresh_token". 207 refresh_request_body (Mapping[str, Any], optional): Custom key value pair that will be added to the refresh token request body. Defaults to None. 208 refresh_request_headers (Mapping[str, Any], optional): Custom key value pair that will be added to the refresh token request headers. Defaults to None. 209 send_refresh_request_as_query_params (bool, optional): When True, the standard refresh args (`grant_type`, `refresh_token`, client credentials when not in an `Authorization` header, scopes, plus any `refresh_request_body` extras) are sent on the URL query string and the request body is emitted empty. Use this for OAuth providers like Gong that document their refresh endpoint with refresh args on the URL query string. Defaults to False. 210 grant_type (str, optional): OAuth grant type. Defaults to "refresh_token". 211 client_id (Optional[str]): The client id to authenticate. If not specified, defaults to credentials.client_id in the config object. 212 client_secret (Optional[str]): The client secret to authenticate. If not specified, defaults to credentials.client_secret in the config object. 213 access_token_config_path (Sequence[str]): Dpath to the access_token field in the connector configuration. Defaults to ("credentials", "access_token"). 214 refresh_token_config_path (Sequence[str]): Dpath to the refresh_token field in the connector configuration. Defaults to ("credentials", "refresh_token"). 215 token_expiry_date_config_path (Sequence[str]): Dpath to the token_expiry_date field in the connector configuration. Defaults to ("credentials", "token_expiry_date"). 216 token_expiry_date_format (Optional[str]): Date format of the token expiry date field (set by expires_in_name). If specified the token expiry date is interpreted as time of expiration, as with token_expiry_is_time_of_expiration, otherwise as number of seconds until expiration. 217 token_expiry_is_time_of_expiration bool: set True it if expires_in is returned as time of expiration instead of the number seconds until expiration 218 message_repository (MessageRepository): The message repository used to emit logs on HTTP requests and control message on config update. 219 emit_control_message_to_message_repository (bool): Whether the message repository additionally receives the CONNECTOR_CONFIG message. This is only for in-process consumers such as the Connector Builder, because messages in the repository are otherwise printed to stdout by the entrypoint and would duplicate the direct stdout emission. Defaults to False. 220 refresh_token_error_messages (Optional[Mapping[str, str]]): Fixed user-facing messages for refresh token errors 221 """ 222 self._connector_config = connector_config 223 self._client_id: str = self._get_config_value_by_path( 224 ("credentials", "client_id"), client_id 225 ) 226 self._client_secret: str = self._get_config_value_by_path( 227 ("credentials", "client_secret"), client_secret 228 ) 229 self._client_id_name = client_id_name 230 self._client_secret_name = client_secret_name 231 self._access_token_config_path = access_token_config_path 232 self._refresh_token_config_path = refresh_token_config_path 233 self._token_expiry_date_config_path = token_expiry_date_config_path 234 self._token_expiry_date_format = token_expiry_date_format 235 self._refresh_token_name = refresh_token_name 236 self._grant_type_name = grant_type_name 237 self._connector_config = connector_config 238 self.__message_repository = message_repository 239 self._emit_control_message_to_message_repository = ( 240 emit_control_message_to_message_repository 241 ) 242 super().__init__( 243 token_refresh_endpoint=token_refresh_endpoint, 244 client_id_name=self._client_id_name, 245 client_id=self._client_id, 246 client_secret_name=self._client_secret_name, 247 client_secret=self._client_secret, 248 refresh_token=self.get_refresh_token(), 249 refresh_token_name=self._refresh_token_name, 250 scopes=scopes, 251 token_expiry_date=self.get_token_expiry_date(), 252 access_token_name=access_token_name, 253 expires_in_name=expires_in_name, 254 refresh_request_body=refresh_request_body, 255 refresh_request_headers=refresh_request_headers, 256 send_refresh_request_as_query_params=send_refresh_request_as_query_params, 257 grant_type_name=self._grant_type_name, 258 grant_type=grant_type, 259 token_expiry_date_format=token_expiry_date_format, 260 token_expiry_is_time_of_expiration=token_expiry_is_time_of_expiration 261 or bool(token_expiry_date_format), 262 refresh_token_error_status_codes=refresh_token_error_status_codes, 263 refresh_token_error_key=refresh_token_error_key, 264 refresh_token_error_values=refresh_token_error_values, 265 refresh_token_error_messages=refresh_token_error_messages, 266 ) 267 268 @property 269 def access_token(self) -> str: 270 """ 271 Retrieve the access token from the configuration. 272 273 Returns: 274 str: The access token. 275 """ 276 return self._get_config_value_by_path(self._access_token_config_path) # type: ignore[return-value] 277 278 @access_token.setter 279 def access_token(self, new_access_token: str) -> None: 280 """ 281 Sets a new access token. 282 283 Args: 284 new_access_token (str): The new access token to be set. 285 """ 286 self._set_config_value_by_path(self._access_token_config_path, new_access_token) 287 288 def get_refresh_token(self) -> str: 289 """ 290 Retrieve the refresh token from the configuration. 291 292 This method fetches the refresh token using the configuration path specified 293 by `_refresh_token_config_path`. 294 295 Returns: 296 str: The refresh token as a string. 297 """ 298 return self._get_config_value_by_path(self._refresh_token_config_path) # type: ignore[return-value] 299 300 def set_refresh_token(self, new_refresh_token: str) -> None: 301 """ 302 Updates the refresh token in the configuration. 303 304 Args: 305 new_refresh_token (str): The new refresh token to be set. 306 """ 307 self._set_config_value_by_path(self._refresh_token_config_path, new_refresh_token) 308 309 def get_token_expiry_date(self) -> AirbyteDateTime: 310 """ 311 Retrieves the token expiry date from the configuration. 312 313 This method fetches the token expiry date from the configuration using the specified path. 314 If the expiry date is an empty string, it returns the current date and time minus one day. 315 Otherwise, it parses the expiry date string into an AirbyteDateTime object. 316 317 Returns: 318 AirbyteDateTime: The parsed or calculated token expiry date. 319 320 Raises: 321 TypeError: If the result is not an instance of AirbyteDateTime. 322 """ 323 expiry_date = self._get_config_value_by_path(self._token_expiry_date_config_path) 324 result = ( 325 ab_datetime_now() - timedelta(days=1) 326 if expiry_date == "" 327 else ab_datetime_parse(str(expiry_date)) 328 ) 329 if isinstance(result, AirbyteDateTime): 330 return result 331 raise TypeError("Invalid datetime conversion") 332 333 def set_token_expiry_date(self, new_token_expiry_date: AirbyteDateTime) -> None: # type: ignore[override] 334 """ 335 Sets the token expiry date in the configuration. 336 337 Args: 338 new_token_expiry_date (AirbyteDateTime): The new expiry date for the token. 339 """ 340 if not ( 341 isinstance(new_token_expiry_date, datetime) or self.token_expiry_is_time_of_expiration 342 ): 343 # Overrides of `refresh_access_token` written before 6.45.5 return the raw `expires_in` 344 # in seconds. Other values are stored as-is and parsed on read. 345 with suppress(ValueError, OverflowError): 346 new_token_expiry_date = self._parse_token_expiration_date(new_token_expiry_date) 347 self._set_config_value_by_path( 348 self._token_expiry_date_config_path, str(new_token_expiry_date) 349 ) 350 351 def token_has_expired(self) -> bool: 352 """Returns True if the token is expired""" 353 return ab_datetime_now() > self.get_token_expiry_date() 354 355 def get_access_token(self) -> str: 356 """Retrieve new access and refresh token if the access token has expired. 357 358 This method uses double-checked locking to ensure thread-safe token refresh. 359 This is especially critical for single-use refresh tokens where concurrent 360 refresh attempts would cause failures as the refresh token is invalidated 361 after first use. 362 363 The new refresh token is persisted with the set_refresh_token function. 364 365 Returns: 366 str: The current access_token, updated if it was previously expired. 367 """ 368 if self.token_has_expired(): 369 with self._token_refresh_lock: 370 # Double-check after acquiring lock - another thread may have already refreshed 371 if self.token_has_expired(): 372 self.refresh_and_set_access_token() 373 return self.access_token 374 375 def refresh_and_set_access_token(self) -> None: 376 """Force refresh the access token and update internal state. 377 378 For single-use refresh tokens, this also persists the new refresh token 379 and emits a control message to update the connector config. If the 380 response omits a refresh token, the existing one is preserved. 381 """ 382 token_before_waiting = self._current_access_token_or_none() 383 with self._token_refresh_lock: 384 if self._current_access_token_or_none() != token_before_waiting: 385 return 386 new_access_token, access_token_expires_in, new_refresh_token = ( 387 self.refresh_access_token() 388 ) 389 self.access_token = new_access_token 390 if new_refresh_token is not None: 391 self.set_refresh_token(new_refresh_token) 392 self.set_token_expiry_date(access_token_expires_in) 393 self._emit_control_message() 394 395 def refresh_access_token(self) -> Tuple[str, AirbyteDateTime, Optional[str]]: # type: ignore[override] 396 """ 397 Refreshes the access token by making a handled request and extracting the necessary token information. 398 399 Returns: 400 A tuple of (access_token, token_expiry_date, refresh_token). The refresh token 401 is `None` when the OAuth provider omits it from the response. 402 """ 403 response_json = self._make_handled_request() 404 return ( 405 self._extract_access_token(response_json), 406 self._extract_token_expiry_date(response_json), 407 self._extract_refresh_token(response_json), 408 ) 409 410 def _set_config_value_by_path(self, config_path: Union[str, Sequence[str]], value: Any) -> None: 411 """ 412 Set a value in the connector configuration at the specified path. 413 414 Args: 415 config_path (Union[str, Sequence[str]]): The path within the configuration where the value should be set. 416 This can be a string representing a single key or a sequence of strings representing a nested path. 417 value (Any): The value to set at the specified path in the configuration. 418 419 Returns: 420 None 421 """ 422 dpath.new(self._connector_config, config_path, value) # type: ignore[arg-type] 423 424 def _get_config_value_by_path( 425 self, config_path: Union[str, Sequence[str]], default: Optional[str] = None 426 ) -> str | Any: 427 """ 428 Retrieve a value from the connector configuration using a specified path. 429 430 Args: 431 config_path (Union[str, Sequence[str]]): The path to the desired configuration value. This can be a string or a sequence of strings. 432 default (Optional[str], optional): The default value to return if the specified path does not exist in the configuration. Defaults to None. 433 434 Returns: 435 Any: The value from the configuration at the specified path, or the default value if the path does not exist. 436 """ 437 return dpath.get( 438 self._connector_config, # type: ignore[arg-type] 439 config_path, 440 default=default if default is not None else "", 441 ) 442 443 def _emit_control_message(self) -> None: 444 """Emit the updated connector config as a CONNECTOR_CONFIG control message. 445 446 The message is always printed to stdout, which is the delivery the platform relies on: it 447 is immediate and independent of whether the source exposes a message repository and of 448 when that repository is drained (for `ConcurrentDeclarativeSource` the internal repository 449 is only drained during `read`, after partitions are processed). The message repository is 450 only used in addition when explicitly requested, for in-process consumers such as the 451 Connector Builder; repository messages are otherwise printed by the entrypoint too and 452 would duplicate the stdout emission. 453 """ 454 emit_configuration_as_airbyte_control_message(self._connector_config) # type: ignore[arg-type] 455 if self._emit_control_message_to_message_repository and not isinstance( 456 self._message_repository, NoopMessageRepository 457 ): 458 self._message_repository.emit_message( 459 create_connector_config_control_message(self._connector_config) # type: ignore[arg-type] 460 ) 461 462 @property 463 def _message_repository(self) -> MessageRepository: 464 """ 465 Overriding AbstractOauth2Authenticator._message_repository to allow for HTTP request logs 466 """ 467 return self.__message_repository
Authenticator that should be used for API implementing single use refresh tokens:
when refreshing access token some API returns a new refresh token that needs to used in the next refresh flow.
This authenticator updates the configuration with new refresh token by emitting Airbyte control message from an observed mutation.
By default, this authenticator expects a connector config with a "credentials" field with the following nested fields: client_id,
client_secret, refresh_token. This behavior can be changed by defining custom config path (using dpath paths) in client_id_config_path,
client_secret_config_path, refresh_token_config_path constructor arguments.
refresh_token_error_messages provides fixed user-facing messages for configured refresh token errors.
170 def __init__( 171 self, 172 connector_config: Mapping[str, Any], 173 token_refresh_endpoint: str, 174 scopes: List[str] | None = None, 175 access_token_name: str = "access_token", 176 expires_in_name: str = "expires_in", 177 refresh_token_name: str = "refresh_token", 178 refresh_request_body: Mapping[str, Any] | None = None, 179 refresh_request_headers: Mapping[str, Any] | None = None, 180 send_refresh_request_as_query_params: bool = False, 181 grant_type_name: str = "grant_type", 182 grant_type: str = "refresh_token", 183 client_id_name: str = "client_id", 184 client_id: Optional[str] = None, 185 client_secret_name: str = "client_secret", 186 client_secret: Optional[str] = None, 187 access_token_config_path: Sequence[str] = ("credentials", "access_token"), 188 refresh_token_config_path: Sequence[str] = ("credentials", "refresh_token"), 189 token_expiry_date_config_path: Sequence[str] = ("credentials", "token_expiry_date"), 190 token_expiry_date_format: Optional[str] = None, 191 message_repository: MessageRepository = NoopMessageRepository(), 192 token_expiry_is_time_of_expiration: bool = False, 193 refresh_token_error_status_codes: Tuple[int, ...] = (), 194 refresh_token_error_key: str = "", 195 refresh_token_error_values: Tuple[str, ...] = (), 196 emit_control_message_to_message_repository: bool = False, 197 refresh_token_error_messages: Optional[Mapping[str, str]] = None, 198 ) -> None: 199 """ 200 Args: 201 connector_config (Mapping[str, Any]): The full connector configuration 202 token_refresh_endpoint (str): Full URL to the token refresh endpoint 203 scopes (List[str], optional): List of OAuth scopes to pass in the refresh token request body. Defaults to None. 204 access_token_name (str, optional): Name of the access token field, used to parse the refresh token response. Defaults to "access_token". 205 expires_in_name (str, optional): Name of the name of the field that characterizes when the current access token will expire, used to parse the refresh token response. Defaults to "expires_in". 206 refresh_token_name (str, optional): Name of the name of the refresh token field, used to parse the refresh token response. Defaults to "refresh_token". 207 refresh_request_body (Mapping[str, Any], optional): Custom key value pair that will be added to the refresh token request body. Defaults to None. 208 refresh_request_headers (Mapping[str, Any], optional): Custom key value pair that will be added to the refresh token request headers. Defaults to None. 209 send_refresh_request_as_query_params (bool, optional): When True, the standard refresh args (`grant_type`, `refresh_token`, client credentials when not in an `Authorization` header, scopes, plus any `refresh_request_body` extras) are sent on the URL query string and the request body is emitted empty. Use this for OAuth providers like Gong that document their refresh endpoint with refresh args on the URL query string. Defaults to False. 210 grant_type (str, optional): OAuth grant type. Defaults to "refresh_token". 211 client_id (Optional[str]): The client id to authenticate. If not specified, defaults to credentials.client_id in the config object. 212 client_secret (Optional[str]): The client secret to authenticate. If not specified, defaults to credentials.client_secret in the config object. 213 access_token_config_path (Sequence[str]): Dpath to the access_token field in the connector configuration. Defaults to ("credentials", "access_token"). 214 refresh_token_config_path (Sequence[str]): Dpath to the refresh_token field in the connector configuration. Defaults to ("credentials", "refresh_token"). 215 token_expiry_date_config_path (Sequence[str]): Dpath to the token_expiry_date field in the connector configuration. Defaults to ("credentials", "token_expiry_date"). 216 token_expiry_date_format (Optional[str]): Date format of the token expiry date field (set by expires_in_name). If specified the token expiry date is interpreted as time of expiration, as with token_expiry_is_time_of_expiration, otherwise as number of seconds until expiration. 217 token_expiry_is_time_of_expiration bool: set True it if expires_in is returned as time of expiration instead of the number seconds until expiration 218 message_repository (MessageRepository): The message repository used to emit logs on HTTP requests and control message on config update. 219 emit_control_message_to_message_repository (bool): Whether the message repository additionally receives the CONNECTOR_CONFIG message. This is only for in-process consumers such as the Connector Builder, because messages in the repository are otherwise printed to stdout by the entrypoint and would duplicate the direct stdout emission. Defaults to False. 220 refresh_token_error_messages (Optional[Mapping[str, str]]): Fixed user-facing messages for refresh token errors 221 """ 222 self._connector_config = connector_config 223 self._client_id: str = self._get_config_value_by_path( 224 ("credentials", "client_id"), client_id 225 ) 226 self._client_secret: str = self._get_config_value_by_path( 227 ("credentials", "client_secret"), client_secret 228 ) 229 self._client_id_name = client_id_name 230 self._client_secret_name = client_secret_name 231 self._access_token_config_path = access_token_config_path 232 self._refresh_token_config_path = refresh_token_config_path 233 self._token_expiry_date_config_path = token_expiry_date_config_path 234 self._token_expiry_date_format = token_expiry_date_format 235 self._refresh_token_name = refresh_token_name 236 self._grant_type_name = grant_type_name 237 self._connector_config = connector_config 238 self.__message_repository = message_repository 239 self._emit_control_message_to_message_repository = ( 240 emit_control_message_to_message_repository 241 ) 242 super().__init__( 243 token_refresh_endpoint=token_refresh_endpoint, 244 client_id_name=self._client_id_name, 245 client_id=self._client_id, 246 client_secret_name=self._client_secret_name, 247 client_secret=self._client_secret, 248 refresh_token=self.get_refresh_token(), 249 refresh_token_name=self._refresh_token_name, 250 scopes=scopes, 251 token_expiry_date=self.get_token_expiry_date(), 252 access_token_name=access_token_name, 253 expires_in_name=expires_in_name, 254 refresh_request_body=refresh_request_body, 255 refresh_request_headers=refresh_request_headers, 256 send_refresh_request_as_query_params=send_refresh_request_as_query_params, 257 grant_type_name=self._grant_type_name, 258 grant_type=grant_type, 259 token_expiry_date_format=token_expiry_date_format, 260 token_expiry_is_time_of_expiration=token_expiry_is_time_of_expiration 261 or bool(token_expiry_date_format), 262 refresh_token_error_status_codes=refresh_token_error_status_codes, 263 refresh_token_error_key=refresh_token_error_key, 264 refresh_token_error_values=refresh_token_error_values, 265 refresh_token_error_messages=refresh_token_error_messages, 266 )
Arguments:
- connector_config (Mapping[str, Any]): The full connector configuration
- token_refresh_endpoint (str): Full URL to the token refresh endpoint
- scopes (List[str], optional): List of OAuth scopes to pass in the refresh token request body. Defaults to None.
- access_token_name (str, optional): Name of the access token field, used to parse the refresh token response. Defaults to "access_token".
- expires_in_name (str, optional): Name of the name of the field that characterizes when the current access token will expire, used to parse the refresh token response. Defaults to "expires_in".
- refresh_token_name (str, optional): Name of the name of the refresh token field, used to parse the refresh token response. Defaults to "refresh_token".
- refresh_request_body (Mapping[str, Any], optional): Custom key value pair that will be added to the refresh token request body. Defaults to None.
- refresh_request_headers (Mapping[str, Any], optional): Custom key value pair that will be added to the refresh token request headers. Defaults to None.
- send_refresh_request_as_query_params (bool, optional): When True, the standard refresh args (
grant_type,refresh_token, client credentials when not in anAuthorizationheader, scopes, plus anyrefresh_request_bodyextras) are sent on the URL query string and the request body is emitted empty. Use this for OAuth providers like Gong that document their refresh endpoint with refresh args on the URL query string. Defaults to False. - grant_type (str, optional): OAuth grant type. Defaults to "refresh_token".
- client_id (Optional[str]): The client id to authenticate. If not specified, defaults to credentials.client_id in the config object.
- client_secret (Optional[str]): The client secret to authenticate. If not specified, defaults to credentials.client_secret in the config object.
- access_token_config_path (Sequence[str]): Dpath to the access_token field in the connector configuration. Defaults to ("credentials", "access_token").
- refresh_token_config_path (Sequence[str]): Dpath to the refresh_token field in the connector configuration. Defaults to ("credentials", "refresh_token").
- token_expiry_date_config_path (Sequence[str]): Dpath to the token_expiry_date field in the connector configuration. Defaults to ("credentials", "token_expiry_date").
- token_expiry_date_format (Optional[str]): Date format of the token expiry date field (set by expires_in_name). If specified the token expiry date is interpreted as time of expiration, as with token_expiry_is_time_of_expiration, otherwise as number of seconds until expiration.
- token_expiry_is_time_of_expiration bool: set True it if expires_in is returned as time of expiration instead of the number seconds until expiration
- message_repository (MessageRepository): The message repository used to emit logs on HTTP requests and control message on config update.
- emit_control_message_to_message_repository (bool): Whether the message repository additionally receives the CONNECTOR_CONFIG message. This is only for in-process consumers such as the Connector Builder, because messages in the repository are otherwise printed to stdout by the entrypoint and would duplicate the direct stdout emission. Defaults to False.
- refresh_token_error_messages (Optional[Mapping[str, str]]): Fixed user-facing messages for refresh token errors
268 @property 269 def access_token(self) -> str: 270 """ 271 Retrieve the access token from the configuration. 272 273 Returns: 274 str: The access token. 275 """ 276 return self._get_config_value_by_path(self._access_token_config_path) # type: ignore[return-value]
Retrieve the access token from the configuration.
Returns:
str: The access token.
288 def get_refresh_token(self) -> str: 289 """ 290 Retrieve the refresh token from the configuration. 291 292 This method fetches the refresh token using the configuration path specified 293 by `_refresh_token_config_path`. 294 295 Returns: 296 str: The refresh token as a string. 297 """ 298 return self._get_config_value_by_path(self._refresh_token_config_path) # type: ignore[return-value]
Retrieve the refresh token from the configuration.
This method fetches the refresh token using the configuration path specified
by _refresh_token_config_path.
Returns:
str: The refresh token as a string.
300 def set_refresh_token(self, new_refresh_token: str) -> None: 301 """ 302 Updates the refresh token in the configuration. 303 304 Args: 305 new_refresh_token (str): The new refresh token to be set. 306 """ 307 self._set_config_value_by_path(self._refresh_token_config_path, new_refresh_token)
Updates the refresh token in the configuration.
Arguments:
- new_refresh_token (str): The new refresh token to be set.
309 def get_token_expiry_date(self) -> AirbyteDateTime: 310 """ 311 Retrieves the token expiry date from the configuration. 312 313 This method fetches the token expiry date from the configuration using the specified path. 314 If the expiry date is an empty string, it returns the current date and time minus one day. 315 Otherwise, it parses the expiry date string into an AirbyteDateTime object. 316 317 Returns: 318 AirbyteDateTime: The parsed or calculated token expiry date. 319 320 Raises: 321 TypeError: If the result is not an instance of AirbyteDateTime. 322 """ 323 expiry_date = self._get_config_value_by_path(self._token_expiry_date_config_path) 324 result = ( 325 ab_datetime_now() - timedelta(days=1) 326 if expiry_date == "" 327 else ab_datetime_parse(str(expiry_date)) 328 ) 329 if isinstance(result, AirbyteDateTime): 330 return result 331 raise TypeError("Invalid datetime conversion")
Retrieves the token expiry date from the configuration.
This method fetches the token expiry date from the configuration using the specified path. If the expiry date is an empty string, it returns the current date and time minus one day. Otherwise, it parses the expiry date string into an AirbyteDateTime object.
Returns:
AirbyteDateTime: The parsed or calculated token expiry date.
Raises:
- TypeError: If the result is not an instance of AirbyteDateTime.
333 def set_token_expiry_date(self, new_token_expiry_date: AirbyteDateTime) -> None: # type: ignore[override] 334 """ 335 Sets the token expiry date in the configuration. 336 337 Args: 338 new_token_expiry_date (AirbyteDateTime): The new expiry date for the token. 339 """ 340 if not ( 341 isinstance(new_token_expiry_date, datetime) or self.token_expiry_is_time_of_expiration 342 ): 343 # Overrides of `refresh_access_token` written before 6.45.5 return the raw `expires_in` 344 # in seconds. Other values are stored as-is and parsed on read. 345 with suppress(ValueError, OverflowError): 346 new_token_expiry_date = self._parse_token_expiration_date(new_token_expiry_date) 347 self._set_config_value_by_path( 348 self._token_expiry_date_config_path, str(new_token_expiry_date) 349 )
Sets the token expiry date in the configuration.
Arguments:
- new_token_expiry_date (AirbyteDateTime): The new expiry date for the token.
351 def token_has_expired(self) -> bool: 352 """Returns True if the token is expired""" 353 return ab_datetime_now() > self.get_token_expiry_date()
Returns True if the token is expired
355 def get_access_token(self) -> str: 356 """Retrieve new access and refresh token if the access token has expired. 357 358 This method uses double-checked locking to ensure thread-safe token refresh. 359 This is especially critical for single-use refresh tokens where concurrent 360 refresh attempts would cause failures as the refresh token is invalidated 361 after first use. 362 363 The new refresh token is persisted with the set_refresh_token function. 364 365 Returns: 366 str: The current access_token, updated if it was previously expired. 367 """ 368 if self.token_has_expired(): 369 with self._token_refresh_lock: 370 # Double-check after acquiring lock - another thread may have already refreshed 371 if self.token_has_expired(): 372 self.refresh_and_set_access_token() 373 return self.access_token
Retrieve new access and refresh token if the access token has expired.
This method uses double-checked locking to ensure thread-safe token refresh. This is especially critical for single-use refresh tokens where concurrent refresh attempts would cause failures as the refresh token is invalidated after first use.
The new refresh token is persisted with the set_refresh_token function.
Returns:
str: The current access_token, updated if it was previously expired.
375 def refresh_and_set_access_token(self) -> None: 376 """Force refresh the access token and update internal state. 377 378 For single-use refresh tokens, this also persists the new refresh token 379 and emits a control message to update the connector config. If the 380 response omits a refresh token, the existing one is preserved. 381 """ 382 token_before_waiting = self._current_access_token_or_none() 383 with self._token_refresh_lock: 384 if self._current_access_token_or_none() != token_before_waiting: 385 return 386 new_access_token, access_token_expires_in, new_refresh_token = ( 387 self.refresh_access_token() 388 ) 389 self.access_token = new_access_token 390 if new_refresh_token is not None: 391 self.set_refresh_token(new_refresh_token) 392 self.set_token_expiry_date(access_token_expires_in) 393 self._emit_control_message()
Force refresh the access token and update internal state.
For single-use refresh tokens, this also persists the new refresh token and emits a control message to update the connector config. If the response omits a refresh token, the existing one is preserved.
395 def refresh_access_token(self) -> Tuple[str, AirbyteDateTime, Optional[str]]: # type: ignore[override] 396 """ 397 Refreshes the access token by making a handled request and extracting the necessary token information. 398 399 Returns: 400 A tuple of (access_token, token_expiry_date, refresh_token). The refresh token 401 is `None` when the OAuth provider omits it from the response. 402 """ 403 response_json = self._make_handled_request() 404 return ( 405 self._extract_access_token(response_json), 406 self._extract_token_expiry_date(response_json), 407 self._extract_refresh_token(response_json), 408 )
Refreshes the access token by making a handled request and extracting the necessary token information.
Returns:
A tuple of (access_token, token_expiry_date, refresh_token). The refresh token is
Nonewhen the OAuth provider omits it from the response.
Inherited Members
- Oauth2Authenticator
- get_token_refresh_endpoint
- get_client_id_name
- get_client_id
- get_client_secret_name
- get_client_secret
- get_refresh_token_name
- get_access_token_name
- get_scopes
- get_expires_in_name
- get_refresh_request_body
- get_refresh_request_headers
- should_send_refresh_request_as_query_params
- get_grant_type_name
- get_grant_type
- token_expiry_is_time_of_expiration
- token_expiry_date_format
39class TokenAuthenticator(AbstractHeaderAuthenticator): 40 """ 41 Builds auth header, based on the token provided. 42 The token is attached to each request via the `auth_header` header. 43 """ 44 45 @property 46 def auth_header(self) -> str: 47 return self._auth_header 48 49 @property 50 def token(self) -> str: 51 return f"{self._auth_method} {self._token}" 52 53 def __init__(self, token: str, auth_method: str = "Bearer", auth_header: str = "Authorization"): 54 self._auth_header = auth_header 55 self._auth_method = auth_method 56 self._token = token
Builds auth header, based on the token provided.
The token is attached to each request via the auth_header header.
Inherited Members
40class UserDefinedBackoffException(BaseBackoffException): 41 """ 42 An exception that exposes how long it attempted to backoff 43 """ 44 45 def __init__( 46 self, 47 backoff: Union[int, float], 48 request: requests.PreparedRequest, 49 response: Optional[Union[requests.Response, Exception]], 50 error_message: str = "", 51 failure_type: Optional[FailureType] = None, 52 ): 53 """ 54 :param backoff: how long to backoff in seconds 55 :param request: the request that triggered this backoff exception 56 :param response: the response that triggered the backoff exception 57 """ 58 self.backoff = backoff 59 super().__init__( 60 request=request, 61 response=response, 62 error_message=error_message, 63 failure_type=failure_type, 64 )
An exception that exposes how long it attempted to backoff
45 def __init__( 46 self, 47 backoff: Union[int, float], 48 request: requests.PreparedRequest, 49 response: Optional[Union[requests.Response, Exception]], 50 error_message: str = "", 51 failure_type: Optional[FailureType] = None, 52 ): 53 """ 54 :param backoff: how long to backoff in seconds 55 :param request: the request that triggered this backoff exception 56 :param response: the response that triggered the backoff exception 57 """ 58 self.backoff = backoff 59 super().__init__( 60 request=request, 61 response=response, 62 error_message=error_message, 63 failure_type=failure_type, 64 )
Parameters
- backoff: how long to backoff in seconds
- request: the request that triggered this backoff exception
- response: the response that triggered the backoff exception
Inherited Members
64class AirbyteLogFormatter(logging.Formatter): 65 """Output log records using AirbyteMessage""" 66 67 # Transforming Python log levels to Airbyte protocol log levels 68 level_mapping = { 69 logging.FATAL: Level.FATAL, 70 logging.ERROR: Level.ERROR, 71 logging.WARNING: Level.WARN, 72 logging.INFO: Level.INFO, 73 logging.DEBUG: Level.DEBUG, 74 } 75 76 def format(self, record: logging.LogRecord) -> str: 77 """Return a JSON representation of the log message""" 78 airbyte_level = self.level_mapping.get(record.levelno, "INFO") 79 if airbyte_level == Level.DEBUG: 80 extras = self.extract_extra_args_from_record(record) 81 if is_platform_debug_log_enabled(): 82 # We have a different behavior between debug logs enabled through `--debug` argument and debug logs 83 # enabled through environment variable. The reason is that for platform logs, we need to have these 84 # printed as AirbyteMessage which is not the case with the current previous implementation. 85 # Why not migrate both to AirbyteMessages then? AirbyteMessages do not support having structured logs. 86 # which means that the DX would be degraded compared to the current solution (devs will need to identify 87 # the `log.message` field and figure out where in this field is the response while the current solution 88 # have a specific field that is structured for extras. 89 message = f"{filter_secrets(record.getMessage())} ///\nExtra logs: {filter_secrets(json.dumps(extras))}" 90 log_message = AirbyteMessage( 91 type=Type.LOG, log=AirbyteLogMessage(level=airbyte_level, message=message) 92 ) 93 return orjson.dumps(AirbyteMessageSerializer.dump(log_message)).decode() 94 else: 95 debug_dict = {"type": "DEBUG", "message": record.getMessage(), "data": extras} 96 return filter_secrets(json.dumps(debug_dict)) 97 else: 98 message = super().format(record) 99 message = filter_secrets(message) 100 log_message = AirbyteMessage( 101 type=Type.LOG, log=AirbyteLogMessage(level=airbyte_level, message=message) 102 ) 103 return orjson.dumps(AirbyteMessageSerializer.dump(log_message)).decode() 104 105 @staticmethod 106 def extract_extra_args_from_record(record: logging.LogRecord) -> Mapping[str, Any]: 107 """ 108 The python logger conflates default args with extra args. We use an empty log record and set operations 109 to isolate fields passed to the log record via extra by the developer. 110 """ 111 default_attrs = logging.LogRecord("", 0, "", 0, None, None, None).__dict__.keys() 112 extra_keys = set(record.__dict__.keys()) - default_attrs 113 return {k: str(getattr(record, k)) for k in extra_keys if hasattr(record, k)}
Output log records using AirbyteMessage
76 def format(self, record: logging.LogRecord) -> str: 77 """Return a JSON representation of the log message""" 78 airbyte_level = self.level_mapping.get(record.levelno, "INFO") 79 if airbyte_level == Level.DEBUG: 80 extras = self.extract_extra_args_from_record(record) 81 if is_platform_debug_log_enabled(): 82 # We have a different behavior between debug logs enabled through `--debug` argument and debug logs 83 # enabled through environment variable. The reason is that for platform logs, we need to have these 84 # printed as AirbyteMessage which is not the case with the current previous implementation. 85 # Why not migrate both to AirbyteMessages then? AirbyteMessages do not support having structured logs. 86 # which means that the DX would be degraded compared to the current solution (devs will need to identify 87 # the `log.message` field and figure out where in this field is the response while the current solution 88 # have a specific field that is structured for extras. 89 message = f"{filter_secrets(record.getMessage())} ///\nExtra logs: {filter_secrets(json.dumps(extras))}" 90 log_message = AirbyteMessage( 91 type=Type.LOG, log=AirbyteLogMessage(level=airbyte_level, message=message) 92 ) 93 return orjson.dumps(AirbyteMessageSerializer.dump(log_message)).decode() 94 else: 95 debug_dict = {"type": "DEBUG", "message": record.getMessage(), "data": extras} 96 return filter_secrets(json.dumps(debug_dict)) 97 else: 98 message = super().format(record) 99 message = filter_secrets(message) 100 log_message = AirbyteMessage( 101 type=Type.LOG, log=AirbyteLogMessage(level=airbyte_level, message=message) 102 ) 103 return orjson.dumps(AirbyteMessageSerializer.dump(log_message)).decode()
Return a JSON representation of the log message
105 @staticmethod 106 def extract_extra_args_from_record(record: logging.LogRecord) -> Mapping[str, Any]: 107 """ 108 The python logger conflates default args with extra args. We use an empty log record and set operations 109 to isolate fields passed to the log record via extra by the developer. 110 """ 111 default_attrs = logging.LogRecord("", 0, "", 0, None, None, None).__dict__.keys() 112 extra_keys = set(record.__dict__.keys()) - default_attrs 113 return {k: str(getattr(record, k)) for k in extra_keys if hasattr(record, k)}
The python logger conflates default args with extra args. We use an empty log record and set operations to isolate fields passed to the log record via extra by the developer.
48def init_logger(name: Optional[str] = None) -> logging.Logger: 49 """Initial set up of logger""" 50 logger = logging.getLogger(name) 51 logger.setLevel(logging.INFO) 52 logging.config.dictConfig(LOGGING_CONFIG) 53 return logger
Initial set up of logger
300@dataclass 301class AirbyteStream: 302 name: str 303 json_schema: Dict[str, Any] 304 supported_sync_modes: List[SyncMode] 305 source_defined_cursor: Optional[bool] = None 306 default_cursor_field: Optional[List[str]] = None 307 source_defined_primary_key: Optional[List[List[str]]] = None 308 namespace: Optional[str] = None 309 is_resumable: Optional[bool] = None 310 is_file_based: Optional[bool] = None
180@dataclass 181class AirbyteConnectionStatus: 182 status: Status 183 message: Optional[str] = None
140@dataclass 141class AirbyteMessage: 142 type: Type # type: ignore [name-defined] 143 log: Optional[AirbyteLogMessage] = None # type: ignore [name-defined] 144 spec: Optional[ConnectorSpecification] = None 145 connectionStatus: Optional[AirbyteConnectionStatus] = None # type: ignore [name-defined] 146 catalog: Optional[AirbyteCatalog] = None # type: ignore [name-defined] 147 record: Optional[AirbyteRecordMessage] = None # type: ignore [name-defined] 148 state: Optional[AirbyteStateMessage] = None 149 trace: Optional[AirbyteTraceMessage] = None # type: ignore [name-defined] 150 control: Optional[AirbyteControlMessage] = None # type: ignore [name-defined]
12class Type(Enum): 13 RECORD = 'RECORD' 14 STATE = 'STATE' 15 LOG = 'LOG' 16 SPEC = 'SPEC' 17 CONNECTION_STATUS = 'CONNECTION_STATUS' 18 CATALOG = 'CATALOG' 19 TRACE = 'TRACE' 20 CONTROL = 'CONTROL' 21 DESTINATION_CATALOG = 'DESTINATION_CATALOG'
313@dataclass 314class ConfiguredAirbyteStream: 315 stream: AirbyteStream 316 sync_mode: SyncMode 317 destination_sync_mode: DestinationSyncMode 318 cursor_field: Optional[List[str]] = None 319 destination_object_name: Optional[str] = None 320 primary_key: Optional[List[List[str]]] = None 321 generation_id: Optional[int] = None 322 minimum_generation_id: Optional[int] = None 323 sync_id: Optional[int] = None 324 include_files: Optional[bool] = None
191class DestinationSyncMode(Enum): 192 append = 'append' 193 overwrite = 'overwrite' 194 append_dedup = 'append_dedup' 195 update = 'update' 196 soft_delete = 'soft_delete'
102class FailureType(Enum): 103 system_error = 'system_error' 104 config_error = 'config_error' 105 transient_error = 'transient_error'
118@dataclass 119class AdvancedAuth: 120 auth_flow_type: Optional[AuthFlowType] = None # type: ignore [name-defined] 121 predicate_key: Optional[List[str]] = None 122 predicate_value: Optional[str] = None 123 oauth_config_specification: Optional[OAuthConfigSpecification] = None
88@dataclass 89class AirbyteLogMessage: 90 level: Level 91 message: str 92 stack_trace: Optional[str] = None
109@dataclass 110class OAuthConfigSpecification: 111 oauth_user_input_from_connector_config_specification: Optional[Dict[str, Any]] = None 112 oauth_connector_input_specification: Optional[OauthConnectorInputSpecification] = None 113 complete_oauth_output_specification: Optional[Dict[str, Any]] = None 114 complete_oauth_server_input_specification: Optional[Dict[str, Any]] = None 115 complete_oauth_server_output_specification: Optional[Dict[str, Any]] = None
126@dataclass 127class ConnectorSpecification: 128 connectionSpecification: Dict[str, Any] 129 documentationUrl: Optional[str] = None 130 changelogUrl: Optional[str] = None 131 supportsIncremental: Optional[bool] = None 132 supportsNormalization: Optional[bool] = False 133 supportsDBT: Optional[bool] = False 134 supported_destination_sync_modes: Optional[List[DestinationSyncMode]] = None # type: ignore [name-defined] 135 authSpecification: Optional[AuthSpecification] = None # type: ignore [name-defined] 136 advanced_auth: Optional[AdvancedAuth] = None 137 protocol_version: Optional[str] = None
79class Level(Enum): 80 FATAL = 'FATAL' 81 ERROR = 'ERROR' 82 WARN = 'WARN' 83 INFO = 'INFO' 84 DEBUG = 'DEBUG' 85 TRACE = 'TRACE'
354@dataclass 355class AirbyteRecordMessage: 356 stream: str 357 data: Dict[str, Any] 358 emitted_at: int 359 namespace: Optional[str] = None 360 meta: Optional[AirbyteRecordMessageMeta] = None 361 file_reference: Optional[AirbyteRecordMessageFileReference] = None
75class InMemoryMessageRepository(MessageRepository): 76 def __init__(self, log_level: Level = Level.INFO) -> None: 77 self._message_queue: Deque[AirbyteMessage] = deque() 78 self._log_level = log_level 79 80 def emit_message(self, message: AirbyteMessage) -> None: 81 self._message_queue.append(message) 82 83 def log_message(self, level: Level, message_provider: Callable[[], LogMessage]) -> None: 84 if _is_severe_enough(self._log_level, level): 85 self.emit_message( 86 AirbyteMessage( 87 type=Type.LOG, 88 log=AirbyteLogMessage( 89 level=level, message=filter_secrets(json.dumps(message_provider())) 90 ), 91 ) 92 ) 93 94 def consume_queue(self) -> Iterable[AirbyteMessage]: 95 while self._message_queue: 96 yield self._message_queue.popleft()
Helper class that provides a standard way to create an ABC using inheritance.
83 def log_message(self, level: Level, message_provider: Callable[[], LogMessage]) -> None: 84 if _is_severe_enough(self._log_level, level): 85 self.emit_message( 86 AirbyteMessage( 87 type=Type.LOG, 88 log=AirbyteLogMessage( 89 level=level, message=filter_secrets(json.dumps(message_provider())) 90 ), 91 ) 92 )
Computing messages can be resource consuming. This method is specialized for logging because we want to allow for lazy evaluation if the log level is less severe than what is configured
46class MessageRepository(ABC): 47 @abstractmethod 48 def emit_message(self, message: AirbyteMessage) -> None: 49 raise NotImplementedError() 50 51 @abstractmethod 52 def log_message(self, level: Level, message_provider: Callable[[], LogMessage]) -> None: 53 """ 54 Computing messages can be resource consuming. This method is specialized for logging because we want to allow for lazy evaluation if 55 the log level is less severe than what is configured 56 """ 57 raise NotImplementedError() 58 59 @abstractmethod 60 def consume_queue(self) -> Iterable[AirbyteMessage]: 61 raise NotImplementedError()
Helper class that provides a standard way to create an ABC using inheritance.
51 @abstractmethod 52 def log_message(self, level: Level, message_provider: Callable[[], LogMessage]) -> None: 53 """ 54 Computing messages can be resource consuming. This method is specialized for logging because we want to allow for lazy evaluation if 55 the log level is less severe than what is configured 56 """ 57 raise NotImplementedError()
Computing messages can be resource consuming. This method is specialized for logging because we want to allow for lazy evaluation if the log level is less severe than what is configured
33class ConnectorStateManager: 34 """ 35 ConnectorStateManager consolidates the various forms of a stream's incoming state message (STREAM / GLOBAL) under a common 36 interface. It also provides methods to extract and update state 37 """ 38 39 def __init__(self, state: Optional[List[AirbyteStateMessage]] = None): 40 shared_state, per_stream_states = self._extract_from_state_message(state) 41 42 # We explicitly throw an error if we receive a GLOBAL state message that contains a shared_state because API sources are 43 # designed to checkpoint state independently of one another. API sources should never be emitting a state message where 44 # shared_state is populated. Rather than define how to handle shared_state without a clear use case, we're opting to throw an 45 # error instead and if/when we find one, we will then implement processing of the shared_state value. 46 if shared_state: 47 raise ValueError( 48 "Received a GLOBAL AirbyteStateMessage that contains a shared_state. This library only ever generates per-STREAM " 49 "STATE messages so this was not generated by this connector. This must be an orchestrator or platform error. GLOBAL " 50 "state messages with shared_state will not be processed correctly. " 51 ) 52 self.per_stream_states = per_stream_states 53 54 def get_stream_state( 55 self, stream_name: str, namespace: Optional[str] 56 ) -> MutableMapping[str, Any]: 57 """ 58 Retrieves the state of a given stream based on its descriptor (name + namespace). 59 :param stream_name: Name of the stream being fetched 60 :param namespace: Namespace of the stream being fetched 61 :return: The per-stream state for a stream 62 """ 63 stream_state: AirbyteStateBlob | None = self.per_stream_states.get( 64 HashableStreamDescriptor(name=stream_name, namespace=namespace) 65 ) 66 if stream_state: 67 return copy.deepcopy({k: v for k, v in stream_state.__dict__.items()}) 68 return {} 69 70 def update_state_for_stream( 71 self, stream_name: str, namespace: Optional[str], value: Mapping[str, Any] 72 ) -> None: 73 """ 74 Overwrites the state blob of a specific stream based on the provided stream name and optional namespace 75 :param stream_name: The name of the stream whose state is being updated 76 :param namespace: The namespace of the stream if it exists 77 :param value: A stream state mapping that is being updated for a stream 78 """ 79 stream_descriptor = HashableStreamDescriptor(name=stream_name, namespace=namespace) 80 self.per_stream_states[stream_descriptor] = AirbyteStateBlob(value) 81 82 def create_state_message(self, stream_name: str, namespace: Optional[str]) -> AirbyteMessage: 83 """ 84 Generates an AirbyteMessage using the current per-stream state of a specified stream 85 :param stream_name: The name of the stream for the message that is being created 86 :param namespace: The namespace of the stream for the message that is being created 87 :return: The Airbyte state message to be emitted by the connector during a sync 88 """ 89 hashable_descriptor = HashableStreamDescriptor(name=stream_name, namespace=namespace) 90 stream_state = self.per_stream_states.get(hashable_descriptor) or AirbyteStateBlob() 91 92 return AirbyteMessage( 93 type=MessageType.STATE, 94 state=AirbyteStateMessage( 95 type=AirbyteStateType.STREAM, 96 stream=AirbyteStreamState( 97 stream_descriptor=StreamDescriptor(name=stream_name, namespace=namespace), 98 stream_state=stream_state, 99 ), 100 ), 101 ) 102 103 @classmethod 104 def _extract_from_state_message( 105 cls, 106 state: Optional[List[AirbyteStateMessage]], 107 ) -> Tuple[ 108 Optional[AirbyteStateBlob], 109 MutableMapping[HashableStreamDescriptor, Optional[AirbyteStateBlob]], 110 ]: 111 """ 112 Takes an incoming list of state messages or a global state message and extracts state attributes according to 113 type which can then be assigned to the new state manager being instantiated 114 :param state: The incoming state input 115 :return: A tuple of shared state and per stream state assembled from the incoming state list 116 """ 117 if state is None: 118 return None, {} 119 120 is_global = cls._is_global_state(state) 121 122 if is_global: 123 # We already validate that this is a global state message, not None: 124 global_state = cast(AirbyteGlobalState, state[0].global_) 125 # global_state has shared_state, also not None: 126 shared_state: AirbyteStateBlob = cast( 127 AirbyteStateBlob, copy.deepcopy(global_state.shared_state, {}) 128 ) 129 streams = { 130 HashableStreamDescriptor( 131 name=per_stream_state.stream_descriptor.name, 132 namespace=per_stream_state.stream_descriptor.namespace, 133 ): per_stream_state.stream_state 134 for per_stream_state in global_state.stream_states # type: ignore[union-attr] # global_state has shared_state 135 } 136 return shared_state, streams 137 else: 138 streams = { 139 HashableStreamDescriptor( 140 name=per_stream_state.stream.stream_descriptor.name, # type: ignore[union-attr] # stream has stream_descriptor 141 namespace=per_stream_state.stream.stream_descriptor.namespace, # type: ignore[union-attr] # stream has stream_descriptor 142 ): per_stream_state.stream.stream_state # type: ignore[union-attr] # stream has stream_state 143 for per_stream_state in state 144 if per_stream_state.type == AirbyteStateType.STREAM 145 and hasattr(per_stream_state, "stream") # type: ignore # state is always a list of AirbyteStateMessage if is_per_stream is True 146 } 147 return None, streams 148 149 @staticmethod 150 def _is_global_state(state: Union[List[AirbyteStateMessage], MutableMapping[str, Any]]) -> bool: 151 return ( 152 isinstance(state, List) 153 and len(state) == 1 154 and isinstance(state[0], AirbyteStateMessage) 155 and state[0].type == AirbyteStateType.GLOBAL 156 ) 157 158 @staticmethod 159 def _is_per_stream_state( 160 state: Union[List[AirbyteStateMessage], MutableMapping[str, Any]], 161 ) -> bool: 162 return isinstance(state, List)
ConnectorStateManager consolidates the various forms of a stream's incoming state message (STREAM / GLOBAL) under a common interface. It also provides methods to extract and update state
39 def __init__(self, state: Optional[List[AirbyteStateMessage]] = None): 40 shared_state, per_stream_states = self._extract_from_state_message(state) 41 42 # We explicitly throw an error if we receive a GLOBAL state message that contains a shared_state because API sources are 43 # designed to checkpoint state independently of one another. API sources should never be emitting a state message where 44 # shared_state is populated. Rather than define how to handle shared_state without a clear use case, we're opting to throw an 45 # error instead and if/when we find one, we will then implement processing of the shared_state value. 46 if shared_state: 47 raise ValueError( 48 "Received a GLOBAL AirbyteStateMessage that contains a shared_state. This library only ever generates per-STREAM " 49 "STATE messages so this was not generated by this connector. This must be an orchestrator or platform error. GLOBAL " 50 "state messages with shared_state will not be processed correctly. " 51 ) 52 self.per_stream_states = per_stream_states
54 def get_stream_state( 55 self, stream_name: str, namespace: Optional[str] 56 ) -> MutableMapping[str, Any]: 57 """ 58 Retrieves the state of a given stream based on its descriptor (name + namespace). 59 :param stream_name: Name of the stream being fetched 60 :param namespace: Namespace of the stream being fetched 61 :return: The per-stream state for a stream 62 """ 63 stream_state: AirbyteStateBlob | None = self.per_stream_states.get( 64 HashableStreamDescriptor(name=stream_name, namespace=namespace) 65 ) 66 if stream_state: 67 return copy.deepcopy({k: v for k, v in stream_state.__dict__.items()}) 68 return {}
Retrieves the state of a given stream based on its descriptor (name + namespace).
Parameters
- stream_name: Name of the stream being fetched
- namespace: Namespace of the stream being fetched
Returns
The per-stream state for a stream
70 def update_state_for_stream( 71 self, stream_name: str, namespace: Optional[str], value: Mapping[str, Any] 72 ) -> None: 73 """ 74 Overwrites the state blob of a specific stream based on the provided stream name and optional namespace 75 :param stream_name: The name of the stream whose state is being updated 76 :param namespace: The namespace of the stream if it exists 77 :param value: A stream state mapping that is being updated for a stream 78 """ 79 stream_descriptor = HashableStreamDescriptor(name=stream_name, namespace=namespace) 80 self.per_stream_states[stream_descriptor] = AirbyteStateBlob(value)
Overwrites the state blob of a specific stream based on the provided stream name and optional namespace
Parameters
- stream_name: The name of the stream whose state is being updated
- namespace: The namespace of the stream if it exists
- value: A stream state mapping that is being updated for a stream
82 def create_state_message(self, stream_name: str, namespace: Optional[str]) -> AirbyteMessage: 83 """ 84 Generates an AirbyteMessage using the current per-stream state of a specified stream 85 :param stream_name: The name of the stream for the message that is being created 86 :param namespace: The namespace of the stream for the message that is being created 87 :return: The Airbyte state message to be emitted by the connector during a sync 88 """ 89 hashable_descriptor = HashableStreamDescriptor(name=stream_name, namespace=namespace) 90 stream_state = self.per_stream_states.get(hashable_descriptor) or AirbyteStateBlob() 91 92 return AirbyteMessage( 93 type=MessageType.STATE, 94 state=AirbyteStateMessage( 95 type=AirbyteStateType.STREAM, 96 stream=AirbyteStreamState( 97 stream_descriptor=StreamDescriptor(name=stream_name, namespace=namespace), 98 stream_state=stream_state, 99 ), 100 ), 101 )
Generates an AirbyteMessage using the current per-stream state of a specified stream
Parameters
- stream_name: The name of the stream for the message that is being created
- namespace: The namespace of the stream for the message that is being created
Returns
The Airbyte state message to be emitted by the connector during a sync
95@deprecated( 96 "Deprecated as of CDK version 0.87.0. " 97 "Deprecated in favor of the `CheckpointMixin` which offers similar functionality." 98) 99class IncrementalMixin(CheckpointMixin, ABC): 100 """Mixin to make stream incremental. 101 102 class IncrementalStream(Stream, IncrementalMixin): 103 @property 104 def state(self): 105 return self._state 106 107 @state.setter 108 def state(self, value): 109 self._state[self.cursor_field] = value[self.cursor_field] 110 """
Mixin to make stream incremental.
class IncrementalStream(Stream, IncrementalMixin): @property def state(self): return self._state
@state.setter
def state(self, value):
self._state[self.cursor_field] = value[self.cursor_field]
Inherited Members
119class Stream(ABC): 120 """ 121 Base abstract class for an Airbyte Stream. Makes no assumption of the Stream's underlying transport protocol. 122 """ 123 124 _configured_json_schema: Optional[Dict[str, Any]] = None 125 _exit_on_rate_limit: bool = False 126 127 # Use self.logger in subclasses to log any messages 128 @property 129 def logger(self) -> logging.Logger: 130 return logging.getLogger(f"airbyte.streams.{self.name}") 131 132 # TypeTransformer object to perform output data transformation 133 transformer: TypeTransformer = TypeTransformer(TransformConfig.NoTransform) 134 135 cursor: Optional[Cursor] = None 136 137 has_multiple_slices = False 138 139 @cached_property 140 def name(self) -> str: 141 """ 142 :return: Stream name. By default this is the implementing class name, but it can be overridden as needed. 143 """ 144 return casing.camel_to_snake(self.__class__.__name__) 145 146 def get_error_display_message(self, exception: BaseException) -> Optional[str]: 147 """ 148 Retrieves the user-friendly display message that corresponds to an exception. 149 This will be called when encountering an exception while reading records from the stream, and used to build the AirbyteTraceMessage. 150 151 The default implementation of this method does not return user-friendly messages for any exception type, but it should be overriden as needed. 152 153 :param exception: The exception that was raised 154 :return: A user-friendly message that indicates the cause of the error 155 """ 156 return None 157 158 def read( # type: ignore # ignoring typing for ConnectorStateManager because of circular dependencies 159 self, 160 configured_stream: ConfiguredAirbyteStream, 161 logger: logging.Logger, 162 slice_logger: SliceLogger, 163 stream_state: MutableMapping[str, Any], 164 state_manager, 165 internal_config: InternalConfig, 166 ) -> Iterable[StreamData]: 167 sync_mode = configured_stream.sync_mode 168 cursor_field = configured_stream.cursor_field 169 self.configured_json_schema = configured_stream.stream.json_schema 170 171 # WARNING: When performing a read() that uses incoming stream state, we MUST use the self.state that is defined as 172 # opposed to the incoming stream_state value. Because some connectors like ones using the file-based CDK modify 173 # state before setting the value on the Stream attribute, the most up-to-date state is derived from Stream.state 174 # instead of the stream_state parameter. This does not apply to legacy connectors using get_updated_state(). 175 try: 176 stream_state = self.state # type: ignore # we know the field might not exist... 177 except AttributeError: 178 pass 179 180 should_checkpoint = bool(state_manager) 181 checkpoint_reader = self._get_checkpoint_reader( 182 logger=logger, cursor_field=cursor_field, sync_mode=sync_mode, stream_state=stream_state 183 ) 184 185 next_slice = checkpoint_reader.next() 186 record_counter = 0 187 stream_state_tracker = copy.deepcopy(stream_state) 188 while next_slice is not None: 189 if slice_logger.should_log_slice_message(logger): 190 yield slice_logger.create_slice_log_message(next_slice) 191 records = self.read_records( 192 sync_mode=sync_mode, # todo: change this interface to no longer rely on sync_mode for behavior 193 stream_slice=next_slice, 194 stream_state=stream_state, 195 cursor_field=cursor_field or None, 196 ) 197 for record_data_or_message in records: 198 yield record_data_or_message 199 if isinstance(record_data_or_message, Mapping) or ( 200 hasattr(record_data_or_message, "type") 201 and record_data_or_message.type == MessageType.RECORD 202 ): 203 record_data = ( 204 record_data_or_message 205 if isinstance(record_data_or_message, Mapping) 206 else record_data_or_message.record 207 ) 208 209 # Thanks I hate it. RFR fundamentally doesn't fit with the concept of the legacy Stream.get_updated_state() 210 # method because RFR streams rely on pagination as a cursor. Stream.get_updated_state() was designed to make 211 # the CDK manage state using specifically the last seen record. don't @ brian.lai 212 # 213 # Also, because the legacy incremental state case decouples observing incoming records from emitting state, it 214 # requires that we separate CheckpointReader.observe() and CheckpointReader.get_checkpoint() which could 215 # otherwise be combined. 216 if self.cursor_field: 217 # Some connectors have streams that implement get_updated_state(), but do not define a cursor_field. This 218 # should be fixed on the stream implementation, but we should also protect against this in the CDK as well 219 stream_state_tracker = self.get_updated_state( 220 stream_state_tracker, 221 record_data, # type: ignore [arg-type] 222 ) 223 self._observe_state(checkpoint_reader, stream_state_tracker) 224 record_counter += 1 225 226 checkpoint_interval = self.state_checkpoint_interval 227 if ( 228 should_checkpoint 229 and checkpoint_interval 230 and record_counter % checkpoint_interval == 0 231 ): 232 checkpoint = checkpoint_reader.get_checkpoint() 233 if checkpoint: 234 airbyte_state_message = self._checkpoint_state( 235 checkpoint, state_manager=state_manager 236 ) 237 yield airbyte_state_message 238 239 if internal_config.is_limit_reached(record_counter): 240 break 241 self._observe_state(checkpoint_reader) 242 checkpoint_state = checkpoint_reader.get_checkpoint() 243 if should_checkpoint and checkpoint_state is not None: 244 airbyte_state_message = self._checkpoint_state( 245 checkpoint_state, state_manager=state_manager 246 ) 247 yield airbyte_state_message 248 249 next_slice = checkpoint_reader.next() 250 251 checkpoint = checkpoint_reader.get_checkpoint() 252 if should_checkpoint and checkpoint is not None: 253 airbyte_state_message = self._checkpoint_state(checkpoint, state_manager=state_manager) 254 yield airbyte_state_message 255 256 def read_only_records(self, state: Optional[Mapping[str, Any]] = None) -> Iterable[StreamData]: 257 """ 258 Helper method that performs a read on a stream with an optional state and emits records. If the parent stream supports 259 incremental, this operation does not update the stream's internal state (if it uses the modern state setter/getter) 260 or emit state messages. 261 """ 262 263 configured_stream = ConfiguredAirbyteStream( 264 stream=AirbyteStream( 265 name=self.name, 266 json_schema={}, 267 supported_sync_modes=[SyncMode.full_refresh, SyncMode.incremental], 268 ), 269 sync_mode=SyncMode.incremental if state else SyncMode.full_refresh, 270 destination_sync_mode=DestinationSyncMode.append, 271 ) 272 273 yield from self.read( 274 configured_stream=configured_stream, 275 logger=self.logger, 276 slice_logger=DebugSliceLogger(), 277 stream_state=dict(state) 278 if state 279 else {}, # read() expects MutableMapping instead of Mapping which is used more often 280 state_manager=None, 281 internal_config=InternalConfig(), # type: ignore [call-arg] 282 ) 283 284 @abstractmethod 285 def read_records( 286 self, 287 sync_mode: SyncMode, 288 cursor_field: Optional[List[str]] = None, 289 stream_slice: Optional[Mapping[str, Any]] = None, 290 stream_state: Optional[Mapping[str, Any]] = None, 291 ) -> Iterable[StreamData]: 292 """ 293 This method should be overridden by subclasses to read records based on the inputs 294 """ 295 296 @lru_cache(maxsize=None) 297 def get_json_schema(self) -> Mapping[str, Any]: 298 """ 299 :return: A dict of the JSON schema representing this stream. 300 301 The default implementation of this method looks for a JSONSchema file with the same name as this stream's "name" property. 302 Override as needed. 303 """ 304 # TODO show an example of using pydantic to define the JSON schema, or reading an OpenAPI spec 305 return ResourceSchemaLoader(package_name_from_class(self.__class__)).get_schema(self.name) 306 307 def as_airbyte_stream(self) -> AirbyteStream: 308 stream = AirbyteStream( 309 name=self.name, 310 json_schema=dict(self.get_json_schema()), 311 supported_sync_modes=[SyncMode.full_refresh], 312 is_resumable=self.is_resumable, 313 ) 314 315 if self.namespace: 316 stream.namespace = self.namespace 317 318 # If we can offer incremental we always should. RFR is always less reliable than incremental which uses a real cursor value 319 if self.supports_incremental: 320 stream.source_defined_cursor = self.source_defined_cursor 321 stream.supported_sync_modes.append(SyncMode.incremental) 322 stream.default_cursor_field = self._wrapped_cursor_field() 323 324 keys = Stream._wrapped_primary_key(self.primary_key) 325 if keys and len(keys) > 0: 326 stream.source_defined_primary_key = keys 327 328 return stream 329 330 @property 331 def supports_incremental(self) -> bool: 332 """ 333 :return: True if this stream supports incrementally reading data 334 """ 335 return len(self._wrapped_cursor_field()) > 0 336 337 @property 338 def is_resumable(self) -> bool: 339 """ 340 :return: True if this stream allows the checkpointing of sync progress and can resume from it on subsequent attempts. 341 This differs from supports_incremental because certain kinds of streams like those supporting resumable full refresh 342 can checkpoint progress in between attempts for improved fault tolerance. However, they will start from the beginning 343 on the next sync job. 344 """ 345 if self.supports_incremental: 346 return True 347 if self.has_multiple_slices: 348 # We temporarily gate substream to not support RFR because puts a pretty high burden on connector developers 349 # to structure stream state in a very specific way. We also can't check for issubclass(HttpSubStream) because 350 # not all substreams implement the interface and it would be a circular dependency so we use parent as a surrogate 351 return False 352 elif hasattr(type(self), "state") and getattr(type(self), "state").fset is not None: 353 # Modern case where a stream manages state using getter/setter 354 return True 355 else: 356 # Legacy case where the CDK manages state via the get_updated_state() method. This is determined by checking if 357 # the stream's get_updated_state() differs from the Stream class and therefore has been overridden 358 return type(self).get_updated_state != Stream.get_updated_state 359 360 def _wrapped_cursor_field(self) -> List[str]: 361 return [self.cursor_field] if isinstance(self.cursor_field, str) else self.cursor_field 362 363 @property 364 def cursor_field(self) -> Union[str, List[str]]: 365 """ 366 Override to return the default cursor field used by this stream e.g: an API entity might always use created_at as the cursor field. 367 :return: The name of the field used as a cursor. If the cursor is nested, return an array consisting of the path to the cursor. 368 """ 369 return [] 370 371 @property 372 def namespace(self) -> Optional[str]: 373 """ 374 Override to return the namespace of this stream, e.g. the Postgres schema which this stream will emit records for. 375 :return: A string containing the name of the namespace. 376 """ 377 return None 378 379 @property 380 def source_defined_cursor(self) -> bool: 381 """ 382 Return False if the cursor can be configured by the user. 383 """ 384 return True 385 386 @property 387 def exit_on_rate_limit(self) -> bool: 388 """Exit on rate limit getter, should return bool value. False if the stream will retry endlessly when rate limited.""" 389 return self._exit_on_rate_limit 390 391 @exit_on_rate_limit.setter 392 def exit_on_rate_limit(self, value: bool) -> None: 393 """Exit on rate limit setter, accept bool value.""" 394 self._exit_on_rate_limit = value 395 396 @property 397 @abstractmethod 398 def primary_key(self) -> Optional[Union[str, List[str], List[List[str]]]]: 399 """ 400 :return: string if single primary key, list of strings if composite primary key, list of list of strings if composite primary key consisting of nested fields. 401 If the stream has no primary keys, return None. 402 """ 403 404 def stream_slices( 405 self, 406 *, 407 sync_mode: SyncMode, 408 cursor_field: Optional[List[str]] = None, 409 stream_state: Optional[Mapping[str, Any]] = None, 410 ) -> Iterable[Optional[Mapping[str, Any]]]: 411 """ 412 Override to define the slices for this stream. See the stream slicing section of the docs for more information. 413 414 :param sync_mode: 415 :param cursor_field: 416 :param stream_state: 417 :return: 418 """ 419 yield StreamSlice(partition={}, cursor_slice={}) 420 421 @property 422 def state_checkpoint_interval(self) -> Optional[int]: 423 """ 424 Decides how often to checkpoint state (i.e: emit a STATE message). E.g: if this returns a value of 100, then state is persisted after reading 425 100 records, then 200, 300, etc.. A good default value is 1000 although your mileage may vary depending on the underlying data source. 426 427 Checkpointing a stream avoids re-reading records in the case a sync is failed or cancelled. 428 429 return None if state should not be checkpointed e.g: because records returned from the underlying data source are not returned in 430 ascending order with respect to the cursor field. This can happen if the source does not support reading records in ascending order of 431 created_at date (or whatever the cursor is). In those cases, state must only be saved once the full stream has been read. 432 """ 433 return None 434 435 # Commented-out to avoid any runtime penalty, since this is used in a hot per-record codepath. 436 # To be evaluated for re-introduction here: https://github.com/airbytehq/airbyte-python-cdk/issues/116 437 # @deprecated( 438 # "Deprecated method `get_updated_state` as of CDK version 0.1.49. " 439 # "Please use explicit state property instead, see `IncrementalMixin` docs." 440 # ) 441 def get_updated_state( 442 self, current_stream_state: MutableMapping[str, Any], latest_record: Mapping[str, Any] 443 ) -> MutableMapping[str, Any]: 444 """DEPRECATED. Please use explicit state property instead, see `IncrementalMixin` docs. 445 446 Override to extract state from the latest record. Needed to implement incremental sync. 447 448 Inspects the latest record extracted from the data source and the current state object and return an updated state object. 449 450 For example: if the state object is based on created_at timestamp, and the current state is {'created_at': 10}, and the latest_record is 451 {'name': 'octavia', 'created_at': 20 } then this method would return {'created_at': 20} to indicate state should be updated to this object. 452 453 :param current_stream_state: The stream's current state object 454 :param latest_record: The latest record extracted from the stream 455 :return: An updated state object 456 """ 457 return {} 458 459 def get_cursor(self) -> Optional[Cursor]: 460 """ 461 A Cursor is an interface that a stream can implement to manage how its internal state is read and updated while 462 reading records. Historically, Python connectors had no concept of a cursor to manage state. Python streams need 463 to define a cursor implementation and override this method to manage state through a Cursor. 464 """ 465 return self.cursor 466 467 def _get_checkpoint_reader( 468 self, 469 logger: logging.Logger, 470 cursor_field: Optional[List[str]], 471 sync_mode: SyncMode, 472 stream_state: MutableMapping[str, Any], 473 ) -> CheckpointReader: 474 mappings_or_slices = self.stream_slices( 475 cursor_field=cursor_field, 476 sync_mode=sync_mode, # todo: change this interface to no longer rely on sync_mode for behavior 477 stream_state=stream_state, 478 ) 479 480 # Because of poor foresight, we wrote the default Stream.stream_slices() method to return [None] which is confusing and 481 # has now normalized this behavior for connector developers. Now some connectors return [None]. This is objectively 482 # misleading and a more ideal interface is [{}] to indicate we still want to iterate over one slice, but with no 483 # specific slice values. None is bad, and now I feel bad that I have to write this hack. 484 if mappings_or_slices == [None]: 485 mappings_or_slices = [{}] 486 487 slices_iterable_copy, iterable_for_detecting_format = itertools.tee(mappings_or_slices, 2) 488 stream_classification = self._classify_stream( 489 mappings_or_slices=iterable_for_detecting_format 490 ) 491 492 # Streams that override has_multiple_slices are explicitly indicating that they will iterate over 493 # multiple partitions. Inspecting slices to automatically apply the correct cursor is only needed as 494 # a backup. So if this value was already assigned to True by the stream, we don't need to reassign it 495 self.has_multiple_slices = ( 496 self.has_multiple_slices or stream_classification.has_multiple_slices 497 ) 498 499 cursor = self.get_cursor() 500 if cursor: 501 cursor.set_initial_state(stream_state=stream_state) 502 503 checkpoint_mode = self._checkpoint_mode 504 505 if cursor and stream_classification.is_legacy_format: 506 return LegacyCursorBasedCheckpointReader( 507 stream_slices=slices_iterable_copy, cursor=cursor, read_state_from_cursor=True 508 ) 509 elif cursor: 510 return CursorBasedCheckpointReader( 511 stream_slices=slices_iterable_copy, 512 cursor=cursor, 513 read_state_from_cursor=checkpoint_mode == CheckpointMode.RESUMABLE_FULL_REFRESH, 514 ) 515 elif checkpoint_mode == CheckpointMode.RESUMABLE_FULL_REFRESH: 516 # Resumable full refresh readers rely on the stream state dynamically being updated during pagination and does 517 # not iterate over a static set of slices. 518 return ResumableFullRefreshCheckpointReader(stream_state=stream_state) 519 elif checkpoint_mode == CheckpointMode.INCREMENTAL: 520 return IncrementalCheckpointReader( 521 stream_slices=slices_iterable_copy, stream_state=stream_state 522 ) 523 else: 524 return FullRefreshCheckpointReader(stream_slices=slices_iterable_copy) 525 526 @property 527 def _checkpoint_mode(self) -> CheckpointMode: 528 if self.is_resumable and len(self._wrapped_cursor_field()) > 0: 529 return CheckpointMode.INCREMENTAL 530 elif self.is_resumable: 531 return CheckpointMode.RESUMABLE_FULL_REFRESH 532 else: 533 return CheckpointMode.FULL_REFRESH 534 535 @staticmethod 536 def _classify_stream( 537 mappings_or_slices: Iterator[Optional[Union[Mapping[str, Any], StreamSlice]]], 538 ) -> StreamClassification: 539 """ 540 This is a bit of a crazy solution, but also the only way we can detect certain attributes about the stream since Python 541 streams do not follow consistent implementation patterns. We care about the following two attributes: 542 - is_substream: Helps to incrementally release changes since substreams w/ parents are much more complicated. Also 543 helps de-risk the release of changes that might impact all connectors 544 - uses_legacy_slice_format: Since the checkpoint reader must manage a complex state object, we opted to have it always 545 use the structured StreamSlice object. However, this requires backwards compatibility with Python sources that only 546 support the legacy mapping object 547 548 Both attributes can eventually be deprecated once stream's define this method deleted once substreams have been implemented and 549 legacy connectors all adhere to the StreamSlice object. 550 """ 551 if not mappings_or_slices: 552 raise ValueError("A stream should always have at least one slice") 553 try: 554 next_slice = next(mappings_or_slices) 555 if isinstance(next_slice, StreamSlice) and next_slice == StreamSlice( 556 partition={}, cursor_slice={} 557 ): 558 is_legacy_format = False 559 slice_has_value = False 560 elif next_slice == {}: 561 is_legacy_format = True 562 slice_has_value = False 563 elif isinstance(next_slice, StreamSlice): 564 is_legacy_format = False 565 slice_has_value = True 566 else: 567 is_legacy_format = True 568 slice_has_value = True 569 except StopIteration: 570 # If the stream has no slices, the format ultimately does not matter since no data will get synced. This is technically 571 # a valid case because it is up to the stream to define its slicing behavior 572 return StreamClassification(is_legacy_format=False, has_multiple_slices=False) 573 574 if slice_has_value: 575 # If the first slice contained a partition value from the result of stream_slices(), this is a substream that might 576 # have multiple parent records to iterate over 577 return StreamClassification( 578 is_legacy_format=is_legacy_format, has_multiple_slices=slice_has_value 579 ) 580 581 try: 582 # If stream_slices() returns multiple slices, this is also a substream that can potentially generate empty slices 583 next(mappings_or_slices) 584 return StreamClassification(is_legacy_format=is_legacy_format, has_multiple_slices=True) 585 except StopIteration: 586 # If the result of stream_slices() only returns a single empty stream slice, then we know this is a regular stream 587 return StreamClassification( 588 is_legacy_format=is_legacy_format, has_multiple_slices=False 589 ) 590 591 def log_stream_sync_configuration(self) -> None: 592 """ 593 Logs the configuration of this stream. 594 """ 595 self.logger.debug( 596 f"Syncing stream instance: {self.name}", 597 extra={ 598 "primary_key": self.primary_key, 599 "cursor_field": self.cursor_field, 600 }, 601 ) 602 603 @staticmethod 604 def _wrapped_primary_key( 605 keys: Optional[Union[str, List[str], List[List[str]]]], 606 ) -> Optional[List[List[str]]]: 607 """ 608 :return: wrap the primary_key property in a list of list of strings required by the Airbyte Stream object. 609 """ 610 if not keys: 611 return None 612 613 if isinstance(keys, str): 614 return [[keys]] 615 elif isinstance(keys, list): 616 wrapped_keys = [] 617 for component in keys: 618 if isinstance(component, str): 619 wrapped_keys.append([component]) 620 elif isinstance(component, list): 621 wrapped_keys.append(component) 622 else: 623 raise ValueError(f"Element must be either list or str. Got: {type(component)}") 624 return wrapped_keys 625 else: 626 raise ValueError(f"Element must be either list or str. Got: {type(keys)}") 627 628 def _observe_state( 629 self, checkpoint_reader: CheckpointReader, stream_state: Optional[Mapping[str, Any]] = None 630 ) -> None: 631 """ 632 Convenience method that attempts to read the Stream's state using the recommended way of connector's managing their 633 own state via state setter/getter. But if we get back an AttributeError, then the legacy Stream.get_updated_state() 634 method is used as a fallback method. 635 """ 636 637 # This is an inversion of the original logic that used to try state getter/setters first. As part of the work to 638 # automatically apply resumable full refresh to all streams, all HttpStream classes implement default state 639 # getter/setter methods, we should default to only using the incoming stream_state parameter value is {} which 640 # indicates the stream does not override the default get_updated_state() implementation. When the default method 641 # is not overridden, then the stream defers to self.state getter 642 if stream_state: 643 checkpoint_reader.observe(stream_state) 644 elif type(self).get_updated_state == Stream.get_updated_state: 645 # We only default to the state getter/setter if the stream does not use the legacy get_updated_state() method 646 try: 647 new_state = self.state # type: ignore # This will always exist on HttpStreams, but may not for Stream 648 if new_state: 649 checkpoint_reader.observe(new_state) 650 except AttributeError: 651 pass 652 653 def _checkpoint_state( # type: ignore # ignoring typing for ConnectorStateManager because of circular dependencies 654 self, 655 stream_state: Mapping[str, Any], 656 state_manager, 657 ) -> AirbyteMessage: 658 # todo: This can be consolidated into one ConnectorStateManager.update_and_create_state_message() method, but I want 659 # to reduce changes right now and this would span concurrent as well 660 state_manager.update_state_for_stream(self.name, self.namespace, stream_state) 661 return state_manager.create_state_message(self.name, self.namespace) # type: ignore [no-any-return] 662 663 @property 664 def configured_json_schema(self) -> Optional[Dict[str, Any]]: 665 """ 666 This property is set from the read method. 667 668 :return Optional[Dict]: JSON schema from configured catalog if provided, otherwise None. 669 """ 670 return self._configured_json_schema 671 672 @configured_json_schema.setter 673 def configured_json_schema(self, json_schema: Dict[str, Any]) -> None: 674 self._configured_json_schema = self._filter_schema_invalid_properties(json_schema) 675 676 def _filter_schema_invalid_properties( 677 self, configured_catalog_json_schema: Dict[str, Any] 678 ) -> Dict[str, Any]: 679 """ 680 Filters the properties in json_schema that are not present in the stream schema. 681 Configured Schemas can have very old fields, so we need to housekeeping ourselves. 682 """ 683 configured_schema: Any = configured_catalog_json_schema.get("properties", {}) 684 stream_schema_properties: Any = self.get_json_schema().get("properties", {}) 685 686 configured_keys = configured_schema.keys() 687 stream_keys = stream_schema_properties.keys() 688 invalid_properties = configured_keys - stream_keys 689 if not invalid_properties: 690 return configured_catalog_json_schema 691 692 self.logger.warning( 693 f"Stream {self.name}: the following fields are deprecated and cannot be synced. {invalid_properties}. Refresh the connection's source schema to resolve this warning." 694 ) 695 696 valid_configured_schema_properties_keys = stream_keys & configured_keys 697 valid_configured_schema_properties = {} 698 699 for configured_schema_property in valid_configured_schema_properties_keys: 700 valid_configured_schema_properties[configured_schema_property] = ( 701 stream_schema_properties[configured_schema_property] 702 ) 703 704 return {**configured_catalog_json_schema, "properties": valid_configured_schema_properties}
Base abstract class for an Airbyte Stream. Makes no assumption of the Stream's underlying transport protocol.
139 @cached_property 140 def name(self) -> str: 141 """ 142 :return: Stream name. By default this is the implementing class name, but it can be overridden as needed. 143 """ 144 return casing.camel_to_snake(self.__class__.__name__)
Returns
Stream name. By default this is the implementing class name, but it can be overridden as needed.
146 def get_error_display_message(self, exception: BaseException) -> Optional[str]: 147 """ 148 Retrieves the user-friendly display message that corresponds to an exception. 149 This will be called when encountering an exception while reading records from the stream, and used to build the AirbyteTraceMessage. 150 151 The default implementation of this method does not return user-friendly messages for any exception type, but it should be overriden as needed. 152 153 :param exception: The exception that was raised 154 :return: A user-friendly message that indicates the cause of the error 155 """ 156 return None
Retrieves the user-friendly display message that corresponds to an exception. This will be called when encountering an exception while reading records from the stream, and used to build the AirbyteTraceMessage.
The default implementation of this method does not return user-friendly messages for any exception type, but it should be overriden as needed.
Parameters
- exception: The exception that was raised
Returns
A user-friendly message that indicates the cause of the error
158 def read( # type: ignore # ignoring typing for ConnectorStateManager because of circular dependencies 159 self, 160 configured_stream: ConfiguredAirbyteStream, 161 logger: logging.Logger, 162 slice_logger: SliceLogger, 163 stream_state: MutableMapping[str, Any], 164 state_manager, 165 internal_config: InternalConfig, 166 ) -> Iterable[StreamData]: 167 sync_mode = configured_stream.sync_mode 168 cursor_field = configured_stream.cursor_field 169 self.configured_json_schema = configured_stream.stream.json_schema 170 171 # WARNING: When performing a read() that uses incoming stream state, we MUST use the self.state that is defined as 172 # opposed to the incoming stream_state value. Because some connectors like ones using the file-based CDK modify 173 # state before setting the value on the Stream attribute, the most up-to-date state is derived from Stream.state 174 # instead of the stream_state parameter. This does not apply to legacy connectors using get_updated_state(). 175 try: 176 stream_state = self.state # type: ignore # we know the field might not exist... 177 except AttributeError: 178 pass 179 180 should_checkpoint = bool(state_manager) 181 checkpoint_reader = self._get_checkpoint_reader( 182 logger=logger, cursor_field=cursor_field, sync_mode=sync_mode, stream_state=stream_state 183 ) 184 185 next_slice = checkpoint_reader.next() 186 record_counter = 0 187 stream_state_tracker = copy.deepcopy(stream_state) 188 while next_slice is not None: 189 if slice_logger.should_log_slice_message(logger): 190 yield slice_logger.create_slice_log_message(next_slice) 191 records = self.read_records( 192 sync_mode=sync_mode, # todo: change this interface to no longer rely on sync_mode for behavior 193 stream_slice=next_slice, 194 stream_state=stream_state, 195 cursor_field=cursor_field or None, 196 ) 197 for record_data_or_message in records: 198 yield record_data_or_message 199 if isinstance(record_data_or_message, Mapping) or ( 200 hasattr(record_data_or_message, "type") 201 and record_data_or_message.type == MessageType.RECORD 202 ): 203 record_data = ( 204 record_data_or_message 205 if isinstance(record_data_or_message, Mapping) 206 else record_data_or_message.record 207 ) 208 209 # Thanks I hate it. RFR fundamentally doesn't fit with the concept of the legacy Stream.get_updated_state() 210 # method because RFR streams rely on pagination as a cursor. Stream.get_updated_state() was designed to make 211 # the CDK manage state using specifically the last seen record. don't @ brian.lai 212 # 213 # Also, because the legacy incremental state case decouples observing incoming records from emitting state, it 214 # requires that we separate CheckpointReader.observe() and CheckpointReader.get_checkpoint() which could 215 # otherwise be combined. 216 if self.cursor_field: 217 # Some connectors have streams that implement get_updated_state(), but do not define a cursor_field. This 218 # should be fixed on the stream implementation, but we should also protect against this in the CDK as well 219 stream_state_tracker = self.get_updated_state( 220 stream_state_tracker, 221 record_data, # type: ignore [arg-type] 222 ) 223 self._observe_state(checkpoint_reader, stream_state_tracker) 224 record_counter += 1 225 226 checkpoint_interval = self.state_checkpoint_interval 227 if ( 228 should_checkpoint 229 and checkpoint_interval 230 and record_counter % checkpoint_interval == 0 231 ): 232 checkpoint = checkpoint_reader.get_checkpoint() 233 if checkpoint: 234 airbyte_state_message = self._checkpoint_state( 235 checkpoint, state_manager=state_manager 236 ) 237 yield airbyte_state_message 238 239 if internal_config.is_limit_reached(record_counter): 240 break 241 self._observe_state(checkpoint_reader) 242 checkpoint_state = checkpoint_reader.get_checkpoint() 243 if should_checkpoint and checkpoint_state is not None: 244 airbyte_state_message = self._checkpoint_state( 245 checkpoint_state, state_manager=state_manager 246 ) 247 yield airbyte_state_message 248 249 next_slice = checkpoint_reader.next() 250 251 checkpoint = checkpoint_reader.get_checkpoint() 252 if should_checkpoint and checkpoint is not None: 253 airbyte_state_message = self._checkpoint_state(checkpoint, state_manager=state_manager) 254 yield airbyte_state_message
256 def read_only_records(self, state: Optional[Mapping[str, Any]] = None) -> Iterable[StreamData]: 257 """ 258 Helper method that performs a read on a stream with an optional state and emits records. If the parent stream supports 259 incremental, this operation does not update the stream's internal state (if it uses the modern state setter/getter) 260 or emit state messages. 261 """ 262 263 configured_stream = ConfiguredAirbyteStream( 264 stream=AirbyteStream( 265 name=self.name, 266 json_schema={}, 267 supported_sync_modes=[SyncMode.full_refresh, SyncMode.incremental], 268 ), 269 sync_mode=SyncMode.incremental if state else SyncMode.full_refresh, 270 destination_sync_mode=DestinationSyncMode.append, 271 ) 272 273 yield from self.read( 274 configured_stream=configured_stream, 275 logger=self.logger, 276 slice_logger=DebugSliceLogger(), 277 stream_state=dict(state) 278 if state 279 else {}, # read() expects MutableMapping instead of Mapping which is used more often 280 state_manager=None, 281 internal_config=InternalConfig(), # type: ignore [call-arg] 282 )
Helper method that performs a read on a stream with an optional state and emits records. If the parent stream supports incremental, this operation does not update the stream's internal state (if it uses the modern state setter/getter) or emit state messages.
284 @abstractmethod 285 def read_records( 286 self, 287 sync_mode: SyncMode, 288 cursor_field: Optional[List[str]] = None, 289 stream_slice: Optional[Mapping[str, Any]] = None, 290 stream_state: Optional[Mapping[str, Any]] = None, 291 ) -> Iterable[StreamData]: 292 """ 293 This method should be overridden by subclasses to read records based on the inputs 294 """
This method should be overridden by subclasses to read records based on the inputs
296 @lru_cache(maxsize=None) 297 def get_json_schema(self) -> Mapping[str, Any]: 298 """ 299 :return: A dict of the JSON schema representing this stream. 300 301 The default implementation of this method looks for a JSONSchema file with the same name as this stream's "name" property. 302 Override as needed. 303 """ 304 # TODO show an example of using pydantic to define the JSON schema, or reading an OpenAPI spec 305 return ResourceSchemaLoader(package_name_from_class(self.__class__)).get_schema(self.name)
Returns
A dict of the JSON schema representing this stream.
The default implementation of this method looks for a JSONSchema file with the same name as this stream's "name" property. Override as needed.
307 def as_airbyte_stream(self) -> AirbyteStream: 308 stream = AirbyteStream( 309 name=self.name, 310 json_schema=dict(self.get_json_schema()), 311 supported_sync_modes=[SyncMode.full_refresh], 312 is_resumable=self.is_resumable, 313 ) 314 315 if self.namespace: 316 stream.namespace = self.namespace 317 318 # If we can offer incremental we always should. RFR is always less reliable than incremental which uses a real cursor value 319 if self.supports_incremental: 320 stream.source_defined_cursor = self.source_defined_cursor 321 stream.supported_sync_modes.append(SyncMode.incremental) 322 stream.default_cursor_field = self._wrapped_cursor_field() 323 324 keys = Stream._wrapped_primary_key(self.primary_key) 325 if keys and len(keys) > 0: 326 stream.source_defined_primary_key = keys 327 328 return stream
330 @property 331 def supports_incremental(self) -> bool: 332 """ 333 :return: True if this stream supports incrementally reading data 334 """ 335 return len(self._wrapped_cursor_field()) > 0
Returns
True if this stream supports incrementally reading data
337 @property 338 def is_resumable(self) -> bool: 339 """ 340 :return: True if this stream allows the checkpointing of sync progress and can resume from it on subsequent attempts. 341 This differs from supports_incremental because certain kinds of streams like those supporting resumable full refresh 342 can checkpoint progress in between attempts for improved fault tolerance. However, they will start from the beginning 343 on the next sync job. 344 """ 345 if self.supports_incremental: 346 return True 347 if self.has_multiple_slices: 348 # We temporarily gate substream to not support RFR because puts a pretty high burden on connector developers 349 # to structure stream state in a very specific way. We also can't check for issubclass(HttpSubStream) because 350 # not all substreams implement the interface and it would be a circular dependency so we use parent as a surrogate 351 return False 352 elif hasattr(type(self), "state") and getattr(type(self), "state").fset is not None: 353 # Modern case where a stream manages state using getter/setter 354 return True 355 else: 356 # Legacy case where the CDK manages state via the get_updated_state() method. This is determined by checking if 357 # the stream's get_updated_state() differs from the Stream class and therefore has been overridden 358 return type(self).get_updated_state != Stream.get_updated_state
Returns
True if this stream allows the checkpointing of sync progress and can resume from it on subsequent attempts. This differs from supports_incremental because certain kinds of streams like those supporting resumable full refresh can checkpoint progress in between attempts for improved fault tolerance. However, they will start from the beginning on the next sync job.
363 @property 364 def cursor_field(self) -> Union[str, List[str]]: 365 """ 366 Override to return the default cursor field used by this stream e.g: an API entity might always use created_at as the cursor field. 367 :return: The name of the field used as a cursor. If the cursor is nested, return an array consisting of the path to the cursor. 368 """ 369 return []
Override to return the default cursor field used by this stream e.g: an API entity might always use created_at as the cursor field.
Returns
The name of the field used as a cursor. If the cursor is nested, return an array consisting of the path to the cursor.
371 @property 372 def namespace(self) -> Optional[str]: 373 """ 374 Override to return the namespace of this stream, e.g. the Postgres schema which this stream will emit records for. 375 :return: A string containing the name of the namespace. 376 """ 377 return None
Override to return the namespace of this stream, e.g. the Postgres schema which this stream will emit records for.
Returns
A string containing the name of the namespace.
379 @property 380 def source_defined_cursor(self) -> bool: 381 """ 382 Return False if the cursor can be configured by the user. 383 """ 384 return True
Return False if the cursor can be configured by the user.
386 @property 387 def exit_on_rate_limit(self) -> bool: 388 """Exit on rate limit getter, should return bool value. False if the stream will retry endlessly when rate limited.""" 389 return self._exit_on_rate_limit
Exit on rate limit getter, should return bool value. False if the stream will retry endlessly when rate limited.
396 @property 397 @abstractmethod 398 def primary_key(self) -> Optional[Union[str, List[str], List[List[str]]]]: 399 """ 400 :return: string if single primary key, list of strings if composite primary key, list of list of strings if composite primary key consisting of nested fields. 401 If the stream has no primary keys, return None. 402 """
Returns
string if single primary key, list of strings if composite primary key, list of list of strings if composite primary key consisting of nested fields. If the stream has no primary keys, return None.
404 def stream_slices( 405 self, 406 *, 407 sync_mode: SyncMode, 408 cursor_field: Optional[List[str]] = None, 409 stream_state: Optional[Mapping[str, Any]] = None, 410 ) -> Iterable[Optional[Mapping[str, Any]]]: 411 """ 412 Override to define the slices for this stream. See the stream slicing section of the docs for more information. 413 414 :param sync_mode: 415 :param cursor_field: 416 :param stream_state: 417 :return: 418 """ 419 yield StreamSlice(partition={}, cursor_slice={})
Override to define the slices for this stream. See the stream slicing section of the docs for more information.
Parameters
- sync_mode:
- cursor_field:
- stream_state:
Returns
421 @property 422 def state_checkpoint_interval(self) -> Optional[int]: 423 """ 424 Decides how often to checkpoint state (i.e: emit a STATE message). E.g: if this returns a value of 100, then state is persisted after reading 425 100 records, then 200, 300, etc.. A good default value is 1000 although your mileage may vary depending on the underlying data source. 426 427 Checkpointing a stream avoids re-reading records in the case a sync is failed or cancelled. 428 429 return None if state should not be checkpointed e.g: because records returned from the underlying data source are not returned in 430 ascending order with respect to the cursor field. This can happen if the source does not support reading records in ascending order of 431 created_at date (or whatever the cursor is). In those cases, state must only be saved once the full stream has been read. 432 """ 433 return None
Decides how often to checkpoint state (i.e: emit a STATE message). E.g: if this returns a value of 100, then state is persisted after reading 100 records, then 200, 300, etc.. A good default value is 1000 although your mileage may vary depending on the underlying data source.
Checkpointing a stream avoids re-reading records in the case a sync is failed or cancelled.
return None if state should not be checkpointed e.g: because records returned from the underlying data source are not returned in ascending order with respect to the cursor field. This can happen if the source does not support reading records in ascending order of created_at date (or whatever the cursor is). In those cases, state must only be saved once the full stream has been read.
441 def get_updated_state( 442 self, current_stream_state: MutableMapping[str, Any], latest_record: Mapping[str, Any] 443 ) -> MutableMapping[str, Any]: 444 """DEPRECATED. Please use explicit state property instead, see `IncrementalMixin` docs. 445 446 Override to extract state from the latest record. Needed to implement incremental sync. 447 448 Inspects the latest record extracted from the data source and the current state object and return an updated state object. 449 450 For example: if the state object is based on created_at timestamp, and the current state is {'created_at': 10}, and the latest_record is 451 {'name': 'octavia', 'created_at': 20 } then this method would return {'created_at': 20} to indicate state should be updated to this object. 452 453 :param current_stream_state: The stream's current state object 454 :param latest_record: The latest record extracted from the stream 455 :return: An updated state object 456 """ 457 return {}
DEPRECATED. Please use explicit state property instead, see IncrementalMixin docs.
Override to extract state from the latest record. Needed to implement incremental sync.
Inspects the latest record extracted from the data source and the current state object and return an updated state object.
For example: if the state object is based on created_at timestamp, and the current state is {'created_at': 10}, and the latest_record is {'name': 'octavia', 'created_at': 20 } then this method would return {'created_at': 20} to indicate state should be updated to this object.
Parameters
- current_stream_state: The stream's current state object
- latest_record: The latest record extracted from the stream
Returns
An updated state object
459 def get_cursor(self) -> Optional[Cursor]: 460 """ 461 A Cursor is an interface that a stream can implement to manage how its internal state is read and updated while 462 reading records. Historically, Python connectors had no concept of a cursor to manage state. Python streams need 463 to define a cursor implementation and override this method to manage state through a Cursor. 464 """ 465 return self.cursor
A Cursor is an interface that a stream can implement to manage how its internal state is read and updated while reading records. Historically, Python connectors had no concept of a cursor to manage state. Python streams need to define a cursor implementation and override this method to manage state through a Cursor.
591 def log_stream_sync_configuration(self) -> None: 592 """ 593 Logs the configuration of this stream. 594 """ 595 self.logger.debug( 596 f"Syncing stream instance: {self.name}", 597 extra={ 598 "primary_key": self.primary_key, 599 "cursor_field": self.cursor_field, 600 }, 601 )
Logs the configuration of this stream.
663 @property 664 def configured_json_schema(self) -> Optional[Dict[str, Any]]: 665 """ 666 This property is set from the read method. 667 668 :return Optional[Dict]: JSON schema from configured catalog if provided, otherwise None. 669 """ 670 return self._configured_json_schema
This property is set from the read method.
Returns
JSON schema from configured catalog if provided, otherwise None.
52def package_name_from_class(cls: object) -> str: 53 """Find the package name given a class name""" 54 module = inspect.getmodule(cls) 55 if module is not None: 56 return module.__name__.split(".")[0] 57 else: 58 raise ValueError(f"Could not find package name for class {cls}")
Find the package name given a class name
26class AirbyteTracedException(Exception): 27 """ 28 An exception that should be emitted as an AirbyteTraceMessage 29 """ 30 31 def __init__( 32 self, 33 internal_message: Optional[str] = None, 34 message: Optional[str] = None, 35 failure_type: FailureType = FailureType.system_error, 36 exception: Optional[BaseException] = None, 37 stream_descriptor: Optional[StreamDescriptor] = None, 38 ): 39 """ 40 :param internal_message: the internal error that caused the failure 41 :param message: a user-friendly message that indicates the cause of the error 42 :param failure_type: the type of error 43 :param exception: the exception that caused the error, from which the stack trace should be retrieved 44 :param stream_descriptor: describe the stream from which the exception comes from 45 """ 46 self.internal_message = internal_message 47 self.message = message 48 self.failure_type = failure_type 49 self._exception = exception 50 self._stream_descriptor = stream_descriptor 51 super().__init__(internal_message) 52 53 def __str__(self) -> str: 54 """Return the user-facing message, falling back to internal_message.""" 55 if self.message is not None: 56 return self.message 57 if self.internal_message is not None: 58 return self.internal_message 59 return "" 60 61 def as_airbyte_message( 62 self, stream_descriptor: Optional[StreamDescriptor] = None 63 ) -> AirbyteMessage: 64 """ 65 Builds an AirbyteTraceMessage from the exception 66 67 :param stream_descriptor is deprecated, please use the stream_description in `__init__ or `from_exception`. If many 68 stream_descriptors are defined, the one from `as_airbyte_message` will be discarded. 69 """ 70 now_millis = time.time_ns() // 1_000_000 71 72 trace_exc = self._exception or self 73 stack_trace_str = "".join(traceback.TracebackException.from_exception(trace_exc).format()) 74 75 trace_message = AirbyteTraceMessage( 76 type=TraceType.ERROR, 77 emitted_at=now_millis, 78 error=AirbyteErrorTraceMessage( 79 message=self.message 80 or "Something went wrong in the connector. See the logs for more details.", 81 internal_message=self.internal_message, 82 failure_type=self.failure_type, 83 stack_trace=stack_trace_str, 84 stream_descriptor=self._stream_descriptor 85 if self._stream_descriptor is not None 86 else stream_descriptor, 87 ), 88 ) 89 90 return AirbyteMessage(type=MessageType.TRACE, trace=trace_message) 91 92 def as_connection_status_message(self) -> Optional[AirbyteMessage]: 93 """Build a failed CONNECTION_STATUS from a config error, with secrets masked. 94 95 The message can quote config values, as a config validation error does, so it goes 96 through the same secret filter as logs and traces. 97 """ 98 if self.failure_type == FailureType.config_error: 99 return AirbyteMessage( 100 type=MessageType.CONNECTION_STATUS, 101 connectionStatus=AirbyteConnectionStatus( 102 status=Status.FAILED, 103 message=filter_secrets(self.message) if self.message else self.message, 104 ), 105 ) 106 return None 107 108 def emit_message(self) -> None: 109 """ 110 Prints the exception as an AirbyteTraceMessage. 111 Note that this will be called automatically on uncaught exceptions when using the airbyte_cdk entrypoint. 112 """ 113 message = orjson.dumps(AirbyteMessageSerializer.dump(self.as_airbyte_message())).decode() 114 filtered_message = filter_secrets(message) 115 print(filtered_message) 116 117 @classmethod 118 def from_exception( 119 cls, 120 exc: BaseException, 121 stream_descriptor: Optional[StreamDescriptor] = None, 122 *args: Any, 123 **kwargs: Any, 124 ) -> "AirbyteTracedException": 125 """ 126 Helper to create an AirbyteTracedException from an existing exception 127 :param exc: the exception that caused the error 128 :param stream_descriptor: describe the stream from which the exception comes from 129 """ 130 if isinstance(exc, AirbyteTracedException): 131 internal_message = exc.internal_message 132 # Preserve the original user-facing message if the caller didn't provide one 133 if "message" not in kwargs: 134 kwargs["message"] = exc.message 135 else: 136 internal_message = str(exc) 137 return cls( 138 internal_message=internal_message, 139 exception=exc, 140 stream_descriptor=stream_descriptor, 141 *args, 142 **kwargs, 143 ) # type: ignore # ignoring because of args and kwargs 144 145 def as_sanitized_airbyte_message( 146 self, stream_descriptor: Optional[StreamDescriptor] = None 147 ) -> AirbyteMessage: 148 """ 149 Builds an AirbyteTraceMessage from the exception and sanitizes any secrets from the message body 150 151 :param stream_descriptor is deprecated, please use the stream_description in `__init__ or `from_exception`. If many 152 stream_descriptors are defined, the one from `as_sanitized_airbyte_message` will be discarded. 153 """ 154 error_message = self.as_airbyte_message(stream_descriptor=stream_descriptor) 155 if error_message.trace.error.message: # type: ignore[union-attr] # AirbyteMessage with MessageType.TRACE has AirbyteTraceMessage 156 error_message.trace.error.message = filter_secrets( # type: ignore[union-attr] 157 error_message.trace.error.message, # type: ignore[union-attr] 158 ) 159 if error_message.trace.error.internal_message: # type: ignore[union-attr] # AirbyteMessage with MessageType.TRACE has AirbyteTraceMessage 160 error_message.trace.error.internal_message = filter_secrets( # type: ignore[union-attr] # AirbyteMessage with MessageType.TRACE has AirbyteTraceMessage 161 error_message.trace.error.internal_message # type: ignore[union-attr] # AirbyteMessage with MessageType.TRACE has AirbyteTraceMessage 162 ) 163 if error_message.trace.error.stack_trace: # type: ignore[union-attr] # AirbyteMessage with MessageType.TRACE has AirbyteTraceMessage 164 error_message.trace.error.stack_trace = filter_secrets( # type: ignore[union-attr] # AirbyteMessage with MessageType.TRACE has AirbyteTraceMessage 165 error_message.trace.error.stack_trace # type: ignore[union-attr] # AirbyteMessage with MessageType.TRACE has AirbyteTraceMessage 166 ) 167 return error_message
An exception that should be emitted as an AirbyteTraceMessage
31 def __init__( 32 self, 33 internal_message: Optional[str] = None, 34 message: Optional[str] = None, 35 failure_type: FailureType = FailureType.system_error, 36 exception: Optional[BaseException] = None, 37 stream_descriptor: Optional[StreamDescriptor] = None, 38 ): 39 """ 40 :param internal_message: the internal error that caused the failure 41 :param message: a user-friendly message that indicates the cause of the error 42 :param failure_type: the type of error 43 :param exception: the exception that caused the error, from which the stack trace should be retrieved 44 :param stream_descriptor: describe the stream from which the exception comes from 45 """ 46 self.internal_message = internal_message 47 self.message = message 48 self.failure_type = failure_type 49 self._exception = exception 50 self._stream_descriptor = stream_descriptor 51 super().__init__(internal_message)
Parameters
- internal_message: the internal error that caused the failure
- message: a user-friendly message that indicates the cause of the error
- failure_type: the type of error
- exception: the exception that caused the error, from which the stack trace should be retrieved
- stream_descriptor: describe the stream from which the exception comes from
61 def as_airbyte_message( 62 self, stream_descriptor: Optional[StreamDescriptor] = None 63 ) -> AirbyteMessage: 64 """ 65 Builds an AirbyteTraceMessage from the exception 66 67 :param stream_descriptor is deprecated, please use the stream_description in `__init__ or `from_exception`. If many 68 stream_descriptors are defined, the one from `as_airbyte_message` will be discarded. 69 """ 70 now_millis = time.time_ns() // 1_000_000 71 72 trace_exc = self._exception or self 73 stack_trace_str = "".join(traceback.TracebackException.from_exception(trace_exc).format()) 74 75 trace_message = AirbyteTraceMessage( 76 type=TraceType.ERROR, 77 emitted_at=now_millis, 78 error=AirbyteErrorTraceMessage( 79 message=self.message 80 or "Something went wrong in the connector. See the logs for more details.", 81 internal_message=self.internal_message, 82 failure_type=self.failure_type, 83 stack_trace=stack_trace_str, 84 stream_descriptor=self._stream_descriptor 85 if self._stream_descriptor is not None 86 else stream_descriptor, 87 ), 88 ) 89 90 return AirbyteMessage(type=MessageType.TRACE, trace=trace_message)
Builds an AirbyteTraceMessage from the exception
:param stream_descriptor is deprecated, please use the stream_description in __init__ orfrom_exception. If many
stream_descriptors are defined, the one fromas_airbyte_message` will be discarded.
92 def as_connection_status_message(self) -> Optional[AirbyteMessage]: 93 """Build a failed CONNECTION_STATUS from a config error, with secrets masked. 94 95 The message can quote config values, as a config validation error does, so it goes 96 through the same secret filter as logs and traces. 97 """ 98 if self.failure_type == FailureType.config_error: 99 return AirbyteMessage( 100 type=MessageType.CONNECTION_STATUS, 101 connectionStatus=AirbyteConnectionStatus( 102 status=Status.FAILED, 103 message=filter_secrets(self.message) if self.message else self.message, 104 ), 105 ) 106 return None
Build a failed CONNECTION_STATUS from a config error, with secrets masked.
The message can quote config values, as a config validation error does, so it goes through the same secret filter as logs and traces.
108 def emit_message(self) -> None: 109 """ 110 Prints the exception as an AirbyteTraceMessage. 111 Note that this will be called automatically on uncaught exceptions when using the airbyte_cdk entrypoint. 112 """ 113 message = orjson.dumps(AirbyteMessageSerializer.dump(self.as_airbyte_message())).decode() 114 filtered_message = filter_secrets(message) 115 print(filtered_message)
Prints the exception as an AirbyteTraceMessage. Note that this will be called automatically on uncaught exceptions when using the airbyte_cdk entrypoint.
117 @classmethod 118 def from_exception( 119 cls, 120 exc: BaseException, 121 stream_descriptor: Optional[StreamDescriptor] = None, 122 *args: Any, 123 **kwargs: Any, 124 ) -> "AirbyteTracedException": 125 """ 126 Helper to create an AirbyteTracedException from an existing exception 127 :param exc: the exception that caused the error 128 :param stream_descriptor: describe the stream from which the exception comes from 129 """ 130 if isinstance(exc, AirbyteTracedException): 131 internal_message = exc.internal_message 132 # Preserve the original user-facing message if the caller didn't provide one 133 if "message" not in kwargs: 134 kwargs["message"] = exc.message 135 else: 136 internal_message = str(exc) 137 return cls( 138 internal_message=internal_message, 139 exception=exc, 140 stream_descriptor=stream_descriptor, 141 *args, 142 **kwargs, 143 ) # type: ignore # ignoring because of args and kwargs
Helper to create an AirbyteTracedException from an existing exception
Parameters
- exc: the exception that caused the error
- stream_descriptor: describe the stream from which the exception comes from
145 def as_sanitized_airbyte_message( 146 self, stream_descriptor: Optional[StreamDescriptor] = None 147 ) -> AirbyteMessage: 148 """ 149 Builds an AirbyteTraceMessage from the exception and sanitizes any secrets from the message body 150 151 :param stream_descriptor is deprecated, please use the stream_description in `__init__ or `from_exception`. If many 152 stream_descriptors are defined, the one from `as_sanitized_airbyte_message` will be discarded. 153 """ 154 error_message = self.as_airbyte_message(stream_descriptor=stream_descriptor) 155 if error_message.trace.error.message: # type: ignore[union-attr] # AirbyteMessage with MessageType.TRACE has AirbyteTraceMessage 156 error_message.trace.error.message = filter_secrets( # type: ignore[union-attr] 157 error_message.trace.error.message, # type: ignore[union-attr] 158 ) 159 if error_message.trace.error.internal_message: # type: ignore[union-attr] # AirbyteMessage with MessageType.TRACE has AirbyteTraceMessage 160 error_message.trace.error.internal_message = filter_secrets( # type: ignore[union-attr] # AirbyteMessage with MessageType.TRACE has AirbyteTraceMessage 161 error_message.trace.error.internal_message # type: ignore[union-attr] # AirbyteMessage with MessageType.TRACE has AirbyteTraceMessage 162 ) 163 if error_message.trace.error.stack_trace: # type: ignore[union-attr] # AirbyteMessage with MessageType.TRACE has AirbyteTraceMessage 164 error_message.trace.error.stack_trace = filter_secrets( # type: ignore[union-attr] # AirbyteMessage with MessageType.TRACE has AirbyteTraceMessage 165 error_message.trace.error.stack_trace # type: ignore[union-attr] # AirbyteMessage with MessageType.TRACE has AirbyteTraceMessage 166 ) 167 return error_message
Builds an AirbyteTraceMessage from the exception and sanitizes any secrets from the message body
:param stream_descriptor is deprecated, please use the stream_description in __init__ orfrom_exception. If many
stream_descriptors are defined, the one fromas_sanitized_airbyte_message` will be discarded.
11def is_cloud_environment() -> bool: 12 """ 13 Returns True if the connector is running in a cloud environment, False otherwise. 14 15 The function checks the value of the DEPLOYMENT_MODE environment variable which is set by the platform. 16 This function can be used to determine whether stricter security measures should be applied. 17 """ 18 deployment_mode = os.environ.get("DEPLOYMENT_MODE", "") 19 return deployment_mode.casefold() == CLOUD_DEPLOYMENT_MODE
Returns True if the connector is running in a cloud environment, False otherwise.
The function checks the value of the DEPLOYMENT_MODE environment variable which is set by the platform. This function can be used to determine whether stricter security measures should be applied.
210class InternalConfig(BaseModel): 211 KEYWORDS: ClassVar[set[str]] = {"_limit", "_page_size"} 212 limit: int = Field(None, alias="_limit") 213 page_size: int = Field(None, alias="_page_size") 214 215 def dict(self, *args: Any, **kwargs: Any) -> dict[str, Any]: 216 kwargs["by_alias"] = True 217 kwargs["exclude_unset"] = True 218 return super().dict(*args, **kwargs) 219 220 def is_limit_reached(self, records_counter: int) -> bool: 221 """ 222 Check if record count reached limit set by internal config. 223 :param records_counter - number of records already red 224 :return True if limit reached, False otherwise 225 """ 226 if self.limit: 227 if records_counter >= self.limit: 228 return True 229 return False
215 def dict(self, *args: Any, **kwargs: Any) -> dict[str, Any]: 216 kwargs["by_alias"] = True 217 kwargs["exclude_unset"] = True 218 return super().dict(*args, **kwargs)
Generate a dictionary representation of the model, optionally specifying which fields to include or exclude.
220 def is_limit_reached(self, records_counter: int) -> bool: 221 """ 222 Check if record count reached limit set by internal config. 223 :param records_counter - number of records already red 224 :return True if limit reached, False otherwise 225 """ 226 if self.limit: 227 if records_counter >= self.limit: 228 return True 229 return False
Check if record count reached limit set by internal config. :param records_counter - number of records already red :return True if limit reached, False otherwise
136class ResourceSchemaLoader: 137 """JSONSchema loader from package resources""" 138 139 def __init__(self, package_name: str): 140 self.package_name = package_name 141 142 def get_schema(self, name: str) -> dict[str, Any]: 143 """ 144 This method retrieves a JSON schema from the schemas/ folder. 145 146 147 The expected file structure is to have all top-level schemas (corresponding to streams) in the "schemas/" folder, with any shared $refs 148 living inside the "schemas/shared/" folder. For example: 149 150 schemas/shared/<shared_definition>.json 151 schemas/<name>.json # contains a $ref to shared_definition 152 schemas/<name2>.json # contains a $ref to shared_definition 153 """ 154 155 schema_filename = f"schemas/{name}.json" 156 raw_file = pkgutil.get_data(self.package_name, schema_filename) 157 if not raw_file: 158 raise IOError(f"Cannot find file {schema_filename}") 159 try: 160 raw_schema = json.loads(raw_file) 161 except ValueError as err: 162 raise RuntimeError(f"Invalid JSON file format for file {schema_filename}") from err 163 164 return self._resolve_schema_references(raw_schema) 165 166 def _resolve_schema_references(self, raw_schema: dict[str, Any]) -> dict[str, Any]: 167 """ 168 Resolve links to external references and move it to local "definitions" map. 169 170 :param raw_schema jsonschema to lookup for external links. 171 :return JSON serializable object with references without external dependencies. 172 """ 173 174 package = importlib.import_module(self.package_name) 175 if package.__file__: 176 base = os.path.dirname(package.__file__) + "/" 177 else: 178 raise ValueError(f"Package {package} does not have a valid __file__ field") 179 resolved = jsonref.JsonRef.replace_refs( 180 raw_schema, loader=JsonFileLoader(base, "schemas/shared"), base_uri=base 181 ) 182 resolved = resolve_ref_links(resolved) 183 if isinstance(resolved, dict): 184 return resolved 185 else: 186 raise ValueError(f"Expected resolved to be a dict. Got {resolved}")
JSONSchema loader from package resources
142 def get_schema(self, name: str) -> dict[str, Any]: 143 """ 144 This method retrieves a JSON schema from the schemas/ folder. 145 146 147 The expected file structure is to have all top-level schemas (corresponding to streams) in the "schemas/" folder, with any shared $refs 148 living inside the "schemas/shared/" folder. For example: 149 150 schemas/shared/<shared_definition>.json 151 schemas/<name>.json # contains a $ref to shared_definition 152 schemas/<name2>.json # contains a $ref to shared_definition 153 """ 154 155 schema_filename = f"schemas/{name}.json" 156 raw_file = pkgutil.get_data(self.package_name, schema_filename) 157 if not raw_file: 158 raise IOError(f"Cannot find file {schema_filename}") 159 try: 160 raw_schema = json.loads(raw_file) 161 except ValueError as err: 162 raise RuntimeError(f"Invalid JSON file format for file {schema_filename}") from err 163 164 return self._resolve_schema_references(raw_schema)
This method retrieves a JSON schema from the schemas/ folder.
The expected file structure is to have all top-level schemas (corresponding to streams) in the "schemas/" folder, with any shared $refs living inside the "schemas/shared/" folder. For example:
schemas/shared/
189def check_config_against_spec_or_exit( 190 config: Mapping[str, Any], spec: ConnectorSpecification 191) -> None: 192 """ 193 Check config object against spec. In case of spec is invalid, throws 194 an exception with validation error description. 195 196 :param config - config loaded from file specified over command line 197 :param spec - spec object generated by connector 198 """ 199 spec_schema = spec.connectionSpecification 200 try: 201 validate(instance=config, schema=spec_schema) 202 except ValidationError as validation_error: 203 raise AirbyteTracedException( 204 message="Config validation error: " + validation_error.message, 205 internal_message=validation_error.message, 206 failure_type=FailureType.config_error, 207 ) from None # required to prevent logging config secrets from the ValidationError's stacktrace
Check config object against spec. In case of spec is invalid, throws an exception with validation error description.
:param config - config loaded from file specified over command line :param spec - spec object generated by connector
232def split_config(config: Mapping[str, Any]) -> Tuple[dict[str, Any], InternalConfig]: 233 """ 234 Break config map object into 2 instances: first is a dict with user defined 235 configuration and second is internal config that contains private keys for 236 acceptance test configuration. 237 238 :param 239 config - Dict object that has been loaded from config file. 240 241 :return tuple of user defined config dict with filtered out internal 242 parameters and connector acceptance test internal config object. 243 """ 244 main_config = {} 245 internal_config = {} 246 for k, v in config.items(): 247 if k in InternalConfig.KEYWORDS: 248 internal_config[k] = v 249 else: 250 main_config[k] = v 251 return main_config, InternalConfig.parse_obj(internal_config)
Break config map object into 2 instances: first is a dict with user defined configuration and second is internal config that contains private keys for acceptance test configuration.
:param config - Dict object that has been loaded from config file.
:return tuple of user defined config dict with filtered out internal parameters and connector acceptance test internal config object.
49class TransformConfig(Flag): 50 """ 51 TypeTransformer class config. Configs can be combined using bitwise or operator e.g. 52 ``` 53 TransformConfig.DefaultSchemaNormalization | TransformConfig.CustomSchemaNormalization 54 ``` 55 """ 56 57 # No action taken, default behavior. Cannot be combined with any other options. 58 NoTransform = auto() 59 # Applies default type casting with default_convert method which converts 60 # values by applying simple type casting to specified jsonschema type. 61 DefaultSchemaNormalization = auto() 62 # Allow registering custom type transformation callback. Can be combined 63 # with DefaultSchemaNormalization. In this case default type casting would 64 # be applied before custom one. 65 CustomSchemaNormalization = auto()
TypeTransformer class config. Configs can be combined using bitwise or operator e.g.
TransformConfig.DefaultSchemaNormalization | TransformConfig.CustomSchemaNormalization
68class TypeTransformer: 69 """ 70 Class for transforming object before output. 71 """ 72 73 _custom_normalizer: Optional[Callable[[Any, Dict[str, Any]], Any]] = None 74 75 def __init__(self, config: TransformConfig): 76 """ 77 Initialize TypeTransformer instance. 78 :param config Transform config that would be applied to object 79 """ 80 if TransformConfig.NoTransform in config and config != TransformConfig.NoTransform: 81 raise Exception("NoTransform option cannot be combined with other flags.") 82 self._config = config 83 all_validators = { 84 key: self.__get_normalizer(key, orig_validator) 85 for key, orig_validator in Draft7Validator.VALIDATORS.items() 86 # Do not validate field we do not transform for maximum performance. 87 if key in ["type", "array", "$ref", "properties", "items"] 88 } 89 self._normalizer = validators.create( 90 meta_schema=Draft7Validator.META_SCHEMA, validators=all_validators 91 ) 92 93 def registerCustomTransform( 94 self, normalization_callback: Callable[[Any, dict[str, Any]], Any] 95 ) -> Callable[[Any, dict[str, Any]], Any]: 96 """ 97 Register custom normalization callback. 98 :param normalization_callback function to be used for value 99 normalization. Takes original value and part type schema. Should return 100 normalized value. See docs/connector-development/cdk-python/schemas.md 101 for details. 102 :return Same callback, this is useful for using registerCustomTransform function as decorator. 103 """ 104 if TransformConfig.CustomSchemaNormalization not in self._config: 105 raise Exception( 106 "Please set TransformConfig.CustomSchemaNormalization config before registering custom normalizer" 107 ) 108 self._custom_normalizer = normalization_callback 109 return normalization_callback 110 111 def __normalize(self, original_item: Any, subschema: Dict[str, Any]) -> Any: 112 """ 113 Applies different transform function to object's field according to config. 114 :param original_item original value of field. 115 :param subschema part of the jsonschema containing field type/format data. 116 :return Final field value. 117 """ 118 if TransformConfig.DefaultSchemaNormalization in self._config: 119 original_item = self.default_convert(original_item, subschema) 120 121 if self._custom_normalizer: 122 original_item = self._custom_normalizer(original_item, subschema) 123 return original_item 124 125 @staticmethod 126 def default_convert(original_item: Any, subschema: Dict[str, Any]) -> Any: 127 """ 128 Default transform function that is used when TransformConfig.DefaultSchemaNormalization flag set. 129 :param original_item original value of field. 130 :param subschema part of the jsonschema containing field type/format data. 131 :return transformed field value. 132 """ 133 target_type = subschema.get("type", []) 134 if original_item is None and "null" in target_type: 135 return None 136 if isinstance(target_type, list): 137 # jsonschema type could either be a single string or array of type 138 # strings. In case if there is some disambigous and more than one 139 # type (except null) do not do any conversion and return original 140 # value. If type array has one type and null i.e. {"type": 141 # ["integer", "null"]}, convert value to specified type. 142 target_type = [t for t in target_type if t != "null"] 143 if len(target_type) != 1: 144 return original_item 145 target_type = target_type[0] 146 try: 147 if target_type == "string": 148 return str(original_item) 149 elif target_type == "number": 150 return float(original_item) 151 elif target_type == "integer": 152 return int(original_item) 153 elif target_type == "boolean": 154 if isinstance(original_item, str): 155 return _strtobool(original_item) == 1 156 return bool(original_item) 157 elif target_type == "array": 158 item_types = set(subschema.get("items", {}).get("type", set())) 159 if ( 160 item_types.issubset(json_to_python_simple) 161 and type(original_item) in json_to_python_simple.values() 162 ): 163 return [original_item] 164 except (ValueError, TypeError): 165 return original_item 166 return original_item 167 168 def __get_normalizer( 169 self, 170 schema_key: str, 171 original_validator: Callable, # type: ignore[type-arg] 172 ) -> Callable[[Any, Any, Any, dict[str, Any]], Generator[Any, Any, None]]: 173 """ 174 Traverse through object fields using native jsonschema validator and apply normalization function. 175 :param schema_key related json schema key that currently being validated/normalized. 176 :original_validator: native jsonschema validator callback. 177 """ 178 179 def normalizator( 180 validator_instance: Validator, 181 property_value: Any, 182 instance: Any, 183 schema: Dict[str, Any], 184 ) -> Generator[Any, Any, None]: 185 """ 186 Jsonschema validator callable it uses for validating instance. We 187 override default Draft7Validator to perform value transformation 188 before validation take place. We do not take any action except 189 logging warn if object does not conform to json schema, just using 190 jsonschema algorithm to traverse through object fields. 191 Look 192 https://python-jsonschema.readthedocs.io/en/stable/creating/?highlight=validators.create#jsonschema.validators.create 193 validators parameter for detailed description. 194 : 195 """ 196 # Transform object and array values before running json schema type checking for each element. 197 # Recursively normalize every value of the "instance" sub-object, 198 # if "instance" is an incorrect type - skip recursive normalization of "instance" 199 if schema_key == "properties" and isinstance(instance, dict): 200 for k, subschema in property_value.items(): 201 if k in instance: 202 instance[k] = self.__normalize(instance[k], subschema) 203 # Recursively normalize every item of the "instance" sub-array, 204 # if "instance" is an incorrect type - skip recursive normalization of "instance" 205 elif schema_key == "items" and isinstance(instance, list): 206 for index, item in enumerate(instance): 207 instance[index] = self.__normalize(item, property_value) 208 209 # Running native jsonschema traverse algorithm after field normalization is done. 210 yield from original_validator( 211 validator_instance, 212 property_value, 213 instance, 214 schema, 215 ) 216 217 return normalizator 218 219 def transform( 220 self, 221 record: Dict[str, Any], 222 schema: Mapping[str, Any], 223 ) -> None: 224 """ 225 Normalize and validate according to config. 226 :param record: record instance for normalization/transformation. All modification are done by modifying existent object. 227 :param schema: object's jsonschema for normalization. 228 """ 229 if TransformConfig.NoTransform in self._config: 230 return 231 normalizer = self._normalizer(schema) 232 for e in normalizer.iter_errors(record): 233 """ 234 just calling normalizer.validate() would throw an exception on 235 first validation occurrences and stop processing rest of schema. 236 """ 237 logger.warning(self.get_error_message(e)) 238 239 def get_error_message(self, e: ValidationError) -> str: 240 """ 241 Construct a sanitized error message from a ValidationError instance. 242 """ 243 field_path = ".".join(map(str, e.path)) 244 type_structure = self._get_type_structure(e.instance) 245 246 return f"Failed to transform value from type '{type_structure}' to type '{e.validator_value}' at path: '{field_path}'" 247 248 def _get_type_structure(self, input_data: Any, current_depth: int = 0) -> Any: 249 """ 250 Get the structure of a given input data for use in error message construction. 251 """ 252 # Handle null values 253 if input_data is None: 254 return "null" 255 256 # Avoid recursing too deep 257 if current_depth >= MAX_NESTING_DEPTH: 258 return "object" if isinstance(input_data, dict) else python_to_json[type(input_data)] 259 260 if isinstance(input_data, dict): 261 return { 262 key: self._get_type_structure(field_value, current_depth + 1) 263 for key, field_value in input_data.items() 264 } 265 266 else: 267 return python_to_json[type(input_data)]
Class for transforming object before output.
75 def __init__(self, config: TransformConfig): 76 """ 77 Initialize TypeTransformer instance. 78 :param config Transform config that would be applied to object 79 """ 80 if TransformConfig.NoTransform in config and config != TransformConfig.NoTransform: 81 raise Exception("NoTransform option cannot be combined with other flags.") 82 self._config = config 83 all_validators = { 84 key: self.__get_normalizer(key, orig_validator) 85 for key, orig_validator in Draft7Validator.VALIDATORS.items() 86 # Do not validate field we do not transform for maximum performance. 87 if key in ["type", "array", "$ref", "properties", "items"] 88 } 89 self._normalizer = validators.create( 90 meta_schema=Draft7Validator.META_SCHEMA, validators=all_validators 91 )
Initialize TypeTransformer instance. :param config Transform config that would be applied to object
93 def registerCustomTransform( 94 self, normalization_callback: Callable[[Any, dict[str, Any]], Any] 95 ) -> Callable[[Any, dict[str, Any]], Any]: 96 """ 97 Register custom normalization callback. 98 :param normalization_callback function to be used for value 99 normalization. Takes original value and part type schema. Should return 100 normalized value. See docs/connector-development/cdk-python/schemas.md 101 for details. 102 :return Same callback, this is useful for using registerCustomTransform function as decorator. 103 """ 104 if TransformConfig.CustomSchemaNormalization not in self._config: 105 raise Exception( 106 "Please set TransformConfig.CustomSchemaNormalization config before registering custom normalizer" 107 ) 108 self._custom_normalizer = normalization_callback 109 return normalization_callback
Register custom normalization callback. :param normalization_callback function to be used for value normalization. Takes original value and part type schema. Should return normalized value. See docs/connector-development/cdk-python/schemas.md for details. :return Same callback, this is useful for using registerCustomTransform function as decorator.
125 @staticmethod 126 def default_convert(original_item: Any, subschema: Dict[str, Any]) -> Any: 127 """ 128 Default transform function that is used when TransformConfig.DefaultSchemaNormalization flag set. 129 :param original_item original value of field. 130 :param subschema part of the jsonschema containing field type/format data. 131 :return transformed field value. 132 """ 133 target_type = subschema.get("type", []) 134 if original_item is None and "null" in target_type: 135 return None 136 if isinstance(target_type, list): 137 # jsonschema type could either be a single string or array of type 138 # strings. In case if there is some disambigous and more than one 139 # type (except null) do not do any conversion and return original 140 # value. If type array has one type and null i.e. {"type": 141 # ["integer", "null"]}, convert value to specified type. 142 target_type = [t for t in target_type if t != "null"] 143 if len(target_type) != 1: 144 return original_item 145 target_type = target_type[0] 146 try: 147 if target_type == "string": 148 return str(original_item) 149 elif target_type == "number": 150 return float(original_item) 151 elif target_type == "integer": 152 return int(original_item) 153 elif target_type == "boolean": 154 if isinstance(original_item, str): 155 return _strtobool(original_item) == 1 156 return bool(original_item) 157 elif target_type == "array": 158 item_types = set(subschema.get("items", {}).get("type", set())) 159 if ( 160 item_types.issubset(json_to_python_simple) 161 and type(original_item) in json_to_python_simple.values() 162 ): 163 return [original_item] 164 except (ValueError, TypeError): 165 return original_item 166 return original_item
Default transform function that is used when TransformConfig.DefaultSchemaNormalization flag set. :param original_item original value of field. :param subschema part of the jsonschema containing field type/format data. :return transformed field value.
219 def transform( 220 self, 221 record: Dict[str, Any], 222 schema: Mapping[str, Any], 223 ) -> None: 224 """ 225 Normalize and validate according to config. 226 :param record: record instance for normalization/transformation. All modification are done by modifying existent object. 227 :param schema: object's jsonschema for normalization. 228 """ 229 if TransformConfig.NoTransform in self._config: 230 return 231 normalizer = self._normalizer(schema) 232 for e in normalizer.iter_errors(record): 233 """ 234 just calling normalizer.validate() would throw an exception on 235 first validation occurrences and stop processing rest of schema. 236 """ 237 logger.warning(self.get_error_message(e))
Normalize and validate according to config.
Parameters
- record: record instance for normalization/transformation. All modification are done by modifying existent object.
- schema: object's jsonschema for normalization.
239 def get_error_message(self, e: ValidationError) -> str: 240 """ 241 Construct a sanitized error message from a ValidationError instance. 242 """ 243 field_path = ".".join(map(str, e.path)) 244 type_structure = self._get_type_structure(e.instance) 245 246 return f"Failed to transform value from type '{type_structure}' to type '{e.validator_value}' at path: '{field_path}'"
Construct a sanitized error message from a ValidationError instance.
80@contextmanager 81def create_timer(name: str) -> Generator[EventTimer, Any, None]: 82 """ 83 Creates a new EventTimer as a context manager to improve code readability. 84 """ 85 a_timer = EventTimer(name) 86 yield a_timer
Creates a new EventTimer as a context manager to improve code readability.
9class OneOfOptionConfig: 10 """ 11 Base class to configure a Pydantic model that's used as a oneOf option in a parent model in a way that's compatible with all Airbyte consumers. 12 13 Inherit from this class in the nested Config class in a model and set title and description (these show up in the UI) and discriminator (this is making sure it's marked as required in the schema). 14 15 Usage: 16 17 ```python 18 class OptionModel(BaseModel): 19 mode: Literal["option_a"] = Field("option_a", const=True) 20 option_a_field: str = Field(...) 21 22 class Config(OneOfOptionConfig): 23 title = "Option A" 24 description = "Option A description" 25 discriminator = "mode" 26 ``` 27 """ 28 29 @staticmethod 30 def schema_extra(schema: Dict[str, Any], model: Any) -> None: 31 if hasattr(model.Config, "description"): 32 schema["description"] = model.Config.description 33 if hasattr(model.Config, "discriminator"): 34 schema.setdefault("required", []).append(model.Config.discriminator)
Base class to configure a Pydantic model that's used as a oneOf option in a parent model in a way that's compatible with all Airbyte consumers.
Inherit from this class in the nested Config class in a model and set title and description (these show up in the UI) and discriminator (this is making sure it's marked as required in the schema).
Usage:
class OptionModel(BaseModel): mode: Literal["option_a"] = Field("option_a", const=True) option_a_field: str = Field(...) class Config(OneOfOptionConfig): title = "Option A" description = "Option A description" discriminator = "mode"
14def resolve_refs(schema: dict[str, Any]) -> dict[str, Any]: 15 """ 16 For spec schemas generated using Pydantic models, the resulting JSON schema can contain refs between object 17 relationships. 18 """ 19 resource = Resource.from_contents(schema, default_specification=DRAFT7) 20 registry = Registry().with_resource("", resource) 21 resolver = registry.resolver() 22 str_schema = json.dumps(schema) 23 for ref_block in re.findall(r'{"\$ref": "#\/definitions\/.+?(?="})"}', str_schema): 24 ref = json.loads(ref_block)["$ref"] 25 resolved = resolver.lookup(ref).contents 26 str_schema = str_schema.replace(ref_block, json.dumps(resolved)) 27 pyschema: dict[str, Any] = json.loads(str_schema) 28 del pyschema["definitions"] 29 return pyschema
For spec schemas generated using Pydantic models, the resulting JSON schema can contain refs between object relationships.
23def as_airbyte_message( 24 stream: Union[AirbyteStream, StreamDescriptor], 25 current_status: AirbyteStreamStatus, 26 reasons: Optional[List[AirbyteStreamStatusReason]] = None, 27) -> AirbyteMessage: 28 """ 29 Builds an AirbyteStreamStatusTraceMessage for the provided stream 30 """ 31 32 now_millis = datetime.now().timestamp() * 1000.0 33 34 trace_message = AirbyteTraceMessage( 35 type=TraceType.STREAM_STATUS, 36 emitted_at=now_millis, 37 stream_status=AirbyteStreamStatusTraceMessage( 38 stream_descriptor=StreamDescriptor(name=stream.name, namespace=stream.namespace), 39 status=current_status, 40 reasons=reasons, 41 ), 42 ) 43 44 return AirbyteMessage(type=MessageType.TRACE, trace=trace_message)
Builds an AirbyteStreamStatusTraceMessage for the provided stream
22class Record(Mapping[str, Any]): 23 def __init__( 24 self, 25 data: Mapping[str, Any], 26 stream_name: str, 27 associated_slice: Optional[StreamSlice] = None, 28 file_reference: Optional[AirbyteRecordMessageFileReference] = None, 29 ): 30 self._data = data 31 self._associated_slice = associated_slice 32 self.stream_name = stream_name 33 self._file_reference = file_reference 34 35 @property 36 def data(self) -> Mapping[str, Any]: 37 return self._data 38 39 @property 40 def associated_slice(self) -> Optional[StreamSlice]: 41 return self._associated_slice 42 43 @property 44 def file_reference(self) -> AirbyteRecordMessageFileReference: 45 return self._file_reference 46 47 @file_reference.setter 48 def file_reference(self, value: AirbyteRecordMessageFileReference) -> None: 49 self._file_reference = value 50 51 def __repr__(self) -> str: 52 return repr(self._data) 53 54 def __getitem__(self, key: str) -> Any: 55 return self._data[key] 56 57 def __len__(self) -> int: 58 return len(self._data) 59 60 def __iter__(self) -> Any: 61 return iter(self._data) 62 63 def __contains__(self, item: object) -> bool: 64 return item in self._data 65 66 def __eq__(self, other: object) -> bool: 67 if isinstance(other, Record): 68 # noinspection PyProtectedMember 69 return self._data == other._data 70 return False 71 72 def __ne__(self, other: object) -> bool: 73 return not self.__eq__(other)
A Mapping is a generic container for associating key/value pairs.
This class provides concrete generic implementations of all methods except for __getitem__, __iter__, and __len__.
23 def __init__( 24 self, 25 data: Mapping[str, Any], 26 stream_name: str, 27 associated_slice: Optional[StreamSlice] = None, 28 file_reference: Optional[AirbyteRecordMessageFileReference] = None, 29 ): 30 self._data = data 31 self._associated_slice = associated_slice 32 self.stream_name = stream_name 33 self._file_reference = file_reference