airbyte_cdk
Welcome to the Airbyte Python CDK!
The Airbyte Python CDK is a Python library that provides a set of tools to help you build connectors for the Airbyte platform.
Building Source Connectors
To build a source connector, you will want to refer to the following classes and modules:
airbyte_cdk.sourcesairbyte_cdk.sources.concurrent_sourceairbyte_cdk.sources.configairbyte_cdk.sources.file_basedairbyte_cdk.sources.streams
Building Destination Connectors
To build a destination connector, you will want to refer to the following classes and modules:
airbyte_cdk.destinationsairbyte_cdk.destinations.Destinationairbyte_cdk.destinations.vector_db_based
Working with Airbyte Protocol Models
The Airbyte CDK provides a set of classes that help you work with the Airbyte protocol models:
Using the CLI (airbyte_cdk.cli)
The Airbyte CDK provides two command-line interfaces (CLIs) for interacting with the framework.
airbyte-cdk: This is the main CLI for the Airbyte CDK. It provides commands for building and testing connectors, as well as other utilities. See theairbyte_cdk.cli.airbyte_cdkmodule for more details.source-declarative-manifest: This command allows you to run declarative manifests directly. See theairbyte_cdk.cli.source_declarative_manifestmodule for more details.
API Reference
1# Copyright (c) 2021 Airbyte, Inc., all rights reserved. 2""" 3# Welcome to the Airbyte Python CDK! 4 5The Airbyte Python CDK is a Python library that provides a set of tools to help you build 6connectors for the Airbyte platform. 7 8## Building Source Connectors 9 10To build a source connector, you will want to refer to 11the following classes and modules: 12 13- `airbyte_cdk.sources` 14- `airbyte_cdk.sources.concurrent_source` 15- `airbyte_cdk.sources.config` 16- `airbyte_cdk.sources.file_based` 17- `airbyte_cdk.sources.streams` 18 19## Building Destination Connectors 20 21To build a destination connector, you will want to refer to 22the following classes and modules: 23 24- `airbyte_cdk.destinations` 25- `airbyte_cdk.destinations.Destination` 26- `airbyte_cdk.destinations.vector_db_based` 27 28## Working with Airbyte Protocol Models 29 30The Airbyte CDK provides a set of classes that help you work with the Airbyte protocol models: 31 32- `airbyte_cdk.models.airbyte_protocol` 33- `airbyte_cdk.models.airbyte_protocol_serializers` 34 35## Using the CLI (`airbyte_cdk.cli`) 36 37The Airbyte CDK provides two command-line interfaces (CLIs) for interacting with the framework. 38 39- `airbyte-cdk`: This is the main CLI for the Airbyte CDK. It provides commands for building 40 and testing connectors, as well as other utilities. See the `airbyte_cdk.cli.airbyte_cdk` module 41 for more details. 42- `source-declarative-manifest`: This command allows you to run declarative manifests directly. 43 See the `airbyte_cdk.cli.source_declarative_manifest` module for more details. 44 45--- 46 47API Reference 48 49--- 50 51""" 52 53# Warning: The below imports are not stable and will cause circular 54# dependencies if auto-sorted with isort. Please keep them in the same order. 55# TODO: Submodules should import from lower-level modules, rather than importing from here. 56# Imports should also be placed in `if TYPE_CHECKING` blocks if they are only used as type 57# hints - again, to avoid circular dependencies. 58# Once those issues are resolved, the below can be sorted with isort. 59import dunamai as _dunamai 60 61from .config_observation import ( 62 create_connector_config_control_message, 63 emit_configuration_as_airbyte_control_message, 64) 65from .connector import BaseConnector, Connector 66from .destinations import Destination 67from .entrypoint import AirbyteEntrypoint, launch 68from .legacy.sources.declarative.declarative_stream import DeclarativeStream 69from .legacy.sources.declarative.incremental import DatetimeBasedCursor 70from .logger import AirbyteLogFormatter, init_logger 71from .models import ( 72 AdvancedAuth, 73 AirbyteConnectionStatus, 74 AirbyteLogMessage, 75 AirbyteMessage, 76 AirbyteRecordMessage, 77 AirbyteStream, 78 ConfiguredAirbyteCatalog, 79 ConfiguredAirbyteStream, 80 ConnectorSpecification, 81 DestinationSyncMode, 82 FailureType, 83 Level, 84 OAuthConfigSpecification, 85 OrchestratorType, 86 Status, 87 SyncMode, 88 Type, 89) 90from .sources import AbstractSource, Source 91from .sources.concurrent_source.concurrent_source import ConcurrentSource 92from .sources.concurrent_source.concurrent_source_adapter import ConcurrentSourceAdapter 93from .sources.config import BaseConfig 94from .sources.connector_state_manager import ConnectorStateManager 95from .sources.declarative.auth import DeclarativeOauth2Authenticator 96from .sources.declarative.auth.declarative_authenticator import DeclarativeAuthenticator, NoAuth 97from .sources.declarative.auth.oauth import DeclarativeSingleUseRefreshTokenOauth2Authenticator 98from .sources.declarative.auth.token import ( 99 ApiKeyAuthenticator, 100 BasicHttpAuthenticator, 101 BearerAuthenticator, 102) 103from .sources.declarative.datetime.min_max_datetime import MinMaxDatetime 104from .sources.declarative.decoders import Decoder, JsonDecoder 105from .sources.declarative.exceptions import ReadException 106from .sources.declarative.extractors import ( 107 CombinedExtractor, 108 CombineMode, 109 DpathExtractor, 110 RecordSelector, 111) 112from .sources.declarative.extractors.record_extractor import RecordExtractor 113from .sources.declarative.extractors.record_filter import RecordFilter 114from .sources.declarative.interpolation import InterpolatedBoolean, InterpolatedString 115from .sources.declarative.migrations.legacy_to_per_partition_state_migration import ( 116 LegacyToPerPartitionStateMigration, 117) 118from .sources.declarative.partition_routers import ( 119 CartesianProductStreamSlicer, 120 SinglePartitionRouter, 121 SubstreamPartitionRouter, 122 UnionPartitionRouter, 123) 124from .sources.declarative.partition_routers.substream_partition_router import ParentStreamConfig 125from .sources.declarative.requesters import HttpRequester, Requester 126from .sources.declarative.requesters.error_handlers import BackoffStrategy 127from .sources.declarative.requesters.paginators import DefaultPaginator, PaginationStrategy 128from .sources.declarative.requesters.paginators.strategies import ( 129 CursorPaginationStrategy, 130 OffsetIncrement, 131 PageIncrement, 132 StopConditionPaginationStrategyDecorator, 133) 134from .sources.declarative.requesters.request_option import RequestOption, RequestOptionType 135from .sources.declarative.requesters.request_options.default_request_options_provider import ( 136 DefaultRequestOptionsProvider, 137) 138from .sources.declarative.requesters.request_options.interpolated_request_input_provider import ( 139 InterpolatedRequestInputProvider, 140) 141from .sources.declarative.requesters.requester import HttpMethod 142from .sources.declarative.retrievers import SimpleRetriever 143from .sources.declarative.schema import JsonFileSchemaLoader 144from .sources.declarative.transformations.add_fields import AddedFieldDefinition, AddFields 145from .sources.declarative.transformations.transformation import RecordTransformation 146from .sources.declarative.types import FieldPointer 147from .sources.declarative.yaml_declarative_source import YamlDeclarativeSource 148from .sources.message import InMemoryMessageRepository, MessageRepository 149from .sources.source import TState 150from .sources.streams.availability_strategy import AvailabilityStrategy 151from .sources.streams.call_rate import ( 152 AbstractAPIBudget, 153 CachedLimiterSession, 154 HttpAPIBudget, 155 HttpRequestMatcher, 156 LimiterSession, 157 MovingWindowCallRatePolicy, 158 Rate, 159) 160from .sources.streams.checkpoint import Cursor as LegacyCursor 161from .sources.streams.checkpoint import ResumableFullRefreshCursor 162from .sources.streams.concurrent.adapters import StreamFacade 163from .sources.streams.concurrent.cursor import ( 164 ConcurrentCursor, 165 Cursor, 166 CursorField, 167 FinalStateCursor, 168) 169from .sources.streams.concurrent.state_converters.datetime_stream_state_converter import ( 170 EpochValueConcurrentStreamStateConverter, 171 IsoMillisConcurrentStreamStateConverter, 172) 173from .sources.streams.core import IncrementalMixin, Stream, package_name_from_class 174from .sources.streams.http import HttpStream, HttpSubStream 175from .sources.streams.http.availability_strategy import HttpAvailabilityStrategy 176from .sources.streams.http.exceptions import ( 177 BaseBackoffException, 178 DefaultBackoffException, 179 UserDefinedBackoffException, 180) 181from .sources.streams.http.rate_limiting import default_backoff_handler 182from .sources.streams.http.requests_native_auth import ( 183 Oauth2Authenticator, 184 SingleUseRefreshTokenOauth2Authenticator, 185 TokenAuthenticator, 186) 187from .sources.streams.http.requests_native_auth.abstract_token import AbstractHeaderAuthenticator 188from .sources.types import Config, Record, StreamSlice 189from .sources.utils import casing 190from .sources.utils.schema_helpers import ( 191 InternalConfig, 192 ResourceSchemaLoader, 193 check_config_against_spec_or_exit, 194 expand_refs, 195 split_config, 196) 197from .sources.utils.transform import TransformConfig, TypeTransformer 198from .utils import AirbyteTracedException, is_cloud_environment 199from .utils.constants import ENV_REQUEST_CACHE_PATH 200from .utils.event_timing import create_timer 201from .utils.oneof_option_config import OneOfOptionConfig 202from .utils.spec_schema_transformations import resolve_refs 203from .utils.stream_status_utils import as_airbyte_message 204 205__all__ = [ 206 # Availability strategy 207 "AvailabilityStrategy", 208 "HttpAvailabilityStrategy", 209 # Checkpoint 210 "LegacyCursor", 211 "ResumableFullRefreshCursor", 212 # Concurrent 213 "ConcurrentCursor", 214 "ConcurrentSource", 215 "ConcurrentSourceAdapter", 216 "Cursor", 217 "CursorField", 218 "DEFAULT_CONCURRENCY", 219 "EpochValueConcurrentStreamStateConverter", 220 "FinalStateCursor", 221 "IsoMillisConcurrentStreamStateConverter", 222 "StreamFacade", 223 # Config observation 224 "create_connector_config_control_message", 225 "emit_configuration_as_airbyte_control_message", 226 # Connector 227 "AbstractSource", 228 "BaseConfig", 229 "BaseConnector", 230 "Connector", 231 "Destination", 232 "Source", 233 "TState", 234 # Declarative 235 "AddFields", 236 "AddedFieldDefinition", 237 "ApiKeyAuthenticator", 238 "BackoffStrategy", 239 "BasicHttpAuthenticator", 240 "BearerAuthenticator", 241 "CartesianProductStreamSlicer", 242 "CombineMode", 243 "CombinedExtractor", 244 "CursorPaginationStrategy", 245 "DatetimeBasedCursor", 246 "DeclarativeAuthenticator", 247 "DeclarativeOauth2Authenticator", 248 "DeclarativeSingleUseRefreshTokenOauth2Authenticator", 249 "DeclarativeStream", 250 "Decoder", 251 "DefaultPaginator", 252 "DefaultRequestOptionsProvider", 253 "DpathExtractor", 254 "FieldPointer", 255 "HttpMethod", 256 "HttpRequester", 257 "InterpolatedBoolean", 258 "InterpolatedRequestInputProvider", 259 "InterpolatedString", 260 "JsonDecoder", 261 "JsonFileSchemaLoader", 262 "LegacyToPerPartitionStateMigration", 263 "MinMaxDatetime", 264 "NoAuth", 265 "OffsetIncrement", 266 "PageIncrement", 267 "PaginationStrategy", 268 "ParentStreamConfig", 269 "ReadException", 270 "RecordExtractor", 271 "RecordFilter", 272 "RecordSelector", 273 "RecordTransformation", 274 "RequestOption", 275 "RequestOptionType", 276 "Requester", 277 "ResponseStatus", 278 "SimpleRetriever", 279 "SinglePartitionRouter", 280 "StopConditionPaginationStrategyDecorator", 281 "StreamSlice", 282 "SubstreamPartitionRouter", 283 "UnionPartitionRouter", 284 "YamlDeclarativeSource", 285 # Entrypoint 286 "launch", 287 "AirbyteEntrypoint", 288 # HTTP 289 "AbstractAPIBudget", 290 "AbstractHeaderAuthenticator", 291 "BaseBackoffException", 292 "CachedLimiterSession", 293 "DefaultBackoffException", 294 "default_backoff_handler", 295 "HttpAPIBudget", 296 "HttpAuthenticator", 297 "HttpRequestMatcher", 298 "HttpStream", 299 "HttpSubStream", 300 "LimiterSession", 301 "MovingWindowCallRatePolicy", 302 "MultipleTokenAuthenticator", 303 "Oauth2Authenticator", 304 "Rate", 305 "SingleUseRefreshTokenOauth2Authenticator", 306 "TokenAuthenticator", 307 "UserDefinedBackoffException", 308 # Logger 309 "AirbyteLogFormatter", 310 "init_logger", 311 # Protocol classes 312 "AirbyteStream", 313 "AirbyteConnectionStatus", 314 "AirbyteMessage", 315 "ConfiguredAirbyteCatalog", 316 "Status", 317 "Type", 318 "OrchestratorType", 319 "ConfiguredAirbyteStream", 320 "DestinationSyncMode", 321 "SyncMode", 322 "FailureType", 323 "AdvancedAuth", 324 "AirbyteLogMessage", 325 "OAuthConfigSpecification", 326 "ConnectorSpecification", 327 "Level", 328 "AirbyteRecordMessage", 329 # Repository 330 "InMemoryMessageRepository", 331 "MessageRepository", 332 # State management 333 "ConnectorStateManager", 334 # Stream 335 "IncrementalMixin", 336 "Stream", 337 "StreamData", 338 "package_name_from_class", 339 # Utils 340 "AirbyteTracedException", 341 "is_cloud_environment", 342 "casing", 343 "InternalConfig", 344 "ResourceSchemaLoader", 345 "check_config_against_spec_or_exit", 346 "split_config", 347 "TransformConfig", 348 "TypeTransformer", 349 "ENV_REQUEST_CACHE_PATH", 350 "create_timer", 351 "OneOfOptionConfig", 352 "resolve_refs", 353 "as_airbyte_message", 354 # Types 355 "Config", 356 "Record", 357 "Source", 358 "StreamSlice", 359] 360 361__version__: str 362"""Version generated by poetry dynamic versioning during publish. 363 364When running in development, dunamai will calculate a new prerelease version 365from existing git release tag info. 366""" 367 368try: 369 __version__ = _dunamai.get_version( 370 "airbyte-cdk", 371 third_choice=_dunamai.Version.from_any_vcs, 372 fallback=_dunamai.Version("0.0.0+dev"), 373 ).serialize() 374except: 375 __version__ = "0.0.0+dev"
19class AvailabilityStrategy(ABC): 20 """ 21 Abstract base class for checking stream availability. 22 """ 23 24 @abstractmethod 25 def check_availability( 26 self, stream: Stream, logger: logging.Logger, source: Optional["Source"] = None 27 ) -> Tuple[bool, Optional[str]]: 28 """ 29 Checks stream availability. 30 31 :param stream: stream 32 :param logger: source logger 33 :param source: (optional) source 34 :return: A tuple of (boolean, str). If boolean is true, then the stream 35 is available, and no str is required. Otherwise, the stream is unavailable 36 for some reason and the str should describe what went wrong and how to 37 resolve the unavailability, if possible. 38 """ 39 40 @staticmethod 41 def get_first_stream_slice(stream: Stream) -> Optional[Mapping[str, Any]]: 42 """ 43 Gets the first stream_slice from a given stream's stream_slices. 44 :param stream: stream 45 :raises StopIteration: if there is no first slice to return (the stream_slices generator is empty) 46 :return: first stream slice from 'stream_slices' generator (`None` is a valid stream slice) 47 """ 48 # We wrap the return output of stream_slices() because some implementations return types that are iterable, 49 # but not iterators such as lists or tuples 50 slices = iter( 51 stream.stream_slices( 52 cursor_field=stream.cursor_field, # type: ignore[arg-type] 53 sync_mode=SyncMode.full_refresh, 54 ) 55 ) 56 return next(slices) 57 58 @staticmethod 59 def get_first_record_for_slice( 60 stream: Stream, stream_slice: Optional[Mapping[str, Any]] 61 ) -> StreamData: 62 """ 63 Gets the first record for a stream_slice of a stream. 64 65 :param stream: stream instance from which to read records 66 :param stream_slice: stream_slice parameters for slicing the stream 67 :raises StopIteration: if there is no first record to return (the read_records generator is empty) 68 :return: StreamData containing the first record in the slice 69 """ 70 # Store the original value of exit_on_rate_limit 71 original_exit_on_rate_limit = stream.exit_on_rate_limit 72 73 try: 74 # Ensure exit_on_rate_limit is safely set to True if possible 75 stream.exit_on_rate_limit = True 76 77 # We wrap the return output of read_records() because some implementations return types that are iterable, 78 # but not iterators such as lists or tuples 79 records_for_slice = iter( 80 stream.read_records(sync_mode=SyncMode.full_refresh, stream_slice=stream_slice) 81 ) 82 83 return next(records_for_slice) 84 finally: 85 # Restore the original exit_on_rate_limit value 86 stream.exit_on_rate_limit = original_exit_on_rate_limit
Abstract base class for checking stream availability.
24 @abstractmethod 25 def check_availability( 26 self, stream: Stream, logger: logging.Logger, source: Optional["Source"] = None 27 ) -> Tuple[bool, Optional[str]]: 28 """ 29 Checks stream availability. 30 31 :param stream: stream 32 :param logger: source logger 33 :param source: (optional) source 34 :return: A tuple of (boolean, str). If boolean is true, then the stream 35 is available, and no str is required. Otherwise, the stream is unavailable 36 for some reason and the str should describe what went wrong and how to 37 resolve the unavailability, if possible. 38 """
Checks stream availability.
Parameters
- stream: stream
- logger: source logger
- source: (optional) source
Returns
A tuple of (boolean, str). If boolean is true, then the stream is available, and no str is required. Otherwise, the stream is unavailable for some reason and the str should describe what went wrong and how to resolve the unavailability, if possible.
40 @staticmethod 41 def get_first_stream_slice(stream: Stream) -> Optional[Mapping[str, Any]]: 42 """ 43 Gets the first stream_slice from a given stream's stream_slices. 44 :param stream: stream 45 :raises StopIteration: if there is no first slice to return (the stream_slices generator is empty) 46 :return: first stream slice from 'stream_slices' generator (`None` is a valid stream slice) 47 """ 48 # We wrap the return output of stream_slices() because some implementations return types that are iterable, 49 # but not iterators such as lists or tuples 50 slices = iter( 51 stream.stream_slices( 52 cursor_field=stream.cursor_field, # type: ignore[arg-type] 53 sync_mode=SyncMode.full_refresh, 54 ) 55 ) 56 return next(slices)
Gets the first stream_slice from a given stream's stream_slices.
Parameters
- stream: stream
Raises
- StopIteration: if there is no first slice to return (the stream_slices generator is empty)
Returns
first stream slice from 'stream_slices' generator (
Noneis a valid stream slice)
58 @staticmethod 59 def get_first_record_for_slice( 60 stream: Stream, stream_slice: Optional[Mapping[str, Any]] 61 ) -> StreamData: 62 """ 63 Gets the first record for a stream_slice of a stream. 64 65 :param stream: stream instance from which to read records 66 :param stream_slice: stream_slice parameters for slicing the stream 67 :raises StopIteration: if there is no first record to return (the read_records generator is empty) 68 :return: StreamData containing the first record in the slice 69 """ 70 # Store the original value of exit_on_rate_limit 71 original_exit_on_rate_limit = stream.exit_on_rate_limit 72 73 try: 74 # Ensure exit_on_rate_limit is safely set to True if possible 75 stream.exit_on_rate_limit = True 76 77 # We wrap the return output of read_records() because some implementations return types that are iterable, 78 # but not iterators such as lists or tuples 79 records_for_slice = iter( 80 stream.read_records(sync_mode=SyncMode.full_refresh, stream_slice=stream_slice) 81 ) 82 83 return next(records_for_slice) 84 finally: 85 # Restore the original exit_on_rate_limit value 86 stream.exit_on_rate_limit = original_exit_on_rate_limit
Gets the first record for a stream_slice of a stream.
Parameters
- stream: stream instance from which to read records
- stream_slice: stream_slice parameters for slicing the stream
Raises
- StopIteration: if there is no first record to return (the read_records generator is empty)
Returns
StreamData containing the first record in the slice
18class HttpAvailabilityStrategy(AvailabilityStrategy): 19 def check_availability( 20 self, stream: Stream, logger: logging.Logger, source: Optional["Source"] = None 21 ) -> Tuple[bool, Optional[str]]: 22 """ 23 Check stream availability by attempting to read the first record of the 24 stream. 25 26 :param stream: stream 27 :param logger: source logger 28 :param source: (optional) source 29 :return: A tuple of (boolean, str). If boolean is true, then the stream 30 is available, and no str is required. Otherwise, the stream is unavailable 31 for some reason and the str should describe what went wrong and how to 32 resolve the unavailability, if possible. 33 """ 34 reason: Optional[str] 35 try: 36 # Some streams need a stream slice to read records (e.g. if they have a SubstreamPartitionRouter) 37 # Streams that don't need a stream slice will return `None` as their first stream slice. 38 stream_slice = self.get_first_stream_slice(stream) 39 except StopIteration: 40 # If stream_slices has no `next()` item (Note - this is different from stream_slices returning [None]!) 41 # This can happen when a substream's `stream_slices` method does a `for record in parent_records: yield <something>` 42 # without accounting for the case in which the parent stream is empty. 43 reason = f"Cannot attempt to connect to stream {stream.name} - no stream slices were found, likely because the parent stream is empty." 44 return False, reason 45 except AirbyteTracedException as error: 46 return False, error.message 47 48 try: 49 self.get_first_record_for_slice(stream, stream_slice) 50 return True, None 51 except StopIteration: 52 logger.info(f"Successfully connected to stream {stream.name}, but got 0 records.") 53 return True, None 54 except AirbyteTracedException as error: 55 return False, error.message
Abstract base class for checking stream availability.
19 def check_availability( 20 self, stream: Stream, logger: logging.Logger, source: Optional["Source"] = None 21 ) -> Tuple[bool, Optional[str]]: 22 """ 23 Check stream availability by attempting to read the first record of the 24 stream. 25 26 :param stream: stream 27 :param logger: source logger 28 :param source: (optional) source 29 :return: A tuple of (boolean, str). If boolean is true, then the stream 30 is available, and no str is required. Otherwise, the stream is unavailable 31 for some reason and the str should describe what went wrong and how to 32 resolve the unavailability, if possible. 33 """ 34 reason: Optional[str] 35 try: 36 # Some streams need a stream slice to read records (e.g. if they have a SubstreamPartitionRouter) 37 # Streams that don't need a stream slice will return `None` as their first stream slice. 38 stream_slice = self.get_first_stream_slice(stream) 39 except StopIteration: 40 # If stream_slices has no `next()` item (Note - this is different from stream_slices returning [None]!) 41 # This can happen when a substream's `stream_slices` method does a `for record in parent_records: yield <something>` 42 # without accounting for the case in which the parent stream is empty. 43 reason = f"Cannot attempt to connect to stream {stream.name} - no stream slices were found, likely because the parent stream is empty." 44 return False, reason 45 except AirbyteTracedException as error: 46 return False, error.message 47 48 try: 49 self.get_first_record_for_slice(stream, stream_slice) 50 return True, None 51 except StopIteration: 52 logger.info(f"Successfully connected to stream {stream.name}, but got 0 records.") 53 return True, None 54 except AirbyteTracedException as error: 55 return False, error.message
Check stream availability by attempting to read the first record of the stream.
Parameters
- stream: stream
- logger: source logger
- source: (optional) source
Returns
A tuple of (boolean, str). If boolean is true, then the stream is available, and no str is required. Otherwise, the stream is unavailable for some reason and the str should describe what went wrong and how to resolve the unavailability, if possible.
Inherited Members
11@dataclass 12class ResumableFullRefreshCursor(Cursor): 13 """ 14 Cursor that allows for the checkpointing of sync progress according to a synthetic cursor based on the pagination state 15 of the stream. Resumable full refresh syncs are only intended to retain state in between sync attempts of the same job 16 with the platform responsible for removing said state. 17 """ 18 19 def __init__(self) -> None: 20 self._cursor: StreamState = {} 21 22 def get_stream_state(self) -> StreamState: 23 return self._cursor 24 25 def set_initial_state(self, stream_state: StreamState) -> None: 26 self._cursor = stream_state 27 28 def observe(self, stream_slice: StreamSlice, record: Record) -> None: 29 """ 30 Resumable full refresh manages state using a page number so it does not need to update state by observing incoming records. 31 """ 32 pass 33 34 def close_slice(self, stream_slice: StreamSlice, *args: Any) -> None: 35 self._cursor = stream_slice.cursor_slice 36 37 def should_be_synced(self, record: Record) -> bool: 38 """ 39 Unlike date-based cursors which filter out records outside slice boundaries, resumable full refresh records exist within pages 40 that don't have filterable bounds. We should always return them. 41 """ 42 return True 43 44 def select_state(self, stream_slice: Optional[StreamSlice] = None) -> Optional[StreamState]: 45 # A top-level RFR cursor only manages the state of a single partition 46 return self._cursor
Cursor that allows for the checkpointing of sync progress according to a synthetic cursor based on the pagination state of the stream. Resumable full refresh syncs are only intended to retain state in between sync attempts of the same job with the platform responsible for removing said state.
Returns the current stream state. We would like to restrict it's usage since it does expose internal of state. As of 2023-06-14, it is used for two things:
- Interpolation of the requests
- Transformation of records
- Saving the state
For the first case, we are probably stuck with exposing the stream state. For the second, we can probably expose a method that allows for emitting the state to the platform.
Cursors are not initialized with their state. As state is needed in order to function properly, this method should be called before calling anything else
Parameters
- stream_state: The state of the stream as returned by get_stream_state
28 def observe(self, stream_slice: StreamSlice, record: Record) -> None: 29 """ 30 Resumable full refresh manages state using a page number so it does not need to update state by observing incoming records. 31 """ 32 pass
Resumable full refresh manages state using a page number so it does not need to update state by observing incoming records.
34 def close_slice(self, stream_slice: StreamSlice, *args: Any) -> None: 35 self._cursor = stream_slice.cursor_slice
Update state based on the stream slice. Note that stream_slice.cursor_slice and most_recent_record.associated_slice are expected
to be the same but we make it explicit here that stream_slice should be leveraged to update the state. We do not pass in the
latest record, since cursor instances should maintain the relevant internal state on their own.
Parameters
- stream_slice: slice to close
37 def should_be_synced(self, record: Record) -> bool: 38 """ 39 Unlike date-based cursors which filter out records outside slice boundaries, resumable full refresh records exist within pages 40 that don't have filterable bounds. We should always return them. 41 """ 42 return True
Unlike date-based cursors which filter out records outside slice boundaries, resumable full refresh records exist within pages that don't have filterable bounds. We should always return them.
44 def select_state(self, stream_slice: Optional[StreamSlice] = None) -> Optional[StreamState]: 45 # A top-level RFR cursor only manages the state of a single partition 46 return self._cursor
Get the state value of a specific stream_slice. For incremental or resumable full refresh cursors which only manage state in a single dimension this is the entire state object. For per-partition cursors used by substreams, this returns the state of a specific parent delineated by the incoming slice's partition object.
179class ConcurrentCursor(Cursor): 180 _START_BOUNDARY = 0 181 _END_BOUNDARY = 1 182 183 def copy_without_state(self) -> "ConcurrentCursor": 184 return self.__class__( 185 stream_name=self._stream_name, 186 stream_namespace=self._stream_namespace, 187 stream_state={}, 188 message_repository=NoopMessageRepository(), 189 connector_state_manager=ConnectorStateManager(), 190 connector_state_converter=self._connector_state_converter, 191 cursor_field=self._cursor_field, 192 slice_boundary_fields=self._slice_boundary_fields, 193 start=self._start, 194 end_provider=self._end_provider, 195 lookback_window=self._lookback_window, 196 slice_range=self._slice_range, 197 cursor_granularity=self._cursor_granularity, 198 clamping_strategy=self._clamping_strategy, 199 is_compare_strictly=self._is_compare_strictly, 200 ) 201 202 def __init__( 203 self, 204 stream_name: str, 205 stream_namespace: Optional[str], 206 stream_state: Any, 207 message_repository: MessageRepository, 208 connector_state_manager: ConnectorStateManager, 209 connector_state_converter: AbstractStreamStateConverter, 210 cursor_field: CursorField, 211 slice_boundary_fields: Optional[Tuple[str, str]], 212 start: Optional[CursorValueType], 213 end_provider: Callable[[], CursorValueType], 214 lookback_window: Optional[GapType] = None, 215 slice_range: Optional[GapType] = None, 216 cursor_granularity: Optional[GapType] = None, 217 clamping_strategy: ClampingStrategy = NoClamping(), 218 is_compare_strictly: bool = False, 219 ) -> None: 220 self._stream_name = stream_name 221 self._stream_namespace = stream_namespace 222 self._message_repository = message_repository 223 self._connector_state_converter = connector_state_converter 224 self._connector_state_manager = connector_state_manager 225 self._cursor_field = cursor_field 226 # To see some example where the slice boundaries might not be defined, check https://github.com/airbytehq/airbyte/blob/1ce84d6396e446e1ac2377362446e3fb94509461/airbyte-integrations/connectors/source-stripe/source_stripe/streams.py#L363-L379 227 self._slice_boundary_fields = slice_boundary_fields 228 self._start = start 229 self._end_provider = end_provider 230 self.start, self._concurrent_state = self._get_concurrent_state(stream_state) 231 self._lookback_window = lookback_window 232 self._slice_range = slice_range 233 self._most_recent_cursor_value_per_partition: MutableMapping[ 234 Union[StreamSlice, Mapping[str, Any], None], Any 235 ] = {} 236 self._has_closed_at_least_one_slice = False 237 self._cursor_granularity = cursor_granularity 238 # Flag to track if the logger has been triggered (per stream) 239 self._should_be_synced_logger_triggered = False 240 self._clamping_strategy = clamping_strategy 241 self._is_compare_strictly = is_compare_strictly 242 self._is_ascending_order = True 243 244 # A lock is required when closing a partition because updating the cursor's concurrent_state is 245 # not thread safe. When multiple partitions are being closed by the cursor at the same time, it is 246 # possible for one partition to update concurrent_state after a second partition has already read 247 # the previous state. This can lead to the second partition overwriting the previous one's state. 248 self._lock = threading.Lock() 249 250 @property 251 def state(self) -> MutableMapping[str, Any]: 252 return self._connector_state_converter.convert_to_state_message( 253 self.cursor_field, self._concurrent_state 254 ) 255 256 @property 257 def cursor_field(self) -> CursorField: 258 return self._cursor_field 259 260 @property 261 def _slice_boundary_fields_wrapper(self) -> Tuple[str, str]: 262 return ( 263 self._slice_boundary_fields 264 if self._slice_boundary_fields 265 else ( 266 self._connector_state_converter.START_KEY, 267 self._connector_state_converter.END_KEY, 268 ) 269 ) 270 271 def _get_concurrent_state( 272 self, state: MutableMapping[str, Any] 273 ) -> Tuple[CursorValueType, MutableMapping[str, Any]]: 274 if self._connector_state_converter.is_state_message_compatible(state): 275 partitioned_state = self._connector_state_converter.deserialize(state) 276 slices_from_partitioned_state = partitioned_state.get("slices", []) 277 278 value_from_partitioned_state = None 279 if slices_from_partitioned_state: 280 # We assume here that the slices have been already merged 281 first_slice = slices_from_partitioned_state[0] 282 value_from_partitioned_state = ( 283 first_slice[self._connector_state_converter.MOST_RECENT_RECORD_KEY] 284 if self._connector_state_converter.MOST_RECENT_RECORD_KEY in first_slice 285 else first_slice[self._connector_state_converter.END_KEY] 286 ) 287 return ( 288 value_from_partitioned_state 289 or self._start 290 or self._connector_state_converter.zero_value, 291 partitioned_state, 292 ) 293 return self._connector_state_converter.convert_from_sequential_state( 294 self._cursor_field, state, self._start 295 ) 296 297 def observe(self, record: Record) -> None: 298 # Because observe writes to the most_recent_cursor_value_per_partition mapping, 299 # it is not thread-safe. However, this shouldn't lead to concurrency issues because 300 # observe() is only invoked by PartitionReader.process_partition(). Since the map is 301 # broken down according to partition, concurrent threads processing only read/write 302 # from different keys which avoids any conflicts. 303 # 304 # If we were to add thread safety, we should implement a lock per-partition 305 # which is instantiated during stream_slices() 306 most_recent_cursor_value = self._most_recent_cursor_value_per_partition.get( 307 record.associated_slice 308 ) 309 try: 310 cursor_value = self._extract_cursor_value(record) 311 312 if most_recent_cursor_value is None or most_recent_cursor_value < cursor_value: 313 self._most_recent_cursor_value_per_partition[record.associated_slice] = cursor_value 314 elif most_recent_cursor_value > cursor_value: 315 self._is_ascending_order = False 316 except ValueError: 317 self._log_for_record_without_cursor_value() 318 319 def _extract_cursor_value(self, record: Record) -> Any: 320 return self._connector_state_converter.parse_value(self._cursor_field.extract_value(record)) 321 322 def close_partition(self, partition: Partition) -> None: 323 with self._lock: 324 slice_count_before = len(self._concurrent_state.get("slices", [])) 325 self._add_slice_to_state(partition) 326 if slice_count_before < len( 327 self._concurrent_state["slices"] 328 ): # only emit if at least one slice has been processed 329 self._merge_partitions() 330 self._emit_state_message() 331 self._has_closed_at_least_one_slice = True 332 333 def _add_slice_to_state(self, partition: Partition) -> None: 334 most_recent_cursor_value = self._most_recent_cursor_value_per_partition.get( 335 partition.to_slice() 336 ) 337 338 if self._slice_boundary_fields: 339 if "slices" not in self._concurrent_state: 340 raise RuntimeError( 341 f"The state for stream {self._stream_name} should have at least one slice to delineate the sync start time, but no slices are present. This is unexpected. Please contact Support." 342 ) 343 self._concurrent_state["slices"].append( 344 { 345 self._connector_state_converter.START_KEY: self._extract_from_slice( 346 partition, self._slice_boundary_fields[self._START_BOUNDARY] 347 ), 348 self._connector_state_converter.END_KEY: self._extract_from_slice( 349 partition, self._slice_boundary_fields[self._END_BOUNDARY] 350 ), 351 self._connector_state_converter.MOST_RECENT_RECORD_KEY: most_recent_cursor_value, 352 } 353 ) 354 elif most_recent_cursor_value: 355 if self._has_closed_at_least_one_slice: 356 # If we track state value using records cursor field, we can only do that if there is one partition. This is because we save 357 # the state every time we close a partition. We assume that if there are multiple slices, they need to be providing 358 # boundaries. There are cases where partitions could not have boundaries: 359 # * The cursor should be per-partition 360 # * The stream state is actually the parent stream state 361 # There might be other cases not listed above. Those are not supported today hence the stream should not use this cursor for 362 # state management. For the specific user that was affected with this issue, we need to: 363 # * Fix state tracking (which is currently broken) 364 # * Make the new version available 365 # * (Probably) ask the user to reset the stream to avoid data loss 366 raise ValueError( 367 "Given that slice_boundary_fields is not defined and that per-partition state is not supported, only one slice is " 368 "expected. Please contact the Airbyte team." 369 ) 370 371 self._concurrent_state["slices"].append( 372 { 373 self._connector_state_converter.START_KEY: self.start, 374 self._connector_state_converter.END_KEY: most_recent_cursor_value, 375 self._connector_state_converter.MOST_RECENT_RECORD_KEY: most_recent_cursor_value, 376 } 377 ) 378 379 def _emit_state_message(self) -> None: 380 self._connector_state_manager.update_state_for_stream( 381 self._stream_name, 382 self._stream_namespace, 383 self.state, 384 ) 385 state_message = self._connector_state_manager.create_state_message( 386 self._stream_name, self._stream_namespace 387 ) 388 self._message_repository.emit_message(state_message) 389 390 def _merge_partitions(self) -> None: 391 self._concurrent_state["slices"] = self._connector_state_converter.merge_intervals( 392 self._concurrent_state["slices"] 393 ) 394 395 def _extract_from_slice(self, partition: Partition, key: str) -> CursorValueType: 396 try: 397 _slice = partition.to_slice() 398 if not _slice: 399 raise KeyError(f"Could not find key `{key}` in empty slice") 400 return self._connector_state_converter.parse_value(_slice[key]) # type: ignore # we expect the devs to specify a key that would return a CursorValueType 401 except KeyError as exception: 402 raise KeyError( 403 f"Partition is expected to have key `{key}` but could not be found" 404 ) from exception 405 406 def ensure_at_least_one_state_emitted(self) -> None: 407 """ 408 The platform expect to have at least one state message on successful syncs. Hence, whatever happens, we expect this method to be 409 called. 410 """ 411 self._emit_state_message() 412 413 def stream_slices(self) -> Iterable[StreamSlice]: 414 """ 415 Generating slices based on a few parameters: 416 * lookback_window: Buffer to remove from END_KEY of the highest slice 417 * slice_range: Max difference between two slices. If the difference between two slices is greater, multiple slices will be created 418 * start: `_split_per_slice_range` will clip any value to `self._start which means that: 419 * if upper is less than self._start, no slices will be generated 420 * if lower is less than self._start, self._start will be used as the lower boundary (lookback_window will not be considered in that case) 421 422 Note that the slices will overlap at their boundaries. We therefore expect to have at least the lower or the upper boundary to be 423 inclusive in the API that is queried. 424 """ 425 self._merge_partitions() 426 427 if self._start is not None and self._is_start_before_first_slice(): 428 yield from self._split_per_slice_range( 429 self._start, 430 self._concurrent_state["slices"][0][self._connector_state_converter.START_KEY], 431 False, 432 ) 433 434 if len(self._concurrent_state["slices"]) == 1: 435 yield from self._split_per_slice_range( 436 self._calculate_lower_boundary_of_last_slice( 437 self._concurrent_state["slices"][0][self._connector_state_converter.END_KEY] 438 ), 439 self._end_provider(), 440 True, 441 ) 442 elif len(self._concurrent_state["slices"]) > 1: 443 for i in range(len(self._concurrent_state["slices"]) - 1): 444 if self._cursor_granularity: 445 yield from self._split_per_slice_range( 446 self._concurrent_state["slices"][i][self._connector_state_converter.END_KEY] 447 + self._cursor_granularity, 448 self._concurrent_state["slices"][i + 1][ 449 self._connector_state_converter.START_KEY 450 ], 451 False, 452 ) 453 else: 454 yield from self._split_per_slice_range( 455 self._concurrent_state["slices"][i][ 456 self._connector_state_converter.END_KEY 457 ], 458 self._concurrent_state["slices"][i + 1][ 459 self._connector_state_converter.START_KEY 460 ], 461 False, 462 ) 463 yield from self._split_per_slice_range( 464 self._calculate_lower_boundary_of_last_slice( 465 self._concurrent_state["slices"][-1][self._connector_state_converter.END_KEY] 466 ), 467 self._end_provider(), 468 True, 469 ) 470 else: 471 raise ValueError("Expected at least one slice") 472 473 def _is_start_before_first_slice(self) -> bool: 474 return ( 475 self._start is not None 476 and self._start 477 < self._concurrent_state["slices"][0][self._connector_state_converter.START_KEY] 478 ) 479 480 def _calculate_lower_boundary_of_last_slice( 481 self, lower_boundary: CursorValueType 482 ) -> CursorValueType: 483 if self._lookback_window: 484 return lower_boundary - self._lookback_window 485 return lower_boundary 486 487 def _split_per_slice_range( 488 self, lower: CursorValueType, upper: CursorValueType, upper_is_end: bool 489 ) -> Iterable[StreamSlice]: 490 if lower > upper or (lower == upper and not self._cursor_granularity): 491 return 492 493 if self._start and upper < self._start: 494 return 495 496 lower = max(lower, self._start) if self._start else lower 497 # Clamp first, else a step shorter than the clamping period collapses the first window 498 lower = self._clamping_strategy.clamp(lower) 499 # With a granularity the last window includes upper, so lower == upper is one unit to read, 500 # unless the API rejects start == end. In a gap, the next slice already reads upper. 501 if lower > upper or (lower == upper and (self._is_compare_strictly or not upper_is_end)): 502 return 503 if not self._slice_range or self._evaluate_upper_safely(lower, self._slice_range) >= upper: 504 # With a granularity upper is the last unit to read: clamping it would read past the end 505 clamped_upper = ( 506 upper 507 if upper_is_end and self._cursor_granularity 508 else self._clamping_strategy.clamp(upper) 509 ) 510 start_value, end_value = ( 511 (lower, clamped_upper - self._cursor_granularity) 512 if self._cursor_granularity and not upper_is_end 513 else (lower, clamped_upper) 514 ) 515 yield StreamSlice( 516 partition={}, 517 cursor_slice={ 518 self._slice_boundary_fields_wrapper[ 519 self._START_BOUNDARY 520 ]: self._connector_state_converter.output_format(start_value), 521 self._slice_boundary_fields_wrapper[ 522 self._END_BOUNDARY 523 ]: self._connector_state_converter.output_format(end_value), 524 }, 525 ) 526 else: 527 stop_processing = False 528 current_lower_boundary = lower 529 while not stop_processing: 530 current_upper_boundary = min( 531 self._evaluate_upper_safely(current_lower_boundary, self._slice_range), upper 532 ) 533 clamped_upper = ( 534 self._clamping_strategy.clamp(current_upper_boundary) 535 if current_upper_boundary != upper 536 else current_upper_boundary 537 ) 538 # A boundary clamped onto upper makes this the last window, else upper is dropped 539 has_reached_upper_boundary = ( 540 current_upper_boundary >= upper or clamped_upper == upper 541 ) 542 clamped_lower = self._clamping_strategy.clamp(current_lower_boundary) 543 if clamped_lower >= clamped_upper: 544 # clamping collapsed both values which means that it is time to stop processing 545 # FIXME should this be replace by proper end_provider 546 break 547 start_value, end_value = ( 548 (clamped_lower, clamped_upper - self._cursor_granularity) 549 if self._cursor_granularity 550 and (not upper_is_end or not has_reached_upper_boundary) 551 else (clamped_lower, clamped_upper) 552 ) 553 yield StreamSlice( 554 partition={}, 555 cursor_slice={ 556 self._slice_boundary_fields_wrapper[ 557 self._START_BOUNDARY 558 ]: self._connector_state_converter.output_format(start_value), 559 self._slice_boundary_fields_wrapper[ 560 self._END_BOUNDARY 561 ]: self._connector_state_converter.output_format(end_value), 562 }, 563 ) 564 current_lower_boundary = clamped_upper 565 if has_reached_upper_boundary: 566 stop_processing = True 567 568 def _evaluate_upper_safely(self, lower: CursorValueType, step: GapType) -> CursorValueType: 569 """ 570 Given that we set the default step at datetime.timedelta.max, we will generate an OverflowError when evaluating the next start_date 571 This method assumes that users would never enter a step that would generate an overflow. Given that would be the case, the code 572 would have broken anyway. 573 """ 574 try: 575 return lower + step 576 except OverflowError: 577 return self._end_provider() 578 579 def should_be_synced(self, record: Record) -> bool: 580 """ 581 Determines if a record should be synced based on its cursor value. 582 :param record: The record to evaluate 583 584 :return: True if the record's cursor value falls within the sync boundaries 585 """ 586 try: 587 record_cursor_value: CursorValueType = self._extract_cursor_value(record) 588 except ValueError: 589 self._log_for_record_without_cursor_value() 590 return True 591 return self._lowest_cursor_value_to_sync() <= record_cursor_value <= self._end_provider() 592 593 def _lowest_cursor_value_to_sync(self) -> CursorValueType: 594 # Keep the records the lookback window re-reads, never below the configured start. If the state is below the 595 # configured start, keep records from the state rather than raising the floor to the configured start. 596 if not self._lookback_window: 597 return self.start 598 try: 599 lower = self.start - self._lookback_window 600 except OverflowError: 601 return self.start 602 return max(lower, min(self.start, self._start)) if self._start else lower 603 604 def _log_for_record_without_cursor_value(self) -> None: 605 if not self._should_be_synced_logger_triggered: 606 LOGGER.warning( 607 f"Could not find cursor field `{self.cursor_field.cursor_field_key}` in record for stream {self._stream_name}. The incremental sync will assume it needs to be synced" 608 ) 609 self._should_be_synced_logger_triggered = True 610 611 def reduce_slice_range(self, stream_slice: StreamSlice) -> StreamSlice: 612 # In theory, we might be more flexible here meaning that it doesn't need to be in ascending order but it just 613 # needs to be ordered. For now though, we will only support ascending order. 614 if not self._is_ascending_order: 615 LOGGER.warning( 616 "Attempting to reduce slice while records are not returned in incremental order might lead to missing records" 617 ) 618 619 if stream_slice in self._most_recent_cursor_value_per_partition: 620 return StreamSlice( 621 partition=stream_slice.partition, 622 cursor_slice={ 623 self._slice_boundary_fields_wrapper[ 624 self._START_BOUNDARY 625 ]: self._connector_state_converter.output_format( 626 self._most_recent_cursor_value_per_partition[stream_slice] 627 ), 628 self._slice_boundary_fields_wrapper[ 629 self._END_BOUNDARY 630 ]: stream_slice.cursor_slice[ 631 self._slice_boundary_fields_wrapper[self._END_BOUNDARY] 632 ], 633 }, 634 extra_fields=stream_slice.extra_fields, 635 ) 636 else: 637 return stream_slice 638 639 def split_request_window( 640 self, stream_slice: StreamSlice, min_split_window: Optional[datetime.timedelta] = None 641 ) -> Optional[List[StreamSlice]]: 642 """ 643 Split `stream_slice` in half along this cursor's own boundary fields, granularity, and output format. 644 645 Returns `None` when the slice cannot be split any further: either it spans at most one 646 `cursor_granularity` unit already (the floor), it is already at or below `min_split_window` (a 647 connector-configured floor independent of and typically looser than `cursor_granularity`), the 648 boundaries are reversed, or no `cursor_granularity` was configured at all (there is then no way to know 649 the smallest addressable unit, or how to keep two children from overlapping at their shared edge). 650 Raises `AirbyteTracedException` instead if a boundary is missing or unparseable: that is a bug in how 651 the slice was produced, not the API rejecting a window, so it must not be reported as one. This is the 652 implementation `SimpleRetriever.request_window_splitter` binds to (see 653 `airbyte_cdk.sources.declarative.retrievers.simple_retriever`) that reuses the same 654 parsing/formatting/comparison operations `_split_per_slice_range` already relies on, so it works for 655 any `CursorValueType`/`GapType` pair this cursor was built with (datetime/timedelta, or int/int), 656 without a second, duplicate implementation of that logic living outside the cursor. 657 """ 658 if not self._cursor_granularity: 659 return None 660 661 start_field, end_field = self._slice_boundary_fields_wrapper 662 try: 663 start_value = self._connector_state_converter.parse_value( 664 stream_slice.cursor_slice[start_field] 665 ) 666 end_value = self._connector_state_converter.parse_value( 667 stream_slice.cursor_slice[end_field] 668 ) 669 except (KeyError, ValueError, TypeError) as exception: 670 # A missing or unparseable boundary means this slice was produced (or passed in) wrong - a bug in 671 # slice generation, not something the API did. Raising here, rather than returning `None` (which 672 # `SimpleRetriever` reports as "the API kept rejecting..."), keeps that distinction visible instead 673 # of blaming the API for our own state. 674 raise AirbyteTracedException( 675 internal_message=( 676 f"Cannot split {stream_slice}: boundary field {start_field!r} or {end_field!r} is missing " 677 f"or not parseable: {exception}" 678 ), 679 message=( 680 f"Stream {self._stream_name} produced a request window with a missing or invalid " 681 f"{start_field!r}/{end_field!r} boundary and could not split it further." 682 ), 683 failure_type=FailureType.system_error, 684 ) from exception 685 686 if start_value >= end_value: 687 # Reversed or degenerate boundaries: nothing safe to split. 688 return None 689 690 span = end_value - start_value # type: ignore[operator] # concretely a GapType (timedelta/int) at runtime 691 if min_split_window is not None and span <= min_split_window: # type: ignore[operator] 692 return None 693 694 total_units = span // self._cursor_granularity + 1 # type: ignore[operator] # inclusive count of granularity units 695 if total_units <= 1: 696 # Already at the minimum granularity and still rejected; nothing left to split. 697 return None 698 699 half_units = total_units // 2 700 first_end = ( 701 start_value + (half_units * self._cursor_granularity) - self._cursor_granularity # type: ignore[operator] 702 ) 703 second_start = first_end + self._cursor_granularity 704 705 if not (start_value <= first_end < second_start <= end_value): 706 # No-progress guard, independent of the granularity check above: refuse to return children that 707 # are not strictly ordered and non-degenerate, rather than risk an identical or overlapping split. 708 return None 709 710 return [ 711 StreamSlice( 712 partition=stream_slice.partition, 713 cursor_slice={ 714 start_field: self._connector_state_converter.output_format(start_value), 715 end_field: self._connector_state_converter.output_format(first_end), 716 }, 717 extra_fields=stream_slice.extra_fields, 718 ), 719 StreamSlice( 720 partition=stream_slice.partition, 721 cursor_slice={ 722 start_field: self._connector_state_converter.output_format(second_start), 723 end_field: self._connector_state_converter.output_format(end_value), 724 }, 725 extra_fields=stream_slice.extra_fields, 726 ), 727 ] 728 729 def get_cursor_datetime_from_state( 730 self, stream_state: Mapping[str, Any] 731 ) -> datetime.datetime | None: 732 """Extract and parse the cursor datetime from the given stream state. 733 734 For concurrent cursors, the state can be in two formats: 735 1. Sequential/legacy format: {cursor_field: cursor_value} 736 2. Concurrent format: {state_type: "date-range", slices: [...]} 737 738 Returns the cursor datetime if present and parseable, otherwise returns None. 739 """ 740 # Check if state is in concurrent format (need to convert to dict for type compatibility) 741 mutable_state: MutableMapping[str, Any] = dict(stream_state) 742 if self._connector_state_converter.is_state_message_compatible(mutable_state): 743 slices = stream_state.get("slices", []) 744 if not slices: 745 return None 746 # Get the most recent cursor value from the first slice (after merging) 747 first_slice = slices[0] 748 cursor_value = first_slice.get( 749 self._connector_state_converter.MOST_RECENT_RECORD_KEY 750 ) or first_slice.get(self._connector_state_converter.END_KEY) 751 if not cursor_value: 752 return None 753 try: 754 parsed_value = self._connector_state_converter.parse_value(cursor_value) 755 if isinstance(parsed_value, datetime.datetime): 756 return parsed_value 757 return None 758 except (ValueError, TypeError): 759 return None 760 761 # Sequential/legacy format: {cursor_field: cursor_value} 762 cursor_value = stream_state.get(self._cursor_field.cursor_field_key) 763 if not cursor_value: 764 return None 765 try: 766 parsed_value = self._connector_state_converter.parse_value(cursor_value) 767 if isinstance(parsed_value, datetime.datetime): 768 return parsed_value 769 return None 770 except (ValueError, TypeError): 771 return None
Slices the stream into chunks that can be fetched independently. Slices enable state checkpointing and data retrieval parallelization.
202 def __init__( 203 self, 204 stream_name: str, 205 stream_namespace: Optional[str], 206 stream_state: Any, 207 message_repository: MessageRepository, 208 connector_state_manager: ConnectorStateManager, 209 connector_state_converter: AbstractStreamStateConverter, 210 cursor_field: CursorField, 211 slice_boundary_fields: Optional[Tuple[str, str]], 212 start: Optional[CursorValueType], 213 end_provider: Callable[[], CursorValueType], 214 lookback_window: Optional[GapType] = None, 215 slice_range: Optional[GapType] = None, 216 cursor_granularity: Optional[GapType] = None, 217 clamping_strategy: ClampingStrategy = NoClamping(), 218 is_compare_strictly: bool = False, 219 ) -> None: 220 self._stream_name = stream_name 221 self._stream_namespace = stream_namespace 222 self._message_repository = message_repository 223 self._connector_state_converter = connector_state_converter 224 self._connector_state_manager = connector_state_manager 225 self._cursor_field = cursor_field 226 # To see some example where the slice boundaries might not be defined, check https://github.com/airbytehq/airbyte/blob/1ce84d6396e446e1ac2377362446e3fb94509461/airbyte-integrations/connectors/source-stripe/source_stripe/streams.py#L363-L379 227 self._slice_boundary_fields = slice_boundary_fields 228 self._start = start 229 self._end_provider = end_provider 230 self.start, self._concurrent_state = self._get_concurrent_state(stream_state) 231 self._lookback_window = lookback_window 232 self._slice_range = slice_range 233 self._most_recent_cursor_value_per_partition: MutableMapping[ 234 Union[StreamSlice, Mapping[str, Any], None], Any 235 ] = {} 236 self._has_closed_at_least_one_slice = False 237 self._cursor_granularity = cursor_granularity 238 # Flag to track if the logger has been triggered (per stream) 239 self._should_be_synced_logger_triggered = False 240 self._clamping_strategy = clamping_strategy 241 self._is_compare_strictly = is_compare_strictly 242 self._is_ascending_order = True 243 244 # A lock is required when closing a partition because updating the cursor's concurrent_state is 245 # not thread safe. When multiple partitions are being closed by the cursor at the same time, it is 246 # possible for one partition to update concurrent_state after a second partition has already read 247 # the previous state. This can lead to the second partition overwriting the previous one's state. 248 self._lock = threading.Lock()
183 def copy_without_state(self) -> "ConcurrentCursor": 184 return self.__class__( 185 stream_name=self._stream_name, 186 stream_namespace=self._stream_namespace, 187 stream_state={}, 188 message_repository=NoopMessageRepository(), 189 connector_state_manager=ConnectorStateManager(), 190 connector_state_converter=self._connector_state_converter, 191 cursor_field=self._cursor_field, 192 slice_boundary_fields=self._slice_boundary_fields, 193 start=self._start, 194 end_provider=self._end_provider, 195 lookback_window=self._lookback_window, 196 slice_range=self._slice_range, 197 cursor_granularity=self._cursor_granularity, 198 clamping_strategy=self._clamping_strategy, 199 is_compare_strictly=self._is_compare_strictly, 200 )
297 def observe(self, record: Record) -> None: 298 # Because observe writes to the most_recent_cursor_value_per_partition mapping, 299 # it is not thread-safe. However, this shouldn't lead to concurrency issues because 300 # observe() is only invoked by PartitionReader.process_partition(). Since the map is 301 # broken down according to partition, concurrent threads processing only read/write 302 # from different keys which avoids any conflicts. 303 # 304 # If we were to add thread safety, we should implement a lock per-partition 305 # which is instantiated during stream_slices() 306 most_recent_cursor_value = self._most_recent_cursor_value_per_partition.get( 307 record.associated_slice 308 ) 309 try: 310 cursor_value = self._extract_cursor_value(record) 311 312 if most_recent_cursor_value is None or most_recent_cursor_value < cursor_value: 313 self._most_recent_cursor_value_per_partition[record.associated_slice] = cursor_value 314 elif most_recent_cursor_value > cursor_value: 315 self._is_ascending_order = False 316 except ValueError: 317 self._log_for_record_without_cursor_value()
Indicate to the cursor that the record has been emitted
322 def close_partition(self, partition: Partition) -> None: 323 with self._lock: 324 slice_count_before = len(self._concurrent_state.get("slices", [])) 325 self._add_slice_to_state(partition) 326 if slice_count_before < len( 327 self._concurrent_state["slices"] 328 ): # only emit if at least one slice has been processed 329 self._merge_partitions() 330 self._emit_state_message() 331 self._has_closed_at_least_one_slice = True
Indicate to the cursor that the partition has been successfully processed
406 def ensure_at_least_one_state_emitted(self) -> None: 407 """ 408 The platform expect to have at least one state message on successful syncs. Hence, whatever happens, we expect this method to be 409 called. 410 """ 411 self._emit_state_message()
The platform expect to have at least one state message on successful syncs. Hence, whatever happens, we expect this method to be called.
413 def stream_slices(self) -> Iterable[StreamSlice]: 414 """ 415 Generating slices based on a few parameters: 416 * lookback_window: Buffer to remove from END_KEY of the highest slice 417 * slice_range: Max difference between two slices. If the difference between two slices is greater, multiple slices will be created 418 * start: `_split_per_slice_range` will clip any value to `self._start which means that: 419 * if upper is less than self._start, no slices will be generated 420 * if lower is less than self._start, self._start will be used as the lower boundary (lookback_window will not be considered in that case) 421 422 Note that the slices will overlap at their boundaries. We therefore expect to have at least the lower or the upper boundary to be 423 inclusive in the API that is queried. 424 """ 425 self._merge_partitions() 426 427 if self._start is not None and self._is_start_before_first_slice(): 428 yield from self._split_per_slice_range( 429 self._start, 430 self._concurrent_state["slices"][0][self._connector_state_converter.START_KEY], 431 False, 432 ) 433 434 if len(self._concurrent_state["slices"]) == 1: 435 yield from self._split_per_slice_range( 436 self._calculate_lower_boundary_of_last_slice( 437 self._concurrent_state["slices"][0][self._connector_state_converter.END_KEY] 438 ), 439 self._end_provider(), 440 True, 441 ) 442 elif len(self._concurrent_state["slices"]) > 1: 443 for i in range(len(self._concurrent_state["slices"]) - 1): 444 if self._cursor_granularity: 445 yield from self._split_per_slice_range( 446 self._concurrent_state["slices"][i][self._connector_state_converter.END_KEY] 447 + self._cursor_granularity, 448 self._concurrent_state["slices"][i + 1][ 449 self._connector_state_converter.START_KEY 450 ], 451 False, 452 ) 453 else: 454 yield from self._split_per_slice_range( 455 self._concurrent_state["slices"][i][ 456 self._connector_state_converter.END_KEY 457 ], 458 self._concurrent_state["slices"][i + 1][ 459 self._connector_state_converter.START_KEY 460 ], 461 False, 462 ) 463 yield from self._split_per_slice_range( 464 self._calculate_lower_boundary_of_last_slice( 465 self._concurrent_state["slices"][-1][self._connector_state_converter.END_KEY] 466 ), 467 self._end_provider(), 468 True, 469 ) 470 else: 471 raise ValueError("Expected at least one slice")
Generating slices based on a few parameters:
- lookback_window: Buffer to remove from END_KEY of the highest slice
- slice_range: Max difference between two slices. If the difference between two slices is greater, multiple slices will be created
- start:
_split_per_slice_rangewill clip any value to `self._start which means that:- if upper is less than self._start, no slices will be generated
- if lower is less than self._start, self._start will be used as the lower boundary (lookback_window will not be considered in that case)
Note that the slices will overlap at their boundaries. We therefore expect to have at least the lower or the upper boundary to be inclusive in the API that is queried.
579 def should_be_synced(self, record: Record) -> bool: 580 """ 581 Determines if a record should be synced based on its cursor value. 582 :param record: The record to evaluate 583 584 :return: True if the record's cursor value falls within the sync boundaries 585 """ 586 try: 587 record_cursor_value: CursorValueType = self._extract_cursor_value(record) 588 except ValueError: 589 self._log_for_record_without_cursor_value() 590 return True 591 return self._lowest_cursor_value_to_sync() <= record_cursor_value <= self._end_provider()
Determines if a record should be synced based on its cursor value.
Parameters
- record: The record to evaluate
Returns
True if the record's cursor value falls within the sync boundaries
611 def reduce_slice_range(self, stream_slice: StreamSlice) -> StreamSlice: 612 # In theory, we might be more flexible here meaning that it doesn't need to be in ascending order but it just 613 # needs to be ordered. For now though, we will only support ascending order. 614 if not self._is_ascending_order: 615 LOGGER.warning( 616 "Attempting to reduce slice while records are not returned in incremental order might lead to missing records" 617 ) 618 619 if stream_slice in self._most_recent_cursor_value_per_partition: 620 return StreamSlice( 621 partition=stream_slice.partition, 622 cursor_slice={ 623 self._slice_boundary_fields_wrapper[ 624 self._START_BOUNDARY 625 ]: self._connector_state_converter.output_format( 626 self._most_recent_cursor_value_per_partition[stream_slice] 627 ), 628 self._slice_boundary_fields_wrapper[ 629 self._END_BOUNDARY 630 ]: stream_slice.cursor_slice[ 631 self._slice_boundary_fields_wrapper[self._END_BOUNDARY] 632 ], 633 }, 634 extra_fields=stream_slice.extra_fields, 635 ) 636 else: 637 return stream_slice
639 def split_request_window( 640 self, stream_slice: StreamSlice, min_split_window: Optional[datetime.timedelta] = None 641 ) -> Optional[List[StreamSlice]]: 642 """ 643 Split `stream_slice` in half along this cursor's own boundary fields, granularity, and output format. 644 645 Returns `None` when the slice cannot be split any further: either it spans at most one 646 `cursor_granularity` unit already (the floor), it is already at or below `min_split_window` (a 647 connector-configured floor independent of and typically looser than `cursor_granularity`), the 648 boundaries are reversed, or no `cursor_granularity` was configured at all (there is then no way to know 649 the smallest addressable unit, or how to keep two children from overlapping at their shared edge). 650 Raises `AirbyteTracedException` instead if a boundary is missing or unparseable: that is a bug in how 651 the slice was produced, not the API rejecting a window, so it must not be reported as one. This is the 652 implementation `SimpleRetriever.request_window_splitter` binds to (see 653 `airbyte_cdk.sources.declarative.retrievers.simple_retriever`) that reuses the same 654 parsing/formatting/comparison operations `_split_per_slice_range` already relies on, so it works for 655 any `CursorValueType`/`GapType` pair this cursor was built with (datetime/timedelta, or int/int), 656 without a second, duplicate implementation of that logic living outside the cursor. 657 """ 658 if not self._cursor_granularity: 659 return None 660 661 start_field, end_field = self._slice_boundary_fields_wrapper 662 try: 663 start_value = self._connector_state_converter.parse_value( 664 stream_slice.cursor_slice[start_field] 665 ) 666 end_value = self._connector_state_converter.parse_value( 667 stream_slice.cursor_slice[end_field] 668 ) 669 except (KeyError, ValueError, TypeError) as exception: 670 # A missing or unparseable boundary means this slice was produced (or passed in) wrong - a bug in 671 # slice generation, not something the API did. Raising here, rather than returning `None` (which 672 # `SimpleRetriever` reports as "the API kept rejecting..."), keeps that distinction visible instead 673 # of blaming the API for our own state. 674 raise AirbyteTracedException( 675 internal_message=( 676 f"Cannot split {stream_slice}: boundary field {start_field!r} or {end_field!r} is missing " 677 f"or not parseable: {exception}" 678 ), 679 message=( 680 f"Stream {self._stream_name} produced a request window with a missing or invalid " 681 f"{start_field!r}/{end_field!r} boundary and could not split it further." 682 ), 683 failure_type=FailureType.system_error, 684 ) from exception 685 686 if start_value >= end_value: 687 # Reversed or degenerate boundaries: nothing safe to split. 688 return None 689 690 span = end_value - start_value # type: ignore[operator] # concretely a GapType (timedelta/int) at runtime 691 if min_split_window is not None and span <= min_split_window: # type: ignore[operator] 692 return None 693 694 total_units = span // self._cursor_granularity + 1 # type: ignore[operator] # inclusive count of granularity units 695 if total_units <= 1: 696 # Already at the minimum granularity and still rejected; nothing left to split. 697 return None 698 699 half_units = total_units // 2 700 first_end = ( 701 start_value + (half_units * self._cursor_granularity) - self._cursor_granularity # type: ignore[operator] 702 ) 703 second_start = first_end + self._cursor_granularity 704 705 if not (start_value <= first_end < second_start <= end_value): 706 # No-progress guard, independent of the granularity check above: refuse to return children that 707 # are not strictly ordered and non-degenerate, rather than risk an identical or overlapping split. 708 return None 709 710 return [ 711 StreamSlice( 712 partition=stream_slice.partition, 713 cursor_slice={ 714 start_field: self._connector_state_converter.output_format(start_value), 715 end_field: self._connector_state_converter.output_format(first_end), 716 }, 717 extra_fields=stream_slice.extra_fields, 718 ), 719 StreamSlice( 720 partition=stream_slice.partition, 721 cursor_slice={ 722 start_field: self._connector_state_converter.output_format(second_start), 723 end_field: self._connector_state_converter.output_format(end_value), 724 }, 725 extra_fields=stream_slice.extra_fields, 726 ), 727 ]
Split stream_slice in half along this cursor's own boundary fields, granularity, and output format.
Returns None when the slice cannot be split any further: either it spans at most one
cursor_granularity unit already (the floor), it is already at or below min_split_window (a
connector-configured floor independent of and typically looser than cursor_granularity), the
boundaries are reversed, or no cursor_granularity was configured at all (there is then no way to know
the smallest addressable unit, or how to keep two children from overlapping at their shared edge).
Raises AirbyteTracedException instead if a boundary is missing or unparseable: that is a bug in how
the slice was produced, not the API rejecting a window, so it must not be reported as one. This is the
implementation SimpleRetriever.request_window_splitter binds to (see
airbyte_cdk.sources.declarative.retrievers.simple_retriever) that reuses the same
parsing/formatting/comparison operations _split_per_slice_range already relies on, so it works for
any CursorValueType/GapType pair this cursor was built with (datetime/timedelta, or int/int),
without a second, duplicate implementation of that logic living outside the cursor.
729 def get_cursor_datetime_from_state( 730 self, stream_state: Mapping[str, Any] 731 ) -> datetime.datetime | None: 732 """Extract and parse the cursor datetime from the given stream state. 733 734 For concurrent cursors, the state can be in two formats: 735 1. Sequential/legacy format: {cursor_field: cursor_value} 736 2. Concurrent format: {state_type: "date-range", slices: [...]} 737 738 Returns the cursor datetime if present and parseable, otherwise returns None. 739 """ 740 # Check if state is in concurrent format (need to convert to dict for type compatibility) 741 mutable_state: MutableMapping[str, Any] = dict(stream_state) 742 if self._connector_state_converter.is_state_message_compatible(mutable_state): 743 slices = stream_state.get("slices", []) 744 if not slices: 745 return None 746 # Get the most recent cursor value from the first slice (after merging) 747 first_slice = slices[0] 748 cursor_value = first_slice.get( 749 self._connector_state_converter.MOST_RECENT_RECORD_KEY 750 ) or first_slice.get(self._connector_state_converter.END_KEY) 751 if not cursor_value: 752 return None 753 try: 754 parsed_value = self._connector_state_converter.parse_value(cursor_value) 755 if isinstance(parsed_value, datetime.datetime): 756 return parsed_value 757 return None 758 except (ValueError, TypeError): 759 return None 760 761 # Sequential/legacy format: {cursor_field: cursor_value} 762 cursor_value = stream_state.get(self._cursor_field.cursor_field_key) 763 if not cursor_value: 764 return None 765 try: 766 parsed_value = self._connector_state_converter.parse_value(cursor_value) 767 if isinstance(parsed_value, datetime.datetime): 768 return parsed_value 769 return None 770 except (ValueError, TypeError): 771 return None
Extract and parse the cursor datetime from the given stream state.
For concurrent cursors, the state can be in two formats:
- Sequential/legacy format: {cursor_field: cursor_value}
- Concurrent format: {state_type: "date-range", slices: [...]}
Returns the cursor datetime if present and parseable, otherwise returns None.
31class ConcurrentSource: 32 """ 33 A Source that reads data from multiple AbstractStreams concurrently. 34 It does so by submitting partition generation, and partition read tasks to a thread pool. 35 The tasks asynchronously add their output to a shared queue. 36 The read is done when all partitions for all streams w ere generated and read. 37 """ 38 39 DEFAULT_TIMEOUT_SECONDS = 900 # Deprecated: default of the ignored `timeout_seconds`. 40 41 @staticmethod 42 def create( 43 num_workers: int, 44 initial_number_of_partitions_to_generate: int, 45 logger: logging.Logger, 46 slice_logger: SliceLogger, 47 message_repository: MessageRepository, 48 queue: Optional[Queue[QueueItem]] = None, 49 timeout_seconds: int = DEFAULT_TIMEOUT_SECONDS, 50 ) -> "ConcurrentSource": 51 if initial_number_of_partitions_to_generate < 1: 52 raise ValueError( 53 f"initial_number_of_partitions_to_generate must be >= 1, got {initial_number_of_partitions_to_generate}" 54 ) 55 is_single_threaded = initial_number_of_partitions_to_generate == 1 and num_workers == 1 56 too_many_generator = ( 57 not is_single_threaded and initial_number_of_partitions_to_generate >= num_workers 58 ) 59 assert not too_many_generator, ( 60 "It is required to have more workers than threads generating partitions" 61 ) 62 threadpool = ThreadPoolManager( 63 concurrent.futures.ThreadPoolExecutor( 64 max_workers=num_workers, thread_name_prefix="workerpool" 65 ), 66 logger, 67 ) 68 return ConcurrentSource( 69 threadpool=threadpool, 70 logger=logger, 71 slice_logger=slice_logger, 72 queue=queue, 73 message_repository=message_repository, 74 initial_number_partitions_to_generate=initial_number_of_partitions_to_generate, 75 timeout_seconds=timeout_seconds, 76 ) 77 78 def __init__( 79 self, 80 threadpool: ThreadPoolManager, 81 logger: logging.Logger, 82 slice_logger: SliceLogger = DebugSliceLogger(), 83 queue: Optional[Queue[QueueItem]] = None, 84 message_repository: MessageRepository = InMemoryMessageRepository(), 85 initial_number_partitions_to_generate: int = 1, 86 timeout_seconds: int = DEFAULT_TIMEOUT_SECONDS, 87 ) -> None: 88 """ 89 :param threadpool: The threadpool to submit tasks to 90 :param logger: The logger to log to 91 :param slice_logger: The slice logger used to create messages on new slices 92 :param message_repository: The repository to emit messages to 93 :param initial_number_partitions_to_generate: The initial number of concurrent partition generation tasks. Limiting this number ensures will limit the latency of the first records emitted. While the latency is not critical, emitting the records early allows the platform and the destination to process them as early as possible. 94 :param timeout_seconds: Deprecated and ignored. Kept so existing callers do not break. 95 """ 96 self._threadpool = threadpool 97 self._logger = logger 98 self._slice_logger = slice_logger 99 self._message_repository = message_repository 100 self._initial_number_partitions_to_generate = initial_number_partitions_to_generate 101 102 # We set a maxsize to for the main thread to process record items when the queue size grows. This assumes that there are less 103 # threads generating partitions that than are max number of workers. If it weren't the case, we could have threads only generating 104 # partitions which would fill the queue. This number is arbitrarily set to 10_000 but will probably need to be changed given more 105 # information and might even need to be configurable depending on the source 106 self._queue = queue or Queue(maxsize=10_000) 107 108 def read( 109 self, 110 streams: List[AbstractStream], 111 ) -> Iterator[AirbyteMessage]: 112 self._logger.info("Starting syncing") 113 concurrent_stream_processor = ConcurrentReadProcessor( 114 streams, 115 PartitionEnqueuer(self._queue, self._threadpool), 116 self._threadpool, 117 self._logger, 118 self._slice_logger, 119 self._message_repository, 120 PartitionReader( 121 self._queue, 122 PartitionLogger(self._slice_logger, self._logger, self._message_repository), 123 ), 124 max_concurrent_partition_generators=self._initial_number_partitions_to_generate, 125 ) 126 127 # Enqueue initial partition generation tasks 128 yield from self._submit_initial_partition_generators(concurrent_stream_processor) 129 130 # Read from the queue until all partitions were generated and read 131 yield from self._consume_from_queue( 132 self._queue, 133 concurrent_stream_processor, 134 ) 135 self._threadpool.check_for_errors_and_shutdown() 136 self._logger.info("Finished syncing") 137 138 def _submit_initial_partition_generators( 139 self, concurrent_stream_processor: ConcurrentReadProcessor 140 ) -> Iterable[AirbyteMessage]: 141 for _ in range(self._initial_number_partitions_to_generate): 142 status_message = concurrent_stream_processor.start_next_partition_generator() 143 if status_message: 144 yield status_message 145 146 def _consume_from_queue( 147 self, 148 queue: Queue[QueueItem], 149 concurrent_stream_processor: ConcurrentReadProcessor, 150 ) -> Iterable[AirbyteMessage]: 151 while airbyte_message_or_record_or_exception := queue.get(): 152 yield from self._handle_item( 153 airbyte_message_or_record_or_exception, 154 concurrent_stream_processor, 155 ) 156 # In the event that a partition raises an exception, anything remaining in 157 # the queue will be missed because is_done() can raise an exception and exit 158 # out of this loop before remaining items are consumed 159 if queue.empty() and concurrent_stream_processor.is_done(): 160 # all partitions were generated and processed. we're done here 161 break 162 163 def _handle_item( 164 self, 165 queue_item: QueueItem, 166 concurrent_stream_processor: ConcurrentReadProcessor, 167 ) -> Iterable[AirbyteMessage]: 168 # handle queue item and call the appropriate handler depending on the type of the queue item 169 if isinstance(queue_item, StreamThreadException): 170 yield from concurrent_stream_processor.on_exception(queue_item) 171 elif isinstance(queue_item, PartitionGenerationCompletedSentinel): 172 yield from concurrent_stream_processor.on_partition_generation_completed(queue_item) 173 elif isinstance(queue_item, Partition): 174 concurrent_stream_processor.on_partition(queue_item) 175 elif isinstance(queue_item, PartitionCompleteSentinel): 176 yield from concurrent_stream_processor.on_partition_complete_sentinel(queue_item) 177 elif isinstance(queue_item, Record): 178 yield from concurrent_stream_processor.on_record(queue_item) 179 elif isinstance(queue_item, AirbyteMessage): 180 yield queue_item 181 else: 182 raise ValueError(f"Unknown queue item type: {type(queue_item)}")
A Source that reads data from multiple AbstractStreams concurrently. It does so by submitting partition generation, and partition read tasks to a thread pool. The tasks asynchronously add their output to a shared queue. The read is done when all partitions for all streams w ere generated and read.
78 def __init__( 79 self, 80 threadpool: ThreadPoolManager, 81 logger: logging.Logger, 82 slice_logger: SliceLogger = DebugSliceLogger(), 83 queue: Optional[Queue[QueueItem]] = None, 84 message_repository: MessageRepository = InMemoryMessageRepository(), 85 initial_number_partitions_to_generate: int = 1, 86 timeout_seconds: int = DEFAULT_TIMEOUT_SECONDS, 87 ) -> None: 88 """ 89 :param threadpool: The threadpool to submit tasks to 90 :param logger: The logger to log to 91 :param slice_logger: The slice logger used to create messages on new slices 92 :param message_repository: The repository to emit messages to 93 :param initial_number_partitions_to_generate: The initial number of concurrent partition generation tasks. Limiting this number ensures will limit the latency of the first records emitted. While the latency is not critical, emitting the records early allows the platform and the destination to process them as early as possible. 94 :param timeout_seconds: Deprecated and ignored. Kept so existing callers do not break. 95 """ 96 self._threadpool = threadpool 97 self._logger = logger 98 self._slice_logger = slice_logger 99 self._message_repository = message_repository 100 self._initial_number_partitions_to_generate = initial_number_partitions_to_generate 101 102 # We set a maxsize to for the main thread to process record items when the queue size grows. This assumes that there are less 103 # threads generating partitions that than are max number of workers. If it weren't the case, we could have threads only generating 104 # partitions which would fill the queue. This number is arbitrarily set to 10_000 but will probably need to be changed given more 105 # information and might even need to be configurable depending on the source 106 self._queue = queue or Queue(maxsize=10_000)
Parameters
- threadpool: The threadpool to submit tasks to
- logger: The logger to log to
- slice_logger: The slice logger used to create messages on new slices
- message_repository: The repository to emit messages to
- initial_number_partitions_to_generate: The initial number of concurrent partition generation tasks. Limiting this number ensures will limit the latency of the first records emitted. While the latency is not critical, emitting the records early allows the platform and the destination to process them as early as possible.
- timeout_seconds: Deprecated and ignored. Kept so existing callers do not break.
41 @staticmethod 42 def create( 43 num_workers: int, 44 initial_number_of_partitions_to_generate: int, 45 logger: logging.Logger, 46 slice_logger: SliceLogger, 47 message_repository: MessageRepository, 48 queue: Optional[Queue[QueueItem]] = None, 49 timeout_seconds: int = DEFAULT_TIMEOUT_SECONDS, 50 ) -> "ConcurrentSource": 51 if initial_number_of_partitions_to_generate < 1: 52 raise ValueError( 53 f"initial_number_of_partitions_to_generate must be >= 1, got {initial_number_of_partitions_to_generate}" 54 ) 55 is_single_threaded = initial_number_of_partitions_to_generate == 1 and num_workers == 1 56 too_many_generator = ( 57 not is_single_threaded and initial_number_of_partitions_to_generate >= num_workers 58 ) 59 assert not too_many_generator, ( 60 "It is required to have more workers than threads generating partitions" 61 ) 62 threadpool = ThreadPoolManager( 63 concurrent.futures.ThreadPoolExecutor( 64 max_workers=num_workers, thread_name_prefix="workerpool" 65 ), 66 logger, 67 ) 68 return ConcurrentSource( 69 threadpool=threadpool, 70 logger=logger, 71 slice_logger=slice_logger, 72 queue=queue, 73 message_repository=message_repository, 74 initial_number_partitions_to_generate=initial_number_of_partitions_to_generate, 75 timeout_seconds=timeout_seconds, 76 )
108 def read( 109 self, 110 streams: List[AbstractStream], 111 ) -> Iterator[AirbyteMessage]: 112 self._logger.info("Starting syncing") 113 concurrent_stream_processor = ConcurrentReadProcessor( 114 streams, 115 PartitionEnqueuer(self._queue, self._threadpool), 116 self._threadpool, 117 self._logger, 118 self._slice_logger, 119 self._message_repository, 120 PartitionReader( 121 self._queue, 122 PartitionLogger(self._slice_logger, self._logger, self._message_repository), 123 ), 124 max_concurrent_partition_generators=self._initial_number_partitions_to_generate, 125 ) 126 127 # Enqueue initial partition generation tasks 128 yield from self._submit_initial_partition_generators(concurrent_stream_processor) 129 130 # Read from the queue until all partitions were generated and read 131 yield from self._consume_from_queue( 132 self._queue, 133 concurrent_stream_processor, 134 ) 135 self._threadpool.check_for_errors_and_shutdown() 136 self._logger.info("Finished syncing")
34class ConcurrentSourceAdapter(AbstractSource, ABC): 35 def __init__(self, concurrent_source: ConcurrentSource, **kwargs: Any) -> None: 36 """ 37 ConcurrentSourceAdapter is a Source that wraps a concurrent source and exposes it as a regular source. 38 39 The source's streams are still defined through the streams() method. 40 Streams wrapped in a StreamFacade will be processed concurrently. 41 Other streams will be processed sequentially as a later step. 42 """ 43 self._concurrent_source = concurrent_source 44 super().__init__(**kwargs) 45 46 def read( 47 self, 48 logger: logging.Logger, 49 config: Mapping[str, Any], 50 catalog: ConfiguredAirbyteCatalog, 51 state: Optional[List[AirbyteStateMessage]] = None, 52 ) -> Iterator[AirbyteMessage]: 53 abstract_streams = self._select_abstract_streams(config, catalog) 54 concurrent_stream_names = {stream.name for stream in abstract_streams} 55 configured_catalog_for_regular_streams = ConfiguredAirbyteCatalog( 56 streams=[ 57 stream 58 for stream in catalog.streams 59 if stream.stream.name not in concurrent_stream_names 60 ] 61 ) 62 if abstract_streams: 63 yield from self._concurrent_source.read(abstract_streams) 64 if configured_catalog_for_regular_streams.streams: 65 yield from super().read(logger, config, configured_catalog_for_regular_streams, state) 66 67 def _select_abstract_streams( 68 self, config: Mapping[str, Any], configured_catalog: ConfiguredAirbyteCatalog 69 ) -> List[AbstractStream]: 70 """ 71 Selects streams that can be processed concurrently and returns their abstract representations. 72 """ 73 all_streams = self.streams(config) 74 stream_name_to_instance: Mapping[str, Stream] = {s.name: s for s in all_streams} 75 abstract_streams: List[AbstractStream] = [] 76 for configured_stream in configured_catalog.streams: 77 stream_instance = stream_name_to_instance.get(configured_stream.stream.name) 78 if not stream_instance: 79 continue 80 81 if isinstance(stream_instance, AbstractStreamFacade): 82 abstract_streams.append(stream_instance.get_underlying_stream()) 83 return abstract_streams 84 85 def convert_to_concurrent_stream( 86 self, 87 logger: logging.Logger, 88 stream: Stream, 89 state_manager: ConnectorStateManager, 90 cursor: Optional[Cursor] = None, 91 ) -> Stream: 92 """ 93 Prepares a stream for concurrent processing by initializing or assigning a cursor, 94 managing the stream's state, and returning an updated Stream instance. 95 """ 96 state: MutableMapping[str, Any] = {} 97 98 if cursor: 99 state = state_manager.get_stream_state(stream.name, stream.namespace) 100 101 stream.cursor = cursor # type: ignore[assignment] # cursor is of type ConcurrentCursor, which inherits from Cursor 102 if hasattr(stream, "parent"): 103 stream.parent.cursor = cursor 104 else: 105 cursor = FinalStateCursor( 106 stream_name=stream.name, 107 stream_namespace=stream.namespace, 108 message_repository=self.message_repository, # type: ignore[arg-type] # _default_message_repository will be returned in the worst case 109 ) 110 return StreamFacade.create_from_stream(stream, self, logger, state, cursor) 111 112 def initialize_cursor( 113 self, 114 stream: Stream, 115 state_manager: ConnectorStateManager, 116 converter: AbstractStreamStateConverter, 117 slice_boundary_fields: Optional[Tuple[str, str]], 118 start: Optional[CursorValueType], 119 end_provider: Callable[[], CursorValueType], 120 lookback_window: Optional[GapType] = None, 121 slice_range: Optional[GapType] = None, 122 ) -> Optional[ConcurrentCursor]: 123 lookback_window = lookback_window or timedelta(seconds=DEFAULT_LOOKBACK_SECONDS) 124 125 cursor_field_name = stream.cursor_field 126 127 if cursor_field_name: 128 if not isinstance(cursor_field_name, str): 129 raise ValueError( 130 f"Cursor field type must be a string, but received {type(cursor_field_name).__name__}." 131 ) 132 133 return ConcurrentCursor( 134 stream.name, 135 stream.namespace, 136 state_manager.get_stream_state(stream.name, stream.namespace), 137 self.message_repository, # type: ignore[arg-type] # _default_message_repository will be returned in the worst case 138 state_manager, 139 converter, 140 CursorField(cursor_field_name), 141 slice_boundary_fields, 142 start, 143 end_provider, 144 lookback_window, 145 slice_range, 146 ) 147 148 return None
Abstract base class for an Airbyte Source. Consumers should implement any abstract methods in this class to create an Airbyte Specification compliant Source.
35 def __init__(self, concurrent_source: ConcurrentSource, **kwargs: Any) -> None: 36 """ 37 ConcurrentSourceAdapter is a Source that wraps a concurrent source and exposes it as a regular source. 38 39 The source's streams are still defined through the streams() method. 40 Streams wrapped in a StreamFacade will be processed concurrently. 41 Other streams will be processed sequentially as a later step. 42 """ 43 self._concurrent_source = concurrent_source 44 super().__init__(**kwargs)
ConcurrentSourceAdapter is a Source that wraps a concurrent source and exposes it as a regular source.
The source's streams are still defined through the streams() method. Streams wrapped in a StreamFacade will be processed concurrently. Other streams will be processed sequentially as a later step.
46 def read( 47 self, 48 logger: logging.Logger, 49 config: Mapping[str, Any], 50 catalog: ConfiguredAirbyteCatalog, 51 state: Optional[List[AirbyteStateMessage]] = None, 52 ) -> Iterator[AirbyteMessage]: 53 abstract_streams = self._select_abstract_streams(config, catalog) 54 concurrent_stream_names = {stream.name for stream in abstract_streams} 55 configured_catalog_for_regular_streams = ConfiguredAirbyteCatalog( 56 streams=[ 57 stream 58 for stream in catalog.streams 59 if stream.stream.name not in concurrent_stream_names 60 ] 61 ) 62 if abstract_streams: 63 yield from self._concurrent_source.read(abstract_streams) 64 if configured_catalog_for_regular_streams.streams: 65 yield from super().read(logger, config, configured_catalog_for_regular_streams, state)
Implements the Read operation from the Airbyte Specification. See https://docs.airbyte.com/understanding-airbyte/airbyte-protocol/.
85 def convert_to_concurrent_stream( 86 self, 87 logger: logging.Logger, 88 stream: Stream, 89 state_manager: ConnectorStateManager, 90 cursor: Optional[Cursor] = None, 91 ) -> Stream: 92 """ 93 Prepares a stream for concurrent processing by initializing or assigning a cursor, 94 managing the stream's state, and returning an updated Stream instance. 95 """ 96 state: MutableMapping[str, Any] = {} 97 98 if cursor: 99 state = state_manager.get_stream_state(stream.name, stream.namespace) 100 101 stream.cursor = cursor # type: ignore[assignment] # cursor is of type ConcurrentCursor, which inherits from Cursor 102 if hasattr(stream, "parent"): 103 stream.parent.cursor = cursor 104 else: 105 cursor = FinalStateCursor( 106 stream_name=stream.name, 107 stream_namespace=stream.namespace, 108 message_repository=self.message_repository, # type: ignore[arg-type] # _default_message_repository will be returned in the worst case 109 ) 110 return StreamFacade.create_from_stream(stream, self, logger, state, cursor)
Prepares a stream for concurrent processing by initializing or assigning a cursor, managing the stream's state, and returning an updated Stream instance.
112 def initialize_cursor( 113 self, 114 stream: Stream, 115 state_manager: ConnectorStateManager, 116 converter: AbstractStreamStateConverter, 117 slice_boundary_fields: Optional[Tuple[str, str]], 118 start: Optional[CursorValueType], 119 end_provider: Callable[[], CursorValueType], 120 lookback_window: Optional[GapType] = None, 121 slice_range: Optional[GapType] = None, 122 ) -> Optional[ConcurrentCursor]: 123 lookback_window = lookback_window or timedelta(seconds=DEFAULT_LOOKBACK_SECONDS) 124 125 cursor_field_name = stream.cursor_field 126 127 if cursor_field_name: 128 if not isinstance(cursor_field_name, str): 129 raise ValueError( 130 f"Cursor field type must be a string, but received {type(cursor_field_name).__name__}." 131 ) 132 133 return ConcurrentCursor( 134 stream.name, 135 stream.namespace, 136 state_manager.get_stream_state(stream.name, stream.namespace), 137 self.message_repository, # type: ignore[arg-type] # _default_message_repository will be returned in the worst case 138 state_manager, 139 converter, 140 CursorField(cursor_field_name), 141 slice_boundary_fields, 142 start, 143 end_provider, 144 lookback_window, 145 slice_range, 146 ) 147 148 return None
Inherited Members
58class Cursor(StreamSlicer, ABC): 59 @property 60 @abstractmethod 61 def state(self) -> MutableMapping[str, Any]: ... 62 63 @abstractmethod 64 def observe(self, record: Record) -> None: 65 """ 66 Indicate to the cursor that the record has been emitted 67 """ 68 raise NotImplementedError() 69 70 @abstractmethod 71 def close_partition(self, partition: Partition) -> None: 72 """ 73 Indicate to the cursor that the partition has been successfully processed 74 """ 75 raise NotImplementedError() 76 77 @abstractmethod 78 def ensure_at_least_one_state_emitted(self) -> None: 79 """ 80 State messages are emitted when a partition is closed. However, the platform expects at least one state to be emitted per sync per 81 stream. Hence, if no partitions are generated, this method needs to be called. 82 """ 83 raise NotImplementedError() 84 85 @abstractmethod 86 def should_be_synced(self, record: Record) -> bool: 87 pass 88 89 def stream_slices(self) -> Iterable[StreamSlice]: 90 """ 91 Default placeholder implementation of generate_slices. 92 Subclasses can override this method to provide actual behavior. 93 """ 94 yield StreamSlice(partition={}, cursor_slice={}) 95 96 def get_cursor_datetime_from_state( 97 self, stream_state: Mapping[str, Any] 98 ) -> datetime.datetime | None: 99 """Extract and parse the cursor datetime from the given stream state. 100 101 This method is used by StateDelegatingStream to validate cursor age against 102 an API's data retention period. Subclasses should implement this method to 103 extract the cursor value from their specific state structure and parse it 104 into a datetime object. 105 106 Returns None if the cursor cannot be extracted or parsed, which will cause 107 StateDelegatingStream to fall back to full refresh (safe default). 108 109 Raises NotImplementedError by default - subclasses must implement this method 110 if they want to support cursor age validation with api_retention_period. 111 """ 112 raise NotImplementedError( 113 f"{self.__class__.__name__} does not implement get_cursor_datetime_from_state. " 114 f"Cursor age validation with api_retention_period is not supported for this cursor type." 115 )
Slices the stream into chunks that can be fetched independently. Slices enable state checkpointing and data retrieval parallelization.
63 @abstractmethod 64 def observe(self, record: Record) -> None: 65 """ 66 Indicate to the cursor that the record has been emitted 67 """ 68 raise NotImplementedError()
Indicate to the cursor that the record has been emitted
70 @abstractmethod 71 def close_partition(self, partition: Partition) -> None: 72 """ 73 Indicate to the cursor that the partition has been successfully processed 74 """ 75 raise NotImplementedError()
Indicate to the cursor that the partition has been successfully processed
77 @abstractmethod 78 def ensure_at_least_one_state_emitted(self) -> None: 79 """ 80 State messages are emitted when a partition is closed. However, the platform expects at least one state to be emitted per sync per 81 stream. Hence, if no partitions are generated, this method needs to be called. 82 """ 83 raise NotImplementedError()
State messages are emitted when a partition is closed. However, the platform expects at least one state to be emitted per sync per stream. Hence, if no partitions are generated, this method needs to be called.
89 def stream_slices(self) -> Iterable[StreamSlice]: 90 """ 91 Default placeholder implementation of generate_slices. 92 Subclasses can override this method to provide actual behavior. 93 """ 94 yield StreamSlice(partition={}, cursor_slice={})
Default placeholder implementation of generate_slices. Subclasses can override this method to provide actual behavior.
96 def get_cursor_datetime_from_state( 97 self, stream_state: Mapping[str, Any] 98 ) -> datetime.datetime | None: 99 """Extract and parse the cursor datetime from the given stream state. 100 101 This method is used by StateDelegatingStream to validate cursor age against 102 an API's data retention period. Subclasses should implement this method to 103 extract the cursor value from their specific state structure and parse it 104 into a datetime object. 105 106 Returns None if the cursor cannot be extracted or parsed, which will cause 107 StateDelegatingStream to fall back to full refresh (safe default). 108 109 Raises NotImplementedError by default - subclasses must implement this method 110 if they want to support cursor age validation with api_retention_period. 111 """ 112 raise NotImplementedError( 113 f"{self.__class__.__name__} does not implement get_cursor_datetime_from_state. " 114 f"Cursor age validation with api_retention_period is not supported for this cursor type." 115 )
Extract and parse the cursor datetime from the given stream state.
This method is used by StateDelegatingStream to validate cursor age against an API's data retention period. Subclasses should implement this method to extract the cursor value from their specific state structure and parse it into a datetime object.
Returns None if the cursor cannot be extracted or parsed, which will cause StateDelegatingStream to fall back to full refresh (safe default).
Raises NotImplementedError by default - subclasses must implement this method if they want to support cursor age validation with api_retention_period.
44class CursorField: 45 def __init__( 46 self, cursor_field_key: str, supports_catalog_defined_cursor_field: bool = False 47 ) -> None: 48 self.cursor_field_key = cursor_field_key 49 self.supports_catalog_defined_cursor_field = supports_catalog_defined_cursor_field 50 51 def extract_value(self, record: Record) -> Any: 52 cursor_value = record.data.get(self.cursor_field_key) 53 if cursor_value is None: 54 raise ValueError(f"Could not find cursor field {self.cursor_field_key} in record") 55 return cursor_value # type: ignore # we assume that the value the path points at is a comparable
51 def extract_value(self, record: Record) -> Any: 52 cursor_value = record.data.get(self.cursor_field_key) 53 if cursor_value is None: 54 raise ValueError(f"Could not find cursor field {self.cursor_field_key} in record") 55 return cursor_value # type: ignore # we assume that the value the path points at is a comparable
115class EpochValueConcurrentStreamStateConverter(DateTimeStreamStateConverter): 116 """ 117 e.g. 118 { "created": 1617030403 } 119 => 120 { 121 "state_type": "date-range", 122 "metadata": { … }, 123 "slices": [ 124 {starts: 0, end: 1617030403, finished_processing: true} 125 ] 126 } 127 """ 128 129 _zero_value = 0 130 131 def increment(self, timestamp: datetime) -> datetime: 132 return timestamp + timedelta(seconds=1) 133 134 def output_format(self, timestamp: datetime) -> int: 135 return int(timestamp.timestamp()) 136 137 def parse_timestamp(self, timestamp: int) -> datetime: 138 dt_object = AirbyteDateTime.fromtimestamp(timestamp, timezone.utc) 139 if not isinstance(dt_object, AirbyteDateTime): 140 raise ValueError( 141 f"AirbyteDateTime object was expected but got {type(dt_object)} from AirbyteDateTime.fromtimestamp({timestamp})" 142 ) 143 return dt_object
e.g. { "created": 1617030403 } => { "state_type": "date-range", "metadata": { … }, "slices": [ {starts: 0, end: 1617030403, finished_processing: true} ] }
131 def increment(self, timestamp: datetime) -> datetime: 132 return timestamp + timedelta(seconds=1)
Increment a timestamp by a single unit.
Convert the cursor value type to a JSON valid type.
137 def parse_timestamp(self, timestamp: int) -> datetime: 138 dt_object = AirbyteDateTime.fromtimestamp(timestamp, timezone.utc) 139 if not isinstance(dt_object, AirbyteDateTime): 140 raise ValueError( 141 f"AirbyteDateTime object was expected but got {type(dt_object)} from AirbyteDateTime.fromtimestamp({timestamp})" 142 ) 143 return dt_object
Inherited Members
118class FinalStateCursor(Cursor): 119 """Cursor that is used to guarantee at least one state message is emitted for a concurrent stream.""" 120 121 def __init__( 122 self, 123 stream_name: str, 124 stream_namespace: Optional[str], 125 message_repository: MessageRepository, 126 ) -> None: 127 self._stream_name = stream_name 128 self._stream_namespace = stream_namespace 129 self._message_repository = message_repository 130 # Normally the connector state manager operates at the source-level. However, we only need it to write the sentinel 131 # state message rather than manage overall source state. This is also only temporary as we move to the resumable 132 # full refresh world where every stream uses a FileBasedConcurrentCursor with incremental state. 133 self._connector_state_manager = ConnectorStateManager() 134 self._has_closed_at_least_one_slice = False 135 136 @property 137 def state(self) -> MutableMapping[str, Any]: 138 return {NO_CURSOR_STATE_KEY: True} 139 140 def observe(self, record: Record) -> None: 141 pass 142 143 def close_partition(self, partition: Partition) -> None: 144 pass 145 146 def ensure_at_least_one_state_emitted(self) -> None: 147 """ 148 Used primarily for full refresh syncs that do not have a valid cursor value to emit at the end of a sync 149 """ 150 151 self._connector_state_manager.update_state_for_stream( 152 self._stream_name, self._stream_namespace, self.state 153 ) 154 state_message = self._connector_state_manager.create_state_message( 155 self._stream_name, self._stream_namespace 156 ) 157 self._message_repository.emit_message(state_message) 158 159 def should_be_synced(self, record: Record) -> bool: 160 return True 161 162 def get_cursor_datetime_from_state( 163 self, stream_state: Mapping[str, Any] 164 ) -> datetime.datetime | None: 165 """Return now() if state indicates a completed full refresh, else None. 166 167 When the state has NO_CURSOR_STATE_KEY: True, it means the previous sync was a 168 completed full refresh. Returning now() indicates the cursor is "current" and 169 within any retention period, so we should use incremental sync. 170 171 For any other state format, return None to indicate this cursor cannot parse it, 172 allowing the incremental cursor to handle the state instead. 173 """ 174 if stream_state.get(NO_CURSOR_STATE_KEY): 175 return datetime.datetime.now(datetime.timezone.utc) 176 return None
Cursor that is used to guarantee at least one state message is emitted for a concurrent stream.
121 def __init__( 122 self, 123 stream_name: str, 124 stream_namespace: Optional[str], 125 message_repository: MessageRepository, 126 ) -> None: 127 self._stream_name = stream_name 128 self._stream_namespace = stream_namespace 129 self._message_repository = message_repository 130 # Normally the connector state manager operates at the source-level. However, we only need it to write the sentinel 131 # state message rather than manage overall source state. This is also only temporary as we move to the resumable 132 # full refresh world where every stream uses a FileBasedConcurrentCursor with incremental state. 133 self._connector_state_manager = ConnectorStateManager() 134 self._has_closed_at_least_one_slice = False
Indicate to the cursor that the partition has been successfully processed
146 def ensure_at_least_one_state_emitted(self) -> None: 147 """ 148 Used primarily for full refresh syncs that do not have a valid cursor value to emit at the end of a sync 149 """ 150 151 self._connector_state_manager.update_state_for_stream( 152 self._stream_name, self._stream_namespace, self.state 153 ) 154 state_message = self._connector_state_manager.create_state_message( 155 self._stream_name, self._stream_namespace 156 ) 157 self._message_repository.emit_message(state_message)
Used primarily for full refresh syncs that do not have a valid cursor value to emit at the end of a sync
162 def get_cursor_datetime_from_state( 163 self, stream_state: Mapping[str, Any] 164 ) -> datetime.datetime | None: 165 """Return now() if state indicates a completed full refresh, else None. 166 167 When the state has NO_CURSOR_STATE_KEY: True, it means the previous sync was a 168 completed full refresh. Returning now() indicates the cursor is "current" and 169 within any retention period, so we should use incremental sync. 170 171 For any other state format, return None to indicate this cursor cannot parse it, 172 allowing the incremental cursor to handle the state instead. 173 """ 174 if stream_state.get(NO_CURSOR_STATE_KEY): 175 return datetime.datetime.now(datetime.timezone.utc) 176 return None
Return now() if state indicates a completed full refresh, else None.
When the state has NO_CURSOR_STATE_KEY: True, it means the previous sync was a completed full refresh. Returning now() indicates the cursor is "current" and within any retention period, so we should use incremental sync.
For any other state format, return None to indicate this cursor cannot parse it, allowing the incremental cursor to handle the state instead.
Inherited Members
146class IsoMillisConcurrentStreamStateConverter(DateTimeStreamStateConverter): 147 """ 148 e.g. 149 { "created": "2021-01-18T21:18:20.000Z" } 150 => 151 { 152 "state_type": "date-range", 153 "metadata": { … }, 154 "slices": [ 155 {starts: "2020-01-18T21:18:20.000Z", end: "2021-01-18T21:18:20.000Z", finished_processing: true} 156 ] 157 } 158 """ 159 160 _zero_value = "0001-01-01T00:00:00.000Z" 161 162 def __init__( 163 self, is_sequential_state: bool = True, cursor_granularity: Optional[timedelta] = None 164 ): 165 super().__init__(is_sequential_state=is_sequential_state) 166 self._cursor_granularity = cursor_granularity or timedelta(milliseconds=1) 167 168 def increment(self, timestamp: datetime) -> datetime: 169 return timestamp + self._cursor_granularity 170 171 def output_format(self, timestamp: datetime) -> str: 172 """Format datetime with milliseconds always included. 173 174 Args: 175 timestamp: The datetime to format. 176 177 Returns: 178 str: ISO8601/RFC3339 formatted string with milliseconds. 179 """ 180 dt = AirbyteDateTime.from_datetime(timestamp) 181 # Always include milliseconds, even if zero 182 millis = dt.microsecond // 1000 if dt.microsecond else 0 183 return f"{dt.year:04d}-{dt.month:02d}-{dt.day:02d}T{dt.hour:02d}:{dt.minute:02d}:{dt.second:02d}.{millis:03d}Z" 184 185 def parse_timestamp(self, timestamp: str) -> datetime: 186 dt_object = ab_datetime_parse(timestamp) 187 if not isinstance(dt_object, AirbyteDateTime): 188 raise ValueError( 189 f"AirbyteDateTime object was expected but got {type(dt_object)} from parse({timestamp})" 190 ) 191 return dt_object
e.g. { "created": "2021-01-18T21:18:20.000Z" } => { "state_type": "date-range", "metadata": { … }, "slices": [ {starts: "2020-01-18T21:18:20.000Z", end: "2021-01-18T21:18:20.000Z", finished_processing: true} ] }
168 def increment(self, timestamp: datetime) -> datetime: 169 return timestamp + self._cursor_granularity
Increment a timestamp by a single unit.
171 def output_format(self, timestamp: datetime) -> str: 172 """Format datetime with milliseconds always included. 173 174 Args: 175 timestamp: The datetime to format. 176 177 Returns: 178 str: ISO8601/RFC3339 formatted string with milliseconds. 179 """ 180 dt = AirbyteDateTime.from_datetime(timestamp) 181 # Always include milliseconds, even if zero 182 millis = dt.microsecond // 1000 if dt.microsecond else 0 183 return f"{dt.year:04d}-{dt.month:02d}-{dt.day:02d}T{dt.hour:02d}:{dt.minute:02d}:{dt.second:02d}.{millis:03d}Z"
Format datetime with milliseconds always included.
Arguments:
- timestamp: The datetime to format.
Returns:
str: ISO8601/RFC3339 formatted string with milliseconds.
Inherited Members
49@deprecated( 50 "This class is experimental. Use at your own risk.", 51 category=ExperimentalClassWarning, 52) 53class StreamFacade(AbstractStreamFacade[DefaultStream], Stream): 54 """ 55 The StreamFacade is a Stream that wraps an AbstractStream and exposes it as a Stream. 56 57 All methods either delegate to the wrapped AbstractStream or provide a default implementation. 58 The default implementations define restrictions imposed on Streams migrated to the new interface. For instance, only source-defined cursors are supported. 59 """ 60 61 @classmethod 62 def create_from_stream( 63 cls, 64 stream: Stream, 65 source: AbstractSource, 66 logger: logging.Logger, 67 state: Optional[MutableMapping[str, Any]], 68 cursor: Cursor, 69 ) -> Stream: 70 """ 71 Create a ConcurrentStream from a Stream object. 72 :param source: The source 73 :param stream: The stream 74 :param max_workers: The maximum number of worker thread to use 75 :return: 76 """ 77 pk = get_primary_key_from_stream(stream.primary_key) 78 cursor_field = get_cursor_field_from_stream(stream) 79 80 if not source.message_repository: 81 raise ValueError( 82 "A message repository is required to emit non-record messages. Please set the message repository on the source." 83 ) 84 85 message_repository = source.message_repository 86 return StreamFacade( 87 DefaultStream( 88 partition_generator=StreamPartitionGenerator( 89 stream, 90 message_repository, 91 SyncMode.full_refresh 92 if isinstance(cursor, FinalStateCursor) 93 else SyncMode.incremental, 94 [cursor_field] if cursor_field is not None else None, 95 state, 96 ), 97 name=stream.name, 98 namespace=stream.namespace, 99 json_schema=stream.get_json_schema(), 100 primary_key=pk, 101 cursor_field=CursorField(cursor_field_key=cursor_field) if cursor_field else None, 102 logger=logger, 103 cursor=cursor, 104 ), 105 stream, 106 cursor, 107 slice_logger=source._slice_logger, 108 logger=logger, 109 ) 110 111 @property 112 def state(self) -> MutableMapping[str, Any]: 113 raise NotImplementedError( 114 "This should not be called as part of the Concurrent CDK code. Please report the problem to Airbyte" 115 ) 116 117 @state.setter 118 def state(self, value: Mapping[str, Any]) -> None: 119 if "state" in dir(self._legacy_stream): 120 self._legacy_stream.state = value # type: ignore # validating `state` is attribute of stream using `if` above 121 122 def __init__( 123 self, 124 stream: DefaultStream, 125 legacy_stream: Stream, 126 cursor: Cursor, 127 slice_logger: SliceLogger, 128 logger: logging.Logger, 129 ): 130 """ 131 :param stream: The underlying AbstractStream 132 """ 133 self._abstract_stream = stream 134 self._legacy_stream = legacy_stream 135 self._cursor = cursor 136 self._slice_logger = slice_logger 137 self._logger = logger 138 139 def read( 140 self, 141 configured_stream: ConfiguredAirbyteStream, 142 logger: logging.Logger, 143 slice_logger: SliceLogger, 144 stream_state: MutableMapping[str, Any], 145 state_manager: ConnectorStateManager, 146 internal_config: InternalConfig, 147 ) -> Iterable[StreamData]: 148 yield from self._read_records() 149 150 def read_records( 151 self, 152 sync_mode: SyncMode, 153 cursor_field: Optional[List[str]] = None, 154 stream_slice: Optional[Mapping[str, Any]] = None, 155 stream_state: Optional[Mapping[str, Any]] = None, 156 ) -> Iterable[StreamData]: 157 try: 158 yield from self._read_records() 159 except Exception as exc: 160 if hasattr(self._cursor, "state"): 161 state = str(self._cursor.state) 162 else: 163 # This shouldn't happen if the ConcurrentCursor was used 164 state = "unknown; no state attribute was available on the cursor" 165 yield AirbyteMessage( 166 type=Type.LOG, 167 log=AirbyteLogMessage( 168 level=Level.ERROR, message=f"Cursor State at time of exception: {state}" 169 ), 170 ) 171 raise exc 172 173 def _read_records(self) -> Iterable[StreamData]: 174 for partition in self._abstract_stream.generate_partitions(): 175 if self._slice_logger.should_log_slice_message(self._logger): 176 yield self._slice_logger.create_slice_log_message(partition.to_slice()) 177 for record in partition.read(): 178 yield record.data 179 180 @property 181 def name(self) -> str: 182 return self._abstract_stream.name 183 184 @property 185 def primary_key(self) -> Optional[Union[str, List[str], List[List[str]]]]: 186 # This method is not expected to be called directly. It is only implemented for backward compatibility with the old interface 187 return self.as_airbyte_stream().source_defined_primary_key # type: ignore # source_defined_primary_key is known to be an Optional[List[List[str]]] 188 189 @property 190 def cursor_field(self) -> Union[str, List[str]]: 191 if self._abstract_stream.cursor_field is None: 192 return [] 193 else: 194 return self._abstract_stream.cursor_field 195 196 @property 197 def cursor(self) -> Optional[Cursor]: # type: ignore[override] # StreamFaced expects to use only airbyte_cdk.sources.streams.concurrent.cursor.Cursor 198 return self._cursor 199 200 @property 201 def block_simultaneous_read(self) -> str: 202 """Returns the blocking group name from the underlying stream""" 203 return self._abstract_stream.block_simultaneous_read 204 205 # FIXME the lru_cache seems to be mostly there because of typing issue 206 @lru_cache(maxsize=None) 207 def get_json_schema(self) -> Mapping[str, Any]: 208 return self._abstract_stream.get_json_schema() 209 210 @property 211 def supports_incremental(self) -> bool: 212 return self._legacy_stream.supports_incremental 213 214 def as_airbyte_stream(self) -> AirbyteStream: 215 return self._abstract_stream.as_airbyte_stream() 216 217 def log_stream_sync_configuration(self) -> None: 218 self._abstract_stream.log_stream_sync_configuration() 219 220 def get_underlying_stream(self) -> DefaultStream: 221 return self._abstract_stream
The StreamFacade is a Stream that wraps an AbstractStream and exposes it as a Stream.
All methods either delegate to the wrapped AbstractStream or provide a default implementation. The default implementations define restrictions imposed on Streams migrated to the new interface. For instance, only source-defined cursors are supported.
122 def __init__( 123 self, 124 stream: DefaultStream, 125 legacy_stream: Stream, 126 cursor: Cursor, 127 slice_logger: SliceLogger, 128 logger: logging.Logger, 129 ): 130 """ 131 :param stream: The underlying AbstractStream 132 """ 133 self._abstract_stream = stream 134 self._legacy_stream = legacy_stream 135 self._cursor = cursor 136 self._slice_logger = slice_logger 137 self._logger = logger
Parameters
- stream: The underlying AbstractStream
61 @classmethod 62 def create_from_stream( 63 cls, 64 stream: Stream, 65 source: AbstractSource, 66 logger: logging.Logger, 67 state: Optional[MutableMapping[str, Any]], 68 cursor: Cursor, 69 ) -> Stream: 70 """ 71 Create a ConcurrentStream from a Stream object. 72 :param source: The source 73 :param stream: The stream 74 :param max_workers: The maximum number of worker thread to use 75 :return: 76 """ 77 pk = get_primary_key_from_stream(stream.primary_key) 78 cursor_field = get_cursor_field_from_stream(stream) 79 80 if not source.message_repository: 81 raise ValueError( 82 "A message repository is required to emit non-record messages. Please set the message repository on the source." 83 ) 84 85 message_repository = source.message_repository 86 return StreamFacade( 87 DefaultStream( 88 partition_generator=StreamPartitionGenerator( 89 stream, 90 message_repository, 91 SyncMode.full_refresh 92 if isinstance(cursor, FinalStateCursor) 93 else SyncMode.incremental, 94 [cursor_field] if cursor_field is not None else None, 95 state, 96 ), 97 name=stream.name, 98 namespace=stream.namespace, 99 json_schema=stream.get_json_schema(), 100 primary_key=pk, 101 cursor_field=CursorField(cursor_field_key=cursor_field) if cursor_field else None, 102 logger=logger, 103 cursor=cursor, 104 ), 105 stream, 106 cursor, 107 slice_logger=source._slice_logger, 108 logger=logger, 109 )
Create a ConcurrentStream from a Stream object.
Parameters
- source: The source
- stream: The stream
- max_workers: The maximum number of worker thread to use
Returns
200 @property 201 def block_simultaneous_read(self) -> str: 202 """Returns the blocking group name from the underlying stream""" 203 return self._abstract_stream.block_simultaneous_read
Returns the blocking group name from the underlying stream
Return the underlying stream facade object.
Inherited Members
- Stream
- logger
- transformer
- cursor
- has_multiple_slices
- name
- get_error_display_message
- read
- read_only_records
- read_records
- get_json_schema
- as_airbyte_stream
- supports_incremental
- is_resumable
- cursor_field
- namespace
- source_defined_cursor
- exit_on_rate_limit
- primary_key
- stream_slices
- state_checkpoint_interval
- get_updated_state
- get_cursor
- log_stream_sync_configuration
- configured_json_schema
106def create_connector_config_control_message(config: MutableMapping[str, Any]) -> AirbyteMessage: 107 control_message = AirbyteControlMessage( 108 type=OrchestratorType.CONNECTOR_CONFIG, 109 emitted_at=time.time() * 1000, 110 connectorConfig=AirbyteControlConnectorConfigMessage(config=config), 111 ) 112 return AirbyteMessage(type=Type.CONTROL, control=control_message)
90def emit_configuration_as_airbyte_control_message(config: MutableMapping[str, Any]) -> None: 91 """Print the connector config as a CONNECTOR_CONFIG control message directly to stdout. 92 93 Single-use OAuth refresh token rotation relies on this direct emission: it reaches the 94 platform for every command (including `check` and `discover`) and does not depend on a 95 message repository being drained. The payload and its newline are written in one `write()` 96 call so a concurrent writer to the shared stdout buffer cannot split the JSON line. 97 """ 98 airbyte_message = create_connector_config_control_message(config) 99 print( 100 f"{orjson.dumps(AirbyteMessageSerializer.dump(airbyte_message)).decode()}\n", 101 end="", 102 flush=True, 103 )
Print the connector config as a CONNECTOR_CONFIG control message directly to stdout.
Single-use OAuth refresh token rotation relies on this direct emission: it reaches the
platform for every command (including check and discover) and does not depend on a
message repository being drained. The payload and its newline are written in one write()
call so a concurrent writer to the shared stdout buffer cannot split the JSON line.
53class AbstractSource(Source, ABC): 54 """ 55 Abstract base class for an Airbyte Source. Consumers should implement any abstract methods 56 in this class to create an Airbyte Specification compliant Source. 57 """ 58 59 @abstractmethod 60 def check_connection( 61 self, logger: logging.Logger, config: Mapping[str, Any] 62 ) -> Tuple[bool, Optional[Any]]: 63 """ 64 :param logger: source logger 65 :param config: The user-provided configuration as specified by the source's spec. 66 This usually contains information required to check connection e.g. tokens, secrets and keys etc. 67 :return: A tuple of (boolean, error). If boolean is true, then the connection check is successful 68 and we can connect to the underlying data source using the provided configuration. 69 Otherwise, the input config cannot be used to connect to the underlying data source, 70 and the "error" object should describe what went wrong. 71 The error object will be cast to string to display the problem to the user. 72 """ 73 74 @abstractmethod 75 def streams(self, config: Mapping[str, Any]) -> List[Stream]: 76 """ 77 :param config: The user-provided configuration as specified by the source's spec. 78 Any stream construction related operation should happen here. 79 :return: A list of the streams in this source connector. 80 """ 81 82 # Stream name to instance map for applying output object transformation 83 _stream_to_instance_map: Dict[str, Stream] = {} 84 _slice_logger: SliceLogger = DebugSliceLogger() 85 86 def discover(self, logger: logging.Logger, config: Mapping[str, Any]) -> AirbyteCatalog: 87 """Implements the Discover operation from the Airbyte Specification. 88 See https://docs.airbyte.com/understanding-airbyte/airbyte-protocol/#discover. 89 """ 90 streams = [stream.as_airbyte_stream() for stream in self.streams(config=config)] 91 return AirbyteCatalog(streams=streams) 92 93 def check(self, logger: logging.Logger, config: Mapping[str, Any]) -> AirbyteConnectionStatus: 94 """Implements the Check Connection operation from the Airbyte Specification. 95 See https://docs.airbyte.com/understanding-airbyte/airbyte-protocol/#check. 96 """ 97 check_succeeded, error = self.check_connection(logger, config) 98 if not check_succeeded: 99 return AirbyteConnectionStatus(status=Status.FAILED, message=repr(error)) 100 return AirbyteConnectionStatus(status=Status.SUCCEEDED) 101 102 def read( 103 self, 104 logger: logging.Logger, 105 config: Mapping[str, Any], 106 catalog: ConfiguredAirbyteCatalog, 107 state: Optional[List[AirbyteStateMessage]] = None, 108 ) -> Iterator[AirbyteMessage]: 109 """Implements the Read operation from the Airbyte Specification. See https://docs.airbyte.com/understanding-airbyte/airbyte-protocol/.""" 110 logger.info(f"Starting syncing {self.name}") 111 config, internal_config = split_config(config) 112 # TODO assert all streams exist in the connector 113 # get the streams once in case the connector needs to make any queries to generate them 114 stream_instances = {s.name: s for s in self.streams(config)} 115 state_manager = ConnectorStateManager(state=state) 116 self._stream_to_instance_map = stream_instances 117 118 stream_name_to_exception: MutableMapping[str, AirbyteTracedException] = {} 119 120 with create_timer(self.name) as timer: 121 for configured_stream in catalog.streams: 122 stream_instance = stream_instances.get(configured_stream.stream.name) 123 is_stream_exist = bool(stream_instance) 124 try: 125 # Used direct reference to `stream_instance` instead of `is_stream_exist` to avoid mypy type checking errors 126 if not stream_instance: 127 if not self.raise_exception_on_missing_stream: 128 yield stream_status_as_airbyte_message( 129 configured_stream.stream, AirbyteStreamStatus.INCOMPLETE 130 ) 131 continue 132 133 error_message = ( 134 f"The stream '{configured_stream.stream.name}' in your connection configuration was not found in the source. " 135 f"Refresh the schema in your replication settings and remove this stream from future sync attempts." 136 ) 137 138 # Use configured_stream as stream_instance to support references in error handling. 139 stream_instance = configured_stream.stream 140 141 raise AirbyteTracedException( 142 message="A stream listed in your configuration was not found in the source. Please check the logs for more " 143 "details.", 144 internal_message=error_message, 145 failure_type=FailureType.config_error, 146 ) 147 148 timer.start_event(f"Syncing stream {configured_stream.stream.name}") 149 logger.info(f"Marking stream {configured_stream.stream.name} as STARTED") 150 yield stream_status_as_airbyte_message( 151 configured_stream.stream, AirbyteStreamStatus.STARTED 152 ) 153 yield from self._read_stream( 154 logger=logger, 155 stream_instance=stream_instance, 156 configured_stream=configured_stream, 157 state_manager=state_manager, 158 internal_config=internal_config, 159 ) 160 logger.info(f"Marking stream {configured_stream.stream.name} as STOPPED") 161 yield stream_status_as_airbyte_message( 162 configured_stream.stream, AirbyteStreamStatus.COMPLETE 163 ) 164 165 except Exception as e: 166 yield from self._emit_queued_messages() 167 logger.exception( 168 f"Encountered an exception while reading stream {configured_stream.stream.name}" 169 ) 170 logger.info(f"Marking stream {configured_stream.stream.name} as STOPPED") 171 yield stream_status_as_airbyte_message( 172 configured_stream.stream, AirbyteStreamStatus.INCOMPLETE 173 ) 174 175 stream_descriptor = StreamDescriptor(name=configured_stream.stream.name) 176 177 if isinstance(e, AirbyteTracedException): 178 traced_exception = e 179 info_message = f"Stopping sync on error from stream {configured_stream.stream.name} because {self.name} does not support continuing syncs on error." 180 else: 181 traced_exception = self._serialize_exception( 182 stream_descriptor, e, stream_instance=stream_instance 183 ) 184 info_message = f"{self.name} does not support continuing syncs on error from stream {configured_stream.stream.name}" 185 186 yield traced_exception.as_sanitized_airbyte_message( 187 stream_descriptor=stream_descriptor 188 ) 189 stream_name_to_exception[stream_instance.name] = traced_exception # type: ignore # use configured_stream if stream_instance is None 190 if self.stop_sync_on_stream_failure: 191 logger.info(info_message) 192 break 193 finally: 194 # Finish read event only if the stream instance exists; 195 # otherwise, there's no need as it never started 196 if is_stream_exist: 197 timer.finish_event() 198 logger.info(f"Finished syncing {configured_stream.stream.name}") 199 logger.info(timer.report()) 200 201 if len(stream_name_to_exception) > 0: 202 error_message = generate_failed_streams_error_message( 203 {key: [value] for key, value in stream_name_to_exception.items()} 204 ) 205 logger.info(error_message) 206 # We still raise at least one exception when a stream raises an exception because the platform currently relies 207 # on a non-zero exit code to determine if a sync attempt has failed. We also raise the exception as a config_error 208 # type because this combined error isn't actionable, but rather the previously emitted individual errors. 209 raise AirbyteTracedException( 210 message=error_message, failure_type=FailureType.config_error 211 ) 212 logger.info(f"Finished syncing {self.name}") 213 214 @staticmethod 215 def _serialize_exception( 216 stream_descriptor: StreamDescriptor, e: Exception, stream_instance: Optional[Stream] = None 217 ) -> AirbyteTracedException: 218 display_message = stream_instance.get_error_display_message(e) if stream_instance else None 219 if display_message: 220 return AirbyteTracedException.from_exception( 221 e, message=display_message, stream_descriptor=stream_descriptor 222 ) 223 return AirbyteTracedException.from_exception(e, stream_descriptor=stream_descriptor) 224 225 @property 226 def raise_exception_on_missing_stream(self) -> bool: 227 return False 228 229 def _read_stream( 230 self, 231 logger: logging.Logger, 232 stream_instance: Stream, 233 configured_stream: ConfiguredAirbyteStream, 234 state_manager: ConnectorStateManager, 235 internal_config: InternalConfig, 236 ) -> Iterator[AirbyteMessage]: 237 if internal_config.page_size and isinstance(stream_instance, HttpStream): 238 logger.info( 239 f"Setting page size for {stream_instance.name} to {internal_config.page_size}" 240 ) 241 stream_instance.page_size = internal_config.page_size 242 logger.debug( 243 f"Syncing configured stream: {configured_stream.stream.name}", 244 extra={ 245 "sync_mode": configured_stream.sync_mode, 246 "primary_key": configured_stream.primary_key, 247 "cursor_field": configured_stream.cursor_field, 248 }, 249 ) 250 stream_instance.log_stream_sync_configuration() 251 252 stream_name = configured_stream.stream.name 253 stream_state = state_manager.get_stream_state(stream_name, stream_instance.namespace) 254 255 # This is a hack. Existing full refresh streams that are converted into resumable full refresh need to discard 256 # the state because the terminal state for a full refresh sync is not compatible with substream resumable full 257 # refresh state. This is only required when running live traffic regression testing since the platform normally 258 # handles whether to pass state 259 if stream_state == {"__ab_no_cursor_state_message": True}: 260 stream_state = {} 261 262 if "state" in dir(stream_instance): 263 stream_instance.state = stream_state # type: ignore # we check that state in the dir(stream_instance) 264 logger.info(f"Setting state of {self.name} stream to {stream_state}") 265 266 record_iterator = stream_instance.read( 267 configured_stream, 268 logger, 269 self._slice_logger, 270 stream_state, 271 state_manager, 272 internal_config, 273 ) 274 275 record_counter = 0 276 logger.info(f"Syncing stream: {stream_name} ") 277 for record_data_or_message in record_iterator: 278 record = self._get_message(record_data_or_message, stream_instance) 279 if record.type == MessageType.RECORD: 280 record_counter += 1 281 if record_counter == 1: 282 logger.info(f"Marking stream {stream_name} as RUNNING") 283 # If we just read the first record of the stream, emit the transition to the RUNNING state 284 yield stream_status_as_airbyte_message( 285 configured_stream.stream, AirbyteStreamStatus.RUNNING 286 ) 287 yield from self._emit_queued_messages() 288 yield record 289 290 logger.info(f"Read {record_counter} records from {stream_name} stream") 291 292 def _emit_queued_messages(self) -> Iterable[AirbyteMessage]: 293 if self.message_repository: 294 yield from self.message_repository.consume_queue() 295 return 296 297 def _get_message( 298 self, record_data_or_message: Union[StreamData, AirbyteMessage], stream: Stream 299 ) -> AirbyteMessage: 300 """ 301 Converts the input to an AirbyteMessage if it is a StreamData. Returns the input as is if it is already an AirbyteMessage 302 """ 303 match record_data_or_message: 304 case AirbyteMessage(): 305 return record_data_or_message 306 case _: 307 return stream_data_to_airbyte_message( 308 stream.name, 309 record_data_or_message, 310 stream.transformer, 311 stream.get_json_schema(), 312 ) 313 314 @property 315 def message_repository(self) -> Union[None, MessageRepository]: 316 return _default_message_repository 317 318 @property 319 def stop_sync_on_stream_failure(self) -> bool: 320 """ 321 WARNING: This function is in-development which means it is subject to change. Use at your own risk. 322 323 By default, when a source encounters an exception while syncing a stream, it will emit an error trace message and then 324 continue syncing the next stream. This can be overwritten on a per-source basis so that the source will stop the sync 325 on the first error seen and emit a single error trace message for that stream. 326 """ 327 return False
Abstract base class for an Airbyte Source. Consumers should implement any abstract methods in this class to create an Airbyte Specification compliant Source.
59 @abstractmethod 60 def check_connection( 61 self, logger: logging.Logger, config: Mapping[str, Any] 62 ) -> Tuple[bool, Optional[Any]]: 63 """ 64 :param logger: source logger 65 :param config: The user-provided configuration as specified by the source's spec. 66 This usually contains information required to check connection e.g. tokens, secrets and keys etc. 67 :return: A tuple of (boolean, error). If boolean is true, then the connection check is successful 68 and we can connect to the underlying data source using the provided configuration. 69 Otherwise, the input config cannot be used to connect to the underlying data source, 70 and the "error" object should describe what went wrong. 71 The error object will be cast to string to display the problem to the user. 72 """
Parameters
- logger: source logger
- config: The user-provided configuration as specified by the source's spec. This usually contains information required to check connection e.g. tokens, secrets and keys etc.
Returns
A tuple of (boolean, error). If boolean is true, then the connection check is successful and we can connect to the underlying data source using the provided configuration. Otherwise, the input config cannot be used to connect to the underlying data source, and the "error" object should describe what went wrong. The error object will be cast to string to display the problem to the user.
74 @abstractmethod 75 def streams(self, config: Mapping[str, Any]) -> List[Stream]: 76 """ 77 :param config: The user-provided configuration as specified by the source's spec. 78 Any stream construction related operation should happen here. 79 :return: A list of the streams in this source connector. 80 """
Parameters
- config: The user-provided configuration as specified by the source's spec. Any stream construction related operation should happen here.
Returns
A list of the streams in this source connector.
86 def discover(self, logger: logging.Logger, config: Mapping[str, Any]) -> AirbyteCatalog: 87 """Implements the Discover operation from the Airbyte Specification. 88 See https://docs.airbyte.com/understanding-airbyte/airbyte-protocol/#discover. 89 """ 90 streams = [stream.as_airbyte_stream() for stream in self.streams(config=config)] 91 return AirbyteCatalog(streams=streams)
Implements the Discover operation from the Airbyte Specification. See https://docs.airbyte.com/understanding-airbyte/airbyte-protocol/#discover.
93 def check(self, logger: logging.Logger, config: Mapping[str, Any]) -> AirbyteConnectionStatus: 94 """Implements the Check Connection operation from the Airbyte Specification. 95 See https://docs.airbyte.com/understanding-airbyte/airbyte-protocol/#check. 96 """ 97 check_succeeded, error = self.check_connection(logger, config) 98 if not check_succeeded: 99 return AirbyteConnectionStatus(status=Status.FAILED, message=repr(error)) 100 return AirbyteConnectionStatus(status=Status.SUCCEEDED)
Implements the Check Connection operation from the Airbyte Specification. See https://docs.airbyte.com/understanding-airbyte/airbyte-protocol/#check.
102 def read( 103 self, 104 logger: logging.Logger, 105 config: Mapping[str, Any], 106 catalog: ConfiguredAirbyteCatalog, 107 state: Optional[List[AirbyteStateMessage]] = None, 108 ) -> Iterator[AirbyteMessage]: 109 """Implements the Read operation from the Airbyte Specification. See https://docs.airbyte.com/understanding-airbyte/airbyte-protocol/.""" 110 logger.info(f"Starting syncing {self.name}") 111 config, internal_config = split_config(config) 112 # TODO assert all streams exist in the connector 113 # get the streams once in case the connector needs to make any queries to generate them 114 stream_instances = {s.name: s for s in self.streams(config)} 115 state_manager = ConnectorStateManager(state=state) 116 self._stream_to_instance_map = stream_instances 117 118 stream_name_to_exception: MutableMapping[str, AirbyteTracedException] = {} 119 120 with create_timer(self.name) as timer: 121 for configured_stream in catalog.streams: 122 stream_instance = stream_instances.get(configured_stream.stream.name) 123 is_stream_exist = bool(stream_instance) 124 try: 125 # Used direct reference to `stream_instance` instead of `is_stream_exist` to avoid mypy type checking errors 126 if not stream_instance: 127 if not self.raise_exception_on_missing_stream: 128 yield stream_status_as_airbyte_message( 129 configured_stream.stream, AirbyteStreamStatus.INCOMPLETE 130 ) 131 continue 132 133 error_message = ( 134 f"The stream '{configured_stream.stream.name}' in your connection configuration was not found in the source. " 135 f"Refresh the schema in your replication settings and remove this stream from future sync attempts." 136 ) 137 138 # Use configured_stream as stream_instance to support references in error handling. 139 stream_instance = configured_stream.stream 140 141 raise AirbyteTracedException( 142 message="A stream listed in your configuration was not found in the source. Please check the logs for more " 143 "details.", 144 internal_message=error_message, 145 failure_type=FailureType.config_error, 146 ) 147 148 timer.start_event(f"Syncing stream {configured_stream.stream.name}") 149 logger.info(f"Marking stream {configured_stream.stream.name} as STARTED") 150 yield stream_status_as_airbyte_message( 151 configured_stream.stream, AirbyteStreamStatus.STARTED 152 ) 153 yield from self._read_stream( 154 logger=logger, 155 stream_instance=stream_instance, 156 configured_stream=configured_stream, 157 state_manager=state_manager, 158 internal_config=internal_config, 159 ) 160 logger.info(f"Marking stream {configured_stream.stream.name} as STOPPED") 161 yield stream_status_as_airbyte_message( 162 configured_stream.stream, AirbyteStreamStatus.COMPLETE 163 ) 164 165 except Exception as e: 166 yield from self._emit_queued_messages() 167 logger.exception( 168 f"Encountered an exception while reading stream {configured_stream.stream.name}" 169 ) 170 logger.info(f"Marking stream {configured_stream.stream.name} as STOPPED") 171 yield stream_status_as_airbyte_message( 172 configured_stream.stream, AirbyteStreamStatus.INCOMPLETE 173 ) 174 175 stream_descriptor = StreamDescriptor(name=configured_stream.stream.name) 176 177 if isinstance(e, AirbyteTracedException): 178 traced_exception = e 179 info_message = f"Stopping sync on error from stream {configured_stream.stream.name} because {self.name} does not support continuing syncs on error." 180 else: 181 traced_exception = self._serialize_exception( 182 stream_descriptor, e, stream_instance=stream_instance 183 ) 184 info_message = f"{self.name} does not support continuing syncs on error from stream {configured_stream.stream.name}" 185 186 yield traced_exception.as_sanitized_airbyte_message( 187 stream_descriptor=stream_descriptor 188 ) 189 stream_name_to_exception[stream_instance.name] = traced_exception # type: ignore # use configured_stream if stream_instance is None 190 if self.stop_sync_on_stream_failure: 191 logger.info(info_message) 192 break 193 finally: 194 # Finish read event only if the stream instance exists; 195 # otherwise, there's no need as it never started 196 if is_stream_exist: 197 timer.finish_event() 198 logger.info(f"Finished syncing {configured_stream.stream.name}") 199 logger.info(timer.report()) 200 201 if len(stream_name_to_exception) > 0: 202 error_message = generate_failed_streams_error_message( 203 {key: [value] for key, value in stream_name_to_exception.items()} 204 ) 205 logger.info(error_message) 206 # We still raise at least one exception when a stream raises an exception because the platform currently relies 207 # on a non-zero exit code to determine if a sync attempt has failed. We also raise the exception as a config_error 208 # type because this combined error isn't actionable, but rather the previously emitted individual errors. 209 raise AirbyteTracedException( 210 message=error_message, failure_type=FailureType.config_error 211 ) 212 logger.info(f"Finished syncing {self.name}")
Implements the Read operation from the Airbyte Specification. See https://docs.airbyte.com/understanding-airbyte/airbyte-protocol/.
318 @property 319 def stop_sync_on_stream_failure(self) -> bool: 320 """ 321 WARNING: This function is in-development which means it is subject to change. Use at your own risk. 322 323 By default, when a source encounters an exception while syncing a stream, it will emit an error trace message and then 324 continue syncing the next stream. This can be overwritten on a per-source basis so that the source will stop the sync 325 on the first error seen and emit a single error trace message for that stream. 326 """ 327 return False
WARNING: This function is in-development which means it is subject to change. Use at your own risk.
By default, when a source encounters an exception while syncing a stream, it will emit an error trace message and then continue syncing the next stream. This can be overwritten on a per-source basis so that the source will stop the sync on the first error seen and emit a single error trace message for that stream.
13class BaseConfig(BaseModel): 14 """Base class for connector spec, adds the following behaviour: 15 16 - resolve $ref and replace it with definition 17 - replace all occurrences of anyOf with oneOf 18 - drop description 19 """ 20 21 @classmethod 22 def schema(cls, *args: Any, **kwargs: Any) -> Dict[str, Any]: 23 """We're overriding the schema classmethod to enable some post-processing""" 24 schema = super().schema(*args, **kwargs) 25 rename_key(schema, old_key="anyOf", new_key="oneOf") # UI supports only oneOf 26 expand_refs(schema) 27 schema.pop("description", None) # description added from the docstring 28 return schema
Base class for connector spec, adds the following behaviour:
- resolve $ref and replace it with definition
- replace all occurrences of anyOf with oneOf
- drop description
21 @classmethod 22 def schema(cls, *args: Any, **kwargs: Any) -> Dict[str, Any]: 23 """We're overriding the schema classmethod to enable some post-processing""" 24 schema = super().schema(*args, **kwargs) 25 rename_key(schema, old_key="anyOf", new_key="oneOf") # UI supports only oneOf 26 expand_refs(schema) 27 schema.pop("description", None) # description added from the docstring 28 return schema
We're overriding the schema classmethod to enable some post-processing
34class BaseConnector(ABC, Generic[TConfig]): 35 # configure whether the `check_config_against_spec_or_exit()` needs to be called 36 check_config_against_spec: bool = True 37 38 @abstractmethod 39 def configure(self, config: Mapping[str, Any], temp_dir: str) -> TConfig: 40 """ 41 Persist config in temporary directory to run the Source job 42 """ 43 44 @staticmethod 45 def read_config(config_path: str) -> MutableMapping[str, Any]: 46 config = BaseConnector._read_json_file(config_path) 47 if isinstance(config, MutableMapping): 48 return config 49 else: 50 raise ValueError( 51 f"The content of {config_path} is not an object and therefore is not a valid config. Please ensure the file represent a config." 52 ) 53 54 @staticmethod 55 def _read_json_file(file_path: str) -> Any: 56 with open(file_path, "r") as file: 57 contents = file.read() 58 59 try: 60 return json.loads(contents) 61 except json.JSONDecodeError as error: 62 raise ValueError( 63 f"Could not read json file {file_path}: {error}. Please ensure that it is a valid JSON." 64 ) 65 66 @staticmethod 67 def write_config(config: TConfig, config_path: str) -> None: 68 with open(config_path, "w") as fh: 69 fh.write(json.dumps(config)) 70 71 def spec(self, logger: logging.Logger) -> ConnectorSpecification: 72 """ 73 Returns the spec for this integration. The spec is a JSON-Schema object describing the required configurations (e.g: username and password) 74 required to run this integration. By default, this will be loaded from a "spec.yaml" or a "spec.json" in the package root. 75 """ 76 77 package = self.__class__.__module__.split(".")[0] 78 79 yaml_spec = load_optional_package_file(package, "spec.yaml") 80 json_spec = load_optional_package_file(package, "spec.json") 81 82 if yaml_spec and json_spec: 83 raise RuntimeError( 84 "Found multiple spec files in the package. Only one of spec.yaml or spec.json should be provided." 85 ) 86 87 if yaml_spec: 88 spec_obj = yaml.load(yaml_spec, Loader=yaml.SafeLoader) 89 elif json_spec: 90 try: 91 spec_obj = json.loads(json_spec) 92 except json.JSONDecodeError as error: 93 raise ValueError( 94 f"Could not read json spec file: {error}. Please ensure that it is a valid JSON." 95 ) 96 else: 97 raise FileNotFoundError("Unable to find spec.yaml or spec.json in the package.") 98 99 return ConnectorSpecificationSerializer.load(spec_obj) 100 101 @abstractmethod 102 def check(self, logger: logging.Logger, config: TConfig) -> AirbyteConnectionStatus: 103 """ 104 Tests if the input configuration can be used to successfully connect to the integration e.g: if a provided Stripe API token can be used to connect 105 to the Stripe API. 106 """
Helper class that provides a standard way to create an ABC using inheritance.
38 @abstractmethod 39 def configure(self, config: Mapping[str, Any], temp_dir: str) -> TConfig: 40 """ 41 Persist config in temporary directory to run the Source job 42 """
Persist config in temporary directory to run the Source job
44 @staticmethod 45 def read_config(config_path: str) -> MutableMapping[str, Any]: 46 config = BaseConnector._read_json_file(config_path) 47 if isinstance(config, MutableMapping): 48 return config 49 else: 50 raise ValueError( 51 f"The content of {config_path} is not an object and therefore is not a valid config. Please ensure the file represent a config." 52 )
71 def spec(self, logger: logging.Logger) -> ConnectorSpecification: 72 """ 73 Returns the spec for this integration. The spec is a JSON-Schema object describing the required configurations (e.g: username and password) 74 required to run this integration. By default, this will be loaded from a "spec.yaml" or a "spec.json" in the package root. 75 """ 76 77 package = self.__class__.__module__.split(".")[0] 78 79 yaml_spec = load_optional_package_file(package, "spec.yaml") 80 json_spec = load_optional_package_file(package, "spec.json") 81 82 if yaml_spec and json_spec: 83 raise RuntimeError( 84 "Found multiple spec files in the package. Only one of spec.yaml or spec.json should be provided." 85 ) 86 87 if yaml_spec: 88 spec_obj = yaml.load(yaml_spec, Loader=yaml.SafeLoader) 89 elif json_spec: 90 try: 91 spec_obj = json.loads(json_spec) 92 except json.JSONDecodeError as error: 93 raise ValueError( 94 f"Could not read json spec file: {error}. Please ensure that it is a valid JSON." 95 ) 96 else: 97 raise FileNotFoundError("Unable to find spec.yaml or spec.json in the package.") 98 99 return ConnectorSpecificationSerializer.load(spec_obj)
Returns the spec for this integration. The spec is a JSON-Schema object describing the required configurations (e.g: username and password) required to run this integration. By default, this will be loaded from a "spec.yaml" or a "spec.json" in the package root.
101 @abstractmethod 102 def check(self, logger: logging.Logger, config: TConfig) -> AirbyteConnectionStatus: 103 """ 104 Tests if the input configuration can be used to successfully connect to the integration e.g: if a provided Stripe API token can be used to connect 105 to the Stripe API. 106 """
Tests if the input configuration can be used to successfully connect to the integration e.g: if a provided Stripe API token can be used to connect to the Stripe API.
124class Connector(DefaultConnectorMixin, BaseConnector[Mapping[str, Any]], ABC): ...
Helper class that provides a standard way to create an ABC using inheritance.
31class Destination(Connector, ABC): 32 VALID_CMDS = {"spec", "check", "write"} 33 34 @abstractmethod 35 def write( 36 self, 37 config: Mapping[str, Any], 38 configured_catalog: ConfiguredAirbyteCatalog, 39 input_messages: Iterable[AirbyteMessage], 40 ) -> Iterable[AirbyteMessage]: 41 """Implement to define how the connector writes data to the destination""" 42 43 def _run_check(self, config: Mapping[str, Any]) -> AirbyteMessage: 44 check_result = self.check(logger, config) 45 return AirbyteMessage(type=Type.CONNECTION_STATUS, connectionStatus=check_result) 46 47 def _parse_input_stream(self, input_stream: io.TextIOWrapper) -> Iterable[AirbyteMessage]: 48 """Reads from stdin, converting to Airbyte messages""" 49 for line in input_stream: 50 try: 51 yield AirbyteMessageSerializer.load(orjson.loads(line)) 52 except orjson.JSONDecodeError: 53 logger.info( 54 f"ignoring input which can't be deserialized as Airbyte Message: {line}" 55 ) 56 57 def _run_write( 58 self, 59 config: Mapping[str, Any], 60 configured_catalog_path: str, 61 input_stream: io.TextIOWrapper, 62 ) -> Iterable[AirbyteMessage]: 63 catalog = ConfiguredAirbyteCatalogSerializer.load( 64 orjson.loads(open(configured_catalog_path).read()) 65 ) 66 input_messages = self._parse_input_stream(input_stream) 67 logger.info("Begin writing to the destination...") 68 yield from self.write( 69 config=config, configured_catalog=catalog, input_messages=input_messages 70 ) 71 logger.info("Writing complete.") 72 73 def parse_args(self, args: List[str]) -> argparse.Namespace: 74 """ 75 :param args: commandline arguments 76 :return: 77 """ 78 79 parent_parser = argparse.ArgumentParser(add_help=False) 80 main_parser = argparse.ArgumentParser() 81 subparsers = main_parser.add_subparsers(title="commands", dest="command") 82 83 # spec 84 subparsers.add_parser( 85 "spec", help="outputs the json configuration specification", parents=[parent_parser] 86 ) 87 88 # check 89 check_parser = subparsers.add_parser( 90 "check", help="checks the config can be used to connect", parents=[parent_parser] 91 ) 92 required_check_parser = check_parser.add_argument_group("required named arguments") 93 required_check_parser.add_argument( 94 "--config", type=str, required=True, help="path to the json configuration file" 95 ) 96 97 # write 98 write_parser = subparsers.add_parser( 99 "write", help="Writes data to the destination", parents=[parent_parser] 100 ) 101 write_required = write_parser.add_argument_group("required named arguments") 102 write_required.add_argument( 103 "--config", type=str, required=True, help="path to the JSON configuration file" 104 ) 105 write_required.add_argument( 106 "--catalog", type=str, required=True, help="path to the configured catalog JSON file" 107 ) 108 109 parsed_args = main_parser.parse_args(args) 110 cmd = parsed_args.command 111 if not cmd: 112 raise Exception("No command entered. ") 113 elif cmd not in ["spec", "check", "write"]: 114 # This is technically dead code since parse_args() would fail if this was the case 115 # But it's non-obvious enough to warrant placing it here anyways 116 raise Exception(f"Unknown command entered: {cmd}") 117 118 return parsed_args 119 120 def run_cmd(self, parsed_args: argparse.Namespace) -> Iterable[AirbyteMessage]: 121 cmd = parsed_args.command 122 if cmd not in self.VALID_CMDS: 123 raise Exception(f"Unrecognized command: {cmd}") 124 125 spec = self.spec(logger) 126 if cmd == "spec": 127 yield AirbyteMessage(type=Type.SPEC, spec=spec) 128 return 129 config = self.read_config(config_path=parsed_args.config) 130 # Register the config's secrets with the log filter before anything can print them, 131 # as the source entrypoint does. A config validation error quotes config values. 132 update_secrets(get_secrets(spec.connectionSpecification, config)) 133 if self.check_config_against_spec or cmd == "check": 134 try: 135 check_config_against_spec_or_exit(config, spec) 136 except AirbyteTracedException as traced_exc: 137 connection_status = traced_exc.as_connection_status_message() 138 if connection_status and cmd == "check": 139 yield connection_status 140 return 141 raise traced_exc 142 143 if cmd == "check": 144 yield self._run_check(config=config) 145 elif cmd == "write": 146 # Wrap in UTF-8 to override any other input encodings 147 wrapped_stdin = io.TextIOWrapper(sys.stdin.buffer, encoding="utf-8") 148 yield from self._run_write( 149 config=config, 150 configured_catalog_path=parsed_args.catalog, 151 input_stream=wrapped_stdin, 152 ) 153 154 def run(self, args: List[str]) -> None: 155 init_uncaught_exception_handler(logger) 156 parsed_args = self.parse_args(args) 157 output_messages = self.run_cmd(parsed_args) 158 for message in output_messages: 159 print(orjson.dumps(AirbyteMessageSerializer.dump(message)).decode())
Helper class that provides a standard way to create an ABC using inheritance.
34 @abstractmethod 35 def write( 36 self, 37 config: Mapping[str, Any], 38 configured_catalog: ConfiguredAirbyteCatalog, 39 input_messages: Iterable[AirbyteMessage], 40 ) -> Iterable[AirbyteMessage]: 41 """Implement to define how the connector writes data to the destination"""
Implement to define how the connector writes data to the destination
73 def parse_args(self, args: List[str]) -> argparse.Namespace: 74 """ 75 :param args: commandline arguments 76 :return: 77 """ 78 79 parent_parser = argparse.ArgumentParser(add_help=False) 80 main_parser = argparse.ArgumentParser() 81 subparsers = main_parser.add_subparsers(title="commands", dest="command") 82 83 # spec 84 subparsers.add_parser( 85 "spec", help="outputs the json configuration specification", parents=[parent_parser] 86 ) 87 88 # check 89 check_parser = subparsers.add_parser( 90 "check", help="checks the config can be used to connect", parents=[parent_parser] 91 ) 92 required_check_parser = check_parser.add_argument_group("required named arguments") 93 required_check_parser.add_argument( 94 "--config", type=str, required=True, help="path to the json configuration file" 95 ) 96 97 # write 98 write_parser = subparsers.add_parser( 99 "write", help="Writes data to the destination", parents=[parent_parser] 100 ) 101 write_required = write_parser.add_argument_group("required named arguments") 102 write_required.add_argument( 103 "--config", type=str, required=True, help="path to the JSON configuration file" 104 ) 105 write_required.add_argument( 106 "--catalog", type=str, required=True, help="path to the configured catalog JSON file" 107 ) 108 109 parsed_args = main_parser.parse_args(args) 110 cmd = parsed_args.command 111 if not cmd: 112 raise Exception("No command entered. ") 113 elif cmd not in ["spec", "check", "write"]: 114 # This is technically dead code since parse_args() would fail if this was the case 115 # But it's non-obvious enough to warrant placing it here anyways 116 raise Exception(f"Unknown command entered: {cmd}") 117 118 return parsed_args
Parameters
- args: commandline arguments
Returns
120 def run_cmd(self, parsed_args: argparse.Namespace) -> Iterable[AirbyteMessage]: 121 cmd = parsed_args.command 122 if cmd not in self.VALID_CMDS: 123 raise Exception(f"Unrecognized command: {cmd}") 124 125 spec = self.spec(logger) 126 if cmd == "spec": 127 yield AirbyteMessage(type=Type.SPEC, spec=spec) 128 return 129 config = self.read_config(config_path=parsed_args.config) 130 # Register the config's secrets with the log filter before anything can print them, 131 # as the source entrypoint does. A config validation error quotes config values. 132 update_secrets(get_secrets(spec.connectionSpecification, config)) 133 if self.check_config_against_spec or cmd == "check": 134 try: 135 check_config_against_spec_or_exit(config, spec) 136 except AirbyteTracedException as traced_exc: 137 connection_status = traced_exc.as_connection_status_message() 138 if connection_status and cmd == "check": 139 yield connection_status 140 return 141 raise traced_exc 142 143 if cmd == "check": 144 yield self._run_check(config=config) 145 elif cmd == "write": 146 # Wrap in UTF-8 to override any other input encodings 147 wrapped_stdin = io.TextIOWrapper(sys.stdin.buffer, encoding="utf-8") 148 yield from self._run_write( 149 config=config, 150 configured_catalog_path=parsed_args.catalog, 151 input_stream=wrapped_stdin, 152 )
56class Source( 57 DefaultConnectorMixin, 58 BaseSource[Mapping[str, Any], List[AirbyteStateMessage], ConfiguredAirbyteCatalog], 59 ABC, 60): 61 # can be overridden to change an input state. 62 @classmethod 63 def read_state(cls, state_path: str) -> List[AirbyteStateMessage]: 64 """ 65 Retrieves the input state of a sync by reading from the specified JSON file. Incoming state can be deserialized into either 66 a JSON object for legacy state input or as a list of AirbyteStateMessages for the per-stream state format. Regardless of the 67 incoming input type, it will always be transformed and output as a list of AirbyteStateMessage(s). 68 :param state_path: The filepath to where the stream states are located 69 :return: The complete stream state based on the connector's previous sync 70 """ 71 parsed_state_messages = [] 72 if state_path: 73 state_obj = BaseConnector._read_json_file(state_path) 74 if state_obj: 75 for state in state_obj: # type: ignore # `isinstance(state_obj, List)` ensures that this is a list 76 parsed_message = AirbyteStateMessageSerializer.load(state) 77 if ( 78 not parsed_message.stream 79 and not parsed_message.data 80 and not parsed_message.global_ 81 ): 82 raise ValueError( 83 "AirbyteStateMessage should contain either a stream, global, or state field" 84 ) 85 parsed_state_messages.append(parsed_message) 86 return parsed_state_messages 87 88 # can be overridden to change an input catalog 89 @classmethod 90 def read_catalog(cls, catalog_path: str) -> ConfiguredAirbyteCatalog: 91 return ConfiguredAirbyteCatalogSerializer.load(cls._read_json_file(catalog_path)) 92 93 @property 94 def name(self) -> str: 95 """Source name""" 96 return self.__class__.__name__
Helper class that provides a standard way to create an ABC using inheritance.
62 @classmethod 63 def read_state(cls, state_path: str) -> List[AirbyteStateMessage]: 64 """ 65 Retrieves the input state of a sync by reading from the specified JSON file. Incoming state can be deserialized into either 66 a JSON object for legacy state input or as a list of AirbyteStateMessages for the per-stream state format. Regardless of the 67 incoming input type, it will always be transformed and output as a list of AirbyteStateMessage(s). 68 :param state_path: The filepath to where the stream states are located 69 :return: The complete stream state based on the connector's previous sync 70 """ 71 parsed_state_messages = [] 72 if state_path: 73 state_obj = BaseConnector._read_json_file(state_path) 74 if state_obj: 75 for state in state_obj: # type: ignore # `isinstance(state_obj, List)` ensures that this is a list 76 parsed_message = AirbyteStateMessageSerializer.load(state) 77 if ( 78 not parsed_message.stream 79 and not parsed_message.data 80 and not parsed_message.global_ 81 ): 82 raise ValueError( 83 "AirbyteStateMessage should contain either a stream, global, or state field" 84 ) 85 parsed_state_messages.append(parsed_message) 86 return parsed_state_messages
Retrieves the input state of a sync by reading from the specified JSON file. Incoming state can be deserialized into either a JSON object for legacy state input or as a list of AirbyteStateMessages for the per-stream state format. Regardless of the incoming input type, it will always be transformed and output as a list of AirbyteStateMessage(s).
Parameters
- state_path: The filepath to where the stream states are located
Returns
The complete stream state based on the connector's previous sync
37@dataclass 38class AddFields(RecordTransformation): 39 """ 40 Transformation which adds field to an output record. The path of the added field can be nested. Adding nested fields will create all 41 necessary parent objects (like mkdir -p). Adding fields to an array will extend the array to that index (filling intermediate 42 indices with null values). So if you add a field at index 5 to the array ["value"], it will become ["value", null, null, null, null, 43 "new_value"]. 44 45 46 This transformation has access to the following contextual values: 47 record: the record about to be output by the connector 48 config: the input configuration provided to a connector 49 stream_state: the current state of the stream 50 stream_slice: the current stream slice being read 51 52 53 54 Examples of instantiating this transformation via YAML: 55 - type: AddFields 56 fields: 57 # hardcoded constant 58 - path: ["path"] 59 value: "static_value" 60 61 # nested path 62 - path: ["path", "to", "field"] 63 value: "static" 64 65 # from config 66 - path: ["shop_id"] 67 value: "{{ config.shop_id }}" 68 69 # from stream_interval 70 - path: ["date"] 71 value: "{{ stream_interval.start_date }}" 72 73 # from record 74 - path: ["unnested_value"] 75 value: {{ record.nested.field }} 76 77 # from stream_slice 78 - path: ["start_date"] 79 value: {{ stream_slice.start_date }} 80 81 # by supplying any valid Jinja template directive or expression https://jinja.palletsprojects.com/en/3.1.x/templates/# 82 - path: ["two_times_two"] 83 value: {{ 2 * 2 }} 84 85 Attributes: 86 fields (List[AddedFieldDefinition]): A list of transformations (path and corresponding value) that will be added to the record 87 """ 88 89 fields: List[AddedFieldDefinition] 90 parameters: InitVar[Mapping[str, Any]] 91 condition: str = "" 92 _parsed_fields: List[ParsedAddFieldDefinition] = field( 93 init=False, repr=False, default_factory=list 94 ) 95 96 def __post_init__(self, parameters: Mapping[str, Any]) -> None: 97 self._filter_interpolator = InterpolatedBoolean( 98 condition=self.condition, parameters=parameters 99 ) 100 101 for add_field in self.fields: 102 if len(add_field.path) < 1: 103 raise ValueError( 104 f"Expected a non-zero-length path for the AddFields transformation {add_field}" 105 ) 106 107 if not isinstance(add_field.value, InterpolatedString): 108 if not isinstance(add_field.value, str): 109 raise f"Expected a string value for the AddFields transformation: {add_field}" 110 else: 111 self._parsed_fields.append( 112 ParsedAddFieldDefinition( 113 add_field.path, 114 InterpolatedString.create(add_field.value, parameters=parameters), 115 value_type=add_field.value_type, 116 parameters=parameters, 117 ) 118 ) 119 else: 120 self._parsed_fields.append( 121 ParsedAddFieldDefinition( 122 add_field.path, 123 add_field.value, 124 value_type=add_field.value_type, 125 parameters={}, 126 ) 127 ) 128 129 def transform( 130 self, 131 record: Dict[str, Any], 132 config: Optional[Config] = None, 133 stream_state: Optional[StreamState] = None, 134 stream_slice: Optional[StreamSlice] = None, 135 ) -> None: 136 if config is None: 137 config = {} 138 kwargs = {"record": record, "stream_slice": stream_slice} 139 for parsed_field in self._parsed_fields: 140 valid_types = (parsed_field.value_type,) if parsed_field.value_type else None 141 value = parsed_field.value.eval(config, valid_types=valid_types, **kwargs) 142 is_empty_condition = not self.condition 143 if is_empty_condition or self._filter_interpolator.eval( 144 config, value=value, path=parsed_field.path, **kwargs 145 ): 146 dpath.new(record, parsed_field.path, value) 147 148 def __eq__(self, other: Any) -> bool: 149 return bool(self.__dict__ == other.__dict__)
Transformation which adds field to an output record. The path of the added field can be nested. Adding nested fields will create all necessary parent objects (like mkdir -p). Adding fields to an array will extend the array to that index (filling intermediate indices with null values). So if you add a field at index 5 to the array ["value"], it will become ["value", null, null, null, null, "new_value"].
This transformation has access to the following contextual values:
record: the record about to be output by the connector config: the input configuration provided to a connector stream_state: the current state of the stream stream_slice: the current stream slice being read
Examples of instantiating this transformation via YAML:
- type: AddFields
fields:
# hardcoded constant
- path: ["path"] value: "static_value"
# nested path
- path: ["path", "to", "field"]
value: "static"
# from config
- path: ["shop_id"]
value: "{{ config.shop_id }}"
# from stream_interval
- path: ["date"]
value: "{{ stream_interval.start_date }}"
# from record
- path: ["unnested_value"]
value: {{ record.nested.field }}
# from stream_slice
- path: ["start_date"]
value: {{ stream_slice.start_date }}
# by supplying any valid Jinja template directive or expression https://jinja.palletsprojects.com/en/3.1.x/templates/#
- path: ["two_times_two"]
value: {{ 2 * 2 }}
Attributes:
- fields (List[AddedFieldDefinition]): A list of transformations (path and corresponding value) that will be added to the record
129 def transform( 130 self, 131 record: Dict[str, Any], 132 config: Optional[Config] = None, 133 stream_state: Optional[StreamState] = None, 134 stream_slice: Optional[StreamSlice] = None, 135 ) -> None: 136 if config is None: 137 config = {} 138 kwargs = {"record": record, "stream_slice": stream_slice} 139 for parsed_field in self._parsed_fields: 140 valid_types = (parsed_field.value_type,) if parsed_field.value_type else None 141 value = parsed_field.value.eval(config, valid_types=valid_types, **kwargs) 142 is_empty_condition = not self.condition 143 if is_empty_condition or self._filter_interpolator.eval( 144 config, value=value, path=parsed_field.path, **kwargs 145 ): 146 dpath.new(record, parsed_field.path, value)
Transform a record by adding, deleting, or mutating fields directly from the record reference passed in argument.
Parameters
- record: The input record to be transformed
- config: The user-provided configuration as specified by the source's spec
- stream_state: The stream state
- stream_slice: The stream slice
Returns
The transformed record
17@dataclass(frozen=True) 18class AddedFieldDefinition: 19 """Defines the field to add on a record""" 20 21 path: FieldPointer 22 value: Union[InterpolatedString, str] 23 value_type: Optional[Type[Any]] 24 parameters: InitVar[Mapping[str, Any]]
Defines the field to add on a record
24@dataclass 25class ApiKeyAuthenticator(DeclarativeAuthenticator): 26 """ 27 ApiKeyAuth sets a request header on the HTTP requests sent. 28 29 The header is of the form: 30 `"<header>": "<token>"` 31 32 For example, 33 `ApiKeyAuthenticator("Authorization", "Bearer hello")` 34 will result in the following header set on the HTTP request 35 `"Authorization": "Bearer hello"` 36 37 Attributes: 38 request_option (RequestOption): request option how to inject the token into the request 39 token_provider (TokenProvider): Provider of the token 40 config (Config): The user-provided configuration as specified by the source's spec 41 parameters (Mapping[str, Any]): Additional runtime parameters to be used for string interpolation 42 """ 43 44 request_option: RequestOption 45 token_provider: TokenProvider 46 config: Config 47 parameters: InitVar[Mapping[str, Any]] 48 49 @property 50 def auth_header(self) -> str: 51 options = self._get_request_options(RequestOptionType.header) 52 return next(iter(options.keys()), "") 53 54 @property 55 def token(self) -> str: 56 return self.token_provider.get_token() 57 58 def _get_request_options(self, option_type: RequestOptionType) -> Mapping[str, Any]: 59 options: MutableMapping[str, Any] = {} 60 if self.request_option.inject_into == option_type: 61 self.request_option.inject_into_request(options, self.token, self.config) 62 return options 63 64 def get_request_params(self) -> Mapping[str, Any]: 65 return self._get_request_options(RequestOptionType.request_parameter) 66 67 def get_request_body_data(self) -> Union[Mapping[str, Any], str]: 68 return self._get_request_options(RequestOptionType.body_data) 69 70 def get_request_body_json(self) -> Mapping[str, Any]: 71 return self._get_request_options(RequestOptionType.body_json)
ApiKeyAuth sets a request header on the HTTP requests sent.
The header is of the form:
"<header>": "<token>"
For example,
ApiKeyAuthenticator("Authorization", "Bearer hello")
will result in the following header set on the HTTP request
"Authorization": "Bearer hello"
Attributes:
- request_option (RequestOption): request option how to inject the token into the request
- token_provider (TokenProvider): Provider of the token
- config (Config): The user-provided configuration as specified by the source's spec
- parameters (Mapping[str, Any]): Additional runtime parameters to be used for string interpolation
49 @property 50 def auth_header(self) -> str: 51 options = self._get_request_options(RequestOptionType.header) 52 return next(iter(options.keys()), "")
HTTP header to set on the requests
64 def get_request_params(self) -> Mapping[str, Any]: 65 return self._get_request_options(RequestOptionType.request_parameter)
HTTP request parameter to add to the requests
67 def get_request_body_data(self) -> Union[Mapping[str, Any], str]: 68 return self._get_request_options(RequestOptionType.body_data)
Form-encoded body data to set on the requests
70 def get_request_body_json(self) -> Mapping[str, Any]: 71 return self._get_request_options(RequestOptionType.body_json)
JSON-encoded body data to set on the requests
Inherited Members
12class BackoffStrategy(ABC): 13 @abstractmethod 14 def backoff_time( 15 self, 16 response_or_exception: Optional[Union[requests.Response, requests.RequestException]], 17 attempt_count: int, 18 ) -> Optional[float]: 19 """ 20 Override this method to dynamically determine backoff time e.g: by reading the X-Retry-After header. 21 22 Not called for every retryable response. `HttpClient` skips the strategies entirely when a 23 rate-limited response can be retried on another credential -- the authenticator says so via 24 `TokenRotatingAuthenticator.has_alternative_token` -- because the wait computed here is 25 derived from the credential that was rejected, and the retry will not use it. Implementations 26 must therefore not rely on being called for side effects such as counting attempts or 27 emitting metrics. 28 29 :param response_or_exception: The response or exception that caused the backoff. 30 :param attempt_count: The number of attempts already performed for this request. 31 :return how long to backoff in seconds. The return value may be a floating point number for subsecond precision. Returning None defers backoff 32 to the default backoff behavior (e.g using an exponential algorithm). 33 """ 34 pass
Helper class that provides a standard way to create an ABC using inheritance.
13 @abstractmethod 14 def backoff_time( 15 self, 16 response_or_exception: Optional[Union[requests.Response, requests.RequestException]], 17 attempt_count: int, 18 ) -> Optional[float]: 19 """ 20 Override this method to dynamically determine backoff time e.g: by reading the X-Retry-After header. 21 22 Not called for every retryable response. `HttpClient` skips the strategies entirely when a 23 rate-limited response can be retried on another credential -- the authenticator says so via 24 `TokenRotatingAuthenticator.has_alternative_token` -- because the wait computed here is 25 derived from the credential that was rejected, and the retry will not use it. Implementations 26 must therefore not rely on being called for side effects such as counting attempts or 27 emitting metrics. 28 29 :param response_or_exception: The response or exception that caused the backoff. 30 :param attempt_count: The number of attempts already performed for this request. 31 :return how long to backoff in seconds. The return value may be a floating point number for subsecond precision. Returning None defers backoff 32 to the default backoff behavior (e.g using an exponential algorithm). 33 """ 34 pass
Override this method to dynamically determine backoff time e.g: by reading the X-Retry-After header.
Not called for every retryable response. HttpClient skips the strategies entirely when a
rate-limited response can be retried on another credential -- the authenticator says so via
TokenRotatingAuthenticator.has_alternative_token -- because the wait computed here is
derived from the credential that was rejected, and the retry will not use it. Implementations
must therefore not rely on being called for side effects such as counting attempts or
emitting metrics.
Parameters
- response_or_exception: The response or exception that caused the backoff.
- attempt_count: The number of attempts already performed for this request. :return how long to backoff in seconds. The return value may be a floating point number for subsecond precision. Returning None defers backoff to the default backoff behavior (e.g using an exponential algorithm).
101@dataclass 102class BasicHttpAuthenticator(DeclarativeAuthenticator): 103 """ 104 Builds auth based off the basic authentication scheme as defined by RFC 7617, which transmits credentials as USER ID/password pairs, encoded using base64 105 https://developer.mozilla.org/en-US/docs/Web/HTTP/Authentication#basic_authentication_scheme 106 107 The header is of the form 108 `"Authorization": "Basic <encoded_credentials>"` 109 110 Attributes: 111 username (Union[InterpolatedString, str]): The username 112 config (Config): The user-provided configuration as specified by the source's spec 113 password (Union[InterpolatedString, str]): The password 114 parameters (Mapping[str, Any]): Additional runtime parameters to be used for string interpolation 115 """ 116 117 username: Union[InterpolatedString, str] 118 config: Config 119 parameters: InitVar[Mapping[str, Any]] 120 password: Union[InterpolatedString, str] = "" 121 122 def __post_init__(self, parameters: Mapping[str, Any]) -> None: 123 self._username = InterpolatedString.create(self.username, parameters=parameters) 124 self._password = InterpolatedString.create(self.password, parameters=parameters) 125 126 @property 127 def auth_header(self) -> str: 128 return "Authorization" 129 130 @property 131 def token(self) -> str: 132 auth_string = ( 133 f"{self._username.eval(self.config)}:{self._password.eval(self.config)}".encode("utf8") 134 ) 135 b64_encoded = base64.b64encode(auth_string).decode("utf8") 136 return f"Basic {b64_encoded}"
Builds auth based off the basic authentication scheme as defined by RFC 7617, which transmits credentials as USER ID/password pairs, encoded using base64 https://developer.mozilla.org/en-US/docs/Web/HTTP/Authentication#basic_authentication_scheme
The header is of the form
"Authorization": "Basic <encoded_credentials>"
Attributes:
- username (Union[InterpolatedString, str]): The username
- config (Config): The user-provided configuration as specified by the source's spec
- password (Union[InterpolatedString, str]): The password
- parameters (Mapping[str, Any]): Additional runtime parameters to be used for string interpolation
130 @property 131 def token(self) -> str: 132 auth_string = ( 133 f"{self._username.eval(self.config)}:{self._password.eval(self.config)}".encode("utf8") 134 ) 135 b64_encoded = base64.b64encode(auth_string).decode("utf8") 136 return f"Basic {b64_encoded}"
The header value to set on outgoing HTTP requests
74@dataclass 75class BearerAuthenticator(DeclarativeAuthenticator): 76 """ 77 Authenticator that sets the Authorization header on the HTTP requests sent. 78 79 The header is of the form: 80 `"Authorization": "Bearer <token>"` 81 82 Attributes: 83 token_provider (TokenProvider): Provider of the token 84 config (Config): The user-provided configuration as specified by the source's spec 85 parameters (Mapping[str, Any]): Additional runtime parameters to be used for string interpolation 86 """ 87 88 token_provider: TokenProvider 89 config: Config 90 parameters: InitVar[Mapping[str, Any]] 91 92 @property 93 def auth_header(self) -> str: 94 return "Authorization" 95 96 @property 97 def token(self) -> str: 98 return f"Bearer {self.token_provider.get_token()}"
Authenticator that sets the Authorization header on the HTTP requests sent.
The header is of the form:
"Authorization": "Bearer <token>"
Attributes:
- token_provider (TokenProvider): Provider of the token
- config (Config): The user-provided configuration as specified by the source's spec
- parameters (Mapping[str, Any]): Additional runtime parameters to be used for string interpolation
40@dataclass 41class CartesianProductStreamSlicer(PartitionRouter): 42 """ 43 Stream slicers that iterates over the cartesian product of input stream slicers 44 Given 2 stream slicers with the following slices: 45 A: [{"i": 0}, {"i": 1}, {"i": 2}] 46 B: [{"s": "hello"}, {"s": "world"}] 47 the resulting stream slices are 48 [ 49 {"i": 0, "s": "hello"}, 50 {"i": 0, "s": "world"}, 51 {"i": 1, "s": "hello"}, 52 {"i": 1, "s": "world"}, 53 {"i": 2, "s": "hello"}, 54 {"i": 2, "s": "world"}, 55 ] 56 57 Attributes: 58 stream_slicers (List[PartitionRouter]): Underlying stream slicers. The RequestOptions (e.g: Request headers, parameters, etc..) returned by this slicer are the combination of the RequestOptions of its input slicers. If there are conflicts e.g: two slicers define the same header or request param, the conflict is resolved by taking the value from the first slicer, where ordering is determined by the order in which slicers were input to this composite slicer. 59 """ 60 61 stream_slicers: List[PartitionRouter] 62 parameters: InitVar[Mapping[str, Any]] 63 64 def __post_init__(self, parameters: Mapping[str, Any]) -> None: 65 check_for_substream_in_slicers(self.stream_slicers, self.logger.warning) 66 67 def get_request_params( 68 self, 69 *, 70 stream_state: Optional[StreamState] = None, 71 stream_slice: Optional[StreamSlice] = None, 72 next_page_token: Optional[Mapping[str, Any]] = None, 73 ) -> Mapping[str, Any]: 74 return dict( 75 ChainMap( 76 *[ # type: ignore # ChainMap expects a MutableMapping[Never, Never] for reasons 77 s.get_request_params( 78 stream_state=stream_state, 79 stream_slice=stream_slice, 80 next_page_token=next_page_token, 81 ) 82 for s in self.stream_slicers 83 ] 84 ) 85 ) 86 87 def get_request_headers( 88 self, 89 *, 90 stream_state: Optional[StreamState] = None, 91 stream_slice: Optional[StreamSlice] = None, 92 next_page_token: Optional[Mapping[str, Any]] = None, 93 ) -> Mapping[str, Any]: 94 return dict( 95 ChainMap( 96 *[ # type: ignore # ChainMap expects a MutableMapping[Never, Never] for reasons 97 s.get_request_headers( 98 stream_state=stream_state, 99 stream_slice=stream_slice, 100 next_page_token=next_page_token, 101 ) 102 for s in self.stream_slicers 103 ] 104 ) 105 ) 106 107 def get_request_body_data( 108 self, 109 *, 110 stream_state: Optional[StreamState] = None, 111 stream_slice: Optional[StreamSlice] = None, 112 next_page_token: Optional[Mapping[str, Any]] = None, 113 ) -> Mapping[str, Any]: 114 return dict( 115 ChainMap( 116 *[ # type: ignore # ChainMap expects a MutableMapping[Never, Never] for reasons 117 s.get_request_body_data( 118 stream_state=stream_state, 119 stream_slice=stream_slice, 120 next_page_token=next_page_token, 121 ) 122 for s in self.stream_slicers 123 ] 124 ) 125 ) 126 127 def get_request_body_json( 128 self, 129 *, 130 stream_state: Optional[StreamState] = None, 131 stream_slice: Optional[StreamSlice] = None, 132 next_page_token: Optional[Mapping[str, Any]] = None, 133 ) -> Mapping[str, Any]: 134 return dict( 135 ChainMap( 136 *[ # type: ignore # ChainMap expects a MutableMapping[Never, Never] for reasons 137 s.get_request_body_json( 138 stream_state=stream_state, 139 stream_slice=stream_slice, 140 next_page_token=next_page_token, 141 ) 142 for s in self.stream_slicers 143 ] 144 ) 145 ) 146 147 def stream_slices(self) -> Iterable[StreamSlice]: 148 sub_slices = (s.stream_slices() for s in self.stream_slicers) 149 product = itertools.product(*sub_slices) 150 for stream_slice_tuple in product: 151 partition = dict(ChainMap(*[s.partition for s in stream_slice_tuple])) # type: ignore # ChainMap expects a MutableMapping[Never, Never] for reasons 152 cursor_slices = [s.cursor_slice for s in stream_slice_tuple if s.cursor_slice] 153 extra_fields = dict(ChainMap(*[s.extra_fields for s in stream_slice_tuple])) # type: ignore # ChainMap expects a MutableMapping[Never, Never] for reasons 154 if len(cursor_slices) > 1: 155 raise ValueError( 156 f"There should only be a single cursor slice. Found {cursor_slices}" 157 ) 158 if cursor_slices: 159 cursor_slice = cursor_slices[0] 160 else: 161 cursor_slice = {} 162 yield StreamSlice( 163 partition=partition, cursor_slice=cursor_slice, extra_fields=extra_fields 164 ) 165 166 def get_stream_state(self) -> Optional[Mapping[str, StreamState]]: 167 """ 168 Parent stream states are not supported for cartesian product stream slicer 169 """ 170 pass 171 172 @property 173 def logger(self) -> logging.Logger: 174 return logging.getLogger("airbyte.CartesianProductStreamSlicer")
Stream slicers that iterates over the cartesian product of input stream slicers Given 2 stream slicers with the following slices: A: [{"i": 0}, {"i": 1}, {"i": 2}] B: [{"s": "hello"}, {"s": "world"}] the resulting stream slices are [ {"i": 0, "s": "hello"}, {"i": 0, "s": "world"}, {"i": 1, "s": "hello"}, {"i": 1, "s": "world"}, {"i": 2, "s": "hello"}, {"i": 2, "s": "world"}, ]
Attributes:
- stream_slicers (List[PartitionRouter]): Underlying stream slicers. The RequestOptions (e.g: Request headers, parameters, etc..) returned by this slicer are the combination of the RequestOptions of its input slicers. If there are conflicts e.g: two slicers define the same header or request param, the conflict is resolved by taking the value from the first slicer, where ordering is determined by the order in which slicers were input to this composite slicer.
67 def get_request_params( 68 self, 69 *, 70 stream_state: Optional[StreamState] = None, 71 stream_slice: Optional[StreamSlice] = None, 72 next_page_token: Optional[Mapping[str, Any]] = None, 73 ) -> Mapping[str, Any]: 74 return dict( 75 ChainMap( 76 *[ # type: ignore # ChainMap expects a MutableMapping[Never, Never] for reasons 77 s.get_request_params( 78 stream_state=stream_state, 79 stream_slice=stream_slice, 80 next_page_token=next_page_token, 81 ) 82 for s in self.stream_slicers 83 ] 84 ) 85 )
Specifies the query parameters that should be set on an outgoing HTTP request given the inputs.
E.g: you might want to define query parameters for paging if next_page_token is not None.
87 def get_request_headers( 88 self, 89 *, 90 stream_state: Optional[StreamState] = None, 91 stream_slice: Optional[StreamSlice] = None, 92 next_page_token: Optional[Mapping[str, Any]] = None, 93 ) -> Mapping[str, Any]: 94 return dict( 95 ChainMap( 96 *[ # type: ignore # ChainMap expects a MutableMapping[Never, Never] for reasons 97 s.get_request_headers( 98 stream_state=stream_state, 99 stream_slice=stream_slice, 100 next_page_token=next_page_token, 101 ) 102 for s in self.stream_slicers 103 ] 104 ) 105 )
Return any non-auth headers. Authentication headers will overwrite any overlapping headers returned from this method.
107 def get_request_body_data( 108 self, 109 *, 110 stream_state: Optional[StreamState] = None, 111 stream_slice: Optional[StreamSlice] = None, 112 next_page_token: Optional[Mapping[str, Any]] = None, 113 ) -> Mapping[str, Any]: 114 return dict( 115 ChainMap( 116 *[ # type: ignore # ChainMap expects a MutableMapping[Never, Never] for reasons 117 s.get_request_body_data( 118 stream_state=stream_state, 119 stream_slice=stream_slice, 120 next_page_token=next_page_token, 121 ) 122 for s in self.stream_slicers 123 ] 124 ) 125 )
Specifies how to populate the body of the request with a non-JSON payload.
If returns a ready text that it will be sent as is. If returns a dict that it will be converted to a urlencoded form. E.g. {"key1": "value1", "key2": "value2"} => "key1=value1&key2=value2"
At the same time only one of the 'request_body_data' and 'request_body_json' functions can be overridden.
127 def get_request_body_json( 128 self, 129 *, 130 stream_state: Optional[StreamState] = None, 131 stream_slice: Optional[StreamSlice] = None, 132 next_page_token: Optional[Mapping[str, Any]] = None, 133 ) -> Mapping[str, Any]: 134 return dict( 135 ChainMap( 136 *[ # type: ignore # ChainMap expects a MutableMapping[Never, Never] for reasons 137 s.get_request_body_json( 138 stream_state=stream_state, 139 stream_slice=stream_slice, 140 next_page_token=next_page_token, 141 ) 142 for s in self.stream_slicers 143 ] 144 ) 145 )
Specifies how to populate the body of the request with a JSON payload.
At the same time only one of the 'request_body_data' and 'request_body_json' functions can be overridden.
147 def stream_slices(self) -> Iterable[StreamSlice]: 148 sub_slices = (s.stream_slices() for s in self.stream_slicers) 149 product = itertools.product(*sub_slices) 150 for stream_slice_tuple in product: 151 partition = dict(ChainMap(*[s.partition for s in stream_slice_tuple])) # type: ignore # ChainMap expects a MutableMapping[Never, Never] for reasons 152 cursor_slices = [s.cursor_slice for s in stream_slice_tuple if s.cursor_slice] 153 extra_fields = dict(ChainMap(*[s.extra_fields for s in stream_slice_tuple])) # type: ignore # ChainMap expects a MutableMapping[Never, Never] for reasons 154 if len(cursor_slices) > 1: 155 raise ValueError( 156 f"There should only be a single cursor slice. Found {cursor_slices}" 157 ) 158 if cursor_slices: 159 cursor_slice = cursor_slices[0] 160 else: 161 cursor_slice = {} 162 yield StreamSlice( 163 partition=partition, cursor_slice=cursor_slice, extra_fields=extra_fields 164 )
Defines stream slices
Returns
An iterable of stream slices
166 def get_stream_state(self) -> Optional[Mapping[str, StreamState]]: 167 """ 168 Parent stream states are not supported for cartesian product stream slicer 169 """ 170 pass
Parent stream states are not supported for cartesian product stream slicer
19class CombineMode(Enum): 20 """ 21 How a `CombinedExtractor` combines the output of its sub-extractors. 22 """ 23 24 union = "union" 25 first_match = "first_match"
How a CombinedExtractor combines the output of its sub-extractors.
46@dataclass 47class CombinedExtractor(RecordExtractor): 48 """Combines the output of several record extractors into a single stream of records. 49 50 A single `DpathExtractor` can only describe one path into the response. Several connectors need 51 more than that and reach for a custom `components.py` today, which keeps them out of the 52 Connector Builder. This component covers the two shapes those connectors implement: 53 54 - `union` (default) — yield every record of every sub-extractor, in sub-extractor order. Use it 55 when one response carries records under several paths, for example a GraphQL document that 56 returns `issues.nodes` and `pullRequests.nodes` side by side. 57 - `first_match` — yield the records of the first sub-extractor that produces at least one 58 record, and skip the remaining sub-extractors. If no sub-extractor produces a record, nothing 59 is yielded. Use it when an API answers with one of several alternative shapes. The winning 60 sub-extractor is only peeked at, never restarted: the peeked record is chained back in front 61 of the remaining ones, so large responses are still streamed lazily and no sub-extractor is 62 ever materialized into a list. 63 64 Examples of instantiating this component: 65 ``` 66 extractor: 67 type: CombinedExtractor 68 mode: union 69 extractors: 70 - type: DpathExtractor 71 field_path: ["data", "repository", "issues", "nodes"] 72 - type: DpathExtractor 73 field_path: ["data", "repository", "pullRequests", "nodes"] 74 ``` 75 76 ``` 77 extractor: 78 type: CombinedExtractor 79 mode: first_match 80 extractors: 81 - type: DpathExtractor 82 field_path: ["data", "boards", "*", "items_page", "items", "*"] 83 - type: DpathExtractor 84 field_path: ["data", "next_items_page", "items"] 85 ``` 86 87 ## Streaming decoders are rejected 88 89 Every sub-extractor is handed the same `requests.Response`, so the response body must be 90 readable more than once. That holds for the decoders that buffer the whole body, which is the 91 common case: `JsonDecoder` and any `CompositeRawDecoder` built with `stream_response=False` 92 read `response.content`, `XmlDecoder` reads `response.text` and `ZipfileDecoder` reads 93 `response.content` — `requests` caches all of those, so every sub-extractor sees the full body. 94 95 It does NOT hold for streaming decoders, i.e. a `CompositeRawDecoder` with 96 `stream_response=True`, which is what `CsvDecoder`, `JsonlDecoder`, `JsonItemsDecoder` and 97 `GzipDecoder` resolve to outside the Connector Builder, nor for `IterableDecoder`. Those read 98 `response.raw` and close it afterwards, so the first sub-extractor drains the stream and the 99 later ones read a dead body. That loss is SILENT rather than loud: `requests` puts a 100 `urllib3.HTTPResponse` in `response.raw`, and reading a closed `urllib3.HTTPResponse` returns 101 an empty body instead of raising, so `union` emits only the first sub-extractor's records and 102 `first_match` emits nothing at all when the first path misses. 103 104 Because of that, `ModelToComponentFactory.create_combined_extractor` refuses to build this 105 component over a streaming decoder and raises a configuration error, naming the decoder in the 106 internal message. The Connector Builder forces those same decoders to `stream_response=False`, 107 so the rejection is applied there too rather than letting a manifest test-read correctly and 108 lose records once published. 109 110 ## Empty records 111 112 `skip_empty_records` drops falsy records — `None`, `{}`, `[]`, `""` — before any of the modes 113 sees them. It is off by default, so a sub-extractor's output is passed through verbatim. 114 115 Turn it on when an API can answer with nulls in the middle of a record list. A GraphQL API 116 that returns a partial response puts `null` in the `data` array for the fields it failed to 117 resolve and describes the failure in a sibling `errors` field; without this flag those nulls 118 reach the record stream and fail schema validation downstream. 119 120 Under `first_match` the flag also changes which sub-extractor wins: a sub-extractor whose 121 records are all empty no longer counts as a match, so the next sub-extractor is tried. That is 122 the behaviour of `source-monday`'s `MondayIncrementalItemsExtractor`, where a page of nulls 123 must fall through to the pagination path rather than lock in the primary path. 124 125 The dropped records still count toward the size of the page, see "Record-counting paginators" 126 below. 127 128 ## Cost 129 130 Each sub-extractor decodes the response independently — `CompositeRawDecoder` re-parses the 131 cached body on every `decode()` call — so a response is parsed once per sub-extractor. 132 Combining three extractors over a large response costs roughly three times the decode time of 133 a single one. 134 135 ## Record-counting paginators 136 137 The count `OffsetIncrement` and `PageIncrement` obtain is the combined count, not the API's 138 page size: 139 140 - `union` returns the SUM over all sub-extractors. An `OffsetIncrement` advances the offset by 141 that sum, so the next page starts past the records that were never read — two sub-extractors 142 returning two records each with `page_size: 2` request offsets 0, 4, 8 instead of 0, 2, 4 and 143 silently drop two thirds of the records. `ModelToComponentFactory.create_offset_increment` 144 therefore rejects a `union` `CombinedExtractor` (nested ones included) with a configuration 145 error. A `PageIncrement` only compares the count against `page_size` to decide whether to 146 stop, so it loses nothing, but it issues one extra request whenever the summed count of the 147 last page happens to equal `page_size`. 148 - `first_match` returns the count of the winning sub-extractor, which does not inflate the 149 count, so it works with either paginator as long as the path the count comes from is the one 150 the API paginates over. 151 152 Empty records dropped by `skip_empty_records` are part of that count. The API returned them as 153 slots of the page, so a full page holding a null must still read as full: counting only the 154 surviving records would make both paginators stop at the first page with a null, and would 155 make an `OffsetIncrement` re-request records it already read. The factory sets 156 `count_dropped_empty_records` on the extractor of a `RecordSelector`, which makes it yield a 157 placeholder for every empty record it drops; the `RecordSelector` counts the placeholders and 158 discards them. The winner under `first_match` is still chosen by the surviving records alone, 159 so the placeholders never change which records are emitted. 160 161 Attributes: 162 extractors (List[RecordExtractor]): The sub-extractors to combine. At least one is required. 163 mode (CombineMode): How the sub-extractor outputs are combined. Defaults to `union`. 164 skip_empty_records (bool): Whether falsy records are dropped before they are combined. 165 Defaults to `False`. 166 count_dropped_empty_records (bool): Set by the factory on the extractor of a 167 `RecordSelector`, which discards the placeholders: yield a placeholder in place of 168 every dropped empty record so it counts toward the page size. Defaults to `False`. 169 """ 170 171 extractors: List[RecordExtractor] 172 parameters: InitVar[Mapping[str, Any]] 173 mode: CombineMode = CombineMode.union 174 skip_empty_records: bool = False 175 count_dropped_empty_records: bool = False 176 177 def __post_init__(self, parameters: Mapping[str, Any]) -> None: 178 if not self.extractors: 179 raise ValueError( 180 "CombinedExtractor requires at least one extractor in its `extractors` field." 181 ) 182 if not isinstance(self.mode, CombineMode): 183 self.mode = CombineMode(self.mode) 184 185 def extract_records(self, response: requests.Response) -> Iterable[Mapping[str, Any]]: 186 if self.mode == CombineMode.first_match: 187 yield from self._extract_first_match(response) 188 else: 189 yield from self._extract_union(response) 190 191 def _extract_union(self, response: requests.Response) -> Iterable[Mapping[str, Any]]: 192 for index in range(len(self.extractors)): 193 yield from self._records_of(index, response) 194 195 def _extract_first_match(self, response: requests.Response) -> Iterable[Mapping[str, Any]]: 196 for index in range(len(self.extractors)): 197 records: Iterator[Mapping[str, Any]] = iter(self._records_of(index, response)) 198 # Placeholders for dropped empty records do not make a sub-extractor win, but they are 199 # kept so the winner's count still includes the ones that came before its first record. 200 dropped_placeholders: List[Mapping[str, Any]] = [] 201 first_record: Union[Mapping[str, Any], _NoRecord] = _NO_RECORD 202 for record in records: 203 if isinstance(record, _DroppedEmptyRecord): 204 dropped_placeholders.append(record) 205 continue 206 first_record = record 207 break 208 if isinstance(first_record, _NoRecord): 209 continue 210 # Chain the peeked record back instead of restarting the extractor: the rest of the 211 # records stay lazy and the response is never read twice. 212 yield from chain(dropped_placeholders, [first_record], records) 213 return 214 215 def _records_of( 216 self, extractor_index: int, response: requests.Response 217 ) -> Iterable[Mapping[str, Any]]: 218 records = self.extractors[extractor_index].extract_records(response) 219 if not self.skip_empty_records: 220 return records 221 return self._without_empty_records(records, extractor_index) 222 223 def _without_empty_records( 224 self, records: Iterable[Mapping[str, Any]], extractor_index: int 225 ) -> Iterator[Mapping[str, Any]]: 226 dropped = 0 227 for record in records: 228 if not record: 229 dropped += 1 230 if self.count_dropped_empty_records: 231 yield _DROPPED_EMPTY_RECORD # type: ignore[misc] # only ever counted, never emitted 232 continue 233 yield record 234 if dropped: 235 logger.warning( 236 "CombinedExtractor dropped %s empty record(s) yielded by sub-extractor %s (%s) " 237 "because `skip_empty_records` is enabled. An API that returns empty records " 238 "alongside real ones usually reports the reason in an error field of the " 239 "response body.", 240 dropped, 241 extractor_index, 242 type(self.extractors[extractor_index]).__name__, 243 )
Combines the output of several record extractors into a single stream of records.
A single DpathExtractor can only describe one path into the response. Several connectors need
more than that and reach for a custom components.py today, which keeps them out of the
Connector Builder. This component covers the two shapes those connectors implement:
union(default) — yield every record of every sub-extractor, in sub-extractor order. Use it when one response carries records under several paths, for example a GraphQL document that returnsissues.nodesandpullRequests.nodesside by side.first_match— yield the records of the first sub-extractor that produces at least one record, and skip the remaining sub-extractors. If no sub-extractor produces a record, nothing is yielded. Use it when an API answers with one of several alternative shapes. The winning sub-extractor is only peeked at, never restarted: the peeked record is chained back in front of the remaining ones, so large responses are still streamed lazily and no sub-extractor is ever materialized into a list.
Examples of instantiating this component:
extractor:
type: CombinedExtractor
mode: union
extractors:
- type: DpathExtractor
field_path: ["data", "repository", "issues", "nodes"]
- type: DpathExtractor
field_path: ["data", "repository", "pullRequests", "nodes"]
extractor:
type: CombinedExtractor
mode: first_match
extractors:
- type: DpathExtractor
field_path: ["data", "boards", "*", "items_page", "items", "*"]
- type: DpathExtractor
field_path: ["data", "next_items_page", "items"]
Streaming decoders are rejected
Every sub-extractor is handed the same requests.Response, so the response body must be
readable more than once. That holds for the decoders that buffer the whole body, which is the
common case: JsonDecoder and any CompositeRawDecoder built with stream_response=False
read response.content, XmlDecoder reads response.text and ZipfileDecoder reads
response.content — requests caches all of those, so every sub-extractor sees the full body.
It does NOT hold for streaming decoders, i.e. a CompositeRawDecoder with
stream_response=True, which is what CsvDecoder, JsonlDecoder, JsonItemsDecoder and
GzipDecoder resolve to outside the Connector Builder, nor for IterableDecoder. Those read
response.raw and close it afterwards, so the first sub-extractor drains the stream and the
later ones read a dead body. That loss is SILENT rather than loud: requests puts a
urllib3.HTTPResponse in response.raw, and reading a closed urllib3.HTTPResponse returns
an empty body instead of raising, so union emits only the first sub-extractor's records and
first_match emits nothing at all when the first path misses.
Because of that, ModelToComponentFactory.create_combined_extractor refuses to build this
component over a streaming decoder and raises a configuration error, naming the decoder in the
internal message. The Connector Builder forces those same decoders to stream_response=False,
so the rejection is applied there too rather than letting a manifest test-read correctly and
lose records once published.
Empty records
skip_empty_records drops falsy records — None, {}, [], "" — before any of the modes
sees them. It is off by default, so a sub-extractor's output is passed through verbatim.
Turn it on when an API can answer with nulls in the middle of a record list. A GraphQL API
that returns a partial response puts null in the data array for the fields it failed to
resolve and describes the failure in a sibling errors field; without this flag those nulls
reach the record stream and fail schema validation downstream.
Under first_match the flag also changes which sub-extractor wins: a sub-extractor whose
records are all empty no longer counts as a match, so the next sub-extractor is tried. That is
the behaviour of source-monday's MondayIncrementalItemsExtractor, where a page of nulls
must fall through to the pagination path rather than lock in the primary path.
The dropped records still count toward the size of the page, see "Record-counting paginators" below.
Cost
Each sub-extractor decodes the response independently — CompositeRawDecoder re-parses the
cached body on every decode() call — so a response is parsed once per sub-extractor.
Combining three extractors over a large response costs roughly three times the decode time of
a single one.
Record-counting paginators
The count OffsetIncrement and PageIncrement obtain is the combined count, not the API's
page size:
unionreturns the SUM over all sub-extractors. AnOffsetIncrementadvances the offset by that sum, so the next page starts past the records that were never read — two sub-extractors returning two records each withpage_size: 2request offsets 0, 4, 8 instead of 0, 2, 4 and silently drop two thirds of the records.ModelToComponentFactory.create_offset_incrementtherefore rejects aunionCombinedExtractor(nested ones included) with a configuration error. APageIncrementonly compares the count againstpage_sizeto decide whether to stop, so it loses nothing, but it issues one extra request whenever the summed count of the last page happens to equalpage_size.first_matchreturns the count of the winning sub-extractor, which does not inflate the count, so it works with either paginator as long as the path the count comes from is the one the API paginates over.
Empty records dropped by skip_empty_records are part of that count. The API returned them as
slots of the page, so a full page holding a null must still read as full: counting only the
surviving records would make both paginators stop at the first page with a null, and would
make an OffsetIncrement re-request records it already read. The factory sets
count_dropped_empty_records on the extractor of a RecordSelector, which makes it yield a
placeholder for every empty record it drops; the RecordSelector counts the placeholders and
discards them. The winner under first_match is still chosen by the surviving records alone,
so the placeholders never change which records are emitted.
Attributes:
- extractors (List[RecordExtractor]): The sub-extractors to combine. At least one is required.
- mode (CombineMode): How the sub-extractor outputs are combined. Defaults to
union. - skip_empty_records (bool): Whether falsy records are dropped before they are combined.
Defaults to
False. - count_dropped_empty_records (bool): Set by the factory on the extractor of a
RecordSelector, which discards the placeholders: yield a placeholder in place of every dropped empty record so it counts toward the page size. Defaults toFalse.
185 def extract_records(self, response: requests.Response) -> Iterable[Mapping[str, Any]]: 186 if self.mode == CombineMode.first_match: 187 yield from self._extract_first_match(response) 188 else: 189 yield from self._extract_union(response)
Selects records from the response
Parameters
- response: The response to extract the records from
Returns
List of Records extracted from the response
28@dataclass 29class CursorPaginationStrategy(PaginationStrategy): 30 """ 31 Pagination strategy that evaluates an interpolated string to define the next page token. 32 33 Attributes: 34 page_size (Optional[Union[str, int]]): the number of records to request 35 cursor_value (Union[InterpolatedString, str]): template string evaluating to the cursor value 36 config (Config): connection config 37 stop_condition (Optional[InterpolatedBoolean]): template string evaluating when to stop paginating 38 decoder (Decoder): decoder to decode the response 39 """ 40 41 cursor_value: Union[InterpolatedString, str] 42 config: Config 43 parameters: InitVar[Mapping[str, Any]] 44 page_size: Optional[Union[str, int]] = None 45 stop_condition: Optional[Union[InterpolatedBoolean, str]] = None 46 decoder: Decoder = field( 47 default_factory=lambda: PaginationDecoderDecorator(decoder=JsonDecoder(parameters={})) 48 ) 49 50 def __post_init__(self, parameters: Mapping[str, Any]) -> None: 51 if isinstance(self.cursor_value, str): 52 self._cursor_value = InterpolatedString.create(self.cursor_value, parameters=parameters) 53 else: 54 self._cursor_value = self.cursor_value 55 if isinstance(self.stop_condition, str): 56 self._stop_condition: Optional[InterpolatedBoolean] = InterpolatedBoolean( 57 condition=self.stop_condition, parameters=parameters 58 ) 59 else: 60 self._stop_condition = self.stop_condition 61 62 if isinstance(self.page_size, int) or (self.page_size is None): 63 self._page_size = self.page_size 64 else: 65 page_size = InterpolatedString(self.page_size, parameters=parameters).eval(self.config) 66 if not isinstance(page_size, int): 67 raise Exception(f"{page_size} is of type {type(page_size)}. Expected {int}") 68 self._page_size = page_size 69 70 if self._page_size is None: 71 self._reject_unbound_page_size_variable() 72 73 def _reject_unbound_page_size_variable(self) -> None: 74 """ 75 Fail construction when an expression reads `page_size` while the strategy declares none. 76 77 The variable holds the page size that was actually requested, so it is the one safe thing to compare 78 `last_page_size` against while a `page_size_reduction` is shrinking the page. Without a declared 79 `page_size` there is nothing to bind it to, and an unbound comparison does not raise: Jinja fails, the 80 interpolation falls back to the raw template string, and a non-empty string is truthy. A 81 `stop_condition` written that way stops after the first page and drops the rest of the partition 82 without failing, which is why this is a construction error rather than a warning. 83 """ 84 for field_name, template in ( 85 ("stop_condition", self.stop_condition), 86 ("cursor_value", self.cursor_value), 87 ): 88 expression = template.string if isinstance(template, InterpolatedString) else template 89 if isinstance(template, InterpolatedBoolean): 90 expression = template.condition 91 if not isinstance(expression, str) or not references_requested_page_size(expression): 92 continue 93 raise ValueError( 94 f"The `{field_name}` {expression!r} reads the `{REQUESTED_PAGE_SIZE_VARIABLE}` interpolation " 95 f"variable, but the CursorPagination strategy it belongs to declares no `page_size`, so there " 96 f"is nothing to bind it to. The comparison would not fail either - it renders as the template " 97 f"string itself, which is truthy - so the pagination would end after the first page and the " 98 f"rest of the partition would be dropped silently. Declare `page_size` on the pagination " 99 f"strategy, or compare against a value the manifest defines, such as " 100 f"`config['page_size']`." 101 ) 102 103 @property 104 def initial_token(self) -> Optional[Any]: 105 """ 106 CursorPaginationStrategy does not have an initial value because the next cursor is typically included 107 in the response of the first request. For Resumable Full Refresh streams that checkpoint the page 108 cursor, the next cursor should be read from the state or stream slice object. 109 """ 110 return None 111 112 def next_page_token( 113 self, 114 response: requests.Response, 115 last_page_size: int, 116 last_record: Optional[Record], 117 last_page_token_value: Optional[Any] = None, 118 page_size_override: Optional[int] = None, 119 stream_slice: Optional[StreamSlice] = None, 120 ) -> Optional[Any]: 121 # The next page is a cursor read from the response, so `page_size_override` does not change how the token 122 # is computed. It is still exposed to the interpolation context as `page_size` because a `stop_condition` 123 # comparing `last_page_size` to a hardcoded page size would read a full reduced page as a short page and 124 # end the pagination early, silently dropping the rest of the partition. Writing the condition as 125 # `{{ last_page_size < page_size }}` keeps it correct while a reduction is in effect. 126 requested_page_size = ( 127 page_size_override if page_size_override is not None else self._page_size 128 ) 129 decoded_response = next(self.decoder.decode(response)) 130 # The default way that link is presented in requests.Response is a string of various links (last, next, etc). This 131 # is not indexable or useful for parsing the cursor, so we replace it with the link dictionary from response.links 132 headers: Dict[str, Any] = dict(response.headers) 133 headers["link"] = response.links 134 if self._stop_condition: 135 should_stop = self._stop_condition.eval( 136 self.config, 137 response=decoded_response, 138 headers=headers, 139 last_record=last_record, 140 last_page_size=last_page_size, 141 page_size=requested_page_size, 142 stream_slice=stream_slice, 143 ) 144 if should_stop: 145 return None 146 token = self._cursor_value.eval( 147 config=self.config, 148 response=decoded_response, 149 headers=headers, 150 last_record=last_record, 151 last_page_size=last_page_size, 152 page_size=requested_page_size, 153 stream_slice=stream_slice, 154 ) 155 return token if token else None 156 157 def get_page_size(self) -> Optional[int]: 158 return self._page_size
Pagination strategy that evaluates an interpolated string to define the next page token.
Attributes:
- page_size (Optional[Union[str, int]]): the number of records to request
- cursor_value (Union[InterpolatedString, str]): template string evaluating to the cursor value
- config (Config): connection config
- stop_condition (Optional[InterpolatedBoolean]): template string evaluating when to stop paginating
- decoder (Decoder): decoder to decode the response
103 @property 104 def initial_token(self) -> Optional[Any]: 105 """ 106 CursorPaginationStrategy does not have an initial value because the next cursor is typically included 107 in the response of the first request. For Resumable Full Refresh streams that checkpoint the page 108 cursor, the next cursor should be read from the state or stream slice object. 109 """ 110 return None
CursorPaginationStrategy does not have an initial value because the next cursor is typically included in the response of the first request. For Resumable Full Refresh streams that checkpoint the page cursor, the next cursor should be read from the state or stream slice object.
112 def next_page_token( 113 self, 114 response: requests.Response, 115 last_page_size: int, 116 last_record: Optional[Record], 117 last_page_token_value: Optional[Any] = None, 118 page_size_override: Optional[int] = None, 119 stream_slice: Optional[StreamSlice] = None, 120 ) -> Optional[Any]: 121 # The next page is a cursor read from the response, so `page_size_override` does not change how the token 122 # is computed. It is still exposed to the interpolation context as `page_size` because a `stop_condition` 123 # comparing `last_page_size` to a hardcoded page size would read a full reduced page as a short page and 124 # end the pagination early, silently dropping the rest of the partition. Writing the condition as 125 # `{{ last_page_size < page_size }}` keeps it correct while a reduction is in effect. 126 requested_page_size = ( 127 page_size_override if page_size_override is not None else self._page_size 128 ) 129 decoded_response = next(self.decoder.decode(response)) 130 # The default way that link is presented in requests.Response is a string of various links (last, next, etc). This 131 # is not indexable or useful for parsing the cursor, so we replace it with the link dictionary from response.links 132 headers: Dict[str, Any] = dict(response.headers) 133 headers["link"] = response.links 134 if self._stop_condition: 135 should_stop = self._stop_condition.eval( 136 self.config, 137 response=decoded_response, 138 headers=headers, 139 last_record=last_record, 140 last_page_size=last_page_size, 141 page_size=requested_page_size, 142 stream_slice=stream_slice, 143 ) 144 if should_stop: 145 return None 146 token = self._cursor_value.eval( 147 config=self.config, 148 response=decoded_response, 149 headers=headers, 150 last_record=last_record, 151 last_page_size=last_page_size, 152 page_size=requested_page_size, 153 stream_slice=stream_slice, 154 ) 155 return token if token else None
Parameters
- response: response to process
- last_page_size: the number of records read from the response
- last_record: the last record extracted from the response
- last_page_token_value: The current value of the page token made on the last request
- page_size_override: the page size that was actually requested, when it differs from the configured
one because of a
REDUCE_PAGE_SIZEresponse action. Strategies that use their configured page size as a stop condition must honor this value, else they end the pagination early and skip records. - stream_slice: the slice the page was read for. Only passed to strategies that declare it, so existing strategies defined outside of the CDK keep working unchanged.
Returns
next page token. Returns None if there are no more pages to fetch
28@dataclass 29class DatetimeBasedCursor(DeclarativeCursor): 30 """ 31 Slices the stream over a datetime range and create a state with format {<cursor_field>: <datetime> } 32 33 Given a start time, end time, a step function, and an optional lookback window, 34 the stream slicer will partition the date range from start time - lookback window to end time. 35 36 The step function is defined as a string of the form ISO8601 duration 37 38 The timestamp format accepts the same format codes as datetime.strfptime, which are 39 all the format codes required by the 1989 C standard. 40 Full list of accepted format codes: https://man7.org/linux/man-pages/man3/strftime.3.html 41 42 Attributes: 43 start_datetime (Union[MinMaxDatetime, str]): the datetime that determines the earliest record that should be synced 44 end_datetime (Optional[Union[MinMaxDatetime, str]]): the datetime that determines the last record that should be synced 45 cursor_field (Union[InterpolatedString, str]): record's cursor field 46 datetime_format (str): format of the datetime 47 step (Optional[str]): size of the timewindow (ISO8601 duration) 48 cursor_granularity (Optional[str]): smallest increment the datetime_format has (ISO 8601 duration) that will be used to ensure that the start of a slice does not overlap with the end of the previous one 49 config (Config): connection config 50 start_time_option (Optional[RequestOption]): request option for start time 51 end_time_option (Optional[RequestOption]): request option for end time 52 partition_field_start (Optional[str]): partition start time field 53 partition_field_end (Optional[str]): stream slice end time field 54 lookback_window (Optional[InterpolatedString]): how many days before start_datetime to read data for (ISO8601 duration) 55 """ 56 57 start_datetime: Union[MinMaxDatetime, str] 58 cursor_field: Union[InterpolatedString, str] 59 datetime_format: str 60 config: Config 61 parameters: InitVar[Mapping[str, Any]] 62 _highest_observed_cursor_field_value: Optional[str] = field( 63 repr=False, default=None 64 ) # tracks the latest observed datetime, which may not be safe to emit in the case of out-of-order records 65 _cursor: Optional[str] = field( 66 repr=False, default=None 67 ) # tracks the latest observed datetime that is appropriate to emit as stream state 68 end_datetime: Optional[Union[MinMaxDatetime, str]] = None 69 step: Optional[Union[InterpolatedString, str]] = None 70 cursor_granularity: Optional[str] = None 71 start_time_option: Optional[RequestOption] = None 72 end_time_option: Optional[RequestOption] = None 73 partition_field_start: Optional[str] = None 74 partition_field_end: Optional[str] = None 75 lookback_window: Optional[Union[InterpolatedString, str]] = None 76 message_repository: Optional[MessageRepository] = None 77 is_compare_strictly: Optional[bool] = False 78 cursor_datetime_formats: List[str] = field(default_factory=lambda: []) 79 80 def __post_init__(self, parameters: Mapping[str, Any]) -> None: 81 if (self.step and not self.cursor_granularity) or ( 82 not self.step and self.cursor_granularity 83 ): 84 raise ValueError( 85 f"If step is defined, cursor_granularity should be as well and vice-versa. " 86 f"Right now, step is `{self.step}` and cursor_granularity is `{self.cursor_granularity}`" 87 ) 88 self._start_datetime = MinMaxDatetime.create(self.start_datetime, parameters) 89 self._end_datetime = ( 90 None if not self.end_datetime else MinMaxDatetime.create(self.end_datetime, parameters) 91 ) 92 93 self._timezone = datetime.timezone.utc 94 self._interpolation = JinjaInterpolation() 95 96 self._step = ( 97 self._parse_timedelta( 98 InterpolatedString.create(self.step, parameters=parameters).eval(self.config) 99 ) 100 if self.step 101 else datetime.timedelta.max 102 ) 103 self._cursor_granularity = self._parse_timedelta(self.cursor_granularity) 104 self.cursor_field = InterpolatedString.create(self.cursor_field, parameters=parameters) 105 self._lookback_window = ( 106 InterpolatedString.create(self.lookback_window, parameters=parameters) 107 if self.lookback_window 108 else None 109 ) 110 self._partition_field_start = InterpolatedString.create( 111 self.partition_field_start or "start_time", parameters=parameters 112 ) 113 self._partition_field_end = InterpolatedString.create( 114 self.partition_field_end or "end_time", parameters=parameters 115 ) 116 self._parser = DatetimeParser() 117 118 # If datetime format is not specified then start/end datetime should inherit it from the stream slicer 119 if not self._start_datetime.datetime_format: 120 self._start_datetime.datetime_format = self.datetime_format 121 if self._end_datetime and not self._end_datetime.datetime_format: 122 self._end_datetime.datetime_format = self.datetime_format 123 124 if not self.cursor_datetime_formats: 125 self.cursor_datetime_formats = [self.datetime_format] 126 127 _validate_component_request_option_paths( 128 self.config, self.start_time_option, self.end_time_option 129 ) 130 131 def get_stream_state(self) -> StreamState: 132 return {self.cursor_field.eval(self.config): self._cursor} if self._cursor else {} # type: ignore # cursor_field is converted to an InterpolatedString in __post_init__ 133 134 def set_initial_state(self, stream_state: StreamState) -> None: 135 """ 136 Cursors are not initialized with their state. As state is needed in order to function properly, this method should be called 137 before calling anything else 138 139 :param stream_state: The state of the stream as returned by get_stream_state 140 """ 141 self._cursor = ( 142 stream_state.get(self.cursor_field.eval(self.config)) if stream_state else None # type: ignore [union-attr] 143 ) 144 145 def observe(self, stream_slice: StreamSlice, record: Record) -> None: 146 """ 147 Register a record with the cursor; the cursor instance can then use it to manage the state of the in-progress stream read. 148 149 :param stream_slice: The current slice, which may or may not contain the most recently observed record 150 :param record: the most recently-read record, which the cursor can use to update the stream state. Outwardly-visible changes to the 151 stream state may need to be deferred depending on whether the source reliably orders records by the cursor field. 152 """ 153 record_cursor_value = record.get(self.cursor_field.eval(self.config)) # type: ignore # cursor_field is converted to an InterpolatedString in __post_init__ 154 # if the current record has no cursor value, we cannot meaningfully update the state based on it, so there is nothing more to do 155 if not record_cursor_value: 156 return 157 158 start_field = self._partition_field_start.eval(self.config) 159 end_field = self._partition_field_end.eval(self.config) 160 is_highest_observed_cursor_value = ( 161 not self._highest_observed_cursor_field_value 162 or self.parse_date(record_cursor_value) 163 > self.parse_date(self._highest_observed_cursor_field_value) 164 ) 165 if ( 166 self._is_within_daterange_boundaries( 167 record, 168 stream_slice.get(start_field), # type: ignore [arg-type] 169 stream_slice.get(end_field), # type: ignore [arg-type] 170 ) 171 and is_highest_observed_cursor_value 172 ): 173 self._highest_observed_cursor_field_value = record_cursor_value 174 175 def close_slice(self, stream_slice: StreamSlice, *args: Any) -> None: 176 if stream_slice.partition: 177 raise ValueError( 178 f"Stream slice {stream_slice} should not have a partition. Got {stream_slice.partition}." 179 ) 180 cursor_value_str_by_cursor_value_datetime = dict( 181 map( 182 # we need to ensure the cursor value is preserved as is in the state else the CATs might complain of something like 183 # 2023-01-04T17:30:19.000Z' <= '2023-01-04T17:30:19.000000Z' 184 lambda datetime_str: (self.parse_date(datetime_str), datetime_str), # type: ignore # because of the filter on the next line, this will only be called with a str 185 filter( 186 lambda item: item, [self._cursor, self._highest_observed_cursor_field_value] 187 ), 188 ) 189 ) 190 self._cursor = ( 191 cursor_value_str_by_cursor_value_datetime[ 192 max(cursor_value_str_by_cursor_value_datetime.keys()) 193 ] 194 if cursor_value_str_by_cursor_value_datetime 195 else None 196 ) 197 198 def stream_slices(self) -> Iterable[StreamSlice]: 199 """ 200 Partition the daterange into slices of size = step. 201 202 The start of the window is the minimum datetime between start_datetime - lookback_window and the stream_state's datetime 203 The end of the window is the minimum datetime between the start of the window and end_datetime. 204 205 :return: 206 """ 207 end_datetime = self.select_best_end_datetime() 208 start_datetime = self._calculate_earliest_possible_value(self.select_best_end_datetime()) 209 return self._partition_daterange(start_datetime, end_datetime, self._step) 210 211 def select_state(self, stream_slice: Optional[StreamSlice] = None) -> Optional[StreamState]: 212 # Datetime based cursors operate over slices made up of datetime ranges. Stream state is based on the progress 213 # through each slice and does not belong to a specific slice. We just return stream state as it is. 214 return self.get_stream_state() 215 216 def _calculate_earliest_possible_value( 217 self, end_datetime: datetime.datetime 218 ) -> datetime.datetime: 219 lookback_delta = self._parse_timedelta( 220 self._lookback_window.eval(self.config) if self._lookback_window else "P0D" 221 ) 222 earliest_possible_start_datetime = min( 223 self._start_datetime.get_datetime(self.config), end_datetime 224 ) 225 try: 226 cursor_datetime = ( 227 self._calculate_cursor_datetime_from_state(self.get_stream_state()) - lookback_delta 228 ) 229 except OverflowError: 230 # cursor_datetime defers to the minimum date if it does not exist in the state. Trying to subtract 231 # a timedelta from the minimum datetime results in an OverflowError 232 cursor_datetime = self._calculate_cursor_datetime_from_state(self.get_stream_state()) 233 return max(earliest_possible_start_datetime, cursor_datetime) 234 235 def select_best_end_datetime(self) -> datetime.datetime: 236 """ 237 Returns the optimal end datetime. 238 This method compares the current datetime with a pre-configured end datetime 239 and returns the earlier of the two. If no pre-configured end datetime is set, 240 the current datetime is returned. 241 242 :return datetime.datetime: The best end datetime, which is either the current datetime or the pre-configured end datetime, whichever is earlier. 243 """ 244 now = datetime.datetime.now(tz=self._timezone) 245 if not self._end_datetime: 246 return now 247 return min(self._end_datetime.get_datetime(self.config), now) 248 249 def _calculate_cursor_datetime_from_state( 250 self, stream_state: Mapping[str, Any] 251 ) -> datetime.datetime: 252 if self.cursor_field.eval(self.config, stream_state=stream_state) in stream_state: # type: ignore # cursor_field is converted to an InterpolatedString in __post_init__ 253 return self.parse_date(stream_state[self.cursor_field.eval(self.config)]) # type: ignore # cursor_field is converted to an InterpolatedString in __post_init__ 254 return datetime.datetime.min.replace(tzinfo=datetime.timezone.utc) 255 256 def _format_datetime(self, dt: datetime.datetime) -> str: 257 return self._parser.format(dt, self.datetime_format) 258 259 def _partition_daterange( 260 self, 261 start: datetime.datetime, 262 end: datetime.datetime, 263 step: Union[datetime.timedelta, Duration], 264 ) -> List[StreamSlice]: 265 start_field = self._partition_field_start.eval(self.config) 266 end_field = self._partition_field_end.eval(self.config) 267 dates = [] 268 269 while self._is_within_date_range(start, end): 270 next_start = self._evaluate_next_start_date_safely(start, step) 271 end_date = self._get_date(next_start - self._cursor_granularity, end, min) 272 dates.append( 273 StreamSlice( 274 partition={}, 275 cursor_slice={ 276 start_field: self._format_datetime(start), 277 end_field: self._format_datetime(end_date), 278 }, 279 ) 280 ) 281 start = next_start 282 return dates 283 284 def _is_within_date_range(self, start: datetime.datetime, end: datetime.datetime) -> bool: 285 if self.is_compare_strictly: 286 return start < end 287 return start <= end 288 289 def _evaluate_next_start_date_safely( 290 self, start: datetime.datetime, step: datetime.timedelta 291 ) -> datetime.datetime: 292 """ 293 Given that we set the default step at datetime.timedelta.max, we will generate an OverflowError when evaluating the next start_date 294 This method assumes that users would never enter a step that would generate an overflow. Given that would be the case, the code 295 would have broken anyway. 296 """ 297 try: 298 return start + step 299 except OverflowError: 300 return datetime.datetime.max.replace(tzinfo=datetime.timezone.utc) 301 302 def _get_date( 303 self, 304 cursor_value: datetime.datetime, 305 default_date: datetime.datetime, 306 comparator: Callable[[datetime.datetime, datetime.datetime], datetime.datetime], 307 ) -> datetime.datetime: 308 cursor_date = cursor_value or default_date 309 return comparator(cursor_date, default_date) 310 311 def parse_date(self, date: str) -> datetime.datetime: 312 for datetime_format in self.cursor_datetime_formats + [self.datetime_format]: 313 try: 314 return self._parser.parse(date, datetime_format) 315 except ValueError: 316 pass 317 raise ValueError(f"No format in {self.cursor_datetime_formats} matching {date}") 318 319 @classmethod 320 def _parse_timedelta(cls, time_str: Optional[str]) -> Union[datetime.timedelta, Duration]: 321 """ 322 :return Parses an ISO 8601 durations into datetime.timedelta or Duration objects. 323 """ 324 if not time_str: 325 return datetime.timedelta(0) 326 return parse_duration(time_str) 327 328 def get_request_params( 329 self, 330 *, 331 stream_state: Optional[StreamState] = None, 332 stream_slice: Optional[StreamSlice] = None, 333 next_page_token: Optional[Mapping[str, Any]] = None, 334 ) -> Mapping[str, Any]: 335 return self._get_request_options(RequestOptionType.request_parameter, stream_slice) 336 337 def get_request_headers( 338 self, 339 *, 340 stream_state: Optional[StreamState] = None, 341 stream_slice: Optional[StreamSlice] = None, 342 next_page_token: Optional[Mapping[str, Any]] = None, 343 ) -> Mapping[str, Any]: 344 return self._get_request_options(RequestOptionType.header, stream_slice) 345 346 def get_request_body_data( 347 self, 348 *, 349 stream_state: Optional[StreamState] = None, 350 stream_slice: Optional[StreamSlice] = None, 351 next_page_token: Optional[Mapping[str, Any]] = None, 352 ) -> Mapping[str, Any]: 353 return self._get_request_options(RequestOptionType.body_data, stream_slice) 354 355 def get_request_body_json( 356 self, 357 *, 358 stream_state: Optional[StreamState] = None, 359 stream_slice: Optional[StreamSlice] = None, 360 next_page_token: Optional[Mapping[str, Any]] = None, 361 ) -> Mapping[str, Any]: 362 return self._get_request_options(RequestOptionType.body_json, stream_slice) 363 364 def request_kwargs(self) -> Mapping[str, Any]: 365 # Never update kwargs 366 return {} 367 368 def _get_request_options( 369 self, option_type: RequestOptionType, stream_slice: Optional[StreamSlice] 370 ) -> Mapping[str, Any]: 371 options: MutableMapping[str, Any] = {} 372 if not stream_slice: 373 return options 374 375 if self.start_time_option and self.start_time_option.inject_into == option_type: 376 start_time_value = stream_slice.get(self._partition_field_start.eval(self.config)) 377 self.start_time_option.inject_into_request(options, start_time_value, self.config) 378 379 if self.end_time_option and self.end_time_option.inject_into == option_type: 380 end_time_value = stream_slice.get(self._partition_field_end.eval(self.config)) 381 self.end_time_option.inject_into_request(options, end_time_value, self.config) 382 383 return options 384 385 def should_be_synced(self, record: Record) -> bool: 386 cursor_field = self.cursor_field.eval(self.config) # type: ignore # cursor_field is converted to an InterpolatedString in __post_init__ 387 record_cursor_value = record.get(cursor_field) 388 if not record_cursor_value: 389 self._send_log( 390 Level.WARN, 391 f"Could not find cursor field `{cursor_field}` in record. The incremental sync will assume it needs to be synced", 392 ) 393 return True 394 latest_possible_cursor_value = self.select_best_end_datetime() 395 earliest_possible_cursor_value = self._calculate_earliest_possible_value( 396 latest_possible_cursor_value 397 ) 398 return self._is_within_daterange_boundaries( 399 record, earliest_possible_cursor_value, latest_possible_cursor_value 400 ) 401 402 def _is_within_daterange_boundaries( 403 self, 404 record: Record, 405 start_datetime_boundary: Union[datetime.datetime, str], 406 end_datetime_boundary: Union[datetime.datetime, str], 407 ) -> bool: 408 cursor_field = self.cursor_field.eval(self.config) # type: ignore # cursor_field is converted to an InterpolatedString in __post_init__ 409 record_cursor_value = record.get(cursor_field) 410 if not record_cursor_value: 411 self._send_log( 412 Level.WARN, 413 f"Could not find cursor field `{cursor_field}` in record. The record will not be considered when emitting sync state", 414 ) 415 return False 416 if isinstance(start_datetime_boundary, str): 417 start_datetime_boundary = self.parse_date(start_datetime_boundary) 418 if isinstance(end_datetime_boundary, str): 419 end_datetime_boundary = self.parse_date(end_datetime_boundary) 420 return ( 421 start_datetime_boundary <= self.parse_date(record_cursor_value) <= end_datetime_boundary 422 ) 423 424 def _send_log(self, level: Level, message: str) -> None: 425 if self.message_repository: 426 self.message_repository.emit_message( 427 AirbyteMessage( 428 type=Type.LOG, 429 log=AirbyteLogMessage(level=level, message=message), 430 ) 431 ) 432 433 def set_runtime_lookback_window(self, lookback_window_in_seconds: int) -> None: 434 """ 435 Updates the lookback window based on a given number of seconds if the new duration 436 is greater than the currently configured lookback window. 437 438 :param lookback_window_in_seconds: The lookback duration in seconds to potentially update to. 439 """ 440 runtime_lookback_window = duration_isoformat(timedelta(seconds=lookback_window_in_seconds)) 441 config_lookback = parse_duration( 442 self._lookback_window.eval(self.config) if self._lookback_window else "P0D" 443 ) 444 445 # Check if the new runtime lookback window is greater than the current config lookback 446 if parse_duration(runtime_lookback_window) > config_lookback: 447 self._lookback_window = InterpolatedString.create( 448 runtime_lookback_window, parameters={} 449 )
Slices the stream over a datetime range and create a state with format {
Given a start time, end time, a step function, and an optional lookback window, the stream slicer will partition the date range from start time - lookback window to end time.
The step function is defined as a string of the form ISO8601 duration
The timestamp format accepts the same format codes as datetime.strfptime, which are all the format codes required by the 1989 C standard. Full list of accepted format codes: https://man7.org/linux/man-pages/man3/strftime.3.html
Attributes:
- start_datetime (Union[MinMaxDatetime, str]): the datetime that determines the earliest record that should be synced
- end_datetime (Optional[Union[MinMaxDatetime, str]]): the datetime that determines the last record that should be synced
- cursor_field (Union[InterpolatedString, str]): record's cursor field
- datetime_format (str): format of the datetime
- step (Optional[str]): size of the timewindow (ISO8601 duration)
- cursor_granularity (Optional[str]): smallest increment the datetime_format has (ISO 8601 duration) that will be used to ensure that the start of a slice does not overlap with the end of the previous one
- config (Config): connection config
- start_time_option (Optional[RequestOption]): request option for start time
- end_time_option (Optional[RequestOption]): request option for end time
- partition_field_start (Optional[str]): partition start time field
- partition_field_end (Optional[str]): stream slice end time field
- lookback_window (Optional[InterpolatedString]): how many days before start_datetime to read data for (ISO8601 duration)
131 def get_stream_state(self) -> StreamState: 132 return {self.cursor_field.eval(self.config): self._cursor} if self._cursor else {} # type: ignore # cursor_field is converted to an InterpolatedString in __post_init__
Returns the current stream state. We would like to restrict it's usage since it does expose internal of state. As of 2023-06-14, it is used for two things:
- Interpolation of the requests
- Transformation of records
- Saving the state
For the first case, we are probably stuck with exposing the stream state. For the second, we can probably expose a method that allows for emitting the state to the platform.
134 def set_initial_state(self, stream_state: StreamState) -> None: 135 """ 136 Cursors are not initialized with their state. As state is needed in order to function properly, this method should be called 137 before calling anything else 138 139 :param stream_state: The state of the stream as returned by get_stream_state 140 """ 141 self._cursor = ( 142 stream_state.get(self.cursor_field.eval(self.config)) if stream_state else None # type: ignore [union-attr] 143 )
Cursors are not initialized with their state. As state is needed in order to function properly, this method should be called before calling anything else
Parameters
- stream_state: The state of the stream as returned by get_stream_state
145 def observe(self, stream_slice: StreamSlice, record: Record) -> None: 146 """ 147 Register a record with the cursor; the cursor instance can then use it to manage the state of the in-progress stream read. 148 149 :param stream_slice: The current slice, which may or may not contain the most recently observed record 150 :param record: the most recently-read record, which the cursor can use to update the stream state. Outwardly-visible changes to the 151 stream state may need to be deferred depending on whether the source reliably orders records by the cursor field. 152 """ 153 record_cursor_value = record.get(self.cursor_field.eval(self.config)) # type: ignore # cursor_field is converted to an InterpolatedString in __post_init__ 154 # if the current record has no cursor value, we cannot meaningfully update the state based on it, so there is nothing more to do 155 if not record_cursor_value: 156 return 157 158 start_field = self._partition_field_start.eval(self.config) 159 end_field = self._partition_field_end.eval(self.config) 160 is_highest_observed_cursor_value = ( 161 not self._highest_observed_cursor_field_value 162 or self.parse_date(record_cursor_value) 163 > self.parse_date(self._highest_observed_cursor_field_value) 164 ) 165 if ( 166 self._is_within_daterange_boundaries( 167 record, 168 stream_slice.get(start_field), # type: ignore [arg-type] 169 stream_slice.get(end_field), # type: ignore [arg-type] 170 ) 171 and is_highest_observed_cursor_value 172 ): 173 self._highest_observed_cursor_field_value = record_cursor_value
Register a record with the cursor; the cursor instance can then use it to manage the state of the in-progress stream read.
Parameters
- stream_slice: The current slice, which may or may not contain the most recently observed record
- record: the most recently-read record, which the cursor can use to update the stream state. Outwardly-visible changes to the stream state may need to be deferred depending on whether the source reliably orders records by the cursor field.
175 def close_slice(self, stream_slice: StreamSlice, *args: Any) -> None: 176 if stream_slice.partition: 177 raise ValueError( 178 f"Stream slice {stream_slice} should not have a partition. Got {stream_slice.partition}." 179 ) 180 cursor_value_str_by_cursor_value_datetime = dict( 181 map( 182 # we need to ensure the cursor value is preserved as is in the state else the CATs might complain of something like 183 # 2023-01-04T17:30:19.000Z' <= '2023-01-04T17:30:19.000000Z' 184 lambda datetime_str: (self.parse_date(datetime_str), datetime_str), # type: ignore # because of the filter on the next line, this will only be called with a str 185 filter( 186 lambda item: item, [self._cursor, self._highest_observed_cursor_field_value] 187 ), 188 ) 189 ) 190 self._cursor = ( 191 cursor_value_str_by_cursor_value_datetime[ 192 max(cursor_value_str_by_cursor_value_datetime.keys()) 193 ] 194 if cursor_value_str_by_cursor_value_datetime 195 else None 196 )
Update state based on the stream slice. Note that stream_slice.cursor_slice and most_recent_record.associated_slice are expected
to be the same but we make it explicit here that stream_slice should be leveraged to update the state. We do not pass in the
latest record, since cursor instances should maintain the relevant internal state on their own.
Parameters
- stream_slice: slice to close
198 def stream_slices(self) -> Iterable[StreamSlice]: 199 """ 200 Partition the daterange into slices of size = step. 201 202 The start of the window is the minimum datetime between start_datetime - lookback_window and the stream_state's datetime 203 The end of the window is the minimum datetime between the start of the window and end_datetime. 204 205 :return: 206 """ 207 end_datetime = self.select_best_end_datetime() 208 start_datetime = self._calculate_earliest_possible_value(self.select_best_end_datetime()) 209 return self._partition_daterange(start_datetime, end_datetime, self._step)
Partition the daterange into slices of size = step.
The start of the window is the minimum datetime between start_datetime - lookback_window and the stream_state's datetime The end of the window is the minimum datetime between the start of the window and end_datetime.
Returns
211 def select_state(self, stream_slice: Optional[StreamSlice] = None) -> Optional[StreamState]: 212 # Datetime based cursors operate over slices made up of datetime ranges. Stream state is based on the progress 213 # through each slice and does not belong to a specific slice. We just return stream state as it is. 214 return self.get_stream_state()
Get the state value of a specific stream_slice. For incremental or resumable full refresh cursors which only manage state in a single dimension this is the entire state object. For per-partition cursors used by substreams, this returns the state of a specific parent delineated by the incoming slice's partition object.
235 def select_best_end_datetime(self) -> datetime.datetime: 236 """ 237 Returns the optimal end datetime. 238 This method compares the current datetime with a pre-configured end datetime 239 and returns the earlier of the two. If no pre-configured end datetime is set, 240 the current datetime is returned. 241 242 :return datetime.datetime: The best end datetime, which is either the current datetime or the pre-configured end datetime, whichever is earlier. 243 """ 244 now = datetime.datetime.now(tz=self._timezone) 245 if not self._end_datetime: 246 return now 247 return min(self._end_datetime.get_datetime(self.config), now)
Returns the optimal end datetime. This method compares the current datetime with a pre-configured end datetime and returns the earlier of the two. If no pre-configured end datetime is set, the current datetime is returned.
Returns
The best end datetime, which is either the current datetime or the pre-configured end datetime, whichever is earlier.
311 def parse_date(self, date: str) -> datetime.datetime: 312 for datetime_format in self.cursor_datetime_formats + [self.datetime_format]: 313 try: 314 return self._parser.parse(date, datetime_format) 315 except ValueError: 316 pass 317 raise ValueError(f"No format in {self.cursor_datetime_formats} matching {date}")
328 def get_request_params( 329 self, 330 *, 331 stream_state: Optional[StreamState] = None, 332 stream_slice: Optional[StreamSlice] = None, 333 next_page_token: Optional[Mapping[str, Any]] = None, 334 ) -> Mapping[str, Any]: 335 return self._get_request_options(RequestOptionType.request_parameter, stream_slice)
Specifies the query parameters that should be set on an outgoing HTTP request given the inputs.
E.g: you might want to define query parameters for paging if next_page_token is not None.
337 def get_request_headers( 338 self, 339 *, 340 stream_state: Optional[StreamState] = None, 341 stream_slice: Optional[StreamSlice] = None, 342 next_page_token: Optional[Mapping[str, Any]] = None, 343 ) -> Mapping[str, Any]: 344 return self._get_request_options(RequestOptionType.header, stream_slice)
Return any non-auth headers. Authentication headers will overwrite any overlapping headers returned from this method.
346 def get_request_body_data( 347 self, 348 *, 349 stream_state: Optional[StreamState] = None, 350 stream_slice: Optional[StreamSlice] = None, 351 next_page_token: Optional[Mapping[str, Any]] = None, 352 ) -> Mapping[str, Any]: 353 return self._get_request_options(RequestOptionType.body_data, stream_slice)
Specifies how to populate the body of the request with a non-JSON payload.
If returns a ready text that it will be sent as is. If returns a dict that it will be converted to a urlencoded form. E.g. {"key1": "value1", "key2": "value2"} => "key1=value1&key2=value2"
At the same time only one of the 'request_body_data' and 'request_body_json' functions can be overridden.
355 def get_request_body_json( 356 self, 357 *, 358 stream_state: Optional[StreamState] = None, 359 stream_slice: Optional[StreamSlice] = None, 360 next_page_token: Optional[Mapping[str, Any]] = None, 361 ) -> Mapping[str, Any]: 362 return self._get_request_options(RequestOptionType.body_json, stream_slice)
Specifies how to populate the body of the request with a JSON payload.
At the same time only one of the 'request_body_data' and 'request_body_json' functions can be overridden.
385 def should_be_synced(self, record: Record) -> bool: 386 cursor_field = self.cursor_field.eval(self.config) # type: ignore # cursor_field is converted to an InterpolatedString in __post_init__ 387 record_cursor_value = record.get(cursor_field) 388 if not record_cursor_value: 389 self._send_log( 390 Level.WARN, 391 f"Could not find cursor field `{cursor_field}` in record. The incremental sync will assume it needs to be synced", 392 ) 393 return True 394 latest_possible_cursor_value = self.select_best_end_datetime() 395 earliest_possible_cursor_value = self._calculate_earliest_possible_value( 396 latest_possible_cursor_value 397 ) 398 return self._is_within_daterange_boundaries( 399 record, earliest_possible_cursor_value, latest_possible_cursor_value 400 )
Evaluating if a record should be synced allows for filtering and stop condition on pagination
433 def set_runtime_lookback_window(self, lookback_window_in_seconds: int) -> None: 434 """ 435 Updates the lookback window based on a given number of seconds if the new duration 436 is greater than the currently configured lookback window. 437 438 :param lookback_window_in_seconds: The lookback duration in seconds to potentially update to. 439 """ 440 runtime_lookback_window = duration_isoformat(timedelta(seconds=lookback_window_in_seconds)) 441 config_lookback = parse_duration( 442 self._lookback_window.eval(self.config) if self._lookback_window else "P0D" 443 ) 444 445 # Check if the new runtime lookback window is greater than the current config lookback 446 if parse_duration(runtime_lookback_window) > config_lookback: 447 self._lookback_window = InterpolatedString.create( 448 runtime_lookback_window, parameters={} 449 )
Updates the lookback window based on a given number of seconds if the new duration is greater than the currently configured lookback window.
Parameters
- lookback_window_in_seconds: The lookback duration in seconds to potentially update to.
14@dataclass 15class DeclarativeAuthenticator(AbstractHeaderAuthenticator): 16 """ 17 Interface used to associate which authenticators can be used as part of the declarative framework 18 """ 19 20 def get_request_params(self) -> Mapping[str, Any]: 21 """HTTP request parameter to add to the requests""" 22 return {} 23 24 def get_request_body_data(self) -> Union[Mapping[str, Any], str]: 25 """Form-encoded body data to set on the requests""" 26 return {} 27 28 def get_request_body_json(self) -> Mapping[str, Any]: 29 """JSON-encoded body data to set on the requests""" 30 return {}
Interface used to associate which authenticators can be used as part of the declarative framework
20 def get_request_params(self) -> Mapping[str, Any]: 21 """HTTP request parameter to add to the requests""" 22 return {}
HTTP request parameter to add to the requests
24 def get_request_body_data(self) -> Union[Mapping[str, Any], str]: 25 """Form-encoded body data to set on the requests""" 26 return {}
Form-encoded body data to set on the requests
28 def get_request_body_json(self) -> Mapping[str, Any]: 29 """JSON-encoded body data to set on the requests""" 30 return {}
JSON-encoded body data to set on the requests
Inherited Members
28@dataclass 29class DeclarativeOauth2Authenticator(AbstractOauth2Authenticator, DeclarativeAuthenticator): 30 """ 31 Generates OAuth2.0 access tokens from an OAuth2.0 refresh token and client credentials based on 32 a declarative connector configuration file. Credentials can be defined explicitly or via interpolation 33 at runtime. The generated access token is attached to each request via the Authorization header. 34 35 Attributes: 36 token_refresh_endpoint (Union[InterpolatedString, str]): The endpoint to refresh the access token 37 client_id (Union[InterpolatedString, str]): The client id 38 client_secret (Union[InterpolatedString, str]): Client secret (can be empty for APIs that support this) 39 refresh_token (Union[InterpolatedString, str]): The token used to refresh the access token 40 access_token_name (Union[InterpolatedString, str]): THe field to extract access token from in the response 41 expires_in_name (Union[InterpolatedString, str]): The field to extract expires_in from in the response 42 config (Mapping[str, Any]): The user-provided configuration as specified by the source's spec 43 scopes (Optional[List[str]]): The scopes to request 44 token_expiry_date (Optional[Union[InterpolatedString, str]]): The access token expiration date 45 token_expiry_date_format str: format of the datetime; provide it if expires_in is returned in datetime instead of seconds 46 token_expiry_is_time_of_expiration bool: set True it if expires_in is returned as time of expiration instead of the number seconds until expiration 47 refresh_request_body (Optional[Mapping[str, Any]]): The request body to send in the refresh request 48 refresh_request_headers (Optional[Mapping[str, Any]]): The request headers to send in the refresh request 49 send_refresh_request_as_query_params (bool): When True, the standard refresh args (`grant_type`, `refresh_token`, client credentials when not in an `Authorization` header, scopes, plus any `refresh_request_body` extras) are sent on the URL query string instead of in the request body, and the body is emitted empty. Use this for OAuth providers like Gong that document their refresh endpoint with refresh args on the URL query string. Defaults to False. 50 grant_type: The grant_type to request for access_token. If set to refresh_token, the refresh_token parameter has to be provided 51 message_repository (MessageRepository): the message repository used to emit logs on HTTP requests 52 refresh_token_error_status_codes (Tuple[int, ...]): Status codes to identify refresh token errors in response 53 refresh_token_error_key (str): Key to identify refresh token error in response 54 refresh_token_error_values (Tuple[str, ...]): List of values to check for exception during token refresh process 55 """ 56 57 config: Mapping[str, Any] 58 parameters: InitVar[Mapping[str, Any]] 59 client_id: Optional[Union[InterpolatedString, str]] = None 60 client_secret: Optional[Union[InterpolatedString, str]] = None 61 token_refresh_endpoint: Optional[Union[InterpolatedString, str]] = None 62 refresh_token: Optional[Union[InterpolatedString, str]] = None 63 scopes: Optional[List[str]] = None 64 token_expiry_date: Optional[Union[InterpolatedString, str]] = None 65 _token_expiry_date: Optional[AirbyteDateTime] = field(init=False, repr=False, default=None) 66 token_expiry_date_format: Optional[str] = None 67 token_expiry_is_time_of_expiration: bool = False 68 access_token_name: Union[InterpolatedString, str] = "access_token" 69 access_token_value: Optional[Union[InterpolatedString, str]] = None 70 client_id_name: Union[InterpolatedString, str] = "client_id" 71 client_secret_name: Union[InterpolatedString, str] = "client_secret" 72 expires_in_name: Union[InterpolatedString, str] = "expires_in" 73 refresh_token_name: Union[InterpolatedString, str] = "refresh_token" 74 refresh_request_body: Optional[Mapping[str, Any]] = None 75 refresh_request_headers: Optional[Mapping[str, Any]] = None 76 send_refresh_request_as_query_params: bool = False 77 grant_type_name: Union[InterpolatedString, str] = "grant_type" 78 grant_type: Union[InterpolatedString, str] = "refresh_token" 79 message_repository: MessageRepository = NoopMessageRepository() 80 profile_assertion: Optional[DeclarativeAuthenticator] = None 81 use_profile_assertion: Optional[Union[InterpolatedBoolean, str, bool]] = False 82 refresh_token_error_status_codes: Tuple[int, ...] = () 83 refresh_token_error_key: str = "" 84 refresh_token_error_values: Tuple[str, ...] = () 85 86 def __post_init__(self, parameters: Mapping[str, Any]) -> None: 87 super().__init__( 88 refresh_token_error_status_codes=self.refresh_token_error_status_codes, 89 refresh_token_error_key=self.refresh_token_error_key, 90 refresh_token_error_values=self.refresh_token_error_values, 91 ) 92 if self.token_refresh_endpoint is not None: 93 self._token_refresh_endpoint: Optional[InterpolatedString] = InterpolatedString.create( 94 self.token_refresh_endpoint, parameters=parameters 95 ) 96 else: 97 self._token_refresh_endpoint = None 98 self._client_id_name = InterpolatedString.create(self.client_id_name, parameters=parameters) 99 self._client_id = ( 100 InterpolatedString.create(self.client_id, parameters=parameters) 101 if self.client_id 102 else self.client_id 103 ) 104 self._client_secret_name = InterpolatedString.create( 105 self.client_secret_name, parameters=parameters 106 ) 107 self._client_secret = ( 108 InterpolatedString.create(self.client_secret, parameters=parameters) 109 if self.client_secret 110 else self.client_secret 111 ) 112 self._refresh_token_name = InterpolatedString.create( 113 self.refresh_token_name, parameters=parameters 114 ) 115 if self.refresh_token is not None: 116 self._refresh_token: Optional[InterpolatedString] = InterpolatedString.create( 117 self.refresh_token, parameters=parameters 118 ) 119 else: 120 self._refresh_token = None 121 self.access_token_name = InterpolatedString.create( 122 self.access_token_name, parameters=parameters 123 ) 124 self.expires_in_name = InterpolatedString.create( 125 self.expires_in_name, parameters=parameters 126 ) 127 self.grant_type_name = InterpolatedString.create( 128 self.grant_type_name, parameters=parameters 129 ) 130 self.grant_type = InterpolatedString.create( 131 "urn:ietf:params:oauth:grant-type:jwt-bearer" 132 if self.use_profile_assertion 133 else self.grant_type, 134 parameters=parameters, 135 ) 136 self._refresh_request_body = InterpolatedMapping( 137 self.refresh_request_body or {}, parameters=parameters 138 ) 139 self._refresh_request_headers = InterpolatedMapping( 140 self.refresh_request_headers or {}, parameters=parameters 141 ) 142 self._send_refresh_request_as_query_params = self.send_refresh_request_as_query_params 143 try: 144 if ( 145 isinstance(self.token_expiry_date, (int, str)) 146 and str(self.token_expiry_date).isdigit() 147 ): 148 self._token_expiry_date = ab_datetime_parse(self.token_expiry_date) 149 else: 150 self._token_expiry_date = ( 151 ab_datetime_parse( 152 InterpolatedString.create( 153 self.token_expiry_date, parameters=parameters 154 ).eval(self.config) 155 ) 156 if self.token_expiry_date 157 else ab_datetime_now() - timedelta(days=1) 158 ) 159 except ValueError as e: 160 raise ValueError(f"Invalid token expiry date format: {e}") 161 self.use_profile_assertion = ( 162 InterpolatedBoolean(self.use_profile_assertion, parameters=parameters) 163 if isinstance(self.use_profile_assertion, str) 164 else self.use_profile_assertion 165 ) 166 self.assertion_name = "assertion" 167 168 if self.access_token_value is not None: 169 self._access_token_value = InterpolatedString.create( 170 self.access_token_value, parameters=parameters 171 ).eval(self.config) 172 else: 173 self._access_token_value = None 174 175 self._access_token: Optional[str] = ( 176 self._access_token_value if self.access_token_value else None 177 ) 178 179 if not self.use_profile_assertion and any( 180 client_creds is None for client_creds in [self.client_id, self.client_secret] 181 ): 182 raise ValueError( 183 "OAuthAuthenticator configuration error: Both 'client_id' and 'client_secret' are required for the " 184 "basic OAuth flow." 185 ) 186 if self.profile_assertion is None and self.use_profile_assertion: 187 raise ValueError( 188 "OAuthAuthenticator configuration error: 'profile_assertion' is required when using the profile assertion flow." 189 ) 190 if self.get_grant_type() == "refresh_token" and self._refresh_token is None: 191 raise ValueError( 192 "OAuthAuthenticator configuration error: A 'refresh_token' is required when the 'grant_type' is set to 'refresh_token'." 193 ) 194 195 def get_token_refresh_endpoint(self) -> Optional[str]: 196 if self._token_refresh_endpoint is not None: 197 refresh_token_endpoint: str = self._token_refresh_endpoint.eval(self.config) 198 if not refresh_token_endpoint: 199 raise ValueError( 200 "OAuthAuthenticator was unable to evaluate token_refresh_endpoint parameter" 201 ) 202 return refresh_token_endpoint 203 return None 204 205 def get_client_id_name(self) -> str: 206 return self._client_id_name.eval(self.config) # type: ignore # eval returns a string in this context 207 208 def get_client_id(self) -> str: 209 client_id = self._client_id.eval(self.config) if self._client_id else self._client_id 210 if not client_id: 211 raise ValueError("OAuthAuthenticator was unable to evaluate client_id parameter") 212 return client_id # type: ignore # value will be returned as a string, or an error will be raised 213 214 def get_client_secret_name(self) -> str: 215 return self._client_secret_name.eval(self.config) # type: ignore # eval returns a string in this context 216 217 def get_client_secret(self) -> str: 218 client_secret = ( 219 self._client_secret.eval(self.config) if self._client_secret else self._client_secret 220 ) 221 if not client_secret: 222 # We've seen some APIs allowing empty client_secret so we will only log here 223 logger.warning( 224 "OAuthAuthenticator was unable to evaluate client_secret parameter hence it'll be empty" 225 ) 226 return client_secret # type: ignore # value will be returned as a string, which might be empty 227 228 def get_refresh_token_name(self) -> str: 229 return self._refresh_token_name.eval(self.config) # type: ignore # eval returns a string in this context 230 231 def get_refresh_token(self) -> Optional[str]: 232 return None if self._refresh_token is None else str(self._refresh_token.eval(self.config)) 233 234 def get_scopes(self) -> List[str]: 235 return self.scopes or [] 236 237 def get_access_token_name(self) -> str: 238 return self.access_token_name.eval(self.config) # type: ignore # eval returns a string in this context 239 240 def get_expires_in_name(self) -> str: 241 return self.expires_in_name.eval(self.config) # type: ignore # eval returns a string in this context 242 243 def get_grant_type_name(self) -> str: 244 return self.grant_type_name.eval(self.config) # type: ignore # eval returns a string in this context 245 246 def get_grant_type(self) -> str: 247 return self.grant_type.eval(self.config) # type: ignore # eval returns a string in this context 248 249 def get_refresh_request_body(self) -> Mapping[str, Any]: 250 return self._refresh_request_body.eval(self.config) 251 252 def get_refresh_request_headers(self) -> Mapping[str, Any]: 253 return self._refresh_request_headers.eval(self.config) 254 255 def should_send_refresh_request_as_query_params(self) -> bool: 256 return self._send_refresh_request_as_query_params 257 258 def get_token_expiry_date(self) -> AirbyteDateTime: 259 if not self._has_access_token_been_initialized(): 260 return AirbyteDateTime.from_datetime(datetime.min) 261 return self._token_expiry_date # type: ignore # _token_expiry_date is an AirbyteDateTime. It is never None despite what mypy thinks 262 263 def _has_access_token_been_initialized(self) -> bool: 264 return self._access_token is not None 265 266 def set_token_expiry_date(self, value: AirbyteDateTime) -> None: 267 # Overrides of `refresh_access_token` written before 6.45.5 return the raw `expires_in`. 268 if not isinstance(value, datetime): 269 with suppress(ValueError, OverflowError): 270 value = self._parse_token_expiration_date(value) 271 self._token_expiry_date = value 272 273 def get_assertion_name(self) -> str: 274 return self.assertion_name 275 276 def get_assertion(self) -> str: 277 if self.profile_assertion is None: 278 raise ValueError("profile_assertion is not set") 279 return self.profile_assertion.token 280 281 def build_refresh_request_body(self) -> Mapping[str, Any]: 282 """ 283 Returns the request body to set on the refresh request 284 285 Override to define additional parameters 286 """ 287 if self.use_profile_assertion: 288 return { 289 self.get_grant_type_name(): self.get_grant_type(), 290 self.get_assertion_name(): self.get_assertion(), 291 } 292 return super().build_refresh_request_body() 293 294 @property 295 def access_token(self) -> str: 296 if self._access_token is None: 297 raise ValueError("access_token is not set") 298 return self._access_token 299 300 @access_token.setter 301 def access_token(self, value: str) -> None: 302 self._access_token = value 303 304 @property 305 def _message_repository(self) -> MessageRepository: 306 """ 307 Overriding AbstractOauth2Authenticator._message_repository to allow for HTTP request logs 308 """ 309 return self.message_repository
Generates OAuth2.0 access tokens from an OAuth2.0 refresh token and client credentials based on a declarative connector configuration file. Credentials can be defined explicitly or via interpolation at runtime. The generated access token is attached to each request via the Authorization header.
Attributes:
- token_refresh_endpoint (Union[InterpolatedString, str]): The endpoint to refresh the access token
- client_id (Union[InterpolatedString, str]): The client id
- client_secret (Union[InterpolatedString, str]): Client secret (can be empty for APIs that support this)
- refresh_token (Union[InterpolatedString, str]): The token used to refresh the access token
- access_token_name (Union[InterpolatedString, str]): THe field to extract access token from in the response
- expires_in_name (Union[InterpolatedString, str]): The field to extract expires_in from in the response
- config (Mapping[str, Any]): The user-provided configuration as specified by the source's spec
- scopes (Optional[List[str]]): The scopes to request
- token_expiry_date (Optional[Union[InterpolatedString, str]]): The access token expiration date
- token_expiry_date_format str: format of the datetime; provide it if expires_in is returned in datetime instead of seconds
- token_expiry_is_time_of_expiration bool: set True it if expires_in is returned as time of expiration instead of the number seconds until expiration
- refresh_request_body (Optional[Mapping[str, Any]]): The request body to send in the refresh request
- refresh_request_headers (Optional[Mapping[str, Any]]): The request headers to send in the refresh request
- send_refresh_request_as_query_params (bool): When True, the standard refresh args (
grant_type,refresh_token, client credentials when not in anAuthorizationheader, scopes, plus anyrefresh_request_bodyextras) are sent on the URL query string instead of in the request body, and the body is emitted empty. Use this for OAuth providers like Gong that document their refresh endpoint with refresh args on the URL query string. Defaults to False. - grant_type: The grant_type to request for access_token. If set to refresh_token, the refresh_token parameter has to be provided
- message_repository (MessageRepository): the message repository used to emit logs on HTTP requests
- refresh_token_error_status_codes (Tuple[int, ...]): Status codes to identify refresh token errors in response
- refresh_token_error_key (str): Key to identify refresh token error in response
- refresh_token_error_values (Tuple[str, ...]): List of values to check for exception during token refresh process
If all of refresh_token_error_status_codes, refresh_token_error_key, and refresh_token_error_values are set, then http errors with such params will be wrapped in AirbyteTracedException.
Format of the datetime; exists it if expires_in is returned as the expiration datetime instead of seconds until it expires
Indicates that the Token Expiry returns the date until which the token will be valid, not the amount of time it will be valid.
195 def get_token_refresh_endpoint(self) -> Optional[str]: 196 if self._token_refresh_endpoint is not None: 197 refresh_token_endpoint: str = self._token_refresh_endpoint.eval(self.config) 198 if not refresh_token_endpoint: 199 raise ValueError( 200 "OAuthAuthenticator was unable to evaluate token_refresh_endpoint parameter" 201 ) 202 return refresh_token_endpoint 203 return None
Returns the endpoint to refresh the access token
205 def get_client_id_name(self) -> str: 206 return self._client_id_name.eval(self.config) # type: ignore # eval returns a string in this context
The client id name to authenticate
208 def get_client_id(self) -> str: 209 client_id = self._client_id.eval(self.config) if self._client_id else self._client_id 210 if not client_id: 211 raise ValueError("OAuthAuthenticator was unable to evaluate client_id parameter") 212 return client_id # type: ignore # value will be returned as a string, or an error will be raised
The client id to authenticate
214 def get_client_secret_name(self) -> str: 215 return self._client_secret_name.eval(self.config) # type: ignore # eval returns a string in this context
The client secret name to authenticate
217 def get_client_secret(self) -> str: 218 client_secret = ( 219 self._client_secret.eval(self.config) if self._client_secret else self._client_secret 220 ) 221 if not client_secret: 222 # We've seen some APIs allowing empty client_secret so we will only log here 223 logger.warning( 224 "OAuthAuthenticator was unable to evaluate client_secret parameter hence it'll be empty" 225 ) 226 return client_secret # type: ignore # value will be returned as a string, which might be empty
The client secret to authenticate
228 def get_refresh_token_name(self) -> str: 229 return self._refresh_token_name.eval(self.config) # type: ignore # eval returns a string in this context
The refresh token name to authenticate
231 def get_refresh_token(self) -> Optional[str]: 232 return None if self._refresh_token is None else str(self._refresh_token.eval(self.config))
The token used to refresh the access token when it expires
237 def get_access_token_name(self) -> str: 238 return self.access_token_name.eval(self.config) # type: ignore # eval returns a string in this context
Field to extract access token from in the response
240 def get_expires_in_name(self) -> str: 241 return self.expires_in_name.eval(self.config) # type: ignore # eval returns a string in this context
Returns the expires_in field name
243 def get_grant_type_name(self) -> str: 244 return self.grant_type_name.eval(self.config) # type: ignore # eval returns a string in this context
Returns grant_type specified name for requesting access_token
246 def get_grant_type(self) -> str: 247 return self.grant_type.eval(self.config) # type: ignore # eval returns a string in this context
Returns grant_type specified for requesting access_token
249 def get_refresh_request_body(self) -> Mapping[str, Any]: 250 return self._refresh_request_body.eval(self.config)
Returns the request body to set on the refresh request
252 def get_refresh_request_headers(self) -> Mapping[str, Any]: 253 return self._refresh_request_headers.eval(self.config)
Returns the request headers to set on the refresh request
255 def should_send_refresh_request_as_query_params(self) -> bool: 256 return self._send_refresh_request_as_query_params
Returns True if the standard refresh args should be sent on the URL
query string instead of in the request body.
Defaults to False so existing authenticators retain their previous
behavior (params in body, no query params on the refresh URL). Subclasses
can override this to opt into the URL-query-string shape required by OAuth
providers like Gong.
258 def get_token_expiry_date(self) -> AirbyteDateTime: 259 if not self._has_access_token_been_initialized(): 260 return AirbyteDateTime.from_datetime(datetime.min) 261 return self._token_expiry_date # type: ignore # _token_expiry_date is an AirbyteDateTime. It is never None despite what mypy thinks
Expiration date of the access token
266 def set_token_expiry_date(self, value: AirbyteDateTime) -> None: 267 # Overrides of `refresh_access_token` written before 6.45.5 return the raw `expires_in`. 268 if not isinstance(value, datetime): 269 with suppress(ValueError, OverflowError): 270 value = self._parse_token_expiration_date(value) 271 self._token_expiry_date = value
Setter for access token expiration date
281 def build_refresh_request_body(self) -> Mapping[str, Any]: 282 """ 283 Returns the request body to set on the refresh request 284 285 Override to define additional parameters 286 """ 287 if self.use_profile_assertion: 288 return { 289 self.get_grant_type_name(): self.get_grant_type(), 290 self.get_assertion_name(): self.get_assertion(), 291 } 292 return super().build_refresh_request_body()
Returns the request body to set on the refresh request
Override to define additional parameters
294 @property 295 def access_token(self) -> str: 296 if self._access_token is None: 297 raise ValueError("access_token is not set") 298 return self._access_token
Returns the access token
Inherited Members
312@dataclass 313class DeclarativeSingleUseRefreshTokenOauth2Authenticator( 314 SingleUseRefreshTokenOauth2Authenticator, DeclarativeAuthenticator 315): 316 """ 317 Declarative version of SingleUseRefreshTokenOauth2Authenticator which can be used in declarative connectors. 318 """ 319 320 def __init__(self, *args: Any, **kwargs: Any) -> None: 321 super().__init__(*args, **kwargs)
Declarative version of SingleUseRefreshTokenOauth2Authenticator which can be used in declarative connectors.
Arguments:
- connector_config (Mapping[str, Any]): The full connector configuration
- token_refresh_endpoint (str): Full URL to the token refresh endpoint
- scopes (List[str], optional): List of OAuth scopes to pass in the refresh token request body. Defaults to None.
- access_token_name (str, optional): Name of the access token field, used to parse the refresh token response. Defaults to "access_token".
- expires_in_name (str, optional): Name of the name of the field that characterizes when the current access token will expire, used to parse the refresh token response. Defaults to "expires_in".
- refresh_token_name (str, optional): Name of the name of the refresh token field, used to parse the refresh token response. Defaults to "refresh_token".
- refresh_request_body (Mapping[str, Any], optional): Custom key value pair that will be added to the refresh token request body. Defaults to None.
- refresh_request_headers (Mapping[str, Any], optional): Custom key value pair that will be added to the refresh token request headers. Defaults to None.
- send_refresh_request_as_query_params (bool, optional): When True, the standard refresh args (
grant_type,refresh_token, client credentials when not in anAuthorizationheader, scopes, plus anyrefresh_request_bodyextras) are sent on the URL query string and the request body is emitted empty. Use this for OAuth providers like Gong that document their refresh endpoint with refresh args on the URL query string. Defaults to False. - grant_type (str, optional): OAuth grant type. Defaults to "refresh_token".
- client_id (Optional[str]): The client id to authenticate. If not specified, defaults to credentials.client_id in the config object.
- client_secret (Optional[str]): The client secret to authenticate. If not specified, defaults to credentials.client_secret in the config object.
- access_token_config_path (Sequence[str]): Dpath to the access_token field in the connector configuration. Defaults to ("credentials", "access_token").
- refresh_token_config_path (Sequence[str]): Dpath to the refresh_token field in the connector configuration. Defaults to ("credentials", "refresh_token").
- token_expiry_date_config_path (Sequence[str]): Dpath to the token_expiry_date field in the connector configuration. Defaults to ("credentials", "token_expiry_date").
- token_expiry_date_format (Optional[str]): Date format of the token expiry date field (set by expires_in_name). If specified the token expiry date is interpreted as time of expiration, as with token_expiry_is_time_of_expiration, otherwise as number of seconds until expiration.
- token_expiry_is_time_of_expiration bool: set True it if expires_in is returned as time of expiration instead of the number seconds until expiration
- message_repository (MessageRepository): The message repository used to emit logs on HTTP requests and control message on config update.
- emit_control_message_to_message_repository (bool): Whether the message repository additionally receives the CONNECTOR_CONFIG message. This is only for in-process consumers such as the Connector Builder, because messages in the repository are otherwise printed to stdout by the entrypoint and would duplicate the direct stdout emission. Defaults to False.
Inherited Members
- SingleUseRefreshTokenOauth2Authenticator
- access_token
- get_refresh_token
- set_refresh_token
- get_token_expiry_date
- set_token_expiry_date
- token_has_expired
- get_access_token
- refresh_and_set_access_token
- refresh_access_token
- Oauth2Authenticator
- get_token_refresh_endpoint
- get_client_id_name
- get_client_id
- get_client_secret_name
- get_client_secret
- get_refresh_token_name
- get_access_token_name
- get_scopes
- get_expires_in_name
- get_refresh_request_body
- get_refresh_request_headers
- should_send_refresh_request_as_query_params
- get_grant_type_name
- get_grant_type
- token_expiry_is_time_of_expiration
- token_expiry_date_format
33@deprecated("DeclarativeStream has been deprecated in favor of the concurrent DefaultStream") 34@dataclass 35class DeclarativeStream(Stream): 36 """ 37 DeclarativeStream is a Stream that delegates most of its logic to its schema_load and retriever 38 39 Attributes: 40 name (str): stream name 41 primary_key (Optional[Union[str, List[str], List[List[str]]]]): the primary key of the stream 42 schema_loader (SchemaLoader): The schema loader 43 retriever (Retriever): The retriever 44 config (Config): The user-provided configuration as specified by the source's spec 45 stream_cursor_field (Optional[Union[InterpolatedString, str]]): The cursor field 46 stream. Transformations are applied in the order in which they are defined. 47 """ 48 49 retriever: Retriever 50 config: Config 51 parameters: InitVar[Mapping[str, Any]] 52 name: str 53 primary_key: Optional[Union[str, List[str], List[List[str]]]] 54 state_migrations: List[StateMigration] = field(repr=True, default_factory=list) 55 schema_loader: Optional[SchemaLoader] = None 56 _name: str = field(init=False, repr=False, default="") 57 _primary_key: str = field(init=False, repr=False, default="") 58 stream_cursor_field: Optional[Union[InterpolatedString, str]] = None 59 60 def __post_init__(self, parameters: Mapping[str, Any]) -> None: 61 self._stream_cursor_field = ( 62 InterpolatedString.create(self.stream_cursor_field, parameters=parameters) 63 if isinstance(self.stream_cursor_field, str) 64 else self.stream_cursor_field 65 ) 66 self._schema_loader = ( 67 self.schema_loader 68 if self.schema_loader 69 else DefaultSchemaLoader(config=self.config, parameters=parameters) 70 ) 71 72 @property # type: ignore 73 def primary_key(self) -> Optional[Union[str, List[str], List[List[str]]]]: 74 return self._primary_key 75 76 @primary_key.setter 77 def primary_key(self, value: str) -> None: 78 if not isinstance(value, property): 79 self._primary_key = value 80 81 @property 82 def exit_on_rate_limit(self) -> bool: 83 if isinstance(self.retriever, AsyncRetriever): 84 return self.retriever.exit_on_rate_limit 85 86 return self.retriever.requester.exit_on_rate_limit # type: ignore # abstract Retriever class has not requester attribute 87 88 @exit_on_rate_limit.setter 89 def exit_on_rate_limit(self, value: bool) -> None: 90 if isinstance(self.retriever, AsyncRetriever): 91 self.retriever.exit_on_rate_limit = value 92 else: 93 self.retriever.requester.exit_on_rate_limit = value # type: ignore[attr-defined] 94 95 @property # type: ignore 96 def name(self) -> str: 97 """ 98 :return: Stream name. By default this is the implementing class name, but it can be overridden as needed. 99 """ 100 return self._name 101 102 @name.setter 103 def name(self, value: str) -> None: 104 if not isinstance(value, property): 105 self._name = value 106 107 @property 108 def state(self) -> MutableMapping[str, Any]: 109 return self.retriever.state # type: ignore 110 111 @state.setter 112 def state(self, value: MutableMapping[str, Any]) -> None: 113 """State setter, accept state serialized by state getter.""" 114 state: Mapping[str, Any] = value 115 if self.state_migrations: 116 for migration in self.state_migrations: 117 if migration.should_migrate(state): 118 state = migration.migrate(state) 119 self.retriever.state = state 120 121 def get_updated_state( 122 self, current_stream_state: MutableMapping[str, Any], latest_record: Mapping[str, Any] 123 ) -> MutableMapping[str, Any]: 124 return self.state 125 126 @property 127 def cursor_field(self) -> Union[str, List[str]]: 128 """ 129 Override to return the default cursor field used by this stream e.g: an API entity might always use created_at as the cursor field. 130 :return: The name of the field used as a cursor. If the cursor is nested, return an array consisting of the path to the cursor. 131 """ 132 cursor = self._stream_cursor_field.eval(self.config) # type: ignore # _stream_cursor_field is always cast to interpolated string 133 return cursor if cursor else [] 134 135 @property 136 def is_resumable(self) -> bool: 137 # Declarative sources always implement state getter/setter, but whether it supports checkpointing is based on 138 # if the retriever has a cursor defined. 139 return self.retriever.cursor is not None if hasattr(self.retriever, "cursor") else False 140 141 def read_records( 142 self, 143 sync_mode: SyncMode, 144 cursor_field: Optional[List[str]] = None, 145 stream_slice: Optional[Mapping[str, Any]] = None, 146 stream_state: Optional[Mapping[str, Any]] = None, 147 ) -> Iterable[Mapping[str, Any]]: 148 """ 149 :param: stream_state We knowingly avoid using stream_state as we want cursors to manage their own state. 150 """ 151 if stream_slice is None or ( 152 not isinstance(stream_slice, StreamSlice) and stream_slice == {} 153 ): 154 # As the parameter is Optional, many would just call `read_records(sync_mode)` during testing without specifying the field 155 # As part of the declarative model without custom components, this should never happen as the CDK would wire up a 156 # SinglePartitionRouter that would create this StreamSlice properly 157 # As part of the declarative model with custom components, a user that would return a `None` slice would now have the default 158 # empty slice which seems to make sense. 159 stream_slice = StreamSlice(partition={}, cursor_slice={}) 160 if not isinstance(stream_slice, StreamSlice): 161 raise ValueError( 162 f"DeclarativeStream does not support stream_slices that are not StreamSlice. Got {stream_slice}" 163 ) 164 yield from self.retriever.read_records(self.get_json_schema(), stream_slice) # type: ignore # records are of the correct type 165 166 def get_json_schema(self) -> Mapping[str, Any]: # type: ignore 167 """ 168 :return: A dict of the JSON schema representing this stream. 169 170 The default implementation of this method looks for a JSONSchema file with the same name as this stream's "name" property. 171 Override as needed. 172 """ 173 return self._schema_loader.get_json_schema() 174 175 def stream_slices( 176 self, 177 *, 178 sync_mode: SyncMode, 179 cursor_field: Optional[List[str]] = None, 180 stream_state: Optional[Mapping[str, Any]] = None, 181 ) -> Iterable[Optional[StreamSlice]]: 182 """ 183 Override to define the slices for this stream. See the stream slicing section of the docs for more information. 184 185 :param sync_mode: 186 :param cursor_field: 187 :param stream_state: we knowingly avoid using stream_state as we want cursors to manage their own state 188 :return: 189 """ 190 return self.retriever.stream_slices() 191 192 @property 193 def state_checkpoint_interval(self) -> Optional[int]: 194 """ 195 We explicitly disable checkpointing here. There are a couple reasons for that and not all are documented here but: 196 * In the case where records are not ordered, the granularity of what is ordered is the slice. Therefore, we will only update the 197 cursor value once at the end of every slice. 198 * Updating the state once every record would generate issues for data feed stop conditions or semi-incremental syncs where the 199 important state is the one at the beginning of the slice 200 """ 201 return None 202 203 def get_cursor(self) -> Optional[Cursor]: 204 return None 205 206 def _get_checkpoint_reader( 207 self, 208 logger: logging.Logger, 209 cursor_field: Optional[List[str]], 210 sync_mode: SyncMode, 211 stream_state: MutableMapping[str, Any], 212 ) -> CheckpointReader: 213 """ 214 This method is overridden to prevent issues with stream slice classification for incremental streams that have parent streams. 215 216 The classification logic, when used with `itertools.tee`, creates a copy of the stream slices. When `stream_slices` is called 217 the second time, the parent records generated during the classification phase are lost. This occurs because `itertools.tee` 218 only buffers the results, meaning the logic in `simple_retriever` that observes and updates the cursor isn't executed again. 219 220 By overriding this method, we ensure that the stream slices are processed correctly and parent records are not lost, 221 allowing the cursor to function as expected. 222 """ 223 mappings_or_slices = self.stream_slices( 224 cursor_field=cursor_field, 225 sync_mode=sync_mode, # todo: change this interface to no longer rely on sync_mode for behavior 226 stream_state=stream_state, 227 ) 228 229 cursor = self.get_cursor() 230 checkpoint_mode = self._checkpoint_mode 231 232 if isinstance( 233 cursor, (GlobalSubstreamCursor, PerPartitionCursor, PerPartitionWithGlobalCursor) 234 ): 235 self.has_multiple_slices = True 236 return CursorBasedCheckpointReader( 237 stream_slices=mappings_or_slices, 238 cursor=cursor, 239 read_state_from_cursor=checkpoint_mode == CheckpointMode.RESUMABLE_FULL_REFRESH, 240 ) 241 242 return super()._get_checkpoint_reader(logger, cursor_field, sync_mode, stream_state)
DeclarativeStream is a Stream that delegates most of its logic to its schema_load and retriever
Attributes:
- name (str): stream name
- primary_key (Optional[Union[str, List[str], List[List[str]]]]): the primary key of the stream
- schema_loader (SchemaLoader): The schema loader
- retriever (Retriever): The retriever
- config (Config): The user-provided configuration as specified by the source's spec
- stream_cursor_field (Optional[Union[InterpolatedString, str]]): The cursor field
- stream. Transformations are applied in the order in which they are defined.
95 @property # type: ignore 96 def name(self) -> str: 97 """ 98 :return: Stream name. By default this is the implementing class name, but it can be overridden as needed. 99 """ 100 return self._name
Returns
Stream name. By default this is the implementing class name, but it can be overridden as needed.
72 @property # type: ignore 73 def primary_key(self) -> Optional[Union[str, List[str], List[List[str]]]]: 74 return self._primary_key
Returns
string if single primary key, list of strings if composite primary key, list of list of strings if composite primary key consisting of nested fields. If the stream has no primary keys, return None.
81 @property 82 def exit_on_rate_limit(self) -> bool: 83 if isinstance(self.retriever, AsyncRetriever): 84 return self.retriever.exit_on_rate_limit 85 86 return self.retriever.requester.exit_on_rate_limit # type: ignore # abstract Retriever class has not requester attribute
Exit on rate limit getter, should return bool value. False if the stream will retry endlessly when rate limited.
107 @property 108 def state(self) -> MutableMapping[str, Any]: 109 return self.retriever.state # type: ignore
State setter, accept state serialized by state getter.
121 def get_updated_state( 122 self, current_stream_state: MutableMapping[str, Any], latest_record: Mapping[str, Any] 123 ) -> MutableMapping[str, Any]: 124 return self.state
DEPRECATED. Please use explicit state property instead, see IncrementalMixin docs.
Override to extract state from the latest record. Needed to implement incremental sync.
Inspects the latest record extracted from the data source and the current state object and return an updated state object.
For example: if the state object is based on created_at timestamp, and the current state is {'created_at': 10}, and the latest_record is {'name': 'octavia', 'created_at': 20 } then this method would return {'created_at': 20} to indicate state should be updated to this object.
Parameters
- current_stream_state: The stream's current state object
- latest_record: The latest record extracted from the stream
Returns
An updated state object
126 @property 127 def cursor_field(self) -> Union[str, List[str]]: 128 """ 129 Override to return the default cursor field used by this stream e.g: an API entity might always use created_at as the cursor field. 130 :return: The name of the field used as a cursor. If the cursor is nested, return an array consisting of the path to the cursor. 131 """ 132 cursor = self._stream_cursor_field.eval(self.config) # type: ignore # _stream_cursor_field is always cast to interpolated string 133 return cursor if cursor else []
Override to return the default cursor field used by this stream e.g: an API entity might always use created_at as the cursor field.
Returns
The name of the field used as a cursor. If the cursor is nested, return an array consisting of the path to the cursor.
135 @property 136 def is_resumable(self) -> bool: 137 # Declarative sources always implement state getter/setter, but whether it supports checkpointing is based on 138 # if the retriever has a cursor defined. 139 return self.retriever.cursor is not None if hasattr(self.retriever, "cursor") else False
Returns
True if this stream allows the checkpointing of sync progress and can resume from it on subsequent attempts. This differs from supports_incremental because certain kinds of streams like those supporting resumable full refresh can checkpoint progress in between attempts for improved fault tolerance. However, they will start from the beginning on the next sync job.
141 def read_records( 142 self, 143 sync_mode: SyncMode, 144 cursor_field: Optional[List[str]] = None, 145 stream_slice: Optional[Mapping[str, Any]] = None, 146 stream_state: Optional[Mapping[str, Any]] = None, 147 ) -> Iterable[Mapping[str, Any]]: 148 """ 149 :param: stream_state We knowingly avoid using stream_state as we want cursors to manage their own state. 150 """ 151 if stream_slice is None or ( 152 not isinstance(stream_slice, StreamSlice) and stream_slice == {} 153 ): 154 # As the parameter is Optional, many would just call `read_records(sync_mode)` during testing without specifying the field 155 # As part of the declarative model without custom components, this should never happen as the CDK would wire up a 156 # SinglePartitionRouter that would create this StreamSlice properly 157 # As part of the declarative model with custom components, a user that would return a `None` slice would now have the default 158 # empty slice which seems to make sense. 159 stream_slice = StreamSlice(partition={}, cursor_slice={}) 160 if not isinstance(stream_slice, StreamSlice): 161 raise ValueError( 162 f"DeclarativeStream does not support stream_slices that are not StreamSlice. Got {stream_slice}" 163 ) 164 yield from self.retriever.read_records(self.get_json_schema(), stream_slice) # type: ignore # records are of the correct type
Parameters
- stream_state We knowingly avoid using stream_state as we want cursors to manage their own state.
166 def get_json_schema(self) -> Mapping[str, Any]: # type: ignore 167 """ 168 :return: A dict of the JSON schema representing this stream. 169 170 The default implementation of this method looks for a JSONSchema file with the same name as this stream's "name" property. 171 Override as needed. 172 """ 173 return self._schema_loader.get_json_schema()
Returns
A dict of the JSON schema representing this stream.
The default implementation of this method looks for a JSONSchema file with the same name as this stream's "name" property. Override as needed.
175 def stream_slices( 176 self, 177 *, 178 sync_mode: SyncMode, 179 cursor_field: Optional[List[str]] = None, 180 stream_state: Optional[Mapping[str, Any]] = None, 181 ) -> Iterable[Optional[StreamSlice]]: 182 """ 183 Override to define the slices for this stream. See the stream slicing section of the docs for more information. 184 185 :param sync_mode: 186 :param cursor_field: 187 :param stream_state: we knowingly avoid using stream_state as we want cursors to manage their own state 188 :return: 189 """ 190 return self.retriever.stream_slices()
Override to define the slices for this stream. See the stream slicing section of the docs for more information.
Parameters
- sync_mode:
- cursor_field:
- stream_state: we knowingly avoid using stream_state as we want cursors to manage their own state
Returns
192 @property 193 def state_checkpoint_interval(self) -> Optional[int]: 194 """ 195 We explicitly disable checkpointing here. There are a couple reasons for that and not all are documented here but: 196 * In the case where records are not ordered, the granularity of what is ordered is the slice. Therefore, we will only update the 197 cursor value once at the end of every slice. 198 * Updating the state once every record would generate issues for data feed stop conditions or semi-incremental syncs where the 199 important state is the one at the beginning of the slice 200 """ 201 return None
We explicitly disable checkpointing here. There are a couple reasons for that and not all are documented here but:
- In the case where records are not ordered, the granularity of what is ordered is the slice. Therefore, we will only update the cursor value once at the end of every slice.
- Updating the state once every record would generate issues for data feed stop conditions or semi-incremental syncs where the important state is the one at the beginning of the slice
A Cursor is an interface that a stream can implement to manage how its internal state is read and updated while reading records. Historically, Python connectors had no concept of a cursor to manage state. Python streams need to define a cursor implementation and override this method to manage state through a Cursor.
15@dataclass 16class Decoder: 17 """ 18 Decoder strategy to transform a requests.Response into a Mapping[str, Any] 19 """ 20 21 @abstractmethod 22 def is_stream_response(self) -> bool: 23 """ 24 Set to True if you'd like to use stream=True option in http requester 25 """ 26 27 @abstractmethod 28 def decode(self, response: requests.Response) -> DECODER_OUTPUT_TYPE: 29 """ 30 Decodes a requests.Response into a Mapping[str, Any] or an array 31 :param response: the response to decode 32 :return: Generator of Mapping describing the response 33 """
Decoder strategy to transform a requests.Response into a Mapping[str, Any]
21 @abstractmethod 22 def is_stream_response(self) -> bool: 23 """ 24 Set to True if you'd like to use stream=True option in http requester 25 """
Set to True if you'd like to use stream=True option in http requester
27 @abstractmethod 28 def decode(self, response: requests.Response) -> DECODER_OUTPUT_TYPE: 29 """ 30 Decodes a requests.Response into a Mapping[str, Any] or an array 31 :param response: the response to decode 32 :return: Generator of Mapping describing the response 33 """
Decodes a requests.Response into a Mapping[str, Any] or an array
Parameters
- response: the response to decode
Returns
Generator of Mapping describing the response
37@dataclass 38class DefaultPaginator(Paginator): 39 """ 40 Default paginator to request pages of results with a fixed size until the pagination strategy no longer returns a next_page_token 41 42 Examples: 43 1. 44 * fetches up to 10 records at a time by setting the "limit" request param to 10 45 * updates the request path with "{{ response._metadata.next }}" 46 ``` 47 paginator: 48 type: "DefaultPaginator" 49 page_size_option: 50 type: RequestOption 51 inject_into: request_parameter 52 field_name: limit 53 page_token_option: 54 type: RequestPath 55 path: "location" 56 pagination_strategy: 57 type: "CursorPagination" 58 cursor_value: "{{ response._metadata.next }}" 59 page_size: 10 60 ``` 61 62 2. 63 * fetches up to 5 records at a time by setting the "page_size" header to 5 64 * increments a record counter and set the request parameter "offset" to the value of the counter 65 ``` 66 paginator: 67 type: "DefaultPaginator" 68 page_size_option: 69 type: RequestOption 70 inject_into: header 71 field_name: page_size 72 pagination_strategy: 73 type: "OffsetIncrement" 74 page_size: 5 75 page_token_option: 76 option_type: "request_parameter" 77 field_name: "offset" 78 ``` 79 80 3. 81 * fetches up to 5 records at a time by setting the "page_size" request param to 5 82 * increments a page counter and set the request parameter "page" to the value of the counter 83 ``` 84 paginator: 85 type: "DefaultPaginator" 86 page_size_option: 87 type: RequestOption 88 inject_into: request_parameter 89 field_name: page_size 90 pagination_strategy: 91 type: "PageIncrement" 92 page_size: 5 93 page_token_option: 94 type: RequestOption 95 option_type: "request_parameter" 96 field_name: "page" 97 ``` 98 Attributes: 99 page_size_option (Optional[RequestOption]): the request option to set the page size. Cannot be injected in the path. 100 page_token_option (Optional[RequestPath, RequestOption]): the request option to set the page token 101 pagination_strategy (PaginationStrategy): Strategy defining how to get the next page token 102 config (Config): connection config 103 url_base (Union[InterpolatedString, str]): endpoint's base url 104 decoder (Decoder): decoder to decode the response 105 """ 106 107 pagination_strategy: PaginationStrategy 108 config: Config 109 url_base: Union[InterpolatedString, str] 110 parameters: InitVar[Mapping[str, Any]] 111 decoder: Decoder = field( 112 default_factory=lambda: PaginationDecoderDecorator(decoder=JsonDecoder(parameters={})) 113 ) 114 page_size_option: Optional[RequestOption] = None 115 page_token_option: Optional[Union[RequestPath, RequestOption]] = None 116 117 def __post_init__(self, parameters: Mapping[str, Any]) -> None: 118 if self.page_size_option and not self.pagination_strategy.get_page_size(): 119 raise ValueError( 120 "page_size_option cannot be set if the pagination strategy does not have a page_size" 121 ) 122 if isinstance(self.url_base, str): 123 self.url_base = InterpolatedString(string=self.url_base, parameters=parameters) 124 125 if self.page_token_option and not isinstance(self.page_token_option, RequestPath): 126 _validate_component_request_option_paths( 127 self.config, 128 self.page_size_option, 129 self.page_token_option, 130 ) 131 132 def get_initial_token(self) -> Optional[Any]: 133 """ 134 Return the page token that should be used for the first request of a stream 135 136 WARNING: get_initial_token() should not be used by streams that use RFR that perform checkpointing 137 of state using page numbers. Because paginators are stateless 138 """ 139 return self.pagination_strategy.initial_token 140 141 def get_page_size(self) -> Optional[int]: 142 return self.pagination_strategy.get_page_size() 143 144 def next_page_token( 145 self, 146 response: requests.Response, 147 last_page_size: int, 148 last_record: Optional[Record], 149 last_page_token_value: Optional[Any] = None, 150 page_size_override: Optional[int] = None, 151 stream_slice: Optional[StreamSlice] = None, 152 ) -> Optional[Mapping[str, Any]]: 153 next_page_token = self.pagination_strategy.next_page_token( 154 response=response, 155 last_page_size=last_page_size, 156 last_record=last_record, 157 last_page_token_value=last_page_token_value, 158 **page_size_override_kwargs(page_size_override), 159 **stream_slice_kwargs(self.pagination_strategy.next_page_token, stream_slice), 160 ) 161 if next_page_token: 162 return {"next_page_token": next_page_token} 163 else: 164 return None 165 166 def path( 167 self, 168 next_page_token: Optional[Mapping[str, Any]], 169 stream_state: Optional[Mapping[str, Any]] = None, 170 stream_slice: Optional[StreamSlice] = None, 171 ) -> Optional[str]: 172 token = next_page_token.get("next_page_token") if next_page_token else None 173 if token and self.page_token_option and isinstance(self.page_token_option, RequestPath): 174 return str(token) 175 else: 176 return None 177 178 def get_request_params( 179 self, 180 *, 181 stream_state: Optional[StreamState] = None, 182 stream_slice: Optional[StreamSlice] = None, 183 next_page_token: Optional[Mapping[str, Any]] = None, 184 page_size_override: Optional[int] = None, 185 ) -> MutableMapping[str, Any]: 186 return self._get_request_options( 187 RequestOptionType.request_parameter, next_page_token, page_size_override 188 ) 189 190 def get_request_headers( 191 self, 192 *, 193 stream_state: Optional[StreamState] = None, 194 stream_slice: Optional[StreamSlice] = None, 195 next_page_token: Optional[Mapping[str, Any]] = None, 196 page_size_override: Optional[int] = None, 197 ) -> Mapping[str, str]: 198 return self._get_request_options( 199 RequestOptionType.header, next_page_token, page_size_override 200 ) 201 202 def get_request_body_data( 203 self, 204 *, 205 stream_state: Optional[StreamState] = None, 206 stream_slice: Optional[StreamSlice] = None, 207 next_page_token: Optional[Mapping[str, Any]] = None, 208 page_size_override: Optional[int] = None, 209 ) -> Mapping[str, Any]: 210 return self._get_request_options( 211 RequestOptionType.body_data, next_page_token, page_size_override 212 ) 213 214 def get_request_body_json( 215 self, 216 *, 217 stream_state: Optional[StreamState] = None, 218 stream_slice: Optional[StreamSlice] = None, 219 next_page_token: Optional[Mapping[str, Any]] = None, 220 page_size_override: Optional[int] = None, 221 ) -> Mapping[str, Any]: 222 return self._get_request_options( 223 RequestOptionType.body_json, next_page_token, page_size_override 224 ) 225 226 def _get_request_options( 227 self, 228 option_type: RequestOptionType, 229 next_page_token: Optional[Mapping[str, Any]], 230 page_size_override: Optional[int] = None, 231 ) -> MutableMapping[str, Any]: 232 options: MutableMapping[str, Any] = {} 233 234 token = next_page_token.get("next_page_token") if next_page_token else None 235 if ( 236 self.page_token_option 237 and token is not None 238 and isinstance(self.page_token_option, RequestOption) 239 and self.page_token_option.inject_into == option_type 240 ): 241 self.page_token_option.inject_into_request(options, token, self.config) 242 243 if self.page_size_option and self.page_size_option.inject_into == option_type: 244 page_size = ( 245 page_size_override 246 if page_size_override is not None 247 else self.pagination_strategy.get_page_size() 248 ) 249 if page_size: 250 self.page_size_option.inject_into_request(options, page_size, self.config) 251 252 return options
Default paginator to request pages of results with a fixed size until the pagination strategy no longer returns a next_page_token
Examples:
1.
- fetches up to 10 records at a time by setting the "limit" request param to 10
- updates the request path with "{{ response._metadata.next }}"
paginator: type: "DefaultPaginator" page_size_option: type: RequestOption inject_into: request_parameter field_name: limit page_token_option: type: RequestPath path: "location" pagination_strategy: type: "CursorPagination" cursor_value: "{{ response._metadata.next }}" page_size: 102.
- fetches up to 5 records at a time by setting the "page_size" header to 5
- increments a record counter and set the request parameter "offset" to the value of the counter
paginator: type: "DefaultPaginator" page_size_option: type: RequestOption inject_into: header field_name: page_size pagination_strategy: type: "OffsetIncrement" page_size: 5 page_token_option: option_type: "request_parameter" field_name: "offset"3.
- fetches up to 5 records at a time by setting the "page_size" request param to 5
- increments a page counter and set the request parameter "page" to the value of the counter
paginator: type: "DefaultPaginator" page_size_option: type: RequestOption inject_into: request_parameter field_name: page_size pagination_strategy: type: "PageIncrement" page_size: 5 page_token_option: type: RequestOption option_type: "request_parameter" field_name: "page"
Attributes:
- page_size_option (Optional[RequestOption]): the request option to set the page size. Cannot be injected in the path.
- page_token_option (Optional[RequestPath, RequestOption]): the request option to set the page token
- pagination_strategy (PaginationStrategy): Strategy defining how to get the next page token
- config (Config): connection config
- url_base (Union[InterpolatedString, str]): endpoint's base url
- decoder (Decoder): decoder to decode the response
132 def get_initial_token(self) -> Optional[Any]: 133 """ 134 Return the page token that should be used for the first request of a stream 135 136 WARNING: get_initial_token() should not be used by streams that use RFR that perform checkpointing 137 of state using page numbers. Because paginators are stateless 138 """ 139 return self.pagination_strategy.initial_token
Return the page token that should be used for the first request of a stream
WARNING: get_initial_token() should not be used by streams that use RFR that perform checkpointing of state using page numbers. Because paginators are stateless
Evaluated against the config alone. A pagination strategy whose page_size template references the
response evaluates it with the response in context inside next_page_token, so the two can disagree -
pre-existing, and only observable for a page_size that is not a constant.
Returns
the number of records this paginator asks for per page, or None if it does not define one
144 def next_page_token( 145 self, 146 response: requests.Response, 147 last_page_size: int, 148 last_record: Optional[Record], 149 last_page_token_value: Optional[Any] = None, 150 page_size_override: Optional[int] = None, 151 stream_slice: Optional[StreamSlice] = None, 152 ) -> Optional[Mapping[str, Any]]: 153 next_page_token = self.pagination_strategy.next_page_token( 154 response=response, 155 last_page_size=last_page_size, 156 last_record=last_record, 157 last_page_token_value=last_page_token_value, 158 **page_size_override_kwargs(page_size_override), 159 **stream_slice_kwargs(self.pagination_strategy.next_page_token, stream_slice), 160 ) 161 if next_page_token: 162 return {"next_page_token": next_page_token} 163 else: 164 return None
Returns the next_page_token to use to fetch the next page of records.
Parameters
- response: the response to process
- last_page_size: the number of records read from the response
- last_record: the last record extracted from the response
- last_page_token_value: The current value of the page token made on the last request
- page_size_override: the page size that was actually requested, when it differs from the configured
one because of a
REDUCE_PAGE_SIZEresponse action - stream_slice: the slice the page was read for, so that a stop condition can compare the page against the slice's own window or partition
Returns
A mapping {"next_page_token":
} for the next page from the input response object. Returning None means there are no more pages to read in this response.
166 def path( 167 self, 168 next_page_token: Optional[Mapping[str, Any]], 169 stream_state: Optional[Mapping[str, Any]] = None, 170 stream_slice: Optional[StreamSlice] = None, 171 ) -> Optional[str]: 172 token = next_page_token.get("next_page_token") if next_page_token else None 173 if token and self.page_token_option and isinstance(self.page_token_option, RequestPath): 174 return str(token) 175 else: 176 return None
Returns the URL path to hit to fetch the next page of records
e.g: if you wanted to hit https://myapi.com/v1/some_entity then this will return "some_entity"
Returns
path to hit to fetch the next request. Returning None means the path is not defined by the next_page_token
178 def get_request_params( 179 self, 180 *, 181 stream_state: Optional[StreamState] = None, 182 stream_slice: Optional[StreamSlice] = None, 183 next_page_token: Optional[Mapping[str, Any]] = None, 184 page_size_override: Optional[int] = None, 185 ) -> MutableMapping[str, Any]: 186 return self._get_request_options( 187 RequestOptionType.request_parameter, next_page_token, page_size_override 188 )
Specifies the query parameters that should be set on an outgoing HTTP request given the inputs.
E.g: you might want to define query parameters for paging if next_page_token is not None.
190 def get_request_headers( 191 self, 192 *, 193 stream_state: Optional[StreamState] = None, 194 stream_slice: Optional[StreamSlice] = None, 195 next_page_token: Optional[Mapping[str, Any]] = None, 196 page_size_override: Optional[int] = None, 197 ) -> Mapping[str, str]: 198 return self._get_request_options( 199 RequestOptionType.header, next_page_token, page_size_override 200 )
Return any non-auth headers. Authentication headers will overwrite any overlapping headers returned from this method.
202 def get_request_body_data( 203 self, 204 *, 205 stream_state: Optional[StreamState] = None, 206 stream_slice: Optional[StreamSlice] = None, 207 next_page_token: Optional[Mapping[str, Any]] = None, 208 page_size_override: Optional[int] = None, 209 ) -> Mapping[str, Any]: 210 return self._get_request_options( 211 RequestOptionType.body_data, next_page_token, page_size_override 212 )
Specifies how to populate the body of the request with a non-JSON payload.
If returns a ready text that it will be sent as is. If returns a dict that it will be converted to a urlencoded form. E.g. {"key1": "value1", "key2": "value2"} => "key1=value1&key2=value2"
At the same time only one of the 'request_body_data' and 'request_body_json' functions can be overridden.
214 def get_request_body_json( 215 self, 216 *, 217 stream_state: Optional[StreamState] = None, 218 stream_slice: Optional[StreamSlice] = None, 219 next_page_token: Optional[Mapping[str, Any]] = None, 220 page_size_override: Optional[int] = None, 221 ) -> Mapping[str, Any]: 222 return self._get_request_options( 223 RequestOptionType.body_json, next_page_token, page_size_override 224 )
Specifies how to populate the body of the request with a JSON payload.
At the same time only one of the 'request_body_data' and 'request_body_json' functions can be overridden.
15@dataclass 16class DefaultRequestOptionsProvider(RequestOptionsProvider): 17 """ 18 Request options provider that extracts fields from the stream_slice and injects them into the respective location in the 19 outbound request being made 20 """ 21 22 parameters: InitVar[Mapping[str, Any]] 23 24 def __post_init__(self, parameters: Mapping[str, Any]) -> None: 25 pass 26 27 def get_request_params( 28 self, 29 *, 30 stream_state: Optional[StreamState] = None, 31 stream_slice: Optional[StreamSlice] = None, 32 next_page_token: Optional[Mapping[str, Any]] = None, 33 ) -> Mapping[str, Any]: 34 return {} 35 36 def get_request_headers( 37 self, 38 *, 39 stream_state: Optional[StreamState] = None, 40 stream_slice: Optional[StreamSlice] = None, 41 next_page_token: Optional[Mapping[str, Any]] = None, 42 ) -> Mapping[str, Any]: 43 return {} 44 45 def get_request_body_data( 46 self, 47 *, 48 stream_state: Optional[StreamState] = None, 49 stream_slice: Optional[StreamSlice] = None, 50 next_page_token: Optional[Mapping[str, Any]] = None, 51 ) -> Union[Mapping[str, Any], str]: 52 return {} 53 54 def get_request_body_json( 55 self, 56 *, 57 stream_state: Optional[StreamState] = None, 58 stream_slice: Optional[StreamSlice] = None, 59 next_page_token: Optional[Mapping[str, Any]] = None, 60 ) -> Mapping[str, Any]: 61 return {}
Request options provider that extracts fields from the stream_slice and injects them into the respective location in the outbound request being made
27 def get_request_params( 28 self, 29 *, 30 stream_state: Optional[StreamState] = None, 31 stream_slice: Optional[StreamSlice] = None, 32 next_page_token: Optional[Mapping[str, Any]] = None, 33 ) -> Mapping[str, Any]: 34 return {}
Specifies the query parameters that should be set on an outgoing HTTP request given the inputs.
E.g: you might want to define query parameters for paging if next_page_token is not None.
36 def get_request_headers( 37 self, 38 *, 39 stream_state: Optional[StreamState] = None, 40 stream_slice: Optional[StreamSlice] = None, 41 next_page_token: Optional[Mapping[str, Any]] = None, 42 ) -> Mapping[str, Any]: 43 return {}
Return any non-auth headers. Authentication headers will overwrite any overlapping headers returned from this method.
45 def get_request_body_data( 46 self, 47 *, 48 stream_state: Optional[StreamState] = None, 49 stream_slice: Optional[StreamSlice] = None, 50 next_page_token: Optional[Mapping[str, Any]] = None, 51 ) -> Union[Mapping[str, Any], str]: 52 return {}
Specifies how to populate the body of the request with a non-JSON payload.
If returns a ready text that it will be sent as is. If returns a dict that it will be converted to a urlencoded form. E.g. {"key1": "value1", "key2": "value2"} => "key1=value1&key2=value2"
At the same time only one of the 'request_body_data' and 'request_body_json' functions can be overridden.
54 def get_request_body_json( 55 self, 56 *, 57 stream_state: Optional[StreamState] = None, 58 stream_slice: Optional[StreamSlice] = None, 59 next_page_token: Optional[Mapping[str, Any]] = None, 60 ) -> Mapping[str, Any]: 61 return {}
Specifies how to populate the body of the request with a JSON payload.
At the same time only one of the 'request_body_data' and 'request_body_json' functions can be overridden.
19@dataclass 20class DpathExtractor(RecordExtractor): 21 """ 22 Record extractor that searches a decoded response over a path defined as an array of fields. 23 24 If the field path points to an array, that array is returned. 25 If the field path points to an object, that object is returned wrapped as an array. 26 If the field path points to an empty object, an empty array is returned. 27 If the field path points to a non-existing path, an empty array is returned. 28 29 Optionally, records can be expanded by providing a RecordExpander component. 30 When record_expander is configured, each extracted record is passed through the 31 expander which extracts items from nested array fields and emits each item as a 32 separate record. 33 34 Examples of instantiating this transform: 35 ``` 36 extractor: 37 type: DpathExtractor 38 field_path: 39 - "root" 40 - "data" 41 ``` 42 43 ``` 44 extractor: 45 type: DpathExtractor 46 field_path: 47 - "root" 48 - "{{ parameters['field'] }}" 49 ``` 50 51 ``` 52 extractor: 53 type: DpathExtractor 54 field_path: [] 55 ``` 56 57 ``` 58 extractor: 59 type: DpathExtractor 60 field_path: 61 - "data" 62 - "object" 63 record_expander: 64 type: RecordExpander 65 expand_records_from_field: 66 - "lines" 67 - "data" 68 remain_original_record: true 69 ``` 70 71 Attributes: 72 field_path (Union[InterpolatedString, str]): Path to the field that should be extracted 73 config (Config): The user-provided configuration as specified by the source's spec 74 decoder (Decoder): The decoder responsible to transfom the response in a Mapping 75 record_expander (Optional[RecordExpander]): Optional component to expand records by extracting items from nested array fields 76 """ 77 78 field_path: List[Union[InterpolatedString, str]] 79 config: Config 80 parameters: InitVar[Mapping[str, Any]] 81 decoder: Decoder = field(default_factory=lambda: JsonDecoder(parameters={})) 82 record_expander: Optional[RecordExpander] = None 83 84 def __post_init__(self, parameters: Mapping[str, Any]) -> None: 85 self._field_path = [ 86 InterpolatedString.create(path, parameters=parameters) for path in self.field_path 87 ] 88 for path_index in range(len(self.field_path)): 89 if isinstance(self.field_path[path_index], str): 90 self._field_path[path_index] = InterpolatedString.create( 91 self.field_path[path_index], parameters=parameters 92 ) 93 94 def extract_records(self, response: requests.Response) -> Iterable[MutableMapping[Any, Any]]: 95 for body in self.decoder.decode(response): 96 if len(self._field_path) == 0: 97 extracted = body 98 else: 99 path = [path.eval(self.config) for path in self._field_path] 100 if "*" in path: 101 extracted = dpath.values(body, path) 102 else: 103 extracted = dpath.get(body, path, default=[]) # type: ignore # extracted will be a MutableMapping, given input data structure 104 if isinstance(extracted, list): 105 if not self.record_expander: 106 yield from extracted 107 else: 108 for record in extracted: 109 yield from self.record_expander.expand_record(record) 110 elif extracted: 111 if self.record_expander: 112 yield from self.record_expander.expand_record(extracted) 113 else: 114 yield extracted 115 else: 116 yield from []
Record extractor that searches a decoded response over a path defined as an array of fields.
If the field path points to an array, that array is returned. If the field path points to an object, that object is returned wrapped as an array. If the field path points to an empty object, an empty array is returned. If the field path points to a non-existing path, an empty array is returned.
Optionally, records can be expanded by providing a RecordExpander component. When record_expander is configured, each extracted record is passed through the expander which extracts items from nested array fields and emits each item as a separate record.
Examples of instantiating this transform:
extractor:
type: DpathExtractor
field_path:
- "root"
- "data"
extractor:
type: DpathExtractor
field_path:
- "root"
- "{{ parameters['field'] }}"
extractor:
type: DpathExtractor
field_path: []
extractor:
type: DpathExtractor
field_path:
- "data"
- "object"
record_expander:
type: RecordExpander
expand_records_from_field:
- "lines"
- "data"
remain_original_record: true
Attributes:
- field_path (Union[InterpolatedString, str]): Path to the field that should be extracted
- config (Config): The user-provided configuration as specified by the source's spec
- decoder (Decoder): The decoder responsible to transfom the response in a Mapping
- record_expander (Optional[RecordExpander]): Optional component to expand records by extracting items from nested array fields
94 def extract_records(self, response: requests.Response) -> Iterable[MutableMapping[Any, Any]]: 95 for body in self.decoder.decode(response): 96 if len(self._field_path) == 0: 97 extracted = body 98 else: 99 path = [path.eval(self.config) for path in self._field_path] 100 if "*" in path: 101 extracted = dpath.values(body, path) 102 else: 103 extracted = dpath.get(body, path, default=[]) # type: ignore # extracted will be a MutableMapping, given input data structure 104 if isinstance(extracted, list): 105 if not self.record_expander: 106 yield from extracted 107 else: 108 for record in extracted: 109 yield from self.record_expander.expand_record(record) 110 elif extracted: 111 if self.record_expander: 112 yield from self.record_expander.expand_record(extracted) 113 else: 114 yield extracted 115 else: 116 yield from []
Selects records from the response
Parameters
- response: The response to extract the records from
Returns
List of Records extracted from the response
19class HttpMethod(Enum): 20 """ 21 Http Method to use when submitting an outgoing HTTP request 22 """ 23 24 DELETE = "DELETE" 25 GET = "GET" 26 PATCH = "PATCH" 27 POST = "POST"
Http Method to use when submitting an outgoing HTTP request
37@dataclass 38class HttpRequester(Requester): 39 """ 40 Default implementation of a Requester 41 42 Attributes: 43 name (str): Name of the stream. Only used for request/response caching 44 url_base (Union[InterpolatedString, str]): Base url to send requests to 45 path (Union[InterpolatedString, str]): Path to send requests to 46 http_method (Union[str, HttpMethod]): HTTP method to use when sending requests 47 request_options_provider (Optional[InterpolatedRequestOptionsProvider]): request option provider defining the options to set on outgoing requests 48 authenticator (DeclarativeAuthenticator): Authenticator defining how to authenticate to the source 49 error_handler (Optional[ErrorHandler]): Error handler defining how to detect and handle errors 50 backoff_strategies (Optional[List[BackoffStrategy]]): List of backoff strategies to use when retrying requests 51 config (Config): The user-provided configuration as specified by the source's spec 52 use_cache (bool): Indicates that data should be cached for this stream 53 """ 54 55 name: str 56 config: Config 57 parameters: InitVar[Mapping[str, Any]] 58 59 url: Optional[Union[InterpolatedString, str]] = None 60 url_base: Optional[Union[InterpolatedString, str]] = None 61 path: Optional[Union[InterpolatedString, str]] = None 62 authenticator: Optional[DeclarativeAuthenticator] = None 63 http_method: Union[str, HttpMethod] = HttpMethod.GET 64 request_options_provider: Optional[InterpolatedRequestOptionsProvider] = None 65 error_handler: Optional[ErrorHandler] = None 66 api_budget: Optional[APIBudget] = None 67 disable_retries: bool = False 68 message_repository: MessageRepository = NoopMessageRepository() 69 use_cache: bool = False 70 connect_timeout_in_seconds: Optional[float] = None 71 read_timeout_in_seconds: Optional[float] = None 72 use_tcp_keepalive: bool = False 73 _exit_on_rate_limit: bool = False 74 stream_response: bool = False 75 decoder: Decoder = field(default_factory=lambda: JsonDecoder(parameters={})) 76 77 def __post_init__(self, parameters: Mapping[str, Any]) -> None: 78 self._url = InterpolatedString.create( 79 self.url if self.url else EmptyString, parameters=parameters 80 ) 81 # deprecated 82 self._url_base = InterpolatedString.create( 83 self.url_base if self.url_base else EmptyString, parameters=parameters 84 ) 85 # deprecated 86 self._path = InterpolatedString.create( 87 self.path if self.path else EmptyString, parameters=parameters 88 ) 89 if self.request_options_provider is None: 90 self._request_options_provider = InterpolatedRequestOptionsProvider( 91 config=self.config, parameters=parameters 92 ) 93 elif isinstance(self.request_options_provider, dict): 94 self._request_options_provider = InterpolatedRequestOptionsProvider( 95 config=self.config, **self.request_options_provider 96 ) 97 else: 98 self._request_options_provider = self.request_options_provider 99 self._authenticator = self.authenticator or NoAuth(parameters=parameters) 100 self._http_method = ( 101 HttpMethod[self.http_method] if isinstance(self.http_method, str) else self.http_method 102 ) 103 self.error_handler = self.error_handler 104 self._parameters = parameters 105 106 if self.error_handler is not None and hasattr(self.error_handler, "backoff_strategies"): 107 backoff_strategies = self.error_handler.backoff_strategies # type: ignore 108 else: 109 backoff_strategies = None 110 111 self._http_client = HttpClient( 112 name=self.name, 113 logger=self.logger, 114 error_handler=self.error_handler, 115 api_budget=self.api_budget, 116 authenticator=self._authenticator, 117 use_cache=self.use_cache, 118 backoff_strategy=backoff_strategies, 119 disable_retries=self.disable_retries, 120 message_repository=self.message_repository, 121 request_timeout=( 122 (self.connect_timeout_in_seconds, self.read_timeout_in_seconds) 123 if self.connect_timeout_in_seconds is not None 124 or self.read_timeout_in_seconds is not None 125 else None 126 ), 127 use_tcp_keepalive=self.use_tcp_keepalive, 128 ) 129 130 @property 131 def exit_on_rate_limit(self) -> bool: 132 return self._exit_on_rate_limit 133 134 @exit_on_rate_limit.setter 135 def exit_on_rate_limit(self, value: bool) -> None: 136 self._exit_on_rate_limit = value 137 138 def get_authenticator(self) -> DeclarativeAuthenticator: 139 return self._authenticator 140 141 def get_url( 142 self, 143 *, 144 stream_state: Optional[StreamState] = None, 145 stream_slice: Optional[StreamSlice] = None, 146 next_page_token: Optional[Mapping[str, Any]] = None, 147 ) -> str: 148 interpolation_context = get_interpolation_context( 149 stream_state=stream_state, 150 stream_slice=stream_slice, 151 next_page_token=next_page_token, 152 ) 153 154 return str(self._url.eval(self.config, **interpolation_context)) 155 156 def _get_url( 157 self, 158 *, 159 path: Optional[str] = None, 160 stream_state: Optional[StreamState] = None, 161 stream_slice: Optional[StreamSlice] = None, 162 next_page_token: Optional[Mapping[str, Any]] = None, 163 ) -> str: 164 url = self.get_url( 165 stream_state=stream_state, 166 stream_slice=stream_slice, 167 next_page_token=next_page_token, 168 ) 169 170 url_base = self.get_url_base( 171 stream_state=stream_state, 172 stream_slice=stream_slice, 173 next_page_token=next_page_token, 174 ) 175 176 path = path or self.get_path( 177 stream_state=stream_state, 178 stream_slice=stream_slice, 179 next_page_token=next_page_token, 180 ) 181 182 full_url = ( 183 self._join_url(url_base, path) 184 if url_base 185 else self._join_url(url, path) 186 if path 187 else url 188 ) 189 190 return full_url 191 192 def get_url_base( 193 self, 194 *, 195 stream_state: Optional[StreamState] = None, 196 stream_slice: Optional[StreamSlice] = None, 197 next_page_token: Optional[Mapping[str, Any]] = None, 198 ) -> str: 199 interpolation_context = get_interpolation_context( 200 stream_state=stream_state, 201 stream_slice=stream_slice, 202 next_page_token=next_page_token, 203 ) 204 return str(self._url_base.eval(self.config, **interpolation_context)) 205 206 def get_path( 207 self, 208 *, 209 stream_state: Optional[StreamState] = None, 210 stream_slice: Optional[StreamSlice] = None, 211 next_page_token: Optional[Mapping[str, Any]] = None, 212 ) -> str: 213 interpolation_context = get_interpolation_context( 214 stream_state=stream_state, 215 stream_slice=stream_slice, 216 next_page_token=next_page_token, 217 ) 218 path = str(self._path.eval(self.config, **interpolation_context)) 219 return path.lstrip("/") 220 221 def get_method(self) -> HttpMethod: 222 return self._http_method 223 224 def get_request_params( 225 self, 226 *, 227 stream_state: Optional[StreamState] = None, 228 stream_slice: Optional[StreamSlice] = None, 229 next_page_token: Optional[Mapping[str, Any]] = None, 230 ) -> MutableMapping[str, Any]: 231 return self._request_options_provider.get_request_params( 232 stream_state=stream_state, 233 stream_slice=stream_slice, 234 next_page_token=next_page_token, 235 ) 236 237 def get_request_headers( 238 self, 239 *, 240 stream_state: Optional[StreamState] = None, 241 stream_slice: Optional[StreamSlice] = None, 242 next_page_token: Optional[Mapping[str, Any]] = None, 243 ) -> Mapping[str, Any]: 244 return self._request_options_provider.get_request_headers( 245 stream_state=stream_state, 246 stream_slice=stream_slice, 247 next_page_token=next_page_token, 248 ) 249 250 # fixing request options provider types has a lot of dependencies 251 def get_request_body_data( # type: ignore 252 self, 253 *, 254 stream_state: Optional[StreamState] = None, 255 stream_slice: Optional[StreamSlice] = None, 256 next_page_token: Optional[Mapping[str, Any]] = None, 257 ) -> Union[Mapping[str, Any], str]: 258 return ( 259 self._request_options_provider.get_request_body_data( 260 stream_state=stream_state, 261 stream_slice=stream_slice, 262 next_page_token=next_page_token, 263 ) 264 or {} 265 ) 266 267 # fixing request options provider types has a lot of dependencies 268 def get_request_body_json( # type: ignore 269 self, 270 *, 271 stream_state: Optional[StreamState] = None, 272 stream_slice: Optional[StreamSlice] = None, 273 next_page_token: Optional[Mapping[str, Any]] = None, 274 ) -> Optional[Mapping[str, Any]]: 275 return self._request_options_provider.get_request_body_json( 276 stream_state=stream_state, 277 stream_slice=stream_slice, 278 next_page_token=next_page_token, 279 ) 280 281 @property 282 def logger(self) -> logging.Logger: 283 return logging.getLogger(f"airbyte.HttpRequester.{self.name}") 284 285 def _get_request_options( 286 self, 287 stream_state: Optional[StreamState], 288 stream_slice: Optional[StreamSlice], 289 next_page_token: Optional[Mapping[str, Any]], 290 requester_method: Callable[..., Optional[Union[Mapping[str, Any], str]]], 291 auth_options_method: Callable[..., Optional[Union[Mapping[str, Any], str]]], 292 extra_options: Optional[Union[Mapping[str, Any], str]] = None, 293 ) -> Union[Mapping[str, Any], str]: 294 """ 295 Get the request_option from the requester, the authenticator and extra_options passed in. 296 Raise a ValueError if there's a key collision 297 Returned merged mapping otherwise 298 """ 299 300 is_body_json = requester_method.__name__ == "get_request_body_json" 301 302 return combine_mappings( 303 [ 304 requester_method( 305 stream_state=stream_state, 306 stream_slice=stream_slice, 307 next_page_token=next_page_token, 308 ), 309 auth_options_method(), 310 extra_options, 311 ], 312 allow_same_value_merge=is_body_json, 313 ) 314 315 def _request_headers( 316 self, 317 stream_state: Optional[StreamState] = None, 318 stream_slice: Optional[StreamSlice] = None, 319 next_page_token: Optional[Mapping[str, Any]] = None, 320 extra_headers: Optional[Mapping[str, Any]] = None, 321 ) -> Mapping[str, Any]: 322 """ 323 Specifies request headers. 324 Authentication headers will overwrite any overlapping headers returned from this method. 325 """ 326 headers = self._get_request_options( 327 stream_state, 328 stream_slice, 329 next_page_token, 330 self.get_request_headers, 331 self.get_authenticator().get_auth_header, 332 extra_headers, 333 ) 334 if isinstance(headers, str): 335 raise ValueError("Request headers cannot be a string") 336 return {str(k): str(v) for k, v in headers.items()} 337 338 def _request_params( 339 self, 340 stream_state: Optional[StreamState], 341 stream_slice: Optional[StreamSlice], 342 next_page_token: Optional[Mapping[str, Any]], 343 extra_params: Optional[Mapping[str, Any]] = None, 344 ) -> Mapping[str, Any]: 345 """ 346 Specifies the query parameters that should be set on an outgoing HTTP request given the inputs. 347 348 E.g: you might want to define query parameters for paging if next_page_token is not None. 349 """ 350 options = self._get_request_options( 351 stream_state, 352 stream_slice, 353 next_page_token, 354 self.get_request_params, 355 self.get_authenticator().get_request_params, 356 extra_params, 357 ) 358 if isinstance(options, str): 359 raise ValueError("Request params cannot be a string") 360 361 for k, v in options.items(): 362 if isinstance(v, (dict,)): 363 raise ValueError( 364 f"Invalid value for `{k}` parameter. The values of request params cannot be an object." 365 ) 366 367 return options 368 369 def _request_body_data( 370 self, 371 stream_state: Optional[StreamState], 372 stream_slice: Optional[StreamSlice], 373 next_page_token: Optional[Mapping[str, Any]], 374 extra_body_data: Optional[Union[Mapping[str, Any], str]] = None, 375 ) -> Optional[Union[Mapping[str, Any], str]]: 376 """ 377 Specifies how to populate the body of the request with a non-JSON payload. 378 379 If returns a ready text that it will be sent as is. 380 If returns a dict that it will be converted to a urlencoded form. 381 E.g. {"key1": "value1", "key2": "value2"} => "key1=value1&key2=value2" 382 383 At the same time only one of the 'request_body_data' and 'request_body_json' functions can be overridden. 384 """ 385 # Warning: use self.state instead of the stream_state passed as argument! 386 return self._get_request_options( 387 stream_state, 388 stream_slice, 389 next_page_token, 390 self.get_request_body_data, 391 self.get_authenticator().get_request_body_data, 392 extra_body_data, 393 ) 394 395 def _request_body_json( 396 self, 397 stream_state: Optional[StreamState], 398 stream_slice: Optional[StreamSlice], 399 next_page_token: Optional[Mapping[str, Any]], 400 extra_body_json: Optional[Mapping[str, Any]] = None, 401 ) -> Optional[Mapping[str, Any]]: 402 """ 403 Specifies how to populate the body of the request with a JSON payload. 404 405 At the same time only one of the 'request_body_data' and 'request_body_json' functions can be overridden. 406 """ 407 # Warning: use self.state instead of the stream_state passed as argument! 408 options = self._get_request_options( 409 stream_state, 410 stream_slice, 411 next_page_token, 412 self.get_request_body_json, 413 self.get_authenticator().get_request_body_json, 414 extra_body_json, 415 ) 416 if isinstance(options, str): 417 raise ValueError("Request body json cannot be a string") 418 return options 419 420 @classmethod 421 def _join_url(cls, url_base: str, path: Optional[str] = None) -> str: 422 """ 423 Joins a base URL with a given path and returns the resulting URL with any trailing slash removed. 424 425 This method ensures that there are no duplicate slashes when concatenating the base URL and the path, 426 which is useful when the full URL is provided from an interpolation context. 427 428 Args: 429 url_base (str): The base URL to which the path will be appended. 430 path (Optional[str]): The path to join with the base URL. 431 432 Returns: 433 str: The resulting joined URL. 434 435 Note: 436 Related issue: https://github.com/airbytehq/airbyte-internal-issues/issues/11869 437 - If the path is an empty string or None, the method returns the base URL with any trailing slash removed. 438 439 Example: 440 1) _join_url("https://example.com/api/", "endpoint") >> 'https://example.com/api/endpoint' 441 2) _join_url("https://example.com/api", "/endpoint") >> 'https://example.com/api/endpoint' 442 3) _join_url("https://example.com/api/", "") >> 'https://example.com/api/' 443 4) _join_url("https://example.com/api", None) >> 'https://example.com/api' 444 """ 445 446 # return a full-url if provided directly from interpolation context 447 if path == EmptyString or path is None: 448 return url_base 449 else: 450 # since we didn't provide a full-url, the url_base might not have a trailing slash 451 # so we join the url_base and path correctly 452 if not url_base.endswith("/"): 453 url_base += "/" 454 455 return urljoin(url_base, path) 456 457 def send_request( 458 self, 459 stream_state: Optional[StreamState] = None, 460 stream_slice: Optional[StreamSlice] = None, 461 next_page_token: Optional[Mapping[str, Any]] = None, 462 path: Optional[str] = None, 463 request_headers: Optional[Mapping[str, Any]] = None, 464 request_params: Optional[Mapping[str, Any]] = None, 465 request_body_data: Optional[Union[Mapping[str, Any], str]] = None, 466 request_body_json: Optional[Mapping[str, Any]] = None, 467 log_formatter: Optional[Callable[[requests.Response], Any]] = None, 468 ) -> Optional[requests.Response]: 469 request, response = self._http_client.send_request( 470 http_method=self.get_method().value, 471 url=self._get_url( 472 path=path, 473 stream_state=stream_state, 474 stream_slice=stream_slice, 475 next_page_token=next_page_token, 476 ), 477 request_kwargs={"stream": self.stream_response}, 478 headers=self._request_headers( 479 stream_state, stream_slice, next_page_token, request_headers 480 ), 481 params=self._request_params( 482 stream_state, stream_slice, next_page_token, request_params 483 ), 484 json=self._request_body_json( 485 stream_state, stream_slice, next_page_token, request_body_json 486 ), 487 data=self._request_body_data( 488 stream_state, stream_slice, next_page_token, request_body_data 489 ), 490 dedupe_query_params=True, 491 log_formatter=log_formatter, 492 exit_on_rate_limit=self._exit_on_rate_limit, 493 ) 494 495 return response
Default implementation of a Requester
Attributes:
- name (str): Name of the stream. Only used for request/response caching
- url_base (Union[InterpolatedString, str]): Base url to send requests to
- path (Union[InterpolatedString, str]): Path to send requests to
- http_method (Union[str, HttpMethod]): HTTP method to use when sending requests
- request_options_provider (Optional[InterpolatedRequestOptionsProvider]): request option provider defining the options to set on outgoing requests
- authenticator (DeclarativeAuthenticator): Authenticator defining how to authenticate to the source
- error_handler (Optional[ErrorHandler]): Error handler defining how to detect and handle errors
- backoff_strategies (Optional[List[BackoffStrategy]]): List of backoff strategies to use when retrying requests
- config (Config): The user-provided configuration as specified by the source's spec
- use_cache (bool): Indicates that data should be cached for this stream
141 def get_url( 142 self, 143 *, 144 stream_state: Optional[StreamState] = None, 145 stream_slice: Optional[StreamSlice] = None, 146 next_page_token: Optional[Mapping[str, Any]] = None, 147 ) -> str: 148 interpolation_context = get_interpolation_context( 149 stream_state=stream_state, 150 stream_slice=stream_slice, 151 next_page_token=next_page_token, 152 ) 153 154 return str(self._url.eval(self.config, **interpolation_context))
Returns
URL base for the API endpoint e.g: if you wanted to hit https://myapi.com/v1/some_entity then this should return "https://myapi.com/v1/"
192 def get_url_base( 193 self, 194 *, 195 stream_state: Optional[StreamState] = None, 196 stream_slice: Optional[StreamSlice] = None, 197 next_page_token: Optional[Mapping[str, Any]] = None, 198 ) -> str: 199 interpolation_context = get_interpolation_context( 200 stream_state=stream_state, 201 stream_slice=stream_slice, 202 next_page_token=next_page_token, 203 ) 204 return str(self._url_base.eval(self.config, **interpolation_context))
Returns
URL base for the API endpoint e.g: if you wanted to hit https://myapi.com/v1/some_entity then this should return "https://myapi.com/v1/"
206 def get_path( 207 self, 208 *, 209 stream_state: Optional[StreamState] = None, 210 stream_slice: Optional[StreamSlice] = None, 211 next_page_token: Optional[Mapping[str, Any]] = None, 212 ) -> str: 213 interpolation_context = get_interpolation_context( 214 stream_state=stream_state, 215 stream_slice=stream_slice, 216 next_page_token=next_page_token, 217 ) 218 path = str(self._path.eval(self.config, **interpolation_context)) 219 return path.lstrip("/")
Returns the URL path for the API endpoint e.g: if you wanted to hit https://myapi.com/v1/some_entity then this should return "some_entity"
224 def get_request_params( 225 self, 226 *, 227 stream_state: Optional[StreamState] = None, 228 stream_slice: Optional[StreamSlice] = None, 229 next_page_token: Optional[Mapping[str, Any]] = None, 230 ) -> MutableMapping[str, Any]: 231 return self._request_options_provider.get_request_params( 232 stream_state=stream_state, 233 stream_slice=stream_slice, 234 next_page_token=next_page_token, 235 )
Specifies the query parameters that should be set on an outgoing HTTP request given the inputs.
E.g: you might want to define query parameters for paging if next_page_token is not None.
237 def get_request_headers( 238 self, 239 *, 240 stream_state: Optional[StreamState] = None, 241 stream_slice: Optional[StreamSlice] = None, 242 next_page_token: Optional[Mapping[str, Any]] = None, 243 ) -> Mapping[str, Any]: 244 return self._request_options_provider.get_request_headers( 245 stream_state=stream_state, 246 stream_slice=stream_slice, 247 next_page_token=next_page_token, 248 )
Return any non-auth headers. Authentication headers will overwrite any overlapping headers returned from this method.
251 def get_request_body_data( # type: ignore 252 self, 253 *, 254 stream_state: Optional[StreamState] = None, 255 stream_slice: Optional[StreamSlice] = None, 256 next_page_token: Optional[Mapping[str, Any]] = None, 257 ) -> Union[Mapping[str, Any], str]: 258 return ( 259 self._request_options_provider.get_request_body_data( 260 stream_state=stream_state, 261 stream_slice=stream_slice, 262 next_page_token=next_page_token, 263 ) 264 or {} 265 )
Specifies how to populate the body of the request with a non-JSON payload.
If returns a ready text that it will be sent as is. If returns a dict that it will be converted to a urlencoded form. E.g. {"key1": "value1", "key2": "value2"} => "key1=value1&key2=value2"
At the same time only one of the 'request_body_data' and 'request_body_json' functions can be overridden.
268 def get_request_body_json( # type: ignore 269 self, 270 *, 271 stream_state: Optional[StreamState] = None, 272 stream_slice: Optional[StreamSlice] = None, 273 next_page_token: Optional[Mapping[str, Any]] = None, 274 ) -> Optional[Mapping[str, Any]]: 275 return self._request_options_provider.get_request_body_json( 276 stream_state=stream_state, 277 stream_slice=stream_slice, 278 next_page_token=next_page_token, 279 )
Specifies how to populate the body of the request with a JSON payload.
At the same time only one of the 'request_body_data' and 'request_body_json' functions can be overridden.
457 def send_request( 458 self, 459 stream_state: Optional[StreamState] = None, 460 stream_slice: Optional[StreamSlice] = None, 461 next_page_token: Optional[Mapping[str, Any]] = None, 462 path: Optional[str] = None, 463 request_headers: Optional[Mapping[str, Any]] = None, 464 request_params: Optional[Mapping[str, Any]] = None, 465 request_body_data: Optional[Union[Mapping[str, Any], str]] = None, 466 request_body_json: Optional[Mapping[str, Any]] = None, 467 log_formatter: Optional[Callable[[requests.Response], Any]] = None, 468 ) -> Optional[requests.Response]: 469 request, response = self._http_client.send_request( 470 http_method=self.get_method().value, 471 url=self._get_url( 472 path=path, 473 stream_state=stream_state, 474 stream_slice=stream_slice, 475 next_page_token=next_page_token, 476 ), 477 request_kwargs={"stream": self.stream_response}, 478 headers=self._request_headers( 479 stream_state, stream_slice, next_page_token, request_headers 480 ), 481 params=self._request_params( 482 stream_state, stream_slice, next_page_token, request_params 483 ), 484 json=self._request_body_json( 485 stream_state, stream_slice, next_page_token, request_body_json 486 ), 487 data=self._request_body_data( 488 stream_state, stream_slice, next_page_token, request_body_data 489 ), 490 dedupe_query_params=True, 491 log_formatter=log_formatter, 492 exit_on_rate_limit=self._exit_on_rate_limit, 493 ) 494 495 return response
Sends a request and returns the response. Might return no response if the error handler chooses to ignore the response or throw an exception in case of an error. If path is set, the path configured on the requester itself is ignored. If header, params and body are set, they are merged with the ones configured on the requester itself.
If a log formatter is provided, it's used to log the performed request and response. If it's not provided, no logging is performed.
30@dataclass 31class InterpolatedBoolean: 32 f""" 33 Wrapper around a string to be evaluated to a boolean value. 34 The string will be evaluated as False if it interpolates to a value in {FALSE_VALUES} 35 36 Attributes: 37 condition (str): The string representing the condition to evaluate to a boolean 38 """ 39 condition: str 40 parameters: InitVar[Mapping[str, Any]] 41 42 def __post_init__(self, parameters: Mapping[str, Any]) -> None: 43 self._default = "False" 44 self._interpolation = JinjaInterpolation() 45 self._parameters = parameters 46 47 def eval(self, config: Config, **additional_parameters: Any) -> bool: 48 """ 49 Interpolates the predicate condition string using the config and other optional arguments passed as parameter. 50 51 :param config: The user-provided configuration as specified by the source's spec 52 :param additional_parameters: Optional parameters used for interpolation 53 :return: The interpolated string 54 """ 55 if isinstance(self.condition, bool): 56 return self.condition 57 else: 58 evaluated = self._interpolation.eval( 59 self.condition, 60 config, 61 self._default, 62 parameters=self._parameters, 63 **additional_parameters, 64 ) 65 if evaluated in FALSE_VALUES: 66 return False 67 # The presence of a value is generally regarded as truthy, so we treat it as such 68 return True
47 def eval(self, config: Config, **additional_parameters: Any) -> bool: 48 """ 49 Interpolates the predicate condition string using the config and other optional arguments passed as parameter. 50 51 :param config: The user-provided configuration as specified by the source's spec 52 :param additional_parameters: Optional parameters used for interpolation 53 :return: The interpolated string 54 """ 55 if isinstance(self.condition, bool): 56 return self.condition 57 else: 58 evaluated = self._interpolation.eval( 59 self.condition, 60 config, 61 self._default, 62 parameters=self._parameters, 63 **additional_parameters, 64 ) 65 if evaluated in FALSE_VALUES: 66 return False 67 # The presence of a value is generally regarded as truthy, so we treat it as such 68 return True
Interpolates the predicate condition string using the config and other optional arguments passed as parameter.
Parameters
- config: The user-provided configuration as specified by the source's spec
- additional_parameters: Optional parameters used for interpolation
Returns
The interpolated string
15@dataclass 16class InterpolatedRequestInputProvider: 17 """ 18 Helper class that generically performs string interpolation on the provided dictionary or string input 19 """ 20 21 parameters: InitVar[Mapping[str, Any]] 22 request_inputs: Optional[Union[str, Mapping[str, str]]] = field(default=None) 23 config: Config = field(default_factory=dict) 24 _interpolator: Optional[Union[InterpolatedString, InterpolatedMapping]] = field( 25 init=False, repr=False, default=None 26 ) 27 _request_inputs: Optional[Union[str, Mapping[str, str]]] = field( 28 init=False, repr=False, default=None 29 ) 30 31 def __post_init__(self, parameters: Mapping[str, Any]) -> None: 32 self._request_inputs = self.request_inputs or {} 33 if isinstance(self._request_inputs, str): 34 self._interpolator = InterpolatedString( 35 self._request_inputs, default="", parameters=parameters 36 ) 37 else: 38 self._interpolator = InterpolatedMapping(self._request_inputs, parameters=parameters) 39 40 def eval_request_inputs( 41 self, 42 stream_slice: Optional[StreamSlice] = None, 43 next_page_token: Optional[Mapping[str, Any]] = None, 44 valid_key_types: Optional[Tuple[Type[Any]]] = None, 45 valid_value_types: Optional[Tuple[Type[Any], ...]] = None, 46 ) -> Mapping[str, Any]: 47 """ 48 Returns the request inputs to set on an outgoing HTTP request 49 50 :param stream_slice: The stream slice 51 :param next_page_token: The pagination token 52 :param valid_key_types: A tuple of types that the interpolator should allow 53 :param valid_value_types: A tuple of types that the interpolator should allow 54 :return: The request inputs to set on an outgoing HTTP request 55 """ 56 kwargs = get_interpolation_context( 57 stream_slice=stream_slice, 58 next_page_token=next_page_token, 59 ) 60 interpolated_value = self._interpolator.eval( # type: ignore # self._interpolator is always initialized with a value and will not be None 61 self.config, 62 valid_key_types=valid_key_types, 63 valid_value_types=valid_value_types, 64 **kwargs, 65 ) 66 67 if isinstance(interpolated_value, dict): 68 non_null_tokens = {k: v for k, v in interpolated_value.items() if v is not None} 69 return non_null_tokens 70 return interpolated_value # type: ignore[no-any-return]
Helper class that generically performs string interpolation on the provided dictionary or string input
40 def eval_request_inputs( 41 self, 42 stream_slice: Optional[StreamSlice] = None, 43 next_page_token: Optional[Mapping[str, Any]] = None, 44 valid_key_types: Optional[Tuple[Type[Any]]] = None, 45 valid_value_types: Optional[Tuple[Type[Any], ...]] = None, 46 ) -> Mapping[str, Any]: 47 """ 48 Returns the request inputs to set on an outgoing HTTP request 49 50 :param stream_slice: The stream slice 51 :param next_page_token: The pagination token 52 :param valid_key_types: A tuple of types that the interpolator should allow 53 :param valid_value_types: A tuple of types that the interpolator should allow 54 :return: The request inputs to set on an outgoing HTTP request 55 """ 56 kwargs = get_interpolation_context( 57 stream_slice=stream_slice, 58 next_page_token=next_page_token, 59 ) 60 interpolated_value = self._interpolator.eval( # type: ignore # self._interpolator is always initialized with a value and will not be None 61 self.config, 62 valid_key_types=valid_key_types, 63 valid_value_types=valid_value_types, 64 **kwargs, 65 ) 66 67 if isinstance(interpolated_value, dict): 68 non_null_tokens = {k: v for k, v in interpolated_value.items() if v is not None} 69 return non_null_tokens 70 return interpolated_value # type: ignore[no-any-return]
Returns the request inputs to set on an outgoing HTTP request
Parameters
- stream_slice: The stream slice
- next_page_token: The pagination token
- valid_key_types: A tuple of types that the interpolator should allow
- valid_value_types: A tuple of types that the interpolator should allow
Returns
The request inputs to set on an outgoing HTTP request
13@dataclass 14class InterpolatedString: 15 """ 16 Wrapper around a raw string to be interpolated with the Jinja2 templating engine 17 18 Attributes: 19 string (str): The string to evalute 20 default (Optional[str]): The default value to return if the evaluation returns an empty string 21 parameters (Mapping[str, Any]): Additional runtime parameters to be used for string interpolation 22 """ 23 24 string: str 25 parameters: InitVar[Mapping[str, Any]] 26 default: Optional[str] = None 27 28 def __post_init__(self, parameters: Mapping[str, Any]) -> None: 29 self.default = self.default or self.string 30 self._interpolation = JinjaInterpolation() 31 self._parameters = parameters 32 # indicates whether passed string is just a plain string, not Jinja template 33 # This allows for optimization, but we do not know it yet at this stage 34 self._is_plain_string = None 35 36 def eval(self, config: Config, **kwargs: Any) -> Any: 37 """ 38 Interpolates the input string using the config and other optional arguments passed as parameter. 39 40 :param config: The user-provided configuration as specified by the source's spec 41 :param kwargs: Optional parameters used for interpolation 42 :return: The interpolated string 43 """ 44 if self._is_plain_string: 45 return self.string 46 if self._is_plain_string is None: 47 # Let's check whether output from evaluation is the same as input. 48 # This indicates occurrence of a plain string, not a template and we can skip Jinja in subsequent runs. 49 evaluated = self._interpolation.eval( 50 self.string, config, self.default, parameters=self._parameters, **kwargs 51 ) 52 self._is_plain_string = self.string == evaluated 53 return evaluated 54 return self._interpolation.eval( 55 self.string, config, self.default, parameters=self._parameters, **kwargs 56 ) 57 58 def __eq__(self, other: Any) -> bool: 59 if not isinstance(other, InterpolatedString): 60 return False 61 return self.string == other.string and self.default == other.default 62 63 @classmethod 64 def create( 65 cls, 66 string_or_interpolated: Union["InterpolatedString", str], 67 *, 68 parameters: Mapping[str, Any], 69 ) -> "InterpolatedString": 70 """ 71 Helper function to obtain an InterpolatedString from either a raw string or an InterpolatedString. 72 73 :param string_or_interpolated: either a raw string or an InterpolatedString. 74 :param parameters: parameters propagated from parent component 75 :return: InterpolatedString representing the input string. 76 """ 77 if isinstance(string_or_interpolated, str): 78 return InterpolatedString(string=string_or_interpolated, parameters=parameters) 79 else: 80 return string_or_interpolated
Wrapper around a raw string to be interpolated with the Jinja2 templating engine
Attributes:
- string (str): The string to evalute
- default (Optional[str]): The default value to return if the evaluation returns an empty string
- parameters (Mapping[str, Any]): Additional runtime parameters to be used for string interpolation
36 def eval(self, config: Config, **kwargs: Any) -> Any: 37 """ 38 Interpolates the input string using the config and other optional arguments passed as parameter. 39 40 :param config: The user-provided configuration as specified by the source's spec 41 :param kwargs: Optional parameters used for interpolation 42 :return: The interpolated string 43 """ 44 if self._is_plain_string: 45 return self.string 46 if self._is_plain_string is None: 47 # Let's check whether output from evaluation is the same as input. 48 # This indicates occurrence of a plain string, not a template and we can skip Jinja in subsequent runs. 49 evaluated = self._interpolation.eval( 50 self.string, config, self.default, parameters=self._parameters, **kwargs 51 ) 52 self._is_plain_string = self.string == evaluated 53 return evaluated 54 return self._interpolation.eval( 55 self.string, config, self.default, parameters=self._parameters, **kwargs 56 )
Interpolates the input string using the config and other optional arguments passed as parameter.
Parameters
- config: The user-provided configuration as specified by the source's spec
- kwargs: Optional parameters used for interpolation
Returns
The interpolated string
63 @classmethod 64 def create( 65 cls, 66 string_or_interpolated: Union["InterpolatedString", str], 67 *, 68 parameters: Mapping[str, Any], 69 ) -> "InterpolatedString": 70 """ 71 Helper function to obtain an InterpolatedString from either a raw string or an InterpolatedString. 72 73 :param string_or_interpolated: either a raw string or an InterpolatedString. 74 :param parameters: parameters propagated from parent component 75 :return: InterpolatedString representing the input string. 76 """ 77 if isinstance(string_or_interpolated, str): 78 return InterpolatedString(string=string_or_interpolated, parameters=parameters) 79 else: 80 return string_or_interpolated
Helper function to obtain an InterpolatedString from either a raw string or an InterpolatedString.
Parameters
- string_or_interpolated: either a raw string or an InterpolatedString.
- parameters: parameters propagated from parent component
Returns
InterpolatedString representing the input string.
20class JsonDecoder(Decoder): 21 """ 22 Decoder strategy that returns the json-encoded content of a response, if any. 23 24 Usually, we would try to instantiate the equivalent `CompositeRawDecoder(parser=JsonParser(), stream_response=False)` but there were specific historical behaviors related to the JsonDecoder that we didn't know if we could remove like the fallback on {} in case of errors. 25 """ 26 27 def __init__(self, parameters: Mapping[str, Any]): 28 self._decoder = CompositeRawDecoder(parser=JsonParser(), stream_response=False) 29 30 def is_stream_response(self) -> bool: 31 return self._decoder.is_stream_response() 32 33 def decode( 34 self, response: requests.Response 35 ) -> Generator[MutableMapping[str, Any], None, None]: 36 """ 37 Given the response is an empty string or an emtpy list, the function will return a generator with an empty mapping. 38 """ 39 has_yielded = False 40 try: 41 for element in self._decoder.decode(response): 42 yield element 43 has_yielded = True 44 except Exception: 45 yield {} 46 47 if not has_yielded: 48 yield {}
Decoder strategy that returns the json-encoded content of a response, if any.
Usually, we would try to instantiate the equivalent CompositeRawDecoder(parser=JsonParser(), stream_response=False) but there were specific historical behaviors related to the JsonDecoder that we didn't know if we could remove like the fallback on {} in case of errors.
Set to True if you'd like to use stream=True option in http requester
33 def decode( 34 self, response: requests.Response 35 ) -> Generator[MutableMapping[str, Any], None, None]: 36 """ 37 Given the response is an empty string or an emtpy list, the function will return a generator with an empty mapping. 38 """ 39 has_yielded = False 40 try: 41 for element in self._decoder.decode(response): 42 yield element 43 has_yielded = True 44 except Exception: 45 yield {} 46 47 if not has_yielded: 48 yield {}
Given the response is an empty string or an emtpy list, the function will return a generator with an empty mapping.
33@dataclass 34class JsonFileSchemaLoader(ResourceSchemaLoader, SchemaLoader): 35 """ 36 Loads the schema from a json file 37 38 Attributes: 39 file_path (Union[InterpolatedString, str]): The path to the json file describing the schema 40 name (str): The stream's name 41 config (Config): The user-provided configuration as specified by the source's spec 42 parameters (Mapping[str, Any]): Additional arguments to pass to the string interpolation if needed 43 """ 44 45 config: Config 46 parameters: InitVar[Mapping[str, Any]] 47 file_path: Union[InterpolatedString, str] = field(default="") 48 49 def __post_init__(self, parameters: Mapping[str, Any]) -> None: 50 if not self.file_path: 51 self.file_path = _default_file_path() 52 self.file_path = InterpolatedString.create(self.file_path, parameters=parameters) 53 54 def get_json_schema(self) -> Mapping[str, Any]: 55 # todo: It is worth revisiting if we can replace file_path with just file_name if every schema is in the /schemas directory 56 # this would require that we find a creative solution to store or retrieve source_name in here since the files are mounted there 57 json_schema_path = self._get_json_filepath() 58 resource, schema_path = self.extract_resource_and_schema_path(json_schema_path) 59 raw_json_file = pkgutil.get_data(resource, schema_path) 60 61 if not raw_json_file: 62 raise IOError(f"Cannot find file {json_schema_path}") 63 try: 64 raw_schema = json.loads(raw_json_file) 65 except ValueError as err: 66 raise RuntimeError(f"Invalid JSON file format for file {json_schema_path}") from err 67 self.package_name = resource 68 return self._resolve_schema_references(raw_schema) 69 70 def _get_json_filepath(self) -> Any: 71 return self.file_path.eval(self.config) # type: ignore # file_path is always cast to an interpolated string 72 73 @staticmethod 74 def extract_resource_and_schema_path(json_schema_path: str) -> Tuple[str, str]: 75 """ 76 When the connector is running on a docker container, package_data is accessible from the resource (source_<name>), so we extract 77 the resource from the first part of the schema path and the remaining path is used to find the schema file. This is a slight 78 hack to identify the source name while we are in the airbyte_cdk module. 79 :param json_schema_path: The path to the schema JSON file 80 :return: Tuple of the resource name and the path to the schema file 81 """ 82 split_path = json_schema_path.split("/") 83 84 if split_path[0] == "" or split_path[0] == ".": 85 split_path = split_path[1:] 86 87 if len(split_path) == 0: 88 return "", "" 89 90 if len(split_path) == 1: 91 return "", split_path[0] 92 93 return split_path[0], "/".join(split_path[1:])
Loads the schema from a json file
Attributes:
- file_path (Union[InterpolatedString, str]): The path to the json file describing the schema
- name (str): The stream's name
- config (Config): The user-provided configuration as specified by the source's spec
- parameters (Mapping[str, Any]): Additional arguments to pass to the string interpolation if needed
54 def get_json_schema(self) -> Mapping[str, Any]: 55 # todo: It is worth revisiting if we can replace file_path with just file_name if every schema is in the /schemas directory 56 # this would require that we find a creative solution to store or retrieve source_name in here since the files are mounted there 57 json_schema_path = self._get_json_filepath() 58 resource, schema_path = self.extract_resource_and_schema_path(json_schema_path) 59 raw_json_file = pkgutil.get_data(resource, schema_path) 60 61 if not raw_json_file: 62 raise IOError(f"Cannot find file {json_schema_path}") 63 try: 64 raw_schema = json.loads(raw_json_file) 65 except ValueError as err: 66 raise RuntimeError(f"Invalid JSON file format for file {json_schema_path}") from err 67 self.package_name = resource 68 return self._resolve_schema_references(raw_schema)
Returns a mapping describing the stream's schema
73 @staticmethod 74 def extract_resource_and_schema_path(json_schema_path: str) -> Tuple[str, str]: 75 """ 76 When the connector is running on a docker container, package_data is accessible from the resource (source_<name>), so we extract 77 the resource from the first part of the schema path and the remaining path is used to find the schema file. This is a slight 78 hack to identify the source name while we are in the airbyte_cdk module. 79 :param json_schema_path: The path to the schema JSON file 80 :return: Tuple of the resource name and the path to the schema file 81 """ 82 split_path = json_schema_path.split("/") 83 84 if split_path[0] == "" or split_path[0] == ".": 85 split_path = split_path[1:] 86 87 if len(split_path) == 0: 88 return "", "" 89 90 if len(split_path) == 1: 91 return "", split_path[0] 92 93 return split_path[0], "/".join(split_path[1:])
When the connector is running on a docker container, package_data is accessible from the resource (source_
Parameters
- json_schema_path: The path to the schema JSON file
Returns
Tuple of the resource name and the path to the schema file
Inherited Members
21class LegacyToPerPartitionStateMigration(StateMigration): 22 """ 23 Transforms the input state for per-partitioned streams from the legacy format to the low-code format. 24 The cursor field and partition ID fields are automatically extracted from the stream's DatetimebasedCursor and SubstreamPartitionRouter. 25 26 Example input state: 27 { 28 "13506132": { 29 "last_changed": "2022-12-27T08:34:39+00:00" 30 } 31 Example output state: 32 { 33 "partition": {"id": "13506132"}, 34 "cursor": {"last_changed": "2022-12-27T08:34:39+00:00"} 35 } 36 """ 37 38 def __init__( 39 self, 40 partition_router: Union[ 41 SubstreamPartitionRouter, UnionPartitionRouter, CustomPartitionRouter 42 ], 43 cursor: DatetimeBasedCursor, 44 config: Mapping[str, Any], 45 parameters: Mapping[str, Any], 46 ): 47 self._partition_router = partition_router 48 self._cursor = cursor 49 self._config = config 50 self._parameters = parameters 51 self._partition_key_field = InterpolatedString.create( 52 self._get_partition_field(partition_router), parameters=self._parameters 53 ).eval(self._config) 54 self._cursor_field = InterpolatedString.create( 55 self._cursor.cursor_field, parameters=self._parameters 56 ).eval(self._config) 57 58 def _get_partition_field( 59 self, 60 partition_router: Union[ 61 SubstreamPartitionRouter, UnionPartitionRouter, CustomPartitionRouter 62 ], 63 ) -> str: 64 if isinstance(partition_router, UnionPartitionRouter): 65 return partition_router.partition_field 66 67 parent_stream_config = partition_router.parent_stream_configs[0] # type: ignore # custom partition routers are expected to expose parent_stream_configs 68 69 # Retrieve the partition field with a condition, as properties are returned as a dictionary for custom components. 70 partition_field = ( 71 parent_stream_config.partition_field 72 if isinstance(parent_stream_config, ParentStreamConfig) 73 else parent_stream_config.get("partition_field") # type: ignore # See above comment on why parent_stream_config might be a dict 74 ) 75 76 return partition_field 77 78 def should_migrate(self, stream_state: Mapping[str, Any]) -> bool: 79 if _is_already_migrated(stream_state): 80 return False 81 82 # UnionPartitionRouter has no parent_stream_configs; its partitions are already 83 # normalized to a single partition field so the parent stream check does not apply. 84 if not isinstance(self._partition_router, UnionPartitionRouter): 85 # There is exactly one parent stream 86 number_of_parent_streams = len(self._partition_router.parent_stream_configs) # type: ignore # custom partition will introduce this attribute if needed 87 if number_of_parent_streams != 1: 88 # There should be exactly one parent stream 89 return False 90 """ 91 The expected state format is 92 "<parent_key_id>" : { 93 "<cursor_field>" : "<cursor_value>" 94 } 95 """ 96 if not stream_state: 97 return False 98 for key, value in stream_state.items(): 99 # it is expected the internal value to be a dictionary according to docstring 100 if not isinstance(value, dict): 101 return False 102 keys = list(value.keys()) 103 if len(keys) != 1: 104 # The input partitioned state should only have one key 105 return False 106 if keys[0] != self._cursor_field: 107 # Unexpected key. Found {keys[0]}. Expected {self._cursor.cursor_field} 108 return False 109 110 return True 111 112 def migrate(self, stream_state: Mapping[str, Any]) -> Mapping[str, Any]: 113 states = [ 114 {"partition": {self._partition_key_field: key}, "cursor": value} 115 for key, value in stream_state.items() 116 ] 117 return {"states": states}
Transforms the input state for per-partitioned streams from the legacy format to the low-code format. The cursor field and partition ID fields are automatically extracted from the stream's DatetimebasedCursor and SubstreamPartitionRouter.
Example input state: { "13506132": { "last_changed": "2022-12-27T08:34:39+00:00" } Example output state: { "partition": {"id": "13506132"}, "cursor": {"last_changed": "2022-12-27T08:34:39+00:00"} }
38 def __init__( 39 self, 40 partition_router: Union[ 41 SubstreamPartitionRouter, UnionPartitionRouter, CustomPartitionRouter 42 ], 43 cursor: DatetimeBasedCursor, 44 config: Mapping[str, Any], 45 parameters: Mapping[str, Any], 46 ): 47 self._partition_router = partition_router 48 self._cursor = cursor 49 self._config = config 50 self._parameters = parameters 51 self._partition_key_field = InterpolatedString.create( 52 self._get_partition_field(partition_router), parameters=self._parameters 53 ).eval(self._config) 54 self._cursor_field = InterpolatedString.create( 55 self._cursor.cursor_field, parameters=self._parameters 56 ).eval(self._config)
78 def should_migrate(self, stream_state: Mapping[str, Any]) -> bool: 79 if _is_already_migrated(stream_state): 80 return False 81 82 # UnionPartitionRouter has no parent_stream_configs; its partitions are already 83 # normalized to a single partition field so the parent stream check does not apply. 84 if not isinstance(self._partition_router, UnionPartitionRouter): 85 # There is exactly one parent stream 86 number_of_parent_streams = len(self._partition_router.parent_stream_configs) # type: ignore # custom partition will introduce this attribute if needed 87 if number_of_parent_streams != 1: 88 # There should be exactly one parent stream 89 return False 90 """ 91 The expected state format is 92 "<parent_key_id>" : { 93 "<cursor_field>" : "<cursor_value>" 94 } 95 """ 96 if not stream_state: 97 return False 98 for key, value in stream_state.items(): 99 # it is expected the internal value to be a dictionary according to docstring 100 if not isinstance(value, dict): 101 return False 102 keys = list(value.keys()) 103 if len(keys) != 1: 104 # The input partitioned state should only have one key 105 return False 106 if keys[0] != self._cursor_field: 107 # Unexpected key. Found {keys[0]}. Expected {self._cursor.cursor_field} 108 return False 109 110 return True
Check if the stream_state should be migrated
Parameters
- stream_state: The stream_state to potentially migrate
Returns
true if the state is of the expected format and should be migrated. False otherwise.
112 def migrate(self, stream_state: Mapping[str, Any]) -> Mapping[str, Any]: 113 states = [ 114 {"partition": {self._partition_key_field: key}, "cursor": value} 115 for key, value in stream_state.items() 116 ] 117 return {"states": states}
Migrate the stream_state. Assumes should_migrate(stream_state) returned True.
Parameters
- stream_state: The stream_state to migrate
Returns
The migrated stream_state
14@dataclass 15class MinMaxDatetime: 16 """ 17 Compares the provided date against optional minimum or maximum times. If date is earlier than 18 min_date, then min_date is returned. If date is greater than max_date, then max_date is returned. 19 If neither, the input date is returned. 20 21 The timestamp format accepts the same format codes as datetime.strfptime, which are 22 all the format codes required by the 1989 C standard. 23 Full list of accepted format codes: https://man7.org/linux/man-pages/man3/strftime.3.html 24 25 Attributes: 26 datetime (Union[InterpolatedString, str]): InterpolatedString or string representing the datetime in the format specified by `datetime_format` 27 datetime_format (str): Format of the datetime passed as argument 28 min_datetime (Union[InterpolatedString, str]): Represents the minimum allowed datetime value. 29 max_datetime (Union[InterpolatedString, str]): Represents the maximum allowed datetime value. 30 """ 31 32 datetime: Union[InterpolatedString, str] 33 parameters: InitVar[Mapping[str, Any]] 34 # datetime_format is a unique case where we inherit it from the parent if it is not specified before using the default value 35 # which is why we need dedicated getter/setter methods and private dataclass field 36 datetime_format: str 37 _datetime_format: str = field(init=False, repr=False, default="") 38 min_datetime: Union[InterpolatedString, str] = "" 39 max_datetime: Union[InterpolatedString, str] = "" 40 41 def __post_init__(self, parameters: Mapping[str, Any]) -> None: 42 self.datetime = InterpolatedString.create(self.datetime, parameters=parameters or {}) 43 self._parser = DatetimeParser() 44 self.min_datetime = ( 45 InterpolatedString.create(self.min_datetime, parameters=parameters) # type: ignore [assignment] # expression has type "InterpolatedString | None", variable has type "InterpolatedString | str" 46 if self.min_datetime 47 else None 48 ) # type: ignore 49 self.max_datetime = ( 50 InterpolatedString.create(self.max_datetime, parameters=parameters) # type: ignore [assignment] # expression has type "InterpolatedString | None", variable has type "InterpolatedString | str" 51 if self.max_datetime 52 else None 53 ) # type: ignore 54 55 def get_datetime( 56 self, config: Mapping[str, Any], **additional_parameters: Mapping[str, Any] 57 ) -> dt.datetime: 58 """ 59 Evaluates and returns the datetime 60 :param config: The user-provided configuration as specified by the source's spec 61 :param additional_parameters: Additional arguments to be passed to the strings for interpolation 62 :return: The evaluated datetime 63 """ 64 # We apply a default datetime format here instead of at instantiation, so it can be set by the parent first 65 datetime_format = self._datetime_format 66 if not datetime_format: 67 datetime_format = "%Y-%m-%dT%H:%M:%S.%f%z" 68 69 time = self._parser.parse( 70 str( 71 self.datetime.eval( # type: ignore[union-attr] # str has no attribute "eval" 72 config, 73 **additional_parameters, 74 ) 75 ), 76 datetime_format, 77 ) # type: ignore # datetime is always cast to an interpolated string 78 79 if self.min_datetime: 80 min_time = str(self.min_datetime.eval(config, **additional_parameters)) # type: ignore # min_datetime is always cast to an interpolated string 81 if min_time: 82 min_datetime = self._parser.parse(min_time, datetime_format) # type: ignore # min_datetime is always cast to an interpolated string 83 time = max(time, min_datetime) 84 if self.max_datetime: 85 max_time = str(self.max_datetime.eval(config, **additional_parameters)) # type: ignore # max_datetime is always cast to an interpolated string 86 if max_time: 87 max_datetime = self._parser.parse(max_time, datetime_format) 88 time = min(time, max_datetime) 89 return time 90 91 @property # type: ignore # properties don't play well with dataclasses... 92 def datetime_format(self) -> str: 93 """The format of the string representing the datetime""" 94 return self._datetime_format 95 96 @datetime_format.setter 97 def datetime_format(self, value: str) -> None: 98 """Setter for the datetime format""" 99 # Covers the case where datetime_format is not provided in the constructor, which causes the property object 100 # to be set which we need to avoid doing 101 if not isinstance(value, property): 102 self._datetime_format = value 103 104 @classmethod 105 def create( 106 cls, 107 interpolated_string_or_min_max_datetime: Union[InterpolatedString, str, "MinMaxDatetime"], 108 parameters: Optional[Mapping[str, Any]] = None, 109 ) -> "MinMaxDatetime": 110 if parameters is None: 111 parameters = {} 112 if isinstance(interpolated_string_or_min_max_datetime, InterpolatedString) or isinstance( 113 interpolated_string_or_min_max_datetime, str 114 ): 115 return MinMaxDatetime( # type: ignore [call-arg] 116 datetime=interpolated_string_or_min_max_datetime, parameters=parameters 117 ) 118 else: 119 return interpolated_string_or_min_max_datetime
Compares the provided date against optional minimum or maximum times. If date is earlier than min_date, then min_date is returned. If date is greater than max_date, then max_date is returned. If neither, the input date is returned.
The timestamp format accepts the same format codes as datetime.strfptime, which are all the format codes required by the 1989 C standard. Full list of accepted format codes: https://man7.org/linux/man-pages/man3/strftime.3.html
Attributes:
- datetime (Union[InterpolatedString, str]): InterpolatedString or string representing the datetime in the format specified by
datetime_format - datetime_format (str): Format of the datetime passed as argument
- min_datetime (Union[InterpolatedString, str]): Represents the minimum allowed datetime value.
- max_datetime (Union[InterpolatedString, str]): Represents the maximum allowed datetime value.
91 @property # type: ignore # properties don't play well with dataclasses... 92 def datetime_format(self) -> str: 93 """The format of the string representing the datetime""" 94 return self._datetime_format
The format of the string representing the datetime
55 def get_datetime( 56 self, config: Mapping[str, Any], **additional_parameters: Mapping[str, Any] 57 ) -> dt.datetime: 58 """ 59 Evaluates and returns the datetime 60 :param config: The user-provided configuration as specified by the source's spec 61 :param additional_parameters: Additional arguments to be passed to the strings for interpolation 62 :return: The evaluated datetime 63 """ 64 # We apply a default datetime format here instead of at instantiation, so it can be set by the parent first 65 datetime_format = self._datetime_format 66 if not datetime_format: 67 datetime_format = "%Y-%m-%dT%H:%M:%S.%f%z" 68 69 time = self._parser.parse( 70 str( 71 self.datetime.eval( # type: ignore[union-attr] # str has no attribute "eval" 72 config, 73 **additional_parameters, 74 ) 75 ), 76 datetime_format, 77 ) # type: ignore # datetime is always cast to an interpolated string 78 79 if self.min_datetime: 80 min_time = str(self.min_datetime.eval(config, **additional_parameters)) # type: ignore # min_datetime is always cast to an interpolated string 81 if min_time: 82 min_datetime = self._parser.parse(min_time, datetime_format) # type: ignore # min_datetime is always cast to an interpolated string 83 time = max(time, min_datetime) 84 if self.max_datetime: 85 max_time = str(self.max_datetime.eval(config, **additional_parameters)) # type: ignore # max_datetime is always cast to an interpolated string 86 if max_time: 87 max_datetime = self._parser.parse(max_time, datetime_format) 88 time = min(time, max_datetime) 89 return time
Evaluates and returns the datetime
Parameters
- config: The user-provided configuration as specified by the source's spec
- additional_parameters: Additional arguments to be passed to the strings for interpolation
Returns
The evaluated datetime
104 @classmethod 105 def create( 106 cls, 107 interpolated_string_or_min_max_datetime: Union[InterpolatedString, str, "MinMaxDatetime"], 108 parameters: Optional[Mapping[str, Any]] = None, 109 ) -> "MinMaxDatetime": 110 if parameters is None: 111 parameters = {} 112 if isinstance(interpolated_string_or_min_max_datetime, InterpolatedString) or isinstance( 113 interpolated_string_or_min_max_datetime, str 114 ): 115 return MinMaxDatetime( # type: ignore [call-arg] 116 datetime=interpolated_string_or_min_max_datetime, parameters=parameters 117 ) 118 else: 119 return interpolated_string_or_min_max_datetime
33@dataclass 34class NoAuth(DeclarativeAuthenticator): 35 parameters: InitVar[Mapping[str, Any]] 36 37 @property 38 def auth_header(self) -> str: 39 return "" 40 41 @property 42 def token(self) -> str: 43 return ""
25@dataclass 26class OffsetIncrement(PaginationStrategy): 27 """ 28 Pagination strategy that returns the number of records reads so far and returns it as the next page token 29 Examples: 30 # page_size to be a constant integer value 31 pagination_strategy: 32 type: OffsetIncrement 33 page_size: 2 34 35 # page_size to be a constant string value 36 pagination_strategy: 37 type: OffsetIncrement 38 page_size: "2" 39 40 # page_size to be an interpolated string value 41 pagination_strategy: 42 type: OffsetIncrement 43 page_size: "{{ parameters['items_per_page'] }}" 44 45 Attributes: 46 page_size (InterpolatedString): the number of records to request 47 extractor (Optional[RecordExtractor]): counts the records of a page by extracting them 48 again. Leave it unset to use the count of the `RecordSelector` that read the page 49 """ 50 51 config: Config 52 page_size: Optional[Union[str, int]] 53 parameters: InitVar[Mapping[str, Any]] 54 extractor: Optional[RecordExtractor] = None 55 decoder: Decoder = field( 56 default_factory=lambda: PaginationDecoderDecorator(decoder=JsonDecoder(parameters={})) 57 ) 58 inject_on_first_request: bool = False 59 60 def __post_init__(self, parameters: Mapping[str, Any]) -> None: 61 page_size = str(self.page_size) if isinstance(self.page_size, int) else self.page_size 62 if page_size: 63 self._page_size: Optional[InterpolatedString] = InterpolatedString( 64 page_size, parameters=parameters 65 ) 66 else: 67 self._page_size = None 68 69 @property 70 def initial_token(self) -> Optional[Any]: 71 if self.inject_on_first_request: 72 return 0 73 return None 74 75 def next_page_token( 76 self, 77 response: requests.Response, 78 last_page_size: int, 79 last_record: Optional[Record], 80 last_page_token_value: Optional[Any] = None, 81 page_size_override: Optional[int] = None, 82 stream_slice: Optional[StreamSlice] = None, 83 ) -> Optional[Any]: 84 # The records the API returned, not the ones emitted: a record filter must neither end the 85 # pagination nor shorten the offset. 86 if self.extractor: 87 last_page_size = len(list(self.extractor.extract_records(response=response))) 88 else: 89 last_page_size = extracted_record_count(response, default=last_page_size) 90 91 # Stop paginating when there are fewer records than the page size or the current page has no records. 92 # The comparison uses the page size that was actually requested: after a `REDUCE_PAGE_SIZE` reduction, a 93 # full page is smaller than the configured page size and comparing against the latter would end the 94 # pagination early and skip records. 95 if page_size_override is not None: 96 requested_page_size: Optional[int] = page_size_override 97 elif isinstance(self.page_size, int): 98 # A constant needs no response, which would otherwise be decoded a second time. 99 requested_page_size = self.page_size 100 elif self._page_size: 101 requested_page_size = self._page_size.eval( 102 self.config, response=next(self.decoder.decode(response)) 103 ) 104 else: 105 requested_page_size = None 106 if (requested_page_size and last_page_size < requested_page_size) or last_page_size == 0: 107 return None 108 elif last_page_token_value is None: 109 # If the OffsetIncrement strategy does not inject on the first request, the incoming last_page_token_value 110 # will be None. For this case, we assume that None was the first page and progress to the next offset 111 return 0 + last_page_size 112 elif not isinstance(last_page_token_value, int): 113 raise ValueError( 114 f"Last page token value {last_page_token_value} for OffsetIncrement pagination strategy was not an integer" 115 ) 116 else: 117 return last_page_token_value + last_page_size 118 119 def get_page_size(self) -> Optional[int]: 120 if self._page_size: 121 page_size = self._page_size.eval(self.config) 122 if not isinstance(page_size, int): 123 raise Exception(f"{page_size} is of type {type(page_size)}. Expected {int}") 124 return page_size 125 else: 126 return None
Pagination strategy that returns the number of records reads so far and returns it as the next page token
Examples:
page_size to be a constant integer value
pagination_strategy: type: OffsetIncrement page_size: 2
page_size to be a constant string value
pagination_strategy: type: OffsetIncrement page_size: "2"
page_size to be an interpolated string value
pagination_strategy: type: OffsetIncrement page_size: "{{ parameters['items_per_page'] }}"
Attributes:
- page_size (InterpolatedString): the number of records to request
- extractor (Optional[RecordExtractor]): counts the records of a page by extracting them
again. Leave it unset to use the count of the
RecordSelectorthat read the page
69 @property 70 def initial_token(self) -> Optional[Any]: 71 if self.inject_on_first_request: 72 return 0 73 return None
Return the initial value of the token
75 def next_page_token( 76 self, 77 response: requests.Response, 78 last_page_size: int, 79 last_record: Optional[Record], 80 last_page_token_value: Optional[Any] = None, 81 page_size_override: Optional[int] = None, 82 stream_slice: Optional[StreamSlice] = None, 83 ) -> Optional[Any]: 84 # The records the API returned, not the ones emitted: a record filter must neither end the 85 # pagination nor shorten the offset. 86 if self.extractor: 87 last_page_size = len(list(self.extractor.extract_records(response=response))) 88 else: 89 last_page_size = extracted_record_count(response, default=last_page_size) 90 91 # Stop paginating when there are fewer records than the page size or the current page has no records. 92 # The comparison uses the page size that was actually requested: after a `REDUCE_PAGE_SIZE` reduction, a 93 # full page is smaller than the configured page size and comparing against the latter would end the 94 # pagination early and skip records. 95 if page_size_override is not None: 96 requested_page_size: Optional[int] = page_size_override 97 elif isinstance(self.page_size, int): 98 # A constant needs no response, which would otherwise be decoded a second time. 99 requested_page_size = self.page_size 100 elif self._page_size: 101 requested_page_size = self._page_size.eval( 102 self.config, response=next(self.decoder.decode(response)) 103 ) 104 else: 105 requested_page_size = None 106 if (requested_page_size and last_page_size < requested_page_size) or last_page_size == 0: 107 return None 108 elif last_page_token_value is None: 109 # If the OffsetIncrement strategy does not inject on the first request, the incoming last_page_token_value 110 # will be None. For this case, we assume that None was the first page and progress to the next offset 111 return 0 + last_page_size 112 elif not isinstance(last_page_token_value, int): 113 raise ValueError( 114 f"Last page token value {last_page_token_value} for OffsetIncrement pagination strategy was not an integer" 115 ) 116 else: 117 return last_page_token_value + last_page_size
Parameters
- response: response to process
- last_page_size: the number of records read from the response
- last_record: the last record extracted from the response
- last_page_token_value: The current value of the page token made on the last request
- page_size_override: the page size that was actually requested, when it differs from the configured
one because of a
REDUCE_PAGE_SIZEresponse action. Strategies that use their configured page size as a stop condition must honor this value, else they end the pagination early and skip records. - stream_slice: the slice the page was read for. Only passed to strategies that declare it, so existing strategies defined outside of the CDK keep working unchanged.
Returns
next page token. Returns None if there are no more pages to fetch
119 def get_page_size(self) -> Optional[int]: 120 if self._page_size: 121 page_size = self._page_size.eval(self.config) 122 if not isinstance(page_size, int): 123 raise Exception(f"{page_size} is of type {type(page_size)}. Expected {int}") 124 return page_size 125 else: 126 return None
Returns
page size: The number of records to fetch in a page. Returns None if unspecified
22@dataclass 23class PageIncrement(PaginationStrategy): 24 """ 25 Pagination strategy that returns the number of pages reads so far and returns it as the next page token 26 27 Attributes: 28 page_size (int): the number of records to request 29 start_from_page (int): number of the initial page 30 extractor (Optional[RecordExtractor]): counts the records of a page by extracting them 31 again. Leave it unset to use the count of the `RecordSelector` that read the page 32 """ 33 34 config: Config 35 page_size: Optional[Union[str, int]] 36 parameters: InitVar[Mapping[str, Any]] 37 start_from_page: int = 0 38 inject_on_first_request: bool = False 39 extractor: Optional[RecordExtractor] = None 40 41 def __post_init__(self, parameters: Mapping[str, Any]) -> None: 42 if isinstance(self.page_size, int) or (self.page_size is None): 43 self._page_size = self.page_size 44 else: 45 page_size = InterpolatedString(self.page_size, parameters=parameters).eval(self.config) 46 if not isinstance(page_size, int): 47 raise Exception(f"{page_size} is of type {type(page_size)}. Expected {int}") 48 self._page_size = page_size 49 50 @property 51 def initial_token(self) -> Optional[Any]: 52 if self.inject_on_first_request: 53 return self.start_from_page 54 return None 55 56 def next_page_token( 57 self, 58 response: requests.Response, 59 last_page_size: int, 60 last_record: Optional[Record], 61 last_page_token_value: Optional[Any], 62 page_size_override: Optional[int] = None, 63 stream_slice: Optional[StreamSlice] = None, 64 ) -> Optional[Any]: 65 if page_size_override is not None: 66 # Reachable only when the factory is bypassed: a manifest naming PageIncrement, and a 67 # CustomPaginationStrategy that subclasses it, are both rejected at config time. An unrecognized 68 # ValueError here would be reported as a generic system error even though the message describes a 69 # configuration mistake. `next_page_token` runs after the page's records have been emitted, so an 70 # out-of-tree caller gets this mid-stream rather than at startup - still better than silently 71 # ignoring the override and skipping records. 72 raise AirbyteTracedException( 73 internal_message="PageIncrement received a page_size_override", 74 message="PageIncrement does not support reducing the page size while paginating: pages are " 75 "addressed as page number * page size, so a smaller page size shifts every following page " 76 "boundary and would skip records. Use OffsetIncrement or CursorPagination instead.", 77 failure_type=FailureType.config_error, 78 ) 79 80 # The record count is dependent on the records returned from the response which may not always 81 # align with the size of pages emitted. For example, a record filter can reduce the number of 82 # records observed below the page size even though the API returned a full page. 83 if self.extractor: 84 last_page_size = len(list(self.extractor.extract_records(response=response))) 85 else: 86 last_page_size = extracted_record_count(response, default=last_page_size) 87 88 # Stop paginating when there are fewer records than the page size or the current page has no records 89 if (self._page_size and last_page_size < self._page_size) or last_page_size == 0: 90 return None 91 elif last_page_token_value is None: 92 # If the PageIncrement strategy does not inject on the first request, the incoming last_page_token_value 93 # may be None. When this is the case, we assume we've already requested the first page specified by 94 # start_from_page and must now get the next page 95 return self.start_from_page + 1 96 elif not isinstance(last_page_token_value, int): 97 raise ValueError( 98 f"Last page token value {last_page_token_value} for PageIncrement pagination strategy was not an integer" 99 ) 100 else: 101 return last_page_token_value + 1 102 103 def get_page_size(self) -> Optional[int]: 104 return self._page_size
Pagination strategy that returns the number of pages reads so far and returns it as the next page token
Attributes:
- page_size (int): the number of records to request
- start_from_page (int): number of the initial page
- extractor (Optional[RecordExtractor]): counts the records of a page by extracting them
again. Leave it unset to use the count of the
RecordSelectorthat read the page
50 @property 51 def initial_token(self) -> Optional[Any]: 52 if self.inject_on_first_request: 53 return self.start_from_page 54 return None
Return the initial value of the token
56 def next_page_token( 57 self, 58 response: requests.Response, 59 last_page_size: int, 60 last_record: Optional[Record], 61 last_page_token_value: Optional[Any], 62 page_size_override: Optional[int] = None, 63 stream_slice: Optional[StreamSlice] = None, 64 ) -> Optional[Any]: 65 if page_size_override is not None: 66 # Reachable only when the factory is bypassed: a manifest naming PageIncrement, and a 67 # CustomPaginationStrategy that subclasses it, are both rejected at config time. An unrecognized 68 # ValueError here would be reported as a generic system error even though the message describes a 69 # configuration mistake. `next_page_token` runs after the page's records have been emitted, so an 70 # out-of-tree caller gets this mid-stream rather than at startup - still better than silently 71 # ignoring the override and skipping records. 72 raise AirbyteTracedException( 73 internal_message="PageIncrement received a page_size_override", 74 message="PageIncrement does not support reducing the page size while paginating: pages are " 75 "addressed as page number * page size, so a smaller page size shifts every following page " 76 "boundary and would skip records. Use OffsetIncrement or CursorPagination instead.", 77 failure_type=FailureType.config_error, 78 ) 79 80 # The record count is dependent on the records returned from the response which may not always 81 # align with the size of pages emitted. For example, a record filter can reduce the number of 82 # records observed below the page size even though the API returned a full page. 83 if self.extractor: 84 last_page_size = len(list(self.extractor.extract_records(response=response))) 85 else: 86 last_page_size = extracted_record_count(response, default=last_page_size) 87 88 # Stop paginating when there are fewer records than the page size or the current page has no records 89 if (self._page_size and last_page_size < self._page_size) or last_page_size == 0: 90 return None 91 elif last_page_token_value is None: 92 # If the PageIncrement strategy does not inject on the first request, the incoming last_page_token_value 93 # may be None. When this is the case, we assume we've already requested the first page specified by 94 # start_from_page and must now get the next page 95 return self.start_from_page + 1 96 elif not isinstance(last_page_token_value, int): 97 raise ValueError( 98 f"Last page token value {last_page_token_value} for PageIncrement pagination strategy was not an integer" 99 ) 100 else: 101 return last_page_token_value + 1
Parameters
- response: response to process
- last_page_size: the number of records read from the response
- last_record: the last record extracted from the response
- last_page_token_value: The current value of the page token made on the last request
- page_size_override: the page size that was actually requested, when it differs from the configured
one because of a
REDUCE_PAGE_SIZEresponse action. Strategies that use their configured page size as a stop condition must honor this value, else they end the pagination early and skip records. - stream_slice: the slice the page was read for. Only passed to strategies that declare it, so existing strategies defined outside of the CDK keep working unchanged.
Returns
next page token. Returns None if there are no more pages to fetch
15@dataclass 16class PaginationStrategy: 17 """ 18 Defines how to get the next page token 19 20 One instance paginates every partition of the stream, several at once, so `next_page_token` must derive the 21 next token from its arguments, such as `last_page_token_value`, rather than from state kept on the instance. 22 """ 23 24 @property 25 @abstractmethod 26 def initial_token(self) -> Optional[Any]: 27 """ 28 Return the initial value of the token 29 """ 30 31 @abstractmethod 32 def next_page_token( 33 self, 34 response: requests.Response, 35 last_page_size: int, 36 last_record: Optional[Record], 37 last_page_token_value: Optional[Any], 38 page_size_override: Optional[int] = None, 39 stream_slice: Optional[StreamSlice] = None, 40 ) -> Optional[Any]: 41 """ 42 :param response: response to process 43 :param last_page_size: the number of records read from the response 44 :param last_record: the last record extracted from the response 45 :param last_page_token_value: The current value of the page token made on the last request 46 :param page_size_override: the page size that was actually requested, when it differs from the configured 47 one because of a `REDUCE_PAGE_SIZE` response action. Strategies that use their configured page size as 48 a stop condition must honor this value, else they end the pagination early and skip records. 49 :param stream_slice: the slice the page was read for. Only passed to strategies that declare it, so existing 50 strategies defined outside of the CDK keep working unchanged. 51 :return: next page token. Returns None if there are no more pages to fetch 52 """ 53 pass 54 55 @abstractmethod 56 def get_page_size(self) -> Optional[int]: 57 """ 58 :return: page size: The number of records to fetch in a page. Returns None if unspecified 59 """
Defines how to get the next page token
One instance paginates every partition of the stream, several at once, so next_page_token must derive the
next token from its arguments, such as last_page_token_value, rather than from state kept on the instance.
24 @property 25 @abstractmethod 26 def initial_token(self) -> Optional[Any]: 27 """ 28 Return the initial value of the token 29 """
Return the initial value of the token
31 @abstractmethod 32 def next_page_token( 33 self, 34 response: requests.Response, 35 last_page_size: int, 36 last_record: Optional[Record], 37 last_page_token_value: Optional[Any], 38 page_size_override: Optional[int] = None, 39 stream_slice: Optional[StreamSlice] = None, 40 ) -> Optional[Any]: 41 """ 42 :param response: response to process 43 :param last_page_size: the number of records read from the response 44 :param last_record: the last record extracted from the response 45 :param last_page_token_value: The current value of the page token made on the last request 46 :param page_size_override: the page size that was actually requested, when it differs from the configured 47 one because of a `REDUCE_PAGE_SIZE` response action. Strategies that use their configured page size as 48 a stop condition must honor this value, else they end the pagination early and skip records. 49 :param stream_slice: the slice the page was read for. Only passed to strategies that declare it, so existing 50 strategies defined outside of the CDK keep working unchanged. 51 :return: next page token. Returns None if there are no more pages to fetch 52 """ 53 pass
Parameters
- response: response to process
- last_page_size: the number of records read from the response
- last_record: the last record extracted from the response
- last_page_token_value: The current value of the page token made on the last request
- page_size_override: the page size that was actually requested, when it differs from the configured
one because of a
REDUCE_PAGE_SIZEresponse action. Strategies that use their configured page size as a stop condition must honor this value, else they end the pagination early and skip records. - stream_slice: the slice the page was read for. Only passed to strategies that declare it, so existing strategies defined outside of the CDK keep working unchanged.
Returns
next page token. Returns None if there are no more pages to fetch
55 @abstractmethod 56 def get_page_size(self) -> Optional[int]: 57 """ 58 :return: page size: The number of records to fetch in a page. Returns None if unspecified 59 """
Returns
page size: The number of records to fetch in a page. Returns None if unspecified
59@dataclass 60class ParentStreamConfig: 61 """ 62 Describes how to create a stream slice from a parent stream 63 64 stream: The stream to read records from 65 parent_key: The key of the parent stream's records that will be the stream slice key 66 partition_field: The partition key 67 extra_fields: Additional field paths to include in the stream slice 68 request_option: How to inject the slice value on an outgoing HTTP request 69 incremental_dependency (bool): Indicates if the parent stream should be read incrementally. 70 """ 71 72 stream: "AbstractStream" 73 parent_key: Union[InterpolatedString, str] 74 partition_field: Union[InterpolatedString, str] 75 config: Config 76 parameters: InitVar[Mapping[str, Any]] 77 extra_fields: Optional[Union[List[List[str]], List[List[InterpolatedString]]]] = ( 78 None # List of field paths (arrays of strings) 79 ) 80 request_option: Optional[RequestOption] = None 81 incremental_dependency: bool = False 82 lazy_read_pointer: Optional[List[Union[InterpolatedString, str]]] = None 83 84 def __post_init__(self, parameters: Mapping[str, Any]) -> None: 85 self.parent_key = InterpolatedString.create(self.parent_key, parameters=parameters) 86 self.partition_field = InterpolatedString.create( 87 self.partition_field, parameters=parameters 88 ) 89 if self.extra_fields: 90 # Create InterpolatedString for each field path in extra_keys 91 self.extra_fields = [ 92 [InterpolatedString.create(path, parameters=parameters) for path in key_path] 93 for key_path in self.extra_fields 94 ] 95 96 self.lazy_read_pointer = ( 97 [ 98 InterpolatedString.create(path, parameters=parameters) 99 if isinstance(path, str) 100 else path 101 for path in self.lazy_read_pointer 102 ] 103 if self.lazy_read_pointer 104 else None 105 )
Describes how to create a stream slice from a parent stream
stream: The stream to read records from parent_key: The key of the parent stream's records that will be the stream slice key partition_field: The partition key extra_fields: Additional field paths to include in the stream slice request_option: How to inject the slice value on an outgoing HTTP request incremental_dependency (bool): Indicates if the parent stream should be read incrementally.
7class ReadException(Exception): 8 """ 9 Raise when there is an error reading data from an API Source 10 """
Raise when there is an error reading data from an API Source
12@dataclass 13class RecordExtractor: 14 """ 15 Responsible for translating an HTTP response into a list of records by extracting records from the response. 16 """ 17 18 @abstractmethod 19 def extract_records( 20 self, 21 response: requests.Response, 22 ) -> Iterable[Mapping[str, Any]]: 23 """ 24 Selects records from the response 25 :param response: The response to extract the records from 26 :return: List of Records extracted from the response 27 """ 28 pass
Responsible for translating an HTTP response into a list of records by extracting records from the response.
18 @abstractmethod 19 def extract_records( 20 self, 21 response: requests.Response, 22 ) -> Iterable[Mapping[str, Any]]: 23 """ 24 Selects records from the response 25 :param response: The response to extract the records from 26 :return: List of Records extracted from the response 27 """ 28 pass
Selects records from the response
Parameters
- response: The response to extract the records from
Returns
List of Records extracted from the response
13@dataclass 14class RecordFilter: 15 """ 16 Filter applied on a list of Records 17 18 config (Config): The user-provided configuration as specified by the source's spec 19 condition (str): The string representing the predicate to filter a record. Records will be removed if evaluated to False 20 """ 21 22 parameters: InitVar[Mapping[str, Any]] 23 config: Config 24 condition: str = "" 25 26 def __post_init__(self, parameters: Mapping[str, Any]) -> None: 27 self._filter_interpolator = InterpolatedBoolean( 28 condition=self.condition, parameters=parameters 29 ) 30 31 def filter_records( 32 self, 33 records: Iterable[Mapping[str, Any]], 34 stream_state: StreamState, 35 stream_slice: Optional[StreamSlice] = None, 36 next_page_token: Optional[Mapping[str, Any]] = None, 37 ) -> Iterable[Mapping[str, Any]]: 38 kwargs = { 39 "stream_state": stream_state, 40 "stream_slice": stream_slice, 41 "next_page_token": next_page_token, 42 "stream_slice.extra_fields": stream_slice.extra_fields if stream_slice else {}, 43 } 44 for record in records: 45 if self._filter_interpolator.eval(self.config, record=record, **kwargs): 46 yield record
Filter applied on a list of Records
config (Config): The user-provided configuration as specified by the source's spec condition (str): The string representing the predicate to filter a record. Records will be removed if evaluated to False
31 def filter_records( 32 self, 33 records: Iterable[Mapping[str, Any]], 34 stream_state: StreamState, 35 stream_slice: Optional[StreamSlice] = None, 36 next_page_token: Optional[Mapping[str, Any]] = None, 37 ) -> Iterable[Mapping[str, Any]]: 38 kwargs = { 39 "stream_state": stream_state, 40 "stream_slice": stream_slice, 41 "next_page_token": next_page_token, 42 "stream_slice.extra_fields": stream_slice.extra_fields if stream_slice else {}, 43 } 44 for record in records: 45 if self._filter_interpolator.eval(self.config, record=record, **kwargs): 46 yield record
39@dataclass 40class RecordSelector(HttpSelector): 41 """ 42 Responsible for translating an HTTP response into a list of records by extracting records from the response and optionally filtering 43 records based on a heuristic. 44 45 Attributes: 46 extractor (RecordExtractor): The record extractor responsible for extracting records from a response 47 schema_normalization (TypeTransformer): The record normalizer responsible for casting record values to stream schema types 48 record_filter (RecordFilter): The record filter responsible for filtering extracted records 49 transformations (List[RecordTransformation]): The transformations to be done on the records 50 """ 51 52 extractor: RecordExtractor 53 config: Config 54 parameters: InitVar[Mapping[str, Any]] 55 schema_normalization: Union[TypeTransformer, DeclarativeTypeTransformer] 56 name: str 57 _name: Union[InterpolatedString, str] = field(init=False, repr=False, default="") 58 record_filter: Optional[RecordFilter] = None 59 transformations: List[RecordTransformation] = field(default_factory=lambda: []) 60 transform_before_filtering: bool = False 61 file_uploader: Optional[DefaultFileUploader] = None 62 63 def __post_init__(self, parameters: Mapping[str, Any]) -> None: 64 self._parameters = parameters 65 self._name = ( 66 InterpolatedString(self._name, parameters=parameters) 67 if isinstance(self._name, str) 68 else self._name 69 ) 70 71 @property # type: ignore 72 def name(self) -> str: 73 """ 74 :return: Stream name 75 """ 76 return ( 77 str(self._name.eval(self.config)) 78 if isinstance(self._name, InterpolatedString) 79 else self._name 80 ) 81 82 @name.setter 83 def name(self, value: str) -> None: 84 if not isinstance(value, property): 85 self._name = value 86 87 def select_records( 88 self, 89 response: requests.Response, 90 stream_state: StreamState, 91 records_schema: Mapping[str, Any], 92 stream_slice: Optional[StreamSlice] = None, 93 next_page_token: Optional[Mapping[str, Any]] = None, 94 ) -> Iterable[Record]: 95 """ 96 Selects records from the response 97 :param response: The response to select the records from 98 :param stream_state: The stream state 99 :param records_schema: json schema of records to return 100 :param stream_slice: The stream slice 101 :param next_page_token: The paginator token 102 :return: List of Records selected from the response 103 """ 104 all_data: Iterable[Mapping[str, Any]] = self._extract_and_count(response) 105 yield from self.filter_and_transform( 106 all_data, stream_state, records_schema, stream_slice, next_page_token 107 ) 108 # A custom record filter can stop reading before the end of the page, which would leave the 109 # page uncounted. 110 for _ in all_data: 111 pass 112 113 def _extract_and_count(self, response: requests.Response) -> Iterable[Mapping[str, Any]]: 114 count = 0 115 for record in self.extractor.extract_records(response): 116 count += 1 117 if not isinstance(record, _DroppedEmptyRecord): 118 yield record 119 # Kept on the response, not on the selector: a stream has one retriever, and so one 120 # selector, reading all of its partitions concurrently. 121 setattr(response, _EXTRACTED_RECORD_COUNT, count) 122 123 def filter_and_transform( 124 self, 125 all_data: Iterable[Mapping[str, Any]], 126 stream_state: StreamState, 127 records_schema: Mapping[str, Any], 128 stream_slice: Optional[StreamSlice] = None, 129 next_page_token: Optional[Mapping[str, Any]] = None, 130 ) -> Iterable[Record]: 131 """ 132 There is an issue with the selector as of 2024-08-30: it does technology-agnostic processing like filtering, transformation and 133 normalization with an API that is technology-specific (as requests.Response is only for HTTP communication using the requests 134 library). 135 136 Until we decide to move this logic away from the selector, we made this method public so that users like AsyncJobRetriever could 137 share the logic of doing transformations on a set of records. 138 """ 139 if self.transform_before_filtering: 140 transformed_data = self._transform(all_data, stream_state, stream_slice) 141 transformed_filtered_data = self._filter( 142 transformed_data, stream_state, stream_slice, next_page_token 143 ) 144 else: 145 filtered_data = self._filter(all_data, stream_state, stream_slice, next_page_token) 146 transformed_filtered_data = self._transform(filtered_data, stream_state, stream_slice) 147 normalized_data = self._normalize_by_schema( 148 transformed_filtered_data, schema=records_schema 149 ) 150 for data in normalized_data: 151 record = Record(data=data, stream_name=self.name, associated_slice=stream_slice) 152 if self.file_uploader: 153 self.file_uploader.upload(record) 154 yield record 155 156 def _normalize_by_schema( 157 self, records: Iterable[Mapping[str, Any]], schema: Optional[Mapping[str, Any]] 158 ) -> Iterable[Mapping[str, Any]]: 159 if schema: 160 # record has type Mapping[str, Any], but dict[str, Any] expected 161 for record in records: 162 normalized_record = dict(record) 163 self.schema_normalization.transform(normalized_record, schema) 164 yield normalized_record 165 else: 166 yield from records 167 168 def _filter( 169 self, 170 records: Iterable[Mapping[str, Any]], 171 stream_state: StreamState, 172 stream_slice: Optional[StreamSlice], 173 next_page_token: Optional[Mapping[str, Any]], 174 ) -> Iterable[Mapping[str, Any]]: 175 if self.record_filter: 176 yield from self.record_filter.filter_records( 177 records, 178 stream_state=stream_state, 179 stream_slice=stream_slice, 180 next_page_token=next_page_token, 181 ) 182 else: 183 yield from records 184 185 def _transform( 186 self, 187 records: Iterable[Mapping[str, Any]], 188 stream_state: StreamState, 189 stream_slice: Optional[StreamSlice] = None, 190 ) -> Iterable[Mapping[str, Any]]: 191 for record in records: 192 for transformation in self.transformations: 193 transformation.transform( 194 record, # type: ignore # record has type Mapping[str, Any], but Dict[str, Any] expected 195 config=self.config, 196 stream_state=stream_state, 197 stream_slice=stream_slice, 198 ) 199 yield record
Responsible for translating an HTTP response into a list of records by extracting records from the response and optionally filtering records based on a heuristic.
Attributes:
- extractor (RecordExtractor): The record extractor responsible for extracting records from a response
- schema_normalization (TypeTransformer): The record normalizer responsible for casting record values to stream schema types
- record_filter (RecordFilter): The record filter responsible for filtering extracted records
- transformations (List[RecordTransformation]): The transformations to be done on the records
71 @property # type: ignore 72 def name(self) -> str: 73 """ 74 :return: Stream name 75 """ 76 return ( 77 str(self._name.eval(self.config)) 78 if isinstance(self._name, InterpolatedString) 79 else self._name 80 )
Returns
Stream name
87 def select_records( 88 self, 89 response: requests.Response, 90 stream_state: StreamState, 91 records_schema: Mapping[str, Any], 92 stream_slice: Optional[StreamSlice] = None, 93 next_page_token: Optional[Mapping[str, Any]] = None, 94 ) -> Iterable[Record]: 95 """ 96 Selects records from the response 97 :param response: The response to select the records from 98 :param stream_state: The stream state 99 :param records_schema: json schema of records to return 100 :param stream_slice: The stream slice 101 :param next_page_token: The paginator token 102 :return: List of Records selected from the response 103 """ 104 all_data: Iterable[Mapping[str, Any]] = self._extract_and_count(response) 105 yield from self.filter_and_transform( 106 all_data, stream_state, records_schema, stream_slice, next_page_token 107 ) 108 # A custom record filter can stop reading before the end of the page, which would leave the 109 # page uncounted. 110 for _ in all_data: 111 pass
Selects records from the response
Parameters
- response: The response to select the records from
- stream_state: The stream state
- records_schema: json schema of records to return
- stream_slice: The stream slice
- next_page_token: The paginator token
Returns
List of Records selected from the response
123 def filter_and_transform( 124 self, 125 all_data: Iterable[Mapping[str, Any]], 126 stream_state: StreamState, 127 records_schema: Mapping[str, Any], 128 stream_slice: Optional[StreamSlice] = None, 129 next_page_token: Optional[Mapping[str, Any]] = None, 130 ) -> Iterable[Record]: 131 """ 132 There is an issue with the selector as of 2024-08-30: it does technology-agnostic processing like filtering, transformation and 133 normalization with an API that is technology-specific (as requests.Response is only for HTTP communication using the requests 134 library). 135 136 Until we decide to move this logic away from the selector, we made this method public so that users like AsyncJobRetriever could 137 share the logic of doing transformations on a set of records. 138 """ 139 if self.transform_before_filtering: 140 transformed_data = self._transform(all_data, stream_state, stream_slice) 141 transformed_filtered_data = self._filter( 142 transformed_data, stream_state, stream_slice, next_page_token 143 ) 144 else: 145 filtered_data = self._filter(all_data, stream_state, stream_slice, next_page_token) 146 transformed_filtered_data = self._transform(filtered_data, stream_state, stream_slice) 147 normalized_data = self._normalize_by_schema( 148 transformed_filtered_data, schema=records_schema 149 ) 150 for data in normalized_data: 151 record = Record(data=data, stream_name=self.name, associated_slice=stream_slice) 152 if self.file_uploader: 153 self.file_uploader.upload(record) 154 yield record
There is an issue with the selector as of 2024-08-30: it does technology-agnostic processing like filtering, transformation and normalization with an API that is technology-specific (as requests.Response is only for HTTP communication using the requests library).
Until we decide to move this logic away from the selector, we made this method public so that users like AsyncJobRetriever could share the logic of doing transformations on a set of records.
13@dataclass 14class RecordTransformation: 15 """ 16 Implementations of this class define transformations that can be applied to records of a stream. 17 """ 18 19 @abstractmethod 20 def transform( 21 self, 22 record: Dict[str, Any], 23 config: Optional[Config] = None, 24 stream_state: Optional[StreamState] = None, 25 stream_slice: Optional[StreamSlice] = None, 26 ) -> None: 27 """ 28 Transform a record by adding, deleting, or mutating fields directly from the record reference passed in argument. 29 30 :param record: The input record to be transformed 31 :param config: The user-provided configuration as specified by the source's spec 32 :param stream_state: The stream state 33 :param stream_slice: The stream slice 34 :return: The transformed record 35 """ 36 37 def __eq__(self, other: object) -> bool: 38 return other.__dict__ == self.__dict__
Implementations of this class define transformations that can be applied to records of a stream.
19 @abstractmethod 20 def transform( 21 self, 22 record: Dict[str, Any], 23 config: Optional[Config] = None, 24 stream_state: Optional[StreamState] = None, 25 stream_slice: Optional[StreamSlice] = None, 26 ) -> None: 27 """ 28 Transform a record by adding, deleting, or mutating fields directly from the record reference passed in argument. 29 30 :param record: The input record to be transformed 31 :param config: The user-provided configuration as specified by the source's spec 32 :param stream_state: The stream state 33 :param stream_slice: The stream slice 34 :return: The transformed record 35 """
Transform a record by adding, deleting, or mutating fields directly from the record reference passed in argument.
Parameters
- record: The input record to be transformed
- config: The user-provided configuration as specified by the source's spec
- stream_state: The stream state
- stream_slice: The stream slice
Returns
The transformed record
25@dataclass 26class RequestOption: 27 """ 28 Describes an option to set on a request 29 30 Attributes: 31 field_name (str): Describes the name of the parameter to inject. Mutually exclusive with field_path. 32 field_path (list(str)): Describes the path to a nested field as a list of field names. 33 Only valid for body_json injection type, and mutually exclusive with field_name. 34 inject_into (RequestOptionType): Describes where in the HTTP request to inject the parameter 35 """ 36 37 inject_into: RequestOptionType 38 parameters: InitVar[Mapping[str, Any]] 39 field_name: Optional[Union[InterpolatedString, str]] = None 40 field_path: Optional[List[Union[InterpolatedString, str]]] = None 41 42 def __post_init__(self, parameters: Mapping[str, Any]) -> None: 43 # Validate inputs. We should expect either field_name or field_path, but not both 44 if self.field_name is None and self.field_path is None: 45 raise ValueError("RequestOption requires either a field_name or field_path") 46 47 if self.field_name is not None and self.field_path is not None: 48 raise ValueError( 49 "Only one of field_name or field_path can be provided to RequestOption" 50 ) 51 52 # Nested field injection is only supported for body JSON injection 53 if self.field_path is not None and self.inject_into != RequestOptionType.body_json: 54 raise ValueError( 55 "Nested field injection is only supported for body JSON injection. Please use a top-level field_name for other injection types." 56 ) 57 58 # Convert field_name and field_path into InterpolatedString objects if they are strings 59 if self.field_name is not None: 60 self.field_name = InterpolatedString.create(self.field_name, parameters=parameters) 61 elif self.field_path is not None: 62 self.field_path = [ 63 InterpolatedString.create(segment, parameters=parameters) 64 for segment in self.field_path 65 ] 66 67 @property 68 def _is_field_path(self) -> bool: 69 """Returns whether this option is a field path (ie, a nested field)""" 70 return self.field_path is not None 71 72 def inject_into_request( 73 self, 74 target: MutableMapping[str, Any], 75 value: Any, 76 config: Config, 77 ) -> None: 78 """ 79 Inject a request option value into a target request structure using either field_name or field_path. 80 For non-body-json injection, only top-level field names are supported. 81 For body-json injection, both field names and nested field paths are supported. 82 83 Args: 84 target: The request structure to inject the value into 85 value: The value to inject 86 config: The config object to use for interpolation 87 """ 88 if self._is_field_path: 89 if self.inject_into != RequestOptionType.body_json: 90 raise ValueError( 91 "Nested field injection is only supported for body JSON injection. Please use a top-level field_name for other injection types." 92 ) 93 94 assert self.field_path is not None # for type checker 95 current = target 96 # Convert path segments into strings, evaluating any interpolated segments 97 # Example: ["data", "{{ config[user_type] }}", "id"] -> ["data", "admin", "id"] 98 *path_parts, final_key = [ 99 str( 100 segment.eval(config=config) 101 if isinstance(segment, InterpolatedString) 102 else segment 103 ) 104 for segment in self.field_path 105 ] 106 107 # Build a nested dictionary structure and set the final value at the deepest level 108 for part in path_parts: 109 current = current.setdefault(part, {}) 110 current[final_key] = value 111 else: 112 # For non-nested fields, evaluate the field name if it's an interpolated string 113 key = ( 114 self.field_name.eval(config=config) 115 if isinstance(self.field_name, InterpolatedString) 116 else self.field_name 117 ) 118 target[str(key)] = value
Describes an option to set on a request
Attributes:
- field_name (str): Describes the name of the parameter to inject. Mutually exclusive with field_path.
- field_path (list(str)): Describes the path to a nested field as a list of field names. Only valid for body_json injection type, and mutually exclusive with field_name.
- inject_into (RequestOptionType): Describes where in the HTTP request to inject the parameter
72 def inject_into_request( 73 self, 74 target: MutableMapping[str, Any], 75 value: Any, 76 config: Config, 77 ) -> None: 78 """ 79 Inject a request option value into a target request structure using either field_name or field_path. 80 For non-body-json injection, only top-level field names are supported. 81 For body-json injection, both field names and nested field paths are supported. 82 83 Args: 84 target: The request structure to inject the value into 85 value: The value to inject 86 config: The config object to use for interpolation 87 """ 88 if self._is_field_path: 89 if self.inject_into != RequestOptionType.body_json: 90 raise ValueError( 91 "Nested field injection is only supported for body JSON injection. Please use a top-level field_name for other injection types." 92 ) 93 94 assert self.field_path is not None # for type checker 95 current = target 96 # Convert path segments into strings, evaluating any interpolated segments 97 # Example: ["data", "{{ config[user_type] }}", "id"] -> ["data", "admin", "id"] 98 *path_parts, final_key = [ 99 str( 100 segment.eval(config=config) 101 if isinstance(segment, InterpolatedString) 102 else segment 103 ) 104 for segment in self.field_path 105 ] 106 107 # Build a nested dictionary structure and set the final value at the deepest level 108 for part in path_parts: 109 current = current.setdefault(part, {}) 110 current[final_key] = value 111 else: 112 # For non-nested fields, evaluate the field name if it's an interpolated string 113 key = ( 114 self.field_name.eval(config=config) 115 if isinstance(self.field_name, InterpolatedString) 116 else self.field_name 117 ) 118 target[str(key)] = value
Inject a request option value into a target request structure using either field_name or field_path. For non-body-json injection, only top-level field names are supported. For body-json injection, both field names and nested field paths are supported.
Arguments:
- target: The request structure to inject the value into
- value: The value to inject
- config: The config object to use for interpolation
14class RequestOptionType(Enum): 15 """ 16 Describes where to set a value on a request 17 """ 18 19 request_parameter = "request_parameter" 20 header = "header" 21 body_data = "body_data" 22 body_json = "body_json"
Describes where to set a value on a request
30class Requester(RequestOptionsProvider): 31 @abstractmethod 32 def get_authenticator(self) -> DeclarativeAuthenticator: 33 """ 34 Specifies the authenticator to use when submitting requests 35 """ 36 pass 37 38 @abstractmethod 39 def get_url( 40 self, 41 *, 42 stream_state: Optional[StreamState], 43 stream_slice: Optional[StreamSlice], 44 next_page_token: Optional[Mapping[str, Any]], 45 ) -> str: 46 """ 47 :return: URL base for the API endpoint e.g: if you wanted to hit https://myapi.com/v1/some_entity then this should return "https://myapi.com/v1/" 48 """ 49 50 @abstractmethod 51 def get_url_base( 52 self, 53 *, 54 stream_state: Optional[StreamState], 55 stream_slice: Optional[StreamSlice], 56 next_page_token: Optional[Mapping[str, Any]], 57 ) -> str: 58 """ 59 :return: URL base for the API endpoint e.g: if you wanted to hit https://myapi.com/v1/some_entity then this should return "https://myapi.com/v1/" 60 """ 61 62 @abstractmethod 63 def get_path( 64 self, 65 *, 66 stream_state: Optional[StreamState], 67 stream_slice: Optional[StreamSlice], 68 next_page_token: Optional[Mapping[str, Any]], 69 ) -> str: 70 """ 71 Returns the URL path for the API endpoint e.g: if you wanted to hit https://myapi.com/v1/some_entity then this should return "some_entity" 72 """ 73 74 @abstractmethod 75 def get_method(self) -> HttpMethod: 76 """ 77 Specifies the HTTP method to use 78 """ 79 80 @abstractmethod 81 def get_request_params( 82 self, 83 *, 84 stream_state: Optional[StreamState] = None, 85 stream_slice: Optional[StreamSlice] = None, 86 next_page_token: Optional[Mapping[str, Any]] = None, 87 ) -> MutableMapping[str, Any]: 88 """ 89 Specifies the query parameters that should be set on an outgoing HTTP request given the inputs. 90 91 E.g: you might want to define query parameters for paging if next_page_token is not None. 92 """ 93 94 @abstractmethod 95 def get_request_headers( 96 self, 97 *, 98 stream_state: Optional[StreamState] = None, 99 stream_slice: Optional[StreamSlice] = None, 100 next_page_token: Optional[Mapping[str, Any]] = None, 101 ) -> Mapping[str, Any]: 102 """ 103 Return any non-auth headers. Authentication headers will overwrite any overlapping headers returned from this method. 104 """ 105 106 @abstractmethod 107 def get_request_body_data( 108 self, 109 *, 110 stream_state: Optional[StreamState] = None, 111 stream_slice: Optional[StreamSlice] = None, 112 next_page_token: Optional[Mapping[str, Any]] = None, 113 ) -> Union[Mapping[str, Any], str]: 114 """ 115 Specifies how to populate the body of the request with a non-JSON payload. 116 117 If returns a ready text that it will be sent as is. 118 If returns a dict that it will be converted to a urlencoded form. 119 E.g. {"key1": "value1", "key2": "value2"} => "key1=value1&key2=value2" 120 121 At the same time only one of the 'request_body_data' and 'request_body_json' functions can be overridden. 122 """ 123 124 @abstractmethod 125 def get_request_body_json( 126 self, 127 *, 128 stream_state: Optional[StreamState] = None, 129 stream_slice: Optional[StreamSlice] = None, 130 next_page_token: Optional[Mapping[str, Any]] = None, 131 ) -> Mapping[str, Any]: 132 """ 133 Specifies how to populate the body of the request with a JSON payload. 134 135 At the same time only one of the 'request_body_data' and 'request_body_json' functions can be overridden. 136 """ 137 138 @abstractmethod 139 def send_request( 140 self, 141 stream_state: Optional[StreamState] = None, 142 stream_slice: Optional[StreamSlice] = None, 143 next_page_token: Optional[Mapping[str, Any]] = None, 144 path: Optional[str] = None, 145 request_headers: Optional[Mapping[str, Any]] = None, 146 request_params: Optional[Mapping[str, Any]] = None, 147 request_body_data: Optional[Union[Mapping[str, Any], str]] = None, 148 request_body_json: Optional[Mapping[str, Any]] = None, 149 log_formatter: Optional[Callable[[requests.Response], Any]] = None, 150 ) -> Optional[requests.Response]: 151 """ 152 Sends a request and returns the response. Might return no response if the error handler chooses to ignore the response or throw an exception in case of an error. 153 If path is set, the path configured on the requester itself is ignored. 154 If header, params and body are set, they are merged with the ones configured on the requester itself. 155 156 If a log formatter is provided, it's used to log the performed request and response. If it's not provided, no logging is performed. 157 """
Defines the request options to set on an outgoing HTTP request
Options can be passed by
- request parameter
- request headers
- body data
- json content
31 @abstractmethod 32 def get_authenticator(self) -> DeclarativeAuthenticator: 33 """ 34 Specifies the authenticator to use when submitting requests 35 """ 36 pass
Specifies the authenticator to use when submitting requests
38 @abstractmethod 39 def get_url( 40 self, 41 *, 42 stream_state: Optional[StreamState], 43 stream_slice: Optional[StreamSlice], 44 next_page_token: Optional[Mapping[str, Any]], 45 ) -> str: 46 """ 47 :return: URL base for the API endpoint e.g: if you wanted to hit https://myapi.com/v1/some_entity then this should return "https://myapi.com/v1/" 48 """
Returns
URL base for the API endpoint e.g: if you wanted to hit https://myapi.com/v1/some_entity then this should return "https://myapi.com/v1/"
50 @abstractmethod 51 def get_url_base( 52 self, 53 *, 54 stream_state: Optional[StreamState], 55 stream_slice: Optional[StreamSlice], 56 next_page_token: Optional[Mapping[str, Any]], 57 ) -> str: 58 """ 59 :return: URL base for the API endpoint e.g: if you wanted to hit https://myapi.com/v1/some_entity then this should return "https://myapi.com/v1/" 60 """
Returns
URL base for the API endpoint e.g: if you wanted to hit https://myapi.com/v1/some_entity then this should return "https://myapi.com/v1/"
62 @abstractmethod 63 def get_path( 64 self, 65 *, 66 stream_state: Optional[StreamState], 67 stream_slice: Optional[StreamSlice], 68 next_page_token: Optional[Mapping[str, Any]], 69 ) -> str: 70 """ 71 Returns the URL path for the API endpoint e.g: if you wanted to hit https://myapi.com/v1/some_entity then this should return "some_entity" 72 """
Returns the URL path for the API endpoint e.g: if you wanted to hit https://myapi.com/v1/some_entity then this should return "some_entity"
74 @abstractmethod 75 def get_method(self) -> HttpMethod: 76 """ 77 Specifies the HTTP method to use 78 """
Specifies the HTTP method to use
80 @abstractmethod 81 def get_request_params( 82 self, 83 *, 84 stream_state: Optional[StreamState] = None, 85 stream_slice: Optional[StreamSlice] = None, 86 next_page_token: Optional[Mapping[str, Any]] = None, 87 ) -> MutableMapping[str, Any]: 88 """ 89 Specifies the query parameters that should be set on an outgoing HTTP request given the inputs. 90 91 E.g: you might want to define query parameters for paging if next_page_token is not None. 92 """
Specifies the query parameters that should be set on an outgoing HTTP request given the inputs.
E.g: you might want to define query parameters for paging if next_page_token is not None.
94 @abstractmethod 95 def get_request_headers( 96 self, 97 *, 98 stream_state: Optional[StreamState] = None, 99 stream_slice: Optional[StreamSlice] = None, 100 next_page_token: Optional[Mapping[str, Any]] = None, 101 ) -> Mapping[str, Any]: 102 """ 103 Return any non-auth headers. Authentication headers will overwrite any overlapping headers returned from this method. 104 """
Return any non-auth headers. Authentication headers will overwrite any overlapping headers returned from this method.
106 @abstractmethod 107 def get_request_body_data( 108 self, 109 *, 110 stream_state: Optional[StreamState] = None, 111 stream_slice: Optional[StreamSlice] = None, 112 next_page_token: Optional[Mapping[str, Any]] = None, 113 ) -> Union[Mapping[str, Any], str]: 114 """ 115 Specifies how to populate the body of the request with a non-JSON payload. 116 117 If returns a ready text that it will be sent as is. 118 If returns a dict that it will be converted to a urlencoded form. 119 E.g. {"key1": "value1", "key2": "value2"} => "key1=value1&key2=value2" 120 121 At the same time only one of the 'request_body_data' and 'request_body_json' functions can be overridden. 122 """
Specifies how to populate the body of the request with a non-JSON payload.
If returns a ready text that it will be sent as is. If returns a dict that it will be converted to a urlencoded form. E.g. {"key1": "value1", "key2": "value2"} => "key1=value1&key2=value2"
At the same time only one of the 'request_body_data' and 'request_body_json' functions can be overridden.
124 @abstractmethod 125 def get_request_body_json( 126 self, 127 *, 128 stream_state: Optional[StreamState] = None, 129 stream_slice: Optional[StreamSlice] = None, 130 next_page_token: Optional[Mapping[str, Any]] = None, 131 ) -> Mapping[str, Any]: 132 """ 133 Specifies how to populate the body of the request with a JSON payload. 134 135 At the same time only one of the 'request_body_data' and 'request_body_json' functions can be overridden. 136 """
Specifies how to populate the body of the request with a JSON payload.
At the same time only one of the 'request_body_data' and 'request_body_json' functions can be overridden.
138 @abstractmethod 139 def send_request( 140 self, 141 stream_state: Optional[StreamState] = None, 142 stream_slice: Optional[StreamSlice] = None, 143 next_page_token: Optional[Mapping[str, Any]] = None, 144 path: Optional[str] = None, 145 request_headers: Optional[Mapping[str, Any]] = None, 146 request_params: Optional[Mapping[str, Any]] = None, 147 request_body_data: Optional[Union[Mapping[str, Any], str]] = None, 148 request_body_json: Optional[Mapping[str, Any]] = None, 149 log_formatter: Optional[Callable[[requests.Response], Any]] = None, 150 ) -> Optional[requests.Response]: 151 """ 152 Sends a request and returns the response. Might return no response if the error handler chooses to ignore the response or throw an exception in case of an error. 153 If path is set, the path configured on the requester itself is ignored. 154 If header, params and body are set, they are merged with the ones configured on the requester itself. 155 156 If a log formatter is provided, it's used to log the performed request and response. If it's not provided, no logging is performed. 157 """
Sends a request and returns the response. Might return no response if the error handler chooses to ignore the response or throw an exception in case of an error. If path is set, the path configured on the requester itself is ignored. If header, params and body are set, they are merged with the ones configured on the requester itself.
If a log formatter is provided, it's used to log the performed request and response. If it's not provided, no logging is performed.
84@dataclass 85class SimpleRetriever(Retriever): 86 """ 87 Retrieves records by synchronously sending requests to fetch records. 88 89 The retriever acts as an orchestrator between the requester, the record selector, the paginator, and the stream slicer. 90 91 For each stream slice, submit requests until there are no more pages of records to fetch. 92 93 This retriever currently inherits from HttpStream to reuse the request submission and pagination machinery. 94 As a result, some of the parameters passed to some methods are unused. 95 The two will be decoupled in a future release. 96 97 Attributes: 98 stream_name (str): The stream's name 99 stream_primary_key (Optional[Union[str, List[str], List[List[str]]]]): The stream's primary key 100 requester (Requester): The HTTP requester 101 record_selector (HttpSelector): The record selector 102 paginator (Optional[Paginator]): The paginator 103 stream_slicer (Optional[StreamSlicer]): The stream slicer 104 parameters (Mapping[str, Any]): Additional runtime parameters to be used for string interpolation 105 post_pagination_filter (Optional[ClientSideIncrementalRecordFilterDecorator]): Set for data feed streams only. 106 Records the cursor considers already synced are dropped once pagination has observed them 107 page_size_reduction (Optional[PageSizeReduction]): How much to shrink the page size when an error handler 108 resolves to `ResponseAction.REDUCE_PAGE_SIZE`. `None` disables page size reduction entirely. 109 It is immutable configuration; the page size in effect lives in a `PageSizeReducer` that 110 `_read_pages` creates per call, so the retriever and its paginator - both shared by every 111 partition of the stream, read concurrently - stay stateless. When `page_size_reduction` is 112 `None` no reducer is created and `_read_pages` keeps its previous behaviour 113 request_window_splitting (Optional[RequestWindowSplitting]): Policy applied when an error handler 114 resolves to `ResponseAction.SPLIT_REQUEST_WINDOW`, or when `RequestWindowSplitRequiredException` 115 is raised directly by custom code. `None` disables request-window splitting entirely: `read_records` 116 converts the exception into `RequestWindowSplitNotSupportedException` instead of attempting a 117 split it cannot perform. 118 request_window_splitter: Bound method - normally the stream's own cursor's `split_request_window` - 119 asked to replace a failing `StreamSlice` with smaller children. `None` has the same effect as 120 `request_window_splitting` being `None`. 121 """ 122 123 requester: Requester 124 record_selector: HttpSelector 125 config: Config 126 parameters: InitVar[Mapping[str, Any]] 127 name: str 128 _name: Union[InterpolatedString, str] = field(init=False, repr=False, default="") 129 primary_key: Optional[Union[str, List[str], List[List[str]]]] 130 _primary_key: str = field(init=False, repr=False, default="") 131 paginator: Optional[Paginator] = None 132 stream_slicer: StreamSlicer = field( 133 default_factory=lambda: SinglePartitionRouter(parameters={}) 134 ) 135 request_option_provider: RequestOptionsProvider = field( 136 default_factory=lambda: DefaultRequestOptionsProvider(parameters={}) 137 ) 138 ignore_stream_slicer_parameters_on_paginated_requests: bool = False 139 additional_query_properties: Optional[QueryProperties] = None 140 log_formatter: Optional[Callable[[requests.Response], Any]] = None 141 pagination_tracker_factory: Callable[[], PaginationTracker] = field( 142 default_factory=lambda: lambda: PaginationTracker() 143 ) 144 post_pagination_filter: Optional[ClientSideIncrementalRecordFilterDecorator] = None 145 page_size_reduction: Optional[PageSizeReduction] = None 146 request_window_splitting: Optional[RequestWindowSplitting] = None 147 request_window_splitter: Optional[ 148 Callable[[StreamSlice, Optional[datetime.timedelta]], Optional[List[StreamSlice]]] 149 ] = None 150 151 def __post_init__(self, parameters: Mapping[str, Any]) -> None: 152 self._paginator = self.paginator or NoPagination(parameters=parameters) 153 self._parameters = parameters 154 self._name = ( 155 InterpolatedString(self._name, parameters=parameters) 156 if isinstance(self._name, str) 157 else self._name 158 ) 159 160 @property # type: ignore 161 def name(self) -> str: 162 """ 163 :return: Stream name 164 """ 165 return ( 166 str(self._name.eval(self.config)) 167 if isinstance(self._name, InterpolatedString) 168 else self._name 169 ) 170 171 @name.setter 172 def name(self, value: str) -> None: 173 if not isinstance(value, property): 174 self._name = value 175 176 def _get_mapping( 177 self, method: Callable[..., Optional[Union[Mapping[str, Any], str]]], **kwargs: Any 178 ) -> Tuple[Union[Mapping[str, Any], str], Set[str]]: 179 """ 180 Get mapping from the provided method, and get the keys of the mapping. 181 If the method returns a string, it will return the string and an empty set. 182 If the method returns a dict, it will return the dict and its keys. 183 """ 184 mapping = method(**kwargs) or {} 185 keys = set(mapping.keys()) if not isinstance(mapping, str) else set() 186 return mapping, keys 187 188 def _get_request_options( 189 self, 190 stream_slice: Optional[StreamSlice], 191 next_page_token: Optional[Mapping[str, Any]], 192 paginator_method: Callable[..., Optional[Union[Mapping[str, Any], str]]], 193 stream_slicer_method: Callable[..., Optional[Union[Mapping[str, Any], str]]], 194 page_size_override: Optional[int] = None, 195 ) -> Union[Mapping[str, Any], str]: 196 """ 197 Get the request_option from the paginator and the stream slicer. 198 Raise a ValueError if there's a key collision 199 Returned merged mapping otherwise 200 """ 201 is_body_json = paginator_method.__name__ == "get_request_body_json" 202 203 mappings = [ 204 paginator_method( 205 stream_slice=stream_slice, 206 next_page_token=next_page_token, 207 **page_size_override_kwargs(page_size_override), 208 ), 209 ] 210 if not next_page_token or not self.ignore_stream_slicer_parameters_on_paginated_requests: 211 mappings.append( 212 stream_slicer_method( 213 stream_slice=stream_slice, 214 next_page_token=next_page_token, 215 ) 216 ) 217 return combine_mappings(mappings, allow_same_value_merge=is_body_json) 218 219 def _request_headers( 220 self, 221 stream_slice: Optional[StreamSlice] = None, 222 next_page_token: Optional[Mapping[str, Any]] = None, 223 page_size_override: Optional[int] = None, 224 ) -> Mapping[str, Any]: 225 """ 226 Specifies request headers. 227 Authentication headers will overwrite any overlapping headers returned from this method. 228 """ 229 headers = self._get_request_options( 230 stream_slice, 231 next_page_token, 232 self._paginator.get_request_headers, 233 self.request_option_provider.get_request_headers, 234 **page_size_override_kwargs(page_size_override), 235 ) 236 if isinstance(headers, str): 237 raise ValueError("Request headers cannot be a string") 238 return {str(k): str(v) for k, v in headers.items()} 239 240 def _request_params( 241 self, 242 stream_slice: Optional[StreamSlice] = None, 243 next_page_token: Optional[Mapping[str, Any]] = None, 244 page_size_override: Optional[int] = None, 245 ) -> Mapping[str, Any]: 246 """ 247 Specifies the query parameters that should be set on an outgoing HTTP request given the inputs. 248 249 E.g: you might want to define query parameters for paging if next_page_token is not None. 250 """ 251 params = self._get_request_options( 252 stream_slice, 253 next_page_token, 254 self._paginator.get_request_params, 255 self.request_option_provider.get_request_params, 256 **page_size_override_kwargs(page_size_override), 257 ) 258 if isinstance(params, str): 259 raise ValueError("Request params cannot be a string") 260 return params 261 262 def _request_body_data( 263 self, 264 stream_slice: Optional[StreamSlice] = None, 265 next_page_token: Optional[Mapping[str, Any]] = None, 266 page_size_override: Optional[int] = None, 267 ) -> Union[Mapping[str, Any], str]: 268 """ 269 Specifies how to populate the body of the request with a non-JSON payload. 270 271 If returns a ready text that it will be sent as is. 272 If returns a dict that it will be converted to a urlencoded form. 273 E.g. {"key1": "value1", "key2": "value2"} => "key1=value1&key2=value2" 274 275 At the same time only one of the 'request_body_data' and 'request_body_json' functions can be overridden. 276 """ 277 return self._get_request_options( 278 stream_slice, 279 next_page_token, 280 self._paginator.get_request_body_data, 281 self.request_option_provider.get_request_body_data, 282 **page_size_override_kwargs(page_size_override), 283 ) 284 285 def _request_body_json( 286 self, 287 stream_slice: Optional[StreamSlice] = None, 288 next_page_token: Optional[Mapping[str, Any]] = None, 289 page_size_override: Optional[int] = None, 290 ) -> Optional[Mapping[str, Any]]: 291 """ 292 Specifies how to populate the body of the request with a JSON payload. 293 294 At the same time only one of the 'request_body_data' and 'request_body_json' functions can be overridden. 295 """ 296 body_json = self._get_request_options( 297 stream_slice, 298 next_page_token, 299 self._paginator.get_request_body_json, 300 self.request_option_provider.get_request_body_json, 301 **page_size_override_kwargs(page_size_override), 302 ) 303 if isinstance(body_json, str): 304 raise ValueError("Request body json cannot be a string") 305 return body_json 306 307 def _paginator_path( 308 self, 309 next_page_token: Optional[Mapping[str, Any]] = None, 310 stream_slice: Optional[StreamSlice] = None, 311 ) -> Optional[str]: 312 """ 313 If the paginator points to a path, follow it, else return nothing so the requester is used. 314 :param next_page_token: 315 :return: 316 """ 317 return self._paginator.path( 318 next_page_token=next_page_token, 319 stream_state={}, # stream_state as an interpolation context is deprecated 320 stream_slice=stream_slice, 321 ) 322 323 def _parse_response( 324 self, 325 response: Optional[requests.Response], 326 records_schema: Mapping[str, Any], 327 stream_slice: Optional[StreamSlice] = None, 328 next_page_token: Optional[Mapping[str, Any]] = None, 329 ) -> Iterable[Record]: 330 if not response: 331 yield from [] 332 else: 333 yield from self.record_selector.select_records( 334 response=response, 335 stream_state={}, # stream_state as an interpolation context is deprecated 336 records_schema=records_schema, 337 stream_slice=stream_slice, 338 next_page_token=next_page_token, 339 ) 340 341 @property # type: ignore 342 def primary_key(self) -> Optional[Union[str, List[str], List[List[str]]]]: 343 """The stream's primary key""" 344 return self._primary_key 345 346 @primary_key.setter 347 def primary_key(self, value: str) -> None: 348 if not isinstance(value, property): 349 self._primary_key = value 350 351 def _next_page_token( 352 self, 353 response: requests.Response, 354 last_page_size: int, 355 last_record: Optional[Record], 356 last_page_token_value: Optional[Any], 357 page_size_override: Optional[int] = None, 358 stream_slice: Optional[StreamSlice] = None, 359 ) -> Optional[Mapping[str, Any]]: 360 """ 361 Specifies a pagination strategy. 362 363 The value returned from this method is passed to most other methods in this class. Use it to form a request e.g: set headers or query params. 364 365 :return: The token for the next page from the input response object. Returning None means there are no more pages to read in this response. 366 """ 367 return self._paginator.next_page_token( 368 response=response, 369 last_page_size=last_page_size, 370 last_record=last_record, 371 last_page_token_value=last_page_token_value, 372 **page_size_override_kwargs(page_size_override), 373 **stream_slice_kwargs(self._paginator.next_page_token, stream_slice), 374 ) 375 376 def _fetch_next_page( 377 self, 378 stream_slice: StreamSlice, 379 next_page_token: Optional[Mapping[str, Any]] = None, 380 page_size_override: Optional[int] = None, 381 ) -> Optional[requests.Response]: 382 return self.requester.send_request( 383 path=self._paginator_path( 384 next_page_token=next_page_token, 385 stream_slice=stream_slice, 386 ), 387 stream_state={}, # stream_state as an interpolation context is deprecated 388 stream_slice=stream_slice, 389 next_page_token=next_page_token, 390 request_headers=self._request_headers( 391 stream_slice=stream_slice, 392 next_page_token=next_page_token, 393 **page_size_override_kwargs(page_size_override), 394 ), 395 request_params=self._request_params( 396 stream_slice=stream_slice, 397 next_page_token=next_page_token, 398 **page_size_override_kwargs(page_size_override), 399 ), 400 request_body_data=self._request_body_data( 401 stream_slice=stream_slice, 402 next_page_token=next_page_token, 403 **page_size_override_kwargs(page_size_override), 404 ), 405 request_body_json=self._request_body_json( 406 stream_slice=stream_slice, 407 next_page_token=next_page_token, 408 **page_size_override_kwargs(page_size_override), 409 ), 410 log_formatter=self.log_formatter, 411 ) 412 413 # This logic is similar to _read_pages in the HttpStream class. When making changes here, consider making changes there as well. 414 def _read_pages( 415 self, 416 records_generator_fn: Callable[[Optional[requests.Response]], Iterable[Record]], 417 stream_slice: StreamSlice, 418 ) -> Iterable[Record]: 419 original_stream_slice = stream_slice 420 pagination_tracker = self.pagination_tracker_factory() 421 page_size_reducer = ( 422 PageSizeReducer( 423 self.page_size_reduction, 424 self._paginator.get_page_size(), 425 stream_name=self.name, 426 ) 427 if self.page_size_reduction 428 else None 429 ) 430 reset_pagination = False 431 reduce_page_size = False 432 next_page_token = self._get_initial_next_page_token() 433 while True: 434 page_size_override = page_size_reducer.page_size_override if page_size_reducer else None 435 merged_records: MutableMapping[str, Any] = defaultdict(dict) 436 last_page_size = 0 437 last_record: Optional[Record] = None 438 439 response = None 440 try: 441 if self.additional_query_properties: 442 for ( 443 properties 444 ) in self.additional_query_properties.get_request_property_chunks(): 445 stream_slice = StreamSlice( 446 partition=stream_slice.partition or {}, 447 cursor_slice=stream_slice.cursor_slice or {}, 448 extra_fields={"query_properties": properties}, 449 ) 450 response = self._fetch_next_page( 451 stream_slice, 452 next_page_token, 453 **page_size_override_kwargs(page_size_override), 454 ) 455 456 for current_record in records_generator_fn(response): 457 if self.additional_query_properties.property_chunking: 458 merge_key = self.additional_query_properties.property_chunking.get_merge_key( 459 current_record 460 ) 461 if merge_key: 462 _deep_merge(merged_records[merge_key], current_record) 463 else: 464 # We should still emit records even if the record did not have a merge key 465 pagination_tracker.observe(current_record) 466 last_page_size += 1 467 last_record = current_record 468 yield current_record 469 else: 470 pagination_tracker.observe(current_record) 471 last_page_size += 1 472 last_record = current_record 473 yield current_record 474 475 for merged_record in merged_records.values(): 476 record = Record( 477 data=merged_record, stream_name=self.name, associated_slice=stream_slice 478 ) 479 pagination_tracker.observe(record) 480 last_page_size += 1 481 last_record = record 482 yield record 483 else: 484 response = self._fetch_next_page( 485 stream_slice, 486 next_page_token, 487 **page_size_override_kwargs(page_size_override), 488 ) 489 for current_record in records_generator_fn(response): 490 pagination_tracker.observe(current_record) 491 last_page_size += 1 492 last_record = current_record 493 yield current_record 494 except PaginationResetRequiredException: 495 reset_pagination = True 496 except PageSizeReductionRequiredException as exception: 497 if page_size_reducer is None: 498 # The action can be attached to a requester we cannot validate at config time, such as one 499 # built by a custom error handler or a CustomRequester. Re-raise as the misconfiguration it 500 # is: the exception being handled is the neutral "the API asked for a smaller page" signal. 501 raise PageSizeReductionNotSupportedException(stream_name=self.name) 502 if last_page_size: 503 # The reduction is safe only because it re-issues a page whose records were not emitted. 504 # Every in-CDK way of reaching this raises from `_fetch_next_page`, before the record loop, 505 # and the factory rejects the manifest constructs that would not, but a custom extractor, 506 # filter or transformation can issue its own request from inside the record generator. 507 # Re-issuing the page then duplicates the records already yielded, so fail instead. 508 raise AirbyteTracedException( 509 internal_message=f"Stream {self.name} requested a page size reduction after {last_page_size} records of the page had already been emitted", 510 message=f"Stream {self.name} asked for a smaller page size in the middle of a page. The page cannot be requested again without duplicating the records already read from it. Move the REDUCE_PAGE_SIZE action to the error handler of the stream's main requester.", 511 failure_type=FailureType.config_error, 512 ) 513 # Raises once the page size cannot be reduced any further and the retries allowed at that 514 # floor are spent, which is what stops the loop when the API keeps failing. 515 page_size_reducer.reduce(exception.error_message) 516 reduce_page_size = True 517 else: 518 if page_size_reducer: 519 page_size_reducer.on_successful_page() 520 if not response: 521 break 522 523 if reduce_page_size: 524 # Retry the very same page: neither the token nor the slice change, only the page size does - 525 # and not even that once the reducer is at its floor and only waiting is left. 526 reduce_page_size = False 527 continue 528 529 if reset_pagination or pagination_tracker.has_reached_limit(): 530 next_page_token = self._get_initial_next_page_token() 531 previous_slice = stream_slice 532 stream_slice = pagination_tracker.reduce_slice_range_if_possible( 533 stream_slice, original_stream_slice 534 ) 535 LOGGER.info( 536 f"Hitting PaginationReset event. StreamSlice used will go from {previous_slice} to {stream_slice}" 537 ) 538 reset_pagination = False 539 else: 540 last_page_token_value = ( 541 next_page_token.get("next_page_token") if next_page_token else None 542 ) 543 next_page_token = self._next_page_token( 544 response=response, # type:ignore # we are breaking from the loop on the try/else if there are no response so this should be fine 545 last_page_size=last_page_size, 546 last_record=last_record, 547 last_page_token_value=last_page_token_value, 548 **page_size_override_kwargs(page_size_override), 549 **stream_slice_kwargs(self._next_page_token, stream_slice), 550 ) 551 if not next_page_token: 552 break 553 554 # Always return an empty generator just in case no records were ever yielded 555 yield from [] 556 557 def _get_initial_next_page_token(self) -> Optional[Mapping[str, Any]]: 558 initial_token = self._paginator.get_initial_token() 559 next_page_token = {"next_page_token": initial_token} if initial_token is not None else None 560 return next_page_token 561 562 def read_records( 563 self, 564 records_schema: Mapping[str, Any], 565 stream_slice: Optional[StreamSlice] = None, 566 ) -> Iterable[StreamData]: 567 """ 568 Fetch a stream's records from an HTTP API source 569 570 :param records_schema: json schema to describe record 571 :param stream_slice: The stream slice to read data for 572 :return: The records read from the API source 573 """ 574 _slice = stream_slice or StreamSlice(partition={}, cursor_slice={}) # None-check 575 yield from self._read_records_or_split_request_window( 576 records_schema, _slice, _slice, depth=0 577 ) 578 579 def _read_records_or_split_request_window( 580 self, 581 records_schema: Mapping[str, Any], 582 stream_slice: StreamSlice, 583 original_slice: StreamSlice, 584 depth: int, 585 ) -> Iterable[StreamData]: 586 """ 587 Read `stream_slice` to completion, replacing it with smaller children and recursing into each of them in 588 turn when a `RequestWindowSplitRequiredException` is raised while reading it. 589 590 Each recursive call re-enters this method - and, through it, `_read_pages` - from scratch for the child 591 slice it is given, so every child gets its own paginator token, `PaginationTracker`, and 592 `PageSizeReducer` for free: nothing here needs to reset that state explicitly. Because the whole 593 recursion lives inside the single generator `DeclarativePartition.read()` consumes, 594 `PartitionReader.process_partition()` only calls `cursor.close_partition()` once, after this generator is 595 fully exhausted - so a failure anywhere in the recursion (a child that cannot be read, or a window that 596 cannot be split any further) propagates out without ever checkpointing the original partition. 597 598 `original_slice` is threaded through the recursion unchanged so every record yielded, however deep the 599 recursion went to produce it, is re-stamped with the partition's own slice rather than the child slice it 600 was actually read against - see `_reassociate_with_original_slice`. `depth` bounds the recursion 601 independently of `request_window_splitter`'s own no-progress guard: it is enforced here so a 602 misbehaving custom cursor cannot recurse indefinitely regardless of what that guard does or does not 603 catch. 604 """ 605 record_generator = partial( 606 self._parse_records, 607 stream_slice=stream_slice, 608 records_schema=records_schema, 609 ) 610 611 emitted_count = 0 612 try: 613 records: Iterable[Mapping[str, Any]] = self._read_pages(record_generator, stream_slice) 614 if self.post_pagination_filter: 615 # A data feed paginates until it reaches a record older than the cursor, so the page that triggers the stop 616 # condition still holds already-synced records. Those are filtered here rather than in the record selector 617 # so that the paginator keeps seeing the whole page: the stop condition is evaluated on the last record of 618 # the page, which is precisely one of the records being dropped. Two consequences of filtering this late: 619 # the pagination tracker observes the dropped records, and a `file_uploader` on the record selector has 620 # already uploaded their files by the time they are dropped. 621 records = self.post_pagination_filter.filter_records( 622 records, 623 # the filter is only used for its cursor comparison, which does not read the stream state 624 stream_state={}, 625 stream_slice=stream_slice, 626 ) 627 for record in records: 628 emitted_count += 1 629 yield self._reassociate_with_original_slice(record, original_slice) 630 except RequestWindowSplitRequiredException as exception: 631 if self.request_window_splitting is None or self.request_window_splitter is None: 632 raise RequestWindowSplitNotSupportedException(stream_name=self.name) from exception 633 634 if emitted_count: 635 # Splitting and re-reading the window re-emits these records: a failed partition is never 636 # checkpointed, so the next attempt would re-emit them anyway. Always replaying makes forward 637 # progress instead of retrying the same oversized window forever; primary-key dedup downstream 638 # handles the duplicates either way. 639 LOGGER.warning( 640 f"Stream {self.name} already emitted {emitted_count} record(s) from {stream_slice} before " 641 f"it was rejected; splitting and re-reading the window may re-emit them." 642 ) 643 644 rejection = f": {exception.error_message}" if exception.error_message else "" 645 if depth >= _MAX_REQUEST_WINDOW_SPLIT_DEPTH: 646 # Logged separately from the exception below so it stays visible even if the trace message's 647 # internal_message isn't surfaced by whatever catches it. 648 LOGGER.warning( 649 f"Stream {self.name} hit the maximum request window split depth " 650 f"({_MAX_REQUEST_WINDOW_SPLIT_DEPTH}) while splitting {original_slice}." 651 ) 652 raise AirbyteTracedException( 653 internal_message=f"Stream {self.name} exceeded the maximum request window split depth of {_MAX_REQUEST_WINDOW_SPLIT_DEPTH} while splitting {original_slice}; the API still rejected {stream_slice}{rejection}", 654 # `transient_error`, so the only remediation is the connector's own, if it defined one. 655 message=self._with_request_window_failure_message( 656 f"Stream {self.name} could not split its request window to a size the API accepts " 657 f"within {_MAX_REQUEST_WINDOW_SPLIT_DEPTH} splits." 658 ), 659 failure_type=FailureType.transient_error, 660 ) from exception 661 662 children = self.request_window_splitter( 663 stream_slice, self.request_window_splitting.min_split_window 664 ) 665 if children is None: 666 min_split_window_note = ( 667 f", its configured `min_split_window` ({self.request_window_splitting.min_split_window})" 668 if self.request_window_splitting.min_split_window 669 else "" 670 ) 671 raise AirbyteTracedException( 672 internal_message=f"Stream {self.name} could not split its request window {stream_slice} any further; the API rejected it{rejection}", 673 # `transient_error` regardless of how the triggering response was classified: exhaustion is 674 # never the user's fault, matching `PageSizeReducer`'s equivalent exhaustion branch. 675 message=self._with_request_window_failure_message( 676 f"The API kept rejecting stream {self.name}'s request window even at the smallest window " 677 f"its cursor allows{min_split_window_note}, or splitting the window further would not " 678 f"make progress." 679 ), 680 failure_type=FailureType.transient_error, 681 ) from exception 682 683 LOGGER.info( 684 f"Stream {self.name}: the API rejected request window {stream_slice} (split depth {depth}); " 685 f"reducing it to {children} and reading each in turn." 686 ) 687 for child in children: 688 yield from self._read_records_or_split_request_window( 689 records_schema, child, original_slice, depth + 1 690 ) 691 692 def _with_request_window_failure_message(self, message: str) -> str: 693 """ 694 :return: the message followed by `request_window_splitting`'s configured `failure_message`, when it 695 defined one - mirroring `PageSizeReducer._with_failure_message` 696 """ 697 assert self.request_window_splitting is not None 698 failure_message = (self.request_window_splitting.failure_message or "").strip() 699 if not failure_message: 700 return message 701 return f"{message} {failure_message}" 702 703 @staticmethod 704 def _reassociate_with_original_slice( 705 record: StreamData, original_slice: StreamSlice 706 ) -> StreamData: 707 """ 708 Records read while recursing into a split child window are parsed against that child's `StreamSlice`, 709 so `RecordSelector` stamps them with the child as `associated_slice`. `DeclarativePartition.read()` 710 passes an already-built `Record` through unchanged rather than re-wrapping it, so left uncorrected the 711 child slice - not the partition's own slice - is what `ConcurrentCursor.observe()` would key its 712 per-partition bookkeeping by. `close_partition()` always looks that bookkeeping up by the partition's own 713 slice, so it would never find it, silently losing the precise most-recently-observed cursor value for a 714 split partition (recoverable in practice today only because callers of that lookup fall back to the 715 slice's end boundary when it is missing). Re-stamping every record with `original_slice` here restores 716 the same tracking a non-split read already gets, rather than relying on that fallback. 717 """ 718 if isinstance(record, Record) and record.associated_slice is not original_slice: 719 return Record( 720 data=record.data, 721 stream_name=record.stream_name, 722 associated_slice=original_slice, 723 file_reference=record.file_reference, 724 ) 725 return record 726 727 def _parse_records( 728 self, 729 response: Optional[requests.Response], 730 records_schema: Mapping[str, Any], 731 stream_slice: Optional[StreamSlice], 732 ) -> Iterable[Record]: 733 yield from self._parse_response( 734 response, 735 stream_slice=stream_slice, 736 records_schema=records_schema, 737 ) 738 739 def must_deduplicate_query_params(self) -> bool: 740 return True 741 742 @staticmethod 743 def _to_partition_key(to_serialize: Any) -> str: 744 # separators have changed in Python 3.4. To avoid being impacted by further change, we explicitly specify our own value 745 return json.dumps(to_serialize, indent=None, separators=(",", ":"), sort_keys=True)
Retrieves records by synchronously sending requests to fetch records.
The retriever acts as an orchestrator between the requester, the record selector, the paginator, and the stream slicer.
For each stream slice, submit requests until there are no more pages of records to fetch.
This retriever currently inherits from HttpStream to reuse the request submission and pagination machinery. As a result, some of the parameters passed to some methods are unused. The two will be decoupled in a future release.
Attributes:
- stream_name (str): The stream's name
- stream_primary_key (Optional[Union[str, List[str], List[List[str]]]]): The stream's primary key
- requester (Requester): The HTTP requester
- record_selector (HttpSelector): The record selector
- paginator (Optional[Paginator]): The paginator
- stream_slicer (Optional[StreamSlicer]): The stream slicer
- parameters (Mapping[str, Any]): Additional runtime parameters to be used for string interpolation
- post_pagination_filter (Optional[ClientSideIncrementalRecordFilterDecorator]): Set for data feed streams only. Records the cursor considers already synced are dropped once pagination has observed them
- page_size_reduction (Optional[PageSizeReduction]): How much to shrink the page size when an error handler
resolves to
ResponseAction.REDUCE_PAGE_SIZE.Nonedisables page size reduction entirely. It is immutable configuration; the page size in effect lives in aPageSizeReducerthat_read_pagescreates per call, so the retriever and its paginator - both shared by every partition of the stream, read concurrently - stay stateless. Whenpage_size_reductionisNoneno reducer is created and_read_pageskeeps its previous behaviour - request_window_splitting (Optional[RequestWindowSplitting]): Policy applied when an error handler
resolves to
ResponseAction.SPLIT_REQUEST_WINDOW, or whenRequestWindowSplitRequiredExceptionis raised directly by custom code.Nonedisables request-window splitting entirely:read_recordsconverts the exception intoRequestWindowSplitNotSupportedExceptioninstead of attempting a split it cannot perform. - request_window_splitter: Bound method - normally the stream's own cursor's
split_request_window- asked to replace a failingStreamSlicewith smaller children.Nonehas the same effect asrequest_window_splittingbeingNone.
160 @property # type: ignore 161 def name(self) -> str: 162 """ 163 :return: Stream name 164 """ 165 return ( 166 str(self._name.eval(self.config)) 167 if isinstance(self._name, InterpolatedString) 168 else self._name 169 )
Returns
Stream name
341 @property # type: ignore 342 def primary_key(self) -> Optional[Union[str, List[str], List[List[str]]]]: 343 """The stream's primary key""" 344 return self._primary_key
The stream's primary key
562 def read_records( 563 self, 564 records_schema: Mapping[str, Any], 565 stream_slice: Optional[StreamSlice] = None, 566 ) -> Iterable[StreamData]: 567 """ 568 Fetch a stream's records from an HTTP API source 569 570 :param records_schema: json schema to describe record 571 :param stream_slice: The stream slice to read data for 572 :return: The records read from the API source 573 """ 574 _slice = stream_slice or StreamSlice(partition={}, cursor_slice={}) # None-check 575 yield from self._read_records_or_split_request_window( 576 records_schema, _slice, _slice, depth=0 577 )
Fetch a stream's records from an HTTP API source
Parameters
- records_schema: json schema to describe record
- stream_slice: The stream slice to read data for
Returns
The records read from the API source
Inherited Members
13@dataclass 14class SinglePartitionRouter(PartitionRouter): 15 """Partition router returning only a stream slice""" 16 17 parameters: InitVar[Mapping[str, Any]] 18 19 def get_request_params( 20 self, 21 stream_state: Optional[StreamState] = None, 22 stream_slice: Optional[StreamSlice] = None, 23 next_page_token: Optional[Mapping[str, Any]] = None, 24 ) -> Mapping[str, Any]: 25 return {} 26 27 def get_request_headers( 28 self, 29 stream_state: Optional[StreamState] = None, 30 stream_slice: Optional[StreamSlice] = None, 31 next_page_token: Optional[Mapping[str, Any]] = None, 32 ) -> Mapping[str, Any]: 33 return {} 34 35 def get_request_body_data( 36 self, 37 stream_state: Optional[StreamState] = None, 38 stream_slice: Optional[StreamSlice] = None, 39 next_page_token: Optional[Mapping[str, Any]] = None, 40 ) -> Mapping[str, Any]: 41 return {} 42 43 def get_request_body_json( 44 self, 45 stream_state: Optional[StreamState] = None, 46 stream_slice: Optional[StreamSlice] = None, 47 next_page_token: Optional[Mapping[str, Any]] = None, 48 ) -> Mapping[str, Any]: 49 return {} 50 51 def stream_slices(self) -> Iterable[StreamSlice]: 52 yield StreamSlice(partition={}, cursor_slice={}) 53 54 def get_stream_state(self) -> Optional[Mapping[str, StreamState]]: 55 """ 56 SinglePartitionRouter doesn't have parent streams 57 """ 58 pass
Partition router returning only a stream slice
19 def get_request_params( 20 self, 21 stream_state: Optional[StreamState] = None, 22 stream_slice: Optional[StreamSlice] = None, 23 next_page_token: Optional[Mapping[str, Any]] = None, 24 ) -> Mapping[str, Any]: 25 return {}
Specifies the query parameters that should be set on an outgoing HTTP request given the inputs.
E.g: you might want to define query parameters for paging if next_page_token is not None.
27 def get_request_headers( 28 self, 29 stream_state: Optional[StreamState] = None, 30 stream_slice: Optional[StreamSlice] = None, 31 next_page_token: Optional[Mapping[str, Any]] = None, 32 ) -> Mapping[str, Any]: 33 return {}
Return any non-auth headers. Authentication headers will overwrite any overlapping headers returned from this method.
35 def get_request_body_data( 36 self, 37 stream_state: Optional[StreamState] = None, 38 stream_slice: Optional[StreamSlice] = None, 39 next_page_token: Optional[Mapping[str, Any]] = None, 40 ) -> Mapping[str, Any]: 41 return {}
Specifies how to populate the body of the request with a non-JSON payload.
If returns a ready text that it will be sent as is. If returns a dict that it will be converted to a urlencoded form. E.g. {"key1": "value1", "key2": "value2"} => "key1=value1&key2=value2"
At the same time only one of the 'request_body_data' and 'request_body_json' functions can be overridden.
43 def get_request_body_json( 44 self, 45 stream_state: Optional[StreamState] = None, 46 stream_slice: Optional[StreamSlice] = None, 47 next_page_token: Optional[Mapping[str, Any]] = None, 48 ) -> Mapping[str, Any]: 49 return {}
Specifies how to populate the body of the request with a JSON payload.
At the same time only one of the 'request_body_data' and 'request_body_json' functions can be overridden.
44class StopConditionPaginationStrategyDecorator(PaginationStrategy): 45 def __init__(self, _delegate: PaginationStrategy, stop_condition: PaginationStopCondition): 46 self._delegate = _delegate 47 self._stop_condition = stop_condition 48 49 def next_page_token( 50 self, 51 response: requests.Response, 52 last_page_size: int, 53 last_record: Optional[Record], 54 last_page_token_value: Optional[Any] = None, 55 page_size_override: Optional[int] = None, 56 stream_slice: Optional[StreamSlice] = None, 57 ) -> Optional[Any]: 58 # We evaluate in reverse order because the assumption is that most of the APIs using data feed structure 59 # will return records in descending order. In terms of performance/memory, we return the records lazily 60 if last_record and self._stop_condition.is_met(last_record): 61 return None 62 return self._delegate.next_page_token( 63 response, 64 last_page_size, 65 last_record, 66 last_page_token_value, 67 **page_size_override_kwargs(page_size_override), 68 **stream_slice_kwargs(self._delegate.next_page_token, stream_slice), 69 ) 70 71 def get_page_size(self) -> Optional[int]: 72 return self._delegate.get_page_size() 73 74 @property 75 def initial_token(self) -> Optional[Any]: 76 return self._delegate.initial_token
Defines how to get the next page token
One instance paginates every partition of the stream, several at once, so next_page_token must derive the
next token from its arguments, such as last_page_token_value, rather than from state kept on the instance.
49 def next_page_token( 50 self, 51 response: requests.Response, 52 last_page_size: int, 53 last_record: Optional[Record], 54 last_page_token_value: Optional[Any] = None, 55 page_size_override: Optional[int] = None, 56 stream_slice: Optional[StreamSlice] = None, 57 ) -> Optional[Any]: 58 # We evaluate in reverse order because the assumption is that most of the APIs using data feed structure 59 # will return records in descending order. In terms of performance/memory, we return the records lazily 60 if last_record and self._stop_condition.is_met(last_record): 61 return None 62 return self._delegate.next_page_token( 63 response, 64 last_page_size, 65 last_record, 66 last_page_token_value, 67 **page_size_override_kwargs(page_size_override), 68 **stream_slice_kwargs(self._delegate.next_page_token, stream_slice), 69 )
Parameters
- response: response to process
- last_page_size: the number of records read from the response
- last_record: the last record extracted from the response
- last_page_token_value: The current value of the page token made on the last request
- page_size_override: the page size that was actually requested, when it differs from the configured
one because of a
REDUCE_PAGE_SIZEresponse action. Strategies that use their configured page size as a stop condition must honor this value, else they end the pagination early and skip records. - stream_slice: the slice the page was read for. Only passed to strategies that declare it, so existing strategies defined outside of the CDK keep working unchanged.
Returns
next page token. Returns None if there are no more pages to fetch
76class StreamSlice(Mapping[str, Any]): 77 def __init__( 78 self, 79 *, 80 partition: Mapping[str, Any], 81 cursor_slice: Mapping[str, Any], 82 extra_fields: Optional[Mapping[str, Any]] = None, 83 ) -> None: 84 """ 85 :param partition: The partition keys representing a unique partition in the stream. 86 :param cursor_slice: The incremental cursor slice keys, such as dates or pagination tokens. 87 :param extra_fields: Additional fields that should not be part of the partition but passed along, such as metadata from the parent stream. 88 """ 89 self._partition = partition 90 self._cursor_slice = cursor_slice 91 self._extra_fields = extra_fields or {} 92 93 # Ensure that partition keys do not overlap with cursor slice keys 94 if partition.keys() & cursor_slice.keys(): 95 raise ValueError("Keys for partition and incremental sync cursor should not overlap") 96 97 self._stream_slice = dict(partition) | dict(cursor_slice) 98 99 @property 100 def partition(self) -> Mapping[str, Any]: 101 """Returns the partition portion of the stream slice.""" 102 p = self._partition 103 while isinstance(p, StreamSlice): 104 p = p.partition 105 return p 106 107 @property 108 def cursor_slice(self) -> Mapping[str, Any]: 109 """Returns the cursor slice portion of the stream slice.""" 110 c = self._cursor_slice 111 while isinstance(c, StreamSlice): 112 c = c.cursor_slice 113 return c 114 115 @property 116 def extra_fields(self) -> Mapping[str, Any]: 117 """Returns the extra fields that are not part of the partition.""" 118 return self._extra_fields 119 120 def __repr__(self) -> str: 121 return repr(self._stream_slice) 122 123 def __setitem__(self, key: str, value: Any) -> None: 124 raise ValueError("StreamSlice is immutable") 125 126 def __getitem__(self, key: str) -> Any: 127 return self._stream_slice[key] 128 129 def __len__(self) -> int: 130 return len(self._stream_slice) 131 132 def __iter__(self) -> Iterator[str]: 133 return iter(self._stream_slice) 134 135 def __contains__(self, item: Any) -> bool: 136 return item in self._stream_slice 137 138 def keys(self) -> KeysView[str]: 139 return self._stream_slice.keys() 140 141 def items(self) -> ItemsView[str, Any]: 142 return self._stream_slice.items() 143 144 def values(self) -> ValuesView[Any]: 145 return self._stream_slice.values() 146 147 def get(self, key: str, default: Any = None) -> Optional[Any]: 148 return self._stream_slice.get(key, default) 149 150 def __eq__(self, other: Any) -> bool: 151 if isinstance(other, dict): 152 return self._stream_slice == other 153 if isinstance(other, StreamSlice): 154 # noinspection PyProtectedMember 155 return self._partition == other._partition and self._cursor_slice == other._cursor_slice 156 return False 157 158 def __ne__(self, other: Any) -> bool: 159 return not self.__eq__(other) 160 161 def __json_serializable__(self) -> Any: 162 return self._stream_slice 163 164 def __hash__(self) -> int: 165 return SliceHasher.hash( 166 stream_slice=self._stream_slice 167 ) # no need to provide stream_name here as this is used for slicing the cursor 168 169 def __bool__(self) -> bool: 170 return bool(self._stream_slice) or bool(self._extra_fields)
A Mapping is a generic container for associating key/value pairs.
This class provides concrete generic implementations of all methods except for __getitem__, __iter__, and __len__.
77 def __init__( 78 self, 79 *, 80 partition: Mapping[str, Any], 81 cursor_slice: Mapping[str, Any], 82 extra_fields: Optional[Mapping[str, Any]] = None, 83 ) -> None: 84 """ 85 :param partition: The partition keys representing a unique partition in the stream. 86 :param cursor_slice: The incremental cursor slice keys, such as dates or pagination tokens. 87 :param extra_fields: Additional fields that should not be part of the partition but passed along, such as metadata from the parent stream. 88 """ 89 self._partition = partition 90 self._cursor_slice = cursor_slice 91 self._extra_fields = extra_fields or {} 92 93 # Ensure that partition keys do not overlap with cursor slice keys 94 if partition.keys() & cursor_slice.keys(): 95 raise ValueError("Keys for partition and incremental sync cursor should not overlap") 96 97 self._stream_slice = dict(partition) | dict(cursor_slice)
Parameters
- partition: The partition keys representing a unique partition in the stream.
- cursor_slice: The incremental cursor slice keys, such as dates or pagination tokens.
- extra_fields: Additional fields that should not be part of the partition but passed along, such as metadata from the parent stream.
99 @property 100 def partition(self) -> Mapping[str, Any]: 101 """Returns the partition portion of the stream slice.""" 102 p = self._partition 103 while isinstance(p, StreamSlice): 104 p = p.partition 105 return p
Returns the partition portion of the stream slice.
107 @property 108 def cursor_slice(self) -> Mapping[str, Any]: 109 """Returns the cursor slice portion of the stream slice.""" 110 c = self._cursor_slice 111 while isinstance(c, StreamSlice): 112 c = c.cursor_slice 113 return c
Returns the cursor slice portion of the stream slice.
115 @property 116 def extra_fields(self) -> Mapping[str, Any]: 117 """Returns the extra fields that are not part of the partition.""" 118 return self._extra_fields
Returns the extra fields that are not part of the partition.
D.items() -> a set-like object providing a view on D's items
108@dataclass 109class SubstreamPartitionRouter(PartitionRouter): 110 """ 111 Partition router that iterates over the parent's stream records and emits slices 112 Will populate the state with `partition_field` and `parent_slice` so they can be accessed by other components 113 114 Attributes: 115 parent_stream_configs (List[ParentStreamConfig]): parent streams to iterate over and their config 116 """ 117 118 parent_stream_configs: List[ParentStreamConfig] 119 config: Config 120 parameters: InitVar[Mapping[str, Any]] 121 122 def __post_init__(self, parameters: Mapping[str, Any]) -> None: 123 if not self.parent_stream_configs: 124 raise ValueError("SubstreamPartitionRouter needs at least 1 parent stream") 125 self._parameters = parameters 126 127 def get_request_params( 128 self, 129 stream_state: Optional[StreamState] = None, 130 stream_slice: Optional[StreamSlice] = None, 131 next_page_token: Optional[Mapping[str, Any]] = None, 132 ) -> Mapping[str, Any]: 133 # Pass the stream_slice from the argument, not the cursor because the cursor is updated after processing the response 134 return self._get_request_option(RequestOptionType.request_parameter, stream_slice) 135 136 def get_request_headers( 137 self, 138 stream_state: Optional[StreamState] = None, 139 stream_slice: Optional[StreamSlice] = None, 140 next_page_token: Optional[Mapping[str, Any]] = None, 141 ) -> Mapping[str, Any]: 142 # Pass the stream_slice from the argument, not the cursor because the cursor is updated after processing the response 143 return self._get_request_option(RequestOptionType.header, stream_slice) 144 145 def get_request_body_data( 146 self, 147 stream_state: Optional[StreamState] = None, 148 stream_slice: Optional[StreamSlice] = None, 149 next_page_token: Optional[Mapping[str, Any]] = None, 150 ) -> Mapping[str, Any]: 151 # Pass the stream_slice from the argument, not the cursor because the cursor is updated after processing the response 152 return self._get_request_option(RequestOptionType.body_data, stream_slice) 153 154 def get_request_body_json( 155 self, 156 stream_state: Optional[StreamState] = None, 157 stream_slice: Optional[StreamSlice] = None, 158 next_page_token: Optional[Mapping[str, Any]] = None, 159 ) -> Mapping[str, Any]: 160 # Pass the stream_slice from the argument, not the cursor because the cursor is updated after processing the response 161 return self._get_request_option(RequestOptionType.body_json, stream_slice) 162 163 def _get_request_option( 164 self, option_type: RequestOptionType, stream_slice: Optional[StreamSlice] 165 ) -> Mapping[str, Any]: 166 params: MutableMapping[str, Any] = {} 167 if stream_slice: 168 for parent_config in self.parent_stream_configs: 169 if ( 170 parent_config.request_option 171 and parent_config.request_option.inject_into == option_type 172 ): 173 key = parent_config.partition_field.eval(self.config) # type: ignore # partition_field is always casted to an interpolated string 174 value = stream_slice.get(key) 175 if value: 176 parent_config.request_option.inject_into_request(params, value, self.config) 177 return params 178 179 def stream_slices(self) -> Iterable[StreamSlice]: 180 """ 181 Iterate over each parent stream's record and create a StreamSlice for each record. 182 183 For each stream, iterate over its stream_slices. 184 For each stream slice, iterate over each record. 185 yield a stream slice for each such records. 186 187 If a parent slice contains no record, emit a slice with parent_record=None. 188 189 The template string can interpolate the following values: 190 - parent_stream_slice: mapping representing the parent's stream slice 191 - parent_record: mapping representing the parent record 192 - parent_stream_name: string representing the parent stream name 193 """ 194 if not self.parent_stream_configs: 195 yield from [] 196 else: 197 for parent_stream_config in self.parent_stream_configs: 198 parent_stream = parent_stream_config.stream 199 parent_field = parent_stream_config.parent_key.eval(self.config) # type: ignore # parent_key is always casted to an interpolated string 200 partition_field = parent_stream_config.partition_field.eval(self.config) # type: ignore # partition_field is always casted to an interpolated string 201 extra_fields = None 202 if parent_stream_config.extra_fields: 203 extra_fields = [ 204 [field_path_part.eval(self.config) for field_path_part in field_path] # type: ignore [union-attr] 205 for field_path in parent_stream_config.extra_fields 206 ] 207 208 for partition, is_last_slice in iterate_with_last_flag( 209 parent_stream.generate_partitions() 210 ): 211 if partition is None: 212 break 213 for parent_record, is_last_record_in_slice in iterate_with_last_flag( 214 partition.read() 215 ): 216 emit_slice = parent_record is not None 217 if parent_record is not None: 218 # In the previous CDK implementation, state management was done internally by the stream. 219 # However, this could cause issues when doing availability check for example as the availability 220 # check would progress the state so state management was moved outside of the read method. 221 # Hence, we need to call the cursor here. 222 # Note that we call observe and close_partition before emitting the associated record as the 223 # ConcurrentPerPartitionCursor will associate a record with the state of the stream after the 224 # record was consumed. 225 parent_stream.cursor.observe(parent_record) 226 parent_partition = ( 227 parent_record.associated_slice.partition 228 if parent_record.associated_slice 229 else {} 230 ) 231 record_data = parent_record.data 232 233 partition_value = dpath.get( 234 record_data, # type: ignore [arg-type] 235 parent_field, 236 default=None, 237 ) 238 if partition_value is None: 239 self.logger.debug( 240 f"Skipping {parent_stream.name} record with missing or null parent key '{parent_field}'" 241 ) 242 emit_slice = False 243 244 if emit_slice: 245 # Add extra fields 246 extracted_extra_fields = self._extract_extra_fields( 247 record_data, extra_fields 248 ) 249 250 if parent_stream_config.lazy_read_pointer: 251 extracted_extra_fields = { 252 "child_response": self._extract_child_response( 253 record_data, 254 parent_stream_config.lazy_read_pointer, # type: ignore[arg-type] # lazy_read_pointer type handeled in __post_init__ of parent_stream_config 255 ), 256 **extracted_extra_fields, 257 } 258 259 if is_last_record_in_slice: 260 parent_stream.cursor.close_partition(partition) 261 if is_last_slice: 262 parent_stream.cursor.ensure_at_least_one_state_emitted() 263 264 if emit_slice: 265 yield StreamSlice( 266 partition={ 267 partition_field: partition_value, 268 "parent_slice": parent_partition or {}, 269 }, 270 cursor_slice={}, 271 extra_fields=extracted_extra_fields, 272 ) 273 274 yield from [] 275 276 def _extract_child_response( 277 self, parent_record: Mapping[str, Any] | AirbyteMessage, pointer: List[InterpolatedString] 278 ) -> requests.Response: 279 """Extract child records from a parent record based on lazy pointers.""" 280 281 def _create_response(data: MutableMapping[str, Any]) -> SafeResponse: 282 """Create a SafeResponse with the given data.""" 283 response = SafeResponse() 284 response.content = json.dumps(data).encode("utf-8") 285 response.status_code = 200 286 return response 287 288 path = [path.eval(self.config) for path in pointer] 289 return _create_response(dpath.get(parent_record, path, default=[])) # type: ignore # argunet will be a MutableMapping, given input data structure 290 291 def _extract_extra_fields( 292 self, 293 parent_record: Mapping[str, Any] | AirbyteMessage, 294 extra_fields: Optional[List[List[str]]] = None, 295 ) -> Mapping[str, Any]: 296 """ 297 Extracts additional fields specified by their paths from the parent record. 298 299 Args: 300 parent_record (Mapping[str, Any]): The record from the parent stream to extract fields from. 301 extra_fields (Optional[List[List[str]]]): A list of field paths (as lists of strings) to extract from the parent record. 302 303 Returns: 304 Mapping[str, Any]: A dictionary containing the extracted fields. 305 The keys are the joined field paths, and the values are the corresponding extracted values. 306 """ 307 extracted_extra_fields = {} 308 if extra_fields: 309 for extra_field_path in extra_fields: 310 try: 311 extra_field_value = dpath.get( 312 parent_record, # type: ignore [arg-type] 313 extra_field_path, 314 ) 315 self.logger.debug( 316 f"Extracted extra_field_path: {extra_field_path} with value: {extra_field_value}" 317 ) 318 except KeyError: 319 self.logger.debug(f"Failed to extract extra_field_path: {extra_field_path}") 320 extra_field_value = None 321 extracted_extra_fields[".".join(extra_field_path)] = extra_field_value 322 return extracted_extra_fields 323 324 def get_stream_state(self) -> Optional[Mapping[str, StreamState]]: 325 """ 326 Get the state of the parent streams. 327 328 Returns: 329 StreamState: The current state of the parent streams. 330 331 Example of state format: 332 { 333 "parent_stream_name1": { 334 "last_updated": "2023-05-27T00:00:00Z" 335 }, 336 "parent_stream_name2": { 337 "last_updated": "2023-05-27T00:00:00Z" 338 } 339 } 340 """ 341 parent_state = {} 342 for parent_config in self.parent_stream_configs: 343 if parent_config.incremental_dependency: 344 parent_state[parent_config.stream.name] = copy.deepcopy( 345 parent_config.stream.cursor.state 346 ) 347 return parent_state 348 349 @property 350 def logger(self) -> logging.Logger: 351 return logging.getLogger("airbyte.SubstreamPartitionRouter")
Partition router that iterates over the parent's stream records and emits slices
Will populate the state with partition_field and parent_slice so they can be accessed by other components
Attributes:
- parent_stream_configs (List[ParentStreamConfig]): parent streams to iterate over and their config
127 def get_request_params( 128 self, 129 stream_state: Optional[StreamState] = None, 130 stream_slice: Optional[StreamSlice] = None, 131 next_page_token: Optional[Mapping[str, Any]] = None, 132 ) -> Mapping[str, Any]: 133 # Pass the stream_slice from the argument, not the cursor because the cursor is updated after processing the response 134 return self._get_request_option(RequestOptionType.request_parameter, stream_slice)
Specifies the query parameters that should be set on an outgoing HTTP request given the inputs.
E.g: you might want to define query parameters for paging if next_page_token is not None.
136 def get_request_headers( 137 self, 138 stream_state: Optional[StreamState] = None, 139 stream_slice: Optional[StreamSlice] = None, 140 next_page_token: Optional[Mapping[str, Any]] = None, 141 ) -> Mapping[str, Any]: 142 # Pass the stream_slice from the argument, not the cursor because the cursor is updated after processing the response 143 return self._get_request_option(RequestOptionType.header, stream_slice)
Return any non-auth headers. Authentication headers will overwrite any overlapping headers returned from this method.
145 def get_request_body_data( 146 self, 147 stream_state: Optional[StreamState] = None, 148 stream_slice: Optional[StreamSlice] = None, 149 next_page_token: Optional[Mapping[str, Any]] = None, 150 ) -> Mapping[str, Any]: 151 # Pass the stream_slice from the argument, not the cursor because the cursor is updated after processing the response 152 return self._get_request_option(RequestOptionType.body_data, stream_slice)
Specifies how to populate the body of the request with a non-JSON payload.
If returns a ready text that it will be sent as is. If returns a dict that it will be converted to a urlencoded form. E.g. {"key1": "value1", "key2": "value2"} => "key1=value1&key2=value2"
At the same time only one of the 'request_body_data' and 'request_body_json' functions can be overridden.
154 def get_request_body_json( 155 self, 156 stream_state: Optional[StreamState] = None, 157 stream_slice: Optional[StreamSlice] = None, 158 next_page_token: Optional[Mapping[str, Any]] = None, 159 ) -> Mapping[str, Any]: 160 # Pass the stream_slice from the argument, not the cursor because the cursor is updated after processing the response 161 return self._get_request_option(RequestOptionType.body_json, stream_slice)
Specifies how to populate the body of the request with a JSON payload.
At the same time only one of the 'request_body_data' and 'request_body_json' functions can be overridden.
179 def stream_slices(self) -> Iterable[StreamSlice]: 180 """ 181 Iterate over each parent stream's record and create a StreamSlice for each record. 182 183 For each stream, iterate over its stream_slices. 184 For each stream slice, iterate over each record. 185 yield a stream slice for each such records. 186 187 If a parent slice contains no record, emit a slice with parent_record=None. 188 189 The template string can interpolate the following values: 190 - parent_stream_slice: mapping representing the parent's stream slice 191 - parent_record: mapping representing the parent record 192 - parent_stream_name: string representing the parent stream name 193 """ 194 if not self.parent_stream_configs: 195 yield from [] 196 else: 197 for parent_stream_config in self.parent_stream_configs: 198 parent_stream = parent_stream_config.stream 199 parent_field = parent_stream_config.parent_key.eval(self.config) # type: ignore # parent_key is always casted to an interpolated string 200 partition_field = parent_stream_config.partition_field.eval(self.config) # type: ignore # partition_field is always casted to an interpolated string 201 extra_fields = None 202 if parent_stream_config.extra_fields: 203 extra_fields = [ 204 [field_path_part.eval(self.config) for field_path_part in field_path] # type: ignore [union-attr] 205 for field_path in parent_stream_config.extra_fields 206 ] 207 208 for partition, is_last_slice in iterate_with_last_flag( 209 parent_stream.generate_partitions() 210 ): 211 if partition is None: 212 break 213 for parent_record, is_last_record_in_slice in iterate_with_last_flag( 214 partition.read() 215 ): 216 emit_slice = parent_record is not None 217 if parent_record is not None: 218 # In the previous CDK implementation, state management was done internally by the stream. 219 # However, this could cause issues when doing availability check for example as the availability 220 # check would progress the state so state management was moved outside of the read method. 221 # Hence, we need to call the cursor here. 222 # Note that we call observe and close_partition before emitting the associated record as the 223 # ConcurrentPerPartitionCursor will associate a record with the state of the stream after the 224 # record was consumed. 225 parent_stream.cursor.observe(parent_record) 226 parent_partition = ( 227 parent_record.associated_slice.partition 228 if parent_record.associated_slice 229 else {} 230 ) 231 record_data = parent_record.data 232 233 partition_value = dpath.get( 234 record_data, # type: ignore [arg-type] 235 parent_field, 236 default=None, 237 ) 238 if partition_value is None: 239 self.logger.debug( 240 f"Skipping {parent_stream.name} record with missing or null parent key '{parent_field}'" 241 ) 242 emit_slice = False 243 244 if emit_slice: 245 # Add extra fields 246 extracted_extra_fields = self._extract_extra_fields( 247 record_data, extra_fields 248 ) 249 250 if parent_stream_config.lazy_read_pointer: 251 extracted_extra_fields = { 252 "child_response": self._extract_child_response( 253 record_data, 254 parent_stream_config.lazy_read_pointer, # type: ignore[arg-type] # lazy_read_pointer type handeled in __post_init__ of parent_stream_config 255 ), 256 **extracted_extra_fields, 257 } 258 259 if is_last_record_in_slice: 260 parent_stream.cursor.close_partition(partition) 261 if is_last_slice: 262 parent_stream.cursor.ensure_at_least_one_state_emitted() 263 264 if emit_slice: 265 yield StreamSlice( 266 partition={ 267 partition_field: partition_value, 268 "parent_slice": parent_partition or {}, 269 }, 270 cursor_slice={}, 271 extra_fields=extracted_extra_fields, 272 ) 273 274 yield from []
Iterate over each parent stream's record and create a StreamSlice for each record.
For each stream, iterate over its stream_slices. For each stream slice, iterate over each record. yield a stream slice for each such records.
If a parent slice contains no record, emit a slice with parent_record=None.
The template string can interpolate the following values:
- parent_stream_slice: mapping representing the parent's stream slice
- parent_record: mapping representing the parent record
- parent_stream_name: string representing the parent stream name
324 def get_stream_state(self) -> Optional[Mapping[str, StreamState]]: 325 """ 326 Get the state of the parent streams. 327 328 Returns: 329 StreamState: The current state of the parent streams. 330 331 Example of state format: 332 { 333 "parent_stream_name1": { 334 "last_updated": "2023-05-27T00:00:00Z" 335 }, 336 "parent_stream_name2": { 337 "last_updated": "2023-05-27T00:00:00Z" 338 } 339 } 340 """ 341 parent_state = {} 342 for parent_config in self.parent_stream_configs: 343 if parent_config.incremental_dependency: 344 parent_state[parent_config.stream.name] = copy.deepcopy( 345 parent_config.stream.cursor.state 346 ) 347 return parent_state
Get the state of the parent streams.
Returns:
StreamState: The current state of the parent streams.
Example of state format: { "parent_stream_name1": { "last_updated": "2023-05-27T00:00:00Z" }, "parent_stream_name2": { "last_updated": "2023-05-27T00:00:00Z" } }
16@dataclass 17class UnionPartitionRouter(PartitionRouter): 18 """ 19 A partition router that yields the deduplicated union of its child partition routers' slices. 20 21 Each emitted slice's partition is normalized to exactly `{partition_field: value}`. Any other 22 partition keys coming from a child router (e.g. a SubstreamPartitionRouter's `parent_slice`) 23 are moved into `extra_fields` so that per-partition state keys stay stable across children. 24 25 Given two child routers producing: 26 A: [{"repository": "org/a"}, {"repository": "org/b"}] 27 B: [{"repository": "org/b", "parent_slice": {...}}, {"repository": "org/c", "parent_slice": {...}}] 28 the union yields: 29 [{"repository": "org/a"}, {"repository": "org/b"}, {"repository": "org/c"}] 30 31 Partition values must be hashable — deduplication holds every distinct value in a set 32 for the duration of slice generation, so memory grows linearly with the number of 33 unique partitions. Note also that a union's partition count is the (deduplicated) sum 34 of its children's, which counts against `ConcurrentPerPartitionCursor`'s per-partition 35 state limit. 36 37 Attributes: 38 partition_routers (List[PartitionRouter]): The child partition routers to union. 39 partition_field (str): The single partition key all child slices are normalized to. 40 """ 41 42 partition_routers: List[PartitionRouter] 43 partition_field: str 44 parameters: InitVar[Mapping[str, Any]] 45 46 def __post_init__(self, parameters: Mapping[str, Any]) -> None: 47 self._parameters = parameters 48 49 def stream_slices(self) -> Iterable[StreamSlice]: 50 """ 51 Iterate over the child partition routers in order, yielding each partition value once. 52 53 The first occurrence of a partition value wins; later duplicates from any child are skipped. 54 """ 55 # The concurrent framework consumes stream_slices() lazily, so this set stays live 56 # (and grows with each distinct partition value) for essentially the whole sync. 57 seen: set[Any] = set() 58 for router in self.partition_routers: 59 for stream_slice in router.stream_slices(): 60 if self.partition_field not in stream_slice.partition: 61 raise ValueError( 62 f"UnionPartitionRouter expects all child partition routers to emit the " 63 f"partition field '{self.partition_field}'. Got {stream_slice.partition}" 64 ) 65 value = stream_slice.partition[self.partition_field] 66 try: 67 if value in seen: 68 continue 69 seen.add(value) 70 except TypeError as exception: 71 raise ValueError( 72 f"UnionPartitionRouter can only deduplicate hashable partition values. " 73 f"Got unhashable value {value!r} for partition field '{self.partition_field}'." 74 ) from exception 75 carried_over_fields = { 76 key: field_value 77 for key, field_value in stream_slice.partition.items() 78 if key != self.partition_field 79 } 80 yield StreamSlice( 81 partition={self.partition_field: value}, 82 cursor_slice={}, 83 extra_fields={ 84 **(dict(stream_slice.extra_fields) if stream_slice.extra_fields else {}), 85 **carried_over_fields, 86 }, 87 ) 88 89 def get_request_params( 90 self, 91 *, 92 stream_state: Optional[StreamState] = None, 93 stream_slice: Optional[StreamSlice] = None, 94 next_page_token: Optional[Mapping[str, Any]] = None, 95 ) -> Mapping[str, Any]: 96 return {} 97 98 def get_request_headers( 99 self, 100 *, 101 stream_state: Optional[StreamState] = None, 102 stream_slice: Optional[StreamSlice] = None, 103 next_page_token: Optional[Mapping[str, Any]] = None, 104 ) -> Mapping[str, Any]: 105 return {} 106 107 def get_request_body_data( 108 self, 109 *, 110 stream_state: Optional[StreamState] = None, 111 stream_slice: Optional[StreamSlice] = None, 112 next_page_token: Optional[Mapping[str, Any]] = None, 113 ) -> Mapping[str, Any]: 114 return {} 115 116 def get_request_body_json( 117 self, 118 *, 119 stream_state: Optional[StreamState] = None, 120 stream_slice: Optional[StreamSlice] = None, 121 next_page_token: Optional[Mapping[str, Any]] = None, 122 ) -> Mapping[str, Any]: 123 return {} 124 125 def get_stream_state(self) -> Optional[Mapping[str, StreamState]]: 126 """ 127 Merge the parent stream states of all child partition routers. 128 129 States are merged with a shallow dict update, which assumes no two child routers 130 reference a parent stream with the same name. If they do, the last child's state 131 for that parent wins and a warning is logged. 132 """ 133 merged_state: dict[str, StreamState] = {} 134 for router in self.partition_routers: 135 child_state = router.get_stream_state() 136 if child_state: 137 colliding_keys = merged_state.keys() & child_state.keys() 138 if colliding_keys: 139 logger.warning( 140 f"UnionPartitionRouter child routers reference the same parent stream(s) " 141 f"{sorted(colliding_keys)}; the last child's state for each colliding parent " 142 f"stream overwrites the previous one." 143 ) 144 merged_state.update(child_state) 145 return merged_state
A partition router that yields the deduplicated union of its child partition routers' slices.
Each emitted slice's partition is normalized to exactly {partition_field: value}. Any other
partition keys coming from a child router (e.g. a SubstreamPartitionRouter's parent_slice)
are moved into extra_fields so that per-partition state keys stay stable across children.
Given two child routers producing: A: [{"repository": "org/a"}, {"repository": "org/b"}] B: [{"repository": "org/b", "parent_slice": {...}}, {"repository": "org/c", "parent_slice": {...}}] the union yields: [{"repository": "org/a"}, {"repository": "org/b"}, {"repository": "org/c"}]
Partition values must be hashable — deduplication holds every distinct value in a set
for the duration of slice generation, so memory grows linearly with the number of
unique partitions. Note also that a union's partition count is the (deduplicated) sum
of its children's, which counts against ConcurrentPerPartitionCursor's per-partition
state limit.
Attributes:
- partition_routers (List[PartitionRouter]): The child partition routers to union.
- partition_field (str): The single partition key all child slices are normalized to.
49 def stream_slices(self) -> Iterable[StreamSlice]: 50 """ 51 Iterate over the child partition routers in order, yielding each partition value once. 52 53 The first occurrence of a partition value wins; later duplicates from any child are skipped. 54 """ 55 # The concurrent framework consumes stream_slices() lazily, so this set stays live 56 # (and grows with each distinct partition value) for essentially the whole sync. 57 seen: set[Any] = set() 58 for router in self.partition_routers: 59 for stream_slice in router.stream_slices(): 60 if self.partition_field not in stream_slice.partition: 61 raise ValueError( 62 f"UnionPartitionRouter expects all child partition routers to emit the " 63 f"partition field '{self.partition_field}'. Got {stream_slice.partition}" 64 ) 65 value = stream_slice.partition[self.partition_field] 66 try: 67 if value in seen: 68 continue 69 seen.add(value) 70 except TypeError as exception: 71 raise ValueError( 72 f"UnionPartitionRouter can only deduplicate hashable partition values. " 73 f"Got unhashable value {value!r} for partition field '{self.partition_field}'." 74 ) from exception 75 carried_over_fields = { 76 key: field_value 77 for key, field_value in stream_slice.partition.items() 78 if key != self.partition_field 79 } 80 yield StreamSlice( 81 partition={self.partition_field: value}, 82 cursor_slice={}, 83 extra_fields={ 84 **(dict(stream_slice.extra_fields) if stream_slice.extra_fields else {}), 85 **carried_over_fields, 86 }, 87 )
Iterate over the child partition routers in order, yielding each partition value once.
The first occurrence of a partition value wins; later duplicates from any child are skipped.
89 def get_request_params( 90 self, 91 *, 92 stream_state: Optional[StreamState] = None, 93 stream_slice: Optional[StreamSlice] = None, 94 next_page_token: Optional[Mapping[str, Any]] = None, 95 ) -> Mapping[str, Any]: 96 return {}
Specifies the query parameters that should be set on an outgoing HTTP request given the inputs.
E.g: you might want to define query parameters for paging if next_page_token is not None.
98 def get_request_headers( 99 self, 100 *, 101 stream_state: Optional[StreamState] = None, 102 stream_slice: Optional[StreamSlice] = None, 103 next_page_token: Optional[Mapping[str, Any]] = None, 104 ) -> Mapping[str, Any]: 105 return {}
Return any non-auth headers. Authentication headers will overwrite any overlapping headers returned from this method.
107 def get_request_body_data( 108 self, 109 *, 110 stream_state: Optional[StreamState] = None, 111 stream_slice: Optional[StreamSlice] = None, 112 next_page_token: Optional[Mapping[str, Any]] = None, 113 ) -> Mapping[str, Any]: 114 return {}
Specifies how to populate the body of the request with a non-JSON payload.
If returns a ready text that it will be sent as is. If returns a dict that it will be converted to a urlencoded form. E.g. {"key1": "value1", "key2": "value2"} => "key1=value1&key2=value2"
At the same time only one of the 'request_body_data' and 'request_body_json' functions can be overridden.
116 def get_request_body_json( 117 self, 118 *, 119 stream_state: Optional[StreamState] = None, 120 stream_slice: Optional[StreamSlice] = None, 121 next_page_token: Optional[Mapping[str, Any]] = None, 122 ) -> Mapping[str, Any]: 123 return {}
Specifies how to populate the body of the request with a JSON payload.
At the same time only one of the 'request_body_data' and 'request_body_json' functions can be overridden.
125 def get_stream_state(self) -> Optional[Mapping[str, StreamState]]: 126 """ 127 Merge the parent stream states of all child partition routers. 128 129 States are merged with a shallow dict update, which assumes no two child routers 130 reference a parent stream with the same name. If they do, the last child's state 131 for that parent wins and a warning is logged. 132 """ 133 merged_state: dict[str, StreamState] = {} 134 for router in self.partition_routers: 135 child_state = router.get_stream_state() 136 if child_state: 137 colliding_keys = merged_state.keys() & child_state.keys() 138 if colliding_keys: 139 logger.warning( 140 f"UnionPartitionRouter child routers reference the same parent stream(s) " 141 f"{sorted(colliding_keys)}; the last child's state for each colliding parent " 142 f"stream overwrites the previous one." 143 ) 144 merged_state.update(child_state) 145 return merged_state
Merge the parent stream states of all child partition routers.
States are merged with a shallow dict update, which assumes no two child routers reference a parent stream with the same name. If they do, the last child's state for that parent wins and a warning is logged.
18class YamlDeclarativeSource(ConcurrentDeclarativeSource): 19 """Declarative source defined by a yaml file""" 20 21 def __init__( 22 self, 23 path_to_yaml: str, 24 debug: bool = False, 25 catalog: Optional[ConfiguredAirbyteCatalog] = None, 26 config: Optional[Mapping[str, Any]] = None, 27 state: Optional[List[AirbyteStateMessage]] = None, 28 config_path: Optional[str] = None, 29 ) -> None: 30 """ 31 :param path_to_yaml: Path to the yaml file describing the source 32 """ 33 self._path_to_yaml = path_to_yaml 34 source_config = self._read_and_parse_yaml_file(path_to_yaml) 35 36 super().__init__( 37 catalog=catalog or ConfiguredAirbyteCatalog(streams=[]), 38 config=config or {}, 39 state=state or [], 40 source_config=source_config, 41 config_path=config_path, 42 ) 43 44 def _read_and_parse_yaml_file(self, path_to_yaml_file: str) -> ConnectionDefinition: 45 try: 46 # For testing purposes, we want to allow to just pass a file 47 with open(path_to_yaml_file, "r") as f: 48 return yaml.safe_load(f) # type: ignore # we assume the yaml represents a ConnectionDefinition 49 except FileNotFoundError: 50 # Running inside the container, the working directory during an operation is not structured the same as the static files 51 package = self.__class__.__module__.split(".")[0] 52 53 yaml_config = pkgutil.get_data(package, path_to_yaml_file) 54 if yaml_config: 55 decoded_yaml = yaml_config.decode() 56 return self._parse(decoded_yaml) 57 return {} 58 59 def _emit_manifest_debug_message(self, extra_args: dict[str, Any]) -> None: 60 extra_args["path_to_yaml"] = self._path_to_yaml 61 62 @staticmethod 63 def _parse(connection_definition_str: str) -> ConnectionDefinition: 64 """ 65 Parses a yaml file into a manifest. Component references still exist in the manifest which will be 66 resolved during the creating of the DeclarativeSource. 67 :param connection_definition_str: yaml string to parse 68 :return: The ConnectionDefinition parsed from connection_definition_str 69 """ 70 return yaml.safe_load(connection_definition_str) # type: ignore # yaml.safe_load doesn't return a type but know it is a Mapping
Declarative source defined by a yaml file
21 def __init__( 22 self, 23 path_to_yaml: str, 24 debug: bool = False, 25 catalog: Optional[ConfiguredAirbyteCatalog] = None, 26 config: Optional[Mapping[str, Any]] = None, 27 state: Optional[List[AirbyteStateMessage]] = None, 28 config_path: Optional[str] = None, 29 ) -> None: 30 """ 31 :param path_to_yaml: Path to the yaml file describing the source 32 """ 33 self._path_to_yaml = path_to_yaml 34 source_config = self._read_and_parse_yaml_file(path_to_yaml) 35 36 super().__init__( 37 catalog=catalog or ConfiguredAirbyteCatalog(streams=[]), 38 config=config or {}, 39 state=state or [], 40 source_config=source_config, 41 config_path=config_path, 42 )
Parameters
- path_to_yaml: Path to the yaml file describing the source
Inherited Members
389def launch(source: Source, args: List[str]) -> None: 390 source_entrypoint = AirbyteEntrypoint(source) 391 parsed_args = source_entrypoint.parse_args(args) 392 # temporarily removes the PrintBuffer because we're seeing weird print behavior for concurrent syncs 393 # Refer to: https://github.com/airbytehq/oncall/issues/6235 394 with PRINT_BUFFER: 395 for message in source_entrypoint.run(parsed_args): 396 # simply printing is creating issues for concurrent CDK as Python uses different two instructions to print: one for the message and 397 # the other for the break line. Adding `\n` to the message ensure that both are printed at the same time 398 print(f"{message}\n", end="")
55class AirbyteEntrypoint(object): 56 def __init__(self, source: Source): 57 init_uncaught_exception_handler(logger) 58 59 # Deployment mode is read when instantiating the entrypoint because it is the common path shared by syncs and connector builder test requests 60 if is_cloud_environment(): 61 _init_internal_request_filter() 62 63 self.source = source 64 self.logger = logging.getLogger(f"airbyte.{getattr(source, 'name', '')}") 65 self._memory_monitor = MemoryMonitor() 66 67 @staticmethod 68 def parse_args(args: List[str]) -> argparse.Namespace: 69 # set up parent parsers 70 parent_parser = argparse.ArgumentParser(add_help=False) 71 parent_parser.add_argument( 72 "--debug", action="store_true", help="enables detailed debug logs related to the sync" 73 ) 74 main_parser = argparse.ArgumentParser() 75 subparsers = main_parser.add_subparsers(title="commands", dest="command") 76 77 # spec 78 subparsers.add_parser( 79 "spec", help="outputs the json configuration specification", parents=[parent_parser] 80 ) 81 82 # check 83 check_parser = subparsers.add_parser( 84 "check", help="checks the config can be used to connect", parents=[parent_parser] 85 ) 86 required_check_parser = check_parser.add_argument_group("required named arguments") 87 required_check_parser.add_argument( 88 "--config", type=str, required=True, help="path to the json configuration file" 89 ) 90 check_parser.add_argument( 91 "--manifest-path", 92 type=str, 93 required=False, 94 help="path to the YAML manifest file to inject into the config", 95 ) 96 check_parser.add_argument( 97 "--components-path", 98 type=str, 99 required=False, 100 help="path to the custom components file, if it exists", 101 ) 102 103 # discover 104 discover_parser = subparsers.add_parser( 105 "discover", 106 help="outputs a catalog describing the source's schema", 107 parents=[parent_parser], 108 ) 109 required_discover_parser = discover_parser.add_argument_group("required named arguments") 110 required_discover_parser.add_argument( 111 "--config", type=str, required=True, help="path to the json configuration file" 112 ) 113 discover_parser.add_argument( 114 "--manifest-path", 115 type=str, 116 required=False, 117 help="path to the YAML manifest file to inject into the config", 118 ) 119 discover_parser.add_argument( 120 "--components-path", 121 type=str, 122 required=False, 123 help="path to the custom components file, if it exists", 124 ) 125 126 # read 127 read_parser = subparsers.add_parser( 128 "read", help="reads the source and outputs messages to STDOUT", parents=[parent_parser] 129 ) 130 131 read_parser.add_argument( 132 "--state", type=str, required=False, help="path to the json-encoded state file" 133 ) 134 required_read_parser = read_parser.add_argument_group("required named arguments") 135 required_read_parser.add_argument( 136 "--config", type=str, required=True, help="path to the json configuration file" 137 ) 138 required_read_parser.add_argument( 139 "--catalog", 140 type=str, 141 required=True, 142 help="path to the catalog used to determine which data to read", 143 ) 144 read_parser.add_argument( 145 "--manifest-path", 146 type=str, 147 required=False, 148 help="path to the YAML manifest file to inject into the config", 149 ) 150 read_parser.add_argument( 151 "--components-path", 152 type=str, 153 required=False, 154 help="path to the custom components file, if it exists", 155 ) 156 157 return main_parser.parse_args(args) 158 159 def run(self, parsed_args: argparse.Namespace) -> Iterable[str]: 160 cmd = parsed_args.command 161 if not cmd: 162 raise Exception("No command passed") 163 164 if (hasattr(parsed_args, "debug") and parsed_args.debug) or is_platform_debug_log_enabled(): 165 self.logger.setLevel(logging.DEBUG) 166 logger.setLevel(logging.DEBUG) 167 self.logger.debug("Debug logs enabled") 168 else: 169 self.logger.setLevel(logging.INFO) 170 171 source_spec: ConnectorSpecification = self.source.spec(self.logger) 172 try: 173 with tempfile.TemporaryDirectory( 174 # Cleanup can fail on Windows due to file locks. Ignore if so, 175 # rather than failing the whole process. 176 ignore_cleanup_errors=True, 177 ) as temp_dir: 178 os.environ[ENV_REQUEST_CACHE_PATH] = ( 179 temp_dir # set this as default directory for request_cache to store *.sqlite files 180 ) 181 if cmd == "spec": 182 message = AirbyteMessage(type=Type.SPEC, spec=source_spec) 183 yield from [ 184 self.airbyte_message_to_string(queued_message) 185 for queued_message in self._emit_queued_messages(self.source) 186 ] 187 yield self.airbyte_message_to_string(message) 188 else: 189 raw_config = self.source.read_config(parsed_args.config) 190 config = self.source.configure(raw_config, temp_dir) 191 192 yield from [ 193 self.airbyte_message_to_string(queued_message) 194 for queued_message in self._emit_queued_messages(self.source) 195 ] 196 if cmd == "check": 197 yield from map( 198 AirbyteEntrypoint.airbyte_message_to_string, 199 self.check(source_spec, config), 200 ) 201 elif cmd == "discover": 202 yield from map( 203 AirbyteEntrypoint.airbyte_message_to_string, 204 self.discover(source_spec, config), 205 ) 206 elif cmd == "read": 207 config_catalog = self.source.read_catalog(parsed_args.catalog) 208 state = self.source.read_state(parsed_args.state) 209 210 yield from map( 211 AirbyteEntrypoint.airbyte_message_to_string, 212 self.read(source_spec, config, config_catalog, state), 213 ) 214 else: 215 raise Exception("Unexpected command " + cmd) 216 finally: 217 yield from [ 218 self.airbyte_message_to_string(queued_message) 219 for queued_message in self._emit_queued_messages(self.source) 220 ] 221 222 def check( 223 self, source_spec: ConnectorSpecification, config: TConfig 224 ) -> Iterable[AirbyteMessage]: 225 self.set_up_secret_filter(config, source_spec.connectionSpecification) 226 try: 227 self.validate_connection(source_spec, config) 228 except AirbyteTracedException as traced_exc: 229 connection_status = traced_exc.as_connection_status_message() 230 # The platform uses the exit code to surface unexpected failures so we raise the exception if the failure type not a config error 231 # If the failure is not exceptional, we'll emit a failed connection status message and return 232 if traced_exc.failure_type != FailureType.config_error: 233 raise traced_exc 234 if connection_status: 235 yield from self._emit_queued_messages(self.source) 236 yield connection_status 237 return 238 239 try: 240 check_result = self.source.check(self.logger, config) 241 except AirbyteTracedException as traced_exc: 242 yield traced_exc.as_sanitized_airbyte_message() 243 # The platform uses the exit code to surface unexpected failures so we raise the exception if the failure type not a config error 244 # If the failure is not exceptional, we'll emit a failed connection status message and return 245 if traced_exc.failure_type != FailureType.config_error: 246 raise traced_exc 247 else: 248 connection_status = traced_exc.as_connection_status_message() 249 if connection_status: 250 yield connection_status 251 return 252 if check_result.status == Status.SUCCEEDED: 253 self.logger.info("Check succeeded") 254 else: 255 self.logger.error("Check failed") 256 257 yield from self._emit_queued_messages(self.source) 258 yield AirbyteMessage(type=Type.CONNECTION_STATUS, connectionStatus=check_result) 259 260 def discover( 261 self, source_spec: ConnectorSpecification, config: TConfig 262 ) -> Iterable[AirbyteMessage]: 263 self.set_up_secret_filter(config, source_spec.connectionSpecification) 264 if self.source.check_config_against_spec: 265 self.validate_connection(source_spec, config) 266 catalog = self.source.discover(self.logger, config) 267 268 yield from self._emit_queued_messages(self.source) 269 yield AirbyteMessage(type=Type.CATALOG, catalog=catalog) 270 271 def read( 272 self, source_spec: ConnectorSpecification, config: TConfig, catalog: Any, state: list[Any] 273 ) -> Iterable[AirbyteMessage]: 274 self.set_up_secret_filter(config, source_spec.connectionSpecification) 275 if self.source.check_config_against_spec: 276 self.validate_connection(source_spec, config) 277 278 # The Airbyte protocol dictates that counts be expressed as float/double to better protect against integer overflows 279 stream_message_counter: DefaultDict[HashableStreamDescriptor, float] = defaultdict(float) 280 for message in self.source.read(self.logger, config, catalog, state): 281 yield self.handle_record_counts(message, stream_message_counter) 282 try: 283 self._memory_monitor.check_memory_usage() 284 except AirbyteTracedException: 285 # Flush queued messages (state checkpoints, logs) before propagating 286 # the memory fail-fast exception, so the platform receives the last 287 # committed state for the next sync. 288 for queued_message in self._emit_queued_messages(self.source): 289 yield self.handle_record_counts(queued_message, stream_message_counter) 290 raise 291 292 # Flush queued messages after normal completion of the read loop. 293 for message in self._emit_queued_messages(self.source): 294 yield self.handle_record_counts(message, stream_message_counter) 295 296 @staticmethod 297 def handle_record_counts( 298 message: AirbyteMessage, stream_message_count: DefaultDict[HashableStreamDescriptor, float] 299 ) -> AirbyteMessage: 300 match message.type: 301 case Type.RECORD: 302 if message.record is None: 303 raise ValueError("Record message must have a record attribute") 304 305 stream_message_count[ 306 HashableStreamDescriptor( 307 name=message.record.stream, # type: ignore[union-attr] # record has `stream` 308 namespace=message.record.namespace, # type: ignore[union-attr] # record has `namespace` 309 ) 310 ] += 1.0 311 case Type.STATE: 312 if message.state is None: 313 raise ValueError("State message must have a state attribute") 314 315 stream_descriptor = message_utils.get_stream_descriptor(message) 316 317 # Set record count from the counter onto the state message 318 message.state.sourceStats = message.state.sourceStats or AirbyteStateStats() # type: ignore[union-attr] # state has `sourceStats` 319 message.state.sourceStats.recordCount = stream_message_count.get( # type: ignore[union-attr] # state has `sourceStats` 320 stream_descriptor, 0.0 321 ) 322 323 # Reset the counter 324 stream_message_count[stream_descriptor] = 0.0 325 return message 326 327 @staticmethod 328 def validate_connection(source_spec: ConnectorSpecification, config: TConfig) -> None: 329 # Remove internal flags from config before validating so 330 # jsonschema's additionalProperties flag won't fail the validation 331 connector_config, _ = split_config(config) 332 check_config_against_spec_or_exit(connector_config, source_spec) 333 334 @staticmethod 335 def set_up_secret_filter(config: TConfig, connection_specification: Mapping[str, Any]) -> None: 336 # Now that we have the config, we can use it to get a list of ai airbyte_secrets 337 # that we should filter in logging to avoid leaking secrets 338 config_secrets = get_secrets(connection_specification, config) 339 update_secrets(config_secrets) 340 341 @staticmethod 342 def airbyte_message_to_string(airbyte_message: AirbyteMessage) -> str: 343 global _HAS_LOGGED_FOR_SERIALIZATION_ERROR 344 serialized_message = AirbyteMessageSerializer.dump(airbyte_message) 345 try: 346 return orjson.dumps(serialized_message).decode() 347 except Exception as exception: 348 if not _HAS_LOGGED_FOR_SERIALIZATION_ERROR: 349 logger.warning( 350 f"There was an error during the serialization of an AirbyteMessage: `{exception}`. This might impact the sync performances." 351 ) 352 _HAS_LOGGED_FOR_SERIALIZATION_ERROR = True 353 try: 354 return json.dumps(serialized_message) 355 except Exception as json_exception: 356 raise AirbyteTracedException( 357 internal_message=f"Failed to serialize AirbyteMessage to JSON: `{json_exception}`", 358 failure_type=FailureType.system_error, 359 message="A record returned from the API failed to be serialized to JSON.", 360 ) from json_exception 361 362 @classmethod 363 def extract_state(cls, args: List[str]) -> Optional[Any]: 364 parsed_args = cls.parse_args(args) 365 if hasattr(parsed_args, "state"): 366 return parsed_args.state 367 return None 368 369 @classmethod 370 def extract_catalog(cls, args: List[str]) -> Optional[Any]: 371 parsed_args = cls.parse_args(args) 372 if hasattr(parsed_args, "catalog"): 373 return parsed_args.catalog 374 return None 375 376 @classmethod 377 def extract_config(cls, args: List[str]) -> Optional[Any]: 378 parsed_args = cls.parse_args(args) 379 if hasattr(parsed_args, "config"): 380 return parsed_args.config 381 return None 382 383 def _emit_queued_messages(self, source: Source) -> Iterable[AirbyteMessage]: 384 if hasattr(source, "message_repository") and source.message_repository: 385 yield from source.message_repository.consume_queue() 386 return
56 def __init__(self, source: Source): 57 init_uncaught_exception_handler(logger) 58 59 # Deployment mode is read when instantiating the entrypoint because it is the common path shared by syncs and connector builder test requests 60 if is_cloud_environment(): 61 _init_internal_request_filter() 62 63 self.source = source 64 self.logger = logging.getLogger(f"airbyte.{getattr(source, 'name', '')}") 65 self._memory_monitor = MemoryMonitor()
67 @staticmethod 68 def parse_args(args: List[str]) -> argparse.Namespace: 69 # set up parent parsers 70 parent_parser = argparse.ArgumentParser(add_help=False) 71 parent_parser.add_argument( 72 "--debug", action="store_true", help="enables detailed debug logs related to the sync" 73 ) 74 main_parser = argparse.ArgumentParser() 75 subparsers = main_parser.add_subparsers(title="commands", dest="command") 76 77 # spec 78 subparsers.add_parser( 79 "spec", help="outputs the json configuration specification", parents=[parent_parser] 80 ) 81 82 # check 83 check_parser = subparsers.add_parser( 84 "check", help="checks the config can be used to connect", parents=[parent_parser] 85 ) 86 required_check_parser = check_parser.add_argument_group("required named arguments") 87 required_check_parser.add_argument( 88 "--config", type=str, required=True, help="path to the json configuration file" 89 ) 90 check_parser.add_argument( 91 "--manifest-path", 92 type=str, 93 required=False, 94 help="path to the YAML manifest file to inject into the config", 95 ) 96 check_parser.add_argument( 97 "--components-path", 98 type=str, 99 required=False, 100 help="path to the custom components file, if it exists", 101 ) 102 103 # discover 104 discover_parser = subparsers.add_parser( 105 "discover", 106 help="outputs a catalog describing the source's schema", 107 parents=[parent_parser], 108 ) 109 required_discover_parser = discover_parser.add_argument_group("required named arguments") 110 required_discover_parser.add_argument( 111 "--config", type=str, required=True, help="path to the json configuration file" 112 ) 113 discover_parser.add_argument( 114 "--manifest-path", 115 type=str, 116 required=False, 117 help="path to the YAML manifest file to inject into the config", 118 ) 119 discover_parser.add_argument( 120 "--components-path", 121 type=str, 122 required=False, 123 help="path to the custom components file, if it exists", 124 ) 125 126 # read 127 read_parser = subparsers.add_parser( 128 "read", help="reads the source and outputs messages to STDOUT", parents=[parent_parser] 129 ) 130 131 read_parser.add_argument( 132 "--state", type=str, required=False, help="path to the json-encoded state file" 133 ) 134 required_read_parser = read_parser.add_argument_group("required named arguments") 135 required_read_parser.add_argument( 136 "--config", type=str, required=True, help="path to the json configuration file" 137 ) 138 required_read_parser.add_argument( 139 "--catalog", 140 type=str, 141 required=True, 142 help="path to the catalog used to determine which data to read", 143 ) 144 read_parser.add_argument( 145 "--manifest-path", 146 type=str, 147 required=False, 148 help="path to the YAML manifest file to inject into the config", 149 ) 150 read_parser.add_argument( 151 "--components-path", 152 type=str, 153 required=False, 154 help="path to the custom components file, if it exists", 155 ) 156 157 return main_parser.parse_args(args)
159 def run(self, parsed_args: argparse.Namespace) -> Iterable[str]: 160 cmd = parsed_args.command 161 if not cmd: 162 raise Exception("No command passed") 163 164 if (hasattr(parsed_args, "debug") and parsed_args.debug) or is_platform_debug_log_enabled(): 165 self.logger.setLevel(logging.DEBUG) 166 logger.setLevel(logging.DEBUG) 167 self.logger.debug("Debug logs enabled") 168 else: 169 self.logger.setLevel(logging.INFO) 170 171 source_spec: ConnectorSpecification = self.source.spec(self.logger) 172 try: 173 with tempfile.TemporaryDirectory( 174 # Cleanup can fail on Windows due to file locks. Ignore if so, 175 # rather than failing the whole process. 176 ignore_cleanup_errors=True, 177 ) as temp_dir: 178 os.environ[ENV_REQUEST_CACHE_PATH] = ( 179 temp_dir # set this as default directory for request_cache to store *.sqlite files 180 ) 181 if cmd == "spec": 182 message = AirbyteMessage(type=Type.SPEC, spec=source_spec) 183 yield from [ 184 self.airbyte_message_to_string(queued_message) 185 for queued_message in self._emit_queued_messages(self.source) 186 ] 187 yield self.airbyte_message_to_string(message) 188 else: 189 raw_config = self.source.read_config(parsed_args.config) 190 config = self.source.configure(raw_config, temp_dir) 191 192 yield from [ 193 self.airbyte_message_to_string(queued_message) 194 for queued_message in self._emit_queued_messages(self.source) 195 ] 196 if cmd == "check": 197 yield from map( 198 AirbyteEntrypoint.airbyte_message_to_string, 199 self.check(source_spec, config), 200 ) 201 elif cmd == "discover": 202 yield from map( 203 AirbyteEntrypoint.airbyte_message_to_string, 204 self.discover(source_spec, config), 205 ) 206 elif cmd == "read": 207 config_catalog = self.source.read_catalog(parsed_args.catalog) 208 state = self.source.read_state(parsed_args.state) 209 210 yield from map( 211 AirbyteEntrypoint.airbyte_message_to_string, 212 self.read(source_spec, config, config_catalog, state), 213 ) 214 else: 215 raise Exception("Unexpected command " + cmd) 216 finally: 217 yield from [ 218 self.airbyte_message_to_string(queued_message) 219 for queued_message in self._emit_queued_messages(self.source) 220 ]
222 def check( 223 self, source_spec: ConnectorSpecification, config: TConfig 224 ) -> Iterable[AirbyteMessage]: 225 self.set_up_secret_filter(config, source_spec.connectionSpecification) 226 try: 227 self.validate_connection(source_spec, config) 228 except AirbyteTracedException as traced_exc: 229 connection_status = traced_exc.as_connection_status_message() 230 # The platform uses the exit code to surface unexpected failures so we raise the exception if the failure type not a config error 231 # If the failure is not exceptional, we'll emit a failed connection status message and return 232 if traced_exc.failure_type != FailureType.config_error: 233 raise traced_exc 234 if connection_status: 235 yield from self._emit_queued_messages(self.source) 236 yield connection_status 237 return 238 239 try: 240 check_result = self.source.check(self.logger, config) 241 except AirbyteTracedException as traced_exc: 242 yield traced_exc.as_sanitized_airbyte_message() 243 # The platform uses the exit code to surface unexpected failures so we raise the exception if the failure type not a config error 244 # If the failure is not exceptional, we'll emit a failed connection status message and return 245 if traced_exc.failure_type != FailureType.config_error: 246 raise traced_exc 247 else: 248 connection_status = traced_exc.as_connection_status_message() 249 if connection_status: 250 yield connection_status 251 return 252 if check_result.status == Status.SUCCEEDED: 253 self.logger.info("Check succeeded") 254 else: 255 self.logger.error("Check failed") 256 257 yield from self._emit_queued_messages(self.source) 258 yield AirbyteMessage(type=Type.CONNECTION_STATUS, connectionStatus=check_result)
260 def discover( 261 self, source_spec: ConnectorSpecification, config: TConfig 262 ) -> Iterable[AirbyteMessage]: 263 self.set_up_secret_filter(config, source_spec.connectionSpecification) 264 if self.source.check_config_against_spec: 265 self.validate_connection(source_spec, config) 266 catalog = self.source.discover(self.logger, config) 267 268 yield from self._emit_queued_messages(self.source) 269 yield AirbyteMessage(type=Type.CATALOG, catalog=catalog)
271 def read( 272 self, source_spec: ConnectorSpecification, config: TConfig, catalog: Any, state: list[Any] 273 ) -> Iterable[AirbyteMessage]: 274 self.set_up_secret_filter(config, source_spec.connectionSpecification) 275 if self.source.check_config_against_spec: 276 self.validate_connection(source_spec, config) 277 278 # The Airbyte protocol dictates that counts be expressed as float/double to better protect against integer overflows 279 stream_message_counter: DefaultDict[HashableStreamDescriptor, float] = defaultdict(float) 280 for message in self.source.read(self.logger, config, catalog, state): 281 yield self.handle_record_counts(message, stream_message_counter) 282 try: 283 self._memory_monitor.check_memory_usage() 284 except AirbyteTracedException: 285 # Flush queued messages (state checkpoints, logs) before propagating 286 # the memory fail-fast exception, so the platform receives the last 287 # committed state for the next sync. 288 for queued_message in self._emit_queued_messages(self.source): 289 yield self.handle_record_counts(queued_message, stream_message_counter) 290 raise 291 292 # Flush queued messages after normal completion of the read loop. 293 for message in self._emit_queued_messages(self.source): 294 yield self.handle_record_counts(message, stream_message_counter)
296 @staticmethod 297 def handle_record_counts( 298 message: AirbyteMessage, stream_message_count: DefaultDict[HashableStreamDescriptor, float] 299 ) -> AirbyteMessage: 300 match message.type: 301 case Type.RECORD: 302 if message.record is None: 303 raise ValueError("Record message must have a record attribute") 304 305 stream_message_count[ 306 HashableStreamDescriptor( 307 name=message.record.stream, # type: ignore[union-attr] # record has `stream` 308 namespace=message.record.namespace, # type: ignore[union-attr] # record has `namespace` 309 ) 310 ] += 1.0 311 case Type.STATE: 312 if message.state is None: 313 raise ValueError("State message must have a state attribute") 314 315 stream_descriptor = message_utils.get_stream_descriptor(message) 316 317 # Set record count from the counter onto the state message 318 message.state.sourceStats = message.state.sourceStats or AirbyteStateStats() # type: ignore[union-attr] # state has `sourceStats` 319 message.state.sourceStats.recordCount = stream_message_count.get( # type: ignore[union-attr] # state has `sourceStats` 320 stream_descriptor, 0.0 321 ) 322 323 # Reset the counter 324 stream_message_count[stream_descriptor] = 0.0 325 return message
327 @staticmethod 328 def validate_connection(source_spec: ConnectorSpecification, config: TConfig) -> None: 329 # Remove internal flags from config before validating so 330 # jsonschema's additionalProperties flag won't fail the validation 331 connector_config, _ = split_config(config) 332 check_config_against_spec_or_exit(connector_config, source_spec)
334 @staticmethod 335 def set_up_secret_filter(config: TConfig, connection_specification: Mapping[str, Any]) -> None: 336 # Now that we have the config, we can use it to get a list of ai airbyte_secrets 337 # that we should filter in logging to avoid leaking secrets 338 config_secrets = get_secrets(connection_specification, config) 339 update_secrets(config_secrets)
341 @staticmethod 342 def airbyte_message_to_string(airbyte_message: AirbyteMessage) -> str: 343 global _HAS_LOGGED_FOR_SERIALIZATION_ERROR 344 serialized_message = AirbyteMessageSerializer.dump(airbyte_message) 345 try: 346 return orjson.dumps(serialized_message).decode() 347 except Exception as exception: 348 if not _HAS_LOGGED_FOR_SERIALIZATION_ERROR: 349 logger.warning( 350 f"There was an error during the serialization of an AirbyteMessage: `{exception}`. This might impact the sync performances." 351 ) 352 _HAS_LOGGED_FOR_SERIALIZATION_ERROR = True 353 try: 354 return json.dumps(serialized_message) 355 except Exception as json_exception: 356 raise AirbyteTracedException( 357 internal_message=f"Failed to serialize AirbyteMessage to JSON: `{json_exception}`", 358 failure_type=FailureType.system_error, 359 message="A record returned from the API failed to be serialized to JSON.", 360 ) from json_exception
528class AbstractAPIBudget(abc.ABC): 529 """Interface to some API where a client allowed to have N calls per T interval. 530 531 Important: APIBudget is not doing any API calls, the end user code is responsible to call this interface 532 to respect call rate limitation of the API. 533 534 It supports multiple policies applied to different group of requests. To distinct these groups we use RequestMatchers. 535 Individual policy represented by MovingWindowCallRatePolicy and currently supports only moving window strategy. 536 """ 537 538 @abc.abstractmethod 539 def acquire_call( 540 self, request: Any, block: bool = True, timeout: Optional[float] = None 541 ) -> None: 542 """Try to get a call from budget, will block by default 543 544 :param request: 545 :param block: when true (default) will block the current thread until call credit is available 546 :param timeout: if set will limit maximum time in block, otherwise will wait until credit is available 547 :raises: CallRateLimitHit - when no credits left and if timeout was set the waiting time exceed the timeout 548 """ 549 550 @abc.abstractmethod 551 def get_matching_policy(self, request: Any) -> Optional[AbstractCallRatePolicy]: 552 """Find matching call rate policy for specific request""" 553 554 @abc.abstractmethod 555 def update_from_response(self, request: Any, response: Any) -> None: 556 """Update budget information based on response from API 557 558 :param request: the initial request that triggered this response 559 :param response: response from the API 560 """
Interface to some API where a client allowed to have N calls per T interval.
Important: APIBudget is not doing any API calls, the end user code is responsible to call this interface to respect call rate limitation of the API.
It supports multiple policies applied to different group of requests. To distinct these groups we use RequestMatchers. Individual policy represented by MovingWindowCallRatePolicy and currently supports only moving window strategy.
538 @abc.abstractmethod 539 def acquire_call( 540 self, request: Any, block: bool = True, timeout: Optional[float] = None 541 ) -> None: 542 """Try to get a call from budget, will block by default 543 544 :param request: 545 :param block: when true (default) will block the current thread until call credit is available 546 :param timeout: if set will limit maximum time in block, otherwise will wait until credit is available 547 :raises: CallRateLimitHit - when no credits left and if timeout was set the waiting time exceed the timeout 548 """
Try to get a call from budget, will block by default
Parameters
- request:
- block: when true (default) will block the current thread until call credit is available
- timeout: if set will limit maximum time in block, otherwise will wait until credit is available
Raises
- CallRateLimitHit - when no credits left and if timeout was set the waiting time exceed the timeout
550 @abc.abstractmethod 551 def get_matching_policy(self, request: Any) -> Optional[AbstractCallRatePolicy]: 552 """Find matching call rate policy for specific request"""
Find matching call rate policy for specific request
554 @abc.abstractmethod 555 def update_from_response(self, request: Any, response: Any) -> None: 556 """Update budget information based on response from API 557 558 :param request: the initial request that triggered this response 559 :param response: response from the API 560 """
Update budget information based on response from API
Parameters
- request: the initial request that triggered this response
- response: response from the API
13class AbstractHeaderAuthenticator(AuthBase): 14 """Abstract class for an header-based authenticators that add a header to outgoing HTTP requests.""" 15 16 def __call__(self, request: requests.PreparedRequest) -> Any: 17 """Attach the HTTP headers required to authenticate on the HTTP request""" 18 request.headers.update(self.get_auth_header()) 19 return request 20 21 def get_auth_header(self) -> Mapping[str, Any]: 22 """The header to set on outgoing HTTP requests""" 23 if self.auth_header: 24 return {self.auth_header: self.token} 25 return {} 26 27 @property 28 @abstractmethod 29 def auth_header(self) -> str: 30 """HTTP header to set on the requests""" 31 32 @property 33 @abstractmethod 34 def token(self) -> str: 35 """The header value to set on outgoing HTTP requests"""
Abstract class for an header-based authenticators that add a header to outgoing HTTP requests.
21 def get_auth_header(self) -> Mapping[str, Any]: 22 """The header to set on outgoing HTTP requests""" 23 if self.auth_header: 24 return {self.auth_header: self.token} 25 return {}
The header to set on outgoing HTTP requests
14class BaseBackoffException(requests.exceptions.HTTPError): 15 def __init__( 16 self, 17 request: requests.PreparedRequest, 18 response: Optional[Union[requests.Response, Exception]], 19 error_message: str = "", 20 failure_type: Optional[FailureType] = None, 21 ): 22 self.failure_type = failure_type 23 if isinstance(response, requests.Response): 24 error_message = ( 25 error_message 26 or f"Request URL: {request.url}, Response Code: {response.status_code}, Response Text: {response.text}" 27 ) 28 super().__init__(error_message, request=request, response=response) 29 else: 30 error_message = error_message or f"Request URL: {request.url}, Exception: {response}" 31 super().__init__(error_message, request=request, response=None)
An HTTP error occurred.
15 def __init__( 16 self, 17 request: requests.PreparedRequest, 18 response: Optional[Union[requests.Response, Exception]], 19 error_message: str = "", 20 failure_type: Optional[FailureType] = None, 21 ): 22 self.failure_type = failure_type 23 if isinstance(response, requests.Response): 24 error_message = ( 25 error_message 26 or f"Request URL: {request.url}, Response Code: {response.status_code}, Response Text: {response.text}" 27 ) 28 super().__init__(error_message, request=request, response=response) 29 else: 30 error_message = error_message or f"Request URL: {request.url}, Exception: {response}" 31 super().__init__(error_message, request=request, response=None)
754class CachedLimiterSession(requests_cache.CacheMixin, LimiterMixin, requests.Session): 755 """Session class with caching and rate-limiting behavior."""
Session class with caching and rate-limiting behavior.
An HTTP error occurred.
Inherited Members
34def default_backoff_handler( 35 max_tries: Optional[int], factor: float, max_time: Optional[int] = None, **kwargs: Any 36) -> Callable[[SendRequestCallableType], SendRequestCallableType]: 37 def log_retry_attempt(details: Mapping[str, Any]) -> None: 38 _, exc, _ = sys.exc_info() 39 if isinstance(exc, RequestException) and exc.response: 40 logger.info( 41 f"Status code: {exc.response.status_code!r}, Response Content: {exc.response.content!r}" 42 ) 43 logger.info( 44 f"Caught retryable error '{str(exc)}' after {details['tries']} tries. Waiting {details['wait']} seconds then retrying..." 45 ) 46 47 def should_give_up(exc: Exception) -> bool: 48 # If a non-rate-limiting related 4XX error makes it this far, it means it was unexpected and probably consistent, so we shouldn't back off 49 if isinstance(exc, RequestException): 50 if exc.response is not None: 51 give_up: bool = ( 52 exc.response is not None 53 and exc.response.status_code != codes.too_many_requests 54 and 400 <= exc.response.status_code < 500 55 ) 56 if give_up: 57 logger.info(f"Giving up for returned HTTP status: {exc.response.status_code!r}") 58 return give_up 59 # Only RequestExceptions are retryable, so if we get here, it's not retryable 60 return False 61 62 return backoff.on_exception( # type: ignore # Decorator function returns a function with a different signature than the input function, so mypy can't infer the type of the returned function 63 backoff.expo, 64 TRANSIENT_EXCEPTIONS, 65 jitter=None, 66 on_backoff=log_retry_attempt, 67 giveup=should_give_up, 68 max_tries=max_tries, 69 max_time=max_time, 70 factor=factor, 71 **kwargs, 72 )
681class HttpAPIBudget(APIBudget): 682 """Implementation of AbstractAPIBudget for HTTP""" 683 684 def __init__( 685 self, 686 ratelimit_reset_header: str = "ratelimit-reset", 687 ratelimit_remaining_header: str = "ratelimit-remaining", 688 status_codes_for_ratelimit_hit: list[int] = [429], 689 **kwargs: Any, 690 ): 691 """Constructor 692 693 :param ratelimit_reset_header: name of the header that has a timestamp of the next reset of call budget 694 :param ratelimit_remaining_header: name of the header that has the number of calls left 695 :param status_codes_for_ratelimit_hit: list of HTTP status codes that signal about rate limit being hit 696 """ 697 self._ratelimit_reset_header = ratelimit_reset_header 698 self._ratelimit_remaining_header = ratelimit_remaining_header 699 self._status_codes_for_ratelimit_hit = status_codes_for_ratelimit_hit 700 super().__init__(**kwargs) 701 702 def update_from_response(self, request: Any, response: Any) -> None: 703 policy = self.get_matching_policy(request) 704 if not policy: 705 return 706 707 if isinstance(response, requests.Response): 708 available_calls = self.get_calls_left_from_response(response) 709 reset_ts = self.get_reset_ts_from_response(response) 710 policy.update(available_calls=available_calls, call_reset_ts=reset_ts) 711 712 def get_reset_ts_from_response( 713 self, response: requests.Response 714 ) -> Optional[datetime.datetime]: 715 if response.headers.get(self._ratelimit_reset_header): 716 return datetime.datetime.fromtimestamp( 717 int(response.headers[self._ratelimit_reset_header]) 718 ) 719 return None 720 721 def get_calls_left_from_response(self, response: requests.Response) -> Optional[int]: 722 if response.headers.get(self._ratelimit_remaining_header): 723 return int(response.headers[self._ratelimit_remaining_header]) 724 725 if response.status_code in self._status_codes_for_ratelimit_hit: 726 return 0 727 728 return None
Implementation of AbstractAPIBudget for HTTP
684 def __init__( 685 self, 686 ratelimit_reset_header: str = "ratelimit-reset", 687 ratelimit_remaining_header: str = "ratelimit-remaining", 688 status_codes_for_ratelimit_hit: list[int] = [429], 689 **kwargs: Any, 690 ): 691 """Constructor 692 693 :param ratelimit_reset_header: name of the header that has a timestamp of the next reset of call budget 694 :param ratelimit_remaining_header: name of the header that has the number of calls left 695 :param status_codes_for_ratelimit_hit: list of HTTP status codes that signal about rate limit being hit 696 """ 697 self._ratelimit_reset_header = ratelimit_reset_header 698 self._ratelimit_remaining_header = ratelimit_remaining_header 699 self._status_codes_for_ratelimit_hit = status_codes_for_ratelimit_hit 700 super().__init__(**kwargs)
Constructor
Parameters
- ratelimit_reset_header: name of the header that has a timestamp of the next reset of call budget
- ratelimit_remaining_header: name of the header that has the number of calls left
- status_codes_for_ratelimit_hit: list of HTTP status codes that signal about rate limit being hit
702 def update_from_response(self, request: Any, response: Any) -> None: 703 policy = self.get_matching_policy(request) 704 if not policy: 705 return 706 707 if isinstance(response, requests.Response): 708 available_calls = self.get_calls_left_from_response(response) 709 reset_ts = self.get_reset_ts_from_response(response) 710 policy.update(available_calls=available_calls, call_reset_ts=reset_ts)
Update budget information based on the API response.
Parameters
- request: the initial request that triggered this response
- response: response from the API
721 def get_calls_left_from_response(self, response: requests.Response) -> Optional[int]: 722 if response.headers.get(self._ratelimit_remaining_header): 723 return int(response.headers[self._ratelimit_remaining_header]) 724 725 if response.status_code in self._status_codes_for_ratelimit_hit: 726 return 0 727 728 return None
Inherited Members
103class HttpRequestMatcher(RequestMatcher): 104 """Simple implementation of RequestMatcher for HTTP requests using HttpRequestRegexMatcher under the hood.""" 105 106 def __init__( 107 self, 108 method: Optional[str] = None, 109 url: Optional[str] = None, 110 params: Optional[Mapping[str, Any]] = None, 111 headers: Optional[Mapping[str, Any]] = None, 112 ): 113 """Constructor 114 115 :param method: HTTP method (e.g., "GET", "POST"). 116 :param url: Full URL to match. 117 :param params: Dictionary of query parameters to match. 118 :param headers: Dictionary of headers to match. 119 """ 120 # Parse the URL to extract the base and path 121 if url: 122 parsed_url = parse.urlsplit(url) 123 url_base = f"{parsed_url.scheme}://{parsed_url.netloc}" 124 url_path = parsed_url.path if parsed_url.path != "/" else None 125 else: 126 url_base = None 127 url_path = None 128 129 # Use HttpRequestRegexMatcher under the hood 130 self._regex_matcher = HttpRequestRegexMatcher( 131 method=method, 132 url_base=url_base, 133 url_path_pattern=re.escape(url_path) if url_path else None, 134 params=params, 135 headers=headers, 136 ) 137 138 def __call__(self, request: Any) -> bool: 139 """ 140 :param request: A requests.Request or requests.PreparedRequest instance. 141 :return: True if the request matches all provided criteria; False otherwise. 142 """ 143 return self._regex_matcher(request) 144 145 def __str__(self) -> str: 146 return ( 147 f"HttpRequestMatcher(method={self._regex_matcher._method}, " 148 f"url={self._regex_matcher._url_base}{self._regex_matcher._url_path_pattern.pattern if self._regex_matcher._url_path_pattern else ''}, " 149 f"params={self._regex_matcher._params}, headers={self._regex_matcher._headers})" 150 )
Simple implementation of RequestMatcher for HTTP requests using HttpRequestRegexMatcher under the hood.
106 def __init__( 107 self, 108 method: Optional[str] = None, 109 url: Optional[str] = None, 110 params: Optional[Mapping[str, Any]] = None, 111 headers: Optional[Mapping[str, Any]] = None, 112 ): 113 """Constructor 114 115 :param method: HTTP method (e.g., "GET", "POST"). 116 :param url: Full URL to match. 117 :param params: Dictionary of query parameters to match. 118 :param headers: Dictionary of headers to match. 119 """ 120 # Parse the URL to extract the base and path 121 if url: 122 parsed_url = parse.urlsplit(url) 123 url_base = f"{parsed_url.scheme}://{parsed_url.netloc}" 124 url_path = parsed_url.path if parsed_url.path != "/" else None 125 else: 126 url_base = None 127 url_path = None 128 129 # Use HttpRequestRegexMatcher under the hood 130 self._regex_matcher = HttpRequestRegexMatcher( 131 method=method, 132 url_base=url_base, 133 url_path_pattern=re.escape(url_path) if url_path else None, 134 params=params, 135 headers=headers, 136 )
Constructor
Parameters
- method: HTTP method (e.g., "GET", "POST").
- url: Full URL to match.
- params: Dictionary of query parameters to match.
- headers: Dictionary of headers to match.
45class HttpStream(Stream, CheckpointMixin, ABC): 46 """ 47 Base abstract class for an Airbyte Stream using the HTTP protocol. Basic building block for users building an Airbyte source for a HTTP API. 48 """ 49 50 source_defined_cursor = True # Most HTTP streams use a source defined cursor (i.e: the user can't configure it like on a SQL table) 51 page_size: Optional[int] = ( 52 None # Use this variable to define page size for API http requests with pagination support 53 ) 54 55 def __init__( 56 self, authenticator: Optional[AuthBase] = None, api_budget: Optional[APIBudget] = None 57 ): 58 self._exit_on_rate_limit: bool = False 59 self._http_client = HttpClient( 60 name=self.name, 61 logger=self.logger, 62 error_handler=self.get_error_handler(), 63 api_budget=api_budget or APIBudget(policies=[]), 64 authenticator=authenticator, 65 use_cache=self.use_cache, 66 backoff_strategy=self.get_backoff_strategy(), 67 message_repository=InMemoryMessageRepository(), 68 ) 69 70 # There are three conditions that dictate if RFR should automatically be applied to a stream 71 # 1. Streams that explicitly initialize their own cursor should defer to it and not automatically apply RFR 72 # 2. Streams with at least one cursor_field are incremental and thus a superior sync to RFR. 73 # 3. Streams overriding read_records() do not guarantee that they will call the parent implementation which can perform 74 # per-page checkpointing so RFR is only supported if a stream use the default `HttpStream.read_records()` method 75 if ( 76 not self.cursor 77 and len(self.cursor_field) == 0 78 and type(self).read_records is HttpStream.read_records 79 ): 80 self.cursor = ResumableFullRefreshCursor() 81 82 @property 83 def exit_on_rate_limit(self) -> bool: 84 """ 85 :return: False if the stream will retry endlessly when rate limited 86 """ 87 return self._exit_on_rate_limit 88 89 @exit_on_rate_limit.setter 90 def exit_on_rate_limit(self, value: bool) -> None: 91 self._exit_on_rate_limit = value 92 93 @property 94 def cache_filename(self) -> str: 95 """ 96 Override if needed. Return the name of cache file 97 Note that if the environment variable REQUEST_CACHE_PATH is not set, the cache will be in-memory only. 98 """ 99 return f"{self.name}.sqlite" 100 101 @property 102 def use_cache(self) -> bool: 103 """ 104 Override if needed. If True, all records will be cached. 105 Note that if the environment variable REQUEST_CACHE_PATH is not set, the cache will be in-memory only. 106 """ 107 return False 108 109 @property 110 @abstractmethod 111 def url_base(self) -> str: 112 """ 113 :return: URL base for the API endpoint e.g: if you wanted to hit https://myapi.com/v1/some_entity then this should return "https://myapi.com/v1/" 114 """ 115 116 @property 117 def http_method(self) -> str: 118 """ 119 Override if needed. See get_request_data/get_request_json if using POST/PUT/PATCH. 120 """ 121 return "GET" 122 123 @property 124 @deprecated( 125 "Deprecated as of CDK version 3.0.0. " 126 "You should set error_handler explicitly in HttpStream.get_error_handler() instead." 127 ) 128 def raise_on_http_errors(self) -> bool: 129 """ 130 Override if needed. If set to False, allows opting-out of raising HTTP code exception. 131 """ 132 return True 133 134 @property 135 @deprecated( 136 "Deprecated as of CDK version 3.0.0. " 137 "You should set backoff_strategies explicitly in HttpStream.get_backoff_strategy() instead." 138 ) 139 def max_retries(self) -> Union[int, None]: 140 """ 141 Override if needed. Specifies maximum amount of retries for backoff policy. Return None for no limit. 142 """ 143 return 5 144 145 @property 146 @deprecated( 147 "Deprecated as of CDK version 3.0.0. " 148 "You should set backoff_strategies explicitly in HttpStream.get_backoff_strategy() instead." 149 ) 150 def max_time(self) -> Union[int, None]: 151 """ 152 Override if needed. Specifies maximum total waiting time (in seconds) for backoff policy. Return None for no limit. 153 """ 154 return 60 * 10 155 156 @property 157 @deprecated( 158 "Deprecated as of CDK version 3.0.0. " 159 "You should set backoff_strategies explicitly in HttpStream.get_backoff_strategy() instead." 160 ) 161 def retry_factor(self) -> float: 162 """ 163 Override if needed. Specifies factor for backoff policy. 164 """ 165 return 5 166 167 @abstractmethod 168 def next_page_token(self, response: requests.Response) -> Optional[Mapping[str, Any]]: 169 """ 170 Override this method to define a pagination strategy. 171 172 The value returned from this method is passed to most other methods in this class. Use it to form a request e.g: set headers or query params. 173 174 :return: The token for the next page from the input response object. Returning None means there are no more pages to read in this response. 175 """ 176 177 @abstractmethod 178 def path( 179 self, 180 *, 181 stream_state: Optional[Mapping[str, Any]] = None, 182 stream_slice: Optional[Mapping[str, Any]] = None, 183 next_page_token: Optional[Mapping[str, Any]] = None, 184 ) -> str: 185 """ 186 Returns the URL path for the API endpoint e.g: if you wanted to hit https://myapi.com/v1/some_entity then this should return "some_entity" 187 """ 188 189 def request_params( 190 self, 191 stream_state: Optional[Mapping[str, Any]], 192 stream_slice: Optional[Mapping[str, Any]] = None, 193 next_page_token: Optional[Mapping[str, Any]] = None, 194 ) -> MutableMapping[str, Any]: 195 """ 196 Override this method to define the query parameters that should be set on an outgoing HTTP request given the inputs. 197 198 E.g: you might want to define query parameters for paging if next_page_token is not None. 199 """ 200 return {} 201 202 def request_headers( 203 self, 204 stream_state: Optional[Mapping[str, Any]], 205 stream_slice: Optional[Mapping[str, Any]] = None, 206 next_page_token: Optional[Mapping[str, Any]] = None, 207 ) -> Mapping[str, Any]: 208 """ 209 Override to return any non-auth headers. Authentication headers will overwrite any overlapping headers returned from this method. 210 """ 211 return {} 212 213 def request_body_data( 214 self, 215 stream_state: Optional[Mapping[str, Any]], 216 stream_slice: Optional[Mapping[str, Any]] = None, 217 next_page_token: Optional[Mapping[str, Any]] = None, 218 ) -> Optional[Union[Mapping[str, Any], str]]: 219 """ 220 Override when creating POST/PUT/PATCH requests to populate the body of the request with a non-JSON payload. 221 222 If returns a ready text that it will be sent as is. 223 If returns a dict that it will be converted to a urlencoded form. 224 E.g. {"key1": "value1", "key2": "value2"} => "key1=value1&key2=value2" 225 226 At the same time only one of the 'request_body_data' and 'request_body_json' functions can be overridden. 227 """ 228 return None 229 230 def request_body_json( 231 self, 232 stream_state: Optional[Mapping[str, Any]], 233 stream_slice: Optional[Mapping[str, Any]] = None, 234 next_page_token: Optional[Mapping[str, Any]] = None, 235 ) -> Optional[Mapping[str, Any]]: 236 """ 237 Override when creating POST/PUT/PATCH requests to populate the body of the request with a JSON payload. 238 239 At the same time only one of the 'request_body_data' and 'request_body_json' functions can be overridden. 240 """ 241 return None 242 243 def request_kwargs( 244 self, 245 stream_state: Optional[Mapping[str, Any]], 246 stream_slice: Optional[Mapping[str, Any]] = None, 247 next_page_token: Optional[Mapping[str, Any]] = None, 248 ) -> Mapping[str, Any]: 249 """ 250 Override to return a mapping of keyword arguments to be used when creating the HTTP request. 251 Any option listed in https://docs.python-requests.org/en/latest/api/#requests.adapters.BaseAdapter.send for can be returned from 252 this method. Note that these options do not conflict with request-level options such as headers, request params, etc.. 253 """ 254 return {} 255 256 @abstractmethod 257 def parse_response( 258 self, 259 response: requests.Response, 260 *, 261 stream_state: Mapping[str, Any], 262 stream_slice: Optional[Mapping[str, Any]] = None, 263 next_page_token: Optional[Mapping[str, Any]] = None, 264 ) -> Iterable[Mapping[str, Any]]: 265 """ 266 Parses the raw response object into a list of records. 267 By default, this returns an iterable containing the input. Override to parse differently. 268 :param response: 269 :param stream_state: 270 :param stream_slice: 271 :param next_page_token: 272 :return: An iterable containing the parsed response 273 """ 274 275 def get_backoff_strategy(self) -> Optional[Union[BackoffStrategy, List[BackoffStrategy]]]: 276 """ 277 Used to initialize Adapter to avoid breaking changes. 278 If Stream has a `backoff_time` method implementation, we know this stream uses old (pre-HTTPClient) backoff handlers and thus an adapter is needed. 279 280 Override to provide custom BackoffStrategy 281 :return Optional[BackoffStrategy]: 282 """ 283 if hasattr(self, "backoff_time"): 284 return HttpStreamAdapterBackoffStrategy(self) 285 else: 286 return None 287 288 def get_error_handler(self) -> Optional[ErrorHandler]: 289 """ 290 Used to initialize Adapter to avoid breaking changes. 291 If Stream has a `should_retry` method implementation, we know this stream uses old (pre-HTTPClient) error handlers and thus an adapter is needed. 292 293 Override to provide custom ErrorHandler 294 :return Optional[ErrorHandler]: 295 """ 296 if hasattr(self, "should_retry"): 297 error_handler = HttpStreamAdapterHttpStatusErrorHandler( 298 stream=self, 299 logger=logging.getLogger(), 300 max_retries=self.max_retries, 301 max_time=timedelta(seconds=self.max_time or 0), 302 ) 303 return error_handler 304 else: 305 return None 306 307 @classmethod 308 def _join_url(cls, url_base: str, path: str) -> str: 309 return urljoin(url_base, path) 310 311 @classmethod 312 def parse_response_error_message(cls, response: requests.Response) -> Optional[str]: 313 """ 314 Parses the raw response object from a failed request into a user-friendly error message. 315 By default, this method tries to grab the error message from JSON responses by following common API patterns. Override to parse differently. 316 317 :param response: 318 :return: A user-friendly message that indicates the cause of the error 319 """ 320 321 # default logic to grab error from common fields 322 def _try_get_error(value: Optional[JsonType]) -> Optional[str]: 323 if isinstance(value, str): 324 return value 325 elif isinstance(value, list): 326 errors_in_value = [_try_get_error(v) for v in value] 327 return ", ".join(v for v in errors_in_value if v is not None) 328 elif isinstance(value, dict): 329 new_value = ( 330 value.get("message") 331 or value.get("messages") 332 or value.get("error") 333 or value.get("errors") 334 or value.get("failures") 335 or value.get("failure") 336 or value.get("detail") 337 ) 338 return _try_get_error(new_value) 339 return None 340 341 try: 342 body = response.json() 343 return _try_get_error(body) 344 except requests.exceptions.JSONDecodeError: 345 return None 346 347 def get_error_display_message(self, exception: BaseException) -> Optional[str]: 348 """ 349 Retrieves the user-friendly display message that corresponds to an exception. 350 This will be called when encountering an exception while reading records from the stream, and used to build the AirbyteTraceMessage. 351 352 The default implementation of this method only handles HTTPErrors by passing the response to self.parse_response_error_message(). 353 The method should be overriden as needed to handle any additional exception types. 354 355 :param exception: The exception that was raised 356 :return: A user-friendly message that indicates the cause of the error 357 """ 358 if isinstance(exception, requests.HTTPError) and exception.response is not None: 359 return self.parse_response_error_message(exception.response) 360 return None 361 362 def read_records( 363 self, 364 sync_mode: SyncMode, 365 cursor_field: Optional[List[str]] = None, 366 stream_slice: Optional[Mapping[str, Any]] = None, 367 stream_state: Optional[Mapping[str, Any]] = None, 368 ) -> Iterable[StreamData]: 369 # A cursor_field indicates this is an incremental stream which offers better checkpointing than RFR enabled via the cursor 370 if self.cursor_field or not isinstance(self.get_cursor(), ResumableFullRefreshCursor): 371 yield from self._read_pages( 372 lambda req, res, state, _slice: self.parse_response( 373 res, stream_slice=_slice, stream_state=state 374 ), 375 stream_slice, 376 stream_state, 377 ) 378 else: 379 yield from self._read_single_page( 380 lambda req, res, state, _slice: self.parse_response( 381 res, stream_slice=_slice, stream_state=state 382 ), 383 stream_slice, 384 stream_state, 385 ) 386 387 @property 388 def state(self) -> MutableMapping[str, Any]: 389 cursor = self.get_cursor() 390 if cursor: 391 return cursor.get_stream_state() # type: ignore 392 return self._state 393 394 @state.setter 395 def state(self, value: MutableMapping[str, Any]) -> None: 396 cursor = self.get_cursor() 397 if cursor: 398 cursor.set_initial_state(value) 399 self._state = value 400 401 def get_cursor(self) -> Optional[Cursor]: 402 # I don't love that this is semi-stateful but not sure what else to do. We don't know exactly what type of cursor to 403 # instantiate when creating the class. We can make a few assumptions like if there is a cursor_field which implies 404 # incremental, but we don't know until runtime if this is a substream. Ideally, a stream should explicitly define 405 # its cursor, but because we're trying to automatically apply RFR we're stuck with this logic where we replace the 406 # cursor at runtime once we detect this is a substream based on self.has_multiple_slices being reassigned 407 if self.has_multiple_slices and isinstance(self.cursor, ResumableFullRefreshCursor): 408 self.cursor = SubstreamResumableFullRefreshCursor() 409 return self.cursor 410 else: 411 return self.cursor 412 413 def _read_pages( 414 self, 415 records_generator_fn: Callable[ 416 [ 417 requests.PreparedRequest, 418 requests.Response, 419 Mapping[str, Any], 420 Optional[Mapping[str, Any]], 421 ], 422 Iterable[StreamData], 423 ], 424 stream_slice: Optional[Mapping[str, Any]] = None, 425 stream_state: Optional[Mapping[str, Any]] = None, 426 ) -> Iterable[StreamData]: 427 stream_state = stream_state or {} 428 pagination_complete = False 429 next_page_token = None 430 while not pagination_complete: 431 request, response = self._fetch_next_page(stream_slice, stream_state, next_page_token) 432 yield from records_generator_fn(request, response, stream_state, stream_slice) 433 434 next_page_token = self.next_page_token(response) 435 if not next_page_token: 436 pagination_complete = True 437 438 cursor = self.get_cursor() 439 if cursor and isinstance(cursor, SubstreamResumableFullRefreshCursor): 440 partition, _, _ = self._extract_slice_fields(stream_slice=stream_slice) 441 # Substreams checkpoint state by marking an entire parent partition as completed so that on the subsequent attempt 442 # after a failure, completed parents are skipped and the sync can make progress 443 cursor.close_slice(StreamSlice(cursor_slice={}, partition=partition)) 444 445 # Always return an empty generator just in case no records were ever yielded 446 yield from [] 447 448 def _read_single_page( 449 self, 450 records_generator_fn: Callable[ 451 [ 452 requests.PreparedRequest, 453 requests.Response, 454 Mapping[str, Any], 455 Optional[Mapping[str, Any]], 456 ], 457 Iterable[StreamData], 458 ], 459 stream_slice: Optional[Mapping[str, Any]] = None, 460 stream_state: Optional[Mapping[str, Any]] = None, 461 ) -> Iterable[StreamData]: 462 partition, cursor_slice, remaining_slice = self._extract_slice_fields( 463 stream_slice=stream_slice 464 ) 465 stream_state = stream_state or {} 466 next_page_token = cursor_slice or None 467 468 request, response = self._fetch_next_page(remaining_slice, stream_state, next_page_token) 469 yield from records_generator_fn(request, response, stream_state, remaining_slice) 470 471 next_page_token = self.next_page_token(response) or { 472 "__ab_full_refresh_sync_complete": True 473 } 474 475 cursor = self.get_cursor() 476 if cursor: 477 cursor.close_slice(StreamSlice(cursor_slice=next_page_token, partition=partition)) 478 479 # Always return an empty generator just in case no records were ever yielded 480 yield from [] 481 482 @staticmethod 483 def _extract_slice_fields( 484 stream_slice: Optional[Mapping[str, Any]], 485 ) -> tuple[Mapping[str, Any], Mapping[str, Any], Mapping[str, Any]]: 486 if not stream_slice: 487 return {}, {}, {} 488 489 if isinstance(stream_slice, StreamSlice): 490 partition = stream_slice.partition 491 cursor_slice = stream_slice.cursor_slice 492 remaining = {k: v for k, v in stream_slice.items()} 493 else: 494 # RFR streams that implement stream_slices() to generate stream slices in the legacy mapping format are converted into a 495 # structured stream slice mapping by the LegacyCursorBasedCheckpointReader. The structured mapping object has separate 496 # fields for the partition and cursor_slice value 497 partition = stream_slice.get("partition", {}) 498 cursor_slice = stream_slice.get("cursor_slice", {}) 499 remaining = { 500 key: val 501 for key, val in stream_slice.items() 502 if key != "partition" and key != "cursor_slice" 503 } 504 return partition, cursor_slice, remaining 505 506 def _fetch_next_page( 507 self, 508 stream_slice: Optional[Mapping[str, Any]] = None, 509 stream_state: Optional[Mapping[str, Any]] = None, 510 next_page_token: Optional[Mapping[str, Any]] = None, 511 ) -> Tuple[requests.PreparedRequest, requests.Response]: 512 request, response = self._http_client.send_request( 513 http_method=self.http_method, 514 url=self._join_url( 515 self.url_base, 516 self.path( 517 stream_state=stream_state, 518 stream_slice=stream_slice, 519 next_page_token=next_page_token, 520 ), 521 ), 522 request_kwargs=self.request_kwargs( 523 stream_state=stream_state, 524 stream_slice=stream_slice, 525 next_page_token=next_page_token, 526 ), 527 headers=self.request_headers( 528 stream_state=stream_state, 529 stream_slice=stream_slice, 530 next_page_token=next_page_token, 531 ), 532 params=self.request_params( 533 stream_state=stream_state, 534 stream_slice=stream_slice, 535 next_page_token=next_page_token, 536 ), 537 json=self.request_body_json( 538 stream_state=stream_state, 539 stream_slice=stream_slice, 540 next_page_token=next_page_token, 541 ), 542 data=self.request_body_data( 543 stream_state=stream_state, 544 stream_slice=stream_slice, 545 next_page_token=next_page_token, 546 ), 547 dedupe_query_params=True, 548 log_formatter=self.get_log_formatter(), 549 exit_on_rate_limit=self.exit_on_rate_limit, 550 ) 551 552 return request, response 553 554 def get_log_formatter(self) -> Optional[Callable[[requests.Response], Any]]: 555 """ 556 557 :return Optional[Callable[[requests.Response], Any]]: Function that will be used in logging inside HttpClient 558 """ 559 return None
Base abstract class for an Airbyte Stream using the HTTP protocol. Basic building block for users building an Airbyte source for a HTTP API.
82 @property 83 def exit_on_rate_limit(self) -> bool: 84 """ 85 :return: False if the stream will retry endlessly when rate limited 86 """ 87 return self._exit_on_rate_limit
Returns
False if the stream will retry endlessly when rate limited
93 @property 94 def cache_filename(self) -> str: 95 """ 96 Override if needed. Return the name of cache file 97 Note that if the environment variable REQUEST_CACHE_PATH is not set, the cache will be in-memory only. 98 """ 99 return f"{self.name}.sqlite"
Override if needed. Return the name of cache file Note that if the environment variable REQUEST_CACHE_PATH is not set, the cache will be in-memory only.
101 @property 102 def use_cache(self) -> bool: 103 """ 104 Override if needed. If True, all records will be cached. 105 Note that if the environment variable REQUEST_CACHE_PATH is not set, the cache will be in-memory only. 106 """ 107 return False
Override if needed. If True, all records will be cached. Note that if the environment variable REQUEST_CACHE_PATH is not set, the cache will be in-memory only.
109 @property 110 @abstractmethod 111 def url_base(self) -> str: 112 """ 113 :return: URL base for the API endpoint e.g: if you wanted to hit https://myapi.com/v1/some_entity then this should return "https://myapi.com/v1/" 114 """
Returns
URL base for the API endpoint e.g: if you wanted to hit https://myapi.com/v1/some_entity then this should return "https://myapi.com/v1/"
116 @property 117 def http_method(self) -> str: 118 """ 119 Override if needed. See get_request_data/get_request_json if using POST/PUT/PATCH. 120 """ 121 return "GET"
Override if needed. See get_request_data/get_request_json if using POST/PUT/PATCH.
123 @property 124 @deprecated( 125 "Deprecated as of CDK version 3.0.0. " 126 "You should set error_handler explicitly in HttpStream.get_error_handler() instead." 127 ) 128 def raise_on_http_errors(self) -> bool: 129 """ 130 Override if needed. If set to False, allows opting-out of raising HTTP code exception. 131 """ 132 return True
Override if needed. If set to False, allows opting-out of raising HTTP code exception.
134 @property 135 @deprecated( 136 "Deprecated as of CDK version 3.0.0. " 137 "You should set backoff_strategies explicitly in HttpStream.get_backoff_strategy() instead." 138 ) 139 def max_retries(self) -> Union[int, None]: 140 """ 141 Override if needed. Specifies maximum amount of retries for backoff policy. Return None for no limit. 142 """ 143 return 5
Override if needed. Specifies maximum amount of retries for backoff policy. Return None for no limit.
145 @property 146 @deprecated( 147 "Deprecated as of CDK version 3.0.0. " 148 "You should set backoff_strategies explicitly in HttpStream.get_backoff_strategy() instead." 149 ) 150 def max_time(self) -> Union[int, None]: 151 """ 152 Override if needed. Specifies maximum total waiting time (in seconds) for backoff policy. Return None for no limit. 153 """ 154 return 60 * 10
Override if needed. Specifies maximum total waiting time (in seconds) for backoff policy. Return None for no limit.
156 @property 157 @deprecated( 158 "Deprecated as of CDK version 3.0.0. " 159 "You should set backoff_strategies explicitly in HttpStream.get_backoff_strategy() instead." 160 ) 161 def retry_factor(self) -> float: 162 """ 163 Override if needed. Specifies factor for backoff policy. 164 """ 165 return 5
Override if needed. Specifies factor for backoff policy.
167 @abstractmethod 168 def next_page_token(self, response: requests.Response) -> Optional[Mapping[str, Any]]: 169 """ 170 Override this method to define a pagination strategy. 171 172 The value returned from this method is passed to most other methods in this class. Use it to form a request e.g: set headers or query params. 173 174 :return: The token for the next page from the input response object. Returning None means there are no more pages to read in this response. 175 """
Override this method to define a pagination strategy.
The value returned from this method is passed to most other methods in this class. Use it to form a request e.g: set headers or query params.
Returns
The token for the next page from the input response object. Returning None means there are no more pages to read in this response.
177 @abstractmethod 178 def path( 179 self, 180 *, 181 stream_state: Optional[Mapping[str, Any]] = None, 182 stream_slice: Optional[Mapping[str, Any]] = None, 183 next_page_token: Optional[Mapping[str, Any]] = None, 184 ) -> str: 185 """ 186 Returns the URL path for the API endpoint e.g: if you wanted to hit https://myapi.com/v1/some_entity then this should return "some_entity" 187 """
Returns the URL path for the API endpoint e.g: if you wanted to hit https://myapi.com/v1/some_entity then this should return "some_entity"
189 def request_params( 190 self, 191 stream_state: Optional[Mapping[str, Any]], 192 stream_slice: Optional[Mapping[str, Any]] = None, 193 next_page_token: Optional[Mapping[str, Any]] = None, 194 ) -> MutableMapping[str, Any]: 195 """ 196 Override this method to define the query parameters that should be set on an outgoing HTTP request given the inputs. 197 198 E.g: you might want to define query parameters for paging if next_page_token is not None. 199 """ 200 return {}
Override this method to define the query parameters that should be set on an outgoing HTTP request given the inputs.
E.g: you might want to define query parameters for paging if next_page_token is not None.
202 def request_headers( 203 self, 204 stream_state: Optional[Mapping[str, Any]], 205 stream_slice: Optional[Mapping[str, Any]] = None, 206 next_page_token: Optional[Mapping[str, Any]] = None, 207 ) -> Mapping[str, Any]: 208 """ 209 Override to return any non-auth headers. Authentication headers will overwrite any overlapping headers returned from this method. 210 """ 211 return {}
Override to return any non-auth headers. Authentication headers will overwrite any overlapping headers returned from this method.
213 def request_body_data( 214 self, 215 stream_state: Optional[Mapping[str, Any]], 216 stream_slice: Optional[Mapping[str, Any]] = None, 217 next_page_token: Optional[Mapping[str, Any]] = None, 218 ) -> Optional[Union[Mapping[str, Any], str]]: 219 """ 220 Override when creating POST/PUT/PATCH requests to populate the body of the request with a non-JSON payload. 221 222 If returns a ready text that it will be sent as is. 223 If returns a dict that it will be converted to a urlencoded form. 224 E.g. {"key1": "value1", "key2": "value2"} => "key1=value1&key2=value2" 225 226 At the same time only one of the 'request_body_data' and 'request_body_json' functions can be overridden. 227 """ 228 return None
Override when creating POST/PUT/PATCH requests to populate the body of the request with a non-JSON payload.
If returns a ready text that it will be sent as is. If returns a dict that it will be converted to a urlencoded form. E.g. {"key1": "value1", "key2": "value2"} => "key1=value1&key2=value2"
At the same time only one of the 'request_body_data' and 'request_body_json' functions can be overridden.
230 def request_body_json( 231 self, 232 stream_state: Optional[Mapping[str, Any]], 233 stream_slice: Optional[Mapping[str, Any]] = None, 234 next_page_token: Optional[Mapping[str, Any]] = None, 235 ) -> Optional[Mapping[str, Any]]: 236 """ 237 Override when creating POST/PUT/PATCH requests to populate the body of the request with a JSON payload. 238 239 At the same time only one of the 'request_body_data' and 'request_body_json' functions can be overridden. 240 """ 241 return None
Override when creating POST/PUT/PATCH requests to populate the body of the request with a JSON payload.
At the same time only one of the 'request_body_data' and 'request_body_json' functions can be overridden.
243 def request_kwargs( 244 self, 245 stream_state: Optional[Mapping[str, Any]], 246 stream_slice: Optional[Mapping[str, Any]] = None, 247 next_page_token: Optional[Mapping[str, Any]] = None, 248 ) -> Mapping[str, Any]: 249 """ 250 Override to return a mapping of keyword arguments to be used when creating the HTTP request. 251 Any option listed in https://docs.python-requests.org/en/latest/api/#requests.adapters.BaseAdapter.send for can be returned from 252 this method. Note that these options do not conflict with request-level options such as headers, request params, etc.. 253 """ 254 return {}
Override to return a mapping of keyword arguments to be used when creating the HTTP request. Any option listed in https://docs.python-requests.org/en/latest/api/#requests.adapters.BaseAdapter.send for can be returned from this method. Note that these options do not conflict with request-level options such as headers, request params, etc..
256 @abstractmethod 257 def parse_response( 258 self, 259 response: requests.Response, 260 *, 261 stream_state: Mapping[str, Any], 262 stream_slice: Optional[Mapping[str, Any]] = None, 263 next_page_token: Optional[Mapping[str, Any]] = None, 264 ) -> Iterable[Mapping[str, Any]]: 265 """ 266 Parses the raw response object into a list of records. 267 By default, this returns an iterable containing the input. Override to parse differently. 268 :param response: 269 :param stream_state: 270 :param stream_slice: 271 :param next_page_token: 272 :return: An iterable containing the parsed response 273 """
Parses the raw response object into a list of records. By default, this returns an iterable containing the input. Override to parse differently.
Parameters
- response:
- stream_state:
- stream_slice:
- next_page_token:
Returns
An iterable containing the parsed response
275 def get_backoff_strategy(self) -> Optional[Union[BackoffStrategy, List[BackoffStrategy]]]: 276 """ 277 Used to initialize Adapter to avoid breaking changes. 278 If Stream has a `backoff_time` method implementation, we know this stream uses old (pre-HTTPClient) backoff handlers and thus an adapter is needed. 279 280 Override to provide custom BackoffStrategy 281 :return Optional[BackoffStrategy]: 282 """ 283 if hasattr(self, "backoff_time"): 284 return HttpStreamAdapterBackoffStrategy(self) 285 else: 286 return None
Used to initialize Adapter to avoid breaking changes.
If Stream has a backoff_time method implementation, we know this stream uses old (pre-HTTPClient) backoff handlers and thus an adapter is needed.
Override to provide custom BackoffStrategy
Returns
288 def get_error_handler(self) -> Optional[ErrorHandler]: 289 """ 290 Used to initialize Adapter to avoid breaking changes. 291 If Stream has a `should_retry` method implementation, we know this stream uses old (pre-HTTPClient) error handlers and thus an adapter is needed. 292 293 Override to provide custom ErrorHandler 294 :return Optional[ErrorHandler]: 295 """ 296 if hasattr(self, "should_retry"): 297 error_handler = HttpStreamAdapterHttpStatusErrorHandler( 298 stream=self, 299 logger=logging.getLogger(), 300 max_retries=self.max_retries, 301 max_time=timedelta(seconds=self.max_time or 0), 302 ) 303 return error_handler 304 else: 305 return None
Used to initialize Adapter to avoid breaking changes.
If Stream has a should_retry method implementation, we know this stream uses old (pre-HTTPClient) error handlers and thus an adapter is needed.
Override to provide custom ErrorHandler
Returns
311 @classmethod 312 def parse_response_error_message(cls, response: requests.Response) -> Optional[str]: 313 """ 314 Parses the raw response object from a failed request into a user-friendly error message. 315 By default, this method tries to grab the error message from JSON responses by following common API patterns. Override to parse differently. 316 317 :param response: 318 :return: A user-friendly message that indicates the cause of the error 319 """ 320 321 # default logic to grab error from common fields 322 def _try_get_error(value: Optional[JsonType]) -> Optional[str]: 323 if isinstance(value, str): 324 return value 325 elif isinstance(value, list): 326 errors_in_value = [_try_get_error(v) for v in value] 327 return ", ".join(v for v in errors_in_value if v is not None) 328 elif isinstance(value, dict): 329 new_value = ( 330 value.get("message") 331 or value.get("messages") 332 or value.get("error") 333 or value.get("errors") 334 or value.get("failures") 335 or value.get("failure") 336 or value.get("detail") 337 ) 338 return _try_get_error(new_value) 339 return None 340 341 try: 342 body = response.json() 343 return _try_get_error(body) 344 except requests.exceptions.JSONDecodeError: 345 return None
Parses the raw response object from a failed request into a user-friendly error message. By default, this method tries to grab the error message from JSON responses by following common API patterns. Override to parse differently.
Parameters
- response:
Returns
A user-friendly message that indicates the cause of the error
347 def get_error_display_message(self, exception: BaseException) -> Optional[str]: 348 """ 349 Retrieves the user-friendly display message that corresponds to an exception. 350 This will be called when encountering an exception while reading records from the stream, and used to build the AirbyteTraceMessage. 351 352 The default implementation of this method only handles HTTPErrors by passing the response to self.parse_response_error_message(). 353 The method should be overriden as needed to handle any additional exception types. 354 355 :param exception: The exception that was raised 356 :return: A user-friendly message that indicates the cause of the error 357 """ 358 if isinstance(exception, requests.HTTPError) and exception.response is not None: 359 return self.parse_response_error_message(exception.response) 360 return None
Retrieves the user-friendly display message that corresponds to an exception. This will be called when encountering an exception while reading records from the stream, and used to build the AirbyteTraceMessage.
The default implementation of this method only handles HTTPErrors by passing the response to self.parse_response_error_message(). The method should be overriden as needed to handle any additional exception types.
Parameters
- exception: The exception that was raised
Returns
A user-friendly message that indicates the cause of the error
362 def read_records( 363 self, 364 sync_mode: SyncMode, 365 cursor_field: Optional[List[str]] = None, 366 stream_slice: Optional[Mapping[str, Any]] = None, 367 stream_state: Optional[Mapping[str, Any]] = None, 368 ) -> Iterable[StreamData]: 369 # A cursor_field indicates this is an incremental stream which offers better checkpointing than RFR enabled via the cursor 370 if self.cursor_field or not isinstance(self.get_cursor(), ResumableFullRefreshCursor): 371 yield from self._read_pages( 372 lambda req, res, state, _slice: self.parse_response( 373 res, stream_slice=_slice, stream_state=state 374 ), 375 stream_slice, 376 stream_state, 377 ) 378 else: 379 yield from self._read_single_page( 380 lambda req, res, state, _slice: self.parse_response( 381 res, stream_slice=_slice, stream_state=state 382 ), 383 stream_slice, 384 stream_state, 385 )
This method should be overridden by subclasses to read records based on the inputs
387 @property 388 def state(self) -> MutableMapping[str, Any]: 389 cursor = self.get_cursor() 390 if cursor: 391 return cursor.get_stream_state() # type: ignore 392 return self._state
State getter, should return state in form that can serialized to a string and send to the output as a STATE AirbyteMessage.
A good example of a state is a cursor_value: { self.cursor_field: "cursor_value" }
State should try to be as small as possible but at the same time descriptive enough to restore syncing process from the point where it stopped.
401 def get_cursor(self) -> Optional[Cursor]: 402 # I don't love that this is semi-stateful but not sure what else to do. We don't know exactly what type of cursor to 403 # instantiate when creating the class. We can make a few assumptions like if there is a cursor_field which implies 404 # incremental, but we don't know until runtime if this is a substream. Ideally, a stream should explicitly define 405 # its cursor, but because we're trying to automatically apply RFR we're stuck with this logic where we replace the 406 # cursor at runtime once we detect this is a substream based on self.has_multiple_slices being reassigned 407 if self.has_multiple_slices and isinstance(self.cursor, ResumableFullRefreshCursor): 408 self.cursor = SubstreamResumableFullRefreshCursor() 409 return self.cursor 410 else: 411 return self.cursor
A Cursor is an interface that a stream can implement to manage how its internal state is read and updated while reading records. Historically, Python connectors had no concept of a cursor to manage state. Python streams need to define a cursor implementation and override this method to manage state through a Cursor.
554 def get_log_formatter(self) -> Optional[Callable[[requests.Response], Any]]: 555 """ 556 557 :return Optional[Callable[[requests.Response], Any]]: Function that will be used in logging inside HttpClient 558 """ 559 return None
Returns
Function that will be used in logging inside HttpClient
Inherited Members
562class HttpSubStream(HttpStream, ABC): 563 def __init__(self, parent: HttpStream, **kwargs: Any): 564 """ 565 :param parent: should be the instance of HttpStream class 566 """ 567 super().__init__(**kwargs) 568 self.parent = parent 569 self.has_multiple_slices = ( 570 True # Substreams are based on parent records which implies there are multiple slices 571 ) 572 573 # There are three conditions that dictate if RFR should automatically be applied to a stream 574 # 1. Streams that explicitly initialize their own cursor should defer to it and not automatically apply RFR 575 # 2. Streams with at least one cursor_field are incremental and thus a superior sync to RFR. 576 # 3. Streams overriding read_records() do not guarantee that they will call the parent implementation which can perform 577 # per-page checkpointing so RFR is only supported if a stream use the default `HttpStream.read_records()` method 578 if ( 579 not self.cursor 580 and len(self.cursor_field) == 0 581 and type(self).read_records is HttpStream.read_records 582 ): 583 self.cursor = SubstreamResumableFullRefreshCursor() 584 585 def stream_slices( 586 self, 587 sync_mode: SyncMode, 588 cursor_field: Optional[List[str]] = None, 589 stream_state: Optional[Mapping[str, Any]] = None, 590 ) -> Iterable[Optional[Mapping[str, Any]]]: 591 # read_stateless() assumes the parent is not concurrent. This is currently okay since the concurrent CDK does 592 # not support either substreams or RFR, but something that needs to be considered once we do 593 for parent_record in self.parent.read_only_records(stream_state): 594 # Skip non-records (eg AirbyteLogMessage) 595 if isinstance(parent_record, AirbyteMessage): 596 if parent_record.type == MessageType.RECORD: 597 parent_record = parent_record.record.data # type: ignore [assignment, union-attr] # Incorrect type for assignment 598 else: 599 continue 600 elif isinstance(parent_record, Record): 601 parent_record = parent_record.data 602 yield {"parent": parent_record}
Base abstract class for an Airbyte Stream using the HTTP protocol. Basic building block for users building an Airbyte source for a HTTP API.
563 def __init__(self, parent: HttpStream, **kwargs: Any): 564 """ 565 :param parent: should be the instance of HttpStream class 566 """ 567 super().__init__(**kwargs) 568 self.parent = parent 569 self.has_multiple_slices = ( 570 True # Substreams are based on parent records which implies there are multiple slices 571 ) 572 573 # There are three conditions that dictate if RFR should automatically be applied to a stream 574 # 1. Streams that explicitly initialize their own cursor should defer to it and not automatically apply RFR 575 # 2. Streams with at least one cursor_field are incremental and thus a superior sync to RFR. 576 # 3. Streams overriding read_records() do not guarantee that they will call the parent implementation which can perform 577 # per-page checkpointing so RFR is only supported if a stream use the default `HttpStream.read_records()` method 578 if ( 579 not self.cursor 580 and len(self.cursor_field) == 0 581 and type(self).read_records is HttpStream.read_records 582 ): 583 self.cursor = SubstreamResumableFullRefreshCursor()
Parameters
- parent: should be the instance of HttpStream class
585 def stream_slices( 586 self, 587 sync_mode: SyncMode, 588 cursor_field: Optional[List[str]] = None, 589 stream_state: Optional[Mapping[str, Any]] = None, 590 ) -> Iterable[Optional[Mapping[str, Any]]]: 591 # read_stateless() assumes the parent is not concurrent. This is currently okay since the concurrent CDK does 592 # not support either substreams or RFR, but something that needs to be considered once we do 593 for parent_record in self.parent.read_only_records(stream_state): 594 # Skip non-records (eg AirbyteLogMessage) 595 if isinstance(parent_record, AirbyteMessage): 596 if parent_record.type == MessageType.RECORD: 597 parent_record = parent_record.record.data # type: ignore [assignment, union-attr] # Incorrect type for assignment 598 else: 599 continue 600 elif isinstance(parent_record, Record): 601 parent_record = parent_record.data 602 yield {"parent": parent_record}
Override to define the slices for this stream. See the stream slicing section of the docs for more information.
Parameters
- sync_mode:
- cursor_field:
- stream_state:
Returns
Inherited Members
- HttpStream
- source_defined_cursor
- page_size
- exit_on_rate_limit
- cache_filename
- use_cache
- url_base
- http_method
- raise_on_http_errors
- max_retries
- max_time
- retry_factor
- next_page_token
- path
- request_params
- request_headers
- request_body_data
- request_body_json
- request_kwargs
- parse_response
- get_backoff_strategy
- get_error_handler
- parse_response_error_message
- get_error_display_message
- read_records
- state
- get_cursor
- get_log_formatter
750class LimiterSession(LimiterMixin, requests.Session): 751 """Session that adds rate-limiting behavior to requests."""
Session that adds rate-limiting behavior to requests.
Inherited Members
425class MovingWindowCallRatePolicy(BaseCallRatePolicy): 426 """ 427 Policy to control requests rate implemented on top of PyRateLimiter lib. 428 The main difference between this policy and FixedWindowCallRatePolicy is that the rate-limiting window 429 is moving along requests that we made, and there is no moment when we reset an available number of calls. 430 This strategy requires saving of timestamps of all requests within a window. 431 """ 432 433 def __init__(self, rates: list[Rate], matchers: list[RequestMatcher]): 434 """Constructor 435 436 :param rates: list of rates, the order is important and must be ascending 437 :param matchers: 438 """ 439 if not rates: 440 raise ValueError("The list of rates can not be empty") 441 pyrate_rates = [ 442 PyRateRate(limit=rate.limit, interval=int(rate.interval.total_seconds() * 1000)) 443 for rate in rates 444 ] 445 self._bucket = InMemoryBucket(pyrate_rates) 446 # Limiter will create the background task that clears old requests in the bucket 447 self._limiter = Limiter(self._bucket) 448 super().__init__(matchers=matchers) 449 450 def try_acquire(self, request: Any, weight: int) -> None: 451 if not self.matches(request): 452 raise ValueError("Request does not match the policy") 453 lowest_limit = min(rate.limit for rate in self._bucket.rates) 454 if weight > lowest_limit: 455 raise ValueError( 456 f"Weight can not exceed the lowest configured rate limit ({lowest_limit})" 457 ) 458 459 try: 460 self._limiter.try_acquire(request, weight=weight) 461 except BucketFullException as exc: 462 item = self._limiter.bucket_factory.wrap_item(request, weight) 463 assert isinstance(item, RateItem) 464 465 with self._limiter.lock: 466 time_to_wait = self._bucket.waiting(item) 467 assert isinstance(time_to_wait, int) 468 469 raise CallRateLimitHit( 470 error=str(exc.meta_info["error"]), 471 item=request, 472 weight=int(exc.meta_info["weight"]), 473 rate=str(exc.meta_info["rate"]), 474 time_to_wait=timedelta(milliseconds=time_to_wait), 475 ) 476 477 def update( 478 self, available_calls: Optional[int], call_reset_ts: Optional[datetime.datetime] 479 ) -> None: 480 """Adjust call bucket to reflect the state of the API server 481 482 :param available_calls: 483 :param call_reset_ts: 484 :return: 485 """ 486 if ( 487 available_calls is not None and call_reset_ts is None 488 ): # we do our best to sync buckets with API 489 if available_calls == 0: 490 with self._limiter.lock: 491 now: int = TimeClock().now() # type: ignore[no-untyped-call] 492 items_to_add = self._get_available_calls(now) 493 if items_to_add > 0: 494 added = self._bucket.put( 495 RateItem(name="dummy", timestamp=now, weight=items_to_add) 496 ) 497 if not added: 498 logger.warning( 499 "Failed to sync rate limit bucket with API: could not add %s items", 500 items_to_add, 501 ) 502 # TODO: add support if needed, it might be that it is not possible to make a good solution for this case 503 # if available_calls is not None and call_reset_ts is not None: 504 # ts = call_reset_ts.timestamp() 505 506 def _get_available_calls(self, now: int) -> int: 507 """Return the number of calls still allowed by the strictest configured rate at the given timestamp (ms).""" 508 return min( 509 rate.limit 510 - sum(1 for item in self._bucket.items if item.timestamp >= now - rate.interval) 511 for rate in self._bucket.rates 512 ) 513 514 def __str__(self) -> str: 515 """Return a human-friendly description of the moving window rate policy for logging purposes.""" 516 rates_info = ", ".join( 517 f"{rate.limit} per {timedelta(milliseconds=rate.interval)}" 518 for rate in self._bucket.rates 519 ) 520 current_bucket_count = self._bucket.count() 521 matcher_str = ", ".join(f"{matcher}" for matcher in self._matchers) 522 return ( 523 f"MovingWindowCallRatePolicy(rates=[{rates_info}], current_bucket_count={current_bucket_count}, " 524 f"matchers=[{matcher_str}])" 525 )
Policy to control requests rate implemented on top of PyRateLimiter lib. The main difference between this policy and FixedWindowCallRatePolicy is that the rate-limiting window is moving along requests that we made, and there is no moment when we reset an available number of calls. This strategy requires saving of timestamps of all requests within a window.
433 def __init__(self, rates: list[Rate], matchers: list[RequestMatcher]): 434 """Constructor 435 436 :param rates: list of rates, the order is important and must be ascending 437 :param matchers: 438 """ 439 if not rates: 440 raise ValueError("The list of rates can not be empty") 441 pyrate_rates = [ 442 PyRateRate(limit=rate.limit, interval=int(rate.interval.total_seconds() * 1000)) 443 for rate in rates 444 ] 445 self._bucket = InMemoryBucket(pyrate_rates) 446 # Limiter will create the background task that clears old requests in the bucket 447 self._limiter = Limiter(self._bucket) 448 super().__init__(matchers=matchers)
Constructor
Parameters
- rates: list of rates, the order is important and must be ascending
- matchers:
450 def try_acquire(self, request: Any, weight: int) -> None: 451 if not self.matches(request): 452 raise ValueError("Request does not match the policy") 453 lowest_limit = min(rate.limit for rate in self._bucket.rates) 454 if weight > lowest_limit: 455 raise ValueError( 456 f"Weight can not exceed the lowest configured rate limit ({lowest_limit})" 457 ) 458 459 try: 460 self._limiter.try_acquire(request, weight=weight) 461 except BucketFullException as exc: 462 item = self._limiter.bucket_factory.wrap_item(request, weight) 463 assert isinstance(item, RateItem) 464 465 with self._limiter.lock: 466 time_to_wait = self._bucket.waiting(item) 467 assert isinstance(time_to_wait, int) 468 469 raise CallRateLimitHit( 470 error=str(exc.meta_info["error"]), 471 item=request, 472 weight=int(exc.meta_info["weight"]), 473 rate=str(exc.meta_info["rate"]), 474 time_to_wait=timedelta(milliseconds=time_to_wait), 475 )
Try to acquire request
Parameters
- request: a request object representing a single call to API
- weight: number of requests to deduct from credit
Returns
477 def update( 478 self, available_calls: Optional[int], call_reset_ts: Optional[datetime.datetime] 479 ) -> None: 480 """Adjust call bucket to reflect the state of the API server 481 482 :param available_calls: 483 :param call_reset_ts: 484 :return: 485 """ 486 if ( 487 available_calls is not None and call_reset_ts is None 488 ): # we do our best to sync buckets with API 489 if available_calls == 0: 490 with self._limiter.lock: 491 now: int = TimeClock().now() # type: ignore[no-untyped-call] 492 items_to_add = self._get_available_calls(now) 493 if items_to_add > 0: 494 added = self._bucket.put( 495 RateItem(name="dummy", timestamp=now, weight=items_to_add) 496 ) 497 if not added: 498 logger.warning( 499 "Failed to sync rate limit bucket with API: could not add %s items", 500 items_to_add, 501 ) 502 # TODO: add support if needed, it might be that it is not possible to make a good solution for this case 503 # if available_calls is not None and call_reset_ts is not None: 504 # ts = call_reset_ts.timestamp()
Adjust call bucket to reflect the state of the API server
Parameters
- available_calls:
- call_reset_ts:
Returns
Inherited Members
27class Oauth2Authenticator(AbstractOauth2Authenticator): 28 """ 29 Generates OAuth2.0 access tokens from an OAuth2.0 refresh token and client credentials. 30 The generated access token is attached to each request via the Authorization header. 31 If a connector_config is provided any mutation of it's value in the scope of this class will emit AirbyteControlConnectorConfigMessage. 32 """ 33 34 def __init__( 35 self, 36 token_refresh_endpoint: str, 37 client_id: str, 38 client_secret: str, 39 refresh_token: str, 40 client_id_name: str = "client_id", 41 client_secret_name: str = "client_secret", 42 refresh_token_name: str = "refresh_token", 43 scopes: List[str] | None = None, 44 token_expiry_date: AirbyteDateTime | None = None, 45 token_expiry_date_format: str | None = None, 46 access_token_name: str = "access_token", 47 expires_in_name: str = "expires_in", 48 refresh_request_body: Mapping[str, Any] | None = None, 49 refresh_request_headers: Mapping[str, Any] | None = None, 50 send_refresh_request_as_query_params: bool = False, 51 grant_type_name: str = "grant_type", 52 grant_type: str = "refresh_token", 53 token_expiry_is_time_of_expiration: bool = False, 54 refresh_token_error_status_codes: Tuple[int, ...] = (), 55 refresh_token_error_key: str = "", 56 refresh_token_error_values: Tuple[str, ...] = (), 57 ) -> None: 58 self._token_refresh_endpoint = token_refresh_endpoint 59 self._client_secret_name = client_secret_name 60 self._client_secret = client_secret 61 self._client_id_name = client_id_name 62 self._client_id = client_id 63 self._refresh_token_name = refresh_token_name 64 self._refresh_token = refresh_token 65 self._scopes = scopes 66 self._access_token_name = access_token_name 67 self._expires_in_name = expires_in_name 68 self._refresh_request_body = refresh_request_body 69 self._refresh_request_headers = refresh_request_headers 70 self._send_refresh_request_as_query_params = send_refresh_request_as_query_params 71 self._grant_type_name = grant_type_name 72 self._grant_type = grant_type 73 74 self._token_expiry_date = token_expiry_date or (ab_datetime_now() - timedelta(days=1)) 75 self._token_expiry_date_format = token_expiry_date_format 76 self._token_expiry_is_time_of_expiration = token_expiry_is_time_of_expiration 77 self._access_token = None 78 super().__init__( 79 refresh_token_error_status_codes, refresh_token_error_key, refresh_token_error_values 80 ) 81 82 def get_token_refresh_endpoint(self) -> str: 83 return self._token_refresh_endpoint 84 85 def get_client_id_name(self) -> str: 86 return self._client_id_name 87 88 def get_client_id(self) -> str: 89 return self._client_id 90 91 def get_client_secret_name(self) -> str: 92 return self._client_secret_name 93 94 def get_client_secret(self) -> str: 95 return self._client_secret 96 97 def get_refresh_token_name(self) -> str: 98 return self._refresh_token_name 99 100 def get_refresh_token(self) -> str: 101 return self._refresh_token 102 103 def get_access_token_name(self) -> str: 104 return self._access_token_name 105 106 def get_scopes(self) -> list[str]: 107 return self._scopes # type: ignore[return-value] 108 109 def get_expires_in_name(self) -> str: 110 return self._expires_in_name 111 112 def get_refresh_request_body(self) -> Mapping[str, Any]: 113 return self._refresh_request_body # type: ignore[return-value] 114 115 def get_refresh_request_headers(self) -> Mapping[str, Any]: 116 return self._refresh_request_headers # type: ignore[return-value] 117 118 def should_send_refresh_request_as_query_params(self) -> bool: 119 return self._send_refresh_request_as_query_params 120 121 def get_grant_type_name(self) -> str: 122 return self._grant_type_name 123 124 def get_grant_type(self) -> str: 125 return self._grant_type 126 127 def get_token_expiry_date(self) -> AirbyteDateTime: 128 return self._token_expiry_date 129 130 def set_token_expiry_date(self, value: AirbyteDateTime) -> None: 131 # Overrides of `refresh_access_token` written before 6.45.5 return the raw `expires_in`. 132 if not isinstance(value, datetime): 133 with suppress(ValueError, OverflowError): 134 value = self._parse_token_expiration_date(value) 135 self._token_expiry_date = value 136 137 @property 138 def token_expiry_is_time_of_expiration(self) -> bool: 139 return self._token_expiry_is_time_of_expiration 140 141 @property 142 def token_expiry_date_format(self) -> Optional[str]: 143 return self._token_expiry_date_format 144 145 @property 146 def access_token(self) -> str: 147 return self._access_token # type: ignore[return-value] 148 149 @access_token.setter 150 def access_token(self, value: str) -> None: 151 self._access_token = value # type: ignore[assignment] # Incorrect type for assignment
Generates OAuth2.0 access tokens from an OAuth2.0 refresh token and client credentials. The generated access token is attached to each request via the Authorization header. If a connector_config is provided any mutation of it's value in the scope of this class will emit AirbyteControlConnectorConfigMessage.
34 def __init__( 35 self, 36 token_refresh_endpoint: str, 37 client_id: str, 38 client_secret: str, 39 refresh_token: str, 40 client_id_name: str = "client_id", 41 client_secret_name: str = "client_secret", 42 refresh_token_name: str = "refresh_token", 43 scopes: List[str] | None = None, 44 token_expiry_date: AirbyteDateTime | None = None, 45 token_expiry_date_format: str | None = None, 46 access_token_name: str = "access_token", 47 expires_in_name: str = "expires_in", 48 refresh_request_body: Mapping[str, Any] | None = None, 49 refresh_request_headers: Mapping[str, Any] | None = None, 50 send_refresh_request_as_query_params: bool = False, 51 grant_type_name: str = "grant_type", 52 grant_type: str = "refresh_token", 53 token_expiry_is_time_of_expiration: bool = False, 54 refresh_token_error_status_codes: Tuple[int, ...] = (), 55 refresh_token_error_key: str = "", 56 refresh_token_error_values: Tuple[str, ...] = (), 57 ) -> None: 58 self._token_refresh_endpoint = token_refresh_endpoint 59 self._client_secret_name = client_secret_name 60 self._client_secret = client_secret 61 self._client_id_name = client_id_name 62 self._client_id = client_id 63 self._refresh_token_name = refresh_token_name 64 self._refresh_token = refresh_token 65 self._scopes = scopes 66 self._access_token_name = access_token_name 67 self._expires_in_name = expires_in_name 68 self._refresh_request_body = refresh_request_body 69 self._refresh_request_headers = refresh_request_headers 70 self._send_refresh_request_as_query_params = send_refresh_request_as_query_params 71 self._grant_type_name = grant_type_name 72 self._grant_type = grant_type 73 74 self._token_expiry_date = token_expiry_date or (ab_datetime_now() - timedelta(days=1)) 75 self._token_expiry_date_format = token_expiry_date_format 76 self._token_expiry_is_time_of_expiration = token_expiry_is_time_of_expiration 77 self._access_token = None 78 super().__init__( 79 refresh_token_error_status_codes, refresh_token_error_key, refresh_token_error_values 80 )
If all of refresh_token_error_status_codes, refresh_token_error_key, and refresh_token_error_values are set, then http errors with such params will be wrapped in AirbyteTracedException.
112 def get_refresh_request_body(self) -> Mapping[str, Any]: 113 return self._refresh_request_body # type: ignore[return-value]
Returns the request body to set on the refresh request
115 def get_refresh_request_headers(self) -> Mapping[str, Any]: 116 return self._refresh_request_headers # type: ignore[return-value]
Returns the request headers to set on the refresh request
118 def should_send_refresh_request_as_query_params(self) -> bool: 119 return self._send_refresh_request_as_query_params
Returns True if the standard refresh args should be sent on the URL
query string instead of in the request body.
Defaults to False so existing authenticators retain their previous
behavior (params in body, no query params on the refresh URL). Subclasses
can override this to opt into the URL-query-string shape required by OAuth
providers like Gong.
130 def set_token_expiry_date(self, value: AirbyteDateTime) -> None: 131 # Overrides of `refresh_access_token` written before 6.45.5 return the raw `expires_in`. 132 if not isinstance(value, datetime): 133 with suppress(ValueError, OverflowError): 134 value = self._parse_token_expiration_date(value) 135 self._token_expiry_date = value
Setter for access token expiration date
137 @property 138 def token_expiry_is_time_of_expiration(self) -> bool: 139 return self._token_expiry_is_time_of_expiration
Indicates that the Token Expiry returns the date until which the token will be valid, not the amount of time it will be valid.
141 @property 142 def token_expiry_date_format(self) -> Optional[str]: 143 return self._token_expiry_date_format
Format of the datetime; exists it if expires_in is returned as the expiration datetime instead of seconds until it expires
145 @property 146 def access_token(self) -> str: 147 return self._access_token # type: ignore[return-value]
Returns the access token
Inherited Members
33@dataclasses.dataclass 34class Rate: 35 """Call rate limit""" 36 37 limit: int 38 interval: timedelta
Call rate limit
154class SingleUseRefreshTokenOauth2Authenticator(Oauth2Authenticator): 155 """ 156 Authenticator that should be used for API implementing single use refresh tokens: 157 when refreshing access token some API returns a new refresh token that needs to used in the next refresh flow. 158 This authenticator updates the configuration with new refresh token by emitting Airbyte control message from an observed mutation. 159 By default, this authenticator expects a connector config with a "credentials" field with the following nested fields: client_id, 160 client_secret, refresh_token. This behavior can be changed by defining custom config path (using dpath paths) in client_id_config_path, 161 client_secret_config_path, refresh_token_config_path constructor arguments. 162 """ 163 164 def __init__( 165 self, 166 connector_config: Mapping[str, Any], 167 token_refresh_endpoint: str, 168 scopes: List[str] | None = None, 169 access_token_name: str = "access_token", 170 expires_in_name: str = "expires_in", 171 refresh_token_name: str = "refresh_token", 172 refresh_request_body: Mapping[str, Any] | None = None, 173 refresh_request_headers: Mapping[str, Any] | None = None, 174 send_refresh_request_as_query_params: bool = False, 175 grant_type_name: str = "grant_type", 176 grant_type: str = "refresh_token", 177 client_id_name: str = "client_id", 178 client_id: Optional[str] = None, 179 client_secret_name: str = "client_secret", 180 client_secret: Optional[str] = None, 181 access_token_config_path: Sequence[str] = ("credentials", "access_token"), 182 refresh_token_config_path: Sequence[str] = ("credentials", "refresh_token"), 183 token_expiry_date_config_path: Sequence[str] = ("credentials", "token_expiry_date"), 184 token_expiry_date_format: Optional[str] = None, 185 message_repository: MessageRepository = NoopMessageRepository(), 186 token_expiry_is_time_of_expiration: bool = False, 187 refresh_token_error_status_codes: Tuple[int, ...] = (), 188 refresh_token_error_key: str = "", 189 refresh_token_error_values: Tuple[str, ...] = (), 190 emit_control_message_to_message_repository: bool = False, 191 ) -> None: 192 """ 193 Args: 194 connector_config (Mapping[str, Any]): The full connector configuration 195 token_refresh_endpoint (str): Full URL to the token refresh endpoint 196 scopes (List[str], optional): List of OAuth scopes to pass in the refresh token request body. Defaults to None. 197 access_token_name (str, optional): Name of the access token field, used to parse the refresh token response. Defaults to "access_token". 198 expires_in_name (str, optional): Name of the name of the field that characterizes when the current access token will expire, used to parse the refresh token response. Defaults to "expires_in". 199 refresh_token_name (str, optional): Name of the name of the refresh token field, used to parse the refresh token response. Defaults to "refresh_token". 200 refresh_request_body (Mapping[str, Any], optional): Custom key value pair that will be added to the refresh token request body. Defaults to None. 201 refresh_request_headers (Mapping[str, Any], optional): Custom key value pair that will be added to the refresh token request headers. Defaults to None. 202 send_refresh_request_as_query_params (bool, optional): When True, the standard refresh args (`grant_type`, `refresh_token`, client credentials when not in an `Authorization` header, scopes, plus any `refresh_request_body` extras) are sent on the URL query string and the request body is emitted empty. Use this for OAuth providers like Gong that document their refresh endpoint with refresh args on the URL query string. Defaults to False. 203 grant_type (str, optional): OAuth grant type. Defaults to "refresh_token". 204 client_id (Optional[str]): The client id to authenticate. If not specified, defaults to credentials.client_id in the config object. 205 client_secret (Optional[str]): The client secret to authenticate. If not specified, defaults to credentials.client_secret in the config object. 206 access_token_config_path (Sequence[str]): Dpath to the access_token field in the connector configuration. Defaults to ("credentials", "access_token"). 207 refresh_token_config_path (Sequence[str]): Dpath to the refresh_token field in the connector configuration. Defaults to ("credentials", "refresh_token"). 208 token_expiry_date_config_path (Sequence[str]): Dpath to the token_expiry_date field in the connector configuration. Defaults to ("credentials", "token_expiry_date"). 209 token_expiry_date_format (Optional[str]): Date format of the token expiry date field (set by expires_in_name). If specified the token expiry date is interpreted as time of expiration, as with token_expiry_is_time_of_expiration, otherwise as number of seconds until expiration. 210 token_expiry_is_time_of_expiration bool: set True it if expires_in is returned as time of expiration instead of the number seconds until expiration 211 message_repository (MessageRepository): The message repository used to emit logs on HTTP requests and control message on config update. 212 emit_control_message_to_message_repository (bool): Whether the message repository additionally receives the CONNECTOR_CONFIG message. This is only for in-process consumers such as the Connector Builder, because messages in the repository are otherwise printed to stdout by the entrypoint and would duplicate the direct stdout emission. Defaults to False. 213 """ 214 self._connector_config = connector_config 215 self._client_id: str = self._get_config_value_by_path( 216 ("credentials", "client_id"), client_id 217 ) 218 self._client_secret: str = self._get_config_value_by_path( 219 ("credentials", "client_secret"), client_secret 220 ) 221 self._client_id_name = client_id_name 222 self._client_secret_name = client_secret_name 223 self._access_token_config_path = access_token_config_path 224 self._refresh_token_config_path = refresh_token_config_path 225 self._token_expiry_date_config_path = token_expiry_date_config_path 226 self._token_expiry_date_format = token_expiry_date_format 227 self._refresh_token_name = refresh_token_name 228 self._grant_type_name = grant_type_name 229 self._connector_config = connector_config 230 self.__message_repository = message_repository 231 self._emit_control_message_to_message_repository = ( 232 emit_control_message_to_message_repository 233 ) 234 super().__init__( 235 token_refresh_endpoint=token_refresh_endpoint, 236 client_id_name=self._client_id_name, 237 client_id=self._client_id, 238 client_secret_name=self._client_secret_name, 239 client_secret=self._client_secret, 240 refresh_token=self.get_refresh_token(), 241 refresh_token_name=self._refresh_token_name, 242 scopes=scopes, 243 token_expiry_date=self.get_token_expiry_date(), 244 access_token_name=access_token_name, 245 expires_in_name=expires_in_name, 246 refresh_request_body=refresh_request_body, 247 refresh_request_headers=refresh_request_headers, 248 send_refresh_request_as_query_params=send_refresh_request_as_query_params, 249 grant_type_name=self._grant_type_name, 250 grant_type=grant_type, 251 token_expiry_date_format=token_expiry_date_format, 252 token_expiry_is_time_of_expiration=token_expiry_is_time_of_expiration 253 or bool(token_expiry_date_format), 254 refresh_token_error_status_codes=refresh_token_error_status_codes, 255 refresh_token_error_key=refresh_token_error_key, 256 refresh_token_error_values=refresh_token_error_values, 257 ) 258 259 @property 260 def access_token(self) -> str: 261 """ 262 Retrieve the access token from the configuration. 263 264 Returns: 265 str: The access token. 266 """ 267 return self._get_config_value_by_path(self._access_token_config_path) # type: ignore[return-value] 268 269 @access_token.setter 270 def access_token(self, new_access_token: str) -> None: 271 """ 272 Sets a new access token. 273 274 Args: 275 new_access_token (str): The new access token to be set. 276 """ 277 self._set_config_value_by_path(self._access_token_config_path, new_access_token) 278 279 def get_refresh_token(self) -> str: 280 """ 281 Retrieve the refresh token from the configuration. 282 283 This method fetches the refresh token using the configuration path specified 284 by `_refresh_token_config_path`. 285 286 Returns: 287 str: The refresh token as a string. 288 """ 289 return self._get_config_value_by_path(self._refresh_token_config_path) # type: ignore[return-value] 290 291 def set_refresh_token(self, new_refresh_token: str) -> None: 292 """ 293 Updates the refresh token in the configuration. 294 295 Args: 296 new_refresh_token (str): The new refresh token to be set. 297 """ 298 self._set_config_value_by_path(self._refresh_token_config_path, new_refresh_token) 299 300 def get_token_expiry_date(self) -> AirbyteDateTime: 301 """ 302 Retrieves the token expiry date from the configuration. 303 304 This method fetches the token expiry date from the configuration using the specified path. 305 If the expiry date is an empty string, it returns the current date and time minus one day. 306 Otherwise, it parses the expiry date string into an AirbyteDateTime object. 307 308 Returns: 309 AirbyteDateTime: The parsed or calculated token expiry date. 310 311 Raises: 312 TypeError: If the result is not an instance of AirbyteDateTime. 313 """ 314 expiry_date = self._get_config_value_by_path(self._token_expiry_date_config_path) 315 result = ( 316 ab_datetime_now() - timedelta(days=1) 317 if expiry_date == "" 318 else ab_datetime_parse(str(expiry_date)) 319 ) 320 if isinstance(result, AirbyteDateTime): 321 return result 322 raise TypeError("Invalid datetime conversion") 323 324 def set_token_expiry_date(self, new_token_expiry_date: AirbyteDateTime) -> None: # type: ignore[override] 325 """ 326 Sets the token expiry date in the configuration. 327 328 Args: 329 new_token_expiry_date (AirbyteDateTime): The new expiry date for the token. 330 """ 331 if not ( 332 isinstance(new_token_expiry_date, datetime) or self.token_expiry_is_time_of_expiration 333 ): 334 # Overrides of `refresh_access_token` written before 6.45.5 return the raw `expires_in` 335 # in seconds. Other values are stored as-is and parsed on read. 336 with suppress(ValueError, OverflowError): 337 new_token_expiry_date = self._parse_token_expiration_date(new_token_expiry_date) 338 self._set_config_value_by_path( 339 self._token_expiry_date_config_path, str(new_token_expiry_date) 340 ) 341 342 def token_has_expired(self) -> bool: 343 """Returns True if the token is expired""" 344 return ab_datetime_now() > self.get_token_expiry_date() 345 346 def get_access_token(self) -> str: 347 """Retrieve new access and refresh token if the access token has expired. 348 349 This method uses double-checked locking to ensure thread-safe token refresh. 350 This is especially critical for single-use refresh tokens where concurrent 351 refresh attempts would cause failures as the refresh token is invalidated 352 after first use. 353 354 The new refresh token is persisted with the set_refresh_token function. 355 356 Returns: 357 str: The current access_token, updated if it was previously expired. 358 """ 359 if self.token_has_expired(): 360 with self._token_refresh_lock: 361 # Double-check after acquiring lock - another thread may have already refreshed 362 if self.token_has_expired(): 363 self.refresh_and_set_access_token() 364 return self.access_token 365 366 def refresh_and_set_access_token(self) -> None: 367 """Force refresh the access token and update internal state. 368 369 For single-use refresh tokens, this also persists the new refresh token 370 and emits a control message to update the connector config. If the 371 response omits a refresh token, the existing one is preserved. 372 """ 373 token_before_waiting = self._current_access_token_or_none() 374 with self._token_refresh_lock: 375 if self._current_access_token_or_none() != token_before_waiting: 376 return 377 new_access_token, access_token_expires_in, new_refresh_token = ( 378 self.refresh_access_token() 379 ) 380 self.access_token = new_access_token 381 if new_refresh_token is not None: 382 self.set_refresh_token(new_refresh_token) 383 self.set_token_expiry_date(access_token_expires_in) 384 self._emit_control_message() 385 386 def refresh_access_token(self) -> Tuple[str, AirbyteDateTime, Optional[str]]: # type: ignore[override] 387 """ 388 Refreshes the access token by making a handled request and extracting the necessary token information. 389 390 Returns: 391 A tuple of (access_token, token_expiry_date, refresh_token). The refresh token 392 is `None` when the OAuth provider omits it from the response. 393 """ 394 response_json = self._make_handled_request() 395 return ( 396 self._extract_access_token(response_json), 397 self._extract_token_expiry_date(response_json), 398 self._extract_refresh_token(response_json), 399 ) 400 401 def _set_config_value_by_path(self, config_path: Union[str, Sequence[str]], value: Any) -> None: 402 """ 403 Set a value in the connector configuration at the specified path. 404 405 Args: 406 config_path (Union[str, Sequence[str]]): The path within the configuration where the value should be set. 407 This can be a string representing a single key or a sequence of strings representing a nested path. 408 value (Any): The value to set at the specified path in the configuration. 409 410 Returns: 411 None 412 """ 413 dpath.new(self._connector_config, config_path, value) # type: ignore[arg-type] 414 415 def _get_config_value_by_path( 416 self, config_path: Union[str, Sequence[str]], default: Optional[str] = None 417 ) -> str | Any: 418 """ 419 Retrieve a value from the connector configuration using a specified path. 420 421 Args: 422 config_path (Union[str, Sequence[str]]): The path to the desired configuration value. This can be a string or a sequence of strings. 423 default (Optional[str], optional): The default value to return if the specified path does not exist in the configuration. Defaults to None. 424 425 Returns: 426 Any: The value from the configuration at the specified path, or the default value if the path does not exist. 427 """ 428 return dpath.get( 429 self._connector_config, # type: ignore[arg-type] 430 config_path, 431 default=default if default is not None else "", 432 ) 433 434 def _emit_control_message(self) -> None: 435 """Emit the updated connector config as a CONNECTOR_CONFIG control message. 436 437 The message is always printed to stdout, which is the delivery the platform relies on: it 438 is immediate and independent of whether the source exposes a message repository and of 439 when that repository is drained (for `ConcurrentDeclarativeSource` the internal repository 440 is only drained during `read`, after partitions are processed). The message repository is 441 only used in addition when explicitly requested, for in-process consumers such as the 442 Connector Builder; repository messages are otherwise printed by the entrypoint too and 443 would duplicate the stdout emission. 444 """ 445 emit_configuration_as_airbyte_control_message(self._connector_config) # type: ignore[arg-type] 446 if self._emit_control_message_to_message_repository and not isinstance( 447 self._message_repository, NoopMessageRepository 448 ): 449 self._message_repository.emit_message( 450 create_connector_config_control_message(self._connector_config) # type: ignore[arg-type] 451 ) 452 453 @property 454 def _message_repository(self) -> MessageRepository: 455 """ 456 Overriding AbstractOauth2Authenticator._message_repository to allow for HTTP request logs 457 """ 458 return self.__message_repository
Authenticator that should be used for API implementing single use refresh tokens: when refreshing access token some API returns a new refresh token that needs to used in the next refresh flow. This authenticator updates the configuration with new refresh token by emitting Airbyte control message from an observed mutation. By default, this authenticator expects a connector config with a "credentials" field with the following nested fields: client_id, client_secret, refresh_token. This behavior can be changed by defining custom config path (using dpath paths) in client_id_config_path, client_secret_config_path, refresh_token_config_path constructor arguments.
164 def __init__( 165 self, 166 connector_config: Mapping[str, Any], 167 token_refresh_endpoint: str, 168 scopes: List[str] | None = None, 169 access_token_name: str = "access_token", 170 expires_in_name: str = "expires_in", 171 refresh_token_name: str = "refresh_token", 172 refresh_request_body: Mapping[str, Any] | None = None, 173 refresh_request_headers: Mapping[str, Any] | None = None, 174 send_refresh_request_as_query_params: bool = False, 175 grant_type_name: str = "grant_type", 176 grant_type: str = "refresh_token", 177 client_id_name: str = "client_id", 178 client_id: Optional[str] = None, 179 client_secret_name: str = "client_secret", 180 client_secret: Optional[str] = None, 181 access_token_config_path: Sequence[str] = ("credentials", "access_token"), 182 refresh_token_config_path: Sequence[str] = ("credentials", "refresh_token"), 183 token_expiry_date_config_path: Sequence[str] = ("credentials", "token_expiry_date"), 184 token_expiry_date_format: Optional[str] = None, 185 message_repository: MessageRepository = NoopMessageRepository(), 186 token_expiry_is_time_of_expiration: bool = False, 187 refresh_token_error_status_codes: Tuple[int, ...] = (), 188 refresh_token_error_key: str = "", 189 refresh_token_error_values: Tuple[str, ...] = (), 190 emit_control_message_to_message_repository: bool = False, 191 ) -> None: 192 """ 193 Args: 194 connector_config (Mapping[str, Any]): The full connector configuration 195 token_refresh_endpoint (str): Full URL to the token refresh endpoint 196 scopes (List[str], optional): List of OAuth scopes to pass in the refresh token request body. Defaults to None. 197 access_token_name (str, optional): Name of the access token field, used to parse the refresh token response. Defaults to "access_token". 198 expires_in_name (str, optional): Name of the name of the field that characterizes when the current access token will expire, used to parse the refresh token response. Defaults to "expires_in". 199 refresh_token_name (str, optional): Name of the name of the refresh token field, used to parse the refresh token response. Defaults to "refresh_token". 200 refresh_request_body (Mapping[str, Any], optional): Custom key value pair that will be added to the refresh token request body. Defaults to None. 201 refresh_request_headers (Mapping[str, Any], optional): Custom key value pair that will be added to the refresh token request headers. Defaults to None. 202 send_refresh_request_as_query_params (bool, optional): When True, the standard refresh args (`grant_type`, `refresh_token`, client credentials when not in an `Authorization` header, scopes, plus any `refresh_request_body` extras) are sent on the URL query string and the request body is emitted empty. Use this for OAuth providers like Gong that document their refresh endpoint with refresh args on the URL query string. Defaults to False. 203 grant_type (str, optional): OAuth grant type. Defaults to "refresh_token". 204 client_id (Optional[str]): The client id to authenticate. If not specified, defaults to credentials.client_id in the config object. 205 client_secret (Optional[str]): The client secret to authenticate. If not specified, defaults to credentials.client_secret in the config object. 206 access_token_config_path (Sequence[str]): Dpath to the access_token field in the connector configuration. Defaults to ("credentials", "access_token"). 207 refresh_token_config_path (Sequence[str]): Dpath to the refresh_token field in the connector configuration. Defaults to ("credentials", "refresh_token"). 208 token_expiry_date_config_path (Sequence[str]): Dpath to the token_expiry_date field in the connector configuration. Defaults to ("credentials", "token_expiry_date"). 209 token_expiry_date_format (Optional[str]): Date format of the token expiry date field (set by expires_in_name). If specified the token expiry date is interpreted as time of expiration, as with token_expiry_is_time_of_expiration, otherwise as number of seconds until expiration. 210 token_expiry_is_time_of_expiration bool: set True it if expires_in is returned as time of expiration instead of the number seconds until expiration 211 message_repository (MessageRepository): The message repository used to emit logs on HTTP requests and control message on config update. 212 emit_control_message_to_message_repository (bool): Whether the message repository additionally receives the CONNECTOR_CONFIG message. This is only for in-process consumers such as the Connector Builder, because messages in the repository are otherwise printed to stdout by the entrypoint and would duplicate the direct stdout emission. Defaults to False. 213 """ 214 self._connector_config = connector_config 215 self._client_id: str = self._get_config_value_by_path( 216 ("credentials", "client_id"), client_id 217 ) 218 self._client_secret: str = self._get_config_value_by_path( 219 ("credentials", "client_secret"), client_secret 220 ) 221 self._client_id_name = client_id_name 222 self._client_secret_name = client_secret_name 223 self._access_token_config_path = access_token_config_path 224 self._refresh_token_config_path = refresh_token_config_path 225 self._token_expiry_date_config_path = token_expiry_date_config_path 226 self._token_expiry_date_format = token_expiry_date_format 227 self._refresh_token_name = refresh_token_name 228 self._grant_type_name = grant_type_name 229 self._connector_config = connector_config 230 self.__message_repository = message_repository 231 self._emit_control_message_to_message_repository = ( 232 emit_control_message_to_message_repository 233 ) 234 super().__init__( 235 token_refresh_endpoint=token_refresh_endpoint, 236 client_id_name=self._client_id_name, 237 client_id=self._client_id, 238 client_secret_name=self._client_secret_name, 239 client_secret=self._client_secret, 240 refresh_token=self.get_refresh_token(), 241 refresh_token_name=self._refresh_token_name, 242 scopes=scopes, 243 token_expiry_date=self.get_token_expiry_date(), 244 access_token_name=access_token_name, 245 expires_in_name=expires_in_name, 246 refresh_request_body=refresh_request_body, 247 refresh_request_headers=refresh_request_headers, 248 send_refresh_request_as_query_params=send_refresh_request_as_query_params, 249 grant_type_name=self._grant_type_name, 250 grant_type=grant_type, 251 token_expiry_date_format=token_expiry_date_format, 252 token_expiry_is_time_of_expiration=token_expiry_is_time_of_expiration 253 or bool(token_expiry_date_format), 254 refresh_token_error_status_codes=refresh_token_error_status_codes, 255 refresh_token_error_key=refresh_token_error_key, 256 refresh_token_error_values=refresh_token_error_values, 257 )
Arguments:
- connector_config (Mapping[str, Any]): The full connector configuration
- token_refresh_endpoint (str): Full URL to the token refresh endpoint
- scopes (List[str], optional): List of OAuth scopes to pass in the refresh token request body. Defaults to None.
- access_token_name (str, optional): Name of the access token field, used to parse the refresh token response. Defaults to "access_token".
- expires_in_name (str, optional): Name of the name of the field that characterizes when the current access token will expire, used to parse the refresh token response. Defaults to "expires_in".
- refresh_token_name (str, optional): Name of the name of the refresh token field, used to parse the refresh token response. Defaults to "refresh_token".
- refresh_request_body (Mapping[str, Any], optional): Custom key value pair that will be added to the refresh token request body. Defaults to None.
- refresh_request_headers (Mapping[str, Any], optional): Custom key value pair that will be added to the refresh token request headers. Defaults to None.
- send_refresh_request_as_query_params (bool, optional): When True, the standard refresh args (
grant_type,refresh_token, client credentials when not in anAuthorizationheader, scopes, plus anyrefresh_request_bodyextras) are sent on the URL query string and the request body is emitted empty. Use this for OAuth providers like Gong that document their refresh endpoint with refresh args on the URL query string. Defaults to False. - grant_type (str, optional): OAuth grant type. Defaults to "refresh_token".
- client_id (Optional[str]): The client id to authenticate. If not specified, defaults to credentials.client_id in the config object.
- client_secret (Optional[str]): The client secret to authenticate. If not specified, defaults to credentials.client_secret in the config object.
- access_token_config_path (Sequence[str]): Dpath to the access_token field in the connector configuration. Defaults to ("credentials", "access_token").
- refresh_token_config_path (Sequence[str]): Dpath to the refresh_token field in the connector configuration. Defaults to ("credentials", "refresh_token").
- token_expiry_date_config_path (Sequence[str]): Dpath to the token_expiry_date field in the connector configuration. Defaults to ("credentials", "token_expiry_date").
- token_expiry_date_format (Optional[str]): Date format of the token expiry date field (set by expires_in_name). If specified the token expiry date is interpreted as time of expiration, as with token_expiry_is_time_of_expiration, otherwise as number of seconds until expiration.
- token_expiry_is_time_of_expiration bool: set True it if expires_in is returned as time of expiration instead of the number seconds until expiration
- message_repository (MessageRepository): The message repository used to emit logs on HTTP requests and control message on config update.
- emit_control_message_to_message_repository (bool): Whether the message repository additionally receives the CONNECTOR_CONFIG message. This is only for in-process consumers such as the Connector Builder, because messages in the repository are otherwise printed to stdout by the entrypoint and would duplicate the direct stdout emission. Defaults to False.
259 @property 260 def access_token(self) -> str: 261 """ 262 Retrieve the access token from the configuration. 263 264 Returns: 265 str: The access token. 266 """ 267 return self._get_config_value_by_path(self._access_token_config_path) # type: ignore[return-value]
Retrieve the access token from the configuration.
Returns:
str: The access token.
279 def get_refresh_token(self) -> str: 280 """ 281 Retrieve the refresh token from the configuration. 282 283 This method fetches the refresh token using the configuration path specified 284 by `_refresh_token_config_path`. 285 286 Returns: 287 str: The refresh token as a string. 288 """ 289 return self._get_config_value_by_path(self._refresh_token_config_path) # type: ignore[return-value]
Retrieve the refresh token from the configuration.
This method fetches the refresh token using the configuration path specified
by _refresh_token_config_path.
Returns:
str: The refresh token as a string.
291 def set_refresh_token(self, new_refresh_token: str) -> None: 292 """ 293 Updates the refresh token in the configuration. 294 295 Args: 296 new_refresh_token (str): The new refresh token to be set. 297 """ 298 self._set_config_value_by_path(self._refresh_token_config_path, new_refresh_token)
Updates the refresh token in the configuration.
Arguments:
- new_refresh_token (str): The new refresh token to be set.
300 def get_token_expiry_date(self) -> AirbyteDateTime: 301 """ 302 Retrieves the token expiry date from the configuration. 303 304 This method fetches the token expiry date from the configuration using the specified path. 305 If the expiry date is an empty string, it returns the current date and time minus one day. 306 Otherwise, it parses the expiry date string into an AirbyteDateTime object. 307 308 Returns: 309 AirbyteDateTime: The parsed or calculated token expiry date. 310 311 Raises: 312 TypeError: If the result is not an instance of AirbyteDateTime. 313 """ 314 expiry_date = self._get_config_value_by_path(self._token_expiry_date_config_path) 315 result = ( 316 ab_datetime_now() - timedelta(days=1) 317 if expiry_date == "" 318 else ab_datetime_parse(str(expiry_date)) 319 ) 320 if isinstance(result, AirbyteDateTime): 321 return result 322 raise TypeError("Invalid datetime conversion")
Retrieves the token expiry date from the configuration.
This method fetches the token expiry date from the configuration using the specified path. If the expiry date is an empty string, it returns the current date and time minus one day. Otherwise, it parses the expiry date string into an AirbyteDateTime object.
Returns:
AirbyteDateTime: The parsed or calculated token expiry date.
Raises:
- TypeError: If the result is not an instance of AirbyteDateTime.
324 def set_token_expiry_date(self, new_token_expiry_date: AirbyteDateTime) -> None: # type: ignore[override] 325 """ 326 Sets the token expiry date in the configuration. 327 328 Args: 329 new_token_expiry_date (AirbyteDateTime): The new expiry date for the token. 330 """ 331 if not ( 332 isinstance(new_token_expiry_date, datetime) or self.token_expiry_is_time_of_expiration 333 ): 334 # Overrides of `refresh_access_token` written before 6.45.5 return the raw `expires_in` 335 # in seconds. Other values are stored as-is and parsed on read. 336 with suppress(ValueError, OverflowError): 337 new_token_expiry_date = self._parse_token_expiration_date(new_token_expiry_date) 338 self._set_config_value_by_path( 339 self._token_expiry_date_config_path, str(new_token_expiry_date) 340 )
Sets the token expiry date in the configuration.
Arguments:
- new_token_expiry_date (AirbyteDateTime): The new expiry date for the token.
342 def token_has_expired(self) -> bool: 343 """Returns True if the token is expired""" 344 return ab_datetime_now() > self.get_token_expiry_date()
Returns True if the token is expired
346 def get_access_token(self) -> str: 347 """Retrieve new access and refresh token if the access token has expired. 348 349 This method uses double-checked locking to ensure thread-safe token refresh. 350 This is especially critical for single-use refresh tokens where concurrent 351 refresh attempts would cause failures as the refresh token is invalidated 352 after first use. 353 354 The new refresh token is persisted with the set_refresh_token function. 355 356 Returns: 357 str: The current access_token, updated if it was previously expired. 358 """ 359 if self.token_has_expired(): 360 with self._token_refresh_lock: 361 # Double-check after acquiring lock - another thread may have already refreshed 362 if self.token_has_expired(): 363 self.refresh_and_set_access_token() 364 return self.access_token
Retrieve new access and refresh token if the access token has expired.
This method uses double-checked locking to ensure thread-safe token refresh. This is especially critical for single-use refresh tokens where concurrent refresh attempts would cause failures as the refresh token is invalidated after first use.
The new refresh token is persisted with the set_refresh_token function.
Returns:
str: The current access_token, updated if it was previously expired.
366 def refresh_and_set_access_token(self) -> None: 367 """Force refresh the access token and update internal state. 368 369 For single-use refresh tokens, this also persists the new refresh token 370 and emits a control message to update the connector config. If the 371 response omits a refresh token, the existing one is preserved. 372 """ 373 token_before_waiting = self._current_access_token_or_none() 374 with self._token_refresh_lock: 375 if self._current_access_token_or_none() != token_before_waiting: 376 return 377 new_access_token, access_token_expires_in, new_refresh_token = ( 378 self.refresh_access_token() 379 ) 380 self.access_token = new_access_token 381 if new_refresh_token is not None: 382 self.set_refresh_token(new_refresh_token) 383 self.set_token_expiry_date(access_token_expires_in) 384 self._emit_control_message()
Force refresh the access token and update internal state.
For single-use refresh tokens, this also persists the new refresh token and emits a control message to update the connector config. If the response omits a refresh token, the existing one is preserved.
386 def refresh_access_token(self) -> Tuple[str, AirbyteDateTime, Optional[str]]: # type: ignore[override] 387 """ 388 Refreshes the access token by making a handled request and extracting the necessary token information. 389 390 Returns: 391 A tuple of (access_token, token_expiry_date, refresh_token). The refresh token 392 is `None` when the OAuth provider omits it from the response. 393 """ 394 response_json = self._make_handled_request() 395 return ( 396 self._extract_access_token(response_json), 397 self._extract_token_expiry_date(response_json), 398 self._extract_refresh_token(response_json), 399 )
Refreshes the access token by making a handled request and extracting the necessary token information.
Returns:
A tuple of (access_token, token_expiry_date, refresh_token). The refresh token is
Nonewhen the OAuth provider omits it from the response.
Inherited Members
- Oauth2Authenticator
- get_token_refresh_endpoint
- get_client_id_name
- get_client_id
- get_client_secret_name
- get_client_secret
- get_refresh_token_name
- get_access_token_name
- get_scopes
- get_expires_in_name
- get_refresh_request_body
- get_refresh_request_headers
- should_send_refresh_request_as_query_params
- get_grant_type_name
- get_grant_type
- token_expiry_is_time_of_expiration
- token_expiry_date_format
39class TokenAuthenticator(AbstractHeaderAuthenticator): 40 """ 41 Builds auth header, based on the token provided. 42 The token is attached to each request via the `auth_header` header. 43 """ 44 45 @property 46 def auth_header(self) -> str: 47 return self._auth_header 48 49 @property 50 def token(self) -> str: 51 return f"{self._auth_method} {self._token}" 52 53 def __init__(self, token: str, auth_method: str = "Bearer", auth_header: str = "Authorization"): 54 self._auth_header = auth_header 55 self._auth_method = auth_method 56 self._token = token
Builds auth header, based on the token provided.
The token is attached to each request via the auth_header header.
Inherited Members
40class UserDefinedBackoffException(BaseBackoffException): 41 """ 42 An exception that exposes how long it attempted to backoff 43 """ 44 45 def __init__( 46 self, 47 backoff: Union[int, float], 48 request: requests.PreparedRequest, 49 response: Optional[Union[requests.Response, Exception]], 50 error_message: str = "", 51 failure_type: Optional[FailureType] = None, 52 ): 53 """ 54 :param backoff: how long to backoff in seconds 55 :param request: the request that triggered this backoff exception 56 :param response: the response that triggered the backoff exception 57 """ 58 self.backoff = backoff 59 super().__init__( 60 request=request, 61 response=response, 62 error_message=error_message, 63 failure_type=failure_type, 64 )
An exception that exposes how long it attempted to backoff
45 def __init__( 46 self, 47 backoff: Union[int, float], 48 request: requests.PreparedRequest, 49 response: Optional[Union[requests.Response, Exception]], 50 error_message: str = "", 51 failure_type: Optional[FailureType] = None, 52 ): 53 """ 54 :param backoff: how long to backoff in seconds 55 :param request: the request that triggered this backoff exception 56 :param response: the response that triggered the backoff exception 57 """ 58 self.backoff = backoff 59 super().__init__( 60 request=request, 61 response=response, 62 error_message=error_message, 63 failure_type=failure_type, 64 )
Parameters
- backoff: how long to backoff in seconds
- request: the request that triggered this backoff exception
- response: the response that triggered the backoff exception
Inherited Members
64class AirbyteLogFormatter(logging.Formatter): 65 """Output log records using AirbyteMessage""" 66 67 # Transforming Python log levels to Airbyte protocol log levels 68 level_mapping = { 69 logging.FATAL: Level.FATAL, 70 logging.ERROR: Level.ERROR, 71 logging.WARNING: Level.WARN, 72 logging.INFO: Level.INFO, 73 logging.DEBUG: Level.DEBUG, 74 } 75 76 def format(self, record: logging.LogRecord) -> str: 77 """Return a JSON representation of the log message""" 78 airbyte_level = self.level_mapping.get(record.levelno, "INFO") 79 if airbyte_level == Level.DEBUG: 80 extras = self.extract_extra_args_from_record(record) 81 if is_platform_debug_log_enabled(): 82 # We have a different behavior between debug logs enabled through `--debug` argument and debug logs 83 # enabled through environment variable. The reason is that for platform logs, we need to have these 84 # printed as AirbyteMessage which is not the case with the current previous implementation. 85 # Why not migrate both to AirbyteMessages then? AirbyteMessages do not support having structured logs. 86 # which means that the DX would be degraded compared to the current solution (devs will need to identify 87 # the `log.message` field and figure out where in this field is the response while the current solution 88 # have a specific field that is structured for extras. 89 message = f"{filter_secrets(record.getMessage())} ///\nExtra logs: {filter_secrets(json.dumps(extras))}" 90 log_message = AirbyteMessage( 91 type=Type.LOG, log=AirbyteLogMessage(level=airbyte_level, message=message) 92 ) 93 return orjson.dumps(AirbyteMessageSerializer.dump(log_message)).decode() 94 else: 95 debug_dict = {"type": "DEBUG", "message": record.getMessage(), "data": extras} 96 return filter_secrets(json.dumps(debug_dict)) 97 else: 98 message = super().format(record) 99 message = filter_secrets(message) 100 log_message = AirbyteMessage( 101 type=Type.LOG, log=AirbyteLogMessage(level=airbyte_level, message=message) 102 ) 103 return orjson.dumps(AirbyteMessageSerializer.dump(log_message)).decode() 104 105 @staticmethod 106 def extract_extra_args_from_record(record: logging.LogRecord) -> Mapping[str, Any]: 107 """ 108 The python logger conflates default args with extra args. We use an empty log record and set operations 109 to isolate fields passed to the log record via extra by the developer. 110 """ 111 default_attrs = logging.LogRecord("", 0, "", 0, None, None, None).__dict__.keys() 112 extra_keys = set(record.__dict__.keys()) - default_attrs 113 return {k: str(getattr(record, k)) for k in extra_keys if hasattr(record, k)}
Output log records using AirbyteMessage
76 def format(self, record: logging.LogRecord) -> str: 77 """Return a JSON representation of the log message""" 78 airbyte_level = self.level_mapping.get(record.levelno, "INFO") 79 if airbyte_level == Level.DEBUG: 80 extras = self.extract_extra_args_from_record(record) 81 if is_platform_debug_log_enabled(): 82 # We have a different behavior between debug logs enabled through `--debug` argument and debug logs 83 # enabled through environment variable. The reason is that for platform logs, we need to have these 84 # printed as AirbyteMessage which is not the case with the current previous implementation. 85 # Why not migrate both to AirbyteMessages then? AirbyteMessages do not support having structured logs. 86 # which means that the DX would be degraded compared to the current solution (devs will need to identify 87 # the `log.message` field and figure out where in this field is the response while the current solution 88 # have a specific field that is structured for extras. 89 message = f"{filter_secrets(record.getMessage())} ///\nExtra logs: {filter_secrets(json.dumps(extras))}" 90 log_message = AirbyteMessage( 91 type=Type.LOG, log=AirbyteLogMessage(level=airbyte_level, message=message) 92 ) 93 return orjson.dumps(AirbyteMessageSerializer.dump(log_message)).decode() 94 else: 95 debug_dict = {"type": "DEBUG", "message": record.getMessage(), "data": extras} 96 return filter_secrets(json.dumps(debug_dict)) 97 else: 98 message = super().format(record) 99 message = filter_secrets(message) 100 log_message = AirbyteMessage( 101 type=Type.LOG, log=AirbyteLogMessage(level=airbyte_level, message=message) 102 ) 103 return orjson.dumps(AirbyteMessageSerializer.dump(log_message)).decode()
Return a JSON representation of the log message
105 @staticmethod 106 def extract_extra_args_from_record(record: logging.LogRecord) -> Mapping[str, Any]: 107 """ 108 The python logger conflates default args with extra args. We use an empty log record and set operations 109 to isolate fields passed to the log record via extra by the developer. 110 """ 111 default_attrs = logging.LogRecord("", 0, "", 0, None, None, None).__dict__.keys() 112 extra_keys = set(record.__dict__.keys()) - default_attrs 113 return {k: str(getattr(record, k)) for k in extra_keys if hasattr(record, k)}
The python logger conflates default args with extra args. We use an empty log record and set operations to isolate fields passed to the log record via extra by the developer.
48def init_logger(name: Optional[str] = None) -> logging.Logger: 49 """Initial set up of logger""" 50 logger = logging.getLogger(name) 51 logger.setLevel(logging.INFO) 52 logging.config.dictConfig(LOGGING_CONFIG) 53 return logger
Initial set up of logger
300@dataclass 301class AirbyteStream: 302 name: str 303 json_schema: Dict[str, Any] 304 supported_sync_modes: List[SyncMode] 305 source_defined_cursor: Optional[bool] = None 306 default_cursor_field: Optional[List[str]] = None 307 source_defined_primary_key: Optional[List[List[str]]] = None 308 namespace: Optional[str] = None 309 is_resumable: Optional[bool] = None 310 is_file_based: Optional[bool] = None
180@dataclass 181class AirbyteConnectionStatus: 182 status: Status 183 message: Optional[str] = None
140@dataclass 141class AirbyteMessage: 142 type: Type # type: ignore [name-defined] 143 log: Optional[AirbyteLogMessage] = None # type: ignore [name-defined] 144 spec: Optional[ConnectorSpecification] = None 145 connectionStatus: Optional[AirbyteConnectionStatus] = None # type: ignore [name-defined] 146 catalog: Optional[AirbyteCatalog] = None # type: ignore [name-defined] 147 record: Optional[AirbyteRecordMessage] = None # type: ignore [name-defined] 148 state: Optional[AirbyteStateMessage] = None 149 trace: Optional[AirbyteTraceMessage] = None # type: ignore [name-defined] 150 control: Optional[AirbyteControlMessage] = None # type: ignore [name-defined]
12class Type(Enum): 13 RECORD = 'RECORD' 14 STATE = 'STATE' 15 LOG = 'LOG' 16 SPEC = 'SPEC' 17 CONNECTION_STATUS = 'CONNECTION_STATUS' 18 CATALOG = 'CATALOG' 19 TRACE = 'TRACE' 20 CONTROL = 'CONTROL' 21 DESTINATION_CATALOG = 'DESTINATION_CATALOG'
313@dataclass 314class ConfiguredAirbyteStream: 315 stream: AirbyteStream 316 sync_mode: SyncMode 317 destination_sync_mode: DestinationSyncMode 318 cursor_field: Optional[List[str]] = None 319 destination_object_name: Optional[str] = None 320 primary_key: Optional[List[List[str]]] = None 321 generation_id: Optional[int] = None 322 minimum_generation_id: Optional[int] = None 323 sync_id: Optional[int] = None 324 include_files: Optional[bool] = None
191class DestinationSyncMode(Enum): 192 append = 'append' 193 overwrite = 'overwrite' 194 append_dedup = 'append_dedup' 195 update = 'update' 196 soft_delete = 'soft_delete'
102class FailureType(Enum): 103 system_error = 'system_error' 104 config_error = 'config_error' 105 transient_error = 'transient_error'
118@dataclass 119class AdvancedAuth: 120 auth_flow_type: Optional[AuthFlowType] = None # type: ignore [name-defined] 121 predicate_key: Optional[List[str]] = None 122 predicate_value: Optional[str] = None 123 oauth_config_specification: Optional[OAuthConfigSpecification] = None
88@dataclass 89class AirbyteLogMessage: 90 level: Level 91 message: str 92 stack_trace: Optional[str] = None
109@dataclass 110class OAuthConfigSpecification: 111 oauth_user_input_from_connector_config_specification: Optional[Dict[str, Any]] = None 112 oauth_connector_input_specification: Optional[OauthConnectorInputSpecification] = None 113 complete_oauth_output_specification: Optional[Dict[str, Any]] = None 114 complete_oauth_server_input_specification: Optional[Dict[str, Any]] = None 115 complete_oauth_server_output_specification: Optional[Dict[str, Any]] = None
126@dataclass 127class ConnectorSpecification: 128 connectionSpecification: Dict[str, Any] 129 documentationUrl: Optional[str] = None 130 changelogUrl: Optional[str] = None 131 supportsIncremental: Optional[bool] = None 132 supportsNormalization: Optional[bool] = False 133 supportsDBT: Optional[bool] = False 134 supported_destination_sync_modes: Optional[List[DestinationSyncMode]] = None # type: ignore [name-defined] 135 authSpecification: Optional[AuthSpecification] = None # type: ignore [name-defined] 136 advanced_auth: Optional[AdvancedAuth] = None 137 protocol_version: Optional[str] = None
79class Level(Enum): 80 FATAL = 'FATAL' 81 ERROR = 'ERROR' 82 WARN = 'WARN' 83 INFO = 'INFO' 84 DEBUG = 'DEBUG' 85 TRACE = 'TRACE'
354@dataclass 355class AirbyteRecordMessage: 356 stream: str 357 data: Dict[str, Any] 358 emitted_at: int 359 namespace: Optional[str] = None 360 meta: Optional[AirbyteRecordMessageMeta] = None 361 file_reference: Optional[AirbyteRecordMessageFileReference] = None
75class InMemoryMessageRepository(MessageRepository): 76 def __init__(self, log_level: Level = Level.INFO) -> None: 77 self._message_queue: Deque[AirbyteMessage] = deque() 78 self._log_level = log_level 79 80 def emit_message(self, message: AirbyteMessage) -> None: 81 self._message_queue.append(message) 82 83 def log_message(self, level: Level, message_provider: Callable[[], LogMessage]) -> None: 84 if _is_severe_enough(self._log_level, level): 85 self.emit_message( 86 AirbyteMessage( 87 type=Type.LOG, 88 log=AirbyteLogMessage( 89 level=level, message=filter_secrets(json.dumps(message_provider())) 90 ), 91 ) 92 ) 93 94 def consume_queue(self) -> Iterable[AirbyteMessage]: 95 while self._message_queue: 96 yield self._message_queue.popleft()
Helper class that provides a standard way to create an ABC using inheritance.
83 def log_message(self, level: Level, message_provider: Callable[[], LogMessage]) -> None: 84 if _is_severe_enough(self._log_level, level): 85 self.emit_message( 86 AirbyteMessage( 87 type=Type.LOG, 88 log=AirbyteLogMessage( 89 level=level, message=filter_secrets(json.dumps(message_provider())) 90 ), 91 ) 92 )
Computing messages can be resource consuming. This method is specialized for logging because we want to allow for lazy evaluation if the log level is less severe than what is configured
46class MessageRepository(ABC): 47 @abstractmethod 48 def emit_message(self, message: AirbyteMessage) -> None: 49 raise NotImplementedError() 50 51 @abstractmethod 52 def log_message(self, level: Level, message_provider: Callable[[], LogMessage]) -> None: 53 """ 54 Computing messages can be resource consuming. This method is specialized for logging because we want to allow for lazy evaluation if 55 the log level is less severe than what is configured 56 """ 57 raise NotImplementedError() 58 59 @abstractmethod 60 def consume_queue(self) -> Iterable[AirbyteMessage]: 61 raise NotImplementedError()
Helper class that provides a standard way to create an ABC using inheritance.
51 @abstractmethod 52 def log_message(self, level: Level, message_provider: Callable[[], LogMessage]) -> None: 53 """ 54 Computing messages can be resource consuming. This method is specialized for logging because we want to allow for lazy evaluation if 55 the log level is less severe than what is configured 56 """ 57 raise NotImplementedError()
Computing messages can be resource consuming. This method is specialized for logging because we want to allow for lazy evaluation if the log level is less severe than what is configured
33class ConnectorStateManager: 34 """ 35 ConnectorStateManager consolidates the various forms of a stream's incoming state message (STREAM / GLOBAL) under a common 36 interface. It also provides methods to extract and update state 37 """ 38 39 def __init__(self, state: Optional[List[AirbyteStateMessage]] = None): 40 shared_state, per_stream_states = self._extract_from_state_message(state) 41 42 # We explicitly throw an error if we receive a GLOBAL state message that contains a shared_state because API sources are 43 # designed to checkpoint state independently of one another. API sources should never be emitting a state message where 44 # shared_state is populated. Rather than define how to handle shared_state without a clear use case, we're opting to throw an 45 # error instead and if/when we find one, we will then implement processing of the shared_state value. 46 if shared_state: 47 raise ValueError( 48 "Received a GLOBAL AirbyteStateMessage that contains a shared_state. This library only ever generates per-STREAM " 49 "STATE messages so this was not generated by this connector. This must be an orchestrator or platform error. GLOBAL " 50 "state messages with shared_state will not be processed correctly. " 51 ) 52 self.per_stream_states = per_stream_states 53 54 def get_stream_state( 55 self, stream_name: str, namespace: Optional[str] 56 ) -> MutableMapping[str, Any]: 57 """ 58 Retrieves the state of a given stream based on its descriptor (name + namespace). 59 :param stream_name: Name of the stream being fetched 60 :param namespace: Namespace of the stream being fetched 61 :return: The per-stream state for a stream 62 """ 63 stream_state: AirbyteStateBlob | None = self.per_stream_states.get( 64 HashableStreamDescriptor(name=stream_name, namespace=namespace) 65 ) 66 if stream_state: 67 return copy.deepcopy({k: v for k, v in stream_state.__dict__.items()}) 68 return {} 69 70 def update_state_for_stream( 71 self, stream_name: str, namespace: Optional[str], value: Mapping[str, Any] 72 ) -> None: 73 """ 74 Overwrites the state blob of a specific stream based on the provided stream name and optional namespace 75 :param stream_name: The name of the stream whose state is being updated 76 :param namespace: The namespace of the stream if it exists 77 :param value: A stream state mapping that is being updated for a stream 78 """ 79 stream_descriptor = HashableStreamDescriptor(name=stream_name, namespace=namespace) 80 self.per_stream_states[stream_descriptor] = AirbyteStateBlob(value) 81 82 def create_state_message(self, stream_name: str, namespace: Optional[str]) -> AirbyteMessage: 83 """ 84 Generates an AirbyteMessage using the current per-stream state of a specified stream 85 :param stream_name: The name of the stream for the message that is being created 86 :param namespace: The namespace of the stream for the message that is being created 87 :return: The Airbyte state message to be emitted by the connector during a sync 88 """ 89 hashable_descriptor = HashableStreamDescriptor(name=stream_name, namespace=namespace) 90 stream_state = self.per_stream_states.get(hashable_descriptor) or AirbyteStateBlob() 91 92 return AirbyteMessage( 93 type=MessageType.STATE, 94 state=AirbyteStateMessage( 95 type=AirbyteStateType.STREAM, 96 stream=AirbyteStreamState( 97 stream_descriptor=StreamDescriptor(name=stream_name, namespace=namespace), 98 stream_state=stream_state, 99 ), 100 ), 101 ) 102 103 @classmethod 104 def _extract_from_state_message( 105 cls, 106 state: Optional[List[AirbyteStateMessage]], 107 ) -> Tuple[ 108 Optional[AirbyteStateBlob], 109 MutableMapping[HashableStreamDescriptor, Optional[AirbyteStateBlob]], 110 ]: 111 """ 112 Takes an incoming list of state messages or a global state message and extracts state attributes according to 113 type which can then be assigned to the new state manager being instantiated 114 :param state: The incoming state input 115 :return: A tuple of shared state and per stream state assembled from the incoming state list 116 """ 117 if state is None: 118 return None, {} 119 120 is_global = cls._is_global_state(state) 121 122 if is_global: 123 # We already validate that this is a global state message, not None: 124 global_state = cast(AirbyteGlobalState, state[0].global_) 125 # global_state has shared_state, also not None: 126 shared_state: AirbyteStateBlob = cast( 127 AirbyteStateBlob, copy.deepcopy(global_state.shared_state, {}) 128 ) 129 streams = { 130 HashableStreamDescriptor( 131 name=per_stream_state.stream_descriptor.name, 132 namespace=per_stream_state.stream_descriptor.namespace, 133 ): per_stream_state.stream_state 134 for per_stream_state in global_state.stream_states # type: ignore[union-attr] # global_state has shared_state 135 } 136 return shared_state, streams 137 else: 138 streams = { 139 HashableStreamDescriptor( 140 name=per_stream_state.stream.stream_descriptor.name, # type: ignore[union-attr] # stream has stream_descriptor 141 namespace=per_stream_state.stream.stream_descriptor.namespace, # type: ignore[union-attr] # stream has stream_descriptor 142 ): per_stream_state.stream.stream_state # type: ignore[union-attr] # stream has stream_state 143 for per_stream_state in state 144 if per_stream_state.type == AirbyteStateType.STREAM 145 and hasattr(per_stream_state, "stream") # type: ignore # state is always a list of AirbyteStateMessage if is_per_stream is True 146 } 147 return None, streams 148 149 @staticmethod 150 def _is_global_state(state: Union[List[AirbyteStateMessage], MutableMapping[str, Any]]) -> bool: 151 return ( 152 isinstance(state, List) 153 and len(state) == 1 154 and isinstance(state[0], AirbyteStateMessage) 155 and state[0].type == AirbyteStateType.GLOBAL 156 ) 157 158 @staticmethod 159 def _is_per_stream_state( 160 state: Union[List[AirbyteStateMessage], MutableMapping[str, Any]], 161 ) -> bool: 162 return isinstance(state, List)
ConnectorStateManager consolidates the various forms of a stream's incoming state message (STREAM / GLOBAL) under a common interface. It also provides methods to extract and update state
39 def __init__(self, state: Optional[List[AirbyteStateMessage]] = None): 40 shared_state, per_stream_states = self._extract_from_state_message(state) 41 42 # We explicitly throw an error if we receive a GLOBAL state message that contains a shared_state because API sources are 43 # designed to checkpoint state independently of one another. API sources should never be emitting a state message where 44 # shared_state is populated. Rather than define how to handle shared_state without a clear use case, we're opting to throw an 45 # error instead and if/when we find one, we will then implement processing of the shared_state value. 46 if shared_state: 47 raise ValueError( 48 "Received a GLOBAL AirbyteStateMessage that contains a shared_state. This library only ever generates per-STREAM " 49 "STATE messages so this was not generated by this connector. This must be an orchestrator or platform error. GLOBAL " 50 "state messages with shared_state will not be processed correctly. " 51 ) 52 self.per_stream_states = per_stream_states
54 def get_stream_state( 55 self, stream_name: str, namespace: Optional[str] 56 ) -> MutableMapping[str, Any]: 57 """ 58 Retrieves the state of a given stream based on its descriptor (name + namespace). 59 :param stream_name: Name of the stream being fetched 60 :param namespace: Namespace of the stream being fetched 61 :return: The per-stream state for a stream 62 """ 63 stream_state: AirbyteStateBlob | None = self.per_stream_states.get( 64 HashableStreamDescriptor(name=stream_name, namespace=namespace) 65 ) 66 if stream_state: 67 return copy.deepcopy({k: v for k, v in stream_state.__dict__.items()}) 68 return {}
Retrieves the state of a given stream based on its descriptor (name + namespace).
Parameters
- stream_name: Name of the stream being fetched
- namespace: Namespace of the stream being fetched
Returns
The per-stream state for a stream
70 def update_state_for_stream( 71 self, stream_name: str, namespace: Optional[str], value: Mapping[str, Any] 72 ) -> None: 73 """ 74 Overwrites the state blob of a specific stream based on the provided stream name and optional namespace 75 :param stream_name: The name of the stream whose state is being updated 76 :param namespace: The namespace of the stream if it exists 77 :param value: A stream state mapping that is being updated for a stream 78 """ 79 stream_descriptor = HashableStreamDescriptor(name=stream_name, namespace=namespace) 80 self.per_stream_states[stream_descriptor] = AirbyteStateBlob(value)
Overwrites the state blob of a specific stream based on the provided stream name and optional namespace
Parameters
- stream_name: The name of the stream whose state is being updated
- namespace: The namespace of the stream if it exists
- value: A stream state mapping that is being updated for a stream
82 def create_state_message(self, stream_name: str, namespace: Optional[str]) -> AirbyteMessage: 83 """ 84 Generates an AirbyteMessage using the current per-stream state of a specified stream 85 :param stream_name: The name of the stream for the message that is being created 86 :param namespace: The namespace of the stream for the message that is being created 87 :return: The Airbyte state message to be emitted by the connector during a sync 88 """ 89 hashable_descriptor = HashableStreamDescriptor(name=stream_name, namespace=namespace) 90 stream_state = self.per_stream_states.get(hashable_descriptor) or AirbyteStateBlob() 91 92 return AirbyteMessage( 93 type=MessageType.STATE, 94 state=AirbyteStateMessage( 95 type=AirbyteStateType.STREAM, 96 stream=AirbyteStreamState( 97 stream_descriptor=StreamDescriptor(name=stream_name, namespace=namespace), 98 stream_state=stream_state, 99 ), 100 ), 101 )
Generates an AirbyteMessage using the current per-stream state of a specified stream
Parameters
- stream_name: The name of the stream for the message that is being created
- namespace: The namespace of the stream for the message that is being created
Returns
The Airbyte state message to be emitted by the connector during a sync
95@deprecated( 96 "Deprecated as of CDK version 0.87.0. " 97 "Deprecated in favor of the `CheckpointMixin` which offers similar functionality." 98) 99class IncrementalMixin(CheckpointMixin, ABC): 100 """Mixin to make stream incremental. 101 102 class IncrementalStream(Stream, IncrementalMixin): 103 @property 104 def state(self): 105 return self._state 106 107 @state.setter 108 def state(self, value): 109 self._state[self.cursor_field] = value[self.cursor_field] 110 """
Mixin to make stream incremental.
class IncrementalStream(Stream, IncrementalMixin): @property def state(self): return self._state
@state.setter
def state(self, value):
self._state[self.cursor_field] = value[self.cursor_field]
Inherited Members
119class Stream(ABC): 120 """ 121 Base abstract class for an Airbyte Stream. Makes no assumption of the Stream's underlying transport protocol. 122 """ 123 124 _configured_json_schema: Optional[Dict[str, Any]] = None 125 _exit_on_rate_limit: bool = False 126 127 # Use self.logger in subclasses to log any messages 128 @property 129 def logger(self) -> logging.Logger: 130 return logging.getLogger(f"airbyte.streams.{self.name}") 131 132 # TypeTransformer object to perform output data transformation 133 transformer: TypeTransformer = TypeTransformer(TransformConfig.NoTransform) 134 135 cursor: Optional[Cursor] = None 136 137 has_multiple_slices = False 138 139 @cached_property 140 def name(self) -> str: 141 """ 142 :return: Stream name. By default this is the implementing class name, but it can be overridden as needed. 143 """ 144 return casing.camel_to_snake(self.__class__.__name__) 145 146 def get_error_display_message(self, exception: BaseException) -> Optional[str]: 147 """ 148 Retrieves the user-friendly display message that corresponds to an exception. 149 This will be called when encountering an exception while reading records from the stream, and used to build the AirbyteTraceMessage. 150 151 The default implementation of this method does not return user-friendly messages for any exception type, but it should be overriden as needed. 152 153 :param exception: The exception that was raised 154 :return: A user-friendly message that indicates the cause of the error 155 """ 156 return None 157 158 def read( # type: ignore # ignoring typing for ConnectorStateManager because of circular dependencies 159 self, 160 configured_stream: ConfiguredAirbyteStream, 161 logger: logging.Logger, 162 slice_logger: SliceLogger, 163 stream_state: MutableMapping[str, Any], 164 state_manager, 165 internal_config: InternalConfig, 166 ) -> Iterable[StreamData]: 167 sync_mode = configured_stream.sync_mode 168 cursor_field = configured_stream.cursor_field 169 self.configured_json_schema = configured_stream.stream.json_schema 170 171 # WARNING: When performing a read() that uses incoming stream state, we MUST use the self.state that is defined as 172 # opposed to the incoming stream_state value. Because some connectors like ones using the file-based CDK modify 173 # state before setting the value on the Stream attribute, the most up-to-date state is derived from Stream.state 174 # instead of the stream_state parameter. This does not apply to legacy connectors using get_updated_state(). 175 try: 176 stream_state = self.state # type: ignore # we know the field might not exist... 177 except AttributeError: 178 pass 179 180 should_checkpoint = bool(state_manager) 181 checkpoint_reader = self._get_checkpoint_reader( 182 logger=logger, cursor_field=cursor_field, sync_mode=sync_mode, stream_state=stream_state 183 ) 184 185 next_slice = checkpoint_reader.next() 186 record_counter = 0 187 stream_state_tracker = copy.deepcopy(stream_state) 188 while next_slice is not None: 189 if slice_logger.should_log_slice_message(logger): 190 yield slice_logger.create_slice_log_message(next_slice) 191 records = self.read_records( 192 sync_mode=sync_mode, # todo: change this interface to no longer rely on sync_mode for behavior 193 stream_slice=next_slice, 194 stream_state=stream_state, 195 cursor_field=cursor_field or None, 196 ) 197 for record_data_or_message in records: 198 yield record_data_or_message 199 if isinstance(record_data_or_message, Mapping) or ( 200 hasattr(record_data_or_message, "type") 201 and record_data_or_message.type == MessageType.RECORD 202 ): 203 record_data = ( 204 record_data_or_message 205 if isinstance(record_data_or_message, Mapping) 206 else record_data_or_message.record 207 ) 208 209 # Thanks I hate it. RFR fundamentally doesn't fit with the concept of the legacy Stream.get_updated_state() 210 # method because RFR streams rely on pagination as a cursor. Stream.get_updated_state() was designed to make 211 # the CDK manage state using specifically the last seen record. don't @ brian.lai 212 # 213 # Also, because the legacy incremental state case decouples observing incoming records from emitting state, it 214 # requires that we separate CheckpointReader.observe() and CheckpointReader.get_checkpoint() which could 215 # otherwise be combined. 216 if self.cursor_field: 217 # Some connectors have streams that implement get_updated_state(), but do not define a cursor_field. This 218 # should be fixed on the stream implementation, but we should also protect against this in the CDK as well 219 stream_state_tracker = self.get_updated_state( 220 stream_state_tracker, 221 record_data, # type: ignore [arg-type] 222 ) 223 self._observe_state(checkpoint_reader, stream_state_tracker) 224 record_counter += 1 225 226 checkpoint_interval = self.state_checkpoint_interval 227 if ( 228 should_checkpoint 229 and checkpoint_interval 230 and record_counter % checkpoint_interval == 0 231 ): 232 checkpoint = checkpoint_reader.get_checkpoint() 233 if checkpoint: 234 airbyte_state_message = self._checkpoint_state( 235 checkpoint, state_manager=state_manager 236 ) 237 yield airbyte_state_message 238 239 if internal_config.is_limit_reached(record_counter): 240 break 241 self._observe_state(checkpoint_reader) 242 checkpoint_state = checkpoint_reader.get_checkpoint() 243 if should_checkpoint and checkpoint_state is not None: 244 airbyte_state_message = self._checkpoint_state( 245 checkpoint_state, state_manager=state_manager 246 ) 247 yield airbyte_state_message 248 249 next_slice = checkpoint_reader.next() 250 251 checkpoint = checkpoint_reader.get_checkpoint() 252 if should_checkpoint and checkpoint is not None: 253 airbyte_state_message = self._checkpoint_state(checkpoint, state_manager=state_manager) 254 yield airbyte_state_message 255 256 def read_only_records(self, state: Optional[Mapping[str, Any]] = None) -> Iterable[StreamData]: 257 """ 258 Helper method that performs a read on a stream with an optional state and emits records. If the parent stream supports 259 incremental, this operation does not update the stream's internal state (if it uses the modern state setter/getter) 260 or emit state messages. 261 """ 262 263 configured_stream = ConfiguredAirbyteStream( 264 stream=AirbyteStream( 265 name=self.name, 266 json_schema={}, 267 supported_sync_modes=[SyncMode.full_refresh, SyncMode.incremental], 268 ), 269 sync_mode=SyncMode.incremental if state else SyncMode.full_refresh, 270 destination_sync_mode=DestinationSyncMode.append, 271 ) 272 273 yield from self.read( 274 configured_stream=configured_stream, 275 logger=self.logger, 276 slice_logger=DebugSliceLogger(), 277 stream_state=dict(state) 278 if state 279 else {}, # read() expects MutableMapping instead of Mapping which is used more often 280 state_manager=None, 281 internal_config=InternalConfig(), # type: ignore [call-arg] 282 ) 283 284 @abstractmethod 285 def read_records( 286 self, 287 sync_mode: SyncMode, 288 cursor_field: Optional[List[str]] = None, 289 stream_slice: Optional[Mapping[str, Any]] = None, 290 stream_state: Optional[Mapping[str, Any]] = None, 291 ) -> Iterable[StreamData]: 292 """ 293 This method should be overridden by subclasses to read records based on the inputs 294 """ 295 296 @lru_cache(maxsize=None) 297 def get_json_schema(self) -> Mapping[str, Any]: 298 """ 299 :return: A dict of the JSON schema representing this stream. 300 301 The default implementation of this method looks for a JSONSchema file with the same name as this stream's "name" property. 302 Override as needed. 303 """ 304 # TODO show an example of using pydantic to define the JSON schema, or reading an OpenAPI spec 305 return ResourceSchemaLoader(package_name_from_class(self.__class__)).get_schema(self.name) 306 307 def as_airbyte_stream(self) -> AirbyteStream: 308 stream = AirbyteStream( 309 name=self.name, 310 json_schema=dict(self.get_json_schema()), 311 supported_sync_modes=[SyncMode.full_refresh], 312 is_resumable=self.is_resumable, 313 ) 314 315 if self.namespace: 316 stream.namespace = self.namespace 317 318 # If we can offer incremental we always should. RFR is always less reliable than incremental which uses a real cursor value 319 if self.supports_incremental: 320 stream.source_defined_cursor = self.source_defined_cursor 321 stream.supported_sync_modes.append(SyncMode.incremental) 322 stream.default_cursor_field = self._wrapped_cursor_field() 323 324 keys = Stream._wrapped_primary_key(self.primary_key) 325 if keys and len(keys) > 0: 326 stream.source_defined_primary_key = keys 327 328 return stream 329 330 @property 331 def supports_incremental(self) -> bool: 332 """ 333 :return: True if this stream supports incrementally reading data 334 """ 335 return len(self._wrapped_cursor_field()) > 0 336 337 @property 338 def is_resumable(self) -> bool: 339 """ 340 :return: True if this stream allows the checkpointing of sync progress and can resume from it on subsequent attempts. 341 This differs from supports_incremental because certain kinds of streams like those supporting resumable full refresh 342 can checkpoint progress in between attempts for improved fault tolerance. However, they will start from the beginning 343 on the next sync job. 344 """ 345 if self.supports_incremental: 346 return True 347 if self.has_multiple_slices: 348 # We temporarily gate substream to not support RFR because puts a pretty high burden on connector developers 349 # to structure stream state in a very specific way. We also can't check for issubclass(HttpSubStream) because 350 # not all substreams implement the interface and it would be a circular dependency so we use parent as a surrogate 351 return False 352 elif hasattr(type(self), "state") and getattr(type(self), "state").fset is not None: 353 # Modern case where a stream manages state using getter/setter 354 return True 355 else: 356 # Legacy case where the CDK manages state via the get_updated_state() method. This is determined by checking if 357 # the stream's get_updated_state() differs from the Stream class and therefore has been overridden 358 return type(self).get_updated_state != Stream.get_updated_state 359 360 def _wrapped_cursor_field(self) -> List[str]: 361 return [self.cursor_field] if isinstance(self.cursor_field, str) else self.cursor_field 362 363 @property 364 def cursor_field(self) -> Union[str, List[str]]: 365 """ 366 Override to return the default cursor field used by this stream e.g: an API entity might always use created_at as the cursor field. 367 :return: The name of the field used as a cursor. If the cursor is nested, return an array consisting of the path to the cursor. 368 """ 369 return [] 370 371 @property 372 def namespace(self) -> Optional[str]: 373 """ 374 Override to return the namespace of this stream, e.g. the Postgres schema which this stream will emit records for. 375 :return: A string containing the name of the namespace. 376 """ 377 return None 378 379 @property 380 def source_defined_cursor(self) -> bool: 381 """ 382 Return False if the cursor can be configured by the user. 383 """ 384 return True 385 386 @property 387 def exit_on_rate_limit(self) -> bool: 388 """Exit on rate limit getter, should return bool value. False if the stream will retry endlessly when rate limited.""" 389 return self._exit_on_rate_limit 390 391 @exit_on_rate_limit.setter 392 def exit_on_rate_limit(self, value: bool) -> None: 393 """Exit on rate limit setter, accept bool value.""" 394 self._exit_on_rate_limit = value 395 396 @property 397 @abstractmethod 398 def primary_key(self) -> Optional[Union[str, List[str], List[List[str]]]]: 399 """ 400 :return: string if single primary key, list of strings if composite primary key, list of list of strings if composite primary key consisting of nested fields. 401 If the stream has no primary keys, return None. 402 """ 403 404 def stream_slices( 405 self, 406 *, 407 sync_mode: SyncMode, 408 cursor_field: Optional[List[str]] = None, 409 stream_state: Optional[Mapping[str, Any]] = None, 410 ) -> Iterable[Optional[Mapping[str, Any]]]: 411 """ 412 Override to define the slices for this stream. See the stream slicing section of the docs for more information. 413 414 :param sync_mode: 415 :param cursor_field: 416 :param stream_state: 417 :return: 418 """ 419 yield StreamSlice(partition={}, cursor_slice={}) 420 421 @property 422 def state_checkpoint_interval(self) -> Optional[int]: 423 """ 424 Decides how often to checkpoint state (i.e: emit a STATE message). E.g: if this returns a value of 100, then state is persisted after reading 425 100 records, then 200, 300, etc.. A good default value is 1000 although your mileage may vary depending on the underlying data source. 426 427 Checkpointing a stream avoids re-reading records in the case a sync is failed or cancelled. 428 429 return None if state should not be checkpointed e.g: because records returned from the underlying data source are not returned in 430 ascending order with respect to the cursor field. This can happen if the source does not support reading records in ascending order of 431 created_at date (or whatever the cursor is). In those cases, state must only be saved once the full stream has been read. 432 """ 433 return None 434 435 # Commented-out to avoid any runtime penalty, since this is used in a hot per-record codepath. 436 # To be evaluated for re-introduction here: https://github.com/airbytehq/airbyte-python-cdk/issues/116 437 # @deprecated( 438 # "Deprecated method `get_updated_state` as of CDK version 0.1.49. " 439 # "Please use explicit state property instead, see `IncrementalMixin` docs." 440 # ) 441 def get_updated_state( 442 self, current_stream_state: MutableMapping[str, Any], latest_record: Mapping[str, Any] 443 ) -> MutableMapping[str, Any]: 444 """DEPRECATED. Please use explicit state property instead, see `IncrementalMixin` docs. 445 446 Override to extract state from the latest record. Needed to implement incremental sync. 447 448 Inspects the latest record extracted from the data source and the current state object and return an updated state object. 449 450 For example: if the state object is based on created_at timestamp, and the current state is {'created_at': 10}, and the latest_record is 451 {'name': 'octavia', 'created_at': 20 } then this method would return {'created_at': 20} to indicate state should be updated to this object. 452 453 :param current_stream_state: The stream's current state object 454 :param latest_record: The latest record extracted from the stream 455 :return: An updated state object 456 """ 457 return {} 458 459 def get_cursor(self) -> Optional[Cursor]: 460 """ 461 A Cursor is an interface that a stream can implement to manage how its internal state is read and updated while 462 reading records. Historically, Python connectors had no concept of a cursor to manage state. Python streams need 463 to define a cursor implementation and override this method to manage state through a Cursor. 464 """ 465 return self.cursor 466 467 def _get_checkpoint_reader( 468 self, 469 logger: logging.Logger, 470 cursor_field: Optional[List[str]], 471 sync_mode: SyncMode, 472 stream_state: MutableMapping[str, Any], 473 ) -> CheckpointReader: 474 mappings_or_slices = self.stream_slices( 475 cursor_field=cursor_field, 476 sync_mode=sync_mode, # todo: change this interface to no longer rely on sync_mode for behavior 477 stream_state=stream_state, 478 ) 479 480 # Because of poor foresight, we wrote the default Stream.stream_slices() method to return [None] which is confusing and 481 # has now normalized this behavior for connector developers. Now some connectors return [None]. This is objectively 482 # misleading and a more ideal interface is [{}] to indicate we still want to iterate over one slice, but with no 483 # specific slice values. None is bad, and now I feel bad that I have to write this hack. 484 if mappings_or_slices == [None]: 485 mappings_or_slices = [{}] 486 487 slices_iterable_copy, iterable_for_detecting_format = itertools.tee(mappings_or_slices, 2) 488 stream_classification = self._classify_stream( 489 mappings_or_slices=iterable_for_detecting_format 490 ) 491 492 # Streams that override has_multiple_slices are explicitly indicating that they will iterate over 493 # multiple partitions. Inspecting slices to automatically apply the correct cursor is only needed as 494 # a backup. So if this value was already assigned to True by the stream, we don't need to reassign it 495 self.has_multiple_slices = ( 496 self.has_multiple_slices or stream_classification.has_multiple_slices 497 ) 498 499 cursor = self.get_cursor() 500 if cursor: 501 cursor.set_initial_state(stream_state=stream_state) 502 503 checkpoint_mode = self._checkpoint_mode 504 505 if cursor and stream_classification.is_legacy_format: 506 return LegacyCursorBasedCheckpointReader( 507 stream_slices=slices_iterable_copy, cursor=cursor, read_state_from_cursor=True 508 ) 509 elif cursor: 510 return CursorBasedCheckpointReader( 511 stream_slices=slices_iterable_copy, 512 cursor=cursor, 513 read_state_from_cursor=checkpoint_mode == CheckpointMode.RESUMABLE_FULL_REFRESH, 514 ) 515 elif checkpoint_mode == CheckpointMode.RESUMABLE_FULL_REFRESH: 516 # Resumable full refresh readers rely on the stream state dynamically being updated during pagination and does 517 # not iterate over a static set of slices. 518 return ResumableFullRefreshCheckpointReader(stream_state=stream_state) 519 elif checkpoint_mode == CheckpointMode.INCREMENTAL: 520 return IncrementalCheckpointReader( 521 stream_slices=slices_iterable_copy, stream_state=stream_state 522 ) 523 else: 524 return FullRefreshCheckpointReader(stream_slices=slices_iterable_copy) 525 526 @property 527 def _checkpoint_mode(self) -> CheckpointMode: 528 if self.is_resumable and len(self._wrapped_cursor_field()) > 0: 529 return CheckpointMode.INCREMENTAL 530 elif self.is_resumable: 531 return CheckpointMode.RESUMABLE_FULL_REFRESH 532 else: 533 return CheckpointMode.FULL_REFRESH 534 535 @staticmethod 536 def _classify_stream( 537 mappings_or_slices: Iterator[Optional[Union[Mapping[str, Any], StreamSlice]]], 538 ) -> StreamClassification: 539 """ 540 This is a bit of a crazy solution, but also the only way we can detect certain attributes about the stream since Python 541 streams do not follow consistent implementation patterns. We care about the following two attributes: 542 - is_substream: Helps to incrementally release changes since substreams w/ parents are much more complicated. Also 543 helps de-risk the release of changes that might impact all connectors 544 - uses_legacy_slice_format: Since the checkpoint reader must manage a complex state object, we opted to have it always 545 use the structured StreamSlice object. However, this requires backwards compatibility with Python sources that only 546 support the legacy mapping object 547 548 Both attributes can eventually be deprecated once stream's define this method deleted once substreams have been implemented and 549 legacy connectors all adhere to the StreamSlice object. 550 """ 551 if not mappings_or_slices: 552 raise ValueError("A stream should always have at least one slice") 553 try: 554 next_slice = next(mappings_or_slices) 555 if isinstance(next_slice, StreamSlice) and next_slice == StreamSlice( 556 partition={}, cursor_slice={} 557 ): 558 is_legacy_format = False 559 slice_has_value = False 560 elif next_slice == {}: 561 is_legacy_format = True 562 slice_has_value = False 563 elif isinstance(next_slice, StreamSlice): 564 is_legacy_format = False 565 slice_has_value = True 566 else: 567 is_legacy_format = True 568 slice_has_value = True 569 except StopIteration: 570 # If the stream has no slices, the format ultimately does not matter since no data will get synced. This is technically 571 # a valid case because it is up to the stream to define its slicing behavior 572 return StreamClassification(is_legacy_format=False, has_multiple_slices=False) 573 574 if slice_has_value: 575 # If the first slice contained a partition value from the result of stream_slices(), this is a substream that might 576 # have multiple parent records to iterate over 577 return StreamClassification( 578 is_legacy_format=is_legacy_format, has_multiple_slices=slice_has_value 579 ) 580 581 try: 582 # If stream_slices() returns multiple slices, this is also a substream that can potentially generate empty slices 583 next(mappings_or_slices) 584 return StreamClassification(is_legacy_format=is_legacy_format, has_multiple_slices=True) 585 except StopIteration: 586 # If the result of stream_slices() only returns a single empty stream slice, then we know this is a regular stream 587 return StreamClassification( 588 is_legacy_format=is_legacy_format, has_multiple_slices=False 589 ) 590 591 def log_stream_sync_configuration(self) -> None: 592 """ 593 Logs the configuration of this stream. 594 """ 595 self.logger.debug( 596 f"Syncing stream instance: {self.name}", 597 extra={ 598 "primary_key": self.primary_key, 599 "cursor_field": self.cursor_field, 600 }, 601 ) 602 603 @staticmethod 604 def _wrapped_primary_key( 605 keys: Optional[Union[str, List[str], List[List[str]]]], 606 ) -> Optional[List[List[str]]]: 607 """ 608 :return: wrap the primary_key property in a list of list of strings required by the Airbyte Stream object. 609 """ 610 if not keys: 611 return None 612 613 if isinstance(keys, str): 614 return [[keys]] 615 elif isinstance(keys, list): 616 wrapped_keys = [] 617 for component in keys: 618 if isinstance(component, str): 619 wrapped_keys.append([component]) 620 elif isinstance(component, list): 621 wrapped_keys.append(component) 622 else: 623 raise ValueError(f"Element must be either list or str. Got: {type(component)}") 624 return wrapped_keys 625 else: 626 raise ValueError(f"Element must be either list or str. Got: {type(keys)}") 627 628 def _observe_state( 629 self, checkpoint_reader: CheckpointReader, stream_state: Optional[Mapping[str, Any]] = None 630 ) -> None: 631 """ 632 Convenience method that attempts to read the Stream's state using the recommended way of connector's managing their 633 own state via state setter/getter. But if we get back an AttributeError, then the legacy Stream.get_updated_state() 634 method is used as a fallback method. 635 """ 636 637 # This is an inversion of the original logic that used to try state getter/setters first. As part of the work to 638 # automatically apply resumable full refresh to all streams, all HttpStream classes implement default state 639 # getter/setter methods, we should default to only using the incoming stream_state parameter value is {} which 640 # indicates the stream does not override the default get_updated_state() implementation. When the default method 641 # is not overridden, then the stream defers to self.state getter 642 if stream_state: 643 checkpoint_reader.observe(stream_state) 644 elif type(self).get_updated_state == Stream.get_updated_state: 645 # We only default to the state getter/setter if the stream does not use the legacy get_updated_state() method 646 try: 647 new_state = self.state # type: ignore # This will always exist on HttpStreams, but may not for Stream 648 if new_state: 649 checkpoint_reader.observe(new_state) 650 except AttributeError: 651 pass 652 653 def _checkpoint_state( # type: ignore # ignoring typing for ConnectorStateManager because of circular dependencies 654 self, 655 stream_state: Mapping[str, Any], 656 state_manager, 657 ) -> AirbyteMessage: 658 # todo: This can be consolidated into one ConnectorStateManager.update_and_create_state_message() method, but I want 659 # to reduce changes right now and this would span concurrent as well 660 state_manager.update_state_for_stream(self.name, self.namespace, stream_state) 661 return state_manager.create_state_message(self.name, self.namespace) # type: ignore [no-any-return] 662 663 @property 664 def configured_json_schema(self) -> Optional[Dict[str, Any]]: 665 """ 666 This property is set from the read method. 667 668 :return Optional[Dict]: JSON schema from configured catalog if provided, otherwise None. 669 """ 670 return self._configured_json_schema 671 672 @configured_json_schema.setter 673 def configured_json_schema(self, json_schema: Dict[str, Any]) -> None: 674 self._configured_json_schema = self._filter_schema_invalid_properties(json_schema) 675 676 def _filter_schema_invalid_properties( 677 self, configured_catalog_json_schema: Dict[str, Any] 678 ) -> Dict[str, Any]: 679 """ 680 Filters the properties in json_schema that are not present in the stream schema. 681 Configured Schemas can have very old fields, so we need to housekeeping ourselves. 682 """ 683 configured_schema: Any = configured_catalog_json_schema.get("properties", {}) 684 stream_schema_properties: Any = self.get_json_schema().get("properties", {}) 685 686 configured_keys = configured_schema.keys() 687 stream_keys = stream_schema_properties.keys() 688 invalid_properties = configured_keys - stream_keys 689 if not invalid_properties: 690 return configured_catalog_json_schema 691 692 self.logger.warning( 693 f"Stream {self.name}: the following fields are deprecated and cannot be synced. {invalid_properties}. Refresh the connection's source schema to resolve this warning." 694 ) 695 696 valid_configured_schema_properties_keys = stream_keys & configured_keys 697 valid_configured_schema_properties = {} 698 699 for configured_schema_property in valid_configured_schema_properties_keys: 700 valid_configured_schema_properties[configured_schema_property] = ( 701 stream_schema_properties[configured_schema_property] 702 ) 703 704 return {**configured_catalog_json_schema, "properties": valid_configured_schema_properties}
Base abstract class for an Airbyte Stream. Makes no assumption of the Stream's underlying transport protocol.
139 @cached_property 140 def name(self) -> str: 141 """ 142 :return: Stream name. By default this is the implementing class name, but it can be overridden as needed. 143 """ 144 return casing.camel_to_snake(self.__class__.__name__)
Returns
Stream name. By default this is the implementing class name, but it can be overridden as needed.
146 def get_error_display_message(self, exception: BaseException) -> Optional[str]: 147 """ 148 Retrieves the user-friendly display message that corresponds to an exception. 149 This will be called when encountering an exception while reading records from the stream, and used to build the AirbyteTraceMessage. 150 151 The default implementation of this method does not return user-friendly messages for any exception type, but it should be overriden as needed. 152 153 :param exception: The exception that was raised 154 :return: A user-friendly message that indicates the cause of the error 155 """ 156 return None
Retrieves the user-friendly display message that corresponds to an exception. This will be called when encountering an exception while reading records from the stream, and used to build the AirbyteTraceMessage.
The default implementation of this method does not return user-friendly messages for any exception type, but it should be overriden as needed.
Parameters
- exception: The exception that was raised
Returns
A user-friendly message that indicates the cause of the error
158 def read( # type: ignore # ignoring typing for ConnectorStateManager because of circular dependencies 159 self, 160 configured_stream: ConfiguredAirbyteStream, 161 logger: logging.Logger, 162 slice_logger: SliceLogger, 163 stream_state: MutableMapping[str, Any], 164 state_manager, 165 internal_config: InternalConfig, 166 ) -> Iterable[StreamData]: 167 sync_mode = configured_stream.sync_mode 168 cursor_field = configured_stream.cursor_field 169 self.configured_json_schema = configured_stream.stream.json_schema 170 171 # WARNING: When performing a read() that uses incoming stream state, we MUST use the self.state that is defined as 172 # opposed to the incoming stream_state value. Because some connectors like ones using the file-based CDK modify 173 # state before setting the value on the Stream attribute, the most up-to-date state is derived from Stream.state 174 # instead of the stream_state parameter. This does not apply to legacy connectors using get_updated_state(). 175 try: 176 stream_state = self.state # type: ignore # we know the field might not exist... 177 except AttributeError: 178 pass 179 180 should_checkpoint = bool(state_manager) 181 checkpoint_reader = self._get_checkpoint_reader( 182 logger=logger, cursor_field=cursor_field, sync_mode=sync_mode, stream_state=stream_state 183 ) 184 185 next_slice = checkpoint_reader.next() 186 record_counter = 0 187 stream_state_tracker = copy.deepcopy(stream_state) 188 while next_slice is not None: 189 if slice_logger.should_log_slice_message(logger): 190 yield slice_logger.create_slice_log_message(next_slice) 191 records = self.read_records( 192 sync_mode=sync_mode, # todo: change this interface to no longer rely on sync_mode for behavior 193 stream_slice=next_slice, 194 stream_state=stream_state, 195 cursor_field=cursor_field or None, 196 ) 197 for record_data_or_message in records: 198 yield record_data_or_message 199 if isinstance(record_data_or_message, Mapping) or ( 200 hasattr(record_data_or_message, "type") 201 and record_data_or_message.type == MessageType.RECORD 202 ): 203 record_data = ( 204 record_data_or_message 205 if isinstance(record_data_or_message, Mapping) 206 else record_data_or_message.record 207 ) 208 209 # Thanks I hate it. RFR fundamentally doesn't fit with the concept of the legacy Stream.get_updated_state() 210 # method because RFR streams rely on pagination as a cursor. Stream.get_updated_state() was designed to make 211 # the CDK manage state using specifically the last seen record. don't @ brian.lai 212 # 213 # Also, because the legacy incremental state case decouples observing incoming records from emitting state, it 214 # requires that we separate CheckpointReader.observe() and CheckpointReader.get_checkpoint() which could 215 # otherwise be combined. 216 if self.cursor_field: 217 # Some connectors have streams that implement get_updated_state(), but do not define a cursor_field. This 218 # should be fixed on the stream implementation, but we should also protect against this in the CDK as well 219 stream_state_tracker = self.get_updated_state( 220 stream_state_tracker, 221 record_data, # type: ignore [arg-type] 222 ) 223 self._observe_state(checkpoint_reader, stream_state_tracker) 224 record_counter += 1 225 226 checkpoint_interval = self.state_checkpoint_interval 227 if ( 228 should_checkpoint 229 and checkpoint_interval 230 and record_counter % checkpoint_interval == 0 231 ): 232 checkpoint = checkpoint_reader.get_checkpoint() 233 if checkpoint: 234 airbyte_state_message = self._checkpoint_state( 235 checkpoint, state_manager=state_manager 236 ) 237 yield airbyte_state_message 238 239 if internal_config.is_limit_reached(record_counter): 240 break 241 self._observe_state(checkpoint_reader) 242 checkpoint_state = checkpoint_reader.get_checkpoint() 243 if should_checkpoint and checkpoint_state is not None: 244 airbyte_state_message = self._checkpoint_state( 245 checkpoint_state, state_manager=state_manager 246 ) 247 yield airbyte_state_message 248 249 next_slice = checkpoint_reader.next() 250 251 checkpoint = checkpoint_reader.get_checkpoint() 252 if should_checkpoint and checkpoint is not None: 253 airbyte_state_message = self._checkpoint_state(checkpoint, state_manager=state_manager) 254 yield airbyte_state_message
256 def read_only_records(self, state: Optional[Mapping[str, Any]] = None) -> Iterable[StreamData]: 257 """ 258 Helper method that performs a read on a stream with an optional state and emits records. If the parent stream supports 259 incremental, this operation does not update the stream's internal state (if it uses the modern state setter/getter) 260 or emit state messages. 261 """ 262 263 configured_stream = ConfiguredAirbyteStream( 264 stream=AirbyteStream( 265 name=self.name, 266 json_schema={}, 267 supported_sync_modes=[SyncMode.full_refresh, SyncMode.incremental], 268 ), 269 sync_mode=SyncMode.incremental if state else SyncMode.full_refresh, 270 destination_sync_mode=DestinationSyncMode.append, 271 ) 272 273 yield from self.read( 274 configured_stream=configured_stream, 275 logger=self.logger, 276 slice_logger=DebugSliceLogger(), 277 stream_state=dict(state) 278 if state 279 else {}, # read() expects MutableMapping instead of Mapping which is used more often 280 state_manager=None, 281 internal_config=InternalConfig(), # type: ignore [call-arg] 282 )
Helper method that performs a read on a stream with an optional state and emits records. If the parent stream supports incremental, this operation does not update the stream's internal state (if it uses the modern state setter/getter) or emit state messages.
284 @abstractmethod 285 def read_records( 286 self, 287 sync_mode: SyncMode, 288 cursor_field: Optional[List[str]] = None, 289 stream_slice: Optional[Mapping[str, Any]] = None, 290 stream_state: Optional[Mapping[str, Any]] = None, 291 ) -> Iterable[StreamData]: 292 """ 293 This method should be overridden by subclasses to read records based on the inputs 294 """
This method should be overridden by subclasses to read records based on the inputs
296 @lru_cache(maxsize=None) 297 def get_json_schema(self) -> Mapping[str, Any]: 298 """ 299 :return: A dict of the JSON schema representing this stream. 300 301 The default implementation of this method looks for a JSONSchema file with the same name as this stream's "name" property. 302 Override as needed. 303 """ 304 # TODO show an example of using pydantic to define the JSON schema, or reading an OpenAPI spec 305 return ResourceSchemaLoader(package_name_from_class(self.__class__)).get_schema(self.name)
Returns
A dict of the JSON schema representing this stream.
The default implementation of this method looks for a JSONSchema file with the same name as this stream's "name" property. Override as needed.
307 def as_airbyte_stream(self) -> AirbyteStream: 308 stream = AirbyteStream( 309 name=self.name, 310 json_schema=dict(self.get_json_schema()), 311 supported_sync_modes=[SyncMode.full_refresh], 312 is_resumable=self.is_resumable, 313 ) 314 315 if self.namespace: 316 stream.namespace = self.namespace 317 318 # If we can offer incremental we always should. RFR is always less reliable than incremental which uses a real cursor value 319 if self.supports_incremental: 320 stream.source_defined_cursor = self.source_defined_cursor 321 stream.supported_sync_modes.append(SyncMode.incremental) 322 stream.default_cursor_field = self._wrapped_cursor_field() 323 324 keys = Stream._wrapped_primary_key(self.primary_key) 325 if keys and len(keys) > 0: 326 stream.source_defined_primary_key = keys 327 328 return stream
330 @property 331 def supports_incremental(self) -> bool: 332 """ 333 :return: True if this stream supports incrementally reading data 334 """ 335 return len(self._wrapped_cursor_field()) > 0
Returns
True if this stream supports incrementally reading data
337 @property 338 def is_resumable(self) -> bool: 339 """ 340 :return: True if this stream allows the checkpointing of sync progress and can resume from it on subsequent attempts. 341 This differs from supports_incremental because certain kinds of streams like those supporting resumable full refresh 342 can checkpoint progress in between attempts for improved fault tolerance. However, they will start from the beginning 343 on the next sync job. 344 """ 345 if self.supports_incremental: 346 return True 347 if self.has_multiple_slices: 348 # We temporarily gate substream to not support RFR because puts a pretty high burden on connector developers 349 # to structure stream state in a very specific way. We also can't check for issubclass(HttpSubStream) because 350 # not all substreams implement the interface and it would be a circular dependency so we use parent as a surrogate 351 return False 352 elif hasattr(type(self), "state") and getattr(type(self), "state").fset is not None: 353 # Modern case where a stream manages state using getter/setter 354 return True 355 else: 356 # Legacy case where the CDK manages state via the get_updated_state() method. This is determined by checking if 357 # the stream's get_updated_state() differs from the Stream class and therefore has been overridden 358 return type(self).get_updated_state != Stream.get_updated_state
Returns
True if this stream allows the checkpointing of sync progress and can resume from it on subsequent attempts. This differs from supports_incremental because certain kinds of streams like those supporting resumable full refresh can checkpoint progress in between attempts for improved fault tolerance. However, they will start from the beginning on the next sync job.
363 @property 364 def cursor_field(self) -> Union[str, List[str]]: 365 """ 366 Override to return the default cursor field used by this stream e.g: an API entity might always use created_at as the cursor field. 367 :return: The name of the field used as a cursor. If the cursor is nested, return an array consisting of the path to the cursor. 368 """ 369 return []
Override to return the default cursor field used by this stream e.g: an API entity might always use created_at as the cursor field.
Returns
The name of the field used as a cursor. If the cursor is nested, return an array consisting of the path to the cursor.
371 @property 372 def namespace(self) -> Optional[str]: 373 """ 374 Override to return the namespace of this stream, e.g. the Postgres schema which this stream will emit records for. 375 :return: A string containing the name of the namespace. 376 """ 377 return None
Override to return the namespace of this stream, e.g. the Postgres schema which this stream will emit records for.
Returns
A string containing the name of the namespace.
379 @property 380 def source_defined_cursor(self) -> bool: 381 """ 382 Return False if the cursor can be configured by the user. 383 """ 384 return True
Return False if the cursor can be configured by the user.
386 @property 387 def exit_on_rate_limit(self) -> bool: 388 """Exit on rate limit getter, should return bool value. False if the stream will retry endlessly when rate limited.""" 389 return self._exit_on_rate_limit
Exit on rate limit getter, should return bool value. False if the stream will retry endlessly when rate limited.
396 @property 397 @abstractmethod 398 def primary_key(self) -> Optional[Union[str, List[str], List[List[str]]]]: 399 """ 400 :return: string if single primary key, list of strings if composite primary key, list of list of strings if composite primary key consisting of nested fields. 401 If the stream has no primary keys, return None. 402 """
Returns
string if single primary key, list of strings if composite primary key, list of list of strings if composite primary key consisting of nested fields. If the stream has no primary keys, return None.
404 def stream_slices( 405 self, 406 *, 407 sync_mode: SyncMode, 408 cursor_field: Optional[List[str]] = None, 409 stream_state: Optional[Mapping[str, Any]] = None, 410 ) -> Iterable[Optional[Mapping[str, Any]]]: 411 """ 412 Override to define the slices for this stream. See the stream slicing section of the docs for more information. 413 414 :param sync_mode: 415 :param cursor_field: 416 :param stream_state: 417 :return: 418 """ 419 yield StreamSlice(partition={}, cursor_slice={})
Override to define the slices for this stream. See the stream slicing section of the docs for more information.
Parameters
- sync_mode:
- cursor_field:
- stream_state:
Returns
421 @property 422 def state_checkpoint_interval(self) -> Optional[int]: 423 """ 424 Decides how often to checkpoint state (i.e: emit a STATE message). E.g: if this returns a value of 100, then state is persisted after reading 425 100 records, then 200, 300, etc.. A good default value is 1000 although your mileage may vary depending on the underlying data source. 426 427 Checkpointing a stream avoids re-reading records in the case a sync is failed or cancelled. 428 429 return None if state should not be checkpointed e.g: because records returned from the underlying data source are not returned in 430 ascending order with respect to the cursor field. This can happen if the source does not support reading records in ascending order of 431 created_at date (or whatever the cursor is). In those cases, state must only be saved once the full stream has been read. 432 """ 433 return None
Decides how often to checkpoint state (i.e: emit a STATE message). E.g: if this returns a value of 100, then state is persisted after reading 100 records, then 200, 300, etc.. A good default value is 1000 although your mileage may vary depending on the underlying data source.
Checkpointing a stream avoids re-reading records in the case a sync is failed or cancelled.
return None if state should not be checkpointed e.g: because records returned from the underlying data source are not returned in ascending order with respect to the cursor field. This can happen if the source does not support reading records in ascending order of created_at date (or whatever the cursor is). In those cases, state must only be saved once the full stream has been read.
441 def get_updated_state( 442 self, current_stream_state: MutableMapping[str, Any], latest_record: Mapping[str, Any] 443 ) -> MutableMapping[str, Any]: 444 """DEPRECATED. Please use explicit state property instead, see `IncrementalMixin` docs. 445 446 Override to extract state from the latest record. Needed to implement incremental sync. 447 448 Inspects the latest record extracted from the data source and the current state object and return an updated state object. 449 450 For example: if the state object is based on created_at timestamp, and the current state is {'created_at': 10}, and the latest_record is 451 {'name': 'octavia', 'created_at': 20 } then this method would return {'created_at': 20} to indicate state should be updated to this object. 452 453 :param current_stream_state: The stream's current state object 454 :param latest_record: The latest record extracted from the stream 455 :return: An updated state object 456 """ 457 return {}
DEPRECATED. Please use explicit state property instead, see IncrementalMixin docs.
Override to extract state from the latest record. Needed to implement incremental sync.
Inspects the latest record extracted from the data source and the current state object and return an updated state object.
For example: if the state object is based on created_at timestamp, and the current state is {'created_at': 10}, and the latest_record is {'name': 'octavia', 'created_at': 20 } then this method would return {'created_at': 20} to indicate state should be updated to this object.
Parameters
- current_stream_state: The stream's current state object
- latest_record: The latest record extracted from the stream
Returns
An updated state object
459 def get_cursor(self) -> Optional[Cursor]: 460 """ 461 A Cursor is an interface that a stream can implement to manage how its internal state is read and updated while 462 reading records. Historically, Python connectors had no concept of a cursor to manage state. Python streams need 463 to define a cursor implementation and override this method to manage state through a Cursor. 464 """ 465 return self.cursor
A Cursor is an interface that a stream can implement to manage how its internal state is read and updated while reading records. Historically, Python connectors had no concept of a cursor to manage state. Python streams need to define a cursor implementation and override this method to manage state through a Cursor.
591 def log_stream_sync_configuration(self) -> None: 592 """ 593 Logs the configuration of this stream. 594 """ 595 self.logger.debug( 596 f"Syncing stream instance: {self.name}", 597 extra={ 598 "primary_key": self.primary_key, 599 "cursor_field": self.cursor_field, 600 }, 601 )
Logs the configuration of this stream.
663 @property 664 def configured_json_schema(self) -> Optional[Dict[str, Any]]: 665 """ 666 This property is set from the read method. 667 668 :return Optional[Dict]: JSON schema from configured catalog if provided, otherwise None. 669 """ 670 return self._configured_json_schema
This property is set from the read method.
Returns
JSON schema from configured catalog if provided, otherwise None.
52def package_name_from_class(cls: object) -> str: 53 """Find the package name given a class name""" 54 module = inspect.getmodule(cls) 55 if module is not None: 56 return module.__name__.split(".")[0] 57 else: 58 raise ValueError(f"Could not find package name for class {cls}")
Find the package name given a class name
26class AirbyteTracedException(Exception): 27 """ 28 An exception that should be emitted as an AirbyteTraceMessage 29 """ 30 31 def __init__( 32 self, 33 internal_message: Optional[str] = None, 34 message: Optional[str] = None, 35 failure_type: FailureType = FailureType.system_error, 36 exception: Optional[BaseException] = None, 37 stream_descriptor: Optional[StreamDescriptor] = None, 38 ): 39 """ 40 :param internal_message: the internal error that caused the failure 41 :param message: a user-friendly message that indicates the cause of the error 42 :param failure_type: the type of error 43 :param exception: the exception that caused the error, from which the stack trace should be retrieved 44 :param stream_descriptor: describe the stream from which the exception comes from 45 """ 46 self.internal_message = internal_message 47 self.message = message 48 self.failure_type = failure_type 49 self._exception = exception 50 self._stream_descriptor = stream_descriptor 51 super().__init__(internal_message) 52 53 def __str__(self) -> str: 54 """Return the user-facing message, falling back to internal_message.""" 55 if self.message is not None: 56 return self.message 57 if self.internal_message is not None: 58 return self.internal_message 59 return "" 60 61 def as_airbyte_message( 62 self, stream_descriptor: Optional[StreamDescriptor] = None 63 ) -> AirbyteMessage: 64 """ 65 Builds an AirbyteTraceMessage from the exception 66 67 :param stream_descriptor is deprecated, please use the stream_description in `__init__ or `from_exception`. If many 68 stream_descriptors are defined, the one from `as_airbyte_message` will be discarded. 69 """ 70 now_millis = time.time_ns() // 1_000_000 71 72 trace_exc = self._exception or self 73 stack_trace_str = "".join(traceback.TracebackException.from_exception(trace_exc).format()) 74 75 trace_message = AirbyteTraceMessage( 76 type=TraceType.ERROR, 77 emitted_at=now_millis, 78 error=AirbyteErrorTraceMessage( 79 message=self.message 80 or "Something went wrong in the connector. See the logs for more details.", 81 internal_message=self.internal_message, 82 failure_type=self.failure_type, 83 stack_trace=stack_trace_str, 84 stream_descriptor=self._stream_descriptor 85 if self._stream_descriptor is not None 86 else stream_descriptor, 87 ), 88 ) 89 90 return AirbyteMessage(type=MessageType.TRACE, trace=trace_message) 91 92 def as_connection_status_message(self) -> Optional[AirbyteMessage]: 93 """Build a failed CONNECTION_STATUS from a config error, with secrets masked. 94 95 The message can quote config values, as a config validation error does, so it goes 96 through the same secret filter as logs and traces. 97 """ 98 if self.failure_type == FailureType.config_error: 99 return AirbyteMessage( 100 type=MessageType.CONNECTION_STATUS, 101 connectionStatus=AirbyteConnectionStatus( 102 status=Status.FAILED, 103 message=filter_secrets(self.message) if self.message else self.message, 104 ), 105 ) 106 return None 107 108 def emit_message(self) -> None: 109 """ 110 Prints the exception as an AirbyteTraceMessage. 111 Note that this will be called automatically on uncaught exceptions when using the airbyte_cdk entrypoint. 112 """ 113 message = orjson.dumps(AirbyteMessageSerializer.dump(self.as_airbyte_message())).decode() 114 filtered_message = filter_secrets(message) 115 print(filtered_message) 116 117 @classmethod 118 def from_exception( 119 cls, 120 exc: BaseException, 121 stream_descriptor: Optional[StreamDescriptor] = None, 122 *args: Any, 123 **kwargs: Any, 124 ) -> "AirbyteTracedException": 125 """ 126 Helper to create an AirbyteTracedException from an existing exception 127 :param exc: the exception that caused the error 128 :param stream_descriptor: describe the stream from which the exception comes from 129 """ 130 if isinstance(exc, AirbyteTracedException): 131 internal_message = exc.internal_message 132 # Preserve the original user-facing message if the caller didn't provide one 133 if "message" not in kwargs: 134 kwargs["message"] = exc.message 135 else: 136 internal_message = str(exc) 137 return cls( 138 internal_message=internal_message, 139 exception=exc, 140 stream_descriptor=stream_descriptor, 141 *args, 142 **kwargs, 143 ) # type: ignore # ignoring because of args and kwargs 144 145 def as_sanitized_airbyte_message( 146 self, stream_descriptor: Optional[StreamDescriptor] = None 147 ) -> AirbyteMessage: 148 """ 149 Builds an AirbyteTraceMessage from the exception and sanitizes any secrets from the message body 150 151 :param stream_descriptor is deprecated, please use the stream_description in `__init__ or `from_exception`. If many 152 stream_descriptors are defined, the one from `as_sanitized_airbyte_message` will be discarded. 153 """ 154 error_message = self.as_airbyte_message(stream_descriptor=stream_descriptor) 155 if error_message.trace.error.message: # type: ignore[union-attr] # AirbyteMessage with MessageType.TRACE has AirbyteTraceMessage 156 error_message.trace.error.message = filter_secrets( # type: ignore[union-attr] 157 error_message.trace.error.message, # type: ignore[union-attr] 158 ) 159 if error_message.trace.error.internal_message: # type: ignore[union-attr] # AirbyteMessage with MessageType.TRACE has AirbyteTraceMessage 160 error_message.trace.error.internal_message = filter_secrets( # type: ignore[union-attr] # AirbyteMessage with MessageType.TRACE has AirbyteTraceMessage 161 error_message.trace.error.internal_message # type: ignore[union-attr] # AirbyteMessage with MessageType.TRACE has AirbyteTraceMessage 162 ) 163 if error_message.trace.error.stack_trace: # type: ignore[union-attr] # AirbyteMessage with MessageType.TRACE has AirbyteTraceMessage 164 error_message.trace.error.stack_trace = filter_secrets( # type: ignore[union-attr] # AirbyteMessage with MessageType.TRACE has AirbyteTraceMessage 165 error_message.trace.error.stack_trace # type: ignore[union-attr] # AirbyteMessage with MessageType.TRACE has AirbyteTraceMessage 166 ) 167 return error_message
An exception that should be emitted as an AirbyteTraceMessage
31 def __init__( 32 self, 33 internal_message: Optional[str] = None, 34 message: Optional[str] = None, 35 failure_type: FailureType = FailureType.system_error, 36 exception: Optional[BaseException] = None, 37 stream_descriptor: Optional[StreamDescriptor] = None, 38 ): 39 """ 40 :param internal_message: the internal error that caused the failure 41 :param message: a user-friendly message that indicates the cause of the error 42 :param failure_type: the type of error 43 :param exception: the exception that caused the error, from which the stack trace should be retrieved 44 :param stream_descriptor: describe the stream from which the exception comes from 45 """ 46 self.internal_message = internal_message 47 self.message = message 48 self.failure_type = failure_type 49 self._exception = exception 50 self._stream_descriptor = stream_descriptor 51 super().__init__(internal_message)
Parameters
- internal_message: the internal error that caused the failure
- message: a user-friendly message that indicates the cause of the error
- failure_type: the type of error
- exception: the exception that caused the error, from which the stack trace should be retrieved
- stream_descriptor: describe the stream from which the exception comes from
61 def as_airbyte_message( 62 self, stream_descriptor: Optional[StreamDescriptor] = None 63 ) -> AirbyteMessage: 64 """ 65 Builds an AirbyteTraceMessage from the exception 66 67 :param stream_descriptor is deprecated, please use the stream_description in `__init__ or `from_exception`. If many 68 stream_descriptors are defined, the one from `as_airbyte_message` will be discarded. 69 """ 70 now_millis = time.time_ns() // 1_000_000 71 72 trace_exc = self._exception or self 73 stack_trace_str = "".join(traceback.TracebackException.from_exception(trace_exc).format()) 74 75 trace_message = AirbyteTraceMessage( 76 type=TraceType.ERROR, 77 emitted_at=now_millis, 78 error=AirbyteErrorTraceMessage( 79 message=self.message 80 or "Something went wrong in the connector. See the logs for more details.", 81 internal_message=self.internal_message, 82 failure_type=self.failure_type, 83 stack_trace=stack_trace_str, 84 stream_descriptor=self._stream_descriptor 85 if self._stream_descriptor is not None 86 else stream_descriptor, 87 ), 88 ) 89 90 return AirbyteMessage(type=MessageType.TRACE, trace=trace_message)
Builds an AirbyteTraceMessage from the exception
:param stream_descriptor is deprecated, please use the stream_description in __init__ orfrom_exception. If many
stream_descriptors are defined, the one fromas_airbyte_message` will be discarded.
92 def as_connection_status_message(self) -> Optional[AirbyteMessage]: 93 """Build a failed CONNECTION_STATUS from a config error, with secrets masked. 94 95 The message can quote config values, as a config validation error does, so it goes 96 through the same secret filter as logs and traces. 97 """ 98 if self.failure_type == FailureType.config_error: 99 return AirbyteMessage( 100 type=MessageType.CONNECTION_STATUS, 101 connectionStatus=AirbyteConnectionStatus( 102 status=Status.FAILED, 103 message=filter_secrets(self.message) if self.message else self.message, 104 ), 105 ) 106 return None
Build a failed CONNECTION_STATUS from a config error, with secrets masked.
The message can quote config values, as a config validation error does, so it goes through the same secret filter as logs and traces.
108 def emit_message(self) -> None: 109 """ 110 Prints the exception as an AirbyteTraceMessage. 111 Note that this will be called automatically on uncaught exceptions when using the airbyte_cdk entrypoint. 112 """ 113 message = orjson.dumps(AirbyteMessageSerializer.dump(self.as_airbyte_message())).decode() 114 filtered_message = filter_secrets(message) 115 print(filtered_message)
Prints the exception as an AirbyteTraceMessage. Note that this will be called automatically on uncaught exceptions when using the airbyte_cdk entrypoint.
117 @classmethod 118 def from_exception( 119 cls, 120 exc: BaseException, 121 stream_descriptor: Optional[StreamDescriptor] = None, 122 *args: Any, 123 **kwargs: Any, 124 ) -> "AirbyteTracedException": 125 """ 126 Helper to create an AirbyteTracedException from an existing exception 127 :param exc: the exception that caused the error 128 :param stream_descriptor: describe the stream from which the exception comes from 129 """ 130 if isinstance(exc, AirbyteTracedException): 131 internal_message = exc.internal_message 132 # Preserve the original user-facing message if the caller didn't provide one 133 if "message" not in kwargs: 134 kwargs["message"] = exc.message 135 else: 136 internal_message = str(exc) 137 return cls( 138 internal_message=internal_message, 139 exception=exc, 140 stream_descriptor=stream_descriptor, 141 *args, 142 **kwargs, 143 ) # type: ignore # ignoring because of args and kwargs
Helper to create an AirbyteTracedException from an existing exception
Parameters
- exc: the exception that caused the error
- stream_descriptor: describe the stream from which the exception comes from
145 def as_sanitized_airbyte_message( 146 self, stream_descriptor: Optional[StreamDescriptor] = None 147 ) -> AirbyteMessage: 148 """ 149 Builds an AirbyteTraceMessage from the exception and sanitizes any secrets from the message body 150 151 :param stream_descriptor is deprecated, please use the stream_description in `__init__ or `from_exception`. If many 152 stream_descriptors are defined, the one from `as_sanitized_airbyte_message` will be discarded. 153 """ 154 error_message = self.as_airbyte_message(stream_descriptor=stream_descriptor) 155 if error_message.trace.error.message: # type: ignore[union-attr] # AirbyteMessage with MessageType.TRACE has AirbyteTraceMessage 156 error_message.trace.error.message = filter_secrets( # type: ignore[union-attr] 157 error_message.trace.error.message, # type: ignore[union-attr] 158 ) 159 if error_message.trace.error.internal_message: # type: ignore[union-attr] # AirbyteMessage with MessageType.TRACE has AirbyteTraceMessage 160 error_message.trace.error.internal_message = filter_secrets( # type: ignore[union-attr] # AirbyteMessage with MessageType.TRACE has AirbyteTraceMessage 161 error_message.trace.error.internal_message # type: ignore[union-attr] # AirbyteMessage with MessageType.TRACE has AirbyteTraceMessage 162 ) 163 if error_message.trace.error.stack_trace: # type: ignore[union-attr] # AirbyteMessage with MessageType.TRACE has AirbyteTraceMessage 164 error_message.trace.error.stack_trace = filter_secrets( # type: ignore[union-attr] # AirbyteMessage with MessageType.TRACE has AirbyteTraceMessage 165 error_message.trace.error.stack_trace # type: ignore[union-attr] # AirbyteMessage with MessageType.TRACE has AirbyteTraceMessage 166 ) 167 return error_message
Builds an AirbyteTraceMessage from the exception and sanitizes any secrets from the message body
:param stream_descriptor is deprecated, please use the stream_description in __init__ orfrom_exception. If many
stream_descriptors are defined, the one fromas_sanitized_airbyte_message` will be discarded.
11def is_cloud_environment() -> bool: 12 """ 13 Returns True if the connector is running in a cloud environment, False otherwise. 14 15 The function checks the value of the DEPLOYMENT_MODE environment variable which is set by the platform. 16 This function can be used to determine whether stricter security measures should be applied. 17 """ 18 deployment_mode = os.environ.get("DEPLOYMENT_MODE", "") 19 return deployment_mode.casefold() == CLOUD_DEPLOYMENT_MODE
Returns True if the connector is running in a cloud environment, False otherwise.
The function checks the value of the DEPLOYMENT_MODE environment variable which is set by the platform. This function can be used to determine whether stricter security measures should be applied.
210class InternalConfig(BaseModel): 211 KEYWORDS: ClassVar[set[str]] = {"_limit", "_page_size"} 212 limit: int = Field(None, alias="_limit") 213 page_size: int = Field(None, alias="_page_size") 214 215 def dict(self, *args: Any, **kwargs: Any) -> dict[str, Any]: 216 kwargs["by_alias"] = True 217 kwargs["exclude_unset"] = True 218 return super().dict(*args, **kwargs) 219 220 def is_limit_reached(self, records_counter: int) -> bool: 221 """ 222 Check if record count reached limit set by internal config. 223 :param records_counter - number of records already red 224 :return True if limit reached, False otherwise 225 """ 226 if self.limit: 227 if records_counter >= self.limit: 228 return True 229 return False
215 def dict(self, *args: Any, **kwargs: Any) -> dict[str, Any]: 216 kwargs["by_alias"] = True 217 kwargs["exclude_unset"] = True 218 return super().dict(*args, **kwargs)
Generate a dictionary representation of the model, optionally specifying which fields to include or exclude.
220 def is_limit_reached(self, records_counter: int) -> bool: 221 """ 222 Check if record count reached limit set by internal config. 223 :param records_counter - number of records already red 224 :return True if limit reached, False otherwise 225 """ 226 if self.limit: 227 if records_counter >= self.limit: 228 return True 229 return False
Check if record count reached limit set by internal config. :param records_counter - number of records already red :return True if limit reached, False otherwise
136class ResourceSchemaLoader: 137 """JSONSchema loader from package resources""" 138 139 def __init__(self, package_name: str): 140 self.package_name = package_name 141 142 def get_schema(self, name: str) -> dict[str, Any]: 143 """ 144 This method retrieves a JSON schema from the schemas/ folder. 145 146 147 The expected file structure is to have all top-level schemas (corresponding to streams) in the "schemas/" folder, with any shared $refs 148 living inside the "schemas/shared/" folder. For example: 149 150 schemas/shared/<shared_definition>.json 151 schemas/<name>.json # contains a $ref to shared_definition 152 schemas/<name2>.json # contains a $ref to shared_definition 153 """ 154 155 schema_filename = f"schemas/{name}.json" 156 raw_file = pkgutil.get_data(self.package_name, schema_filename) 157 if not raw_file: 158 raise IOError(f"Cannot find file {schema_filename}") 159 try: 160 raw_schema = json.loads(raw_file) 161 except ValueError as err: 162 raise RuntimeError(f"Invalid JSON file format for file {schema_filename}") from err 163 164 return self._resolve_schema_references(raw_schema) 165 166 def _resolve_schema_references(self, raw_schema: dict[str, Any]) -> dict[str, Any]: 167 """ 168 Resolve links to external references and move it to local "definitions" map. 169 170 :param raw_schema jsonschema to lookup for external links. 171 :return JSON serializable object with references without external dependencies. 172 """ 173 174 package = importlib.import_module(self.package_name) 175 if package.__file__: 176 base = os.path.dirname(package.__file__) + "/" 177 else: 178 raise ValueError(f"Package {package} does not have a valid __file__ field") 179 resolved = jsonref.JsonRef.replace_refs( 180 raw_schema, loader=JsonFileLoader(base, "schemas/shared"), base_uri=base 181 ) 182 resolved = resolve_ref_links(resolved) 183 if isinstance(resolved, dict): 184 return resolved 185 else: 186 raise ValueError(f"Expected resolved to be a dict. Got {resolved}")
JSONSchema loader from package resources
142 def get_schema(self, name: str) -> dict[str, Any]: 143 """ 144 This method retrieves a JSON schema from the schemas/ folder. 145 146 147 The expected file structure is to have all top-level schemas (corresponding to streams) in the "schemas/" folder, with any shared $refs 148 living inside the "schemas/shared/" folder. For example: 149 150 schemas/shared/<shared_definition>.json 151 schemas/<name>.json # contains a $ref to shared_definition 152 schemas/<name2>.json # contains a $ref to shared_definition 153 """ 154 155 schema_filename = f"schemas/{name}.json" 156 raw_file = pkgutil.get_data(self.package_name, schema_filename) 157 if not raw_file: 158 raise IOError(f"Cannot find file {schema_filename}") 159 try: 160 raw_schema = json.loads(raw_file) 161 except ValueError as err: 162 raise RuntimeError(f"Invalid JSON file format for file {schema_filename}") from err 163 164 return self._resolve_schema_references(raw_schema)
This method retrieves a JSON schema from the schemas/ folder.
The expected file structure is to have all top-level schemas (corresponding to streams) in the "schemas/" folder, with any shared $refs living inside the "schemas/shared/" folder. For example:
schemas/shared/
189def check_config_against_spec_or_exit( 190 config: Mapping[str, Any], spec: ConnectorSpecification 191) -> None: 192 """ 193 Check config object against spec. In case of spec is invalid, throws 194 an exception with validation error description. 195 196 :param config - config loaded from file specified over command line 197 :param spec - spec object generated by connector 198 """ 199 spec_schema = spec.connectionSpecification 200 try: 201 validate(instance=config, schema=spec_schema) 202 except ValidationError as validation_error: 203 raise AirbyteTracedException( 204 message="Config validation error: " + validation_error.message, 205 internal_message=validation_error.message, 206 failure_type=FailureType.config_error, 207 ) from None # required to prevent logging config secrets from the ValidationError's stacktrace
Check config object against spec. In case of spec is invalid, throws an exception with validation error description.
:param config - config loaded from file specified over command line :param spec - spec object generated by connector
232def split_config(config: Mapping[str, Any]) -> Tuple[dict[str, Any], InternalConfig]: 233 """ 234 Break config map object into 2 instances: first is a dict with user defined 235 configuration and second is internal config that contains private keys for 236 acceptance test configuration. 237 238 :param 239 config - Dict object that has been loaded from config file. 240 241 :return tuple of user defined config dict with filtered out internal 242 parameters and connector acceptance test internal config object. 243 """ 244 main_config = {} 245 internal_config = {} 246 for k, v in config.items(): 247 if k in InternalConfig.KEYWORDS: 248 internal_config[k] = v 249 else: 250 main_config[k] = v 251 return main_config, InternalConfig.parse_obj(internal_config)
Break config map object into 2 instances: first is a dict with user defined configuration and second is internal config that contains private keys for acceptance test configuration.
:param config - Dict object that has been loaded from config file.
:return tuple of user defined config dict with filtered out internal parameters and connector acceptance test internal config object.
49class TransformConfig(Flag): 50 """ 51 TypeTransformer class config. Configs can be combined using bitwise or operator e.g. 52 ``` 53 TransformConfig.DefaultSchemaNormalization | TransformConfig.CustomSchemaNormalization 54 ``` 55 """ 56 57 # No action taken, default behavior. Cannot be combined with any other options. 58 NoTransform = auto() 59 # Applies default type casting with default_convert method which converts 60 # values by applying simple type casting to specified jsonschema type. 61 DefaultSchemaNormalization = auto() 62 # Allow registering custom type transformation callback. Can be combined 63 # with DefaultSchemaNormalization. In this case default type casting would 64 # be applied before custom one. 65 CustomSchemaNormalization = auto()
TypeTransformer class config. Configs can be combined using bitwise or operator e.g.
TransformConfig.DefaultSchemaNormalization | TransformConfig.CustomSchemaNormalization
68class TypeTransformer: 69 """ 70 Class for transforming object before output. 71 """ 72 73 _custom_normalizer: Optional[Callable[[Any, Dict[str, Any]], Any]] = None 74 75 def __init__(self, config: TransformConfig): 76 """ 77 Initialize TypeTransformer instance. 78 :param config Transform config that would be applied to object 79 """ 80 if TransformConfig.NoTransform in config and config != TransformConfig.NoTransform: 81 raise Exception("NoTransform option cannot be combined with other flags.") 82 self._config = config 83 all_validators = { 84 key: self.__get_normalizer(key, orig_validator) 85 for key, orig_validator in Draft7Validator.VALIDATORS.items() 86 # Do not validate field we do not transform for maximum performance. 87 if key in ["type", "array", "$ref", "properties", "items"] 88 } 89 self._normalizer = validators.create( 90 meta_schema=Draft7Validator.META_SCHEMA, validators=all_validators 91 ) 92 93 def registerCustomTransform( 94 self, normalization_callback: Callable[[Any, dict[str, Any]], Any] 95 ) -> Callable[[Any, dict[str, Any]], Any]: 96 """ 97 Register custom normalization callback. 98 :param normalization_callback function to be used for value 99 normalization. Takes original value and part type schema. Should return 100 normalized value. See docs/connector-development/cdk-python/schemas.md 101 for details. 102 :return Same callback, this is useful for using registerCustomTransform function as decorator. 103 """ 104 if TransformConfig.CustomSchemaNormalization not in self._config: 105 raise Exception( 106 "Please set TransformConfig.CustomSchemaNormalization config before registering custom normalizer" 107 ) 108 self._custom_normalizer = normalization_callback 109 return normalization_callback 110 111 def __normalize(self, original_item: Any, subschema: Dict[str, Any]) -> Any: 112 """ 113 Applies different transform function to object's field according to config. 114 :param original_item original value of field. 115 :param subschema part of the jsonschema containing field type/format data. 116 :return Final field value. 117 """ 118 if TransformConfig.DefaultSchemaNormalization in self._config: 119 original_item = self.default_convert(original_item, subschema) 120 121 if self._custom_normalizer: 122 original_item = self._custom_normalizer(original_item, subschema) 123 return original_item 124 125 @staticmethod 126 def default_convert(original_item: Any, subschema: Dict[str, Any]) -> Any: 127 """ 128 Default transform function that is used when TransformConfig.DefaultSchemaNormalization flag set. 129 :param original_item original value of field. 130 :param subschema part of the jsonschema containing field type/format data. 131 :return transformed field value. 132 """ 133 target_type = subschema.get("type", []) 134 if original_item is None and "null" in target_type: 135 return None 136 if isinstance(target_type, list): 137 # jsonschema type could either be a single string or array of type 138 # strings. In case if there is some disambigous and more than one 139 # type (except null) do not do any conversion and return original 140 # value. If type array has one type and null i.e. {"type": 141 # ["integer", "null"]}, convert value to specified type. 142 target_type = [t for t in target_type if t != "null"] 143 if len(target_type) != 1: 144 return original_item 145 target_type = target_type[0] 146 try: 147 if target_type == "string": 148 return str(original_item) 149 elif target_type == "number": 150 return float(original_item) 151 elif target_type == "integer": 152 return int(original_item) 153 elif target_type == "boolean": 154 if isinstance(original_item, str): 155 return _strtobool(original_item) == 1 156 return bool(original_item) 157 elif target_type == "array": 158 item_types = set(subschema.get("items", {}).get("type", set())) 159 if ( 160 item_types.issubset(json_to_python_simple) 161 and type(original_item) in json_to_python_simple.values() 162 ): 163 return [original_item] 164 except (ValueError, TypeError): 165 return original_item 166 return original_item 167 168 def __get_normalizer( 169 self, 170 schema_key: str, 171 original_validator: Callable, # type: ignore[type-arg] 172 ) -> Callable[[Any, Any, Any, dict[str, Any]], Generator[Any, Any, None]]: 173 """ 174 Traverse through object fields using native jsonschema validator and apply normalization function. 175 :param schema_key related json schema key that currently being validated/normalized. 176 :original_validator: native jsonschema validator callback. 177 """ 178 179 def normalizator( 180 validator_instance: Validator, 181 property_value: Any, 182 instance: Any, 183 schema: Dict[str, Any], 184 ) -> Generator[Any, Any, None]: 185 """ 186 Jsonschema validator callable it uses for validating instance. We 187 override default Draft7Validator to perform value transformation 188 before validation take place. We do not take any action except 189 logging warn if object does not conform to json schema, just using 190 jsonschema algorithm to traverse through object fields. 191 Look 192 https://python-jsonschema.readthedocs.io/en/stable/creating/?highlight=validators.create#jsonschema.validators.create 193 validators parameter for detailed description. 194 : 195 """ 196 # Transform object and array values before running json schema type checking for each element. 197 # Recursively normalize every value of the "instance" sub-object, 198 # if "instance" is an incorrect type - skip recursive normalization of "instance" 199 if schema_key == "properties" and isinstance(instance, dict): 200 for k, subschema in property_value.items(): 201 if k in instance: 202 instance[k] = self.__normalize(instance[k], subschema) 203 # Recursively normalize every item of the "instance" sub-array, 204 # if "instance" is an incorrect type - skip recursive normalization of "instance" 205 elif schema_key == "items" and isinstance(instance, list): 206 for index, item in enumerate(instance): 207 instance[index] = self.__normalize(item, property_value) 208 209 # Running native jsonschema traverse algorithm after field normalization is done. 210 yield from original_validator( 211 validator_instance, 212 property_value, 213 instance, 214 schema, 215 ) 216 217 return normalizator 218 219 def transform( 220 self, 221 record: Dict[str, Any], 222 schema: Mapping[str, Any], 223 ) -> None: 224 """ 225 Normalize and validate according to config. 226 :param record: record instance for normalization/transformation. All modification are done by modifying existent object. 227 :param schema: object's jsonschema for normalization. 228 """ 229 if TransformConfig.NoTransform in self._config: 230 return 231 normalizer = self._normalizer(schema) 232 for e in normalizer.iter_errors(record): 233 """ 234 just calling normalizer.validate() would throw an exception on 235 first validation occurrences and stop processing rest of schema. 236 """ 237 logger.warning(self.get_error_message(e)) 238 239 def get_error_message(self, e: ValidationError) -> str: 240 """ 241 Construct a sanitized error message from a ValidationError instance. 242 """ 243 field_path = ".".join(map(str, e.path)) 244 type_structure = self._get_type_structure(e.instance) 245 246 return f"Failed to transform value from type '{type_structure}' to type '{e.validator_value}' at path: '{field_path}'" 247 248 def _get_type_structure(self, input_data: Any, current_depth: int = 0) -> Any: 249 """ 250 Get the structure of a given input data for use in error message construction. 251 """ 252 # Handle null values 253 if input_data is None: 254 return "null" 255 256 # Avoid recursing too deep 257 if current_depth >= MAX_NESTING_DEPTH: 258 return "object" if isinstance(input_data, dict) else python_to_json[type(input_data)] 259 260 if isinstance(input_data, dict): 261 return { 262 key: self._get_type_structure(field_value, current_depth + 1) 263 for key, field_value in input_data.items() 264 } 265 266 else: 267 return python_to_json[type(input_data)]
Class for transforming object before output.
75 def __init__(self, config: TransformConfig): 76 """ 77 Initialize TypeTransformer instance. 78 :param config Transform config that would be applied to object 79 """ 80 if TransformConfig.NoTransform in config and config != TransformConfig.NoTransform: 81 raise Exception("NoTransform option cannot be combined with other flags.") 82 self._config = config 83 all_validators = { 84 key: self.__get_normalizer(key, orig_validator) 85 for key, orig_validator in Draft7Validator.VALIDATORS.items() 86 # Do not validate field we do not transform for maximum performance. 87 if key in ["type", "array", "$ref", "properties", "items"] 88 } 89 self._normalizer = validators.create( 90 meta_schema=Draft7Validator.META_SCHEMA, validators=all_validators 91 )
Initialize TypeTransformer instance. :param config Transform config that would be applied to object
93 def registerCustomTransform( 94 self, normalization_callback: Callable[[Any, dict[str, Any]], Any] 95 ) -> Callable[[Any, dict[str, Any]], Any]: 96 """ 97 Register custom normalization callback. 98 :param normalization_callback function to be used for value 99 normalization. Takes original value and part type schema. Should return 100 normalized value. See docs/connector-development/cdk-python/schemas.md 101 for details. 102 :return Same callback, this is useful for using registerCustomTransform function as decorator. 103 """ 104 if TransformConfig.CustomSchemaNormalization not in self._config: 105 raise Exception( 106 "Please set TransformConfig.CustomSchemaNormalization config before registering custom normalizer" 107 ) 108 self._custom_normalizer = normalization_callback 109 return normalization_callback
Register custom normalization callback. :param normalization_callback function to be used for value normalization. Takes original value and part type schema. Should return normalized value. See docs/connector-development/cdk-python/schemas.md for details. :return Same callback, this is useful for using registerCustomTransform function as decorator.
125 @staticmethod 126 def default_convert(original_item: Any, subschema: Dict[str, Any]) -> Any: 127 """ 128 Default transform function that is used when TransformConfig.DefaultSchemaNormalization flag set. 129 :param original_item original value of field. 130 :param subschema part of the jsonschema containing field type/format data. 131 :return transformed field value. 132 """ 133 target_type = subschema.get("type", []) 134 if original_item is None and "null" in target_type: 135 return None 136 if isinstance(target_type, list): 137 # jsonschema type could either be a single string or array of type 138 # strings. In case if there is some disambigous and more than one 139 # type (except null) do not do any conversion and return original 140 # value. If type array has one type and null i.e. {"type": 141 # ["integer", "null"]}, convert value to specified type. 142 target_type = [t for t in target_type if t != "null"] 143 if len(target_type) != 1: 144 return original_item 145 target_type = target_type[0] 146 try: 147 if target_type == "string": 148 return str(original_item) 149 elif target_type == "number": 150 return float(original_item) 151 elif target_type == "integer": 152 return int(original_item) 153 elif target_type == "boolean": 154 if isinstance(original_item, str): 155 return _strtobool(original_item) == 1 156 return bool(original_item) 157 elif target_type == "array": 158 item_types = set(subschema.get("items", {}).get("type", set())) 159 if ( 160 item_types.issubset(json_to_python_simple) 161 and type(original_item) in json_to_python_simple.values() 162 ): 163 return [original_item] 164 except (ValueError, TypeError): 165 return original_item 166 return original_item
Default transform function that is used when TransformConfig.DefaultSchemaNormalization flag set. :param original_item original value of field. :param subschema part of the jsonschema containing field type/format data. :return transformed field value.
219 def transform( 220 self, 221 record: Dict[str, Any], 222 schema: Mapping[str, Any], 223 ) -> None: 224 """ 225 Normalize and validate according to config. 226 :param record: record instance for normalization/transformation. All modification are done by modifying existent object. 227 :param schema: object's jsonschema for normalization. 228 """ 229 if TransformConfig.NoTransform in self._config: 230 return 231 normalizer = self._normalizer(schema) 232 for e in normalizer.iter_errors(record): 233 """ 234 just calling normalizer.validate() would throw an exception on 235 first validation occurrences and stop processing rest of schema. 236 """ 237 logger.warning(self.get_error_message(e))
Normalize and validate according to config.
Parameters
- record: record instance for normalization/transformation. All modification are done by modifying existent object.
- schema: object's jsonschema for normalization.
239 def get_error_message(self, e: ValidationError) -> str: 240 """ 241 Construct a sanitized error message from a ValidationError instance. 242 """ 243 field_path = ".".join(map(str, e.path)) 244 type_structure = self._get_type_structure(e.instance) 245 246 return f"Failed to transform value from type '{type_structure}' to type '{e.validator_value}' at path: '{field_path}'"
Construct a sanitized error message from a ValidationError instance.
80@contextmanager 81def create_timer(name: str) -> Generator[EventTimer, Any, None]: 82 """ 83 Creates a new EventTimer as a context manager to improve code readability. 84 """ 85 a_timer = EventTimer(name) 86 yield a_timer
Creates a new EventTimer as a context manager to improve code readability.
9class OneOfOptionConfig: 10 """ 11 Base class to configure a Pydantic model that's used as a oneOf option in a parent model in a way that's compatible with all Airbyte consumers. 12 13 Inherit from this class in the nested Config class in a model and set title and description (these show up in the UI) and discriminator (this is making sure it's marked as required in the schema). 14 15 Usage: 16 17 ```python 18 class OptionModel(BaseModel): 19 mode: Literal["option_a"] = Field("option_a", const=True) 20 option_a_field: str = Field(...) 21 22 class Config(OneOfOptionConfig): 23 title = "Option A" 24 description = "Option A description" 25 discriminator = "mode" 26 ``` 27 """ 28 29 @staticmethod 30 def schema_extra(schema: Dict[str, Any], model: Any) -> None: 31 if hasattr(model.Config, "description"): 32 schema["description"] = model.Config.description 33 if hasattr(model.Config, "discriminator"): 34 schema.setdefault("required", []).append(model.Config.discriminator)
Base class to configure a Pydantic model that's used as a oneOf option in a parent model in a way that's compatible with all Airbyte consumers.
Inherit from this class in the nested Config class in a model and set title and description (these show up in the UI) and discriminator (this is making sure it's marked as required in the schema).
Usage:
class OptionModel(BaseModel): mode: Literal["option_a"] = Field("option_a", const=True) option_a_field: str = Field(...) class Config(OneOfOptionConfig): title = "Option A" description = "Option A description" discriminator = "mode"
14def resolve_refs(schema: dict[str, Any]) -> dict[str, Any]: 15 """ 16 For spec schemas generated using Pydantic models, the resulting JSON schema can contain refs between object 17 relationships. 18 """ 19 resource = Resource.from_contents(schema, default_specification=DRAFT7) 20 registry = Registry().with_resource("", resource) 21 resolver = registry.resolver() 22 str_schema = json.dumps(schema) 23 for ref_block in re.findall(r'{"\$ref": "#\/definitions\/.+?(?="})"}', str_schema): 24 ref = json.loads(ref_block)["$ref"] 25 resolved = resolver.lookup(ref).contents 26 str_schema = str_schema.replace(ref_block, json.dumps(resolved)) 27 pyschema: dict[str, Any] = json.loads(str_schema) 28 del pyschema["definitions"] 29 return pyschema
For spec schemas generated using Pydantic models, the resulting JSON schema can contain refs between object relationships.
23def as_airbyte_message( 24 stream: Union[AirbyteStream, StreamDescriptor], 25 current_status: AirbyteStreamStatus, 26 reasons: Optional[List[AirbyteStreamStatusReason]] = None, 27) -> AirbyteMessage: 28 """ 29 Builds an AirbyteStreamStatusTraceMessage for the provided stream 30 """ 31 32 now_millis = datetime.now().timestamp() * 1000.0 33 34 trace_message = AirbyteTraceMessage( 35 type=TraceType.STREAM_STATUS, 36 emitted_at=now_millis, 37 stream_status=AirbyteStreamStatusTraceMessage( 38 stream_descriptor=StreamDescriptor(name=stream.name, namespace=stream.namespace), 39 status=current_status, 40 reasons=reasons, 41 ), 42 ) 43 44 return AirbyteMessage(type=MessageType.TRACE, trace=trace_message)
Builds an AirbyteStreamStatusTraceMessage for the provided stream
22class Record(Mapping[str, Any]): 23 def __init__( 24 self, 25 data: Mapping[str, Any], 26 stream_name: str, 27 associated_slice: Optional[StreamSlice] = None, 28 file_reference: Optional[AirbyteRecordMessageFileReference] = None, 29 ): 30 self._data = data 31 self._associated_slice = associated_slice 32 self.stream_name = stream_name 33 self._file_reference = file_reference 34 35 @property 36 def data(self) -> Mapping[str, Any]: 37 return self._data 38 39 @property 40 def associated_slice(self) -> Optional[StreamSlice]: 41 return self._associated_slice 42 43 @property 44 def file_reference(self) -> AirbyteRecordMessageFileReference: 45 return self._file_reference 46 47 @file_reference.setter 48 def file_reference(self, value: AirbyteRecordMessageFileReference) -> None: 49 self._file_reference = value 50 51 def __repr__(self) -> str: 52 return repr(self._data) 53 54 def __getitem__(self, key: str) -> Any: 55 return self._data[key] 56 57 def __len__(self) -> int: 58 return len(self._data) 59 60 def __iter__(self) -> Any: 61 return iter(self._data) 62 63 def __contains__(self, item: object) -> bool: 64 return item in self._data 65 66 def __eq__(self, other: object) -> bool: 67 if isinstance(other, Record): 68 # noinspection PyProtectedMember 69 return self._data == other._data 70 return False 71 72 def __ne__(self, other: object) -> bool: 73 return not self.__eq__(other)
A Mapping is a generic container for associating key/value pairs.
This class provides concrete generic implementations of all methods except for __getitem__, __iter__, and __len__.
23 def __init__( 24 self, 25 data: Mapping[str, Any], 26 stream_name: str, 27 associated_slice: Optional[StreamSlice] = None, 28 file_reference: Optional[AirbyteRecordMessageFileReference] = None, 29 ): 30 self._data = data 31 self._associated_slice = associated_slice 32 self.stream_name = stream_name 33 self._file_reference = file_reference