airbyte_cdk.sources.streams.http.error_handlers.backoff_strategy

View Source

 1#
 2# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
 3#
 4
 5from abc import ABC, abstractmethod
 6from typing import Optional, Union
 7
 8import requests
 9
10
11class BackoffStrategy(ABC):
12    @abstractmethod
13    def backoff_time(
14        self,
15        response_or_exception: Optional[Union[requests.Response, requests.RequestException]],
16        attempt_count: int,
17    ) -> Optional[float]:
18        """
19        Override this method to dynamically determine backoff time e.g: by reading the X-Retry-After header.
20
21        Not called for every retryable response. `HttpClient` skips the strategies entirely when a
22        rate-limited response can be retried on another credential -- the authenticator says so via
23        `TokenRotatingAuthenticator.has_alternative_token` -- because the wait computed here is
24        derived from the credential that was rejected, and the retry will not use it. Implementations
25        must therefore not rely on being called for side effects such as counting attempts or
26        emitting metrics.
27
28        :param response_or_exception: The response or exception that caused the backoff.
29        :param attempt_count: The number of attempts already performed for this request.
30        :return how long to backoff in seconds. The return value may be a floating point number for subsecond precision. Returning None defers backoff
31        to the default backoff behavior (e.g using an exponential algorithm).
32        """
33        pass

class BackoffStrategy(abc.ABC): View Source

12class BackoffStrategy(ABC):
13    @abstractmethod
14    def backoff_time(
15        self,
16        response_or_exception: Optional[Union[requests.Response, requests.RequestException]],
17        attempt_count: int,
18    ) -> Optional[float]:
19        """
20        Override this method to dynamically determine backoff time e.g: by reading the X-Retry-After header.
21
22        Not called for every retryable response. `HttpClient` skips the strategies entirely when a
23        rate-limited response can be retried on another credential -- the authenticator says so via
24        `TokenRotatingAuthenticator.has_alternative_token` -- because the wait computed here is
25        derived from the credential that was rejected, and the retry will not use it. Implementations
26        must therefore not rely on being called for side effects such as counting attempts or
27        emitting metrics.
28
29        :param response_or_exception: The response or exception that caused the backoff.
30        :param attempt_count: The number of attempts already performed for this request.
31        :return how long to backoff in seconds. The return value may be a floating point number for subsecond precision. Returning None defers backoff
32        to the default backoff behavior (e.g using an exponential algorithm).
33        """
34        pass

Helper class that provides a standard way to create an ABC using inheritance.

@abstractmethod

def backoff_time( self, response_or_exception: Union[requests.models.Response, requests.exceptions.RequestException, NoneType], attempt_count: int) -> Optional[float]: View Source

13    @abstractmethod
14    def backoff_time(
15        self,
16        response_or_exception: Optional[Union[requests.Response, requests.RequestException]],
17        attempt_count: int,
18    ) -> Optional[float]:
19        """
20        Override this method to dynamically determine backoff time e.g: by reading the X-Retry-After header.
21
22        Not called for every retryable response. `HttpClient` skips the strategies entirely when a
23        rate-limited response can be retried on another credential -- the authenticator says so via
24        `TokenRotatingAuthenticator.has_alternative_token` -- because the wait computed here is
25        derived from the credential that was rejected, and the retry will not use it. Implementations
26        must therefore not rely on being called for side effects such as counting attempts or
27        emitting metrics.
28
29        :param response_or_exception: The response or exception that caused the backoff.
30        :param attempt_count: The number of attempts already performed for this request.
31        :return how long to backoff in seconds. The return value may be a floating point number for subsecond precision. Returning None defers backoff
32        to the default backoff behavior (e.g using an exponential algorithm).
33        """
34        pass

Override this method to dynamically determine backoff time e.g: by reading the X-Retry-After header.

Not called for every retryable response. HttpClient skips the strategies entirely when a rate-limited response can be retried on another credential -- the authenticator says so via TokenRotatingAuthenticator.has_alternative_token -- because the wait computed here is derived from the credential that was rejected, and the retry will not use it. Implementations must therefore not rely on being called for side effects such as counting attempts or emitting metrics.

Parameters

response_or_exception: The response or exception that caused the backoff.
attempt_count: The number of attempts already performed for this request. :return how long to backoff in seconds. The return value may be a floating point number for subsecond precision. Returning None defers backoff to the default backoff behavior (e.g using an exponential algorithm).