Skip to main content

Low-code connector development

Airbyte’s low-code framework enables you to build source connectors for REST APIs by modifying boilerplate YAML files.

danger

The low-code framework is in alpha, which means it’s still in active development and may include backward-incompatible changes. Share feedback and requests with us on our Slack channel or email us at feedback@airbyte.io

What connectors can I build using the low-code framework?

Refer to the REST API documentation for the source you want to build the connector for and answer the following questions:

If the answer to all questions is yes, you can use the low-code framework to build a connector for the source. If not, use the Python CDK.

Prerequisites

  • An API key for the source you want to build a connector for
  • Python >= 3.9
  • Docker
  • NodeJS

Overview of the process

To use the low-code framework to build an REST API Source connector:

  1. Generate the API key for the source you want to build a connector for
  2. Set up the project on your local machine
  3. Set up your local development environment
  4. Update the connector spec and config​uration
  5. Update the connector definition
  6. Test the connector
  7. Add the connector to the Airbyte platform

For a step-by-step tutorial, refer to the Getting Started tutorial or the video tutorial

Configuring the YAML file

The low-code framework involves editing a boilerplate YAML file. The general structure of the YAML file is as follows:

version: "0.1.0"
definitions:
<key-value pairs defining objects which will be reused in the YAML connector>
streams:
<list stream definitions>
check:
<definition of connection checker>

The following table describes the components of the YAML file:

ComponentDescription
versionIndicates the framework version
definitionsDescribes the objects to be reused in the YAML connector
streamsLists the streams of the source
checkDescribes how to test the connection to the source by trying to read a record from a specified list of streams and failing if no records could be read
tip

Streams define the schema of the data to sync, as well as how to read it from the underlying API source. A stream generally corresponds to a resource within the API. They are analogous to tables for a relational database source.

For each stream, configure the following components:

ComponentSub-componentDescription
NameName of the stream
Primary key (Optional)Used to uniquely identify records, enabling deduplication. Can be a string for single primary keys, a list of strings for composite primary keys, or a list of list of strings for composite primary keys consisting of nested fields
SchemaDescribes the data to sync
Data retrieverDescribes how to retrieve data from the API
RequesterDescribes how to prepare HTTP requests to send to the source API and defines the base URL and path, the request options provider, the HTTP method, authenticator, error handler components
PaginationDescribes how to navigate through the API's pages
Record SelectorDescribes how to extract records from a HTTP response
Stream SlicerDescribes how to partition the stream, enabling incremental syncs and checkpointing
Cursor fieldField to use as stream cursor. Can either be a string, or a list of strings if the cursor is a nested field.
TransformationsA set of transformations to be applied on the records read from the source before emitting them to the destination
Checkpoint intervalDefines the interval, in number of records, at which incremental syncs should be checkpointed

For a deep dive into each of the components, refer to Understanding the YAML file or the full YAML Schema definition

Tutorial

This section a tutorial that will guide you through the end-to-end process of implementing a low-code connector.

  1. Getting started
  2. Creating a source
  3. Installing dependencies
  4. Connecting to the API
  5. Reading data
  6. Incremental reads
  7. Testing

Sample connectors

For examples of production-ready config-based connectors, refer to: