airbyte.documents
This module contains the Documents class for converting Airbyte records into documents.
Generally you will not create Documents objects directly. Instead, you can use one of the
following methods to generate documents from records:
Source.get_documents(): Get an iterable of documents from a source.Dataset.to_documents(): Get an iterable of documents from a dataset.
1# Copyright (c) 2024 Airbyte, Inc., all rights reserved. 2"""This module contains the `Documents` class for converting Airbyte records into documents. 3 4Generally you will not create `Documents` objects directly. Instead, you can use one of the 5following methods to generate documents from records: 6 7- `Source.get_documents()`: Get an iterable of documents from a source. 8- `Dataset.to_documents()`: Get an iterable of documents from a dataset. 9""" 10 11from __future__ import annotations 12 13import datetime # noqa: TC003 # Pydantic resolves this annotation at runtime. 14from typing import Any 15 16from pydantic import BaseModel, Field 17 18 19MAX_SINGLE_LINE_LENGTH = 60 20AIRBYTE_DOCUMENT_RENDERING = "airbyte_document_rendering" 21TITLE_PROPERTY = "title_property" 22CONTENT_PROPS = "content_properties" 23METADATA_PROPERTIES = "metadata_properties" 24 25 26class Document(BaseModel): 27 """A PyAirbyte document is a specific projection on top of a record. 28 29 Documents have the following structure: 30 - id (str): A unique string identifier for the document. 31 - content (str): A string representing the record when rendered as a document. 32 - metadata (dict[str, Any]): Associated metadata about the document, such as the record's IDs 33 and/or URLs. 34 35 This class is duck-typed to be compatible with LangChain project's `Document` class. 36 """ 37 38 id: str | None = Field(default=None) 39 content: str 40 metadata: dict[str, Any] 41 last_modified: datetime.datetime | None = Field(default=None) 42 43 def __str__(self) -> str: 44 """Return a string representation of the document.""" 45 return self.content 46 47 @property 48 def page_content(self) -> str: 49 """Return the content of the document. 50 51 This is an alias for the `content` property, and is provided for duck-type compatibility 52 with the LangChain project's `Document` class. 53 """ 54 return self.content 55 56 57__all__ = [ 58 "Document", 59]
class
Document(pydantic.main.BaseModel):
27class Document(BaseModel): 28 """A PyAirbyte document is a specific projection on top of a record. 29 30 Documents have the following structure: 31 - id (str): A unique string identifier for the document. 32 - content (str): A string representing the record when rendered as a document. 33 - metadata (dict[str, Any]): Associated metadata about the document, such as the record's IDs 34 and/or URLs. 35 36 This class is duck-typed to be compatible with LangChain project's `Document` class. 37 """ 38 39 id: str | None = Field(default=None) 40 content: str 41 metadata: dict[str, Any] 42 last_modified: datetime.datetime | None = Field(default=None) 43 44 def __str__(self) -> str: 45 """Return a string representation of the document.""" 46 return self.content 47 48 @property 49 def page_content(self) -> str: 50 """Return the content of the document. 51 52 This is an alias for the `content` property, and is provided for duck-type compatibility 53 with the LangChain project's `Document` class. 54 """ 55 return self.content
A PyAirbyte document is a specific projection on top of a record.
Documents have the following structure:
- id (str): A unique string identifier for the document.
- content (str): A string representing the record when rendered as a document.
- metadata (dict[str, Any]): Associated metadata about the document, such as the record's IDs and/or URLs.
This class is duck-typed to be compatible with LangChain project's Document class.