feat(storage): Implement storage capabilities and pipelines for sync and async drivers #231
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…and async drivers - Added `select_to_storage` method to `AdbcDriver`, `AsyncpgDriver`, and `DuckDBDriver` for streaming query results to storage. - Introduced `StorageDriverMixin` to provide common storage functionality across sync and async drivers. - Enhanced `DatabaseConfigProtocol` to include storage capabilities and caching mechanisms. - Created `StoragePipeline` classes for handling storage operations, including writing rows and Arrow tables. - Added telemetry support for storage operations to track metrics such as bytes processed and duration. - Implemented error handling for storage capabilities with `StorageCapabilityError`. - Added unit tests for storage capabilities in configuration.
…dapters - Added support for native Arrow and Parquet export in Aiosqlite, Asyncmy, Asyncpg, BigQuery, DuckDB, OracleDB, Psqlpy, and Psycopg configurations. - Implemented `select_to_storage` method in Aiosqlite, Asyncmy, Asyncpg, BigQuery, DuckDB, OracleDB, Psqlpy, and Psycopg drivers to facilitate streaming Arrow results to storage. - Introduced `load_from_arrow` and `load_from_storage` methods in DuckDB driver for loading Arrow data and reading from storage. - Enhanced storage pipeline with methods for reading Arrow data from storage synchronously and asynchronously. - Updated storage mixin to include Arrow table coercion and telemetry building for ingest operations.
…ed type handling and error management
- Simplified method calls in PsycopgSyncDriver and PsycopgAsyncDriver for better readability. - Added support for native Arrow import and export in SqliteConfig. - Implemented load_from_arrow and load_from_storage methods in SqliteDriver to handle Arrow data ingestion. - Introduced telemetry enhancements to track source and backend information during storage operations. - Added integration tests for AioSQLite, AsyncMy, and SQLite adapters to validate storage bridge functionality. - Improved diagnostics for storage bridge metrics and serializer cache. - Cleaned up redundant code in various driver classes and storage pipeline methods.
…ration tests for various adapters
…ridge tests refactor(tests): Simplify connection casting in SQLite driver tests
… separate function
…nd enhance telemetry output
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
select_to_storagemethod to to all backendsStorageDriverMixinto provide common storage functionality across sync and async drivers.DatabaseConfigProtocolto include storage capabilities and caching mechanisms.StoragePipelineclasses for handling storage operations, including writing rows and Arrow tables.StorageCapabilityError.