feat: add COPY support for GCS/Azblob with OSS/GCS/Azblob integration coverage

[WenyXu]

I hereby agree to the terms of the GreptimeDB CLA.

Refer to a related PR or issue link (optional)

What's changed and what's your intention?

This PR completes object storage support parity for COPY workflows and improves CLI ergonomics:

• Add gcs:// and azblob:// backend support in COPY datasource path.
• Add integration tests for:
  • COPY TO/FROM oss://
  • COPY TO/FROM gcs://
  • COPY TO/FROM azblob://
• Refactor datasource object-store layering to reuse existing shared layer utilities.
• Hide GCS credential_path from CLI user-facing options, while keeping internal compatibility.

PR Checklist

Please convert it to a draft if some of the following conditions are not met.

• I have written the necessary rustdoc comments.
• I have added the necessary unit tests and integration tests.
• This PR requires documentation updates.
• API changes are backward compatible.
• Schema or data changes are backward compatible.

[WenyXu]

@codex review

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly expands the data import/export capabilities by introducing native support for Google Cloud Storage and Azure Blob Storage within the COPY command. It ensures full parity for object storage workflows, allowing users to seamlessly move data to and from these popular cloud providers. The changes also include a refactoring of the underlying object storage integration, centralizing common utilities and enhancing the overall robustness and user experience of the CLI.

Highlights

• Object Storage Support: Added support for Google Cloud Storage (GCS) and Azure Blob Storage (Azblob) as backends for COPY operations, enhancing cloud storage integration.
• Integration Tests: Implemented comprehensive integration tests for COPY TO/FROM operations across OSS, GCS, and Azblob to ensure robust functionality.
• Code Refactoring: Refactored the object-store layering to reuse existing shared utility functions, improving code maintainability and consistency.
• CLI Ergonomics: Improved CLI ergonomics by hiding the GCS credential_path option from user-facing interfaces while maintaining internal compatibility.

Changelog

• src/cli/src/common/object_store.rs
  • Modified the wrap_with_clap_prefix macro to support a #[hide] attribute for CLI options.
  • Applied the #[hide = true] attribute to the credential_path field within the GCS object store configuration, making it non-user-facing.
• src/cli/src/data/export.rs
  • Removed gcs-credential-path arguments from GCS-related test commands, aligning with the hidden CLI option.
• src/cli/src/data/storage_export.rs
  • Removed the logic for handling and exposing gcs_credential_path when building GCS connection options and masking sensitive information.
• src/common/datasource/src/object_store.rs
  • Added new modules for azblob and gcs object storage backends.
  • Defined new schema constants GCS_SCHEMA and AZBLOB_SCHEMA.
  • Integrated build_gcs_backend and build_azblob_backend into the build_backend function, enabling dynamic backend creation for GCS and Azure Blob.
• src/common/datasource/src/object_store/azblob.rs
  • Added a new module to implement the Azure Blob storage backend, including functions to check supported keys and build the ObjectStore.
• src/common/datasource/src/object_store/fs.rs
  • Refactored the build_fs_backend function to utilize the shared with_instrument_layers utility for consistent logging, tracing, and metrics.
• src/common/datasource/src/object_store/gcs.rs
  • Added a new module to implement the Google Cloud Storage backend, including functions to check supported keys and build the ObjectStore.
• src/common/datasource/src/object_store/oss.rs
  • Refactored the build_oss_backend function to utilize shared with_instrument_layers and with_retry_layers utilities, removing redundant layer definitions.
• src/common/datasource/src/object_store/s3.rs
  • Refactored the build_s3_backend function to utilize shared with_instrument_layers and with_retry_layers utilities, removing redundant layer definitions.
• tests-integration/src/tests/instance_test.rs
  • Added new integration tests for COPY TO and COPY FROM operations using OSS (Aliyun Object Storage Service).
  • Added new integration tests for COPY TO and COPY FROM operations using GCS (Google Cloud Storage).
  • Added new integration tests for COPY TO and COPY FROM operations using Azblob (Azure Blob Storage).

Activity

• No human activity detected in the pull request yet.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7f4eeb4e5b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[gemini-code-assist]

Code Review

This pull request adds support for GCS and Azblob to the COPY command, along with integration tests. The changes look good overall. The object store backend creation has been nicely refactored to reuse common layering logic. I've added a few suggestions in the integration tests to reduce code duplication and improve how connection strings are constructed, which should enhance maintainability.

_{Note: Security Review is unavailable for this PR.}

[fengjiachun]

Others LGTM