Skip to content

fix: s3 table upload location - #1376

Merged
jczhong84 merged 9 commits into
pinterest:masterfrom
jczhong84:jc/tableupload
Nov 29, 2023
Merged

jczhong84 merged 9 commits into
pinterest:masterfrom
jczhong84:jc/tableupload

Conversation

@jczhong84

Copy link
Copy Markdown
Collaborator
  • Explicitly call out that the table is managed when use_schema_location=False for s3 table uploader.
  • Get the actual table location from metastore for data uploading for managed tables
  • Fix the logic of checking is_external. Previously is_external will be False when use_schema_location is True, which supposed to be True (external table), as it is specifying location.
    is_external = not self._exporter_config.get("use_schema_location", False)

@jczhong84
jczhong84 requested a review from czgu November 28, 2023 06:27
Comment thread docs_website/docs/integrations/add_table_upload.md
Comment thread querybook/server/lib/metastore/loaders/hive_metastore_loader.py Outdated
Comment thread querybook/server/lib/table_upload/exporter/s3_exporter.py Outdated
Comment thread querybook/server/lib/table_upload/exporter/s3_exporter.py Outdated
Comment thread querybook/server/lib/table_upload/exporter/s3_exporter.py Outdated
Comment thread querybook/server/lib/table_upload/exporter/s3_exporter.py Outdated
Comment thread querybook/webapp/components/TableUploader/TableUploaderForm.tsx

Made below changes for `S3BaseExporter` (csv table uploader feature):

- Both `s3_path` and `use_schema_location` are optional now

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you need to mention if use_schema_location=True before 3.29.0, it would create managed table, now it creates external

czgu
czgu previously approved these changes Nov 29, 2023
@jczhong84
jczhong84 merged commit 0b5699a into pinterest:master Nov 29, 2023
aidenprice pushed a commit to arrowtail-precision/querybook that referenced this pull request Jan 3, 2024
* fix: s3 table upload location

* fix linter

* sanitize s3 path

* disable redirect

* fix linter

* add break change & bump version

* typing

* remove `

* breaking change
@jczhong84
jczhong84 deleted the jc/tableupload branch March 6, 2025 07:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants