4.0.4 Release Notes

[yiguolei]

What's Changed

Feature

• Enable system table all_manifests for Iceberg tables ([fix](iceberg) Enable system table all_manifests for iceberg #60279)

• Support bucket-domain-name for OSS ([feat](oss)Support bucket-domain-name #59755)

• Support flexible partial update for routine load (Support flexible partial update for routine load #60128)

• Implement expire_snapshots procedure for Iceberg tables ([Feature](Iceberg) Implement expire_snapshots procedure for Iceberg tables #59979)

• Support PREVIOUS_DAY function ([Feature](func) Support function PREVIOUS_DAY #60680)

• Support 3 spatial functions: ST_Distance, ST_GeometryType, ST_Length ([feature](geo) support 3 spatial functions: ST_Distance, ST_GeometryType, ST_Length #60170)

• Support more INTERVAL time-unit ([Feature](function) Support more INTERVAL time-unit #60347)

• Add OzoneProperties to support Apache Ozone ([feature](storage) Add OzoneProperties to support Apache Ozone #60809)

• Support MaxCompute ram_role_arn and ecs_ram_role ([feature](maxcompute)support maxcompute ram_role_arn and ecs_ram_role authenticate. #60649)

• Support schema change for complex types in Iceberg external tables ([Feature](iceberg) Support schema change for complex types in Iceberg external tables #60169)

• Support score range filter pushdown (min_score semantics) (branch-4.0: [feature](bm25) support score range filter pushdown (min_score semantics) #60997)

• Support multiple tokenize index in one column ([branch-4.0](inverted index) support multiple tokenize index in one column #60415)

• Add x509 cert based auth framework ([secrity](auth) Add x509 cert based auth framework #60098)

• Support dfs.client.use.datanode.hostname for libhdfs3 ([feat](libhdfs3) support dfs.client.use.datanode.hostname for libhdfs3 #59915)

Improvement

• Improve password validation to align with MySQL STRONG policy ([Enhancement](auth) Improve password validation to align with MySQL STRONG policy #60188)

• Optimize the performance of parquet dict decoder ([opt](parquet) opt the performance of dict decoder #59681)

• Supports dynamically changing the rate limiter config ([Enhancement](client) Supports dynamically changing the rate limiter config #59465)

• Split metadata scan ranges per split ([enhance](multi-catalog) Split metadata scan ranges per split #60257)

• Optimize Iceberg rewrite_data_files to avoid generating excessive small files ([opt](iceberg) optimize rewrite_data_files to avoid generating excessive small files #60063)

• Use COUNT_ON_INDEX on variant subcolumns ([Improve](Variant) use COUNT_ON_INDEX on variant subcolumns #60404)

• Optimize file split size for multi-catalog (branch-4.0: [opt](multi-catalog) Optimize file split size.  #60637)

• Limit the download rate of warmup task ([Opt](warmup) Limit the download rate of warmup task #60180)

• Optimize performance of certain time field functions used with FROM_UNIXTIME (branch-4.0: [Opt](function) opt of certain time field functions used in conjunction with FROM_UNIXTIME. #60843)

• Add fold constant for PREVIOUS_DAY ([Enhancement](func) Add fold constant for PREVIOUS_DAY #60755)

• Support PostgreSQL partition table sync for streaming job ([Improve](streaming job) support postgres partition table sync #60560)

Bugfix

• Fix crash caused by concurrent compaction accessing shared sample_infos ([fix](compaction) Fix crash caused by concurrent compaction accessing shared sample_infos #60376)

• Fix unnecessary conflict range in lazy commit ([fix](cloud) fix unnecessary conflict range in lazy commit #60274)

• Fix routine load task schedule stuck after create task fail ([fix](job) fix routine load task schedule stuck after create task fail #60143)

• Fix DELETE/UPDATE cannot resolve column when table alias uses AS keyword ([fix](parser) fix DELETE/UPDATE cannot resolve column when table alias uses AS keyword  #60335)

• Fix boolean query AllScorer combination handling (branch-4.0: [fix](inverted-index) Fix boolean query AllScorer combination handling #60438)

• Fix query cache not hit when use sort and one phase aggregation ([fix](query cache) fix query cache not hit when use sort and one phase aggregation #60298)

• Modify ORC reader to make error reporting more accurate ([fix](orc)Modifying the orc reader makes error reporting more accurate. #60234)

• Fix prune nested column maybe throw NullPointerException ([fix](prune nested column) fix prune nested column maybe throw NullPointerException #60395)

• Fix partitions function throw error when query internal table without specifying catalog condition ([Fix](function) fix partitions function throw error when query internal table without specifying catalog condition, this incompatible with show partitions command #60247)

• Fix backends udf return rows incompatible with show backends command ([Fix](udf) fix backends udf return rows incompatible with show backends command #60210)

• Fix AggregateNode compute digest of query cache should consider sortByGroupKey ([fix](query cache) AggregateNode compute digest of query cache should consider sortByGroupKey #60431)

• Fix predicate should be pushed down when conjunct contains CAST for Variant (branch4.0 [Fix](Variant) predicate should be pushed down when conjunct contains CAST #60485)

• Treat no such key as empty response when listing objects (part 2) for S3 ([fix](s3) Treat no such key as empty response when listing objects (part 2) #60286)

• Fix empty string MATCH on keyword index returning wrong results ([fix](inverted index) Fix empty string MATCH on keyword index returning wrong results #60500)

• Fix PhysicalDictionarySink.resetLogicalProperties() does not reset logical properties properly ([fix](nereids) PhysicalDictionarySink.resetLogicalProperties() does not reset logical properties properly #60495)

• Fix %f (microseconds) format specifier in str_to_date ([Fix](datetime) fix %f (microseconds) format specifier in str_to_date #60632)

• Throw exception for date_floor/ceil instead of return NULL ([fix](datetime) throw exception for date_floor/ceil instead of return NULL #60633)

• Fix S3-compatible storage must support temporary credentials when vendor credentials are configured at catalog level ([Fix](RestCatalog)When vendor credentials are configured at the catalog level, S3-compatible storage must also support temporary credentials #60232)

• Fix empty InsertIntoDictionaryCommand originSql ([Fix](dictionary) Fix empty InsertIntoDictionaryCommand originSql #60631)

• Fix width_bucket did not enforce that the fourth argument must be a constant ([fix](function)width_bucket did not enforce that the fourth argument must be a constant. #60643)

• Validate mode parameter in search() DSL options ([fix](search) Validate mode parameter in search() DSL options #60785)

• Fix range search prepare failed on NULL literal for ANN ([fix](ann range search) range search prepare failed on NULL literal #60564)

• Make dot match newline in regexp_fn by default ([Fix](regexp) make dot match newline in regexp_fn by default #60831)

• Fix Azure Storage Vault endpoint always using HTTP instead of HTTPS ([fix](storage vault) fix Azure Storage Vault endpoint always using HTTP instead of HTTPS #60854)

• Add rest external catalog for gson compatibility for Paimon ([fix](paimon) add rest external catalog for gson comaptiblity #60917)

• Disable strict mode for variant internal cast to fix INSERT INTO SELECT returning all NULLs (cherry-pick branch4.0 : [fix](variant) Disable strict mode for variant internal cast to fix INSERT INTO SELECT returning all NULLs  #60900)

• Default to the Default chain when S3 role_arn is set without provider_type ([fix](be) Default to the Default chain when S3 role_arn is set without provider_type #60822)

• Fix parquet reader lazy materialization cannot filter ([fix](parquet)fix parquet reader lazy materialization cannot filter. #60474)

• Fix point query ignoring session timezone for functions like from_unixtime ([fix](point-query) Fix point query ignoring session timezone for functions like from_unixtime #60913)

• Make AND/OR/NOT operators case-sensitive in search DSL ([fix](search) Make AND/OR/NOT operators case-sensitive in search DSL #59747)

• Decouple min pipeline executor size from ConnectContext ([Fix](Cloud)decouple min pipeline executor size from ConnectContext #60958)

• Fix Azure endpoint detection for sovereign clouds with force-global config support ([fix](Azure)Fix Azure endpoint detection for sovereign clouds with force-global config support #60903)

• Backport search() function improvements and bug fixes (branch-4.0: [fix](search) Backport search() function improvements and bug fixes #61028)

• Fix encrypt key is not case sensitive ([fix](encrypt)encrypt key is not case sensitive #60288)

• Make hive compress split assertion BE-aware ([fix](test) make hive compress split assertion BE-aware #60947)

• Support CLIENT_DEPRECATE_EOF to fix empty result with MySQL driver 9.5.0 ([fix](protocol) Support CLIENT_DEPRECATE_EOF to fix empty result with MySQL driver 9.5.0 #61050)

• Fix basic and icu tokenizer can not be customized in custom analyzer ([fix](custom analyzer) Fix basic and icu tokenizer can not be customized #60506)