4.0.1 Release Notes

[yiguolei]

Behavior Changes

• The SHOW PARTITIONS command no longer supports Iceberg tables. Please directly use Iceberg's $partitions system table to view partitions. [opt](show-partitions) remove show partitions support for iceberg #56985

New Features

• Added the mmh64_v2 function to generate hash results consistent with those from other third-party libraries. [Feature](func) Support function mmh64_v2 #57180
• Added the json_hash function to generate hash values for jsonb types. [feature](function) support json_hash function  #56962
• Added the binary data type, along with a series of related functions: length, from_base64_binary, to_base64_binary, and sub_binary. [Feature](varbinary) add length, from_base64_binary, to_base64_bianry, sub_binary for VarBinary type #56648
• Added the sort_json_object_keys/normalize_json_numbers_to_double function for sorting the keys of jsonb.
• Added several MySQL-compatible time functions: UTC_DATE, UTC_TIME, and UTC_TIMESTAMP. branch-4.0: [Feature](func) Support function UTC_DATE & UTC_TIME, and UTC_TIMESTAMP with precision #57347 #57443
• Added support for MaxCompute Schema hierarchy. https://github.com/apache/doris/pull/56874Documentation: https://doris.apache.org/docs/3.x/lakehouse/catalogs/maxcompute-catalog#hierarchical-mapping
• The JSON_OBJECT function now supports using * as a parameter. [feature](analyzer) support * as json_object's parameter #57256

Improvement

AI & Search

• Added support for phrase queries, wildcard queries, and regular expression queries to the SEARCH function. [feature](search) add phrase/wildcard/regex support for search and refact some code #57372 [feature](inverted index) query_v2 add regexp, wildcard, phrase query #57007
• Extended the SEARCH function with 2 new parameters: an optional default_field parameter (default column) and a default_operator parameter (specifying the boolean operator for multi-column queries as "and" or "or"). [feature](search) add another two params for search #57312
• The SEARCH function now supports searching sub-columns of the variant type, allowing direct search of specific fields in JSON paths via dot notation (e.g., variantColumn.subcolumn:keyword).
• Upgraded the default storage format of the inverted index from V2 to V3. [opt](inverted index) default index format uses v3 #57140
• Enhanced support for custom tokenizer pipelines by adding the char_filter component; introduced two built-in tokenizers (basic tokenizer and ICU tokenizer) in the analyzer framework; added aliases for built-in tokenizers and supported component configuration with the same name to optimize the unified analyzer framework. [opt](inverted index) add custom analyzer support with char_filter, basic and icu tokenizer #56243 [opt](inverted index) Add built_in_analyzer parser alias and modify custom analyzer logic #57055

Lakehouse

• Added the session variable merge_io_read_slice_size_bytes to address the issue of severe read amplification in external table Merge IO under certain conditions.Documentation: https://doris.apache.org/docs/3.x/lakehouse/best-practices/optimization#merge-io-optimization

Query

• Optimized the JOIN shuffle selection algorithm. [opt](nereids) optimize bs downgrade alg. #56279

Others

• Optimized the size of Runtime Filter serialization information in physical plans. [fix](nereids)set RuntimeFilterInfo only on BE which is merge node #57108 [opt](nereids)move runtime filter info from instance level to BE level #56978

Bugfix

AI & Search

• Fixed the issue with search query results for non-tokenized fields and enabled the execution of the search function on MOW tables. [fix](search) fix search result for non-tokenizer match #56914 [fix](search) fix mow support for search function #56927
• Fixed the calculation error in the inverted index when filtering with the IS NULL predicate. [fix](inverted index) fix is null predicate for inverted index evaluate #56964

Lakehouse

• Fixed the issue where predicate pushdown could not use the Parquet Page Index under certain conditions. [External](parquet) pass non predicates column's offset index to RowGroupReader #55795
• Fixed the issue of missing shard reads in external table queries under certain conditions. [fix](parquet) incorrect skip files when returning eof #57071
• Fixed the issue where modifying Catalog properties did not take effect when the Hadoop file system cache was enabled under certain conditions. [fix](catalog)Disable Hadoop FileSystem cache for multi-configurationobject storage catalogs #57063
• Fixed the issue of metadata replay failure due to connection property validation during upgrades from older versions under certain conditions. [Fix](Repository)Handle non-StorageException errors in Storage parameter validation #56929
• Fixed the issue of FE thread deadlock caused by Refresh Catalog under certain conditions. [fix](external catalog) Distinguish between functions used to refresh cache and change catalog properties #56639
• Fixed the issue of being unable to read Iceberg tables converted from Hive. [fix](iceberg) support reading iceberg partition table without partition value stored in file #56918
• Fixed the issue of BE crashes caused by collecting Query Profiles under certain conditions. [fix](hdfs)Fix be coredump in HDFS reader during profile collection. #56806

Query

• Fixed the incorrect result of datetime type during timezone-related cast operations under boundary conditions. [Fix](cast) Fix wrong corner case of datetime conversion about timezone convert #57422
• Fixed the incorrect precision derivation of results for some datetime-related functions. [fix](fold constant) fix result's scale of some datetime related function #56671
• Fixed the core dump issue when inf is used as a predicate condition for float types. [fix](olap) Need to handle floating-point Infinity values in the predicate #57100
• Fixed the core dump issue of the explode function with variable parameters. [fix](function) Crash caused by explode function #56991
• Fixed the instability issue when casting decimal256 to float types. [fix](decimal256) fix unstable overflow error when casting decimal256 to float #56848
• Fixed the potential core dump issue caused by repeated scheduling during spill disk operations. [fix](pipeline) Crashing caused by repeated spill operations #56755
• Fixed the occasional incorrect adjustment of the order between mark join and other joins. [fix](nereids)ReorderJoin rule should not absorb mark join to multi join #56837
• Fixed the issue where some commands were not correctly forwarded to the master frontend for execution. [fix](nereids)fix some show command's redirect policy #55185
• Fixed the occasional incorrect generation of partition topn by window functions. [fix](nereids) partition topn optimization requires all window expressions are in the same order #56622
• Fixed the potential query error when synchronized mv definitions contain keywords. [fix](nereids)fix sync mv parse error if it contains sql KEYWORD #57052

Others

• Prohibited creating another synchronized mv based on a synchronized mv. [fix](nereids)forbid create sync mv based on annother sync mv #56912
• Fixed the issue of delayed memory release in profiles. [fix](nereids)relase physical plan in Profile.releaseMemory() #57257