Skip to content

branch-4.0: [Opt](function) opt of certain time field functions used in conjunction with FROM_UNIXTIME.#60843

Merged
yiguolei merged 2 commits intoapache:branch-4.0from
linrrzqqq:pick-unixtime-fn-branch-4.0
Feb 26, 2026
Merged

branch-4.0: [Opt](function) opt of certain time field functions used in conjunction with FROM_UNIXTIME.#60843
yiguolei merged 2 commits intoapache:branch-4.0from
linrrzqqq:pick-unixtime-fn-branch-4.0

Conversation

@linrrzqqq
Copy link
Copy Markdown
Contributor

@linrrzqqq linrrzqqq commented Feb 26, 2026

pick: #57941 and #60829

@linrrzqqq linrrzqqq requested a review from yiguolei as a code owner February 26, 2026 03:45
@hello-stephen
Copy link
Copy Markdown
Contributor

hello-stephen commented Feb 26, 2026

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@linrrzqqq
[Opt](function) opt of certain time field functions used in conjuncti…
ee6ac3b 
…on with FROM_UNIXTIME. (apache#57941)

opt of certain time field functions(`HOUR`, `MINUTE`, `SECOND`, `MICROSECOND`) used in conjunction with `FROM_UNIXTIME`

Take `HOUR(FROM_UNIXTIME(ts))` as an example:
The `hour(from_unixtime(xxx))` function is slow because
`from_unixtime`needs to extract the full yyyy-MM-dd HH:mm:ss format from
the timestamp, which is not necessary. By calculating only the required
fields directly from the timestamp, the process can be significantly
faster.

Add a function `hour_from_unixtime(ts)` to extract the hour from a unix
timestamp, which is timezone aware.

Implementation:
1. Lookup the timezone offset with cctz library
2. Calculate the hour from local unixtime
```cpp
int64_t local_unixtime = unixtime + timezone.lookup_offset(unixtime);
int hour = (local_unixtime % (24 * 3600)) / 3600
```

Performance:

Before VS After:
```text
-- HOUR
Doris> SELECT COUNT(HOUR(FROM_UNIXTIME(ts))) FROM test_hour_from_unixtime;
+--------------------------------+
| COUNT(HOUR(FROM_UNIXTIME(ts))) |
+--------------------------------+
|                      100000000 |
+--------------------------------+
1 row in set (9.51 sec)

Doris> SELECT COUNT(HOUR(FROM_UNIXTIME(ts))) FROM test_hour_from_unixtime;
+--------------------------------+
| COUNT(HOUR(FROM_UNIXTIME(ts))) |
+--------------------------------+
|                      100000000 |
+--------------------------------+
1 row in set (0.96 sec)

-- MINUTE
Doris> SELECT COUNT(MINUTE(FROM_UNIXTIME(ts))) FROM test_hour_from_unixtime;
+----------------------------------+
| COUNT(MINUTE(FROM_UNIXTIME(ts))) |
+----------------------------------+
|                        100000000 |
+----------------------------------+
1 row in set (10.98 sec)

Doris> SELECT COUNT(MINUTE(FROM_UNIXTIME(ts))) FROM test_hour_from_unixtime;
+----------------------------------+
| COUNT(MINUTE(FROM_UNIXTIME(ts))) |
+----------------------------------+
|                        100000000 |
+----------------------------------+
1 row in set (1.00 sec)

-- SECOND
Doris> SELECT COUNT(SECOND(FROM_UNIXTIME(ts))) FROM test_hour_from_unixtime;
+----------------------------------+
| COUNT(SECOND(FROM_UNIXTIME(ts))) |
+----------------------------------+
|                        100000000 |
+----------------------------------+
1 row in set (10.01 sec)

Doris> SELECT COUNT(SECOND(FROM_UNIXTIME(ts))) FROM test_hour_from_unixtime;
+----------------------------------+
| COUNT(SECOND(FROM_UNIXTIME(ts))) |
+----------------------------------+
|                        100000000 |
+----------------------------------+
1 row in set (0.90 sec)

-- MICROSECOND
Doris> SELECT COUNT(MICROSECOND(FROM_UNIXTIME(ts))) FROM test_hour_from_unixtime;
+---------------------------------------+
| COUNT(MICROSECOND(FROM_UNIXTIME(ts))) |
+---------------------------------------+
|                             100000000 |
+---------------------------------------+
1 row in set (9.75 sec)

Doris> SELECT COUNT(MICROSECOND(FROM_UNIXTIME(ts))) FROM test_hour_from_unixtime;
+---------------------------------------+
| COUNT(MICROSECOND(FROM_UNIXTIME(ts))) |
+---------------------------------------+
|                             100000000 |
+---------------------------------------+
1 row in set (1.24 sec)
```
@linrrzqqq
[Fix](func) Fixing microsecond_from_unixtime decimal arg type to DECI…
f482b62 
…MAL(18, 6) (apache#60829)

### What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

```sql
mysql> SELECT MICROSECOND_FROM_UNIXTIME(1145.14);
+------------------------------------+
| MICROSECOND_FROM_UNIXTIME(1145.14) |
+------------------------------------+
|                             140000 |
+------------------------------------+
1 row in set (0.02 sec)

mysql> select @@debug_skip_fold_constant;
+----------------------------+
| @@debug_skip_fold_constant |
+----------------------------+
|                          0 |
+----------------------------+
1 row in set (0.01 sec)

mysql> set debug_skip_fold_constant=true;
Query OK, 0 rows affected (0.02 sec)

mysql> SELECT MICROSECOND_FROM_UNIXTIME(1145.14);
ERROR 1105 (HY000): RpcException, msg: send fragments failed. io.grpc.StatusRuntimeException: UNAVAILABLE: Keepalive failed. The connection is likely gone, host: 10.16.10.3
```
```text
F20260225 00:04:40.698660 618581 status.h:467] Bad cast from type:doris::vectorized::ColumnDecimal<(doris::PrimitiveType)28>* to doris::vectorized::ColumnDecimal<(doris::PrimitiveType)29> const*
*** Check failure stack trace: ***
    @     0x55dbc5c4dc4f  google::LogMessage::SendToLog()
    @     0x55dbc5c44260  google::LogMessage::Flush()
    @     0x55dbc5c47959  google::LogMessageFatal::~LogMessageFatal()
    @     0x55db8f09709b  doris::Status::FatalError<>()
    @     0x55dbb7936b66  _ZZ11assert_castIPKN5doris10vectorized13ColumnDecimalILNS0_13PrimitiveTypeE29EEEL18TypeCheckOnRelease1EPKNS1_7IColumnEET_OT1_ENKUlOSB_E_clISA_EES6_SE_
    @     0x55dbb7935f74  assert_cast<>()
    @     0x55dbbcbf2f1f  doris::vectorized::FunctionTimeFieldFromUnixtime<>::execute_impl()
    @     0x55dbbcbf3762  doris::vectorized::FunctionTimeFieldFromUnixtime<>::execute_impl()
    @     0x55dbba5ca2d8  doris::vectorized::PreparedFunctionImpl::_execute_skipped_constant_deal()
    @     0x55dbba5ad094  doris::vectorized::PreparedFunctionImpl::default_implementation_for_constant_arguments()
    @     0x55dbba5af1fd  doris::vectorized::PreparedFunctionImpl::default_execute()
    @     0x55dbba5af624  doris::vectorized::PreparedFunctionImpl::execute()
```

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
@linrrzqqq linrrzqqq force-pushed the pick-unixtime-fn-branch-4.0 branch from 9cb475b to f482b62 Compare February 26, 2026 04:31
@linrrzqqq
Copy link
Copy Markdown
Contributor Author

linrrzqqq commented Feb 26, 2026

run buildall

@hello-stephen
Copy link
Copy Markdown
Contributor

hello-stephen commented Feb 26, 2026

BE UT Coverage Report

Increment line coverage 7.06% (6/85) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 52.98% (19129/36104)
Line Coverage 36.17% (178114/492428)
Region Coverage 32.77% (137986/421035)
Branch Coverage 33.74% (59914/177598)

@yiguolei yiguolei merged commit 7362354 into apache:branch-4.0 Feb 26, 2026
23 of 26 checks passed
@linrrzqqq linrrzqqq deleted the pick-unixtime-fn-branch-4.0 branch February 26, 2026 09:12
ybtsdst pushed a commit to ybtsdst/doris that referenced this pull request Feb 27, 2026
@linrrzqqq
branch-4.0: [Opt](function) opt of certain time field functions used …
2fb9bbe 
…in conjunction with FROM_UNIXTIME. (apache#60843)

pick: apache#57941 and
apache#60829
@yiguolei yiguolei mentioned this pull request Mar 7, 2026
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@yiguolei yiguolei yiguolei approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@linrrzqqq @hello-stephen @yiguolei