Skip to content

[fix](inverted index) Fix skipping data reads for columns when index is hit#57461

Merged
eldenmoon merged 2 commits intoapache:masterfrom
csun5285:fix_select
Oct 29, 2025
Merged

[fix](inverted index) Fix skipping data reads for columns when index is hit#57461
eldenmoon merged 2 commits intoapache:masterfrom
csun5285:fix_select

Conversation

@csun5285
Copy link
Copy Markdown
Contributor

@csun5285 csun5285 commented Oct 29, 2025

What problem does this PR solve?

Different subcolumns from variant share the same parent_unique_id, so we choose to abandon this optimization.

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Copy Markdown
Contributor

Thearas commented Oct 29, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@csun5285
Copy link
Copy Markdown
Contributor Author

run buildall

@doris-robot
Copy link
Copy Markdown

ClickBench: Total hot run time: 29.23 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 47655a2f48aa848564d1625ae1c577d159f6ccca, data reload: false

query1	0.07	0.05	0.06
query2	0.11	0.06	0.05
query3	0.26	0.10	0.10
query4	1.62	0.13	0.13
query5	0.30	0.29	0.29
query6	1.22	0.68	0.68
query7	0.04	0.03	0.03
query8	0.06	0.05	0.05
query9	0.67	0.58	0.60
query10	0.63	0.64	0.62
query11	0.19	0.13	0.14
query12	0.19	0.14	0.14
query13	0.64	0.61	0.61
query14	1.05	1.04	1.03
query15	0.92	0.91	0.90
query16	0.44	0.43	0.45
query17	1.11	1.24	1.15
query18	0.24	0.22	0.22
query19	2.04	1.93	1.93
query20	0.01	0.02	0.01
query21	15.36	0.20	0.16
query22	5.02	0.08	0.05
query23	15.64	0.31	0.12
query24	2.65	0.60	0.51
query25	0.08	0.07	0.07
query26	0.16	0.15	0.16
query27	0.07	0.06	0.05
query28	4.57	1.19	0.96
query29	12.59	4.57	3.77
query30	0.32	0.15	0.12
query31	2.83	0.63	0.42
query32	3.24	0.59	0.49
query33	3.16	3.19	3.11
query34	15.92	5.29	4.55
query35	4.66	4.71	4.64
query36	0.70	0.53	0.53
query37	0.11	0.08	0.07
query38	0.06	0.05	0.05
query39	0.04	0.03	0.03
query40	0.20	0.15	0.14
query41	0.10	0.04	0.03
query42	0.05	0.04	0.04
query43	0.05	0.04	0.04
Total cold run time: 99.39 s
Total hot run time: 29.23 s

@doris-robot
Copy link
Copy Markdown

BE UT Coverage Report

Increment line coverage 100.00% (3/3) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 52.75% (18064/34243)
Line Coverage 37.99% (163807/431226)
Region Coverage 32.37% (124904/385851)
Branch Coverage 33.72% (54606/161931)

@hello-stephen
Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 100.00% (3/3) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 71.40% (23954/33548)
Line Coverage 57.85% (249347/431038)
Region Coverage 53.04% (207183/390613)
Branch Coverage 54.71% (89017/162714)

@csun5285
Copy link
Copy Markdown
Contributor Author

run buildall

@doris-robot
Copy link
Copy Markdown

TPC-DS: Total hot run time: 190741 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f0fb29ae2bc229983be87b7de6bd7928eb45d03d, data reload: false

query1	1071	438	405	405
query2	6572	1718	1695	1695
query3	6757	219	223	219
query4	25766	23864	23187	23187
query5	5925	668	500	500
query6	348	242	225	225
query7	4652	514	302	302
query8	321	273	257	257
query9	8693	2619	2582	2582
query10	581	364	305	305
query11	15859	14996	14931	14931
query12	188	124	120	120
query13	1700	593	465	465
query14	12697	9309	9294	9294
query15	264	191	172	172
query16	7836	708	539	539
query17	1604	845	673	673
query18	2240	481	365	365
query19	324	225	200	200
query20	152	144	138	138
query21	231	138	123	123
query22	4732	4562	4764	4562
query23	34869	33661	33797	33661
query24	8142	2514	2501	2501
query25	591	530	437	437
query26	1273	280	159	159
query27	2799	521	373	373
query28	4968	2267	2312	2267
query29	830	661	569	569
query30	307	250	210	210
query31	910	836	789	789
query32	84	78	75	75
query33	602	397	369	369
query34	837	880	555	555
query35	845	884	820	820
query36	987	1004	947	947
query37	145	109	89	89
query38	3774	3694	3646	3646
query39	1554	1527	1487	1487
query40	239	123	118	118
query41	71	67	67	67
query42	128	108	114	108
query43	497	481	476	476
query44	1219	754	721	721
query45	183	185	173	173
query46	895	991	650	650
query47	1773	1794	1721	1721
query48	413	417	338	338
query49	750	495	414	414
query50	638	684	423	423
query51	3886	3908	3833	3833
query52	114	116	94	94
query53	245	271	193	193
query54	608	613	541	541
query55	87	88	92	88
query56	334	314	317	314
query57	1166	1193	1120	1120
query58	287	276	286	276
query59	2510	2626	2535	2535
query60	360	342	319	319
query61	164	161	165	161
query62	790	760	671	671
query63	230	190	192	190
query64	4416	1144	880	880
query65	4083	3928	3935	3928
query66	1050	430	322	322
query67	15416	15270	15023	15023
query68	7771	895	596	596
query69	498	330	288	288
query70	1414	1270	1305	1270
query71	499	339	320	320
query72	5841	4915	4853	4853
query73	543	571	363	363
query74	8877	9089	9121	9089
query75	4278	3328	2852	2852
query76	3267	1185	730	730
query77	855	417	344	344
query78	9500	9728	9015	9015
query79	2297	802	582	582
query80	684	565	512	512
query81	492	261	241	241
query82	423	166	131	131
query83	295	267	251	251
query84	311	114	95	95
query85	877	474	430	430
query86	337	339	301	301
query87	3704	3741	3614	3614
query88	3122	2258	2263	2258
query89	381	319	299	299
query90	2041	221	226	221
query91	164	185	139	139
query92	84	75	70	70
query93	1161	973	642	642
query94	684	452	343	343
query95	414	336	317	317
query96	490	584	285	285
query97	2998	2985	2875	2875
query98	234	244	218	218
query99	1430	1404	1287	1287
Total cold run time: 280548 ms
Total hot run time: 190741 ms

@doris-robot
Copy link
Copy Markdown

ClickBench: Total hot run time: 28.07 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f0fb29ae2bc229983be87b7de6bd7928eb45d03d, data reload: false

query1	0.05	0.05	0.05
query2	0.10	0.05	0.06
query3	0.26	0.08	0.09
query4	1.61	0.12	0.12
query5	0.29	0.26	0.26
query6	1.20	0.66	0.67
query7	0.03	0.02	0.02
query8	0.05	0.04	0.04
query9	0.62	0.53	0.51
query10	0.57	0.58	0.57
query11	0.17	0.11	0.12
query12	0.17	0.12	0.12
query13	0.62	0.60	0.60
query14	1.06	1.01	1.01
query15	0.85	0.84	0.84
query16	0.39	0.40	0.38
query17	1.03	1.00	1.00
query18	0.21	0.20	0.20
query19	1.86	1.77	1.86
query20	0.02	0.01	0.01
query21	15.46	0.17	0.14
query22	5.13	0.06	0.04
query23	15.68	0.26	0.11
query24	3.30	1.06	0.93
query25	0.08	0.06	0.06
query26	0.15	0.15	0.14
query27	0.06	0.06	0.05
query28	5.02	1.16	0.94
query29	12.58	3.99	3.26
query30	0.29	0.13	0.12
query31	2.81	0.59	0.39
query32	3.23	0.54	0.47
query33	2.98	3.02	3.03
query34	15.86	5.13	4.57
query35	4.58	4.58	4.58
query36	0.69	0.51	0.49
query37	0.09	0.06	0.06
query38	0.06	0.04	0.04
query39	0.04	0.03	0.03
query40	0.18	0.15	0.14
query41	0.10	0.03	0.04
query42	0.04	0.03	0.03
query43	0.04	0.04	0.03
Total cold run time: 99.61 s
Total hot run time: 28.07 s

Copy link
Copy Markdown
Member

@airborne12 airborne12 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Oct 29, 2025
@github-actions
Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@hello-stephen
Copy link
Copy Markdown
Contributor

BE UT Coverage Report

Increment line coverage 60.00% (3/5) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 52.76% (18065/34243)
Line Coverage 37.99% (163824/431228)
Region Coverage 32.37% (124917/385851)
Branch Coverage 33.73% (54613/161931)

Copy link
Copy Markdown
Member

@eldenmoon eldenmoon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@hello-stephen
Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 100.00% (5/5) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 71.40% (23953/33548)
Line Coverage 57.80% (249142/431040)
Region Coverage 52.99% (206969/390613)
Branch Coverage 54.67% (88951/162714)

@eldenmoon eldenmoon merged commit 3b1d999 into apache:master Oct 29, 2025
27 of 29 checks passed
github-actions bot pushed a commit that referenced this pull request Oct 29, 2025
…is hit (#57461)

Different subcolumns from variant share the same parent_unique_id, so we
choose to abandon this optimization.
github-actions bot pushed a commit that referenced this pull request Oct 29, 2025
…is hit (#57461)

Different subcolumns from variant share the same parent_unique_id, so we
choose to abandon this optimization.
yiguolei pushed a commit that referenced this pull request Oct 30, 2025
… when index is hit #57461 (#57476)

Cherry-picked from #57461

Co-authored-by: Sun Chenyang <sunchenyang@selectdb.com>
dwdwqfwe pushed a commit to dwdwqfwe/doris that referenced this pull request Oct 31, 2025
…is hit (apache#57461)

Different subcolumns from variant share the same parent_unique_id, so we
choose to abandon this optimization.
morrySnow pushed a commit that referenced this pull request Nov 3, 2025
… when index is hit #57461 (#57475)

Cherry-picked from #57461

Co-authored-by: Sun Chenyang <sunchenyang@selectdb.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/3.1.3-merged dev/4.0.1-merged reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants