Add dictionary-dependent index validation for table configs

[rsrkpatwari1234]

Partially Fixes #17398

Problem

Currently, Pinot allows invalid table configurations where indexes that require dictionary encoding (inverted index, range index, FST index, inverted FST index) are enabled on columns with noDictionaryColumns set. This leads to segment creation failures at runtime rather than failing fast at table config validation time.

Solution

In this PR, I have added validation logic in TableConfigUtils.validateIndexTypeCompatibility() to detect and reject incompatible index configurations during table creation/update, providing clear, actionable error messages to users.

Testing

Added unit tests

enhancement user-experience

[codecov-commenter]

❌ 36 Tests Failed:

Tests completed	Failed	Passed	Skipped
16514	36	16478	40

View the top 3 failed test(s) by shortest run time

org.apache.pinot.core.data.manager.TableIndexingTest::testAddIndex

Stack Traces | 0.001s run time

Expected error: "Cannot create inverted index for raw index column: col" for test case: raw_sv_DOUBLE,inverted_index but got: "Column 'col' has inverted_index enabled but dictionary is disabled. Inverted index requires dictionary encoding to be enabled."

org.apache.pinot.segment.local.utils.TableConfigUtilsTest::testInvertedFSTIndexWithoutDictionary

Stack Traces | 0.002s run time

The exception was thrown with the wrong message: expected ".*inverted FST.*dictionary.*" but got "Column 'test_col' has inverted_index enabled but dictionary is disabled. Inverted index requires dictionary encoding to be enabled."

org.apache.pinot.segment.local.utils.TableConfigUtilsTest::testFSTIndexWithoutDictionary

Stack Traces | 0.004s run time

The exception was thrown with the wrong message: expected ".*fst_index.*dictionary.*" but got "Cannot create FST index on column: test_col without dictionary"

org.apache.pinot.segment.local.utils.TableConfigUtilsTest::testValidateFieldConfig

Stack Traces | 0.008s run time

Range index with forward index disabled no dictionary column is allowed

org.apache.pinot.integration.tests.UpsertTableIntegrationTest::testUpsertCompactionWithSoftDelete

Stack Traces | 606s run time

Failed to meet condition in 600000ms, error message: Failed to load all documents

View the full list of 31 ❄️ flaky test(s)

org.apache.pinot.core.data.manager.TableIndexingTest::testAddIndex[raw_mv_BOOLEAN,inverted_index](233)

Flake rate in main: 89.74% (Passed 4 times, Failed 35 times)

Stack Traces | 0s run time

Expected error: "Cannot create inverted index for raw index column: col" for test case: raw_mv_BOOLEAN,inverted_index but got: "Column 'col' has inverted_index enabled but dictionary is disabled. Inverted index requires dictionary encoding to be enabled."

org.apache.pinot.core.data.manager.TableIndexingTest::testAddIndex[raw_mv_BOOLEAN,range_index](237)

Flake rate in main: 89.74% (Passed 4 times, Failed 35 times)

Stack Traces | 0.002s run time

Expected success for test case: raw_mv_BOOLEAN,range_index but got error: java.lang.IllegalStateException: Column 'col' has range_index enabled but dictionary is disabled. Range index requires dictionary encoding to be enabled.

org.apache.pinot.core.data.manager.TableIndexingTest::testAddIndex[raw_mv_BYTES,inverted_index](401)

Flake rate in main: 89.74% (Passed 4 times, Failed 35 times)

Stack Traces | 0s run time

Expected error: "Caught exception while reading data" for test case: raw_mv_BYTES,inverted_index but got: "Column 'col' has inverted_index enabled but dictionary is disabled. Inverted index requires dictionary encoding to be enabled."

org.apache.pinot.core.data.manager.TableIndexingTest::testAddIndex[raw_mv_DOUBLE,inverted_index](161)

Flake rate in main: 89.74% (Passed 4 times, Failed 35 times)

Stack Traces | 0.001s run time

Expected error: "Cannot create inverted index for raw index column: col" for test case: raw_mv_DOUBLE,inverted_index but got: "Column 'col' has inverted_index enabled but dictionary is disabled. Inverted index requires dictionary encoding to be enabled."

org.apache.pinot.core.data.manager.TableIndexingTest::testAddIndex[raw_mv_DOUBLE,range_index](165)

Flake rate in main: 89.74% (Passed 4 times, Failed 35 times)

Stack Traces | 0.001s run time

Expected success for test case: raw_mv_DOUBLE,range_index but got error: java.lang.IllegalStateException: Column 'col' has range_index enabled but dictionary is disabled. Range index requires dictionary encoding to be enabled.

org.apache.pinot.core.data.manager.TableIndexingTest::testAddIndex[raw_mv_FLOAT,inverted_index](113)

Flake rate in main: 89.74% (Passed 4 times, Failed 35 times)

Stack Traces | 0.001s run time

Expected error: "Cannot create inverted index for raw index column: col" for test case: raw_mv_FLOAT,inverted_index but got: "Column 'col' has inverted_index enabled but dictionary is disabled. Inverted index requires dictionary encoding to be enabled."

org.apache.pinot.core.data.manager.TableIndexingTest::testAddIndex[raw_mv_FLOAT,range_index](117)

Flake rate in main: 89.74% (Passed 4 times, Failed 35 times)

Stack Traces | 0.001s run time

Expected success for test case: raw_mv_FLOAT,range_index but got error: java.lang.IllegalStateException: Column 'col' has range_index enabled but dictionary is disabled. Range index requires dictionary encoding to be enabled.

org.apache.pinot.core.data.manager.TableIndexingTest::testAddIndex[raw_mv_INT,inverted_index](17)

Flake rate in main: 89.74% (Passed 4 times, Failed 35 times)

Stack Traces | 0.003s run time

Expected error: "Cannot create inverted index for raw index column: col" for test case: raw_mv_INT,inverted_index but got: "Column 'col' has inverted_index enabled but dictionary is disabled. Inverted index requires dictionary encoding to be enabled."

org.apache.pinot.core.data.manager.TableIndexingTest::testAddIndex[raw_mv_INT,range_index](21)

Flake rate in main: 89.74% (Passed 4 times, Failed 35 times)

Stack Traces | 0.001s run time

Expected success for test case: raw_mv_INT,range_index but got error: java.lang.IllegalStateException: Column 'col' has range_index enabled but dictionary is disabled. Range index requires dictionary encoding to be enabled.

org.apache.pinot.core.data.manager.TableIndexingTest::testAddIndex[raw_mv_LONG,inverted_index](65)

Flake rate in main: 89.74% (Passed 4 times, Failed 35 times)

Stack Traces | 0.001s run time

Expected error: "Cannot create inverted index for raw index column: col" for test case: raw_mv_LONG,inverted_index but got: "Column 'col' has inverted_index enabled but dictionary is disabled. Inverted index requires dictionary encoding to be enabled."

org.apache.pinot.core.data.manager.TableIndexingTest::testAddIndex[raw_mv_LONG,range_index](69)

Flake rate in main: 89.74% (Passed 4 times, Failed 35 times)

Stack Traces | 0.001s run time

Expected success for test case: raw_mv_LONG,range_index but got error: java.lang.IllegalStateException: Column 'col' has range_index enabled but dictionary is disabled. Range index requires dictionary encoding to be enabled.

org.apache.pinot.core.data.manager.TableIndexingTest::testAddIndex[raw_mv_STRING,inverted_index](329)

Flake rate in main: 89.74% (Passed 4 times, Failed 35 times)

Stack Traces | 0s run time

Expected error: "Cannot create inverted index for raw index column: col" for test case: raw_mv_STRING,inverted_index but got: "Column 'col' has inverted_index enabled but dictionary is disabled. Inverted index requires dictionary encoding to be enabled."

org.apache.pinot.core.data.manager.TableIndexingTest::testAddIndex[raw_mv_TIMESTAMP,inverted_index](281)

Flake rate in main: 89.74% (Passed 4 times, Failed 35 times)

Stack Traces | 0.001s run time

Expected error: "Cannot create inverted index for raw index column: col" for test case: raw_mv_TIMESTAMP,inverted_index but got: "Column 'col' has inverted_index enabled but dictionary is disabled. Inverted index requires dictionary encoding to be enabled."

org.apache.pinot.core.data.manager.TableIndexingTest::testAddIndex[raw_mv_TIMESTAMP,range_index](285)

Flake rate in main: 89.74% (Passed 4 times, Failed 35 times)

Stack Traces | 0s run time

Expected success for test case: raw_mv_TIMESTAMP,range_index but got error: java.lang.IllegalStateException: Column 'col' has range_index enabled but dictionary is disabled. Range index requires dictionary encoding to be enabled.

org.apache.pinot.core.data.manager.TableIndexingTest::testAddIndex[raw_sv_BIG_DECIMAL,inverted_index](197)

Flake rate in main: 89.74% (Passed 4 times, Failed 35 times)

Stack Traces | 0.001s run time

Expected error: "Cannot create inverted index for raw index column: col" for test case: raw_sv_BIG_DECIMAL,inverted_index but got: "Column 'col' has inverted_index enabled but dictionary is disabled. Inverted index requires dictionary encoding to be enabled."

org.apache.pinot.core.data.manager.TableIndexingTest::testAddIndex[raw_sv_BIG_DECIMAL,range_index](201)

Flake rate in main: 89.74% (Passed 4 times, Failed 35 times)

Stack Traces | 0.001s run time

Expected error: "Unsupported data type: BIG_DECIMAL" for test case: raw_sv_BIG_DECIMAL,range_index but got: "Column 'col' has range_index enabled but dictionary is disabled. Range index requires dictionary encoding to be enabled."

org.apache.pinot.core.data.manager.TableIndexingTest::testAddIndex[raw_sv_BOOLEAN,inverted_index](221)

Flake rate in main: 89.74% (Passed 4 times, Failed 35 times)

Stack Traces | 0s run time

Expected error: "Cannot create inverted index for raw index column: col" for test case: raw_sv_BOOLEAN,inverted_index but got: "Column 'col' has inverted_index enabled but dictionary is disabled. Inverted index requires dictionary encoding to be enabled."

org.apache.pinot.core.data.manager.TableIndexingTest::testAddIndex[raw_sv_BOOLEAN,range_index](225)

Flake rate in main: 89.74% (Passed 4 times, Failed 35 times)

Stack Traces | 0.002s run time

Expected success for test case: raw_sv_BOOLEAN,range_index but got error: java.lang.IllegalStateException: Column 'col' has range_index enabled but dictionary is disabled. Range index requires dictionary encoding to be enabled.

org.apache.pinot.core.data.manager.TableIndexingTest::testAddIndex[raw_sv_BYTES,inverted_index](389)

Flake rate in main: 89.74% (Passed 4 times, Failed 35 times)

Stack Traces | 0s run time

Expected error: "Caught exception while reading data" for test case: raw_sv_BYTES,inverted_index but got: "Column 'col' has inverted_index enabled but dictionary is disabled. Inverted index requires dictionary encoding to be enabled."

org.apache.pinot.core.data.manager.TableIndexingTest::testAddIndex[raw_sv_DOUBLE,inverted_index](149)

Flake rate in main: 89.74% (Passed 4 times, Failed 35 times)

Stack Traces | 0.001s run time

Expected error: "Cannot create inverted index for raw index column: col" for test case: raw_sv_DOUBLE,inverted_index but got: "Column 'col' has inverted_index enabled but dictionary is disabled. Inverted index requires dictionary encoding to be enabled."

org.apache.pinot.core.data.manager.TableIndexingTest::testAddIndex[raw_sv_DOUBLE,range_index](153)

Flake rate in main: 89.74% (Passed 4 times, Failed 35 times)

Stack Traces | 0.001s run time

Expected success for test case: raw_sv_DOUBLE,range_index but got error: java.lang.IllegalStateException: Column 'col' has range_index enabled but dictionary is disabled. Range index requires dictionary encoding to be enabled.

org.apache.pinot.core.data.manager.TableIndexingTest::testAddIndex[raw_sv_FLOAT,inverted_index](101)

Flake rate in main: 89.74% (Passed 4 times, Failed 35 times)

Stack Traces | 0.001s run time

Expected error: "Cannot create inverted index for raw index column: col" for test case: raw_sv_FLOAT,inverted_index but got: "Column 'col' has inverted_index enabled but dictionary is disabled. Inverted index requires dictionary encoding to be enabled."

org.apache.pinot.core.data.manager.TableIndexingTest::testAddIndex[raw_sv_FLOAT,range_index](105)

Flake rate in main: 89.74% (Passed 4 times, Failed 35 times)

Stack Traces | 0.001s run time

Expected success for test case: raw_sv_FLOAT,range_index but got error: java.lang.IllegalStateException: Column 'col' has range_index enabled but dictionary is disabled. Range index requires dictionary encoding to be enabled.

org.apache.pinot.core.data.manager.TableIndexingTest::testAddIndex[raw_sv_INT,inverted_index](5)

Flake rate in main: 89.74% (Passed 4 times, Failed 35 times)

Stack Traces | 0.016s run time

Expected error: "Cannot create inverted index for raw index column: col" for test case: raw_sv_INT,inverted_index but got: "Column 'col' has inverted_index enabled but dictionary is disabled. Inverted index requires dictionary encoding to be enabled."

org.apache.pinot.core.data.manager.TableIndexingTest::testAddIndex[raw_sv_INT,range_index](9)

Flake rate in main: 89.74% (Passed 4 times, Failed 35 times)

Stack Traces | 0.001s run time

Expected success for test case: raw_sv_INT,range_index but got error: java.lang.IllegalStateException: Column 'col' has range_index enabled but dictionary is disabled. Range index requires dictionary encoding to be enabled.

org.apache.pinot.core.data.manager.TableIndexingTest::testAddIndex[raw_sv_JSON,inverted_index](365)

Flake rate in main: 89.74% (Passed 4 times, Failed 35 times)

Stack Traces | 0s run time

Expected error: "Cannot create inverted index for raw index column: col" for test case: raw_sv_JSON,inverted_index but got: "Column 'col' has inverted_index enabled but dictionary is disabled. Inverted index requires dictionary encoding to be enabled."

org.apache.pinot.core.data.manager.TableIndexingTest::testAddIndex[raw_sv_LONG,inverted_index](53)

Flake rate in main: 89.74% (Passed 4 times, Failed 35 times)

Stack Traces | 0.001s run time

Expected error: "Cannot create inverted index for raw index column: col" for test case: raw_sv_LONG,inverted_index but got: "Column 'col' has inverted_index enabled but dictionary is disabled. Inverted index requires dictionary encoding to be enabled."

org.apache.pinot.core.data.manager.TableIndexingTest::testAddIndex[raw_sv_LONG,range_index](57)

Flake rate in main: 89.74% (Passed 4 times, Failed 35 times)

Stack Traces | 0s run time

Expected success for test case: raw_sv_LONG,range_index but got error: java.lang.IllegalStateException: Column 'col' has range_index enabled but dictionary is disabled. Range index requires dictionary encoding to be enabled.

org.apache.pinot.core.data.manager.TableIndexingTest::testAddIndex[raw_sv_STRING,inverted_index](317)

Flake rate in main: 89.74% (Passed 4 times, Failed 35 times)

Stack Traces | 0.001s run time

Expected error: "Cannot create inverted index for raw index column: col" for test case: raw_sv_STRING,inverted_index but got: "Column 'col' has inverted_index enabled but dictionary is disabled. Inverted index requires dictionary encoding to be enabled."

org.apache.pinot.core.data.manager.TableIndexingTest::testAddIndex[raw_sv_TIMESTAMP,inverted_index](269)

Flake rate in main: 89.74% (Passed 4 times, Failed 35 times)

Stack Traces | 0.001s run time

Expected error: "Cannot create inverted index for raw index column: col" for test case: raw_sv_TIMESTAMP,inverted_index but got: "Column 'col' has inverted_index enabled but dictionary is disabled. Inverted index requires dictionary encoding to be enabled."

org.apache.pinot.core.data.manager.TableIndexingTest::testAddIndex[raw_sv_TIMESTAMP,range_index](273)

Flake rate in main: 89.74% (Passed 4 times, Failed 35 times)

Stack Traces | 0.001s run time

Expected success for test case: raw_sv_TIMESTAMP,range_index but got error: java.lang.IllegalStateException: Column 'col' has range_index enabled but dictionary is disabled. Range index requires dictionary encoding to be enabled.

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}