fix(bloom_filter): skip 1-byte marker in unfreeze_from_arrays to fix version mismatch#1979
Open
BrytonLee wants to merge 1 commit intoapache:masterfrom
Open
fix(bloom_filter): skip 1-byte marker in unfreeze_from_arrays to fix version mismatch#1979BrytonLee wants to merge 1 commit intoapache:masterfrom
BrytonLee wants to merge 1 commit intoapache:masterfrom
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR fixes a version mismatch error in the bloom filter aggregation by ensuring the unfreeze_from_arrays method correctly skips the 1-byte marker that freeze_to_arrays writes before the BloomFilter data. The bug caused "unsupported version: 16777216" errors because the marker byte (0x01) was being incorrectly interpreted as part of the version field when read as a big-endian i32.
Changes:
- Added marker byte skipping in unfreeze_from_arrays to match the serialization format used by freeze_to_arrays
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
richox
reviewed
Feb 2, 2026
Contributor
There was a problem hiding this comment.
we can remove this byte writing, it is redundant now.
richox
reviewed
Feb 2, 2026
| if let Some(w) = v { | ||
| let mut cursor = Cursor::new(w); | ||
| // Skip the 1-byte marker written by freeze_to_arrays | ||
| cursor.read_u8()?; |
Contributor
There was a problem hiding this comment.
and we dont have to skip this byte reading
The 1-byte marker written in freeze_to_arrays was redundant because BinaryArray already uses null values to distinguish Some(BloomFilter) from None. This caused SparkBloomFilter::read_from to read the marker as part of the version field, resulting in "unsupported version: 16777216" error. The marker is only needed in spill/unspill (continuous byte stream) but not in freeze_to_arrays/unfreeze_from_arrays (BinaryArray with native null support).
BrytonLee
force-pushed
the
fix/bloom-filter-unfreeze
branch
from
8a1c8a3 to
041c7fc
Compare
richox
approved these changes
Feb 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
freeze_to_arrays writes a 1-byte marker (0x01) before the BloomFilter data, but unfreeze_from_arrays was reading directly without skipping this marker. This caused SparkBloomFilter::read_from to interpret the marker byte as part of the version field, resulting in "unsupported version: 16777216" error (0x01000000 when read as big-endian i32).
The fix adds cursor.read_u8() to skip the marker before reading the BloomFilter.