[AURON #1850] Add FlinkArrowUtils for Flink-Arrow type conversion by x-tong · Pull Request #1959 · apache/auron

x-tong · 2026-01-26T10:18:26Z

Summary

Part 1/3 of Flink RowData to Arrow conversion implementation (split from #1930).

This PR adds the foundational type conversion utilities:

FlinkArrowUtils: Conversion form Flink RowData to Arrow types
Support for all common Flink types including primitives, temporal, and complex types
Comprehensive unit tests for type conversion

Follow-up PRs

Part 2: FlinkArrowFieldWriter + FlinkArrowWriter
Part 3: FlinkArrowFFIExporter

Test plan

Unit tests for FlinkArrowUtils type conversion
Build passes with ./auron-build.sh --pre --sparkver 3.5 --scalaver 2.12 -Pflink-1.18 -DskipBuildNative
Code formatted with ./dev/reformat

Related: #1850

Part 1 of Flink RowData to Arrow conversion implementation. This PR adds the foundational type conversion utilities: - FlinkArrowUtils: Bidirectional conversion between Flink LogicalType and Arrow types - Support for all common Flink types including primitives, temporal, and complex types - Comprehensive unit tests for type conversion

Copilot

Pull request overview

This PR introduces foundational type conversion utilities for Flink-Arrow integration as part 1 of a 3-part implementation series. It adds FlinkArrowUtils with methods to convert Flink LogicalTypes to Arrow types, supporting primitives, temporal types, and complex structures (arrays, maps, rows).

Changes:

Added FlinkArrowUtils class with one-way conversion from Flink LogicalType to Arrow types
Added comprehensive unit tests covering all supported type conversions
Added Arrow and Flink table dependencies to support the conversion utilities

Reviewed changes

Copilot reviewed 3 out of 4 changed files in this pull request and generated 2 comments.

File	Description
auron-flink-extension/auron-flink-runtime/src/main/java/org/apache/auron/flink/arrow/FlinkArrowUtils.java	Core utility class implementing Flink-to-Arrow type conversion with methods for types, fields, and schemas, plus a shared ROOT_ALLOCATOR for Arrow memory management
auron-flink-extension/auron-flink-runtime/src/test/java/org/apache/auron/flink/arrow/FlinkArrowUtilsTest.java	Comprehensive test suite covering basic types, complex types (arrays, rows, maps), temporal types, and error handling
auron-flink-extension/auron-flink-runtime/pom.xml	Added Arrow dependencies (arrow-c-data, arrow-memory-unsafe, arrow-vector) and test dependencies (junit-jupiter-api)
.gitignore	Added LSP-related files (*.prefs) to ignore list

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

...xtension/auron-flink-runtime/src/main/java/org/apache/auron/flink/arrow/FlinkArrowUtils.java

…ache/auron/flink/arrow/FlinkArrowUtils.java Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

ShreyeshArangath

Overall LGTM, just a couple minor comments

Also, from the description, should this not be built with flink instead of Spark?

./auron-build.sh --pre --sparkver 3.5 --scalaver 2.12 -DskipBuildNative

...xtension/auron-flink-runtime/src/main/java/org/apache/auron/flink/arrow/FlinkArrowUtils.java

- Mark class as final to prevent subclassing - Add null check for logicalType parameter to throw IllegalArgumentException instead of NPE

x-tong · 2026-01-27T18:13:19Z

Overall LGTM, just a couple minor comments

Also, from the description, should this not be built with flink instead of Spark?

./auron-build.sh --pre --sparkver 3.5 --scalaver 2.12 -DskipBuildNative

Thank you for your review, yes, that's the way it should be done, I have made the changes.

Copilot

Pull request overview

Copilot reviewed 3 out of 4 changed files in this pull request and generated 2 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

...sion/auron-flink-runtime/src/test/java/org/apache/auron/flink/arrow/FlinkArrowUtilsTest.java

- Add test case for NullType to Arrow conversion - Add assertion to verify Decimal bitWidth is 128

ShreyeshArangath

LGTM, thanks for adding this

Tartarus0zm

Thanks for your contribution!
Overall LGTM, just a couple minor comments.

...xtension/auron-flink-runtime/src/main/java/org/apache/auron/flink/arrow/FlinkArrowUtils.java

- Remove redundant nullable parameter from toArrowField method - Use logicalType.isNullable() to determine field nullability - Improve TimeType conversion with precision-based Arrow type selection - Update tests to reflect API changes

…lZonedTimestampType Reference Flink's ArrowUtils implementation to select appropriate Arrow TimeUnit based on timestamp precision: - precision 0: SECOND - precision 1-3: MILLISECOND - precision 4-6: MICROSECOND - precision 7+: NANOSECOND

...xtension/auron-flink-runtime/src/main/java/org/apache/auron/flink/arrow/FlinkArrowUtils.java

…mplementation

Tartarus0zm

thanks for your contribute!
LGTM

Tartarus0zm · 2026-02-04T08:16:06Z

@x-tong thanks for your contribute! merged!

x-tong mentioned this pull request Jan 26, 2026

[AURON #1850] Introduce Flink RowData to Arrow conversion #1930

Open

5 tasks

github-actions bot added infra build flink labels Jan 26, 2026

cxzl25 requested a review from Copilot January 26, 2026 12:34

Copilot started reviewing on behalf of cxzl25 January 26, 2026 12:35 View session

Copilot AI reviewed Jan 26, 2026

View reviewed changes

...xtension/auron-flink-runtime/src/main/java/org/apache/auron/flink/arrow/FlinkArrowUtils.java Outdated Show resolved Hide resolved

...xtension/auron-flink-runtime/src/main/java/org/apache/auron/flink/arrow/FlinkArrowUtils.java Outdated Show resolved Hide resolved

x-tong and others added 2 commits January 26, 2026 22:11

Update auron-flink-extension/auron-flink-runtime/src/main/java/org/ap…

7de5c88

…ache/auron/flink/arrow/FlinkArrowUtils.java Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Fix code formatting

666fc85

ShreyeshArangath reviewed Jan 27, 2026

View reviewed changes

...xtension/auron-flink-runtime/src/main/java/org/apache/auron/flink/arrow/FlinkArrowUtils.java Outdated Show resolved Hide resolved

...xtension/auron-flink-runtime/src/main/java/org/apache/auron/flink/arrow/FlinkArrowUtils.java Show resolved Hide resolved

Address PR review comments for FlinkArrowUtils

9f6cfc6

- Mark class as final to prevent subclassing - Add null check for logicalType parameter to throw IllegalArgumentException instead of NPE

x-tong force-pushed the feature/issue-1850-flink-arrow-part1 branch from e6be0b8 to 9f6cfc6 Compare January 27, 2026 18:09

cxzl25 requested a review from Copilot January 28, 2026 05:59

Copilot started reviewing on behalf of cxzl25 January 28, 2026 05:59 View session

Copilot AI reviewed Jan 28, 2026

View reviewed changes

...sion/auron-flink-runtime/src/test/java/org/apache/auron/flink/arrow/FlinkArrowUtilsTest.java Show resolved Hide resolved

...sion/auron-flink-runtime/src/test/java/org/apache/auron/flink/arrow/FlinkArrowUtilsTest.java Show resolved Hide resolved

Add test coverage for NullType and Decimal bitWidth

54875fe

- Add test case for NullType to Arrow conversion - Add assertion to verify Decimal bitWidth is 128

x-tong requested a review from ShreyeshArangath January 28, 2026 07:56

ShreyeshArangath approved these changes Jan 28, 2026

View reviewed changes

richox approved these changes Jan 29, 2026

View reviewed changes

Tartarus0zm requested changes Jan 30, 2026

View reviewed changes

...xtension/auron-flink-runtime/src/main/java/org/apache/auron/flink/arrow/FlinkArrowUtils.java Outdated Show resolved Hide resolved

x-tong and others added 2 commits January 30, 2026 17:55

Tartarus0zm reviewed Feb 2, 2026

View reviewed changes

...xtension/auron-flink-runtime/src/main/java/org/apache/auron/flink/arrow/FlinkArrowUtils.java Outdated Show resolved Hide resolved

Use null timezone for LocalZonedTimestampType to align with Flink's i…

1e4590b

…mplementation

Tartarus0zm approved these changes Feb 4, 2026

View reviewed changes

Tartarus0zm merged commit a9adc70 into apache:master Feb 4, 2026
187 of 188 checks passed

Conversation

x-tong commented Jan 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Follow-up PRs

Test plan

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

ShreyeshArangath left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

x-tong commented Jan 27, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

ShreyeshArangath left a comment

Choose a reason for hiding this comment

Uh oh!

Tartarus0zm left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Tartarus0zm left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Tartarus0zm commented Feb 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

x-tong commented Jan 26, 2026 •

edited

Loading