[Real-time Table Replication X clusters][2/n] Create a Controller Job for historical segments backfill

[wirybeaver]

• Create an async controller zk jobs to deeply backfill historical segments prior to the inducted consumer watermarks. The deep copy means replicating segments to a standalone remote storage path.
• Introduced TableReplicator to fetch segment ZK metadata from the source, register a controller job, and submit 4 worker tasks (can be fine tuned in the future) that call a SegmentCopier for each segment; errors immediately trigger job updates and completed segments are batched for ZK updates.
• Added ZkBasedTableReplicationObserver and TableReplicationProgressStats to keep in-memory progress (remaining count and failed segment list) and persist job metadata to ZK (including a SEGMENTS_TO_BE_COPIED field) every 100 completions or immediately on error.
• Implemented SegmentCopier interface with RealtimeSegmentCopier (deep-copy mode) that copies segments between the same PinotFS scheme and uploads the segment URI to the destination controller; if a segment lacks a deep-store download URL the code logs the problem and records failures.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 66.35071% with 71 lines in your changes missing coverage. Please review.
✅ Project coverage is 63.15%. Comparing base (36ac509) to head (9c2eb01).

Files with missing lines	Patch %	Lines
.../helix/core/replication/RealtimeSegmentCopier.java	57.57%	23 Missing and 5 partials ⚠️
...roller/helix/core/replication/TableReplicator.java	60.41%	17 Missing and 2 partials ⚠️
...ntroller/helix/core/PinotHelixResourceManager.java	42.30%	13 Missing and 2 partials ⚠️
...oller/api/resources/PinotTableRestletResource.java	0.00%	4 Missing ⚠️
...e/replication/ZkBasedTableReplicationObserver.java	90.62%	2 Missing and 1 partial ⚠️
...ller/helix/core/replication/NoOpSegmentCopier.java	0.00%	2 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #17521      +/-   ##
============================================
- Coverage     63.18%   63.15%   -0.04%     
  Complexity     1479     1479              
============================================
  Files          3173     3179       +6     
  Lines        189917   190124     +207     
  Branches      29064    29081      +17     
============================================
+ Hits         120002   120072      +70     
- Misses        60575    60720     +145     
+ Partials       9340     9332       -8     

Flag	Coverage Δ
custom-integration1	100.00% <ø> (ø)
integration	100.00% <ø> (ø)
integration1	100.00% <ø> (ø)
integration2	0.00% <ø> (ø)
java-11	63.13% <66.35%> (-0.04%)	⬇️
java-21	63.12% <66.35%> (-0.01%)	⬇️
temurin	63.15% <66.35%> (-0.04%)	⬇️
unittests	63.15% <66.35%> (-0.04%)	⬇️
unittests1	55.52% <0.00%> (+0.02%)	⬆️
unittests2	34.05% <66.35%> (-0.02%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.