Skip to content

[Fix] Isolate FileReaderManager reader I/O with per-file locks - #18788

Merged
JackieTien97 merged 9 commits into
apache:masterfrom
JackieTien97:fix/file-reader-manager-per-file-locks-cloud
Oct 9, 2026
Merged

JackieTien97 merged 9 commits into
apache:masterfrom
JackieTien97:fix/file-reader-manager-per-file-locks-cloud

Conversation

@JackieTien97

@JackieTien97 JackieTien97 commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

Description

A slow reader open or close currently holds the FileReaderManager monitor, stalling unrelated files. Reference registration can also hold a DataRegion read lock while waiting for reader I/O, delaying writes and new queries in that region. This change serializes reader I/O per file and keeps reference registration in a short registry critical section, so it does not wait for an opening or closing reader.

Reader lifecycle and locking

  • Keep separate closed/unclosed slots per internal TsFileID and a separate path namespace for external files. Operation pins include entry-lock waiters and prevent premature entry replacement.
  • Protect reference counts with the registry lock independently of the entry I/O monitor. A last release rechecks references after waiting for I/O, so a newly registered query retains its reader. Releases select the same closed/unclosed slot as registration; an empty unclosed slot never consumes another query's closed reference.
  • Discard a reader from the cache after any close attempt, including failures. Log close errors and propagate them from explicit close operations, while subsequent queries can reopen a healthy file. This preserves the original best-effort close behavior without retaining poisoned slots or abandoned unclosed readers.
  • Preserve serialized global test cleanup and its final pin drain. Nested finally blocks reopen admission after snapshot or reclamation failures; reference registration remains blocked until cleanup completes. Callers must stop reader use before global cleanup.
  • Base reader-count warnings on the atomic increment result, aggregate close errors without mutating reader-supplied exceptions, and log both the reader state and zero-reference close reason with localized FileMonitor messages.

Reference ownership

Remove failed registrations from fragment release sets to prevent a second read-lock release. The lightweight fragment context records the closed/unclosed state at registration and rolls back partial initialization, so sealing a file cannot redirect its release to another query's slot. TsFileLock preserves read/write exclusion when a waiting thread is interrupted and restores interrupt status only after acquisition.

Deleted-file releases and initialization rollback run after releasing the DataRegion read lock. Initialization-local pending references retain their file locks until cleanup, including references from a batch that fails before publishing its lease. Cleanup attempts every queued release and preserves the original initialization exception. Region scans also release the region lock when acquisition consumes the remaining time slice.

Validation

Validated on the PR branch with Amazon Corretto 17.0.5, Maven 3.9.9, and cached TsFile 2.4.1-260915-SNAPSHOT artifacts. Maven ran offline with Develocity disabled; builds, test data, and temporary files stayed on the external test volume.

  • Clean dependency-reactor regression: 128 tests, 0 failures, 0 errors, 1 skipped, across 17 classes. The skip is the existing performance test in TimeSeriesMetadataCacheTest.
  • FileReaderManagerConcurrencyTest: 23/23, including real DataRegion read/write locks under gated open and close, last-release/new-reference interleaving, close-failure recovery for internal and external readers, unclosed-slot reclamation, independent exception aggregation, cleanup gate restoration after injected errors, and isolation of a new closed reader from a late unclosed release after test cleanup.
  • FileReaderReferenceLifecycleTest: 4/4; TsFileLockTest: 2/2.
  • RegionQueryInitializationLockTest: 6/6, covering shared queries, batches, both region-scan modes, and failed initialization while reader close is blocked. RegionScanTimeSliceTest: 1/1, covering the exhausted-time-slice return after a real region lock acquisition.
  • mvn clean test-compile -DskipTests: 53 modules successful.
  • mvn clean test-compile -P with-zh-locale -DskipTests: 53 modules successful.
  • mvn spotless:apply -pl iotdb-core/datanode, full-reactor formatting checks in both compile runs, and git diff --check: passed.

Distributed-cluster and object-storage end-to-end tests were not run.

@JackieTien97
JackieTien97 marked this pull request as ready for review October 8, 2026 07:30
@JackieTien97
JackieTien97 merged commit 48233ea into apache:master Oct 9, 2026
42 of 43 checks passed
@JackieTien97
JackieTien97 deleted the fix/file-reader-manager-per-file-locks-cloud branch October 9, 2026 02:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant