Conversation
The table format commit hook fires for delta commits and clustering but not for compaction or log compaction, so a pluggable table format never learns that base files were rewritten. Adds the table format completion action to the compaction and log compaction timeline transitions and invokes the hook from CompactHelpers. The compaction transitions now also return the completed instant carrying its completion time, matching the other transitions, since the completion action consumes it.
…ts on merge-on-read tables Adds hoodie.write.updates.as.deletes.and.inserts: when enabled on a MOR table, an upsert rewrites each tagged update into a positional delete to the record's current file group plus an insert of the new version routed by the insert partitioner, so log files carry only positional deletes and every live record stays in a base file. This makes the storage layout representable exactly by table formats based on deletion vectors.
…rge-on-read instant The merge-on-read rollback executor reverted the instant only in the native timeline, so a pluggable table format never reverted its own metadata and was left referencing files the rollback then deleted. Hoists the copy-on-write executor's un-publish handling, which already reverts the table format and handles instants still completed in the native timeline, into the base rollback executor and uses it for both table types.
…en routing updates as deletes and inserts Under hoodie.write.updates.as.deletes.and.inserts a key moves file groups on every update while older file groups keep a tombstoned physical copy, so a base-file-scanning index tags an incoming record once per remaining copy and a repeated update of the same key fans out into duplicates. Reduce tagged records by key keeping the copy with the latest location before decomposing, so exactly one positional delete and one insert are produced per key.
…rt write mode Small-file handling routes merge-on-read updates through a merge handle and assigns inserts into existing file groups, which silently bypasses the positional-delete log path the write mode requires. With the mode enabled the delta commit executors always append updates and the upsert partitioners stop offering small files, so inserts land in base files of new file groups.
…e Spark write path Exercises decomposition, repeated updates of the same keys, and the small-file routing gates through SparkRDDWriteClient on a merge-on-read table, asserting one positional delete and one insert per key per round and delete-only log files.
…insert write mode The decomposed delete unconditionally tombstones the record's current version, so under event-time ordering a late-arriving update with a lower ordering value would incorrectly win; the write path now rejects the mode unless the table uses commit-time ordering. Log compaction scheduling is also rejected since it rewrites delete blocks and breaks the delete-only, positionally consistent log layout the mode maintains.
…iew; persist pluggable table format settings A table shared with another table format can hold data files with foreign names; they belong to no file group and made buildFileGroups fail on a null file id. TableBuilder.set now keeps hoodie.table.format.* keys so a pluggable table format receives its settings from the first commit on.
…ions in the update-as-delete-insert write mode
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #20167 +/- ##
============================================
+ Coverage 78.31% 80.42% +2.10%
- Complexity 33929 34935 +1006
============================================
Files 2544 2547 +3
Lines 141886 143034 +1148
Branches 17213 18728 +1515
============================================
+ Hits 111125 115041 +3916
+ Misses 23056 20072 -2984
- Partials 7705 7921 +216
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
2 of 3 tasks
Collaborator
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Describe the issue this Pull Request addresses
issue: #16073, #20175
Opened as a draft to capture the current state of the work. The design is written up in RFC-114 (#20181; number claimed in #20174).
An external table format (RFC-93) whose storage model is immutable data files plus positional delete files can only represent a merge-on-read table exactly if every live record sits in a base file and log files carry positional deletes alone. Hudi's upsert appends the new version of an updated record to the current file group's log, which such a format cannot express between compactions.
Summary and Changelog
hoodie.write.updates.as.deletes.and.inserts(default false). On a merge-on-read table, an upsert routes each tagged update as a positional delete against the record's current file group plus an untagged insert that the insert partitioner routes to a fresh file slice. Log files then hold positional deletes only.HoodieTableFormatis now notified on compaction completion and on merge-on-read rollback, so an external table format plugin can drop the positional deletes of rewritten or rolled-back base files. Table format settings persist in the table config, and the file-system view ignores data files Hudi did not name. The view change is open for discussion: it breaks Apache XTable's Hudi conversion target, which registers source-named files as Hudi base files; RFC-114 records the better shape (hide foreign files through the format's own metadata).TestHoodieJavaClientUpdatesAsDeletesAndInserts(decomposition, repeated updates of the same keys, default behavior unchanged, compaction after decomposed updates, the rejections),TestSparkUpdatesAsDeletesAndInsertsthrough the Spark write client, andTestHoodieJavaClientMergeOnReadForTestFormatfor the table-format hooks.Impact
No behavior change unless the new config is enabled. Pluggable table format implementations now receive compaction and rollback notifications.
Risk Level
low: the write mode is opt-in and guarded; the table-format notifications are no-ops for the native format.
Documentation Update
The new config carries its description in
HoodieWriteConfig. A website page will be needed once the feature is finalized.Contributor's checklist