Use data source label in distributed HeadNode factory #15554
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
vepadulano
force-pushed
the
distrdf-headnode-from-ds-label
branch
from
06cebe3
to
20966e8
Compare
Test Results 12 files 12 suites 2d 21h 39m 15s ⏱️ For more details on these failures, see this check. Results for commit 5717ec1. ♻️ This comment has been updated with latest results. |
Introduce a new method to get a label for the data source that the current RDataFrame is processing. There are three main types: * The dataframe will process a TTree dataset * The dataframe will process an empty dataset * The dataframe will process data from an RDataSource The function returns a label with the suffix "DS" also for the first two cases, to be aligned as much as possible with the RDataSource infrastructure.
Make use of the new function to get the data source label in distributed RDataFrame to create the headnode of the Python computation graph. This also avoids extra parsing in the factory function which includes opening the first input file once more to distinguish between TTree or RNTuple input (in case the first input argument is a string).
vepadulano
force-pushed
the
distrdf-headnode-from-ds-label
branch
from
20966e8
to
5717ec1
Compare
martamaja10
approved these changes
May 28, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Introduce a new method to get a label for the data source that the current RDataFrame is processing. There are three major types:
The function returns a label with the suffix "DS" also for the first two cases, to be aligned as much as possible with the RDataSource infrastructure.
Make use of this function in distributed RDataFrame to create the headnode of the Python computation graph. This also avoids extra parsing in the factory function which includes opening the first input file once more to distinguish between TTree or RNTuple input (in case the first input argument is a string).