Skip to content

Hyrax 2294 - #1441

Merged
kyang2014 merged 23 commits into
masterfrom
hyrax-2294
Oct 14, 2026
Merged

kyang2014 merged 23 commits into
masterfrom
hyrax-2294

Conversation

@kyang2014

@kyang2014 kyang2014 commented Oct 12, 2026 •

Copy link
Copy Markdown
Collaborator

Description

Reference ticket: HYRAX-2294

Implement a way to switch from buffer chunk to super chunk based on the number of chunks and the data transfer size.
Highlights:

  1. An experimental formula based on the performance measurements with real NASA files is used to calculate the estimated data transfer time based on the super chunk and the buffer chunk.
  2. Given the consideration of system load in an operational environment, the parallelism is downplayed when calculating the estimated data transfer time.
  3. The time to build the buffer chunk queue and to obtain the super chunk information is reduced to minimal due to the reuse of the built buffer chunk queue.
  4. The parameters that are used to calculate the estimated data transfer time are configurable by using the BES keys.
  5. The algorithm is tested with real NASA files and the software correctly picks up the right decision for big files like PACE and TEMPO.

Tasks

  • Ticket exists and is linked in title
  • No TODOs added

…mrpparray class members to avoid the duplication of calculating the buffer chunk size twice.
…have to calculate the buffer size before hand to determine if we need to switch to the buffer chunk.
…nk and buffer chunk queues only once and also add a method to clean up the queue after a variable is done.
…with the deflate filter and the fletcher32 filter is applied at last.
…unk. Also make several parameters as BES keys so that they can be changed as configuration parameters.
@kyang2014
kyang2014 marked this pull request as draft October 12, 2026 21:14
@kyang2014
kyang2014 marked this pull request as ready for review October 13, 2026 20:37

@jgallagher59701 jgallagher59701 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good. Approved. There are few sonarcloud issues that you might look at, but some are too picky. For example, "complexity is 27 instead of 25 or less". I'd ignore that one. Your call on those issues.

@sonarqubecloud

Copy link
Copy Markdown

@sonarqubecloud

Copy link
Copy Markdown

@sonarqubecloud

Copy link
Copy Markdown

@kyang2014
kyang2014 merged commit 55a0df4 into master Oct 14, 2026
5 checks passed
@kyang2014
kyang2014 deleted the hyrax-2294 branch October 14, 2026 16:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants