httpreader: random access over an HTTP resource - #1953
Open
hdonnay wants to merge 5 commits into
Open
Conversation
Member
Author
|
Conflicts with #1948 |
hdonnay
force-pushed
the
hack/httpreader
branch
6 times, most recently
from
July 22, 2026 16:41
9c3ce05 to
719faf9
Compare
hdonnay
force-pushed
the
hack/httpreader
branch
2 times, most recently
from
July 27, 2026 20:06
7635a0a to
01c8352
Compare
BradLugo
self-requested a review
July 27, 2026 20:12
BradLugo
requested changes
Jul 27, 2026
BradLugo
left a comment
BradLugo
left a comment
Contributor
There was a problem hiding this comment.
I think there's enough to address for now. I'll do another deep dive in the next review
| } | ||
|
|
||
| //go:generate sh testdata/create_paxsize.sh | ||
| func TestPAXSize(t *testing.T) { |
Contributor
There was a problem hiding this comment.
Looks like this test fails without 5d4f550. Again, not really a blocker for me, mostly fyi.
Comment on lines
+155
to
+164
| for s.Scan() { | ||
| fs := strings.FieldsFunc(s.Text(), func(r rune) bool { return r == ' ' || r == '=' }) | ||
| if fs[1] == "size" { | ||
| // This is *not* the stringified octal madness of a real header. | ||
| *p, err = strconv.ParseInt(fs[2], 10, 64) | ||
| if err != nil { | ||
| return nil, parseErr("bad block at %d: weird PAX header: bad size: %v", off, err) | ||
| } | ||
| } | ||
| } |
Contributor
There was a problem hiding this comment.
I've never really looked at the pax spec before, but after briefly reading about it, it seems like we'd at least need to break after finding size so we don't hit one of the extended headers, which might cause a an error: https://pubs.opengroup.org/onlinepubs/009695399/utilities/pax.html#tag_04_100_13_03
Member
Author
There was a problem hiding this comment.
I'm not sure what you mean. This is handling the extended headers.
This package does an io.ReaderAt over an HTTP resource. It includes a novel way to determine the end of a resource for partially-compliant servers. Using wholly-compliant servers is recommended. Signed-off-by: Hank Donnay <hdonnay@redhat.com> Change-Id: I08d6671217535ac897a2ed5c9cd20aa06a6a6964
Signed-off-by: Hank Donnay <hdonnay@redhat.com> Change-Id: I162632409abf01efcec4633713c9f0d06a6a6964
Signed-off-by: Hank Donnay <hdonnay@redhat.com> Change-Id: I8bd87cb66de93dad080c1d6d8aac34a96a6a6964
Signed-off-by: Hank Donnay <hdonnay@redhat.com> Change-Id: I6c2d6cfae9f6e742056c157e38b8b6156a6a6964
This handles the "easy" case of simply proxying reads for uncompressed tar archives to range requests. Future improvements would move the "spooling" out of this package and into the `fs.FS` implementation. Signed-off-by: Hank Donnay <hdonnay@redhat.com> Change-Id: I9d200dd841954054df0b187b9fd160a56a6a6964
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This adds a package (
internal/httpreader) that implementsio.ReaderAtover HTTP requests. It incorporates a lot of tricks I know from a past life dealing with RFC7233 non- and selectively- compliant servers.Then, the
fetcherpackage gains the capability to use anhttpreader.Readerwhen it notices that a layer is uncompressed.Future work may involve the
tarfslayer doing transparent caching and being able to handle compressed layers directly.