Repository navigation
s3d - no more hosts available #235
Description
Testing the rclone s3d connected to sia.storage.
Files test-2000MB.bin and test-3000MB.bin were uploaded without problem.
Generating 100 x 5MB files resulted in the error message : No more hosts available.
ERROR backend.sia failed to finalize upload {"error": "failed to upload slab: shard upload failed: no more hosts available"}
WARN sdk failed to refresh hosts {"error": "failed to fetch hosts from indexer: Get \"https://sia.storage/hosts?limit=100&offset=0&sc=...\": dial tcp: lookup sia.storage on 127.0.0.11:53: server misbehaving"}
Metadata
Metadata
Assignees
Labels
No labels
Activity
Didrole commentedon Aug 6, 2026
Your logs indicate that a DNS resolution issue made the request to get a hosts list from the indexer fail.
Maybe be the DNS server you're using had an issue.
Is it always reproducible?
chris124567 commentedon Aug 7, 2026
Thank you for bringing this to our attention. I think this might have to do with failing to fetch the host list as the above commenter said. 1 failure should not be a big deal because if I remember correctly we only update the list when it succeeds and it's refreshed every 10 minutes. But there could be a bug somewhere so I will take a look and also see if I can reproduce this.
chris124567 commentedon Aug 7, 2026
I was able to reproduce this. We have an open PR in the Go SDK (which s3d depends on) that fixes this in my testing: SiaFoundation/sia-storage-go#42
I will update here when it is merged and when we release new versions. Thank you again
stealthx commentedon Aug 7, 2026
Good to see you have a fix.
I deleted Docker and started over from scratch.
I have succesfully uploaded :
test-30MB.bin
test-40MB.bin
test-50MB.bin
folder with 10 x 5MB files
When I tried with a folder containing 20 x 5 MB files, I encountered the again.
chris124567 commentedon Aug 7, 2026
We need to review and merge the PR then release a new version to fix the issue for end users. So it may be a few days
chris124567 commentedon Aug 12, 2026
This should be fixed in v0.1.2
stealthx commentedon Aug 16, 2026
S3d Version 0.1.2, Commit: 9004737
It looks like I still have No more hosts available error:
chris124567 commentedon Aug 17, 2026
Is this from uploading 100 x 5MB files again?
I couldn't reproduce this on 0.1.2 but I will try again tomorrow
chris124567 commentedon Aug 17, 2026
OK, I was able to reproduce this if I introduced a decent amount of network instability. There are a few changes I can think of for
sia-storage-goto make this less likely to happen so I will do that this week.stealthx commentedon Aug 17, 2026
Yes, my Wi-Fi speed is problematic now, but I can still upload files to Google Drive, watch YouTube 720p and browse the web.
The s3d test only involved 1 x 5 MB and 4 x 10 MB generated files. Perhaps the same problem occurs with the Android app that I posted with logs in the Discord - development yesterday.
chris124567 commentedon Aug 19, 2026
Status update: SiaFoundation/sia-storage-go#56 should fix this issue once it is merged and we bump the versions
MrSliff commentedon Aug 31, 2026
I’m seeing the same
failed to upload slab: shard upload failed: no more hosts availableissue with s3d v0.1.2 against sia.storage on a paid 5 TB plan.The local queue persists and retries, but affected upload groups repeatedly fail to finalize.
Does SiaFoundation/sia-storage-go#56 address this s3d scenario? If so, is there an expected s3d release or recommended mitigation until the dependency update lands?
ChrisSchinnerl commentedon Aug 31, 2026
@MrSliff should be fixed in v0.1.3
stealthx commentedon Aug 31, 2026
S3d Version 0.1.3 is better. For this test, my internet speed is stable 39 / 10 Mbps.
However, I did receive the
No more hostsmessage once when uploading 500 MB (100 x 5 MB).100 MB (20 x 5 MB) test ended successfully, it consumed 522 MB.
200 MB (40 x 5 MB) test ended successfully, but it consumed 1.9 GB.
400 MB (80 x 5 MB) test ended successfully with a WARN and has consumed 10.1 GB:
There is a new message regarding the GO version and the fallback to encoding/json:
chris124567 commentedon Sep 1, 2026
Thank you again for reporting this. The encoding/json warning can be ignored. Looking into this (though I am unable to reproduce at the moment) and will hopefully come up with a more permanent solution
stealthx commentedon Sep 23, 2026
I was able to upload 2.6 GB of data (23159 files) without any error messages. S3d looks solid after the updates.
The following setting was used in s3d.yml: uploadThreads: 2
chris124567 commentedon Sep 23, 2026
That's great to hear. Right now we are doing bindings so that the Rust SDK can be used in Go to reduce duplicated efforts and we have merged some additional fixes in that repo that should eliminate this entirely as we have found it can pop up in unusual and hard to reproduce ways
stealthx commentedon Oct 8, 2026
I have a new error. Could it be related to the previous error?
failed to fetch object eventsandfailed to refresh hostschris124567 commentedon Oct 8, 2026
Looking at those errors, it seems that they all say "lookup sia.storage on 127.0.0.11:53" or "read: connection reset by peer." That suggests your connection went down and that the DNS resolver is failing to do lookups as a result. Have you had any connection issues lately (or have you been running software such as dnsmasq and updating/reconfiguring it)?
stealthx commentedon Oct 8, 2026
The internet was working. There was a strange pattern to the errors. First two occurred in hours, and then suddenly by the minute.
I didn't do anything with dnsmasq. I'll see if the errors show up again.