Repository navigation
Solr backups - #1418
Open
amywieliczka wants to merge 24 commits into
Open
Solr backups#1418amywieliczka wants to merge 24 commits into
amywieliczka wants to merge 24 commits into
Conversation
amywieliczka
force-pushed
the
solr-backups
branch
2 times, most recently
from
September 23, 2026 18:41
60ec770 to
963f556
Compare
amywieliczka
commented
Sep 23, 2026
amywieliczka
commented
Sep 23, 2026
amywieliczka
force-pushed
the
solr-backups
branch
from
September 24, 2026 17:59
963f556 to
87c6784
Compare
barbarahui
approved these changes
Sep 25, 2026
barbarahui
left a comment
barbarahui
left a comment
Contributor
There was a problem hiding this comment.
The README is great, very clear!
Phew, it took me a while to wrap my head around all of this. Your README answered all my "huh?" questions though.
|
|
||
| 1. Eventbridge Scheduler > Run Task > One-Off ECS Task with security group permissions allowing requests to http://solr-leader:8983/solr/arclight/replication?command=backup API endpoint. The ArcLight image already has this security group...we could add a leader-backup script to that image and override the entrypoint command? Seems odd that the ArcLight image would have the solr backup script, though. We could also put the solr backup script in the solr image and override the entrypoint on the solr image so that solr never starts, and use that container to query the Solr leader's replication API. There must already be a security group allowing arbitrary Solr containers to query the Solr leader's replication API (perhaps auto-magically already handled by the ECS App Mesh). | ||
| 2. Cron sidecar container defined in the task definition with network permissions to hit http://solr-leader:8983/solr/arclight/replication? endpoint - in awsvpc mode (which we already use), there are no further special permissions necessary - all containers in a single task definition have the same `localhost` handled by ECS magic. | ||
| 3. (Currently Implemented) Running cron on the Solr container itself. Involves some trickiness regarding starting the container as root so we can start cron as root and then dropping into the Solr user to run Solr. Also some trickiness regarding backup script output tailed into the solr logs themselves. But also this kind of seems like the right place for it? This was definitely the easiest to develop, since I can run this setup locally in docker with minio, and it doesn't depend on any AWS or ECS features. Typically, running cron on a container doesn't make sense, but in this specific case, since we only ever have a maximum of 1 leader and Solr's lockfile forces us to fully spin it down before starting up a new leader, it is also safe - there's no race cases between crons on different containers. |
Contributor
There was a problem hiding this comment.
I do like the fact that this doesn't depend on any AWS features. Especially since you've already worked out the tricky bits, this seems like a fine solution to me. Thanks for laying out the other options -- if we run into problems with this approach then we can always try another.
amywieliczka
force-pushed
the
solr-backups
branch
from
October 6, 2026 18:54
0d0f1f9 to
1c7c418
Compare
amywieliczka
marked this pull request as ready for review
October 6, 2026 19:14
bibliotechy
reviewed
Oct 6, 2026
bibliotechy
left a comment
bibliotechy
left a comment
Contributor
There was a problem hiding this comment.
Looks good. A few minor things to verify and maybe update, but overall, I'm excited to see this merged!
bibliotechy
approved these changes
Oct 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is finally "ready for review" but we should be careful about actually merging this in. Rough order of operations - starting in on experimentation with cinco-stage cluster.