Repository navigation
Hangup when build with large files #75
Description
I am building a large project with many sub-projects (the file structures are complex) and the habitus hangup in long period.
2017/07/25 02:14:33 ▶ Using '/habitus/build.yml' as build file
2017/07/25 02:14:33 ▶ Collecting artifact information
2017/07/25 02:14:33 ▶ Building 1 steps
2017/07/25 02:14:33 ▶ Step 1 - build-db-migrator, image-name = 'test:test'
2017/07/25 02:14:33 ▶ Step 1 - Build for test:test
2017/07/25 02:14:33 ▶ Step 1 - Building test:test from context '/habitus'
2017/07/25 02:14:33 ▶ Step 1 - Parsing and converting 'db-migrator/Dockerfile'
2017/07/25 02:14:33 ▶ Step 1 - Writing the new Dockerfile into '/habitus/db-migrator/Dockerfile.generated'
2017/07/25 02:14:33 ▶ Step 1 - Building the test:test image from /habitus/db-migrator/Dockerfile.generated
# The progress never update
My build.yml is simple for test:
build:
version: 2016-03-14
steps:
build-db-migrator:
name: test:test
dockerfile: db-migrator/Dockerfile
I run the habitus in my local image habitus:runner built based on the latest source code.
And run the script to start building:
readonly _ROOT_DIR="$(cd -P -- "$(dirname -- "$0")" && pwd -P)"
readonly WORKDIR="/habitus"
docker run -it --rm \
--workdir $WORKDIR \
-v /var/run/docker.sock:/var/run/docker.sock \
-v "$_ROOT_DIR":$WORKDIR \
habitus:runner \
habitus -env build_version=testI have tried to build a simple repo with the same docker image and shell scripts, it works.
I noticed that the building get slower when I generated a 1GB file with truncate -s 1G block in the work directory. It will get back to normal when set .dockerignore to ignore the block file.
Although I have ignored many things in the large project via .dockerignore, it still hang.
How can I debug the habitus server?
Activity
adoyle-h commentedon Jul 25, 2017
I think it is due to some large files still omitted to ignore.
Is there any method to help detect the problem? Maybe
Sending build context to Docker daemon XXX MBmessage is helpful.foldingbeauty commentedon Jul 27, 2017
Hi @adoyle-h. Hi need to investigate if I get the the size of the building context from the docker deamon API. Will put this on my todo list. Thanks for reaching out.
mumoshu commentedon Mar 8, 2018
For anyone interested: The docker-build implementation seems to be here: https://github.com/docker/cli/blob/82a80858857aeeb37008bd7d2f4afcda3d39c966/cli/command/image/build.go#L333
mumoshu commentedon Mar 13, 2018
Okay, so it turns out that habitus relies on go-dockerclient and passes build context as a directory path rather than
io.Readerwhich is required in order to calculate the progress.Habitus: https://github.com/cloud66/habitus/blob/5d45985f7af33d1a6bcbb5c782f80fcf39b829da/build/builder.go#L267
go-dockerclient: https://github.com/fsouza/go-dockerclient/blob/cb0ae363fa010610c248eb6a397aa72345b4b406/image.go#L489
Official docker client: https://github.com/moby/moby/blob/3a633a712c8bbb863fe7e57ec132dd87a9c4eff7/client/image_build.go#L21
Fortunately, we can switch habitus to pass an
InputStream. The input stream could be created from a tar stream and can we wrapped with a progress reader to achieve what we want.