54

I recently switched to multi-stage docker builds, and it doesn't appear that there's any caching on intermediate builds. I'm not sure if this is a docker limitation, something which just isn't available or whether I'm doing something wrong.

I am pulling down the final build and doing a --cache-from at the start of the new build, but it always runs the full build.

Nestor Sokil
  • 2,162
  • 12
  • 28
Matthew Goslett
  • 1,127
  • 1
  • 7
  • 6
  • 1
    Please provide some output or logs. Why do you think there is no caching? – Nestor Sokil Oct 04 '18 at 12:20
  • 3
    @NestorSokil The intermediate stages run irrespective of any changes being made which would affect those docker layers. The "hello world" at https://docs.docker.com/develop/develop-images/multistage-build/#name-your-build-stages even does the same thing. – Matthew Goslett Oct 04 '18 at 12:40
  • The layer from the earlier stages are not in the final stage, so using `--cache-from` with that won't help the earlier stages. Saving the previous stages can work, but it appears to only work and match the layers if you keep building using the same computer/filesystem, so doesn't actually help CI builds :( – Aaron Jul 06 '19 at 01:53

3 Answers3

57

This appears to be a limitation of docker itself and is described under this issue - https://github.com/moby/moby/issues/34715

The workaround is to:

  1. Build the intermediate stages with a --target
  2. Push the intermediate images to the registry
  3. Build the final image with a --target and use multiple --cache-from paths, listing all the intermediate images and the final image
  4. Push the final image to the registry
  5. For subsequent builds, pull the intermediate + final images down from the registry first
Matthew Goslett
  • 1,127
  • 1
  • 7
  • 6
  • 1
    It uses the first match, so list the list the final image first, then intermediates. This isn't working in CI situations where you a pull to a different machine from the push, some bugs in Docker --cache-from handling. https://github.com/moby/moby/issues/34715 – Aaron Jul 06 '19 at 01:44
  • 5
    How can you use this with `docker-compose`? I have a cached mid-stage image which i `docker load` and set `cache_from` to the same image in `docker-compose.yml` but it rebuilds all stages anyway. – thisismydesign Jan 30 '20 at 16:55
22

Since the previous answer was posted, there is now a solution using the BuildKit backend:

This involves passing the argument --build-arg BUILDKIT_INLINE_CACHE=1 to your docker build command. You will also need to ensure BuildKit is being used by setting the environment variable DOCKER_BUILDKIT=1 (on Linux; I think BuildKit might be the default backend on Windows when using recent versions of Docker Desktop). A complete command line solution for CI might look something like:

export DOCKER_BUILDKIT=1
    
# Use cache from remote repository, tag as latest, keep cache metadata
docker build -t yourname/yourapp:latest \
      --cache-from yourname/yourapp:latest \
      --build-arg BUILDKIT_INLINE_CACHE=1 .
    
# Push new build up to remote repository replacing latest
docker push yourname/yourapp:latest

Some of the other commenters are asking about docker-compose. It works for this too, although you need to additionally specify the environment variable COMPOSE_DOCKER_CLI_BUILD=1 to ensure docker-compose uses the docker CLI (with BuildKit thanks to DOCKER_BUILDKIT=1) and then you can set BUILDKIT_INLINE_CACHE: 1 in the args: section of the build: section of your YAML file to ensure the required --build-arg is set.

For reference:

malat
  • 12,152
  • 13
  • 89
  • 158
Eric Greveson
  • 221
  • 2
  • 2
6

I'd like to add another important point to the answer

Wrong answer

--build-arg BUILDKIT_INLINE_CACHE=1 caches only the last layer, and works only in cases when nothing is changed in the whole Dockerfile

So, to enable the caching of layers for the whole build, this argument should be replaced by --cache-to type=inline,mode=max. See the documentation

Correct answer

The documentation above as on 2023-03-28 states:

When generating a cache output, the --cache-to argument accepts a mode option for defining which layers to include in the exported cache. This is supported by all cache backends except for the inline cache.

That means, for the cached intermediate states with all layers one needs to use registry cache backend.

I plan to use the same image:tag as pushed at the build time, but with suffix -buildcache. So, --cache-from type=registry,ref=org/image:tag-buildcache,mode=max --cache-to type=registry,ref=org/image:tag-buildcache,mode=max

Felixoid
  • 431
  • 4
  • 10
  • Actually, you should switch to the `registry` cache-backend instead of the `inline` cache-backend to cache all the layers. From [the documentation](https://docs.docker.com/build/cache/backends/registry/): - Allows for separating the cache and resulting image artifacts so that you can distribute your final image without the cache inside. - It can efficiently cache multi-stage builds in max mode, instead of only the final stage. - It works with other exporters for more flexibility, instead of only the image exporter. – dom Mar 27 '23 at 09:47
  • You need a separate image or tag to store the cache then. That's 100% opposite to the purpose, no? – Felixoid Mar 27 '23 at 10:39
  • I think having the cache inside the container that launches your app isn't really a requirement. Being able to cache all layers in a multi-stage build is much more important to me. It is necessary to use a separate image anyway if you want to cache all stages of your multi-stage build. It seems like they're working on Azure blob storage, Amazon S3, and Github actions as viable backends but these are either beta or unreleased. – dom Mar 27 '23 at 12:40
  • 1
    Ok, either something changed recently, or I've missed the important note from the [documentation](https://docs.docker.com/build/cache/backends/#cache-mode) that I've provided. Specifically, `mode is supported by all cache backends except for the inline cache.` – Felixoid Mar 28 '23 at 21:33