SOCI-indexed images cut SageMaker job startup, unless your code reads the whole image at once

Before your training job on Amazon SageMaker AI (AWS’s service for training and running models) runs a single line of your code, it can spend minutes downloading a container image (a prepackaged bundle of software holding the libraries and environment the job runs in). On the AWS blog, AWS says that for deep learning images over 10 GB this download adds minutes to every job, and you pay for it as job time. The fix AWS has just announced is a SOCI index (Seekable OCI, a lazy-loading technique: fetch only the files needed to start, and pull the rest in the background while your code runs). The index only pays off if your code does not read most of the image at startup, and you can check that with two runs of the same job.

What AWS says, and what it measured

According to AWS, a SOCI index is metadata (a map of where every file sits inside the image) stored in Amazon ECR (AWS’s container image registry) as an artifact linked to your original image, so the image digest (its unique identifier) and any existing signatures remain valid. If an image has no index, or the index cannot be read, the job falls back to a full download and still runs.

The figures below are AWS’s own measurements, on compressed images from 1.8 to 35.4 GB, taken as medians of repeated runs. The steps in the next section come from AWS’s guidance, and this article has not rerun them:

• 1.8 GB image: standard download 48 seconds, lazy loading 3–11 seconds.
• 13.2 GB image: 3 minutes 42 seconds versus about 6 seconds.
• 35.4 GB image: 7 minutes 17 seconds versus 3–11 seconds.
• On each test training job (ResNet-50 image classification and fine-tuning of a small language model), the first epoch (one pass over the whole training dataset) ran 3–16 seconds longer. Net of that, AWS reports each job finished 1–6 minutes sooner.

AWS also says the improvement grows with image size, and images smaller than roughly 4 GB see relatively small improvements. The point AWS makes itself, and the core of this article: if your entry point (the first command the job runs) reads most of the image before training starts, for example by scanning, checksumming (computing an integrity code for files) or preloading its contents, the first epoch waits for roughly the same bytes a full pull would have fetched, and the net improvement is small.

Try it on one of your jobs

Start with a job that is not critical.

Input: a SageMaker job that uses an image of roughly 4 GB or more and runs repeatedly (AWS says hyperparameter tuning sweeps and short experiments are where image download weighs most). The image must be in an Amazon ECR private repository. AWS says images in other registries, or pulled through a private Docker registry with TrainingRepositoryAccessMode set to Vpc, are not eligible for lazy loading.

1. Run the job as it is today and record two numbers: the time from image pull to container ready, and the total job time. This is your baseline.
2. Pick one of two routes. If you use a Deep Learning Container (a prebuilt AWS image for PyTorch, TensorFlow, vLLM or SGLang), change the image tag (the part after the colon in the image name, which picks the version to pull) to the variant ending in -soci, for example pytorch-training:2.9-gpu-py312-cu130-ubuntu22.04-sagemaker-v1-soci. If you use your own image, create the index on a Linux host with containerd (software that runs containers), the SOCI snapshotter (the containerd component that loads images on demand) and nerdctl (a command-line tool for containerd): nerdctl pull, then nerdctl image convert --soci to a new tag, then nerdctl push. If your image is built on a DLC base image, you must index the final image, because the base image’s index does not carry down.
3. Check that the new tag really has an index. AWS gives this command; the expected output is a list of manifests (files describing the parts of an image) that carry the SOCI index artifactType:

aws ecr batch-get-image --repository-name "$REPOSITORY_NAME" \
  --image-ids imageTag="$SOCI_IMAGE_TAG" \
  --accepted-media-types "application/vnd.oci.image.index.v1+json" \
  --query 'images[0].imageManifest' --output text \
  | jq '.manifests[] | {mediaType, artifactType}'

4. Open the job’s entry point and setup scripts and look for code that calls the Docker CLI, reads /var/run/docker.sock, or derives identifiers from cgroup paths. AWS warns that SOCI-indexed images run on a containerd-based runtime, so anything that relies on Docker may break. Fix it before you run.
5. Rerun exactly the same job with only the image tag changed, and record the same two numbers plus the first-epoch time.

Reading the result, and what to do when it fails

You are done when the wait for the container to become ready drops from minutes to seconds and total job time is below your baseline, even if the first epoch is a few seconds slower. If the first epoch is much slower than AWS’s 3–16 seconds, check whether your entry point reads whole library directories or preloads data from the image. That is exactly the case where AWS warns the net benefit is small, so keep the old pull or drop the preloading step.

If download time did not change, check in this order. First, if the image is not in an Amazon ECR private repository, or the job sets TrainingRepositoryAccessMode to Vpc, the image is not eligible and changing the tag will not help. Next, ask whether the image is smaller than roughly 4 GB, where AWS says the improvement is small. Only then go back to step 3, because the tag may point at an image with no index. When an image has no index or the index cannot be read, the job does a full download as usual, so nothing breaks and nothing is gained. If you only want to test locally before submitting a job, AWS points to a Finch (AWS’s open-source container tool) lazy-loading walkthrough in the resources section of the original post.

Sources

1. AWS Machine Learning Blog, “Announcing SOCI-indexed images for faster SageMaker AI jobs” — source of the figures, the command and the warnings above.