Propose OCI-packaging changes

[Silvanoc]

Description

I'd like to propose some changes in the OCI-packaging format implemented in #

Container image packaging looks like this:

---
  config:
    class:
      hideEmptyMembersBox: true
---
classDiagram
    namespace ContainerImages {
        class mai["Multi-Arch Container Image"]:::ociIndex {
            <<OCI index>>
            manifests
        }

        class x86i["x86-64 Container Image"]:::ociManifest {
            <<OCI manifest>>
            config
            layers
        }

        class armi["ARM64 Container Image"]:::ociManifest {
            <<OCI manifest>>
            config
            layers
        }    
    }

%%    namespace Compositions {
%%        class ci["Multi-Arch Composition"]:::ociIndex {
%%           <<OCI index>>
%%            subject
%%            manifests
%%        }

%%        class cx86m["x86-64 Composition"]:::ociManifest {
%%            <<OCI manifest>>
%%            subject
%%            layers
%%        }

%%        class compx86["x86-64 compose.yaml"] {
%%            <<OCI blob>>
%%        }

%%        class carmm["ARM64 Composition"]:::ociManifest {
%%            <<OCI manifest>>
%%            subject
%%            layers
%%        }

%%        class comparm["ARM64 compose.yaml"] {
%%            <<OCI blob>>
%%        }
%%    }

    namespace Tags {
        class latestIm["latest"]:::ociTag {
            <<OCI tag>>
        }

%%        class latestComp["latest-composition"]:::ociTag {
%%            <<OCI tag>>
%%        }        
    }

    classDef ociTag fill:#09f,stroke:#444,stroke-width:4px;
    classDef ociIndex fill:#f9f,stroke:#444,stroke-width:4px;
    classDef ociManifest fill:#f9a,stroke:#444,stroke-width:2px;

    latestIm --> mai : digest
    mai o-- x86i : manifest
    mai o-- armi : manifest
%%    latestComp --> ci : digest
%%    ci o-- cx86m : manifest
%%    ci o-- carmm : manifest
%%    ci --> mai : subject
%%    cx86m o-- compx86 : layer
%%    cx86m --> x86i : subject
%%    carmm o-- comparm : layer
%%    carmm --> armi : subject

Loading

---

‼️ IMPORTANT ‼️

Following proposal enables:

1. "attaching" a compose.yaml to an existing container image without modifying any of its OCI index or manifests
2. composing multiple container images copying them to a single OCI repository, will be called "bundled composition" and is similar to a phone app
3. composing multiple container images contained in other OCI repositories, will be called "loose composition" and is similar to a Helm Chart

---

Bundled Composition

This would be the case when the presence of all the needed parts needs to be guaranteed by the OCI registry. But also when "attaching" a compose.yaml to an existing container image.

It modifies the OCI-reference to the container images, but not the images themselves. Therefore it is changing the compose.yaml file.

It provides better guarantees that other container-based packages like "Helm Charts" and therefore it can be seen as the "appification" of a composition. The availability of the OCI repository guarantees the availability of the application.

Here is where the beauty of the subject jumps in:

---
  config:
    class:
      hideEmptyMembersBox: true
---
classDiagram
    namespace ContainerImages {
        class mai["Multi-Arch Container Image"]:::ociIndex {
            <<OCI index>>
            manifests
        }

        class x86i["x86-64 Container Image"]:::ociManifest {
            <<OCI manifest>>
            config
            layers
        }

        class armi["ARM64 Container Image"]:::ociManifest {
            <<OCI manifest>>
            config
            layers
        }    
    }

    namespace Compositions {
%%        class ci["Multi-Arch Composition"]:::ociIndex {
%%           <<OCI index>>
%%            subject
%%            manifests
%%        }

        class cx86m["x86-64 Composition"]:::ociManifest {
            <<OCI manifest>>
            subject
            layers
        }

        class compx86["x86-64 compose.yaml"] {
            <<OCI blob>>
        }

%%        class carmm["ARM64 Composition"]:::ociManifest {
%%            <<OCI manifest>>
%%            subject
%%            layers
%%        }

%%        class comparm["ARM64 compose.yaml"] {
%%            <<OCI blob>>
%%        }
    }

    namespace Tags {
        class latestIm["latest"]:::ociTag {
            <<OCI tag>>
        }

%%        class latestComp["latest-composition"]:::ociTag {
%%            <<OCI tag>>
%%        }        
    }

    classDef ociTag fill:#09f,stroke:#444,stroke-width:4px;
    classDef ociIndex fill:#f9f,stroke:#444,stroke-width:4px;
    classDef ociManifest fill:#f9a,stroke:#444,stroke-width:2px;

    latestIm --> mai : digest
    mai o-- x86i : manifest
    mai o-- armi : manifest
%%    latestComp --> ci : digest
%%    ci o-- cx86m : manifest
%%    ci o-- carmm : manifest
%%    ci --> mai : subject
    cx86m o-- compx86 : layer
    cx86m --> x86i : subject
%%    carmm o-- comparm : layer
%%    carmm --> armi : subject

Loading

You wouldn't even need to add a tag to the compose (though you could). The referrers API (or its fallback) would let you "discover" the composition querying for the container image.

But what if you want to provide multi-arch compositions for a multi-arch image? You simply apply the same pattern:

---
  config:
    class:
      hideEmptyMembersBox: true
---
classDiagram
    namespace ContainerImages {
        class mai["Multi-Arch Container Image"]:::ociIndex {
            <<OCI index>>
            manifests
        }

        class x86i["x86-64 Container Image"]:::ociManifest {
            <<OCI manifest>>
            config
            layers
        }

        class armi["ARM64 Container Image"]:::ociManifest {
            <<OCI manifest>>
            config
            layers
        }    
    }

    namespace Compositions {
        class ci["Multi-Arch Composition"]:::ociIndex {
           <<OCI index>>
            subject
            manifests
        }

        class cx86m["x86-64 Composition"]:::ociManifest {
            <<OCI manifest>>
            subject
            layers
        }

        class compx86["x86-64 compose.yaml"] {
            <<OCI blob>>
        }

        class carmm["ARM64 Composition"]:::ociManifest {
            <<OCI manifest>>
            subject
            layers
        }

        class comparm["ARM64 compose.yaml"] {
            <<OCI blob>>
        }
    }

    namespace Tags {
        class latestIm["latest"]:::ociTag {
            <<OCI tag>>
        }

%%        class latestComp["latest-composition"]:::ociTag {
%%            <<OCI tag>>
%%        }        
    }

    classDef ociTag fill:#09f,stroke:#444,stroke-width:4px;
    classDef ociIndex fill:#f9f,stroke:#444,stroke-width:4px;
    classDef ociManifest fill:#f9a,stroke:#444,stroke-width:2px;

    latestIm --> mai : digest
    mai o-- x86i : manifest
    mai o-- armi : manifest
%%    latestComp --> ci : digest
    ci o-- cx86m : manifest
    ci o-- carmm : manifest
    ci --> mai : subject
    cx86m o-- compx86 : layer
    cx86m --> x86i : subject
    carmm o-- comparm : layer
    carmm --> armi : subject

Loading

No matter which image reference you have (tag to the index, index digest, arch-specific manifest digest,...) you can always discover the corresponding composition.

Finally a tag can be added also to the composition, providing two different entry points:

1. only for the container image
2. for the container image with a compose.yaml providing some help to run a container using the image

---
  config:
    class:
      hideEmptyMembersBox: true
---
classDiagram
    namespace ContainerImages {
        class mai["Multi-Arch Container Image"]:::ociIndex {
            <<OCI index>>
            manifests
        }

        class x86i["x86-64 Container Image"]:::ociManifest {
            <<OCI manifest>>
            config
            layers
        }

        class armi["ARM64 Container Image"]:::ociManifest {
            <<OCI manifest>>
            config
            layers
        }    
    }

    namespace Compositions {
        class ci["Multi-Arch Composition"]:::ociIndex {
           <<OCI index>>
            subject
            manifests
        }

        class cx86m["x86-64 Composition"]:::ociManifest {
            <<OCI manifest>>
            subject
            layers
        }

        class compx86["x86-64 compose.yaml"] {
            <<OCI blob>>
        }

        class carmm["ARM64 Composition"]:::ociManifest {
            <<OCI manifest>>
            subject
            layers
        }

        class comparm["ARM64 compose.yaml"] {
            <<OCI blob>>
        }
    }

    namespace Tags {
        class latestIm["latest"]:::ociTag {
            <<OCI tag>>
        }

        class latestComp["latest-composition"]:::ociTag {
            <<OCI tag>>
        }        
    }

    classDef ociTag fill:#09f,stroke:#444,stroke-width:4px;
    classDef ociIndex fill:#f9f,stroke:#444,stroke-width:4px;
    classDef ociManifest fill:#f9a,stroke:#444,stroke-width:2px;

    latestIm --> mai : digest
    mai o-- x86i : manifest
    mai o-- armi : manifest
    latestComp --> ci : digest
    ci o-- cx86m : manifest
    ci o-- carmm : manifest
    ci --> mai : subject
    cx86m o-- compx86 : layer
    cx86m --> x86i : subject
    carmm o-- comparm : layer
    carmm --> armi : subject

Loading

Loose Composition

In this case the presence of the needed container images is not guaranteed by the OCI registry. It is equivalent to a "Helm Chart" in that sense. If an OCI repository hosting one of the required container images is deleted, the composition is broken.

It does not change the compose.yaml file in any way.

The references to the container images are not explicit in the OCI artifact, but "embedded" in the compose.yaml file and in the image-digests.yaml file (which is used to ensure the integrity of the images).

---
  config:
    class:
      hideEmptyMembersBox: true
---
classDiagram
    namespace Image1 {
        class mai["Multi-Arch Container Image"]:::ociIndex {
            <<OCI index>>
            manifests
        }

        class x86i["x86-64 Container Image"]:::ociManifest {
            <<OCI manifest>>
            config
            layers
        }

        class armi["ARM64 Container Image"]:::ociManifest {
            <<OCI manifest>>
            config
            layers
        }    

        class latestIm["latest"]:::ociTag {
            <<OCI tag>>
        }
    }

    namespace CompositionRepository {
        class ci["Multi-Arch Composition"]:::ociIndex {
           <<OCI index>>
            subject
            manifests
        }

        class cx86m["x86-64 Composition"]:::ociManifest {
            <<OCI manifest>>
            subject
            layers
        }

        class compx86["x86-64 compose.yaml"] {
            <<OCI blob>>
        }

        class idx86["x86-64 image-digests.yaml"] {
            <<OCI blob>>
        }

        class carmm["ARM64 Composition"]:::ociManifest {
            <<OCI manifest>>
            subject
            layers
        }

        class comparm["ARM64 compose.yaml"] {
            <<OCI blob>>
        }

        class idarm["ARM64 image-digests.yaml"] {
            <<OCI blob>>
        }

        class latestComp["latest"]:::ociTag {
            <<OCI tag>>
        }        
    }

    classDef ociTag fill:#09f,stroke:#444,stroke-width:4px;
    classDef ociIndex fill:#f9f,stroke:#444,stroke-width:4px;
    classDef ociManifest fill:#f9a,stroke:#444,stroke-width:2px;

    latestIm --> mai : digest
    mai o-- x86i : manifest
    mai o-- armi : manifest
    latestComp --> ci : digest
    ci o-- cx86m : manifest
    ci o-- carmm : manifest
    cx86m o-- compx86 : layer
    cx86m o-- idx86 : layer
    carmm o-- comparm : layer
    carmm o-- idarm : layer
    compx86 ..> x86i
    idx86 ..> x86i
    comparm ..> armi
    idarm ..> armi

Loading

[ndeloof]

about multi-arch: why refer by subject to platform specific images and not the multi-plaform index-list ? Doing so you don't need platform-specific compose.yaml (which would have the exact same content)

"Loose Composition" you describe is what we have when a compose file is published without the --app flag, but relying on index-list for multi-platform

"Bundled Composition" you describe allows a compose OCI artifact to be subject of an image but not a set of images, as required by a compose application. Could use a top-level index-list to group such images, but then you basically get the same solution as publish --app with subject relation reverted.

The main issue I can see with your approach is that the application tag doesn't target the actual compose OCI artifact. Then there's no way to refer to it for a consumer to use your "bundle" application.

[ndeloof]

I'm closing this issue as "not planned"

[Silvanoc]

about multi-arch: why refer by subject to platform specific images and not the multi-plaform index-list ? Doing so you don't need platform-specific compose.yaml (which would have the exact same content)

Having platform-specific compose.yaml would be optional. Ideally it's not needed. But it might be needed under certain circumstances. If all platform-specific manifest are in fact the very same, then it's only explicitly stating the supported platform without any need for duplication of the compose.yaml (what is anyway impossible with the OCI CAS).

"Loose Composition" you describe is what we have when a compose file is published without the --app flag, but relying on index-list for multi-platform

I might have overseen the details. I thought that no OCI-artifact would be created without the --app flag. If the result without the --app flag is a manifest + compose.yaml, then it's almost the same. Having a similar approach to the container images one (you can have only a manifest or an index to have multi-platform) would be the only addition of my proposal.

"Bundled Composition" you describe allows a compose OCI artifact to be subject of an image but not a set of images, as required by a compose application. Could use a top-level index-list to group such images, but then you basically get the same solution as publish --app with subject relation reverted.

I realize that while writing I've merged what I had as two different use-cases in my head:

1. Add a compose.yaml to an existing container image as some sort of documentation or easy to consume execution format. I should rename it to "Attached Compose".
2. What I would also call somehow an "app", which what I would say most of us expect from an app. I should simply rename it to "App".

In the "Attached Compose" the subject is not the compose.yaml, but the other way around.

In the case of an "App" you have a very good point: you can only have a single subject per referrer. As a consequence it should not be a referrer-subject relationship, but an index referring to the index/manifest of the compose.yaml and to the manifests/indexes of the individual container images.

In any case, I'm still convinced that making the container images referrers to the compose.yaml (which becomes their subject) is not the right approach. It forces you to either add a new index to contain the Subject or modify the already existing image index/manifest (changing its digest).

The main issue I can see with your approach is that the application tag doesn't target the actual compose OCI artifact. Then there's no way to refer to it for a consumer to use your "bundle" application.

In the "Attached Compose" I would see the container image as THE focus (or subject) of the OCI artifact. And therefore the typical tags (latest, versions,...) are kept associated to it. Tags associated to such an attachment as the compose.yaml would be just some convenience that is not really needed if referrers can be listed.

In the "App" I fully agree with you. THE focus (or subject) of the OCI artifact is the composition itself and therefore it gets the tags.

[Silvanoc]

I'm closing this issue as "not planned"

@ndeloof if you leave it as it is, you are creating an cyclic dependency! I'll try to illustrate what I mean with an example.

Let's suppose that we have a composition using Grafana v12.1.4 (current index digest is sha256:553160ed532b89bac217ffd424c2d7fb62a1dfbc49685d29993795b77e9cea9a).
If you create a compose.yaml with "digest pinning", then you'd write image: grafana@sha256:553160ed532b89bac217ffd424c2d7fb62a1dfbc49685d29993795b77e9cea9a. And the manifest referring to that compose.yaml index/manifest would be pinning it with its digest. But since you are putting that digest in the Subject of the container image index (or platform manifest), you are changing that index/manifest (so the container image pinning is now wrong). So you need to adapt the compose.yaml, which then gets a new digest which would require a change in the image Subject.

[ndeloof]

since you are putting that digest in the Subject of the container image index (or platform manifest), you are changing that index/manifest (so the container image pinning is now wrong

nope, the image index is not use to refer to the image, but to create a logical group with references to all images used by the compose application (comparable to docker/app design a few years ago). This image index is not expected to be used directly, the image manifest (or index-list for multi-platform) is still the one expected to be used