Adventures in a Cloud Native Landscape

Web User Interfaces for Managing Flux GitOps Environments

Fri, 14 Mar 2025 14:04:56 +0000

Whenever I hear a debate about the pros and cons of Argo CD and Flux as preferred GitOps solutions, invariably it’s Argo CD that wins out. Often, but not always, it’s Argo CD’s web user interface that is cited as the big differentiator between the two tools. Now, I could (but shouldn’t) wax lyrical about the clickops approach being a potential anti-pattern to the GitOps way of doing things. GitOps tools use abstractions represented as Kubernetes config to model the environment; an Application or ApplicationSet, a GitRepository, a HelmChart, a Kustomization, and so on. Just like the Kubernetes config that represents the application that is to be managed using GitOps, the Gitops abstractions need to be expressed declaratively in immutable, versioned storage (such as a version control system, like Git), too. Otherwise, they will suffer the ambiguity and lack of reliability associated with configuration drift. Clickops is fine if these abstractions are managed using GitOps principles, but not OK if they’re created in the cluster, in an imperative fashion through a web UI. Enough!

There is an obvious attraction to a web UI for would be adopters set free in an unfamiliar environment. For developers, whose only want is to get their application deployed, having to get your head around the complexities of Kubernetes and the machinations of a GitOps tool, is an unnecessary distraction. A web UI that enables you to operate these environments, and to gain appropriate, timely feedback on deployments from, is all too valuable. This is the great upside to the web UI that Argo CD provides out-of-the-box. But, what about Flux?

Flux, perhaps because of the clickops anti-pattern, doesn’t come bundled with a web UI like Argo CD¹. However, it has a feature-rich command line interface instead (as does Argo CD, by the way), which enables the creation of the configuration for its abstractions, amongst other useful things. A comprehensive CLI is right up my Straße, but clearly it’s not for everyone, so what can you do to minimise your exposure to it, and still get an optimal experience with Flux? Well, all is not lost, because there are a few community-provided web UI solutions for Flux to try out!

Flux Plugin for Headlamp Link to heading

Headlmap is a general, open-source web UI for Kubernetes, and is a sandbox project of the Cloud Native Computing Foundation. It enables the management of Kubernetes clusters and the workloads deployed therein, and even has a map view (beta) of the objects and their relationships, much the same as can be found in Argo CD’s web UI. Headlamp is designed to be extensible through a plugin architecture, with individual plugins created as Javascript bundles. Currently, Headlamp has plugins for Prometheus, Backstage, cert-manager, OpenCost, and Flux, amongst others. It’s a growing list.

The plugin for Flux adds a sidebar entry for managing the objects that relate to Flux abstractions; things like a HelmRelease, a Kustomization, or an ImagePolicy. In tabular form, it provides the summary details of each object and its status, with the ability to drill down further to retrieve more granular information when needed. This allows for determining the state of the system without having to use the command line to interrogate the cluster directly.

It’s also possible to suspend reconciliation, or force an asynchronous sync with the ‘source’ if required. Headlamp’s UI adapts to the permissions available to the authenticated user, according to the privileges afforded to the user via RBAC. Hence, users get to see and do only what their role permits them to see and do.

Weave GitOps Link to heading

Flux originated from a company called Weaveworks, which sadly closed down in 2024. Amongst the numerous open-source tools initiated by Weaveworks, was Weave GitOps, which included a graphical ‘management overlay’ for Flux-based GitOps environments. After the demise of Weaveworks, the Weave GitOps open-source project lay abandoned for almost a year, before it was uncoupled from the defunct organisation and donated to a community-driven entity of the same name. The first commit in the resurrection was made in November 2024. After a couple of months of work, a 0.39.0 Release Candidate (RC.1) surfaced on GitHub.

Weave GitOps is very similar in look, feel and function to the Flux Plugin for Headlamp, with some minor differences. It won’t be a surprise to learn that the Headlamp plugin was inspired to some degree by Weave GitOps, as evidenced by this comment.

Whilst the web UI provided by Weave GitOps is functioning again as a viable front end to Flux, at present it feels as if there is still a fair bit of work to be done to get it to a stable and viable proposition. It will be interesting to see how these two similar user interfaces progress, and whether they start to diverge at some point.

Capacitor Link to heading

Another option for graphically viewing Flux environments, is Capacitor, an open-source web UI that originated from a company called Gimlet. It has a similar goal to the other user interfaces; to provide simple, graphical feedback on the state of the applications running in a cluster, being managed by Flux. And, to see the state of Flux itself, too.

Capacitor arrived in a fanfare in early 2024, before the Headlamp plugin was released, and before work commenced on the resurrection of Weave GitOps. For a while it was the only web UI available for Flux. In recent times, however, there appears to have been a hiatus in its maintenance, with the last commit to Capacitor’s GitHub repo having been made in October 2024. It’s unclear what the long term future holds for Capacitor.

Conclusion Link to heading

All of these user interfaces for Flux are designed in a similar vein; to provide a simple, utilitarian view on the state of deployed applications, and the Flux runtime environment that manages them. In the main, the interfaces are read-only, which removes the temptation to revert to clickops-style management. Unlike the web UI provided by Argo CD, none of them have the means to create any abstractions for managing applications.

Without the ability to create the abstractions, Flux adopters are left to craft YAML definitions for these abstractions by hand. Or, the Flux CLI can be used to create them according to the arguments that are provided to the sub-command in question (e.g. flux create source oci). Just be sure to export the definitions rather than have them created imperatively in the cluster, and then commit them to the source that the Flux runtime is watching.

It’s a little too early to know how this landscape will shift over time, but for those with a preference for a web UI over a CLI, Headlamp seems the best bet at present. It has a wider focus than just Flux, it’s well maintained, and is a registered sandbox project of the CNCF.

It’s worth mentioning Flamingo, a Flux Subsystem for Argo, which integrates Flux and Argo CD, and thereby provides the Argo CD web UI to Flux users. Unfortunately, the project is currently stalled as it seeks sponsorship for its continuation. ↩︎

Using Frp to Publicly Expose Services in a Local Kubernetes Cluster

Thu, 27 Jun 2024 09:04:15 +0000

I recently had cause to test out cert-manager using the Kubernetes Gateway API, but wanted to do this using a local cluster on my laptop, based on kind. I wanted cert-manager to automatically acquire an X.509 certificate on behalf of an application service running in the cluster, using the ACME protocol. This isn’t straightforward to achieve, as:

The cluster, hosting cert-manager as an ACME client, runs on a laptop on a private network behind a router, using NAT.
The certificate authority (CA) issuing the X.509 certificate, which provides the ACME server component, needs to present a domain validation challenge to cert-manager, from the internet.

Essentially, the problem is that the cluster is on a private network, but needs to be addressable via a registered domain name, from the internet. How best to achieve this?

Options Link to heading

There are probably a million and one ways to achieve this, all with varying degrees of complexity. We could use SSH reverse tunnelling, or a commercial offering like Cloudflare Tunnel, or one of the myriad of open source tunnelling solutions available. I chose to use frp, “a fast reverse proxy that allows you to expose a local server located behind a NAT or firewall to the internet”. With 82,000 GitHub stars, it seems popular!

Frp Link to heading

Frp uses a client/server model to establish a connection at either end of a tunnel; the server component at the end that is publicly exposed to the internet, and the client on the private network behind the router. Client traffic arriving at the frp server end is routed to the frp client through the tunnel, according to the configuration provided for the client and server components.

Frp Server Configuration Link to heading

For my scenario, I chose to host the frp server on a DigitalOcean droplet, with a domain name set to resolve to the droplet’s public IP address. This is the domain name that will appear in the certificate’s Subject Alternative Name (SAN). The configuration file for the server looks like this:

# Server configuration file -> /home/frps/frps.toml

# Bind address and port for frp server and client communication
bindAddr = "0.0.0.0"
bindPort = 7000

# Token for authenticating with client
auth.token = "CH6JuHAJFDNoieah"

# Configuration for frp server dashboard (optional)
webServer.addr = "0.0.0.0"
webServer.port = 7500
webServer.user = "admin"
webServer.password = "NGe1EFQ7w0q0smJm"

# Ports for virtual hosts (applications running in Kubernetes)
vhostHTTPPort = 80
vhostHTTPSPort = 443

In this simple scenario, the configuration provides:

the interfaces and port number through which the frp client interacts with the server
a token used by the client and the server for authenticating with each other
access details for the server dashboard that shows active connections
the ports the server will listen on for virtual host traffic (80 for HTTP and 443 for HTTPS)

Because this setup is temporary, and to make things relatively easy, the frp server can be run using a container rather than installing the binary to the host:

docker run -d --restart always --name frps \
    -p 7000:7000 \
    -p 7500:7500 \
    -p 80:80 \
    -p 443:443 \
    -v /home/frps/frps.toml:/etc/frps.toml \
    ghcr.io/fatedier/frps:v0.58.1 -c /etc/frps.toml

A container image for the frp server is provided by the maintainer of frp, as a GitHub package. The Dockerfile from which the image is built, can also be found in the repo.

Kind Cluster Link to heading

Before discussing the client setup, let’s just describe how the cluster is configured on the private network.

$ docker network inspect -f "{{json .IPAM.Config}}" kind | jq '.[0]'
{
  "Subnet": "172.18.0.0/16",
  "Gateway": "172.18.0.1"
}

Kind uses containers as Kubernetes nodes, which communicate using a (virtual) Docker network provisioned for the purpose, called ‘kind’. In this scenario, it uses the local subnet 172.18.0.0/16. Let’s keep this in the forefront of our minds for a moment, but turn to what’s running in the cluster.

$ kubectl -n envoy-gateway-system get pods,deployments,services
NAME                                            READY   STATUS    RESTARTS   AGE
pod/envoy-default-gw-3d45476e-b5474cb59-cdjps   2/2     Running   0          73m
pod/envoy-gateway-7f58b69497-xxjw5              1/1     Running   0          16h

NAME                                        READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/envoy-default-gw-3d45476e   1/1     1            1           73m
deployment.apps/envoy-gateway               1/1     1            1           16h

NAME                                    TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)               AGE
service/envoy-default-gw-3d45476e       LoadBalancer   10.96.160.26   172.18.0.6    80:30610/TCP          73m
service/envoy-gateway                   ClusterIP      10.96.19.152   <none>        18000/TCP,18001/TCP   16h
service/envoy-gateway-metrics-service   ClusterIP      10.96.89.244   <none>        19001/TCP             16h

The target application is running in the default namespace in the cluster, and is exposed using the Envoy Gateway, acting as a gateway controller for Gateway API objects. The Kubernetes Gateway API supersedes the Ingress API. The Envoy Gateway provisions a deployment of Envoy, which proxies HTTP/S requests for the application, using a Service of type LoadbLanacer. The Cloud Provider for Kind is also used to emulate the provisioning of a ‘cloud’ load balancer, which exposes the application beyond the cluster boundary with an IP address. The IP address is 172.18.0.6, and is on the subnet associated with the ‘kind’ network. Remember, this IP address is still inaccessible from the internet, because it’s on the private network, behind the router.

If the frp client can route traffic received from the frp server, for the domain name, to this IP address on the ‘kind’ network, it should be possible to use cert-manager to request an X.509 certificate using the ACME protocol. Further, it’ll enable anonymous, internet-facing clients to consume applications running in the cluster on the private network, too.

Frp Client Configuration Link to heading

Just as the frp server running on the droplet needs a configuration file, so does the frp client running on the laptop.

# Client configuration file -> /home/frpc/frpc.toml

# Address of the frp server (taken from the environment),
# along with its port
serverAddr = "{{ .Envs.FRP_SERVER_ADDR }}"
serverPort = 7000

# Token for authenticating with server
auth.token = "CH6JuHAJFDNoieah"

# Proxy definition for 'https' traffic, with the destination
# IP address taken from the environment
[[proxies]]
name = "https"
type = "https"
localIP = "{{ .Envs.FRP_PROXY_LOCAL_IP }}"
localPort = 443
customDomains = ["myhost.xyz"]

# Proxy definition for 'http' traffic, with the destination
# IP address taken from the environment
[[proxies]]
name = "http"
type = "http"
localIP = "{{ .Envs.FRP_PROXY_LOCAL_IP }}"
localPort = 80
customDomains = ["myhost.xyz"]

The configuration file content is reasonably self-explanatory, but there are a couple of things to point out:

For flexibility, the IP address of the frp server is configured using an environment variable rather than being hard-coded.
The file contains proxy definitions for both, HTTP and HTTPS traffic, for the domain myhost.xyz. The destination IP address for this proxied traffic is also taken from the environment (which evaluates to 172.18.0.6 in this particular scenario).

As the frp client is getting some of its configuration from the environment, the relevant environment variables need to be set. In this case, the frp server is running on a DigitalOcean droplet, which requires doctl in order to interact with the DigitalOcean API:

export FRP_SERVER_ADDR="$(doctl compute droplet list --tag-name kind-lab --format PublicIPv4 --no-header)"

We know the local target IP address already, but this may be different in subsequent test scenarios, so it’s best to query the cluster to retrieve the IP address and set the variable accordingly:

export FRP_PROXY_LOCAL_IP="$(kubectl get gtw gw -o yaml | yq '.status.addresses.[] | select(.type == "IPAddress") | .value')"

Just as the frp server was deployed as a container, so too can the frp client (Docker image for the client is here, and the Dockerfile here):

docker run -d --restart always --name frpc \
    --network kind \
    -p 7000:7000 \
    -v /home/frpc/frpc.toml:/etc/frpc.toml \
    -e FRP_SERVER_ADDR \
    -e FRP_PROXY_LOCAL_IP \
    ghcr.io/fatedier/frpc:v0.58.1 -c /etc/frpc.toml

The frpc client container is attached to the ‘kind’ network, so that the traffic that it proxies can be routed to the IP address defined in the FRP_PROXY_LOCAL_IP variable; 172.18.0.6. Once deployed, the frp server and client establish a tunnel that proxies HTTP/S requests to the exposed Service in the cluster.

This enables cert-manager to initiate certificate requests for suitably configured Gateway API objects using the ACME protocol. But, it also allows a CA (for example, Let’s Encrypt), to challenge cert-manager with an HTTP-01 or DNS-01 challenge for proof of domain control. In turn, cert-manager is able to respond to the challenge, and then establish a Kubernetes secret with the TLS artifacts (X.509 certificate and private key). The secret can then be used to establish secure TLS-encrypted communication between clients and the target application in the cluster on the private network.

Conclusion Link to heading

Not everyone wants to spin up a cloud-provided Kubernetes cluster for testing purposes; it can get expensive. Local development cluster tools, such as kind, are designed for just such requirements. But, you’ll always need to satisfy that one scenario where you need to access the local cluster from the internet, and sometimes with an addressable domain name. Frp is just one solution available, but it’s a comprehensive solution with a lot more features that haven’t been discussed here. Just to be clear, you should read up on securing the connection between the client and server, to ensure no eavesdropping on the traffic flow.

JSON Pointer Syntax in JSON Patches with Kustomize

Tue, 21 May 2024 15:28:54 +0000

I was recently struggling to add a single annotation to a Kubernetes object definition, using a JSON Patch with Kustomize, when I stumbled on the JSON Pointer syntax. It’s fleetingly referenced in the Kustomize documentation, and can be useful in some circumstances. So, here’s a summary of what it is, and what it does.

The Problem Link to heading

My challenge was to add an annotation to a single Kubernetes object with Kustomize. The commonAnnotations field adds annotations to all rendered objects, so that’s no good for the job at hand, and the builtin Annotations Transformer doesn’t cut it either. We could use a Strategic Merge patch instead, but where would the fun be in that?! A JSON Patch, targeting the specific object definition, seems the best way to go. Let’s suppose we have an object definition in the Kustomize base that contains the following:

apiVersion: gateway.networking.k8s.io
kind: Gateway
metadata:
  name: https-gw
  annotations:
    foo: bar

<snip>

A JSON Patch can be defined to add the annotation cert-manager.io/issuer: letsencrypt-issuer at the path (target location) in the JSON document that represents the object. We could then perform a Kustomize build, and apply the JSON Patch defined in a Kustomization:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
patches:
  - patch: |-
      - op: add
        path: /metadata/annotations
        value:
          cert-manager.io/issuer: letsencrypt-issuer
    target:
      apiVersion: gateway.networking.k8s.io
      kind: Gateway
      name: https-gw

We’d hope that this renders the following from the build:

apiVersion: gateway.networking.k8s.io
kind: Gateway
metadata:
  name: https-gw
  annotations:
    foo: bar
    cert-manager.io/issuer: letsencrypt-issuer

<snip>

Alas, not so. Instead, we get the following, with the original foo: bar annotation replaced with the annotation defined in the patch:

apiVersion: gateway.networking.k8s.io
kind: Gateway
metadata:
  name: https-gw
  annotations:
    cert-manager.io/issuer: letsencrypt-issuer

The reason for this is explained in the JSON Patch IETF RFC6902 standard for the ‘add’ operation:

If the target location specifies an object member that does exist, that member’s value is replaced.

Because the target location specified by the patch’s path (/metadata/annotations) already has a value in the Kustomize base definition (foo: bar), it gets clobbered with the value defined in the patch. That’s not what we want.

Potential Solution Link to heading

In the search for a solution, we could draw on another one of the IETF RFC6902 functions for the ‘add’ operation:

If the target location specifies an object member that does not already exist, a new member is added to the object.

The patch could be amended to include the annotation’s key in the path, which should get added to the object with the value specified:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
patches:
  - patch: |-
      - op: add
        path: /metadata/annotations/cert-manager.io/issuer
        value: letsencrypt-issuer
    target:
      apiVersion: gateway.networking.k8s.io
      kind: Gateway
      name: https-gw

But, this time, there is an error:

Error: add operation does not apply: doc is missing path: "/metadata/annotations/cert-manager.io/issuer": missing value

The proposed key for the annotation, cert-manager.io/issuer, has been interpreted as two different components of the path to the target location in the JSON document. In effect, we’re trying to add the ‘object member’ issuer to a target location that doesn’t exist.

The Solution Link to heading

The JSON Patch specification requires the value of the path to be a string containing a JSON Pointer; a sequence of zero or more reference tokens, each prefixed by a /. The characters / and ~ both have speical meanings in JSON Pointer. So, if one of the reference tokens contains a /, as in our annotation example, it needs to be escaped so that it can be interpreted literally. The sequence ~1 is used in place of / where it is to be interpreted literally. The JSON Patch needs to be re-defined, thus:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
patches:
  - patch: |-
      - op: add
        path: /metadata/annotations/cert-manager.io~1issuer
        value: letsencrypt-issuer
    target:
      apiVersion: gateway.networking.k8s.io
      kind: Gateway
      name: https-gw

Now we’re adding the object member cert-manager.io/issuer to the object. Rendering this Kustomization with a Kustomize build, gives us just what we need:

apiVersion: gateway.networking.k8s.io
kind: Gateway
metadata:
  name: https-gw
  annotations:
    foo: bar
    cert-manager.io/issuer: letsencrypt-issuer

<snip>

Conclusion Link to heading

I’ve always viewed the JSON Patch mechanism in Kustomize as a bit of a black art. Sometimes it pays to go and read up on what you’re actually trying to do! RFCs may not be everyone’s cup of tea, so https://jsonpatch.com/ is a little easier on the eye.

Installing ESO with a Custom Namespace Using Kustomize

Sun, 03 Dec 2023 14:35:54 +0000

I’m putting this here in case anyone runs into the same problem as I did (and, seemingly, others).

TLDR Link to heading

If you want to install the External Secrets Operator (ESO);

using Kustomize instead of Helm,
intend to use the external-secrets.yaml file from the assets on the GitHub release page
want to deploy to a custom namespace (instead of ‘default’)

some patching is required to get the pods running.

The Problem Link to heading

The External Secrets Operator is generally deployed using Helm, with a chart provided and maintained by the project community. For those who don’t use Helm, and would prefer to work with YAML instead, an external-secrets.yaml file is provided in the assets on the project’s GitHub release page. It contains the CRDs as well as the various manifests for getting ESO deployed. The different manifests have been rendered with the assumption that ESO is being deployed to the ‘default’ namespace on Kubernetes.

If you’d prefer to deploy ESO to a different, custom namespace, you might use Kustomize to patch the YAML before using kubectl to apply it to the cluster. A Kustomization could be created to define the namespace, and to have it applied to all object definitions:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - namespace.yaml
  - https://github.com/external-secrets/external-secrets/releases/download/v0.9.9/external-secrets.yaml
namespace: eso

However, this will result in the failure of the TLS bootstrapping mechanism for the validating webhook, which, in turn, will inhibit the ability to create any SecretStore or ExternalSecret objects:

$ kubectl -n eso get po
NAME                                               READY   STATUS             RESTARTS     AGE
external-secrets-9f6b87d4b-f44vx                   1/1     Running            0            21m
external-secrets-cert-controller-8d89c6584-ncbz2   0/1     Running            0            21m
external-secrets-webhook-5857bd5fff-rndcf          0/1     CrashLoopBackOff   6 (2m ago)   21m

If you switch back to deploying in the ‘default’ namespace, all is rosy again. Perusing the documentation, it’s not entirely clear what configuration change is required to prevent this problem occurring, but the logs from the pods suggest that a Kubernetes secret with the TLS credentials for the webhook is missing.

The Solution Link to heading

It turns out the fix requires some additional options provided for the cert-controller and webhook containers, that relate to the custom namespace. With these added, the secret with the TLS credentials appears in the custom namespace, and all is well:

$ kubectl -n eso get secrets external-secrets-webhook
NAME                       TYPE     DATA   AGE
external-secrets-webhook   Opaque   4      15s

A patch can be defined in the Kustomization to effect this:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

<snip>

patches:
  - patch: |-
      - op: replace
        path: /spec/template/spec/containers/0/args/3
        value: "--service-namespace=eso"
      - op: replace
        path: /spec/template/spec/containers/0/args/5
        value: "--secret-namespace=eso"
    target:
      kind: Deployment
      name: external-secrets-cert-controller
  - patch: |-
      - op: replace
        path: /spec/template/spec/containers/0/args/2
        value: "--dns-name=external-secrets-webhook.eso.svc"
    target:
      kind: Deployment
      name: external-secrets-webhook

I hope this helps someone out there!

Bootstrapping Argo CD

Mon, 19 Dec 2022 15:18:55 +0000

This article follows on from an introductory article that discussed the chicken or the egg paradox when it comes to bootstrapping GitOps agents into Kubernetes clusters. The article discusses how this relates to one of the popular GitOps solutions commonly used to automate application deployments to Kubernetes, Argo CD. Argo CD is one of the prominent GitOps solutions that pioneers this evolving approach to application delivery, and is part of a family of tools that co-exist under the Argo umbrella. Collectively, the Argo toolset is a recently graduated project of the Cloud Native Computing Foundation (CNCF).

Installing Argo CD Link to heading

When it comes to bootstrapping Argo CD into a cluster to act as a GitOps agent, there are detailed instructions for installation according to the preferred setup (i.e. multi-tenant, high availability and so on). Kubernetes configuration files (including its custom resource definitions) are maintained at Argo CD’s GitHub repo, and these can be applied directly with kubectl, or through a Kustomization definition. There’s a Helm chart, too, for anyone who prefers to use the chart packaging metaphor for applications. Using one of these techniques gets Argo CD running in a target cluster.

So, we know how to install Argo CD, but, what’s less clear is how or if Argo CD can manage itself according to GitOps principles. The project’s documentation is a little opaque in this regard. But, if you look hard enough, you’ll find a reference to managing Argo CD with Argo CD. It suggests using a Kustomization to define how Argo CD is configured to run in a Kubernetes cluster, with the config stored in a Git repo, which is monitored by the installed instance of Argo CD. There’s even a live online example of Argo CD managing itself, alongside a bunch of other applications.

A deployed agent, with its configuration held in versioned storage, configured to fetch and reconcile that same configuration, fulfils the GitOps principles. What’s missing from this tantalising glimpse of self-management, however, is how this works in practice. Let’s see if we can unpick this.

Applications in Argo CD Link to heading

GitOps agents work with instances of their own custom resources, that enable them to manage applications according to the GitOps principles. For Argo CD, the main custom resource it extends the Kubernetes API with, is the ‘Application’. In defining an Application object, and having it applied to the cluster, we provide Argo CD with the information it needs to manage that application. Some of the information defined in the object is mandatory, and some is optional. For example, the source location of the application’s remote Git or Helm repository that contains its configuration, and the coordinates of the target cluster where the application is to be deployed, are mandatory. Here’s an example Application definition that could be used by Argo CD for managing itself;

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: argocd
  namespace: argocd
spec:
  destination:
    namespace: argocd
    server: https://kubernetes.default.svc
  project: default
  source:
    path: argocd-bootstrap/argocd
    repoURL: https://github.com/nbrownuk/gitops-bootstrap.git
    targetRevision: HEAD
  syncPolicy:
    automated:
      allowEmpty: true
      prune: true
      selfHeal: true

This object definition would need to be applied to a cluster running Argo CD, which would then be enacted upon by Argo CD’s application controller, resulting in periodic fetches of the configuration from the source repo. Any changes made to Argo CD’s configuration in the repo (for example, a change in app version), would then be applied to the cluster, which would result in a rolling update of Argo CD. Hence, Argo CD is managing itself.

But, let’s stop to think for a moment; isn’t the definition of the Application part of the overall configuration? We’ve manually applied the Application object definition to the cluster, but it’s not stored in our ‘source of truth’ in the Git repo. If we had to re-create the cluster, would we remember the Git reference to use for the targetRevision; was it the HEAD, a branch, a specific commit? What was the sync policy for imperative changes made inside the cluster? This is one of the typical scenarios that inspired the GitOps philosophy - to take away the ambiguity of configuration, by using declarative configuration residing in versioned storage. This includes the configuration that drives the actions of the GitOps agent itself.

App of Apps Pattern Link to heading

Using a pattern called ‘app of apps’, Argo CD enables Argo CD to manage itself declaratively. The pattern is more general purpose in utility, in that a single ‘root’ Application definition serves as a pointer to a Git directory path containing one or more further Application definitions. Applying the ‘root’ definition results in the creation of all the subordinate Application objects in the cluster, which are then acted upon by Argo CD’s application controller. One of the subordinate Application objects might be one that defines Argo CD itself.

.
├── managedapps
│  ├── argocd
│  │  ├── kustomization.yaml
│  │  └── namespace.yaml
│  └── podinfo
│     └── kustomization.yaml
├── rootapp
│  ├── kustomization.yaml
│  └── rootapp.yaml
└── subapps
   ├── argocd.yaml
   ├── kustomization.yaml
   └── podinfo.yaml

Here, the ‘rootapp’ Application definition (contained in rootapp.yaml) references the path, ‘subapps’, and each of the Application definitions at this location (argocd.yaml, podinfo.yaml) reference the corresponding sub-directories under ‘managedapps’.

If there are just a few applications, this pattern works well. But, if there are a multitude of applications, then maintenance of the plethora of Application definitions can become burdensome. As an enhancement to this pattern, the Argo CD project introduced the ApplicationSet custom resource and controller. An ApplicationSet object defines one or more ‘generators’, which generate key/value pairs called parameters. The definition will also contain a ‘template’, which the ApplicationSet controller will render with the corresponding parameters. In this way, a single ApplicationSet object can automatically spawn numerous Application resources, saving on the administrative overhead of maintaining lots of Application definitions by hand.

An ApplicationSet definition can work equally as well as the app of apps pattern for managing Argo CD itself. Maybe you’d want to lean towards the ApplicationSet solution, as it’s seen as an evolution of the app of apps pattern.

Argo CD Autopilot Link to heading

One of the problems with setting up the ‘app of apps’ or ApplicationSet configuration for Argo CD self-management, is that there are a number of manual steps involved:

Establish Git repo with app configuration
Install Argo CD
Create secret for trusted access to Git repo
Apply ‘root’ Application or ApplicationSet object definition

None of these steps are particularly onerous, but where there are manual steps there is always the opportunity to introduce error and ambiguity. The more automated everything is, the better chance of a successful outcome. This is where Argo CD Autopilot adds some value.

Argo CD Autopilot is a command line tool for bootstrapping Argo CD and managed applications into Kubernetes clusters. It originated from engineers at Codefresh as an ancillary project to Argo CD. And, it seeks to take away some of the complexity of bootstrapping an Argo CD GitOps implementation, as well as to provide a sane repo structure for the configuration of the managed applications. It still uses ApplicationSet and Application objects under the covers, but hides the complexity behind its CLI. The price you pay for this simplified user experience, is an opinionated approach to the repo structure, and the way you might approach your deployment workflows.

In terms of bootstrapping Argo CD itself, however, it’s as simple as issuing a command:

$ argocd-autopilot repo bootstrap \
    --repo https://github.com/nbrownuk/argocd-bootstrap \
    --git-token "$(< ./github-token)"

The net result is the creation of a Git repo (hence, the need to supply a personal access token with ‘repo’ privileges) containing a prescribed directory structure, with all of the necessary configuration elements to enable self-management of Argo CD. It also results in the deployment of Argo CD to the cluster addressed by your current context. From here you can use the CLI to create projects and applications for management by Argo CD, with the necessary configuration automatically created in the Git repo.

.
├── apps
│  ├── podinfo
│  │  ├── base
│  │  │  └── kustomization.yaml
│  │  └── overlays
│  │     └── prod
│  │        ├── config.json
│  │        └── kustomization.yaml
│  └── README.md
├── bootstrap
│  ├── argo-cd
│  │  └── kustomization.yaml
│  ├── argo-cd.yaml
│  ├── cluster-resources
│  │  ├── in-cluster
│  │  │  ├── argocd-ns.yaml
│  │  │  └── README.md
│  │  └── in-cluster.json
│  ├── cluster-resources.yaml
│  └── root.yaml
└── projects
   ├── prod.yaml
   └── README.md

The directory structure above is an example of a configured Git repo when using Argo CD Autopilot.

It’s early days for the Argo CD Autopilot project, with some features still missing. For example, its CLI doesn’t yet support the use of Helm charts as the embodiment of application configuration, although it’s possible to work around this using the limited support provided by Kustomize. Argo CD Autopilot isn’t necessarily for everybody, especially given its opinionated approach. But, if you’re looking for a convenient method for bootstrapping Argo CD, it does the job perfectly. It even allows you to recover from the loss of a cluster using the CLI, by bootstrapping with reference to the original Git repo as the source of truth.

Bootstrapping Flux

Mon, 12 Dec 2022 12:13:28 +0000

Following on from an introductory article on the merits of bootstrapping GitOps agents into Kubernetes clusters, this article looks into how this can be achieved with Flux. Flux is a collective name to describe several discrete Kubernetes controllers (also known as the GitOps Toolkit), that each perform a separate, specific GitOps-oriented function. Flux is a graduated Cloud Native Computing Foundation (CNCF) project.

Installing Flux Link to heading

Bootstrapping involves installation, and Flux can be installed into a Kubernetes cluster in a variety of ways. A set of YAML manifests are maintained in the project’s GitHub repo:

$ kustomize build https://github.com/fluxcd/flux2/manifests/install?ref=v0.37.0 | \
    kubectl apply -f -

But, if Helm is your thing, then there’s a packaged Helm chart maintained for installing Flux, too. There’s even a command line tool, unsurprisingly called Flux, that can also be used to get Flux up and running in a cluster:

$ flux install

Performing any of these actions would provide us with a working Flux deployment in a cluster, consisting of the set of controllers that govern its GitOps actions.

$ kubectl -n flux-system get po
NAME                                          READY   STATUS    RESTARTS   AGE
helm-controller-7b85c84687-dzdbr              1/1     Running   0          81s
image-automation-controller-f45c4b86b-sqqlf   1/1     Running   0          81s
image-reflector-controller-59c894c647-lzb76   1/1     Running   0          80s
kustomize-controller-d88d76876-dqw2m          1/1     Running   0          80s
notification-controller-55df97fbb9-jbrqb      1/1     Running   0          80s
source-controller-5bb5c7b9bd-dsd6w            1/1     Running   0          80s

But, it doesn’t provide us with a self-managed setup, using immutable, declarative configuration stored under version control.

It turns out that the Flux project considers that bootstrapping, self-management and recovery, are primary concerns of GitOps deployments based on Flux. And, as such, it provides an installation mechanism that speaks to each of these concerns, which is described prominently in the documentation. So, how does Flux go about bootstrapping?

Bootstrapping with the Flux CLI Link to heading

The Flux CLI is a like a Swiss army knife, in that it provides a large number of useful, utilitarian features. As well as using it to install Flux’s controllers into a cluster, you can use it to create the manifests for the custom resource objects that the controllers use to perform GitOps actions, for example. And a whole lot more, besides.

Perhaps one of its most important functions, however, is its ability to perform a bootstrap of a GitOps environment, using a single command:

$ flux bootstrap github \
    --repository flux-bootstrap \
    --owner nbrownuk \
    --personal true

This is a fairly vanilla example of the use of the flux bootstrap command, but there are a ton of other flags available to nuance the outcome. The command performs a lot of work behind the scenes; it:

creates a remote Git repo (GitHub, GitLab, AWS CodeCommit etc.)
clones the repo
generates the manifests for each of Flux’s controllers
commits the manifests and pushes them to the remote repo
installs the controllers into the cluster using the generated manifests
creates an ssh key pair for trusted communication between Flux’s source controller and the remote repo (if you don’t bring your own)
creates a secret embodying the private key, and adds the public key to the remote Git repo as a ‘deploy key’
generates manifests (based on Flux’s custom resources) for the purposes of syncing desired state from the remote repo to the cluster
commits sync manifests and pushes them to the remote repo
applies the sync manifests to cluster, and waits for reconciliation

That’s a lot of heavy lifting done on our behalf, and results in a repo structure that contains all that’s needed for Flux to manage itself:

.
└── flux-system
   ├── gotk-components.yaml
   ├── gotk-sync.yaml
   └── kustomization.yaml

The ‘gotk-components.yaml’ manifest contains the custom resource definitions, and the Kubernetes object definitions for each of the controllers. And, the ‘gotk-sync.yaml’ manifest contains custom resource object definitions, providing Flux with the location of the repo, and the path within the repo that contains the configuration that is to be applied to the cluster. That’s all Flux needs to manage itself. And, should you be unfortunate enough to lose you cluster, the bootstrap command can be run again to provision Flux to a new cluster, using the configuration in the repo.

Updating Flux Link to heading

As I write, even though Flux has gained graduate status with the CNCF, and has acquired a sizeable community of adopters, the project is still working towards General Availability. This means that Flux gets updated on a regular basis, which begs the question, how does Flux get updated after bootstrap?

Firstly, with each release, a new version of the Flux CLI is provided¹. And so, to achieve an update to Flux after an initial bootstrap, all that’s required is a re-run of the bootstrap command with the updated version of the CLI. Execution of the command results in an update to the controller definitions in the ‘gotk-components.yaml’ file in the remote Git repo, which is subsequently fetched and applied to the cluster by Flux. And this results in a rolling update of each controller running in the cluster.

It may be more preferable to gate this change in the source Git repo using a pull request. The revised manifest for the new controller versions could be generated in advance using the following command:

$ flux install --export > ./gotk-components.yaml

If you’re using GitHub as the Git host provider, this could also be automated using GitHub Actions, which could easily be adapted for use in other CI/CD systems.

Bootstrapping and updating in this manner is all well and good, if we’re happy to use a command line interface to achieve this. Sometimes, DevOps teams prefer to bundle the installation of core services into the infrastructure layer. How do we manage this for Flux?

Bootstrapping with Terraform Link to heading

When provisioning an entire environment for running cloud-native applications, an interesting question arises. What delineates infrastructure from applications? This line can get quite blurred, and different people will have different answers as to where to draw the line. But, the question is an important one, because different tools are generally used for managing the different layers. Terraform is pretty ubiquitous when it comes to managing infrastructure, and as we know, Flux is used to manage application deployments to Kubernetes.

Flux sits on the boundary; you can’t manage applications with Flux until it’s provisioned, and you can’t provision Flux until the Kubernetes cluster is provisioned. Many will choose to drop Flux into the infrastructure layer, and have Terraform handle its bootstrapping, after which, Flux is able to manage itself. Anticipating the need, and recognising the importance of catering for this scenario, the Flux project has provided a Flux Terraform provider for provisioning Flux.

Up to at least v0.21.0, the Flux provider consisted of two data sources; one for generating the YAML for the CRDs and deployment configuration, and one for generating the YAML for the sync activity. The code used to achieve this is the same code used in the CLI for bootstrapping Flux. But, generation of the Kubernetes configuration doesn’t get Flux bootstrapped, and the user is required to make use of other Terraform providers to store the configuration in a Git source, and then to get it applied to a cluster.

The official Terraform Kubernetes provider, which is needed to apply the generated configuration to the cluster, comes with some drawbacks. And, further, the notion that the Flux provider doesn’t itself handle the bootstrapping, might be considered a sub-optimal user experience. This has prompted the Flux project to overhaul the Flux provider, so that it performs the bootstrap process in its entirety. At the time of writing, a new bootstrap resource is in development for the Flux provider. With this important revision, the initial terraform apply will:

generate the necessary configuration,
push the configuration to the Git repo,
apply the configuration to the cluster, and wait for reconciliation

Subsequently, Terraform is responsible for managing the configuration in the repo, and Flux is responible for fetching and applying the configuration from the repo. This provides a clear delineation of responsibility, and a much improved user experience.

For the sake of clarity, the version of the CLI reflects the Flux release version, whereas the individual controllers have different, independent release versions. For example, in Flux v0.37.0, the source controller version is v0.32.1, whilst the helm controller is v0.27.0. ↩︎

Bootstrapping GitOps Agents

Sun, 04 Dec 2022 11:47:54 +0000

Since the term was originally coined back in 2017 by Alexis Richardson (CEO of Weaveworks), the GitOps philosophy has struggled to find a coherent, consensual definition. This fostered some ambiguity, and occasionally some contradiction. For example, push-based deployments were once categorised as a valid approach to GitOps deployments, but now they’re not. There was going to be a common GitOps engine for the good of the community, and then there wasn’t. Flux was in the vanguard of software tools that exemplified GitOps, and then it needed to be rewritten from the ground up. And, so on.

It’s encouraging, then, that the last 12 months or so have seen a gradual maturing of the concept, with a common understanding of what we mean by the term ‘GitOps’. And the community has started working together across competing technologies, all under the umbrella of the Cloud Native Computing Foundation (CNCF). Some of the tooling has even acquired ‘graduated’ project status with the CNCF. All is good in GitOps land.

As the work to define standards has unfolded, and the development of software tools to meet those standards has ensued, one aspect of GitOps has consistently bugged me. It’s an implementation detail that doesn’t seem to get a lot of attention.

GitOps Agents Link to heading

First, let’s briefly outline what a GitOps agent is in the context of a Kubernetes cluster. A GitOps agent is a software application that (at least in part) implements the Kubernetes controller pattern, and is responsible for reconciling the declared ‘desired state’ with the actual cluster state. That is, the Kubernetes configuration for an application stored in a version control system (VCS), is periodically fetched and applied to the cluster, resulting in automated deployments. Change is initiated through the VCS, and explicitly not through direct, imperative change in the cluster. This allows application deployments to be managed using the inherent control features of the hosting VCS (i.e. pull or merge requests).

Chicken or the Egg Link to heading

So, GitOps agents allow us to automate application installs and updates in a Kubernetes cluster, according to the GitOps principles. But, aren’t the agents themselves, software applications? They might be controllers, but they’re still just software applications, requiring; installation, update, reconfiguration, recovery, and so on. This begs the question, how do GitOps agents get deployed to a cluster? Can they be deployed to a cluster according to GitOps principles, like the apps they manage? Can they be used to manage themselves? Seemingly, this is an example of the classic chicken or the egg paradox!

If we have to manually install or update a GitOps agent, then it’s deployment is open to all of the issues that the GitOps principles seek to resolve; ambiguous state, configuration drift, unsolicited change, and much more. This effectively makes the state of the whole system as frail as if there were no GitOps agent at all. To that end, it’s crucial that GitOps solutions address this problem head on, and provide a means for bootstrapping their agents, whilst allowing for their ongoing management, in line with the GitOps principles that underpin the whole approach.

This is the first article in a brief series looking into the different approaches taken to the bootstrapping conundrum.

Up Next Link to heading

The two leading projects in the GitOps domain (arguably), Flux and ArgoCD, go about bootstrapping in different ways. First, we’ll discuss how the Flux project goes about bootstrapping its numerous controllers, and then we’ll lift the lid on ArgoCD, to see how it gets things done.

Customising Docker CLI Output

Thu, 03 Aug 2017 12:52:09 +0000

Docker provides a comprehensive API and CLI to its platform. This article is concerned with customising the output returned by Docker CLI commands.

There are a large number of Docker client CLI commands, which provide information relating to various Docker objects on a given Docker host or Swarm cluster. Generally, this output is provided in a tabular format. An example, which all Docker users will have come across, is the docker container ls command, which provides a list of running containers:

$ docker container ls
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                  NAMES
43195e559b42        wordpress           "docker-entrypoint..."   47 seconds ago      Up 46 seconds       0.0.0.0:8080->80/tcp   wp
f7926468281f        mariadb             "docker-entrypoint..."   2 minutes ago       Up 2 minutes        3306/tcp               mysql

Customising Command Output Link to heading

Sometimes, all of this information is too much, and you may find yourself wanting to format the output just how you’d like it. You might want to do this to de-clutter the output, for aesthetic purposes, or for formatting output as input to scripts. This is quite straightforward to do, as a large number of CLI commands have a config option, --format, just for this purpose. The format of the output needs to be specified using a Golang template, which translates a JSON object into the desired format. For example, if we’re only interested in the container ID, image, status, exposed ports and name, we could get this with the following (the \t specifies a tab):

$ docker container ls --format '{{.ID}}\t{{.Image}}\t{{.Status}}\t{{.Ports}}\t{{.Names}}'
43195e559b42    wordpress    Up 41 minutes    0.0.0.0:8080->80/tcp    wp
f7926468281f    mariadb    Up 43 minutes    3306/tcp    mysql

This provides us with the reduced amount of information we specified, but it looks a bit shoddy. We can add the table directive to improve the look:

$ docker container ls --format 'table {{.ID}}\t{{.Image}}\t{{.Status}}\t{{.Ports}}\t{{.Names}}'
CONTAINER ID        IMAGE               STATUS              PORTS                  NAMES
43195e559b42        wordpress           Up About an hour    0.0.0.0:8080->80/tcp   wp
f7926468281f        mariadb             Up About an hour    3306/tcp               mysql

Docker actually uses a template applied to a JSON object, to generate the default output you see when no user-defined formatting is applied. The default table format for listing all of the container objects is:

table {{.ID}}\t{{.Image}}\t{{.Command}}\t{{.RunningFor}}\t{{.Status}}\t{{.Ports}}\t{{.Names}}

These are not the complete set of fields available in the output, however. We can find all of the fields associated with the container object, with:

$ docker container ls --format '{{json .}}' | jq '.'
{
  "Command": "\"docker-entrypoint...\"",
  "CreatedAt": "2017-07-24 16:23:25 +0100 BST",
  "ID": "43195e559b42",
  "Image": "wordpress",
  "Labels": "",
  "LocalVolumes": "1",
  "Mounts": "c71e998f250e...",
  "Names": "wp",
  "Networks": "wp",
  "Ports": "0.0.0.0:8080->80/tcp",
  "RunningFor": "About an hour ago",
  "Size": "0B",
  "Status": "Up About an hour"
}
{
  "Command": "\"docker-entrypoint...\"",
  "CreatedAt": "2017-07-24 16:21:33 +0100 BST",
  "ID": "f7926468281f",
  "Image": "mariadb",
  "Labels": "",
  "LocalVolumes": "1",
  "Mounts": "acaa1732009a...",
  "Names": "mysql",
  "Networks": "wp",
  "Ports": "3306/tcp",
  "RunningFor": "About an hour ago",
  "Size": "0B",
  "Status": "Up About an hour"
}

Notice that there are some keys in each of the objects, missing from the default output; Labels, LocalVolumes, Mounts and Networks, to name a few. Hence, we could customise our output further, by replacing the Status field with the Networks field:

$ docker container ls --format 'table {{.ID}}\t{{.Image}}\t{{.Networks}}\t{{.Ports}}\t{{.Names}}'
CONTAINER ID        IMAGE               NETWORKS            PORTS                  NAMES
43195e559b42        wordpress           bridgey,wp          0.0.0.0:8080->80/tcp   wp
f7926468281f        mariadb             wp                  3306/tcp               mysql

Making a Customisation Permanent Link to heading

The --formatconfig option is great if you want to customise the output in a specific way for a particular use case. It would be a significant PITA, however, if you had to remember this syntax each time you issued a command, if you wanted to perpetually have customised output. You would of course, create an alias, or a script. Docker, however, allows you to make this customisation more permanent, with the use of a configuration file. When a user on a Docker host logs in to the Docker Hub for the very first time, using the docker login command, a file called config.json is created in a directory called .docker in the user’s home directory. This file is used by Docker to hold JSON encoded properties, including a user’s credentials. It can also be used to hold the format template for the docker container ls command, using the psFormat property. The property is called psFormat, after the old version of the command name, docker ps. A config.json file might look like this:

$ cat config.json
{
    "auths": {},
    "psFormat": "table {{.ID}}\t{{.Image}}\t{{.Status}}\t{{.Ports}}\t{{.Names}}\t{{.Networks}}"
}

The psFormat property is the JSON key, whilst the value is the required template for configuring the command output.

With the psFormat property defined, every time you use the docker container ls command, you’ll get the customised output you desire. It’s possible to override the customisation on a case by case basis, simply by using the --format config option, which takes precedence. Take care when editing the config file; incorrect syntax could render all properties invalid.

Valid Command Customisation Properties Link to heading

Whilst the output for a large number of commands can be formatted using the --format config option, permanent customisation via a property defined in the config.json file, is mainly reserved for commands listing particular objects. A complete list of the commands, their relevant config property, and default template, are provided in the table below:

Command	Property	Default Template
`docker container ls`	`psFormat`	`table {{.ID}}\t{{.Image}}\t{{.Command}}\t{{.RunningFor}}\t{{.Status}}\t{{.Ports}}\t{{.Names}}`
`docker image ls`	`imagesFormat`	`table {{.Repository}}\t{{.Tag}}\t{{.ID}}\t{{.CreatedSince}}\t{{.Size}}`
`docker network ls`	`networksFormat`	`table {{.ID}}\t{{.Name}}\t{{.Driver}}\t{{.Scope}}`
`docker plugin ls`	`pluginsFormat`	`table {{.ID}}\t{{.Name}}\t{{.Description}}\t{{.Enabled}}`
`docker volume ls`	`volumesFormat`	`table {{.Driver}}\t{{.Name}}`

This table omits a number of templates for Docker CLI commands that relate to ‘swarm mode’. Finally, the output of the docker stats command can be defined in the config.json file, too. This command provides rudimentary, real-time resource consumption for running containers, and the statsFormat property allows for customising which metrics are displayed:

Command	Property	Default Template
`docker stats`	`statsFormat`	`table {{.Container}}\t{{.CPUPerc}}\t{{.MemUsage}}\t{{.MemPerc}}\t{{.NetIO}}\t{{.BlockIO}}\t{{.PIDs}}`

Explaining Docker Image IDs

Wed, 08 Jun 2016 12:18:47 +0000

When Docker v1.10 came along, there was a fairly seismic change with the way the Docker Engine handles images. Whilst this was publicised well, and there was little impact on the general usage of Docker (image migration, aside), there were some UI changes which sparked some confusion. So, what was the change, and why does the docker history command show some IDs as <missing>?

$ docker history debian
IMAGE               CREATED             CREATED BY                                      SIZE                COMMENT
1742affe03b5        10 days ago         /bin/sh -c #(nop) CMD ["/bin/bash"]             0 B                 
<missing>           10 days ago         /bin/sh -c #(nop) ADD file:5d8521419ad6cfb695   125.1 MB

Preamble Link to heading

First, some background. A docker image is a read-only template for creating containers, and provides a filesystem based on an ordered union of multiple layers of files and directories, which can be shared with other images and containers. Sharing of image layers is a fundamental component of the Docker platform, and is possible through the implementation of a copy-on-write (COW) mechanism. During its lifetime, if a container needs to change a file from the read-only image that provides its filesystem, it copies the file up to its own private read-write layer before making the change.

A layer or ‘diff’ is created during the Docker image build process, and results when commands are run in a container, which produce new or modified files and directories. These new or modified files and directories are ‘committed’ as a new layer. The output of the docker history command above shows that the debian image has two layers.

Historical Perspective Link to heading

Historically (pre Docker v1.10), each time a new layer was created as a result of a commit action, Docker also created a corresponding image, which was identified by a randomly generated 256-bit UUID, usually referred to as an image ID (presented in the UI as either a short 12-digit hex string, or a long 64-digit hex string). Docker stored the layer contents in a directory with a name synonymous with the image ID. Internally, the image consisted of a configuration object, which held the characteristics of the image, including its ID, and the ID of the image’s parent image. In this way, Docker was able to construct a filesystem for a container, with each image in turn referencing its parent and the corresponding layer content, until the base image was reached which had no parent. Optionally, each image could also be tagged with a meaningful name (e.g. my_image:1.0), but this was usually reserved for the leaf image. This is depicted in the diagram below:

Using the docker inspect command would yield:

$ docker inspect my_image:1.0
[
    {
        "Id": "ca1f5f48ef431c0818d5e8797dfe707557bdc728fe7c3027c75de18f934a3b76",
        "Parent": "91bac885982d2d564c0e1869e8b8827c435eead714c06d4c670aaae616c1542c"
        ...
        ...

This method served Docker well for a sustained period, but over time was perceived to be sub-optimal for a variety of reasons. One of the big drivers for change, came from the lack of a means of detecting whether an image’s contents had been tampered with during a push to or pull from a registry, such as the Docker Hub. This led to robust criticism from the community at large, and led to a series of changes, culminating in content addressable IDs.

Content Addressable IDs Link to heading

Since Docker v1.10, generally, images and layers are no longer synonymous. Instead, an image directly references one or more layers that eventually contribute to a derived container’s filesystem.

Layers are now identified by a digest, which takes the form algorithm:hex; for example:

sha256:fc92eec5cac70b0c324cec2933cd7db1c0eae7c9e2649e42d02e77eb6da0d15f

The hex element is calculated by applying the algorithm (SHA256) to a layer’s content. If the content changes, then the computed digest will also change, meaning that Docker can check the retrieved contents of a layer with its published digest in order to verify its content. Layers have no notion of an image or of belonging to an image, they are merely collections of files and directories.

A Docker image now consists of a configuration object, which (amongst other things) contains an ordered list of layer digests, which enables the Docker Engine to assemble a container’s filesystem with reference to layer digests rather than parent images. The image ID is also a digest, and is a computed SHA256 hash of the image configuration object, which contains the digests of the layers that contribute to the image’s filesystem definition. The following diagram depicts the relationship between image and layers post Docker v1.10:

The digests for the image and layers have been shortened for readability.

The diff directory for storing the layer content, is now named after a randomly generated ‘cache ID’, and the Docker Engine maintains the link between the layer and its cache ID, so that it knows where to locate the layer’s content on disk.

So, when a Docker image is pulled from a registry, and the docker history command is used to reveal its contents, the output provides something similar to:

$ docker history swarm
IMAGE               CREATED             CREATED BY                                      SIZE                COMMENT
c54bba046158        9 days ago          /bin/sh -c #(nop) CMD ["--help"]                0 B                 
<missing>           9 days ago          /bin/sh -c #(nop) ENTRYPOINT &{["/swarm"]}      0 B                 
<missing>           9 days ago          /bin/sh -c #(nop) VOLUME [/.swarm]              0 B                 
<missing>           9 days ago          /bin/sh -c #(nop) EXPOSE 2375/tcp               0 B                 
<missing>           9 days ago          /bin/sh -c #(nop) ENV SWARM_HOST=:2375          0 B                 
<missing>           9 days ago          /bin/sh -c #(nop) COPY dir:b76b2255a3b423981a   0 B                 
<missing>           9 days ago          /bin/sh -c #(nop) COPY file:5acf949e76228329d   277.2 kB            
<missing>           9 days ago          /bin/sh -c #(nop) COPY file:a2157cec2320f541a   19.06 MB

The command provides detail about the image and the layers it is composed of. The <missing> value in the IMAGE field for all but one of the layers of the image, is misleading and a little unfortunate. It conveys the suggestion of an error, but there is no error as layers are no longer synonymous with a corresponding image and ID. I think it would have been more appropriate to have left the field blank. Also, the image ID appears to be associated with the uppermost layer, but in fact, the image ID doesn’t ‘belong’ to any of the layers. Rather, the layers collectively belong to the image, and provide its filesystem definition.

Locally Built Images Link to heading

Whilst this narrative for content addressable images holds true for all Docker images post Docker v1.10, locally built images on a Docker host are treated slightly differently. The generic content of an image built locally remains the same - it is a configuration object containing configuration items, including an ordered list of layer digests.

However, when a layer is committed during an image build on a local Docker host, an ‘intermediate’ image is created at the same time. Just like all other images, it has a configuration item which is a list of the layer digests that are to be incorporated as part of the image, and its ID or digest contains a hash of the configuration object. Intermediate images aren’t tagged with a name, but, they do have a ‘Parent’ key, which contains the ID of the parent image.

The purpose of the intermediate images and the reference to parent images, is to facilitate the use of Docker’s build cache. The build cache is another important feature of the Docker platform, and is used to help the Docker Engine make use of pre-existing layer content, rather than regenerating the content needlessly for an identical build command. It makes the build process more efficient. When an image is built locally, the docker history command might provide output similar to the following:

$ docker history jbloggs/my_image:latest 
IMAGE               CREATED             CREATED BY                                      SIZE                COMMENT
26cca5b0c787        52 seconds ago      /bin/sh -c #(nop) CMD ["/bin/sh" "-c" "/bin/b   0 B                 
97e47fb9e0a6        52 seconds ago      /bin/sh -c apt-get update &&     apt-get inst   16.98 MB            
1742affe03b5        13 days ago         /bin/sh -c #(nop) CMD ["/bin/bash"]             0 B                 
<missing>           13 days ago         /bin/sh -c #(nop) ADD file:5d8521419ad6cfb695   125.1 MB

In this example, the top two layers are created during the local image build, whilst the bottom layers came from the base image for the build (e.g. Dockerfile instruction FROM debian). We can use the docker inspect command to review the layer digests associated with the image:

$ docker inspect jboggs/my_image:latest 
[
    {
        ...
        ...
        "RootFS": {
            "Type": "layers",
            "Layers": [
                "sha256:4dcab49015d47e8f300ec33400a02cebc7b54cadd09c37e49eccbc655279da90",
                "sha256:f22bfbc1df820c465d94215e45bf9b1f0ae0fe3435a90dc5296a4b55712f46e7"
            ]
        }
    }
]

The docker history command shows the image as having four layers, but docker inspect suggests just two layers. This is because the two CMD instructions produce metadata for the image, don’t add any content, and therefore the ‘diff’ is empty.

When a locally built image is pushed to a registry, it is only the leaf image that is uploaded along with its constituent layers, and a subsequent pull by another Docker host will not yield any intermediate parent images. This is because once the image is made available to other potential users on different Docker hosts via a registry, it effectively becomes read-only, and the components that support the build cache are no longer required. Instead of the image ID, <missing> is inserted into the history output in its place.

Pushing the image to a registry might yield:

$ docker push jbloggs/my_image:latest
The push refers to a repository [docker.io/jbloggs/my_image]
f22bfbc1df82: Pushed 
5f70bf18a086: Layer already exists 
4dcab49015d4: Layer already exists 
latest: digest: sha256:7f63e3661b1377e2658e458ac1ff6d5e0079f0cfd9ff2830786d1b45ae1bb820 size: 3147

In this example, only one layer has been pushed, as two of the layers already exist in the registry, referenced by one or more other images which use the same content.

A Final Twist Link to heading

The digests that Docker uses for layer ‘diffs’ on a Docker host, contain the sha256 hash of the tar archived content of the diff. Before the layer is uploaded to a registry as part of a push, it is compressed for bandwidth efficiency. A manifest is also created to describe the contents of the image, and it contains the digests of the compressed layer content. Consequently, the digests for the layers in the manifest are different to those generated in their uncompressed state. The manifest is also pushed to the registry.

The digest of a compressed layer diff can be referred to as a ‘distribution digest’, whilst the digest for the uncompressed layer diff can be referred to as a ‘content digest’. Hence, when we pull our example image on a different Docker host, the docker pull command gives the following output:

$ docker pull jbloggs/my_image
Using default tag: latest
latest: Pulling from jbloggs/my_image

51f5c6a04d83: Pull complete 
a3ed95caeb02: Pull complete 
9a246d793396: Pull complete 
Digest: sha256:7f63e3661b1377e2658e458ac1ff6d5e0079f0cfd9ff2830786d1b45ae1bb820
Status: Downloaded newer image for jbloggs/my_image:latest

The distribution digests in the output of the docker pull command, are very different to the digests reported by the docker push command. But, the pull will decompress the layers, and the output of a docker inspect command will provide the familiar content digests that we saw after the image build.

Summary Link to heading

Following the changes to image and layer handling in Docker v1.10:

A Docker image provides a filesystem for a derived container based on the references it stores to layer diffs
Layer diffs are referenced using a digest, which contains an SHA256 hash of an archive of the diff’s contents
A Docker image’s ID is a digest, which contains an SHA256 hash of the image’s JSON configuration object
Docker creates intermediate images during a local image build, for the purposes of maintaining a build cache
An image manifest is created and pushed to a Docker registry when an image is pushed
An image manifest contains digests of the image’s layers, which contain the SHA256 hashes of the compressed, archived diff contents

Referencing Docker Images

Mon, 27 Apr 2015 16:39:42 +0000

Docker images are the templates that derive the nature and behaviour of a container, and Docker stores these images in repositories on hosted registries. An official registry, called the Docker Hub Registry, is hosted by Docker Inc., which contains:

A set of official, certified Docker repositories, which are curated by the Docker community
Publicly accessible repositories, provided by any individual or organisation with a Docker Hub account
Private repositories for individuals and organisations who purchase one of the available plans provided by Docker Inc

The Docker Hub Registry is an incredibly valuable resource, with over 7 million publicly available repositories of Docker images. But what if you’re a security conscious corporation, that wants to keep your intellectual property proprietary, behind a corporate firewall? Or you’re a third-party wanting to provide a value add service to your customers? You have a choice; you can either purchase a commercial product like Red Hat Quay, or you can deploy an open-source solution such as Harbor inside your corporate firewall.

All of these options, however, pose a serious question - how do I address the correct image that I need for my container? For example, how do I make sure that the MySQL image I use for my application is the one that has been carefully crafted by the Database Administrators inside my organisation, rather than the official MySQL image on the public Docker Hub Registry, or even some other random MySQL image provided by an unknown entity on the Docker Hub Registry? This all comes down to specifying the correct image name when you retrieve an image or invoke a container using the Docker CLI or API, and there is a format that needs to be adhered to. A fully qualified image name (FQIN) consists of three main components; a registry location (with an optional port specification), a username, and a repository name (with an optional tag specification):

hostname[:port]/username/reponame[:tag]

The hostname and optional port specify the location of the registry, and if these are omitted then Docker defaults to the Docker Hub Registry at index.docker.io. The next element in the image name is a username, and once again, if this is omitted, it corresponds to a special username called library. In the Docker Hub Registry, the library username is for the officially, curated Docker images. Finally, a repository name needs to be specified, and optionally an image tag to identify the specific image from its related images in the repository (if the tag is omitted, Docker assumes the tag latest).

Library Images Link to heading

In order to ‘pull’ the latest official Ubuntu image, the following Docker CLI command can be invoked:

docker pull ubuntu

In this format, the registry location, username and tag have been omitted. The shortened image name directs the Docker engine to pull the latest library image from the ubuntu repository on the Docker Hub Registry. This could also have been achieved using the longhand format:

docker pull index.docker.io/library/ubuntu:latest

User Images Link to heading

In order to pull the latest version of an image called pxe that belongs to the user jpetazzo on the Docker Hub Registry, the following command can be used:

docker pull jpetazzo/pxe

In this example, the registry location has been omitted, and so the default Docker Hub Registry is the target for the Docker engine.

Images on Third-Party Registries Link to heading

Some third-party organisations host their own Docker registries independent of Docker Inc, which they make available to their customers. In order to pull an image that resides on a third party registry (such as r.j3ss.co), the registry location needs to be supplied along with the username and repository, e.g.:

docker pull r.j3ss.co/tor-browser:stable

In this case, a tag has been specified as part of the image name in order to differentiate it from other versions of the image.

Images on Self-Hosted Registries Link to heading

Finally, we can reference an image that resides on a locally configured, self-hosted registry by specifying the registry location and the repository required:

docker pull internal.mycorp.com:5000/revealjs:latest

The docker-proxy

Wed, 15 Apr 2015 12:11:31 +0000

Containers created and managed by the Docker engine are able to provide the service that is running inside the container, not only to other co-located containers, but also to remote hosts. Docker achieves this with port forwarding.

When a container starts with its port mapped to the Docker host on which it runs, in addition to the new process that runs inside the container, you may have noticed an additional process on the Docker host called docker-proxy:

  PID TTY      STAT   TIME COMMAND
 8006 ?        Sl     0:00 docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 8000 -container-ip 172.17.0.2 -container-port 8000

The purpose of this process is to enable a service consumer to communicate with the service providing container …. but it’s only used in particular circumstances. The docker-proxy operates in userland, and simply receives any packets arriving at the host’s specified port, that the kernel hasn’t ‘dropped’ or forwarded, and redirects them to the container’s port.

In order to understand why this process exists, we first need to understand a little about Docker’s networking configuration. The default modus operandi for a Docker host is to create a virtual ethernet bridge (called docker0), attach each container’s network interface to the bridge, and to use network address translation (NAT) when containers need to make themselves visible to the Docker host and beyond:

Controlling access to a container’s service is controlled with rules associated with the host’s netfilter framework, in both the NAT and filter tables. The general processing flow of packets by netfilter is depicted in this diagram.

If a container’s port 172.17.0.2:8000 is to be mapped to the host as port 8000, then Docker adds some rules to netfilter’s NAT table, enabling the container to ‘masquerade’ as the host using NAT:

Chain PREROUTING (policy ACCEPT 49 packets, 9985 bytes)
num   pkts bytes target     prot opt in     out       source               destination         
1       80  4152 DOCKER     all  --  *      *         0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL

Chain OUTPUT (policy ACCEPT 1436 packets, 151K bytes)
num   pkts bytes target     prot opt in     out       source               destination         
1      274 56172 DOCKER     all  --  *      *         0.0.0.0/0           !127.0.0.0/8          ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT 1369 packets, 137K bytes)
num   pkts bytes target     prot opt in     out       source               destination         
1      274 56172 MASQUERADE all  --  *      !docker0  172.17.0.0/16        0.0.0.0/0                   
2        0     0 MASQUERADE tcp  --  *      *         172.17.0.2           172.17.0.2           tcp dpt:8000

Chain DOCKER (2 references)
num   pkts bytes target     prot opt in       out     source               destination         
1        0     0 DNAT       tcp  --  !docker0 *       0.0.0.0/0            0.0.0.0/0            tcp dpt:8000 to:172.17.0.2:8000

Netfilter is stateful, which means that it can track connections that have already been established, and in such circumstances it bypasses the NAT table rules. But in order for a connection to be established in the first place, packets are subjected to the scrutiny of the rules in the NAT and filter tables.

Packets destined for the host’s socket (the container’s mapped port) are processed by netfilter and tested against the rules in the PREROUTING chain of the NAT table, and provided the destination address of a packet is local to the Docker host (which it is), netfilter jumps to the DOCKER chain for further processing. As long as the packet didn’t arrive from the ethernet bridge (i.e. from a container), and provided the packet is addressed to TCP port 8000 on the Docker host, then its destination is changed to 172.17.0.2:8000 by the DNAT target - which is the container socket. As the packet needs to be routed to the container, the rules in the FORWARD chain of the filter table are assessed:

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
num   pkts bytes target     prot opt in       out      source              destination         
1       63 10326 DOCKER     all  --  *        docker0  0.0.0.0/0           0.0.0.0/0           
2       50  9618 ACCEPT     all  --  *        docker0  0.0.0.0/0           0.0.0.0/0            ctstate RELATED,ESTABLISHED
3       61  5675 ACCEPT     all  --  docker0 !docker0  0.0.0.0/0           0.0.0.0/0           
4        0     0 ACCEPT     all  --  docker0  docker0  0.0.0.0/0           0.0.0.0/0           

Chain DOCKER (1 references)
num   pkts bytes target     prot opt in       out      source              destination         
1        0     0 ACCEPT     tcp  --  !docker0 docker0  0.0.0.0/0           172.17.0.2           tcp dpt:8000

The first rule applies, which forces a jump to the DOCKER chain, and the single rule in the chain matches the characteristics of the packet, and ‘accepts’ the packet for forwarding on to the container’s socket. Hence, a remote service consuming process thinks it is communicating with the Docker host, but is being serviced by the container instead.

Similarly, when a container initiates a dialogue with a remote service provider, netfilter’s NAT POSTROUTING chain changes the source IP address of packets from the container’s IP address, to the address of the host’s network interface that is responsible for routing the packets to their required destination. This is achieved with netfilter’s MASQUERADE target.

A Docker host makes significant use of netfilter rules to aid NAT, and to control access to the containers it hosts, and the docker-proxy mechanism isn’t always required. However, there are certain circumstances where this method of control is not available, which is why Docker also creates an instance of the docker-proxy whenever a container’s port is forwarded to the Docker host.

Firstly, in order for a remote host to consume a container’s service, the Docker host must act like a router, forwarding traffic to the network associated with the ethernet bridge. A Linux host is not normally configured to be a router, so the kernel parameter net.ipv4.ip_forward needs to be set to 1 (net.ipv6.conf.default.forwarding and net.ipv6.conf.all.forwarding for IPv6). Docker takes care of this if its daemon is started with default settings. If, however, the daemon is started with the --ip-forward and/or --iptables command line options set to false, then Docker can’t make use of netfilter rules and has to fall back on the docker-proxy. This scenario is probably quite rare, but it is conceivable that some corporate security policies may impose this constraint.

Secondly, even when Docker is able to forward packets using netfilter rules, there is one circumstance where it is not possible to apply netfilter rules. Unless told otherwise, when a container’s port is forwarded to the Docker host, it will be forwarded to all of the host’s interfaces, including its loopback interface. But the Linux kernel does not allow the routing of loopback traffic, and therefore it’s not possible to apply netfilter NAT rules to packets originating from 127.0.0.0/8. Instead, netfilter sends packets through the filter table’s INPUT chain to a local process listening on the designated port - the docker-proxy¹.

The docker-proxy, then, is a ‘catch all’ method for allowing container port forwarding to the Docker host. However, it’s generally considered that the docker-proxy is an inelegant solution to the problems highlighted above, and when a large range of container ports are exposed, it consumes considerable memory. An attempt was previously made to remove the dependency for the docker-proxy, but this fell foul of the limitations of the aged kernel in RHEL 6.x and CentOS 6.x, which the Docker project feels duty bound to support. Hence, the docker-proxy remains a major constituent part of the Docker experience in all Docker versions up to the current version 1.5. As I write, version 1.6 is due for imminent release, and there have been moves to remove the automatic requirement for the docker-proxy, which I’ll cover in another article.

Update - Hairpin NAT Support ↩︎

A Basic Container

Wed, 25 Mar 2015 10:59:59 +0000

We’ve looked at five of the six available namespaces provided by the Linux kernel in a series of previous articles, and we’ll take a look at the final namespace, the USER namespace, in a future article. This article looks at how we can combine a number of the namespaces with a specially prepared directory, in which we’ll ‘jail’ our process using the chroot system call. Although our implementation will be missing a few key features that normally accompany container implementations (e.g. cgroups), the resulting environment in which our process will run, can be considered a very rudimentary container of sorts. It isolates the process from several different system resources, and contains the process within a limited filesystem.

The first thing we need to do is to prepare a directory on the host which will become the root filesystem for the container, which will be located at /var/local/jail. We’re going to provide just a few binaries for the container to use; env, bash, ps, ls and top¹.

It’s not just a simple matter of copying the binaries to /var/local/jail, each binary relies on shared libraries, and we also need to ensure they are available in the appropriate directory in the container’s filesystem. To do this, we can make use of the ldd command, whose purpose is to provide information regarding the shared libraries used by a particular binary. I’ve created a script called binlibdepcp.sh, which takes care of determining the library dependencies for a binary, and then copying them along with the relevant libraries to the correct locations in the container’s filesystem. It also copies the ld.so.cache file, which is the list of directories that is searched for libraries, in the event that a required library does not reside in /lib or /usr/lib. The script is available on GitHub.

Let’s demonstrate this for the env binary, which is located at /usr/bin/env. Having previously created the /var/local/jail directory, the env binary and libraries are copied to the correct location under /var/local/jail with the following:

$ sudo ./binlibdepcp.sh /usr/bin/env /var/local/jail
[sudo] password for wolf:
Copying ...

                      env : [OK]
                libc.so.6 : [OK]
     ld-linux-x86-64.so.2 : [OK]
              ld.so.cache : [OK]

...... Done

We can repeat this exercise for the other binaries we intend to use within the container, which, more likely than not, are located in either /usr/bin or /bin. Additionally, so that our commands will display nicely when we run them inside the container, we need to provide the relevant portion of the terminfo database. Assuming we have an xterm, we can copy this into the jail (the location of the terminfo database varies from Linux distro to distro, so make sure to copy the files to the correct directory under /var/local/jail):

$ sudo mkdir -p /var/local/jail/lib/terminfo/x
[sudo] password for wolf:
$ sudo cp -p /lib/terminfo/x/* /var/local/jail/lib/terminfo/x

That’s the container’s filesystem prepared. Now we need to amend the program we have slowly been developing whilst we’ve been looking at the properties of containers. The changes are available in the invoke_ns6.c source file, which can be found here.

The first change is to add a new command line option, -c which must be accompanied with a directory path, which will be the root of the jail:

// Parse command line options and construct arguments
// to be passed to childFunction
while ((option = getopt(argc, argv, "+hvpmu:ni:c:")) != -1) {
    switch (option) {
        case 'c':
            args.jail = 1;
            args.path = malloc(sizeof(char *) * (strlen(optarg) + 1));
            strcpy(args.path, optarg);
            break;
        case 'i':
            if (strcmp("no", optarg) != 0 && strcmp("yes", optarg) != 0) {
                fprintf(stderr, "%s: option requires valid argument -- 'i'\n", argv[0]);
                usage(argv[0]);
                exit(EXIT_FAILURE);
            }
            else
                if (strcmp("yes", optarg) == 0)
                    flags |= CLONE_NEWIPC;
            args.ipc = 1;
            break;
        case 'n':
            flags |= CLONE_NEWNET;
            break;
        case 'u':
            flags |= CLONE_NEWUTS;
            args.hostname = malloc(sizeof(char *) * (strlen(optarg) + 1));
            strcpy(args.hostname, optarg);
            break;
        case 'm':
            flags |= CLONE_NEWNS;
            break;
        case 'p':
            flags |= CLONE_NEWPID;
            break;
        case 'v':
            args.verbose = 1;
            break;
        case 'h':
            usage(argv[0]);
            exit(EXIT_SUCCESS);
        default:
            usage(argv[0]);
            exit(EXIT_FAILURE);
    }
}

The other main change is to add some code to ensure that if the -c option is supplied, the cloned child process is jailed inside the directory with the chroot system call. The chroot system call changes the root directory of the child process, and we change the current working directory to that root directory, and then create a /proc directory within it:

// If specified, place process in chroot jail
if (args->jail) {
    if (args->verbose)
        printf(" Child: creating chroot jail\n");
    if (chroot(args->path) == -1) {
        perror(" Child: childFunction: chroot");
        exit(EXIT_FAILURE);
    }
    else {
        if (args->verbose)
            printf(" Child: changing directory into chroot jail\n");
        if (chdir("/") == -1) {
            perror(" Child: childFunction: chdir");
            exit(EXIT_FAILURE);
        }
        if (access("/proc", F_OK) != 0)
            if (mkdir("/proc", 0555) == -1) {
                perror(" Child: childFunction: mkdir");
                exit(EXIT_FAILURE);
            }
    }
}

We can now invoke our container (with a customised command prompt) with the following command:

$ sudo ./invoke_ns -vpmu calculus -c /var/local/jail env PS1="\h [\W] " TERM=$TERM bash --norc
[sudo] password for wolf:
calculus [/]

Now that we have an interactive bash command shell running inside the container, we can use the ls, ps and top commands to verify we have a very minimal operating environment, if not a very useful one! It doesn’t take much imagination, however, to see the possibilities for containing independent workloads in minimal, lightweight containers.

In reality, a process inside a container needs a few more things than we have provided in our rudimentary version. Thankfully, the excellent work that has been conducted in the open source community with projects like Docker and LXC, have taken the hard work out of creating and manipulating workloads within containers.

You could also use the content of a root filesystem obtained from a suitable distro, e.g. Ubuntu ↩︎

IPC Namespace

Mon, 16 Feb 2015 16:02:00 +0000

The IPC namespace is used for isolating System V IPC objects, and POSIX message queues. The clone flag used to achieve this is CLONE_NEWIPC. We’ve adapted our program from previous articles, to create a POSIX message queue, which will be used to pass a message between a process and its cloned child process.

To demonstrate how the IPC namespace establishes isolation of a POSIX message queue, we’ll create a message queue in the parent process, and attempt to open it from within the child process, in order to send a message from the child to the parent process. We’ll do this once when the cloned child process remains in the global IPC namespace, and then when it is created within a new IPC namespace.

So, first off, we need to embellish the program, which is available here (invoke_ns5.c), with a new command line option, -i. The -i option must be accompanied with a string of either ‘yes’ or ‘no’. If ‘yes’ is specified, the child process will be cloned in a new IPC namespace, and if ‘no’ is specified, it remains in the global IPC namespace.

The command line parsing now looks like this:

// Parse command line options and construct arguments
// to be passed to childFunction
while ((option = getopt(argc, argv, "+hvpmu:ni:")) != -1) {
    switch (option) {
        case 'i':
            if (strcmp("no", optarg) != 0 && strcmp("yes", optarg) != 0) {
                fprintf(stderr, "%s: option requires valid argument -- 'i'\n", argv[0]);
                usage(argv[0]);
                exit(EXIT_FAILURE);
            }
            else
                if (strcmp("yes", optarg) == 0)
                    flags |= CLONE_NEWIPC;
            args.ipc = 1;
            break;
        case 'n':
            flags |= CLONE_NEWNET;
            break;
        case 'u':
            flags |= CLONE_NEWUTS;
            args.hostname = malloc(sizeof(char) * (strlen(optarg) + 1));
            strcpy(args.hostname, optarg);
            break;
        case 'm':
            flags |= CLONE_NEWNS;
            break;
        case 'p':
            flags |= CLONE_NEWPID;
            break;
        case 'v':
            args.verbose = 1;
            break;
        case 'h':
            usage(argv[0]);
            exit(EXIT_SUCCESS);
        default:
            usage(argv[0]);
            exit(EXIT_FAILURE);
    }
}

We also need to define the message queue name:

// Define message queue name
const char *mq_name = "/ipc_namespace";

If the -i command line option has been specified, before making the call to clone, the parent calls the prepareMQ function:

// Prepare message queue
if (args.ipc) {
    mq = prepareMQ(&args);
    if (mq == -1)
        exit(EXIT_FAILURE);
}

The prepareMQ function creates a message queue (in the global IPC namespace), and opens it for reading and writing, with a maximum message size of 81 bytes (an extra byte for termination):

// Prepare message queue
mqd_t prepareMQ(void *child_args)
{
    struct arguments *args = child_args;
    mqd_t            mq;
    int              oflags = O_CREAT | O_RDWR;
    struct mq_attr   attr;

    attr.mq_flags   = 0;
    attr.mq_maxmsg  = 10;
    attr.mq_msgsize = 81;
    attr.mq_curmsgs = 0;

    mq = mq_open(mq_name, oflags, 0644, &attr);
    if (mq != -1) {
        if (args->verbose)
            printf("Parent: opening message queue %s\n", mq_name);
    }
    else
        perror("Parent: prepareMQ: mq_open");

    return mq;
}

Once the message queue has been established, the child process is created with the clone system call, and then the parent waits 60 seconds for a message from the child process, before timing out. If a message is received from the child process, the parent process echoes this to stdout:

// Read message from child on message queue
if (args.ipc) {
    if (clock_gettime(CLOCK_REALTIME, &timeout) == 0) {
        timeout.tv_sec += 60;
        if (mq_getattr(mq, &attr) != -1) {
            msg = malloc(attr.mq_msgsize);
            if (mq_timedreceive(mq, msg, attr.mq_msgsize, NULL, &timeout) != -1) {
                if (args.verbose)
                    printf("Parent: received message from child\n");
                printf("\n    Parent: the following message was received from the child\n     >> %s\n\n", msg);
            }
            else
                perror("Parent: main: mq_timedreceive");
            free(msg);
        }
        else
            perror("Parent: main: mq_getattr");
    }
    else
        perror("Parent: main: clock_gettime");
}

Once the child process has terminated, the parent closes and removes the message queue:

// Remove message queue
if (args.ipc) {
    if (args.verbose)
        printf("Parent: closing message queue %s\n", mq_name);
    if (mq_close(mq) == -1)
        perror("Parent: main: mq_close");
    if (args.verbose)
        printf("Parent: removing message queue %s\n", mq_name);
    if (mq_unlink(mq_name) == -1)
        perror("Parent: main: mq_unlink");
}

With regard to the cloned child, we have added some additional code to the childFunction, which gets executed when the child process is created. If the -i command line option has been specified, the child attempts to open the message queue created by the parent, and prompts the user to enter a message, which is then sent to the message queue. If the child has got this far, it closes the message queue and continues to process the other namespace requirements:

// Send message to parent if -i option provided
if (args->ipc) {
    if (args->verbose)
        printf(" Child: opening message queue %s\n", mq_name);
    mq = mq_open(mq_name, oflags);
    if (mq != -1) {
        if (mq_getattr(mq, &attr) != -1) {
            msg = malloc(attr.mq_msgsize);
            printf("\n     Child: enter a message to send to the parent process (MAX 80 chars)\n     >> ");
            if (fgets(msg, attr.mq_msgsize, stdin) != NULL) {
                msg[strcspn(msg, "\n")] = '\0';
                printf("\n");
                if (args->verbose)
                    printf(" Child: sending message to parent\n");
                if (mq_send(mq, msg, attr.mq_msgsize,0) == -1)
                    perror(" Child: childFunction: mq_send");
            }
            else
                perror(" Child: childFunction: fgets");
            if (args->verbose)
                printf(" Child: closing message queue %s\n", mq_name);
            if (mq_close(mq) == -1)
                perror(" Child: childFunction: mq_close");
            free(msg);
        }
        else
            perror(" Child: childFunction: mq_getattr");
    }
    else
        perror(" Child: childFunction: mq_open");
}

Now let’s see what happens when we execute the program. First, we need to compile it, remembering to specify -lrt in order to link the POSIX real-time library into the executable, which contains the message queue API. In the first instance we’ll specify -i no, which means the cloned child process remains in the same IPC namespace as its parent, the global IPC namespace:

$ sudo ./invoke_ns -vi no
[sudo] password for wolf:
Parent: PID of parent is 11944
Parent: opening message queue /ipc_namespace
Parent: PID of child is 11945
 Child: PID of child is 11945
 Child: opening message queue /ipc_namespace

     Child: enter a message to send to the parent process (MAX 80 chars)
     >>

At this point, we need to type a message of up to 80 characters, perhaps ….

I will return to haunt you with peculiar piano riffs

After hitting return, the message is sent to the message queue, which the parent reads, and echoes to stdout. In this instance, because we have only specified the -i no option, the child process exits, which triggers the parent to close and remove the message queue, before it exits:

     Child: enter a message to send to the parent process (MAX 80 chars)
     >> I will return to haunt you with peculiar piano riffs

 Child: sending message to parent
 Child: closing message queue /ipc_namespace
Parent: received message from child

    Parent: the following message was received from the child
     >> I will return to haunt you with peculiar piano riffs

Parent: closing message queue /ipc_namespace
Parent: Removing message queue /ipc_namespace
Parent: ./invoke_ns - Finishing up

If we now run the program again, but this time specify -i yes as the command line option, the child process will be created in a new IPC namespace and will not be able to open the message queue, because it can’t see it:

$ sudo ./invoke_ns -vi yes
[sudo] password for wolf:
Parent: PID of parent is 12508
Parent: opening message queue /ipc_namespace
Parent: PID of child is 12509
 Child: PID of child is 12509
 Child: opening message queue /ipc_namespace
 Child: mq_open: No such file or directory

After 60 seconds, the parent’s wait for reading the message queue times out:

Parent: mq_timedreceive: Connection timed out
Parent: closing message queue /ipc_namespace
Parent: Removing message queue /ipc_namespace
Parent: ./invoke_ns - Finishing up

Each IPC namespace has its own message queue filesystem, which is only visible to processes residing in the same IPC namespace.

In the next article, we’ll look at cloning a process into several namespaces, along with a minimal chroot filesystem, in order to create a ‘rustic’ container.

Network Namespace

Wed, 14 Jan 2015 20:04:00 +0000

So far in this series, we’ve looked at isolating processes in PID, mount and UTS namespaces. The next namespace in this sequence is the network namespace, which allows you to isolate a process in terms of its network stack. That is, a process that is cast into a new network namespace using one of the system calls clone or unshare, or an existing network namespace with setns, are isolated from the host’s network stack, but enjoy their own private network stack associated with the namespace. The resources that are isolated include routes, firewall rules, network devices, and ports.

We’ve been developing a program to demonstrate how each of the namespaces work, and the changes that incorporate the network namespace can be found in invoke_ns4.c, which is available here. The changes that have been made from the last iteration are trivial; we’ve added a new command line option, -n, which specifies that the cloned process is to be created in a new network namespace, and the addition of the CLONE_NEWNET constant to the clone flags when parsing the command line options:

// Parse command line options and construct arguments
// to be passed to childFunction
while ((option = getopt(argc, argv, "+hvpmu:n")) != -1) {
    switch (option) {
        case 'n':
            flags |= CLONE_NEWNET;
            break;
        case 'u':
            flags |= CLONE_NEWUTS;
            args.hostname = malloc(sizeof(char) * (strlen(optarg) + 1));
            strcpy(args.hostname, optarg);
            break;
        case 'm':
            flags |= CLONE_NEWNS;
            break;
        case 'p':
            flags |= CLONE_NEWPID;
            break;
        case 'v':
            args.verbose = 1;
            break;
        case 'h':
            usage(argv[0]);
            exit(EXIT_SUCCESS);
        default:
            usage(argv[0]);
            exit(EXIT_FAILURE);
    }
}

Once the new version of the program has been compiled, we can invoke it with the following command line options and arguments:

$ sudo ./invoke_ns -vpmnu calculus env PS1="\h\[\] [\[\]\w\[\]] " PATH=$PATH bash --norc
[sudo] password for wolf:
Parent: PID of parent is 6142
Parent: PID of child is 6143
 Child: PID of child is 1
 Child: Executing command bash ...
calculus [/home/wolf]

The cloned process is created in a number of namespaces, including a new network namespace. Just as we’ve done in the previous examples, we can confirm that the process resides in its own network namespace, distinct from the global network namespace of all the other processes running on the system:

$ sudo ls -l /proc/6142/ns && sudo ls -l /proc/6143/ns
[sudo] password for wolf:
total 0
lrwxrwxrwx 1 root root 0 Mar 12 12:21 ipc -> ipc:[4026531839]
lrwxrwxrwx 1 root root 0 Mar 12 12:21 mnt -> mnt:[4026531840]
lrwxrwxrwx 1 root root 0 Mar 12 12:21 net -> net:[4026531956]
lrwxrwxrwx 1 root root 0 Mar 12 12:21 pid -> pid:[4026531836]
lrwxrwxrwx 1 root root 0 Mar 12 12:21 user -> user:[4026531837]
lrwxrwxrwx 1 root root 0 Mar 12 12:21 uts -> uts:[4026531838]
total 0
lrwxrwxrwx 1 root root 0 Mar 12 12:21 ipc -> ipc:[4026531839]
lrwxrwxrwx 1 root root 0 Mar 12 12:21 mnt -> mnt:[4026532430]
lrwxrwxrwx 1 root root 0 Mar 12 12:21 net -> net:[4026532434]
lrwxrwxrwx 1 root root 0 Mar 12 12:21 pid -> pid:[4026532432]
lrwxrwxrwx 1 root root 0 Mar 12 12:21 user -> user:[4026531837]
lrwxrwxrwx 1 root root 0 Mar 12 12:21 uts -> uts:[4026532431]

At the bash command prompt of the cloned process, we can see what the new network namespace provides the process:

calculus [/home/wolf] ip link show
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN mode DEFAULT qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

The kernel has simply provided a loopback interface, which we can bring up with ip link set lo up.

This is very spartan, and it is up to the initiator to configure a meaningful network stack for the namespace. Assuming that we want any processes running in the new network namespace to be able to communicate beyond the boundaries of the namespace, then we need to establish a ’tunnel’ between the global network namespace and our newly created network namespace.

To do this, we can create a pair of virtual ethernet interfaces (veth pair), one which will reside in the new network namespace, and one that will reside in the global network namespace. The pair of interfaces are linked to one another like a pipe, and whatever appears at one end, finds its way to the other end.

The veth pair is created in the global network namespace, and then we need to move one of the interfaces into our new network namespace. We’ll call the interface that will reside in the global network namespace, ‘host’, whilst the interface that is to be moved to the new network namespace will be called ‘guest’. In a bash command shell which resides in the global network namespace, we use the following to create the veth pair:

$ sudo ip link add host type veth peer name guest

When this is done, we can see the newly created veth pair by listing the interfaces available in the global network namespace:

$ ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT qlen 1000
    link/ether 08:00:27:fd:d7:6a brd ff:ff:ff:ff:ff:ff
3: guest@host: <BROADCAST,MULTICAST,M-DOWN> mtu 1500 qdisc noop state DOWN mode DEFAULT qlen 1000
    link/ether 4a:47:28:07:d4:d6 brd ff:ff:ff:ff:ff:ff
4: host@guest: <BROADCAST,MULTICAST,M-DOWN> mtu 1500 qdisc noop state DOWN mode DEFAULT qlen 1000
    link/ether de:7b:af:36:05:e3 brd ff:ff:ff:ff:ff:ff

In order to move the ‘guest’ interface into the new network namespace, we need to use the PID of the cloned child process, as it is in the global network namespace, that is 6143. The command simply moves the interface called ‘guest’ into the network namespace which PID 6143 resides in:

$ sudo ip link set guest netns 6143

We can check that this has been accomplished by issuing the ip link show command in both network namespaces. In the global network namespace:

$ ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT qlen 1000
    link/ether 08:00:27:fd:d7:6a brd ff:ff:ff:ff:ff:ff
4: host@if3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT qlen 1000
    link/ether de:7b:af:36:05:e3 brd ff:ff:ff:ff:ff:ff link-netnsid 0

And in the new network namespace:

calculus [/home/wolf] ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
3: guest@if4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT qlen 1000
    link/ether 4a:47:28:07:d4:d6 brd ff:ff:ff:ff:ff:ff link-netnsid 0

We can assign an IP address to each of the interfaces in the veth pair. In the new network namespace, we can execute the following:

calculus [/home/wolf] ip addr add 10.0.0.2/24 dev guest
calculus [/home/wolf] ip link set guest up
calculus [/home/wolf] ip addr show guest
3: guest@if4: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state LOWERLAYERDOWN qlen 1000
    link/ether 4a:47:28:07:d4:d6 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.0.0.2/24 scope global guest
       valid_lft forever preferred_lft forever

And in the global network namespace:

$ sudo ip addr add 10.0.0.1/24 dev host
[wolf@centos ~]$ sudo ip link set host up
[wolf@centos ~]$ ip addr show host
4: host@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
    link/ether de:7b:af:36:05:e3 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.0.0.1/24 scope global host
       valid_lft forever preferred_lft forever
    inet6 fe80::dc7b:afff:fe36:5e3/64 scope link 
       valid_lft forever preferred_lft forever

Finally, we can check that each interface is reachable from the other by pinging from each end.

In order to make a network namespace more useful, additional work would need to be carried out to establish a suitable network configuration for a specific purpose, e.g. creating a bridge on the host for bridging to a physical network interface.

In the next article, we’ll take a look at the IPC namespace.

Additional notes on network namespaces

Physical network interfaces can only exist in one network namespace at a time, and are returned to the global network namespace as and when the network namespace they reside in, terminates.

UTS Namespace

Tue, 30 Dec 2014 17:11:00 +0000

The UTS namespace is used to isolate two specific elements of the system that relate to the uname system call. UTS is an abbreviation of UNIX Time Sharing, a term that dates back to the fledgling days of UNIX, when multi-user, multi-tasking operating systems were a novelty.

Ken Thompson & Dennis Ritchie

The UTS namespace is named after the data structure used to store information returned by the uname system call. Specifically, the UTS namespace isolates the hostname and the NIS domain name. NIS, an abbreviation of Network Information Service, is an outdated directory service created by Sun Microsystems for the Solaris operating system, which was discontinued by Oracle after its acquisition of Sun Microsystems. In short, the UTS namespace is about isolating hostnames.

To demonstrate its use, the program we used in the last article about the mount namespace, has been further adapted as invoke_ns3.c, and is available here. Unsurprisingly, we’ve added a new command line option to our program, -u, which must be accompanied with a string that will be used to set the hostname in our newly created UTS namespace:

$ ./invoke_ns -h
Usage: ./invoke_ns [options] [cmd [arg...]]
Options can be:
    -h           display this help message
    -v           display verbose messages
    -p           new PID namespace
    -m           new MNT namespace
    -u hostname  new UTS namespace with associated hostname

We’ve added some code to parse the command line when -u is specified, which adds the CLONE_NEWUTS constant to the clone flags, and stores the hostname in some dynamically allocated memory. The pointer to the memory is passed as part of the args structure to the clone system call, which provides the arguments to childFunction:

// Parse command line options and construct arguments
// to be passed to childFunction
while ((option = getopt(argc, argv, "+hvpmu:")) != -1) {
    switch (option) {
        case 'u':
            flags |= CLONE_NEWUTS;
            args.hostname = malloc(sizeof(char) * (strlen(optarg) + 1));
            strcpy(args.hostname, optarg);
            break;
        case 'm':
            flags |= CLONE_NEWNS;
            break;
        case 'p':
            flags |= CLONE_NEWPID;
            break;
        case 'v':
            args.verbose = 1;
            break;
        case 'h':
            usage(argv[0]);
            exit(EXIT_SUCCESS);
        default:
            usage(argv[0]);
            exit(EXIT_FAILURE);
    }
}

In order to set the hostname inside the new UTS namespace, we need to add some code to childFunction using the sethostname system call:

// Set new hostname in UTS namespace if applicable
if (args->flags & CLONE_NEWUTS)
    if (sethostname(args->hostname, strlen(args->hostname)) == -1)
        perror(" Child: childFunction: sethostname");

Having compiled invoke_ns3.c, we can run the program with sudo ./invoke_ns -vpmu calculus env PS1='\h\[\] [\[\]\w\[\]] ' bash, which will set the hostname inside the new UTS namespace as ‘calculus’. As part of the command that’s specified for the cloned process to run, we’ve set the bash shell prompt (using the environment variable PS1) to include the hostname, which is, of course, ‘calculus’. We can also use the hostname command to echo the hostname set for the UTS namespace:

$ sudo ./invoke_ns -vpmu calculus env PS1='\h\[\] [\[\]\w\[\]] ' bash --norc
Parent: PID of parent is 16375
Parent: PID of child is 16376
 Child: PID of child is 1
 Child: executing command env ...
calculus [/home/wolf] hostname
calculus

Hence, it’s possible to provide processes running on a given host system, an entirely different identity in terms of hostname, provided they exist in their own unique UTS namespace.

The next article will cover the NET namespace.

Mount Namespace

Fri, 19 Dec 2014 11:49:00 +0000

In the last article about namespaces, we looked at the PID namespace. This time, we’ll take a look at the mount namespace.

Mount namespaces isolate a set of mount points for a process or processes in a given namespace, providing the opportunity for different processes on a system to have different views of the host’s filesystem. When a new mount namespace is created with either of the system calls clone or unshare, it is done so with a copy of the set of mounts that exist in the mount namespace where the calling process resides. Any subsequent changes to the set of mounts in the ‘parent’ or ‘child’ mount namespaces, are not reflected in the other …. unless the mount is a shared mount.

In the article where we discussed PID namespaces, we discovered that the process we cloned into a new PID namespace, appeared as if it resided in the PID namespace of its parent. This is because the parent and child processes share the same mount namespace and view of the filesystem, and when we listed the contents of /proc/$$/ns for the process in the new PID namespace (which has PID 1), we actually listed the namespaces for the ‘init’ process in the global PID namespace! That’s very confusing.

To get around this problem, as well as cloning our process into a new PID namespace, we’ll also create a new mount namespace for it at the same time, and mount a new procfs. The program we started with in the previous article, has been adapted as invoke_ns2.c, and can be downloaded here.

There are a couple of trivial changes; the first is the addition of the -m command line option to specify creation of a new mount namespace, and the second is the parsing of that option, which adds the CLONE_NEWNS constant to the clone flags:

// Parse command line options and construct arguments
// to be passed to childFunction
while ((option = getopt(argc, argv, "+hvpm")) != -1) {
    switch (option) {
        case 'm':
            flags |= CLONE_NEWNS;
            break;
        case 'p':
            flags |= CLONE_NEWPID;
            break;
        case 'v':
            args.verbose = 1;
            break;
        case 'h':
            usage(argv[0]);
            exit(EXIT_SUCCESS);
        default:
            usage(argv[0]);
            exit(EXIT_FAILURE);
    }
}

Notice that the clone flag constant for the mount namespace does not follow the convention associated with the constants for the other namespace types, i.e. CLONE_NEWTYPE. Instead of CLONE_NEWMNT, it is CLONE_NEWNS, and that’s because it was the first to appear in the kernel, and points to the notion that there may not have been other namespaces planned at that point in time.

The bigger change to the program, is the following addition to the childFunction, which gets called by the child process:

// Mount new proc instance in new mount namespace if and only if
// the child exists in both a new PID and mount namespace
if ((args->flags & CLONE_NEWPID) && (args->flags & CLONE_NEWNS)) {
    if (mount("none", "/proc", "", MS_REC|MS_PRIVATE, NULL) == -1)
        perror(" Child: childFunction: mount");
    if (mount("proc", "/proc", "proc", 0, NULL) == -1)
        perror(" Child: childFunction: mount");
}

This code makes the mount private to the new namespace (systemd makes all mounts ‘shared’ by default), and mounts a new procfs at /proc within the newly created mount namespace. Having compiled the modified code contained in invoke_ns2.c, we can run the program with the additional argument:

$ sudo ./invoke_ns -vpm env PS1="ns # " bash --norc
[sudo] password for wolf: 
Parent: PID of parent is 16179
Parent: PID of child is 16180
 Child: PID of child is 1
 Child: executing command env ...
ns #

On the face of it, nothing is any different to the version we ran when we were looking at PID namespaces. Listing the namespace directories for the parent and child in the global namespace yields:

$ sudo ls -l /proc/16179/ns && sudo ls -l /proc/16180/ns
[sudo] password for wolf: 
total 0
lrwxrwxrwx. 1 root root 0 Sep  7 13:07 ipc -> ipc:[4026531839]
lrwxrwxrwx. 1 root root 0 Sep  7 13:07 mnt -> mnt:[4026531840]
lrwxrwxrwx. 1 root root 0 Sep  7 13:07 net -> net:[4026531956]
lrwxrwxrwx. 1 root root 0 Sep  7 13:07 pid -> pid:[4026531836]
lrwxrwxrwx. 1 root root 0 Sep  7 13:07 user -> user:[4026531837]
lrwxrwxrwx. 1 root root 0 Sep  7 13:07 uts -> uts:[4026531838]
total 0
lrwxrwxrwx. 1 root root 0 Sep  7 13:07 ipc -> ipc:[4026531839]
lrwxrwxrwx. 1 root root 0 Sep  7 13:07 mnt -> mnt:[4026532118]
lrwxrwxrwx. 1 root root 0 Sep  7 13:07 net -> net:[4026531956]
lrwxrwxrwx. 1 root root 0 Sep  7 13:07 pid -> pid:[4026532173]
lrwxrwxrwx. 1 root root 0 Sep  7 13:07 user -> user:[4026531837]
lrwxrwxrwx. 1 root root 0 Sep  7 13:07 uts -> uts:[4026531838]

As expected, the parent and child processes reside in different PID and mount namespaces. When listing the namespaces from the bash command shell prompt of the child process within the new namespaces, however, we get what we expect this time:

ns # ls -l /proc/$$/ns
total 0
lrwxrwxrwx. 1 root root 0 Sep  7 13:09 ipc -> ipc:[4026531839]
lrwxrwxrwx. 1 root root 0 Sep  7 13:09 mnt -> mnt:[4026532118]
lrwxrwxrwx. 1 root root 0 Sep  7 13:09 net -> net:[4026531956]
lrwxrwxrwx. 1 root root 0 Sep  7 13:09 pid -> pid:[4026532173]
lrwxrwxrwx. 1 root root 0 Sep  7 13:09 user -> user:[4026531837]
lrwxrwxrwx. 1 root root 0 Sep  7 13:09 uts -> uts:[4026531838]

PID 1 within the new PID and mount namespaces, reports exactly the same as PID 16180 in the global namespaces; they are a representation of the same process. Additionally, if we do a process listing inside the new namespaces, we get just two processes, the bash command shell and ps itself:

ns # ps
  PID TTY          TIME CMD
    1 pts/0    00:00:00 bash
    3 pts/0    00:00:00 ps

In the next article, we’ll take a look at the UTS namespace.

PID Namespace

Sun, 14 Dec 2014 14:26:00 +0000

A PID namespace isolates process ID (PID) numbers, thereby allowing different processes running on the same Linux system to have the same PID. Processes with the same PID, don’t clash however, because they are isolated from each other by virtue of existing in different PID namespaces.

To demonstrate how they work, we’ll use a C program using the clone system call, which can be downloaded from GitHub. We’ll develop the program in future articles to demonstrate the other namespaces. Let’s take a look at the key parts of the first iteration of the program, invoke_ns1.c.

Firstly, having compiled it gcc -o invoke_ns ./invoke_ns1.c, use of the program can be established by executing ./invoke_ns -h. This provides:

Usage: ./invoke_ns [options] [cmd [arg...]]
Options can be:
    -h           display this help message
    -v           display verbose messages
    -p           new PID namespace

As well as the -h option, the program also takes a -p option to specify the creation of a new PID namespace for the new cloned process, and a -v option to specify verbosity, which is useful for understanding what is happening as the program executes. The program also takes an optional command and associated arguments to execute in the cloned process within the new namespace.

If the -p command line option is used, a flag is set with the CLONE_NEWPID constant, which will ultimately get passed to the clone system call:

// Parse command line options and construct arguments
// to be passed to childFunction
while ((option = getopt(argc, argv, "+hvp")) != -1) {  
    switch (option) {
        case 'p':
            flags |= CLONE_NEWPID;
            break;
        case 'v':
            args.verbose = 1;
            break;
        case 'h':
            usage(argv[0]);
            exit(EXIT_SUCCESS);
        default:
            usage(argv[0]);
            exit(EXIT_FAILURE);
    }
}

Once the command line options have been processed, if a command and associated arguments have been provided to the program at the command line, these are copied into some memory allocated for the purpose. The pointer to the memory block, args.command, along with some other information, will be another argument to the clone system call:

// Assemble command to be executed in namespace
if(optind != argc) {
    args.command = malloc(sizeof(char *) * (argc - optind + 1));
    for (i = optind; i < argc; i++) {
        args.command[i - optind] = malloc(strlen(argv[i]) + 1);
        strcpy(args.command[i - optind], argv[i]);
    }
    args.command[argc - optind] = NULL;
}

Just before the call to clone, we need to allocate a unique memory block to act as the stack for the child process that will get created, so that it doesn’t interfere with the stack associated with the parent:

// Allocate heap for child's stack
child_stack = malloc(STACK_SIZE);
if (child_stack == NULL) {
    perror("Parent: main: malloc");
    exit(EXIT_FAILURE);
}

Finally, we get to execute the clone system call:

// Clone child process
child = clone(childFunction, child_stack + STACK_SIZE, flags | SIGCHLD, &args);

The childFunction argument is a function that gets executed by the child process after it has been created. We’ll discuss this further in a moment. The other arguments include a pointer to the child’s stack, the ‘clone flags’, and our command and arguments and ancillary information, which will be passed to the childFunction.

Staying with the parent process for the moment, if the clone system call is successful, it waits for the child process to terminate, before the program exits:

// Wait for child to finish
if (waitpid(child, NULL, 0) == -1) {
    perror("Parent: main: waitpid");
    exit(EXIT_FAILURE);
}

In the meantime, the clone system call has created a new, child process within a new PID namespace, which calls the function childFunction:

// Function passed to the clone system call
int childFunction(void *child_args)
{
    struct arguments *args = child_args;

    if (args->verbose)
        printf(" Child: PID of child is %d\n", getpid());

    // Execute command if given
    if (args->command != NULL) {
        if (clearenv() != 0)
            fprintf(stderr, " Child: childFunction: couldn't clear environment\n");
        if (args->verbose)
            printf(" Child: executing command %s ...\n", args->command[0]);
        execvp(args->command[0], &args->command[0]);
    }
    else
        exit(EXIT_SUCCESS);

    perror(" Child: childFunction: execvp");
    exit(EXIT_FAILURE);
}

Its purpose is to simply execute the command and its arguments, if one has been provided at the command line. It does this using the execvp system call, which replaces the process image with a new one based on the command provided. That’s it! Let’s see how it works in practice.

First, some information regarding how to determine which namespace a process is running in. Each namespace for a given process is represented by a file in the procfs directory /proc/$$/ns where $$ is the PID of the process in question. For example:

$ ls -l /proc/$$/ns
total 0
lrwxrwxrwx 1 wolf wolf 0 Mar  9 15:29 ipc -> ipc:[4026531839]
lrwxrwxrwx 1 wolf wolf 0 Mar  9 15:29 mnt -> mnt:[4026531840]
lrwxrwxrwx 1 wolf wolf 0 Mar  9 15:29 net -> net:[4026531956]
lrwxrwxrwx 1 wolf wolf 0 Mar  9 15:29 pid -> pid:[4026531836]
lrwxrwxrwx 1 wolf wolf 0 Mar  9 15:29 user -> user:[4026531837]
lrwxrwxrwx 1 wolf wolf 0 Mar  9 15:29 uts -> uts:[4026531838]

The files are special symlinks bearing the namespace type (e.g. mnt), which point to inodes which represent unique namespaces, which the kernel reports as a conjugation of the namespace type and the inode number. Hence, if two different processes exist in the same namespace, their symlinks will point to the same inodes.

Every process exists in one namespace of each type, whether it is the global namespace (sometimes called the default or root namespace), or one that has been subsequently created using clone or unshare. You can test this out by starting two separate command shells and issuing the command ls -l /proc/$$/ns in each, and comparing the results - the output from each should be identical, as both processes belong in the global PID namespace.

Let’s create a new PID namespace for a cloned process using invoke_ns, and let’s ascertain whether the child process exists in a new PID namespace. We can create the new PID namespace specifying that the child process execute an interactive bash command shell:

$ sudo ./invoke_ns -vp env PS1="ns # " bash --norc
[sudo] password for wolf: 
Parent: PID of parent is 16034
Parent: PID of child is 16035
 Child: PID of child is 1
 Child: executing command env ...
ns #

Note that we have to be a privileged user to create the PID namespace, the CAP_SYS_ADMIN capability is required. We’ve created the new PID namespace and specified that an interactive bash command shell be executed using the command prompt ‘ns #’. The parent process reports that it has a PID of 16034, and that the cloned child process has a PID of 16035. The child, however, reports that it has a PID of 1, and is executing the command ‘env ...’. Although the child process exists in the newly created namespace, it also has a PID in the namespace from which it was created, in this case the global PID namespace.

In a bash command shell in the global PID namespace (i.e. not at the prompt ‘ns #’), we can list the namespaces for the parent (16034) and child (16035) processes:

$ sudo ls -l /proc/16034/ns && sudo ls -l /proc/16035/ns
[sudo] password for wolf: 
total 0
lrwxrwxrwx. 1 root root 0 Sep  7 10:52 ipc -> ipc:[4026531839]
lrwxrwxrwx. 1 root root 0 Sep  7 10:52 mnt -> mnt:[4026531840]
lrwxrwxrwx. 1 root root 0 Sep  7 10:52 net -> net:[4026531956]
lrwxrwxrwx. 1 root root 0 Sep  7 10:52 pid -> pid:[4026531836]
lrwxrwxrwx. 1 root root 0 Sep  7 10:52 user -> user:[4026531837]
lrwxrwxrwx. 1 root root 0 Sep  7 10:52 uts -> uts:[4026531838]
total 0
lrwxrwxrwx. 1 root root 0 Sep  7 10:52 ipc -> ipc:[4026531839]
lrwxrwxrwx. 1 root root 0 Sep  7 10:52 mnt -> mnt:[4026531840]
lrwxrwxrwx. 1 root root 0 Sep  7 10:52 net -> net:[4026531956]
lrwxrwxrwx. 1 root root 0 Sep  7 10:52 pid -> pid:[4026532118]
lrwxrwxrwx. 1 root root 0 Sep  7 10:52 user -> user:[4026531837]
lrwxrwxrwx. 1 root root 0 Sep  7 10:52 uts -> uts:[4026531838]

All of the namespaces are identical, apart from the PID namespaces, which indicates that the the parent and child processes are in different PID namespaces. A strange thing happens, however, if we list the namespaces for a process from within our bash command shell in the new PID namespace:

ns # ls -l /proc/$$/ns
total 0
lrwxrwxrwx 1 root root 0 Mar  9 17:03 ipc -> ipc:[4026531839]
lrwxrwxrwx 1 root root 0 Mar  9 17:03 mnt -> mnt:[4026531840]
lrwxrwxrwx 1 root root 0 Mar  9 17:03 net -> net:[4026531956]
lrwxrwxrwx 1 root root 0 Mar  9 17:03 pid -> pid:[4026531836]
lrwxrwxrwx 1 root root 0 Mar  9 17:03 user -> user:[4026531837]
lrwxrwxrwx 1 root root 0 Mar  9 17:03 uts -> uts:[4026531838]

From within the new PID namespace, listing the namespaces suggests that the child process still exists in the global PID namespace (4026531836). Clearly this is a contradiction, and we’ll explain why this is in the next article, which discusses the Mount namespace.

Additional notes on PID namespaces

When a new PID namespace is created, the first process that is created in the namespace is ‘init-like’ in nature, and is given the PID 1. It is ‘init-like’ in behaviour, because it assumes parenthood of any orphaned processes in the namespace, and when it terminates, all other processes which belong in the same namespace are terminated with the SIGKILL signal, and the namespace is subsequently removed.

PID namespaces can be nested up to a depth of 32, and any given process in a descendent namespace, will have as many different PIDs as it has ancestors + a PID for the namespace in which it resides. A PID in a descendent namespace can be ‘seen’ or operated on (subject to permissions) by processes in ancestor namespaces, and those in its own namespace, but not processes in sibling namespaces or descendent namespaces.

Introducing Namespaces

Mon, 08 Dec 2014 16:09:00 +0000

The container abstraction has made a big entry into the world of software development, offering a method for streamlined feature delivery. You don’t necessarily need to know about the inner workings of containers in order to use them, but it can help when it comes to getting the best from them.

In a series of articles I’ll be exploring one of the Linux kernel features that make containers possible; namespaces. Linux namespaces provide the isolation features that are so important to the container abstraction. The series uncovers the different namespaces in turn, but starts with a general intro:

Introducing Namespaces (this article)
PID Namespace
Mount Namespace
UTS Namespace
Network Namespace
IPC Namespace
A Basic Container (putting it all together)

A namespace is effectively a means of isolating a set of names or identifiers, and is very familiar to software developers, as the concept is widely used, either explicitly or implicitly, in many different programming languages. Linux namespaces isolate processes, so that different processes have a different view of various system resources, according to the namespaces they belong to.

Since the first namespace arrived in the Linux kernel in version 2.4.19 in 2002, as many as ten different namespaces have been proposed, but to date just six¹ have made it into the mainstream kernel. The namespaces that have been implemented are the Mount namespace, the UTS namespace, the PID namespace, the Network namespace, the IPC namespace, and the User namespace. Those missing from the original list are the security namespace, the security keys namespace, the device namespace, and the time namespace.

The following table shows when each namespace arrived in the mainstream Linux kernel:

Namespace	Clone Flag	Kernel	Year
Mount	CLONE_NEWNS	2.4.19	2002
UTS	CLONE_NEWUTS	2.6.19	2006
PID	CLONE_NEWPID	2.6.24	2008
Network	CLONE_NEWNET	2.6.29	2009
IPC	CLONE_NEWIPC	2.6.30	2009
User	CLONE_NEWUSER	3.8	2013

So, how do you place processes into namespaces? This is achieved with three different Linux system calls: clone, unshare and setns.

The clone system call is a general purpose function for creating a new process (‘child’) based on the calling process’ (‘parent’) context, which can be modified based on the flags passed, and can be used to specify a new namespace(s) in which the child is to exist.

The unshare system call is also a general purpose function, which can be used by a calling process to disassociate its shared context from other processes, and for our interest, place itself in a new namespace(s). In essence, the clone system call creates a new process, whereas the unshare system call doesn’t.

Finally, the setns system call allows a calling process to join a different, existing namespace, by specifying a file descriptor that relates to the namespace in question.

The next article looks at the PID namespace in more detail, and how the clone system call is used to create a new PID namespace.

Update 2020-10-21; the Linux kernel now carries a Cgroup and Time namespace. ↩︎

Mon, 01 Jan 0001 00:00:00 +0000

About Link to heading

Thanks for dropping by! I’m Nigel Brown, and I’m an independent trainer, author and consultant, with a particular interest in cloud native computing. I love technology and working out how things work, and my main aim here is to provide some technical insight and opinion into popular open source software projects.

I’m also for hire, and here are some of things I work on with cloud-native companies who I partner with.

Technical Writing Link to heading

I do a lot of technical writing in the form of blog articles, tutorials and technical guides. This usually involves picking apart a tool or an approach, and then describing how it works, and the benefits it provides (or otherwise). Sometimes I do this in my own name, and other times I ghostwrite the content on behalf of someone else.

We brought on Nigel as a 3rd party writer to assist with articles, checklists, and guides for our website blog that specializes in up-to-date Cloud-Native technologies. Nigel delivers 3-4 articles monthly covering very niche topics in the Cloud-Native ecosystem that we are able to use for education of our audience as well as capture leads from downloadable gated content. Nigel is very knowledgeable in our space and delivers work quickly and accurately. Our Developers never have changes to make to the content he produces, which is astounding. Highly recommended!

Tommy Hobin, Inbound Marketing Manager, Giant Swarm

Course Authoring Link to heading

I’ve written numerous technical training courses, both for myself and for partners. The material I create tends to relate to cloud-native technologies, and covers a number of different formats; slide creation, lab exercises, workshops, and video content.

Training Delivery Link to heading

As well as authoring course material, I also deliver technical training courses directly to my customers, as well as through the partners I work with. Training is delivered in classroom format, both on-site as well as in a virtual setting.

I really enjoy working with Nigel. Over the last few years he has contributed greatly to training delivery at CS with workshop facilitation and content creation. He is a really reliable, attentive and thoughtful collaborator, and we have built some truly great things together.

Ben Fakalogotoa, Training Delivery Manager, Container Solutions

If you’re short on bandwidth and need some reliable help with creating technical content, then I’d love to help! You can get in touch at nigel at windsock.io.

Mon, 01 Jan 0001 00:00:00 +0000

Here are the video-based training courses that I’ve created; they’re hosted on Pluralsight’s technology learning platform. It requires a subscription, but you can try it out with a free 10-day trial.

Cloud Native Training

Kubernetes Gateway API: Role-based Configuration of Ingress

This course is for anyone who needs to delegate the configuration of ingress between different teams using the Gateway API.
Kubernetes Gateway API: Getting Started

This course is for anyone who is involved in configuring external client access to applications running in Kubernetes clusters.
Getting Started with Kustomize

This course is for anyone who needs to deploy applications to Kubernetes clusters, but prefers patching YAML definitions to create variants, rather than writing complex templates to achieve the same.
Using GitOps to Automate Kubernetes Deployments with Flux 2

This course is for DevOps teams responsible for managing the deployments of cloud-native applications to Kubernetes clusters.
Developing Docker Apps: Core Principles

This course is for software developers who are looking to build on their basic understanding of application containers, to start to use Docker in earnest to build cloud native application services.
Automating Kubernetes Deployments Using a GitOps Workflow

This course is for DevOps teams responsible for managing the deployments of cloud-native applications to Kubernetes clusters.
Managing Ingress Traffic Patterns for Kubernetes Services

This course is for DevOps teams responsible for configuring Kubernetes clusters to expose workloads beyond the cluster.
Securing the Docker Platform

This course is for DevOps and IT Ops teams who are responsible for providing a secure platform for container operations.
Securing Docker Container Workloads

This course is for DevOps and IT Ops teams who have a responsibility for securing the software applications they deploy into production environments.
Containerizing a Software Application with Docker

This course is for software developers seeking to adopt the microservices architecture pattern, using the Docker platform as the packaging and distribution mechanism.