Kubernetes Made Easy for Cluster Admins

In my previous post I wrote about the challenges we faced moving some cloud applications to a new Kubernetes cluster, and described the process that led to us build a new Kubernetes management application, code named Harbor Pilot.

Harbor Pilot makes deployment of a new application on Kubernetes much easier for developers, while at the same time giving cluster admins tools and controls to make sure applications are being deployed in accordance with standards. This article goes a little deeper into the cluster admin's view of the world and describes how Harbor Pilot can make management of a Kubernetes cluster easier.

Recap: Developer Objectives

Harbor Pilot helps developers:

Run and manage applications in Kubernetes without having to learn a lot of new concepts and commands.
Easily provision and manage application dependencies (databases, utilities, etc.)
Do 1 and 2 first and then learn Kubernetes as time permits (maybe never)

This post covers the developer experience in detail including manage apps and dependencies across multiple environments:

and deploying new apps and/or dependencies from the catalog with one click:

That's cool for developers but as a cluster admin, how do I manage this catalog and control the way that the applications are deployed? Keep reading because this post is for you :).

Harbor Pilot Cluster Admin Objectives

Harbor Pilot is designed to help the cluster administrator:

Spend less time hand-holding developers and answering silly questions (addressed by developer objectives above)
Make sure that apps are deployed the way that I want them to be (the subject of this post)
Maximize the value my cluster delivers by minimizing wasted resources (the subject of a future post)

Configuration Concepts Preview

Harbor Pilot provides the cluster admin a set of simple but powerful concepts to control how apps are deployed:

Manage configuration information (including resources, parameters and overrides) at the cluster, tenant and environment levels
Manage a catalog of app types with configurations, dependencies and outputs
Define app configurations with user inputs, configuration values, environment variables, additional overrides, additional resources and exposed values

Layered Configuration

In Harbor Pilot, the cluster admin defines configuration information in layers. Most of the configuration is defined at the global (cluster-wide) level:

Additions can be made at the tenant level (a group of users and apps). In this example, the cluster admin added a "Charge Code" label override for the "WATD" tenant:

Additions can also be made at the environment (dev, test, prod for example) level. In this example, the cluster admin added a Service Level label to all manifests in the production environment:

Now let's look at each type of configuration information.

Configuration Resource

A configuration resource is a parametrized template for a Kubernetes manifest. It is analogous to a Helm chart template. Here is an example:

A configuration resource can be conditional. The persistent volume example above has this condition Persistent Storage = true which only includes the resource when the configuration requires persistent storage.

Notice that the configuration is JSON5 instead of YAML. We made this choice to simplify editing, syntax checking and automatic formatting. Don't worry if you find a YAML example you want to use. You can paste YAML in the field and it will automatically be parsed and converted into JSON5.

Notice that the storage property of the resource request has the value ${Disk Space}Gi'. "Disk Space" is a configuration parameter defined by the cluster administrator.

Configuration Parameters

A configuration parameter is an input to the configuration process. You can think of the configuration parameters like the Helm values file. Here is an example:

A configuration parameter has the following properties:

Name: The name of the parameter.
Question: A question that will be displayed when a user is prompted for a value.
Condition: An optional logical expression that defines when the question is included. For example, the user should should only be asked for disk space if the app requires persistent storage.
Type: The type of parameter: String, Number, or Boolean. Types like Image Name and Image Tag are on the roadmap.
Default: The default value for the parameter.
Category: The category that the question is presented in.
Secret: When checked, the value will be presented and stored in a secure way.
Optional: When checked, the parameter is not required.

Configuration Overrides

You may have noticed that the configuration resource above was a little "sparse". Important things like metadata.name were missing. To simplify resource editing and comprehension, configuration that is shared across multiple resource types is normalized out into a configuration override. A configuration override is analogous to Kustomize cross-cutting field. Here is a simple configuration override:

This applies the value of the App Type Name configuration parameter as a label. Notice the path expression:

*..metadata.labels[app.kubernetes.io/name]

It is JSONPath Plus syntax and here is the breakdown:

* matches all manifests
..metadata matches all metadata node at any level in the resource
.labels matches the labels node
[app.kubernetes.io/name] matches the property app.kubernetes.io/name

A configuration override can apply a value to many paths:

A configuration override can be conditional, apply a parametrized object and target specific manifest types:

App Types

The cluster admin defines a catalog of app types. Each app type has one or more app configuration:

In this example, the Postgres app type has three app configurations: one for a single node instance run inside the cluster, one for a Compose managed instance and one for an AWS Relational Database Service managed instance.

Notice that the Postgres app type also has a set of outputs. The app type makes these values available to other apps that use it as a dependency.

An app type can also have dependencies (which are other app types). Here is an example of a Chatwoot app type that depends on a Postgres database and a Redis cache:

User Inputs

An app configuration can have user inputs. When a user deploys an instance of the app configuration, the will be prompted for these inputs. Here are the user inputs for the Postgres Single Node configuration example:

Configuration Values

An app configuration has a configuration value for each configuration parameter defined at the global, tenant or environment level:

Environment Variables

An app configuration can have environment variables with parametrized values. The environment variables can be marked secret so that they are stored in a Kubernetes secret - all others are put in a Kubernetes config map. Here are some of the environment variables for the Postgres Single Node configuration example:

Additional Overrides

An app configuration can have additional overrides beyond the ones defined at the global, tenant and environment levels. In this example, the deployment strategy is changed to recreate so that we don't end up with two pods connected to the same persistent volume at the same time:

Exposed to Dependent Apps

Last but not least, an app configuration defines which environment variable is used to fulfill each app type outputs:

Summary