Connect a Kubernetes cluster

Ctadel reads your Kubernetes clusters through their API and installs nothing in them: it only lists and reads objects. The identity it uses depends on the cloud and is not always limited to reading; see the EKS case and the limits below. A managed cluster is discovered through the cloud account that hosts it; any other cluster is connected by providing a kubeconfig.

Two ways to connect a cluster

A provided kubeconfig takes precedence over the cloud account's credentials, including for a managed cluster that was already discovered.

Clusters discovered through a cloud account

Clusters appear under "Settings", tab "Integrations", sub-tab "Kubernetes" as soon as the cloud account's inventory has read them. For each one, Ctadel builds a temporary kubeconfig from the cloud API; that kubeconfig is not kept.

A cluster that is being created or deleted answers with an incomplete kubeconfig; the scan fails cleanly and runs again at the next inventory.

EKS and the ctadel-scanner group

On EKS, a kubeconfig is not enough: the cluster has to accept Ctadel's IAM identity. There are two paths.

Default path. The CloudFormation template of the Ctadel role has an EnableKubernetesAccess option, true by default. It grants eks:CreateAccessEntry, eks:DescribeAccessEntry and eks:AssociateAccessPolicy. At the first scan, Ctadel registers its own identity as an access entry of the cluster and associates the AmazonEKSClusterAdminPolicy policy with it. This path only works on clusters whose authentication mode is API or API_AND_CONFIG_MAP.

Recommended path. Set EnableKubernetesAccess to false and grant access yourself with a read-only role. This is what the "Instructions" button offers, shown on an EKS cluster in the "Needs Config" state:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: ctadel-scanner
rules:
  - apiGroups: [""]
    resources: [pods, nodes, namespaces, services, serviceaccounts, secrets, configmaps, limitranges, resourcequotas, persistentvolumes, persistentvolumeclaims]
    verbs: [get, list]
  - apiGroups: [apps]
    resources: [deployments, daemonsets, statefulsets, replicasets]
    verbs: [get, list]
  - apiGroups: [batch]
    resources: [cronjobs, jobs]
    verbs: [get, list]
  - apiGroups: [networking.k8s.io]
    resources: [ingresses, networkpolicies]
    verbs: [get, list]
  - apiGroups: [rbac.authorization.k8s.io]
    resources: [roles, rolebindings, clusterroles, clusterrolebindings]
    verbs: [get, list]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: ctadel-scanner
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: ctadel-scanner
subjects:
  - kind: Group
    name: ctadel-scanner
    apiGroup: rbac.authorization.k8s.io

Then map Ctadel's IAM identity to that group. The ARN to use is shown under "Error details:" in the same dialog.

Clusters in API or API_AND_CONFIG_MAP mode (the default):

aws eks create-access-entry --cluster-name <CLUSTER> --principal-arn <IAM_ARN> --kubernetes-groups ctadel-scanner

Clusters still in CONFIG_MAP mode, in the aws-auth ConfigMap of kube-system, under mapRoles or mapUsers:

- rolearn: <IAM_ARN>
  username: ctadel-scanner
  groups:
    - ctadel-scanner

Once access is granted, click "Rescan" on that cluster.

Connect a cluster with a kubeconfig

1. Open "Settings", tab "Integrations", sub-tab "Kubernetes".
2. Click "Connect Cluster". For a cluster that was already discovered, use "Connect with kubeconfig" on its row instead: its id is kept and its findings stay on one entry.
3. Fill in "Cluster Name", optionally "Cluster ID (optional)" and "Region (optional)", then pick the "Cloud Provider".
4. Paste the content into "Kubeconfig". The command kubectl config view --raw --minify prints the current context with its data inline.
5. Click "Connect Cluster", then "Scan Now".

The kubeconfig is encrypted (AES-256-GCM) before it is written to your instance's database. "Disconnect" erases it.

The identity carried by the kubeconfig should have the permissions of the ctadel-scanner role above, for example a service account bound to that ClusterRole, and nothing more.

What a kubeconfig may not contain

The kubeconfig is checked on entry, then at every scan. Anything that would run a command or read a file on your Ctadel instance is refused.

The file must define at least one cluster and one user. A refused kubeconfig is refused on entry, with the offending field named in the message.

What the scanner reads

The Kubernetes scanner only does get and list on the objects of the role above: pods, nodes, namespaces, services, service accounts, secrets, ConfigMaps, LimitRanges, ResourceQuotas, persistent volumes and their claims, Deployments, DaemonSets, StatefulSets, ReplicaSets, CronJobs, Jobs, Ingresses, NetworkPolicies, and every RBAC role and binding. Secrets are read for the checks that concern them and for the registry credentials of the pods (imagePullSecrets); those credentials never leave the Kubernetes scanner and are not kept after the scan.

Pod images

After the configuration pass, each distinct image found in the pods is analysed (packages, vulnerabilities, secrets in the layers), with a four-minute limit per image. To pull private images, the Kubernetes scanner assembles registry credentials for the duration of the scan, erased afterwards, made of:

• the pods' imagePullSecrets (types dockerconfigjson and dockercfg);
• the cloud account's credentials for its own registry, when no secret covers the host: an ECR token (AWS), the service principal's client id and secret on ACR (Azure, which requires the AcrPull role), the service account key on Artifact Registry and Container Registry (Google Cloud), the secret key on the Scaleway registry.

An image that cannot be pulled is counted and named in the result; it contributes no package and must not be read as a clean image.

Following a cluster's state

Each row carries a state: "Not Scanned", "Scanning...", "Connected", "Needs Config" or "Error", with the error message when there is one. The list refreshes on its own while a scan is running. "Needs Config" covers authorization refusals from the cluster, a failed EKS access entry registration, EKS clusters in CONFIG_MAP mode and a refused Azure service principal secret; everything else is "Error".

Limits

• The "Cloud Provider" list of the connect dialog offers AWS, Scaleway, GCP, Azure and "Other / On-premise"; for an OVHcloud MKS or OUTSCALE OKS cluster connected with a kubeconfig, pick "Other / On-premise", or connect it from its row if it was discovered.
• A cluster connected with a kubeconfig only, with no cloud account behind it, is not picked up by the automatic inventory. Run it again with "Rescan".
• On EKS, with EnableKubernetesAccess left at true, Ctadel's identity receives the cluster's AmazonEKSClusterAdminPolicy, not a read-only role. Set the option to false and apply the ctadel-scanner role to limit the permissions.
• On AKS, Ctadel asks for the cluster's administrator kubeconfig (listClusterAdminCredential); the service principal must be allowed to do so.
• The API server must be reachable from your Ctadel instance: no proxy or tunnel is accepted in the kubeconfig.
• An image whose analysis exceeds four minutes, or whose reference is not a plausible image reference, is marked as failed, with no package.
• EKS clusters in CONFIG_MAP mode require editing the aws-auth ConfigMap by hand.
• Runtime event detection inside clusters is in internal validation and not available to customers.