featured image Kubernetes AppOps Security: Network Policies Part 1 - Good Practices

10/29/2019 by Johannes Schnatterer in Software Craftsmanship

Kubernetes AppOps Security: Network Policies Part 1 - Good Practices

+++You can download the original article (German), published in JavaSPEKTRUM 05/2019.+++

The Kubernetes network model may seem unusual to many: It requires a flat, non-hierarchical network in which everything (nodes, pods, kubelets, etc.) can communicate with everything else. The traffic in the cluster is not restricted by default, including even between namespaces. This is the point when traditional sysadmins ask in vain about network segments and firewalls. But Kubernetes offers the “network policy” resource that can be used to restrict network connections. The rules for incoming and outgoing traffic are declared in a network policy for ISO/OSI layers 3 and 4 (IP addresses and ports), and these rules are assigned to a group of pods using labels. The rules are then queried from the Kubernetes API and enforced by the Container Networking Interface (CNI) plugin (e.g., Calico, Cilium).

So, pod traffic is not restricted if

  • no network policies exist
  • the pod has no label that matches a network policy, or
  • the utilized CNI plugin does not implement network policies

Unauthorized network access that can be prevented using network policies

This default behavior is not ideal in terms of security. In particular, when multiple applications, teams or clients are running on one cluster, a single vulnerable application can provide an opening for attacks. Figure 1 provides an example: An attacker uses the vulnerable application for “Team B” to obtain customer data from the “Team A” database. Although it is not directly accessible via the internet, it is misconfigured and requires no authentication. Cases like these are not academic! For example, MongoDB instances have been operated thousands of times without authentication and were directly accessible via the internet.

Also, outgoing traffic can present a threat. For example, a compromised host might try to connect to a command server. This can be mitigated by blocking outgoing connections.

General capabilities and syntax

Network policies allow you to restrict traffic and thereby limit the impact of an attack. This allows inbound (“ingress”) and/or outbound (“egress”) traffic to be allowed for specific pods, entire namespaces or IP address ranges (egress only). Optionally, traffic can also be restricted by port.

As is customary with Kubernetes, the desired state of the cluster is defined declaratively in YAML. A simple example of a network policy is shown in Listing 1. The assignment of the pods is loosely tied to the respective label. This applies both to the pods to which the network policy is applied (“podSelector”) as well as to the specification of rules (which is below “podSelector” in the example, starting after “ingress”).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: db-allow
  namespace: team-a
spec:
  podSelector:
    matchLabels:
      db: a
  ingress:
  - from:
      - podSelector:
          matchLabels:
            app: a

Listing 1: Simple network policy that prevents unwanted access from Figure 1

The network policy in Listing 1 applies to all pods with the label “db: a”, and it only allows inbound traffic from pods that have the label “app: a”. In general, network policies apply to pods within the namespace in which they are applied (in Listing 1, “team-a”). Nevertheless, it is possible to specify “ingress” or “egress” rules that are applied to other namespaces (you can learn more about this in the next section).

At the beginning it takes a little bit of time to get used to reading the syntax. “Network policy recipes” will help introduce you to the topic: Here, network policy examples ranging from simple to complex use cases are vividly explained using animated GIFs.

Protecting individual pods as shown in Listing 1 is a start. But how can we ensure that no unwanted network accesses occur in larger clusters and complex application landscapes? One way to solve this problem will be shown below by way of the following examples. If you want to try it yourself in a defined environment, you will find complete examples with instructions in the “cloudogu/k8s-security-demos” repository on GitHub.

Whitelisting of incoming traffic

A sensible first step is the whitelisting of incoming traffic.

1
2
3
4
5
6
7
8
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: ingress-default-deny-all
  namespace: team-a
spec:
  podSelector: {}
ingress: []

Listing 2: Network policy that prohibits all communication with pods in a namespace

This can be implemented using a network policy for each namespace, which selects all pods from a namespace and blocks all incoming traffic. Listing 2 shows how it works: All pods are selected, but no allowed “ingress” rules are described. After that, pods within the namespace may not be accessed unless explicitly allowed, as Listing 1 shows.

If this is applied to each namespace (except initially for “kube-system”; see the next section for more information), only explicitly allowed connections between pods in the cluster can be established. This method produces a great effect with a manageable amount of effort.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: ingress-allow-traefik-to-access-app-a
  namespace:  team-a
spec:
  podSelector:
    matchLabels:
      app: a
  ingress:
  - ports:
      - port: 8080
    from:
      - namespaceSelector:
          matchLabels:
            namespace: kube-system
        podSelector:
          matchLabels:
            app: traefik

Listing 3: Network policy that allows inbound traffic from the ingress controller

However, what should not be forgotten is the whitelisting of traffic from

  • the ingress controller (for example NGINX or Traefik), since no further incoming requests can be made (Listing 3 demonstrates whitelisting for Traefik), and
  • the observability infrastructure, since otherwise Prometheus (Listing 4) will stop collecting application metrics, for example.

When selecting namespaces, please note that they have no labels by default. For example, to access listing 3, you need to add a “namespace: kube-system” label to the “kube-system” namespace. The advantage of assigning labels instead of names is that a label can also be assigned to multiple namespaces. This makes it possible, for example, to apply a network policy to all productive namespaces.

The simultaneous use of “namespaceSelector” and “podSelector”, as used in Listing 3 and 4, is only possible starting with Kubernetes version 1.11.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: ingress-allow-prometheus-to-access-db-a
  namespace: production
spec:
  podSelector:
    matchLabels:
      db: a
  ingress:
  - ports:
      - port: 9080
    from:
      - namespaceSelector:
          matchLabels:
            namespace: monitoring
        podSelector:
          matchLabels:
              app: prometheus
              component: server

Listing 4: Network policy that allows Prometheus to monitor an application

For advanced users: the “kube-system” namespace

As already mentioned, whitelisting the “kube-system” namespace requires some more attention than whitelisting other namespaces.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: allow-kube-dns-all-namespaces
  namespace: kube-system
spec:
  podSelector:
    matchLabels:
      k8s-app: kube-dns
  ingress:
    - ports:
      - protocol: UDP
        port: 53
      from:
        - namespaceSelector: {}

Listing 5: Network policy that allows the DNS service of kube-dns to be accessed

If any incoming traffic in the “kube-system” namespace is disallowed, this will affect, among other things:

  • name resolution with the DNS addon (for example, kube-dns)
  • and incoming traffic from the outside to the ingress controller, which is often operated in this namespace.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: ingress-allow-traefik-external
  namespace: kube-system
spec:
  podSelector:
    matchLabels:
      app: traefik
  ingress:
    - ports:
        - port: 80
        - port: 443
      from: []

Listing 6: Network policy that allows HTTP access to the Traefik ingress controller

Listing 5 shows how DNS access to the kube-dns pods can generally be authorized: UDP traffic is allowed on port 53 from all namespaces to the pods. Listing 6 shows an example of a Traefik HTTP reverse proxy, where accesses on ports 80 (this traffic is usually redirected to port 443) and 443 are allowed without needing to specify a concrete “from”.

For advanced users: Restricting outgoing traffic

If incoming traffic is whitelisted in all namespaces, unauthorized access can no longer take place within the cluster: The outbound traffic of a pod is also the inbound traffic of another pod that is whitelisted. However, there is yet another source of outgoing traffic that requires a closer look: Traffic from the cluster to surrounding networks, such as the internet or corporate network. Many applications do not need to communicate with the world outside the cluster, or they only need to communicate with selected hosts. Examples of permitted accesses are the loading of plugins (Jenkins, SonarQube, etc.) or access to authentication infrastructure (OAuth providers such as Facebook, GitHub, etc. or corporate LDAP/Active Directory). As described above, attackers can exploit an internet connection or access to a corporate network, which can be used as a gateway to other targets. These attacks can be prevented since Kubernetes version 1.8 through the restriction of “egress” traffic.

1
2
3
4
5
6
7
8
9
10
11
12
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: egress-allow-internal-only
  namespace: team-a
spec:
  policyTypes:
    - Egress
  podSelector: {}
  egress:
    - to:
    - namespaceSelector: {}

Listing 7: Network policy that allows outgoing traffic only within the cluster

Again, it is possible to apply the whitelisting procedure that we are familiar with from “ingress”. However, in that case all “ingress” rules would have to be repeated for the internal “egress” traffic. It is therefore more pragmatic to only allow outgoing traffic within the cluster with one network policy per namespace and then allow individual pods to access the outside. Listing 7 shows a network policy that does this by applying it to all pods and allowing “egress” to all namespaces. Everything outside the pod network has no namespace and is therefore rejected.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: egress-allow-all-egress
  namespace: team-a
spec:
  policyTypes:
    - Egress
  podSelector:
    matchLabels:
      app: a
  egress:
    - to:
        - ipBlock:
            cidr: 0.0.0.0/0

Listing 8: Network policy that allows all outbound traffic for specific pods

Whitelisting then takes place in a way that is similar to the “ingress” through additional network policies. However, the “egress” applies to IP address blocks using Classless Inter-Domain Routing (CIDR) notation. An example that allows all traffic for the specific pods is shown in Listing 8. Here you can make even more granular adjustments by allowing selected IP addresses or ranges.

When whitelisting the “egress”, it is important to not forget cloud-native applications that speak directly to the Kubernetes API. Since the API server is not in the pod network, it is not affected by the “namespaceSelector”. For pods that need it, it may be a good idea to whitelist the entire internal address space of the cluster, such as, for example, by using CIDR “10.0.0.0/8”. The implementation shows, however, that “egress” was added to the network policies only later, and it obviously does not yet have the same degree of maturity: For example, this does not work in the Calico CNI plugin. The only option to work around this problem is to allow all traffic for these pods, as shown in Listing 8.

Conclusion and recommendations

This article describes how standard Kubernetes network policies can be deployed with a manageable amount of effort to establish a solid layer of protection for the cluster.

However, the learning curve is not flat. Therefore, if you want to increase security with minimal effort, you should at least whitelist the “ingress” traffic in all namespaces except for the “kube-system”. This allows the namespaces to be isolated to a certain degree. In addition, it provides a good overview of who is communicating with whom in the cluster. Although network policies for the “kube-system” namespace and “egress” traffic are more complex, they improve security even more and are therefore also recommended.

For trying this out on your own, the “cloudogu/k8s-security-demos” repository offers an approach in which a cluster is protected step by step using the procedure described in this article.

Of course, there are pitfalls when using network policies, and we will provide some practical tips on testing and debugging in the next article of this series.


Johannes Schnatterer
Johannes Schnatterer

- Solution Architect -

Johannes has a passion for continuous delivery, open source and software quality. He believes that concise documentation can make a difference.