From 0dd95b0bce0c56f8bfdc4474f0604583e30889e7 Mon Sep 17 00:00:00 2001 From: Aaron Rueth Date: Mon, 5 Aug 2019 08:41:53 -0700 Subject: [PATCH 1/6] Adding docker0 and docker_gwbridge information UCP plan-installation section --- ee/ucp/admin/install/plan-installation.md | 49 ++++++++++++++++++++++- 1 file changed, 47 insertions(+), 2 deletions(-) diff --git a/ee/ucp/admin/install/plan-installation.md b/ee/ucp/admin/install/plan-installation.md index bf1e1b404f..6a946d17b0 100644 --- a/ee/ucp/admin/install/plan-installation.md +++ b/ee/ucp/admin/install/plan-installation.md @@ -42,11 +42,56 @@ this. ## Avoid IP range conflicts +### Engine + +There are two IP ranges used by the engine for the `docker0` and `docker_gwbridge` interface: + +#### docker0 + +By default, the Docker creates and configures the host system with a network interface called `docker0`, which is an ethernet bridge device. If you don’t specify a different network when starting a container, the container is connected to the bridge and all traffic coming from and going to the container flows over the bridge to the Docker engine, which handles routing on behalf of the container. + +Docker configures `docker0` with a configurable IP range. Containers which are connected to the default bridge are allocated IP addresses within this range. Certain default settings apply to `docker` unless you specify otherwise. The default subnet for `docker0` is `172.17.0.0/16` and the default maximum transmission unit (`MTU`) is `1500` bytes. + +The recommended way configure the `docker0` settings is to use the `daemon.json` file. You can specify one or more of the following settings to configure the `docker0` network: + +``` +{ + "bip": "172.17.0.1/16", + "fixed-cidr": "172.17.0.0/16", + "mtu": 1500 +} +``` + +`bip`: Supply a specific bridge IP range for the `docker0` interface, using standard CIDR notation. Default is `172.17.0.1/16`. + +`fixed-cidr`: Restrict the IP range for `docker0`, using standard CIDR notation. Default is `172.17.0.0/16`. +This range must be an IPv4 range for fixed IPs, and must be a subset of the bridge IP range (`bip` in `daemon.json`). For example, with `172.17.0.0/17`, IPs for your containers will be chosen from the first half of addresses(`172.17.0.1` - `172.17.127.254`) included in the `bip`(`172.17.0.0/16`) subnet. + +`mtu`: Set the maximum packet size in bytes for `docker0`. + +#### docker_gwbridge + + The `docker_gwbridge` is a virtual bridge that connects the overlay networks (including the `ingress` network) to an individual Docker engine's physical network. Docker creates it automatically when you initialize a swarm or join a Docker host to a swarm, but it is not a Docker device. It exists in the kernel of the Docker host. The default subnet for `docker_gwbridge` is `172.18.0.0/16`. + + If you need to customize its settings, you must do so before joining the Docker host to the swarm, or after temporarily removing the host from the swarm. The recommended way to configure the `docker_gwbridge` settings is to use the `daemon.json` file. You can specify one or more of the following settings to configure the network: + + ``` + { + "default-address-pools": [ + {"base":"172.18.0.0/16","size":16} + ] + } + ``` + +`default-address-pools`: Set the default address pools for local node networks. + +### Swarm + Swarm uses a default address pool of `10.0.0.0/8` for its overlay networks. If this conflicts with your current network implementation, please use a custom IP address pool. To specify a custom IP address pool, use the `--default-address-pool` command line option during [Swarm initialization](../../../../engine/swarm/swarm-mode.md). > **Note**: Currently, the UCP installation process does not support this flag. To deploy with a custom IP pool, Swarm must first be installed using this flag and UCP must be installed on top of it. -### Kubernetes IP Range Conflicts +### Kubernetes There are 2 internal IP ranges used within Kubernetes that may overlap and conflict with the underlying infrastructure: @@ -54,7 +99,7 @@ conflict with the underlying infrastructure: - The Pod Network - Each Pod in Kubernetes is given an IP address from either the Calico or Azure IPAM services. In a default installation Pods are given IP addresses on the `192.168.0.0/16` range. This can be customised at install - time using the `--pod-cidr` flag. + time using the `--pod-cidr` flag. - The Services Network - When a user exposes a Service in Kubernetes it is accesible via a VIP, this VIP comes from a Cluster IP Range. By default on UCP From ab81d0e43b24fcecee37ab853edbc4bad9a8fe13 Mon Sep 17 00:00:00 2001 From: Aaron Rueth Date: Tue, 13 Aug 2019 07:52:23 -0700 Subject: [PATCH 2/6] Updates from review --- ee/ucp/admin/install/plan-installation.md | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-) diff --git a/ee/ucp/admin/install/plan-installation.md b/ee/ucp/admin/install/plan-installation.md index 6a946d17b0..468dc0e566 100644 --- a/ee/ucp/admin/install/plan-installation.md +++ b/ee/ucp/admin/install/plan-installation.md @@ -45,14 +45,14 @@ this. ### Engine There are two IP ranges used by the engine for the `docker0` and `docker_gwbridge` interface: - + #### docker0 -By default, the Docker creates and configures the host system with a network interface called `docker0`, which is an ethernet bridge device. If you don’t specify a different network when starting a container, the container is connected to the bridge and all traffic coming from and going to the container flows over the bridge to the Docker engine, which handles routing on behalf of the container. +By default, the Docker engine creates and configures the host system with a network interface called `docker0`, which is an ethernet bridge device. If you don't specify a different network when starting a container, the container is connected to the bridge and all traffic coming from and going to the container flows over the bridge to the Docker engine, which handles routing on behalf of the container. -Docker configures `docker0` with a configurable IP range. Containers which are connected to the default bridge are allocated IP addresses within this range. Certain default settings apply to `docker` unless you specify otherwise. The default subnet for `docker0` is `172.17.0.0/16` and the default maximum transmission unit (`MTU`) is `1500` bytes. +Docker engine creates `docker0` with a configurable IP range. Containers which are connected to the default bridge are allocated IP addresses within this range. Certain default settings apply to `docker` unless you specify otherwise. The default subnet for `docker0` is `172.17.0.0/16` and the default maximum transmission unit (`MTU`) is `1500` bytes. -The recommended way configure the `docker0` settings is to use the `daemon.json` file. You can specify one or more of the following settings to configure the `docker0` network: +The recommended way to configure the `docker0` settings is to use the `daemon.json` file. You can specify one or more of the following settings to configure the `docker0` interface: ``` { @@ -73,7 +73,9 @@ This range must be an IPv4 range for fixed IPs, and must be a subset of the brid The `docker_gwbridge` is a virtual bridge that connects the overlay networks (including the `ingress` network) to an individual Docker engine's physical network. Docker creates it automatically when you initialize a swarm or join a Docker host to a swarm, but it is not a Docker device. It exists in the kernel of the Docker host. The default subnet for `docker_gwbridge` is `172.18.0.0/16`. - If you need to customize its settings, you must do so before joining the Docker host to the swarm, or after temporarily removing the host from the swarm. The recommended way to configure the `docker_gwbridge` settings is to use the `daemon.json` file. You can specify one or more of the following settings to configure the network: + > **Note**: If you need to customize the `docker_gwbridge` settings, you must do so before joining the host to the swarm, or after temporarily removing the host from the swarm. + + The recommended way to configure the `docker_gwbridge` settings is to use the `daemon.json` file. You can specify one or more of the following settings to configure the interface: ``` { @@ -87,7 +89,7 @@ This range must be an IPv4 range for fixed IPs, and must be a subset of the brid ### Swarm -Swarm uses a default address pool of `10.0.0.0/8` for its overlay networks. If this conflicts with your current network implementation, please use a custom IP address pool. To specify a custom IP address pool, use the `--default-address-pool` command line option during [Swarm initialization](../../../../engine/swarm/swarm-mode.md). +Swarm uses a default address pool of `10.0.0.0/8` for its overlay networks. If this conflicts with your current network implementation, please use a custom IP address pool. To specify a custom IP address pool, use the `--default-address-pool` command line option during [Swarm initialization](../../../../engine/swarm/swarm-mode.md). > **Note**: Currently, the UCP installation process does not support this flag. To deploy with a custom IP pool, Swarm must first be installed using this flag and UCP must be installed on top of it. @@ -98,7 +100,7 @@ conflict with the underlying infrastructure: - The Pod Network - Each Pod in Kubernetes is given an IP address from either the Calico or Azure IPAM services. In a default installation Pods are given - IP addresses on the `192.168.0.0/16` range. This can be customised at install + IP addresses on the `192.168.0.0/16` range. This can be customized at install time using the `--pod-cidr` flag. - The Services Network - When a user exposes a Service in Kubernetes it is From 6e7614d3f5d8ad3035f7167a17b6c46c693f0095 Mon Sep 17 00:00:00 2001 From: Aaron Rueth Date: Tue, 13 Aug 2019 08:01:20 -0700 Subject: [PATCH 3/6] Cleaned up markdown --- ee/ucp/admin/install/plan-installation.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/ee/ucp/admin/install/plan-installation.md b/ee/ucp/admin/install/plan-installation.md index 468dc0e566..0bd4deca06 100644 --- a/ee/ucp/admin/install/plan-installation.md +++ b/ee/ucp/admin/install/plan-installation.md @@ -54,7 +54,7 @@ Docker engine creates `docker0` with a configurable IP range. Containers which a The recommended way to configure the `docker0` settings is to use the `daemon.json` file. You can specify one or more of the following settings to configure the `docker0` interface: -``` +```json { "bip": "172.17.0.1/16", "fixed-cidr": "172.17.0.0/16", @@ -77,7 +77,7 @@ This range must be an IPv4 range for fixed IPs, and must be a subset of the brid The recommended way to configure the `docker_gwbridge` settings is to use the `daemon.json` file. You can specify one or more of the following settings to configure the interface: - ``` + ```json { "default-address-pools": [ {"base":"172.18.0.0/16","size":16} @@ -98,13 +98,13 @@ Swarm uses a default address pool of `10.0.0.0/8` for its overlay networks. If t There are 2 internal IP ranges used within Kubernetes that may overlap and conflict with the underlying infrastructure: -- The Pod Network - Each Pod in Kubernetes is given an IP address from either +* The Pod Network - Each Pod in Kubernetes is given an IP address from either the Calico or Azure IPAM services. In a default installation Pods are given IP addresses on the `192.168.0.0/16` range. This can be customized at install time using the `--pod-cidr` flag. -- The Services Network - When a user exposes a Service in Kubernetes it is - accesible via a VIP, this VIP comes from a Cluster IP Range. By default on UCP +* The Services Network - When a user exposes a Service in Kubernetes it is + accessible via a VIP, this VIP comes from a Cluster IP Range. By default on UCP this range is `10.96.0.0/16`. From UCP 3.1.8 and onwards this value can be changed at install time with the `--service-cluster-ip-range` flag. @@ -143,8 +143,8 @@ DTR, your load balancer needs to distinguish traffic between the two by IP address or port number. * If you want to configure your load balancer to listen on port 443: - * Use one load balancer for UCP and another for DTR, - * Use the same load balancer with multiple virtual IPs. + * Use one load balancer for UCP and another for DTR, + * Use the same load balancer with multiple virtual IPs. * Configure your load balancer to expose UCP or DTR on a port other than 443. If you want to install UCP in a high-availability configuration that uses @@ -180,6 +180,6 @@ manager nodes joining the cluster or being promoted to a manager role. ## Where to go next -- [System requirements](system-requirements.md) -- [Install UCP](index.md) +* [System requirements](system-requirements.md) +* [Install UCP](index.md) From 59fe504b3fbf7b0ea77a34233ef57656ca4becd9 Mon Sep 17 00:00:00 2001 From: Aaron Rueth Date: Mon, 19 Aug 2019 10:08:21 -0700 Subject: [PATCH 4/6] Made updates based on customer feedback --- ee/ucp/admin/install/plan-installation.md | 26 ++++++++++++++++------- 1 file changed, 18 insertions(+), 8 deletions(-) diff --git a/ee/ucp/admin/install/plan-installation.md b/ee/ucp/admin/install/plan-installation.md index 0bd4deca06..1c9ea51193 100644 --- a/ee/ucp/admin/install/plan-installation.md +++ b/ee/ucp/admin/install/plan-installation.md @@ -42,6 +42,12 @@ this. ## Avoid IP range conflicts +Engine `fixed-cidr` - CIDR range for `docker0` interface and local containers, default `172.17.0.0/16`. +Engine `default-address-pools` - CIDR range for `docker_gwbridge` interface and bridge networks, default `172.18.0.0/16`. +Swarm `default-addr-pool` - CIDR range for Swarm overlay networks, default `10.0.0.0/8`. +Kubernetes `pod-cidr` - CIDR range for Kubernetes pods, default `192.168.0.0/16`. +Kubernetes `service-cluster-ip-range` - CIDR range for Kubernetes services + ### Engine There are two IP ranges used by the engine for the `docker0` and `docker_gwbridge` interface: @@ -50,7 +56,7 @@ There are two IP ranges used by the engine for the `docker0` and `docker_gwbridg By default, the Docker engine creates and configures the host system with a network interface called `docker0`, which is an ethernet bridge device. If you don't specify a different network when starting a container, the container is connected to the bridge and all traffic coming from and going to the container flows over the bridge to the Docker engine, which handles routing on behalf of the container. -Docker engine creates `docker0` with a configurable IP range. Containers which are connected to the default bridge are allocated IP addresses within this range. Certain default settings apply to `docker` unless you specify otherwise. The default subnet for `docker0` is `172.17.0.0/16` and the default maximum transmission unit (`MTU`) is `1500` bytes. +Docker engine creates `docker0` with a configurable IP range. Containers which are connected to the default bridge are allocated IP addresses within this range. Certain default settings apply to `docker` unless you specify otherwise. The default subnet for `docker0` is `172.17.0.0/16`. The recommended way to configure the `docker0` settings is to use the `daemon.json` file. You can specify one or more of the following settings to configure the `docker0` interface: @@ -58,7 +64,6 @@ The recommended way to configure the `docker0` settings is to use the `daemon.js { "bip": "172.17.0.1/16", "fixed-cidr": "172.17.0.0/16", - "mtu": 1500 } ``` @@ -67,8 +72,6 @@ The recommended way to configure the `docker0` settings is to use the `daemon.js `fixed-cidr`: Restrict the IP range for `docker0`, using standard CIDR notation. Default is `172.17.0.0/16`. This range must be an IPv4 range for fixed IPs, and must be a subset of the bridge IP range (`bip` in `daemon.json`). For example, with `172.17.0.0/17`, IPs for your containers will be chosen from the first half of addresses(`172.17.0.1` - `172.17.127.254`) included in the `bip`(`172.17.0.0/16`) subnet. -`mtu`: Set the maximum packet size in bytes for `docker0`. - #### docker_gwbridge The `docker_gwbridge` is a virtual bridge that connects the overlay networks (including the `ingress` network) to an individual Docker engine's physical network. Docker creates it automatically when you initialize a swarm or join a Docker host to a swarm, but it is not a Docker device. It exists in the kernel of the Docker host. The default subnet for `docker_gwbridge` is `172.18.0.0/16`. @@ -80,18 +83,25 @@ This range must be an IPv4 range for fixed IPs, and must be a subset of the brid ```json { "default-address-pools": [ - {"base":"172.18.0.0/16","size":16} + {"base":"172.18.0.0/16","size":24}, + {"base":"###.###.###.###/##","size":##} ] } ``` -`default-address-pools`: Set the default address pools for local node networks. +`default-address-pools`: A list of IP address pools for local bridge networks, the default is a single pool `{"base":"172.18.0.0/16","size":24}`. This allocates `/24` network from the `172.18.0.0/16` CIDR range for local bridge networks. Each entry in the list contain the following: + +`base`: CIDR range to be divided up for bridge networks, the default is `172.18.0.0/16` + +`size`: CIDR netmask that determines the default overlay network size to allocate from the `base` pool, the default is `24` ### Swarm -Swarm uses a default address pool of `10.0.0.0/8` for its overlay networks. If this conflicts with your current network implementation, please use a custom IP address pool. To specify a custom IP address pool, use the `--default-address-pool` command line option during [Swarm initialization](../../../../engine/swarm/swarm-mode.md). +Swarm uses a default address pool of `10.0.0.0/8` for its overlay networks. If this conflicts with your current network implementation, please use a custom IP address pool. To specify a custom IP address pool, use the `--default-addr-pool` command line option during [Swarm initialization](../../../../engine/swarm/swarm-mode.md). -> **Note**: Currently, the UCP installation process does not support this flag. To deploy with a custom IP pool, Swarm must first be installed using this flag and UCP must be installed on top of it. +> **Note**: The Swarm `default-addr-pool` setting is separate from the Docker engine `default-address-pools` setting. They are two separate ranges that are used for different purposes. + +> **Note**: Currently, the UCP installation process does not support this flag. To deploy with a custom IP pool, Swarm must first be initialized using this flag and UCP must be installed on top of it. ### Kubernetes From e42615d6ccaab9ca74a9fefbee03fe12c54bce47 Mon Sep 17 00:00:00 2001 From: Aaron Rueth Date: Mon, 19 Aug 2019 10:11:52 -0700 Subject: [PATCH 5/6] Added default for service-cluster-ip-range --- ee/ucp/admin/install/plan-installation.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ee/ucp/admin/install/plan-installation.md b/ee/ucp/admin/install/plan-installation.md index 1c9ea51193..a4d3047fa1 100644 --- a/ee/ucp/admin/install/plan-installation.md +++ b/ee/ucp/admin/install/plan-installation.md @@ -46,7 +46,7 @@ Engine `fixed-cidr` - CIDR range for `docker0` interface and local containers, d Engine `default-address-pools` - CIDR range for `docker_gwbridge` interface and bridge networks, default `172.18.0.0/16`. Swarm `default-addr-pool` - CIDR range for Swarm overlay networks, default `10.0.0.0/8`. Kubernetes `pod-cidr` - CIDR range for Kubernetes pods, default `192.168.0.0/16`. -Kubernetes `service-cluster-ip-range` - CIDR range for Kubernetes services +Kubernetes `service-cluster-ip-range` - CIDR range for Kubernetes services, default `10.96.0.0/16`. ### Engine From f16a7ad1d932d7ace92ca35399cca26b1d70ca33 Mon Sep 17 00:00:00 2001 From: Aaron Rueth Date: Tue, 20 Aug 2019 09:46:23 -0700 Subject: [PATCH 6/6] Removed overlay reference in bridge network section --- ee/ucp/admin/install/plan-installation.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ee/ucp/admin/install/plan-installation.md b/ee/ucp/admin/install/plan-installation.md index a4d3047fa1..97bbb34606 100644 --- a/ee/ucp/admin/install/plan-installation.md +++ b/ee/ucp/admin/install/plan-installation.md @@ -93,7 +93,7 @@ This range must be an IPv4 range for fixed IPs, and must be a subset of the brid `base`: CIDR range to be divided up for bridge networks, the default is `172.18.0.0/16` -`size`: CIDR netmask that determines the default overlay network size to allocate from the `base` pool, the default is `24` +`size`: CIDR netmask that determines the default network size to allocate from the `base` pool, the default is `24` ### Swarm