The previous article demonstrated how to implement cross-cluster Anthos Service Mesh (ASM) in a single network. In this article we will illustrate the mesh service communication using different VPC network. The use case will focus on setting up two GKE cluster with ASM installed in two different VPCs. As the clusters are in different networks, the services in a mesh will communicate with each other via internal east-west ingress gateway. The communication will happen over the Internet.
The mesh topology will look like the following:
Prerequisites
- You have already setup a Google project with two VPCs in separate regions. You can opt for any available two regions to implement this use case. For this article we will take Mumbai as one region and Singapore as another.
- You have Anthos GKE cluster setup in both the VPCs along with ASM version 1.8.3 installed. To understand how to install ASM, follow this link. You will also have to install istio operator that will represent ingress gateway to serve the east-west traffic across two clusters. See section Set up east-west gateway below.
- The ASM Certificate Authority (CA) used will be Mesh CA (only available for GKE clusters). You could also use Citadel CA as an alternate option.
Set the cluster context
As a first step, identify the context of each cluster. The below command will lists the different cluster context.
1 2 3 | kubectl config get-contexts -o namegke_sandbox-111111_asia_south1-a_cluster-1gke_sandbox-111111_asia_southeast1-a_cluster-2 |
The cluster context name follows a pattern: project-id_cluster-location_cluster_name. Assign the context name to $ctx1 and $ctx2 environment variables, each representing cluster one and two respectively.
1 | export ctx1=gke_sandbox-111111_asia_south1-a_cluster-1 export ctx2=gke_sandbox-111111_asia_southeast1-a_cluster-2 |
Setup endpoint discovery between clusters
In this step you will enable each cluster to discover service endpoints of their counterpart, so that cluster one will discover service endpoints of the second cluster and vice versa.
1 | istioctl x create-remote-secret --context=$ctx1 --name=cluster-1 | \ kubectl apply -f - --context=$ctx2 istioctl x create-remote-secret --context=$ctx2 --name=cluster-2 | \ kubectl apply -f - --context=$ctx1 |
You enable this by creating secrets for each cluster that grants access to kube API server of that cluster. Each secret is the certificate derived from the common root CA, in this case Mesh CA. You then apply the secret to the other cluster. In that way secrets are exchanged and the clusters are able to see the service endpoints of each other. In this case the endpoint of ingress gateway that will serve the east-west traffic will be discovered.
Set up east-west gateway
Before installing ASM, create an istio operator config file in both the clusters which will set up an ingress gateway for east-west traffic (cross-cluster). The Mumbai cluster will represent west side of the traffic while the Singapore cluster will represent east side of the traffic. The config file for the Mumbai cluster will look like the following:
01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 | apiVersion: install.istio.io/v1alpha1 kind: IstioOperator spec: values: global: meshID: mesh1 multiCluster: clusterName: cluster-1 network: vpc-1 components: ingressGateways: - name: istio-eastwestgateway label: istio: eastwestgateway app: istio-eastwestgateway topology.istio.io/network: vpc-1 enabled: true k8s: env: - name: ISTIO_META_ROUTER_MODE value: "sni-dnat" - name: ISTIO_META_REQUESTED_NETWORK_VIEW value: vpc-1 service: ports: - name: status-port port: 15021 targetPort: 15021 - name: tls port: 15443 targetPort: 15443 - name: tls-istiod port: 15012 targetPort: 15012 - name: tls-webhook port: 15017 targetPort: 15017 |
You can apply the above configuration as an overlay file while installing ASM using the -f flag
1 | istioctl install --context=$ctx1 -f istio-ingress-eastwest.yaml |
Important thing to note is the port 15443 which is used for cross cluster communication. In order to expose this port for mesh services to communicate, you will have to create a custom gateway that will use Service Name Indication (SNI) routing with TLS extension on this port. The below code depicts the custom gateway that will actually facilitate east-west routing.
01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 | apiVersion: networking.istio.io/v1alpha3 kind: Gateway metadata: name: multi-network namespace: istio-system spec: selector: istio: eastwestgateway servers: - port: number: 15443 name: tls protocol: TLS tls: mode: AUTO_PASSTHROUGH hosts: - "*.local" |
The above gateway named multi-network exposes port 15443 using SNI (with TLS extension) based routing allowing Kubernetes services with .local domain to communicate with Mumbai cluster mesh via this gateway. The TLS mode used is AUTO_PASSTHROUGH. This will, in effect, not terminate the TLS connection at the gateway and instead pass on the request to the target backend service. This mode also assumes that the client or the service from the calling cluster has initiated a TLS handshake. So if the service from the Singapore cluster uses plain vanilla http:// and not https://, the gateway will still pass through the request. This can create security loophole if the target backend does not handle TLS termination. We have kept it simple, as the purpose of this use case is to simply demonstrate cross cluster communication between two VPCs.
Apply both the above config file in the Singapore cluster (just change the name of the network) and you should be all set. To setup the sample application and test the service to service communication across clusters you can refer to the previous article.
Published on Java Code Geeks with permission by Rajeev Hathi, partner at our JCG program. See the original article here: Anthos ASM: Cross cluster mesh with different network Opinions expressed by Java Code Geeks contributors are their own. |
