This is second in a series of posts exploring service to service call patterns in some of the application runtimes on Google Cloud. The first in the series explored service to service call patterns in GKE.
This post will expand on it by adding in a Service Mesh, specifically Anthos Service Mesh, and explore how the service to service patterns change in the presence of a mesh. The service to service call with be across services in a single cluster. The next post will explore services deployed to multiple GKE clusters.
Set-Up
The steps to set-up a GKE cluster and install Anthos service mesh on top of it is described in this document – https://cloud.google.com/service-mesh/docs/unified-install/install, in brief these are the commands that I had to run in my GCP Project to get a cluster running:
export PROJECT=$(gcloud config get-value project)
export ZONE="us-west1-a"
# Create a GKE Standard cluster named "solo"
gcloud container clusters create solo \
--project=${PROJECT} \
--zone=${ZONE} \
--enable-ip-alias \
--machine-type=e2-standard-4 \
--num-nodes=2 \
--workload-pool=${PROJECT}.svc.id.goog
# Download the asmcli utility
curl https://storage.googleapis.com/csm-artifacts/asm/asmcli_1.11 > asmcli
# Install service mesh on the newly created "solo" cluster
./asmcli install \
--project_id ${PROJECT} \
--cluster_name solo \
--cluster_location us-west1-a \
--fleet_id ${PROJECT} \
--enable_all \
--ca mesh_ca
# Determine the ASM revision
ASM_REVISION=$(kubectl get deploy -n istio-system -l app=istiod -o jsonpath={.items[*].metadata.labels.'istio\.io\/rev'}'{"\n"}')
# Install an ingress gateway that can work with the service mesh
kubectl create namespace gw-namespace
kubectl label namespace gw-namespace \
istio.io/rev=${ASM_REVISION} --overwrite
git clone https://github.com/GoogleCloudPlatform/anthos-service-mesh-packages.git
kubectl apply -n gw-namespace \
-f anthos-service-mesh-packages/samples/gateways/istio-ingressgateway
# Create a namespace to host applications with mesh proxy injected in
kubectl create namespace istio-apps
kubectl label namespace istio-apps istio-injection- istio.io/rev=${ASM_REVISION} --overwrite
If the installation of cluster and the mesh has run through cleanly, a good way to verify the installation is to see if the cluster gets registered as a Anthos managed cluster in the Google Cloud Console.
The services that I will be installing is fairly simple and looks like this:
Using a UI, the caller can make the producer behave in certain ways:
- Introduce response time delays
- Respond with certain status codes
This will help check how the mesh environment will behave in the face of these behaviors.
The codebase for the “caller” and “producer” are in
this repository – https://github.com/bijukunjummen/sample-service-to-service, there are kubernetes manifests available in the repository to bring up these services.
Behavior 1 – Mutual TLS
The first behavior that I want to see is for the the caller and the producer to verify each others identities by presenting and validating their certificates.
This can be done by adding in a istio DestinationRule for the producer, along these lines:
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: sample-producer-dl
namespace: istio-apps
spec:
host: sample-producer.istio-apps.svc.cluster.local
trafficPolicy:
tls:
mode: ISTIO_MUTUAL
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: sample-caller-dl
namespace: istio-apps
spec:
host: sample-caller.istio-apps.svc.cluster.local
trafficPolicy:
tls:
mode: ISTIO_MUTUAL
This also adds in the DestinationRule for the caller, this is because the caller gets the call from the browser via an
Ingress Gateway and even this call needs to be authenticated using mtls
Alright now that the set-up in place, the following is what gets captured as the request flows from the Browser to the Ingress Gateway to the Caller to the Producer.
The sign that the mTLS works is seeing the “x-forwarded-client-cert” header, this is in both the Callers headers coming in from Ingress-gateway, and in the “Producers” headers coming in from the Caller.
Behavior 2 – Timeout
The second behavior that I want to explore is the timeouts. A request timeout can be set for the call from the Caller to Producer by creating a Virtual Service for the Producer with the value set, along these lines:
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: sample-producer-route
namespace: istio-apps
spec:
hosts:
- "sample-producer.istio-apps.svc.cluster.local"
http:
- timeout: 5s
route:
- destination:
host: sample-producer
port:
number: 8080
With this configuration in place a request from the caller with a delay of 6 seconds, causes the Mesh to timeout and present an error that looks like this:
The mesh responds with a http status code of 504 with a message of “Upstream timed out”.
Behavior 3 – Circuit Breaker
Circuit breaker is implemented using a Destination Rule resource.
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: sample-producer-dl
namespace: istio-apps
spec:
host: sample-producer.istio-apps.svc.cluster.local
trafficPolicy:
tls:
mode: ISTIO_MUTUAL
connectionPool:
outlierDetection:
consecutive5xxErrors: 3
interval: 15s
baseEjectionTime: 15s
Here I have configuration which breaks the circuit if 3 continuous 5XX responses are received from the Producer in a 15 second interval, and then does not make a request for another 15 seconds
With this configuration in place a request with broken circuit looks like this:
The mesh responds with a http status code of 503 and a message of “no healthy upstream”
Conclusion
The neat thing is that in all scenarios so far, the way the Caller calls the Producer remains exactly the same, it is the mesh which injects in the appropriate security controls through mTLS and the resilience of calling service through timeouts and circuit breaker.
Published on Java Code Geeks with permission by Biju Kunjummen, partner at our JCG program. See the original article here: Service to Service call patterns – GKE with Anthos Service Mesh on a single cluster Opinions expressed by Java Code Geeks contributors are their own. |
