Chaos Testing

Apply some chaos engineering by throwing in some HTTP errors or network delays. Understanding failure scenarios is a critical aspect of microservices architecture (aka distributed computing)

Before Start

You should have NO virtualservice nor destinationrule (in tutorial namespace) kubectl get virtualservice kubectl get destinationrule if so run:

./scripts/clean.sh tutorial

HTTP Error 503

By default, recommendation v1 and v2 are being randomly load-balanced as that is the default behavior in Kubernetes/OpenShift

kubectl get pods -l app=recommendation -n tutorial

NAME                                  READY     STATUS    RESTARTS   AGE
recommendation-v1-3719512284-7mlzw   2/2       Running   6          18h
recommendation-v2-2815683430-vn77w   2/2       Running   0          3h

You can inject 503’s, for approximately 50% of the requests

kubectl create -f istiofiles/destination-rule-recommendation-v1-v2.yml -n tutorial

kubectl create -f istiofiles/virtual-service-recommendation-503.yml -n tutorial

./scripts/run.sh $GATEWAY_URL/customer

customer => Error: 503 - preference => Error: 503 - fault filter abort
customer => Error: 503 - preference => Error: 503 - fault filter abort
customer => Error: 503 - preference => Error: 503 - fault filter abort
customer => Error: 503 - preference => Error: 503 - fault filter abort

Clean up

kubectl delete -f istiofiles/virtual-service-recommendation-503.yml -n tutorial

Delay

The most insidious of possible distributed computing faults is not a "down" service but a service that is responding slowly, potentially causing a cascading failure in your network of services.

kubectl replace -f istiofiles/destination-rule-recommendation.yml -n tutorial

kubectl create -f istiofiles/virtual-service-recommendation-delay.yml -n tutorial

And hit the customer endpoint

./scripts/run.sh $GATEWAY_URL/customer

You will notice many requests to the customer endpoint now have a delay. If you are monitoring the logs for recommendation v1 and v2, you will also see the delay happens BEFORE the recommendation service is actually called

stern recommendation -n tutorial

Clean up

kubectl delete -f istiofiles/destination-rule-recommendation.yml -n tutorial

kubectl delete -f istiofiles/virtual-service-recommendation-delay.yml -n tutorial