Hi, We are facing one more issue with GoodData version 3.24.1. After upgrading the Kubernetes node, 2 out of 3 etcd pods are keep restarting. The log is attached below. Please let us know how to address this issue.

Hi Noushad, hopefully my colleague @Robert Moucha may have some insight and can help with this 🙂

This may occasionally happen, older etcd chart versions are susceptible to this issue. Etcd is designed to keep data as long as the majority of pods (2 of 3, in your case) are running. It cannot survive loss of 2 or more pods. In case it already happened, please do the following: 1. make sure the env variable ETCD_INITIAL_CLUSTER_STATE is set to

new

in gooddata-cn-etcd StatefulSet. If it is set to

existing

, patch the StatefulSet and set it to

new

. 2. delete all 3 PVCs belonging to etcd:

kubectl -n YOUR-NAMESPACE delete pvc  -l <http://app.kubernetes.io/component=etcd|app.kubernetes.io/component=etcd> --wait=false

. The argument

--wait=true

is important. 3. delete all etcd pods at once:

kubectl -n YOUR-NAMESPACE delete pod -l <http://app.kubernetes.io/component=etcd|app.kubernetes.io/component=etcd>

Pods will be deleted, along with their PVCs, and then created, and resulting cluster will be operational. 4. patch etcd StatefulSet and set ETCD_INITIAL_CLUSTER_STATE to

existing

Thank you @Robert Moucha 🙂 we will try it and update you

Sorry, wait=false is correct

Thank you, the solution you suggested worked 🙂