UPDATE: The issue seems to have resolved itself. H...
# gooddata-cn
UPDATE: The issue seems to have resolved itself. Had and upgrading issue After performing the v1.6.0 upgrade (from 1.5.1, first upgrading to pulsar 2.7.8 (from 2.7.2), then running upgrade for GD.CN 1.6.0, seeing dashboards rendering errors like the image below. In Chrome network tab, seeing net::ERR_NETWORK_CHANGED and http error 500 and 502 in API responses. Any ideas?
This seems to be resolved now. After 15 minutes, we still saw this problem. Running
kubectl get pods --namespace gooddata-cn
listed some pods in
state, rest in
. After the 15 minutes I rolled back to 1.5.1, still saw the same problem, upgraded back to 1.6. Now it looks fine... So probably just should have waited until some transient processes finish.
@Robert Moucha @Milan Sladký this should not happen during upgrade, right? Can we investigate it in any way?
Right, it should not happen. The Helm tool performs rolling upgrade of all Deployment entities, so it adds a new pod with new image, waits until it comes up, terminates one of older pods and then continues the same way with the remaining pods. The "Terminating" state is transient - it means that this is older pod and no traffic should be directed to it, and it will eventually disappear. While it may take several minutes to perform upgrade (depending on cluster workload and network conditions), 15 minutes seem to be way too long. BTW,
helm upgrade
parameter that waits until the whole process is done. It's recommended to use it together with
--timeout NN
(wait upto NN minutes).