https://www.gooddata.com/ logo
Join Slack
Powered by
UPDATE: The issue seems to have resolved itself. H...
# gooddata-cn
v
UPDATE: The issue seems to have resolved itself. Had and upgrading issue After performing the v1.6.0 upgrade (from 1.5.1, first upgrading to pulsar 2.7.8 (from 2.7.2), then running upgrade for GD.CN 1.6.0, seeing dashboards rendering errors like the image below. In Chrome network tab, seeing net::ERR_NETWORK_CHANGED and http error 500 and 502 in API responses. Any ideas?
This seems to be resolved now. After 15 minutes, we still saw this problem. Running 
kubectl get pods --namespace gooddata-cn
listed some pods in 
Terminating
state, rest in 
Running
. After the 15 minutes I rolled back to 1.5.1, still saw the same problem, upgraded back to 1.6. Now it looks fine... So probably just should have waited until some transient processes finish.
j
@Robert Moucha @Milan Sladký this should not happen during upgrade, right? Can we investigate it in any way?
r
Right, it should not happen. The Helm tool performs rolling upgrade of all Deployment entities, so it adds a new pod with new image, waits until it comes up, terminates one of older pods and then continues the same way with the remaining pods. The "Terminating" state is transient - it means that this is older pod and no traffic should be directed to it, and it will eventually disappear. While it may take several minutes to perform upgrade (depending on cluster workload and network conditions), 15 minutes seem to be way too long. BTW, 
helm upgrade
supports 
--wait
parameter that waits until the whole process is done. It's recommended to use it together with 
--timeout NN
(wait upto NN minutes).
Open in Slack
PreviousNext
Powered by