So we are running into an issue with gooddata-cn where we are not able to get data back from our BigQuery Data Source Connection. and the UI keeps hanging on us It is able to successfully make the LDM from the datasource. So we know they keyfile and simba drivers were installed correctly. We tried to use the analyze functionality in the GUI to see some data but we get an error.( a 500 from https://gooddata-cn.mydomain.com/api/actions/workspaces/999999ffffffff0fcbd99af4fedd1ca99/execution/afm/execute ) When looking in the GCP console we see query requests related to pulling the Dataset Schema, but nothing related to actually pulling in data from specific tables. And now we cant even get into a workspace. we can only login, then select a workspace and then we get a never ending spinning bar. The js console is showing 500 errors when trying to hit: https://gooddata-cn.mydomain.com/api/profile as well as Message: {“status”: 500, “detail”: “Server-side problem. Contact support.“, “traceId”: “378a0b90b9dee3d2"} the gooddata-cn and pulsar pods are all up and running, and when looking through their logs we dont really find any ERROR level logs that would suggest there is an issue. But the gooddata-cn-metadata-api-xxxx pods have been restarted >20 times in the last 4 days. Are there any good docs in how to debug this kind of problem and how to search for traceids? We are not sure where to look and how to fix this particular issue.

Hello Peter, every 5xx HTTP status code is accompanied with related backend error, possibly with some exception stack frame (stored in

exc

field of JSON-formatted log). A single user request may generate several log records from multiple backend services, but all of them are sharing common traceId value. so it is possible to use your existing logging facility to search all relevant records with the same traceId. As far as the metadata-api restarts are concerned, the most common cause is the pod gets restarted by kubelet because it exceeded its resource limits. But the real reason might be visible in

kubectl events

or

kubectl describe your-failing-pod

We figured out the problem by the way. The redis connection was not working. we updated it to the proper IP and it started working again.

glad to hear that you've found the problem. Thanks for info.