Hey all, is anyone seeing performance issues with ...
# gooddata-cloud
s
Hey all, is anyone seeing performance issues with Goodata Cloud today? We're seeing slowness in our UI.
@Aaron Gunn fyi
k
Yes, there have been several recent reports of performance issues and outages with GoodData Cloud, including slowness in the UI and problems loading dashboards. These issues have sometimes been related to ongoing releases, temporary backend problems, or infrastructure incidents. For example, users have reported: - Dashboards and workspaces not loading or loading very slowly, especially during or after platform releases. In some cases, the slowness was tied to missing indexes in the metadata database, which engineers resolved, restoring normal execution times. If you are still experiencing slowness, it may be related to these recent events, and you may want to check if the issue persists now that fixes have been applied [Unexpected GoodData Cloud Issue on IAD1 Datacenter](https://support.gooddata.com/hc/en-us/articles/21439244799763-Thu-October-5th-2023-GoodData-Cloud-Release) [Slack thread](https://gooddataconnect.slack.com/archives/C04S1MSLEAW/p1696496758533829?thread_ts=1696496758.533829&cid=C04S1MSLEAW). - There have also been reports of temporary outages or degraded performance in specific regions (e.g., EU or US datacenters), which were acknowledged and resolved by GoodData engineers. If you are affected, it is recommended to monitor the official GoodData status or announcement pages for updates [Unexpected GoodData Cloud Issue on DUB1 Datacenter](https://support.gooddata.com/hc/en-us/articles/38288731795987-Thu-Feb-6th-2025-Unexpected-GoodData-Cloud-Issue-on-DUB1-Datacenter-eu-west-1) [Slack thread](https://gooddataconnect.slack.com/archives/C04S1MSLEAW/p1738832323116349?thread_ts=1738832323.116349&cid=C04S1MSLEAW). If your slowness is ongoing, you may want to provide a trace ID or example dashboard URL to GoodData support, as this can help them investigate further. Also, sometimes performance issues can be user- or environment-specific, so checking with other users in your organization or trying a different browser/network can help isolate the problem. If you need real-time updates, check the GoodData support announcements or status page, as incidents are often posted there as they occur.
i
Hello Steve, Our AI is still in development and I would like to step in and inform that there has not been any known performance issues lately. Nevertheless, we are looking into it and will update you here. May I ask if you still observe some performance issues?
Hello Steve, I have just used impersonation in your domain and could not observe and performance issues. Could you please describe the performance issues you had? Thank you
s
I haven't checked but w were getting reports of 60s + loading of our dashboards.
i
Hello Steve, Could you kindly check if you observe any performance issues right now? If yes, could you please explain so we can investigate it further? Thank you for your cooperation.
s
We are generally seeing average response times from the calls to gooddata cloud take > 800-1000 ms per request. I have some har files we can share. When there are 5-10 of these requests all together we get lots of just loading/waiting for information. We don't see any differences/performance with our Datasources at the same time respectively so they seem to be related to the Gooddata cloud services. We cannot see these resources or have any way to deterministically point to these things except that everything else is the same from our perspective except the part we do not have access or information about ( ie - > Gooddata Cloud resources)
Adding to this • in the past we have seen issues where the data center gets overloaded • or other customers impact our performance because we are all using shared resources • etc.
i
Hello Steve, I am sorry that you are experiencing performance issues. Could you please share the HAR files (If you would like to you can send me those in DM) and in case it is not enough, could you please also share a link where you see some of the performance issues so I can test it? Thank you once again for your cooperation.
d
Draft message: Hi Steve, I've taken the longest request in your har file, which lasted a bit over five seconds, and analyzed its execution. It's in workspace
***_40417
, dashboard ID
***-d73aa79ccc48
. I also ran the dashboard myself to get a clean picture. Altogether, the dashboard sends around 25 requests to SnowFlake. The requests take between 0.7 and 2.4 seconds each to process. (They run partially in parallel, the times don't add-up.) Considering all this, the overall time of five seconds to display the whole dashboard doesn't seem surprising. Could you please give us an example of a slow operation you've observed? It can be a .har file - but an accurate time when the slow response was observed will be enough. I can get the rest from the system logs, as long as I have an accurate start- or end-time (plus the time-zone). Can you send me the time of an example? (Alternatively, request ID is ideal for diagnosing - but whatever is easier for you.) Thanks.
s
Are we saying this is a snowflake issue and not a Gooddata issue? The attribute requests seem to be related to getting the PDM and/or Analytical model. We can take a look, but how do we know/understand what part of this process is Gooddata resources vs Snowflake resources in our instance? This isn't the first time we've tried to better understand this relationship. Main thing we're seeing now is periodically is that a lot of the calls to gooddata for dashboards seem to average > 1s. What is the expected response times from gooddata?
d
Hi Steve. We are NOT saying that. We are saying we need to investigate before we can make a conclusion. I did a rather detailed analysis of the 5-second dashboard that I found in your HAR files. What I'm asking from you now are some examples of slowness that I could investigate next. How do we know what part is GD/Snowflake resources? By looking into the logs of several incidents of slowness and analyzing what was being done and for how long. In addition to GD/Snowflake resources, we can also find out that we need to modify the LDM or metrics or change some filters. I really don't know now. There is no single figure as an expected response time. Fetching information from metadata should be a fraction of a second. Computation of a dashboard, on the other hand, can be anything - depending on the dashboard's complexity. If the dashboard contains computations that need to full-scan the data a few times it WILL take long. The only thing I can guarantee is that it won't be more than 150 seconds - that's currently the timeout for dashboard computation. Can you get an example of a long computation for me, please? Date + time + timezone are crucial; traceID, duration and some description are optional. Many thanks.
s
@Daniel Stourac I don't have a test case for you of this scenario, but I'm in gooddata trying to author right now and I'm constantly getting hung up and having no ability to use the system. I'll grab a har file from it.
In fact, its just frozen, I cannot do anything once I select a dashboard. Is there a gooddata issue right now?
👀 1
d
Hi Steve. No, there is no issue. I have just checked my own test environment and all seems normal.
My env is located in the same datacenter as yours and I'm using Snowflake as the source DB.
s
I get to a dashboard to edit it and it freezes, never resolves the requests.
I can't even save he 2nd har file cause the UI is frozen
maybe a cache issue, incognito seems to work a bit better....
d
I'll look at the empty dashboard. Can you figure out when exactly it was recorded?
s
just now
or 5 minutes ago
d
Could you please also try a different browser to verify if the issue is in the browser or in the backend?
s
I tried incognito and I'm through where I was stuck. So likely a cache issue with Chrome.
I'll try safari if you think thats helpful
d
Please try. Just for comparison. It will help us pinpoint the cause of the issue.
If you can also get the precise time, it would help A LOT. There are thousands of events in your environment between "just now" and "5 minutes ago". Exact time helps tremendously in my investigation. Thanks.
s
yeah safari works pretty well, just my normal instance of chrome. I was using 2-3 tabs looking at the same models
I finished recording a 3:59pm CT today
👍 1
so the 4-5 minutes before that
👍 1
Workspace name is: lmn_single_authoring
if that helps
d
The number of tabs won't be the issue. Something in the cache is more likely.
The timing helps a LOT. So does the WS name. Thanks.
Please, bear with me until tomorrow to investigate it. Also, we should switch to DM to avoid leaking some sensitive info. (Please check that the video doesn't contain some ...)
s
it shouldn't but fair point
👍 1