Hi Team, All of a sudden, we see the below errors in afm-exec-api log for the average load on GoodData-cn, it is causing lots of slowness in GoodData-cn. Any idea what is this error and any quick fix for it? {"exc":"kotlinx.coroutines.JobCancellationException: MonoCoroutine was cancelled; job=MonoCoroutine{Cancelling}@690f491 ", "level":"ERROR", "logger":"com.gooddata.tiger.grpc.client.resultcache.ResultCacheClient", "msg":"getResultCacheResponse", "spanId":"ef4d670de5af5ff0", "thread":"reactor-http-epoll-1", "traceId":"ef4d670de5af5ff0", "ts":"2022-10-17 235519.975"} { insertId: "3rgglj1fewcaj2m0" jsonPayload: { exc: "kotlinx.coroutines.JobCancellationException: MonoCoroutine was cancelled; job=MonoCoroutine{Cancelling}@690f491 " level: "ERROR" logger: "com.gooddata.tiger.grpc.client.resultcache.ResultCacheClient" msg: "getResultCacheResponse" spanId: "ef4d670de5af5ff0" thread: "reactor-http-epoll-1" traceId: "ef4d670de5af5ff0" ts: "2022-10-17 235519.975" } labels: {5} logName: "xxxx/logs/stdout" receiveTimestamp: "2022-10-17T235524.874567330Z" resource: {2} severity: "ERROR" timestamp: "2022-10-17T235519.976262607Z" }

@Martin Blazik please, help

@Kshirod Mohanty would you give me a clue where do you run it? I need more detail log to investigate the issue

We have installed GoodData-CN in GCP K8s environment. Our client is a GCP CF function which makes a series of call to get the insights from GoodData-cn. Please let me know which log you are looking for. I can provide.

the error message above is just a grpc timeout while retrieving data from result-cache component that retrieves data from redis and also performs final data transformation (cross-tabulation). Is it possible that this error is somehow related to redis error you had 3 days ago? BTW, did you checked or set the redis maxmemory-policy to

allkeys-lru

as I suggested? Please check logs from result-cache pods for exceptions. it's also possible to trace down the request path using the same

traceId

that is present in the most of logs.

Here you go. @Martin Blazik @Robert Moucha

@Kshirod Mohanty Do you have the Redis maxmemory-policy set to

allkeys-lru

as I suggested? It will configure Redis instance to delete least recently used keys so you will not need to delete the keys manually. According to the log, it looks there is inconsistent status among several keys in Redis that coordinate report computation and result delivery. In order to recover from this error perform the following steps: 1. flush all keys in redis. 2. exec into pulsar/pulsar-pulsar-0 pod and run:

apache-pulsar/bin/pulsar-admin topics delete "<persistent://gooddata-cn/gooddata-cn/sql.select>"
apache-pulsar/bin/pulsar-admin topics delete "<persistent://gooddata-cn/gooddata-cn/compute.calcique>"
apache-pulsar/bin/pulsar-admin topics delete "<persistent://gooddata-cn/gooddata-cn/result.xtab>"

Pulsar topics will be recreated immediately. Unprocessed messages will be lost and it will cause all running computations to fail. But when you reload your dashboard, data will be recalculated and redis cache populated.

@Robert Moucha thanks for the quick response. I could not find any backlogs for these topics. But I could see good different between msgInCounter and msgOutCounter for both result.xtab and compute.calcique topics. What is the reason behind delete/recreate of these topics? Just curious can we skip-all old messages instead?

HTTP 412 Precondition Failed

Right, when apps are connected to the topics, force flag needs to be set (apache-pulsar/bin/pulsar-admin topics delete -f <<some-topic>>) msgInCounter and msgOutCounter may differ because a single incoming message may be redelivered when error ocurred during the first processing attempts. But if backlogSize is zero, it's not necessary to recreate topic. It's possible to clear backlog of topic subscription (without actual topic deletion) using pulsar-admin topics clear-backlog, but you need to pass the subscription name as well (these can be seen in "topics stats" output. As far as backlog is empty, it's not necessary to truncate the backlog by any means. Please respond to my question: did you checked or set the redis maxmemory-policy to

allkeys-lru

as I suggested?

@Robert Moucha I have changed maxmemory-policy to allkeys-lru. I did a very small load test, I don't see any error in afm service log. Here is our configuration of redis: Single node Capacity 150 GB, Max memory 150 GB. Eviction policy takes effect when maxmemory-gb reached the limit that is 150GB. I am sure this is not optimal. We bumped up memory from 5GB to 50GB ,then to 150GB for the recent gooddata-cn slowness we have during load testing (Test scenarios 500). From the metrics below, for test scenarios of size 20, I see the number of keys is 1.5K. I ran the test 5PM San Francisco time. I can still see(6 hours after the test run) the keys are active and only few with smaller TTL were evicted. Our client creates 75 number of keys for each test scenario.So running a load of 500, it will create 35K keys in a span of 30mins. With this trend(keys are not actually evicted) and from our recent experience we will again hit by the gooddata-cn slowness when the number of keys will be around 20K. Can you please recommend the capacity of Redis we should use for gooddata-cn ?

our application uses only basic Hash and Set operations in Redis, most of them (may be all of them) have

O(1)

algorithmic complexity, so the processing time is not affected by the amount of stored keys. So there's no reason why would the amount of keys affect performance. For your reference, here are charts from one regular perf test run in our environment. While the memory is "full" and number of keys reach to 700K keys, the redis response time usually remains below 20μs.