Hi! Sorry, one more question. I’m going through the service logs and I’ve noticed the following error every minute:

ERROR org.apache.pulsar.broker.service.persistent.PersistentDispatcherMultipleConsumers - [<persistent://public/default/result.xtab.DLQ> / Result-Cache listener-dead-letter] Error reading entries at 107:1 : Cursor was already closed, Read Type Normal

Any ideas what that is? Thanks!

Hi Tomáš, this appears to be an error related to the

Result-Cache

microservice, which is responsible for caching report results and such. It’s a bit generic, and thus not very helpful, but it is related to the service failing to read certain entries, and probably retrying it repeatedly, since you see them every minute. The error by itself doesn’t tell us very much, though. Are you experiencing any other issues? Are there other related errors showing up?

Thanks for the response, Francisco! Well, from time to time some queries seem to fail and return 500 on the client. And also sometimes we see filters on the filter bar fail to load and display an error (due to a failed collectLabelElements). But mostly I’m trying to find out if there’s anything we can do about performance of queries (it’s super slow after cache is flushed) and so I’m looking at the logs. There’s another one that occurs often:

ts="2023-06-29 13:26:50.673" level=ERROR msg="Failed to call onRow" logger=com.gooddata.tiger.grpc.client.rawcache.RawCacheStreamHandlerForStore thread=DefaultDispatcher-worker-13 action=grpcClientCall orgId=default spanId=775148994045db86 traceId=db6427964fcf64b2 userId=dbc252a2-d812-41f1-b864-cbe2e09b3a91 exc="errorType=com.gooddata.tiger.grpc.client.rawcache.RawCacheStreamHandlerForStore.OverLimitException, message=Reached limit of max size of data returned from database. Over limit 104857887

These last two errors are caused by queries returning too much data, exceeding the Execution limits. They could be related to the Cache issues as well, and probably are related to the performance slowdowns you noticed. As for improving the performance, have you taken a look at the recommendations in Improve Database Performance?

Thanks for the link on the limits! Is there a way to tell which one of these 3 limits we’re actually hitting?

I would start by checking the traceId of the errors and seeing if you can locate which endpoint is being called before the error occurs.

Unfortunately, when I search for the traceId, I can’t find a reference to the original request and since the query takes a long time, it’s hard to pinpoint it to a particular request in the sea of logs, but I’ll manage somehow hopefully 🙂 Just to confirm, you can adjust these limits only when deploying on K8s?

Hi Tomáš, yes, your assumption is correct. 🙂