Hi, I am seeing what looks to be an OOM exception on loading an insight in the AD on GD CN 2.2. Any help or guidance is appreciated

"msg":"Failed to get SQL query for label elements request","exc":"errorType=com.gooddata.tiger.grpc.error.GrpcPropagatedServerException, message=org.springframework.data.redis.RedisSystemException: Error in execution; nested exception is io.lettuce.core.RedisCommandExecutionException: OOM command not allowed when used memory > 'maxmemory'.,<no detail>\n\tat com.gooddata.tiger.grpc.error.ErrorPropagationKt.convertFromKnownException(ErrorPropagation.kt:234)\n\tat com.gooddata.tiger.grpc.error.ErrorPropagationKt.convertToTransferableException(ErrorPropagation.kt:204)\n\tat com.gooddata.tiger.grpc.error.ErrorPropagationKt.clientCatching(ErrorPropagation.kt:104)\n\tat com.gooddata.tiger.grpc.error.ErrorPropagationKt$clientCatching$1.invokeSuspend(ErrorPropagation.kt)\n\tat kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)\n\tat kotlinx.coroutines.DispatchedTaskKt.resume(DispatchedTask.kt:175)\n\tat kotlinx.coroutines.DispatchedTaskKt.resumeUnconfined(DispatchedTask.kt:137)\n\tat kotlinx.coroutines.DispatchedTaskKt.dispatch(DispatchedTask.kt:108)\n\tat kotlinx.coroutines.CancellableContinuationImpl.dispatchResume(CancellableContinuationImpl.kt:308)\n\tat kotlinx.coroutines.CancellableContinuationImpl.resumeImpl(CancellableContinuationImpl.kt:318)\n\tat kotlinx.coroutines.CancellableContinuationImpl.resumeWith(CancellableContinuationImpl.kt:250)\n\tat com.github.marcoferrer.krotoplus.coroutines.client.SuspendingUnaryObserver.onError(SuspendingUnaryObserver.kt:34)\n\tat io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:487)\n\tat brave.grpc.TracingClientInterceptor$TracingClientCallListener.onClose(TracingClientInterceptor.java:202)\n\tat io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)\n\tat io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)\n\tat io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)\n\tat net.devh.boot.grpc.client.metric.MetricCollectingClientCallListener.onClose(MetricCollectingClientCallListener.java:59)\n\tat io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:562)\n\tat io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:70)\n\tat io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:743)\n\tat io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:722)\n\tat io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)\n\tat io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)\n\tat java.base/java.lang.Thread.run(Unknown Source)\n\tSuppressed: The stacktrace has been enhanced by Reactor, refer to additional information below: \nError has been observed at the following site(s):\n\t*__checkpoint ⇢ Handler com.gooddata.tiger.afm.controller.ElementsController#processElementsRequestPost(String, int, int, boolean, ElementsRequest, ServerHttpRequest, Continuation) [DispatcherHandler]\nOriginal Stack Trace:\n\t\t... 27 more\n"}

Hi, what deployment type are you talking about? Did you installed helm chart to your cluster and configured gooddata-cn to access your custom redis instance?

Yes

If yes, can you please check the setting of

maxmemory-policy

? it shoudl be set to

allkeys-lru

Will do and keep you posted ... I need to work with my dev-ops team on this.

It's redis setting

So what you're saying is the redis memory needs to be bumped up or the lru setting needs to be setup to allkeys

increasing redis mem will not resolve the issue

Any idea why this might suddenly have popped up? The AD was working fine yesterday

gooddata-cn relies on automated cache eviction based on LRU algorithm. If you don't have this setting turned on, the memory just fills up and apps stop working

@Robert Moucha Thank you changing the redis parameter solved the problem