To GoodData homepage
Question

GoodData.CN Internal Server Error

N
  • NickK
  • Known Participant
  • 22 replies

Hi!

We’ve encountered some strange error and we cannot find any way to identify the problem.

We’ve using GoodData.CN development edition in docker container.

The issue comes up after a minute or so after opening “Analyze” tab:

We can’t do anything after that.

Here are the logs from the server:

ts="2021-06-30 08:17:07.585" level=ERROR msg="Failed to start computation" logger=com.gooddata.tiger.grpc.client.aqe.AqeClient thread=grpc-default-executor-6 spanId=13ea4090fa414b5e traceId=13ea4090fa414b5e exc="errorType=com.gooddata.tiger.grpc.error.GrpcPropagatedServerException, message=UNKNOWN: ,<no detail>
	at com.gooddata.tiger.grpc.error.ErrorPropagationKt.convertFromUnknownException(ErrorPropagation.kt:108)
	Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException: 
Error has been observed at the following site(s):
	|_ checkpoint ⇢ Handler com.gooddata.tiger.afm.controller.ElementsController#processElementsRequest(String, String, ElementsOrder, boolean, boolean, String, int, int, boolean, ServerHttpRequest, Continuation) [DispatcherHandler]
Stack trace:
		at com.gooddata.tiger.grpc.error.ErrorPropagationKt.convertFromUnknownException(ErrorPropagation.kt:108)
		at com.gooddata.tiger.grpc.error.ErrorPropagationKt.convertToTransferableException(ErrorPropagation.kt:98)
		at com.gooddata.tiger.grpc.error.ErrorPropagationKt.clientCatching(ErrorPropagation.kt:72)
		at com.gooddata.tiger.grpc.error.ErrorPropagationKt$clientCatching$1.invokeSuspend(ErrorPropagation.kt)
		at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
		at kotlinx.coroutines.DispatchedTaskKt.resume(DispatchedTask.kt:175)
		at kotlinx.coroutines.DispatchedTaskKt.resumeUnconfined(DispatchedTask.kt:137)
		at kotlinx.coroutines.DispatchedTaskKt.dispatch(DispatchedTask.kt:108)
		at kotlinx.coroutines.CancellableContinuationImpl.dispatchResume(CancellableContinuationImpl.kt:308)
		at kotlinx.coroutines.CancellableContinuationImpl.resumeImpl(CancellableContinuationImpl.kt:318)
		at kotlinx.coroutines.CancellableContinuationImpl.resumeWith(CancellableContinuationImpl.kt:250)
		at com.github.marcoferrer.krotoplus.coroutines.client.SuspendingUnaryObserver.onError(SuspendingUnaryObserver.kt:34)
		at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:478)
		at brave.grpc.TracingClientInterceptor$TracingClientCallListener.onClose(TracingClientInterceptor.java:202)
		at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
		at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
		at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
		at net.devh.boot.grpc.client.metric.MetricCollectingClientCallListener.onClose(MetricCollectingClientCallListener.java:65)
		at io.grpc.internal.DelayedClientCall$DelayedListener$3.run(DelayedClientCall.java:464)
		at io.grpc.internal.DelayedClientCall$DelayedListener.delayOrExecute(DelayedClientCall.java:428)
		at io.grpc.internal.DelayedClientCall$DelayedListener.onClose(DelayedClientCall.java:461)
		at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:617)
		at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:70)
		at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:803)
		at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:782)
		at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
		at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
		at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
		at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
		at java.base/java.lang.Thread.run(Unknown Source)
"
ts="2021-06-30 08:17:07.637" level=ERROR msg="Internal Server Error" logger=org.zalando.problem.spring.common.AdviceTraits thread=grpc-default-executor-6 orgId=default spanId=13ea4090fa414b5e traceId=13ea4090fa414b5e userId=demo exc="errorType=com.gooddata.tiger.grpc.error.GrpcPropagatedServerException, message=UNKNOWN: ,<no detail>
	at com.gooddata.tiger.grpc.error.ErrorPropagationKt.convertFromUnknownException(ErrorPropagation.kt:108)
	Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException: 
Error has been observed at the following site(s):
	|_ checkpoint ⇢ Handler com.gooddata.tiger.afm.controller.ElementsController#processElementsRequest(String, String, ElementsOrder, boolean, boolean, String, int, int, boolean, ServerHttpRequest, Continuation) [DispatcherHandler]
Stack trace:
		at com.gooddata.tiger.grpc.error.ErrorPropagationKt.convertFromUnknownException(ErrorPropagation.kt:108)
		at com.gooddata.tiger.grpc.error.ErrorPropagationKt.convertToTransferableException(ErrorPropagation.kt:98)
		at com.gooddata.tiger.grpc.error.ErrorPropagationKt.clientCatching(ErrorPropagation.kt:72)
		at com.gooddata.tiger.grpc.error.ErrorPropagationKt$clientCatching$1.invokeSuspend(ErrorPropagation.kt)
		at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
		at kotlinx.coroutines.DispatchedTaskKt.resume(DispatchedTask.kt:175)
		at kotlinx.coroutines.DispatchedTaskKt.resumeUnconfined(DispatchedTask.kt:137)
		at kotlinx.coroutines.DispatchedTaskKt.dispatch(DispatchedTask.kt:108)
		at kotlinx.coroutines.CancellableContinuationImpl.dispatchResume(CancellableContinuationImpl.kt:308)
		at kotlinx.coroutines.CancellableContinuationImpl.resumeImpl(CancellableContinuationImpl.kt:318)
		at kotlinx.coroutines.CancellableContinuationImpl.resumeWith(CancellableContinuationImpl.kt:250)
		at com.github.marcoferrer.krotoplus.coroutines.client.SuspendingUnaryObserver.onError(SuspendingUnaryObserver.kt:34)
		at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:478)
		at brave.grpc.TracingClientInterceptor$TracingClientCallListener.onClose(TracingClientInterceptor.java:202)
		at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
		at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
		at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
		at net.devh.boot.grpc.client.metric.MetricCollectingClientCallListener.onClose(MetricCollectingClientCallListener.java:65)
		at io.grpc.internal.DelayedClientCall$DelayedListener$3.run(DelayedClientCall.java:464)
		at io.grpc.internal.DelayedClientCall$DelayedListener.delayOrExecute(DelayedClientCall.java:428)
		at io.grpc.internal.DelayedClientCall$DelayedListener.onClose(DelayedClientCall.java:461)
		at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:617)
		at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:70)
		at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:803)
		at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:782)
		at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
		at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
		at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
		at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
		at java.base/java.lang.Thread.run(Unknown Source)
"
172.24.0.1 - - [30/Jun/2021:08:17:07 +0000] "GET /api/actions/workspaces/d2df1e7a8d814d22956e0c27410f0cf4/execution/collectLabelElements?label=aca_dependent.coverage&offset=0&limit=5 HTTP/1.1" 500 124 "https://gooddata-dev.criterionhcm.com/analyze/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36"
ts="2021-06-30 08:17:07.875802+00:00" level=error traceId="13ea4090fa414b5e" spanId="e38823a8d79e4587" action="aqe_server" func="aqe_server:try_catch_log_error/2 (/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl:136)" reason="uncaught exception" details="{error,function_clause,[{aqe_grpc,check_grpc_result,[tiger_metadata_store_service_client,get_datasets,{error,timeout}],[{file,\\\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/util/aqe_grpc.erl\\\"},{line,88}]},{aqe_tracing,external_call_wrapper,2,[{file,\\\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/util/aqe_tracing.erl\\\"},{line,76}]},{aqe_workspace_models,build_models_from_md,2,[{file,\\\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_workspace_models.erl\\\"},{line,51}]},{aqe_server,ensure_model_cache,2,[{file,\\\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\\\"},{line,309}]},{aqe_server,get_model_id,3,[{file,\\\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\\\"},{line,291}]},{aqe_server,compute_raw_id,4,[{file,\\\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\\\"},{line,164}]},{aqe_server,'-handle_call/3-fun-0-',4,[{file,\\\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\\\"},{line,80}]},{aqe_server,try_catch_log_error,2,[{file,\\\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\\\"},{line,119}]}]}"
ts="2021-06-30 08:17:08.892692+00:00" level=error client_info="{<0.1859.0>,dead}" label="{gen_server,terminate}" last_message="{compute,#{context => #{organizationId => <<\"default\">>,workspaceId => <<\"d2df1e7a8d814d22956e0c27410f0cf4\">>},maql => #{asts => [],auxMetrics => [],dimension => [#{id => <<\"aca_dependent.coverage\">>,space => <<\"d2df1e7a8d814d22956e0c27410f0cf4\">>,type => 'LABEL'},#{id => <<\"aca_dependent.coverage\">>,space => <<\"d2df1e7a8d814d22956e0c27410f0cf4\">>,type => 'LABEL'}],filters => [],having => [],metrics => [],timestamp => <<\"2021-06-30T08:17:02.556854Z\">>},skipCache => false},#{<<\":authority\">> => <<\"localhost:6569\">>,<<\":method\">> => <<\"POST\">>,<<\":path\">> => <<\"/tiger.AqeService/compute\">>,<<\":scheme\">> => <<\"http\">>,<<\"grpc-accept-encoding\">> => <<\"gzip\">>,<<\"user-agent\">> => <<\"afm-exec-api grpc-java-netty/1.34.1\">>,<<\"x-b3-parentspanid\">> => <<\"13ea4090fa414b5e\">>,<<\"x-b3-sampled\">> => <<\"0\">>,<<\"x-b3-spanid\">> => <<\"0c907423d586544f\">>,<<\"x-b3-traceid\">> => <<\"13ea4090fa414b5e\">>,<<\"x-tiger-organization-id\">> => <<\"default\">>,<<\"x-tiger-user-id\">> => <<\"demo\">>,<<\"x-tiger-user-name\">> => <<\"Demo User\">>}}" log="[]" name="aqe_server" reason="{function_clause,[{aqe_grpc,check_grpc_result,[tiger_metadata_store_service_client,get_datasets,{error,timeout}],[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/util/aqe_grpc.erl\"},{line,88}]},{aqe_tracing,external_call_wrapper,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/util/aqe_tracing.erl\"},{line,76}]},{aqe_workspace_models,build_models_from_md,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_workspace_models.erl\"},{line,51}]},{aqe_server,ensure_model_cache,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,309}]},{aqe_server,get_model_id,3,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,291}]},{aqe_server,compute_raw_id,4,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,164}]},{aqe_server,'-handle_call/3-fun-0-',4,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,80}]},{aqe_server,try_catch_log_error,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,119}]}]}" state="{state,<0.731.0>,<0.737.0>,<0.739.0>,<0.740.0>}"
ts="2021-06-30 08:17:08.894333+00:00" level=error label="{proc_lib,crash}" report="[[{initial_call,{aqe_server,init,['Argument__1']}},{pid,<0.730.0>},{registered_name,aqe_server},{error_info,{error,function_clause,[{aqe_grpc,check_grpc_result,[tiger_metadata_store_service_client,get_datasets,{error,timeout}],[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/util/aqe_grpc.erl\"},{line,88}]},{aqe_tracing,external_call_wrapper,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/util/aqe_tracing.erl\"},{line,76}]},{aqe_workspace_models,build_models_from_md,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_workspace_models.erl\"},{line,51}]},{aqe_server,ensure_model_cache,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,309}]},{aqe_server,get_model_id,3,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,291}]},{aqe_server,compute_raw_id,4,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,164}]},{aqe_server,'-handle_call/3-fun-0-',4,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,80}]},{aqe_server,try_catch_log_error,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,119}]}]}},{ancestors,[aqe_sup,<0.710.0>]},{message_queue_len,0},{messages,[]},{links,[<0.728.0>]},{dictionary,[{otter_span_information,{span,1625041022572933,1435030422688975710,\"compute_request\",16395393651521242503,905351222356497487,[{\"lc\",\"grpc_listener\"}],[{1625041022572964,\"sr\"}],undefined}},{aqe_log,{log,{erlang_logger},\"aqe_server\",{1625,40911,95305}}}]},{trap_exit,true},{status,running},{heap_size,4185},{stack_size,28},{reductions,101608}],[]]"
ts="2021-06-30 08:17:08.896340+00:00" level=error label="{proc_lib,crash}" report="[[{initial_call,{gun,proc_lib_hack,['Argument__1','Argument__2','Argument__3','Argument__4','Argument__5']}},{pid,<0.731.0>},{registered_name,[]},{error_info,{exit,{{owner_gone,{function_clause,[{aqe_grpc,check_grpc_result,[tiger_metadata_store_service_client,get_datasets,{error,timeout}],[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/util/aqe_grpc.erl\"},{line,88}]},{aqe_tracing,external_call_wrapper,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/util/aqe_tracing.erl\"},{line,76}]},{aqe_workspace_models,build_models_from_md,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_workspace_models.erl\"},{line,51}]},{aqe_server,ensure_model_cache,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,309}]},{aqe_server,get_model_id,3,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,291}]},{aqe_server,compute_raw_id,4,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,164}]},{aqe_server,'-handle_call/3-fun-0-',4,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,80}]},{aqe_server,try_catch_log_error,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,119}]}]}},[]},[{gun,proc_lib_hack,5,[{file,\"/builds/TuxBtSUr/2/gooddata/gdc-nas/microservices/aqe/aqe-erlang/_build/default/lib/gun/src/gun.erl\"},{line,654}]},{proc_lib,init_p_do_apply,3,[{file,\"proc_lib.erl\"},{line,226}]}]}},{ancestors,[gun_sup,<0.673.0>]},{message_queue_len,0},{messages,[]},{links,[<0.674.0>]},{dictionary,[]},{trap_exit,false},{status,running},{heap_size,2586},{stack_size,28},{reductions,2770}],[]]"
ts="2021-06-30 08:17:08.913046+00:00" level=error label="{supervisor,child_terminated}" report="[{supervisor,{local,gun_sup}},{errorContext,child_terminated},{reason,{{owner_gone,{function_clause,[{aqe_grpc,check_grpc_result,[tiger_metadata_store_service_client,get_datasets,{error,timeout}],[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/util/aqe_grpc.erl\"},{line,88}]},{aqe_tracing,external_call_wrapper,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/util/aqe_tracing.erl\"},{line,76}]},{aqe_workspace_models,build_models_from_md,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_workspace_models.erl\"},{line,51}]},{aqe_server,ensure_model_cache,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,309}]},{aqe_server,get_model_id,3,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,291}]},{aqe_server,compute_raw_id,4,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,164}]},{aqe_server,'-handle_call/3-fun-0-',4,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,80}]},{aqe_server,try_catch_log_error,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,119}]}]}},[]}},{offender,[{pid,<0.731.0>},{id,gun},{mfargs,{gun,start_link,undefined}},{restart_type,temporary},{shutdown,5000},{child_type,worker}]}]"
ts="2021-06-30 08:17:08.912917+00:00" level=error label="{proc_lib,crash}" report="[[{initial_call,{gun,proc_lib_hack,['Argument__1','Argument__2','Argument__3','Argument__4','Argument__5']}},{pid,<0.740.0>},{registered_name,[]},{error_info,{exit,{{owner_gone,{function_clause,[{aqe_grpc,check_grpc_result,[tiger_metadata_store_service_client,get_datasets,{error,timeout}],[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/util/aqe_grpc.erl\"},{line,88}]},{aqe_tracing,external_call_wrapper,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/util/aqe_tracing.erl\"},{line,76}]},{aqe_workspace_models,build_models_from_md,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_workspace_models.erl\"},{line,51}]},{aqe_server,ensure_model_cache,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,309}]},{aqe_server,get_model_id,3,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,291}]},{aqe_server,compute_raw_id,4,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,164}]},{aqe_server,'-handle_call/3-fun-0-',4,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,80}]},{aqe_server,try_catch_log_error,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,119}]}]}},[]},[{gun,proc_lib_hack,5,[{file,\"/builds/TuxBtSUr/2/gooddata/gdc-nas/microservices/aqe/aqe-erlang/_build/default/lib/gun/src/gun.erl\"},{line,654}]},{proc_lib,init_p_do_apply,3,[{file,\"proc_lib.erl\"},{line,226}]}]}},{ancestors,[gun_sup,<0.673.0>]},{message_queue_len,0},{messages,[]},{links,[<0.674.0>]},{dictionary,[]},{trap_exit,false},{status,running},{heap_size,2586},{stack_size,28},{reductions,2765}],[]]"
ts="2021-06-30 08:17:08.922346+00:00" level=error label="{supervisor,child_terminated}" report="[{supervisor,{local,gun_sup}},{errorContext,child_terminated},{reason,{{owner_gone,{function_clause,[{aqe_grpc,check_grpc_result,[tiger_metadata_store_service_client,get_datasets,{error,timeout}],[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/util/aqe_grpc.erl\"},{line,88}]},{aqe_tracing,external_call_wrapper,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/util/aqe_tracing.erl\"},{line,76}]},{aqe_workspace_models,build_models_from_md,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_workspace_models.erl\"},{line,51}]},{aqe_server,ensure_model_cache,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,309}]},{aqe_server,get_model_id,3,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,291}]},{aqe_server,compute_raw_id,4,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,164}]},{aqe_server,'-handle_call/3-fun-0-',4,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,80}]},{aqe_server,try_catch_log_error,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,119}]}]}},[]}},{offender,[{pid,<0.740.0>},{id,gun},{mfargs,{gun,start_link,undefined}},{restart_type,temporary},{shutdown,5000},{child_type,worker}]}]"
ts="2021-06-30 08:17:08.896960+00:00" level=error label="{proc_lib,crash}" report="[[{initial_call,{gun,proc_lib_hack,['Argument__1','Argument__2','Argument__3','Argument__4','Argument__5']}},{pid,<0.737.0>},{registered_name,[]},{error_info,{exit,{{owner_gone,{function_clause,[{aqe_grpc,check_grpc_result,[tiger_metadata_store_service_client,get_datasets,{error,timeout}],[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/util/aqe_grpc.erl\"},{line,88}]},{aqe_tracing,external_call_wrapper,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/util/aqe_tracing.erl\"},{line,76}]},{aqe_workspace_models,build_models_from_md,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_workspace_models.erl\"},{line,51}]},{aqe_server,ensure_model_cache,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,309}]},{aqe_server,get_model_id,3,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,291}]},{aqe_server,compute_raw_id,4,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,164}]},{aqe_server,'-handle_call/3-fun-0-',4,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,80}]},{aqe_server,try_catch_log_error,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,119}]}]}},[]},[{gun,proc_lib_hack,5,[{file,\"/builds/TuxBtSUr/2/gooddata/gdc-nas/microservices/aqe/aqe-erlang/_build/default/lib/gun/src/gun.erl\"},{line,654}]},{proc_lib,init_p_do_apply,3,[{file,\"proc_lib.erl\"},{line,226}]}]}},{ancestors,[gun_sup,<0.673.0>]},{message_queue_len,0},{messages,[]},{links,[<0.674.0>]},{dictionary,[]},{trap_exit,false},{status,running},{heap_size,2586},{stack_size,28},{reductions,2801}],[]]"
ts="2021-06-30 08:17:08.924399+00:00" level=error label="{supervisor,child_terminated}" report="[{supervisor,{local,gun_sup}},{errorContext,child_terminated},{reason,{{owner_gone,{function_clause,[{aqe_grpc,check_grpc_result,[tiger_metadata_store_service_client,get_datasets,{error,timeout}],[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/util/aqe_grpc.erl\"},{line,88}]},{aqe_tracing,external_call_wrapper,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/util/aqe_tracing.erl\"},{line,76}]},{aqe_workspace_models,build_models_from_md,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_workspace_models.erl\"},{line,51}]},{aqe_server,ensure_model_cache,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,309}]},{aqe_server,get_model_id,3,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,291}]},{aqe_server,compute_raw_id,4,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,164}]},{aqe_server,'-handle_call/3-fun-0-',4,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,80}]},{aqe_server,try_catch_log_error,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,119}]}]}},[]}},{offender,[{pid,<0.737.0>},{id,gun},{mfargs,{gun,start_link,undefined}},{restart_type,temporary},{shutdown,5000},{child_type,worker}]}]"
ts="2021-06-30 08:17:08.895753+00:00" level=error label="{supervisor,child_terminated}" report="[{supervisor,{local,aqe_sup}},{errorContext,child_terminated},{reason,{function_clause,[{aqe_grpc,check_grpc_result,[tiger_metadata_store_service_client,get_datasets,{error,timeout}],[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/util/aqe_grpc.erl\"},{line,88}]},{aqe_tracing,external_call_wrapper,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/util/aqe_tracing.erl\"},{line,76}]},{aqe_workspace_models,build_models_from_md,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_workspace_models.erl\"},{line,51}]},{aqe_server,ensure_model_cache,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,309}]},{aqe_server,get_model_id,3,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,291}]},{aqe_server,compute_raw_id,4,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,164}]},{aqe_server,'-handle_call/3-fun-0-',4,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,80}]},{aqe_server,try_catch_log_error,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,119}]}]}},{offender,[{pid,<0.730.0>},{id,aqe_server},{mfargs,{aqe_server,start_link,[]}},{restart_type,permanent},{shutdown,5000},{child_type,worker}]}]"
ts="2021-06-30 08:17:08.912730+00:00" level=error label="{proc_lib,crash}" report="[[{initial_call,{gun,proc_lib_hack,['Argument__1','Argument__2','Argument__3','Argument__4','Argument__5']}},{pid,<0.739.0>},{registered_name,[]},{error_info,{exit,{{owner_gone,{function_clause,[{aqe_grpc,check_grpc_result,[tiger_metadata_store_service_client,get_datasets,{error,timeout}],[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/util/aqe_grpc.erl\"},{line,88}]},{aqe_tracing,external_call_wrapper,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/util/aqe_tracing.erl\"},{line,76}]},{aqe_workspace_models,build_models_from_md,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_workspace_models.erl\"},{line,51}]},{aqe_server,ensure_model_cache,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,309}]},{aqe_server,get_model_id,3,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,291}]},{aqe_server,compute_raw_id,4,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,164}]},{aqe_server,'-handle_call/3-fun-0-',4,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,80}]},{aqe_server,try_catch_log_error,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,119}]}]}},[]},[{gun,proc_lib_hack,5,[{file,\"/builds/TuxBtSUr/2/gooddata/gdc-nas/microservices/aqe/aqe-erlang/_build/default/lib/gun/src/gun.erl\"},{line,654}]},{proc_lib,init_p_do_apply,3,[{file,\"proc_lib.erl\"},{line,226}]}]}},{ancestors,[gun_sup,<0.673.0>]},{message_queue_len,0},{messages,[]},{links,[<0.674.0>]},{dictionary,[]},{trap_exit,false},{status,running},{heap_size,2586},{stack_size,28},{reductions,2768}],[]]"
ts="2021-06-30 08:17:08.936901+00:00" level=error label="{supervisor,child_terminated}" report="[{supervisor,{local,gun_sup}},{errorContext,child_terminated},{reason,{{owner_gone,{function_clause,[{aqe_grpc,check_grpc_result,[tiger_metadata_store_service_client,get_datasets,{error,timeout}],[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/util/aqe_grpc.erl\"},{line,88}]},{aqe_tracing,external_call_wrapper,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/util/aqe_tracing.erl\"},{line,76}]},{aqe_workspace_models,build_models_from_md,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_workspace_models.erl\"},{line,51}]},{aqe_server,ensure_model_cache,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,309}]},{aqe_server,get_model_id,3,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,291}]},{aqe_server,compute_raw_id,4,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,164}]},{aqe_server,'-handle_call/3-fun-0-',4,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,80}]},{aqe_server,try_catch_log_error,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,119}]}]}},[]}},{offender,[{pid,<0.739.0>},{id,gun},{mfargs,{gun,start_link,undefined}},{restart_type,temporary},{shutdown,5000},{child_type,worker}]}]"

Any help would be appreciated.

This topic has been closed for comments

24 replies

D
Rank

Hi Nick,
From the screenshot and the logs, we can see that there's a timeout while loading the label aca_dependent_coverage.

Can I know if the error happened as soon as the "Analyze" screen started loading or was it when you tried to drag and drop a measure/attribute from the list on the left?

And lastly, can you check aca_dependent_coverage on the Logical Data Model? Is it loading correctly? And may I know how many values does it have?

D
Rank

Hi Nick,

We have been checking further on this and looks like you have too many datasets loaded, so you hit a timeout when loading it all in “Analyze”.

Please review your model and remove unused and un-necessary datasets.

V

Hi Daniela

I removed unused datasets here https://gooddata-dev.criterionhcm.com/dashboards/#/project/daceb3e88abd4bd5a7af036e9876a837/dashboard/dbec2742-0423-498a-979d-5af79037c52c

Now I can see the mesures in Analyze tab

 But I still can’t see them on the dashboard

 

Mort
Rank

 But I still can’t see them on the dashboard

Hi Vasiliy,

Isn’t it possible that you don’t have any data for this month (that started today)? By default the insights and KPIs on Dashboard are filtered by date filter if you did not explicitly disconnected them in the dashboard widget configuration.

V

 But I still can’t see them on the dashboard

Hi Vasiliy,

Isn’t it possible that you don’t have any data for this month (that started today)? By default the insights and KPIs on Dashboard are filtered by date filter if you did not explicitly disconnected them in the dashboard widget configuration.

How can I explicitly disconnect default filtering by date in the dashboard widget configuration?

Mort
Rank

 But I still can’t see them on the dashboard

Hi Vasiliy,

Isn’t it possible that you don’t have any data for this month (that started today)? By default the insights and KPIs on Dashboard are filtered by date filter if you did not explicitly disconnected them in the dashboard widget configuration.

How can I explicitly disconnect default filtering by date in the dashboard widget configuration?

Click on Edit button in top right corner to edit the dashboard, then click on KPI you want to setup. You should see the configuration panel of that KPI. Uncheck Date in Filter by section and then apply and save your changes. 

V

Hi

I’m still encoutering problems with insights.

The dashboard https://gooddata-dev.criterionhcm.com/dashboards/#/project/daceb3e88abd4bd5a7af036e9876a837/dashboard/dbec2742-0423-498a-979d-5af79037c52c shows the items partially

And if I try to open an insight for editing

I get this error

 

Iva Gasparekova
Rank
Userlevel 3

Unfortunately, this might be caused by the recent changes of your model and removal of some datasets. Would it be possible to rebuild one of your insights and test it again, please?

Iva
N

We also encounter sudden HTTP 500 from time to time.

While investigating, we’ve noticed that there are 2 postgres processes inside GD.CN docker container each eats 100% of 1 CPU core:


Here is postgres activity graph:
 


We’ve also noticed that postgres is configured with default minimum memory settings:
/etc/postgresql/11/main/conf.d/cloudnative-pg.conf:
 

listen_addresses = '*'
data_directory = '/data/postgresql/' # use data in another directory
shared_buffers = 128MB                  # min 128kB
#temp_buffers = 8MB                     # min 800kB
#work_mem = 4MB                         # min 64kB
#maintenance_work_mem = 64MB            # min 1MB

Would be nice to have instructions on how to tune those. We did it like this:
 

listen_addresses = '*'
data_directory = '/data/postgresql/' # use data in another directory
shared_buffers = 12GB                  # min 128kB
temp_buffers = 80MB                     # min 800kB
work_mem = 40MB                         # min 64kB
maintenance_work_mem = 640MB            # min 1MB

But unfortunately that did not help :-(

Attached last 1000 lines from docker logs. They are not helpful for us unfortunately...

1 Attachment
J
Rank

Hi @NickK,

Unfortunately we did not see any similar issues in our testing instances, so we are not sure what is the root cause of the problem (we are still investigating)

Anyway - if you can respond to following questions, it could help investigation:

  • Do you know what activity is performed on GoodData.CN during these problems? Only report calculation? Or creation of metrics, visualization objects or dashboard? Or manipulation with model (scanning data source, ...)?
  • These 2 stuck “postgres” process - are they stuck for long time (so you expect them to be stuck indefinitely), or only for some long amount of time (minutes, tenths of minutes)?
  • Can you connect to running PostgreSQL (credentials to internal PG are “postgres” / “topsecret”) and check “pg_stat_activity” table? Is there anything interesting? Is it changing, when you try after few minutes later?
N

Hi @Jan Chochol,

We’ve checked nginx logs and it seems like the problem was caused by a lot of HTTP POST requests like 

/api/actions/workspaces/…/execution/afm/execute

The response from GoodData was HTTP 500, and it seems like UI retransmitted those requests every 5 seconds. Seems like there were 2 such sessions, which is why there were 2 stuck postgres processes. They were always busy with processing such requests.

We were able to clear this issue by exporting all workspaces to JSON and then importing just one that we needed at the moment.

We also tuned PostgreSQL memory and upgraded EC2 instance.

J
Rank

Thanks @NickK.

We are aware of the fact, that performance is not ideal and we are trying to figure some solution.

Also we would try to prevent starvation by repeated same requests.

D
Rank

@NickK May I ask you to still check postgres database if this kind of issue ever happens again? Especially contents of “pg_stat_activity” table. Thanks

N

@NickKMay I ask you to still check postgres database if this kind of issue ever happens again? Especially contents of “pg_stat_activity” table. Thanks

Sure, will check that if it happens again.

V

Insight can’t get data from successfully published dataset. We are also encountering UI issues.

 

D
Rank

@Vasiliy Ostrovsky Can you please attach log events with relevant errors to your issue? Also what is the content of “pg_stat_activity” table when this issue happens?

Y

Hi @David Kubecka,

We are getting consistent timeout exception after ~5000ms for next URLs:
> https://gooddata-dev.criterionhcm.com/api/actions/workspaces/daceb3e88abd4bd5a7af036e9876a837/execution/afm/computeValidObjects
> https://gooddata-dev.criterionhcm.com/api/actions/workspaces/daceb3e88abd4bd5a7af036e9876a837/execution/afm/execute

Is there any way to increase the execution timeout?
Please correct me if I’m wrong and this exception has nothing to do with timeout.

ts="2021-07-23 13:22:28.575078+00:00" level=error client_info="{<0.17986.1>,dead}" label="{gen_server,terminate}" last_message="{valid_objects,#{context => #{organizationId => <<\"default\">>,workspaceId => <<\"daceb3e88abd4bd5a7af036e9876a837\">>},maql => #{asts => [#{ast => #{content => [#{content => [#{content => [#{content => [#{content => [],type => <<\"type\">>,value => <<\"fact\">>},#{content => [],type => <<\"id\">>,value => <<\"position.is_active\">>},#{content => [],type => <<\"space\">>,value => <<\"daceb3e88abd4bd5a7af036e9876a837\">>}],type => <<\"identifier\">>,value => <<>>}],type => <<\"function\">>,value => <<\"SUM\">>}],type => <<\"expression\">>,value => <<>>}],type => <<\"metric\">>,value => <<>>},identifier => #{id => <<\"default-f9403f441e614089a69ce9206361d934\">>,space => <<\"**local**\">>,type => 'METRIC'}}],auxMetrics => [],dimension => [],filters => [],having => [],metrics => [#{id => <<\"default-f9403f441e614089a69ce9206361d934\">>,space => <<\"**local**\">>,type => 'METRIC'}],timestamp => <<\"2021-07-23T13:22:22.56737Z\">>},types => [attributes]},#{<<\":authority\">> => <<\"localhost:6569\">>,<<\":method\">> => <<\"POST\">>,<<\":path\">> => <<\"/tiger.AqeService/validObjects\">>,<<\":scheme\">> => <<\"http\">>,<<\"grpc-accept-encoding\">> => <<\"gzip\">>,<<\"user-agent\">> => <<\"afm-exec-api grpc-java-netty/1.34.1\">>,<<\"x-b3-parentspanid\">> => <<\"ce830c2eac114b40\">>,<<\"x-b3-sampled\">> => <<\"0\">>,<<\"x-b3-spanid\">> => <<\"3b71123506aab72e\">>,<<\"x-b3-traceid\">> => <<\"ce830c2eac114b40\">>,<<\"x-tiger-organization-id\">> => <<\"default\">>,<<\"x-tiger-user-id\">> => <<\"demo\">>,<<\"x-tiger-user-name\">> => <<\"Demo User\">>}}" log="[]" name="aqe_server" reason="{function_clause,[{aqe_grpc,check_grpc_result,[tiger_metadata_store_service_client,get_datasets,{error,timeout}],[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/util/aqe_grpc.erl\"},{line,88}]},{aqe_tracing,external_call_wrapper,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/util/aqe_tracing.erl\"},{line,76}]},{aqe_workspace_models,build_models_from_md,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_workspace_models.erl\"},{line,51}]},{aqe_server,ensure_model_cache,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,309}]},{aqe_server,get_models_by_types,4,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,287}]},{aqe_server,process_valid_objects,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,277}]},{aqe_server,'-handle_call/3-fun-1-',2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,98}]},{aqe_server,try_catch_log_error,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,119}]}]}" state="{state,<0.8886.1>,<0.8887.1>,<0.8888.1>,<0.8889.1>}"
ts="2021-07-23 13:22:28.576238+00:00" level=error label="{proc_lib,crash}" report="[[{initial_call,{aqe_server,init,['Argument__1']}},{pid,<0.8885.1>},{registered_name,aqe_server},{error_info,{error,function_clause,[{aqe_grpc,check_grpc_result,[tiger_metadata_store_service_client,get_datasets,{error,timeout}],[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/util/aqe_grpc.erl\"},{line,88}]},{aqe_tracing,external_call_wrapper,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/util/aqe_tracing.erl\"},{line,76}]},{aqe_workspace_models,build_models_from_md,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_workspace_models.erl\"},{line,51}]},{aqe_server,ensure_model_cache,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,309}]},{aqe_server,get_models_by_types,4,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,287}]},{aqe_server,process_valid_objects,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,277}]},{aqe_server,'-handle_call/3-fun-1-',2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,98}]},{aqe_server,try_catch_log_error,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,119}]}]}},{ancestors,[aqe_sup,<0.710.0>]},{message_queue_len,0},{messages,[]},{links,[<0.728.0>]},{dictionary,[{otter_span_information,{span,1627046542572392,14880750988357094208,\"valid_objects\",4021662233256807475,4283224739560470318,[{\"lc\",\"grpc_listener\"}],[{1627046542572423,\"sr\"}],undefined}},{aqe_log,{log,{erlang_logger},\"aqe_server\",{1627,45633,777867}}}]},{trap_exit,true},{status,running},{heap_size,46422},{stack_size,28},{reductions,115659}],[]]"
ts="2021-07-23 13:22:28.577105+00:00" level=error label="{proc_lib,crash}" report="[[{initial_call,{gun,proc_lib_hack,['Argument__1','Argument__2','Argument__3','Argument__4','Argument__5']}},{pid,<0.8888.1>},{registered_name,[]},{error_info,{exit,{{owner_gone,{function_clause,[{aqe_grpc,check_grpc_result,[tiger_metadata_store_service_client,get_datasets,{error,timeout}],[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/util/aqe_grpc.erl\"},{line,88}]},{aqe_tracing,external_call_wrapper,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/util/aqe_tracing.erl\"},{line,76}]},{aqe_workspace_models,build_models_from_md,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_workspace_models.erl\"},{line,51}]},{aqe_server,ensure_model_cache,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,309}]},{aqe_server,get_models_by_types,4,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,287}]},{aqe_server,process_valid_objects,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,277}]},{aqe_server,'-handle_call/3-fun-1-',2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,98}]},{aqe_server,try_catch_log_error,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,119}]}]}},[]},[{gun,proc_lib_hack,5,[{file,\"/builds/TuxBtSUr/2/gooddata/gdc-nas/microservices/aqe/aqe-erlang/_build/default/lib/gun/src/gun.erl\"},{line,654}]},{proc_lib,init_p_do_apply,3,[{file,\"proc_lib.erl\"},{line,226}]}]}},{ancestors,[gun_sup,<0.673.0>]},{message_queue_len,0},{messages,[]},{links,[<0.674.0>]},{dictionary,[]},{trap_exit,false},{status,running},{heap_size,2586},{stack_size,28},{reductions,8707}],[]]"
ts="2021-07-23 13:22:28.576969+00:00" level=error label="{supervisor,child_terminated}" report="[{supervisor,{local,aqe_sup}},{errorContext,child_terminated},{reason,{function_clause,[{aqe_grpc,check_grpc_result,[tiger_metadata_store_service_client,get_datasets,{error,timeout}],[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/util/aqe_grpc.erl\"},{line,88}]},{aqe_tracing,external_call_wrapper,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/util/aqe_tracing.erl\"},{line,76}]},{aqe_workspace_models,build_models_from_md,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_workspace_models.erl\"},{line,51}]},{aqe_server,ensure_model_cache,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,309}]},{aqe_server,get_models_by_types,4,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,287}]},{aqe_server,process_valid_objects,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,277}]},{aqe_server,'-handle_call/3-fun-1-',2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,98}]},{aqe_server,try_catch_log_error,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,119}]}]}},{offender,[{pid,<0.8885.1>},{id,aqe_server},{mfargs,{aqe_server,start_link,[]}},{restart_type,permanent},{shutdown,5000},{child_type,worker}]}]"
ts="2021-07-23 13:22:28.577418+00:00" level=error label="{proc_lib,crash}" report="[[{initial_call,{gun,proc_lib_hack,['Argument__1','Argument__2','Argument__3','Argument__4','Argument__5']}},{pid,<0.8887.1>},{registered_name,[]},{error_info,{exit,{{owner_gone,{function_clause,[{aqe_grpc,check_grpc_result,[tiger_metadata_store_service_client,get_datasets,{error,timeout}],[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/util/aqe_grpc.erl\"},{line,88}]},{aqe_tracing,external_call_wrapper,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/util/aqe_tracing.erl\"},{line,76}]},{aqe_workspace_models,build_models_from_md,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_workspace_models.erl\"},{line,51}]},{aqe_server,ensure_model_cache,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,309}]},{aqe_server,get_models_by_types,4,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,287}]},{aqe_server,process_valid_objects,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,277}]},{aqe_server,'-handle_call/3-fun-1-',2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,98}]},{aqe_server,try_catch_log_error,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,119}]}]}},[]},[{gun,proc_lib_hack,5,[{file,\"/builds/TuxBtSUr/2/gooddata/gdc-nas/microservices/aqe/aqe-erlang/_build/default/lib/gun/src/gun.erl\"},{line,654}]},{proc_lib,init_p_do_apply,3,[{file,\"proc_lib.erl\"},{line,226}]}]}},{ancestors,[gun_sup,<0.673.0>]},{message_queue_len,0},{messages,[]},{links,[<0.674.0>]},{dictionary,[]},{trap_exit,false},{status,running},{heap_size,2586},{stack_size,28},{reductions,8703}],[]]"
ts="2021-07-23 13:22:28.577203+00:00" level=error label="{proc_lib,crash}" report="[[{initial_call,{gun,proc_lib_hack,['Argument__1','Argument__2','Argument__3','Argument__4','Argument__5']}},{pid,<0.8886.1>},{registered_name,[]},{error_info,{exit,{{owner_gone,{function_clause,[{aqe_grpc,check_grpc_result,[tiger_metadata_store_service_client,get_datasets,{error,timeout}],[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/util/aqe_grpc.erl\"},{line,88}]},{aqe_tracing,external_call_wrapper,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/util/aqe_tracing.erl\"},{line,76}]},{aqe_workspace_models,build_models_from_md,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_workspace_models.erl\"},{line,51}]},{aqe_server,ensure_model_cache,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,309}]},{aqe_server,get_models_by_types,4,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,287}]},{aqe_server,process_valid_objects,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,277}]},{aqe_server,'-handle_call/3-fun-1-',2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,98}]},{aqe_server,try_catch_log_error,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,119}]}]}},[]},[{gun,proc_lib_hack,5,[{file,\"/builds/TuxBtSUr/2/gooddata/gdc-nas/microservices/aqe/aqe-erlang/_build/default/lib/gun/src/gun.erl\"},{line,654}]},{proc_lib,init_p_do_apply,3,[{file,\"proc_lib.erl\"},{line,226}]}]}},{ancestors,[gun_sup,<0.673.0>]},{message_queue_len,0},{messages,[]},{links,[<0.674.0>]},{dictionary,[]},{trap_exit,false},{status,running},{heap_size,2586},{stack_size,28},{reductions,8703}],[]]"
ts="2021-07-23 13:22:28.578361+00:00" level=error label="{supervisor,child_terminated}" report="[{supervisor,{local,gun_sup}},{errorContext,child_terminated},{reason,{{owner_gone,{function_clause,[{aqe_grpc,check_grpc_result,[tiger_metadata_store_service_client,get_datasets,{error,timeout}],[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/util/aqe_grpc.erl\"},{line,88}]},{aqe_tracing,external_call_wrapper,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/util/aqe_tracing.erl\"},{line,76}]},{aqe_workspace_models,build_models_from_md,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_workspace_models.erl\"},{line,51}]},{aqe_server,ensure_model_cache,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,309}]},{aqe_server,get_models_by_types,4,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,287}]},{aqe_server,process_valid_objects,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,277}]},{aqe_server,'-handle_call/3-fun-1-',2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,98}]},{aqe_server,try_catch_log_error,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,119}]}]}},[]}},{offender,[{pid,<0.8888.1>},{id,gun},{mfargs,{gun,start_link,undefined}},{restart_type,temporary},{shutdown,5000},{child_type,worker}]}]"
ts="2021-07-23 13:22:28.577610+00:00" level=error label="{proc_lib,crash}" report="[[{initial_call,{gun,proc_lib_hack,['Argument__1','Argument__2','Argument__3','Argument__4','Argument__5']}},{pid,<0.8889.1>},{registered_name,[]},{error_info,{exit,{{owner_gone,{function_clause,[{aqe_grpc,check_grpc_result,[tiger_metadata_store_service_client,get_datasets,{error,timeout}],[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/util/aqe_grpc.erl\"},{line,88}]},{aqe_tracing,external_call_wrapper,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/util/aqe_tracing.erl\"},{line,76}]},{aqe_workspace_models,build_models_from_md,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_workspace_models.erl\"},{line,51}]},{aqe_server,ensure_model_cache,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,309}]},{aqe_server,get_models_by_types,4,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,287}]},{aqe_server,process_valid_objects,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,277}]},{aqe_server,'-handle_call/3-fun-1-',2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,98}]},{aqe_server,try_catch_log_error,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,119}]}]}},[]},[{gun,proc_lib_hack,5,[{file,\"/builds/TuxBtSUr/2/gooddata/gdc-nas/microservices/aqe/aqe-erlang/_build/default/lib/gun/src/gun.erl\"},{line,654}]},{proc_lib,init_p_do_apply,3,[{file,\"proc_lib.erl\"},{line,226}]}]}},{ancestors,[gun_sup,<0.673.0>]},{message_queue_len,0},{messages,[]},{links,[<0.674.0>]},{dictionary,[]},{trap_exit,false},{status,running},{heap_size,2586},{stack_size,28},{reductions,8692}],[]]"
ts="2021-07-23 13:22:28.579400+00:00" level=error label="{supervisor,child_terminated}" report="[{supervisor,{local,gun_sup}},{errorContext,child_terminated},{reason,{{owner_gone,{function_clause,[{aqe_grpc,check_grpc_result,[tiger_metadata_store_service_client,get_datasets,{error,timeout}],[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/util/aqe_grpc.erl\"},{line,88}]},{aqe_tracing,external_call_wrapper,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/util/aqe_tracing.erl\"},{line,76}]},{aqe_workspace_models,build_models_from_md,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_workspace_models.erl\"},{line,51}]},{aqe_server,ensure_model_cache,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,309}]},{aqe_server,get_models_by_types,4,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,287}]},{aqe_server,process_valid_objects,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,277}]},{aqe_server,'-handle_call/3-fun-1-',2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,98}]},{aqe_server,try_catch_log_error,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,119}]}]}},[]}},{offender,[{pid,<0.8887.1>},{id,gun},{mfargs,{gun,start_link,undefined}},{restart_type,temporary},{shutdown,5000},{child_type,worker}]}]"
ts="2021-07-23 13:22:28.580127+00:00" level=error label="{supervisor,child_terminated}" report="[{supervisor,{local,gun_sup}},{errorContext,child_terminated},{reason,{{owner_gone,{function_clause,[{aqe_grpc,check_grpc_result,[tiger_metadata_store_service_client,get_datasets,{error,timeout}],[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/util/aqe_grpc.erl\"},{line,88}]},{aqe_tracing,external_call_wrapper,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/util/aqe_tracing.erl\"},{line,76}]},{aqe_workspace_models,build_models_from_md,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_workspace_models.erl\"},{line,51}]},{aqe_server,ensure_model_cache,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,309}]},{aqe_server,get_models_by_types,4,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,287}]},{aqe_server,process_valid_objects,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,277}]},{aqe_server,'-handle_call/3-fun-1-',2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,98}]},{aqe_server,try_catch_log_error,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,119}]}]}},[]}},{offender,[{pid,<0.8886.1>},{id,gun},{mfargs,{gun,start_link,undefined}},{restart_type,temporary},{shutdown,5000},{child_type,worker}]}]"
ts="2021-07-23 13:22:28.580758+00:00" level=error label="{supervisor,child_terminated}" report="[{supervisor,{local,gun_sup}},{errorContext,child_terminated},{reason,{{owner_gone,{function_clause,[{aqe_grpc,check_grpc_result,[tiger_metadata_store_service_client,get_datasets,{error,timeout}],[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/util/aqe_grpc.erl\"},{line,88}]},{aqe_tracing,external_call_wrapper,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/util/aqe_tracing.erl\"},{line,76}]},{aqe_workspace_models,build_models_from_md,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_workspace_models.erl\"},{line,51}]},{aqe_server,ensure_model_cache,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,309}]},{aqe_server,get_models_by_types,4,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,287}]},{aqe_server,process_valid_objects,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,277}]},{aqe_server,'-handle_call/3-fun-1-',2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,98}]},{aqe_server,try_catch_log_error,2,[{file,\"/builds/TuxBtSUr/0/gooddata/gdc-nas/microservices/aqe/aqe-erlang/src/worker/aqe_server.erl\"},{line,119}]}]}},[]}},{offender,[{pid,<0.8889.1>},{id,gun},{mfargs,{gun,start_link,undefined}},{restart_type,temporary},{shutdown,5000},{child_type,worker}]}]"

 

D
Rank

@Yuriy Balakhonov 

The log exceptions are indeed related to timeout but unfortunately they do not point in any way to the root cause which most probably lies in the overloaded DB (hypothesis). That’s why it’s important for us if you would be able to check postgres database traffic. Especially contents of “pg_stat_activity” table might be of interest.

Regarding configuration: currently it’s not possible to configure timeouts for this particular service.

D
Rank

@Yuriy Balakhonov@Vasiliy Ostrovsky@NickK We have made some progress on this issue and would like to test whether it helps in your particular case. For that we would need the dump of your metadata, namely logical (LDM) and physical (PDM) models.

PDM can be obtained at

/api/layout/dataSources/{data_source_id}/physicalModel

and LDM at

/api/layout/workspaces/{workspaceId}/logicalModel

Also can you describe the usage pattern (concurrency) at the time you hit those issues?

Thanks

Martin Burian
Rank
Userlevel 2

Hello @Yuriy Balakhonov @Vasiliy Ostrovsky @NickK.

We have released a performance optimization in 1.4, which should resolve the issue with the 5s timeouts. If you still see the timeouts in 1.4, please provide the information requested in previous post by @David Kubecka .

Thank you.

Martin

Y

Hello, @Martin Burian 
Unfortunately we still receive the ‘Internal Server Error’


logs:

ts="2021-12-19 20:13:31.062" level=ERROR msg="Internal Server Error" logger=com.gooddata.tiger.web.exception.BaseExceptionHandling thread=grpc-default-executor-443 orgId=default spanId=5cc3f434ec8938d9 traceId=5cc3f434ec8938d9 userId=demo exc="errorType=com.gooddata.tiger.grpc.error.GrpcPropagatedServerException, message=UNKNOWN: ,<no detail>
	at com.gooddata.tiger.grpc.error.ErrorPropagationKt.convertFromUnknownException(ErrorPropagation.kt:108)
	Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException: 
Error has been observed at the following site(s):
	|_ checkpoint ⇢ Handler com.gooddata.tiger.afm.controller.ValidObjectsController#processAfmValidObjectsQuery(String, AfmValidObjectsQuery, Continuation) [DispatcherHandler]
Stack trace:
		at com.gooddata.tiger.grpc.error.ErrorPropagationKt.convertFromUnknownException(ErrorPropagation.kt:108)
		at com.gooddata.tiger.grpc.error.ErrorPropagationKt.convertToTransferableException(ErrorPropagation.kt:98)
		at com.gooddata.tiger.grpc.error.ErrorPropagationKt.clientCatching(ErrorPropagation.kt:72)
		at com.gooddata.tiger.grpc.error.ErrorPropagationKt$clientCatching$1.invokeSuspend(ErrorPropagation.kt)
		at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
		at kotlinx.coroutines.DispatchedTaskKt.resume(DispatchedTask.kt:175)
		at kotlinx.coroutines.DispatchedTaskKt.resumeUnconfined(DispatchedTask.kt:137)
		at kotlinx.coroutines.DispatchedTaskKt.dispatch(DispatchedTask.kt:108)
		at kotlinx.coroutines.CancellableContinuationImpl.dispatchResume(CancellableContinuationImpl.kt:308)
		at kotlinx.coroutines.CancellableContinuationImpl.resumeImpl(CancellableContinuationImpl.kt:318)
		at kotlinx.coroutines.CancellableContinuationImpl.resumeWith(CancellableContinuationImpl.kt:250)
		at com.github.marcoferrer.krotoplus.coroutines.client.SuspendingUnaryObserver.onError(SuspendingUnaryObserver.kt:34)
		at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:478)
		at brave.grpc.TracingClientInterceptor$TracingClientCallListener.onClose(TracingClientInterceptor.java:202)
		at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
		at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
		at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
		at net.devh.boot.grpc.client.metric.MetricCollectingClientCallListener.onClose(MetricCollectingClientCallListener.java:65)
		at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:617)
		at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:70)
		at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:803)
		at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:782)
		at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
		at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
		at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
		at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
		at java.base/java.lang.Thread.run(Unknown Source)
"
172.21.0.1 - - [19/Dec/2021:20:13:31 +0000] "POST /api/actions/workspaces/658b61ad-bda7-42ce-91ca-5ae13fc1bbb9/execution/afm/computeValidObjects HTTP/1.1" 500 124 "https://gooddata-dev.criterionhcm.com/analyze/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.1.2 Safari/605.1.15"

 

Martin Burian
Rank
Userlevel 2

Hello @Yuriy Balakhonov, @Vasiliy Ostrovsky, @NickK.

Unfortunately we were not able to find the root cause of the issue based on the error message.

We would need following information to be able to investigate further:

  • pg_stat_activity
  • physicalModel
  • usage pattern

Could you please share the information?

What we could also do to organize a call with our engineering to check the issue online. Please let us know.

Thank you.

Best regards,

Martin

Martin Burian
Rank
Userlevel 2

Hi @Yuriy Balakhonov, @Vasiliy Ostrovsky, @NickK.

I would like to review if you were able to resolve the issue or it still exists. If the issue still exists could you please provide:

  • pg_stat_activity
  • physicalModel
  • usage pattern

We could also organize a call to check it online with you.

Please let us know.

Thank you.

Martin

Martin Burian
Rank
Userlevel 2

Closing for inactivity.

Join us on
Welcome to the GoodData Community!Cookie settings