We're seeing an error in the afm-exec-api logs related to timing out requests in the GD.CN UI:

{
  "ts": "2024-01-22 20:14:55.724",
  "level": "ERROR",
  "logger": "com.gooddata.tiger.grpc.client.resultcache.ResultCacheClient",
  "thread": "DefaultDispatcher-worker-1",
  "traceId": "2098d9d948e8cfb5",
  "spanId": "2098d9d948e8cfb5",
  "userId": "james.lee",
  "orgId": "eb6f13c6-4b73-4a09-af89-adfb572d972e",
  "tokenId": "",
  "msg": "getResultCacheResponse",
  "action": "grpcClientCall",
  "exc": "kotlinx.coroutines.JobCancellationException: MonoCoroutine was cancelled; job=MonoCoroutine{Cancelling}@77b61a6c\n"
}

Repro steps: 1. Log into a particular tenant with LDM referencing our Snowflake data source 2. Click the "Analyze" tab, observe "Untitled insight" and a list of fields on the left 3. Drag one of the columns with data to the "ROWS" field in the new insight. In the display area, observe the "Computing" progress indicator 4. The operation appears to time out with the above error in the logs and 404 response from the following endpoint:

https://[hostname]/api/v1/actions/workspaces/b922ec833042440287f8e8837607fb6b/execution/afm/execute/result/6a0c80d11e875a869be6dd2f3c8b727b05046bb4?offset=0%2C0&limit=100%2C1000

The 404 response has the following content:

{
  "title": "Not Found",
  "status": 404,
  "detail": "Result not found in result cache",
  "resultId": "6a0c80d11e875a869be6dd2f3c8b727b05046bb4",
  "traceId": "76ad754f13aaf80a"
}

Note: this procedure works with the same data source on our GCP instance of the same tenant / organization, but times out on our AWS instance of this tenant / organization. The insights were migrated from v2.3.2 on GCP to v3.1.0 on AWS using the AIO method provided in the above thread. The timeout occurs at exactly 3 minutes. I'm attaching the logs for afm-exec-api and result-cache for the relevant time period. Please let us know if any additional info would be helpful to debug this issue. Thanks for any help.

Hi Pete, I believe 3 minutes is timeout set on nginx and when appropriate response is not received the request will timeout. Question is why the insight took that long to compute. To that I’m not sure though. Result ID (

6a0c80d11e875a869be6dd2f3c8b727b05046bb4

in this case) is generated by afm-exec-api during insight execution and it is registered into result cache. If a result is already in result cache in it’ll get picked up immediately and isight is displayed in the UI. If result is not in the result cache sql queries are generated and executed against a datasource. In the attached logs there is only the afm execution registering the result id into result cache and polling for result which ends up in 404 because it can’t be found after the said 3 minutes. I would recommend to: • for the sake of completeness clear datasource caches hitting uploadNotification API to clear caches • Try to execute the insight in analytical designer again (clearing caches often times to resolve various issues) and you can replace

/edit

part of URL by

/debug

. It will download a zip file with afm explanation resources including sql file containing query which is being executed in datasource. • You can try to execute the query directly in the datasource to see how long it takes to finish and/or compare it with the explain resource in your other deployment where it works • It might be worth to check SQL_EXECUTOR pods logs if there are any timeouts on sql queries.

Thanks, Jan. We will investigate.

under the normal circumstances the debug file is downloaded automatically and no other steps are needed, but it looks like there is something else going on… by the nginx log entry it looks like the requests are timing out further down on upstream server in this case I guess result cache service.. but to be honest I’m not sure what could be causing it, I’ll check internally.

Right, we're seeing the debug file on GCP but not on AWS, trying to figure out which part of our AWS deploy is not configured properly

Hi Pete, sorry for being late here, I’m glad you were able to resolve the issue.