Greetings I have a condition on <http GoodData CN|GoodData C GoodData #gooddata-cn

Join Slack

Greetings... I have a condition on <GoodData.CN> c...

# gooddata-cn

Vincil Bishop

08/20/2022, 2:03 PM

Greetings... I have a condition on GoodData.CN community edition, that is really confusing me

Vincil Bishop

08/20/2022, 2:03 PM

I have a data model with several levels... org, project, repository, contributor, with the commit object being at the center. commit -> repo -> project -> org

Vincil Bishop

08/20/2022, 2:04 PM

I have a page that queries activity in a week... so it fires off potentially 7 column charts at once

Vincil Bishop

08/20/2022, 2:05 PM

this query works fine when filtering by org Id, but it fails miserably when filtering by project, repo, or contributor

Vincil Bishop

08/20/2022, 2:05 PM

I am filtering normally on all those attributes in probably 20 or more other places

Vincil Bishop

08/20/2022, 2:06 PM

But this chart fails, by locking up the UI

Vincil Bishop

08/20/2022, 2:06 PM

and these messages are given in the GD server log

Vincil Bishop

08/20/2022, 2:06 PM

Copy code

skan-gd | ts="2022-08-20 13:57:38.756" level=WARN msg="Unable to send telemetry data" logger=com.gooddata.tiger.telemetry.MatomoTelemetryReporter thread=pool-5-thread-1 exc="java.util.concurrent.ExecutionException: java.net.ConnectException: Timeout connecting to [<http://matomo.anywhere.gooddata.com/3.69.247.149:443|matomo.anywhere.gooddata.com/3.69.247.149:443>]
skan-gd | 	at org.apache.http.concurrent.BasicFuture.getResult(BasicFuture.java:71)
skan-gd | 	at org.apache.http.concurrent.BasicFuture.get(BasicFuture.java:84)
skan-gd | 	at org.apache.http.impl.nio.client.FutureWrapper.get(FutureWrapper.java:70)
skan-gd | 	at com.gooddata.tiger.telemetry.MatomoTelemetryReporter.logError(MatomoTelemetryReporter.kt:262)
skan-gd | 	at com.gooddata.tiger.telemetry.MatomoTelemetryReporter.access$logError(MatomoTelemetryReporter.kt:25)
skan-gd | 	at com.gooddata.tiger.telemetry.MatomoTelemetryReporter$logError$1.invokeSuspend(MatomoTelemetryReporter.kt)
skan-gd | 	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
skan-gd | 	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:56)
skan-gd | 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
skan-gd | 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
skan-gd | 	at java.base/java.lang.Thread.run(Unknown Source)
skan-gd | Caused by: java.net.ConnectException: Timeout connecting to [<http://matomo.anywhere.gooddata.com/3.69.247.149:443|matomo.anywhere.gooddata.com/3.69.247.149:443>]
skan-gd | 	at org.apache.http.nio.pool.RouteSpecificPool.timeout(RouteSpecificPool.java:169)
skan-gd | 	at org.apache.http.nio.pool.AbstractNIOConnPool.requestTimeout(AbstractNIOConnPool.java:632)
skan-gd | 	at org.apache.http.nio.pool.AbstractNIOConnPool$InternalSessionRequestCallback.timeout(AbstractNIOConnPool.java:898)
skan-gd | 	at org.apache.http.impl.nio.reactor.SessionRequestImpl.timeout(SessionRequestImpl.java:198)
skan-gd | 	at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processTimeouts(DefaultConnectingIOReactor.java:213)
skan-gd | 	at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:158)
skan-gd | 	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:351)
skan-gd | 	at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:221)
skan-gd | 	at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)
skan-gd | 	... 1 more

Vincil Bishop

08/20/2022, 2:06 PM

Copy code

skan-gd | 172.28.0.1 - - [20/Aug/2022:13:57:29 +0000] "OPTIONS /api/v1/actions/workspaces/app/execution/afm/execute/result/e23ed2beb1cf72a29c7b12ce3f54bdf5240f3384 HTTP/1.1" 204 0 "<http://localhost:4200/>" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36"
skan-gd | ts="2022-08-20 13:57:29.755" level=WARN msg="Unable to send telemetry data" logger=com.gooddata.tiger.telemetry.MatomoTelemetryReporter thread=pool-5-thread-1 exc="java.util.concurrent.ExecutionException: java.net.ConnectException: Timeout connecting to [<http://matomo.anywhere.gooddata.com/3.69.247.149:443|matomo.anywhere.gooddata.com/3.69.247.149:443>]
skan-gd | 	at org.apache.http.concurrent.BasicFuture.getResult(BasicFuture.java:71)
skan-gd | 	at org.apache.http.concurrent.BasicFuture.get(BasicFuture.java:84)
skan-gd | 	at org.apache.http.impl.nio.client.FutureWrapper.get(FutureWrapper.java:70)
skan-gd | 	at com.gooddata.tiger.telemetry.MatomoTelemetryReporter.logError(MatomoTelemetryReporter.kt:262)
skan-gd | 	at com.gooddata.tiger.telemetry.MatomoTelemetryReporter.access$logError(MatomoTelemetryReporter.kt:25)
skan-gd | 	at com.gooddata.tiger.telemetry.MatomoTelemetryReporter$logError$1.invokeSuspend(MatomoTelemetryReporter.kt)
skan-gd | 	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
skan-gd | 	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:56)
skan-gd | 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
skan-gd | 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
skan-gd | 	at java.base/java.lang.Thread.run(Unknown Source)
skan-gd | Caused by: java.net.ConnectException: Timeout connecting to [<http://matomo.anywhere.gooddata.com/3.69.247.149:443|matomo.anywhere.gooddata.com/3.69.247.149:443>]
skan-gd | 	at org.apache.http.nio.pool.RouteSpecificPool.timeout(RouteSpecificPool.java:169)
skan-gd | 	at org.apache.http.nio.pool.AbstractNIOConnPool.requestTimeout(AbstractNIOConnPool.java:632)
skan-gd | 	at org.apache.http.nio.pool.AbstractNIOConnPool$InternalSessionRequestCallback.timeout(AbstractNIOConnPool.java:898)
skan-gd | 	at org.apache.http.impl.nio.reactor.SessionRequestImpl.timeout(SessionRequestImpl.java:198)
skan-gd | 	at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processTimeouts(DefaultConnectingIOReactor.java:213)
skan-gd | 	at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:158)
skan-gd | 	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:351)
skan-gd | 	at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:221)
skan-gd | 	at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)
skan-gd | 	... 1 more
skan-gd | "

Vincil Bishop

08/20/2022, 2:07 PM

The messages in the server log scroll like there are too many requests made when requesting the project, repo, or contributor context, but it works fine with the org context

Vincil Bishop

08/20/2022, 2:07 PM

I am stumped

Vincil Bishop

08/20/2022, 2:07 PM

any insight would be appreciated

Vincil Bishop

08/20/2022, 2:18 PM

@hunger regnuh ^^

Robert Moucha

08/20/2022, 2:50 PM

@Vincil Bishop Hi vincil, the error you see was related to telemetry component. There was a maintenance performed on our side a few hours ago that caused the telemetry service becoming unavailable. Now it should be OK, please try again. Sorry for incovenience.

Robert Moucha

08/20/2022, 2:52 PM

On the other hand, the client-side telemetry component should fail fast when telemetry server is not accessible. Unfortunately this is not the case. I will report it to our dev team to avoid this condition next time.

Vincil Bishop

08/20/2022, 4:02 PM

But my question would be...why does it work when filtering for one attribute, but not for the others?

Vincil Bishop

08/20/2022, 4:05 PM

I just did the query again and the same result...is this something I would need to restart the server to correct?

Robert Moucha

08/20/2022, 4:08 PM

Honestly, I don't know. Backend sends telemetry data asynchronously so it should not affect report computation if the telemetry service is unavailable. But there's also telemetry collector running in UI app (in browser). I don't have much info about its implementation, sorry. I will ask UI developers on Monday

Vincil Bishop

08/20/2022, 4:10 PM

appreciate the weekend response... this is not super super blocking... I'd like to get it working, now more just to understand what I did to break it so I can understand more about the capabilities of the system

Vincil Bishop

08/20/2022, 4:10 PM

In a way I don't think it's due to an external component being down... as the query works like clockwork in the org context, but no others

Vincil Bishop

08/20/2022, 4:10 PM

it's not like this is holding up production or anything

Vincil Bishop

08/20/2022, 4:11 PM

super weird and not something easy to troubleshoot

Vincil Bishop

08/20/2022, 4:11 PM

a moment will come soon when all will be revealed, hahaha

Vincil Bishop

08/20/2022, 4:11 PM

we always figure it out eventually

hunger regnuh

08/20/2022, 4:11 PM

can’t you enable log trace and have it show the sql query?

Robert Moucha

08/20/2022, 4:12 PM

Yes it is possible

Vincil Bishop

08/20/2022, 4:12 PM

yikes...like on the... gd postgres instance?

hunger regnuh

08/20/2022, 4:13 PM

yep, then just run it natively in pgsql and trace the problem that way - i’m thinking there’s some kind of recursive lookup happening

Vincil Bishop

08/20/2022, 4:13 PM

possibly

Vincil Bishop

08/20/2022, 4:13 PM

but I cleaned up the LDM to remove any duplicate paths

Vincil Bishop

08/20/2022, 4:13 PM

it's all pretty straightforward now

Vincil Bishop

08/20/2022, 4:14 PM

it's like it kicks off too many requests, then a backend mechanism gets overloaded...then the timeouts start

Robert Moucha

08/20/2022, 4:14 PM

Either on postgres, or you can start docker container with "-e APP_LOGLEVEL=INFO" (default is WARN) to emit the sql queries that are generated by Calcique component.

Vincil Bishop

08/20/2022, 4:15 PM

Got it... will try the enhanced log level

Vincil Bishop

08/20/2022, 4:15 PM

and this is on local machine BTW

Vincil Bishop

08/20/2022, 4:15 PM

in docker compose

Robert Moucha

08/20/2022, 4:15 PM

check out https://www.gooddata.com/developers/cloud-native/doc/cloud/deploy-and-install/community-edition/advanced-config/ for further config options for performance improvements

Vincil Bishop

08/20/2022, 4:15 PM

but it also happens in the cloud

Vincil Bishop

08/20/2022, 4:16 PM

a few nice gems in the advanced config doc

Robert Moucha

08/20/2022, 4:17 PM

We're going to write detailed article dedicated to performance, but it will be focused on helm chart version. gooddata-cn-ce (the single docker image) is a bit limited by the used technology.

Vincil Bishop

08/20/2022, 4:18 PM

sure

Robert Moucha

08/20/2022, 4:18 PM

I plan to make some improvements, w.r.t. memory tuning for specific microservices.

Robert Moucha

08/20/2022, 4:20 PM

currently you can only set up JVM parameters for all services altogether. But if you have plenty of RAM, you could give it a try.

Vincil Bishop

08/20/2022, 4:20 PM

I think it's model related

Vincil Bishop

08/20/2022, 4:20 PM

as it works on one context, organization

Vincil Bishop

08/20/2022, 4:20 PM

and not others... project, repo, contributor

Robert Moucha

08/20/2022, 4:21 PM

I see, can you share the LDM screenshot? It looks like github stats or so

Robert Moucha

08/20/2022, 4:22 PM

one of my colleagues just published an article on this topic: https://medium.com/gooddata-developers/how-to-build-a-modern-data-pipeline-cfdd9d14fbea

Robert Moucha

08/20/2022, 5:15 PM

I can't identify any weak spot at the moment. when you increase the log level, you can grab the sqls from log. Alternatively, there's so-called "explainAFM" resource on our API https://www.gooddata.com/developers/cloud-native/doc/cloud/api-and-sdk/api/api_reference_all/#/actions/explainAFM If you are using Analyze tab, you can get the same relevant details on that page if you replace the "edit" part of URL with "debug". A zip file containing diagnostic information will be downloaded, including the SQL

Vincil Bishop

08/20/2022, 5:16 PM

thanks abunch... will give that a try and let you know

Open in Slack

Previous Next