Solved

Can't initialize BigQuery connection - "Headless Exception" error

  • 23 August 2021
  • 18 replies
  • 268 views

We’re on the final steps of setting up GoodData.CN, using BigQuery as our data source.

Everything has been going according to the documentation, up until the JDBC driver part. It does seem to load, but now we’re getting the following errors:

  1. when testing the connection via API: 
    {
    "error": "Could not initialize connection to data source; caused by java.sql.SQLException: [JDBC Driver]null; caused by java.awt.HeadlessException",
    "successful": false
    }

     

  2. in the front end: 
    scan error={\"detail\":\"Could not initialize connection to data source with id = sandbox-analytics-01; caused by java.sql.SQLException: [JDBC Driver]null; caused by java.awt.HeadlessException\",\"status\":400,\"title\":\"Bad Request\",\"traceId\":\"c61e3d6f6bdc4b5c\"}"

     

  3. and in the k8s pod: 
    {"ts":"2021-08-23 14:03:55.725","level":"ERROR","logger":"com.zaxxer.hikari.pool.HikariPool","thread":"DefaultDispatcher-worker-2","traceId":"1d81ad1e182ebdab","spanId":"e2268e780ca7b1da","userId":"admin","orgId":"syn","msg":"HikariPool-4 - Exception during pool initialization.","exc":"java.sql.SQLException: [JDBC Driver]null\n\tat java.desktop/java.awt.GraphicsEnvironment.checkHeadless(Unknown Source)\n\tat java.desktop/java.awt.Window.<init>(Unknown Source)\n\tat java.desktop/java.awt.Frame.<init>(Unknown Source)\n\tat java.desktop/javax.swing.JFrame.<init>(Unknown Source)\n\tat com.simba.googlebigquery.googlebigquery.client.BQClient.userAccountOAuth(Unknown Source)\n\tat com.simba.googlebigquery.googlebigquery.core.BQConnection.connect(Unknown Source)\n\tat com.simba.googlebigquery.jdbc.common.BaseConnectionFactory.doConnect(Unknown Source)\n\tat com.simba.googlebigquery.jdbc.common.[...]

 

Any ideas what might be causing this?

icon

Best answer by jku 26 August 2021, 15:18

View original

18 replies

Hello @dastein,

it looks like you’re using the User Account OAuth from the k8s pod stacktrace you provided. However GoodData.CN currently only supports Service Account OAuth as stated in BigQuery - Data source details.

Please, let me know if that lead helped or it is not the case.

Regards,
Dominik

Hi @Dominik,

thanks for the quick response: We do specify AuthType=0 in the JDBC URL, shouldn’t that be enough to set Service Account OAuth?

 

Best,

David

 

Adding the 

OAuthType=0

to the JDBC URL should be, by all accounts, sufficient. According to the driver developer docs, it should even be the default. So you can try removing it from your URL and it should technically work. It is weird that userAccountOAuth method was invoked in the stacktrace, serviceAccountOAuth should have been called.

Would you mind sharing the JDBC URL of the datasource that is failing the connection test? 

Dominik

Certainly:

{
"schema": "dbt_production",
"type": "BIGQUERY",
"url": "jdbc:bigquery://https://www.googleapis.com/bigquery/v2:443;ProjectId=sandbox-analytics-01;OAuthType=0",
"username": "",
"password": "",
"token": "ewog....."
}

 

Userlevel 1

We produce a log message starting with “BQ connection config” containing JDBC URL generated by our platform.

Can you please find it in your log and comment it here?

Where would I find this log message? In which pod/service?

 

Userlevel 1

sql-executor

Here are the captured logs from the two pods. Both show errors. The file ending with pod1 shows logs after I hit the button below "Data Source connected. Now scan the data." The file ending with pod2 show logs after I did the Data Source test over the API (with the same settings as the stored datasource").
 
sql-executor-pod1.log#85
jdbc:bigquery://https://www.googleapis.com/bigquery/v2:443;ProjectId=sandbox-analytics-01;OAuthType=0;OAuthServiceAcctEmail=user@gooddata.com;DefaultDataset=dbt_production;OAuthPvtKeyPath=/tmp/tmp9519416683194855305bigquery_token.json
 
When I run
kubectl exec -it pod/gooddata-cn-sql-executor-5b9bd57544-s5n4g  -n gooddata-cn -c sql-executor -- cat /tmp/tmp9519416683194855305bigquery_token.json

it outputs the service account file (see below) in JSON.

 
 
sql-executor-pod2.log#69
jdbc:bigquery://https://www.googleapis.com/bigquery/v2:443;ProjectId=sandbox-analytics-01;OAuthType=0;OAuthServiceAcctEmail=user@gooddata.com;DefaultDataset=dbt_production;OAuthPvtKeyPath=/tmp/tmp7897009948474282971bigquery_token.json

 

When I run..
kubectl exec -it pod/gooddata-cn-sql-executor-5b9bd57544-x4v89  -n gooddata-cn -c sql-executor -- ls -alg /tmp/tmp7897009948474282971bigquery_token.json
-rw------- 1 bin 0 Aug 23 17:31 /tmp/tmp7897009948474282971bigquery_token.json

I can see this tmp file is empty.

 
Service Account File
Below is the service account file (with private key removed here)
{
  "type": "service_account",
  "project_id": "sandbox-analytics-01",
  "private_key_id": "e00376deb35096646b18b34c9c1f16fb85c5db4b",
  "private_key": "...",
  "client_email": "gooddata@sandbox-analytics-01.iam.gserviceaccount.com",
  "client_id": "105292823964650792634",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://oauth2.googleapis.com/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/gooddata%40sandbox-analytics-01.iam.gserviceaccount.com"
}

 

 
I observe that OAuthServiceAcctEmail parameter in JDBC URLs do not match the client_email in Service Account File (that was base64 encoded and stored as token in the Data Source).
 
Maybe these oberservations help?!
Userlevel 1

We tested this setup against our BigQuery instance.

We realized that OAuthServiceAcctEmail in URL has no effect unless it is not present at all, then server returns error.

So we decided to set it to “noop” value “user@gooddata.com” and not to extract it from service account files.

May I kindly ask you test this setup (with noop email in URL) from any Database Management tool like Dbeaver / Datagrip?

Let me know if it works against your environment.

If it does not work, test it also with correct email in URL.

Let me know the results here.

Thanks!

Test seems to work.

Userlevel 1

We tried to reproduce your issue in our K8S deployment with multiple replicas of sql-executor, but everything is working properly on our side so far.

We are digging deeper and will let you know once we find something interesting.

Userlevel 2

Indeed, there’s an error in /api/actions/dataSource/test API preventing a proper operation of this test API on BigQuery datasources.

It also explains why the tempfile is empty, because token is mistakenly ignored. I have filed internal issue and we will fix it in upcomming release.

 

But it still doesn’t explain the original error java.awt.HeadlessException induced by bigquery driver’s attempt to render user login dialogue.

 

Could you please try to reproduce the issue the following way:

  1. delete your data source (DELETE /api/entities/dataSources/YOUR-BQ-DATASOURCE-ID)
  2. construct body of createDatasource API request:
{
"data": {
"attributes": {
"name": "YOUR BQ DATASOURCE",
"url": "jdbc:bigquery://https://www.googleapis.com/bigquery/v2:443;ProjectId=sandbox-analytics-01;OAuthType=0",
"schema": "dbt_production",
"type": "BIGQUERY",
"username": "",
"password": "",
"token": "ewog..."
},
"id": "YOUR-BQ-DATASOURCE-ID",
"type": "dataSource"
}
}
  1. Submit this request using POST /api/entities/dataSources (with Content-Type: application/vnd.gooddata.api+json)
  2. perform Logical Data Model scan of your datasource in “Data” tab in UI

We tried to reproduce your issue in our K8S deployment with multiple replicas of sql-executor, but everything is working properly on our side so far.

We are digging deeper and will let you know once we find something interesting.


Hi!

Could you please provide the link to the exact driver/version you are using for BigQuery so we test with the exact same one?

Best - j

Userlevel 2

For us, it works fine with the older version 1.2.4.1007 as well as with the latest 1.2.19.1023.

You’re using 1.2.16.1020 so I guess it should work as well.

 

For us, it works fine with the older version 1.2.4.1007 as well as with the latest 1.2.19.1023.

You’re using 1.2.16.1020 so I guess it should work as well.

 


Okay, so we tested in k8s setup with 1.2.14.1017 and just now also with 1.2.19.1023. Both downloaded from https://storage.googleapis.com/simba-bq-release/jdbc/SimbaJDBCDriverforGoogleBigQuery42_1.2.14.1017.zip
https://storage.googleapis.com/simba-bq-release/jdbc/SimbaJDBCDriverforGoogleBigQuery42_1.2.19.1023.zip

Both produce the same kind of exceptions..

It says “HeadlessException” in the thread title… the actual exceptions @dastein and me get look like this though (see below).

Any ideas..?

 

{"ts":"2021-08-26 11:15:05.325","level":"WARN","logger":"com.github.benmanes.caffeine.cache.LocalAsyncCache","thread":"DefaultDispatcher-worker-2","traceId":"04a19400a60fde7e","spanId":"99ffe5696ee014b8","userId":"jku","orgId":"syn","msg":"Exception thrown during asynchronous load","exc":"java.util.ServiceConfigurationError: io.grpc.ManagedChannelProvider: io.grpc.netty.NettyChannelProvider not a subtype\n\tat java.base/java.util.ServiceLoader.fail(Unknown Source)\n\tat java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNextService(Unknown Source)\n\tat java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNext(Unknown Source)\n\tat java.base/java.util.ServiceLoader$2.hasNext(Unknown Source)\n\tat java.base/java.util.ServiceLoader$3.hasNext(Unknown Source)\n\tat io.grpc.ServiceProviders.getCandidatesViaServiceLoader(ServiceProviders.java:111)\n\tat io.grpc.ServiceProviders.loadAll(ServiceProviders.java:64)\n\tat io.grpc.ManagedChannelRegistry.getDefaultRegistry(ManagedChannelRegistry.java:96)\n\tat io.grpc.ManagedChannelProvider.provider(ManagedChannelProvider.java:41)\n\tat io.grpc.ManagedChannelBuilder.forAddress(ManagedChannelBuilder.java:39)\n\tat com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.createSingleChannel(InstantiatingGrpcChannelProvider.java:293)\n\tat com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.access$1700(InstantiatingGrpcChannelProvider.java:74)\n\tat com.google.api.gax.grpc.InstantiatingGrpcChannelProvider$1.createSingleChannel(InstantiatingGrpcChannelProvider.java:220)\n\tat com.google.api.gax.grpc.ChannelPool.create(ChannelPool.java:72)\n\tat com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.createChannel(InstantiatingGrpcChannelProvider.java:227)\n\tat com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.getTransportChannel(InstantiatingGrpcChannelProvider.java:210)\n\tat com.google.api.gax.rpc.ClientContext.create(ClientContext.java:169)\n\tat com.google.cloud.bigquery.storage.v1.stub.EnhancedBigQueryReadStub.create(EnhancedBigQueryReadStub.java:89)\n\tat com.google.cloud.bigquery.storage.v1.BigQueryReadClient.<init>(BigQueryReadClient.java:129)\n\tat com.google.cloud.bigquery.storage.v1.BigQueryReadClient.create(BigQueryReadClient.java:110)\n\tat com.simba.googlebigquery.googlebigquery.client.BQClient.createBigQueryStorageClient(Unknown Source)\n\tat com.simba.googlebigquery.googlebigquery.client.BQClient.serviceAccountOAuth(Unknown Source)\n\tat com.simba.googlebigquery.googlebigquery.core.BQConnection.connect(Unknown Source)\n\tat com.simba.googlebigquery.jdbc.common.BaseConnectionFactory.doConnect(Unknown Source)\n\tat com.simba.googlebigquery.jdbc.common.AbstractDriver.connect(Unknown Source)\n\tat com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:138)\n\tat com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:364)\n\tat com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:206)\n\tat com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:476)\n\tat com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:561)\n\tat com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:115)\n\tat com.zaxxer.hikari.HikariDataSource.<init>(HikariDataSource.java:81)\n\tat com.gooddata.tiger.sqlexecutor.db.ConnectedDataSource.<init>(ConnectedDataSource.kt:22)\n\tat com.gooddata.tiger.sqlexecutor.db.DataSourceMapper$getConnectedDataSource$1.invoke(DataSourceMapper.kt:50)\n\tat com.gooddata.tiger.sqlexecutor.db.DataSourceMapper$getConnectedDataSource$1.invoke(DataSourceMapper.kt:22)\n\tat com.gooddata.tiger.sqlexecutor.db.DataSourceMapper.getDatabaseVendor(DataSourceMapper.kt:64)\n\tat com.gooddata.tiger.sqlexecutor.db.DataSourceMapper.getConnectedDataSource(DataSourceMapper.kt:48)\n\tat com.gooddata.tiger.sqlexecutor.db.ConnectionService$getDataSource$2.invokeSuspend(ConnectionService.kt:58)\n\tat kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)\n\tat kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:56)\n\tat kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571)\n\tat kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:738)\n\tat kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678)\n\tat kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)\n"}
{"ts":"2021-08-26 11:15:05.334","level":"ERROR","logger":"com.gooddata.tiger.sqlexecutor.datasource.metadata.service.DataSourceMetadataService","thread":"DefaultDispatcher-worker-4","traceId":"04a19400a60fde7e","spanId":"99ffe5696ee014b8","userId":"jku","orgId":"syn","msg":"gRPC server call","exc":"java.util.ServiceConfigurationError: io.grpc.ManagedChannelProvider: io.grpc.netty.NettyChannelProvider not a subtype\n\tat java.base/java.util.ServiceLoader.fail(Unknown Source)\n\tat java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNextService(Unknown Source)\n\tat java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNext(Unknown Source)\n\tat java.base/java.util.ServiceLoader$2.hasNext(Unknown Source)\n\tat java.base/java.util.ServiceLoader$3.hasNext(Unknown Source)\n\tat io.grpc.ServiceProviders.getCandidatesViaServiceLoader(ServiceProviders.java:111)\n\tat io.grpc.ServiceProviders.loadAll(ServiceProviders.java:64)\n\tat io.grpc.ManagedChannelRegistry.getDefaultRegistry(ManagedChannelRegistry.java:96)\n\tat io.grpc.ManagedChannelProvider.provider(ManagedChannelProvider.java:41)\n\tat io.grpc.ManagedChannelBuilder.forAddress(ManagedChannelBuilder.java:39)\n\tat com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.createSingleChannel(InstantiatingGrpcChannelProvider.java:293)\n\tat com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.access$1700(InstantiatingGrpcChannelProvider.java:74)\n\tat com.google.api.gax.grpc.InstantiatingGrpcChannelProvider$1.createSingleChannel(InstantiatingGrpcChannelProvider.java:220)\n\tat com.google.api.gax.grpc.ChannelPool.create(ChannelPool.java:72)\n\tat com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.createChannel(InstantiatingGrpcChannelProvider.java:227)\n\tat com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.getTransportChannel(InstantiatingGrpcChannelProvider.java:210)\n\tat com.google.api.gax.rpc.ClientContext.create(ClientContext.java:169)\n\tat com.google.cloud.bigquery.storage.v1.stub.EnhancedBigQueryReadStub.create(EnhancedBigQueryReadStub.java:89)\n\tat com.google.cloud.bigquery.storage.v1.BigQueryReadClient.<init>(BigQueryReadClient.java:129)\n\tat com.google.cloud.bigquery.storage.v1.BigQueryReadClient.create(BigQueryReadClient.java:110)\n\tat com.simba.googlebigquery.googlebigquery.client.BQClient.createBigQueryStorageClient(Unknown Source)\n\tat com.simba.googlebigquery.googlebigquery.client.BQClient.serviceAccountOAuth(Unknown Source)\n\tat com.simba.googlebigquery.googlebigquery.core.BQConnection.connect(Unknown Source)\n\tat com.simba.googlebigquery.jdbc.common.BaseConnectionFactory.doConnect(Unknown Source)\n\tat com.simba.googlebigquery.jdbc.common.AbstractDriver.connect(Unknown Source)\n\tat com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:138)\n\tat com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:364)\n\tat com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:206)\n\tat com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:476)\n\tat com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:561)\n\tat com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:115)\n\tat com.zaxxer.hikari.HikariDataSource.<init>(HikariDataSource.java:81)\n\tat com.gooddata.tiger.sqlexecutor.db.ConnectedDataSource.<init>(ConnectedDataSource.kt:22)\n\tat com.gooddata.tiger.sqlexecutor.db.DataSourceMapper$getConnectedDataSource$1.invoke(DataSourceMapper.kt:50)\n\tat com.gooddata.tiger.sqlexecutor.db.DataSourceMapper$getConnectedDataSource$1.invoke(DataSourceMapper.kt:22)\n\tat com.gooddata.tiger.sqlexecutor.db.DataSourceMapper.getDatabaseVendor(DataSourceMapper.kt:64)\n\tat com.gooddata.tiger.sqlexecutor.db.DataSourceMapper.getConnectedDataSource(DataSourceMapper.kt:48)\n\tat com.gooddata.tiger.sqlexecutor.db.ConnectionService$getDataSource$2.invokeSuspend(ConnectionService.kt:58)\n\tat kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)\n\tat kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:56)\n\tat kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571)\n\tat kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:738)\n\tat kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678)\n\tat kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)\n"}

 

Userlevel 2

@jku - thank you for the full exception stack, it was actually very helpful. The error shows a classloading confilict in gRPC library.

 

Are you sure you properly followed a procedure described in https://www.gooddata.com/developers/cloud-native/doc/1.3/administration/data-sources/supported/bigquery/#prerequisite and removed the gRPC API library from Simba BigQuery driver, before building an image with db-drivers?

cd <folder where driver is unpacked>

rm grpc-api-*.jar

 

 

Userlevel 2

FYI, in the upcoming release 1.4, this extra step with file removal will not be required.

That was it! :boom::rocket::earth_africa:


I removed the grpc-api-*.jar file, rebuild the image and reinstalled the helm chart.

So why did I not followed a procedure [1] properly?

At first I worked with the non-offcial dbschema.com drivers and than switched to the offical drivers.

And I admit I had not taken the step 3 seriously. To me “Execute the following script to delete the libraries that aren’t required for JDBC functionality:” had read like an optional step – rather than a mandatory one that would cause serious errors if one ignores it.

So probably a clear statement about the error potential would be a helpful improvement of the docu.

Thanks @jacek and @Robert Moucha and @Dominik Labuda for your help!

[1] https://www.gooddata.com/developers/cloud-native/doc/1.3/administration/data-sources/supported/bigquery/#prerequisite

Reply