https://www.gooddata.com/ logo
Join Slack
Powered by
Hi Team, {"title":"Internal Server Error","status"...
# gooddata-cn
h
Hi Team, {"title":"Internal Server Error","status":500,"detail":"Server-side problem. Contact support.","traceId":"839116efa8672905"} All pods are running fine only
Copy code
gooddata-cn-metadata-api 0/1 
gooddata-cn-sql-executor 0/1
Copy code
Normal  Pulled   5m38s          kubelet Successfully pulled image "gooddata/sql-executor:1.7.0" in 160.257957ms
 Normal  Started  5m37s (x2 over 8m35s)  kubelet Started container sql-executor
 Warning Unhealthy 3m28s (x22 over 7m58s) kubelet Startup probe failed: Get "<http://10.0.2.4:9101/actuator/health/liveness>": dial tcp 10.0.2.4:9101: connect: connection refused
Any Idea?
r
Hi Hamid, I can't say much without providing logs from the failing containers. But my guess is that these pods have problems while applying DB schema change. Please check the logs and if by any chance there are messages like "waiting for database change lock" (sorry I don't remember the exact phrase), please let me know. I will provide you details on how to fix that issue. It will require connection to the "md" and "execution" databases.
h
Hi Robert
For metadata:
Copy code
{"ts":"2022-10-12 10:56:23.849","level":"INFO","logger":"com.zaxxer.hikari.HikariDataSource","thread":"main","msg":"HikariPool-1 - Starting..."}
{"ts":"2022-10-12 10:56:24.220","level":"INFO","logger":"com.zaxxer.hikari.HikariDataSource","thread":"main","msg":"HikariPool-1 - Start completed."}
{"ts":"2022-10-12 10:56:24.849","level":"INFO","logger":"liquibase.lockservice","thread":"main","msg":"Waiting for changelog lock...."}
{"ts":"2022-10-12 10:56:34.855","level":"INFO","logger":"liquibase.lockservice","thread":"main","msg":"Waiting for changelog lock...."}
{"ts":"2022-10-12 10:56:44.861","level":"INFO","logger":"liquibase.lockservice","thread":"main","msg":"Waiting for changelog lock...."}
{"ts":"2022-10-12 10:56:54.867","level":"INFO","logger":"liquibase.lockservice","thread":"main","msg":"Waiting for changelog lock...."}
{"ts":"2022-10-12 10:57:04.874","level":"INFO","logger":"liquibase.lockservice","thread":"main","msg":"Waiting for changelog lock...."}
{"ts":"2022-10-12 10:57:14.880","level":"INFO","logger":"liquibase.lockservice","thread":"main","msg":"Waiting for changelog lock...."}
{"ts":"2022-10-12 10:57:24.886","level":"INFO","logger":"liquibase.lockservice","thread":"main","msg":"Waiting for changelog lock...."}
{"ts":"2022-10-12 10:57:34.891","level":"INFO","logger":"liquibase.lockservice","thread":"main","msg":"Waiting for changelog lock...."}
{"ts":"2022-10-12 10:57:44.896","level":"INFO","logger":"liquibase.lockservice","thread":"main","msg":"Waiting for changelog lock...."}
{"ts":"2022-10-12 10:57:54.901","level":"INFO","logger":"liquibase.lockservice","thread":"main","msg":"Waiting for changelog lock...."}
{"ts":"2022-10-12 10:58:04.907","level":"INFO","logger":"liquibase.lockservice","thread":"main","msg":"Waiting for changelog lock...."}
{"ts":"2022-10-12 10:58:14.912","level":"INFO","logger":"liquibase.lockservice","thread":"main","msg":"Waiting for changelog lock...."}
For sql-executor
Copy code
{"ts":"2022-10-12 11:04:13.570","level":"INFO","logger":"oadd.org.apache.drill.common.util.GuavaPatcher","thread":"main","msg":"Google's Closeables patched for old HBase Guava version."}
{"ts":"2022-10-12 11:04:13.957","level":"INFO","logger":"com.zaxxer.hikari.HikariDataSource","thread":"main","msg":"HikariPool-1 - Start completed."}
{"ts":"2022-10-12 11:04:14.589","level":"INFO","logger":"liquibase.lockservice","thread":"main","msg":"Waiting for changelog lock...."}
{"ts":"2022-10-12 11:04:24.595","level":"INFO","logger":"liquibase.lockservice","thread":"main","msg":"Waiting for changelog lock...."}
{"ts":"2022-10-12 11:04:34.605","level":"INFO","logger":"liquibase.lockservice","thread":"main","msg":"Waiting for changelog lock...."}
r
yes, that's it what I meant
h
Thank you, I do have access to DB
r
great. Connect to db (if you're using postgresql-ha subchart, connect through pgpool pod so your sql commands will be directed to the primary db)
connect to 
md
database
h
I am using pgadmin4
r
it doesn't matter what client are you using.
👍 1
Just run SQL command:
Copy code
delete from databasechangeloglock;
in md database. Run the same command in 
execution
database. Eventually, both sql-executor and metadata-api will start.
You can make them to start faster, if you delete the crash-looping pods.
they will be started by k8s and correctly grab the db lock.
h
Screen Shot 2022-10-12 at 4.13.13 AM.png
👍 1
r
Yes, this is expected result. One deleted row.
🙌 1
h
Thank you so much @Robert Moucha, Issue resolved
👍 1
🙌 1
Open in Slack
PreviousNext
Powered by