Hi. I am trying to install GDCN v3.20.0 in an AWS EKS cluster. Everything seems to be running OK, but there are a few pods that cannot changes to "ready" state. I uninstalled and reinstalled many times, but I don't seem to get anywhere. Here is a list of pods in gooddata-cn namespace. $ kubectl get pod -n gooddata-cn NAME READY STATUS RESTARTS AGE gooddata-cn-afm-exec-api-85b4fff6f4-2lncv 0/1 Running 0 131m gooddata-cn-afm-exec-api-85b4fff6f4-xclh7 0/1 Running 0 131m gooddata-cn-analytical-designer-78d9b9b86-7xjck 1/1 Running 0 131m gooddata-cn-analytical-designer-78d9b9b86-bmp7f 1/1 Running 0 131m gooddata-cn-api-gateway-6dc4586b57-582sm 1/1 Running 0 131m gooddata-cn-api-gateway-6dc4586b57-wz4w4 1/1 Running 0 131m gooddata-cn-apidocs-8bd8cc86-8hb2b 1/1 Running 0 131m gooddata-cn-apidocs-8bd8cc86-wdgkp 1/1 Running 0 131m gooddata-cn-auth-service-856569c6cc-cjlvn 1/1 Running 0 131m gooddata-cn-auth-service-856569c6cc-m87rk 1/1 Running 0 131m gooddata-cn-automation-6f6fb7fcb6-2pgnf 1/1 Running 0 131m gooddata-cn-automation-6f6fb7fcb6-889pp 1/1 Running 0 131m gooddata-cn-calcique-64fc55b64f-gnkz7 0/1 Running 0 131m gooddata-cn-calcique-64fc55b64f-qkjl9 0/1 Running 0 131m gooddata-cn-dashboards-7d485c4867-77pzs 1/1 Running 0 131m gooddata-cn-dashboards-7d485c4867-hbh8c 1/1 Running 0 131m gooddata-cn-dex-5d677f8db5-h4ndq 1/1 Running 0 131m gooddata-cn-dex-5d677f8db5-m6jmc 1/1 Running 0 131m gooddata-cn-etcd-0 1/1 Running 0 131m gooddata-cn-etcd-1 1/1 Running 0 131m gooddata-cn-etcd-2 1/1 Running 0 131m gooddata-cn-export-controller-7bb6f578cc-cwngk 1/1 Running 0 131m gooddata-cn-export-controller-7bb6f578cc-fjff8 1/1 Running 0 131m gooddata-cn-home-ui-85ccc74c56-6fds6 1/1 Running 0 131m gooddata-cn-home-ui-85ccc74c56-wztvh 1/1 Running 0 131m gooddata-cn-ldm-modeler-77d774d8d8-86454 1/1 Running 0 131m gooddata-cn-ldm-modeler-77d774d8d8-r8fx5 1/1 Running 0 131m gooddata-cn-measure-editor-f458f6f7c-jkjbp 1/1 Running 0 131m gooddata-cn-measure-editor-f458f6f7c-vbql8 1/1 Running 0 131m gooddata-cn-metadata-api-7ffb999677-krgg7 0/1 Running 23 (2m51s ago) 131m gooddata-cn-metadata-api-7ffb999677-l2smr 0/1 Running 4 (3m10s ago) 18m gooddata-cn-organization-controller-66955c995b-t5cnt 1/1 Running 0 131m gooddata-cn-organization-controller-66955c995b-zxlgh 1/1 Running 0 131m gooddata-cn-pdf-stapler-service-dbffdd97b-dkspc 1/1 Running 1 (130m ago) 131m gooddata-cn-pdf-stapler-service-dbffdd97b-rw752 1/1 Running 2 (129m ago) 131m gooddata-cn-quiver-cache-7677c8d796-bjcjc 1/1 Running 7 (117m ago) 131m gooddata-cn-quiver-cache-7677c8d796-lpv69 1/1 Running 2 (127m ago) 131m gooddata-cn-quiver-ml-fc5684646-2msg4 1/1 Running 0 131m gooddata-cn-quiver-ml-fc5684646-5l8lb 1/1 Running 0 131m gooddata-cn-quiver-xtab-8fbd94f78-2z9hb 1/1 Running 0 131m gooddata-cn-quiver-xtab-8fbd94f78-cwf6p 1/1 Running 0 131m gooddata-cn-result-cache-866674d7b4-4xlf4 1/1 Running 0 131m gooddata-cn-result-cache-866674d7b4-8kdcr 1/1 Running 0 131m gooddata-cn-scan-model-58c6849dc7-2z27r 1/1 Running 0 131m gooddata-cn-scan-model-58c6849dc7-mfbtq 1/1 Running 0 131m gooddata-cn-sql-executor-6457bc8cd-4jpk5 1/1 Running 1 (129m ago) 131m gooddata-cn-sql-executor-6457bc8cd-qjvxq 1/1 Running 0 131m gooddata-cn-tabular-exporter-7697865869-qzrjd 1/1 Running 0 131m gooddata-cn-tabular-exporter-7697865869-w88f7 1/1 Running 0 131m gooddata-cn-tools-6db9cfb645-c6jbr 1/1 Running 0 131m gooddata-cn-visual-exporter-service-bb9cb76cf-bf8x5 3/3 Running 0 131m gooddata-cn-visual-exporter-service-bb9cb76cf-rj64x 3/3 Running 0 131m gooddata-cn-web-components-87b4dc5dd-fnt76 1/1 Running 0 131m gooddata-cn-web-components-87b4dc5dd-ztbrt 1/1 Running 0 131m $ From the logs, I think afm-exec-api depends on calcique, which in turn depends on metadata-api. But metadata-api is never ready. It restarted many times as you can see above. In the attached metadata-api.log, it always ends with {"ts":"2024-10-19 002608.276","level":"INFO","logger":"liquibase.lockservice","thread":"main","msg":"Waiting for changelog lock...."} Stream closed EOF for gooddata-cn/gooddata-cn-metadata-api-7ffb999677-l2smr (metadata-api) What is that "lock"? How do we resolve this? Any other reasons? Thanks in advance.

Hi, for

{
  "ts": "2024-10-19 00:26:08.276",
  "level": "INFO",
  "logger": "liquibase.lockservice",
  "thread": "main",
  "msg": "Waiting for changelog lock...."
}

Metadata-api uses Liquibase for DB schema management. This tool uses a special database table to prevent accidental simultaneous schema modification from multiple places. When metadata-api pod starts and initializes liquibase, tool tries to insert row to table

dbchangeloglock

. The problem happens when pod is restarted by k8s (e.g. by startup probe timeout) and lock row remains in db. other pods will keep waiting for the lock to be released, killed by k8s liveness probe eventually. The only known solution is to connect to the md datatabase and remove the offending lock using command:

DELETE FROM dbchangeloglock;

Thanks, Jan. That solves it.