Getting the below error while creating / updating an organization. All services and pods are running healthy. Org ingress is getting created but if I open the URL it gives 404 with message No organization found for hostname. From the organization-controller log the below error is found. [2022-05-04 202713,720] kopf.objects [INFO ] [gooddata-cn/test-2 starting create_fn [2022-05-04 202713,886] kopf.objects [INFO ] [gooddata-cn/test-2] Creating Ingress for organization 'test-2' [2022-05-04 202714,675] kopf.objects [INFO ] [gooddata-cn/test-2] Organization "Test2" will be created/updated. [2022-05-04 202714,685] kopf.objects [ERROR ] [gooddata-cn/test-2] Handler 'create_fn' failed with an exception. Will retry. Traceback (most recent call last): File "/usr/local/lib/python3.8/site-packages/kopf/_core/actions/execution.py", line 378, in invoke_handler result = await invocation.invoke( File "/usr/local/lib/python3.8/site-packages/kopf/_core/actions/invocation.py", line 140, in invoke await asyncio.shield(future) # slightly expensive: creates tasks File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 57, in run result = self.fn(*self.args, **self.kwargs) File "/app/controller.py", line 274, in create_fn org_exist = ensure_organization(logger, spec) File "/app/controller.py", line 78, in ensure_organization metadata_grpc_client.create_organization(id, org_name, hostname, adminGroup, adminUser, File "/app/metadata_client.py", line 69, in create_organization self.stub.ensureOrganizationExists(org_msg, metadata=self.call_metadata) File "/usr/local/lib/python3.8/site-packages/grpc/_channel.py", line 946, in call return _end_unary_response_blocking(state, call, False, None) File "/usr/local/lib/python3.8/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking raise _InactiveRpcError(state) grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with: status = StatusCode.INTERNAL details = "UNAVAILABLE: io exception" debug_error_string = "{"created":"@1651697072.585607257","description":"Error received from peer ipv4&lt;meta api service ip address&gt;6572","file":"src/core/lib/surface/call.cc","file_line":1069,"grpc_message":"UNAVAILABLE: io exception","grpc_status":13}"

Hi, this error suggests the metadata-api deployment is not accessible. please check logs from both gooddata-cn-metadata-api logs. It is also possible that metadata pods are not reachable from org-controller. If this is a transient network error, this request is retried by org-controller every minute until it eventually succeeds. You'll see it reported in active org-controller pod logs as:

Handler 'create_fn' succeeded.               
Creation is processed: 1 succeeded; 0 failed.

But check metadata-api logs first because it is the most common source of issues in similar cases