We tried upgrading GD.CN in our dev environment from version 3.1 to 3.5.5 and encountered issues with quiver-xtab and quiver-cache. In quiver-xtab, we see the following error:

2024-03-21T00:09:31.405665124Z {"module":"quiver_dataframe","logger":"quiver.server","level":"critical","timestamp":"2024-03-21T00:09:31.403544Z","exception":"Traceback (most recent call last):\n  File \"/usr/local/lib/python3.11/site-packages/quiver_dataframe/module_config.py\", line 144, in _parse_memory_usage_limit\n    limit_bytes = qm.convert_limit_to_bytes(val)\n                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.11/site-packages/quiver_core/config/utils.py\", line 77, in convert_limit_to_bytes\n    raise ValueError(f\"The limit must not be negative. Got: '{amount}'.\")\nValueError: The limit must not be negative. Got: '-254'.\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"/usr/local/lib/python3.11/site-packages/quiver_node/node_main.py\", line 41, in _modules_from_configuration\n    module_instance = im.QUIVER_MODULE.create(server_params, args, settings)\n                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.11/site-packages/quiver_dataframe/module.py\", line 57, in create\n    config = create_config(\n             ^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.11/site-packages/quiver_dataframe/module_config.py\", line 242, in create_config\n    limit_bytes, limit_perc = _parse_memory_usage_limit(memory_usage_limit)\n                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.11/site-packages/quiver_dataframe/module_config.py\", line 146, in _parse_memory_usage_limit\n    limit_perc = _parse_memory_usage_perc_limit(val)\n                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.11/site-packages/quiver_dataframe/module_config.py\", line 118, in _parse_memory_usage_perc_limit\n    raise ValidationError(_LIMIT_E
RROR_MSG)\ndynaconf.validator.ValidationError: Value of 'dataframe_mem_limit' must be in format #<k|M|G|%>. When using %, the value must be from interval <1, 75>.","action":"module_initialization_failed"}
Loading settings from ('/tmp/quiver-xtab.conf.toml',)
Failed to instantiate module 'quiver_dataframe'

In quiver-cache, we see the following:

{"module":"quiver_shard","logger":"quiver.server","level":"critical","timestamp":"2024-03-21T00:13:32.044639Z","exception":"Traceback (most recent call last):\n  File \"/usr/local/lib/python3.11/site-packages/quiver_node/node_main.py\", line 41, in _modules_from_configuration\n    module_instance = im.QUIVER_MODULE.create(server_params, args, settings)\n                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.11/site-packages/quiver_shard/module.py\", line 66, in create\n    config = create_config(\n             ^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.11/site-packages/quiver_shard/module_config.py\", line 156, in create_config\n    storage_config = qc.get_valid_storage_service_config(settings)\n                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.11/site-packages/quiver_core/config/local_storage_config.py\", line 422, in get_valid_storage_service_config\n    settings.validators.validate()\n  File \"/usr/local/lib/python3.11/site-packages/dynaconf/validator.py\", line 492, in validate\n    validator.validate(\n  File \"/usr/local/lib/python3.11/site-packages/dynaconf/validator.py\", line 216, in validate\n    self._validate_items(\n  File \"/usr/local/lib/python3.11/site-packages/dynaconf/validator.py\", line 306, in _validate_items\n    raise ValidationError(_message, details=[(self, _message)])\ndynaconf.validator.ValidationError: Specify MEMORY_CACHE_SIZE in format '<amount><M|G>'","action":"module_initialization_failed"}
Loading settings from ('/tmp/quiver-cache.conf.toml',)
2024-03-21T00:13:32.047544013Z Failed to instantiate module 'quiver_shard'

It seems there's an issue with our configuration but we're unsure which variable in the helm corresponds to the variable that doesn't validate in the code. Attaching our values.yaml file (values for all envs are declared under helm.values in fleet.yaml). Thanks for suggestions. 😀

Hi, there's a bug in helm chart. Please convert values of .quiver.resources.cache.limits.memory and .quiver.resources.xtab.limits.memory from GiB to MiB

Perfect, thanks Robert!

sorry for troubles, I will file internal issue to fix that But for now, just use 2048Mi instead of 2Gi etc.

will do, thank you for the guidance

Hi Pete, the second error might be related to

deployQuiverDatasourceFs

the following settings can be removed from values files

quiver:
      replicaCount:
        datasource: 2
      resources:
        datasource:
          limits:
            cpu: 600m
            memory: 4Gi
          requests:
            cpu: 600m
            memory: 4Gi

s3DatasourceFsStorage:
    s3Bucket: "gooddata-datasource-fs-storage-beta"

deployQuiverDatasourceFs: true

These settings are used us internally for CSV uploads functionality and currently are not used in CN deployments

no, both errors are caused by mem given in GiB

makes sense, thanks for the explanation 🙏

I stand corrected, thanks Robert