Hi Gooddata!
We noticed that our filters that use ...
# gooddata-cnn
Nuno Salgado
04/03/2023, 12:54 PMHi Gooddata!
We noticed that our filters that use dimension tables were not properly filtering the data. Although the fact tables were being filtered correctly, we found that all entries on dimensions were visible to everyone on workspaces. Our database dropped all tables and data, but the table schema remained the same. After the last database update, we called the api/v1/actions/dataSources/{}/uploadNotification endpoint. Do you have any tips on what could have caused this issue?
j
Jan Soubusta
04/03/2023, 2:13 PMAre you aware of the concept of Physical Data Model?
If you change the relational model in you database, you have to re-scan it in GoodData.
We cache the model on our side to do not query DB catalogs too often (it is costly, esp. on clustered DBs like Snowflake).
n
Nuno Salgado
04/03/2023, 2:28 PM@Jan Soubusta, thank you for your response. In this case, we removed the tables, but when we applied the migration, all the tables and columns remained the same. However, something went wrong, and the WDF_fields were not respected, even though they existed in the tables. Regarding automation, which command and API should we use to correctly rescan the data?
j
Jan Soubusta
04/03/2023, 2:35 PMHm, I am wrong, sorry
Jan Soubusta
04/03/2023, 2:39 PMBasically, there is a dropdown(picker) in the left panel of the modelling UI application, containing the list of available data sources.
Jan Soubusta
04/03/2023, 2:40 PMThe popup with the message is displayed when you hover over the underlying button
Jan Soubusta
04/03/2023, 2:40 PMThe button re-scans the data source and refreshes the PDM.
n
Nuno Salgado
04/03/2023, 2:41 PMYes, but we want do this via api or something not on ui. Cause we have a pipeline to send the data, and then we want to call rescan. Basically we and this “re-scans the data source and refreshes the PDM.” using commands or API
j
Jan Soubusta
04/03/2023, 2:41 PMI recommend to use our Python SDK
Jan Soubusta
04/03/2023, 2:43 PMHere is an open sourced repository containing end-to-end demo:
https://gitlab.com/patrikbraborec/gooddata-data-pipeline
The main README contains a description of onboarding and links to articles.
Jan Soubusta
04/03/2023, 2:46 PMOr, you can basically read just the official DOC:
https://www.gooddata.com/docs/python-sdk/1.3/data/physical-data-model/scan_and_put_pdm/
https://www.gooddata.com/docs/python-sdk/1.3/data/data-source/generate_logical_model/
https://www.gooddata.com/docs/python-sdk/1.3/workspace-content/logical-data-model/load_and_put_declarative_ldm/
etc,,
n
Nuno Salgado
04/03/2023, 2:53 PMThanks for your response @Jan Soubusta will take a look on this!