https://www.gooddata.com/ logo
Join Slack
Powered by
Hello, my org is migrating from BigQuery to Snowfl...
# gooddata-cn
s
Hello, my org is migrating from BigQuery to Snowflake, and switching from GCP to AWS and we are looking for guidance: • What do we need to do to migrate insights? Preferably we don't have to recreate them manually • What is your recommendation to migrate GD-CN from GCP to AWS without having to recreate the insights manually? Any advice is appreciated
d
Hi Sheila! Since the LDM in the layout includes explicit references to the data source used in the source workspace, without the references, it can break. One possibility would be to clone the project and re-map the new datasource. Unfortunately, we don’t have this steps (re-mapping) documented yet. I have requested internally to have them soon in our Community/Documentation and I will be happy to let you know once we have this steps.
s
Thank you! Please keep us posted. Is there a ticket we can track? We can also create a zendesk.
j
Hi, it will take some time to document everything related (and maybe even deliver changes into our platform to improve UX). Please, give me few hours to prepare a script demonstrating how to achieve what you need. Meanwhile, can you investigate on your side if you plan to change the case of table/column names because the migration to Snowflake, which provides upper-case by default(while all other databases provide lower-case)?
OK, here is a script re-mapping LDM to different data source:
Copy code
GOODDATA_LAYOUTS_DIR = Path("gooddata_layouts")
GD_HOST = os.getenv("GOODDATA_HOST", "<http://localhost:3000>")
GD_TOKEN = os.getenv("GOODDATA_TOKEN", "YWRtaW46Ym9vdHN0cmFwOmFkbWluMTIz")
WORKSPACE_ID = "cicd_demo_development"
DS_MAPPING = {"cicd_staging": "cicd_dev"}

# Change data source mapped to LDM
sdk = GoodDataSdk.create(host_=GD_HOST, token_=GD_TOKEN)
ldm = sdk.catalog_workspace_content.get_declarative_ldm(WORKSPACE_ID)
ldm.modify_mapped_data_source(DS_MAPPING)
sdk.catalog_workspace_content.put_declarative_ldm(WORKSPACE_ID, ldm)
and here is a script changing the case of table/column names in LDM:
Copy code
# When migrating from/to Snowflake, the case of DB object names can be changed
# Snowflake default is upper-case, while it is lower-case in all other databases we support
# Alternatively, you can force lower-case in Snowflake by enclosing all object names in DDL statements
# GoodData LDM is mapped to PDM tables/columns, which case can be changed
# This script updates case of all relevant objects
FORCE_UPPER = True
FORCE_LOWER = False

def change_case(object_name: str) -> str:
    if object_name:
        if FORCE_LOWER:
            return object_name.lower()
        else:
            return object_name.upper()

if FORCE_LOWER or FORCE_UPPER:
    ldm = sdk.catalog_workspace_content.get_declarative_ldm(WORKSPACE_ID)
    for dataset in ldm.ldm.datasets:
        dataset.data_source_table_id.id = change_case(dataset.data_source_table_id.id)
        for attribute in dataset.attributes:
            attribute.source_column = change_case(attribute.source_column)
            attribute.sort_column = change_case(attribute.sort_column)
            for label in attribute.labels:
                label.source_column = change_case(label.source_column)
        for fact in dataset.facts:
            fact.source_column = change_case(fact.source_column)
        for reference in dataset.references:
            new_columns = []
            for reference_column in reference.source_columns:
                new_columns.append(change_case(reference_column))
            reference.source_columns = new_columns

    sdk.catalog_workspace_content.put_declarative_ldm(WORKSPACE_ID, ldm)
We will incorporate the second script into the Python SDK, so it is accessible by calling a single function.
I just finished implementation of related functionality in Python SDK. Newly, you can clone workspace in the following simple way:
Copy code
data_source_mapping = { "MyPostgres": "MySnowflake" }
sdk.catalog_workspace.clone_workspace("source_workspace_id", data_source_mapping=data_source_mapping, upper_case=True)
# If you want to set custom target workspace ID/name:
sdk.catalog_workspace.clone_workspace("source_workspace_id", "target_workspace_id", "target_workspace_name", data_source_mapping=data_source_mapping, upper_case=True)
# If you want to rewrite the target (already existing) workspace:
sdk.catalog_workspace.clone_workspace("source_workspace_id", "target_workspace_id", "target_workspace_name", data_source_mapping=data_source_mapping, upper_case=True, overwrite_existing=True)
Currently, you can utllize it if you install the Python SDK from the Github repository (master branch). It should be released to PyPi circa in the beginning of March.
Open in Slack
PreviousNext
Powered by