Hi everyone, I'm loading up data to a model via the REST API using these directions: https://help.gooddata.com/doc/enterprise/en/data-integration/data-preparation-and-dis[…]ion/additional-data-load-reference/loading-data-via-rest-api , everything is working fine except how do I load data incrementally? It seems everytime I do a data push, it replaces the previous data automatically? From reading this document here: https://help.gooddata.com/doc/free/en/data-integration/data-preparation-and-distribut[…]/delete-old-data-while-loading-new-data-to-a-dataset-via-api I need to specify that I need to delete old data, but that's not what I want to do and these params aren't anywhere in my manifest. How do I get it to stop replacing data and instead append to it?

Hi Shyam, as mentioned in this doc: https://help.gooddata.com/pages/viewpage.action?pageId=86796461 To upload data incrementally, use

"mode": "INCREMENTAL"

.

@Michael Ullock thanks I did that and now gooddata shows "partial upload" but my dataset is still the same as the last time I uploaded on full, ie new data hasn't been added in it's just the old one that's still there

If you do the FULL load it will load all rows in source data - After this, if you then do INCREMENTAL load after this, it will upload only new and updated records only

@Michael Ullock I tried this, and it doesn't seem to update any of the new data afterwards.. ie it's still got the original 1000 records and no others even when I put new records in. Do I name the file the same, or any other differences?

Is this a cache thing regarding needing to flush the cache after data uploads?

@Paul Muharsky possibly.. how would I go about flushing it?

Sorry, I have no Idea! also new to the platform, just came across several mentions of needing to flush things after data uploads while browsing slack channels...

I saw this: https://www.gooddata.com/developers/cloud-native/doc/1.5/administration/data-sources/notification/ I'll try it to see if it helps

Hi @Shyam Ramani, let me try to help here. I believe there might be several different concepts mixed up here so let me try to explain it. I understand that it might be a bit confusing. First the GoodData edition - there are two major editions which differ (among other things) in how they handle the data loads: • GoodData Platform - in which you physically load the data to its internal database via API (or other tools) • GoodData.CN (Cloud Native) - in which the data is not being loaded anywhere and it works directly on top of the connected supported data source Based on the screenshots and the API you mentioned, you seem to be using the GoodData Platform. So the “Data Source Notification” article for Cloud Native does not apply to you. In fact any article which is in the Cloud Native section does not apply to you. So let’s talk about data loads to the GoodData Platform now. The API you mentioned you are using (https://help.gooddata.com/doc/enterprise/en/data-integration/data-preparation-and-dis[…]ion/additional-data-load-reference/loading-data-via-rest-api) is a low-level API to load data to a specific workspace and it does load the data either fully or incrementally for you, but it does not determine the increment. This low level API does not use the x__timestamp column. If you want to incrementally load only the data which has changed, you should send only the new/modified records in the CSV file. From the screenshot you shared, it seems you are now really loading incrementally (=upsert) but the data size uploaded remained the same 371.7 kB. So I assume you told the system to upload incrementally, but provided ALL the data (not just the increment). As I said this API is NOT using the x__timestamp column. The x__timestamp column (or in case of flat files the naming convention of the file with timestamp in its name) is a feature of another tool for uploading data - the Automated Data Distribution which is higher-level and internally uses the same API, but apart from it, it provides service for automatically handling the increments, load data to multiple workspaces etc. So depending on what you want to achieve and works best for you, I would recommend to either: • keep using the API to upload the data, but for the incremental load make sure you only provide the new and modified records in your CSV file in case of incremental load (and feel free to remove the x__timestamp column as this API is not using it) (make sure you have a key defined in your dataset for incremental load to avoid duplicities) • OR explore the possibilities of the Automated Data distribution and use the x__timestamp or timestamp in the filename to handle the increments

got it, figured it out in the end