Hey All, Got a data refresh question for you all and not sure if we're thinking about it in the best way • We used to sync our data 1x a day and when we completed we would purge the data source cache through the gooddata API • Worked ok, obviously slow Now our issue is we sync a lot more frequently, but we are now in a situation where if customers are using the data source while refreshing, they could be getting partially completed data • the cache helps with performance, but doesn't help for data not in the cache We've got 2 thoughts on how to approach this and curious on what others might be doing • pre-fill the gooddata cache ◦ could be hard to put all data possible in the cache • data source iterations ◦ every pipeline run creates a new data set schema ◦ then we update the gd datasource to point to the new schema

You’re running into the classic “data freshness vs. consistency” vs. “performance” triangle. Based on the docs and community answers, here’s what is clearly supported and recommended, and where the gaps are. 1. How GoodData cache behaves (important for your design) • GoodData’s FlexCache stores both raw and final query results; when a result is in cache, it’s returned immediately; otherwise GoodData hits your DB. [FlexCache] • Caches *have no TTL*; they are evicted by LRU only. They are not auto-refreshed on a schedule. [FlexCache; Community cache TTL] • To ensure new data is used, you must invalidate the data source cache (uploadNotification / Clear cache). [Reload cached data] • After invalidation, the next execution of a visualization or AFM query recomputes from the DB and repopulates cache. [Reload cached data; AFM warm-up discussion] So: if you invalidate cache during your ETL, any query that misses cache will hit a partially loaded DB and see inconsistent data. ------------------------- 2. Best-practice pattern from docs & community The documented / recommended pattern is: 1. Run your ETL/ELT to completion in the source DB. 2. Only after the load is fully finished and consistent, call:

POST /api/v1/actions/dataSources/{dataSourceId}/uploadNotification

or use the UI “Clear cache” / Python SDK. [Reload cached data] 3. Let the first post-refresh executions repopulate FlexCache. This is explicitly called out as the expectation: “Whenever an ELT finishes, you need to perform cache invalidation… It is expected to invalidate data source once backing DB data changes are finished so that reports are consistent.” [FlexCache; Snowflake cache discussion] That pattern avoids partially completed data, but you’re right: during the ETL window, any cache miss will still see “old” data until you invalidate. ------------------------- 3. Your two ideas vs. what’s documented A. Pre-fill (warm) the GoodData cache This is supported and aligns with docs: • After invalidation, you can warm FlexCache by executing AFM queries that match your key insights/dashboards (e.g., via

POST /execution/afm/execute

). [AFM warm-up discussion; FlexCache final results; Slack AFM warm-up] • There is no documented API to “execute insight by ID” directly for cache warm-up; you must either: ◦ Use GoodData.UI components/hooks (which execute the insight and thus warm cache), or ◦ Manually construct AFM queries that mirror your important insights. [Slack AFM warm-up] Limitations (from sources): • You’re correct: you *cannot realistically pre-fill all possible combinations*; you typically warm only the most-used dashboards/filters. • Docs do not describe any built-in “global prefill” mechanism; it’s all “execute queries to fill cache”. So: pre-fill is a documented, supported pattern, but you must choose a subset of queries to warm. B. “Data source iterations” (new schema per run, then repoint) The knowledge sources do not describe or endorse a pattern where each pipeline run creates a new schema and you repoint the GoodData data source to it. What is documented: • Changing the data source definition (including URL/connection) is one of the operations that triggers new queries / invalidates cache. [Snowflake cache discussion] • But there is no explicit guidance on using rotating schemas as a consistency strategy, nor any examples of that pattern. So I cannot confirm from the docs that “schema-per-run + repoint” is a recommended or supported best practice; it may work, but it’s not covered in the provided materials. ------------------------- 4. What others are doing (per community threads) From the community discussions you have: • Many customers: ◦ Run ETL on a schedule. ◦ After ETL completes, call

uploadNotification

(or Clear cache) to invalidate. [Reload cached data; Community best practice] • Some then warm the cache by executing key queries (AFM or via UI) so that end users don’t pay the full cost on first load. [Slack AFM warm-up] • Support explicitly confirms: ◦ There is no automatic data source cache refresh. ◦ Caches are valid until you invalidate (or change LDM, metrics, data source definition). [Community cache TTL; Snowflake cache discussion] ------------------------- 5. Practical pattern you can safely adopt (within what’s documented) Within the limits of the docs, a “safe” approach looks like: 1. Run ETL into a consistent state in your DB (however you implement staging/atomicity is outside GoodData’s scope; the sources don’t cover that). 2. Only after ETL is fully done, call:

POST /api/v1/actions/dataSources/{dataSourceId}/uploadNotification

or equivalent. [Reload cached data] 3. Optionally, warm FlexCache by: ◦ Executing AFM queries for your most-used dashboards/filters via

POST /execution/afm/execute

, or ◦ Programmatically loading those dashboards/insights via GoodData.UI in a headless/scripted context. [Slack AFM warm-up] 4. Let users read from cache between refreshes; they will always see a fully consistent snapshot (pre- or post-refresh), never a partial load, as long as you: ◦ Don’t invalidate cache until the DB is consistent. ◦ Don’t change the data source definition mid-load. If you need near real-time and can tolerate more load on your DB, you can also consider DirectQuery (bypasses cache), but that’s a separate trade-off and is only briefly mentioned in the docs. [Cache management overview] ------------------------- 6. Summary vs. your two options • Pre-fill cache: Supported and aligns with docs; you warm key queries after each

uploadNotification

. You’re right that you can’t cover every possible query, but this is the documented way to combine freshness + performance. • Data source iterations (new schema per run): Not described in the provided documentation; I cannot confirm it as a recommended pattern. The only documented “switch” is invalidating cache or changing the data source definition, not rotating schemas per run. If you share more about how your ETL ensures atomicity (e.g., staging + swap), you can likely keep your current single-schema approach and just tighten the timing of

uploadNotification

+ optional cache warm-up. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups. 👀 Our Support team monitors all messages and will jump in if needed.

Hi Steve, we have reviewed the response from our AI Assistant - But please feel free to follow-up if you need any further assistance

Today we don't have atomicity until the process completes. So if we do need/want to keep data as accurate all the time, even as we process data more efficiently -> we would almost always be in a state of flux. The cache could help, but its no different than today

To avoid partially loaded data, GoodData needs to read from a DB state that is already consistent. The platform itself doesn’t provide atomicity during your load; that has to be implemented on the DB side. A common pattern is: 1. Load into staging tables or a staging schema until the refresh is fully complete and validated. 2. Perform an atomic switch so that GoodData always reads from a stable snapshot (for example, swap/rename tables, or have stable views/synonyms that you repoint to the new tables/schema). 3. Only after this switch, call

POST /api/v1/actions/dataSources/{dataSourceId}/uploadNotification

so GoodData invalidates FlexCache and recomputes results against the new, complete snapshot. [*Reload cached data*; FlexCache] With this setup, users always see either the previous complete snapshot or the new complete snapshot—never an in‑between state. Cache warming after invalidation (via AFM or by programmatically loading key dashboards) can then be used to improve performance for the most-used queries, but it won’t by itself solve partial reads unless the DB-side publish step is atomic. [Slack AFM warm-up]

@Steve Fox have you considered a blue/green swap in the target schema? That approach has been working well for us.

Thats what I want to do@Marek Horvát . Mind me asking how you do that today?

@Steve Fox here is how our approach works: • First, our data transformation jobs create temporary versions of the final tables and populate them with new data. • Once the load completes, we drop the old tables and rename the temporary ones. • Finally, we refresh the cache on the GD Cloud side. Depending on your use case, you might want to implement a more atomic approach - such as triggering the cache refresh after each individual table is swapped. Hope this makes sense!

Interesting thank you!