Solved

Getting data into GoodData from multiple applications

  • 16 February 2021
  • 2 replies
  • 165 views

Hello. I am looking around to choose an analytic platform for our customers. The use case is that we have severalĀ applications from which we need to take data and use themĀ to build the analytics for the customers. What capabilities GoodData platform has to get the data in? I do not have one data source, but we have several applications running on our servers and in public cloud. Thank you.Ā Tom

icon

Best answer by ZDS 16 February 2021, 15:03

View original

2 replies

Hi Tom,

Ā 

GoodData supports two high-level architecture options:

  1. Ingesting data from an existing data warehouse (e.g. Snowflake, Redshift, BigQuery, S3 files, etc.) where you consolidate data from your sources yourself. You can choose the data warehouse, connectors, and data transformation tools that best suit your needs. The data are distributed to individual customerā€™s workspaces where your customerā€™s users execute their reports, dashboards, etc..Ā 
  2. Ingesting data directly from your data sources, staging them in GoodData data warehouse (called ADS), and distributing it to individual customerā€™s workspaces.Ā 

This documentation topic https://help.gooddata.com/doc/en/building-on-gooddata-platform/data-preparation-and-distributionĀ  provides more details.

In April we are going to launch a container-based deployment of the GoodData platform that youā€™ll be able to deploy to your datacenter (local, public or private cloud) side by side with your data source (again cloud data warehouses like Snowflake and many other databases like Postgres, etc.). Stay tuned.

Ā 

ZDĀ 

@Tom HarrisĀ one thing you may want to consider is scope vs. cost of your project and how tied to a specific vendor you want to be.

For example, letā€™s say you decide to go with ProviderAĀ for your data ingestion of all your various sources.Ā  Ā You setup a data pipeline to perform timed data harvests from each source and youā€™ve got it configured to transform the data so you can perform analytics and dashboarding duties.

2-3Ā years down the line, youā€™ve learned a lot and you realize ProviderA wasnā€™t really the right fit for your evolving needs.Ā  Now you need to find someone else.Ā  Ā Because ProviderA also did all the data ingestion andĀ transformation, youā€™re going to have to spend a lot of time redoing that data pipeline.Ā  Depending on how well that project was documented and managed, you may have to spend the same amount of time (again) understanding how you need to transform the data from those various sources so it can work together.

Alternatively, you could consider managing the data pipeline separately and manage your own data warehouse; which gives you a little more flexibility with whatā€™s on the market today and may make it a little easier to implement the analytics and visualization aspect, but that will force some design thought around data security andĀ data literacy by the managing group(s).

Thereā€™s advantages and disadvantages to both approaches and a poor implementation plan will cost you more in the end no matter what you do.Ā  If youā€™re looking around today, I hope that youā€™ve clearly defined what ā€œsuccessā€ looks like for your project and that youā€™ve got a general exit strategy on what to do if ā€œsuccessā€ is not met.

Ā 

Ā 

Reply