Hi GD support team, hope everyone is doing well 😄 Im currently trying to use a FlexConnect as a data source. However, I noticed that if it returns a size of ~200k rows (with say about 10-15 columns), it would give this error

Reached limit of max size of data returned from data source. The limit is 20971520 bytes

This can occur even if I select 1 column to be placed under

Rows

on GD (unless I should go change my code to return only for that column). Then again, even if so, this can arise if the number of rows is large. Is there any way to resolve this? I didnt run into this issue when I was using CSV files as data source. Perhaps does GD support pagination or returning data in batches? (actually I've tried the latter, by changing from PyArrow Data and turned it into batches prior to returning, but still run into this issue). Any assistance will be much appreciated! 🙏

Hi, The limit for size of dataset returned by FlexConnect in GD Cloud is indeed 20MB while for CSV dataset the current limit is 200MB (1TB for all the files in total). We may consider to increase the limit to 200MB as for CSV files. Would it be enough for your use case? One way to mitigate the problem is to analyze

columns

in an execution context of FlexConnect table function and return only columns which are needed for the SQL query which is executed on top of returned data. You can return only a single column even if the table function is declared with 100 columns, but visualization displays just one. This can help to reduce the size in case of wide dataset with many columns.

Hello Jakub, thank you for your reply. Yes is it possible to also increase the limit of the returned payload? I see, so there's an additional SQL layer being done on the returned data. I was suspecting if I could do that, because I noticed that I was only requesting for 1 column via GD, but I could return the entire table and it worked. Ok I will also add this optimization piece

The limit can be currently increased by configuration change only in GoodData CN and not on shared clusters of GD Cloud. Would 200MB satisfy your needs?

Yes, I think yes, let's try with 200 MB for now. Understood and thanks for the example on the above, I will try to add these optimizations in the code

What is the use case? We have 180 seconds timeout which should be enough for interactive applications. The limit is there to prevent overloading of underlying databases by bad queries which do not provide data in time.

Hmm it's at 180s? I am currently setting the error show up within 25-30s. When I query off a smaller table then it wouldn't show the error. But yes I do understand the intention. I'll investigate further and provide evidence on this

Hi Alson, at this time we have no ETA on when this change will be made. However, I will update our internal ticket with our Engineers now to see if I can get more details. Thanks for bearing with us in the meantime.

Thanks Michael, I hope that this change wouldnt be too big and wouldnt trouble you guys too much. But I think it's kinda important, I have some data that Im unable to show right now, even though I've just performed the optimizations suggested by Jakub Is there a way for me to keep track of this ticket or change?

Hi Alson, I hope it's OK for me to step in here. I am afraid that the ticket is internal and cannot be viewed on your side. However, as my colleague said, we will keep you posted. Thank you for your understanding.

Noted Moises, thanks for the assistance, looking forward to further updates 😄

Hi Alson, thank you for checking in on this feature. I checked our internal ticket, and while it’s currently on our radar, it hasn’t been prioritised yet as our team is focused on other high-impact initiatives at the moment. If this functionality is important to you, we’d recommend reaching out to your Account Owner to discuss this and who can help push this update for you.

Thanks for your reply and the details, Michael. Sure understood, I will reach out to her, this is quite important for us as we may have columns that have high cardinality, resulting in a larger than 20MB sized data.