To GoodData homepage

Replies posted by Simonbee

1 Reply

S
SimonbeeNew Participant
posted in Data sources

Loading Apache Spark CSV data from S3 with ADD

Hi Boris, Thanks for the quick response. The files I want to read are written by Apache Spark using the Scala API described here:https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/DataFrameWriter.htmlTypically Spark breaks a CSV file into multiple parts and saves them in a folder with the name passed to the save() call. The number of files written depends on the number of partitions in which Spark stores the data, or in the case of an “append” write, a new CSV file is written for each append.Reading the Spark docs, I can see that this is an underlying Hadoop naming convention and Spark does not allow it to be changed at the point of writing. There are some workarounds available such as this one. I’ll proceed along these lines for now.Simon

2
S
3 years ago
Welcome to the GoodData Community!Cookie settings