# Best practice for large csv imports? **URL:** \ **Category:** Ask the community\ **Created:** [March 25, 2024, 8:50am UTC](https://community.appfarm.io/t/best-practice-for-large-csv-imports/1132 "2024-03-25T08:50:10Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![matt](https://dub1.discourse-cdn.com/flex013/user_avatar/community.appfarm.io/matt/32/188_2.png) [@matt](https://community.appfarm.io/u/matt)\ **Post date:** [March 25, 2024, 8:50am UTC](https://community.appfarm.io/t/best-practice-for-large-csv-imports/1132/1 "2024-03-25T08:50:10Z")
I have a 140MB csv file that I’d like to import, and it seems to import into frontend memory fine but fails to persist that data to the server afterwards. ![image](https://europe1.discourse-cdn.com/flex013/uploads/appfarm/original/2X/d/d29d6b355f78bf2bcdb3b53411ff28da487ea05c.png) I didn’t see file size limitations in the docs, but I might have missed it. Do you have a recommended best practice for dealing with large imports like this? I could do it with GraphQL, but would prefer to make it easier for less technical users to also perform the same task. This is the action being run, and it works for smaller data sets without any problems. ![image](https://europe1.discourse-cdn.com/flex013/uploads/appfarm/original/2X/0/0041868d13fd09f9e8e9e86196162cd4991fa3b0.png) ---
**Author:** ![ErikAKSkallevold](https://dub1.discourse-cdn.com/flex013/user_avatar/community.appfarm.io/erikakskallevold/32/1378_2.png) [@ErikAKSkallevold](https://community.appfarm.io/u/ErikAKSkallevold)\ **Post date:** [March 25, 2024, 9:19am UTC](https://community.appfarm.io/t/best-practice-for-large-csv-imports/1132/2 "2024-03-25T09:19:52Z")
Hello Matt, When I need to persist large amounts of data, I like to do it in chunks of 1000 objects at a time. You can achieve this by using a while-loop and two counters stored in App Variables. On the first iteration you set` Counter Start` to 0 and `Counter End` to 1000. Then select all objects where `Index >= Counter Start && Index < Counter End.` This will select object 0-999. For the next iteration, increment both counters by 1000. The Select will now select objects 1000-1999. Again, persist only the selected objects. Do this until you have reached the end of the array. When the end is reached, break out of the loop. ![bilde](https://europe1.discourse-cdn.com/flex013/uploads/appfarm/original/2X/2/2ef440221de92f8e6d6f8bdb669c9153d9c5e6f1.png) ![bilde](https://europe1.discourse-cdn.com/flex013/uploads/appfarm/original/2X/4/4dbdaf5dbbb5143a5ea8fa884e9919c2adb225b5.png) Hope this helps 🙂 //Erik ---
**Author:** ![matt](https://dub1.discourse-cdn.com/flex013/user_avatar/community.appfarm.io/matt/32/188_2.png) [@matt](https://community.appfarm.io/u/matt)\ **Post date:** [March 25, 2024, 11:07am UTC](https://community.appfarm.io/t/best-practice-for-large-csv-imports/1132/3 "2024-03-25T11:07:22Z")
Thanks, it helps! But the performance is really horrible. I followed your lead like this: ![image](https://europe1.discourse-cdn.com/flex013/uploads/appfarm/original/2X/1/162b42dd2946d6054fee8909569dbbafccd801d5.png) but each round of persisting 1k objects takes ~10 seconds. So importing this file is going to take at least 90 minutes. I’ll try again with 10k objects at a time and see how much that helps I guess. ![image](https://europe1.discourse-cdn.com/flex013/uploads/appfarm/original/2X/1/15d79d126342ec66a7f9456fe50dd2f3265eda57.jpeg) ---
**Author:** ![ErikAKSkallevold](https://dub1.discourse-cdn.com/flex013/user_avatar/community.appfarm.io/erikakskallevold/32/1378_2.png) [@ErikAKSkallevold](https://community.appfarm.io/u/ErikAKSkallevold)\ **Post date:** [March 25, 2024, 2:34pm UTC](https://community.appfarm.io/t/best-practice-for-large-csv-imports/1132/4 "2024-03-25T14:34:26Z")
90 minutes is a long time, even for a large CSV-file. Is this import located in a big app? If it is, are there other datasources that are affected when persisting the data? This could explain the poor performance as they would need to recalculate runtime properties and dependencies after each persist. I recommend moving the import job to a service. With this approach you can persist the CSV-file in the app and read it from a service later. The service can then process the CSV-file in the background without the users having to monitor the app. You could also move the import job to a new app with less datasources, but this would require the user to not leave the app while persisting. //Erik ---
**Author:** ![matt](https://dub1.discourse-cdn.com/flex013/user_avatar/community.appfarm.io/matt/32/188_2.png) [@matt](https://community.appfarm.io/u/matt)\ **Post date:** [March 26, 2024, 10:12am UTC](https://community.appfarm.io/t/best-practice-for-large-csv-imports/1132/5 "2024-03-26T10:12:58Z")
This is a new, very small app, and there shouldn’t be any other datasources affected when persisting the data. I’ll look into the service approach a bit and also try importing via GraphQL I guess. I would expect this to work fine using the normal tools though. ☹ ---
**Author:** ![matt](https://dub1.discourse-cdn.com/flex013/user_avatar/community.appfarm.io/matt/32/188_2.png) [@matt](https://community.appfarm.io/u/matt)\ **Post date:** [March 26, 2024, 11:19am UTC](https://community.appfarm.io/t/best-practice-for-large-csv-imports/1132/6 "2024-03-26T11:19:12Z")
Do you have any examples of this kind of setup, where a user uploads a CSV file from the app, which triggers a separate service to persist the data? ---
**Author:** ![ErikAKSkallevold](https://dub1.discourse-cdn.com/flex013/user_avatar/community.appfarm.io/erikakskallevold/32/1378_2.png) [@ErikAKSkallevold](https://community.appfarm.io/u/ErikAKSkallevold)\ **Post date:** [March 26, 2024, 7:29pm UTC](https://community.appfarm.io/t/best-practice-for-large-csv-imports/1132/7 "2024-03-26T19:29:28Z")
Yes, I have a very simple showcase here. It consists of an app where the user can upload a file. When the file has been persisted a service responsible for processing the file is called. It takes the `File Content URL` of the file as an Query Param. Here are some images of the setup: 1. The app-action for uploading the file and calling the service 2. The setup of the Service Endpoint that is responsible for handling the query params. Note that the Query Param is stored in Service Variables. The variable has the type: `Internet URL`. ![bilde](https://europe1.discourse-cdn.com/flex013/uploads/appfarm/original/2X/f/f94e53c6b6fcfd33877b6f099ff0e96ae26fa19b.png) 1. The action that is called ran when the Endpoint is called. The Internet URL is passed as an Action Param and used in the `Import Data`-action node to retrieve the file. You may ofcourse add additional logic to this action. ![bilde](https://europe1.discourse-cdn.com/flex013/uploads/appfarm/original/2X/0/0344ed8ff486909c4255dd73ff8c12ea656c39ec.png) ![Service Action Param. Bound to the Service Variable: Param - URL](https://europe1.discourse-cdn.com/flex013/uploads/appfarm/original/2X/2/2b24033665a6e46968e6d1dd6c6f80405aae451f.png) Since you expect the Service to take a while to process, you should probably enable `Run Async` on the Service Endpoint. This will make it so that the App does not wait for the Service to complete before going to the next action node. Please set the timeout accordingly as the default is 5 minutes. If you’re worried about data races or if the service should not be run concurrently, turn on `No Concurrency` instead. //Erik ---
**Author:** ![matt](https://dub1.discourse-cdn.com/flex013/user_avatar/community.appfarm.io/matt/32/188_2.png) [@matt](https://community.appfarm.io/u/matt)\ **Post date:** [March 27, 2024, 10:35am UTC](https://community.appfarm.io/t/best-practice-for-large-csv-imports/1132/8 "2024-03-27T10:35:22Z")
It looks like you are recasting the URL to a String in your service action parameter - is that necessary instead of using URLs throughout? I _almost_ have this working…I think…but I am getting 403 errors when the service is trying to run. My user _should_ have access to run this service though, so I don’t understand why it’s giving me a 403. Do I have my data model objects setup incorrectly or something, so that the File URL is wrong? I’m using a database connected object’s File Content URL here as shown in the screenshot. ![image](https://europe1.discourse-cdn.com/flex013/uploads/appfarm/original/2X/d/d8b4c289640a2ea7d5b3a3250e8123c1ea5106cb.png) ![image](https://europe1.discourse-cdn.com/flex013/uploads/appfarm/original/2X/8/83c4a3e2817e338a8a035784f2f28da60d63d17c.png) ![image](https://europe1.discourse-cdn.com/flex013/uploads/appfarm/original/2X/0/005bb679daf4f81faabc752ca0d8ede6cca347dd.png) ![image](https://europe1.discourse-cdn.com/flex013/uploads/appfarm/original/2X/a/a0b24bfbbe5bfab3141e95fb0ea7a56c66a43b2d.png) ![image](https://europe1.discourse-cdn.com/flex013/uploads/appfarm/original/2X/e/e144dba6acaaa3364624eb66280ddada6fb5db96.png) ![image](https://europe1.discourse-cdn.com/flex013/uploads/appfarm/original/2X/1/1a637f8a1cf52a057075399462072ac236b0e012.png) ![image](https://europe1.discourse-cdn.com/flex013/uploads/appfarm/original/2X/4/43f327983ec11839106ab9ea641eac11ab4f9415.png) ---
**Author:** ![ErikAKSkallevold](https://dub1.discourse-cdn.com/flex013/user_avatar/community.appfarm.io/erikakskallevold/32/1378_2.png) [@ErikAKSkallevold](https://community.appfarm.io/u/ErikAKSkallevold)\ **Post date:** [March 27, 2024, 12:39pm UTC](https://community.appfarm.io/t/best-practice-for-large-csv-imports/1132/9 "2024-03-27T12:39:35Z")
To others who might have the same issue, the solution to this problem was to use the `File Content URL` of the runtime object and not persisted object when calling the service 🙂 //Erik