Scheduled Service to Update large datasets

Hi,
What are the implications of running a service with many actions on a daily schedule vs splitting them up into several services?

In my case I need to run a service that creates an employee object in our DB by getting info from several different endpoints and storing the info in one object. Unfortunately, we are not provided with a modified or created data as fields from the endpoint, so we need to read through a lot of data in the service.

My specific issue atm is that I get this error although the service keep on running in the background. However the last actions are running very slow, so the setup is not sufficient.

Is the clue to split up into several different schedules that run with some time in between on a schedule? Other tips also appreciated :slight_smile:

Hi!

We recommend breaking it down into smaller Actions for different functionalities or integrations. This way you can use “Run other action”. You can also consider using separate services where appropriate and use “Run Service” to enable communication between them.

In general, we recommend against running a long action with web requests, iterations and CRUD operations without any form of exception handling. You can read more here about our recommended approach for setting up an Event Log.

Since we don’t have detailed insight into your service, it’s difficult to recommend specific improvements, but we hope this helps guide you in the right direction.

Ok. In my case where i need to update a large dataset with several web-requests I still run into the issue I described above with the service not getting a response in time. This happens also when i split the service up with several actions and call another action.

Is there anything I can do for the service to not error out. As you see in the background here, the service is still running it seems. But for me to check and validate the data sett i need to get rid of the popup, and the only way to get rid of the popup is to reload the page. If I do that I loose all the runtime data…

I have experienced this issue with the debugger. If possible I would try to limit the dataset when testing with debugger to check the base service functionality. When base functionality is working I would set up temporary schedules to do one large scale test comparing datasets before and after run without the debugger.

Thank you Sondre. Are you experiencing this issue only when working with the debugger. Meaning. If you test on a small dataset using the debugger tool, then schedule the service one the real (larger) dataset, it will work fine, in your experience?

I’ve created a short loom here describing my issue. Hopefully I can get some help fixing this issue / understanding how the service is working “under the hood”

No problem! Yes the timeout is in my experience only using debugger. It seems the our connection to the server times out, and i have experienced the service running further after the timout, but not allways to the end.

Summarized the debugger is inconsistent on large datasets.

And i have never had the same issue with scheduled test runs no. Both Test and Staging is the same in this regard. Debugger dodgy, scheduled good :slight_smile:

I agree with Sondre’s considerations here. The service debugger client tends to time out on large datasets. Can you try to

  1. Add a minor Sleep inside the iteration (or after the web request)
  2. Test executing the same service from /api/services/ (the swagger tool) to see if you get the same ping timeout there?