It seems like the parallell excecution of for loop is great for transforming data from one object class to another quickly. For instance from Contract to Orders. As we are considering optimizing some previously implementet sequential for loops it would be nice to know where it limitations are on “sideways” scalability.
Does someone have experience with this? Say 500 objects in parallell to 500 others.
What if you run 500 Contracts in parallell to 500 Orders and for each contract 5 Contract Lines to 5 Order Lines also in parallell.
Ps. parallell works great on small scale object creation in app
If I understand you correctly, I would say that we’ve been implementing the parallell execution of for-loops kind of what you’re describing.
The limitations, as far as we’ve observed, comes when trying to transform data from 2 data sources (order and order lines) to another two (contract and contract lines) and running these in two nested for-loops, both in parallell execution. We experienced some performance issues when running this setup, and chose to change the setup to instead run the inner for-loop in parallell execution and the outer for-loop sequentially. After some trial and error, we found a sweet spot for this setup that limits it to a maximum of 100 executions in parallell. Essentially iterating sequentially over a maximum of 50 orders, doing some transformations on contracts, and for each order iterate in parallell over a maximum of 100 order lines and making some transformations on the relevant contract lines.
Another operation we’re running uses nested for-loop, both running with parallell execution. However, this action are only creating new data and are not performing any heavy data transformations. The action are running at most 15 iterations in parallell where each of these executions are running, at most, 10 iterations in parallell. If I understand the concept of parallell executions correctly, this means we’re able to run up to 150 executions simultaneously with almost none failures.
We’re running both of these operations on Services, and I belive the limitations might vary depending on the number of resources allocated for your solution.
