Delta load using Data Extract API

Ask the community
1

Hello, I just tested the Data Extract API that seems very useful for reporting purpose.
I have one question about the most appropriate way of using this feature.
I have a 5 m objects object class that I want to use in my reporting. Data will be stored in a DW.
This 5m rows table is only containing data can not be modified or deleted once created.

Today, to transfert the data from Appfarm to the DW, i’m using graphQL and filter on the af_createdDate.

I really like how hast and how easy is it with the Data Extract API, but how can it be possible to know the cursor if I want to load only the data that was created from

Example:
Today, I’m loading the entire data set: no problem with the Data Extract API
Tomorrow: I only want to add the data created during the day: what’s the best way to achieve this ? reload the entire data set ?

2

Hi!

Have you tried using the ID of the last returned object as the cursor in your next request? In theory, the object IDs are automatically increasing, meaning that if you use the ID of the last object from yesterday in your request today, you should only receive objects created since your last request. The data is not necessarily ordered on createdDate, but all the object should be created after your previous cursor.

I hope that works; if not, please let us know.

3

I was thinking about it, but when using the last ID (last of day one) as the start cursor for the next day (day two) I’ve this specific record loaded twice.
But I can also handle it by removing the duplicates