I have found out that first round of data scrape/ingestion is pretty easy.

The "black box" (for me) represents the next update. How can I avoid having to download all the data again and just use the new additions?

Do you have a plan for updates?