In Sitefinity, several instances require the need to import large amounts of data, such as moving the website from another CMS to Sitefinity, or importing data into Sitefinity from another source. Importing thousands or millions of pieces of data is always time consuming and there is a possibility of a timeout. Consider processing each item that takes a lot of time or each item that is process intensive, like adding an image or document or a video to a dynamic item.
Avoiding Timeouts: Best Practices for Large Data Imports in Sitefinity
The idea is to loop through each record quickly, delaying the actual processing of each item, and leave it to Sitefinity’s Scheduled Task to handle the heavy processing. Each time the scheduled task is called, there is an entry made in the database in the table sf_scheduled_task. While querying this table, it will be interesting to see entries for scheduled tasks used by Sitefinity itself internally, like a scheduled task for synchronization, sitemap generation, item deletion, etc.
This particular Sitefinity CMS tutorial is very helpful: http://docs.sitefinity.com/for-developers-publishing-system/tutorial-schedule-a-timed-task-to-upload-content.
In order to do that we need pass custom data to the scheduled task. In the following code snippet, item title and file path of the pdf file that should be attached as a related media file for the dynamic module item is passed on as custom data. This custom data is set as a string separated by a tab.
In the example below, for simplicity and code reusability, we use comma separated data format (CSV) for the data to be imported. With that, we can use TextFieldParser to read the records and for each record, we call the scheduled task to take care of the time consuming operation like extracting text from a pdf file and attaching pdf file as a related media to the dynamic item. Doing this will reduce the time to loop through all the records. In this case, we reuse the scheduled task by passing custom data, just like we pass variables to a method.
It is helpful to query the sf_scheduled_task table to inspect the status of each task. There are two columns in the table – status and status_message, which give information about the task status and error messages, if any. Note that a task which successfully completes is deleted from the table, so only those errored out and those that are scheduled to run remain on the table. There is another useful column called progress. Note that the custom data we pass on is stored in task_data. There are also timestamp columns like execute_time. Understanding this table will provide valuable information about the data that is being imported. We can inspect for those with errors and we can reschedule to import that data again. We can also build a dashboard page that queries this table, visually shows the progress, and lists the data waiting to be executed.
