Common Import Questions - Importing New Participants and Households

When adding new Participants and Households, how much can I import at once? Are there limits? Guidelines?

What Size Import Files Do you Recommend for Creating New Records?

• Initial test import: 5 records
• First real import: 100 - 250 records.
• If that goes well, scale up to 500 records
• Then try 750.
• If that goes as swiftly and smoothly as you'd like, try 1,000 records. 

For the initial test import, start with just a few records so you can review the validation feedback and make adjustments to your data set. From there, start with 100-250 records at a time. Scale up in increments of 250 to 500 then 750 and then 1,000.  

But How Many Records Can I Actually Import at a Time?

The New Participants, Update by Participant ID, and Update by Record ID imports all allow for importing more than 100 records at a time. Compyle will break up large record sets into batches of 100 to process at a time. 

So More than 100, but how many? Why is this hard to answer?

There is a not a system defined limit to how many records you can import at once, but there are functional limitations to consider.

For instance, consider field validation. Compyle validates the data entries for each of the individual fields and flags any anomalies. An import with 1,000 participants that only fills out 5 fields requires the same amount of validation as an import with 100 participants and 50 fields. The validation for the values is one part of creating participants via import, but there are other processes as well, such as duplicate check evaluations, and making sure the file is formatted appropriately. 

In general, the upward limit of how many records you can create through import will depend on many factors, including:

• the available environment resources, which vary based on how many users are actively working at any given time and what they are doing
• how many different fields you are populating with the import
• the upload speed for your internet connection
• the resources available on your device, such as available working memory, etc
• how many different fields/file columns you are adding data to and what types of fields they are
• how clean the information is/how well the data values align with field options and types
• whether or not you are using duplicate check, which in turn varies depending on how many entity records are already in your site
• whether or not the new records are being assigned to a group other than the upload group