Data import workflow
3 minute read
A reliable import is iterative: prepare one dependency, search existing shared records, send a small batch, recover generated IDs, reconcile them with the source table, and validate the result before importing dependent objects.
Prepare and order dependencies
Define the destination Project and Dataset, confirm permissions, preserve an
immutable source copy, and add a unique local key such as source_row_id.
Normalize encoding, dates, missing values, decimals, and column names. Search
shared People, References, Taxons, Locations, and Traits before creating them,
and check the POST API. Keep both the source key and ODB
ID or UUID.
| Stage | Prepare or find | Used later by |
|---|---|---|
| 1 | People and Bibliographic References | collection, identification, measurements, Taxons, Datasets |
| 2 | Taxons | identifications, measurements, vernacular names |
| 3 | Locations | Individuals, measurements, spatial validation |
| 4 | Traits, units, categories | measurements and forms |
| 5 | Project and Dataset | Individuals, Vouchers, measurements, media |
| 6 | Individuals and occurrences | Vouchers, identifications, measurements, media |
| 7 | Vouchers and identification history | measurements, media, references |
| 8 | Measurements, Media, vernacular names | final dataset |
Validate coordinates first
POST locations-validation accepts decimal latitude and longitude and
reports registered Locations containing each point. Use it before creating
Individuals or automatic point Locations to find swapped axes, wrong signs,
unexpected administrative areas or protected areas, duplicates, and unsuitable
precision. It reports relationships with existing Locations; it does not decide
whether a coordinate is scientifically correct.
The UserJob cycle
- Submit representative pilot rows and save the UserJob ID.
- Follow status, progress, and logs; do not resubmit while processing.
- Review every row’s status, ID, recognition fields, errors, and warnings.
- Join results back to
source_row_id; never rely on position after sorting. - GET created or reused records and compare essential fields and relationships.
- Correct and resubmit only pending rows, preserving every job ID.
- Use validated IDs in the next dependency stage.
A successful job may still contain warnings or reused records. A useful working
table keeps source_row_id, odb_status, odb_id, odb_uuid, odb_error, and
odb_warning. An import is complete only when every row is documented, warnings
reviewed, IDs reconciled, server records checked, counts and permissions
verified, and source files and UserJob results preserved.
Continue with the R import tutorials. For an unfamiliar spreadsheet, use the guarded AI-assisted planning prompt and review its proposed workflow.
Importing media ZIPs
The ZIP is saved in private temporary storage and extracted once. The metadata
table matches each media file through filename. Processing runs in chunks
within the same UserJob, preserving progress and incomplete dates.
After a transient failure or cancellation, resume the job while its temporary files remain available: confirmed rows are not imported again. Resuming does not fix invalid metadata; to correct input data, prepare a new batch containing only the required rows. Source files are removed once every row referencing them has been confirmed. Temporary cleanup does not remove imported final media.
Inactive failed or cancelled jobs have configurable retention (7 days in the template). After expiration, upload the outstanding files again. Installation limits and administrator settings are documented in Administrator configuration.