Data import workflow
2 minute read
A reliable import is iterative: prepare one dependency, search existing shared records, send a small batch, recover generated IDs, reconcile them with the source table, and validate the result before importing dependent objects.
Prepare and order dependencies
Define the destination Project and Dataset, confirm permissions, preserve an
immutable source copy, and add a unique local key such as source_row_id.
Normalize encoding, dates, missing values, decimals, and column names. Search
shared People, References, Taxons, Locations, and Traits before creating them,
and check the POST API. Keep both the source key and ODB
ID or UUID.
| Stage | Prepare or find | Used later by |
|---|---|---|
| 1 | People and Bibliographic References | collection, identification, measurements, Taxons, Datasets |
| 2 | Taxons | identifications, measurements, vernacular names |
| 3 | Locations | Individuals, measurements, spatial validation |
| 4 | Traits, units, categories | measurements and forms |
| 5 | Project and Dataset | Individuals, Vouchers, measurements, media |
| 6 | Individuals and occurrences | Vouchers, identifications, measurements, media |
| 7 | Vouchers and identification history | measurements, media, references |
| 8 | Measurements, Media, vernacular names | final dataset |
Validate coordinates first
POST locations-validation accepts decimal latitude and longitude and
reports registered Locations containing each point. Use it before creating
Individuals or automatic point Locations to find swapped axes, wrong signs,
unexpected administrative areas or protected areas, duplicates, and unsuitable
precision. It reports relationships with existing Locations; it does not decide
whether a coordinate is scientifically correct.
The UserJob cycle
- Submit representative pilot rows and save the UserJob ID.
- Follow status, progress, and logs; do not resubmit while processing.
- Review every row’s status, ID, recognition fields, errors, and warnings.
- Join results back to
source_row_id; never rely on position after sorting. - GET created or reused records and compare essential fields and relationships.
- Correct and resubmit only pending rows, preserving every job ID.
- Use validated IDs in the next dependency stage.
A successful job may still contain warnings or reused records. A useful working
table keeps source_row_id, odb_status, odb_id, odb_uuid, odb_error, and
odb_warning. An import is complete only when every row is documented, warnings
reviewed, IDs reconciled, server records checked, counts and permissions
verified, and source files and UserJob results preserved.
Continue with the R import tutorials.