Getting Started

• 1: First time users
• 2: Apache Installation
• 3: Docker Installation
• 4: Customize Installation

1 - First time users

OpenDataBio is software to be used online. Local installations are for testing or development, although it could be used for a single-user production localhost environment.

User roles

• If you are installing, the first login to an OpenDataBio installation must be done with the default super-admin user: admin@example.org and password1. These settings should be changed or the installation will be open to anyone reading the docs;
• Self-registrations only grant access to datasets with privacy set to registered users and allows user do download data of open-access, but do not allow the user to edit nor add data;
• Only full users can contribute with data.
• But only super admin can grant full-user role to registered users - different OpenDataBio installations may have different policies as to how you may gain full-user access. Here is not the place to find that info.

See also User Model.

Prep your full-user account

1. Register yourself as Person and assign it as your user default person, creating a link between your user and yourself as collector.
2. You need at least a dataset to enter your own data
3. When becoming a full-user, a restricted-access Dataset and Project will be automatically created for you (your Workspaces). You may modify these entities to fit your personal needs.
4. You may create as many Projects and Datasets as needed. So, understand how they work and which data they control access to.

Entering data

There three main ways to import data into OpenDataBio:

1. One by one through the web Interface
2. Using the OpenDataBio POST API services:
   1. importing from a spreadsheet file (CSV, XLXS or ODS) using the web Interface
   2. using the OpenDataBio R package client
3. When using the OpenDataBio API services you must prep your data or file to import according to the field options of the POST verb for the specific ’endpoint’ your are trying to import.

Tips for entering data

1. If first time entering data, you should use the web interface and create at least one record for each model needed to fit your needs. Then play with the privacy settings of your Workspace Dataset, and check whether you can access the data when logged in and when not logged in.
2. Use Dataset for a self-contained set of data that should be distributed as a group. Datasets are dynamic publications, have author, data, and title.
3. Although ODB attempt to minimize redundancy, giving users flexibility comes with a cost, and some definitions, like that of Traits or Persons may receive duplicated entries. So, care must be taken when creating such records. Administrators may create a ‘code of conduct’ for the users of an ODB installation to minimize such redundancy.
4. Follow an order for importation of new data, starting from the libraries of common use. For example, you should first register Locations, Taxons, Persons, Traits and any other common library before importing Individuals or Measurements
5. There is no need to import POINT locations before importing Individuals because ODB creates the location for you when you inform latitude and longitude, and will detect for you to which parent location your individual belongs to. However, if you want to validate your points (understand where such point location will placed), you may use the Location API with querytype parameter specified for this.
6. There are different ways to create PLOT and TRANSECT locations - see here Locations if that is your case
7. Creating Taxons require only the specification of a name - ODB will search nomenclature services for you, find the name, metadata and parents and import all of the them if needed. If you are importing published names, just inform this single attribute. Else, if the name is unpublished, you need to inform additional fields. So, separate the batch importation of published and unpublished names into two sets.
8. The notes field of any model is for both plain text or JSON object string formatted data. The Json option allows you to store custom structured data any model having the notes field. You may, for example, store as notes some secondary fields from original sources when importing data, but may store any additional data that is not provided by the ODB database structure. Such data will not be validate by ODB and standardization of both tags and values depends on you. Json notes will be imported and exported as a JSON string, and will be presented in the interface as a formatted table; URLs in your Json will be presented as links.

2 - Apache Installation

These instructions are for an apache-based installation, but can be easily tuned to work with nginx.

Server requirements

1. The minimum supported PHP version is 8.0
2. The web server may be apache or nginx. For nginx, check configuration in the docker files to tune these instructions, which are for apache.
3. It requires a SQL database, MySQL and MariaDB have been tested, but may also work with Postgres. Tested with MYSQL.v8 and MariaDB.v15.1.
4. PHP extensions required ‘openssl’, ‘pdo’, ‘pdo_mysql’, ‘mbstring’, ’tokenizer’, ‘xlm’, ‘dom’, ‘gd’, ’exif’, ‘bcmath’, ‘zip’
5. Pandoc is used to translate LaTeX code used in the bibliographic references. It is not necessary for the installation, but it is suggested for a better user experience.
6. Requires Supervisor, which is needed background jobs

Create Dedicated User

The recommended way to install OpenDataBio for production is using a dedicated system user. In this instructions this user is odbserver.

Download OpenDataBio

Login as your Dedicated User and download or clone this software to where you want to install it. Here we assume this is /home/odbserver/opendatabio so that the installation files will reside in this directory. If this is not your path, change below whenever it applies.

Prep the Server

First, install the prerequisite software: Apache, MySQL, PHP, Pandoc and Supervisor. On a Debian system, you need to install some PHP extensions as well and enable them:

sudo apt-get install software-properties-common
sudo add-apt-repository ppa:ondrej/php
sudo add-apt-repository ppa:ondrej/php ppa:ondrej/apache2
sudo add-apt-repository ppa:ondrej/php
sudo add-apt-repository ppa:ondrej/apache2

sudo apt-get install mysql-server php8.0 libapache2-mod-php8.0 php8.0-intl \
 php8.0-mysql php8.0-sqlite3 php8.0-gd php8.0-mysql php8.0-cli pandoc \
 php8.0-mbstring php8.0-xml php8.0-gd php8.0-bcmath php8.0-zip php8.0-curl \
 supervisor

sudo a2enmod php8.0
sudo phpenmod mbstring
sudo phpenmod xml
sudo phpenmod dom
sudo phpenmod gd
sudo a2enmod rewrite
sudo a2ensite
sudo systemctl restart apache2.service



#To check if they are installed:
php -m | grep -E 'mbstring|cli|xml|gd|mysql|pandoc|supervisord|bcmath|pcntl|zip'

Add the following to your Apache configuration.

• Change /home/odbserver/opendatabio to your path (the files must be accessible by apache)
• You may create a new file in the sites-available folder: /etc/apache2/sites-available/opendatabio.conf and place the following code in it.

touch /etc/apache2/sites-available/opendatabio.conf
echo '<IfModule alias_module>
        Alias /opendatabio      /home/odbserver/opendatabio/public/
        Alias /fonts /home/odbserver/opendatabio/public/fonts
        Alias /images /home/odbserver/opendatabio/public/images
        <Directory "/home/odbserver/opendatabio/public">
                Require all granted
                AllowOverride All
        </Directory>
</IfModule>' > /etc/apache2/sites-available/opendatabio.conf

This will cause Apache to redirect all requests for / to the correct folder, and also allow the provided .htaccess file to handle the rewrite rules, so that the URLs will be pretty. If you would like to access the file when pointing the browser to the server root, add the following directive as well:

RedirectMatch ^/$ /

Configure your php.ini file. The installer may complain about missing PHP extensions, so remember to activate them in both the cli (/etc/php/8.0/cli/php.ini) and the web ini (/etc/php/8.0/fpm/php.ini) files for PHP!

Update the values for the following variables:

Find files:
php -i | grep 'Configuration File'

Change in them:
	memory_limit should be at least 512M
	post_max_size should be at least 30M
	upload_max_filesize should be at least 30M

Something like:

[PHP]
allow_url_fopen=1
memory_limit = 512M

post_max_size = 100M
upload_max_filesize = 100M

Enable the Apache modules ‘mod_rewrite’ and ‘mod_alias’ and restart your Server:

sudo a2enmod rewrite
sudo a2ensite
sudo systemctl restart apache2.service

Mysql Charset and Collation

1. You should add the following to your configuration file (mariadb.cnf or my.cnf), i.e. the Charset and Collation you choose for your installation must match that in the ‘config/database.php’

[mysqld]
character-set-client-handshake = FALSE  #without this, there is no effect of the init_connect
collation-server      = utf8mb4_unicode_ci
init-connect          = "SET NAMES utf8mb4 COLLATE utf8mb4_unicode_ci"
character-set-server  = utf8mb4
log-bin-trust-function-creators = 1
sort_buffer_size = 4294967295  #this is needed for geometry (bug in mysql:8)

[mariadb] or [mysql]
max_allowed_packet=100M
innodb_log_file_size=300M  #no use for mysql

2. If using MariaDB and you still have problems of type #1267 Illegal mix of collations, then check here on how to fix that,

Configure supervisord

Configure Supervisor, which is required for jobs. Create a file name opendatabio-worker.conf in the Supervisor configuration folder /etc/supervisor/conf.d/opendatabio-worker.conf with the following content:

touch /etc/supervisor/conf.d/opendatabio-worker.conf
echo ";--------------
[program:opendatabio-worker]
process_name=%(program_name)s_%(process_num)02d
command=php /home/odbserver/opendatabio/artisan queue:work --sleep=3 --tries=1 --timeout=0 --memory=512
autostart=true
autorestart=true
user=odbserver
numprocs=8
redirect_stderr=true
stdout_logfile=/home/odbserver/opendatabio/storage/logs/supervisor.log
;--------------" > /etc/supervisor/conf.d/opendatabio-worker.conf

Folder permissions

• Folders storage and bootstrap/cache must be writable by the Server user (usually www-data). Set 0755 permission to these directories.
• Config .env file requires 0640 permission.
• This link has different ways to set up permissions for files and folders of a Laravel application. Below the preferred method:

cd /home/odbserver

#give write permissions to odbserver user and the apache user
sudo chown -R odbserver:www-data opendatabio
sudo find ./opendatabio -type f -exec chmod 644 {} \;
sudo find ./opendatabio -type d -exec chmod 755 {} \;  

#in these folders the server stores data and files.
#Make sure their permission is correct
cd /home/odbserver/opendatabio
sudo chgrp -R www-data storage bootstrap/cache
sudo chmod -R ug+rwx storage bootstrap/cache

#make sure the .env file has 640 permission
sudo chmod 640 ./.env

Install OpenDataBio

1. Many Linux distributions (most notably Ubuntu and Debian) have different php.ini files for the command line interface and the Apache plugin. It is recommended to use the configuration file for Apache when running the install script, so it will be able to correctly point out missing extensions or configurations. To do so, find the correct path to the .ini file, and export it before using the php install command.

For example,

export PHPRC=/etc/php/7.4/apache2/php.ini

2. The installation script will download the Composer dependency manager and all required PHP libraries listed in the composer.json file. However, if your server is behind a proxy, you should install and configure Composer independently. We have implemented PROXY configuration, but we are not using it anymore and have not tested properly (if you require adjustments, place an issue on GitLab).

3. The script will prompt you configurations options, which are stored in the environment .env file in the application root folder.

You may, optionally, configure this file before running the installer:

• Create a .env file with the contents of the provided cp .env.example .env
• Read the comments in this file and adjust accordingly.

4. Run the installer:

cd /home/odbserver/opendatabio
php install

5. Seed data - the script above will ask if you want to install seed data for Locations and Taxons - seed data is version specific. Check the seed data repository version notes.

Installation issues

There are countless possible ways to install the application, but they may involve more steps and configurations.

• if you browser return 500|SERVER ERROR you should look to the last error in storage/logs/laravel.log. If you have ERROR: No application encryption key has been specified run:

php artisan key:generate
php artisan config:cache

• If you receive the error “failed to open stream: Connection timed out” while running the installer, this indicates a misconfiguration of your IPv6 routing. The easiest fix is to disable IPv6 routing on the server.
• If you receive errors during the random seeding of the database, you may attempt to remove the database entirely and rebuild it. Of course, do not run this on a production installation.

php artisan migrate:fresh

• You may also replace the Locations and Taxons tables with seed data after a fresh migration using:

php seedodb

Post-install configs

• If your import/export jobs are not being processed, make sure Supervisor is running systemctl start supervisord && systemctl enable supervisord, and check the log files at storage/logs/supervisor.log.
• You can change several configuration variables for the application. The most important of those are probably set by the installer, and include database configuration and proxy settings, but many more exist in the .env and config/app.php files. In particular, you may want to change the language, timezone and e-mail settings. Run php artisan config:cache after updating the config files.
• In order to stop search engine crawlers from indexing your database, add the following to your “robots.txt” in your server root folder (in Debian, /var/www/html):

User-agent: *
Disallow: /

Storage & Backups

You may change storage configurations in config/filesystem.php, where you may define cloud based storage, which may be needed if have many users submitting media files, requiring lots of drive space.

1. Data downloads are queue as jobs and a file is written in a temporary folder, and the file is deleted when the job is deleted by the user. This folder is defined as the download disk in filesystem.php config file, which point to storage/app/public/downloads. UserJobs web interface difficult navigation will force users to delete old jobs, but a cron cleaning job may be advisable to implement in your installation;
2. Media files are by default stored in the media disk, which place files in folder storage/app/public/media;
3. For regular configuration create both directories storage/app/public/downloads and storage/app/public/media with writable permissions by the Server user, see below topic;
4. Remember to include media folder in a backup plan;

3 - Docker Installation

The easiest way to install and run OpenDataBio is using Docker and the docker configuration files provided, which contain all the needed configurations to run ODB. Uses nginx and mysql, and supervisor for queues

Docker files

laraverl-app/
----docker/*
----./env.docker
----docker-compose.yml
----Dockerfile
----Makefile

These are adapted from this link, where you find a production setting as well.

Installation

1. Make sure you have Docker and Docker-compose installed in your system;
2. Check if your user is in the docker group created during docker installation;
3. Download or clone the OpenDataBio in your machine;
4. Make sure your user is the owner of the opendatabio folder created and contents, else change ownership and group recursively to your user
5. Enter the opendatabio directory
6. Edit and adjust the environment file name .env.docker (optional)
7. To install locally for development just adjust the following variables in the Dockerfile, which are needed to map the files owners to a docker user;
   1. UID the numeric user your are logged in and which is the owner of all files and directories in the opendatabio directory.
   2. GDI the numeric group the user belongs, usually same as UID.
8. File Makefile contains shortcuts to the docker-compose commands used to build the services configured in the docker-compose.yml and auxiliary files in the docker folder.
9. Build the docker containers using the shortcuts (read the Makefile to undersand the commands)

make build

1. Start the implemented docker Services

make start

1. See the containers and try log into the laravel container

docker ps
make ssh #to enter the container shell
make ssh-mysql #to enter the mysql container, where you may access the database shell using `mysql -uroot -p` or use the laravel user

1. Install composer dependencies

make composer-install

1. Migrate the database

make migrate

1. You may also replace the Locations and Taxons tables with seed data:

make seed-odb

1. If worked, then Opendatabio will be available in your browser http::/localhost:8080.
2. Login with superuser admin@example.org and password password1
3. Additional configurations in these files are required for a production environment and deployment;

Data persistence

The docker images may be deleted without loosing any data. The mysql tables are stored in a volume. You may change to a local path bind.

docker volume list

Using

See the contents of Makefile

make stop
make start
make restart
docker ps
...

If you have issues and changed the docker files, you may need to rebuild:

#delete all images without loosing data
make stop
docker system prune -a  #and accepts Yes
make build
make start

4 - Customize Installation

Simple changes that can be implemented in the layout of a OpenDataBio web site

Logo and BackGround Image

To replace the Navigation bar logo and the image of the landing page, just put your image files replacing the files in /public/custom/ without changing their names.

Texts and Info

To change the welcome text of the landing page, change the values of the array keys in the following files:

• /resources/lang/en/customs.php
• /resources/lang/pt/customs.php
• Do not remove the entry keys. Set to null to suppress from appearing in the footer and landing page.

NavBar and Footer

1. If you want to change the color of the top navigation bar and the footer, just replace css Boostrap 5 class in the corresponding tags and files in folder /resources/view/layout.
2. You may add additional html to the footer and navbar, change logo size, etc… as you wish.