BIDScoin: Coin your imaging data to BIDS¶
BIDScoin is a user friendly open-source Python application that converts (“coins”) source-level (raw) neuroimaging data-sets to standardized data-sets that are organized according to the Brain Imaging Data Structure (BIDS) specification. Rather then depending on complex programmatic logic for source data-type identification, BIDScoin uses a mapping approach to discover the different source data types in your repository and convert them into BIDS data types. Different runs of source data are uniquely identified by their file system properties (e.g. file name or size) and by their attributes (e.g. ProtocolName from the DICOM header). Mapping information can be pre-specified (e.g. per site), allowing BIDScoin to make intelligent first suggestions on how to classify and convert the data. While this command-line procedure exploits all information available on disk, BIDScoin presents a Graphical User Interface (GUI) for researchers to check and edit these mappings – bringing in the missing knowledge that often exists only in their heads.
Data conversions are performed within plugins, such as plugins that employ dcm2niix, spec2nii or nibabel.
BIDScoin requires no programming knowledge in order to use it, but users can use regular expression and plug-ins to further enhance BIDScoin’s power and flexibility, and readily handle a wide variety of source data types.
BIDScoin is developed at the Donders Institute of the Radboud University.
Features¶
[x] DICOM source data
[x] PAR/REC source data (Philips)
[x] NIfTI source data
[x] Physiological logging data*
[x] MR Spectroscopy data**
[x] PET data
[x] Field maps*
[x] Multi-echo data*
[x] Multi-coil data*
[ ] Stimulus/behavioural logfiles
[x] Plug-ins
* = Only DICOM source data** = Only Twix, SDAT/SPAR and P-file source data
Are you a Python programmer with an interest in BIDS who knows all about GE and / or Philips data?
Are you experienced with parsing stimulus presentation log-files? Or do you have ideas to improve
the this toolkit or its documentation? Have you come across bugs? Then you are highly encouraged to
provide feedback or contribute to this project on https://github.com/Donders-Institute/bidscoin.
Note:¶
The full BIDScoin documentation is hosted at Read the Docs
For citation and more information, see our BIDScoin publication in Frontiers in Neuroinformatics (doi: 10.3389/fninf.2021.770608)
Issues or questions can be posted at Github
Installation¶
Python installation¶
BIDScoin is a Python 3 package and can be installed on Linux, MS Windows and on OS-X computers, as long as a Python interpreter (v3.8 or higher) is available on the system. On Linux and OS-X this is usually already the case, but MS Windows users may need to first install Python themselves. See e.g. this Python 3 distribution for instructions.
BIDScoin installation¶
To install BIDScoin on your system run one of the following commands in your command-line interface / shell (tip: you may want or need to install bidscoin in a virtual / conda Python environment):
$ pip install bidscoin # Use this when you want to convert conventional MR imaging data with the dcm2niix2bids plugin
$ pip install bidscoin[spec2nii2bids] # Use this when you want to convert MR spectroscopy data with the spec2nii2bids plugin
$ pip install bidscoin[deface] # Use this when you want to deface anatomical MRI scans. NB: Requires FSL to be installed on your system
$ pip install bidscoin[deface,pet2bids] # Use this when you want to deface anatomical MRI scans and convert PET data with the pet2bids plugin
$ pip install bidscoin[all] # Use this to install all extra packages
These install commands can be run independently and will give you the latest stable release of BIDScoin and its plugins. Alternatively, if you need to use the very latest (development / unstable) version of the software, you can also install BIDScoin directly from the github source code repository:
$ pip install git+https://github.com/Donders-Institute/bidscoin
If you do not have git (or any other version control system) installed you can download and unzip the code yourself in a folder named e.g. ‘bidscoin’ and run:
$ pip install ./bidscoin
Updating BIDScoin¶
Run your pip install command as before with the additional --upgrade or --force-reinstall option, e.g.:
$ pip install --upgrade bidscoin # The latest stable release
$ pip install --force-reinstall git+https://github.com/Donders-Institute/bidscoin # The latest code (add ``--no-deps`` to only upgrade the bidscoin package)
Caution
The bidsmaps are not guaranteed to be compatible between different BIDScoin versions
After a successful BIDScoin installation or upgrade, it may be needed to (re)do any adjustments that were done on your template bidsmap (so make a back-up of it before you upgrade)
Dcm2niix installation¶
Unfortunately the pip installer can only install Python software and the default ‘dcm2niix2bids’ plugin relies on an external application named dcm2niix to convert DICOM and PAR/REC source data to NIfTI. To make use of the dcm2niix2bids plugin you should therefore download and install dcm2niix yourself according to the instructions. When done, make sure that the dcm2niix executable is on your user or system path (Windows users can add the path permanently, e.g. by running: setx path "%path%;C:\Program Files\dcm2niix"). Otherwise, make sure that the command to run the dcm2niix executable (exactly as if you would run it yourself in your command terminal) is set correctly in the Options section in your bidsmap. This can be done in two ways:
Open your template bidsmap with a text editor and adjust the settings as needed. The default template bidsmap is located in the [path_to_bidscoin]/heuristics subfolder – see the output of
bidscoin -pfor the fullpath location on your system.Go to the Options tab the first time the BIDScoin GUI is launched and adjust the settings as needed. Then click the [Set as default] button to save the settings to your default template bidsmap.
Testing BIDScoin¶
You can run the ‘bidscoin’ utility to test the installation of your BIDScoin installation and settings:
$ bidscoin -t # Test with the default template bidsmap
$ bidscoin -t my_template_bidsmap # Test with your custom template bidsmap
See also the Troubleshooting guide for more information on potential installation issues.
Using an Apptainer (Singularity) container¶
An alternative for installing Python, BIDScoin and it’s dependencies yourself is to execute BIDScoin commands using an Apptainer container. Executing BIDScoin commands via a container is less simple than running them directly on your host computer, read the official documentation for installation and usage instructions. NB: “Singularity” has been rebranded as “Apptainer”, so Singularity users should replace apptainer for singularity in the commands given below.
The Apptainer current image includes:
Debian Linux (see https://hub.docker.com/_/python)
the latest version of dcm2niix
the latest stable release of BIDScoin and its plugins
The current image does not include this (non-free) software needed for some bidsapps:
FSL (needed for
defaceandslicereport)Freesurfer/synthstrip (needed for
skullstrip)
Building the container image¶
Download the Apptainer definition file and execute the following command to build a BIDScoin container image:
$ sudo apptainer build bidscoin.sif apptainer.def
Alternatively, you can first build a Docker image (see instructions in the section below), save it to e.g. bidscoin.tar and then convert it into a Apptainer image using:
$ sudo apptainer build bidscoin.sif bidscoin.tar
Run BIDScoin tools in the container¶
You can use the following command syntax to execute BIDScoin tools in the container:
$ apptainer exec bidscoin.sif <bidscoin_tool> <bidscoin_tool_args>
Where <bidscoin_tool> is a BIDScoin tool (e.g., bidsmapper, bidscoiner, dicomsort) and <bidscoin_tool_args> are the tool’s arguments. So for instance, if you have source data in myhome/data/raw, instead of running bidsmapper data/raw data/bids and then bidsmapper data/raw data/bids from your home directory, you now execute:
$ xhost +
$ apptainer exec bidscoin.sif bidsmapper data/raw data/bids
$ xhost -
$ apptainer exec bidscoin.sif bidscoiner data/raw data/bids
The xhost + command allows Apptainer to open a graphical display on your computer and normally needs to be run once before launching a GUI application, i.e. is needed for running the bidseditor.
If your data doesn’t reside in your home folder, then you need to add a --bind <host_dir>:<container_dir> Apptainer argument which maps a folder from the host system to a folder inside the Apptainer container. So if yuor data is in /myproject/raw, you run:
$ apptainer exec bidscoin.sif --bind /myproject <bidscoin_tool> <bidscoin_tool_args>
See the documentation for usage and setting environment variables to automatically bind your root paths for all containers.
Using a Docker container¶
If the Apptainer container is not working for you, it is also possible to use a Docker container. The Docker versus Apptainer image and container usage are very similar, and both have their pros and cons. A fundamental argument for using Apptainer is that it doesn’t require root permission (admin rights), whereas a fundamental argument for using Docker is that it is not limited to Linux hosts.
The current Docker image includes the same as the Apptainer image:
Debian Linux (see https://hub.docker.com/_/python)
the latest version of dcm2niix
the latest stable release of BIDScoin and its plugins
Likewise, the current image does not include this (non-free) software needed for some bidsapps:
FSL (needed for
defaceandslicereport)Freesurfer/synthstrip (needed for
skullstrip)
Building the container image¶
Download the Dockerfile and execute the following command to build a BIDScoin container image:
$ sudo docker build -t bidscoin .
Run BIDScoin tools in the container¶
Executing BIDScoin commands via Docker is less simple than via Apptainer (and surely less simple than running them directly on your host computer). For instance, it is typically needed to bind-mount your data folder(s) in the container and, for the bidseditor, to bind-mount an x-server socket to display the GUI in your host computer. The syntax to run dockerized bidscoin tools is:
$ docker run --rm -v <bind_mount> bidscoin <bidscoin_tool> <bidscoin_tool_args>
If you have source data in /my/data/raw, instead of running bidsmapper /my/data/raw /my/data/bids and then bidsmapper /my/data/raw /my/data/bids, you now execute for instance:
$ xhost +
$ sudo docker run --rm -e DISPLAY=$DISPLAY -v /tmp/.X11-unix:/tmp/.X11-unix -v /my/data:/mnt bidscoin bidsmapper /my/data/raw /my/data/bids
$ xhost -
$ sudo docker run --rm -v /my/data:/my/data bidscoin bidscoiner /my/data/raw /my/data/bids
As for Apptainer, the xhost + is normally needed to be launching a GUI application, but a few more arguments are now required, i.e. -e for setting the display number and -v for binding the data volume and for binding the x-server socket (see the documentation for usage and configuring bind propagation).
Data preparation¶
Supported source data structures¶
Out of the box, BIDScoin requires that the source data repository is organized according to a subject/[session]/data structure (the session subfolder is always optional). The data folder(s) can be structured in various ways (depending on the plugin and/or dataformat), as illustrated by the following examples:
1. A DICOM Series layout¶
The data folder is organized in multiple series subfolders, each of which that contains a single data type that is typically acquired in a single run – a.k.a ‘Series’ in DICOM speak. This is how users receive their data from the (Siemens) scanners at the DCCN:
sourcedata
|-- sub-001
| |-- ses-mri01
| | |-- 001-localizer
| | | |-- 00001_1.3.12.2.1107.5.2.19.45416.2017121914582956872274162.IMA
| | | |-- 00002_1.3.12.2.1107.5.2.19.45416.2017121914583757650874172.IMA
| | | `-- 00003_1.3.12.2.1107.5.2.19.45416.2017121914583358068374167.IMA
| | |
| | |-- 002-t1_mprage_sag_p2_iso_1.0
| | | |-- 00002_1.3.12.2.1107.5.2.19.45416.2017121915051526005675150.IMA
| | | |-- 00003_1.3.12.2.1107.5.2.19.45416.2017121915051520026075138.IMA
| | | |-- 00004_1.3.12.2.1107.5.2.19.45416.2017121915051515689275130.IMA
| | | [..]
| | [..]
| |
| `-- ses-mri02
| |-- 001-localizer
| | |-- 00001_1.3.12.2.1107.5.2.19.45416.2017121914582956872274162.IMA
| | |-- 00002_1.3.12.2.1107.5.2.19.45416.2017121914583757650874172.IMA
| | `-- 00003_1.3.12.2.1107.5.2.19.45416.2017121914583358068374167.IMA
| [..]
|
|-- sub-002
| `-- ses-mri01
| |-- 001-localizer
| | |-- 00001_1.3.12.2.1107.5.2.19.45416.2017121914582956872274162.IMA
| | |-- 00002_1.3.12.2.1107.5.2.19.45416.2017121914583757650874172.IMA
| | `-- 00003_1.3.12.2.1107.5.2.19.45416.2017121914583358068374167.IMA
| [..]
[..]
2. A DICOMDIR layout¶
The data folder contains a DICOMDIR file and multiple subfolders. A DICOMDIR is dictionary-file that indicates the various places where the DICOM files are stored. DICOMDIRs are often used in clinical settings and may look like this (example is without the optional session subfolders):
sourcedata
|-- sub-001
| |-- DICOM
| | `-- 00001EE9
| | `-- AAFC99B8
| | `-- AA547EAB
| | |-- 00000025
| | | |-- EE008C45
| | | |-- EE027F55
| | | |-- EE03D17C
| | | [..]
| | |
| | |-- 000000B4
| | | |-- EE07CCDA
| | | |-- EE0E0701
| | | |-- EE0E200A
| | | [..]
| | [..]
| `-- DICOMDIR
|
|-- sub-002
| [..]
[..]
The above organization of one DICOMDIR file per subject or session is supported out of the box by the bidscoiner and bidsmapper. If you have a single multi-subject DICOMDIR file for your entire repository you can reorganize your data by running the `dicomsort <utilities.html#dicomsort>`__ utility beforehand.
3. A flat DICOM layout¶
In a flat DICOM layout the data folder contains all the DICOM files of all the different Series without any subfolders. This layout is sometimes used when exporting data in clinical settings (NB: in this example, non-default ‘pat^’ and ‘ses^’ subject/session prefixes are used):
sourcedata
|-- pat^001
| `-- ses^01
| |-- IM_0001.dcm
| |-- IM_0002.dcm
| |-- IM_0003.dcm
| [..]
|
|-- pat^002
| `-- ses^01
| |-- IM_0001.dcm
| |-- IM_0002.dcm
| |-- IM_0003.dcm
| [..]
[..]
4. A PAR/REC layout¶
All PAR/REC(/XML) files of all the different Series are contained in the data folder (without subfolders). This layout is how users often export their data from Philips scanners in research settings (optional json sidecar files were added in this example to complement the PAR-files):
sourcedata
|-- sub-001
| `-- ses-mri01
| |-- TCHC_066_1_WIP_Hanneke_Block_2_SENSE_4_1.PAR
| |-- TCHC_066_1_WIP_Hanneke_Block_2_SENSE_4_1.REC
| |-- TCHC_066_1_WIP_Hanneke_Block_2_SENSE_4_1.json
| |-- TCHC_066_1_WIP_IDED_SENSE_6_1.PAR
| |-- TCHC_066_1_WIP_IDED_SENSE_6_1.REC
| |-- TCHC_066_1_WIP_IDED_SENSE_6_1.json
| [..]
|
|-- sub-002
| `-- ses-mri01
| |-- TCHC_066_1_WIP_Hanneke_Block_2_SENSE_4_1.PAR
| |-- TCHC_066_1_WIP_Hanneke_Block_2_SENSE_4_1.REC
| |-- TCHC_066_1_WIP_Hanneke_Block_2_SENSE_4_1.json
| |-- TCHC_066_1_WIP_IDED_SENSE_6_1.PAR
| |-- TCHC_066_1_WIP_IDED_SENSE_6_1.REC
| |-- TCHC_066_1_WIP_IDED_SENSE_6_1.json
| [..]
[..]
5. Miscellaneous layouts¶
The above layouts are supported by the (default) dcm2niix2bids plugin. The other pre-installed plugins search the data folder recursively for source files. For instance, the nested mix of NIfTI and MRS source data in the example below can be converted to BIDS using the nibabel2bids plugin in combination with the spec2nii2bids plugin (NB: example is with a non-default ‘’ (empty) subject prefix):
sourcedata
|-- 001
| |-- anat
| | `-- 35-365_t1_structural_20150418_03.nii
| | `-- 35-365_t1_structural_20150418_03.json
| |-- met
| | |-- 35-365_MRS_ACC_supressed_20150418_04.SDAT
| | `-- 35-365_MRS_ACC_supressed_20150418_04.SPAR
| `-- water
| |-- 35-365_MRS_ACC_20150418_05.SDAT
| `-- 35-365_MRS_ACC_20150418_05.SPAR
|
|-- 002
| `-- 35-365.tar.gz
|
|-- 003
| |-- TCHC_066_1_WIP_Hanneke_Block_2_SENSE_4_1.nii.gz
| |-- TCHC_066_1_WIP_Hanneke_Block_2_SENSE_4_1.json
| [..]
[..]
Note
You can store your session data in any of the above data layouts as zipped (.zip) or tarzipped (e.g. .tar.gz) archive files. BIDScoin workflow tools will automatically unpack/unzip those archive files in a temporary folder and then process your session data from there. For flat/DICOMDIR data, BIDScoin tools (i.e. the bidsmapper and the bidscoiner) will automatically run dicomsort in a temporary folder to sort them in seriesfolders. Depending on the data and file system, repeatedly unzipping data in the workflow may come with a significant processing speed penalty.
Tip
BIDScoin plugins will typically skip (Linux-style hidden) files and folders of which the name starts with a . (dot) character. You can use this feature to flexibly omit subjects, sessions or runs from your bids repository, for instance when you restarted an MRI scan because something went wrong with the stimulus presentation and you don’t want that data to be converted and enumerated as run-1, run-2.
Recommended data acquisition conventions¶
BIDScoin can automatically recognize source datatypes on the basis of it’s properties and attributes. Typically, in the DCCN users name their MR scan protocols in a meaningful way, which is therefore used as a basis for intelligent source datatype identification. For instance, if a functional fmri protocol is named “StopTask” or “fMRI_Stroop”, the default bidsmap_dccn template will yield a positive ‘func/bold’ match, as it has the “task” and “fMRI” keywords in it’s run-item regular expression: {ProtocolName: (?i).*(f.?MRI|task|BOLD|func|rest|RSN|CMRR.*_TR).*}. Similarly, anatomical scans that have T1w or MPRAGE in their protocol name are identified as anat/T1w items, and field maps that have fmap, fieldmap or B0map in their protocol name are identified as field maps. On the other hand, if a functional scan is just named Stop, the datatype cannot be correctly identified (at least not by the default template) and needs to be manually changed in the bidseditor from extra_data to func. A robust way to acquire and convert your data is hence to use (BIDS-like) descriptive names for your protocols, or for any other attribute or property (such as filenames) that you may use to manage your data. For more details and keywords, see e.g. the DCCN template bidsmap.
The BIDScoin workflow¶
With a sufficiently organized source data folder, the data conversion to BIDS can be performed by running the (1a) the bidsmapper, (1b) the bidseditor and (2) the bidscoiner command-line tools. The bidsmapper starts by building a map of the different kind of data types (scans) in your source dataset, which you can then edit with the bidseditor. The bidscoiner reads this so-called study bidsmap, which tells it how exactly to convert (“coin”) the source data into a BIDS data repository.
Creation and application of a study bidsmap¶
By default, when finished the bidsmapper automatically launches the bidseditor, so in it’s simplest form, all you need to do to convert your raw source data into BIDS is to run two simple shell commands, e.g.:
$ bidsmapper sourcefolder bidsfolder # Scans your data and creates a study bidsmap
$ bidscoiner sourcefolder bidsfolder # Converts your data to BIDS using the study bidsmap
If you add new subjects all you need to do is re-run the bidscoiner – unless the scan protocol was changed, then you also need to first re-run the bidsmapper to add the new samples to the study bidsmap. The paragraphs below describe the BIDScoin workflow in more detail.
Tip
If you don’t know what shell command to use or what to do, run the bidscoin command to give you a workflow overview
Step 1a: Running the bidsmapper¶
usage: bidsmapper.py [-h] [-b BIDSMAP] [-t TEMPLATE] [-p PLUGINS [PLUGINS ...]] [-n SUBPREFIX]
[-m SESPREFIX] [-u UNZIP] [-s] [-a] [-f] [--no-update]
sourcefolder bidsfolder
The bidsmapper scans your source data repository to identify different data types by matching
them against the run-items in the template bidsmap. Once a match is found, a mapping to BIDS
output data types is made and the run-item is added to the study bidsmap. You can check and
edit these generated bids-mappings to your needs with the (automatically launched) bidseditor.
Re-run the bidsmapper whenever something was changed in your data acquisition protocol and
edit the new data type to your needs (your existing bidsmap will be re-used).
The bidsmapper uses plugins, as stored in the bidsmap['Options'], to do the actual work
positional arguments:
sourcefolder The study root folder containing the raw source data folders
bidsfolder The destination folder with the (future) bids data and the
bidsfolder/code/bidscoin/bidsmap.yaml output file
options:
-h, --help show this help message and exit
-b BIDSMAP, --bidsmap BIDSMAP
The study bidsmap file with the mapping heuristics. If the bidsmap filename
is relative (i.e. no '/' in the name) then it is assumed to be located in
bidsfolder/code/bidscoin. Default: bidsmap.yaml
-t TEMPLATE, --template TEMPLATE
The bidsmap template file with the default heuristics (this could be provided
by your institute). If the bidsmap filename is relative (i.e. no '/' in the
name) then it is assumed to be located in bidsfolder/code/bidscoin. Default:
bidsmap_dccn
-p PLUGINS [PLUGINS ...], --plugins PLUGINS [PLUGINS ...]
List of plugins to be used. Default: the plugin list of the study/template
bidsmap)
-n SUBPREFIX, --subprefix SUBPREFIX
The prefix common for all the source subject-folders (e.g. 'Pt' is the
subprefix if subject folders are named 'Pt018', 'Pt019', ...). Use '*' when
your subject folders do not have a prefix. Default: the value of the
study/template bidsmap, e.g. 'sub-'
-m SESPREFIX, --sesprefix SESPREFIX
The prefix common for all the source session-folders (e.g. 'M_' is the
subprefix if session folders are named 'M_pre', 'M_post', ..). Use '*' when
your session folders do not have a prefix. Default: the value of the
study/template bidsmap, e.g. 'ses-'
-u UNZIP, --unzip UNZIP
Wildcard pattern to unpack tarball/zip-files in the sub/ses sourcefolder that
need to be unzipped (in a tempdir) to make the data readable. Default: the
value of the study/template bidsmap
-s, --store Store provenance data samples in the bidsfolder/code/provenance folder
(useful for inspecting e.g. zipped or transfered datasets)
-a, --automated Save the automatically generated bidsmap to disk and without interactively
tweaking it with the bidseditor
-f, --force Discard the previously saved bidsmap and logfile
--no-update Do not update any sub/sesprefixes in or prepend the sourcefolder name to the
<<filepath:regexp>> expression that extracts the subject/session labels. This
is normally done to make the extraction more robust, but could cause problems
for certain use cases
examples:
bidsmapper myproject/raw myproject/bids
bidsmapper myproject/raw myproject/bids -t bidsmap_custom # Uses a template bidsmap of choice
bidsmapper myproject/raw myproject/bids -p nibabel2bids # Uses a plugin of choice
bidsmapper myproject/raw myproject/bids -u '*.tar.gz' # Unzip tarball sourcefiles
After the source data has been scanned, the bidsmapper will automatically launch step 1b to let the user check and edit the automatically generated study bidsmap. For a fully automated workflow users can skip this interactive step using the -i option (see above).
Tip
The default template bidsmap (-t bidsmap_dccn) is customized for acquisitions at the DCCN. If this bidsmap is not working well for you, consider adapting it to your needs so that the bidsmapper can recognize more of your scans and automatically map them to BIDS the way you prefer.
Step 1b: Running the bidseditor¶
usage: bidseditor.py [-h] [-b BIDSMAP] [-t TEMPLATE] bidsfolder
This application launches a graphical user interface for editing the bidsmap that is produced
by the bidsmapper. You can edit the BIDS data types and entities until all run-items have a
meaningful and nicely readable BIDS output name. The (saved) bidsmap.yaml output file will be
used by the bidscoiner to do the conversion of the source data to BIDS.
You can hoover with your mouse over items to get help text (pop-up tooltips).
positional arguments:
bidsfolder The destination folder with the (future) bids data
options:
-h, --help show this help message and exit
-b BIDSMAP, --bidsmap BIDSMAP
The study bidsmap file with the mapping heuristics. If the bidsmap filename
is relative (i.e. no "/" in the name) then it is assumed to be located in
bidsfolder/code/bidscoin. Default: bidsmap.yaml
-t TEMPLATE, --template TEMPLATE
The template bidsmap file with the default heuristics (this could be provided
by your institute). If the bidsmap filename is relative (i.e. no "/" in the
name) then it is assumed to be located in bidsfolder/code/bidscoin. Default:
bidsmap_dccn
examples:
bidseditor myproject/bids
bidseditor myproject/bids -t bidsmap_dccn.yaml
bidseditor myproject/bids -b my/custom/bidsmap.yaml
Main window¶
As shown below, the main window of the bidseditor opens with separate data mapping tabs for each data format that is present in the bidsmap (here DICOM mappings and PAR mappings). The data mapping tabs consist of a Participant labels table and a Data samples table. By default, the participant table contains dynamic <<filepath:regexp>> property values, which are used to extract the subject and session labels from the path of the source data during bidscoiner runtime. Alternatively, you can put a dynamic attribute value there (e.g. <<PatientName>>) if you want to extract that information from the source header. The data samples table shows a list of input files (left side) that uniquely represent all the different data types in the sourcedata repository, in conjunction with a preview of their BIDS output names (right side). The BIDS output names are shown in red if they are not BIDS compliant, crossed-out gray when the runs will be ignored / skipped in the conversion to BIDS, otherwise it is colored green.
The main window with the DICOM mappings tab, an Options tab and a Data browser tab. The selected DICOM mappings tab shows an overview of how DICOM source data types (left) are mapped to BIDS output data (right). The BIDScoin settings used for this study can be adjusted in the Options tab and the Data browser tab can be used to inspect the source data structure.¶
Tip
If the default subject/session expression (e.g. /sub-(.*?)/ where sub- can be substituted by your prefix) fails to parse the subject or session label, try prepending (a part of) the sourcefolder path, e.g. if your data is in /project/sourcedata/s001/.. and your subject prefix is s, try <<filepath:/sourcedata/s(.*?)/>> for extracting the 001 subject label. This is especially useful if your subject folders have no or a very short prefix.
Tip
Clear the session label field if you have data with only one session. This will remove the optional session label from the BIDS output name
Edit window¶
In the main window, you can double-click the BIDS output name of a data sample or click the [Edit] button next to it (NB: the * in this button indicates that attention is required) to open a new window, as shown below. In this new window, the full bids-mapping info of the clicked data-sample (AKA run-item) is shown, with the filesystem Properties and file Attributes input on the left, and, most importantly, the associated BIDS Data type, Data filename and Meta data output on the right. Editing the properties and attributes is usually not necessary and considered advanced usage, so you can focus on the BIDS output tables on the right. You should first make sure the BIDS Data type (drop down menu) and its suffix label (drop down menu) are set correctly, and then you should edit the (automatically generated) BIDS values that you think are not optimal or incorrect (double-click the cell). Each time an item is edited, a new Data filename preview is shown (green or red text indicates that the name is BIDS compliant or not). In the Meta data table (see the figure below) you can enter key-value pairs that you like to be stored as BIDS meta-data in the json sidecar file. Right-clicking the meta table allows you to import meta-data from JSON/YAML/CSV/TSV files on disk.
If the preview of the BIDS filename and meta-data both look good, you can store the data in the bidsmap by clicking the [OK] button.
The edit window for customizing a bidsmap run item, featuring the DICOM attributes mapped onto BIDS values and (dynamic) metadata values (e.g. Comments). BIDS values that are restricted to a limited set are presented with a drop-down menu.¶
Finally, if all BIDS output names in the main window are fine, you can click on the [Save] button and proceed with running the bidscoiner tool (step 2). Note that re-running the bidsmapper or bidseditor is always a safe thing to do since these tools will re-use the existing bidsmap yaml-file and will not delete or write anything to disk except to the bidsmap yaml-file.
Field maps¶
Field maps are acquired and stored in various (sequences and manufacturer dependent) ways and may require some special treatment. For instance, it could be that you have magnitude1 and magnitude2 data in one series-folder (which is what Siemens can do). In that case you should select the magnitude1 suffix and let bidscoiner automatically pick up the magnitude2 during runtime (or vice versa). The same holds for phase1 and phase2 data. The suffix magnitude can be selected for sequences that save field maps directly. See the BIDS specification for more details on field-map suffixes.
Field maps are typically acquired to be applied to specific other scans from the same session. The BIDS specification provides two meta-data mechanisms to store this semantic meta data (NB: BIDS-apps may not use your field map at all if you do not specify anything):
First there is the older
IntendedFormechanism that can handle more basic use cases, i.e. it explicitly specifies the path of the target images to which the field map should be applied, but it is left implicit from which images the field map is to be computed. You can enter a dynamicIntendedForsearch string in theMeta datatable to have BIDScoin automatically fill out this field for you. For instance you can simply usetask-Stop*_boldas a search pattern to specify all functional runs in the BIDS session that havetask-Stopand_boldas part of their filename. For more advanced usage and explanation, see the special bidsmap features sectionSecond, there is the new and more flexible
B0Fieldmapmechanism that uses aB0FieldIdentifierto group all the images from which the field map can be computed, and aB0FieldSourceto indicate which field map should be used to correct the image. For instance, you could use{B0FieldIdentifier: sbref_fmap}in yourAPandPAPE-polarsbrefimages, in conjunction with{B0FieldSource: sbref_fmap}in your associatedAPPE-polarboldimage.
Tip
The BIDScoin GUI features several ways to help you setting the right values: * Double-clicking an input filename pops-up an inspection window with the full header information (e.g. useful for checking attributes that are not (yet) in your bidsmap) * Hoovering with your mouse over a cell pops-up a tooltip with more background information (e.g. from the BIDS specifications) * Always check the terminal output and make sure there are no warnings or error messages there (a summary of them is printed when exiting the application)
Step 2: Running the bidscoiner¶
usage: bidscoiner.py [-h] [-p PARTICIPANT_LABEL [PARTICIPANT_LABEL ...]] [-b BIDSMAP] [-f]
sourcefolder bidsfolder
Converts ("coins") your source datasets to NIfTI/json/tsv BIDS datasets using the mapping
information from the bidsmap.yaml file. Edit this bidsmap to your needs using the bidseditor
tool before running this function or (re-)run the bidsmapper whenever you encounter unexpected
data. You can run bidscoiner after all data has been collected, or run / re-run it whenever
new data has been added to your source folder (presuming the scan protocol hasn't changed).
Also, if you delete a subject/session folder from the bidsfolder, it will simply be re-created
from the sourcefolder the next time you run the bidscoiner.
The bidscoiner uses plugins, as stored in the bidsmap['Options'], to do the actual work
Provenance information, warnings and error messages are stored in the
bidsfolder/code/bidscoin/bidscoiner.log file.
positional arguments:
sourcefolder The study root folder containing the raw source data
bidsfolder The destination / output folder with the bids data
options:
-h, --help show this help message and exit
-p PARTICIPANT_LABEL [PARTICIPANT_LABEL ...], --participant_label PARTICIPANT_LABEL [PARTICIPANT_LABEL ...]
Space separated list of selected sub-# names / folders to be processed (the
sub-prefix can be removed). Otherwise all subjects in the sourcefolder will
be selected
-b BIDSMAP, --bidsmap BIDSMAP
The study bidsmap file with the mapping heuristics. If the bidsmap filename
is relative (i.e. no "/" in the name) then it is assumed to be located in
bidsfolder/code/bidscoin. Default: bidsmap.yaml
-f, --force Process all subjects, regardless of existing subject folders in the
bidsfolder. Otherwise these subject folders will be skipped
examples:
bidscoiner myproject/raw myproject/bids
bidscoiner -f myproject/raw myproject/bids -p sub-009 sub-030
Tip
Always check the terminal output for possible warnings or errors (a summary of them is printed at the end)
Check your json sidecar files of your field maps, in particular see if they have the expected
IntendedFor/B0FieldIdentifiervalues
Note
The provenance of the produced BIDS data-sets is stored in the [bidsfolder]/code/bidscoin/bidscoiner.log file. This file is also very useful for debugging / tracking down bidscoin issues.
Finishing up¶
After a successful run of bidscoiner, the work to convert your data in a fully compliant BIDS dataset is usually not fully over and, depending on the complexity of your data-set, additional tools may need to be run to post-process (e.g. deface) your data or convert datatypes not supported by the standard BIDScoin plugins (e.g. EEG data). BIDScoin comes with some bidsapps that can help you finishing up your bids data repository.
Adding more meta-data¶
To make your dataset reproducable and shareable, you should add study-level meta-data in the modality agnostic BIDS files (BIDScoin saves stub versions of them). For instance, you should update the content of the dataset_description.json and README files in your bids folder and you may need to provide e.g. additional *_sessions.tsv or participants.json files (see the BIDS specification for more information). Moreover, if you have behavioural log-files you will find that BIDScoin does not (yet) support converting these into BIDS compliant *_events.tsv/json files (advanced users are encouraged to use the bidscoiner plug-in option and write their own log-file parser).
BIDS validation¶
If all of the above work is done, you can (and should) run the web-based bidsvalidator to check for inconsistencies or missing files in your bids data-set (NB: the bidsvalidator also exists as a command-line tool).
Note
Privacy-sensitive source data samples may be stored in [bidsfolder]/code/bidscoin/provenance (see the -s option in the bidsmapper).
The bidsmap explained¶
Structure and content¶
A central concept in BIDScoin is the so-called bidsmap. Generally speaking, a bidsmap is a collection of run-items that define how source data types (e.g. a T1w- or a T2w-scan) should be converted to BIDS data types. As illustrated in the figure below (but see also the screenshot of the edit window), run-items consist of a ‘provenance’ field and a ‘properties’, ‘attributes’, ‘bids’ and a ‘meta’ dictionary (a set of key-value pairs):
The provenance field contains the pathname of a source data sample that is representative for the run-item. The provenance data is not strictly necessary but very useful for deeper inspection of the source data and for tracing back the conversion process, e.g. in case of encountering unexpected results
The properties dictionary contains file system properties of the data sample, i.e. the file path, the file name, the file size on disk and the number of files in the containing folder. Depending on your data management, this information allows or can help to identify different datatypes in your source data repository
The attributes dictionary contains attributes from the source data itself, such as the ‘ProtocolName’ from the DICOM header. The source attributes are a very rich source of information of which a minimal subset is normally sufficient to identify the different datatypes in your source data repository. The attributes are read from (the header of) the source file itself or, if present, from an accompanying sidecar file. This sidecar file transparently extends (or overrule) the available source attributes, as if that data would have been written to (the header of) the source data file itself. The name of the sidecar file should be the same as the name of the first associated source file and have a
.jsonfile extension. For instance, the001.dcm,002.dcm,003.dcm, [..], DICOM source images can have a sidecar file in the same directory named001.json(e.g. containing metadata that is not available in the DICOM header or that must be overruled). It should be noted that BIDScoin plugins will copy the extended attribute data over to the json sidecar files in your BIDS output folder, giving you additional control to generate your BIDS sidecar files (in addition to the meta dictionary described in point 5 below).The bids dictionary contains the BIDS datatype and entities that determine the filename of the BIDS output data. The values in this dictionary are encouraged to be edited by the user
The meta dictionary contains custom key-value pairs that are added to the json sidecar file by the BIDScoin plugins. Meta data may well vary from session to session, hence this dictionary often contains dynamic attribute values that are evaluated during bidscoiner runtime (see the special features below)
In sum, a run-item contains a single bids-mapping, which links the input dictionaries (2) and (3) to the output dictionaries (4) and (5).
A snippet of study bidsmap in YAML format. The bidsmap contains separate sections for each source data format (here ‘DICOM’) and sub-sections for the BIDS datatypes (here ‘anat’). The arrow illustrates how the ‘properties’ and ‘attributes’ input dictionaries are mapped onto the ‘bids’ and ‘meta’ output dictionaries. Note that the ‘part’ value in the bids dictionary is a list, which is presented in the bidseditor GUI as a drop-down menu (with the first empty item being selected). Also note the special double bracket dynamic values (<<..>>), which are explained below.¶
At the root level, a bidsmap is hierarchically organized in data format sections, such as ‘DICOM’ and ‘PAR’, which in turn contain subsections for the ‘participant_label’ and ‘session_label’, and subsections for the BIDS datatypes (such as ‘fmap’, ‘anat’, ‘func’) and for the ‘exclude’ and ‘extra_data’ datatypes. The ‘exclude’ datatype contains a list of run-items for source data that need to be omitted when converting the source data to BIDS and the ‘extra_data’ datatype contains a list of run-items for including miscellaneous data that is not (yet) defined in the BIDS specifications. In general you can think of the BIDS subsections as run-item lists of source datatypes that should be converted to it. The participant- and session-label subsections contain key-value pairs for setting the BIDS subject and session labels. Next to the data format sections there is a general ‘Options’ section, that accommodates BIDScoin and plugin settings.
When BIDScoin workflow routines process source data, they will scan the entire repository and take samples of the data and compare them with the run-items in the bidsmap until they come across a run-item of which all (non-empty) properties and attribute values match (using fullmatch) with the values extracted from the data sample at hand. At that point a run-item match is established, i.e. BIDScoin then knows precisely how to convert the data sample to BIDS. Bidsmaps can contain an unlimited number of run-items, including multiple run-items mapping onto the same BIDS target (e.g. when you renamed your DICOM scan protocol halfway your study and you don’t want that irrelevant change to be reflected in the BIDS output).
From template to study bidsmap¶
In BIDScoin a bidsmap can either be a template bidsmap or a study bidsmap. A template bidsmap contains a comprehensive set of run-items (one run-item for each BIDS target), each of which containing the prior knowledge about the source data properties and attributes that typically belong to a BIDS target. A study bidsmap, on the other hand, contains only run-items that matched positively with the source data of a study, i.e. it represents all the source data types present in the study (but nothing more). Moreover, this shortlist of run-items can be edited by the user (adding the posterior knowledge) to get the optimal mapping to BIDS for the data at hand. In the workflow (see figure below), the bidsmapper takes the template bidsmap and source data as input to automatically produce a first version of a study bidsmap. The runs in this study bidsmap are taken from the template bidsmap, with the difference that all attribute values of the matching run-item (including empty values and values with regular expressions) are replaced with the attribute values of the data samples. In this way, the run-items will uniquely match to the data samples, providing a complete mapping of all source data types to BIDS data types. These mappings can be edited by the user with the bidseditor and then given to the bidscoiner to do the actual conversion of the source data to BIDS.
Users normally don’t have to know about or interact with the template bidsmap, and only see study bidsmaps (in the bidseditor). To have a fully automated workflow, users with a very good template bidsmap and standardized data acquisition protocol can rely on the automated mappings and skip the bidseditor step (and hence don’t see the study bidsmap). In fact, it is then even possible (but certainly not recommended) to skip the bidsmapping step and pass the template bidsmap directly to the bidscoiner (for this to work you need to be sure beforehand that all source datatypes have matching run-items in the template bidsmap).
Creation and application of a study bidsmap¶
Special bidsmap features¶
The dictionary values in a bidsmap are not simple strings but have some special features that make BIDScoin powerful, flexible and helpful:
Run-item matching¶
Source property and attribute values of run-items in a bidsmap are interpreted as regular expression patterns when they are matched with your source data samples. For instance, a key-value pair of an attribute dictionary in your template bidsmap could be {ProtocolName: .*(mprage|T1w).*}, which would test if the extracted attribute string for ‘ProtocolName’ from the DICOM header of a data sample contains either a ‘mprage’ or a ‘T1w’ substring. More precisely, the Python expression that is evaluated is: match = re.fullmatch('.*(mprage|T1w).*', 't1_mprage_sag_p2_iso_1.0')) if the ProtocolName of the data sample is ‘t1_mprage_sag_p2_iso_1.0’.
Dynamic values¶
Dictionary values can be static, in which case the value is just a normal string, or dynamic, when the string is enclosed with single or double pointy brackets. In case of single pointy brackets the bids value will be replaced during bidsmapper, bidseditor and bidscoiner runtime by the value of the source attribute or property of the data sample at hand. It is also possible to then extract a substring from the source string by adding a colon-separated regular expression to the bids value. For instance the two dynamic values in {acq: <MRAcquisitionType>Demo<SeriesDescription:t1_(.*?)_sag>} will be replaced by {acq: 3DDemoMPRAGE} if the ‘MRAcquisitionType’ of the data sample is ‘3D’ and ‘SeriesDescription’ is ‘t1_MPRAGE_sag_p2_iso_1.0’. More precisely, the Python expression that is evaluated for the second dynamic ‘SeriesDescription’ value is: substring = re.findall('t1_(.*?)_sag', 't1_mprage_sag_p2_iso_1.0'). If dynamic values are enclosed with double pointy brackets, the only difference is that they will be replaced only during bidscoiner runtime – this is useful for bids values that are subject/session dependent. Double bracket dynamic values can for instance be used to add DICOM meta data that is not saved by default in the json sidecar files, such as <<ImageComments>> or <<RadionuclideTotalDose>>. Another example is the extraction of the subject and session label. For instance, you can use <<filepath:/sub-(.*?)/>> to extract ‘003’ (i.e. the shortest string between /sub- and /) if the data for that subject is in /data/raw/sub-003/ses-01. Alternatively, if the subject label is encoded in the DICOM PatientName as e.g. ID_003_anon, then <<PatientName:ID_(.*?)_>> would likewise extract ‘003’. To test out dynamic values (either with or without appended regular expressions), you can handily enter them in the bidseditor within single brackets to instantly obtain their resulting value.
Run-index¶
Dynamic values can handle many use cases and can be used throughout BIDScoin. Yet there are two exceptions that cannot always be handled directly with dynamic values. The first exception is the ‘run’-index in the bids output dictionary, since this index number cannot usually be determined from the data file alone. In that case, if the run-index is a dynamic number (e.g. {run: <<1>>}) and another output file with that run-index already exists, then during bidscoiner runtime this number will be incremented in compliance with the BIDS standard (e.g. to {run: 2}). If the run index is encoded in the header or filename, then the index can unambiguously be extracted using dynamic values. For instance, using {run: <<ProtocolName:run-(.*?)_>>} will give {run: 3} if the DICOM ProtocolName is t1_mprage_sag_run-3_iso_1.0.
Fieldmaps: IntendedFor¶
The other exception not covered by dynamic values is the ‘IntendedFor’ value in the meta dictionary of field maps. The IntendedFor value is a list of associated output files that you can specify within a dynamic value using Unix shell-style wildcards. In that way, the bidscoiner will lookup the path of these images on disk using the Python glob <https://docs.python.org/3.8/library/pathlib.html#pathlib.Path.glob>`__(*dynamic_value*) expression. For instance, using a simple ``{IntendedFor: <<task>>}` value will lookup all functional runs in the BIDS subject[/session] folder (since in BIDS these runs always have ‘task’ in their filename), whereas a more specific {IntendedFor: <<func/*Stop*Go_bold><func/*Reward*_bold>>} value will select all ‘Stop1Go’-, ‘Stop2Go’- and ‘Reward’ bold-runs in the func sub-folder. In case duplicated field maps are acquired (e.g. when a scan failed or a session was interrupted) you can limit the search scope by appending a colon-separated “bounding” term to the search pattern. E.g. {IntendedFor: <<task:[]>>} will bound the wildcard search to files that are ‘uninterruptedly connected’ to the current field map, i.e. without there being another run of the field map in between. The bounded search can be further constrained by limiting the maximum number of matches, indicated with lower and upper limits. For instance {IntendedFor: <<task:[-3:0]>>} will limit the bounded search to maximally three runs preceding the field map. Similarly, {IntendedFor: <<task:[-2:2]>>} will limit the bounded search to maximally two preceding and two subsequent runs, and {IntendedFor: <<task:[0:]>>} will limit the bounded search to all matches acquired after the field map. In this latter case, for the first field map, only task-Stop_run-1 and task-Stop_run-2 will match the bounded search if the 5 collected runs were named: 1) fieldmap_run-1, 2) task-Stop_run-1, 3) task-Stop_run-2, 4) fieldmap_run-2, 5) task-Stop_run-3. The second run of the field map will match with task-Stop_run-3 only (note that the second field map would have matched all task runs if the bounding term would have been [], [:] or [-2:2]).
Note
The IntendedFor field is a legacy way to deal with field maps. Instead, it is recommended to use the B0FieldIdentifier and B0FieldSource fields that were introduced with BIDS 1.7
BIDS value lists. Instead of a normal string, a bids dictionary value can also be a list of strings, with the last list item being the (zero-based) list index that selects the actual value from the list. For instance the list
{part: ['', 'mag', 'phase', 'real', 'imag', 2]}would select ‘phase’ as the value belonging to ‘part’. A bids value list is made visible in the bidseditor as a drop-down menu in which the user can select the value (i.e. set the list index).
Tip
In addition to DICOM attribute names, the more advanced / unambiguous pydicom-style tag numbers can also be used for indexing a DICOM header. For instance, the PatientName, 0x00100010, 0x10,0x10, (0x10, 0x10), and (0010, 0010) index keys are all equivalent.
Building your own template bidsmap¶
The run-items in the default ‘bidsmap_dccn’ template bidsmap have values that are tailored to MRI acquisitions in the Donders Institute. Hence, if you are using different protocol parameters that do not match with these template values or you are using e.g. filenames instead of header information to typify your data, then your runs will initially be data (mis)typed by the bidsmapper as miscellaneous ‘extra_data’ – which you then need to correct afterwards yourself. To improve that initial data typing and further automate your workflow, you may consider creating your own customized template bidsmap. Here are some things to keep in mind when building your own template bidsmap:
To get the correct match for every source data type, run-items of template (but not study) bidsmaps typically contain regular expressions in their property and/or in attribute values. These regular expressions should best be designed to broadly but uniquely match the values in the source data, i.e. they should match with all variations of the same source data type, but never match with any other source data type. The expressions can be considered as prior knowledge about the data, and can be dependent on your data acquisition protocol.
When matching a data sample to run-items in a bidsmap, the search order is such that data samples will first be matched to the ‘exclude’ run-items, then, if they don’t match, to the BIDS run-items (the items in ‘fmap’, ‘anat’, ‘func’, etc) and finally, if none of those match either, to the ‘extra_data’ run-items. The search order for the list of run-items within each BIDS datatype is from top to bottom. The search order can play a role (and can be exploited) if you have run-items that are very similar, i.e. have (partly) overlapping properties or attributes. You can use this to your advantage by placing certain run-items before others. For instance, if you are adding run-items for multi-band EPI pulse sequences, you may want to put your ‘SBREF’ run-item before your ‘MB’ run-item and put a minor extra property and/or attribute that is unique to the additionally acquired single-band reference image. So if the SeriesDescription is “task_fMRI” for the MB sequence and “task_fMRISBREF” for the SBREF sequence, then you can have
{SeriesDescription: .*fMRI.*}for the MB run-item while narrowing down the matching pattern of the SBREF to{SeriesDescription: .*fMRISBREF.*}. MB data samples will not match the latter pattern but will match with the MB run-item. SBREF samples will match with both run-items, but only the SBREF run-item will be copied over to the study bidsmap because it is encountered before the MB run-item (BIDScoin stops searching the bidsmap if it finds a match).In your template bidsmap you can populate your run-items with any set of
propertiesand/orattributes. For instance if in your centre you are using the “PerformedProcedureStepDescription” DICOM attribute instead of “SeriesDescription” to store your metadata then you can (probably should) include that attribute to get more successful matches for your run-items. What you should not include there are properties or attributes that vary between repeats of the same acquisition, e.g. the DICOM ‘AcquisitionTime’ attribute (that makes every data sample unique and will hence give you a very long list of mostly redundant run-items in your study bidsmap). It is however perfectly fine to use such varying properties or attributes in dynamic values of thebidsandmetarun-item dictionaries (see below).Single dynamic brackets containing source properties or attributes can be used in the bids and meta dictionary, to have them show up in the bidseditor as pre-filled proposals for BIDS labels and/or sidecar meta data values. For instance, if you put
{ContrastName: <ContrastAgent>}in a meta-dictionary in your template bidsmap, it will show up in the bidseditor GUI as{ContrastName: PureGadolinium}. Double dynamic brackets can also be used, but these remain unevaluated until bidscoiner runtime. Double brackets are therefore only needed when the property or attribute value varies from subject to subject (such as “<<Age>>”) or from acquisition to acquisition (such as “<<InjectedMass>>”).Finally, it is a good practice for the first run-item in each BIDS datatype section of your template bidsmap to have all empty properties and attributes values. The benefit of this is that you can dereference (‘copy’) it in other run-items (see the editing section below), and in this way improve your consistency and reduce the maintenance burden of keeping your template bidsmap up-to-date. The first run-item is also the item that is selected when a user manually sets the run-item to this BIDS datatype in the bidseditor GUI.
Tip
Make a copy of the DCCN template (
[path_to_bidscoin]/heuristics/bidsmap_dccn.yaml) as a starting point for your own template bidsmap, and adapt it to your needs.The power of regular expressions is nearly unlimited, you can e.g. use negative look aheads to not match (exclude) certain strings
When creating new run-items, make sure to adhere to the YAML format and to the definitions in the BIDS schema files (
[path_to_bidscoin]/bidscoin/schema/datatypes). You can test your YAML syntax using an online YAML-validator and your compliance with the BIDS standard withbidscoin -t your_template_bidsmap. If all seems well you can install it usingbidscoin -i your_template_bidsmap.
Editing the template bidsmap¶
Using the bidseditor. While this is certainly not recommended for most use cases (as it may break), the easiest (quick and dirty) way to create a bidsmap template is to use the bidseditor GUI. If you have a run item in your study bidsmap that you would like to be automatically mapped in other / future studies you can simply append that run to any template bidsmap by using the [Export] button in the GUI (see screenshot below). However, as explained above, before you do that you should always clear the attribute values (e.g. ‘EchoTime’) that vary across repeats of the same or similar acquisitions. If you want to make the run-item more generic, note that you can still use regular expressions as
propertiesandattributesmatching patterns. Instead of exporting to a template bidsmap, you can also open (and edit) the template bidsmap itself with the bidseditor. An important limitation of exporting run items is that they are appended to a bidsmap template, meaning that they are last in line (for that datatype) when the bidsmapper searches for a matching run-item. Another limitation is that with the GUI you cannot make use of YAML anchors and references, giving you a less clearly formatted bidsmap that is harder to maintain. Both limitations are overcome when directly editing the template bidsmap yourself using a text editor.Using a text editor. The advised way to create or modify template bidsmaps is to use a text editor and edit the raw bidsmap data directly. The bidsmap data is stored in YAML format, so you do have to have some basic understanding of this data-serialization language. As can be seen from the template snippet below, YAML format is quite human-friendly and human-readable, but there are a few things you should be aware of, most notably the use of anchors and aliases. The dccn template bidsmap uses anchors in the first run-item of a BIDS datatype, and aliases in the others (to dereference the content of the anchors). And because all values of the
propertiesandattributesdictionary are empty, in the other run-items all you have to declare are the (non-empty) key-value pairs that you want to use for matching your source data types.
anat: # ----------------------- All anatomical runs --------------------
- provenance: # The fullpath name of the DICOM file from which the attributes are read. Serves also as a look-up key to find a run in the bidsmap
properties: &fileattr # This is an optional (stub) entry of filesystem matching (could be added to any run-item)
filepath: # File folder, e.g. ".*Parkinson.*" or ".*(phantom|bottle).*"
filename: # File name, e.g. ".*fmap.*" or ".*(fmap|field.?map|B0.?map).*"
filesize: # File size, e.g. "2[4-6]\d MB" for matching files between 240-269 MB
nrfiles: # Number of files in the folder that match the above criteria, e.g. "5/d/d" for matching a number between 500-599
attributes: &anat_dicomattr # An empty / non-matching "reference" dictionary that can be derefenced in other run-items of this data type
Modality:
ProtocolName:
SeriesDescription:
ImageType:
SequenceName:
SequenceVariant:
ScanningSequence:
MRAcquisitionType:
SliceThickness:
FlipAngle:
EchoNumbers:
EchoTime:
RepetitionTime:
InPlanePhaseEncodingDirection:
bids: &anat_dicoment_nonparametric # See: schema/datatypes/anat.yaml
acq: <SeriesDescription> # This will be expanded by the bidsmapper (so the user can edit it in the bidseditor)
ce:
rec:
run: <<1>> # This will be updated dynamically during bidscoiner runtime (as it depends on the already existing files)
part: ['', 'mag', 'phase', 'real', 'imag', 0] # This BIDS value list will be shown as a dropdown menu in the bidseditor with the first (empty) item selected (as indicated by the last item, i.e. 0)
suffix: T1w
meta: # This is an optional entry for meta-data that will be appended to the json sidecar files produced by dcm2niix
- provenance:
properties:
<<: *fileattr
nrfiles: [1-3]/d/d # Number of files in the folder that match the above criteria, e.g. "5/d/d" for matching a number between 500-599
attributes:
<<: *anat_dicomattr
ProtocolName: '(?i).*(MPRAGE|T1w).*'
MRAcquisitionType: '3D'
bids: *anat_dicoment_nonparametric
meta:
Comments: <<ImageComments>> # This will be expanded dynamically during bidscoiner runtime (as it may vary from session to session)
- provenance:
attributes:
<<: *anat_dicomattr
ProtocolName: '(?i).*T2w.*'
SequenceVariant: '[''SK'', ''SP'']' # NB: Uses a yaml single-quote escape
bids:
<<: *anat_dicoment_nonparametric
suffix: T2w
Snippet derived from the bidsmap_dccn template, showing a “DICOM” section with a void “anat” run-item and two normal run-items that dereference the first run-item (e.g. the &anatattributes_dicom anchor is dereferenced with the <<: *anatattributes_dicom alias)
Options¶
BIDScoin has different options and settings (see below) that can be adjusted per study bidsmap or, when you want to customize the default, set as default in the template bidsmap. There are separate settings for BIDScoin and for the individual plugins that can be edited by double clicking the corresponding fields. Installed plugins can be removed or added to extend BIDScoin’s functionality.
The bidseditor options window with the different settings for BIDScoin and its plugins. The user can manage the plugins that will be used with the [Add] (as shown here) and [Remove] buttons, and save the current options to the template bidsmap by using the [Set as default] button.¶
BIDScoin¶
These setting can be used by all the BIDScoin tools:
version: Used to check for version conflicts between the installed version (see ../bidscoin/version.txt) and the version that was used to create the bidsmap, or between the installed version and the latest online version.bidsignore: Semicolon-separated list of (non-BIDS) datatypes that you want to include but that do not pass a BIDS validation test. These files are added to the.bidsignorefile. Example:bidsignore: extra_data/;rTMS/;myfile.txt;yourfile.csvsubprefix: The prefix before the subject label in the source data folder, e.g. ‘patient-‘ if the source data is inraw/patient-001/ses-01/..sesprefix: Idem for the session labeldatatypes: Datatypes that are converted to BIDS. This can be useful for ignoring / excluding specific datatypes (without changing their mappings)unknowntypes: Datatypes that are not part of BIDS but that are converted to a BIDS-like entries in the BIDS folderignoretypes: Datatypes that are excluded / not converted”””zip: Wildcard pattern to select tarball/zip-files in the source folders that need to be unzipped (in a tempdir) to expose the data. Use for instance ‘*.tar.gz’ if your source data looks like sub-0101_MPRAGEdcmfiles.tar.gz, etc
The core working of BIDScoin and its plugins can be tested by clicking the corresponding [Test] button and inspection of the terminal output.
dcm2niix2bids - plugin¶
The dcm2niix2bids plugin is the default bidscoiner plugin that converts DICOM and PAR/REC data to BIDS. It relies on dcm2niix, for which you can set the following options:
command: Command to run dcm2niix from the terminal, such as:dcm2niix(if the executable is already present on your path)module add dcm2niix/v1.0.20210317; dcm2niix(if you use a module system)PATH=/opt/dcm2niix/bin:$PATH; dcm2niix(prepend the path to your executable)/opt/dcm2niix/bin/dcm2niix(specify the fullpath to the executable)C:\"Program Files"\dcm2niix\dcm2niix.exe(use quotes to deal with whitespaces in your fullpath)
args: Argument string that is passed as input to dcm2niix to customize its behavior, e.g.-z n -i yfor ignoring derived data and having uncompressed output data.anon: Set this anonymization flag to ‘y’ to round off age and to discard acquisition date from the meta datameta: The file extensions of the associated / equally named (meta)data sourcefiles that are copied over as BIDS (sidecar) files, such as['.json', '.tsv', '.tsv.gz']. You can use this to enrich json sidecar files or add data that is not supported by this plugin. For instance, with each PET DICOM image you can put a small json file with key-value pairs that are not contained in the DICOM header (such as{InjectedRadioactivity: 400, InjectedMass: 10}). NB: Data entered in the meta table of the bidseditor GUI always has priority over data in source json files, which itself has priority over dcm2niix-generated json data.
Tip
Use the [Set as default] button to put your custom dcm2niix command in your template bidsmap so that you don’t have to adjust it anymore for every new study
SPM users may want to use ‘-z n’, which produces unzipped NIfTI’s
Install
pigzto speed-up dm2niix (search the web for more help)
spec2nii2bids - plugin¶
The spec2nii2bids plugin is an optional bidscoiner plugin for spectroscopy data. It relies on spec2nii, for which you can set the following options:
command: Command to run spec2nii, such asspec2nii(see the dcm2niix plugin for more examples to set the path)args: Argument string that is passed as input to spec2nii to customize its behavioranon: Set this anonymization flag to ‘y’ to round off age and to discard acquisition date from the meta datameta: The file extensions of the associated / equally named (meta)data sourcefiles that are copied over as BIDS (sidecar) files, such as['.json', '.tsv', '.tsv.gz']. You can use this to enrich json sidecar files or add data that is not supported by this plugin. NB: Data entered in the meta table of the bidseditor GUI always has priority over data in source json files, which itself has priority over dcm2niix-generated json data.multiraid: The mapVBVD argument for selecting the multiraid Twix file to load (default = 2, i.e. 2nd file)
pet2bids - plugin¶
The pet2bids plugin is an optional bidscoiner plugin for PET data that is still under development. It relies on PET2BIDS, for which you can set the following options:
command: Command to run pet2bids, such asdcm2niix4pet(see the dcm2niix plugin for more examples to set the path)args: Reserved for future releasesanon: Set this anonymization flag to ‘y’ to round off age and to discard acquisition date from the meta datameta: The file extensions of the associated / equally named (meta)data sourcefiles that are copied over as BIDS (sidecar) files, such as['.json', '.tsv', '.xls', '.xlsx']. You can use this to enrich json sidecar files or add data that is not supported by this plugin. NB: Data entered in the meta table of the bidseditor GUI always has priority over data in source json files, which itself has priority over dcm2niix-generated json data.
nibabel2bids - plugin¶
The nibabel2bids plugin is an optional bidscoiner plugin that converts the wide variety of nibabel datatypes to BIDS. The following options can be set:
ext: The (nibabel) file extension of the output data, i.e..nii.gzor.niimeta: The file extensions of the associated / equally named (meta)data sourcefiles that are copied over as BIDS (sidecar) files, such as['.json', '.tsv', '.tsv.gz', '.bval', '.bvec']. You can use this to enrich json sidecar files or add data that is not supported by this plugin. For instance, in this way you can still convert a NIfTI dataset that was previously created with dcm2niix to BIDS. NB: Data entered in the meta table of the bidseditor GUI always has priority over data in source json files, which itself has priority over dcm2niix-generated json data.
Note
Typically, nibabel2bids does not produce any json sidecar files, so as a user you need to provide for that yourself. You can look up the fields required by the BIDS specification and enter that information in the meta data tables of the bidseditor or put it in json files next to your source data.
Plugins¶
As shown in the figure below, all interactions of BIDScoin routines with source data are done via a plugin layer that abstracts away differences between source data formats. The bidsmapper and bidscoiner tools loop over the subjects/sessions in your source data repository and then use the plugins that are listed in the bidsmap to do the actual work.
The BIDScoin architecture and dataflow, showing different layers of abstraction. The BIDScoin layer interacts with the plugins using a single programming interface (API), which in turn interact with the source data in a dataformat dependent way. The BIDScoin layer also interacts with the metadata layer, where all prior knowledge and mapping information is stored.¶
You can use the bidscoin utility to list, install or uninstall BIDScoin plugins, but the following plugins come pre-installed:
Dcm2niix2bids: a plugin for DICOM and PAR/XML data¶
The ‘dcm2niix2bids’ plugin is a wrapper around the well-known pydicom, nibabel and (in particular) dcm2niix tools to interact with and convert DICOM and Philips PAR(/REC)/XML source data. Pydicom is used to read DICOM attributes, nibabel is used to read PAR/XML attribute values and dcm2niix is used to convert the DICOM and PAR/XML source data to NIfTI and create BIDS sidecar files. Personal data from the source header (e.g. Age, Sex) is added to the BIDS participants.tsv file.
Spec2nii2bids: a plugin for MR spectroscopy data¶
The ‘spec2nii2bids’ plugin is a wrapper around the recent spec2nii Python library to interact with and convert MR spectroscopy source data. Presently, the spec2nii2bids plugin is a first implementation that supports the conversion of Philips SPAR/SDAT files, Siemens Twix files and GE P-files to NIfTI, in conjunction with BIDS sidecar files.
Nibabel2bids: a generic plugin for imaging data¶
The nibabel2bids plugin wraps around the flexible nibabel tool to convert a wide variety of data formats into NIfTI-files. Currently, the default template bidsmap is tailored to NIfTI source data only (but this can readily be extended), and BIDS sidecar files are not automatically produced by nibabel (but see the note further below).
Pet2bids: a plugin for PET data¶
The ‘pet2bids’ plugin is a wrapper around the PET2BIDS tool. PET2BIDS accepts PET imaging and blood data as inputs (e.g. DICOM, ECAT, spreadsheets) and delivers BIDS formatted outputs. An installation of dcm2niix (https://github.com/rordenlab/dcm2niix) is required to convert DICOM data.
Note
Out of the box, BIDScoin plugins typically produce sidecar files that contain metadata from the source headers. However, when such meta-data is missing (e.g. as for nibabel2bids), or when it needs to be appended or overruled, then users can add sidecar files to the source data (as explained here) or add that meta-data using the bidseditor (the latter takes precedence).
The plugin programming interface¶
This paragraph describes the requirements and structure of plugins in order to allow advanced users and developers to write their own plugin and extent or customize BIDScoin to their needs.
The main task of a plugin is to perform the actual conversion of the source data into a format that is part of the BIDS standard. BIDScoin offers the library module named ‘bids’ to interact with bidsmaps and to provide the intended output names and meta data. Moreover, the bids library contains a class named DataSource() that is (strongly) recommended to be used for reading the source data attributes and properties. In this way, the extended source data attributes and dynamic values (including regular expressions) are handled transparently, ensuring consistent results throughout BIDScoin.
However, if you have a long list of extended attributes it can be cumbersome to put them all in the metadata table (even though you can right-click to import data there). This is where the copymetadata() function comes in. This is meant to be used by the plugins during bidscoiner runtime, and just transfers all extra source metadata to the BIDS folder (for each plugin, the user can control this if needed in the bidseditor by setting the allowed file extensions of the extra source metadata files). A scenario where copymetadata is very useful is e.g. when you have a non-BIDS nifti dataset that was created using dcm2niix and now you want to convert that to BIDS (in that case you just want to copy over all the metadata in the json files). The nibabel2bids plugin can do exactly that. NB: copymetadata is still a very rudimentary function (I never needed it myself so I didn’t develop it very much)
If a field is present in the metadata table and also in the metadata transferred by copymetadata, then the metadata table should take precedence (this should be implemented as such in the plugin), so that the user gets what he sees in the bidseditor (WYSIWYG).
As can be seen in the API code snippet below (but also see the default plugins for reference implementation), a BIDScoin plugin is a Python module with the following programming interface (functions):
"""
This module contains placeholder code demonstrating the bidscoin plugin API, both for the bidsmapper and for
the bidscoiner. The functions in this module are called if the basename of this module (when located in the
plugins-folder; otherwise the full path must be provided) is listed in the bidsmap. The following plugin functions
are expected to be present:
- test: A test function for the plugin + its bidsmap options. Can be called by the user from the bidseditor and the bidscoin utility
- is_sourcefile: A function to assess whether a source file is supported by the plugin. The return value should correspond to a data format section in the bidsmap
- get_attribute: A function to read an attribute value from a source file
- bidsmapper_plugin: A function to discover BIDS-mappings in a source data session
- bidscoiner_plugin: A function to convert a single source data session to bids according to the specified BIDS-mappings
To avoid code duplications and minimize plugin development time, various support functions are available in
BIDScoin's library modules named 'bidscoin' and, most notably, 'bids'
"""
import logging
from pathlib import Path
LOGGER = logging.getLogger(__name__)
# The default options that are set when installing the plugin
OPTIONS = {'command': 'demo', # Plugin option
'args': 'foo bar'} # Another plugin option
# The default bids-mappings that are added when installing the plugin
BIDSMAP = {'DemoFormat':{
'subject': '<<filepath:/sub-(.*?)/>>', # This filesystem property extracts the subject label from the source directory. NB: Any property or attribute can be used, e.g. <PatientID>
'session': '<<filepath:/sub-.*?/ses-(.*?)/>>', # This filesystem property extracts the session label from the source directory. NB: Any property or attribute can be used, e.g. <StudyID>
'func': [ # ----------------------- All functional runs --------------------
{'provenance': '', # The fullpath name of the source file from which the attributes and properties are read. Serves also as a look-up key to find a run in the bidsmap
'properties': # The matching (regexp) criteria go in here
{'filepath': '', # File folder, e.g. ".*Parkinson.*" or ".*(phantom|bottle).*"
'filename': '', # File name, e.g. ".*fmap.*" or ".*(fmap|field.?map|B0.?map).*"
'filesize': '', # File size, e.g. "2[4-6]\d MB" for matching files between 240-269 MB
'nrfiles': ''}, # Number of files in the folder that match the above criteria, e.g. "5/d/d" for matching a number between 500-599
'attributes': # The matching (regexp) criteria go in here
{'ch_num': '.*',
'filetype': '.*',
'freq': '.*',
'ch_name': '.*',
'units': '.*',
'trigger_idx': '.*'},
'bids':
{'task': '',
'acq': '',
'ce': '',
'dir': '',
'rec': '',
'run': '<<1>>', # This will be updated during bidscoiner runtime (as it depends on the already existing files)
'recording': '',
'suffix': 'physio'},
'meta': # This is an optional entry for meta-data dictionary that are appended to the json sidecar files
{'TriggerChannel': '<<trigger_idx>>',
'ExpectedTimepoints': '<<num_timepoints_found>>',
'ChannelNames': '<<ch_name>>',
'Threshold': '<<thr>>',
'TimeOffset': '<<time_offset>>'}}],
[...]
'exclude': [ # ----------------------- Data that will be left out -------------
{'provenance': '',
'properties':
{'filepath': '',
'filename': '',
'filesize': '',
'nrfiles': ''},
'attributes':
{'ch_num': '.*',
'filetype': '.*',
'freq': '.*',
'ch_name': '.*',
'units': '.*',
'trigger_idx': '.*'},
'bids':
{'task': '',
'acq': '',
'ce': '',
'dir': '',
'rec': '',
'run': '<<1>>',
'recording': '',
'suffix': 'physio'},
'meta':
{'TriggerChannel': '<<trigger_idx>>',
'ExpectedTimepoints': '<<num_timepoints_found>>',
'ChannelNames': '<<ch_name>>',
'Threshold': '<<thr>>',
'TimeOffset': '<<time_offset>>'}}]}}
def test(options: dict=OPTIONS) -> bool:
"""
Performs a runtime/integration test of the working of the plugin + its bidsmap options
:param options: A dictionary with the plugin options, e.g. taken from the bidsmap['Options']['plugins']['README']
:return: The errorcode (e.g 0 if the tool generated the expected result, > 0 if there was a tool error)
"""
LOGGER.info(f'This is a demo-plugin test routine, validating its working with options: {options}')
return 0
def is_sourcefile(file: Path) -> str:
"""
This plugin function assesses whether a sourcefile is of a supported dataformat
:param file: The sourcefile that is assessed
:return: The valid / supported dataformat of the sourcefile
"""
if file.is_file():
LOGGER.verbose(f'This is a demo-plugin is_sourcefile routine, assessing whether "{file}" has a valid dataformat')
return 'dataformat' if file == 'supportedformat' else ''
return ''
def get_attribute(dataformat: str, sourcefile: Path, attribute: str, options: dict) -> str:
"""
This plugin function reads attributes from the supported sourcefile
:param dataformat: The bidsmap-dataformat of the sourcefile, e.g. DICOM of PAR
:param sourcefile: The sourcefile from which the attribute value should be read
:param attribute: The attribute key for which the value should be read
:param options: A dictionary with the plugin options, e.g. taken from the bidsmap['Options']
:return: The attribute value
"""
if dataformat in ('DICOM','PAR'):
LOGGER.verbose(f'This is a demo-plugin get_attribute routine, reading the {dataformat} "{attribute}" attribute value from "{sourcefile}"')
return ''
def bidsmapper_plugin(session: Path, bidsmap_new: dict, bidsmap_old: dict, template: dict, store: dict) -> None:
"""
All the logic to map the Philips PAR/XML fields onto bids labels go into this plugin function. The function is
expecte to update / append new runs to the bidsmap_new data structure. The bidsmap options for this plugin can
be found in:
bidsmap_new/old['Options']['plugins']['README']
See also the dcm2niix2bids plugin for reference implementation
:param session: The full-path name of the subject/session raw data source folder
:param bidsmap_new: The new study bidsmap that we are building
:param bidsmap_old: The previous study bidsmap that has precedence over the template bidsmap
:param template: The template bidsmap with the default heuristics
:param store: The paths of the source- and target-folder
:return:
"""
LOGGER.verbose(f'This is a bidsmapper demo-plugin working on: {session}')
def bidscoiner_plugin(session: Path, bidsmap: dict, bidsses: Path) -> None:
"""
The plugin to convert the runs in the source folder and save them in the bids folder. Each saved datafile should be
accompanied by a json sidecar file. The bidsmap options for this plugin can be found in:
bidsmap_new/old['Options']['plugins']['README']
See also the dcm2niix2bids plugin for reference implementation
:param session: The full-path name of the subject/session source folder
:param bidsmap: The full mapping heuristics from the bidsmap YAML-file
:param bidsses: The full-path name of the BIDS output 'ses-' folder
:return: Nothing
"""
LOGGER.debug(f'This is a bidscoiner demo-plugin working on: {session} -> {bidsfolder}')
The README plugin placeholder code
BIDS-apps¶
Multi-echo combination¶
Before sharing or pre-processing their images, users may want to combine the separate the individual echos of multi-echo MRI acquisitions. The echcombine-tool is a wrapper around mecombine that writes BIDS valid output.
usage: echocombine [-h] [-p PARTICIPANT_LABEL [PARTICIPANT_LABEL ...]] [-o OUTPUT]
[-a {PAID,TE,average}] [-w [WEIGHTS ...]] [-f]
bidsfolder pattern
A wrapper around the 'mecombine' multi-echo combination tool
(https://github.com/Donders-Institute/multiecho).
Except for BIDS inheritances, this wrapper is BIDS-aware (a 'bidsapp') and writes BIDS
compliant output
positional arguments:
bidsfolder The bids-directory with the (multi-echo) subject data
pattern Globlike recursive search pattern (relative to the subject/session
folder) to select the first echo of the images that need to be
combined, e.g. '*task-*echo-1*'
optional arguments:
-h, --help show this help message and exit
-p PARTICIPANT_LABEL [PARTICIPANT_LABEL ...], --participant_label PARTICIPANT_LABEL [PARTICIPANT_LABEL ...]
Space separated list of sub-# identifiers to be processed (the sub-
prefix can be left out). If not specified then all sub-folders in the
bidsfolder will be processed (default: None)
-o OUTPUT, --output OUTPUT
A string that determines where the output is saved. It can be the
name of a BIDS datatype folder, such as 'func', or of the derivatives
folder, i.e. 'derivatives'. If output = [the name of the input
datatype folder] then the original echo images are replaced by one
combined image. If output is left empty then the combined image is
saved in the input datatype folder and the original echo images are
moved to the extra_data folder (default: )
-a {PAID,TE,average}, --algorithm {PAID,TE,average}
Combination algorithm (default: TE)
-w [WEIGHTS ...], --weights [WEIGHTS ...]
Weights for each echo (default: None)
-f, --force If this flag is given subjects will be processed, regardless of
existing target files already exist. Otherwise the echo-combination
will be skipped (default: False)
examples:
echocombine myproject/bids func/*task-stroop*echo-1*
echocombine myproject/bids *task-stroop*echo-1* -p 001 003
echocombine myproject/bids func/*task-*echo-1* -o func
echocombine myproject/bids func/*task-*echo-1* -o derivatives -w 13 26 39 52
echocombine myproject/bids func/*task-*echo-1* -a PAID
Defacing¶
Before sharing or pre-processing their images, users may want to deface their anatomical MRI acquisitions to protect the privacy of their subjects. The deface-tool is a wrapper around pydeface that writes BIDS valid output. NB: pydeface requires FSL to be installed on the system.
usage: deface [-h] [-p PARTICIPANT_LABEL [PARTICIPANT_LABEL ...]] [-o OUTPUT] [-c]
[-n NATIVESPEC] [-a ARGS] [-f]
bidsfolder pattern
A wrapper around the 'pydeface' defacing tool (https://github.com/poldracklab/pydeface).
Except for BIDS inheritances and IntendedFor usage, this wrapper is BIDS-aware (a 'bidsapp')
and writes BIDS compliant output
Linux users can distribute the computations to their HPC compute cluster if the DRMAA
libraries are installed and the DRMAA_LIBRARY_PATH environment variable set
For multi-echo data see ``medeface``
positional arguments:
bidsfolder The bids-directory with the subject data
pattern Globlike search pattern (relative to the subject/session folder) to
select the images that need to be defaced, e.g. 'anat/*_T1w*'
optional arguments:
-h, --help show this help message and exit
-p PARTICIPANT_LABEL [PARTICIPANT_LABEL ...], --participant_label PARTICIPANT_LABEL [PARTICIPANT_LABEL ...]
Space separated list of sub-# identifiers to be processed (the sub-
prefix can be left out). If not specified then all sub-folders in the
bidsfolder will be processed (default: None)
-o OUTPUT, --output OUTPUT
A string that determines where the defaced images are saved. It can
be the name of a BIDS datatype folder, such as 'anat', or of the
derivatives folder, i.e. 'derivatives'. If output is left empty then
the original images are replaced by the defaced images (default:
None)
-c, --cluster Flag to use the DRMAA library to submit the deface jobs to a high-
performance compute (HPC) cluster (default: False)
-n NATIVESPEC, --nativespec NATIVESPEC
DRMAA native specifications for submitting deface jobs to the HPC
cluster (default: -l walltime=00:30:00,mem=2gb)
-a ARGS, --args ARGS Additional arguments (in dict/json-style) that are passed to
pydeface. See examples for usage (default: {})
-f, --force If this flag is given images will be processed, regardless if images
have already been defaced (i.e. if {"Defaced": True} in the json
sidecar file) (default: False)
examples:
deface myproject/bids anat/*_T1w*
deface myproject/bids anat/*_T1w* -p 001 003 -o derivatives
deface myproject/bids anat/*_T1w* -c -n "-l walltime=00:60:00,mem=4gb"
deface myproject/bids anat/*_T1w* -a '{"cost": "corratio", "verbose": ""}'
Multi-echo defacing¶
This utility is very similar to the deface utility above, except that it can handle multi-echo data.
usage: medeface [-h] [-m MASKPATTERN] [-p PARTICIPANT_LABEL [PARTICIPANT_LABEL ...]]
[-o OUTPUT] [-c] [-n NATIVESPEC] [-a ARGS] [-f]
bidsfolder pattern
A wrapper around the 'pydeface' defacing tool (https://github.com/poldracklab/pydeface) that
computes a defacing mask on a (temporary) echo-combined image and then applies it to each
individual echo-image.
Except for BIDS inheritances and IntendedFor usage, this wrapper is BIDS-aware (a 'bidsapp')
and writes BIDS compliant output
Linux users can distribute the computations to their HPC compute cluster if the DRMAA
libraries are installed and the DRMAA_LIBRARY_PATH environment variable set
For single-echo data see ``deface``
positional arguments:
bidsfolder The bids-directory with the (multi-echo) subject data
pattern Globlike search pattern (relative to the subject/session folder) to
select the images that need to be defaced, e.g. 'anat/*_T2starw*'
optional arguments:
-h, --help show this help message and exit
-m MASKPATTERN, --maskpattern MASKPATTERN
Globlike search pattern (relative to the subject/session folder) to
select the images from which the defacemask is computed, e.g.
'anat/*_part-mag_*_T2starw*'. If not given then 'pattern' is used
(default: None)
-p PARTICIPANT_LABEL [PARTICIPANT_LABEL ...], --participant_label PARTICIPANT_LABEL [PARTICIPANT_LABEL ...]
Space separated list of sub-# identifiers to be processed (the sub-
prefix can be left out). If not specified then all sub-folders in the
bidsfolder will be processed (default: None)
-o OUTPUT, --output OUTPUT
A string that determines where the defaced images are saved. It can
be the name of a BIDS datatype folder, such as 'anat', or of the
derivatives folder, i.e. 'derivatives'. If output is left empty then
the original images are replaced by the defaced images (default:
None)
-c, --cluster Flag to submit the deface jobs to a high-performance compute (HPC)
cluster (default: False)
-n NATIVESPEC, --nativespec NATIVESPEC
DRMAA native specifications for submitting deface jobs to the HPC
cluster (default: -l walltime=00:30:00,mem=2gb)
-a ARGS, --args ARGS Additional arguments (in dict/json-style) that are passed to
pydeface. See examples for usage (default: {})
-f, --force If this flag is given images will be processed, regardless if images
have already been defaced (i.e. if {"Defaced": True} in the json
sidecar file) (default: False)
examples:
medeface myproject/bids anat/*_T1w*
medeface myproject/bids anat/*_T1w* -p 001 003 -o derivatives
medeface myproject/bids anat/*_T1w* -c -n "-l walltime=00:60:00,mem=4gb"
medeface myproject/bids anat/*acq-GRE* -m anat/*acq-GRE*magnitude*"
medeface myproject/bids anat/*_FLAIR* -a '{"cost": "corratio", "verbose": ""}'
Skull-stripping¶
The skullstrip-tool is a wrapper around the synthstrip tool that writes BIDS valid output
usage: skullstrip [-h] [-p PARTICIPANT_LABEL [PARTICIPANT_LABEL ...]] [-m MASKED]
[-o OUTPUT [OUTPUT ...]] [-f] [-a ARGS]
bidsfolder pattern
A wrapper around FreeSurfer's 'synthstrip' skull stripping tool
(https://surfer.nmr.mgh.harvard.edu/docs/synthstrip). Except for BIDS inheritances,
this wrapper is BIDS-aware (a 'bidsapp') and writes BIDS compliant output
The corresponding brain mask is saved in the bids/derivatives/synthstrip folder
Assumes the installation of FreeSurfer v7.3.2 or higher
positional arguments:
bidsfolder The bids-directory with the subject data
pattern Globlike search pattern (relative to the subject/session folder) to
select the (3D) images that need to be skullstripped, e.g.
'anat/*_T1w*'
optional arguments:
-h, --help show this help message and exit
-p PARTICIPANT_LABEL [PARTICIPANT_LABEL ...], --participant_label PARTICIPANT_LABEL [PARTICIPANT_LABEL ...]
Space separated list of sub-# identifiers to be processed (the sub-
prefix can be left out). If not specified then all sub-folders in the
bidsfolder will be processed (default: None)
-m MASKED, --masked MASKED
Globlike search pattern (relative to the subject/session folder) to
select additional (3D/4D) images from the same space that need to be
masked with the same mask, e.g. 'fmap/*_phasediff'. NB: This option
can only be used if pattern yields a single file per session (default:
None)
-o OUTPUT [OUTPUT ...], --output OUTPUT [OUTPUT ...]
One or two output strings that determine where the skullstripped +
additional masked images are saved. Each output string can be the
name of a BIDS datatype folder, such as 'anat', or of the derivatives
folder, i.e. 'derivatives' (default). If the output string is the
same as the datatype then the original images are replaced by the
skullstripped images (default: None)
-f, --force Process images, regardless whether images have already been
skullstripped (i.e. if {'SkullStripped': True} in the json sidecar
file) (default: False)
-a ARGS, --args ARGS Additional arguments that are passed to synthstrip (NB: Use quotes
and a leading space to prevent unintended argument parsing)
-c, --cluster Use `qsub` to submit the skullstrip jobs to a high-performance compute
(HPC) cluster. Can only be used if `--masked` is left empty (default: False)
examples:
skullstrip myproject/bids anat/*_T1w*
skullstrip myproject/bids anat/*_T1w* -p 001 003 -a ' --no-csf'
skullstrip myproject/bids fmap/*_magnitude1* -m fmap/*_phasediff -o extra_data fmap
skullstrip myproject/bids fmap/*_acq-mylabel*_magnitude1* -m fmap/*_acq-mylabel_* -o fmap
Quality control¶
Slicereport is a very flexible QC report generator for doing visual inspections on your BIDS data.
usage: slicereport.py [-h] [-o OUTLINEPATTERN] [-i OUTLINEIMAGE]
[-p PARTICIPANT_LABEL [PARTICIPANT_LABEL ...]] [-r REPORTFOLDER]
[-q QCSCORES [QCSCORES ...]] [-c] [--options OPTIONS [OPTIONS ...]]
[--outputs OUTPUTS [OUTPUTS ...]] [--suboptions SUBOPTIONS [SUBOPTIONS ...]]
[--suboutputs SUBOUTPUTS [SUBOUTPUTS ...]]
bidsfolder pattern
A wrapper around the 'slicer' imaging tool (https://fsl.fmrib.ox.ac.uk/fsl/fslwiki/Miscvis)
to generate a web page with a row of image slices for each subject in the BIDS repository, as
well as individual sub-pages displaying more detailed information. The input images are
selectable using wildcards, and the output images are configurable via various user options,
allowing you to quickly create a custom 'slicer' report to do visual quality control on any
datatype in your repository.
Requires an existing installation of FSL/slicer
positional arguments:
bidsfolder The bids-directory with the subject data
pattern Globlike search pattern to select the images in bidsfolder to be reported,
e.g. 'anat/*_T2starw*'
options:
-h, --help show this help message and exit
-o OUTLINEPATTERN, --outlinepattern OUTLINEPATTERN
Globlike search pattern to select red outline images that are projected on top
of the reported images (i.e. 'outlinepattern' must yield the same number of
images as 'pattern'. Prepend `outlinedir:` if your outline images are in
`outlinedir` instead of `bidsdir` (see examples below)`
-i OUTLINEIMAGE, --outlineimage OUTLINEIMAGE
A common red-outline image that is projected on top of all images
-p PARTICIPANT_LABEL [PARTICIPANT_LABEL ...], --participant_label PARTICIPANT_LABEL [PARTICIPANT_LABEL ...]
Space separated list of sub-# identifiers to be processed (the sub-prefix can
be left out). If not specified then all sub-folders in the bidsfolder will be
processed
-r REPORTFOLDER, --reportfolder REPORTFOLDER
The folder where the report is saved (default:
bidsfolder/derivatives/slicereport)
-x XLINKFOLDER [XLINKFOLDER ...], --xlinkfolder XLINKFOLDER [XLINKFOLDER ...]
A (list of) QC report folder(s) with cross-linkable sub-reports, e.g.
bidsfolder/derivatives/mriqc
-q QCSCORES [QCSCORES ...], --qcscores QCSCORES [QCSCORES ...]
Column names for creating an accompanying tsv-file to store QC-rating scores
(default: rating_overall)
-c, --cluster Use `qsub` to submit the slicer jobs to a high-performance compute (HPC) cluster
--options OPTIONS [OPTIONS ...]
Main options of slicer (see below). (default: "s 1")
--outputs OUTPUTS [OUTPUTS ...]
Output options of slicer (see below). (default: "x 0.4 x 0.5 x 0.6 y 0.4 y 0.5
y 0.6 z 0.4 z 0.5 z 0.6")
--suboptions SUBOPTIONS [SUBOPTIONS ...]
Main options of slicer for creating the sub-reports (same as OPTIONS, see
below). (default: OPTIONS)
--suboutputs SUBOUTPUTS [SUBOUTPUTS ...]
Output options of slicer for creating the sub-reports (same as OUTPUTS, see
below). (default: "S 4 1600")
OPTIONS:
L : Label slices with slice number.
l [LUT] : Use a different colour map from that specified in the header.
i [MIN] [MAX] : Specify intensity min and max for display range.
e [THR] : Use the specified threshold for edges (if > 0 use this proportion of max-min,
if < 0, use the absolute value)
t : Produce semi-transparent (dithered) edges.
n : Use nearest-neighbour interpolation for output.
u : Do not put left-right labels in output.
s : Size scaling factor
c : Add a red dot marker to top right of image
OUTPUTS:
x/y/z [SLICE] [..] : Output sagittal, coronal or axial slice (if [SLICE] > 0 it is a
fraction of image dimension, if < 0, it is an absolute slice number)
a : Output mid-sagittal, -coronal and -axial slices into one image
A [WIDTH] : Output _all_ axial slices into one image of _max_ width [WIDTH]
S [SAMPLE] [WIDTH] : As `A` but only include every [SAMPLE]'th slice
LF : Start a new line (i.e. works like a row break)
examples:
slicereport myproject/bids anat/*_T1w*
slicereport myproject/bids anat/*_T2w* -r myproject/QC/slicereport_T2 -x myproject/QC/slicereport_T1
slicereport myproject/bids fmap/*_phasediff* -o fmap/*_magnitude1*
slicereport myproject/bids/derivatives/fmriprep anat/*run-?_desc-preproc_T1w* -o anat/*run-?_label-GM*
slicereport myproject/bids/derivatives/deface anat/*_T1w* -o myproject/bids:anat/*_T1w* --options L e 0.05
slicereport myproject/bids anat/*_T1w* --outputs x 0.3 x 0.4 x 0.5 x 0.6 x 0.7 LF z 0.3 z 0.4 z 0.5 z 0.6 z 0.7
Snippet of a slicereport for doing quality control on skullstrip output images (see above). The
background image shows the skull-stripped image in the extra_data folder, and the red outline image
on top shows the contours of the original image in the anat folder. Users can click on an image to
navigate to the individual (more detailed) slicereport of that subject. This example can be generated
from scratch with just two commands:¶
$ skullstrip bids anat/*run-1_T1w* -o extra_data
$ slicereport bids extra_data/*run-1_T1w* -o anat/*run-1_T1w*
Utilities¶
bidscoin¶
The bidscoin command-line utility serves as a central starting point to test and manage your BIDScoin installation
usage: bidscoin [-h] [-l] [-p] [-i INSTALL [INSTALL ...]] [-u UNINSTALL [UNINSTALL ...]]
[-d DOWNLOAD] [-t [TEST]] [-b BIDSMAPTEST] [-v]
BIDScoin is a toolkit to convert and organize raw data-sets according to the Brain Imaging
Data Structure (BIDS)
The basic workflow is to run these two tools:
$ bidsmapper sourcefolder bidsfolder # This produces a study bidsmap and launches a GUI
$ bidscoiner sourcefolder bidsfolder # This converts your data to BIDS according to the study bidsmap
Set the environment variable BIDSCOIN_DEBUG=TRUE in your console to run BIDScoin in it's more
verbose DEBUG logging mode
For more documentation see: https://bidscoin.readthedocs.io
optional arguments:
-h, --help show this help message and exit
-l, --list List all executables (i.e. the apps, bidsapps and utilities)
-p, --plugins List all installed plugins and template bidsmaps
-i INSTALL [INSTALL ...], --install INSTALL [INSTALL ...]
A list of template bidsmaps and/or bidscoin plugins to install
-u UNINSTALL [UNINSTALL ...], --uninstall UNINSTALL [UNINSTALL ...]
A list of template bidsmaps and/or bidscoin plugins to uninstall
-d DOWNLOAD, --download DOWNLOAD
Download folder. If given, tutorial MRI data will be downloaded here
-t [TEST], --test [TEST]
Test the bidscoin installation and template bidsmap
-b BIDSMAPTEST, --bidsmaptest BIDSMAPTEST
Test the run-items and their bidsnames of all normal runs in the
study bidsmap. Provide the bids-folder or the bidsmap filepath
-v, --version Show the installed version and check for updates
examples:
bidscoin -l
bidscoin -d data/bidscoin_tutorial
bidscoin -t
bidscoin -t my_template_bidsmap
bidscoin -b my_study_bidsmap
bidscoin -i data/my_template_bidsmap.yaml downloads/my_plugin.py
dicomsort¶
The dicomsort command-line tool is a utility to move your flat- or DICOMDIR-organized files (see above) into a ‘seriesfolder’ organization. This can be useful to organise your source data in a more convenient and human readable way (DICOMDIR or flat DICOM directories can often be hard to comprehend). The BIDScoin tools will run dicomsort in a temporary folder if your data is not already organized in series-folders, so in principle you don’t really need to run it yourself (unless when you have a single multi-subject DICOMDIR file for your entire repository). Running dicomsort beforehand does, however, give you more flexibility in handling special cases that are not handled properly and it can also give you a speed benefit. If dicomsort do not satisfy your needs, then have a look at this reorganize_dicom_files tool.
usage: dicomsort [-h] [-i SUBPREFIX] [-j SESPREFIX] [-f FOLDERSCHEME] [-n NAMESCHEME]
[-p PATTERN] [--force] [-d]
dicomsource
Sorts and/or renames DICOM files into local subfolders, e.g. with 3-digit SeriesNumber-SeriesDescription
folder names (i.e. following the same listing as on the scanner console)
Supports flat DICOM as well as multi-subject/session DICOMDIR file structures.
positional arguments:
dicomsource The root folder containing the dicomsource/[sub/][ses/] dicomfiles or
the DICOMDIR file
optional arguments:
-h, --help show this help message and exit
-i SUBPREFIX, --subprefix SUBPREFIX
Provide a prefix string for recursive sorting of dicomsource/subject
subfolders (e.g. "sub-") (default: None)
-j SESPREFIX, --sesprefix SESPREFIX
Provide a prefix string for recursive sorting of
dicomsource/subject/session subfolders (e.g. "ses-") (default: None)
-f FOLDERSCHEME, --folderscheme FOLDERSCHEME
Naming scheme for the sorted DICOM Series subfolders. Follows the
Python string formatting syntax with DICOM field names in curly
bracers with an optional number of digits for numeric fields. Sorting
in subfolders is skipped when an empty folderscheme is given (but
note that renaming the filenames can still be performed) (default:
{SeriesNumber:03d}-{SeriesDescription})
-n NAMESCHEME, --namescheme NAMESCHEME
Optional naming scheme that can be provided to rename the DICOM
files. Follows the Python string formatting syntax with DICOM field
names in curly bracers with an optional number of digits for numeric
fields. Use e.g. "{PatientName}_{SeriesNumber:03d}_{SeriesDescription
}_{AcquisitionNumber:05d}_{InstanceNumber:05d}.dcm" or
"{InstanceNumber:05d}_{SOPInstanceUID}.IMA" for default names
(default: None)
-p PATTERN, --pattern PATTERN
The regular expression pattern used in re.match(pattern, dicomfile)
to select the dicom files (default: .*\.(IMA|dcm)$)
--force Sort the DICOM data even the DICOM fields of the folder/name scheme
are not in the data (default: False)
-d, --dryrun Add this flag to just print the dicomsort commands without actually
doing anything (default: False)
examples:
dicomsort sub-011/ses-mri01
dicomsort sub-011/ses-mri01/DICOMDIR -n {AcquisitionNumber:05d}_{InstanceNumber:05d}.dcm
dicomsort myproject/raw/DICOMDIR --subprefix pat^ --sesprefix
rawmapper¶
Another command-line utility that can be helpful in organizing your source data is rawmapper. This utility can show you an overview (map) of all the values of DICOM-attributes of interest in your data-set and, optionally, used to rename your source data sub-folders. The latter option can be handy e.g. if you manually entered subject-identifiers as [Additional info] at the scanner console and you want to use these to rename your subject folders.
usage: rawmapper [-h] [-s SESSIONS [SESSIONS ...]] [-f FIELD [FIELD ...]] [-w WILDCARD]
[-o OUTFOLDER] [-r] [-c] [-n SUBPREFIX] [-m [SESPREFIX]] [-d]
sourcefolder
Maps out the values of a dicom attribute of all subjects in the sourcefolder, saves the result
in a mapper-file and, optionally, uses the dicom values to rename the sub-/ses-id's of the
subfolders. This latter option can be used, e.g. when an alternative subject id was entered in
the [Additional info] field during subject registration at the scanner console (i.e. this data
is stored in the dicom attribute named 'PatientComments')
positional arguments:
sourcefolder The source folder with the raw data in sub-#/ses-#/series
organisation
optional arguments:
-h, --help show this help message and exit
-s SESSIONS [SESSIONS ...], --sessions SESSIONS [SESSIONS ...]
Space separated list of selected sub-#/ses-# names / folders to be
processed. Otherwise all sessions in the bidsfolder will be selected
(default: None)
-f FIELD [FIELD ...], --field FIELD [FIELD ...]
The fieldname(s) of the dicom attribute(s) used to rename or map the
subid/sesid foldernames (default: ['PatientComments',
'ImageComments'])
-w WILDCARD, --wildcard WILDCARD
The Unix style pathname pattern expansion that is used to select the
series from which the dicomfield is being mapped (can contain
wildcards) (default: *)
-o OUTFOLDER, --outfolder OUTFOLDER
The mapper-file is normally saved in sourcefolder or, when using this
option, in outfolder (default: None)
-r, --rename If this flag is given sub-subid/ses-sesid directories in the
sourcefolder will be renamed to sub-dcmval/ses-dcmval (default:
False)
-c, --clobber Flag to rename the directories, even if the target-directory already
exists (default: False)
-n SUBPREFIX, --subprefix SUBPREFIX
The prefix common for all the source subject-folders. Use a '*'
wildcard if there is no prefix (default: sub-)
-m [SESPREFIX], --sesprefix [SESPREFIX]
The prefix common for all the source session-folders. Use a '*'
wildcard if there is no prefix or an empty value if there are no
sessions (default: ses-)
-d, --dryrun Add this flag to dryrun (test) the mapping or renaming of the sub-
subid/ses-sesid directories (i.e. nothing is stored on disk and
directory names are not actually changed)) (default: False)
examples:
rawmapper myproject/raw
rawmapper myproject/raw -f AcquisitionDate
rawmapper myproject/raw -s sub-100/ses-mri01 sub-126/ses-mri01
rawmapper myproject/raw -r -f ManufacturerModelName AcquisitionDate --dryrun
rawmapper myproject/raw -r -s sub-1*/* sub-2*/ses-mri01 --dryrun
rawmapper -f EchoTime -w *fMRI* myproject/raw
bidsparticipants¶
The bidsparticipants tool is useful for (re-)generating a participants.tsv file from your source data (without having to run bidscoiner)
usage: bidsparticipants [-h] [-k KEYS [KEYS ...]] [-d] [-b BIDSMAP] [-v]
sourcefolder bidsfolder
(Re)scans data sets in the source folder for subject meta data to populate the participants.tsv
file in the bids directory, e.g. after you renamed (be careful there!), added or deleted data
in the bids folder yourself.
Provenance information, warnings and error messages are stored in the
bidsfolder/code/bidscoin/bidsparticipants.log file.
positional arguments:
sourcefolder The study root folder containing the raw source data folders
bidsfolder The destination / output folder with the bids data
optional arguments:
-h, --help show this help message and exit
-k KEYS [KEYS ...], --keys KEYS [KEYS ...]
Space separated list of the participants.tsv columns. Default:
'session_id' 'age' 'sex' 'size' 'weight'
-d, --dryrun Add this flag to only print the participants info on screen
-b BIDSMAP, --bidsmap BIDSMAP
The study bidsmap file with the mapping heuristics. If the bidsmap
filename is relative (i.e. no "/" in the name) then it is assumed to
be located in bidsfolder/code/bidscoin. Default: bidsmap.yaml
-v, --version Show the BIDS and BIDScoin version
examples:
bidsparticipants myproject/raw myproject/bids
bidsparticipants myproject/raw myproject/bids -k participant_id age sex
Screenshots¶
The BIDScoin architecture¶
The BIDScoin architecture and dataflow, showing different layers of abstraction. The BIDScoin layer interacts with the plugins using a single programming interface (API), which in turn interact with the source data in a dataformat dependent way. The BIDScoin layer also interacts with the metadata layer, where all prior knowledge and mapping information is stored.¶
The bidseditor¶
The bidseditor main window with an overview of the data types in the source data (left column) with a preview of the BIDS output names (right column). The green or red color indicates whether manual editing of the BIDS-mapping is necessary, while the strikeout text indicates that the datatype will not be converted, which is useful for handling irrelevant data. The user can edit the subject and session property values if needed (session can be left empty to be omitted) and the result is immediately reflected in the preview. Different tabs represent different data formats in the source dataset, i.e. DICOM and PAR, which are represented as separate sections in the bidsmap. In addition, there is a tab to edit the study specific options and a tab in which the user can browse the organization of the source data and inspect the data.¶
The BIDS-mapping edit window featuring file name matching (.*\.IMA) and dynamic metadata values (e.g. TimeZero). BIDS values that are restricted to a limited set are presented with a drop-down menu (here labeled [Data type], [rec] and [suffix]). The user can immediately see the results of their edits in the preview of the BIDS output filename. A green filename indicates that the name is compliant with the BIDS standard, whereas a red name indicates that the user still needs to fill out one or more compulsory bids values (with a pop-up window appearing if the user ignores it). Hoovering with the mouse over features explanatory text from the BIDS schema files. Double clicking on the DICOM filename opens a new window displaying the full header information with all attributes. The user can export the customized mapping to a different bidsmap on disk.¶
The bidsmap options for BIDScoin and its plugins. The user can manage the plugins that will be used with the [Add] and [Remove] buttons, and save the current options to the template bidsmap by using the [Set as default] button.¶
Tutorial¶
BIDS introduction and BIDScoin demo¶
A good starting point to learn more about BIDS and BIDScoin is to watch this presentation from the OpenMR Benelux 2020 meeting (slides). The first 14 minutes Robert Oostenveld provides a general overview of the BIDS standard, after which Marcel Zwiers presents the design of BIDScoin and demonstrates hands-on how you can use it to convert a dataset to BIDS.
BIDScoin tutorial¶
The tutorial below was written for an older version of BIDScoin with the DCCN user in mind. Nevertheless, the main principles still apply, and you are encouraged to try out the assignments
1. Setting up the environment¶
Depending on how BIDScoin was installed, you may have to set your Python environment settings before you can run BIDScoin commands from your command-line interface / shell. In the DCCN compute cluster example below it is assumed that an environment module is used to load your Linux Anaconda Python installation and that BIDScoin is installed in a conda environment named “bidscoin”. Run or adjust these commands to your computer system if needed:
$ module add bidscoin # Load the DCCN bidscoin module with the PATH settings and Anaconda environment
$ source activate /opt/bidscoin # Activate the Python virtual environment with the BIDScoin Python packages
Now you should be able to execute BIDScoin commands. Test this by running bidscoin to get a general workflow overview. Can you generate a list of all BIDScoin tools? What about the plugins?
2. Data preparation¶
Create a tutorial playground folder by executing these shell commands:
$ bidscoin --download . # Download the tutorial data (use a "." for the current folder or a pathname of choice to save it elsewhere)
$ cd ./bidscointutorial # Go to the downloaded data (replace "." with the full pathname if your data was saved elsewhere)
The new bidscointutorial folder contains a raw source-data folder and a bids_ref reference BIDS folder, i.e. the intended end product of this tutorial. In the raw folder you will find these DICOM series (aka “runs”):
001-localizer_32ch-head A localizer scan that is not scientifically relevant and can be left out of the BIDS dataset
002-AAHead_Scout_32ch-head A localizer scan that is not scientifically relevant and can be left out of the BIDS dataset
007-t1_mprage_sag_ipat2_1p0iso An anatomical T1-weighted scan
047-cmrr_2p4iso_mb8_TR0700_SBRef A single-band reference scan of the subsequent multi-band functional MRI scan
048-cmrr_2p4iso_mb8_TR0700 A multi-band functional MRI scan
049-field_map_2p4iso The field-map magnitude images of the first and second echo. Set as "magnitude1", bidscoiner will recognize the format. This field map is intended for the previous functional MRI scan
050-field_map_2p4iso The field-map phase difference image of the first and second echo
059-cmrr_2p5iso_mb3me3_TR1500_SBRef A single-band reference scan of the subsequent multi-echo functional MRI scan
060-cmrr_2p5iso_mb3me3_TR1500 A multi-band multi-echo functional MRI scan
061-field_map_2p5iso Idem, the field-map magnitude images of the first and second echo, intended for the previous functional MRI scan
062-field_map_2p5iso Idem, the field-map phase difference image of the first and second echo
Let’s begin with inspecting this new raw data collection:
Are the DICOM files for all the
bids/sub-*folders organized in series-subfolders (e.g.sub-001/ses-01/003-T1MPRAGE/0001.dcmetc)? Use dicomsort if this is not the case (hint: it’s not the case). A help text for all BIDScoin tools is available by running the tool with the-hflag (e.g.rawmapper -h)Use the rawmapper command to print out the DICOM values of the “EchoTime”, “Sex” and “AcquisitionDate” of the fMRI series in the
rawfolder
3. BIDS mapping¶
Now we can make a study bidsmap, i.e. the mapping from DICOM source-files to BIDS target-files. To that end, scan all folders in the raw data collection by running the bidsmapper command:
$ bidsmapper raw bids
In the GUI that appears at the end, edit the task and acquisition labels of the functional scans into something more readable, e.g.
task-Rewardfor theacq-mb8scans andtask-Stopfor theacq-mb3me3 scans. Also make the name of the T1 scan more user friendly, e.g. by naming the acquisition label simplyacq-mprage.Add a search pattern to the
IntendedForfield such that the first field map will select yourRewardruns and the second field map yourStopruns (see the bidseditor field map notes for more details)Since for this dataset we only have one session per subject, remove the session label (and note how the output names simplify, omitting the session subfolders and labels)
When all done, go to the
Optionstab and change thedcm2niixsettings to get non-zipped NIfTI output data (i.e.*.niiinstead of*.nii.gz). Test the tool to see if it can run and, as a final step, save your bidsmap. You can always go back later to change any of your edits by running the bidseditor command line tool directly. Try that.
4. BIDS coining¶
The next step, converting the source data into a BIDS collection, is very simple to do (and can be repeated whenever new data has come in). To do this run the bidscoiner command-line tool (note that the input is the same as for the bidsmapper):
$ bidscoiner raw bids
Check your
bids/code/bidscoin/bidscoiner.log(the complete terminal output) andbids/code/bidscoin/bidscoiner.errors(the summary that is also printed at the end) files for any errors or warnings. You shouldn’t have any :-)Compare the results in your
bids/sub-*subject folders with the inbids_refreference result. Are the file and folder names the same (don’t worry about the multi-echo images and theextra_dataimages, they are combined/generated as described below)? Also check the json sidecar files of the field maps. Do they have the rightEchoTimeandIntendedForfields?What happens if you re-run the
bidscoinercommand? Are the same subjects processed again? Re-runsub-001.
5. Finishing up¶
Now that you have converted the data to BIDS, you still need to do some manual work to make it fully ready for data analysis and sharing
Combine the echos using the echocombine tool, such that the individual echo images are replaced by the echo-combined image
Deface the anatomical scans using the deface tool. This will take a while, but will obviously not work well for our phantom dataset. Therefore store the ‘defaced’ output in the
derivativesfolder (instead of e.g. overwriting the existing images)Inspect the
bids/participants.tsvfile and decide if it is ok.Update the
dataset_description.jsonandREADMEfiles in yourbidsfolderAs a final step, run the bids-validator on your
~/bids_tutorialfolder. Are you completely ready now to share this dataset?
Troubleshooting¶
Installation¶
A first step when encountering execution errors is to test whether your installation is working correctly. An easy way to test the working of various BIDScoin components is to run bidscoin -t in your terminal. Some commonly seen messages are:
The “dcm2niix” command is not recognized¶
This is an installation problem and means that bidscoin can’t find your dcm2niix executable (just carefully follow the installation instructions)
Could not load the Qt platform plugin “xcb”¶
This error message may occur on certain Linux platforms when opening the bidseditor. This is an installation issue that may be solved by downgrading your PyQt5 library, e.g. by running pip install --upgrade pyqt5==5.14 in your terminal environment. Another solution might be to use your Linux package manager to install PyQt5, e.g. like this: apt-get install python3-pyqt5 python3-pyqt5.qtx11extras
My Apptainer/Singularity container fails¶
When running apptainer/singularity on on systems with an older Linux kernel (e.g. older than 3.15) you may get errors such as ImportError: libQt5Core.so.5: cannot open shared object file: No such file or directory. A working fix may be to add the following line at the end of %post section of the singularity.def file.
strip --remove-section=.note.ABI-tag /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
The fix comes from these resources:
Workflow¶
The first step in troubleshooting is to look at the warnings and messages printed out in the terminal (they are also save to disk in the bidsfolder/code/bidscoin output folder). Make sure you are ok with the warnings (they are meaningful and not to be ignored) and do not continue with a next step until all errors are resolved.
My bidsmap is empty¶
After running the bidsmapper, the bidseditor shows an empty bidsmap (i.e no data samples). The most likely cause is that the structure of your raw data repository is not understood by BIDScoin (see data preparation for more info). Another likely cause is that the sub-/ses- prefixes need to be adjusted to your folder names (e.g. when your ). Install and/or add the plugin.
My subject/session labels are wrong¶
Everything seems to work but the sub-/ses- BIDS labels are not what I want. In the bidseditor main window, play around with the subject regular expressions.
I got an “Unexpected postfix / file conversion result”¶
This bidscoiner warning message means that the source data was not properly recognised / converted by the plugin. Please search and/or report it on Github issue to resolve it.
I only see “_magnitude1” or “_magnitude2” run-items in my bidsmap¶
Siemens (and perhaps other manufacturers too) stores all field-map Series in a single Series folder. Hence, when BIDScoin takes a sample from that folder it only sees one of these Series. You don’t need to worry about this, because the dcm2niix plugin will accommodate for this and will look-up the other samples during bidscoiner runtime.
My source-files can no longer be found¶
You may get the warning “Cannot reliably change the datatype and/or suffix because the source file ‘..’ can no longer be found”. This warning is generated when (1) your source data moved to a different location, or (2) your data is zipped or in DICOMDIR format. This warning can be ignored if you do not need to change the datatype of your run-items anymore (in the bidseditor), because in that case BIDScoin may need access to the source data (to read new properties or attributes). To restore data access for (1), move the data to it’s original location and for (2) use the –store option of bidsmapper to store local copies of the source data samples in the bids output folder.
I have duplicated field maps because of an interrupted session¶
It may happen that due to irregularities during data acquisition you had to reacquire your field-map for part of your data. In that case the IntendedFor and B0FieldIdentifier/’B0FieldSource` semantics become ambiguous. To handle this situation, you can use json sidecar files to extend the source attributes (see below) or use the limited IntendedFor search as described here and here.
The bidscoiner says that the IntendedFor search gave no results¶
Even if you have specified the IntendedFor value in the bidseditor, you still get Empty ‘IntendedFor’ field map value in {..}: the search for {..} gave no results. This may be because you hardcoded the IntendedFor value instead of providing a search pattern. Or it may be that you provided a correct search pattern but that for some subjects the target images were not acquired or could not be found (e.g. due to irregularities in the acquisition). Check out the BIDS output session(s) mentioned in the warning(s) and see if and how you should update your IntendedFor search pattern accordingly.
The data of some subjects need to be treated (mapped) differently¶
Sometimes you may have irregularities in your data that make that you would like make exceptions for run-items of certain subjects. There are different ways to do this but most likely the best way to do this is to add a json sidecar file to the source data of those run-items. In the json sidecar file you can store an attribute key-value pair to overrule or extend the original attribute value of the source data. For instance, if your fmri run was acquired with the wrong task presentation, e.g. task2 instead of task1, you can add SeriesDescription: task2 to the sidecar file to overrule SeriesDescription: task1 in the DICOM header (to make a more specific exception that shows up as a new run-item in the bidsmap you can change it to e.g. task1_exception).
More help¶
If this guide doesn’t help to solve your problem, then you can search on github for open and/or closed issues to see if anyone else has encountered similar problems before. If not, feel free to help yourself and others by opening a new github issue.
Changelog¶
All notable changes to this project will be documented in this file. The format is based on Keep a Changelog
4.0.0 - 2023-03-18¶
Added¶
Support for BIDS 1.8.0
Support for handling of non-alphanumeric characters in sub/ses prefixes
A new (optional) pet2bids plugin
A
skullstriptool (a bids-wrapper around FreeSurfer/synthstrip)A
slicereportQC tool (a bids-wrapper around FSL/slicesdir)A unit/integration test & CI framework
New Dockerfile and Apptainer/Singularity definition files to run BIDScoin in a container
A BIDSCOIN_DEBUG environment variable to run bidscoin in a more verbose debug mode
VERBOSE and SUCCESS logging levels
Checks using the bids-validator python module
Options to perform study/template bidsmap tests with
bidscoin -b / -tOption to list and install template bidsmaps
Many bugfixes, user interface improvements and tweaks
Changed¶
The default behaviour is now to not unzip data in the sourcefolders anymore. Instead, users can use the
--unzipoption in the bidsmapperThe spec2nii plugin is no longer added by default but is optional
The dcm2niix2bids plugin no longer handles PET data (this is now handled by the pet2bids plugin)
Removed BIDScoin’s redundant/confusing
datatypesoption from the bidsmapRemoved the obscure
participantsoption from bidscoinerMajor code refactoring
3.7.4 - 2022-10-21¶
Added¶
Added support for the ABCD GE pepolar pulse sequence
Use an orange bidsname font in the bidseditor for
.bidsignoredatatypesA (right-click) context menu in the bidseditor to import meta-data from disk into the meta-table
Changed¶
Remove the DCCN specific dcm2niix module usage in the bidsmap template
Add
-l nto the dcm2niix arguments to revert old UIN16 -> INT16 behaviour (otherwise fmriprep outputs are twice as large)No longer enforce BIDS compliance on
.bidsignoredatatypes
Fixed¶
Sorting flat DICOM data (in a temporary working directory)
The dcm2niix module (if not removed) raised an error that prevented handling dcm2niix suffixes
3.7.3 - 2022-07-13¶
Added¶
The usage of json sidecar files as a datasource for attribute values
A template bidsmap for the ScanSessionTool
Changed¶
Dicomsort now searches recursively over the sessionfolder
The dcm2niix2bids plugin now searches recursively for DICOM Series folders
Images that have already been defaced are now skipped
Prepend the rawfolder name & subprefix for more robust subject- / session-label filepath extraction
Fixed¶
Pydeface not parsing subject / session labels from the filepath
The non-HPC use of pydeface no longer requires DRMAA installation
Account for
*and?wildcards in the sub/ses prefixes in the bidsmapperAccount for dynamic values with non-matching regular expressions (special thanks to Mateusz Pawlik)
Various minor bugs
3.7.1 - 2022-03-11¶
Added¶
IntendedFor can now be appended with a “bounding” term to deal with duplicated field maps from interrupted sessions
The possibility to process subject folders without prefix
Support for BIDS 1.7 (e.g. for the new
B0FieldSourceandB0FieldIdentifierfield-map meta fields)A
nibabel2bidsplugin (e.g. to convert NIfTI datasets to BIDS)Plugin
metaoption setting to enrich json sidecar files or add data that is not supported
Changed¶
Removed / changed redundant subject/session prefix input arguments (now stored in the bidsmap)
The
IntendedForsearch feature now works independent of plugins
Fixed¶
The bidscoin installation test in the bidseditor
The IntendedFor list when combining echos
3.7.0 - 2021-12-20¶
Added¶
A BIDScoin installation test (
bidscoin -t)Option to install extra packages, such as phys2bids
A bidseditor button to save the Options to a (default) template bidsmap
Sub-/ses-prefix settings and BIDS / extra_data / excluded datatypes in bidsmap[‘Options’][‘bidscoin’]
Regular expressions for extracting property and attribute substrings from dynamic values via a <<key:regular_expression>> syntax
A plugin for spec2nii to convert MR spectroscopy data
An experimental plugin for phys2bids to convert physiological data
An experimental plugin for pet2bids to convert MR spectroscopy data
Added a multi-echo deface function
medefacethat uses the same defacemask for all echo-imagesThe possibility to extract DICOM values using pydicom-style tag numbers (in addition to the attribute name)
The possibility for plugins to set default bidsmappings and Options when installed
A Singularity container configuration file
Improved (more fine-grained) plugin installation procedures
The option to remove decimals from age and discard acquisition dates from the metadata
Changed¶
Plugins should now have a
is_sourcefileand aget_attributefunction and have a simpler/changed API (-> DataSource class)The intricate filtering of the
nrfilesproperty by the other filesystem properties has been removed and is now a pure/unfiltered file-system propertyThe default
<<SourceFilePath>>keyword has been replaced by the more flexible <<filepath:/sub-(.*?)/>> property to extract the subject / session labelThe dcm2bidsmap and the dcm2niix2bids plugins have been merged
The dicomsort utility has new naming-scheme functionality
Removed the obsolete bidsmap_template.yaml file
Fixed¶
Avoid storing Python literal structures as strings
3.6.3 - 2021-06-14¶
Fixed¶
Remove regular expression metacharacters from the source attribute if needed (could cause a regexp compile error)
Fixed for list of dynamic <
3.6.2 - 2021-05-31¶
Fixed¶
Removed the redundant importlib dependency from the requirements (could cause an installation error)
3.6.1 - 2021-05-20¶
Fixed¶
The bidscoiner no longer sometimes crashes when dcm2niix produces custom suffixes (e.g. for multi-echo data)
3.6.0 - 2021-05-13¶
Added¶
Support for BIDS v1.6.0 (-> PET)
Separate tabs for DICOM and PAR to edit all the mappings of mixed datasets in a single bidseditor session
Run-item matching on filesystem properties, i.e. on the pathname, filename and filesize and nr of files in the folder. This can be used in conjunction with the (DICOM/PAR) attributes
A meta-data dictionary that can be edited with the bidseditor and that will be added to the json sidecar files by the bidscoiner
More user feedback in the GUI for new BIDS-compliancy checks on missing or invalid bids data
A right-click menu option to remove a run-item from the bidsmap (advanced usage)
The option to load a new bidsmap in the bidseditor
Enable the user to edit json, yaml, tsv and other non-DICOM / non-PAR files with double-clicks in the data browser
A central ‘bidscoin’ package function with various utilities, such as listing and installing BIDScoin plugins or executables
Plugins can have their own ‘test’ routine that can be called from the bidseditor
Changed¶
Using regular expressions instead of fnmatch to match (template bidsmap) attribute values. This makes the templates more powerful and flexible
Moved the bidsmapping and bidscoining functionality to stand-alone plugins (changed API), making plugins a first-class BIDScoin citizen
The plugins have moved to the bidsmap[‘Options’], where they have their own key-value options dictionary (changed API)
Move IntendedFor field over to the new meta-data dictionary
Renamed the
leave_outdatatype toexcludeRe-introduced skipping hidden folders (hidden files are also skipped)
Moved the ‘pulltutorial’ function over to the new ‘bidscoin’ function
Removed¶
P7 and NIfTI support (it was never implemented anyhow)
The option to edit new mappings on-the-fly in the bidsmapper (
-i 2)
3.5.3 - 2021-04-13¶
Fixed¶
Save non-standard field maps in the derivative folder
Add ‘AcquisitionTime’ to physio json-files and add the physio-files to the *_scans.tsv file
3.5.1 - 2021-03-21¶
Added¶
BIDScoin version update checks
Fixed¶
Speed optimizations
Code clean-up
More robust dcm2niix output handling
3.5 - 2021-03-08¶
A significant rewrite and evolution of BIDScoin!
Added¶
Support for BIDS v1.5
Support for Siemens advanced physiological logging data
Improved GUI help tooltips and user feedback
Improved feedback and control for invalid bidsnames
Validation of run-items and bidsmaps against the BIDS schema
Changed¶
Use the DCCN template bidsmap as the default
Fixed¶
Simplified and improved (hopefully) handling of fieldmaps
3.0.5 - 2020-08-05¶
Added¶
A download tool for tutorial data
A tool for regenerating the participants.tsv file
Fixed¶
Various bugs
3.0.4 - 2020-05-14¶
Added¶
Exportfunction in the bidseditor to allow for adding run items to existing (template) bidsmapsSupport for Unix-shell style wildcards for matching run items in the bidsmap
Changed¶
Improved DCCN example template bidsmap
Fixed¶
Various minor bugs
3.0.3 - 2020-04-14¶
Fixed¶
A small bugfix to properly handle appending dcm2niix suffices to the BIDS acq-label
3.0.1 - 2020-04-04¶
Added¶
A ‘provenance store’ in the
bidsmapperto fix a bug (#56 ) and allow for moving the bids-folder aroundSupport for zipped/tarred DICOM directories
3.0 - 2020-04-01¶
A Significant rewrite to make BIDScoin more robust, user-friendly and feature-rich :-)
Added¶
First support for Philips PAR / REC data format
A BIDS compliant defacing tool
A BIDS compliant multi-echo combination tool
Much improved documentation (https://bidscoin.readthedocs.io)
2.3.1 - 2019-09-12¶
Fixed¶
a small but important bug that caused datasets without field maps to crash (my test datasets all had field maps :-))
2.3 - 2019-08-29¶
A lot of improvements have landed in 2.3, making it the best release of the 2-series by far!
Added¶
The possibility to edit Participant labels
Various tests and checks in Options to ensure creating good working bidsmaps / BIDS output data
Upgraded compliance with bids v1.2.1
The possibility to leave-out certain data types / runs
Changed¶
A new workflow that is easier and more consistent
Greatly improved graphical user interface and error/warning reporting
Improved bidsmap_dccn template
Fixed¶
Significant code refactoring to squash a number of important bugs and make the code more robust and maintainable
2.2 - 2019-07-11¶
Added¶
Options tab to edit and test the bidscoin Options
A leave-out option (to ignore runs / prevent them from showing up in the BIDS directory)
A graphical interface to the bidsmapper
Improved logging
Improved the DICOM attribute wildcard feature
Changed¶
New layout of the main and edit windows
Fixed¶
Various bugfixes
2.1 - 2019-06-23¶
Added¶
Editing of bidsmap Options
Fixed¶
IntendedForin field map json sidecar filesCode redundancy
2.0 - 2019-06-18¶
A major release and rewrite with important user-facing improvements
Added¶
A shiny GUI :-)
A new and much easier workflow
Fixed¶
Various bugfixes
1.5 - 2019-03-06¶
Added¶
Support for PET scans
Support for DICOMDIR data
Saving of template sidecar files in the bids output directory
Changed¶
increased flexibility for renaming / reorganising the raw (input) data structure
Added provenance data to the bidsmap/yaml files
Fixed¶
various bugfixes