The BIDScoin workflow¶
With a sufficiently organized source data folder, the data conversion to BIDS can be performed by running the (1a) the bidsmapper, (1b) the bidseditor and (2) the bidscoiner command-line tools. The bidsmapper starts by building a map of the different kind of data types (scans) in your source dataset, which you can then edit with the bidseditor. The bidscoiner reads this so-called study bidsmap, which tells it how exactly to convert (“coin”) the source data into a BIDS data repository.
By default, step 1a automatically launches step 1b, so in it’s simplest form, all you need to do to convert your raw source data into BIDS is to run two simple shell commands, e.g.:
$ bidsmapper sourcefolder bidsfolder # Scans your data and creates a study bidsmap
$ bidscoiner sourcefolder bidsfolder # Converts your data to BIDS using the study bidsmap
If you add new subjects all you need to do is re-run the bidscoiner – unless the scan protocol was changed, then you also need to first re-run the bidsmapper to add the new samples to the study bidsmap. The paragraphs below describe the BIDScoin worklow in more detail.
Tip
If you don’t know what shell command to use or what to do, run the bidscoin
command to give you a workflow overview
Step 1a: Running the bidsmapper¶
usage: bidsmapper.py [-h] [-b BIDSMAP] [-t TEMPLATE] [-p PLUGINS [PLUGINS ...]] [-n SUBPREFIX]
[-m SESPREFIX] [-s] [-a] [-f] [-v]
sourcefolder bidsfolder
The bidsmapper scans your source data repository to identify different data types by matching
them against the run-items in the template bidsmap. Once a match is found, a mapping to BIDS
output data types is made and the run-item is added to the study bidsmap. You can check and edit
these generated bids-mappings to your needs with the (automatically launched) bidseditor. Re-run
the bidsmapper whenever something was changed in your data acquisition protocol and edit the new
data type to your needs (your existing bidsmap will be re-used).
The bidsmapper uses plugins, as stored in the bidsmap['Options'], to do the actual work
positional arguments:
sourcefolder The study root folder containing the raw source data folders
bidsfolder The destination folder with the (future) bids data and the
bidsfolder/code/bidscoin/bidsmap.yaml output file
optional arguments:
-h, --help show this help message and exit
-b BIDSMAP, --bidsmap BIDSMAP
The study bidsmap file with the mapping heuristics. If the bidsmap filename
is relative (i.e. no "/" in the name) then it is assumed to be located in
bidsfolder/code/bidscoin. Default: bidsmap.yaml
-t TEMPLATE, --template TEMPLATE
The bidsmap template file with the default heuristics (this could be
provided by your institute). If the bidsmap filename is relative (i.e. no
"/" in the name) then it is assumed to be located in
bidsfolder/code/bidscoin. Default: bidsmap_dccn.yaml
-p PLUGINS [PLUGINS ...], --plugins PLUGINS [PLUGINS ...]
List of plugins to be used (with default options, overrules the plugin list
in the study/template bidsmaps)
-n SUBPREFIX, --subprefix SUBPREFIX
The prefix common for all the source subject-folders (e.g. 'Pt' is the
subprefix if subject folders are named 'Pt018', 'Pt019', ...). Use '*' when
your subject folders do not have a prefix. Default: the value of the study
or template bidsmap, e.g. 'sub-'
-m SESPREFIX, --sesprefix SESPREFIX
The prefix common for all the source session-folders (e.g. 'M_' is the
subprefix if session folders are named 'M_pre', 'M_post', ...). Use '*'
when your session folders do not have a prefix. Default: the value of the
study or template bidsmap, e.g. 'ses-'
-s, --store Flag to store provenance data samples in the bidsfolder/'code'/'provenance'
folder (useful for inspecting e.g. zipped or transfered datasets)
-a, --automated Flag to save the automatically generated bidsmap to disk and without
interactively tweaking it with the bidseditor
-f, --force Flag to discard the previously saved bidsmap and logfile
-v, --version Show the installed version and check for updates
examples:
bidsmapper /project/foo/raw /project/foo/bids
bidsmapper /project/foo/raw /project/foo/bids -t bidsmap_dccn
bidsmapper /project/foo/raw /project/foo/bids -p nibabel2bids
After the source data has been scanned, the bidsmapper will automatically launch step 1b to let the user check and edit the automatically generated study bidsmap. For a fully automated workflow users can skip this interactive step using the -i
option (see above).
Tip
The default template bidsmap (-t bidsmap_dccn
) is customized for acquisitions at the DCCN. If this bidsmap is not working well for you, consider adapting it to your needs so that the bidsmapper can recognize more of your scans and automatically map them to BIDS the way you prefer.
Step 1b: Running the bidseditor¶
usage: bidseditor.py [-h] [-b BIDSMAP] [-t TEMPLATE] bidsfolder
This application launches a graphical user interface for editing the bidsmap that is produced by
the bidsmapper. You can edit the BIDS data types and entities until all run-items have a meaningful
and nicely readable BIDS output name. The (saved) bidsmap.yaml output file will be used by the
bidscoiner to do the conversion conversion of the source data to BIDS.
You can hoover with your mouse over items to get help text (pop-up tooltips).
positional arguments:
bidsfolder The destination folder with the (future) bids data
optional arguments:
-h, --help show this help message and exit
-b BIDSMAP, --bidsmap BIDSMAP
The study bidsmap file with the mapping heuristics. If the bidsmap filename
is relative (i.e. no "/" in the name) then it is assumed to be located in
bidsfolder/code/bidscoin. Default: bidsmap.yaml
-t TEMPLATE, --template TEMPLATE
The template bidsmap file with the default heuristics (this could be
provided by your institute). If the bidsmap filename is relative (i.e. no
"/" in the name) then it is assumed to be located in
bidsfolder/code/bidscoin. Default: bidsmap_dccn.yaml
examples:
bidseditor /project/foo/bids
bidseditor /project/foo/bids -t bidsmap_dccn.yaml
bidseditor /project/foo/bids -b my/custom/bidsmap.yaml
Main window¶
As shown below, the main window of the bidseditor opens with separate data mapping tabs for each data format that is present in the bidsmap (here DICOM mappings
and PAR mappings
). The data mapping tabs consist of a Participant labels
table and a Data samples
table. By default, the participant table contains dynamic <<filepath:regexp>>
property values, which are used to extract the subject and session labels from the path of the source data during bidscoiner runtime. Alternatively, you can put a dynamic attribute value there (e.g. <<PatientName>>) if you want to extract that information from the source header. The data samples table shows a list of input files (left side) that uniquely represent all the different data types in the sourcedata repository, in conjunction with a preview of their BIDS output
names (right side). The BIDS output names are shown in red if they are not BIDS compliant, striked-out gray when the runs will be ignored / skipped in the conversion to BIDS, otherwise it is colored green.
Tip
If the default subject/session expression (e.g. /sub-(.*?)/
where sub-
can be substituted by your prefix) fails to parse the subject or session label, try prepending (a part of) the sourcefolder path, e.g. if your data is in /project/sourcedata/s001/..
and your subject prefix is s
, try <<filepath:/sourcedata/s(.*?)/>>
for extracting the 001
subject label. This is especially useful if your subject folders have no or a very short prefix.
Tip
Clear the session
label field if you have data with only one session. This will remove the optional session label from the BIDS ouptput name
Edit window¶
In the main window, you can double-click the BIDS output name of a data sample or click the [Edit] button next to it (NB: the * in this button indicates that attention is required) to open a new window, as shown below. In this new window, the full bids-mapping info of the clicked data-sample (AKA run-item) is shown, with the filesystem Properties
and file Attributes
input on the left, and, most importantly, the associated BIDS Data type
, Data filename
and Meta data
output on the right. You should first make sure the BIDS Data type
(drop down menu) and its suffix
label (drop down menu) are set correctly, and then you should edit the (automatically generated) BIDS values that you think are not optimal or incorrect (double-click the cell). Each time an item is edited, a new Data filename
preview is shown (green or red text indicates that the name is BIDS compliant or not). In the Meta data
table (see the figure below) you can enter key-value pairs that you like to to be appended (by the standard dcm2niix2bids
plugin) to the standard meta-data in the json sidecar file. Right-clicking the meta table allows you to import meta-data from JSON/YAML/CSV/TSV files on disk. Editing the source properties and attributes of a study bidsmap is usually not necessary and considered advanced usage.
If the preview of the BIDS filename and meta-data both look good, you can store the data in the bidsmap by clicking the [OK] button.
Finally, if all BIDS output names in the main window are fine, you can click on the [Save] button and proceed with running the bidscoiner tool (step 2). Note that re-running the bidsmapper or bidseditor is always a safe thing to do since these tools will re-use the existing bidsmap yaml-file and will not delete or write anything to disk except to the bidsmap yaml-file.
Fieldmaps¶
Fieldmaps are acquired and stored in various (sequences and manufacturer dependent) ways and may require some special treatment. For instance, it could be that you have magnitude1
and magnitude2
data in one series-folder (which is what Siemens can do). In that case you should select the magnitude1
suffix and let bidscoiner automatically pick up the magnitude2
during runtime (or vice versa). The same holds for phase1
and phase2
data. The suffix magnitude
can be selected for sequences that save fieldmaps directly. See the BIDS specification for more details on fieldmap suffixes.
Fieldmaps are typically acquired to be applied to specific other scans from the same session. The BIDS specification provides two meta-data mechanisms to store this semantic meta data (NB: BIDS-apps may not use your fieldmap at all if you do not specify anything):
First there is the older
IntendedFor
mechanism that can handle more basic use cases, i.e. it explicitly specifies the path of the target images to which the fieldmap should be applied, but it is left implicit from which images the fieldmap is to be computed. You can enter a dynamicIntendedFor
search string in theMeta data
table to have BIDScoin automatically fill out this field for you. For instance you can simply usetask-Stop*_bold
as a search pattern to specify all functional runs in the BIDS session that havetask-Stop
and_bold
as part of their filename. For more advanced usage and explanation, see the special bidsmap features sectionSecond, there is the new and more flexible
B0Fieldmap
mechanism that uses aB0FieldIdentifier
to group all the images from which the fieldmap can be computed, and aB0FieldSource
to indicate which fieldmap should be used to correct the image. For instance, you could use{B0FieldIdentifier: sbref_fmap}
in yourAP
andPA
PE-polarsbref
images, in conjunction with{B0FieldSource: sbref_fmap}
in your associatedAP
PE-polarbold
image.
Tip
The BIDScoin GUI features several ways to help you setting the right values: * Double-clicking an input filename pops-up an inspection window with the full header information (e.g. useful for checking attributes that are not (yet) in your bidsmap) * Hoovering with your mouse over a cell pops-up a tooltip with more background information (e.g. from the BIDS specifications) * Always check the terminal output and make sure there are no warnings or error messages there (a summary of them is printed when exiting the application)
Step 2: Running the bidscoiner¶
usage: bidscoiner.py [-h] [-p PARTICIPANT_LABEL [PARTICIPANT_LABEL ...]] [-f] [-s] [-b BIDSMAP]
[-v]
sourcefolder bidsfolder
Converts ("coins") your source datasets to nifti / json / tsv BIDS datasets using
the information from the bidsmap.yaml file. Edit this bidsmap to your needs using the
bidseditor tool before running this function or (re-)run the bidsmapper whenever you
encounter unexpected data. You can run bidscoiner after all data has been collected,
or run / re-run it whenever new data has been added to your source folder (presuming
the scan protocol hasn't changed). Also, if you delete a subject/session folder from
the bidsfolder, it will simply be re-created from the sourcefolder the next time you
run the bidscoiner.
The bidscoiner uses plugins, as stored in the bidsmap['Options'], to do the actual work
Provenance information, warnings and error messages are stored in the
bidsfolder/code/bidscoin/bidscoiner.log file.
positional arguments:
sourcefolder The study root folder containing the raw source data
bidsfolder The destination / output folder with the bids data
optional arguments:
-h, --help show this help message and exit
-p PARTICIPANT_LABEL [PARTICIPANT_LABEL ...], --participant_label PARTICIPANT_LABEL [PARTICIPANT_LABEL ...]
Space separated list of selected sub-# names / folders to be processed (the
sub- prefix can be removed). Otherwise all subjects in the sourcefolder
will be selected
-f, --force If this flag is given subjects will be processed, regardless of existing
folders in the bidsfolder. Otherwise existing folders will be skipped
-s, --skip_participants
If this flag is given those subjects that are in participants.tsv will not
be processed (also when the --force flag is given). Otherwise the
participants.tsv table is ignored
-b BIDSMAP, --bidsmap BIDSMAP
The study bidsmap file with the mapping heuristics. If the bidsmap filename
is relative (i.e. no "/" in the name) then it is assumed to be located in
bidsfolder/code/bidscoin. Default: bidsmap.yaml
-v, --version Show the installed version and check for updates
examples:
bidscoiner /project/foo/raw /project/foo/bids
bidscoiner -f /project/foo/raw /project/foo/bids -p sub-009 sub-030
Tip
Always check the terminal output for possible warnings or errors (a summary of them is printed at the end)
Check your json sidecar files of your fieldmaps, in particular see if they have the expected
IntendedFor
/B0FieldIdentifier
values
Note
The provenance of the produced BIDS data-sets is stored in the [bidsfolder]/code/bidscoin/bidscoiner.log
file. This file is also very useful for debugging / tracking down bidscoin issues.