Results
1MRI Systems¶
Most MRI development is done on commercial systems using proprietary hardware and software. Peeking inside the black boxes that generate the images is non-trivial, but it is essential for promoting reproducibility in MRI.
Quantitative MRI articles are powerfulexcellent showcases of reproducible research practices, as they usually come with fitting models that can be shared on public code repositories. The applications range from MR spectroscopy Clarke et al., 2021Riemann et al., 2022Songeon et al., 2023Wilson2021-xg} to ASL {cite:p}{Woods2022-ud}, diffusion MRI (Cai et al., 2021; Tristán-Vega et al., 2022), CEST (Huang et al., 2022), magnetization transfer (Assländer et al., 2022; Malik et al., 2020; Rowley et al., 2021), B1 mapping (Delgado et al., 2020) and relaxometry (Balbastre et al., 2022; Kapre et al., 2020; J. Lee et al., 2020; Whitaker et al., 2020).
Transparent reconstruction pipelines are also prominently featured in the reproducible research insights, including real-time MRI (Zhao et al., 2021), parallel imaging (Hess et al., 2021), large-scale volumetric dynamic imaging (Ong et al., 2020), phase unwrapping (Dymerska et al., 2021), hyperpolarized MRI (Tustison et al., 2021), Dixon imaging (Rydén et al., 2020) and X-nuclei imaging (McCallister et al., 2021). Deep learning is increasingly present in the reproducibility conversation, as MRI researchers are trying to shine a light on AI-driven workflows for phase-focused applications (Cole et al., 2021), CEST (Huang et al., 2022), diffusion-weighted imaging (Barbieri et al., 2020), myelin water imaging (J. Lee et al., 2020), B1 estimation (Abbasi-Rad et al., 2021), and tissue segmentation (Estrada et al., 2020).
Reproducibility of MRI hardware is still in its infancy, but a recent study integrated RF coils with commercial field cameras for ultrahigh-field MRI, exemplifying the coupling of hardware advancements with software solutions. The authors shared the design CAD files, performance data, and image reconstruction code, ensuring that hardware innovations can be reproduced and utilized by other researchers (Gilbert et al., 2022).
Finally, vendor-neutral pulse sequences are putting interoperability and transparency at the center of the reproducibility landscape. Pulseq and gammaSTAR are vendor-neutral platforms enabling the creation of MRI pulse sequences that are compatible with three major MRI vendors (Layton et al., 2017); (Cordes et al., 2020). In addition, VENUS is an end-to-end vendor-neutral workflow that was shown to reduce inter-vendor variability in quantitative MRI measurements of myelin, thereby strengthening the reproducibility of quantitative MRI research and facilitating multicenter clinical trials
2Reproducibility tools¶
There is a growing number of studies providing access to raw imaging data, pre-processing pipelines, and post-analysis results. Repositories like Zenodo, XNAT, and the Open Science Framework (OSF) serve as vital resources for housing and curating MRI data. Data sharing is also made easier thanks to unifiedstandardized data representationsformats, such as the ISMRM raw data format (Inati et al., 2017) for standardizing k-space datarepresentations, and the Brain Imaging Data Structure (BIDS) for organizing complex datasets (Gorgolewski et al., 2016) and their derivatives (Karakuzu, Appelhoff, et al., 2022). Software repositories such as Github and Gitlab are making it easier to centralize processing routines and to adopt version control, unit tests and other robust software development practices. The introduction of tools for automated QA processes, as seen in the development of platforms like PreQual for DWI analysis (Cai et al., 2021), signifies an emphasis on interoperability and standardization.
The increasing adoption of containerization and virtual environments makes workflows transparent and easy to execute. Tools like Docker and Singularity are used to package computing environments, making them portableshippable and reproducible across different systems. Studies employing these tools enable MRI researchers to replicate computational processing pipelines without dealing with dependency issues in local computational environments (Cordes et al., 2020; Estrada et al., 2020; Karakuzu, Biswas, et al., 2022).
The rise of machine learning and artificial intelligence in MRI necessitates rigorous evaluation to ensure reproducibility. Studies that use deep learning are beginning to supplement their methodological descriptions with the open-source code, trained models, and simulation tools that underpin their algorithms. Algorithms such aslike DeepCEST, developed for B1 inhomogeneity correction at 7T, showcase how clinical research can be improved by reproducible research practices (Huang et al., 2022). Sharing these algorithms allows others to perform direct comparisons and apply them to new datasets.
Finally, pulse-sequence and hardware descriptions are slowly entering the public domain (Cordes et al., 2020; Gilbert et al., 2022; Karakuzu, Biswas, et al., 2022; Layton et al., 2017). For a long time MRI vendors have been reluctant to open up their systems (Stikov & Karakuzu, 2023), but standardized phantoms (Stupic et al., 2021) are creating benchmarks that require transparency and reproducibility. This is particularly relevant for quantitative MRI applications, where scanner upgrades and variabilities across sites are a major hurdle to wider clinical adoption (Boudreau et al., 2023; Keenan et al., 2019; Y. Lee et al., 2019).
Reproducibility is also bolstered by interactive documentation and tools such as Jupyter Notebooks, allowing for dynamic presentation and hands-on engagement with data and methods. Platforms incorporating such interactive elements are being utilized with greater frequency, providing real-time demonstration of analysis techniques and enabling peer-led validation. Resources such as MRHub, MRPub, and Open Source Imaging and NeuroLibre (Karakuzu et al. 2022) serve as a gateway to a wide range of tools and tutorials that promote reproducibility in MRI.
3Discussion and Future Directions¶
The progress towards reproducibility in neuroimaging research points to a distinct cultural shift in the scientific community. The move towards open-access publishing, code-sharing platforms, and data repositories reflects a concerted effort to uphold the reproducibility of complex imaging studies. Adopting containerization technologies, pushing for standardization, and consistently focusing on quality assurance are key drivers that will continue to improve reproducibility standards in MRI research. Figure 2 is a word cloud generated from the articles included in this scoping review, highlighting the concepts and vocabulary that is driving reproducibility in MRI.
# REQUIRED CODE CELL
import os
import re
from PIL import Image
import numpy as np
import matplotlib.pyplot as plt
from scipy.ndimage import gaussian_gradient_magnitude
from wordcloud import WordCloud, ImageColorGenerator, STOPWORDS
np.seterr(divide = 'ignore')
DATA_ROOT = "../data/repro-mri-scoping/repro_mri_scoping"
# Purify and merge text
def read_file(file_path):
with open(file_path, 'r', encoding='utf-8') as file:
content = file.read()
return content
def write_file(file_path, content):
with open(file_path, 'w') as file:
file.write(content)
def remove_enumeration_lines(text):
# Define the pattern for lines starting with enumeration and ending with a question mark
pattern = r'^\s*\d+\.\s.*\?$'
# Use re.MULTILINE to apply the pattern to each line in the input text
result = re.sub(pattern, '', text, flags=re.MULTILINE)
return result
def remove_by_pattern(input_text, patterns_to_remove):
for pattern in patterns_to_remove:
input_text = re.sub(pattern, '', input_text)
return input_text.strip()
def get_output_dir(file_name):
op_dir = "../output"
if not os.path.exists(op_dir):
os.mkdir(op_dir)
return os.path.join(op_dir,file_name)directory_path = os.path.join(DATA_ROOT,"repro_insights_parsed_nov23")
input_files = [os.path.join(directory_path, f) for f in os.listdir(directory_path) if f.endswith('.txt')]
patterns_to_remove = [
re.compile(r'Questions about the specific reproducible research habit'),
re.compile(r'Title:'),
re.compile(r'TLDR:'),
re.compile(r'Abstract:'),
re.compile(r'Reproducibility Insights:'),
re.compile(r'General questions'),
re.compile(r'Questions about the specific reproducible research habit'),
re.compile(r'This MRM Reproducible Research Insights interview'),
re.compile(r'This work was singled out because it demonstrated exemplary reproducible research practices')
]
all_text = ''
for cur_file in input_files:
cur_content = read_file(cur_file)
cur_content = remove_enumeration_lines(cur_content)
cur_content = remove_by_pattern(cur_content, patterns_to_remove)
all_text = all_text + "\n" + cur_contenttext = all_text
stopwords = set(STOPWORDS)
stopwords.add("will")
stopwords.add("you")
stopwords.add("others")
stopwords.add("people")
stopwords.add("using")
stopwords.add("By")
stopwords.add("Mathieu Boudreau")
stopwords.add("Agah Karakuzu")
stopwords.add("Pinar S. Ozbay")
brain_color = np.array(Image.open(os.path.join(DATA_ROOT, "brain_image.png")))
brain_color = brain_color[::3, ::3]
brain_mask = brain_color.copy()
brain_mask[brain_mask.sum(axis=2) == 0] = 255
# some finesse: we enforce boundaries between colors so they get less washed out.
# For that we do some edge detection in the image
edges = np.mean([gaussian_gradient_magnitude(brain_color[:, :, i] / 255., 2) for i in range(3)], axis=0)
brain_mask[edges > .08] = 255
# create wordcloud. A bit sluggish, you can subsample more strongly for quicker rendering
# relative_scaling=0 means the frequencies in the data are reflected less
# acurately but it makes a better picture
wc = WordCloud(max_words=2000, mask=brain_mask, max_font_size=50, min_font_size=5, random_state=42, relative_scaling=0.2, stopwords=stopwords)
# generate word cloud
wc.generate(text)
plt.imshow(wc)
# create coloring from image
image_colors = ImageColorGenerator(brain_color)
wc.recolor(color_func=image_colors)
plt.figure(figsize=(10, 10))
plt.imshow(wc, interpolation="bilinear")
wc.to_file(get_output_dir("brain_wordcloud.png"))<wordcloud.wordcloud.WordCloud at 0x756a69f9c370>
Figure 1:A word cloud generated from the 31 reproducible research insights published by Magnetic Resonance in Medicine Highlights.
As can be seen from the figure, the components of reproducibility in MRI research are multifaceted, integrating not just data and code, but also the analytical pipelines and hardware configurations. The shift towards comprehensive sharing is motivated both by a scientific ethic of transparency and the practical need for rigorous validation of complex methodologies (Boudreau et al., 2021, 2022).
However, this shift is not without challenges (Niso et al. 2022). Variations in data acquisition and analysis methodologies limit cross-study comparisons. Sensitivity to software and hardware versions can impede direct reproducibility. Privacy concerns and data protection regulations can be barriers to data sharing, particularly with clinical images.
While challenges persist, steps are taken by individual researchers and institutions to prioritize reproducibility. Moving forward, the MRI community should work collectively to overcome barriers, institutionalize reproducible practices, and constructively address data sharing concerns to further the discipline’s progress.
The initiatives and tools identified in this review serve as a blueprint for future studies to replicate successful practices, safeguard against bias, and accelerate neuroscientific discovery. As MRI research continues to advance, upholding the principles of reproducibility will be essential to maintaining the integrity and translational potential of its findings.
- Clarke, W. T., Stagg, C. J., & Jbabdi, S. (2021). FSL-MRS: An end-to-end spectroscopy analysis package. Magn. Reson. Med., 85(6), 2950–2964.
- Riemann, L. T., Aigner, C. S., Mekle, R., Speck, O., Rose, G., Ittermann, B., Schmitter, S., & Fillmer, A. (2022). Fourier-based decomposition for simultaneous 2-voxel MRS acquisition with 2SPECIAL. Magn. Reson. Med., 88(5), 1978–1993.
- Songeon, J., Courvoisier, S., Xin, L., Agius, T., Dabrowski, O., Longchamp, A., Lazeyras, F., & Klauser, A. (2023). In vivo magnetic resonance 31 P-Spectral Analysis With Neural Networks: 31P-SPAWNN. Magn. Reson. Med., 89(1), 40–53.