NeuroFusion: A Unified Framework for Generalized Visual Stimulus Decoding from fMRI Across Datasets and Subjects

Clinical Snapshot

Key Findings

• P Value: Not reported

• Effect Size: Up to 94.8% CLIP similarity on NSD dataset with augmentation (NSD AUG condition); SSIM of 0.403 on BOLD5000 after lightweight fine-tuning of alignment layers

• Primary Outcome: Semantic reconstruction fidelity measured by CLIP similarity score across datasets and subjects using the NeuroFusion unified framework

• Nnt Or Sensitivity: Not applicable in traditional clinical sense; no sensitivity/specificity or NNT reported. Cross-dataset transfer performance metrics serve as the functional analogue — SSIM 0.403 on BOLD5000 represents the key cross-dataset generalisation benchmark

• Confidence Interval: Not reported

Clinical Application

Clinical deployment is not currently feasible. The framework requires high-quality research-grade fMRI data, substantial computational infrastructure for model training, and lightweight fine-tuning on target subjects — all of which exceed routine clinical neuroimaging workflows. The small subject numbers in validation datasets preclude robust clinical benchmarking. This technology has no current PBS listing, TGA regulatory pathway, or RACGP guideline relevance. Australian clinical fMRI practice (primarily pre-surgical mapping and research) does not yet incorporate neural decoding frameworks of this type. Future relevance may emerge in the context of assistive communication devices for patients with severe motor impairment, which would require engagement with the TGA's Software as a Medical Device (SaMD) regulatory framework and NHMRC ethics oversight for brain data privacy. Australian neuroimaging research consortia (e.g., MASSIVE, Australian Imaging Biomarkers and Lifestyle study infrastructure) could provide larger validation datasets in future work. Currently applicable only to research participants undergoing task-based fMRI with visual stimulation in controlled laboratory settings. Not validated for clinical populations including patients with stroke, disorders of consciousness, locked-in syndrome, or other neurological conditions where brain-computer interface applications would be most relevant.

Abstract

Recent advancements in neural decoding have shown promising results in reconstructing visual experiences from brain activity. However, existing approaches focus primarily on decoding within a single dataset or subject, which limits generalization across various sources of neuroimaging. In this work, we propose a novel framework for the decoding of visual stimuli between subjects and between data sets, integrating neural recordings from multiple publicly available fMRI datasets. To address inherent intersubject and interdataset variability, we introduce a contrastive learning-based alignment strategy using image embeddings from a pre-trained IP-Adapter model. Our approach learns a shared latent space by aligning subject-specific neural representations with image features, enabling generalized decoding across both subjects and datasets. In addition, we propose a simple yet effective data augmentation method using ridge regression. This method synthesizes realistic fMRI-like signals from novel images by predicting voxel activity and injecting learned noise distributions, thus enhancing training diversity and model robustness. To the best of our knowledge, while several recent studies have explored cross-subject decoding, we extend recent cross-subject decoding efforts by training a single unified framework jointly across multiple public fMRI datasets and subjects, enabling cross-dataset transfer in addition to cross-subject generalization. We distinguish this multi-dataset unified training setting, where each dataset contributes training data, from a stricter leave-one-dataset-out transfer setting in which the target dataset is excluded from source pretraining and used only for lightweight alignment-layer adaptation. Empirically, our unified model achieves strong semantic reconstruction across datasets (e.g., up to 94.8% CLIP similarity on NSD (AUG) and 0.403 SSIM on BOLD5000 after lightweight finetuning), demonstrating robust cross-subject and cross-dataset transfer.

References

1. 1.Kashif, M., Ferrante, M., & Toschi, N. (2026). NeuroFusion: A unified framework for generalized visual stimulus decoding from fMRI across datasets and subjects. Neuroinformatics. https://doi.org/10.1109/JPROC.2020.3004555 [Note: DOI as supplied by source metadata — independent verification recommended prior to citation]