Information

Convention for selection of summary slices in fMRI

Convention for selection of summary slices in fMRI



We are searching data for your request:

Forums and discussions:
Manuals and reference books:
Data from registers:
Wait the end of the search in all databases.
Upon completion, a link will appear to access the found materials.

Looking at papers, I often see an fMRI volume summarised by taking a single axial, sagittal and coronal slice (typically in MNI152 space). For an example see Figure 1 in de Bie et al, 2011 [1]. The text of a number of papers I have browsed doesn't explain how the slices are selected.

My question is what methods are conventional for selecting the x,y,z slice co-ordinates in such figures?

[1] https://www.researchgate.net/profile/Alle_meije_Wink/publication/51077270_Resting-state_networks_in_awake_five-_to_eight-year_old_children/links/0fcfd50a5fafa7a0bb000000.pdf


EEG Signatures of Dynamic Functional Network Connectivity States

The human brain operates by dynamically modulating different neural populations to enable goal directed behavior. The synchrony or lack thereof between different brain regions is thought to correspond to observed functional connectivity dynamics in resting state brain imaging data. In a large sample of healthy human adult subjects and utilizing a sliding windowed correlation method on functional imaging data, earlier we demonstrated the presence of seven distinct functional connectivity states/patterns between different brain networks that reliably occur across time and subjects. Whether these connectivity states correspond to meaningful electrophysiological signatures was not clear. In this study, using a dataset with concurrent EEG and resting state functional imaging data acquired during eyes open and eyes closed states, we demonstrate the replicability of previous findings in an independent sample, and identify EEG spectral signatures associated with these functional network connectivity changes. Eyes open and eyes closed conditions show common and different connectivity patterns that are associated with distinct EEG spectral signatures. Certain connectivity states are more prevalent in the eyes open case and some occur only in eyes closed state. Both conditions exhibit a state of increased thalamocortical anticorrelation associated with reduced EEG spectral alpha power and increased delta and theta power possibly reflecting drowsiness. This state occurs more frequently in the eyes closed state. In summary, we find a link between dynamic connectivity in fMRI data and concurrently collected EEG data, including a large effect of vigilance on functional connectivity. As demonstrated with EEG and fMRI, the stationarity of connectivity cannot be assumed, even for relatively short periods.

This is a preview of subscription content, access via your institution.


An fMRI Study of the Interactions Between the Attention and the Gustatory Networks

In a prior study, we showed that trying to detect a taste in a tasteless solution results in enhanced activity in the gustatory and attention networks. The aim of the current study was to use connectivity analyses to test if and how these networks interact during directed attention to taste. We predicted that the attention network modulates taste cortex, reflecting top-down enhancement of incoming sensory signals that are relevant to goal-directed behavior. fMRI was used to measure brain responses in 14 subjects as they performed two different tasks: (1) trying to detect a taste in a solution or (2) passively perceiving the same solution. We used psychophysiological interaction analysis to identify regions demonstrating increased connectivity during a taste attention task compared to passive tasting. We observed greater connectivity between the anterior cingulate cortex and the frontal eye fields, posterior parietal cortex, and parietal operculum and between the anterior cingulate cortex and the right anterior insula and frontal operculum. These results suggested that selective attention to taste is mediated by a hierarchical circuit in which signals are first sent from the frontal eye fields, posterior parietal cortex, and parietal operculum to the anterior cingulate cortex, which in turn modulates responses in the anterior insula and frontal operculum. We then tested this prediction using dynamic causal modeling. This analysis confirmed a model of indirect modulation of the gustatory cortex, with the strongest influence coming from the frontal eye fields via the anterior cingulate cortex. In summary, the results indicate that the attention network modulates the gustatory cortex during attention to taste and that the anterior cingulate cortex acts as an intermediary processing hub between the attention network and the gustatory cortex.

This is a preview of subscription content, access via your institution.


Results

Behavioral Data

The mean proportion of old responses made on the recognition test are shown in Figure 2 as a function of word type (R, F, Foil) these data represent hits for R and F words and false alarms for Foil words. A within-subjects analysis of variance (ANOVA) on these data revealed a significant main effect of word type, F2,18 = 29.94, mean squared error = 0.027, P < 0.01. Planned contrasts revealed a significant directed forgetting effect, wherein participants had fewer recognition hits to F than to R words, F1,18 = 6.02, P < 0.03. Even so, the proportion of “old” responses made to F words was significantly greater than the proportion of such responses made to Foil words, F1,18 = 26.31, P < 0.01. This latter finding argues that the near-chance hit rate for F words (0.57) does not reflect random responding, although participants recognized fewer F than R words they did, in fact, recognize a significant number of F words.

Proportion old responses on the recognition test, as a function of word type (R, F, Foil). Bars depict average group data lines depict individual performance for each of the 10 participants.

Proportion old responses on the recognition test, as a function of word type (R, F, Foil). Bars depict average group data lines depict individual performance for each of the 10 participants.

FMRI Data

Scans acquired during the study phase were sorted post hoc based on hits/misses made during the recognition of R and F words. (Sorting the data in this way meant that, for some participants, there were relatively few trials in each cell of the design matrix. To assess the power of the effects reported here, we calculated the effect size (Cohen's d) for our results. The effect sizes ranged from 2.18 to greater than 5. That is, even the smallest effect size in these data was very robust (a “large” effect size is d ≥ 0.8.) This allowed us to determine how activations revealed during the study were related to the intention to remember or forget and to the success in instantiating those intentions. These data are summarized in Table 1.

The data were analyzed using a 3-factor ANOVA (3dANOVA3, part of the AFNI suite of image analysis programs). The factors were Memory Instruction (R vs. F), Outcome (Remember vs. Forget) and Subject, which was a random factor. In addition to the main effects and interaction, 4 planned contrasts were included: R-Remember–R-Forget F-Forget–F-Remember R-Remember–F-Remember and F-Forget–R-Forget.

Memory Instruction

Here, we examined the difference in activation on R instruction and F instruction trials. Whereas greater activity for R than F trials was found in middle frontal gyrus and insula (see also Reber et al. 2002), we found greater activity for F than R trials in frontal and medial temporal areas (see Fig. 3). As shown in Table 1, these areas included activations in middle frontal gyrus, middle cingulate gyrus, middle temporal gyrus, and parahippocampal gyrus. If intentional forgetting were achieved simply by passive decay (i.e., the obverse of intentional remembering), this comparison would have revealed few, if any, unique activations. The fact that there were unique activations—and in areas involved in memory formation (see Buckner et al. 1999)—supports the view that active cognitive processes are engaged by an F instruction intentional forgetting is not simply a failure to intentionally remember.

The main effect of Memory Instruction. The orthogonal pictures at the left detail the hippocampal activation. The orthogonal pictures at the right detail the frontal activation. These activations are also shown on the reconstruction in the center.

The main effect of Memory Instruction. The orthogonal pictures at the left detail the hippocampal activation. The orthogonal pictures at the right detail the frontal activation. These activations are also shown on the reconstruction in the center.

Encoding activations to survive correction for multiple comparisons

BA X Y Z Volume
Main effects
F-instructions > R-instructions
Superior medial, frontal gyrus BA10 −6 67 12 6574
Middle frontal gyrus BA6 −34 27 60 1835
Middle cingulate gyrus BA31 18 −33 40 2293
Middle/superior temporal gyrus BA39 58 −65 24 5198
Middle temporal gyrus BA21 66 −17 −8 3134
Parahippocampal gyrus BA34 −14 −5 −20 5580
Parahippocampal gyrus BA35 18 −25 −20 3669
R-instructions > F-instructions
Middle frontal gyrusBA9−263516192
InsulaBA1330716512
Forgotten > Remembered
Inferior frontal gyrus BA47 42 23 −8 1987
Putamen −26 −1 4 1605
Remembered > Forgotten
Inferior frontal gyrusBA45−58150192
Inferior parietal lobule BA40 −34 −37 32 1605
Parahippocampal gyrus BA30 −26 −53 12 1758
Parahippocampal gyrus/hippocampusBA3626330640
Parahippocampal gyrus BA19 34 −53 0 1529
Lingual gyrus/cerebellum (declive) BA18 6 −69 −24 2752
Interaction
Instruction × outcome
Inferior frontal gyrus BA47 50 19 −8 1835
Insula/Inferior frontal gyrus BA13 46 3 16 1987
Inferior parietal lobule BA40 −38 −29 28 1605
Thalamus/pulvinar −6 −21 12 1911
Inferior frontal gyrus BA45 −54 35 0 1452
Parahippocampal gyrus BA36 −34 −33 −20 2675
Superior frontal gyrus BA10 −30 59 28 1376
Inferior temporal gyrus BA20 54 −21 −20 1682
Postcentral Gyrus BA1/3 −54 −17 52 2446
Contrasts
R-Remember > R-Forget
Middle frontal gyrus BA46 −54 35 32 1835
Inferior parietal lobule BA40 −34 −37 32 1987
Cingulate gyrus BA32 26 15 24 2675
Parahippocampal gyrus BA30 −10 −29 44 2370
Perirhinal cortexBA20−36−10−22192
Middle occipital gyrus BA18 −26 −89 4 2905
Caudate body −14 −13 24 1529
Precuneus/caudate tail −26 −49 16 2370
Thalamus/pulvinar −2 −25 8 2370
Cerebellum/declive −34 −77 −28 2523
F-Forget > F-Remember
Superior/middle frontal gyrus BA10 −30 55 24 1452
Inferior frontal gyrus BA47 46 23 –8 6650
Postcentral gyrus BA4 54 −13 32 1758
Putamen −26 −1 4 1835
R-Remember > F-Remember
Superior frontal gyrus BA9 −34 35 32 2140
Postcentral gyrus BA3 −46 −13 56 1376
F-Forget > R-Forget
Superior frontal gyrus BA10/11 18 63 −12 2599
Inferior/middle frontal gyrus BA9 26 15 28 1376
Inferior parietal lobule BA40 −58 −57 48 2599
Parahippocampal gyrus/hippocampus BA35 22 −17 −12 4204
Posterior cingulateBA3122−2940320
BA X Y Z Volume
Main effects
F-instructions > R-instructions
Superior medial, frontal gyrus BA10 −6 67 12 6574
Middle frontal gyrus BA6 −34 27 60 1835
Middle cingulate gyrus BA31 18 −33 40 2293
Middle/superior temporal gyrus BA39 58 −65 24 5198
Middle temporal gyrus BA21 66 −17 −8 3134
Parahippocampal gyrus BA34 −14 −5 −20 5580
Parahippocampal gyrus BA35 18 −25 −20 3669
R-instructions > F-instructions
Middle frontal gyrusBA9−263516192
InsulaBA1330716512
Forgotten > Remembered
Inferior frontal gyrus BA47 42 23 −8 1987
Putamen −26 −1 4 1605
Remembered > Forgotten
Inferior frontal gyrusBA45−58150192
Inferior parietal lobule BA40 −34 −37 32 1605
Parahippocampal gyrus BA30 −26 −53 12 1758
Parahippocampal gyrus/hippocampusBA3626330640
Parahippocampal gyrus BA19 34 −53 0 1529
Lingual gyrus/cerebellum (declive) BA18 6 −69 −24 2752
Interaction
Instruction × outcome
Inferior frontal gyrus BA47 50 19 −8 1835
Insula/Inferior frontal gyrus BA13 46 3 16 1987
Inferior parietal lobule BA40 −38 −29 28 1605
Thalamus/pulvinar −6 −21 12 1911
Inferior frontal gyrus BA45 −54 35 0 1452
Parahippocampal gyrus BA36 −34 −33 −20 2675
Superior frontal gyrus BA10 −30 59 28 1376
Inferior temporal gyrus BA20 54 −21 −20 1682
Postcentral Gyrus BA1/3 −54 −17 52 2446
Contrasts
R-Remember > R-Forget
Middle frontal gyrus BA46 −54 35 32 1835
Inferior parietal lobule BA40 −34 −37 32 1987
Cingulate gyrus BA32 26 15 24 2675
Parahippocampal gyrus BA30 −10 −29 44 2370
Perirhinal cortexBA20−36−10−22192
Middle occipital gyrus BA18 −26 −89 4 2905
Caudate body −14 −13 24 1529
Precuneus/caudate tail −26 −49 16 2370
Thalamus/pulvinar −2 −25 8 2370
Cerebellum/declive −34 −77 −28 2523
F-Forget > F-Remember
Superior/middle frontal gyrus BA10 −30 55 24 1452
Inferior frontal gyrus BA47 46 23 –8 6650
Postcentral gyrus BA4 54 −13 32 1758
Putamen −26 −1 4 1835
R-Remember > F-Remember
Superior frontal gyrus BA9 −34 35 32 2140
Postcentral gyrus BA3 −46 −13 56 1376
F-Forget > R-Forget
Superior frontal gyrus BA10/11 18 63 −12 2599
Inferior/middle frontal gyrus BA9 26 15 28 1376
Inferior parietal lobule BA40 −58 −57 48 2599
Parahippocampal gyrus/hippocampus BA35 22 −17 −12 4204
Posterior cingulateBA3122−2940320

Note: Activations are given in Talairach coordinates (x, y, z) and volume is in mm 3 . Areas in italics are replications of previous findings.

Encoding activations to survive correction for multiple comparisons

BA X Y Z Volume
Main effects
F-instructions > R-instructions
Superior medial, frontal gyrus BA10 −6 67 12 6574
Middle frontal gyrus BA6 −34 27 60 1835
Middle cingulate gyrus BA31 18 −33 40 2293
Middle/superior temporal gyrus BA39 58 −65 24 5198
Middle temporal gyrus BA21 66 −17 −8 3134
Parahippocampal gyrus BA34 −14 −5 −20 5580
Parahippocampal gyrus BA35 18 −25 −20 3669
R-instructions > F-instructions
Middle frontal gyrusBA9−263516192
InsulaBA1330716512
Forgotten > Remembered
Inferior frontal gyrus BA47 42 23 −8 1987
Putamen −26 −1 4 1605
Remembered > Forgotten
Inferior frontal gyrusBA45−58150192
Inferior parietal lobule BA40 −34 −37 32 1605
Parahippocampal gyrus BA30 −26 −53 12 1758
Parahippocampal gyrus/hippocampusBA3626330640
Parahippocampal gyrus BA19 34 −53 0 1529
Lingual gyrus/cerebellum (declive) BA18 6 −69 −24 2752
Interaction
Instruction × outcome
Inferior frontal gyrus BA47 50 19 −8 1835
Insula/Inferior frontal gyrus BA13 46 3 16 1987
Inferior parietal lobule BA40 −38 −29 28 1605
Thalamus/pulvinar −6 −21 12 1911
Inferior frontal gyrus BA45 −54 35 0 1452
Parahippocampal gyrus BA36 −34 −33 −20 2675
Superior frontal gyrus BA10 −30 59 28 1376
Inferior temporal gyrus BA20 54 −21 −20 1682
Postcentral Gyrus BA1/3 −54 −17 52 2446
Contrasts
R-Remember > R-Forget
Middle frontal gyrus BA46 −54 35 32 1835
Inferior parietal lobule BA40 −34 −37 32 1987
Cingulate gyrus BA32 26 15 24 2675
Parahippocampal gyrus BA30 −10 −29 44 2370
Perirhinal cortexBA20−36−10−22192
Middle occipital gyrus BA18 −26 −89 4 2905
Caudate body −14 −13 24 1529
Precuneus/caudate tail −26 −49 16 2370
Thalamus/pulvinar −2 −25 8 2370
Cerebellum/declive −34 −77 −28 2523
F-Forget > F-Remember
Superior/middle frontal gyrus BA10 −30 55 24 1452
Inferior frontal gyrus BA47 46 23 –8 6650
Postcentral gyrus BA4 54 −13 32 1758
Putamen −26 −1 4 1835
R-Remember > F-Remember
Superior frontal gyrus BA9 −34 35 32 2140
Postcentral gyrus BA3 −46 −13 56 1376
F-Forget > R-Forget
Superior frontal gyrus BA10/11 18 63 −12 2599
Inferior/middle frontal gyrus BA9 26 15 28 1376
Inferior parietal lobule BA40 −58 −57 48 2599
Parahippocampal gyrus/hippocampus BA35 22 −17 −12 4204
Posterior cingulateBA3122−2940320
BA X Y Z Volume
Main effects
F-instructions > R-instructions
Superior medial, frontal gyrus BA10 −6 67 12 6574
Middle frontal gyrus BA6 −34 27 60 1835
Middle cingulate gyrus BA31 18 −33 40 2293
Middle/superior temporal gyrus BA39 58 −65 24 5198
Middle temporal gyrus BA21 66 −17 −8 3134
Parahippocampal gyrus BA34 −14 −5 −20 5580
Parahippocampal gyrus BA35 18 −25 −20 3669
R-instructions > F-instructions
Middle frontal gyrusBA9−263516192
InsulaBA1330716512
Forgotten > Remembered
Inferior frontal gyrus BA47 42 23 −8 1987
Putamen −26 −1 4 1605
Remembered > Forgotten
Inferior frontal gyrusBA45−58150192
Inferior parietal lobule BA40 −34 −37 32 1605
Parahippocampal gyrus BA30 −26 −53 12 1758
Parahippocampal gyrus/hippocampusBA3626330640
Parahippocampal gyrus BA19 34 −53 0 1529
Lingual gyrus/cerebellum (declive) BA18 6 −69 −24 2752
Interaction
Instruction × outcome
Inferior frontal gyrus BA47 50 19 −8 1835
Insula/Inferior frontal gyrus BA13 46 3 16 1987
Inferior parietal lobule BA40 −38 −29 28 1605
Thalamus/pulvinar −6 −21 12 1911
Inferior frontal gyrus BA45 −54 35 0 1452
Parahippocampal gyrus BA36 −34 −33 −20 2675
Superior frontal gyrus BA10 −30 59 28 1376
Inferior temporal gyrus BA20 54 −21 −20 1682
Postcentral Gyrus BA1/3 −54 −17 52 2446
Contrasts
R-Remember > R-Forget
Middle frontal gyrus BA46 −54 35 32 1835
Inferior parietal lobule BA40 −34 −37 32 1987
Cingulate gyrus BA32 26 15 24 2675
Parahippocampal gyrus BA30 −10 −29 44 2370
Perirhinal cortexBA20−36−10−22192
Middle occipital gyrus BA18 −26 −89 4 2905
Caudate body −14 −13 24 1529
Precuneus/caudate tail −26 −49 16 2370
Thalamus/pulvinar −2 −25 8 2370
Cerebellum/declive −34 −77 −28 2523
F-Forget > F-Remember
Superior/middle frontal gyrus BA10 −30 55 24 1452
Inferior frontal gyrus BA47 46 23 –8 6650
Postcentral gyrus BA4 54 −13 32 1758
Putamen −26 −1 4 1835
R-Remember > F-Remember
Superior frontal gyrus BA9 −34 35 32 2140
Postcentral gyrus BA3 −46 −13 56 1376
F-Forget > R-Forget
Superior frontal gyrus BA10/11 18 63 −12 2599
Inferior/middle frontal gyrus BA9 26 15 28 1376
Inferior parietal lobule BA40 −58 −57 48 2599
Parahippocampal gyrus/hippocampus BA35 22 −17 −12 4204
Posterior cingulateBA3122−2940320

Note: Activations are given in Talairach coordinates (x, y, z) and volume is in mm 3 . Areas in italics are replications of previous findings.

Memory Outcome

The results of our ANOVA also revealed main effects of memory outcome. As shown in Table 1, frontal and medial temporal areas were distinguished between encoding trials that resulted in later recognition from encoding trials that did not. The inferior frontal gyrus, in particular, showed increased activation on encoding trials that led to later forgetting versus those that led to remembering (see also Reber et al. 2002). Conversely, the inferior parietal lobule and parahippocampal gyrus showed greater activity on encoding trials for which the later outcome would be remembering versus trials for which the later outcome would be forgetting.

Memory Instruction by Outcome

Memory instruction and outcome interacted significantly in several key memory-related areas. These included inferior frontal gyrus, insula, inferior parietal lobule, thalamus/pulvinar, parahippocampal gyrus, superior frontal gyrus, and inferior temporal gyrus (see also Table 1). Of these activations, only that in right, inferior parietal regions (Broadmann area [BA] 47) showed increased activity that was specific to the F-Forget condition (see Fig. 4). The insula (BA13), left-sided inferior parietal (BA40), and thalamus/pulvinar regions showed increased activity that might be characterized as predicting successful instantiation of intentions (see Fig. 5). These regions showed more activity in the F-Forget than the F-Remember condition and more activity in the R-Remember than the R-Forget condition. Thus, regardless of whether the intention was to forget or remember the item, these areas were more active when the intention was later successfully implemented relative to when it was not. Conversely, activity in the inferior frontal (BA45) and parahippocampal (BA 36) gyri appeared to predict failure to instantiate an intention (see Fig. 6). Activity in these areas was greater for F-Remember than for F-Forget and for R-Forget than for R-Remember. Contrast analyses were conducted to better understand the activity in the regions showing reliable interaction effects.

The interaction between Memory Instruction (Cue) and Outcome. The crosshairs indicate an area in right inferior frontal gyrus (BA47) that is preferentially active in the F-Forget (directed forgetting) condition.

The interaction between Memory Instruction (Cue) and Outcome. The crosshairs indicate an area in right inferior frontal gyrus (BA47) that is preferentially active in the F-Forget (directed forgetting) condition.

The interaction between Memory Instruction (Cue) and Outcome. The crosshairs indicate an area in right insula/inferior frontal gyrus that was more active when participants were going to succeed in their intention (F-forget or R-remember) than when they were going to fail (R-forget or F-remember). The activation in this area is graphed in the lower right. The same pattern was evident in the thalamus, which can be seen in the axial slice and is denoted by the arrow.

The interaction between Memory Instruction (Cue) and Outcome. The crosshairs indicate an area in right insula/inferior frontal gyrus that was more active when participants were going to succeed in their intention (F-forget or R-remember) than when they were going to fail (R-forget or F-remember). The activation in this area is graphed in the lower right. The same pattern was evident in the thalamus, which can be seen in the axial slice and is denoted by the arrow.

The interaction between Memory Instruction (Cue) and Outcome. The crosshairs indicate an area in left inferior frontal gyrus that was more active when participants were going to fail in their intention (R-forget or F-remember) than when they were going to succeed (F-forget or R-remember). The activation in this area is graphed in the lower right.

The interaction between Memory Instruction (Cue) and Outcome. The crosshairs indicate an area in left inferior frontal gyrus that was more active when participants were going to fail in their intention (R-forget or F-remember) than when they were going to succeed (F-forget or R-remember). The activation in this area is graphed in the lower right.

Intentional Remembering

To assess the network underlying successful remembering, we compared study trials on which the attempt to intentionally commit a word to memory later resulted in recognition success to those that later resulted in failure (R-Remember–R-Forget). Our results replicate previous findings of activity in perirhinal cortex ( Davachi et al. 2003). This contrast also revealed that in addition to activity in visual areas and the cerebellum, there was more activity in the middle frontal gyrus and in the cingulate gyrus while participants were encoding an R word they would later correctly recognize compared with one they would later forget (see Table 1). The same was true of the inferior parietal lobule and the thalamus/pulvinar.

Intentional Forgetting

To assess the areas associated with intentional forgetting, we compared F trials at study that later resulted in successful forgetting with those that resulted in failure to forget (F-Forget–F-Remember). According to the common incarnation of the selective rehearsal account, forgetting occurs in an item-method paradigm because participants drop F items from the rehearsal set and therefore fail to fully encode them (passive decay). If this were the case, then the incidental remembering that occurred for F-Remember items should be associated with mechanisms involved in episodic encoding by comparison, successful forgetting in the F-Forget condition should either not involve these mechanisms at all or should engage them weakly and/or for a shorter duration. Thus, if successful intentional forgetting were due simply to a lack of encoding, then subtracting the activation associated with F-Remember items from the activation associated with F-Forget items should result in no positive activations. Instead, one would expect this contrast to yield “negative” activations in areas associated with episodic encoding. This is not what we found, however. When the intention to forget successfully prevented a word from being committed to memory, there was more activity in several areas compared with when the intention failed and the word was later recognized (see Table 1). These areas included the superior/middle frontal gyrus and inferior frontal gyrus. Importantly, inferior frontal gyrus has also been implicated in successful stopping of overt behavior (e.g., Aron et al. 2003).

Intentional Forgetting versus Unintentional Forgetting

If it were the case that intentional forgetting resulted from a simple failure to encode the forgotten F word, then similar failures to encode R words would be expected to result in similar patterns of neural activity. This was not the case, however. We compared study trials on which participants were instructed to forget the word and subsequently failed to recognize it with study trials on which they were instructed to remember the word, but subsequently failed to recognize it (F-Forget–R-Forget). As shown in Figure 7 (and also in Table 1), this comparison revealed increased activity in the parahippocampal gyrus/hippocampus and the superior frontal gyrus. The fact that unique activations were revealed after this subtraction confirms that intentional forgetting and unintentional forgetting are distinguishable. Nevertheless, this comparison revealed that some of the activity associated with intentional forgetting occurred in the network associated with unintentional forgetting ( Wagner and Davachi 2001) of the regions previously implicated in unintentional forgetting (posterior cingulate, bilateral inferior parietal, and medial parietal cortices), we found activity in the inferior parietal lobule and posterior cingulate cortex.

The F-Forget–R-Forget comparison. Activations to survive multiple corrections included hippocampus and frontal gyrus.

The F-Forget–R-Forget comparison. Activations to survive multiple corrections included hippocampus and frontal gyrus.

Intentional Remembering versus Unintentional Remembering

A similar (exploratory) contrast was performed to compare intentional and unintentional remembering. We examined whether activity during study differentiates remembering that occurs intentionally and remembering that occurs in the absence of such an intention (R-Remember–F-Remember). In both cases, the behavioral outcome was the same: A study word was correctly recognized as old at test. However, in the case of the R-Remember condition, the word was remembered because the participant intended to remember it in the case of the F-Remember condition, the word was remembered despite the fact that the participant intended to forget it. In this comparison, intentional remembering was associated with greater activity in superior frontal gyrus and in postcentral gyrus (see Table 1).


1 Introduction

Since its introduction, multivariate pattern analysis (MVPA)—or informally, neural ‘decoding’—has had a transformative influence on cognitive neuroscience. Methodologically, it is a veritable multi-tool that provides a unified approach for analysing data from cellular recordings, fMRI, EEG, and MEG, which can also be paired with computational modelling and behavioural paradigms (Kriegeskorte et al. [2008]). Theoretically, it is often presented as a means for investigating the structure and content of the brain's population code, thereby unifying psychological and neuroscientific explanations while predicting behavioural performance (Kriegeskorte and Kievit [2013] Haxby et al. [2014]). More ambitiously still, decoding methods are advertised as a means of ‘reading’ the brain and ‘listening’ in on the mind (Haynes and Rees [2006] Norman et al. [2006]).

Underlying these bold pronouncements is a crucial inference, which we call the decoder's dictum:

If information can be decoded from patterns of neural activity, then this provides strong evidence about what information those patterns represent.

The decoder’s dictum should interest philosophers for two reasons. First, a central philosophical issue with neuroimaging is its use in ‘reverse inferences’ about mental function (Poldrack [2006] Klein [2010]). The decoder's dictum is a similar but more nuanced form of inference, so it deserves careful scrutiny. Second, decoding results are some of the most compelling in cognitive neuroscience, and offer a wellspring of findings that philosophers may want to tap into when defending theoretical claims about the architecture of the mind and brain. 1 It is therefore worth clarifying what decoding can really show.

We argue that the decoder’s dictum is false. The dictum is underwritten by the idea that uncovering information in neural activity patterns, using ‘biologically plausible’ MVPA methods that are similar to the decoding procedures of the brain, is sufficient to show that this information is neurally represented and functionally exploitable. However, as we are typically ignorant of the precise information exploited by these methods, we cannot infer that the information decoded is the same information the brain exploits. Thus decodability is not (by itself) a reliable guide to neural representation. Our goal is not to reprimand neuroscientists for how they currently employ and interpret MVPA. Rather, what follows will clarify the conditions under which decoding could provide evidence about neural representation.

By analogy, consider research on brain–machine interface (BMI) systems, which use decoding to generate control signals for computer cursors or prosthetic limbs (Hatsopoulos and Donoghue [2009]). Largely because of BMI’s engineering and translational objectives, however, little attention is paid to the biological plausibility of decoding methods. Consequently, BMI research does not involve inferences about neural function based on decodability. We believe that, epistemically, decoding in cognitive neuroscience is typically no better off than in BMI research, and so forms a thin basis for drawing inferences about neural representation.

Our focus is on how MVPA is used to investigate neural representations. Since talk of representation is itself philosophically contentious, we assume a relatively lightweight notion that is consistent with usage in the relevant sectors of neuroscience: a representation is any internal state of a complex system that serves as a vehicle for informational content and plays a functional role within the system based on the information that it carries (Bechtel [1998]). This notion of representation is built into the idea that an internal state of a system encodes information (that is, represents informational content), which is then decoded for later use by a system (based on the functional import of the information that is encoded). 2 Thus, in talking of a ‘representation with informational content’, we simply have in mind a state that encodes information for subsequent decoding. As we shall see, some researchers talk of decoding mental representations. We assume they have in mind at least the notion of internal representation we have articulated, so our arguments apply to their claims as well.

We focus on neural representations that take the form of population codes. A population code represents information through distributed patterns of activity occurring across a number of neurons. In typical population coding models, each individual neuron exhibits a distribution of responses over some set of inputs, and for any given input, the joint or combined response across the entire neural population encodes information about the input parameters (Pouget et al. [2000]).

Our critique of the dictum will take some set-up. In Section 2, we provide a brief introduction to decoding methods. In Section 3, we argue that the dictum is false: the presence of decodable information in patterns of neural activity does not show that the brain represents that information. Section 4 expands on this argument by considering possible objections. In Section 5, we suggest a way to move beyond the dictum. Section 6 concludes the article.


There is considerable evidence that there are anatomically and functionally distinct pathways for action and object recognition. However, little is known about how information about action and objects is integrated. This study provides fMRI evidence for task-based selection of brain regions associated with action and object processing, and on how the congruency between the action and the object modulates neural response. Participants viewed videos of objects used in congruent or incongruent actions and attended either to the action or the object in a one-back procedure. Attending to the action led to increased responses in a fronto-parietal action-associated network. Attending to the object activated regions within a fronto-inferior temporal network. Stronger responses for congruent action–object clips occurred in bilateral parietal, inferior temporal, and putamen. Distinct cortical and thalamic regions were modulated by congruency in the different tasks. The results suggest that (i) selective attention to action and object information is mediated through separate networks, (ii) object–action congruency evokes responses in action planning regions, and (iii) the selective activation of nuclei within the thalamus provides a mechanism to integrate task goals in relation to the congruency of the perceptual information presented to the observer.

The ability to manipulate objects and tools is crucial for everyday life. This ability depends on an interaction between the properties of the object and the required action. Object-related actions involve on-line guidance of our effectors in response to these objects, thus object affordances (the potency of the object for action) may be an inherent part of object perception (Tucker & Ellis, 1998 Gibson, 1979). This study is concerned with how we process action, objects, and their relations.

Traditionally, theories have stressed that the retrieval of an appropriate action for an object is guided by access to semantic knowledge based on an object's associations and its abstract function (e.g., Ochipa, Rothi, & Heilman, 1992 Roy & Square, 1985). For instance, a cup activates the action of drinking through access to semantic knowledge based on our prior associations with how cups are used and what they are used for. However, there is increasing neuropsychological and experimental evidence that access to action information can be evoked by the visual properties of objects in a relatively direct way, without the necessary involvement of semantic (associative) memory (see Humphreys et al., 2010 Humphreys & Riddoch, 2003, for reviews also see Barsalou, 1999, for a similar view derived from a different literature). This evidence provides the backdrop for this study, in which we had participants make 1-back judgments on images of objects being manipulated by congruent or incongruent actions. We examine first the distinct neural correlates of action and object-related processing invoked by attention to actions and to objects. Subsequently, we examine the neural basis of the interaction between action and object pathways driven by the congruency of the action–object pairing.

Neuropsychological research suggests that knowledge of actions (how to manipulate objects) is dissociated from knowledge of objects (what the object is). Ferreira and colleagues (Ferreira, Giusiano, Ceccaldi, & Poncet, 1997) report a patient who was impaired at naming objects but unimpaired at naming actions and who was also able to name the objects from gesture information. Additional patients with damage to the left occipito-temporal cortex, who have the clinical presentation of optic aphasia, are able to gesture the correct action to objects they cannot recognize, even when access to associative semantic information is impaired (Hillis & Caramazza, 1995 Riddoch & Humphreys, 1987). In such cases, vision is not simply used for on-line control of prehensile actions but also to access information about the category of action that can be performed. The results indicate that responding to action-related associations to objects can be dissociated from semantic recognition processes.

In contrast, there are other patients who have an intact access to semantic knowledge of objects, showing intact recognition and naming. These patients can produce correct action in response to the object's name, but they nevertheless fail to correctly act when objects are presented visually (e.g., Pilgrim & Humphreys, 1991 Riddoch, Humphreys, & Price, 1989 DeRenzi, Faglioni, & Sorgato, 1982). In these cases, visual access to semantics is preserved but visual access to action-related responses is blocked, though they are able to access action information via the indirect semantic route (e.g., from the object's name).

Behavioral experiments with healthy participants also provide evidence for direct action-related and indirect semantic-related responses to objects (e.g., Yoon, Humphreys, & Riddoch, 2010 Yoon & Humphreys, 2005, 2007 Chainay & Humphreys, 2002). Decisions about which action to perform on an object are faster when made on pictures of objects than on object names, whereas decisions about the semantic context associated with objects are not affected by the mode of presentation (Chainay & Humphreys, 2002). Also in contrast with semantic categorical decisions, action decisions are not affected by semantic priming but are affected by the orientation of the handle to viewers (Yoon & Humphreys, 2007) and by the correct relative positions of paired objects within an observer's egocentric reference frame (Yoon et al., 2010).

The evidence indicate that access to action and semantic information about objects can dissociate. The interrelations between objects and action, however, can affect responses based on the action and semantic recognition routes (Yoon & Humphreys, 2005). Notably, using stimuli similar to those employed in our study, Yoon and Humphreys found that both action and semantic decisions to objects are facilitated if stimuli are presented with a congruent object grip and if the object is used appropriately compared with when the grip or the action is inappropriate to the object.

We note that the behavioral differences observed for accessing action-related versus semantic contextual knowledge occur even when explicit actions are not made by the participants to the objects. It is nevertheless possible that action-related effects are contingent on the activation of motor actions to objects modulated through the so-called mirror neuron system, commonly involved in action production and recognition. There is evidence that viewing actions evokes a simulated response in motor cortex (for a recent review, Rizzolatti & Sinigaglia, 2010). The neural areas comprising the action observation network (Grafton, 2009) include the bilateral STS, the inferior parietal lobule (IPL), the inferior frontal gyrus (IFG), and the premotor cortex (PM). In addition, the SMA (Dayan et al., 2007 Hamilton & Grafton, 2007), BG, and cerebellum (Blakemore, Frith, & Wolpert, 2001 Wolpert, Miall, & Kawato, 1998) have also been implicated in action simulation. Interestingly, Humphreys et al. (2010) report increased activity in dorsal PM when participants view objects gripped in a congruent relative to an incongruent manner, suggesting that motor-based simulation may also be sensitive to the interaction between action and object information.

Access to action-related information (how/where to grasp objects) and to semantic knowledge (what) has been linked to the dorsal and ventral visual streams (Milner & Goodale, 1995 Ungerleider & Mishkin, 1982). For example, Shmuelof and Zohary (2005) reported that, when participants see objects being manipulated, then recognition of the action is associated with activity in dorsal areas, including the anterior intraparietal sulcus (aIPS) while in contrast, recognition of the object is linked to activity within ventral regions including the fusiform gyrus (e.g., Grill-Spector, 2003 Chao, Haxby, & Martin, 1999 Grill-Spector, Kushnir, Edelman, Itzchak, & Malach, 1998). Consistent with this, the aIPS shows adaptation of activity when the same grasp information is viewed repeatedly, whereas adaptation in the fusiform gyrus depends on the repetition of the identity and form of the object (Shmuelof & Zohary, 2005). The aIPS appears insensitive to the long-term familiarity of the grasp, including whether a grasp is congruent with the correct action (Valyear & Culham, 2009). In other studies, observation of object manipulation has been associated with bilateral activation of the PM along with the IPS (e.g., Tunik, Rice, Hamilton, & Grafton, 2007 Buccino et al., 2001, for reviews), whereas the IFG is bilaterally involved when participants view static images of an object being grasped by a hand irrespective of whether it is held correctly or incorrectly for object use (Johnson-Frey et al., 2003). Valyear and Culham (2009), however, argue that grasp information is also encoded by the ventral stream regions. Using an ROI approach, they report that regions sensitive to objects, body parts, and motions in the occipital–temporal cortex are sensitive to whether stimuli are correctly grasped for actions.

In addition to cases where participants see actions performed on objects, objects that are inherently associated with actions, tools, and manipulable objects are reported to elicit responses in the dorsal action associated network. Specifically, the left anterior supramarginal gyrus (SMG the rostral part of IPL) is suggested to code motor programs for object use (Peeters et al., 2009). The left anterior SMG, along with the posterior SMG/angular gyrus, is associated with planning the use of familiar objects (Johnson-Frey, Newman-Norlund, & Grafton, 2005). Damage to the SMG (e.g., Sunderland, Wilkins, & Dineen, 2011 Randerath, Goldenberg, Spijkers, Li, & Hermsdoerfer, 2010), along with virtual lesions of this region (following TMS), is linked to poor use of tools. Observation of manipulable objects has also been shown to elicit activity in additional regions including the left inferior frontal lobe, the posterior middle temporal gyrus, and the posterior parietal cortex (e.g., Beauchamp, Lee, Haxby, & Martin, 2002 Devlin et al., 2002 Grèzes & Decety, 2002). However, it is unclear how responses in these regions are affected by attention and the congruency of action–object relations.

In the current study, we investigated the neural correlates of action and object processing and their interaction in response to video clips of objects being used in a congruent or incongruent manner. To selectively evoke action versus object (semantic) processing, we manipulated the attended property of the combined stimuli (similar to Yoon & Humphreys, 2005). On identical video stimuli, participants were asked to perform a 1-back task on the action or on the object. To further facilitate the involvement of the semantic route and to ensure that the comparison across objects could not be based on simple visual features, the object-related 1-back task required object recognition of two different exemplars of an object category (e.g., two different cups). The congruency of the action was manipulated to gain a better understanding of how action and object information is integrated. We manipulated the congruency of the action to the objects. Thus, actions could be congruent with the identity of the object (hitting with a hammer) or incongruent (twisting a hammer). We hypothesized that responses would be elicited in the dorsal action network when participants selectively attended to the action, whereas the ventral object network would be activated by attending to the objects. We further hypothesized that the involvement of these networks would be greater for congruent than for incongruent object–action relations, as congruent action–object relations lend themselves more easily to motor simulations and facilitate action and object recognition (Yoon & Humphreys, 2005). A final question of interest concerned the interaction between task and congruency. What brain areas enable information about the congruency of the action to modulate the object network and are different regions recruited to enable the congruency of the object to modulate the action network? Here we expected to find regions that take information about both objects and actions and respond differentially according to whether this information is congruent or incongruent. We speculated that such regions will be interconnected with the action observation and object-associated networks.


Results

Characteristics of Included Studies

Fifty-three high-quality functional task contrasts from 16 independent samples from 24 fMRI studies were included in the main meta-analysis. The main meta-analysis comprised 338 youths with disruptive behavior disorder or conduct problems (the disruptive/conduct problems group) (mean age, 15.2 years mean age range, 11.9–17.7 years 80% male) and 298 control subjects (mean age, 15.0 years mean age range, 11.3–17.9 years 80% male), taking overlaps into account (Table 1 see also Table S1 in the online data supplement). Five studies (four testing emotion processing and one testing pain empathic processing) assessed conduct problems dimensionally without providing a clinical diagnosis (26, 27, 32, 37, 44). Across nine studies, there were 108 participants with disruptive behavior or conduct problems and psychopathic traits and 115 healthy control subjects. Most (N=11) but not all studies (18, 26, 27, 32, 37, 43, 44) reported ADHD comorbidity rates (0%−88% most were greater than 50%). Twenty-two hot executive function task contrasts were used to create 10 independent brain maps (171 cases, 177 controls), 10 cool executive function task contrasts created four independent brain maps (60 cases, 70 controls), and 17 emotion processing contrasts created eight independent brain maps (169 cases, 130 controls).

Main Meta-Analysis

The disruptive/conduct problems group, compared with the control group, showed significantly decreased activation in a cluster comprising the dorsal and rostral anterior cingulate and medial prefrontal cortex, extending into the supplementary motor area and ventral caudate. Case subjects, compared with control subjects, showed no significantly increased activations (Table 2A, Figure 1A, and Figure 2A).

TABLE 2. Results of the Meta-Analysis of Whole-Brain fMRI Studies in Youths With Disruptive Behavior Disorder or Severe Conduct Problems (DBD/CP) Compared With Healthy Control Subjects Including All Tasks, by Cognitive Subdomain and Presence of Psychopathic Traits a

a BA=Brodmann’s area dACC=dorsal anterior cingulate cortex DLPFC=dorsolateral prefrontal cortex dMPFC=dorsomedial prefrontal cortex FG=fusiform gyrus ITG=inferior temporal gyrus L=left MNI=Montreal Neurological Institute MTG=middle temporal gyrus PT=psychopathic traits/callous unemotional traits R=right rACC=rostral anterior cingulate cortex rMPFC=rostral medial prefrontal cortex SMA=supplementary motor area STG=superior temporal gyrus vMPFC=ventromedial prefrontal cortex VS=ventral striatum.

b Confidence intervals estimated using the inverse of the normal distribution of the p values.

TABLE 2. Results of the Meta-Analysis of Whole-Brain fMRI Studies in Youths With Disruptive Behavior Disorder or Severe Conduct Problems (DBD/CP) Compared With Healthy Control Subjects Including All Tasks, by Cognitive Subdomain and Presence of Psychopathic Traits a

FIGURE 1. Results of the Main Meta-Analysis and of the Subgroup Meta-Analysis of Youths With Disruptive Behavior Disorder or Severe Conduct Problems With Psychopathic Traits a

a In panel A, decreased activation in youths with disruptive behavior disorder or conduct problems compared with healthy control subjects is shown in red in the dorsal and rostral anterior cingulate cortex (ACC), in the dorsal and rostral medial prefrontal cortex (MPFC), and in the supplementary motor area and ventral caudate. In panel B, decreased activation in youths with disruptive behavior disorder or conduct problems with psychopathic traits compared with healthy controls is shown in red in the hypothalamus and thalamus extending into the ventral medial prefrontal cortex and ventral striatum. Increased activation is shown in green in the dorsolateral prefrontal cortex (DLPFC). The increased dorsal caudate activation finding is not shown in Figure 1 but in Figure 2.

FIGURE 2. Axial Sections Showing Regions That Were Significantly Reduced (Red) and Increased (Green) in Youths With Disruptive Behavior Disorder or Conduct Problems (DBD/CP) Relative to Healthy Control Subjects a

a Montreal Neurological Institute z coordinates are indicated for slice distance (in mm) from the intercommissural line. The right side of the image corresponds to the right side of the brain.

Cognitive Subdomain Meta-Analyses

The subgroup meta-analyses showed that, compared with control subjects, youths with disruptive behavior and conduct problems across all hot executive function fMRI data sets had decreased activation in the dorsal anterior cingulate and dorso-medial prefrontal cortex extending into the supplementary motor area, along with increased right dorsal caudate activation (Table 2B, Figure 2B). Across all cool executive function fMRI data sets, they had decreased activation in the right superior and middle temporal gyrus, posterior insula, and putamen (Table 2C, Figure 2C). Across all emotion processing fMRI data sets, they had decreased activation in the right dorsolateral prefrontal cortex and left temporal pole (Table 2D, Figure 2D).

Subgroup Meta-Analysis in the Disruptive/Conduct Problems Group With Psychopathic Traits

The subgroup meta-analysis including only youths with disruptive/conduct problems with psychopathic traits showed decreased activation relative to control subjects in a cluster comprising the hypothalamus and thalamus extending into the ventral striatum and ventromedial prefrontal cortex, in addition to increased activation in the rostral dorsolateral prefrontal cortex and right dorsal caudate (Table 2E, Figure 1B, and Figure 2E).

Findings remained significant when studies with nondiagnosed youths with conduct problems were excluded.

Meta-Regression Analyses of Effects of Age, Medication, Gender, and ADHD

The meta-regression analyses showed that increasing age was associated with progressive hypoactivation in the right dorsolateral prefrontal cortex (Montreal Neurological Institute coordinates: x=50, y=28, z=36 16 voxels), which overlapped with the reduced cluster during emotion processing that medication was associated with increased activation in the temporal and medial frontal regions bilaterally, the cerebellar vermis, and the posterior cingulate/precuneus and with decreased activation in the cerebellar vermis, right insula, and left hippocampus (see Figure S1 in the online data supplement), none of which overlapped with any group difference clusters that male gender was associated with lower activation (i.e., more severe dysfunction than females) in the left anterior cingulate in the disruptive/conduct problems group relative to the control group and that ADHD comorbidity across the 11 available studies with this information was not significantly correlated with neural underactivation relative to control subjects.

Reliability Analyses

Whole-brain jackknife sensitivity analyses showed that the main meta-analysis finding in the dorso-rostral anterior cingulate, medial prefrontal cortex, and ventral caudate was robust and replicable (Table 3), as it was preserved in all but two brain map combinations. For the subgroup meta-analyses, the brain difference findings were preserved in all but one or two combinations of brain maps (see Tables S2–S5 in the online data supplement).

TABLE 3. Results of the Jackknife Reliability Analyses of the Main Meta-Analysis Findings Based on 52 Different Task Contrast Results From 16 Independent Samples a

a ACC=anterior cingulate cortex D=dorsal MNI=Montreal Neurological Institute PFC=prefrontal cortex R=rostral SMA=supplementary motor area yes=brain region remains significantly decreased in the jackknife analysis when the independent sample in question is excluded from the meta-analysis no=brain region is no longer significantly decreased when the independent sample in question is excluded.

TABLE 3. Results of the Jackknife Reliability Analyses of the Main Meta-Analysis Findings Based on 52 Different Task Contrast Results From 16 Independent Samples a

Publication Bias

Funnel plots showed that studies with smaller sample sizes were associated with smaller effect sizes, which is opposite to the association observed in publication bias.


5 Responses to “Using Caret for fMRI Visualization”

Hey man,
what a great tutorial!!
Thanks for sharing your knowledge

sawfoot - September 29th, 2011

Thanks for the tutorial, it was very helpful (and still works in 2011!).

Can anyone help me with the working of CARET. I have already done a meta-analysis so I have all my coordinates in a text document. Can anyone tell me how to put in CARET?

How great this tutorial!
May I translate it into Chinese version and share in my blog?
Thank you!

Can you share the processing steps of adding a paint file? Thank you very much!


Deep learning networks

Deep learning networks (DLNs) have led to a revolution in machine learning and artificial intelligence (Krizhevsky et al., 2012 LeCun et al., 1998 Serre et al., 2007 Szegedy et al., 2015a). DLNs outperform existing approaches on object recognition tasks by training complex multi-layer networks with millions of parameters (i.e., weights) on large databases of natural images. Recently, neuroscientists have become interested in how the computations and representations in these models relate to the ventral stream in monkeys and humans (Cadieu et al., 2014 Dubois et al., 2015 Guclu and van Gerven, 2015 Hong et al., 2016 Khaligh-Razavi and Kriegeskorte, 2014 Yamins et al., 2014 Yamins and DiCarlo, 2016). For these reasons, we choose to examine these models, which also allow for RSA at multiple representational levels.

In this contribution, one key question is whether functional smoothness breaks down at more advanced layers in DLNs as it did in the untrained random neural networks considered in the previous section. We address this question by presenting natural image stimuli (i.e., novel photographs) to a trained DLN, specifically Inception-v3 GoogLeNet (Szegedy et al., 2015b), and applying RSA to evaluate whether the similarity structure of items would be recoverable using fMRI.

Architecture

The DLN we consider, Inception-v3 GoogLeNet, is a convolutional neural network (CNN), which is a type of DLN especially adept at classification and recognition of visual inputs. CNNs excel in computer vision, learning from huge amounts of data. For example, human-like accuracy on test sets has been achieved by: LeNet, a pioneering CNN that identifies handwritten digits (LeCun et al., 1998) HMAX, trained to detect objects, e.g., faces, in cluttered environments (Serre et al., 2007) and AlexNet, which classifies photographs into 1000 categories (Krizhevsky et al., 2012).

The high-level architecture of CNNs consists of many layers (Szegedy et al., 2015a). These are stacked on top of each other, in much the same way as the stacked multilevel perceptrons described previously. A key difference is that CNNs have more variety especially in breadth (number of units) between layers.

In many CNNs, some of the network’s layers are convolutional, which contain components that do not receive input from the whole of the previous layer, but a small subset of it (Szegedy et al., 2015b). Many convolutional components are required to process the whole of the previous layer by creating an overlapping tiling of small patches. Often convolutional layers are interleaved with max-pooling layers (Lecun et al., 1998), which also contain tile-like components that act as local filters over the previous layer. This type of processing and architecture is both empirically driven by what works best, as well as inspired by the visual ventral stream, specifically receptive fields (Fukushima, 1980 Hubel and Wiesel, 1959, 1968 Serre et al., 2007).

Convolutional and max-pooling layers provide a structure that is inherently hierarchical. Lower layers perform computations on small localized patches of the input, while deeper layers perform computations on increasingly larger, more global, areas of the stimuli. After such localized processing, it is typical to include layers that are fully-connected, i.e., are more classically connectionist. And finally, a layer with the required output structure, e.g., units that represent classes or a yes/no response as appropriate.

Inception-v3 GoogLeNet uses a specific arrangement of these aforementioned layers, connected both in series and in parallel (Szegedy et al., 2015b, 2015a, 2016). In total it has 26 layers and 25 million parameters inclusive of connection weights (Szegedy et al., 2015b). The final layer is a softmax layer that is trained to activate a single unit per class. These units correspond to labels that have been applied to sets of photographs by humans, e.g., ‘space shuttle’, ‘ice cream’, ‘sock’, within the ImageNet database (Russakovsky et al., 2015).

Inception-v3 GoogLeNet has been trained on millions of human-labeled photographs from 1000 of ImageNet’s synsets (sets of photographs). The 1000 -unit wide output produced by the network when presented with a photograph represents the probabilities of the input belonging to each of those classes. For example, if the network is given a photograph of a moped it may also activate the output unit that corresponds to bicycle with activation 0.03 . This is interpreted as the network expressing the belief that there is a 3 % probability that the appropriate label for the input is ‘bicycle’. In addition, this interpretation is useful because it allows for multiple classes to co-exist within a single input. For example, a photo with a guillotine and a wig in it will cause it to activate both corresponding output units. Thus the network is held to have learned a distribution of appropriate labels that reflect the most salient items in a scene. Inception-v3 GoogLeNet, achieves human levels of accuracy on test sets, producing the correct label in its five most probable guesses approximately 95 % of the time (Szegedy et al., 2015b).

Deep learning network simulation

We consider whether functional smoothness declines as inputs are processed by the more advanced layers of Inception-v3 GoogLeNet. If so, fMRI should be less successful in brain regions that instantiate computations analogous to the more advanced layers of such networks. Unlike the previous simulations, we present novel photographs of natural categories to these networks. The key question is whether items from related categories (e.g., banjos and guitars) will be similar at various network layers. The 40 photographs (i.e., stimuli) are divided equally amongst 8 subordinate categories: banjos, guitars, mopeds, sportscars, lions, tigers, robins, and partridges, which in turn aggregate into 4 basic-level categories: musical instruments, vehicles, mammals, and birds which in turn aggregate into 2 superordinates: animate and inanimate.

We consider how similar the internal network representations are for pairs of stimuli by comparing the resulting network activity, which is analogous to comparing neural activity over voxels in RSA. Correlations for all possible pairings of the 40 stimuli were calculated for both a mid and a later network layer (see Figure 4).

Similarity structure becomes more difficult to recover in the more advanced layers of the DLN.

(A) The similarity structure in a middle layer of a DLN, Inception-v3 GoogLeNet. The mammals (lions and tigers) and birds (robins and partridges) correlate forming a high-level domain, rendering the upper-left quadrant a darker shade of red. Whereas the vehicles (sportscars and mopeds) and musical instruments (guitars and banjos) form two high-level categories. (B) In contrast, at a later layer in this network, the similarity space shows high within-category correlations and weakened correlations between categories. While some structure between categories is preserved, mopeds are no more similar to sportscars than they are to robins.

The middle layer (Figure 4A) reveals cross-category similarity at both the basic and superordinate level. For example, lions are more like robins than guitars. However, at the later layer (Figure 4B) the similarity structure has broken down such that subordinate category similarity dominates (i.e., a lion is like another lion, but not so much like a tiger). Interestingly, the decline in functional smoothness is not a consequence of sparseness at the later layer as the Gini coefficient, a measure of sparseness (Gini, 1909), is 0.947 for the earlier middle layer (Figure 4A) and 0.579 for the later advanced layer (Figure 4B), indicating that network representations are distributed in general and even more so at the later layer. Thus, the decline in functional smoothness at later layers does not appear to be a straightforward consequence of training these networks to classify stimuli, although it would be interesting to compare to unsupervised approaches that can perform at equivalent accuracy levels (no such network currently exists).

These DLN results are directly analogous to those with random untrained networks (see Figure 2). In those simulations, similar input patterns mapped to orthogonal (i.e., dissimilar) internal representations in later layers. Likewise, the trained DLN at later layers can only capture similarity structure within subordinate categories (e.g., a tiger is like another tiger) which the network was trained to classify. The effect of training the network was to create equivalence classes based on the training label (e.g., tiger) such that members of that category are mapped to similar network states. Violating functional smoothness, all other similarity structure is discarded such that a tiger is no more similar to a lion than to a banjo from the network’s perspective. Should brain regions operate in a similar fashion, fMRI would not be successful in recovering similarity structure therein. In the Discussion, we consider the implications of these findings on our understanding of the ventral stream and the prospects for fMRI.


Psychopathy: the behavioral profile

Psychopathy is a disorder characterized by pronounced emotional deficits, marked by reduction in guilt and empathy, and involves increased risk for displaying antisocial behavior. 5 The disorder is developmental. Psychopathic traits, particularly the emotional component, are relatively stable from childhood into adulthood. 6,7 One reason for the attention this classification receives is its strong predictive utility for institutional adjustment and recidivism (ie, reoffending). 8 Individuals with psychopathy are approximately three times more likely to reoffend than those with low psychopathic traits, and four times more likely to reoffend violently. 9 Admittedly, it is the past antisocial behavior, indexed by psychopathy assessments, that is particularly important in predicting future criminal activity. 8 However, it is the emotional component that characterizes psychopathy high levels of antisocial behavior can develop from other neurobiological and socio-environmental risk factors. 10 Psychopathy is not equivalent to the DSM-IV diagnosis of conduct disorder or antisocial personality disorder (ASPD) or their ICD-10 counterparts. The psychiatric diagnoses focus on antisocial behavior rather than underlying causes ie, the emotion dysfunction seen in psychopathy. 11 As a consequence, individuals meeting the criteria for antisocial personality disorder are more heterogeneous in their pathophysiology than individuals meeting criteria for psychopathy. 12


Watch the video: FMRI (August 2022).