2.1. Participants

A total of 134 L1 German L2 English speakers participated online (n = 78) or in the lab (n = 56). All subjects were recruited from a population of students of English linguistics at TU Dortmund University. In the web-based experiment, 13 participants were excluded from further analyses due to missing data and/or technical issues. Additionally, participants were excluded because they were L1 English (2), not L1 German (1), did not study English (7) or had an eye disease (1) that impeded their participation. In the lab-based experiment, participants were excluded due to missing data (1) or previous participation in the web-based study (3). Thus, 54 online participants (43 female, 10 male, 1 nonbinary) and 52 lab participants (41 female, 8 male, nonbinary) remained for further analyses. All participants filled in a background questionnaire assessing their first language(s) and further language learning history, as well as their social background (based on the Language and Social Background Questionnaire, LSBQ; Anderson et al., Reference Anderson, Mak, Keyvani Chahi and Bialystok2018). Thirteen participants from each group reported having another L1 in addition to German. Furthermore, participants’ language proficiencies were assessed through the German and English versions of the LexTALE (Lemhöfer & Broersma, Reference Lemhöfer and Broersma2012), a vocabulary test in which participants have to distinguish words from pseudowords.Footnote ² Participants were advanced learners of L2 English, indicated by a mean English LexTALE score above 80, which roughly corresponds to C1-level advanced learners according to the Common European Framework (CEF; see Lemhöfer & Broersma, Reference Lemhöfer and Broersma2012, for a correlation between LexTALE scores and CEF proficiency levels). Participant characteristics are summarized in Table 1. Independent sample t-tests showed no significant differences in background measures between the groups (all ps > .33). Participation was voluntary, and participants joined the study either as part of a seminar or received a small monetary compensation. Informed consent was secured from all participants. Ethical approval was granted by the Ethics Committee of TU Dortmund University (ethics vote no. GEKTUDO_2022_38 & GEKTUDO_2022_39), and the study followed the principles of the 1964 Declaration of Helsinki.

Table 1. Participant characteristics (n = 106)

^a English use was aggregated across five situations (with family, with friends, at university, using media such as movies and reading, on social media) on a 10-point rank scale (ranging from 1 = I do not use English in these situations at all. to 10 = I exclusively use English in these situations.)

^b Flanker effect as a measure of inhibitory control, calculated by subtracting congruent from incongruent reaction times in the Flanker task.

2.2. Materials

The experimental stimuli consisted of 160 letter strings, including 80 words and 80 nonwords.

2.3. Procedure

Participants performed a visual English Lexical Decision task in which they had to decide as quickly and accurately as possible whether a word presented on screen was an existing English word or not. Decisions were made by pressing one of two designated keys on the keyboard in the web-based experiment or one of two designated buttons on a MilliKey MH-5 button box in the lab-based experiment. For the online experiment, the keys “f” (for NO-presses) and “j” (for YES-presses) were chosen. These keys are comparably positioned across keyboards as they serve as the position keys for typewriting (and hence, usually have small bumps on them). Both groups were presented with on-screen instructions. The stimuli were presented in white ink (#FFFFFF), 40 px Arial font, on a black background (#000000) at the center of the screen. All stimuli were displayed in capitals to avoid language cues, as in German (unlike English), all nouns are always capitalized (see Lemhöfer et al., Reference Lemhöfer, Huestegge and Mulder2018). Each trial started with a 500 ms fixation cross at screen center. The target stimulus followed and remained on screen for a maximum of 3000 ms or until a key/button was pressed. No feedback regarding participants’ responses was provided. The experiment was presented in five blocks: a ten-trial practice block followed by four experimental blocks of 40 items each. The first three experimental blocks were followed by a short break that allowed the participants to rest until they pressed a key/button to continue. Stimuli were presented in a different randomized order per participant. The experiment took approximately 5 minutes and was programmed in OpenSesame (version 3.3.11; Mathôt et al., Reference Mathôt, Schreij and Theeuwes2012).

In addition to the Lexical Decision task and the self-paced reading experiment (see Experiment 2), the participants completed several background tasks, the results of which are not covered in this paper. The order of tasks was as follows: (1) Lexical Decision task, (2) Flanker task, (3) Self-paced Reading task, (4) German LexTALE, (5) English LexTALE, (6) Reading Span task, (7) Questionnaire. In total, the testing session took approximately 70 minutes.

2.4. Results

Accuracy rates and reaction times (RTs) for words/nonwords and cognates/noncognates were analyzed in R (version 4.4.0; R Core Team, Reference Team2024). One noncognate item (EN: pupil) was removed because it is a homonym with both a noncognate translation (Schüler:in) and a cognate one (Pupille) in German. For the accuracy analysis, RTs below 200 ms were coded as false alarms (web: n = 1; lab: n = 0). For the RT analysis, incorrect button presses and extreme RTs above 2000 ms (web: n = 115 [1.39%]; lab: n = 104 [1.31%]) were excluded. Table 2 lists the accuracy rates and RTs for the respective conditions.

Table 2. Mean accuracies (proportions) and reaction times (in milliseconds) by group and condition

Note: Standard deviations are in parentheses.

Accuracy and RT data were analyzed using mixed-effects regressions with the aid of lme4 (Bates et al., Reference Bates, Mächler, Bolker and Walker2015) and lmertest (Kuznetsova et al., Reference Kuznetsova, Brockhoff and Christensen2017). Separate models were run for the analyses of word status and cognate status. For the analyses of words and nonwords, word status, setting and their interaction were entered as fixed effects. For the analyses of cognates and noncognates, nonwords were removed from the data set, and cognate status, setting, as well as their interaction, were entered as fixed effects. All two-level fixed effects were sum-coded (setting: web = 1, lab = −1; word status: nonword = 1, word = −1; cognate status: cognate = 1, noncognate = −1). To identify the maximal converging random effect structure, the “order” function in the buildmer package was used for all mixed-effect models (Voeten, Reference Voeten2021).Footnote ³ Effect sizes were calculated with the effectsize package (Ben-Shachar et al., Reference Ben-Shachar, Lüdecke and Makowski2020). Overall, accuracies were close to the ceiling, which speaks to the participants’ high L2 proficiency. For words and nonwords, the model returned a significant main effect of word status for accuracy (ß = −0.61, SE = 0.12, z = −4.94, p < .001, d = −0.96) and RT data (ß = 111.01, SE = 9.04, t = 12.28, p < .001, d = 2.4), with words being processed more accurately and faster than nonwords. Furthermore, accuracy analyses of cognates and noncognates yielded a moderate main effect of cognate status (ß = 0.32, SE = 0.12, z = 2.60, p = .009, d = 0.51), with cognates being processed more accurately than noncognates. RT analyses yielded neither main effects nor an interaction. Note that none of the models yielded a main effect of or an interaction with setting. Detailed model output is displayed in Table 3.

Table 3. LDT model outputs for word status (word versus nonword) and cognate status (cognate versus noncognate) after “buildmer” model optimization

Note: Significant effects in bold.

2.5. Discussion

Experiment 1 investigated the replicability of the word/nonword effect as well as the effect of cognate status during isolated L2 lexical processing. Besides the replicability of these well-established effects, the comparability of L2 word recognition across the two settings was examined (RQ1). For word versus nonword processing, differences in accuracies and RTs were observed between the two conditions in both settings, replicating previous lab-based results. Although participants’ decision accuracies were above 90% in both conditions, analyses still yielded a significant difference between conditions, with comparatively lower accuracies for nonwords compared to words. Similar to previous findings, RT analyses revealed a processing disadvantage for nonwords over words, with pseudowords exhibiting a delay in processing. Additionally, overall mean RTs differed by only 6 ms between settings. This confirms previous observations by Hilbig (Reference Hilbig2016) and Kim et al. (Reference Kim, Lowder and Choi2023), who likewise demonstrated no significant difference in RT effects in web-based versus lab-based lexical decision.

For cognate versus noncognate processing, analyses of participants’ decision accuracies revealed a main effect of cognate status. With overall accuracies of 98% and 99% per condition, participants’ accuracy was high across the board, thus corroborating their overall high L2 English proficiency as evidenced by their English LexTALE results. The descriptively rather small difference of 1% yielded statistical significance, providing evidence for a cognate facilitation effect. However, contrary to initial predictions, the cognate facilitation effect could not be replicated in participants’ response times. Nonetheless, this pertains to both settings and thus suggests that the null results are not setting-induced (see also Semmelmann & Weigelt, Reference Semmelmann and Weigelt2017, who reported null effects with priming across three settings). The absence of a cognate facilitation effect in RTs may be a ceiling effect caused by participants’ overall high L2 proficiency (Bultena et al., Reference Bultena, Dijkstra and Van Hell2014) or the stimuli’s lexical frequency. The cognate and noncognate nouns used in the LDT were predominantly high-frequency nouns (mean SUBTLEX-US log10 = 3.25), which may have disguised potential cognate effects (Peeters et al., Reference Peeters, Dijkstra and Grainger2013). Consequently, future studies could further explore potentially modulating factors in cognate processing and limitations to the cognate facilitation effect. Although the cognate effect could not be replicated in the present study’s RT analysis, the overall findings of Experiment 1 suggest comparability of the results obtained in both the online and laboratory settings. The lexical decision manipulation itself was successful, as evidenced by the word/nonword RT effect observed in both settings.

In summary, these findings show that with overall ceiling performance in accuracies across settings, online testing did not prove to adversely affect participants’ concentration on ultimate decision-making. Additionally, RT results did not differ across groups, neither for relative RTs between conditions nor for absolute overall mean RTs. Thus, the results expand the findings of previous research by providing comparable evidence of L2 lexical processing across both web-based and lab-based experimental settings.

3.1. Participants

The same participants who took part in Experiment 1 also participated in Experiment 2.

3.2. Materials

Experimental sentences were constructed based on a 2 (Sentence type: SRC versus ORC) × 2 (Plausibility: plausible versus implausible) × 2 (Cognate status: cognate versus noncognate) design. This resulted in 40 sentence quadruplets in the cognate condition and 40 sentence quadruplets in the noncognate condition (see 8a–h for examples; slashes indicate the phrases in which the sentences were presented). Additionally, 120 plausible and implausible filler sentences were created (see Appendix S1 in Supplementary Materials for details).

3.3. Procedure

Subsequent to the Lexical Decision task and the Flanker task, participants performed a noncumulative self-paced reading task in which they read English sentences phrase-by-phrase and rated the plausibility of each sentence immediately after having read it. Implausibility was defined as the event described in the sentence being very unlikely or even impossible to occur. Instructions were presented on screen. Phrases were presented in white ink (#FFFFFF), 40 px Arial font, on a black background (#000000) at the center of the screen, using the stationary window method. Trials were initiated with a 500 ms fixation cross at screen center. Subsequently, the sentences were displayed in phrases of different lengths (varying from one to four words) with a maximum of six phrases. Experimental sentences always consisted of four phrases. Each phrase remained on screen until a designated key was pressed (the space bar online; a central button on the button box in the lab). The first phrase always began with a capital letter. The last phrase always ended with a full stop to indicate the end of the sentence. It was followed by a visual display of a question mark that prompted participants to judge the plausibility of the previously read sentence by pressing one of two designated keys. Online, participants pressed the “f” (for implausible) and “j” (for plausible) keys on their keyboard (see Experiment 1). In the laboratory, a MilliKey MH-5 button box was used. No feedback regarding responses was provided. The experiment was presented in six blocks: A six-trial practice block preceded five experimental blocks of 40 sentences each. The total number of 200 sentences was randomly distributed across these five blocks. The first four experimental blocks were followed by short breaks that allowed participants to rest until they pressed a key to continue the experiment. The experiment lasted approximately 15–20 minutes and was programmed in OpenSesame (version 3.3.11; Mathôt et al., Reference Mathôt, Schreij and Theeuwes2012).

3.4. Results

Plausibility judgment accuracies and reading times were analyzed in R (version 4.4.0; R Core Team, Reference Team2024). Two noncognate items that contained the noun pupil were excluded from further analyses (see Experiment 1). Additionally, one item with a low plausibility judgment accuracy of 54% was removed. The remaining items had an average accuracy of 93% (range: 77–99%). For the analyses of reading times, the focus was on two regions of interest: (i) the critical phrase containing the RC and (ii) the post-critical phrase immediately following the RC. For reading time analyses, all items with incorrect plausibility judgment were excluded. Additionally, phrases with reading times below 200 ms and above 5000 ms were removed from further analyses (see Klassen et al., Reference Klassen, Kolb, Hopp and Westergaard2022, for a similar procedure). For the critical phrase, this resulted in the removal of 44 trials (web: n = 34 [0.89%]; lab: n = 10 [0.27%]). For the post-critical phrase, 77 trials were removed (web: n = 44 [1.15%]; lab: n = 33 [0.88%]). Table 4 lists the mean plausibility judgment accuracies and reading times for the critical and post-critical phrases per condition for both groups.

Table 4. Mean plausibility judgment accuracies (proportions) and reading times (in milliseconds) for critical and post-critical phrases per condition by group

Note: Standard deviations are in parentheses.

Accuracy and reading time data were analyzed using mixed-effects regressions (lme4, Bates et al., Reference Bates, Mächler, Bolker and Walker2015; lmertest, Kuznetsova et al., Reference Kuznetsova, Brockhoff and Christensen2017). Setting, sentence type, plausibility and cognate status, as well as their interactions, were entered as fixed effects into the models. Sum-coding was applied to the two-level fixed effects (setting: web = 1, lab = −1; sentence type: SRC = 1, ORC = −1; plausibility: plausible = 1, implausible = −1; cognate status: cognate = 1, noncognate = −1). By means of the “order” function in the buildmer R-package (Voeten, Reference Voeten2021), the maximal converging random effect structure was identified. Effect sizes were calculated with the effectsize package (Ben-Shachar et al., Reference Ben-Shachar, Lüdecke and Makowski2020). Plausibility judgment accuracy was overall high across the board, suggesting that in both groups, participants were able to successfully parse L2 noncanonical sentence structures. Still, generalized linear mixed effects analyses revealed a main effect of plausibility, with implausible sentences being processed more accurately than plausible ones (ß = −0.45, SE = 0.15, z = −3.09, p = .002, d = −0.6). For reading times in the critical phrase, analyses yielded significant main effects for plausibility (ß = −74.11, SE = 8.23, t = −9.01, p < .001, d = −1.76) and sentence type (ß = −48.27, SE = 7.50, t = −6.44, p < .001, d = −1.26), indicating that in the critical RC region, plausible sentences were overall processed faster than implausible ones and SRCs faster than ORCs. Analyses of reading times in the post-critical phrase revealed main effects of plausibility (ß = 47.11, SE = 16.21, t = 2.91, p = .004, d = 0.57) and sentence type (ß = −42.50, SE = 7.34, t = −5.79, p < .001, d = −1.13) and an interaction of both fixed effects (ß = −30.00, SE = 7.04, t = −4.26, p < .001, d = −0.83). In the post-critical region, implausible sentences were processed faster than plausible sentences – opposite to processing patterns in the critical phrase. Similar to the critical phrase, SRCs were processed faster than ORCs in the region following the RC. While there was a clear advantage for SRCs over ORCs for plausible sentences, the difference between these two conditions was much smaller for implausible sentences. Neither the main effects of cognate status or setting nor interactions with these two factors were found in any of the three dependent measures. Participants’ mean reading times for the critical and post-critical phrases are plotted in Figures 1 and 2. Note that these times were collapsed across cognate status to allow for better visualization, as this factor proved not to have a significant influence. Detailed model output is displayed in Table 5.

Figure 1. Mean reading times (in milliseconds) for the critical phrase by word order and by plausibility for both settings.

Note: Error bars show the 95% confidence interval.

Figure 2. Mean reading times (in milliseconds) for the post-critical phrase by word order and by plausibility for both settings.

Note: Error bars show the 95% confidence interval.

Table 5. SPR model outputs after “buildmer” model optimization

Note: Significant effects in bold.

Finally, the overall mean reading speed was compared between settings. Mean reading speed was calculated across the sum of reading times of all phrases for both experimental and filler sentences. Incorrect trials were excluded from further analyses, and extreme values above 10000 ms were considered outliers and thus removed. The fixed factor setting was sum-coded. Whereas the two groups differed descriptively by about 200 ms, with the online group (M = 3005 ms) being faster than the lab group (M = 3212 ms), linear mixed effects analysisFootnote ⁴ on overall mean reading speed revealed no significant impact of setting on overall reading times (ß = −103.08, SE = 68.51, t = −1.51, p = .132, d = −0.29).

3.5. Discussion

Experiment 2 examined the replicability of cognate effects in L2 sentence context (RQ2) and that of word order and plausibility effects during L2 sentence processing (RQ3). Moreover, the comparability of online and laboratory testing was further investigated based on these linguistic phenomena. Whereas lexical co-activation through cognates as observed in prior studies could not be replicated, the present study observed effects of both word order and plausibility on sentence processing. Importantly, the study found similar results for both settings in the accuracy of sentence final judgments as well as two reading time measures, suggesting the comparability of both testing contexts. In the following, we will first discuss the absence of cross-linguistic lexical effects induced by cognates during L2 sentence processing and then review the effects of word order and plausibility on L2 sentence comprehension. Both research questions (RQ2 and RQ3) will be considered against the methodological backdrop of the testing environment.

In view of RQ2 concerning lexical processing in L2 sentence context, the results resemble the null findings for RTs obtained in Experiment 1. Similarly, no effect of cognate status was found in plausibility judgment accuracies or reading times of either the critical or the post-critical phrase in either of the two settings. Thus, in contrast to previous studies (Hopp, Reference Hopp2017; Miller, Reference Miller2014), a cognate facilitation effect during L2 sentence processing could not be replicated. Instead, descriptively, critical phrases containing cognate words were processed more slowly than those containing noncognate words across all conditions. While this cognate disadvantage was not statistically significant, it is a pattern that can be observed across both groups. The fact that results from both settings showed similar descriptive and inferential results provides further evidence of the comparability of the web-based and lab-based settings. Regarding linguistic implications, it can neither be confirmed nor ruled out whether syntactic or semantic context or cross-linguistic syntactic L1 interference induced the absence of cognate facilitation (Dirix et al., Reference Dirix, Brysbaert and Duyck2019; Lauro & Schwartz, Reference Lauro and Schwartz2017). In fact, the syntactic environment is rather unlikely to have caused the null findings since no cognate facilitation was found for these words out of context either (see Experiment 1). Further research is needed to better understand the interplay of lexical and syntactic L1 activation and its influence on L2 sentence processing.

With respect to RQ3, the present study identified the impacts of word order and plausibility on L2 English relative clause processing. In general, accuracies for plausibility judgment were high across the board in all conditions for both groups, reflecting participants’ advanced L2 sentence comprehension. Nevertheless, the plausibility judgment accuracies revealed that implausible sentences were processed more accurately than plausible ones, contradicting the assumption of a processing advantage for semantically plausible sentences. The disadvantage of plausible sentences was particularly evident in the processing of plausible ORCs by the online group, with comparatively lower accuracies in this condition (M = 88%). A possible explanation could lie in thematic roles and positing a somewhat enhanced agent-first preference in these learners. Such a preference is characterized by the first-mentioned noun phrase (NP) of a sentence being more likely to be interpreted as the agent (and not patient) of a transitive event (Bever, Reference Bever and Hayes1970; VanPatten, Reference VanPatten, VanPatten and Williams2015). Applying this to plausible ORCs (first NP is patient and not agent), the initial interpretation tends to be implausible. However, with target-like syntactic parsing, the initial implausible interpretation needs to be overcome, forcing learners to reanalyze the sentence to allow successful interpretation. In this case, it seemed more difficult to judge a sentence that had already been assessed implausible as still plausible than vice versa (i.e., judging a sentence that had been assessed plausible as implausible, e.g., when erroneously applying the agent-first strategy to implausible ORCs that start with the patient). Applying such a processing heuristic thus leads to more revision difficulties when the sentence has already been discarded. However, this descriptive disadvantage for plausible ORC accuracies is only present in the online group. This might indicate a general speed–accuracy trade-off in web-based participants’ reading times: The online group may have failed in reanalyzing the initially seemingly implausible sentence as plausible slightly more often than the lab-based group because they were reading descriptively faster, which applies to both overall reading times and to RTs in the critical and post-critical phrase.

With regard to reading times of the critical phrase, analyses revealed main effects of plausibility and sentence type. The present study thus replicated previous in-person research by demonstrating processing advantages for plausible sentences over implausible ones and advantages for SRCs over ORCs in both settings (for similar findings, see Lim & Christianson, Reference Lim and Christianson2013). No significant difference in RTs between settings was found.

With regard to reading times of the post-critical phrase, analyses yielded main effects of plausibility and sentence type as well as an interaction of the two factors. Overall, implausible sentences were processed faster than plausible ones. This processing advantage for implausible sentences contrasts with the pattern observed in the critical phrase, which is most likely due to the nature of the plausibility judgment task itself. As soon as an implausible sentence has been deemed implausible, for which participants already have sufficient information in the critical phrase, there is no reason to pay further attention to subsequent regions. Thus, in this case, readers are more likely to skim-read or even skip the post-critical phrase. In contrast, a plausible sentence may turn into an implausible one at any moment depending on incoming information. This may provide a general advantage for implausible sentences on final sentence segments (for similar task effects, see Williams, Reference Williams2006). Additionally, similarly to the critical phrase, SRCs were processed faster than ORCs. This is driven mostly by the plausible sentences, expressed by the two-way interaction, and thus (again) reflecting plausible ORCs as the most difficult condition. As suggested above, the particular difficulty of plausible ORCs is likely due to these sentences initially seeming implausible when following a heuristic agent-first strategy, and this initial judgment needs to be revised to correctly judge these sentences as plausible. As for the group, mixed effects analyses once more revealed no differences across settings and no interaction with any of the experimental manipulations.

With respect to the reading times of both the critical and post-critical phrases and the mean reading speed across all phrases, the web-based group demonstrated descriptively faster RTs than the lab-based group. Note, however, that this difference in absolute RTs only appeared descriptively, and neither a main effect of setting nor an interaction with it was found in any of the abovementioned analyses. Since significant effects of word order and plausibility were found for all three dependent measures in both settings, it can be concluded that the self-paced reading data collected online is on par with the data collected in the lab.

In summary, the self-paced reading results complement the findings from lexical decision (Experiment 1) and extend previous research demonstrating comparable findings of behavioral data collections conducted via the web and in the lab (Hilbig, Reference Hilbig2016; Kim et al., Reference Kim, Lowder and Choi2023; Patterson & Nicklin, Reference Patterson and Nicklin2023). The results of Experiment 2 revealed that the linguistic effects of sentence plausibility and sentence structure obtained in the lab can be replicated via online data collection. Methodological consequences resulting from the decision to conduct web-based research and important aspects that need to be considered for the preparation and implementation of online testing will be discussed in more detail below.