Information

Free software to run questionnaire based experiments

Free software to run questionnaire based experiments



We are searching data for your request:

Forums and discussions:
Manuals and reference books:
Data from registers:
Wait the end of the search in all databases.
Upon completion, a link will appear to access the found materials.

I am looking for free software to build experiments that do not heavily rely on accurate timing (where I use PsychoPy and heard a lot of good things about OpenSesame) but more on an easy way to implement different types of items and response options.

That is, for example, an experiment which starts in presenting some texts to the participants, then some questions contingent on this text and on what participants entered before (potentially more than one question per page with different response options) followed by some standard questionnaires. How would you implement such a study?


Background and motivation

I used to do such things using MediaLab which unfortunately is proprietary software. However, implementing items and questionnaires was easy. You could program a simple html page (even using variables and placeholders) using html forms and MediaLab would present this page using ie engine in fullscreen and collect the responses. Those responses could then be used or handed over to DirectRT for response time experiments.

Currently I am using PsychoPy to control the screen and data collection and code all questionnaires and such stuff per hand using wxPython. It works, but designing questionnairs in wxPython is a lot less handy and more difficult than programming html pages with html forms and definitely nothing for people not too profficient in coding.

Any ideas or hints would be really appreciated.

Note that I read "What's the best program for creating computer based psychology experiments under OS X?", but my question concerns specific implementations. Furthermore, I work on Windows 7.


It sounds like you're looking for a platform on which to implement computerized adaptive tests (since subsequent questions are contingent on prior responses).

I found Concerto, which is based in R and MySQL, but allows some flexibility in presentation (it says it uses HTML directly, but you could probably couple it with another language).


You'll want to look into PsychToolbox 3, a very capable Matlab/Octave toolbox intended for running high-precision behavioral psychology experiments. If you don't have access to Matlab, you can use Octave (which is available for Windows 7; see link) to run your experiments.

The toolbox itself is a platform for creating interactive experiments, ranging from simple question-based surveys to highly complex, precision visual displays. For the purpose at hand, I would build a simple function that displays text using PsychToolbox's DrawText or DrawFormattedText functions and monitors responses using theKb*functions; see the KbDemopage for an overview. Each of these has their own demo showing how to use the function (the aptly-named DrawSomeTextDemo and DrawFormattedTextDemo functions, and theKbDemoone mentioned earlier); simply check the source code of the demos to see how they work.

I will mention that PsychToolbox has learning curve, and requires familiarity with Matlab/Octave programming. That being said, with the above functions, it should be pretty easy to create a generic "survey" function which accepts however many lines of text (one question, however many answers), displays them to the user, and monitors and records feedback.


Experiment Builder

SR Research Experiment Builder is a sophisticated and intuitive drag-and-drop graphical programming environment for creating computer-based psychology and neuroscience experiments.

  • No hidden fees. Updates are always free and always fully backwards compatible.
  • Cross-platform compatible for Windows (32-bit and 64-bit) and macOS (OSX).
  • Delivers complex visual and auditory stimuli and registers responses with extremely high levels of timing precision.
  • Multi-language text presentation , with full control of text properties, html markup , and automatic interest area segmentation.
  • Displays video content with precise timing and frame-logging.
  • Powerful randomization options , including blocking, run-length control, and list counter-balancing.
  • Hundreds of existing templates that can be easily modified (e.g., change blindness, smooth pursuit, pro-saccade task, Stroop task and many more) are available through the SR Research Support Forum.
  • Fully integrated with EyeLink Data Viewer.
  • Compatible with all EyeLink trackers, facilitating cross laboratory research collaboration.
  • Supports both eye tracking (EyeLink) and non-eye tracking experiments.

If you would like to find out more about Experiment Builder and how it can help your eye-tracking research, please get in touch with us! You can also download the latest version of Experiment Builder by clicking the button below (you will need to be registered with our support forum for the link to work):


Introduction

The advent of the Internet opened new avenues of exploration for us as psychological researchers. Internet-based experimental instruments allow us to conduct experiments with demographically and culturally diverse samples, to recruit large subject pools in less time, to avoid organisational issues such as scheduling conflicts, to save costs related to laboratory space, equipment, personnel hours, and administration, and to increase our ability to conduct international experiments [1,2,3]. For these benefits to be worthwhile we must be able to trust Internet-based instruments to accurately record participants’ responses, in terms of both the actual responses as well as their intrinsic characteristics, such as response times. The current study investigates this particular issue by testing the replicability of the Internet-based implementation of PsyToolkit for use with paradigms requiring complex Choice Response Time (CRT) tasks.

It has been argued that, for instruments found to reliably record participants’ responses, Internet-based experimentation has three main advantages over laboratory-based experimentation [2] increased generalisability, increased voluntariness, and increased ecological validity. Increased generalisability refers to participants being able to be recruited from much broader demographic and/or geographic backgrounds, meaning that the sample is more likely to be truly representative of society. Increased voluntariness refers to participants having fewer constraints on their decisions to participate and to continue to participate as, for example, there is no researcher whose presence might socially pressure a participant to continue. Further, responses may be more authentic when participants are more comfortable in their ability to stop the experiment [2]. Ecological validity is a measure of the level to which participant behaviour in an experiment resembles their behaviour in a naturalistic setting. The closer to reality an experiment can be, the higher the level of ecological validity the experiment is said to have, and the more we can be confident that the results obtained reflect the participant’s real-world behaviours. As an example, driving simulators attempt to simulate, to different degrees, the feeling of driving a real car. The closer the simulator is to the experience of naturalistically driving a car, the higher the level of ecological validity. As such, an experiment in which you sit inside an actual car, observe a scene projected on the wall in front and to the sides of you, and respond using the car’s steering wheel, accelerator, and break is likely to have a higher level of ecological validity than an experiment in which you sit in front of a computer screen, observe a scene shown on the screen, and respond using controllers shaped like a steering wheel, accelerator, and break, which in turn is likely to have a higher level of ecological validity than an experiment in which you sit in front of a computer screen, observe a scene shown on the screen, and respond by moving the mouse on the screen to control direction and speed. With reference to internet-based studies, it has been argued that the ability for participants to take part in experiments in environments (and using equipment) that they are familiar with, and the ability for participants to undertake experiments without the presence of a researcher in the room, lead to increased ecological validity [2].

The ability to undertake experiments in familiar environments, and with familiar equipment, has the potential to enhance ecological validity in at least two manners increased familiarity and reduced cognitive load. Increased familiarity refers to the fact that participants can choose the time, place, and surroundings in which to undertake the experiment, ensuring that any effects found cannot be attributed to being in an unfamiliar setting [2]. Cognitive load refers to the amount of cognitive resources required, out of a limited pool, to fulfil the requirements of mentally demanding tasks [4]. In experimental terms, increasing levels of cognitive load are associated with increased reaction times, as participants have less cognitive resources available for undertaking experimental tasks. Unfamiliar environmental factors are known to increase cognitive load, as the level to which the brain actively monitors the environment is higher, which in turn reduces the cognitive resources available for other tasks. As such, the more familiar an individual is with their surroundings, the less cognitive resources are utilised in monitoring the environment, meaning that there are more cognitive resources available for focusing on the experimental task with which they are presented.

The lack of a researcher present has the potential to enhance ecological validity through reduced social desirability bias and reduced cognitive load. Social desirability bias refers to a cognitive bias in which individuals act to increase the level to which answers they give are in line with social norms in order to present themselves in the best possible light [4, 5]. The level to which this bias occurs is, among other factors, heightened in the presence of others [6]. As such, responses given in the absence of researchers are more likely indicative of how an individual truly feels about the subject, leading to higher ecological validity. Cognitive load is also reduced in the absence of a researcher, as the presence of others when undertaking a task divides attention, at least to some degree, between the experimental task and anyone else present [7, 8].

While increasing ecological validity is an important factor for experimental design, laboratory-based experiments also have advantages over internet-based experiments. Firstly, laboratory-based experiments have a higher range of possible research approaches. This is primarily due to equipment requirements. It is not reasonable, for example, to expect participants recruited from the general populace to all own eye tracking equipment as such, it is more logical to undertake experiments in which eye tracking is included in laboratory conditions. Further, hardware and software related issues have historically introduced a high level of error noise into results obtained through internet-based instruments compared to those obtained through laboratory-based instrumnets, primarily observable as response time noise. A wide variety of factors can affect response time recording, such as hardware timing features, device driver issues and interactions, script errors, operating system variability, interactions with other software, tools to construct the paradigm, interactions with other hardware, and configuration of settings and levels [3,9]. In laboratory-based experiments these sources of noise are less likely to affect the final results of the experiment, as all participants undertake the experiment with the same hardware, software, device drivers, operating system, and system configuration. In internet-based experiments, however, there are large potential differences in these elements between participants’ computers, which can lead to a higher level of noise within the results obtained. Further, responses given via the internet are also affected by the amount of time it takes for the website hosting the experiment to successfully send an image to the participants’ computer, and then, after responding, by the amount of time it takes for the response to be sent from the participants’ computer to the website hosting the experiment [10]. As high noise levels can obscure small effects and give the illusion of heterogeneous responses, care must be taken when analysing results obtained through internet-based instruments to ensure that an increase in heterogeneous responses are due to ecological validity improving rather than noise level increasing. However, technology continues to evolve, and recent advances in the design of internet-based experimental tools–such as has occurred with PsyToolkit, the instrument we present next–may have significantly reduced error noise compared to older internet-based instruments, even to the point of bringing them fully in line with laboratory-based instruments. As such, for instruments with which there is minimal Internet-related noise, if ecological validity was indeed increased we could (for example) expect participants to respond to items in a less self-monitored and/or socially accepted manner, with participants displaying wider response choice variability and overall faster response times.

PsyToolkit is an open-access psychological instrument developed to allow researchers, including student researchers, to easily program and run experimental psychological experiments and surveys for both laboratory and Internet settings [11,12]. Two versions of PsyToolkit are available a laboratory-based version that runs on Linux, and an Internet-based version that is Javascript based and can run on modern browsers without participants needing to download any programs. The Internet-based version of the instrument is specifically aimed at addressing financial and technical limitations commonly faced by students, as it is free software that has specifically been designed for running online questionnaires, Simple Response Time (SRT) tasks, and Choice Response Time tasks (CRT) [12]. A SRT is an experimental task in which a single stimulus, and only that stimulus, is presented repeatedly at the same on-screen location, with participants tasked with responding to every presentation of the stimulus in the exact same manner and quickly as possible [13]. An example of this is participants being instructed to watch an LED and to press a specific button as quickly as possible whenever the LED lights up. A CRT is an experimental task in which instead multiple stimuli are shown, and/or stimuli are presented on different areas of the screen, and the participant is tasked with responding in different manners depending on the nature of each presentation (e.g., Zajdel and Nowak [13]). An example of this is participants being instructed to look at a screen on which letters will appear, with the task of pressing the corresponding letter on a keyboard. CRTs can also differ in complexity. Simple CRTs, such as in the above example, require participants to recognise the stimuli and respond accordingly. More complex CRTs require participants to also make judgements about the nature of the stimuli.

In the present experiment, participants were instructed to look at a screen on which first names paired with role nouns appeared, with the task of pressing one of two buttons depending on whether they believed that it made logical sense for someone with the name shown to hold the role shown. Stoet [12] states that PsyToolkit is designed for a teaching environment, with minimal technical barriers and free web-based hosting of their studies. A library of existing psychological scales and experiments is available for students to examine and adapt, and extensive online documentation and tutorials are available to assist if students face any issues. Further, Stoet [12] states that PsyToolkit is designed to allow for students to randomise item order in both questionnaires and in cognitive experiments, to allow for a convenient way of scoring, and to give feedback to participants about their test scores options not available in all Internet-based instruments. All users of the Internet-based version must register an account to be able to create experiments, but accounts are free. Randomisation is possible in both the survey and the experiment, and partial randomisation is also possible for if one wishes for only certain portions of the survey and/or experiment to be randomised. Further, alternate versions of the experiment can be created, with participants randomly assigned between versions. In terms of reliability, Stoet [14] states that both the Internet and Linux versions of PsyToolkit can reliably measure small effects of less than 50ms, with the Linux version being more precise. However, to our knowledge, currently no research has been published examining the replicability of the Internet-based version of PsyToolkit.

As PsyToolkit is intended to be a student-focused instrument, and many universities do not set up experimental computers with Linux for their students, it was decided to compare results obtained through the Internet-based implementation of PsyToolkit to results obtained through E-Prime 3.0 in a laboratory setting. E-Prime was chosen as it is a commonly used psychological research tool in university settings, including in teaching environments, and, like PsyToolkit, it has a low barrier to entry and has an experiment library. Further, Stoet [14] states that the Linux-based version of PsyToolkit is on par with E-Prime, so, while there is likely to be noise due to differences in software, this is expected to be minimal.

While the replicability of PsyToolkit has not been examined, the replicability of other Internet-based instruments has been tested through CRT tasks (e.g., Reimers and Stewart [15] Schubert, Murteira, Collins, & Lopes [16]). Reimers and Stewart [15] used a CRT task to test the replicability of an experiment in the Internet-based version of Adobe Flash compared to the same experiment in a laboratory-based version of Adobe Flash, with the same experiment coded in C used as a baseline. Participants were shown green and red rectangles and were required to press buttons corresponding to the colour of the rectangle on the screen. They found that, compared to the baseline, (a) response times of the laboratory-based version of Flash were 10ms longer, (b) response times of the Internet-based version of Flash were 30-40ms longer, and (c) there were no significant differences in Response Time standard errors across conditions. Schubert et al. [16] used both SRT and CRT experiments in a study testing the replicability of ScriptingRT to Flash. Six experiments were conducted over the course of their study. The first three studies used SRT tasks but were automated to test specific aspects of ScriptingRT. The last three studies used CRT tasks, specifically a version of the Stroop task, where participants were presented with either the words “red” or “blue”, or a neutral letter string, in either red or blue on a white background. Participants were instructed to press keys corresponding to the colour of the word or neutral letter string shown. Experiment 4 tested the Internet-based version of ScriptingRT by itself, while Experiment 5 compared ScriptingRT to the same experiment coded in DMDX (a laboratory-based instrument [17]) with participants undertaking both tasks on the same computer, and Experiment 6 compared ScriptingRT to Inquisit Web Edition, both running via the Internet. In Experiment 5, the experiment of most interest to the present experiment as it compares an Internet-based implementation of the instrument to a laboratory-based one, Schubert et al. [16] found that the size of the Stroop effect was not affected by which software was used.

Historically, psycholinguistic research has not relied upon Internet based testing, as it often relies upon small differences in response times in CRT tasks to detect effects [18] and is strongly affected by response time noise. Recent research (e.g., Enochson & Culbertson [18]) has found that some modern Internet-based instruments are reliably able to test these small differences, meaning that modern psycholinguistic research may safely utilise Internet based tools that have been properly validated. Some researchers have suggested that PsyToolkit may be a delicate enough tool for psycholinguistic experimentation (e.g., Sampaio [19]). An opportunity arises therefore to test both general replicability and psycholinguistic specific replicability of PsyToolkit through a psycholinguistic experimental paradigm.

The present study was designed to compare responses and Response Times measured by the Internet-based implementation of PsyToolkit with those measured by the laboratory-based implementation of E-Prime 3.0 using a complex CRT task composed of an existing and published psycholinguistic paradigm (i.e., Gygax & Gabriel [20]) to test replicability between the Internet-based implementation of PsyToolkit (Version 2.4.3) and E-Prime (Version 3.0.3.31). The paradigm uses a between-subjects two-alternative forced choice design, with a CRT task in which participants are shown pairs of terms (in the present experiment a first name and a role noun e.g., ‘Kate–Chefs’) and are then required to, as quickly as possible, make a judgement as to whether the pairing makes logical sense (i.e., could someone named Kate be a member of a group of chefs). Experimental item pairings were composed of first names paired with professional roles that vary in gender stereotypicality. As logically any individual can hold any professional role, filler item pairings were included to prevent participants of developing a strategy of always answering positively to all roles seen. The filler items were first names paired with gender-marked kinship terms, with both congruent (e.g., ‘Kate–Mothers’) and incongruent (e.g., ‘Kate–Fathers’) pairings shown to prevent participants from developing a strategy of answering positively to professional roles and negatively to familial roles.

The paradigm we utilise is more complex than those used by Reimers and Stewart [15] and by Schubert et al. [16], as the paradigm used in the current study requires participants to make subjective judgements of the items presented before responding, while the paradigms used by Reimers and Stewart [15] and Schubert et al. [16] required that participants responded based on the colour, an objective quality, of the items presented to them. One can therefore expect that overall response times will be longer for this study than those found by Reimers and Stewart [15] and Schubert et al. [16], and, compared to Reimers and Stewart [15], it is likely that Response Time standard errors will be larger. Further, if the results indicate that there is a high level of replicability between PsyToolkit and E-Prime, then it may be possible to determine whether the results offer any support for the concept of increased ecological validity in Internet-based experiments. If the results obtained in PsyToolkit do have a higher level of environmental validity than the results obtained in E-Prime, we would expect that participants who undertake the PsyToolkit version of the experiment would be more likely to respond negatively, and would overall respond more quickly (i.e., more spontaneously), than those who undertake the E-Prime version of the experiment.

It is worth noting that Norwegian is considered a semi-gendered language. This is because some, but not all, nouns have associated gender markers. Specifically, only nouns that refer to living beings, especially humans, are gendered in Norwegian. Further, the majority of role nouns in the plural form are the same as the masculine-specific singular form. This is due in part to a linguistic policy of gender neutralisation [21], under which the masculine grammatically marked form of role nouns are actively encouraged to become the main linguistic device to refer to the majority of roles [22].


Statistics itself has developed a lot in recent years. Prominent statisticians around the world have introduced various new tests and analysis types, thereby adding new aspects and dimensions to the field of statistical analysis. Statistics involves multiple tests, correlations, variable-analysis, and hypothesis testing, that makes it a complicated process.

A statistical analysis software has the following features to make complicated statistical functions easy.

Below is the list of 10 free and open source statistical software

  1. JASP
  2. SOFA
  3. GNU PSPP
  4. SCI LABS
  5. Jamovi
  6. MacAnova
  7. Past
  8. Develve
  9. Invivostat
  10. IBM SPSS

4 False Consensus Effect

Develop a short questionnaire with simple questions, such as, "I think the government should spend less money on security and more on helping the homeless." The questionnaire should be short and one that can be answered quickly. After people answer the questionnaires by themselves, ask them to guess the number of others in their group who have the same opinion on each question. Then ask participants to raise their hands to indicate their responses on the questions. People have a tendency to overestimate how many others have the same opinion they do. The False Consensus Effect demonstrates the idea that people think those who are similar to themselves also have similar attitudes.


Introduction

Humans constantly interact with their environment. Picking up the phone and pressing a computer key are actions that people perform almost unconsciously in their daily lives. These actions have an observable effect the phone stops ringing or a character appears on the screen. Thus, people are active agents in the control of events in their environments.

The study of how people detect that they are agents of certain effects, and of how they perceive the relationship between their actions and their potential effects has been an interesting and productive research topic since the early times of experimental psychology. Among the questions that have been explored are the perception of the causal relationship between an action and its effect, consciousness and willingness to initiate voluntary acts, as well as the perceived temporal distance between actions and their consequences.

Wilhelm Wundt had already reported the seminal experimental work in this area as early as 1887 (see 1 ). He designed a complex apparatus in which a clock’s hand rotated through a sphere, with the occasional presentation of an auditory stimulus. Wundt asked his experimental participants to indicate the position of the clock hand when the tone was presented. In this way he could investigate whether the participant’s subjective perception of the tone coincided with its objective timing.

This methodology was then adapted in the twentieth century by Libet and his colleagues, who used an oscilloscope with a dot rotating around a sphere 2 . We will refer to this method as Libet’s clock. Variations of this procedure (including current computerized versions), have been used to investigate a number of interesting research phenomena in psychology and neuroscience 3,4,5,6,7,8 . For instance, using Libet’s clock, Haggard and his colleagues asked their experimental participants to press a key in a computer keyboard anytime they wished while a dot was rotating around a sphere in the computer screen 3 . Upon pressing the key, a 250-millisecond delay was followed by the presentation of a tone. Participants were asked to estimate the position of the dot at two different moments - either when the tone sounded or when they pressed the key. The results showed that when participants were asked to estimate the dot’s position at the time of the key press, there was a forward shift in this estimation (that is, they judged the key press had occurred later in time). However, when participants were asked to estimate the position of the dot when the tone sounded, participants judged that it occurred earlier than the time of its actual onset. The misjudgment of the dot’s position in both situations is known as temporal binding, a subjective reduction of the temporal delay between the two events. In other words, the action and its consequence are perceived as being closer to each other than they actually are. This effect has also been called intentional binding 3 and it is the subject of intense theoretical debate 9,10 : Some researchers have argued that it is causation, rather than intentionality, that is critical in producing this phenomenon 11 , while others contend that both intentionality and causality are critical 12 . Interestingly, it has also been shown that when people get used to a certain delay between their actions and their effects, presenting the effects with a shorter delay may even reverse the perception of the events, so that the tone might be perceived as occurring before the action 13 . Other “time reversal paradoxes” have been described in the literature, including Warren’s phonemic restoration effect 14 , which also works “backwards” in time by restoring earlier missing information in a heard sentence based on later-arriving information. There is a large class of perceptual restoration effects, including the famous blind spot phenomena where we routinely fill in visual field percepts in the scene, when that patch of input is actually a scotoma induced by the missing retinal receptors where we might expect receptors to exist 15 . These are not so much “illusions” as “restorations” of the expected perceptual content, in spite of missing input information

Using Libet’s clock together with electroencephalographic (EEG) recording, a delay in the conscious experience of the perception of a stimulus has also been reported 16 , along with the timing of conscious decisions 2 . In those experiments, the participants’ brain activity was monitored while they were asked to estimate the position of the dot in a Libet’s clock when they “felt the will” to press a key, when they pressed it, or when a tone sounded after their key-press. The results showed that their brain activity was first in the temporal chain, with their “feeling the will” to act occurring after their brain activity had already started, and their perception of the action following their “feeling the will”. Their conscious perception of the sound closed the chain 2,16 . These experiments have fueled a very interesting debate on the notions of conscious will and sense of agency, suggesting that conscious will might be a by-product of brain activity.

Regardless of the merits of the various theoretical interpretations and debates that have flourished around these experiments using Libet’s clock, and even though some alternative procedures have also been developed (for alternative proposals see 1,17,18 ), it is clear that Libet’s clock is still a popular experimental paradigm that is used to study a number of interesting questions in psychology and neuroscience, such as the perception of external stimuli like tones, and internal cues such as the sense of agency and the timing of conscious decisions.

To the best of our knowledge, however, there is not a standardized version of Libet’s procedure that could be simply programed to fit one’s experiment. Such version would greatly facilitate replicability and homogeneity and should probably not be expensive to use in today’s computerized laboratories. By contrast, as we have been able to learn, each research group seems to have developed their own software over the years, which makes this procedure difficult to share among researchers (we were not able to obtain a copy), due to the “home-made”, personal, features of most current versions. In addition, little technical details are typically provided with manuscripts, which, given the extreme precision and accuracy that are required in timing studies, makes replicability difficult and opens the doors to potential software and timing errors (both at the operating system level, which may be open to timing issues, and at the level of the programming language used to develop the experimental paradigm). Even though it is possible that those potential errors are absent in most previous research in the literature, one cannot discard their existence when the articles do not provide the necessary details for code inspection and accurate reproducibility of the experiments.

Thus, having a public and standardized version of Libet’s clock which could be used freely to conduct experiments on sense of agency, intentional binding, and related phenomena should facilitate enormously the advancement of knowledge in these areas, the development of new experiments and the replicability of findings. With this in mind, we developed Labclock Web, a tool that researchers could use freely to conduct experiments. Labclock Web offers three main improvements when compared with current tools used to conduct experiments with Libet’s procedure. First, Labclock Web is open source software. It is free and is highly flexible, as its code is public and can be adapted according to the aims of each experiment (e.g., it might be connected to additional apparatus). If errors are present, anyone could detect them, report them, and solve them. Second, using external configuration files, experimental tasks are easily programmed by non-expert programmers. Although some basic computing skills are required, tasks can be adapted to each experimental situation without extensive knowledge of programming. Finally, Labclock Web operates in web browsers, a feature which allows for experiments to be conducted online.

Labclock Web takes advantage of the latest web standards to provide a multi-platform application for conducting online experiments. Below we describe the technical features of this tool and the tests that we have conducted to guarantee the accuracy of stimulus presentation–an issue of critical importance in experiments of this sort. In addition, and because the potential problems in running these experiments arise not only from technical aspects, but also from the behavioral ones (particularly in those experiments conducted through the Internet), we also conducted two experiments on intentional binding with human participants. The first one was conducted in the laboratory, whilst the second was carried out online. The results confirmed that Labclock Web is a reliable tool for conducting experiments in this area. They also show that reliable experiments can be carried out online but that some additional cautionary measures need to be taken into account in those cases.


What makes SmartSurvey right for you?

From event feedback to sophisticated market research, SmartSurvey is your best choice. Get our best features to reduce

Polls, surveys, questionnaires, online forms, consultations, quizzes, you name it, we have it.

No complicated installations or downloads, just sign up and create polls through your browser.

Intuitive user-led design, with easy to follow step-by-step guides.

Data encrypted during transit and at rest. ISO27001 and Cyber Security Essentials Plus certified.

Connect your data with APIs, Zapier, Salesforce, Google Analytics, and more.

Help when you need it, through email and telephone.

Security is our most important feature and we take it seriously

Security Features

Enable SSL encryption on any survey. Apply password protection and IP restriction on user accounts and survey responses.

ISO 27001 Certified

The highest possible standard for data security. An internationally recognised system for keeping information assets secure.

UK/EU Based Servers

All data is stored and backed up on UK/EU-based servers.

Data Protection Act

Fully compliant with EU Privacy Laws and registered under the Data Protection Act.


Experiment Builder

SR Research Experiment Builder is a sophisticated and intuitive drag-and-drop graphical programming environment for creating computer-based psychology and neuroscience experiments.

  • No hidden fees. Updates are always free and always fully backwards compatible.
  • Cross-platform compatible for Windows (32-bit and 64-bit) and macOS (OSX).
  • Delivers complex visual and auditory stimuli and registers responses with extremely high levels of timing precision.
  • Multi-language text presentation , with full control of text properties, html markup , and automatic interest area segmentation.
  • Displays video content with precise timing and frame-logging.
  • Powerful randomization options , including blocking, run-length control, and list counter-balancing.
  • Hundreds of existing templates that can be easily modified (e.g., change blindness, smooth pursuit, pro-saccade task, Stroop task and many more) are available through the SR Research Support Forum.
  • Fully integrated with EyeLink Data Viewer.
  • Compatible with all EyeLink trackers, facilitating cross laboratory research collaboration.
  • Supports both eye tracking (EyeLink) and non-eye tracking experiments.

If you would like to find out more about Experiment Builder and how it can help your eye-tracking research, please get in touch with us! You can also download the latest version of Experiment Builder by clicking the button below (you will need to be registered with our support forum for the link to work):


Introduction

The advent of the Internet opened new avenues of exploration for us as psychological researchers. Internet-based experimental instruments allow us to conduct experiments with demographically and culturally diverse samples, to recruit large subject pools in less time, to avoid organisational issues such as scheduling conflicts, to save costs related to laboratory space, equipment, personnel hours, and administration, and to increase our ability to conduct international experiments [1,2,3]. For these benefits to be worthwhile we must be able to trust Internet-based instruments to accurately record participants’ responses, in terms of both the actual responses as well as their intrinsic characteristics, such as response times. The current study investigates this particular issue by testing the replicability of the Internet-based implementation of PsyToolkit for use with paradigms requiring complex Choice Response Time (CRT) tasks.

It has been argued that, for instruments found to reliably record participants’ responses, Internet-based experimentation has three main advantages over laboratory-based experimentation [2] increased generalisability, increased voluntariness, and increased ecological validity. Increased generalisability refers to participants being able to be recruited from much broader demographic and/or geographic backgrounds, meaning that the sample is more likely to be truly representative of society. Increased voluntariness refers to participants having fewer constraints on their decisions to participate and to continue to participate as, for example, there is no researcher whose presence might socially pressure a participant to continue. Further, responses may be more authentic when participants are more comfortable in their ability to stop the experiment [2]. Ecological validity is a measure of the level to which participant behaviour in an experiment resembles their behaviour in a naturalistic setting. The closer to reality an experiment can be, the higher the level of ecological validity the experiment is said to have, and the more we can be confident that the results obtained reflect the participant’s real-world behaviours. As an example, driving simulators attempt to simulate, to different degrees, the feeling of driving a real car. The closer the simulator is to the experience of naturalistically driving a car, the higher the level of ecological validity. As such, an experiment in which you sit inside an actual car, observe a scene projected on the wall in front and to the sides of you, and respond using the car’s steering wheel, accelerator, and break is likely to have a higher level of ecological validity than an experiment in which you sit in front of a computer screen, observe a scene shown on the screen, and respond using controllers shaped like a steering wheel, accelerator, and break, which in turn is likely to have a higher level of ecological validity than an experiment in which you sit in front of a computer screen, observe a scene shown on the screen, and respond by moving the mouse on the screen to control direction and speed. With reference to internet-based studies, it has been argued that the ability for participants to take part in experiments in environments (and using equipment) that they are familiar with, and the ability for participants to undertake experiments without the presence of a researcher in the room, lead to increased ecological validity [2].

The ability to undertake experiments in familiar environments, and with familiar equipment, has the potential to enhance ecological validity in at least two manners increased familiarity and reduced cognitive load. Increased familiarity refers to the fact that participants can choose the time, place, and surroundings in which to undertake the experiment, ensuring that any effects found cannot be attributed to being in an unfamiliar setting [2]. Cognitive load refers to the amount of cognitive resources required, out of a limited pool, to fulfil the requirements of mentally demanding tasks [4]. In experimental terms, increasing levels of cognitive load are associated with increased reaction times, as participants have less cognitive resources available for undertaking experimental tasks. Unfamiliar environmental factors are known to increase cognitive load, as the level to which the brain actively monitors the environment is higher, which in turn reduces the cognitive resources available for other tasks. As such, the more familiar an individual is with their surroundings, the less cognitive resources are utilised in monitoring the environment, meaning that there are more cognitive resources available for focusing on the experimental task with which they are presented.

The lack of a researcher present has the potential to enhance ecological validity through reduced social desirability bias and reduced cognitive load. Social desirability bias refers to a cognitive bias in which individuals act to increase the level to which answers they give are in line with social norms in order to present themselves in the best possible light [4, 5]. The level to which this bias occurs is, among other factors, heightened in the presence of others [6]. As such, responses given in the absence of researchers are more likely indicative of how an individual truly feels about the subject, leading to higher ecological validity. Cognitive load is also reduced in the absence of a researcher, as the presence of others when undertaking a task divides attention, at least to some degree, between the experimental task and anyone else present [7, 8].

While increasing ecological validity is an important factor for experimental design, laboratory-based experiments also have advantages over internet-based experiments. Firstly, laboratory-based experiments have a higher range of possible research approaches. This is primarily due to equipment requirements. It is not reasonable, for example, to expect participants recruited from the general populace to all own eye tracking equipment as such, it is more logical to undertake experiments in which eye tracking is included in laboratory conditions. Further, hardware and software related issues have historically introduced a high level of error noise into results obtained through internet-based instruments compared to those obtained through laboratory-based instrumnets, primarily observable as response time noise. A wide variety of factors can affect response time recording, such as hardware timing features, device driver issues and interactions, script errors, operating system variability, interactions with other software, tools to construct the paradigm, interactions with other hardware, and configuration of settings and levels [3,9]. In laboratory-based experiments these sources of noise are less likely to affect the final results of the experiment, as all participants undertake the experiment with the same hardware, software, device drivers, operating system, and system configuration. In internet-based experiments, however, there are large potential differences in these elements between participants’ computers, which can lead to a higher level of noise within the results obtained. Further, responses given via the internet are also affected by the amount of time it takes for the website hosting the experiment to successfully send an image to the participants’ computer, and then, after responding, by the amount of time it takes for the response to be sent from the participants’ computer to the website hosting the experiment [10]. As high noise levels can obscure small effects and give the illusion of heterogeneous responses, care must be taken when analysing results obtained through internet-based instruments to ensure that an increase in heterogeneous responses are due to ecological validity improving rather than noise level increasing. However, technology continues to evolve, and recent advances in the design of internet-based experimental tools–such as has occurred with PsyToolkit, the instrument we present next–may have significantly reduced error noise compared to older internet-based instruments, even to the point of bringing them fully in line with laboratory-based instruments. As such, for instruments with which there is minimal Internet-related noise, if ecological validity was indeed increased we could (for example) expect participants to respond to items in a less self-monitored and/or socially accepted manner, with participants displaying wider response choice variability and overall faster response times.

PsyToolkit is an open-access psychological instrument developed to allow researchers, including student researchers, to easily program and run experimental psychological experiments and surveys for both laboratory and Internet settings [11,12]. Two versions of PsyToolkit are available a laboratory-based version that runs on Linux, and an Internet-based version that is Javascript based and can run on modern browsers without participants needing to download any programs. The Internet-based version of the instrument is specifically aimed at addressing financial and technical limitations commonly faced by students, as it is free software that has specifically been designed for running online questionnaires, Simple Response Time (SRT) tasks, and Choice Response Time tasks (CRT) [12]. A SRT is an experimental task in which a single stimulus, and only that stimulus, is presented repeatedly at the same on-screen location, with participants tasked with responding to every presentation of the stimulus in the exact same manner and quickly as possible [13]. An example of this is participants being instructed to watch an LED and to press a specific button as quickly as possible whenever the LED lights up. A CRT is an experimental task in which instead multiple stimuli are shown, and/or stimuli are presented on different areas of the screen, and the participant is tasked with responding in different manners depending on the nature of each presentation (e.g., Zajdel and Nowak [13]). An example of this is participants being instructed to look at a screen on which letters will appear, with the task of pressing the corresponding letter on a keyboard. CRTs can also differ in complexity. Simple CRTs, such as in the above example, require participants to recognise the stimuli and respond accordingly. More complex CRTs require participants to also make judgements about the nature of the stimuli.

In the present experiment, participants were instructed to look at a screen on which first names paired with role nouns appeared, with the task of pressing one of two buttons depending on whether they believed that it made logical sense for someone with the name shown to hold the role shown. Stoet [12] states that PsyToolkit is designed for a teaching environment, with minimal technical barriers and free web-based hosting of their studies. A library of existing psychological scales and experiments is available for students to examine and adapt, and extensive online documentation and tutorials are available to assist if students face any issues. Further, Stoet [12] states that PsyToolkit is designed to allow for students to randomise item order in both questionnaires and in cognitive experiments, to allow for a convenient way of scoring, and to give feedback to participants about their test scores options not available in all Internet-based instruments. All users of the Internet-based version must register an account to be able to create experiments, but accounts are free. Randomisation is possible in both the survey and the experiment, and partial randomisation is also possible for if one wishes for only certain portions of the survey and/or experiment to be randomised. Further, alternate versions of the experiment can be created, with participants randomly assigned between versions. In terms of reliability, Stoet [14] states that both the Internet and Linux versions of PsyToolkit can reliably measure small effects of less than 50ms, with the Linux version being more precise. However, to our knowledge, currently no research has been published examining the replicability of the Internet-based version of PsyToolkit.

As PsyToolkit is intended to be a student-focused instrument, and many universities do not set up experimental computers with Linux for their students, it was decided to compare results obtained through the Internet-based implementation of PsyToolkit to results obtained through E-Prime 3.0 in a laboratory setting. E-Prime was chosen as it is a commonly used psychological research tool in university settings, including in teaching environments, and, like PsyToolkit, it has a low barrier to entry and has an experiment library. Further, Stoet [14] states that the Linux-based version of PsyToolkit is on par with E-Prime, so, while there is likely to be noise due to differences in software, this is expected to be minimal.

While the replicability of PsyToolkit has not been examined, the replicability of other Internet-based instruments has been tested through CRT tasks (e.g., Reimers and Stewart [15] Schubert, Murteira, Collins, & Lopes [16]). Reimers and Stewart [15] used a CRT task to test the replicability of an experiment in the Internet-based version of Adobe Flash compared to the same experiment in a laboratory-based version of Adobe Flash, with the same experiment coded in C used as a baseline. Participants were shown green and red rectangles and were required to press buttons corresponding to the colour of the rectangle on the screen. They found that, compared to the baseline, (a) response times of the laboratory-based version of Flash were 10ms longer, (b) response times of the Internet-based version of Flash were 30-40ms longer, and (c) there were no significant differences in Response Time standard errors across conditions. Schubert et al. [16] used both SRT and CRT experiments in a study testing the replicability of ScriptingRT to Flash. Six experiments were conducted over the course of their study. The first three studies used SRT tasks but were automated to test specific aspects of ScriptingRT. The last three studies used CRT tasks, specifically a version of the Stroop task, where participants were presented with either the words “red” or “blue”, or a neutral letter string, in either red or blue on a white background. Participants were instructed to press keys corresponding to the colour of the word or neutral letter string shown. Experiment 4 tested the Internet-based version of ScriptingRT by itself, while Experiment 5 compared ScriptingRT to the same experiment coded in DMDX (a laboratory-based instrument [17]) with participants undertaking both tasks on the same computer, and Experiment 6 compared ScriptingRT to Inquisit Web Edition, both running via the Internet. In Experiment 5, the experiment of most interest to the present experiment as it compares an Internet-based implementation of the instrument to a laboratory-based one, Schubert et al. [16] found that the size of the Stroop effect was not affected by which software was used.

Historically, psycholinguistic research has not relied upon Internet based testing, as it often relies upon small differences in response times in CRT tasks to detect effects [18] and is strongly affected by response time noise. Recent research (e.g., Enochson & Culbertson [18]) has found that some modern Internet-based instruments are reliably able to test these small differences, meaning that modern psycholinguistic research may safely utilise Internet based tools that have been properly validated. Some researchers have suggested that PsyToolkit may be a delicate enough tool for psycholinguistic experimentation (e.g., Sampaio [19]). An opportunity arises therefore to test both general replicability and psycholinguistic specific replicability of PsyToolkit through a psycholinguistic experimental paradigm.

The present study was designed to compare responses and Response Times measured by the Internet-based implementation of PsyToolkit with those measured by the laboratory-based implementation of E-Prime 3.0 using a complex CRT task composed of an existing and published psycholinguistic paradigm (i.e., Gygax & Gabriel [20]) to test replicability between the Internet-based implementation of PsyToolkit (Version 2.4.3) and E-Prime (Version 3.0.3.31). The paradigm uses a between-subjects two-alternative forced choice design, with a CRT task in which participants are shown pairs of terms (in the present experiment a first name and a role noun e.g., ‘Kate–Chefs’) and are then required to, as quickly as possible, make a judgement as to whether the pairing makes logical sense (i.e., could someone named Kate be a member of a group of chefs). Experimental item pairings were composed of first names paired with professional roles that vary in gender stereotypicality. As logically any individual can hold any professional role, filler item pairings were included to prevent participants of developing a strategy of always answering positively to all roles seen. The filler items were first names paired with gender-marked kinship terms, with both congruent (e.g., ‘Kate–Mothers’) and incongruent (e.g., ‘Kate–Fathers’) pairings shown to prevent participants from developing a strategy of answering positively to professional roles and negatively to familial roles.

The paradigm we utilise is more complex than those used by Reimers and Stewart [15] and by Schubert et al. [16], as the paradigm used in the current study requires participants to make subjective judgements of the items presented before responding, while the paradigms used by Reimers and Stewart [15] and Schubert et al. [16] required that participants responded based on the colour, an objective quality, of the items presented to them. One can therefore expect that overall response times will be longer for this study than those found by Reimers and Stewart [15] and Schubert et al. [16], and, compared to Reimers and Stewart [15], it is likely that Response Time standard errors will be larger. Further, if the results indicate that there is a high level of replicability between PsyToolkit and E-Prime, then it may be possible to determine whether the results offer any support for the concept of increased ecological validity in Internet-based experiments. If the results obtained in PsyToolkit do have a higher level of environmental validity than the results obtained in E-Prime, we would expect that participants who undertake the PsyToolkit version of the experiment would be more likely to respond negatively, and would overall respond more quickly (i.e., more spontaneously), than those who undertake the E-Prime version of the experiment.

It is worth noting that Norwegian is considered a semi-gendered language. This is because some, but not all, nouns have associated gender markers. Specifically, only nouns that refer to living beings, especially humans, are gendered in Norwegian. Further, the majority of role nouns in the plural form are the same as the masculine-specific singular form. This is due in part to a linguistic policy of gender neutralisation [21], under which the masculine grammatically marked form of role nouns are actively encouraged to become the main linguistic device to refer to the majority of roles [22].


Statistics itself has developed a lot in recent years. Prominent statisticians around the world have introduced various new tests and analysis types, thereby adding new aspects and dimensions to the field of statistical analysis. Statistics involves multiple tests, correlations, variable-analysis, and hypothesis testing, that makes it a complicated process.

A statistical analysis software has the following features to make complicated statistical functions easy.

Below is the list of 10 free and open source statistical software

  1. JASP
  2. SOFA
  3. GNU PSPP
  4. SCI LABS
  5. Jamovi
  6. MacAnova
  7. Past
  8. Develve
  9. Invivostat
  10. IBM SPSS

Sona around the World

Social for Personality and Social Psychology (SPSP), March 2018, Atlanta, GA

Southeastern Psychological Association (SEPA), March 2019, Jacksonville, FL

European Association of Social Psychology (EASP), July 2017, Granada, Spain

Social for Personality and Social Psychology (SPSP), January 2017, San Antonio, TX

Chinese Psychological Society (CPS), October 2016, Xi'an, China

International Congress of Psychology (ICP), July 2016, Yokohama, Japan

Japanese Psychological Association (JPA), September 2015, Nagoya, Japan

Chinese Psychological Society (CPS), October 2015, Tianjin, China

Association for Psychological Science (APS), May 2015, New York City

Social for Personality and Social Psychology (SPSP), February 2015, Long Beach, California

European Association of Social Psychology (EASP), July 2014, Amsterdam

Japanese Psychological Association (JPA), September 2014, Kyoto, Japan

APS Annual Convention, May 2013, Washington, DC

CPS Annual Meeting, November 2013, Nanjing, China

SPSP Annual Meeting, January 2013, New Orleans, Louisiana

How Does an Exam Software Work?

Popularly used by recruiters and educational institutions, exam software is used for setting up online exams. The best online examination software helps with the following procedures:

    Students’ Registration

Online exam software helps with the registration process of students and generates unique IDs for them.

You can create a subjective, objective, multiple-choice, and other types of questions online and ensure zero spam.

Students can take tests from anywhere with a stable internet connection and a system. Similarly, teachers can invigilate directly through the system.

Teachers don’t need to evaluate answers manually, as the exam software helps analyze students’ performance digitally.

YouTube broadcast software enables users to list their live streams as videos on their channels. This way the live stream can be seen even after it ended.

The performance reports include detailed info about the strengths and weaknesses of every student. Accordingly, teachers can make the improvement plan.


4 Activities & Exercises

1. Solution-focused art therapy/ letter writing

A powerful in-session task is to request a client to draw or write about one of the following, as part of art therapy:

  • a picture of their miracle
  • something the client does well
  • a day when everything went well. What was different about that day?
  • a special person in their life

2. Strengths Finders

Have a client focus on a time when they felt their strongest. Ask them to highlight what strengths were present when things were going well. This can be an illuminating activity that helps clients focus on the strengths they already have inside of them.

A variation of this task is to have a client ask people who are important in their lives to tell them how they view the client’s strengths. Collecting strengths from another’s perspective can be very illuminating and helpful in bringing a client into a strength perspective.

3. Solution Mind Mapping

A creative way to guide a client into a brainstorm of solutions is by mind mapping. Have the miracle at the center of the mind map. From the center, have a client create branches of solutions to make that miracle happen. By exploring solution options, a client will self-generate and be more connected to the outcome.

4. Experiment Journals

Encourage clients to do experiments in real-life settings concerning the presenting problem. Have the client keep track of what works from an approach perspective. Reassure the client that a variety of experiments is a helpful approach.