Yesterday I watched a friend suffer the consequences of not knowing the best way to get Qualtrics to spit out data in the easiest form. Instead of having a single outcome variable (what he was measuring to gauge the effects of a manipulation) and a single between-groups variable (one that reflected the random assignment), he had many variables, some of which were coded differently, that he had to combine.

That’s a headache, in any statistical software. So here I will walk you through how to set up a simple experiment on Qualtrics in a way that will give you a single outcome variable and a single predictor variable (factor or between-groups variable). My blog post on using logic can help with more complex designs. At a later date I might cover how to make sure all similar variables are coded the same way in Qualtrics.

Our example research question:

What kind of media makes people more anxious, the news or reality TV? We will have people self-report anxiety for the outcome variable.
Participants will be randomly assigned to one of two conditions, News or Reality TV.
Participants will watch a 3 minute clip (news or reality TV) and then complete the anxiety scale.
In the real world, we’d also be collecting demographics and possibly some other information. These will be included in our first block.

Basic set up for random assignment in Qualtrics:

You will need four blocks on Qualtrics

1. Consent and demographics.
2. News stimulus block
3. Reality TV stimulus block.
4. Outcome variable block.

First step: set up your blocks in the initial screen you get after creating a new survey:

Once you have your four blocks, select Survey Flow from the top tabs:

You will then be able to set up your random assignment on this screen:

Blocks are presented to participants in order.

1. Add a Randomizer block below your Consent by selecting “Add Below”
2. You will get lots of options for Add Below. Select randomizer
3. Qualtrics will randomly assign participants to one or more of anything you put below this. You will need to click and drag your conditions to where it says “Add new element here)
4. Below, you can see that I have dragged both stimuli blocks below the randomizer
5. If you leave it like this, all participants will see both clips:
   

   Select the quantity of blocks you wish each participant to view.

6. You will need to set that to “1” to present only one clip. You can also select “Evenly present elements,” which is minor violation of random assignment, but will prevent randomicity from happening in a disagreeable fashion (e.g., getting 75 people in group 1 and 25 in group 2. The larger the sample, the less you need to worry about this).
   Final view of randomizing block below.
7. As it is set up, participants will view (and complete) block one (consent and demographics), and then be randomly assigned to one of the two conditions.
   They will then all be presented with the Anxiety block (and complete the Anxiety Scale).
   When you download the data, you will get separates variable for the people assigned to News and TV (e.g., the variable that indicates the stimulus has been presented, and anything else, such as a timing variable that is on the separate blocks).
   You will get single variables for the Anxiety Scale (e.g., everyone will have A1, A2, A3, A4… completed in the same columns)
8. If you have any other blocks (e.g., questionnaires for covariates), you can put them below the Anxiety block, and everyone will see them. Or you can just add an end of survey block, as I have done below:
9. In real life, I have a customized end of survey message, but above it is just the Qualtrics standard.

End of Survey block

If you put an “End of Survey” block below a randomizer or any other sub-block, participants will not be presented with any further blocks (higher up in tree, but below the End block).

There are plenty of reasons to do this, but I won’t go into it now.

If you want to add a control condition, there are several ways to do it. You can set up a control block, with an innocuous message (matched in experimental blocks), that will be part of the random assignment below our randomizing block. Or you can set up a series of if-then statements, but that’s a bit more work.

To get a variable that specifies what condition each participant was assigned to, you need to select “export viewing order data for randomized surveys” when you download the data (export function on the Data and Analyses tab). You can then either use this variable (it will state the block each participant was assigned to) or recode it into a numerical variable, depending on the needs of your analyses and statistical software.

I’ll stop here for now, because this is the basic format. If you have a question, feel free to comment and I’ll try to answer it.

For questions concerning logic (conditional display of questions, etc) see my new post: Qualtrics tips: Using Logic.

To learn how to efficiently export data from randomization, see my blog post on Downloading Data from Qualtrics.

Feeling grateful for all the time I’ve just saved you? Please buy me a coffee via Ko-Fi by clicking on the link below!