Yesterday someone asked me how to ensure that gender was equally balanced across conditions within random assignment, and that reminded me of my intention to write a blog on using logic (if-then statements) in Qualtrics. Of course, as soon as you interfere with random assignment, it’s no longer entirely random, but that’s another issue, and you can (semi) randomly assign within condition such that you get a balance. Doing so requires using logic.

(see my blog post with tips about Random Assignment in Qualtrics)

Logic simply refers to showing questions according to previously determined conditions (if…., then). There are various ways to set up logic in Qualtrics. You can do it at the level of individual questions (using Display logic or Skip logic), or at the level of Blocks in Survey Flow, using Branch logic.  There are almost always multiple ways to get the same (or similar) result. I am going to use several examples to illustrate how you might go about setting conditions for the display of questions to survey takers.

Table of Contents

Logic with the Survey
Logic from Survey Flow
End of Survey Customization
Using time to screen participants
How to ensure gender balance across conditions

Logic within the survey.

The first place you will need to use logic is your very first question: consent

Anyone who selects “disagree” must be redirected the the end of the survey. In the above image, I have already applied skip logic to the question, as follows:

1. Click on the gear icon under the Question name (Q2 here).
2. Select “Add skip logic”

   

3. Select “Do not agree” and set it to skip to the end of survey.
   

   

This works well, as long as the end of survey doesn’t redirect to some place that will give the participant credit for participating. If you need the non-participants to go to a different end of survey message, you will need to set up branch logic in the Survey flow (explained later).

You can also use skip logic to have certain participants “skip” questions. For example, if want to ask follow-up questions based on an initial response, as I have in the image below:

The arrows will correspond to similarly colored arrows at the skip destination:

You’ll get color coded arrows for each set (skip logic and destination). It’s a good idea to follow them through to make sure you’ve done it correctly.

The question “Ride” at the right is the destination of two prior skip logic questions, coded in dark green and blue.

You must put a page break after the skip logic question, and separating each set of relevant destination questions.

Display logic, which you also access by clicking on the gear, can be used similarly. (You cannot use both skip and display logic though). To use display logic, click on the gear, select display, and choose the condition that must be met for the item to be displayed. In the example below, participants have to answer “yes” to a prior question (“Have you owned horses in the past?”) for it to be displayed.

Again, you must have page breaks separating everything that depends on conditional statements.

Skip Logic and Display Logic are fine for simple conditionals. However, if you want to make presentation of questions or sets of questions (or assignment to groups) contingent upon prior answers, then you must use branch logic in survey flow, which is much more flexible.

Logic from Survey Flow.

Go to Survey flow by clicking the tab at the top of the page:

Let’s start with the example of routing people who select “I disagree” to a separate end of survey (and not back to SONA to get credit for participating).

In the image below, you will see the flow of my sample survey. I have placed the Embedded Data block on top. (I am collecting the original source of the participant–covered in my prior blog on Embedded Data as well as assigned a random number that I can use to track them through a series of surveys). The second block, which I never bothered to rename, is the original first block, where my information sheet and consent form is. Select Add Below (you will want it immediately under the consent block).

You will need to choose the type of element to add: Add “Branch”

You will then need to add a condition:

Select “Question” and then your consent item, and then if “I disagree” (or similar) is selected:

Hit OK. Now when someone selects “Disagree” they will be shown whatever is under this Branch. Right now nothing is there, so they would simply go on to the next block. The flow simply goes to the next element down after completing everything that is under that branch unless you put an “End of Survey” in that branch. That is what you need to do here.

This time, select “End of Survey”

End of Survey Customization

This will take them to whatever End of Survey you have set up in Survey Options. If you haven’t done anything, it will be the default Qualtrics message. If you have set it up to automatically redirect to your departmental research pool system (SONA), then it will take them there. Smart undergrads might figure out they don’t need to do anything and will still get credit. You don’t want that, so you must customize the End of Survey:

Selecting customize will bring the pop-up below, and you must tick “override survey options.” Then you can select Custom End of Survey Message, or whatever you like, depending on your needs. In our example, we just don’t want them redirected to the SONA systems URL. But if you had other reasons for shunting certain participants off, you might want to redirect to a specific URL (maybe an external survey).

What about waiting till the end of the survey and not assigning credit if participants fail to pass certain criteria?

Go to the end of your survey flow (and make sure you haven’t added “End of Survey” to any preceding blocks). Add a Branch as explained above. If there is just one attention check, you can set it to select only those participants who fail it (assuming the end of this branch is a failure to complete and no reroute to credit). I do not advise this, because anyone can make a mistake. In my experience, collecting reams of data from voluntary participants, even the most well-intentioned person can fail to pass your attention check. I would recommend using at least three checks, and discarding only those participants who fail all three.

To do this, set the first condition, and then click on the green plus (+) sign to the right.

You can then add a second condition. If you choose “AND,” both conditions must be met to trigger the branch. If you select “OR,” then one (or both) must be met to trigger the branch.  Then just set the condition you want.

For example, if you have an item embedded in a larger questionnaire that says “Please select “Disagree,” then you will want to have everyone who did NOT select disagree trigger the condition.  Qualtrics allows to to set the conditional at the level of item response:

Using time to screen participants.

You can also eliminate people who fail to read your stimuli based on time. For example, say you want to eliminate everyone who spend less than 30 seconds reading a stimulus.

First, you will need to add a timing question to your survey if you haven’t already:

This will create a question that records the time spent on that page (so use page breaks intelligently. Randomizing questions with a block will invalidate timing questions).

Qualtrics outputs 4 variables with a timing question: time (in seconds) till first click, time till last click, time till page submit, and number of clicks on the page.. This is all potentially useful information, but we will focus on time till page submit.

In Survey flow, you can add a condition based on time (in my example, ti’s the third condition):

In the above example, the branch will be triggered if participants spend less than 30 seconds before proceeding to the next page.

Finally, you must add a customized End of Survey element beneath the branch to boot people out to a specified (non-credit-granting) end:

Don’t forget to Override survey options and customize the end:

I’ve saved several custom End-of-Survey messages to my library, and I can add the appropriate one wherever I want.

What about making sure you get a balance of females and males across randomly assigned experimental conditions?

Note: As soon as you interfere with random assignment, it’s no longer truly random.  But you could talk about matched pairs, etc. I don’t actually do this myself (just as I do not set up survey flow to automatically eliminate participants), but I’ve been asked how to do it, so here it is.

Essentially, you will use branch logic to separate demographic groups of interest (in this simple example, males and females). Then you will randomize within each group.

First, set up two branches, one for females and one for males. I always have an option for people to select “other” and specify, but we’ll ignore that here for simplicity’s sake.

Next, add a Randomizer element beneath each branch (see my prior blog on Random Assignment in Qualtrics). For example:

In the above example, I have interfered with randomicity twice. First, I separated males and females. Second, I selected “Evenly present Elements,” which means Qualtrics will balance across the two conditions (here, NEWS and Reality TV).

The final Survey flow would look like this:

Anyone who did not select male or female would simply proceed to the next block. Therefore, you should probably set up a third branch with the condition that neither male nor female is selected. Use “AND” and “not selected” twice, with a randomizer beneath it, for everyone else.

Place your outcome variable(s) beneath the series of branches and their sub-elements (randomizers).

That’s all folks.

If you’ve made it this far, please LIKE Qualtrics tips: Using logic and if you’re feeling generous, please click on the link below to buy me a coffee via Ko-fi 🙂