I’ve got to explain to my labmates how to use Amazon.com’s Mechanical Turk to collect data, so I thought I’d write a blog with some basic tips on using MTurk with Qualtrics.

Table of Contents

Potential Issues
Setting up Qualtrics
Setting up your Study on MTurk
Other Tips

(MTurk Requester UI Guide)

Academic MTurk Users Facebook page

Some time ago, Amazon doubled their fees on HITs with ten or more assignments.. it is 20% for up to nine assignments, and after that 40%.  This was disastrous for those of us in academia who have low budgets (and when we’ve got more money to spend, we often prefer to spend it on in-lab controlled experiments).

Luckily, there are many workarounds if you know a bit of coding, or if you know where to find coding.  When I first had to deal with this, I had to reference about 20 sites to work it all out. Since then, several people have been kind enough to post instructions.  This is a particularly clear explanation covering how to avoid that 40% fee (if that’s all you need, or if you  need more detail than I provide, I recommend you go to that webpage).

Here, I will provide general instructions on how to use MTurk, in conjunction with Qualtrics, to collect data. (More posts on Qualtrics: Random Assignment, Embedded Data, Using Logic)

See my post on how to avoid bad data when collecting data on MTurk.

Some issues for which you will need to prepare:

1. Workers taking your survey more than once: this will violate assumptions of independence, among other things.
2. Workers getting the Survey Code from other workers (there are many fora) and simply inputting it on amazon in order to get paid without ever even opening your survey (yes it happens, even though most mTurk workers are honest).
3. You not allowing enough time.
   First, remember that the MTurk workers must complete your survey, get the code, and input on amazon.
   Second, remember that undergrads complete questionnaires at light speed, or at least, in about one quarter to half the amount of time people “in the real world” will need. This means not only will you have to allow enough time on the MTurk side, but you will have to pay enough.  I’m not going to tell you how much to pay, but if you are paying 25 cents for 5 minutes (a typical amount), that’s $3/hr.  You will get lots of participants if you offer anything over $0.25 in my experience, but you’re going to piss a lot of people off if they end up spending 10 minutes for $0.25.  It’s easy to miscalculate, I’ve done it… just keep an eye on timing and adjust payment accordingly.

On Qualtrics:

1. Insert a “Text Entry” question at the end of your survey asking for the participants Worker ID (e.g., “Please enter your MTurk Worker ID.”).
   You will use this to verify that each worker has in fact completed your survey.
   Make sure you specify on the MTurk project that they will need to input their Worker ID.
   When you are ready to pay your Workers, download the Survey data file on Qualtrics, and use the Worker ID variable to make sure each worker has completed the survey and input his or her ID.  I copy the ID from MTurk, and use “Find” to locate it in the appropriate SPSS or excel column.
2. Set up a Custom End of Survey message.
   1. In “Survey Options—Survey Termination” select “Custom end of survey message.”
      

      

   2. You can see that I already have several such messages, one of which is “MTurk Numeracy.” They are stored in your Library.  You can create a new one from here.
   3. Use the Custom End of Survey message to give workers a code to input on the MTurk side in order to be paid.
      For example:
      Thank you! Please use the following code to validate your participation on mTurk: “HappyPuppy8” **See below for updated advice on codes.**

Assigning unique codes to workers (UPDATE)

I learned fairly quickly after writing this blog post that you cannot use just one code for workers to input on MTurk (people will input the code without having completed your survey. You should give each worker a unique code. You can do this via Survey Flow (embedded data) and piped text (see images below). (My blog post on Embedded Data.)

• 

• 

After you set up the Embedded Data, go to the last page in your Survey. Add a Descriptive Text Question with the words “Your unique code is” (for example), and then you will need to pipe the number assigned in Survey Flow into the text. TO do this, click on “Rich Text Editor.” Next, click on Piped Text –> Embedded Data Field, and select the field you created previously.

• 

• 

On Amazon’s MTurk site:

1. Sign up to be a Requester at https://requester.mturk.com/
   I use my regular amazon account for convenience, but you may want a lab or business account.
2. Click on “create”
   1. Select “New project”

   

    

3. Click on Survey link
4. Select “create project”

On “enter properties tab”

1. Name, etc., is obvious.
2. On “setting up your hit”
   1. Be sure to set “Number assignments per HIT” at 9. Once you do ten or more, the charge goes from 20% to 40%.
   2. Give them enough time to finish your survey and get back to input the code.***
   3. Set the HIT to expire in a reasonable time frame. I set mine at 4 hours.
   4. I leave “Autopay workers” at 3 days, in case I die.

      

***Sometimes people do not manage to make it back in time to submit the HIT.  If they have completed the survey and input their worker ID, I pay them anyway if they email me.
*** Make sure you go through and verify participation before those 3 days are up, preferably ASAP.  Download the SPSS (or excel) file, and crosscheck Worker IDs before paying.

For Worker Requirements:

I wouldn’t require Master workers. These are people rated “Master” at identifying colors or whatever for marketing firms. Not worth the extra money.  There is an article in an academic journal that supports this 😉

Add more criteria:

I set HIT approval rate at 95%

You can also set number of HITs approved, but remember there may be newbies who are very good, and not yet jaundiced.

Set location if you want to limit to US or English speaking countries, e.g.

Once you have workers to pay/rate, you can set Qualification Types.
Go to “Manage-Qualification Types” and create a new one:

You can give it a name; for example, “Personality Survey” Then, you can assign it to all workers who took that survey.

If you click on WORKER ID on the approve/reject screen, you will get to the above screen. For example, if you are going to run the same questionnaire for the same study at multiple times, you won’t want the same people to take it (otherwise, lack of independence).  So when you pay (or not) each worker, you can assign them a qualification type that means “already took this survey.”***

You can also set a qualification that specifies “bad worker” or “good worker”

5. Select “Only Workers who qualify to do my HITs can preview my HITs.”  Otherwise you are going to get people who do them anyway, and then get mad if you don’t pay them.

*** This is very important if you do not plan to collect ALL your data at once, or if you plan to use the same stimuli or dependent variables and don’t want participants to be familiar with them.

On Design Layout tab:

1. Provide survey link.
2. Describe survey in the instructions. Include “Please do not take this survey more than once.”  I state EXPLICITLY that I will only pay ONCE.  The way I set them up, they shouldn’t be able to do the HIT more than once, but some people are very creative.
3. After you have set up the description, select “Source”

   

   After you click on “source” it will look like:

   

4. Go to http://uniqueturker.myleott.com/
   (This will limit number of times workers can access the batch of HITs.  They are all published consecutively, and a person could feasibly do your survey many times.)  

1. Put your Survey name (eg) in the first field. (you don’t have to; I do it so I know what I am looking at if I save the script)
2. Click on “Get Script”

   

3. Copy the script and

Paste at the TOP of the source code on the MTurk page

Go to bottom of source code:

Add: <!–${hits}–> (that is two en-dashes: ! – – $ and } – -> without spaces)

You don’t have to use “hits”. You can call it anything. This is just the name of the column in the csv file that MTurk creates for you to use to set batches.  This will make it possible to create multiple HITS instead of just one.

Hit “Source” again and select “Preview and Finish.”

On the Preview and Finish Tab:

1. Click Download sample. A csv file will download.
2. Open csv file.
   • In A1, it will say “hits” (or whatever you put at the bottom of source code for MTurk)

• Underneath “hits” put a 0 (e.g.; you can put anything).
• Drag to fill the cells underneath with 0s
  • You need as many zeros as the desired sample size/9.
  • e. g., if you want 72 participants, you need 8 zeros in the A column underneath “Hits”

3. Save as csv file. You can rename it, just make sure you know where it’s at.

Go back to MTurk requester site. Save project if you haven’t already (select “Finish”)

• You will be taken to the Create page. Select ‘Publish batch’
  • You will be asked to upload a csv file. Upload the one you just prepared.
• After your file is uploaded and verified, you’ll be taken to the payment page:

“Number of hits in this batch” will be the amount of zeros you put in the csv file. I forgot to save my file with 8 zeros, and the one you download has 3 rows filled in, so here it says 3 HITs.

Number of assignments per HIT is 9, as per “Setting up your HIT” section above. This ensures you will be charged the 20% fee.

Make sure it’s charging you 20%!  (and not 5% more for Masters because you forgot to deselect “Masters only,” or 40% because you did it wrong and are publishing more than 9 assignments per HIT)

***You will of course have to fund your account first!  Follow links to do so.

Other tips:

• Check the email you’ve associated with your MTurk account frequently so that you may resolve problems as they arise.
• Pay people if they’ve done your survey, even if they don’t make it back to MTurk to put in the code.  In fact, if there is any possibility that a person completed your survey, pay him or her.
• Answer inquiries immediately, and pay as soon as possible.  They already earn little money (even if you are generous by MTurk standards), so don’t make workers wait.
• Thanks to a worker who took the time to explain how things work from their end, I know that there is a site where requesters are reviewed by workers: Turkopticon
• I advise keeping an eye on it… I had one worker–who had not actually DONE the hit, but just seen it criticized on the forum–complaining about one of my surveys (advertised as taking no more than five minutes for $0.35) because it took over ten minutes.  However, unless you sat for five minutes on the consent form, it couldn’t take that long, because I had set time limits on the tasks.  I believe they had somehow ended up on another (non Mturk) survey I had running.  If I had known about the incident, I could have cleared it up earlier.  As it was, the complainer apologized and offered advice.
• Mostly, workers are good people, but the reason I advise collecting their IDs and cross-checking in your data-file before paying, and watching the fora, is that some are not.  Because when in doubt you must pay, it is better to remain free of doubt as much as possible!
• Make sure your Qualtrics survey is free of errors. It would suck to pay 100 workers only to find you have to throw out the data!
• Amazon does not keep records of your payments for more than 18 months, so if you need receipts to collect grant money, save them from the beginning.
• Thanks to someone who shared this post, I was reminded of another useful site: Turker Nation

If you want to collect worker ids automatically…

Follow the instructions here.

You can do that before you follow the instructions above for avoiding the 40% fee and ensuring no repeat takers.

That’s about it…

I’ve just transferred these posts to this new blog. I lost some formatting. If there is information missing also, please let me know in the comments and I will try to fix it.

I will update if I find I have forgotten something. (and updates are needed now, coming soon.) I have not actually used MTurk since September, so there may be details I have forgotten!
Feeling grateful for all the time I’ve just saved you? Please buy me a coffee via Ko-Fi by clicking on the link below!