{"id":807,"date":"2016-02-14T01:41:39","date_gmt":"2016-02-14T01:41:39","guid":{"rendered":"https:\/\/jessicaeblack.wordpress.com\/?p=807"},"modified":"2026-04-09T09:40:05","modified_gmt":"2026-04-09T16:40:05","slug":"tips-on-using-mturk-with-qualtrics","status":"publish","type":"post","link":"https:\/\/jessicaeblack.org\/research\/tips-on-using-mturk-with-qualtrics\/","title":{"rendered":"Tips on using MTurk with Qualtrics"},"content":{"rendered":"\n

I’ve got to explain to my labmates how to use Amazon.com’s Mechanical Turk<\/a> to collect data, so I thought I’d write a blog with some basic tips on using MTurk with Qualtrics.<\/p>\n\n\n\n

Table of Contents<\/h5>\n\n\n\n

Potential Issues<\/a>
Setting up Qualtrics<\/a>
Setting up your Study on MTurk<\/a>
Other Tips<\/a><\/p>\n\n\n\n

(MTurk Requester UI Guide<\/a>)<\/p>\n\n\n\n

Academic MTurk Users Facebook page<\/a><\/p>\n\n\n\n

Some time ago, Amazon doubled their fees on HITs with ten or more assignments.. it is 20% for up to nine assignments, and after that 40%.  This was disastrous for those of us in academia who have low budgets (and when we’ve got more money to spend, we often prefer to spend it on in-lab controlled experiments).<\/p>\n\n\n\n

Luckily, there are many workarounds if you know a bit of coding, or<\/em> if you know where to find coding.  When I first had to deal with this, I had to reference about 20 sites to work it all out. Since then, several people have been kind enough to post instructions.  This<\/a> is a particularly clear explanation covering how to avoid that 40% fee (if that’s all you need, or if you  need more detail than I provide, I recommend you go to that webpage<\/a>).<\/p>\n\n\n\n

Here, I will provide general<\/strong> instructions on how to use MTurk, in conjunction with Qualtrics, to collect data. (More posts on Qualtrics: Random Assignment<\/a>, Embedded Data<\/a>, Using Logic<\/a>)<\/p>\n\n\n\n

See my post on how to avoid bad data<\/a> when collecting data on MTurk.<\/p>\n\n\n\n

Some issues for which you will need to prepare:<\/h3>\n\n\n\n
    \n
  1. Workers taking your survey more than once: this will violate assumptions of independence, among other things.<\/li>\n\n\n\n
  2. Workers getting the Survey Code from other workers (there are many fora) and simply inputting it on amazon in order to get paid without ever even opening your survey (yes<\/em><\/strong> it happens, even though most mTurk workers are honest).<\/li>\n\n\n\n
  3. You<\/em><\/strong> not allowing enough time.
    First<\/span>, remember that the MTurk workers must complete your survey, get the code, and input on amazon.
    Second<\/span>, remember that undergrads complete questionnaires at light speed, or at least, in about one quarter to half the amount of time people “in the real world” will need. This means not only will you have to allow enough time on the MTurk side, but you will have to pay enough. \u00a0I’m not going to tell you how much to pay, but if you are paying 25 cents for 5 minutes (a typical amount), that’s $3\/hr. \u00a0You will get lots of participants if you offer anything over $0.25 in my experience, but you’re going to piss a lot of people off if they end up spending 10 minutes for $0.25. \u00a0It’s easy to miscalculate, I’ve done it… just keep an eye on timing and adjust payment accordingly.<\/li>\n<\/ol>\n\n\n\n

    On Qualtrics:<\/strong><\/h3>\n\n\n\n
      \n
    1. Insert a “Text Entry” question at the end of your survey asking for the participants Worker ID (e.g., \u201cPlease enter your MTurk Worker ID.\u201d).
      You will use this to verify that each worker has in fact completed your survey.
      Make sure you specify on the MTurk project that they will need to input their Worker ID.
      When you are ready to pay your Workers, download the Survey data file on Qualtrics, and use the Worker ID variable to make sure each worker has completed the survey and input his or her ID. \u00a0I copy the ID from MTurk, and use “Find” to locate it in the appropriate SPSS or excel column.<\/span><\/span><\/span><\/li>\n\n\n\n
    2. Set up a Custom End of Survey message.\n
        \n
      1. In \u201cSurvey Options\u2014Survey Termination\u201d select \u201cCustom end of survey message.\u201d
        \"qualtrics\"<\/figure><\/figure> <\/li>\n\n\n\n
      2. You can see that I already have several such messages, one of which is “MTurk Numeracy.” They are stored in your Library. \u00a0You can create a new one from here.<\/li>\n\n\n\n
      3. Use the Custom End of Survey message to give workers a code<\/span> to input on the MTurk side in order to be paid.
        For example:
        Thank you! Please use the following code to validate your participation on mTurk:\u00a0“HappyPuppy8” **See below for updated advice on codes.**<\/li>\n<\/ol>\n<\/li>\n<\/ol>\n\n\n\n
        Assigning unique codes to workers (UPDATE)<\/h5>\n\n\n\n

        I learned fairly quickly after writing this blog post that you cannot use just one code for workers to input on MTurk (people will input the code without having completed your survey. You should give each worker a unique code. You can do this via Survey Flow (embedded data) and piped text (see images below). (My blog post on Embedded Data.<\/a>)<\/p>\n\n\n\n

        \n
        \"\"<\/figure>\n\n\n\n
        \"\"<\/figure>\n
        In Survey Flow, create an Embedded Data field (here, I named it RandomID). Use the drop down menus to select Custom Value –> Piped Text –> Random Number. Make sure you set the minimum and maximum to allow as many numbers as possible (to avoid repeats.)<\/figcaption><\/figure>\n\n\n
        \n
        \"\"
        The final set-up. Participants will be randomly assigned a number between 1000000 and 9999999.<\/figcaption><\/figure>\n<\/div>\n\n\n

        After you set up the Embedded Data, go to the last page in your Survey. Add a Descriptive Text Question with the words “Your unique code is” (for example), and then you will need to pipe the number assigned in Survey Flow into the text. TO do this, click on “Rich Text Editor.” Next, click on Piped Text –> Embedded Data Field, and select the field you created previously.<\/p>\n\n\n\n

        \n
        \"\"<\/figure>\n\n\n\n
        \"\"<\/figure>\n
        To insert the Random Code you assigned in Survey Flow, click on Piped Text –> Embedded Data Field and select the Random number field you created.<\/figcaption><\/figure>\n\n\n\n

        On Amazon’s MTurk site<\/a>:<\/strong><\/h3>\n\n\n\n
          \n
        1. Sign up to be a Requester at https:\/\/requester.mturk.com\/<\/a>
          I use my regular amazon account for convenience, but you may want a lab or business account.<\/li>\n\n\n\n
        2. Click on \u201ccreate<\/span><\/strong>\u201d

          1. Select \u201cNew project\u201d<\/li>
            <\/ol>
            \"iamge1\"<\/figure><\/figure>

            \u00a0<\/p>
            <\/li>\n\n\n\n

          2. Click on Survey link<\/li>\n\n\n\n
          3. Select \u201ccreate project\u201d<\/li>\n<\/ol>\n\n\n\n

            On \u201center properties<\/span> tab<\/strong>\u201d<\/h4>\n\n\n\n
              \n
            1. Name, etc., is obvious.<\/li>\n\n\n\n
            2. On \u201csetting up your hit<\/strong><\/span>\u201d\n
                \n
              1. Be sure to set \u201cNumber assignments per HIT\u201d at 9. Once you do ten or more, the charge goes from 20% to 40%.<\/li>\n\n\n\n
              2. Give them enough time to finish your survey and get back to input the code.***<\/li>\n\n\n\n
              3. Set the HIT to expire in a reasonable time frame. I set mine at 4 hours.<\/li>\n\n\n\n
              4. I leave \u201cAutopay workers\u201d at 3 days, in case I die.
                \"setting<\/figure><\/figure>
                <\/li>\n<\/ol>\n<\/li>\n<\/ol>\n\n\n\n

                ***Sometimes people do not manage to make it back in time to submit the HIT.  If they have completed the survey and input their worker ID, I pay them anyway if they email me.
                *** Make sure you go through and verify participation before those 3 days are up, preferably ASAP.  Download the SPSS (or excel) file, and crosscheck Worker IDs before paying.<\/p>\n\n\n\n

                For Worker Requirements<\/span><\/strong>:<\/h4>\n\n\n\n

                I wouldn\u2019t require Master workers. These are people rated \u201cMaster\u201d at identifying colors or whatever for marketing firms. Not worth the extra money.  There is an article in an academic journal that supports this \ud83d\ude09<\/p>\n\n\n\n

                Add more criteria: <\/p>\n\n\n\n

                I set HIT approval rate at 95%<\/p>\n\n\n\n

                You can also set number of HITs approved, but remember there may be newbies who are very good, and not yet jaundiced.<\/p>\n\n\n\n

                Set location if you want to limit to US or English speaking countries, e.g.<\/p>\n\n\n\n

                Once you have workers to pay\/rate, you can set Qualification Types<\/strong><\/span>.
                Go to “Manage-Qualification Types” and create a new one:<\/p>\n\n\n

                \n
                \"QUalitftypes\"<\/figure>\n<\/div>\n\n\n

                You can give it a name; for example, “Personality Survey” Then, you can assign it to all workers who took that survey.<\/p>\n\n\n

                \n
                \"qual<\/figure>\n<\/div>\n\n\n

                If you click on WORKER ID on the approve\/reject screen, you will get to the above screen. For example, if you are going to run the same questionnaire for the same study at multiple times, you won\u2019t want the same people to take it (otherwise, lack of independence).  So when you pay (or not) each worker, you can assign them a qualification type that means \u201calready took this survey.\u201d***<\/p>\n\n\n\n

                You can also set a qualification that specifies \u201cbad worker\u201d or \u201cgood worker\u201d <\/h5>\n\n\n
                \n
                \"using
                This worker completed a survey that included imaginative resistance scenarios, and gave good data. I would have assigned a zero if the data had been bad.<\/figcaption><\/figure>\n<\/div>\n\n\n

                5. Select \u201cOnly Workers who qualify to do my HITs can preview my HITs.\u201d  Otherwise you are going to get people who do them anyway,<\/em> and then get mad if you don\u2019t pay them.<\/p>\n\n\n\n

                *** This is very important<\/strong> if you do not plan to collect ALL<\/strong> your data at once, or if you plan to use the same stimuli or dependent variables and don’t want participants to be familiar with them.<\/p>\n\n\n\n

                On Design Layout<\/span><\/strong> tab:<\/h4>\n\n\n\n
                  \n
                1. Provide survey link.<\/li>\n\n\n\n
                2. Describe survey in the instructions. Include \u201cPlease do not take this survey more than once.\u201d\u00a0 I state EXPLICITLY that I will only pay ONCE.\u00a0 The way I set them up, they shouldn\u2019t be able<\/em><\/strong> to do the HIT more than once, but some people are very creative.<\/li>\n\n\n\n
                3. After you have set up the description, select \u201cSource\u201d
                  \"srouce\"<\/figure><\/figure> After you click on “source” it will look like:
                  \"Srouce<\/figure><\/figure> <\/li>\n\n\n\n
                4. Go to http:\/\/uniqueturker.myleott.com\/
                  <\/a>(This will limit number of times workers can access the batch of HITs. \u00a0They are all published consecutively, and a person could feasibly do your survey many times.) \u00a0 <\/li>\n<\/ol>\n\n\n
                  \n
                  \"\"<\/figure>\n<\/div>\n\n\n
                    \n
                  1. Put your Survey name (eg) in the first field. (you don\u2019t have to; I do it so I know what I am looking at if I save the script)<\/li>\n\n\n\n
                  2. Click on \u201cGet Script\u201d
                    \"unique<\/figure><\/figure>
                    <\/li>\n\n\n\n
                  3. Copy the script and<\/li>\n<\/ol>\n\n\n\n
                    Paste at the TOP<\/strong> of the source code on the MTurk page
                    <\/span><\/span><\/span>
                    \"paste<\/figure> <\/h5>\n\n\n\n
                    Go to bottom of source code:<\/strong> <\/h5>\n\n\n\n

                    Add: <!–${hits}–><\/strong> (that is two en-dashes: ! – – $ and } – -> without spaces)<\/p>\n\n\n\n

                    You don\u2019t have to use \u201chits\u201d. You can call it anything. This is just the name of the column in the csv file that MTurk creates for you to use to set batches.  This will make it possible to create multiple HITS instead of just one. <\/p>\n\n\n\n

                    Hit \u201cSource\u201d again and select \u201cPreview and Finish<\/span><\/strong>.\u201d<\/p>\n\n\n\n

                    On the Preview and Finish<\/strong><\/span> Tab:<\/p>\n\n\n\n

                      \n
                    1. Click Download sample<\/strong>. A csv file will download.<\/li>\n\n\n\n
                    2. Open csv file.\n