-
Notifications
You must be signed in to change notification settings - Fork 5
/
Copy pathindex.html
1809 lines (1163 loc) · 41.6 KB
/
index.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<meta name="description" content="behavioral data collection, mechanical turk, psychology">
<meta name="author" content="todd m. gureckis">
<title></title>
<script src="http://code.jquery.com/jquery-1.10.1.min.js"></script>
<script src="http://d3js.org/d3.v3.min.js" charset="utf-8"></script>
<script src="js/remark.js" type="text/javascript"></script>
<!-- Bootstrap core CSS -->
<link href="css/bootstrap.css" rel="stylesheet">
<!-- Custom styles for this template -->
<link href="css/cdbo.css" rel="stylesheet">
</head>
<body>
<textarea id="source">
name: title
class: center, middle
<img src="images/jspsych+psiturk.png" width="650">
# Using .blue[Mechanical Turk] and .blue[jspsych+psiTurk] for Dynamic Web Experiments
<hr>
.author[
**Josh de Leeuw**, organizer
_Indiana University_
<hr>
**Todd Gureckis** ([@todd_gureckis](http://twitter.com/todd_gureckis)),
Anna Coenen ([@AnnaCoenen](http://twitter.com/AnnaCoenen)), **Doug Markant** ([@dougmarkant](http://twitter.com/dougmarkant)), Jay Martin, John McDonnell ([@johnvmcdonnell](http://twitter.com/johnvmcdonnell)), Alex Rich, and many github contributors!
_[Computation and Cognition Lab](http://gureckislab.org)_, _New York University_
]
---
class: middle
# Agenda
1. **Overview, What is Mechanical Turk?** What is important to know about running psychology experiments online?
2. **Basics of using Amazon Mechanical Turk**
3. Building dynamic web experiments using **jsPsych**.
4. Introduction and overview of **psiTurk**
5. We will get a quick tour of **psiTurk**'s experiment exchange.
6. Using **psiTurk** to put our experiment on AMT.
7. General Q & A with any remaining time.
---
class: middle
# What you will need to work hands-on
1. The tutorial materials from [www.jspsych.org/cogsci2014](http://www.jspsych.org/cogsci2014)
2. A programming-friendly text editor.
3. For **psiTurk** portions, it would be best to be running Linux or OSX to install on your machine. Windows users can run PsiTurk on a cloud server using OpenShift.
4. Register for an account at [psiturk.org/register](psiturk.org/register).
5. You will need an Amazon Web Services account to use Mechanical Turk.
<hr>
Feel free to leave any subpart of this out and just watch. Both libraries
are extensively documented online and so you can use the workshop just to get
oriented to the key concepts.
---
class: middle
# If you need help
1. All tutorial materials are available online. [www.jspsych.org/cogsci2014](http://www.jspsych.org/cogsci2014)
2. Interrupt and ask questions
3. Find a code-buddy
---
name: whatisamt
class: center, middle
.task[Part 1: What is AMT?]
# Online data collection?
---
class: middle
.task[Part 1: What is AMT?]
.left-column[
<img src="images/Computer_Workstation_Variables.jpg" width="300">
]
.right-column[
# typical laboratory-based experiment
<hr>
1. Office-style interaction
1. General control over lighting, temperature, viewing angle, distance
1. A few people at a time (1-5/hour), each on individual workstations
1. Data saved to local disc or to a local file server
]
---
class: middle
.task[Part 1: What is AMT?]
.left-column[
.center[
<img src="images/womenonlaptop.jpg" width="200"><img src="images/TabletUser.jpg" width="200">
<img src="images/smartphone.jpg" width="200"><img src="images/OutsideUser.jpg" width="200">
]
]
.right-column[
# typical online experiment
<hr>
1. Variable lighting, conditions, computer system, etc...
1. International subject pool
1. Many people at the same time (10-100/hour), each talking to a centralized server
1. Data saved in database to allow concurrent reading/writing
<br><br>
**This may not be all bad actually depending on your research project!!**
]
---
name: whatisamt
class: center, middle
# What is ".blue[Mechanical Turk]"?
---
.task[Part 1: What is AMT?]
# The history
- Developed by [Amazon.com](http://amazon.com)
- Originally for in-house use to detect duplicate product postings on Amazon's site .red[*]
<br><br>
.center[<img src="images/AMTLogo.png" width="300">
]
.footnote[.red[*] Nice summary in the [New York Times](http://www.nytimes.com/2007/03/25/business/yourmoney/25Stream.html)]
---
name: terminology
class: shortlist, middle
.task[Part 1: What is AMT?]
# Key terminology
- **HIT** = Human Intelligence Task (a unit of work, e.g. a trial or an entire sequence of trials in an experiment)
- .orange[**Requester**] = an entity (e.g., researcher) who posts HITs
- .blue[**Worker**] = a person who performs the task
---
.task[Part 1: What is AMT?]
# What kinds of tasks?
- Difficult for computer or machine learning systems
- Provide three key words to describe an image
- Does this photograph contain a car?
- What event does this twitter search refer to?
- Traditional work
- Write a positive review for this product online
- Translate this text from English to Spanish
- .green[Help with science!]
- **Participate in my human cognition/perception/learning experiment!**
---
.task[Part 1: What is AMT?]
# Who are the workers?
- 46.80% US, 34% India, 19.20% Other
- United States demographic
- 55-65% female
- Most make <$60k/year
- Median age of 30
- Hold bachelor's and are young
- Distribution mostly similar to US internet pop.
- See Ipeirotis, et al. (2010) or Mason and Siddharth (2011).
---
.task[Part 1: What is AMT?]
# World distribution
<img src="images/world-turk-distribution.jpg" width="700">
Tamir (2011)
---
class: middle
# Stephanie Costello
<hr>
"Costello lives in a trailer at the edge of a desert town in the Southwest. She is 50 and has an associate’s degree in nursing, but she has been unable to find suitable work as a nurse. In 2007, Costello was working at a boring office job and, in slow periods, earning extra money by doing online surveys on Mechanical Turk. When she lost her job at the start of the 2008 recession, she took to Turking full time, often more. What started as a source of extra cash suddenly turned into her main source of income. According to the 2010 study, Costello’s situation may be representative of approximately one in eight Turkers in America, or one in five worldwide." (from [Marvit, 2014](http://www.thenation.com/article/178241/how-crowdworkers-became-ghosts-digital-machine))
---
class: middle
# Why are people doing it?
- One word alluded to in previous passage: <b>money</b>
- Not just money but access to jobs that are open, flexible, challenging, changing, committement-free
- No travel costs, workplace politics
- For fun? Enjoyment of mental activities? To fill up an otherwise boring day/evening?
- For science: people generally may view participating in science studies rewarding
- "Money, and the best way to earn it, underpins much of the discussion about AMT" (on Turker Nation forum)
---
class: middle
.task[Part 1: What is AMT?]
# Worth letting these point sink in...
<hr>
- Despite being a simple mouse click away, crowds are composed of **people**
- Online labor markets naturally are appealing to people without other income means
- In many ways this is **good**
- At same time risk of depersonalization due to anonymity. These are real people and deserve respect and
thanks for participating in research
---
.task[Part 1: What is AMT?]
# Compensation
- Median wage is $1.38/hour
- Short tasks (~5 mins) award around 10 cents
- Requester can reject work and revoke payment
- And can also **award bonuses**
- Amazon takes 10% of payments
- Amazon tries to stay out of disputes
---
.task[Part 1: What is AMT?]
# Outside of the US
- Unfortunately, getting a requester account from outside the US can be a bit tricky
- Requires US billing address and US debit or credit card
- If you can't get a US credit card, setting up the account with a US American collaborator is an option
- Or consider international/European alternatives to AMT, such as [clickworker](http://www.clickworker.com), [crowdflower](http://crowdflower.com). The latter actually uses AMT workers, but is accessible worldwide
- (the psiTurk toolbox could in principle be adapted to work with other platforms)
---
.task[Part 1: What is AMT?]
# Why use AMT-based experiments?
- **Convenient** And the jspsych+psiTurk toolbox will help you even more
- **Fast** Collect a lot of data quickly, also good for piloting
- **Affordable** Workers will sign up for $2.00 for 15-25 minute session
- **Anonymous** Subject never meets experimenter
- **Replicable** Very easy to share your experiment code
---
.task[Part 1: What is AMT?]
# Why not?
Numerous concerns have been raised about AMT data:
<br>
- **Selection Bias:** No control over who takes the experiment
- 53% are self-identified liberals, 25% conservatives, 73% voted for Obama, see details [here](http://themonkeycage.org/2012/12/19/how-representative-are-amazon-mechanical-turk-workers)
- Makes AMT sample probably less useful for research on ideology, or political attitudes (see [this](http://www.culturalcognition.net/blog/2013/7/10/fooled-twice-shame-on-who-problems-with-mechanical-turk-stud.html) blogpost for a discussion)
- On the other hand, 20 year old university students are a very special sample too!
- **Population size:**
- Amazon claims 0.5 million registered users
- Effective population size of about 8000-10000 users
---
.task[Part 1: What is AMT?]
# Why not?
- **Contamination of subject base**
- Recent [study](http://www.jessechandler.com/uploads/2/8/0/5/2805897/13_chandler_mueller__paolacci.pdf) has addressed this question
- Can be a problem for *very* widely used paradigms
- 56% had participated in Prisoner's Dilemma, 52% in Ultimatum game, 30% Trolley problem
- Cross talk (e.g. in AMT forums) seems to be rare (workers share information about payment, duration, etc.)
- For specific studies repeated participation can be prevented by recording worker's IDs
- .blue[PsiTurk toolbox helps with this!]
---
.task[Part 1: What is AMT?]
# Why not?
- **Non-lab setting **
- [This](http://www.jessechandler.com/uploads/2/8/0/5/2805897/13_chandler_mueller__paolacci.pdf) study showed
- 27% were not alone while working on the HIT
- 18% were watching tv
- 14% were listening to music
- Recording how often participants change windows/take breaks might help clean data
- .blue[PsiTurk toolbox helps with this!]
- Still, the lack of experimental control over a worker's environment might make some studies unsuitable for AMT
---
class: center, middle
# How does the data compare to that collected in the lab?
---
name: replicationstudies
.task[Part 1: What is AMT?]
# Lots of interest in this...
.refs[
- Gosling, S.D., Vazire, S., Srivastava, S., & John, O.P. (2004). [Should we trust web-based studies? A comparative analysis of six preconceptions about Internet questionnaires](http://ww.w.simine.com/docs/Gosling_et_al_AP_2004.pdf). _American Psychologist_, 59, 2, 93-104.
- Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010). [Running experiments on Amazon Mechanical Turk](http://repub.eur.nl/res/pub/31983/jdm10630a[1].pdf). _Judgment and Decision Making_, 5, 411-419.
- Buhrmester, M., Kwang, T., & Gosling, S. D. (2011). [Amazon's Mechanical Turk A New Source of Inexpensive, Yet High-Quality, Data?](http://pps.sagepub.com/content/6/1/3.full). _Perspectives on Psychological Science_, 6(1), 3-5.
- Germine, L., Nakayama, K., Duchaine, B.C., Chabris, C.F., Chatterjee, G. & Wilmer, J.B. (2012). [Is the Web as good as the lab? Comparable performance from Web and lab in cognitive/perceptual experiments](http://www.springerlink.com/content/f0244t772070138w/) _Psychonomic Bulletin & Review_, 19.5.
- Shapiro, D. N., Chandler, J., & Mueller, P. A. (2013). [Using Mechanical Turk to Study Clinical Populations](http://s3.amazonaws.com/academia.edu.documents/30554524/Clinical_Psychological_Science-2013-Shapiro-2167702612469015.pdf?AWSAccessKeyId=AKIAIR6FSIMDFXPEERSA&Expires=1374090987&Signature=%2B4nErhKWOQhoWYY9gpgV0EbvVa0%3D&response-content-disposition=inline). _Clinical Psychological Science_, 1(2), 213-220.
]
---
.task[Part 1: What is AMT?]
# Do classic findings replicate?
- Some members of our lab recently contributed to this study that tested some classic psychology experiments on AMT.red[*]
- Crump, M. J., McDonnell, J. V., & Gureckis, T. M. (2013). [Evaluating Amazon's Mechanical Turk as a tool for experimental behavioral research.](http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjournal.pone.0057410) PloS one, 8(3).
- Focus was on reaction time findings that require sustained attention from subjects and precise recording of responses
- And learning experiments involving more cognitive effort
.footnote[.red[*] For a summary of the paper see this [blog post](http://gureckislab.org/blog/?p=1297)]
---
.task[Part 1: What is AMT?]
# Stimulus Detection
.left-column[
Stroop
- People respond faster to congruent (e.g. .blue[blue]) than to incongruent (.green[blue]) items
Task Switching
- Faster RT if task stays the same (e.g. "Odd or Even?") than when tasks alternate ("Odd or Even?" / "Small or Large?")
Flanker
- When identifying a target (e.g. "h"), congruent flankers ("hhhhh") lead to faster RTs than incongruent ("ffhhff") ones
]
.center[.right-column[
<img src="images/StroopResult.png" width="160">
<img src="images/TaskSwitchingResult.png" width="160">
<img src="images/FlankerResult.png" width="160">
]]
---
.task[Part 1: What is AMT?]
# Stimulus Detection
.left-column[
Simon
- Targets that are spatially compatible with response key are identified faster
Visual Cuing
- Stimulus detection is faster for cued versus uncued targets if cue is presented after short interval (<=300ms) and slower after longer interval (>=400ms)
Attentional Blink
- During rapid serial visual presentation visual target detection is impaired if target is displayed 100-500ms after another one
]
.center[.right-column[
<img src="images/SimonResult.png" width="160">
<img src="images/ContCueing.png" width="160">
<img src="images/AttentionalBlink.png" width="160">
]]
---
.task[Part 1: What is AMT?]
# Subliminal Perception
Masked priming
- Responding to arrow probes that are either compatibly (prime: >>, probe: >>) or incompatibly primed (prime: <<, probe: >>) at durations of 16, 32, 48, 64, 80, and 96 ms
- Previous finding was that compatibility effects are negative for very short prime durations and positive for longer ones
.center[
<img src="images/PrimingResult.png" width="330">
]
- Only partially replicated: no negative compatibility effect
- Lower bound on display duration in browser!
---
.task[Part 1: What is AMT?]
# Category Learning
- Classic Shepard, Hovland, & Jenkins (1961) paper
- Six different ways of classifying the same 8 items (geometric figures with 3 binary dimensions) into deterministic categories
- Generally Type I is learned faster than Type II which is faster that Types III-V, with Type VI the hardest
.center[
<img src="images/abstractstructureSHJ.png" width="150"><img src="images/dimstructureSHJ.png" width="170">
<img src="images/SHJDifficulty.png" width="550">
]
---
.task[Part 1: What is AMT?]
# Results
Although type I was learned easily, performance on all other types was significantly worse than in the original study
<br><br>
.center[
<img src="images/exp8-nosofsky.png" height="290"><img src="images/exp8-amt.png" height="290">
]
---
.task[Part 1: What is AMT?]
# Incentives too low?
There was little difference between high ($2 to $4.50 with bonus), medium ($1.50), and low-payment ($.75) groups
<br><br>
<img src="images/exp9-typeII.png" height="290">
<img src="images/exp9-typeIV.png" height="290">
---
.task[Part 1: What is AMT?]
# Bad instructions?
Including manipulation checks that asked non-trivial questions about the task improved performance on types II to VI
<br><br>
.center[<img src="images/exp10-l-vs-amt.png" height="310">]
---
.task[Part 1: What is AMT?]
# Replication in other Areas
1. Clinical (Shapiro, et al, 2013)
- Scores on a range of psychometric tests had high internal consistency and test-retest reliability (r =.87)
2. Personality (Buhrmeser, et al, 2011)
- No difference in consistency on personality questionnaires at different payment levels, and absolute levels of consistency and test-retest reliability were high (r = .88)
3. JDM (Paolacci, et al, 2010)
- Compared classic JDM studies (Asian Disease, Linda, Physicians Problem) on AMT and university subject pool
- No difference in failure rate of catch trials and pattern of behavior in studies but AMT subjects were more risk averse
---
.task[Part 1: What is AMT?]
# A word on Ethics
- How much should you pay?
- As much as the market supports or should it match what you pay in the lab (roughly)?
- AMT has been criticized for incentivizing companies to outsource labor without adequate compensation (for discussions see for example [here](http://priceonomics.com/who-makes-below-minimum-wage-in-the-mechanical/?utm_source=buffer&utm_campaign=Buffer&utm_content=buffer2d486&utm_medium=twitter), [here](http://economix.blogs.nytimes.com/2013/03/18/the-unregulated-work-of-mechanical-turk/), and [here](http://www.huffingtonpost.com/julian-dobson/mechanical-turk-amazons-underclass_b_2687431.html))
- Can you get IRB approval for your experiment?
- Arguably, workers have more freedom of choice than lab participants (can stop experiment any time)
- Wasn't very difficult to add to an existing IRB at NYU
- Common at many, many universities now.
---
.task[Part 1: What is AMT?]
# Be a good requester
- If your task breaks, workers *will* e-mail you!
- They also share information about good and bad requesters:
<br><br>
.center[
<img src="images/Turkopticon.png" height="350">
]
---
.task[Part 1: What is AMT?]
# Be a good requester
A few recommendations to keep workers happy...
1. Before running, .blue[test your experiment] on different browsers and the sandbox (AMT's testing environment)
1. Give explicit instructions and .blue[show warnings] about closing the page, going back in the browsers, etc.
1. Start collecting data in .blue[small batches]
1. If something goes wrong, provide an .blue[alternative payment method] for workers who are unable to finish or submit HIT
- One way of doing this is to provide a HIT with no content specifically aimed at workers that had problems and pay them using a "bonus"
---
.task[Part 1: What is AMT?]
# Part 1: Summary
1. AMT allows you to collect data quickly and conveniently
1. Web experiments can prevent experimenter effects and increase anonymity
1. Many classic psychology findings replicate with high fidelity
- But it's important to check for browser limits regarding presentation time, etc.
- It's also helpful to check participants' understanding of the task
1. Changing payment does not have a large effect performance but does affect signup and dropout rates
1. Make sure to build up a good reputation with AMT workers to make your life easier
<!-- - Note: dropout rates are much higher than in the lab, so maybe consider reporting those if they might interact with performance
-->
---
class: center, middle
.task[Part 1: What is AMT?]
# End of .blue[Part 1]
Questions?
---
name: part2
class: center, middle
.task[Part 2: Using AMT]
# Using AMT as a Worker and Requester
---
class: middle
.task[Part 2: Using AMT]
# Two perspectives:
1. What is it like to use AMT as a [**Worker**](#terminology)?
2. What is it like to use AMT as a [**Requester**](#terminology)?
# Two platforms:
1. The sandbox
- Worker sandbox ([workersandbox.mturk.com](http://workersandbox.mturk.com))
- Requester sandbox ([requestersandbox.mturk.com](http://requestersandbox.mturk.com))
2. The real deal ([www.mturk.com](http://www.mturk.com))
---
.task[Part 2: Using AMT]
# What is a HIT ?
- A unit of work, or a "Human Intelligence Task"
- In our experiments, a HIT is typically an entire experiment (as might be done in the lab)
- However, it is up to the **Requester** to determine what constitutes a HIT
- Could be one trial in an experiment, and you just offer the .blue[**Worker**] as many HITs as they care to do
---
.task[Part 2: Using AMT]
# Using AMT as a .blue[**Worker**]
1. Sign up on Amazon's site
2. Search for tasks (i.e., HITs) to do online through the AMT interface.
3. Worker accepts a HIT. The task appears in a frame within the AMT website.
3. Accepting a HIT means it is removed from the .orange[**Requester**] account. If the Worker decides not to complete the HIT, they have to _return_ it to the .orange[**Requester**] so someone else can do it
4. After completing the HIT, the worker signals to Amazon that they have completed the task
5. Payment will be credited to a worker's account some time after completion
---
.task[Part 2: Using AMT]
# Using AMT as a .orange[**Requester**]
1. Open a Amazon Web Services (AWS) account: [Subscribe here](http://aws.amazon.com)
1. Get a Mechanical Turk Requester Account: [Subscribe here](http://requester.mturk.com)
1. Create a "dummy" email account which you can use to deal with inevitable issues that come up with Workers
- Workers ***will*** contact you when they have issues
- Need a way to deal with this efficiently and quickly (your reputation depends on it!)
1. Use a built-in Amazon template or "External Question" approach to design your experiment
1. Test your code on the "sandbox"
1. Go live!
---
class: center, middle
# Let's request some HITs
---
.task[Part 2: Using AMT]
# AMT's built-in question templates
###Why use them?
- Hosted directly on Amazon's servers
- Good for simple surveys (multiple choice, text fields, radio buttons)
- Easy to use (don't need your own server)
- Tools for helping you "build" the question
- Basic HTML can be included (to format texts, or load data, images, etc. from a separate datafile, for example)
- HITs can be individualized with different content for each
---
.task[Part 2: Using AMT]
# AMT's built-in question templates
###But...
- Dynamic elements cannot be included!
- For those purposes, you want to use AMTs **external question** type
---
.task[Part 2: The mechanics]
# External questions
- You are responsible for:
- Hosting the experiment
- **At minimum** you need a computer with a web-accessible HTTP server running (a static IP address may simplify things)
- You probably want to avoid trying to host it over WiFi
- Posting your link to Amazon
- Communicating back to Amazon when the HIT has been completed
- Approving completed HITs for payment in a timely manner
- Dealing quickly with worker issues
- Saving your data in a secure place, etc..
---
.task[Part 2: Using AMT]
# External questions
- Interactions with AMT that psiTurk handles:
- Submit a request for HITs
- Provide a link to an "ad" that describes the HIT for Workers (this appears within the AMT site when they click 'View a HIT in this group')
- After HIT is accepted, provide a link to the experiment
- After HIT is complete, provide a button Worker will click to inform Amazon that they have finished. This submits a form with their identifying information to AMT.
<!--
.footnote[
psiTurk hint .red[[1]]: see templates/mturkindex.html<br />
psiTurk hint .red[[2]]: see templates/thanks.html
]
-->
---
class: center, middle
# End of .blue[Part 2]
### .gray[Time for a break!]
---
name: part3
class: center, middle
.task[Part 3: Building dynamic web experiments]
# .blue[Part 3]: Building dynamic web experiments using jsPsych
---
.task[Part 3: Building dynamic web experiments]
# Programming for the web:<br> .blue[The Good]
- Web programming was developed specifically for user interaction.
- Web experiments will run in the lab too!
- Tons of existing resources, both .blue[libraries] and .blue[tutorials].
---
.task[Part 3: Building dynamic web experiments]
# Programming for the web: <br>.blue[The Challenge]
Web development has a lot of moving parts:
- Code a .blue[web page]
- .blue[Custom web server] to serve the experiment and interact with Amazon.
- .blue[Database] for the server to store data.
---
.task[Part 3: Building dynamic web experiments]
# Programming for the web: <br>.blue[The Challenge]
Web development has a lot of moving parts:
- Code a .blue[web page] ✓
- .blue[Custom web server] to serve the experiment and interact with Amazon.
- .blue[Database] for the server to store data.
<hr>
**jsPsych** takes care of building the dynamic webpage.
<br>
<img src="images/jspsych.png" width="300">
---
.task[Part 3: Building dynamic web experiments]
# Programming for the web: <br>.blue[The Challenge]
Web development has a lot of moving parts:
- Code a .blue[web page]
- .blue[Custom web server] to serve the experiment and interact with Amazon. ✓
- .blue[Database] for the server to store data. ✓
<hr>
**psiturk** takes care of the server, data storage, and interaction
with Amazon Mechanical turk.
<br>
<img src="images/psiturk.png" width="400">
---
.task[Part 3: Building dynamic web experiments]
# Programming for the web: <br>.blue[The Challenge]
- Code a .blue[web page]:
- Structure coded in <span class="blue">HTML</span>.
- Style defined in <span class="blue">CSS</span>.
- Programs written in <span class="blue">Javascript</span>.
---
.task[Part 3: Building dynamic web experiments]
# What is JavaScript?
- For one thing, it has .blue[nothing to do] with Java.
- It is the .blue[only language] that browsers understand natively
- (but flash is also an option).
- These slides were compiled in Javascript!
- Javascript's role in a behavioral experiment:
- Control the <span class="blue">task logic</span>: condition, etc.
- Control <span class="blue">what subjects see</span> and when.
- Collect and store <span class="blue">data</span>, then send it to the server.
---
.task[Part 3: Building dynamic web experiments]
# What Javascript .blue[can] do
- Interact with a web server (to send or request data for instance)
- Track clicks, mouse movements, etc.
- Record response times
- Track changes in focus and window size
- Enable group experiments
---
.task[Part 3: Building dynamic web experiments]
# What Javascript .blue[can't] do
- Control focus (lock participants in a window)
- Disable browser actions (reloading, going back)
- Saving directly to a file (server interaction is required)
---
.task[Part 3: Building dynamic web experiments]
# Learning Javascript
Lots of resources for learning: E.g., [codecademy.com](http://codecademy.com):
<img src="images/codecademy.png" width="750" />
???
No time to teach javascript
One of the most commonly used languages.
One resource I recommend for beginners is codecademy.
Doesn't assume programming experimence.
Guides you through Javascript interactively.
---
class: center, middle
.jspsych[jsPsych]
<br><br><br>
A JavaScript library for creating behavioral experiments that run in a web browser.
---
class: middle
## jsPsych separates the structure and content of experiments
.left-half[
##Structure
* Show instructions
* Show a stimulus and collect a keyboard response
* Give feedback on a response
* Show a visual search array]
.right-half[
##Content
* What the instructions say
* How long is the stimulus displayed
* Which keys are acceptable responses
* How many items are in the visual search array]
---
class: middle
# jsPsych provides the structure, you provide the content
---
class: middle
# Structures are created by defining plugins
Plugins can be assembled together in any combination to create an experiment.
---
class: middle
# Why is jsPsych helpful?
* We tend to repeat the same structures across experiments, with variations in content.
* Other researchers often want to use the same structures to replicate a finding.
* jsPsych lets you rapidly assemble new experiments by providing all the functionality necessary to glue different structures together.
* jsPsych also provides a hanful of functions for interacting with data, randomizing factors, and developing new plugins.