ࡱ> @ %jbjb "8hhl ,}FFFFFF` l:<<<<<<,g \htFFtthFFFtRFF:""t:x>>2*>>}}>  > March 17th, 2009 MEMORANDUM To: Dr. Mary L. Spencer, Dean, College of Liberal Arts and Social Sciences Dr. James Sellmann, Associate Dean, College of Liberal Arts and Social Sciences Dr. Lilnabeth Somera, Associate Dean, College of Liberal Arts and Social Sciences Faculty in Psychology From: Dr. Kyle Smith, Professor of Psychology Re; Report of an assessment of hypothesis testing skills among students enrolled in upper division courses in psychology This report describes the design of, and results currently available from, a data collection assessing students' skills in hypothesis testing: a critical component of one of ten American Psychological Association (APA)-endorsed classes of goals for undergraduates in psychology, and of one (Critical Thinking Skills) in two that have been adopted as current priorities of the UOG Psychology Program. The report also describes consequent changes to the Psychology Program's approach to teaching skills in hypothesis testing. Although students in virtually all psychology courses read and hear about hypothesis testing, traditional assessments in those courses (e.g., examinations) frequently focus on the outcomes of tests of specific hypotheses (e.g., learning that making new material self-relevant increases the accuracy of memories for that material.) Most courses place lesser emphasis on the method by which hypothesis tests are performed. Of the current courses in UOG Psychology's curriculum, only PY413 Research Methodology in the Behavioral Sciences makes regular use of assessments designed to test the student's skill in conducting and interpreting tests of hypotheses. Because success (e.g., as researchers or consultants) for psychology degree holders often depends on skills in hypothesis testing, and because potential employers frequently demand such skills, it is important to assess these skills regularly, and to reinforce our curriculum to redress any deficiencies in teaching them. The instrument appended to this report, which is designed to measure several elements of hypothesis testing skills proposed by APA officer and critical thinking specialist Dr. Diane Halpern and by H. R. Riggio (2003) in their text, Thinking Critically about Critical Thinking, was administered to 18 students enrolled in PY440/G, a course that emphasizes empirical, quantitative research, 2.5 weeks prior to the end of Fall Semester 2008. Hypothesis testing involves operationally defining the variables of interest, proposing a specific and falsifiable relation between those variables (e.g., alcohol intake (as measured by blood alcohol level) correlates positively with the frequency of reports of disturbed sleep), and comparing the results predicted by the hypothesis with results obtained in research. Hypothesis testing also requires considering competing explanations for a given set of results, and if possible, eliminating them (e.g., through controlled experiments). The items in the assessment instrument were adapted from actual cases of media-reported hypothesis tests included in Halpern and Riggio's text as problems for testing and developing hypothesis testing skills. Adaptations converted subject matter to psychology specific content (e.g., medical hypotheses to psychological hypotheses. Students' answers were scored using the rubrics listed below: LEARNING OUTCOMEITEM 1ITEM 2ITEM 3ITEM 4ITEM 5ITEM 6ITEM 7ITEM 8Demonstrates understanding of the need to isolate and control variables in order to make strong causal claims0 to 20 to 30 to 40 to 50 to 30 to 34 (alt.)N/AChecks for adequate sample size and possible bias before generalizing0 to 1N/AN/AN/A0 to 2N/AN/A0 to 5Demonstrates an understanding of the limits of correlational reasoningN/A0 to 3 (5 if both)N/A0 to 3 (alt.)0 to 2 (alt. To sample)N/A2 (alt.)2 (alt.)Demonstrates an understanding of the limits of anecdotal reasoningN/AN/A0 to 4 (5 if both)N/AN/AN/A0 to 5N/ARecognizes biased information searches, including confirmation bias0 to 2N/AN/AN/AN/A0 to 2N/AN/A As an example, consider the first item. Dr. Hasslein, a clinical psychologist, sometimes uses a controversial technique in therapy. He uses this technique, a form of "implosion therapy," with those of his clients who are suicidal because they've lost a loved one. He forces the suicidal client to speak to a photo of the lost relative or partner, saying repeatedly, "I've got to send you away. I know that I will never, ever see you again." When some of his clients recover from their depression and suicidal thoughts, Hasslein attributes their recovery to the technique. When some of his clients commit suicide anyway, he concludes that their depression was so severe that even the implosion technique could not help them. Using your hypothesis testing skills, comment on Hasslein's observations and conclusions. Hasslein's procedure and conclusions (which parallel an actual case study of a physician's procedures and conclusions) are riddled with flaws. Hasslein has not compared the outcomes for his clients with those for clients who do not receive implosion therapy, to determine whether the former improve more often (i.e., he did not control the variables in question). An experiment, with clients randomly assigned to experimental and control (comparison) groups, would be required in order to make any causal claims. In addition, there is no evidence that Hasslein has applied his technique to a representative sample of clients. He may, for example, have applied his technique only to clients whom he believed to have strong prospects for recovery. If that sort of bias exists, any conclusions from the study could not be generalized to other groups. More importantly, Hasslein demonstrates a classic confirmation bias: attending to cases that support his hypothesis, and dismissing cases that weigh against it. Students obtained points for describing any of these problems, which are common in everyday reasoning and even in "research" considered by institutions and policy-makers. An answer considering all three problems would receive the maximum score of 5, as indicated in the rubric. Students also indicated which courses in psychology they had completed or were completing. Results The mean score (out of a possible 40) was 12.7, with a standard deviation of 7.27. Scores ranged from 2 to 25. Ideally, the more courses in psychologyand especially, the more upper division courses in psychologya student completes, the better his/her skills in testing hypotheses, and in critically analyzing well-designed and poorly designed tests of hypotheses, should become. However, no such reliable trend appeared in the present data. Students who had completed more upper-division courses in psychology did not obtain reliably higher total scores on the assessment: r (17) = .22, a nonsignificant value. Neither did total numbers of courses (100 to 400 level) in psychology predict total scores on the assessment, r (17) = .11, ns. Nor did the number of methods-based courses completed in psychology reliably predict total scores, r (17) = .29, ns. Changes in the Curriculum In response to these results, I developed a system for teaching and providing practice in hypothesis testing skills, to be used in preparing course specific modules. I have also prepared such modules (PowerPoint slides and exercises) for PY420 Abnormal Psychology (appended) and for PY440/G Social Psychology. At individual faculty members' requests, I will prepare additional modules specific to testing hypotheses relevant to other Psychology Program courses. Once more data using the assessment instrument are in hand, we may conduct a factor analysis of responses, to identify specific and largely independent elements in critical thinking skills (e.g., sets of items in the assessment that the student tends to pass or fail as a group), to allow us to select more specific targets for future revisions of the curriculum. I appreciate the College's support of assessment and of improvements to teaching. Attachments  cd  yz$l/ !!!!!!"" "$"%6CJCJ CJOJQJ6 jU& !l%&TU s t op+ $$Ifa$ $0^`0a$$`a$$a$ !l%&TU s t op+29@GNU\cd  PW[_cjnryz\`dw{#$01*+g  c+29@GNU\cd  PW[_cjnFf $$Ifa$Ff $$Ifa$nryz\`dw{|ެFfFf_ $$Ifa$Ff, $$Ifa$#$01*+g h " "$"$a$Ff $$Ifa$g h " "$"##a%b%%%% $"##a%b%%%%%%%%$a$ / =!"#$%Dd$pF  c "AC:\Documents and Settings\Cathleen Moore-Linn\My Documents\Letterhead Templates\CLASS\CLASS-SOCIAL BEHAVIORAL SCIENCES.psd.jpgR,>g4J|{08|DF>g4J|{08|JFIF,,C    $.' ",#(7),01444'9=82<.342C  2!!22222222222222222222222222222222222222222222222222" }!1AQa"q2#BR$3br %&'()*456789:CDEFGHIJSTUVWXYZcdefghijstuvwxyz w!1AQaq"2B #3Rbr $4%&'()*56789:CDEFGHIJSTUVWXYZcdefghijstuvwxyz ?( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( +<;& f$v^Uv.ǢXEHO""\=N;Wq_G9!DE7?:A|N>e'? igP[Cs $md`5=QAEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPES-tG8-<^%?W,> YN3;{M&=C<%й컒.] O|_2Lt zU;\9}WXl.w'|?QqSoډ|.|f3rl {70֍e6S̖pqZ)&hEEQ@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@r0=p,?uns=+k?4Ʃ "(d'M:GMPd {"%+hTּK+\FK&Dl$ޢaqi&Ku(63z֡ Y`d?AWFtKh([|cq p F36t_ִ'^yg>y-Դ9r찪֮?[x2ڷ M휳;(ތ>ҽ>۶9Y48״ܱɱ"9kj(߶Ĺ,,.ՄҰ7ٻf@98|;rΙ-C<ުa:G ]9~׮dr |E_xgPK ?w⼺kEt~~ERe)[F}EEx+W> ~a*5<d{v{/{Þ?Mb*yKnM"jzht<4V43Mbe}[[3i|A^O^OD滕cO¶-448-.5y`Jy15kK(V;)*ͯҪ]$i$Hz0?:WҨ[ YGJபԜiFדOY>W+kG>F*;ZFsVנPb 98>aB;84ӞϗfQET]LsHClDG2} xGjv wèxWf<?ب+$Ck5Rw{~5oD7If?¼`׍H,efMqy5I$S  U~-|2ŢkrXv=?ʾT%x:0e^#A$>ʵZFϴϰϰϰϰϰϰϰϰϰϰϰϰϰϰϰϰϰϰϰϰϰϰϰϰϰϰϰϰϰϰϰϰϰϰϰϰϰϰϰϰϰϰϰϰϰϰϰϰϰϰϰϰϰϰϰϰϰϰϰϰϰϰϰϰϰϰϰϰϰϰϰϰϰϰϰϰϰϰϰϰϰϰϰϰϰϰϰϰϰϰϰϰϰϰϰϰϰϰϰϰϰϰϰϰϰϰϰϰϰϰZj7ڴ簿5:ά;)lONș;#>%6#Ӱk[Ʋ9=]זdٰՓ=Ȯ>L^*&.$\ͭ.?T,r5?9Q51>a 2>n 5l\m.[mӏQA=P>\@ *֗XI"7cNŮ6z5㽝rٍI0aР'M5.nW,AE>rХ> qP:K湴[h^Y+p޴ӭWYP; SN38a4ļ`4lFrj #[ < r%Ǡ뻂T乇cOo40 ܚI=ƃ d7|%vpxG*,JVpxzZׇ5mRшhQi_Z躵jV0΁QVXQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEF0𿄮.c|]K?= WO/H'摽kwγEk]?@/Mg/t,-nS Z `A͞J̞CG'xCAqF^jm;XL|70F@'LWkyIy5M[־rմULe-wB=N:;?:ce#W#޾4]Jld0'#Ÿnc" ztaewS rjn-S0jւ8#$48d-fG.IUrrzҢ D;J+NF%֋pt`ǭZ#6K#`t~JӞ@_8v˨L`V8+-MSr}mG'{–=7ƝpxqZqHHێԡ^5&ԣ. thDjq $=1Gim^T5uKX&NExB{ЬNT:Z_o jkEloUy~'#?$q™fXƍsiWc~?{še .ׁ_@̯pcFΡ,1HHͷ}ǵtTm;s'KpĜ00+":tGhN$"k> kAКK93ZA{j}yEcsW^,(0ζ+c`((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((/>u I [ϳiVZ,Rpl&'dy?;wO=ݬw#L?5?< _ 69Z#feCgзK͋v{S4w 6'VdmAи?s:UG*>54+-ǵּٰݎˑ&2+g<'ҬbWڠlH>鴡S$×ʻvͥFűI$յ<<(|a+Ϋ[ބ2ٱt(_$Z$㭰YU?"+Of<HL׍x-I0 f_sQG&yZmK*v ;>ʟ(EM &k|)ah>ִⲹ$>NҰQ7Բѿ4\ߣⰨ5ʑ𷔳xD\3ɵ3*V8;N,璄ή47Mݎ<1Glo$Z1zW5|'WLҜf|aоғ[Li͏Ńhiz5Xʛvi98^}gs̅>T6~Ӡ1?|cUmrv0K'u'' GI$kۭ'DmfKlg,=w>ir|{Z{| {; V)8?í`8إo*LeOO־.(Š(((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((ac$VѤqdYd_DZ+ƺ?|:#[-ówq:ObZ uߥC A]ŧ]iDZ4Di$qӨjՎZj'D-aFg}GcfP ,kF+>iMx 7-Yv s_F1fBԃ CM>qgjEF~uM(-\`;t#nSX+]4cC+yqkq}* e_2|~H>L)w8tmaY7 \K2 O2:T_=͒!0yzíR oG: dP%/e:!3G=y|k &v5\10-׻i@~:?MZ8ۘeWhk6h$c"y*?Z8T`1a:8ןBѨ ~EUWL1}h/azXZq^[Q6ݏʠ=I j'tҠ 4c=+/??h՜J9E~:t!|WZ4`F;VR>G8}3~dugg:/jpÆǢEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEINa RaW˾-yUZM r! kGmTFX<+r,E(Ğ99FsxEsΡcpFD{5|@E xn|'$]*#[x_T.{r++FxcAy>dʿ'od9 +ԵYy) }޼W bѭ%u38a=P9_xo/Q}w^*NuA{%䝑]c27Os ܬ])>mO/iw Y1O-id4-^(Ƴ=bDm@ARmI;Md34m?*)skyC&o JӗEaJoluemWFHTyXatOEӚ=&v @C*g^ {hXmctӵ"WKCr⧊md!h[ܝ;?j( D@ޢ(V;)ʍwaQ6dU2}[SCO3_Aσ>-new$d{ IŻѨօ78 tU|Cc GZoX«2Q׼#4sq5ƙg(&KK#kXhTR7>\+&léd8ܻd_σ#Eɍ#us3J`de ҕf;#eV+FqN/h(((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((14bb04bah=@))j>G",ժ׏. lFf27PUE"*V]'zci)J%YHaRVF|ZGEݔ:F3ן|fU+#c:λVWOoqpwdʭ_obF#/GN7Kf%Uiċ\ $З2tRvWoR\U+ɧ(6:d8<mxL18!8Ue|$ jFܰp6[,vpR%<(QE"yq<H._בۼ0!Vg# v#v-׊Ҧn͹ny#Ө֎/C/ܭ%G]y׆dH"c0ʙ/˳ٰ4ܳݛ|v8חv3U3[b vՒ{3%ѴR@AN"u\2au_;ʏI睿K۱Z=[g?tKCdv3^B'}|14⟝݇գ]$ݴvoƽMS?ʼJC]yϡ77$چW2<$Ѭ>s$!V$)Zsm?dݏJr,=9vg sL,X%'rM[m\C2]F^GһV\"{wtdsyHZ_k׆=OϦֵeSm(:((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((( p Y‘"tW>&źUY`}?_6ݴR1R mpeКʪΪПV> G2sۼ_7|L`D+"0{^JNLOּ❳˫l;˭taœ'Nye(A*sWid3oߕyЖ22=^+c, rV| Nr+ijjf,2VfmΟH"/f?ZumjmWKm9tVI>r~u0PiATڙ-fz֦ΕFE597a|Vc++Yc%'/c^u뒚46q.e  iͤxSLBK,kz+%ݢ+(((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((+~;iMޕ@ |wrOuVq{fy?)+d,ź,Ⱦ`(yFtŻlZI-mfdSkM6я0>z]*,cwI.I+g&h-w!R`ث 3~"S~F:~#|?d',7rZصf=Uv[=wի}6pZާ5fP!0AԓBе&|CvOQQbv/<=X3K@]`q3g9x>UFJ?AtJ2?ZSlm4v[wd6I#;dV| Aay8uG( 2ɍ㲯v&ݎ|#fpkˋX'jamǭtjH鄹Ϥ4y(8G$Ӵ簷.a z p<'@kyh=-%GVgr3T*RvW:?E{a.FVWh{CszѹOkR9%.gvmâEtxW$K͵H;SUA#طTI*+wާ4\ k6 <> #+kNj(а(((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((R#E|'45?f8>c/ oQMז:+ JJ)Q7WBCHVv Y@&޼'qKLu?]6SoZok4M#Wi3ZcB@s]:3{K}.+]>Y(cruvC+⧚#T冟 DghqUD1^BW޲LtL?jܡ%{ pO2/{O^=סXXX:Uy'B-.}G@Z7x ޺/v搋>ҁ($+KO{Lդ*`[RODեKsD0{Uoyj0F$uj5ubn7+DOC&#M6خx jdoXyV?"r>$,̱F3#"&ѰKʸ}:VݼJI:к2<+ˎyMb;ŲH哢_ZjjOJ8j:̷+%~Htwk&wF܋i| 8l1M>WRѯN5/mܵˤ@Çӭ44{87.1<[(Š(((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((`~%xI)yKtL Rxvk 𸩒&JwEе/j eZn8@N2Oa^CL_v(;;cָ^*(d1'1Fzx\qF+;պ"2w  \d/ğ1x5 (;_zY(cojҝBHOzה6x_bkMVwEC"2:0!OjƬEf5]p&H9!Gƹ:[ 4(_r9CJ"񍭟ǡ#WdYra=O^sg +G#4ֵ7I4LD8 \o~7lؙc;",귷cwd8RxcM[6- zǕSɭ K'h6Ͻ,%ʢ4ժM󱫑Y*$ij™VRO\v&f#mJEUQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEyGş[}sNEfUUeN{\5IJH6 N;sҾկ 颕C~nt?ֳz=QE@Qĺr`2aЎc=C+ ަrzڻ^n,lmAI" 3ӽJIjLR ^}sesec ׭zt#UtV;D$nrk5b7r,I G>]ղݮv܌9<m?[gaʤnA#ߥx>_koGA袤[=o?lԖ'減l_ztT5sTl5ፎ%䰬T=y>*/mP\o 8UK⿅!g\\$ QꦘzGj˪%G:]?#AgQMo^ԑŽCs jI4ɥ@͐+kxl5 (G`t((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((()ԫ 0E:o 'McÐ ?h84[ffSڹN0B;*"f8UQ֥قNj:?'(|f;]8W fG3Kw=ifLA!6wԺ/l> A?ӳճ:#է§Q̳徱:#:ŷ7WZ=5_)?z<]IJ_dn쨄(u?ҩFZ?ÿ-KgvZ:/ﴛ㽶r}jonlahn"mں D;yd)!Ȍ f z7~-ax^JaM!}w9u#F%ņbރڽ;hX@DQS;0G鶺MVVQ,V "v+SP((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((ý#D->[_; O}qTtFxsz^H0Rr\o+Ҿxv+f]{P v zq[2&S̽̄rl?֬_x)I?h, 9vW籬ҳ+=N$-ٖ94q^?-iAϩ5VLb( bkM/;;R\bBy1Ꮚ?dBfŕf-Muç7n zZZ&te8#PoۢUg٣'|z<V6i$߶U$oλ |n@cx (wo{ǷW49H#B'(4yw<6>DֻmB=,d$ZVhՅQTPQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEV>+6Fi̿Cڶ(?]ǜ=yfjz5Q$\f̪wu9aCC|uajW" Y&"]ƽsNkŚY%āO{$>Ne ~+NPsnhBjgaQEYaEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPQ,q.dtAIYz$:wQܢAo!9Q-S]#趬 g׃O 6ta $﷮%JMUݰ87?bot{(am>6>pNFI%+kJ6RXٿ+/~O ݦ:7һ^u}V->4Ԣɉ'.Rɰ)쳦j)& <<yԵE5my5{X)$Vsgo$Um}~bиI+ R}sVEɶdl#wJO Aȳ zT䧥@e}>k?1btG ?Z&,6&-Jy>FHqe9_qBmmO G*1ed82H?lW75bn,JJM_AҵV+ѷ(}M1t]]3햟+/xnIđ@ ^ | mo~lJW:ԑ%GVjJvVupX eCvQ^}j/rS}΄6֧ed?nba/'ŲxOqjGfq 'SO4]WTO ^L\ێ[}l;dukL,1j4.ZJ!r ~ɡ#)juޗ{+IT>rCQ+c\[e}~ Ft=OPKDSXXzW"𽗈Ԡ76vs;۽ݝ̩nI_7UuOBzj3?ء{giB9d#~]Du f|WΞ%&MN="}bvFV_Z?Q.u٥534/#; W+doKߜi4R"dM߄ |*X2u/^\A L{-Ős^m*9$%#V5|;>2O=F't?c]6iݯٵ+Xav?#4^ ;x\|#cM{5>|>~5m/IY8CnS{#|/1 滙)Y f$%&'"ObWOOѼkjzMnm?1bk^Aljvq~ƾ_̒|Dss[)|MNV>hDqnE֏4yAar|~kgjnSjshB(^KX4';U-ؗ+QM29uQD2\AĒ"\!KXV>- _;SӢ+KI𧌢GWw~uS-Kr^."O"_bY/ע׆59({{ FZY! Ĺ[C긦QGꖲ4nUV!ˈޭz(((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((ȋs_)^^YD[AJ:QQsxn.hKʮ2o2-Zgd{WkYt_hz6y )Zj3!?qjRB14;coJ@5M~x츊YY] xpni3W@G"<úZgÌObq\v⯉!fV3efTGߓU1Qi:l:Fmanb0u+WSv_{{EmtO4Q`3OifUPI&E &2^<0$`6*}OKg%*# Lv&7aq4WF0a\í"} 6:uEF@qyzW[TGۿcTA_=|Hmnʻzύ1gbzǙ">c)iٳ+6B iBG+Ѽ x7O6HEprEr l7ڶyWKIks>679ْ)[𬿆4}ڜ[BR(=z챳Ib-CNBi{.YIbaAUl_xv-^WTg-lu_o/?DeqW_u/o^ABZ6`/:{!={R{?T (,ǟ]z@A_yK6'@ QZXO?I1LA"AZ^oO 1"aЊ?60sľvgJ^xAKf5jrߺ迅vtcKVJT5[4*W[4j(Iz)5Zx"Fq\*/.k & } Zyi-9)EXO'- %!EYCtime4p,ǘs j D::ce_O$f$jq½`|21_Paӛ1t>\Ա+˼-<5"Ey424d* W"W`G4HŸ**$ҽ wLtm:P io|%SL>,%p zeyȮZ#DQ^Dvv ygm$"A#4w[þ;w\% Z\ccHKs+ξ57*Au=>;HAY=k$R#+"פכ| ?멯I;|k' u_}_:RBg**VRNbK pqc[##nˍ! u7^ƢF{G澞"]e3ƞLe~5u|? !p ۀ%+zqwEEJav=&w<>ڷ;vyz/g]"w`Ip ,a~Aguhf#&Wv!7x9,?o$,Aq^_5x&o|PMs2Hm89WĊpx{÷6a,x("8|aOUiPw\@ݙX(mNT\TiMkկ7sG*}!Uu}}ϢKZQ@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@I"E;j7@+G[cbIw[sK[kվ)j :ե,b;5|"i]fȆ%WbKŶJ0{dz/>5mʇi8u?I uoJcE,Zx᧊M.f/2*>>+)JƗy+)'Hi)[ܲڒͻͳɏnD =vY'1s4,:;ubcSܞ' G]'=[|O5е6Tq3 'h!ynե}xᎳfeJ0G?d%+#*+G5h@ZZY7=VŞ0Ft+,lJr )XstNV)DQ_;?G6fftRL*x G~wQu^#؂ WuWqKiΡ=4K~z|IykGbC uRz&>$\lv>^\yJ5O>64JK# X#[49y|.i$Ga-#l𾴾"Ն8a^#G?DQEexZ|@M!ҊB&C081׳ڴ:{"=xmt[ v+4vw5}菫loa,--geCdU/N6&\_~>籷ϸfD2!դ>.UҳM7岵ڳܲ|ỳA:ֱQ^n~zl$di8u_5xvƿoOK9yKgUztMsLaEF-̒wqҚJQETMap,tbo%d+'5E.k1.=T4x~1M']$XiLa ~Uw>o;>Kyh&FXw/?~ i:u1f_1_ܵ۳MR,k)FV&X\ox>.>tt__NhR's~~4N #H7ZO'9aեfY8b=Uz5G״-f; z$H)=3?Vf_/+JF`桢OK5g IT<+[-5}2(O0gsV9xvj14V>[)@sJ -rO_>YԎ[1{3Ğ@>֢T5YnVVZ ~$ϊƉ{Kɓ2ه {zzQM^fkQw{nVxSq#Wtn8ahRCĿi-K$H῔ȣ'3r>~:M$~7??夕| ֡!ujҙmE1fLG'rd(?J3. Q?yDbFżd~RN,[Spi^_7xPI=ĞIE|l_ j[K$l8E_֙qᆻ I}hc,r<-ʟȣ9(<~kH6qDZ9?~ i:u1f_1_ܵ7&u|5YjzP#+IJG,l~.EzO9rܶdnus?)_,vcmѝK}q:Wmo}đIf W&25L_$-v+ED:5=Gŝ#ٳA|:xGPpf/6WoGǫPCwh"-zy_v&[XiK=7^v5jb(cވn BɑC< SZ-, HW* c:WB!qGs3~u(~RXww־wx.)_rO_Sv4Hun%}Qwǁ4|%]h7Y:Rx?Khِ.v\w5k+Z'qR2}X+N3tyQ ?lזiSNb+j[0]5_QKLFu?-a6Pjdw+MS牵R}FUf.>WjzCIo!DPD1ȩxA}GT%fad5EM[XnmJF@=`+{'YT?ГNcBh#R23l TtW@OWqMO=[6ʗdMEP7UrQki/..'D3"RHMv6hRgiL-q鰶#w[io*QDUp+kq%ws? j$?ۓ={Wu= ~".!!\xZloDueq;T t[X:DwVY< |v5_C+V?]kXSԭ>5Z7=$#fqΒ(,_/5?-75xbE614x)| yPtZ3V ծfn ךC#?5W6}}"Ak qtڭ+K GH%J>^o>y]⾲נVǓ#)rZcPxk˶O2;Tg1k gt--*%߄4O+$j;nJډS" ;Wο7qq *۹m?Z.E6-լ"j@Os9-4GZiʲ+b<9kAiI77տTPQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEYkφN^!3fz^V3b]p፭|{մ7v[O Q`F5xψ43Z堻nO>j%2b|J]|-n=vW?\LKQY 9l>-L>*rzťE̞p#Ih} zD''bcDZ^#} 2g.4YpDwٿ}Z[X-ksI~l|uD Iw>1e4Vtw|sc5 UxS^7zso&o#j\oIn൝!}PK[wɞ=zO"o=HC2H[k\$ "p^8F{{?ڔٯ4PQ>?N-Ի|)M5SQg)r&#osׁ 7:?^HhݺnM_#^>*O-كI#e_t0ڼﲵMmҶWufʲ$G3О7-Imٳ;6~%+/mҰo-妹.;f+վEsڐ_E9dܽA}ךWO_;Yh֚dOb$i qwⷊ sN`AO')) REO?댟Wu\_{;OEk{ye3lr=+A_R#E@AGV.?g׿߃=lrH[;KJLחxz# sސrdG##Uw^/}FIh#lqIs@~)yz Ϝ\]Śj2[X(Qg u Z-Zw22hala+!K4g&PSOx}ڵ Zax~c%ttiuU1^ BQV]M1G9Q\M}\Eނa|e7,>νE-'my$je lr)7J<Th;,)/FfP=+WV4kiS8̈Tg̏k&gH[?!/GImlX#_h]ƿԒIJ,fI63] 5"3۸fڲ9\ҽŞ),A"k<#C#ynӿoY޹u ) 'nЕ8<~]wX4,>\*e2Nsz\zX_BLXZU[G ҭQGڟC~*泧ɣC"yd0V'q=GZ".E 2/+ۆw`zӽDْٗ>O CiHe=`qZodRvZ,B]_OhVZqymy;ϧn%(Ԭ-}<T䍿GSd=}Lx).Ewxao|S'XV,)+)"c+A\ק|1[[2Ǩ"3qOC?{B-$dJ} .Ok_þ!_ӫ0-] ?L"~:jBL;@T?Sef Y_9|>={O/mOwZݬځeW]xm;W/-4$Fd r9tv"xS4Q<ƕ"$q\$k3ꗾ$. @YqQx"Ex<c`ђU/W7>e,R)eeYH>*99㇆ N\I[c(C1C/O=ʊ߅5~rʗ>Z*id=. OC~k8썴u`s̖ޡߊ.΋_+Q2-ǒQ_ᯂkũj99MkO/QsӰ_ZF ~9p'Ќ6v=Ɗk?ڕٯTP_>76G4-˘ 뱰AIi 1dx—ܧZB\uwDth,'lC_S Uq!s,XTROOhv E}%qE:X'?UώVwW>,iж>w'_tJR3ǚ ׌KW-^ihZDv;qs[u~(k^]XhƒXy#溯|CO|c dJ"d=}1JH~ 7pZΐ>V %譻ƽ'?7eWu $-z8At#=꽟mJ ?M(L`tyʧD'D._ |SgkMTt\1l5/©M UKhr?[5>*O-كI#W|-{&Ϣ)(ְ;ʵ◵'% <^p̗G<$‹+‹+‹+‹+‹+?a*,9I_*9_*iZiX[ߥ _7|-Vy/G/^υ_'M _E{?Q?Wdk~(/Q`gjOTiU_:oZߥ ?tKX9YړyUsyUzN>7|-Vy/GWdk~(/Q`giTjOU_:oZߥ ?tKX9YړyUw/^υ_'M _E{qTiU_:oZߥ ?tKX9YsyUړyUzN>7|-Vy/G/^υ_'M _E{?Q?Wdk~(/Q`giTjOU_:oZߥ ?tKX9YsyUړyUzN>7|-Vy/G/^υ_'M _E{?Q?Wdk~(/Q`gjOTjOU_:oZߥ ?tKX9YړyUړyUzN>7|-Vy/G/^υ_'M _E|~/# ܅1?Ο <tKdk~(sRVj_o'M _GN>€r_j_o-Ky[ 5dk~(/P}K-Ky[ 4e+oƺ7|-:oZߥ Ϲe+oƏL/m|7[υ_'M _@Y9/L/m|7 <tKdk~( >% <?2Կ畷]oN>7|-g2Կ畷G&Zk/Qυ_,&ZhRVu:oZߥ ?tKsRVj_o'M _GN>€r_j_o-Ky[ 5dk~(/P}K-Ky[ 4e+oƺ7|-:oZߥ Ϲe+oƏL/m|7[υ_'M _@Y9/L/m|7 <tKdk~( >% <?2Կ畷]oN>7|-g2Կ畷G&Zk/Qυ_,&ZhRVu:oZߥ ?tKsRVj_o'M _GN>€r_j_o-Ky[ 5dk~(/P}K-Ky[ 4e+oƺ7|-:oZߥ Ϲe+oƏL/m|7[υ_'M _@Y9/L/m|7 <tKdk~( >% <?2Կ畷]oN>7|-g2Կ畷G&Zk/Qυ_,&ZhRVu:oZߥ ?tKsRVj_o'M _GN>€r_j_o-Ky[ 5dk~(/P}K-Ky[ 4e+oƺ7|-:oZߥ Ϲe+oƏL/m|7[υ_'M _@Y9/L/m|7 <tKdk~( >% <?2Կ畷]oN>7|-g2Կ畷G&Zk/Qυ_,&ZhRVu:oZߥ ?tKsRVj_o'M _GN>€r_j_o-Ky[ 5dk~(/P}K-Ky[ 4e+oƺ7|-:oZߥ Ϲe+oƏL/m|7[υ_'M _@Y9/L/m|7 <tKdk~( >% <?2Կ畷]oN>7|-g2Կ畷G&Zk/Qυ_,&ZhRVu:oZߥ ?tKsRVj_o'M _GN>€r_j_o-Ky[ 5dk~(/P}K-Ky[ 4e+oƺ7|-:oZߥ Ϲe+oƏL/m|7[υ_'M _@Y9/L/m|7 <tKdk~( >% <?2Կ畷]oN>7|-g2Կ畷G&Zk/Qυ_,&ZhRVu:oZߥ ?tKsRVj_o'M _GN>€r_j_o-Ky[ 5dk~(/P}K-Ky[ 4e+oƺ7|-:oZߥ Ϲe+oƏL/m|7[υ_'M _@Y9/L/m|7 <tKdk~( >% <?2Կ畷]oN>7|-g2Կ畷G&Zk/Qυ_,&ZhRVu:oZߥ ?tKsRVj_o'M _GN>€r_j_o-Ky[ 5dk~(/P}K-Ky[ 4e+oƺ7|-:oZߥ Ϲe+oƏL/m|7[υ_'M _@Y9/L/m|7 <tKdk~( >% <?2Կ畷]oN>7|-g2Կ畷G&Zk/Qυ_,&ZhRVu:oZߥ ?tKsRVj_o'M _GN>€r_j_o-Ky[ 5dk~(/P}K-Ky[ 4e+oƺ7|-:oZߥ Ϲe+oƏL/m|7[υ_'M _@Y9/L/m|7 <tKdk~( >% <?2Կ畷]oN>7|-g2Կ畷G&Zk/Qυ_,&ZhRVu:oZߥ ?tKsRVj_o'M _GN>€r_j_o-Ky[ 5dk~(/P}K-Ky[ 4e+oƺ7|-:oZߥ Ϲe+oƏL/m|7[υ_'M _@Y9/L/m|7 <tKdk~( >% <?2Կ畷]oN>7|-g2Կ畷G&Zk/Qυ_,&ZhRVu:oZߥ ?tKrQLZX)cշ(oQ_F˟WG~&,oafF*wrk,׿fS]kH.utֈ±`Cݯ&n]Yrha'ck_#kٟd}q~;՟kk| z!tQE2Z)i5`uYOWRD\(BkCc#7+-o݆Pv:(ylRWMe}li"6o'ެx J#?Ȳjv.=Dd @&k-1L ? tk3eu{N-ǒMԷ\Z<:t05`USVZfOuCgm*IsoWXȬs+okFIX'@QP2LSS,A5֗s Є>A#?ܛ8^ aG5r[GwdWݟZL{ac ?U^.c3uLc[l'# ]4QE2Š(u/isIieV>Zw5&twdYN>xfI%ubiK 瓼WIvjĨ89 l$jۘyX+KI[hyZ_~WOq o둦7WD*kW ڼEFrXտ62\.g_u )h~&MnHV؁X] y'u\^BחQZIq'AY3أӐ;;~ݑڬz sZU >p'QWRAtx4Y#VywV\'LRvG[ߍKMPgi9#|F$F֭/ `|3woX).ƹMf9Qސ74XGtqU-~Lh+[סфA0[5&?uNb_lui1IyV%s avӺ kRiFceH`ޯ'׉]~샆dzLRv=bKkc}wx/u[3&,{&>=2(PQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEW? 1$oA]x2hmـMfL~ݐ?{|-|6 ՚rpk blc2oN3OJuhڟ"<sیnrs޺iKgo*,HQr~pW.Q^/uZf.ѴTjo|`[ĩKGD^5d1X=G22=a=;`$CnzCU}$z7!Op` i,-!QKq^iொž$m%l U7%|}7>o~)S18 ?LºJߋwR:7ە+c^O_5RvCn_ krxöګOy۞p:GǿJK"S#M.IӞ6vYina1@Si{2Ge)#HPϹ ғi N ==[o R/?V5/iZ|e }A"獼;xy46xH" 37j~xjY${%|sӅMBoT˨_N~3Icw2c { fw\4:GX- :4{ C̃MX~u "ж>P~~2|ic]_[@J3II|eÛs,pC^a?xF;D($.0p4Ϥ(/sBo2iZ0^̑9FӒO|T&2pqW2ukEx]/Q7,Ҡ9?.k/Ûo Lp.d.dz'Ŀ^ UHfd59HX5xWÚiz;NwGKmճRM0Pмy^?%U1bFG'u Q:eqY6skK BMڭE HFbzz__Vk?h/H?H> EJ/tdyEVC>s!s#بG%Fe5Ss+)Y rxşi(_&IWw|m#އ$W/u∴xf1#I Kɕv('ڼ⿌٥Þkh1I)VVQjnIi\/į\x+I.&(PJ x#O_k\ȱ,f*sroLjt9 Kl삥K9itzu{{vQqr#$@qیb,FixktT ѪEJQu9t {&$( }½Sv݉ղƅYj2[vedFG;:Cڗ2̏ǟ<٪ = 2⧏t45mKyT~9#(.&7?*)"W9o0l=9>,U{#0ID~f$=X0=Ȁs_x~U[7weEW%\/zsnU (cF۟W~Z,$d ׏j,kS0~Rif`M7uνRGVRC2%8dשЇ+C ܏){^^k]\s]t<|NW3v w(UHYnzk4'kןiIm8']$A-YES5< 9[jV ʼ+6U R"[heo῵{iQ5^>}yv:4}8!8U5ϯg &Dԯ]}=>ĐF?x]yfbr8RB ۯʼFv|3^67/5?r#>\c8{1>|ȟjX:wښ+)9@nyώ?2?z5yύրT1Oc e\SW)@Nvƪ~b/okFAT]&\夕 GjՑzZ-STP⣹m)&Q.#Ђqj{chd 6¨Y-i,a.yڵ?&IRς5:K7l'+즺y4O"M+T/XH[!+"^ks*(pINnfgU@F}Izϋ4綂R݅;JEh|M"r5Tu[S}ڎ!oc/gr/PC +$gε忰{khYw,{R!5a<!;?* < a* #\5ЊrH!w8U$23kd϶&|ݔWY-@Z?gV\֋W0p.s1KWdE6"Eq2+.)פYiVneS =uq=Z} #:|yVC]p=OM sة+Ї;_28܍.>2O?j_\Id*J,נdz?^}MV%‚? jz(QEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEVVXxKNԭİ?S꧱^C⟊_wg&eڱg8`܂3ӊMKs^#/$0 : ~(Dٌj*ʫ_r%&ZM] TF.gz>^U2$㬵/~)k^z un Bn]dN=~uM DG*i$?W@j{I#ǛWI46uH ~XkkH^;Zމ&I!wIą(#"XҼ3!ZOE@;7pNwOwG Z~X<'}zGqtP1Z[C,=ҖoCWANm3iX1aΟh+^J(y@k-Ǘm=Ż,3dmlU_>%hPi*5#'|ҦL3м6wyK&sj AL>$ ֵu+|BZB3yL z5Y sC*309ػH˛l&…ݏ`/<xy8}#sem,I[5$ǏaWhH\\LsSՍȕDTW^Ѭu QT϶q5܉qgH=[ֽOg ^x4i "*@k:Ekt='>ޔkI,&M"܅@$0MyO ks[d瞝 ׮xfӼţurrWjn? 'I4QʻZI;gڍH7I"[>r\y iѴ]IӬV"}Zmn^4k *;qnzc!v>}sxN$!#USׯl|ŤklΠ RO^I_Dty%R t(~*.ZF>w )K2vHӽ^U'Q^>(jz̜InGfHOuv-Đ!eA\&{K-b #idHHNz(z7F/ǛImٗdi4x@ t .RY\O^ligIqx3jKXz[>~l3룲Ѱ3ǹ1}4۲.Q[.XÛba\Î{x=[/|mtA|j3,z_?Rkѫ J|B@oNvkzψ4)o/J !d`.:yTZqkc};R5˯.+] ${p+uC=eǘDl9088k(׶{XG|' nÏ{|JW K]Ӹ]+,K_UEpn .WxdBܿ͌rjna6=Im&Q;:ǽhVtft)^Kxt,rN=kzQE0 ?.$fD߆kvb~])uŠE +8hpPw֍Vt5ڊ_4#A =F(̗C'7Ғ9ȭ:('дiay嘎Mi@qxIa 8 FQ@VsUU('T5>ʵVD0) H4P:SQE!FV#]MKminQh@Q@\[AwxD=A|)+vdUʵ"L ;ժ(c5 8ʐAXΌ:iUEc'Ur&%Hn"Ee&)!Vڠ)PP\[CunNU= OEQl,q1%GjE>c ,rMh*EP0:*六%Pr EUʵ"^TQ@Q@Q@g_hun`7zѢ0º26fh-ᶌG j:*j(QE ;4qlU;[tP+!FpQE(]V7ukåXTF!+Z]֚EK[Xr0J~*ޗckX+pW-ᴁ`EHaTt^J^9YFaU( {xmaBEOEhk۷n#֝P.P(vS;w->Mkvf+8qQw֍Vt=w7C}+F(((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((1$$If zJ!Z$*'8'6$$$$2 2 22 22 22 22 22 22 22 22 4a1$$Ifp zJ!Z$*'8'6$$$$2 2 22 22 22 22 22 22 22 22 4a1$$If zJ!Z$*'8'6$$$$2 2 22 22 22 22 22 22 22 22 4a1$$If zJ!Z$*'8'6$$$$2 2 22 22 22 22 22 22 22 22 4a1$$If0 zJ!Z$*'8'6$$$$2 2 22 22 22 22 22 22 22 22 4a1$$If0 zJ!Z$*'8'6$$$$2 2 22 22 22 22 22 22 22 22 4a i4@4NormalCJOJPJQJmH <A@<Default Paragraph Font8 !z z zo +s%+n$"%g % OLE_LINK1t~X,7YY Kyle D. SmithAMacintosh HD:Users:kylesmith:Desktop:Critical Thinking report.doc Kyle D. SmithZMacintosh HD:Users:kylesmith:Documents:Microsoft User Data:AutoRecovery save of Critical T + 2 9 @ G N U \ c d   PW[_cjnryz\`dw{@| @GTimes New Roman5Symbol3 Arial3Times 1h FFxF>H@ '$0dUr@This report describes the design and results currently available from a data collection assessing students' skills in hypothesi Kyle D. Smith Kyle D. Smith Oh+'0 ,DP` x    'This report describes the design and results currently available from a data collection assessing students' skills in hypothesihisKyle D. SmithscyleNormal Kyle D. Smithsc62eMicrosoft Word 10.1@ @H@ y@fw@ ՜.+,D՜.+, hp  l''Psychology Program, 91첥g' r This report describes the design and results currently available from a data collection assessing students' skills in hypothesi Title 8@ _PID_HLINKS'A8weC:\Documents and Settings\Cathleen Moore-Linn\My Documents\Letterhead Templates\CLASS\CLASS-SOCIAL BEHAVIORAL SCIENCES.psd.jpg  !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~Root Entry FData 1TableWordDocument"8SummaryInformation(DocumentSummaryInformation8CompObjX FMicrosoft Word DocumentNB6WWord.Document.8