ࡱ> #` bjbjmm .8`F}}}8(~<d~ {|("ƀƀƀƀƀƀ  $mhխ0ƀƀ0ƀƀ5Xƀƀ zƀp y1gN}FfK0{`cZc4cpƀfr؁\4ƀƀƀ00AXƀƀƀ{@HD5H  Direct and Indirect Assessment Strategies Mary J. Allen,  HYPERLINK "mailto:mallen@csub.edu" mallen@csub.edu, February 2008 Two Basic Ways to Assess Student Learning: Direct The assessment is based on an analysis of student behaviors or products in which they demonstrate how well they have mastered learning outcomes. Indirect The assessment is based on an analysis of reported perceptions about student mastery of learning outcomes. The perceptions may be self-reports by students, or they may be made by others, such as alumni, fieldwork supervisors, employers, or faculty. Properties of Good Assessment Techniques Validdirectly reflects the learning outcome being assessed Reliableespecially inter-rater reliability when subjective judgments are made Actionableresults help faculty identify what students are learning well and what requires more attention Efficient and cost-effective in time and money Engaging to students and other respondentsso theyll demonstrate the extent of their learning Interesting to faculty and other stakeholdersthey care about results and are willing to act on them Triangulationmultiple lines of evidence point to the same conclusion Strategies for Direct Assessment of Student Learning Published Tests Locally-Developed Tests Embedded Assignments and Course Activities Portfolios Collective Portfolios Examples of Published Tests Some Examples of Published TestsMeasure of Academic Proficiency and Progress (MAPP; replaced the Academic Profile in Jan. 2006) College-level reading, mathematics, writing, and critical thinking in the context of the humanities, social sciences, and natural sciences (can be taken and scored online; essay section is optional)http://www.ets.orgCollegiate Learning Assessment (CLA)critical thinking, analytic reasoning, writing skills; based on open-ended questionshttp://www.cae.org/content/pro_collegiate.htmCollegiate Assessment of Academic Proficiency (CAAP)assesses college students academic achievement in core general education skills (writing, reading, math, science reasoning, and critical thinking) HYPERLINK "http://www.act.org/caap/index.html" http://www.act.org/caap/index.html Steps in Selecting a Published Test Identify a possible test. Consider published reviews of this test, such as reviews in the Mental Measurements Yearbook. Order a specimen set from the publisher. Take the test and consider the appropriateness of its format and content. Consider the tests relationship to your learning outcomes. Consider the depth of processing of the items (e.g., analyze items using Blooms taxonomy). Consider the publication date and currency of the items. How many scores are provided? Will these scores be useful? How? Look at the test manual. Were test development procedures reasonable? What is the evidence for the tests reliability and validity for the intended use? If you will be using the norms, consider their relevance for your purpose. Consider practicalities, e.g., timing, test proctoring, and test scoring requirements. Verify that faculty are willing to act on results. Published Test Strengths and WeaknessesPotential StrengthsPotential WeaknessesCan provide direct evidence of student mastery of learning outcomes. They generally are carefully developed, highly reliable, professionally scored, and nationally normed. They frequently provide a number of norm groups, such as norms for community colleges, liberal arts colleges, and comprehensive universities. Online versions of tests are increasingly available, and some provide immediate scoring. Some publishers allow faculty to supplement tests with their own items, so tests can be adapted to better serve local needs.Students may not take the test seriously if test results have no impact on their lives. These tests are not useful as direct measures for program assessment if they do not align with local curricula and learning outcomes. Test scores may reflect criteria that are too broad for meaningful assessment. Most published tests rely heavily on multiple-choice items which often focus on specific facts, but program learning outcomes more often emphasize higher-level skills. If the test does not reflect the learning outcomes that faculty value and the curricula that students experience, results are likely to be discounted and inconsequential. Tests can be expensive. The marginal gain from annual testing may be low. Faculty may object to standardized exam scores on general principles, leading them to ignore results. Locally-Developed Tests Common Test Item FormatsItem TypeCharacteristics and SuggestionsCompletionThese items require students to fill-in-the-blank with appropriate terms or phrases. They appear to be best for testing vocabulary and basic knowledge, and they avoid giving students credit for guessing by requiring recall, rather than recognition. Scoring can be difficult if more than one answer can be correct. EssayEssay questions are very popular and can be used to assess higher-order thinking skills. They generally ask for explanations and justifications, rather than memorized lists. Key words in essay questions are summarize, evaluate, contrast, explain, describe, define, compare, discuss, criticize, justify, trace, interpret, prove, and illustrate (Moss & Holder, 1988). MatchingUsually these questions are presented as two columns, and students are required to associate elements in column B with elements in column A. Such items are easy to score, but they are relatively difficult to construct and they seem best suited for testing knowledge of factual information, rather than deeper levels of understanding. Multiple-ChoiceMultiple-choice questions are popular because they can measure many concepts in a short period of time, and they generally are better than other objective questions at assessing higher-order thinking. They are easy to score, and item banks associated with popular textbooks are often available. Writing good items takes time, and there is strong temptation to emphasize facts, rather than understanding. True-FalseTrue-false items are relatively easy to construct and grade, but they appear to be best at assessing factual knowledge, rather than deep understanding.  Possible Format for Authentic Essay Questions Give students a task, a role, and an audience. Locally-Developed Test Strengths and WeaknessesPotential StrengthsPotential WeaknessesCan provide direct evidence of student mastery of learning outcomes. Appropriate mixes of essay and objective questions allow faculty to address various types of learning outcomes. Students generally are motivated to display the extent of their learning. If well-constructed, they are likely to have good validity. Because local faculty write the exam, they are likely to be interested in results and willing to use them. Can be integrated into routine faculty workloads. The evaluation process should directly lead faculty into discussions of student learning, curriculum, pedagogy, and student support services.These exams are likely to be less reliable than published exams. Reliability and validity generally are unknown. Creating and scoring exams takes time. Traditional testing methods have been criticized for not being  authentic. Norms generally are not available. Embedded Assignments and Course Activities % Community-service learning and other fieldwork activities % Culminating projects, such as papers in capstone courses % Exams or parts of exams % Group projects % Homework assignments % In-class presentations % Student recitals and exhibitions Assignments and activities are purposefully created to collect information relevant to specific program learning outcomes. Results are pooled across courses and instructors to indicate program accomplishments, not just the learning of students in specific courses. Consider integrating signature assignments into the curriculum, i.e., assignments designed to assess specific learning outcomes. This is embedded assessment. Assignments might be developed as threshold, milestone, or capstone assessments [AAC&U (2005) Liberal Education Outcomes: A Preliminary Report on Student Achievement in College]. Embedded Assignments and Course Activities Strengths and WeaknessesPotential StrengthsPotential WeaknessesCan provide direct evidence of student mastery of learning outcomes. Out-of-class assignments are not restricted to time constraints typical for exams. Students are generally motivated to demonstrate the extent of their learning. Can provide authentic assessment of learning outcomes. Can involve CSL or other fieldwork activities and ratings by fieldwork supervisors. Can provide a context for assessing communication and teamwork skills. Can be used for grading as well as assessment. Faculty who develop the procedures are likely to be interested in results and willing to use them. The evaluation process should directly lead faculty into discussions of student learning, curriculum, pedagogy, and student support services. Data collection is unobtrusive to students.Requires time to develop and coordinate. Requires faculty trust that the program will be assessed, not individual teachers. Reliability and validity generally are unknown. Norms generally are not available. Portfolios Showcase vs. Developmental Portfolios: best work vs. evidence of growth Workload and storage demands for large programs can be overwhelming! Some Questions to Answer Before Assigning Portfolios What is the purpose of the requirementto document student learning, to demonstrate student development, to learn about students reflections on their learning, to create a document useful to students, to help students grow through personal reflection on their personal goals? When and how will students be told about the requirement, including what materials they need to collect or to produce for it? Will the portfolios be used developmentally or will they be submitted only as students near graduation? Will portfolios be showcase or developmental? Are there minimum and maximum lengths or sizes for portfolios? Who will decide which materials will be included in portfoliosfaculty or students? What elements will be required in the portfolioevidence only from courses in the discipline, other types of evidence, evidence directly tied to learning outcomes, previously graded products or clean copies? Will students be graded on the portfolios? If so, how and by whom? How will the portfolios be assessed to evaluate and improve the program? What can be done for students who have inadequate evidence through no fault of their own? What will motivate students to take the portfolio assignment seriously? How will the portfolio be submittedhard copy or electronic copy? Who owns the portfoliosstudents or the program? Who has access to the portfolios and for what purposes? How will student privacy and confidentiality be protected? Portfolio Strengths and WeaknessesPotential StrengthsPotential WeaknessesCan provide direct evidence of student mastery of learning outcomes. Students are encouraged to take responsibility for and pride in their learning. Students may become more aware of their own academic growth. Can be used for developmental assessment and can be integrated into the advising process to individualize student planning. Can help faculty identify curriculum gaps, lack of alignment with outcomes. Students can use portfolios and the portfolio process to prepare for graduate school or career applications. The evaluation process should directly lead faculty into discussions of student learning, curriculum, pedagogy, and student support services. E-portfolios or CD-ROMs can be easily viewed, duplicated, and stored.Requires faculty time to prepare the portfolio assignment and assist students as they prepare them. Requires faculty analysis and, if graded, faculty time to assign grades. May be difficult to motivate students to take the task seriously. May be more difficult for transfer students to assemble the portfolio if they havent saved relevant materials. Students may refrain from criticizing the program if their portfolio is graded or if their names will be associated with portfolios during the review. Collective Portfolios Some of the benefits of traditional portfolios, with much less work! Collective Portfolio Strengths and WeaknessesPotential StrengthsPotential WeaknessesCan provide direct evidence of student mastery of learning outcomes. Students generally are motivated to display the extent of their learning. Workload demands generally are more manageable than traditional portfolios. Can help faculty identify curriculum gaps, lack of alignment with outcomes. Students are not required to do extra work. The evaluation process should directly lead faculty into discussions of student learning, curriculum, pedagogy, and student support services. Data collection is unobtrusive to students.If assignments are not aligned with the outcomes being examined, evidence may be problematic. If sampling is not done well, results may not generalize to the entire program. Reviewing the materials takes time and planning. Strategies for Indirect Assessment of Student Learning Surveys Interviews Focus Groups Surveys Point-of-contact surveys Online, e-mailed, registration, or grad check surveys Keep it simple! Common Survey FormatsType of ItemExampleCheck listPlease indicate which of the activities you feel competent to perform. __ Develop an investment plan __ Interpret a financial report __ Provide feedback about an employees performance __ Write a case study ClassificationOrganization of the paper: _____ Confusing, unclear _____ Generally clear, minor points of confusion _____ Clear, logical, easy to follow FrequencyIn a typical term, I used the departments computer lab: Never 1-2 times 3-5 times 6 or more times ImportanceHow important is it for the department to provide career counseling? Unimportant Slightly Moderately Very Extremely Important Important Important Important LikelihoodHow likely are you to apply to a graduate program in the next five years? Very Unlikely Slightly Unlikely Uncertain Slightly Likely Very Likely Linear rating scaleAbility to compose paragraphs in standard, written English. Unsatisfactory ____ | ____ | ____ | ____ | ____ | ____ | _____ Excellent Likert scaleI am able to write a research paper using MLA standards. Strongly Disagree Disagree Neutral Agree Strongly Agree Open-ended Please describe the most important concepts you learned in the program. Partially close-endedPlease check the most important factor that led you to major in engineering. ___ Experience in a specific course ___ Experience with a specific instructor ___ Work experience in this or a related field ___ Advice from a career planning office or consultant ___ Advice from family member or friend ___ Personal interest ___ Other: please explain QualityPlease indicate the quality of instruction in the general education program. Very Poor Poor Good Very Good Excellent RankingPlease indicate your ranking of the importance of the following learning outcomes by assigning ranks from 1 to 4, where a 1 is most important and 4 is least important. ___ Computing ___ Critical thinking ___ Speaking ___ Writing  Gap Analysis Sometimes it is useful to ask respondents to rate a set of items twice: once to indicate their importance and once to indicate the extent of their achievement. Differences (gaps) between the two ratings receive particular attention when interpreting results, especially items that are judged to be important and not well achieved. Survey Strengths and WeaknessesPotential StrengthsPotential Weaknesses% Are flexible in format and can include questions about many issues. % Can be administered to large groups of respondents. % Can easily assess the views of various stakeholders. % Usually has face validity the questions generally have a clear relationship to the outcomes being assessed. % Tend to be inexpensive to administer. % Can be conducted relatively quickly. % Responses to close-ended questions are easy to tabulate and to report in tables or graphs. % Open-ended questions allow faculty to uncover unanticipated results. % Can be used to track opinions across time to explore trends. % Are amenable to different formats, such as paper-and-pencil or online formats. % Can be used to collect opinions from respondents at distant sites.% Provides indirect evidence about student learning. % Their validity depends on the quality of the questions and response options. % Conclusions can be inaccurate if biased samples are obtained. % Results might not include the full array of opinions if the sample is small. % What people say they do or know may be inconsistent with what they actually do or know. % Open-ended responses can be difficult and time-consuming to analyze.  Interviews Interviews can be conducted one-on-one, in small groups, or over the phone. Interviews can be structured (with specified questions) or unstructured (a more open process). Questions can be close-ended (e.g., multiple-choice style) or open-ended (respondents construct a response). Can target students, graduating seniors, alumni, employers, community members, faculty, etc. Can do exit interviews or pre-post interviews. Can focus on student experiences, concerns, or attitudes related to the program being assessed. Generally should be conducted by neutral parties to avoid bias and conflict of interest. Some Tips for Effective Interviewing% Conduct the interview in an environment that allows the interaction to be confidential and uninterrupted. % Demonstrate respect for the respondents as participants in the assessment process rather than as subjects. Explain the purpose of the project, how the data will be used, how the respondent s anonymity or confidentiality will be maintained, and the respondents rights as participants. Ask if they have any questions. % Put the respondents at ease. Do more listening than talking. Allow respondents to finish their statements without interruption. % Match follow-up questions to the project s objectives. For example, if the objective is to obtain student feedback about student advising, don t spend time pursuing other topics. % Do not argue with the respondent s point of view, even if you are convinced that the viewpoint is incorrect. Your role is to obtain the respondents opinions, not to convert them to your perspective. % Allow respondents time to process the question. They may not have thought about the issue before, and they may require time to develop a thoughtful response. % Paraphrase to verify that you have understood the respondent s comments. Respondents will sometimes realize that what they said isn t what they meant, or you may have misunderstood them. Paraphrasing provides an opportunity to improve the accuracy of the data. % Make sure you know how to record the data and include a backup system. You may be using a tape recorder if so, consider supplementing the tape with written notes in case the recorder fails or the tape is faulty. Always build in a system for verifying that the tape is functioning or that other data recording procedures are working. Dont forget your pencil and paper! Interview Strengths and WeaknessesPotential StrengthsPotential Weaknesses% Are flexible in format and can include questions about many issues. % Can assess the views of various stakeholders. % Usually has face validity the questions generally have a clear relationship to the outcomes being assessed. % Can provide insights into the reasons for participants beliefs, attitudes, and experiences. % Interviewers can prompt respondents to provide more detailed responses. % Interviewers can respond to questions and clarify misunderstandings. % Telephone interviews can be used to reach distant respondents. % Can provide a sense of immediacy and personal attention for respondents. % Open-ended questions allow faculty to uncover unanticipated results.% Generally provides indirect evidence about student learning.  +,;<`apqs I Q M O y  n T U V W X }~IJz|jh)OJQJUh`h)OJQJh\h+_h)5CJ\h)5OJQJ\h)OJQJ h)5\h)hT n hhjhUhh'QZ5CJ$aJ$h)h)5CJ$aJ$ h1$h)h1$h)OJQJ0,I M N O P y z  o  & F hh^hgd)gd) & F ^`gd)$&d P ]a$gdgd)gd)&d P ]gd). b  3 > T U W X t u $Ifgd<gd) & F hph^hgd)gd)gd)$a$gd)&d P ]gd) & F hh^hgd)  $Ifgd<dkd$$Ifl4(##0#4 laf4yt)O}wnnn $Ifgd<kd$$IflF/((# 0#    4 layt)}~Iwnnn $Ifgd<kd6$$IflF/((# 0#    4 layt)z{|">2346]_mn01bc*F !!E!f%h%j%%''ظحآآ؝ h)CJh h)>* h)>*h)6OJQJ]h)5OJQJ\h)5CJ\ h)5\ h)6]h)5CJ\h)OJQJh)jh)OJQJUjh)OJQJU7@wf^YTAA & F h8h^hgd)gd)gd)$a$gd)&d P ]gd)kd$$IflF/((# 0#    4 layt)@iK]2346^$$If`a$gd< 8^gd)$a$gd)&d P ]gd) & F h8h ^h` gd) & F h8h^hgd)^_s$$If`a$gd<fkd$$Ifl4#H$0H$64 laf4yt)5wn1ppppppppppp & F $If^`gd)vkd$$Ifl0#L0H$64 layt) 1cp_WRMgd)gd)$a$gd)&d P ]gd)vkd$$Ifl0#L0H$64 layt) & F $If^`gd) )fkdT$$Ifl4,"h%064 laf4yt< $$Ifa$gd<)*5pxo $Ifgd<  !$Ifgd<ykd$$Ifl40,"F" 064 laf4yt<pqwxo $Ifgd<  !$Ifgd<ykd$$Ifl40,"F" 064 laf4yt<?xo $Ifgd<  !$Ifgd<ykd!$$Ifl40,"F" 064 laf4yt<?@Pxo $Ifgd<  !$Ifgd<ykd$$Ifl40,"F" 064 laf4yt< xo $Ifgd<  !$Ifgd<ykdc$$Ifl40,"F" 064 laf4yt< !yi$$If`a$gd<$a$gd)gd)ykd $$Ifl40,"F" 064 laf4yt<!!/!D!$$If`a$gd<fkd $$Ifl4%t%0t%64 laf4yt<D!E!!!E"""###:$$ %ppppppppppp & F $If^`gd)vkd0 $$Ifl0p% 0t%64 layt< %f%h%j%l%%%<&p_WRJA gd)dgd)gd)$a$gd)&d P ]gd)vkd $$Ifl0p% 0t%64 layt< & F $If^`gd)<&&&'@'t'''((?*@*A*B**$$If`a$gd<gd)&d P ]gd)gd)dgd)  !gd) gd)'(());*?*@*A*B****....."/#/$/X/4445!5K577: :6:~::::==== >!>">*>>>> ?|?}???.@/@ƴƮեեhSRh)CJaJ h)CJhSRh)CJaJh)5CJ\ h)CJhLOJQJh)CJ OJQJh)OJQJ h)5\ h)CJh)CJOJQJh[h)6h7ph)5h) h>)6****$$$<fkdf $$Ifl4$$0$64 laf4yt<***I+++#,j,,,---ppppppppppp & F $If^`gd)vkd $$Ifl0$ 0$64 layt< -2.b.......p__ZM &dPgd)gd)&d P ]gd)vkd $$Ifl0$ 0$64 layt< & F $If^`gd)..."/#/$/Y/n00T11122(3q334U444444 & F hh^hgd)&d P ]gd) & Fgd)gd)4 5!555J5akd' $$Ifl4$$0$64 laf4yt<$$If`a$gd<J5K555666R77&88899uuuuuuuuuuuu & F $If^`gd)qkd $$Ifl0H$H0$64 layt< 9::: :6:7:|:~:ud__ZRZ$a$gd)gd)gd)&d P ]gd)qkdM$$Ifl0H$H0$64 layt< & F $If^`gd)~:::::akd$$Ifl4$$0$64 laf4yt<$$If`a$gd<::;f;;;*<<<B===uuuuuuuuuu & F $If^`gd)qkds$$Ifl0H$H0$64 layt< ======>> >!>">*>}xsxggg}xbgd) & FP `P gd)gd)gd)&d P ]gd)qkd$$Ifl0H$H0$64 layt< *>+>D>z>>>>>>]fkd$$Ifl4%t%0t%64 laf4yt<$ !h$If^ha$gd<  8^8`gd) & F  8`8gd)gd)>>>>> ?)?I?}??aPPPPPP$If^`gd<vkdB$$Ifl0%0t%64 layt<$$If`a$gd<$ $If^`a$gd< ?????? @.@/@xcccc $If^`gd<vkd$$Ifl0%0t%64 layt<$If^`gd</@0@:@s@@@xxxx$If^`gd<vkdx$$Ifl0%0t%64 layt</@s@@@@"A#AAAA2B3BBBBBCC C!C(C)C*CrCCCCDD}DEEEEFFFF G G>?ʳ>ıII崳JJKrMtMPPPP2Q4QQQRRSSSS h)o(h)PJ^J h)5\h)CJOJQJh`h)5h)CJPJ^J h)CJ h)CJhSRh)CJaJ h)CJh)E@@@"A{AAAxxxxx$If^`gd<vkd$$Ifl0%0t%64 layt<AAA2B3BBBxxxxx$If^`gd<vkd$$Ifl0%0t%64 layt<BBBBBBBBCCCCCxxjjjjjjjjj $$Ifa$gd<K$$If^`gd<vkdI$$Ifl0%0t%64 layt< C C!C"C#C$C%C&C'C(C)C*C+C$If^`gd<Ff0 $$Ifa$gd<K$Ffi +C,C9CrCCCxxxx$If^`gd<vkdr$$Ifl0%0t%64 layt<CCCDDDDDxxxxxx$If^`gd<vkd $$Ifl0%0t%64 layt<DD0D}DDDD1EYEoEEExxxxxxxxxx$If^`gd<vkd$$Ifl0%0t%64 layt< EEEEEFFFxxxx$If^`gd<vkdC$$Ifl0%0t%64 layt<FFGFOFF G$G1G>?$<vkd$$Ifl0%0t%64 layt<?G@GAGBGCGPG8I:IKLjLLrMN~N OO}}}}}}}}}}} $If^` gd<qkd $$Ifl0%L(0t%64 layt< OPP4QQRSSSShW&d P ]gd)qkd:!$$Ifl0%L(0t%64 layt< $If^`gd<$If^`gd< SS2S4SSETTU?UUUUUUBV$$If`a$gd<gd)&d P ]gd) & F hhd]^hgd)gd)gd)$a$gd)SS2S4SUUUUBVvWWWW\\^^``bbbbccdeefffgggg*h.hhhLiiitvz@BF(*.VTh)CJOJQJUh)PJ^J h)6] h)CJ h)CJh)OJQJ h)5\h)h)5CJ\JBVDVWYZ \]^`b$If^`gd<^kd!$$Ifl,"h%064 layt< bbbc$$If`a$gd<`gd)^kdZ"$$Ifl,"h%064 layt<cc%c:c$$If`a$gd<fkd"$$Ifl4$$0$64 laf4yt<:c;cddefgg*hhNiiv||||||||||||| $If`gd<vkdj#$$Ifl0h$0$64 layt<% Their validity depends on the quality of the questions. % Poor interviewer skills can generate limited or useless information. % Can be difficult to obtain a representative sample of respondents. % What people say they do or know may be inconsistent with what they actually do or know. % Can be relatively time-consuming and expensive to conduct, especially if interviewers and interviewees are paid or if the no-show rate for scheduled interviews is high. % The process can intimidate some respondents, especially if asked about sensitive information and their identity is known to the interviewer. % Results can be difficult and time-consuming to analyze. % Transcriptions of interviews can be time-consuming and costly. Focus Groups Traditional focus groups are free-flowing discussions among small, homogeneous groups (typically from 6 to 10 participants), guided by a skilled facilitator who subtly directs the discussion in accordance with pre-determined objectives. This process leads to in-depth responses to questions, generally with full participation from all group members. The facilitator departs from the script to follow promising leads that arise during the interaction. Structured group interviews are less interactive than traditional focus groups and can be facilitated by people with less training in group dynamics and traditional focus group methodology. The group interview is highly structured, and the report generally provides a few core findings, rather than an in-depth analysis. Sample Focus Group QuestionsPurpose of QuestionExamplesWarm-up% I d like everyone to start out by stating a word or phrase that best describes your view of the program.Issue 1: Career Preparation% Please tell us what career you are interested in pursuing after graduation. % How has the program helped you prepare for your career or future activities?Issue 2: Advising% We are interested in your advising experiences in the program. Could you tell us about your first advising experience in the department? % What did you find most useful in your interactions with your advisor? % What would you like our advisors to do differently?Issue 3: Curriculum% Thinking about the curriculum and the required courses, how well do you think they prepared you for upper-division work? % What should be changed about the curriculum to better prepare you for your career or for graduate school?Closing% We ve covered a lot of ground today, but we know you might still have other input about the program. Is there anything you would like to say about the program that hasn t been discussed already? Focus Group Strengths and WeaknessesPotential StrengthsPotential Weaknesses% Are flexible in format and can include questions about many issues. % Can provide in-depth exploration of issues. % Usually has face validity the questions generally have a clear relationship to the outcomes being assessed. % Can be combined with other techniques, such as surveys. % The process allows faculty to uncover unanticipated results. % Can provide insights into the reasons for participants beliefs, attitudes, and experiences. % Can be conducted within courses. % Participants have the opportunity to react to each other s ideas, providing an opportunity to uncover the degree of consensus on ideas that emerge during the discussion. % Generally provides indirect evidence about student learning. % Requires a skilled, unbiased facilitator. % Their validity depends on the quality of the questions. % Results might not include the full array of opinions if only one focus group is conducted. % What people say they do or know may be inconsistent with what they actually do or know. % Recruiting and scheduling the groups can be difficult. % Time-consuming to collect and analyze data.  References Allen, M. J. (2004) Assessing Academic Programs in Higher Education. Bolton, MA: Sage. Allen, M. J. (2006). Assessing General Education Programs. Bolton, MA: Sage. Krueger, R.A., & Casey M.A. (2000). Focus Groups: A Practical Guide for Applied Research (3rd ed.). Thousand Oaks, CA: Sage.     PAGE  Direct and Indirect Assessment - PAGE 1 B*kZUgd)&d P ]gd)vkd$$$Ifl0h$0$64 layt< $If`gd< $If`gd<VIfkd$$$Ifl4,""064 laf4yt)$$If`a$gd<&d P ]gd)$ & F hh&dP^ha$gd)gd) &dPgd)@RTd:eTG $If^gd<$If^`gd<vkd+%$$Ifl0w,"064 layt)$$If`a$gd<$$If^`a$gd<:<txkk $If^gd<$If^`gd<vkd%$$Ifl0w,"064 layt)|xkkk $If^gd<$If^`gd<vkda&$$Ifl0w,"064 layt)xkk $If^gd<$If^`gd<vkd&$$Ifl0w,"064 layt)|xk $If^gd<$If^`gd<vkd'$$Ifl0w,"064 layt)|~t$$If`a$gd<gd)vkd2($$Ifl0w,"064 layt) $$If`a$gd<akd($$Ifl4$$0$64 laf4yt<"& XZ\`< >dh jDF|,.046:<@BFH̸̰̩jh+_0JUh_jh_U hEhahEhaH*hEha6hLhaha6hahahE5haha5hEh)5CJ\h)PJ^J h)5\h)7 " \8>d $Ifgd<qkdX)$$Ifl0$L80$64 layt<d24{vvvvvvqvgdagdEgd)gd)qkd)$$Ifl0$L80$64 layt< $Ifgd< 4.248:>@DFXZ\ &`#$gd<gd)gdEHTVX\ hEhah_h_h[0JmHnHu h_h[0J h g0Jh+_jh+_0JU h+_0J,1h/ =!"#$% $$If!vh5##v#:V l40#5#4f4yt)$$If!vh55 5#v#v #v:V l0#55 54yt)$$If!vh55 5#v#v #v:V l0#55 54yt)DyK #http://www.act.org/caap/index.htmlyK Fhttp://www.act.org/caap/index.html$$If!vh55 5#v#v #v:V l0#55 54yt)$$If!vh5H$#vH$:V l40H$65H$4f4yt)$$If!vh5L5#vL#v:V l0H$65L54yt)$$If!vh5L5#vL#v:V l0H$65L54yt)$$If!vh5"#v":V l4065h%4f4yt<$$If!vh5F5R#vF#vR:V l4065F5" 4f4yt<$$If!vh5F5R#vF#vR:V l4065F5" 4f4yt<$$If!vh5F5R#vF#vR:V l4065F5" 4f4yt<$$If!vh5F5R#vF#vR:V l4065F5" 4f4yt<$$If!vh5F5R#vF#vR:V l4065F5" 4f4yt<$$If!vh5F5R#vF#vR:V l4065F5" 4f4yt<$$If!vh5t%#vt%:V l40t%65t%4f4yt<$$If!vh55 #v#v :V l0t%655 4yt<$$If!vh55 #v#v :V l0t%655 4yt<$$If!vh5$#v$:V l40$65$4f4yt<$$If!vh5 5#v #v:V l0$65 54yt<$$If!vh5 5#v #v:V l0$65 54yt<$$If!vh5$#v$:V l40$65$4f4yt<$$If!vh55H#v#vH:V l0$655H4yt<$$If!vh55H#v#vH:V l0$655H4yt<$$If!vh5$#v$:V l40$65$4f4yt<$$If!vh55H#v#vH:V l0$655H4yt<$$If!vh55H#v#vH:V l0$655H4yt<$$If!vh5t%#vt%:V l40t%65t%/ 4f4yt<$$If!vh55#v#v:V l0t%6554yt<$$If!vh55#v#v:V l0t%6554yt<$$If!vh55#v#v:V l0t%6554yt<$$If!vh55#v#v:V l0t%6554yt<$$If!vh55#v#v:V l0t%6554yt<$$If!vh55#v#v:V l0t%6554yt<$IfK$L$!v h5@55H5H55555 #v@#v#vH#v#v :V h6, 5@55H55 9 / 2 22 22 22 22 22 22 22 22 2 4 yt<@kd$IfK$L$h @X p@'@''H'H'''''6$$$$2 22 22 22 22 22 22 22 22 2 4ayt<$IfK$L$!v h5@55H5H55555 #v@#v#vH#v#v :V h6, 5@55H55 9 / 2 22 22 22 22 22 22 22 22 2 4 yt<@kd$IfK$L$h @X p@'@''H'H'''''6$$$$2 22 22 22 22 22 22 22 22 2 4ayt<$$If!vh55#v#v:V l0t%6554yt<$$If!vh55#v#v:V l0t%6554yt<$$If!vh55#v#v:V l0t%6554yt<$$If!vh55#v#v:V l0t%6554yt<$$If!vh55#v#v:V l0t%6554yt<$$If!vh55#v#v:V l0t%6554yt<$$If!vh5t%#vt%:V l40t%65t%4f4yt<$$If!vh5L5(#vL#v(:V l0t%65L5(4yt<$$If!vh5L5(#vL#v(:V l0t%65L5(4yt<$$If!vh5"#v":V l065h%4yt<$$If!vh5"#v":V l065h%4yt<$$If!vh5$#v$:V l40$65$4f4yt<$$If!vh55#v#v:V l0$6554yt<$$If!vh55#v#v:V l0$6554yt<$$If!vh5"#v":V l4065"4f4yt)$$If!vh55#v#v:V l06554yt)$$If!vh55#v#v:V l06554yt)$$If!vh55#v#v:V l06554yt)$$If!vh55#v#v:V l06554yt)$$If!vh55#v#v:V l06554yt)$$If!vh55#v#v:V l06554yt)$$If!vh5$#v$:V l40$65$4f4yt<$$If!vh5L58#vL#v8:V l0$65L584yt<$$If!vh5L58#vL#v8:V l0$65L584yt<@@@ )NormalCJ_HaJmH sH tH >@> ) Heading 1$$@&a$CJ`Z@Z ) Heading 3'$$&d @&P ]a$5\@@@ ) Heading 4$$@&a$5\T@T ) Heading 5!$&d @&P ]5\>@> ) Heading 8$$@&a$CJDA@D Default Paragraph FontRi@R  Table Normal4 l4a (k(No List8B@8 ) Body Text$a$CJ84 @4 )Footer  !4@4 )Header  !LC@"L )Body Text Indent^`XR@2X )Body Text Indent 2 ^`HS@BH )Body Text Indent 3 h`h6U@Q6 ) Hyperlink >*B*ph.)@a. ) Page NumberHrH L Balloon TextCJOJQJ^JaJZY@Z  g Document Map-D M CJOJQJ^JaJ`,IMNOPyzob3>TUWXtuO}~I@ i K ] 2 3 4 6 ^ _ s 5 wn1c )*5pqw?@P/DEEDZu? @ A B I!!!#"j"""###2$b$$$$$$$$$"%#%$%Y%n&&T'''((()q))*U****** +!+5+J+K+++,,,R--&...//000 06070|0~0000001f111*222B3333333344 4!4"4*4+4D4z4444444444 5)5I5}5555555 6.6/606:6s66666"7{77777283888888888899999 9!9"9#9$9%9&9'9(9)9*9+9,999r99999::::::0:}::::1;Y;o;;;;;;E<F<G<O<< =$=1=>=?=@=A=B=C=P=>>>>>>>>>>2?h?? @5@\@@A?AAA BZBBBCCCCCCCCCCEDDE?EEEEEE!F"FFGPHIIpJxKLLLMM%M:M;MMMN~NNOPOOO!P\PPPBQQ|RRRRRRSST VVV,V-VAVJVKVSVVVV)WxWyWWX_XXXX%YYYY_Z`ZaZbZZZZZZZ&[[[\m\\=]|]]]@^^^_________k_l___8`:`;`=`>`@`A`C`D`M`N`O`{`|`}``00000000 0 000000P 0P 0P 0P 0P 0P 0P 0P0P000 0 0 0 0 000080W0X0X 0X 0X0X 0X 0X 0X 0X 0X 0X 0X 0X 0X 0X 0X 0X000 0 0 0 0 0 0 0 0 0 0  0  0 0003 03  03  03  03  03   03  03  03  03  03   03  03  03  03  0 3  0 3  0 3  0 3  03  03 0000 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 000800 0 0 0 0  0  0 0 0 0 0 0  0 0 0 0 0 0 000000000000000000@ 0@  0@  0@  0@  0@   0@  0@  0@  0@  0@  0@  0@  0 @  0!@  0"@   0#@  0$@  0%@  0&@  0@  0@ 0@ 0@ (0@ 0$ 0$ 0$0$0$0$ 0$ 0$ 0$ 0$ 0$ 0$ 0$ 0$ 0$ 0 $ 0 $ 0 $ 0 $ 0 $ 0$0$0$0$ 0$ 0$ 0$ 0$  0'$ 0($ 0)$ 0*$ 0+$ 0,$ 0-$ 0.$  0/$ 00$ 01$ 02$ 03$ 0$ 0$000 00 00 00 0 0 0 0 0 0 0 0 0  04 0 05 0 06 0 07 0 08 0 09 0 0: 0  0; 0 0< 0 0= 0 0 0 0 00 0003 03 03 030303x030"4 0"4 0"4 0"40"40"40"4 0"4 0"4 0"4 0"4 0"4 0"40"40"40"40"40"4 0"4 0"4 0"40"40"40"40"4 0"4 0"4 0"40"40"4 0"4 0"4 0"40"40"40"4 0"4 0"4 0"40"40"40"4 0"4 0"4 0"40"4 0"4 0"4 0"4 0"4 0"4 0"4 0"4 0"4 0"4 0"4 0"4 0"4 0"4 0"4 0"4 0"4 0"4 0"4 0"4 0"4 0"4 0"4 0"40"40"4 0"4 0"4 0"40"40"40"40"4 0"4 0"4 0"40"40"40"40"40"40"40"40"4 0"4 0"4 0"40"40"4 0"4 0"4 0"40"40"40"40"40"4 0"4 0"40"40"40"40"40"40"40"40"40"4 0"4 0"4 0"4 0"4 0"40"40"40"40"40"40"40"40"40"40"4 0"40"40"40"40"40"40"4 0"4 0"4080C0C 0C 0C 0C 0C 0C 0C 0C0C0C0C0C 0C 0C0C0C0C0C0C0C0C 0C 0C0C 0C 0C 0C 0C 0C0C0C0C0C0C0C0C0C 0C0C0C0C0C0C0C0C0C 0C 0C0C(0C0R( 0C( 0C0T0T0T 0T 0T 0T 0T 0T 0T 0T 0T 0T0T 0T 0T 0T0T0T 0T 0T 0T0T 0T 0T 0T 0T 0T0T0T 0T 0T 0T 0T 0T0T0T0T0T0T0T0T 0T0T0T0T0T0T0T0T 0T 0TH0T0_0_0_0_0_0_0_0_0_@000@000@000@000@0@0@0@0@0@000,IMNOyzob3>TUWXtuI@ i K ] 2 3 4 6 ^ _ s 5 wn1c )*5pqw?@P/DEEDZu@ A B I!!!#"j"""###2$b$$$$$$$$$"%#%$%Y%n&&T'''((()q))*U****** +!+5+J+K+++,,,R--&...//000 06070|0~0000001f111*222B3333333344 4!4"4*4+4D4z44444455/606:6s666677889 9+9,999:::1;Y;o;;;;;;E<F<G< =$=1=?=@=B=C=P=>>>>>>>>>>2?h?? @5@\@@A?AAA BZBBBCCCCCCCCCCEDDE?EEEEEE!F"FFGPHIIpJxKLLLMM%M:M;MMMN~NNOPOOO!P\PPPBQQ|RRRRRRSST VVV,V-VAVJVKVSVVVV)WxWyWWX_XXXX%YYYY_Z`ZaZbZZZZZZZ&[[[\m\\=]|]]]@^^^____`@01L@01L00@01L@01L@01L@01L@ 01L@ 01L@01L@01L @0@03V@ 03V@ 03V@ 03V@ 03V@ 03V@ 03V@ 03V@03V @0 @0@0nX@ 0nX@ 0nX@ 0nX@ 0nX@ 0nX@0nX@0nX @0:@0Y@0Y@0Y@0YK02f@0Y@0Y@0Y@0Y@0Y@0Y@0Y@0Y@0Y @0 @0@0\@ 0\@ 0\@ 0\@ 0\@ 0\@ 0\@ 0\@ 0\@ 0\@ 0 \@ 0 \@ 0 \@0\ @0@0/`@0/`@0/`@0/`@0/`@0/`@ 0/`@ 0/`@ 0/`@ 0/`@ 0/`@ 0/`@ 0/`@ 0/`@ 0/`@ 0 /`@ 0 /`@ 0 /`@ 0 /`@0/`@0/` @0 @0@0e@0e@0e@0e@0e@0e@0e@0e@0e@0e@0e@0e@0e@0e@0e@0e@0e@0e@0e@0e@0e:@0e@0l@0l@0l@0l@0l@ 0 l@ 0l@ 0l@ 0l@ 0l@ 0l@ 0l@ 0l@ 0l@ 0l@ 0l@ 0l@0l@0l @0 @0@0Wp@0Wp@0Wp@0Wp@0Wp@0Wp@0Wp@0Wp@0Wp@0Wp @0@0s@0s@0s@0s@0s@0s@ 0s@ 0s@ 0s@ 0s@ 0s@ 0s@ 0s@ 0 s@ 0!s@ 0"s@ 0#s@ 0$s@ 0%s@ 0&s@0s@0w@0w@0w*@0w@0w@ 0w@ 0w@0w@0w@0w@ 0w@ 0w@ 0w@ 0w@ 0w@ 0w@ 0w@ 0w@ 0w@ 0 w@ 0 w@ 0 w@ 0 w@ 0 w@ 0w@0w@0w@0w@0w@0w@0w@0w@ 0'w@ 0(w@ 0)w@ 0*w@ 0+w@ 0,w@ 0-w@ 0.w@ 0/w@ 00w@ 01w@ 02w@ 03w@0w@0w @0 @0@0@0@0@0@0@0@0@0@ 04@ 05@ 06@ 07@ 08@ 09@ 0:@ 0;@ 0<@ 0=@0@0@0 @0@0-@ 0-@ 0-@ 0-@0-@0-z@0-@0@ 0@ 0@ 0K00G@0 K00G@#0$9:K00G@0K00G@0gdK0\1sb@0K00K00 Gb@0NK0 0 GA0K0 0 GpK00G@0@ K00G@0@K00G@0(K00G@0`K0m1hnK0m1gI0m1e01@0I00 @0 K0l1`@0K00K00GL0 K0l1[mK0l1ZK0l1YK00 @0 01'0101@0@0@0@0@0@0@0@0@0@0@0@0@0@0@0@0@0@0@0@0@0@0@0@0@0@0@0@0@0@0 @0:@0@0@ 0@ 0@ 0@ 0@ 0@ 0@ 0@0@0@0@0@0@0@0@0@0@0@0@0@0@0@0@0@0@0@0@0@0@0@0@0@0@0@0@0@0@0@0@0@0@0@0@0@0@0@0@0@0*@0@0P*@ 0*@ 0@0"@0"@0h@0h@0h@0h@0h@0h@0h@0h@0h@0h@0h@0h@0h@0h@0h@0h@0h@0h@0h@0h@0h@0h@0h@0h@0h@0h@0h@0h@0h@0h@0h@0h@0h@0h@0h@0h@0h@0h@0h@0h@0h@0h@0h@0h@0h@0h@0h@0h@0h0 EEEHz'/@SH5<M\k }@^1)p? !D! %<&**-.4J59~::=*>>?/@@ABC+CCDEFF?GBIIOSBVbc:c:| d4689:;=>?@ABCDEFGHIJKLNOPQRSTUVWXYZ[]^_`abcdefghijlmno7;`pI{`XX 8?AH!! K: K:l K:, K: K: K:l K:, K: K: K:l K:, X_X_`____``.``  ^_b_b____,`0`0`` 9*urn:schemas-microsoft-com:office:smarttagsState8*urn:schemas-microsoft-com:office:smarttagsCity9 *urn:schemas-microsoft-com:office:smarttagsplace; *urn:schemas-microsoft-com:office:smarttagsaddress: *urn:schemas-microsoft-com:office:smarttagsStreet <   8`8`:`:`;`;`=`>`@`A`C`D`}``4    ""=%C%II____``7`8`8`:`:`;`;`=`>`@`A`C`D`O`p`}``33333333n**--|0}0:@=A=A=>>>>CCCCEELLRR V VVV_7`8`8`:`:`;`;`=`>`@`A`C`D`L`O`p`z``8`8`:`:`;`;`=`>`@`A`C`D`}`` yP* ) N$!%,4H䘺dJx<.OjWfhZddԨ[eyhh^h`B*CJOJQJo(ph ^`OJQJo(o pp^p`OJQJo( @ @ ^@ `OJQJo( ^`OJQJo(o ^`OJQJo( ^`OJQJo( ^`OJQJo(o PP^P`OJQJo(h ^`OJQJo(h ^`OJQJo(oh pp^p`OJQJo(h @ @ ^@ `OJQJo(h ^`OJQJo(oh ^`OJQJo(h ^`OJQJo(h ^`OJQJo(oh PP^P`OJQJo(h^`.h^`.hpLp^p`L.h@ @ ^@ `.h^`.hL^`L.h^`.h^`.hPLP^P`L.h ^`o(hH.h ^`hH.h pLp^p`LhH.h @ @ ^@ `hH.h ^`hH.h L^`LhH.h ^`hH.h ^`hH.h PLP^P`LhH.h ^`OJQJo(h ^`OJQJo(oh pp^p`OJQJo(h @ @ ^@ `OJQJo(h ^`OJQJo(oh ^`OJQJo(h ^`OJQJo(h ^`OJQJo(oh PP^P`OJQJo(h 88^8`OJQJo(h ^`OJQJo(oh   ^ `OJQJo(h   ^ `OJQJo(h xx^x`OJQJo(oh HH^H`OJQJo(h ^`OJQJo(h ^`OJQJo(oh ^`OJQJo(^`o(.^`.pLp^p`L.@ @ ^@ `.^`.L^`L.^`.^`.PLP^P`L.^`o(.^`.pLp^p`L.@ @ ^@ `.^`.L^`L.^`.^`.PLP^P`L.hh^h`B*CJOJQJo(ph ^`OJQJo(o pp^p`OJQJo( @ @ ^@ `OJQJo( ^`OJQJo(o ^`OJQJo( ^`OJQJo( ^`OJQJo(o PP^P`OJQJo(h ^`OJQJo(h ^`OJQJo(oh pp^p`OJQJo(h @ @ ^@ `OJQJo(h ^`OJQJo(oh ^`OJQJo(h ^`OJQJo(h ^`OJQJo(oh PP^P`OJQJo( W[ey.O*  yZdd4HdJ%, \                                                                       \                 E$ +_Z g_$)HRQ'QZ_h[_mT npq<E\z:BLa_uO}~I6 ^ _ s  )*5pqw?@P/DE@ A B #$$** +!+5+J+K+&.00 0~000000233 4"44444444555/606:66667778888888899999 9!9"9#9$9%9&9'9(9)9*9+9,999999::0:;;;F<G<O<?=@=>>>>>>>>>ACCEE!F"FLLLMM%M:M;MORRVV,V-VAVJVKVSVVVVxWyWWXXXYYY_Z`ZbZZZZZZ=]___7``9oym0ym0@|1>?@BCDFGHIJKL;M;N;O!P`@@@$@&@P@@H@J@L@P@R@@@V@X@Z@\@^@`@@@d@f@h@UnknownGz Times New Roman5Symbol3& z Arial3z TimesU Arial Unicode MSArial5& zaTahoma?5 z Courier New;Wingdings"1hù&\Q1\Q1!4``2qHX ?)2Direct and Indirect Assessment Mary Allen Mary Allen4         Oh+'0 $0 P \ h t Direct and Indirect Assessment Mary AllenNormal Mary Allen5Microsoft Office Word@F#@@*eN@gN\Q՜.+,D՜.+,P  hp  CSUB1`' Direct and Indirect Assessment Title, 8@ _PID_HLINKSA !&#http://www.act.org/caap/index.htmlIuGmailto:mallen@csub.eduI  !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~     Root Entry F1gNData *1TableWordDocument.SummaryInformation(DocumentSummaryInformation8CompObjq  FMicrosoft Office Word Document MSWordDocWord.Document.89qRoot Entry Fp\Data *1TableWordDocument.՜.+,D՜.+,P  hp  CSUB1`' Direct and Indirect Assessment Title, 8@ _PID_HLINKSA !&#http://www.act.org/caap/index.htmlIuGmailto:mallen@csub.eduISummaryInformation(DocumentSummaryInformation8CompObjq