PK
CM^2' ' mimetypeapplication/vnd.oasis.opendocument.textPK
CMNܑ content.xmlInnovative Learning and Teaching: Experiments Across the DisciplinesInnovative Learning and Teaching: Experiments Across the DisciplinesIlene D. Alexander and Robert K. Poch, EditorsInnovative Learning and Teaching: Experiments Across the Disciplines by Individual authors is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License, except where otherwise noted.Publishers To Toni A. H. McNaronAs a founding director of the University of Minnesota’s Center for Teaching and Learning Services, and creator of the “Early Career Teaching Program: Pursuing Excellence in Multicultural Education,” Toni exemplified her belief that institutional excellence derives from “curriculum, instruction, and scholarship that are culturally informed and inclusive of the students and larger society the University serves.”As a University teacher, Toni expected that her students would do brilliant things. She challenged and supported them in their individual as well as collective learning journeys. As teachers, these beliefs and expectations as integral to our teaching and learning work; as editors, we find them within the chapters of this book.In her mentor role, Toni’s words and deeds, equally, call attention to the possibilities that emerge when teachers talk with one another – through reflection, story, testimony, and scholarship – especially when in these dialogues teachers seek to understand learning, to see learners, and to expand teaching practices..I dwell in Possibility – A fairer House than Prose – More numerous of Windows – Superior – for Doors –(from Emily Dickinson, poem 657) And…To my students across the decades and the disciplines, especially Cherry Muhanji, Mary Bucklin, Jamie Peterson, Alex Fink, and Tyler Cessor. (Ilene)To LeRoy Gardner, Herman G. Green, Fred R. Sheheen, and Julia E. Wells who lit the path toward more inclusive, just, and joyful learning spaces. (Bob) ContentsAcknowledgementsCo-EditorsForeword: What Makes a University Course Good?Annotated Table of ContentsExpanding Active LearningPublic Lands, Virtual Classrooms: Teaching a Digitally Networked Field CourseUsing Game Dynamics to Teach Applied Statistics Lessons LearnedA Case-Based Course on Developing Interprofessional Health CompetenciesPlanning, Developing, and Implementing Nursing Telehealth SimulationBuilding New Course StructuresBuilding Understanding and Appreciation: Collaborative Work in General ChemistryTeaching Undergraduate History: A Problem-Based ApproachFlipping the Classroom and the Genetic Counseling Clinic: Online Branching CasesLearning through Generative ExplorationTranslating Knowledge to Engage Global Grand Challenges: A Case StudyReframing AssessmentSelf-Assessment in Language Courses: Does In-Class Support Make a Difference?Developing an Easy-to-Use Learning Analytics Tool to Facilitate Effective Course and Curriculum DesignImproving Performance and Reflective Learning through Video TechnologiesWeb Mapping Tools and Pedagogical Materials to Support Spatial ThinkingStudent Lecture Viewing: Learning from an Online Health Psychology MinorUnder the Watchful Eye of Online ProctoringAcknowledgementsSeeing Colleagues’ Teaching WorkFaculty awarded Provost’s “Experiments in Learning Innovations” grants during Fall 2014 to pursue innovative undergraduate teaching and learning at the course or curricular level were required to (1) engage in an 18-month process of designing, developing, launching, and revising teaching/learning projects in project teams that included pedagogy and technology consultants, and (2) wrap the project by providing a short statement reflecting on the project and process overall.Happily, most faculty completing the Innovations Grant cycle opted to participate in the Provost’s April 2016 “Innovations in Teaching Showcase,” from which this monograph emerged. Faculty moving among the posters seemed to remark, most of all, on what they were gaining in hearing about colleagues’ pedagogical frameworks, teaching practices, and integration of new approaches at course and curricular levels. One comment specifically sparked the monograph idea for Ilene: “It’s unusual – and nice – that this grant has us talking with other teachers. I like learning about my colleagues’ teaching work.” In May 2016 we sent an “Invitation to Publish” to recipients of Innovations Grants and of earlier Digital Technology Grants.Acknowledging Colleagues’ SupportNow November 2017, we’ve arrived at publishing Innovative Learning and Teaching: Experiments Across the Disciplines through the University of Minnesota Libraries Publishing Services’ open book and open textbook initiative. We’ve gotten here, first of all, thanks to the combined efforts of teachers who persisted beyond grant requirements to compose these chapters, and of consultants from the Center for Educational Innovation and Academic Technology Support Services whose expertise was integral to each project.More specifically, as co-editors we are grateful for colleagues who lightened the lift as we moved from “Let’s make a book” to “Here’s the book”:Danny Smith, who served as our Production Manager, converting a collection of documents into an organized, visually pleasing, accessibly-formatted monograph, and Mason Owens who coordinated the final text updates.The peer reviewers/first responders to manuscripts: CEI staffers Paul Baepler, Al Beitz, Paul Ching, Kris Gorman, Mary Jetter, David Langley, Jeff Lindgren, Kate Martin, Jane O’Brien, Christina Petersen, Bill Rozaitis; and longtime CEI partners Patrick Nunnally, and Tim Kamenar.The incredibly supportive Publishing Initiative duo of Shane Nackerud and Emma Molls whose initial warm welcome developed into ongoing support as we learned the publishing platform and moved through each development stage.Finally, we thank Kate Martin and Rebecca Ropers-Huilman for their constancy as well as their significant editorial insights and contributions.Co-EditorsIlene Dawn Alexander (alexa032@umn.edu) is a teaching consultant and Preparing Future Faculty instructor at the Center for Educational Innovation (Office of the Executive Vice President and Provost, University of Minnesota). She also serves also as an assistant faculty coordinator in the University’s College in the Schools program, mentoring high school teachers of the “University Writing” course, and as affiliate faculty in the College of Education and Human Development. Her research interests, as well as consulting and teaching practice, focus on multicultural, inclusive, accessible learning and teaching as these intersect with learning science, diversity science, networked learning, and various pedagogies with critical social justice orientations. Ilene earned MAs in English and Women’s Studies at Mankato State University, and completed a University of Iowa PhD in American Studies, and was supported by reflective, generous, fully-human teaching mentors at every stage.A photographer who got her first camera in grade school, Ilene shares the monograph’s cover photograph, Bright Weisman, under a Creative Commons Attribution 4.0 International License.Robert K. Poch (pochx001@umn.edu) is a senior fellow and researcher in the Department of Curriculum and Instruction at the University of Minnesota. Bob teaches undergraduate history and graduate courses in postsecondary multicultural teaching and learning, and college student development theory. His current research focuses on problem-based approaches to teaching history within diverse classrooms and biographical studies of largely unknown U.S. civil rights leaders. Bob is a recipient of the Horace T. Morse University of Minnesota Alumni Association Award for Outstanding Contributions to Undergraduate Education. He holds a PhD in higher education from the University of Virginia.Foreword: What Makes a University Course Good?Rebecca Ropers-Huilman What makes a university course good? The extent to which course experiences facilitate students’ learning of predetermined objectives? The ways in which the course pushes students beyond their boundaries to develop both content and communication skills? How the course ultimately prepares people to meet society’s needs? The likelihood that the course is sustainable over time? Innovative Learning and Teaching: Experiments Across the Disciplines addresses all of these questions through the lenses of university instructors grappling with challenges they experienced in their classes, and thinking creatively about how to overcome those challenges.This book is about teachers who care about students’ learning and are motivated to take risks and actively reflect on how to best facilitate that learning. What is most resonant for me after reading “Innovative Learning and Teaching” is how important relationships and communities are to effective higher education.Opening Part 1, “Expanding Active Learning,” Mark Pedelty describes the challenges and advantages of a digitally networked field course on environmental communication and suggests that learning can be enriched if done in the contexts being considered in the class. In other words, learning in communities strengthens the learning. Laure Charleux describes the gamification of her GIS statistics course and notes the importance of classroom community and students’ abilities to rely on each other, even if key activities in the course rely on individual effort. Karin Hamilton and Tricia Todd reflect on how bringing real world health challenges as case studies into their classroom facilitate students’ knowledge of how to work in interprofessional teams and develop their identities both as health professionals and as team members. They stress that learning how to be in relationship with one another is critically important to developing an appropriate response to health-related challenges. Mary M. Rowan, Mary Steffes, Carol Flaten, Lori Rhudy and Nima Salehi illustrate how changes in a profession necessitate changes in pedagogical approaches. They explain how using simulations with nursing students help to ensure that those students can participate adequately in a technology that is increasingly used in that field. These simulations help future nurses hone communication and relational skills that they will use as they serve their broader communities.The chapters of Part 2, “Building New Course Structures,” expand this focus on the importance of relationships in learning, and focuses more directly on how the structures of the courses can facilitate learning developed through those relationships. Reflecting on his teaching of Chemistry, Brian Gute writes, “I became convinced that if I was going to improve the experience for my students, I needed some way to connect with them as individuals. That connection would also provide me with a lens into their learning, giving me a clearer picture of what they understood and where they were still struggling.” He elected to implement a flipped classroom model that facilitates students’ learning about teams, ensures students’ accountability for their own learning, and makes the learning fun for students and teacher alike. Robert K. Poch and Eskender Yousuf describe their desire to help students “experience the excitement of historical discovery and personal meaning-making” through incorporating problem-based learning into their teaching of History. By asking students what problems they want to investigate and focusing on materials from and about that time period, students develop relationships with their instructors as guides rather than conveyors of “truth.” They further come to develop independent skills as emerging historians. Alex Cummings, Andrea Stoddard, Pat McCarthy Veach, Bonnie LeRoy and Heather Zierhut structure their course around case-based learning where students observe recorded genetic counseling sessions and respond to questions associated with those sessions. While the authors note limitations of this approach, they believe that it enhances students’ abilities to work with future clients. Similar to Poch and Yousuf’s desire, Brad Hokanson and Jody Lawrence intended that their students “be personally interested and invested in their work, and for them to be motivated and self-driven.” To accomplish this, they use generative learning that fosters both creativity and confidence in the community that is created by the class structure. Daniel Philippon, Barrett Colombo, Fred Rose and Julian Marshall describe a structure to implement a curriculum that is focused on meeting the grand challenges of our larger society. Among other strategies, they explicitly teach students how to transform “Knowledge to Impact” so that they learn about identified challenges and take action to address those challenges.Part 3, “Reframing Assessment,” brings together chapters focusing on how assessment tools can help students develop their skills and knowledge related to both course content and, in some cases, learning in collaboration with peers. Gabriela Sweet, Sara Mack and Anna Olivero-Agney advocate for tools that put the learner at the center of courses (in this case, language courses) and help learners reflect purposefully about their learning. By including a “Human Dimension” in this assessment, students develop awareness of how language skills can help them develop both self-awareness and meaningful interactions with others. Xavier Prat-Resina, Molly Dingel and Robert Dunbar describe how faculty at an innovative teaching and learning campus developed a tool to help them determine how to structure their courses to maximize students’ success. This chapter describes how a faculty can work together to create learning tools that benefit both students’ and faculty members’ learning. Mary M. Rowan, Carol Flaten, Lori Rhudy and Nima Salehi explain their use of video, self-review, peer response, and instructor feedback in areas related to key nursing skills. They found that when students had hands-on experience with multiple levels of feedback, they learned from others how to refine their practice. Steven M. Manson, Melinda Kernik, Dudley Bonsal, Laura Matson, Eric Deluca, Ashwini Srinivasamohan and Sophia Strosberg outline how they used mapping, “as a tool to examine interrelationships among individuals, institutions, structures, events, ideas, and technology.” Advocating that maps can be used to tell stories, they present the wide range of uses that students can experience through technology associated with mapping to engage in spatial thinking. Focusing on the use and value of lectures in online courses, Thomas Brothen, Penny Nichol and Esther Joy Steenlage Maruani analyzed three Psychology courses that were taught entirely online. They found that lectures were not a significant predictor of most learning measures and advocated that lectures be seen as a supplement to, rather than a centerpiece of, university courses. Finally, Daniel Woldeab, Thomas Lindsay and Thomas Brothen review the relatively new assessment tool of online proctoring. To ensure academic integrity during assessments taken outside a traditional, place-based class setting, online proctoring serves as a way to expand the reach of university course offerings to those who are geographically dispersed or otherwise unable to take an assessment in a traditional setting.Throughout Innovative Learning and Teaching: Experiments Across the Disciplines, three things are clear. First, relationships matter. The relationships between students and teachers are key to learning; similarly, the relationships among faculty who come together to improve their teaching are also instrumental to teaching innovation. Second, institutional support matters. Many authors referred to institutional support that facilitated their learning innovations. Administrators at several levels can incentivize the development of new teaching approaches, both through funds to experiment as well as through an encouragement to take teaching risks that may encounter unanticipated obstacles. Finally, it is clear that the teacher’s role in advancing student learning is of utmost importance. This is true both in commonly understood ways (such as clarity of expectations and the crafting of experiences that advance clearly articulated course objectives) as well as in less commonly discussed ways (such as the teacher’s passion, creativity, and interpersonal relationships with students).Collectively, the authors in this volume demonstrate a commitment to on-going responsiveness to challenges as well as a desire to incorporate opportunities made available by technological developments. Readers will benefit from both specific ideas about course design and the active reflections of teachers who care deeply about their relationships with students, their disciplines, and the broader world.Annotated Table of ContentsForeword: What Makes a University Course Good?Rebecca Ropers-Huilman, Vice Provost for Faculty and Academic Affairs, and Professor of Higher Education, ropers@umn.eduThe authors in this volume demonstrate a commitment to on-going responsiveness to challenges as well as a desire to incorporate opportunities made available by technological developments. Readers will benefit from both specific ideas about course design and the active reflections of teachers who care deeply about their relationships with students, their disciplines, and the broader world.I. Expanding Active LearningPublic Lands, Virtual Classroom: Teaching a Digitally Networked Field CourseMark Pedelty, Communication Studies, pedelty@umn.eduEnvironmental Communication was transformed from a classroom-based course into a “digitally networked field course.” Each student now chooses a public land field site to start the semester and develops a site. Specific”interpretive talk”, with these aspects of the course linked together into a virtual classroom using an e-learning platform and students’ smartphones. This chapter provides a realistic description of the rewards and challenges that accompany this relatively new methodology.Using Game Dynamics to Teach Applied Statistics Lessons LearnedLaure Charleux, Department of Geography, UMD, lcharleu@d.umn.eduNoting students’ difficulty acquiring a working knowledge of applied statistics for GIS analyses in her traditionally taught course, the author describes and reflects upon what happened when she designed game dynamics to improve student learning.A Case-Based Course on Developing Interprofessional Health CompetenciesKarin Hamilton, Department of Veterinary Population Medicine, khamilton@umn.edu, and Tricia Todd, Health Careers Center; todd0002@umn.eduThis chapter describes the three-year evolution of a case-based pedagogical approach where students compete on interprofessional teams to solve complex health challenges while developing the skills, attitudes, and behaviors to work effectively in future health careers.Planning, Development, and Implementation Nursing Telehealth SimulationMary M. Rowan, rowan005@umn.edu, Mary Steffes, steffes005@umn.edu, Carol Flaten, cflaten@umn.edu, Lori Rhudy, rhudy002@umn.edu, and Nima Salehi, sale0012@umn.edu; all School of NursingThe authors chronicle the development, implementation, and learning outcomes of a high fidelity telehealth home visit simulation for undergraduate nursing students. Students learned about telehealth regulations and the challenges of communicating information using lay language.II. Building New Course StructuresBuilding Understanding and Appreciation: Collaborative Work in General ChemistryBrian D. Gute, Department of Chemistry and Biochemistry, UMD, bgute2@d.umn.eduThis chapter describes the steps taken to address the instructional challenges in General Chemistry II and to improve the student learning experience through the design and construction of a fully flipped course.Teaching Undergraduate History: A Problem-Based ApproachRobert K. Poch, Department of Curriculum and Instruction, pochx001@umn.edu, and Eskender Yousuf, McNair Scholar Program, yousu014@umn.eduThe authors describe the conversion of a traditionally taught history survey course into a problem-based course that enables students to experience historical inquiry through well-defined historical thinking skill development and the application of those skills to problems that historians encounter as they interact with primary and secondary sources.Flipping the Classroom and the Genetic Counseling Clinic: Online Branching CasesAlex Cummings, East Tennessee Children’s Hospital-Knoxville Genetics Center, cummi427@umn.edu; Andrea Stoddard astodard93@gmail.com, Pat McCarthy Veach veach001@umn.edu, Bonnie LeRoy leroy001@umn.edu, and Heather Zierhut zier0034@umn.edu; all Department of Genetics, Cell Biology, DevelopmentSimulated genetic counseling sessions are one way to increase access to clinical experiences without increasing the time required for direct supervision. The authors report on developing three hypothetical genetic counseling situations to further learners’ understanding of clinical situations, and summarize student survey data regarding implementation of one module.Learning Through Generative ExplorationBrad Hokanson, Department of Design, Housing, and Apparel, brad@umn.edu, and Jody Nyboer, School of Design, Syracuse University, jlnyboer@syr.eduThis chapter presents the methods of a course that develops the creativity of learners through generative exploration, a process of learning in which learners solve problems, develop ideas, and stretch limits. Observations of course operation and theory are presented along with student comments.Translating Knowledge to Engage Global Grand Challenges: A Case StudyDaniel Philippon, English Department, danp@umn.edu, Barrett Colombo, Institute on the Environment, colom008@umn.edu, Fred Rose, Institute on the Environment, rosex122@umn.edu, and Julian Marshall, Civil and Environmental Engineering, University of Washington, jdmarsh@uw.eduThis article chronicles the pilot efforts of an interdisciplinary team of faculty to develop a co-curriculum for trans-disciplinary skills and foundational competencies that prepare students to effectively engage a broad range of grand challenges, work in small teams to develop viable solutions (e.g., a social venture plan or policy intervention), and propose these solutions to an audience beyond their particular course.III. Reframing AssessmentSelf-Assessment in Language Courses: Does In-Class Support Make a Difference?Gabriela Sweet, CLA Language Center, sweet003@umn.edu, Sara Mack, Department of Spanish and Portuguese Studies, mack@umn.edu, and Anna Olivero-Agney, CLA Language Center, olive152@umn.eduThe authors describe a language learning self-assessment protocol and compare data on the effects of two different implementations of the protocol on students’ analyses of their knowledge and skills.Developing an Easy-to-Use Learning Analytics Tools to Facilitate Effective Course and Curriculum DesignXavier Prat-Resina pratr001@r.umn.edu, Molly Dingel dinge016@r.umn.edu, and Robert Dunbar dunb0011@r.umn.edu; all Center for Learning Innovation, UMRThis chapter reports on the faculty-driven development of an institutionally-supported research tool that allows the exploration of student learning related data, which can facilitate hypothesis generation and evaluation. The chapter summarizes the trajectory of the project from faculty idea to institutionally supported research tool, discusses the many valuable partnerships and collaborations required for success, and links to a beta version of the tool accessible for review.Improving Performance and Reflective Learning through Video TechnologiesMary M. Rowan, rowan005@umn.edu, Carol Flaten, cflaten@umn.edu, Lori Rhudy, rhudy002@umn.edu, and Nima Salehi, sale0012@umn.edu; all School of NursingThe authors describe how video recordings of students practicing nursing skills in laboratory courses can build on and enhance skill development through the use of self-reflection activities and peer feedback.Web Mapping Tools and Pedagogical Material to Support Spatial ThinkingSteven M. Manson manson@umn.edu, Melinda Kernik, kerni016@umn.edu, Dudley Bonsal, bons0015@umn.edu, Laura Matson, matso092@umn.edu, Eric Deluca, delu0027@umn.edu, Ashwini Srinivasamohan, srini134@umn.edu, and Sophia Strosberg, stros023@umn.edu; all Department of GeographyThe authors describe how web-mapping tools and pedagogical material they have developed frees spatial thinking instruction from the classroom and encourages students to creatively engage in the spaces around them in new ways.Student Lecture Viewing: Learning from an Online Health Psychology MinorThomas Brothen, broth001@umn.edu, Penny Nichol, nich0185@umn.edu, and Esther Joy Steenlage Maruani, esther@umn.edu; all Department of PsychologyThe authors report on their use of lecture-viewing analytics to determine how lectures compared to other course activities in their effect on learning outcomes in an online course.Under the Watchful Eye of Online ProctoringDaniel Woldeab, College of Individualized Studies, Metropolitan State University, daniel.woldeab@metrostate.edu, Thomas Lindsay, Liberal Arts Technologies Innovation Services, lindsayt@umn.edu, and Thomas Brothen, Department of Psychology, broth001@umn.eduThis chapter presents data on how students and instructors perceived online proctoring in three online or hybrid psychology courses. While instructors were generally positive about the proctoring service, students conveyed numerous concerns.PART IExpanding Active LearningChapter 1Public Lands, Virtual Classrooms: Teaching a Digitally Networked Field CourseMark PedeltyKeywordsdistance education, field instruction, environmental studies, environmental communication, flipped classroom IntroductionThings were going well in our windowless classroom deep in the basement of Ford Hall. I was teaching Environmental Communication for the sixth time in about as many years. Students were presenting what they had learned in experiential projects conducted outside the classroom when suddenly it struck me, “Why are we doing this in a classroom?” As I had discovered when teaching a study abroad version of the course, and in related projects in the local community, the subject of environmental communication lends itself well to field-based teaching. Furthermore, new technologies allow us to create dynamic and mobile virtual classrooms, freeing us to teach and learn in rich contexts, the places were environmental communication takes place as a matter of course every day. Therefore, why not teach the course in a state park, superfund site, or nature center where professional environmental communicators work everyday? It seemed as if the only major weak link in the class was that we were in a basement classroom rather than out learning in the wide variety of locations where environmental communication comes alive as a matter of course. That move seemed to be the natural next step for the course. So, I set out to turn COMM 4250 into a “digitally networked field course.”Specifically, students in the Environmental Communication course gain a broad overview of environmental communication as a field of research and as a profession, starting with theoretical foundations and then stepping through individual areas of research and practice, including but not limited to: environmental discourses, news, advocacy, advertising, public participation processes, risk communication, science communication, and creative mediation of environmental questions in film, television, music, and the arts. Fortunately, the course text, Environmental Communication and the Public Sphere (Cox and Pezzullo, 2015), provides an excellent resource to organize and support that contextual overview and casework. That aspect of the course has not changed with the redesign. Instead, adding the field element has made those topics come alive on-site and in the students’ field-based project, which is an “interpretive talk” about their site. Each student also chooses to explore an environmental subject relevant to that site (e.g., fen preservation at Fort Snelling State Park or toxin containment at an urban superfund site). In the first offering students could choose any public land or water site in Minnesota, but by the second we were limiting students mainly to those with interpretive resources on site, including a nature center and professional staff. We found that students choosing sites without those resources were at a major disadvantage and that nearly all needed that extra support, especially as a way to orient themselves to the local ecosystem and to learn about relevant challenges and questions. Students could choose a site anywhere in Minnesota, but we recommended sites near to their homes so that several visits might be possible as opposed to a few intensive encounters. The more distant sites did work for students willing to make at least three extensive trips to their site.The course is based on a praxis model. Reading and theoretical knowledge enrich the students’ practical work on site. In turn, developing a project and exploring a public land site brings the course materials and concepts alive for students. Skills-building is not central to the course per se—we do less in terms of media production training, for example, than might be preferable and possible in a two semester course sequence—but students do learn valuable presentational skills in the process of researching, writing, rehearsing, and recording interpretive talks over the course of the semester, using expert models, peer and teacher input as a resource.Instructors must learn and develop as well. We are human beings, not teaching machines. A teacher’s emotional investment and sense of intellectual challenge is crucial to sustaining a quality course, and therefore worth exploring.Fortunately, several people helped me make the transition from the classroom to a digitally networked field course. Sara Schoen, Paul Ching, Joy Hamilton, and several other UMN faculty, staff, and students helped me convert the course over a period of months drawing on support from the Experiments in Learning Innovation (ELI) program. The result was a course redesign in which each student chooses a public land field site. All 100 plus sites and students were linked together into one big virtual classroom using Moodle (a e-learning platform) and students’ smartphones.We completed a quantitative and qualitative assessment of the course and wrote about the results in a book chapter (Pedelty and Hamilton, 2017) for Milstein and Pileggi’s Environmental Communication Pedagogy and Practice. However, we did not discuss one of the most important aspects of the course: teaching. The change in teaching methods and the instructor’s experience is perhaps one of the most important innovations in the course, a radical change in how, where, and why we teach the course. Here I will describe what it is like to teach a digitally networked field course. I do so knowing that colleagues might consider adapting a similar format and therefore would like to offer additional information and ideas based on this transformative teaching experience.Teaching MattersIn an age of student-centered learning we often forget about the teacher’s perspective, speaking and writing about teaching and learning as if the student experience is the sole determinant of what creates a quality course and curriculum. Therefore, we often fail to discuss factors like emotional investment, intellectual motivation, experience, and even logistics. In truth, a course could offer an extremely effective learning experience, but if it is not also a quality teaching experience the instructor might never teach it again. Many instructors have crafted an ideal course only to find that it was unsustainable term-to-term for reasons that go beyond student learning outcomes.There are many considerations that go into determining whether or not a given pedagogical practice will actually work best in a give situation and for a given teacher. Does the style of teaching and delivery match the instructor’s skills, knowledge, personality, and interests? Is there a high enough level of intellectual challenge, emotional investment and perhaps even pleasure to make the instructor want to teach the course again and again?I do not want to imply that instructors are selfish. Most of our gratification comes from seeing students learn and succeed regardless whether or not the method fits our own proclivities. However, even the most effective course cannot be sustained if the experience of actually teaching the class—the method, context, and subject matter—do not continually reignite the professor’s passion for teaching the course year-in-year-out. “Passion in teaching is not a luxury or a frill that we can do without,” note medical education specialists Harden and Laidlaw, “it is the key element in students’ learning.” They argue that a teacher’s “passion” is more important “than the teaching strategy adopted” as well as “the learning technology incorporated” (Harden Laidlaw, 2012, 20).Yet, reading the pedagogical research literature one might be led to believe that college instructors are endlessly flexible professionals who institute the best teaching practices, regardless what a given method means for them or requires of them. We all know from teaching, talking with colleagues, and negotiating the distribution of teaching “loads” across our faculties that there is far more to the effective teaching and learning equation. Instructors must learn and develop as well. We are human beings, not teaching machines. A teacher’s emotional investment and sense of intellectual challenge is crucial to sustaining a quality course, and therefore worth exploring. After all, even “the perfect” course is useless if no instructor is willing to teach it more than once. A quality teaching experience is one of many prerequisites for an effective course and positive student learning outcomes.This is not to say that teaching environmental communication semester-in-and-semester-out in the classroom was becoming a drag : it was by far one of the highlights of my yearly teaching cycle already. However, it was time to take the course to the next step for sake of student learning and, not inconsequentially, to best utilize my skills, interests, and passions in relation to the subject. While some instructors would find the extra logistical challenges of field teaching an annoyance, and rightfully so, for me the teaching and learning benefits of the digitally networked field course more than compensated for the extra time and complexity involved.The CourseWhen I tell colleagues that I am teaching a digitally networked field course, two questions generally arise: “What is it?” and then “What’s it like to teach it?” In the next two sections I will answer both questions, starting with an overview of the course. However, please do keep in mind that this is just one of many ways to teach a digitally networked field course. If the classroom can be adapted to a wide range of subject matter, formats, and teaching styles, then the world outside it is at least as malleable and variable.In the first three weeks of the course, students choose and begin to explore a public land site in Minnesota. They start with on-site discovery, supplemented by library research. In Fall 2015 students chose field sites ranging from the Mississippi River Flats, just off campus, to the Boundary Waters Canoe Area Wilderness, 270 miles to the North.They each researched a specific environmental issue related to their site and then developed an “interpretive talk” (Henker and Brown, 2011), what many of us colloquially refer to as a “ranger talk.” By the end of the semester students had written, rehearsed, performed, and filmed their interpretive talk at their site. In other words, they learned about environmental communication by doing it. They also read articles and books about topics in environmental communication related to a given stage of their project (Cox and Pezullo, 2015), took weekly quizzes covering the readings, and wrote short weekly reports that integrated reading assignments, online discussions, and onsite fieldwork.By the end of the semester students had written, rehearsed, performed, and filmed their interpretive talk on site. In other words, they learned about environmental communication by doing it. They also read articles and books about topics in environmental communication related to a given stage of their project, took weekly quizzes covering the readings, and wrote short weekly reports that integrated reading assignments, online discussions, and onsite fieldwork.The interpretive talk assignment is at the core of the course. Developing that project was part of each week’s assignment, a scaffolded approach that required students to become environmental communicators one step at a time. One of the best indicators of learning is whether or not a student can perform what is expected of a professional in a given subject. The interpretive talk assignment teaches students how to become skilled environmental communicators and allows them to demonstrate their learning directly.In order to focus on content and performance, students were required to video a live performance. Students were not allowed to edit their videos or do any other post-production work on the live video. The goal was to learn and demonstrate performative communication abilities rather than develop technical production skills. Nor did I want students with advanced video production skills to “hide” weaknesses or poor preparation for the live performance via clever editing.Weekly lessons and materials are delivered via videos we produced specifically for the course. Each week students watch one video on Moodle, a total of fifteen over the course of the semester. A few are solo lectures that I delivered in studio, but most involved interviews we conducted with local experts in environmental communication, including a number of UMN scholars in related fields, experts all in delivering environmental information to publics and policy makers. Andrew Matthews, an expert in pedagogical film production, filmed and edited the lectures that required green screen or other augmentation. The following website provides an overview of the Moodle site, sample lecture and interview videos, and a Google Map with sample student videos: https://sites.google.com/a/umn.edu/4250-moodle-screenshots/.Students also listen to a weekly podcast created by the instructor. Podcasting allows for creative adaptation to students’ specific needs at the moment. Each podcast was recorded at a public land site, most often a Minnesota State Park. In addition to “housekeeping” matters, such as specific advice for completing that week’s report or how to improve quiz scores, the podcasts would include field interviews and present advice to help students complete a given stage of the interpretive talk project. In both formats, the instructor modeled the interpretive talk project directly by performing a range of possibilities. The goal in presenting interpretive talks ourselves was not to prefigure the students’ own interpretive talks, but rather to provide a range of options and ideas to inspire their individual creative work in the field.Naturally, after the first iteration of the redesigned course, changes are underway for future semesters, but a surprisingly high proportion of the course was retained for 2017 and is continuing on. Much of the course was developed over seven iterations of the classroom course, but a great deal was new as well. Most of the readings, Moodle site, videos, weekly podcast, and students’ onsite research and interpretive talk assignment will stay as is, having proven their worth in the 2016 experiment. However, as will be explained at the end of the chapter, the interpretive talk assignment was changed from video to audio.Once again, a more rigorous assessment and explanation of student learning outcomes is available elsewhere (Pedelty and Hamilton 2017). Having provided an overview of the course here, I will move on to the main purpose of this chapter: to describe what it was like to teach a digitally networked field course.The Teaching ExperienceAs explained earlier, even the illusory “perfect course” (Miller 1993) would be useless if it did not offer a fulfilling experience for the instructor. For me, teaching a digitally networked field course was a revelatory experience in that regard. Although it was more time consuming than most of the classroom courses I have taught recently, benefits far outweighed the costs. It was more fulfilling and intellectually stimulating than most of the classroom-based courses I have taught. Consider the weekly podcast. Each week I would devote an entire day to traveling to relatively remote public land sites, presenting podcasts that stepped students through the weekly report assignment, helping students develop their interpretive talks, and providing model talks using specific examples. For example, one podcast covered legal issues surrounding nitrates in the Des Moines River. That talk was presented from the banks of the Des Moines, in Southwestern Minnesota’s Kilen Woods State Park. Another podcast involved an interview with Park Ranger Mark Crawford about his efforts to protect the endangered Blanding’s turtle at Lake Maria State Park.Having advanced beyond simplistic, Cartesian binaries between mind vs. body or positivistic conceptions of intellect vs. emotion, the academy as a whole is rediscovering that there is nothing magical about sitting around the seminarians’ table. Neither movement nor the senses (i.e., empiricism) are antithetical to thought. Instead, they can become part of the discovery process.As another example, when traveling to the National Communication Association Annual Convention in Las Vegas (2015) I made a side trip to Red Rock Canyon National Conservation Area to talk about efforts there to interpret human impacts in the area, from water scarcity to damage of the surrounding cliffs by rock climbers. Each week involved another topic, another park, and another stage of the students’ projects. Sound like fun? It was.As I suspected when first considering the idea of breaking down the classroom walls—and as many of us learn time and again when teaching field or study abroad courses (Milstein, 2005)—there is nothing like a rich learning context to bring learning alive (Varopoulou, 2010). The logistics are often daunting, but well worth it. Even with flipped versions of the course I had started to sense that the weakest link in COMM 4250 was the comparatively stale, controlled, and sensory-deprived context of the classroom. The “environment” for our environmental communication course was a lifeless, beige, windowless room full of rolling desk chairs. Now it is hundreds of open spaces where students explore the questions in the reading, videos, and podcast in the kinds of places reference in those work: learning in a rich context. The “classroom” is a digitally networked set of public lands and waterways.Yet, distance education is itself often criticized as sensory-deprived. That is because it often can become so. Students stare at their screens taking tests and reading assignments rather than working face-to-face with an instructor in real time and space as a classroom allows. However, it is important to remember that campuses and classrooms developed for much the same reason as digital distance education: economy of scale. It was less expensive to deliver instruction to a large population of students corralled into common classroom spaces than to teach individuals or small groups in rich contexts where a given subject is practiced. New communication technologies can take us out of the classroom and back into the places where apprenticeship and direct discovery once again become possible.To reverse Sherry Turkle’s (2012) pessimistic phrase, new learning technologies allow us to be “together alone,” learning individually in the field yet linked together via virtual classrooms. From boardrooms to farm fields the subject matter comes alive. Students and instructors alike can feel a more visceral connection to learning, seeing its relevance directly instead of one step removed in the virtual space of the classroom. It allows for students and teachers to use their entire bodies, to move about the country and campus rather than sublimating their senses to purely cerebral pursuits with prone body postures in fixed locations. The mind comes alive with the body.Having advanced beyond simplistic, Cartesian binaries between mind vs. body or positivistic conceptions of intellect vs. emotion, the academy as a whole is rediscovering that there is nothing magical about sitting around the seminarians’ table. Neither movement nor the senses (i.e., empiricism) are antithetical to thought. Instead, they can become part of the discovery process.This is not to denigrate our typical mode of scholarly exchange and learning—text and talk—but rather to suggest that we could be seizing new opportunities to augment classroom learning, academic conferencing, and other forms of exchange with styles of teaching and learning that take us into rich contexts. While the classroom will remain central to many if not most subjects, when it comes to environmental studies we need to find more ways to get students out of the classroom and into the world.Challenges and ModificationsOf course, not everything worked well in the first attempt at digitally networking the field course. Our assessment chapter (Pedelty and Hamilton 2017) details some of the main challenges we faced in teaching the course in a digitally networked field format. Among those was the first that I will address in this section: the technical challenges faced by students as they attempted to film their interpretive talks onsite. Even with minimal tech expectations, students had trouble filming and performing their talks. My assumptions about digital natives did not hold up across the board and with the already demanding requirements for writing, rehearsing, and performing a talk, time did not allow for quality video recording in most cases. Too many prerequisite skills were taken for granted in planning the course. There was a bit of piling on relative to reasonable course expectations. An advanced course is warranted for those who would like more advanced training in environmental field performance and media production.Likewise, it was difficult to teach performance-based skills at a distance without the controlled support of the face-to-face physical classroom space for weekly performance workshops (Pedelty and Hamilton, 2017). Just as I took basic smartphone-level video skills for granted, so too I forgot how much of course learning in past courses had been predicated on face-to-face performance training with peers and the instructor. Those students who needed extra tech and performance assistance suffered somewhat in the digitally networked version of the course. For that purpose, we needed to be in the same space at the same time or we needed more time for an iterative approach, with sufficient time to present very early work and gradually develop more advanced performance skills via remote instruction. A “rough cut” relatively late in the game helped, but was not enough for many students in comparison to the previous classroom-based course.The main fix for the challenges posed by these tech and performance gaps, therefore, was to the move into audio-only podcasting for the Spring, 2017 version of the course. Audio has allowed for greater on-site dexterity, weekly workshops, and more gradual staging, training and development. In many cases videography and visual performance distracted students and instructors from core learning goals. While video would work very well in a longer, more advanced course in environmental communication, it was much less useful in this introductory course designed for a broad student population hailing from a variety of majors and with little time to develop technical skills.Which brings us to the third and final course weakness. Perhaps the biggest challenge was that of teaching 120 students (100 by the end). Three talented Teaching Assistants were essential to making that work, but the scale of operation was at times too great to realize course goals. Although increased scale and efficiency were initial goals for the transformed course, they were not the primary impetus for the revision and those goals will now be dropped. A 25 to 40-student enrollment cap is ideal for the classroom version of the course and the same appears to hold true for the digitally networked format. For example, much of the interactive online teaching takes place via an instructor’s assessment of each student’s weekly reports. Once the enrollment exceeds 40 students, it becomes difficult to effectively grade weekly reports, including the crucial first and final cuts of the interpretive talk. Of course, that is a problem for any large course, but the performance skills development goals of COMM 4250 makes large enrollments a particularly deep deficit in digitally networked field courses. Much like other field and skills-based courses, this one needs to have a favorable teacher-to-student ratio.I met with every student in person during the third week of classes to establish a face-to-face relationship and I met with students during office hours to read their drafts and reports. The TA’s (3 in the very large first course, 1 in the second smaller offering) also made themselves available for appointments.Finally, one other aspect of the course failed. In fact, it failed before the semester even started. I had hoped to make this the first system-wide course at the University of Minnesota. After all, a digital field connection should allow us to network, teach, and learn across all five campuses without worrying about geographic distance. Having taught at University of Minnesota, Morris in the 1990’s, and having interacted with University of Minnesota Duluth students and faculty during my eighteen years on the Twin Cities campus, I very much wanted all University of Minnesota students to have access to this course.Unfortunately, students on any given U of M campus must petition to take a course at another U of M campus, a ponderous obstacle to organic interactivity across campuses. After discussion with administrators in charge of facilitating potential system-wide learning, correspondence with other campus officials, and travel to UMD to set-up a more permanent exchange, it became apparent that the challenges of creating a course with significant enrollments of students outside the Twin Cities campus would be too great to overcome in any reasonable time budget. While a few non-TC students took the steps of finding out about the course via campus flyers, visiting their advisors for approval, and submitting petitions to register for the course, in all practicality those procedures work against the goal of creating a system-wide course. Plus, the 120-student enrollment cap was quickly reached with UMTC students. However, turning this into a system-wide course remains a future goal and COMM 4250 seems like a natural pilot for that eventual effort.Each of the above course components has been assessed and reconsidered. As a result, the next course offering involved a smaller enrollment cap, audio-only main projects, and further restrictions on field site selection. Students were required to choose a site that has interpretive resources available from the start, including national and state parks as well as county parks with nature centers and professional or volunteer interpretive staff on site. We discovered that most students benefited greatly from interpretive resources to focus their initial discovery research and to supplement course materials and library research. With some remarkable exceptions, students who chose public lands and waterways without on-site interpretive resources—nature centers, placards, exhibits, self-guided trails, and rangers—were at a disadvantage relative to peers who chose parks, preserves, and public lands that contained those useful resources.Fortunately, most of the above challenges were overcome and the second iteration of the digitally networked field course seems to be working even better. The next iteration will involve relatively small tweaks rather than significant changes. The current students are producing exceptional interpretive talks that I will assign in the next class. They will serve as individual models and give them a sense of the wide range of possibilities that the genre affords.ConclusionThe digitally networked field course has a number of moving parts and takes a great deal of work to prepare and teach, but it is highly pleasurable, rewarding, and intellectually challenging format. It simultaneously advances student learning and fully engages the instructor. Course evaluations support that conclusion, with students offering up positive intellectual as well as emotional assessments of the course (Pedelty and Hamilton 2017). We taught and learned within a digital classroom that brought us into contact with some of the most biodiverse, publically relevant, and fascinating places in our state.This chapter began with description and end in advocacy. The digitally networked field model worked very well, which is one of the reasons I want to share the experience with colleagues. With the right conditions and subject matter, students learn important lessons when given the freedom to choose their field site and share what they learn there. It would not work for every topic, but it has worked for environmental communication. For many instructors, digitally networked field instruction it is the type of teaching that can continually reignite a professor’s passion for teaching and student’s interest in learning, drawing upon the pedagogical agency of place.ReferencesCox, R. and Pezzullo, P. eds. (2015) Environmental Communication and the Public Sphere (4th ed.) Thousand Oaks, CA: Sage Publications.Harden, R. M., Laidlaw, J. M. (2012). Essential Skills for a Medical Teacher: An Introduction to Teaching and Learning in Medicine. Edinburgh: Churchill Livingstone.Henker, K. B., Brown, G. (2011) As Good as the Real Thing? A Comparative Study of Interpretive Podcasts and Traditional Ranger Talks. Editorial Assistant, 7.Kolb, D. A. (2014) Experiential Learning: Experience as the Source of Learning and Development. New York: Pearson Education.Miller, J. E. (1993) “Tradeoffs in Student Satisfaction: Is the “Perfect” Course an Illusion?” Journal on Excellence in College Teaching 4 27-47.Milstein, T. (2005) “Transformation Abroad: Sojourning and the Perceived Enhancement of Self-Efficacy” International Journal of Intercultural Relations 29 2 217-238.Milstein, T. and Pileggi, M. (2017) Environmental Communication Pedagogy and Practice. Routledge, New York.Pedelty, M. and Hamilton, J. (forthcoming, 2017) ““Further Afield: Performance Pedagogy, Fieldwork, and Distance Learning in Environmental Communication Courses” In Milstein, T. and Pileggi, M. (2017) Environmental Communication Pedagogy and Practice. Routledge, New York.Roberts, J. W. (2015) Experiential Education in the College Context: What it is, How it Works, and Why it Matters Routledge, New YorkTurkle, S. (2012) Alone Together: Why we Expect More from Technology and Less From Each Other. New York: Basic books.Varopoulou, H., (2010) “Landscape with Students and Teachers: Landscape, Environment and Performance Pedagogy in a Greek Summer Academy” Performance Research 15 4 123-130. Chapter 2Using Game Dynamics to Teach Applied Statistics Lessons LearnedLaure CharleuxKeywordsgamification, moodleTeaching Applied Stats to GIS StudentsGeographical Information Science is about collecting data and analyzing them in relation to their location in order to create new information. For instance, GIS students might learn how to integrate data from the park and recreation service of a city with census data to calculate and map the ratio of public green spaces to the number of inhabitants within a certain reach. Quickly though, some amount of basic statistical knowledge is needed to carry out interesting analyses and draw meaningful conclusions. For instance, students need to know about spatial correlation to decide whether or not the poorest areas of the city also tend to be the ones with the least access to public green spaces.Until recently, academic GIS programs have mainly been at the graduate level but they are trickling down to the undergraduate level at an accelerating pace. While it is standard in many disciplines to expect graduate students to learn how to carry-out statistical analyses, the endeavor is more challenging at the undergraduate level. With GIS especially, many students choose the field because it is a technology-oriented discipline that is perceived as requiring little to no math. As a result, in my “Applied Statistics for GIS” class includes some seniors who are at a loss for calculating percentages. This mandatory class hits them off-guard, as they are required to calculate not only percentages (or other indicators), but also estimates of percentages and their confidence intervals, and to test how significantly they differ from one another or from a reference level. Students are also introduced to bivariate analysis and have to carry out and interpret a couple of regression analyses. Confronted with a higher level of math than expected, some students become defensive.In truth, I also get students who are really good at math. Some of them are even double majors in another field, like biology, where they have learned some statistics. But they too are disturbed by the applied nature of this class. While they are used to and good at neat exercises where everything works according to the book, this course requires they work with “real world” scenarios. The praxis of GIS data analysis is messy, and it is important to introduce students to this messiness: How does one make the call to exclude outliers from the analysis? How strong does a correlation need to be to be deemed worthy of further exploration? When is it okay to recognize one does not have enough information to draw conclusions and how does one report this? For some of my students who feel comfortable with math as a sequential process of reviewing a problem, selecting a formula and calculating the correct solution become very disturbed when confronted by the murky reality of praxis.When I first started teaching this class, I used a traditional outline, focusing on a different method every week or every other week, with lecture, exercises and lab assignment for each method. My stated goal was to introduce students to a series of important methods, to get them thinking about uses for these methods, and how to use them. The outcomes obtained with this approach were unsatisfactory on several levels, and I will mention only a few here. For instance, this course layout did not allow for the proper emphasis on the messiness of praxis. When students work on a new method every week, as teachers we don’t want them distracted too much by the weird formatting of real-life data, so we end-up pre-formatting the data for them. In this set up, it becomes questionable whether they would be able to apply a particular method when confronted with unformatted data. Additionally, grading each lab only once was not the proper means to improve their statistical writing. As part of learning to analyse and report on data, it is crucial that students learn to write the interpretation of their statistical results in plain language to be understood by non-specialist audiences. But this requires, like any writing assignment, some back and forth between students and instructor, something not allowed by the weekly labs set up. Most importantly, weekly labs on various methods, based on exercises unrelated to one another, did not allow for information to sink in, for real knowledge to form, and for students to digest feedback on ideas or writing. It gave the class the feel of a superficial sampler and student evaluations almost systematically pointed out that they thought they didn’t learn one thing well in the class, even when they had good grades. Obviously, something needed to change.The Idea of Gamification EmergesAfter three years trying various ways to make the class work in the traditional weekly topic layout, I resolved that the layout itself was problematic. This resulted from a goal that was somewhat misplaced. Instead of wanting to introduce students to a series of statistical methods, putting an emphasis on content delivery, my primary goal should be to introduce them to the praxis of statistical analysis in GIS, putting an emphasis on engaging students in learning-by-doing experiences.The focus on praxis led to the idea of organizing the course through a workshop/studio approach, using experiential learning to provide a meaningful context to help make things stick, contrary to the “sampler” approach. Instead of chaining unrelated, convenient labs to illustrate pre-selected methods, I decided to create a couple of sustained experiences where students would have to learn and use several statistical methods to tackle real-world problems. The problems would be chosen based on the opportunities they would offer to practice essential statistical methods, making these problems the drivers of the learning experience, not the methods.A difficulty I could anticipate with a workshop-based course was that students working in groups would set up plans for division and coordination of tasks. Rather than this, I wanted each student to intimately grapple with the whole problem at stake. For an applied statistics class I saw this intimate dimension as vital: I wanted every student to be given the opportunity to experience the messiness of data first hand…. and to exert their creativity to find ways to clean, reshape and adapt data to fit analysis needs, while coming to terms with the fact that this sometimes proves impossible and that some questions cannot be answered due to the lack of proper data. Working closely with the problems, each student would need to address the reality that a number of statistical notions are rather counter-intuitive: How come we’d rather use one-tailed tests, which seem to give a less specific information but actually reduce the estimated risk of error? What is the difference between type I and type II errors and how do problems need to be formulated and results interpreted knowing that the most common tests estimate only the type I error? These are examples of notions that students typically struggle with until they have an “aha” moment, which may only come if each student is intimately engaged in the problems. Another difficulty with group work is that students need to keep pace with each other and, from the heterogeneity I had witnessed in previous years, I thought this would just be too discouraging for some and frustrating for others. I wanted to create a learning environment where the pace of each student would be respected.I was in the midst of these reflections when serendipity struck and elements of my several lives collided. I had recently (finally) purchased my first smart-phone and my kids had gotten hooked on Candy Crush. I was struggling with them spending hours trying to advance through outrageously and nonsensically difficult levels of the game and getting happily frustrated about it. At the same period, procrastinating from grading fall semester projects, I stumbled on a blog post about using Moodle (the content management system – CMS – used at the University of Minnesota) to implement gamification strategies (http://z.umn.edu/17gz). As the light bulb turned on I saw how I could actually set up my class to harness the power of gamification – like Candy Crush, and make an environment in which my students would be happy to work on difficult and frustrating applied GIS problems.Indeed, Candy Crush is one example of a “railroad” game, where participants advance from station (level) to station until the end of the line. This is compatible with a workshop approach, which requires that students engage in sustained and authentic activities; it only requires parsing the workshop into elemental stages that constitute the different stations of the game. And this transforms the workshop into an individual experience, where all students are on the same journey but can travel at their own pace. One crucially appealing aspect of the railroad game dynamic is that players get to return to each station again and again until they succeed and journey to the next one. I fancied that it would mean the end of superficial C-level work: students would have to actually learn at a satisfactory level before moving to the next station. Not every student would complete the entire game, but at least everybody would get to learn some things well.Finally, I loved the idea that gamifying the class would almost naturally displace my position from content expert who dispenses knowledge to “game master” who facilitates and coaches, to move from “sage-on-a-stage” to “mentor-from-the-middle” (see Sweet et al., Teaching Applied Creative Thinking, Stillwater, OK: New Forums Press, 2013). While this game would be much more individualized and constrained than a role-game and while teaching stats is not exactly the same as teaching creative thinking, I envisioned that I would be a facilitator and coach to the players/students and that the game would provide ample opportunities for unstructured interactions so that I could model as a GIS professional and a scholar.How I Did ItIf students are to experience the class as a game, it has to look and feel like a game. The trick is to use the class CMS a little differently, showing students a sophisticated and aesthetically pleasing front end only, with links to the actual assignments, quizzes, etc., hidden backstage. I will show how this was implemented in Moodle 2.8 (later transferred without major issue to 3.0) but I expect that the same principles could be transferred to a different CMS. I think the two critical components needed are, in the first place, a completion tracking system to condition the availability of specific items/activities to students having attempted, failed or passed prerequisite activities and, in the second place, a html item type, whose code can be modified by the instructor (“labels” in Moodle). Figure 1: Tabs in the “Onetopic” format of a Moodle courseFirst of all, content in Moodle can be organized into different sections and I chose to use the “Onetopic” format, which displays one section at a time, using tabs for navigation between sections. This format, displayed in Figure 1, makes it easy to actually hide from students the last sections of a course using +/- buttons.When the “ + ” button is clicked, the “Backstage” section becomes visible. When the “ – ” button is clicked, it gets hidden again, but all the elements located in this section are still available, if you know their URLs. This backstage section (Figure 2) contains and hides from the students’ direct view all the activities and resources that make up the course. The core of the game/class are workshops, which I called “Quests”, parsed into several labs, called “levels”. Each quest corresponds to an interactive map in Moodle, as shown in Figure 3.These backstage activities and resources become accessible to students via hyperlinks embedded in the visible elements of the game.There are 6 levels in this first quest and this student has already completed successfully levels 1 and 2 (green circles). They have already turned in level 3, but I was not totally satisfied, gave them feedback and a grade less than 100%. Their submission is automatically re-opened and I will be informed when they submit a new version. The appearance of the map will be updated with a green circle around level 3 once they reach 100% for this level.Technically, these quest levels are “assignments” in Moodle, all hidden from view in the backstage section. As the Figure 3 screenshot shows in the grey highlighted line at the bottom, clicking on level 2 will open assignment #963120. If the student were to click on level 4, they would get a message that particular resource is currently unavailable, as the student had not yet completed level 3.The image also displays pop-ups when hovering over levels, to indicate whether or not they require a piece of theoretical knowledge to be acquired before unlocking.There are 6 levels in this first quest and this student has already completed successfully levels 1 and 2 (green circles). They have already turned in level 3, but I was not totally satisfied, gave them feedback and a grade less than 100%. Their submission is automatically re-opened and I will be informed when they submit a new version. The appearance of the map will be updated with a green circle around level 3 once they reach 100% for this level.Figure 2: All the activities as resources for the game are gathered in a hidden section of the Moodle courseTechnically, these quest levels are “assignments” in Moodle, all hidden from view in the backstage section. As the Figure 3 screenshot shows in the grey highlighted line at the bottom, clicking on level 2 will open assignment #963120. If the student were to click on level 4, they would get a message that particular resource is currently unavailable, as the student had not yet completed level 3.The image also displays pop-ups when hovering over levels, to indicate whether or not they require a piece of theoretical knowledge to be acquired before unlocking.Behind the scenes, the Figure 3 map is in fact made-up of several “label” elements in Moodle. Figure 4 displays how I see things as an instructor.The map is in fact made up of six banners, one for each level, and each banner is repeated in 3 different labels, the only difference being no circle, or a red or green circle around the level. Which labels are visible is controlled by the “activity completion tracking” features of Moodle, with parameters available in the settings for each label. Figure 3: The first “quest map” in Moodle Figure 5 displays the setting page for the “failed” level 1 label.First of all, under the “general” tab, the html view was toggled by clicking on the “” symbol. Each label is in fact a svg (scalable vector graphics format) drawing, initially created in Inkscape. The code is then simply copy-pasted in Moodle, after toggling the html mode. Highlighted in yellow are the lines of code that set up the link to the assignment and the pop-up message.The “restrict access” tab shows that the assignment called “Quest stage 1: putting data together” must be completed with a fail grade for the label to be accessed/visible. The struck out eye icon on the left indicates that the label is totally hidden when not accessible. Figure 4: Instructor view of the first “quest map” in MoodleElements of wisdom are the second most important piece of the game/class. Most quest levels require some new knowledge to be acquired before being unlocked. This is kept to a minimum though, the focus being on learning through actually practicing analyses. Most elements of wisdom are “lesson” activities in Moodle. Lessons allow chaining content and questions, in order to check that basic concepts are understood before letting students move to more advanced notions. But “calculated questions” are not available in Moodle “lessons”, which is why I had to combine a “lesson” and a “quiz” for some elements of wisdom.While I always write at least some contextual information myself, many lessons rely extensively on content from the internet. The idea is to introduce students to various sources of knowledge they will be able to draw from after the class is over, when they need to investigate a new statistical method for instance.Visually, elements of wisdom are displayed using the same principles as for quest maps, with “labels” indicating whether or not each element of wisdom has been attempted yet, or has been attempted but failed or has been attempted and passed. As with quest level assignments, each label links to the corresponding lesson or lesson/quiz combo, which are located in the hidden backstage section. Figure 5: Settings of a “quest map” label in MoodleStudents start the game with a grade of 0 and each activity they complete earn them points. A total of 100 points are possible. Quest levels count for 50 points, elements of wisdom for 30 and the final exam (“Nirvana”, the end of the cycle of quests and elements of wisdom) for 10. To earn more than 90 points and get an A, or to make up for deficiencies, students have to go one step further and earn up to 10 points in “medals”. Medals, like scouting badges, can be earned on various occasions scattered throughout the game. One major way is to help other students: there is a “good deeds repository”, handled through a Google form where students record how much time another student just spent to help them. Some quest levels include an extra task for a medal. Medals can also be earned for posting useful resources on a forum (“loot chest”), for proposing good questions for the final exam, and for anything relevant that occurs or is suggested during the course of the class.What I learnedThe first important lesson I learned from gamifying this class is that, as hoped for, gamification does work as a grit-enhancer and provides students with feedback – and support – as they move through and beyond frustration. I call this the “yeah-factor” effect.Picture a student who just read about a statistical test and the formula to apply who is then asked to turn in a short application exercise. They are not too sure they understood the reading properly but they don’t want to bother too much for a simple exercise and just turn in something they think might work. The instructor grades the exercise: the student did not get it right; they got a poor grade. The student still doesn’t know how to apply the formula, and likely doesn’t like the instructor very much. Now picture a player who just read about a statistical test and the formula to apply who is then asked to play a short application game. They enter their answer. They did not get it right, the computer asks them to try again, with new numbers. Since they have to get it right to advance, they go back to the reading to figure out what they did not understand, or ask help from a peer, try it again and this time it works. They feel great. Or, perhaps, after reading and trying time and time again, they cannot get it right; however, rather than just being frustrated, they ask the game master for help at the next class session. With an encouraging game master, the students are willing to show how they are doing it. The game master offers feedback on their effort and drops a clue. Their eyes brighten, they utter a thank you. As the game master walks away to help another player, she hears a loud and radiant “Yeah!”.Figure 6: Instructor view of the “elements of wisdom” in Moodle Not only does the game help everybody feel good through and beyond frustration, there are additional learning benefits. Repeating application exercises helps notions sink in more deeply. Going back and improving analyses based on mentor feedback until they are of professional quality develops the sense of the craft that is data analysis, of the respect owed to the final audience and of the value of a work well done in general.This might be the most important lesson I learned about teaching a gamified class, somewhat unexpectedly: group dynamics matter a lot, even though the game is played individually.Looking at outcomes more formally, there is a clear trade-off between breadth and depth, between the non-gamified and the gamified version of the class. Fewer statistical methods were covered in the gamified version, but students performed better on the final exam, even though more advanced questions were introduced. The reduction in breadth is even greater if one takes into consideration that not all students complete the game. Feedback from students made it clear that it was easy to slack off without deadlines for the completion of each level. The second time I taught the gamified version, I introduced indicative dates at which each level should be completed if students wanted to stay on track to complete the game, and completion rates improved a little (⅓ of students completed 90% of the game the first year, 3|5 the second year; ⅔ completed 70% of the game the first year, 4|5 the second year).From the instructor’s perspective, teaching a gamified class is different on many levels. First of all, the entire game must be set up before the class starts. Whatever the content management system used, there is a fair amount of tedious work to control which image is displayed and which activities are unlocked after each activity is completed. One could theoretically get away with setting up the shell only before the class starts, and writing the labs and lessons just ahead of the first player. But this means taking a chance on something not working out well at the end. This initial effort is rewarded by a much more relaxed semester, where the traditional hassle of grading is replaced by organic feedback to students. At the end of the day, students will all get to 100% on each quest level in order to advance to the next one. This removes the hassle of coming up with a “fair” grading scheme that works for all. Instead, the instructor can really focus on identifying elements worth improving in the work of each student without restraint to preselected dimensions, as it would be the case in a “real” professional situation. My single preferred thing with this system is that I don’t get stacks of labs to grade all at the same time: after a while, quest level submissions and resubmissions trickle-in rather constantly and I can spend half an hour every morning giving real time feedback to students, rather than cramming half a day of grading every week. In addition to this written feedback on quest level submissions, class time is typically unstructured, spent mostly on mentoring. Mentoring is partially individual, helping students one at a time, but there are a lot of opportunities for sharing as well: sharing individual questions that are of interest for everybody, sharing the experience of players that are further ahead in the game or sharing one’s own experiences related to the methods or the topics studied. The time spent telling those “stories” did feel slightly digressive at the time but, looking back, many student evaluations spontaneously mentioned them as one of their main source of learning in the class, maybe not directly and tightly related to statistics, but on what it means to be a GIS professional in general. Opportunities for sharing do not arise automatically though; a lot depends on the group dynamic.This might be the most important lesson I learned about teaching a gamified class, somewhat unexpectedly: group dynamics matter a lot, even though the game is played individually. I have taught the gamified version of the class twice. The first time 15 students took the class, almost all were GIS majors, almost all had several friends in the class and almost all had already taken two or more of my classes. By contrast, I knew only half of the 12 students who took the class the following year, only half of them were GIS majors and most of them knew nobody or just one person in the class. Adding to that, because of some renovation taking place, the working lab reserved for students who take GIS classes to do their homework was not available the second year. Students had reserved workstations in the library, but it was in a silent space, making collaboration difficult. On top of everything, I caught a bad virus at the beginning of the semester and was not my usual self for nearly two months. As a result of all these circumstances, group cohesiveness never built up. A first symptom was that students hardly helped one another, whereas they had appeared to be constantly helping each other the first year. The ramifications were multiple. First of all, they obviously didn’t benefit from the documented advantages of peer-tutoring. Then, since they were not helping each other, they had to wait for me to help them when they got stuck, leading to frustrating down time. This significantly reduced their gratefulness for the help and the “yeah-factor” effect. Also, this mechanically implied that I had to spend more time providing individual support rather than sharing things with the entire group, which in turn didn’t help group comfort and cohesiveness. All this didn’t have direct damageable effects on the immediate measured outcomes: the final exam results were similar to the previous year and the completion rates were up (due to new indicative deadlines). But this led to a less enjoyable experience for everyone, less side-line learning and a sense of lesser accomplishment – as attested by student evaluations that were a lot less enthusiastic than the previous year (though still better than with the non-gamified version).This lack of group dynamic took me by surprise. I was unprepared and kidded myself for too long that it would eventually all fall together. Looking back, I have identified the factors why it didn’t work out (see above), and I will have a panoply of community building activities and devices at the ready next year, to be used if similar difficulties arise again. I will also work with students to help identify times in their schedules where they could meet and work together outside of class time, not in the library but in a collaboration-friendly space. I think it is ultimately the instructor/game master’s responsibility to build the group cohesiveness, because the spirit of the game is at stake. At the end of the day, gamification is just an illusion, literally a mind-trick that generates dopamine secretion to help students through challenging concepts and tasks. Collective buy-in strengthens the illusion, so that when somebody starts wondering whether the emperor has clothes, everybody else reinforces that he does. If the illusion starts to unravel, then the entire game setup becomes a series of cumbersome convolutions, which probably raises the stress level rather than helps, and fuels resentment.Because gamification ultimately relies on the students’ willingness to be tricked, my final piece of advice is that it should be used with parsimony. I expect that if students were in several gamified classes, their enthusiasm would falter and they would be less willing to “play the game”, or require even more sophisticated setups. Until studies showing otherwise are available, I will personally reserve gamification for these extra-challenging classes where students most need a grit boost.Chapter 3A Case-Based Course on Developing Interprofessional Health CompetenciesKarin Hamilton and Tricia ToddKeywordsinterprofessional, team, health profession, course, case studyHistory and purpose of the courseThe New Health Professions Team course began as a conversation between two instructors – one from the field of public health, and the other from the field of veterinary medicine. The result of the conversation was an agreement that the current common focus on interprofessional teams as strictly teams, of health professionals providing direct patient care in a clinical setting, was limiting. Having worked on complex professional challenges ranging from zoonotic diseases to foodborne outbreaks, the instructors wanted to demonstrate how the skills and competencies of working in interprofessional health teams go beyond the hospital walls.In addition to expanding the scope of interprofessional health teams, the instructors wanted to apply the concepts of interprofessional education to undergraduate student learning. The goal was to help undergraduate students learn about the competencies (skills, attitudes, and behaviors) needed to work effectively in teams made up of diverse professionals striving to solve complex health problems. The challenge was to teach the interprofessional competencies to students before they enrolled in health professional programs.Undergraduates are in a unique situation as they have little to no professional identity or role affiliation. Often, teaching them about interprofessional team dynamics is challenging because their own past experiences do not represent the complexities of an interprofessional team. In addition, the many skills and competencies interprofessional research identifies as vital often fall into the broad and elusive “soft skills” category. With soft skills often best gained through experiences, teaching the competencies we identified required us as instructors to bring simulation, cases, role play, and other interactive activities into undergraduate, professional practice courses.Goals and Learning Objectives of the CourseThe goals of the course were to raise awareness and build interprofessional competency among pre-health students. The original competency domains and learning objectives for year one, Spring 2015, were developed by reviewing the competencies identified in the Interprofessional Education Collaborative (IPEC) Report on Core Competencies for Interprofessional Collaborative Practice (IPEC, 2011) and the University of Minnesota Student Learning and Development Outcomes (UMN SLO, UMN SDO). Due to the narrow focus on patient-centered teams in the IPEC report, the instructors expanded the learning objectives to apply to interdisciplinary teams addressing public health challenges. This resulted in fourteen learning objectives within the four IPEC domains of Values and Ethics for Interprofessional Practice, Roles and Responsibilities, Interprofessional Communication and Team and Teamwork.The following year, Spring, 2016, the objectives were reorganized into content specific objectives and personal growth objectives. The content specific objectives focused on the four IPEC domains, while the personal growth objectives focused on self-assessment and building self-awareness around development of interprofessional competency. This reorganization was done to better align the learning objectives with the course goals and subsequent activities while also making the learning objectives clearer to the students and more measurable.In 2016, an updated edition of the IPEC report (IPEC, 2016) was published with the language sufficiently broadened so that the IPEC domains will now provide the foundation for the 2017 course objectives. imageFigure 1: Students participating in the Marshmallow Challenge activityEvolution of the courseSpring 2015 Class The course was developed in Summer and Fall 2014 and first taught in Spring 2015. The semester was designed to include a mix of theory around interprofessional competencies, real life examples, discussion, and activities for each class. Some examples of the activities used include: 1) practicing giving and receiving feedback; 2) completing the Marshmallow Challenge (Figure 1) (Wujec, 2015); and 3) watching and discussing selected videos demonstrating both good and bad communication skills. In order to connect the interprofessional competencies to teams working in the health professions, sites of the students’ future careers, two realistic disease outbreak case scenarios were created and used throughout the course: one based on rabies and the second on Lyme disease. Students were split into four teams, two of which worked through the rabies case and two of which worked through the Lyme case. The students were assigned to teams with the goal of maximizing team diversity in gender, college, degree program, and health career interest. The instructors provided teams with a weekly update on the progression of their respective case.While the students were learning about the cases, they were doing so in teams and reflecting on team dynamics. Therefore, the assignments focused primarily on (1) self-reflection and evaluation, (2) writing assignments about the interprofessional competencies within the case scenarios, (3) obtaining external feedback on one’s interprofessional competency level, (4) providing peer evaluations of team members, and (5) a final team presentation. For their final team presentations, teams were tasked with describing the stakeholders and their roles and responsibilities with regard to the assigned scenario. They were asked to provide examples in the scenario where interprofessional competencies were demonstrated well and where they could have been improved. They also proposed possible solutions to the case.Early in the semester, the instructors quickly learned how challenging it is to teach the concepts of interprofessional teamwork to pre-professional students. Students typically relied on previous team-based experiences, often in sports settings, to assess their current knowledge and skill levels as a team member. While there are some similarities, the dynamics at play in interprofessional teams are far more nuanced and often linked to roles, responsibilities, and power. An additional challenge the instructors observed was the lack of accurate self-awareness among many undergraduates. Frequently, the students ranked themselves at the highest level of competency in interprofessional skills when in reality most did not have sufficient experience or evidence to support this level of competency. As the interprofessional competencies are less “concrete” than other forms of knowledge, such as descriptions of different types of bacteria, it was also challenging to write the assignment and activity instructions with enough detail to satisfy the students, while leaving enough room to encourage students to think independently and reflect at a deeper level.In this class with pre-health students the cases had a health focus. The intention, however, was to have the students focus more on the interprofessional dynamics versus solving the health problem. To keep students from focusing on solving the case quickly, they were given updates at the end of each week to further unfold the case. This resulted in the cases and related assignments being relatively disconnected from the theory and activities completed in class. Ultimately, there was less interest and engagement in the cases and the final presentations did not reach the level of depth and reflection desired by the instructors.After teaching the first year, the instructors decided to push the course to a new level and applied for an Experiments in Learning Innovations grant. The vision was to create an ongoing, experiential-based case where students would work in interprofessional teams throughout the semester to problem solve, and while doing so, would be made aware of interprofessional challenges within a team, as well as practice strategies to overcome them. The instructors were awarded the grant, and proceeded to initiate the development of their “big, hairy, and audacious goal” (BHAG). And as happens, life intervened. Weddings, surgeries, global travel, and a variety of other situations caused the instructors to tone down the goals for the Spring 2016 class and create a two year implementation plan instead.Spring 2016 Class For the 2016 class, the instructors consulted with the Minnesota Department of Health and decided to modify a pre-existing foodborne disease outbreak training scenario based upon real outbreak experiences (US FDA, 2015). The most significant modification was adapting the case scenario to occur on campus so the students could have a context that also allowed for interactions with local health professionals. The course was broken into four phases:Phase 1 (4 weeks): Interprofessional Competencies In this phase, the students were given interactive class-based activities to learn about concepts such as sharing power, roles and responsibilities, communication, developing group norms, team dynamics, and cultural humility. This phase was similar to the structure from the original Spring 2015 course.Phase 2 (3 weeks): Subject Matter Expert (SME) Teams The students were assigned into uniprofessional SME teams and first introduced to the fictional on-campus foodborne disease outbreak scenario within these teams so that they could initially learn about the roles and responsibilities of the various experts who would be engaged in the case. There were three teams: Hospital/Laboratory, Epidemiology/Public Health, and Environmental Health/Food Safety. As in 2015, the students were assigned to teams with the goal of maximizing team diversity in gender, college, degree program, and health career interest. The first class in this phase started with time for the students to meet their teammates and establish their new teams with the optimistic hope that they would use some of the skills and activities introduced in Phase 1 of the course. However, rather than taking time to team-build, most teams jumped right into the case scenario. When brought to their attention that they skipped right over the processes of team-building identified in the first few weeks of class, they all made the connection and asked to have a second chance.Figure 2: Example of interprofessional team case mapStudents were also introduced to stakeholder and systems mapping and given the option of several free tools (Diagramo, Draw IO, Excel, Lucid) to choose from as they were tasked to create team maps of the outbreak scenario. Each week the teams received SME specific updates for the case which they added to their respective case maps (Figure 2). One additional assignment during this phase was for students to interview a health professional in the same field as each team’s SME to help them develop a better understanding of the roles and responsibilities of that particular health profession. The health professionals were from the community and volunteered their time to talk with the students. During the last class of this phase, the students were surprised with an unannounced press conference. Volunteers role-played as reporters and interviewed the teams on what was happening in the case at that particular time (students had already received communication training and practice in Phase 1). The volunteer reporters created and shared some very interesting headlines from the information they received during the press conference (Figure 3). Phase 2 ended with self and peer evaluation on the interprofessional competencies within their SME teams. Figure 3: Example headlines created by role playing reporters after an unannounced press conference activity (left) and a second announced press conference (right)Phase 3 (4 weeks): Interprofessional Teams At this point in the semester, the students were reassigned to interprofessional teams to continue problem solving the case. An interprofessional team consisted of 1-2 members from each of the previous SME teams. Once again, the students jumped right into the case instead of team building. Again the issue was raised, and the students explained that because they knew each other, they had skipped over the more formal processes of team-building.Each new interprofessional team was challenged to combine the three SME team case maps into one new interprofessional team case map. Again, local health professionals volunteered to meet with the students in this phase. This time volunteers played roles within the case scenario and provided information and updates the interprofessional teams needed to continue mapping and figuring out the case. Another press conference occurred during Phase 3, but this time the teams were given advance notice to prepare a statement and be ready to answer questions. These press conferences were included to help them understand and deal with the public communication necessary when working on a case where the public has a vested interest. Phase 3 ended with the conclusion to the foodborne disease outbreak case scenario.Phase 4 (3 weeks): Wrap Up Throughout the course, students completed individual writing assignments about the interprofessional competencies, incorporating self-reflection and evaluation, and peer evaluation. In addition, each interprofessional team was tasked to reflect on the interprofessional competencies, both as found in the case scenario and as experienced working in their teams throughout the third phase. Finally, each team led the class through a new activity to introduce, practice, and debrief about one of the interprofessional competencies they found to be most important. Several of the health professional volunteers participated in a panel discussion about their real life experiences working in interprofessional teams. The course wrapped up with a summary and debrief about what the students liked and what they would change in the course.Instructor reflections on lessons learned from the courseTeaching Interprofessional Team Skills Whether educating undergraduate students or those in health professions programs, teaching and evaluating interprofessional team skills is challenging. Each semester we have tightened and clarified the learning objectives in an effort to help students understand the focus, purpose, goals, and expected outcomes of the course. A specific challenge that is not unique to undergraduates, but also pertains to professional students and professionals, is the tendency to want to focus on the scientific problem, not the interprofessional challenges. Building intentional interprofessional challenges into the fictional case scenarios has helped students to understand and appreciate the importance of trust, communication, roles, responsibilities and the team-building processes; however, it is an ongoing challenge to find the balance between interest and motivation to solve the case versus learning about and developing interprofessional competencies.Solving the Case vs. Building the Competencies In an effort to make the course more engaging and interactive, we used two variations of multi-week cases that the students needed to solve, all the while paying attention to both the case and the interprofessional challenges. While providing a realistic scenario helped demonstrate the interprofessional team competencies, the students generally lacked the knowledge and experience with their subject matter expertise roles, often leaving them confused and disengaged. Students tended to stay in their student role and wanted to solve the case rather than Instead of taking on the role they were assigned, instead of taking on the role they were assigned, and focusing on the process and team dynamics. This is a normal tendency, even in a real team situation, as often team dynamics are not as clear, or as easy to address as the scientific problem at hand.Real World Perspective As mentioned previously, finding the “right” case that students can relate to without having had significant real world experience in the health field is an ongoing challenge. We started in 2015 with two cases, one focusing on rabies and the other on Lyme disease, which had mixed results. In 2016, we used a fictional on campus foodborne disease outbreak which again had mixed results. Some of the students thoroughly enjoyed learning about and role-playing in different health professionals than they had previously been aware of whereas other students were frustrated by not be allowed to focus on their specific health profession interest. In an effort to address this, the course has changed each semester it has been taught, to provide an engaging realistic case that both fits in students’ future career interest while expanding their experience and perspectives of interprofessional teams working on health challenges.Interacting with Health Professionals To address the lack of familiarity with the professions, we utilized real health professionals as volunteer role players within the case. Our volunteers for 2016 included a physician, state and county epidemiologists, and graduate students from “Team Diarrhea” at the Minnesota Department of Health. The student teams took different approaches to interacting with the health professionals with some meeting in person as a team, others sending one or two student representatives, and some communicating via email or the telephone. The interactions by students who met in person with the health professionals seemed to have had the greatest positive impact both on the students and the volunteers. The volunteers generally enjoyed communicating with the students as they appreciate our interest in building interprofessional competence in future health professionals.Systems Thinking In addition to learning about interprofessional teams, we are helping students understand how systems (e.g. public health systems, healthcare systems) function. Previously, this was a challenge because students had a limited understanding of the various stakeholders needed to solve complex health challenges. Systems thinking requires enough awareness of the parts of the system to really appreciate how they impact each other, and where there will be similar challenges to those found in interprofessional teams. In Spring 2016, the students were taught how to develop a case map. This activity was appreciated some students who saw the map as a useful tool and dreaded by others as a “busy work” task.Self-Reflection The most important goal of the course is to help students learn about and practice the skills, attitudes, behaviors, and competencies needed to work effectively in interprofessional teams. This requires students to spend time reflecting on their own actions and the actions of others. We have built in a significant number of reflection activities throughout the semester. One observation is that students generally lack the ability to reflect with any real depth or even accuracy. Therefore, it is essential to give students sequenced guided reflection questions to help them reflect at a deeper level. In both 2015 and 2016, we found students tended to have an inflated view of their own interprofessional skills. We also learned that students do not like to provide feedback to each other, particularly if it has any sense of criticism. Teaching students how to critique and give feedback is an ongoing challenge and yet an essential skill as a member of an interprofessional team.Incorporating Technology One of the new goals after receiving the Experiments in Learning Innovations grant was to incorporate technology to support the class. We learned a number ways to use Moodle to help provide a clearer course structure using the four phases, and added a new mapping tool that students could use to record case progress, tracking stakeholders, communications, decisions, and so on. While we feel the case mapping is a useful skill, particularly in following an extended case scenario, we learned through reflection assignments and a feedback session that not all of the students shared that opinion. As we move into the next iteration of the class, the plan is to continue to explore technology that can assist the students in the learning process.Assessing Student Learning Student learning has been assessed formally and informally through a variety of methods including writing assignments, self and peer evaluations, self-reflection, team case mapping, final team presentations, and feedback sessions. Grading rubrics were used for formally assessed writing and final project assignments. We continue to address the challenge to effectively articulate the reflection and final team project assignments so that they convey, guide, and support students in developing the depth of critical thinking we desired.Next StepsThe Experiments in Learning Innovations grant offered us the opportunity to think creatively about the course: how to make it more experiential, and how to better integrate interprofessional competency theory with real world experiences. While the Spring 2016 class has shown significant improvements from the Spring 2015 class, we recognize the opportunities for improvement and look forward to adding some additional activities in the third iteration of the course in Spring 2017 based on our experiences and student feedback. The core interprofessional competencies will remain the basis for the course, and each class will continue to feature an activity that engages students in practicing those competencies. To allow for more in-class time for the activities, case studies, discussion and reflection, the classroom so in the sense that students will complete readings or learning activities about the competencies before in-class activities each week. Some of the new learning activities will be set up as interactive online modules created using Articulate Storyline software, and other video resources.Getting the case study “right” has been one of the biggest challenges with this course. We started with two infectious disease case studies that two teams each worked through over the semester in a relatively passive sense. In the second iteration, we used a foodborne disease outbreak case study that required more interaction and integration into the course. For the case components of the Spring 2017 course, we plan to have the students participate in two distinct case modes. One will involve working through weekly cases involving interprofessional health teams, with each case study tied to one of four interprofessional competency domains. We hope having a larger variety of case scenarios will allow students the comfort of thinking about interprofessional teams through their future health profession lens, as well as pushing them to learn about and appreciate the contributions of other health professions they may work with in future careers.Second, student working in teams will problem solve a relevant health challenge that they may encounter on campus. For example, they could choose binge drinking, mental health, or some other topic that is both relevant and requires working with a diverse group of people to problem solve. Having all groups focus on the same challenge will hopefully create some competitive tension resulting in deeper team dynamic experiences. Each group will be tasked to develop a proposal on how to approach the complex health challenge on the campus. The interprofessional competencies will need to be integrated into their approach and proposal. The groups will use the case mapping tool from the Spring 2016 course to identify stakeholders and approach the health challenge with systems thinking. Students will also need to interact with local health professionals, as well as others, while developing the proposal to address the health challenge. We will continue to explore new ways to assess student learning of the interprofessional competencies as well.The initial intention, to raise awareness and develop interprofessional competencies among pre-health profession students, remained the same throughout the evolution of the course. The question of how to best do this has been a constant challenge for us. One factor that keeps us motivated is the recognition that this challenge is not unique to teaching interprofessional competencies to undergraduate students, but is an ongoing challenge among health professionals as well. Feedback from students’ reflections (some sample quotes below) through writing assignments and a feedback session on the final day of class has helped us to recognize that even if the course is not perfect, students are learning and so we continue to push on modifying and teaching this course. The following comments reflect students’ comments overall:Previous to this course, I had considered communication to be active talking and active listening, but it is so much more.I have learned to be more self-aware and self-critical, but have a long way to go to fully develop reflective skills that reveal five levels of why.Because of my experience in this class, I think I will be better outfitted in the real world for interprofessional work as it has helped me understand the hardships and false assumptions that can be made when working with others not in my field of “expertise” per se at least in the scenarios presented to us.Entering this class I thought I already had a fairly decent foundation for working in teams. Throughout school and extracurricular activities I have always worked in teams in some sort of setting. Little did I know I had a weak foundation that needed to be broken down and rebuilt if I wanted to see any measurable improvement in my interprofessional team skills.ReferencesInterprofessional Education Collaborative (IPEC) Expert Panel (2011). Core Competencies for Interprofessional Collaborative Practice: Report of an Expert Panel. Washington, D.C.: Interprofessional Education Collaborative.Interprofessional Education Collaborative (IPEC) (2016). Core Competencies for Interprofessional Collaborative Practice: 2016 update. Washington, DC: Interprofessional Education Collaborative.U.S. Food and Drug Administration (FDA). ‘Stealthy Situation’ of Food Related Emergency Exercise Bundle (FREE-B) http://www.fda.gov/Food/FoodDefense/ToolsEducationalMaterials/ucm295902.htm. Accessed March 15, 2015.University of Minnesota (UMN). Student Development Outcomes (SDO). http://www.sdo.umn.edu/. Accessed March 15, 2015.University of Minnesota (UMN). Student Learning Outcomes (SLO). http://academic.umn.edu/provost/teaching/cesl_loutcomes.html Accessed March 15, 2015.Wujec, Tom. Marshmallow challenge. http://www.tomwujec.com/design-projects/marshmallow-challenge/. Accessed March 15, 2015.AcknowledgementsWe would like to acknowledge and thank the following people for supporting us in this course development and evolution and/or volunteering to interact with our students during the course.Judy Beniak, DNP, MPH, RN, Director, Health Careers CenterChristina Petersen, PhD, Education Program Specialist, Center for Educational Innovation, Office of the Vice Provost for Faculty and Academic AffairsYelena Yan, Instructional Designer, Academic Technology Support Services, Office of Information TechnologyDave Johnson, MPH, Supervisor, Hennepin County EpidemiologyDave Golden, Director of Public Health and Communications, Boynton Health Services, University of MinnesotaCarol Nelson, MD, Physician, Boynton Health Services, University of MinnesotaJoshua Rounds, MPH and Dana Eikmeier, MPH, Epidemiologists, Minnesota Department of HealthTeam Diarrhea students, Minnesota Department of HealthTracey Dutcher, DVM, MS, DACVPM, One Health Coordination Center, US Department of Agriculture: Animal and Plant Health Inspection Service: Veterinary ServicesChapter 4Planning, Developing, and Implementing Nursing Telehealth SimulationMary M. Rowan, Mary Steffes, Carol Flaten, Lori Rhudy and Nima SalehiKeywordssimulation, telehealth, telemedicine, telenursing, nursing education, chronic disease management Telehealth: A New Model of CareNurses entering the workforce in the 21st century face unprecedented challenges. The Affordable Care Act (ACA) has given millions of people access to the healthcare system. Furthermore, the changing demographics in the US are widely understood, with an aging population raising concerns about the cost of healthcare. At the same time, the nursing profession faces significant shortages. According to the American Nurses Association, the average age of employed registered nurses is now 50 years (http://www.nursingworld.org/nursingshortage). Workforce projections point to a 574,000 new RN positions by 2022. The changing access to health care services and an aging population, coupled with the shortage of nurses could spell a crisis. New models of care and the related technological tools are being developed to manage the projected increase in demand for services. One such model is telehealth—a broad term that includes the provision of healthcare when the provider and patient are at different locations.According to the Centers for Disease Control and Prevention, 70% of deaths in the US are due to preventable conditions (www.cdc.gov). Healthcare in the US is undergoing a shift from hospital-based tertiary care to community-based health promotion and disease prevention. This shift has been accelerated by changes in reimbursement policies under the Patient Protection Affordable Care Act of 2010 (42 U.S.C. § 18001). Nurses entering the workforce today must be prepared for a major role in delivering community-based or home-based care as the focus on disease prevention, health promotion and chronic disease management becomes prominent. A significant number of nurses are expected to be engaged in providing chronic disease management that is aimed at reducing the frequency of hospitalization for this population. These nurses must be competent in using telehealth technologies to provide safe, quality nursing care to patients located at a distance from the care provider (e.g., patients in their home). The technology extends the reach of the healthcare professional and allows efficient and effective interaction with people seeking healthcare services. It is an essential component of the new models of care being developed for the 21st century (Benhuri, 2010; Murray, 2013; Sevean, Dampier, Spadoni, Strickland Pilatzke, 2008).Telehealth: What is it?Telehealth is a broad term encompassing everything from tele-Intensive Care Units to remote monitoring of the status of individuals with chronic diseases living in residential facilities or at home. Sometimes communication is between healthcare providers and patients, but telehealth also makes it possible for providers to consult with specialists located elsewhere.Telehealth is real-time two-way, interactive audio and visual communication, including the application of secure video conferencing or store-and-forward technology to provide or support health care delivery. Minnesota has been a leader in adopting policies to enable the use of telehealth technology. Under Minnesota law telehealth is defined as “the delivery of health care services or consultations while the patient is at an originating site and the licensed health care provider is at a distant site” (MN Telemedicine Act 2016). Depending on the types of providers engaged in telehealth, various modifications of the term may be used. For example, telenursing indicates that the provider is a nursing professional interacting with a patient who is located at a distant site while telemedicine might be the preferred term by physicians (Grady, 2014).Telehealth technologies make use of a wide variety of tools and equipment that enable long-distance assessment, clinical decision-making, care coordination, consultation, and communication between patients and healthcare personnel. Current technology allows for very high quality videoconferencing directly with a patient or between providers. Telehealth technology also includes peripherals such as special stethoscopes, otoscopes, ophthalmoscopes, ultrasound probes, exam cameras, and weight scales, to name a few. These specialized peripherals send high quality digital data instantaneously to the provider. The caveat is that all of the technology used for sending or receiving personal health information (PHI) must be compliant with the Health Insurance Portability and Accountability Act of 1996 (HIPAA). Protecting PHI is a key tenet of HIPAA. The complexities of telehealth—everything from using the new technologies to understanding the regulations, call for additions to traditional nursing curricula.Nursing Education for the 21st CenturyThe School of Nursing received funding via the Provost’s initiative Enhancement of Academic Programs Using Digital Technology. One of the goals was to develop a new high-fidelity simulation opportunity, including the development of the necessary supporting content for the curriculum. The planning for the telehealth simulation began in 2013, with a goal of implementing a simulation in 2015.Prior to the beginning of this project, faculty investigated the literature and determined that telehealth competencies for nursing were not well-defined. Furthermore, there was no delineation of the appropriate competencies for various levels of the baccalaureate nursing student (sophomore, junior, or senior). In order to determine competencies, specific telehealth activities that registered nurses currently perform were identified. From this basis, faculty defined core competencies that our pre-licensure nursing students need in order to be able to perform current nursing activities in telehealth as well as predicted future activities.The following activities were identified:Assess a patient’s capacity to use telehealthTrain and support patients and families using telehealth technologiesTriage incoming calls and alarms in telemonitoring centersAnalyze and interpret incoming biometric data derived from self-measurement devicesPerform a home environment assessment remotely using telehealth technologiesPerform a focused physical assessment of a patient in a remote location using telehealth technologiesUse therapeutic communication to educate and support health promotion activities via videoconferenceUse technology to ensure interprofessional coordination of careEvaluate and adjust a patient’s plan of careWe found a gap in our curriculum regarding knowledge, skills, and abilities related to telehealth technology in general. Ultimately, this project facilitated the development of content and learning materials needed for senior nursing students to gain an understanding of telehealth and its use for chronic disease management.Project Planning: Creating a High Fidelity SimulationThe faculty team developed a project plan and began creating the necessary supporting documents for the telehealth simulation. Students had previously completed courses in nursing informatics and public health nursing. They had completed both content and experiential learning with in-person home visits or community-based nursing care. They had also studied content about chronic health conditions and completed clinical experiences with older adults. This previous learning provided a frame for the simulation experience, and we developed more specific telehealth content to enhance the curriculum. The materials we developed to support the telehealth simulation included the following:Project planning guidePreparation materials for students including class content specific to telehealth nursingA case study for use with the simulation (older adult with cardiac failure)A script for the standardized patient actorA script for the student in the role of family member (adult child of the patient)Observer guides for faculty and student peersOrientation guide for preparing students for the 3 roles: observer, family member, visiting nurseObjectives for the “visiting nurse” including guidelines for conducting a teleconference home visitFaculty debriefing guideEvaluation surveyWe explored various types of telehealth equipment currently used for interacting with patients in their homes. After exploring costs, ease of use, and other features of the available equipment we determined that we could simulate the telehealth visit using equipment and services readily available at the university. The WebEx web-conferencing system is HIPAA-compliant and supported by the university and was a cost-effective and reliable option for connecting the student nurse with the standardized patient (actor).Additionally, we planned a schedule and workflow for the event that included an orientation for the learners, the plan for implementing the nurse-patient encounter, and the guide for the post-simulation debriefing. Logistical planning included testing the WebEx for use with the computer in the nurse location and testing computer tablets for use in the in the simulated patient’s home setting. We prepared a plan for using telephone back-up in the event that the WebEx or internet connection failed.Image 1: Student Nurse videoconferencing with Patient in Telenursing Control Center Image 2: Patient Actor videoconferencing with Nurse in Simulated Home SettingImplementationWe implemented the inaugural telehealth simulation with 141 senior nursing students participating. The students were located on the Minneapolis and Rochester campuses. The standardized patient was located in Minneapolis.This simulation event took place in the School of Nursing Bentson Healthy Communities Innovation Center, a 10,000 square foot simulation and laboratory facility. We initiated a teleconference using WebEx and located the student in the role of the public health visiting nurse in a room simulating a telehealth monitoring center. The student sat in front of a computer connected to the internet in order to initiate the web-based videoconference (Image 1).The Bentson Center has two rooms designed to look like a home setting. The standardized patient and a student portraying his adult child were in this room with the computer tablet connected to the internet for access to the web-based video conferencing system (Image 2). In a third room, faculty and student observers watched the videoconference interaction on a computer screen in real-time. We modified the set-up slightly for our students located on our campus in Rochester. In this instance, two students were in the nurse role and interacted with the patient located in Minneapolis via a computer and WebEx. Two students observed on a computer screen from a separate location. The arrangements on both campuses worked equally well.Additional faculty and staff were present to orient students to the process, direct them to the various locations depending on their role, and to the debriefing room following their encounter. Faculty observers facilitated the debriefing session in a meeting space at each campus location.The entire senior nursing class participated in the simulation. Each student had a 15 minute orientation that included time for the students to read through their role instructions. The orientation was followed by a 20 minute encounter between the student nurse(s) and patient. Following the encounter, students were debriefed by faculty for approximately 20-30 minutes. Typically we had 8 students in each debrief session.Our primary goal was for students to learn about telehealth as a viable strategy for working with people who have chronic illnesses. Prior to the encounter, students received simulated data about the patient’s health status (vital signs and weight showing a rapid gain in a short period of time – indicative of decompensation and fluid retention). Based on the information faculty provided about the case, the likely cause of the weight gain was fluid retention related to dietary sodium. The students needed to interpret the data and determine their plan for providing care.Engaging in the simulation helped the students learned about telehealth and the various requirements, including the types of equipment and transmissions that were HIPAA compliant. However, the simulation turned out to be a rich educational experience beyond the primary goal of the activity. For example, students did not anticipate that challenges of talking with patients using lay language or conducting an assessment from a distance.The patient script included prompts from the standardized patient about the recent loss of his wife as well as conflicting statements from the patient and his daughter about the quality of his diet. The patient described the challenges of learning to cook for himself since the loss of his wife. He stated a preference for eating at the local café and described his choices of food (high sodium). The patient also talked about the snacks he preferred (salty snacks). The daughter described her efforts to help by stocking her father’s freezer with prepared dinners (likely high in sodium). Tensions between the father’s self-determination and the daughter’s concern were apparent.These prompts in the script provided openings for students to explore the patient’s situation, assess his level of edema and evaluate his dietary sodium intake. Furthermore, students had the opportunity to use therapeutic communication strategies including expressions of empathy and active listening techniques.The DebriefAn advocacy-inquiry approach is used for debriefing simulations in the School of Nursing (Rudolph, J., Simon, R., Dufresne, R., Raemer, D., 2006). Faculty who observe the simulation serve as the facilitators for the debriefing session. During the typical debrief session a conversation occurs among the group members. The focus is on the exploration of the thinking behind the various actions taken by the student nurses. For example, a faculty member may have observed that the student nurse didn’t respond to a question asked by the patient. The observation would be described along with an inquiry about what the student was thinking at that moment. Often this type of approach uncovers confusion or inaccurate assumptions that the learner holds. This uncovered assumption (mental model) can be corrected by the faculty member leading to new understanding by the learners. The group discussion is designed to encourage deeper reflection about the various choices made by the learners and allow them to self-identify gaps in their knowledge.ResultsThe goal for students to learn about telehealth as a viable strategy for working with people who have chronic illnesses was met. The students improved their understanding of telehealth as a strategy for patient care. They became familiar with the various technical and legal requirements, including the types of equipment and transmissions that were HIPAA compliant. They discovered areas for improvement, such as how to communicate effectively.The patient scenario provided multiple opportunities for students to demonstrate their ability to use professional, therapeutic communication techniques. Students also learned new ways of beginning and ending an interaction with a client. Students were challenged to stay focused on the patient via teleconference and realized that strategies needed to convey active listening and ask clarifying or relevant questions when communicating with a patient via computer are quite different from communication strategies used during in-person encounters.Ninety percent of the students who completed the post-simulation evaluation (128 evaluations) indicated that they had significantly increased their awareness of telehealth as a method to provide health care. Eighty-seven percent said that the simulation significantly increased their awareness of the need to use plain language to teach patients. Overall, students received first-hand experience with the challenges of interacting with patients located at a distance from the nurse. In addition, the debriefing provided the opportunity for students to correct their mental models and increase their understanding of disease management while properly protecting personal health information. We repeated this simulation in the spring of 2016 with graduating seniors and had similar positive results.SummaryAn exploration of the nursing curriculum revealed a gap regarding telenursing as a viable strategy for the purpose of chronic disease management. Competencies for nursing students were identified and content was developed for existing courses. A high-fidelity simulation was designed to add an experiential component to the telehealth content in the curriculum. The simulated telehealth encounter engaged student nurses with a challenge to assess a patient located at a distance. Student attempted to interpret data, plan care, and implement appropriate strategies to maintain the health of a patient living with heart failure. The debriefing sessions led to new insights for the students.Using available technology supported by the university made the simulation feasible and cost effective. The goals of the project were achieved. Lessons learned from the first implementation led to modifications of the simulation that streamlined some elements of the process, which is now included in the curriculum. We believe that this simulation was an important addition to the curriculum and helped to fill a gap that has been identified between typical nursing curricula and the experience of nurses in the field who are using these telehealth approaches to efficiently and effectively provide care to a variety of patients.ReferencesBenhuri, G. 2010. Teaching Community Telenursing with Simulation. Clinical Simulation in Nursing, 6(4), pp. e161-e163. doi:10.1016/j.ecns.2009.11.011Grady, J. 2014. Telehealth: A Case Study in Disruptive Innovation. American Journal of Nursing, 114(4).Minnesota Telemedicine Act. 2016. MN Statute 62A.671 Definitions § Subd 9.Murray, T. A. 2013. Innovations in Nursing Education: The State of the Art. Journal of Nursing Regulation, 3(4), pp. 25-31.Rudolph, J., Simon, R., Dufresne, R., Raemer, D. 2006. There’s no Such Thing as “Nonjudgmental” Debriefing: A Theory and Method for Debriefing with Good Judgment. Simulation in Healthcare, 1(1), pp. 49-55.Sevean, P., Dampier, S., Spadoni M., Strickland, S., Pilatzke, S. September (2008). Bridging the Distance: Educating Nurses for Telehealth Practice. The Journal of Continuing Education in Nursing, 39(9), pp. 413-8AcknowledgementsWe thank the University of Minnesota Provost for the 2013 Enhancement of Academic Programs Using Digital Technology grants program that funded this project.We would like to acknowledge Jeanne Pfeiffer, DNP, MPH, RN, CIC, FAAN, Clinical Associate Professor, University of Minnesota School of Nursing, for her contributions regarding the development of telehealth competencies for prelicensure nursing students.PART IIBuilding New Course StructuresChapter 5Building Understanding and Appreciation: Collaborative Work in General ChemistryBrian D. GuteKeywordsflipped classroom, active learning, student teams, general chemistry Challenges of Second-Semester General ChemistryDuring my first semester of teaching at the University of Minnesota Duluth (UMD), I had the opportunity (some might call it a challenge) to teach both General Chemistry I (Chem1) and General Chemistry II (Chem2), the first and second semesters of the foundational chemistry course for science majors. As the semester progressed, I noticed that students in Chem2 struggled much more than the students in Chem1. When I mentioned this to some colleagues, they rationalized it as the difference between the “on-cycle” students in Chem1 and the “off-cycle” students in Chem2. After teaching the second-semester course multiple times, I recognize that in general there are differences in the preparation of so-called “on-cycle” and “off-cycle” students, however, the greater difference is in the complexity of the course content and the level of exposure that students already have to the material. A significant amount of material in Chem1 is covered in most high school chemistry courses, while the majority of students have never seen the material in Chem2. One unintended consequence is that many students who do well in the first-semester course rely in some part on their previous knowledge and do not learn the college study skills that they will need to succeed in more advanced courses.Many students struggle in Chem2 because they are faced with completely new material, more challenging math (more complex algebra than needed in Chem1), and a faster pace than what they are used to when covering new material. As they do not realize that their study skills are not up to the task at hand, students find themselves struggling with multiple challenging factors all at the same time.As I considered these issues, it became clear to me that if I wanted students to be more successful in Chem2, I needed to find ways to better support their learning, to help them develop better study skills, and begin to learn on their own. Over the course of the next five years, I read some of the scholarly literature on teaching, especially articles focused on chemistry education and active learning (Abraham, 2005; Bretz, 2005; Byers Eilks, 2009; Cooper, 2005; Cracolice, 2005, 2009; Eberlein et al, 2008; Eilks et al, 2009; Farrell et al, 1999; Floriano et al, 2009; Gallagher-Bolos Smithenry, 2004; Geiger et al, 2009; Greenbowe Hand, 2005; Moog Farrell, 2011; Moog et al, 2009; Spencer, 1999), and I experimented with a number of techniques designed to improve student engagement and success.Could Flipping Address the Problem?Ultimately, I wanted to change the way that I taught large lecture courses. While I enjoy lecturing, and feel energized after delivering what (in my mind) is a particularly engaging lecture, I am just as often disheartened by the questions I get after such “exceptional” lectures. It quickly becomes evident that my message was not as well received as I had imagined and that some of my students are just as confused about the material as they were at the beginning of the class period. I want my students to understand the fundamental chemical principles that we are covering and I want them to succeed in my class, but obviously I am not always getting through to them. Also, I believe that the human connection between the instructor and the student is an important component of motivation and learning, large lectures make it difficult to develop that connection. In large lecture settings, students become nameless faces amongst a sea of more nameless faces. I became convinced that if I was going to improve the experience for my students, I needed some way to connect with them as individuals. That connection would also provide me with a lens into their learning, giving me a clearer picture of what they understood and where they were still struggling.In my efforts to innovate, I tried increasing the number of chemical demonstrations that I did in class, using active learning activities in their discussion sections and active learning techniques in lecture, including a student response system. While these changes helped, I did not see significant overall improvements. Often, the activities seemed to create more of a division within the class, helping the students who were already succeeding while doing little to nothing for the struggling students.When I first heard about “flipped classes” I, like many of my colleagues, was skeptical. It sounded like the way college classes are supposed to work. Students come to class prepared, having read ahead in their books, and class time focuses on points of confusion and the more challenging aspects of the material. However, like other cynical academics, I am fairly certain that the majority of my students do not come to class prepared, and as a result, I feel compelled to cover all of the material since I know they are not covering it on their own.Furthermore, I had a hard time picturing what a flipped class would look like. I had taken classes in college that were largely discussion-based, or spent equal amounts of time on lecture (to fill in needed background material) and discussion, but the vast majority of my experience with science courses was in a purely lecture format. I simply did not have the proper frame of reference to understand what a flipped class would look like and what we would do during class. At the same time there was something about the inclusion of technology in the learning process and the promise of more time to interact directly with my students that intrigued me enough that I started to delve deeper into the literature (and other resources) on flipped classes (Bergmann Sams, 2012; Berrett, 2012; Bishop Verleger, 2013; Educause Learning Initiative, 2012; Gimbar, 2011; Mangan, 2013; Musallam, 2011a-d; Neshyba, 2013; Smith, 2013; Sowash, 2012; Talbert, 2014a, 2014b).In addition to my research, I participated in a faculty learning community on flipped classes and tried my hand at designing my own flipped lessons. I also experimented with process-oriented guided inquiry learning (POGIL) activities (Moog Farrell, 2011) to get a sense of how students could spend class time constructing their understanding of the course content. For several years, my students were reluctant guinea pigs as I worked to better understand the ins-and-outs of the flipped classroom.What I learned quickly was that the “flip” flopped if the instructor was not successful in getting the students to “buy-in” to this instructional approach, failed to hold the students accountable for the pre-class work (which is achieved by having some accountability measure or by reviewing the pre-class work at the beginning of class, thus inviting students to come prepared), and did not conduct the “flipped” lessons on a regular schedule, or as the standard approach to the course. To my mind, this last point was the most important. Flipped activities are very different from what students are familiar with and if they happen to infrequently, the students never get used to doing them. Every time you introduce another flipped lesson, there are challenges with getting the students to remember to do the pre-work and with getting them to willingly work in small teams. These challenges go away as soon as the flipped lessons begin to happen on a regular schedule and students know to plan for them.I also devoted some of my time to seeking out “good” chemistry videos online, which was really an exercise in identifying what I did and did not like in “content-delivery” videos. There are many (in my opinion) really bad videos on Youtube and elsewhere, but I also found some gems with high production quality. Some of the best videos for general chemistry are the Crash Course Chemistry videos (Green, 2014) and the Khan Academy Chemistry videos (www.khanacademy.org/science/chemistry), though many of the Khan Academy videos are longer than I would like. Additionally, Nivaldo Tro has created some “Key Concept Videos” in support of his textbooks. Tro designed the videos to target key concepts that students tend to struggle with and they incorporate a final concept check question at the end. The Key Concept Videos can be accessed through Tro’s publisher, Pearson Education.While exploring the flipped classroom approach, I also spent some of my time learning about the technologies that were available to help educators create videos. One of the most useful resources I stumbled across was a series of Youtube videos by Ramsey Musallam (2011a-d). These videos record a Youtube Teacher’s Studio course that he taught focused on the educational theory behind flipped teaching and the tools available to create videos in support of flipped teaching.What I also learned was that preparing flipped lessons was time consuming. While one could simply record lectures based on existing lecture slides, the overwhelming message in the literature is that video “lectures” for flipped classes need to be short and to the point (Bergmann Sams, 2012; Guo et al, 2014; Mangan, 2013; Mills, 1977; Musallam, 2011a-d; Pettis, 2015; Schwartz et al, 2009; Smith, 2013; Sowash, 2012). For me, this meant significantly reworking my existing lecture notes to make short segments emphasizing a specific concept or application in order to keep the videos at reasonable lengths (not more than 10–12 minutes, but preferably 5–8 minutes). Also, I needed activities to engage students with the concepts during class time for every flipped class period. Based on my experiences, I knew that if I was going to take the plunge, I wanted to flip the entire class rather than slowly work to incrementally build flipped materials. To make that happen, I would need some support.At this point, I was not sure how I would manage the work involved in flipping the entire course, until several opportunities converged. I had already leveraged some small funding opportunities in my efforts to experiment with the approach and had acquired a writing capture tablet, a webcam for my desktop computer, a headset with microphone, and copies of Camtasia Studio and SnagIt software from TechSmith. These tools gave me better control over the quality (and editing) of my videos than the various free video creation tools and made it possible for me to create tutorial videos showing students how to work through mathematical problems using the writing capture tablet. Next, the new Dean of the Swenson College of Science and Engineering (SCSE) at UMD was promoting a shift within the college to widespread use of active learning techniques. And finally, a University of Minnesota system-wide call for Experiments in Learning Innovations (ELI) proposals came out from the Center for Educational Innovation (CEI) on the Twin Cities campus.Making It HappenAt this point, I had all of the necessary tools to create “content delivery” videos for the course. The remaining challenges were to identify and create all of the necessary videos, organize my Moodle course site to support the flipped class approach, and develop the assessments that would hold students responsible for watching the videos in advance of class as well as the in-class activities that would replace lecture. Looking at this “short” list, I realized just what a monumental undertaking it would be to create a fully flipped course. In order to pull it off, I needed support from people who were well-versed in educational methods and the flipped classroom approach. The ELI request for proposals offered just that, access to educational experts from CEI and to educational technologists from the University’s Academic Technology Support Services (ATSS). With a successful proposal I might even have access to the resources necessary to create higher production quality videos than I would otherwise be able to make on my own.In order to be competitive for the funding (and resources) from CEI, I proposed to do something that I had not seen in the literature, a direct comparison of lecture versus the flipped approach. This presented me with three challenges: traditionally, my Department only offers a single section of Chem2 in the fall; I wanted to make sure that the two sections I taught were roughly of the same size; and I wanted to teach the flipped class in a space with movable desks or tables to better facilitate team work. While my proposal was successful, I had some real challenges moving my vision for the project forward. UMD does not have many large classrooms that are not large lecture halls and rooms for the fall and spring had already been scheduled two months before it was announced that my project had been funded. This meant scrambling for whatever spaces might be available for the fall and ultimately limited the flipped class to a maximum enrollment of 54 students, while the lecture section enrollment was essentially unlimited since it was scheduled in a large lecture hall. Additionally, the creation of extra sections of the course in both the fall and spring semesters impacted my department’s budget and those costs had to be mitigated with funds from my budget for the project. That said, the ELI grant made the project possible, and provided me with a dedicated project manager to help me plan, identify available resources, and organize my work.At this point, what I lacked was a plan for the overall structure of a totally flipped class. I knew that the “experts” all said that students need to be held accountable for doing the pre-class work, but how would I do that? And, how would I connect the accountability measure with the videos to create “packaged” pre-class work assignments that could be accessed on a single page of the LMS? These were all questions that I started to explore with my project manager as I started working on developing the course.Figure 1: Example of a pre-class work (PCW) activity within the course Moodle site including learning goals, the alternate assigned reading, two embedded videos hosted on Youtube.Designing and Managing Pre-Class ActivitiesWith the guidance of my project manager, I started by writing specific student learning outcomes for each of the chapters to be covered and identifying topics that would require content videos. This daunting task led me to write 167 specific student learning outcomes and identify 63 unique topics that required videos. With that first task completed, I started to look for existing resources and identified 34 videos on Youtube and Khan Academy, leaving a minimum of 29 videos that I needed to create for the class. However, once I was done, I had a grand total of 65 videos, 31 of which I created using PowerPoint, a Surface Pro 3 (that I purchased to make writing capture easier), and Camtasia Studio. These new videos are hosted privately on Youtube and accessed through individual pre-class activities pages on the Moodle course site. The course site was rounded out with an additional five videos on relevant “topics of interest” intended to show students interesting applications of the concepts we were covering and three chemistry infographics from Compound Interest (www.compoundchem.com).Figure 2: An example of the GoogleForm serving as a review quiz.One of my goals in creating the pre-class activities (referred to as pre-class work or PCWs in class) was for all components of an activity to be available on the same page so students could see everything they needed to do to complete the assignment at a glance. After some discussion with my project manager, we agreed that each PCW should include a list of the student learning outcomes for the activity, the assigned videos (or the relevant pages within the textbook for students who preferred reading to watching videos), and a follow-up quiz to check for conceptual understanding and completion of the assignment. Because I wanted students to be able to access all three of these components for a given PCW in one place, each PCW resides on its own page in Moodle. This also meant that I could not use the quiz tool in Moodle since I could not create a quiz on a subpage and the quiz activity would take the students away from the page with the learning goals and the videos. Figure 1 presents an example of a PCW, with a list of specific student learning outcomes for the activity, a list of optional resources within the textbook that students could use (rather than the videos), embedded videos, and an embedded GoogleForm used as the quiz. I should mention that during the first two semesters of the flipped class, GoogleForms were just “dumb” surveys that collected data in a spreadsheet. I had to work within the spreadsheet to score the quiz after the deadline. As of late spring or early summer of 2016, a quizzing feature became available as part of the GoogleForm and made these GoogleForms much more useful for my class. Not only is the quiz self-scoring, but I can also set the quiz to show students which questions they answered incorrectly. In total, I created 38 PCWs for the fall semester of 2015, with some minor modifications for spring semester due to a textbook change.Designing and Delivering In-Class ActivitiesI have to admit, when I proposed this project (and even into the first few months of work) I had not given much thought to what we would do in class, I was mostly focused on establishing learning outcomes for each of the chapters and the need to identify and create content videos. When I had given some thought to in-class activities, it was in very general terms. Further into the project it came time to start working on these activities, which I call in-class work activities (ICWs), and I really needed to do some thinking. While I dislike “worksheets” in college-level classes, I needed to give all of the students a set of problems/activities to spend time on in class.It was during this stage of the project while I was discussing this issue with my project manager that I had the opportunity to sit in on Dr. Michelle Driesen’s flipped Chem2 course. Dr. Driesen’s students received their in-class activities in the form of electronic “worksheets” that they had to complete and that were graded on a regular basis. In all honesty, I was trying to avoid creating more work that would need to be graded (I was already planning to have graded homework), but quickly recognized that my students would need some incentive to complete the ICWs. After all, the whole point of the activities was for students to spend the time in class applying what they had learned from the videos while a teaching assistant and I were present to coach them through the work.I needed a way to provide my students with these activities. I did not want to print out a new stack of activities for every class and I certainly did not want to regularly collect, grade, and redistribute a stack assignments. I needed a way to distribute and collect the activities electronically, and even grade and comment on the assignments electronically. I also wanted to have my students work with data and practice creating and interpreting graphs. I also wanted to be able to include images in the activities to promote visual thinking.I had attended a few workshops and community of practice meetings at UMD where other faculty talked about using GoogleDocs to work collaboratively. All of our students have access to GoogleDrive and GoogleDocs, so using these electronic resources would only require some additional training and no additional cost for my students. At this point, my concept of working collaboratively in a GoogleDoc was still naïve (at best), but I knew that there was potential. Also, I could include full-color images in the GoogleDocs and if I shared the documents with my students, there would not be any reason to print the images/activities in color. Additionally, GoogleSheets would allow students to manipulate tabulated data and create graphs. This seemed like a solution that met all of my needs for the ICWs.Each ICW is designed to accompany one of the pre-class assignments and the specific set of the student learning objectives for that PCW. Roughly 30-40% of each ICW focuses on conceptual material designed to help students build familiarity with the concepts and practice using the chemistry specific terminology to build their vocabulary. Generally, these conceptual sections then lead into specific applications, either creating or interpreting graphical representations of chemical data or practicing specific calculations that are central to applying or making predictions relevant to these concepts. One of my goals was to focus student attention on the foundational skills that they needed to be successful in the course, with a few more challenging application problems that often required them to work together and push their understanding a little further.I initially started designing and creating ICWs with the assumption that we would need one for every day of class and I persisted in this delusional thought until my first set of students convinced me that some of my activities were far too long for a single 50-minute class period. At that point, I started to adjust my existing activities and design new ones based on the number of concepts to be covered per chapter and the depth to which we needed to cover any particular concept. All in all, this resulted in the creation of 40 in-class work activities for fall 2015.Figure 2: Example of the introductory section of an in-class work (ICW) for General Chemistry II.To help students keep things in context, each ICW was titled and included the chapter and specific sections covered at the top of the activity. Additionally, each ICW includes a set of standard instructions, a table for including the names of the team members actively contributing to the work on the activity, and specific student learning goals related to the pre-class work and the in-class work (as shown in Figure 2).Additionally, each ICW was divided into parts that covered specific concepts and those parts included three types of color-coded activities: Questions, Tasks, and Problems.Figure 3 presents an example of a problem from an ICW. Note that problems were always color coded red and involved calculations. For every calculation problem, I included the numerical answer so students could quickly check their work to see if they had done the problem correctly. This was extremely helpful as it cut down on the number of questions about the “right” answer and allowed my teaching assistant and me to focus on helping students who were stuck or had made some mistake in their calculations. Questions (color coded purple) asked students to answer a specific conceptual question or provide a definition for an important term or concept. Finally, Tasks (color coded green) required students to do something else, such as graphing data points provided in the exercise and making specific observations related to the shape of the graph, or sketching and labeling a diagram illustrating a chemical process or important chemical apparatus.One of the things I was most concerned about with online activities was having students show their work. After all, they should have a record of their work that they could return to for studying and since I was providing the answers to the problems, I wanted to emphasize the importance of the process of reaching the answer over simply the correct numerical value. The use of GoogleDocs greatly simplified this issue. Students could take a picture of their work using a cell phone or their laptop’s camera and embed the image in the GoogleDoc. And similarly, they could create graphs in GoogleSheets and copy and paste their work into the GoogleDoc.Figure 3: Example of a problem included in one of the in-class work (ICW) activity for General Chemistry II.Successful implementation of the ICWs required trial and error. I had decided that students would work in teams of 3–4 and turn in a single copy of the ICW for their team. I initially hoped that I could give everyone access to a single copy of the GoogleDoc and have them all make their own copy, which was a huge disaster on the first day of class. Everyone, at essentially the same time, created a new file called “Copy of …” and almost everyone started working in that same document. It became clear that I needed to create a copy for each team. An ICW link from the Moodle site led to a shared GoogleDrive folder specific to that ICW, and the folder contained one copy of the ICW for each team. It might have been an inelegant solution, but it worked.Who Needs to Manage Team Work? Apparently, I Do.When I set out to flip Chem2, I knew that I had a lot of work to do to create all of the materials for the course. However, until I started creating the ICWs, I had not spent much time thinking about how I would organize class time. Similarly, I had not spent much time thinking about what would need to be graded, about whether students would work individually or in teams, or about classroom dynamics. Perhaps, subconsciously, I hoped that those issues would all take care of themselves while I was busy creating content for the course.As I mentioned above, I eventually decided that students would work in teams. I made this choice for several reasons: 1) it would reduce the amount of grading that a teaching assistant or I would need to do; 2) students would be able to help each other learn the material; and 3) I was aware of some of the literature on the benefits of cooperative learning (Cooper, 2005; Eilks et al, 2009; Geiger et al, 2009). However, with so much of my attention focused on developing the materials and getting ready for my comparative study, I approached the creation of teams as a triviality and let my students pick their own teams. This had mixed results and the chaos of the first semester was heightened by the dysfunctionality of some of the teams. While some new friendships were formed that semester, I think others ended horribly.When I mentioned this to my project manager, she was anything but surprised and simply asked me how many teams I had in total and how many of them were truly dysfunctional (in my lay assessment). When I told her that I had two truly dysfunctional teams (and perhaps one slightly dysfunctional team) out of eighteen total teams, she just looked at me and said, straight-faced, “That sounds about right.” Apparently, it is common for approximately 10% of teams to be dysfunctional. While that might be an accurate statistic, it did nothing to account for the fact that students in those teams were miserable and dealing with that drama, in addition to everything else was not doing much for me either.That said, the class as a whole refused to change teams part of the way through the semester. And, in spite of the issues and the drama, in general the teams worked well together. One of the stranger dynamics that I witnessed during the first semester had to do with team-to-team interactions. We began the semester in a fairly traditional classroom with rectangular tables, and while there was a “front” to the room, I rarely used it as such. However, the students all sat facing the “front” of the room with team members sitting next to each other. While there were often members of two teams sitting side by side, there was rarely any cross talk between teams. Later in the semester, the class was moved into an “active learning” classroom with round tables that seated nine students each. Suddenly, two or three teams were forced to sit at the same table. The round tables allowed for easier interactions and the teams started to work collaboratively. This new collaboration between teams meant that students were helping each other even more, and my teaching assistant and I only needed to step in when the entire table misunderstood a concept or when the ICW problems became really challenging.My teaching assistant noticed other interactions taking shape during the first semester. Always in the classroom before me, since my Chem2 lecture class was across campus and scheduled immediately before the flipped class, she noticed that most of the students arrived early and almost immediately started talking about chemistry, usually focusing on the videos or the ICW, but occasionally they were discussing how the class concepts related to other topics that interested them. These same interactions were observed during the second semester of the flipped class by a different teaching assistant. Going back to the team dynamics problem, obviously, I needed to do something with future classes to improve the team work experience for everyone. I once again turned to my project manager for advice, along with other colleagues who regularly use team work in their classes. They in turn gave me some useful suggestions and pointed me toward more of the literature on team work (Cooper, 2005; Dasgupta et al, 2015; Eilks et al, 2009; Geiger et al, 2009; Layton et al, 2010; Loughry et al, 2007; Loughry et al, 2014; Oakley et al, 2004; Ohland et al, 2012). I learned that: 1) I should be creating the teams, not allowing students to form their own teams; 2) long-term teams should be formed carefully, taking specific factors into account; 3) team work, when done well in STEM courses, can greatly benefit students from underrepresented groups (Dasgupta et al, 2015); and 4) ultimately the functioning of a team comes down to everyone agreeing to a set of shared expectations (Oakley et al, 2004).This was all fairly overwhelming until I was pointed to the team formation tool (CATME Team-Maker), the peer evaluation tool (CATME Peer Evaluation) on the CATME SMARTER Teamwork website (www.catme.org), and the social contract creation resources within the manuscript by Oakley et al (2004). I implemented these tools in spring 2016 and had much better success with the teams. CATME Team-Maker helped me to form teams in which female students and students from other underrepresented groups were paired-up within the teams of three and to form teams with a range of academic levels (based on reported GPA and grade in the first-semester General Chemistry course). Once the teams were formed, each team was required to create a set of guiding principles that everyone in the team agreed to (a social contract) and to keep a copy for themselves and provide me with a copy as well. During this second offering of the flipped class, I only had issues with one team. That team had a late addition who was not present to participate in the creation of the team’s social contract and who was most likely was never told about the team contract by the other members of the team.The current state of the flipped classWe are now well into the third iteration of Chem2 as a flipped class and the class has changed quite a bit. The first two semesters were relatively small sections, with 49 and 26 students respectively. Issues with the registration system and miscommunication about the course significantly limited enrollment in the spring of 2016 and marginalized my ability to get statistically meaningful results. Two flipped sections were offered in the fall of 2016 with initial enrollments of 90 students per section.One challenge for me was that I needed to submit a proposal on moving forward with the flipped class approach during the 2016-17 academic year in November of 2015 – before I had any results from my study whatsoever. However, I had anecdotal information from discussions with the students and some practical experience with the class. One message my students sent me loud and clear was that the class periods needed to be longer. Several of the teams commented that they often felt like their team was just starting to “get it” and make really progress on an ICW when it was time to pack-up for the day.In response to these comments, I proposed that we change the course from 50-minutes five times per week (four lecture periods and one discussion period) to 90-minutes twice per week, thinking that a 90-minute class period would give students plenty of time to work and would even make it possible to take a break during the middle of class. The 90-minute proposal was quickly shot down as a non-standard scheduling request and I was given the options for reducing class meetings of three 50-minute class periods per week or two 75-minute periods per week, which meant that I would effectively lose 100 minutes of class time per week.My counter proposal was that I would make the two 75-minute class periods work so long as exams were conducted outside of normal class time, so that we did not lose even more instructional time throughout the semester. We settled on this schedule for the course, which meant that I needed to retool all of the activities to fit a two-day per week model instead of the five-days per week the class started with. And even though I have given up significant class time, I think it was worth it. The two-day per week model allows for a much more predictable schedule of assignments, quizzes, and exams, and students are more willing to finish their ICWs outside of class if they run out of time since they only come to class two days per week.Team creation using CATME Team-Maker appears to work well and the team contracts have really helped to improve interactions within the teams. I know that in my section this fall (2016), out of 32 teams I have two or three teams that have some issues, but the issues are much less severe than those encountered during the fall semester and the students seem to have worked the issues out largely on their own. Oddly, from my perspective there have been more issues with students coming late to class or leaving early, but any time I have asked about it, the other team members tell me they know about the absence and have already had a conversation about how the missing member will contribute to the team’s work on that day’s ICW.Of course, retooling the course to a two-day per week schedule meant combining or reconfiguring many of the PCWs and ICWs. I have also continued to create more videos and refine my existing videos, while also incorporating more “topic of interest” videos and additional infographics from Compound Interest (www.compoundchem.com). Table 1 summarizes how the number of “activities” associated with the course has changed over the course of three semesters.Table 1: Summary of the number of activities, videos and infographics associated with the flipped course over three successive semesters. Fall 2015Spring 2016Fall 2016Pre-class works (PCWs)383427-Videos Created313551-Other videos333641In-class works (ICWs)403427“Topic of interest” videos5915Infographs377Table 1 summarizes how the number of “activities” associated with the course has changed over the course of three semesters.Another substantive change has been the implementation of structured classroom facilitation and more standardized large group clarifications. One of the challenges I ran into during the first two semesters was keeping the teams “on track.” Some teams worked very efficiently, while others were not nearly as good at managing their time. Another challenge was that many student teams were uncomfortable with the open-ended conceptual questions in the ICWs and they sometimes blamed their poor performance on the conceptual exam questions on the fact that they never got the correct answers to those questions. Borrowing ideas from a workshop on facilitating POGIL activities (Schneider and Lovitt, 2016), I created short sets of slides for each class period that included how much time the teams would get to work on each section of the ICW and some follow-up questions for us to review as a class. I also added a final slide with daily reminders about activity deadlines. In addition to creating a more structured feel to the 75-minute class period and conveying to students how much time I thought they should need to complete each portion of the class activity, it helped to standardize what we were doing in class every day. This was especially important during the fall 2016 semester when I shared the teaching responsibility for the flipped class with a colleague. We were each teaching our own section of flipped Chem2, but the structured approach helped us to ensure that the student experience was essentially the same between both of the sections.Lesson’s LearnedFlipping takes a lot of work. There is a lot of work involved in preparing for a semester-long, fully-flipped class. The prep work for this project required quite a bit of time during the preceding spring semester and summer and still required a significant amount of work during fall semester. Completely flipping your course all at once is not for the faint of heart.Flipped classes are fun. While it was a lot of work, teaching the flipped class was the most fun I have had in my eight years of teaching. Instead of lecturing to students, I spent my time talking with students and answering their questions. It seemed like any time the work was starting to get to me, something happened in class that made my day and reminded me why I was putting so much work into this new class format.Flipped classes can be fun for everyone. It’s a much less formal setting, students are constantly learning from each other, catching mistakes and misconceptions early on, and they are more comfortable asking each other and me for help when they get stuck. And, stepping into the classroom five minutes before class to find students already talking about chemistry is one of the coolest things I’ve experienced during my teaching career. It is really encouraging to see how engaged the students become with the material.Clear communication is critical. Of course, there were learning opportunities for all of us. One of the most important lessons that I learned was the need to clearly communicate course expectations to the students. They are much more receptive to a new style of teaching if they have a clear understanding of what to expect from the very beginning. I was not as clear as I could have been with students in the fall 2015 section, and while they were good sports, they made it clear to me that they felt I had not been completely upfront with them about how class was going to work. Taking that lesson to heart, I spent more time talking to the students in the following two semesters and in each of those semesters, everyone had a much better understanding of what to expect from the class and as a result, I have had much less pushback from students related to the course format.Less is more. By the end of the first month, it was also clear that five class periods per week with pre-class work (PCW), in-class work (ICW), and follow-up homework assignments for every day, or even most days, was too much for everyone to keep up with, including me. While I could not change the class schedule during the first two semesters, I did start to revise my approach to activities throughout the semester. I moved to fewer, but somewhat longer pre-class assignments and in-class activities designed to span several days. This reduced the number of days per week that students had required activities to complete outside of class.This changed even more drastically in fall 2016 when the course switched to two days per week. The fact that there was a PCW and ICW for every class period and an associated homework assignment created a routine that students could plan around. It also removed any of the uncertainty from the first two semesters about when the next PCW or ICW was due.Successful team work requires upfront preparation. Team dynamics need to be managed properly. I made some mistakes the first semester with how teams were formed and in not providing students with more guidance in setting team expectations. Giving students some choice, but creating assigned teams in a purposeful fashion in spring 2016 and fall 2016 led to much better results. Implementing team-developed social contracts for team expectations has also made a great improvement.Relax and be flexible. Nothing is ever perfect the first time and most likely it still is not perfect the second (or even third) time either. If you feel that your class needs to run like a well-oiled machine and that you need to be in control of everything that happens in the classroom, a flipped class is not for you. More importantly with the flipped format, every team is going to be different, so you need to shift gears quickly to address issues and misunderstandings as they come up. Of course, this is also one of the strengths of the approach. We deal with misunderstandings as they arise to help students better learn the material. It is also much easier to see when students are struggling with the material.Future DirectionsThe data from the first two semesters of this study are encouraging. The impact on the number of students withdrawing or ending the semester with D’s or F’s is incredibly exciting. In both the fall and spring semesters, the DFW rates (percentage wise) in the lecture section were twice the DFW rates in the flipped sections. Furthermore, attitude surveys conducted as part of the study show that students in the flipped class leave the course with a more positive opinion of chemistry in general, and that their experience with the course is also much more positive. With continued support from my Department and the Dean of SCSE, two flipped sections of Chem2 will be offered this spring (2017) and in the fall and spring of the next academic year (2017-18). Out of necessity, the flipped sections will continue to have enrollments of 90 to 100 students per section, but based on experiences during fall 2016, I know that this is manageable with sufficient instructional support from teaching assistants.ReferencesAbraham, M.R. (2005). Inquiry and the Learning Cycle Approach. In N.J. Pienta, M.M. Cooper T.J. Greenbowe (Eds.), Chemists’ Guide to Effective Teaching. (pp 41–52). Upper Saddle River, NJ: Pearson Education, Inc.Bergmann, J., Sams, A. (2012). Flip Your Classroom: Reach Every Student in Every Class Every Day. Eugene, OR: International Society for Technology in Education (iste).Berrett, D. (2012). How ‘Flipping’ the Classroom can Improve the Traditional Lecture. In A Guide to the Flipped Classroom. (2-6). The Chronicle of Higher Education, http://www.chronicle.com/resource/a-guide-to-the-flipped-classro/5882/ [viewed 28 Nov 2016].Bishop, J.L., Verleger, M. (2013). The Flipped Classroom: A Survey of the Research. In Proceedings, 120th American Society for Engineering Education Annual Conference Exposition, https://peer.asee.org/the-flipped-classroom-a-survey-of-the-research [viewed 28 Nov 2016].Bretz, S.L. (2005). All Students are not Created Equal: Learning Styles in the Chemistry Classroom. In N.J. Pienta, M.M. Cooper T.J. Greenbowe (Eds.), Chemists’ Guide to Effective Teaching. (pp 28–40). Upper Saddle River, NJ: Pearson Education.Byers, B., Eilks, I. (2009). The Need for Innovations in Higher Level Chemistry Education – A Pedagogical Justification. In I. Eilks B. Byers (Eds.), Innovative Methods of Teaching and Learning Chemistry in Higher Education. (pp 5–22). Cambridge: The Royal Society of Chemistry.Cooper, M.M. (2005). An Introduction to Small-Group Learning. In N.J. Pienta, M.M. Cooper T.J. Greenbowe (Eds.), Chemists’ Guide to Effective Teaching. (pp 117–128). Upper Saddle River, NJ: Pearson Education, Inc.Cracolice, M.S. (2005). How Students Learn: Knowledge Construction in College Chemistry Courses. In N.J. Pienta, M.M. Cooper T.J. Greenbowe (Eds.), Chemists’ Guide to Effective Teaching. (pp 12–27). Upper Saddle River, NJ: Pearson Education, Inc.Cracolice, M.S. (2009). Guided Inquiry and the Learning Cycle. In N.J. Pienta, M.M. Cooper T.J. Greenbowe (Eds.), Chemists’ Guide to Effective Teaching: Volume II. (pp 20–34). Upper Saddle River, NJ: Pearson Education, Inc.Dasgupta, N., McManus Scircle, M., Hunsinger, M. (2015). Female Peers in Small Work Groups Enhance Women’s Motivation, Verbal Participation, and Career Aspirations in Engineering. Proceedings of the National Academy of Science of the United States of America, 112(16), 4988-4993.Eberlein, T., Kampmeier, J., Minderhout, V., Moog, R.S., Platt, T., Varma-Nelson, P., White, H.B. (2008). Pedagogies of Engagement in Science: A Comparison of PBL, POGIL, and PLTL. Biochemistry and Molecular Biology Education, 36(4), 262-273.Educause Learning Initiative (ELI). (2012). 7 Things You Should Know About Flipped Classrooms. https://library.educause.edu/resources/2012/2/7-things-you-should-know-about-flipped-classrooms [viewed 28 Nov 2016].Eilks, I., Markic, S., Bäumer, M., Schanze, S. (2009). Cooperative Learning in Higher Level Chemistry Education. In I. Eilks B. Byers (Eds.), Innovative Methods of Teaching and Learning Chemistry in Higher Education. (pp 103–122). Cambridge: The Royal Society of Chemistry.Farrell, J.J., Moog, R.S., Spencer, J.N. (1999). A Guided Inquiry General Chemistry Course. Journal of Chemical Education, 76, 570-574.Floriano, M.A., Reiners, C.S., Markic, S., Avitabile, G. (2009). The Uniqueness of Teaching and Learning Chemistry. In I. Eilks B. Byers (Eds.), Innovative Methods of Teaching and Learning Chemistry in Higher Education. (pp 23–41). Cambridge: The Royal Society of Chemistry.Gallagher-Bolos, J.A., Smithenry, D.W. (2004). Teaching Inquiry-Based Chemistry: Creating Student-Led Scientific Communities. Portsmouth, NH: Heinemann.Geiger, L., Jones, L., Karre, I. (2009). Transforming Lecture Halls with Cooperative Learning. In N.J. Pienta, M.M. Cooper T.J. Greenbowe (Eds.), Chemists’ Guide to Effective Teaching: Volume II. (pp 49–70). Upper Saddle River, NJ: Pearson Education, Inc.Gimbar, K. [Lodge McCammon]. (2011, May 2). Why I Flipped My Classroom Retrieved from https://youtu.be/9aGuLuipTwgGreen, H. [CrashCourse]. (2014, Jan 13). Crash Course Chemistry Retrieved from https://youtu.be/FSyAehMdpyI?list=PL8dPuuaLjXtPHzzYuWy6fYEaX9mQQ8oGrGreenbowe, T.J., Hand, B. (2005). Introduction to the Science Writing Heuristic. In N.J. Pienta, M.M. Cooper T.J. Greenbowe (Eds.), Chemists’ Guide to Effective Teaching. (pp 140–154). Upper Saddle River, NJ: Pearson Education, Inc.Guo, P.J., Kim, J., Rubin, R. (2014). How Video Production Affects Student Engagement: An Empirical Study of MOOC Videos. In Proceedings of the First ACM Conference on Learning @ Scale Conference (L@S ’14). (pp 41-50). New York: ACM.Layton, R.A., Loughry, M.L., Ohland, M.W., Ricco, G.D. (2010). Design and Validation of a Web-Based System for Assigning Members to Teams Using Instructor-Specified Criteria. Advances in Engineering Education, 2 (1), 1-28.Loughry, M.L., Ohland, M.W., Moore, D.D. (2007). Development of a Theory-Based Assessment of Team Member Effectiveness. Educational and Psychological Measurement, 67, 505-524.Loughry, M.L., Ohland, M.L., Woehr, D.J. (2014). Assessing Teamwork Skills for Assurance of Learning Using CATME Team Tools. Journal of Marketing Education, 36(1), 5-19.Mangan, K. (2013). Inside the Flipped Classroom. In A Guide to the Flipped Classroom. (7-9). The Chronicle of Higher Education, http://www.chronicle.com/resource/a-guide-to-the-flipped-classro/5882/ [viewed 28 Nov 2016].Mills, H.R. (1977). Techniques of Technical Training, 3rd edition. London: MacMillan.Moog, R.S., Farrell, J.J. (2011). Chemistry: A Guided Inquiry. Hoboken, NJ: John Wiley Sons, Inc.Moog, R.S., Creegan, F.J., Hanson, D.M., Spencer, J.N., Straumanis, A., Bunce, D.M., Wolfskill, T. (2009). POGIL: Process-Oriented Guided-Inquiry Learning. In N.J. Pienta, M.M. Cooper T.J. Greenbowe (Eds.), Chemists’ Guide to Effective Teaching: Volume II. (pp 90–107). Upper Saddle River, NJ: Pearson Education, Inc.Musallam, R. [teach’s channel]. (2011a, Sep 15). Ramsey Musallam: Flipteaching – Part 1 Retrieved from https://youtu.be/rub1VNq2NvMMusallam, R. [teach’s channel]. (2011b, Sep 15). Ramsey Musallam: Flipteaching – Part 2 Retrieved from https://youtu.be/FIwKVRQJ1PEMusallam, R. [teach’s channel]. (2011c, Sep 15). Ramsey Musallam: Flipteaching – Part 3 Retrieved from https://youtu.be/jjElGeJaHZcMusallam, R. [teach’s channel]. (2011d, Sep 15). Ramsey Musallam: Flipteaching – Part 4. Retrieved from https://youtu.be/9MzxczsF9bANeshyba, S. (2013). It’s a Flipping Revolution. In A Guide to the Flipped Classroom. (10-12). The Chronicle of Higher Education, http://www.chronicle.com/resource/a-guide-to-the-flipped-classro/5882/ [viewed 28 Nov 2016].Oakley, B., Felder, R.M., Brent, R., Elhajj, I. (2004). Turning Student Groups Into Effective Teams. Journal of Student Centered Learning, 2(1), 9-34.Ohland, M.W., Loughry, M.L., Woehr, D.J., Finelli, C.J., Bullard, L.G., Felder, R.M., Layton, R.A., Pomeranz, H.R., Schmucker, D.G. (2012). The Comprehensive Assessment of Team Member Effectiveness: Development of a Behaviorally Anchored Rating Scale for Self and Peer Evaluation. Academy of Management Learning Education, 11(4), 609-630.Pettis, T. (2015, Jan 13). Flipping with Kirch: Designing a Flipped Learning Environment Retrieved from https://www.sophia.org/tutorials/flipping-with-kirch-webinar-011315Schneider, J., Lovitt, C. (2016). The POGIL Project Workshop: Classroom Facilitation, Presented at the 2016 Biennial Conference on Chemical Education (BCCE), Greeley, CO, July 31-August 4, 2016.Schwartz, M.S., Sadler, P.M., Sonnert, G., Tai, R.H. (2009). Depth Versus Breadth: How Content Coverage in High School Science Courses Related to Later Success in College Science Coursework. Science Education, 93(5), 798-826.Smith, J.D. (2013). Student Attitudes Toward Flipping the General Chemistry Classroom. Chemical Education Research and Practice, 14, 607-614.Sowash, J. [GoEd Online’s channel]. (2012, Apr 23). 5 Things I Wish I Knew When I Flipped My Class Retrieved from https://youtu.be/4JPdGlyt6ggSpencer, J.N. (1999). New Directions in Teaching Chemistry. Journal of Chemical Education, 76, 566-569.Talbert, R. (2014a). Toward a Common Definition of ‘Flipped Learning’. In A Guide to the Flipped Classroom. (13-14). The Chronicle of Higher Education, http://www.chronicle.com/resource/a-guide-to-the-flipped-classro/5882/ [viewed 28 Nov 2016].Talbert, R. (2014b). Flipped Learning Skepticism: Do Students Want to Have Lectures? In A Guide to the Flipped Classroom. (15-17). The Chronicle of Higher Education, http://www.chronicle.com/resource/a-guide-to-the-flipped-classro/5882/ [viewed 28 Nov 2016].AcknowledgementsI would like to take the opportunity to thank the University of Minnesota Center for Educational Innovation for supporting my Educational Learning Innovations research project and, especially, Kris Gorman, J.D. Walker, and Paul Baepler for their invaluable assistance with the design and analysis of the project. I would also like to thank the members of the first Swenson College of Science and Engineering Active Learning Cohort for their friendship and support throughout this endeavor, and Drs. Bilin Tsai, Elizabeth Austin-Minor, and Joshua Hamilton for their confidence in me and support, both personal and administrative, throughout this grand experiment.Chapter 6Teaching Undergraduate History: A Problem-Based ApproachRobert K. Poch and Eskender YousufKeywordsproblem solving, historical thinking skills, learning assessment IntroductionAmong the challenges that faculty encounter is facilitating active engagement with their discipline within classrooms of diverse undergraduate students (Calder, 2006; Rendon, 2009). We face this challenge regularly when teaching history. Within a mostly lecture-based format, it is easy to deny students opportunities to engage the discipline as historians.While we concentrate primarily on the discipline of history, this approach can be applied within other disciplines where the aim is to provide diverse undergraduate students with the opportunity to “do” the work of those disciplines and find personal connection within them. Historians discover and use primary source documents, confront vexing contextual and interpretational problems, experience the diverse perspectives of peers, and so too can students. For many faculty, the temptation in undergraduate survey courses is to place full emphasis on content coverage and to ignore or minimize development of discipline-based skills (Calder, 2006; Sipress and Voelker, 2009). This produces poor results if our instructional goals include providing a more genuine experience with our discipline, developing analytic skills, and engaging students meaningfully (Weimer 2002). When students are passive recipients of disciplinary information with no apparent connection to themselves, they withdraw intellectually and emotionally (Freire 1970; Langer 1997).However, it is possible for highly diverse students to experience the dynamic nature of disciplines such as history by “doing” – that is, by actively developing the skills and engaging the processes and problems involved in being a practitioner of the discipline (Sipress and Voelker, 2009; Weimer, 2002). This can happen in large and small classes and also in survey courses. We want students to encounter history as historians. We also want them to experience the excitement of historical discovery and personal meaning-making that first drew us into the work. In doing so, students also learn and retain substantive course content.We focus below on a “problem-based” approach to teaching and learning history where students are active disciplinary practitioners engaged in addressing problem topics of relevance and connection to their diverse lives. In doing so, the following questions are addressed:What are some of the core elements of historical inquiry? What do skilled historians actually do?What actions encourage students from different cultural and disciplinary backgrounds to engage the core elements of historical inquiry as practitioners?How can course pedagogy, assignments, and assessments become consistent with what skilled historians do and foster student engagement?How can the results of these efforts be assessed? What are some of the assessment results from using a problem-based approach to learning history?While we concentrate primarily on the discipline of history, this approach can be applied within other disciplines where the aim is to provide diverse undergraduate students with the opportunity to “do” the work of those disciplines and find personal connection within them (Gurung, Chick, and Haynie, 2009).Identifying and Using Core Elements of Historical InquiryIn creating learning environments where students become historians, it is necessary to consider what it is that historians do and what skills are necessary to be practitioners of the discipline. In a macro sense, many historians describe their work as problem solving guided by active questioning (Elton, 1967; Fischer, 1970; Marius and Page, 2005; Nevins, 1963). That is, questions are posed; sources and facts are collected, critically read, contextualized, and organized; and, an interpretation of the past is formed while recognizing that the complexities of history defy easy explanations (Ayers, 2006; Commager, 1965; Wineburg, 1991, 1999). Whenever possible, historians utilize primary sources to form their own interpretations rather than relying mostly on the interpretations of other historians.Historians are regularly challenged to analyze, contextualize, and interpret the past from incomplete disparate sources. Such actions require a series of more discrete skills including the critical evaluation, interpretation, and communication of evidence; the detection of bias; and careful consideration of historical causation (Ayers, 2005; Barzun and Graff, 1977; Bloch, 1953; Carr, 1961; Commager, 1965; Elton, 1967; Evans, 1999; Fischer, 1970; Lerner, 1997; Nevins, 1963; Wood, 2008). These skills are expressed concisely as the “5Cs” of historical thinking: “change over time, causality, context, complexity, and contingency” (Andrews and Burke, 2007, 1).Through using the 5Cs, students become increasingly aware of how much can change over time – such as political systems, landscapes, and social values – while, simultaneously, acknowledging retention of strong elements of the past such as holidays and the rituals surrounding them. Further, once developed through classroom engagement with historical sources, the elements of causality, context, complexity, and contingency enable students to identify and appreciate the incomplete nature of historical records and the intricate, simultaneous, and broad scale human interactions and competing interests in history (Ayers, 2006; Nevins, 1963; Wood, 2008). The awareness of complexity causes professional historians and students to probe more deeply into explanations of causality and to reject simplistic reasoning.To have students engage history problems in a manner that is relevant and meaningful, it is necessary to also think carefully about how to invite students and their interests into the process of historical inquiry. While engineering, mathematics, physics and accounting are experienced as real, relevant, and practical, history is not experienced that way. Students often encounter it as an abstruse, fact-laden, memorization-based, irrelevant, impersonal discipline. We must, therefore, address how to engage students in being practitioners of historical inquiry and interpretation.These are some of the key components of the real work of historians and the historical reasoning used to create interpretations of the past. In working with undergraduate students – some of whom just graduated from high school – the 5Cs are a useful, understandable, and easy to remember toolset for engaging historical inquiry. With these parts of the work of historians and historical thinking in mind, it is possible to design classroom experiences that bring students into the dynamic nature of this work and its associated challenges. Students can then experience the discipline of history more fully and also learn how to create their own historical meaning from available sources (Sipress and Voelker, 2009). These components of historical thinking help to form the “history problems” that we utilize in our U.S. history classroom and which we describe further below.However, to have students engage history problems in a manner that is relevant and meaningful, it is necessary to also think carefully about how to invite students and their interests into the process of historical inquiry. While engineering, mathematics, physics and accounting are experienced as real, relevant, and practical, history is not experienced that way. Students often encounter it as an abstruse, fact-laden, memorization-based, irrelevant, impersonal discipline. We must, therefore, address how to engage students in being practitioners of historical inquiry and interpretation.Engaging Students in Core Elements of Historical Inquiry as PractitionersThe invitation to participate in our class, “America’s Past and Present: Multicultural Perspectives,” is underscored by bringing students into direct interaction with historical thinking skills, primary source materials that are reflective of multiple cultures on the American landscape, and with complex historical issues and problems that invite students to interpret history with their own voices rather than having a textbook or the instructor be the sole interpretive voices. We want students to gain more elegant and inclusive views of history that expose the complex dynamics between people over time and which stimulate curiosity about how life was experienced and interpreted by different diverse populations. This is enabled in part by the diversity of our students. The multiple complexities of persons in the past are reflected in our students. As observed by Lee, Poch, Shaw, and Williams (2012),“…we have observed our institution’s student population become increasingly diverse in terms of racial and ethnic demographics. Historically, generalized categories of racial and ethnic identity have become more diffuse and complex. We are also more mindful of the often less visible forms of difference that are present in any learning environment, such as socioeconomic status, sexual orientation, religion, disability, and many others”As students engage each other in class and discover how their classmates form different historical interpretations based in part on their different lived experiences, it stimulates and reinforces an understanding of the different perspectives and lived experiences of persons throughout history.To better ensure the relevance of the history problems to diverse student interests, students are asked on the first day of the course what part of U.S. history between the Civil War and present time is of greatest interest to them. While some students do not know how to answer such a question at first, many others have some notion of their interests. Examples of these student responses from spring semester 2015 are as follows:World War IIU.S. Civil Rights movements (including the role of youth in such movements)Vietnam WarThe Great DepressionThe 1920s9/11 and its effect on the worldSpace raceOther countries and perceptions of the U.S.Native Americans and TribesImmigrantsCivil War differing economiesWhen we, as instructors, are responsive to such interests, students more fully engage with the topics and are willing to invest the energy to do the challenging work of historical meaning-making using disciplinary thinking skills.These interests are invaluable in making the course ours – that is, a shared experience of historical investigation that reflects mutual interests rather than those of the instructor alone. It is a powerful opportunity to communicate to students at the beginning of the course that the instructors are engaged co-investigators of historical topics that the students suggest and that student interests are of great value. We use student historical interests to create substantive class discussion questions, short reading and writing assignments, and further engagement with historical thinking skills through lengthier history problems that come later in the semester. While assembling sources related to the interests, we spend the first three to four weeks of the course introducing and practicing the 5Cs of historical thinking. Initial reading and writing assignments selected before the course starts enable students to begin the process of understanding, recognizing, and using context, causality, complexity, change and continuity over time, and contingency. These historical thinking skills are then used to explore more deeply and intentionally the historical topics that students suggest. In doing so, student interests become integrated and useful parts of the course experience.Students also tell us that they want to explore historical themes and issues that are not commonly approached in historical texts or rooted in fact memorization. For example, one student expressed in a topical interest survey that she wanted “…to learn the truth about history. The real original text. I want to find it and research it. Interest in causality and what caused all the events in history to happen? WHYYY! The reason things went down the way they did!” Students express that they want to explore the meaning and use of racism, the rise of feminism, and the perspectives of other nations whose histories intersect with those of the United States. They want to do so in a way that engages history through interesting questions full of encounters with ordinary people who experienced the past in powerful but mostly unknown ways. When we, as instructors, are responsive to such interests, students more fully engage with the topics and are willing to invest the energy to do the challenging work of historical meaning-making using disciplinary thinking skills.Making Pedagogy, Assignments, and Assessments Consistent with what Historians Actually DoEngaging students as historians takes careful thought and planning. Core elements of course design are important parts of this work. The course curriculum must provide space for the development of student evaluative and interpretive skills. This often comes with winnowing some course content as traditionally delivered through lengthy lectures (Calder, 2006). The process of winnowing involved using part of a summer break to critically review course materials to identify where unnecessary content was located that cluttered class time and reduced the capacity to develop student historical thinking skills. For example, a discussion of Civil War medicine and pro- and anti-U.S. imperialist arguments were removed given that they were peripheral to more important course themes. Further, those subjects tended to lead to more lecturing rather than active discussion. Rendon (1993) observed that “…many culturally diverse students do not learn best through lecture. Instead, we should focus on collaborative learning and dialogue that promote critical thinking, interpretation and diversity of opinion” (10).Students can mine rich primary sources of the period as guided by research questions collaboratively developed by students and the instructor.Lectures are balanced with skill-building by doing – actively engaging students in learning how to develop researchable questions, engaging primary and secondary texts with critical lenses, forming interpretations from available evidence, and presenting results. Further, significant thought must be invested in designing course assignments and resources that enable skill development to occur and be assessed. Assessments must be constructed to evaluate student work in a manner consistent with skill development expectations. Class time invested in developing the foundational skills used within the discipline is necessary given that “…history teachers cannot simply present students with documents, tell them what to do, and then expect magical gains in the development of students’ historical sense. Much more elaborate and carefully thought out ‘scaffolding’ is needed to realize the potential of this approach” (Calder et al., 2002, 59).This approach has significant student developmental implications. It may involve moving students and the course structure away from a dualistic form of learning history where questions are framed in terms of right and wrong response outcomes and there is strong dependency upon the instructor. Instead, there will be movement toward a course design wherein students are met with formulating questions within a course-related area of personal historical interest that has interpretive complexities associated within it (Donald, 2002, 3; Evans et al, 2010).For example, rather than presenting students with a course design that asks them to identify within an exam three major outcomes of Reconstruction after the U.S. Civil War from lecture notes, students can experience the real problems of Reconstruction in depth by reading conflicting newspaper accounts in the North and the South regarding the political enfranchisement of African American men and the political balances of power that were at play (Langer, 1989). Rather than searching secondary and tertiary sources (such as many textbooks and lectures) alone for such information, students can mine rich primary sources of the period as guided by research questions collaboratively developed by students and the instructor. One research question that was developed in this manner focused on the tactics that some southern states utilized to stymie the voting capacity of black males following passage of the Fifteenth Amendment. In response, students were able to find and analyze different literacy tests for voting (the class even tried taking some of the tests which produced a high failure rate) and also details on the administration of poll taxes.Such collaboration and student interpretive responsibilities can lead to movement from what psychologist Ellen Langer refers to as “mindlessness” wherein students are stuck with rote memorization and the search for the “right” answer rather than experiencing the rich contexts and possibilities that exist as part of the act of discovering and making meaning within disciplines (Langer, 1997). Langer notes that, “In math, teaching for understanding involves teaching students to think about what a problem means and to look for multiple solutions. Studies have confirmed that science is better taught through hands-on research and discovery than through memorization alone. In English, teaching for understanding means emphasizing the process of writing and exploring literature rather than memorizing grammar rules and doing drills. Understanding is encouraged in history by turning students into junior historians” (Langer, 1997, 71, 72). It is in that spirit that we developed history problems.History Problems “As an interpretive historian using the primary source readings that are provided in this problem, how do you define Jim Crow?” This question, which seems deceptively simple at first, quickly exposes the complexities of Jim Crow as a comprehensive system within American society that touched every aspect of life.Each history problem is comprised of three essential parts: an introduction to the problem with concise contextual information; open-ended problem questions designed to provide students with interpretive space to utilize their voice and perspectives (rather than the instructor’s voice or that of the textbook); and a set of primary source materials that reflect diverse authors and views. Students are given three history problems throughout the semester and they have approximately four weeks to complete them given the complexity of the readings. Although there is no required page length for the problem responses, students often write seven pages or more for each problem. The history problems used during the Spring 2015 semester were based on student interests expressed at the beginning of the semester and involved the following topics: “The challenges of Jim Crow and the dynamics found within it;” “’Equal protection under the law:’ The challenges of separate but equal – The struggle for Brown v. Board of Education;” and, “September 11, 2001.”The history problem questions provide students with the ability to create responses based on their own interpretation of the material. The questions replicate real challenges and problems for historians that are consistent with the 5Cs of historical thinking that we use in class. Some questions expose students to the complexities of powerful systems of racial oppression such as Jim Crow. Other questions focus on establishing context or examining change over time. For example, in the problem examining Jim Crow, students were asked the following: “As an interpretive historian using the primary source readings that are provided in this problem, how do you define Jim Crow?” This question, which seems deceptively simple at first, quickly exposes the complexities of Jim Crow as a comprehensive system within American society that touched every aspect of life.To assist in developing a definition of Jim Crow, the history problem packet includes a variety of primary sources that include memoirs, excerpts from scholarly books and novels, and a 1949 travel guide for African American motorists. Within this particular problem packet, the sources included pieces from W.E.B. Dubois’ The Souls of Black Folk (1903); Ralph Ellison’s Invisible Man (1952); John Hope Franklin’s autobiography, Mirror to America (2005); Howard Thurman’s The Luminous Darkness: A Personal Interpretation of the Anatomy of Segregation and the Ground of Hope (1965); Richard Wright’s Uncle Tom’s Children (1940); and, The Negro Motorist Green Book (1949). The Green Book was published to provide African American travelers with “…information that will keep him from running into difficulties, embarrassments and to make his trips more enjoyable” (1). These sources provided different views of and experiences with Jim Crow and, unlike a textbook, did not provide the definition and interpretation of Jim Crow for the students. Instead, the students worked with the different texts, situated them contextually in their particular time and place and with consideration of who wrote them, and gradually developed their own definition of Jim Crow. Further, the sources spanned a number of decades so that some consideration could be given to change over time in addition to complexity. The sources worked well in providing multiple perspectives of how Jim Crow, as a system, affected different parts of life and also a sense of the varieties of materials that historians use.The same history problem also asked students to consider how the sources in the packet related to any prior readings we had used in the course (such as Frederick Douglass’ 1865 speech, “What the Black Man Wants”), so that students could further analyze and gain familiarity with context, change over time, complexity, contingency, and causality. Douglass’ speech was useful not only as an earlier expression of the challenges and contradictions that Jim Crow created within a nation that professed freedom and democracy, but also served as a source to explore the challenging concept of contingency. By expressing how black men wanted political participation through receipt of the right to vote and full recognition for their intellectual capacity to be informed contributors to democracy, Douglass’ speech highlighted contingencies necessary for breaking explicit bonds of enslavement and more diffuse societal systems of oppression. These varied course and problem-based primary sources enabled students to make complex connections between forms of evidence and further solidified historical thinking skills as expressed within the 5Cs. Providing approximately four weeks to work with each history problem gave plenty of in-class time to further discuss the sources, the context of the sources, and to practice the skills necessary to utilize them effectively.Assessing the Results of Using a Problem-Based Approach to Learning HistoryWe utilized several forms of assessment to evaluate the problem-based approach to learning history: 1) the evaluation of student written responses to history problems, 2) individual interviews with students, and, 3) an independently conducted end-of-semester student survey. In each assessment approach, it was important that student voices were prominent and listened to attentively (Patton, 1980). Each of these assessment forms is discussed below.Written responses to history problemsStudent written responses to the four history problems were evaluated carefully for progressive use of the elements of historical thinking. Each of the papers went through two evaluative reviews – one by the course teaching assistant and the other by the primary instructor. The papers were scored on a standard A-F grading scale and were preceded by smaller writing assignments that practiced discrete elements of five identified historical thinking skills. For example, the Frederick Douglass speech, “What the Black Man Wants,” was used early in the semester in part so students could practice establishing and expressing context and recognizing some elements of contingency. This was done with multiple other pieces of short reading and writing exercises that involved the voices and writing of diverse speakers and authors. With this practice experience in place, students could move with greater confidence in addressing contextual issues in the first history problem and those that followed. The papers were also useful in assessing student command or struggle with certain components of historical thinking. We discovered in multiple early papers that students did not fully understand the idea of contingency and, in response, were able to spend more time discussing and practicing it in class.Toward the end of the semester as students worked with perhaps the most challenging history problem involving the September 11, 2001 attacks, we could detect that students were engaging in far more sophisticated historical reasoning and explicit use of historical thinking skills. For example, one student, a freshman, having studied the presence and role of the United States in the Middle East since the early twentieth-century (using maps, documents, interviews, reports, and political cartoons provided in the fourth history problem), constructed a complex contextual background in her written response to one set of the problem questions (“Using information from our class sessions and the materials provided within this problem packet, describe the relationships that existed and some of the events that occurred between the United States and the ‘Middle East’ region prior to 9/11. With these sources in mind, what are some of the possible motives for the 9/11 attack?”). She responded in part in her introduction,The events of September 11th, 2001 came as a shock to millions of American citizens; however, a complex history of rocky foreign relations combined with the struggles regarding religion and government in the ‘Middle East’ suggest that the attack was only one part of several interconnected issues. As we examine the context of the events surrounding the 9/11 attacks, the complexity of the United States’ position in world affairs, the major causes leading up to the attack, and contingency of other nations’ histories on our own, we can begin to analyze the affect that each of these has had on the aftermath of 9/11 over time… [the] ten to fifteen years before the attacks on the Twin Towers… show a deeply complex relationship between different nations. While the Saudi Arabian government aided the United States [in the Gulf War by providing a U.S. military staging ground along the border with Kuwait] there were other entities such as Iraq that were pitted against the United States, resulting in conflict between the nations in that area regarding involvement of the United States. To add to this complexity is the idea of a theocracy and questions on how to rule a nation when military forces within those nations do not share the political philosophy or religious beliefs of those nations.In this brief excerpt which was supported by lengthier supporting text and examples, we could easily detect the use of historical thinking skills (some of which were explicitly mentioned), including greater recognition of the deep complexities that long preceded the events of 9/11.The written responses to the history problem questions were returned to the students with evaluative comments that served, in part, to prepare students for individual meetings with the course instructor and the graduate research assistant. With highly diverse students from different nations, it was important to provide different opportunities for the students to express how they approached the problems that extended beyond their written responses.Individual meetings with students about the history problemsIndividual student interviews were conducted on two of the problem set essays (history problems one and three). Each student was given the opportunity to schedule a conversation with us to discuss their responses and to further sharpen their historical thinking skills based on the 5Cs. During these meetings, the students could gain additional points (but no subtraction of points) by further clarifying their written responses and the processes that they used to construct them. The meeting questions were provided to each student in advance and included: Where did you encounter the greatest challenges in responding to this history problem? How did you approach the challenges? What do you believe you learned through engaging in this history problem? An opening question was designed to further probe each student’s writing by asking: “We found some engaging interpretations within your paper [if this was truthful] and also some places where we would like to know more. Can you further describe [this was customized for each student paper]….” The questions presented opportunities for students to further explain their own interpretations and how they approached the problems over time. Through the questions, students reflected on and expressed their own historical interpretations in a manner that replicates much of the way that historians utilize peers within their professional communities.Within the second set of individual meetings on the third history problem, students began to express more of the 5Cs of historical thinking.During the first set of interviews regarding the “Challenges of Jim Crow and the dynamics found within it,” we met individually with twenty-three students who expressed a wide array of historical thinking skills. For example, one student, in expanding upon her interpretation of Jim Crow, remarked that developing her own definition of Jim Crow enabled her to “…go far below the surface to see the complexity of Jim Crow and its relationship to definitions of racism.” Further, the student explained that examining the use of Jim Crow-related art revealed to her the complex strategies of Jim Crow systems of oppression. Another student observed that reading primary source accounts of African Americans who lived within Jim Crow brought forth “contradictions” within the imagery and terms used within Jim Crow such as using ideas of “light” and “visibility” to describe American society while those who lived under the weight of Jim Crow described darkness, shadows, and invisibility. Within these comments, and many others, we observed students expressing different elements of historical thinking including complexity (very explicitly), as well as change over time, and causality as students considered and described the structures of Jim Crow messaging and how the messaging was delivered in different ways during specific spans of time.Within the second set of individual meetings on the third history problem, students began to express more of the 5Cs of historical thinking. We used a slightly modified set of questions that included: Did you feel that you were able to utilize any particular historical thinking skills in this problem? Students commented more extensively and easily about historical thinking skills and demonstrated within their papers greater complexity in historical thinking. For example, one student expressed complex differences in how African American’s responded to racial oppression over time. She interpreted primary sources in the first history problem as being “defensive in nature – how persons responded or protected themselves within the Jim Crow system” whereas in the third history problem on the legal strategies that African American attorneys used to eventually prevail in the Brown v. Board of Education decision, the strategy was “more offensive in nature in that it showed black persons taking back their rights and being more confident in doing so.” This student, and others, expressed greater awareness and mastery of complexity, causality, and change over time in their interpretations of primary historical source materials.Independently conducted end-of-semester student surveyAt the request of the instructor, a grant-supported survey was developed and given to students in the history course at the very end of spring semester 2015. Among those who responded to the survey (50% of a class of 24 first-year students where this problem-based approach was fully implemented), the following are representative of their responses:Question: What parts of the course were particularly effective for you in developing historical thinking skills and the capacity to be an effective historian? What parts were particularly ineffective?“The parts of this course that were effective in developing historical thinking skills were definitely the history problems and also the 5Cs. I learned so much through the history problems that I would have never learned through a test and I will remember the information much better by writing about it in a history problem. I didn’t feel like any part was ineffective.”“The history problems were effective because they allowed us to give our own opinion on the matter and we got involved instead of just mindless memorization.”“I think that the most effective things that we did in class to develop my historical thinking skills were definitely the problem sets and our class discussions. Both of those two platforms pushed us to think for ourselves and contribute to a larger group discussion. I loved the problem sets because they forced me to think and form my own opinions using the historical thinking skills that we were given.”“The history problems really helped me see how contemporary historians actually applied their skills to modern problems.”Question: Do you believe that you know and can apply the essential elements of historical thinking, as a result of this course? Please explain.“Yes, I have already applied it to other classes and feel very comfortable doing it.”“Yes, the history problems gave us that opportunity.”“I do believe that I could apply the elements of historical thinking into other classes and in my everyday life as a historian. I feel confident that I know the 5Cs of historical thinking and could use them in other situations. They were drilled into us, I won’t forget them.”“Yes. I believe that using the 5Cs from the beginning of this course helped me to gain further knowledge and also to help me dig deeper into historical problems and questions.”“I definitely feel that this course has aided my skills in critical analysis and historical thinking and I can see how to use these skills in different contexts and subjects besides history.”Student survey responses found that 1) history thinking elements of the 5C’s gave them a grasp of historical thinking skills; 2) established a process whereby students could formulate their own interpretations of historical sources thereby moving from mindless memorization to mindfulness and, 3) students were able to utilize their own lived experiences and interests as historians engaged in investigating historical problems.ConclusionThe use of history problems in our course stems from a twofold purpose. First, we want students to experience the discipline of history as actively engaged historians who use primary sources in addressing challenging questions of historical interpretation through use of well-defined historical thinking skills. Second, we want to facilitate personal interaction with the discipline by using sources and stories that are reflective of the diversity of our students and enabling their interpretive voices to emerge and be respected in our assessments of their learning. Through the use of history problems, unlike our past exams, we noticed the disappearance of instructor voice in student interpretations of historical source materials and an increase in deliberate use of the 5Cs of historical thinking: context, change over time, contingency, complexity, and causality. Continued exploration of the use of history problems in developing more focused development of particular historical thinking skills will occur in our future work as will the capacity to assess those skills effectively through combinations of written work and in-person conversational interactions with students.ReferencesAndrews, T., Burke, F. (2007). “What Does it Mean to Think Historically?” Perspectives on History: The Newsmagazine of the American Historical Association 45 (1), 32-35. Washington, DC: American Historical Association.Ayers, E. L. (2005). What caused the Civil War?: Reflections on the South and southern history. New York: W.W. Norton CompanyBarzun, J., and Graff, H. F. (1977). The modern researcher. New York: Harcourt Brace Jovanovich, Inc.Bloch, M. (1953). The Historian’s Craft. New York: Vintage Books.Calder, L., Cutler, W. W., Mills Kelly, T. (2002). “History Lessons: Historians and the Scholarship of Teaching and Learning.” In M. T. Huber S.P. Morreale (Eds.). Disciplinary styles in the scholarship of teaching and learning: Exploring common ground. (pp. 45-68). Washington, DC: American Association for Higher Education The Carnegie Foundation for the Advancement of Teaching.Calder, L. (2006). “Uncoverage: Toward a Signature Pedagogy for the History Survey.” The Journal of American History, 1358-1370. .Carr, E. (1961). What is History? New York: Vintage Books.Commager, H. S. (1965). The nature and the study of history. Columbus, OH: Charles E. Merrill Books, Inc.Deloria Jr., V., and Wildcat, D. R. (2001). Power and place: Indian education in America. Golden, CO: American Indian Graduate Center and Fulcrum Resources.Donald, J. G. (2002). Learning to think: Disciplinary perspectives. San Francisco: Jossey-Bass.Dubois, W.E.B. (1903). The souls of black folk. Chicago: A.C. McClurg and Co.Ellison, R. (1952). Invisible man. New York: Random House.Elton, G. R. (1967). The Practice of History. New York: Thomas Y. Cromwell Company.Evans, N. J., et al (2010). Student Development in College: Theory, Research, and Practice. San Francisco: John Wiley Sonsn.Evans, R. J. (1999). In Defense of History. New York: W.W. Norton Company.Fischer, D. H. (1970). Historians’ Fallacies: Toward a Logic of Historical Thought. New York: HarperPerennial.Franklin, J. H. (2005). Mirror to America. New York: Farrar, Straus and Giroux.Freire, P. (1970). Pedagogy of the Oppressed. New York: Penguin Books.Guring, A.R.; Chick, N. L.; and Haynie, A., eds. (2009). Exploring Signature Pedagogies: Approaches to Teaching Disciplinary Habits of Mind. Reagan Sterling, VA: Stylus Publishing.Ladson-Billings, G. (Autumn, 1995). “Toward a Theory of Culturally Relevant Pedagogy,” American Educational Research Journal, Vol. 32, No. 3, 465-491.Langer, E. J. (1989). Mindfulness. New York: Addison-Wesley Publishing Company.Langer, E. J. (1997). The Power of Mindful Learning. New York: Addison-Wesley Publishing Company.Lerner, G. (1997). Why History Matters. New York: Oxford University Press.Marius, R., and Page, M. E. (2005). A Short Guide to Writing about History. New York: Pearson Longman.Nevins, A. (1963). The Gateway to History. Chicago: Quadrangle Books.Patton, M. Q. (1980). Qualitative Evaluation Methods. Beverly Hills, CA: Sage Publications.Rendon, L. I. (2009). Sentipensante (Sensing/Thinking) Pedagogy: Educating for Wholeness, Social Justice and Liberation. Sterling, VA: Stylus Publishing.Rendon, L. I. (February 1993). “Validating Culturally Diverse Students.” Paper Presented at the Annual Conference of the Community College Chairs, Phoenix, Arizona.Sipress, J. M., and Voelker, D. J. (2009). “From Learning History to Doing History.” Exploring Signature Pedagogies: Approaches to Teaching Disciplinary Habits of Mind. Reagan A.R. Guring, Nancy L. Chick, and Aeron Haynie, eds. Sterling, VA: Stylus Publishing.The Negro Motorist Green Book (1949). New York: Victor H. Green Co., Publishers. Accessed at: http://www.autolife.umd.umich.edu/Race/R_Casestudy/87_135_1736_GreenBk.pdfThurman, H. (1965). The Luminous Darkness: A Personal Interpretation of the Anatomy of Segregation and the Ground of Hope. New York: Harper Row.Wineburg, S. (1991). “Historical Problem Solving: A Study of the Cognitive ProcessesWineburg, S. (March 1999). “Historical Thinking and Other Unnatural Acts.” The Phi Delta Kappan, vol. 80, No. 7, pp. 488-499.Weimer, M. (2002). Learner-Centered Teaching: Five Key Changes to Practice. San Francisco: Jossey-Bass.Wood, G. (2008). The Purpose of the Past: Reflections on the Uses of History. New York: The Penguin Press.Wright, R. (1940). Uncle Tom’s Children. New York: Harper Brothers.AcknowledgementsFacilitation and evaluation of this project was enabled by the invaluable support and guidance of Ilene Alexander, Jeff Lindgren, and J.D. Walker. We also acknowledge the work and influence of the following excellent undergraduate teaching assistants who, over the last seven years, contributed to the development, implementation, and the strengthening of the problem based approach to teaching and learning: Emily DePalma, Emily McCune, Julianna Ryburn, Jade Beauclair Sandstrom, and Chris StewartChapter 7Flipping the Classroom and the Genetic Counseling Clinic: Online Branching CasesAlex Cummings, Andrea Stoddard, Pat McCarthy Veach, Bonnie LeRoy and Heather ZierhutKeywordsgenetics, genetic counseling, case-based learning, on-line learning, flipping the classroom, teaching modules IntroductionGenetic counseling professionals are in high demand in the United States and that demand is growing. Given the current number of graduates matriculating annually, the increased need for genetic counselors likely will not be met. Recent studies show there are more than twice as many jobs available as there are graduates to fill them. Yet, it is difficult to increase class sizes within genetic counseling graduate programs given the limited resources available to fulfill intensive educational requirements in the clinical setting. As with most human services professions, training involves several supervised clinical rotations. These rotations represent different genetic counseling specialties in which students actively participate in patient care under the direct supervision of certified genetic counselors. Students typically begin their clinical rotations towards the end of the first year in their genetic counseling program and their initial rotation is observational in nature. In subsequent rotations, students assume increasing responsibility for patient care, resulting in the need for even more intensive supervision by certified genetic counselors. In addition to the intensive supervision needed, there are a limited number of clinical rotation sites available for genetic counseling students. These factors limit the numbers of students that can be admitted to genetic counseling programs.One way to address these issues is by introducing simulated clinic situations into the class, in essence, providing some supplemental observational experiences that occur during initial clinical rotations. Accordingly, we created a series of online modules consisting of genetic counseling cases accompanied by stimulus questions for use in genetic counseling curricula. These modules effectively provide an opportunity for first year students to gain exposure to prenatal, cancer, and pediatric clinical scenarios without taking up valuable clinic supervision spots. To best recreate clinical situations, we chose a case-based learning approach. Case-based learning involves the use of cases representative of real life situations to elicit appropriate learning and understanding of desired curricular points. Unlike problem-based learning, case-based learning allows for preparation prior to discussion and is a more guided inquiry process, with said direction provided by facilitators/instructors.¹ Case-based learning has been implemented within all levels of education and has been shown to be especially effective within the sciences.2-4 Studies suggest case-based learning increases student motivation and engagement, as well as short-term retention of knowledge.4,5Activities involving virtual viewing of and reflection about cases are intended to reduce the amount of time students spend in physical clinical observation. We developed three modules comprised of video recordings of simulated genetic counseling sessions that represent the primary clinical genetic counseling specialties – prenatal, pediatric, and cancer genetic counseling. The modules will initially be introduced into two genetic counseling courses at the University of Minnesota (GCD 8911, GCD 8912) that are designed to prepare students for learning in the clinical setting.Module DescriptionEach module contains segments of genetic counseling sessions (scenarios) representing major components (e.g., taking an efficient family history, contracting [mutual agenda setting], discussing testing options, decision making, and providing results). All modules consist of at least two genetic counseling appointments (pre- and post- test). Table 1 shows the structure and major components of the modules.Students are instructed to watch full length clips of the major components of the genetic counseling sessions (~15 to 40 minutes) and then answer a series of questions associated with segments of the videos (~30 seconds to 2 minutes). Accompanying questions include multiple choice items, short answer, and essay questions addressing both genetics knowledge and psychosocial counseling issues and skills. Progression through the scenarios within each module is comparable to that of an entire genetic counseling session.The modules and corresponding assignments can be used within the classroom and/or they can be completed outside of class. Students are able to watch, re-watch, and progress through each case at their own pace. By introducing three different cases based upon genetic counseling specialties within the classroom setting, we aim to better prepare students for seeing similar types of patients in subsequent clinical rotations.Table 1. Genetic Counseling Sessions StructurePediatric Module1 Pediatric Counseling 1: Genetic Counseling: What to Expect? 2 Pediatric Counseling 2: Discussing Duchenne Muscular Dystrophy and Inheritance 3 Pediatric Counseling 3: Considerations for Carrier Testing 4 Pediatric Counseling 4: Carrier Test Results-1 5 Pediatric Counseling 5: Carrier Test Results-2Cancer Module6 Cancer Counseling 1: Pretest Counseling Session 7 Cancer Counseling 2 – Return Appointment 8 Cancer Counseling 3: Results AppointmentPrenatal Module9 Prenatal Counseling 1: Family History Part A 10 Prenatal Counseling 2: Family History Part B 11 Prenatal Counseling 3: Gina Declines Testing 12 Prenatal Counseling 4: Gina is Unsure about Testing 13 Prenatal Counseling 5: Gina Wants Testing 14 Prenatal Counseling 6: Gina’s Reaction 1 15 Prenatal Counseling 7: Gina’s Reaction 2 16 Prenatal Counseling 8: Preliminary Test Results 17 Prenatal Counseling 9: Final Test ResultsPrenatal Module ImplementationAfter reviewing several online applications, we decided to initially host the modules within a centralized learning management system. Implementation of the prenatal module began within the genetic counseling curriculum at the University of Minnesota in Fall 2016. The first-year genetic counseling students (N = 10) were required to watch the prenatal module videos over a four-week period and answer the corresponding questions. The prenatal module was divided into four segments:Collecting Family History Information (Family History Part A and B)Discussing Genetic Testing Options (Gina Declines Testing, Gina is Unsure about Testing, Gina Wants Testing)Test Results (Gina’s Reaction 1 and 2)Discussing Results and Next Steps (Preliminary Test Results, Final Test Results)Each segment contains two or three videos representing the different components of a counseling session. Initially, students watched the videos in their entirety and then watched short clips with corresponding questions.After the students completed the entire prenatal module, they were given a 13 question online survey to assess their perceptions of this teaching tool. The survey included six closed-ended questions in which students rated the module on a five-point Likert scale (excellent, good, adequate, marginal and poor). This portion of the survey addressed:How well the stimulus questions related to the video clipsThe clarity and difficulty level of the questionsWhether the module helped students apply knowledge gained in clinical observation and classroom instructionThe overall usefulness of the moduleTwo additional questions, rated on a four-point scale (very comfortable, comfortable, somewhat comfortable, and little or not at all comfortable) assessed students’ comfort level in addressing patients’ informational and psychosocial needs before and after completing the module. The survey questions align with the course objectives, namely, to provide students with background information and skills needed to begin the clinical training component of their education as well as with an opportunity to practice the skills needed in the clinical training component of their education.A forced-choice (yes/no) question was included in the survey to determine whether students would recommend this teaching tool for future genetic counseling students. The remaining four questions were open ended, asking about students’ perceptions of the advantages and disadvantages of the prenatal module compared to clinical observations and classroom instruction (lectures, class discussions, and role plays). At the end of the survey, students had the opportunity to provide additional comments about the module.Student Assessment of Prenatal ModuleAll ten of the genetic counseling students completed the prenatal module, and nine completed the survey (Table 2). The prenatal module assessments were rated on average as “good” with respect to difficulty level and the ability to apply classroom material and clinical observation experience to completing the on-line clinical scenario questions. Ratings of the module questions’ clarity and extent to which they related to the video clips were on average between adequate-to-good.Table 2. Student Evaluations of Genetic Counseling Prenatal Module (N = 9)QuestionMean (SD)MedianExtent to which questions relate to video clips3.11 (0.93)3Clarity of question wording3.56 (0.73)3Variation in question difficulty level4.11 (0.78)4Students’ perceived ability to apply classroom information to the on-line clinical scenario3.78 (0.67)4Students’ perceived ability to apply information learned from clinical observations to on-line clinical scenario4 (0.71)4Using a 5 point Likert Scale: Excellent = 5, Good = 4, Adequate = 3, Marginal = 2, Poor = 1Prior to module completion, the average student comfort level addressing patients’ psychosocial needs was somewhat comfortable (mean = 2, SD = 0.5, median = somewhat comfortable) (Scale: 4= very comfortable to 1 = little or not at all comfortable). After the module was completed student comfort level increased by 0.67 points to an average of 2.67 (SD= 0.5; median = comfortable).Students identified several advantages to the online module over clinical observations. Almost everyone mentioned they liked the capability to “pause or re-watch a section that may have been confus[ing]…” and make observations or connections they otherwise would have missed. One student wrote, “[You] catch different aspects of the session, such as the way things are phrased that you might not catch in a clinical observation.” Additional advantages include: the modules presented the opportunity to “really analyze the situation…” and allowed students to “assess their knowledge and understanding by completing the questions.” One student mentioned that a major advantage of the module is the ability to “follow a patient through all of their appointments with the genetic counselor to see the full picture, whereas in clinic you might not get the chance to be there for every appointment.” Several students also mentioned that they liked the ability to think through and consider how they would respond to certain situations without the pressure of being in clinic.In addition to advantages over the clinical observations, students identified advantages of the module over classroom instruction as follows: Most commented that they enjoyed the ability to work on the module on their own schedule and were not limited by time restraints. A typical classroom challenge concerns the “difficult[y] to review topics multiple times” and “sometimes things go too fast [in the classroom] and a student is unable to take detailed notes.” In contrast, using the module, students could review the material as many times as they wanted. Additional comments included that the module was superior to classroom instruction in regards to practicality and application. Students commented:The classroom instruction is not as practical as the module. With the module you can see how the information you learn in class is applied to the profession of genetic counselingThe module was a more realistic example rather than talking about sessions in classWith classroom instruction, it is harder to imagine actual sessions and what they would look like or the various ways that patients might respond to informationThe students also noted several disadvantages. The main disadvantage noted was the inability to talk to one’s “supervisor or instructor before and after the session.” Introducing the online modules removed students from the physical environment of a clinic. As genetic counseling encompasses more than the actual sessions with patients, removal from the physical setting precludes student observation of the preparation for a genetic counseling session and post-session follow-up. The module only shows the direct contact with the patient in the sessions and does not illustrate the larger scope of a genetic counselor’s responsibilities within the clinic. Being in clinic also “allows you to see other aspects such as working with sonographers and other paperwork…” In the future, additional vignettes could be added to these types of clinic activities and interactions could be added as separate vignettes to show these “behind the scenes” types of activities and interactions. Other disadvantages included the hypothetical nature of the module: the lack of a “real interaction between patients and a counselor,” and “’[actual] patient emotion[s]… are much more intense… in a clinic setting.” In summary, perceived disadvantages are that the online module eliminates immediate communication with supervisors, diminishes collaboration between students, lecturers and supervisors, fails to portray other genetic counselor responsibilities, and is somewhat contrived.Although the module had its drawbacks, every student indicated it should be used as a teaching tool for future genetic counseling students. To improve its effectiveness, the students suggested completion of the module prior to their clinical observations. Another common suggestion was having students watch the video on their own time and then to review it “… [during] class so we could talk about ideas” and “come up with questions.” Regarding module formatting, students offered a few suggestions. For time management purposes, they recommended adding the number of questions and estimated time for completion to the beginning of each module section, or playing the entire video and having it “pause and pop up a comment box when there is a question” associated with that portion of the video.Discussion/Future DirectionsIn summary, students perceived the module as clearly executed and comprehensible. By flipping the clinic and the classroom, we have enabled students to use precious classroom time to review clinical cases aligned with relevant course work. Students regarded stimulus questions associated with the counseling sessions as useful in allowing them to directly apply the knowledge they had gained from their clinical observations and classroom instruction to a clinical scenario. The module appeared to increase students’ comfort level addressing patients’ psychosocial needs, although further research including a control or comparison group is necessary in order to conclude that the differences in comfort level are attributable to the module. Finally, the students perceived more advantages to using the module than disadvantages.A limitation to the assessment of the prenatal module is the limited number of responses due to small class size. Only ten first year genetic counseling students viewed the module, and of these, only nine completed the survey, resulting in limited generalizability of the findings. A small sample of this type may also be more vulnerable to “demand characteristics” (e.g., giving more favorable ratings as the course instructor is one of the authors of the module).Improvements to the website will be addressed in future versions of the modules. There were technical difficulties such as one of the video clips not aligning with the question being asked. This could have altered students’ views of the module and resulted in lower ratings for module clarity. In order to improve upon the prenatal module, the video clip will be fixed and the number of questions associated with each section and the approximate time to completion will be added.The next step in flipping the classroom requires implementation of the pediatric and cancer modules. If these modules prove to effective, like the prenatal module, then all three modules can be used as a prototype for the development of additional cases representing different, more diverse medical indications, as well as cases from more specialized genetic counseling clinics. These modules will aim to obviate some of the observational time needed during initial clinical rotations, thus freeing up clinical sites and supervisors.ConclusionsBy flipping the clinic, we hoped to enhance student engagement and learning, and provide an orientation to clinical practice. While there are advantages and disadvantages to the online prenatal module, overall the module appears to be beneficial and a “value-added” supplement to both in-class and clinical supervision experiences. Module based learning cannot replace clinical observation and classroom instruction. However, the types of genetic counseling modules described herein may reduce the amount of clinical observation time needed and thus help with diminishing the supervision bottleneck. Module based learning is a valuable tool that can be used in conjunction with previously established teaching strategies in order to increase class sizes in genetic counseling graduate programs.References1. Srinivasan M, Wilkes M, Stevenson F, Nguyen T, Slavin S. Comparing Problem-Based Learning with Case-Based Learning: Effects of a Major Curricular Shift at Two Institutions. Acad Med. 2007;82(1):74-82. doi:10.1097/01.ACM.0000249963.93776.aa.2. Jamkar A V, Burdick W, Morahan P, Yemul VY, Sarmukadum, Singh G. Proposed Model of Case Based Learning for Training Undergraduate Medical Student in Surgery. Indian J Surg. 2007;69(5):176-183. doi:10.1007/s12262-007-0016-23. Mishra K, Snow-Lisy DC, Ross J, Goldfarb DA, Goldman H, Campbell SC. Evaluation of a Case-Based Urology Learning Program. Urology. 2013;82(6):1207-1210. doi:10.1016/j.urology.2013.09.007.4. Van Dijken PC, Thévoz S, Jucker-Kupper P, Feihl F, Bonvin R, Waeber B. Evaluation of an Online, Case-Based Interactive Approach to Teaching Pathophysiology. Med Teach. 2008;30(5):e131-6. doi:10.1080/01421590801932210.5. Rybarczyk BJ, Baines AT, McVey M, Thompson JT, Wilkins H. A Case-Based Approach Increases Student Learning Outcomes and Comprehension of Cellular Respiration Concepts. Biochem Mol Biol Educ. 2007;35(3):181-186. doi:10.1002/bmb.40.6. Qualtrics. Qualtrics. http://www.qualtrics.com. Published 2005.Chapter 8Learning through Generative ExplorationBrad Hokanson and Jody NyboerKeywordscreativity, generative learning theory, learner generated content, divergent thinking, creative problem solving, situated cognition, learning community Introduction How does one develop creativity in learners? How does one get learners to have the courage to find their own way and to make their own discoveries? How do you get them to openly research and experiment with ideas, and to generate their own learning? How could learners, guided by faculty, generate the content of a course? Here we present some of the “how” learning occurs in our own course through exercises, problems, and challenges.When I first signed up for this class, I wasn’t really sure what to expect. … I knew I wasn’t the most creative person in the world, there was a lot for me to learn, but I didn’t expect to learn as much as I did from this class…. Overall this class has taught me a lot more about myself than I thought it would, and it has taught me things I didn’t expect. (student comment)Beginning from our experience in design studios, our pedagogy is based on learners developing much of the content. Learning must be an exploration where it is better to discover than to be told. We want our students to be personally interested and invested in their work and for them to be motivated and self-driven.Generative learning theory describes the process of learners constructing meaning by making connections between their existing knowledge, beliefs and experiences and stimuli (Hanke, 2012; Wittrock, 1991). Here we are using “generative” to describe the creative actions and problem solving of the learners. It is a process where learners generate the ideas, the results, as well as much of the content materials of the course.Current practices in educationSociety often views education as being the dissemination of information. And many courses follow this pedagogical model, centered on presenting content. The learning of students and the success of a course are commonly evaluated by the amount of information that is retained. Economics and class size encourage simplified evaluation models, which are often automated. And, for that matter, many of our students have been trained to expect education in a simplistic format and corresponding styles of instruction.Generative learning theory describes the process of learners constructing meaning by making connections between their existing knowledge, beliefs and experiences and stimuli. Here we are using “generative” to describe the creative actions and problem solving of the learners. It is a process where learners generate the ideas, the results, as well as much of the content materials of the course.This same curricular texture can be observed both online and offline, as online courses often develop from face-to-face models. In both, the focus is on declarative knowledge and the learning activities are generally passive. Students receive information through lectures, live or on video, and through reading, participate in some discussion-based activities, and are tested in their knowledge of the material. While there is a range of activities such as written assignments, most learning is in terms of reception and retention. The learning is often low-level and ineffective. And cheap.For a generation, a stated goal of education has been higher order thinking skills, to be achieved through the development of curricula, course work, assignments, and active learning that is complex and grounded. However, regardless of which version of Bloom’s Taxonomy (Bloom, 1956, or Anderson, et. al., 2001) is used as a basis, today most learning experiences continue to focus on remembering and retention. That is, declarative knowledge.In more progressive learning experiences, “active” learning is typically thought of as discussing and writing-based work, but it seldom goes beyond those methods. An attitude of passive learning still restrains the curiosity of students, and limits an aptitude for investigation.In design and problem-based learning, the complexity and grounded nature of the projects adds a depth, a complexity to the experience. Learning then includes the information about the problem, the development of investigative skills, and the building of problem solving capabilities. The skills developed in such a learning environments frequently are long lasting.There are domains of learning that employ more challenging and complex learning models, disciplines which focus on active learning. They include music, art, theatre, dance, writing, lab sciences and other problem oriented fields. Our experience is in the field of design, where faculty assign projects tailored to the skills and understanding of the learners. These are generative forms of learning. Necessarily, learning is scaffolded to match learning capability. In design and problem-based learning, the complexity and grounded nature of the projects adds a depth and a complexity to the experience. Learning then includes the information about the problem, the development of investigative skills, and the building of problem solving capabilities. The skills developed in such a learning environments frequently are long lasting.Generative learning experiences help students gain initiative and confidence in their own explorations and experiments. They are richer and more authentic. The secondary learning that occurs changes their personal epistemology, as investigation and initiative are more inherent in their knowing, and which are applied to other areas of study.Another instance where this class has affected the rest of my life would be my overall confidence when I’m in public places has been very positively affected. I used to be shy when I was around people I didn’t know but after having to go out in public and do things I would have never even thought about doing before has helped me to be more outgoing. (student comment)Generative Learning Applied: Course FocusThis writing focuses on a University of Minnesota course on creativity. Creativity as a skill is highly regarded but seldom taught. Academic standards generally give lip service to developing creativity as a broad based skill, but it’s not one which is commonly taught like other skills such as writing, public speaking, or piano. Creativity is a valuable skill which is useful in all domains, and one which can be developed in learners (Scott, et. al., 2004).What makes our class different from most other courses at the university is its goal, the development of a skill in the learner. The course is not about creativity, rather is focused on making learners more creative. The methods of the course include a variety of challenges, exercises, and projects that engage the learner. This is learning through generative exploration, where learners’ ideas, projects, and findings provide much of the learning material for the course. As with a traditional design studio, the learners themselves generate the subjects of discussion and interaction. In this process, there are responses to challenges posed by faculty, who in-turn offer guidance, not answers.Creative Problem Solving has taught me how to stretch beyond my limits in other aspects of my life…I have learned something immensely important when it comes to problem solving: not always going with your first idea. Because of the divergent and convergent thinking we have been using throughout the semester, I have been applying those same concepts into my other classes as well as my job as a Features Intern at the Pioneer Press. When I have an initial idea for an assignment or a pitch or article I plan on writing, I will stop and take some time to develop the idea and see if I can build off of it in any other ways. Knowing that the first idea is hardly ever the best idea, I have been able to drastically improve my work in other aspects of my life. (student comment)Our course, both online and in person, focuses on learning that is active, engaged, and grounded. The teaching methodology is based on studio courses, and has evolved from our backgrounds in design and architecture.The course seeks to involve learners directly, physically, and personally. Most of the work of the assignments, even in the online versions, must be completed in person. This builds on the theory of situated cognition where all knowledge is “situated” in activity, where actions must be completed for learning to occur (Brown, Collins, Duguid, 1989).The course builds understanding by exploration and experimentation. Education in some other fields shares much of this idea; for example, the exploration that happens in dance can be compared to the learning that occurs in the painting studio. It’s easy to see a parallel with creative writing classes with repeated drafts, editing, presentations, and completion of finished results.The creativity class is understood to be playful, personal, exploratory, and experimental. Intrinsically, it is personally engaging. We contend the resultant playfulness and exploration leads to much deeper learning and deep personal engagement.Course MethodsThe course presents a conceptual shift, an different orientation from most other courses. The students in the class must develop the habit to go beyond the first acceptable answer, and to seek other answers that might be better in some way. This is centered in the phrase “the only wrong answer is one answer.” While grammatically awkward, it pushes learners to generate more possibilities. Victor Papanek put it better when discussing the design fields: “Design as a problem-solving activity can never, by definition, yield the one right answer: it will always produce an infinite number of answers…” (1971, p. 5).This is a change in the paradigm of the solving of problems, where the goal is often to find the solution to a given problem. This new paradigm recognizes that there are many possible, alternative solutions, and that one can develop many answers and must make selections using ambiguous, ill-defined, and changing criteria. This is a mature form of thinking better aligned with the higher order thinking skills of Bloom’s Taxonomy, for it requires synthesis and analysis regarding problems and situations, and identification of why, where and when more than the one right answer is appropriate.People who pursue creative skills will gain a fluency with ideas, an ability to generate numerous alternatives to a given prompt. They will also develop a tolerance of ambiguity and an ability to see the value in alternative solutions. They will grow in the ability to cognitively elaborate and to generate different types of ideas and solutions. And they will discover the courage to share and improve these ideas. Developing these traits shapes the methodology of the course to be very active, both in the classroom and online.Every aspect of the course is designed to build the skill in learners to develop and consider more ideas, alternatives, and possibilities. For example, attendance is taken by idea-generation exercise such as the development of a number of thumbnail sketches for t-shirts, instead of a simple roll call. This is practicing divergent thinking, which is a central aspect of creativity.Learners develop a tolerance for change and ambiguity, for generating ideas that are different from their peers and to build a courage for new ideas. The habit to vary is a long-term goal of the course. Students are trained to make different choices and to expose themselves to different ideas.The design of the course seeks higher levels of learning, through a learning process that is more complex and challenging, and subsequently, learning that will increase retention and which will develops skills such as synthesis, analysis, and creativity.A Focus on Divergent ThinkingIn class and online, students engaged exercises to practice and stretch their capability to generate multiple possible solutions. These are varieties of the classic Alternative Uses Test, where the challenge is to generate several possible uses for a variety of objects, such as bricks, blankets, newspapers, and paper clips. Integrated into each lecture, and available as an online drill, it provides development and reinforcement of skills in divergent thinking. Using that basic technique, I have been able to integrate this course’s material into everyday life. One example of this was at work – when my boss and I were inquiring with each other to solve a manufacturing problem that came about, I literally said, “this is what I learned in class,” and I wrote down 5 different ways the problem could be solved. 4 of the 5 potential solutions had not even crossed his mind. This example shows how simple a string of thoughts can be created – having one good idea is having many good ideas, as we’ve learned in class. Listing the initial idea, then tweaking that idea 4 times ended up saving the company a significant amount of money and is still being implemented today. (student comment) The play and playfulness that are inherent in the course open the imagination. This attitude allows learners to be less dogmatic about their thinking and less parochial. It allows ambiguity and alternatives to creep into their mindset, and to accept absurdities as part of the world. This develops the ability to find and define problems, and to eventually solve them in a creative manner.This process of learning builds an initiative for investigation and an internal motivation, a curiosity, as it were. Students develop an understanding of investigation, an aptitude toward trying out ideas, and skills such as synthesis, analysis and problem solving. We have not seen this initiative in our other courses.Principal Course AssignmentsThese challenges are all designed to be completed in learners’ own context; in school, at work, or at home. Their creativity becomes something that is integrated in their lives, authentic, and very personal. These challenges are similar to an online soil science educator asking learners to complete experiments on a container of dirt where they live. Investigations can then be conducted to better understand the nature of soil (or challenge), an understanding which is “grounded” in their own context.Most of the course focuses on a series of challenges called Do Something Differents or DSDs. Each DSD has a topic, such as to “eat” or “wear” that is actively solved by the learner. In completing the project, a process for creativity is modeled through the assignment structure. For each challenge, the learners must develop a number of different concepts, initiate and plan the experience, photograph the results and prepare a written description. Presenting their DSD in front of their peers helps students become less sensitive to possible embarrassment, and builds a psychological callus.These challenges are broadly defined, and learners provide their own constraints and definitions. They learn to redefine problems and find new challenges. The flexibility of assignments encourages students to develop their own ideas, to define and redefine the challenges they are facing. Shared with the rest of the class, the project results provide the material for the course. Like a design studio, the work is discussed and critiqued. The complexity of this set of assignments grows through the term. The DSDs become more complex changing from the simple challenge to “wear” something different to a conceptually more difficult such as “give”. Later assignments involve others as partners, as inspiration, and as non-class members who are convinced to be different in some manner.These challenges are all designed to be completed in learners’ own context; in school, at work, or at home. Their creativity becomes something that is integrated in their lives, authentic, and very personal. These challenges are similar to an online soil science educator asking learners to complete experiments on a container of dirt where they live. Investigations can then be conducted to better understand the nature of soil (or challenge), an understanding which is “grounded” in their own context.Critical to the success of the model is for learners to actually execute their ideas. Doing the projects of the class embodies the thinking, and builds on the complexity of grounded problems. Ideas often fail when they face real life and need to be adjusted, just as experiments don’t always work out in the science lab. This helps develop an ability to adjust and change, as well as courage in the face of failure.It is situated cognition, in that knowing is inseparable from doing. Cognition cannot be separated from context. They’ve got to ride the bike, do the dance, and walk the walk.Creativity in a Learning CommunityBy nature of a definitional inclusion of originality, creativity involves doing things that are contrary to popular choice or expectation. It is about ideas that are out of the mainstream of a “culture of conformity” (Sternberg Lubart, 1995), against a current which is particularly strong for undergraduates. They seek to fit in with their peers and to not be different. More than in other courses, the support from other students in the course for others’ divergent efforts is essential. However, with an on-campus but completely online course, developing the normal in-person social engagements of class work is a challenge.Developing a learning community is valuable, beyond the discussion boards prevalent in online classes. The course assignments help learners to connect with each other through pairs, teams, and shared experiences. Building a real, physical community in an online class is difficult but very rewarding. While students in a face-to-face class have the opportunity to meet one another, online students on campus are often invisible to each other. Our response to this is the Purple Ribbon Assignment which began in an online version of the course. Each student was given a length of purple ribbon at the beginning of the term and told to wear the ribbon around campus. The ribbon, often tied to their knapsack or coat, identified members of the class. Each student then took selfies around campus with other ribbon wearers, fostering personal connections that can’t be made online. The assignment tagged students as being members of a select group, providing an open “introduction” springboard to many new connections. Friendships, study groups, and lunch buddies all developed, adding some of the qualities of an in-person course. This has subsequently been adopted in the face-to-face course with comparable success.I made a lot of connections through this assignment, and met a lot of different people with different majors. I think what I found the coolest about this assignment was being able to spread my own community at the U and who I interact with, which is a hard thing to do at a school of this size. It was a good way to break the ice, and now I have more connections of people I run into regularly in my schedule. It is always good to make new friends. (student comment) Changing Course StructuresThere are substantial shifts that occur with this class, moving from passive to active learning, and from simple dispensed information to complex, student-constructed learning. Learners generating materials is part of the process; this requires guidance by faculty with an eye for scaffolding learners, and not toward covering specific content. In this, learners can develop a richer learning experience, a broader ecology of their knowledge.Our research shows that the creative skills learners develop are retained after the course. Testing of past course participants has shown that over 90% of the measured increase in creative skills is still present two years after completion of the course (Im, Hokanson, Johnson, 2015).As other faculty also seek deeper and more complex learning in students, it’s a valid question whether such techniques such as student-generated content, problem-based learning and social tagging can or should be applied to their courses. While the specifics of any topic will affect which modes are applicable, each of the techniques may have some value to other disciplines and can be transferred.Coming into the course, I had no idea just how much my mindset would change through this class. I have primarily noticed how my standards of doing even the practical things, as well as my confidence among others have greatly changed and increased. My standards of doing the practical have become higher in their originality. That is, what I once considered routine, I now see as an opportunity to be a new experience.(student comment)As faculty, as teachers, each of us must strive to more fully engage learners. And we must find effective ways for engagement including grounded challenges and problems that are personal, engaging, and generative. Making and creating solutions involves learners cognitively and deeply. We observe that these techniques have value in our courses, and look to hear of comparable developments in other disciplines.ReferencesAnderson, L. W., Krathwohl, D. R., Bloom, B. S. (2001). A Taxonomy for Learning, Teaching, and Assessing: A Revision of Bloom’s Taxonomy of Educational Objectives. Allyn BaconBloom, B. S. (1956). Taxonomy of Educational Objectives: The Classification of Educational Goals: Cognitive Domain. Longman.Brown, J. S.; Collins, A.; Duguid, S. (1989). “Situated Cognition and the Culture of Learning.” Educational Researcher. 18 (1): 32–42.Hanke, U. (2012). Generative Learning. In Encyclopedia of the Sciences of Learning (pp. 1356-1358). Springer.Im, H., Hokanson, B. Johnson, K. (2015). Teaching Creative Problem Solving Skills: A Longitudinal Study, Clothing and Textiles Research Journal. [33] 129-142.Papanek, V. (1971), Design for the Real World: Human Ecology and Social Change. Chicago Review Press.Scott, G., Leritz, L. E., Mumford, M. D. (2004). The Effectiveness of Creativity Training: A Quantitative Review. Creativity Research Journal, 16(4), 361-388.Sternberg, R. J., Lubart, T. I. (1995). Defying the Crowd: Cultivating Creativity in a Culture of Conformity. Free Press.Chapter 9Translating Knowledge to Engage Global Grand Challenges: A Case StudyDaniel Philippon, Barrett Colombo, Fred Rose and Julian MarshallKeywordsinterdisciplinary, trans-disciplinary, active learning. competencies, grand challenges, co-curriculum IntroductionIn fall 2014, the University of Minnesota adopted a new strategic plan to transform both research and teaching to address society’s grand challenges. This plan outlined a series of adaptive changes to infuse grand challenges across the curriculum. Recommendations included a new vision for liberal education requirements, including the development of introductory to advanced course categories that, through the arc of a student’s career, would allow varied and sophisticated engagement in a challenge. To accomplish these goals, the University created a Grand Challenge Curriculum (GCC) designator, placed at the Provost level, and started a phased implementation of new cross-collegiate curricular and co-curricular arrangements for revenue and cost-sharing.In fall 2015, the University focused its efforts on testing and developing promising models that could bring the plan’s ambitious goals to scale across one of the largest land-grant research universities in the U.S. In particular, an initial priority was to develop new introductory seminars focused on a variety of grand challenges. Each of these introductory seminars was meant to integrate multiple disciplines into course design and pedagogy and promote “active learning” approaches in the classroom.At the same time, no matter the particular grand challenge, the University emphasized that each introductory GCC seminar should focus on “competencies that prepare students to recognize grand challenges, assess possible points of intervention, and take action” (Cheng, Schively Slotterback, et al). As a result, while the content of each course varied greatly, each course was to have a common interest in building students’ ability to take practical, real-world action in response to the challenge and to communicate their ideas effectively.This article outlines the efforts of a group of faculty and instructional staff teaching across a number of GCC courses to develop appropriate curricular and co-curricular structures to support these foundational competencies.This instructional team eventually coordinated the instruction of four courses during Fall 2015:GCC 3001 – Can We Feed the World Without Destroying It? Profs. Jason Hill (College of Food, Agricultural, and Natural Resource Sciences) G. David Tilman (College of Biological Sciences)GCC 5003 – Seeking Solutions to Global Health Issues Prof. Cheryl Robertson (College of Nursing)GCC 5005 – Global Venture Design—What Impact Will You Make? Prof. Julian Marshall (College of Science Engineering) and Fred Rose (University of Minnesota Institute on the Environment)LA 3003/5003 – Climate Adaptation for Minneapolis Vincent deBritto (College of Design)Three of the courses were part of the University’s GCC. The GCC has a number of requirements for course design that promote an interdisciplinary approach to problem-solving, including a requirement of co-instruction from faculty across at least two colleges and the expectation that faculty actively collaborate (and not simply switch off) on instruction during class. Further, as introductions to a particular challenge, all three GCC courses had no content prerequisites—advanced undergraduates or graduate students were welcome to apply, no matter their major program of study. The fourth course, though taught from the Dept. of Landscape Architecture, shared many of the same educational objectives of the other courses.Project BackgroundDistinct content needs As a specialized introduction to a global grand challenge, each course necessarily devoted 30-70 percent of course hours to developing students’ literacy in the challenge itself. While some content did overlap from course to course, the specific challenge addressed by each course generally required that students be immersed in distinct disciplinary and professional knowledge. For example, in “Can we feed the world without destroying it?,” students were introduced to literature from ecology, population studies, the history of science, agricultural policy, genetic engineering, agroecology, biosystems engineering, and political economy. In contrast, “Seeking Solutions to Global Health Issues” introduced students to cutting-edge understandings of emerging pandemic threats, disease ecology, refugee policy, women’s and gender studies, the history of immigration, health systems policy and institutions, public health policy, international political economy, and veterinary population health.Despite these differences, instructors felt that some overlap in content was necessary for students to engage in each of their grand challenges. Most notably, nearly all courses required that students study one particular overarching grand challenge—the impacts and feedbacks affecting their issue due to global climate change.Common emphasis on real-world, practical impact Despite widely different kinds of content, all of the courses emphasized the need to move from a survey of the challenge—in effect, an understanding of the problem—to taking some kind of action to respond to the challenge.The co-curricular pilot project originated from the shared belief that, despite the differences between particular grand challenges, students would require a common set of trans-disciplinary skills to develop solutions to these challenges. In other words, the assumption was that common approaches might exist in helping students translate their knowledge about grand challenges into creating some tangible impact on those challengesInstructors also assumed that practical impact could take many forms, depending on the challenge and also on the students’ theory of change. Social change can occur from a broad range of actions. However, whether developing a public art installation, a public health implementation plan, a social venture business plan, or a narrative essay, the team assumed that there might be some common approaches in moving from problem to action.Curricular StructureWhile taught by separate instructional teams, and designed according to the goals of those instructors, all courses were to feature the following common four core elements:Creating a Knowledge to Impact Skills Lab: A new “skills lab” featuring modules focused on different trans-disciplinary skills, to be integrated within each course, or taught separately as a 1-credit option.Proposing solutions that respond to some aspect of the wider challenge:Within each course, interdisciplinary teams of students would work in a studio environment, coached by instructors and mentors, to create solutions that could potentially be implemented.Holding a common workshop event across multiple GCC courses:All courses would emphasize early feedback for student teams through studios and presentations. In addition, all students would present at a final workshop event, including students from other challenge courses, and proposals would be reviewed by experts drawn from UMN faculty and the MSP professional community.Supporting students in developing solutions beyond the class: Students would be guided toward funding to implement their ideas, enroll in other courses at the University, etc.These goals and this curricular structure remained largely intact, with some modifications:Creating a Knowledge to Impact Skills Lab We developed and piloted the “Knowledge To Impact” (KTI) curriculum to prepare students with skills and frameworks that would enable them to use the knowledge gained in their GCC course to create plausible solutions. Solutions could take varied forms according to the needs of the challenge, including sustainable business or non-profit models, public policy, public health interventions, media, or technical solutions. To more easily meet the needs of varied GCC course structure and timelines, KTI was developed in a modular fashion, with each module focused on a different trans-disciplinary skill. The only prerequisite for KTI was that students were currently enrolled in another GCC course.The lab introduced students to skills for engaging with other people effectively to address a challenge. These included both oral and written communication training, developing effective presentations, elements of storytelling, and integrative leadership. The lab also provided introductions to skills that allow students to better approach complex problems. These included systems dynamics, quantitative reasoning, and design thinking and its array of skills (i.e., problem scoping, stakeholder mapping, prototype and iteration).Figure 1: Curricular Structure Overview for Core GCC Course Design1Proposing solutions that respond to some aspect of the wider challengeOur project team shared the following learning outcomes. Students will:understand the path from idea to impact and apply this knowledge through a project in their GCC courseapply design thinking to identify a suitable problem relevant to their Grand Challengeidentify a problem they wish to focus on, and create an actionable problem statement relevant to their Grand Challengepropose a solution that addresses the problem in some wayshare their proposal with practitioners in their challenge area who could advise on how to improve itAcross all four courses, students worked in teams of 3-5 to develop projects designed around these outcomes. Examples of proposals include:A food delivery social venture plan to deliver produce to university studentsA pilot social network focused on farmers in the Upper Midwest around the issue of aquifer sustainabilityA food waste reduction curriculum pilot for Minneapolis Public SchoolsA public health intervention plan to improve patient satisfaction in UgandaA microfinance business plan designed to reduce incidence of diarrhea in Uganda Holding a common workshop event across multiple GCC courses Our end-of-semester KTI workshop event featured approximately 125 students from across three GCC courses, plus the participating LA 3003 course. The goals of the workshop included:Students’ awareness of their participation in the University’s broad effort to address society’s grand challenges through teaching and researchAn emphasis on providing feedback for refining each team’s projectsA video, produced by the University of Minnesota, featured GCC students preparing for and participating in the event.Figure 2: Curricular Structure for Team Project Development2Supporting students in developing solutions beyond the classThe UMN Strategic Report on Grand Challenge Curriculum notes that students ultimately require an arc of opportunities for developing their capacity and knowledge. To move toward this integration, our suite of courses adopted the following strategies:A Grand Challenge Studio Session: Scheduled office hours with instructors/mentors to review developing proposals. Mentors were drawn from the MSP professional community and available to interested students beyond their particular course.Deliberate efforts to make students aware of other course and non-course opportunities (i.e. funding competitions, co-curricular opportunities, etc.) at the University.EvaluationWe conducted a post-class evaluation in the three GCC classes, in part by asking students to describe their experiences on post-it notes. Six themes emerged:“Challenging.”The most common word students used to describe the Grand Challenge Courses was “challenging.” Twenty students across the three classes wrote “challenging” and others added a host of similar words such as “stressful,” “overwhelming,” “intense,” and even “nerve wracking.” Although the term “Grand Challenge” may have originally been meant to describe large, complex global problems, it simultaneously came to represent the steep learning curve that many students faced in these courses.“Eye-opening.”The Grand Challenge Courses opened some students’ eyes to new perspectives on themselves, the world, and how change really happens. A number of students found the classes to be innovative (8), eye-opening (7), thought-provoking (5), provocative (3), and similar words, including “mind-blowing” and “game-changing.” When these students placed their post-it notes describing their GGC, we asked if similar words would describe most of their other UM experiences. In each case, they quickly responded (sometimes with laughter) that they would not. Verbally, many shared that it changed how they saw themselves and what it takes to change the world.“Inspiring.”The Grand Challenge Courses motivated some students to be global change makers. Some students’ words reflected that they were personally inspired by the GCC. They found the experience “inspiring” (9) and used other related words such as “beyond important,” “meaningful,” “character-building,” and “empowering.” As one student shared: “I came to the class having an idea of how complicated the problems are, and was curious how the class would break down those problems. What changed for me personally was the idea that complicated problems can be addressed if we work on them relentlessly.”“Enjoyable.”Some students expressed that they enjoyed the experience. Multiple students found the Grand Challenge Course format enjoyable. Despite the experience being sometimes “stressful,” students expressed that it was “fun” (6) and used similar words such as “engaging,” “refreshing,” “amazing,” and “rewarding.”Unconventional Class Structure. Twenty students, especially in the “Feed the World” class, chose to comment that they found the class format to be “unconventional.” Students also selected words indicating that the structure of a GCC was different from the norm. They called the GCCs “unorthodox,” “unconventional,” and even “crazy.” There were also a number of words commenting on the “collaborative” (5) and “hands-on” class structure.Metacognitive Class Content.Some students focused their word choices on the type of thinking that the Grand Challenge Courses required. Student words also focused on the metacognitive skills needed to be global change-makers. In the “Feeding the World” class, seven students wrote the class was “scientific.” Other words reflected the need for “empathy,” “creativity,” “ideation,” and “deep thinking.”Limitations of EvaluationIt should be noted that this report represents a snapshot of student reflections and is not meant to substitute for a formal class evaluation. Also, the students presented their feedback with the instructors present, which may have constrained critical commentary. What students did share, however, reveals important thoughts about the Grand Challenge Course potential that should be combined with the standard Student Rating of Teaching (SRT) survey to gain a fuller view of the GCC progress and challenges. As one student commented: “We really are the guinea pigs of the university Grand Challenge Courses. It’s neat to be part of the beginning!”Future DirectionsCurricular ideasAdapt the curriculum to other kinds of challenges, especially non-environmental grand challenges.Encourage teams to develop other types of “products” where a social venture plan, policy intervention, etc. may not be not appropriate. Other “products” that are equally likely to create change include narrative pieces (i.e., videos, articles, photo-essays, etc.) and epistemologies for approaching these challenges. This would mirror the approach of a pilot GCC course, HCOL 3805H – Our Common Waters: Making Sense of the Great Lakes, taught by Prof. Dan Philippon (CLA) and Prof. Deborah Swackhamer (School of Public Health and Humphrey School).Implementation ideasRefine the core aspect of the project with the existing set of courses, and integrate in other courses. We integrated this curriculum into one GCC course in Spring 2016 and, to varying degrees, is part of all four GCC courses during the current Fall 2016 semester.Expand the Studio concept for Fall 2016 to include sessions for four GCC courses, as well as students in other curricular and co-curricular programs.Continue the GCC end-of-semester workshop.Adjust the quantity of KTI content to fit instructors’ needs: where possible, integrate further, and where KTI is competing with other priorities, reduce the number of hours devoted to KTI.Increase internship opportunities for students to continue work past the course.ReferencesCheng, R., Schively Slotterback, C., et al. “Grand Challenges—Curriculum. Report of the Strategic Planning Workgroup University of Minnesota Twin Cities.”AcknowledgementsThe authors would like to acknowledge the participation of a broad group of instructors essential for implementing this project. The instructors in the three courses that participated in this pilot included Prof. Vincent deBritto, Prof. Jason Hill, Prof. Cheryl Robertson, and Prof. G. David Tilman. Prof. Jim Toole developed the evaluation framework, facilitated feedback, and summarized the results. Institute on the Environment Managing Director Lewis Gilbert provided strategic advice on the project. In addition, Prof. Bill Arnold and Prof. Elizabeth Wilson were part of the original proposal team and advised on the early formation of the project. Finally, we are grateful for project funding from the University of Minnesota Institute on the Environment and the University of Minnesota’s Center for Educational Innovation.1Like a GCC course that covers a myriad of topics and their complex interactions relative to the grand challenge at hand (the vertical part of the “T” in this visual), the KTI likewise introduced a suite of capacities necessary for acting on these challenges (the horizontal parts of the “T”). Clearly a one-credit class (or equivalent) cannot begin to cover all trans-disciplinary capacities required to meet a challenge in any significant depth, as each is a discipline in and of itself. However, the intent was to introduce these skills so students could gain basic competency, apply these skills to the problem at hand, and most importantly, gain awareness of how to use these skills to tackle big projects in the future. The “Core GCC Course Design: Interdisciplinary grounding in a particular grand challenge” represented here is an example from GCC 5003: Seeking Solutions to Global Health Issues.2An illustration of a student team’s path from a general interest in a problem to a refined proposal for taking action. The structure of the Knowledge to Impact co-curriculum relies on techniques from design thinking, organizational development, and communications. While proposal development is cyclical, relying on multiple iterations based on feedback, ultimately teams must consider questions outlined in the figure.PART IIIReframing AssessmentChapter 10Self-Assessment in Language Courses: Does In-Class Support Make a Difference?Gabriela Sweet, Sara Mack and Anna Olivero-AgneyKeywordsself-assessment, language learning, learner agency, Fink’s taxonomy of learning, tools for practice and training in evaluating skills, guided sequence of in-class activities and collaborative learning, student-reported benefit, accuracy in self-assessing, self-regulatory learning BackgroundThe fact that language learners benefit from a critical examination of their own strengths and weaknesses is well established. Oskarsson (1984) conducted a literature review of this “relatively new field” (p. 1) over 30 years ago. By that time it was already documented that learners who are trained and supported in self-assessment become reliable evaluators, with judgments correlating closely to external measures such as standardized tests and instructor estimates. Additionally, Oscarson (1989) suggested a list of key rationale for using self-assessment procedures in language learning. In the intervening years, most of these once “speculative” (p. 3) rationale have been confirmed empirically, showing that self-assessment increases the level of awareness and improves goal orientation (Moeller, Theiler Wu 2012). The most recent studies continue the trend of documenting self-assessment as a powerful tool to enhance language learning (Ziegler 2014; Dolosic et al 2016).While the efficacy of self-assessment is now widely accepted in general terms, few studies have examined large-scale efforts to provide training and support in self-assessment for learners at the postsecondary level. With approximately 1.5 million students in college-level language courses other than English annually (MLA 2015), there is a clear need to more closely examine self-assessment in this context, not only documenting its benefits, but critically analyzing how self-assessment is conducted and what procedures are most beneficial to this population of learners. To address this question, this paper reports on an experiment using two versions of a large-scale self-assessment protocol developed at the University of Minnesota, Basic Outcomes Student Self-Assessment (BOSSA).BOSSA is a proficiency-based standardized protocol for language learning that makes self-assessment meaningful to students through concrete grounding in performance. It consists of tools for speaking and writing, appropriate for language learners at different proficiency levels. The core BOSSA experience is generally delivered in a computer lab and lasts approximately 50 minutes. Its innovative design consists of an articulated, guided sequence of activities plus a self-assessment questionnaire. Students benefit from trying out language tasks related to course outcomes and practice evaluating their skills as they identify their language strengths and areas that need work, and set specific goals (personal or course-related) to address the gaps they perceive.Further, the evidence-generating protocol provides actionable feedback to students, instructors, and programs. Learners are provided with an individual data report, and can monitor their progress over time as they work towards achieving goals. Instructors and programs can use aggregate data from a class-level learner profile to reflect on course and program proficiency goals. Measurable outcomes include increased awareness, learner agency, and engagement in learning. The evidence-generating protocol provides actionable feedback to students, instructors, and programs. Learners are provided with an individual data report, and can monitor their progress over time as they work towards achieving goals. Instructors and programs can use aggregate data from a class-level learner profile to reflect on course and program proficiency goals. Measurable outcomes include increased awareness, learner agency, and engagement in learning. This article focuses on the BOSSA protocol for examining speaking skills. There is also a full BOSSA protocol for writing that similarly pairs language performance tasks, opportunities for reflection, and a self-assessment questionnaire. In addition, there are self-assessment questionnaires for listening and reading, with plans to develop the full protocol for these skills. The speaking protocol includes the elements of BOSSA noted above, including communicative language performance tasks, reflection first alone and then with peers, practice in developing the skill of self-assessment, and culminates with students completing the self-assessment questionnaire, rating their language skills. As with the BOSSA writing protocol, the questionnaire results are electronically tabulated and learners receive a customized report of how their (self-assessed) skills compare with course goals, along with suggestions on how to improve.The BOSSA experience has resulted in a paradigm shift, promoting shared responsibility for learning between students and instructors. This aligns with Fink’s (2013) call for a shift from a content-centered paradigm to one that puts the learner at the center. He suggests that course design be grounded in the premise that learning effect real and lasting change in one’s life. In the content-centered approach, instructors “respond to the question of what students should learn by describing the topics or content that will be included in the course” (p. 60). In contrast, Fink’s learner-centered focus creates a more favorable environment for learners to retain what they learn in the class and increasing the likelihood that they will have the tools and motivation needed to keep learning after the course ends (p. 63).The shift initiated by BOSSA corresponds to several categories of Fink’s learner-centered paradigm. These categories (Integration, The Human Dimension, and Learning How to Learn) emphasize higher-order metacognitive skills. BOSSA’s guided discussions and self-assessment questionnaires align with Fink’s concept of Integration, providing an opportunity for students to transfer what they are learning in class to other contexts. Learning becomes more meaningful when students can align course objectives with personal goals.Those same BOSSA elements also highlight the Human Dimension, raising student awareness that learning another language can, in Fink’s words, “…affect their own lives and their interactions with others” (p. 90). In providing a frame for students to reflect on their learning and fill the gaps– and provide each other with support and feedback in that process– BOSSA promotes the value of collaborative learning and problem-solving.The most significant connection between BOSSA and the learner-centered paradigm is in the Learning How to Learn category. Fink (2013) outlines three distinct forms in this category: “becoming a better student, learning how to inquire about […] particular subject matter, and becoming a self-directed learner” (p. 91). BOSSA supports all three of these aspects, focusing on the process of learning, via its integrated set of activities. In other words, through BOSSA, students are at the center, learning to articulate not only what but how they learn language, thus taking charge of the trajectory. In the words of one student, “It was a huge wake-up call for where I was in my language learning. This has definitely helped with my proficiency and confidence.” And from an instructor, “I see that my students are much more self-aware of their own skills, struggles, and goals in relation to the target language. I think this self-awareness is empowering.”The BOSSA protocol was originally developed for students in Intermediate Spanish courses, then extended to serve students in French, German and Italian, and today also supports students of Arabic, Chinese, English, Hmong, Korean, Portuguese, and Russian. After its first year of development and use, Language Flagship funding for the Proficiency Assessment for Curricular Enhancement (PACE) project supported further development of BOSSA. BOSSA currently serves an average of 3,000 students each academic year. This large-scale use within and across languages has operationalized self-assessment in second language classes at the University of Minnesota, and standardized the process so the benefits of self-assessment are available to students regardless of the language they study.Original Version of the BOSSA ProtocolThe original version of the BOSSA procedure requires learners to complete the majority of the protocol tasks in a computer lab. Many courses integrate two 50-minute class periods into the course calendar for the protocol, one at the beginning of the semester and one at the end, when students can see the progress that they have made.The lab session has six components. First, students watch a two-minute video that introduces them to self-assessment and familiarizes them with criteria they will use later to evaluate their skills. The video standardizes delivery of the message across languages and levels. Instructors facilitate the session, rather than being at the center as “teacher”. Students warm up with a short conversation activity, activating their second language prior knowledge. The instructor provides the topic, usually based on what students have been working on in class at that point of the semester.The second component is the Speaking Practice Task (SPT). Students complete the SPT via computer, using headphones and microphones. The in-house created content supporting this component is delivered through an easily navigable web interface that guides learners through the language performance task. Each SPT is contextualized, giving students a reason to use the language, and consists of three steps. Each step has a slightly different communicative context. Students have the opportunity to show what they can do by responding to prompts, within the frame of a general topic and task. For example, beginning students introduce themselves and talk about their likes and dislikes, while students at the advanced level make explanations and provide solutions for situations that include an unexpected complication. Backward design aligns tasks (topics and functions) with what students are expected to be able to do by the end of the instructional level. So the SPTs are tied to specific, tangible outcomes or objectives on the course, program, and national standards levels. Responses are recorded. There are four SPTs, each calibrated to a different proficiency level; they last between nine and fourteen minutes (depending on level).Listening and using criteria to reflect is where practice and training with self-assessment begins. After completing this third component, the SPT, students listen to their own recordings and use criteria to reflect on how well they spoke in completing the language tasks. The criteria address the following areas: vocabulary and fluency, complexity, accuracy and comprehensibility, and communication strategies. They are aligned with course objectives and criteria used for summative assessments. A short description explains to students what each criterion entails. The recordings provide students with something concrete for them to refer to, making it easier to reflect on their abilities. The criteria introduce students to thinking about, evaluating, how they use language; for example, whether they actually do use complete sentences when they speak. Again, this is what makes BOSSA unique: it integrates a real time performance opportunity with self-assessment. Students report that being able to hear themselves speak is very valuable; for some, it’s the first time they’ve done this. For most, this quick check-up serves as a wake-up call that comes from themselves, not from their instructor. And thus, self-awareness begins. This is what makes BOSSA unique: it integrates a proximal performance opportunity with self-assessment. Students report that being able to hear themselves speak is very valuable; for some, it’s the first time they’ve done this. It’s like giving themselves a quick check-up; a wake-up call that comes from themselves, not from their instructor. And thus, self-awareness begins. At this point in the experience, students have started to think about what they can do, grounding their awareness via reflection on their performance on a set of concrete tasks. Now they are ready to work in pairs in the fourth component, deepening that metacognitive self-awareness by processing ideas with a classmate. This is where the paradigm starts to shift: as students reflect, they begin to think about their studies as a place where they can be in charge and where they can take control. The collaborative pair work is an additional opportunity for students to think about how language learning works, and share notes about their strengths and challenges and set specific, realistic goals for improvement. They begin to see that it’s up to them to do the work, taking charge to make changes, translating reflection into action. This is all done in English, the native language of the majority of our students, so that students can reflect more deeply.The fifth component is class discussion. Students lead the discussion about strengths and challenges within the larger framework of proficiency in their own words. As they talk, instructors record the discussion on the whiteboard and reformulate what students say in terms of the criteria, thus providing language that allows them to articulate and reflect on how they learn with concrete examples. The board is photographed so that students have a record later. The instructor also facilitates the class discussion, as needed, to help students understand what is realistic in terms of proficiency goals per course expectations. Students report that this is a very valuable component of the protocol: “In your head you think that everyone is better than you, and then in class discussion… you don’t feel as bad, now I am not alone and I can work on that.” By finding common ground and exchanging suggestions, students take ownership and gain autonomy. Doing something individually, then in pairs, and then, in the large group, builds confidence.In the sixth, and final component, students use an online, self-assessment questionnaire to rate their speaking ability. This step brings together all of the knowledge gained in the lab session: students have a specific idea of their skills in light of their actual speaking performance in the SPT, and they have new knowledge (from the discussion) that helps them assess those skills in terms of general language learning goals, goals specific to the course, and their own individual goals.After the students complete the online self-assessment questionnaire, a summary of their results appears on-screen. Results consist of two main parts: a score and a proficiency level based on their self-reported performance rating, and a description of what language learners like themselves can do at their self-rated level of proficiency (additional support toward realistic expectations). Students also get results via email, which includes their responses to individual items, specific learning strategies for their individual learning preferences and goals, and information on other proficiency levels (not just the one at which self-assessed).The Modified BOSSA ProtocolIn Spring 2016, a pilot funded by an Experiments in Learning Innovation (ELI) grant explored whether learners can use the self-assessment protocol, forging their own path to self-awareness and self-regulated learning – on their own turf and on their own terms – as effectively as in the classroom. This effort to adapt the BOSSA protocol for use outside of the computer lab was in response to feedback on the original BOSSA protocol gathered in focus groups with students and instructors. The focus group data indicated that instructors were concerned that making time for the computerized self-assessment protocol in the lab would decrease time available for other course content. In addition, they wanted students actively using and reflecting on language use outside the classroom. Students wanted to have their own recordings, something difficult to arrange given the current delivery model in the lab. Furthermore, lab time in the computer labs can be difficult to book as nearly all language programs at the University of Minnesota frequently make use of the technology labs have to offer throughout the semester.The modified at-home delivery could not guarantee the built-in speaking practice and training in self-assessment before students evaluate their skills, precisely the element which sets BOSSA apart from other self-assessment protocols.In this modified BOSSA approach, the second round (near the end of the semester) takes place both at home and in the classroom, rather than completing all steps in the language lab. Students complete the SPTs at home, listening to their recordings and evaluating their skills using a worksheet. They also reflect on how they have improved since the first time they completed the SPT and what they did to improve. The following day, in class, students continue the reflective activity in pairs, sharing their notes. Class discussion follows and at the end of the class session students use computers, phones, or other electronic devices to access and complete the online self-assessment questionnaire, as in the original format of the protocol. Instructors were free to use a variety of approaches to realize the reflection. For example, some instructors assigned students to do the entire process (including completing the online self-assessment questionnaire) at home, and followed up with a brief discussion during the next class session. Others made the SPT optional, requiring only that students complete the online self-assessment questionnaire as a homework assignment. Instructors later reported partial completion of the assigned tasks, as it turned out that many students had opted not to do the SPT at home, or had not listened to their recordings and evaluated their skills. Therefore, they were not prepared to reflect on the proximal performance experience of the SPT, making it difficult to realize the benefit of collaborative learning during class discussion. In effect, the modified at-home delivery could not guarantee the built-in speaking practice and training in self-assessment before students evaluate their skills, precisely the element which sets BOSSA apart from other self-assessment protocols.Procedures, Results, and AnalysisQuantitative data analyses compared the efficacy of the modified BOSSA delivery with that of the original format. Approximately 340 students of a variety of languages and levels participated, with half completing the original, completely in the lab format (henceforth referred to as “R2”, for “Round 2 in lab”) and half completing the modified, mixed at-home and in-class format (“R2M”, for “Round 2 modified”). Data were gathered via a survey measuring how self-assessment impacts student self-awareness, self-regulatory learning, and performance. Survey responses were also analyzed to determine to what extent the awareness students gain through doing BOSSA practice activities is associated with increased accuracy in self-assessment. To measure this, students’ self-assessment was compared with their skills as measured by the American Council on the Teaching of Foreign Languages (ACTFL) test battery.Preliminary analyses show that in both versions of the BOSSA protocol, self-assessment in language learning leads to a higher level of learner agency and awareness of the language learning process (greater than 70% in each). However, the effect is especially strong when learners are provided with training in rating their skills through the original format in which all six components are done in the lab during class time.On a survey collecting student-reported benefit to using the protocol after the end-of-semester BOSSA session, students using the two formats responded very differently to an item focused on the practice and training aspects of BOSSA (see Figure 1, next page). Three-quarters of those using the R2 format said that the opportunities to practice (including the SPT, pair work, and class discussion) helped them feel prepared to complete the online self-assessment questionnaire, while only two-thirds of those using the R2M format reported that practice helped. Students qualified the overall experience similarly, with 74% of those using the R2 format reporting that self-assessment helped their language learning, as compared to 67% of those using the R2M format. Further, 82% who did the BOSSA protocol in the lab said that they could identify both strengths and things they needed to work on, while 76% of the mixed at-home and in-class users reported self-awareness around these areas. In addition, both groups reported that they had made changes in their language learning practices in response to using self-assessment, with slightly higher reported benefit from those using the R2 format.In terms of accuracy, learners provided with training and regular opportunities to rate their skills in the lab setting self-assess more accurately than those who engage in the process in a mixed at-home and lab setting. ACTFL proficiency ratings using the Oral Proficiency Interview – Computer (OPIc) test were examined together with students’ self-assessed proficiency ratings to determine to what extent students’ self-evaluations of their skills matched up with their performance. Data were analyzed first by aggregating per semester of instruction, and then comparing both overall mean (second-semester and fourth-semester levels) and by-person ratings from the two groups. The R2 group for whom there were both ACTFL data and self-assessment data consisted of 182 learners of Arabic, German, and Portuguese (78 second-semester and 104 fourth-semester students); the R2M group was made up of 161 learners of French, Korean, and Russian (38 second-semester and 123 fourth-semester students) for whom there were both ACTFL data and self-assessment data.Figure 1: Student-reported benefit: Using self-assessment to support language learningStudents in both the R2 and R2M groups tend to assess their speaking skills lower than how they are rated by ACTFL (as illustrated in Figure 2) with fourth-semester learners (S4) in general self-assessing a bit more accurately than second-semester learners (S2). Representing ACTFL proficiency levels using integers (e.g., 3=Novice High, 4=Intermediate Low, 5=Intermediate Mid), and based on semester of instruction mean ratings, learners who had the full support of an integrated BOSSA session (R2) in the lab evaluated their speaking skills within .36 of how they were rated, while those using the mixed format (R2M) evaluated their skills with less accuracy, or within .39 of their ACTFL rating (aggregating second- and fourth-semester data).Looking more closely at how individual students self-assess their speaking skills as compared to how they are rated by ACTFL, the data show a high degree of accuracy at or within one sub-level on the proficiency scale (for example, self-assessing at the Intermediate Low level and being ACTFL rated Intermediate Low or Intermediate Mid) for all learners, using both formats (see table below). Those completing all activities in the lab self-assess slightly more accurately than those who did some activities at home and some in class. The margin for second-semester learners (N=78) was very slight, with only a tenth of a percentage point of difference: R2 users (students of Arabic) self-assess with 94.9% accuracy at or within one sub-level as compared to R2M users, (students of Russian) at 94.8% accuracy. There is more divergence in the data for fourth-semester learners (N=227): R2 users (students of German and Portuguese) self-assess with 99% accuracy at or within one sub-level as compared to R2M users, (students of French, Korean, and Russian) at 96.6% accuracy (see Table 1).Figure 2: Comparison of ACTFL OPIc and self-assessed mean ratings: Semester-level accuracy (R2 R2M)Table 1: Person-level accuracy in self-assessment (R2 R2M)Too Low (2 or + sub-levels)Too Low (1 sub-level)ACCURATEToo High (1 sub-level)Too High (2 or + sub-levels)Semester 2 R2 n = 781.333.344.916.73.8Semester 4 R2 n = 104021.144.233.71Semester 2 R2M n = 382.631.642.121.12.6Semester 4 R2M n = 123121.959.315.42.4Discussion and Conclusions3.1 Limitations of the study Data from the student feedback survey and performance data, comparing ACTFL proficiency ratings to how students self-assess, show that the fully supported session in the lab yields higher student-reported benefit, learner agency, and accuracy. However, the high degree of variation present in the mixed at-home and in-class R2M format limit the generalizability of these data. There were differences in terms of administrative oversight, specific methods of delivery, student compliance, and, to some degree, deviation from original purpose in terms of BOSSA goals (all elements that were not possible to control in this study situation). At the same time, this kind of variation is to be expected in real-life circumstances; administrators of any large-scale delivery will encounter similar variation.3.2 Future directions Through the Experiments in Learning Innovation grant, the researchers were able to look more closely at how BOSSA works for students and programs. It is clear that while the BOSSA protocol is adaptable for use outside of the language lab setting, more investigation is indicated in order to increase the efficacy of the modified delivery mode. BOSSA was created with the understanding that it would be an ongoing iterative dialogue between end-users and developers. As the protocol’s reach continues to expand as a support for language learning at the University of Minnesota, it has potential to serve as a model that is easily adaptable for use by other programs of study.At the same time, we – the instructors and researchers who created BOSSA – need to continue to grow through exploration and research, refining theory and design through our findings, and implementing improvements. BOSSA is a workable solution for a wide range of language programs that are charged with supporting students of all levels, using a standardized approach that is highly structured but still able to be customized to address specific needs. We recognize that this balance between standardization and customization is vital. For example, we have learned that some students don’t feel qualified to rate themselves; they don’t see yet that self-assessment is a skill like any other, which improves with practice. It would serve those students well to focus more on building confidence through regular use of the self-assessment process. And instructors, while acknowledging the clear benefit of the protocol to students, say they would like support in terms of how and when to use additional self-assessment opportunities throughout the semester. BOSSA helps students to become agents of their learning through the cycle of using the tool for themselves and reflecting analytically. In the future, increased practice, training, and regular use of self-assessment, especially those integrated as class activities, seem key to continuing to foster learner agency.Conclusion We’ve seen clear benefits of using self-assessment to support second language learning at the University of Minnesota. However, a close analysis of how students experience self-assessment –whether in a supported, guided, lab class session atmosphere using standardized delivery, or through completing some activities on their own and others with peers in the classroom setting – shows that a climate that supports reflective learning in the lab class session setting results in higher learner agency and awareness, as well as accuracy in learners evaluating what they are able to do. We’ve learned that self-assessment must strike a balance between students realizing that they can take control of their learning, while still benefiting from external guidance. This guidance can be from instructors directly or through the messaging that BOSSA’s lab delivery provides, especially in the careful scaffolding of activities where reflection leads to self-awareness, and training prepares students to complete the self-assessment questionnaire at the end of the session.In contrast, data show that a lack of standardized delivery in terms of how self-assessment is introduced and conducted results in students experiencing less benefit insofar as how they see their own ability to be in charge of their learning and in their ability to accurately evaluate their skills.An examination of what procedures are most beneficial to our learner population indicates that students benefit from the guided critical examination of their own strengths and weaknesses in the lab setting. Here is feedback from a student after her first time participating in a BOSSA lab session:With that type of power [from a BOSSA session] and the change in your consciousness, it’ll make you pursue your learning in a different way, in a more intentional way, because not only do you have the tools to track what you’re doing, but you want to do better. And you are also able to identify your personal goals. At the same time it makes your learning more practical, because you think, “How is going to apply in my life, outside of the classroom?” Not just “How well am I going to do on the next test?” It’s just really extremely effective as a learning technique […] because it makes us intentional learners.In conclusion, the guided training and support that students receive in self-assessment in a lab setting, integrating a proximal performance experience as basis for reflection on ability and coupled with collaborative metacognitive discussions of learning, work in tandem to promote increased learner agency, empowerment, and awareness about what learners themselves can do. While there are multiple ways to deliver and manage this process, this study suggests that the most beneficial method is one in which learners learning about self-assessment, and have opportunities to build their self-assessment skills via interaction with their peers in the lab class session setting.ReferencesDolosic, H. N., Brantmeier, C., Strube, M., Hogrebe, M. C. (2016). Living Language: Self‐Assessment, Oral Production, and Domestic Immersion. Foreign Language Annals, 49, 302–316.Enrollments in Languages Other Than English in United States Institutions of Higher Education, Fall 2013. David Goldberg, Dennis Looney, and Natalia Lusin.Fink, L. D. (2013). Creating Significant Learning Experiences: An Integrated Approach to Designing College Courses. John Wiley Sons.Moeller, A. J., Theiler, J. M., Wu, C. (2012). Goal Setting and Student Achievement: A Longitudinal Study. The Modern Language Journal, 96(2), 153-169.Oskarsson, M. (1984). Self-Assessment of Foreign Language Skills: A Survey of Research and Development Work. Council for Cultural Cooperation.Oscarson, M. (1989). Self-assessment of Language Proficiency: Rationale and Applications. Language Testing, 6(1), 1-13.Ziegler, N. A. (2014). Fostering Self‐Regulated Learning Through the European Language Portfolio: An Embedded Mixed Methods Study. The Modern Language Journal, 98(4), 921-936.This research was supported in part by grants from the Center for Educational Innovation, University of Minnesota, and the Language Flagship Program Initiative of the National Security Education Program, U.S. Department of Defense.Chapter 11Developing an Easy-to-Use Learning Analytics Tool to Facilitate Effective Course and Curriculum DesignXavier Prat-Resina, Molly Dingel and Robert Dunbar Keywordslearning data analysis, curriculum development Environment that Inspired Tool DevelopmentThe University of Minnesota Rochester (UMR) opened its doors to students for a Bachelor of Health Sciences (BSHS) degree program in fall 2009 with the mission of utilizing evidence based teaching strategies and providing a personalized learning environment for students. This mission was supported by a requirement that the UMR tenure track faculty conduct research in the area of teaching and learning, and an institutional commitment to generating a large database of student data that could be used for that faculty research. Initially, the database was envisioned to be one part of a larger system named iSEAL (intelligent System for Education, Assessment, and Learning) (Neuhauser, 2012) which would serve as a “curriculum delivery and research tool” (iSEAL Design Documentation, 2010) in the place of Moodle, Blackboard, or similar Learning Management Systems (LMS). However, more than just an LMS, the goals for iSEAL were to store data and assess student progress (Neuhauser, Mondal, Lin, 2009, slide 7). In practice, iSEAL was storing a lot of data about student performance, but accessing that data was challenging.In early 2013, faculty took the initiative to try to build a tool that would allow us to share, visualize and start understanding the student data that we were accumulating. A typical challenge when implementing learning analytics at the department or curriculum level is that those interested in answering the learning research questions (usually researchers) are not necessarily the same people who deliver the courses (faculty). Also, these faculty and/or researchers are seldom involved in developing the source code for data representation, storage, and analysis. We believe that at UMR we were in a privileged situation where the three groups of people described above were either the same group of people or at least could work very closely.Our primary goal was to bring together an interdisciplinary group of faculty to identify a set of relevant research questions, indicators, and databases in order to develop and test a tool that would allow faculty to connect learning analytics with course design. We proposed that developing an effective and easy to use learning analytics tool that allows faculty to intelligently and dynamically explore the large amount of student and course data would increase the probability of faculty adoption. This Browser of Student and Course Objects (BoSCO) would help faculty connect course design and learning outcomes at both the course and curricular level and, therefore, increase both the efficiency and effectiveness of course and curriculum design (Dunbar, Dingel, Prat-Resina, 2014).Development of Tool with Focus on Faculty ResearchOur project, funded by the University of Minnesota Provost’s Undergraduate Digital Technology Grant, began with a focus group in which the entire research faculty at the University of Minnesota Rochester met to identify types of data they would like to collect related to their areas of research. Specifically, UMR’s ten tenure-track faculty met on 19 June 2013 for 4 hours. During the meeting, each completed a survey about types of data they would be interested in, noted where that data was located, completed a survey about characteristics of a successful curriculum, and participated in a mediated discussion about their responses on each survey item. Following an analysis of the data collected from the surveys as well as the subsequent discussion, it was discovered that our faculty wanted data in five priority areas (listed from highest to lowest): Information about student development and attitudes, course content and curriculum, metrics of performance, demographic information, and postgraduate information.Collecting, storing, and visualizing the types of data prioritized by our faculty involved significant challenges that could be grouped into three broad categories: Primary among these, variability in faculty data priorities meant that all data needed to be accessible through multiple databases or collected into a single database. Second, the faculty needed a convenient way to explore the database(s). Third, the data explored by faculty needed to be secure and managed according to our group’s existing IRB protocols. Given that our group already had active IRB protocols, we focused our tool development on collecting and visualizing data relevant to our faculty. Once we understood faculty data needs, an evaluation of the characteristics of existing resources ultimately resulted in the separation of the iSEAL database from the iSEAL LMS functions and construction of software (BoSCO) that allowed faculty to explore the iSEAL database.Development of BoSCOThe iSEAL “all in one” model was a necessary step in realizing that we were trying to carry out too many distinct projects simultaneously. There were at least three different products in one: developing a competent LMS from scratch; building a database to consolidate longitudinal student and course data; and designing a learning analytics tool for faculty research. The obvious limitations in budget and the fact that the three products required very different technologies made it clear that we needed split the project into three separate segments, and to outsource as much development as possible so that we could focus on the learning analytics research.This change of direction in the project was motivated by time and budget considerations, but was also a natural step that loosely fits the model-view-controller (MVC) software architectural pattern. In the MVC model different representations of information are separated into different interconnected parts. These three connected parts or levels could be labeled as “data collection”, “data storage” and “data representation” as they represented in figure 1. Below we describe in detail the characteristics of each of these three levels.Level 1. Data collection: The LMS market is very dynamic as it tries to adapt to new technologies emerging every year in higher education. The University of Minnesota officially adopts a LMS to support system-wide, and employs a team of developers to maintain a working LMS for all students in all of the UMN campuses. (In 2017-2018, the University will switch from Moodle to Canvas as the system-supported LMS.) It became obvious that UMR had no resources to compete with, or even match, this expensive and fast development type of project. Therefore, it was appropriate to take advantage of the resources that the University of Minnesota system offered to our campus. By doing so, we could outsource the LMS development by relying on the system-supported, and invest time in connecting the external LMS with student data. In addition, outsourcing the LMS also gives the necessary academic freedom to faculty to choose any software that they see fit to deliver the pedagogical experience they consider most appropriate.Figure 1: A general scheme of the project divides it in three interconnected levels, data collection, data storage, and data representation.Thanks to the collaboration with the University of Minnesota Moodle team, our own web development team built an interface to query Moodle’s API (Application Programmer’s Interface) and, at the end of each semester, import the student grades from Moodle. In principle, this same strategy could be implemented with any other LMS that our faculty would choose.As seen in Figure 1, demographic data usually resides in the data warehouse used by admissions and the student development teams across the university system. Data such as ethnicity, sex, age, high school background and other demographic indicators are typical fields in the dataset, which we imported into our local database.Finally, there is another source of data that must be imported into our database from the “data collection” level (see Figure 1). Surveys and attitudinal student data may reside outside of the LMS because they are administered by other software (Qualtrics, Act engage, Google Forms, SurveyMonkey, MS Excel…), the survey is given in paper, or is given to a group of students that are not linked to a specific course. Survey data could also be gathered from a whole cohort, alumni or any other subgroup of students. It was, therefore, necessary to be able to import other kinds of data into our system.Importing student, demographics and survey data along with IRB consent tracking constitutes the main content of our local database that we labeled as the “Academic Analytics Data Warehouse” (AADW) described in the next paragraph.Level 2. Data storage: The essential design of the Academic Analytics Data Warehouse was already in place several years ago when iSEAL was the LMS of choice. It is important to point out that the structure of the AADW could not be similar to a regular database of a LMS. That is, in a regular LMS the same course offered in different years or different instructors become two unrelated instances in the database. In fact, the Moodle software at the UMN system changes the entire database each academic year as new Moodle versions get upgraded. Our AADW could not change every year; it had to remain the same database to preserve the longitudinal picture. Each offering of a course in our campus had to be related to the course delivered in previous years. By doing so, each semester would add another layer of courses in the pre-existing scaffold of the BSHS curriculum.In addition, as part of its initial development, iSEAL’s web interface houses a copy of the IRB protocols, and offers students the option to opt in or out of the research study at any point during the semester. Each semester the system keeps track of the students that have given their consent to be part of the study.Even though a lot of work and good ideas had been implemented in the original iSEAL, a new database had to be created, the AADW. We needed the flexibility to import the different types external data listed in the “data collection” section, as well as to connect the learning and course objects with summative and formative assessment. In other words, in the initial iSEAL, the course activities and the student grades associated to those activities were not linked, and that had to be corrected.Level 3. Data representation: A key component of this whole endeavour is finding an answer to the question “How should we make data available to faculty?” First and foremost, it is fundamental that FERPA, the IRB protocol, and faculty data ownership are respected throughout the whole process.As figure 2 schematically shows, there are currently two types of access to the data, a manual download/upload and an automatic querying through Tableau visualization software (www.tableau.com). These two types of access to the data are used in the two different versions of BoSCO and they serve different purposes.BoSCO beta version: The manual download/upload via BoSCO was the first strategy developed, which is described elsewhere (Dunbar, Dingel, and Prat-Resina, 2014). BoSCO beta is kept available and under development as it currently works as a beta version for testing new ways to represent data. One of the advantages of this more manual process is that an instructor can add additional data that currently does not reside in the AADW database. To that end, we have made available a “deidentifier” tool: faculty can upload an identified external grades file and the tool can substitute students’ identity with an unidentifiable code. The returning deidentified file also lists whether or not the students gave their consent to participate in the research protocol during that semester.The download option is also used when faculty want the deidentified data to be used for further analysis in other statistical software. For example, as the download format is one csv file for each course it makes it straightforward to upload data in the R statistical package.Figure 2: The two different strategies for data representation, “BoSCO on Tableau” and “BoSCO beta” have similar graphs but serve different purposes.One last advantage of this version of BoSCO is that since it is the user that uploads the data to the BoSCO beta interface there is no problem of data ownership because it never connects directly to the database, the user can erase the visualized data at any time.BoSCO on Tableau: The second way to visualize the AADW data, and the second version of BoSCO, is through Tableau, which is a more robust, stable interface and ready for mass consumption. Tableau directly queries AADW and, upon login, makes available to the user only the data to which they have ownership or access. Because different faculty own different types of data, one needs to build in Tableau a “viewer page” for each faculty member or each data owner.The two versions of BoSCO presented here serve two different purposes. Although licensing costs may make the Tableau interface less attractive, it is important for a development project like ours to keep a more stable and robust product, like the BoSCO on Tableau, separate from the more under development, internal use and testing “sandbox” software like the BoSCO beta version.Both versions of BoSCO shown in figure 2 have very similar types of graphical representations. Upon the selection of one or several courses, semesters and/or assignments, the user can represent the data using bar graphs and xy-scatter plots. In all cases, the selected data can be filtered by demographics indicators (sex, race, etc.) or by performance in other assignments such as ACT scores (See Dunbar, Dingel, and Prat-Resina, 2014 for a further description on types of representations).Even though BoSCO will provide a summary of basic statistical information about the selected student data (like means and standard deviation), as we have previously stated, BoSCO is not a statistical tool and contains no algorithms for learning analytics. Rather, it tries to represent in a simple and intuitive way the large amount of student data our institution accumulates through the years.Project OutcomeHypothesis Catalyst: This project has led to the creation of a tool that is at its core a way for faculty researchers to quickly explore large amounts of student and course related data. The tool that emerged does basic statistics and correlations but comprehensive analysis is necessarily dependent on the specific research questions of each faculty member. In the world of academic analytics, therefore BoSCO serves a distinctive but very useful role as a “hypothesis catalyst.” Using BoSCO, our faculty are finally able to navigate and visualize large amounts of data of various types. While this exploration does not provide a direct measure of learning, it can help formulate meaningful questions based on the landscape of student and course data. In this way, BoSCO is not dissimilar from a telescope that does not provide a direct explanation of how the craters of the moon are formed, but does let us know they are there.What follows is a specific example of how BoSCO has worked as hypothesis catalyst in our courses. For example, in one of UMR’s Chemistry courses, we offer students homework quizzes with several attempts. Students had often complained about the homework not preparing them well for the exam. Using BoSCO we were able to see that, in fact, the homework scores did not correlate with exam grades or final grades. All students obtained pretty high grades in quizzes but not necessarily in exams. However, BoSCO also showed us that the quiz score on the first attempt did correlate with exam grades. Students who performed poorly in the course would require more quiz attempts to achieve the same score as high course performing students. This phenomenon would give a false sense of understanding and accomplishment to low performing students. Therefore, we used BoSCO to hypothesize that if we decrease the number of possible attempts in quizzes in the forthcoming semester, the homework grade would better inform students, and make them aware of their realistic understanding of the subject.BoSCO has also been used to compare student’s grades across different courses in our curriculum, allowing us to see how students’ demographics, attitudes, or aptitudes correlate with performance in different subjects and over time. While the exploration of data at this level has not specifically led to the generation of any hypothesis or change in the curriculum, it has provided an opening for us to begin to think about changes we might want to make (and investigate) to support student success.OverviewIn this paper, we described the development of a learning analytics tool that may help faculty interested in visualizing and understanding the basics of course and curriculum student data. In our description, we paid special attention to the different stakeholders involved in the development, the types of data and their storage, as well as the necessary IRB protocols and data deidentifcation that must prevail at all times. For institutions interested in using our BoSCO approach, we made available the source code of the BoSCO’s version referred above as “beta version.” It is free to download on the GitHub software manager at https://github.com/xavierprat/BoSCO and can be installed on any modern web server. At this URL there is also a video tutorial regarding how to use BoSCO and the type and format of student data that it requires.ReferencesDunbar, R. L., Dingel, M. J., Prat-Resina, X. (2014). Connecting Analytics and Curriculum Design: Process and Outcomes of Building a Tool to Browse Data Relevant to Course Designers. Journal of Learning Analytics, 1(3), 220–240.iSEAL Design Documentation. (2010). iSEAL Design Documentation. University of Minnesota Board of Regents.Lund, B. (2012, November 16). UMR Software Breaks New Ground. Rochester Post Bulletin. Rochester, Minnesota. Retrieved from http://www.postbulletin.com/business/umr-software-breaks-new-ground/article_15bb3858-8db5-5fb2-a7b8-c9b74f1100b1.htmlNeuhauser, C. (2012). From Academic Analytics to Individualized Education. In A. H. Duin, E. A. Nater, F. X. Anklesaria (Eds.), Cultivating Change in the Academy: 50+ Stories from the Digital Frontlines at the University of Minnesota in 2012, An Open-Source eBook (pp. 188–193). University of Minnesota. Retrieved from http://conservancy.umn.edu/handle/125273Neuhauser, C., Mondal, A., Lin, Z. (2009, January). ISEAL. Rochester, MinnesotaAcknowledgementsWe would like to thank the Provost Office for financial support (The Undergraduate Digital Technology Grant). The authors are grateful to the University of Minnesota Rochester web development team, IT staff on the Twin Cities Campus, and all the faculty of the CLI whose input throughout the process contributed to the development of this project.Chapter 12Improving Performance and Reflective Learning through Video TechnologiesMary M. Rowan, Carol Flaten, Lori Rhudy and Nima SalehiKeywordsdigital technology integration, psychomotor skills video, self-assessment, peer feedback, reflection, instructor feedback, video annotation IntroductionIn 2007 the Carnegie Foundation for the Advancement of Teaching established a program for the comprehensive study of preparation in five professions, including nursing. In the report on nursing education, the researchers noted that graduates of nursing programs are “ill-prepared for the profound changes in science, technology and the nature and settings of nursing practice” and made several recommendations for transforming nursing education (Benner, Sutphan, Leonard, Day, 2010). At the time, Benner et al. found that much of the academic work provided theoretical knowledge for nursing but clinical practice was not well integrated. Faculty were encouraged to look for ways to more closely tie coursework to what actually happens in patient care. One of the specific recommendations was for increasing experiential teaching and creating opportunities for situated cognition, thinking-in-action, to address students’ need to understand the scope and purpose of nursing care interventions and how to analyze their own progress toward achieving safe patient-care goals. Furthermore, making explicit the connections from classroom content and laboratory practice to actual clinical performance would help students move beyond acquiring knowledge into actions requiring students to us knowledge, thereby integrating the academic work with clinical practice (Benner et al., 2010).The Carnegie report catalyzed our school to complete a major redesign of the curriculum with a plan for a shift in the approach to experiential learning activities. At about the same time, the School of Nursing initiated a significant remodel of laboratory and classroom spaces and redesigned some of the teaching strategies used in the new spaces. Plans included increasing the use of flipped classroom techniques, metacognitive activities, and opportunities for reflection. Shortly after the school’s newly redesigned laboratory opened, the authors received a grant to develop learning activities using digital technologies designed to enhance teaching and learning. One part of this grant project was the integration of student use of tablet computers for video recording of basic nursing skills as a form of performance evaluation and as a way to extend learning beyond the actual hands-on practice time spent in the laboratory. The pilot project for using video plus annotation is reported here.Use of Video Recording and AnnotationVideo of performance has long been used as a way to document progress and as a tool for providing peer or instructor feedback. In health sciences, video is often used as a component of high fidelity simulation experiences. However, the authors could find no published examples of the use of video in the context of a basic nursing interventions laboratory or examples of recordings paired with annotation as a reflective activity.Assisting students in their development as reflective practitioners has long been a challenge in undergraduate education. Metacognitive strategies are commonly used in classroom settings to deepen learning, but are less likely to be used in laboratory courses. The focus in most nursing lab courses is to teach students how to perform basic nursing interventions and psychomotor skills.Laboratory courses in nursing consist of experiential activities and a type of virtual clinical practice. The laboratory environment provides students with safe and accessible exposure to health care skills practice in order to prepare them for actual patient care in clinics, hospitals or home care settings. The laboratory experiences are typically limited to a few hours each week. Instructors have been challenged to find ways to extend the learning beyond the actual hands on practice during the lab session. Although video has long been used in education, the relatively recent availability of easy-to-use video annotation tools provides new opportunities (Rich Hannafin, 2009). Integrating video and annotation activities for self-evaluation, peer response and instructor feedback provides students with a way to extend and enhance the impact of laboratory practice. In addition, video recording of psychomotor skills practice sessions provides a concrete review of common health care errors and a more in-depth understanding of how individual students might be able to improve their nursing abilities.Self-Reflection and Peer ResponseNursing standards of practice in the U.S. are regulated by each state’s Board of Nursing and guided by the state’s Nurse Practice Act. Healthcare professionals (nurses, physicians and other healthcare professionals) are well aware of the need to reflect on and regulate their own practice. Nursing students must learn how to critically reflect on and monitor their practice in order to meet the standards of providing safe, high quality nursing care. There is an expectation that professional nurses will receive ongoing feedback from peers on performance. Furthermore, nurses need to know how to provide feedback to peers. Self-reflection and peer response activities during nursing laboratory courses prepare students for real life peer feedback in the nursing field. For this reason we developed course materials to teach reflection and peer response as part of the video project.Courses SelectedTwo courses were selected for the purposes of this project.Assessment and Beginning Interventions: Nursing Lab INursing Interventions Lab IIThese sequential courses are required during the first year of the nursing major, prior to any activities with actual patients. Each of the two courses identified included 7 sections of students, ranging from 15 to 24 per section for a total of 146 students. The ratio of instructors to students was approximately 1:8 in lab sections. Students spend 4 hours per week in the laboratory courses.Beginning nursing students need to achieve a minimum level of competence before they can begin clinical rotations in patient care settings. The selected courses included a wide range of hands-on practice activities that had the potential to be enhanced by the use of video for performance review.The intent of the project was to use digital technologies to enhance students’ awareness of their kinetic movements and manipulation of specialized equipment. This increased awareness would provide the platform for critical reflection. A challenge was determining how to integrate video activities into the selected courses. Several questions were identified.Recording which nursing interventions (skills) would provide the most benefit to students?What types of technologies would effectively support recording, annotating, and sharing activities?What preparation would faculty and students need to facilitate these activities?How could we assess the effectiveness of these activities?Faculty believed that teaching certain specific skill sets, such as sterile technique, could be improved. Sterile technique is an example of a complex skill set that requires understanding how one physically moves in and through an environment. Students often underestimate the complexity. Furthermore, faculty were aware that students struggled with sterile technique when they were in actual patient care settings and believed that a video with annotation would offer an opportunity to deepen the students’ mastery of the skills beyond actual hands-on practice in the laboratory. This topic was determined to be an excellent fit for the initial of the use of video paired with annotation for self-review, peer response, and instructor feedback. Other activities such as medication administration, urinary catheterization, and intravenous medication infusion were also selected. These four activities selected for the project are considered foundational skills for nursing practice, and embed the core concepts of sterile technique and safe medication administration. The plan was to limit each video to 3-5 minutes of performance and to focus on components of the skill that students had difficulty executing. The use of video was not initiated until the fourth week of the course to give students time to get acclimated to the lab environment and protocols. (After the initial pilot project the intravenous infusion video recording was dropped since it involved multiple tasks and required longer recording times.)Activity Logistics and Technology UseThe technology used for video and review included the following:Camera “app” on computer tablets (Apple iPad®)In class computer and projector (for small/large group debriefing)Fuse app for Camtasia (Techsmith® product used to upload video file to media server)Media Mill – university media serverVideoAnt – web-based annotation application for mobile and desktop devices that was developed at the College of Education and Human Development, University of MinnesotaFaculty designed a developmental approach to introduce the video activities to students. During the first experience, the focus was on using the technology. With each subsequent recording activity, the focus on assessment and reflection increased. Students worked in pairs, with each student taking a turn to practice the skill while their partner recorded the activity using the iPad® camera. If a third student was part of the team, they were directed to observe and take notes for feedback purposes until it was their turn to practice or record. After each activity was practiced and recorded, students met in pairs to watch the videos. Faculty provided a checklist that outlined critical elements of each skill. Students reviewed each video to determine where they had met criteria based on the checklists. Students were also required to assess the video for areas of deficiency. After the initial review students uploaded their video file to a university media server where it was linked to an annotation application and shared with an assigned peer responder, and with the instructor.Figure 1: Sample of video with AnnotationLinks to each video were added to an online collaborative document in the course website. Due to sharing settings in annotation program (VideoAnt), each video linked in the document could only be viewed by 1) the student, 2) the peer response student, and 3) the instructor. Peer responders and faculty would select the video, review it, and annotate the recording to provide feedback based on provided guidelines. Aspects of the skill performed well and areas for improvement were identified.Originally students uploaded the video files to Media Mill with technical support staff processing the videos, linking them to the web-based video annotation application, and adding links to the course website. This process proved to be very time consuming. During subsequent semesters the use of uploaded video diminished and, instead, emphasis was placed on using time during the laboratory session for students to debrief videos in pairs, then small 8-person groups with the instructor. In the small groups, instructors reviewed one or two of the videos as a basis for discussion and to respond to questions and on skills methods and approaches. To extend the learning, students were required to access their video after the laboratory in order to:Complete self-assessment reflection by assigned dateComplete peer response for partner by assigned dateView instructor feedback posted later during the weekPreparation and Support for FacultyAt the beginning of the semester the course coordinator and a technology support person met with the instructional team to review curriculum and video activities. The team reviewed assignment guidelines, discussed the purpose of integrating video recording, and practiced using the iPad® and VideoAnt annotation applications. An instruction guide was developed and provided for each application and the plan for technical assistance and support was described. All support materials were available to the instructional team and the students in the course website. Meetings to review plans for using VideoAnt technology were also scheduled for later in the semester.Preparation and Support for StudentsStudents were provided an overview of the purpose of video activities at the beginning of the semester. (Sample information in Appendix A.) The planned activities were identified in the weekly course outline accessible via the course learning platform. Course content was adapted to include information about the purpose of the video integration, self-assessment, peer response and instructor feedback. Clear, concise one page instructions were provided for the use of the iPad® and the process for uploading the recordings to the media repository. A sample video of the activity was linked in the directions on the course website. Another instruction document guided them through using the VideoAnt annotation application. Technical assistance was provided during each video-enhanced lab session. A tech support email and phone number was provided for the online self-assessment and peer response annotation activity that students completed after the lab session.Evaluation DataStudents were given surveys pre and post video recording activities at the beginning and the end of the semester for both courses. The pre survey provided baseline information. The post survey results after the first course showed an increase from pre to post responses indicating agreement with the benefits of video recording, self-assessment, and instructor feedback. Students were less positive about the benefits of peer response. This was also reflected during follow up focus group sessions with students noting that peers needed more instructor modeling and overall training to provide effective review responses. Survey questions and results are in Appendix B.Student focus group resultsStudents in focus groups expressed that the video recording process was helpful in learning and reviewing skills practice. They enjoyed reviewing the videos at their own pace. They liked having online feedback from instructors, and said they would like more instructor feedback. Many said they enjoyed both receiving and providing feedback from peers, noting that this gave them greater confidence and helped their learning. Overall focus group participants were happy with the video and annotation activities. Comments from students included:The feedback from instructors was really great.Good critiques by peers were very helpful.Made me more confident, could see improvementEnjoyed sharing it with my family. This gave me a feeling of pride.VideoAnt was easy to use. I was worried until I used it and then very happy that it was easy.Faculty Focus Group ResultsFaculty focus group results indicated that instructors felt students improved in their ability to provide peer feedback after they viewed faculty feedback. They suggested that faculty model feedback initially before expecting students to provide responses so there would be greater clarity on what constitutes effective feedback. Faculty felt that in-lab debriefing with small groups of 8 students provided the best use of the video activities, with the quality of student questions enhanced by being able to use actual student videos to elicit questions and discussions. Faculty observed that online instructor response using VideoAnt was beneficial but labor intensive, and noted that other faculty assignment feedback could suffer as a consequence. Therefore some time management considerations needed to be taken into account as video recording activities were integrated. Some faculty mentioned the following:Students became more critical in their peer feedback over time.Online instructor feedback allows faculty to give more feedback to individual students.Online feedback made faculty feel like they were not as “present” compared to providing feedback in person.With this activity, instructor feedback was delayed until several days after the lab activity occurred.Both faculty and students felt that the video recordings provided helpful information about motor skills and about more subtle body language and communication skills. Both focus groups indicated that the smaller debriefing groups of 8 students were preferable to having a larger whole class group for getting instructor input and making students feel comfortable about sharing videos with peers.Technology IssuesIssues encountered during the video process were with the following:Wi-fi connections were not always reliable, requiring students to log back in before continuing an upload to the media server.The upload time for the 3 – 5 minute video file was often long, and sometimes file uploads stalled in the process. A workaround for this issue was to have lab and tech support staff upload the video files after students had completed the lab and left the area. Complete technical directions coupled with onsite technical support staff helped mitigate these problems.A few students did not add their name or section to the uploaded video title as instructed, so it took some time to locate student and course section names to link it in the proper section of the course site.Processing video and using the technology was extremely time-consuming.Exploration of ways to streamline the media server upload and processing workflow is ongoing. Determining this workflow needs dedicated time, training and technical support.ConclusionsInstructors found that the use of annotated video with follow up analysis, self-assessment, peer response, and instructor feedback helped students understand kinetic/movement requirements and how to manipulate equipment. Students also had an increased appreciation for the complexity of the nursing interventions they were learning.Students valued both the self-reflection and faculty feedback components. Many of them indicated that they wanted more video recording activities and instructor feedback. Students appreciated the opportunity to review their performance multiple times and focus on a different aspect of their performance each time. They felt this was an enhancement that improved performance. Because students were novices, peer feedback in some cases was less valued, and students mentioned that peer response was inconsistent. They suggested some additional modeling or training could enhance the value of peer response. Some students indicated that they weren’t sure what feedback to provide peers even though they had been given a skills checklist and general guidelines to note both areas where tasks were performed well and areas to improve. However, many students indicated that they learned a lot from watching other student videos, in particular that they learned alternate ways to complete health care tasks.Nursing students need to become adept with complex procedures that include operating specialized equipment, manipulating objects and performing skills while remaining aware of nursing environmental concerns such as protection of a sterile field. Observing recorded activities with guidance directing them to attend to specific behaviors (often unconscious) helped students learn how to perform nursing skills. Students felt that video activities were particularly effective in providing visual feedback on their performance, and that this in turn helped them improve and enhance their abilities.The general consensus from faculty and students was that use of video technology promotes reflective practice and enhances performance of nursing activities. Incorporating peer observation and response provided valuable immediate input and early socialization to the role they would be expected to play in their professional environment.Post Project Use of Video Recording and ReflectionThe initial project occurred in 2013. Despite the early technical challenges, the activities described proved pivotal in changing the approach used for teaching basic nursing interventions. The value of reflection for enhancing and extending learning was clear. The faculty considered how to build more metacognitive and reflexive activities into the weekly lab sessions and into the high-fidelity simulation experiences that were included in the course. The annotation processes have been changed and improved to make the annotation option less labor intensive. This tool holds promise as an important reflective activity.Each lab session now ends with a debriefing element. A variety of approaches are used to strengthen the learning from that session. In general, the model used is to consider the activity from three perspectives: Thinking-in-action, Thinking-on-action, and Thinking-beyond-action (Dreifuerst, 2009). Using all three of these elements asks students to consider what they were thinking while they were engaged in the activity, to consider the activity retrospectively, and then to consider what they learned that can be used in the future as they encounter novel situations. It is difficult to measure the results beyond the performance in the actual laboratory course, but clinical instructors are reporting that the students are better prepared and have more confidence in their ability than was observed in the past.ReferencesBenner, P. Sutphan, M., Leonard, V., Day, L. (2010). Educating Nurses: A Call for Radical Transformation. Carnegie Foundation for the Advancement of Teaching. San Francisco: Jossey-Bass.Dreifuerst, K. (2009). Essentials of Debriefing in Simulation Learning: A Concept Analysis. Nursing Education Perspectives, 30(2), pp. 109-114.Rich, P. Hannafin, M. (2009). Video Annotation Tools: Technologies to Scaffold, Structure, and Transform Teacher Reflection. Journal of Teacher Education, 60, pp. 52-67, DOI: 10.1177/0022487108328486AcknowledgementsWe thank the University of Minnesota Provost for the 2013 Enhancement of Academic Programs Using Digital Technology grants program that funded this project. We also wish to acknowledge Dr. Jeff Lindgren, assistant director of the Center for Educational Innovation.Appendix A1. Sample Instructions or Self-Reflection/Peer ResponseSelf-Reflection and Peer Response Overview In this course you will be completing several learning activities that include self-reflection and peer response. Here is a brief overview of the learning theories and practices associated with this type of learning activity.What is Self-Reflection?Self-Reflection is the process where the student who performed an activity or wrote a paper, critically reviews their own work. A set of criteria or guidelines are provided for the student. The student is then able to look closely to identify strengths or weaknesses.Why is self-reflection important in the learning process?Self-reflection allows the student to gain greater insight into their own behaviors and decision making process. This ability is a life-long learning skill that will be necessary as new situations are presented to the student/nurse throughout one’s career.How do I do Self-Reflection?In your lab or classroom course your faculty will guide you through the self-reflection process. Typically you will have a document that will provide a combination of specific criteria that are important for you to consider in your self-assessment (for example, did you keep the equipment sterile or are your written sentences complete) and criteria that are broader (such as your communication style with the client and/or family). You will likely write a reflection response that identifies insights about observed behavior and how you might approach a similar activity in the future based on this observation.What is the value of Peer Response?Peer Response and Self-Reflection are lifelong professional abilities that you will practice throughout your career.What is Peer Response?In a lab or classroom setting you will work in pairs or small groups. Within these small groups or pairs, you will observe or read your peers’ work and provide comments back to your peers regarding what you observed. This is an active process for both the student completing the activity and the student providing comments. As a result, both students are actively engaged in the learning process. After you have had an opportunity to do the activity you may also complete a self-reflection activity.Why is Peer Response Important?Peer response has been shown to empower learners and improve the quality of learning (Rush, Firth, Burke Marks-Maran, 2012). Peer response also helps learners develop the skills required to make judgments or decisions that inform critical thinking abilities important in nursing. Students are also able to identify their own strengths and weaknesses, which supports their development in the learning process.Engaging in meaningful feedback at all levels of learning and in professional nursing practice supports not only the individual nurse but the entire team of health care providers. Practicing peer response as a nursing student is a foundational ability for professional nursing practice.When and where would Peer Response be used?As a student you will be provided with activities that will include a Peer Response component. These activities may be in a lab course, classroom course, on-line course or clinical course. Direct observation of an activity such as hand washing technique or reading the first draft of a classmate’s written assignment may be assigned. These activities may occur in or outside of class time. Each instructor will provide specific directions.What do I know about the activity in order to provide Peer Response?You may be thinking, “I am not an expert on that activity, how can I provide any meaningful Peer Response comments?” You are smarter than you think! In preparation for activities, you will have read about the activity, likely watch a short video (several times), possibly read a procedure/guideline form, and will use your observation skills. Based on your preparation you will have a good knowledge base from which to provide feedback to a peer. You and your peer or small group will actively be engaged in the learning process.How Do I Do Peer Response?Responsibilities:Come prepared: Complete assigned work before class time. Review any assigned handouts or guides that help you provide specific comments. Be respectful of all students in the course. Each student comes to an assignment or activity with a different set of background experiences. Be respectful of differences and learn from everyone’s experiences.Communication:Communication is crucial in the Peer Response process. Peer Response needs to include affirming or positive information and information about specific components of the activity that did not work well. Sometimes errors will occur.Below are a set of suggested responses to help you provide comments to a peer. The ability to provide this type of feedback required in the Peer Response process will take practice. Your instructors will assist you.Clear, specific comments are the most helpful. For example,All of the steps were completed in the correct orderWhen you put on the second sterile glove, your right hand touched the table, contaminating that glove.These comments are more helpful than “Great job!” Other phrases to consider:I noticed the cover fell on the floor…I am curious about where you placed the sterile dressing…The nurse in the video did ___ a bit differently. How did ____ work for you?Appendix References:Rush, S., Firth, T., Burke, L., Marks-Maran, D. (2012). Implementation and evaluation of peer assessment of clinical skills for first year student nurses. Nurse Education in Practice, 12. Retrieved from www.elsevier.com.nepr.Additional resources that informed this paper:Boehm,H. Bonnel, W. (2010). The use of peer review in nursing education and clinical practice. Journal for Nurses in Staff Development, 26(3), 108-115.Price, B. (2005). Self-assessment and reflection in nurse education. Nursing Standard, 19(29), 33-37.2. Sample Directions for ActivitySelf Assessment and Peer Response DirectionsNote: “Patients” are mannequins for this activity This week you will doing self-assessment and providing peer response for group members during medication administration learning activities. Read the Self-Assessment and Peer Response Guidelines in the General section of the course first. A key concept to remember is that both the student performing the skill and the peer responder will be enhancing their understanding from this analysis.Self AssessmentDuring this activity one of your group members will videotape your activity. Afterwards you will analyze the process using the skills evaluation checklist. Provide team members with your thoughts and comments to help them understand your analysis of your performance.Peer ResponseVideotape your partner using provided equipment and direction sheets. After videotaping briefly review skills practice using the video and the skills evaluation checklist. Remember to keep your comments tactful, but provide honest reflections on what you’ve observed.3. Self-Assessment and Peer Response ExamplesSample Student Self-reflectionI forgot to ask the patient whether she had any latex or iodine allergies. I also forgot to put her legs apart before providing the cleaning care (I remembered in the middle of the process and then I did it). I poured the lubricant on the sterile field but forgot to lubricate the catheter. On the video I could not hear myself well throughout the procedure; however, I remember greeting her and identifying her with two identifiers. I did hand hygiene, provided patient privacy, provided comfort. I was having a hard time putting the sterile gloves. I should have removed my rings before putting the gloves on.Sample Peer ResponseWhat went well: You entered the room and introduced yourself and performed hand hygiene, which was very good. You verified the patient’s identity and informed them on what procedure they were going to have today. You opened the sterile package properly.Areas of ImprovementWhen getting the patient ready to bathe you did not tell her that you were placing a bath blanket on her, and you did not clean her properly with different sides of the cloth so as not to contaminate clean areas. When the catheter kit was open you reached over the kit and caused a break in the sterile field, so be aware not to lean over a sterile field. Also remember to place the patient’s legs in the proper position for cleaning. Assist the patient if needed.Technical SupportDuring the first week using video recording the lab assistant will erase all video segments at the end of the session. In subsequent weeks the videos will be uploaded for deeper self-reflection and feedback.Appendix B: Pre and Post Survey ComparisonEvaluating a video recording of myself practicing activities in the lab will enhance/enhanced my learningI believe that reflecting on my video recorded skill practice is an important learning activityFeedback from my peers is/was important to my learningObserving a peer’s performance and providing feedback to them enhances/enhanced my learningObservation of my own video recorded activity is likely to improve/improved my confidence in my ability to perform that particular activity.Reflecting on my own video recorded activity is likely to improve/improved my confidence in my ability to perform that particular activity.I regularly reflect on my learning activities in the lab as a way to improve(Pre question only) I have used video recording to evaluate my performance.(Post question only) Feedback from my faculty was important to my learningTable 1: Survey Results with Percentage of Students Selecting Each ItemPercentStrongly AgreeAgreeUndecidedDisagreeStrongly DisagreeTotalQ1/1Pre29.255.514.60.70137Post47.646.92.13.50143Q2/2Pre28.253.516.91.40142Post40496.24.80145Q3/3Pre37.253.16.92.10.7145Post34.848.911.350141Q4 (only post)PreN/AN/AN/AN/AN/AN/APost76.422.20.70.70144Q4/5Pre36.659.23.50.70142Post32.452.8122.80142Q5/6Pre31.250.414.24.30141Post39.4507.72.80142Q6/7Pre25.561.710.62.10141Post3552.19.33.60140Q7/8Pre19.578.22.300133Post23.262.310.63.50142Chapter 13Web Mapping Tools and Pedagogical Materials to Support Spatial ThinkingSteven M. Manson, Melinda Kernik, Dudley Bonsal, Laura Matson, Eric Deluca, Ashwini Srinivasamohan and Sophia StrosbergKeywordsweb mapping, computer-aided cartography, experiential learning Web mapping for spatial thinkingA growing number of disciplines are finding that spatial thinking is a powerful but underexplored mode of inquiry applicable to a range of problems. The last twenty years have seen remarkable growth in spatial technologies, such as digital mapping and earth satellite imaging, along with developments in instructional technology such as computing and web learning. The National Research Council (NRC) report, Learning to Think Spatially (2006), describes the combination of instructional and spatial technology as spatial thinking, an integrator and a facilitator for problem solving across many parts of the curriculum, and one that is predicted to play a significant role in the information-based economy of the coming decades.Web mapping and geospatial technology more generally offers a way for students and instructors to achieve many key goals of a broader liberal education, meant in the broad sense of both STEM (Science, Technology, Engineering, and Mathematics) and the liberal arts. Web mapping and spatial thinking are entry points into a trillion dollar industry. Billions of people use technologies such as Global Positioning Systems (GPS), Google Maps, Yelp, and Uber. Governments use mapping to identify crime hot-spots, plan social interventions, and identify routes to evacuate vulnerable populations from harm. Companies use spatial analysis to site stores, evaluate supply chains, and determine how much to charge for goods and services. Web mapping is often seen as a way of introducing STEM into a variety of different courses, but it is also an entry point into a classic liberal education that demonstrates specific ways in which creative insights and knowledge are acquired and used, as well as how they change over time. Mapping Our World uses web mapping and cognate technologies examines how scholars use mapping to understand a range of individuals and the social systems of which they are part, along with addressing how technology and society are mutual constitutive.Web mapping is often seen as a way of introducing STEM into a variety of different courses, but it is also an entry point into a classic liberal education that demonstrates specific ways in which creative insights and knowledge are acquired and used, as well as how they change over time.The NRC report also describes the profound and largely unmet need for approaches to enact spatial thinking in the classroom. People have long used maps, from scratching their worldview on clay tablets thousands of years ago through to people today creating sophisticated web-based maps to change their societies. Mapping is an essential form of inquiry in social science that ranges from the use of geospatial technologies to gather data on people and places through to conducting sophisticated analyses. Mapping also represents the interplay of society and technology as mapping technologies—ranging from the earliest forms of writing to modern satellite imaging and web-based social networks—spring from and play out in a social context. The standard approach in many courses is to use specialized mapping software to explore and analyze spatial data on a range of topics. Application is limited because the software is expensive, entails extensive student and teacher training, and usually requires desktop computers in dedicated computer labs.As we describe below, a promising avenue for incorporating spatial thinking in the classroom is with web mapping. Web mapping involves using maps over the internet on a mobile device or personal computer. Web mapping promises to require fewer resources than traditional desktop software while replicating much of its functionality. Importantly, web mapping also offers exciting opportunities for creative activities that go beyond standard approaches, freeing spatial thinking from the classroom and encouraging students to creatively engage in the spaces around them in new ways. In the last few years, the capacity of web mapping has gone beyond simple display of pre-existing maps to become, in essence, web-based Geographic Information Systems (GIS) that collect, manipulate, analyze, and visualize spatial data.Web Mapping in PracticeWe focused on developing web mapping technology and curricular material to support spatial thinking. We designed, implemented, and evaluated these materials in Mapping Our World, which enrolls about 300 students per year, and are rolling them out to a broader array of courses. Students who successfully complete this course are able to read, use, and create maps informed by a contextual understanding of how maps reflect the relationship between society and technology and how mapping is an essential form of social science inquiry. Students gain hands-on experience with making their own maps with a variety of technologies, including web-based maps and satellite data, to address a variety of societal issues, such as disaster response and political gerrymandering. The course can be used to gain insight into the technical underpinnings of mapping as a social science approach for later courses, complement on-going interest and activities, or provide an applied focus for research or policy.In terms of structure, each week centers on a single core theme in mapping. We start with a lecture, online activities on mid-week, and a weekly interactive online laboratory session that students work on all week with several scheduled interactive times (Table 1). In addition, weekly readings and a final lab project build on, and bring together, themes from the entire course as follows:Lectures build on the readings to provide the big picture for the week’s activity and lab. Over the course of the semester we cover key skills in mapmaking such as map design symbolization and then move into broader issues such as analysis, lying with maps, and maps in society. Students are encouraged do the readings for each week before the lecture. These readings are on the course website and available for free.Activities amplify one or two specific aspects of the week’s topic and are conduced in person and online. Earlier in the semester we focus on making particular styles of maps, while later, we cover topics like how mapping technologies are used to push propaganda and rig elections. In-person activities are conducted in a classroom setting and typically use paper hand-outs. Online activities utilize a mix of small and large group work. Students are automatically assigned to small groups in a learning-management system, Moodle, if the activity that week requires a group. Activities generally start on Wednesday and are usually due on Thursday. We kick off activities during one of several scheduled times where students work as a group with the instructor and other students. Some activities require additional time to complete. For example, some activities require students to spend twenty or thirty minutes doing work and then posting their work to a forum. They then may have to wait a few minutes for someone to respond, and then spend another ten minutes to respond in turn.Weekly labs focus on using a range of online tools to explore data, create maps, and understand our world. One week we use mapping technologies to track actual people, for example, while in another we use census data to understand how the Twin Cities have changed over time.We examine how social scientists use mapping technologies to describe and analyze human experiences and behavior. In addition to learning the fundamentals of mapping, we spend a good deal of time and effort exploring how maps are used in the social sciences, as both tools and objects of analysis. For example, in addition to making their own maps to analyze data, students look at how organizations use maps to tell lies, sell products, win elections, and save lives. In the lectures Analysis and Social maps, for example, we examine how GPS and mapping technologies are designed and used in several contexts, and the discussion includes explicit consideration of multiple social science approaches to understanding social processes. In the labs Simplification and Analysis we examine how researchers use mapping and other approaches to understand social issues such as poverty and income inequality in a broader social context. Most lectures and many activities feature opportunities to discuss and debate a range of issues from a social science perspective. Some are freewheeling, as in the lecture Surveillance when we discuss different theories on how a society should balance privacy and security. Other venues are more structured, as with the lab Tracking people, where students learn to use different methods to examine data on the actual locations of four subjects collected over the course of three days to and describe the subjects’ experiences and behavior.Students learn how to manipulate primary or secondary data. The first half of the course focuses on the fundamentals of mapping, albeit with constant thread of social elements, while the second half examines the social dimensions of these fundamentals in depth. Mapping relies on a number of basic tasks, from data collection to foundational social science approaches, ranging from mapping data, using symbolization to best represent data, simplifying and classifying data via visual and statistical methods, and then finally analyzing data and maps with a range of methods. We also distinguish between ‘raw’ primary data collected by the analyst and secondary data, or those collected by another party or for another purpose. One week focuses on data, ranging from technical issues of collection to social dimensions such as the changing definition of race and ethnicity in census data over time and what it means for social science analysis. Following weeks examine foundational social science approaches, ranging from how to map these data, use symbolization to best represent these data, simplify and classify these data via visual and statistical methods, and then finally analyze data and maps with methods including analyses of proximity, clustering, and correlation. Later weeks amplify various features of the social science data-analysis workflow developed in the first half of the class, including ways that the same underlying data can be used to develop different narratives and the attendant ethical responsibilities that social scientists must meet.Table 1: Course structure by weekReadings/LectureActivityLabMaps! General overview of maps and their history.Geospatial revolution. Students watch a video on mapping technologies and answer questions.Software introduction. Guided tutorial to the web-mapping software.Data. Data types, sources, pitfalls, and uses.Data scenarios. Students work together to assess a set of land use planning issues and the data needed to understand them.Mapping data. Mapping basic data and learning about how to find and critique data.Scale + Projection. Mechanics of moving from a 3d world to 2d maps.Scale and projection. Students work in small groups to assess the utility of different projections.Scale and projection. Identifying the best scales at which to map certain kinds of data.Symbolization. How are map symbols used and for what purposes?Bad maps. Groups identify problems with mapsSymbolization. Experimenting with mapping styles and colors.Simplification. Approaches to simplifying and codifying data.Simplification. Groups classify and map information.Classification. Learning different ways of sorting and binning data.Analysis. Basic approaches to map analysis and examples of how they are used.Analysis. Students assess ways of solving mapping problemsMaps in the wild. Students find and share maps from real-world settings.Lying maps. Ways in which maps are used for advertising, propaganda, and other uses.Gerrymandering. Students complete an online exercise to gerrymander political districts.Lying maps. Students developed different story lines with the same underlying data.Surveillance. Exploring how mapping and other spatial technologies are used to spy on people.Surveillance scenarios. Groups discuss campus-based surveillance scenarios.Tracking people. Students use mapping to track people in time and space.Social maps. How are maps used in society?Where 2.0. Students watch and discuss videos on mapping and geospatial technology.Crowd sourcing. Students crowd-source a map of cameras on campus.Project work. Students develop a final project.Project work. Students work in small groups to refine their project ideas.Final project. Develop two or three maps for the final project.In addition to making their own maps to analyze data, students look at how organizations use maps to tell lies, sell products, win elections, and save lives.We identify key mapping resources and evaluate their quality. We examine the social science dimensions of data and mapping technologies in order to better understand their promises and pitfalls. In the lecture Data, for example, we examine how the Global Positioning System (GPS) works and the limitations of its functionality. We discuss the extent to which users, including social scientists, must deal with the limitations of these and related data such as those collected by satellite imaging. In the lecture Social maps and attendant lab Crowd sourcing, we examine how crowd sourced maps (i.e., those where the public provide the data) compare to more traditional data sources in their accuracy and adaptability, and discuss how these different methods of gathering mapped data affect the way these maps are used. More broadly, over the course of the semester we discuss the role of various individuals and organizations in providing map-based social science analyses, including assessments of data quality, on a range of issues.Mapping Our World offers multiple instances for students to use mapping to examine the roles that individuals play in their cultural, social, economic, and political worlds. The course uses mapping as a tool to examine interrelationships among individuals, institutions, structures, events, ideas, and technology. In Social maps, we examine the social and technical dimensions of how mapping technologies such as satellite imaging of the earth, GPS units, or the spatially-aware internet owe their existence and continued evolution to military research. We discuss how mapping is a trillion-dollar enterprise that fuses fast-evolving technologies together with rapidly changing societal practices. We examine the social and technical dimensions of mapping. For example, what are the implications for individuals and society of the fact that private companies, and not governments, are now the largest collectors of spatial data on people and places? In the lecture Data and its attendant activity Data scenarios, for example, we examine these implications for individuals and society in a variety of ways. In the lecture Maps! and activity Geospatial revolution, we take the long view on myriad interrelationships among individuals, institutions, and events by examining how mapping has been used as a tool for, and served as an expression of, social goals such as exploration and contestation of ideas around sovereignty and colonialism.Mapping Our World offers multiple instances for students to use mapping to examine the roles that individuals play in their cultural, social, economic, and political worlds. We ask questions in the week Surveillance about what surveillance practices mean for individuals and society. What does it mean that most mobile phones and many web-applications like Facebook can track location in a way that was not possible just a few years ago? The Social maps week similarly asks students to consider how they as individuals can take advantage of the ubiquity of both web-based and mobile mapping technologies to change the way we understand and relate to the world. We also examine, in weeks led off by the lectures on Simplification and Symbolization and their accompanying labs, how individuals and organizations make many choices in how the make maps, and how these choices reflect their cultural, social, economic, and political worlds.Students enjoyed becoming creators and producers of their own data and maps. During the final project period of the course, students spend two weeks developing an astounding array of data and maps on topics from national sports teams to race and ethnicity to urban renewal to environmental change. These problems are often “ill-structured problems” that require discovering multiple forms of data (including developing their own) and reconciling different perspectives. The students critically evaluate the problem, sift through multiple solutions, and then apply technology to develop new views on the world. Interactive mapping, for example, allows students to engage in digital storytelling and adaptive learning, such as through use of dynamic ‘story maps’ that have students describe and analyze real-world problems like climate change or poverty.Mapping offers a multidisciplinary framework with which to understand a range of local, national, and global issues by engaging in a process of critical evaluation of maps produced by various social groups and researchers. Students develop a framework with which to evaluate the truth claims about a range of local, national, and global issues by engaging in a process of critical evaluation of maps produced by various social groups and researchers. In several activities and most labs, students create their own maps and experiment with different technologies of map creation while examining how different fields—among them anthropology, geography, philosophy, political science, sociology, statistics, and women’s studies—develop a view on the social world. Other engagements are more abstract, as when we examine how mapping is used in fields such as advertising or international relations. In Lying maps, for example, we look at propaganda over the last hundred years, while in the lecture Surveillance, we deal with how social sciences engage with issues of territory and territoriality in the post 9-11 era. We also examine the politics around mapping and how social issues such as immigration or political issues around elections must be synthesized and analyzed at local, national, and global scales.Mapping Our World employs several strategies to help students to work collaboratively and individually to construct new knowledge. Over half of the activities and all the labs have students use software to create new data, maps, and analyses. While the labs are done individually, in the activities students collaborative to create new knowledge. Students work in small groups of three to ten students to tackle a range of problems. Several activities involve students working together to develop a new and hopefully better understanding of the world, such as finding different ways gerrymandering is done in the real world or how scale and projection can be used to create differing conceptions of issues such as poverty, income, ethnicity, or migration. In another activity, Maps in the wild, students find a map online or in another parts of the public sphere and then analyze the map. Other students in turn provide feedback on the map and the attendant analysis. The final two labs together constitute the final project, which is an opportunity for students to use the many approaches from this class to dive deep into a single problem or issue.In addition having focusing on mapping as a social science approach, Mapping Our World focuses on how mapping technologies affect and are affected by society in very specific ways and help introduce students to core STEM methods and concepts. We examine the measurable impacts mapping technologies have on society. In the lectures Maps! and Analysis we discuss how mapping is a trillion-dollar enterprise that fuses fast-evolving technologies and with rapidly changing societal practices. We ask questions in Surveillance about what it means for individuals and society that most mobile phones and many web-applications like Facebook can track location in a way that was not possible just a few years ago. Should the government to track our every movement? Should parents or schools track their children? Companies their employees or customers? We examine how these are not hypothetical questions given that tens of millions of people are being tracked at any given moment, including most people in the class. The Social maps week similarly asks students to consider how the ubiquity of both web-based and mobile mapping technologies has impacted the way we understand and relate to the world. Throughout the course, we discuss the role that society has played in fostering the development of mapping technology….We examine how various social groups put mapping technologies to use. We examine the science and engineering behind mapping technologies in order to better understand their promise and pitfalls. In the lecture Data, for example, we examine how the Global Positioning System (GPS) works and the limitations of its functionality. Perhaps more importantly, we discuss the extent to which limitations are made clear to its users, and pitfalls these users face as a result of not knowing these limitations. In the lab Tracking people, for example, we examine how the specifics of GPS technology play out in following people in real-world situations, such as the transition from indoors to outdoors or how GPS deals with different modes of travel. We look at similar issues around the basic science and engineering behind the technology behind collecting data from space-borne platforms, drones, and ground based cameras in many of the readings as well as the Data, Analysis, and Surveillance lectures; activities including Data and Scale and projection; and several labs.Throughout the course, we discuss the role that society has played in fostering the development of mapping technology. In the lectures Maps! and Data, we examine the long-standing impact that society has had on developing mapping technologies, ranging from the impact of the printing press on paper maps to development of computers and satellites on digital mapping. In these lectures and in Social maps, we examine the social and technical dimensions of the fact that mapping technologies such as satellite imaging of the earth, GPS units, or the spatially-aware internet owe their existence and continued evolution to military research. We also examine how mapping practices and related technologies, such as the use of Google map for searching for businesses, are actually shaping our world in many subtle and not-so-subtle ways. In the lecture Data and attendant activity Mapping data, we examine the implications for individuals and society that private companies, and not governments, are now the largest collectors of spatial data about people and places. In Scale and projection, we examine how the basic features of online maps, chosen by the companies that maintain them, have may have unintended affects how readers view the world.We examine how various social groups put mapping technologies to use. We spend a good deal of time and effort exploring how maps are used in many ways, including to tell lies, sell products, win elections, and save lives. In the lectures Social maps, for example, we watch a video that describes how GPS and mapping technology are designed and used in several contexts, and the discussion includes explicit consideration of multiple perspectives on the same technologies. In Lying maps, we examined how many different cartographic methods can be used to bend the truth, sell a specific perspective, or simply lie about the state of the world. We look at mapping from multiple perspectives, hearing from developers of technology through to users and other people affected by the technology. In several readings and lectures, we also look mapping and geospatial technologies from the perspectives of application developers and mapping companies through to users such as real estate agents and police officers as well as members of the general public.We develop a framework with which to evaluate new technology now and into the future by engaging in a process of critical evaluation of maps produced by various social groups, along with several conversations around how to critically evaluate how these technologies play out in, and are affected by, a social context. Asking and answering questions with no easy answers gives us the opportunity to develop skills in evaluating conflicting views on existing and emerging features of mapping technology. In several of the labs – including Mapping data, Scale and projection, Symbolization, and Analysis – students create their own maps and experiment with different technologies of map creation. In other labs and activities – such as Gerrymandering, Surveillance scenarios, and Tracking people – we step through questions necessary to understanding the ethical dilemmas posed by situations such as security camera surveillance, crime mapping, and gerrymandering electoral districts. Mapping Our World offers a lens through which to understand interactions between society and technology, as mapping technologies are driven by social imperatives and these technologies in turn change with society.Lessons LearnedWe assessed different web-mapping platforms to identify their visualization and analysis capacity and associated data. We examined how these platforms meet pedagogical needs (after Manson et al. 2014). The University of Minnesota’s spatial science infrastructure, U-Spatial, provides staff, hardware, and software for web mapping and integration with UMN’s broader computing ecosystem, such as x.500 authentication (see http://uspatial.umn.edu). We elected to use two robust platforms, Esri ArcGIS Online and Social Explorer, to develop exercises and online material.ArcGIS Online is similar to Google Maps but offers many more advantages because we work with educators to identify features necessary for instruction (Figure 1). Key among these features is a broad range of data, interactivity, security, ease of use, customization, analytical capabilities, low resource demands, and sustainability.Social Explorer is a demographic data visualization and research website that allows users to develop dynamic maps and customizable reports (Figure 2).Figure 1: ArcGIS Online map in which students have collaboratively identified and mapped the locations of surveillance camerasWe tested web mapping in the classroom and could draw on the fact that U-Spatial implemented a web-mapping system for campus. First, we have tested web-based activities over the last three years with the 300 students annually who take Geography 1502: Mapping Our World. Second, U-Spatial manages ArcGIS Online for the university, making it available for research use (Harvey 2013). U-Spatial was invited to present on emerging best practices for ArcGIS Online use and management at Esri’s Education Conference (where Esri is the company that makes ArcGIS Online), and U-Spatial was called on to meet with its lead developers to discuss issues specific to higher education.In addition to examining and testing underlying web-mapping platforms, we expanded and refined curricular material. In addition to creating material for standard spatial-thinking activities, such as cartographic guides, we took advantage of the unique opportunities web mapping offers spatial thinking. This included making exercises for flipped classrooms in which students engage in self-guided work, such as mapping the location of security cameras on campus (Figure 1) and then analyzing and interpreting these data in class as part of a larger discussion about spatial aspects of privacy and security in the digital age. These curricular materials also leverage interactive mapping to allow students to engage in digital storytelling and adaptive learning, such as through use of dynamic ‘story maps’ that have students describe and analyze real-world problems like climate change or poverty. We implemented and tested new technology and pedagogical materials in Mapping Our World. We used student and instructor assessments via surveys, focus groups, and interviews as described below. Figure 2: Social Explorer offers mapping and visualization capability for census data.We tested web mapping in the classroom and could draw on the fact that U-Spatial implemented a web-mapping system for campus. First, we have tested web-based activities over the last three years with the 300 students annually who take Geography 1502: Mapping Our World. Second, U-Spatial manages ArcGIS Online for the university, making it available for research use (Harvey 2013). U-Spatial was invited to present on emerging best practices for ArcGIS Online use and management at Esri’s Education Conference (where Esri is the company that makes ArcGIS Online), and U-Spatial was called on to meet with its lead developers to discuss issues specific to higher education.In addition to examining and testing underlying web-mapping platforms, we expanded and refined curricular material. In addition to creating material for standard spatial-thinking activities, such as cartographic guides, we took advantage of the unique opportunities web mapping offers spatial thinking. This included making exercises for flipped classrooms in which students engage in self-guided work, such as mapping the location of security cameras on campus (Figure 1) and then analyzing and interpreting these data in class as part of a larger discussion about spatial aspects of privacy and security in the digital age. These curricular materials also leverage interactive mapping to allow students to engage in digital storytelling and adaptive learning, such as through use of dynamic ‘story maps’ that have students describe and analyze real-world problems like climate change or poverty. We implemented and tested new technology and pedagogical materials in Mapping Our World. We used student and instructor assessments via surveys, focus groups, and interviews as described below. Table 2: Web mapping responses coded by sentiment (n = 104)Questions with coded responsesStudents(%)What do you like best about the web mapping applications used in class?Easy and/or quick to use55Ability to analyze and visualize spatial data with a range of tools28Ability to customize map appearance an/or purpose25Large amounts of data available online15Generally enjoyable or useful13Free access to an online interface12Educates about how mapping works10Being able to compare different data sets/interoperability4What is one thing you would change about the web mapping applications used in class?Occasional software glitches31Software can be confusing at times28Allow for more customization or have more features22Difficult to find data in online catalogs27Nothing20More hands-on software instruction12Better quality datasets and metadata10More opportunities to develop paper maps by hand4Students like web mapping but there is room to improve instructional materials on the systems themselves. Survey responses across offerings indicate students generally enjoy the mapping experience (Table 2). They enjoy web mapping because it is quick and easy, allows them to do interesting things, and they can customize the experience among other reasons.Beyond a few technical issues, one drawback is that a sizable minority of students expressed interest in more instruction with the web mapping tools before using them for assignments. We are planning to develop more, and better, guides and tutorials. We are also working with the creators of these web-based tools to identify areas where they can provide online tutorials and other materials what can help train students. Finally, we have developed several tutorials that focus on areas where students have problems (e.g., finding specific types of mapping tools on a site) as well as expanded documentation on the labs themselves.In addition to students finding web mapping useful, the large majority valued the online materials, including a draft version of an online textbook for the course. The U of M Libraries conducted a survey on these materials and student responses included the following open-ended responses. All students surveyed replied ‘yes’ to the question “would you take another course using material like this?” Common answers to the question “what worked well for you when using the digital course material?” included that they were free, easy to access, well organized, easy to use, accessible any time, and tied well to the course content. Most students reported no concerns, although two commented that it would have been more convenient to have paper copies of the reading when they there was no internet access (e.g., it becomes unavailable at home).Table 3 offers insight into the mostly positive experience students have in using the free course content assigned for this class. The sample size in this survey is smaller than the survey described by Table 2 because it was administered closer to the end of the semester and, unlike that above, was not assigned course credit.In addition to high-levels of student satisfaction, we found a number of advantages for instructors, who can develop innovate assignments for which students can make creative solutions because the system offers several key characteristics. Among these are:Analysis. Both applications, ArcGIS in particular, offer sophisticated visualization and analysis tools that are usually found only in specialized and difficult-to-learn workstation software. Students can add, edit, manipulate, and analyze data.Data. ArcGIS Online has data ranging from census information and street networks to protected wetlands and remotely sensed imagery. While an instructor can focus on particular kinds of data, students can also easily gain access to literally thousands of datasets encompassing a wide array of human and environmental topics.Low Resource Demands. Our experience points to mapping capacity that requires fewer hardware and software resources than full-fledged mapping or GIS software, which often requires expensive dedicated hardware and much time to learn basics. Learning the rudiments of ArcGIS Desktop, for example, requires a four-credit course in a specialized lab. In contrast, web mapping is easy to pick up and can be used on any computer with an internet connection; students and instructors with no mapping experience can learn ArcGIS Online or Social Explorer in about twenty minutes on pretty much any computer. As above, one challenge is that a minority of students report some problems with learning the software, but judging from the queries we get in office hours, these problems tend to be very short-term and specific instances, and we are developing materials to address this issue. More broadly, students and instructors can engage in a broader array of activities because they are not tied down to a physical lab or constrained to using software on campus due to licensing.Interactivity. Tools to allow students and faculty to upload their own data via onscreen digitizing, global positioning system (GPS) handsets, or third-party digital maps available over the internet. This includes uploading and searching for images, video, audio, and other data. This range allows students to develop innovative solutions to a range of problems posed by assignments and meeting their own interests.Sustainability. Web mapping can be sustainable through various strategies, but of particular emphasis at the University of Minnesota is the existence of Enterprise GIS (a division of U Services that supports enterprise-scale geographical information system support) and U-Spatial to administer and maintain ArcGIS Online in the long term. The platform is part of the larger ArcGIS family of GIS software for which the university has site license, which means instructors can get full-time technical support. Overall, the project is sustainable by virtue of having low resource demands, as noted above, making it more likely to be adopted and funded by a range of actors on campus.Table 3. Responses to the question: Based on your experience using the free/affordable course content assigned for this class, rank the following statements (n = 24)QuestionStrongly DisagreeDisagreeNeutralAgreeStrongly AgreeThe materials for this class were easy to use0001014The materials for this class were easy to access0001014The resources covered material in a way that I understood001914I read more of the assigned readings because the materials were free003813I read more of the assigned readings because they were available online011913I studied using the online readings/videos020913 Future DirectionsIn addition to expanding our guides and material around basis map use, we would like to work with instructors who have asked U-Spatial for help in enacting spatial thinking in their courses. This outreach has potential for high impact because it encourages spatial thinking in the hundreds of students from dozens of departments who take Mapping Our World, but it has even greater potential (and meets correspondingly low resistance) by satisfying broad faculty interest in spatial thinking on campus.Now is an ideal time to extend the use of web-mapping to classrooms all across campus. Over 2000 staff, faculty, and students at the university already use ArcGIS for research use. Many of these scholars, and others besides, have expressed interest in using the system for teaching purposes. In response to this demand, and in order to take advantage of existing web mapping infrastructure, U-Spatial will provide the backend technical infrastructure and support, while other groups such as ourselves will engage in several activities to bring ArcGIS Online to teaching. We are working on: 1) readying ArcGIS Online for broader classroom adoption, such as integration with Moodle and offering several simple interfaces; 2) writing curricular templates and how-to guides and generic labs that can be adapted by any instructor; and 3) working with CLA faculty for the first year to roll-out ArcGIS Online and these curricular templates in the classroom.ReferencesHarvey, F., Kne, L., Manson, S. (2013). U-Spatial: A Consortium for the Spatial University. ESRI, Redlands.Manson, S., Shannon, J., Eria, S., Kne, L., Dyke, K., Nelson, S., … Matson, L. (2014). Resource Needs and Pedagogical Value of Web Mapping for Spatial Thinking. Journal of Geography, 113(3), 107-117.National Research Council (US). Committee on Support for Thinking Spatially (2006). Learning to Think Spatially. National Academies Press.AcknowledgementsDeveloping spatial thinking materials require a team-based approach. Steven Manson is a mapping researcher and the primary instructor of Mapping Our World, the course in which we developed material. Len Kne is the Associate Director of U-Spatial, U of M’s spatial technology infrastructure and leads its web mapping team. Eric DeLuca, Melinda Kernik, Laura Matson, Ashwini Srinivasamohan, and Sophia Strosberg collectively served twelve semesters as teaching assistants for Mapping Our World and were integral to designing and implementing web mapping technology in the classroom. Paul Ching, from the Center for Educational Innovation, and Susan Tade, from Academic Technology Support Services, are experts in pedagogy and technology and provided advice and guidance at all steps along the way. Kristi Jensen and Shane Nackerud from the eLearning Support Initiative of the University Libraries offered expertise on copyright, elearning, and information technology with support from the Partnership for Affordable Content program.Chapter 14Student Lecture Viewing: Learning from an Online Health Psychology MinorThomas Brothen, Penny Nichol and Esther Joy Steenlage MaruaniKeywordsonline lectures, lecture capture, lecture viewing, learning outcomes, health psychologyIntroductionThe availability of and registration in online courses is expanding. Basic Internet searches using terms such as online courses, online colleges, or online degrees yield millions of hits, suggesting that interest in them is also very high. Drouin (2013) reviewed literature on this issue and concluded that the increase is real and significant. And although the popularity of the MOOC phenomenon may have peaked, Hill (2016) credited that online course structure with helping to validate online education as a useful resource. Hill also concluded from his research that many students prefer online courses and possibly half of the students enrolled in them would not take a traditional, live course for a number of reasons. This finding suggests that post-secondary educational institutions need to develop online courses to serve their current and prospective student populations. Although the issue is not settled (c.f., Figlio, Rush, Lin, 2010), general support for adding online courses came in a 26 June 2009 U.S. Department of Education meta-analysis of studies comparing online and traditional courses. The researchers found that “on average, students in online learning conditions performed better than those receiving face-to-face instruction.” Consistent with the findings reviewed above, the report also noted that the prevalence of online courses has been increasing greatly.As the popularity of online courses continues to increase and colleges see them as a way to attract and maintain student enrollments, instructors are faced with choices as to how to structure them. Because the lecture method is still the dominant feature of college instruction, instructors designing online courses are likely to include recorded lectures even though research supporting lectures’ effectiveness is mixed at best (Costin 1972; Freeman et al., 2014; McKeachie Hofer, 2002). Additionally, alternatives to lecturing such as the Personalized System of Instruction (PSI; Keller, 1968) have been shown to be consistently superior to traditional lecture teaching methods (Kulik, Kulik, Bangert-Drowns, 1990). Further, mastery learning systems also have led to superior student learning (Bloom, 1976; Kulik et al.) compared to the traditional lecture method. There also has been a more recent movement in the direction away from lecturing toward cooperative learning and peer discussion (Johnson Johnson, 2009; Mazur, 2009). Nonetheless, lecture remains the dominant instructional technique in colleges and this chapter reviews our evaluation of that method in three online courses that are part of an online minor in health psychology.Our primary question is how much educational value lectures added to students’ learning outcomes in our courses. Our second approach was to determine how lectures compared to other course activities in their effect on students’ learning. To answer that question we first needed to determine more precisely how much of the lectures available to them students actually viewed. Most studies of this assess the answer to that question by basically asking students if they watched the lectures (e.g., Bosshardt Chiang, 2016). We assessed that variable more objectively. We performed this study in three courses situated within a new program in our Department. Our primary question is how much educational value lectures added to students’ learning outcomes in our courses. Our second approach was to determine how lectures compared to other course activities in their effect on students’ learning. The Department of Psychology offers two majors (BA and BS) in addition to a minor in psychology. The Department has the largest undergraduate program in the liberal arts college with nearly 1500 majors in BA and BS programs and an additional number of students pursuing minors. Several years ago, difficulty arose in meeting student demand for core courses in the curriculum required of students pursuing both majors and minors. Consequently, the Department requested to drop the minor because of completion for these core courses from students pursuing the minor. After pushback from the College administration and the College Assembly, the Department reaffirmed the minor and attempted to meet student demand by initiating an online minor in 2012. The minor created at that time consists of several online courses and focuses on the theme of Health Psychology.The minor in Health Psychology includes six courses, each of which is taught in a fully online version that duplicates the content and course goals of already existing live courses. This chapter presents data from an analysis of three of these courses: Introduction to Psychology, Introduction to Psychological Measurement and Data Analysis, and Introduction to Research Methods. Our primary goals in the research described here are to determine first how much students actually watched lectures in our online courses and then whether the lectures add educational value. Our overall goal is to provide a perspective on how much students value online lectures and thus help instructors to decide whether they should use them in their online courses. To accomplish these goals, we set out to gather data that is more valid than the questionnaire data typically gathered by researchers to assess amount of online lecture watching and to relate that data to data on other course elements.MethodFor each of the three courses described here, we recorded lectures with the Mediasite lecture capture system (http://www.sonicfoundry.com) and made them available to students as links through the Moodle course management system (CMS). The Mediasite platform is a lecture capture system that is within “a subset of streaming products… designed specifically for capture and management of classroom content. These products rose from the…need for educational institutions to record and archive content” (http://www.sonicfoundry.com/resource/wainhousereport/?aliId=62648363). Students watch the lectures on one window on their screens and the Powerpoint slides through a second window. They can stop at any time during the lecture and when they come back, they are brought back to the point where they stopped previously. The built-in analytics track students’ usage and “shows a specific user’s (or group of users’) viewership over any time period, including videos watched, viewing activity, durations, registration data and more” (http://www.sonicfoundry.com/mediasite/manage/analytics/). Mediasite keeps a record of how much of each lecture each student viewed on one or more occasions, adding only the unviewed sections into the calculation. For this study, we obtained the total percentage watched for each lecture by all students and computed an average percentage watched (from 0 to 100%) for all lectures in each of the courses. Thus, each student had a value of 0-100% watched for each lecture and for the semester as well. We also collected the calculated percentages watched for each student on each lecture and then computed a Total Lecture Viewing average that also ranged from 0 to 100% for all the lectures.PSY 3801: Introduction to Psychological Measurement and Data AnalysisThe Introduction to Psychological Measurement and Data Analysis course (PSY 3801) is offered in two formats taught by different instructors – a traditional, in-person lecture and laboratory format and a completely online format. In the semester during which this study was conducted, the traditional format began with 360 students, who attended 50-minute lectures three times a week and a 75-minute laboratory session once a week. Forty-six students started the online version of the course. Those students watched all lectures online and participated in online laboratory work. The lecture modules (i.e., short videos that focused on a specific topic, subtopic, or computational example) were recorded specifically for the online section in a studio using Mediasite lecture capture. Overall, 98 modules were available for viewing, running roughly 3 minutes to 30 minutes in length, with an average module length of 13.22 minutes (SD = 6.40). Approximately 11% of the modules focused on review material or introductions to topics, 49% covered new course content, and 40% consisted of computational examples of the statistical tests.Students in the online section completed:10 problem set assignments,13 chapter quizzes,two midterm exams, anda final examStudents also could earn extra credit by completing additional assignments related to course workFor this chapter, assessment information examined included:students’ chapter quiz score;total chapter quiz score;each individual exam score, including a total exam score;extra credit score; andtotal points for the coursePSY 3001W: Introduction to Research MethodsThe Introduction to Research Methods course (PSY 3001W) is a writing intensive course offered in two formats taught by different instructors – a traditional, in-person lecture and laboratory format and a completely online format. The traditional format offered during the semester included in this study began with 288 students who attended 50-minute lectures two times a week and a 105-minute laboratory session once a week. Twenty-four students began the semester in the online course. These online students watched all lectures online and participated in online laboratory work, including a group research project. As with the data analysis course, online lectures were broken down into modules by topic or subtopic. For this course, 111 modules were available for viewing, running roughly 2 minutes to 29 minutes in length, with an average module length of 11.10 minutes (SD = 5.95). Eight percent of the modules focused on review material or introductions to topics, 76% covered research methods content, and 26% consisted of writing tips and APA style guidelines.The online students completed:writing assignments mostly related to an APA-style research paper,weekly lab participation assignments,weekly quizzestwo midterm exams, anda final examonline students also could earn extra credit by completing additional assignments related to course workFor this chapter, assessment information examined:students’ paper-related writing score total;overall writing score total; total quiz score;total participation score;each individual exam score, including a total exam scoreextra credit score; andtotal points for the coursePSY 1001: Introduction to PsychologyThe Introduction to Psychology course (PSY 1001) is offered each semester in a live/hybrid/online format with students being able to select from three basically different formats. Although the formats are different, they cover the exact same material with all students using the same textbook, completing basically the same assignments, taking the same three mid-semester exams and final exam in a computerized testing center. All students attended or watched all the same 39 fifty minute lectures. For the semester in which this analysis was performed, a total of 1207 students began the course. In the first variation of the course, 582 students finished by completing the work and taking the final exam. They attended live one hour lectures three days/week and a live one hour discussion section one day/week. In the second hybrid live/online version, students watched the lectures online and attended a live one hour discussion section one day/week – 369 finished the course. In the third totally online version, students watched the lectures online and participated in an online, asynchronous discussion section each week – 125 finished. Finally, 68 students from the University Honors Program enrolled in a variant of the hybrid version in which they watched the lectures online and attended a live two hour discussion section each week in which the regular discussion activities were handled in more depth. Because of the higher academic proficiency and status of the Honors students, that section was eliminated from all subsequent analyses.For the introductory psychology class, we collected Mediasite lecture viewing data for each of 39 fifty-minute class lectures that were given live in class, then recorded, and made available to all students within 5 minutes of each lecture’s conclusion. In addition to the total percentage lecture viewing variable, we collected two other input variables that we predicted would contribute to course performance: discussion activity performance, measured by total points obtained, and total points obtained on practice exams. Students had practice exams available for each of three mid-semester exams and the final exam. The purpose of the practice exam was for students to assess their knowledge and get direction as to what they should restudy. Students could repeat them as many times as they liked. The practice exam items are drawn randomly from large item pools that measure what also is measured on the exams. In this and previous semesters, the correlations between points gained on exams and points on the practice exams has been consistently in the +.70 correlation range. We also collected two student academic measures from the Registrar’s office – cumulative college GPA and ACT Comprehensive exam score. Finally, to determine if personality type moderated any of the other variables, we recorded students’ scores on a Big5 Personality Scale (DeYoung, Quilty, Peterson, 2007) that we had administered online as part of a class assignment. All data collection for this study was consistent with our University Human Subjects protocols.We collected data from two versions of the introduction to psychology course that utilize online lectures. Students in the totally online section completed weekly graded asynchronous discussion assignments worth approximately 17% of their grade, and took 3 mid-semester and 1 final exam worth 51% of their grade. Students were told that exams were structured around the lectures and some course material could only be gained through them. To help them prepare for exams, students had practice exams covering the same concepts tested on the exams but with different items drawn randomly from large pools so they never got the same exam twice. Students in the hybrid section we investigated had live discussion sections once each week and operated under the same grading process. An ANOVA comparing total exam points between the live, hybrid, and online sections showed they did not differ (F(2, 1049) = 1.04, n.s.). Because this and subsequent analyses determined that the same pattern of results occurred in the hybrid and online variations, we combined the data to obtain more statistical power for the final analyses we performed for this study.ResultsPSY 3801: Introduction to Psychological Measurement and Data AnalysisFor the measurement and data analysis course, 40 students completed the course. Their percentage of total lecture watching averaged 28.28% for the semester. The average percentage of lecture module viewing was correlated with total quiz score, total problem set score, individual exam scores, total exam score, extra credit score, and overall total points in the course, respectively. The relationships between average percent viewing and total quiz score (r(38) = .175, n.s.), total problem set score (r(37) = -.131, n.s.), midterm exam 1 score (r(38) = .142, n.s.), midterm exam 2 score (r(38) = .257, n.s.), final exam score (r(38) = .215, n.s.), total exam score (r(38) = .235, n.s.), and extra credit score (r(38) = .261, n.s.) were all nonsignificant. However, the relationship between average percent viewing and overall total points in the course was significant, r(38) = .322, p .05. Throughout the course of the semester, the average percent of each module watched decreased from a high of 74% for the first lecture module to a low of 5% for the last lecture module. There was a significant decrease in average lecture viewing when comparing the first week to the last week of the course (t(16) = 9.40, p .001).PSY 3001W: Introduction to Research MethodsTwenty-three students completed the research methods course. Their percentage of total lecture watching averaged 34.87% for the semester. Almost all of the course assessments were not significantly related to the average percentage of lecture module viewing. For writing, the paper components (r(21) = .084, n.s.) and total writing score (r(21) = .086, n.s.) showed virtually no correlation with total viewing; the correlation between participation and viewing (r(21) = .083, n.s.) was roughly the same as that of the writing components. The correlations between total quiz scores and viewing and between midterm exam 2 score and viewing were the same (r(21) = .325, n.s.). The correlation between extra credit score and average viewing was negative (r(21) = -.204, n.s.). Unlike the measurement and data analysis course, total course points and average viewing were not significantly correlated (r(21) = .135, n.s.), while midterm Exam 1 scores (r(21) = .434, p .05) and final exam scores (r(21) = .466, p .05) were significantly correlated with lecture viewing. A pattern in the average percentage of each module watched similar to that of the measurement and data analysis course was observed with a higher percentage having watched near the beginning (75% for the first module of week 2) and a lower percentage at the end (15% for the second to last module) of the course. Again, there was a significant decrease in average lecture module viewing when comparing the first week to the last week of the course (t(14) = 11.13, p .001). One key difference was a spike in lecture module viewing after the first exam (75% for Introduction Formatting Guidelines), but this might have been due to the nature of the module – the formatting guidelines for the students’ first major writing assignment. Additionally, average lecture watching for the research methods course was higher throughout the course of the semester than lecture watching for the measurement and data analysis course.PSY 1001: Introduction to PsychologyThe results from the data analysis and research methods courses were replicated and extended in the introductory psychology course. We first examined the percentage of lectures watched by students. As described above, we obtained the Mediasite calculated percentages for each student on each lecture and also computed a Total Lecture Viewing average that ranged from 0 to 100%. Students watched an average over all 39 recorded lectures of 37.38%. The average watched declined throughout the semester. The average percentage for the first three lectures was 44% and for the last three, 36%. The linear trend for the decline in all 39 lectures was significant with F(1, 493) = 38.59 (p .001). Clearly, lecture watching was much less than 100% and declined as the semester proceeded. We next examined the relationships between the six variables listed above.Correlating the six variables (lecture viewing, discussion activity points, exam points, practice exam points, cumulative GPA, ACT Comp score) revealed a value of r(494) = .421, p .001 between Total Lecture Viewing and Exam Total. This suggested a positive effect of viewing lectures and exam performance. We then examined the relationship of the other variables with Total Lecture Viewing: Participation Points Total correlated r(494) = .261, p .001, Practice Exam Total r(494) = .217, p .001, cumulative GPA r(494) = .392, p .001, and ACT Comprehensive score r(494) = .097, p .05. To assess how much the other variables were associated with exam total, we next assessed the correlations of Exam Total with Participation Points Total r(494) = .455, p .001, Practice Exam Total r(494) = .630, p .001, cumulative GPA r(494) = .706, p .001, and ACT Comprehensive score r(494) =.375, p .001. Personality variables measured by the Big5 scale (Openness to New Experience, Conscientiousness, Extraversion, Agreeableness, and Neuroticism) did not correlate with Total Lecture Viewing and thus were dropped from further analysis.For the next analysis, we defined Exam Total as the outcome variable and the others (lecture viewing, discussion activity points, exam points, practice exam points, cumulative GPA, ACT Comp score) as predictor variables to assess how much lecture viewing affected students’ learning performance. To control for dependencies between the predictor variables on the criterion variable, we performed a stepwise linear regression between the outcome and predictor variables. The results indicated the overall R = .826 (p .001). This analysis accounted for a very substantial 68.2% of the total variance with four of the five predictor variables accounting for significant (p .05) variance added. The first variable in the equation was Cumulative GPA which accounted for 45.8% (p .001) of the variance. Second was Practice Exam Total, which accounted for 14.2% (p .001) additional variance, the third was ACT Comprehensive score (7.6%; p .001), and last was Total Lecture Viewing (.7%; p .05). Participation points was not a significant predictor in the analysis.SummaryAcross all three classes, lecture viewing was low (well less than 40%, on average). Viewing also declined significantly during the semester for all three classes. Overall, the positive effects of lecture viewing were small and not statistically significant in most cases. In the introductory psychology course, our data revealed that lecture viewing had some impact on exam performance but other activities had more.DiscussionGysbers, Johnston, Hancock, and Denyer (2011) asked Australian Biochemistry and Molecular Biology students who had access to online lectures why they still came to live lectures. They found that students believed live lectures provided a better explanation of the material and provided a better social environment or simply attended lectures because that seemed the right thing to do (see also Jensen, 2011). This illustrates that lectures are traditionally favored by both instructors and students but this begs the question of how useful they really are. This study explicitly addressed that question. The low percentages of lecture viewing suggest that students actually value watching lectures much less than they say they do, and that their central place in courses would suggest. The three courses represented in this study had hybrid or online versions that had to meet the same Departmental requirements for content learning. If the live course covered material in lectures, it also had to be covered in the hybrid or online versions. This requirement for covering all the course material is handled differently in the three courses—from specially created modules in the Data and Research courses to video copies of the live lectures in the Introduction course. But our data reveals very similar results for them. In the introductory psychology course, the measurement and data analysis course, and the research methods course, percentage of viewing averaged only 37.5%, 28%, and 35%, respectively, and in all three courses, lecture viewing declined throughout the semester. Parenthetically, in a past semester’s live lecture version of the introductory psychology course, lecture attendance also declined throughout the semester to a low of about 40% at the end of the semester (Wu, 2015). The low percentages of lecture viewing suggest that students actually value watching lectures much less than they say they do, and that their central place in courses would suggest. We return to our rather narrow original question as to how much value students apparently see in watching online lectures and how much lecture viewing adds to student learning.Studies such as that done by Bosshardt and Chiang (2016) have utilized mostly quasi-experimental manipulations of live lecture vs online lecture course sections and found no effects or data favoring one over the other. Such studies have not provided a good answer as to baseline lecture viewing by students. This study looked more carefully at actual student behavior as tracked by the Mediasite platform and we found that percentage of the lectures watched was below 40% for the introductory psychology course for most of the semester. For the measurement and data analysis course, average lecture watching was higher for the first half of the semester (48%) than the second half of the semester (21%). This trend also held in the research methods course with an average of 51% lecture viewing in the first half of the semester and 26% in the second half of the semester. We also believe that if there truly is benefit to students from lectures, instructors should think of them as study aids that some students might find useful rather than making them the course centerpiece. Although lecture viewing had a positive effect on student learning based on exam points, that effect was very small—less than 1% of variance accounted for in total exam points for the introductory psychology course. The exams in the introductory psychology and research methods courses were primarily lecture-based, rather than textbook-based and lecture viewing correlated significantly with most of the exam-related components of these two courses. Clearly, students were getting that information from somewhere other than from watching the lectures. We suspect that many students used resources such as the textbook or internet sites related to the material or from other students’ lecture notes. We see this as a subject for further research.Most who have taught college courses have felt the pressure to include lectures. They either know or soon find out that students expect lectures and get anxious if they are not included as a major part of the course. New and different forms of instruction that do not include lecturing are undertaken at the instructor’s peril. The first author of this chapter has in his files a comment by a student made on a course evaluation in a course taught by the learning group method in a computer lab. The student wrote “All the instructor does in this class is walk around while the students do all the work.” For this type of reason and many others, lectures have remained central to college teaching. Even MOOCs, the “newest” wrinkle in the higher education fabric, originally were almost entirely recorded lectures although they have lately become more diverse in their approach (Ossiannilsson, Altinay, Altinay, 2016).The data obtained in this study suggests that activities such as problem sets and practice exams might be much more useful for student learning. Students occasionally remark during evaluations of our courses that they would rather learn from the textbook or find the textbook more valuable than lectures even though we tell them that our exams draw from the lectures. Based on our findings in this study, we have little concrete advice for instructors as to how they should structure their lectures but we do advise them to consult the large literature on increasing student learning (e.g., McKeachie Hofer, 2002) to find ways to improve their students’ learning outcomes. We also believe that if there truly is benefit to students from lectures, instructors should think of them as study aids that some students might find useful rather than making them the course centerpiece.ReferencesBloom, B. S. (1976). Human characteristics and school learning. New York: McGraw-Hill.Bosshardt, W., Chiang, E. P. (2016). Targeting teaching lecture capture learning: Do students perform better compared to face-to-face classes? Southern Economic Journal, 82, 1021–1038. doi:10.1002/soej.12084Costin, F. (1972). Lecturing versus other methods of teaching: A review of research. British Journal of Educational Technology, 3(1), 4-31. doi:10.1111/j.1467-8535.1972.tb00570.xDeYoung, C. G. , Quilty, L. C., Peterson, J. B. (2007). Between facets and domains: 10 aspects of the Big Five. Journal of Personality and Social Psychology, 93, 880 – 896. doi: 10.1037/0022-3514.93.5.880Drouin, M. A. (2013). If you record it, some won’t come: Using lecture capture in introductory psychology. Teaching of Psychology, 41(1), 11-19. doi:10.1177/0098628313514172Figlio, D., Rush, M., Yin, L. (2010). Is it live or is it internet? Experimental estimates of the effects of online instruction on student learning. doi:10.3386/w16089Freeman, S., Eddy, S. L., Mcdonough, M., Smith, M. K., Okoroafor, N., Jordt, H., Wenderoth, M. P. (2014). Active learning increases student performance in science, engineering, and mathematics. Proceedings of the National Academy of Sciences, 111(23), 8410-8415. doi:10.1073/pnas.1319030111Gysbers, V., Johnston, J., Hancock, D., Denyer, G. (2011). Why do students still bother coming to lectures, when everything is available online? International Journal of Innovation in Science and Mathematics Education, 19(3), 20-36.Hill, B. P. (2016). MOOCs Are Dead. Long Live Online Higher Education. The Chronicle of Higher EducationJensen, S. A. (2011). In-class versus online video lectures: Similar learning outcomes, but a preference for in-class. Teaching of Psychology, 38, 298-302. doi:10.1177/0098628311421336Johnson, D. W., Johnson, R. T. (2009). An educational psychology success story: Social interdependence theory and cooperative learning. Educational Researcher, 38, 365-379. doi:10.3102/0013189×09339057Keller, F. S. (1968). “Good-bye, teacher…” Journal of Applied Behavior Analysis, 1, 79–89.http://doi.org/10.1901/jaba.1968.1-79Kulik, C. C., Kulik, J. A., Bangert-Drowns, R. L. (1990). Effectiveness of mastery learning programs: A meta-analysis. Review of Educational Research, 60, 265. doi:10.2307/1170612Mazur, E. (2009, January). Farewell, lecture? Science, 323(5910), 50-51. Retrieved from http://www.jstor.org/stable/20177113McKeachie, W. J., Hofer, B. K. (2002). McKeachie’s Teaching Tips: Strategies, Research, and Theory for College and University Teachers (11th ed.). Boston, MA: Houghton Mifflin Company.Means, B., Toyama, Y., Murphy, R., Bakia, M., Jones, K. (2009). Evaluation of evidence based practices in online learning: A meta analysis and review of online learning studies. Washington D.C.: U.S. Department of Education Office of Planning, Evaluation, and Policy Development.Ossiannilsson, E., Altinay, F., Altinay, Z. (2016). MOOCs as change agents to boost innovation in higher education learning arenas. Education Sciences, 6(3), 25. doi:10.3390/educsci6030025Wu, W. (2015). Correlational relationship between class attendance and class performance in college. Unpublished manuscript, Department of Psychology, University of Minnesota.Correspondence regarding this manuscript should be directed to Thomas Brothen, Department of Psychology, University of Minnesota, Twin Cities. This chapter was supported in part by National Science Foundation Grant NSF/IIS-1447788 to the first author.Chapter 15Under the Watchful Eye of Online ProctoringDaniel Woldeab, Thomas Lindsay and Thomas BrothenKeywordsonline, proctoring, student, exams, academic integrity IntroductionOver the last two decades the steady increase of online learning in public higher education institutions has brought about the challenge of how best to assess students’ progress online while safeguarding the integrity of exams. To address this challenge, many institutions have outsourced the proctoring of student exams to online proctoring providers, which provides students with a way of taking exams from a distance through a secure platform. Students need to have a working computer with functioning webcam, and access to internet. The online proctoring service monitors students while they are taking the exam, and allows students to take the exam from any location Maintaining honesty in the learning environment becomes even more significant, as one of the fundamental aspects of instruction is the assessment of student learning. The purpose of this study therefore, is to examine student and faculty satisfaction with a solution to that problem–online proctoring.they choose: from their own home and supervised online, to a testing center where supervision is provided by live proctors. This study used ProctorU as the proctoring service provider. Students in fully online courses took their exams online, and were thus supervised remotely. Students in traditional classroom, or in hybrid courses (a mix of online and face-to-face) took their exams in a testing center, where they were supervised by in person proctors.It is well documented that there is an increasing trend of U.S. public universities and colleges offering online courses. Students appreciate the convenience that online learning gives them, while institutions can use these environments to broaden their reach, expand and diversify their offerings. Allen and Seaman (2010) reported that 30 percent of U.S. college and university students are enrolled in at least one online course. The authors also reported that online enrollments in higher education in the U.S. grew at a much faster rate than overall enrollments in traditional university classes (Allen Seaman, 2010).For the most part, the students who are enrolled in online courses today are not the conventional distance learners – i.e., adult learners mostly living and working far from a university campus. Geographical distance has very little to do with today’s online learners; the majority are students who live on or around university campuses. Mann and Henneberry (2012) noted that many of the students that are taking online courses are of more traditional college age, between 18 and 24. For the most part, these students gravitate toward online offerings because the format fits their work and other obligations; without this convenience it would be very difficult for them to make the progress needed in their programs, and hence to complete and graduate on time. To meet this demand for flexibility, many institutions offer a considerable number of online courses (Allen Seaman, 2011).However, as these institutions continue to grow their online offerings, educators are increasingly concerned about how best to ensure academic integrity (Barnes Paris, 2013). Maintaining honesty in the learning environment becomes even more significant, as one of the fundamental aspects of instruction is the assessment of student learning. The purpose of this study therefore, is to examine student and faculty satisfaction with a solution to that problem–online proctoring.Review of Related LiteratureIn the last two decades, online education has made room for many private startup online colleges and universities throughout the United States, and a growing number of American traditional public higher education institutions have also entered into providing online education. These institutions face questions of how to navigate this change, and whether they can meet the needs of a contemporary student body that seeks flexible ways to attain their educational goals. At the same time, they struggle with ever-shrinking local and federal budgets, which may push them to expand their online education offerings as a way to grow budgets and shrink costs.Online education seems poised to become an educational norm in the years to come. A 2011 study by Ambient Insight Research noted that at the time of the study 1.25 million higher education students took all of their courses online, and 10.65 million students took some of their courses online (Adkins, 2011). Indeed, technology can greatly enhance the learning experience – bringing concepts and curricula to life in new ways. Likewise, when used effectively, online opportunities have provided higher education institutions with flexible options to expand their offerings into the global market (Casey, 2008). Harden (2013) also argues that the college classroom itself will in part become virtual, due to the advancements in information technology, the continued tuition increases that outpace inflation, and the country’s massive student loan debt.Much of the literature shows that the shift across higher education is forward-looking – toward online instruction. If this becomes the teaching platform of the future “as the greater interactivity and global connectivity that future technology will afford, the gap between the online experience and the in-person experience will continue to close” (Harden, 2013, p. 56). Allen and Seaman (2015) of the Babson Survey Research Group state that online learning growth accounted for nearly three-quarters of all U.S. higher education enrollment increases in 2014.More recently, Carey (2015) – after going back centuries, scrutinizing the model upon which America’s higher education is based, and that he considers flawed – stresses that information technology is capable of providing quality and affordable forms of higher education. Equally, DeMillo (2011) contends that the new information technology forces, which are moving higher education toward more virtual spaces, will challenge institutions clinging to centuries-old models of higher education.As public higher education institutions gradually embrace online education, maintaining academic integrity in cyberspace has added another challenging dimension, which is of great concern to educators and institutions alike. In one of the most comprehensive studies looking into academic dishonesty, Bill Bowers (1964) looked at 99 U.S. universities and colleges comprising 5,000 students and revealed that 75 percent of institutions surveyed had one or more incidents of academic dishonesty. This trend has been increasing: a study conducted by McCabe, Trevino, and Butterfield (2001) shows that both the magnitude and severity of academic dishonesty on college campuses have greatly increased in the last decades. Likewise, a study conducted throughout the United States and Canada by McCabe (2005) and encompassing 80,000 students from 83 different colleges and universities, found that “one in five students (21%) has engaged in at least one …. serious form of test or exam cheating” (p.3). Even more dramatically, Gabriel (2010) reported that “in surveys of 14,000 undergraduates over the last four years, an average of 61 percent admitted to cheating on assignments and exams” (p.2).There are relatively few articles written about academic dishonesty regarding unproctored online exams (e.g., Ercegovac Richardson, 2004; Strengold, 2004); and comparisons of academic dishonesty in online and traditional bricks and mortar environments have also been explored (e.g., Grijalva, Nowell, Kerkvliet, 2006; Shaw, 2004). However, our understanding about faculty and student satisfaction with online proctoring is very much unknown. …the literature is inconsistent when it comes to examining the severity and magnitude of academic dishonesty taking place online versus in a traditional setting. While studies done by Lanier (2006) found higher rates of cheating online, (Harmon Lambrinos, 2008; Grijalva, Nowell, Kerkvliet 2006), others such as Hart and Morgan (2010) and Stuber-McEwen, Wisely, and Hoggatt (2009) found lower rates of academic dishonesty online compared to those in traditional settings.Others have looked at performance levels on proctored and unproctored exams as indicators of academic dishonesty. For example, a Schultz, Schultz, and Gallogly (2007) study comparing unproctored online exams and proctored paper and pencil exams of the same course reported that those in unproctored online exams performed significantly higher than those in the proctored setting. Similarly, the study of Carstairs and Myors (2009) confirmed the above findings that students who took their exams with an unproctored method performed significantly higher than those who took proctored exams. Other studies took the general approach of arguing that since most online exams are unproctored, it is difficult to verify exam takers’ identity and deter cheating (Reynolds and Weiner 2009).Further, those taking online unproctored exams may have easier access to prohibited information during the exam (Reynolds Weiner, 2009). However, the limited literature that exists seems to indicate that regardless of where the exam is taking place (in an online or brick-and-mortar setting), academic dishonesty occurs. Indeed, factors contributing to academic dishonesty in the traditional classroom are also present in the online teaching and learning environment, and educators’ concerns are well warranted.It is well documented that online proctoring offers both faculty and students considerable advantages. Kinney (2001) noted that online proctoring is a valuable option for students who are geographically dispersed, and where it is not feasible for them to take their exams while on campus. Likewise, Tao and Li (2012) stated that when online proctoring is used to assess students attending conventional brick and mortar classes, online proctoring reduces instructional time dedicated to testing. This allows educators and students to engage more with the course contents. Furthermore, Naglieri, Drasgow, Schmit, Handler, Prifitera, Margolis, and Velasquez (2004) asserted that online exams are more scalable and efficient than pencil-and-paper exams. The next section, on methodology, discusses the characteristics of research participants and scale we used to assess student and faculty satisfaction with online proctoring.MethodologyTo understand students’ decision to participate in, and their satisfaction with classes that use online proctoring of exams, we assessed three courses in a single department over two consecutive semesters: two upper-level courses that were conducted entirely online and one large introductory course with online, face-to-face, and hybrid sections. Instructors agreed to participate in a focus group about their experiences, and to offer students extra course credit for participation. A total of 865 students consented to participate in the study: 339 in fully-online courses with online-proctored exams, 357 in traditional lecture courses with exams in a testing center, and 169 in hybrid courses with a mix of in-person and online course delivery, and exams in a testing center. Students taking their exams through the online proctoring service first were to check to insure their equipment met the service’s requirements of a webcam and microphone. Next, they established an account with the company and scheduled their exams during the exam time window established for the class. When they went to the proctoring site for their exams, a proctor checked to see they were in a reasonably private testing location, administered an identification protocol, and then entered the password into the Moodle exam. For the duration of the exam, live proctors monitored the students and the exam was video recorded in case there were any possible security breaches.Table 1: Participant detailsNumber of students in fully online coursesNumber of students in face-to-face classesNumber of students in hybrid coursesConsented to participate in study339357169Completed the post-survey316351169Completed both pre- and post-surveys114264126We assessed three courses in a single department over two consecutive semesters: two upper-level courses that were conducted entirely online and one large introductory course with online, face-to-face, and hybrid sections.We asked students to consent to participate in the study and to share their course grades with the researchers. Consenting participants completed a pre-survey near the beginning of the semester, before any exams had been given, as well as a post-survey immediately following the final exam. All procedures were approved by the Institutional Review Board. The 836 (316 online, 351 face-to-face, 169 hybrid) included in this study are the total number who both consented and completed the final post-survey. For comparisons that require both the pre-survey and the final data, the number reduces to 504 students (114, 264, 126).Both surveys followed the same structure, divided into five general topical areas: scheduling, format, technology readiness, questions about the exam, and the experience of being monitored. The pre-survey asked about previous experiences, reasons for choosing the specific class and its exam format, and expectations for online-proctored examination. The post-survey followed up on the specific experiences of the exam they had just completed, how it aligned with their expectations, and their considerations for future online-proctored exams. The course instructors’ role in the research study was to help gain student participation and also serve as research participants themselves. We asked instructors to keep notes about questions and concerns they might observe during the semester. They also participated in a focus group at the end of the first semester to discuss their views and experiences with online proctoring, insights into student experiences, and any effects of online proctoring on their classes.FindingsPre-Survey In this chapter, we report only results for those students taking their exams through the online proctoring service. Of the 114 online-proctoring students who took the pre-survey, 81% reported that they had previously taken exams online or on a computer, and the same percentage claimed no concerns about the technology they would need for exams. Despite this, more than one third (38%) reported being somewhat or not at all confident that they had the necessary equipment, and a majority (52%) reported being somewhat or not at all confident that they had the expertise to set up, use, and/or navigate any technological aspects of the exam environment (see table 2a). Only 40% of students reported being comfortable or very comfortable taking exams in the course’s testing environment, with 44% being somewhat comfortable and 16% selecting “not at all comfortable” (see table 2b).Table 2a: Pre-Survey Confidence in Online Proctoring TechnologyNot at all or somewhat confidentConfident or very confidentHave all equipment needed38%62%Have expertise to set up, use, and/or navigate52%48%Table 2b: Pre-Survey Comfort with Online Proctoring EnvironmentNot at all or somewhat comfortableComfortable or very comfortableComfort with taking exams in course’s testing environment60%40%Comfort with presence of proctor70%30%Half (50%) of online-proctoring students reported that the format of the class (online lectures and discussions, with proctored online exams) was their first choice of course format. In open-ended responses participants cited several reasons why the format was or was not their first choice, but even the most common themes—convenience of scheduling, preference for face-to-face conversation—were only mentioned by a small number of participants. Scheduling was cited by some as an advantage of online proctoring, but others viewed it as a disadvantage: while only 24% indicated concerns about scheduling exams for the course, participants were almost evenly split on how well scheduling options would work for them, with 53% responding that options work “extremely well” or “well”, and 47% responding that options work only somewhat or do not work. Similarly, when asked if they expected exam scheduling to be easier or harder than other courses they had taken, a third (33%) expected it to be about the same, one quarter (25%) expected it to be harder, and 40% expected it to be easier.In a series of questions about their expectations of being monitored during the exams, students expressed a significant level of discomfort with online proctoring. Of respondents, 70% expressed that they were not at all or somewhat comfortable, while only 30% indicated they were either comfortable or very comfortable with the presence of proctors during their online exams (see table 2b). While a majority (56%) expected the overall level of monitoring to be “about right,” fully 42% thought it would be “too much”, and very few participants (3%) thought it would be “too little”. Students were not concerned about the proctors distracting them from their exams, however: a majority (59%) indicated that in prior exams, proctors were never distracting, and a third (33%) selected “somewhat or sometimes distracting”, while just 9% stated that proctors were often or always distracting.While students did not expect to ask many questions of the proctors, the ability to do so was considered important to a substantial number. Just a small minority of students expected it to be likely that they would ask either procedural or content questions of the proctors, with the overwhelming majority on both topics considering it only somewhat or not at all likely (see table 3). About a quarter of students (26%) expected the proctors to be helpful or very helpful, and a similar number (24%) expressed that proctors for previous exams had been usually or always helpful (see table 4). Despite students’ relatively low expectation of their likelihood to ask a question or of the proctor’s helpfulness in addressing it, over a third (35%) of participants indicated that the ability to ask these sorts of questions was important or very important.Table 3: Questions Asked of ProctorsPre-Test Survey Reported LikelihoodPost-Survey Incidence of Asking questionsLikely or Very LikelySomewhat LikelyNot at all LikelyQuestion AskedAnswered SatisfactorilyProcedural Question18%51%31%9%71%Content Questions10%50%40%4%Table 4: Expectations and Experiences of Proctors’ HelpfulnessNot at all or somewhat helpfulHelpful or very helpfulPre-survey expectation74%26%Pre-survey prior experiences76%24%Post-survey experience64%36%Post-Survey With the background of expectations laid out, the post-survey (316 online-proctoring student participants) was gathered after the final exam for the classes being studied, and demonstrates some of the complexities, successes, and challenges of online exam proctoring. While the majority of students (63%) reported no problems with the testing environment, a sizeable minority (37%) reported a variety of problems. Most of these problems were minor issues with setup or scheduling, but a small number of significant problems were reported. A similar number found the test environment (e.g., trouble with scheduling, or with connecting to the test) to be “somewhat conducive” (32%) or “not at all conducive” (6%) to the students’ test-taking.Scheduling and unexpected wait times appear to be significant challenges for students using online proctoring. Student wait times after scheduled exam start times were approximately evenly distributed across “less than five minutes” (19%), “five to ten minutes” (27%), “ten to fifteen minutes” (20%), “fifteen to twenty minutes” (16%), and “more than twenty minutes” (18%). Half of students who waited at least five minutes (51%) thought the wait was acceptable, but nearly two-thirds of that same group (65%) found that the wait induced stress.On the other hand, the benefits of flexible scheduling were significant to many online-proctoring students. An eighth of participants (13%) were outside the metropolitan area in which the University is located at the time they took their final exam, and 3% report being outside the United States. While a small majority (53%) of students reported that they could have easily taken the exam in person, 34% indicated that this would have been more trouble, and 13% reported that they could not have taken the exam at all if they had needed to travel to the University to take it (see table 5a). Similarly, while a majority (56%) of respondents report that if the online proctoring tool were not available they would have had no trouble taking the exam, it would have been “somewhat more difficult” for 35% of students or “much more difficult” for 8% to take the exams; and 2% of respondents indicated that without online proctoring they would have been unable to complete the exams (see table 5b).Table 5a: Ability to Take Exams in Person if Online Proctoring were not availableEasily ableAble, but more troubleNot able53%34%13%Table 5b: Difficulty of Taking Exams by Any Other MethodNo more difficultSomewhat more difficultMuch more difficultNot possible56%35%8%2%Unlike traditional, in-person proctored exams, online proctoring presented participants with both the benefit of being able to choose a time for their exam, as well as the inconvenience of needing to find a time that would work. The data indicate that the benefits far outweighed the drawbacks. Very few students (6%) reported a schedule that was “not at all flexible”, with the vast majority (72%) reporting their schedule for exams as “somewhat flexible” and about a fifth (22%) claiming to be “very flexible”. In line with this, specific scheduling of online-proctored exams worked well, with more than four-fifths of students (82%) reporting that the scheduled time worked “extremely well” (28%) or “reasonably well” (54%). Nearly all participants (96%) reported that they were able to take the exam in a time and place that worked for them.The post-survey data indicate that some students became more comfortable with online proctoring, but that many students still had concerns. Whereas 70% had expressed that they were “not at all comfortable” or “somewhat comfortable” with the presence of proctors in the pre-survey, that number dropped slightly, to 63%, when reporting on their comfort during their final exam. Concern over the level of monitoring also improved, with 71% marking the level as “about right”, compared to 56% before the first exam. While 42% thought it would be “too much” before the first exam, only 26% reported that it had been “too much” during their final exam, and very few participants (3%) thought it was be “too little” (see table 6). Online proctors were rated by students as not especially distracting, and about as much so as proctors in other (presumably face-to-face) exams they had taken: a strong majority (62%, vs. 59% in the pre-survey) reported the proctors as “never or not at all distracting”, with over a quarter more (27%, vs. 33% in the pre-survey) selecting “somewhat or sometimes distracting”, for a total of 89% (vs. 92% in the pre-survey) indicating a low level of distraction.Table 6: Level of MonitoringToo MuchAbout RightToo LittlePre-survey expectation42%56%3%Post-survey experience26%71%3%Overall, even fewer students asked proctors procedural or content questions than they had anticipated (see table 3). A majority (71%) of students who asked questions reported that the proctor was able to answer their question(s) adequately. However, students reported that the proctors were not particularly helpful: a majority (63%) rated them not at all or somewhat helpful, while just a third (36%) rated them helpful or very helpful. This number, while low, still represented a substantial improvement over pre-survey expectations (see table 4).After taking a course for a semester using online-proctored exams, students remained divided on the relative merits and tradeoffs of this exam format. A small majority (58%) indicated that they would choose to use this testing environment in future classes. However, this response appears to be closely linked to whether or not the online-proctored format was or was not their first choice of course format. A strong majority of those who had reported this at the beginning of the semester to be their first choice of format (75%) reported that they would choose this testing environment again in the future, while an almost equally strong majority of those who had reported it to not be their first choice (70%) reported that they would not choose this testing environment again in the future.Faculty Focus Group Two overall themes became apparent in the analysis of the instructor focus group. First, they expressed three reasons for giving proctored exams: to assess student retention of information and their ability to remember significant portions of the content and relay that information without notes and supplemental information; to more or less force students to keep up with content as the course is progressing; and to encourage students to not just keep up with reading, but rather work through the material. They generally agreed that online proctoring is a must for online courses. However, they also expressed concerns that due to issues involved with setting up online exam proctoring, they are using fewer high-stakes assessments, reducing the overall number of exams, and looking for lower-stakes methods of assessment. The main challenge with low-stake online exams through the course management system (i.e., Moodle) is uncertainty that the students are doing the exam work by themselves.One instructor said:I had students who were looking for a way to compensate for some poor performance and I wanted to try to offer students something and I considered the possibility of adding another exam into our schedule … and I think the way I’m going to handle it is to do an unproctored exam and so that it is sort of like an assignment because I have to assume under those circumstances they will be using notes and textbook information in order to go through the exam material. So I would say it is a factor, though it’s not huge.Another instructor stated that:It really is about cost of proctoring and that’s why we are challenged with thinking about fewer assessments, because it’s expensive. I mean the challenge when we talk about having fewer exams, in my class there is a tremendous amount of material we have to cover. And to reduce the number of exams, means there will be more material on each exam, so it’s not a very attractive option. I’ve thought of open book exams as another possibility, so that’s why it’s not proctored, because they can just do whatever. But in the end you do want to know that the person who is responding is the person who is doing it.In addition to the overall themes identified above, instructors’ answers to the four questions of interest can be summarized as follows:Instructors’ experience with resolution of academic dishonesty: Our respondents reported no known academic dishonesty concerning exams delivered through our online testing procedures. There was a single incident where a proctor reported suspicious behavior, but upon further analysis, the incident was found to be nothing.Instructors’ experience with prevention of academic dishonesty: Our respondents expressed that online testing/proctoring is one way to assure the prevention of academic dishonesty. This way we know the person who took the exam is indeed the person for which the exam is intended.Instructors’ experience with time commitments: Our respondents did not hide the fact that there is quite a learning curve with online testing/proctoring. However, by the second or third exam, faculty reported that they were comfortable with the procedures.The first exam we had was really rocky, we had some issues, there was a password that did not match up, and there was a timing issue and students didn’t know what to expect and they were being delayed and I don’t think ProctorU was telling them the reason, so it took them a half hour/ 45 minutes to get it sorted out. So the second exam was better, and by the time we got to the third exam it went almost perfectly.For students who had not before experienced online proctoring, the idea of someone watching them on their computer might well produce anxiety, and our students’ responses supported that assumption.Faculty overall experience with online testing/proctoring: In addition to positive aspects of using it, faculty reported some administrative issues. One incident reported was that an online proctor cut a student off after a time period, even though it was an untimed test. Another more serious incident involved three occasions of a proctor telling a student of Muslim faith to take off her scarf, which she repeatedly refused. Our respondents stressed that the service should train their staff in a way that reflects our university’s respect for diversity. However, they also noted that the service’s staff has been responsive and accommodating, and when an issue was flagged for which they were responsible, they were quick to work with the students to find resolution.DiscussionThe results of our pre-survey of student expectations and concerns revealed several important issues that were prevalent in the three courses we studied. First, students did not express great concerns about taking their exams online. However, they did express concerns about whether their own equipment would be adequate for matching the online testing service’s technical requirements. The requirements for an operative webcam and other computer features may seem obvious to instructors and technically savvy students but complex to at least some of them and lead to uncertainty on their part. Second, we found the lack of students’ desire for scheduling flexibility to be somewhat surprising. This may be due to the alternatives students had, which gave them available times throughout the exam testing periods in addition to a choice of taking their exams in our on-campus testing center. It would likely have been a greater issue if, as in the typical situation, opportunities to take their exam were more restricted. Third, even though students’ revelations about their previous experiences with proctored online exams did not indicate this issue, a large number of them expressed concern that the proctors would distract them during the exam.Fourth, it appeared that whereas students didn’t actually expect to need the help of proctors during their exams, they attached importance to having them available. The difficulty might be the unusual situation students encounter when, instead of their instructor or at least a teaching assistant present during the exam, there is a disembodied presence monitoring them. And because the proctor is someone other than a member of the teaching staff, students fear they will be unable to get help if they need it.The post-survey showed a number of relevant findings: First, a significant percentage of students expressed dissatisfaction with delays they experienced in actually starting their exams and indicated they suffered stress on that account. We suspect that some of this problem may be due to students making reservations on short notice prior to their exam times. This would make it difficult for the proctoring service to schedule sufficient proctors to meet demand.Second, nearly half the students said they would have had problems coming to campus to take a traditional on-campus exam. Most of the students enrolled in the classes in this study lived near or on campus but some clearly resided elsewhere, making the online service necessary for them. Third, students expressed general satisfaction with the flexibility provided by online testing for scheduling their exams. Even though they initially said it would not have been a problem, they apparently thought differently in retrospect.Third, students expressed general satisfaction with the flexibility provided by online testing for scheduling their exams. Even though they initially said it would not have been a problem, they apparently thought differently in retrospect.Fourth, echoing a concern expressed in the pre-survey, many students said their biggest dislike for the online exam was that the proctors were distracting to them. Again, this is an issue that instructors could address early on during orientation or instruction on this type of exam format.Fifth, unlike anticipated concerns expressed on the pre-survey, students actually asked few questions of the proctors—thus not seeing that as a necessity. Most instructors strive to create exams that are clear to students and don’t require explanation. We believe this was the case for the courses in this study and students ultimately appreciated that.Finally, there were decidedly mixed results as to whether students would choose the online testing service as opposed to an on-campus exam in their future courses. It appears that students would choose the online service primarily if coming to campus was not a good option for them. Our findings suggest that there are several things that instructors should consider when deciding to use an online testing and proctoring service. Our findings suggest that there are several things that instructors should consider when deciding to use an online testing and proctoring service. Our student questionnaire data indicated that online education may not be as much about distance education overall, but rather that it provides learners with the convenience that fits their work and other obligations, without which it would be very difficult for them to take our courses or complete their programs. Our instructors’ view is fairly consistent with findings from the student surveys. Instructors repeatedly stressed that there are situations where it is really nice to have online options for students. And, it comes down to what the University has in mind in offering online courses: If the objective is to broaden the availability of our courses to meet the daily life circumstances of our students, then having online proctoring is completely necessary for those students. Having undertaken this study, we argue that fully online courses require online proctoring.Our faculty focus group revealed few problems experienced by, or greatly concerning to, instructors. It was clear that a learning process is necessary for faculty to develop procedures and expectations related to using online testing services. The instructors of the courses in this study were generally positive about the online testing service and were unanimous in their intention to continue using it. This is not to say there were no problems but they turned out to be manageable and the instructors generally agreed that the benefits certainly outweighed them.First, we suggest that instructors thoroughly familiarize themselves with how the services work so they can anticipate students’ concerns.Second, instructors should identify students’ technical difficulties and try to address them by spending time familiarizing students with how to get ready for and ultimately take their exams.Third, instructors need to anticipate what scheduling issues students might have. At our university, traditional in-class exams are managed by a central office so that students know when their exams are scheduled and know that they do not conflict with their other classes’ exams. Freeing up the process with an online testing service may seem to make things easier but it’s possible that too many choices may complicate matters.Fourth, depending on the online service they choose, instructors need to anticipate students’ fear of proctor intrusiveness and develop ways to address it. We found this concern on both our pre- and post-surveys. Different online services now offer options ranging from only live proctors, to only recorded sessions with no live proctor contact, to a mix of both. Not having a live proctor might eliminate the worry of some students that someone is spying on them, but might also make them feel abandoned if by chance they need help with the technology or clarification of an exam item. This issue is closely related to the delays in starting their exams, which some students expressed. Online exam vendors working with live proctors need to schedule an adequate number of proctors, and at very busy times wait lengths are likely to increase. Services that simply record may not have these problems. We recommend that instructors consider these issues and how they can make their exam instructions and questions as “transparent” as possible so that students don’t feel left on their own.Fifth, instructors should endeavor to determine, early on, where students are taking the course from. Knowing how many are restricted from coming to campus (if given a choice between campus and online exams) helps with planning and avoiding problems. Determining other motives students might have for taking either online or on-campus exams may prove valuable as well.Limitations and Recommendations for Future StudiesOne limitation of this study has to do with the focus group discussion analysis. The focus group findings reported here are the results of a first phase data analysis (i.e., visual data analysis – isolation of themes and sentences). We did not deploy qualitative analysis software to transcribe and analyze our focus group findings, and we consider this as a limitation that should be addressed in future research.To reiterate a point made above, many of our respondents reported a significant level of discomfort with online proctoring. Therefore, we think future studies in this area could consider examining antecedents such as anxiety induced by remote online proctoring and its effect on students’ exam performance. One approach might be studies that compare face-to-face and online proctoring environments from a test anxiety perspective, as well as its implications on students’ exam performance. In addition, whether online proctoring was a student’s first choice of test taking format seems to have a significant impact on how they react to the testing environment. This warrants further consideration and research. As online courses increase in number, online exams seem likely to increase as well; therefore, it will be important to make the process of taking them go as smoothly as possible for both students and their instructors.ReferencesAdkins, S. (2011). The US Market for Self-Paced eLearning Products and Services: 2010-2015 Forecast and Analysis. Ambient Insight.Allen, I., Seaman, J. (2015). Grade level: Tracking Online Education in the United States. Babson Survey Research Group.Allen, I. and Seaman, J. (2011). Going the Distance: Online Education in the United States, 2011. Sloan Consortium. Babson Survey Research Group.Allen, I. E., Seaman, J. (2010). Class Differences: Online Education in the Unite States, 2010. Needham, MA: Babson Survey Research Group.Barnes, C., Paris, B. L. (2013). An Analysis of Academic Integrity Techniques Used in Online Courses at a Southern University. Northwest Decision Sciences Institute Annual Meeting Proceedings.Bowers, W. J. (1964). Student Dishonesty and its Control in College. New York: Bureau of Applied Social Research, Columbia University.Carey, K. (2016). The End of College: Creating the Future of Learning and the University of Everywhere. Riverhead Books (Hardcover).Carstairs, J., Myors, B. (2009). Internet Testing: A Natural Experiment Reveals Test Score Inflation on a High-Stakes, Unproctored Cognitive Test. Computers in Human Behavior, 25(3), 738-742.Casey, D. (2008). A Journey to Legitimacy: The Historical Development of Distance Education Through Technology. TechTrends, 52(2), 45-51.DeMillo, R. A. (2011). Abelard to Apple: The Fate of American Colleges and Universities. MIT Press.Ercegovac, Z., Richardson, J. V. (2004). Academic Dishonesty, Plagiarism Included, in the Digital Age: A Literature Review. College Research Libraries, 65(4), 301-318.Gabriel, T. (2010). To Stop Cheats, Colleges Learn Their Trickery. New York Times, 5.Grijalva, T. C., Nowell, C., Kerkvliet, J. (2006). Academic Honesty and Online Courses. College Student Journal, 40(1), 180-185.Harmon, O. R., Lambrinos, J. (2008). Are Online Exams an Invitation to Cheat?. The Journal of Economic Education, 39(2), 116-125.Hart, L., Morgan, L. (2010). Academic Integrity in an Online Registered Nurse to Baccalaureate in Nursing Program. The Journal of Continuing Education in Nursing, 41(11), 498-505.Harden, N. (2013). The end of the University as We Know it. The American Interest, 8(3), 54-62.Kinney, N. E. (2001). A Guide to Design and Testing in Online Psychology Courses. Psychology Learning Teaching, 1(1), 16-20.Mann, J. T., Henneberry, S. R. (2012). What Characteristics of College Students Influence Their Decisions to Select Online Courses?. Online Journal of Distance Learning Administration, 15(4), n4.McCabe, D. L. (2005). Cheating Among College and University Students: A North American Perspective. International Journal for Educational Integrity, 1(1).McCabe, D. L., Treviño, L. K., Butterfield, K. D. (2001). Cheating in Academic Institutions: A Decade of Research. Ethics Behavior, 11(3), 219-232.Naglieri, J. A., Drasgow, F., Schmit, M., Handler, L., Prifitera, A., Margolis, A., Velasquez, R. (2004). Psychological Testing on the Internet: New Problems, Old Issues. American Psychologist, 59(3),150.Reynolds, D. H., Weiner, J. A. (2009). Online Recruiting and Selection: Innovations in Talent Acquisition. John Wiley Sons.Schultz, M. C., Schultz, J. T., Gallogly, J. (2007). The Management of Testing in Distance Learning Environments. Journal of College Teaching Learning, 4(9), 19-26.Shaw, D. C. (2004). Academic Dishonesty in Traditional and Online Courses as Self-Reported by Students in Online Courses.Sterngold, A. (2004). Confronting Plagiarism: How Conventional Teaching Invites Cyber-Cheating. Change: The Magazine of Higher Learning, 36(3), 16-21.Stuber-McEwen, D., Wiseley, P., Hoggatt, S. (2009). Point, Click, and Cheat: Frequency and Type of Academic Dishonesty in the Virtual Classroom. Online Journal of Distance Learning Administration, 12(3), 1-10.Tao, J., Li, Z. (2012). A Case Study on Computerized Take-Home Testing: Benefits and Pitfalls. International Journal of Technology in Teaching and Learning, 8(1), 33-43.Communications regarding this chapter should be sent to Daniel Woldeab, College of Individualized Studies, Metropolitan State University, daniel.woldeab@metrostate.eduPK
CMڮ meta.xmlMicrosoftOffice/14.0 MicrosoftWordThiyagarajanThiyagarajan2015-02-27T16:23:00Z2015-02-27T16:24:00Z1PT60SPK
CM}t settings.xmlPK
CMjC C
styles.xml1 PK
CM media/PK
CM?t2q media/phillippon2.pngPNG
IHDR g sRGB gAMA a pHYs od IDATx^UU?1,cu>?}>owl{mQ+:"VT MA@"(6Dł%߳.{ν{ν듬v[g~(G# Hl:DYµ#_~epik֬)_xqlC^;3Kw\0bĈ5W^1۰z?x0x`
608VZeW^p f]xo} f̘Qږ%n;ݬtu$)5+W4%g/Y$ܛ/rnĽkD"ю8tUy7rD;hD(ӧOOE/ǡhG`?FаH#_/lbŊpOe
xy뭷µhH$XIG 922z#0+Ƿ~V<>p-}2D<وJ$Y&dDD`R`Z9^ZR9?E6`YzF1#~k0;iw}>˟̾4H5iskMy'n8!H>=hVVҊ"Q^O>*:Q#'L%E.\)DyC1Kxdϝُ4i$FN#K+~gMRS{6vK#q.QS)47o?\n߿ttm:u8c?H0x, z"p9NQH4ƏS,Q3TDJ={rHxºƄZH,Z!aԩ zH$ eGL$%}pOGHՃ%Z]CG
ģH4V#:X$*yõ:5uE<
|E;/3<{Q udM|JMm
rfǥ\Z(r*GH9bp6$j\j-#!%;S̺|:(b5+np