130 W. Nippon St.
Philadelphia, P.A.
U.S.A. 19119
Phone: (215) 248-1248
Email: celebration@pobox.upenn.edu
HomePage
Mark, M.A. The VCR Tutor: Design and evaluation of an intelligent tutoring system. Master's Thesis, University of Saskatchewan, Saskatoon, Saskatchewan, 1991. Advisor: Prof. J. E. Greer, Department of Computational Science. (Also available as ARIES Research Report 91-7, ARIES Laboratory, Department of Computational Science, University of Saskatchewan, Saskatoon, Saskatchewan, 1991.)
This thesis describes the design, implementation and evaluation of a tutoring system for instruction in device operation. While design and implementation are important components of this research, a major focus is evaluation. Evaluation is used to compare versions of the system and examine psychological and educational theories underlying system development.
The purpose of the tutoring system is to teach an adult how to program a video cassette recorder (VCR) to automatically record television programs. Actions involved in operating a VCR are procedural, but their execution may depend upon cognitive knowledge of the device, its behaviour, and the relationships between device features and device actions. Questions about the relationship of knowledge to learning in the VCR domain include: "Does the domain involve only procedural knowledge, or factual and referential knowledge as well?" "Will knowledgeable feedback in a computer-based instructional system support learning more effectively than other types of feedback?" "How should instructional feedback be designed to best facilitate learning of procedures in this domain?"
To examine these questions, prompting, sequence-based, device-based, and knowledgeable versions of a VCR tutoring system have been implemented. They incorporate different tutorial approaches and types of knowledge. An experimental study has compared the effectiveness of these versions. Subjects who used the knowledgeable tutor learned to program a VCR using fewer steps and with fewer errors than those who used the prompting version. Students with knowledgeable tutoring showed fewer types of errors than prompting students and completed the testing task more quickly. Subjects who used the device-based version of the tutor executed the task with fewer types of errors than prompting students, but took significantly longer than prompting students to learn the task.
These results have implications for the design and development of instructional systems for the VCR domain. They support the view that the VCR programming domain involves factual and referential as well as procedural knowledge. They demonstrate that knowledgeable feedback supports learning in the VCR domain more effectively than other types of feedback, and they suggest that the presentation of knowledgeable feedback is a desirable design for a tutoring system in this type of domain.
Mark, M. A., & Koedinger, K. R. Strategic Support of Algebraic Expression-Writing. To appear in the Proceedings of the Seventh International Conference on User Modeling. Banff, Alberta, Canada, June 20-24, 1999. [UM99 Website]
In completing a problem with the PAT Algebra I Tutor, students solve a number of algebraically related tasks. We have observed a characteristic pattern of students' success rates at these tasks. We have also observed that students' success on specific skills may differ depending on whether students previously carried out algebraically related activities in the same problem. Finally, the sequences in which tasks are completed may indicate overall problem-solving strategies of the student. These observations have important implications for the future design of the PAT Tutor and its user model.
Mark, M. A., Koedinger, K. R., & Hadley, W. H. Elaborating Models of Algebraic Expression-Writing. Proceedings of the Fourth International Conference on Intelligent Tutoring Systems. San Antonio, Texas, August 16-19, 1998.
This paper discusses the refinement of one part of the cognitive model for the PAT Algebra I tutor, algebraic expression-writing, in the context of developing the overall PAT tutor. During the tutor's use in Pittsburgh schools, from 1992 to 1997, considerable elaboration of our original production rule models has occurred. Expression-writing skills have been a particularly rich focus of development. We discuss development of the expression-writing model over time, from our initial model of expression-writing as a single skill, to our current characterization of expressions in terms of features which cause students difficulty, supporting these changes in the model with formative analyses. Direct observation of students in class, step-by-step PAT tutor protocols tracing student behavior, and statistical analysis of protocol data, have all informed our understanding of students' skills, and our refinement of this part of PAT's cognitive model.
Koedinger, K.R., Anderson, J. R., Hadley, W. H. & Mark, M.A. Intelligent tutoring goes to school in the big city. International Journal of Artificial Intelligence in Education (Special Issue of AI-ED 95 best papers), 1997, 8, 30-43.
Koedinger, K.R., Anderson, J. R., Hadley, W. H. & Mark, M.A. Intelligent tutoring goes to school in the big city. Proceedings of the Seventh World Conference on Artificial Intelligence in Education. Washington, D.C., August 16-19, 1995. (Awarded 1 of 2 Best Paper Awards.)
This paper reports on a large-scale experiment introducing and evaluating intelligent tutoring in an urban High School setting. Critical to the success of this project has been a client-centered design approach that has matched our client's expertise in curricular objectives and classroom teaching with our expertise in artificial intelligence and cognitive psychology. The Pittsburgh Urban Mathematics Project (PUMP) has produced an algebra curriculum that is centrally focused on mathematical analysis of real world situations and the use of computational tools. We have built an intelligent tutor, called PAT, that supports this curriculum and has been made a regular part of 9th grade Algebra in 3 Pittsburgh schools. In the 1993-1994 school year, we evaluated the effect of the PUMP curriculum and and PAT tutor use. On average, the 470 students in experimental classes outperformed students in comparison classes by 15% on standardized tests and 100% on tests targeting the PUMP objectives. This study provides further evidence that laboratory tutoring systems can be scaled up and made to work, both technically and pedagogically, in real and unforgiving settings like urban high schools.
Mark, M.A. & Greer, J.E. The VCR tutor: Effective instruction for device operation. Journal of the Learning Sciences, 1995, 4(2), 209-246.
A critical issue for intelligent tutoring systems is whether knowledgeable diagnosis and feedback offers instructional benefits compared with less knowledge-intensive instructional methods. The effectiveness of different instructional methods is examined in a device operation task: setting a VCR to record a program. Four tutoring simulations use different instructional approaches emphasizing specific types of knowledge. The possibility of acquiring this knowledge is manipulated by increasing opportunities to interact with the simulation and by explicitly providing knowledgeable diagnosis and feedback. Eighty novice adults were randomly assigned to the four tutors for training. Planned comparisons of learner achievement indicate greater learning with knowledgeable diagnosis and feedback, compared to correctness feedback. Analysis of variance to detect predicted linear trends shows increased instructional effectiveness across versions. Results are not simply a function of differences in training time. They support the contention that knowledgeable diagnosis and feedback can offer benefits over other instructional methods.
Mark, M.A.& Greer, J.E. Evaluation methodologies for intelligent tutoring systems. Journal of Artificial Intelligence in Education (Special Issue on Evaluation), 1993, 4 (2/3), 129-153. [one of the top 20 most referenced papers of the International Journal of Artificial Intelligence and Education]
As intelligent tutoring system (ITS) issues are investigated and intelligent tutoring systems are developed, evaluation methodology becomes important. Basic researchers, system developers, and educators working with ITS all have motives for becoming involved in ITS evaluation. In formative evaluation, researchers examine a system under development, to identify problems and guide modifications. By contrast, summative evaluation is carried out to support formal claims about the construction, behaviour of, or outcomes associated with a completed system. Different methodologies are suitable for different types of evaluation, some focusing on internal considerations, such as architecture and behaviour, others on external considerations, such as educational impact.
This paper draws upon the areas of intelligent tutoring systems research, expert systems design, computer-based instruction, education, and psychology to identify techniques for the formative and summative evaluation of ITS. Evaluation techniques are discussed in terms of the motivations for using them, situations for which they are particularly suitable, and their strengths and weaknesses. Techniques are illustrated with reference to actual studies whenever possible.
Winne, P.H., Butler, D.L., McGinn, M., Sugarman, J.H., Jones, M., Mark, M., & Field, D. "STUDY: A tool for authoring adaptive learning environments and for advancing instructional research." ITS '92: System Demonstrations (Second International Conference on Intelligent Tutoring Systems). Montreal, June 10-12, 1992.
STUDY is a tool for authoring Adaptive Learning Environments. We refer to the system as state-of-the-art because it is based fundamentally on knowledge of computing and instructional psychology that is available now. Rather than offering a revolutionary re-definition of education or implying radically new views of learning and the learner, STUDY has been designed and implemented to step modestly beyond high quality instruction that can be provided by a superlative textbook and its accessories. To achieve this goal, STUDY can be used to create tutorials in which students develop more than just knowledge of the subject matter provided in the text. STUDY also serves as a valuable research tool for bootstrapping research regarding the nature of effective instruction in Adaptive Learning Environments.
Mark, M.A.& Greer, J.E. "Methods for intelligent tutoring system evaluation." Proceedings of the East West Conference on Emerging Computer Technologies in Education, 6-9 April 1992, Moscow, Russia. ICSTI: Moscow, Russia, 1992. (Awarded 1 of 5 Best Paper Awards, for ITS-related papers presented at the conference.)
As intelligent tutoring system (ITS) issues are investigated and intelligent tutoring systems are developed, the issue of evaluation begins to receive attention. In formative evaluation, researchers examine a system under development, to identify problems and guide modifications. Summative evaluation is carried out to support formal claims about the construction, behaviour of, or outcomes associated with a completed system. Different methodologies are suitable for different types of evaluation, some focusing on internal considerations such as architecture and behaviour; others on external considerations such as educational impact. This paper draws upon the areas of ITS research, expert systems design, computer-based instruction, education and psychology to identify techniques for the formative and summative evaluation of ITS. Evaluation techniques are discussed in terms of motivations for using them, situations for which they are particularly suitable, and strengths and weaknesses. Techniques are illustrated with reference to actual studies whenever possible.
Jones, M., Mark, M., Field, D., & Winne, P.H. The STUDY system: A learning environment to facilitate reading comprehension and studying. Paper presented at Survival Skills in the Nineties: A conference about media and information literacies, Edmonton, Alberta, November 1 - 2, 1991.
This abstract is not currently available.
Mark, M.A.& Greer, J.E. "The VCR Tutor: Evaluating instructional effectiveness." Proceedings of the 13th Annual Conference of the Cognitive Science Society, August 7-10, 1991. COGSCI: Chicago, Illinois, 1991. (Also available as ARIES Research Report 91-5, ARIES Laboratory, Department of Computational Science, University of Saskatchewan, Saskatoon, Saskatchewan, 1991.)
People use a wide variety of devices. Operation of a device can usually be described in terms of knowledge of specific procedural sequences. However, execution of procedures may also depend upon knowledge of the device, its behaviour, and the relationships between device features and device actions. A video cassette recorder (VCR) is one commonly used device. Programming a VCR to automatically record a chosen television program is an example of a device manipulation task. In designing a device tutor, it is relevant to ask how instruction about device operation should be designed, and to ask whether knowledge engineering for a device tutor should focus on procedural knowledge or involve factual and referential knowledge as well. Four versions of a tutoring system for the VCR device and programming task have been implemented, incorporating different tutorial approaches using different types of knowledge. The effectiveness of these versions has been examined experimentally. Subjects who used the knowledgeable tutoring version learned to program a VCR simulation using fewer steps and with fewer errors and error types than subjects who used a prompting version of the tutor.
Mark, M.A.& Greer, J.E. "Tutoring systems for device operation." Proceedings of the International Conference for Cognitive Science for the Development of Organizations (ICO '91), Montreal, May 2-4, 1991. (Also available as ARIES Research Report 91-2, ARIES Laboratory, Department of Computational Science, University of Saskatchewan, Saskatoon, Saskatchewan, 1991.)
People use a wide variety of devices. Operation of a device can usually be described in terms of knowledge of specific procedural sequences. However, execution of such procedures may also depend upon knowledge of the device, its behaviour, and the relationships between device features and device actions. It is relevant to ask how instruction about device operation should be designed, and whether knowledge engineering for a device tutor should involve only procedural knowledge, or factual and referential knowledge as well.
A video cassette recorder (VCR) is one example of a commonly used device. Programming a VCR to automatically record a chosen television program is an example of a device manipulation task. Four versions of a tutoring system for this task and device have been implemented, incorporating different tutorial approaches and types of knowledge. The effectiveness of these versions has been examined in a one-way ANOVA with four experimental groups. Students who used the knowledgeable tutoring version were found to be most successful at carrying out the task. They used less steps to program the tutor, and programmed the tutor in less time, than any of the other groups.
Mark, M.A. "Evaluation of intelligent tutoring systems." Proceedings of the Second Annual Graduate Symposium on Computational Science, April 5, 1990, Saskatoon, Saskatchewan. Saskatoon, Saskatchewan: Department of Computational Science, 1990. (Also available in extended form as ARIES Research Report 90-2, ARIES Laboratory, Department of Computational Science, University of Saskatchewan, Saskatoon, Saskatchewan, 1990.)
As intelligent tutoring system (ITS) issues are investigated and intelligent tutoring systems are developed, the issue of evaluation begins to receive attention. Basic researchers, system developers, and educators working with ITS all have motives for becoming involved in their evaluation. At different times in the life cycle of a project, researchers may be concerned with different types of evaluation. In formative evaluation, researchers examine a system under development, to identify problems and guide modifications. In contrast, summative evaluation is carried out to support formal claims about the construction, behaviour of, or outcomes associated with a completed system.
This paper draws upon the areas of ITS research, expert systems design, computer-based instruction, education and psychology to identify techniques for the formative and summative evaluation of ITS. Evaluation techniques are discussed in terms of the motivations for using them, situations for which they are particularly suitable, and their strengths and weaknesses. Techniques are illustrated with reference to actual studies whenever possible.
Greer, J. E., Mark, M.A., & McCalla, G.I. "Incorporating granularity-based recognition into SCENT." Proceedings of the Fourth International Conference on Artificial Intelligence and Education, Amsterdam. May, 1989, 107-115.
Understanding the strategies that underlie students' problem-solving behaviour in complex domains requires significant expertise and large amounts of domain specific knowledge. In this paper, a computational model for granularity-based recognition of strategies in LISP programming is described in accordance with theoretical principles governing granularity in knowledge representation. The implemented system recognizes students' programming strategies at various levels of abstraction and aggregation and provides an overall diagnosis. The design of the system guarantees that full recognition will occur at some grain size and that partial recognition at finer grain sizes is probable. The combination of these two types of recognition provides a more robust understanding of students' strategies. Granularity-based strategy recognition seems to be an appropriate model for diagnosis in the SCENT advisor.
Coulman, R., Bhuiyan, S., Mark, M., Prasad, B., Schweigardt, M., Greer, J., & McCalla, G. "The Circuits Intelligent Tutoring System." Proceedings of the Sixth Canadian Symposium on Instructional Technology, Halifax, Canada, May 1989.
This abstract is not currently available.
McCalla, G., Greer, J., & Members of the SCENT Research Team. "Intelligent advising in problem-solving domains: The SCENT-3 architecture." In Proceedings of Intelligent Tutoring Systems, ITS-88. Montreal, June 1-3, 1988.
The SCENT project has focussed on developing an intelligent advising environment for students learning to program in Lisp. This paper presents a status report on SCENT, a summary of recent research contributions, and the design of the new SCENT-3 system. SCENT-3 is an architecture for a full scale student advising system. Work on SCENT-3 is progressing concurrently on many fronts with new research contributions in dynamic planning and blackboard control, student modelling, strategy judging and diagnosis, and program analysis. The SCENT-3 architecture has been designed to achieve the goal of creating an intelligent advising system for higher order problem-solving activities with minimal domain dependence. Much more research remains to be done to fully prove out SCENT-3, but the investigations carried out so far are promising.
Mary Mark Ockerbloom celebration@pobox.upenn.edu