Submission Type: Short Paper about work-in-progress

Submission Title: TEACHING THE TUTOR: ACQUIRING DOMAIN KNOWLEDGE THROUGH INTERACTIVE  INSTRUCTION

Topic Areas: Authoring, Architectures for Educational Technology Systems, Interactive Learning Environments, Intelligent Environments

Presenter: R. Morelli

Department of Computer Science

Trinity College, Hartford, CT 06106

ralph.morelli@mail.trincoll.edu

860-297-2220

Contact:  R. Morelli

TEACHING THE TUTOR: ACQUIRING DOMAIN KNOWLEDGE THROUGH INTERACTIVE INSTRUCTION

1. Introduction

Intelligent Tutoring Systems (ITSs) and other knowledge based systems are characterized by their representation of domain knowledge in a form that can be processed and manipulated by the system. A main bottleneck in developing such systems is the knowledge acquisition problem -- the problem of acquiring and encoding the knowledge base. Most ITSs have relied on some form of preprogramming in which the knowledge engineer acquires the information from a domain expert and programs it directly into the system. This approach is labor intensive and requires a skilled programmer. A teacher who wishes to develop an ITS cannot easily input the required knowledge directly. What if there were a way for the domain expert to teach the system the requisite knowledge?

This work-in-progress reports on research that addresses the knowledge acquisition problem. The primary issue is to determine to what degree an ITS architecture can learn its knowledge base. Soar Instructable Teaching Agent (SoarITA) is a prototype of an architecture that learns a significant part of its domain by taking natural language instructions from a human teacher.

SoarITA is based on Instructo-Soar, a machine learning architecture that supports situated, interactive tutorial instruction within an ongoing agent (Huffman and Laird, 1995). Instructo-Soar successfully demonstrated a number of important knowledge acquisition features that appear to be applicable to tutoring and help systems:

• It can learn general knowledge from specific instructions in an efficient manner and integrate its learned knowledge with its prior knowledge. Once Instructo-Soar is taught how to “Pick up the green block,” it knows how to pick up a block of any color.

• It can take instructions that apply to its current situation or to counterfactual situations. It can handle instructions such as “To turn off the light, push the red button” and “If the light is bright, then dim the light.”

• There is no inherent limitation on the type of knowledge it can learn. This is particularly important for the tutoring domain which typically consists of knowledge about the student, the interface, the task domain, the curriculum, and tutoring principles.

The research to be reported at the conference consists of adapting Instructo-Soar’s machine learning capability for the ITS domain. This has involved reorganizing Instructo-Soar into domain-specific and domain-independent components and adding new knowledge for SoarITA’s domain. Work to date has focused on determining that SoarITA can learn and integrate the types of knowledge important in ITSs -- knowledge about the task, about tutoring, and about the interface.

2. Teaching SoarITA Logic Proofs.

The ITS framework being used in this project is part of an interactive learning environment (ILE) that supports instruction in elementary symbolic logic [Morelli, Pelton and Dinkins, 1995]. The knowledge acquisition work described here should be incorporated into the ILE by the time of the conference.

2.1 SoarITA’s Built-in Knowledge. SoarITA contains the following initial knowledge:

• Natural language processing knowledge, originally developed for NL-Soar [Lewis, 1993] and used in Instructo-Soar.

• Instruction processing knowledge that maps instructions -- “Write the first premise in the proof window!” into the agent’s internal representation language.

• Task knowledge, including knowledge of logic objects (arguments, premises, inference rules), interface objects (windows, buttons), tutoring objects (explanations, exercises), the various relationships among them (Argument-1 has premise-1), as well as a set of primitive operators that the system knows how to perform -- e.g., write a premise in the proof window.

2.2 The Logic Domain. One of the most difficult skills taught in an introductory logic course is that of developing deductive logic proofs. For example, consider the problem of constructing a proof for the following argument:

A -> (C&D)

A / Conclusion: C

A correct solution to this problem would be the following proof:

1. A -> (C&D)

2. A / Conclusion: C

3. C&D Justification: Follows from steps 1 and 2 by modus ponens

4. C Justification: Follows from step 3 by simplification

In this example, lines 1-2 represent the premises of the argument, with the conclusion being represented on the right side of line 2. Proving the validity of the argument is a matter of showing that its conclusion can be derived from its premises by application of valid inference rules. Steps 3 and 4 are derived by applying known inference rules, in this case modus ponens or simplification. The system’s interface consists of a problem window, which presents the problem, and a proof window, into which the student is supposed to write the proof.

2.3 Teaching SoarITA To Do Logic Proofs. The following example illustrates how SoarITA is taught the new operation, prove (arg) where arg is the above argument.

1 Prove the validity of the argument. How do I do that?

1.1 Write the first premise in the proof window. OK.

1.2 Write the second premise in the proof window. OK

1.3 Write the conclusion in the proof window. OK

1.4 Apply modus ponens to lines 1 and 2. OK

1.5 Apply simplification to line 3. OK

1.6 The operator is finished. OK

When the first instruction is given, SoarITA comprehends it by creating a semantic structure in which “the argument” is mapped to an object in its environment. However, since this is an unknown operation, the semantic structure does not map to any known operator and therefore the system begins accepting instructions. Each subsequent (indented) instruction is comprehended, remembered and executed, and as a result the system learns the implementation of prove (arg) as an over-specific, lock-step procedure. When the instruction sequence is “finished” the goal of the new operator is induced by collecting those relevant features that changed between its initial and final states. In this case the key change is that the conclusion of the argument (“C”) now occurs as the last line of the proof; thus the goal of the new operator becomes, in part, to derive “C.” Once SoarITA knows the new operator’s goal and its implementation sequence, it can “explain” how each step in the implementation leads to the goal of proving arg. It uses a from of explanation based learning (EBL) to learn a more general form of the new operator. [0] The result of this process is the learning of a set of production rules that implement the new operator. For example, the following production proposes the first step of prove(arg):

if the goal is new-op-15 (?arg)

and ?arg has first premise ?prem

and ?prem is not written in the proof-window

then propose operator write (?prem, proof-window)

Of course, some of the individual steps in the above instruction sequence could be broken down into more primitive steps. Learning a more fine-grained representation within a given domain -- something that human tutors are frequently called upon to do -- is straightforward for SoarITA, provided it is equipped with the appropriate lexical and ontological knowledge.

2.4 Teaching SoarITA To Teach Logic Proofs. Learning how to do proofs is different from teaching someone else how to do a proof. For an ITS, the type of knowledge learned in the previous example could be used as an example of a correct logic proof (“Watch how I do it”) or as an answer if the student gives up (“The correct proof is...”). But, as the following example illustrates, to equip SoarITA to teach a student how to do that proof merely requires that SoarITA be taught a different set of operations:

Teach how to prove the argument. How do I do that?

Say “first i copy the argument into the proof window.” OK

Set up the proof window with the argument. OK.

Say “now I will apply modus ponens to lines 1 and 2” OK

Show how to apply modus ponens to lines 1 and 2. How do I do that?

Say “notice that line 1 matches the first premise of modus ponens.” OK

Show that line 1 matches the first premise of modus ponens OK

The operator is finished. OK

...

The operator is finished. OK

3. Discussion of Preliminary Results and Future Work.

Our work so far indicates that it is possible to teach an ITS the appropriate kinds of task and tutoring knowledge, and it does seem feasible that such a system could serve as a basis for an authoring environment for ITSs and other interactive help systems. The following comments identify a couple of important research issues.

3.1 The Transfer Problem. To what degree does knowledge learned in one situation transfer to another situation? Obviously it is desirable to get as much transfer as possible. Our preliminary results indicate that certain types of transfer are easy -- e.g., every logic proof starts by copying the argument to the proof window -- but that other types are not -- e.g.the inference rules used in one argument won’t easily transfer to another. A major question for the ITS context, is whether any useful transfer can be achieved between learning how to do a task and how to teach the task. Data on these types of questions will be presented at the conference.

3.2 The Initial Knowledge Problem. Our work so far indicates that it is relatively straightforward to add new levels of generality to the knowledge base, either finer or courser grained. This kind of flexibility should lead to ITSs that can be more easily adapted to fit the needs of the students. As the above examples suggest, this flexibility depends upon built-in ontological and lexical knowledge. Our preliminary efforts indicate that some of this knowledge can be acquired automatically either through instruction or through other methods. By conference time it should be possible to indicate what the limits are in this area.

3.3 Related Work. There have been other efforts to use machine learning in ITS development, including COACH, a tutoring system that monitors and adapts its user model [Selker, 1994]. Similarly, personal assistant agents such as that found in [Maes, 1994] and programming by demonstrations systems [Smith et al., 1994] utilize machine learning techniques in instructional or help systems. There are also efforts, such as REDEEM [Major, 1995], that provide environments and tools for ITS authoring, although these do not involve machine learning. What makes SoarITA unique is the effort to use machine learning to create an authoring environment built around the idea of teaching the tutor -- i.e., using full-fledged natural language instructions to teach the system its knowledge base.

4. References.

Huffman, S.B. and Laird, J.E. (1995). Flexibly Instructable Agents. JAIR 3, 271-324.

Newell, A. (1990). Unified Theories of Cognition. Cambridge, MA: Harvard University Press.

Lewis, R.L. (1993). An Architecturally-Based Theory of Human Sentence Comprehension. Ph.D. thesis. School of Computer Science, Carnegie Mellon University, Pittsburgh, PA.

Major, N. (1995). REDEEM: Creating Reusable Intelligent Courseware, Proc. of AIED-95, World Conference on AI in Education, Washington, D.C., 75-82.

Maes, P. (1994). Agents that reduce work and information overload. CACM 37(7), 30-40.

Morelli, R., Pelton, G. and Dinkins, B. (1995). A tutoring architecture that learns. Proc. of AIED-95, pg. 586.

Selker, P. (1994). COACH: A tutoring agent that learns. CACM 37(7), 92-99.

Smith, D.C., Cypher, A., Spohrer, J. (1994). KIDSIM: Programming agents without a programming language.CACM 37(7), 55-67.

---

[0] Instructo-Soar’s situated explanation method [Huffman and Laird, 1995] uses both analytic and inductive learning algorithms and heuristics , which will be explained more fully at the conference. Also as their names suggest, both Instructo-Soar and SoarITA are based on the Soar cognitive architecture [Newell, 1990], which will also be described at the conference.