5/23/00

Kathleen O'Toole, News Service (650) 725-1939; e-mail: kathleen.otoole@stanford.edu

Internet grading service reduces tedium for teachers, students

After dinner when he needs a pick-me-up, John Etchemendy often sneaks off to watch students submit their homework. Tapping into the Internet from his home computer, the professor of philosophy and author of logic textbooks and software reaches one of two Sun workstations named Grade Grinder. He can watch as the wannabe historians and lawyers taking logic from Professor John Justice at Randolph-Macon Woman's College and the tech majors taking logic from Professor Selmer Bringsjord at Rensselaer Polytechnic Institute (RPI) submit answers to their problem sets.

These students and others taking introductory logic courses around the country use the Internet to interact with Grade Grinder, a robotic teaching assistant that doesn't give them answers to problems but hints and reminders of principles they have previously encountered. The robot's advice is personalized to address the specific shortcomings of the last answer each student has submitted, and it is delivered by e-mail in seconds. That compares to the week or more that is typical of feedback from a human grader.

Grade Grinder is an Internet grading service that is provided with purchase of a new textbook, Language Proof and Logic, and four pieces of software. Etchemendy co-authored the package with the late Jon Barwise of the University of Indiana and formerly of Stanford, and a team of researchers at Stanford's Center for the Study of Language and Information (CSLI) and Indiana's Visual Inference Laboratory. Co-published last fall by CSLI Publications and a commercial textbook house, Seven Bridges Press, the textbook/software package for introductory logic is priced at $43.95, slightly less than most logic textbooks alone. There is one catch: Because the purchaser is buying lifetime tutoring help from Grade Grinder, each educational package comes with a unique registration number, and the student cannot resell that ID to another student. But the textbook covers more ground than most introductory courses, so the student can continue to receive tutoring from Grade Grinder years after taking a formal course.

"We are grading about half of the students' exercises live at a central point, so I get to watch their progress, and that is one of the most rewarding things," says Etchemendy, who has authored other textbooks and software but without this interactive component. (He also chaired the university's Commission on Technology in Teaching and Learning, which funded proposals for developing learning technologies on campus.) "First you'll see a student submit an answer that is wildly incorrect, then get some feedback from Grade Grinder and keep resubmitting until it's correct. As a textbook author, you don't usually get that chance to see how students learn from it, except your own."

When he finds a pattern of trouble, Etchemendy goes into Grade Grinder's Java-language software and tinkers. For example, he noticed a pattern last fall of students at California State University-Northridge having difficulties with word problems that took the form "neither . . . nor." Many of the students there acquired English as their second language, and the grammatical formulation confused them. Seeing the overall pattern allowed Etchemendy to improve Grade Grinder's advice both to the students and their professor.

Accidental origins

Etchemendy got into this business by accident, but it has turned out to fit well with his research on reasoning systems that use multiple forms of representation. He began building teaching software in the 1980s out of frustration with some of the mistakes students made in logic courses. Formal logic requires translating English sentences into a language that lacks the ambiguities and subtleties of natural languages. "As an example," he says, "I use the old 'Saturday Night Live' joke that goes, 'Every five minutes, a man is mugged in New York City. We're going to interview him tonight.'"

The joke is based on what logicians call a "quantifier scope ambiguity." "People don't even recognize that the English sentence has this ambiguity because we understand it correctly in context. If I said to you, 'Every five minutes, a man from the L.A. Times has been calling,' you would immediately interpret the sentence differently than you interpreted the one about the man being mugged. But when students try to learn an unambiguous language, they have problems because they lack this understanding of English ambiguities."

Grade Grinder uses sophisticated computer algorithms to check such things as the logical equivalence of the student's sentence with expected answers, and its truth or falsity in a large number of contexts. It can check files created using programs packaged with the textbook -- Tarski's World, Fitch and Boole. It performs this check much faster and with fewer errors than even an expert human logician, says Dave Barker-Plummer, a logician and senior research scientist at CSLI.

Grade Grinder most likely never will replace a human instructor, Barker-Plummer and Etchemendy say, but it can free instructors and students of their most tedious teaching and learning tasks. Instructors still grade about half of the homework. "This isn't classical distance education, because we think that no amount of technology can replace an instructor's interactions with students when they are trying to understand deeper conceptual issues," Barker-Plummer says.

The development team "stumbled on this minimally invasive approach," Etchemendy says, and "we also didn't see in advance that part of the value of Grade Grinder would be allowing us to centrally analyze common mistakes."

About a dozen of the professors who used Language Proof and Logic this year were polled by e-mail for this article. They gave it high marks, some saying it was a "revolutionary" development in the use of technology in the classroom. All who responded said the grading service made teaching logic easier on them and learning it easier on their students.

"The automated grader worked flawlessly and freed my TA to help with substantive issues in logic, rather than mechanical checking of proofs. It's like having another TA -- for free," said Bringsjord, director of the Minds and Machines Lab at RPI. "I believe this is the future. It's the start of tutoring agents that handle parts of teaching traditionally done by humans. My students also loved it."

Justice of the Department of Philosophy at Randolph-Macon agreed, adding, "With the Grade Grinder always available on the Internet, the student can know within seconds if she is doing the work correctly. What's more, she can correct her work before she asks the Grade Grinder to forward a report to the instructor. This instant feedback makes learning logic quicker and less frustrating for the student."

Not all the students in the classes of Professor Tom Burke at the University of South Carolina were pleased, however. "Some students hate the software precisely because one cannot indulge in shortcuts or sloppiness that pencil-and-paper homework easily permits," he said.

Burke said his only frustration with the program was "handling the massive amount of information I get via e-mail on students' progress through the homework. I wish I could get this information in a form that could be easily imported into a database."

The CSLI team is addressing that issue. "We're hoping to have web access to all the data for a particular instructor's course in a compact form by the end of summer," Etchemendy says.

Foiling cheaters, adapting to other courses

Instructors in the past also have been concerned that automated assessment systems might increase opportunities for student cheating, Barker-Plummer says. To address that concern, all Grade Grinder homework has a time stamp that makes it highly unlikely for students to share their homework. "We've made it difficult enough that any student who is savvy enough to circumvent the system will probably find it easier and less time consuming to do the work."

The Grade Grinder software could be adapted for use by other courses, Etchemendy and Barker-Plummer believe. "Any type of computer file that a computer can do something sensible to and give useful feedback could use the Grade Grinder framework," Etchemendy says. "All you have to do is write a single grading module for that type of file. Down the road, we could supply chemistry professors with a software framework in which to plug in their chemistry module, for example."

The grader would be especially useful, Barker-Plummer says, in scientific courses where the range of possible answers is so large that it is difficult to tell if a student has found a correct one. "I can imagine situations in chemistry where you are asked to write down a formula for a molecule, and there may be hundreds of ways to do it."

CSLI Publications

The logic course package was sought after by commercial publishers, but Etchemendy says he felt it would be "unfair if not immoral" to turn over the rights to a product that was developed with university resources. "Fortunately, CSLI has a press that publishes academic books, and we were able to have them publish it, through an arrangement with another press to do textbook marketing and distribution." About $2 from the sale of each textbook/software package is set aside to cover the expected operation and maintenance costs of the grader.

Dikran Karagueuzian, who directs CSLI Publications, says he views Language Proof and Logic as a breakthrough in educational technology. An academic press, CSLI Publications publishes 35 to 40 titles a year, many of them books in the cognitive sciences and all of which are reviewed by experts in their fields before acceptance (see http://csli-publications.stanford.edu/). Karagueuzian has rejected other courseware proposals because referees found their quality spotty, he says. "A lot of academic software is like a homemade motorcycle. You can't take it on the freeway. The programs can be used on the campus where they were born, but they can't pass the campus boundaries.

"In this case, we have a package that doesn't annoy the best students at the Harvards and Stanfords but which also doesn't alienate community college students. The authors used lots of concrete examples that don't go above the users' heads. The ideas sort of unfurl, and the really nice thing is the instant feedback."

-30-

By Kathleen O'Toole