'Grade grinder' looks at
logic behind students' errors
BY KATHLEEN O'TOOLE
 After dinner when he needs a pick-me-up,
John Etchemendy often sneaks off to watch students submit
their homework. Tapping into the Internet from his home
computer, the professor of philosophy and author of logic
textbooks and software reaches one of two Sun
workstations named Grade Grinder. He can watch as the
wannabe historians and lawyers taking logic from
Professor John Justice at Randolph-Macon Woman's College
and the tech majors taking logic from Professor Selmer
Bringsjord at Rensselaer Polytechnic Institute (RPI)
submit answers to their problem sets.
These students and others
taking introductory logic courses around the country use
the Internet to interact with Grade Grinder, a robotic
teaching assistant that doesn't give them answers to
problems, but gives them hints and reminders of
principles they have previously encountered. The robot's
advice is personalized to address the specific
shortcomings of the last answer each student has
submitted, and it is delivered by e-mail in seconds. That
compares to the week or more that is typical of feedback
from a human grader.
Grade Grinder is an
Internet grading service that is provided with purchase
of a new textbook, Language Proof and Logic, and
four pieces of software. Etchemendy co-authored the
package with the late Jon Barwise of the University of
Indiana and formerly of Stanford, and a team of
researchers at Stanford's Center for the Study of
Language and Information (CSLI) and Indiana's Visual
Inference Laboratory. Co-published last fall by CSLI
Publications and a commercial textbook house, Seven
Bridges Press, the textbook/software package for
introductory logic is priced at $43.95, slightly less
than most logic textbooks alone. There is one catch:
Because the purchaser is buying lifetime tutoring help
from Grade Grinder, each educational package comes with a
unique registration number, and the student cannot resell
that ID to another student. But the textbook covers more
ground than most introductory courses, so the student can
continue to receive tutoring from Grade Grinder years
after taking a formal course.
"We are grading about
half of the students' exercises live at a central point,
so I get to watch their progress, and that is one of the
most rewarding things," says Etchemendy, who has
authored other textbooks and software but without this
interactive component. (He also chaired the university's
Commission on Technology in Teaching and Learning, which
funded proposals for developing learning technologies on
campus.) "First you'll see a student submit an
answer that is wildly incorrect, then get some feedback
from Grade Grinder and keep resubmitting until it's
correct. As a textbook author, you don't usually get that
chance to see how students learn from it, except your
own."
When he finds a pattern of
trouble, Etchemendy goes into Grade Grinder's
Java-language software and tinkers. For example, he
noticed a pattern last fall of students at California
State University-Northridge having difficulties with word
problems that took the form "neither . . .
nor." Many of the students there acquired English as
their second language, and the grammatical formulation
confused them. Seeing the overall pattern allowed
Etchemendy to improve Grade Grinder's advice both to the
students and their professor.
Accidental origins
Etchemendy got into this
business by accident, but it has turned out to fit well
with his research on reasoning systems that use multiple
forms of representation. He began building teaching
software in the 1980s out of frustration with some of the
mistakes students made in logic courses. Formal logic
requires translating English sentences into a language
that lacks the ambiguities and subtleties of natural
languages. "As an example," he says, "I
use the old 'Saturday Night Live' joke that goes, 'Every
five minutes, a man is mugged in New York City. We're
going to interview him tonight.'"
The joke is based on what
logicians call a "quantifier scope ambiguity."
"People don't even recognize that the English
sentence has this ambiguity because we understand it
correctly in context. If I said to you, 'Every five
minutes, a man from the L.A. Times has been
calling,' you would immediately interpret the sentence
differently than you interpreted the one about the man
being mugged. But when students try to learn an
unambiguous language, they have problems because they
lack this understanding of English ambiguities."
Grade Grinder uses
sophisticated computer algorithms to check such things as
the logical equivalence of the student's sentence with
expected answers, and its truth or falsity in a large
number of contexts. It can check files created using
programs packaged with the textbook -- Tarski's World,
Fitch and Boole. It performs this check much faster and
with fewer errors than even an expert human logician,
says Dave Barker-Plummer, a logician and senior research
scientist at CSLI.
Grade Grinder most likely
never will replace a human instructor, Barker-Plummer and
Etchemendy say, but it can free instructors and students
of their most tedious teaching and learning tasks.
Instructors still grade about half of the homework.
"This isn't classical distance education, because we
think that no amount of technology can replace an
instructor's interactions with students when they are
trying to understand deeper conceptual issues,"
Barker-Plummer says.
The development team
"stumbled on this minimally invasive approach,"
Etchemendy says, and "we also didn't see in advance
that part of the value of Grade Grinder would be allowing
us to centrally analyze common mistakes."
About a dozen of the
professors who used Language Proof and Logic this
year were polled by e-mail for this article. They gave it
high marks, some saying it was a
"revolutionary" development in the use of
technology in the classroom. All who responded said the
grading service made teaching logic easier on them and
learning it easier on their students.
"The automated grader
worked flawlessly and freed my TA to help with
substantive issues in logic, rather than mechanical
checking of proofs. It's like having another TA -- for
free," said Bringsjord, director of the Minds and
Machines Lab at RPI. "I believe this is the future.
It's the start of tutoring agents that handle parts of
teaching traditionally done by humans. My students also
loved it."
Justice of the Department
of Philosophy at Randolph-Macon agreed, adding,
"With the Grade Grinder always available on the
Internet, the student can know within seconds if she is
doing the work correctly. What's more, she can correct
her work before she asks the Grade Grinder to forward a
report to the instructor. This instant feedback makes
learning logic quicker and less frustrating for the
student."
Not all the students in
the classes of Professor Tom Burke at the University of
South Carolina were pleased, however. "Some students
hate the software precisely because one cannot indulge in
shortcuts or sloppiness that pencil-and-paper homework
easily permits," he said.
Burke said his only
frustration with the program was "handling the
massive amount of information I get via e-mail on
students' progress through the homework. I wish I could
get this information in a form that could be easily
imported into a database."
The CSLI team is
addressing that issue. "We're hoping to have web
access to all the data for a particular instructor's
course in a compact form by the end of summer,"
Etchemendy says.
Foiling cheaters,
adapting to other courses
Instructors in the past
also have been concerned that automated assessment
systems might increase opportunities for student
cheating, Barker-Plummer says. To address that concern,
all Grade Grinder homework has a time stamp that makes it
highly unlikely for students to share their homework.
"We've made it difficult enough that any student who
is savvy enough to circumvent the system will probably
find it easier and less time consuming to do the
work."
The Grade Grinder software
could be adapted for use by other courses, Etchemendy and
Barker-Plummer believe. "Any type of computer file
that a computer can do something sensible to and give
useful feedback could use the Grade Grinder
framework," Etchemendy says. "All you have to
do is write a single grading module for that type of
file. Down the road, we could supply chemistry professors
with a software framework in which to plug in their
chemistry module, for example."
The grader would be
especially useful, Barker-Plummer says, in scientific
courses where the range of possible answers is so large
that it is difficult to tell if a student has found a
correct one. "I can imagine situations in chemistry
where you are asked to write down a formula for a
molecule, and there may be hundreds of ways to do
it."
CSLI Publications
The logic course package
was sought after by commercial publishers, but Etchemendy
says he felt it would be "unfair if not
immoral" to turn over the rights to a product that
was developed with university resources.
"Fortunately, CSLI has a press that publishes
academic books, and we were able to have them publish it,
through an arrangement with another press to do textbook
marketing and distribution." About $2 from the sale
of each textbook/software package is set aside to cover
the expected operation and maintenance costs of the
grader.
Dikran Karagueuzian, who
directs CSLI Publications, says he views Language
Proof and Logic as a breakthrough in educational
technology. An academic press, CSLI Publications
publishes 35 to 40 titles a year, many of them books in
the cognitive sciences and all of which are reviewed by
experts in their fields before acceptance (see
http://csli-publications.stanford.edu/). Karagueuzian
has rejected other courseware proposals because referees
found their quality spotty, he says. "A lot of
academic software is like a homemade motorcycle. You
can't take it on the freeway. The programs can be used on
the campus where they were born, but they can't pass the
campus boundaries.
"In this case, we
have a package that doesn't annoy the best students at
the Harvards and Stanfords but which also doesn't
alienate community college students. The authors used
lots of concrete examples that don't go above the users'
heads. The ideas sort of unfurl, and the really nice
thing is the instant feedback." SR
|
