TOWARD A THEORY OF CRITIQUING HUMAN ERRORS IN COMPUTER PROGRAMMING

Toufic  M. MEZHER

The purpose of this research is to pose and test various theories of expert critics in computer programming. The goal is to find out how a computer critic can best impact programmers’ behavior during programming tasks.

Three empirical studies were conducted. The first study consisted of a control group. Its purpose was to find syntax errors that programmers make when programming. To this end, subjects were given a programming task. Thirteen different kinds of syntax errors were made by the subjects; errors per subject ranging between one and eleven. These results revealed not only different kinds of syntax errors, but those commons to most subjects.

In the second study, subjects were given the same programming task, but this time had the assistance of a critic. The purpose of this critic was to prevent, detect and correct syntax errors during programming tasks. Several dozen critic networks were implemented based on the following critiquing theory: (1) timing of the critic was before, during and after task, (2) process of the critic was incremental, (3) mode of the critic was active, (4) knowledge in the critic was both deep and shallow, (5) heuristics algorithms were used, (6) interface of the critic was direct manipulation driven, and (7) strategies used in the critic were influencers, directors, and debiasers. The results of the second study showed that critics reduced the programmers' syntax errors by 85%. The only drawback was that the critic slowed programmers down. Therefore, experienced programmers disliked using the critic, although novice programmers found it helpful during programming tasks.

The third study was conducted to evaluate both the effectiveness of the critic's textual information and its usability. To evaluate effectiveness of the textual information, the critic only signaled users if an error occurred, but did not provide the source of the error. These results showed that subjects performed much better with textual information. In addition, a survey questionnaire designed to evaluate the critic's usability and effectiveness and the best critiquing approach and strategies in computer programming was completed by subjects. This questionnaire also evaluated the software covering keystrokes, clarity, and sequence of operations of the critic. These results showed: (1) Programmers like to be critiqued during and after the task. They like to be critiqued before the task only to receive a warning about consequences of their next possible action. Programmers like to be critiqued before, during, and after the task when learning a new f unction/ language. Programmers whose skills place them between "proficient" and "expert" don't like to be critiqued at all because they make few errors. (2) Some programmers choose incremental critics, others choose batch. (3) Some programmers prefer the critic to be active, others choose passive. Regarding critiquing strategies, the questionnaire has shown that when subjects first started learning how to program in COPE, they liked all three strategies (influencers, directors, and debiasers. After the subjects became more familiar with the computer language, they only liked to see directors and debiasers. They prefer influencers only when they deal with cautions regarding their next action or when they are learning new functions/language. Proficient and expert programmers don't like using the critic.