The experiments performed consisted of comparisons of the classification accuracies obtained by the three approaches described earlier. The same data was used in all experiments. Three medical domain datasets (obtained from Ljubljana) -Lymphography, Breast Cancer and Primary Tumour were used in these comparisons. Each of the datasets was divided in two subsets, one for learning and other for testing (70% for learning and 30% for testing). The three classification strategies were tried using the same learned theory. Table 1 presents the average of ten repetitions of these experiments (standard deviations are between brackets). In order to examine the differences, t-tests with a 95% confidence level were performed. The values which represented a significant difference are in italics on the table.
Table 1. Results of experiments.
MYCIN-like PROSPECTOR-like Best Quality
Lymphography 78% (5%) 63% (9%) 81% (3%)
Breast Cancer 67% (6%) 78% (3%) 77% (4%)
Primary Tumour 23% (4%) 33% (6%) 32% (7%)
The results of table 1 were quite surprising. The best rule strategy was expected to be the worst since it does not take into account combinations of opinions. MYCIN's certainty factors performed worse than the others, with the exception of Lymphography dataset. PROSPECTOR's approach performed quite badly on Lymphography dataset. Both Breast Cancer and Primary Tumour datasets are known to be rather noisy. In this context the results suggest that the degree of uncertainty of the dataset counteracts in some way the advantages of combination of rules (at least for PROSPECTOR's approach as MYCIN's approach is always bad).
These differences show that the classification strategy can significantly affect the accuracy obtained by learning systems. These experiments seem to indicate that the best rule strategy can be a good strategy especially if we take in to account its simplicity when compared to other methods.