Paul A. Fortier
Department of french, Spanish, and Italian
University of Manitoba
Canada R3T 2N2
The only practical response is disambiguation by human informants. But the reliability of such a process is a concern. Singh (1986) and Fokkema (1988) discuss the matter in a theoretical and speculative mode without systematic empirical evidence. Nothing bearing directly on the topic can be found in recent issues of Computers and the Humanities or Literary and Linguistic Computing.
The correctness of a given choice by an individual informant or rater is generally unknowable. Usually, such a choice is a matter of degree and judgment within the cultural context of a language. The consistency of markup from one informant to another is a likely reasonable touchstone for assessing the reliability of the disambiguation process.
French thesauri identify seventy words (in the sense of lemmata) related to the concept of solitude. These strings were used to search the texts for words potentially related to solitude in the ARTFL database. Words centered in 60 characters of context were given to six informants with minimal instructions: mark the words, which from a reading of the context, do not evoke the concept of human solitude and go back to the ARTFL database for more context in doubtful cases.
Table 1 summarizes the number of allusions to human solitude. Note that the range of the number of counts of allusions to solitude varies widely according to the text examined.
To assess the impact of providing a greater amount of context to informants, 300 characters of context from the original text were obtained using the same set of strings as for 60 characters of context. Identical instructions were given to six informants--two of whom participated in the first project.
Table 2 summarizes the number of allusions to human solitude for a context of 300 characters. When a larger context is provided, the range of variability in the results increases. On the other hand, the consistency of the raters is slightly improved.
The measure of reliability of a sum of responses from a random sample of a set of a fixed number of raters drawn from a population of raters is known as the Spearman-Brown prophecy after Spearman (1910) and Brown (1910). Cronbach's alpha, due to Cronbach (1951), is an estimate of the Spearman-Brown prophecy.
Both the Spearman-Brown prophecy and Cronbach's alpha are increasing functions of the intraclass correlation--parameter and statistic, respectively--and each asymptotically equal 100% in the limit as the number of raters increases without bound despite the consideration that adding more raters only increases the chances that everybody will not agree.
Statements made regarding a population based on a sample must be qualified by a probability clause. The standard choice for a probability clause is such that the statement be correct in the long run, if the statistical procedure were to be done unboundedly many times, 95% of the time--or nineteen times out of twenty. So a measure of the degree of difficulty associated with a specific text in determining whether there is an allusion to solitude is given by the minimum number of informants to achieve a value of 95% for Cronbach's alpha nineteen times out of twenty.
The 95% confidence intervals for the intraclass correlation coefficient are given in Table 3 for both 60 character and 300 character groups surrounding each type. Note that for each novel the confidence interval for rater reliability is shifted to the left with the 300 character groups compared to the 60 character groups. With more context, the estimate of reliability becomes lower.
Note the lack of overlap for the 95% confidence intervals for the two sizes of character groups for each of the novels by Bernanos, Céline, and Proust. It is conjectured that as the sample size surrounding each type increases, there is more opportunity for subjectivity based on personal opinion.
Table 4 presents the number of informants required to achieve 95% for Cronbach's alpha nineteen times out of twenty for 60 and 300 character contexts. For each novel, as the size of context increases, the number of informants required to achieve 95% for Cronbach's alpha increases. Moreover, the size of a jury to decide whether a word alludes to human solitude with 300 characters of context ought to be no fewer than 45 based on the novels surveyed.
With respect to literary analysis, note that the novels of Camus have a greater spread in jury size than those of Gide.
These results suggest a high degree of subjectivity when a single individual scores the semantic content of literary data. The meaning of individual words in context is a matter of opinion, and cannot be taken as definitive until a high degree of consensus among a large number of raters or informants is achieved.
Scores for Human Solitude with 60 Characters of Context
Scores for Human Solitude with 300 Characters of Context
95% confidence interval: intraclass correlation coefficient
|Texts||60 Characters||300 characters|
Number of informants required for a value of 95% or more for
Cronbach's alpha nineteen times out of twenty.
|Texts||60 Characters||300 Characters|
Explanation of Symbols used in the Tables Texts:
BJC: Bernanos, Journal d'un curé de campagne
CET: Camus, L'Étranger
CCH: Camus, La Chute
CVN: Céline, Voyage au bout de la nuit
GIM: Gide, L'Immoraliste
GPE: Gide, La Porte étroite
MNV: Mauriac, Le Noeud de Vipères
PFU: Proust, La Fugitive
SNA: Sartre, La Nausée.
raw: raw solitude data as downloaded from the ARTFL database
s(1-7): French Literature Graduate Student
p(1-5): French Literature Professor