|
28 |
|
|
|
|
The model also assumes that a document can be about a word to some degree.
This implies that in general a document collection can be broken up into subsets; each subset being made up of documents that are about a given word to the same degree.
The fundamental hypothesis made now is that a content-bearing word is a word that distinguishes more than one class of documents with respect to the extent to which the topic referred to by the word is treated in the documents in each class.
It is precisely these words that are the candidates for index terms.
These content-bearing words can be mechanically detected by measuring the extent to which their distributions deviate from that expected under a Poisson process.
In this model the status of one of these content words within a subset of documents of the same 'aboutness' is one of non-content-bearing, that is, within the given subset it does not discriminate between further subsets.
Harter[31] has identified two assumptions, based upon which the above ideas can be used to provide a method of automatic indexing. The aim is to specify a rule that for any given document will assign it index terms selected from the list of candidates. The assumptions are:
(1) The probability that a document will be found relevant to a request for information on a subject is a function of the relative extent to which the topic is treated in the document. (2) The number of tokens in a document is a function* of the extent to which the subject referred to by the word is treated in the document.
In these assumptions a 'topic' is identified with the 'subject of the request' and with the 'subject referred to by the word'. Also, only single word requests are considered, although Bookstein and Kraft[35] in a more recent paper have attempted an extension to multi-word requests. The indexing rule based on these assumptions indexes a document with word w if and only if the probability of the document being judged relevant to a request for information on w exceeds some cost function. To calculate the required probability of relevance for a content-bearing word we need to postulate what its distribution would look like. We know that it cannot be a single Poisson distribution, and that it is intrinsic to a content-bearing word that it will distinguish between subsets of documents differing in the extent to which they treat the topic specified by the word. By assumption (2), within one of these subsets the distribution of a content-bearing can however be described by a Poisson process. Therefore, if there are only two such |
|
|
28 |
|
|
|