| Similarity |
Page |
Snapshot |
| 126 |
In general we would have two tables of this kind when setting up our function g x,one for estimating the parameters associated with P x w 1 and one for P x w 2
...The estimates shown above are examples of point estimates
...Two basic assumptions made in deriving any estimation rule through Bayesian decision theory are:1 the form of the prior distribution on the parameter space,i
...probability distribution on the possible values of the binomial parameter;and 2 the form of the loss function used to measure the error made in estimating the parameter
...Once these two assumptions are made explicit by defining the form of the distribution and loss function,then,together with Bayes Principle which seeks to minimise the posterior conditional expected loss given the observations,we can derive a number of different estimation rules
...where x is the number of successes in n trials,and a and b are parameters dictated by the particularcombination of prior and loss |
| 114 |
the system to its user will be the best that is obtainable on the basis of those data
...Of course this principle raises many questions as to the acceptability of the assumptions
...The probability ranking principle assumes that we can calculate P relevance document,not only that,it assumes that we can do it accurately
...So returning now to the immediate problem which is to calculate,or estimate,P relevance document
... |
| 133 |
3
...It must be emphasised that in the non linear case the estimation of the parameters for g x will ideally involve a different MST for each of P x w 1 and P x w 2
...There is a choice of how one would implement the model for g x depending on whether one is interested in setting the cut off a prior or a posteriori
...If one assumes that the cut off is set a posteriori then we can rank the documents according to P w 1 x and leave the user to decide when he has seen enough
...to calculate estimate the probability of relevance for each document x
... |
| 129 |
we work with the ratio In the latter case we do not see the retrieval problem as one of discriminating between relevant and non relevant documents,instead we merely wish to compute the P relevance x for each document x and present the user with documents in decreasing order of this probability
...The decision rules derived above are couched in terms of P x wi
...I will now proceed to discuss ways of using this probabilistic model of retrieval and at the same time discuss some of the practical problems that arise
...The curse of dimensionality In deriving the decision rules I assumed that a document is represented by an n dimensional vector where n is the size of the index term vocabulary
... |
| 128 |
objected to on the same grounds that one might object to the probability of Newton s Second Law of Motion being the case
...To approach the problem in this way would be useless unless one believed that for many index terms the distribution over the relevant documents is different from that over the non relevant documents
...The elaboration in terms of ranking rather than just discrimination is trivial:the cut off set by the constant in g x is gradually relaxed thereby increasing the number of documents retrieved or assigned to the relevant category
...If one is prepared to let the user set the cut off after retrieval has taken place then the need for a theory about cut off disappears
... |
| 113 |
any given document whether it is relevant or non relevant
...PQ relevance document where the Q is meant to emphasise that it is for a specific query
...P relevance document
...Let us now assume following Robertson [7]that:1 The relevance of a document to a request is independent of other documents in the collection
...With this assumption we can now state a principle,in terms of probability of relevance,which shows that probabilistic information can be used in an optimal manner in retrieval
...The probability ranking principle
... |
| 119 |
and The importance of writing it this way,apart from its simplicity,is that for each document x to calculate g x we simply add the coefficients ci for those index terms that are present,i
...The constant C which has been assumed the same for all documents x will of course vary from query to query,but it can be interpreted as the cut off applied to the retrieval function
...Let us now turn to the other part of g x,namely ci and let us try and interpret it in terms of the conventional contingency table
...There will be one such table for each index term;I have shown it for the index term i although the subscript i has not been used in the cells
...This is in fact the weighting formula F 4 used by Robertson and Sparck Jones 1 in their so called retrospective experiments
... |
| 29 |
subsets differing in the extent to which they are about a word w then the distribution of w can be described by a mixture of two Poisson distributions
...here p 1 is the probability of a random document belonging to one of the subsets and x 1 and x 2 are the mean occurrences in the two classes
...Although Harter [31]uses function in his wording of this assumption,I think measure would have been more appropriate
...assumption 1 we can calculate the probability of relevance for any document from one of these classes
...that is used to make the decision whether to assign an index term w that occurs k times in a document
...Finally,although tests have shown that this model assigns sensible index terms,it has not been tested from the point of view of its effectiveness in retrieval
...Discrimination and or representation There are two conflicting ways of looking at the problem of characterising documents for retrieval
... |
| 112 |
of presenting the basic theory;I have chosen to present it in such a way that connections with other fields such as pattern recognition are easily made
...The fundamental mathematical tool for this chapter is Bayes Theorem:most of the equations derive directly from it
...This was recognised by Maron in his The Logic Behind a Probabilistic Interpretation as early as 1964 [4]...Remember that the basic instrument we have for trying to separate the relevant from the non relevant documents is a matching function,whether it be that we are in a clustered environment or an unstructured one
...It will be assumed in the sequel that the documents are described by binary state attributes,that is,absence or presence of index terms
...Estimation or calculation of relevance When we search a document collection,we attempt to retrieve relevant documents without retrieving non relevant ones
... |
