Page 148 Concepts and similar pages

Concepts

Similarity Concept
Retrieval effectiveness
Effectiveness
Measures of effectiveness
Measurement of effectiveness
Recall
Precision
Contingency table
Information measure
Probabilistic retrieval
Information retrieval definition

Similar pages

Similarity Page Snapshot
10 In the past there has been much debate about the validity of evaluations based on relevance judgments provided by erring human beings ...Effectiveness and efficiency Much of the research and development in information retrieval is aimed at improving the effectiveness and efficiency of retrieval ...
146 There has been much debate in the past as to whether precision and recall are in fact the appropriate quantities to use as measures of effectiveness ...1 the most commonly used pair;2 fairly well understood quantities ...The final question How to evaluate?has a large technical answer ...Before proceeding to the technical details relating to the measurement of effectiveness it is as well to examine more closely the concept of relevance which underlies it ...Relevance Relevance is a subjective notion ...
145 automatic and interactive retrieval system?Studies to gauge this are going on but results are hard to interpret ...It should be apparent now that in evaluating an information retrieval system we are mainly concerned with providing data so that users can make a decision as to 1 whether they want such a system social question and 2 whether it will be worth it ...The second question what to evaluate?boils down to what can we measure that will reflect the ability of the system to satisfy the user ...1 The coverage of the collection,that is,the extent to which the system includes relevant matter;2 the time lag,that is,the average interval between the time the search request is made and the time an answer is given;3 the form of presentation of the output;4 the effort involved on the part of the user in obtaining answers to his search requests;5 the recall of the system,that is,the proportion of relevant material actually retrieved in answer to a search request;6 the precision of the system,that is,the proportion of retrieved material that is actually relevant ...It is claimed that 1 4 are readily assessed ...
189 The time is ripe for another attempt at using natural language to represent documents inside a computer ...It has never been assumed that a retrieval system should attempt to understand the content of a document ...Such an approach would make feedback a major tool ...Future developments Much of the work in IR has suffered from the difficulty of comparing retrieval results ...
5 frequency of occurrence and co occurrence of index terms in the relevant and non relevant documents ...Chapter 7:Evaluation here I give a traditional view of the measurement of effectiveness followed by an explanation of some of the more promising attempts at improving the art ...Chapter 8:The Future contains some speculation about the future of IR and tries to pinpoint some areas of research where further work is desperately needed ...Information retrieval Since the 1940 s the problem of information storage and retrieval has attracted increasing attention ...In principle,information storage and retrieval is simple ...When high speed computers became available for non numerical work,many thought that a computer would be able to read an entire document collection to extract the relevant documents ...
181 A short paper by Good [44]which is in sympathy with the approach based on a theory of measurement given here,discusses the evaluation of retrieval systems in terms of expected utility ...One conspicuous omission from this chapter is any discussion of cost effectiveness ...References 1 ...2 ...3 ...4 ...5 ...6 ...7 ...8 ...26,321 343 1975 ...9 ...10 ...11 ...12 ...13 ...14 ...15 ...16 ...17 ...18 ...
188 In basing a theory of evaluation on the theory of measurement,is it possible to devise a measure of effectiveness not starting with precision and recall but simply with the set of relevant documents and the set of retrieved documents?If so,can we generalise such a measure to take account of degree of relevance?An alternative derivation of an E type measure could be done in terms of recall and fallout ...Up to now the measurement of effectiveness has proved fairly intractable to statistical analysis ...I think the Robertson model described in Chapter 7 goes some way to being considered as a reasonable statistical model ...There may be laws of retrieval such as the well known trade off between precision and recall that are worth establishing either empirically or by theoretical argument ...6 ...There is a need for more intensive research into the problems of what to use to represent the content of documents in a computer ...Information retrieval systems,both operational and experimental,have been keyword based ...The major reason for this rather simple minded approach to document retrieval is a very good one ...
147 now have the situation where a number of questions exist for which the correct responses are known ...There is a concept of relevance which can be said to be objective and which deserves mention as an interesting source of speculation ...Logical relevance is most easily explicated if the questions are restricted to the yes no type ...A stored sentence is logically relevant to a representation of an information need if and only if it is a member of some minimal premiss set of stored sentences for some component statement of that need ...Although logical relevance is initially only defined between sentences it can easily be extended to apply to stored documents ...
180 effectiveness can be calculated to infinite precision we may be insisting on a difference when in fact it only occurs in the tenth decimal place ...Finally,although I have just explained the use of the sign test in terms of single number measures,it is also used to detect a significant difference between precision recall graphs ...Bibliographic remarks Quite a number of references to the work on evaluation have already been given in the main body of the chapter ...Buried in the report by Keen Digger [32]Chapter 16 is an excellent discussion of the desirable properties of any measure of effectiveness ...A parameter which I have mentioned in passing but which deserves closer study in generality ...The trade off between precision and recall has for a long time been the subject of debate ...Guazzo [39]describe an approach to the measurement of retrieval effectiveness based on information theory ...The notion of relevance has at all times attracted much discussion ...