Search the other textbookusing the following terms: Zipfian distribution; Distribution joint probability;
Distribution prior;
Distribution stationary;
Distribution Zipflan;
Joint probability distribution;
Prior distribution;
Pages related to Zipfian distributionThese pages belong to the same textbook
| 1 | 81 | most modern applications (i.e., with the huge disk volumes now common), NDoc >> Vocab. This is one of the most important ways in which experimental collections (including AIT) differ from real corpora. A useful indexing vocabulary can be expected to be of a relatively constant size, Vocab =~ 10^3 to 10^5, while corpora sizes are likely to vary dramatically, NDoc =~ 10^4 to 10^9.
Along similar lines, it is always useful to think about what this means in the context of the WWW, where the notion of a closed corpus disappears. The WWW is an organic, constantly and growing |
| 2 | 62 | In fact, many influential thinkers have looked at such patterns among symbols. Going back to some of our most ancient writings suggests that statistical analyses of the original Hebrew characters and their positions within the two-dimensional array of the page reveals new codes.
Donald Knuth, one of computer science's most reknowned theoreticians, has analyzed an apparently random verse (Chapter 3, verse 16) from 59 of the Bible's books and used these as the basis of stratified sampling of the approximately 30000 Biblical verses. He found, for example, that the 3:16 verses were particulary richin occurrances of YHWH , the |
| 3 | 67 | is immediately replaced by a private one through a string of nerve impulses.... This recorded message presumably uses fewer signs than the incoming one; therefore when a given message reaches a higher level it will have been reduced to a choice between a few possibilities only without the extreme redundancy of the sounds. The last stages are idea stages, where not only the public representation has been lost, but also the public elements of information.
Mandelbrot makes other provocative suggestions, for example that schizophrenics provide the best test of his theory since these individuals impose fewest semantic constraints on |
| 4 | 294 | assumed that queries were fairly rich, structured expressions. At least at the moment, these assumptions do not seem to hold for most Web searching.
But despite the relatively simple form of most queries, the third interesting fact is that Web queries are rarely repeated. Even folding case and ignoring word order, only one third of queries appeared more than once in the billion queries; only 14% occurred more than three times. These statistics are especially significant in the face of new services such as AskJeeves which focus on providing especially relevant answers for a restricted set of anticipated |
| 5 | 80 | Figure 3.5: Indexing Graph
number of occurrences varies dramatically from one keyword to another. Once we make an assumption about how keywords occur within separate documents, we can derive the distribution of keywords across documents. But the distribution of keywords assigned to documents can be expected to be much more uniform - documents are about a nearly unform or constant number of topics. Figure 3.5 represents the index as a graph, where edges connect keyword nodes on the left with document nodes on the right. The Index graph is a bipartite graph, with its nodes divided into two subsets (keywords |
| 6 | 57 | Figure 2.7 Quoted Lines in an Email Message
how large a number this must be, whether your machine/compiler efficiently supports integers this large (or whether you are better off keeping the two numbers separate) will vary considerably. For this reason it makes good sense to isolate these issues in a separate routine.
Dependencies on document type
The process of indexing has been idealized, as having a first stage where we worry about what kind of document it is (e.g., whether it's a thesis or an email message), and then assuming subsequent processing is completely independent of document |
| 7 | 85 | low-frequency terms that are likely to be of particular importance in identifying relevant material. This is because the number of documents relevant to a query is generally small, and thus any frequently occurring terms must necessarily occur in many irrelevant documents; infrequently occurring terms have a greater probability of occurring in relevant documents --- and should thus be considered as being of greater potential when searching a database.
Rather than looking at the raw occurrence frequencies, we will aggregate occurrences within any document and consider only the number of documents} in which a keyword occurs. IDF proposes, again using |
| 8 | 274 | Figure 7.7: Covering Algorithms
which provide the most information gain . Finally, Cohen adapted these rule learning techniques to the text domain by adding set valued attributes. These special attributes collapse a document's representation to be simply the set of words it contains. Ripper's rules can then include tests for sets of words, rather than having to test the presence/absence of each word individually.
When irrelevant attributes abound
In documents where irrelevant attributes abound' (e.g., when any one document contains a small fraction of the full vocabulary but still more than are important in a classifier) this |
| 9 | 126 | Table 4.2: Tho Hypothetical Retrievals
retrieval in Rank order, and plotting each and every point in this fashion gives the Re/Pre curve shown in Figure 4.10.
At this point we can already make several observations. Asymptotically, we know that the final recall must go to one; once we have retrieved every document we've also retrieved every relevant document. The precision will be the ratio of the number of relevant documents to the total corpus size. Ordinarily, unless we are interested in very general queries, and/or very small sets of documents, this ratio will be very close to zero.
|
| 10 | 9 | used to count words. Even earlier, the related discipline of Library Science had developed many automated techniques for efficiently storing, cataloging and retrieving the physical materials so that browsing patrons could find them; many of these methods can be applied to the digital documents held within computers. IR has also borrowed heavily from the field of linguistics, especially computational linguistics.
The primary journals in the field and most important conferences Processing Management, the ACM's Transactions on Information Systems and the Journal of the American Society for Information Science (JASIS) are some of the central journals; meetings of the American |
|