The exact rank-frequency function and size-frequency function of N-grams and N-word phrases with applications

N-grams are generalized words consisting of N consecutive symbols (letters), as they are used in a text. N-word phrases are general concepts consisting of N consecutive words, also as used in a text. Given the rank-frequency function of single letters (i.e., one-grams) or of single words (i.e., one-word phrases) being Zipfian, we determine in this paper, the exact rank-frequency function (i.e., the occurrence of N-grams or N-word phrases on each rank) and size-frequency distribution (i.e., the density of N-grams or N-word phrases on each occurrence density) of these N-grams and N-word phrases. This paper distinguishes itself from other ones on this topic by allowing no approximations in the calculations. This leads to an intricate rank-frequency function for N-grams and N-word phrases (as we knew before from unpublished calculations) but leads surprisingly, to a very simple size-frequency function f(N) for N-grams or N-word phrases of the form fN(j) = F/j(1+1/beta) ln(N-1) (G/j), where the Zipflan distribution of single letters or words is proportional to 1/r(beta). The paper closes with the calculation of type/token averages AN and type/token-taken averages mu(N)*, for N-grams and N-word phrases, where we also verify the theoretically proved result mu(N)* >= mu(N) but where we also give estimates for the differences mu(N)* - mu(N). (c) 2005 Elsevier Ltd. All rights reserved.