頁籤選單縮合
題 名 | Extension of Zipf's Law to Word and Character N-grams for English and Chinese |
---|---|
作 者 | Ha,Le Quan; Sicilia-Garcia,E. I.; Ming,Ji; Smith,F. J.; | 書刊名 | International Journal of Computational Linguistics & Chinese Language Processing |
卷 期 | 8:1 2003.02[民92.02] |
頁 次 | 頁77-101 |
分類號 | 312.23 |
關鍵詞 | Zipf's law; Chinese character; Chinese compound word; n-grams; Phrases; |
語 文 | 英文(English) |
英文摘要 | It is shown that for a large corpus, Zipf 's law for both words in English and characters in Chinese does not hold for all ranks. The frequency falls below the frequency predicted by Zipf's law for English words for rank greater than about 5,000 and for Chinese characters for rank greater than about 1,000. However, when single words or characters are combined together with n-gram words or characters in one list and put in order of frequency, the frequency of tokens in the combined list follows Zipf’s law approximately with the slope close to -1 on a loglog plot for all n-grams, down to the lowest frequencies in both languages. This behaviour is also found for English 2-byte and 3-byte word fragments. It only happens when all n-grams are used, including semantically incomplete n-grams. Previous theories do not predict this behaviour, possibly because conditional probabilities of tokens have not been properly represented. |
本系統中英文摘要資訊取自各篇刊載內容。