With over 1,000,000 words in the English language, why is it that we tend to use the same words over & over? This blog shows a hierarchical approach to help you branch out and choose more descriptive words. But first, to get you into the mood for a blog about […]
Analytics claim this is the 20th most used word in English writing. What word, you might ask? This word. Which one? This one right here! You might think I'm trying to lead into an Abbott & Costello-style comedy routine, but I literally mean this word ... the word 'this'! As you can […]
The post Analytics claim this is the 20th most used word in English writing appeared first on SAS Learning Post.
~ This article is co-authored by Biljana Belamaric Wilsey and Teresa Jade, both of whom are linguists in SAS' Text Analytics R&D. When I learned to program in Python, I was reminded that you have to tell the computer everything explicitly; it does not understand the human world of nuance […]
The post Text analytics through linguists’ eyes: When is a period not a full stop? appeared first on The Text Frontier.
Similarly, in Text Analytics, the "Corpus" is the "huge switching station" that tells us the meaning of words and how to associate different forms of words to the items of interest that we are trying to extract from text.
The Wall Street Journal’s “Numbers Guy” -- Carl Bialik -- quoted Mike Calcagno, general manager of the Microsoft group that manages Word. Calgagno says "Text corpora is the lifeblood of most of our development and testing processes."
"Microsoft has licensed over one trillion words of English text in each of the past two years, and bolsters its collection with emails exchanged on its Hotmail program, with identifying details removed", according to a Microsoft spokeswoman.
SAS's own Enterprise Content Categorization maintains huge corpora in various languages. As Bialik notes in the Wall Street Journal article ("Making Every Word Count" , Sept. 12, 2008): "Without enough spoken-language data, subtleties may not emerge."
"The word 'rife' only occurs in negative contexts," says Anne O'Keeffe, a linguist at Mary Immaculate College, the University of Limerick, Ireland. "We are never rife with money," despite that affliction's appeal.
In spite of their utility, publicly-available Corpora are hard to come by and even harder to update.
The largest public collection may be the British National Corpus, which was assembled in the early 1990s. The BNC included the recorded conversations of 200 Britons. The intended American counterpart to the BNC --the American National Corpus -- is a collection of text that includes the 9/11 Commission Report and Berlitz travel guides. With only 22 million words, the ANC is small when compared to the BNC.
Copora and associated taxonomies are extremely valuable components of a robust text mining/text analytics solution. We are fortunate to have these assets available to us in support of our text mining/analytics tasks.