A Data-Intensive Approach to Named Entity Recognition Combining Contextual and Intrinsic Indicators

Introduction

Innovations in information technology have increased the amount and variety of information accessible by organizations over the past decades. Consumer opinions in online forums and behaviors in online social networking websites are examples of sources of valuable information for businesses intelligence. The use of births, marriages and deaths information from public announcements to update proprietary business databases and specific marketing campaigns are specific examples of how such information can be applied in a business organization. Although effectively analyzing this wide array of information sources can provide businesses competitive advantage in a consumer-centric market environment (Godbole & Roy, 2008), the existence of over 80% of the available information in unstructured form¹ (Shilakes & Tylman, 1998) and ineffectiveness of most of the existing automated decision making and business intelligence tools (which are primarily designed for structured data) in analyzing unstructured textual information (UTI) is a major challenge (Raghuveer, Jindal, Mokbel, Debnath, & Du, 2007). Several natural language processing (NLP) (Klein & Simmons, 1963) and text mining methods (Church, 1989) have been researched towards solving this problem. Similar to (Wu, et al., 2007), our approach identifies entities as well as their roles within UTI in a process known as semantic named entity recognition (NER) and applies this natural ontology in creating structure for the text. By transforming UTI to structured information, existing business analytics and intelligence extraction processes can be used for processing the information.

Our approach to NER is a language and domain independent data-intensive method. This approach neither utilizes explicit grammar rules nor relies on external resources such as dictionaries and gazettes, making it seamlessly adaptable across different business groups. It derives all the information needed from annotated example documents contained in a knowledgebase. This data-intensive NER approach takes advantage of the ever increasing amount of information available (Cisco Systems, 2007) and the significantly lower cost of high-performance computing resources (Kurzweil, 2005). These two conditions make the capacity to rapidly process this huge volume of information available to businesses with limited resources. Consequently, some difficult problems that were previously solved using complex algorithmic and statistical models are now being approached by more data-intensive solutions that leverages fast computational speeds to extract prior known solutions/information contained in a vast corpora of relevant information (Anderson, 2008). Pioneering this solutions approach are organizations like Google, Yahoo!, Bing and AOL that have access to massive amounts of data. For example, instead of using dictionaries to suggest spelling corrections to search terms, they now use huge amount of information from previous searches, Google translates languages by finding instances of past translations of the same words and phrases (Anderson, 2008). Furthermore, research such as map-reduce by Google (Yang, Dasdan, Hsiao, & Parker, 2007) provides an effective way of utilizing huge volumes of data. This trend towards data-intensive solution can be captured using Peter Norvig, Google’s Research Director’s statement that “all models are wrong, and increasingly you can succeed without them” (Anderson, 2008).

We describe a named entity recognition (NER) approach in which the opportunity for improvement in performance increases along with the volume of information from which knowledge can be extracted. We also discussed its strength and weaknesses using experiments based on a relatively limited knowledgebase.