Keyword indexing is the problem of assigning keywords to text documents. It is an important task as keywords play crucial roles in several information retrieval tasks. The problem is also challenging as the number of text documents is increasing, and such documents come in different forms (i.e., scientific papers, online news articles, and microblog posts). This chapter provides an overview of keyword indexing and elaborates on keyword extraction techniques. The authors provide the general motivations behind the supervised and the unsupervised keyword extraction and enumerate several pioneering and state-of-the-art techniques. Feature engineering, evaluation metrics, and benchmark datasets used to evaluate the performance of keyword extraction systems are also discussed.
TopIntroduction
Keywords1 are salient words that best describe the content of a document. Keywords are particularly important as they provide readers with the concept of a document and have applications in document clustering (Kang, 2003; Christy, Gandhi, and Vaithyasubramanian, 2019), searching (Liu, Do, and Cao, 2020), meta-data enrichment (Al-Natsheh, Martinet, Muhlenbach, Rico, and Zighed, 2018), and summarizing (Choi, Kim, and Lee, 2020).
The number of online documents is increasing at a high rate and a few of them come with keywords. Turney (2002) cites that some 20% of full-text academic journals do not contain author-assigned keywords, which goes up to 30% for web pages. As manual keyword extraction of keywords is a time-consuming, error-prone, and expensive task, automatic keyword indexing has become an active research topic. Automatic indexing of documents with keywords can be achieved in two ways, either via keyword assignment or keyword extraction. Keyword assignment is concerned with indexing a document with keywords that are not necessarily present in the document. It requires external resources such as ontology and thesaurus (C. Zhang and Xu, 2009). This approach is cited as particularly useful for short texts such as microblog feeds, as they do not contain enough structural, linguistic, and statistical information (Singhal, Kasturi, Sharma, and Srivastava, 2017). The primary advantage of keyword assignment is that documents can be indexed with keywords that are not present in the text but better describe the content of the document than in-text words. Primary limitations of keyword assignment, on the other hand, include their dependence on external resources and being language-dependent. However, multilingual vocabularies such as EUROVOC and AGROVOC may help overcome the second limitation (Steinberger, 2001; Balaji et al., 2010).
Keyword extraction methods aim to index a document with words that are present in the document. Keyword extraction systems are classified based on several aspects. One classification is based on the learning algorithm they employ: supervised, unsupervised. Supervised keyword extraction can be considered a binary classification problem, where words in a document are assigned either into the keyword class or the non-keyword class (Witten, Paynter, Frank, Gutwin, and Nevill-Manning, 1999). Unsupervised keyword extraction methods use unsupervised learning methods for keyword extraction. Simple statistic approaches make use of n-gram statistics (Suleiman and Awajan, 2017), term frequency (Hulth, 2003), and term frequency-inverse document frequency (Sun, Wang, and Xia, 2017). Linguistic approaches incorporate NLP tools such as lexical analysis (Ercan and Cicekli, 2007), syntactic analysis (Hulth, 2003).
Keyword extraction systems also differ by the number of documents they work on. Some keyword extraction systems (Matsuo and Ishizuka, 2004; Campos et al., 2020) consider the content of a single document to extract keywords, while some others (C. Zhang and Xu, 2009; T. D. Nguyen and Luong, 2010) consider the content of multiple documents. Multi-document approaches benefit from a richer source of information, i.e., learn structures of documents to improve keyword extraction (T. D. Nguyen and Luong, 2010), verify single document-based findings against other documents in the corpus (Wartena, Brussee, and Slakhorst, 2010), and build domain knowledge (C. Zhang and Xu, 2009). However, building and annotating, in case of supervised learning, a corpus is a difficult task.