codingbooks
diff --git a/‎05-document-term-matrices.Rmd
Lines changed: 10 additions & 2 deletions b/‎05-document-term-matrices.Rmd
Lines changed: 10 additions & 2 deletions
diff --git a/‎06-topic-models.Rmd
Lines changed: 12 additions & 5 deletions b/‎06-topic-models.Rmd
Lines changed: 12 additions & 5 deletions
diff --git a/‎09-usenet.Rmd
Lines changed: 16 additions & 4 deletions b/‎09-usenet.Rmd
Lines changed: 16 additions & 4 deletions
diff --git a/‎images/flowchart.png
1.65 MB b/‎images/flowchart.png
1.65 MB
@@ -14,7 +14,13 @@ In the previous chapters, we've been analyzing text arranged in the tidy text fo
 
 However, most of the existing R tools for natural language processing, besides the tidytext package, aren't compatible with this format. The [CRAN Task View for Natural Language Processing](https://cran.r-project.org/web/views/NaturalLanguageProcessing.html) lists a large selection of packages that take other structures of input and provide non-tidy outputs. These packages are very useful in text mining applications, and many existing text datasets are structured according to these formats.
 
-Computer scientist Hal Abelson has observed that "No matter how complex and polished the individual operations are, it is often the quality of the glue that most directly determines the power of the system" [@Friedman:2008:EP
10000
L:1378240]. In that spirit, this chapter will discuss the "glue" that connects the tidy text format with other important packages and data structures, allowing you to rely on both existing text mining packages and the suite of tidy tools to perform your analysis. In particular, we'll examine the process of tidying document-term matrices, as well as casting a tidy data frame into a sparse matrix.
+Computer scientist Hal Abelson has observed that "No matter how complex and polished the individual operations are, it is often the quality of the glue that most directly determines the power of the system" [@Friedman:2008:EPL:1378240]. In that spirit, this chapter will discuss the "glue" that connects the tidy text format with other important packages and data structures, allowing you to rely on both existing text mining packages and the suite of tidy tools to perform your analysis.
+
+```{r flowchart, echo = FALSE, out.width = '100%', fig.cap = "A flowchart of a typical text analysis that combines tidytext with other tools and data formats. This chapter shows how to convert between document-term matrices and tidy data frames, as well as converting from a Corpus object to a text data frame. The topicmodels and mallet packages are explored in Chapter 6."}
+knitr::include_graphics("images/flowchart.png")
+```
+
+Figure \@ref(fig:flowchart) illustrates how an analysis can might switch between tidy and non-tidy data structures and tools, a process that will be explored in these next two chapters. This chapter will focus on the process of tidying document-term matrices, as well as casting a tidy data frame into a sparse matrix. We'll also expore how to tidy Corpus objects into text data frames, including a case study of ingesting and analyzing financial articles.
 
 ## Tidying a document-term matrix {#tidy-dtm}
 
@@ -31,6 +37,8 @@ DTM objects cannot be used directly with tidy tools, just as tidy data frames ca
 * `tidy()` turns a document-term matrix into a tidy data frame. This verb comes from the broom package [@R-broom], which provides similar tidying functions for many statistical models and objects.
 * `cast()` turns a tidy one-term-per-row data frame into a matrix. tidytext provides three variations of this verb, each converting to a different type of matrix: `cast_sparse()` (converting to a sparse matrix from the Matrix package), `cast_dtm()` (converting to a `DocumentTermMatrix` object from tm), and `cast_dfm()` (converting to a `dfm` object from quanteda).
 
+As shown in Figure \@ref(fig:flowchart), a DTM is typically comparable to a tidy data frame after a `count` or a `group_by`/`summarize` that contains counts or another statistic for each combination of a term and document.
+
 ### Tidying DocumentTermMatrix objects
 
 Perhaps the most widely used implementation of DTMs in R is the `DocumentTermMatrix` class in the tm package. Many available text mining datasets are provided in this format. For example, consider the collection of Associated Press newspaper articles included in the topicmodels package.
@@ -353,7 +361,7 @@ stock_tokens %>%
   labs(y = "Frequency of word * AFINN score")
 ```
 
-In the context of these financial articles, there are a few big red flags here. The words "share" and "shares" are counted as positive verbs by the AFINN lexicon ("Alice will **share** her cake with Bob"), but they're actually neutral nouns ("The stock price is $12 per **share**") that could just as easily be in a positive sentence as a negative one. The word "fool" is even more deceptive: it refers to Motley Fool, a financial services company. In short, we can see that the AFINN sentiment lexicon is entirely unsuited to the context of financial data (as are the NRC and Bing).
+In the context of these financial articles, there are a few big red flags here. The words "share" and "shares" are counted as positive verbs by the AFINN lexicon ("Alice will **share** her cake with Bob"), but they're actually neutral nouns ("The stock price is $12 per **share**") that could just as easily be in a positive sentence as a negative one. The word "fool" is even more deceptive: it refers to Motley Fool, a financial services company. In short, we can see that the AFINN sentiment lexicon is entirely unsuited to the context of financial data (as are the NRC and Bing lexicons).
 
 Instead, we introduce another sentiment lexicon: the Loughran and McDonald dictionary of financial sentiment terms [@loughran2011liability]. This dictionary was developed based on analyses of financial reports, and intentionally avoids words like "share" and "fool", as well as subtler terms like "liability" and "risk" that may not have a negative meaning in a financial context.
 
 
@@ -14,7 +14,7 @@ In text mining, we often have collections of documents, such as blog posts or ne
 
 Latent Dirichlet allocation (LDA) is a particularly popular method for fitting a topic model. It treats each document as a mixture of topics, and each topic as a mixture of words. This allows documents to "overlap" each other in terms of content, rather than being separated into discrete groups, in a way that mirrors typical use of natural language.
 
-We can use tidy text principles to approach topic modeling with the same set of tidy tools we've used throughout this book. In this chapter, we'll learn to tidy `LDA` objects from the [topicmodels package](https://cran.r-project.org/package=topicmodels) to examine model results using ggplot2 and dplyr. We'll also explore an example of clustering chapters from several books, where we can see that a topic model "learns" to tell the difference between the four books based on the text content.
+We can use tidy text principles to approach topic modeling with the same set of tidy tools we've used throughout this book. In this chapter, we'll learn to work with `LDA` objects from the [topicmodels package](https://cran.r-project.org/package=topicmodels). As shown in Figure \@ref(fig:flowchart) in the last chapter, the LDA package produces a model object that can be tidied with tidytext to make it compatible with ggplot2 and dplyr. We'll also explore an example of clustering chapters from several books, where we can see that a topic model "learns" to tell the difference between the four books based on the text content.
 
 ## Latent Dirichlet allocation
 
@@ -34,7 +34,13 @@ data("AssociatedPress")
 AssociatedPress
 ```
 
-We can use the `LDA()` function from the topicmodels package, setting `k = 2`, to create a two-topic LDA model. (Almost any topic model in practice will use a larger `k`, but we will see in Chapter \@ref(library-heist) that this analysis approach extends to a larger number of topics). This function returns an object containing the full details of the model fit, such as how words are associated with topics and how topics are associated with documents.
+We can use the `LDA()` function from the topicmodels package, setting `k = 2`, to create a two-topic LDA model.
+
+```{block, type = "rmdnote"}
+Almost any topic model in practice will use a larger `k`, but we will soon see that this analysis approach extends to a larger number of topics
+```
+
+This function returns an object containing the full details of the model fit, such as how words are associated with topics and how topics are associated with documents.
 
 ```{r ap_lda}
 # set a seed so that the output of the model is predictable
@@ -253,7 +259,7 @@ chapters_gamma
 
 Each of these values is an estimated proportion of words from that document that are generated from that topic. For example, the model estimates that each word in the `r chapters_gamma$document[1]` document has only a `r percent(chapters_gamma$gamma[1])` probability of coming from topic 1 (Pride and Prejudice).
 
-Now that we have these topic-probabilities, we can see how well our unsupervised learning did at distinguishing the four books. We'd expect that chapters within a book would be found to be mostly (or entirely), generated from the corresponding topic.
+Now that we have these topic probabilities, we can see how well our unsupervised learning did at distinguishing the four books. We'd expect that chapters within a book would be found to be mostly (or entirely), generated from the corresponding topic
8000
.
 
 First we re-separate the document name into title and chapter, after which we can visualize the per-document-per-topic probability for each (Figure \@ref(fig:chaptersldagamma)).
 
@@ -340,7 +346,7 @@ assignments %>%
        fill = "% of assignments")
 ```
 
-We notice that almost all the words for *Pride and Prejudice*, *Twenty Thousand Leagues Under the Sea*, and *War of the Worlds* were correctly assigned, while *Great Expectations* had a fair amount of misassigned words (which, as we saw above, led to two chapters getting misclassified).
+We notice that almost all the words for *Pride and Prejudice*, *Twenty Thousand Leagues Under the Sea*, and *War of the Worlds* were correctly assigned, while *Great Expectations* had a fair number of misassigned words (which, as we saw above, led to two chapters getting misclassified).
 
 What were the most commonly mistaken words?
 
@@ -392,6 +398,7 @@ collapsed <- by_chapter_word %>%
   group_by(document) %>%
   summarize(text = paste(word, collapse = " "))
 
+# create an empty file of "stopwords"
 file.create(empty_file <- tempfile())
 docs <- mallet.import(collapsed$document, collapsed$text, empty_file)
 
@@ -418,4 +425,4 @@ We could use ggplot2 to explore and visualize the model in the same way we did t
 
 ## Summary
 
-This chapter introduces topic modeling for finding clusters of words within documents, and shows how the `tidy()` verb lets us explore and understand these models using dplyr and ggplot2. This is one of the advantages of the tidy approach to model exploration; the challenges of different output formats are handled by the tidying functions, and we can explore model results using a standard set of tools. In particular, we examined 
+This chapter introduces topic modeling for finding clusters of words that characterize a set of documents, and shows how the `tidy()` verb lets us explore and understand these models using dplyr and ggplot2. This is one of the advantages of the tidy approach to model exploration: the challenges of different output formats are handled by the tidying functions, and we can explore model results using a standard set of tools. In particular, we saw that topic modeling is able to separate and distinguish chapters from four separate books, and explored the limitations of the model by finding words and chapters that it assigned incorrectly.
@@ -13,7 +13,11 @@ In our final chapter, we'll use what we've learned in this book to perform a sta
 
 ## Pre-processing
 
-We'll start by reading in all the messages from the `20news-bydate` folder, which are organized in sub-folders with one file for each message. We can read in files like these with a combination of `read_lines()`, `map()` and `unnest()`. Note that this step takes several minutes to read all the documents.
+We'll start by reading in all the messages from the `20news-bydate` folder, which are organized in sub-folders with one file for each message. We can read in files like these with a combination of `read_lines()`, `map()` and `unnest()`.
+
+```{block, type = "rmdwarning"}
+Note that this step may take several minutes to read all the documents.
+```
 
 ```{r libraries}
 library(dplyr)
@@ -80,7 +84,11 @@ cleaned_text <- raw_text %>%
   ungroup()
 ```
 
-Many lines also have nested text representing quotes from other users, typically starting with a line like "so-and-so writes..." These can be removed with a few regular expressions. (We also choose to manually remove two messages, `9704` and `9985` that contained a large amount of non-text content).
+Many lines also have nested text representing quotes from other users, typically starting with a line like "so-and-so writes..." These can be removed with a few regular expressions.
+
+```{block, type = "rmdnote"}
+We also choose to manually remove two messages, `9704` and `9985` that contained a large amount of non-text content.
+```
 
 ```{r cleaned_text2, dependson = "cleaned_text1"}
 cleaned_text <- cleaned_text %>%
@@ -145,7 +153,7 @@ tf_idf %>%
   coord_flip()
 ```
 
-We see lots of characteristic words specific to particular newsgroup, such as "wiring" and "circuit" on the sci.electronics topic and "orbit" and "lunar" for the space newsgroup. You could use this same code to explore other newsgroups yourself.
+We see lots of characteristic words specific to a particular newsgroup, such as "wiring" and "circuit" on the sci.electronics topic and "orbit" and "lunar" for the space newsgroup. You could use this same code to explore other newsgroups yourself.
 
 ```{r, dependson = "tf_idf", echo = FALSE, fig.width=9, fig.height=8, eval = FALSE, echo = FALSE}
 plot_tf_idf <- function(d) {
@@ -350,7 +358,11 @@ sentiment_messages <- usenet_words %>%
   filter(words >= 5)
 ```
 
-As a simple measure to reduce the role of randomness, we filtered out messages that had fewer than five words that contributed to sentiment. What were the most positive messages?
+```{block, type = "rmdnote"}
+As a simple measure to reduce the role of randomness, we filtered out messages that had fewer than five words that contributed to sentiment.
+```
+
+What were the most positive messages?
 
 ```{r dependson = "sentiment_messages"}
 sentiment_messages %>%