Before beginning there are three important preliminary definitions one will need in any text analysis:
Document: a distinct text object that one wishes to analyze. This could be a paper, a paragraph, etc.
Term: an individual word in a given document.
Corpus: The set of all documents.
Topic Models
Topic Modeling is a statistical model that attempts to cluster the words found in a document into various “topics”. The hope is that these topics would capture some underlying subject contained within the text. It is important to note that Topic Models are fuzzy in the sense that documents are assumed to be comprised of multiple topics with varying degrees of membership to each.
Latent Dirichlet Allocation
Perhaps the most famous topic model is Latent Dirichlet Allocation (LDA) (Blei et al., 2003). It is a three-level hierarchical Bayesian model that defines a collection of documents as a random mixture over some underlying topic set where each topic is itself a distribution over our vocabulary.
LDA works by assuming a data generating process (DGP) for our documents and then employs inference to discover the most likely parameters (Grimmer et al., 2022).
Blei, Ng, and Jordan define the following DGP (2003):
Choose \(N \sim\) Poisson(\(\xi\))
Choose \(\theta \sim\) Dir(\(\alpha\))
For each of the \(N\) words \(w_n\):
Choose a topic \(z_n\sim\)Multinomial(\(\theta\))
Choose a word \(w_n\) from \(p(w_n|z_n,\beta)\), a multinomial probability conditioned on the topic \(z_n\)
For the following analysis we estimate the parameters using the collapsed Gibbs sampling method, though it is important to note there are others that yield varied results. To further complicate things, when fitting LDA in R one must predefine the number of topics for the model. Finding a good estimate for the number of topics is paramount and many methods are explored.
Once a model is fitted, it is common to extract 2 matrices \(\Phi\) and \(\Theta\). These are also commonly referred to as \(\beta\) and \(\gamma\) respectively. Both are given by the priors of LDA. \(\Phi\) represents the probability mass over the terms. \(\Theta\) represents the probability mass function over the topics.
Feature Extraction and Preprocessing
Document-Term Matrices
There are many methods of feature extraction from text; we opt for the bag of words model. To construct this model simply define a common set of words shared between documents and store a count of word appearances. Commonly this is stored as a Document-Term Matrix (DTM), for example, given the documents:
Corpus
Document 1
“I am happy”
Document 2
“I am sad”
the corresponding DTM would be
I
am
happy
sad
Doc 1
1
1
1
0
Doc 2
1
1
0
1
In R we can us the tm package to create our Corpus and DTM objects. For these examples scrapped Data Science course information from various universities will be preprocessed.
doc_id
1 Computer Science 2210A/B
2 Computer Science 2211A/B
3 Computer Science 2212A/B/Y
4 Computer Science 2214A/B
5 Computer Science 3319A/B
6 Computer Science 3340A/B
text
1 Lists, stacks, queues, priority queues, trees, graphs, and their associated algorithms; file structures; sorting, searching, and hashing techniques; time and space complexity.
2 An introduction to software tools and systems programming. Topics include: understanding how programs execute (compilation, linking and loading); an introduction to a complex operating system (UNIX); scripting languages; the C programming language; system calls; memory management; libraries; multi-component program organization and builds; version control; debuggers and profilers.
3 A team project course that provides practical experience in the software engineering field. Introduction to the structure and unique characteristics of large software systems, and concepts and techniques in the design, management and implementation of large software systems.
4 This course presents an introduction to the mathematical foundations of computer science, with an emphasis on mathematical reasoning, combinatorial analysis, discrete structures, applications and modeling, and algorithmic thinking. Topics include sets, functions, relations, algorithms, number theory, matrices, mathematical reasoning, counting, graphs and trees.
5 A study of relational databases. Theoretical concepts will be covered, including relational algebra and relational calculus. Commercially available database systems will be used to demonstrate concepts such as Structured-Query-Language (SQL), writing code to connect and query a database, query optimization, Atomicity-Consistency-Isolation-Durability (ACID) concepts, and database design.
6 Upper and lower time and space bounds; levels of intractability; graph algorithms; greedy algorithms; dynamic algorithms; exhaustive search techniques; parallel algorithms.
<<SimpleCorpus>>
Metadata: corpus specific: 1, document level (indexed): 0
Content: documents: 1
Computer Science 2210A/B
Lists, stacks, queues, priority queues, trees, graphs, and their associated algorithms; file structures; sorting, searching, and hashing techniques; time and space complexity.
As shown the first element of the corpus is the course Computer Science 2210A/B and it holds a course description.
DTM Construction
Text Preprocessing
Raw text data lends itself to difficult analysis. NLP posses several best practices for cleaning text:
Punctation/Number Removal: deleting any non alphabetic characters.
Lowercasing: sending all words to lowercase.
Stopword Removal: stopwords are words used often in sentences that give little to no information, e.g., articles such as the, a, etc.
Tokenizing: dividing a document into a set of individual words, note we opt to combine synonymous tokens in our analysis as well
Note
TF-IDF and stemming are also very popular techniques. We found for our analysis these both lead to cumbersome topic interpretation, and we therefore omit them, though they should both be attempted in any complete analysis.
<<SimpleCorpus>>
Metadata: corpus specific: 1, document level (indexed): 0
Content: documents: 1
Computer Science 2210A/B
lists queues priority queues trees graphs associated algorithms structures sorting searching hashing time space complexity
Collocations
Collocations are terms that have a higher probability of appearing together adjacently than they do individually. Rather than just have a model of singular terms, unigrams, we can incorporate the most likely paired terms as their own token— bigrams. Lau et al. showed including Bigrams can signifigcantly improve topic intelligibility and leads to more parsimonious models (2013).
collocation count count_nested length lambda z
1 math math 31 0 2 7.510804 13.64619
2 data science 27 0 2 3.062254 11.75981
3 time series 9 0 2 5.838259 11.02856
4 differential equations 10 0 2 6.696623 10.83258
5 graph theory 9 0 2 5.202422 10.51883
6 hypothesis testing 10 0 2 8.155490 10.12133
Quanteda has functions to produce the most common collocations with statistical significance. Some of the terms are nonsensical like math math and are simply a product of our text preprocessing: likely stopword removal. However, we can examine the list and choose bigrams that we think ought to be included. Notable significant bigrams can be visualized:
LDA, like many unsupervised learning methods, is extremely dependent on its parameters. One such parameter that requires meticulous tuning is the number of topics. In the literature there exist many metrics for discovering the optimal number of topics with several being readily available in R.
Perplexity
Perplexity is a measure of a model’s ability to generalize to unseen data. It is the log-likelihood of a test set of data and is given by Pleplé:
\(\alpha\): the hyperparameter for the topic distributions
The lower our perplexity score the better. We can use 3-fold cross validation and plot our perplexities for various topic numbers.
set.seed(87460945)# Calculate foldsidxs <-sample(seq_len(9))folds <-split(idxs, rep(1:3, each =3, length.out =9))# Define number of topicstopics <-seq(2, 50, 1)# Create data frame for storing resultsresults <-data.frame()# Perform cross validationfor (k in topics) { scores <-c()for (i in1:3) { test_idx <- folds[[i]] train_idx <-setdiff(unlist(folds, use.names =FALSE), test_idx) test <- dtm[test_idx, ] train <- dtm[train_idx, ] LDA.out <-LDA(dtm, k, method ="Gibbs") p <-perplexity(LDA.out, newdata = test) scores <-c(scores, p) } temp <-data.frame("K"= k, "Perplexity"=mean(scores)) results <-rbind(results, temp)}# Plot Perplexity vs. Kggplot(results, aes(x=K, y=Perplexity)) +geom_line() +ggtitle("Perplexity vs. Number of Topics K")+theme(plot.title =element_text(hjust =0.5))
# Retrieving K for minimum Perplexityresults[which.min(results$Perplexity),"K"]
[1] 14
Looks like in terms of Perplexity 14 is our optimal number of topics though there is a sizable initial tip at \(K=8\) that is work exploring.
ldatuning
The ldatuning package provides us with several metrics to evaluate the number topics, all of which rely on finding extrema. We seek to minimize Arun2010 and CaoJuan2009 and maximize Deveaud2014 and Griffiths2004.
# Find number of topicsresult <-FindTopicsNumber( dtm,topics =seq(from =2, to =40, by =1),metrics =c("Griffiths2004", "CaoJuan2009", "Arun2010", "Deveaud2014"),method ="Gibbs",control =list(seed =77),mc.cores = 2L,)# Plot metricsFindTopicsNumber_plot(result)
It seems no topic numbers over 25 are viable. Let us zoom in on the 2-25 range.
# Find number of topicsresult <-FindTopicsNumber( dtm,topics =seq(from =2, to =25, by =1),metrics =c("Griffiths2004", "CaoJuan2009", "Arun2010", "Deveaud2014"),method ="Gibbs",control =list(seed =77),mc.cores = 2L,)# Plot metricsFindTopicsNumber_plot(result)
For our minima we see CaoJuan2009 suggests 3 and 9 while Arun2010 promotes somewhere in the 20-25 range. Examining our maxima we find Deveaud2014 also suggests around 3 and Griffiths2004 recommends 15 or 18. This provides us with a good range of topic numbers to explore, \(k = 3,5,8,9,12,14,15,20,25\).
Topic Visualization
Rather than individually fit a model for each topic number and print the most likely generated terms, the R package LDAvis creates fantastic intractable visualizations. LDAvis tries to answer 3 questions (Sievert & Shirley, 2014):
How prevalent is each topic?
How do the topics relate to each other?
How do the topics relate to each other?
To interpret these visualizations we fist need to define several quantities:
Relevance - Sievert and Shirley proposed relevance as a metric to rank terms based on their usefulness for interpreting topics (2014). They define it as follows:
\(\phi_{kw}\): the probability of term \(w\) for topic \(k\).
\(p_w\): the marginal probability of term \(w\) in the corpus.
\(\lambda\): a weight parameter such that \(0 \leq \lambda \leq 1\).
\(\text{log}\left( \frac{\phi_{kw}}{p_w}\right)\): the lift of a term.
A \(\lambda\) of 1 orders terms based on their topic-specific probability whereas a \(\lambda\) of 0 orders term based on their lift. Sievert and Shirley suggest \(\lambda = 0.6\). The higher this metric the better.
Lift - As seen above Lift is the ratio of a term’s probability in a given topic to its marginal probability. The metric downweights terms that are frequent throughout an entire corpus (Sievert & Shirley 2014).
Saliency - Introuced by Chuang, Manning, and Heer, it is defined as follows (2012): \[\begin{align} saliency(w) &= P(W) \times distinctiveness(w) \\
distinctiveness(w) &= \sum_T P(T|w)\text{log}\frac{P(T|w)}{P(T)}
\end{align}\]
\(P(T|w)\): the probability a word \(w\) was genereted by a topic \(T\)
\(P(T)\): the marginal probability of the topic, i.e., the probability a word was produced by topic \(T\)
The general idea is that distinctiveness is the Kullback-Leibler divergence between \(P(T|w)\) and \(P(T)\) and informs us how useful a \(w\) is for determining a generating topic. Multiplying this by \(P(w)\) yields saliency; a metric that ranks terms in their usefulness in identifying topics.
With these we can understand our visuals:
Topic circles: circles are drawn equal to the number of topics. Their areas are proportional to the estimated number of words produced by that topic. The centers of the circles are determined by computing the distances between topics which are then projected onto a 2D plane. The larger a circle is, the more prevalent it is.
Red Bars: the estimated number of times a term was created by a topic. When you select a circle the red bars for the most relevant terms will be highlighted.
Blue Bars: each bar represents the frequency of each term. When no topic is selected the blue bars for the most salient terms are displayed. When a topic is selected the blue bars are the frequency of the most relevant terms.
Topic-Term Circles: When you highlight a term the circles’ areas will change. These circles’ areas are proportional to the number of times a topic generated that specific term.
If \(\lambda=1\) terms are ranked in decreasing order of their topic-specific probability. \(\lambda=0\) ranks terms solely by their lift. The suggested \(\lambda\) is \(\lambda=0.6\) though in our experimentation we found \(\lambda \in [0.6,0.8]\) tends to provide good results.
K=3 Solution
# Fit largest modellda.out <-LDA(dtm, 3, method ="Gibbs",control =list(seed =87460945))# Find required quantitiesphi <-posterior(lda.out)$terms %>% as.matrixtheta <-posterior(lda.out)$topics %>% as.matrixvocab <-colnames(phi)doc_length <-vector()for (i in1:length(corpus)) { temp <-paste(corpus[[i]]$content, collapse =' ') doc_length <-c(doc_length, stri_count(temp, regex ='\\S+'))}temp_frequency <-as.matrix(dtm)freq_matrix <-data.frame(ST =colnames(temp_frequency),Freq =colSums(temp_frequency))# Convert to jsonjson_lda <- LDAvis::createJSON(phi = phi,theta = theta,vocab = vocab,doc.length = doc_length,term.frequency = freq_matrix$Freq)# Open server for visualization# serVis(json_lda, out.dir = "./TestVis/K_3", open.browser = FALSE)# serVis(json_lda, open.browser = TRUE)
As we might expect the overarching topics of any Data Science curriculum are programming, statistics, and mathematics. Interestingly, models actually beats out statistics indicating that data scientists are primarily concerned with modelling as opposed to inference or estimation.
K=5 Solution
# Fit largest modellda.out <-LDA(dtm, 5, method ="Gibbs",control =list(seed =87460945))# Find required quantitiesphi <-posterior(lda.out)$terms %>% as.matrixtheta <-posterior(lda.out)$topics %>% as.matrixvocab <-colnames(phi)doc_length <-vector()for (i in1:length(corpus)) { temp <-paste(corpus[[i]]$content, collapse =' ') doc_length <-c(doc_length, stri_count(temp, regex ='\\S+'))}temp_frequency <-as.matrix(dtm)freq_matrix <-data.frame(ST =colnames(temp_frequency),Freq =colSums(temp_frequency))# Convert to jsonjson_lda <- LDAvis::createJSON(phi = phi,theta = theta,vocab = vocab,doc.length = doc_length,term.frequency = freq_matrix$Freq)# Open server for visualization# serVis(json_lda, out.dir = "./TestVis/K_5", open.browser = FALSE)# serVis(json_lda, open.browser = TRUE)
Topic 1: statistics comes to the top with applications, random, theory, probability, estimation, and distributions trailing behind. Clearly a statistics topic.
Topic 2: software, database, and systems come to the top of this topic. Databases are an important part of any data scientist’s job and it makes sense this would be put into its own topic- its quite distinct from any of the others.
Topic 3: models, analysis, learning, regression, clustering, classification, and machine_learning. Very clearly a machine learning topic.
Topic 4: mathematics is by in large the most relevant term. This coupled with words such as calculus, equations, linear, integration shows this is a mathematics topic.
Topic 5: programming, algorithms, computer, objectorientated, this is likely the computer science and programming aspect of any data science program.
K=8 Solution
# Fit largest modellda.out <-LDA(dtm, 8, method ="Gibbs",control =list(seed =87460945))# Find required quantitiesphi <-posterior(lda.out)$terms %>% as.matrixtheta <-posterior(lda.out)$topics %>% as.matrixvocab <-colnames(phi)doc_length <-vector()for (i in1:length(corpus)) { temp <-paste(corpus[[i]]$content, collapse =' ') doc_length <-c(doc_length, stri_count(temp, regex ='\\S+'))}temp_frequency <-as.matrix(dtm)freq_matrix <-data.frame(ST =colnames(temp_frequency),Freq =colSums(temp_frequency))# Convert to jsonjson_lda <- LDAvis::createJSON(phi = phi,theta = theta,vocab = vocab,doc.length = doc_length,term.frequency = freq_matrix$Freq)# Open server for visualization# serVis(json_lda, out.dir = "./TestVis/K_8", open.browser = FALSE)# serVis(json_lda, open.browser = TRUE)
For the sake of brevity I will simply list what I consider to be the most likely topics from here on out. Note asterisks indicate noisey topics I am unsure of and a ? indicates the topic is uninterpretable.
Topic 1: statistics
Topic 2: machine learning
Topic 3: business
Topic 4: mathematics
Topic 5: data analysis/inference
Topic 6: databases
Topic 7: computer science
Topic 8: linear algebra
K=9 Solution
# Fit largest modellda.out <-LDA(dtm, 9, method ="Gibbs",control =list(seed =87460945))# Find required quantitiesphi <-posterior(lda.out)$terms %>% as.matrixtheta <-posterior(lda.out)$topics %>% as.matrixvocab <-colnames(phi)doc_length <-vector()for (i in1:length(corpus)) { temp <-paste(corpus[[i]]$content, collapse =' ') doc_length <-c(doc_length, stri_count(temp, regex ='\\S+'))}temp_frequency <-as.matrix(dtm)freq_matrix <-data.frame(ST =colnames(temp_frequency),Freq =colSums(temp_frequency))# Convert to jsonjson_lda <- LDAvis::createJSON(phi = phi,theta = theta,vocab = vocab,doc.length = doc_length,term.frequency = freq_matrix$Freq)# Open server for visualization# serVis(json_lda, out.dir = "./TestVis/K_9", open.browser = FALSE)# serVis(json_lda, open.browser = TRUE)
Currently we are using 1 sampled degree pathway from each university. It would be interesting to see what happens if we sample say 100 students from each university. Will the variance in electives lead to different topics?
<<SimpleCorpus>>
Metadata: corpus specific: 1, document level (indexed): 0
Content: documents: 1
Western Computer Science 2210A/B (1)
Lists, stacks, queues, priority queues, trees, graphs, and their associated algorithms; file structures; sorting, searching, and hashing techniques; time and space complexity.
Now we must discover topic numbers. Unfortunately, we cannot use Perplexity here as the model is too big and the code will take too long to run. However, ldatuning is still an option.
# Find number of topicsresult <-FindTopicsNumber( dtm,topics =seq(from =2, to =20, by =1),metrics =c("Griffiths2004", "CaoJuan2009", "Arun2010", "Deveaud2014"),method ="Gibbs",control =list(seed =77),mc.cores = 2L,)# Plot metricsFindTopicsNumber_plot(result)
# Find number of topicsresult <-FindTopicsNumber( dtm,topics =seq(from =20, to =40, by =1),metrics =c("Griffiths2004", "CaoJuan2009", "Arun2010", "Deveaud2014"),method ="Gibbs",control =list(seed =77),mc.cores = 2L,)# Plot metricsFindTopicsNumber_plot(result)
Potential Topic numbers from CaoJuan2009 are 23, 4. Arun2010 hits 0 at 20 and 40. Deveaud2014 suggests 3 and 23. Griffiths plateaus around 24.
K = 3 Solution
We can start with the 3 topic solution to confirm our model is working good. We expect to see something similar to before: statistics, mathematics, and programming/computer science.
# Fit largest modellda.out <-LDA(dtm, 3, method ="Gibbs", control =list(seed =87460945))# Find required quantitiesphi <-posterior(lda.out)$terms %>% as.matrixtheta <-posterior(lda.out)$topics %>% as.matrixvocab <-colnames(phi)doc_length <-vector()for (i in1:length(corpus)) { temp <-paste(corpus[[i]]$content, collapse =' ') doc_length <-c(doc_length, stri_count(temp, regex ='\\S+'))}temp_frequency <-as.matrix(dtm)freq_matrix <-data.frame(ST =colnames(temp_frequency),Freq =colSums(temp_frequency))# Convert to jsonjson_lda <- LDAvis::createJSON(phi = phi,theta = theta,vocab = vocab,doc.length = doc_length,term.frequency = freq_matrix$Freq)# Open server for visualization# serVis(json_lda, out.dir = "./totalk3", open.browser = FALSE)# serVis(json_lda, open.browser = TRUE)
Overall there are some topic differences but not too much change overall.
University-Topic Composition
Another interesting question to explore is which topics do universities prioritize? Is one university’s Data Science program more heavily biased towards Math? How about Statistics? Recall the \(\gamma\) matrix provides us with the per-document-per-topic probabilities. That is, it provides us with a percentage break down of which topics generated each document.
Now that we have a few models and labelled topics we can look at what topics specific universities focus on. Let’s start with the 3 group solution.
Given our optimal models we may now move onto exploring what are the main words that comprise each of the topics. To do so we explore \(\beta\) values using wordclouds. Recall that the $\beta$ matrix provides us with the per-topic-per-word probabilities.
A topic model for movie reviews. (n.d.). Retrieved March 20, 2023, from
https://ldavis.cpsievert.me/reviews/reviews.html
Blei, D. M., Ng, A. Y., & Jordan, M. I. (2003). Latent dirichlet allocation. Journal of machine Learning research, 3(Jan), 993-1022.
Fernandes, E. (2019, November 2). Text preprocessing for NLP and Machine Learning Using R. DataMathStat. Retrieved March 20, 2023, from https://datamathstat.wordpress.com/2019/10/25/text-preprocessing-for-nlp-and-machine-learning-using-r/
Gandrud, C. (2015, May 8). A Link Between topicmodels LDA and LDAvis. R. Retrieved March 20, 2023, from https://www.r-bloggers.com/2015/05/a-link-between-topicmodels-lda-and-ldavis/
Grimmer, J., Roberts, M. E., & Stewart, B. M. (2022). Text as data: A New Framework for Machine Learning and the Social Sciences. Princeton University Press.
Lau, J. H., Baldwin, T., & Newman, D. (2013). On collocations and topic models. ACM Transactions onSpeech and Language Processing, 10(3), 1–14. https://doi.org/10.1145/2483969.2483972
Meza, D. (2015, July 20). Topic modeling using R · Knowledger. knowledgeR. Retrieved March 20, 2023,from https://knowledger.rbind.io/post/topic-modeling-using-r/
Pleplé, Q. (n.d.). Perplexity To Evaluate Topic Models. qpleple. Retrieved March 20, 2023, from http://qpleple.com/perplexity-to-evaluate-topic-models/
Sievert, C., & Shirley, K. (2014, June). LDAvis: A method for visualizing and interpreting topics. In Proceedings of the workshop on interactive language learning, visualization, and interfaces (pp. 63-70).
Silge, J., Robinson, D., & ProQuest (Firm). (2017). Text mining with R: A tidy approach (First ed.). O’Reilly Media.
Tufts, C. (n.d.). The Little Book of LDA. Latest Posts – Mining the Details. Retrieved March 20, 2023, from https://miningthedetails.com/LDA_Inference_Book/lda-inference.html
