%\VignetteIndexEntry{Neural Text Models with R package ruimtehol}

\documentclass[nojss]{jss}
\title{Neural Text Models with R package ruimtehol}
\author{Jan Wijffels}
\Plainauthor{Jan Wijffels}
\Abstract{
Ruimtehol is a comprehensive R package which wraps the StarSpace C++ library (\url{https://github.com/facebookresearch/StarSpace}). Starspace is a neural natural language modelling toolkit which allows you to calculate word, sentence, article, document, webpage, link and entity 'embeddings'. By using the 'embeddings', you can perform text based multi-label classification, find similarities between texts and categories, do collaborative-filtering based recommendation as well as content-based recommendation, find out relations between entities, calculate graph 'embeddings' as well as perform semi-supervised learning and multi-task learning on plain text. 
The techniques are explained in detail in the paper: \emph{StarSpace: Embed All The Things!}, available at (\url{https://arxiv.org/abs/1709.03856}).}
\Keywords{Starspace, NLP, embed, embedding, neural, text}
\Plainkeywords{Starspace, NLP, embed, embedding, neural, text}
\Address{
  BNOSAC - Open Analytical Helpers\\
  E-mail: \email{jwijffels@bnosac.be}\\
  URL: \url{http://www.bnosac.be}\\
}

\begin{document}
\SweaveOpts{concordance=TRUE}

<<preliminaries, echo=FALSE, results=hide>>=
options(prompt = "R> ", continue = "+   ")
options(prompt = " ", continue = "   ")
set.seed(123456789)
@

\section{Ground control to ruimtehol}
\subsection{Overview}
The ruimtehol R package which wraps the StarSpace C++ library (\url{https://github.com/facebookresearch/StarSpace}), focussing on building and utilising embedding models for natural language. 
It allows you to do the following Natural Language Processing tasks

\begin{itemize}
\item{Text classification}
\item{Learning word, sentence or document level embeddings}
\item{Finding sentence or document similarity}
\item{Ranking web documents}
\item{Content-based recommendation (e.g. recommend text/music based on the content)}
\item{Collaborative filtering based recommendation (e.g. recommend text/music based on interest)}
\item{Identification of entity relationships}
\end{itemize}

\subsubsection{Source code repository}
The source code of the package is on github at \url{https://github.com/bnosac/ruimtehol}.\\
It uses Starspace version 'STARSPACE-2017-2' cloned in September 2018 to \url{https://github.com/bnosac-dev/StarSpace}. Modifications were done in order to make it CRAN compliant and to make the usage more straightforward for R users. 
The R package is distributed under the Mozilla Public License version 2.0.

\subsection{Functionalities} \label{functionalities}
The R package allows you to have a low-level access to the C++ library for fine-grained control and provides as well high-level interfaces for commonly used tasks. Functionalities are also included for saving and loading models, using the models to predict and general embedding similarity and ranking functions.

\begin{tabular}{l | l}
\hline	
starspace                    & Low-level interface to build a Starspace model                 \\
starspace\_load\_model       & Load a pre-trained model or a tab-separated file               \\
starspace\_save\_model       & Save a Starspace model                                         \\
starspace\_embedding         & Get embeddings of documents/words/ngrams/labels                \\
starspace\_knn               & Find k-nearest neighbouring information for new text           \\
starspace\_dictonary         & Get words/labels part of the model dictionary                  \\
predict.textspace            & Get predictions along a Starspace model                        \\
as.matrix                    & Get words and label embeddings                                 \\
embedding\_similarity        & Cosine/dot product similarity between embeddings - top-n most similar text       \\
embed\_wordspace             & Build a Starspace model which calculates word/ngram embeddings                   \\
embed\_sentencespace         & Build a Starspace model which calculates sentence embeddings                     \\
embed\_articlespace          & Build a Starspace model for embedding articles - sentence-article similarities   \\
embed\_tagspace              & Build a Starspace model for multi-label classification                           \\
embed\_docspace              & Build a Starspace model for content-based recommendation                         \\
embed\_pagespace             & Build a Starspace model for interest-based recommendation                        \\
embed\_entityrelationspace   & Build a Starspace model for entity relationship completion                       \\
\hline
\end{tabular}

\begin{center}
\includegraphics[width=12cm]{logo-ruimtehol.png}
\end{center}

\newpage
\subsection{Example data}
In what follows below, we showcase some of the use cases of the R package. In order to do that, we will use text with questions and answers from the Belgian parliament. This dataset was collected as open data under the CC0 license from \url{http://data.dekamer.be}

The data contains Dutch questions asked in the year 2017 by members in the National Belgian parliament. Each question was categorised alongside several themes and the dataset also contains the answer which was given by the department or minister who was responsible for that topic. We also know to which political party the person belonged to who was asking the question.

<<>>=
library(ruimtehol)
data("dekamer", package = "ruimtehol")
str(dekamer)
@

\newpage
\section{Text classification}
In this example, we will perform text classification. Each question in parliament can be labelled with one or more tags. A tagspace model is constructed which can be used to tag new questions with the learned tagset.

\subsection{Data preparation}
The following data preparation is done first
\begin{itemize}
  \item make sure all the text is separated by spaces, this is the format which Starspace needs
  \item make sure the response is a list of all categories in case you do multi-label classification and each category should not contain spaces (in the below example spaces are replaced with a dash)
\end{itemize}
The Starspace C++ library internally uses spaces as a separator, make sure each token is separated by a space if you want to include it in the model and that the labels do not contain spaces.

<<>>=
dekamer$x <- strsplit(dekamer$question, "\\W")
dekamer$x <- lapply(dekamer$x, FUN = function(x) x[x != ""])
dekamer$x <- sapply(dekamer$x, FUN = function(x) paste(x, collapse = " "))
dekamer$x <- tolower(dekamer$x)
dekamer$y <- strsplit(dekamer$question_theme, split = ",")
dekamer$y <- lapply(dekamer$y, FUN=function(x) gsub(" ", "-", x))
dekamer$x[1:2]
dekamer$y[1:2]
@

\newpage
\subsection{Model building} \label{embedtagspace}

Next a model is constructed, in this case a tagspace model which is a model which can be used for simple classification as well as multi-label classification (which is the case here).  

<<results=hide>>=
set.seed(123456789)
model <- embed_tagspace(x = dekamer$x, y = dekamer$y, 
                        early_stopping = 0.8, validationPatience = 10,
                        dim = 50, 
                        lr = 0.01, epoch = 40, loss = "softmax", adagrad = TRUE, 
                        similarity = "cosine", negSearchLimit = 50,
                        ngrams = 2, minCount = 2, bucket = 100000,
                        maxTrainTime = 2 * 60)
@
<<>>=
model
@

The code above has trained a model with the following relevant arguments

\begin{itemize}
\item  \textbf{early\_stopping}: data is split in a training (80 \%) and a validation set (20 \%)
\item  \textbf{dim}: the dimension of the embedding is set to 50
\item  optimisation is done with \textbf{adagrad}, during 40 \textbf{epochs}, starting with a learning rate (\textbf{lr}) of 0.01 and each time decreasing the learning rate by 1/epoch. 
\item  The \textbf{loss} which is optimised is softmax loss. If the loss has not decreased during 10 epochs as set with \textbf{validationPatience}, training is stopped.
\item  Similarity between positive and negative labels is done by \textbf{cosine} similarity
\item  \textbf{negSearchLimit} indicates the number of negative samples which are taken (for each question we know which labels were given (positive) and we sample from the list of labels which were not given a bunch of negatives)
\item  model is trained on bigrams (argument \textbf{ngrams}) and these should occur at least twice (argument \textbf{minCount}), embeddings of bigrams are hashed to 100000 buckets
\item  model is trained for maximum 120 seconds (argument \textbf{maxTrainTime}))
\end{itemize}

\subsection{Model inspection}
\subsubsection{Loss evolution}
The plot shows the evolution of the loss over the epochs on the training set and the validation set. This should generally decrease steadily and stabilise on the validation set which indicates it has learned the embeddings well.

<<fig=TRUE>>=
plot(model)
@

\subsubsection{Dictionary}
Starspace models the embeddings of the labels and the words in the same embedding space allowing you to compute similarities across labels and text. To get the dictionary of all terms in the model (words and labels), you can do the following. 

<<>>=
dict <- starspace_dictionary(model)
str(dict)
length(dict$labels)
@

The dictionary element contains a data.frame with all words as well as all labels which were found in the data and for which embeddings are calculated.\\ 
Note that bigrams or ngrams are not stored and you'll also see that labels are prefixed with the text \_\_label\_\_, allowing them to be distinguished from plain words. 

\subsection{Embeddings of the dictionary}
You can get the learned embeddings of the words and the labels which are part of the dictionary as follows. 

<<>>=
emb <- as.matrix(model)
dim(emb)
@

If you only want embeddings of the words or of the labels, you can set the \emph{type} argument in the function \emph{as.matrix} or you can use the function \emph{starspace\_embedding} directly. Below the embedding of the word 'geld' is retrieved as well as the embedding of the label 'VERVOERBELEID'.

<<>>=
emb_words  <- as.matrix(model, type = "words")
emb_labels <- as.matrix(model, type = "labels", prefix = FALSE)
e <- starspace_embedding(model, x = c("__label__VERVOERBELEID", "geld"), type = "ngram")
@

If you trained a model with $ngrams > 1$, you can get the embedding of a bigram/ngram also. For this Starspace uses a hashing trick from \emph{fastText} and gets - based on the words which define the bigram/ngram - the embedding of the hashed bucket of the combined term. If you specify to retrieve this type of embedding, you can not have more words than you specified \emph{ngrams} when doing the training.

<<>>=
e <- starspace_embedding(model, c("nationale loterij"), type = "ngram")
@

\subsection{Embeddings of full text} \label{embeddingsfulltext}

For retrieving embeddings of full text, use \emph{type = 'document'} which is the default of \emph{starspace\_embedding}. It aggregates the embeddings of the words which are part of the text. Note that this aggregation is governed by the normalization parameter p which you can set when you build the model. The sum of the embeddings of the words which are in the text is normalised by dividing by $\#words^p$. If $p=1$, this is equivalent to taking the average of the embeddings. When $p=0$, this is equivalent to taking sum of the embeddings. The default is $p=0.5$ indicating a mixture of both.

<<>>=
text <- c("de nmbs heeft het treinaanbod uitgebreid via onteigening ...",
          "de migranten komen naar europa de asielcentra ...")
emb_text <- starspace_embedding(model, text)
dim(emb_text)
@

\subsection{Predict/Similarities/Ranking}
You can extract predictions and get embedding similarities for information retrieval and ranking.
Function \emph{predict} gets the document embeddings of the text and find the closest among the labels.

<<>>=
predict(model, "de migranten komen naar europa de asielcentra ...")
@

The same can also be achieved with the function \emph{embedding\_similarity} which provides cosine and dot product similarities. They give the same numbers as the predict functionality.

<<>>=
embedding_similarity(emb_text, emb_labels, type = "cosine", top_n = 5)
@

Shorthands for the knn Starspace functionality is also provided. This function allows you to answer things like 'What does this look like'. It shows the nearest neighbour of text to the dictionary.

<<>>=
starspace_knn(model, "de migranten komen naar europa de asielcentra ...", k = 5)
@

\subsection{Customising}

As the words and labels are in the same embedding space, you can interpret the predict and similarity functionalities in a broad way. This is shown below where the target documents to compare with are changed. 

<<results=hide>>=
targetdocs <- c("__label__FISCALITEIT", 
                "__label__OVERHEIDSADMINISTRATIE", 
                "__label__MIGRATIEBELEID", 
                "__label__POLITIE", 
                "__label__BUITENLANDS-BELEID", 
                "__label__ECONOMISCH-BELEID", 
                "de migranten komen naar europa ZZZ", 
                "__label__PERSONEEL")
predict(model, "de migranten komen naar europa de asielcentra ...", 
        basedoc = targetdocs)
embedding_similarity(
  starspace_embedding(model, "de migranten komen naar europa de asielcentra ..."),
  starspace_embedding(model, targetdocs), top_n = 3)
@


\subsection{Save/Load model}
Saving the model consists of saving the embeddings of the words and the labels as well as storing the model parameters. The saved model can next be loaded back in and used for information retrieval.
The following approach is the advised approach to save and reload models.

<<>>=
starspace_save_model(model, file = "textspace.ruimtehol")
model <- starspace_load_model("textspace.ruimtehol")
@

<<echo=FALSE>>=
invisible(file.remove("textspace.ruimtehol"))
@


\newpage
\section{Other models}
The ruimtehol package contains many more models. It is advised to just inspect the help of the functions listed up in section \ref{functionalities}.
\begin{itemize}
  \item Embeddings of word, sentences, articles, documents, webpages, links and entities: See the examples in the package.
  \item Ranking and information retrieval: See the examples in the package.
  \item Collaborative filtering: See the examples in the package.
\end{itemize}

\section{Semi-supervised learning}
In the example in this vignette on classification modelling (embed\_tagspace shown in section \ref{embedtagspace}), Starspace was used in a completely supervised setting. If you have a look at the documentation of the functions embed\_wordspace, embed\_sentencespace, embed\_articlespace, you will notice that these are completely unsupervised. Starspace also allows to do a combination of both, namely you can perform semi-supervised learning. This is shown below where we randomly remove some data with text and some data of the labels as well and still learn on the full data. As long as we have information on the labels or on the terms or both, we can learn embeddings.

<<results=hide>>=
set.seed(321)
dekamer <- dekamer[order(rnorm(n = nrow(dekamer))), ]
X <- dekamer$x
Y <- dekamer$y
X[1:250]   <- NA
Y[251:500] <- NA
model <- embed_tagspace(x = X, y = Y, 
                        early_stopping = 0.8, validationPatience = 10,
                        dim = 50, 
                        lr = 0.01, epoch = 40, loss = "softmax", adagrad = TRUE, 
                        similarity = "cosine", negSearchLimit = 50,
                        ngrams = 2, minCount = 2,
                        maxTrainTime = 2 * 60)
@

\newpage
\section{Transfer learning}
The ruimtehol R package also allows to do transfer learning. In transfer learning, knowledge gained while learning on other data can be transferred to new data. The typical use case of this is the case where we have already pretrained embeddings available. Several authors have provided embeddings which were trained on different sources (e.g. Wikipedia / Open databases / Google or Bing queries / Gigaword / Open CONLLU corpora). These are mostly monolingual resources but also cross-lingual embeddings are common now. There are 2 use cases of such pretrained embeddings, namely.
\begin{enumerate}
  \item You can use these embedding as is
  \item You can use these embedding as starting point to train and customise them on your data (transfer learning)
\end{enumerate}
In the examples below, both will be shown. In either case, we need to provide the argument \emph{embeddings} which should be a pretrained embedding matrix where the rownames of the matrix are the terms or labels from the model. 

\subsection{Transfer learning - use embeddings as is}
In order to show the first use case we generate some random embedding matrix and feed it to function \emph{starspace} which is the main workhorse behind all embed\_ \ldots functions.
The important arguments that you need to give are
\begin{itemize}
  \item embeddings: a matrix of embeddings where the row names indicate the terms or label
  \item similarity: how you want to calculate similarities between embeddings and documents
  \item ngrams: when calculating similarities, shall we consider more than unigrams only
  \item p: normalisation parameter as explained in section \ref{embeddingsfulltext}
  \item trainMode: either 0 (tagspace), 1 (pagespace), 2 (articlespace), 3 (sentence space), 4 (multi-relational graphspace), 5 (word embeddings)
\end{itemize}

<<>>=
pretrained <- matrix(data = rnorm(1000 * 100), nrow = 1000, ncol = 100, 
                     dimnames = list(term = sprintf("word%s", 1:1000)))
model <- starspace(embeddings = pretrained, 
                   similarity = "cosine", p = 0.5, ngrams = 1, trainMode = 5)
predict(model, newdata = c("word5 word1 word5 word3"), type = "knn")
@

\subsection{Transfer learning - use embeddings as starting values for training}
Now for the second use case: \textbf{transfer learning}. Below we build a simple wordspace model to extract word embeddings. These embeddings will be passed on as starting values for another tagspace model.

<<results=hide>>=
set.seed(321)
model <- embed_wordspace(dekamer$x, 
                         dim = 50, ws = 7, epoch = 5, ngrams = 2, adagrad = FALSE,
                         margin = 0.8, negSearchLimit = 10,
                         maxTrainTime = 2 * 60)
pretrained_words  <- as.matrix(model)
@

It's important to note that we need to pass on pretrained embeddings \textbf{for all terms as well as all labels which we want to keep training upon}. As we haven't got the embeddings of the labels, in the example below, we assign some random starting values to the embedding of the labels. (Small note: in business settings it makes sense to let these labels start from a sensible combination of the embeddings of words which you think are similar to that label). Starspace uses the default prefix \_\_label\_\_ for identifying a label, which we need to add as prefix to the rownames from the label embeddings. 

<<>>=
labels            <- sort(unique(unlist(dekamer$y)))
pretrained_labels <- matrix(data = rnorm(n = length(labels) * 50, 
                                         mean = mean(pretrained_words), 
                                         sd = sd(pretrained_words)), 
                            nrow = length(labels), 
                            ncol = 50, 
                            dimnames = list(term = sprintf("__label__%s", labels)))
pretrained        <- rbind(pretrained_words, pretrained_labels)
@

Once we have these pretrained embeddings, we can use them as starting values for the training.
Note that we do not need to provide the dim argument as this argument is governed by the dimension of the pretrained embedding matrix.

<<results=hide>>=
set.seed(321)
model <- embed_tagspace(x = dekamer$x, y = dekamer$y, 
                        embeddings = pretrained,
                        early_stopping = 0.8, validationPatience = 10,
                        dim = 50, 
                        lr = 0.01, epoch = 40, loss = "softmax", adagrad = TRUE, 
                        similarity = "cosine", negSearchLimit = 50,
                        ngrams = 2, minCount = 2,
                        maxTrainTime = 2 * 60)
embedding <- as.matrix(model)
@

<<fig=TRUE, fig.align="center", out.width="3cm">>=
plot(model)
starspace_knn(model, "__label__FISCALITEIT", k = 10)
@

And now we have the model ready for doing further work based on the enhanced embedding matrix.

\end{document}