# How to best manage computationally heavy analyses

Many NLP analyses require a lot of computational resources; but standard sized datasets in psychology can be analysed with text on a standard laptop. It is rather the computation of the pre-trained language models that require a huge amount of computational resources. Wolf et al., (2020) exemplify this by pointing out that the RoBERTa language model:

> “was trained on 160 GB of text using 1024 32GB V100. On Amazon-Web-Services cloud computing (AWS), such a pretraining would cost approximately 100K USD.” (p. 2)

Hence, there are a lot of computations behind the word embeddings. In text, the most computationally heavy and time consuming elements are the process of retrieving word embeddings using `textEmbedLayersOutput` (which is also used in `textEmbed`). Retrieving word embeddings for a standard dataset with a few hundred participants may take between 15 minutes to an hour. Hence, it is worth planning analyses. A few time and resource management advice include:

1. Testing: Before you run the analyses on your entire dataset, ensure that everything first runs smoothly on a small part of the data set (e.g., 20 rows of your data). Note that this should not be a process of testing different setting, but only to see that everything works.
2. Timekeeping: When running different analyses have the computer take time (see code example below), so that you get a better understanding of how long time different analyses take.
3. Scheduling: Run those analyses that take longer time over a coffee break or over night. So for example, it might be worth retrieving all word embeddings at a separate time.

    library(text)
    # Save starting time
    T1 <- Sys.time()
    textEmbed(Language_based_assessment_data_8_10[1,1],
              layers = 12,
              decontexts = FALSE)
    # Save stoping time
    T2 <- Sys.time()

    # Compute time taken to run above function
    T2-T1

### Your system’s capacity

Thinking about your computer’s memory capacity may become important if you have a lot of data and use multiple layers with many dimensions. For example consider that one sentence of 10 words/tokens, which are each represented by 12 layers a 768 dimensions results in 92 160 values (i.e., 10 x 12 x 768). To avoid running out of memory and get analyses to run faster, consider to:

1. Only retrieve the layers that you plan to use (e.g., `layers = 11:12`) rather than retrieving all layers (i.e., `layers = 'all'`).
2. Only retrieve tokens if you are planing to use them; otherwise set `return_tokens = FALSE`.
3. Do not ask for decontextualized word embeddings if you are not going to us them (e.g., in plotting).

### References

Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., & Funtowicz, M. (2019). Huggingface’s transformers: State-of-the-art natural language processing.