Ndtyjky

I am working on a project in which I use several language detection algorithms, such as Textcat and CLD3. I have a dataframe in which I recorded what language a piece of text was written in, what the guess of each algorithm was and whether that guess was correct.

Because the length of the strings varies greatly, I want to evaluate the performance of each algorithm over a moving threshold (such as for all strings with more than 5 words, then more than 10 words, etc)

The data look like this:

Text    Language CLD Textcat Word_count CLD_correct Textcat_correct 

String1 EN       en  en      20         1           1

String2 EN       NA  fr      5          0           0

String3 FR       fr  es      10         1           0

String4 ES       ca  es      7          0           1

What I would dearly like to do is to plot the accuracy for each threshold in terms of the number of words. For example, I found that overall CLD labels the language correctly in 75% of cases. However, when considering only strings with 7 words or more, this goes up to 85%.

So on the x-axis I want to plot the number of words for the threshold, on the y-axis the percentage of correct guesses made by the algorithm.

I know how to do this by hand (subset the dataframe for value Word_count > x, calculate the accuracy for each algorithm, store those in a data frame, calculate for Word_count > y, and so on, and then plot it), but because my sample is very large, it would take a gargantuan amount of work to get this all done, and there must be a more intelligent way to do this. I considered iterating over different thresholds with a for-loop to calculate values for each and then storing those, but a large part of the strings in this data set can be over 100 words, and I am considering to do the same for character length.

Does someone know how this could be fixed in a more automated fashion?

First define a vector of used algorithms

algorithmrithms <- c('Textcat_correct', 'CLD_correct')

Then create a vector with the number of words for which you want to see the accuracy

word.size <- seq(5, 20, 5)

Now you can use the package dplyr and lapply to get a list for each word amount and algorithm.

library(dplyr)

resultList <- lapply(word.size, function(y) { 

    lapply(algorithm, function(x) { 

        df %>%

        rename(algorithm = x) %>%

        filter(Word_count >= y) %>%

        group_by(algorithm) %>%

         summarise(all = sum(Word_count)) %>%

         mutate(accuracy = all/sum(all)*100) %>%

         filter(algorithm == 1) %>%

         mutate(algorithm=replace(algorithm, algorithm == 1, x)) %>%

         mutate(words = y) })

    })

This list you can convert to a dataframe

df2 <- as.data.frame(do.call(rbind, unlist(resultList, recursive=F)))

And now you can plot your results

library(ggplot2)

ggplot(df2, aes(words, accuracy, fill=algorithm)) + 

    geom_bar(stat="identity", position="dodge")

As result you get this