Why is as.factor() in R so slow and can it be improved?

I've recently discovered that as.factor() operates very slowly, particularly on character vectors with long character strings. This seems to be a particular problem inside dplyr::mutate() statements; operation on vectors outside of mutate() statements seems to be much faster. Is there some way to speed up performance of this function or to substitute a faster one?

Using the factor() function with the "levels" argument is much faster. Using the "levels" argument is key; without it, factor() is just as slow as as.factor(). An example is below:

require(microbenchmark)
require(tidyverse)

#Generate a random vector of 22-character long strings consisting of numeric characters
random_char_vec = sprintf("%022.0f", runif(1e7)*1e22)

#Put it into a tibble
random_num_tibble = tibble(random_char_vec = random_char_vec)

#The problem seems to be when the character string is very long; 
#if each element of random_char_vec is only five characters 
#this takes no time at all; 
#at 22 digits it takes over two minutes.

microbenchmark(
  {
    factor_random_num = as.factor(random_char_vec)
  },
  times=1)
Unit: seconds
                                                   expr      min       lq     mean
 {     factor_random_num = as.factor(random_char_vec) } 146.2098 146.2098 146.2098
   median       uq      max neval
 146.2098 146.2098 146.2098     1

#This takes two seconds.
microbenchmark(
  {
    factor_random_num = factor(random_char_vec, levels = unique(random_char_vec))
  },
  times=1)
Unit: seconds
                                                                                  expr
 {     factor_random_num = factor(random_char_vec, levels = unique(random_char_vec)) }
      min       lq     mean   median       uq      max neval
 1.796813 1.796813 1.796813 1.796813 1.796813 1.796813     1

#The key to the speedup is precomputing the levels; without setting levels, no speedup.
microbenchmark(
  {
    factor_random_num = factor(random_char_vec)
  },
  times=1)
Unit: seconds
                                                expr      min       lq     mean   median
 {     factor_random_num = factor(random_char_vec) } 123.8821 123.8821 123.8821 123.8821
       uq      max neval
 123.8821 123.8821     1

I don't know why putting the call to factor() inside the mutate() statement sometimes leads to such slow performance.

Hope this helps someone encountering the same issue!

tl;dr base functions can often be sped up by doing less, but it is often not worth the effort. data.table provides an alternative to as.factor.

Generally, if the function body, (your_func without parentheses in the console) does not say something like .Primitive, or .Internal then significant speed ups are possible. The question is, is it worth the effort?

A base case, 200 level factor of length 100 million, base as.factor takes ~3 seconds on my machine.

library(data.table)
library(stringi)
s <- stri_rand_strings(200, 20)
set.seed(1)
chr <- sample(s, 1e8, TRUE)
dt <- data.table(chr = chr)

bench::mark(
  transform(dt, chr = as.factor(chr)),
  dplyr::mutate(dt, chr = as.factor(chr)), 
  check = FALSE
)

Switching to a data.table approac, using (inadvisable) a package internal function, this can be roughly cut in half.

start = proc.time()
dt[,chr := data.table:::as_factor(chr)]
timetaken(start)

Unless it is frequently run, the conversion is not worth the time it takes to type it.