Reading, wRiting, and pRogramming

R and RStudio do not exist in isolation. Much of the time, we use it to read in files, write to new files, or do various programmatic conversions.

Take this simple example. We want to save a dataframe that includes non-English characters. The function, write.csv(), takes the system’s native encoding by default, whereas write_csv() supports only UTF-8.

library(readr)

df <- data.frame(food = c("Crêpe", "Spätzle", "Smørrebrød", "חומוס"))
write.csv(df, "native.csv", row.names = FALSE)
write_csv(df, "utf8.csv")

Everything looks fine when we use the corresponding read functions on the files, but if we switch them around, we run into problems:

read.csv("utf8.csv")

##         food
## 1      Crêpe
## 2    Spätzle
## 3 Smørrebrød
## 4      חומוס

read_csv("native.csv")

## Parsed with column specification:
## cols(
##   food = col_character()
## )

## # A tibble: 4 x 1
##   food      
##   <chr>     
## 1 Crêpe     
## 2 Spätzle   
## 3 Smørrebrød
## 4 חומוס

We can work around these problems using the fileEncoding parameter in read.csv() or the locale parameter in read_csv().²

However, some conversion processes rely on base-R commands that translate to/from native encodings, resulting in “forced round-trips.” Often, there is no workaround unless you dig into C. Many rendering functions, such as rmarkdown::render() or reprex::reprex(), get stuck because of this combination of base R defaults and lack of Windows UTF-8 support.

In fact, many, many encoding issues ultimately drill down to the same few problematic base R functions, which include sink(), source(), writeLines(),³ and format().

Beyond functions, packages have a few extra sets of restrictions. Special (non-English) characters are not allowed in package names, nor do they always display properly in search results. Take a look at Colin Fay’s proustr package documentation, for example. The help table of contents is garbled, but the help pages themselves are mostly fine.

Enter right-to-left (RTL) languages

Things sometimes get trickier when you work with right-to-left (RTL) languages. By RTL, I mean that most of the language is written from right to left, but numbers or URL’s or code or whatever else are often still left-to-right (or in English characters). Fortunately, there are standards for bidirectional, or “BiDi”, text. In fact, there are even UTF-8 codes for defining text directions that:

• use “isolates” or “embeddings” to set a base direction and let the BiDi algorithm take it from there
• automatically determine text direction according to the first strongly-typed character (e.g. “a” or “א” but not “!”)
• force a direction with “override” codes that ignore the BiDi algorithm

If you’re like me, reading through these guidelines the first time around doesn’t make much sense, so let’s take a look at how Excel handles a few different scenarios:

There’s a lot going on there, but it becomes clear pretty quickly that even without using English, the combination of Hebrew letters, numbers, and punctuation requires a set of rules for sensible display.

These rules also bleed into data frames and the like. In some ways, the following bug makes sense. After all, if everything was in Hebrew, you’d want columns to be displayed RTL.

As a non-native RTL-er, these issues are a source of frustration but also great fascination for me. I encourage those of you more fluent in RTL languages than I to weigh in on issues related to the IDE, plotting, and elsewhere. If you need guidance on producing a minimal reproducible example of your problem, check out the reprex package or Yihui Xie’s Minimal Reproducible Example Paradox blogpost.

A New Hope

There is beta UTF-8 support on Windows! Does this solve all my problems? No, not yet.

However, UTF-8 efforts from both the Windows and R world (e.g. utf8) are making progress in this domain. In the mean time, the rest of us can continue to file issues, make PRs, and avoid base R functions that ignore our wish for a UTF-8 world.

Thanks

I may not have managed any PR’s at the Tidyverse Developer Day after the rstudio::conf this year, but I had the opportunity to connect with several patient and kind encoding pros, including Colin Fay, Christophe Dervieux, Yoni Sidi (extra thanks to Yoni for reviewing my draft of this post!), and Kirill Müller. They gave me the motivation to read up on encoding yet again and assemble my thoughts and learnings here in this post.