Encoding Resilience

25 Years of Press Freedom Index Data

Because I had to update the data annually in my pressfreedom while developing pressfreedom.data, an R data package, I learned that writing error-free code is not enough to meet current challenges. While building a pipeline for 24 years of press freedom data, I discovered that even the most evidence-based assumptions can fail if they don’t account for the uncertainty of what comes next.

The data I relied on came from Reporters Without Borders (RSF). Unfortunately, I couldn’t download one clean dataset, but had to stitch together press freedom index data from 2002 to 2026

Starting in spring 2025 with {pressfreedom.data}, I had to stitch together press freedom index data from 2002 to 2024. The data are from Reporters Without Borders (RSF).

< “The World Press Freedom Index (WPFI) is an annual ranking of 180 countries compiled and published by Reporters Without Borders (RSF) since 2002 based upon the assessment of the non-governmental organization as well as surveys of professionals around the world, of the countries’ press freedom records in the previous year. It intends to reflect the degree of freedom that journalists, news organizations, and netizens have in each country, and the efforts made by authorities to respect this freedom. Wikipedia”

Three different methodological approaches

My first difficulty was that I couldn’t take the data as a single consolidated file, but had to download it year by year. After some spot tests, it was evident that the data structures had changed over the years.

• Period 1 – From 2002-2012 (2011 is missing): The index focused primarily on physical violence, censorship, and media control, including threats against journalists and restrictions on internet access (see Questionnaire for compiling the 2010 Press Freedom Index). The scoring scale was neither reproducible nor metric. The higher the score, the worse the press freedom, with 0 representing the best possible score but no upper limit for the worst situation.
• Period 2 – From 2013-2021: The methodology shifted toward measuring pluralism, media independence, and the legislative and infrastructural environments (see RSF SURVEY 2016). This era introduced a scoring scale, in which 100 represents the ideal country and 0 the worst possible scenario.
• Period 3 – From 2022 onwards: A major overhaul occurred, moving to a much more granular 118-question questionnaire. This modern approach also evaluates five key contextual components: political, legal, economic, sociocultural, and safety, with each component receiving a score from 0 to 100 (see World Press Freedom Index: Questionnaire 2022 and Methodology used for compiling the World Press Freedom Index 2025.

My wrong assumption

My first download function was based on careful spot-checking I’d done years before, when 2024 was still the latest year. I’d taken random samples from different periods — some early years for period 1, like 2002 and 2012; some middle years for period 2, like 2013 and 2021; and some recent years, like 2022 and 2024 — and checked their actual encoding with readr::guess_encoding(). Every example I tested came back as UTF-8. So I hardcoded UTF-8 as the default and moved on.

---

download_rwb_data <- function(years) {
  for (year in years) {
    url <- rsf_url_for(year)
    
    raw <- readr::read_delim(
      url,
      delim = ";",
      locale = readr::locale(encoding = "UTF-8", decimal_mark = ",")
    )

    readr::write_delim(raw, here::here("inst/extdata", glue::glue("rwb{year}.csv")))
  }
}

Listing 1: My frist (wrong) approach

---

Note 1: Where does the URL for every year come from?

The RSF Press Freedom Index files are hosted at a predictable URL pattern. For example, the 2025 data is at https://rsf.org/sites/default/files/import_classement/2025.csv. rsf_url_for(year) below stands in for the (trivial) helper that builds this URL from a given year: https://rsf.org/sites/default/files/import_classement/{year}.csv.

This felt justified: I had tested multiple years across the full span of the dataset, found them all to be UTF-8, and made a decision based on the evidence. One encoding for all years. Done.

I knew that UTF-8 has become the de facto standard for the web and modern software development. In contrast, other encoding schemes, such as ISO-8859-1 (also known as Latin-1), are largely restricted to legacy systems. It therefore seemed natural to me that RSF uses UTF-8 for all files.

It worked fine until it didn’t. Years later, in summer 2026, I resumed the work. Two new years (2025 and 2026) were published in the meantime. The moment I opened the raw CSV files, I found garbled text. There’s even a term for this: mojibake (文字化け) — Japanese for “corrupted characters.” It’s what you get when text encoded in one character set gets decoded as if it were a different one.

But when I checked my spot-sample years again, they were all still UTF-8. That’s when it hit me: I had tested 2002, 2013, 2022, and 2024. But I’d never tested 2025 and 2026. Those years didn’t exist when I did my encoding check. And RSF, without any announcement, had suddenly changed their export encoding from UTF-8 to ISO 8859-1 (Latin-1) after 22 years. The assumption I’d made by checking the data was correct — but only for the data I’d actually checked.

The error occurred silently and was very difficult to detect: the garbled text appeared only in the last two files and, in my case, because I had chosen only the column with English country names, only in some (remaining French) zone names.

First part of the solution

I realized that it was very important for reproducibility not to fiddle with the original files and to save them really as raw files. This required stopping guessing at download time entirely. What I came up with was the following code chunk:

utils::download.file(
  url,
  destfile = filename,
  mode = "wb",   # write binary -- the whole point
  quiet = TRUE
)

Listing 2: The first part of the solution

Note 2: Where do URL and filename come from?

URL and filename come from the same for-loop-over-years structure as download_rwb_data() above, e.g. url <- rsf_url_for(year) and filename <- here::here("inst/extdata", glue::glue("rwb{year}.csv")).

The essential difference from my first approach was to avoid any file conversion during the download. This meant not using any function of the {readr} package, as it forces you to make a decision by using locale() as a parameter. The coercion to choose a locale reflects the Tidyverse philosophy of explicitness over implicitness and is normally a good thing. I will need this feature anyway when working with the data (see Second part of the solution). But here was the goal: Download the files without any intervention whatsoever.

mode = "wb" copies bytes with no text-mode translation at all — no read_delim(), no locale, no decision. Whatever RSF’s server sends is exactly what lands on disk. The encoding guess moves downstream, to the moment the file is actually parsed using {readr}.

In my previous design, I baked a fixed encoding guess into the download function that wrote the silent conversion back to disk as fact. By the time I went looking at the “raw” file, it wasn’t raw anymore. Under the new design, the raw file is always, permanently, exactly what RSF sent. I can re-run, fix, or improve the detection logic forever because the ground truth was never overwritten in the first place.

Second part of the solution

I found the following second part of the solution to my encoding problem using LLMs via Posit Assistant. I have to confess that at first I didn’t fully comprehend the code chunk. I felt it introduced unnecessary complexity. But after I understood read::guess_encoding(), I appreciated the following few code lines:

detect_csv_encoding <- function(filepath) {                    # (1)
  guesses <- readr::guess_encoding(filepath)                   # (2)
  if (nrow(guesses) == 0) {                                    # (3)
    stop("Could not detect encoding for: ", filepath)          # (3)
  }

  top_guess <- guesses$encoding[1]                             # (4)

  if (grepl("^(UTF-8|US-ASCII|ASCII)$",                        # (5)
            top_guess, ignore.case = TRUE)) {                  # (5)
    return("UTF-8")                                            # (5)
  }
  if (grepl("^(ISO-8859-1|windows-1252|latin1)$",              # (6)
            top_guess, ignore.case = TRUE)) {                  # (6)
    return("ISO-8859-1")                                       # (6)
  }

  stop(                                                        # (7)
    "Unexpected encoding '", top_guess, "' detected for: ",    # (7)
    filepath, ". Expected UTF-8 or ISO-8859-1 family."         # (7)
  )
}

1. With the help of AI (Claude Sonnet 5), I designed a new function named detect_csv_encoding(). It takes one argument, filepath, which should be a string representing the path to the file you want to inspect.
2. Then the function guesses the encoding with guesses <- readr::guess_encoding(filepath). The guess_encoding() function is from the {readr} package (part of the Tidyverse). It reads the file’s raw bytes and compares them against known patterns to determine which encoding was used. It returns a data frame (specifically a tibble) containing columns for the encoding name and the confidence level (0 to 1).
3. if (nrow(guesses) == 0) stop(...) is a safety check. If {readr} cannot find any recognizable pattern at all and returns an empty data frame, the function stops with an informative error instead of silently guessing.
4. Since readr::guess_encoding() returns multiple possibilities sorted by confidence, we only care about the most likely one. top_guess <- guesses$encoding[1] extracts the string from the encoding column of the first row (the one with the highest confidence, the best guess).
5. The first grepl() explicitly checks whether the top guess is UTF-8, US-ASCII, or ASCII (all strict subsets of UTF-8; readr sometimes detects pure-ASCII text as ASCII). If so, the function returns "UTF-8".
6. The second grepl() checks whether the top guess belongs to the Western European Latin-1 family: ISO-8859-1, windows-1252, or latin1. (More details on the regex at the end of this listing.) Even though these three are slightly different, they are often interchangeable for basic Western text. By forcing them all to "ISO-8859-1", the rest of the code can rely on a single, predictable string.
7. If neither branch matched, the function stop()s with the actual detected encoding name. This is a deliberate design choice: earlier versions of this function fell through to a silent return("UTF-8") default here, which would have hidden the very problem this whole post is about. See the callout below for why this matters.

---

Explanation of the Regex lines

• grepl(…): A function that returns TRUE if a pattern is found in a string.
• ^: Matches the start of the string.
• (ISO-8859-1|windows-1252|latin1): The pipe | acts as an OR operator. It checks whether the string is exactly ISO-8859-1, windows-1252, or latin1.
• $: Matches the end of the string (ensuring no extra characters are present).
• ignore.case = TRUE: Makes the check case-insensitive (e.g., “LATIN1” would match “latin1”).

Note 3: Why only two encodings — and why fail instead of guess?

It’s tempting to think this detector should be extended to recognize every possible encoding. But readr::guess_encoding() (built on stringi/ICU) already returns a ranked, probabilistic guess across many candidate encodings — it’s not a hard classifier, and widening the regex to match more encoding names wouldn’t make the underlying guess any more certain. It would just add complexity without addressing the actual risk.

Restricting the detector’s known-good outcomes to UTF-8 and ISO-8859-1 is instead a deliberate, evidence-based decision: RSF’s files have only ever used these two encodings across more than two decades of data. Encoding that as domain knowledge is reasonable.

The part that actually mattered was the fallback. My original version treated “not confidently Latin-1” as “must be UTF-8” — silently defaulting instead of checking. That’s precisely the assumption that broke earlier in this post. The fixed version above checks for UTF-8 and ISO-8859-1 explicitly and stop()s on anything else (including an empty guess). If RSF ever introduces a third encoding, or ships a corrupted file, the function now fails loudly at the moment of detection — instead of mislabeling it and letting bad data slip downstream.

The main idea behind this code snippet is to determine the encoding scheme for each file rather than assuming a fixed one.

Demonstration: Testing the detector on real files

To see this in action, here’s a working example using actual RSF data files. The files rwb2024.csv (UTF-8) and rwb2025.csv (ISO-8859-1) demonstrate how the detector automatically identifies the encoding of each file.

readr::guess_encoding()

The first demonstration shows how ´readr::guess_encoding()` works:

file_path_24 <- here::here("posts/2026-08-12-encoding-resilience/rwb2024.csv")
file_path_25 <- here::here("posts/2026-08-12-encoding-resilience/rwb2025.csv")

knitr::kable(readr::guess_encoding(file_path_24))
knitr::kable(readr::guess_encoding(file_path_25))

Table 1: Guessing the type of encoding

(a) rwb2024.csv (UTF-8)

encoding	confidence
UTF-8	1.00
windows-1252	0.31

(b) rwb2025.csv (ISO-8859-1)

encoding	confidence
ISO-8859-1	0.39
ISO-8859-2	0.22

detect_csv_encoding()

The second demonstrations puts readr::guess_encoding() in the context of detect_csv_encoding() function.

detect_csv_encoding <- function(filepath) {
  guesses <- readr::guess_encoding(filepath)
  if (nrow(guesses) == 0) {
    stop("Could not detect encoding for: ", filepath)
  }

  top_guess <- guesses$encoding[1]

  if (grepl("^(UTF-8|US-ASCII|ASCII)$", top_guess, ignore.case = TRUE)) {
    return("UTF-8")
  }
  if (grepl("^(ISO-8859-1|windows-1252|latin1)$", top_guess, ignore.case = TRUE)) {
    return("ISO-8859-1")
  }

  stop(
    "Unexpected encoding '", top_guess, "' detected for: ", filepath,
    ". Expected UTF-8 or ISO-8859-1 family."
  )
}

detect_csv_encoding(file_path_24)
detect_csv_encoding(file_path_25)

[1] "UTF-8"
[1] "ISO-8859-1"

The results of detect_csv_encoding() demonstrate the key benefit: the same function, applied to files with different encodings, correctly identifies each one without needing to hard-code a single assumption. RSF changed their encoding between 2024 and 2025 (from UTF-8 to ISO-8859-1), and the detector handled it automatically.

Conclusion

Testing what you have is not the same as testing what might arrive in the future. Evidence-based assumptions are solid when they’re checked against current data, but they’re incomplete when that data is constantly growing. My spot-check approach had worked perfectly for every year I tested, but RSF changed their encoding after I’d finished testing, so the assumption remained incomplete. That was the moment I realized: design for re-checkability, not just correctness. Keep ground truth sacred by avoiding encoding decisions at download time, push detection to the parsing stage, and make every decision verifiable per file, not assumed once and trusted forever.