Automation versus Judgment Calls

Annual updates that sometimes change the data format

I’ve been building pressfreedom.data, an R data package that collects Reporters Without Borders (RSF) press freedom data from 2002 through 2026. There is no consolidated dataset available because RSF publishes its press freedom data annually as a separate CSV file. This was a major task for me, as I explained in Encoding Resilience, and When Data Cleaning Becomes Decision-Making. It involved not only downloading raw CSVs with inconsistent column names, but I had to normalize 24 years of data into one unified structure, standardize over 200 country-name variants, and assigning ISO codes — a sequence of decisions and transformations that took me several months of iteration to get it right.

By late 2025, I had it finally working—the package compiled. The tests passed. The dataset was clean.

But in the meantime, RSF released a new dataset in June 2026. I could download, inspect, and manually clean it, as I had done before. Then I thought: RSF will publish a new dataset again in 2027. When that happens, what do I do?

The honest answer was: I’d do it all again. Manually. Re-read my own notes to remember what I’d done and why, almost certainly break something, and spend a day or two getting back to where I already was. That’s not a pipeline. That’s a recipe I’d memorized badly enough that I’d need to re-read the card every time I wanted to cook the dish.

This post is about what I tried, what didn’t fit, and what I actually built.

Automated pipeline versus manual guided procedure

Workflow process in {pressfreedom.data}

The work I’d done over those months had four natural phases:

1. Download — fetch the raw CSV from RSF’s website: See Encoding Resilience
2. Clean — normalize column names, fix data types, handle year-specific quirks See When Data Cleaning Becomes Decision-Making
3. Combine — merge all 24 yearly files into one dataset
4. Standardize — reconcile country names (See When Data Cleaning Becomes Decision-Making), assign ISO codes, validate

But on my first try, I didn’t design a structured workflow to process and prepare raw data for analysis. Instead, I ran four separate R scripts in order by hand in RStudio, keeping notes as script comments and in a text file about what I needed to do differently for each year.

That’s not an automated pipeline but a manual controlled procedure.

The distinction matters because manually guided processes are time-sensitive. I can follow a procedure I documented yesterday. Six months from now I’ll probably skip a step, misread a note, or make a judgment call that quietly contradicts a decision from months earlier. The code might run without error and produce subtly wrong data, and I won’t notice until something downstream looks off.

I needed to turn the manually guided procedure into a function — something I could call once that would do the right thing without relying on my memory and dispersed notes of what “right” was.

Reaching for {targets} — and what I learned from it

The first tool I seriously considered was {targets}, an R package for workflow management. The idea is straightforward: you describe your analysis as a collection of targets — each one a named function call with a declared input. targets figures out which targets are out of date and reruns only what’s needed. A target is cached when it succeeds; it only reruns if its upstream inputs change.

Here’s what a minimal targets pipeline for my problem would have looked like:

R Code 1 : Minimal targets pipeline for pressfreedom.data

# _targets.R
library(targets)

list(
  tar_target(raw_2026, download_rwb_csv(2026)),
  tar_target(clean_2026, clean_period_3(raw_2026, year = 2026)),
  tar_target(combined,   combine_cleaned_periods()),
  tar_target(final,      standardize_rwb_countries(combined))
)

With this in place, running targets::tar_make() would execute only the targets that have changed. If I’d already downloaded and cleaned the 2026 data, re-running would skip those steps and jump straight to combining and standardizing. The pipeline is self-documenting: the dependency graph is explicit in the code.

And that dependency graph — a directed acyclic graph (DAG) — is exactly what targets is designed to manage. A DAG is a flowchart where each node is a step and arrows show what feeds into what, with no cycles (nothing feeds back into itself). The tool shines when a pipeline has genuine branching: parallel inputs that converge, shared intermediates used by multiple downstream steps, or conditional paths depending on what’s available.

graph LR
    A[raw_2026.csv] --> C[clean_2026]
    B[raw_2025.csv] --> D[clean_2025]
    C --> E[combined]
    D --> E
    E --> F[standardized]
    E --> G[summary_report]
    F --> H[package_data]

Figure 1: What a real DAG looks like: two input streams, shared intermediate, branching output

I spent some time learning targets before concluding it wasn’t the right fit for my problem. That time wasn’t wasted — I now have a much clearer picture of when targets is the right choice.

NoteWhen targets is the right choice

targets pays off when your pipeline has one or more of these properties:

• Branching computation — multiple input sources that converge on shared intermediates, or one intermediate used by several downstream steps
• Expensive steps — if fitting a model takes 40 minutes, you want caching
• High iteration rate — running the pipeline daily or per code change, where skipping unchanged targets saves meaningful time
• Multiple contributors — the dependency graph serves as documentation for a team, not just yourself
• Reproducibility requirements — the graph makes repetitions replicable

For my linear pipeline, which runs in less than 30 seconds and only once a year, these benefits do not apply.

My pipeline turns out not to have a graph. It has a sequence:

graph LR
    A[detect] --> B[download] --> C[clean] --> D[combine] --> E[standardize]

Figure 2: My actual pipeline: a strict linear sequence with no branching

No branching. No parallel inputs. No shared intermediates used by multiple steps.

The entire run takes less than 30 seconds, and I will use it once a year. Introducing targets would have added complexity — a _targets.R file, a targets store, a different execution model — without any of the benefits that justify that complexity.

The other temptation: automate everything

Having ruled out targets, I briefly considered the opposite approach: writing a script that handled every decision automatically. Detect when RSF published new data, download it, clean it, push the commit.

I discarded this version fast. RSF has changed their column names and methodology several times. A fully automated pipeline would be dangerous, either breaking silently on those changes or hiding them behind a “success” status that wasn’t actually warranted.

ImportantGood automation removes busywork, but it shouldn’t remove judgment calls.

What I actually built

The function I ended up with is update_rwb_data(). Its job: detect which years of RSF data are missing from the package, download them, clean them, recombine everything, and restandardize. What follows is a mockup of how my function for the yearly update works:

R Code 2 : RWB Data Update Report (mockup)

# Run once per year when RSF publishes new rankings
result <- update_rwb_data(auto_commit = FALSE)
print(result)

=== RWB Data Update Report ===

Status: SUCCESS
Years downloaded: 2026
Years cleaned: 2026
Rows before: 4,192
Rows after: 4,382
Row increase: 190
Consolidation rules applied: 246
Validation passed: true
Git commit: (skipped)

Messages:
  - Detected 1 missing year(s): 2026
  - Downloaded rwb2026.csv (190 rows)
  - Cleaned year 2026 (period 3)
  - Combined all periods: 4,382 rows
  - Standardization complete: 191 unique countries
  - All validation checks passed

That report is the function’s way of saying: here is what I did, here is what changed, here is whether the result looks plausible. One call per year.

To show what the pipeline actually produces — the dataset itself — here’s what rwb_standardized looks like after all four phases have run:

R Code 3 : The dataset that the pipeline produces

library(pressfreedom.data)
dplyr::glimpse(rwb_standardized)

Rows: 4,183
Columns: 20
$ year_n            <dbl> 2002, 2002, 2002, 2002, 2002, 2002, 2002, 2002, 2002…
$ iso               <chr> "AFG", "DZA", "AGO", "ARG", "AUS", "AUT", "AZE", "BH…
$ score             <dbl> 35.50, 31.00, 30.17, 12.00, 3.50, 7.50, 34.50, 23.00…
$ rank              <dbl> 104, 95, 93, 42, 12, 27, 101, 67, 118, 124, 13, 21, …
$ political_context <dbl> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, …
$ rank_pol          <dbl> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, …
$ economic_context  <dbl> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, …
$ rank_eco          <dbl> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, …
$ legal_context     <dbl> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, …
$ rank_leg          <dbl> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, …
$ social_context    <dbl> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, …
$ rank_soc          <dbl> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, …
$ safety            <dbl> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, …
$ rank_saf          <dbl> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, …
$ zone              <chr> "Asia-Pacific", "Middle East & North Africa", "Afric…
$ rank_n_1          <dbl> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, …
$ rank_evolution    <dbl> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, …
$ score_n_1         <dbl> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, …
$ score_evolution   <dbl> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, …
$ country_en        <chr> "Afghanistan", "Algeria", "Angola", "Argentina", "Au…

That dataset is the output of the whole pipeline — 24 years of press freedom rankings, normalized, combined, standardized. Everything in the update_rwb_data() function exists to keep that dataset up to date without having to recreate it from scratch each time.

How the five phases fit together

The five internal phases of update_rwb_data()

1. Detect — check which years are missing from data/cleaned/; abort early if nothing is missing
2. Download — fetch the new year’s CSV from RSF’s server; validate that it has the expected structure before writing it to disk
3. Clean — normalize column names, convert data types, handle year-specific quirks; applies per-year override rules for cases where RSF renamed a column
4. Combine — merge all cleaned years into one dataset; recalculate year-over-year evolution columns (which depend on the full history, not just the new year)
5. Standardize — apply the country-consolidation mapping, assign ISO codes, validate the result

Phases 4 and 5 always run, even if no new year was added. They take about three seconds and they’re fast enough that skipping them to save time would be a false economy — and risky, since their outputs depend on having consistent inputs.

Asymmetric error handling

The design that took longest to get right wasn’t the happy path — it was what happens when something goes wrong. Here the LLM helped me a lot.

CautionNot all failures should behave the same way

The function uses two different failure modes, deliberately:

• Hard abort (download or cleaning failure): the pipeline stops immediately with an informative error message. A partial update — a new year downloaded but not cleaned, or cleaned but not combined — is worse than no update, because it looks complete when it isn’t.

• Warning and continue (validation failure): if the row count is slightly off or a check produces an unexpected result, the function reports it and keeps going. Validation failures often mean “something unexpected happened, please look at this” — not “the data is corrupt, stop everything.” I want the output so I can inspect it, not a locked door.

The distinguishing question: can I trust the intermediate output if I let this continue? If yes, warn. If no, abort.

This distinction didn’t come to me immediately. My first instinct was to abort on anything unexpected. It seemed to me the safest option. Working through specific cases with the AI assistant made me realize that the word “unexpected” covers a wide range. A missing column in the raw CSV is a hard error. A row count that’s slightly higher or lower is not a reason to block the entire update procedure. It is important to know about this so that I can look it up and find out what the reason for this change is. It might be that RSF quietly dropped a territory or they added a new country. This is worth knowing about and evaluating, but it is not worth blocking the whole update process.

The error messages themselves are part of the design. When a column name has changed — as RSF has done sometimes without notice in past years — the abort message names the year, the column it expected, and what it actually found. That takes a few lines to write and saves an hour of debugging when it triggers six months from now.

How the conversation shaped the design

The design of update_rwb_data() came partly from working through it with an AI assistant — not because the AI generated the solution, but because the conversation forced me to articulate decisions I’d been leaving implicit.

The asymmetric error handling popped up during the conversation with the AI assistant. At first I wanted to abort on anything unexpected. But the LLM walked me through specific cases: what if RSF adds a new territory? What if a country’s score is missing for one year? I realized: Some of those are errors; some are expected variation that I want to be told about. Spelling out the cases made the distinction visible.

The function interface — a single function with named logical parameters — also emerged from the conversation. My first sketches had separate scripts for each phase. The AI asked what I’d want to do if the download succeeded but the cleaning failed. That question made clear I needed a single entry point with shared state, not four independent scripts with no coordination between them.

Lesson learned

I built update_rwb_data() for one specific package, {pressfreedom.data}, but in retrospect I believe the reasoning behind it applies to other multi-step data pipelines that run on a fixed schedule:

• Don’t automate before you understand the failure modes. The automation design came after understanding where each step could break, not before.
• Match the tool to the problem. targets is excellent for what it does; my pipeline didn’t have the properties that justify it. Spending time with the wrong tool isn’t wasted if it teaches you why it’s wrong.
• Leave room for judgment. The hardest part of designing automation is identifying which decisions can be scripted, and which need a human. If “who decides this is correct?” has no clear answer, it probably shouldn’t be automated.
• Error messages are part of the interface. This was one of the most impressive learnings for me. I haven’t used so many distinctive (failure) messages so far. Either I tried to fix the problem behind the scenes, in a way imperceptible to the user, or I reported a generic failure message, such as “cleaning failed”. A failure message that names the year, the expected column, and what it found is much better — especially when you’re reading it six months after you wrote the code.