Immich Traditional Chinese Geodata, Part 2: The Data Pipeline

The previous post in this series, How Reverse Geocoding Works, took apart the way Immich reads geographic data: a handful of plain text files get imported into PostgreSQL at startup, and a nearest-neighbour query resolves a place name when a photo is uploaded. Since swapping the files is enough to swap what gets displayed, the remaining question is about those “better files” themselves.

This post takes apart the immich-geodata-zh-tw pipeline, from each country’s official map data all the way to the release.tar.gz that users download.

Two Tracks

#

The whole pipeline splits into two tracks, each with its own job:

• extract: turns one country’s official map data into an intermediate CSV. It runs once per country that has dedicated processing logic, and each run is independent.
• release: merges every intermediate CSV back into the GeoNames data, translates it, prunes redundant points, and packs a release. Seven stages run in order.

extract only serves countries with a dedicated handler, and qualifying takes two things at once: usable official administrative-boundary data for that country, plus processing logic written in the project to read it.

Every other country skips extract entirely. But “no extract” does not mean “nothing happened”. The published data actually comes in three levels of accuracy, and the difference is how many processing steps a given country went through.

The most accurate level covers the five regions that go through extract. Administrative names and representative point coordinates are both rebuilt from official map data, and point density rises along with it (Taiwan goes all the way down to the village level). The price is that the country needs usable official data, and someone has to write a handler for its particular format.

The middle level covers countries that go through the locationiq lookup. Coordinates stay exactly as GeoNames has them, but every point is sent through one reverse lookup, and the administrative levels that come back overwrite the original fields. Besides picking up Chinese place-name translations, this level also fixes cases where the point sits in the right spot but upstream assigned it to the wrong administrative unit, or where the name itself is simply wrong. GeoNames is a global database, so this kind of drift is not rare. To reach this level you have to name the country with --country-code at run time and supply an API key.

The baseline level covers everything else. A country that is never named stops here: administrative assignment and point density stay exactly as GeoNames has them, and only the place names get swapped for Chinese translations during the translate stage.

Put differently, more accurate data costs more processing. Do nothing and you get the baseline. Spend API quota on reverse lookups and you can correct upstream administrative errors. Go one level higher and you have to find that country’s official map data and write dedicated processing logic for it.

When release runs, it checks whether data/handler/ holds an intermediate CSV for a given country and picks its path accordingly. The output of extract is committed to version control, and official map data rarely changes, so a release only needs to read the CSVs that are already there instead of reprocessing every country’s map data each time.

Track One: extract

#

The input is a country’s official map data; the output is an intermediate CSV in a fixed format:

cargo run --release -- extract --country TW \
  --shapefile <path to NLSC village boundary data>

Each row of the output CSV is one administrative unit, carrying the name at every level plus a computed representative coordinate. Taiwan, for example, produces 7,986 rows, one per village. That makes it possible to merge the CSV back into the GeoNames cities500.txt during release, replacing the original points and administrative fields.

All the country-specific differences are concentrated in this step. Taiwan reads three fields straight out of the NLSC village boundary data. Japan has to distinguish ordinary cities, government-designated cities, and towns and villages under a district. South Korea has to pull Hanja spellings from the Korean Wikipedia. Thailand and Indonesia both need Wikidata queries and P131 membership validation. Those pieces of logic are the subject of later posts in this series, so treat them as a black box for now.

Coordinate computation, by contrast, is shared: project into a coordinate system suited to that country, compute the geometric centre there, then convert back to WGS84. The reason is that computing a centre directly in latitude and longitude distorts the result, which the Taiwan post covers in detail.

What a Handler Actually Owns

#

The countries differ a lot, but internally extract is a fixed pipeline, and each country’s handler only supplies logic at a few specific points along it:

• load_context: loads the lookup tables and caches that country needs. Taiwan needs none; Thailand and Indonesia have to set up the environment for Wikidata queries.
• apply_country_centroids: projects according to the centroid_pipeline the country declares, then computes the centre. Taiwan uses a fixed EPSG:3826, Japan and South Korea use dynamic UTM, and Thailand and Indonesia each use their own Albers projection.
• rows_from_features: field mapping and name resolution. Taiwan only reads COUNTYNAME, TOWNNAME, and VILLNAME.

Everything else is shared across all countries: reading the file, parsing features, sorting, rounding coordinates, and writing the CSV with its uniform columns. There is also one conditional stage, split_parts, which splits a multipart geometry into one row per part, and only Indonesia enables it today.

Which countries have a handler is recorded in the Country enum. Country::ALL is the single source of truth, the CLI list is derived from it, and adding a country neither requires nor permits keeping a second copy in sync.

For a look at what a complete handler looks like end to end, the Taiwan post walks through one from start to finish.

Track Two: The Six Stages of release

#

cargo run --release -- release \
  --locationiq-api-key "YOUR_API_KEY" \
  --country-code "US"

That single command runs six stages in order:

Every stage can run on its own, and every stage can be skipped with --pass-<stage>.

cleanup: The Starting Point for Idempotency

#

This step does one simple thing: it wipes and rebuilds the output/ directory so that every run starts from a clean state.

prepare: Fetching the Raw Material

#

Three source files come from GeoNames: cities500.txt (settlements and administrative seats with a population of 500 or more, roughly 200,000 entries), admin1CodesASCII.txt (first-level administrative codes mapped to names), and admin2Codes.txt (second-level units, which this project does not process and only keeps so the file structure stays complete).

Files that already exist are skipped by default rather than downloaded again. cities500.txt runs to several hundred MB once decompressed, so that cache is very noticeable when you rerun the pipeline.

enhance: The Core Stage

#

This step folds two jobs together:

1. Merging country data: it reads the intermediate CSVs under meta_data/ and writes data from countries with dedicated processing logic (a handler) into admin1CodesASCII.txt and cities500.txt, replacing the original GeoNames points.
2. Allocating IDs: newly added rows must not collide with existing geoname_id values.

In practice, Immich never validates a geoname_id and never stores one; it only stores the resolved place name. Internally, though, a city has to pair with its corresponding admin 1 entry during lookup, so IDs still have to stay unique and correctly paired.

To get that right, the program first computes the global maximum ID present in the data, then allocates from that maximum plus one onward: admin1CodesASCII.txt takes a block first, and cities500.txt continues from there. The benefit of not hard-coding a number range is that when GeoNames expands its data later, the new rows still will not overwrite any official existing points.

The output is cities500_optimized.txt and admin1CodesASCII_optimized.txt.

locationiq: Correcting Administrative Units for Named Countries

#

This stage only handles countries named explicitly at run time; it is not applied to the global dataset. A country has to be listed with --country-code in the command (several are allowed), and the five countries that already have a dedicated handler are excluded automatically, because their data was finished during extract and there is no point spending API quota on it again.

Once named, every location in that country has its coordinates sent to LocationIQ for one reverse lookup, and the administrative levels that come back overwrite the original GeoNames fields. This is the middle level described earlier: coordinates stay put, but cases where upstream filed a point under the wrong administrative unit, or got the name wrong, get corrected. Countries that are not named never touch this step at all.

This is the only place in the whole pipeline that can be blocked by an external service. LocationIQ enforces a daily request limit, and a single country can have tens of thousands of locations to look up, so the pipeline is designed to be interrupted and resumed:

• Query progress is recorded in meta_data/<country code>.csv, and coordinates that have already been looked up are skipped automatically.
• When the daily limit is hit, switch API keys or rerun the same command the next day.
• Add --pass-cleanup to keep the existing intermediate artifacts in output/, saving another download and another round of preprocessing.

The “stages can be skipped individually” design mentioned earlier exists to solve exactly this.

translate: Deciding the Final Display Name

#

Countries with a dedicated handler were settled back in extract and never reach this stage. For everything else, the Chinese name might come from several places, and this stage decides which one wins.

The candidate sources, in order:

1. The name returned by the LocationIQ lookup. The previous stage sends accept-language: zh,en with each request, so the administrative names that come back may already be in Chinese.
2. A Chinese name filtered out of the GeoNames alias file. alternateNamesV2.txt, downloaded during prepare, holds aliases in many languages; the program filters the Chinese entries and ranks them zh-Hant → zh-TW → zh-HK → zh → zh-Hans → zh-CN → zh-SG, producing alternate_chinese_name.csv as a lookup table. This is an intermediate artifact regenerated on every run, not a hand-maintained list.
3. The alternatenames column that ships inside cities500.txt. This is the last Chinese source when the previous one misses. A single location often carries dozens of aliases, and the program picks out the Chinese one.

Once a candidate is in hand, the National Academy for Educational Research’s foreign place-name glossary makes the final call: with enough confidence it overwrites the existing name outright; with low confidence it only fills the gap when there was no Chinese name to begin with; and for questionable matches it keeps whatever was already there. All three outcomes are counted separately in the log, which makes it easy to check how much this layer changed before publishing.

If every source misses, the original text stays. Obscure locations can still show up in English inside Immich, and that is deliberate.

The output is cities500_translated.txt and admin1CodesASCII_translated.txt.

OpenCC Is Not a Blind Conversion Step

#

The Chinese names collected in the earlier steps may be written in Simplified Chinese, so this step tries to bring everything over to Traditional Chinese. But not every Chinese string gets converted: each one is inspected first. The program keeps both conversion directions ready, Simplified to Traditional (s2t) and Traditional to Simplified (t2s), and compares round-trip results to work out what a given string already is:

• text == t2s(text) → the string is already Simplified, so s2t is called to convert it to Traditional
• text == s2t(text) → the string is already Traditional, so it is kept untouched

The same rule applies when picking a candidate out of alternatenames: a candidate that is already Traditional is taken as is, and only Simplified candidates need converting.

The check has to come before the conversion because applying s2t unconditionally corrupts characters that were already correct. Some Simplified characters cover several meanings with one glyph while Traditional Chinese uses different characters for each meaning, so a string wrongly judged as Simplified and converted anyway ends up damaging translations that were fine already: 「里」 becomes 「裏」, 「占」 becomes 「佔」, and so on.

prune: Pruning Redundant Place Points Before Release

#

This is a critical stage in the release pipeline, located after translate and before pack.

Why Pruning Is Necessary

#

As the project integrated high-precision official map data from multiple countries, cities500.txt swelled rapidly to nearly 490,000 rows (Japan and Indonesia alone account for over 230,000 rows). This brought three heavy burdens:

1. Database Footprint: PostgreSQL’s geodata_places table and its spatial indexes expanded to over 220 MB.
2. Query Latency: Immich triggers a nearest-neighbour query for every uploaded photo. In dense regions like Japan, each query scanned an average of 242 rows within a 25 km radius, yet many of those rows were merely “different representative points within the same administrative division” — regardless of which point won, the returned administrative name was completely identical.
3. Download Size: The uncompressed plain-text file reached 64 MB.

The goal of pruning is to eliminate redundant place points while guaranteeing that 100% of reverse geocoding query results remain completely unchanged.

Keep It Unless Proven Otherwise

#

Immich’s reverse lookup relies under the hood on PostgreSQL’s earth_box(ll_to_earth_public(lat, lng), 25000). earth_box creates an axis-aligned cube in geocentric Cartesian coordinates, not a spherical disc! This means a point 25.5 km due north falls outside the box, while a point offset diagonally by 28.3 km still falls inside. Consequently, any simplistic reasoning based on “nearest-neighbour replacement” fails to guarantee identical SQL query behaviour.

Therefore, the pruner adheres to a strict “keep it unless proven otherwise” principle, proving for each candidate point $p$ that: after deleting $p$, every query location that could possibly be affected still receives the exact same administrative name:

• T0 (Local Identity): All Delaunay triangulation neighbours of candidate point $p$ must belong to the exact same administrative division.
• T1 (Regional Coverage): Every location within the affected region $R_p$ remains covered within 25 km by other kept points sharing the same name.
• T2 (Boundary Consistency): When falling outside the 25 km threshold and triggering fallback to Natural Earth country names, the result remains completely identical to the original behavior.

To verify continuous regions, the program uses gnomonic projection to perform recursive quadtree subdivision. Gnomonic projection maps great circles to straight lines, ensuring that spherical polygons possess geodesic convexity so that region corner vertices can safely bound the entire area.

Furthermore, the T0 argument requires that neighbours remain alive; deleting adjacent points simultaneously invalidates the proof. Thus, a single pass can only delete a “Maximal Independent Set” of mutually non-adjacent points. After each pass, the spherical Delaunay adjacency is rebuilt and re-evaluated, requiring 20 iterative passes to reach full convergence.

Pruning Results

#

Benchmark results (subdivision budget 2048, multi-pass convergence):

• 33.5% Row Reduction: Global place points decreased from 487,372 to 324,296 (a net reduction of 163,076 points).
• Size and Index Savings: Uncompressed cities500.txt shrank by 25.2% (down 47.8 MB), PostgreSQL’s geodata_places table and indexes dropped by 25.8% (down 164 MB), and the GiST spatial index alone dropped by 46.1% (down 41 MB).
• 3x Faster Queries: Average reverse geocoding latency in dense regions dropped from 3.36 ms to 1.15 ms (a 66% drop, with p50 down to 0.63 ms).
• Zero-Diff Verification: In differential testing across 6.35 million sample coordinates against a real PostgreSQL instance checking (country, state, city), label changes were 0, and newly introduced null results were 0.
• Inherent Protection for Sparse Regions: Sparsely populated countries like Canada and Russia were preserved entirely because they could not satisfy the T1 coverage proof (out of 59 countries generating candidate points, only 6 dense countries had points deleted), requiring zero manual country whitelists.

Pruning is a default stage in the release workflow. It can be skipped with --pass-prune or run standalone via cargo run --release -- prune.

pack: Packing

#

This stage arranges the pruned and translated files, data/vendor/i18n-iso-countries/ (the country name mapping), LICENSE, and NOTICE.md into the release directory structure, writes geodata-date.txt, and finally produces release.tar.gz and release.zip.

The geodata/countryInfo.txt that Immich reads for country names is also produced in this stage (see How Reverse Geocoding Works for the background). pack takes the original GeoNames file under data/vendor/geonames/, pairs it with the Traditional Chinese names from en.json, replaces the country name in column 5, and places the result in the release’s geodata/. Every other column and the header comments are preserved byte for byte. This base file is committed to version control rather than downloaded in the prepare stage, because it rarely changes and only needs updating when a country is added, removed, or renamed.

This stage treats a missing translation the opposite way translate does. translate keeps the original text when it finds no Chinese name, and a rare place showing English is intentional. pack instead fails outright, listing the missing codes, when the base file contains a current country code that en.json lacks. Only the dissolved legacy codes AN and CS are exempt. The reason is that if GeoNames added a country and nobody noticed, that country would quietly fall back to English, and users would see mixed languages instead of a clear error.

That bundle is exactly what the update_data.sh install script downloads, and its directory structure maps directly onto where the files go inside Immich at install time. The release ships both countryInfo.txt and i18n-iso-countries: the former for Immich v3.3.0 and later, the latter for earlier versions. The script decides which one to install by checking which files actually exist in the target environment, not by looking at the version number. In testing, the v3.2.4 image already contained countryInfo.txt; it just was not read yet.

Verifying Without Calling External Services

#

The awkward part of this pipeline is that it depends on both network downloads and a paid API, so running the full thing to verify every change is out of the question. The CLI therefore offers two verification modes.

dry-run: verifies the stage orchestration of a release without downloading data or calling the API.

cargo run -- release --dry-run \
  --locationiq-api-key "fixture" \
  --country-code "KR" "TH" \
  --batch-size 100 --locationiq-qps 2

--batch-size and --locationiq-qps are throttling parameters for the locationiq stage (queries per batch, requests per second). A dry run never issues real requests; passing them is only there to make the orchestration follow the same path.

fixture mode: uses local fixed test data to produce a smoke artifact, which verifies that the release archive and the directory structure update_data.sh expects are both correct.

cargo run -- release --fixture-mode \
  --pass-locationiq \
  --output-folder /tmp/rust-release-smoke

The official release and nightly workflows both run the real pipeline, but they run the fixture release smoke first as a preflight check.

The next post, Five Regions, Five Answers, moves into the per-country logic: why one shared pipeline ends up producing five different display strategies for five regions.

References

#

• Local data processing - extract commands per country and instructions for the full pipeline
• Point Pruning Documentation - Full geometric proofs and differential testing details for the prune stage
• City-Level Selection Criteria - Adjudication rules for city-level administrative names across regions
• GeoNames Documentation - file formats of the source data
• LocationIQ Documentation - Reverse Geocoding API