Updating Subdivision Codes

Main Process

Obtain a password for iso.org
Create a directory named cldr-private, if it doesn’t already exist. The default location is ../cldr-private, relative to the cldr directory. If you want a different location, you can specify -DCLDR_PRIVATE_DATA=... when running GenerateSubdivisions
Navigate to https://www.iso.org/obp/ui/ and enter the password
Click on My Account, then Country Codes Collection, and click on the XML button where it says “Download all country codes in XML”
Save the file in the cldr-private directory and unzip it
Open iso_country_codes.xml in that folder. Find the generated line, e.g., <country-codes generated=“2016-12-09T08:22:27.508296+01:00”>
Add that date to the folder name, e.g., 2016-12-09_iso_country_code_ALL_xml
Move that folder into cldr-private/external/iso_country_codes if not already there
Also copy the contents of the folder to cldr-private/iso_country_codes/iso_country_codes.xml (overriding current contents, if any)
~~Diff just to see what’s new.~~
- Actually, this step is too painful, because ISO doesn’t have a canonical XML format. So elements of a table come in random order… Sometimes
  - <subdivision-code footnote=”*”>AZ-ORD</subdivision-code>
  - <subdivision-code footnote=”*”>AZ-SAD</subdivision-code>
- And sometimes the reverse!
- May add diffs generation to GenerateSubdivisions…
Run GenerateSubdivisions
- cd cldr
- mvn --file=tools/pom.xml -pl cldr-rdf compile -DskipTests=true exec:java -Dexec.mainClass=org.unicode.cldr.tool.GenerateSubdivisions
It will create a number of files in ../Generated (see CLDRPaths.GEN_DIRECTORY). The important ones are:
- ../Generated/cldr/subdivision/subdivisions.xml
- ../Generated/cldr/subdivision/subdivisionAliases.txt
- ../Generated/cldr/subdivision/en.xml
Run diff common/supplemental/subdivisions.xml ../Generated/cldr/subdivision/subdivisions.xml
1. If they are not different (other than date/version/revision), skip to “Open … supplementalMetadata.xml” below
2. Copy the generated contents into the cldr file, and save
3. Make sure the added IDs make sense
4. Verify that we NEVER remove an ID. See #8735
  1. An ID may be deprecated; in that case it should show up in subdivisionAliases.txt if there is a good substitute
  2. We may need to add a 4-letter code in case ISO messes up
  3. In either of these cases, change GenerateSubdivisions.java to do the right thing
5. Save the Diffs, since they are useful for updating aliases. See example at end. [This instruction needs clarification: save the diffs how and where?]
Open common/supplemental/supplementalMetadata.xml
1. Search for 
2. Replace subsequent lines, up to and including the line before , with the contents of ../Generated/cldr/subdivision/subdivisionAliases.txt
3. Do a diff with the last release version. The new file should preserve the old aliases.
  1. Note: there is a tool problem where some lines are duplicated. For now, check and fix them.
  2. If a line is duplicated, when you run the tests they will show as errors.
  3. Make sure the changes make sense.
  4. IN PARTICULAR, make sure that NO former types (in uncommented lines) disappear! That is, restore any such lines before committing.) Put them below the line:
    - 
  5. (Ideally the tool would do that, but we’re not quite there.)
4. Use the names to add more aliases. (See Fixing). Check https://www.iso.org/obp/ui/#iso:code:3166:TW (replacing TW by the country code) to see notes there.
Copy the generated en.xml into the cldr repo common/subdivisions
- cp ../Generated/cldr/subdivision/en.xml common/subdivisions/
- You’ll overwrite the one there. The new one reuses all the old names where they exist
- Do a diff with the last release (typically git diff common/subdivisions/en.xml)
- Make sure the added names (from ISO) are consistent
- Verify that we NEVER remove an ID. (The deprecated ones move down, but don’t disappear)
Run the Update Validity XML steps to produce a new common/validity/subdivision.xml
- Don’t bother with the others, but diff and update that one
- A code may move to deprecated, but it should never disappear. If you find that, then revisit the step “Open … supplementalMetadata.xml” above
- The instructions for Update Validity XML mention running TestValidity; note that running all tests will automatically include running TestValidity
Run the tests
1. Fix any collisions in English
2. Google various combinations like [country code <first> <second>] to find articles like ISO_3166-2:UG, then make a fix
3. Often a sub-subdivision has the same name as a subdivision. When that is the case add a qualifier to the lesser know one, like “City” or “District”
4. Sometimes a name will change in ISO to correct a mistake, which can cause a collision
Fix the ?? in supplemental data (where possible; see below)

Fixing ??

If there are not known new subdivisions that the old ones should map to, you’ll see commented-out lines in supplementalMetadata like:
-
As many of these as possible, see if there is a mapping to one or more new subdivisions. That is, where possible, track down the best code(s) to map all of these to, and uncomment the line, and move BELOW 
- Note that for the name comment, change <!- - to <!--
First step is to go to https://www.iso.org/obp/ui/#iso:code:3166:CZ (replacing CZ by the country code in question).
1. Look at the change history at the bottom.
Other sources, in case #3 doesn’t pan out.
1. Go to Wikipedia for the country, and looking for “administrative divisions”. That can lead you to sections like: https://en.wikipedia.org/wiki/Local_government_in_Northern_Ireland#History, to track down what happened to the old subdivisions.
2. In some cases it’s very clear; a new major subdivision has the same contents as an old one, so it is just a rename. The following describes a process for handling that.

Old data

<subgroup type=“CZ” subtype=“JC” contains=“311 312 313 314 315 316 317”/>

<subgroup type=“CZ” subtype=“KA” contains=“411 412 413”/>

…

New data

<subgroup type=“CZ” subtype=“31” contains=“311 312 313 314 315 316 317”/>

<subgroup type=“CZ” subtype=“41” contains=“411 412 413”/>

…

Exact matches

From this, we can see that items have been renamed. Easiest to add the type values and contains values to a spreadsheet (use regex to extract), marking with old/new. Then sort, and pick out the ones that match.

Source		old	new	contents	Mechanical
<subgroup type=”FR” subtype=”H” contains=”2A 2B”/>	FR	“H”		“2A 2B”	<subdivisionAlias type=”FR-H” replacement=”FR-COR” reason=”deprecated”/>
<subgroup type=”FR” subtype=”COR” contains=”2A 2B”/>	FR		“COR”	“2A 2B”

Partial Matches

Rearrange the leftovers to see if there is any OLD => NEW1+NEW2... cases or OLD1 = NEW, OLD2=NEW cases. For example, for FR we get Q=>NOR and P=>NOR. Remember that these are “best fit”, so there may be small discrepancies.

Source		old	new	contents	Mechanical
Source		old	new	contents	Mechanical	Fixed ?? cases
<subgroup type=”FR” subtype=”Q” contains=”27 76”/>	FR	“Q”		“27 76”	<subdivisionAlias type=”FR-Q” replacement=”FR-?? reason=”deprecated”/>	<subdivisionAlias type=”FR-Q” replacement=”FR-NOR” reason=”deprecated”/>
<subgroup type=”FR” subtype=”P” contains=”14 50 61”/>	FR	“P”		“14 50 61”	<subdivisionAlias type=”FR-P” replacement=”FR-?? reason=”deprecated”/>	<subdivisionAlias type=”FR-P” replacement=”FR-NOR” reason=”deprecated”/>
<subgroup type=”FR” subtype=”NOR” contains=”14 27 50 61 76”/>	FR		“NOR”	“14 27 50 61 76”