Main Process
This section describes how to run the CLDRModify passes for the mechanical cleanup before the release.
Before Starting
- Diff common/main with incoming/common/main
- Spot check for problems.
- Run the ConsoleCheckCLDR (with options below) to see if there are errors.
Successive PassesYou will then run CLDRModify with different options, in multiple passes.
- With each step, sanity check the results (as described below), fix any problems, tag, and check in as described below.
- This sanity check is important, since the regularizing often reveals problems in the original. For example, a date format like MMM'-yy regularizes to MMM-'yy' -- but the original was clearly an error.
- If you need to do a single file over again (eg resolving conflicts), use the -m option on CLDRModify, as described below.
After passes- Search "en.xml" and "root.xml" to verify that there are zero occurrences of "draft=". If not true, circulate a proposal to the cldr list for resolution.
- After you are all done, run ConsoleCheckCLDR once more, to make sure you didn't introduce any new errors.
Details
For the purpose of this document, we'll assume you are generating into {cldrdata}/dropbox/gen/main/ as the target directory. Change any instance below to the directory that you actually use.
Passes
- Each pass will build into {cldrdata}/dropbox/gen/main/
- Each time you will run CLDRModify with the standard options, plus one of the listed Options bulleted below.
- After building, remember to refresh data files with F5 if you are using Eclipse, after building.
- Compare against the CVS files (see above) for sanity - it is best to diff every file and review.
- Tag existing files with "preBRS16-fp" (or something like it) so that we can backout changes.
- Then copy the gen/main files to the common/main files
- Run ConsoleCheckCLDR to verify that no errors have been introduced. Standard options, plus:
- If ok, check in (see How to check in consistently after each pass).
- Otherwise, either manually patch, or restore from CVS Head files to get back to a clean state, fix, and repeat.
Options
Standard Options: -Dfile.encoding=UTF-8 -Xmx512M -DSHOW_FILES plus your choice of source/target directories.
Options for each pass:
- First change draft="true" to draft="unconfirmed" with Search&Replace // run on ALL of common *.xml
- <none> - do it with nothing but standard options first.
- This will regularize the structure, so that's not mixed in with later changes
- If any WARNING occurs, fix it manually (this is when two items have the same path when reading in the file).
- -fp : do the input processing (normally done by Survey Tool, but this catches manual insertions)
- -ff : fix NFC
- -f_ : remove superfluous compound language translations (like "Portuguese (Brazilian)")
- -fn - fix the number formats to be uniform
- Sanity-check this one over carefully, since sometimes a vetter will put in a localized decimal point or thousands separator instead of the neutral, pattern one!
- -fw : change problematic alt="proposed" cases.
- -fi : fix identical children (this copies up items that are the same in all child locales)
JCE: The -fi option has some significant problems and needs to be reworked. Recommend not to use it in most BRS situations.
- Optional - review the following and see if any apply. Most require code changes to work!!
-
-fx: This lets you remove paths that match a pattern. Only use this if you are making a structural change
- -fv: only do this if ConsoleCheckCLDR fails with lots of attribute errors. And if you do, you'll have to look at the code and maybe make some fixes, since it is old. Diff the results carefully!
- -fy:
remove deprecated. Only use this pass if the structure needs to be
changed. You'll need to modify the code to make the changes that you
want.
- There were also one-time fixes, that you only have to apply in special cases.
- -r minimalize data
JCE: Be SURE to diff these carefully. You may choose to keep certain data, especially in places where the stuff being minimized is part of a larger group ( such as date/time formats, available formats, or month names.
- -fw : Run again to make sure that no unconfirmed valued got orphaned during minimilization.
You will have to repeat this cycle if any outside changes are made to the data!
One-Time FixesThese should be commented out by having if (false) fixList.add(...). The code remains in case we want to adapt for future cases, but don't use them unless you fix the code to do what you want, and carefully diff the results. - -fk: this was set up to fix the skeletons so that they matched the patterns.
- -fe: fix Interindic. If you need to make changes in transliteration, you might want to modify this and run it.
- -fs: Fix the stand-alone narrow values.
- -fu: Fix the unit patterns.
- -fd: Fix dates
- -fz: Fix exemplars
- -fr: Fix references and standard
How to check in consistently after each pass
Sanity Check
- The console will list changes made, such as:
- Creating File: {cldrdata}/dropbox\gen\main\zh_Hant.xml
*Renaming old {cldrdata}/dropbox\gen\main\zh_Hant.xml
%zh_Hant_HK - Replacing: <yy'年'M'月'd'日'> by <yy年M月d日> at: //ldml/dates/calendars/calendar[@type="gregorian"]/dateFormats/dateFormatLength[@type="short"]/dateFormat[@type="standard"]/pattern[@type="standard"]
- The diff folder in the output has CompareIt! bat files for each change, or you can use CVS diff after moving to the CVS folder by doing the Copy and then checking.
Copy Files
- Copy from the target directory (eg {cldrdata}/dropbox\gen\main) to the cvs directory (eg {cldrdata}/common\main)
- skip the file log.txt, and the files _unchanged_.....xml
- Run CLDRModify again, with nothing but standard options. This will verify that there are no xml errors.
- There should be no change in the output; thus you should not see anything but _unchanged_.....xml files in the target directory.
Now ready to check in
In WinCVS do the following (on the console, do the equivalent)
- Refresh (F5), then click "Show committable only" icon (red)
- Diff the changed files (red) as a sanity check.
- Select main; Modify>Create tag> (eg pre-NFC1-4)
- Check in
- Select main; Modify>Create tag> (eg post-NFC1-4)
If someone checks in a change in the middle of one of your passes, it is generally easier to check in the rest of the changes, check out a clean copy of that file, and return the pass with only that file. The -m(uk) option can be used to restrict the pass to only uk.xml, for example. CLDRModify Configuration FileThe CLDRModify tool can be used to make changes to a number of files, based on a configuration tool. - Use the -fk option
- Either put your changes into cldr/tool/modify_config.txt, or use -k config_file to specify a different file. The path is relative to cldr/tool/
- Run CLDRModify, and as usual, diff your changes.
The current format of the configuration file is a list of lines.
action=delete; locale= sv ; value= YER ; path= //ldml/numbers/currencies/currency[@type="YER"]/symbol ;
If there is a match for the value and path and locale, then the action is taken. Only 'delete' is supported now. If the path is empty, any path matches. If the value is empty, any value matches. If the locale is *, then any locale matches.
The format may change in the future!
|