[apple/icu.git] / icuSources / data / cldr-icu-readme.txt

# Copyright (C) 2010, International Business Machines Corporation and others.
# All Rights Reserved.                  
#
# Commands for regenerating ICU4C locale data (.txt files) from CLDR.
#
# The process requires local copies of
#    - CLDR (the source of most of the data, and some Java tools)
#    - ICU4J  (used by the conversion tools)
#    - ICU4C  (the destination for the new data, and the source for some of it)
#
# For an official CLDR data integration into ICU, these should be clean, freshly
# checked-out. For released CLDR sources, an alternative to checking out sources
# for a given version is downloading the zipped sources for the common (core.zip)
# and tools (tools.zip) directory subtrees from the Data column in
# [http://cldr.unicode.org/index/downloads].
#
# The versions of each of these must match. Included with the release notes for
# ICU is the version number and/or a CLDR svn tag name for the revision of CLDR
# that was the source of the data for that release of ICU.
#
# Note: Some versions of the OpenJDK will not build the CLDR java utilities.
#   If you see compilation errors complaining about type incompatibilities with
#   functions on generic classes, try switching to the Sun JDK.
#
# Besides a standard JDK, the process also requires ant
# (http://ant.apache.org/),
# plus the xml-apis.jar from the Apache xalan package
# (http://xml.apache.org/xalan-j/downloads.html).
#
# Note: Enough things can (and will) fail in this process that it is best to
#   run the commands separately from an interactive shell. They should all
#   copy and paste without problems.
#
# It is often useful to save logs of the output of many of the steps in this
# process. The commands below put log files in /tmp; you may want to put them
# somewhere else.
#
#----
#
# There are several environment variables that need to be defined.
#
# a) Java- and ant-related variables
#
# JAVA_HOME:     Path to JDK (a directory, containing e.g. bin/java, bin/javac,
#                etc.); on many systems this can be set using
#                `/usr/libexec/java_home`.
#
# XML_APIS_JAR:  Complete path to xml-apis.jar (at the end of the path)
#
# ANT_OPTS:      Two important options you may want to set:
#
#                -Xmx1024m, to give Java more memory; otherwise it may run out
#                 of heap.
#
#                -DCLDR_DTD_CACHE=/tmp/cldrdtd, to set up a temp directory for
#                 caching the CLDR xml dtd, to speed up data generation.
#
# b) CLDR-related variables
#
# CLDR_DIR:      Path to root of CLDR sources, below which are the common and
#                tools directories. Two other variables will be defined based
#                on this:
# CLDR_MAIN, CLDR_JAR are defined relative to CLDR_DIR.
#
# c) ICU-related variables:
#
# ICU4C_DIR:     Path to root of ICU4C sources, below which is the source dir.
#
# ICU4J_ROOT:    Path to root of ICU4J sources, below which is the main dir.
#                Two other variables will be defined based on this:
# ICU4J_JAR, UTILITIES_JAR are defined relative to ICU4J_ROOT.
#
#----
#
# If you are adding or removing locales, or specific kinds of locale data,
# there are some xml files in the ICU sources that need to be updated (these xml
# files are used in addition to the CLDR files as inputs to the CLDR data build
# process for ICU):
#
#    icu/trunk/source/data/icu-config.xml - Update <locales> to add or remove
#                CLDR locales for inclusion in ICU. Update <paths> to prefer
#                alt forms for certain paths, or to exclude certain paths; note
#                that <paths> items can only have draft or alt attributes.
#
#    icu/trunk/source/data/build.xml - If you are adding or removing break
#                iterators, you need to update  <fileset id="brkitr" ...> under
#                <target name="clean" ...> to clean the correct set of files.
#
#    icu/trunk/source/data/xml/      - If you are adding a new locale, break
#                iterator, collattion tailoring, or rule-based number formatter,
#                you need to add a corresponding xml file in (respectively) the
#                main/, brkitr/, collation/, or rbnf/ subdirectory here.
#
#----
#
# For an official CLDR data integration into ICU, there are some additional
# considerations:
#
# a) Don't commit anything in ICU sources (and possibly any changes in CLDR
#    sources, depending on their nature) until you have finished testing and
#    resolving build issues and test failures for both ICU4C and ICU4J.
#
# b) There are version numbers that may need manual updating in CLDR (other
#    version numbers get updated automatically, based on these):
#
#    common/dtd/ldml.dtd                            - update cldrVersion
#    common/dtd/ldmlBCP47.dtd                       - update cldrVersion
#    common/dtd/ldmlSupplemental.dtd                - update cldrVersion
#    tools/java/org/unicode/cldr/util/CLDRFile.java - update GEN_VERSION
#
# c) After everything is committed, you will need to tag the CLDR, ICU4J, and
#    ICU4C sources that ended up being used for the integration; see step 17
#    below.
#
################################################################################

# 1a. Java and ant variables, adjust for your system
# (here XML_APIS_JAR is set as on Mac OSX).

export JAVA_HOME=`/usr/libexec/java_home`
export XML_APIS_JAR=/Library/Java/Extensions/xml-apis.jar
mkdir /tmp/cldrdtd
export ANT_OPTS="-Xmx1024m -DCLDR_DTD_CACHE=/tmp/cldrdtd"

# 1b. CLDR variables, adjust for your setup; with cygwin it might be e.g.
# CLDR_DIR=`cygpath -wp /build/cldr`

export CLDR_DIR=$HOME/cldr/trunk
export CLDR_MAIN=$CLDR_DIR/common/main
export CLDR_JAR=$CLDR_DIR/tools/java/cldr.jar
#export CLDR_CLASSES=$CLDR_DIR/tools/java/classes

# 1c. ICU variables

export ICU4C_DIR=$HOME/icu/icu/trunk
export ICU4J_ROOT=$HOME/icu/icu4j/trunk
export ICU4J_JAR=$ICU4J_ROOT/icu4j.jar
export UTILITIES_JAR=$ICU4J_ROOT/out/cldr_util/lib/utilities.jar
#export ICU4J_CLASSES=$ICU4J_ROOT/out/cldr_util/bin

# 2. Build ICU4J, including the cldr utilities.

cd $ICU4J_ROOT
ant clean all jar cldrUtil

# 3. Build the CLDR Java tools

cd $CLDR_DIR/tools/java
ant jar

# 4. Configure ICU4C, build and test without new data first, to verify that
# there are no pre-existing errors (configure shown here for MacOSX, adjust
# for your platform).

cd $ICU4C_DIR/source
./runConfigureICU MacOSX
make all 2>&1 | tee /tmp/icu4c-oldData-makeAll.txt
make check 2>&1 | tee /tmp/icu4c-oldData-makeCheck.txt

# 5. Build the new ICU4C data files; these include .txt files and .mk files.
# These new files will replace whatever was already present in the ICU4C sources.
# This process uses ant with ICU's data/build.xml and data/icu-config.xml to
# operate (via CLDR's ant/CLDRConverterTool.java and ant/CLDRBuild.java) the
# necessary CLDR tools including LDML2ICUConverter, ConvertTransforms, etc.
# This process will take several minutes (CLDR_DTD_CACHE helps reduce this).
# Keep a log so you can investigate anything that looks suspicious.

cd $ICU4C_DIR/source/data
ant clean
ant all 2>&1 | tee /tmp/cldrNN-buildLog.txt

# 6. Check which data files have modifications, which have been added or removed
# (if there are no changes, you may not need to proceed further). Make sure the
# list seems reasonable.

svn status

# 7. Fix any errors, investigate any warnings. Some warnings are expected,
# including  warnings for missing versions in locale names which specify some
# collationvariants, e.g.
#   [cldr-build] WARNING (ja_JP_TRADITIONAL): No version #??
#   [cldr-build] WARNING (zh_TW_STROKE): No version #??
# and warnings for some empty collation bundles, e.g.
#   [cldr-build] WARNING (en):  warning: No collations found. Bundle will ...
#   [cldr-build] WARNING (to):  warning: No collations found. Bundle will ...
#
# Fixing may entail modifying CLDR source data or tools - for example,
# updating the validSubLocales for collation data (file a bug f appropriate).
# Repeat steps 5-7 until there are no build errors and no unexpected
# warnings.

# 8. Now rebuild ICU4C with the new data and run make check tests.
# Again, keep a log so you can investigate the errors.

cd $ICU4C_DIR/source
make check 2>&1 | tee /tmp/icu4c-newData-makeCheck.txt

# 9. Investigate each test case failure. The first run processing new CLDR data
# from the Survey Tool can result in thousands of failures (in many cases, one
# CLDR data fix can resolve hundres of test failures). If the error is caused
# by bad CLDR data, then file a CLDR bug, fix the data, and regenerate from
# step 5a. If the data is OK but the testcase needs to be updated because the
# data has legitimately changed, then update the testcase. You will check in
# the updated testcases along with the new ICU data at the end of this process.
# Repeat steps 5-8 until there are no errors.

# 10. Now run the make check tests in exhaustive mode:

cd $ICU4C_DIR/source
export INTLTEST_OPTS="-e"
export CINTLTST_OPTS="-e"
make check 2>&1 | tee /tmp/icu4c-newData-makeCheckEx.txt

# 11. Again, investigate each failure, fixing CLDR data or ICU test cases as
# appropriate, and repeating steps 5-7 and 10 until there are no errors.

# 12. Now with ICU4J, build and test without new data first, to verify that
# there are no pre-existing errors (or at least to have the pre-existing errors
# as a base for comparison):

cd $ICU4J_ROOT
ant all 2>&1 | tee /tmp/icu4j-oldData-antAll.txt
ant check 2>&1 | tee /tmp/icu4j-oldData-antCheck.txt

# 13. Now build the new data for ICU4J

cd $ICU4C_DIR/source/data
make icu4j-data-install

# 14. Now rebuild ICU4J with the new data and run tests:
# Keep a log so you can investigate the errors.

cd $ICU4J_ROOT
ant check 2>&1 | tee /tmp/icu4j-newData-antCheck.txt

# 15. Investigate test case failures; fix test cases and repeat from step 14,
# or fix CLDR data and repeat from step 5, as appropriate, unitl; there are no
# more failures in ICU4C or ICU4J (except failures that were present before you
# began testing the new CLDR data).

# 16. Check the file changes; then svn add or svn remove as necessary, and
# commit the changes.

cd $ICU4C_DIR/source/
svn status
# add or remove as necessary, then commit

cd $ICU4J_ROOT
svn status
# add or remove as necessary, then commit

# 17. For an official CLDR data integration into ICU, now tag the CLDR, ICU4J,
# and ICU4C sources with an appropriate CLDR milestone (you can check previous
# tags for format), e.g.:

svn copy svn+ssh://unicode.org/repos/cldr/trunk \
svn+ssh://unicode.org/repos/cldr/tags/release-NNN \
--parents -m "cldrbug nnnn: tag cldr sources for NNN"

svn copy svn+ssh://source.icu-project.org/repos/icu/icu4j/trunk \
svn+ssh://source.icu-project.org/repos/icu/icu4j/tags/cldr-NNN \
--parents -m 'ticket:mmmm: tag the version used for integrating CLDR NNN'

svn copy svn+ssh://source.icu-project.org/repos/icu/icu/trunk \
svn+ssh://source.icu-project.org/repos/icu/icu/tags/cldr-NNN \
--parents -m 'ticket:mmmm: tag the version used for integrating CLDR NNN'
Commit	Line	Data
729e4ab9 A	1	# Copyright (C) 2010, International Business Machines Corporation and others.
	2	# All Rights Reserved.
	3	#
	4	# Commands for regenerating ICU4C locale data (.txt files) from CLDR.
	5	#
	6	# The process requires local copies of
	7	# - CLDR (the source of most of the data, and some Java tools)
	8	# - ICU4J (used by the conversion tools)
	9	# - ICU4C (the destination for the new data, and the source for some of it)
	10	#
	11	# For an official CLDR data integration into ICU, these should be clean, freshly
	12	# checked-out. For released CLDR sources, an alternative to checking out sources
	13	# for a given version is downloading the zipped sources for the common (core.zip)
	14	# and tools (tools.zip) directory subtrees from the Data column in
	15	# [http://cldr.unicode.org/index/downloads].
	16	#
	17	# The versions of each of these must match. Included with the release notes for
	18	# ICU is the version number and/or a CLDR svn tag name for the revision of CLDR
	19	# that was the source of the data for that release of ICU.
	20	#
	21	# Note: Some versions of the OpenJDK will not build the CLDR java utilities.
	22	# If you see compilation errors complaining about type incompatibilities with
	23	# functions on generic classes, try switching to the Sun JDK.
	24	#
	25	# Besides a standard JDK, the process also requires ant
	26	# (http://ant.apache.org/),
	27	# plus the xml-apis.jar from the Apache xalan package
	28	# (http://xml.apache.org/xalan-j/downloads.html).
	29	#
	30	# Note: Enough things can (and will) fail in this process that it is best to
	31	# run the commands separately from an interactive shell. They should all
	32	# copy and paste without problems.
	33	#
	34	# It is often useful to save logs of the output of many of the steps in this
	35	# process. The commands below put log files in /tmp; you may want to put them
	36	# somewhere else.
	37	#
	38	#----
	39	#
	40	# There are several environment variables that need to be defined.
	41	#
	42	# a) Java- and ant-related variables
	43	#
	44	# JAVA_HOME: Path to JDK (a directory, containing e.g. bin/java, bin/javac,
	45	# etc.); on many systems this can be set using
	46	# `/usr/libexec/java_home`.
	47	#
	48	# XML_APIS_JAR: Complete path to xml-apis.jar (at the end of the path)
	49	#
	50	# ANT_OPTS: Two important options you may want to set:
	51	#
	52	# -Xmx1024m, to give Java more memory; otherwise it may run out
	53	# of heap.
	54	#
	55	# -DCLDR_DTD_CACHE=/tmp/cldrdtd, to set up a temp directory for
	56	# caching the CLDR xml dtd, to speed up data generation.
	57	#
	58	# b) CLDR-related variables
	59	#
	60	# CLDR_DIR: Path to root of CLDR sources, below which are the common and
	61	# tools directories. Two other variables will be defined based
	62	# on this:
	63	# CLDR_MAIN, CLDR_JAR are defined relative to CLDR_DIR.
	64	#
65	# c) ICU-related variables:
66	#
67	# ICU4C_DIR: Path to root of ICU4C sources, below which is the source dir.
68	#
69	# ICU4J_ROOT: Path to root of ICU4J sources, below which is the main dir.
70	# Two other variables will be defined based on this:
71	# ICU4J_JAR, UTILITIES_JAR are defined relative to ICU4J_ROOT.
72	#
73	#----
74	#
75	# If you are adding or removing locales, or specific kinds of locale data,
76	# there are some xml files in the ICU sources that need to be updated (these xml
77	# files are used in addition to the CLDR files as inputs to the CLDR data build
78	# process for ICU):
79	#
80	# icu/trunk/source/data/icu-config.xml - Update <locales> to add or remove
81	# CLDR locales for inclusion in ICU. Update <paths> to prefer
82	# alt forms for certain paths, or to exclude certain paths; note
83	# that <paths> items can only have draft or alt attributes.
84	#
85	# icu/trunk/source/data/build.xml - If you are adding or removing break
86	# iterators, you need to update <fileset id="brkitr" ...> under
87	# <target name="clean" ...> to clean the correct set of files.
88	#
89	# icu/trunk/source/data/xml/ - If you are adding a new locale, break
90	# iterator, collattion tailoring, or rule-based number formatter,
91	# you need to add a corresponding xml file in (respectively) the
92	# main/, brkitr/, collation/, or rbnf/ subdirectory here.
93	#
94	#----
95	#
96	# For an official CLDR data integration into ICU, there are some additional
97	# considerations:
98	#
99	# a) Don't commit anything in ICU sources (and possibly any changes in CLDR
100	# sources, depending on their nature) until you have finished testing and
101	# resolving build issues and test failures for both ICU4C and ICU4J.
102	#
103	# b) There are version numbers that may need manual updating in CLDR (other
104	# version numbers get updated automatically, based on these):
105	#
106	# common/dtd/ldml.dtd - update cldrVersion
107	# common/dtd/ldmlBCP47.dtd - update cldrVersion
108	# common/dtd/ldmlSupplemental.dtd - update cldrVersion
109	# tools/java/org/unicode/cldr/util/CLDRFile.java - update GEN_VERSION
110	#
111	# c) After everything is committed, you will need to tag the CLDR, ICU4J, and
112	# ICU4C sources that ended up being used for the integration; see step 17
113	# below.
114	#
115	################################################################################
116
117	# 1a. Java and ant variables, adjust for your system
118	# (here XML_APIS_JAR is set as on Mac OSX).
119
120	export JAVA_HOME=`/usr/libexec/java_home`
121	export XML_APIS_JAR=/Library/Java/Extensions/xml-apis.jar
122	mkdir /tmp/cldrdtd
123	export ANT_OPTS="-Xmx1024m -DCLDR_DTD_CACHE=/tmp/cldrdtd"
124
125	# 1b. CLDR variables, adjust for your setup; with cygwin it might be e.g.
126	# CLDR_DIR=`cygpath -wp /build/cldr`
127
128	export CLDR_DIR=$HOME/cldr/trunk
129	export CLDR_MAIN=$CLDR_DIR/common/main
130	export CLDR_JAR=$CLDR_DIR/tools/java/cldr.jar
131	#export CLDR_CLASSES=$CLDR_DIR/tools/java/classes
132
133	# 1c. ICU variables
134
135	export ICU4C_DIR=$HOME/icu/icu/trunk
136	export ICU4J_ROOT=$HOME/icu/icu4j/trunk
137	export ICU4J_JAR=$ICU4J_ROOT/icu4j.jar
138	export UTILITIES_JAR=$ICU4J_ROOT/out/cldr_util/lib/utilities.jar
139	#export ICU4J_CLASSES=$ICU4J_ROOT/out/cldr_util/bin
140
141	# 2. Build ICU4J, including the cldr utilities.
142
143	cd $ICU4J_ROOT
144	ant clean all jar cldrUtil
145
146	# 3. Build the CLDR Java tools
147
148	cd $CLDR_DIR/tools/java
149	ant jar
150
151	# 4. Configure ICU4C, build and test without new data first, to verify that
152	# there are no pre-existing errors (configure shown here for MacOSX, adjust
153	# for your platform).
154
155	cd $ICU4C_DIR/source
156	./runConfigureICU MacOSX
157	make all 2>&1 \| tee /tmp/icu4c-oldData-makeAll.txt
158	make check 2>&1 \| tee /tmp/icu4c-oldData-makeCheck.txt
159
160	# 5. Build the new ICU4C data files; these include .txt files and .mk files.
161	# These new files will replace whatever was already present in the ICU4C sources.
162	# This process uses ant with ICU's data/build.xml and data/icu-config.xml to
163	# operate (via CLDR's ant/CLDRConverterTool.java and ant/CLDRBuild.java) the
164	# necessary CLDR tools including LDML2ICUConverter, ConvertTransforms, etc.
165	# This process will take several minutes (CLDR_DTD_CACHE helps reduce this).
166	# Keep a log so you can investigate anything that looks suspicious.
167
168	cd $ICU4C_DIR/source/data
169	ant clean
170	ant all 2>&1 \| tee /tmp/cldrNN-buildLog.txt
171
172	# 6. Check which data files have modifications, which have been added or removed
173	# (if there are no changes, you may not need to proceed further). Make sure the
174	# list seems reasonable.
175
176	svn status
177
178	# 7. Fix any errors, investigate any warnings. Some warnings are expected,
179	# including warnings for missing versions in locale names which specify some
180	# collationvariants, e.g.
181	# [cldr-build] WARNING (ja_JP_TRADITIONAL): No version #??
182	# [cldr-build] WARNING (zh_TW_STROKE): No version #??
183	# and warnings for some empty collation bundles, e.g.
184	# [cldr-build] WARNING (en): warning: No collations found. Bundle will ...
185	# [cldr-build] WARNING (to): warning: No collations found. Bundle will ...
186	#
187	# Fixing may entail modifying CLDR source data or tools - for example,
188	# updating the validSubLocales for collation data (file a bug f appropriate).
189	# Repeat steps 5-7 until there are no build errors and no unexpected
190	# warnings.
191
192	# 8. Now rebuild ICU4C with the new data and run make check tests.
193	# Again, keep a log so you can investigate the errors.
194
195	cd $ICU4C_DIR/source
196	make check 2>&1 \| tee /tmp/icu4c-newData-makeCheck.txt
197
198	# 9. Investigate each test case failure. The first run processing new CLDR data
199	# from the Survey Tool can result in thousands of failures (in many cases, one
200	# CLDR data fix can resolve hundres of test failures). If the error is caused
201	# by bad CLDR data, then file a CLDR bug, fix the data, and regenerate from
202	# step 5a. If the data is OK but the testcase needs to be updated because the
203	# data has legitimately changed, then update the testcase. You will check in
204	# the updated testcases along with the new ICU data at the end of this process.
205	# Repeat steps 5-8 until there are no errors.
206
207	# 10. Now run the make check tests in exhaustive mode:
208
209	cd $ICU4C_DIR/source
210	export INTLTEST_OPTS="-e"
211	export CINTLTST_OPTS="-e"
212	make check 2>&1 \| tee /tmp/icu4c-newData-makeCheckEx.txt
213
214	# 11. Again, investigate each failure, fixing CLDR data or ICU test cases as
215	# appropriate, and repeating steps 5-7 and 10 until there are no errors.
216
217	# 12. Now with ICU4J, build and test without new data first, to verify that
218	# there are no pre-existing errors (or at least to have the pre-existing errors
219	# as a base for comparison):
220
221	cd $ICU4J_ROOT
222	ant all 2>&1 \| tee /tmp/icu4j-oldData-antAll.txt
223	ant check 2>&1 \| tee /tmp/icu4j-oldData-antCheck.txt
224
225	# 13. Now build the new data for ICU4J
226
227	cd $ICU4C_DIR/source/data
228	make icu4j-data-install
229
230	# 14. Now rebuild ICU4J with the new data and run tests:
231	# Keep a log so you can investigate the errors.
232
233	cd $ICU4J_ROOT
234	ant check 2>&1 \| tee /tmp/icu4j-newData-antCheck.txt
235
236	# 15. Investigate test case failures; fix test cases and repeat from step 14,
237	# or fix CLDR data and repeat from step 5, as appropriate, unitl; there are no
238	# more failures in ICU4C or ICU4J (except failures that were present before you
239	# began testing the new CLDR data).
240
241	# 16. Check the file changes; then svn add or svn remove as necessary, and
242	# commit the changes.
243
244	cd $ICU4C_DIR/source/
245	svn status
246	# add or remove as necessary, then commit
247
248	cd $ICU4J_ROOT
249	svn status
250	# add or remove as necessary, then commit
251
252	# 17. For an official CLDR data integration into ICU, now tag the CLDR, ICU4J,
253	# and ICU4C sources with an appropriate CLDR milestone (you can check previous
254	# tags for format), e.g.:
255
256	svn copy svn+ssh://unicode.org/repos/cldr/trunk \
257	svn+ssh://unicode.org/repos/cldr/tags/release-NNN \
258	--parents -m "cldrbug nnnn: tag cldr sources for NNN"
259
260	svn copy svn+ssh://source.icu-project.org/repos/icu/icu4j/trunk \
261	svn+ssh://source.icu-project.org/repos/icu/icu4j/tags/cldr-NNN \
262	--parents -m 'ticket:mmmm: tag the version used for integrating CLDR NNN'
263
264	svn copy svn+ssh://source.icu-project.org/repos/icu/icu/trunk \
265	svn+ssh://source.icu-project.org/repos/icu/icu/tags/cldr-NNN \
266	--parents -m 'ticket:mmmm: tag the version used for integrating CLDR NNN'
267