// Purpose: topic overview
// Author: wxWidgets team
// RCS-ID: $Id$
-// Licence: wxWindows license
+// Licence: wxWindows licence
/////////////////////////////////////////////////////////////////////////////
/**
@page overview_unicode Unicode Support in wxWidgets
+@tableofcontents
+
This section describes how does wxWidgets support Unicode and how can it affect
your programs.
these changes.
You can skip the first two sections if you're already familiar with Unicode and
-wish to jump directly in the details of its support in the library:
-@li @ref overview_unicode_what
-@li @ref overview_unicode_encodings
-@li @ref overview_unicode_supportin
-@li @ref overview_unicode_pitfalls
-@li @ref overview_unicode_supportout
-@li @ref overview_unicode_settings
+wish to jump directly in the details of its support in the library.
-<hr>
@section overview_unicode_what What is Unicode?
When working with Unicode, it's important to define the meaning of some terms.
-A @e glyph is a particular image that represents a @e character or part of a character.
+A <b><em>glyph</em></b> is a particular image (usually part of a font) that
+represents a character or part of a character.
Any character may have one or more glyph associated; e.g. some of the possible
glyphs for the capital letter 'A' are:
@image html overview_unicode_glyphs.png
Unicode assigns each character of almost any existing alphabet/script a number,
-which is called <em>code point</em>; it's typically indicated in documentation
+which is called <b><em>code point</em></b>; it's typically indicated in documentation
manuals and in the Unicode website as @c U+xxxx where @c xxxx is an hexadecimal number.
-The Unicode standard divides the space of all possible code points in @e planes;
+Note that typically one character is assigned exactly one code point, but there
+are exceptions; the so-called <em>precomposed characters</em>
+(see http://en.wikipedia.org/wiki/Precomposed_character) or the <em>ligatures</em>.
+In these cases a single "character" may be mapped to more than one code point or
+viceversa more characters may be mapped to a single code point.
+
+The Unicode standard divides the space of all possible code points in <b><em>planes</em></b>;
a plane is a range of 65,536 (1000016) contiguous Unicode code points.
Planes are numbered from 0 to 16, where the first one is the @e BMP, or Basic
Multilingual Plane.
+The BMP contains characters for all modern languages, and a large number of
+special characters. The other planes in fact contain mainly historic scripts,
+special-purpose characters or are unused.
Code points are represented in computer memory as a sequence of one or more
-<em>code units</em>, where a code unit is a unit of memory: 8, 16, or 32 bits.
+<b><em>code units</em></b>, where a code unit is a unit of memory: 8, 16, or 32 bits.
More precisely, a code unit is the minimal bit combination that can represent a
unit of encoded text for processing or interchange.
-The @e UTF or Unicode Transformation Formats are algorithms mapping the Unicode
+The <b><em>UTF</em></b> or Unicode Transformation Formats are algorithms mapping the Unicode
code points to code unit sequences. The simplest of them is <b>UTF-32</b> where
-each code unit is composed by 32 bits (4 bytes) and each code point is represented
-by a single code unit.
+each code unit is composed by 32 bits (4 bytes) and each code point is always
+represented by a single code unit (fixed length encoding).
(Note that even UTF-32 is still not completely trivial as the mapping is different
for little and big-endian architectures). UTF-32 is commonly used under Unix systems for
internal representation of Unicode strings.
it encodes the first (approximately) 64 thousands of Unicode code points
(the BMP plane) using 16-bit code units (2 bytes) and uses a pair of 16-bit code
units to encode the characters beyond this. These pairs are called @e surrogate.
+Thus UTF16 uses a variable number of code units to encode each code point.
Finally, the most widespread encoding used for the external Unicode storage
(e.g. files and network protocols) is <b>UTF-8</b> which is byte-oriented and so
@c char are 8bit wide on almost all systems; when using UTF16 typically code
units are stored into @c wchar_t types since @c wchar_t is at least 16bits on
all systems. This is also the approach used by wxString.
-See @ref overview_wxstring for more info.
+See @ref overview_string for more info.
See also http://unicode.org/glossary/ for the official definitions of the
terms reported above.
@section overview_unicode_supportin Unicode Support in wxWidgets
-Since wxWidgets 3.0 Unicode support is always enabled and building the library
-without it is not recommended any longer and will cease to be supported in the
-near future. This means that internally only Unicode strings are used and that,
-under Microsoft Windows, Unicode system API is used which means that wxWidgets
-programs require the Microsoft Layer for Unicode to run on Windows 95/98/ME.
+@subsection overview_unicode_support_default Unicode is Always Used by Default
+
+Since wxWidgets 3.0 Unicode support is always enabled and while building the
+library without it is still possible, it is not recommended any longer and will
+cease to be supported in the near future. This means that internally only
+Unicode strings are used and that, under Microsoft Windows, Unicode system API
+is used which means that wxWidgets programs require the Microsoft Layer for
+Unicode to run on Windows 95/98/ME.
However, unlike the Unicode build mode of the previous versions of wxWidgets, this
support is mostly transparent: you can still continue to work with the @b narrow
-(i.e. current-locale-encoded @c char*) strings even if @b wide
-(i.e. UTF16/UCS2-encoded @c wchar_t* or UTF8-encoded @c char) strings are also
+(i.e. current locale-encoded @c char*) strings even if @b wide
+(i.e. UTF16-encoded @c wchar_t* or UTF8-encoded @c char*) strings are also
supported. Any wxWidgets function accepts arguments of either type as both
kinds of strings are implicitly converted to wxString, so both
@code
@endcode
and the somewhat less usual
@code
-wxMessageBox(L"Salut \u00e0 toi!"); // 00E0 is "Latin Small Letter a with Grave"
+wxMessageBox(L"Salut \u00E0 toi!"); // U+00E0 is "Latin Small Letter a with Grave"
@endcode
work as expected.
modern Unix systems is UTF-8 and as the string above is not a valid UTF-8 byte
sequence, nothing would be displayed at all in this case. Thus it is important
to <b>never use 8-bit (instead of 7-bit) characters directly in the program source</b>
-but use wide strings or, alternatively, write
+but use wide strings or, alternatively, write:
@code
-wxMessageBox(wxString::FromUTF8("Salut \xc3\xa0 toi!"));
+wxMessageBox(wxString::FromUTF8("Salut \xC3\xA0 toi!"));
+ // in UTF8 the character U+00E0 is encoded as 0xC3A0
@endcode
In a similar way, wxString provides access to its contents as either @c wchar_t or
aware of the potential problems covered by the following section.
+@subsection overview_unicode_support_utf Choosing Unicode Representation
+
+wxWidgets uses the system @c wchar_t in wxString implementation by default
+under all systems. Thus, under Microsoft Windows, UCS-2 (simplified version of
+UTF-16 without support for surrogate characters) is used as @c wchar_t is 2
+bytes on this platform. Under Unix systems, including Mac OS X, UCS-4 (also
+known as UTF-32) is used by default, however it is also possible to build
+wxWidgets to use UTF-8 internally by passing @c --enable-utf8 option to
+configure.
+
+The interface provided by wxString is the same independently of the format used
+internally. However different formats have specific advantages and
+disadvantages. Notably, under Unix, the underlying graphical toolkit (e.g.
+GTK+) usually uses UTF-8 encoded strings and using the same representations for
+the strings in wxWidgets allows to avoid conversion from UTF-32 to UTF-8 and
+vice versa each time a string is shown in the UI or retrieved from it. The
+overhead of such conversions is usually negligible for small strings but may be
+important for some programs. If you believe that it would be advantageous to
+use UTF-8 for the strings in your particular application, you may rebuild
+wxWidgets to use UTF-8 as explained above (notice that this is currently not
+supported under Microsoft Windows and arguably doesn't make much sense there as
+Windows itself uses UTF-16 and not UTF-8) but be sure to be aware of the
+performance implications (see @ref overview_unicode_performance) of using UTF-8
+in wxString before doing this!
+
+Generally speaking you should only use non-default UTF-8 build in specific
+circumstances e.g. building for resource-constrained systems where the overhead
+of conversions (and also reduced memory usage of UTF-8 compared to UTF-32 for
+the European languages) can be important. If the environment in which your
+program is running is under your control -- as is quite often the case in such
+scenarios -- consider ensuring that the system always uses UTF-8 locale and
+use @c --enable-utf8only configure option to disable support for the other
+locales and consider all strings to be in UTF-8. This further reduces the code
+size and removes the need for conversions in more cases.
+
+
+@subsection overview_unicode_settings Unicode Related Preprocessor Symbols
+
+@c wxUSE_UNICODE is defined as 1 now to indicate Unicode support. It can be
+explicitly set to 0 in @c setup.h under MSW or you can use @c --disable-unicode
+under Unix but doing this is strongly discouraged. By default, @c
+wxUSE_UNICODE_WCHAR is also defined as 1, however in UTF-8 build (described in
+the previous section), it is set to 0 and @c wxUSE_UNICODE_UTF8, which is
+usually 0, is set to 1 instead. In the latter case, @c wxUSE_UTF8_LOCALE_ONLY
+can also be set to 1 to indicate that all strings are considered to be in UTF-8.
+
+
+
@section overview_unicode_pitfalls Potential Unicode Pitfalls
The problems can be separated into three broad classes:
n:
- Writing @code switch ( s[n] ) @endcode doesn't work because the argument of
- the switch statement must an integer expression so you need to replace
+ the switch statement must be an integer expression so you need to replace
@c s[n] with @code s[n].GetValue() @endcode. You may also force the
conversion to @c char or @c wchar_t by using an explicit cast but beware that
converting the value to char uses the conversion to current locale and may
- Using a cast to force the issue (listed only for completeness):
@code printf("Hello, %s", (const char *)s.c_str()) @endcode
- - The result of @c c_str() can not be cast to @c char* but only to @c const @c
+ - The result of @c c_str() cannot be cast to @c char* but only to @c const @c
@c char*. Of course, modifying the string via the pointer returned by this
method has never been possible but unfortunately it was occasionally useful
to use a @c const_cast here to pass the value to const-incorrect functions.
avoiding implicit conversions to @c char*.
-@subsection overview_unicode_performance Unicode Performance Implications
+@subsection overview_unicode_performance Performance Implications of Using UTF-8
-Under Unix systems wxString class uses variable-width UTF-8 encoding for
-internal representation and this implies that it can't guarantee constant-time
-access to N-th element of the string any longer as to find the position of this
-character in the string we have to examine all the preceding ones. Usually this
-doesn't matter much because most algorithms used on the strings examine them
-sequentially anyhow and because wxString implements a cache for iterating over
-the string by index but it can have serious consequences for algorithms
-using random access to string elements as they typically acquire O(N^2) time
-complexity instead of O(N) where N is the length of the string.
+As mentioned above, under Unix systems wxString class can use variable-width
+UTF-8 encoding for internal representation. In this case it can't guarantee
+constant-time access to N-th element of the string any longer as to find the
+position of this character in the string we have to examine all the preceding
+ones. Usually this doesn't matter much because most algorithms used on the
+strings examine them sequentially anyhow and because wxString implements a
+cache for iterating over the string by index but it can have serious
+consequences for algorithms using random access to string elements as they
+typically acquire O(N^2) time complexity instead of O(N) where N is the length
+of the string.
Even despite caching the index, indexed access should be replaced with
sequential access using string iterators. For example a typical loop:
representations and the wxString methods wxString::ToAscii(), wxString::ToUTF8()
(or its synonym wxString::utf8_str()), wxString::mb_str(), wxString::c_str() and
wxString::wc_str() can be used for this.
+
The first of them should be only used for the string containing 7-bit ASCII characters
only, anything else will be replaced by some substitution character.
wxString::mb_str() converts the string to the encoding used by the current locale
puts(p); // or call any other function taking const char *
@endcode
does @b not work because the temporary buffer returned by wxString::ToUTF8() is
-destroyed and @c p is left pointing nowhere. To correct this you may use
+destroyed and @c p is left pointing nowhere. To correct this you should use
@code
-wxCharBuffer p(s.ToUTF8());
+const wxScopedCharBuffer p(s.ToUTF8());
puts(p);
@endcode
-which does work but results in an unnecessary copy of string data in the build
-configurations when wxString::ToUTF8() returns the pointer to internal string buffer.
-If this inefficiency is important you may write
-@code
-const wxUTF8Buf p(s.ToUTF8());
-puts(p);
-@endcode
-where @c wxUTF8Buf is the type corresponding to the real return type of wxString::ToUTF8().
+which does work.
+
Similarly, wxWX2WCbuf can be used for the return type of wxString::wc_str().
But, once again, none of these cryptic types is really needed if you just pass
the return value of any of the functions mentioned in this section to another
function directly.
-@section overview_unicode_settings Unicode Related Compilation Settings
-
-@c wxUSE_UNICODE is now defined as 1 by default to indicate Unicode support.
-If UTF-8 is used for the internal storage in wxString, @c wxUSE_UNICODE_UTF8 is
-also defined, otherwise @c wxUSE_UNICODE_WCHAR is.
-
-You are encouraged to always use the default build settings of wxWidgets; this avoids
-the need of different builds of the same application/library because of different
-"build modes".
-
*/
-
projects / wxWidgets.git / blobdiff