]> git.saurik.com Git - wxWidgets.git/blame_incremental - docs/doxygen/overviews/unicode.h
wxMessageBox off the main thread lost result code.
[wxWidgets.git] / docs / doxygen / overviews / unicode.h
... / ...
CommitLineData
1/////////////////////////////////////////////////////////////////////////////
2// Name: unicode.h
3// Purpose: topic overview
4// Author: wxWidgets team
5// Licence: wxWindows licence
6/////////////////////////////////////////////////////////////////////////////
7
8/**
9
10@page overview_unicode Unicode Support in wxWidgets
11
12@tableofcontents
13
14This section describes how does wxWidgets support Unicode and how can it affect
15your programs.
16
17Notice that Unicode support has changed radically in wxWidgets 3.0 and a lot of
18existing material pertaining to the previous versions of the library is not
19correct any more. Please see @ref overview_changes_unicode for the details of
20these changes.
21
22You can skip the first two sections if you're already familiar with Unicode and
23wish to jump directly in the details of its support in the library.
24
25
26
27@section overview_unicode_what What is Unicode?
28
29Unicode is a standard for character encoding which addresses the shortcomings
30of the previous standards (e.g. the ASCII standard), by using 8, 16 or 32 bits
31for encoding each character.
32This allows enough code points (see below for the definition) sufficient to
33encode all of the world languages at once.
34More details about Unicode may be found at http://www.unicode.org/.
35
36From a practical point of view, using Unicode is almost a requirement when
37writing applications for international audience. Moreover, any application
38reading files which it didn't produce or receiving data from the network from
39other services should be ready to deal with Unicode.
40
41
42@section overview_unicode_encodings Unicode Representations and Terminology
43
44When working with Unicode, it's important to define the meaning of some terms.
45
46A <b><em>glyph</em></b> is a particular image (usually part of a font) that
47represents a character or part of a character.
48Any character may have one or more glyph associated; e.g. some of the possible
49glyphs for the capital letter 'A' are:
50
51@image html overview_unicode_glyphs.png
52
53Unicode assigns each character of almost any existing alphabet/script a number,
54which is called <b><em>code point</em></b>; it's typically indicated in documentation
55manuals and in the Unicode website as @c U+xxxx where @c xxxx is an hexadecimal number.
56
57Note that typically one character is assigned exactly one code point, but there
58are exceptions; the so-called <em>precomposed characters</em>
59(see http://en.wikipedia.org/wiki/Precomposed_character) or the <em>ligatures</em>.
60In these cases a single "character" may be mapped to more than one code point or
61viceversa more characters may be mapped to a single code point.
62
63The Unicode standard divides the space of all possible code points in <b><em>planes</em></b>;
64a plane is a range of 65,536 (1000016) contiguous Unicode code points.
65Planes are numbered from 0 to 16, where the first one is the @e BMP, or Basic
66Multilingual Plane.
67The BMP contains characters for all modern languages, and a large number of
68special characters. The other planes in fact contain mainly historic scripts,
69special-purpose characters or are unused.
70
71Code points are represented in computer memory as a sequence of one or more
72<b><em>code units</em></b>, where a code unit is a unit of memory: 8, 16, or 32 bits.
73More precisely, a code unit is the minimal bit combination that can represent a
74unit of encoded text for processing or interchange.
75
76The <b><em>UTF</em></b> or Unicode Transformation Formats are algorithms mapping the Unicode
77code points to code unit sequences. The simplest of them is <b>UTF-32</b> where
78each code unit is composed by 32 bits (4 bytes) and each code point is always
79represented by a single code unit (fixed length encoding).
80(Note that even UTF-32 is still not completely trivial as the mapping is different
81for little and big-endian architectures). UTF-32 is commonly used under Unix systems for
82internal representation of Unicode strings.
83
84Another very widespread standard is <b>UTF-16</b> which is used by Microsoft Windows:
85it encodes the first (approximately) 64 thousands of Unicode code points
86(the BMP plane) using 16-bit code units (2 bytes) and uses a pair of 16-bit code
87units to encode the characters beyond this. These pairs are called @e surrogate.
88Thus UTF16 uses a variable number of code units to encode each code point.
89
90Finally, the most widespread encoding used for the external Unicode storage
91(e.g. files and network protocols) is <b>UTF-8</b> which is byte-oriented and so
92avoids the endianness ambiguities of UTF-16 and UTF-32.
93UTF-8 uses code units of 8 bits (1 byte); code points beyond the usual english
94alphabet are represented using a variable number of bytes, which makes it less
95efficient than UTF-32 for internal representation.
96
97As visual aid to understand the differences between the various concepts described
98so far, look at the different UTF representations of the same code point:
99
100@image html overview_unicode_codes.png
101
102In this particular case UTF8 requires more space than UTF16 (3 bytes instead of 2).
103
104Note that from the C/C++ programmer perspective the situation is further complicated
105by the fact that the standard type @c wchar_t which is usually used to represent the
106Unicode ("wide") strings in C/C++ doesn't have the same size on all platforms.
107It is 4 bytes under Unix systems, corresponding to the tradition of using
108UTF-32, but only 2 bytes under Windows which is required by compatibility with
109the OS which uses UTF-16.
110
111Typically when UTF8 is used, code units are stored into @c char types, since
112@c char are 8bit wide on almost all systems; when using UTF16 typically code
113units are stored into @c wchar_t types since @c wchar_t is at least 16bits on
114all systems. This is also the approach used by wxString.
115See @ref overview_string for more info.
116
117See also http://unicode.org/glossary/ for the official definitions of the
118terms reported above.
119
120
121@section overview_unicode_supportin Unicode Support in wxWidgets
122
123@subsection overview_unicode_support_default Unicode is Always Used by Default
124
125Since wxWidgets 3.0 Unicode support is always enabled and while building the
126library without it is still possible, it is not recommended any longer and will
127cease to be supported in the near future. This means that internally only
128Unicode strings are used and that, under Microsoft Windows, Unicode system API
129is used which means that wxWidgets programs require the Microsoft Layer for
130Unicode to run on Windows 95/98/ME.
131
132However, unlike the Unicode build mode of the previous versions of wxWidgets, this
133support is mostly transparent: you can still continue to work with the @b narrow
134(i.e. current locale-encoded @c char*) strings even if @b wide
135(i.e. UTF16-encoded @c wchar_t* or UTF8-encoded @c char*) strings are also
136supported. Any wxWidgets function accepts arguments of either type as both
137kinds of strings are implicitly converted to wxString, so both
138@code
139wxMessageBox("Hello, world!");
140@endcode
141and the somewhat less usual
142@code
143wxMessageBox(L"Salut \u00E0 toi!"); // U+00E0 is "Latin Small Letter a with Grave"
144@endcode
145work as expected.
146
147Notice that the narrow strings used with wxWidgets are @e always assumed to be
148in the current locale encoding, so writing
149@code
150wxMessageBox("Salut à toi!");
151@endcode
152wouldn't work if the encoding used on the user system is incompatible with
153ISO-8859-1 (or even if the sources were compiled under different locale
154in the case of gcc). In particular, the most common encoding used under
155modern Unix systems is UTF-8 and as the string above is not a valid UTF-8 byte
156sequence, nothing would be displayed at all in this case. Thus it is important
157to <b>never use 8-bit (instead of 7-bit) characters directly in the program source</b>
158but use wide strings or, alternatively, write:
159@code
160wxMessageBox(wxString::FromUTF8("Salut \xC3\xA0 toi!"));
161 // in UTF8 the character U+00E0 is encoded as 0xC3A0
162@endcode
163
164In a similar way, wxString provides access to its contents as either @c wchar_t or
165@c char character buffer. Of course, the latter only works if the string contains
166data representable in the current locale encoding. This will always be the case
167if the string had been initially constructed from a narrow string or if it
168contains only 7-bit ASCII data but otherwise this conversion is not guaranteed
169to succeed. And as with wxString::FromUTF8() example above, you can always use
170wxString::ToUTF8() to retrieve the string contents in UTF-8 encoding -- this,
171unlike converting to @c char* using the current locale, never fails.
172
173For more info about how wxString works, please see the @ref overview_string.
174
175To summarize, Unicode support in wxWidgets is mostly @b transparent for the
176application and if you use wxString objects for storing all the character data
177in your program there is really nothing special to do. However you should be
178aware of the potential problems covered by the following section.
179
180
181@subsection overview_unicode_support_utf Choosing Unicode Representation
182
183wxWidgets uses the system @c wchar_t in wxString implementation by default
184under all systems. Thus, under Microsoft Windows, UCS-2 (simplified version of
185UTF-16 without support for surrogate characters) is used as @c wchar_t is 2
186bytes on this platform. Under Unix systems, including Mac OS X, UCS-4 (also
187known as UTF-32) is used by default, however it is also possible to build
188wxWidgets to use UTF-8 internally by passing @c --enable-utf8 option to
189configure.
190
191The interface provided by wxString is the same independently of the format used
192internally. However different formats have specific advantages and
193disadvantages. Notably, under Unix, the underlying graphical toolkit (e.g.
194GTK+) usually uses UTF-8 encoded strings and using the same representations for
195the strings in wxWidgets allows to avoid conversion from UTF-32 to UTF-8 and
196vice versa each time a string is shown in the UI or retrieved from it. The
197overhead of such conversions is usually negligible for small strings but may be
198important for some programs. If you believe that it would be advantageous to
199use UTF-8 for the strings in your particular application, you may rebuild
200wxWidgets to use UTF-8 as explained above (notice that this is currently not
201supported under Microsoft Windows and arguably doesn't make much sense there as
202Windows itself uses UTF-16 and not UTF-8) but be sure to be aware of the
203performance implications (see @ref overview_unicode_performance) of using UTF-8
204in wxString before doing this!
205
206Generally speaking you should only use non-default UTF-8 build in specific
207circumstances e.g. building for resource-constrained systems where the overhead
208of conversions (and also reduced memory usage of UTF-8 compared to UTF-32 for
209the European languages) can be important. If the environment in which your
210program is running is under your control -- as is quite often the case in such
211scenarios -- consider ensuring that the system always uses UTF-8 locale and
212use @c --enable-utf8only configure option to disable support for the other
213locales and consider all strings to be in UTF-8. This further reduces the code
214size and removes the need for conversions in more cases.
215
216
217@subsection overview_unicode_settings Unicode Related Preprocessor Symbols
218
219@c wxUSE_UNICODE is defined as 1 now to indicate Unicode support. It can be
220explicitly set to 0 in @c setup.h under MSW or you can use @c --disable-unicode
221under Unix but doing this is strongly discouraged. By default, @c
222wxUSE_UNICODE_WCHAR is also defined as 1, however in UTF-8 build (described in
223the previous section), it is set to 0 and @c wxUSE_UNICODE_UTF8, which is
224usually 0, is set to 1 instead. In the latter case, @c wxUSE_UTF8_LOCALE_ONLY
225can also be set to 1 to indicate that all strings are considered to be in UTF-8.
226
227
228
229@section overview_unicode_pitfalls Potential Unicode Pitfalls
230
231The problems can be separated into three broad classes:
232
233@subsection overview_unicode_compilation_errors Unicode-Related Compilation Errors
234
235Because of the need to support implicit conversions to both @c char and
236@c wchar_t, wxString implementation is rather involved and many of its operators
237don't return the types which they could be naively expected to return.
238For example, the @c operator[] doesn't return neither a @c char nor a @c wchar_t
239but an object of a helper class wxUniChar or wxUniCharRef which is implicitly
240convertible to either. Usually you don't need to worry about this as the
241conversions do their work behind the scenes however in some cases it doesn't
242work. Here are some examples, using a wxString object @c s and some integer @c
243n:
244
245 - Writing @code switch ( s[n] ) @endcode doesn't work because the argument of
246 the switch statement must be an integer expression so you need to replace
247 @c s[n] with @code s[n].GetValue() @endcode. You may also force the
248 conversion to @c char or @c wchar_t by using an explicit cast but beware that
249 converting the value to char uses the conversion to current locale and may
250 return 0 if it fails. Finally notice that writing @code (wxChar)s[n] @endcode
251 works both with wxWidgets 3.0 and previous library versions and so should be
252 used for writing code which should be compatible with both 2.8 and 3.0.
253
254 - Similarly, @code &s[n] @endcode doesn't yield a pointer to char so you may
255 not pass it to functions expecting @c char* or @c wchar_t*. Consider using
256 string iterators instead if possible or replace this expression with
257 @code s.c_str() + n @endcode otherwise.
258
259Another class of problems is related to the fact that the value returned by
260@c c_str() itself is also not just a pointer to a buffer but a value of helper
261class wxCStrData which is implicitly convertible to both narrow and wide
262strings. Again, this mostly will be unnoticeable but can result in some
263problems:
264
265 - You shouldn't pass @c c_str() result to vararg functions such as standard
266 @c printf(). Some compilers (notably g++) warn about this but even if they
267 don't, this @code printf("Hello, %s", s.c_str()) @endcode is not going to
268 work. It can be corrected in one of the following ways:
269
270 - Preferred: @code wxPrintf("Hello, %s", s) @endcode (notice the absence
271 of @c c_str(), it is not needed at all with wxWidgets functions)
272 - Compatible with wxWidgets 2.8: @code wxPrintf("Hello, %s", s.c_str()) @endcode
273 - Using an explicit conversion to narrow, multibyte, string:
274 @code printf("Hello, %s", (const char *)s.mb_str()) @endcode
275 - Using a cast to force the issue (listed only for completeness):
276 @code printf("Hello, %s", (const char *)s.c_str()) @endcode
277
278 - The result of @c c_str() cannot be cast to @c char* but only to @c const @c
279 @c char*. Of course, modifying the string via the pointer returned by this
280 method has never been possible but unfortunately it was occasionally useful
281 to use a @c const_cast here to pass the value to const-incorrect functions.
282 This can be done either using new wxString::char_str() (and matching
283 wchar_str()) method or by writing a double cast:
284 @code (char *)(const char *)s.c_str() @endcode
285
286 - One of the unfortunate consequences of the possibility to pass wxString to
287 @c wxPrintf() without using @c c_str() is that it is now impossible to pass
288 the elements of unnamed enumerations to @c wxPrintf() and other similar
289 vararg functions, i.e.
290 @code
291 enum { Red, Green, Blue };
292 wxPrintf("Red is %d", Red);
293 @endcode
294 doesn't compile. The easiest workaround is to give a name to the enum.
295
296Other unexpected compilation errors may arise but they should happen even more
297rarely than the above-mentioned ones and the solution should usually be quite
298simple: just use the explicit methods of wxUniChar and wxCStrData classes
299instead of relying on their implicit conversions if the compiler can't choose
300among them.
301
302
303@subsection overview_unicode_data_loss Data Loss due To Unicode Conversion Errors
304
305wxString API provides implicit conversion of the internal Unicode string
306contents to narrow, char strings. This can be very convenient and is absolutely
307necessary for backwards compatibility with the existing code using wxWidgets
308however it is a rather dangerous operation as it can easily give unexpected
309results if the string contents isn't convertible to the current locale.
310
311To be precise, the conversion will always succeed if the string was created
312from a narrow string initially. It will also succeed if the current encoding is
313UTF-8 as all Unicode strings are representable in this encoding. However
314initializing the string using wxString::FromUTF8() method and then accessing it
315as a char string via its wxString::c_str() method is a recipe for disaster as the
316program may work perfectly well during testing on Unix systems using UTF-8 locale
317but completely fail under Windows where UTF-8 locales are never used because
318wxString::c_str() would return an empty string.
319
320The simplest way to ensure that this doesn't happen is to avoid conversions to
321@c char* completely by using wxString throughout your program. However if the
322program never manipulates 8 bit strings internally, using @c char* pointers is
323safe as well. So the existing code needs to be reviewed when upgrading to
324wxWidgets 3.0 and the new code should be used with this in mind and ideally
325avoiding implicit conversions to @c char*.
326
327
328@subsection overview_unicode_performance Performance Implications of Using UTF-8
329
330As mentioned above, under Unix systems wxString class can use variable-width
331UTF-8 encoding for internal representation. In this case it can't guarantee
332constant-time access to N-th element of the string any longer as to find the
333position of this character in the string we have to examine all the preceding
334ones. Usually this doesn't matter much because most algorithms used on the
335strings examine them sequentially anyhow and because wxString implements a
336cache for iterating over the string by index but it can have serious
337consequences for algorithms using random access to string elements as they
338typically acquire O(N^2) time complexity instead of O(N) where N is the length
339of the string.
340
341Even despite caching the index, indexed access should be replaced with
342sequential access using string iterators. For example a typical loop:
343@code
344wxString s("hello");
345for ( size_t i = 0; i < s.length(); i++ )
346{
347 wchar_t ch = s[i];
348
349 // do something with it
350}
351@endcode
352should be rewritten as
353@code
354wxString s("hello");
355for ( wxString::const_iterator i = s.begin(); i != s.end(); ++i )
356{
357 wchar_t ch = *i
358
359 // do something with it
360}
361@endcode
362
363Another, similar, alternative is to use pointer arithmetic:
364@code
365wxString s("hello");
366for ( const wchar_t *p = s.wc_str(); *p; p++ )
367{
368 wchar_t ch = *i
369
370 // do something with it
371}
372@endcode
373however this doesn't work correctly for strings with embedded @c NUL characters
374and the use of iterators is generally preferred as they provide some run-time
375checks (at least in debug build) unlike the raw pointers. But if you do use
376them, it is better to use @c wchar_t pointers rather than @c char ones to avoid the
377data loss problems due to conversion as discussed in the previous section.
378
379
380@section overview_unicode_supportout Unicode and the Outside World
381
382Even though wxWidgets always uses Unicode internally, not all the other
383libraries and programs do and even those that do use Unicode may use a
384different encoding of it. So you need to be able to convert the data to various
385representations and the wxString methods wxString::ToAscii(), wxString::ToUTF8()
386(or its synonym wxString::utf8_str()), wxString::mb_str(), wxString::c_str() and
387wxString::wc_str() can be used for this.
388
389The first of them should be only used for the string containing 7-bit ASCII characters
390only, anything else will be replaced by some substitution character.
391wxString::mb_str() converts the string to the encoding used by the current locale
392and so can return an empty string if the string contains characters not representable in
393it as explained in @ref overview_unicode_data_loss. The same applies to wxString::c_str()
394if its result is used as a narrow string. Finally, wxString::ToUTF8() and wxString::wc_str()
395functions never fail and always return a pointer to char string containing the
396UTF-8 representation of the string or @c wchar_t string.
397
398wxString also provides two convenience functions: wxString::From8BitData() and
399wxString::To8BitData(). They can be used to create a wxString from arbitrary binary
400data without supposing that it is in current locale encoding, and then get it back,
401again, without any conversion or, rather, undoing the conversion used by
402wxString::From8BitData(). Because of this you should only use wxString::From8BitData()
403for the strings created using wxString::To8BitData(). Also notice that in spite
404of the availability of these functions, wxString is not the ideal class for storing
405arbitrary binary data as they can take up to 4 times more space than needed
406(when using @c wchar_t internal representation on the systems where size of
407wide characters is 4 bytes) and you should consider using wxMemoryBuffer
408instead.
409
410Final word of caution: most of these functions may return either directly the
411pointer to internal string buffer or a temporary wxCharBuffer or wxWCharBuffer
412object. Such objects are implicitly convertible to @c char and @c wchar_t pointers,
413respectively, and so the result of, for example, wxString::ToUTF8() can always be
414passed directly to a function taking <tt>const char*</tt>. However code such as
415@code
416const char *p = s.ToUTF8();
417...
418puts(p); // or call any other function taking const char *
419@endcode
420does @b not work because the temporary buffer returned by wxString::ToUTF8() is
421destroyed and @c p is left pointing nowhere. To correct this you should use
422@code
423const wxScopedCharBuffer p(s.ToUTF8());
424puts(p);
425@endcode
426which does work.
427
428Similarly, wxWX2WCbuf can be used for the return type of wxString::wc_str().
429But, once again, none of these cryptic types is really needed if you just pass
430the return value of any of the functions mentioned in this section to another
431function directly.
432
433*/