Logo Search packages:      
Sourcecode: icu version File versions

utf.h File Reference

Detailed Description

C API: UChar and UChar32 data types and UTF macros for C Unicode string handling.

This file defines the UChar and UChar32 data types for Unicode code units and code points, as well as macros for efficiently getting code points in and out of a string.

utf.h is included by utypes.h and itself includes the utfXX.h after some common definitions. Those files define the macros for each UTF-size.

The original concept for these files was for ICU to allow in principle to set which UTF (UTF-8/16/32) is used internally by defining UTF_SIZE to either 8, 16, or 32. utf.h would then define the UChar type accordingly. UTF-16 was the default.

This concept has been abandoned. A lot of the ICU source code — especially low-level code like conversion, normalization, and collation — assumes UTF-16, utf.h enforces the default of UTF-16. The UTF-8 and UTF-32 macros remain for now for completeness and backward compatibility.

Accordingly, utf.h defines UChar to be an unsigned 16-bit integer. If this matches wchar_t, then UChar is defined to be exactly wchar_t, otherwise uint16_t.

UChar32 is always defined to be a 32-bit integer to be large enough for a 21-bit Unicode code point (Unicode scalar value, 0..0x10ffff). If wchar_t is a 32-bit type, then UChar32 is defined to be exactly wchar_t, regardless of whether wchar_t is signed or unsigned. This means that UChar32 may be signed or unsigned depending on the platform! If wchar_t is not a 32-bit type, then UChar32 is defined to be uint32_t.

utf.h also defines a number of C macros for handling single Unicode code points and for using UTF Unicode strings. It includes utf8.h, utf16.h, and utf32.h for the actual implementations of those macros and then aliases one set of them (for UTF-16) for general use. The UTF-specific macros have the UTF size in the macro name prefixes (UTF16_...), while the general alias macros always begin with UTF_...

Many string operations can be done with or without error checking. Where such a distinction is useful, there are two versions of the macros, "unsafe" and "safe" ones with ..._UNSAFE and ..._SAFE suffixes. The unsafe macros are fast but may cause program failures if the strings are not well-formed. The safe macros have an additional, boolean parameter "strict". If strict is FALSE, then only illegal sequences are detected. Otherwise, irregular sequences and non-characters are detected as well (like single surrogates). Safe macros return special error code points for illegal/irregular sequences: Typically, U+ffff, or values that would result in a code unit sequence of the same length as the erroneous input sequence.
Note that _UNSAFE macros have fewer parameters: They do not have the strictness parameter, and they do not have start/length parameters for boundary checking.

Here, the macros are aliased in two steps: In the first step, the UTF-specific macros with UTF16_ prefix and _UNSAFE and _SAFE suffixes are aliased according to the UTF_SIZE to macros with UTF_ prefix and the same suffixes and signatures. Then, in a second step, the default, general alias macros are set to use either the unsafe or the safe/not strict (default) or the safe/strict macro; these general macros do not have a strictness parameter.

It is possible to change the default choice for the general alias macros to be unsafe, safe/not strict or safe/strict. The default is safe/not strict. It is not recommended to select the unsafe macros as the basis for Unicode string handling in ICU! To select this, define UTF_SAFE, UTF_STRICT, or UTF_UNSAFE.

For general use, one should use the default, general macros with UTF_ prefix and no _SAFE/_UNSAFE suffix. Only in some cases it may be necessary to control the choice of macro directly and use a less generic alias. For example, if it can be assumed that a string is well-formed and the index will stay within the bounds, then the _UNSAFE version may be used. If a UTF-8 string is to be processed, then the macros with UTF8_ prefixes need to be used.

Usage: ICU coding guidelines for if() statements should be followed when using these macros. Compound statements (curly braces {}) must be used for if-else-while... bodies and all macro statements should be terminated with semicolon.

Definition in file utf.h.

#include <stddef.h>
#include "unicode/umachine.h"
#include "unicode/utf8.h"
#include "unicode/utf16.h"
#include "unicode/utf32.h"

Go to the source code of this file.


#define U_HAVE_WCHAR_H   1
#define U_SIZEOF_UCHAR   (UTF_SIZE>>3)
#define U_SIZEOF_WCHAR_T   4
#define UTF8_ERROR_VALUE_1   0x15
#define UTF8_ERROR_VALUE_2   0x9f
#define UTF_APPEND_CHAR(s, i, length, c)   UTF_APPEND_CHAR_SAFE(s, i, length, c)
#define UTF_APPEND_CHAR_SAFE(s, i, length, c)   UTF16_APPEND_CHAR_SAFE(s, i, length, c)
#define UTF_ARRAY_SIZE(size)   UTF16_ARRAY_SIZE(size)
#define UTF_BACK_1(s, start, i)   UTF_BACK_1_SAFE(s, start, i)
#define UTF_BACK_1_SAFE(s, start, i)   UTF16_BACK_1_SAFE(s, start, i)
#define UTF_BACK_1_UNSAFE(s, i)   UTF16_BACK_1_UNSAFE(s, i)
#define UTF_BACK_N(s, start, i, n)   UTF_BACK_N_SAFE(s, start, i, n)
#define UTF_BACK_N_SAFE(s, start, i, n)   UTF16_BACK_N_SAFE(s, start, i, n)
#define UTF_BACK_N_UNSAFE(s, i, n)   UTF16_BACK_N_UNSAFE(s, i, n)
#define UTF_ERROR_VALUE   0xffff
#define UTF_FWD_1(s, i, length)   UTF_FWD_1_SAFE(s, i, length)
#define UTF_FWD_1_SAFE(s, i, length)   UTF16_FWD_1_SAFE(s, i, length)
#define UTF_FWD_1_UNSAFE(s, i)   UTF16_FWD_1_UNSAFE(s, i)
#define UTF_FWD_N(s, i, length, n)   UTF_FWD_N_SAFE(s, i, length, n)
#define UTF_FWD_N_SAFE(s, i, length, n)   UTF16_FWD_N_SAFE(s, i, length, n)
#define UTF_FWD_N_UNSAFE(s, i, n)   UTF16_FWD_N_UNSAFE(s, i, n)
#define UTF_GET_CHAR(s, start, i, length, c)   UTF_GET_CHAR_SAFE(s, start, i, length, c, FALSE)
#define UTF_GET_CHAR_SAFE(s, start, i, length, c, strict)   UTF16_GET_CHAR_SAFE(s, start, i, length, c, strict)
#define UTF_GET_CHAR_UNSAFE(s, i, c)   UTF16_GET_CHAR_UNSAFE(s, i, c)
#define UTF_IS_ERROR(c)   (((c)&0xfffe)==0xfffe || (c)==UTF8_ERROR_VALUE_1 || (c)==UTF8_ERROR_VALUE_2)
#define UTF_IS_LEAD(uchar)   UTF16_IS_LEAD(uchar)
#define UTF_IS_SINGLE(uchar)   UTF16_IS_SINGLE(uchar)
#define UTF_IS_SURROGATE(uchar)   (((uchar)&0xfffff800)==0xd800)
#define UTF_IS_TRAIL(uchar)   UTF16_IS_TRAIL(uchar)
#define UTF_IS_VALID(c)
#define UTF_NEXT_CHAR(s, i, length, c)   UTF_NEXT_CHAR_SAFE(s, i, length, c, FALSE)
#define UTF_NEXT_CHAR_SAFE(s, i, length, c, strict)   UTF16_NEXT_CHAR_SAFE(s, i, length, c, strict)
#define UTF_NEXT_CHAR_UNSAFE(s, i, c)   UTF16_NEXT_CHAR_UNSAFE(s, i, c)
#define UTF_PREV_CHAR(s, start, i, c)   UTF_PREV_CHAR_SAFE(s, start, i, c, FALSE)
#define UTF_PREV_CHAR_SAFE(s, start, i, c, strict)   UTF16_PREV_CHAR_SAFE(s, start, i, c, strict)
#define UTF_PREV_CHAR_UNSAFE(s, i, c)   UTF16_PREV_CHAR_UNSAFE(s, i, c)
#define UTF_SAFE
#define UTF_SET_CHAR_LIMIT(s, start, i, length)   UTF_SET_CHAR_LIMIT_SAFE(s, start, i, length)
#define UTF_SET_CHAR_LIMIT_SAFE(s, start, i, length)   UTF16_SET_CHAR_LIMIT_SAFE(s, start, i, length)
#define UTF_SET_CHAR_START(s, start, i)   UTF_SET_CHAR_START_SAFE(s, start, i)
#define UTF_SET_CHAR_START_SAFE(s, start, i)   UTF16_SET_CHAR_START_SAFE(s, start, i)
#define UTF_SIZE   16


typedef uint16_t UChar
typedef wchar_t UChar32
typedef int32_t UTextOffset

Generated by  Doxygen 1.6.0   Back to index