- * These functions are intended to check strings, typically
- * identifiers of some type, such as URLs, for the presence of
- * characters that are likely to be visually confusing -
- * for cases where the displayed form of an identifier may
- * not be what it appears to be.
- *
- * Unicode Technical Report #36, http://unicode.org/reports/tr36, and
- * Unicode Technical Standard #39, http://unicode.org/reports/tr39
- * "Unicode security considerations", give more background on
- * security an spoofing issues with Unicode identifiers.
- * The tests and checks provided by this module implement the recommendations
- * from those Unicode documents.
- *
- * The tests available on identifiers fall into two general categories:
- * -# Single identifier tests. Check whether an identifier is
- * potentially confusable with any other string, or is suspicious
- * for other reasons.
- * -# Two identifier tests. Check whether two specific identifiers are confusable.
- * This does not consider whether either of strings is potentially
- * confusable with any string other than the exact one specified.
- *
- * The steps to perform confusability testing are
- * -# Open a USpoofChecker.
- * -# Configure the USPoofChecker for the desired set of tests. The tests that will
- * be performed are specified by a set of USpoofChecks flags.
- * -# Perform the checks using the pre-configured USpoofChecker. The results indicate
- * which (if any) of the selected tests have identified possible problems with the identifier.
- * Results are reported as a set of USpoofChecks flags; this mirrors the form in which
- * the set of tests to perform was originally specified to the USpoofChecker.
- *
- * A USpoofChecker may be used repeatedly to perform checks on any number of identifiers.
- *
- * Thread Safety: The test functions for checking a single identifier, or for testing
- * whether two identifiers are possible confusable, are thread safe.
- * They may called concurrently, from multiple threads, using the same USpoofChecker instance.
- *
- * More generally, the standard ICU thread safety rules apply: functions that take a
- * const USpoofChecker parameter are thread safe. Those that take a non-const
- * USpoofChecier are not thread safe.
- *
- *
- * Descriptions of the available checks.
- *
- * When testing whether pairs of identifiers are confusable, with the uspoof_areConfusable()
- * family of functions, the relevant tests are
- *
- * -# USPOOF_SINGLE_SCRIPT_CONFUSABLE: All of the characters from the two identifiers are
- * from a single script, and the two identifiers are visually confusable.
- * -# USPOOF_MIXED_SCRIPT_CONFUSABLE: At least one of the identifiers contains characters
- * from more than one script, and the two identifiers are visually confusable.
- * -# USPOOF_WHOLE_SCRIPT_CONFUSABLE: Each of the two identifiers is of a single script, but
- * the two identifiers are from different scripts, and they are visually confusable.
- *
- * The safest approach is to enable all three of these checks as a group.
- *
- * USPOOF_ANY_CASE is a modifier for the above tests. If the identifiers being checked can
- * be of mixed case and are used in a case-sensitive manner, this option should be specified.
- *
- * If the identifiers being checked are used in a case-insensitive manner, and if they are
- * displayed to users in lower-case form only, the USPOOF_ANY_CASE option should not be
- * specified. Confusabality issues involving upper case letters will not be reported.
- *
- * When performing tests on a single identifier, with the uspoof_check() family of functions,
- * the relevant tests are:
- *
- * -# USPOOF_MIXED_SCRIPT_CONFUSABLE: the identifier contains characters from multiple
- * scripts, and there exists an identifier of a single script that is visually confusable.
- * -# USPOOF_WHOLE_SCRIPT_CONFUSABLE: the identifier consists of characters from a single
- * script, and there exists a visually confusable identifier.
- * The visually confusable identifier also consists of characters from a single script.
- * but not the same script as the identifier being checked.
- * -# USPOOF_ANY_CASE: modifies the mixed script and whole script confusables tests. If
- * specified, the checks will consider confusable characters of any case. If this flag is not
- * set, the test is performed assuming case folded identifiers.
- * -# USPOOF_SINGLE_SCRIPT: check that the identifier contains only characters from a
- * single script. (Characters from the 'common' and 'inherited' scripts are ignored.)
- * This is not a test for confusable identifiers
- * -# USPOOF_INVISIBLE: check an identifier for the presence of invisible characters,
- * such as zero-width spaces, or character sequences that are
- * likely not to display, such as multiple occurrences of the same
- * non-spacing mark. This check does not test the input string as a whole
- * for conformance to any particular syntax for identifiers.
- * -# USPOOF_CHAR_LIMIT: check that an identifier contains only characters from a specified set
- * of acceptable characters. See uspoof_setAllowedChars() and
- * uspoof_setAllowedLocales().
- *
- * Note on Scripts:
- * Characters from the Unicode Scripts "Common" and "Inherited" are ignored when considering
- * the script of an identifier. Common characters include digits and symbols that
- * are normally used with text from more than one script.
- *
- * Identifier Skeletons: A skeleton is a transformation of an identifier, such that
- * all identifiers that are confusable with each other have the same skeleton.
- * Using skeletons, it is possible to build a dictionary data structure for
- * a set of identifiers, and then quickly test whether a new identifier is
- * confusable with an identifier already in the set. The uspoof_getSkeleton()
- * family of functions will produce the skeleton from an identifier.
- *
- * Note that skeletons are not guaranteed to be stable between versions
- * of Unicode or ICU, so an applications should not rely on creating a permanent,
- * or difficult to update, database of skeletons. Instabilities result from
- * identifying new pairs or sequences of characters that are visually
- * confusable, and thus must be mapped to the same skeleton character(s).
+ * <p>
+ * This class, based on <a href="http://unicode.org/reports/tr36">Unicode Technical Report #36</a> and
+ * <a href="http://unicode.org/reports/tr39">Unicode Technical Standard #39</a>, has two main functions:
+ *
+ * <ol>
+ * <li>Checking whether two strings are visually <em>confusable</em> with each other, such as "Harvest" and
+ * "Ηarvest", where the second string starts with the Greek capital letter Eta.</li>
+ * <li>Checking whether an individual string is likely to be an attempt at confusing the reader (<em>spoof
+ * detection</em>), such as "paypal" with some Latin characters substituted with Cyrillic look-alikes.</li>
+ * </ol>
+ *
+ * <p>
+ * Although originally designed as a method for flagging suspicious identifier strings such as URLs,
+ * <code>USpoofChecker</code> has a number of other practical use cases, such as preventing attempts to evade bad-word
+ * content filters.
+ *
+ * <p>
+ * The functions of this class are exposed as C API, with a handful of syntactical conveniences for C++.
+ *
+ * <h2>Confusables</h2>
+ *
+ * <p>
+ * The following example shows how to use <code>USpoofChecker</code> to check for confusability between two strings:
+ *
+ * \code{.c}
+ * UErrorCode status = U_ZERO_ERROR;
+ * UChar* str1 = (UChar*) u"Harvest";
+ * UChar* str2 = (UChar*) u"\u0397arvest"; // with U+0397 GREEK CAPITAL LETTER ETA
+ *
+ * USpoofChecker* sc = uspoof_open(&status);
+ * uspoof_setChecks(sc, USPOOF_CONFUSABLE, &status);
+ *
+ * int32_t bitmask = uspoof_areConfusable(sc, str1, -1, str2, -1, &status);
+ * UBool result = bitmask != 0;
+ * // areConfusable: 1 (status: U_ZERO_ERROR)
+ * printf("areConfusable: %d (status: %s)\n", result, u_errorName(status));
+ * uspoof_close(sc);
+ * \endcode
+ *
+ * <p>
+ * The call to {@link uspoof_open} creates a <code>USpoofChecker</code> object; the call to {@link uspoof_setChecks}
+ * enables confusable checking and disables all other checks; the call to {@link uspoof_areConfusable} performs the
+ * confusability test; and the following line extracts the result out of the return value. For best performance,
+ * the instance should be created once (e.g., upon application startup), and the efficient
+ * {@link uspoof_areConfusable} method can be used at runtime.
+ *
+ * <p>
+ * The type {@link LocalUSpoofCheckerPointer} is exposed for C++ programmers. It will automatically call
+ * {@link uspoof_close} when the object goes out of scope:
+ *
+ * \code{.cpp}
+ * UErrorCode status = U_ZERO_ERROR;
+ * LocalUSpoofCheckerPointer sc(uspoof_open(&status));
+ * uspoof_setChecks(sc.getAlias(), USPOOF_CONFUSABLE, &status);
+ * // ...
+ * \endcode
+ *
+ * <p>
+ * UTS 39 defines two strings to be <em>confusable</em> if they map to the same <em>skeleton string</em>. A skeleton can
+ * be thought of as a "hash code". {@link uspoof_getSkeleton} computes the skeleton for a particular string, so
+ * the following snippet is equivalent to the example above:
+ *
+ * \code{.c}
+ * UErrorCode status = U_ZERO_ERROR;
+ * UChar* str1 = (UChar*) u"Harvest";
+ * UChar* str2 = (UChar*) u"\u0397arvest"; // with U+0397 GREEK CAPITAL LETTER ETA
+ *
+ * USpoofChecker* sc = uspoof_open(&status);
+ * uspoof_setChecks(sc, USPOOF_CONFUSABLE, &status);
+ *
+ * // Get skeleton 1
+ * int32_t skel1Len = uspoof_getSkeleton(sc, 0, str1, -1, NULL, 0, &status);
+ * UChar* skel1 = (UChar*) malloc(++skel1Len * sizeof(UChar));
+ * status = U_ZERO_ERROR;
+ * uspoof_getSkeleton(sc, 0, str1, -1, skel1, skel1Len, &status);
+ *
+ * // Get skeleton 2
+ * int32_t skel2Len = uspoof_getSkeleton(sc, 0, str2, -1, NULL, 0, &status);
+ * UChar* skel2 = (UChar*) malloc(++skel2Len * sizeof(UChar));
+ * status = U_ZERO_ERROR;
+ * uspoof_getSkeleton(sc, 0, str2, -1, skel2, skel2Len, &status);
+ *
+ * // Are the skeletons the same?
+ * UBool result = u_strcmp(skel1, skel2) == 0;
+ * // areConfusable: 1 (status: U_ZERO_ERROR)
+ * printf("areConfusable: %d (status: %s)\n", result, u_errorName(status));
+ * uspoof_close(sc);
+ * free(skel1);
+ * free(skel2);
+ * \endcode
+ *
+ * <p>
+ * If you need to check if a string is confusable with any string in a dictionary of many strings, rather than calling
+ * {@link uspoof_areConfusable} many times in a loop, {@link uspoof_getSkeleton} can be used instead, as shown below:
+ *
+ * \code{.c}
+ * UErrorCode status = U_ZERO_ERROR;
+ * #define DICTIONARY_LENGTH 2
+ * UChar* dictionary[DICTIONARY_LENGTH] = { (UChar*) u"lorem", (UChar*) u"ipsum" };
+ * UChar* skeletons[DICTIONARY_LENGTH];
+ * UChar* str = (UChar*) u"1orern";
+ *
+ * // Setup:
+ * USpoofChecker* sc = uspoof_open(&status);
+ * uspoof_setChecks(sc, USPOOF_CONFUSABLE, &status);
+ * for (size_t i=0; i<DICTIONARY_LENGTH; i++) {
+ * UChar* word = dictionary[i];
+ * int32_t len = uspoof_getSkeleton(sc, 0, word, -1, NULL, 0, &status);
+ * skeletons[i] = (UChar*) malloc(++len * sizeof(UChar));
+ * status = U_ZERO_ERROR;
+ * uspoof_getSkeleton(sc, 0, word, -1, skeletons[i], len, &status);
+ * }
+ *
+ * // Live Check:
+ * {
+ * int32_t len = uspoof_getSkeleton(sc, 0, str, -1, NULL, 0, &status);
+ * UChar* skel = (UChar*) malloc(++len * sizeof(UChar));
+ * status = U_ZERO_ERROR;
+ * uspoof_getSkeleton(sc, 0, str, -1, skel, len, &status);
+ * UBool result = FALSE;
+ * for (size_t i=0; i<DICTIONARY_LENGTH; i++) {
+ * result = u_strcmp(skel, skeletons[i]) == 0;
+ * if (result == TRUE) { break; }
+ * }
+ * // Has confusable in dictionary: 1 (status: U_ZERO_ERROR)
+ * printf("Has confusable in dictionary: %d (status: %s)\n", result, u_errorName(status));
+ * free(skel);
+ * }
+ *
+ * for (size_t i=0; i<DICTIONARY_LENGTH; i++) {
+ * free(skeletons[i]);
+ * }
+ * uspoof_close(sc);
+ * \endcode
+ *
+ * <p>
+ * <b>Note:</b> Since the Unicode confusables mapping table is frequently updated, confusable skeletons are <em>not</em>
+ * guaranteed to be the same between ICU releases. We therefore recommend that you always compute confusable skeletons
+ * at runtime and do not rely on creating a permanent, or difficult to update, database of skeletons.
+ *
+ * <h2>Spoof Detection</h2>
+ *
+ * <p>
+ * The following snippet shows a minimal example of using <code>USpoofChecker</code> to perform spoof detection on a
+ * string:
+ *
+ * \code{.c}
+ * UErrorCode status = U_ZERO_ERROR;
+ * UChar* str = (UChar*) u"p\u0430ypal"; // with U+0430 CYRILLIC SMALL LETTER A
+ *
+ * // Get the default set of allowable characters:
+ * USet* allowed = uset_openEmpty();
+ * uset_addAll(allowed, uspoof_getRecommendedSet(&status));
+ * uset_addAll(allowed, uspoof_getInclusionSet(&status));
+ *
+ * USpoofChecker* sc = uspoof_open(&status);
+ * uspoof_setAllowedChars(sc, allowed, &status);
+ * uspoof_setRestrictionLevel(sc, USPOOF_MODERATELY_RESTRICTIVE);
+ *
+ * int32_t bitmask = uspoof_check(sc, str, -1, NULL, &status);
+ * UBool result = bitmask != 0;
+ * // fails checks: 1 (status: U_ZERO_ERROR)
+ * printf("fails checks: %d (status: %s)\n", result, u_errorName(status));
+ * uspoof_close(sc);
+ * uset_close(allowed);
+ * \endcode
+ *
+ * <p>
+ * As in the case for confusability checking, it is good practice to create one <code>USpoofChecker</code> instance at
+ * startup, and call the cheaper {@link uspoof_check} online. We specify the set of
+ * allowed characters to be those with type RECOMMENDED or INCLUSION, according to the recommendation in UTS 39.
+ *
+ * <p>
+ * In addition to {@link uspoof_check}, the function {@link uspoof_checkUTF8} is exposed for UTF8-encoded char* strings,
+ * and {@link uspoof_checkUnicodeString} is exposed for C++ programmers.
+ *
+ * <p>
+ * If the {@link USPOOF_AUX_INFO} check is enabled, a limited amount of information on why a string failed the checks
+ * is available in the returned bitmask. For complete information, use the {@link uspoof_check2} class of functions
+ * with a {@link USpoofCheckResult} parameter:
+ *
+ * \code{.c}
+ * UErrorCode status = U_ZERO_ERROR;
+ * UChar* str = (UChar*) u"p\u0430ypal"; // with U+0430 CYRILLIC SMALL LETTER A
+ *
+ * // Get the default set of allowable characters:
+ * USet* allowed = uset_openEmpty();
+ * uset_addAll(allowed, uspoof_getRecommendedSet(&status));
+ * uset_addAll(allowed, uspoof_getInclusionSet(&status));
+ *
+ * USpoofChecker* sc = uspoof_open(&status);
+ * uspoof_setAllowedChars(sc, allowed, &status);
+ * uspoof_setRestrictionLevel(sc, USPOOF_MODERATELY_RESTRICTIVE);
projects / apple / icu.git / blobdiff