Awesome Unicode

A curated list of delightful Unicode tidbits, packages and resources.

Please read the contribution guidelines before contributing. Key Unicode terminology is defined in the glossary.

Cross posted to Wisdom’s Dev Blog

Foreword

Unicode is Awesome! Prior to Unicode, international communication was grueling- everyone had defined their separate extended character set in the upperhalf of ASCII (called Code Pages) that would conflict- Just think, German speakers coordinating with Korean speakers over which 127 character Code Page to use. Thankfully the Unicode standard caught on and unified communication. Unicode 8.0 standardizes over 120,000 characters from over 129 scripts - some modern, some ancient, and some still undeciphered. Unicode handles left-to-right and right-to-left text, combining marks, and includes diverse cultural, political, religious characters and emojis. Unicode is awesomely human - and ultimately underappreciated.

Contents

• Quick Unicode Background
  • What Characters Does the Unicode Standard Include?
  • Unicode Character Encodings
  • Lets talk Numbers
  • UTF-16 Surrogate Pairs
  • Calculating Surrogate Pairs
  • Composing & Decomposing
  • Myths of Unicode
  • Applied Unicode Encodings
  • Source Code
• Awesome Characters List
  • Special Characters
  • Variable identifiers can effectively include whitespace!
  • Modifiers
  • Uppercase Transformation Collisions
  • Lowercase Transformation Collisions
• Quirks and Troubleshooting
  • One-To-Many Case Mappings
• Awesome Packages & Libraries
• Emojis
  • Diversity
• Creatively Naming Variables and Methods
  • Recursive HTML Tag Renaming Script
• Unicode Fonts
• More Reading
• Exploring Deeper into Unicode Yourself
• Overview Map
  • A map of the Basic Multilingual Plane
  • Unicode Blocks
• Principles of the Unicode Standard
• Unicode Versions
• Contributing
• Code of Conduct
• License

Quick Unicode Background

What Characters Does the Unicode Standard Include?

The Unicode Standard defines codes for characters used in all the major languages written today. Scripts include the European alphabetic scripts, Middle Eastern right-to-left scripts, and many scripts of Asia.

The Unicode Standard further includes punctuation marks, diacritics, mathematical symbols, technical symbols, arrows, dingbats, emoji, etc. It provides codes for diacritics, which are modifying character marks such as the tilde (~), that are used in conjunction with base characters to represent accented letters (ñ, for example). In all, the Unicode Standard, Version 9.0 provides codes for 128,172 characters from the world’s alphabets, ideograph sets, and symbol collections.

The majority of common-use characters fit into the first 64K code points, an area of the codespace that is called the basic multilingual plane, or BMP for short. There are sixteen other supplementary planes available for encoding other characters, with currently over 850,000 unused code points. More characters are under consideration for addition to future versions of the standard.

The Unicode Standard also reserves code points for private use. Vendors or end users can assign these internally for their own characters and symbols, or use them with specialized fonts. There are 6,400 private use code points on the BMP and another 131,068 supplementary private use code points, should 6,400 be insufficient for particular applications.

Unicode Character Encodings

Character encoding standards define not only the identity of each character and its numeric value, or code point, but also how this value is represented in bits.

The Unicode Standard defines three encoding forms that allow the same data to be transmitted in a byte, word or double word oriented format (i.e. in 8, 16 or 32-bits per code unit). All three encoding forms encode the same common character repertoire and can be efficiently transformed into one another without loss of data. The Unicode Consortium fully endorses the use of any of these encoding forms as a conformant way of implementing the Unicode Standard.

UTF-8 is popular for HTML and similar protocols. UTF-8 is a way of transforming all Unicode characters into a variable length encoding of bytes. It has the advantages that the Unicode characters corresponding to the familiar ASCII set have the same byte values as ASCII, and that Unicode characters transformed into UTF-8 can be used with much existing software without extensive software rewrites.

UTF-16 is popular in many environments that need to balance efficient access to characters with economical use of storage. It is reasonably compact and all the heavily used characters fit into a single 16-bit code unit, while all other characters are accessible via pairs of 16-bit code units.

UTF-32 is useful where memory space is no concern, but fixed width, single code unit access to characters is desired. Each Unicode character is encoded in a single 32-bit code unit when using UTF-32.

All three encoding forms need at most 4 bytes (or 32-bits) of data for each character.

Lets talk Numbers

The Unicode characterset is divided into 17 core segments called “planes”, which are further divided into blocks. Each plane has space for 65,536 (2¹⁶) codepoints, supporting a grand total of 1,114,112 codepoints. There are two “Private Use Area” planes (#16 & #17) that are allocated to be used however one wishes. These two Private Use planes account for 131,072 codepoints.

The first plane is called the Basic Multilingual Plane or BMP. It contains the code points from U+0000 to U+FFFF, which are the most frequently used characters. The other sixteen planes (U+010000 → U+10FFFF) are called supplementary planes or astral planes.

UTF-16 Surrogate Pairs

Characters outside the BMP, e.g. U+1D306 tetragram for centre (𝌆), can only be encoded in UTF-16 using two 16-bit code units: 0xD834 0xDF06. This is called a surrogate pair. Note that a surrogate pair only represents a single character. The first code unit of a surrogate pair is always in the range from 0xD800 to 0xDBFF, and is called a high surrogate or a lead surrogate. The second code unit of a surrogate pair is always in the range from 0xDC00 to 0xDFFF, and is called a low surrogate or a trail surrogate.

– Mathias Bynens

Surrogate pair: A representation for a single abstract character that consists of a sequence of two 16-bit code units, where the first value of the pair is a high-surrogate code unit and the second value is a low-surrogate code unit. Surrogate pairs are used only in UTF-16. (See Section 3.9, Unicode Encoding Forms.) – Unicode 8.0.0 Chapter 3 - Surrogates

Calculating Surrogate Pairs

The Unicode character 💩 Pile of Poo (U+1F4A9) in UTF-16 must be encoded as a surrogate pair, i.e. two surrogates. To convert any code point to a surrogate pair, use the following algorithm (in JavaScript). Keep in mind that we’re using hexidecimal notation.

 var High_Surrogate = function(Code_Point){ return Math.floor((Code_Point - 0x10000) / 0x400) + 0xD800 };
 var Low_Surrogate  = function(Code_Point){ return (Code_Point - 0x10000) % 0x400 + 0xDC00 };

 // Reverses The Conversion
 var Code_Point = function(High_Surrogate, Low_Surrogate){
    return (High_Surrogate - 0xD800) * 0x400 + Low_Surrogate - 0xDC00 + 0x10000;
 };

 > var codepoint = 0x1F4A9;                                 // 0x1F4A9 == 128169
 > High_Surrogate(codepoint).toString(16)
 "d83d"                                                     // 0xD83D == 55357
 > Low_Surrogate(codepoint).toString(16)
 "dca9"                                                     // 0xDCA9 == 56489

 > String.fromCharCode(  High_Surrogate(codepoint) , Low_Surrogate(codepoint) );
  "💩"
> String.fromCodePoint(0x1F4A9)
  "💩"
 > '\ud83d\udca9'
  "💩"

Composing & Decomposing

Unicode includes a mechanism for modifying character shape that greatly extends the supported glyph repertoire. This covers the use of combining diacritical marks. They are inserted after the main character. Multiple combining diacritics may be stacked over the same character. Unicode also contains precomposed versions of most letter/diacritic combinations in normal use.

Certain sequences of characters can also be represented as a single character, called a precomposed character (or composite or decomposible character). For example, the character “ü” can be encoded as the single code point U+00FC “ü” or as the base character U+0075 “u” followed by the non-spacing character U+0308 “¨”. The Unicode Standard encodes precomposed characters for compatibility with established standards such as Latin 1, which includes many precomposed characters such as “ü” and “ñ”.

Precomposed characters may be decomposed for consistency or analysis. For example, in alphabetizing (collating) a list of names, the character “ü” may be decomposed into a “u” followed by the non-spacing character “¨”. Once the character has been decomposed, it may be easier for the collation to work with the character because it can be processed as a “u” with modifications. This allows easier alphabetical sorting for languages where character modifiers do not affect alphabetical order. The Unicode Standard defines the decompositions for all precomposed characters. It also defines normalization forms to provide for unique representations of characters.

Myths of Unicode

From Mark Davis’s Unicode Myths slides. - Unicode is simply a 16-bit code - Some people are under the misconception that Unicode is simply a 16-bit code where each character takes 16 bits and therefore there are 65,536 possible characters. This is not, actually, correct. It is the single most common myth about Unicode, so if you thought that, don’t feel bad.

• You can use any unassigned codepoint for internal use - No. Eventually that hole will be filled with a different character. Instead use private use or noncharacters.

• Every Unicode code point represents a character - No. There are lots of nonCharacters (FFFE, FFFF, 1FFFE,…) There are also surrogate code points, private and unassigned codepoints, and control/format “characters” (RLM, ZWNJ,…)

• Unicode will run out of space - If it were linear, we would run out in 2140 AD. But it isn’t linear. See http://www.unicode.org/roadmaps/

• Case mappings are 1-1 - No. They can also be:
  • One-to-many: (ß → SS )
  • Contextual: (…Σ ↔ …ς AND …ΣΤ… ↔ …στ… )
  • Locale-sensitive: ( I ↔ ı AND İ ↔ i )

Applied Unicode Encodings

Source Code

Awesome Characters List

Special Characters

The Unicode Consortium published a general punctuation chart where you can find more details.

#### Wait a second… what did I just read?

## Variable identifiers can effectively include whitespace!

The U+3164 HANGUL FILLER character displays as an advancing whitespace character. The character is rendered as completely invisible (and non advancing, i.e. “zero width”), if not explicitly supported in rendering. That means the ugly character replacement (�) symbol should never be displayed.

I’m not yet sure why U+3164 was specified to behave this way. Interestingly, U+3164 was added to Unicode in version 1.1 (1993)- so the consortium must have had a lot of time to think it through. Anyway, here are a few examples.

> var ᅟ = 'foo';
undefined
> ᅟ
'foo'


> var ㅤ= alert;
undefined
> var foo = 'bar'
undefined
> if ( foo ===ㅤ`baz` ){}    // alert
undefined


> var varㅤfooㅤ\u{A60C}ㅤπ = 'bar';
undefined
> varㅤfooㅤꘌㅤπ
'bar'

NOTE: I’ve tested U+3164 rendering on Ubuntu and OS X with the following: node, php, ruby, python3.5, scala ,vim, cat, chrome+github gist. Atom is the only system that fails by (incorrectly) displaying empty boxes. I have yet to test it out on Emacs and Sublime. From what I understand, the Unicode Consortium will not reassign or rename characters or codepoints, but may be convinced to change character properties like ID_Start/ID_Continue.

Modifiers

The zero-width joiner (ZWJ) is a non-printing character used in the computerized typesetting of some complex scripts such as the Arabic script or any Indic script. When placed between two characters that would otherwise not be connected, a ZWJ causes them to be printed in their connected forms.

The zero-width non-joiner (ZWNJ) is a non-printing character used in the computerization of writing systems that make use of ligatures. When placed between two characters that would otherwise be connected into a ligature, a ZWNJ causes them to be printed in their final and initial forms, respectively. This is also an effect of a space character, but a ZWNJ is used when it is desirable to keep the words closer together or to connect a word with its morpheme.

> 'a'
 "a"

> 'a\u{0308}'
 "ä"

> 'a\u{20DE}\u{0308}'
 "a⃞̈"

> 'a\u{20DE}\u{0308}\u{20DD}'
 "a⃞̈⃝"

// Modifying Invisible Characters
> '\u{200E}\u{200E}\u{200E}\u{200E}\u{200E}\u{200E}\u{200E}\u{200E}\u{200E}\u{200E}'
 "‎‎‎‎‎‎‎‎‎‎"

> '\u{200E}\u{200E}\u{200E}\u{200E}\u{200E}\u{200E}\u{200E}\u{200E}\u{200E}\u{200E}'.length
 10

:collision: Uppercase Transformation Collisions

:collision: Lowercase Transformation Collisions

Quirks and Troubleshooting

• String length is typically determined by counting codepoints. This means that surrogate pairs would count as two characters. Combining multiple diacritics may be stacked over the same character. a + ̈ == ̈a, increasing length, while only producing a single character.

• Similarily, reversing strings often is a non-trivial task. Again, surrogate pairs and diacritics must be reversed together. ES Reverser provides a pretty good solution.

• Upper and lower case mappings are not always one-to-one. They can also be:
  • One-to-many: (ß → SS )
  • Contextual: (…Σ ↔ …ς AND …ΣΤ… ↔ …στ… )
  • Locale-sensitive: ( I ↔ ı AND İ ↔ i )

One-To-Many Case Mappings

Most of the below characters express their one-to-many case mappings when uppercased- while others should be lowercased. This list should be split up

Awesome Packages & Libraries

• PhantomScript - :ghost: :flashlight: Invisible JavaScript code execution & social engineering
• ESReverser - A Unicode-aware string reverser written in JavaScript.
• mimic - [ab]using Unicode to create tragedy
• python-ftfy - Given Unicode text, make its representation consistent and possibly less broken.
• vim-troll-stopper - Stop Unicode trolls from messing with your code.

Emojis

• Unicode Consortium’s Emoji Chart
• Emojipedia - Information about specific emoji, news blog.
• emojitracker - Realtime emoji use on Twitter.
• World Translation Foundation - A way to promote, explore, and translate the written word into the pictorial alphabet of Emoji.
• Can I Emoji? - Displays the current status of native Emoji support across iOS, Android and Windows.
• How to register an emoji URL

Diversity

The Unicode Consortium has made a huge effort better reflect and incorporate human diversity, including cultural practices. Here is the Consortium’s diversity report.

Emojis of mixed gender situations are now available, such as same sex families, holding hands, and kissing. The real kicker are Emoji combined sequences. Basically:

Code Points	Recipe	Combined
U+1F469 U+200D U+2764 U+FE0F U+200D U+1F469
U+1F468 U+200D U+1F468 U+200D U+1F467 U+200D U+1F466

Further, emojis now support skin color modifiers.

Five symbol modifier characters that provide for a range of skin tones for human emoji were released in Unicode Version 8.0 (mid-2015). These characters are based on the six tones of the Fitzpatrick scale, a recognized standard for dermatology (there are many examples of this scale online, such as FitzpatrickSkinType.pdf). The exact shades may vary between implementations. – Unicode Consortium’s Diversity report

Code	Name	Samples
U+1F3FB	EMOJI MODIFIER FITZPATRICK TYPE-1-2
U+1F3FC	EMOJI MODIFIER FITZPATRICK TYPE-3
U+1F3FD	EMOJI MODIFIER FITZPATRICK TYPE-4
U+1F3FE	EMOJI MODIFIER FITZPATRICK TYPE-5
U+1F3FF	EMOJI MODIFIER FITZPATRICK TYPE-6

Just follow the desired Emoji with one of the skin color modifiers \u{1F466}\u{1F3FE}.

+   → 

Creatively Naming Variables and Methods

Examples are written in JavaScript (ES6)

In general, characters designated the ID_START property may be used at the beggining of a variable name. Characters designated with the ID_CONTINUE property may be used after the first character of a variable.

function rand(μ,σ){ ... };

String.prototype.reverseⵑ = function(){..};

Number.prototype.isTrueɁ = function(){..};

var WhatDoesThisDoɁɁɁɁ = 42

Here are some really creative variable names from Mathias Bynes

// How convenient!
var π = Math.PI;

// Sometimes, you just have to use the Bad Parts of JavaScript:
var ಠ_ಠ = eval;

// Code, Y U NO WORK?!
var ლ_ಠ益ಠ_ლ = 42;

// How about a JavaScript library for functional programming?
var λ = function() {};

// Obfuscate boring variable names for great justice
var \u006C\u006F\u006C\u0077\u0061\u0074 = 'heh';

// …or just make up random ones
var Ꙭൽↈⴱ = 'huh';

// While perfectly valid, this doesn’t work in most browsers:
var foo\u200Cbar = 42;

// This is *not* a bitwise left shift (`<<`):
var 〱〱 = 2;
// This is, though:
〱〱 << 〱〱; // 8

// Give yourself a discount:
var price_9̶9̶_89 = 'cheap';

// Fun with Roman numerals
var Ⅳ = 4;
var Ⅴ = 5;
Ⅳ + Ⅴ; // 9

// Cthulhu was here
var Hͫ̆̒̐ͣ̊̄ͯ͗͏̵̗̻̰̠̬͝ͅE̴̷̬͎̱̘͇͍̾ͦ͊͒͊̓̓̐_̫̠̱̩̭̤͈̑̎̋ͮͩ̒͑̾͋͘Ç̳͕̯̭̱̲̣̠̜͋̍O̴̦̗̯̹̼ͭ̐ͨ̊̈͘͠M̶̝̠̭̭̤̻͓͑̓̊ͣͤ̎͟͠E̢̞̮̹͍̞̳̣ͣͪ͐̈T̡̯̳̭̜̠͕͌̈́̽̿ͤ̿̅̑Ḧ̱̱̺̰̳̹̘̰́̏ͪ̂̽͂̀͠ = 'Zalgo';

And here’s some Unicode CSS Classes from David Walsh

<!-- place this within the document head -->
<meta charset="UTF-8" />

<!-- error message -->
<div class="ಠ_ಠ">You do not have access to this page.</div>

<!-- success message -->
<div class="❤">Your changes have been saved successfully!</div>

.ಠ_ಠ {
    border: 1px solid #f00;
}

.❤ {
    background: lightgreen;
}

Recursive HTML Tag Renaming Script

If you want to rename all your HTML tags to what appears as nothing, the following script is just what your looking for.

Do note however that HTML does not support all unicode characters.

// U+1160 HANGUL JUNGSEONG FILLER
transformAllTags('ᅠ');

// An actual HTML element node designed to look like a comment node, using the U+01C3 LATIN LETTER RETROFLEX CLICK 
//  <ǃ-- name="viewport" content="width=device-width"></ǃ-->
transformAllTags('ǃ--');

// or even <ᅠ⃝
transformAllTags('\u{1160}\u{20dd}');

// and for a bonus, all existing tag names will have each character ensquared. h⃞t⃞m⃞l⃞
transformAllTags();


function transformAllTags (newName){
   // querySelectorAll doesn't actually return an array.
   Array.from(document.querySelectorAll('*'))
     .forEach(function(x){
         transformTag(x, newName);
   });
}

function wonky(str){
  return str.split('').join('\u{20de}') + '\u{20de}';
}

function transformTag(tagIdOrElem, tagType){
    var elem = (tagIdOrElem instanceof HTMLElement) ? tagIdOrElem : document.getElementById(tagIdOrElem);
    if(!elem || !(elem instanceof HTMLElement))return;
    var children = elem.childNodes;
    var parent = elem.parentNode;
    var newNode = document.createElement(tagType||wonky(elem.tagName));
    for(var a=0;a<elem.attributes.length;a++){
        newNode.setAttribute(elem.attributes[a].nodeName, elem.attributes[a].value);
    }
    for(var i= 0,clen=children.length;i<clen;i++){
        newNode.appendChild(children[0]); //0...always point to the first non-moved element
    }
    newNode.style.cssText = elem.style.cssText;
    parent.replaceChild(newNode,elem);
}

Here is what it does support:

function testBegin(str){
 try{
    eval(`document.createElement( '${str}' );`)
    return true;
 }
 catch(e){ return false; }
}

function testContinue(str){
 try{
    eval(`document.createElement( 'a${str}' );`)
    return true;
 }
 catch(e){ return false; }
}

And heres some basic results

// Test if dashes can start an HTML Tag
> testBegin('-')
< false

> testContinue('-')
< true

> testBegin('ᅠ-')   // Prepend dash with U+1160 HANGUL JUNGSEONG FILLER
< true

Unicode Fonts

A single TrueType / OpenType font format cannot cover all UTF-8 characters as there is a hard limit of 65535 glyphs in a font. Since there are over 1.1 million UTF-8 glphys, you will need to use a font-family to cover them all. - https://en.wikipedia.org/wiki/Unicode_font#List_of_Unicode_fonts - http://www.unifont.org/fontguide/

More Reading

• The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets - By Joel Spolsky
• What Every Programmer Absolutely, Positively Needs To Know About Encodings And Character Sets To Work With Text
• The Unicode Consortium’s Recommended Reading List
• Space Yourself - Smashing Magazine’s Spacing Guide
• JavaScript has a Unicode Problem
• Creative usernames and Spotify account hijacking

Exploring Deeper into Unicode Yourself

• Shapecatcher - Draw the character you’re looking for.
• Confusable Unicode Characters
• Unicode Character Database
• Database Dumps of Codepoints.net
• Unicode Blocks List
• Unicode Character Code Charts
• Unicode Case Charts
• Unicode Normalization Chart
• Unicode FAQ

Overview Map

A map of the Basic Multilingual Plane

The Chinese, Japanese and Korean (CJK) scripts share a common background, collectively known as CJK characters. In the process called Han unification, the common (shared) characters were identified and named “CJK Unified Ideographs”.

Unicode Blocks

The Unicode standard arranges groups of characters together in blocks. This is the complete list of blocks across all 17 planes.

Principles of the Unicode Standard

The Unicode Standard set forth the following fundamental principles:

• Universal repertoire - Every writing system ever used shall be respected and represented in the standard
• Logical order - In bidirectional text are the characters stored in logical order, not in a way that the representaion
• Efficiency - The documentation must be efficient and complete.
• Unification - Where different cultures or languages use the same character, it shall be only included once. This point is
• Characters, not glyphs - Only characters, not glyphs shall be encoded. In a nutshell, glyphs are the actual graphical
• Dynamic composition - New characters can be composed of other, already standardized characters. For example, the character “Ä” can be composed of an “A” and a dieresis sign (“ ¨ ”).
• Semantics - Included characters must be well defined and distinguished from others.
• Stability - Once defined characters shall never be removed or their codepoints reassigned. In the case of an error, a codepoint shall be deprecated.
• Plain Text - Characters in the standard are text and never mark-up or metacharacters.
• Convertibility - Every other used encoding shall be representable in terms of a Unicode encoding.

Note: Principle descriptions are from codepoints.net

Unicode Versions

• Version 9.0.0 (Latest Version, August 2016 - adds exactly 7,500 characters)
• Version 8.0.0
• Version 7.0.0
• Version 6.3.0
• Version 6.2.0
• Version 6.1.0
• Version 6.0.0
• Version 5.2.0
• Version 5.1.0
• Version 5.0.0 (unavailable)
• Version 4.0.1
• Version 4.0.0

Contributing

See the Awesome Unicode contribution guide for details on how to contribute.

Code of Conduct

See the Code of Conduct for details. Basically it comes down to: >In the interest of fostering an open and welcoming environment, we as contributors and maintainers pledge to making participation in our project and our community a harassment-free experience for everyone, regardless of age, body size, disability, ethnicity, gender identity and expression, level of experience, nationality, personal appearance, race, religion, or sexual identity and orientation.

License

To the extent possible under law, the contributors have waived all copyright and related or neighboring rights to this work. See the license file for details.