2.2.1 Characters and Character Encodings

A character can be (rather loosely) defined as basic element of a script.[2] In other words, characters are the elements from which larger textual units like words are created.

An important point here is that a character is not the same as a glyph in a font. A glyph is a specific visual shape that represents a character. A glyph representing the same character could be rendered in many different ways (see Figure 2-2). One way of understanding the difference is that a character is an abstract quantity which could be represented by one or a combination of glyphs in a font.

The first step in representing a script on a computer is to list all the valid characters that comprise a script. For characters that in common use, this is straightforward, but for scripts and languages that have been around for centuries, there will be historical characters that need to be either included or excluded from our list of valid characters. Similarly there could be variants of the same basic character in use in different regions; we need to decide whether such variants deserve to be treated separately or whether they can be lumped together as one character itself.

Most scripts have distinct classes of characters with distinct meanings. Some characters are used to separate words (i.e. punctuation) and some denote numbers and some denote vowels and some consonants.

Indic scripts tends to have more kinds of characters.

• Some script elements serve the role of intonation modifiers: for example, they indicate that a vowel sound has to be prolonged.

• Other script elements are modifiers, meaning that they do not occur by themselves, but only in conjunction with another, more primary character.

• Some Indic scripts have two classes of otherwise phonetically similar characters: one class has characters which can combine with other characters and the other comprises of characters that always occur in uncombined form.

Deciding the set of characters of a script and their semantic properties requires linguistic expertise. Once this list is prepared, we refer to it as character set[3].

A character set has the following properties:

• It contains all the distinct elements of the script that are of interest to us.

• It does not contain trivial variations of a character (say, alternate visual forms). Each character in a character set has a distinct semantic meaning.

• All the different semantic classes of characters of the script are identified and represented.

After the composition of a character set has been decided, we can then represent it in a form suitable for processing in a digital computer. Computers, you may recall from Section 2.1, process symbols or bit patterns. The process of representing the characters of a script involves mapping them to bit-patterns that a computer can then manipulate.

For example, we could choose the bit pattern 01100001 to represent the character ' a ' ( LATIN SMALL LETTER A ) in the Roman script[4]. Since it is not very convenient to use binary strings, we normally write these bit-patterns in numeric form: 01100001 is written as decimal 97 or as hexadecimal 0x61.

The mapping between a bit pattern used and the character it represents is called a character encoding. The mapping for all the characters in a character set is similarly called a character set encoding[5]. The numeric value of each bit pattern is termed its code point.

Now, if we can get industry wide acceptance of our character encoding scheme, then it would be possible for us to transfer data from one computer to another without loss of information, since both the sender and recipient would have agreed on the mappings between the bit-patterns in the data and the characters they are to represent. A widely accepted character encoding is called a standard encoding.

The important thing to note here is that encodings are just conventions. For example, decimal 97 or 01100001 denotes the character ' a ' in the ASCII character set encoding and the character ' / ' (SOLIDUS) in the IBM EBCDIC character set encoding.

Further, a given language can have more than one encoding. For example, the Chinese language has multiple encodings in widespread use; for example: Big5 and EUC.

Why would a script have multiple encodings? A given script may have multiple encodings on account of many reasons. Historical reasons: different manufacturers may have designed and implemented their own encodings before standardization took place. For example, IBM's EBCDIC character set encoding was implemented before the ASCII encoding was formalized. The different encodings could have been designed with different goals in mind: for space efficiency (for example, the Shift-JIS encoding for Japanese, is more space efficient than the EUC encoding), or perhaps for the ease of sorting and collating text, or for compatibility with an existing encoding. There could have been errors in the standard encoding requiring a new encoding to be defined. For example, the ISCII (Indian Standard Code for Information Interchange) standard's support for Tamil and Kannada was sufficiently inferior that these respective language groups formulated their own character encoding standards (TSCII and KSCLP respectively).