2.1 Introduction

Computers are introduced to neophytes as ``number crunching machines'', as devices related to the ancient abacus, only more electronic. Yet, for the most part, the modern computer spends an insignificant portion of its time doing numeric calculation. Instead, the great utility of computers comes from the fact that they are excellent symbol manipulation machines.

To a computer, human readable text is merely a sequence of symbols; symbols that it can process and transform at blinding speed, in the precise manner written down in the software program that it is executing.

Here is a list of a few typical text transformation processes that computers are called on to do:

• They render text into readable ``characters'' either on a display device or onto a printed page. We expect computers to do this correctly, taking into account complex typographical features like ligatures and context dependent rendering (see the sidebar Ligatures and context dependent rendering).

  Ligatures and context dependent rendering Many scripts have complex rendering conventions. For example, contrast the two ways the strings 'fi' and 'fl' in the Roman script have been typeset below. The ligatured forms are used in high-quality typography. Scripts like Arabic use ligatures extensively. For such scripts, the ability of software to render ligatures is a crucial requirement. Text without ligatures Text with ligatures Further, the correct way to render a character can depend on the surrounding context. The German character Eszett has two valid displayed forms: depending on the surrounding context it should be displayed as the two letters 'ss' or as the letter shape 'ß'.

• We expect the computer to be able to change the rendered appearance of the text: for example, we may want a particular part of the displayed text to be rendered with a different weight (bold, medium, etc.) or with a different font entirely.

• Computers take in input from a keyboard or other input device. For some scripts and languages this process is straightforward: a user would type at a keyboard that possesses one key per ``character'' of the script.

  However, some scripts are so complex that this simple approach is out of the question. Indic and Far-Eastern scripts require the computer to give considerable assistance to the user keying in data. For example, Kanji (Chinese and Japanese) ideograms are input by requiring the user to select the required character from thousands of alternatives.

• Computers are expected to assist in the process of selection and editing portions of the displayed text.

  The end-user expects this process to be straightforward; in typical graphical user interfaces regions of text are manipulated using mouse gestures (clicking, dragging etc).

  However, the process of mapping between the display and the underlying characters is not straightforward for some scripts. The Indic scripts, for example, use a visual ordering that is different from the manner text is represented internally in the computer. For such scripts the mapping between the internal representation and the visual presentation can be quite complex.

• Computers are expected to know how to determine what constitutes a ``word'', a ``sentence'', a ``stanza'' or ``paragraph'' in the text.

  We expect the computer to be able to analyze the morphological structure of text: for example, extract word roots, stems and suffixes, perform grammar checks on sentences and the like.

  This requirement implies that the controlling software needs to know about the linguistic properties of the text elements that it is manipulating.

• Similarly, we expect software to be able to break lines correctly, and to hyphenate words correctly.

  Some languages like Sanskrit, when written using the Devanagari script tend to have very long ``words''. For such scripts, the rules that determine where a word can be hyphenated can be quite complex (see Figure 2-1).

  Figure 2-1. Hyphenation positions (Devanagari)

  

• Computers are expected to easily sort, filter, and search in textual data; indeed much of the value of information technology comes from the ability of computers to do these tasks well.

  These functions are not always easy to implement correctly as some scripts in use in the world today have multiple conflicting ways of being represented inside the computer.

• Computers are expected to be able to compress text or otherwise prepare it for transmission, archival and other kinds of processing. Further we expect to be able to recover the original text from the transformed version without error.

Clearly, these diverse tasks are possible only if the underlying software architecture has been designed to enable these in the first place. In subsequent sections of this tutorial we will examine the foundations over which computer processing of text is built.