Background     Problem Defnition     Solution     Issues     Prototype     Current Work    Publications

Multimodal Interface


Today, there are a lot of local language applications coming up.  Till date, keyboard has been used as an input interface.  But, typically in any Indian language, there are about 3000 character clusters.  All the characters of an Indian language cannot be mapped to 256 scan codes provided by our normal keyboard.  Moreover, designing a keyboard with keys for each character becomes unmanageable.  Hence, we use a sequence of key strokes to type a character.  Thus, data entry in local language becomes a pain.  More details available here

Problem Definition

Hence, the primary focus is to develop a more natural, and user-friendly interface for interaction with the system, mainly for local language applications.

Our Solution

In Multimodal Interface (MMI), speech, and handwriting interfaces are provided in addition to the keyboard.  They supplement the keyboard, but donot replace it.

Our goal is to develop a multimodal interface to the computer that is relevant for India, i.e., one that enables Indic computing.  The components of this Indian language interface are:


All three interfaces - keyboard, handwriting, and speech has been integrated seamlessly at the ToolKit level (Qt has been used for this).  The interface currently used redirects the recognized output to the control having focus. 

Speech Interface

Isolated word recognition has been used for the speech interface.  Here, user utters a word, and it is recognized using the word models, generated earlier using train data.  It is vocabulary dependent, but gives higher recognition rates for limited vocabulary tasks.

Handwriting Interface

Online handwritten character recognition (OHCR) has been employed for the handwriting interface.  Here, the user writes a character using a tablet, on the HWR screen.  It is recognized based on the component set of strokes.  A stroke is the meaningful collection of points, between the pen down and the most earliest pen up events, which make a part of any character.

Keyboard Interface

For typing in local language, our X level modification has been used.  It is application independent. 


A prototype for multimodal interface has been developed for Railway Availability Enquiry System .  

Current Work

Current work focusses on enabling the web-based forms with multimodal interface.  It means that any online form can be filled with the multimodal interface, thereby easing the data entry.      More details


  1. N Hemalatha, M Kasirajan, M Palani Selvam, M S Vidhya, G Vijay Prakash, T Nagarajan, and Hema A Murthy, "Multimodal Interface Enabled Web-based Form Filling Applications"