Sign language is the primary means of communication among the deaf community and the deaf with the normal people. It is the natural language for this community. They communicate using hand movements and other gestures. In this research, we try to develop a system that can help the deaf community communicate better with the rest of the world and amongst themselves. Building an accurate system that translates speech to sign language is of a great importance in order to facilitate the communication of this community. In this research, an Avatar Based Translation System from Amharic Speech to the Ethiopian Sign languageis developed for the Deaf people. According to WHO and World Bank report [68]; from the total population of Ethiopia 17.6% (about 14 million) have some kind of disabilities. Among these 3.5% (2.8 million) belongs to Deaf. It is believed that the Ethiopian Sign Language (ESL) has its origin in the American Sign Language with some influence form the Nordic countries [20]. Ethiopian Sign Language has noits own well studied grammar unlike Amharic language. Oursystem is made up of a speech recognizer (for decoding the spoken utterance into a word sequence), a natural language translator (for converting a word sequence into a sequence of signs belonging to the sign language), and a 3D avatar animation module (for playing back the hand movements). The system has beenevaluated in three steps. First the speech recognition module has evaluated separately and we get an accuracy of 6.88% Best WER with SGMM+MMI setup. With this research 238 Amharic Alphabets and Numbersand 417 Amharic Stem words animation scripts using SiGML format has developed. Those animations have evaluated with professional sign language teacher and 5 selected students and 83.94% of those signs are constructed correctly with suggestions to modify some sign animations. After separate evaluation of the two modules, the overall system has been evaluated by randomly selected 15 simple sentences that are read by three different test readers who did not participate in reading the training data of the ASR.From this final evaluation, we achieved 79.6% accuracy translating Amharic speech ESL. Wehave studied why the overall system performance has degraded compared to the separated modules evaluation and finally some future works and recommendation has suggested.