Speaker recognition is the technique of recognizing automatically, who is speaking on the source of individual information built-in in speech waves. It uses the speaker's voice to confirm their identity and provides control access for services such as voice calls, database access facilities, information services, voice mail, and security control for private information areas, and remote access to computers, and for other several fields. Research within the area of speaker recognition has now crossed over five decades and has shown positive results under controlled conditions and having big data sets. Besides, almost all of the previous study uses traditional methods like MFCC and LPCC as feature extraction and GMM, VQ and HMM, etc. as classifiers. These features generally require the domain knowledge for researchers to be designed and take a long time to be tested and selected for high classification performance. And also concerning the requirement of realistic applications, the system should take into account the problems related to memory and computational resource limitation. So, in this thesis, we tried to address the short utterance data sets in the real circumstance by applying an end-to-end Convolutional Neural Network and CNN with SVM. Since CNN training usually requires large sample data and has the overfitting problem. On the other hand, SVM has good generalization ability and can solve the small sample problem. For the preparation of the data, audio signals are gathered from the Bahir Dar city by directly recording sounds from Amharic language speakers using an audio recorder. A total of 50 datasets were collected having 10 samples in each speaker. Each speaker had 10 seconds of waiting time to speak over the recorder, after being collected and pre-processed; each pre-processed voice was transformed into a spectrogram image. Then, the spectrogram images were used as an input to the proposed model to learn speaker-specific spectral features. To see the effect of gender on speaker identification for the Amharic language, we have experimented with 25 male and 25 female speakers. Then, we have found an accuracy of 76 % and 92 % for males and females respectively. Finally, we compared the end-to-end and combination of CNN with SVM using a 50 speakers' voice dataset. Then we have achieved an accuracy of 82%, and 95%, for an end-to-end Convolutional Neural Network, and convolutional neural network with a support vector machine model respectively.