Habesha Kemis is the conventional clothing of ladies in Ethiopia which is worn during ceremonial and formal events. Identifying and inspecting Habesha Kemis Cloth fabric is based on the human experts. Various researchers have been done the identification of clothing whether it is a dress, t-shirt, skirt, trouser, shoes, jacket, jeans, leaser and the like. Automatic identification of nearly the same color cloths was neglected in previous studies. In addition most of the previously proposed methods for identifying cloths have focused hand crafted feature extraction techniques. Thus, in this work, we present a system for automatic identification of Habesha Kemis Cloth fabrics which have almost nearly the same color and we have used the combination of CNN and handcrafted texture descriptor. The proposed system has three main components: preprocessing, feature extraction and classification. In the preprocessing stage, we normalize the image to a standard size, converting the color image into Grayscale, and contrast enhancement using SSR and MSR techniques. In feature extraction, we apply Gabor filter, LBP, GLCM and CNN on the image dataset of Habesha Kemis Cloth fabrics. It is used to select the important features that account for the identification of Habesha Kemis Cloth fabric. For classification, we have used Convolutional Neural Network (CNN) and Support Vector Machine (SVM). In case of end-to-end CNN, a five-way softmax classifier was used for categorizing into specific classes (i.e, Abujedie, Fasha, Magg, Mennen and Shash). The proposed model is implemented using keras (with Tensorflow as a back end) in python 3.7 and tasted with sample images taken from Habesha Kemis shops found in Gondar and Bahir Dar. Our CNN model achieved 96% testing accuracy. SVM model using each of the feature extractors that we have used as feature extractor (i. e CNN, GLCM, LBP and Gabor filter) feature vectors achieved, 96%, 90%, 87%, 93% accuracy respectively and with the combined feature vectors of (CNN and Gabor filter) it was achieved 99% accuracy. Keywords: Habesha Kemis, Deep learning, CNN, GLCM, LBP, Gabor filter, SSR and MSR