Our country Ethiopia has many cultures, from this Habesha Kemis is one of the clothing cultures that the women wear in most places and it is common in Amhara regional state. This Habesha Kemis seems the same but it has different texture, shapes, and colors on the embroidery design of Habesha Kemis. This embroidery design represents the culture of Gojjam, Gondar, and Wollo Habesha Kemis. However, due to the development of fashion design on clothes, the pure cultural embroidery design of the Habesha Kemis becomes replaced by different fashion style and it loses its cultural identity. There are many types of research on cloth pattern and color detection, recognition, and classification, but clothes that have the same pattern and color are difficult to classify into a different class. This research focused on classifying Gondar, Gojjam, and Wollo Habesha Kemis based on the embroidery design by using computer vision and machine learning approach. The image of Habesha Kemis was captured from Bahir Dar and Addis Ababa Habesha Kemis shops. The size of these collected images is large and reduced into 224x224 during the image preprocessing stage. We used a Conventional neural network to extract deep features and we used handcrafted Gabor filter algorithm for texture features extraction. In this study, we have used end to end CNN model and RBF kernel function of SVM for classification of Habesha Kemis into a specified class. Accordingly, the proposed end to end CNN model achieved 92% accuracy and then by integrating Gabor filter with this end to end CNN model an improvement of 95% accuracy registered. In addition, Gabor feature extraction, CNN feature extraction, and combined feature vectors are experimented by using SVM classifier, in which 93%, 95%, and 96% accuracy achieved, respectively. The performance of the proposed approach is affected by the similarity of color for different classes. The major challenge of this research is segmenting the embroidery design part of Habesha Kemis and hence there is a need to find appropriate segmentation without affecting the embroidery design color and texture.