The economic development of most developing countries including Ethiopia is extremely dependent on agricultural production. From those agricultural products pepper is a high-value vegetable and spice crop in Ethiopia for both domestic market and export. However, it has been infected by different diseases. The spread of pepper plant diseases is the most frequent reason for the decline in pepper yield. To avoid a decline in productivity, timely and precise detection of plant diseases is essential. Various image processing researches have been made for early identification and diagnosis of pepper plant diseases using different approaches. CNN was proposed for extracting meaningful deep high level features. However, using CNN with small dataset is difficult for extracting enough features. We develop a model that works with a small sample dataset by merging local feature Grey Level Co-occurrence Matrix (GLCM), Histogram Oriented Gradient (HOG), Local Binary Pattern (LBP), and high-level Convolutional Neural Network (CNN) for the identification of pepper leaf disease. We used an end-to-end Convolutional neural network using a Softmax classifier and Support Vector Machine (SVM) for the classification of three pepper leaf disease classes. This research work focused on designing hybrid feature extraction for pepper plant leaf disease identification that specified the disease into leaf spot, mosaic, and wilt pepper leaf disease effectively. The image of the pepper leaf was captured from Burie and Woreta. To propose the best feature extractor, we performed a series of experiments on LBP, HOG, and GLCM, and achieved 81.25%, 83.33%, and 84.9% respectively. We have hybrid three local features with CNN feature vector to get more discriminative features of pepper leaf, then given to SVM classifier to classify the hybrid feature vector. Among the hybrid of three hand-crafted features with CNN, the combination of CNN with the GLCM feature vector achieved a better performance that is 94.79%. In addition, we achieved an accuracy of 86.96% on end-to-end CNN using Softmax classifier. Therefore, a hybrid of GLCM with CNN feature results in good performance rather than using them individually. Keywords: GLCM, CNN, K-means, feature extraction, SVM