Sesame is one of the most important high value edible oil crops in Ethiopia with considerably high contribution to foreign currency and wide applications. The major constraints in sesame productions are diseases and insect pests causing severe yield losses. For experts, identifying the sesame disease by naked eyes is challenging. Moreover, identifying those diseases with the help of laboratory is time-consuming and costly. In this study, an attempt was made to identify Bacterial Blight (Xanthomonas campestris pv. sesami), Cercospora leaf spot (Cercospora sesame) and Sesame Gall Midge (Asphondylia sesami) using a deep learning model. The dataset is acquired form Ethiopian Institute of Agricultural Research Area Humera, Metema and Assossa. The data is captured using high resolution of 1440 x 2960 digital camera and the “.jpg” file format images are collected. The dataset was preprocessed by image resizing, Histogram equalization, noise reduction, as well as feature extraction, and Harris Corner Detection. Then, 70% and 30% of the data were used for training and testing respectively. Convolutional neural network (CNN), AlexNet and GoogLeNet of deep learning architecture is proposed for sesame’s leaf disease identification, and the performance was compared with each other. The models were trained with original and augmented datasets having 3,500 for training set and 1500 for test set Bacterial Blight, Cercospora, Gall midge and Healthy images respectively. Tensor flow, Keras, and Theano were used for model development. Hyper parameter tuning was applied to enhance the performance of the model. The experimental results showed that the accuracies of convolutional neural network (CNN), AlexNet and GoogLeNet are 89.4%, 81.6% and 95.5% respectively. This shows that GoogLeNet, with an accuracy of 95.5%, outperformed the other models. Finally, we develop a prototype test system by using Flask web application. The GoogLeNet model was used for the prototype test.