Stem rust is the most common and severe wheat fungal disease which appears worldwide. Stem rust is a very critical disease that is caused by a fungus called (Puccinia graminisf. sp. tritici), which typically causes losses of 10-50%, but mostly it reduces grain yield by up to 90% if it is not controlled. Examination for stem rust signs involves careful inspection of part of the plant affected (stems, leaf sheaths, and leaf blades, which will also affect the part of a wheat head), shape and distribution of lesions (oval-shaped or elongated), lesion color (Orange-red), and degree of damage (tearing of outer layer of plant tissue). Pathologists and trained surveyors visually inspect clinical signs of stem rust. However, manual diagnosis and identification is very expensive, time-consuming, and sometimes leading to errors. Moreover, lacks of pathologists are the main problems in detecting and grading wheat stem rust, especially in rural areas. Various researches have been done to solve problems in the detection of wheat rust disease. However, they did not provide a solution for grading the severity levels of stem rust. Thus, in this work, we present a system for automatic detection and grading of wheat stem rust in line with the CIMMYT guideline The proposed system has four components: preprocessing, segmentation, feature extraction, and classification. In image preprocessing, we normalize the image to a standard size. segmentation is used for region of interest (ROI) extraction and we used Adaptive Thresholding. In feature extraction, we propose to apply a Gabor filter on the image for texture feature enhancement, used for detecting and selecting important features of the disease. For classification, we use a convolutional neural network. A 12-way Softmax is used for grading into a specific class (Resistant (tR), Moderately resistant (MR), and Susceptible (S)). Data augmentation and dropout techniques are applied to overcome the overfitting problem in the model. The proposed system is implemented using Keras (using TensorFlow as a backend) in Python. It is trained and tested using an image dataset collected and prepared with the help of pathologists from Haramaya University. The model achieved an accuracy of 92.02% for training and 92.01 for testing to detect and grade wheat stem rust. Our model was faster to train and improves its performance from state-of-the-art CNN models