Stripe or yellow rust is the most commonly occurring disease in wheat caused by fungus called Puccinia striiformis. It affects the productivity and quality of wheat throughout the world. Detection of yellow rust in early stage is challenging activity because of high spread rate in few days. Experts use necked eye inspection to detect yellow rust after the diseases reach at higher stage. Those experts do not work effectively due to tiredness and bias. Therefore, many researchers are motivated for the development of detection model based on machine learning algorithms. But, in the study they considered manual feature extraction method and in early stage symptoms of yellow rust (yellow spots) are invisible for manual feature extraction. Besides, similar with other wheat disease so these method prone to error and becomes subjective. It is therefore the aim of this study to develop wheat yellow rust detection model. The proposed system integrates components, such as preprocessing (Image resizing, Histogram equalization, and Noise removal), segmentation, feature extraction and classification. In Image resizing we normalize the image to a standard size(224×224), Histogram equalization is used to produce an output image whose histogram is uniform, In Noise removal, we apply Median filtering technique, For segmentation we use K-means segmentation technique by partition an image into k clusters, and we use CNN for feature extraction and classification. Implementation were performed using Keras (with TensorFlow as a backend) in Python and Sample wheat leafs were taken from South Gonder Zone from specific place, Gobgob Kebele. 1661 images were taken from each of the two classes (Healthy and infected). The total number of images taken was 3322 containing wheat leaf. To build the detection models for prediction of wheat yellow rust, Softmax and SVM classifiers are investigated. In addition the effect of K-means segmentation on the overall performance of the classification model also experimented. Experimental results show that, SVM classifier outperforms Softmax classifier. Quantitatively, an overall accuracy of 98.04% is achieved by using SVM classifier; on the other hand Softmax classifier achieves accuracy 97.74%. K-means segmentation is used to improve the performance. Identifying K values for K-means segmentation was challenging task for properly detect the region of interest in the given leaf. Here we used visualization methods to identify suitable K values and hence we recommend as way forward to design an automatic setting of optimal K value for the k-means segmentation to work adaptively.