Bananas are the most important and extensively consumed fruit in the world, and they are vital to the economies of many nations. On the other hand, a number of illnesses can seriously impair the quantity and quality of banana crops. Making use of a laboratory and naked eyes of observation, plant pathologists can classify banana disease. This could take a while and cause additional flows to be lost. This paper describes classification of banana fruit diseases that affect the fruit part that is common fungi diseases, which are anthracnose, crown rot and freckle fruit. As we have developed a CNN prototype for automate categorization of banana fruit fungi diseases. Preprocessing, segmentation, feature extraction and classification are the four components of the proposed framework. During image preprocessing, the image is normalized to a standard size and reducing noise. Segmentation is used for threshold-based extraction. In order to extract texture features from raw images, we suggested using the Gabor approach in feature extraction. For classification, we used Softmax that classifying into a specific class (Healthy, Anthracnose, Crown rot and Freckle fruit). Python's Keras is used to implement the suggested system and a sample image collection that was gathered from the ET fruit center in Bahir Dar. 80% is used for training and the remaining 20% is used for testing. The Banana Net prototype achieved accuracy of 96.57% for training and 96.56% for testing to classify banana disease. Compared to the VGG16 and VGG19 models, our model was smaller and easier to train. Keywords: Deep learning, CNN, Banana disease, Feature learning, Segmentation,