The brain is the most important organ in our body, and it regulates all functions in the body, such as thoughts, memory, speech, and movement of arms and legs. However, different diseases affect the human brain. Brain abscess is one of the brain diseases that need accurate and timely detection vital for this disease to reduce the death rate of people. Computed Tomography is one of the top imaging methods in the medical field. Manually reading and naming the images is very challenging to do. Hence, it requires high time and extensive knowledge for perfect detection. Many researchers work on the detection and classification of brain diseases using machine learning and deep learning approaches in previous studies. Almost all studies were concentrated on the detection of tumors, stroke, hemorrhage, and other brain diseases. Brain abscess detection was neglected in previous studies. The proposed model used Histogram of Oriented Gradient and Convolutional Neural Network for feature extraction and Support Vector Machine for classification. Convolutional Neural Network is used for deep feature extractions and HOG used for local feature descriptors. Hence, Histogram of Oriented Gradient is a grateful local feature descriptor. Support Vector Machine is high performance for the classification of these features. The model classifies early cerebritis, late cerebritis, early capsule, and late capsule are the stages of the brain abscess and normal brain images. The dataset was collected from patients at Bahir Dar Felege_Hiwot General Specialized Hospital and Gondar General specialized hospital to perform a model using Keras, TensorFlow backend in a python environment. The model used 2000 images, from this dataset, used 80% of images (1600) for the training set and 20% of images (400) for the testing set, from the training dataset 20% of the images (320) for the validation set. The model attained an accuracy of 99.25% for training and 99.00% for detecting brain abscess. We have compared the proposed model with the current state of the models, LeNet and AlexNet. So, the proposed model was better for the detection of brain abscess.