Late blight and early blight diseases are caused by fungal pathogen.These diseases affect potato which is the fourth major crop of the world. The first symptom of these diseases appear in the leaf of potato. Detecting the disease as early as possible help the farmers from economic loss. Most of the time plant disease can be detected by using naked eye observation of farmers (gardeners) or by the experts. The necked eye observation is not perfect and also the detection by the experts will be time consuming and tiresome. So researchers come to the concept of using automatic detection of plant disease. Most researchers those uses Convolutional neural network(CNN) prefer to use a sequential layer which lead to learn features in ordinary manner. We have developed a CNN model for automatic detection of early blight and late blight diseases. The proposed system include steps like image preprocessing, data augmentation, feature extraction and classification. Image preprocessing include resizing of an image to pixel of 256 * 256 and removal of noise by using 3 * 3 pixel window median filter. We have used image data generator for data augmentation. For the feature extraction and classification we have build a CNN model. The proposed model contain two main modules which are the concatenating module and residual module in order to fill the gap of using sequential layer. In concatenating module the input from previous layer give to two convolution layer of different filters. Then the output of two convolutional layers are concatenating using concatenate layer which is used to get a multi level feature. In residual module the input from the previous layer follow feed forward manner and also skip using skipping connection, which add new features. The proposed model is implemented in Keras using TenserFlow as a backend in python. It is tested with a 2152 dataset taken from Plant village and 330 from farm land located in Merawi, south of Bahir Dar, Ethiopia .Our model is evaluated using performance metrics like accuracy, precision, recall and f1-score. From the optimizers tested in our model adamax perform the best. Our model with adamax optimizer achieve 99.53% train accuracy, 99.48% validation accuracy and 99.53% test accuracy with a batch size of 32 for 100 epoch.