The recent field focuses on DWDM (Dense Wavelength Division Multiplexing), which uses the bandwidth provided by optical fiber to increase data rate requirements. In this thesis analysis, the DWDM system using Erbium Doped Fiber Amplifier (EDFA) with PMD (Polarization Mode Dispersion) is carried out in C-band. The performance of a 4-channel DWDM system with high-performance data rates of 10 Gbps and 20 Gbps has been analyzed. Modelling an optical fiber connection with the program OPTISYSTEM is used to investigate the performance of optical communication system by using RZ (Return to Zero), NRZ (Non-Return to Zero), and DB (Duo-Binary) modulation formats. According to the modulated outputs, a comprehensive comparison in terms of Q- factor (Quality factor), BER (Bit Error Rate), and OSNR (Optical Signal to Noise Ratio) is developed to establish the advantages and disadvantages of the coding formats in short and long- haul optical fiber communication system. For fixed input power (0dBm) and variable fiber length at 10 Gbps data rate with and without PMD, RZ performs well for short and NRZ for long distance communication. For the same condition at the data rate of 20 Gbps, NRZ works well for short and NRZ-DB for long distance communication. Also for variable input power and fixed fiber length (100km) at 10 Gbps data rate with and without PMD, NRZ performs well for the given input powers. For the same condition, NRZ-DB works well for the given input powers at the data rate of 20 Gbps and RZ without PMD and NRZ with PMD has a strong OSNR value for this data rate. Generally, PMD affects the performance of transmission because it decreases the Q-factor and OSNR value of our transmission for the PMD coefficient of 0.5ps/sqrt (km).Also this PMD coefficient influences NRZ to have no long distance performance and RZ to have no good performance for the given input powers at data rate of 20Gbps Keywords: -RZ, NRZ, DB, DWDM, EDFA, Modulators, Optisystem.