The design of cost effective Optical Line Terminal (OLT) for TWDM-PON based arrayed waveguide grating (AWG) is designed. However, while providing high data rate and longer transmission distance, TWDM-PON suffers from fiber dispersion and attenuation, which several limits on system performance. The performance of the different modulation formats on the proposed improved TWDMPON architecture transmission system is examined. In this thesis, TWDM-PON architecture based on Fiber Bragg Grating (FBG) and different Optical Amplifier (OA) for extended transmission distance is proposed. FBG dispersion compensation is considered in order to tackle the dispersion problem and increase transmission distance. And the optical signal needs to be amplified by using SOA, Raman and Erbium Doped Fiber Amplifier (EDFA) in order to compensate for the losses in the transmission. The performance of the proposed system is analyzed and the performance of different modulation formats is measured and analyzed in terms of BER and Q factor values. One multicarrier input from the PM which provides seven different wavelengths from 186.4 – 187.4 THz with 100 GHz frequency spacing. AWG routs each wavelength frequency of 187 THz, 187.1 THz, 187.2THz, and 187.3 THz. At 45 km single laser source TWDM-PON achieves Max Q factor of 6.119989 for 10 dBm input optical power and TWDM-PON with four laser sources gives Q factor of 6.232882 which shows relatively same performance. Duo Binary modulation format shows better performance in achieving 5.75 E-10 Min BER at 50 km transmission distance. The EDFA outperforms SOA and Raman in terms of Q factor and Min BER for fiber lengths ranging from 5 to 50 km, having a 4.50E-10 and 06.24 Min BER and Q factor value respectively, EDFA achieves 49 km. The Min BER was 6.67E-10 for EDFA and 2.02E-09 for RAMAN with best acceptable value at 100 km and the SOA Min BER value was 1.19E-08 with FBG dispersion compensation