With the increasing demand for high-speed data transmission in wireless communication systems, innovative solutions are essential to meet these needs. This thesis explores the potential of Radio over Fiber (RoF) systems to increase wireless system capacity and efficiency. RoF systems integrate fiber optic links with radio waves to deliver highcapacity radio frequency (RF) modulation channels, aiming to support the next generation of broadband wireless systems with high-speed data transmission capabilities. The study focuses on two key techniques for improving RoF system performance: Coherent Optical Orthogonal Frequency Division Multiplexing (CO-OFDM) and Multi Input Multi Output (MIMO). CO-OFDM combines coherent detection with OFDM to enhance robustness against polarization dispersion and optimize power and spectral efficiency. MIMO increases wireless capacity by employing multiple antennas at both the receiver and transmitter, utilizing spatial multiplexing or enhancing system robustness through antenna diversity. We analyzed a 70 GHz RoF system using modulation formats 16QAM, 64QAM, and 256QAM to evaluate these techniques. Parameters such as data rate, optical signal-to-noise ratio (OSNR), input optical power, and fiber length were varied to assess their impact on system performance. Simulation results indicated that the 64QAM modulation format in CO-OFDM achieved the most favorable outcomes concerning bit error rate (BER), OSNR, and constellation diagrams. Notably, the 64QAM format is projected to reach the performance level of 256QAM, indicating a 33% enhancement in maximum capacity through optimized link budget. The findings suggest that RoF systems employing CO-OFDM and MIMO techniques significantly enhance wireless system performance and capacity. This technology shows great promise for future high-speed broadband wireless systems. Radio over Fiber, CO-OFDM, MIMO, wireless system capacity, 16QAM, 64QAM, 256QAM, optical signal-to-noise ratio (OSNR), bit error rate (BER), high-speed data transmission. Keywords: ROF, 5G, QAM, CO-OFDM