Intelligent reflecting surface (IRS) is mainly made from a metasurface which can be programmable to adjust the phase shift and amplitude of the incoming signal. It is an enabling technology for beyond fifth generation (B5G) wireless communication system that controls the propagation environment. Due to high coupling between active beamforming (ABF) and passive beamforming (PBF) joint beamforming optimization is mandatory in IRS-based wireless communication system. Additionally, in practical scenarios, PBF is discrete and the reflected signal strength depends on IRS phase shift. In this thesis, minimum mean square error (MMSE), zero forcing (ZF), fractional programming (FP), particle swarm optimization (PSO), cuckoo search optimization (CSO), and hybrid PSO with CSO (CSOPSO) algorithms are used to optimize jointly both ABF and PBF to maximize spectral efficiency (SE) and energy efficiency (EE) for B5G wireless communication systems with continuous and discrete phase shift models. From these algorithms ZF, MMSE and FP are used for ABF where as PSO, CSO and CSOPSO are used for PBF in both continuous and discrete phase shift optimizations. Rate splitting multiple access (RSMA) technique is applied to the considered optimization algorithms with lower complexity. ABF and PBF matrices updated in alternating manner with help of their closed form expressions till end of iteration. Numerical simulation is done to validate the optimization algorithms performance in maximizing SE and EE. The SE and EE optimized using FP and PSO algorithms converged faster than CSO and CSOPSO with lower complexity. Continuous phase shift has better performance than discrete, and a system with a minimum reflection coefficient of 1 has better performance than with 0.75. Velocity scaling improved the EE and SE of the system and a system with RSMA has better performance than without RSMA. MMSE has better performance than ZF in all cases, especially at lower SNR and higher numbers of users. CSOPSO performed well in terms of complexity and efficiency since a 99.323%, 97.834% and 90.515% of MMSE-CSO achieved with MMSE-CSOPSO, MMSE-PSO and MMSE-random heuristic optimizations, respectively. Key Words: ABF, CSO, FP, IRS, MMSE, PBF, PSO and RSMA.