A Mobile Ad-hoc Network (MANET) is a collection of mobile nodes where the mobile nodes are act as router and forming a communications network without pre-existing network infrastructure and any centralized monitoring. MANET is highly vulnerable to security attacks so that the performance of congestion control variant of Transmission Control Protocol (TCP) and routing protocols are affected by the malicious node. TCP assumes that packet drop or loss is an indication of network congestion, consequently it wrongly retransmits the lost packet and reduces the packet sending rate, which degrades its performance significantly. In this regard, an investigation is made into how well these variants and routing protocol respond to different network conditions particularly with respect to extension of network size and variation of smart Grey-hole attack. This research aims to analyse four TCP variants (TCP Reno, NewReno, Vegas and Westwood+) with considering three routing protocol which are Ad-hoc On-demand Distance Vector (AODV), Ad-hoc On-demand Multipath Distance Vector (AOMDV) and Destination Sequence Distance Vector (DSDV). Simulation is conducted with lower and higher node densities and increasing the percentage of malicious smart Grey-hole attack from 0%- 35%. The simulation result reveal that TCP Vegas with AOMDV is the ideal combination of transport and routing protocol which can adapt fairly well to the changing network size despite Vegas has a high energy usage especially with AODV routing protocol. However, Vegas has outperformed other variants with low energy usage in the presence of malicious nodes. Alongside Vegas with greyholeAODV is the absolute combination of transport and routing protocols which is less vulnerable for random dropping of packets. On the other hand, the research affirms the superiority of reactive protocols, over proactive ones with extension of network size and variation of smart Grey-hole attack. Among the reactive protocols, AOMDV has been found to be remarkable with extension of network size. Nevertheless, greyholeAODV outperforms other routing protocols under smart Grey-hole attack. In terms of communication energy cost, DSDV is the best routing protocol when deploying 0% attack as well us under 10% to 35% of Grey-hole nodes.