Kelantan is the only Malaysian state led by the opposition Islamist party, PAS.
Hassan Mohamood, who heads Kelantan's Islamic affairs committee, told The Associated Press on Wednesday that the stiffer laws are useful "as a form of deterrence".
The previous maximum penalty was two years in prison and a fine of RM5,000 ($1,400).
Proselytising of Muslims is forbidden under federal laws, but the recent case of Line Joy, a Malay-Muslim woman who sought legal recognition of her right to pick her religion of choice, raised fears among some in Malaysia over mass conversion.
Attempts to convert Muslims to other faiths are rare, and people found guilty face prison terms in most states in the country where just over half the 27 million population are Muslims with a sizeable number of Buddhists, Christians and Hindus.
According to the latest US state department data on Malaysia, ethnic Malays – who are Muslims by law - make up about half the population but are officially grouped together as Bumiputera, or sons of the soil, with indigenous groups who make up 11 per cent of the population, not all of whom are Muslims.
Religious conflicts
The amended penalties in Kelantan, where PAS has been in control since 1990, are believed to be the heaviest nationwide.
In Malaysia, Islam comes under state-level jurisdiction and religious authorities often send Malay Muslims who try to leave the faith for counselling and rehabilitation.
Some have also been imprisoned for apostasy.
In the Lina Joy case, the woman who was born to Muslim parents failed to get the country's highest civil court to recognise her right to choose her own faith.
The Malaysian constitution guarantees freedom of worship for all citizens but the country's parallel Islamic legal system often gives rise to religious conflicts.