Speaking at a news conference in Johannesburg, South Africa, Annan said: "We had to cancel our visit because the government made it very clear that it will not co-operate."
Annan and Carter had been due to travel with Graca Machel, an advocate of women's and children's rights and the wife of Nelson Mandela, the former president of South Africa who set up The Elders in 2007 to tackle global problems.
Women's rights
The group had hoped to assess the deteriorating economic situation in Zimbabwe and its effects on the population.
Christopher Mutsvangwa, a spokesperson for Zimbabwe's governing Zanu-PF, told Al Jazeera the delegation was denied several documents because "one of the three - Kofi Annan - specifically" was up to "mischief".
"He is trying to ride on the back of a well-meaning lady [Graca Machel] who is the first lady of two presidents of the region," he said.
Mutsvangwa said Carter has "an impeccable standing in Zimbabwe because he helped Zimbabwe get independence, and was the first western leader to recognise land as the key issue in the Zimbabwe conflict as far back as 1979 and 1980".
He added: "Kofi Annan, as secretary-general of the UN, never did anything to help Zimbabwe's cause.
"He was involved in a conspiracy ... which legitimised sanctions on Zimbabwe which were patently illegal and imposed unilaterally by western countries."
'Food shortages'
Annan said on The Elders' website: "Food shortages, a lack of seed and fertiliser for planting and the breakdown in health services are all having a serious effect on the people.
"We understand that the situation requires an urgent response and that delays will only prolong the people's suffering."
On Friday, the World Health Organisation (WHO) reported that nearly 300 people have recently died from a cholera outbreak.
Fadela Chaib, a WHO spokesperson, said 6,072 cases had been reported between the start of August and November 18, with a rise in the number of reported cases in the past two weeks.
Doctors and medical staff had protested in Harare, the capital, earlier in the week, saying the healthcare system is suffering from severe shortages.