Al-Zawahiri referred to the conflict in Pakistan as "the American crusader manipulation of Pakistan's destiny" and warned it could break up the nuclear-armed nation.
"The Americans are today occupying Afghanistan and Pakistan, so it is the duty of every Muslim in Pakistan to rise up to fight them," he said in the eight-minute, 49-second recording, posted on the internet.
Al Jazeera's Imran Khan, reporting from Islamabad, said al-Zawahiri was hoping to provoke a reaction to Pakistan's campaign against Taliban fighters in Swat.
"He's talking about the American 'occupation' of Pakistan ... Pakistan isn't occupied by anybody. What he means is the operations against the Pakistani Taliban by the Pakistani army.
"He says that these operations are run by the US - that this is not a Pakistani war, this is a US proxy war. It's that kind of inflammatory language that he's hoping will get people riled up."
Declining influence
The al-Qaeda deputy last addressed a message in English to Pakistanis in August 2008, calling for a "jihad" in the South Asian nation.
But many analysts in Pakistan say al-Qaeda has little influence in the country.
"There is this idea that al-Qaeda is a diminishing force within Pakistani society," our correspondent said.
"But that doesn't stop them from planning, gaining support networks and doing what the Americans say they are doing - which is attacking them abroad."
According to the SITE intelligence group, which monitors material from such sources, al-Zawahiri's audio message was released on Tuesday.
In it, al-Zawahri said: "[If] we stand by passively without offering due support to the mujahidin, we shall not only contribute to the destruction of Pakistan and Afghanistan, but we shall also deserve the painful punishment of almighty Allah."
US influence
Under heavy US pressure to act, Pakistan's army launched a massive offensive in late April in Buner and Lower Dir, before focusing on fighters in Swat, where the Taliban concentrated a two-year campaign to impose its strict interpretation of sharia, or Islamic law.
The conflict has displaced nearly two million people, but officials have said that the pace of returns to the area has grown, though many civilians are fearful about security in the area.
Last week, Yusuf Raza Gilani, Pakistan's prime minister, unveiled plans to start sending home many of those displaced by the conflict and said the military had "eliminated" the fighters.
Elsewhere in North West Frontier Province, two policemen were killed and five wounded in a bomb explosion in the town of Bannu.