A second crude bomb found near the steps of Mount Carmel church across from a petrol station was safely detonated by troops.
Police also discovered a suspect package at a store about a block away from the church.
The attack comes two days after a bomb exploded outside a cathedral in Cotabato City on Mindanao, killing five and wounding more than 50 others.
'Critical condition'
Two men were killed on the spot in Tuesday's bomb, Lieutenant-Colonel Edgard Arevalo, a Philippine navy spokesman, told reporters adding that the death toll could rise.
"Many of the wounded are in critical condition," he said, adding that three police officers were among those seriously wounded.
"We are still investigating who carried out the bombing."
Muslim-majority Jolo is a stronghold of separatists linked to Jemaah Islamiyah (JI), the Southeast Asian group that Philippine and Western intelligence officials have linked with al-Qaeda.
No group has claimed responsibility for the Jolo attack but security forces suspect it to be the work of the Abu Sayyaf, a small rebel group with ties to the JI.
A similar attack was reported barely two hours later in the port city of Iligan on the main island of Mindanao, where a bomb left inside a car parked near an army vehicle exploded leaving at least seven people including two soldiers wounded.
Philippine authorities said it was not clear whether the attacks in Jolo and Iligan were related.