Police were called to Binfield Street in Tipton around 1pm after residents reported hearing a loud bang.
Officers said the explosion is being treated as a terrorist incident and some residents have reported finding nails and other debris in the area.
No-one is thought to have been injured but a cordon was set up in part of the town and several streets have been sealed off.
The blast was believed to have happened on a disused railway line behind the mosque.
It was reported near the Kanz-ul-Iman Muslim Welfare Association Central Jamia Mosque in Binfield Street, local councillors said.
Tipton Green councillor Ian Jones said residents in the area had described hearing a "loud thud", while councillor Syeda Amina Khatun said she was not aware of any previous attacks on the mosque.
"The explosion has gone off on the disused railway line behind the mosque," she said.
