Areas of non-specular on Bumped Specular material

I’m working on a top-down RPG and to make the walls feel solid, I’ve made the back-side of each wall model use a black texture to blend in with the black background colour of the camera. This effect works well when I used the Bumped Diffuse shader, but it messes up with the Bumped Specular shader.

I’m wondering if there is any way to create an area of non-specular on the default Bumped Specular shader. I’ve tried making the area black on the normal map but that gives the result shown in the picture below.

An example of the issue in action can be seen here:

Bumped Diffuse

Bumped Specular (the specular in the top-right is on the black parts)

I’ve done a bit more research and it appears that the specular map is generated from the alpha channel of the main texture rather than the normal map as I previously suspected. Ticking the “Alpha from Grayscale” box on the texture importer seems to have fixed it.