The problem you have is that it depends on how much material is under the bolt seat. If you remove too much of the back pad, you will often leave very little under the bolt, which limits how much you can remove.
You may not even be able to take 5mm off safely. You'd to remove the wheel and have a close look.