1. By metaphysics I do not mean "theology". Let's leave God and Religion out of this: but I write as a Catholic, remember!
2. By metaphysics I mean those underlying non-empirical assumptions which underpin Physics and which most physicists take for granted. Such as:
A) The Cosmos is governed by Laws.
B) The Cosmos is real.
C) Things and events have causes:
"nothing comes from nothing; nothing ever could!"
D) Causality only happens one way in time - if this is true!
E) Everything has a rational explanation.
Such statements are not Physics statements and I suspect never can/will be so. They go beyond the phenomena and experience with which Physics deals. They motivate Physics and also facilitate it. If one didn't believe these statements (or others similar to them: I am not meaning to be exact here) then one would have no purpose in doing Physics and no means to do it either.
Hence, Physics without Metaphysics has no foundation.
Mathematics is a wonderful language. As a Platonist I hope that the entire structure of the material world can be described mathematically. However, mathematics cannot deal with the kind of issues which I have just mentioned. Perhaps it can express them (this would not be so surprising as English can do so) but it cannot answer the questions raised.
Mathematics can not justify the validity of mathematics. Mathematics requires an axiomatic basis and why one set of axioms is to be preferred over another is not a mathematical question, but a metamathematical question - or so it seems to me.
Even if it turns out that there is only one set of consistent axioms (which would seem to be false, because the empty set of axioms seems pretty self-consistent to me!) then this fact would not explain why that set of axioms is actualised in physical reality.
Mathematics does not of itself describe reality; it only deals with the relationships and interactions and implications of axioms. Physical reality is more than this. It is actually real, whatever that means - but that is a metaphysical question.