They are wrong. No matter the pride or alcohol consumption or weird mental alpha-beast desire, they were and still are are wrong. Yes, there are numbers in music, and yes I do, on occasion, count a rhythm, but that is not math, and last time I checked, to play my senior Liszt repertoire, I did not have to prove Cauchy's Group Theorem, which states that, if I have a group, G, of finite order with identity e, and prime p divides the order of G, then G has an element of order p. (I apparently learned this once, then forgot it.
Was that facetious—perhaps a little aggressive—of me? Yes. But what if we turn this around? If I approached a psychologist and said "I had these dreams last night, and I think I'm repressing traumatic memories," and the psychologist explained how that's false pop psychology derived from the false claims of Sigmund Freud, I, the non-psychologist with no scientific knowledge of the field, would not say, "actually, Freud's theories have backings in . . . and therefore I'm right and you're wrong." I'm not right; I'm just an asshole.
People might think math (or physics or engineering) and music fit together due to confirmation bias. Many of us play music. Some are intellectuals, and some are mathematicians. Remember the Mozart effect? People believed kids who grew up listening to Mozart became more intelligent, so Mozart and intelligence appeared to go hand in hand. (This was false causation.) We ignored Beethoven and intelligent kids, Rachmaninov and intelligent kids, even Captain Beefheart & His Magic Band and intelligent kids. Maybe there is a correlation between musicians and mathematicians, but that is far from proving they fit together. It simply means that, of all the people who play music, some of them are mathematicians. Maybe all the mathematicians who happen to be musicians are lazy mathematicians, not doing their work, and thus we, the common-folk, are more exposed to them than we are to the good mathematicians.
I suppose writing and music have a lot in common too, but nobody has mentioned that yet. Or art and math, writing and math, thinking and math, math and math . . . damn, should've thought of that one earlier.
It may be better, the next time a math-musician crosses your path, to express interest in either his or her mathematical works or musical interests. And if you're uninterested, don't comment.