Why should it be your highest priority? As long as you're not in over your head and are paying it off on schedule, all you lose is a little future money to interest payments. If what you get by not paying it off faster is sufficiently valuable to you, it might very well be a good bargain.

I mean, do you really think that as soon as you buy a house (which pretty much anyone not absurdly rich — and many who are — will take on debt to do), you must immediately neglect your family and pursue money at all costs? That hardly seems like a tenable position. People who have grave problems with debt have those problems not because debt is inherently a grave problem, but because they took on debt they couldn't handle.