I think I can understand this side of the argument, but I can't say as I agree with it. Notwithstanding dealing with varmints or troublesome predatory animals, this is my view:
For me, it goes back to a number of things. For example, historic practice. One of the main arguments that I have heard from hunting supporters is that it is our historic right to do so. Okay, I don't disagree with that. However, for the most part our ancestors didn't go out and shoot a deer or a moose just because they could - by and large they did it to feed and provide clothing for their families.
Another argument I have heard from time to time is that it is natural predation, just like you'd see a wolf or cougar do - that humans are just another predator in the food web. You certainly don't see wild animals walk away from a fresh kill, satisfied because they managed to fell the largest buck.
For me, people who hunt solely for trophies are in it for nothing more than their egos. They are no part of natural predation or "grandfathered" (not the right term I know, but I'm drawing a black atm) historical hunting traditions.
I would also respectfully disagree that it's "your" animal either. Yes, you have been given the right/chance/priviledge to shoot one, but they all belong to all of us, hunters and non-hunters alike. Why should any one of us have to go without a freezer full of meat because someone else wanted to get a rack of antlers and let the meat be eaten by the critters? I would respectfully suggest that if the meat is not to the hunter's liking, then they should select a different animal to hunt.
In some places, you have to be in a draw for years before you get selected for a licence. Trophy killing without making use of the rest of the animal, for me, just harkens back to the 1800s where over-killing nearly wiped out many species just because rich gentlemen wanted to show how grand they were by having a room full of exotic, stuffed animals. It was not responsible hunting and in a lot of ways it is one of the reasons why responsible hunters today have to justify their sport.
As someone else said, if you are out hunting and get a nice trophy, then bonus. I do not think it should be the goal unto itself.
BTW, I'm not pointing this argument at you specifically or attacking/flaming you. I am only trying to clarify my view in realtion to yours.