You know, in the end, the decision rests with yourself. Only you know how hard it was to find that particular rifle. You have to ask yourself - how hard will it be in the future? How much will the future price have increased, if any?
In terms of the vendor, it is not very likely that this business knows you. I spend time to chat (politics, news events, guns, ammo) with a local gunshop owner and we know each other. If either of us noticed the imperfections, we would have worked something out because he values my business and friendship. Big box stores or infrequent visits to this vendor would result in a lessor tendency be skeptical. Perhaps, if it was in a manufacturers box, they likely did not even open it before sending to you (I know of several that operate like that) and even if they did, some of those imperfections would have been difficult to spot or even noticed by an inexperienced stockroom person or clerk (my bias there - as that is reality from my experience). They offered to take it back - I think that is fair. If it really bothers you to the point of losing sleep, then send it back.
I liken the "new" factor to the new car or truck. It is immaculately babied until that first cup of coffee soaks into the carpet or that first tree branch scratches the fender. Then it's well - just a car or truck now. At that point, one doesn't really care if it is now washed at the Petro-Can whirling "soft" cloth car washes. Same thing with the rifle. In the field - it gonna rub, fall or otherwise get marred by something. If the rifle is that important to you, then pretend the first field bumps have happened and get a good night's sleep.