Certainly, if you're a lawyer, a ruthless cop who does his job by running after criminals and constantly pissing them off or maybe your job entails stepping on people who have the means and reasons to hurt you, than I can totally see the concern.
Otherwise if this is just in case scenario. Just in case I am on the way home from playing golf and I run into some gangsters who are going to take my money and kill me. Then think about it, is it just good old paranoia?
If you feel more secure with a gun in your car than go ahead. If you don't mind scaring someone as you carelessly show off your gun to someone and then have them call the police. And trust me when I say this if there's an anonymous call that comes in and it traces back to your house. Guess who is going to be answering a lot of unnecessary questions and I really do hope that you have everything in order. Again take my word for it, but you'd be in the back of the cruiser in no time.
Social responsibility is not some foreign object, it's something that everyone should be always concerned about. That applies to doing dangerous things and scaring civilians. Everyone has different sensitivity threshold. Some can handle it and some can't, if you scare some poor mother with 3 kids with your gun as you take it out of your trunk, than shame on you.
Remember, people in general are scared of guns and gun owners, be cool and let them be while enjoying your firearms. Otherwise don't scare them further and give them reasons to pursue retarded legislations.