Hi Halrath,
The models others have listed above are all fine handguns to consider but I would be less concerned with individual models than ergonomics. I would recommend giving
this article a gander. It discusses some important points that you should try to keep in mind when looking for a handgun. How well you can hold and operate the gun is really going to dictate how much you like shooting it.
Also, I wouldn't worry about wanting something bigger than a 9mm. You will likely want to try different calibers but I've been shooting exclusively 9mm semi-automatics for some time and have no real interest in changing calibers. I'm more interested in mastering what I have. Some people are more interested in a diverse range of experience. When you're learning to shoot a pistol, though, for the first few months the last thing you'll probably be thinking is 'man, I wish I had something with more recoil and higher ammo cost' unless you're incredibly talented, independently wealthy, or some Tony Stark-esque combination of the two.
I'd also strongly suggest keeping an eye on the used market. A lot of guns are sold at great discounts that are barely used at all. The number of people who seem to buy guns and just cycle them through their collection with less than 500 rounds is extraordinary. Even if they have a good number of rounds through the (I'm talking 5000+), if that causes a serious quality issue with the gun, I wouldn't have wanted it even new. Unless something is really hard to find and I haven't seen it used for more than six months, I don't even really consider new guns anymore. The savings translate into an enormous amount of ammo. Then again, I can understand the interest in something brand new for a first purchase because, hey, why not?
In terms of specific guns, I would also consider whether you want to do any competition shooting down the road. You said you want to look at guns you won't want to ditch later and knowing you're buying a gun that will work reasonably well for competition is a good long-term idea. Sports like IPSC are one of the only things we're allowed to do with our handguns in Canada and really help maintain an interest in a given piece, but if you have the 'wrong' gun for a class you want to shoot, you might end up looking for something different. For example, I know quite a few guys into action shooting who abandoned their hammer-fired DA/SA guns for Glocks or M&Ps because of IPSC rules, and others who just ended up buying CZ Shadows.