I find the time that is burnt is at the loading bench and the multiple sighting in sessions, not bothered by and actually rather enjoy having different mechanisms about, I only hunt with one per season anyhow so no change ups. Much prefer a single rifle and bullets tailored to the game personally, and likewise appreciate one wonderful gun to numerous decent production rifles- though I subscribe to both "systems" clearly by a look in the safe. See what you're saying, I'm just on the other side of the fence, less rifles less cartridges but more heavily invested in them and tailored to you.
I suppose it depends on what your hunting season entails... if you big hunt were for bison as opposed to say, a stone sheep, it is likely that your tool would be different.
I only hunted with my 7X57 No.1 this fall but only took a meat bear and a meat doe... the rest was spent with archery gear. Everyone has to be comfortable with their choices... I love time at the loading bench and at the range, not as much as hunting, but I don't have to sacrifice one for the other... I can also hunt with a multitude of weapons... I don't imagine that will change. I also know that you have a young family, and when my three children were young a lot of things took the back seat...