$2600? Nope, not with care.
Baikal SXS shottie that I cut down to 20 inch barrels- $300
Two slightly used Uberti Remington percussion revolvers- $650
A new, and possibly the last of, Rossi Puma .357 lever rifle- just over $600 after taxes.
So that's just a hair over $1500 all in.
But now comes the holster belt rig and cowboy hat... Oh, OK. There goes the budget....
With care and patience I suspect you can get into cowboy shooting for around $1800 total or maybe even a bit less with luck. But if the spark is lit and the need to jump in is great then yeah, the sky is the limit if you rush out and buy all new.