Yeah, I've switched out factory barrels before but I usually end up going back to the original. Unless you're trying to hit super precise targets, a decent factory barrel usually works just fine. The aftermarket stuff can be awesome but it's not a miracle cure
I would swap out only if necessary. I have had a lot of good guns and none needed a swap barrel. Most guns with factory barrels shoot better than the owner is able to. I have done plenty of work on rifles to help accuracy, improvements to meet my needs of accuracy.
I'd swap if I was customizing. I've thought about putting a 20 inch fluted heavy barrel on my 5.56/223 coyote gun instead of the stock M4 barrel it came with. But that stock barrel does less than 1.75 at 200 yards so I've never switched. Don't fix what ain't broken, you know?