I went from a 650 V-Star to the Deluxe and although I have only been riding for about 4 years the change was positive. I thought the same thing you did but found that the bike in stock form was only 75 pounds heavier than my V-Star. The bigger bike actually handled and performed much better and the feel is awesome.
I test drove the fat boy because it is the bike I always wanted but once I rode on the Deluxe I was sold. If you are 5ยด9" or shorter, like I am, this is the perfect bike. It fits like it was custome made for me. Good Luck