The good news is that apparently your starter and your ignition systems work. DO NOT keep your starter going longer than a few seconds and DO NOT KEEP TURNING UNTIL YOU KILL THE BATTERY. You are going to melt your solenoid switch and damage your starter if you haven\\\\\\\´t already done so..
My guess from your description is that you need to balance your carburators.
BUT, before playing around with the carburators,
I recommend that you do the following in order ( or let a friendly motorcycle repair shop do it):
Note. I\\\\\\\´m assuming you got a repair manual that can give details.
1. Check the spark plugs and gap. I think you are going to find one or both black and wet.
2. Check the compression. If you don\\\\\\\´t have a compression gage, buy one. For $ 30. or so bucks is a good investment. If the readings are more than 10 psi different,or lower than 150 psi reset the valves.
3.Balance your carburators. This is done for idling (Likely your problem) and for 3500rpm.
This is best done by a motorcycle shop, since it requires mercury manometers and vacuum lines connections and a somewhat cumbersome rigging of gasoline feed to the carburators while the motorcycle seat and gas tank are removed ( The adjustment screw for the carburators balance is in an impossible place to reach with the tank in place. Note: You or your motorcycle shop( mechanic) may want to remove and check the carburators ( and setting float levels) outside the bike before reinstalling them and attempting to balance. 
We invite you to introduce yourself by a few sentences if you are new to
this discussion group. Enter your e-mail to join the forum and get notifcations on new messages. Or post your question,
answer or comment as a guest. We have just improved this discussion forum and expect the number of users to increase.
More 1982 Yamaha XV 750 SE information
Our discussion groups are intended for short messages. We also welcome more extensive motorcycle reviews.
Bikez also recommends that you check out the technical
data and the riders' rating for the 82 Yamaha XV 750 SE.
You can also rate it yourself. Use our efficient motorcycle
classifieds to sell or
buy
this bike. Your motorbike gets a high quality presentation. Ads are free.