I was Aero Engineering but had roommates that did physics and EE. I'd go with what you are interested in. My take was that both are pretty darned hard but people get degrees in them every year and it sounds like you have the prereq's to succeed if you are interested. EE has more of a future if you got out and wanted to work doing that kind of stuff, seems like everything from cell phones to CPUs to cars needs a few EEs to help design it, can't say the same for physics though. Be advised too that a major in physics is very different than the Newtonian stuff you did in high schoool &/or AP physics. I always thought that kind of stuff is pretty fun, mass/acceleration/etc, real stuff. I think that around junior year in college you'll be into the 'black magic' part of physics, lots of equations, no numbers, yuck. Of course, aero and EE are similar, once you get past the basic sophmore type courses, most engineering majors get very heavy into math, not surprising, but of little value when it comes to flying a plane.
So, go for a major do what interests you/you'd enjoy; don't necessarily go business or something just because it seems easier or someone said to, if you are studying what you really care about, it will be better in the long run. That being said, don't get into one of the harder majors (EE/physics, for example) if you don't think you can pull pretty decent grades. A 2.0 in EE will still put you near the bottom of your class, you'll have learned a bunch but may have trouble service selecting what you want.
About schools, I'd throw USNA into the mix (I went to canoe U myself so I'm biased) but I figured I could fly out of either USAFA or USNA and I decided I'd rather sail than ski most of the time, thus, annapolis. Plus if the flying didn't work out for some reason (eyes the most likely) I could have gone Subs, Surface or USMC, whereas at AF, I'd think you're off to a missile silo or some other equally tempting desk type job.
About VMI/Citadel, I'm of the opinion if you are gonna go get abused for 4 years, at least get the gov't to pay for it. VMI/Citadel have great reputations in their geographic regions when it comes to producing businessmen, i.e. great networking if you do 4 years in the service then go join some regional company, otherwise, I'd say go to the source, USNA/USAFA. Just an opinion, I'm sure some Citadel/VMI guys will jump on about how wonderful it is to pay 12k a year for all the 'fun.' if you pick up a ROTC scholarship, i guess it will still be free, but still, if you want 4 years of military, I'd vote USAFA/USNA.
State college could be very good too. Much more fun while you're there, more distractions if you choose a hard major though, and unlike at USAFA, there will be plenty of people that could care less about their class rank/service selection that will be tugging you to go party when you might be better served keeping your grades up. There are tons of military pilots from all the places, you mention, it really boils down to what you want? academy, military school, or rotc at civilian? all will get you to the same place in roughly the same shape, but you'll have different experiences along the way.
And I'm sure you know, and maybe have already applied, but if not, I thin, time is getting short for USNA/USAFA/ROTC. Back in the day they did a rolling admission starting in January or thereabouts, but there is a bunch of stuff extra you have to do for academies, so if you haven't already, get that stuff working.
I got the degree with the intent of eventually getting my master's in business management and moving from operations into more of a management role as my career progresses. 
