Fly_Dude-
First, you ought to read the "Fighters or Heavies" thread in this Military section. It'll give you a great idea of what it takes to get where you want to go in the military, especially in the A.F.
As for your question, I have to say that flying fighters is the greatest experience, short of sex, that one can imagine. Fighters are amazingly easy to fly (in most cases) and frighteningly difficult to employ well.
By that I mean that most fighters are designed to fly easily. I have often said that once someone got used to the final approach speeds (around 155 kts or so), flying an Eagle would be a cinch; a grandmother could do it in a snap. The control harmonies are great, although they are computer aided, but what the heck...it still contains an awesome amount of power and maneuverablility. The first time I soloed an Eagle I darn near couldn't make my oxygen mask seal because I was grinning too much. The systems in fighters are complex, but not so much that it takes a lot of brain cells to work it inflight. Fuel management is a no-brainer---if you have fuel, it will feed the engines. Engine management is pretty simple if you don't horse around with the throttles too much at high altitude (keep in mind I flew "steam-driven" Eagles with the old P&W F-100 engine, which had an analog electronic engine control. The F-16 has a digital engine control that is much better at maintaining sanity).
In stark contrast, employing an Eagle, or any other fighter out there, is very difficult to do well. There are stacks upon stacks of energy diagrams at various configurations to study, threats and counter-tactics, weapons employment and theory, radar theory, Eagle tactics, employment standards, communications standards and nomenclature, large force employment considerations, yada yada yada. It gets complicated.
Suffice it to say that:
a. Yep, it is difficult, but not prohibitively so. Heck, how hard can it be if I did it? If you can handle a complex aircraft, you can fly a fighter. The problem is getting there (see the Fighters vs. Heavies thread for more info. Also check out the "Question for the A.F." thread by the kid who wants to be a pilot when he grows up). You have to be able to handle formation and aerobatics. The latter is pretty easy, but formation takes a little more skill and is a good indicator if you are able to handle the mental gymnastics of three dimensional maneuvering with energy manangement thrown in.
b. It is getting to the point that ol' Ronnie Reagan had it right when he said that we were growing a generation of future warriors because they spent a lot of time behind a Nintendo cotrolling mayhem with buttons. The Eagle and Viper both have a tremendous number of buttons and functions that work without putting your head inside the cockpit. Software updates to the computers change these functions routinely, so there is always a challenge to learn how to fight again. Once you do put your cranium inside the jet, information overload from onboard systems (radar, radar warning receiver, data link, etc.) and prioritization are two competing forces wrestling for domination in your overworked brain. That part is very demanding.
c. The physical challenges of a modern fighter cockpit are enormous in some cases. The F-15 and 16 have tremendous G capabilities, and pulling those Gz for long periods of time (such as during a maneuvering 1 vs. 1 fight we call Basic Fighter Maneuvers, or BFM) can be quite daunting. Although most fighters can pull 9 Gz, they can usually only do so for short periods of time. The problem is that they can pull 7 Gz for a couple of minutes at a whack during a maneuvering fight. Trying to stay awake (not passing out due to lack of blood flow to the brain) during that period is physically demanding, and I found it more difficult to do it after I turned 40. At that point, I had to rely more on guile and cunning than raw strength...(More often than not I ran out of guile before the other guy ran out of strength, unfortunately!)
Is it worth it? I cannot answer that for you. I can only say that flying fighters, serving my country and defending what I believe is right has been the best part of my life, and I would not change a thing in my career given the chance to do things over.
Best of luck with your training, and I hope this answered some of your questions.