Don't let people scare you. It is pretty safe, just like all big cities there are problems. The train system is very good. Take a day trip to Toledo is a must. It is just over an hour train ride. Pick up a good spanish sword while your there..
Also another good day trip is El Escorial, and on the way is Valle de los Caidos. It is amazing! Biggest cross in the world built by franco for his tomb.
If you are really want to see some sites away from tourists take a bus or rent a car to Chinchon, about 45 min. away. Come to think of it, just rent a car. Very exciting.