It's a matter of everyone staying vigilant and informed. When I was a student pilot back in the mid 70s, my instructor had shown me on the sectional where a low-level military route was located near my home town. The route made a dogleg at the tip of a lake about 15 miles north of my high school. I used to occasionally get time off from school to do some of my solo cross countries (principal liked me, and he had flown B-29s), and I sometimes flew over the school just to "show off".
One day while I was in the school parking lot, three ANG A-7s came scorching right overhead at less than 500ft, pulling hard in 90 degree bank turns. I'd never seen that before or after, so I guess its possible that they missed the turn by a few seconds. For me it was a wake up call to keep my eyes open, because anyone can make a mistake and it can all be over in an instant.
Years later, while doing touch-and-goes, a helicopter went barreling straight through the traffic pattern, missing me by maybe a couple hundred feet horizontally. I never saw him, only finding out when I landed from an instructor who saw him pass behind me. Even if you're exactly where you are supposed to be, doing everything right, you can still end up in a smoking hole.