Security is everyone's concern. I have flown some very recognizable people, Clint Eastwood, Harrison Ford, Mel Gibson, Maddona, Henry Kissinger, Sly Stallone and most were good people. Some not so good. Telling your family and your neighbors should be no problem unless you are flying for a cartel down south or a Langley based government firm. I think at a FBO or anywhere else and it is someone you don't know, just a polite answer like just an individual that no one has ever heard of, would work. If they get pushy then you really might consider a call to 911. I have had to do that three times and each time the responder took the person away for one reason or another.