If I were running the bubble and someone started deviating without asking, say 1/2 mile, it would be a big deal. But most controllers won't run traffic real close if there is weather in the area because some pilots will just turn without asking. If I do run traffic tight in weather, I will assign present heading to one or both.
As far as you owning 4 miles on both sides of an airway, that is kinda an outdated ruleing in my opinion. Its in the book. But its in the book under the nonradar section (chapter 6 section 5 for the doubters). As soon as you hear those precious radar contact words, I wouldn't turn more than a few degrees for even 30 seconds without advising.
Under the same section (6-5-4) it says that (using angles) an aircraft cleared from a point to another 260 miles away owns, at the widest part, 10 miles both sides of a direct line. That's alot of airspace, and again, why it is under the nonradar section.
Longstory short, 1/2 mile and they guy sounds bored, tell him. 1/2 mile and the controller is busy, advise him, and say it will only be for a mile or so.