Actually, no. If the supervisor investigating such a loss of separation finds that the controller clearly intended for you to be where you were, he couldn't care less about whether there's a "cleared..." on the tape-- heck, he probably wouldn't even notice it.
In my 25 year ATC career, almost all pilot violations that went to FSDO (or GADO, or ACDO) were extremely solid ones that also caused a loss of separation-- altitude busts, etc. It's a lot easier to just write up the controller than to convince a FSDO inspector to pursue a case against a pilot who was doing exactly what he was told to do-- particularly if the controller used improper phraseology to tell him!
That said, I agree that "affirmative", without a call sign, was insufficient and required further confirmation.