There are exceptions to any rule, of course, but the exceptions here are really quite limited. Perhaps you are also a U.S. marshal and are attending an informal gathering of U.S. marshals during which shorts-yanking horseplay might be expected. Or perhaps you have obtained prior consent (preferably in writing).
Those are really the only two exceptions that come to mind.
Actually, I would suggest similar limitations on this activity under any circumstances, for a couple of reasons: (1) doing this could be rude and/or frightening; and (2) one may not know in advance that the target is a U.S. marshal.
These lessons were learned the hard way by a 19-year-old Pittsburgh man, who yanked down a woman's shorts while she was running on a riverside trail last Tuesday. The trail is reportedly a busy one and there were other people around, plus he ran off right after the yanking, so that seems to be all he was after. He got a lot more than that, though.
The woman is in fact a deputy U.S. marshal, and she was not about to let the assault go unpunished. Yelling for others to call 911, she took off after the assailant, screaming "Federal marshal! Stop!" Although she is only 5' 5" and he is 6' 4", 210 pounds, she not only caught him but also seems to have subdued him pretty easily: "Police said the woman 'feared that [he] was going to assault her again' so she kicked him in the crotch, grabbed him by the shoulder and punched him in the face." Police arrived shortly thereafter.
No one was seriously injured in the incident, although the report says the woman "hurt her left pinkie," presumably while punching the suspect in the face. He faces several charges including, for some reason, "escape," although he didn't.