Imagine if you are playing a game of chess, which requires a lot of skill and concentration, not unlike magic, and your opponent is constantly tapping on the table, snapping etc. to make noise. Not necessarily to mess with you, but because some people can't sit still. I would (and do) find it annoying regardless of their intentions. Personally I don't pay attention to a card that someone has been holding onto for the entire game. If I bounce a card to their hand, I'll notice that I haven't seen it again, so it is still there, but I don't actively pay attention to what cards they're holding onto, and I don't think many people in the casual or FMN levels of magic do, so there isn't really any strategic gain by doing it aside from annoying your opponent if that is what you are going for. I would easily mess up a complex combo even if I know it well because of the flicking.
It doesn't need to be done. If you ask them to stop and they don't (not because they instinctively do it, but because they are trying to mess with you), you know what type of game your playing, and I personally wouldn't be having fun. If i'm not having fun, I leave. They can "shuffle" their hand quietly. I usually hold my cards so that I am essentially holding a stack of cards, and I fan them out if I need to. Don't do it as part of your strategy. If you can't beat them playing the game, get better. Don't be a [insert derogatory term of your choosing.]