I get the feeling that Qhorin would not stand and trade blows like a dumb Tyrell bannerman. It seems that Qhorin was the crow most feared by the wildlings, even more than Ben Stark. I also do not feel that Qhorin was a man to let his emotions get the better of him in a fight, i.e. I don't think he'd be scared by red handed reavers nor get puffed up too proud when his family's honor was being tarnished. In my opinion, his fight with Jon was a huge sham... I don't think he even really tried (if he had, he'd have killed Ghost first and then turned on Jon).
As far as Victarion goes... how well does he fight when the enemy is not scared witless by his fleet's sudden attack? Does he fight effectively one on one when his flanks are not protected by a score of berzerkers? Can he fight on land when his opponent is not at a disadvantage?