Abstract unarmed combat

When it comes to fighting without weapons, D&D tends to go a bit insane. Besides often coming out overly fiddly and complex, subsystems for unarmed combat tend to mesh very poorly with the standard (armed) combat rules. Game designers, it seems, have tried to model a set of Hollywood fight tropes which don't fit well with the abstractions and assumptions of D&D combat. Consider, for instance, the ever-popular notion of knocking out an opponent with a single punch. In a game where clubs and maces can only whittle away the opponent's hit points, a rule allowing knockout punches produces results that are often counterintuitive, if not outright ridiculous. 

I've made my own attempts on more than one occasion to construct a set of viable grappling/unarmed combat rules, but now I'm coming at it from a very different angle.

The main problems with unarmed combat subsystems are A) treating armed and unarmed combat as fundamentally distinct from each other when in most respects they are not, and B) making unarmed combat less abstract than standard armed combat in a misguided effort to model those Hollywood fight tropes mentioned above. 

A) The line between armed and unarmed combat is a lot blurrier than all the crazy subsystems would have us believe. In fact, armed melee often involves a lot of pushing, shoving, tripping, hooking, and strikes with fists, feet, knees, elbows, and heads. Unarmed combat is often considered to be non-lethal, but many people are killed by unarmed pummeling or choking, even in our real world where the assailants are very unlikely to be wearing armored gauntlets. D&D also has rules for inflicting non-lethal damage with weapons, so there's a lot of overlap going on. Bottom line, in D&D, damage is damage. Only the final damage roll that reduces the target to zero hp really matters as far as lethal vs. non-lethal.

B) The combat system in games like B/X D&D is highly abstract. A combat round is ten seconds long (varying from 6 seconds to a full minute in other editions or clones) and an attack roll does not represent one discrete swing. Instead, the attack roll represents ten seconds' worth of feints, parries, dodges, lunges, jockeying for position, as well as (possibly multiple) actual earnest attempts to strike the opponent. It's possible that one or both combatants may inflict brief stuns or dazes on each other, knock each other down, stomp each other's toes, throw dust in each other's eyes, and so forth, but all of this is abstracted into a single attack roll, and summarized in the damage roll (or lack thereof). 

Bolting a subsystem onto that abstract chassis which adds stuns, knockdowns, trips, and so on is a mismatch. It tries to make explicit and game-mechanical what in the base system is implicit and narrative. A hit with a club absolutely can stun or knock down an opponent; those effects are implied and abstracted into the final damage rolls. A hit with a fist or foot or elbow can also stun or knock down an opponent, but instead of using the same assumptions as we do with the club, we apply game-mechanical effects to them and wonder why our system feels so off-kilter. 

* * *

Of course, we could regain the armed-unarmed symmetry by applying the same subsystems for stun, knockdown, and other status effects to armed combat as we do to unarmed, but at the cost of torpedoing the elegant, streamlined abstraction of standard D&D combat. It's also a lot of extra complexity and emphasis on combat for a game that, at its most fun and playable, is about much more than combat.

But, assuming we like simplicity and abstraction over a ton of extra crunch in combat, what, if anything, do we need unarmed combat options to do that isn't already subsumed in the abstraction of the attack roll/damage roll procedure?

The most obvious thing is to affect an opponent's movement or position in specific ways. You can grab onto it to keep it from breaking away (or in the case of very large opponents, to climb on). Or you can force it to move somewhere it would rather not, like into a fire pit or over a precipice. Either way, we can resolve those attempts without a lot of special rules and no additional subsystems by using the ascending AC/attack bonus framework and a simple hack.

First, the player declares that the character is trying to grab, grapple, or move the opponent. The character must have at least one hand free, and ideally two for most maneuvers. Still, the possibilities of one-handed grappling provide a legitimate reason to use a one-handed weapon without a shield. 

The DM makes a quick judgment as to whether the desired result is even possible; e.g. a human fighter trying to shove an elephant over a ledge is completely ridiculous, unless the fighter is wearing a girdle of giant strength or something. 

The DM decides the standard of success: either the character's attack roll must exceed the opponent's roll, or equal or exceed a total of 20, or both. The DM may also impose disadvantage on the roll if the attacker has only one hand free, unless the maneuver is relatively simple. In almost all cases, if the attack roll succeeds against the target's AC, normal damage can be inflicted, regardless of whether the special maneuver succeeds, but depending on the standard of success, the maneuver might succeed without beating the target's AC and inflicting damage. For example:

• A player whose character has a dagger in one hand and the other hand free wants to grab his opponent. The DM rules he only has to exceed his opponent's attack roll (or highest attack roll, in the case of opponents with multiple attacks) and since it's a fairly basic move, no disadvantage applies for using one hand. If he does, he grabs his opponent by a limb. Whether or not he beats the opponent's roll, he can do normal dagger damage if his roll beats the opponent's AC, and whether or not he beats the opponent's AC, he can grab it if his attack roll exceeds its attack roll. It's possible he could do damage, secure the grab, both, or neither, depending on the dice.
• The same character, still holding onto his dagger and the foe's arm at once, wants to fling the foe into a nearby pit. Since he's still using only one hand, the DM rules he must roll at disadvantage, but still must only beat the opponent's attack roll to succeed. The DM also rules that if the move is successful, no weapon damage can be inflicted; after all, the foe is flung out of range.
• A fighter wants to lock an opponent in a damaging hold. He opts to go completely unarmed. The DM rules that this requires him to exceed his opponent's attack roll and score a total of 20 or more, since the desired result is to completely restrain the opponent while inflicting ongoing damage. If he successfully locks in the hold, he can continue inflicting damage each round as long as he beats the opponent's roll. The opponent can prevent damage that round by exceeding the attacker's roll with his own, and may escape completely by scoring a total of 20 or more. 
• Another fighter wishes to shove an ogre into a pit of acid. She sheathes her weapon so she can use the strength of both arms. The DM rules that her roll must both exceed 20, since an ogre is a big, heavy opponent who is very difficult to shift, and beat the ogre's attack roll. If her roll beats the ogre's AC, she can inflict normal unarmed damage, even if she wasn't able to throw it into the pit. Had she attempted the same move on a foe her own size or smaller, the DM might have ruled she only needed to beat its attack roll. 
• What if two or more characters team up to shove the ogre into the pit? Roll once with advantage, or with a bonus, or roll each attack and apply the highest. 

So, there it is. It's rudimentary, and has a few wrinkles to iron out, but I like it for its simplicity and flexibility, and its easy integration with the basic combat rules.