Just because I had an idea doesn't make it a good one.
I thought about this a lot in the car today, because I'd like to reuse what you come up with in the future. Achieving balance is the tough part here, because even if players aren't minmaxers, they'll still end up getting biased by exceptionally good/bad skills. Here are a LOT of thoughts. I had a lot of time in the car.
Here's what I came up with to help shape my thinking:
- a skill is either always available or situational. Weapon specialization is always available; favored enemy is situational.
- for combat bonuses, always available should be equivalent to a +1 stat bonus e.g. +1 to hit and +1 damage; or +2 to hit and +0 damage.
- for combat bonuses, situational should be +4 (or more).
- I've always treated +2 to-hit equivalent to +10% to hit, which would mean the expected damage would increase by 10%. A fighter brandishing a longsword who strikes 25% of the time (16-20) would have an expected damage/round of 1.125 (0.25*4.5). The same fighter who gets a +2 to hit (14-20) would see his damage go up to 1.575 hp/round. Using a 50% chance, he'd go from 2.25 to 2.7. +2 damage is harder for my brain to compute, because it depends on the likelihood of hitting. I usually assume 25% likelihood (16-20) and a longsword: 4.5*.25 -> 1.125 damage/round. If the fighter gets +2 damage, that's 5.5*1.375 (less than 1.575) damage/round. If the same fighter has a 50% chance (11-20), the +2 damage would take him from 2.25 hp/round to 2.75 hp/round. Slightly higher, but you never seem to throw creatures at me where I have a 50% chance of hit.
- whatever I said before about scaling with level gain, ignore. I think it's a good idea. I think it's a lot harder to achieve than #1 and #2.
Now feedback on the specific skills, using the above general guidelines:
Weapon specialization: seems fine to me. Something below on ranks though.
You stated the skill gain for martial characters, but what about non-martial characters? I think you want them to also get combat skills since Armor and Medium|Large weapons are in the list.
Brute: Feels weaker given both the math above (third point) and my second point. Why +1 and not +2?
Defender: I had a totally different idea for defender, something more like bodyguard. I may not have expressed it well. One problem I have with D&D (opposed to 4e) is, why monsters don't skip the tin cans and first eliminate the bigger threats, like wizards. Wizards can hit harder and go down faster, so save the tin cans for a few rounds later. Tin cans tend to run away after half of the party ends up in the monsters' stomachs anyway. So my original idea: the bonus to the armor class would go to the target character that the defender was defending, like Kilian defending Tiana. This is a situational skill, hence being willing to go with the +4 AC opposed to a lower AC.
Favored enemy: see math above. To hit > damage IMO.
Two weapon fighting (2WF): Works for me, though I have concerns about the second attack. There's a forum post somewhere about secondary attacks for fighters and that changing a lot more than just a bit of damage. I can't find it now. I found it solely by browsing and being bored. This turns a d6 or d8 attack into a 2d6 attack. That's pretty huge. I think there was a penalty to the second attack, though I forget what it was. The penalty probably offsets the extra attack. I think this would end up as a +1 AC or +2(ish) damage, which is compliant with the guidelines above.
Shield bash: You saw the math. This is awesome. +1 AC and a free d4 attack, and STR bonuses should apply (IMO). I'm not sure what I'd change. Bash only if the target attacked the character and the attack missed? Allow a saving throw against DEX or paralysis as well as the to-hit attempt? Maybe it's ok, but it's +1 AC and +2(ish) damage. The and vs or is what makes me like 2WF and not shield bash. /shrug
Defense: That's it. It'll never be chosen, especially if non-fighters can pick up Armor and fighters are picking up shield bash.
Toughness: +2 hp at level 1 is awesome. Those same +2 hp by level 10 is a poor bargain, especially since other skills (shield bash, 2WF) scale with the character level.
Medium and large weapons: Yep, no problem there. Mages should be able to pick it up after medium weapons. I don't know if it will ever happen, but a wizard swinging a claymore would be funny to see.
Sneak attack: I thought we removed this for our version of scouts. Right?
Armor: Cool! This benefits both thieves and magic-users. Do you intend this to be wear without class penalties for wearing? The thief supplement currently has penalties for wearing > leather armor, so I'd expect this skill to either offset or negate those penalties. Negate might be a little high. As for magic-users, negating the "can't cast in armor" restriction might be really steep. I like it. I'm just wondering about class balance.
Point blank: I agree with all three. #3 might provide too many benefits relative to others.
Skill progression:
I like AD&D 2.0 when they added proficiencies. It's very much like your house rules. I also like how additional skill-ups allowed for either deeper growth in one area, or breadth in other areas. You first made me think about proficiencies with the comment about weapon specialization (+1 attack at three ranks).
If you want to allow multiple ranks, you can solve this by doubling the skill points gained at 3/5/7/... but allowing only one skill to be new -- the other has to be depth. At level three, a character would have a rank-2 and a rank-1. At level 5, it'd be 3/1/1 or 2/2/1.
Or, just leave it flat and don't allow for depth on skills like specialization and defender. Not my preference but definitely easier.
The break-even point is around 30% to hit, or 15-20.