What edition are you referring to? (Remember, there are 6 editions out there, and things change between editions)
Not sure what you mean by "ability damage from physical effects" either, as most ability damage comes from magical attacks (nerve strike, reduce attribute spell, toxins too I think off hand)
Are you referring to the dice modifiers from wounds?