You sound just like my Dad... and well, I happen to agree with you: my first-hand experiences as a passenger lend themselves to that last line of yours, that those unable to drive stick tend to be lacking in certain driving skills.Some of the best rides out there don't come equipped with an automatic transmission, which keeps them out of the hands of those not skilled enough to handle them in the first place.
You need to be capable of driving either, in my opinion, or you are lacking in certain skills that I believe all drivers should have.