Yes I find it so annoying! There have been a few times where I nearly side-swiped a person because they were in my blind spot even though I was looking to make sure no one was there. I think a lot of times it is people just being jerks and not wanting you to get in front of them so they speed up.
I will check my side, look forward to make sure I'm not getting too close to the car in front of me, and by the time I look to my side again as I move over someone will be right there speeding so I don't get in front of them. Why can't people just respect the fact that I have my turn signal on and I am going to get over? California drivers are the worst!!