Trouble understanding clearly when there is background noise

My aunt, my mother, their mother (my grandmother), and I all seem to have the same hearing dissability and none of us have a clue what it is or what can be done about it.  I can't find any similarities to our problem on any forum.

I have a terrible time understanding people who are talking if there is the slightest background noise.  I also have tinnitus which I have learned to accept and deal with.  I went to a hearing aid specialist today who performed a typical hearing test.  My results are far better than anybody in my similar age group.  In fact, being in my mid-forties, I performed far better than many 20 year olds.  I can hear a pin drop in the next room, but I have a terrible time hearing people speak clearly.  Add the smallest amount of background noise and my ability to understand what is being said becomes a severe problem.  The best way I can describe this is blurred, garbled, or distorted hearing.  The doctor thinks that I was just there to get a free hearing test.  I am serious that this is a huge problem for my relatives and I.  I was hoping that a hearing aid would help me, but they said it won't help at all, and could make things sound worse.

What is going on?  This condition has very gradually worsened over the past 20-30 years or so, and people keep saying that I have selective hearing.  I am becoming very good at understanding people if I focus on their lips, but if they turn around I often lose what they say to me.  If it is perfectly silent when they are talking I don't seem to have as much of a problem understanding the person.

It sounds like you have unbalanced hearing, when the two ears hear "differently" for some reason.

there is a nerve called Auditory Nerve that goes from the ears to the brain.  Over time this can start going bad.  I dont know if you should see a neurologist or neurotologist for that.  Mine is diseased a little from ear infections and I have a problem hearing words also in a crowded room.  If it is extremely loud, I cant make out anything but others can.  Even the TV being too loud makes this worse.  I also have tinnitus and sometimes its louder than the words being spoken.  Dont know if this will help and this is all I know.

I tried plugging one ear and then the other ear to see if listening with one ear made any difference and it didn't.  Good suggestion though.  It was something I hadn't thought of trying yet.