I think your drums are "out of round" and need to either be machined or replaced fully.
You said your "shoes" looked ok. Maybe they are not as ok as you thought.
The drum is full of springs and stuff and they get exposed to a lot of heat, wear and tear, and the likes and may need to be replaced also.
Check the insides of both your rear rims. Is there a lot of black brake dust?
If so then your shoes are rubbing while you are driving also and the reason you don't hear that is because sound travels backwards and it only becomes noticable when you are slowing down.
Lastly, after a long drive put your hand on both drums. If they are scalding hot, then your brakes are rubbing.
I had this problem also last summer. Needed a brake overhaul on both rears. Installed new springs, replaced the emergency brake cable and machined the drums. Now, no more problems.