For me, the most irritating thing is not really a performance related issue, but the dreaded ground-loop. That annoying 50 Hz mains hum is enough to make you want to throw your set out of the window. And as luck has it, whenever the suspected culprit is back with the technician / dealer, it will behave perfectly.
When I was running audio and video systems together, the culprit was almost always the external antenna, or the SCV cable point. Disconnect it, and the hum disappears. I've since separated the systems, but a brute-force method of resolving this would be to use an antenna isolator.
For audio, this often happened when the components were connected to different wall switches. As far as possible nowadays, I try to make sure that all components are plugged into the same distributor.