The processing of any noisy stimuli requires extra means from the nervous system. Several electrophysiological studies have proved that the processing of noise starts approximately 150–200 milliseconds after stimulus onset. This is suggested by the fact that the amplitude of the face-selective N170 component has been reduced and its latency prolonged by adding Gaussian noise to a face stimulus while the earlier P100 component is not influenced by added noise. While previous studies tested the electrophysiological correlates of noisy face perception in humans, the nature of added noise has not been studied yet in relation to face processing. Our goal was to distinguish the effects of added phase-noise from that of another irrelevant, overlapping non-face object (a car) on the electrophysiological correlates of human face processing. Subjects performed a two-alternative gender discrimination task with seven levels of difficulty, equalised between the phase-noise and overlapping object conditions. We found that the first component where an amplitude reduction took place was the N170 component when adding phase-noise to the face stimuli but not in the overlapping object condition. The amplitude of the later positive P220 component, on the other hand, increased significantly in both noisy conditions. Our results suggest that the processing of phase-noise starts in the time range, reflected by the N170 component, while the interaction of a face and of another overlapping irrelevant stimulus does not happen until the P220 component.