It's the documentary feeling, the film stock that has heavy grain.
Plus - and this is why it is genius - Friedkin incrementally increased the exposure to scenes of possessed Regan.
The first time - a minute, the second time- minute and 1\2, third time- 2 minutes, etc.
He keeps slowly increasing the exposure until you get the full-on final showdown.