When performing a user test with eye tracking we recommend the principle of Retrospective Think Aloud. It basically means that the test is done and recorded in a settled and relaxed environment, that encourages the participant to act as natural as possible. Afterwards, you'll review the recording of the test together and explore usability pitfalls, design inconsistencies and other issues
To make things a bit easier, here are a few tips:
It is OK if they speak during the test. Retrospective Think Aloud suggests that you encourage the users to perform the test without comments and review it together afterwards. However, some people want to speak their mind during the test. That's OK, and here's why: What they want to say is probably important feedback to you, and there’s a risk that they’ll forget it if they're asked to bring it up in the retrospective phase. Also, making people feel comfortable is more important than invariably sticking to "the protocol".
Don't take it personal, when they bash your design. If you work in smaller teams, chances are that you're the one who has created the design that is being tested. People will have opinions, and that is partly why they're here. When they unload a pile of unpleasantness at your lovely design, don't try to defend it or explain whatever "technical CMS related reason" that's behind your inconsistent UI. What they're actually doing is helping you create a better product. And that should be appreciated.
Note-taking can be done, both during the test, and while replaying the video recording. Even though you are recording the test, you should be aware and alert. Some user comments, hiccups, or sudden insight might be good to note right away, as they might not be caught on the recording.
When the task is completed and you have stopped the recording, it's a good opportunity for a quick debrief before jumping in to the replaying. The debriefing could just be a quick question about their general impression of the test. Or, it could be refilling their coffee. There are actually two practical reasons for doing this:
It gets people comfortable and ready to review the recording.
It allows you to think about something else, while the recording is being uploaded.