At the Attendees page, you are able to determine who will be invited to your meeting.

The people you choose will receive email notifications about the meeting in the "Pubish" section.

You can choose attendees either by their role, or individually.  Once you add them to the list, they will no longer appear in the "Individuals Not Yet Invited" list.  This feature was designed to allow you to talk about people behind their backs and ensure they won't be attending (or at least you didn't invite them).

To add users, select them and press the ">>" button to move them to the "Invited Attendees" list.

To remove attendees from the list, check the respective checkbox and press the "<<" button.

Entire groups of people can be added very easily.  Removing entire groups from the meeting is not supported.  Sorry.