Simply go to edit your forum page, head to Access Restriction. It must be a drop down menu which displays Make Public, Allow Everyone(if your forum restriction is currently open for everyone), click on that and find the Group which you allow viewing access.
In addition, if you wish to keep the forum for public viewing but don't want everyone to post a message in it, go down to Forum setting and look for Group Only Posting: drop down and select which group you want to have posting access. Lastly, choose the level of the group members you are granting a post access under Group Level Required To Post: drop down. Then save.