Depending on how you want to restrict access and to what extent, choose one of the following options: You can also restrict access to your computer in different ways. I think this article from TeamViewer is what are you looking for: How can I restrict access for TeamViewer connections to my computer? To achieve your goal, you should use TeamViewer's way. You make a connection to a Teamviewer ID, and TeamViewer passes the connection down through the TeamViewer client's established tunnel to the destination and you then you are prompted for password and then the connection establishes afterwards. When you run TeamViewer, you are assigned an ID on their broker server. You can see the answer of kobaltz for the question How exactly does a remote program like Team Viewer work?
TeamViewer users connect each other by using a third party server in the middle. When you connect someone with TeamViewer, you don't do it directly. I don't think you can do this with firewalls, at the networking layer.
