I pinned all modules at first. Then closed pins through the X in their own little menu to the right of the module viewer. The button to pin/ unpin them all doesn't change by doing this. In the second screenshot you can see that I unpinned them and in the module viewer it still says unpin.
Using the button only in the module viewer changes the button state.