The rubber gasket around the parking brake is attached to a plastic frame which attaches to the console via some plastic tabs that are melted to hold them in place. You're supposed to heat the tabs up enough to almost melt them and then straighten them out so you can pop that frame off, then melt them again once you've put the new rubber gasket. The factory vinyl & leather boots attach the same way if you want to replace it with one of those.
Your other option is to replace it with an aftermarket leather boot which will still require removing that piece (but you can just cut off the tabs) and gluing the new boot in place.