The critical issue is whether your signal when you are transmitting is higher than the signals you normally receive. If it is not then the muting won't work. If it is then it may work. You have to juggle the mute level until your TX signal just blanks it. I have made mute work (simplex) with an external HT on VHF. So the concept is strong but does require your TX signal into the dongle be somewhat (10dB or more) higher than the signals you want to hear on receive.
(As a side issue you MAY be trying to push an RTLSDR dongle a little harder than it can handle. They have very small dynamic range. They are nice "get your feet wet" tools. But it's difficult to swim in a wading pool.)
{^_^}