Queue might be assigned for queue transfers, so we need to unref it if the memory has been assigned to a specific queue. Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/5079>