For anyone else, I had the same experience with the dropdown search/filtering not working in Bootstrap modal (Bootstrap v4.4). I ended up just using the multi-select listbox https://blazor.radzen.com/listbox in the Bootstrap modal and that worked. It doesn't give the same dropdown selection experience but I can at least search and select items in a list.
But it sounds like using the Radzen DialogService https://blazor.radzen.com/dialog modal to display the dropdown instead of Bootstraps modal would work as well, just haven't tried that.