You are definitely using themes if you see this. You can't use Radzen.Blazor components without a theme.
You can disable preloading via the corresponding parameter of RadzenTheme.
UPDATE: Nevermind. I found what you mean - there is a warning in the developer tools console. We will address that with the next release. Meanwhile you can set the PreloadIconFont property of RadzenTheme to false:
If you don't have RadzenTheme then you are probably preloading the font manually. We can't help in this case. Anyway the getting started instructions is a good start: Get Started | Free UI Components by Radzen