Gambit might be misremembering or there might be a second valid method. The way that worked for me was:

Load a .txt file, such as SomeDungeon.txt

SomeDungeon.txt will now show up in your recent files

Delete SomeDungeon.txt, rename File.rtc to SomeDungeon.txt and put it in the same place

Load it from the recent files, and the editor will load it
As Gambit has noted, this will work, but there will probably be some small glitches and resources that are compiled into the RTC file won't be preserved.
Anyway, there were tools for converting RTC text files to DSB, although they're pretty obsolete these days. No attempt was ever made to make it parse binary RTC files, though. There are enough differences between how RTC and DSB do things internally that it'd probably need a bunch of hacks, like there are for converted DM/CSB.
In addition to copyrighted code, the other issue is that RTC still contains the copyrighted graphics and sounds from DM. This is also the reason why DSB also can't be considered "free and open-source" despite the source code being available.