I think I'm gonna have to do it anyway, because some players claim they get nausea when playing at such low resolution (320x240), and the only way to give them higher resolutions that perform reasonably is to have it hardware accelerated.
Renderer is abstracted away already, but the real difference would probably be occlusion culling... With raycasting, I get it for free, but if I'd go down the hardware accelerated path I'd have to pick something more clever.
Raycasting and software rendering in general tends to scale poorly with resolution, even with vectorization and all the bells and whistles of modern CPUs.
As an example this[0] video shows the benchmark from Post Apocalyptic Petra running on my previous GPU (RX 5700 XT) which all it does is build a per-frame (client-side) vertex-buffer in OpenGL 1.1 (the engine was made for actual retro PCs running DOS and Win9x so it does some rudimentary occlusion culling but that mainly affects 90s hardware, not anything released since 2000 or so). If anything, the rendering has so little overhead that half of the framerate is "eaten" by the FPS counter overlay :-P.
Thinking about modern games, a single character model probably has more vertices than my entire level (and yours probably), so it's definitely reasonable to expect occlusion culling for such simple geometry might actually reduce performance rather than increase it.
But ignoring the GPU you have on your system is boring