Only for my curiosity do you have used topaz ?
I have tried to use topaz to upscale pixel art content (xwing cutscene) like this:
with poor result.
I did - specifically, Topaz Video Enhance AI using the Proteus 6-Paramater model (with lots of tinkering with the paramaters until I found a good balance).
With captured footage of VGA games like WC2 and X-Wing, there's a trick you have to do before you start in Topaz VEAI. VGA games of the era were typically rendered to screen at double their actual native resolution using nearest-neighbour scaling (hence the visible pixelation). So first, you need to halve the dimensions
of your captured footage to get it back to its native resolution - otherwise, Topaz VEAI will see pixelation as part of the source footage and try to maintain it
So, for example, if your captured footage has dimensions of 640x480 or 640x400, you need to first scale it down to 320x240 or 320x200, then
run it through Topaz VEAI. You'll probably find the results much improved.
In the case of my WC2 and Undertale experiments, I used Adobe After Effects to scale down the footage. I use AE specifically because I could set the layer quality to "draft," which made sure the footage got downscaled
using nearest-neighbour scaling:
I chose this method because it was the first way I thought of to downscale the footage using nearest-neighbour scaling, which is important to ensure no smoothing is added to the downscale and you get a 100% mathematically-accurate recreation of the game's native resolution. I wanted to avoid any smoothing calculations being applied (bilinear, bicubic, Lanczos etc.), because they would also see the pixelation as part of the image and lead to results that aren't as crisp.
I went with After Effects rather than Handbrake or Adobe Media Encoder because they don't offer nearest-neighbour scaling. You don't have to use AE, but do try to find some method that doesn't apply any smoothing algorithms. You could probably use FFMPEG or something else. I'm sure some Googling will find a solution :^)