It does work, but it's very awkward to do unfortunately, since you will roll and change speed as long as you hold the button down.. ie, you can't maneuver and afterburn at the same time as long as you stay in mouse flight mode. In WC1, I remember using the mouse in one hand, and having the other on the keyboard, before I got a joystick. The tab key would activate the afterburners, even when flying with the mouse. It seems they disabled that ability for mouse mode in WCIII, or it causes a conflict of some sort.
As for my system specs:
Pentium 4 3.4ghz with HT
WinXP Pro SP2
1 gig ddr2 ram
ATI X800XT 256mb gfx card
Audigy2 sound card
DirectX version 9.0c
Joystick is a Saitek X45 usb
The joystick may be causing some of the issues, I borrowed a friend's Logitech 3d joystick (also usb) and tested with it, and the jumping was a little better, but still a problem. I've got DosBox 0.65, and it does run stably, but the joystick refuses to calibrate at all in that version, and sticks in the lower right corner. Version 0.63 on the other hand gave me about half the normal joystick range, but crashed frequently. I'll keep experimenting though, VDMSound seems to be giving me the best results now.
Edit: Is there any way to assign a deadzone of some type to the joystick settings in VDMSound?