This happens for me as well, in 0.74 from the main site. Not in any previous releases. I was initially hoping that it was only Magic Carpet 2, but it seems to be happening in X-Com 1.2, as well. I'm running:
XP SP2 32-bit
3.4 GHz P4
2GB RAM
It takes 2-3 minutes for mine to drop to single digit FPS, which it does rather suddenly once I notice a slowdown. If I only capture a couple of minutes worth, I don't notice any slowdown.
Tried it with Mechwarrior 2 as well. With that, performance went down sooner (was noticeable in under a minute) but more gradually. Became unplayable at about the same file size as the X-Com movie, about 16 MB.
For Mech2 (which has a higher memory limit set that normal in my config files):
DOSBox version 0.74
Copyright 2002-2010 DOSBox Team, published under GNU GPL.
---
CONFIG:Loading primary settings from config file mech2.conf
Memory sizes above 31 MB are NOT recommended.
Stick with the default values unless you are absolutely certain.
MIDI:Opened device:win32
DOSBox switched to max cycles, because of the setting: cycles=auto. If the game
runs too fast try a fixed cycles amount in DOSBox's options.
Capturing Video to capture\mw2_000.avi
Stopped capturing video.
Trying again with X-Com using fixed cycles caused it to last a lot longer than with everything on auto. The video got to around 7 minutes before it got really bad (music stuttering horribly). It didn't even register 5% CPU usage in task manager though, and adjusting the cycles seemed to have no effect either way. Wound up with a 50MB avi from that.
DOSBox version 0.74
Copyright 2002-2010 DOSBox Team, published under GNU GPL.
---
CONFIG:Loading primary settings from config file xcom.conf
MIDI:Opened device:win32
SHELL:Redirect output to nul
SHELL:Redirect output to xcomutil.log
Capturing Video to capture\geoscape_001.avi
CPU speed: fixed 10000 cycles.
CPU speed: fixed 12000 cycles.
CPU speed: fixed 14000 cycles.
CPU speed: fixed 16000 cycles. If you need more than 20000, try core=dynamic in
DOSBox's options.
CPU speed: fixed 18000 cycles. If you need more than 20000, try core=dynamic in
DOSBox's options.
CPU speed: fixed 20000 cycles. If you need more than 20000, try core=dynamic in
DOSBox's options.
CPU speed: fixed 22000 cycles. If you need more than 20000, try core=dynamic in
DOSBox's options.
CPU speed: fixed 24000 cycles. If you need more than 20000, try core=dynamic in
DOSBox's options.
CPU speed: fixed 26000 cycles. If you need more than 20000, try core=dynamic in
DOSBox's options.
CPU speed: fixed 28000 cycles. If you need more than 20000, try core=dynamic in
DOSBox's options.
CPU speed: fixed 30000 cycles. If you need more than 20000, try core=dynamic in
DOSBox's options.
CPU speed: fixed 29000 cycles.
CPU speed: fixed 28000 cycles.
CPU speed: fixed 27000 cycles.
CPU speed: fixed 26000 cycles.
Stopped capturing video.