did you try frameskip=1 ?

That works!

Thank you so much!

Okay, I thought this was resolved, but it isn't.

Frameskip at 1 makes it so it takes longer to get worse, but it still gets worse, eventually to the point where playing the game is hard. At the same time, the recording itself is still fine.

I think I'm gonna have to go back to 0.73. I have no clue what's wrong with 0.74, it's almost like the recorder has an internal memory leak that only affects DOSBox and not the rest of my computer.

How much free space do you have on your hard disk?

90 GB

how long does it take to start working not properly?

what is your dosbox version and where have you got it from?
what is your Operating System?
what are your PC specifications?

post content of "DOSBox Status Window" (in Windows: right click on its title bar -> edit)
enclose your dosbox's config file

1. About 7 minutes
2. DOSBox version is in the title... 0.74 from the official website
3. Windows Vista
4. My processor is an AMD Turion(tm) X2 Dual Core Mobile RM-74
My RAM is 4 GB
and I've got a 64 bit operating system

5. "DOSBox version 0.74
Copyright 2002-2010 DOSBox Team, published under GNU GPL.
---
CONFIG:Loading primary settings from config file C:\Users\Zachski\AppData\Local\
DOSBox\dosbox-0.74.conf
MIDI:Opened device:win32
CPU speed: fixed 3300 cycles.
CPU speed: fixed 3630 cycles.
CPU speed: fixed 3993 cycles.
CPU speed: fixed 4392 cycles.
CPU speed: fixed 4831 cycles.
CPU speed: fixed 5314 cycles.
CPU speed: fixed 5845 cycles.
Capturing Video to C:\Users\Zachski\AppData\Local\DOSBox\capture\sierra_000.avi"

However, I should add it lags even when the CPU speed is set to 3000 (and I have to increase the CPU speed for my game, otherwise autosaves take forever)
6. See attachment

Does it happen with other games too?

This happens for me as well, in 0.74 from the main site. Not in any previous releases. I was initially hoping that it was only Magic Carpet 2, but it seems to be happening in X-Com 1.2, as well. I'm running:
XP SP2 32-bit
3.4 GHz P4
2GB RAM

It takes 2-3 minutes for mine to drop to single digit FPS, which it does rather suddenly once I notice a slowdown. If I only capture a couple of minutes worth, I don't notice any slowdown.

Tried it with Mechwarrior 2 as well. With that, performance went down sooner (was noticeable in under a minute) but more gradually. Became unplayable at about the same file size as the X-Com movie, about 16 MB.

For Mech2 (which has a higher memory limit set that normal in my config files):
DOSBox version 0.74
Copyright 2002-2010 DOSBox Team, published under GNU GPL.
---
CONFIG:Loading primary settings from config file mech2.conf
Memory sizes above 31 MB are NOT recommended.
Stick with the default values unless you are absolutely certain.
MIDI:Opened device:win32
DOSBox switched to max cycles, because of the setting: cycles=auto. If the game
runs too fast try a fixed cycles amount in DOSBox's options.
Capturing Video to capture\mw2_000.avi
Stopped capturing video.

Trying again with X-Com using fixed cycles caused it to last a lot longer than with everything on auto. The video got to around 7 minutes before it got really bad (music stuttering horribly). It didn't even register 5% CPU usage in task manager though, and adjusting the cycles seemed to have no effect either way. Wound up with a 50MB avi from that.

DOSBox version 0.74
Copyright 2002-2010 DOSBox Team, published under GNU GPL.
---
CONFIG:Loading primary settings from config file xcom.conf
MIDI:Opened device:win32
SHELL:Redirect output to nul
SHELL:Redirect output to xcomutil.log
Capturing Video to capture\geoscape_001.avi
CPU speed: fixed 10000 cycles.
CPU speed: fixed 12000 cycles.
CPU speed: fixed 14000 cycles.
CPU speed: fixed 16000 cycles. If you need more than 20000, try core=dynamic in
DOSBox's options.
CPU speed: fixed 18000 cycles. If you need more than 20000, try core=dynamic in
DOSBox's options.
CPU speed: fixed 20000 cycles. If you need more than 20000, try core=dynamic in
DOSBox's options.
CPU speed: fixed 22000 cycles. If you need more than 20000, try core=dynamic in
DOSBox's options.
CPU speed: fixed 24000 cycles. If you need more than 20000, try core=dynamic in
DOSBox's options.
CPU speed: fixed 26000 cycles. If you need more than 20000, try core=dynamic in
DOSBox's options.
CPU speed: fixed 28000 cycles. If you need more than 20000, try core=dynamic in
DOSBox's options.
CPU speed: fixed 30000 cycles. If you need more than 20000, try core=dynamic in
DOSBox's options.
CPU speed: fixed 29000 cycles.
CPU speed: fixed 28000 cycles.
CPU speed: fixed 27000 cycles.
CPU speed: fixed 26000 cycles.
Stopped capturing video.

What's DOSBox's memory usage in the task manager?

Checked this with mech2 until my frame rate stayed below 1 fps. Memory usage didn't go up significantly. Virtual size hovered around 154000K the whole time, working set went from a little under 80000k to a little over 82000k during the capture, and dropped back down to 81832k after turning off capture.