Actually depends on what you are distributing 😀. You are free to distribute the sources of your patched dosbox (obviously). You would be free to distribute your proprietary binary DLL (provided you did not include any dosbox headers or link with any other GPL software in the making - kinda hard if you are making a new cpu core, but let's say it works 😁) in a separate package under your license. But the minute you distribute a compiled and linked dosbox executable that has your DLL as a requirement, you are making an extended version of dosbox that is one whole program and has to be GPL. Simply wrapping proprietary code into a DLL does not get you off or everybody would do it. That's why LGPL was invented. (Plain)GPL libraries can only be used in GPL programs!
Now, the license cannot distinguish what was first - your DLL or dosbox, or in this case is dosbox linked to Steam or Steam to dosbox. It only says that "Linking ABC statically or dynamically with other modules is making a combined work based on ABC." So linking with Steam libraries makes that a combined work based on DOSBox and such a GPL work. That's why GPL is sometimes considered a "viral" license - it "infects" everything it touches 😀
To distribute non-free programs with GPL software, "...you must make sure that the free and non-free programs communicate at arms length, that they are not combined in a way that would make them effectively a single program." This is clearly not the case when a modified dosbox version needs steam.dll to function properly.
Now, there was an interesting post on the steam forums, I thought I'd post it here before it gets lost:
ATimson wrote:If it's truely a binary wrapper, injected into a compiled EXE and not edited into the source code, then I think the following could be considered as applying:
"In addition, mere aggregation of another work not based on the Program with the Program (or with a work based on the Program) on a volume of a storage or distribution medium does not bring the other work under the scope of this License."
If it can be unassociated into a binary identical to the unwrapped one, there's a definite argument that the wrapper constitutes a distribution medium.
If this really holds...I don't know 😀