(My personal 2c worth)
I would actually recommend starting with a very good book on the "C" programming language (to understand what the code does and also debugging and the "stack" concepts). ( I spent plenty of time in 8086 books, but although I could write simulators, could not understand why code was compiled the way it was ).

On the 8086 side, you will need to know the "usual" ports and addresses, and know that "int xx" were treated as macros (with AX as selector and other registers as variables).
Ralf Brown's Interrupt List sheds some light into what those did.
(Initially the INTs provided routines to service "interrupts" or real-world events signalling the cpu that something needed attention)

Hardware devices are mostly separated out, but dosbox only emulates essential "noticeable" features (we can ignore timings and real-world synchronizing issues, because we can force the order of events and also leave the underlying operating system to take care of some of those issues).

The 'demo scene' has source code that explains some of the tricks and how they push the hardware to achieve them.

There are a lot of dos "features" that had hidden or unusual effects that have been captured by trial and error (and lots of debugging), so dosbox has plenty of rewritten dos calls (int 21h) emulated as well.

The SDL library provides a nice cross platform handler for events (keyboard, graphics, sound etc); so understanding that side will help understand how we glue the simulator side back onto the operating system.