The following explanations are overviews. In the interests of brevity, they neglect some corner cases.

Definitions:

Video memory: For our purposes, memory accessible by the GPU. For details, please see the “Graphics Memory Reporting through WDDM” whitepaper (http://www.microsoft.com/whdc/device/display/graphicsmemory.mspx).
Hardware acceleration: The process of performing computations on the GPU instead of the CPU.
GDI: Graphics Device Interface – a 2D rendering API.
Surface: A chunk of memory representing a 2D array of pixels.
Present (verb): A function used to request that the contents of a backbuffer be displayed on screen. E.g. In DXGI, IDXGISwapChain::Present.
GPU acceleration: Also known as hardware acceleration. This means performing work on the Graphics Processing Unit (GPU) instead of on the Central Processing Unit (CPU). When the GPU can be used, it is typically much faster than the CPU, hence the term “acceleration”.

Overview:

XP uses the XP driver model (XPDM) and video memory is a fixed/finite resource. Windows Vista and Windows 7 use the Windows Display Driver Model (WDDM) where graphics memory is virtualized. This means that in XPDM you are limited to the amount by the amount of physical memory on the graphics card, but in WDDM you are not. Therefore WDDM allows running either more applications or more resource intensive applications than was possible with XPDM. You can read more about WDDM athttp://msdn.microsoft.com/en-us/library/aa480220.aspx. There is no hard limit of video memory in either WDDM or XPDM, but depending on the driver, there are other limits. For example, some drivers have limits on the number of surfaces that can be created.

In Windows Vista and Windows 7 the Desktop Window Manager (DWM) is used. Instead of presenting directly to the front buffer as in Windows XP, apps present to a DWM buffer. The DWM then composes the buffers together to create the image you see on your screen.

Taking advantage of GPU acceleration:
In XP GDI is GPU accelerated to various degrees depending on how the OS is configured or the device driver (for details see Hooking Versus Punting: http://msdn.microsoft.com/en-us/library/ms799743.aspx). In Vista, GDI is not GPU accelerated however the performance difference is usually not perceptible by the user. In Windows 7, some limited GPU acceleration for GDI was added to enable some video memory optimizations. Direct3D and WPF are GPU accelerated on all 3 OS’s. Direct2D is GPU accelerated too, but it is currently available on Windows 7 only. Microsoft has announced that it will releasing Direct2D on Windows Vista, and it will be GPU accelerated there too.

The Desktop Window Manager uses GPU acceleration, so apps on Windows Vista and Windows 7 benefit automatically. For example, when you drag a window in XP the app receives a request to redraw the window. In Windows Vista and Windows 7, the DWM maintains a copy of the window contents in graphics memory, so the app doesn’t need to redraw the window.

If you notice an app performing worse in Windows Vista as compared to Windows XP, please don’t assume that there is nothing you can do about it. For example there used to be a performance problem with windbg.exe on Vista that was caused by an interaction with the DWM. Turning off the DWM did make the problem go away, but a change to windbg.exe fixed the problem for good. If you have performance problems, as a first step I’d recommend profiling your app to see what is hogging the CPU (especially compared to XP).

Virtual memory considerations:

Wikipedia has a great description of virtual memory, but here’s a quick overview: Processes store and retrieve information through addresses similar to the way people use phone numbers when making phone calls. Virtual memory addresses are to physical memory addresses as speed dial numbers are to a full phone numbers. Just as different people have the speed dial number “4” mapped to different full phone numbers, different processes have the virtual address “0x00A4B3C0” mapped to different physical memory addresses. Speed dial numbers are used as shortcuts, but virtual memory addresses are often longer than needed to address all of physical memory. So why are virtual memory addresses used?
1. Process isolation: Even assuming that you don’t have any malware on your computer, the computer equivalent of accidentally dialing the wrong phone number has devastating consequences. We restrict processes to using virtual memory addresses so they can’t interfere with memory belonging to other processes.
2. Contiguous memory: Processes often depend on having large sections of sequential addresses available.
3. Resource sharing: The amount of RAM installed on a typical PC is small enough that it’s possible to exhaust it by running many processes at the same time. If the processes used physical memory addresses, they would just fail when they ran exhausted physical memory. With virtual addresses, the OS can use both physical memory and hard disk space to hold the information to which a virtual address refers.

Virtual memory problems start when an app is compiled as 32-bit and it needs more than 2GB of virtual address space. You may wonder why 2GB? After all 2 raised to the 32 is 4GB. The answer is because the other 2GB of virtual address space is reserved for the kernel. There are ways to give your app slightly more user mode virtual address space at the cost of less kernel address space, but the best solution is to use a 64-bit OS and compile your app as 64-bit. See 64-bit programming for Game Developers (http://msdn.microsoft.com/en-us/library/bb147385.aspx) for more information (the material was written for game developers but it’s applicable to any app).

If you really, really can’t use a 64-bit OS, then you can try to look at ways to reduce virtual memory usage - and I say “try” because there’s no easy way to do that. So what does this have to do with video memory? 
1. Large mapable surfaces consume large amounts of virtual address space. In order to use video memory, the CPU has to populate it. One of the mechanisms for populating a surface is to “map”* it to a range of virtual addresses. When you create a mapable surface, Direct3D allocates a range of virtual addresses for this purpose. It does not do it at the time you map the surface because there might not be a large enough contiguous chunk of virtual memory available.
2. Even non-mapable surfaces may consume virtual address space under XPDM. It depends on the driver. Non-mapable surfaces consume virtual address space in Windows Vista RTM, but this was changed in Vista SP1 (and Windows 7) because it was causing problems for games.
3. WDDM drivers have a user mode component, so I’d expect that apps would use more user mode address space and less kernel address space on Windows Vista and Windows 7 as compared to Windows XP.
4. WPF is GPU accelerated, but it keeps around a system memory copy of resources, so virtual address space is used.
5. Direct2D does not keep a system memory copy of resources and uses non-mapable Direct3D surfaces internally, so virtual address space is conserved. (The tradeoff is that Direct2D apps must handle device removed errors)
6. As mentioned earlier, on Windows XP GDI calls may or may not be handled by graphics card driver. For device-managed surfaces (see http://msdn.microsoft.com/en-us/library/ms799615.aspx for information about GDI surface types) typically no virtual memory is used. On Windows Vista, GDI calls are never handled by the graphics card, so GDI objects do consume virtual memory. On Windows 7 there is some GDI acceleration, so some apps will consume less virtual memory as compared to Windows Vista.

Also good to know:
1. There are tools that let you view virtual address space - for example, Process Explorer (http://technet.microsoft.com/en-us/sysinternals/bb896653.aspx). If you can figure out what is eating up your address space, you might be able to take action.
2. Remember that virtual memory and physical memory are two separate things. You can use virtual memory address space without using physical memory (e.g. by calling VirtualAlloc with the MEM_RESERVE flag) and you can use physical memory without using virtual address space (e.g. by allocating non-lockable surfaces in Direct3D).
3. This white paper might be helpful: Virtual Address Space Usage in Windows Game Development (http://www.microsoft.com/whdc/device/display/WDDM_VA.mspx)

I hope this information helps you to not exceed your virtual memory limits, but if you still have problems you’ll have to switch to 64-bit Windows and compile your app as 64-bit.

* Map is the D3D10 API. The D3D9 API was LockRect.