Here's a detailed blog post on how the Qubes distribution is working to isolate the graphical interface from the rest of the system. "The upcoming 4.1 release changes this protocol to a more flexible form. It will no longer use direct memory addresses, but an abstract mechanism in which the qube has to explicitly allow access to a particular memory page. In our current implementation — under Xen — we use the grant tables mechanism, which provides a separate memory allocation API and allows working on grants and not directly on memory pages. Other implementations will also be possible: whether for another hypervisor (e.g. KVM) or for a completely different architecture not based on shared memory (e.g. directly sending frames to another machine)."