This week the Windows version of the Replay browser is entering beta. Until now, we’ve only supported recording on two operating systems in the Unix family: macOS and Linux. Windows and Unix have little in common, but we can still record and replay what happens in a Windows application by following the same methodology used for Unix operating systems. In this post we describe the challenges in handling these applications.
Linking Windows Applications
As described in Recording and Replaying, Replay’s recorder acts a lot like the operating system’s dynamic linker in order to change references to library calls so that they call shims which record their behavior. In order to do this, the recorder needs to understand the format of the binary files which have been loaded into memory. Windows binaries use the PE executable format, which is completely different from the Mach-O format used on macOS and the ELF format used on Linux. These formats are all fundamentally quite similar, however, and basically describe how to map the executable file into memory, which symbols it exports and where references to imported symbols can be found. The fine details are pretty intricate, but thanks to the extensive documentation on MSDN and elsewhere, finding the symbols in a PE binary isn’t too difficult.
Windows System Library API
Once we’re able to find the symbols referenced by a binary, we need to replace references to system library functions with recorder shims that record the inputs and outputs to those functions when they are called. We need to understand the behavior of these library functions, which have almost no overlap with the functions in macOS or Linux system libraries. For example, while a Unix program will call
open to create or open a file, a Windows program will call CreateFile. These analogous functions can be handled in analogous ways, and while Gecko (the browser engine which Replay uses, along with Firefox) can call several hundred different Windows system library functions, adding support for these is largely a straightforward process.Windows ABI
The library API describes how to call into Windows system libraries by writing some C/C++ code, but in order to record the behavior of these calls Replay’s recorder needs to understand the binary interface used by these library functions. Also known as calling conventions, this interface describes which arguments are held in which registers, how any extra arguments are laid out on the stack, how values are returned, and what registers can be clobbered by the function. macOS and Linux both use the System V ABI, which is nice for simplicity, but Windows uses its own ABI. The recorder already generates snippets of machine code to capture information about calls made using the System V ABI, and this can be extended to the Windows ABI as well, though the presence of multiple ABIs adds a fair amount of complexity in places where we need to inter-convert between the ABIs in order to replay on Linux.
Replaying on Linux
The main reason why supporting Windows hasn’t required a ton of work is that we only had to change a few parts of the recorder and backend replaying components in order to handle the above differences vs. macOS and Linux. So few parts needed changing because we replay all recordings on Linux, regardless of whether they were originally made on macOS, Linux, or Windows. Using a single environment for replaying is hugely beneficial for simplicity and reliability, and is possible because of the commonalities between applications compiled for the different operating systems. Binaries for each of these are basically just big blobs of x64 machine code. Despite the differences in the binary format used to load them and the APIs/ABIs which they use, once they are loaded they will run just fine regardless of the host operating system, so long as the system library calls they make behave as expected.
Running software compiled for one operating system on a different one is not a new idea. The Wine project is a similar layer used to run Windows applications on Linux, and Darling is able to run macOS applications on Linux. These projects have been ongoing for many years, even decades; emulating the full functionality of the system library functions is a complex and very difficult task, e.g. emulating something as simple as the
CreateFile function above needs to model the API’s semantics and associated differences in filesystem behavior between Linux and Windows. In order to replay a call to CreateFile from a recording we only need to understand what the call’s inputs and outputs are. This is a simpler take on the problem which Wine and Darling are tackling; instead of translating between two operating systems, when replaying we isolate the application from the underlying operating system using data from the recording, so that we can ensure it will behave consistently and can analyze its behavior in detail.