Tuesday, September 06, 2011

Case Study: Python as Secret Weapon for C++ Windows Programming

One of my favorite features of Python is its interactive shell. If you want to try something, you type in the code and try it immediately. For someone whose first coding environment was the equally-immediate Applesoft Basic, this is just as natural. But if your introduction to programming was C, C++, or Java, the benefits might not be apparent, especially if you're trying to do exploratory coding in one of those languages.

So I'm going to walk through a recent experience as a case study.

The Problem

At work we develop a Windows program that talks to certain devices via serial cables. Those devices also come in wireless Bluetooth flavors, and we connect to them via a "virtual serial port". To the program running, it looks as if the Bluetooth device is plugged into a real serial port, because all of the wireless connectivity is abstracted away by Windows. These devices are unidirectional--they transmit data to the Windows program, which passively reads it.

If you power off and restart one of these wired devices, it will start chattering away at the Windows program with hardly a hiccup--our program never even sees a disconnect. However, we noticed that this didn't happen with the Bluetooth devices: powering down one of those requires the Windows app to reconnect. But the Windows app didn't even seem to get any notification that the device disconnected. So how do you solve this chicken and egg problem?


It had been a while since I'd done any actual hardware serial programming, so I started with some documentation, and remembered that the RS-232 serial spec included a line called DCD, or Data Carrier Detect (also called RLSD, for Receive Line Signal Detect). Back in the dinosaur days, this signal meant that your modem was connected to the remote modem, and was able to start communicating back and forth.

Sure enough, a search brought up the right bit of Win32 API documentation, which told me how to detect an RLSD change on a physical serial port using the SetCommMask and WaitCommEvent calls. The question now became "does the Microsoft virtual serial port for Bluetooth support RLSD"?


At this point I could have started up Visual Studio, created a scratch project, written a couple dozen lines of C++ code, compiled and linked, fixed the compile errors, compiled and linked again, run the program, fixed the inevitable errors that the compiler didn't catch, and then had my answer.

But I'm too impatient to wait for Visual Studio to start up, too lazy to write C++ when I don't have to, and I have the hubris to think I can come up with something better than the obvious solution. Programmers are funny like that.

So instead, I cranked up DreamPie.

Secret Weapon #1: DreamPie

DreamPie is, very simply, my favorite cross-platform interactive Python interpreter. It began life as a fork of Python's built-in IDLE command shell, and from there it's never looked back. It has excellent interactive completion for packages (so you can type "from sys import s" and get a list of "stdin, stdout, stderr").

Even better, it does completion when you're typing file paths in arbitrary strings. I use this a lot to get to modules I'm trying to test: "import os,sys; sys.path.append('c:/src/'" gives me a list of all the directories in in "c:/src".

It also has a slick separation of (typed) input and (generated) output, and a neat "copy only code" feature that makes it perfect for "try this code interactively, and when it works the way I want it, yank it into the actual source file" exploration.

DreamPie works pretty much the same on both Linux and Windows systems. It's reputed to work well on Mac systems, too, but I don't use them for day-to-day development.

So where I'd normally crank up the Python command interpreter for interactive exploratory coding, I usually reach for DreamPie instead.

But what I needed to explore now was the Windows API as called from C++, not Python.

Secret Weapon #2: ctypes

ctypes is a "foreign function interface" (FFI) that's been part of Python since version 2.5. An FFI is just a way to call code that isn't written in your current programming language. In our case, the functions I wanted to call in order to test out serial port notification are in the kernel32.dll library, which is part of Windows. ctypes makes this really easy. Well, easy if you happen to have the Windows API documentation and all of the correct C header files handy, and if you know exactly what you're looking for:

>>> import ctypes
... file_mode = 0x80000000 # GENERIC_READ from <winnt.h>
... open_existing = 3 # from <winbase.h>
... buffer = ctypes.create_string_buffer(100)
... bytes_read = ctypes.c_ulong(0)
... hfile = ctypes.windll.kernel32.CreateFileW(r'\\.\COM17', file_mode, 0, None, open_existing, 0, None)
... ctypes.windll.kernel32.ReadFile(hfile, buffer, 100, ctypes.byref(bytes_read), None)
... buffer.value
0: b'\r\n052100746029\r\n'

Hooray. We can call the Win32 API functions to open the serial port and read from it, just like we would from C code.

But... that's an awful lot of crap to remember and type. I had to know exactly the C code I wanted to write. I had to know the Windows API well enough to find the constants and the functions to call. I had to know the ctypes API well enough to wire up Python to the C return values via ctypes buffers.

What a chore. Did I mention I'm lazy?

ctypes is the universal adapter--it can connect Python code to anything. But if you're specifically looking to call the Windows API, there's an even better tool:

Secret Weapon #3: PyWin32

PyWin32 predates ctypes, but it has a similar goal: gluing Python to something else. In this case, something else is specifically the entire Win32 API. PyWin32 consists of about two dozen modules, for example, "win32print" for printing, or "win32gui" for window handling, which wrap a good portion of the Win32 API.

The documentation is rather Spartan, but if you know the Win32 API side, you can map those calls to the PyWin32 modules without too much pain. The 8-line, hard-to-remember ctypes example above turns into just four lines of simpler code using PyWin32:

>>> import win32file # for CreateFile
... import win32con # for constants
... hfile = win32file.CreateFileW(r'\\.\COM17',
... win32con.GENERIC_READ,
... 0,
... None,
... win32con.OPEN_EXISTING,
... 0,
... None)
... win32file.ReadFile(hfile, 50, None)
0: (0, b'\r\n052100746029\r\n')

The Final Secret Weapon

My actual exploratory DreamPie session to see if Window's virtual Bluetooth serial port supported RLSD looked like this:

>>> import win32api, win32file, win32con
>>> hfile = win32file.CreateFileW(r'\\.\COM17', win32con.GENERIC_READ | win32con.GENERIC_WRITE, 0, None, win32con.OPEN_EXISTING, 0, None)
>>> win32file.GetCommMask(hfile)
0: 0
>>> win32file.SetCommMask(hfile, win32con.EV_RLSD)
Traceback (most recent call last):
File "", line 1, in
win32file.SetCommMask(hfile, win32con.EV_RLSD)
AttributeError: 'module' object has no attribute 'EV_RLSD'
>>> win32file.SetCommMask(hfile, win32file.EV_RLSD)
>>> win32file.GetCommMask(hfile)
1: 32
>>> win32file.EV_RLSD
2: 32
>>> win32file.WaitCommEvent(hfile)
3: (0, 32)

This is an actual copy of the DreamPie buffer from my test session, mistakes and all. This is what really happened when I tried to figure out if RLSD would work:
  1. I typed up the code to open the serial port, which I knew should succeed, and it did.
  2. I looked up the Win32 API call to get the "event mask", or the set of events that were being watched on the serial port handle, and saw that it was "GetCommMask". I blindly typed "win32file.GetCo", and lo and behold, DreamPie brought up a list of completions, which assured me that GetCommMask was there.
  3. The Win32 API said that GetCommMask returned its result in a buffer passed into the call. Knowing that PyWin32 usually does a pretty good job of hiding return buffers, I decided to just try calling it with the input parameter, and got back zero. That made sense, if the serial port wasn't being monitored for events.
  4. So I decided to push my luck: if GetCommMask worked, SetCommMask should work, too. A quick peek at the documentation, and... hrm. win32con didn't contain the "EV_RLSD" constant I was looking for to monitor the RLSD signal.
  5. Well, I could have just typed the exact value (0x020) from the Windows docs... or I could just retype the line and use PyWin's autocompletion to see if win32file has the constant. I typed "win32file.EV_", and I had my answer. Then a quick re-test of GetCommMask() showed that the value was set.
  6. The API docs claimed that WaitCommEvent should wait for one of the masked events to occur, and then return which one occurred. But the documentation showed that it took another of those return buffers. Thinking that PyWin32 might help me here, too: I typed "win32file.WaitCommEvent(hfile)", and the call appeared to block.
  7. So I powered down the device, and within a few seconds, I was rewarded with the return value from WaitCommEvent: (0, 32). Aha. This meant that the Windows API version of WaitCommEvent returned 0 (for success), and that the return buffer contained 32, or EV_RLSD.

I included all the steps, including the mistakes, to show the last secret weapon: flexibility. Be willing to bounce back and forth between the documentation, the code you think should work, and the feedback you get both from the code under test and the tools you're using--and be willing to change your mental model based on that feedback.

In reality, this whole test took under five minutes from "Hmm... I wonder if I can use the DCD signal" to "Aha, looks like it works! Time to test it in C++." To be honest, I didn't even type out the whole ctypes version while testing--I started on it, realized that I'd have to look up and type all the constants by hand, then restarted DreamPie to jump over to PyWin32. Remembering that you can switch tools on the fly keeps you from getting stuck in ratholes that aren't directly related to the task at hand.

Flexibility is the key to fast and efficient exploratory coding. Using an interactive language like Python with a good set of support tools and libraries can be a secret weapon for speeding up exploratory coding--even when your target language is C++.


Anonymous said...

how does dreampie compare to ipython?

Tim Lesher said...

Anonymous: I used iPython for some time a few years ago. I found that, while it did some things like completion very well, it was rather unstable, at least under Windows.

In particular, it had a nasty intermittent bug where exiting the interpreter didn't quite completely close the process, and it was indeterminate which of the exiting iPython shell or the cmd.exe shell would receive a given line of stdin!

Additionally, iPython has (or at least had) a ton of functionality I didn't really need. I wasn't looking for a replacement system shell, and it seemed at the time that most of the development effort on iPython was focused on supporting high-performance parallel computing and scientific computing rather than general purpose computing.

Things may be different now, of course--I haven't looked at iPython in some time.

Anonymous said...

I suggest that you take a look ay bpython for when you need a dreampie like equivalent in console mode (f.e. integrated into your favorite editor) but other then that, dreampie rocks.

paul c said...

bpython states its for unix-like systems, so I guess not so good for windows.