It's probably been done.

I started programming in junior high, writing BASIC code on a Commodore 128. At the time the only learning resource I had was the owner's manual and some issues of Compute!'s Gazette, so I was almost completely self-taught. I don't think I had actually heard the words "computer science".

If you've never programmed in BASIC of that era, understand that:

1. There were no "functions"; a program consisted of a list of steps, executed linearly, with some control structures sprinkled in and plenty of GOTO jumps
2. Variables were all global
3. Variable names were limited to two letters

There was a limited bit of "subroutine" support: GOSUB was like a GOTO that remembered where it came from, and a RETURN would start executing from the next line after the GOSUB.  But since all variables were global, subroutines couldn't be reentrant.

So I got the clever idea that instead of using variables in a GOSUB routine, you could use arrays, and then have a variable that kept track of how many times you'd entered the subroutine. Sure, you had to define the maximum number of times you expected to GOSUB before you started returning, but at least you didn't have to remember whether or not the variable you were about to mutate was going to munge a previous call.

It was brilliant.

And when I got to college I learned that I'd "invented" the call stack and recursive functions. Rather a let-down, that.

So, grab your 8502 and load up a JMP 2011, when I wrote a blog post on "Stages of Competency":

After doing "this programmer thing" for a few years now, I've noticed a pattern in how I acquire skills and techniques. It's surprisingly consistent, and consists of these stages:

I thought it was pretty remarkable how consistently I saw the progression from "awareness" to "familiarity" to "functional understanding" to "understanding" to "competence" in myself and in others.

Pretty insightful, huh?

Turns out that pattern is a simplified subset of an educational classification system called Bloom's Taxonomy.

And it was first proposed in 1956.

So, yeah, that insight you thought you just had? It's probably been done.

 

3 Simple Rules That Will Make You a REAL Superstar Developer

In my experience there are two kinds of "rock star" software developers.  There's the Neil Peart rock star developer, who combines a natural blessing of talent and intelligence with a relentless work ethic and humble attitude, and over time becomes the developer that people not only want to hire, but want to be.  And then there's the "prima donna" rock star developer, who combines a modicum of raw talent with sheer attitude and self-promotion into the programming equivalent of a hotel-room-trashing, sex-and-drugs-and-rock-and-roll tabloid icon (I won't name an equivalent musician--use your favorite example).

Last year a tweet by Zed Shaw pointed me to a brilliant piece of satire called 3 Simple Rules That Will Make You a 'Superstar' Developer that gave three simple rules and two deeper principles for becoming a hard-living, Type 2 rock star programmer.  It's remarkably concise and accurate.  But what struck me is how close the three rules and two principles are to rules and principles for becoming a real Professor on the Programming Drums.

"Prima Donna" Rule 1: Write lots of code.

Have to fix a small bug in an area someone else has written? Don't waste time trying to understand it or their motivations for writing it that way. Just rewrite the lot as you think it ought to work. Call it refactoring if anyone asks.

"Neil Peart" Rule 1: Read lots of code.

You will spend more of your career reading code than writing code. Learn how to do it well. That means doing it a lot.  Read code even when you don't absolutely have to, and understand it deeply even when you think a shallow once-over will tell you all you need to know. Have a spare half hour?  Read the last couple checkins from other people on the team, even if you don't need to. You will learn more about the system faster, you might find issues earlier, and you will probably learn something or see a technique you didn't know about.  Have a spare afternoon?  Find an open source project and start reading code. Copiously reading both awful code and good code will help hone your internal sense of the difference.

Prima Donna Rule 2: Write your code quickly.  

Touch lots of files, and include every one of them in the ChangeLog. Don't worry about accidentally introducing hard-to-find bugs; they'll actually help you later on, as long as they're actually hard to find. Avoid introducing trivial bugs.

Neil Peart Rule 2: Finish your code quickly.  

"Done" is a Boolean state, and work isn't done until you would be surprised to have to revisit it again in a few weeks.  Minimize your personal work-in-progress.

Don't let 90%-finished tasks rot outside source control in a local directory, because you will forget the details. Don't check something in, thinking "this will do for now; we'll get to hardening it later", because you *will* forget the details.  And it's not finished until it's tested to your team's standards, documented to your team's standards, and understood well enough that if you get hit by a bus on your way home tonight, someone else can take your place.

Prima Donna Rule 3: Don't take time to document your code.

And don't add little comments explaining potential pitfalls in modifying some of the less clear statements you've introduced. You don't need them--you wrote the code.

Neil Peart Rule 3: Document your code with a single-minded purpose.

Obvious code with only as much documentation as is needed is the Holy Grail. Undocumented and unclear code is as bad, but over-documented code can be worse because it becomes a crutch ("so what if the code is ugly: that's why I commented it!").

You already know that when you write code, you put your reputation on the line that it is correct.  But when you document code, you put your reputation on the line that not only is it correct and sufficient now, but it
will be correct and sufficient when someone looks at it down the road. So minimize your risk of shame. Boilerplate comments, rambling exposition, comments that duplicate the code, and commented-out code
left in "just in case" are signs of laziness hiding behind the mantra of "comments are good; more must be better!"

Behold, the Underlying Principles

Strikingly, each set of rules emerge from one technical principle and one social principle, and each set of principles is a mirror image of the other:

The Prima Donna Technical Principle: You're 10x as productive when you're working on code you wrote as on code you didn't write.  

So, maximize your opportunity to work in code that you wrote, no matter the consequences.

The Neil Peart Technical Principle: You're 10x as productive when you have full awareness and mastery of your environment.  

Yes, you can achieve that by always working on your own code. That will mean being so prolific that all the problems you're fixing are your own creations.

Or you can achieve it by having a deep and total understanding of as much of your team's project and tool set as possible.  And the cleaner a design or a development process is, the more of it you can fit in your head at once.


The Prima Donna Social Principle: You win The Game by improving your reputation to superstar guru levels.

Your programming ability is judged by how much code you write, how quickly you finish features and fix critical bugs and how often your insights are necessary to solve problems.

The Neil Peart Social Principle: Optimize your life for value, not perceived ability. 

Your value to your project and your team is only partially related to your programming ability (perceived or real).  It's directly proportional to your ability to add value to your project and your team.  The more deeply you understand your project, your team, your code, and your tools, the more value you can add.  Conversely, any technical debt you create will be repaid either by you or those who follow you, and servicing that debt reduces the mental "capital" you have available toadd value.

In many ways, the "prima donna" and "Neil Peart" principles differ only subtly.  Maybe that's why it's so easy to find yourself on one path, when you really think you're on the other.

Case Study: Python as Secret Weapon for C++ Windows Programming

One of my favorite features of Python is its interactive shell. If you want to try something, you type in the code and try it immediately. For someone whose first coding environment was the equally-immediate Applesoft Basic, this is just as natural. But if your introduction to programming was C, C++, or Java, the benefits might not be apparent, especially if you're trying to do exploratory coding in one of those languages.

So I'm going to walk through a recent experience as a case study.

The Problem

At work we develop a Windows program that talks to certain devices via serial cables. Those devices also come in wireless Bluetooth flavors, and we connect to them via a "virtual serial port". To the program running, it looks as if the Bluetooth device is plugged into a real serial port, because all of the wireless connectivity is abstracted away by Windows. These devices are unidirectional--they transmit data to the Windows program, which passively reads it.

If you power off and restart one of these wired devices, it will start chattering away at the Windows program with hardly a hiccup--our program never even sees a disconnect. However, we noticed that this didn't happen with the Bluetooth devices: powering down one of those requires the Windows app to reconnect. But the Windows app didn't even seem to get any notification that the device disconnected. So how do you solve this chicken and egg problem?

Research

It had been a while since I'd done any actual hardware serial programming, so I started with some documentation, and remembered that the RS-232 serial spec included a line called DCD, or Data Carrier Detect (also called RLSD, for Receive Line Signal Detect). Back in the dinosaur days, this signal meant that your modem was connected to the remote modem, and was able to start communicating back and forth.

Sure enough, a search brought up the right bit of Win32 API documentation, which told me how to detect an RLSD change on a physical serial port using the SetCommMask and WaitCommEvent calls. The question now became "does the Microsoft virtual serial port for Bluetooth support RLSD"?

Exploration

At this point I could have started up Visual Studio, created a scratch project, written a couple dozen lines of C++ code, compiled and linked, fixed the compile errors, compiled and linked again, run the program, fixed the inevitable errors that the compiler didn't catch, and then had my answer.

But I'm too impatient to wait for Visual Studio to start up, too lazy to write C++ when I don't have to, and I have the hubris to think I can come up with something better than the obvious solution. Programmers are funny like that.

So instead, I cranked up DreamPie.

Secret Weapon #1: DreamPie

DreamPie is, very simply, my favorite cross-platform interactive Python interpreter. It began life as a fork of Python's built-in IDLE command shell, and from there it's never looked back. It has excellent interactive completion for packages (so you can type "from sys import s" and get a list of "stdin, stdout, stderr").

Even better, it does completion when you're typing file paths in arbitrary strings. I use this a lot to get to modules I'm trying to test: "import os,sys; sys.path.append('c:/src/'" gives me a list of all the directories in in "c:/src".

It also has a slick separation of (typed) input and (generated) output, and a neat "copy only code" feature that makes it perfect for "try this code interactively, and when it works the way I want it, yank it into the actual source file" exploration.

DreamPie works pretty much the same on both Linux and Windows systems. It's reputed to work well on Mac systems, too, but I don't use them for day-to-day development.

So where I'd normally crank up the Python command interpreter for interactive exploratory coding, I usually reach for DreamPie instead.

But what I needed to explore now was the Windows API as called from C++, not Python.

Secret Weapon #2: ctypes

ctypes is a "foreign function interface" (FFI) that's been part of Python since version 2.5. An FFI is just a way to call code that isn't written in your current programming language. In our case, the functions I wanted to call in order to test out serial port notification are in the kernel32.dll library, which is part of Windows. ctypes makes this really easy. Well, easy if you happen to have the Windows API documentation and all of the correct C header files handy, and if you know exactly what you're looking for:

>>> import ctypes

... file_mode = 0x80000000 # GENERIC_READ from <winnt.h>

... open_existing = 3 # from <winbase.h>

... buffer = ctypes.create_string_buffer(100)

... bytes_read = ctypes.c_ulong(0)

... hfile = ctypes.windll.kernel32.CreateFileW(r'\\.\COM17', file_mode, 0, None, open_existing, 0, None)

... ctypes.windll.kernel32.ReadFile(hfile, buffer, 100, ctypes.byref(bytes_read), None)

... buffer.value

0: b'\r\n052100746029\r\n'

>>>

Hooray. We can call the Win32 API functions to open the serial port and read from it, just like we would from C code.

But... that's an awful lot of crap to remember and type. I had to know exactly the C code I wanted to write. I had to know the Windows API well enough to find the constants and the functions to call. I had to know the ctypes API well enough to wire up Python to the C return values via ctypes buffers.

What a chore. Did I mention I'm lazy?

ctypes is the universal adapter--it can connect Python code to anything. But if you're specifically looking to call the Windows API, there's an even better tool:

Secret Weapon #3: PyWin32

PyWin32 predates ctypes, but it has a similar goal: gluing Python to something else. In this case, something else is specifically the entire Win32 API. PyWin32 consists of about two dozen modules, for example, "win32print" for printing, or "win32gui" for window handling, which wrap a good portion of the Win32 API.

The documentation is rather Spartan, but if you know the Win32 API side, you can map those calls to the PyWin32 modules without too much pain. The 8-line, hard-to-remember ctypes example above turns into just four lines of simpler code using PyWin32:

>>> import win32file # for CreateFile

... import win32con # for constants

... hfile = win32file.CreateFileW(r'\\.\COM17',

... win32con.GENERIC_READ,

... 0,

... None,

... win32con.OPEN_EXISTING,

... 0,

... None)

... win32file.ReadFile(hfile, 50, None)

0: (0, b'\r\n052100746029\r\n')

The Final Secret Weapon

My actual exploratory DreamPie session to see if Window's virtual Bluetooth serial port supported RLSD looked like this:

>>> import win32api, win32file, win32con

>>> hfile = win32file.CreateFileW(r'\\.\COM17', win32con.GENERIC_READ | win32con.GENERIC_WRITE, 0, None, win32con.OPEN_EXISTING, 0, None)

>>> win32file.GetCommMask(hfile)

0: 0

>>> win32file.SetCommMask(hfile, win32con.EV_RLSD)

Traceback (most recent call last):

File "", line 1, in

win32file.SetCommMask(hfile, win32con.EV_RLSD)

AttributeError: 'module' object has no attribute 'EV_RLSD'

>>> win32file.SetCommMask(hfile, win32file.EV_RLSD)

>>> win32file.GetCommMask(hfile)

1: 32

>>> win32file.EV_RLSD

2: 32

>>> win32file.WaitCommEvent(hfile)

3: (0, 32)

>>>

This is an actual copy of the DreamPie buffer from my test session, mistakes and all. This is what really happened when I tried to figure out if RLSD would work:

1. I typed up the code to open the serial port, which I knew should succeed, and it did.
2. I looked up the Win32 API call to get the "event mask", or the set of events that were being watched on the serial port handle, and saw that it was "GetCommMask". I blindly typed "win32file.GetCo", and lo and behold, DreamPie brought up a list of completions, which assured me that GetCommMask was there.
3. The Win32 API said that GetCommMask returned its result in a buffer passed into the call. Knowing that PyWin32 usually does a pretty good job of hiding return buffers, I decided to just try calling it with the input parameter, and got back zero. That made sense, if the serial port wasn't being monitored for events.
4. So I decided to push my luck: if GetCommMask worked, SetCommMask should work, too. A quick peek at the documentation, and... hrm. win32con didn't contain the "EV_RLSD" constant I was looking for to monitor the RLSD signal.
5. Well, I could have just typed the exact value (0x020) from the Windows docs... or I could just retype the line and use PyWin's autocompletion to see if win32file has the constant. I typed "win32file.EV_", and I had my answer. Then a quick re-test of GetCommMask() showed that the value was set.
6. The API docs claimed that WaitCommEvent should wait for one of the masked events to occur, and then return which one occurred. But the documentation showed that it took another of those return buffers. Thinking that PyWin32 might help me here, too: I typed "win32file.WaitCommEvent(hfile)", and the call appeared to block.
7. So I powered down the device, and within a few seconds, I was rewarded with the return value from WaitCommEvent: (0, 32). Aha. This meant that the Windows API version of WaitCommEvent returned 0 (for success), and that the return buffer contained 32, or EV_RLSD.

I included all the steps, including the mistakes, to show the last secret weapon: flexibility. Be willing to bounce back and forth between the documentation, the code you think should work, and the feedback you get both from the code under test and the tools you're using--and be willing to change your mental model based on that feedback.

In reality, this whole test took under five minutes from "Hmm... I wonder if I can use the DCD signal" to "Aha, looks like it works! Time to test it in C++." To be honest, I didn't even type out the whole ctypes version while testing--I started on it, realized that I'd have to look up and type all the constants by hand, then restarted DreamPie to jump over to PyWin32. Remembering that you can switch tools on the fly keeps you from getting stuck in ratholes that aren't directly related to the task at hand.

Flexibility is the key to fast and efficient exploratory coding. Using an interactive language like Python with a good set of support tools and libraries can be a secret weapon for speeding up exploratory coding--even when your target language is C++.

Stages of Competency

After doing "this programmer thing" for a few years now, I've noticed a pattern in how I acquire skills and techniques. It's surprisingly consistent, and consists of these stages:

0: Awareness

I've heard of the technique and can regurgitate a definition and a couple of use cases. I can probably pass a really bad phone screen (and in my experience, most of them are).

1: Familiarity

It's intrigued me enough that I've read up on it. I've probably looked at some code that uses it, and I can pick it out of a crowd, but I still mentally skip over it when reading its code (a bad habit that makes it harder to get past this phase).

At this point, if I were asked "what is X" or "how does X work" in an interview, I can probably pass the question, as long as there isn't a followup involving coding or something like "what are the pitfalls of using X over the long term", which is why I don't use questions like that in interviews anymore!

2: Functional Understanding

At this point I've either had to work with someone else's code that uses it, or else I've gone through an article that shows how to use it. I don't mentally skip over it anymore, and I can debug and modify it with some difficulty. Importantly, I can tell someone else what it's doing, but I will probably get embarrassed if I try to get into the details or (worse) debug it with them.

But I can't usefully synthesize anything with it. I get to a point in code and think, "Ah, this is a good place to use X!". Two hours (or more) later, I have bruises on my forehead from bashing it into the desk, I'm thinking "THIS CAN NOT BE THAT HARD", and I start wondering why I don't stay with the subset of techniques I know like the back of my hand. That's really tempting.

I get stuck in stage 2 a lot. I was there with C++ template metaprogramming for about five years, and I'm still there right now with Python metaclasses.

3: Understanding

After several frustrating episodes in stage 2, I do exactly the same thing in another context and... it makes sense. It works. I don't believe it, so I tweak things that should make it break, and it breaks in predictable ways. And I can reverse the tweaks and have it work again, predictably.

At this point, I always have the same three internal questions: a) do I really understand this? b) how did I not really understand this before? and c) what am I missing? I get uncomfortable not knowing how I know something.

Then all is well until I try to teach it to someone else, and we end up in another multi-hour WTF session.

What I've really learned at this stage is a single "groove" that works. As long as I don't deviate too much from the way I've used the technique, everything is fine. I think that subconsciously I know the limitations of that "groove", so I don't tend to make the little changes that expose the rough corners of my understanding. When I'm working with someone else, they have different edges to their own understanding. That's when I get this "uh-oh" feeling that tells me I really don't know what's going to happen when we do this.

Absent working with other people, I still think I understand it, which is a dangerous bit of self-delusion, and the biggest reason I'd rather work with a team than solo.

4: Competence

I don't know how I get here either, except maybe via repetitions of stage 3. In fact I don't usually notice even getting to this stage. The sign is usually that I'm having to do something outside the "groove" of my usual use of a technique, and that little "uh-oh" goes off, and then... it still works. Or else someone asks me about what would happen in a nasty corner case, and what comes out of my mouth is a better explanation of the details than I thought I could come up with.

This is also the point at which I finally feel comfortable writing about the technique, showing someone else how to use it, or trying to extend or modify it. The irony of it is that unless I do those things earlier, when I don't feel competent to do so, I tend not to get to this stage.

---

The funniest thing about this model is that if I look at code I've written in the past, I can usually pick out where I was on the scale when I wrote it. Again, I can't say exactly what the "tells" are, but when I get to stage 4 on something and look back at earlier code, I can think "ahh, ok, I was stuck in stage 2 at the time, and the places this code will break are probably X, Y, and Z."... and they usually are.

Forget owner's manuals--I wish brains came with source code. This progression would make a lot more sense then.

Switchpy

One of the consequences of the 2.x-to-3.x Python changeover is that I need to keep both versions around for a while on my Windows dev workstation.

Actually, strike that: I need to keep many versions around:

• 2.5.4, because that's the earliest version we support at work for some internal tools
• 2.6.6, because one particular internal tool jumped the gun and started using the "with" statement before we migrated to...
• 2.7, because that's what we're migrating those internal tools to (slowly)
• 3.1.2, because that's what we're targeting for new development
• A "special" 3.1.2, which mimics the version we've modified for use in our embedded devices
• The most recent 3.2 alpha, for testing
• A 3.2 trunk install, for testing patches

Virtualenv doesn't exactly do what I want: you have to install it from within an already-installed version of Python, and it doesn't support Python 3 yet (although there is a fork that does). Plus it doesn't handle anything other than environment variables--it doesn't understand Windows' defaults.

Ned Batchelder wrote a neat script that does some of that, but again, it doesn't handle everything.

So starting from Ned's script, I came up with switchpy:

• Supports Windows Python versions from 2.5 up to 3.2
• Changes the local PATH environment in the current shell (via the same batchfile trick as mpath)
• Updates the Registry-based associations (via code from Ned's script)
• Pings Explorer so that if you run "python.exe" from the Start | Run command, it notices the update
• Automatically reads installed official versions from the Registry, so you can say "switchpy 31" instead of "switchpy c:\python31"

So now, testing scripts in multiple versions of Python is as easy as:

C:\src\myscript>switchpy 25
Switching to Python at C:\Python25\...
Python is now C:\Python25\

C:\src\myscript>py.test
============================= test session starts =============================
python: platform win32 -- Python 2.5.4 -- pytest-1.3.0
test object 1: C:\src\myscript

myscript\tests\test_script.py ...

========================== 3 passed in 0.03 seconds ===========================

C:\src\myscript>switchpy 31
Switching to Python at C:\Python31\...
Python is now C:\Python31\

C:\src\myscript>py.test
============================= test session starts =============================
platform win32 -- Python 3.1.2 -- pytest-1.3.1
test object 1: C:\src\myscript

myscript\tests\test_script.py ...

========================== 3 passed in 0.03 seconds ===========================

For now, you can find switchpy in the same bitbucket repo as mpath; if I add any more scripts, I'll probably end up making it a more general repo.

Language of the Year

Several some many years ago, I decided to learn a new programming language a year. I don't subscribe to the "once you're a programmer, you can learn any new language in a few weeks" theory, so I didn't intend to get to guru status on any of them in 52 weekends and a few nights. Instead, the idea was to get good enough to be able to read code well, synthesize code at a decent level of competency, and understand the ideas behind the language, then move on.

If memory serves me correctly, I started with Java, then moved through REBOL, Python, Ruby, Scheme, and Lua before I dropped the practice. I tried getting back into it with Haskell a few years ago, but didn't take it to the "ok, I get it now" level of proficiency--mostly because of time constraints of changing jobs rather than any issue with the language.

I did learn something from each of these, which is the real point of the exercise. Now I'm thinking about starting a (belated) Language of the Year for 2010. The contenders are:

Erlang

Pro: interesting concurrency (independent processes) and data (immutable) models.

Con: might be too big to get my head around in nights and weekends

Io

Pro: Small surface area; looks like "Lua with a prototype-based object model" so far; might be useful practically.

Con: Not sure there's enough new there after doing Lua and REBOL.

Clojure

Pro: Interesting transactional memory model. Also, parentheses.

Con: I'm wondering how much time I'll spend re-learning the Java environment vs. learning Clojure.

Go

Pro: Interesting concurrency model; might be useful practically.

Con: Not sure there's enough different there, compared to Clojure or Erlang.

Part of the problem is picking a language that different enough to make it worth learning, but not so different that I can't use it for small, practical tools that I can actually use day-to-day. The Haskell and Scheme experiences showed me that if I can't use what I'm practicing day-to-day, I find it hard to keep devoting time to the project.

At the moment, it's looking like Clojure > Go > Erlang > Io. I'll probably pick up a distribution for each, get as far as "hello world", and then decide.

Mpath: command-line path manipulation for Windows

I'm a command line geek. Windows' style of installing everything in its own directory makes it easier to clean up after uninstallation, but it makes for very long PATH environment variables. If I put every directory containing command line tools in the system path, it gets too long for Windows to handle. So I usually end up doing "PATH=%PATH%;c:\somethingelse\bin" just before I use it. That also makes for long paths over long runtimes, especially when you use it in batch files (since you end up with PATH=c:\somethingelse\bin;c:\somethingelse\bin;c:\somethingelse\bin;[rest of path] after multiple invocations).

So I wrote mpath. Mpath is a combination batch file and Python script that takes advantage of some quirks of the Windows command shell, to let a child process alter the environment of a parent command shell process (something that you typically can't do in win32, but mpath gets around it by creating a temporary batch file that gets executed in the parent process).

Syntax:

• mpath pathname : prepends pathname to the current command shell's PATH, if it doesn't already exist.
• mpath + pathname : appends pathname to the current command shell's PATH, if it doesn't already exist.
• mpath - pathname : removes pathname from the current command shell's PATH, if it exists.

A quick demo:

C:\> PATH=C:

C:\> PATH
PATH=C:

C:\> mpath c:\foo (prepend c:\foo to the path)

C:\> PATH
PATH=c:\foo;C:

C:\> mpath - C:\FOO (take it off the path--note case insensitivity)

C:\> PATH
PATH=C:

C:\> mpath + c:\foo (append c:\foo to the path)

C:\> PATH
PATH=C:\;c:\foo

C:\> mpath c:\foo (try to prepend it again--mpath knows it's already there)
c:\foo already in path.

C:\> PATH=%PATH%;c:\foo (silly user should have used mpath...)

C:\> PATH
PATH=C:\;c:\foo;c:\foo (now there are two copies!)

C:\> mpath - C:\FOO (but mpath takes care of that.)

C:\> PATH
PATH=C:

I've tested Mpath with Windows XP running Python 2.5 and 2.6. I know it doesn't work on 3.x; I plan on fixing that at some point when I need it.

Update: mpath is now tested on 2.5, 2.6, 2.7, and 3.1.

Five Pycon 2010 Talks I Need to See

Following the example of Catherine Devlin and Carl Trachte, I thought I'd put together a list of the five Pycon talks I need to see in 2010. But I couldn't--I struggled to get below a dozen. So here are the top five I need to see, plus the ones I'll probably kick myself for not seeing because they're undoubtedly going to be scheduled in the same slots as the top five:

1. Import this, that, and the other thing: custom importers (Brett Cannon)
This is an easy choice, because I'm about to be implementing one of these for work. Would have been be nicer if Pycon 2010 had been scheduled for September 2009, but I'll take what I can get.

2. Understanding the Python GIL (David Beazley)
Another easy choice. After reading lots of code and debugging thread issues in our embedded Python interpreter at work, I think have a decent grasp of the GIL implementation. Given David's mindbending generators tutorial last year and his GIL presentation from ChiPy, I expect this talk to be rich in things I will be disturbed to have learned.

3. Powerful Pythonic Patterns (Alex Martelli)
Alex's talk last year, Abstractions as Leverage, was curiously satisfying. He didn't present any facts I hadn't already heard or read, but his presentation made some new connections for me (in a "My God, it's full of stars!" way).

4. Threading is Not a Model (Joe Gregorio)
In the last few years, I've begun to see pervasive threading as a placebo more than a solution. To paraphrase JWZ, some people, when confronted with a problem, think, "I know, I'll spin up a new thread." Now they have two problems. In reality, they've usually created an unknown number of problems, bounded only at the lower end by the number two. I'm really interested in seeing what Joe brings to the discussion beyond the usual "threads, select(), or fork()" question.

5. Turtles All The Way Down: Demystifying Deferreds, Decorators, and Declarations (Glyf Lefkowitz)
I have a long history of utter contempt for the practice of using syntactic sugar to "re-define the language in order to provide a more concise, natural style" for a given purpose. Glyf says he "will try to convince you that all of this wonderful magic isn't all that weird". Sounds like a challenge. If you're not continually questioning your own biases, you're heading for a mental rut, so I'm going to try to attend this with an open mind (and probably leave with a thoroughly-bitten tongue).

These are the ones I will move heaven, earth, and lunch plans to see. The others I really want to attend are:

• How Are Large Applications Embedding Python? (Peter Shinners). Totally relevant for work, but probably more elementary than I'd want.
• What Every Developer Should Know About Database Scalability (Jonathan Ellis). Totally irrelevant for my current work, but I've had to work in this area in the past, so it's somewhat interesting, and I'm curious about what's changed lately.
• Optimizations and Micro-Optimizations in CPython (Larry Hastings). Pure geeky personal interest.
• New *and* Improved: Coming changes to unittest, the standard library test framework (Michael Foord). I'm not quite a test-driven development zealot, but I'm about as close as you can get without applying for membership.
• Python Metaprogramming (Nicolas Lara). More pure geeky goodness.
• Eventlet: Asynchronous I/O with a Synchronous Interface (Donovan Preston). I can't quite decide whether this is applicable to work or not, and there's only one way to find out.
• Seattle: A Python-based Platform for Easy Development and Deployment of Networked Systems and Applications (Ivan Beschastnikh). I was quite disappointed by last year's sandboxing talk (the description didn't really let on that it was all about PyPy), so I'm hoping I can pick up more from this one.
• Tests and Testability (Ned Batchelder). Probably more elementary-level than I'd like, but might have some good discussion.
• On the Subject of Source Code (Ian Bicking). Another blue-sky talk by Ian? Yes, please.
• Python's Dusty Corners (Jack Diederich). I have a feeling this will be like Doug Hellman's PyModule of The Week: 80% of it is "yeah, yeah, I knew that," and 20% is "oh, wow, how did I not know that?"

I wasn't terribly impressed by the tutorial list (other than the compiled Python one), so I'll probably pass on them, but the talks look even better than last year. See you in Atlanta!

Five Things I Hate About C++

A few years ago, the "five things I hate about my favorite programming language" went around. I think it originated with Brian D. Foy's post on Perl. I like his reasoning: if you can't think of five things you don't like about it, you probably don't know enough about it to advocate for it.

Peter Siebel's recent post about the opinions of the folks he interviewed for Coders at Work made me remember it again. While I often reach for Python as the top tool in my toolbox these days, I've been writing C++ for most of my career, so I thought I'd take a crack at C++ first.

So:

5. No consistent ABI

C++ doesn't define a standard application binary interface (a standard for how the binaries produced from source code are laid out or linked together). If you're writing code to link against a pre-built library, then unless you're using the same version of the same compiler, you can't guarantee that your code will work correctly. (Technically, C doesn't either, but for practical purposes, though, C is in much better shape, mostly because C++'s features provide far more opportunities for implementations to disagree.)

The practical result is often that C linkage is considered "safe" and C++ linkage is considered "unsafe", which means that C linkage is the lingua franca for object-level interoperability, and no one really pushes for compatible C++ linkage--which in turn means that it doesn't happen. (While my experience is mostly in Windows programming, the situation seems to be better in the g++ and GNU/Linux world--as is often the case.)

4. Sorta-kinda safety

The first benefit of C++ over C for me wasn't object-orientation. It was that C++ seemed to be much better at catching the kinds of low-level programmer errors I tended to make back then. Class member protection, type-checked function parameters, exceptions that (unlike return codes) can't accidentally be ignored, constructors and destructors that are guaranteed to be called at the right time, improved casting operations--what's not to like?

The problem is that most of the safety features aren't really safe, they're just a little safer, often due to the desire for source code compatibility with C or concerns about run-time performance. You can probably argue that C++ is safer than C, but I believe it's "just safer enough" that C++ programmers get complacent.

Plus the interaction language of features makes it much easier to commit horrible, higher-level design mistakes that are harder to see when reading the code, particularly with things like non-trivial constructors and destructors, misused (and overused!) inheritance, and non-obvious method overrides.

3. Textual macros

The LISP world has had the "hygienic vs unhygienic macros" argument for a long time. In a (grossly oversimplified) nutshell, hygienic macros allow you to define new, reusable bits of language without worrying about the context in which they'll be evaluated. This makes for safer macro definitions, but precludes some very useful techniques that unhygienic macros allow--for example, enabling the code in the expanded macro use and affect variables in the context in which it's expanded.

But the C/C++ macro implementation makes LISP's unhygienic macros look like an Intel cleanroom. That's because they're not even really part of the C language syntax: they're just a simple, dumb textual replacement done in a preprocessing step, before compilation even occurs.

This feature inherited from C is so error prone that C++ added features like "inline" and namespacing to try to approximate the most common use cases for C macros, so that we wouldn't have to deal with them. It still didn't want to touch preprocessor macros for fear of breaking backwards compatibility, though, so now we have the worst of both worlds: a dangerous feature implemented outside the language syntax, with some of its bits duplicated in the language syntax, and guidance that says "sorry about the mess--here's some partial replacements that don't quite cover the gamut, but that's all you get. Have a nice day."

2. Worst-of-both-worlds standardization

C was born as an in-house development language in an AT&T lab in the late 1960s, and was used in anger almost from day one (for reimplementing the UNIX operating system). By the time standardization started, the language feature set was fairly solid and well-proven, and implementators already had real-world knowledge of the features.

C++, on the other hand, didn't go through this process. While C was designed as a language for implementing operating systems (and applications), C++ was designed as a language for implementing language features. It wasn't used (as far as I know) as the backbone of a single, well-known system in the way C was, so the language was free to evolve more divergently and more slowly.

Worse, the development of the language seems to have been driven by the the design and evolution of the specification, rather than by things tried and lessons learned in implementation. In some cases, features were added to the language specification before they were even implemented, in the hopes that smart compiler vendors would figure something out.

As a result, we have features that don't work like you'd expect (like std::vector or auto_ptr<>), features that don't interact well (like templates and class inheritance), and even features that, well, just don't work (like export, which was in the standard speculatively for years before its first attempted implementation, and which as far as I know has never been fully and correctly implemented by anyone).

On the other hand, while C was standardized after it had mostly stabilized, the C++ standardization process started while the language was still very much in flux. As a result, the core language is full of weirdnesses that are explainable only when you know the political situation at the time.

For example, the construction "virtual void foo() = 0;" is a pretty weird way to spell "pure virtual". In The Design and Evolution of C++, Bjarne Stroustrup reveals that the "=0" construction is there because he wanted to get pure virtual functions into the language specification, but a committee meeting was coming up soon, and he didn't think he could convince enough people to get behind adding a new "pure" keyword.

This leads right into...

1. C++ tries to be all things for all people

I think this one is the root of most of C++'s problems. C++ is and has always been a "more-is-better" language. If you like C, we'll make sure you like C++ by bending over backwards to make C code still work (except when it doesn't) and by making efficiency our top, err, one of our top-ten priorities. If you like object oriented programming, we've added classes and inheritance. Oh, multiple inheritance? Yep, we heard that works well, so we'll add it in there too. Parametric polymorphism? Multiple dispatch? Currying? Oh, hrm, we seem to have painted ourselves into a corner... but we can bodge most of that in with templates and partial template specialization. Oh, and guess what? We just figured out that you can use templates to do metaprogramming, so you get that feature for free! Free is good, right?

This results in two, mostly-correct perceptions:

1) C++ is a big grab-bag of language features, some of which are razor-sharp and don't really hang together coherently, but work great so long as you're really, really careful.
2) C++ is more-or-less better than C, so long as you stay with a "sane subset" of its features.

But what is that sane subset? That depends entirely on who you talk to, and the subset that they choose usually reveals more about their own priorities and experience than anything about the language itself.

All that being said, I still choose C++ (or my own trusted subset of it, at least) over C because of the convenience of constructors and destructors, the expressiveness of templates, and the confidence I get from RAII. I still choose it over Java because I don't need to worry about a runtime VM, because I can access platform-specific APIs and native libraries at will, and no checked-exception silliness.

But that doesn't mean I don't sigh a little every time I burn multiple days chasing down an intermittent memory leak, or that I don't steal a surreptitious glance at younger, better-looking languages with less emotional baggage from time to time.

Buying or Building... Furniture

Most software developers are familiar with the "buy-or-build" question: is it more effective to find existing software and try to make it work in your situation, or to build it to your exact specification and take on the burden of maintaining it? But sometimes it comes up in other contexts.

Like office furniture.

My current project at work is winding down, and I'm rolling over to a new one. As part of the transition, I'm moving from my old, two-person office into the new team's bullpen environment.

It might seem like a poor trade, but this team chose to trade in their fairly nice offices because they valued the higher conversational bandwidth they got in a bullpen. Yes, it's a bit noisier, but most of the noise is project-related, and results in quicker and more complete information dispersal both among developers and between developers and SQA engineers (who also share the space).

The big win for me is that it reduces the barrier to pair-programming to the cost of mumbling, "Uh... can anyone take a look at this with me?" And we still have the offices for when we need to make a phone call or do an interview.

One of the stipulations on building out the bullpen was that we had to use existing furniture. Unfortunately, while our current furniture is nice (and somewhat pricey, from what I'm told), it's optimized for a one-person or two-person office. We each get a curvy desk, a table with attached bookshelf that fits the curvy desk as an extension, and a funky rolling file cabinet. But the curvaceousness of the furniture means that it only fits well in a few prescribed configurations--none of which match a bullpen where you want to pair-program!

So the current bullpen, built from curvy bits loosely jammed together, isn't big enough to hold more people. And naturally, the people who handle furniture and facilities wouldn't be terribly happy with us saying, "Oh, this expensive furniture is nice. Now would you mind finding some place in our already-filled building to store it, and buy us some additional expensive furniture just like it, but without curvy bits?"

So our manager/Scrum Master, being the pragmatist that he is, decided we should build our own. From scratch.

Actually, "scratch" in this case really means heavy, solid-core interior doors for tabletops, and prebuilt folding-table legs to hold them up. Assembly is trivial, the surfaces are generous, prefinished, and attractive, and the cost was just a fraction of what we'd have paid for non-curvy versions of our standard furniture (which keeps the facilities folks happy... or at least happier).

There are, of course, some drawbacks. Making single large pairing stations means that you have to choose a single table height. In our case, it was chosen for us by the height of the prefab table legs.

However, my current programming partner suffers from an unfortunate and tragic genetic defect that caused his growth to continue far beyond normal human levels (the medical term is, I believe, "freakishly tall"). I, on the other hand, boast a full 5'3" of height, which seems far more normal to me, all things being relative.

So our alternative solution was to just tear the bookshelves off two small tables (again, that storage problem!), and then use one for each person, moving the tables around when we need to. The works great if the tables have cool adjustable legs like ours do.

(Mine, of course, is the station on the right.)

How to be the most awesome Dad ever

To be the most awesome Dad ever, capable of carrying out feats of skill and mastery usually reserved for the likes of the Avatar, James Bond, or the Doctor, requires just a few common ingredients:

1. The locked, most secret diary of a pre-adolescent daughter (who has lost the key),
   
2. The knowledge that all such cheap locks are the same,
   
3. A set of cheap luggage locks with keys,
   
4. A frantic pre-adolescent daughter in possession of #1 but not #2 or #3, and
   
5. A flair for the dramatic, with which one discloses that one knows how to pick locks, but it's a secret handed down from master spy to master spy, therefore the work must be done behind a locked door (which neatly conceals the fact that you're rummaging around in your bedroom drawer to find #3).
   

Fun with itertools

Sometimes it's hard to shake old habits, especially when you've burned them into your brain as the "standard" way to do things. For example, I've been doing network programming with C and C++ for a very long time. One of the standard pieces of code I've written again and again is the "connect with backoff" pattern.

If a program needs a continuous network connection, and that connection is lost, it should try to reconnect. On the one hand, you want to reconnect as quickly as possible; on the other hand, you don't want to keep retrying (and failing) in a tight loop. So you use a "backoff" timer: after each attempt, you wait longer (up to a maximum limit).

As a C programmer, I would implement an algorithm that resembles this Python-like pseudocode:

# After the first failure wait half a second before retrying;
# double this each time up to eight seconds.
backoff_times = [.5, 1, 2, 4, 8]
cur_backoff = None

while 1:
   try:
       # Try to connect
       connect()
   except ConnectionError:
       # Failed; update the backoff counter
       if cur_backoff is None:
           cur_backoff = 0
       else:
           cur_backoff = min(len(backoff_times)-1, cur_backoff+1)
       # Wait to retry
       time.sleep(backoff_times[cur_backoff])
   else:
       # Success; reset the backoff timer
       cur_backoff = None

But in Python the code to manage the current backoff timer looks out of place.

In a high level language, when the ratio of "code that says what I want" to "code that tells the language how to do what I want" gets too low, you're doing it wrong. It means that you're spending too many mental cycles on the "how," and not enough on the "what".

In this case, Python gives me a better way to tell it just "what" I want it to do: use an iterator.

import itertools

def iter_pegged(seq):
   """Return an iterator that walks the sequence, and then 'pegs' on the last item."""
   return itertools.chain(seq, itertools.repeat(seq[-1]))

backoff_times = [.5, 1, 2, 4, 8]
cur_backoff = iter_pegged(backoff_times)

while 1:
   try:
       # Try to connect
       connect()
   except ConnectionError:
       # Wait to retry
       time.sleep(cur_backoff.next())
   else:
       # Success; reset the backoff timer
       cur_backoff = iter_pegged(backoff_times)

Other than the definition of iter_pegged, each line of code says what only what it wants to do, not how it wants to do it.

And that's what coding in a high level language is all about, no?

Git: what he said.

About three months ago, I started writing a blog post on why my (development) life has changed for the better since I started using git for version control. It's been stewing in my "Drafts" folder for lack of time and attention.

My main point was that git is optimized for messy reality, as opposed to an idealized view of software development that never really happened that way, but looks better when you draw it on a white board.

A few weeks ago, Ryan Tomayko said most of what I intended to say, in The Thing About Git.

So, "what he said". In particular:

• Git means never having to say, “you should have”

• git --rebase interactive is “a bit like git commit --amend hopped up on acid and holding a chainsaw - completely insane and quite dangerous but capable of exposing entirely new states of mind.”

That Looks About Right

tlesher@badwolf:~$ history | awk '{print $2}' | sort | uniq -c | sort -rn | head
    109 ls
    107 cd
     78 git
     26 sudo
     14 rm
     12 ssh
     11 source
     11 more
     10 wget
      9 ipython

(via Chris and Tom)

Oh, look! A startup!

This December, I took a deep breath and jumped. I left a great job at a great company, working on cool projects with fun people, and executed a perfect swan dive back into the tech startup blender.

Now why would a thirty-something refugee from the tech bubble with a wife, two young kids, and a mortgage give up a good salary and stable, interesting job for the stress, uncertainty, and heavier workload of a new startup?

For me, it's all about aligning outcomes with effort. In a very early stage startup, the link between "what is about to happen" and "what am I doing" is as close to direct as you can possibly get. The outcome for the company is directly tied to what you're doing and how well you're doing it: no excuses, and no wiggle room. Naturally, there are external risks (like being run over by a company so big they barely notice the bump), you see the results of your effort clearly.

Working for a BigCo, Inc. is very different. Expending effort in a large company is like pulling on a bungie cord attached to a rock. Pull moderately, and nothing happens. Pull a little more, and the the rock comes along, but it tends to wiggle around in directions you don't intend. Yank really hard, and the rock has an annoying tendency to fly up and smack you in the head.

Now replace the bungie cord with a stick attached to the weight. You pull a little, the weight moves a little. You pull a lot, the weight moves a lot. You push, and the weight moves the opposite direction. The outcome is directly tied to your effort.

To be fair, my last company wasn't quite to the "bungie cord" stage. But when a former colleague approached me about his idea for a startup, I was drawn back in again.

Of course, to (ahem) stretch the analogy, sticks do tend to break more easily, but hey: that's the risk you take.

Sad Mac

Now that, my friends, is a Sad Mac.

10 Things I Learned at MEDC

1. In person, Sue Loh sounds exactly like you'd imagine from reading the CE blog.
2. In person, Doug Boling sounds exactly like you'd imagine from reading his book (except his humor comes off better in person).
3. Windows Mobile 6 isn't as big a deal as it first seemed (unless you happen to be a managed code developer, which I'm not).
4. The Tao is worth seeing (if you can either get in on your own coolness, or else persuade a multi-billion-dollar company to slip the owner some cash).
5. You never know which of your mild-mannered colleagues will turn out to be a raving, iPod-dancing, Viva-Las-Vegas-singing Elvis fan.
6. Managed code (C#) actually runs on "real" embedded devices that are too small for even CE.
7. No matter how awesome your hotel looks, you still need to provide running water to your guests, or they get cranky.
8. When attending a conference, make sure you introduce yourself and describe your company to as many people as you can: you never know which one will suggest a very cool opportunity a few days later.
9. Windows Smartphones are now outselling Blackberry devices, and 90% of them are being sold to consumers rather than businesses.
10. Despite pushing most of the cool new features onto Windows Mobile first, Microsoft still isn't abandoning CE devices just yet. Thanks for that.
    

Ok, I give in.

Just call me a sheep.

O Sole Mio...

Ok, so I've missed PyCon for the past three years running because of work conflicts. But at least this year they've been nice enough to send me to the Microsoft Mobile and Embedded DevCon (MEDC) at The Venetian in Las Vegas.

No "one more thing" announcements from today's keynote, but we did get an earful of how Microsoft wants to position the Windows Mobile family: "it's not just for business anymore". Robbie Bach, the president of Microsoft's Entertainment & Devices Division, claims that Win Mobile-based smart phones are outselling Blackberry devices, and that 90% of those sales are actually to consumers (not businesses).

That sounds great, but the massive emphasis on "all things smartphone" makes me a bit nervous. Microsoft is a smart optimizer when it comes to business strategies: what happens when their CE OEMs (like us) become 2% of their developer base, and their Windows Mobile Pro/Standard OEMs are 98%?

Won't someone think of the poor headless CE devices? Just try using your fancy QVGA display in a freezer for a few hours...

Your Memex is here. Are you using it?

Microsoft Research has put significant effort into implementing a near-literal version of Vannevar Bush's "memex" in its MyLifeBits project. I think we have the memex already: we just don't realize it.

In 1945, the Atlantic Journal published "As We May Think", in which Vannevar Bush speculated that in the future, a machine--the "memex," or "memory extender"--would assist researchers by storing, indexing, and retrieving every piece of information they could possibly need. A user could also add his own text, images, or recordings, and could record notes and comments on the content. And it all fit within a large desk.

This was strong stuff for the time: understand that the state of the art was the Harvard Mark I: a 50-foot-long, 10,000-pound, four-function calculator that could divide at the blinding speed of four operations per minute. To put it in perspective, Bush's prediction was made when my grandparents were not yet old enough to drive a car.

Since 2002, Microsoft Research has been working on implementing MyLifeBits, their version of the memex. And after five years of effort, they now have a one-user prototype to show for their efforts. So don't expect to be shelling out for the Microsoft Memex anytime soon.

But a few weeks ago, I had a realization. I went back to the original, 60-year-old article, and read over the description of the memex again:

A memex is a device in which an individual stores all his books, records, and communications, and which is mechanized so that it may be consulted with exceeding speed and flexibility....

It consists of a desk, and while it can presumably be operated from a distance, it is primarily the piece of furniture at which he works....

In one end is the stored material. The matter of bulk is well taken care of by improved microfilm. ...

Most of the memex contents are purchased on microfilm ready for insertion. Books of all sorts, pictures, current periodicals, newspapers, are thus obtained and dropped into place....

All this is conventional, except for the projection forward of present-day mechanisms and gadgetry. It affords an immediate step, however, to associative indexing, the basic idea of which is a provision whereby any item may be caused at will to select immediately and automatically another....It is exactly as though the physical items had been gathered together from widely separated sources and bound together to form a new book. It is more than this, for any item can be joined into numerous trails.... And his trails do not fade.

There are more parallels, but that's a good start.

So on the one hand, we have a research project to create a literal implementation of the memex that might exist sometime in the future, or a distributed, chaotic, mashup of individual technologies that together get about 90% of the way there today.

Any bets on which will get there first?

More importantly, what are you waiting for that isn't there yet?