Stages of Competency

After doing "this programmer thing" for a few years now, I've noticed a pattern in how I acquire skills and techniques. It's surprisingly consistent, and consists of these stages:

0: Awareness

I've heard of the technique and can regurgitate a definition and a couple of use cases. I can probably pass a really bad phone screen (and in my experience, most of them are).

1: Familiarity

It's intrigued me enough that I've read up on it. I've probably looked at some code that uses it, and I can pick it out of a crowd, but I still mentally skip over it when reading its code (a bad habit that makes it harder to get past this phase).

At this point, if I were asked "what is X" or "how does X work" in an interview, I can probably pass the question, as long as there isn't a followup involving coding or something like "what are the pitfalls of using X over the long term", which is why I don't use questions like that in interviews anymore!

2: Functional Understanding

At this point I've either had to work with someone else's code that uses it, or else I've gone through an article that shows how to use it. I don't mentally skip over it anymore, and I can debug and modify it with some difficulty. Importantly, I can tell someone else what it's doing, but I will probably get embarrassed if I try to get into the details or (worse) debug it with them.

But I can't usefully synthesize anything with it. I get to a point in code and think, "Ah, this is a good place to use X!". Two hours (or more) later, I have bruises on my forehead from bashing it into the desk, I'm thinking "THIS CAN NOT BE THAT HARD", and I start wondering why I don't stay with the subset of techniques I know like the back of my hand. That's really tempting.

I get stuck in stage 2 a lot. I was there with C++ template metaprogramming for about five years, and I'm still there right now with Python metaclasses.

3: Understanding

After several frustrating episodes in stage 2, I do exactly the same thing in another context and... it makes sense. It works. I don't believe it, so I tweak things that should make it break, and it breaks in predictable ways. And I can reverse the tweaks and have it work again, predictably.

At this point, I always have the same three internal questions: a) do I really understand this? b) how did I not really understand this before? and c) what am I missing? I get uncomfortable not knowing how I know something.

Then all is well until I try to teach it to someone else, and we end up in another multi-hour WTF session.

What I've really learned at this stage is a single "groove" that works. As long as I don't deviate too much from the way I've used the technique, everything is fine. I think that subconsciously I know the limitations of that "groove", so I don't tend to make the little changes that expose the rough corners of my understanding. When I'm working with someone else, they have different edges to their own understanding. That's when I get this "uh-oh" feeling that tells me I really don't know what's going to happen when we do this.

Absent working with other people, I still think I understand it, which is a dangerous bit of self-delusion, and the biggest reason I'd rather work with a team than solo.

4: Competence

I don't know how I get here either, except maybe via repetitions of stage 3. In fact I don't usually notice even getting to this stage. The sign is usually that I'm having to do something outside the "groove" of my usual use of a technique, and that little "uh-oh" goes off, and then... it still works. Or else someone asks me about what would happen in a nasty corner case, and what comes out of my mouth is a better explanation of the details than I thought I could come up with.

This is also the point at which I finally feel comfortable writing about the technique, showing someone else how to use it, or trying to extend or modify it. The irony of it is that unless I do those things earlier, when I don't feel competent to do so, I tend not to get to this stage.

---

The funniest thing about this model is that if I look at code I've written in the past, I can usually pick out where I was on the scale when I wrote it. Again, I can't say exactly what the "tells" are, but when I get to stage 4 on something and look back at earlier code, I can think "ahh, ok, I was stuck in stage 2 at the time, and the places this code will break are probably X, Y, and Z."... and they usually are.

Forget owner's manuals--I wish brains came with source code. This progression would make a lot more sense then.

Switchpy

One of the consequences of the 2.x-to-3.x Python changeover is that I need to keep both versions around for a while on my Windows dev workstation.

Actually, strike that: I need to keep many versions around:

• 2.5.4, because that's the earliest version we support at work for some internal tools
• 2.6.6, because one particular internal tool jumped the gun and started using the "with" statement before we migrated to...
• 2.7, because that's what we're migrating those internal tools to (slowly)
• 3.1.2, because that's what we're targeting for new development
• A "special" 3.1.2, which mimics the version we've modified for use in our embedded devices
• The most recent 3.2 alpha, for testing
• A 3.2 trunk install, for testing patches

Virtualenv doesn't exactly do what I want: you have to install it from within an already-installed version of Python, and it doesn't support Python 3 yet (although there is a fork that does). Plus it doesn't handle anything other than environment variables--it doesn't understand Windows' defaults.

Ned Batchelder wrote a neat script that does some of that, but again, it doesn't handle everything.

So starting from Ned's script, I came up with switchpy:

• Supports Windows Python versions from 2.5 up to 3.2
• Changes the local PATH environment in the current shell (via the same batchfile trick as mpath)
• Updates the Registry-based associations (via code from Ned's script)
• Pings Explorer so that if you run "python.exe" from the Start | Run command, it notices the update
• Automatically reads installed official versions from the Registry, so you can say "switchpy 31" instead of "switchpy c:\python31"

So now, testing scripts in multiple versions of Python is as easy as:

C:\src\myscript>switchpy 25
Switching to Python at C:\Python25\...
Python is now C:\Python25\

C:\src\myscript>py.test
============================= test session starts =============================
python: platform win32 -- Python 2.5.4 -- pytest-1.3.0
test object 1: C:\src\myscript

myscript\tests\test_script.py ...

========================== 3 passed in 0.03 seconds ===========================

C:\src\myscript>switchpy 31
Switching to Python at C:\Python31\...
Python is now C:\Python31\

C:\src\myscript>py.test
============================= test session starts =============================
platform win32 -- Python 3.1.2 -- pytest-1.3.1
test object 1: C:\src\myscript

myscript\tests\test_script.py ...

========================== 3 passed in 0.03 seconds ===========================

For now, you can find switchpy in the same bitbucket repo as mpath; if I add any more scripts, I'll probably end up making it a more general repo.

Language of the Year

Several some many years ago, I decided to learn a new programming language a year. I don't subscribe to the "once you're a programmer, you can learn any new language in a few weeks" theory, so I didn't intend to get to guru status on any of them in 52 weekends and a few nights. Instead, the idea was to get good enough to be able to read code well, synthesize code at a decent level of competency, and understand the ideas behind the language, then move on.

If memory serves me correctly, I started with Java, then moved through REBOL, Python, Ruby, Scheme, and Lua before I dropped the practice. I tried getting back into it with Haskell a few years ago, but didn't take it to the "ok, I get it now" level of proficiency--mostly because of time constraints of changing jobs rather than any issue with the language.

I did learn something from each of these, which is the real point of the exercise. Now I'm thinking about starting a (belated) Language of the Year for 2010. The contenders are:

Erlang

Pro: interesting concurrency (independent processes) and data (immutable) models.

Con: might be too big to get my head around in nights and weekends

Io

Pro: Small surface area; looks like "Lua with a prototype-based object model" so far; might be useful practically.

Con: Not sure there's enough new there after doing Lua and REBOL.

Clojure

Pro: Interesting transactional memory model. Also, parentheses.

Con: I'm wondering how much time I'll spend re-learning the Java environment vs. learning Clojure.

Go

Pro: Interesting concurrency model; might be useful practically.

Con: Not sure there's enough different there, compared to Clojure or Erlang.

Part of the problem is picking a language that different enough to make it worth learning, but not so different that I can't use it for small, practical tools that I can actually use day-to-day. The Haskell and Scheme experiences showed me that if I can't use what I'm practicing day-to-day, I find it hard to keep devoting time to the project.

At the moment, it's looking like Clojure > Go > Erlang > Io. I'll probably pick up a distribution for each, get as far as "hello world", and then decide.

Mpath: command-line path manipulation for Windows

I'm a command line geek. Windows' style of installing everything in its own directory makes it easier to clean up after uninstallation, but it makes for very long PATH environment variables. If I put every directory containing command line tools in the system path, it gets too long for Windows to handle. So I usually end up doing "PATH=%PATH%;c:\somethingelse\bin" just before I use it. That also makes for long paths over long runtimes, especially when you use it in batch files (since you end up with PATH=c:\somethingelse\bin;c:\somethingelse\bin;c:\somethingelse\bin;[rest of path] after multiple invocations).

So I wrote mpath. Mpath is a combination batch file and Python script that takes advantage of some quirks of the Windows command shell, to let a child process alter the environment of a parent command shell process (something that you typically can't do in win32, but mpath gets around it by creating a temporary batch file that gets executed in the parent process).

Syntax:

• mpath pathname : prepends pathname to the current command shell's PATH, if it doesn't already exist.
• mpath + pathname : appends pathname to the current command shell's PATH, if it doesn't already exist.
• mpath - pathname : removes pathname from the current command shell's PATH, if it exists.

A quick demo:

C:\> PATH=C:

C:\> PATH
PATH=C:

C:\> mpath c:\foo (prepend c:\foo to the path)

C:\> PATH
PATH=c:\foo;C:

C:\> mpath - C:\FOO (take it off the path--note case insensitivity)

C:\> PATH
PATH=C:

C:\> mpath + c:\foo (append c:\foo to the path)

C:\> PATH
PATH=C:\;c:\foo

C:\> mpath c:\foo (try to prepend it again--mpath knows it's already there)
c:\foo already in path.

C:\> PATH=%PATH%;c:\foo (silly user should have used mpath...)

C:\> PATH
PATH=C:\;c:\foo;c:\foo (now there are two copies!)

C:\> mpath - C:\FOO (but mpath takes care of that.)

C:\> PATH
PATH=C:

I've tested Mpath with Windows XP running Python 2.5 and 2.6. I know it doesn't work on 3.x; I plan on fixing that at some point when I need it.

Update: mpath is now tested on 2.5, 2.6, 2.7, and 3.1.

Five Pycon 2010 Talks I Need to See

Following the example of Catherine Devlin and Carl Trachte, I thought I'd put together a list of the five Pycon talks I need to see in 2010. But I couldn't--I struggled to get below a dozen. So here are the top five I need to see, plus the ones I'll probably kick myself for not seeing because they're undoubtedly going to be scheduled in the same slots as the top five:

1. Import this, that, and the other thing: custom importers (Brett Cannon)
This is an easy choice, because I'm about to be implementing one of these for work. Would have been be nicer if Pycon 2010 had been scheduled for September 2009, but I'll take what I can get.

2. Understanding the Python GIL (David Beazley)
Another easy choice. After reading lots of code and debugging thread issues in our embedded Python interpreter at work, I think have a decent grasp of the GIL implementation. Given David's mindbending generators tutorial last year and his GIL presentation from ChiPy, I expect this talk to be rich in things I will be disturbed to have learned.

3. Powerful Pythonic Patterns (Alex Martelli)
Alex's talk last year, Abstractions as Leverage, was curiously satisfying. He didn't present any facts I hadn't already heard or read, but his presentation made some new connections for me (in a "My God, it's full of stars!" way).

4. Threading is Not a Model (Joe Gregorio)
In the last few years, I've begun to see pervasive threading as a placebo more than a solution. To paraphrase JWZ, some people, when confronted with a problem, think, "I know, I'll spin up a new thread." Now they have two problems. In reality, they've usually created an unknown number of problems, bounded only at the lower end by the number two. I'm really interested in seeing what Joe brings to the discussion beyond the usual "threads, select(), or fork()" question.

5. Turtles All The Way Down: Demystifying Deferreds, Decorators, and Declarations (Glyf Lefkowitz)
I have a long history of utter contempt for the practice of using syntactic sugar to "re-define the language in order to provide a more concise, natural style" for a given purpose. Glyf says he "will try to convince you that all of this wonderful magic isn't all that weird". Sounds like a challenge. If you're not continually questioning your own biases, you're heading for a mental rut, so I'm going to try to attend this with an open mind (and probably leave with a thoroughly-bitten tongue).

These are the ones I will move heaven, earth, and lunch plans to see. The others I really want to attend are:

• How Are Large Applications Embedding Python? (Peter Shinners). Totally relevant for work, but probably more elementary than I'd want.
• What Every Developer Should Know About Database Scalability (Jonathan Ellis). Totally irrelevant for my current work, but I've had to work in this area in the past, so it's somewhat interesting, and I'm curious about what's changed lately.
• Optimizations and Micro-Optimizations in CPython (Larry Hastings). Pure geeky personal interest.
• New *and* Improved: Coming changes to unittest, the standard library test framework (Michael Foord). I'm not quite a test-driven development zealot, but I'm about as close as you can get without applying for membership.
• Python Metaprogramming (Nicolas Lara). More pure geeky goodness.
• Eventlet: Asynchronous I/O with a Synchronous Interface (Donovan Preston). I can't quite decide whether this is applicable to work or not, and there's only one way to find out.
• Seattle: A Python-based Platform for Easy Development and Deployment of Networked Systems and Applications (Ivan Beschastnikh). I was quite disappointed by last year's sandboxing talk (the description didn't really let on that it was all about PyPy), so I'm hoping I can pick up more from this one.
• Tests and Testability (Ned Batchelder). Probably more elementary-level than I'd like, but might have some good discussion.
• On the Subject of Source Code (Ian Bicking). Another blue-sky talk by Ian? Yes, please.
• Python's Dusty Corners (Jack Diederich). I have a feeling this will be like Doug Hellman's PyModule of The Week: 80% of it is "yeah, yeah, I knew that," and 20% is "oh, wow, how did I not know that?"

I wasn't terribly impressed by the tutorial list (other than the compiled Python one), so I'll probably pass on them, but the talks look even better than last year. See you in Atlanta!

Five Things I Hate About C++

A few years ago, the "five things I hate about my favorite programming language" went around. I think it originated with Brian D. Foy's post on Perl. I like his reasoning: if you can't think of five things you don't like about it, you probably don't know enough about it to advocate for it.

Peter Siebel's recent post about the opinions of the folks he interviewed for Coders at Work made me remember it again. While I often reach for Python as the top tool in my toolbox these days, I've been writing C++ for most of my career, so I thought I'd take a crack at C++ first.

So:

5. No consistent ABI

C++ doesn't define a standard application binary interface (a standard for how the binaries produced from source code are laid out or linked together). If you're writing code to link against a pre-built library, then unless you're using the same version of the same compiler, you can't guarantee that your code will work correctly. (Technically, C doesn't either, but for practical purposes, though, C is in much better shape, mostly because C++'s features provide far more opportunities for implementations to disagree.)

The practical result is often that C linkage is considered "safe" and C++ linkage is considered "unsafe", which means that C linkage is the lingua franca for object-level interoperability, and no one really pushes for compatible C++ linkage--which in turn means that it doesn't happen. (While my experience is mostly in Windows programming, the situation seems to be better in the g++ and GNU/Linux world--as is often the case.)

4. Sorta-kinda safety

The first benefit of C++ over C for me wasn't object-orientation. It was that C++ seemed to be much better at catching the kinds of low-level programmer errors I tended to make back then. Class member protection, type-checked function parameters, exceptions that (unlike return codes) can't accidentally be ignored, constructors and destructors that are guaranteed to be called at the right time, improved casting operations--what's not to like?

The problem is that most of the safety features aren't really safe, they're just a little safer, often due to the desire for source code compatibility with C or concerns about run-time performance. You can probably argue that C++ is safer than C, but I believe it's "just safer enough" that C++ programmers get complacent.

Plus the interaction language of features makes it much easier to commit horrible, higher-level design mistakes that are harder to see when reading the code, particularly with things like non-trivial constructors and destructors, misused (and overused!) inheritance, and non-obvious method overrides.

3. Textual macros

The LISP world has had the "hygienic vs unhygienic macros" argument for a long time. In a (grossly oversimplified) nutshell, hygienic macros allow you to define new, reusable bits of language without worrying about the context in which they'll be evaluated. This makes for safer macro definitions, but precludes some very useful techniques that unhygienic macros allow--for example, enabling the code in the expanded macro use and affect variables in the context in which it's expanded.

But the C/C++ macro implementation makes LISP's unhygienic macros look like an Intel cleanroom. That's because they're not even really part of the C language syntax: they're just a simple, dumb textual replacement done in a preprocessing step, before compilation even occurs.

This feature inherited from C is so error prone that C++ added features like "inline" and namespacing to try to approximate the most common use cases for C macros, so that we wouldn't have to deal with them. It still didn't want to touch preprocessor macros for fear of breaking backwards compatibility, though, so now we have the worst of both worlds: a dangerous feature implemented outside the language syntax, with some of its bits duplicated in the language syntax, and guidance that says "sorry about the mess--here's some partial replacements that don't quite cover the gamut, but that's all you get. Have a nice day."

2. Worst-of-both-worlds standardization

C was born as an in-house development language in an AT&T lab in the late 1960s, and was used in anger almost from day one (for reimplementing the UNIX operating system). By the time standardization started, the language feature set was fairly solid and well-proven, and implementators already had real-world knowledge of the features.

C++, on the other hand, didn't go through this process. While C was designed as a language for implementing operating systems (and applications), C++ was designed as a language for implementing language features. It wasn't used (as far as I know) as the backbone of a single, well-known system in the way C was, so the language was free to evolve more divergently and more slowly.

Worse, the development of the language seems to have been driven by the the design and evolution of the specification, rather than by things tried and lessons learned in implementation. In some cases, features were added to the language specification before they were even implemented, in the hopes that smart compiler vendors would figure something out.

As a result, we have features that don't work like you'd expect (like std::vector or auto_ptr<>), features that don't interact well (like templates and class inheritance), and even features that, well, just don't work (like export, which was in the standard speculatively for years before its first attempted implementation, and which as far as I know has never been fully and correctly implemented by anyone).

On the other hand, while C was standardized after it had mostly stabilized, the C++ standardization process started while the language was still very much in flux. As a result, the core language is full of weirdnesses that are explainable only when you know the political situation at the time.

For example, the construction "virtual void foo() = 0;" is a pretty weird way to spell "pure virtual". In The Design and Evolution of C++, Bjarne Stroustrup reveals that the "=0" construction is there because he wanted to get pure virtual functions into the language specification, but a committee meeting was coming up soon, and he didn't think he could convince enough people to get behind adding a new "pure" keyword.

This leads right into...

1. C++ tries to be all things for all people

I think this one is the root of most of C++'s problems. C++ is and has always been a "more-is-better" language. If you like C, we'll make sure you like C++ by bending over backwards to make C code still work (except when it doesn't) and by making efficiency our top, err, one of our top-ten priorities. If you like object oriented programming, we've added classes and inheritance. Oh, multiple inheritance? Yep, we heard that works well, so we'll add it in there too. Parametric polymorphism? Multiple dispatch? Currying? Oh, hrm, we seem to have painted ourselves into a corner... but we can bodge most of that in with templates and partial template specialization. Oh, and guess what? We just figured out that you can use templates to do metaprogramming, so you get that feature for free! Free is good, right?

This results in two, mostly-correct perceptions:

1) C++ is a big grab-bag of language features, some of which are razor-sharp and don't really hang together coherently, but work great so long as you're really, really careful.
2) C++ is more-or-less better than C, so long as you stay with a "sane subset" of its features.

But what is that sane subset? That depends entirely on who you talk to, and the subset that they choose usually reveals more about their own priorities and experience than anything about the language itself.

All that being said, I still choose C++ (or my own trusted subset of it, at least) over C because of the convenience of constructors and destructors, the expressiveness of templates, and the confidence I get from RAII. I still choose it over Java because I don't need to worry about a runtime VM, because I can access platform-specific APIs and native libraries at will, and no checked-exception silliness.

But that doesn't mean I don't sigh a little every time I burn multiple days chasing down an intermittent memory leak, or that I don't steal a surreptitious glance at younger, better-looking languages with less emotional baggage from time to time.

Buying or Building... Furniture

Most software developers are familiar with the "buy-or-build" question: is it more effective to find existing software and try to make it work in your situation, or to build it to your exact specification and take on the burden of maintaining it? But sometimes it comes up in other contexts.

Like office furniture.

My current project at work is winding down, and I'm rolling over to a new one. As part of the transition, I'm moving from my old, two-person office into the new team's bullpen environment.

It might seem like a poor trade, but this team chose to trade in their fairly nice offices because they valued the higher conversational bandwidth they got in a bullpen. Yes, it's a bit noisier, but most of the noise is project-related, and results in quicker and more complete information dispersal both among developers and between developers and SQA engineers (who also share the space).

The big win for me is that it reduces the barrier to pair-programming to the cost of mumbling, "Uh... can anyone take a look at this with me?" And we still have the offices for when we need to make a phone call or do an interview.

One of the stipulations on building out the bullpen was that we had to use existing furniture. Unfortunately, while our current furniture is nice (and somewhat pricey, from what I'm told), it's optimized for a one-person or two-person office. We each get a curvy desk, a table with attached bookshelf that fits the curvy desk as an extension, and a funky rolling file cabinet. But the curvaceousness of the furniture means that it only fits well in a few prescribed configurations--none of which match a bullpen where you want to pair-program!

So the current bullpen, built from curvy bits loosely jammed together, isn't big enough to hold more people. And naturally, the people who handle furniture and facilities wouldn't be terribly happy with us saying, "Oh, this expensive furniture is nice. Now would you mind finding some place in our already-filled building to store it, and buy us some additional expensive furniture just like it, but without curvy bits?"

So our manager/Scrum Master, being the pragmatist that he is, decided we should build our own. From scratch.

Actually, "scratch" in this case really means heavy, solid-core interior doors for tabletops, and prebuilt folding-table legs to hold them up. Assembly is trivial, the surfaces are generous, prefinished, and attractive, and the cost was just a fraction of what we'd have paid for non-curvy versions of our standard furniture (which keeps the facilities folks happy... or at least happier).

There are, of course, some drawbacks. Making single large pairing stations means that you have to choose a single table height. In our case, it was chosen for us by the height of the prefab table legs.

However, my current programming partner suffers from an unfortunate and tragic genetic defect that caused his growth to continue far beyond normal human levels (the medical term is, I believe, "freakishly tall"). I, on the other hand, boast a full 5'3" of height, which seems far more normal to me, all things being relative.

So our alternative solution was to just tear the bookshelves off two small tables (again, that storage problem!), and then use one for each person, moving the tables around when we need to. The works great if the tables have cool adjustable legs like ours do.

(Mine, of course, is the station on the right.)

How to be the most awesome Dad ever

To be the most awesome Dad ever, capable of carrying out feats of skill and mastery usually reserved for the likes of the Avatar, James Bond, or the Doctor, requires just a few common ingredients:

1. The locked, most secret diary of a pre-adolescent daughter (who has lost the key),
   
2. The knowledge that all such cheap locks are the same,
   
3. A set of cheap luggage locks with keys,
   
4. A frantic pre-adolescent daughter in possession of #1 but not #2 or #3, and
   
5. A flair for the dramatic, with which one discloses that one knows how to pick locks, but it's a secret handed down from master spy to master spy, therefore the work must be done behind a locked door (which neatly conceals the fact that you're rummaging around in your bedroom drawer to find #3).
   

Fun with itertools

Sometimes it's hard to shake old habits, especially when you've burned them into your brain as the "standard" way to do things. For example, I've been doing network programming with C and C++ for a very long time. One of the standard pieces of code I've written again and again is the "connect with backoff" pattern.

If a program needs a continuous network connection, and that connection is lost, it should try to reconnect. On the one hand, you want to reconnect as quickly as possible; on the other hand, you don't want to keep retrying (and failing) in a tight loop. So you use a "backoff" timer: after each attempt, you wait longer (up to a maximum limit).

As a C programmer, I would implement an algorithm that resembles this Python-like pseudocode:

# After the first failure wait half a second before retrying;
# double this each time up to eight seconds.
backoff_times = [.5, 1, 2, 4, 8]
cur_backoff = None

while 1:
   try:
       # Try to connect
       connect()
   except ConnectionError:
       # Failed; update the backoff counter
       if cur_backoff is None:
           cur_backoff = 0
       else:
           cur_backoff = min(len(backoff_times)-1, cur_backoff+1)
       # Wait to retry
       time.sleep(backoff_times[cur_backoff])
   else:
       # Success; reset the backoff timer
       cur_backoff = None

But in Python the code to manage the current backoff timer looks out of place.

In a high level language, when the ratio of "code that says what I want" to "code that tells the language how to do what I want" gets too low, you're doing it wrong. It means that you're spending too many mental cycles on the "how," and not enough on the "what".

In this case, Python gives me a better way to tell it just "what" I want it to do: use an iterator.

import itertools

def iter_pegged(seq):
   """Return an iterator that walks the sequence, and then 'pegs' on the last item."""
   return itertools.chain(seq, itertools.repeat(seq[-1]))

backoff_times = [.5, 1, 2, 4, 8]
cur_backoff = iter_pegged(backoff_times)

while 1:
   try:
       # Try to connect
       connect()
   except ConnectionError:
       # Wait to retry
       time.sleep(cur_backoff.next())
   else:
       # Success; reset the backoff timer
       cur_backoff = iter_pegged(backoff_times)

Other than the definition of iter_pegged, each line of code says what only what it wants to do, not how it wants to do it.

And that's what coding in a high level language is all about, no?

Git: what he said.

About three months ago, I started writing a blog post on why my (development) life has changed for the better since I started using git for version control. It's been stewing in my "Drafts" folder for lack of time and attention.

My main point was that git is optimized for messy reality, as opposed to an idealized view of software development that never really happened that way, but looks better when you draw it on a white board.

A few weeks ago, Ryan Tomayko said most of what I intended to say, in The Thing About Git.

So, "what he said". In particular:

• Git means never having to say, “you should have”

• git --rebase interactive is “a bit like git commit --amend hopped up on acid and holding a chainsaw - completely insane and quite dangerous but capable of exposing entirely new states of mind.”

That Looks About Right

tlesher@badwolf:~$ history | awk '{print $2}' | sort | uniq -c | sort -rn | head
    109 ls
    107 cd
     78 git
     26 sudo
     14 rm
     12 ssh
     11 source
     11 more
     10 wget
      9 ipython

(via Chris and Tom)

Oh, look! A startup!

This December, I took a deep breath and jumped. I left a great job at a great company, working on cool projects with fun people, and executed a perfect swan dive back into the tech startup blender.

Now why would a thirty-something refugee from the tech bubble with a wife, two young kids, and a mortgage give up a good salary and stable, interesting job for the stress, uncertainty, and heavier workload of a new startup?

For me, it's all about aligning outcomes with effort. In a very early stage startup, the link between "what is about to happen" and "what am I doing" is as close to direct as you can possibly get. The outcome for the company is directly tied to what you're doing and how well you're doing it: no excuses, and no wiggle room. Naturally, there are external risks (like being run over by a company so big they barely notice the bump), you see the results of your effort clearly.

Working for a BigCo, Inc. is very different. Expending effort in a large company is like pulling on a bungie cord attached to a rock. Pull moderately, and nothing happens. Pull a little more, and the the rock comes along, but it tends to wiggle around in directions you don't intend. Yank really hard, and the rock has an annoying tendency to fly up and smack you in the head.

Now replace the bungie cord with a stick attached to the weight. You pull a little, the weight moves a little. You pull a lot, the weight moves a lot. You push, and the weight moves the opposite direction. The outcome is directly tied to your effort.

To be fair, my last company wasn't quite to the "bungie cord" stage. But when a former colleague approached me about his idea for a startup, I was drawn back in again.

Of course, to (ahem) stretch the analogy, sticks do tend to break more easily, but hey: that's the risk you take.

Sad Mac

Now that, my friends, is a Sad Mac.

10 Things I Learned at MEDC

1. In person, Sue Loh sounds exactly like you'd imagine from reading the CE blog.
2. In person, Doug Boling sounds exactly like you'd imagine from reading his book (except his humor comes off better in person).
3. Windows Mobile 6 isn't as big a deal as it first seemed (unless you happen to be a managed code developer, which I'm not).
4. The Tao is worth seeing (if you can either get in on your own coolness, or else persuade a multi-billion-dollar company to slip the owner some cash).
5. You never know which of your mild-mannered colleagues will turn out to be a raving, iPod-dancing, Viva-Las-Vegas-singing Elvis fan.
6. Managed code (C#) actually runs on "real" embedded devices that are too small for even CE.
7. No matter how awesome your hotel looks, you still need to provide running water to your guests, or they get cranky.
8. When attending a conference, make sure you introduce yourself and describe your company to as many people as you can: you never know which one will suggest a very cool opportunity a few days later.
9. Windows Smartphones are now outselling Blackberry devices, and 90% of them are being sold to consumers rather than businesses.
10. Despite pushing most of the cool new features onto Windows Mobile first, Microsoft still isn't abandoning CE devices just yet. Thanks for that.
    

Ok, I give in.

Just call me a sheep.

O Sole Mio...

Ok, so I've missed PyCon for the past three years running because of work conflicts. But at least this year they've been nice enough to send me to the Microsoft Mobile and Embedded DevCon (MEDC) at The Venetian in Las Vegas.

No "one more thing" announcements from today's keynote, but we did get an earful of how Microsoft wants to position the Windows Mobile family: "it's not just for business anymore". Robbie Bach, the president of Microsoft's Entertainment & Devices Division, claims that Win Mobile-based smart phones are outselling Blackberry devices, and that 90% of those sales are actually to consumers (not businesses).

That sounds great, but the massive emphasis on "all things smartphone" makes me a bit nervous. Microsoft is a smart optimizer when it comes to business strategies: what happens when their CE OEMs (like us) become 2% of their developer base, and their Windows Mobile Pro/Standard OEMs are 98%?

Won't someone think of the poor headless CE devices? Just try using your fancy QVGA display in a freezer for a few hours...

Your Memex is here. Are you using it?

Microsoft Research has put significant effort into implementing a near-literal version of Vannevar Bush's "memex" in its MyLifeBits project. I think we have the memex already: we just don't realize it.

In 1945, the Atlantic Journal published "As We May Think", in which Vannevar Bush speculated that in the future, a machine--the "memex," or "memory extender"--would assist researchers by storing, indexing, and retrieving every piece of information they could possibly need. A user could also add his own text, images, or recordings, and could record notes and comments on the content. And it all fit within a large desk.

This was strong stuff for the time: understand that the state of the art was the Harvard Mark I: a 50-foot-long, 10,000-pound, four-function calculator that could divide at the blinding speed of four operations per minute. To put it in perspective, Bush's prediction was made when my grandparents were not yet old enough to drive a car.

Since 2002, Microsoft Research has been working on implementing MyLifeBits, their version of the memex. And after five years of effort, they now have a one-user prototype to show for their efforts. So don't expect to be shelling out for the Microsoft Memex anytime soon.

But a few weeks ago, I had a realization. I went back to the original, 60-year-old article, and read over the description of the memex again:

A memex is a device in which an individual stores all his books, records, and communications, and which is mechanized so that it may be consulted with exceeding speed and flexibility....

It consists of a desk, and while it can presumably be operated from a distance, it is primarily the piece of furniture at which he works....

In one end is the stored material. The matter of bulk is well taken care of by improved microfilm. ...

Most of the memex contents are purchased on microfilm ready for insertion. Books of all sorts, pictures, current periodicals, newspapers, are thus obtained and dropped into place....

All this is conventional, except for the projection forward of present-day mechanisms and gadgetry. It affords an immediate step, however, to associative indexing, the basic idea of which is a provision whereby any item may be caused at will to select immediately and automatically another....It is exactly as though the physical items had been gathered together from widely separated sources and bound together to form a new book. It is more than this, for any item can be joined into numerous trails.... And his trails do not fade.

There are more parallels, but that's a good start.

So on the one hand, we have a research project to create a literal implementation of the memex that might exist sometime in the future, or a distributed, chaotic, mashup of individual technologies that together get about 90% of the way there today.

Any bets on which will get there first?

More importantly, what are you waiting for that isn't there yet?

Slightly less perpetually behind

Adam Rifkin says, "Having a blog means feeling perpetually behind." Well, today I'm slightly less behind. The Pipes have been blocked for a few months (you might notice that this happens every time a project at work goes into the home stretch), but now that pressure is off, and I'll be able to (finally) polish and post some items that have been languishing in the Drafts folder for a while.

TurboGears decorator madness: linkify

One of the neat features in TurboGears is the @jsonify decorator. It uses RuleDispatch to define generic functions to convert data model objects into JSON notation for use in AJAXish applications. For example, TurboGears provides this default converter for the User identity class:

@jsonify.when('isinstance(obj, User)')
def jsonify_user(obj):
   result = jsonify_sqlobject( obj )
   del result['password']
   result["groups"] = [g.group_name for g in obj.groups]
   result["permissions"] = [p.permission_name for p in obj.permissions]
   return result

The first line lets the default JSONifier rules handle the object; after that, it removes the "password" field for security reasons, and then adds support for fields that the default rules can't handle (like joins). The @jsonify.when decorator handles mapping the default jsonify() function to the type-specific version, so when you want to return a User object converted to JSON, you just return "jsonify(myUser)" and you're done.

This approach can be used for other purposes. For example, in one project, I kept running across is the need to render references to objects as links to view that object. For example, say you have a app that renders the text

Last updated at 12:00 by Joe

with the template snippet:

<p>Last updated at ${thing.last_update_time} by ${thing.update_user.display_name}</p>

Easy and straightforward. But if you want to link "Joe" to Joe's user profile page, then every time you want to do this, you end up writing something like:

<p>Last updated at ${thing.last_update_time} by 
  <a href="${'/users/%d' % thing.update_user.id}" 
    title="User profile for ${thing.update_user.display_name}"
    ${thing.update_user.display_name}
  </a>
</p>

Then hours later you kick yourself because you find one place out of 20 where you made a typo in this monstrosity (see if you can find the one in the example above!).

So I "borrowed" jsonify's approach and created linkify.py for the project:

import dispatch
import model
import types

from elementtree import ElementTree

# Linkify generic methods... modeled after jsonify

@dispatch.generic()
def linkify(obj):
   raise NotImplementedError

@linkify.when('isinstance(obj, model.User)')
def linkify_user(user):
   link = ElementTree.Element('a',
                              href='/user/%d' % user.id,
                              title='User Profile for "%s"' % user.display_name)
   link.text = user.display_name
   return link

Then, in your controllers.py, you can make this available to templates:

# Add linkify to tg namespace
import linkify
def provide_linkify(vars):
   vars['linkify'] = linkify.linkify
turbogears.view.variable_providers.append(provide_linkify)

And now, in your template, you just write:

<p>Last updated at ${thing.last_update_time} by ${tg.linkify(thing.update_user)}</p>

Much, much nicer.

Yes, games ARE different.

In the late 1990s, the game industry was starting to tackle "best practices". Basic techniques common in the rest of the software world (and mostly taken for granted now) weren't getting much traction in the game business. I often heard fellow game developers complain that those practices wouldn't work for games, because the industry was "just different." A decade later, I think they were right--but not in the way they thought they were.

When will Duke Nukem Forever be released? When it's done. DNF is the most popular whipping boy for a game development schedule gone horribly wrong, but there have been plenty of similar examples (including the original Unreal). Game developers are notorious for scope creep and gold-plating, as well as massive mid-stream changes in design and architecture. Why is that?

I've become convinced it's not solely due to "unprofessionalism" or lack of effective project management. You see, if you're creating a business application, or a device driver, or control software, you can specify the requirements, from which you can create a design, from which you can create code, which you can trace back to requirements. It's not hard to write effective, measurable requirements for this type of software, which means it's not hard to justify or drop a given feature with objectivity.

But games have a damnable, hidden, unwritten Requirement Zero:

Thy Game Shall Be Fun.

Requirement Zero is a real killer. Because it's not objectively measurable, it's not easy to manage. You can pick on Daikatana because of its schedule slips, or because you don't like its designer, but the bottom line was it just wasn't a fun game, and for more reasons than the obvious "sidekicks get squished by doors" bugs.

I once had a conversation with Danielle Bunten Berry (of M.U.L.E. fame) and a colleague about a particular game. It was gorgeous, free of technical glitches, pushed the envelope on technology, and had a decent storyline. But my colleague summed it up this way: "There's a big hole where the fun should be".

You see, users don't expect their word processor, or their spreadsheet, or the software that dispenses their soda at the local mini-mart to be fun. It's sufficient for it to be "fun-neutral": so long as it isn't so badly designed, defective, or poorly-performing as to be "un-fun", it's acceptable.

You can create a game that completely satisfies the spec, is totally free of bugs, finishes on-time, stays within its budget, and has a flawless marketing program--and it can still be a miserable failure if it isn't fun.

Yes, games ARE different.