Wednesday, May 25, 2005

Unsung Heroes of Python: asynchat/asyncore

Python is known for its "batteries-included" nature. One of the batteries that gets too little attention, I think, the asynchat module.

One of my pastimes is playing a certain web game, which features live chat. In a previous incarnation of the game, one of the players wrote a very useful "bot"--an automated pseudo-player that sat around in chat, and provided useful information when queried. She quit before the newest revision of the game was released, though, and some of the players were missing the bot.

So, I pulled out the asynchat module, launched Ethereal, and started reverse-engineering the chat protocol (source and some documentation are available, but the version I was talking to seems to be somewhat customized).

45 minutes later, I had a fully-functional bot. Another hour, and I had a nicely-factored module from which you could build a whole new ultra-wizzy chat client. (Teaching the bot all about the game took another eight hours, of course, but I don't think any libraries would help with that...)

Now the cooler-than-thou Pythonistas out there are probably saying, "Bah! Twisted Rules!". That's nice. Twisted may be sexy, but asynchat/asyncore has some advantages:
  • It's simple. Two modules, under 1k lines of code (as opposed to a raft of modules and 80k lines of code). No surprises.
  • It's documented, so I don't have to hang out in an IRC room or grovel through thousands of lines of code to figure out what's wrong.
  • It's included with Python, so I know it's tested--no surprises when I try it on a new machine.
  • It's written in everyday bog-standard Python, not its own framework on top of Python, so there's no prerequisite learning to do.
  • I'm reasonably sure that it's not going to be drastically changed.

So, for grinding out TCP/IP-based tools quickly, nothing beats asynchat. It probably would have taken me three times as long in C or C++, and if I'd started with Twisted, I'd still be reading Twisted source code...

18 comments:

Glyph Lefkowitz said...

Yes, but if I'd started with Twisted, I'd be done before you ;-)

Glyph Lefkowitz said...

But, seriously. I'm glad you've had a good time writing a quick hack with asyncore, but ... no reason to put down Twisted. I don't mean to offend, but some of your comments are self-sustaining FUD that people keep repeating because they heard it somewhere else...

- Twisted is documented. As I've pointed out in other fora, there is a huge amount of documentation. If you're just trying to bang out a protocol you only need to understand about 3 interfaces and there is a very thorough "writing clients and servers" document. If you want to do just what asynchat does, basically, take a look here: http://twistedmatrix.com/projects/core/documentation/howto/servers.html

- The split that you're referring to as a "drastic change" has broken no backwards compatibility that I'm aware of. If it has please point it out so that we can issue a maintenance release.

- Asyncore doesn't deal with cross-platform issues at all - it's just not part of its charter. You will have "surprises when you try it on a new machine", like these cross platform issues, for example: http://www.itamarst.org/writings/win32sockets.html

This is one of the main reasons why I didn't use it for Twisted in the first place, in fact.

- Twisted is pretty well-tested, at least the parts that mimic asyncore's behavior. Take a look: http://twistedmatrix.com/buildbot/

- The interface you're referring as a "framework on top of Python" to was removed in the 2.0 split, replaced by a backwards-compatibility layer which now uses the new zope3 interfaces module. While not yet included with Python, it is likely to become a standard of some kind, and the Twisted team is engaged with the rest of the Python community to try to build some consensus about what the framework features it shares in common with other systems should look like.

Florian said...

Glyph twisted is quite nice, tried&true, sturdy and a good lad overall. I've no concrete reason to medlle in the worriesome and imperfect code I usually do to make networking happen.

About the only reason I can put forward is that it feels wrong. For me, personally.

Everytime I touch networking personally though, twisted feels a little less wrong. I estimate now it can only be a matter of a few months to a year before I see reason and be converted, though I haven't reached this point yet.

Oh, and I must mention that I find #twisted is a major bummer, culturally, and thus contributes perhaps a fair deal of antipathy towards the whole thing for me.

Fredrik said...

"This is one of the main reasons why I didn't use it for Twisted in the first place"

Which is, of course, a typical trait for the Twisted camp -- instead of working with other developers and building on the Python core, you ran away, built a new thing, spent lots of energy slamming asyncore and medusa (if you're offended by someone ranking asyncore over twisted for a given project, maybe you twisted folks should look yourself in the mirror; FUD is extremely common among twisted zealots), and contributed absolutely nothing to the core.

And yes, asyncore works perfectly fine on multiple platforms. Skip the FUD, skip the CADT development approach and the "stab people in the face" line of advocacy, and people make take you a bit more seriously.

Tim Lesher said...

I think you're missing the point, glyph.

The title of the entry says nothing about Twisted. The bulk of the entry says nothing about Twisted. In fact, the first draft of the entry didn't mention Twisted at all...

...but I had a nagging feeling that, seconds after I posted it, someone would have made a comment like "it would have been one 20 minutes if you had used Twisted."

Just, in fact, like your first comment.

Jon-Pierre Gentil said...

I'm sensing metaFUD right about now. Too much FUD about FUD slinging.

Honestly, both twisted and asyncore work well for what was accomplished.

People need to stop being so butthurt over some opinions. He even said that twisted is sexy. Sheesh.

He just wanted to do something without needing a large framework, that's all.

Jonathan Ellis said...

Incidently, the version of chat on CB is customized only in the sense of "running the latest code from cvs."

Somebody should get those damn lazy nfc chat developers to do a release one of these days.

Andrew said...

The title of the entry says nothing about Twisted. The bulk of the entry says nothing about Twisted. In fact, the first draft of the entry didn't mention Twisted at all...

That's a bit unfair. You explicitly list five whole bullet points -- and nearly half your entry -- of complaints directly squarely at Twisted, so I think it's reasonable to get a response from the Twisted community -- particularly because some of the points are misinformed.

Florian said...

Ah the nice deduction by guilt.

Here's what he did. He mentioned five good points about asyncore, not five bad ones about Twisted.

Then it happend again, human nature. Somebody said "This five things are good about asyncore.". Another one heard "This five things are bad about twisted."
This kind of thing is common. The interesting bit is why it happens.

Mostly it happens because there's an expectation to hear this kind of critics, and thus you tend to hear it even when it's not present.

There's not many reasony why you would expect to hear this kind of critics. Either you get it a lot, or you feel yourself guilty on the points ( which is a specialization of "you hear it a lot" )

Now if twisted could defy this critics with ease, there'd be nothing to worry about.

Or is there something to worry about?

Anonymous said...

Considering the article has that line just before the 5 good points of asyncore :
Twisted may be sexy, but asynchat/asyncore has some advantages

It's easy to understand why some people would interpret them as bad things about Twisted

Anonymous said...

Well the sad thing wth blogs is that instead of commenting on the good points of an article, people just waste their time with lame comments.

Frederick is right, nothing mentioned against Twisted. But this is so typical in the Python community. As soon as a project is big enough, people gets really protective and don't accept that other projects might do better in some aspects.

Another example. People are also talking a lotabout Rails, not only because it has good points but also because the ruby community is much open to critics.

That being said, I am a hug fan of Python and use it on a daily basis. I just tend to be careful with its community.

Andrew said...

Florian:

Here's what he did. He mentioned five good points about asyncore, not five bad ones about Twisted.

Actually, he did both at the same time :)

I strongly suspect that if he hadn't mentioned Twisted at all that one of two things would have happened: 1) No-one would have said anything about Twisted, or 2) Someone would say "Btw, do you know about Twisted?" "Yes." "Fair enough then."

It's a blog, with a link inviting people to post comments. People have done so. It's not a big deal :)

Fredrik:

Which is, of course, a typical trait for the Twisted camp -- instead of working with other developers and building on the Python core, you ran away, built a new thing, spent lots of energy slamming asyncore and medusa [...] and contributed absolutely nothing to the core.

Since well before Twisted 1.0 (as the CVS/SVN history will show) there have been attempts to provide smooth compatibility for asyncore code with the Twisted core, and it was simply too hard -- the way asyncore works is just not flexible enough to do this. The scope of Twisted is also different -- asyncore/asynchat don't come with batteries included in terms of providing support for common protocols and protocol types that Twisted does.

There are definitely advantages to the asyncore way -- a smaller, simpler library is easier to learn and understand. But Twisted is sufficiently different that it's not a simple matter of "well, they should have just improved asyncore." It's not that easy.

I also think the Twisted community has done a reasonable job of contributing bug reports, bug fixes, and feature ideas and the like to back to Python.

By the way, I think you misunderstood what Glyph meant about asyncore and cross-platform issues. He didn't say it doesn't work -- he said that it's not something asyncore itself deals with, the applications built on it have to deal with the annoying differences in socket behaviour directly, whereas Twisted tries hard to make sure you don't have to know or care about them.

Anonymous said...

I have to disagree with Glyph when he says that Twisted is documented. In my opinion, the incomplete, out-of-date and incorrect documentation distributed by the Twisted folks is shameful parody of documentation. I tried to use it to get started with Twisted to do a simple web server that I planned to extend into a web application. It took me days to figure out how to do this and only because I finally found an example on somebody else's website. While I was googling for info, I was amazed to see how little information on Twisted was available on the web! Isn't anybody else using it? Or is it only the toy for the in-group of Twisted developers?

I did frequently come across references to IRC but that doesn't help me much, because IRC disappears as soon as it is written. Unlike web pages, blogs, etc. So, IRC doesn't count as documentation. And many of us don't use or have access to IRC either so this is worse than useless.

Anonymous said...

"So, IRC doesn't count as documentation. And many of us don't use or have access to IRC either so this is worse than useless."

Yep, IRC is a great clique builder. And even when you do have access to IRC logs, the information density is usually so low, even if you disregard the banter and (apparently) the offensive attitudes, that most people would be turned off from just using the software in question, let alone becoming part of the community.

Rod Senra said...

The comments theme drove away from the blog entry theme. Nevertheless, my testmony in favour of Twisted is this:
First time I used Twisted, it took me 40 minutes to build a fully-functional
transparent proxy, that time span includes reading docs, samples, and code. I do not believe the same could be accomplished with asynchat/asyncore
in the same time frame in the same conditions.

Marc said...

Just gotta stop a sec to say "hah!"

Twisted well documented?? Twisted is a *mess* (both in documentation and in code)... we have 2 projects that are based on twisted (one large, one very small) and while both work admirably well (robust and scalable beyond my hopes) it was not an easy path; I often found myself digging through the (messy -- sorry but it is) source to figure out what one obscure function/variable did (or didn't do, as parts weren't always completed or were not used anymore). Truth is, it ain't a pleasure to get to know twisted, and I think thats what the author was getting at ("quickly").

Of course, now that I know twisted I rely on it for just about everything :-)

Oh and I am mainly referring to twisted 1.0/1.1/1.3... I have no experience with twisted 2.0 as of yet.

Anonymous said...

That is so great!
It's also possible to run python in parallel on SMP: Parallel Python

Anonymous said...

As of today, documentation for Twisted 2.* is still pure crap and almost non-existant...