jmoiron.net

Python has lots of built-in serialization methods. It has had a standard module called pickle for a very long time, and exposes its built-in trusted data marshaling routines in the marshal module. Since the release of Python 2.6 in October, 2008 it has also had a standard, well-tested json library.

Ted Dzubia, an internet-notorious opinionated windbag and adept comedic writer, recently wrote a scathing post about Node.js, which was half in reply to a somewhat bizarre rant by its author Ryan Dahl on the state of software and a desire to retreat to the simplicity and stability of the Unix Philosophy.

I've been writing a bit in nodejs recently, writing an extended version of etsy's brilliant statsd which handles logging as well called logd. While writing it, I made a couple of mistakes that are probably quite common even to people who know JavaScript quite well but have not used it much outside the browser environment.

I started on a project recently that required some HTML scraping, which itself requires some kind of web crawler. There are good projects out there to facilitate this in python, especially scrapy, which I recommend, but for a variety of reasons which include NIH but is not limited to that, I decided to write my own crawl portion. About 75% of the way through, I thought it'd make a good blog post, since not everyone needs the sophistication scrapy provides, and the basic techniques and design considerations are generally useful.

I want to explore some trends in software development that I am somewhat conscious of as ripples in the web development community. Some of these philosophies I recognize from stories that were told to me by my elders, and some are either new or represent some other facet of past ground covered but then forgotten.

For a long time, the root of my website just forwarded you to this blog. The whole website itself was just a thin conceit for this blog, and I did very little else that warranted inclusion on the website.

In response to @ericholscher and his recent blog post "Virtualenv Tips", I've decided to describe the way I use virtualenv in the hopes that someone might find it useful. Before you even bother reading this, though, you should install virtualenvwrapper, as most of my customizations are built into the various extra hooks it provides.

This is probably old news to most, but I've discovered recently that github has won.

While integrating celery and django-celery into a project at work recently, it gave a warning about the transaction isolation level on our database:

This interesting post on .NET and its development community really struck a tone with me.

I've been pleasantly surprised with the amount of interest in johnny-cache since Jeremy and I released it this past weekend. A lot of the comments revealed that perhaps the documentation is missing an important discussion on the repercussions of using Johnny. They are also pretty positive about the name :)

I've been waiting a long time to write about this. Johnny Cache is now released upon the world. It's a drop-in caching library/framework for Django that will cache all of your querysets forever in a consistent and safe manner. You can install it via pip install johnny-cache.

A friend and colleague Jeremy Self, who I've been working with the past few weeks on a project that will probably be released shortly, told me the interesting results of some profiling he was doing on a lazy-evaluated data structure:

As mentioned in yet-unresolved #7835, I've been writing a sort of meta-application recently that doesn't provide any models of its own but absolutely requires models (and data) of various types to test against. I ended up adopting julien's technique, which puts the test-only application inside the tests module, and then overrides setUp and tearDown to monkey patch your settings to include the test-only application and then run a syncdb command. This approach started out working very well, but I soon ran into a fairly major problem: fixtures failed to load.

For some background, I've been working with Django for a few years now and for the past year have been a developer on a rather large django application at work. I did a few projects with SQLAlchemy about 2 years ago when I was put on a project whose requirements seemed to imply the need for a database but had no web component. I know of and have used other python ORMs (notably SQLObject), but this post is about SQLA and Django's ORMs, and how I consider the differences between them to be important but also neutral.

With my recent release of a new version of this site, I finally made the jump from using mod_python to using mod_wsgi. This change had been a long time coming, but was made slightly less than straight forward by my desire to deploy to virtualenv environments.

Spent a few hours today trying to figure out how to do QuerySet caching in django in some kind of transparent manner. We are working on a multi-tiered caching system, sort of like that which was released by the pownce guys a few weeks ago, except with a bit more there there.

Python metaclasses can be really intense. SQLObject's declarative metaclass base, popularized by Django's ORM, has been as useful a DSL as the python world has had in a long time. But please be careful when using metaclasses in a way that makes it difficult to extend via subclassing!

I've been in search of the "holy grail" of plain text markup for years. Before the first django version of this site, I was writing a custom mod_python handler to run a server-page style site and as part of that writing my own wiki creole. Since then, I've used TracWiki, reStructured Text, Markdown, MediaWiki, and Textile but have been left slightly disappointed by them all.

Unfortunately for those seeking pimps or the services of their employees, this is sadly about editors. Seriously.

Django 1.0 has some nice new features and some better ways of doing things we used to do manually. The admin app getting decoupled from the ORM has really paved the way for a lot of customization to go into the admin app; so much so that the last 10% it used to be missing for most CMS needs is now probably easily realized via small decoupled customizations.

From some talks with Greg Deangelis, who seems to have taken Jeremy's design aesthetic to heart, I have decided that I should fix this broken site. Fixing it means a lot of things:

Editors Note: This entry was published late due to laziness and technical difficulties, and first appeared here on September 7th, 2008.

You might have noticed that the site has seen a few minor changes. This reflects a large-scale refactoring of the entire blog application (in order to fix a queryset caching issue), and the simplification and consolidation of a lot of crufty templates. You can see my progress via the hg repo.

After a few days of actually getting things done, I relapsed into staring lazily into the depths of the internet and finding entertainment in what most people would probably consider the most mundane of places. This time, it was one of the rare interesting reddit articles in which a guy mails typographers asking them for handwriting samples.

I suppose this post is long overdue in a lot of ways. I've ended my employment at Attila Technologies and started a position at Advance Internet. I've quit my job as a toolsmith/utility/architect and have become a python/django developer. I started my new job not in Journal Square (where the company is located), but in Austin, Texas.

I've disabled comments because I have some neat captcha-less ideas for how to tell bots to fuck the hell off from my comments section but I don't feel like implementing them for django. The new system (mostly homegrown glue w/ selector, beaker, mako & werkzeug thrown in) has been in heavy development the past few weeks and commenting should be available by the summer. Sorry Mike; now you've no reason to come here.

This past week/end, I set up Mercurial on my server and started a few projects there. One is called slipcover, which I talked about last time. The other project, which I've started more recently, isn't really called anything. It was started as a direct result of me using Hg and I must say that so far I am pretty happy.

I've been doing a lot with CouchDB and WSGI the past few months, with positive and negative results. I'm finally getting the "hang" of Document Orited Databases (DODB? Rubbish acronym), and starting to understand what is possible and what isn't, how to do one-to-many and many-to-many relationships without having lots of queries. It struck me that although the per-query time in CouchDB is thusfar a lot slower than relational databases, the overall database time for individual webpage loads is far less, and there are far less queries and complex information joins going on.

Happy Anniversary to me!

I've been thinking about WSGI and CouchDB recently, while on the subject of digital inflexibility. First, I want to clarify a few things about what I mean by flexibility with respect to an application, and how the current crop of frameworks approach this problem. If you want to follow this musing well, I highly suggest reading "What PHP Deployment Gets Right" by Ian Bicking; or just his entire blog, and most of the crosstalk on the web about REST, Web Services, and the evolution of the WWW.

I've been playing with a few ideas recently, stemming from the perceived inflexibility in my blog software and my inability to develop a gallery over the past, oh, year and a half.

I have been programming in Python for quite some time now, and I've been doing it professionally for over 2 years. Despite this, I am not nearly as proficient at the language as I could be, probably because I am using it professionally and have to devote time I could be using learning the language to solving problems.

I am taking a class in Quantitative Software Engineering. It's a discipline that likes to throw around all these numbers, but is mostly posturing as something more defined and understood than it really is. The professor is smart and encouraging and I like to write on the discussion boards.

I just came off a day at work, a week at work, that was legendary in frustration and triumph. I'm pretty sure that most of the details of what I struggled through, heroically, are confidential at worst and tangential to an NDA at best, so I'll gloss over things in generic but jargon laden terms.

Unfortunately for my family my updates recently are not about me, but about things I'm doing. I've been pretty prolific as far as coding is concerned thusfar this year, both writing programs at work and on my own. Because I spend so much time at my girlfriend's place, and she has a windows machine, I've had to impose a few hopefully non-annoying programs on here, but I finally found the last missing piece: PieTTY. Now that unicode things work in my console, I'm able to develop on her computer and actually verify what I'm doing.

You might have noticed that the word finished was in quotes in my previous post. This was on purpose, and anyone who is a hacker (or a perfectionist, with, say, deadlines) will know why and roughly the content of this post. Ninrename (which is a terrible name that is more and more nondescript of the programs capabilities) has been expanded upon quite a lot in the past week (or two). Features are being added left, right, and center, and the desire is there to make it actually very good and not just a one off.

Just "finished" hacking together some code that started out as a small desire and has ended up an obsession of sorts. Initially, it was just going to be a smart, specialized file renamer. It has ballooned into a poorly written (but fairly solid) beast of a program with crc checking and unrar/zipfile support. When I say it's poorly written, I mean it's not beautiful like my xdccq module is, for instance. It isn't elegant in the least, does things in a way that is acknowledged as poor design decisions, and the main dispatch is a giant ugly conditional mess. But it works pretty well!

On friday November 17th, I started the first of what will be 2 two week vacations in 2 months. That night, sitting at my desk in the office at 7:00, I decided to fire up monodevelop and try out the Boo Language. I had been using Banshee since fixing the support for it in my xchat irc script, and was somewhat impressed with the startup speed of mono gtk applications when compared to Python. Although I had just finally gotten a pygtk app to run in both windows and linux (unmodified! it was kind of exciting) I decided to go ahead and play around with the CLR's library a little.

A month ago while I was visiting home I put on my girlfriends glasses and lo, I was able to see much better than normal. I had noticed slight deterioration of my vision about a year earlier, as signs that were far away were slightly fuzzy, but I saw far better with the glasses on than without. This was a bit alarming.

I spent the last few days worth of free time tearing my hair out with respect to the todo list and it's general lack of performance. The problem seems to be that nested sortables (Sortables that contain other Sortables) are extremely slow in Firefox on linux. In a quest to get out of the prototype/scriptaculous world and also fix my application, I spent 3 hours surveying various ajax toolkits.

Tonight, a night in which I wrote source code to achieve some purpose, felt good. It felt real good. There are a few things going down around the development-sphere that somewhat might indirectly involve me (or not), and here they are in no particular order:

1.0 of this site (what you might have been looking at for the past few months) was simply the migration from my own mod_python framework to django. 1.1 is the the culmination of about 3 months of work, mostly on the todo list application, but also a great deal on reorganizing the backend to make redeployment easier. There is still a great deal more work to do in that area, and the gallery is still yet to come, but development pacing has started to get pretty fast now that I'm very comfortable with all of django's nuances and idoioms.

Although it seemed like it might be complete and utter fallacy up until about August 30th, I actually did manage to not only sign up for 2 graduate courses but also Japanese levels 5 & 6, thus saving myself from being a total liar.

I've been in some kind of groove recently. It isn't too productive of a groove, but my mind has been racing like I'm on speed. If only insane hyperactivity came with the same metabolism benefits as methamphetamine. Another drawback is that I haven't come down for the past week, and have been going to sleep on average 20 minutes later each night.

This one's a quickie. I promised to make available some changes I made to Gene Cash's EXIF.py library. I'm only providing them as a diff patch because ceache is working on improving the performance of the library extensively by using PIL's exif parsing code with EXIF.py's MakerNote deciphering code.

Feels like I've been doing less than nothing for the past 10 days.

I missed what seems like quite a few get-togethers of jeremy's, and when people asked my why the only excuse I could muster was that I don't like sushi. It's a pretty valid excuse, but I think if the company had been different I would have gone.

It started out as an insertcredit forum post, but the promise of extreme difficulty coupled with my complete amnesia about completing Super Mario Bros. 2 made me very interested. To throw some fun into the mix, and as a little academic exersize, I wrote the mario challenge! website for me and some friends to track our progress and win fabulous prizes.

I spent most of today investigating different ways to read EXIF data in Python for the up and coming gallery. A few days ago, as noted elsewhere, I spent a day investigating Mono and writing trivially simple GUI applications in boo. These were kind of liesurely activities, but their usage was immediately relevant to anyone.. even a non-coder. When my images are immediately available to everyone along with the ISO setting, shutter speed, and timestamp, normal people will understand. When I show my parents GUI apps that I have written, they understand its uses.

Palomar played tonight at Mercury Lounge with Hockey Night and some 2 piece that I didn't stick around to see. The playfulness and joy that Palomar show when they're playing is totally infectious; Randy told me that they were going to be playing in New Brunswick next week and I am really looking forward to bumming a ride from someone and catching that show. Their new album is coming out "Sometime..."

Jwz, who wrote or contributed heavily to XEmacs, Xscreensaver and Netscape, said in one of his many insightful blog entries that if you are trying to write social software, your main focus should be "How will this software get my users laid?" I haven't been so concerned with that recently (although it's an area that deserves some attention), but I've come to the decision after 2 or 3 years that I need something to keep track of me and tell me "You will feel bad if you do not accomplish this." I think others have come to this decision, too.

I finally understand apache, and what really happens internally when a request is made. I wasn't able to understand mod_python until today. Until the limitations of the server page model are ground against in such a way that you actually wonder, "There has to be a better way!" you aren't really ready to write your own Apache request handler. Even if it's totally fucking easy.

Since most every other part of this website is a hastily cobbled together piece of crap, I see no reason for the title I've selected not to fail. If it doesn't, let it be known that its due to no merit of my own. My site has shown the capacity to display Unicode Japanese (日本語) characters in the past, so perhaps this won't be any different.

The task now, as it has been and most likely will continue to be, is to design something to facilitate me in keeping a todo list and actually using it. There are a number of things that a todo list that I'd actually use must do; here they are in no particular order:

Every once in a while I get to disliking the code on my website, but I never dislike it enough to do anything drastic. I'm in such a state right now; its basically a Hodge podge of python scripts, none of which I care for all that much. There's nothing exactly technical about anything either; a bunch of parameterized strings with very static HTML code mixed in an inconsistent jumble. The only thing that is of particular note at this point is the design that I tweaked quite a bit to accomplish (although still rough around the edges), and finally the problem isn't that I don't like the way it looks, but I don't like the way that it works.. that it is.

For most of today I was trying my hand at developing an object oriented framwork that would result in generated code that would look as if it was handwritten. My preliminary results are pretty encouraging. The main idea comes from the the design of PHPhtmlLib, but I wasn't happy with the generated code from that library and wanted something a little more XHTML friendly.

For those of you possibly uninterested, this is going to almost entirely dwell on esoteric computer usability discussion.

Been using ViM for a while, and decided to use the X version (gViM) for my Java class. The X version comes with a fairly gorgeous color theme, and I wanted it to actually come up every time I used it. I searched quickly for some way to do this in the GUI, but since its Vim I figured (rightly) that most things were done through the RC file. A little research on the Vim irc wiki got me a few options that I've wanted for a while, so I figured I'd share them:

Sometimes I wish that one toolkit would just win, and then I wouldn't need to waffle back and forth between them. They can look the same and function quite similarly; it IS possible; but in the example image, I had to wrestle with spacing issues in Qt for quite some time in order to get it identical to the Gtk version. Part of me has just wanted Gtk to get better than Qt so that I could actually use it and not feel like I'm implementing parts that should be done for me (like cut&paste).

I probably should be sleeping right now, but instead I'm here (on my bed), struggling to think of something to write about. This isn't supposed to happen; I'm supposed to actually have something to write about. I do, indeed; but unfortunately I just can't seem to put any words down on the subject.

My life for the past weeks has consisted of the following things:

My time management lately has been impeccable. I've been going to work via bus, which requires quite the strategic outlook as far as schedules are concerned. I've been working 10 hour days and am still technically without a contract (although continually assured that the paper work is at its last stage, however many steps that might be from me), leaving myself both time and money to go to a myriad of new york city concerts. And yet, with all of this activity, I still have no time to do the things that I want to do besides these things; one of the continual victims is sleep.

I finished my last final ever as an undergraduate May 14th, 2004. Let the celebration begin.

I read the whole archive of Elf Only Inn the other day, mostly at the impersonal behest of Tycho, who's on comic has left me somewhat empty these days. This archive reading of course reminded me that I haven't read the Thin H Line/Sexy Losers in a while, so maybe I'll eventually get to doing that.

PacMan; the Slackware Package Manager; version 0.3, previously codenamed "Dartmouth", has been completed. I figure I can have codenames if Intel can. Here's a changelog since v0.2:

Why the hell can I not design context menu's in designer? The idea is so idiotic that I've spent 2 days trying to figure out how, because it's something I think just has to be there. I'm just missing it, or something, because I'm new. But after surfing freenode for a while and asking some questions there, I couldn't figure it out. It has become a major roadblock in the first steps I was about to take towards making my program actually do anything.

I am no GUI designer. I have new found respect for all of the people from whom I've consciously or unconsciously stolen the ideas that go into the screens coming up shortly. Actually, I always had respect for a well built program, but at this point I think I have more respect for poorly built programs: in order for them to get wherever they are, a lot of people had to think quite a bit.

I mentioned that I hated having to install Eric3 in order to compile the *.ui files that QtDesigner creates. I did some exploration and realized that Eric3 used "pyuic" (PYthon User Interface Compiler), which, wouldn't you know it, comes standard with PyQt. There goes one huge complaint I had about the whole system as well as 3 packages. I find myself thinking that PyQt did it right, and that libglade should develop a python generator for the glade UI files.

When I first started using python+gtk+glade/libglade, I didn't think that any other programmers had it better than me. I was still in a platform dependent stupor, a sublime state of anti-idiocy insanity where my own idiocy was incalculable and yet undetectable, but then again I had never had any experience developing Gui's even though simple tools were most likely available. My PyGTK Tutorials, available still in the docs section, are a testament to how much I enjoyed the system, as well as how much I began to understand it.

I'm posting this for some feedback on the project I was thinking about yesterday. Now that I've had some time to start the backend part of it (it will be useful for scripting regardless of whether or not I go for the full glory), what do you think the usefulness of a slackware package manager front end that integrates with swaret and provides much of the information in a quick, visual way? For instance; click on a package and get what the latest version is (if its not installed, you can upgrade), the package description, the package size, files that it has in /usr/bin, packages it depends on (that you have or do not have), popular packages that depend on it.

I've been trying out a few console only programs in the past week; a few programs that I had been meaning to try for some time. One of them was swaret, the SlackWARE Tool, and the other is Screen, the UNIX tool.

The need for a subtitle, a la "or how I learned to stop worrying and love the bomb", which oddly enough came up in political conversation, family banter, and the thanksgiving festivities last night, suddenly presents itself. The need that is, in case you forgot within mhy first of what will probably be many rambling run-on sentences. After all, although most will understand, I'm sure I can probably name at least one person who won't get the title.

Still pretty insane, although, quite oddly, a healthy mixture of jerumizing, IRCing, and a visit to my orthropedic surgeon have somewhat grounded me in a state of somewhat benevolent happiness. My apathy can't keep a smile from my face when my homeade tutorials gain international attention. Even just the little things like this and my much sought after anjuta packages make me happy to know that at least the work I put toward this site is for more than mere personal edification.

In the past few weeks, lots has been happening, not that you'd know. There has been little progress on this site, although what little there has been would count as huge amounts elsewhere. As I wonder whether or not Jerumu could even understand his own code if he goes back to it at this point, I realize I have more important things to worry about.

I got JTEX to version 0.8; cut/copy/paste functions were just implemented in PyGTK (go figure) and I needed to recompile from CVS HEAD. I'll put some packages up that have all of the latest patches shortly (Update: The package is there now -- J)(and label them accordingly); anyway, the only thing left for JTEX is the preferences file handling (which has as of yet undetermined difficulty attached to it) and undo/redo. At this point, I think that making the tutorial will be once again more fun than actually making the program, as there is really very little left to actually deal with PyGTK or LibGlade. Only time will tell, however; but I'm trying to set a goal of 1 page on the tutorial a week.

Updated quite a few things on the site, even since the redesign, which happened half a week from now. Here's a list of some of the things I fixed:

You know what they say; One new theme deserves another. For the most part, it's true.

It is my contention that Lorde Omlette is a fake; a ninny with soiled knickers. Let me explain, please.

In an effort to keep the design of Jtex simple, and further reinforce that it is not competition for other projects, I've taken a few features out of the release plan. The project is more exciting to me as a tutorial to myself and others on building PyGTK applications rather than reinventing the text editor yet again. The removed features follow:

I can't seem to remember what it is that I wanted to have done by Friday night, but whatever it is, I probably don't have it done. The tutorial is still languishing on page 3, and although I didn't get much actual development done, I did learn quite a bit about how to use pango to change fonts in the TextView, among other things.