jmoiron.net

Python has lots of built-in serialization methods. It has had a standard module called pickle for a very long time, and exposes its built-in trusted data marshaling routines in the marshal module. Since the release of Python 2.6 in October, 2008 it has also had a standard, well-tested json library.

I started on a project recently that required some HTML scraping, which itself requires some kind of web crawler. There are good projects out there to facilitate this in python, especially scrapy, which I recommend, but for a variety of reasons which include NIH but is not limited to that, I decided to write my own crawl portion. About 75% of the way through, I thought it'd make a good blog post, since not everyone needs the sophistication scrapy provides, and the basic techniques and design considerations are generally useful.

While integrating celery and django-celery into a project at work recently, it gave a warning about the transaction isolation level on our database:

About a week ago, I released Johnny-Cache v0.2, which includes a few important bugfixes over 0.1 and adds a blacklist option called MAN_IN_BLACKLIST, which is a tuple of table names that you don't want Johnny to touch. Note that any query that references that table will also not be touched, so don't use it on tables that you might want to join against.

I've been pleasantly surprised with the amount of interest in johnny-cache since Jeremy and I released it this past weekend. A lot of the comments revealed that perhaps the documentation is missing an important discussion on the repercussions of using Johnny. They are also pretty positive about the name :)

I've been waiting a long time to write about this. Johnny Cache is now released upon the world. It's a drop-in caching library/framework for Django that will cache all of your querysets forever in a consistent and safe manner. You can install it via pip install johnny-cache.

A friend and colleague Jeremy Self, who I've been working with the past few weeks on a project that will probably be released shortly, told me the interesting results of some profiling he was doing on a lazy-evaluated data structure:

As mentioned in yet-unresolved #7835, I've been writing a sort of meta-application recently that doesn't provide any models of its own but absolutely requires models (and data) of various types to test against. I ended up adopting julien's technique, which puts the test-only application inside the tests module, and then overrides setUp and tearDown to monkey patch your settings to include the test-only application and then run a syncdb command. This approach started out working very well, but I soon ran into a fairly major problem: fixtures failed to load.

Since Audacious crashes in Ubuntu 9.10 when playing NSF files, I was compelled to compile a newer version for personal use, and I noticed that their build system is tricked out with very little extraneous geek-cred output and some colored output. I have been writing lots of little one-off bulk job scripts lately, and when I realized the answer to the question "Why don't I do this?" was "I don't know how", I decided to do some google jump roping and figure it out.

Been having a sort of chronic problem with my phone over the past couple weeks where songs would skip. Didn't think it was too big of a deal, but transferring some more music over to it the other day somehow my pictures all disappeared. I take fairly frequent backups, but this caught a few that weren't on the memory card. I checked dmesg, saw end_request: I/O error, dev sdg, sector 48728, and knew that I was probably in for some problems.

With my recent release of a new version of this site, I finally made the jump from using mod_python to using mod_wsgi. This change had been a long time coming, but was made slightly less than straight forward by my desire to deploy to virtualenv environments.

Spent a few hours today trying to figure out how to do QuerySet caching in django in some kind of transparent manner. We are working on a multi-tiered caching system, sort of like that which was released by the pownce guys a few weeks ago, except with a bit more there there.

Python metaclasses can be really intense. SQLObject's declarative metaclass base, popularized by Django's ORM, has been as useful a DSL as the python world has had in a long time. But please be careful when using metaclasses in a way that makes it difficult to extend via subclassing!

I've been in search of the "holy grail" of plain text markup for years. Before the first django version of this site, I was writing a custom mod_python handler to run a server-page style site and as part of that writing my own wiki creole. Since then, I've used TracWiki, reStructured Text, Markdown, MediaWiki, and Textile but have been left slightly disappointed by them all.

If you go to the py7zlib page you will see a comment describing a problem with the timestamps. I was having a similar problem with the timestamps, except the comment didn't provide a patch or a way to convert time time. This problem exists in svn trunk as of this writing, although it was discovered a year ago. The issue is perhaps due to a slight fileformat shift, but essentially there is an extra byte (or 9) that are not being read and shifting the time results.

The past 2 days I've had a few perplexing technical problems that have had interesting answers. After a long weekend of drinking, eating, and lest I forget drinking, I got back to Cecilia's late last night and felt like doing a little digital maintenance on my computer, when I noticed that it was down. It's never down.

I've disabled comments because I have some neat captcha-less ideas for how to tell bots to fuck the hell off from my comments section but I don't feel like implementing them for django. The new system (mostly homegrown glue w/ selector, beaker, mako & werkzeug thrown in) has been in heavy development the past few weeks and commenting should be available by the summer. Sorry Mike; now you've no reason to come here.

I've been doing a lot with CouchDB and WSGI the past few months, with positive and negative results. I'm finally getting the "hang" of Document Orited Databases (DODB? Rubbish acronym), and starting to understand what is possible and what isn't, how to do one-to-many and many-to-many relationships without having lots of queries. It struck me that although the per-query time in CouchDB is thusfar a lot slower than relational databases, the overall database time for individual webpage loads is far less, and there are far less queries and complex information joins going on.

Happy Anniversary to me!

I've been thinking about WSGI and CouchDB recently, while on the subject of digital inflexibility. First, I want to clarify a few things about what I mean by flexibility with respect to an application, and how the current crop of frameworks approach this problem. If you want to follow this musing well, I highly suggest reading "What PHP Deployment Gets Right" by Ian Bicking; or just his entire blog, and most of the crosstalk on the web about REST, Web Services, and the evolution of the WWW.

I have been programming in Python for quite some time now, and I've been doing it professionally for over 2 years. Despite this, I am not nearly as proficient at the language as I could be, probably because I am using it professionally and have to devote time I could be using learning the language to solving problems.

Just "finished" hacking together some code that started out as a small desire and has ended up an obsession of sorts. Initially, it was just going to be a smart, specialized file renamer. It has ballooned into a poorly written (but fairly solid) beast of a program with crc checking and unrar/zipfile support. When I say it's poorly written, I mean it's not beautiful like my xdccq module is, for instance. It isn't elegant in the least, does things in a way that is acknowledged as poor design decisions, and the main dispatch is a giant ugly conditional mess. But it works pretty well!

I mentioned before that I had splintered off my blog developing efforts into a project I named saudade. The first somewhat releasable ball of code is now finished and in operation at jmoiron.net. This is a (thusfar modest) blog engine utilizing Django and focusing on levereging existing technologies, best practices, and a slim codebase. Some features saudade has over my previous software:

This one's a quickie. I promised to make available some changes I made to Gene Cash's EXIF.py library. I'm only providing them as a diff patch because ceache is working on improving the performance of the library extensively by using PIL's exif parsing code with EXIF.py's MakerNote deciphering code.

It started out as an insertcredit forum post, but the promise of extreme difficulty coupled with my complete amnesia about completing Super Mario Bros. 2 made me very interested. To throw some fun into the mix, and as a little academic exersize, I wrote the mario challenge! website for me and some friends to track our progress and win fabulous prizes.

I spent most of today investigating different ways to read EXIF data in Python for the up and coming gallery. A few days ago, as noted elsewhere, I spent a day investigating Mono and writing trivially simple GUI applications in boo. These were kind of liesurely activities, but their usage was immediately relevant to anyone.. even a non-coder. When my images are immediately available to everyone along with the ISO setting, shutter speed, and timestamp, normal people will understand. When I show my parents GUI apps that I have written, they understand its uses.

I finally understand apache, and what really happens internally when a request is made. I wasn't able to understand mod_python until today. Until the limitations of the server page model are ground against in such a way that you actually wonder, "There has to be a better way!" you aren't really ready to write your own Apache request handler. Even if it's totally fucking easy.

The average latency for blogposts has increased for me and for others, but the bandwidth has stayed about the same, regardless of what format I'd rather employ. The problem lately is that I detest my ¼ complete backend and the entire structure behind it; Jerm's decision, although heavily ridiculed by myself and yet others, was the right one, because he and I are of a particular breed of perfectionist that can either have everything absolutely perfect, or nothing being satisfactory at all. I am just not capable of doing small, incremental design changes; when I get into a process, I put my all into it, and when I back out, I forget about it and use that space for other things. I work best in batch mode, a-la Knuth.

I read the whole archive of Elf Only Inn the other day, mostly at the impersonal behest of Tycho, who's on comic has left me somewhat empty these days. This archive reading of course reminded me that I haven't read the Thin H Line/Sexy Losers in a while, so maybe I'll eventually get to doing that.

Why the hell can I not design context menu's in designer? The idea is so idiotic that I've spent 2 days trying to figure out how, because it's something I think just has to be there. I'm just missing it, or something, because I'm new. But after surfing freenode for a while and asking some questions there, I couldn't figure it out. It has become a major roadblock in the first steps I was about to take towards making my program actually do anything.

I am no GUI designer. I have new found respect for all of the people from whom I've consciously or unconsciously stolen the ideas that go into the screens coming up shortly. Actually, I always had respect for a well built program, but at this point I think I have more respect for poorly built programs: in order for them to get wherever they are, a lot of people had to think quite a bit.

I mentioned that I hated having to install Eric3 in order to compile the *.ui files that QtDesigner creates. I did some exploration and realized that Eric3 used "pyuic" (PYthon User Interface Compiler), which, wouldn't you know it, comes standard with PyQt. There goes one huge complaint I had about the whole system as well as 3 packages. I find myself thinking that PyQt did it right, and that libglade should develop a python generator for the glade UI files.

When I first started using python+gtk+glade/libglade, I didn't think that any other programmers had it better than me. I was still in a platform dependent stupor, a sublime state of anti-idiocy insanity where my own idiocy was incalculable and yet undetectable, but then again I had never had any experience developing Gui's even though simple tools were most likely available. My PyGTK Tutorials, available still in the docs section, are a testament to how much I enjoyed the system, as well as how much I began to understand it.

I've been trying out a few console only programs in the past week; a few programs that I had been meaning to try for some time. One of them was swaret, the SlackWARE Tool, and the other is Screen, the UNIX tool.

Ahh, its that time of year again, when the freshman orientation week ends, they get their laptops, and then fuck our connection right in the balloon knot. If this was last year at this time, only this year (bear with me), I'd still be fine with this, because I'd be downloading packages pertaining to whatever interest I have this week at lightning I2 speeds. Unfortunately, Stevens needed the valueble I2 network money to water the grass and plant flowers.

It is my contention that Lorde Omlette is a fake; a ninny with soiled knickers. Let me explain, please.

Its been quite a few days since I set out to rewrite this site in python, and so far, I have only constructed a minimal scaffolding where I want my great statue to stand. How the brilliant fire of fervor can die down to the smoking head of a recently lit match, blown out by the wind or put out of its misery by a single rain drop, is beyond me. How my mind that seemingly lives only in pursuit of its own destruction due to overactivity manages to latch on to something so completely only to let it go is also not yet within my understanding.

It would appear that submission is working, although in a non-final rudimentary form. Still, something had to be done, and even more will soon have to be done because the main page and other pages work off of different databases entirely. Unfortunately, there are no plans to get any of the functionality in the comments or rant section ported to the new system within the next 5 days; but I don't think that will kill anybody.

In an effort to keep the design of Jtex simple, and further reinforce that it is not competition for other projects, I've taken a few features out of the release plan. The project is more exciting to me as a tutorial to myself and others on building PyGTK applications rather than reinventing the text editor yet again. The removed features follow:

I can't seem to remember what it is that I wanted to have done by Friday night, but whatever it is, I probably don't have it done. The tutorial is still languishing on page 3, and although I didn't get much actual development done, I did learn quite a bit about how to use pango to change fonts in the TextView, among other things.

While thinking over what to include as features for the Jonas Text Editor Experiment, XML struck my fancy as a way of creating a simple properties interface that was as extensible as it was clean and easy to parse. And then XML took my fancy and beat the ever living shit out of it.

In an Ideal world, here is what the schedule looks like. I hope to have 0.2 complete by tonight, and 1.0 released by mid to late june. PySCP should start up heavy sometime early july, and I expect to have working scp code by mid to late august. There's also another project on the way... and I need to re-write my website to adjust for my newfound thirst for development.

Pango is most interesting in its infinite uses. Check out a Pango Tutorial if you are interested; using glade with pango can be very enriching. As soon as I find out how to find the windows and dialogs i create with gtk.glade.XML, I'll write something up on that. It would obviously have something to do with an event; in this case, I'm just connecting an about dialogue, so its going to be an onclick signal. Even so, I find glade an excellent tool for RAD, and the whole glade + python thing looks like it could be extremely powerful in well developed hands.

Now, part 2 of the old tutorial (the one that I was porting) runs and displays properly, with signals connected properly; but displays a warning that is just as useless as the previous one I was getting.

To Do:

Yesterday, Phil (firu) and I spent about 4 hours in dazed mixture of drowsiness, cleaning, and chilling. We managed to remove and organize the metric fuck-ton of shit that was blocking the windows, and ended up chilling on the radiator, right next to the windows, on the 3rd story. The newly opened windows (and newly opened line of sight to said opened windows) have revealed unto me a most amazing view of the rooftops of hoboken from my seat, and since their opening I've spent many an idle minute gazing out at the tree with the strange hairy seeds and cranes out near Jersey City.

My thumbs seem to be any color except green, a contributing factor to the apparent slow and painful death of my new foliage. They are, however, rather adept at writing compilers; as I was able to finally compile useful code today with my pascal compiler (fibonacci and factorial). Will it ever be completed? The world will never know.