Note to job seekers

When applying for a programming job, listing Dreamweaver as a skill is an automatic 50 demerits.

I'm going to be a digital librarian!

As of March 15th, I will be working for the UNC Library as a digital library programmer. I'm going to miss Lulu a lot. It's been a wonderful environment to work in, with people I'm going to find hard to leave. But working with collections like Documenting the American South is a text geek's Nirvana, so it was far too good an opportunity to pass up...

How has Ruby blown your mind?

...asks Pat Eyler

I had the opportunity to learn Ruby as part of a work project last year and was immediately impressed by its object-orientation, its use of blocks, the straightforward way it handles multiple inheritance with modules, and just the elegance and speed with which I could work in it. The moment that really changed the way I saw the language came when I had to generate previews of Word and OpenDocument (ODT) documents uploaded to the site I was working on. Converting Word to ODT seemed like the way to go, since ODT has a zipped XML format, and can therefore be transformed to XHTML. I have a lot of experience using XSLT to transform XML from one vocabulary to another, so this seemed like well explored territory to me, even if it would take a fair amount of work to accomplish. As usual, I did some web-trolling to see who had dealt with this issue before me, in case the problem was already solved. Google pointed me at J. David Eisenberg's ruby_odt_to_xhtml, which looked like a good start. It didn't do everything I wanted, in particular it didn't handle footnotes adequately, but I didnt expect it would be too hard to modify. The surprises came when I looked at the code...

The first surprise was the utter lack of XSLT. Not a huge surprise, perhaps. I'd already gathered that Rubyists viewed XML with a somewhat jaundiced eye. Tim Bray has lamented the state of XML support in Ruby as well. Tim is quite right about the relative weakness of XML support in Ruby, even though I absolutely agree with the practice of avoiding XML configuration files. There is a perfectly good Ruby frontend to libxslt, however, so it's use is not out of the question. But there it was: for whatever reason, the author had decided not to use the technology I was familiar with...why would he do that, and could I still use his tool?

The mind expansion came about when I started figuring out how to extend odt_to_xhtml to handle notes, which it was basically ignoring. I wanted to turn ODT footnotes into endnotes with named anchors at the bottom of the page, links in the text to the corresponding anchor, and backlinks from the note to its link in the text. Before describing what I found, I should give a little background on XSLT:

At its most basic, XSLT expects input in the form of an XML document, and produces either XML or text output. In XSLT, the functions are called templates. Templates respond either to calls (as do functions in most languages) or, more often, to matches on the input XML document. So a template like

<xsl:template match="text:p">
  <p><xsl:apply-templates/></p>
</xsl:template>

would be triggered every time a paragraph element in an OpenDocument content.xml is encountered and would output a <p> tag, yield to any other matching templates, and then close the <p> tag.

As I looked at JDE's code, I saw lots of methods like this:

def process_text_list_item( element, output_node )
style_name = register_style( element )
item = emit_element( output_node, "li", {"class" => style_name} )
process_children( element, item )
end

emit_element does what it sounds like it does, adds a child element to the element passed in to the method with a hash of attribute name/value pairs. It's process_children that really interests me:

#  Process an element's children
#  node: the context node
#  output_node: the node to which to add the children
#  xpath_expr: which children to process (default is all)
#
#  Algorithm:
#  If the node is a text node, output to the destination.
#  If it's an element, munge its name into
#  <tt>process_prefix_elementname</tt>. If that
#  method exists, call it to handle the element. Otherwise,
#  process this node's children recursively.
#
def process_children( node, output_node, xpath_expr="node()" )
REXML::XPath.each( node, xpath_expr ) do |item|
  if (item.kind_of?(REXML::Element)) then
    str = "process_" + @namespace_urn[item.namespace] + "_" + item.name.tr_s(":-", "__")
    if ODT_to_XHTML.method_defined?( str ) then
      self.send( str, item, output_node )
    else
      process_children(item, output_node)
    end
  elsif (item.kind_of?(REXML::Text) && !item.value.match(/^\s*$/))
    output_node.add_text(item.value)
  end
end
#
#  If it's empty, add a null string to force a begin and end
#  tag to be generated
if (!output_node.has_elements? && !output_node.has_text?) then
  output_node.add_text("")
end
end

Mind expansion ensued. This Ruby class was doing exactly the same thing that I'd expect an XSLT stylesheet to do, with the help of a few lines of code to keep it going! process_text_list_item is a template! Coming from Java and then PHP, I'd have no hesitation switching to XSLT to accomplish a bit of XML processing like this, but in Ruby, there really wasn't any need. I could write XSLT-like code perfectly naturally without ever leaving Ruby!

Now, I still like XSLT, and I'd still use it in many cases like this, because it's portable across different lanaguages and platforms. But here, where there are other considerations, it's wonderful that I'm not forced to step outside the language I'm working in to accomplish what I want. In order to extend the code to handle notes, I just added some new template-like methods to match on notes and note-citations, e.g.:

def process_text_note( element, output_node )
process_children(element, output_node, "#{text_ns}:note-citation")
end

In OpenDocument, notes are inline structures. The note is embedded within the text at the point where the citation occurs, so to create endnotes, you need to split the note into a citation link and a note that is placed at the end of the output document. To add the endnotes, I borrowed a trick from XSLT: modes. If an XSL template has a mode="something" attribute, then that template will not match on an input node unless it was dispatched with an <apply-templates mode="something"/>. So I did the same thing, e.g.:

def process_text_note_mode_endnote( element, output_node )
p = emit_element(output_node, "p", {"class" => "footnote"})
process_children(element, p, "#{@text_ns}:note-citation", "endnote")
process_text_s(element, p)
process_children(element, p, "#{@text_ns}:note-body/#{@text_ns}:p[1]/node()")
process_children(element, p, "#{@text_ns}:note-body/#{@text_ns}:p[1]/following-sibling::*")
end

The method that controls the processing flow in JDE's code is called analyze_content_xml. I just added a call to my moded methods in analyze_content_xml and modified process_children to take a mode parameter.

def process_children( node, output_node, xpath_expr="node()", mode=nil )
if xpath_expr.nil?
  xpath_expr = "node()"
end
REXML::XPath.each( node, xpath_expr ) do |item|
  if (item.kind_of?(REXML::Element)) then
    str = "process_" + @namespace_urn[item.namespace] + "_" + item.name.tr_s(":-", "__")
    if mode
      str += "_mode_#{mode}"
    end
    if ODT_to_XHTML.method_defined?( str ) then
      self.send( str, item, output_node )
    else
      process_children(item, output_node)
    end
  elsif (item.kind_of?(REXML::Text) && !item.value.match(/^\s*$/))
    output_node.add_text(item.value)
  end
end
#
#  If it's empty, add a null string to force a begin and end
#  tag to be generated
if (!output_node.has_elements? && !output_node.has_text?) then
  output_node.add_text("")
end
end

Done. Easy. Blew my mind.

Prototype grows up

http://prototypejs.org is the new site for Prototype 1.5. As the Ajaxian blog noted: Now with Documentation! Of course, Prototype always had some documentation; quite good documentation at that, even though there were substantial pieces missing and you had to go digging sometimes.

Prototype played a big part in reawakening my interest in Javascript as a programming language. I was rather anti-Javascript for a while, having fought many bloody battles with cross-browser incompatibilities in the early 2000's (UNC Chapel Hill, my then employer, had standardized, somewhat foolishly, on Netscape 4.7, but of course we had to support IE too -- nightmare). I got back into it seriously when I started to notice all of the AJAXy and Web 2.0-ish stuff going on. I've learned a lot from digging around in the Prototype source code, so the spotty documentation actually did me some good.

Kudos to the Prototype development team and the contributors to the documentation effort. You've done us all a great service! I look forward to using 1.5...

XSL-FO 2.0 Workshop 2006

International Workshop on the future of the Extensible Stylesheet Language (XSL-FO) Version 2.0

XSL-FO 2.0 Workshop 2006

I have two suggestions:

• use CSS instead of weird attribute-based style declarations.


• for God's sake, have a reference implementation.

Boycott Blackboard!

I knew Blackboard had a patent application for their LMS, but apparently it has been granted and their first act was to file a lawsuit against one of their competitors. This is terrible on many levels. Not least that such a stupid patent should never have been granted,. Of course, the USPTO would probably let me patent my nose hair.

I certainly won't be using Blackboard for my XML class in the Fall, and I'd encourage other instructors to drop it too.

Google Library

I just read John Battelle's post on the AAP's lawsuit. The comments are particularly interesting, with a couple of very strident ones criticising Google. I have a theory about how Google plans to justify their actions:

1. Libraries are allowed, under copyright law, to make a single copy of any work in their possession. This is called the Library Exemption. There is a nice outline of the terms here. The libraries themselves can't get in trouble for contracting with Google to do this for them, because they are receiving no commercial advantage from it. Google clearly is receiving a competitive advantage from it, BUT:
   
2. They may be able to make a good case for Fair Use, depending on the nature of what they keep from the book. There are four aspects to be weighed in any Fair Use defense (see Wikipedia):
1. the purpose and character of the use, including whether such use is of a commercial nature or is for nonprofit educational purposes;
2. the nature of the copyrighted work;
3. the amount and substantiality of the portion used in relation to the copyrighted work as a whole; and
4. the effect of the use upon the potential market for or value of the copyrighted work.

Clearly Google hopes for commercial advantage from the use of the scanned books, so they might fail the first test. The second doesn't really apply: these are clearly books subject fully to copyright law. It's the third and fourth aspects that I think are the center of Google's defense. A copy is a copy, but a searchable index created from a scanned copy is arguably a transformative use of the book. A human being can neither read the index, nor reconstruct the original from it, so Google may be able to successfully defend themselves on aspect #3. Their main weakness is the existence of page images from the original scan. These may or may not be stored and accessible in such a way that a whole copy of the original could be reconstructed and read. Aspect #4 is another winner for Google. The clear effect of this system will be to sell more copies of the publishers' books. The only (theoretical) commercial harm caused to the publishers is that they are effectively prevented from rolling a Google Print of their own, which might bring them in more money than simply selling their books. So Google wins on at least two of the four counts, and the act of copying itself is protected under the Library Exemption.

I suspect the AAP would have an uphill battle in winning this one. I wouldn't be surprised if they wanted Google to license their books for their index at some fairly exorbitant rate, and Google refused to pay because they're doing the publishers a favor. That would make the lawsuit a negotiating tactic.

When SEOs Attack

Search Engine Foo: iUniverse Book Publishing: Book Publisher for Self Publishing and Print on Demand. Care to guess what terms they're optimizing for? It does seem to work. They show up #1 for a Google search on "self publishing." So clearly this sort of spamming works. But it leads to pretty hilarious prose:

iUniverse, the leading online book publisher, offers the most comprehensive book publishing services in the self-publishing industry—awarded the Editor's Choice award by PC Magazine and chosen by thousands of satisfied authors as the leading print-on-demand book publisher.

We help authors to prepare a manuscript, design and self-publish a book of professional quality, publicize and market their book, and print copies of their book for sale online and in bookstores around the world.

As an innovative book publisher, we also offer exclusive services such as our acclaimed Editorial Review and our revolutionary Star Program, designed to discover and nurture exceptional new talent within our growing author community.

Don't wait any longer to get that manuscript off your desk and into the marketplace. With iUniverse as your book publisher, you can become a published author in a matter of weeks. Why not get started today?

Yes, indeed. Publish your book with a publishing publisher and be published. Ouch. Not sure I'd pay them an exorbitant fee to edit my book.

Writing Code

I'm coming to the conclusion that writing code, as an activity, really is like writing prose. I find myself treating code projects just like writing projects:

1. I spend the first part of the project thinking about it and being (apparently) very unproductive. (25-30%)
   
2. After I reach some sort of critical mass in my thinking, I very quickly pour out everything into code/onto the page. The project is 80% done as far as volume goes at this point. (10-20%)
   
3. I spend the rest of the time editing, bugfixing, refining, etc. (50-60%)

For larger projects, this cycle gets repeated for each component of the project. This is precisely the pattern I followed when writing my dissertation. I don't know if this kind of working method is in any way typical, but it does seem to produce the desired results. It makes giving project completion estimates next to impossible though, because I really have no idea how long the project will take until I enter the hyper-productive phase, and when that's complete, I often still have a lot of work to do, even though the bulk of the code/writing is done.

This is why it's best for me if I've got some variety in a job. The hyper-productive phase really can't overlap with anything else: if I'm interrupted then, I'll get off track and it may blow the whole day, but in phase 1 or 3 I'm better off not spending all my time focusing on the project, because I'll just end up web surfing. Or blogging.

Tagging Notes

From a conversation this afternoon: the tags used in folksonomies are deliberately stupid. They are atomic units of information. So a tag can be any atomic unit--it doesn't have to be a word, it could be a URL, a zip code, or anything else you can think of that isn't reducible.

adaptive path » ajax: a new approach to web applications

adaptive path » ajax: a new approach to web applications. Web application development is starting to get really exciting again. The funny thing is that a lot of this technology has been around for a while, and even though IE supported it, you didn't see tools like these. I wonder whether the development of Firefox is really what's pushed it. Certainly nearly all the developers I know shun IE. So perhaps having a capable Free browser was what sparked all this innovation.

Blog-binding

Recently, I've been seeing a number of companies and projects springing up around the idea of publishing blogs as books. The examples I'm aware of are Blogbinders, qoop, LJBook, and most recently, book this blog, but I'd bet there are more. What I'm wondering about is how useful the blog-directly-to-book pathway is. Wouldn't an application that aggregates your blog posts into an editing environment (like Word or OpenOffice) be more useful? Can we really smooth over the formatting differences between web and print well enough to produce (automatically) a nice-looking book 100% of the time? I'm a little skeptical.

From my (admittedly cursory) browsing, it looks like blogbinders have a human in the middle of the process, and qoop certainly did for their only title so far, John Battelle's SearchBlog. Requiring a human being in the loop raises costs and introduces scaling issues.

There's a fine tradition of publishing diaries, going back at least to Caesar, but unless you happen to be famous, or have a blog that's truly interesting a high percentage of the time, the way to monetize blogs is more likely to be on the Hardball Times model, where you build up an audience, and then sell them work they're interested in.

But if blogs really are the ultimate vanity presses, then there may indeed be money in printing them, if you charge the blogger enough up front. It will be interesting to see how all this shakes out.

Reflexive link

Just noted on my work blog the Wired article by Hilary Rosen on Creative Commons. Always nice to see a new convert.

First Post

Just trying this again. Oddly, Camino wouldn't let me type in the text area, but Firefox will. Curious....