First, background: A “Sitemap” is a file or set of files that list the URLs of pages in a web site that you want to be sure get indexed by search engines. You can also use them to provide hints that help page indexing, like when a page was last changed, an anticipated average frequency of change, relative “priority” (importance) of content, etc. Sitemaps are simple XML files that will be fetched from a site by search engines as part of their spidering process, and while they could be pre-generated and then served as a static file, it seemed to me that a Rails web app ought to generate its own on demand.

Moving to implementation, while you can theoretically choose any name and location for your sitemap file, it really isn’t practical. In particular, URLs pointed to from a sitemap must be contained in the same directory or a sub-directory on the server where it is located. So, if your Sitemap is going to describe your whole site, it has to be served from the root of the server’s path hierarchy. Which means that you can’t use a RESTful route. So, we add the following in config/routes.rb:

  map.sitemap '/sitemap.xml', :controller => :admin, :action => :sitemap

This maps request for the Sitemap (called sitemap.xml, which is the default name that search engines will look for if you don’t explicitly specify a name in your robots.txt file or through a specific engine’s web UI) to the Rails controller admin. (I’m not the only one who has a catch-all controller for all the odd stuff in their Ruby-on-Rails application, right?)

Next is the actual implementation of the sitemap action. For this example, I’ve assumed that we’re build a sitemap for the comment model that we’re all familiar with from the Rails Getting Started Guide.

  def sitemap
    result_text = <<PLAIN_XML
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>#{comments_url}</loc>
    <changefreq>monthly</changefreq>
    <priority>0.2</priority>
  </url>
PLAIN_XML

    Comment.all.each do |comment|
      result_text += "  <url>\n"
      result_text += "    <loc>#{comment_url(comment)}</loc>\n"
      result_text += "    <lastmod>#{comment.updated_at.to_s(:db) }</lastmod>\n"
      result_text += "    <changefreq>monthly</changefreq>\n"
      result_text += "    <priority>0.5</priority>\n"
      result_text += "  </url>\n"
    end

    result_text += "</urlset>\n"
    render :text => result_text
  end

The first few lines, dealing with the here-document PLAIN_XML generate the fixed file header for the Sitemap, as well as the fixed block that points to the index action (page) in the comment controller. For my sitemap, I didn’t bother trying to figure out a modification time for it, and assume that the content of the index page will be a less-good match for any search query than the page for a specific comment (comments/show).

Then the block starting from Comment.all.each loops over all the comment model instances stored in the database, and generates the XML package that describes the URL for each. The XML schema is very simple: each URL is represented by the content of a url tag, the loc tag contains the URL itself, and the remainder of the tags within the url provide metadata associated with the URL.

The last modification time value, which goes into the lastmod tag, is generated by Rails’ automatic model (database record) update time tracking widgetry. If you aren’t already using it, you simply have to add a datetime column named updated_at to your database schema, and Rails will take care of the rest. (If you’ve got a running system, make sure your migration fills in an initial value for the new updated_at column or you ensure your code handles nil updated_at values.)

Finally, the end of the method generates the closing XML tag for the sitemap, and then outputs everything. It’s important to use render :text to avoid having the intended content of the sitemap wrapped inside of one of your application’s Rails layouts.

So, there it is, nothing to it. Didn’t take any longer to write the code than it did to write the post to this point. Alas, the code is wrong. It turns out that, once you’re past the year/month/day part of the modification time, the date format generated by :db isn’t compatible with the spec for a Sitemap, which requires a W3C Datetime date. Now a quick web search will turn up a couple of things that are related. First, from http://swik.net/XML/del.icio.us%2Ftag%2Fxml/Ruby+W3C+Datetime+Format+(Google+Sitemaps+with+Ruby+on+Rails)/b1hvy, comes the snipet:

  def w3c_date(date) date.utc.strftime("%Y-%m-%dT%H:%M:%S+00:00") end

Which works just great, but as a method, putting it into the above code block is kind of ugly. And I seem to be willing to invest an inordinate amount of time to avoid “ugly.” Then, from the Rails RDoc comes:

Module ActiveSupport::CoreExtensions::Date::Conversions

Adding your own time formats to to_formatted_s

You can add your own formats to the Date::DATE_FORMATS hash. Use the format name as the hash key and either a strftime string or Proc instance that takes a date argument as the value.

    # config/initializers/time_formats.rb
    Date::DATE_FORMATS[:month_and_year] = "%B %Y"
  

And that ought to be that, right? Just stuff the format string from one into the DATE_FORMATS hash and everything should be fine. But no. The trick, which took twice as long to figure out as implementing everything else, is that Date::DATE_FORMATS is the wrong hash. That may be the right hash for classes that have the ...Date::Conversions module mixed in, but not here. The updated_at attribute returns a Rails TimeWithZone object, which is no surprise given that this is Rails’ preferred time class. What is a surprise, revealed only after you read the source for TimeWithZone is that the hash that it uses for formatting is Time::DATE_FORMATS. So, in my project, I’ve got the file config/initializers/extensions.rb that I use for little Rails customizations like that, and it now includes:

  Time::DATE_FORMATS[:w3c] =
    lambda { |date| date.strftime( "%Y-%m-%dT%H:%M:%SZ" ) }

Which lets me generate the lastmode tag’s content with:

  result_text += "    <lastmod>#{comment.updated_at.to_s(:w3c) }</lastmodd>>\n"

Which is clear, and pretty, and if enough of you read this post, might even be worth the time it took for me to figure it out.

Oh, one more thing: I know pretty much nothing about time-zone handling in Rails. The Sitemap format wants last-modification times in UTC, and the format string constant I include above states that the time is UTC (the ending “Z” character), and the above works for me. But, perhaps it wouldn’t work for someone that had different time-zone settings in Rails or their database. Anyway, if this produces timestamps that seem off to you, change the content of the lambda to “date.utc.strftime()” and all should be well.

Lori Garver

(This post is for Ada Lovelace Day, March 24th, a day to commemorate women in technology and science. You can find a List of other ALD posts at FindingAda.com, and a community-contributed listing of notable women in technology and science at AdasDaughters.org, including the entry I made for Lori Garver, which includes some links I didn’t get to use here.)

Lori Garver is the Deputy Administrator of NASA, and one of the shapers of NASA’s change in direction from the too-slow and too-expensive one-shot “Constellation program” toward fostering a sustainable industrial base that will provide for ongoing and continuous manned space missions within the budget NASA has historically received.

Garver rejoined NASA as Deputy Administrator in July of 2009, after leading the Obama Administration’s NASA “transition team.” Within NASA there is only one Deputy, a position analogous to a corporate CEO (in comparison to the Administrator which is currently similar to a commercial President/Chairman). NASA says:

As deputy administrator, Garver is NASA’s second in command. She is responsible to the administrator for providing overall leadership, planning, and policy direction for the agency. Garver represents NASA to the Executive Office of the President, Congress, heads of government agencies, international organizations, and external organizations and communities. She also oversees the work of NASA’s functional offices, including the Office of the Chief Financial Officer, Office of General Counsel and Office of Strategic Communications.

Garver’s work on space policy began after she was employed in the John Glenn Presidential campaign (1983-84). Later she started with the American “National Space Society” at its founding, and served as its second Executive Director from 1987 to 1996. She then worked in NASA’s administration, finishing as Associate Administrator of the Office of Policy and Plans, and leaving NASA in 2001. Between ending her first tenure at NASA and rejoining as an Obama appointee, she worked in a series of space policy and consultancy organizations (Capital Space, Avascent Group, and DFI Corporate Services), was an active advocate for space exploration, and served as the space policy advisor to the John Kerry, Hillary Clinton, and Barack Obama Presidential campaigns.

And somewhere amid all that, she also worked to assemble a sponsor-funded private trip to the International Space Station, served on the NASA Advisory Council, was a guest lecturer at the International Space University, was both president and board member of Women in Aerospace, and was president of the American Astronautical Society.

When I first heard about Ada Lovelace Day, I immediately wanted to write about Lori Garver, although I’ve neither met her nor been fortunate enough to hear her speak in person. As I started researching for this piece, I realized that I didn’t know exactly why she resonated so much with me. (She isn’t even the first woman to be NASA Deputy Administrator. That was Shana Dale, her immediate predecessor.) But I think I’ve figured it out, and it has to do with both our agreement on technical and policy issues, and on how what she’s gone through in public the last few years mirrors the struggles I’ve seen in female colleagues.

Campaign ’08

After serving as the space policy advisor to the John Kerry campaign in 2004, Lori Garver held essentially the same position in Hillary Clinton’s campaign in 2007/08 until Clinton was defeated in the primaries. In May of 2008, she spoke for the Clinton campaign in a “debate” between representatives of Clinton, Obama, and McCain (The so-so space debate). After Clinton’s loss, Garver became the Obama campaign’s space advisor (they hadn’t really had one previously, which had been evident in May). In August, another debate-of-proxies was held, this time with Garver representing Obama, and again she “won” (Space policy heats up this summer, Who’s advising McCain on energy and climate?).

I don’t know how much of the similarity between the Kerry, Clinton, and Obama space policy positions was due to the candidates own similarities which then caused them to select someone that agreed with them, or to the fact that Garver presumably gave them all similar (sound) advice. In any event, the cores of the different policies were similar both to each other and to my own views: NASA is important, but it’s unrealistic to imagine it getting significantly more resources in the future than it has in the past (after Apollo). Manned space flight is important, but unless costs are reduced, it is unlikely to be more frequent or to go farther than the Space Shuttle did. NASA may be a federal government agency, but that doesn’t mean it’s acceptable for the number-one concern to be maximizing the number of Congressional districts in which it spends money.

In short, Garver advocated for a space program that would actually do interesting, worth-while, and exciting things. As opposed to the program we’ve had for decades, which operated as if it’s primary goal was to maintain NASA’s “workforce” and cost-plus contractors.

Obama Presidential Transition

She has been underestimated many times, but has maintained a laser-like focus in her determination to affect change in the U.S. space program.

As part of the “transition” effort, starting after his election and continuing into the first few months of his administration, Obama had fact-finding teams visit virtually all of the agencies and departments of the federal government. Lori Garver headed the small “Agency Review Team” that was responsible for assessing the current state of NASA and making recommendations to the Presidential transition.

At the beginning of President Obama’s term, NASA had been working on the Constellation Program for three and a half years. “Constellation” was a combination of new rockets and spacecraft with a plan for how they would be used for flights to Earth orbit and subsequent Moon landings. It was formulated under NASA Administrator Michael Griffin in order to implement George W. Bush’s NASA policy, the “Vision for Space Exploration” (itself conceived under Griffin’s predecessor, Sean O’Keefe). Setting aside the question of whether Constellation was or was not a faithful execution of The Vision, by the time the new administration was coming in, it was clear that NASA was not receiving the budget increases that had been assumed when Constellation was planned, that it was suffering from the same organizational failures that had increased the cost and reduced the performance of the Space Shuttle and International Space Station, and that the hardware systems it was developing would have neither cost nor capability improvements over the Space Shuttle or the Apollo-era Saturn V.

Given all of this, and his own hopes to remain as NASA Administrator under President Obama, Griffin responded somewhat defensively to Garver’s team’s efforts to ascertain the real state of affairs within NASA. As reported in NASA has become a transition problem for Obama:

NASA administrator Mike Griffin is not cooperating with President-elect Barack Obama’s transition team, is obstructing its efforts to get information and has told its leader that she is “not qualified” to judge his rocket program, the Orlando Sentinel has learned.

In a heated 40-minute conversation last week with Lori Garver, a former NASA associate administrator who heads the space transition team, a red-faced Griffin demanded to speak directly to Obama, according to witnesses.

And this is one of the reasons I wanted to write about Lori Garver. Not just that I admire the work she’s done and positions she’s taken, but that her experiences underline both what has improved, and what has not, in the way our culture treats “women in technology and science.” Now, it is clear to me that Griffin would have opposed the efforts of any transition team, whether led by a woman or not. But the form of that opposition in this case is identical to what I’ve seen repeatedly happening to women I’ve worked with. Time and again I’ve seen people duck the merits of a technical argument and retreat into attacking a woman’s right to be in the argument in the first place. Especially in the case of women who find themselves “in technology” without having the proper pedigree and certificates. After all, Garver’s degrees (yes, more than one) are “only” in Political Science and in Science, Technology and Public Policy. With credentials like that, and having only worked in space policy for 25 years, how could she possibly be qualified to review the progress of a government technology program?

As disappointing as it is to see this kind of thing still happening, overall I think that Garver’s role here is a positive sign: she got and carried out the assignment, she clearly influenced Obama and the transition team, and in the end I believe that Griffin’s behavior toward her and the transition process are part of the reason that he did not continue on as NASA Administrator. And there’s plenty of coverage on the web that looks at it this way ([1] [2]). But it is still disappointing. And, corners of the Internet being the cesspools that they are, it can still be really disappointing:

Obama is flooding NASA with annoying & incompetent women and men

Is Mike Griffin supposed to pretend that he believes that Ms Lori Garver can tell him whether it is a good idea to replace some rockets by Ares I or by the Orion crew capsule? Isn’t it a little bit over the edge?

(I’m not including a link to this and some other quotes below on purpose. I’m sure Google will help you out if you’re masochistic enough to want the full versions.)

The Obama NASA FY11 Budget

“We’ve been trying to relive Apollo for 40 years, unsuccessfully. For too long NASA over-promised and under-delivered, and now we will be doing things differently.”

At the conclusion of the NASA transition, President Obama nominated Lori Garver as NASA Deputy Administrator along with ex-astronaut Charles Bolden as Administrator. And in the interim, the Obama Administration had chartered the Review of Human Space Flight Committee, which identified options for NASA including several alternatives to continuing with Constellation. (During hearings Jeff Greason, one of the Committee members, concluded “If Santa Claus gave us [the Constellation] system fully developed tomorrow, the first thing we’d have to do is cancel it, because we can’t afford to fly it.”)

So, I was pleased and hardly surprised when the Obama administration’s proposals for changes at NASA, which were announced as part of the overall 2011 budget proposal, called for shutting down Constellation and having NASA procure flights to space (crewed and cargo) competitively from commercial enterprises. But many in NASA and Congress, unable to conceive of non-business-as-usual at NASA, were surprised. And many senators and representatives from “the Space Coast” reacted to save their astro-pork. In particular:

Senator’s attack on NASA deputy chief Lori Garver backfires

Vitter accused Garver — who was not present at the hearing — of orchestrating the cancellation of Constellation. He also seemed to suggest that Garver was running the agency, and not Administrator Charlie Bolden. Bolden later called Vitter’s comment “unfair.”

Not only were administration officials outraged by Vitter’s remarks but several female civil servants and women executives in aerospace companies who have known Garver for years felt compelled to send their complaints to Senate staff Wednesday afternoon.

and

Attacks on Lori Garver Backfiring

Garver has become a target of critics of the proposal. Homer Hickam, author of October Sky and the subject of the motion picture Rocket Boys, called Garver a “gadfly” who should resign. He said nothing about Bolden.

But comments on blog posts seem to always be more vitriolic than the original content. This is from a comment posted in response to the above blog:

“The problem with Lori Garver is that she is not qualified for the position that she is in. The fact that the recommendations she makes are given the weight that they are is going to be very destructive to the ability of the United States to get to LEO and beyond.”

As with Griffin earlier, there’s plenty of support for and defense of both Garver and the changes at NASA from (what I consider to be) the reasonable side of the argument ([3] [4] [5]). As well as more sludge. In the comments posted on a newspaper article:

A journalist with any ounce of integrity and interest should investigate NASA Deputy Administrator Lori Garver’s unseemly and unethical connections to the very space start-up companies she’s now benefitting by cancelling Constellation programs. They once funded her quest to become a Russian cosmonaut. She’s a political appointtee with a dearth of actual space experience. She should be fired for recommending the nuclear-like damage to our NASA manned space program for the next two decades, allowing the Russians, Chinese and India to take-over space pre-eminence. Yea, thanks a lot Obama. Hope and change, allright.

On the one hand, this is just another case of Griffin’s attempts to avoid an argument he can’t win by attacking his opponent personally, albeit on a grander scale. On the other, I think the fact that the attackers ignored both Obama and Bolden, both theoretically “more responsible” for the changes than Garver, is telling. It is another sad fact about today’s culture in America that it has become common to argue against a policy by attacking the character of someone associated with it. And while it can’t be the way that anyone wants to have their importance validated, if Garver weren’t perceived to be playing a significant role in shaping NASA’s future, she wouldn’t be worth attacking. So, while the grounds her attackers are using are far too familiar and unquestionably chauvinistic, I think that it’s a good sign that she’s in the game. And it’s an even better sign that so far, she’s winning.

But since then, I’ve been frustrated at how slowly my pages have loaded. Both Amazon and Google ads integrate into web pages by having “the publisher” (you and me) include a block of HTML/JavaScript that they generate into our pages at the locations where the ads are supposed to appear. Unfortunately, when most browsers run into a tag that tells them to fetch a remote JavaScript file in the middle of the page, they stop laying out that page on the screen until they’ve finished fetching the requested JavaScript. So, even though my pages are relatively small and download quickly, only a tiny bit of them was actually being drawn to the screen before the browser tried to fetch an ad. It was especially bad in pages where I’d tried to “float” an ad into the upper-right corner: the screen would blank (my page had been received from my server) and then not a single character of text would be shown for the half-second or second it took for the ad to be fetched. And to compound problems, a page containing multiple ads would be blocked in its rendering separately for each ad—the browser wouldn’t fetch them in parallel.

So, eventually I got frustrated, decided that ads wouldn’t do me any good if their delays drove people from my site, and got down to the business of trying to restructure things so that I wouldn’t pay the ad-fetch penalty until my page’s actual content had been displayed. This turned out to be trickier that I’d planned. I spent quite a while messing around with techniques where I’d cause the ads to be downloaded by adding their HTML tags to the page’s Document Object Model (DOM) only after the basic structure of the page was done. I got several versions of this that almost worked, but it seemed that there were scoping problems caused by the difference between a JavaScript node being inserted into a web page after the fact rather than being parsed on the first pass through, and though they downloaded, I couldn’t get either Amazon’s or Google’s ad code to run when inserting it into the page “late.”

Eventually, I found a trick that did work: putting the ad code in HTML <div> blocks at the very end of the page. Of course, this isn’t where I actually wanted the ads to appear, so I also put empty <div>s at the places within my pages where I did want the ads to appear. Also at the bottom of the page, right between the actual ads and the </body> tag, I included a little bit of JavaScript that takes each ad, removes the DOM object representing the <div> that contains the ad from the page, and then re-inserts the ad’s div object within the empty place-holder div that I’d put earlier in the page. With a tiny bit of polish (setting display:none on the ad-loading divs, and setting the height and width on the place-holder divs so that the page wouldn’t re-layout when they were moved up), I had a solution that had my pages being displayed in full as quick as they were before the ads were included, and with the ads popping into view even quicker.

As for the details, I ended up with a single standard piece of JavaScript for moving ads around within a page, regardless of how many there are or where they’re located:

    function moveAndDisplayDelayedDivs(){
      if (typeof divsToMove !== 'undefined')
	for (var c =0; c < divsToMove.length; c++){
	  var targetDiv = document.getElementById( divsToMove[c] );
	  var divWithAd = document.getElementById( divsToMove[c] + '-content' );

	  // remove the ad div from the main document
	  var parent = divWithAd.parentNode;
	  parent.removeChild( divWithAd );
	  // put the ad div into its target location on the page
	  targetDiv.appendChild( divWithAd );
	  // and make it visible
	  divWithAd.style.display = 'block';
	}
    }

This code relies on a couple of conventions. First, somewhere earlier in the pages, there must have been code that made divsToMove an array containing the IDs of the HTML div tags into which the ads must be moved. To avoid having to have to arrays, I established the convention that if id="foo" is the div where an ad is ultimately supposed to be displayed, then the div containing the third-party markup for that ad would always be id="foo-content". After that, it’s a straight-forward bit of DOM manipulation to move the -content divs up the page into their intended display locations.

I chose to build up the divsToMove array incrementally, making one addition to it in each place in the page where I wanted an ad to be located. So, I made sure the following trivial function is defined at the top of my pages:

    function addToDivsToMove(divId){
      if (typeof divsToMove === 'undefined')
	divsToMove = new Array();
      divsToMove[ divsToMove.length ] = divId;
    }

And then the declaration of a point that will eventually display an ad looks like:

    <div id="google-ad" style="width: 160; height: 600px;"></div>
    <script>
      addToDivsToMove('google-ad');
    </script>

With the third-party ad code itself appearing near the bottom of the page like this:

    <div id="google-ad-content" style="display: none;">
      <script type="text/javascript"><!--
	google_ad_client = "my publisher_id";
	google_ad_slot = "my page_slot";
	google_ad_width = 160;
	google_ad_height = 600;
	//--></script>
      <script type="text/javascript"
	      src="http://pagead2.googlesyndication.com/pagead/show_ads.js" >
      </script>
    </div>

The one thing that is worth repeating/noting is that, in order to avoid the page’s layout being changed when the browser reaches the bottom of the page and executes the code to move the ads up to their final locations, the divs that the ads are going to be moved into need to have their size explicitly set to match the ad. I don’t find this to be problematic, as both Amazon and Google require you to choose from a handful of fixed sizes for their advertisements, anyway. So, as long as the size is fixed anyway, it’s only a minor inconvenience to have to repeat it between the first div‘s declaration and the third-party markup.

And that’s that. But, I have the great good fortune to be implementing my web application in Ruby-on-Rails. So, while people unlucky enough to be putting their page templates together by hand might have a problem making sure that the “moving parts” of the two divs that the ads are split between match up, I can actually put those parts in the same source file right next to each other, and depend on Rails to separate them out for me. Below is the equivalent source for including an Amazon “Omakase” ad block into my Rails-generated pages:

    <% unless @this_is_non_information_page or ENV['RAILS_ENV']=='test' %>
      <% if WontoMedia.ads.amazon and
	    WontoMedia.ads.amazon.associate_id %>

	<div id="block-amazon-ad" class="box"
	    style="width: 99%; height: 600px; text-align: center;"></div>
	<script>
	  addToDivsToMove('block-amazon-ad');
	</script>

	lt;% content_for :last_bottom_page_js do %>
	  <div id="block-amazon-ad-content" style="display: none;">
	    <script type="text/javascript"><!--
	      amazon_ad_tag = "<%= WontoMedia.ads.amazon.associate_id -%>";
	      amazon_ad_height = "600";
	      amazon_ad_height = "160";
	      //-->
	    </script>
	    <script type="text/javascript"
		    src="http://www.assoc-amazon.com/s/ads.js"></script>
	  </div>
	<% end %>
      <% end %>
    <% end %>

(Note that this code comes from my live project, WontoMedia, and is licensed under AGPLv3.)

There are several bits of Rails coolness that this takes advantage of or relates to. First, this blob of code and markup is the sole content of a Rails layout partial, so I can incorporate an Amazon ad into any location of any of my pages by rendering that partial into the desired display location within a page layout. Second, Rails allows multiple independent streams of information that will eventually make up a page to be composed in parallel. Most people don’t take advantage of this, and all their templates are composed together linearly in the order in which different parts include each other. However, by using content_for and having a matching yield in my top-level layout, this one file can generate HTML that is part of Rails’ default content flow (the div that the ad will eventually display in when it’s loaded) and part of the flow I’ve labeled last_bottom_page_js. (Yes, I do have other ..._bottom_page_js flows, so what?)

I also use the embedded-Ruby (ERB) that Rails provides in page layouts and partials to:

• manage which of my pages actually have ads displayed (only those that don’t have the Ruby variable @this_is_non_information_page set for them)
• suppress the inclusion of ads when my pages are loaded during unit testing (yes, the ad-loading delays were a pain when visiting my site, but adding a few seconds per page to a few hundred pages loaded during regression testing nearly made me walk off a pier)
• allow for customization of my web application:
  1. The WontoMedia object is a Ruby OpenStruct which I use to hold configuration information for a particular installation of my web application. When I install my app on sites I run, I fill in the advertising config info with the values associated with my Amazon and Google accounts. If someone else installs WontoMedia, they can run it with no advertising (simply by not including these constants in their config file) or set it up to point to their own advertising accounts.
  2. The actual configuration values are then inserted into the boiler-plate code provided by the ad vendor using Rails’ ERB notation for including the result of evaluating a Ruby expression into the output page: <%= expr -%>.

All-in-all, a relatively satisfying solution to my problems with slow page loading due to external ads.

Before I’d paid attention to test_helper to the extent that I’ve added stuff to it, and knew I needed to require it at the top of those files that used my additions. But back when I started with Rails, figuring out the magic included in the boiler-place created in test_helper by scaffolding wasn’t my most pressing issue. I mostly stopped thinking about it, and all of my tests have been just running, and all was well.

But then I tried running my project’s automated tests on a new system. And they just wouldn’t.

At first, I looked in lots of the wrong places, because this wasn’t the first time I’d moved my project between systems, and platform issues weren’t my first suspects. I still don’t know what caused things to be reliable on my first development systems, but it turned out that the difference that caused my problem to be revealed was deep under the hood in rake: on my original system, rake chose to run my tests in an order that coincidentally placed one of the files that did have require 'test_helper' first in its execution order. And because all of the tests that rake executes run within the same Ruby interpreter, all of the subsequent tests get the benefit of that one original require. And I continued to get lucky in running my project’s tests on different systems, right up until the last one. On that one, rake picked a different order to execute the tests—an order in which a test that I left the require out of was run first. And it failed. Duh.

Of course, ActionController isn’t the only “uninitialized constant” error you might get if you’ve had this particular brain-fade. The class named could be any Rails class, but if you’re having problems running tests, then they’ll likely by a Ruby or Rails base test class like Test, ActiveSupport, ActionView.

I’m a bit embarrassed to admit to this error, to having gotten this far along without understanding that test_helper was the sole link between my tests and the rest of Rails. On the other hand, if someone else had written this post and Google had found it, I’d have solved my problem in a few minutes instead of the three hours that it did take. So, here it is, and my best to you if you’ve happened onto this because you’re having the same problem.

Types of Free and Open Source Licensing

Bruce Perens argues that there are basically only three types of Free or Open Source software license:

• the gift license — essentially any non-copyleft free license, giving others the right to modify, redistribute, and incorporate in any context
• sharing-with-rules — is Perens’ term for copyleft licenses, emphasizing that they effectively create a collaboration between original author and modifiers
• the “in-between” license — a category that encompasses the GNU Lesser General Public License, which was created to provide copyleft licensing for the body of a reusable software component, while allowing that component to be incorporated into other, non-copyleft, software.

As an example, the most popular gift license is the BSD License. And for the balance of this post, I’ll ignore the distinctions between the non-gift license types (e.g., GPL vs. LGPL). Both of these licenses serve to mandate a population of re-publishing modification authors, which is the significant difference between copyleft and gift licenses.

While the original authors of free software are certainly members of the same community as other authors, users, and “modifiers”, they continue to have a special legal relationship with their creations. The author (technically, the copyright owner) of a piece of software is not bound by the license terms that apply to others. An author of software released under a copyleft license could keep some modifications private, can re-release the software under another license, and can sell non-free licenses to the software if they choose.

A number of companies based on the development of free software use their special relationship with the software’s copyright as a key element of their business models. These companies use “dual licensing”, where the software is made available simultaneously under different licensing terms to different audiences. Most commonly, the source is publicly released using a copyleft license, but the owner also makes another license available for a fee, that doesn’t require the licensee to republish modifications. A variation of this is where the copyright owner does not publish all of the software they have available, keeping the source code for a few “value added” features private. Then the commercially-licensed (fee-based) alternative to the free software version of the package may be valuable to customers who don’t plan to make their own modifications.

In a dual-licensing case, the people using the free software version/license to a package benefit by not having to pay the fee, and the company benefits by being able to incorporate modifications published by others into the free version. Companies paying for private licenses to the package can also benefit from the availability of the free-software version: the overall quality of the package is typically higher than a closed-source alternative, and the license cost is typically lower. (Lower license costs are enabled by the contributions of the free software community to the package, and enforced by the fact that any sufficiently-motivated developer could create their own implementation of privately-licensed features.)

A World Without Copyleft

If copyright were abolished without the simultaneous creation of a new legal basis for copyleft licensing, the software industry would be shifted into an environment where all of the published source code in the world was effectively “gift licensed”. By comparison, today 65%+ of the open-source software in the world is licensed with a GNU Public License, with perhaps 75% being under some type of copyleft license, GNU and non-GNU combined. I also believe that copyleft licensing’s effect on the community is even higher than would be expected from that percentage. Given the percentage of key software infrastructure components and tools (from the Linux kernel and GNU operating system on down) that are used via copyleft licensing, it is unlikely that there is a software developer in the world who isn’t aware of the terms of copyleft or who uses copyleft’ed software.

So, in a world without the ability to enforce copyleft, what would change? First, those companies who rely on dual-licensed software for their economic viability would gradually expire. No copyleft enforcement would immediately remove the incentive of all customers who are paying for the privilege of not having to republish their modifications to a piece of software. The profitability of licensing proprietary add-on features would follow soon after, first for those packages written in interpreted languages, and then the rest (as DRM for computer software is notoriously brittle and expensive now, even with the underlying support of copyright law).

The next obvious problem is that the entire corpus of free software would become vulnerable to Embrace-and-Extend. Without the requirement to republish modifications, the fragile balance of competition that has grown up between the free software community and (especially large) closed-source software publishers would be disrupted. Every software enhancement a community member publishes would be a direct reduction to the expenses of closed-source companies.

In a sense, without the protections of copyleft, free software would become the victim of a “tragedy of the commons.” At first, this comparison may seem invalid. After all, how can the common resource of “free software” be depleted by more people making copies when (as a copyright abolitionist would argue) making a copy of someone’s software doesn’t stop them from still having their copy. But the “commons” isn’t the software; the important shared resource, that can either be developed or degraded, is the population of free software developers. Zed Shaw has written a quite personal essay describing his negative experiences from releasing software under a gift license and defending his exclusive use of the GPL in the future. I believe that these experiences would become the norm if copyleft licensing were no longer possible, and that without the option Shaw had of moving to copyleft licenses, most developers would abandon publishing free software altogether.

Just because publishers of free software don’t receive cash payments from the people who use or modify their code, it is wrong to assume that there is no value exchanged, or that eliminating that exchange wouldn’t have an effect on on publishing.

Finally, in The Cathedral & the Bazaar Eric Raymond points out that most software is developed within organizations for their own use or for incorporation into other products. Development efforts where software is the end product are actually very much in the minority. Raymond argues that free software is good for most organizations developing internal-use software, as it allows the development costs of the software to be shared among peer and competitor organizations, providing that the software doesn’t form the primary basis on which competition occurs. However, even these efforts would be affected if copyright (and copyleft) were eliminated. Without the guarantee that other organizations will re-publish modifications, the risk that internal-use software will become the basis on which companies compete increases significantly, with a matching reduction in the incentive for an original author to publish.

When Copyright isn’t Copyright

Finally, a quick diversion into the land of people who want to abolish copyright but somehow keep copyleft or other open/commons licenses. For example, the post Copyright and Open Source suggests that what copyright abolitionists want is copyright reform so extreme that it would be wrong to call the reformed legal framework “copyright” at all. It seems to me that this is a confusing distinction to make. That post claims a distinction between “credit right” (which is apparently Good™ and moral to enforce) and copyright (which is an unreasonable legal fiction). In particular, there’s the common argument against copyright: “If I steal your bicycle, now you have no bicycle; if I copy your computer program, now we both have it.” Unfortunately, the problem of eliminating copyright doesn’t arise because “you” have my computer program. It arises because Microsoft has it and will turn around resell it as proprietary to thousands of others.

Taking away copyright and replacing it with statues protecting “creditright” is not a good trade. Yes, if copyright disappeared, I suppose it might be an interesting consolation prize for people stealing my work to have to include my name on the mile-long list of everyone they’ve stolen from. It might be something (a hypothetical) I could point to in my resume, as I try to find a new job at a closed-source company after the open-source company I was working for collapses. Credit—being able to build a reputation from the work a programmer publishes—is certainly an important part of the value that is exchanged when someone makes free software available. But it isn’t the only part, or sufficient to sustain a free software “ecosystem” comparable to what we have today.

In Summary

So, hopefully I’ve convinced you that, if you like having access to free and open-source software, you need to defend the GPL (even if you choose not to use it yourself), and you cannot join the ranks of the (software) copyright abolitionists. If not, perhaps a quick recap will push you over the edge:

• Copyright law, while having significant flaws in the U.S. at the moment, is necessary to allow authors to set the terms of exchange for their work. Not all compensation is monetary, and copyright is essential in protecting the in-kind exchanges (software modifications, reputation and acknowledgement, etc.) within the free software community. And while credit can be part of that compensation, it is far from the only part that needs to be legally protected.
• The fact that a fraction of the available free software uses gift licenses does not prove that free software is sustainable without the option of copyleft licensing. Not only does gift-licensed software make up a minority of the free software “market”, but even if it were a small majority, copyleft-licensed software controls the dynamics of that market.
• Yes, Richard Stallman and the GNU Project may have political/cultural goals more extreme than yours, but nothing in copyright law or copyleft licenses is coercive. Just like any other contract, a free software license is an exchange voluntarily entered into by both parties.

The terms of copyleft are never forced on anyone—if you feel like you’re being “forced” to use software distributed under a copyleft-license, look at your assumptions again. What are your alternatives, really? Are there other things that you could do, but you’ve excluded them from consideration because of prohibitive cost or inconvenience? What if the piece of free software you “have to” use didn’t exist? If you have any alternative to using a piece of GPL’ed software besides “curl up in the corner and die,” then you’re not being forced, you’re being cheap.

Yes, there’s lots of really good software available under copyleft licenses. Yes, it’s really tempting to use it in your projects—that was one of Stallman’s original criteria for deciding what packages the GNU Project should work on, in an effort to improve adoption of free software. But when making a copy of my work deprives me of the benefit that I want to receive in exchange for that work, it seems like theft to me, even if I “still have my copy.” And given the diverse range of values that developers and organizations seek by publishing free software, undermining copyright will allow perceived theft and depress or destroy the production of new free software.

Well, thanks for playing—I hope you found something here to set you thinking. And if you’re still interested in this topic, here are a couple of other posts I ran into while researching for this piece:

• Why Open Source Vendors Will Continue to Select the GPL
• Why I Use the GPL

For and Against: Copyright Law and Copyleft

Many people advocate reforming copyright law, many of them from the software industry and related fields. And, depending on the reform, it would be fair to count me as one of them. There is also a vocal group who believe that copyright is anachronistic (digital copies have virtually no cost) or contrary to the social good (the free-flow of information is more important than ever in today’s society). Anti-copyright advocates will often use the slogan “information wants to be free,” and maintain that attempts to enforce copyright law are counter to the flow of progress and doomed to failure.

As you may have guessed, I don’t agree with the copyright abolitionists. I also don’t agree with a related set of people, mostly programmers, who are in favor of Open Source software, but feel that the copyleft provisions of the GPL are (somehow) “wrong.” Given the subset of these programmers I’ve known personally, I think that for the majority of them, the objections are relatively shallow: the fact that some software uses a copyleft license imposes extra work on them (making their modified version available publicly) that they’d rather not do, or forces them to talk to their employer’s legal department, or prevents them from using the software entirely because of their company’s policies/decisions. I’ve met virtually no developers who are self-employed, or not employed as programmers at all, who have these objections.

The rest of the anti-copyleft programmers I’ve known and read make arguments in the same famillies as the justifications for copyright law: moral and pragmatic/economic. The moral/ethical arguments against copyleft are much like those of the copyright abolitionists: “software wants to be free” and that it’s unreasonable to call something completely intangible “property.” Somehow, though, they feel that these arguments apply to software more than they do to other copyrighted works. Perhaps they see software more as a collection of useful parts than as an integrated whole like a novel or song. (Although I would imagine that sampling musicians would disagree.)

One interesting anti-copyleft argument is that the GPL (and similar licenses) are not “free” because they deny other people the “freedom” to keep their own modifications to a software package private/proprietary. At first, this argument might be appealing, but it is essentially analogous to saying that a tolerant person must also tolerate someone else’s intolerance. These people are worried about their “freedoms” with respect to something that first had to be created by somebody else. And even though they don’t have to sign a document before copying the original author’s work, the license is still a contract between the two. And if they don’t want to have their “freedom” somehow “taken away” by that contract, all they have to do is not copy the other person’s work in the first place.

Some people who complain about the GPL taking away their freedoms subscribe to the anti-copyright argument that an unsanctioned copy doesn’t take anything “real” away from the original author. But to them I say “How can you complain about the GPL requiring you to let other people copy your work if you believe all restrictions on copying are wrong?” At which point, we arrive at the root of the matter. The people concerned over the GPL’s “taking of their freedom,” underneath all of the moral and economic arguments, actual fall into two groups:

• the lazy: they don’t actually object to other people copying their work, they object to having to go to all the trouble to publish their work for others. Yet they want to benefit from the (presumably much greater) work of original authors who did publish.
• the intellectually dishonest: they’re working in a compiled programming language, so they know that even if copyright were abolished, as long as they release only executable binaries, nobody else can build on their work, even though they’re building on the original author who published their code as source.

Finally, the argument against “depriving people of freedom” cuts both ways. If the copyright abolitionists succeed (either completely, or by carving out some kind of software-specific exception), then they will have deprived me of my “freedom” to only share software source code with those who will share with me in return.

Embrace, Extend, Extinguish

“Embrace, Extend, and Extinguish” is the alleged internal name of Microsoft’s strategy for moving existing standards and technology from being competitive to being Microsoft-propriety. The method is practiced as follows: (1) wait for somebody else to implement something cool, (2) implement a clone of it yourself, (3) add a handful of minor tweaks so that your clone isn’t compatible with the original, (4) use your existing dominance to bootstrap your clone into the marketplace. After step 4, Microsoft’s market share grows monotonically barring a major success by a competitor, because each customer that changes to the MS-specific version creates for themselves a significant cost to ever change (back) to a competing version.

Clearly, the existence of software licensed using copyleft in a particular technical niche doesn’t prevent Embrace&Extend entirely. Any organization large enough to practice this strategy is large enough to implement a clone entirely from scratch. However, having public, non-copyleft sources certainly gives the “extender” a leg up; even more so if the public version of the technology continues to be maintained. Without a copyleft license, and the underlying copyright law to enforce it, the existence of public sources for a technology is an even greater boon to predators and exploiters than it is to the cooperative community. Who knows, if the GNU software wasn’t licensed under GPL, perhaps Microsoft would have started using it as the basis for Windows in the late ’80s or early ’90s?

The article Why I Don’t Use the GPL asks the question “what harm is really done [if a company uses non-copyleft source code] in their product?” The author concludes that there isn’t any harm worth considering (“The open-source project still exists.”), and that there would be benefits to the community-at-large if proprietary software contained larger fractions of open-source code. But in the end, I think that this is a naive analysis. Perhaps it would be accurate if the biggest software companies were what we now consider mid-size, or if they were less likely to look for opportunities to gain advantage exploitatively.

This concludes in Part 3, with a discussion of the different types of Open Source licensing, and what the open-source world might look like in the absence of copyright law.

So, that’s the chain of thinking that led me to the point, the other day, when I was trying to add Amazon text-link ads to text I was writing in MediaWiki. And it turned out to be far trickier than I ever would have guessed.

The HTML that Amazon generates for you to use for a text-link advertisement contains the markup for a link and for the inclusion of a single-pixel image. The link is displayed to the user and how they go from seeing the ad to Amazon’s web site, and the image is used by Amazon’s analytics for impression counting. Because I wanted to embed these ads into a wiki page, I had to convert the HTML generated by Amazon into MediaWiki wiki markup. For example, the following is the HTML for a text advertisement generated by Amazon (spacing modified for clarity):

<a
  href="http://www.amazon.com/gp/product/0132350882?ie=UTF8&tag=wontology-20&linkCode=as2&camp=1789&creative=390957&creativeASIN=0132350882">
    Clean Code: A Handbook of Agile Software Craftsmanship
</a>
<img src="http://www.assoc-amazon.com/e/ir?t=wontology-20&l=as2&o=1&a=0132350882"
  width="1" height="1" border="0" alt=""
  style="border:none !important; margin:0px !important;" />

The first tag is the anchor creating the link, and the second causes the browser to fetch the invisible image from Amazon’s servers any time a page containing the link is displayed. The ad is rewritten in wiki markup as:

[href="http://www.amazon.com/gp/product/0132350882?ie=UTF8&tag=wontology-20&linkCode=as2&camp=1789&creative=390957&creativeASIN=0132350882
  Clean Code: A Handbook of Agile Software Craftsmanship]

http://www.assoc-amazon.com/e/ir?t=wontology-20&l=as2&o=1&a=0132350882

The URL in brackets is rendered using the text following it in the brackets. A bare URL that points to a web page would be displayed using the text of the URL itself as a link, but bare URLs for images are handled differently. By default, MediaWiki requires images that are to be displayed within its pages to first be uploaded to the wiki server, and then incorporated in pages using wiki markup like [[File: ...]] or [[Image: ...]]. However, copying the single-pixel image from Amazon’s servers and uploading it to the wiki would defeat the purpose. The user isn’t supposed to see the image, the important part is that the image be fetched by a user’s browser every time the page containing the image is shown. So, MediaWiki must embed the image into the wiki page with a URL that points to a server other than itself. The MediaWiki documentation refers to this as an “external image.”

When MediaWiki is installed external images are disabled by default. Turning support for them on is accomplished, like most other MediaWiki customizations, by modifying the LocalSettings.php source file. There are two settings that can be added to enable external images in a MediaWiki installation, $wgAllowExternalImages and $wgAllowExternalImagesFrom. The former is simply set to true to enable inline images fetched from any external server, while the latter can be initialized with an array listing the domains from which external image fetches should be permitted. (The preceding links are to the respective MediaWiki manual pages.)

So, I modified my MediaWiki server’s configuration, put my modified ad link into my text, and instead of my pages containing an invisible single-pixel image following each ad link, they displayed the full text of the image’s URL. Hmmmm….

For a while I was puzzled, but eventually figured out that my problem wasn’t that I hadn’t enabled external images correctly, it was that MediaWiki couldn’t identify Amazon’s URL as pointing to an image. When MediaWiki parses the wiki text of a page, it distinguishes image URLs from page URLs based on the presence of a file name extension matching one of the well-known image formats. Amazon’s image link has no extension, and since MediaWiki doesn’t actually fetch from the link, it doesn’t receive the MIME type of the image. It therefore assumes that the URL points to an external web page, and presents it accordingly.

The code that checks this is located in the file includes/parser/Parser.php, and looks like:

    function maybeMakeExternalImage( $url ) {
        ##snip##
        if ( $this->mOptions->getAllowExternalImages()
             || ( $imagesexception && $imagematch ) ) {
            if ( preg_match( self::EXT_IMAGE_REGEX, $url ) ) {
                # Image found
                $text = $sk->makeExternalImage( $url );
            }
        }
        ##snip##

The set of URLs that are identified as pointing to images is determined by the regular expression EXT_IMAGE_REGEX, defined earlier in the same source file. Modifying the regular expression to identify the Amazon image URLs would have been prohibitively complex—the “or” terms required to identify a host name as an alternative to a file name extension would have more than tripled the length of the expression. Instead, I decided to simply modify the detection code, like this:

    function maybeMakeExternalImage( $url ) {
        ##snip##
        if ( $this->mOptions->getAllowExternalImages()
             || ( $imagesexception && $imagematch ) ) {
            if ( preg_match( self::EXT_IMAGE_REGEX, $url ) ||
                 preg_match( "/assoc-amazon\.com/", $url ) ) {
                # Image found
                $text = $sk->makeExternalImage( $url );
            }
        }
        ##snip##

The regular expression I actually use also checks for my Amazon advertiser ID, and perhaps is too specific. If I were going to create a change that could be submitted to the MediaWiki project based on this fix, I’d create a new configuration variable that would contain the regular expression to be used, so that it could be set by wiki site administrators from LocalSettings.php without having to hack a source file.

So, if you’ve been having problems getting external images to work in your MediaWiki installation, whether they’re ads or not, this might be the fix you need.

Copyright In Brief

What It Is

First, a quick review: “copyright” is the legal basis allowing the author/creator of a “work” to control and limit the creation of copies of that work. Copyright pertains to lots of things (writing, music, computer software, video, art, etc.), but protects only the form of those things and not the ideas that they contain.

For example, you and I could both sit down right now and write physics text books. Assuming we both did a good job, the ideas communicated by the two books would be very similar or identical. But we’d almost certainly have created a different presentation and used different words, and both our books would be protected by copyright. Other people couldn’t copy your book without your permission, but I could make copies of mine because I created it myself and didn’t copy you, even though both books describe the same things. (Technical ideas, rather than particular presentations of ideas, are protected by patents, a completely separate field of law complementary to, but not connected to, copyright law.)

Though there are procedures and reasons to register your authorship of a work, it isn’t actually necessary in the U.S. Your right to control the copying of your work exists simply because you created it. There are limits to copyright. First, it eventually expires. Second, there are limited types of copying and excerpting described as “fair use” that cannot be prevented by asserting copyright, such as brief quotations or use of portions for the purpose of satire.

Why We Have Copyright Law

Copyright is so ubiquitous most people don’t think about why it exists or why it is enforced by the government. But there actually are reasons. One family of reasons are basically moral/ethical. People argue that people automatically own what they create, even if what’s created is more idea than object, and that making an unauthorized copy of something is essentially theft. And just like other thefts, it is the government’s place to establish the precise definition of the crime and to protect each of us from everyone else. This is a fine argument as far as it goes, but other people argue that it isn’t reasonable to equate taking a loaf of bread form someone with making a loaf of bread from someone else’s recipe without their permission.

Another set of arguments is more pragmatic, and looks at the question from the perspective of how the existence of copyright for an individual contributes to the public good as a whole. In this line of reasoning, by protecting an author’s ability to limit and control the number of copies of their work, it makes it easier for the author to make a living, and hopefully authors will therefore (on average, across society) produce more works than they would otherwise. Counter arguments here primarily question the degree to which creative works would actually fail to be created in the absence of copyright, and whether the restrictions on distribution of information that copyright creates damage society more than the actual benefits from enforcing it.

Copyright’s Part in Free Software

A computer program is Free Software if it is (1) publicly available for use and modification, and (2) people modifying it are obligated to also make their changes publicly available. Making software publicly available is something the author would do, and isn’t at all difficult today. Imposing the obligation to redistribute on people who author modifications to the software was the unique contribution that Richard Stallman when he coined the phrase Free Software and started writing and distributing his own Free Software as part of the GNU Project.

The obligation of modification authors to redistribute their work (and under the same terms as the original) is accomplished using copyright law. Because the original software is copyrighted by its original author (under U.S. law, whether the author wants it to be or not), the author must explicitly grant others the right to copy that software through a license before they can use it. Whether or not something falls under the name Free Software depends on the terms of that license.

Stallman created the GNU General Public License (the GPL) and uses it for the software he writes. The GPL obligates people who are modifying their copies to in turn publicly distribute and license their modifications under the same license as the original. He refers to this license as a copyleft, because while copyright law allows authors to restrict access to their work, the GPL mandates that the authors of modifications share their work.

Not all Free Software licenses are copyleft licenses. As long as a license gives people the freedom to redistribute copies of software (modified or not), it is a Free license. It is only if the license mandates distribution of modifications that it is a copy-left license.

Next the is Part 2 for some arguments against copyleft specifically, and a description of the “Embrace, Extend, and Extinguish” product strategy and Part 3 addresses different licenses and summarizes.

Hi, I’m Glen, this is November 2009, and I’ve committed to the “National Blog Post Month” (nablopomo.com) and am going to write 30 blog posts in the 30 days this month. I’d planned to write a post about writing posts after I was a few more days in. When I got up this morning, I was planning to write about copyright, Free Software, and information “wanting to be free.” But, it’s gotten late, and I don’t have time to do that justice tonight. So, instead, a quick introduction to me, why I’m doing this, and what you migth find here in the next few weeks.

I’m a programmer. Actually, a lot more things than a programmer. But you know the saying “when the only tool you have is a hammer, all your problems look like nails”? Well, when I bump into a problem, my first impulse it to try to figure out how to solve it with a new piece of software. So: I’m a programmer, or at least an engineer.

I’m a bit over 41, married, have three cats and no kids. I’ve been in computers professionally since I left college (note that I didn’t say “graduated”), first in chip design but mostly in software. If my wandering career is somehow of interest, you can find the details on LinkedIn. Up until this past February I’ve always been a regular employee–occasionally I’ve done some contracting on the side, but the most “entrepreneurial” I’ve been is working for start-up companies. But early this year, I struck out on my own: I resigned from my full-time job, and I’ve been developing a web application since.

Which brings us to the double-edged sword of this whole blog-post-a-day thing. On the one hand, there are probably “better” things I could be doing with the time I’m writing this. On the other, there are a number of things I’ve bumped into during my work that I’ve “been meaning to” write up. And I think I need some structure. It’s been almost nine months that I’ve been working on my idea of The Next Medium-Sized Thing, and I still don’t have it to a point where I think it would be useful to others (although I am getting close). So I’m really hoping that getting one of these out each day will restore a missing sense of completing short-term things.

I’m probably also doing it a tiny bit out of envy for all of the people doing National Novel Writing Month. Now that’s something that definitely isn’t justifiable for someone self-”employed” and living out of savings. But it sure would be fun….

But, I think this will be fun too. There are going to be a fair number of technical/programming articles, stuff on copyright and patents and software (oh, my!), and almost certainly more thoughts on space travel. Probably also some stuff about the way the world works (systems thinking, emergence, process control, sense making, etc.), education, or maybe sustainability (Permaculture, anyone?). Perhaps a personal anecdote or two. And if I absolutely can’t resit, maybe a touch of politics. So, if some of that sounds like it might be interesting to you, then come on back later in the month. If not, well, at least fair warning has been given.

And that brings me happily to the end of Day Two. (Oh, by the way, the timezone in WordPress was miss-set when I posted yesterday, it was actually 11:45pm here. But, I don’t plan to hold myself to a midnight cut-off. All else being equal, I’m both a procrastinator and a night-owl, and as far as I’m concerned the blog-posting day ends when I go to sleep, regardless of what the clock says.)

In building my Rails application, I needed to pre-load data into the same tables that it uses to store user-created objects in operation. The data had to be pre-loaded, because there had to be special records that the application could have hard-coded references to, and I didn’t want to risk re-creating them wrong by hand after database wipes, or on new installations. Here I describe the thrashing around I went through, and the solution that I finally came up with. Which I think are pretty good. But since I originally implemented it, an alternative has started to be build into Rails itself. This is described in places like Seed Data in Rails 3, Railscasts #179: Seed Data, and Google’s results for the release of Rails v2.3.4.

Prior to v2.3.4, most Rails applications that needed seed data used static inserts in a database migration file to put the database records in place. This approach has a number of problems. First, the preferred Rails way for loading a database in a new installation is by loading the db/schema.rb file, instead of by loading all of an applications historic migration files. This is not only more efficient, but often as a Rails application evolves its older migrations can no longer be run (they have become incompatible with the current application source). Encapsulating seed data in migrations requires that new installations be initialized by re-running the migrations, which may eventually stop being possible. In addition, if the database schema is updated after the seed data creation statements are created (a virtual certainty), the seed data will have to be updated, either by editing the older migrations (a version-control nightmare) or by creating migrations that alter the seed data (effectively scattering the data across multiple migration files, a significant barrier to comprehension if anyone ever needs to read the data).

Another problem with using migrations is that, unlike Rails mechanism for loading fixture data, there is no support for creating IDs that create associations between database records in different tables (database table foreign keys). Given that my seed data needed to go into two tables, with records from one referring to records in the other, not having fixture-like automatic generation of association IDs would have made maintaining my seed data very difficult.

So, what I really wanted was something that worked without using migrations, and with ID creation/mapping like fixtures. Which led me to find a fixture- and rake-based solution posted in a couple of different places on the web. The code I wrote based on their ideas starts with a couple of rake tasks:

namespace :db do
  desc "Load YAML seed data from db/fixtures"
  task :seed => :environment do
    load_all_YAML_globbed_from(
      File.join( RAILS_ROOT, 'db', 'fixtures', '**', '*.yml' ))
  end

  desc "This drops the db, builds the db, and seeds the data."
  task :reseed => [:environment, 'db:reset', 'db:seed']
end

I keep the do-it method and its dependencies along with these tasks in lib/tasks/db.rake. The “seed” task will load al of the YAML files located in db/fixtures. In the common case where there is a uniqueness constraint on one or more database columns that are populated from the fixture data, it won’t be possible to execute “seed” more than once on the same database. Therefore, the “reseed” task is provided to empty the database, rebuild its schema from the current, and then load the seed data.

The rest of the code to implement this task isn’t that complicated. It builds on Rails’ built-in database connection and fixture loading:

require 'active_record'
require 'active_record/fixtures'

def load_all_YAML_globbed_from(pattern)
  ensure_ActiveRecord_available
  Dir.glob(pattern).each do |path|
    load_YAML_from path
  end
end

def ensure_ActiveRecord_available
  unless ActiveRecord::Base.connected?
    ActiveRecord::Base.establish_connection(
      ActiveRecord::Base.configurations[RAILS_ENV] )
  end
end

def load_YAML_from(path)
  path =~ %r%([^/_]+)\.yml$%
  table = $1
  path.sub!(/\.yml$/, "")

  begin
    f = Fixtures.new( ActiveRecord::Base.connection, table, nil, path )
    f.insert_fixtures
  rescue ActiveRecord::StatementInvalid
    # must have already loaded this data
  end
end

ensure_ActiveRecord_available makes sure that the current execution context has a valid database connection, and that it matches thee current Rails environment (e.g., production, development, or test). The biggest difference between this code and the portions of Rails that are responsible for loading YAML test fixtures are the first three lines of load_YAML_from. Rails uses the entire base name of the YAML file (that is, everything but the ending file extension “.yml“) as the name of the database table to load. Rails also loads fixtures only from a single directory (not recursively from an entire directory tree, as the File.join() in the “seed” task does). This has the cumulative effect of requiring all of the data that is to be loaded into the same table to come from a single file. Allowing YAML files to be distributed between multiple files, and for multiple files to load into the same database table, allows large sets of seed data to be broken up across multiple files.

Having a Rake task that loads seed data solves the problem for installing new application instances and for interactive development. Unfortunately, Rails’ built-in machinery for setting up the database used for automated tests knows how to clone the schema from the development database, but not its content. And if an application really needs its seed data to operate, then its unit tests shouldn’t pass if the database is empty.

To load the seed data into the test database, a mechanism similar to the above is necessary when running unit tests, and the preferred way to hook into this is using the standard test_helper file. For cleanliness, I chose to put most of the seed-related code outside the test helper and require it:

test/test_helper.rb:

class ActiveSupport::TestCase
  require File.join( File.dirname(__FILE__), 'seed_helper' )
  setup :load_app_seed_data
end

test/seed_helper.rb:

require 'active_record'
require 'active_record/fixtures'

# define method in any context where 'require'ed; no fixed namespace/class
def load_app_seed_data
  ensure_connection

  Dir.glob(File.join( File.dirname(__FILE__), "..", "db", "fixtures",
                      "**", "*.yml" )).each do |path|
    path =~ %r%([^/_]+)\.yml$%
    table = $1
    path.sub!(/\.yml$/, "")

    begin
      f = Fixtures.new( @helper_db_connection, table, nil, path )
      f.insert_fixtures
    rescue ActiveRecord::StatementInvalid
      # must have already loaded this data
    end
  end
end

def ensure_connection
  unless ActiveRecord::Base.connected?
    ActiveRecord::Base.establish_connection(
      ActiveRecord::Base.configurations['test'])
  end
end

This code is essentially the same as the content of the Rake task (and, now that I look at it again, perhaps with a bit of refactoring I could require it from db.rake instead of duplicating, hmmm….). The key here is the setup call in test_helper.rb. This connects the method that loads the seed data to the setup-phase callback Rails (Test::Unit) provides in the flow of running unit tests.

Note the rescue that catches StatementInvalid exceptions. This is present/duplicated in the Rake task above, but is most important here. It prevents attempts to reload the same fixtures from causing a test run to fail in the event that the callback is repeated.

Alas, a Rails project is likely to use additional test frameworks beyond the built-in Test::Unit and its derivatives. For my project, I use Cucumber, a Ruby automated functional test framework, and Blue Ridge, a Rails plugin for JavaScript testing. In order to support running tests through these frameworks via Rake individually, they also need hooks to load the seed data into the test database if it hasn’t been already. The code for these two frameworks are given as examples, both for themselves and to guide integration into other frameworks.

For Blue Ridge, I created a locally-modified version of their rake tasks in lib/tasks/javascript_testing_tasks.rake:

require File.join( RAILS_ROOT, 'test', 'seed_helper' )

begin   # don't force Blue Ridge dependency on non-developers
  namespace :test do
    task :javascripts => [ :environment ] do
      prepare_and_load_database

      Dir.chdir("#{RAILS_ROOT}/test/javascript") do
        begin
          load_app_seed_data
          # rest of the rake task was here
        ensure
          # shutdown/cleanup here
        end
    end
  end

  def prepare_and_load_database
    # ensure that we start with a clean database
    Rake::Task["db:test:prepare"].invoke
    # plus our test "fixture" data
    Rake::Task["db:test_fixtures"].invoke
  end

  def cleanup_database
    # ensure that we *leave* a clean database, too
    Rake::Task["db:test:prepare"].invoke
  end

rescue LoadError
  puts "WARNING: Missing development dependency.  'Blue Ridge' or a dependency is not available."
end

At this point, there isn’t much to say about the call to load_app_seed_data. However, the other two *_database calls are new. These are present not because the seed data loading needs them, but because I wanted to have the same test fixture data present for reference by my JavaScript tests as I use in my Ruby/Rails unit tests.

For Cucumber, the hooks to load seed data are placed into a standard extension file that Cucumber supports, features/support/local.rb. In fact, the entirety of my project’s local.rb are the two lines:

require File.join( File.dirname(__FILE__), 'seed_helper' )
load_app_seed_data

For Cucumber, I use special steps in each scenario to create any non-seed data needed for the test, so there is no code to load the test fixture data.

So, even with Rails incorporating an “official” seed-data mechanism for Rails 3, you may find this code useful if you too want to use multiple YAML files per database table, or to have your seed data available to tests running in add-on frameworks. Please comment with any questions.