First vendors start releasing things with SD card readers, and people quickly realise Linux can't use them. Then several very talented hackers start a project <http://mmc.drzeus.cx/wiki/Linux/Drivers/sdhci> to reverse engineer SDHCI controllers, since the SD Card Association won't release specs publically (apparently citing "trade secrets").

Talented hackers get things working so well that it makes it into 2.6.17-rc1 <http://www.kernel.org/git/gitweb.cgi?p=linux/kernel/git/torvalds/linux-2.6.git;a=history;h=6246b6128bbe34d0752f119cf7c5111c85fe481d;f=drivers/mmc/sdhci.c>.

After all this is done, the SD Card Assosociation, seemingly without any prior communication, release a simplified spec http://www.sdcard.org/HostController/index.html> now that the community has figured it out.

Of course it acts as a good check that what was reverse engineered is correct, and reverse engineering things can certainly be a lot of fun. But really, what a lot of wasted resources thanks to a backwards vendor. What exactly have the SD Card Association gained, apart from bad will?

I toyed with the idea of vectorising this, but it seemed too hard.

In essence, SAL (System Abstraction Layer) starts the machine with one processor running (the boot processor) and the others asleep.

The boot processor jumps to _start in head.S, which checks a variable task_for_booting_cpu to see if it is the boot processor or not.

After that, it jumps into platform specific code and gets the machine ready to go. Eventually we fall into smp_init() which starts the other processors. For each other processor in the system we call do_boot_cpu which sends an IPI (inter-processor interrupt) wakeup to the other processor.

The other processor wakes up, and again jumps into _start, however this time when it checks the task_for_booting_cpu it will be set as the idle thread, so it knows it is not the boot processor. It jumps into start_secondary but largely follows more or less the same path, but skipping the platform setup stuff. Eventually it calls smp_callin to flag back to the boot processor that it is alive and sitting in the idle thread.

The boot processor waits a few seconds for each CPU to check in as alive before assuming the worse and moving on. Once all the CPUs are online, the system is pretty much booted.

Have you ever wondered what a frequency histogram of nibbles in a dump of an uncompressed Itanium Linux kernel image is? Well, wonder no more

0x0     6175200  37.0%  *******************************************************
0x1     1612596   9.7%  ***************
0x2     1261945   7.6%  ************
0x3     632509    3.8%  ******
0x4     1098759   6.6%  **********
0x5     590878    3.5%  ******
0x6     753510    4.5%  *******
0x7     576100    3.4%  ******
0x8     1194081   7.1%  ***********
0x9     462349    2.8%  *****
0xa     408003    2.4%  ****
0xb     275851    1.7%  ***
0xc     439190    2.6%  ****
0xd     263658    1.6%  ***
0xe     353211    2.1%  ****
0xf     607788    3.6%  ******

I swear I was not procrastinating when I came up with this; I was (am) trying to find a bug where I get a totally crap value in a register, and I wondered if that value was somehow being pulled out of the kernel code.

Martin Pool reminded me of The Innovator's Dilemma by Clayton M. Christensen.

This book examines why great companies miss the boat with new technology. He talks about disruptive technologies; those technologies that start out under performing established offerings. They're usually cheaper, smaller, simpler and have features that appeal to fringe customers (i.e. those customers who haven't already bought existing offerings).

These disruptive technologies often creep up on established offerings; as illustrated in the graph below.

The problem is that the disruptive technology doesn't become viable for all customers for some long period of time. Companies who listen to their customers end up developing their existing lines (to meet existing customer demand) until suddenly that customer demand is now met by the disruptive technology. At that point, the sustaining technology is out of businesses; nobody wants it. Christensen gives many examples, especially with disk manufacturers. Companies making 14 inch platters were catering to mainframe customers who needed storage in the 300Mb range, not the 30Mb range 8 inch platters of the time could offer. However, it wasn't long before the 8 inch platters were in that range, and by that stage it was too late for the 14 inch platter companies.

Disruptive technologies grow by opening up new markets. The book gives the example of hydraulics and digging equipment; initially hydraulic arms could not dig as much dirt as the steam and cable counterparts, however house builders, interesting in digging shallow, narrow ditches for the post-war housing boom, snapped up the new offerings. It sure beat digging ditches by hand. Eventually the disruptive technology of hydraulics caught up to cable based diggers, which were less reliable and more unsafe (imagine a cable snapping with tons of load on it), putting most of the cable based digger companies out of business.

But the cable based companies could not offer hydraulic diggers to their customers; for a long time they could not dig as much so those customers who need to excavate a lot of dirt would not buy them. It would have been a very bad managerial decision to move the company to hydraulic diggers and loose all your big customers. They followed what their customers wanted and went out of business for it.

The book goes on to talk about strategies for companies to manage this change. However, Linux in specific and open source software in general is clearly a disruptive technology. Where exactly Linux is on the line is an open question, I think somewhere after the red circle but before the green circle.

The evangelism prevalent in some areas of Linux and Open Source development has always struck a difficult chord with me. Concerted efforts to sell open source as something better because it is open source have never really made sense to me.

This graph illustrates to me why. Big gigantic changes can't be made by big gigantic companies. Switching to what I (we) can see as a disruptive technology is not in their interest, because it probably does not meet their customers needs. They essentially need to wait until it is too late; to when the disruptive technology has caught up to them sufficiently to provide an alternative to the status quo.

As a disruptive technology you grow by opening up new markets on your way to "world domination". You need to focus on your technology, and keep pushing up the development line to get closer and closer to that green circle. Marketing of course plays a part, but the performance of the technology has to be there. I strongly believe that sufficiently good technology starts to speak for itself -- the bottom line with software is that it does what you need it to do in a way that helps you do it. It is for this reason I think a Freshmeat announcement of version N+1 is infinitely better than a press release about Linux/Open Source blah blah blah.

I think it is important to keep this in mind. People digging the biggest holes couldn't move away from cable based excavators until hydraulics could move as much dirt as the old technology. With software, moving the dirt equates to features, support, ease of use, compatibility and a myriad of other challenges. I look forward to the day Linux is the biggest shovel of all!

Unless you are one of the core kernel developers, you always communicate your changes via patches posted to mailing lists. However, if your patch is going to take more than ten minutes to write you need to keep up to date with the upstream kernel source, because otherwise you'll end up hopelessly behind. The problem is that you always need to keep your changes on the top, because you need to be easily able to export them as a patch against the upstream tree at any point.

I call this being on the "edge" of kernel development -- there is nobody further out from the middle than you. Generally, I have found git and any of the wrappers quite painful, but thanks to Matt Chapman I gave Mercurial (hg) a go, and it is perfect.

Firstly, clone a new tree

hg clone http://www.kernel.org/hg/linux-2.6/

This tree you will never touch directly; it is a copy of where Linus is at any point in time. Next, clone a working directory where you will be doing your development.

hg clone linux-2.6 linux-2.6-working

Start doing your work on the linux-2.6-working tree; make commits as necessary as you develop major parts of your work.

You will want to tag your first commit to make it easier to create diffs of your work. Use a local tag, because it is only for your reference within the local tree. So after your first commit do

hg tag -l kernel-import

You can then get all your changes against the kernel you have downloaded from with

hg diff kernel-import:tip

Now several hours/days/weeks later you need to update your changes against the latest upstream versions. Firstly, update the upstream tree

cd linux-2.6
hg pull

There shouldn't be any conflicts or issues, because you have not changed anything locally. Now create a new update tree, cloned from the latest upstream version (just as you did when you started).

hg clone linux-2.6 linux-2.6-update

Now pull into the update tree your development tree. This will (should) have the effect of just grabbing all your changesets.

cd linux-2.6-update
hg pull ../linux-2.6-working

Now your project has multiple heads -- one from your old working tree and one from the new tree you are merging into. You will now need to merge the pulled changes into the new update tree. Do this with

hg update -m

Hopefully there won't be any conflicts, but if there are you will have to resolve them. Once done, the final step is to commit your changes into the new tree.

hg commit -m "merge to new linus tree"

Now create a local tag at the start of your changesets by looking at hg log and using hg tag -l kernel-import first-of-your-changesets-revision-number (is there a better way to do this?).

hg tag kernel-import

So your changesets should now be on the the tip of the update tree. So the update tree becomes the new working tree, and you can archive the old working tree. You can export your patches (for sending to lists, etc) with

hg export kernel-import:tip

Now continue your work in the update tree you just created. Commit as much as you like doing all the development you require, at any time you can use the export from the tag you created to get a patch of your work.

Eventually you will need to re-sync with upstream again. At this point repeat the process; make a new update tree and pull your working tree into it. Archive the old working tree and continue development on the new 'update' tree.

I've been hacking a lot of kernel headers lately, which gives you plenty of time to think as the rebuild magic happens. So I thought I'd try to come up to speed a bit more on git, the new kernel SCM.

Which lead me to attempt a diagram explaining how things fit together. As usual with these things, I'm not sure if it makes things clearer or more confusing.

xfig source available if you want to make it better.

• The Definitive Guide is a good start, but probably won't help you actually make a document.

• You have two choices, XML or SGML (see the guide). I've always used SGML, but make it like XML (see below). This is because doing things like inserting a text file or program listing from a foreign file with XML requires all sorts of playing with tools that seem mostly to be based on Java, which sucks on any platform !x86, which I use a lot. It seems much easier to just use jade to get everything done.

• Use the XML declaration for SGML so you can have closed tags (e.g. <tag />). I think this turns it into something defined as WebSGML. Close all your tags (such as <imagedata>) and make sure all your tags are lower case. This means if you want to make it XML later, you can. I've also found xsltproc does a better job at finding problems than jade.

• You'll want a CSS file to style your HTML output. Create a DSSSL script that looks like below.

  <!DOCTYPE style-sheet PUBLIC "-//James Clark//DTD DSSSL Style Sheet//EN" [
  <!ENTITY docbook.dsl PUBLIC "-//Norman Walsh//DOCUMENT DocBook HTML Stylesheet//EN" CDATA dsssl>
  ]>
  <style-sheet>
  
  <style-specification id="html" use="docbook">
  <style-specification-body>
  
  (define %stylesheet% "stylesheetname.css")
  
  </style-specification-body>
  </style-specification>
  
  <external-specification id="docbook" document="docbook.dsl">
  
  </style-sheet>
  

• Put your figures in with something like below and you can create both PDF and HTML output

  <figure>
       <title>Figure title</title>
       <mediaobject>
         <imageobject>
           <imagedata fileref="figures/figure.eps" format="EPS" />
         </imageobject>
         <imageobject>
           <imagedata fileref="figures/figure.png" format="PNG" />
         </imageobject>
         <textobject>
           <phrase>About the figure.</phrase>
         </textobject>
       </mediaobject>
     </figure>
  

• Create your output with the jw wrapper around jade.

  HTML
  $ jw -o html -d stylesheet.dsl -f docbook -b html -l /usr/share/xml/declaration/xml.dcl docbook-file.sgml
  
  PDF
  $ jw -f docbook -b dvi -l /usr/share/xml/declaration/xml.dcl docbook-file.sgml
  $ dvipdf docbook-file.dvi
  

The short version is : pull power from the box, setup Z-modem to send the new firmware file, turn the box on. That's it -- no Tridge style hacking required.

For those of you who missed the BBS experience and skipped straight to the world of TCP/IP, in more detail I did

$ wget http://www.kristal.com.au/opentel/downloads/ODT4200PVR_A_KRISTAL_V202.zip
$ unzip ODT4200PVR_A_KRISTAL_V202.zip
$ stty -F /dev/ttyS0 ispeed 115200 ospeed 115200
$ sz -vvv -b ODT4200PVR_A_KRISTAL_V202.pgm < /dev/ttyS0 > /dev/ttyS0
[walk over to box, pull out of power point and put back in, it downloads software and reboots]

sz is in the lrzsz package in Debian.

Running strings over the firmware doesn't uncover anything extraordinarily interesting; anyone else pulled it apart any further?

Update : the latest release appears to be a semi-public 2.04 version which can be downloaded directly from Oznetics.