In this post Mike introduces the concept of an "Open Source Bigot" in response to this article. A perhaps more reasoned article appears here. The thread that runs between them is that you may not be able to make a profit by releasing your software as free software. Note I mean free in the FSF sense, and in the sense a open source bigot would mean. Below I refer to non-free open source software; basically I mean this to encompass any license that doesn't require you redistribute modified source code.

Something didn't sit right with me when I was reading these articles. It took me a bus trip home to think of why. In economics you study the "tragedy of the commons". Most everyone is aware of this theory these days; if you have a common resource it doesn't cost you anything to use more of it, but degrades the overall opportunity for everyone to use it. The usual example is the village field, where everyone sees it as advantageous to breed one more head of cattle, even though their actions combined will lead to overgrazing.

We learn that to avoid this you need some form of regulation over the resource. You might charge carbon dioxide output and setup a carbon credits market, or make it illegal to (and fine people who) dump into rivers.

But my use of non-free open source software doesn't influence anyone else's ability to use that software! The tragedy of the commons doesn't apply! Indeed, software is unique in that it costs basically nothing in terms of resources to duplicate and distribute it, and it doesn't cost any more for one person or a million to use it. In this case, the source code isn't the river or the village field. But there is something that is like the village field -- those ideas that the software manifests. The software does something; it may do it not so well, or very well indeed. When it doesn't do it well, we call it a bug or a crappy API; when it does we call it efficient or a nice API.

If everyone just uses the software, non-free open source is a great boon. It costs nothing to distribute and the people using it are reaping benefits they otherwise wouldn't have had. If they contribute back to their community, so much the better.

But when a for profit company takes non-free open source software and uses it in a for profit product, they might also find and fix bugs or enhance the operation of the software. Leveraging the non-free open source software to do something better for you is analogous to leveraging the nice field to have one more head of cattle. This company now has a competitive advantage. Not only this, but a bug isn't a bug until you find it is a bug (a little philosophical, but I think you get the point) meaning that once it is found, while it remains unfixed it represents a regression. It's a similar situation with an unclean API; once someone writes a cleaner one the old one becomes obsolete.

I mentioned that to overcome the commons problem we need some sort of regulation. Herein lies the GPL and "open source bigots". The GPL requires you to give back those changes to the community you got them from. When you release your code under some sort of BSD license and it's picked up by a for profit commercial vendor, you've effectively made your own little tragedy of the commons. You're left with a product is that is worse thanks to the (newly found, but unreported and unfixed) bugs and unclean API's (that now have good replacements) and can't expect to get any changes back.

And why should I expect anyone to give me back changes when it is not in their interest to do so? As Mike says, "Open Source isn't a knife, competition is a knife". Why sharpen the knife of the competitor? I know Mike and the Atlassian guys give their changes back; but if their changes are going to make Bugzilla a better competitor to them, what logical reason do they have to give them back? Ethics? Ethics is nice but not dependable. It is said that the market is never wrong and everything about markets says you don't give your competitive advantages to your competitors, ethics or not. Using the GPL pushes this from an ethical requirement to something enforceable, which changes the equation.

So you may not be able to make a profit releasing your software as free software, but if you're using non-free open source components in your proprietary, commercial product I think you are in a precarious situation. Can you say you'll always pay back the community for what they gave you? Can you say the same about everyone else?

It took us a long time to realise that pumping carbon dioxide into the atmosphere was even likely to be doing any harm. Will we some day realise that releasing code with no quid pro quo ended up disadvantaging us more than it helped? There are many licenses that don't require you to return your changes to the community you took them from and many developers prefer distribute code with such licenses. People develop software for many reasons, sometimes just to scratch an itch, sometimes just for fun and sometimes as a challenge. Who am I to say anyone's choice of license is wrong? Maybe I'm just an open source bigot.

I've recently got great service from two companies online, so credit where credit is due.

• www.camerastore.com.au : I ordered a Canon IXUS 30 camera and it arrived at about 9:00am the next morning. They were also the cheapest price I could find!
• I ordered some business cards from Salt Media in Western Australia. The first batch were wrong because of some stupid overprint thing that only shows up when you print, the guys kindly offered a half price reprint even though it was really my fault. Tip: ask for a proof, always. Great price, too.

James Wilson was kind enough to point me to some of the problems that gcc has with the peculiarities of IA64. Take, for example, the following little function :

void function(int *a, int *b)
{
    *a = 1;
    *b = 2;
}

Now let's inspect the code that gcc generates for this on 386 and IA64

--- 386 ---
push   %ebp
mov    %esp,%ebp
mov    0x8(%ebp),%edx
movl   $0x3,(%edx)
mov    0xc(%ebp),%edx
movl   $0x4,(%edx)
pop    %ebp
ret

--- ia64 ---
[MMI]       mov r2=3;;
            st4 [r32]=r2
            mov r2=4;;
[MIB]       st4 [r33]=r2
            nop.i 0x0
            br.ret.sptk.many b0;;

Now those of you who know IA64 assembly will see that that function takes three cycles when it really only needs to take two. The ;; signifies a stop, which means that there is a dependency -- i.e. after the mov you need to put a stop before loading it into r32. This is called a "read after write" dependency; you have to tell the processor so that pipeline gets things right. On x86 you don't have to worry about that, as internally the processor breaks down instructions to smaller RISC operations and will schedule the two independently -- the whole point of IA64 and EPIC is that the compiler should figure this out for you.

What we could do is put the two mov instructions into separate registers (remember, we have plenty) and then stop, saving a cycle. Indeed, the Intel compiler does as good a job as one could have by hand :

[MII]       alloc r8=ar.pfs,2,2,0
            mov r3=3
            mov r2=4;;
[MMB]       st4 [r32]=r3
            st4 [r33]=r2
            br.ret.sptk.many b0;;

Now, if you're really good (like James, who did this) you can inspect the gcc optimisation paths (via the -d flags) and realise that at first gcc gets it right, putting the two loads before a stop. But the register optimisation pass jumps in and tries to reduce the lifespan of a registers with a constants in them, breaking up the instructions. On x86 you want this -- it reduces register pressure at the cost of instruction level parallelism; a trade off you are willing to take. On IA64, with plenty of registers and relying on ILP to run fast, it hurts.

People are starting to look at issues like this; but if you've ever peered under the hood of gcc you'll know that it's not for the faint of heart!

Attendend a talk by Rob Pike from Google (who is important enough to have his own Wikipedia entry).

In short, the talk was on a language he jointly wrote which I imagine is the tool people at Google use to get figures like the Zeitgeist numbers (I don't think he actually mentioned the name, something like sawzill? Whatever it is I think it's also the name of a regular expression function in the language). Actually Google have lots of indexes, not just the entire internet but all the search queries, machine logs, developer check-in and build logs and probably what people eat for lunch. You can imagine it would be cool to see this data in interesting ways.

No database is going to handle all this data. In fact, a complex relational model database isn't really needed because most of the data can be thought of as associative ie. (result a + result b) + result c = result a + (result b + result c) and communicative ie. result a + result b + result c = result c + result b + result a.

The beauty of it is that you only ever think of things one record at a time. You then "emit" your transformation of the record upwards to an aggregator, which, as the name suggests, aggregates your results into something interesting. These aggreagtors plug in with the MapReduce implementation, which would be another few talks, although it can apparently sort a terrabyte of data in minutes. There are lots of aggregators, like sum, top, histograms, sort etc. All these communications happen through a Google standardised IDL layer called "protocol buffers" which have bindings into many languages like C/C++, Python, etc. It was suggested you can think of this IDL layer as "XML without the XML crap".

So as I understand it, you run Rob's tool over your index extracting those that you want. This is extremely parrellisable, since you only consider one record at a time and the results are associative/communicative. You pass this upwards to your aggregators, which plug into the MapReduce stuff which is also highly parallelised. They have plans to "pipe" these together (analogous to a Unix pipe), though they haven't done that yet (I imagine you'd want that for something like "most increasing searches" ... find all searches for last week, then all for this week, then take the difference of the two sets, then sort the result set, etc). Rob has a job he runs every morning to check stuff which takes a week of CPU time, and completes in a few minutes.

One interesting part is handling of crap index entries -- the language runs in two modes; one where you crash out for entries that you can't handle and one where they are ignored but logged. This means you test it, but when you run it on the google indexes where there are obviously pathalogical cases you didn't think about (a query string with a million 'a's for example) the program keeps running.

It was pointed out this model (consider only one record at a time) doesn't fit well with a graph structure of data, such as for calculating PageRank ... indeed Rob responded that alternative mechanisms are indeed used to calculate PageRank type figures. But you can certainly check the PageRank of a page via this tool, and easily run queries like "what is the highest PageRanked page on a site" (for example, adobe.com has the Acrobat Reader download page as the highest ranked page).

As a background you need to understand the instruction format of IA64. Itanium groups instructions into groups of three called "bundles". Each of the three instructions are in a slot (slot0-2). Each instruction is 41 bits, and there is 5 bits of template information (making for 128 bit bundles). There are rules about what instructions can be bundled together and in what order they come (the templates). This allows the compiler to determine optimal bundling ... the theory being that the compiler has more information about what is happening (having access to the source code) so it can make best use of the processor resources rather than the processor having to guess what is happening at runtime. This is why it is important to use a good compiler to get good results out of the Itanium processor.

Using the linux ptrace() call we can step through the instructions a program is executing.

#include <errno.h>
#include <signal.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>

#include <sys/ptrace.h>
#include <sys/wait.h>

char *prog_name;

#include <asm/ptrace.h>
#include <asm/ptrace_offsets.h>

static int lastslot = 0;

union bundle_t {
        struct {
                struct {
                        unsigned long template  : 5 ;
                        unsigned long slot0     : 41;
                        unsigned long bot_slot1 : 18;
                } word0;
                struct  {
                        unsigned long top_slot1 : 23;
                        unsigned long slot2     : 41;
                } word1;
        } bitfield;
        unsigned long array[2];
};

void
print_instruction (int child_pid, int state)
{
        long scnum;
        long ip, slot;
        union bundle_t bundle;

        ip = ptrace (PTRACE_PEEKUSER, child_pid, PT_CR_IIP, 0);
        slot = (ptrace (PTRACE_PEEKUSER, child_pid, PT_CR_IPSR, 0) >> 41) & 0x3;
        scnum = ptrace (PTRACE_PEEKUSER, child_pid, PT_R15, 0);

        printf("%lx %d\n", ip, slot);
}

int
main (int argc, char **argv, char **envp)
{
        int status, pid, child_pid, state = 1, arg = 1;

        prog_name = argv[0];

        child_pid = fork ();
        if (child_pid == 0)
        {
                ptrace (PTRACE_TRACEME, 0, 0, 0);
                execve (argv[arg], argv + arg, envp);
                printf ("%s: execve failed (errno=%d)\n", prog_name, errno);
                exit(-2);
        }

        while (1)
        {
                pid = wait4 (-1, &status, 0, 0);
                if (pid == -1)
                {
                        if (errno == EINTR)
                                continue;

                        printf ("%s: wait4() failed (errno=%d)\n", prog_name, errno);
                }

                if (WIFSIGNALED (status) || WIFEXITED (status)
                    || (WIFSTOPPED (status) && WSTOPSIG (status) != SIGTRAP))
                {
                        if (WIFEXITED (status))
                        {
                                printf ("%s: exit status %d\n", prog_name, WEXITSTATUS (status));
                                break;
                        }
                        else if (WIFSIGNALED (status))
                        {
                                printf ("%s: terminated by signal %d\n",
                                        prog_name, WTERMSIG (status));
                        }
                        else
                                printf ("%s: got signal %d\n", prog_name, WSTOPSIG (status));
                }

                print_instruction (child_pid, state);
                ptrace (PTRACE_SINGLESTEP, child_pid, 0, 0);

        }
        return 0;
}

This will produce a lot of output which should show you increasing instruction pointer values and slot values. Each exception happens after the instruction, so to see what you just executed you have to go back one slot.

...
20000000000043d0 0
20000000000043d0 1
20000000000043d0 2
20000000000043e0 0
20000000000043e0 1
20000000000043e0 2
...

So far, that's all quite straight forward. The only tricky bit comes around system calls. To make a system call on IA64 you put the system call number into r15 and execute a break 0x100000 call (well, you used to, until fast system calls were introduced).

But you will have issues single stepping around the break system call, because it has a higher priority than the single step exception. This means that the system call will be handled, the instruction pointer updated, the next instruction executed and then you'll get your fault.

To illustrate with an example, image a function like

400000000000e500 :
400000000000e500:       01 10 24 02 80 05       [MII]       alloc r2=ar.pfs,9,1,0
400000000000e506:       f0 00 80 00 42 00                   mov r15=r32
400000000000e50c:       00 00 00 08                         break.i 0x100000;;
400000000000e510:       13 00 fc 15 06 bb       [MBB]       cmp.eq p0,p6=-1,r10
400000000000e516:       41 00 00 42 00 00             (p06) br.ret.sptk.few b0
400000000000e51c:       30 51 00 41                         br.cond.spnt.few 4000000000013640 <__syscall_error>;;

You're going to see output like

400000000000e500 0 <--- after call into function
400000000000e500 1 <--- slot 0 of syscall first bundle
400000000000e500 2 <--- slot 1 of syscall first bundle
                   <--- system call handled (nothing printed)
400000000000e510 1 <--- slot 0 of syscall second bundle
400000000000e510 2 <--- etc

Of course, you can change the ptrace argument to PTRACE_SYSCALL and you will get two faults at 0x400000000000e50c ... one on entry and one on exit.

There are some attempts to show Debian usage by architecture via mirror statistics. Dirk Eddelbuettel writes

Lastly, some concerns were raised about various biases from local mirrors, web caches, multiple installs and what have you. These are fair questions as they all affect how Debian is obtained, installed and updated. But for as long as we don't know why that should be different across architectures, this is not a concern for the question at hand. The concerns reflect uncertaintly about the absolute level of users, but barring additonal information (or hypotheses), they do not affect the distribution of users across architectures which is what this exercise is about in the first place.

I don't see how this can be right. For example, I use an IA64 desktop, so does one other person in my group, and we run two IA64 production servers (this isn't counting the many development boxes). These have all been pulled from apt-proxy on one machine which caches. So my updates at home on my PowerPC would be seen, but only one of the IA64 ones were (the first update to the apt-cache), even though relatively there are more users. So how can mirror statistics really show you anything?

I'd think they types of people running IA64 boxes will be at large institutions and generally updated via local mirrors somehow. They're also likely to have many boxes, rather than just one or two. So I'd seriously doubt the stats for IA64 mean much.

I recently picked up the Yamaha TSS-15W from Dick Smith.

I had previously invested in a Joytech Switch Box which is a great product at $100 (I've seen them Electronics Boutique and some Harvey Normans, where they were slightly cheaper) which switches 4 SCART, S-Video, Composite Video and, most impressivley Optical Audio.

Seeing as my digitial set top box and DVD had optical output, my primary requirement was that the speaker system have optical inputs -- the TSS-15 fits this perfectly with two optical inputs and one coaxial digital input (plus one normal stero input).

Then an Xbox advanced av pack, a few optical and s-video cables later and I'm really happy with the results. If you are buying optical cables, the cheapest ones I could find were at JB Hi-Fi and were labeled as minidisk cables ($12), but as long as it has the TOS-link end you should be right. I saw some that had gold plated ends -- on an optical cable! People will pay extra for anything.

The TSS-15 does everything you could want -- it does all the usual Dolby this and that, including the Pro Logic modes that turn 2 channel input into 5.1 channel output (by default it automatically switches around). The speakers are really small but surprisingly loud; for normal living room use they are great and go up louder than I can bear to listen to them. The subwoofer is also fine, though it's not separately powered (passive subwoofer?) it has some special design that makes it good, apparently.

The remote is quite functional, giving you control over the current input, Dobly modes and a special "night mode" which seems to just cut the bass and up the treble.

For me, the upgrade to S-Video everywhere and surround sound has been fantasitic. The separate switch box with optical switching has allowed me to upgrade really easily -- for example many of the cheaper dvd all in one systems (~$800) only come with one or two digial inputs -- not enough if you want the x-box or other extras.

The only problems with the system are minor -- the ends of the speakers all have an RCA connector, which means you have to have a very wide speaker stand if you are to run the wire down the middle of it; else you have to cut and re-solder the ends (as I did). I picked up some very cheap stands also from Dick Smith, really designed for another model. With a bit of double sided tape they stick on, but don't look perfect. If Yamaha had created stands for the speakers I'm sure we would have bought them.

One thing I have noticed is that when Digital TV advertises the separate "Dolby" audio channel, it always seems to be only 2 channel output. This means that the Yamaha correctly detects the input as 5.1 channel input, but only two of the speakers are working. I'm not sure if this is the TV broadcasting or the Opentel box.

In a previous post I mentioned importing a python file as a config file via ihooks.

Craig Ringer sent me a note, pasted below

I think there's a considerably simpler approach than the one you've taken, as well as an alternative to ihooks that's quite a bit tidier to use.

First, the simple alternative. The `import' statement pretty much just calls __builtin__.__import__(), and you're quite free to use that in your code directly. This lets you do something like this for your config importer (based off your posted code):

    import os
    import __builtin__

    try:
        config = __builtin__.__import__('config', globals(), {}, [])
    except Exception:
        # We catch any Exception because the user could do all sorts
        # of things wrong in the import. We can't even guarantee that
        # an ImportError comes from our __import__ call and not an import
        # attempt inside the config file without delving into the backtrace
        # using the traceback module.
        print "An error occurred while importing the config file."
        print "Traceback follows."
        raise

    def config_example():
        print config.some_config_dictionary["option"]

    if __name__ == "__main__":
            config_example()

It's worth noting that if I recall correctly the \_\_builtin\_\_
should be used; \_\_builtins\_\_ is specific to the CPython
implementation.

If you start having to accept an explicit path for the config file
from the user or something then ihooks is probably better. For
simple cases, \_\_builtin\_\_.\_\_import\_\_() is much quicker and
easier to understand though.

If you do decide you need more than \_\_builtin\_\_.\_\_import\_\_()
has to offer, it may be better to look into the PEP 302 import
hooks, which are much cleaner and simpler than the rather ancient
ihooks module. I should probably submit a patch to the ihook docs to
make them refer to PEP 302, actually.

Some info on PEP 302:
`http://www.python.org/doc/2.3.5/whatsnew/section-pep302.html <http://www.python.org/doc/2.3.5/whatsnew/section-pep302.html>`__

Indeed, __import__ was my first attempt at doing what I wanted; but it is a little off the semantics I wanted. I wanted to be able to specify a file on the command line with an argument --config=/path/to/file/config.py and have that loaded in. For example, __import__("/path/to/config.py") doesn't work.

Import hooks I looked into and are a possibility, but these also don't quite give the semantics I wanted. Here's a minimal example of how it might work

import sys
import imp
import os

path_to_config_file = "./testdir/config.py"

class Config:
    def __init__(self, path):
        print path
        self.path = path

    def _get_code(self, fullname):
        # path_to_config_file might have come in via our options, etc
        f = open(path_to_config_file)
        # read in the file and compile it
        c = ""
        for line in f.readlines():
            c = c + line
        code = compile(c, fullname, 'exec')
        # false means this is not a package, code is the compiled code
        return False, code

    def find_module(self, fullname):
        # sufficient to return ourselves
        print fullname
        return self

    def load_module(self, fullname):
        # get the code
        ispkg, code = self._get_code(fullname)

        # create a new module
        mod = imp.new_module(fullname)
        # add it to the modules list
        sys.modules[fullname] = mod
        # set the file
        mod.__file__ = "<%s>" % (path_to_config_file)
        # set the module loader
        mod.__loader__ = self

        # exec code
        exec code in mod.__dict__

        # return the module object
        return mod

sys.path_hooks.append(Config)

import aconfigfile

print aconfigfile

aconfigfile.hello()

The issue I see with this is that the import aconfigfile isn't really clearly showing what I want. I really want to be able to pass any arbitrary file name to be imported, e.g. something like import ./a/config/config.file. As far as I can see, import hooks are more designed to import files held in, say, a zip or pak archive.