Friday, 30 April 2010

Shared Libraries n Stuff

Hmm, another long weekend followed by what turned into a long week. Damn I'm tired.

Well I gave up on building XBMC on the beagleboard for now, and I haven't had time to work on setting up a cross compilation environment yet either. Work is getting in the way too much, just too worn out to concentrate enough. I should try this weekend, although I need to get outside a bit too for some exercise and my own sanity.

I spent the rest of the long weekend past trying to grok shared libraries and thinking about how to get newlib into one for WoofƆs. I didn't make a lot of progress. It would be nice just to use something like Amiga libraries which are more like objects, including an instance pointer to all functions, but that just makes it too hard to re-use existing code. Partly because all the code also must be re-entrant and not reference .data or.bss, not to mention the object pointer. Easier to map/load, not so easy to port existing libraries. I just about had a handle on the whole elf-shared-library thing, but I was a bit too tired/distracted so it's still a bit up in the air.

Well at least one nice thing is it only takes 17 seconds(!) to do a complete build of newlib on my workstation, althoughI fear I will need a few iterations before I work it out. I was trying to enable _REENT_ONLY, but it seems to be an old option which is no longer supported and fixing it all up will be more hassle than it's worth, so I will probably just have to work out elf shared libraries to make any progress there.

It's not that I really need a C library just yet, but I was thinking about how to write the application server and how to hook system startup into application startup, i.e. accessing dynamic memory and so on, and so I was just thinking of killing two birds with one stone. Might have to leave that on the back burner for a bit, and try something a bit more modest. Actually I should probably put it all on the backburner for a wile, I have other things that need attention.

I didn't do anything on the yard at all last weekend - a bit hungover Saturday and then the weather turned. It's been a bit cold and crappy most of the week (in relative terms) which is always pretty draining. And the days are so short already ... (particularly when you get up at 11 or 12!)

Sleep. Damn. Eating poorly, sleeping worse. It adds up. My mouth has been a bit sore so I have avoided using the mouth splint every night - but boy do I notice if I don't use it. It's like the `Rain God' truck driver in So Long and Thanks for All the Fish who upon experiencing rain so heavy he imagines it makes no difference whether the wipers are on or off tries turning them off, only to discover it really does make it a lot worse. However to his dismay it doesn't get any better when he turns them on again. That's exactly what it is like with the mouth splint and sleep apnoea.

Well i'm glad I stayed in rather than going to the pub tonight - starting to feel a bit weird, and last weekend was too depressing - my drinking buddy went off to a show and I had a couple more at another pub, and what I can remember got pretty boring by the end.

Tuesday, 27 April 2010

Accepted for GSOC 2010, XBMC acceleration

Well the students and projects have been announced for the GSOC 2010, and the beagleboard project ended up with 6 slots, which is quite good considering it is their first year (and I was expecting only 4).

And one of the projects I put my name down as a mentor made the cut - the XBMC acceleration project which seems to have been a favourite amongst all potential mentors. Since we knew what projects were in the top running I have already had some contact with Tobias in preparation, and he seems both keen and also good at communicating with the XBMC project which will of course be vital. Well I'm not sure what I will have to do to help as it is not a project I've worked on but I guess I'll learn as I go too.

I have also been trying to build XBMC on a beagleboard (silly I know - I need to set up a cross-dev environment for it) - wow, what a beast of an app. I spent about 3 days just setting up a new angstrom and the build dependencies (I just did a bit when I walked past and noticed it'd finished the last bit/failed again). Very slow - also doing it over NFS so that probably isn't helping. And I don't really need to, as I'm using my old PS3 hard drive as the system drive - it's much faster/more reliable than flash cards are.

Just as well I don't have 'a life' - i've got plenty of time to play with this stuff without any distractions.

PS Sorry to those who didn't make the cut this year - there were a lot of applications and plenty of competition.

Friday, 23 April 2010

Java n i/o

I needed to write an application to process very large files, and it had to run on M$ stuff. Well for me that means Java since my workstation is GNU.

I would normally use C, and would definitely use GNU/Linux or some other system with a modern filesystem on a task like this - NTFS is so god-awful slow. But that wasn't my choice.

Just to make sure i/o would be as minimal a bottleneck as possible I wrote a very simple async i/o loop - I don't even know if makes a lot of difference because Linux does quite aggressive read-ahead anyway, but it certainly doesn't hurt it. Java 7 is supposed to have an async i/o interface that maps more directly to the OS one, but that isn't available so I just rolled my own.

I used the 'nio' interfaces - just for the buffers really to avoid excess copying. Oddly searching for 'async i/o' brings them up but they only have a non-blocking select type thing which has no use in this application.

It's pretty simple anyway. First a bit of setup in the processing thread.
        final ByteBuffer sentinal = ByteBuffer.allocate(0);
final ArrayBlockingQueue queue = new ArrayBlockingQueue(numBuffers+1);
final ArrayBlockingQueue done = new ArrayBlockingQueue(numBuffers+1);

// must be one less than queue size - room for sentinal
for (int i = 0; i < numBuffers; i++) {

The main idea is using the blocking queues as a read-ahead mechanism. They seem to have a fair bit of overhead but it can be easily tuned by using different sized blocks.

The sentinal is used to mark end of file.

Then a thread is setup with a simple loop:

                    while (true) {
ByteBuffer bin = done.take();
int len =;

if (len > 0) {
} else if (len == -1) {
} else {

It took me a little while to realise how to use the ByteBuffer properly - to reset it after reading so the processing loop can read the same data, but it's pretty straightforward.

The processing loop is then simply:
            while ((bb = queue.take()) != sentinal) {
... do stuff ...

The processing loop can get a bit 'hairy' if you want to avoid copying data around too much, but this is the same in any language.

For comparison I whipped up a simple C version that did the same processing but with just the Linux read-ahead. The overall processing time (taking out java startup) is identical because - surprise surprise - the system is completely i/o bound. The Linux read-ahead is obviously quite aggressive too - which might not be so good if you're doing more random access.

The C version used 1/10th of the memory and about 1/4 of the CPU time ... but it had a better inner-loop, and the Java has a nice GUI as well. The C one only took about 20 minutes to write vs a few hours for Java - but I'm more familiar with C, and particularly writing this sort of stream processor to run efficiently in C.

Anyway it was an interesting diversion from my current work, and a good learning opportunity.

Another week down

Blah. Been a long week - got a bit caught up with hacking away so had some pretty late nights, and then early mornings from phone calls or visitors. Hmm.

Some friends are in town from OS next week so that should be nice, hopefully I catch up with them and their new twins. I'd been trying to call for the last few weeks - after not having seen them for about a year - and finally got through at some silly time on Saturday morning. We also tried the PS3 video phone application (alas, i upgraded my firmware in the end ...) which wasn't too bad - a bit like those old 'hello from space' videos, but a bit clearer and sometimes the framerate was very good. Audio delay was noticeable though. First time i'e used a video phone, and it's a bit weird talking to someone through a window in another lounge room (maybe all the stranger stranger for it being 8am in the morning after being up all night). Didn't know where to look.

We've just had some great weather and it's a pity it looks like they'll arrive on ANZAC day - which is typically the obvious start to winter, and it looks like this this year will be a typical year. After hibernating for a Chicago winter i'm sure the week of 25+ we just had would've been nice.

On the international traveller front it sounds like my sister has had enough of the ignorance and absurdity of the country, and sick of not being able to get decent work and is heading back from America fairly soon. She's only been away for a few years but she might get a bit of a shock to see how far down that path we have also gone. Even if she doesn't decide to stay long, I'm hoping she cleans up my 'other house' that's sat empty for a couple of years collecting dust anyway!

Speaking of ANZAC day - that's a pretty good example of how the nation has become a bit feral. How a total rout and a day meant to remember the utter stupidity of war got turned into a day of nationhood and the glorification of `fighting for ones country' i'll never know. I always thought it was about remembering the poor diggers who were mis-used as pawns in political games or by incompetent chains of command, and the 'least we forget' was about the stupidity of sending them in the first place, not the 'honour of their sacrifice' - typical right wing bullshit. Now it all seems to be about draping yourself in the flag (a flagrant violation of the honour of any nation's flag - and they didn't even fight under that flag anyway - it was all as British subjects) in a sort of quasi-national day and thinking of it as the day 'we grew up' - I can't really say if it was, but we surely have 'grown down' since if this is the thought of the common man these days.

Monday, 19 April 2010

Hello world

Well I integrated a timer interrupt into WoofƆs and got that to work eventually - the MMU makes things like this very fun as I have to map everything I need to use. A few silly bugs in the MMU code didn't help either.

After a 7 hour stint last night I finally booted up an executed a 'user process', including sending it the initial welcome message. It doesn't do much other than call a debug printf function to say hello, but given that it is running in it's own address space successfully that seems a reasonable achievement.

Interrupts are going to be a bit of a pain. Because I want user code to deal with interrupts the kernel has to clear the hardware interrupt status bits itself - which means it needs to know about all the devices that might produce interrupts and how to turn them off. I can't see any way around this right now. I also need some way of letting the user-code know what triggered the interrupt, which is another issue - even with 96 interrupts there is some double-up on the interrupt source. I guess there's always the mailbox idea, or a syscall to query it. There might be some hardware where the interrupt needs processing immediately too, but i'll come to that when I find it.

Hmm, next thing might be to think about devices and get a serial device working.

Saturday, 17 April 2010


Finally got timers `worked out' - i'd played with it before but couldn't get it to work - but it turned out that the code was basically fine, but I had a bug in the irq `enable' routine ... der. They are really rather simple devices that work much the same in almost all microprocessors ever made ..., so it was a bit embarrassing it took this long.

It's using the 32 768Hz clock - which means you have to use some extra hardware logic to get a true 1Khz divisor, but the manual explains that pretty clearly as referenced in the source.

Source is in irq-timer.c.

I'm at the point in WoofƆs that I need at least one timer to get much further so it seemed a good time to poke it into puppybits. I managed to squeeze in a little time to play with WoofƆs over the last few nights and got the code that i'd written a few days ago to load and run - after a stack of mucking about as usual. It doesn't do much - with the trivial round-robin scheduler and no interrupts once the memserver task goes to sleep the idle task starts and is never pre-empted :). I had some linking issues, but the main problem was that for some silly reason I was calling some assembly from C as a function *pointer* not a function and it took me far too long to spot it.

But there's an excuse this time. It's been a really long week - 49 hours of work, quite a few very late nights, an on-site meeting I had to wake up at 7am for, a couple of midnight meetings for GSOC - one of which I missed because I fell asleep, and then I woke up at 2am and didn't crash again till after 8am this morning. Not sure 'tired' covers it. The sunrise wasn't bad though.

Thursday, 15 April 2010

Guess which application?

"I have fixed the score on that particular proposal to match the score allotted to it. It is a known issue that occurs randomly sometimes because this particular action is not run in a transaction."

No comment necessary.

Tuesday, 13 April 2010


Damnit. Damnit. I need a beer. Oh hang on, I already have one.

Turns out that the algorithm I was converting ... was broken on the C end. No matter how much I tried testing the individual parts, when I turned on the full processing loop it just didn't match. After re-implementing the entire algorithm in both java and c (removing all the layers of abstraction i'd added) they all just checked out - but both produced crap - the original just wasn't doing the same thing at all. Well that was pretty much 2 days wasted, ending in a nasty headache at that.

Well probably more time wasted, as the algorithm I thought actually did something wasn't doing anything useful after-all. Well at least I can re-focus on that, and not waste my time.

Along the way I had a bit of a fight with the swing layout classes. Oddly enough the main memory I have from very first GUI programming I did in uni was how much we all had to fight with the toolkit layout system (well, after just how dreadfully slow it was - something beyond the complexity of gtk but running on a single sparc with 30 other students). Layout always seems to be an issue ... although TBH I think Gtk+ did a reasonable job for the most part - but then again 6 years of programming in it probably helped me forget it's weirdness. I kind of have it working but now when I use a different theme the content renders over the pane border.

Mid-afternoon being stumped I did some work in the yard - clearing some crap out of the way so I can level things off, and started said levelling, so at least my day wasn't a total loss. Although I think I have a bit more earth than I thought I did, so I might have to find somewhere to put it ... I have a rectangle of space I want to put a raised garden bed in, so I might have to look at putting a wall around that now too to hold some of the excess dirt (i'd hoped to leave that for later). I had a dead apricot tree there which i'd almost hacked out (without an axe), and today I managed to break the rest of the roots with some brute force so maybe i've just made myself a heap more work to do! While laying a string-line to get the levels I pinched my finger with a retaining wall block - barely hurt at the time but now I have a huge blood blister on it.

Hmm, the beer's finally starting to work. For a bit there I thought I might need something a bit stronger ...

Melange stinks

So the GSOC project is run on this half-arsed pile of shit called `melange' ('the spice of creation' / 'soc' / geddit?). I've been so utterly and literally dumbfounded by just how poor this application is that I simply haven't been able to find the words to describe it, but I think i'm finally willing to give it a go. But I wont do it on the mailing lists because someone will get offended by the naked truth, and there's already been enough of a shitstorm any time it's mentioned.

... half-arsed
pile of shit
called `melange'

It's very very slow, very ugly, lacks almost all of the basic features you'd expect in any web application developed since about1998, uses javascript where it doesn't need it - interfering with basic browser operation, and is generally an exercise in frustration trying to use it to do basic tasks. And on top of all that the project maintainers seem to be yahoos who like to do things like drop in new versions in a live system in the middle of critical periods of the process, and don't see 6 hours of downtime during said critical limited-time phase as any sort of problem.

Yet when anyone criticises it, the tired old canards of 'send us patches then' 'give us feature requests', and so on get dragged out - at best. Or they get all defensive and agro about it. I believe it's written in Python, so that vastly limits the pool of competent programmers even able to send patches. I believe it runs on googles app server which further limits the programmer pool - both for physical reasons and political ones ('fog computing' et al). And since the application is so poor at demonstrating its basic worth or indeed raison d'être - just why would anyone bother? The latter point is of particular note when it comes to feature requests - if they can't even manage the basics well (after two years?), I have zero confidence they could ever get anything more complex achieved.

This is not some toy project someone is doing in their spare time - this is a production system being used by thousands of people relying on it to ease what is a pretty difficult and tedious process to start with, and to decide how to dole out millions of dollars of real money. It shouldn't be getting in the way at every turn. We shouldn't all be beta testing what is barely a beta-level prototype at this point - this is the second year of it's use, and if that's all I came up with after all that time i'd be seriously consider whether I was capable of ever getting it to work. Even after just one day of failures (like yesterday) i'm questioning my own path.

Some of the mentors were discussing the provision of basic features like history and version control a-la-every-other-document-management-system-since-1970 and Chris Dibona had the inanity to suggest the application needs to `get simpler', not add features! What sort of nonsense is that? The app is difficult to use because it is too simple, remove any one `feature' and it would be utterly worthless - it really does the absolute bare minimum, and does that poorly.

I think what most frustrates me is this attitude more than the application itself - one, that being 'open sauce' you should expect to be a permanent beta-tester of alpha-grade software, and by implication that 'open sauce software' is always barely usable crap (which is clearly quite FUDish). Two, that you're all so privileged that Google is allowing you to bask in their glorious name in any way that you should be thankful for anything they do provide. Third, that is basically works so what are you complaining about. And finally that it's better than what they had before even if it's only because it's not proprietary.

It might not be a Google product, but their name is all over this whole thing and all it does is tarnish that name. As it does of their application server. I just hope I don't have to continue to deal with it beyond the acceptance stage (if I end up being a mentor), but I fear it is a critical component of the entire programme.


Shit what a day. I've been trying to get some @#$@ algorithm working in Java after converting it from C. I've verified that every stage produces the same numerical result for the same test data, but when run in the application all it produces is total rubbish. I know when I find the problem it'll turn out to be a one line bug somewhere ... i've had more than enough now, i've already spent 5 hours on it, or 7. Time to give it a rest, although I guess it's something to 'look forward to' tomorrow too, because it's a core component and I have to get it working asap. @!#$@

I haven't had much time left over lately to do much beagle hacking but i did a bit more last night - really just getting the last lot of stuff I did to compile, and filling out a few more little details. Although all that late night and tired hacking is certainly going to make for broken code. Next i'll have to get it running on the board - which I imagine will take some time. This includes a user-process as well, so once I get that running I can really start to test everything - mmu code, ipc, and so on. Hmm, still need a jiffy clock though ...

I spent a few hours on the weekend pounding dirt into the dirt - compacting the ground in preparation for levelling it off. Quite surprised at how soft it was - it generally compacted up to 50mm. And this ground was under paving originally ... Also pulled out an old wisteria that i'd soaked up with glycophosphate and thought i'd killed - although it was still well alive under-ground and shooting all along the roots. Bloody hell talk about strong roots (and lots of them), even a pencil sized root was too strong for me to break when it ran too deep to pull further - and I don't think i've put as much effort into anything for quite some time - even the spade had trouble cutting through it.

Cat jumped up on me this morning ... and bloody slipped! In his falling he tried to grab on harder ... damn his claws are sharp. Nice big scratch across my waist ... Still, the Java ended up more painful today.

Thursday, 8 April 2010

Java, FFT, Stuff

Just received what appears to be Intel hiring spam on the Beagleboard list. I hope technology like the beagleboard eventually wipes their sorry criminal arses off the planet, the last thing I want is to hear about their stinking jobs.


Anyway. Java n stuff.

I kept working on my prototype and added the Apache Maths library mostly because I wanted a simple Complex type that was supposedly written by Java people who know how to write such things. Got a bunch of algorithms converted and just ran it a few times looking at the printf's just to admire my handiwork.

Then I saw the profiler button and thought what the hell, lets see how it works. Well apart from being pretty friggan kick-arse (I can't even remember the last time I ran a profiler over C, but I can tell you it sucked - Visual Studio had absolutely naught without 'paying extra' for some shareware crap) I noticed huge bottlenecks all over the place. For one my image loading was taking about 70% of the time (loads 10 images, does a few fft's over part of them). Well I was flying blind on that one so made a bit of a mess of it. Anyway, did something about that .. next ... the complex arithmetic and the array references were taking all the time. SST arrays are accessed through multi-dimensional capable getters, which are abstracted a couple of levels to an index calculation that loops over an array of strides. Yes very efficient. For some reason I imagined the 'hotspot' compiler could remove all that, but alas it cannot.

Anyway ... to cut a long and rather boring story short, I've thrown away apache.maths's Complex type and written my own that has mutable elements and is used just to hold values. I found a decent FFT library in Java and since that was pretty much the only reason I was using SST I also got rid of that and wrote my own 2D array classes, and re-wrote most of the code to use simple one-dimensional arrays directly.

Man it's ugly code - but it was pretty fugly using apache.maths.complex too - the C is much, much cleaner for the inner loops. For all that it's about 10x faster than it was and on par roughly with the C code (don't quite have the same processing to really compare yet), which i think is worth it. Even if it took most of the day, although I cleaned stuff up as I went too so it wasn't wasted. Well i'm about where I was with the C a week ago, but in the meantime there's been a 4 day long-weekend and now I can easily add GUI. I looked at gtk+ very briefly but it just looked like no fun at all. And besides being cross platform at this point is a big plus.

So along the way I found JTransforms - this is the nice Java FFT library. I'm not sure why it took so long to find - I looked all over the place and all I could find were dinky little implementations. And lots of people looking for decent implementations. And really old stuff. Maybe it's the age of Java, or maybe google just failed me yet again as a search service.

Another tool I found along the way was ImageJ - actually I'd looked at it a few weeks ago but not in any great detail. The AWT interface is a bit clunky, but it is pretty fast. From that I worked out how to access the image buffers directly so sped up my image loading about 100x (well I was using 'getpixel' basically, now just a byte array directly). I don't really know if the gimp's implementations are particularly good/bad/or whatever, but doing the same sized guassian filter for example over a 1024x768 image noticeably faster in ImageJ.

... Visual Studio ...
... a rotting pile of
pig shit

Oh here's a funny thing which will probably be very easy to get used to. In the debugger (or even running the code), `fatal' errors like array bounds exceptions don't mean the end of execution. Oh yes I know, welcome to the late 90s, but .NET in VS was a pig, nay a bloaty fat diseased pig, oh lets not mince words; a rotting pile of pig shit, when trying to debug, and fatal errors were fatal too, so I forgot it doesn't have to be that way in so-called 'modern' languages. Not that Netbeans has been too kind to me, I've had a few pointer grabs and it brought system down twice (or maybe SELinux did in its own 'denial of service' attack for added security, if i understood the stack trace at all - it's now disabled), one pointer grab required hitting the power button to reboot, and the other required the reset button as trying to run any programme ended in an oops. Not having much luck debugging threaded code either (it just 'vanishes' if it has a fatal error).

Spent a bunch of time going through some GSOC proposals ... but I don't really know what to say and don't have any examples to go on. Boy that 'melange' thing they're using to track everything is a real crock of shit, really really slow and painful to use - I can't imagine how it works on a project with dozens of proposals. I didn't even realise there were more applications on another page of that fugly 'macos themed' javascript spreadsheet/table view thing it lists everything in.

Oh yeah and so much for getting some dry-work done in the back yard. Later that night it started bucketing down, and bucketing down, and it then rained a lot - around 32mm in a few hours - a lot of rain for here, and particularly loud on a tin roof. Barely a drop for months and then all that. The yard is still totally soaked 2 days later.

Monday, 5 April 2010

That's it for summer I guess.

Daylight savings just finished here - and it's already throwing me out. After plugging away at the computer all afternoon I thought it must've been 10 or something as it seemed to have been dark for so long, but it was only 7:30 ... Good time to call it a day I guess.

Yesterday I got a whole lot of work out of the way on the retaining wall, I should've done more today but I was too lazy. The weather was too nice too - although then I just spent most of the day inside instead of taking advantage of it. Once I sit down to hack it's pretty well game over ... Probably should've made the effort though - looks like weather is turning wet soon, and I need to get some of the stuff done whilst it's still try. Maybe an hour or two over the next two days might do it.

Well that's Easter done, back to work tomorrow.


Was up pretty late hacking away on some WoofƆs stuff, and then continued it for most of today. For the most part it's just re-arranging stuff I already have from puppybits, or my previous x86 hacking. Pretty time consuming though, trying to tie it all together. It also needed quite a bit of re-thinking and re-jigging along the way.

I've got it all building, and it currently launches two tasks, one which immediately goes to sleep waiting for work to do, and the idle task which blinks the led. But that isn't really any more than I already have in puppybits, so I need to start testing the other stuff - per-process virtual memory, message passing, and so on. These things are a bit hard to play with in isolation since so much support is needed first. Hmm, I really need to get timers and timer interrupts working too, but I can test that in isolation.

The architecture which i've thought about so far does seem to be holding together at least - no big surprises have come along ... yet. Although that assumes the stuff i've written actually works too. So far I have a 'memserver' which is tightly bound to the kernel - it is basically a kernel thread since it accesses the kernel memory directly. It is used to create all in-kernel objects - directly adding them to the kernel in some cases, or where necessary through lightweight system calls to register the new resource. This lets me avoid any dynamic memory in the kernel itself, and by using the right data structures or the occasional lightweight system call I don't have to worry about serialisation either (well, once i've got it right).

I had been thinking about all sort of exotic data structures like trees or hash tables to locate resources based on an object id - but these all have various issues. Execution time, serialisation, and so on (I have implementations that use no dynamic memory, so that wasn't an issue). So in the end I settled for a simple array for many of the objects - it probably uses less memory anyway, and certainly needs less code. It also allows me to update and access it atomically from multiple threads of execution without trouble.

The whole 'kernel' is only 10 system calls so far - and although I still need a few more for interrupts, it shouldn't be many ... I think. Of course most of the work is in the servers, and the kernel is just passing data around. Even in those 10 I already have some 'helper' syscalls too - they aren't strictly necessary but combine a couple of system calls into one, there is probably scope for some more of those.

I had a little chicken and egg problem with message allocating - you need to send a message to the memory server to allocate new messages before they can be sent via the kernel. I think I have worked out a solution - part of the process start-up will be to send a system message to the new process. This message can then be used to ask for more, or just as a general purpose message container. Still, I might need some other more direct mechanism since message ownership passes every time they are forwarded, and it is possible for them to get `lost'.

I might have to look into a mailbox mechanism I guess, although I don't want to have too many options for IPC. After I'd written the basic message allocation system I thought it would be a bit bulky to do too often - well the point is not to in the first place, but it seemed a bit heavy. I looked into a simpler mechanism of message passing using limited registered arguments. But it just didn't seem that useful - because you can only pass primitive non-pointer types. As soon as you want to pass buffers around the mechanism falls down, and you need even more complex support code (like 'far' copies), which then needs to perform extra security checks, and so on. One `freebie' of the message passing mechanism as i've envisioned it so far is it sort of self-checks. Nothing the kernel deals with needs checking in detail because any addresses are from trusted sources or have already been verified, or just integer representations of virtual addresses it never looks at. And servers can perform fairly cheap checking since all data must fit in a fixed bound.

Might sit on all that for a while now.

Sunday, 4 April 2010

New project

Finally cleaned up and checked in the kernel start code, and a simple demo which uses it.

I still need to do some prototyping of various low-level components before I can make a whole, but I might kick off the part of the whole I can do already because i'm getting a bit bored of just working on these small fragments.

Keeping with the canine theme, i'm going to call it ...


Or maybe WʊfƆs, although I think the larger one is more aesthetically pleasing (the letters are based on the international phonetic alphabet if you were wondering).

I'll probably just keep it in the puppybits project, although keep the code separate. With work and home being as they are I haven't had much time nor energy to work on this stuff, so things aren't going to progress at any great rate. Speaking of which I think it's time to hit the yard and get dirty shoveling. Two days of Easter down and all i've managed so far is ~97km of cycling, a hangover, and an improved tan.

Thursday, 1 April 2010


CentOS isn't worth the paper ... oh hang on

After a few more hiccups I finally gave up on CentOS on my workstation last week - I don't mean to diss on the CentOS project and people as such, but I have to say I had a pretty negative experience with it and probably wont be considering it again (there just isn't any need). I hadn't tried it before, and thought i'd give it a go, mostly because I was expecting simple reliability.

Unfortunately not.
Lack of packages
Just how limited was a little bit of a shock ... but this was something entirely reasonable and had no major problem with this.

Lack of stability
This was both shocking and surprising. I had a lot of stability issues from day one. Primarily that xterm's constantly locked up when trying to do ordinary things like 'man blah' or 'less foo' (and I need to use the real xterm - the other `desktop project' knockoff's are unusably slow at full screen, and barely usable even at 80x24). I got around it using info, or emacs shell but it got old very fast. Firefox also had a habit of grabbing the pointer if I accidentally dragged something which is too easy to do. For the most part I loathe DnD because generally it's too easy to trigger and used in some really stupid places, and where it could be useful - e.g. dropping files into an application or filename selector, it isn't implemented, sigh. I wish GUI designers of the free `desktops' had a wider GUI experience than just MacOS and Windows 95.

Bloody slow (not just slow)
Ok I'm a bit spoilt with my home workstation and it is decidedly slower hardware, but this was a little ridiculous. I can forgive some of it because I chose to encrypt the home partition, but I really have no feel on how much that makes a difference. I'm sure EXT3 is something to blame here too - if anything demonstrates that the Linux project is no true meritocracy it is that this steaming pile is still the main filesystem of choice when so many more advanced and reliable filesystems have existed for years!

Updates broke it
What finally broke the camel's back was turning it on one morning and having no X any more because the update manager prodded me to install a different Linux and I wasn't paying attention. I am 'partly to blame' because I am using the proprietary nvidia driver (the other one just didn't work, and I need OpenCL/CUDA or something down the line too) from livna or something or other - but I haven't had quite such problems doing similar things in the past that were so difficult to resolve. After mucking about for an hour or two I gave up and just installed Fedora 11 (the newer ones are just fucked up too much to be slavagable within my patience band). That ended up taking the whole day (made a few mistakes along the way), but it was definitely worth it.

Now I don't know how much CentOS differs from RHL, but this doesn't leave a good impression of that either. The whole point of using old software with limited packages is for stability and reliability and it was on those two particular points that I had the most negative experience. Probably the worst I've had with any OS since automated package management existed (~10 years).

I've moved to Fedora 11 and have had no issues since. Faster, more stable, more packages, blah blah blah.

X, Java, Netbeans

Well, while i'm dissing on free software projects ... I don't know what they're doing wrong but OpenJDK has some major performance issues that simply make Java (and anything that uses it) look really bad (in all fairness I am using Fedora 11, so it is certainly quite an old release ...). I am doing some stuff in netbeans (6.8) and although usable for the most part, had a decidedly lurcherous gait which only got worse over time till it was barely usable. I resorted to using emacs for editing (it's still a much better basic editor), jumping back to netbeans only to look up methods (JavaDocs as useful as they are are a bit unwieldy to navigate, particularly in a web browser). I suspect it has more to do with something basic like the interaction with X than anything else - and we all know what a mess of The X Windows System that f.d.o are making so it probably isn't entirely `their fault'.

So I finally got sick of that and after a bit of searching on the net and trying "-Dsun.java2d.pmoffscreen=false" to no big effect, the only thing seemed to be to try the Sun JDK. *Cough* Oracle JDK. Apart from a different way of selecting fonts that initially made it look a bit crap, and a generally lighter appearance in the text you'd think I just upgraded the computer to see the difference.

It's like night and day. It might use scads more memory than a C app but it feels just as snappy and responsive (at least in comparison to how it was, hard to guage). The thing is, having such a poor user experience from the OpenJDK just continues to give Java a bad name when it just isn't warranted, and hasn't been for a long time now. I was originally just going to implement a GUI in Java (or maybe just a prototype) but I might try a few tests of the algorithms to see how it compares to the C code - it spends most of it's time in FFTW anyway at present. It's a real pity that it doesn't support a primitive complex type, but that isn't the end of the world. I'm using the Shared Scientific Toolkit at the moment, although mostly just for the arrays and fft ops (that's all the code really needs).


Oh god what is this pain in my head ... yet another messed up build system. One that provides all the craptascialness of Ant for your C++ projects too! Jesus, who comes up with this shit? I had some issues building SST - it turned out I just needed to install the static FFTW library, but along the way I had my first experience of CMake. About the only thing going for it is that it doesn't use XML. What I don't understand is why these things come along and break the whole point of make files? During the build something failed - so I worked out how to manually run the command and the command ran ok. Fine then, just run make and let it continue ... oh no, that would be too fucking simple wouldnt it? It uses some fucked up meta-system for tracking build dependencies, so it just re-builds the whole directory again, again failing on the final link. Then I ran a different make target which built ok ... but wasn't quite what I needed. No problem, try the original build target ... 'target up to date' ... huh?


The one and only point for any make system to exist in the first place is to guarantee your builds are consistent without having to recompile everything every time. For everything else you may as well just use shell scripts - which are conveniently embedded inside a Makefile. I understand CMake does some other stuff, and also provides a `shell script' environment that supports broken operating systems, and that idea has some merit - should you wish to interact with such a broken system at least - but if it can't get the basics right what use is it?

Ant is another nightmare all of its own - not only does it not do any dependency checking whatsoever (one of the most critical features of any build system), simple human readable shell scripting is replaced not only by a disastrous and unparsable XML scripting language but also by dynamically loaded Java modules! I can understand wanting to use your favourite language for everything, but Java is not a scripting language and sometimes there is a right tool for the job.

completely broken ...
... simply worthless.
... CMake ... Ant ...

I consider any tool where you must occasionally `make clean' to get a reliable build to be completely broken and simply worthless. CMake seems to get around this absurdity by abusing 'ccache' - another painful bit of kit that shouldn't need to exist (the limited use-cases for which ccache provides any service can mostly be done in other much simpler ways - assuming you have a reliable dependency mechanism in the first place). And Ant gets around this by the fact the java compiler is quite fast anyway, and already does the dependency checking - but only for the Java sources, and any real project has to do more than just compile objects and link them together. One often has to 'clean and rebuild' in all the GUI IDEs i've used, but there should be no reason to ever need this unless you're manipulating files outside of it.

Is this you?
What I find disappointing (vexatious, alarming, and upsetting too) is that people refuse to learn a basic reliable, flexible tool, and then come up with their own which doesn't even solve the original problem. `I can't use an editor to control the tabs in my file' is really an idiotic and puerile argument (if you do think that, then yes i am in fact calling you an idiot. Idiot.) I suspect if they understood the original problem they wouldn't have bothered to inflict all this crap upon themselves or the rest of us in the first place and wouldn't the world would be a better place for that?

I understand some people think make is a bit arcane .. but it isn't. XML is the definition of arcane - lets design a format which makes it easy to write parsers, for developers. You know that tiny bit of code that gets written once and forgotten about because nobody fucking cares once it works. It's not like they even got that bit right - the parsers are large and complex and the code you write using them ends up being large and complex too (and it's all slow). But the real winner is that this then inflicts all the nitty-gritty which makes it 'easy to parse' onto the user for whom it is decidedly not easy to parse, or write, or even design. Now that's arcane.

But I digress.

The auto*tools are a big bit of nasty, but at least it's only one (perhaps broken) system to learn, not 'n' (definitely broken) systems where 'n' is monotonically increasing. And there are plenty of application projects for which it shouldn't be needed (but gets used anyway, sigh).

Woke up too early

Shit, I was supposed to have some bricks delivered this morning - I was up way too late and got little sleep worrying about making sure I tell them to put it in the driveway to avoid 3 hours of back breaking work moving them. I suppose i should go get some groceries, or I might just avoid the shops which turn into a nightmare of panic buying just before Easter since many shops are shut - GOOD HEAVENS - for 2 whole days in a row. How ever will we all survive for so long without being able to buy more shit!