(tl;dr: use latest rdup for backups, and thecus has problems with some drives, even from the compatibility list, if so, try updating firmware).

So I recently saw a reasonably good deal on the Thecus N5550, a 5-bay hot-swappable NAS machine. This seemed as good a time as any to finally clean up my backups and Do Them Right.

So the plan is to see how much the already excellent rdup has progressed (last time I used it was at version 0.5, it is now at 1.1.13, the version from the repositories on my systems was 1.0.5).

Initial setup for the N5550 is quite OK, I setup NTP, NFS access, Samba access, and enabled SSH login to have a lowlevel look at what's going on inside.

Setting up the RAID array was a breeze; For my first experiment I chose a RAID-5 array of 3 Seagate 3TB disks, totalling in about 5.5TB of space.

I installed rdup on my vps, and on a local always-on machine (a lowpower fanless box acting as a fancy gateway), and initially I mounted the big drive as a samba mount (there were some unidentified problems with nfs4, see below). Read and write speed were not as high as I would have thought, about 1-3 MB/s. But that was a problem for later, and I don't really need that much speed anyway.

I only needed simple incremental backups, so the rdup-simple script should be fine.

Then the first problem arised; there were a lot of 'no such file or directory' errors. After retracing the steps rdup-simple takes manually, I found out that sometimes rdup-up missed creating a directory. And then of course when it tries to write files in it it fails. This happens for different directories on every run, so I suspect this is some form of race condition.

I asked around and a friend who uses it says he had never seen this with his version (1.1.11 IIRC).

Time to upgrade.

Ater getting rid of the repository versions, and building a fresh rdup from the latest release (long live apt-get build-dep!), rdup-simple proceeded without a hitch.

I also has a number of media files I wanted to copy, so I simply scp'd them to the mounted share.

And then the real trouble started.

While it was copying one of the video files, scp aborted with an I/O error. I tried to copy again, and this time the failed file worked. But shortly after that, while copying another big file, the thecus started beeping continuously.

Looking at the diagnostics, it reported a failing disk (drive 2). Since I had already run the smart checks on this brand new disk, I didn't think it was actually dying on me, so I tried to find what was really happening. Good thing I had that SSH access; dmesg showed lots and lots of problems:

Periodic exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen from sata drive
hard resetting link

followed by a number of messages:

device reported invalid CHS sector 0

And of course the RAID was in a degraded mode. Telling it to fix itself started a rebuild. As you may now, RAID rebuilds take a long long time, and after about 10 hours (at 92 percent), disks 1 and 3 dropped with similar errors. Bye bye RAID.

This seems to be a problem with the driver, as similar problems for other systems are all over the web. It is often blamed on bad cabling, but there are no cables in this machine (the sata connections are directly wired on a board, and if that board was bad, I'd expect to see much, much more complaints on the Thecus forum).

Another thing I noticed is that these are 6GB/s disks, and the thecus should handle them fine (they are on the compatibility list), but they are connected at 3.0GB/s, not 6.0.

This might be related.

So, I almost contemplated getting rid of the firmware and trying to install a real OS on the system. But there was one thing else to try first.

The version of the disk firmware was CC4B, and there were some reports with other Thecus machines that depending on the firmware version, some disks of the same model worked and some did not.

I put my disks in another machine and updated firmware to CC4H. Fun!

Steps updating seagate disks: - Boot my main Linux machine to windows for Seagate firmware update tool - firmware update tool reboots the machine into a mini linux version - which updates the disk - then reboots the machine (defaulting to main linux) - Repeat for other disks

Shoving the disks back into the Thecus, I got a small disappointment; the disks were still negotiated to 3 GB/s. Oh well.

Rebuilt the RAID array (this time as a RAID-1, I care more about redundancy than write speed and size). And this time I found out the nfs problem (or at least a workaround); when mounting with nfs4, the shares got a weird UID, and Invalid Argument if you try to chown something. However, mounting it using nfs3 makes it work.

Using RAID-1 and nfs(3) gave me 10 MB/s write speed and 2 GB/s read speed. Now that is more like it! (Again, don't care too much about write speed, but 10 MB is a lot better than 1). (EDIT: hmm, that was with a zeroed test file I just created, for normal files it appears to be about 10 MB/s read speed as well).

I have been stresstesting it all day now, and have just started the backups again. No disk I/O errors so far.

So if you get those, before doing anything drastic, check your disk firmware updates.

Happy RAIDing!

The wiki trap

Or: "Yo Dawg, I put a wiki in your wiki, so you can point to the wiki that points to your wiki."

A lot of projects use a wiki as a central repository for information about the project, which can be a blessing or a curse. I'll refrain from naming the specific project and wiki, but condensed, I recently had a discussion that went a bit like this:

A: Hey, I need to know how to do X, and more generally what the policy on Y is.

B: It's on the wiki. Look here <>.

A: No it's not, I've been there and it talks about the subject, but does not actually say how to do X.

B: Well then it should be, and I don't know either. It's a wiki, so please improve it.

A: Sure, but I can't if I don't know how to do X.

B: Please ask C.

A: Hi C, I need to know how to do X, and more generally what the policy on Y is.

C: It's on the wiki. Look here <>.

This went on for at least 3 rounds, until I finally found someone who actually knew the procedure and policy. He extensively explained it (thanks, btw), and then ended with:

You should have found this on the wiki.

After that I was, to put it mildly, a bit miffed.

Everyone was of course right, it should have been on that wiki, since that wiki was the project's central repository for exactly this kind of information. I'm quite sure I'm far from the only one who has had this experience. I've been thinking about it for a bit, an here's my impression of the general misconceptions on wikis, and the things you need to realize when you decide to use one.

Wikis are not magic

A wiki can be a powerful tool; all information instantly available, everyone with access can improve upon it, and since history is retained, erroneous additions and alterations can easily be put back. But they are not magic. Setting up a wiki for your project or company does not mean you will automatically end up with all information everyone needs.

Wikis do not reduce the amount of work

Somehow, it seems that people are under the impression that a wiki takes less work than 'traditional' forms of (internal) documentation. This is not true. If anything, on the whole a wiki takes more work; the major difference is that this work can be distributed better over the group of people that need and use it.

Wikis do not stay up to date

This might be the biggest problem. Wikis do not stay up to date. They age as easily as other forms of documentation. In fact, it's quite easy to add information in a place that might not easily be found by someone who wants to update it, prompting them to add it somewhere else. And then you'e even worse off. Once you have found a piece of information somewhere, do you check through the rest of it to see if anything else is more recent?

So while a wiki can be a powerful tool, it can also just as easily (or even more easily) transform into a rotten mess, where information is missing, unfindable, contradictory, or plain wrong.

So how does one prevent this?

Use it. Seriously.

The first step, even before you actually start using a wiki, is to decide to use it. And by use it, I don't mean install some software, put in some starters, and declare that this is now the Official Way. When I say use it, I mean use it. Everyone involved in the project should commit to writing, reading, and updating. Especially in the beginning, or once you have come to the realization that your wiki is a jumbled mess of loose pages, contradictory information and indecipherable explanations, make sure everyone commits to use it. Daily. For everything that is related to the information in it. Even if they are quite sure they know it, they should still check the site if the information there matches their view of it.

The point here is not just to get the wrong bits out and the right bits in. The point is to engrain everyone involved with the notion that this is an integral part of the process. Things change. Processes change, requirements change, and the wiki is the place where you decided this information should be should change with it. This really only goes right once it is such a central notion that you don't really have to realize it anymore, and the checking and updating of small things does happen automatically.

Have some structure

Second important thing is to have a Plan. You'll need a wide overview on what information is supposed to go in, and where it should go. One of the powerful features is that you do not have to design for every little detail, since the users can add those themselves, but you do have to come up with the global structure. If it is not quickly clear where one is supposed to put a piece of information, they will put it where it 'seems right'. And a lot of different 'seems rights' can get very wrong. But if you have the right basic structure in place, things tend to fall into place much more easily. In the end, this will ease up both searching for information and updating it.

Update it

And the third one, if something is changed, make it a task for someone to update the wiki on that regard. If something is decided by multiple people (like in a meeting), immediately discuss who should update the information. If something is decided by someone by himself, he is responsible for updating it. This too goes much easier once the first stap has been done; it should happen automatically, but still explicitely.

Take time every once in a while to get a feel of the overall structure. Call it a structural review, if you will. Sit down with someone else and try to find random bits of information. Click through it and see if things seem out of place. Clear your mind for this, it's quite easy to miss things that are weird but you have gotten used to.

A wiki is not a static piece of design; it's not supposed to. You don't need to look into every little change to see if it's right. And its power comes from exactly that. But like a lot of things, it does need a little careful grooming for it not to become one giant mess. And if it is regularly tended to, it can approach what seems like magic. Fairies will fly by and sprinkle your documentation with magical information-dust. And all live happily ever after.

Oh, and yes, I have updated that original page :)

Happy editing!

Disabling Sense on the HTC Desire

I have been playing around with my HTC Desire today, and wanted to see what it was like without the Sense interface. If found several small tutorials online on how to disable (en re-enable!) it, which contained about the same steps, differed a little bit on how extensive those steps were described, and were all completely similar in that none of them actually worked.

Shortly the steps are described as follows; In Sense, press Menu Select Settings Select Applications Select Manage Applications Find "HTC Sense" Click 'clear defaults' Click 'stop application' Press Home button * Select the non-Sense option

Now all of those are indeed correct, but they seem to be based on a pre-retail version of the Desire. When you try this it all goes well until you press the Home button. At that point the Sense UI simply gets restarted again.

The reason for this is that apparently HTC has removed the Home application, and the trick to disable the Sense UI is to install it again. Now I could not find Home for download itself (it's expected to be included I guess), but there is a Home++ beta application, which seems to work fine.

If you download Home++ from the Android Market and then perform the steps described above it works like a charm.

You do want to be a bit careful here, I also tried AdvancedLauncher, but it crashes on startup on this phone, resulting in an endless crash-and-start loop that I couldn't get out of except by removing the phone's battery.

Also note that this only disables the UI, not other Sense-related background tasks. Still if you just want to try it out, now you know how :)

It's here! It got in yesterday, making the total count 134 days.

I see that by now the game has gotten cheaper, though the package was gracefully accompanied by no apologies at all.

Thanks Play.

Play.com: 3-5 days delivery. Unless it's longer. A lot longer.

Being a pretty regular customer at play.com, with around 60 orders placed over the last couple of years, I never really had to deal with their customer support. Once, quite some time ago, I had an order that didn't arrive. I sent them an e-mail, and they sent another copy. Very nice.

So a while ago a friend and I decided it was time that we go and tackle Resident Evil 5 together. We both ordered it (of course at play!) and agreed to set apart a bit of time to play the coop mode, since the single-player version isn't that great, if the demo was anything to go by.

Being in the Netherlands, we're used to orders taking up to a few weeks sometimes, I guess it's something to do with customs over here. So when the order didn't arrive, I waited until more than a month had passed before i reported the item as missing.

Full of hope I went back to the site to find the e-mail address I used last time to report a missing order. Turns out you now have to apply for a form, which is then sent to you, you have to print, sign and send back over snail-mail.

But sure, I can see that they don't want everyone to report fake missing items, and that there should be a bit of a hurdle to get your order resent, so I just printed and signed it, and sent it back.

And sure enough, after about a week I got an e-mail:

We are sorry to hear that you have not yet received your order. A
replacement has been ordered and will be dispatched as soon as possible.

Should you receive the missing order in the future please contact our
Customer Service Team on 0845 800 1020 (UK only) or +44 (0)1534 877 595
(outside UK) between the hours of 9.00am to 8.00pm Monday to Friday, and
9.00am to 5.00pm weekends.

Please accept our apologies for any inconvenience this may have caused you
and thank you for your patience and valued custom.

Kind Regards,

Play

"Cool", I thought, "We're gonna be playing this somewhere next week."

That is now exactly 58 days ago.

Turns out, by the time my missing item form got processed, the original one was out of stock. And apparently having outstanding (missing item) orders isn't really a reason to be quick with restocking.

After a few weeks I sent them an e-mail (yes, contrary to what the site seems to say, you can actually e-mail them), but of course I got a standard reply'

Please be advised that your outstanding order is currently in the process
of re-stocking.

Whilst we cannot provide you with a restock date at this time your order
will be charged and dispatched as soon as it becomes available.

Please accept our apologies for any inconvenience this may have caused you
and thank you for your patience and valued custom.

Kind Regards,

Play

Never mind that they had another version of the same game for the same price, still in stock (just checked it, by now that one is also sold out).

All this time, every few days my friend, who already has the game and wants to start playing, asks me whether it's in yet. How nice of him to wait for me :)

Of course I could just ask for a refund, but by now I'm actually wondering how long this is going to take.

Bets may be placed.

Since the original order, we're now at exactly one hundred days, and counting.