I was visiting Tesco Campona for my favorite blend of Trappista cheese (Füredi), and as I was looking for a copy of Restaurant Empire II, I found something interesting that made me think about software prices.

Right next to each other, there were three items.

1. for 11 Euro: Gothic 3
2. for 14 Euro: Gothic 3: Forsaken Gods
3. for 10 Euro: Gothic 3 Gold (Gothic 3 plus Gothic 3: Forsaken Gods in one box)

Of course, it would make sense if this would be the Matrix vs. the Matrix triology ;)

There is a firm who likes Linux SW firewalls. As they are doing quite complicated filtering (DHCP bridging over VLANs and other nifty stuff) and they have gigabit and fiber connections and as the are cost-savvy, this is OK.

Yet, one day, something went wrong: the firewall, which was working flawlessly for a while, refused to start and had RAID problems. The cause was a bit suprising: the UPS, which was connected with USB to the firewall, caused the problems - more precisly, if that UPS was plugged into the back USB ports of the machine, the kernel (2.6.8) would fail to load - of course, when it was plugged into the USB port on the front, it worked like a charm. And as the service personnel used the front port for "testing" and the rear port for "production", the kernel failed to load after every long blackout and the RAID become dirty from the half-boot.

The solution was a sticker on the rear USB ports with "do not use these ports" written on it.

A task that sounds simple: install a new vserver on a SW RAID.

It is not so simple, though.
When you want to create the mount point with "mkdir /var/lib/vservers/nvs", you get a permission denied even though you are root.

Some SysAdmins might spend hours with Google, instead of simply

lsattr /var/lib | grep vservers
chattr -i /var/lib/vservers
mkdir /var/lib/vservers/nvs
chattr +i /var/lib/vservers

A self-employed accountant client had a 250 GB HDD partitioned into four partitions, storing some valuable (accounting) data on the fourth partition and some family photos on the third. Then, data hit the fans.

Suddenly, when the faulty video card borked the motherboard, an "expert", who had an older motherboard without LBA48 support, tried to reinstall Windows XP after replacing the MoBo. Of course, only the first partition was visible and "all data were lost".

After finding out that there were no backup, the desperate client asked our help. Nothing else was neeeded than a simple

apt-get install gpart
gpart -b backup.bin -W /dev/hdc /dev/hdc

If you want to play with it, gpart offers more options, but in this case, it was not necessary. We first tried testdisk, which was able to copy files from the second partition without changing anything, but was unable to find the third and the fourth partition (at least, not as easily and quickly as gpart did).

What use is documentation that is unavailable?

Not much, so I am linking theese two here (might you find them with the aid of Google!):
How to compile your custom kernel vith vserver support under Debian Etch
How to patch and build OpenOffice.org on Debian Etch

Theese OOo guys are quite funny - after I reported a bug to them (and to Debian, as well), they kept marking it as invalid as it was not in the "core" part of OOo, just in the part which was used on Suse, Novell, Debian and Windows.

Well, after accepting my patch I sent them at 2007/Sep/14, they closed the bug with "closed, not relevant for official OOo". Well, at least not "invalid, not relevant for official OOo"

Anyway, the patch is in Debian Lenny and some OOo 2.2+ branches.

Having an Etch with kernel 2.6.22 coming from Etch backports caused me some extra work when I wanted to add KQemu to Qemu.

Qemu is a very nice virtualization tool - but in itself, it is quite slow on older CPUs. (Particularly, if you have HW virtualization support with Turion X2, Barcelona, Core2Duo or dual-core Xeon, you should consider KVM, XEN or wait for the KVM-QEMU merge). There is a KQemu kernel module, however, which speeds things up considerably. This KQemu was not GPL until recently, so in Etch, it was non-free and just recently changed to free, thus, now we want to install it with module-assistant (a nice tool to install kernel source packages).

So, after installing your linux-image-XXX and your linux-headers, you use module-assistant like:

apt-get remove kqemu-source
apt-get install kqemu-source
cd /usr/src
m-a prepare
m-a build kqemu
dpkg -i kqemu-modules-XXX*
echo kqemu >> /etc/modules
modprobe kqemu

Now, all you have to watch out is to load the KQemu kernel module and check that your qemu uses it. Hints:
- If lsmod | grep kqemu ends with 0 when qemu is running, kqemu is not in use.
- On AMD64, set machine type to x86_64 (or use qemu-system-x86_64 instead of qemu).
- Disable qemuctl from qemu-launcher to see what is going on in the background and to provide full KQemu support.
- If you do not use qemuctl, omit the ampersand (&) from the end of the command line of qemu-launcher to keep the Qemu monitor open, so you can suspend the VM, change CD images, send Ctrl-Alt-Del, etc.
- Full KQemu support seems to increase the chance of BSODs and Kernel Panics on a HP NX6125. What is more, Windows XP Pro SP2 can not be installed in full KQemu mode. Of course, after installation, it is OK.
- Do not set KQemu support to full, but leave it on enabled, unless you want see some kernel panic, if you are using the Etch Qemu package on AMD64. You can compile 0.9.0 as described by Weiqi Gao.
- If you don't have a /dev/kqemu, try modprobe kqemu.

Error checking is necessary to save time - in this case, an apt-get dist-upgrade on a Debian Etch, plus a manual install of SUN JDK 6u3 made a home-brown application go belly-up with:

java.net.SocketException: java.security.NoSuchAlgorithmException: 
Error constructing implementation (algorithm: Default, provider: SunJSSE, 
class: com.sun.net.ssl.internal.ssl.DefaultSSLContextImpl)

The problem was caused by two bugs and a default behaviour:
1. The key generator script did not check for an existing keytool in the path
2. The application have not checked if the key file is 0-byte
3. With the clean-up after the dist-upgrade, symlinks from /etc/alternatives were removed and with the manual install, there were no symlinks created from /usr/bin to the JDK's keytool.

Of course, it would not been so interesting if the error message would give *any* hint on the error.

Moving a vserver from one host to another produced an interesting problem: MySQL would not start.

After copying (tarred, of course) the config files form /etc/vservers/xxx and the root file system from /var/lib/vservers/xxx/ (which was a dedicated SW RAID partition), and modifying a few small settings (interface name, path to root filesystem), MySQL refused to start - the file socket was not created.

Although only the symptom was discovered, not the cause, as this was a production server and a live migration, we were happy enough to find an ugly workaround:

apt-get remove mysql-server-5.0
apt-get install mysql-server-5.0

At one place, there was an interesting problem with changing MAC addresses.

There was a Debian server with udev, used as a vserver host with some vservers having dedicated Fast Ethernet (100 mbps) ports. These ports were on QFE cards (Quad Fast Ethernet PCI NICs) which were acquired second-hand, and from different sources, all without warranty.

One of these cards was a Sun QFE Happy Meal (?), drived with sunhme. This particular card set random MAC addresses (HW adresses) during each reboot and udev recognized the card as "four new eth interfaces" as they were not found in the MAC address database.

The problem was that the vservers falled back to eth0 as their dedicated interface (example: eth5) did not exist.

The solution was to tinker a little bit with the udev config file /etc/udev/rules.d/z25_persistent-net.rules and bind the eth5...eth8 interfaces to PCI bus IDs - which is a rather harsh solution, and works only as long as the card is not moved to a different PCI socket.

BUS=="pci", ID=="0000:04:00.1", NAME="eth5"
BUS=="pci", ID=="0000:04:01.1", NAME="eth6"
BUS=="pci", ID=="0000:04:02.1", NAME="eth7"
BUS=="pci", ID=="0000:04:03.1", NAME="eth8"

When we have RDBMS-es (like, uhhm, Oracle), we often have databases in them. What is more, we have indexes (or indices, we use both as a plural form of "index") for those databases, and we expect our RDBMS to rewrite our queries in a way that it will use those indexes.

In one of the databases, there was a table with around 3 million records, with an index on primary key. When we did a "select X from Y where REC_KEY like '123456%';", it fetched 10 hits in 0.1 seconds. Then we did a "select X from Y where REC_KEY like '123456%' and REC_TYPE='Z';", and it took more than 12 seconds to return the 3 hits.

The problem is that every sane person would use the index to fetch all the records with "REC_KEY like '123456%';", and then filter out those that are not "REC_TYPE='Z'". Of course, this particular instance did not - it used no index at all. You would think that *any* index should be better than a full table scan, but noooo, RDBMSes are smarter and know what to do. Or at least, they should, but appereantly it was not properly maintained and the statistics were not updated. And because of the support contract with a third party, we were not allowed to do it, yet there was a need for a queries that won't take 12 seconds for each record.

The solution was to set autotrace and force the use of the REC_KEY index with index hints. This needed some fight because, of course, in this RDBMS, you need to run some scripts and set some privileges to run an "explain", and there was this support contract again.

As a developer, I got frustated by bug reports that end up being false alarms. If I'm lucky, the user did not found the manual in the docs directory - if I'm unlucky, they have not found out that there are directories at all (yep, there are plenty of jar files out there).

Today, I met the other edge of the axe, namely that my bug report on OpenOffice (using the platform-default character encoding, but showing another one in the combobox) was closed 3 times before I hammered them down with my devotion (and some help from the Debian staff).

Sometimes I wonder how often people mix up the "number of closed bugs" with the "number of fixed bugs".

Also, I wonder why would you run for these numbers if your salary does not depend on you and deep inside, you know you are not on the right path.

When another SysAdm wrote me that "The good news: it will not be hard to convince the boss to buy new UPS-es", I knew that something hit the fan. My Sarge-Etch mix server hosting 4 Etch virtual servers was crashed when the UPS overloaded and comitted emergency shutdown. No notification to the servers before pulling the plug, of course.

First, we booted up - ext3fs and fsck saved the day again.

The software RAID take some time to reassemble - we have to dig up the UIDs of the disks and re-generate some config files, but it worked at the end.

Even the mysql tables were rescued with some forced checks (we had to rebuilt the indices, but who cares?).

But the vservers. Those made me work for 20 hours. It all started by rather inoccent looking error when I typed vserver karvaly start which was caused by some version incompatibilites between the old vserver patch in the kernel and the new vserver-utils in Etch. As I learned later, guys hurried up the new version of vserver-utils because we "need" it in Etch.

Next logical step: Get new kernel.

Stock kernel now includes vserver (old needed kernel-patch-vserver), so it is even better. Download new kernel, install new kernel, reboot.
Of course, stock kernel can not see the RAID (SCSI controller? Ext3fs? ) and the operators can not understand any of those pretty error messages. After a wonderful telephone call, they learn the basics of lilo and boot back to the old kernel.

Next logical step: Compile new kernel.

Download, bunzip2, untar, zcat /proc/config.gz, make-kpkg, and the result is:
fs/built-in.o: In function `version_read_proc':
proc_misc.c:(.text+0x3b3e2): undefined reference to `vx_linux_banner'
make[1]: *** [.tmp_vmlinux1] Error 1
make[1]: Leaving directory `/usr/src/linux-source-2.6.18'
make: *** [debian/stamp-build-kernel] Error 2


A few hours later, I found the trick.

In the old days, vserver was not part of the kernel tree, so you needed a patch. But in 2.6, especially in Etch, it became upstream, so the kernel-patch-vserver was disappeared.
This means that if you download linux-tree-2.6.18, you will get linux-source-2.6.18 and linux-patch-debian-2.6.18. This latter contain all the patches you need to apply to get the stock kernel images, ordered in series (containing a few ten patcheses).
Most of these patches are already applied in linux-source, you can play with /usr/src/kernel-patches/all/2.6.18/apply/debian and /usr/src/kernel-patches/all/2.6.18/unpatch/debian. However, if you take a look deeper:
cd /usr/src/kernel-patches/
find . -name "*vserver*" -ls
cat all/2.6.18/debian/series/* | grep vserver
So, there are different flavours of linux-images you can build, from different patchsets, but only one flavour of linux-source, which needs additional patches to compile.

So, you need to apply this 21 line vserver-version.patch or else compilation will fail at the very end, namely linking, with an error message pointing to the file-system's built-in functions (gruesome!).

Oh yeah, so, to make it work, you need to apply this vserver-version.patch patch too, before starting to compile the kernel (there are some other patches, too).

There are languages like C, C++ or Java, which have different "equal" symbols namely "=" as value assignment (let it be equal!) and "==" as an binary operator, which returns a boolean (are these equal?).

In SQL, the same symbol "=" is used in both cases, the former is in update, the latter is in select.

We (as humanity) spend millions of dollars to build a great SQL (RDBMS) server, which is bundled with other softwares (which increase our productivity) on a DVD (even though 640 MB should be enough for anyone).

And, of course, we are still unable to write a parser that accepts "==" instead of "=" in a where clause.

Nor are we able (that would really increase productivity) to at least return a comprehensible error message instead of ORA-00936: missing expression or ERROR 1064 (42000): You have an error in your SQL syntax;.

Surely empty strings are very practical, even if some people think that they should be omitted from RDBMS-based systems and nulls should be used instead. Surely it is very handy to have char and varchar types. While chars are not compressed (thus some index operations may be faster), varchars are compressed, thus loading them into memory may be faster.

But then, you have this funny thing in this Oracle9i Enterprise Edition Release 9.2.0.7.0 - 64bit I work with sometimes:

ORA> select length('') from dual;
LENGTH('')
----------

1 row selected.

ORA> select length('SE') from dual;
LENGTH('SE')
------------
	     2
1 row selected.

So, if you happen to use "update ... when length('')=0;", you will have a bad day (especially, if you are stuck with a minimalistic fprint-output script, without being able to use control loops). So, there might be a point in getting those certifications, after all - if you are not certified, how would you know all those ill-logical system-specific behavior? (Yes, I know, this behavior is widely documented all over the net.)

The solution for this feature (oh God, I love this expression) is:

ORA> select length(''||'X')-1 from dual;
LENGTH(''||'X')-1
-----------------
		    0
1 row selected.

ORA> select length('SE'||'X')-1 from dual;
LENGTH('SE'||'X')-1
-------------------
			2
1 row selected.

Of course, you have to watch out for the character encoding if you are living outside the world USA:

ORA> select length(''||'X')-length('X') as zero from dual;
	ZERO
----------
	   0
1 row selected.

ORA> select length('SE'||'X')-length('X') as two from dual;
	 TWO
----------
	   2
1 row selected.

And yes, of course, in my MySQL:

mysql> select length('') from dual;
+------------+
| length('') |
+------------+
|          0 |
+------------+
1 row in set (0.00 sec)

Grown up using no clipboard (DOS), getting used to a single-application clipboard (NC), then to a whole-OS clipboard (a Sun X-terminal) and then multiple clipboards in one application (MS Word 97?), I was a little bit confused when met X's full clipboard mechanism on long-term.

To put it simple, there are two areas, a real clipboard (used by graphical applications in a Ctrl-C Ctrl-V style) and a selection (used by command-line applications in Mouse-1 Mouse-3 style, with Mouse-1+Mouse-2 emulating Mouse-3 in most configurations).

Yet, I had a problem with copying command lines from OpenOffice documents to aterm windows. I started using aterm because I could easily set up the font colors by a command-line parameter as opposed to gnome-terminal and it was a very handy way to provide visual feedback on which server/user runs the terminal (green for local me, red for local root, orange for University account, etc.). Later I switched to OpenBox as I have not needed all the eye candy in Gnome but could use that extra 100 MB of memory. Then, I found out that after booting, I can use "Ctrl-C" to copy in OpenOffice and then "Mouse 2" in aterm to paste - but only as long as I don't use "Mouse 1" to copy. This happens because aterm (Enlightment? OpenBox? X?) is smart and gets me access to the GUI clipboard (officially: Primary clipboard) as long as the CLI clipboard selection (officially: Selection) is empty. But when I put something in the Selection, I could not us Ctrl-C Shift-Insert anymore to copy text from OpenOffice to aterm (Shift-Insert is a shortcut to ... Mouse-2?). I had to restart X or use gnome-terminal.

Finally, I found out the trick: Select the text in OpenOffice, but instead of Ctrl-C, just switch to aterm and press Mouse-3 (defaults to the middle button). Voila. No need to press all those buttons and overstress your Ctrl button anyway, right?

You might need to empty your clipboard first, probably with xclipboard.