I've just released qcontrol 0.5.1. Changes since the last release:
- Add build targets to enable static linking (Original patch by Frans Pop).
- Wake-on-Lan and EUP control (by Michael Stapelberg, Debian bug #703888).
- Updated example configurations (based on Debian package).
- Support loading configuration snippets from a directory (Ian Campbell, Debian bug #697574).
- Various other bug fixes.
I also put together a very basic homepage.
Get it from gitorious or http://www.hellion.org.uk/qcontrol/releases/0.5.1/.
The Debian package will be uploaded shortly.
Taking over Qcontrol upstream
When I took over the qcontrol package in Debian, back in October 2012, I was aware that upstream hadn't been active for quite a while (since late 2009). I figured that would be OK since the software was pretty simple and pretty much feature complete.
Nevertheless since I've been maintaining the package I've had a small number of wishlist bugs (many with patches, thanks guys!) which are really upstream issues. While I could carry them as patches in the Debian package they would really be better taken care of by an upstream.
With that in mind I mailed the previous upstream (and original author) Byron Bradley back in February to ask if he would mind if I took over as upstream. Unfortunately I haven't yet heard back, given how long upstream has been inactive for this isn't terribly surprising so having waited several months I have decided to just go ahead and take over upstream development. Thanks Byron for all your work on the project, I hope you don't mind me taking over.
Since it is unlikely that I will be able to access the old website or Subversion repository I have converted the repository to git and uploaded it to gitorious:
I don't expect I'll be doing an awful lot of proactive upstream development but at least now I have a hat I can put on when someone reports an "upstream" issue against qcontrol and somewhere which can accept upstream patches from myself and others.
I'm still deciding what to do about a website etc. I may just enable the gitorious wiki or I may setup an ikiwiki software site type setup, which has the advantage of being a little more flexible and providing a neat way of dealing with bug reports without being too much overhead to setup up.
In the meantime its been almost 5 years since the last release, so...
New Release: qcontrol 0.5.0
This is a rollup of some changes which were made in the old upstream SVN repository but never released and some patches which had been made in the Debian packaging. What's here corresponds to the Debian 0.4.2+svn-r40-3 package.
The 0.4.2 release was untagged in SVN, but corresponds to r14, new stuff since then includes:
- Support for more hardware (TS-119, TS-219, TS-41x, all by Martin Michlmayr)
- Support auto power on feature (by Martin Michlmayr)
- Support for the A125 LCD on TS-419P devices (by Bernhard R. Link)
- Support for a daemon mode, including syslog (Byron Bradley)
- Support for disabling the watchdog on TS-219P II and TS-419P II (Helmut Pozimski)
- Various other fixes (Loïc Minier, Martin, Bernhard, Byron)
As well as the change of maintainer I think the addition of the daemon mode warrants the bump to 0.5.0.
Get the new release from git or http://www.hellion.org.uk/qcontrol/releases/0.5.0/.
Working with virtualisation I find myself occasionally needing to generate a random MAC address for use with a virtual machine. I have a stupid little local script which spits one out, which is all fine and dandy for my purposes. However this week one of my colleagues was writing a wiki page and needed to include instructions on how to set the MAC address on the Arndale board. He wanted to include a shell snippet to generate one and I suggested that there must be websites which will generate you a suitable address. But when I came to look we found that not a single one of the half a dozen site which I looked at handled the locally administered or multicast bits correctly, meaning that they would randomly generate either multicast addresses or addresses using assigned OUI (or both). Using a random OUI may not cause too much trouble in practice but using a multicast address is sure to lead to strange behaviour, and in any case it's kind of lame.
So last night I sat down and wrote a quick hack cgi script to generate random addresses and tonight I deployed it at: http://www.hellion.org.uk/cgi-bin/randmac.pl.
This is where the world and his dog leaves a comment pointing me to the existing service my searches missed...
I've just got back from Linaro Connect Asia 2013 in Hong Kong. This was my first time back in Hong Kong since 1995 when I left after living there for 10 years. It was really interesting to see how much the place had changed, as well as how much it hadn't!
I started the week by giving a talk together with Stefano Stabellini Introducing Xen on ARM to a full house. The slides can be found on the event page or google docs. There is a video on youtube but due to the cameras being setup for a fish-bowl style panel it is unfortunately mostly of my backside. The reception was excellent and the talk seeded a large number of hallway track conversations with all the many folks who are interested in Xen on the ARM platform.
The day after our talk Lars Kurth (Xen.org community manager) and Mark Heath (VP of XenServer Engineering, and my boss) gave a keynote (video) announced Citrix's intention to join the Linaro Enterprise Group (AKA "LEG") on the 1st of April. This is pretty exciting for me since it means I'll get to work much more closely with the Linaro and ARM communities and spend even more of my time hacking on Xen on ARM!
At some point Stefano and myself were also video interviewed by Charbax of http://armdevices.net/ but it doesn't appear to have gone online yet. Lars and Mark also gave interviews which are online already.
One nice thing about Connect is that the afternoons are left free of talks so I managed to find time in the hackrooms to implement SMP host support and 64-bit domain 0 support for Xen on ARM as well as getting multiarch cross building working for upstream Xen with Wookey's kind assistance (no patches for this yet since I still need to clean them up). It was also useful to catch up with the ARM kernel developers and KVM guys to see what they are up to and share ideas etc.
Finally right at the end Stefano and myself showed Xen on ARM at Demo Friday. We had Xen running on the Arndale Board, ARM Chromebook and 64-bit ARMv8 using the emulators (no hardware being available yet).
All in all I had an excellent (if tiring) week, there was loads of interest in Xen on ARM and in Citrix joining LEG and everyone was very welcoming. I'm really looking forward to the next Connect in Dublin.
Outside of Connect I managed to find time to visit Sham Shui Po at look at all the gadgets but managed to resist buying anything (Stefano got a Thinkpad X230, which was enough retail therapy for both of us!). I caught up with my sister (who lives there again), her boyfriend and my parents (who happened to be visiting her that week). I also managed have some drinks with some old school friends, most of whom I've not seen for nearly twenty years, but it was just like old times ;-).
I'm off again to Hong Kong next week with my family to visit my sister (yes, there was some bad planning involved here...). This time I plan to do all the toursity type stuff that I never did while I lived there. I've also got tickets for the Hong Kong Rubgy Sevens and hopefully I'll manage catch up with some more old friends. I probably won't be able to resist buying gadgets this time around though.
My colleague Stefano Stabellini and I are in the process of putting the finishing touches to our presentation which we are giving at Linaro Connect next week in Hong Kong. If you are there come and see Introducing XEN on ARM on Monday at 1100, we'll be running through some of the basics of Xen as well as the specifics of how this translates into use on the ARM virtualised platform and talking about our status and roadmap going forward.
As well as that we will also be demoing Xen on ARM, both on v7 hardware platforms and on v8 software models, at Demo Friday. I've been mostly working on the 64-bit port recently and the initial set of 45(!) patches was committed at the end of last week --- just in time!
The other main Xen on ARM event at this years Connect is a keynote on the Tuesday morning given by Lars Kurth (Xen.org Community Manager) and Mark Heath (XenServer VP of Engineering, and also my boss).
Other than that my schedule seems to be running over with other interesting sounding talks and there are certainly going to be plenty of people to catch up with, but hopefully I'll have enough time to put the finishing touches onto ARMv8 64-bit guest support in the hackrooms like I'm planning...
Looking forward to seeing some of you there!
So as previously mentioned I gave a talk at FOSDEM this year on the future of paravirtualisation under Xen. I thought the talk was reasonably well attended despite being at six o'clock in the evening. In the end I underran my half hour slot but fortunately people had plenty of interesting questions! I've posted my slides on my xenbits space.
I spent a fair chunk of the rest of the time on the Xen.org booth, which is hard work but it is always nice to speak to users and other developers. I also managed to catch up with various friends from other projects and managed to meet a few folks who I've only ever met in email before.
I didn't get to see as many talks as I wanted (for example I missed pretty much all of the ARM track :-() due to clashes with my stints on the booth, my own talk, and talks clashing with each other. Still it was all in all a worthwhile, hopefully I'll be able to make it again next year!
Next up: LinaroConnect in Hong Kong!
Like many people I'll be going to FOSDEM again this year.
I'll be giving a talk on Saturday at 1800 in the Virtualisation devroom on Evolving Xen Paravirtualization. I'll also be manning the Xen.org booth at least some of the time but hopefully I'll get to see some of the other talks too.
Hope to see everyone there!
What is Valgrind?
Valgrind is a framework for building dynamic analysis tools. Several useful tools are included in the Valgrind distribution including tools to check dynamic memory usage (memcheck), a cache profiler (cachegrind), heap profiler (massif) and thread debugger (helgrind) among others. Valgrind also provides a framework which can be used to build other tools.
The Valgrind tool which I find most useful and the one which I have most experience with is memcheck. This tool can detect all manner of memory management problems, including use after free, using uninitialized data, memory leaks, double free. Between them these can result in savings of many hours of staring a core dumps and gdb backtraces.
How Does memcheck Work?
At its core Valgrind is a dynamic binary translation engine, which is used to instrument the code at runtime. In the case of memcheck this is used to track properties of the memory in a process, including aspects such as whether each bit (yes, bit) of memory has been initialized since it was allocated. It also tracks this information as data is loaded into registers, so it can know if a given register is currently tainted with uninitialized data or not. As well as instrumentation through binary translation Valgrind also includes instrumented versions of the C dynamic memory allocation functions which are used to track whether a each bit of memory is currently considered free or allocated, as well as tainting registers when they contain a pointer to memory which has been freed.
A large part of memcheck's functionality is built upon Valgrind's ability to determine when memory has been initialized. However although binary translation can be used to instrument when the application itself has initialized some memory it is not currently possible to instrument the behaviour of system calls, in other words Valgrind cannot automatically determine when memory has been initialised e.g. after a read(2) system call. For this reason Valgrind has baked in knowledge about the behaviour of system calls on various platforms.
For example lets consider the read(2) system call. The prototype of read(2) is:
ssize_t read(int fd, void *buf, size_t count);
This system call reads up to count bytes from the file descriptor fd into the memory pointed to by buf and returns the number of bytes read. Here is the code within Valgrind which handles this system call (found in coregrind/m_syswrap/syswrap-generic.c):
PRE(sys_read)
{
*flags |= SfMayBlock;
PRINT("sys_read ( %ld, %#lx, %llu )", ARG1, ARG2, (ULong)ARG3);
PRE_REG_READ3(ssize_t, "read",
unsigned int, fd, char *, buf, vki_size_t, count);
if (!ML_(fd_allowed)(ARG1, "read", tid, False))
SET_STATUS_Failure( VKI_EBADF );
else
PRE_MEM_WRITE( "read(buf)", ARG2, ARG3 );
}
POST(sys_read)
{
vg_assert(SUCCESS);
POST_MEM_WRITE( ARG2, RES );
}
These two functions are called before and after the system call respectively. The main piece of functionality is the calls to PRE_MEM_WRITE and POST_MEM_WRITE. In this case PRE_MEM_WRITE is used to check that the memory starting at ARG2 (recall that this is the supplied buffer) and extending for ARG3 bytes (the system calls count argument) consists of allocated memory. After
the system call in complete the post function calls POST_MEM_WRITE to indicate to Valgrind that RES bytes of the buffer have know been initialized.
The above is a relatively simple case, but the above two hooks must be supplied for each system call on each platform which Valgrind wishes to support. One of the most problematic system calls is the ioctl(2) system call. The ioctl call takes a file descriptor, a request number and a pointer to a per-ioctl argument structure and implements per-device control requests. There
are literally dozens of devices many with their own specific ioctls. For this reason a sizable portion of Valgrind code is dedicate to decoding the behaviour of the myriad of potential ioctls
and providing pre- and post-call instrumentation for them.
Interacting with the Hypervisor from Userspace
Under Xen the toolstack is a normal userspace process running in the control domain (typically domain 0). As part of its operation the toolstack needs to communicate with the hypervisor and to do this it uses hypercalls. However a userspace process cannot simply make hypercalls on its own, instead it must request that the OS kernel make the hypercall for it. This prevents just any userspace process making a hypercall, since the kernel will first check the privilege of the process making the request.
Under Linux a userspace process makes hypercalls by using the ioctl(2) system call on a special "Privileged Command" (privcmd) device. In this case the toolstack uses the IOCTL_PRIVCMD_HYPERCALL ioctl request which takes a hypercall number and the hypercall arguments as its argument. Other OSes which can be used as a control domain posses similar interfaces.
Making memcheck Work With Xen Toolstacks?
As might be expected the majority of the work to allow Valgrind to work on Xen toolstack processes was teaching it about the IOCTL_PRIVCMD_HYPERCALL ioctl. Fortunately it was not necessary to teach Valgrind about every hypercall but only about those which are used by toolstacks. This is a smallish subset of all hypercalls including the domctl interfaces and a handful of others.
The initial batch of patches cover all of the system calls needed to start new domains, shut them down and migrate them using the XL toolstack. New hypercalls will be added as they are discovered to be missing.
Getting and Using This Functionality
The initial set of patches were accepted into Valgrind's Subversion repository trunk revision 13081 in October 2012 and are expected to be part of the 3.9.0 release.
Until 3.9.0 is released it will be necessary to get this functionality from SVN. Fortunately this is relatively simple and follows the usual autofoo dance. Although the usual caveats regarding running unreleased software apply.
$ svn co svn://svn.valgrind.org/valgrind/trunk valgrind $ cd valgrind $ ./autogen.sh $ ./configure $ make $ sudo make install
Once Valgrind is built and installed simply call it passing the command to be debugged as an option:
# valgrind xl list ==16186== Memcheck, a memory error detector ==16186== Copyright (C) 2002-2012, and GNU GPL'd, by Julian Seward et al. ==16186== Using Valgrind-3.9.0.SVN and LibVEX; rerun with -h for copyright info ==16186== Command: xl list ==16186== Name ID Mem VCPUs State Time(s) Domain-0 0 512 4 r----- 332.5 ==16186== ==16186== HEAP SUMMARY: ==16186== in use at exit: 0 bytes in 0 blocks ==16186== total heap usage: 16 allocs, 16 frees, 93,529 bytes allocated ==16186== ==16186== All heap blocks were freed -- no leaks are possible ==16186== ==16186== For counts of detected and suppressed errors, rerun with: -v ==16186== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 39 from 6)
Here we see that xl list has no leaks! However if I comment out the call to libxl_dominfo_list_free in tools/libxl/xl_cmdimpl.c:main_list() then instead Valgrind reports:
[...] ==16203== HEAP SUMMARY: ==16203== in use at exit: 90,112 bytes in 1 blocks ==16203== total heap usage: 16 allocs, 15 frees, 93,529 bytes allocated ==16203== ==16203== LEAK SUMMARY: ==16203== definitely lost: 0 bytes in 0 blocks ==16203== indirectly lost: 0 bytes in 0 blocks ==16203== possibly lost: 90,112 bytes in 1 blocks ==16203== still reachable: 0 bytes in 0 blocks ==16203== suppressed: 0 bytes in 0 blocks ==16203== Rerun with --leak-check=full to see details of leaked memory ==16203== ==16203== For counts of detected and suppressed errors, rerun with: -v ==16203== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 39 from 6) [...]
So it has caught the memory leak! Rerunning with the suggests --leak-check=full goes even further and tells me where the leaked memory was allocated:
[...] ==16204== 90,112 bytes in 1 blocks are possibly lost in loss record 1 of 1 ==16204== at 0x4024480: calloc (vg_replace_malloc.c:593) ==16204== by 0x4050CE4: libxl_list_domain (libxl.c:548) ==16204== by 0x805EEC7: main_list (xl_cmdimpl.c:3931) ==16204== by 0x804D6DD: main (xl.c:285) [...]
To ease debugging of domain creation I find it useful to use the -F option to xl create to stop xl from forking and daemonising. Most xl subcommands do not daemonize and so need no special treatment.
For more information on running Valgrind see the Valgrind Documentation.
Conclusion
Valgrind is a powerful tool in the arsenal of the C programmer, and can be used to catch a large number of hard to debug and common issues before they happen can save many hours of tedious debugging. The ability to use tools such as this on Xen toolstack processes is of enormous benefit and has already found several bugs in the XL toolstack.
The QNAP TS-XXX NAS boxes are nice little systems but one thing I really miss is a console. Martin Michlmayr has some instructions (TS-119, TS-219, TS-41x) on his excellent Debian On QNAP pages on how to build a suitable adaptor but sadly even though I've managed to get all the parts I've been too lazy to actually solder the thing together (it doesn't help that I nearly always burn myself when I use a soldering iron!).
This became more pressing when someone reported that Debian bug #693263 in qcontrol was not fixed in Wheezy since the issue appeared to be in the initramfs hook. Worse it seemed like something in my proposed fix was causing the system to not boot at all!
Having finally found the hardware needed to reproduce the issue my
first thought was to try netconsole. To do this I edited
/etc/initramfs-tools/modules to add:
mv643xx_eth
netconsole netconsole=@<IP>/eth0,@<DST-IP>/<DST-MAC>
The option syntax is described in
netconsole.txt
but briefly: <IP> is the address of the TS-419P II I'm debugging on,
and <DST-IP> and <DST-MAC> are the IP and MAC address of another
machine on the network.
Having done that, running update-initramfs -u and rebooting I can
use netcat -u -l -p 6666 on the other machine to see all the kernel
messages.
So far so good but this doesn't get me any debugging from the
userspace portions of the initramfs. To get those we have to
get a bit hacky by editing /usr/share/initramfs-tools/init, first to
change:
# Parse command line options
-for x in $(cat /proc/cmdline); do
+for x in $(cat /proc/cmdline) debug; do
case $x in
and secondly:
debug)
debug=y
quiet=n
- exec >/run/initramfs/initramfs.debug 2>&1
+ exec >/dev/kmsg 2>&1
set -x
;;
The first of these simulates adding debug to the kernel command line
(which can't otherwise easily be edited on these systems) and the
second redirects the initramfs process's output to the kernel log. The
overall effect is that the output of the initramfs processes appears
over netcat.
Since I've been working on qcontrol I've had need of a way to build packages in an ARM environment. Since all my native ARM hardware is "production" (at least so far as something can be production in a home environment) I prefer to avoid installing development tools on them. When I started looking into this all my ARM systems ran from relatively small flash devices which also ruled out the possibility of a development chroot. After search the web a bit I discovered that it is possible to use qemu-user-static together with [[!binfmt-support ]] to create cross architecture chroots which can be entered mostly transparently. It is also possible to combine this with schroot and sbuild which makes the day to day use pretty much the same as a native or biarch chroot.
Since I've now had to rediscover how to do this twice (once for Sid and then again when I realized I also needed a Wheezy environment) I thought I should write it down for the benefit of my future self.
First things first lets install the various packages which we need. As
well as debootstrap to create the chroot we need
qemu-user-static and binfmt-support to allow us to run ARM
binaries:
$ sudo apt-get install debootstrap qemu-user-static binfmt-support
I'm using schroot's lvm-snapshot mode so we need an LVM volume,
formatted with a filesystem and mounted in a temporary location:
$ sudo lvcreate -L8G -n sbuild-wheezy-armel /dev/VG
$ sudo mkfs.ext3 /dev/VG/sbuild-wheezy-armel
$ sudo mount /dev/VG/sbuild-wheezy-armel /mnt/
Next we need to create the chroot. For this we use debootstrap in
foreign mode which does a 2 stage setup, firstly by using the
--foreign option:
$ sudo debootstrap --variant=buildd --include=fakeroot,build-essential \
--arch=armel --foreign \
wheezy /mnt http://<DEBIAN-MIRROR>/debian/
Now we (temporarily) copy the qemu-arm-static binary into the chroot
(at the place where binfmt-support expects it), change into the
chroot and run debootstrap's second stage to finish the basic chroot
setup:
$ sudo cp /usr/bin/qemu-arm-static /mnt/usr/bin/
$ sudo chroot /mnt
(chroot)# ./debootstrap/debootstrap --second-stage
(chroot)# exit
Now that we have a basic chroot in place it needs a little bit of
tailoring. I've found the easiest way to do this is to use
sbuild-createchroot and then tweak the result:
$ sudo sbuild-createchroot --arch=armel --foreign --setup-only \
wheezy /mnt http://<DEBIAN-MIRROR>/debian/
If we were building a native chroot then we could have skipped the
debootstrap stuff and let sbuild-createchroot handle
things. Unfortunately this doesn't seem to play nicely with
--foreign.
Now we can remove the qemu-arm-static binary from the
chroot. schroot will automatically bind mount the version from the
host into the chroot which ensures it remains up to date. We can also
unmount the chroot since we will let schroot manage that from here
on:
$ sudo rm /mnt/usr/bin/qemu-arm-static
$ sudo umount /mnt
Next we need to tweak the resulting schroot/sbuild configuration
since sbuild-createchroot assumes the use of schroot's directory
mode. First remove the symlink to /mnt which it created:
$ sudo rm /etc/sbuild/chroot/wheezy-armel-sbuild
Lastly we need to edit /etc/schroot/chroot.d/wheezy-armel-sbuild-<SUFFIX>
(where is randomly generated) and reconfigure it to use
lvm-snapshot mode, by apply the following changes:
[wheezy-armel-sbuild]
-type=directory
-directory=/mnt
+type=lvm-snapshot
+device=/dev/VG/sbuild-wheezy-armel
+lvm-snapshot-options=--size 2G
description=Debian wheezy/armel autobuilder
groups=root,sbuild
root-groups=root,sbuild
profile=sbuild
That's it, we now have a chroot which we can trivially invoke via
sbuild to build armel packages:
$ sbuild --arch=armel --dist=wheezy <... other options...>
We can also use sbuild-update to keep the chroot up to date:
$ sudo sbuild-update -udcar wheezy-armel
You can also use schroot to login to the chroot directly, although
for that case you'll likely want to create a different chroot using
schroot's directory or block-device mode instead lvm-snapshot
and drop some of the sbuild specific setup. I expect I'll blog about
this configuration in some more detail at some point in the future
since I'll want to document it to give a more convenient build
environment for Xen on
ARM
than the build
farm
which we have at work.
This blog is powered by ikiwiki.