The last couple of weeks development on my TTY layer has slown down a little bit. Some say the last 10% of your software always takes 90% of the time. Today I'm working on timer support inside the TTY layer (the VTIME and VMIN fields inside termios).

Most of the time I just use the FreeBSD source code as documentation on how TTY's should be implemented inside the kernel. In a lot of cases this has shown to be useful, but in this case I just did it by reading POSIX. The FreeBSD code works pretty well, but the source code is so complex, without a real reason. The old FreeBSD TTY layer has a single read() routine that is called in both canonical and non-canonical mode, with or without timers (see ttread()). This routine is about 300 lines big, with a lot of goto statements in it. Just awful.

The TTY layer should work pretty well now. I'm now already further than I could imagine. One of the major features that is still missing, is proper support for carrier/connection state tracking. My TTY layer just blindly assumes there is a connection, which means I can only connect to getty's that are run on callout devices. I hope to get this working in a couple of weeks, but I do have to spend more time on writing reports for school. We'll see. :-)

Next month I'm going to BSDCan as well, so I can chat with other FreeBSD folks about my work. Shibby!

One of the nice things about my assignment (rewriting the TTY layer), is that I'm now often in the position to fix bugs that have been a pain for many years. One of them is the well know Jail leak.

The FreeBSD kernel stores process credentials in a structure called struct ucred. This structure is passed through the kernel on many different locations to implement the privilege model that's available in the kernel. One feature that's unique among the BSD's, is support for Jails, which is actually a chroot() on steroids. The credentials structure may have a reference to a struct prison to limit access to certain features within the kernel.

Normally, when a credentials structure isn't properly freed within the kernel, you'll end up having a Zombie Jail. There have been some bugs in the past that caused them (i.e. leaking in the socket code), but one of them hasn't been fixed, because it was just impossible to do so. When a process creates a pseudo-TTY within a jail, a device node is created, with refers to the credentials structure. Because the old TTY code wasn't capable of destroying devices properly, the prison was referenced indefinitely, which means that jails could not be removed from the system anymore.

Today I've been working on a new driver called ptycompat(4), which adds support for the legacy BSD-style PTY naming scheme. Because my new TTY layer properly performs reference counting on TTY's, they are now properly garbage collected, which means prisons will get destroyed as well. Hooray!

Some time ago, Florent Thoumie added me to Planet xBSD, so I could blog about my TTY hacking, but so far I didn't post anything yet. I've been too busy writing code, I guess... :-)

Anyway, I started working at Snow 7 weeks ago, working on my internship, which is to improve FreeBSD's TTY layer. I did a lot of research before I even started working on my assignment; unfortunately the current TTY layer has so many problems of its own, I decided to entirely redesign it. Today I'll write something about the great improvements I made with respect to buffering inside the TTY layer.

Old TTY buffering has overhead

One of the things I discovered while reading the old TTY source code, was how inefficient data is currently stored within the TTY layer. The old TTY layer uses three buffers per TTY, called the outq, canq and rawq. The meaning of the outq is obvious: it is used to store data that's on its way out (from the process to the serial port, SSH daemon or terminal emulator). The canq and rawq are both used for character input. The canq contains lines of text that are already canonicalized (ready to be sent to the application), while the rawq contains a partial line. The partial line is copied to the canq when the user presses the Return key.

Because data in those queues always has a different alignment, copying data from queue to queue has to be performed per byte, which is awfully slow. This is why I came up with a two-queue model, the inq and outq. The outq is a light-weight queue that cannot store any parity information (there can't be any parity errors on our way out), while the inq is a heavy-weight queue. The inq has the ability to canonicalize lines as an O(1) operation (just update an offset).

Replacing the queues was quite experimental, because I wasn't exactly sure if it would fit. The inq doesn't allow data to be sparsely stored, which means new data cannot be stored before the partial line. It turned out to work great. So far there doesn't seem to be any requirement for that.

Buffered copying to userspace

I also noticed the current TTY code buffers data before it copies it back to userspace. We cannot hold any locks, because there is no guarantee pages are actually available at that moment. I came up with the idea to temporarily remove blocks from the buffer queues when we have to copy data back to userspace. We can't perform this trick in all cases, because when we only want to partially read data in the current block, other threads might want to read the other data while we're sleeping.

I added some counters to the block queue code, so I can gather some statistics on how many times we can actually perform this trick. It seems to work quite good in practice:

kern.tty_inq_nslow: 35
kern.tty_inq_nfast: 136
kern.tty_outq_nslow: 14
kern.tty_outq_nfast: 1824

The reason why it seems to work better for the output queue, is because the input queue will store the partial lines inside the same block as the already canonicalized input, so it cannot perform the trick yet. It's still a pretty good result, though. OpenSSH and xterm probably use such big buffers that reads of partial blocks will almost never happen.

So how far am I?

I've only been coding 5 weeks, but so far I:

• wrote a new TTY layer, which includes the buffer systems I described above. It's still quite incomplete, but it should already be usable.
• wrote a new pseudo-TTY driver, which can be allocated through the new system call posix_openpt(2). This is quite unique, because most other operating systems implement this feature through a C library routine.
• ported syscons to the new TTY layer. It already seems to work fine. It still uses Giant, but that can be fixed later on.
• somewhat ported the UART driver. It only seems to work when I boot into single user mode, so I'm close.
• nuked the old TTY layer. My system still boots.

I still have about 10 weeks to hack on my code, so I should get pretty far, I guess.

If you're a daredevil, you can already try my code, which is stored in FreeBSD's Perforce server under the directory //depot/user/ed/mpsafetty. If you're an i386/amd64 user and only use syscons and pseudo-TTY's, it should already be somewhat usable. Thanks! :-)

This week I started on my internship at Snow, which was great. Snow has more than a hundred people employed, so I had a real hard time remembering all the names. ;-)

I already started working on my assignment (revamping the FreeBSD TTY layer). The first weeks I'm not going to write a lot of cool stuff, because I have to document everything properly on beforehand. By the end of this week I'm probably going to start on the design documents that I can release to the public. This means other FreeBSD developers can (dis)approve my work before I'm going to hack on it.

I started using Perforce and already committed a random TTY patch I had laying around. It works great, but somehow it has to refresh all files that have been affected by a commit, which meant it had to perform a second complete checkout of the FreeBSD source tree after I made a new branch. Another thing that bit me, was that on my desktop at home, my system crashed while I was performing a p4 sync because I was doing something silly. After I rebooted and ran another p4 sync, it didn't sync the files again which got lost by the system crash, which meant I had to force a complete synchronisation again, which took another hour. Apart from these small issues, Perforce is a lot better to use than CVS. ;-)

Yesterday was the last day we had to go to school to give a demo on a project we have been working on for the last half year. We had to develop a hovercraft that could be remotely controlled, based on an embedded Linux device - a Texas Instruments OMAP5912 OSK.

It all went pretty good, though we did suffer from the typical Demo Effect. When everyone was ready to see our hovercraft in a head to head contest against the other team, our skirt engine seemed to be a little burnt. This didn't turn out to be a big problem, because we had a spare one laying around in our closet.

Anyway, we had a lot of fun working on the project. It was my last project at Fontys Hogescholen in Eindhoven, which now means I only have to go on my second internship to graduate. On Monday I'm starting at the office, which means I'm finished somewhere in July.

As usual I'm going to FOSDEM again next month. I'm not sure, but this is probably the fifth time for me now. The schedule looks interesting already. I saw there will be a FreeBSD/PostgreSQL room this year. I'll probably visit that room a lot.

I also saw that the folks from BSDA will be there. When I went to LinuxTag last year, I took a beta exam, which means I'll get a 50% discount when I take the final exam at FOSDEM this year. Sounds like a good deal. I did a pretty good job on the beta exam, except for the questions about pkgsrc (and there were a lot of them).

This year I'm going to Brussels by car. As usual I'm teaming up with the people from Dispuut Interlink and MCGV Stack, but this year some people from TCCN are going with me as well!

As promised in my last post, I was going to tell something about my assignment I'll be working on as of next month. I'm going to work at a company called Snow B.V., located in Waardenburg near Zaltbommel. I'm going to work on FreeBSD's TTY in-kernel implementation. I was thinking about improving the following things:

• Improve locking in the TTY layer. Remove the usage of the Giant and give each TTY device its own lock. This is going to improve performance a lot, because Giant is currently being picked up in routines like exit1(), which is quite silly.
• Improve the reference counting on TTY's in the kernel, which should improve garbage collecting on terminal devices, which causes PTY's to be really deallocated when the master device is closed.
• When time permits, improve the buffering scheme used in the TTY code (the so-called clists). Currently all buffers are allocated on beforehand, which means that a low-traffic PTY already allocates about 10 KB of memory, while there is a great chance it will never reach its watermark. Because the PTY driver never runs an interrupt context, we could allocate the buffers when needed.
• Maybe port the remaining applications from sgtty to termios, so we could remove the old interface one day.

So far the people at Snow and FreeBSD have been very helpful. I've got a Perforce account at FreeBSD, which means you folks can track my work during my internship.

So as I said before, I haven't written a weblog for a year or so, so there are a lot of things to catch up!

Internship at TUNIX

Almost exactly one year ago, I started at an IT firm called TUNIX in Nijmegen. I had a great time over there. I could just go to their office in like 10-15 minutes bus and my assignments were just awesome. I mainly did research on network redundancy protocols to improve the availability for their solutions.

What I really liked about my internship as well, was that they gave me a lot of time to go to conferences and such. I went to FOSDEM in Brussels, Sun's Virtualisation in One Day at their main office in the Netherlands, LinuxTag in Berlin and the NLUUG spring conference.

Australia!

After I finished my internship, I went to Australia with my girlfriend. We really had a fun time, because I finally saw my mom's part of the family again, which I didn't see for like 4-5 years. Most of the time we spent in Victoria with my family, but we did go on a small trip to Sydney with my sister. We had such a great time.

Les Trophées du Libre

When I went to LinuxTag earlier last year, there was this booth from the people of Les Trophées du Libre, which is an international event to reward new pieces of Open Source Software. By the beginning of November, I got this message from the jury that my application Herrie was among the finalists in the category Multimedia & Games! Can you imagine a command line music player among the finalists in the category Multimedia & Games?

Anyway, dad and I went on a nice trip to France by car. We had such a lovely time over there and we met some really nice other contestants as well. Herrie won the 3rd prize, which was really sweet!

Well, that's about it... I'm now quite busy at school to finish off the last projects, so I can start working on my next assignment, which I will unveil when I've organised everything.

About a year ago I removed the weblog from my website, because I couldn't update it enough to be relevant. I'm now in my last year at school, which means I'm going on my second internship. Next month, to be precise. That's why I've revamped by weblog again, so I can write about the cool stuff I'm going to do.