Improve the Debian boot process - blog

Etch boots as fast as Woody...

2007-04-11T14:14:00.000-07:00

With the Post-etch frenzy, I installed Debian 4.0 stable as soon as I could. I used the same system used a few months ago for the google Summer of Code. Under the same conditions, the system required 32 seconds until KDE started. Compared with the previous stable releases:

Woody - 32 seconds
Sarge - 44 seconds
Etch - 32 seconds

Nevertheless, Etch has significantly more process started at boot time and the CPU usage is more intensive --with still some room for improvement --. Follow this link for the bootcharts and more detailed information.

As I starting point, using dash instead of bash, the boot time is reduced to 30 seconds and it could be further improved but there is some "hald" process (maybe part of dbus) that doesn't seem to want to run in parallel. See bootcharts. Any ideas?

A weird effect of discover (SoC2006)

2006-09-08T13:33:00.000-07:00

It is funny to see that when I remove discover (/etc/rcS.d/S36discover) from the boot, a disk utilisation and cpu usage peak appear at the end of loading KDE. Would it be that discover is loaded by KDE if it wasn't loaded? Or a discover ghost :p. Well, the bootcharts for the two different cases are available here.

Prelinking during the boot (SoC2006)

2006-09-08T11:28:00.000-07:00

I've tried prelinking with the aim of making the boot faster. I've used the package of prelink from debian in unstable and we get no significant time improvement. Prelinking was added after readahead in a custom script run during the boot in /etc/rcS.d/S70prelink. What the script does is to prelink: kdeinit, Xorg, kwin, kdesktop, kicker, artsd, kaccess, ktip, klipper and korgac. The bootcharts are availabe here.

8 second reduction (SoC2006)

2006-09-08T06:35:00.000-07:00

By removing useless init-scripts from the symbolic link pool (/etc/rc?.d) we got a further second improvement in the boot time. The two init-scripts removed were hotkey-setup and gdm. The former can be removed as I'm not using a laptop and the latter because I'm using KDE. The boot time was reduced from 46 to 45 seconds. See the new bootchart and compare with the reference bootchart.

Trying to integrate Ubuntu's readahead (SoC2006).

2006-09-08T00:11:00.000-07:00

I've been playing a bit with Ubuntu's readahead to get a better boot time on debian. At least they seem to get it. I've used strace to get the list of files for different init-scripts and generate the required /etc/readahead/boot file - containing the files to load to memory. The advantage of readahead over preload so far is that it does not affect the boot time when it is not active. I consider readahead inactive when the boot file is empty and preload when the preload script is removed from the symbolic link pools (/etc/rcX.d).

By using readahead shortly after the system starts to boot (near udev), the boot time is even increased by 2 seconds (e.g. S03readahead in /etc/rcS.d). On the other side, by using some of the files used by KDE during startup there is almost a one second improvement as may be seen in the bootchart. We use readahead in /etc/rcS.d position S42. The list of files and the bootcharts may be found here.

Besides, I've corrected the deliverable 5 including the fact that lintian incorporates now a check for lsb compliance based on our patch! The deliverable is available here

7 seconds less on the boot time (SoC2006)

2006-09-05T16:52:00.000-07:00

With the alioth sever down, it seems our project webpage is down as well. Anyway, we are currently on a 7 second improvement of the boot time by using reordering, dash as /bin/sh and hwclock in the background. The bootchart is available here while that of the original system here. No time was won using preload 0.4 with varied configuration combinations. Curious to see that the time is 2 seconds more even without preload in the symbolic link farm (/etc/rcX.d) when preload is installed.

Deliverables 3 and 5 (SoC2006)

2006-08-31T10:17:00.000-07:00

The new deliverables 3 and 5 were published. In the third deliverable we mention the effect of most hotspots, the procedure used for testing them and their bootcharts. Besides, it contains a group of combinations done with the hotspots and their effect. Deliverable 5 simply consists of the LSB guide for maintainers and the lintian patch to check LSB compliance. Both issues were published some weeks ago and this deliverable was just published to comply with the original deliverable list. Still two deliverables are missing: Deliverable 4: a proposal to change the boot system in debian according to the results from Deliverable 3 and further research, and Deliverable 6: the final report of the project. They are available in the deliverables webpage

Insserv reordering and repeated init-scripts (SoC2006)

2006-08-17T01:08:00.000-07:00

Some time ago I posted having obtained a 2 second time decrease in the boot time using insserv reordering. In that occasion, I had already deleted some repeated init scripts from the insserv modified /etc/rc2.d directory. Recently, I had problems using insserv as I thought the major change was just to change the order of some of the init-scripts (stop-bootlogd, sysklogd, klogd). Now, from a system starting in 53 seconds I got a 2 second improvement by using the same script order used some time ago. Besides, I thought I shouldn't bother to delete the repeated init-scripts as /sbin/init is supposed to ignore them. Nevertheless, by removing the remaining repeated scripts I got a further 2 second time improvement, i.e., a 4 seconds improvement! The bootcharts and init order are available at the project webpage.

System reinstalled and testing preload again (SoC 2006)

2006-08-15T05:42:00.000-07:00

Currently recovering from a disk failure with lots of data lost (i'll backup more often now), I'm trying to tackle with the inconsistencies sketched in the previous blog: preload and parallel booting hotspots don't perform as before. I guess the discrepancies came from playing too much with the boot process.
The new system has originally a boot time of 52 seconds and the installation of preload 0.4 on a freshly installed system gave 1 second longer boot time during the first reboot and came back to the original time after 4 reboots and goes back to 53 seconds afterwards.
The bootchart is available on the bootcharts section of the project webpage for two preload configurations.

Besides, unverbose booting was tested. Well, simply verbose was changed to "VERBOSE=no" and "VERBOSE=quiet" just making the system slower! What could this be? Manual tweaking may be probably needed. The bootcharts for the different cases are available on the bootcharts section of the project webpage

Finally, a laptop testing system was prepared for comparison. It has also a fresh debian sid installation on an external USB hard drive. The results using the laptop test bed were similar for unverbose booting although the boot time seem to be unconsistent! Around 2 second changes without networking and higher with networking. I _guess_ a possible reason is by using an usb hard drive.

Comparing preload 0.4 and 0.2 (SoC 2006)

2006-08-09T01:56:00.000-07:00

I've tried preload 0.2 and 0.4 to compare its effects. Unfortunately my results don't hold with the results obtained some blogs ago: preload 0.2 reduced the boot time by 2 seconds. In this occasion, time was even lost (making me remember my first tests). This may be due to some upgrades had been made to sid since.
First I tried preload 0.2 with default with 1 second lost and a modified configuration with no boot time change.
Then I tried preload 0.4 with default and modified configuration both with no time change.
The bootcharts and log file are availabe at the project webpage.
Yesterday I presented a table with hotspots combinates and unlike it was expected, preload may not be helping anymore (preload 0.4 was used).

Table preview of hotspots (SoC 2006)

2006-08-07T16:02:00.000-07:00

Some hotpots combinations were tested. Unexpectedly, the use of several hotspots together didn't cause an arithmetic sum of boot time improvements. For example, let's consider the following hotspots and their corresponding boot times:

hotspot	time
DASH AS SH	4
PRELOADING	2
NETWORKING	0
HWCLOCK	2
REORDERING	2
PARALLEL	4

On the other side, while trying them together we get:

hotspot	Test#1	Test#2	Test#3	Test#4	Test#5	Test#6
INSSERV	X	X	X	X	X
DASH	X	X	X	X	X	X
NETWORKING	X		X	X
HWCLOCK	X		X	X
PRELOAD		X	X	X	X	X
PARALLEL		X	X		X	X
time improve	4	6	6	6	6	4

Lintian patch for LSB-compliance (SoC 2006)

2006-07-30T04:22:00.000-07:00

A patch for lintian to check runtime dependency in the init scripts is now availabe at the project webpage. It proposes modifications to two files in the lintian/checks directory and was already tested. It reports incomplete and missing dependency information as well as individual warnings per headers missing. It was reported as a patch for the #377740 bug previously reported.

Now preloading worked (SoC 2006)

2006-07-22T11:40:00.000-07:00

One again I've tried the preload program from Behdad Esfahbod (a result from SoC2005) but this time positive results. The cause of the difference might have been an incorrect deinstallation of parallel booting (insserv/startpar). The boot time showed a 2 second time improvement. The preload.conf parameters mapprefix and exeprefix were set to empty such that all files would be accepted. The results were compared with mapprefix being set to the directories used by some of the init scripts (strace -f initscript), adjusting the quantum time of preload (cycle parameter) and changing the position of the initscript to start just after hwclockfirst.sh. The results were the same: a 2 second time improvent. See the bootcharts here.

Now Parallel execution works better (SoC 2006)

2006-07-20T16:36:00.000-07:00

A bug was found when doing parallel execution with startpar. The scripts seem to have been executing twice and it is being fixed by Petter Reinholdtsen. To compare the effect of the bug, the parallel execution boot time using startpar was compared with the one using "shell". The main difference between using CONCURRENCY=startpar and CONCURRENCY=shell in /etc/default/rcS is that the latter doesn't care about the order in which messages are printed to the screen.

The issue made me try things again and this time I didn't accept the default script reorder from insserv. Insserv seems to have the following reordering issues:

a)repeates scripts in /etc/rc2.d that were already started in /etc/rcS.d
b)doesn't reorder /etc/rcS.d
c)puts stop-bootlogd near the beginning in /etc/rc2.d and bootlogd while sysklogd and klogd are at the middle.
d)includes scripts present in /etc/init.d that were not previously executed at boot time.

Issues (a) and (c) were corrected and the boot time was 2 seconds faster than without insserv(See bootcharts).
While testing parallel execution, using CONCURRENCY=startpar, there was no improvement (the bug hasn't been fixed) and while testing with CONCURRENCY=shell there was a 4 second time improvement from the original system without insserv. (See bootcharts)

Meeting with my mentor in Dublin (SoC 2006)

2006-07-19T17:14:00.000-07:00

In one of this extremely weird sunny and warm days in Dublin, I've met my google SoC mentor Petter Reinholdtsen. We've discussed several issues of the project and here are the main points:

1. Some small issues in the LSB compliance list needed to be corrected. Some packages didn't showed their package. It was solved and now all the init-scripts have the package they belong to. See LSB-compliance list.

2. It was important to compare the results from two different ways to use dash instead of bash in the init-scripts. See discussion in debian-devel. The two approaches are:

a) make /bin/sh point to /bin/dash - using the procedure described in /usr/share/doc/bash/README.Debian.gz, we obtain a time reduction of 4 seconds.
b) change #!/bin/sh for #!/bin/dash in the init-scripts - using the substitution function of sed, the initscripts were changed to use dash. A reduction of 2 seconds was obtained.

The bootcharts are available here. The difference should be because some programs are still using bash as just the first line of the initscripts was changed.

Automatic update for the LSB compliance list (SoC 2006)

2006-07-15T12:47:00.000-07:00

We have published a list for LSB-compliance in here. The script to create it was now modified to be easily updated automatically considering changes in the initscript files and new reported bugs. It will also consider special cases like with the scripts bootlogd and hostname.sh that wouldn't normally be considered as LSB compliant. The scripts are available under the GNU license at the project repository.

testing ubuntu's readahead (SoC2006)

2006-07-11T19:02:00.000-07:00

The base distribution I'm using for Sid changed slightly by installing ssh. There was no boot time difference. The new base bootchart (same time 49 sec) is here.

Based on a discussion in initscripts-ng-devel between Petter Reinholdtsen and Erich Schubert, I decided to try ubuntu's readahead to see if we will get a better or worst boot time. After installing ubuntu's readahead 1.0.1-2 but noticed not time difference. No difference was obtained by changing the boot order (S39 to S19 in rcS.d) nor by adding files read at /etc/readahead/readahead. The bootchart is here

Also tried the newer readahead from ubuntu readahead 0.20050517.0220-0ubuntu3 but now with a loss of 1 second. This includes two scripts S01readahead and S39readahead-desktop in rcS.d and S99stop-readahead in rc2.d. The bootchart is here

Next I'll try to analyze the SUSE static preload.

Besides I filed an bug (377941) to initramfs-tool I had with installing Etch on my laptop with a usb harddisk. Tried adding sleep 5 to /usr/share/initramfs-tools/scripts/init-premount/udev, then dpkg-reconfigure linux-image-.... but still fails :(

More bug reports and benchmark (SoC2006)

2006-07-11T01:52:00.000-07:00

Yesterday I filed a bug report for lintian as it doesn't check for LSB compliance. Many scripts would be fixed at the moment we get an LSB check in lintian and normally we will be providing them with a patch for it with our Deliverable 5 in a few weeks. Bug report #377740
I'm installing sid in my laptop --originally-- to use as benchmarking for the boot process. The installation procedure was the same as the one explained in the project webpage for sid except that it also includes the Laptop package subset. As I'm using a usb harddisk, it seems there is a problem at the boottime as /dev/sda1 takes to long to be detected and when root wants to mount it hasn't been detected yet. I currently skip this problem this by doing CTRL-S at the beginning to give it more time but this won't make a good boot process benchmark.
New bug reports were filed:
alsa-base #377716
alsa-utils #377717

LSB- howto, status list and check script (SoC 2006)

2006-07-08T11:51:00.000-07:00

A guide for debian package maintainers to make the scripts LSB compliant is available in here. It is not a substitute of the LSB 3.1 but intends to be used as a quick reference to create LSB-compliant init-scripts.

The script to check the LSB compliance was changed to reflect the init script status according to:
+No LSB headers - If "### BEGIN INIT INFO" is not found.
+LSB compliance - If the headers Provides, Required-Start, Required-Stop, Default-Start and Default-Stop exist and have arguments. Besides, if Default-Start is equal to 0 or to 6, LSB compliance is satisfied only with Provides, Required-Start and Default-Start.
+Missing LSB headers in the other cases.

Finally, a list with the current LSB-compliance status of Debian init-scripts is available here. It was automatically generated using a modified version of the script mentioned above and includes a link to the bug report filed to correct the scripts. The aim of this webpage is to give an overview of the work that needs to be done in Debian for LSB compliance.

Bug reports for packages with missing LSB headers (SoC2006)

2006-07-05T17:10:00.000-07:00

Yesterday I started to prepare bug reports about missing LSB headers. Since then I reported:
-acpid script #376778
-atd script #376780
-bittorrent script #376944
-exim4 script #376953
-hotkey-setup script #376955
-inetd script #376956
-kdm script #376958
-lpr script #376960
-makedev script #376992
-nfs-common script #376976.
A summary of the bug reports with missing lsb headers can be found in here
Besides, we've raised the issue of some scripts missing dependencies in the bootclean init script and the missing Should-Start dependency of bootmisc.sh in the sysvinit-devel mailing list.

Deliverable on line and LSB list

2006-07-04T11:55:00.000-07:00

The second deliverable of the "Improve debian boot" project is available to check since yesterday night in the webpage. In this report, a comparison between the boot process of the debian releases from woody to sid is presented. Besides, the first hotspots are investigated.

On the other hand,I've updated the LSB compliance of the scripts in Sid and added some related bug reports to some of the scripts. Besides, I've worked on new bugs on the script for LSB compliance by checking with all the scripts.

btw, anybody interested on this project can always join us every day at irc.debian.org at the channel #pkg-sysvinit (Europe daytime normally :D ).

Preload status and back to bechmarking (SoC 2006)

2006-07-02T06:49:00.000-07:00

I've went back to the subject of benchmarking Debian releases in order to prepare the second deliverable. It should be available tonight for review in the project webpage.

Besides, we are trying to explore how to make the debian package of preload to work properly such that we can notice any time improvement. The default configuration and with little variations show no time difference. Anybody out there knows about some examples of how to configure preload?

preload and LSB compliance webpage

2006-06-30T01:59:00.000-07:00

I started to try another hotspot: preload. Unfortunately, I noticed no change on the boot time after installing preload. Not even after changing its boot order from /etc/rc2.d/ to /etc/rcS.d/ after the dependencies were met: in the init script it claims to require $time, $remote_fs and $local_fs. I believe seeing it in ubuntu at just the beginning of /etc/rcS.d/. The bootcharts and test procedures can be seen here. This is an issue that has to be further investigated.

On the other side, the lsb check script was debugged a little bit more. A list with the compliance of the init scripts in sid is available here. The few LSB compliant scripts are on the top, the big incomplete majority in the middle and those without LSB headers at the bottom.

Debugging the LSB check in lintian/linda.

2006-06-29T01:33:00.000-07:00

With run-time dependencies in the init scripts (as LSB headers) we can verify the correct boot order of the system. Currently, lintian just checks the init scripts to provide start, stop, restart and force-reload actions. A small modification could allow it to check for LSB compliance as well and decrease the big amount of scripts lacking run-time dependencies until now.
Currently we have a script to verify LSB compliance and is on a debugging phase. One of the issues is when to warn about a missing information on the stop phase, like Required-Stop, Should-Stop and Default-Stop. Changes we made were not to warn (at least for Default-Stop) when the system default start is only runtime S (initialization).

Finally, there are now some way to track the bugs related to this project with BTS usertags. Most of them have been submitted by Petter Reinholdtsen. Follow this link to see the bugs.

Script for checking LSB compliance (SoC 2006)

2006-06-28T03:42:00.000-07:00

Based on a script by Petter Reinholdtsen to check the correct init script order, I've made a script to check for LSB compliance. It is available on the project webpage. The script checks for the existance of each of the LSB headers and considers 4 main output cases:
+ No LSB headers (when the START INIT INFO header is not found)
+ Basic LSB compliance (when Should-Start, Should-Stop and the Description have no arguments.
+ Full LSB compliance (when all headers have arguments)
+ Partial compliance (in the other cases).
Besides, it sends to to STDERR comments on missing headers or missing arguments in them.