bzip.org changes hands [LWN.net]

The bzip2 compression algorithm has been slowly falling out of favor, but is still used heavily across the net. A search for "bzip2 source" returns bzip.org as the first three results. But it would seem that the owner of this domain has let it go, and it is now parked and running ads. So we no longer have an official home for bzip2. If a new repository or tarball does turn up at that domain, it should be looked at closely before being trusted. (Thanks to Jason Kushmaul).

to post comments

bzip.org changes hands

Posted Aug 9, 2018 22:20 UTC (Thu) by birdie (guest, #114905) [Link] (4 responses)

A Git page would be enough for this project. There's no need to pay for a domain name and hosting.

bzip.org changes hands

Posted Aug 9, 2018 22:56 UTC (Thu) by cesarb (subscriber, #6266) [Link] (3 responses)

This project long predates Git. It predates Subversion. It predates even SourceForge.

bzip.org changes hands

Posted Aug 9, 2018 23:09 UTC (Thu) by stumbles (guest, #8796) [Link] (2 responses)

I fail to see the relevance.

bzip.org changes hands

Posted Aug 10, 2018 15:47 UTC (Fri) by briangmaddox (guest, #39279) [Link] (1 responses)

I believe the relevance is there is more to do than simply taking a zip of the code and sticking it into Github (ie, the history of commit messages from other version control systems need to be carried forward, etc).

bzip.org changes hands

Posted Aug 12, 2018 23:42 UTC (Sun) by ceplm (subscriber, #41334) [Link]

Certainly, and where does pray that history and older commits live?

bzip.org changes hands

Posted Aug 9, 2018 22:51 UTC (Thu) by cesarb (subscriber, #6266) [Link] (20 responses)

> A search for "bzip2 source" returns bzip.org as the first three results.

I normally use "yum info" when I want to know the official homepage of a project. On my machine, "yum info bzip2" does return "URL: http:// www.bzip.org/" (space added to prevent linking).

bzip.org changes hands

Posted Aug 9, 2018 23:11 UTC (Thu) by karkhaz (subscriber, #99844) [Link] (19 responses)

...possibly raise a bug with your distro's maintainer please?

Packages with stale information are bad news. At some point, I attempted to build most of the Arch Linux packages from scratch, in dependency order. There were 140 of them that didn't build from source. In many cases this was because the PKGBUILD (Arch Linux's package build manifest) specified a URL to download the source code from that was stale or misspelled. For example, one of these packages was `licenses`, which is a package that contains the texts of the text of all known FOSS licenses---it was not building because it could not retrieve the text of Mozilla's MPL, since Mozilla had broken the URL where it was hosted since the package was last built.

I thought this was a bit worrisome from a legal perspective because many FOSS licenses stipulate that a distributor of binaries must also distribute the source code---but in the case of those 140 packages, whose binaries are hosted by Arch's mirrors, the source code was not to be found at the specified URL.

(the list of such broken packages is here if anybody's curious, they've all since been fixed: https://www.archlinux.org/todo/packages-with-missing-sour...)

bzip.org changes hands

Posted Aug 9, 2018 23:23 UTC (Thu) by rahulsundaram (subscriber, #21946) [Link] (4 responses)

> Packages with stale information are bad news

Yes they should be fixed but Fedora doesn't depend directly on upstream sources to do the build. The equivalent source is always in the SRPM. FYI.

bzip.org changes hands

Posted Aug 9, 2018 23:33 UTC (Thu) by karkhaz (subscriber, #99844) [Link] (3 responses)

Yes, and I suppose that the other large distros do the same---it seems sensible, both for resiliency against projects dying, and for fulfilling the obligations of the GPL etc. Not sure why Arch doesn't, but IIRC the size of all the source packages together was about 350 GB, so maybe they just don't have the resources to host them all.

bzip.org changes hands

Posted Aug 10, 2018 8:30 UTC (Fri) by Sesse (subscriber, #53779) [Link] (2 responses)

Storing 350 GB of data in S3 is $8/month, so surely the storage costs isn't the big issue. If it's a bandwidth problem, I'm fairly certain they could get something sponsored.

The hidden cost of mirroring

Posted Aug 10, 2018 15:58 UTC (Fri) by smoogen (subscriber, #97) [Link]

Static storage and CPU costs are usually the least amount of costs at any 'cloud' provider. What kills putting mirrored content up there is the constantly changing datasets which eat into both upload/download and various IO costs that different providers have. A couple of people who have tried to mirror just Fedora rawhide (which would be the 'closest' to Arch) have found that their 6 month planned budgets eaten in a month (or less) due to the amount of change in a distribution.

Sponsorship helps a lot but the provider usually will need to know how they are getting their money's worth from this because it is lost profit in a business that they make pennies on the dollar of revenue.

bzip.org changes hands

Posted Aug 13, 2018 9:46 UTC (Mon) by giantcheeseburger (guest, #125574) [Link]

Hosting can be even cheaper than that, free even. The Open Source Lab at Oregon State University (OSUOSL)[1] offers hosting to free and open source projects absolutely free of charge.

[1] http://osuosl.org/services/hosting/

bzip.org changes hands

Posted Aug 10, 2018 6:32 UTC (Fri) by epa (subscriber, #39769) [Link] (1 responses)

Surely the definitive copy of the licence text should itself be in a git repository. Then the package info can give both the location of the repository and the SHA.

(Is there a URI scheme for an object in a git repository which is itself accessed over http/s or ssh?)

bzip.org changes hands

Posted Aug 10, 2018 8:36 UTC (Fri) by karkhaz (subscriber, #99844) [Link]

The source package for `licenses` [1] seems to already contain copies of some of the licenses, and retrieves the others over the network at build time. (I think this is another no-no at least for Debian packages, right? Source packages should contain everything they need to be built without accessing the net.)

The URLs for the absent licenses are all fairly short and descriptive [2]; if there were a URI scheme for accessing git objects, it would be nice if they could be similarly succinct. There may be web interfaces that give you these URLs, for example it looks like Arch Linux uses cgit for their PKGBUILDS, [3] is the URL for a particular file in the licenses package.

[1] https://git.archlinux.org/svntogit/packages.git/tree/trun...
[2] https://www.gnu.org/licenses/gpl-3.0.txt
[3] https://git.archlinux.org/svntogit/packages.git/tree/trun...

bzip.org changes hands

Posted Aug 10, 2018 11:54 UTC (Fri) by amarao (guest, #87073) [Link] (11 responses)

As usual, the Debian approach looks the best. Even outside of 'disappearing URLs', Debian requires it packages to be rebuild without network access, and all sources to be provided from within Debian repositories.

There was once the event, when it helped me a lot when we lost source code for one of the libraries for our inhouse application. I was not able to find the original git anywhere, but we've used a private deb repositories (aptly) and git-build-package had built both binary and source packages, and we uploaded them. I was able to recover source code from there.

I see no big difference between our sloppiness with source code and upstream revolution causing of disappearance of the source code. Both leads to the same result, and Debian model allows to survive such hiccups.

bzip.org changes hands

Posted Aug 11, 2018 4:19 UTC (Sat) by pabs (subscriber, #43278) [Link]

In addition, Debian also stores all old versions (both binary and source):

https://snapshot.debian.org/

bzip.org changes hands

Posted Aug 12, 2018 20:05 UTC (Sun) by ceplm (subscriber, #41334) [Link] (9 responses)

As usual, Debian fanboys don't look outside of their own distro. Of course, every distro I know caches source tarballs and builds on systems without network access (I know for sure about Fedora/RHEL, *SUSE, but I am certain that all major distros do the same).

bzip.org changes hands

Posted Aug 12, 2018 21:27 UTC (Sun) by k8to (guest, #15413) [Link] (1 responses)

Other distributions have source packages for sure.

Many other distributions, though, don't usually consider it a severe bug when a package fails to build from source and actively test them to see that they still do.

bzip.org changes hands

Posted Aug 12, 2018 23:44 UTC (Sun) by ceplm (subscriber, #41334) [Link]

??? Which distribution doesn't consider FTBFS a serious bug needed to be fixed fast?

bzip.org changes hands

Posted Aug 14, 2018 15:09 UTC (Tue) by jwilk (subscriber, #63328) [Link] (1 responses)

Do Debian buildds actually block external network access? I haven't seen any evidence of this, but maybe something changed when I wasn't paying attention.

bzip.org changes hands

Posted Aug 16, 2018 7:12 UTC (Thu) by smcv (subscriber, #53363) [Link]

Unfortunately not (but reproducible-builds.org does, so their systematic rebuilds of the archive would catch network accesses).

bzip.org changes hands

Posted Aug 15, 2018 18:22 UTC (Wed) by deepfire (guest, #26138) [Link] (2 responses)

NixOS:

1. Defines hashed upstream sources as inputs to the derivations, so changes cannot slip by inadvertently.
2. Transparently caches upstream sources for release channels, which provides a degree of protection against upstreams going AWOL.
3. Builds everything in a network-isolated sandbox.
4. Provides deterministic builds.

bzip.org changes hands

Posted Aug 15, 2018 21:32 UTC (Wed) by amarao (guest, #87073) [Link] (1 responses)

Are deterministic builds of NixOS the same as reproducible build of Debian? (That means, byte-to-byte match of rebuild package of the package to another rebuild of the same package of the same version on another machine)

As far as I know, Debian is really moving toward that goal, but there are many packages which give trouble to be rebuild with exactly same layout of everything.

bzip.org changes hands

Posted Aug 15, 2018 22:55 UTC (Wed) by deepfire (guest, #26138) [Link]

Yes, that is the point, and yes it's not a 100% deterministic story yet, though -- while NixOS has probably the most systematic and advanced machinery to support build purity, there are exceptions, such as PGO builds in GCC, which yield different results on different CPUs.

One solution, of course, is to disable PGO in GCC, at the cost of performance.

bzip.org changes hands

Posted Aug 16, 2018 19:27 UTC (Thu) by jond (subscriber, #37669) [Link] (1 responses)

Seeing as this sub-thread started as a discussion of a distribution not doing this, no, they don't all do this.

bzip.org changes hands

Posted Aug 16, 2018 20:09 UTC (Thu) by ceplm (subscriber, #41334) [Link]

You are right, sorry. That is quite horrible news to hear.

bzip.org changes hands

Posted Aug 9, 2018 23:39 UTC (Thu) by synaxin (guest, #122012) [Link] (3 responses)

I made this when I found it and reported it here
https://gitlab.com/bzip/bzip2

bzip.org changes hands

Posted Aug 10, 2018 5:13 UTC (Fri) by burki99 (subscriber, #17149) [Link]

Julian Seward is still active: https://sourceware.org/git/?p=valgrind.git;a=commit;h=e52...
Why not contact him directly and ask if he will reactivate the bzip domain or move the code to sourceware or some other git host of his choice.

bzip.org changes hands

Posted Aug 10, 2018 7:03 UTC (Fri) by sscherfke (guest, #126305) [Link] (1 responses)

Thx for that. I also found https://github.com/enthought/bzip2-1.0.6 (which is ~5y old). Both repos contain exactly the same files, so I guess we can trust them.

bzip.org changes hands

Posted Aug 11, 2018 15:08 UTC (Sat) by khim (subscriber, #9252) [Link]

Why not grab file from something like https://web.archive.org/web/20140801000000/http://www.bzip.org/1.0.6/bzip2-1.0.6.tar.gz?

That one is highly unlinkely to be wrong one... Versions from 1.0.1 to 1.0.5 are also there. And couple of historic versions are on sourceware.

But yeah, there are no canonical page. Not anymore...

bzip.org changes hands

Posted Aug 10, 2018 0:30 UTC (Fri) by Beolach (guest, #77384) [Link]

The Richard Yao message thread quoted in this week's Distribution quotes of the week is relevant to this.

bzip2 will move back to sourceware.org

Posted Aug 10, 2018 15:41 UTC (Fri) by mjw (subscriber, #16740) [Link]

bzip.org is unfortunately no longer available for the bzip2 project. It will soon be restored at its former project home at https://sourceware.org/bzip2 (which currently only hosts older releases).

bzip.org changes hands

Posted Aug 11, 2018 11:45 UTC (Sat) by geuder (subscriber, #62854) [Link] (27 responses)

Do we still need bzip2? If I want good compression I use xz. If I want speed (e.g compressing disk images) I use lzo.

Sometimes I have used pbzip2, because it runs in parallel on several cores. I have not checked what is the relationship between the classic and the parallel implementation.

bzip.org changes hands

Posted Aug 11, 2018 13:22 UTC (Sat) by pr1268 (guest, #24648) [Link] (1 responses)

The practical need for bz2 (or lack thereof) is not the point here; note our editor's caution about checking the veracity and security of any content downloaded from the bzip.org domain should it reappear. Such quiet "drive-by-night" domain changes are a big red flag for unknowingly downloading malicious or nefarious content.

Also, despite xz's better performance, and even though you prefer xz (as do I), there are still thousands of software programs / packages, etc. that still use and rely on bzip compression. If not the command-line utility itself, then certainly the supporting libraries.

Based on the other comments above, it seems important that the valid sources for bzip2 be freely available from a centralized and properly-sanctioned repository. Why the maintainer(s) abandoned this domain? I don't know.

bzip.org changes hands

Posted Aug 11, 2018 15:09 UTC (Sat) by geuder (subscriber, #62854) [Link]

True.

My yocto build also has

SRC_URI = "http://www.bzip.org/${PV}/${BP}.tar.gz \

I wouldn't have noticed the problem before the disk containing my source mirror crashes. And even then the build should not have accepted a fake download, because there is a checksum.

Out of the 195 tarballs my build uses, 43 are compressed using bzip2. Still 22%.

On the machine I type this I get

$ sudo rpm -e --test libbz2-1 2>&1 | grep -c "is needed"
31

Amongst them systemd, rpm, Python, Perl, genisoimage, and libavformat.

So even if the format feels useless legacy, we won't get rid of it in the next 10 years even if we tried hard.

bzip.org changes hands

Posted Aug 11, 2018 17:21 UTC (Sat) by josh (subscriber, #17465) [Link] (18 responses)

> If I want good compression I use xz. If I want speed (e.g compressing disk images) I use lzo.

These days, the answer tends to be "if you want good compression, use zstd; if you want speed, also use zstd; if you want compatibility with DEFLATE, use zlib".

bzip.org changes hands

Posted Aug 11, 2018 23:44 UTC (Sat) by geuder (subscriber, #62854) [Link]

> use zstd

Interesting. I vaguely remember having read about it in the past. But I have never bumped into any file compressed by it. Was not even installed on my machine. Now I installed it. Let's see whether I'll remember to use it or forget again...

bzip.org changes hands

Posted Aug 12, 2018 8:58 UTC (Sun) by ncm (guest, #165) [Link] (15 responses)

Lz4 is probably faster -- I get 1GB/s decompression from it.

xz, as a file format, has severe enough problems that it should never be used for archiving or distribution.

bzip.org changes hands

Posted Aug 12, 2018 18:11 UTC (Sun) by ncm (guest, #165) [Link] (3 responses)

It looks like zstd decompresses faster than lz4. That is very fast indeed.

bzip.org changes hands

Posted Aug 15, 2018 0:45 UTC (Wed) by ncm (guest, #165) [Link] (2 responses)

Actually I am wrong again -- lz4 is quite a bit faster than the fastest mode ("-1") of zstd, both compressing and decompressing. lz4 just doesn't compress as well. But "zstd -c -1" is not *much* slower, and compresses quite a lot better, so it's not an easy choice. But it is a very easy choice to switch from almost any other general-purpose lossless compressor.

I particularly like that if you concatenate .zst files, the decompressed output is the concatenation of the original input. I imagine that is a side effect of supporting "-T0" (use all available CPUs, in parallel), but it has many other uses.

bzip.org changes hands

Posted Aug 16, 2018 8:59 UTC (Thu) by epa (subscriber, #39769) [Link] (1 responses)

You can concatenate .gz files too.

bzip.org changes hands

Posted Aug 18, 2018 4:11 UTC (Sat) by ncm (guest, #165) [Link]

One may learn something new every day, and like it.

bzip.org changes hands

Posted Aug 13, 2018 9:40 UTC (Mon) by vadim (subscriber, #35271) [Link] (10 responses)

Interesting, what are those problems?

bzip.org changes hands

Posted Aug 13, 2018 9:56 UTC (Mon) by giantcheeseburger (guest, #125574) [Link] (9 responses)

While I'm not personally able to speak to the validity of the arguments I believe that "Xz format inadequate for long-term archiving" outlines the issues with the xz format.

bzip.org changes hands

Posted Aug 13, 2018 10:38 UTC (Mon) by josh (subscriber, #17465) [Link] (3 responses)

Those arguments have been mostly debunked a while back, when that article was first posted to various places including debian-devel. (I don't have the links handy for that debunking, unfortunately.) That article is more or less propaganda for the much more obscure lzip format.

bzip.org changes hands

Posted Aug 13, 2018 15:02 UTC (Mon) by mirabilos (subscriber, #84359) [Link] (2 responses)

I’d be interested in actual debunking arguments, as António and someone else have made their points to me in a decent enough manner, and from what I remember on the Debian side, there was much FUD from xz fans involved as well. So I’d prefer to compare myself.

xz is just as obscure a format, it simply was adopted by some key players earlier. And have you read its format specification? I did, early, and I was not impressed by how the container structure is laid out, nor its complexity. (I still adopted xz for some uses simply due to its improvements over bzip2 and LZMA-Alone.)

bzip.org changes hands

Posted Aug 13, 2018 16:10 UTC (Mon) by rahulsundaram (subscriber, #21946) [Link] (1 responses)

> xz is just as obscure a format, it simply was adopted by some key players earlier

That seems a self contradiction. If it was adopted by key players, it is no longer obscure.

bzip.org changes hands

Posted Aug 15, 2018 18:30 UTC (Wed) by deepfire (guest, #26138) [Link]

Obscure as:

1. not widely heralded
2. not easy to understand, which also could be:
2.1 gratuitiously complicated
2.2 complications mirror objective necessities

..a variety of options.

bzip.org changes hands

Posted Aug 14, 2018 5:51 UTC (Tue) by epa (subscriber, #39769) [Link] (4 responses)

It always surprises me that the gzip format, which has provision for different compression methods, was never more widely used. (The files could have a different extension from .gz to avoid confusing people, but otherwise follow the RFC 1952 format, with a different value in the CM (compression method) byte.) Perhaps the gzip format is often a bit more complex than you'd want to support when first hacking up a compression tool, but it's certainly preferable to inventing yet another container format.

bzip.org changes hands

Posted Aug 16, 2018 19:30 UTC (Thu) by jond (subscriber, #37669) [Link] (3 responses)

But unless I'm mistaken, gzip is *not* a container format. (nor is bzip2).

bzip.org changes hands

Posted Aug 18, 2018 4:10 UTC (Sat) by ncm (guest, #165) [Link] (1 responses)

It is necessarily a container for a compressed stream. You perhaps mean that it is not an archiver for a collection of files, but we have no need for that in a compression container, because we may compress a tar or cpio archive into it. Combined archiver / compressors only arose in environments that lacked useful archive formats, such as windos and macos.

What makes it a container is the header and metadata format that, as noted has a field for the compression algorithm used.

bzip.org changes hands

Posted Aug 18, 2018 6:38 UTC (Sat) by Cyberax (✭ supporter ✭, #52523) [Link]

Actually, combined archives arose because they are better than gzipped tarballs. In particular, ZIP allows editing files in-place.

This is not a problem these days, but back at the 80286 era it was much easier and faster to deal with ZIP files than tarballs.

bzip.org changes hands

Posted Aug 19, 2018 9:40 UTC (Sun) by epa (subscriber, #39769) [Link]

I meant for the Unix-style compression programs that just compress a single input to a single output. Obviously a program like 7-zip, which can combine multiple files into an archive, needs a different format. But most of the programs we're talking about here (gzip, bzip2, xz, zstd, lrzip, lz4 and so on) are most often used in single-file mode on Linux. If you want an archive, it's more common (for better or worse) to make a tar archive and compress that.

bzip.org changes hands

Posted Aug 12, 2018 20:17 UTC (Sun) by flussence (guest, #85566) [Link]

If you want compatibility with medium-age distros, use zstd too. It compresses xz files better than xz does in a lot of cases.

bzip.org changes hands

Posted Aug 16, 2018 19:28 UTC (Thu) by jond (subscriber, #37669) [Link] (2 responses)

> Do we still need bzip2? If I want good compression I use xz.

I don't need it for compression, but I sure hope I can always get my hands on it for decompression.

bzip.org changes hands

Posted Aug 18, 2018 4:02 UTC (Sat) by ncm (guest, #165) [Link] (1 responses)

Zstandard will decompress both, so you may disregard them freely.

bzip.org changes hands

Posted Aug 18, 2018 21:15 UTC (Sat) by lsl (subscriber, #86508) [Link]

Hmm, this doesn't seem to be true.

> % echo abc >a && bzip2 a
> % unzstd a.bz2
> zstd: a.bz2: unknown suffix (.zst expected) -- ignored
> % zstdcat <a.bz2
> [garbage]
> % zstd --version
> *** zstd command line interface 64-bits v1.3.5, by Yann Collet ***

The man page doesn't mention this capability either.

bzip.org changes hands

Posted Aug 18, 2018 8:11 UTC (Sat) by jezuch (subscriber, #52988) [Link] (2 responses)

Well, due to the nature of the compression algorithm (Burrows-Wheeler Transform), bzip2 may still be better than xz still, for example for text data. But these are rare cases nowadays as bzip2 definitely shows its age (900kB max block size for example - for BWT it's the bigger the better). Oh, and it's sloooow :)

bzip.org changes hands

Posted Aug 18, 2018 10:58 UTC (Sat) by diegor (subscriber, #1967) [Link] (1 responses)

bzip2 is slow, but still way faster of xz at the same compression ratio.

bzip.org changes hands

Posted Aug 21, 2018 7:22 UTC (Tue) by jezuch (subscriber, #52988) [Link]

The thing is that with bzip2 the decompression is slow as well. LZMA used in xz is a lz77-type algorithm which is usually slow (arbitrarily slow, one can say) when compressing but decompression is relatively fast (algorithmically it's almost trivial). Bzip2 during decompression does basically the same thing as when compressing only in reverse. It's infuriating at times :)

bzip.org changes hands

Posted Aug 13, 2018 16:38 UTC (Mon) by felix.s (guest, #104710) [Link]

Given that it's just the domain name that's expired, you can always add a line to your hosts file to access the original site:

46.235.226.80 bzip.org

Of course, adding lines to hosts file given to you by random strangers is not always the wisest idea. You can always check the tarballs against the Debian .orig.tar.bz2 package just to be sure.

bzip.org changes hands

Posted Aug 13, 2018 20:05 UTC (Mon) by Conan_Kudo (subscriber, #103240) [Link]

My understanding is that bzip2 has never been version controlled officially. Does anyone know otherwise?