LaForge's home page

Security Issues regarding GSMA eSIMs / eUICCs + Javacard

Harald Welte — Tue, 08 Jul 2025 22:00:00 GMT

The independent security researcher Adam Gowdiak has published an extensive report on flaws he found in some eUICCs (the chips used to store eSIM profiles within the GSMA eSIM architecture). While the specific demonstrable exploit was in a product of one specific CardOS Vendor (Kigen, formerly part of ARM), the fundamental underlying issue is actually an architectural one.

The Oracle Javacard [memory] safety architecture relies on a so-called bytecode verifier which is a program that you run after compiling an application, but before executing the code on the Javacard. The specifications allow for both on-card and off-card verification. However, the computational complexity of this verifier is generally assumed to exceed the resources available inside many microcontrollers used to implement java cards. Such microcontrollers often are ARM SC000 (Cortex-M0 based) or SC300 (Cortex-M3 based) based, with only tens of kilobytes of RAM and hundreds of kilobytes of flash.

Javacard was originally developed for use cases within the banking/payment industry. In that industry, the card-issuing bank is the sole entity that has the keys to load java applets onto a card. That entity is of course interested in the security of the card, and will hence always run an off-card bytecode verifier. In a world of physical SIM/USIM cards issued by a single mobile operator, the situation is the same: The card-issuing MNO/MVNO is the only entity with key materials to install additional java applets on the card.

This fundamental problem became already apparent by earlier findings by Adam Gowdiak in 2019, but at least in terms of public responses by Oracle and Gemalto back then, they mostly did hand-waving and/or made lame excuses.

However, when the industry represented in GSMA standardized the eSIM architecture, this changed. Suddenly we have various eSIM profiles of various different operators, each holding key material to install Java applets on the shared card. In such an environment, it is no longer safe to assume that every MNO/MVNO can be trusted to be non-adverserial and hence trusted to run that off-card bytecode verifier before loading applets onto the card.

If the Javacard runtime on the existing card/chip itself cannot autonomously perform those verification tasks, I don't see how the problem can ever be solved short of completely removing/disabling Javacard support in such eUICCs. Luckily it is an optional feature and not a mandatory requirement for an eUICC to be approved/accredited. Sadly many MNOs/MVNOS however will mandate Javacard support in their eSIM profiles and hence refuse to install into an eUICC without it :(

In my opinion, the solution to the problem can only be to either make the GSMA require full on-card bytecode verification on all eUICCs, or to remove Javacard support from the eUICC.

We have to keep in mind that there are hundreds if not thousands of MVNOs around the planet, and all of them are subject to whatever local jurisdiction they operate in, and also subject to whatever government pressure (e.g from intelligence agencies).

In hindsight, anyone familiar with the 2019 work by Gowdiak and an understanding of the fundamental change to multiple stakeholders in an eUICC (compared to classic SIM/USIM) should have arrived at the conclusion that there is a serious problem that needs addressing. I think the 2019 work had not been publicized and covered significantly enough to make sure that everyone in the industry was made aware of the problems. And that in turn is mostly a result of Oracle + Gemalto downplaying the 2019 findings back in the day, rather than raising awareness within all relevant groups and bodies of the industry.

Mitigation via TS.48 key diversification

The specific attack presented was using a GSMA TS.48 test profile to install the malicious java bytecode; those TS.48 profiles are standardized profiles used by the industry for cellular testing; until the attack they contained well-known static OTA key material. The mitigation to randomize/diversity those keys in TS.48v7 closes that particular vector, but the attack itself is not dependent on test profiles. Any MNO/NVNO (or rather, anyone with access to a commercial service of a SM-DP+ accredited by GSMA) obviously has the ability to load java applets into the eSIM profile that they create, using keys that they themselves specify.

What IMHO ought to be done

Oracle should get off their we only provide a reference implementation and vendors should invent their own prporietary verification mechanisms horse. This is just covering their own ass and not helping any of their downstream users/implementers. The reference implementation should show how proper verification can be done in the most resource-constrained environment of cards (it's JavaCard, after all!), and any advances of the verifier should happen once at Oracle, and then used by all the implementers (CardOS vendors). Anyone who really cares about security of a standardized platform (like Javacard) should never leave key aspects of it up to each and every implementer, but rather should solve the problem once, publicly, with validation and testing tools, independent 3rd party penetration testing and then ensure that every implementer uses that proven implementation.
GSMA should have security requirements (and mandatory penetration tests) specifically regarding the JVM/JRE of each card that gets SAS-UP accredited.
GSMA should require that Javacard support should be disabled on all existing eUICCs that cannot legitimately claim/demonstrate that they are performing full bytecode verification entirely on-card.
GSMA should refuse any future SAS-UP accreditation to any product that requires off-card bytecode verification
The entire industry should find a way to think beyond Javacard, or in fact any technology whose security requires verification of the executable program that is too complex to perform on-card on the targeted microcontrollers.

On Linux MAINTAINERS file removal of Russian developers

Harald Welte — Wed, 23 Oct 2024 22:00:00 GMT

I sincerely regret to see Linux kernel patches like this one removing Russian developers from the MAINTAINERS file. To me, it is a sign or maybe even a symbol of how far the Linux kernel developer community I remember from ~ 20 years ago has changed, and how much it has alienated itself from what I remember back in the day.

In my opinion this commit is wrong at so many different levels:

it is intransparent. Initially it gave no explanation whatsoever (other than some compliance hand-waving). There was some follow-up paraphrasing one paragraph of presumed legal advice that was given presumably by Linux Foundation to Linus. That's not a thorough legal analysis at all. It doesn't even say to whom it was given, and who (the individual developers? Linux Foundation? Distributors?) is presumed to be subject to the unspecified regulations in which specific jurisdiction
it discriminates developers based on their presumed [Russian] nationality based on their name, e-mail address domain name or employer.

A later post in the thread has clarified that it's about an U.S. embargo list against certain Russian individuals / companies. It is news to me that the MAINTAINERS file was usually containing Companies or that the Linux kernel development is Companies engaging with each other. I was under the naive assumption that it's individual developers who work together, and their employers do not really matter. Contributions are judged by their merit, and not by the author or their employer / affiliation. In the super unlikely case that indeed those individual developers removed from the MAINTAINERS file would be personally listed in the embargo list: Then yes, of course, I agree, they'd have to be removed. But then the commit log should of course point to [the version] of that list and explicitly mention that they were personally listed there.

And no, I am of course not a friend of the Russian government at all. They are committing war crimes, no doubt about it. But since when has the collaboration of individual developers in an open source project been something related to actions completely unrelated to those individuals? Should I as a German developer be excluded due to the track record of Germany having started two world wars killing millions? Should Americans be excluded due to a very extensive track record of violating international law? Should we exclude Palestinians? Israelis? Syrians? Iranians? [In case it's not obvious: Those are rhetorical questions, my position is of course no to all of them].

I just think there's nothing more wrong than discriminating against people just because of their passport, their employer or their place of residence. Maybe it's my German upbringing/socialization, but we've had multiple times in our history where the concept of **Sippenhaft** (kin liability) existed. In those dark ages of history you could be prosecuted for crimes committed by other family members.

Now of course removal from the MAINTAINERS file or any other exclusion from the Linux kernel development process is of course not in any way comparable to prosecution like imprisonment or execution. However, the principle seems the same: An individual is punished for mere association with some others who happen to be committing crimes.

Now if there really was a compelling legal argument for this (I doubt it, but let's assume for a second there is): In that case I'd expect a broad discussion against it; a reluctance to comply with it; a search for a way to circumvent said legal requirement; a petition or political movement against that requirement.

Even if there was absolutely no way around performing such a "removal of names": At the very least I'd expect some civil disobedience by at least then introducing a statement into the file that one would have hoped to still be listing those individuals as co-maintainers but one was forced by [regulation, court order, ...] to remove them.

But the least I would expect is for senior Kernel developers to simply do apply the patch with a one-sentence commit log message and thereby disrespect the work of said [presumed] Russian developers. All that does is to alienate individuals of the developer community. Not just those who are subject to said treatment today, but any others who see this sad example how Linux developers treat each other and feel discouraged from becoming or remaining active in a community with such behaviour.

It literally hurts me personally to see this happening. It's like a kick in the gut. I used to be proud about having had an involvement with the Linux kernel community in a previous life. This doesn't feel like the community I remember being part of.

Back to Taiwan the first time after 5 years

Harald Welte — Tue, 22 Oct 2024 22:00:00 GMT

Some of the readers of this blog know that I have a very special relationship with Taiwan. As a teenager, it was the magical far-away country that built most of the PC components in all my PCs since my first 286-16 I got in 1989. Around 2006-2008 I had the very unexpected opportunity to work in Taiwan for some time (mainly for Openmoko, later some consulting for VIA). During that time I have always felt most welcome in and fascinated by the small island nation who managed to turn themselves into a high-tech development and manufacturing site for ever more complex electronics. And who managed to evolve from decades of military dictatorship and turn into a true democracy - all the while being discriminated by pretty much all of the countries around the world, as everybody wanted to benefit from cheap manufacturing in mainland China and hence expel democratic Taiwan from the united nations in favour of communist mainland Chine.

I have the deepest admiration for Taiwan to manage all of their economic success and progress in terms of democracy and freedom despite the political situation across the Taiwan strait, and despite everything that comes along with it. May they continue to have the chance of continuing their path.

Setting economy, society and politics behind: On a more personal level I've enjoyed their culinary marvels from excellent dumplings around every street corner to niu rou mien (beef noodle soup) to ma la huo guo (spicy hot pot). Plus then the natural beauty, particularly of the rural mountainous regions once you leave the densely populated areas around the coast line and the plains of the north west.

While working in Taiwan in 2006/2007 I decided to buy a motorbike. Using that bike I've first made humble day trips and later (once I was no longer busy with stressful work at Openmoko) multiple week-long road trips around the island, riding on virtually any passable road you can find. My typical routing algorithm is "take the smallest possible road from A to B".

So even after concluding my work in Taiwan, I returned again and again for holidays, each one with more road trips. For some time, Taiwan had literally become my second home. I had my favorite restaurants, shops, as well as some places around the rural parts of the Island I cam back to several times. I even managed to take up some mandarin classes, something I never had the time for while doing [more than] full time work. To my big regret, it's still very humble beginner level; I guess had I not co-started a company (sysmocom) in Berlin in 2011, I'd have spent more time for a more serious story.

In any case, I have nothing but the fondest memory of Taiwan. My frequent visits cam to a forcible halt with the COVID-19 pandemic, Taiwan was in full isolation in 2020/21, and even irrespective of government regulations, I've been very cautious about travel and contact. Plus of course, there's always the bad conscience of frequent intercontinental air travel.

Originally I was planning to finally go on an extended Taiwan holiday in Summer 2024, but then the island was hit by a relatively serious earthquake in April, affecting particularly many of the remote mountain regions that are of main interest to me. There are some roads that I'd have wanted to ride ever since 2008, but which had been closed every successive year when I went there, due to years of reconstructions after [mostly landslides following] earthquakes and typhoons. So I decided to postpone it for another year to 2025.

However, in an unexpected change of faith, the opportunity arose to give the opening Keyonte at the 2024 Open Compliance Summit in Japan, and along with that the opportunity to do a stop-over in Taiwan. It will just be a few days of Taipei this time (no motorbike trips), but I'm very much looking forward to being back in the city I probably know second or third-best on the planet (after Berlin, my home for 23 years, as well as Nuernberg, my place of birth). Let's see what is still the same and what has changed during the past 5 years!

Oral history transcripts: Pioneers of Taiwans Chip + PC industry

Harald Welte — Tue, 22 Oct 2024 22:00:00 GMT

During the preparation of my current brief visit to Taiwan, I've more or less by coincidence stumbled on several transcripts of oral history interviews with pioneers of the Taiwanese Chip and PC industry (click on the individual transcripts in the Related Records section at the bottom). They have been recorded, transcribed and translated in 2011 by the Computer History Museum under funding from the National Science Council, Taiwan, R.O.C..

As some of you know, I've been spending a lot of time in recent years researching (and practically exploring + re-implementing) historical telecommunications with my retronetworking project.

Retrocomputing itself is not my main focus. I usually feel there's more than enough people operating, repairing, documenting at least many older computers, as well as keeping archives of related software and continuing to spread knowledge on how they operated. Nevertheless, it is a very interesting topic - I just decided that with my limited spare time I want to focus on retro-communications which is under-explored and under-represented.

What's equally important than keeping the old technology alive, is keeping the knowledge around its creation alive. How did it happen that certain technologies were created and became successful or not? How where they key people behind it? etc.

Given my personal history with Taiwan during the last 18 years, it's actually surprising I haven't yet given thought on how or where the history of the Taiwanese IT industry is documented or kept alive. So far I didn't know of any computer museums that would focus especially on the Taiwanese developments. It didn't even occur to me to even check if there are any.

During my work in Taiwan I've had the chance to briefly meet a few senior people at FIC (large mainboard maker that made many PC mainboards I personally used) and both at VIA (chipset + CPU maker). But I didn't ever have a chance to talk about the history.

In any case, I now found those transcripts of interviews. And what a trove of interesting first-hand information they are! If you have an interest in computer history, and want to understand how it came about that Taiwan became such a major player in either the PC industry or in the semiconductor design + manufacturing, then I believe those transcripts are a "must read".

Now they've made me interested to learn more. I have little hope of many books being published on that subject, particularly in a Language I can read (i.e. English, not mandarin Chinese). But I shall research that subject. I'd also be interested to hear about any other information, like collections of historical artifacts, archives, libraries, etc. So in the unlikely case anybody reading this has some pointers on information about the history of the Taiwanese Chip and Computer history, please by all means do reach out and share!.

Once I have sufficiently prepared myself in reading whatever I can find in terms of written materials, I might be tempted to try to reach out and see if I can find some first-hand witnesses who'd want to share their stories on a future trip to Taiwan...

OsmoDevCon 2024: "Introduction to XDP, eBPF and AF_XDP"

Harald Welte — Sat, 04 May 2024 22:00:00 GMT

I've presented a talk Introduction to XDP, eBPF and AF_XDP as part of the OsmoDevCon 2024 conference on Open Source Mobile Communications.

This talk provides a generic introduction to a set of modern Linux kernel technologies:

eBPF (extended Berkeley Packet Filter) is a kind of virtual machine that runs sandboxed programs inside the Linux kernel.
XDP (eXpress Data Path) is a framework for eBPF that enables high-performance programmable packet processing in the Linux kernel
AF_XDP is an address family that is optimized for high-performance packet processing. It allows in-kernel XDP eBPF programs to efficiently pass packets to userspace via memory-mapped ring buffers.

The talk provides a high-level overview. It should provide some basics before the other/later talks on bpftrace and eUPF.

You can find the video recording at https://media.ccc.de/v/osmodevcon2024-204-introduction-to-xdp-ebpf-and-afxdp

OsmoDevCon 2024: "Using bpftrace to analyze osmocom performance"

Harald Welte — Sat, 04 May 2024 22:00:00 GMT

I've presented a talk Using bpftrace to analyze osmocom performance as part of the OsmoDevCon 2024 conference on Open Source Mobile Communications.

bpftrace is a utility that uses the Linux kernel tracing infrastructure (and eBPF) in order to provide tracing capabilities within the kernel, like uprobe, kprobe, tracepoints, etc.

bpftrace can help us to analyze the performance of [unmodified] Osmocom programs and quickly provide information like, for example:

Histogram of time spent in a specific system call
Histogram of any argument or return value of any system call

You can find the video recording at https://media.ccc.de/v/osmodevcon2024-203-using-bpftrace-to-analyze-osmocom-performance

OsmoDevCon 2024: "Anatomy of the eSIM Profile"

Harald Welte — Fri, 03 May 2024 22:00:00 GMT

I've presented a talk Anatomy of the eSIM Profile as part of the OsmoDevCon 2024 conference on Open Source Mobile Communications.

In the eSIM universe, eSIM profiles are the virtualised content of a classic USIM (possibly with ISIM, CSIM, applets, etc.).

Let's have a look what an eSIM profile is:

how is the data structured / organized?
what data can be represented in it?
how to handle features provided by eUICC, how can the eSIM profile mandate some of them?
how does personalization of eSIM profiles work?

There is also hands-on navigation through profiles, based on the pySim.esim.saip module.

You can find the video recording at https://media.ccc.de/v/osmodevcon2024-174-anatomy-of-the-esim-profile

OsmoDevCon 2024: "Detailed workings of OTA for SIM/USIM/eUICC"

Harald Welte — Fri, 03 May 2024 22:00:00 GMT

I've presented a talk Detailed workings of OTA for SIM/USIM/eUICC as part of the OsmoDevCon 2024 conference on Open Source Mobile Communications.

Everyone knows that OTA (over the air) access to SIM cards exists for decades, and that somehow authenticated APDUs can be sent via SMS.

But let's look at the OTA architecture in more detail:

OTA transport (SCP80) over SMS, USSD, CellBroadcast, CAT-TP, BIP
The new SCP81 transport (HTTPS via TLS-PSK)
how to address individal applications on the card via their TAR
common applications like RFM and RAM
custom applications on the card
OTA in the world of eUICCs
talking to the ECASD
talking to the ISD-R
talking to the ISD-P/MNO-SD or applications therein

You can find the video recording at https://media.ccc.de/v/osmodevcon2024-175-detailed-workings-of-ota-for-sim-usim-euicc

OsmoDevCon 2024: "GlobalPlatform in USIM and eUICC"

Harald Welte — Fri, 03 May 2024 22:00:00 GMT

I've presented a talk GlobalPlatform in USIM and eUICC as part of the OsmoDevCon 2024 conference on Open Source Mobile Communications.

The GlobalPlatform Card Specification and its many amendments play a significant role in most real-world USIM/ISIM, and even more so in eUICC.

The talk will try to provide an overview of what GlobalPlatform does in the telecommunications context.

Topics include:

security domains
key loading
card and application life cycle
loading and installation of applications
Secure Channel Protocols SCP02, SCP03

You can find the video recording at https://media.ccc.de/v/osmodevcon2024-173-globalplatform-in-usim-and-euicc

OsmoDevCon 2024: "High-performance I/O using io_uring via osmo_io"

Harald Welte — Thu, 02 May 2024 22:00:00 GMT

I've co-presented a talk (together with Andreas Eversberg High-performance I/O using io_uring via osmo_io as part of the OsmoDevCon 2024 conference on Open Source Mobile Communications.

Traditional socket I/O via read/write/recvfrom/sendto/recvmsg/sendmsg and friends creates a very high system call load. A highly-loaded osmo-bsc spends most of its time in syscall entry and syscall exit.

io_uring is a modern Linux kernel mechanism to avoid this syscall overhead. We have introduced the osmo_io`API to libosmocore as a generic back-end for non-blocking/asynchronous I/O and a back-end for our classic `osmo_fd / poll approach as well as a new backend for io_uring.

The talk will cover

a very basic io_uring introduction
a description of the osmo_io API
the difficulties porting from osmo_fd to osmo_io
status of porting various sub-systems over to osmo_io

You can find the video recording at https://media.ccc.de/v/osmodevcon2024-209-high-performance-i-o-using-iouring-via-osmoio

Gradual migration of IP address/port between servers

Harald Welte — Fri, 29 Mar 2024 23:00:00 GMT

I'm a strong proponent of self-hosting all your services, if not on your own hardware than at least on dedicated rented hardware. For IT nerds of my generation, this has been the norm sicne the early 1990s: If you wante to run your own webserver/mailserver/... back then, the only way was to self-host.

So over the last 30 years, I've always been running a fleet of machines, some my own hardware colocated, and during the past ~18 years also some rented dedicated "root servers". They run a plethora of services for either my personal stuff (like this blog, or my personal email server), or any of the IT services of the open source projects I'm involved in (like osmocom) or the company I co-founded and run (sysmocom).

Every few years there's the need to migrate to new hardware. Either due to power consumption/efficiency, or to increase performance, or to simply avoid aging hardware that may be dying soon.

When upgrading from one [hosted] server to another [hosted] server, there's always the question of how to manage the migration with minimal interruption to services. For very simple services like http/https, it can usually be done entirely within DNS: You reduce the TTL of the records, bring up the service on the new server (with a new IP), make the change in the DNS and once the TTL of the DNS record is expired in all caches, everybody will access the new server/IP.

However, there are services where the IP address must be retained. SMTP is a prime example of that. Given how spam filtering works, you certainly will not want to give up your years if not decadeds of good reputation for your IP address. As a result, you will want to keep the IP address while doing the migration.

If it's a physical machine in colocation or your home, you can of course do that all rather easily under your control. You can synchronize the various steps from stopping the services on the old machine, rsync'ing over the spool files to the new, then migrate the IP over to the new machine.

However, if it's a rented "root" server at a company like Hetzner or KVH, then you do not have full control over when exactly the IP address will be migrated over to the new server.

Also, if there are many different services on that same physical machine, running on a variety of different IPv4/IPv6 addresess and ports, it may be difficult to migrate all of them at once. It would be much more practical, if individual services could be migrated step by step.

The poor man's approach would be to use port-forwarding / reverse-proxying. In this case, the client establishes a TCP connection to the old IP address on the old server, and a small port-forward proxy accepts that TCP connectin, creates a second TCP connection to the new server, and bridges those two together. This approach only works for the most simplistic of services (like web servers), where

there are only inbound connections from remote clients (as outbound connections from the new server would originate from the new IP, not the old one), and
where the source IP of the client doesn't matter. To the new server all connections' source IP addresses suddenly are masked and there's only one source IP (the old server) for all connections.

For more sophisticated serviecs (like e-mail/SMTP, again), this is not an option. The SMTP client IP address matters for whitelists/blacklists/relay rules, etc. And of course there are also plenty of outbound SMTP connections which need to originate from the old IP, not the new IP.

So in bed last night [don't ask], I was brainstorming if the goal of fully transparent migration of individual TCP+UDP/IP (IPv4+IPv6) services could be made between and old and new server. In theory it's rather simple, but in practice the IP stack is not really designed for this, and we're breaking a lot of the assumptions and principles of IP networking.

After some playing around earlier today, I was actually able to create a working setup!

It fulfills the followign goals / exhibits the following properties:

old and new server run concurrently for any amount of time
individual IP:port tuples can be migrated from old to new server, as services are migrated step by step
fully transparent to any remote peer: Old IP:port of server visible to client
fully transparent to the local service: Real client IP:port of client visible to server
no constraints on whether or not the old and new IPs are in the same subnet, link-layer, data centre, ...
use only stock features of the Linux kernel, no custom software, kernel patches, ...
no requirement for controlling a router in front of either old or new server

General Idea

The general idea is to receive and classify incoming packets on the old server, and then selectively tunnel some of them via a GRE tunnel from the old machine to the new machine, where they are decapsulated and passed to local processes on the new server. Any packets generated by the service on the new server (responses to clients or outbound connections to remote serveers) will take the opposite route: They will be encapsulated on the new server, passed through that GRE tunnel back to the old server, from where they will be sent off to the internet.

That sounds simple in theory, but it poses a number of challenges:

packets destined for a local IP address of the old server need to be re-routed/forwarded, not delivered to local sockets. This is easily done with fwmark, multiple routing tables and a rule, similar ro many other policy routing setups.
FIXME

RetroNetCall: "Datex-L, the German CSPDN"

Harald Welte — Tue, 31 Jan 2023 23:00:00 GMT

I've presented about Datex-L, the German CSPDN (Circuit Switched Public Data Network) as part of the RetroNetCall talk series on retro-networking technology.

You can find the video recording at https://media.ccc.de/v/retronetcall-20230201-laforge-datex-l-cspdn

I've always been fascinated by "ancient" data communications - both the kind that I personally consciously witnessed from the late 1980s as well as the kind that I never experienced myself (like Datex-L).

I am not an expert in the subject by all means, as I was never involved in its design, implementation or even used it. However, given that there's very few public information online about Datex-L and/or other CSPDNs, I thought I could improve the situation by presenting about it.

RetroNetCall: "ISDN B-Channel protocols"

Harald Welte — Tue, 06 Dec 2022 23:00:00 GMT

I've presented about ISDN B-Channel protocols (X.75, V.120, V.110, T.70, ...) as part of the RetroNetCall talk series on retro-networking technology.

You can find the video recording at https://media.ccc.de/v/retronetcall-20221207-laforge-isdn-b-channel-protocols

Many people with some kind of telecom background are familiar with D-channel (signalling) protocols of ISDN, and you can find many publications on that topic. Surprisingly, much less publications are talking about the B-channel protocols used for data transmission, like VX.75, V.110, V.120, T.70, ...

RetroNetCall: "OCTOI project status update"

Harald Welte — Tue, 08 Nov 2022 23:00:00 GMT

I've presented an OCTOI project status update (Nov 2022) as part of the RetroNetCall talk series on retro-networking technology.

You can find the video recording at https://media.ccc.de/v/retronetcall-20221109-laforge-octoi-status-update

OsmoDevCall: "Osmocom SIMtrace2 Tutorial"

Harald Welte — Tue, 18 Oct 2022 22:00:00 GMT

I've presented an SIMtrace2 Tutorial - SIM protocol tracing: how and why as part of the OsmoDevCall talk series on Osmocom and related technology.

You can find the video recording at https://media.ccc.de/v/osmodevcall-20221019-laforge-simtrace2-tutorial

Deployment of future community TDMoIP hub

Harald Welte — Sun, 18 Sep 2022 22:00:00 GMT

I've mentioned some of my various retronetworking projects in some past blog posts. One of those projects is Osmocom Community TDM over IP (OCTOI). During the past 5 or so months, we have been using a number of GPS-synchronized open source icE1usb interconnected by a new, efficient but strill transparent TDMoIP protocol in order to run a distributed TDM/PDH network. This network is currently only used to provide ISDN services to retronetworking enthusiasts, but other uses like frame relay have also been validated.

So far, the central hub of this OCTOI network has been operating in the basement of my home, behind a consumer-grade DOCSIS cable modem connection. Given that TDMoIP is relatively sensitive to packet loss, this has been sub-optimal.

Luckily some of my old friends at noris.net have agreed to host a new OCTOI hub free of charge in one of their ultra-reliable co-location data centres. I'm already hosting some other machines there for 20+ years, and noris.net is a good fit given that they were - in their early days as an ISP - the driving force in the early 90s behind one of the Linux kernel ISDN stracks called u-isdn. So after many decades, ISDN returns to them in a very different way.

Side note: In case you're curious, a reconstructed partial release history of the u-isdn code can be found on gitea.osmocom.org

But I digress. So today, there was the installation of this new OCTOI hub setup. It has been prepared for several weeks in advance, and the hub contains two circuit boards designed entirely only for this use case. The most difficult challenge was the fact that this data centre has no existing GPS RF distribution, and the roof is ~ 100m of CAT5 cable (no fiber!) away from the roof. So we faced the challenge of passing the 1PPS (1 pulse per second) signal reliably through several steps of lightning/over-voltage protection into the icE1usb whose internal GPS-DO serves as a grandmaster clock for the TDM network.

The equipment deployed in this installation currently contains:

a rather beefy Supermicro 2U server with EPYC 7113P CPU and 4x PCIe, two of which are populated with Digium TE820 cards resulting in a total of 16 E1 ports
an icE1usb with RS422 interface board connected via 100m RS422 to an Ericsson GPS03 receiver. There's two layers of of over-voltage protection on the RS422 (each with gas discharge tubes and TVS) and two stages of over-voltage protection in the coaxial cable between antenna and GPS receiver.
a Livingston Portmaster3 RAS server
a Cisco AS5400 RAS server

For more details, see this wiki page and this ticket

Now that the physical deployment has been made, the next steps will be to migrate all the TDMoIP links from the existing user base over to the new hub. We hope the reliability and performance will be much better than behind DOCSIS.

In any case, this new setup for sure has a lot of capacity to connect many more more users to this network. At this point we can still only offer E1 PRI interfaces. I expect that at some point during the coming winter the project for remote TDMoIP BRI (S/T, S0-Bus) connectivity will become available.

Acknowledgements

I'd like to thank anyone helping this effort, specifically * Sylvain "tnt" Munaut for his work on the RS422 interface board (+ gateware/firmware) * noris.net for sponsoring the co-location * sysmocom for sponsoring the EPYC server hardware

Clock sync trouble with Digium cards and timing cables

Harald Welte — Thu, 08 Sep 2022 22:00:00 GMT

If you have ever worked with Digium (now part of Sangoma) digital telephony interface cards such as the TE110/410/420/820 (single to octal E1/T1/J1 PRI cards), you will probably have seen that they always have a timing connector, where the timing information can be passed from one card to another.

In PDH/ISDN (or even SDH) networks, it is very important to have a synchronized clock across the network. If the clocks are drifting, there will be underruns or overruns, with associated phase jumps that are particularly dangerous when analog modem calls are transported.

In traditional ISDN use cases, the clock is always provided by the network operator, and any customer/user side equipment is expected to synchronize to that clock.

So this Digium timing cable is needed in applications where you have more PRI lines than possible with one card, but only a subset of your lines (spans) are connected to the public operator. The timing cable should make sure that the clock received on one port from the public operator should be used as transmit bit-clock on all of the other ports, no matter on which card.

Unfortunately this decades-old Digium timing cable approach seems to suffer from some problems.

bursty bit clock changes until link is up

The first problem is that downstream port transmit bit clock was jumping around in bursts every two or so seconds. You can see an oscillogram of the E1 master signal (yellow) received by one TE820 card and the transmit of the slave ports on the other card at https://people.osmocom.org/laforge/photos/te820_timingcable_problem.mp4

As you can see, for some seconds the two clocks seem to be in perfect lock/sync, but in between there are periods of immense clock drift.

What I'd have expected is the behavior that can be seen at https://people.osmocom.org/laforge/photos/te820_notimingcable_loopback.mp4 - which shows a similar setup but without the use of a timing cable: Both the master clock input and the clock output were connected on the same TE820 card.

As I found out much later, this problem only occurs until any of the downstream/slave ports is fully OK/GREEN.

This is surprising, as any other E1 equipment I've seen always transmits at a constant bit clock irrespective whether there's any signal in the opposite direction, and irrespective of whether any other ports are up/aligned or not.

But ok, once you adjust your expectations to this Digium peculiarity, you can actually proceed.

clock drift between master and slave cards

Once any of the spans of a slave card on the timing bus are fully aligned, the transmit bit clocks of all of its ports appear to be in sync/lock - yay - but unfortunately only at the very first glance.

When looking at it for more than a few seconds, one can see a slow, continuous drift of the slave bit clocks compared to the master :(

Some initial measurements show that the clock of the slave card of the timing cable is drifting at about 12.5 ppb (parts per billion) when compared against the master clock reference.

This is rather disappointing, given that the whole point of a timing cable is to ensure you have one reference clock with all signals locked to it.

The work-around

If you are willing to sacrifice one port (span) of each card, you can work around that slow-clock-drift issue by connecting an external loopback cable. So the master card is configured to use the clock provided by the upstream provider. Its other ports (spans) will transmit at the exact recovered clock rate with no drift. You can use any of those ports to provide the clock reference to a port on the slave card using an external loopback cable.

In this setup, your slave card[s] will have perfect bit clock sync/lock.

Its just rather sad that you need to sacrifice ports just for achieving proper clock sync - something that the timing connectors and cables claim to do, but in reality don't achieve, at least not in my setup with the most modern and high-end octal-port PCIe cards (TE820).

Progress on the ITU-T V5 access network front

Harald Welte — Thu, 08 Sep 2022 22:00:00 GMT

Almost one year after my post regarding first steps towards a V5 implementation, some friends and I were finally able to visit Wobcom, a small German city carrier and pick up a lot of decommissioned POTS/ISDN/PDH/SDH equipment, primarily V5 access networks.

This means that a number of retronetworking enthusiasts now have a chance to play with Siemens Fastlink, Nokia EKSOS and DeTeWe ALIAN access networks/multiplexers.

My primary interest is in Nokia EKSOS, which looks like an rather easy, low-complexity target. As one of the first steps, I took PCB photographs of the various modules/cards in the shelf, take note of the main chip designations and started to search for the related data sheets.

The results can be found in the Osmocom retronetworking wiki, with https://osmocom.org/projects/retronetworking/wiki/Nokia_EKSOS being the main entry page, and sub-pages about

In short: Unsurprisingly, a lot of Infineon analog and digital ICs for the POTS and ISDN ports, as well as a number of Motorola M68k based QUICC32 microprocessors and several unknown ASICs.

So with V5 hardware at my disposal, I've slowly re-started my efforts to implement the LE (local exchange) side of the V5 protocol stack, with the goal of eventually being able to interface those V5 AN with the Osmocom Community TDM over IP network. Once that is in place, we should also be able to offer real ISDN Uk0 (BRI) and POTS lines at retrocomputing events or hacker camps in the coming years.

Retronetworking at VCFB 2022

Harald Welte — Thu, 08 Sep 2022 22:00:00 GMT

I'm happy to announce active participation at the Vintage Computing Festival Berlin 2022 in two ways:

Running a retronetworking exhibit on Modem and ISDN dial-up
Giving a talk on the Osmocom Community TDMoIP netwokr

The exhibit will be similar to the exhibit at the retrocomputing village of the last CCC congress (36C3): A digital telephony network with ISDN BRI and POTS lines providing services to a number of laptops with Modems and ISDN terminal adapters.

We plan to demo the following things: * analog modem and ISDN dial-up into BBSs ** text / ANSI interfaces via Telix, Telemate, Terminate ** RIPterm graphical interfaces * analog modem and ISDN dial-up IP/internet * ISDN video telephony

The client computers will be contemporary 486/Pentium machines wit DOS, Windows 3.11 and OS/2.

OsmoDevCall: "Advanced SIM card topics: GlobalPlatform SCP02, OTA, ARA-M, ISIM"

Harald Welte — Thu, 24 Feb 2022 23:00:00 GMT

I've presented on Advanced SIM card topics: GlobalPlatform SCP02, OTA, ARA-M, ISIM as part of the OsmoDevCall talk series on Osmocom and related technology.

You can find the video recording at https://media.ccc.de/v/osmodevcall-20220225-laforge-advanced-sim-topics

OsmoDevCall: "Control/User Plane Separation (CUPS) and PFCP"

Harald Welte — Mon, 27 Dec 2021 23:00:00 GMT

I've presented on Control/User Plane Separation (CUPS) and PFCP as part of the OsmoDevCall talk series on Osmocom and related technology.

You can find the video recording at https://media.ccc.de/v/osmodevcall-20211125-laforge-cups-pfcp

OsmoDevCall: "Retronetworking: V5 interfaces in ISDN/PSTN"

Harald Welte — Mon, 27 Dec 2021 23:00:00 GMT

I've presented on Retronetworking: V5 interfaces in ISDN/PSTN as part of the OsmoDevCall talk series on Osmocom and related technology.

You can find the video recording at https://media.ccc.de/v/osmodevcall-20211228-laforge-retro-isdn-v5

OsmoDevCall: "E1, TDM, PDH, SDH, Basics"

Harald Welte — Thu, 11 Nov 2021 23:00:00 GMT

I've presented on E1, TDM, PDH, SDH, Basics as part of the OsmoDevCall talk series on Osmocom and related technology.

You can find the video recording at https://media.ccc.de/v/osmodevcall-20211112-laforge-tdm

OsmoDevCall: "SIM card profile creation, personalization, production"

Harald Welte — Thu, 11 Nov 2021 23:00:00 GMT

I've presented on SIM card profile creation, personalization, production as part of the OsmoDevCall talk series on Osmocom and related technology.

You can find the video recording at https://media.ccc.de/v/osmodevcall-20211022-laforge-sim

First steps towards an ITU-T V5.1 / V5.2 implementation

Harald Welte — Sun, 10 Oct 2021 22:00:00 GMT

As some of you may know, I've been starting to collect "vintage" telecommunications equipment starting from analog modems to ISDN adapters, but also PBXs and even SDH equipment. The goal is to keep this equipment (and related software) alive for demonstration and practical exploration.

Some [incomplete] information can be found at https://osmocom.org/projects/retro-bbs/wiki/

Working with PBXs to simulate the PSTN (ISDN/POTS) network is fine to some extent, but it's of course not the real deal. You only get S0-buses and no actual Uk0 like actual ISDN lines of the late 80ies and 90ies. You have problems with modems not liking the PBX dialtone, etc.

Hence, I've always wanted to get my hand on some more real-world central-office telephone network equipment, and I finally have a source for so-called V5.1/V5.2 access multiplexers. Those are like remote extension boxes for the central office switch (like EWSD or System 12). They aggregate/multiplex a number of analog or ISDN BRI subscriber lines into E1 lines, while not implementing any of the actual call control or ISDN signalling logic. All of that is provided by the actual telephone switch/exchange.

So in order to integrate such access multiplexers in my retronetworking setup, I will have to implement the LE (local exchange) side of the V5.1 and/or V5.2 protocols, as specified in ITU-T G.964 and G.965.

In the limited spare time I have next to my dayjob and various FOSS projects, progress will likely be slow. Nonetheless I started with an implementation now, and I already had a lot of fun learning about more details of those interfaces and their related protocols.

One of the unresolved questions is to what kind of software I would want to integrate once the V5.x part is resolved.

lcr would probably be the most ISDN-native approach, but it is mostly unused and quite EOL.
Asterisk or FreeSWITCH would of course be obvious candidates, but they are all relatively alien to ISDN, and hence not very transparent once you start to do anything but voice calls (e.g. dialup ISDN data calls in various forms).
yate is another potential candidate. It already supports classic SS7 including ISUP, so it would be a good candidate to build an actual ISDN exchange with V5.2 access multiplexers on the customer-facing side (Q.921+Q.931 on it) and SS7/ISUP towards other exchanges.

For now I think yate would be the most promising approach. Time will tell.

The final goal would then be to have a setup [e.g. at a future CCC congress] where we would have SDH add/drop multiplexers in several halls, and V5.x access multiplexers attached to that, connecting analog and ISDN BRI lines from individual participants to a software-defined central exchange. Ideally actually multiple exchanges, so we can show the signaling on the V5.x side, the Q.921/Q.931 side and the SS7/ISUP between the exchanges.

Given that the next CCC congress is not before December 2022, there is a chance to actually implement this before then ;)

OsmoDevCall: "osmo-remsim in practice"

Harald Welte — Thu, 26 Aug 2021 22:00:00 GMT

I've presented on osmo-remsim in practice as part of the OsmoDevCall talk series on Osmocom and related technology.

You can find the video recording at https://media.ccc.de/v/osmodevcall-20210827-laforge-osmo-remsim

OsmoDevCall: "GSM-R and how it differs from GSM"

Harald Welte — Thu, 12 Aug 2021 22:00:00 GMT

I've presented on GSM-R and how it differs from GSM as part of the OsmoDevCall talk series on Osmocom and related technology.

You can find the video recording at https://media.ccc.de/v/osmodevcall-20210813-laforge-gsm-r

OsmoDevCall: "High-Level intro to IMS, VoLTE & VoWiFi"

Harald Welte — Thu, 22 Jul 2021 22:00:00 GMT

I've presented a talk High-Level intro to IMS, VoLTE & VoWiFi as part of the OsmoDevCall talk series on Osmocom and related technology.

You can find the video recording at https://media.ccc.de/v/osmodevcall-20210723-laforge-ims-volte-vowifi

Notfallwarnung im Mobilfunknetz + Cell Broadcast (Teil 2)

Harald Welte — Mon, 19 Jul 2021 22:00:00 GMT

[excuse this German-language post, this is targeted at the current German public discourse]

Ein paar Ergänzungen zu meinem blog-post gestern.

Ich benutzt den generischen Begriff PWS statt SMSCB, weil SMSCB strikt genommen nur im 2G-System existiert, und nur ein Layer ist, der dort für Notfallalarmierung verwendet wird.

Zu Notfallwarn-Apps

Natürlich sind spezielle, nationale Deutsche Katastrophenschutz-Apps auch nützlich! Aber diese sollten allenfalls zusätzlich angeboten werden, nachdem man erstmal die grundlegende Alarmierung konform der relevanten internationalen (und auch EU-)Standards via Cell Broadcast / PWS realisiert. Man sagt ja auch nicht: Nachrichtensendungen braucht man im Radio nicht mehr, weil man die bereits im Fernsehen hat. Man will auf allen verfügbaren Kanälen senden, und zunächst jene mit möglichst universeller Reichweite und klaren technischen Vorteilen benutzen, bevor man dann zusätzlich auch auf anderen Kanälen alarmiert.

Wie sieht PWS für mich als Anwender aus

Hier scheint es größere Missverständnisse zu geben, wie das auf dem Telefon letztlich aussieht. Ist ja auch verständlich, hierzulande sieht man das nie, ausser man ist zufällig in einem Labor/Basttel-Netz z.B. auf einer CCC-Veranstaltung unterwegs, in der das Osmocom-Projekt mal solche Nachrichten versendet hat.

Die PWS (ETWS, CMAS, WEA, KPAS, EU-ALERT, ...) nachrichten werden vom Telefon empfangen, und dann je nach Konfiguration und Priorität behandelt. Für die USA ist im WEA vorgeschrieben, dass Alarme einer bestimmten Prioritatsklasse (z.B. der Presidential Level Alert) immer zwangsweise zur Anzeige gebracht werden und immer mit einem lauten sirenenartigen Alarmton einhergehen. Es ist sogar explizit verboten, dass der Anwender diese Alarme irgendwo ausstellen, stumm schalten o.ä. kann. Insofern spielt es keine Rolle, ob das Telefon gerade Lautlos gestellt ist, oder es nicht gerade unmittelbar bei mir ist.

Bei manchen Geräten werden die Warnungen sogar mittels einer text2speech-Engine laut über den Lautsprecher vorgelesen, nachdem der Alarmton erscheint. Ob das eine regulatorische Anforderung eines der nationalen System ist, weiss ich nicht - ich habe es jedenfalls bereits in manchen Fällen gesehen, als ich mittels Osmocom-Software solche Alarme in privaten Labornetzen versandt habe.

Noch ein paar technische Details

PWS-Nachrichten werden auch dann noch ausgestrahlt, wenn die Zelle ihre Netzanbindung verloren hat. Wenn also z.B. das Glasfaserkabel zum Kernnetz bereits weg ist, aber noch Strom da ist, werden bereits vorher vom CBC (Cell Broadcast Centre) an die Mobilfunkzelle übermittelte Warnungen entsprechend ihrer Gültigkeitsdauer weiter autonom von der Zelle ausgesendet Das ist wieder ein inhärenter technischer Vorteil, der niemals mit einer App erreichbar ist, weil diese erfordert dass das komplette Mobilfunknetz mit allen internen Verbindungen und dem Kernnetz sowie die Internetverbindung vom Netzbetreiber zum Server des App-Anbieters durchgehend funktioniert.
PWS-Nachrichten können zumindest technisch auch von Telefonen empfangen werden, die garnicht im Netz eingebucht sind, oder die keine SIM eingelegt haben. Ob dies in den Standards gefordert wird, und/oder ob dies die jeweilige Telefonsoftware das so umsetzt, weiss ich nicht und müsste man prüfen. Technisch liegt es nahe, ähnlich wie das Absetzen von Notrufen, das ja auch technisch in diesen Fällen möglich ist.

Zu den Kosten

Wenn - wie in der idealen Welt - das Vorhalten von Notfallalarmierung eine Vorgabe bereits zum Zeitpunkt der Lizenzvergabe für Funkfrequenzen gewesen wäre, wäre das alles einfach ganz lautlos von Anfang an immer unterstützt gewesen. Keiner hätte extra Geld investieren müssen, weil diese minimale technische Vorgabe dann ja bereits Teil der Ausschreibungen der Betreiber für den Einkauf ihres Equipments gewesen wäre. Zudem hatten wir ja bereits in der Vergangenheit Cell Brodacast in allen drei Deutschen Netzen, d.h. die Technik war mal [aus ganz andern Gründen] vorhanden aber wurde irgendwann weggespart.

Das jetzt nachträglich einzuführen heisst natürlich, dass es niemand eingeplant hat, und dass jeder beteiligte am Markt sich das vergolden lassen will. Die Hersteller freuen sich in etwa wie "Oh, Ihr wollt jetzt mehr als ihr damals beim Einkauf spezifiziert habt? Schön, dann schreiben wir mal ein Angebot".

Technisch ist das alles ein Klacks. Die komplette Entwicklung aller Bestandteile für PWS in 2G/3G/4G/5G würde ich auf einen niedrigen einmaligen sechsstelligen Betrag schätzen. Und das ist die einmalige Investition in der Entwicklung, welche dann über alle Geräte/Länder/Netze umgebrochen wird. Bei den Milliarden, die in Entwicklung und Anschaffung von Mobilfunktechnik investiert wird, ist das ein Witz.

Die Geräte wie Basisstationen aller relevanten Hersteller unterstützen natürlich von Haus aus PWS. Die bauen für Deutschland ja nicht andere Geräte, als jene, die in UK, NL, RO, US, ... verbaut werden. Der Markt ist international, die gleiche Technik steht überall.

Weil man jetzt zu spät ist, wird das natürlich von allen Seiten ausgenutzt. Jeder Basisstationshersteller wird die Hand aufhalten und sagen, das kostet jetzt pro Zelle X EUR im Jahr zusätzliche Lizenzgebühren. Und die Anbieter der zentralen Komponente CBC werden auch branchenüblich die Hand aufhalten, mit satten jährlichen Lizenzgebühren. Und die Consultants werden auch alle die Hand aufhalten, weil es gibt wieder etwas zu Integrieren, zu testen, ... Das CBC ist keine komplexe Technik. Wenn man das einmalig als Open Source entwickeln lässt, und in allen Netzen einsetzt, bekommt man es quasi zum Nulltarif. Aber das würde ja Voraussetzen, dass man sich wirklich mit der Technik befasst, versteht um welch simple Software es hier geht, und dass man mal andere Wege in der Beschaffung geht, als nur mal eben bei seinen existierenden 3 Lieferanten anzurufen, die sich dann eine goldene Nase verdienen wollen.

In der öffentlichen Diskussion wird von 20-40 Millionen EUR gesprochen. Das sind überzogene Forderungen der Marktteilnehmer, nichts sonst. Aber selbst wenn man der Meinung ist, dass man lieber das Geld zum Fenster hinauswerfen will, statt Open Source Alternativen zu [ver]suchen, dann ist auch diese Größenordnung etwas, dass im Vergleich zu den sonstigen Anschaffungs- und Betriebskosten eines Mobilfunknetzes verschwindend gering ist. Ganz zu schweigen von den Folgekosten im Bereich Bergung/Rettung, Personenschäden, etc. die sich dadurch mittelfristig bei Katastrophen einsparen lassen.

Oder anders betrachtet: Wenn sogar das wirtschaftlich viel schwächere Rumänien sich sowas leisten kann, dann wird es wohl auch die Bundesrepublik Deutschland stemmen können.

Notfallwarnung im Mobilfunknetz + Cell Broadcast

Harald Welte — Sun, 18 Jul 2021 22:00:00 GMT

[excuse this German-language post, this is targeted at the current German public discourse]

In mehrerern Gegenden Deutschlands gab es verheerende Hochwasser, und die Öffentlichkeit diskutiert deshalb mal wieder die gute alte Frage nach dem adäquaten Mittel der Alarmierung der Bevölkerung.

Es ist einfach nur ein gigantisches Trauerspiel, wie sehr die Deutsche Politik und Verwaltung in diesem Punkt inzwischen seit Jahrzehnten sämtliche relevanten Standards verpennt, und dann immer wieder öffentlich durch fachlich falsche und völlig uninformierte Aussagen auffällt.

Das Thema wurde vor dem aktuellen Hochwasser bereits letztes Jahr im Rahmen des sog. WarnTag öffentlich diskutiert. Auch hier von Seiten der öffentlichen Hand ausschliesslich mit falschen Aussagen, wie z.B. dass es bei Cell Broadcast Datenschutzprobleme gibt. Dabei ist Cell Broadcast die einzige Technologie, wo keine Rückmeldung des einzelnen Netzteilnehmers erfolgt, und das Netz nichtmal weiss, wer die Nachricht empfangen hat, und wo dieser Empfang stattgefunden hat. Ganz wie beim UKW-Radio.

Fakt ist, dass alle digitalen Mobilfunkstandards seit GSM/2G, d.h. seit 1991 die Möglichkeit mitbringen, effizient, schnell und datensparsam alle Nutzer (einer bestimmten geographischen Region) mit sogenannten broadcast Nachrichten zu informieren. Diese Technik, in GSM/2G genannt Cell Broacast (oder auch _SMSCB_), unterscheidet sich Grundlegend von allen anderen Kommunikationsformen im Mobilfunknetz, wie Anrufe und herkömmliche SMS (offiziell SMS-PP). Anrufe, SMS und auch mobile Paketdaten (Internet) werden immer für jeden Teilnehmer individuell auf ihm zugewiesenen Funkressourcen übermittelt. Diese Ressourcen sind beschränkt. Es können in keinem Mobilfunknetz der Welt alle Teilnehmer gleichzeitig telefonieren, oder gleichzeitig SMS empfangen.

Stattdessen benutzt Cell Broadcast - wie der Name bereits unmissverständlich klar macht - Einen broadcast, d.h. Rundsendemechanismus. Eine Nachricht wird einmal gesendet, benötigt also nur eine geteilte Ressource auf der Luftschnittstelle, und wird dann von allen Geräten im Empfangsbereich zeitgleich empfangen und dekodiert. Das ist wie UKW-Radio oder klassisches terrestrisches Fernsehen.

Cell Broadcast wurde bereits in den 1990er Jahren von Deutschen Netzbetreibern benutzt. Und zwar nicht für etwas lebensnotwendiges wie die Notfallsignalisierung, sondern für so banale Dinge wie die Liste jener Vorwahlen, zu denen gerade ein vergünstigter "wandernder Ortstarif" Besteht. Ja, sowas gab es mal bei Vodafone. Oder bei O2 wurden über lange Zeit (aus unbekannten Gründen) die GPS-Koordinaten der jeweiligen Basisstation als Cell Broadcast versendet.

In der folgenden (nun fast abgeschalteten) Mobilfunkgeneration 3G wurde Cell Broadcast leicht umbenannt als Service Area Broadcast beibehalten. Schliesslich gibt es ja Länder mit - anders als in Deutschland - funktionierender und kompetenter Regulierung des Telekommunikationsmarktes, und die langjährig bestehenden gesetzlichen Anforderungen solcher Länder zwingen die Netzbetreiber und auch die Ausrüster der Neztbetreiber, neue Mobilfunkstandards so zu entwickeln, dass die gesetzlichen Vorgaben bzgl. der Alarmierung der Bevölkerung im Notfall funktioniert.

Im Rahmen dieser Standardisierung haben eine Reihe von Ländern innerhalb der 3GPP-Standardisierung (zuständig für 2G, 3G, 4G, 5G) sogenannte Public Warning Systems (PWS) standardisiert. Zu diesen gehören z.B. das Japanische ETWAS (Earthquake and Tsunami Warning System), das Koreanische KPAS (Korean Public Alerting System), das US-Amerikanische WEA (Wireless Emergency Alerts, früher bekannt als CMAS) und auch das EU-ALERT mit den nationalen Implementationen NL-ALERT (Niederlande) und UK-ALERT (Großbritannien) sowie RO-ALERT (Rumänien).

Die zahlreichen Studien und Untersuchungen, die zur Gestaltung obiger Systeme und der internationalen Standards im Mobilfunk geführt haben, weisen auch nochmal nach, was sowieso vorher jedem Techniker offensichtlich erscheint: Eine schelle Alarmierung aller Teilnehmer (einer Region) kann nur über einen Broadcast-Mechanismus erfolgen. In Japan war die Zielvorgabe, die Alarmierung in Erdbebenfällen innerhalb von weniger als 4 Sekunden an die gesamte betroffene Bevölkerung zu übertragen. Und das ist mit PWS möglich!

Die relevanten PWS-Standards in 2G/3G/4G/5G bieten jede Menge nützliche Funktionen:

Benachrichtigung in bestimmten geographischen Regionen
Interoperable Schnittstellen, so dass Netzwerkelemente unterschiedlicher Hersteller miteinander kommunizieren
Konfigurierbare Benachrichtigungstexte, nicht nur in der primären Landessprache, sondern auch in mehreren anderen Sprachen, die dann automatisch je nach Spracheinstellung des Telefons wiedergegeben werden
Unterschiedliche Schweregrade von Alarmierungen
Übermittlung nicht nur im Broadcast, sondern auch im Unicast an jeden Teilnehmer, der gerade in einem Telefongespräch ist, und dessen Telefon gerade währenddessen aus technischen Gründen den Broadcast nicht empfangen würde
Unterschied zwischen Wiederholung einer Übertragung ohne Änderung des Inhalts und einer übertragung mit geändertem Inhalt

Es gibt also seit vielen Jahren internationale Standards, wie sämtliche heute eingesetzten Mobilfunktechniken zur schnellen, effizienten und datensparsamen Alarmierung der Bevölkerung eingesetzt werden können.

Es gibt zahlreiche Länder, die diese Systeme seit langem einsetzen. Das US-Amerikanische WEA wurde nach eigenen Angaben seit 2012 bereits mehr als 61.000 Mal benutzt, um Menschen vor Unwetter oder anderen Katastrophen zu warnen.

Sogar innerhalb der EU hat man das EU-ALERT System spezifiziert, welches weitgehend mit dem amerikanischen WEA identisch ist, und auf die gleichen Techniken aufbaut.

Und dann gibt es Länder wie Deutschland, die es seit genauso vielen Jahren vermissen lassen, durch Gesetze oder Vorschriften

die Netzbetreiber zum Betrieb dieser Broadcast-Technologien in ihrem Netz verpflichtet
die Netzbetreiber zur Bereitstellung von standardisierten Schnittstellen gegenüber den Behörden wie Zivilschutz / Katastrophenschutz zu verpflichten, so das diese selbständig über alle Netzbetreiber Warnungen versenden können
die Gerätehersteller z.B. über Vorschriften des FTEG (Gesetz über Funkanlagen und Telekommunikationsendeinrichtungen) zu Verpflichten, die PWS-Nachrichten anzuzeigen

In den USA, dem vermeintlich viel mehr dem Freien Markt und dem Kapitalismus anhängenden System ist all dies der Regulierungsbehörde FCC möglich. In Deutschland mit seiner sozialen Marktwirtschaft ist es anscheinend unmöglich, den Markt entsprechend zu regulieren. Eine solche Regulierung schafft man in Deutschland nur für wirklich wichtige Themen wie zur Durchsetzung der Bereitstellung von Schnittstellen für die Telekommunikationsüberwachung. Bei so irrelevanten Themen wie dem Katastrophenschutz und der Alarmierung der Bevölkerung braucht man den Markt nicht zu regulieren. Wenn die Netzbetreiber kein PWS anbieten wollen, dann ist das einfach so Gottgegeben, und man kann da ja nichts machen.

Falls jemand sich SMSCB und PWS technisch näher ansehen will: In 2019 haben wir im Osmocom-Projekt eine Open Source Implementation des kompletten Systems von BTS über BSC bis zum CBC, sowie der dazwischen befindlichen Protokolle wie CBSP vorgenommen. Dies wurde freundlicherweise durch den Prototype Fund mit EUR 35k finanziert. Ja, so günstig kann man die nötige Technik zumindest für eine einzelne Mobilfunkgeneration entwickeln...

Man kann also in einem selbst betriebenen Labor-Mobilfunknetz, welches auf Open Source Software basiert mehr in Punkt standardkonformer Notfallalarmierung, als die Deutsche Politik, Verwaltung und Netzbetreiber zusammen hinbekommen.

Wir haben in Deutschland Leute, die diese Standards in und auswendig kennen, sogar daran mitgearbeitet haben. Wir haben Entwickler, die diese Standards implementiert haben. Aber wir schaffen es nicht, das auch mal selbst praktisch zu benutzen - das überlassen wir lieber den anderen Ländern. Wir lassen lieber zuerst die ganze Katastrophenalarmierung mittels Sirenen vergammeln, machen den Netzbetreibern keine Vorgaben, entwicklen komische Apps, die Anwender extra installieren müssen, die prinzipbedingt nicht skalieren und beim Test (WarnTag) nicht funktionieren.

Was für eine Glanzleistung für den hochentwickelten Techhologie-Standort Deutschland.

OsmoDevCall: "SS7 and SIGTRAN in 2G/3G networks"

Harald Welte — Sun, 23 May 2021 22:00:00 GMT

I've presented a talk SS7 and SIGTRAN in 2G/3G networks as part of the OsmoDevCall talk series on Osmocom and related technology.

You can find the video recording at https://media.ccc.de/v/osmodevcall-20210514-laforge-ss7-sigtran

OsmoDevCall: "pySim-shell - next generation SIM configuration tool"

Harald Welte — Thu, 08 Apr 2021 22:00:00 GMT

I've presented a talk SS7 and SIGTRAN in 2G/3G networks as part of the OsmoDevCall talk series on Osmocom and related technology.

You can find the video recording at https://media.ccc.de/v/osmodevcall-20210409-laforge-pysim-shell

OsmoDevCall: "Osmocom 2020 review"

Harald Welte — Thu, 11 Mar 2021 23:00:00 GMT

I've presented a talk Osmocom 2020 review as part of the OsmoDevCall talk series on Osmocom and related technology.

You can find the video recording at https://media.ccc.de/v/osmodevcall-20210312-laforge-overview-of-2020

36C3 Talks on SIM card technology / Mitel DECT

Harald Welte — Sat, 04 Jan 2020 23:00:00 GMT

At 36C3 in December 2019 I had the pleasure of presenting: One full talk about SIM card technology from A to Z and another talk where I presented together with eventphone team members about Security issues in the Mitel SIP-DECT system.

The SIM card talk was surprisingly successful, both in terms of a full audience on-site, as well as in terms of the number of viewers of the recordings on media.ccc.de. SIM cards are a rather niche topic in the wider IT industry, and my talk was not covering any vulnerabilities or the like. Also, there was nothing novel in the talk: SIM cards have been around for decades, and not much has changed (except maybe eSIM and TLS) in recent years.

In any case, I'm of course happy that it was well received. So far I've received lots of positive feedback.

As I'm working [more than] full time in cellular technology for almost 15 years now, it's sometimes hard to imagine what kind of topics people might be interested in. If you have some kind of suggestion on what kind of subject within my area of expertise you'd like me to talk about, please don't hesitate to reach out.

The Mitel DECT talk also went quite well. I covered about 10 minutes of technical details regarding the reverse engineering of the firmware and the communication protocols of the device. Thanks again to Dieter Spaar for helping with that. He is and remains the best reverse engineer I have met, and it's always a privilege to collaborate on any project. It was of course also nice to see what kind of useful (and/or fun) things the eventphone team have built on top of the knowledge that was gained by protocol-level reverse engineering.

If you want to know more low-level technical detail than the 36C3 talk, I recommend my earlier talk at the OsmoDevCon 2019 about Aastra/Mitel DET base station dissection.

If only I had more time, I would love to work on improving the lack of Free / Open Source Software realted to the DECT protocol family. There's the abandoned deDECTed.org, and the equally abandoned dect.osmocom.org project. The former only deals with the loewst levels of DECT (PHY/MAC). The latter is to a large extent implemented as part of an ancient version of the Linux kernel (I would say this should all run in userspace, like we run all of GSM/UMTS/LTE in userspace today).

If anyone wants to help out, I still think working on the DECT DLC and NWK dissectors for wireshark is the best way to start. It will create a tool that's important for anyone working with the DECT protocols, and it will be more or less a requirement for development and debugging should anyone ever go further in terms of implementing those protocols on either the PP or FP side. You can find my humble beginnings of the related dissectors in the laforge/dect branch of osmocom.org/wireshark.git.

Retronetworking / BBS-Revival setup at #36C3

Harald Welte — Sat, 04 Jan 2020 23:00:00 GMT

After many years of being involved in various projects at the annual Chaos Communication Congress (starting from the audio/vidoe recording team at 15C3), I've finally also departed the GSM team, i.e. the people who operate (Osmocom based) cellular networks at CCC events.

The CCC Camp in August 2019 was slightly different: Instead of helping an Osmocom based 2G/3G network, I decided to put up a nextepc-based LTE network and make that use the 2G/3G HLR (osmo-hlr) via a newly-written DIAMETER-to-GSUP proxy. After lots of hacking on that proxy and fixing various bugs in nextepc (see my laforge/cccamp2019 branch here) this was working rather fine.

For 36C3 in December 2019 I had something different in mind: It was supposed to be the first actual demo of the retronetworking / bbs-revival setup I've been working on during past months. This setup in turn is sort-of a continuation of my talk at 34C3 two years ago: BBSs and early Intenet access in the 1990ies.

Rather than just talking about it, I wanted to be able to show people the real thing: Actual client PCs running (mainly) DOS, dialling over analog modems and phone lines as well as ISDN-TAs and ISDN lines into BBSs, together with early Interent access using SLIP and PPP over the same dial-up lines.

The actual setup can be seen at the Dialup Network In A Box wiki page, together with the 36C3 specific wiki page.

What took most of the time was - interestingly - mainly two topics:

A 1U rack-mount system with four E1 ports. I had lots of old Sangoma Quad-E1 cards in PCI form-factor available, but wanted to use a PC with a more modern/faster CPU than those old first-generation Atom boxes that still had actual PCI slots. Those new mainboards don't have PCI but PCIe. There are plenty of PCIe to PCI bridges and associated products on the market, which worked fine with virtually any PCI card I could find, but not with the Sangoma AFT PCI cards I wanted to use. Seconds to minutes after boot, the PCI-PCIe bridges would always forget their secondary bus number. I suspected excessive power consumption or glitches, but couldn't find anything wrong when looking at the power rails with a scope. Adding additional capacitors on every rail also didn't change it. The !RESET line is also clean. It remains a mystery. I then finally decided to but a new (expensive) DAHDI 4-port E1 PCIe card to move ahead. What a waste of money if you have tons of other E1 cards around.
Various trouble with FreeSWITCH. All I wanted/needed was some simple emulation of a PSTN/ISDN switch, operating in NT mode towards both the Livingston Portmaster 3 RAS and the Auerswald PBX. I would have used lcr, but it supports neither DAHDI nor Sangoma, but only mISDN - and there are no mISDN cards with four E1 ports :( So I decided to go for FreeSWITCH, knowing it has had a long history of ISDN/PRI/E1 support. However, it was a big disappointment. First, there were some segfaults due to a classic pointer deref before NULL-check. Next, libpri and FreeSWITCH have a different idea how channel (timeslot) numbers are structured, rendering any call attempt to fail. Finally, FreeSWITCH decided to blindly overwrite any bearer capabilities IE with 'speech', even if an ISDN dialup call (unrestricted digital information) was being handled. The FreeSWITCH documentation contains tons of references on channel input/output variables related to that - but it turns out their libpri integration doesn't set any of those, nor use any of them on the outbound side.

Anyway, after a lot more time than expected the setup was operational, and we could establish modem calls as well as ISDN dialup calls between the clients and the Portmaster3. The PM3 in turn then was configured to forward the dialup sessions via telnet to a variety of BBSs around the internet. Some exist still (or again) on the public internet. Some others were explicitly (re)created by 36C3 participants for this very BBS-Revival setup.

My personal favorite was finding ACiD Underworld 2.0, one of the few BBSs out there today who support RIPscrip, a protocol used to render vector graphics, text and even mouse-clickable UI via modem connection to a DOS/EGA client program called RIPterm. So we had one RIPterm installation on Novell DOS7 that was just used for dialling into ACiD Underworld 2.0.

Among other things we also tested interoperability between the 1980ies CCC DIY accoustic coupler "Datenklo" and the Portmaster, and confirmed that Windows 2000 could establish multilink-PPP not only over two B-channels (128 kbps) but also over 3 B-Channels (192).

Running this setup for four days meant 36C3 was a quite different experience than many previous CCC congresses:

I was less stressed as I wasn't involved in operating a service that many people would want to use (GSM).
I got engaged with many more people with whom I would normally not have entered a conversation, as they were watching the exhibits/demos and we got to chat about the technology involved and the 'good old days'.

So all in all, despite the last minute FreeSWITCH-patching, it was a much more relaxing and rewarding experience for me.

Special thanks to

Sylvain "tnt" Munaut for spending a lot of time with me at the retronetworking assembly. The fact that I had an E1 interface around was a good way for him to continue development on his ICE40 based bi-directional E1 wiretap. He also helped with setup and teardown.
miaoski and evanslify for reviving two of their old BBSs from Taiwan so we could use them at this event

The retronetworking setup is intended to operate at many other future events, whether CCC related, Vintage Computing or otherwise. It's relatively small and portable.

I'm very much looking forward to the next incarnations. Until then, I will hopefully have more software configured and operational, including a variety of local BBSs (running in VMs/containers), together with the respective networking (FTN, ZConnect, ...) and point software like CrossPoint.

If you are interested in helping out with this project: I'm very much looking for help. It doesn't matter if you're old and have had BBS experience back in the day, or if you're a younger person who wants to learn about communications history. Any help is appreciated. Please reach out to the bbs-revival@lists.osmocom.org mailing list, or directly to me via e-mail.

Software Freedom Podcast #3 about Free Software mobile phone communication

Harald Welte — Thu, 19 Dec 2019 23:00:00 GMT

Recently I had the pleasure of being part of the 3rd incarnation of a new podcast series by the Free Software Foundation Europe: The Software Freedom Podcast.

In episode 3, Matthias and Bonnie of the FSFE are interviewing me about various high-level topics related to [the lack of] Free Software in cellular telephony, as well as some of the projects that I was involved in (Openmoko, Osmocom).

We've also touched the current mainstream / political debate about Huawei and 5G networks, where my position can only be summarized as: It doesn't matter much in which country the related proprietary software is being developed. What we need is Free / Open Source software that can be publicly audited, and we need a method by which the operator can ensure that a given version of that FOSS software is actually executing on his equipment.

Thanks to the FSFE for covering such underdeveloped areas of Free Software, and to use their podcast to distribute related information and ideas.

Sometimes software development is a struggle

Harald Welte — Sat, 28 Sep 2019 22:00:00 GMT

I'm currently working on the firmware for a new project, an 8-slot smart card reader. I will share more about the architecture and design ideas behind this project soon, but today I'll simply write about how hard it sometimes is to actually get software development done. Seemingly trivial things suddenly take ages. I guess everyone writing code knows this, but today I felt like I had to share this story.

Chapter 1 - Introduction

As I'm quite convinced of test-driven development these days, I don't want to simply write firmware code that can only execute in the target, but I'm actually working on a USB CCID (USb Class for Smart Card readers) stack which is hardware-independent, and which can also run entirely in userspace on a Linux device with USB gadget (device) controller. This way it's much easier to instrument, trace, introspect and test the code base, and tests with actual target board hardware are limited to those functions provided by the board.

So the current architecture for development of the CCID implementation looks like this:

Implement the USB CCID device using FunctionFS (I did this some months ago, and in fact developing this was a similarly much more time consuming task than expected, maybe I find time to expand on that)
Attach this USB gadget to a virtual USB bus + host controller using the Linux kernel dummy_hcd module
Talk to a dumb phoenix style serial SIM card reader attached to a USB UART, which is connected to an actual SIM card (or any smart card, for that matter)

By using a "stupid" UART based smart card reader, I am very close to the target environment on a Cortex-M microcntroller, where I also have to talk to a UART and hence implement all the beauty of ISO 7816-3. Hence, the test / mock / development environment is as close as possible to the target environment.

So I implemented the various bits and pieces and ended up at a point where I wanted to test. And I'm not getting any response from the UART / SIM card at all. I check all my code, add lots of debugging, play around with various RTS / DTR / ... handshake settings (which sometimes control power) - no avail.

In the end, after many hours of trial + error I actually inserted a different SIM card and finally, I got an ATR from the card. In more than 20 years of working with smart cards and SIM cards, this is the first time I've actually seen a SIM card die in front of me, with no response whatsoever from the card.

Chapter 2 - Linux is broken

Anyway, the next step was to get the T=0 protocol of ISO 7816-3 going. Since there is only one I/O line between SIM card and reader for both directions, the protocol is a half-duplex protocol. This is unlike "normal" RS232-style UART communication, where you have a separate Rx and Tx line.

On the hardware side, this is most often implemented by simply connecting both the Rx and Tx line of the UART to the SIM I/O pin. This in turn means that you're always getting an echo back for every byte you write.

One could discard such bytes, but then I'm targeting a microcontroller, which should be running eight cards in parallel, at preferably baud-rates up to ~1 megabit speeds, so having to read and discard all those bytes seems like a big waste of resources.

The obvious solution around that is to disable the receiver inside the UART before you start transmitting, and re-enable it after you're done transmitting. This is typically done rather easily, as most UART registers in hardware provide some way to selectively enable transmitter and/or receiver independently.

But since I'm working in Linux userspace in my development environment: How do I approximate this kind of behavior? At least the older readers of this blog will remember something called the CREAD flag of termios. Clearing that flag will disable the receiver. Back in the 1990ies, I did tons of work with serial ports, and I remembered there was such a flag.

So I implement my userspace UART backend and somehow it simply doesn't want to work. Again of course I assume I must be doing something wrong. I'm using strace, I'm single-stepping through code - no avail.

In the end, it turns out that I've just found a bug in the Linux kernel, one that appears to be there at least ever since the git history of linux-2.6.git started. Almost all USB serial device drivers do not implement CREAD, and there is no sotware fall-back implemented in the core serial (or usb-serial) handling that would discard any received bytes inside the kernel if CREAD is cleared. Interestingly, the non-USB serial drivers for classic UARTs attached to local bus, PCI, ... seem to support it.

The problem would be half as much of a problem if the syscall to clear CREAD would actually fail with an error. But no, it simply returns success but bytes continue to be received from the UART/tty :/

So that's the second big surprise of this weekend...

Chapter 3 - Again a broken card?

So I settle for implementing the 'receive as many characters as you wrote' work-around. Once that is done, I continue to test the code. And what happens? Somehow my state machine (implemented using osmo-fsm, of course) for reading the ATR (code found here) somehow never wants to complete. The last byte of the ATR always is missing. How can that be?

Well, guess what, the second SIM card I used is sending a broken, non-spec compliant ATR where the header indicates 9 historical bytes are present, but then in reality only 8 bytes are sent by the card.

Of course every reader has a timeout at that point, but that timeout was not yet implemented in my code, and I also wasn't expecting to hit that timeout.

So after using yet another SIM card (now a sysmoUSIM-SJS1, not sure why I didn't even start with that one), it suddenly works.

After a weekend of detours, each of which I would not have assumed at all before, I finally have code that can obtain the ATR and exchange T=0 TPDUs with cards. Of course I could have had that very easily if I wanted (we do have code in pySim for this, e.g.) but not in the architecture that is as close as it gets to the firmware environment of the microcontroller of my target board.

Fernvale Kits - Lack of Interest - Discount

Harald Welte — Fri, 28 Sep 2018 22:00:00 GMT

Back in December 2014 at 31C3, bunnie and xobs presented about their exciting Fernvale project, how they reverse engineered parts of the MT6260 ARM SoC, which also happens to contain a Mediatek GSM baseband.

Thousands (at least hundreds) of people have seen that talk live. To date, 2506 people (or AIs?) have watched the recordings on youtube, 4859 more people on media.ccc.de.

Given that Fernvale was the closest you could get to having a hackable baseband processor / phone chip, I expected at least as much interest into this project as we received four years earlier with OsmocomBB.

As a result, in early 2015, sysmocom decided to order 50 units of Fernvale DVT2 evaluation kits from bunnie, and to offer them in the sysmocom webshop to ensure the wider community would be able to get the boards they need for research into widely available, inexpensive 2G baseband chips.

This decision was made purely for the perceived benefit of the community: Make an exciting project available for anyone. With that kind of complexity and component density, it's unlikely anyone would ever solder a board themselves. So somebody has to build some and make it available. The mark-up sysmocom put on top of bunnie's manufacturing cost was super minimal, only covering customs/import/shipping fees to Germany, as well as minimal overhead for packing/picking and accounting.

Now it's almost four years after bunnie + xobs' presentation, and of those 50 Fernvale boards, we still have 34 (!) units in stock. That means, only 16 people on this planet ever had an interest in playing with what at the time I thought was one of the most exciting pieces of equipment to play with.

So we lost somewhere on the order of close to 3600 EUR in dead inventory, for something that never was supposed to be a business anyway. That sucks, but I still think it was worth it.

In order to minimize the losses, sysmocom has now discounted the boards and reduced the price from EUR 110 to to EUR 58.82 (excluding VAT). I have very limited hope that this will increase the amount of interest in this project, but well, you got to try :)

In case you're thinking "oh, let's wait some more time, until they hand them out for free", let me tell you: If money is the issue that prevents you from playing with a Fernvale, then please contact me with the details about what you'd want to do with it, and we can see about providing them for free or at substantially reduced cost.

In the worst case, it was ~ 3600 EUR we could have invested in implementing more Osmocom software, which is sad. But would I do it again if I saw a very exciting project? Definitely!

The lesson learned here is probably that even a technically very exciting project backed by world-renowned hackers like bunnie doesn't mean that anyone will actually ever do anything with it, unless they get everything handed on a silver plate, i.e. all the software/reversing work is already done for them by others. And that actually makes me much more sad than the loss of those ~ 3600 EUR in sysmocom's balance sheet.

I also feel even more sorry for bunnie + xobs. They've invested time, money and passion into a project that nobody really seemed to want to get involved and/or take further. ("nobody" is meant figuratively. I know there were/are some enthusiasts who did pick up. I'm talking about the big picture). My condolences to bunnie + xobs!

Wireshark dissector for 3GPP CBSP - traces wanted!

Harald Welte — Tue, 18 Sep 2018 22:00:00 GMT

I recently was reading 3GPP TS 48.049, the specification for the CBSP (Cell Broadcast Service Protocol), which is the protocol between the BSC (Base Station Controller) and the CBC (Cell Broadcast Centre). It is how the CBC according to spec is instructing the BSCs to broadcast the various cell broadcast messages to their respective geographic scope.

While OsmoBTS and OsmoBSC do have support for SMSCB on the CBCH, there is no real interface in OsmoBSC yet on how any external application would instruct it tot send cell broadcasts. The only existing interface is a VTY command, which is nice for testing and development, but hardly a scalable solution.

So I was reading up on the specs, discovered CBSP and thought one good way to get familiar with it is to write a wireshark dissector for it. You can find the result at https://code.wireshark.org/review/#/c/29745/

Now my main problem is that as usual there appear to be no open source implementations of this protocol, so I cannot generate any traces myself. More surprising is that it's not even possible to find any real-world CBSP traces out there. So I'm facing a chicken-and-egg problem. I can only test / verify my wireshark dissector if I find some traces.

So if you happen to have done any work on cell broadcast in 2G network and have a CBSP trace around (or can generate one): Please send it to me, thanks!

Alternatively, you can of course also use the patch linked above, build your own wireshark from scratch, test it and provide feedback. Thanks in either case!

Still alive, just not blogging

Harald Welte — Sun, 26 Aug 2018 22:00:00 GMT

It's been months without any update to this blog, and I feel sad about that. Nothing particular has happened to me, everything is proceeding as usual.

At the Osmocom project we've been making great progress on a variety of fronts, including

3GPP LCLS (Local Call, Local Switch)
Inter-BSC hand-over in osmo-bsc
load Based hand-over in osmo-bsc
reintroducing SCCPlite compatibility to the new BSC code in osmo-bsc / libosmo-sigtran
finishing the first release of the SIMtrace2 firmware
extending test coverage on all fronts, particularly in our TTCN-3 test suites
tons of fixes to the osmo-bts measurement processing / reporting
higher precision time of arrival reporting in osmo-bts
migrating osmocom.org services to new, faster servers

At sysmocom, next to the Osmocom topics above, we've

made the sysmoQMOD remote SIM firmware much more robust and reliable
after months of delays, finally SIMtrace2 hardware kits are available again
created autoamtic testing of pySim-prog and sysmo-usim-util
extended our osmo-gsm-tester based automatic testing setup to include multi-TRX nanoBTS setups

In terms of other topic,

my wife and I have been to a three week motorbike tour all over the Alps in July
I've done tons of servicing (brake piston fittings, brake tubes, fuel line, fixing rust/paint, replacing clutch cable, choke cable, transmission chain, replacing several rusted/worn-out needle bearings, and much more) on my 22year old BMW F650ST to prepare it for many more yers to come. As some type-specific spare parts (mostly plastic parts) are becoming rarer, it was best to take care of replacements sooner than later
some servicing/repairs to my 19 year old Audi A4 car (which passed German mandatory inspection without any deficiency at the first attempt!)
some servicing of my Yamaha FZ6
repaired my Fairphone 2 by swapping the microphone module (mike was mute)
I've re-vamped a lot of the physical/hardware infrastructure for gnumonks.org and other sites I run, which was triggered by having to move racks

Re-launching openmoko USB Product ID and Ethernet OUI registry

Harald Welte — Fri, 08 Jun 2018 22:00:00 GMT

Some time after Openmoko went out of business, they made available their USB Vendor IDs and IEEE OUI (Ethernet MAC address prefix) available to Open Source Hardware / FOSS projects.

After maintaining that for some years myself, I was unable to find time to continue the work and I had handed it over some time ago to two volunteers. However, as things go, those volunteers also stopped to respond to PID / OUI requests, and we're now launching the third attempt of continuing this service.

As the openmoko.org wiki will soon be moved into an archive of static web pages only, we're also moving the list of allocated PID and OUIs into a git repository.

Since git.openmoko.org is also about to be decommissioned, the repository is now at https://github.com/openmoko/openmoko-usb-oui, next to all the archived openmoko.org repository mirrors.

This also means that in addition to sending an e-mail application for getting an allocation in those ranges, you can now send a pull-request via github.

Thanks to cuvoodoo for volunteering to maintain the Openmoko USB PID and IEEE OUI allocations from now on!

openmoko.org archive down due to datacenter issues

Harald Welte — Thu, 24 May 2018 22:00:00 GMT

Unfortunately, since about 11:30 am CEST on MAy 24, openmoko.org is down due to some power outage related issues at Hetzner, the hosting company at which openmoko.org has been hosting for more than a decade now.

The problem seems to have caused quite a lot of fall-out tom many servers (Hetzner is hosting some 200k machines, not sure how many affected, though), and Hetzner is anything but verbose when it comes to actually explaining what the issue is.

All they have published is https://www.hetzner-status.de/en.html#8842 - which is rather tight lipped about some power grid issues. But then, what do you have UPSs for if not for "a strong voltage reduction in the local power grid"?

The openmoko.org archive machine is running in Hetzner DC10, by the way. This is where they've had the largest number of tickets.

In any case, we'll have to wait for them to resolve their tickets. They appear to be working day and night on that.

I have a number of machines hosted at Hetzner, and I'm actually rather happy that none of the more important systems were affected that long. Some machines simply lost their uplink connectivity for some minutes, while some others were rebooted (power outage). The openmoko.org archive is the only machine that didn't automatically boot after the outage, maybe the power supply needs replacement.

In any case, I hope the service will be back up again soon.

btw: Guess who's been paying for hosting costs ever since Openmoko, Inc. has shut down? Yes, yours truly. It was OK for something like 9 years, but I want to recursively pull the dynamic content through some cache, which can then be made permanent. The resulting static archive can then be moved to some VM somewhere, without requiring a dedicated root server. That should reduce the costs down to almost nothing.

OsmoCon 2018 CfP closes on 2018-05-30

Harald Welte — Thu, 24 May 2018 22:00:00 GMT

One of the difficulties with OsmoCon2017 last year was that almost nobody submitted talks / discussions within the deadline, early enough to allow for proper planning.

This lad to the situation where the sysmocom team had to come up with a schedule/agenda on their own. Later on much after the CfP deadline,people then squeezed in talks, making the overall schedule too full.

It is up to you to avoid this situation again in 2018 at OsmoCon2018 by submitting your talk RIGHT NOW. We will be very strict regarding late submissions. So if you would like to shape the Agenda of OsmoCon 2018, this is your chance. Please use it.

We will have to create a schedule soon, as [almost] nobody will register to a conference unless the schedule is known. If there's not sufficient contribution in terms of CfP response from the wider community, don't complain later that 90% of the talks are from sysmocom team members and only about the Cellular Network Infrastructure topics.

You have been warned. Please make your CfP submission in time at https://pretalx.sysmocom.de/osmocon2018/cfp before the CfP deadline on 2018-05-30 23:59 (Europe/Berlin)

Mailing List hosting for FOSS Projects

Harald Welte — Wed, 23 May 2018 22:00:00 GMT

Recently I've encountered several occasions in which a FOSS project would have been interested in some reliable, independent mailing list hosting for their project communication.

I was surprised how difficult it was to find anyone running such a service.

From the user / FOSS project point of view, the criteria that I would have are:

operated by some respected entity that is unlikely to turn hostile, discontinue the service or go out of business altogether
free of any type of advertisements (we all know how annoying those are)
cares about privacy, i.e. doesn't sell the subscriber lists or non-public archives
use FOSS to run the service itself, such as GNU mailman, listserv, ezmlm, ...
an easy path to migrate away to another service (or self-hosting) as they grow or their requirements change. A simple mail forward to that new address for the related addresses is typically sufficient for that

If you think mailing lists serve no purpose these days anyways, and everyone is on github: Please have a look at the many thousands of FOSS project mailing lists out there still in use. Not everyone wants to introduce a dependency to the whim of a proprietary software-as-a-service provider.

I never had this problem as I always hosted my own mailman instance on lists.gnumonks.org anyway, and all the entities that I've been involved in (whether non-profit or businesses) had their own mailing list hosts. From franken.de in the 1990ies to netfilter.org, openmoko.org and now osmocom.org, we all pride oursevles in self-hosting.

But then there are plenty of smaller projects that neither have the skills nor the funding available. So they go to yahoo groups or some other service that will then hold them hostage without a way to switch their list archives from private to public, without downloadable archives or forwarding in the case they want to move away :(

Of course the larger FOSS projects also have their own list servers, starting from vger.kernel.org to Linux distributions like Debian GNU/Linux. But what if your FOSS project is not specifically Linux related?

The sort-of obvious candidates that I found all don't really fit:

https://lists.gnu.org/ is for official GNU projects
https://lists.nongnu.org/ is for projects on Savannah (which is much more than just a mailing list service, many projects don't need or want a "Forge")
https://vger.kernel.org/ is specifically for Linux kernel development
https://lists.freedesktop.org/ is specifically for desktop/UI related projects
https://mail.kde.org/mailman/listinfo/ is hosting lists for KDE related projects
https://mail.gnome.org/mailman/listinfo/ likewise for Gnome projects
http://lists.fsfe.org/ appears to be for FSFE specific/internal lists only, and not a public service
http://lists.spi-inc.org/ likewise is for SPI's own lists only
https://opensource.org/lists also only runs lists for themselves
http://lists.digitalfreedomfoundation.org/lists/listinfo has no public lists
http://lists.jpberlin.de/ hosts tons of mailing lists for any kind of topic, but is a paid service
https://wiki.list.org/COM/Mailman%20hosting%20services lists various other paid services

Now don't get me wrong, I'm of course not expecting that there are commercial entities operating free-of charge list hosting services where you neither pay with money, nor your data, nor by becoming a spam receiver.

But still, in the wider context of the Free Software community, I'm seriously surprised that none of the various non-for-profit / non-commercial foundations or associations are offering a public mailing list hosting service for FOSS projects.

One can of course always ask any from the above list and ask for a mailing list even though it's strictly speaking off-topic to them. But who will do that, if he has to ask uninvited for a favor?

I think there's something missing. I don't have the time to set up a related service, but I would certainly want to contribute in terms of funding in case any existing FOSS related legal entity wanted to expand. If you already have a legal entity, abuse contacts, a team of sysadmins, then it's only half the required effort.

OsmoDevCon 2018 retrospective

Harald Welte — Sun, 29 Apr 2018 22:00:00 GMT

One week ago, the annual Osmocom developer meeting (OsmoDevCon 2018) concluded after four long and intense days with old and new friends (schedule can be seen here).

It was already the 7th incarnation of OsmoDevCon, and I have to say that it's really great to see the core Osmocom community come together every year, to share their work and experience with their fellow hackers.

Ever since the beginning we've had the tradition that we look beyond our own projects. In 2012, David Burgess was presenting on OpenBTS. In 2016, Ismael Gomez presented about srsUE + srsLTE, and this year we've had the pleasure of having Sukchan Kim coming all the way from Korea to talk to us about his nextepc project (a FOSS implementation of the Evolved Packet Core, the 4G core network).

What has also been a regular "entertainment" part in recent years are the field trip reports to various [former] satellite/SIGINT/... sites by Dimitri Stolnikov.

All in all, the event has become at least as much about the people than about technology. It's a community of like-minded people that to some part are still working on joint projects, but often work independently and scratch their own itch - whether open source mobile comms related or not.

After some criticism last year, the so-called "unstructured" part of OsmoDevCon has received more time again this year, allowing for exchange among the participants irrespective of any formal / scheduled talk or discussion topic.

In 2018, with the help of c3voc, for the first time ever, we've recorded most of the presentations on video. The results are still in the process of being cut, but are starting to appear at https://media.ccc.de/c/osmodevcon2018.

If you want to join a future OsmoDevCon in person: Make sure you start contributing to any of the many Osmocom member projects now to become eligible. We need you!

Now the sad part is that it will take one entire year until we'll reconvene. May the Osmocom Developer community live long and prosper. I want to meet you guys for many more years at OsmoDevCon!

There is of course the user-oriented OsmoCon 2018 in October, but that's a much larger event with a different audience.

Nevertheless, I'm very much looking forward to that, too.

The OsmoCon 2018 Call for Participation is still running. Please consider submitting talks if you have anything open source mobile communications related to share!

osmo-fl2k - Using USB-VGA dongles as SDR transmitter

Harald Welte — Sun, 22 Apr 2018 22:00:00 GMT

Yesterday, during OsmoDevCon 2018, Steve Markgraf released osmo-fl2k, a new Osmocom member project which enables the use of FL2000 USB-VGA adapters as ultra-low-cost SDR transmitters.

How does it work?

A major part of any VGA card has always been a rather fast DAC which generates the 8-bit analog values for (each) red, green and blue at the pixel clock. Given that fast DACs were very rare/expensive (and still are to some extent), the idea of (ab)using the VGA DAC to transmit radio has been followed by many earlier, mostly proof-of-concept projects, such as Tempest for Eliza in 2001.

However, with osmo-fl2k, for the first time it was possible to completely disable the horizontal and vertical blanking, resulting in a continuous stream of pixels (samples). Furthermore, as the supported devices have no frame buffer memory, the samples are streamed directly from host RAM.

As most USB-VGA adapters appear to have no low-pass filters on their DAC outputs, it is possible to use any of the harmonics to transmit signals at much higher frequencies than normally possible within the baseband of the (max) 157 Mega-Samples per seconds that can be achieved.

osmo-fl2k and rtl-sdr

Steve is the creator of the earlier, complementary rtl-sdr software, which since 2012 transforms USB DVB adapters into general-purpose SDR receivers.

Today, six years later, it is hard to think of where SDR would be without rtl-sdr. Reducing the entry cost of SDR receivers nearly down to zero has done a lot for democratization of SDR technology.

There is hence a big chance that his osmo-fl2k project will attain a similar popularity. Having a SDR transmitter for as little as USD 5 is an amazing proposition.

free riders

Please keep in mind that Steve has done rtl-sdr just for fun, to scratch his own itch and for the "hack value". He chose to share his work with the wider public, in source code, under a free software license. He's a very humble person, he doesn't need to stand in the limelight.

Many other people since have built a business around rtl-sdr. They have grabbed domains with his project name, etc. They are now earning money based on what he has done and shared selflessly, without ever contributing back to the pioneering developers who brought this to all of us in the first place.

So, do we want to bet if history repeats itself? How long will it take for vendors showing up online advertising the USB VGA dongles as "SDR transmitter", possibly even with a surcharge? How long will it take for them to include Steve's software without giving proper attribution? How long until they will violate the GNU GPL by not providing the complete corresponding source code to derivative versions they create?

If you want to thank Steve for his amazing work

reach out to him personally
contribute to his work, e.g.
help to maintain it
package it for distributions
send patches (via osmocom-sdr mailing list)
register an osmocom.org account and update the wiki with more information

And last, but not least, carry on the spirit of "hack value" and democratization of software defined radio.

Thank you, Steve! After rtl-sdr and osmo-fl2k, it's hard to guess what will come next :)

udtrace - Unix domain socket tracing

Harald Welte — Thu, 29 Mar 2018 22:00:00 GMT

When developing applications that exchange data over sockets, every so often you'd like to analyze exactly what kind of data is exchanged over the socket.

For TCP/UDP/SCTP/DCCP or other IP-based sockets, this is rather easy by means of libpcap and tools like tcpdump, tshark or wireshark. However, for unix domain socket, unfortunately no such general capture/tracing infrastructure exists in the Linux kernel.

Interestingly, even after searching for quite a bit I couldn't find any existing tools for this. This is surprising, as unix domain sockets are used by a variety of programs, from sql servers to bind8 ndc all the way to the systemctl tool to manage systemd.

In absence of any kernel support, the two technologies I can think of to implement this is either systemtap or a LD_PRELOAD wrapper.

However, I couldn't find an example for using either of those two to get traces of unix domain soocket communications.

Ok, so I get to write my own. My first idea hence was to implement something based on top of systemtap, the Linux kernel tracing framework. Unfortunately, systemtap was broken in Debian unstable (which I use for decades) at the time, so I went back to the good old LD_PRELOAD shim library / wrapper approach.

The result is called udtrace and can be found at

git clone https://gitea.osmocom.org/laforge/udtrace

or alternatively via its github mirror.

Below is a copy+paste of its README file. Let's hope this tool is useful to other developers, too:

udtrace - Unix Domain socket tracing

This is a LD_PRELOAD wrapper library which can be used to trace the data sent and/or received via unix domain sockets.

Unlike IP based communication that can be captured/traced with pcap programs like tcpdump or wireshark, there is no similar mechanism available for unix domain sockets.

This LD_PRELOAD library intercepts the C library function calls of dynamically linked programs. It will detect all file descriptors representing unix domain sockets and will then print traces of all data sent/received via the socket.

Usage

Simply build libudtrace.so using the make command, and then start your to-be-traced program with

LD_PRELOAD=libudtrace.os

e.g.

LD_PRELOAD=libudtrace.so systemctl status

which will produce output like this:

>>> UDTRACE: Unix Domain Socket Trace initialized (TITAN support DISABLED)
>>> UDTRACE: Adding FD 4
>>> UDTRACE: connect(4, "/run/dbus/system_bus_socket")
4 sendmsg W 00415554482045585445524e414c20
4 sendmsg W 3331333033303330
4 sendmsg W 0d0a4e45474f54494154455f554e49585f46440d0a424547494e0d0a
[...]

Output Format

Currently, udtrace will produc the following output:

At time a FD for a unix domain socket is created:

>>> UDTRACE: Adding FD 8

At time a FD for a unix domain socket is closed:

>>> UDTRACE: Removing FD 8

At time a FD for a unix domain socket is bound or connected:

>>> UDTRACE: connect(9, "/tmp/mncc")

When data is read from the socket:

9 read R 00040000050000004403000008000000680000001c0300002c03000000000000

When data is written to the socket:

9 write W 00040000050000004403000008000000680000001c0300002c03000000000000

Where

9 is the file dsecriptor on which the event happened
read/write is the name of the syscall, could e.g. also be sendmsg / readv / etc.
R|W is Read / Write (from the process point of view)
followed by a hex-dump of the raw data. Only data successfully written (or read) will be printed, not the entire buffer passed to the syscall. The rationale is to only print data that was actually sent to or received from the socket.

TITAN decoder support

Getting hex-dumps is nice and fine, but normally one wants to have a more detailed decode of the data that is being passed on the socket.

For TCP based protocols, there is wireshark. But most protocols on unix domain sockets don't follow inter-operable / public standards, so even if one was to pass the traces into wireshark somehow, there would be no decoder.

In the Osmocom project, we already had some type definitions and decoders for our protocols written in the TTCN-3 programming language, using Eclipse TITAN. In order to build those decoders fro MNCC and PCUIF, please use

make ENABLE_TITAN=1

when building the code.

Please note that this introduces a run-time dependency to libttcn3-dynamic.so, which is (at least on Debian GNU/Linux) not installed in a default library search path, so you will have to use something like:

LD_LIBRARY_PATH=/usr/lib/titan LD_PRELOAD=libudtrace.so systemctl status

Report from the Geniatech vs. McHardy GPL violation court hearing

Harald Welte — Tue, 06 Mar 2018 23:00:00 GMT

Today, I took some time off to attend the court hearing in the appeal hearing related to a GPL infringement dispute between former netfilter colleague Partrick McHardy and Geniatech Europe

I am not in any way legally involved in the lawsuit on either the plaintiff or the defendant side. However, as a fellow (former) Linux kernel developer myself, and a long-term Free Software community member who strongly believes in the copyleft model, I of course am very interested in this case.

History of the Case

This case is about GPL infringements in consumer electronics devices based on a GNU/Linux operating system, including the Linux kernel and at least some devices netfilter/iptables. The specific devices in question are a series of satellite TV receivers built by a Shenzhen (China) based company Geniatech, which is represented in Europe by Germany-based Geniatech Europe GmbH.

The Geniatech Europe CEO has openly admitted (out of court) that they had some GPL incompliance in the past, and that there was failure on their part that needed to be fixed. However, he was not willing to accept an overly wide claim in the preliminary injunction against his company.

The history of the case is that at some point in July 2017, Patrick McHardy has made a test purchase of a Geniatech Europe product, and found it infringing the GNU General Public License v2. Apparently no source code (and/or written offer) had been provide alongside the binary - a straight-forward violation of the license terms and hence a violation of copyright. The plaintiff then asked the regional court of Cologne to issue a preliminary injunction against the defendant, which was granted on September 8th,2017.

In terms of legal procedure, in Germany, when a plaintiff applies for a preliminary injunction, it is immediately granted by the court after brief review of the filing, without previously hearing the defendant in an oral hearing. If the defendant (like in this case) wishes to appeal the preliminary injunction, it files an appeal which then results in an oral hearing. This is what happened, after which the district court of cologne (Landgericht Koeln) on October 20, 2017 issued ruling 14 O 188/17 partially upholding the injunction.

All in all, nothing particularly unusual about this. There is no dispute about a copyright infringement having existed, and this generally grants any of the copyright holders the right to have the infringing party to cease and desist from any further infringement.

However, this injunction has a very wide scope, stating that the defendant was to cease and desist not only from ever publishing, selling, offering for download any version of Linux (unless being compliant to the license). It furthermore asked the defendant to cease and desist

from putting hyperlinks on their website to any version of Linux
from asking users to download any version of Linux

unless the conditions of the GPL are met, particularly the clauses related to providing the complete and corresponding source code.

The appeals case at OLG Cologne

The defendant now escalated this to the next higher court, the higher regional court of Cologne (OLG Koeln), asking to withdraw the earlier ruling of the lower court, i.e. removing the injunction with its current scope.

The first very positive surprise at the hearing was the depth in which the OLG court has studied the subject matter of the dispute prior to the hearing. In the many GPL related court cases that I witnessed so far, it was by far the most precise analysis of how Linux kernel development works, and this despite the more than 1000 pages of filings that parties had made to the court to this point.

Just to give you some examples:

the court understood that Linux was created by Linus Torvalds in 1991 and released under GPL to facilitate the open and collaborative development
the court recognized that there is no co-authorship / joint authorship (German: Miturheber) in the Linux kernel as a whole, as it was not a group of people planning+developing a given program together, but it is a program that has been released by Linus Torvalds and has since been edited by more than 15.000 developers without any "grand joint plan" but rather in successive iterations. This situation constitutes "editing authorship" (German: Bearbeiterurheber)
the court further recognized that being listed as "head of the netfilter core team" or a "subsystem maintainer" doesn't necessarily mean that one is contributing copyrightable works. Reviewing thousands of patches doesn't mean you own copyright on them, drawing an analogy to an editorial office at a publisher.
the court understood there are plenty of Linux versions that may not even contain any of Patric McHardy's code (such as older versions)

After about 35 minutes of the presiding judge explaining the court's understanding of the case (and how kernel development works), it went on to summarize the summary of their internal elaboration at the court prior to the meeting.

In this summary, the presiding judge stated very clearly that they believe there is some merit to the arguments of the defendant, and that they would be inclined in a ruling favorable to the defendant based on their current understanding of the case.

He cited the following main reasons:

The Linux kernel development model does not support the claim of Patrick McHardy having co-authored Linux. In so far, he is only an editing author (Bearbeiterurheber), and not a co-author. Nevertheless, even an editing author has the right to ask for cease and desist, but only on those portions that he authored/edited, and not on the entire Linux kernel.
The plaintiff did not sufficiently show what exactly his contributions were and how they were forming themselves copyrightable works
The plaintiff did not substantiate what copyrightable contributions he has made outside of netfilter/iptables. His mere listing as general networking subsystem maintainer does not clarify what his copyrightable contributions were
The plaintiff being a member of the netfilter core team or even the head of the core team still doesn't support the claim of being a co-author, as netfilter substantially existed since 1999, three years before Patrick's first contribution to netfilter, and five years before joining the core team in 2004.

So all in all, it was clear that the court also thought the ruling on all of Linux was too far-fetching.

The court suggested that it might be better to have regular main proceedings, in which expert witnesses can be called and real evidence has to be provided, as opposed to the constraints of the preliminary procedure that was applied currently.

Some other details that were mentioned somewhere during the hearing:

Patrick McHardy apparently unilaterally terminated the license to his works in an e-mail dated 26th of July 2017 towards the defendant. According to the defendant (and general legal opinion, including my own position), this is in turn a violation of the GPLv2, as it only allowed plaintiff to create and publish modified versions of Linux under the obligation that he licenses his works under GPLv2 to any third party, including the defendant. The defendant believes this is abuse of his rights (German: Rechtsmissbraeuchlich).
sworn affidavits of senior kernel developer Greg Kroah-Hartman and current netfilter maintainer Pablo Neira were presented in support of some of the defendants claims. The contents of those are unfortunately not public, neither is the contents of the sworn affidavists presented by the plaintiff.
The defendant has made substantiated claims in his filings that Patrick McHardy would perform his enforcement activities not with the primary motivation of achieving license compliance, but as a method to generate monetary gain. Such claims include that McHardy has acted in more than 38 cases, in at least one of which he has requested a contractual penalty of 1.8 million EUR. The total amount of monies received as contractual penalties was quoted as over 2 million EUR to this point. Please note that those are claims made by the defendant, which were just reproduced by the court. The court has not assessed their validity. However, the presiding judge explicitly stated that he received a phone calls about this case from a lawyer known to him personally, who supported that large contractual penalties are being paid in other related cases.
One argument by the plaintiff seems to center around being listed as a general kernel networking maintainer until 2017 (despite his latest patches being from 2015, and those were netfilter only)

Withdrawal by Patrick McHardy

At some point, the court hearing was temporarily suspended to provide the legal representation of the plaintiff with the opportunity to have a Phone call with the plaintiff to decide if they would want to continue with their request to uphold the preliminary injunction. After a few minutes, the hearing was resumed, with the plaintiff withdrawing their request to uphold the injunction.

As a result, the injunction is now withdrawn, and the plaintiff has to bear all legal costs (court fees, lawyers costs on both sides).

Personal Opinion

For me, this is all of course a difficult topic. With my history of being the first to enforce the GNU GPLv2 in (equally German) court, it is unsurprising that I am in favor of license enforcement being performed by copyright holders.

I believe individual developers who have contributed to the Linux kernel should have the right to enforce the license, if needed. It is important to have distributed copyright, and to avoid a situation where only one (possibly industry friendly) entity would be able to take [legal] action.

I'm not arguing for a "too soft" approach. It's almost 15 years since the first court cases on license violations on (embedded) Linux, and the fact that the problem still exists today clearly shows the industry is very far from having solved a seemingly rather simple problem.

On the other hand, such activities must always be oriented to compliance, and compliance only. Collecting huge amounts of contractual penalties is questionable. And if it was necessary to collect such huge amounts to motivate large corporations to be compliant, then this must be done in the open, with the community knowing about it, and the proceeds of such contractual penalties must be donated to free software related entities to prove that personal financial gain is not a motivation.

The rumors of Patrick performing GPL enforcement for personal financial gain have been around for years. It was initially very hard for me to believe. But as more and more about this became known, and Patrick would refuse to any contact requests by his former netfilter team-mates as well as the wider kernel community make it hard to avoid drawing related conclusions.

We do need enforcement, both out of court and in court. But we need it to happen out of the closet, with the community in the picture, and without financial gain to individuals. The "principles of community oriented enforcement" of the Software Freedom Conservancy as well as the more recent (but much less substantial) kernel enforcement statement represent the most sane and fair approach for how we as a community should deal with license violations.

So am I happy with the outcome? Not entirely. It's good that an over-reaching injunction was removed. But then, a lot of money and effort was wasted on this, without any verdict/ruling. It would have been IMHO better to have a court ruling published, in which the injunction is substantially reduced in scope (e.g. only about netfilter, or specific versions of the kernel, or specific products, not about placing hyperlinks, etc.). It would also have been useful to have some of the other arguments end up in a written ruling of a court, rather than more or less "evaporating" in the spoken word of the hearing today, without advancing legal precedent.

Lessons learned for the developer community

In the absence of detailed knowledge on computer programming, legal folks tend to look at "metadata" more, as this is what they can understand.
It matters who has which title and when. Should somebody not be an active maintainer, make sure he's not listed as such.
If somebody ceases to be a maintainer or developer of a project, remove him or her from the respective lists immediately, not just several years later.
Copyright statements do matter. Make sure you don't merge any patches adding copyright statements without being sure they are actually valid.

Lessons learned for the IT industry

There may be people doing GPL enforcement for not-so-noble motives
Defending yourself against claims in court can very well be worth it, as opposed to simply settling out of court (presumably for some money). The Telefonica case in 2016 <>_ has shown this, as has this current Geniatech case. The legal system can work, if you give it a chance.
Nevertheless, if you have violated the license, and one of the copyright holders makes a properly substantiated claim, you still will get injunctions granted against you (and rightfully so). This was just not done in this case (not properly substantiated, scope of injunction too wide/coarse).

Dear Patrick

For years, your former netfilter colleagues and friends wanted to have a conversation with you. You have not returned our invitation so far. Please do reach out to us. We won't bite, but we want to share our views with you, and show you what implications your actions have not only on Linux, but also particularly on the personal and professional lives of the very developers that you worked hand-in-hand with for a decade. It's your decision what you do with that information afterwards, but please do give us a chance to talk. We would greatly appreciate if you'd take up that invitation for such a conversation. Thanks.

34C3 and its Osmocom GSM/UMTS network

Harald Welte — Sun, 31 Dec 2017 23:00:00 GMT

At the 34th annual Chaos Communication Congress, a team of Osmocom folks continued the many years old tradition of operating an experimental Osmocom based GSM network at the event. Though I've originally started that tradition, I'm not involved in installation and/or operation of that network, all the credits go to Lynxis, neels, tsaitgaist and the larger team of volunteers surrounding them. My involvement was only to answer the occasional technical question and to look at bugs that show up in the software during operation, and if possible fix them on-site.

34C3 marks two significant changes in terms of its cellular network:

the new post-nitb Osmocom stack was used, with OsmoBSC, OsmoMSC and OsmoHLR
both an GSM/GPRS network (on 1800 MHz) was operated ,as well as (for the first time) an UMTS network (in the 850 MHz band)

The good news is: The team did great work building this network from scratch, in a new venue, and without relying on people that have significant experience in network operation. Definitely, the team was considerably larger and more distributed than at the time when I was still running that network.

The bad news is: There was a seemingly endless number of bugs that were discovered while operating this network. Some shortcomings were known before, but the extent and number of bugs uncovered all across the stack was quite devastating to me. Sure, at some point from day 2 onwards we had a network that provided [some level of] service, and as far as I've heard, some ~ 23k calls were switched over it. But that was after more than two days of debugging + bug fixing, and we still saw unexplained behavior and crashes later on.

This is such a big surprise as we have put a lot of effort into testing over the last years. This starts from the osmo-gsm-tester software and continuously running test setup, and continues with the osmo-ttcn3-hacks integration tests that mainly I wrote during the last few months. Both us and some of our users have also (successfully!) performed interoperability testing with other vendors' implementations such as MSCs. And last, but not least, the individual Osmocom developers had been using the new post-NITB stack on their personal machines.

So what does this mean?

I'm sorry about the sub-standard state of the software and the resulting problems we've experienced in the 34C3 network. The extent of problems surprised me (and I presume everyone else involved)
I'm grateful that we've had the opportunity to discover all those bugs, thanks to the GSM team at 34C3, as well as Deutsche Telekom for donating 3 ARFCNs from their spectrum, as well as the German regulatory authority Bundesnetzagentur for providing the experimental license in the 850 MHz spectrum.
We need to have even more focus on automatic testing than we had so far. None of the components should be without exhaustive test coverage on at least the most common transactions, including all their failure modes (such as timeouts, rejects, ...)

My preferred method of integration testing has been by using TTCN-3 and Eclipse TITAN to emulate all the interfaces surrounding a single of the Osmocom programs (like OsmoBSC) and then test both valid and invalid transactions. For the BSC, this means emulating MS+BTS on Abis; emulating MSC on A; emulating the MGW, as well as the CTRL and VTY interfaces.

I currently see the following areas in biggest need of integration testing:

OsmoHLR (which needs a GSUP implementation in TTCN-3, which I've created on the spot at 34C3) where we e.g. discovered that updates to the subscriber via VTY/CTRL would surprisingly not result in an InsertSubscriberData to VLR+SGSN
OsmoMSC, particularly when used with external MNCC handlers, which was so far blocked by the lack of a MNCC implementation in TTCN-3, which I've been working on both on-site and after returning back home.
user plane testing for OsmoMGW and other components. We currently only test the control plane (MGCP), but not the actual user plane e.g. on the RTP side between the elements
UMTS related testing on OsmoHNBGW, OsmoMSC and OsmoSGSN. We currently have no automatic testing at all in these areas.

Even before 34C3 and the above-mentioned experiences, I concluded that for 2018 we will pursue a test-driven development approach for all new features added by the sysmocom team to the Osmocom code base. The experience with the many issues at 34C3 has just confirmed that approach. In parallel, we will have to improve test coverage on the existing code base, as outlined above. The biggest challenge will of course be to convince our paying customers of this approach, but I see very little alternative if we want to ensure production quality of our cellular stack.

So here we come: 2018, The year of testing.

Osmocom Review 2017

Harald Welte — Sun, 31 Dec 2017 23:00:00 GMT

As 2017 has just concluded, let's have a look at the major events and improvements in the Osmocom Cellular Infrastructure projects (i.e. those projects dealing with building protocol stacks and network elements for mobile network infrastructure.

I've prepared a detailed year 2017 summary at the osmocom.org website, but let me write a bit about the most note-worthy topics here.

NITB Split

Once upon a time, we implemented everything needed to operate a GSM network inside a single process called OsmoNITB. Those days are now gone, and we have separate OsmoBSC, OsmoMSC, OsmoHLR, OsmoSTP processes, which use interfaces that are interoperable with non-Osmocom implementations (which is what some of our users require).

This change is certainly the most significant change in the close-to-10-year history of the project. However, we have tried to make it as non-intrusive as possible, by using default point codes and IP addresses which will make the individual processes magically talk to each other if installed on a single machine.

We've also released a OsmoNITB Migration Guide, as well as our usual set of user manuals in order to help our users.

We'll continue to improve the user experience, to re-introduce some of the features lost in the split, such as the ability to attach names to the subscribers.

Testing

We have osmo-gsm-tester together with the two physical setups at the sysmocom office, which continuously run the latest Osmocom components and test an entire matrix of different BTSs, software configurations and modems. However, this tests at super low load, and it tests only signalling so far, not user plane yet. Hence, coverage is limited.

We also have unit tests as part of the 'make check' process, Jenkins based build verification before merging any patches, as well as integration tests for some of the network elements in TTCN-3. This is much more than we had until 2016, but still by far not enough, as we had just seen at the fall-out from the sub-optimal 34C3 event network.

OsmoCon

2017 also marks the year where we've for the first time organized a user-oriented event. It was a huge success, and we will for sure have another OsmoCon incarnation in 2018 (most likely in May or June). It will not be back-to-back with the developer conference OsmoDevCon this time.

SIGTRAN stack

We have a new SIGTRAN stakc with SUA, M3UA and SCCP as well as OsmoSTP. This has been lacking a long time.

OsmoGGSN

We have converted OpenGGSN into a true member of the Osmocom family, thereby deprecating OpenGGSN which we had earlier adopted and maintained.

On the Linux Kernel Enforcement Statement

Harald Welte — Mon, 06 Nov 2017 23:00:00 GMT

I'm late with covering this here, but work overload is having its toll on my ability to blog.

On October 16th, key Linux Kernel developers have released and anounced the Linux Kernel Community Enforcement Statemnt.

In its actual text, those key kernel developers cover

compliance with the reciprocal sharing obligations of GPLv2 is critical and mandatory
acknowledgement to the right to enforce
expression of interest to ensure that enforcement actions are conducted in a manner beneficial to the larger community
a method to provide reinstatement of rights after ceasing a license violation (see below)
that legal action is a last resort
that after resolving any non-compliance, the formerly incompliant user is welcome to the community

I wholeheartedly agree with those. This should be no surprise as I've been one of the initiators and signatories of the earlier statement of the netfilter project on GPL enforcement.

On the reinstatement of rights

The enforcement statement then specifically expresses the view of the signatories on the specific aspect of the license termination. Particularly in the US, among legal scholars there is a strong opinion that if the rights under the GPLv2 are terminated due to non-compliance, the infringing entity needs an explicit reinstatement of rights from the copyright holder. The enforcement statement now basically states that the signatories believe the rights should automatically be re-instated if the license violation ceases within 30 days of being notified of the license violation

To people like me living in the European (and particularly German) legal framework, this has very little to no implications. It has been the major legal position that any user, even an infringing user can automatically obtain a new license as soon as he no longer violates. He just (really or imaginary) obtains a new copy of the source code, at which time he again gets a new license from the copyright holders, as long as he fulfills the license conditions.

So my personal opinion as a non-legal person active in GPL compliance on the reinstatement statement is that it changes little to nothing regarding the jurisdiction that I operate in. It merely expresses that other developers express their intent and interest to a similar approach in other jurisdictions.

SFLC sues SFC over trademark infringement

Harald Welte — Mon, 06 Nov 2017 23:00:00 GMT

As the Software Freedom Conservancy (SFC) has publicly disclosed on their website, it appears that Software Freedom Law Center (SFLC) has filed for a trademark infringement lawsuit against SFC.

SFLC has launched SFC in 2006, and SFLC has helped and endorsed SFC in the past.

This lawsuit is hard to believe. What has this community come to, if its various members - who used all to be respected equally - start filing law suits against each other?

It's of course not known what kind of negotiations might have happened out-of-court before an actual lawsuit has been filed. Nevertheless, one would have hoped that people are able to talk to each other, and that the mutual respect for working at different aspects and with possibly slightly different strategies would have resulted in a less confrontational approach to resolving any dispute.

To me, this story just looks like there can only be losers on all sides, by far not just limited to the two entities in question.

On lwn.net some people, including high-ranking members of the FOSS community have started to spread conspiracy theories as to whether there's any secret scheming behind the scenes, particularly from the Linux Foundation towards SFLC to cause trouble towards the SFC and their possibly-not-overly-enjoyed-by-everyone enforcement activities.

I think this is complete rubbish. Neither have I ever had the impression that the LF is completely opposed to license enforcement to begin with, nor do I have remotely enough phantasy to see them engage in such malicious scheming.

What motivates SFLC and/or Eben to attack their former offspring is however unexplainable to the bystander. One hopes there is no connection to his departure from FSF about one year ago, where he served as general counsel for more than two decades.

Obtaining the local IP address of an unbound UDP socket

Harald Welte — Thu, 19 Oct 2017 22:00:00 GMT

Sometimes one is finding an interesting problem and is surprised that there is not a multitude of blog post, stackoverflow answers or the like about it.

A (I think) not so uncommon problem when working with datagram sockets is that you may want to know the local IP address that the OS/kernel chooses when sending a packet to a given destination.

In an unbound UDP socket, you basically send and receive packets with any number of peers from a single socket. When sending a packet to destination Y, you simply pass the destination address/port into the sendto() socket function, and the OS/kernel will figure out which of its local IP addresses will be used for reaching this particular destination.

If you're a dumb host with a single default router, then the answer to that question is simple. But in any reasonably non-trivial use case, your host will have a variety of physical and/or virtual network devices with any number of addresses on them.

Why would you want to know that address? Because maybe you need to encode that address as part of a packet payload. In the current use case that we have, it is the OsmoMGW, implementing the IETF MGCP Media Gateway Control Protocol.

So what can you do? You can actually create a new "trial" socket, not bind it to any specific local address/port, but connect() it to the destination of your IP packets. Then you do a getsockname(), which will give you the local address/port the kernel has selected for this socket. And that's exactly the answer to your question. You can now close the "trial" socket and have learned which local IP address the kernel would use if you were to send a packet to that destination.

At least on Linux, this works. While getsockname() is standard BSD sockets API, I'm not sure how portable it is to use it on a socket that has not been explicitly bound by a prior call to bind().

Invited keynote + TTCN-3 talk at netdevconf 2.2 in Seoul

Harald Welte — Mon, 09 Oct 2017 22:00:00 GMT

It was a big surprise that I've recently been invited to give a keynote on netfilter history at netdevconf 2.2.

First of all, I wouldn't have expected netfilter to be that relevant next to all the other [core] networking topics at netdevconf. Secondly, I've not been doing any work on netfilter for about a decade now, so my memory is a bit rusty by now ;)

Speaking of Rusty: Timing wise there is apparently a nice coincidence that I'll be able to meet up with him in Berlin later this month, i.e. hopefully we can spend some time reminiscing about old times and see what kind of useful input he has for the keynote.

I'm also asking my former colleagues and successors in the netfilter project to share with me any note-worthy events or anecdotes, particularly also covering the time after my retirement from the core team. So if you have something that you believe shouldn't miss in a keynote on netfilter project history: Please reach out to me by e-mail ASAP and let me know about it.

To try to fend off the elder[ly] statesmen image that goes along with being invited to give keynotes about the history of projects you were working on a long time ago, I also submitted an actual technical talk: TTCN-3 and Eclipse Titan for testing protocol stacks, in which I'll cover my recent journey into TTCN-3 and TITAN land, and how I think those tools can help us in the Linux [kernel] networking community to productively produce tests for the various protocols.

As usual for netdevconf, there are plenty of other exciting talks in the schedule

I'm very much looking forward to both visiting Seoul again, as well as meeting lots of the excellent people involved in the Linux networking subsystems. See ya!

Ten years Openmoko Neo1973 release anniversary dinner

Harald Welte — Sun, 08 Oct 2017 22:00:00 GMT

As I noted earlier this year, 2017 marks the tenth anniversary of shipping the first Openmoko phone, the Neo1973.

On this occasion, a number of the key people managed to gather for an anniversary dinner in Taipei. Thanks for everyone who could make it, it was very good to see them together again. Sadly, by far not everyone could attend. You have been missed!

The award for the most crazy attendee of the meeting goes out to my friend Milosch, who has actually flown from his home in the UK to Taiwan, only to meet up with old friends and attend the anniversary dinner.

You can some pictures in Milosch's related tweet.

On Vacation

Harald Welte — Wed, 04 Oct 2017 22:00:00 GMT

In case you're wondering about the lack of activity not only on this blog but also in git repositories, mailing lists and the like: I've been on vacation since September 13. It's my usual "one month in Taiwan" routine, during which I spend some time in Taipei, but also take several long motorbike tours around mostly rural Taiwan.

You can find the occasional snapshot in my twitter feed, such as the, pictures, here and there.

Purism Librem 5 campaign

Harald Welte — Sat, 02 Sep 2017 22:00:00 GMT

There's a new project currently undergoing crowd funding that might be of interest to the former Openmoko community: The Purism Librem 5 campaign.

Similar to Openmoko a decade ago, they are aiming to build a FOSS based smartphone built on GNU/Linux without any proprietary drivers/blobs on the application processor, from bootloader to userspace.

Furthermore (just like Openmoko) the baseband processor is fully isolated, with no shared memory and with the Linux-running application processor being in full control.

They go beyond what we wanted to do at Openmoko in offering hardware kill switches for camera/phone/baseband/bluetooth. During Openmoko days we assumed it is sufficient to simply control all those bits from the trusted Linux domain, but of course once that might be compromised, a physical kill switch provides a completely different level of security.

I wish them all the best, and hope they can leave a better track record than Openmoko. Sure, we sold some thousands of phones, but the company quickly died, and the state of software was far from end-user-ready. I think the primary obstacles/complexities are verification of the hardware design as well as the software stack all the way up to the UI.

The budget of ~ 1.5 million seems extremely tight from my point of view, but then I have no information about how much Puri.sm is able to invest from other sources outside of the campaign.

If you're a FOSS developer with a strong interest in a Free/Open privacy-first smartphone, please note that they have several job openings, from Kernel Developer to OS Developer to UI Developer. I'd love to see some talents at work in that area.

It's a bit of a pity that almost all of the actual technical details are unspecified at this point (except RAM/flash/main-cpu). No details on the cellular modem/chipset used, no details on the camera, neither on the bluetooth chipset, wifi chipset, etc. This might be an indication of the early stage of their plannings. I would have expected that one has ironed out those questions before looking for funding - but then, it's their campaign and they can run it as they see it fit!

I for my part have just put in a pledge for one phone. Let's see what will come of it. In case you feel motivated by this post to join in: Please keep in mind that any crowdfunding campaign bears significant financial risks. So please make sure you made up your mind and don't blame my blog post for luring you into spending money :)

First actual XMOS / XCORE project

Harald Welte — Fri, 01 Sep 2017 22:00:00 GMT

For many years I've been fascinated by the XMOS XCore architecture. It offers a surprisingly refreshing alternative virtually any other classic microcontroller architectures out there. However, despite reading a lot about it years ago, being fascinated by it, and even giving a short informal presentation about it once, I've so far never used it. Too much "real" work imposes a high barrier to spending time learning about new architectures, languages, toolchains and the like.

Introduction into XCore

Rather than having lots of fixed-purpose built-in "hard core" peripherals for interfaces such as SPI, I2C, I2S, etc. the XCore controllers have a combination of

I/O ports for 1/4/8/16/32 bit wide signals, with SERDES, FIFO, hardware strobe generation, etc
Clock blocks for using/dividing internal or external clocks
hardware multi-threading that presents 8 logical threads on each core
xCONNECT links that can be used to connect multiple processors over 2 or 5 wires per direction
channels as a means of communication (similar to sockets) between threads, whether on the same xCORE or a remote core via xCONNECT
an extended C (xC) programming language to make use of parallelism, channels and the I/O ports

In spirit, it is like a 21st century implementation of some of the concepts established first with Transputers.

My main interest in xMOS has been the flexibility that you get in implementing not-so-standard electronics interfaces. For regular I2C, UART, SPI, etc. there is of course no such need. But every so often one encounters some interface that's very rately found (like the output of an E1/T1 Line Interface Unit).

Also, quite often I run into use cases where it's simply impossible to find a microcontroller with a sufficient number of the related peripherals built-in. Try finding a microcontroller with 8 UARTs, for example. Or one with four different PCM/I2S interfaces, which all can run in different clock domains.

The existing options of solving such problems basically boil down to either implementing it in hard-wired logic (unrealistic, complex, expensive) or going to programmable logic with CPLD or FPGAs. While the latter is certainly also quite interesting, the learning curve is steep, the tools anything but easy to use and the synthesising time (and thus development cycles) long. Furthermore, your board design will be more complex as you have that FPGA/CPLD and a microcontroller, need to interface the two, etc (yes, in high-end use cases there's the Zynq, but I'm thinking of several orders of magnitude less complex designs).

Of course one can also take a "pure software" approach and go for high-speed bit-banging. There are some ARM SoCs that can toggle their pins. People have reported rates like 14 MHz being possible on a Raspberry Pi. However, when running a general-purpose OS in parallel, this kind of speed is hard to do reliably over long term, and the related software implementations are going to be anything but nice to write.

So the XCore is looking like a nice alternative for a lot of those use cases. Where you want a microcontroller with more programmability in terms of its I/O capabilities, but not go as far as to go full-on with FPGA/CPLD development in Verilog or VHDL.

My current use case

My current use case is to implement a board that can accept four independent PCM inputs (all in slave mode, i.e. clock provided by external master) and present them via USB to a host PC. The final goal is to have a board that can be combined with the sysmoQMOD and which can interface the PCM audio of four cellular modems concurrently.

While XMOS is quite strong in the Audio field and you can find existing examples and app notes for I2S and S/PDIF, I couldn't find any existing code for a PCM slave of the given requirements (short frame sync, 8kHz sample rate, 16bit samples, 2.048 MHz bit clock, MSB first).

I wanted to get a feeling how well one can implement the related PCM slave. In order to test the slave, I decided to develop the matching PCM master and run the two against each other. Despite having never written any code for XMOS before, nor having used any of the toolchain, I was able to implement the PCM master and PCM slave within something like ~6 hours, including simulation and verification. Sure, one can certainly do that in much less time, but only once you're familiar with the tools, programming environment, language, etc. I think it's not bad.

The biggest problem was that the clock phase for a clocked output port cannot be configured, i.e. the XCore insists on always clocking out a new bit at the falling edge, while my use case of course required the opposite: Clocking oout new signals at the rising edge. I had to use a second clock block to generate the inverted clock in order to achieve that goal.

Beyond that 4xPCM use case, I also have other ideas like finally putting the osmo-e1-xcvr to use by combining it with an XMOS device to build a portable E1-to-USB adapter. I have no clue if and when I'll find time for that, but if somebody wants to join in: Let me know!

The good parts

Documentation excellent

I found the various pieces of documentation extremely useful and very well written.

Fast progress

I was able to make fast progress in solving the first task using the XMOS / Xcore approach.

Soft Cores developed in public, with commit log

You can find plenty of soft cores that XMOS has been developing on github at https://github.com/xcore, including the full commit history.

This type of development is a big improvement over what most vendors of smaller microcontrollers like Atmel are doing (infrequent tar-ball code-drops without commit history). And in the case of the classic uC vendors, we're talking about drivers only. In the XMOS case it's about the entire logic of the peripheral!

You can for example see that for their I2C core, the very active commit history goes back to January 2011.

xSIM simulation extremely helpful

The xTIMEcomposer IDE (based on Eclipse) contains extensive tracing support and an extensible near cycle accurate simulator (xSIM). I've implemented a PCM mater and PCM slave in xC and was able to simulate the program while looking at the waveforms of the logic signals between those two.

The bad parts

Unfortunately, my extremely enthusiastic reception of XMOS has suffered quite a bit over time. Let me explain why:

Hard to get XCore chips

While the product portfolio on on the xMOS website looks extremely comprehensive, the vast majority of the parts is not available from stock at distributors. You won't even get samples, and lead times are 12 weeks (!). If you check at digikey, they have listed a total of 302 different XMOS controllers, but only 35 of them are in stock. USB capable are 15. With other distributors like Farnell it's even worse.

I've seen this with other semiconductor vendors before, but never to such a large extent. Sure, some packages/configurations are not standard products, but having only 11% of the portfolio actually available is pretty bad.

In such situations, where it's difficult to convince distributors to stock parts, it would be a good idea for XMOS to stock parts themselves and provide samples / low quantities directly. Not everyone is able to order large trays and/or capable to wait 12 weeks, especially during the R&D phase of a board.

Extremely limited number of single-bit ports

In the smaller / lower pin-count parts, like the XU[F]-208 series in QFN/LQFP-64, the number of usable, exposed single-bit ports is ridiculously low. Out of the total 33 I/O lines available, only 7 can be used as single-bit I/O ports. All other lines can only be used for 4-, 8-, or 16-bit ports. If you're dealing primarily with serial interfaces like I2C, SPI, I2S, UART/USART and the like, those parallel ports are of no use, and you have to go for a mechanically much larger part (like XU[F]-216 in TQFP-128) in order to have a decent number of single-bit ports exposed. Those parts also come with twice the number of cores, memory, etc- which you don't need for slow-speed serial interfaces...

Insufficient number of exposed xLINKs

The smaller parts like XU[F]-208 only have one xLINK exposed. Of what use is that? If you don't have at least two links available, you cannot daisy-chain them to each other, let alone build more complex structures like cubes (at least 3 xLINKs).

So once again you have to go to much larger packages, where you will not use 80% of the pins or resources, just to get the required number of xLINKs for interconnection.

Change to a non-FOSS License

XMOS deserved a lot of praise for releasing all their soft IP cores as Free / Open Source Software on github at https://github.com/xcore. The License has basically been a 3-clause BSD license. This was a good move, as it meant that anyone could create derivative versions, whether proprietary or FOSS, and there would be virtually no license incompatibilities with whatever code people wanted to write.

However, to my very big disappointment, more recently XMOS seems to have changed their policy on this. New soft cores (released at https://github.com/xmos as opposed to the old https://github.com/xcore) are made available under a non-free license. This license is nothing like BSD 3-clause license or any other Free Software or Open Source license. It restricts the license to use the code together with an XMOS product, requires the user to contribute fixes back to XMOS and contains references to importand export control. This license is incopatible with probably any FOSS license in existance, making it impossible to write FOSS code on XMOS while using any of the new soft cores released by XMOS.

But even beyond that license change, not even all code is provided in source code format anymore. The new USB library (lib_usb) is provided as binary-only library, for example.

If you know anyone at XMOS management or XMOS legal with whom I could raise this topic of license change when transitioning from older sc_* software to later lib_* code, I would appreciate this a lot.

Proprietary Compiler

While a lot of the toolchain and IDE is based on open source (Eclipse, LLVM, ...), the actual xC compiler is proprietary.

The sad state of voice support in cellular modems

Harald Welte — Fri, 01 Sep 2017 22:00:00 GMT

Cellular modems have existed for decades and come in many shapes and kinds. They contain the cellular baseband processor, RF frontend, protocol stack software and anything else required to communicate with a cellular network. Basically a phone without display or input.

During the last decade or so, the vast majority of cellular modems come as LGA modules, i.e. a small PCB with all components on the top side (and a shielding can), which has contact pads on the bottom so you can solder it onto your mainboard. You can obtain them from vendors such as Sierra Wireless, u-blox, Quectel, ZTE, Huawei, Telit, Gemalto, and many others.

In most cases, the vendors now also solder those modules to small adapter boards to offer the same product in mPCIe form-factor. Other modems are directly manufactured in mPCIe or NGFF aka m.2 form-factor.

As long as those modems were still 2G / 2.5G / 2.75G, the main interconnection with the host (often some embedded system) was a serial UART. The Audio input/output for voice calls was made available as analog signals, ready to connect a microphone and spekaer, as that's what the cellular chipsets were designed for in the smartphones. In the Openmoko phones we also interfaced the audio of the cellular modem in analog, exactly for that reason.

From 3G onwards, the primary interface towards the host is now USB, with the modem running as a USB device. If your laptop contains a cellular modem, you will see it show up in the lsusb output.

From that point onwards, it would have made a lot of sense to simply expose the audio also via USB. Simply offer a multi-function USB device that has both whatever virutal serial ports for AT commands and network device for IP, and add a USB Audio device to it. It would simply show up as a "USB sound card" to the host, with all standard drivers working as expected. Sadly, nobody seems to have implemented this, at least not in a supported production version of their product

Instead, what some modem vendors have implemented as an ugly hack is the transport of 8kHz 16bit PCM samples over one of the UARTs. See for example the Quectel UC-20 or the Simcom SIM7100 which implement such a method.

All the others ignore any acess to the audio stream from software to a large part. One wonders why that is. From a software and systems architecture perspective it would be super easy. Instead, what most vendors do, is to expose a digital PCM interface. This is suboptimal in many ways:

there is no mPCIe standard on which pins PCM should be exposed
no standard product (like laptop, router, ...) with mPCIe slot will have anything connected to those PCM pins

Furthermore, each manufacturer / modem seems to support a different subset of dialect of the PCM interface in terms of

voltage (almost all of them are 1.8V, while mPCIe signals normally are 3.3V logic level)
master/slave (almost all of them insist on being a clock master)
sample format (alaw/ulaw/linear)
clock/bit rate (mostly 2.048 MHz, but can be as low as 128kHz)
frame sync (mostly short frame sync that ends before the first bit of the sample)
endianness (mostly MSB first)
clock phase (mostly change signals at rising edge; sample at falling edge)

It's a real nightmare, when it could be so simple. If they implemented USB-Audio, you could plug a cellular modem into any board with a mPCIe slot and it would simply work. As they don't, you need a specially designed mainboard that implements exactly the specific dialect/version of PCM of the given modem.

By the way, the most "amazing" vendor seems to be u-blox. Their Modems support PCM audio, but only the solder-type version. They simply didn't route those signals to the mPCIe slot, making audio impossible to use when using a connectorized modem. How inconvenient.

Summary

If you want to access the audio signals of a cellular modem from software, then you either

have standard hardware and pick one very specific modem model and hope this is available sufficiently long during your application, or
build your own hardware implementing a PCM slave interface and then pick + choose your cellular modem

On the Osmocom mpcie-breakout board and the sysmocom QMOD board we have exposed the PCM related pins on 2.54mm headers to allow for some separate board to pick up that PCM and offer it to the host system. However, such separate board hasn't been developed so far.

Osmocom jenkins test suite execution

Harald Welte — Sat, 19 Aug 2017 22:00:00 GMT

Automatic Testing in Osmocom

So far, in many Osmocom projects we have unit tests next to the code. Those unit tests are executing test on a per-C-function basis, and typically use the respective function directly from a small test program, executed at make check time. The actual main program (like OsmoBSC or OsmoBTS) is not executed at that time.

We also have VTY testing, which specifically tests that the VTY has proper documentation for all nodes of all commands.

Then there's a big gap, and we have osmo-gsm-tester for testing a full cellular network end-to-end. It includes physical GSM modesm, coaxial distribution network, attenuators, splitter/combiners, real BTS hardware and logic to run the full network, from OsmoBTS to the core - both for OsmoNITB and OsmoMSC+OsmoHLR based networks.

However, I think a lot of testing falls somewhere in between, where you want to run the program-under-test (e.g. OsmoBSC), but you don't want to run the MS, BTS and MSC that normally surroudns it. You want to test it by emulating the BTS on the Abis sid and the MSC on the A side, and just test Abis and A interface transactions.

For this kind of testing, I have recently started to investigate available options and tools.

OsmoSTP (M3UA/SUA)

Several months ago, during the development of OsmoSTP, I disovered that the Network Programming Lab of Münster University of Applied Sciences led by Michael Tuexen had released implementations of the ETSI test suite for the M3UA and SUA members of the SIGTRAN protocol family.

The somewhat difficult part is that they are implemented in scheme, using the guile interpreter/compiler, as well as a C-language based execution wrapper, which then is again called by another guile wrapper script.

I've reimplemented the test executor in python and added JUnitXML output to it. This means it can feed the test results directly into Jenkins.

I've also cleaned up the Dockerfiles and related image generation for the osmo-stp-master, m3ua-test and sua-test images, as well as some scripts to actually execute them on one of the Builders. You can find related Dockerfiles as well as associtaed Makfiles in http://git.osmocom.org/docker-playground

The end result after integration with Osmocom jenkins can be seen in the following examples on jenkins.osmocom.org for M3UA and for SUA

Triggering the builds is currently periodic once per night, but we could of course also trigger them automatically at some later point.

OpenGGSN (GTP)

For OpenGGSN, during the development of IPv6 PDP context support, I wrote some test infrastructure and test cases in TTCN-3. Those test cases can be found at http://git.osmocom.org/osmo-ttcn3-hacks/tree/ggsn_tests

I've also packaged the GGSN and the test cases each into separate Docker containers called osmo-ggsn-latest and ggsn-test. Related Dockerfiles and Makefiles can again be found in http://git.osmocom.org/docker-playground - together with a Eclipse TITAN Docker base image using Debian Stretch called debian-stretch-titan

Using those TTCN-3 test cases with the TITAN JUnitXML logger plugin we can again integrate the results directly into Jenkins, whose results you can see at https://jenkins.osmocom.org/jenkins/view/TTCN3/job/ttcn3-ggsn-test/14/testReport/(root)/GGSN_Tests/

Further Work

I've built some infrastructure for Gb (NS/BSSGP), VirtualUm and other testing, but yet have to build Docker images and related jenkins integration for it. Stay tuned about that. Also, lots more actual tests cases are required. I'm very much looking forward to any contributions.

IPv6 User Plane support in Osmocom

Harald Welte — Tue, 08 Aug 2017 22:00:00 GMT

Preface

Cellular systems ever since GPRS are using a tunnel based architecture to provide IP connectivity to cellular terminals such as phones, modems, M2M/IoT devices and the like. The MS/UE establishes a PDP context between itself and the GGSN on the other end of the cellular network. The GGSN then is the first IP-level router, and the entire cellular network is abstracted away from the User-IP point of view.

This architecture didn't change with EGPRS, and not with UMTS, HSxPA and even survived conceptually in LTE/4G.

While the concept of a PDP context / tunnel exists to de-couple the transport layer from the structure and type of data inside the tunneled data, the primary user plane so far has been IPv4.

In Osmocom, we made sure that there are no impairments / assumptions about the contents of the tunnel, so OsmoPCU and OsmoSGSN do not care at all what bits and bytes are transmitted in the tunnel.

The only Osmocom component dealing with the type of tunnel and its payload structure is OpenGGSN. The GGSN must allocate the address/prefix assigned to each individual MS/UE, perform routing between the external IP network and the cellular network and hence is at the heart of this. Sadly, OpenGGSN was an abandoned project for many years until Osmocom adopted it, and it only implemented IPv4.

This is actually a big surprise to me. Many of the users of the Osmocom stack are from the IT security area. They use the Osmocom stack to test mobile phones for vulnerabilities, analyze mobile malware and the like. As any penetration tester should be interested in analyzing all of the attack surface exposed by a given device-under-test, I would have assumed that testing just on IPv4 would be insufficient and over the past 9 years, somebody should have come around and implemented the missing bits for IPv6 so they can test on IPv6, too.

In reality, it seems nobody appears to have shared line of thinking and invested a bit of time in growing the tools used. Or if they did, they didn't share the related code.

In June 2017, Gerrie Roos submitted a patch for OpenGGSN IPv6 support that raised hopes about soon being able to close that gap. However, at closer sight it turns out that the code was written against a more than 7 years old version of OpenGGSN, and it seems to primarily focus on IPv6 on the outer (transport) layer, rather than on the inner (user) layer.

OpenGGSN IPv6 PDP Context Support

So in July 2017, I started to work on IPv6 PDP support in OpenGGSN.

Initially I thought How hard can it be? It's not like IPv6 is new to me (I joined 6bone under 3ffe prefixes back in the 1990ies and worked on IPv6 support in ip6tables ages ago. And aside from allocating/matching longer addresses, what kind of complexity does one expect?

After my initial attempt of implementation, partially mislead by the patch that was contributed against that 2010-or-older version of OpenGGSN, I'm surprised how wrong I was.

In IPv4 PDP contexts, the process of establishing a PDP context is simple:

Request establishment of a PDP context, set the type to IETF IPv4
Receive an allocated IPv4 End User Address
Optionally use IPCP (part of PPP) to reques and receive DNS Server IP addresses

So I implemented the identical approach for IPv6. Maintain a pool of IPv6 addresses, allocate one, and use IPCP for DNS. And nothing worked.

IPv6 PDP contexts assign a /64 prefix, not a single address or a smaller prefix
The End User Address that's part of the Signalling plane of Layer 3 Session Management and GTP is not the actual address, but just serves to generate the interface identifier portion of a link-local IPv6 address
IPv6 stateless autoconfiguration is used with this link-local IPv6 address inside the User Plane, after the control plane signaling to establish the PDP context has completed. This means the GGSN needs to parse ICMPv6 router solicitations and generate ICMPV6 router advertisements.

To make things worse, the stateless autoconfiguration is modified in some subtle ways to make it different from the normal SLAAC used on Ethernet and other media:

the timers / lifetimes are different
only one prefix is permitted
only a prefix length of 64 is permitted

A few days later I implemented all of that, but it still didn't work. The problem was with DNS server adresses. In IPv4, the 3GPP protocols simply tunnel IPCP frames for this. This makes a lot of sense, as IPCP is designed for point-to-point interfaces, and this is exactly what a PDP context is.

In IPv6, the corresponding IP6CP protocol does not have the capability to provision DNS server addresses to a PPP client. WTF? The IETF seriously requires implementations to do DHCPv6 over PPP, after establishing a point-to-point connection, only to get DNS server information?!? Some people suggested an IETF draft to change this butthe draft has expired in 2011 and we're still stuck.

While 3GPP permits the use of DHCPv6 in some scenarios, support in phones/modems for it is not mandatory. Rather, the 3GPP has come up with their own mechanism on how to communicate DNS server IPv6 addresses during PDP context activation: The use of containers as part of the PCO Information Element used in L3-SM and GTP (see Section 10.5.6.3 of 3GPP TS 24.008. They by the way also specified the same mechanism for IPv4, so there's now two competing methods on how to provision IPv4 DNS server information: IPCP and the new method.

In any case, after some more hacking, OpenGGSN can now also provide DNS server information to the MS/UE. And once that was implemented, I had actual live uesr IPv6 data over a full Osmocom cellular stack!

Summary

We now have working IPv6 User IP in OpenGGSN. Together with the rest of the Osmocom stack you can operate a private GPRS, EGPRS, UMTS or HSPA network that provide end-to-end transparent, routed IPv6 connectivity to mobile devices.

All in all, it took much longer than nneeded, and the following questions remain in my mind:

why did the IETF not specify IP6CP capabilities to configure DNS servers?
why the complex two-stage address configuration with PDP EUA allocation for the link-local address first and then stateless autoconfiguration?
why don't we simply allocate the entire prefix via the End User Address information element on the signaling plane? For sure next to the 16byte address we could have put one byte for prefix-length?
why do I see duplication detection flavour neighbour solicitations from Qualcomm based phones on what is a point-to-point link with exactly two devices: The UE and the GGSN?
why do I see link-layer source address options inside the ICMPv6 neighbor and router solicitation from mobile phones, when that option is specifically not to be used on point-to-point links?
why is the smallest prefix that can be allocated a /64? That's such a waste for a point-to-point link with a single device on the other end, and in times of billions of connected IoT devices it will just encourage the use of non-public IPv6 space (i.e. SNAT/MASQUERADING) while wasting large parts of the address space

Some of those choices would have made sense if one would have made it fully compatible with normal IPv6 like e.g. on Ethernet. But implementing ICMPv6 router and neighbor solicitation without getting any benefit such as ability to have multiple prefixes, prefixes of different lengths, I just don't understand why anyone ever thought You can find the code at http://git.osmocom.org/openggsn/log/?h=laforge/ipv6 and the related ticket at https://osmocom.org/issues/2418

Virtual Um interface between OsmoBTS and OsmocomBB

Harald Welte — Tue, 18 Jul 2017 22:00:00 GMT

During the last couple of days, I've been working on completing, cleaning up and merging a Virtual Um interface (i.e. virtual radio layer) between OsmoBTS and OsmocomBB. After I started with the implementation and left it in an early stage in January 2016, Sebastian Stumpf has been completing it around early 2017, with now some subsequent fixes and improvements by me. The combined result allows us to run a complete GSM network with 1-N BTSs and 1-M MSs without any actual radio hardware, which is of course excellent for all kinds of testing scenarios.

The Virtual Um layer is based on sending L2 frames (blocks) encapsulated via GSMTAP UDP multicast packets. There are two separate multicast groups, one for uplink and one for downlink. The multicast nature simulates the shared medium and enables any simulated phone to receive the signal from multiple BTSs via the downlink multicast group.

In OsmoBTS, this is implemented via the new osmo-bts-virtual BTS model.

In OsmocomBB, this is realized by adding virtphy virtual L1, which speaks the same L1CTL protocol that is used between the real OsmcoomBB Layer1 and the Layer2/3 programs such as mobile and the like.

Now many people would argue that GSM without the radio and actual handsets is no fun. I tend to agree, as I'm a hardware person at heart and I am not a big fan of simulation.

Nevertheless, this forms the basis of all kinds of possibilities for automatized (regression) testing in a way and for layers/interfaces that osmo-gsm-tester cannot cover as it uses a black-box proprietary mobile phone (modem). It is also pretty useful if you're traveling a lot and don't want to carry around a BTS and phones all the time, or get some development done in airplanes or other places where operating a radio transmitter is not really a (viable) option.

If you're curious and want to give it a shot, I've put together some setup instructions at the Virtual Um page of the Osmocom Wiki.

Ten years after first shipping Openmoko Neo1973

Harald Welte — Sun, 09 Jul 2017 14:00:00 GMT

Exactly 10 years ago, on July 9th, 2007 we started to sell+ship the first Openmoko Neo1973. To be more precise, the webshop actually opened a few hours early, depending on your time zone. Sean announced the availability in this mailing list post

I don't really have to add much to my ten years [of starting to work on] Openmoko anniversary blog post a year ago, but still thought it's worth while to point out the tenth anniversary.

It was exciting times, and there was a lot of pioneering spirit: Building a Linux based smartphone with a 100% FOSS software stack on the application processor, including all drivers, userland, applications - at a time before Android was known or announced. As history shows, we'd been working in parallel with Apple on the iPhone, and Google on Android. Of course there's little chance that a small taiwanese company can compete with the endless resources of the big industry giants, and the many Neo1973 delays meant we had missed the window of opportunity to be the first on the market.

It's sad that Openmoko (or similar projects) have not survived even as a special-interest project for FOSS enthusiasts. Today, virtually all options of smartphones are encumbered with way more proprietary blobs than we could ever imagine back then.

In any case, the tenth anniversary of trying to change the amount of Free Softwware in the smartphone world is worth some celebration. I'm reaching out to old friends and colleagues, and I guess we'll have somewhat of a celebration party both in Germany and in Taiwan (where I'll be for my holidays from mid-September to mid-October).

FOSS misconceptions, still in 2017

Harald Welte — Thu, 15 Jun 2017 22:00:00 GMT

The lack of basic FOSS understanding in Telecom

Given that the Free and Open Source movement has been around at least since the 1980ies, it puzzles me that people still seem to have such fundamental misconceptions about it.

Something that really triggered me was an article at LightReading [1] which quotes Ulf Ewaldsson, a leading Ericsson excecutive with

"I have yet to understand why we would open source something we think is really good software"

This completely misses the point. FOSS is not about making a charity donation of a finished product to the planet.

FOSS is about sharing the development costs among multiple players, and avoiding that everyone has to reimplement the wheel. Macro-Economically, it is complete and utter nonsense that each 3GPP specification gets implemented two dozens of times, by at least a dozen of different entities. As a result, products are way more expensive than needed.

If large Telco players (whether operators or equipment manufacturers) were to collaboratively develop code just as much as they collaboratively develop the protocol specifications, there would be no need for replicating all of this work.

As a result, everyone could produce cellular network elements at reduced cost, sharing the R&D expenses, and competing in key areas, such as who can come up with the most energy-efficient implementation, or can produce the most reliable hardware, the best receiver sensitivity, the best and most fair scheduling implementation, or whatever else. But some 80% of the code could probably be shared, as e.g. encoding and decoding messages according to a given publicly released 3GPP specification document is not where those equipment suppliers actually compete.

So my dear cellular operator executives: Next time you're cursing about the prohibitively expensive pricing that your equipment suppliers quote you: You only have to pay that much because everyone is reimplementing the wheel over and over again.

Equally, my dear cellular infrastructure suppliers: You are all dying one by one, as it's hard to develop everything from scratch. Over the years, many of you have died. One wonders, if we might still have more players left, if some of you had started to cooperate in developing FOSS at least in those areas where you're not competing. You could replicate what Linux is doing in the operating system market. There's no need in having a phalanx of different proprietary flavors of Unix-like OSs. It's way too expansive, and it's not an area in which most companies need to or want to compete anyway.

Management Summary

You don't first develop and entire product until it is finished and then release it as open source. This makes little economic sense in a lot of cases, as you've already invested into developing 100% of it. Instead, you actually develop a new product collaboratively as FOSS in order to not have to invest 100% but maybe only 30% or even less. You get a multitude of your R&D investment back, because you're not only getting your own code, but all the other code that other community members implemented. You of course also get other benefits, such as peer review of the code, more ideas (not all bright people work inside one given company), etc.

How the Osmocom GSM stack is funded

Harald Welte — Thu, 15 Jun 2017 22:00:00 GMT

As the topic has been raised on twitter, I thought I might share a bit of insight into the funding of the Osmocom Cellular Infrastructure Projects.

Keep in mind: Osmocom is a much larger umbrella project, and beyond the Networks-side cellular stack is home many different community-based projects around open source mobile communications. All of those have started more or less as just for fun projects, nothing serious, just a hobby [1]

The projects implementing the network-side protocol stacks and network elements of GSM/GPRS/EGPRS/UMTS cellular networks are somewhat the exception to that, as they have evolved to some extent professionalized. We call those projects collectively the Cellular Infrastructure projects inside Osmocom. This post is about that part of Osmocom only

History

From late 2008 through 2009, People like Holger and I were working on bs11-abis and later OpenBSC only in our spare time. The name Osmocom didn't even exist back then. There was a strong technical community with contributions from Sylvain Munaut, Andreas Eversberg, Daniel Willmann, Jan Luebbe and a few others. None of this would have been possible if it wasn't for all the help we got from Dieter Spaar with the BS-11 [2]. We all had our dayjob in other places, and OpenBSC work was really just a hobby. People were working on it, because it was where no FOSS hacker has gone before. It was cool. It was a big and pleasant challenge to enter the closed telecom space as pure autodidacts.

Holger and I were doing freelance contract development work on Open Source projects for many years before. I was mostly doing Linux related contracting, while Holger has been active in all kinds of areas throughout the FOSS software stack.

In 2010, Holger and I saw some first interest by companies into OpenBSC, including Netzing AG and On-Waves ehf. So we were able to spend at least some of our paid time on OpenBSC/Osmocom related contract work, and were thus able to do less other work. We also continued to spend tons of spare time in bringing Osmocom forward. Also, the amount of contract work we did was only a fraction of the many more hours of spare time.

In 2011, Holger and I decided to start the company sysmocom in order to generate more funding for the Osmocom GSM projects by means of financing software development by product sales. So rather than doing freelance work for companies who bought their BTS hardware from other places (and spent huge amounts of cash on that), we decided that we wanted to be a full solution supplier, who can offer a complete product based on all hardware and software required to run small GSM networks.

The only problem is: We still needed an actual BTS for that. Through some reverse engineering of existing products we figured out who one of the ODM suppliers for the hardware + PHY layer was, and decided to develop the OsmoBTS software to do so. We inherited some of the early code from work done by Andreas Eversberg on the jolly/bts branch of OsmocomBB (thanks), but much was missing at the time.

What follows was Holger and me working several years for free [3], without any salary, in order to complete the OsmoBTS software, build an embedded Linux distribution around it based on OE/poky, write documentation, etc. and complete the first sysmocom product: The sysmoBTS 1002

We did that not because we want to get rich, or because we want to run a business. We did it simply because we saw an opportunity to generate funding for the Osmocom projects and make them more sustainable and successful. And because we believe there is a big, gaping, huge vacuum in terms of absence of FOSS in the cellular telecom sphere.

Funding by means of sysmocom product sales

Once we started to sell the sysmoBTS products, we were able to fund Osmocom related development from the profits made on hardware / full-system product sales. Every single unit sold made a big contribution towards funding both the maintenance as well as the ongoing development on new features.

This source of funding continues to be an important factor today.

Funding by means of R&D contracts

The probably best and most welcome method of funding Osmocom related work is by means of R&D projects in which a customer funds our work to extend the Osmocom GSM stack in one particular area where he has a particular need that the existing code cannot fulfill yet.

This kind of project is the ideal match, as it shows where the true strength of FOSS is: Each of those customers did not have to fund the development of a GSM stack from scratch. Rather, they only had to fund those bits that were missing for their particular application.

Our reference for this is and has been On-Waves, who have been funding development of their required features (and bug fixing etc.) since 2010.

We've of course had many other projects from a variety of customers over over the years. Last, but not least, we had a customer who willingly co-funded (together with funds from NLnet foundation and lots of unpaid effort by sysmocom) the 3G/3.5G support in the Osmocom stack.

The problem here is:

we have not been able to secure anywhere nearly as many of those R&D projects within the cellular industry, despite believing we have a very good foundation upon which we can built. I've been writing many exciting technical project proposals
you almost exclusively get funding only for new features. But it's very hard to get funding for the core maintenance work. The bug-fixing, code review, code refactoring, testing, etc.

So as a result, the profit margin you have on selling R&D projects is basically used to (do a bad job of) fund those bits and pieces that nobody wants to pay for.

Funding by means of customer support

There is a way to generate funding for development by providing support services. We've had some success with this, but primarily alongside the actual hardware/system sales - not so much in terms of pure software-only support.

Also, providing support services from a R&D company means:

either you distract your developers by handling support inquiries. This means they will have less time to work on actual code, and likely get side tracked by too many issues that make it hard to focus
or you have to hire separate support staff. This of course means that the size of the support business has to be sufficiently large to not only cover the cots of hiring + training support staff, but also still generate funding for the actual software R&D.

We've tried shortly with the second option, but fallen back to the first for now. There's simply not sufficient user/admin type support business to rectify dedicated staff for that.

Funding by means of cross-subsizing from other business areas

sysmocom also started to do some non-Osmocom projects in order to generate revenue that we can feed again into Osmocom projects. I'm not at liberty to discuss them in detail, but basically we've been doing pretty much anything from

custom embedded Linux board designs
M2M devices with GSM modems
consulting gigs
public tendered research projects

Profits from all those areas went again into Osmocom development.

Last, but not least, we also operate the sysmocom webshop. The profit we make on those products also is again immediately re-invested into Osmocom development.

Funding by grants

We've had some success in securing funding from NLnet Foundation for specific features. While this is useful, the size of their projects grants of up to EUR 30k is not a good fit for the scale of the tasks we have at hand inside Osmocom. You may think that's a considerable amount of money? Well, that translates to 2-3 man-months of work at a bare cost-covering rate. At a team size of 6 developers, you would theoretically have churned through that in two weeks. Also, their focus is (understandably) on Internet and IT security, and not so much cellular communications.

There are of course other options for grants, such as government research grants and the like. However, they require long-term planning, they require you to match (i.e. pay yourself) a significant portion, and basically mandate that you hire one extra person for doing all the required paperwork and reporting. So all in all, not a particularly attractive option for a very small company consisting of die hard engineers.

Funding by more BTS ports

At sysmocom, we've been doing some ports of the OsmoBTS + OsmoPCU software to other hardware, and supporting those other BTS vendors with porting, R&D and support services.

If sysmocom was a classic BTS vendor, we would not help our "competition". However, we are not. sysmocom exists to help Osmocom, and we strongly believe in open systems and architectures, without a single point of failure, a single supplier for any component or any type of vendor lock-in.

So we happily help third parties to get Osmocom running on their hardware, either with a proprietary PHY or with OsmoTRX.

However, we expect that those BTS vendors also understand their responsibility to share the development and maintenance effort of the stack. Preferably by dedicating some of their own staff to work in the Osmocom community. Alternatively, sysmocom can perform that work as paid service. But that's a double-edged sword: We don't want to be a single point of failure.

Osmocom funding outside of sysmocom

Osmocom is of course more than sysmocom. Even for the cellular infrastructure projects inside Osmocom is true: They are true, community-based, open, collaborative development projects. Anyone can contribute.

Over the years, there have been code contributions by e.g. Fairwaves. They, too, build GSM base station hardware and use that as a means to not only recover the R&D on the hardware, but also to contribute to Osmocom. At some point a few years ago, there was a lot of work from them in the area of OsmoTRX, OsmoBTS and OsmoPCU. Unfortunately, in more recent years, they have not been able to keep up the level of contributions.

There are other companies engaged in activities with and around Osmcoom. There's Rhizomatica, an NGO helping indigenous communities to run their own cellular networks. They have been funding some of our efforts, but being an NGO helping rural regions in developing countries, they of course also don't have the deep pockets. Ideally, we'd want to be the ones contributing to them, not the other way around.

State of funding

We're making some progress in securing funding from players we cannot name [4] during recent years. We're also making occasional progress in convincing BTS suppliers to chip in their share. Unfortunately there are more who don't live up to their responsibility than those who do. I might start calling them out by name one day. The wider community and the public actually deserves to know who plays by FOSS rules and who doesn't. That's not shaming, it's just stating bare facts.

Which brings us to:

sysmocom is in an office that's actually too small for the team, equipment and stock. But we certainly cannot afford more space.
we cannot pay our employees what they could earn working at similar positions in other companies. So working at sysmocom requires dedication to the cause :)
Holger and I have invested way more time than we have ever paid us, even more so considering the opportunity cost of what we would have earned if we'd continued our freelance Open Source hacker path
we're [just barely] managing to pay for 6 developers dedicated to Osmocom development on our payroll based on the various funding sources indicated above

Nevertheless, I doubt that any such a small team has ever implemented an end-to-end GSM/GPRS/EGPRS network from RAN to Core at comparative feature set. My deepest respects to everyone involved. The big task now is to make it sustainable.

Summary

So as you can see, there's quite a bit of funding around. However, it always falls short of what's needed to implement all parts properly, and even not quite sufficient to keep maintaining the status quo in a proper and tested way. That can often be frustrating (mostly to us but sometimes also to users who run into regressions and oter bugs). There's so much more potential. So many things we wanted to add or clean up for a long time, but too little people interested in joining in, helping out - financially or by writing code.

On thing that is often a challenge when dealing with traditional customers: We are not developing a product and then selling a ready-made product. In fact, in FOSS this would be more or less suicidal: We'd have to invest man-years upfront, but then once it is finished, everyone can use it without having to partake in that investment.

So instead, the FOSS model requires the customers/users to chip in early during the R&D phase, in order to then subsequently harvest the fruits of that.

I think the lack of a FOSS mindset across the cellular / telecom industry is the biggest constraining factor here. I've seen that some 20-15 years ago in the Linux world. Trust me, it takes a lot of dedication to the cause to endure this lack of comprehension so many years later.

Playing back GSM RTP streams, RTP-HR bugs

Harald Welte — Sun, 28 May 2017 22:00:00 GMT

Chapter 0: Problem Statement

In an all-IP GSM network, where we use Abis, A and other interfaces within the cellular network over IP transport, the audio of voice calls is transported inside RTP frames. The codec payload in those RTP frames is the actual codec frame of the respective cellular voice codec. In GSM, there are four relevant codecs: FR, HR, EFR and AMR.

Every so often during the (meanwhile many years of ) development of Osmocom cellular infrastructure software it would have been useful to be able to quickly play back the audio for analysis of given issues.

However, until now we didn't have that capability. The reason is relatively simple: In Osmocom, we genally don't do transcoding but simply pass the voice codec frames from left to right. They're only transcoded inside the phones or inside some external media gateway (in case of larger networks).

Chapter 1: GSM Audio Pocket Knife

Back in 2010, when we were very actively working on OsmocomBB, the telephone-side GSM protocol stack implementation, Sylvain Munaut wrote the GSM Audio Pocket Knife (gapk) in order to be able to convert between different formats (representations) of codec frames. In cellular communcations, everyoe is coming up with their own representation for the codec frames: The way they look on E1 as a TRAU frame is completely different from how RTP payload looks like, or what the TI Calypso DSP uses internally, or what a GSM Tester like the Racal 61x3 uses. The differences are mostly about data types used, bit-endinanness as well as padding and headers. And of course those different formats exist for each of the four codecs :/

In 2013 I first added simplistic RTP support for FR-GSM to gapk, which was sufficient for my debugging needs back then. Still, you had to save the decoded PCM output to a file and play that back, or use a pipe into aplay.

Last week, I picked up this subject again and added a long series of patches to gapk:

support for variable-length codec frames (required for AMR support)
support for AMR codec encode/decode using libopencore-amrnb
support of all known RTP payload formats for all four codecs
support for direct live playback to a sound card via ALSA

All of the above can now be combined to make GAPK bind to a specified UDP port and play back the RTP codec frames that anyone sends to that port using a command like this:

$ gapk -I 0.0.0.0/30000 -f rtp-amr -A default -g rawpcm-s16le

I've also merged a chance to OsmoBSC/OsmoNITB which allows the administrator to re-direct the voice of any active voice channel towards a user-specified IP address and port. Using that you can simply disconnect the voice stream from its normal destination and play back the audio via your sound card.

Chapter 2: Bugs in OsmoBTS GSM-HR

While going through the exercise of implementing the above extension to gapk, I had lots of trouble to get it to work for GSM-HR.

After some more digging, it seems there are two conflicting specification on how to format the RTP payload for half-rate GSM:

ETSI TS 101 318 from 1998
IETF RFC 5993 from 2010

In Osmocom, we claim to implement RFC5993, but it turned out that (at least) osmo-bts-sysmo (for sysmoBTS) was actually implementing the ETSI format instead.

And even worse, osmo-bts-sysmo gets event the ETSI format wrong. Each of the codec parameters (which are unaligned bit-fields) are in the wrong bit-endianness :(

Both the above were coincidentially also discovered by Sylvain Munaut during operating of the 32C3 GSM network in December 2015 and resulted the two following "work around" patches: * HACK for HR * HACK: Fix the bit order in HR frames

Those merely worked around those issues in the rtp_proxy of OsmoNITB, rather than addressing the real issue. That's ok, they were "quick" hacks to get something working at all during a four-day conference. I'm now working on "real" fixes in osmo-bts-sysmo. The devil is of course in the details, when people upgrade one BTS but not the other and want to inter-operate, ...

It yet remains to be investigated how osmo-bts-trx and other osmo-bts ports behave in this regard.

Chapter 3: Conclusions

Most definitely it is once again a very clear sign that more testing is required. It's tricky to see even wih osmo-gsm-tester, as GSM-HR works between two phones or even two instances of osmo-bts-sysmo, as both sides of the implementation have the same (wrong) understanding of the spec.

Given that we can only catch this kind of bug together with the hardware (the DSP runs the PHY code), pure unit tests wouldn't catch it. And the end-to-end test is also not very well suited to it. It seems to call for something in betewen. Something like an A-bis interface level test.

We need more (automatic) testing. I cannot say that often enough. The big challenge is how to convince contributors and customers that they should invest their time and money there, rather than yet-another (not automatically tested) feature?

Power-cycling a USB port should be simple, right?

Harald Welte — Tue, 23 May 2017 22:00:00 GMT

Every so often I happen to be involved in designing electronics equipment that's supposed to run reliably remotely in inaccessible locations,without any ability for "remote hands" to perform things like power-cycling or the like. I'm talking about really remote locations, possible with no but limited back-haul, and a very high cost of ever sending somebody there for remote maintenance.

Given that a lot of computer peripherals (chips, modules, ...) use USB these days, this is often some kind of an embedded ARM (rarely x86) SoM or SBC, which is hooked up to a custom board that contains a USB hub chip as well as a line of peripherals.

One of the most important lectures I've learned from experience is: Never trust reset signals / lines, always include power-switching capability. There are many chips and electronics modules available on the market that have either no RESET, or even might claim to have a hardware RESET line which you later (painfully) discover just to be a GPIO polled by software which can get stuck, and hence no way to really hard-reset the given component.

In the case of a USB-attached device (even though the USB might only exist on a circuit board between two ICs), this is typically rather easy: The USB hub is generally capable of switching the power of its downstream ports. Many cheap USB hubs don't implement this at all, or implement only ganged switching, but if you carefully select your USB hub (or in the case of a custom PCB), you can make sure that the given USB hub supports individual port power switching.

Now the next step is how to actually use this from your (embedded) Linux system. It turns out to be harder than expected. After all, we're talking about a standard feature that's present in the USB specifications since USB 1.x in the late 1990ies. So the expectation is that it should be straight-forward to do with any decent operating system.

I don't know how it's on other operating systems, but on Linux I couldn't really find a proper way how to do this in a clean way. For more details, please read my post to the linux-usb mailing list.

Why am I running into this now? Is it such a strange idea? I mean, power-cycling a device should be the most simple and straight-forward thing to do in order to recover from any kind of "stuck state" or other related issue. Logical enabling/disabling of the port, resetting the USB device via USB protocol, etc. are all just "soft" forms of a reset which at best help with USB related issues, but not with any other part of a USB device.

And in the case of e.g. an USB-attached cellular modem, we're actually talking about a multi-processor system with multiple built-in micro-controllers, at least one DSP, an ARM core that might run another Linux itself (to implement the USB gadget), ... - certainly enough complex software that you would want to be able to power-cycle it...

I'm curious what the response of the Linux USB gurus is.

OsmoDevCon 2017 Review

Harald Welte — Tue, 02 May 2017 22:00:00 GMT

After the public user-oriented OsmoCon 2017, we also recently had the 6th incarnation of our annual contributors-only Osmocom Developer Conference: The OsmoDevCon 2017.

This is a much smaller group, typically about 20 people, and is limited to actual developers who have a past record of contributing to any of the many Osmocom projects.

We had a large number of presentation and discussions. In fact, so large that the schedule of talks extended from 10am to midnight on some days. While this is great, it also means that there was definitely too little time for more informal conversations, chatting or even actual work on code.

We also have such a wide range of topics and scope inside Osmocom, that the traditional ad-hoch scheduling approach no longer seems to be working as it used to. Not everyone is interested in (or has time for) all the topics, so we should group them according to their topic/subject on a given day or half-day. This will enable people to attend only those days that are relevant to them, and spend the remaining day in an adjacent room hacking away on code.

It's sad that we only have OsmoDevCon once per year. Maybe that's actually also something to think about. Rather than having 4 days once per year, maybe have two weekends per year.

Always in motion the future is.

Overhyped Docker

Harald Welte — Tue, 02 May 2017 22:00:00 GMT

Overhyped Docker missing the most basic features

I've always been extremely skeptical of suddenly emerging over-hyped technologies, particularly if they advertise to solve problems by adding yet another layer to systems that are already sufficiently complex themselves.

There are of course many issues with containers, ranging from replicated system libraries and the basic underlying statement that you're giving up on the system packet manager to properly deal with dependencies.

I'm also highly skeptical of FOSS projects that are primarily driven by one (VC funded?) company. Especially if their offering includes a so-called cloud service which they can stop to operate at any given point in time, or (more realistically) first get everybody to use and then start charging for.

But well, despite all the bad things I read about it over the years, on one day in May 2017 I finally thought let's give it a try. My problem to solve as a test balloon is fairly simple.

My basic use case

The plan is to start OsmoSTP, the m3ua-testtool and the sua-testtool, which both connect to OsmoSTP. By running this setup inside containers and inside an internal network, we could then execute the entire testsuite e.g. during jenkins test without having IP address or port number conflicts. It could even run multiple times in parallel on one buildhost, verifying different patches as part of the continuous integration setup.

This application is not so complex. All it needs is three containers, an internal network and some connections in between. Should be a piece of cake, right?

But enter the world of buzzword-fueled web-4000.0 software-defined virtualised and orchestrated container NFW + SDN vodoo: It turns out to be impossible, at least not with the preferred tools they advertise.

Dockerfiles

The part that worked relatively easily was writing a few Dockerfiles to build the actual containers. All based on debian:jessie from the library.

As m3ua-testsuite is written in guile, and needs to build some guile plugin/extension, I had to actually include guile-2.0-dev and other packages in the container, making it a bit bloated.

I couldn't immediately find a nice example Dockerfile recipe that would allow me to build stuff from source outside of the container, and then install the resulting binaries into the container. This seems to be a somewhat weak spot, where more support/infrastructure would be helpful. I guess the idea is that you simply install applications via package feeds and apt-get. But I digress.

So after some tinkering, I ended up with three docker containers:

one running OsmoSTP
one running m3ua-testtool
one running sua-testtool

I also managed to create an internal bridged network between the containers, so the containers could talk to one another.

However, I have to manually start each of the containers with ugly long command line arguments, such as docker run --network sigtran --ip 172.18.0.200 -it osmo-stp-master. This is of course sub-optimal, and what Docker Services + Stacks should resolve.

Services + Stacks

The idea seems good: A service defines how a given container is run, and a stack defines multiple containers and their relation to each other. So it should be simple to define a stack with three services, right?

Well, it turns out that it is not. Docker documents that you can configure a static ipv4_address [1] for each service/container, but it seems related configuration statements are simply silently ignored/discarded [2], [3], [4].

This seems to be related that for some strange reason stacks can (at least in later versions of docker) only use overlay type networks, rather than the much simpler bridge networks. And while bridge networks appear to support static IP address allocations, overlay apparently doesn't.

I still have a hard time grasping that something that considers itself a serious product for production use (by a company with estimated value over a billion USD, not by a few hobbyists) that has no support for running containers on static IP addresses. that. How many applications out there have I seen that require static IP address configuration? How much simpler do setups get, if you don't have to rely on things like dynamic DNS updates (or DNS availability at all)?

So I'm stuck with having to manually configure the network between my containers, and manually starting them by clumsy shell scripts, rather than having a proper abstraction for all of that. Well done :/

Exposing Ports

Unrelated to all of the above: If you run some software inside containers, you will pretty soon want to expose some network services from containers. This should also be the most basic task on the planet.

However, it seems that the creators of docker live in the early 1980ies, where only TCP and UDP transport protocols existed. They seem to have missed that by the late 1990ies to early 2000s, protocols like SCTP or DCCP were invented.

But yet, in 2017, Docker chooses to

blindly assume TCP in https://docs.docker.com/engine/reference/builder/#expose without even mentioning it (or designing the syntax to accept any specification of the protocol)
design a syntax (/tcp) in the command-line parsing (see https://docs.docker.com/engine/reference/run/#expose-incoming-ports), but then only parse tcp and udp, despite people requesting support for other protocols like SCTP as early as three years ago

Now some of the readers may think 'who uses SCTP anyway'. I will give you a straight answer: Everyone who has a mobile phone uses SCTP. This is due to the fact that pretty much all the connections inside cellular networks (at least for 3G/4G networks, and in reality also for many 2G networks) are using SCTP as underlying transport protocol, from the radio access network into the core network. So every time you switch your phone on, or do anything with it, you are using SCTP. Not on your phone itself, but by all the systems that form the network that you're using. And with the drive to C-RAN, NFV, SDN and all the other buzzwords also appearing in the Cellular Telecom field, people should actually worry about it, if they want to be a part of the software stack that is used in future cellular telecom systems.

Summary

After spending the better part of a day to do something that seemed like the most basic use case for running three networked containers using Docker, I'm back to step one: Most likely inventing some custom scripts based on unshare to run my three test programs in a separate network namespace for isolated execution of test suite execution as part of a Jenkins CI setup :/

It's also clear that Docker apparently don't care much about playing a role in the Cellular Telecom world, which is increasingly moving away from proprietary and hardware-based systems (like STPs) to virtualised, software-based systems.

Book on Practical GPL Compliance

Harald Welte — Mon, 01 May 2017 22:00:00 GMT

My former gpl-violations.org colleague Armijn Hemel and Shane Coughlan (former coordinator of the FSFE Legal Network) have written a book on practical GPL compliance issues.

I've read through it (in the bath tub of course, what better place to read technical literature), and I can agree wholeheartedly with its contents. For those who have been involved in GPL compliance engineering there shouldn't be much new - but for the vast majority of developers out there who have had little exposure to the bread-and-butter work of providing complete an corresponding source code, it makes an excellent introductory text.

The book focuses on compliance with GPLv2, which is probably not too surprising given that it's published by the Linux foundation, and Linux being GPLv2.

You can download an electronic copy of the book from https://www.linuxfoundation.org/news-media/research/practical-gpl-compliance

Given the subject matter is Free Software, and the book is written by long-time community members, I cannot help to notice a bit of a surprise about the fact that the book is released in classic copyright under All rights reserved with no freedom to the user.

Considering the sensitive legal topics touched, I can understand the possible motivation by the authors to not permit derivative works. But then, there still are licenses such as CC-BY-ND which prevent derivative works but still permit users to make and distribute copies of the work itself. I've made that recommendation / request to Shane, let's see if they can arrange for some more freedom for their readers.

OsmoCon 2017 Review

Harald Welte — Sun, 30 Apr 2017 22:00:00 GMT

It's already one week past the event, so I really have to sit down and write some rewview on the first public Osmocom Conference ever: OsmoCon 2017.

The event was a huge success, by all accounts.

We've not only been sold out, but we also had to turn down some last minute registrations due to the venue being beyond capacity (60 seats). People traveled from Japan, India, the US, Mexico and many other places to attend.
We've had an amazing audience ranging from commercial operators to community cellular operators to professional developers doing work relate to osmocom, academia, IT security crowds and last but not least enthusiasts/hobbyists, with whom the project[s] started.
I've received exclusively positive feedback from many attendees
We've had a great programme. Some part of it was of introductory nature and probably not too interesting if you've been in Osmocom for a few years. However, the work on 3G as well as the current roadmap was probably not as widely known yet. Also, I really loved to see Roch's talk about Running a commercial cellular network with Osmocom software as well as the talk on Facebook's OpenCellular BTS hardware and the Community Cellular Manager.
We have very professional live streaming + video recordings courtesy of the C3VOC team. Thanks a lot for your support and for having the video recordings of all talks online already at the next day after the event.

We also received some requests for improvements, many of which we will hopefully consider before the next Osmocom Conference:

have a multiple day event. Particularly if you're traveling long-distance, it is a lot of overhead for a single-day event. We of course fully understand that. On the other hand, it was the first Osmocom Conference, and hence it was a test balloon where it was initially unclear if we'll be able to get a reasonable number of attendees interested at all, or not. And organizing an event with venue and talks for multiple days if in the end only 10 people attend would have been a lot of effort and financial risk. But now that we know there are interested folks, we can definitely think of a multiple day event next time
Signs indicating venue details on the last meters. I agree, this cold have been better. The address of the venue was published, but we could have had some signs/posters at the door pointing you to the right meeting room inside the venue. Sorry for that.
Better internet connectivity. This is a double-edged sword. Of course we want our audience to be primarily focused on the talks and not distracted :P I would hope that most people are able to survive a one day event without good connectivity, but for sure we will have to improve in case of a multiple-day event in the future

In terms of my requests to the attendees, I only have one

Participate in the discussions on the schedule/programme while it is still possible to influence it. When we started to put together the programme, I posted about it on the openbsc mailing list and invited feedback. Still, most people seem to have missed the time window during which talks could have been submitted and the schedule still influenced before finalizing it
Register in time. We have had almost no registrations until about two weeks ahead of the event (and I was considering to cancel it), and then suddenly were sold out in the week ahead of the event. We've had people who first booked their tickets, only to learn that the tickets were sold out. I guess we will introduce early bird pricing and add a very expensive last minute ticket option next year in order to increase motivation to register early and thus give us flexibility regarding venue planning.

Thanks again to everyone involved in OsmoCon 2017!

Ok, now, all of you who missed the event: Go to https://media.ccc.de/c/osmocon17 and check out the recordings. Have fun!

Things you find when using SCTP on Linux

Harald Welte — Sun, 16 Apr 2017 22:00:00 GMT

Observations on SCTP and Linux

When I was still doing Linux kernel work with netfilter/iptables in the early 2000's, I was somebody who actually regularly had a look at the new RFCs that came out. So I saw the SCTP RFCs, SIGTRAN RFCs, SIP and RTP, etc. all released during those years. I was quite happy to see that for new protocols like SCTP and later DCCP, Linux quickly received a mainline implementation.

Now most people won't have used SCTP so far, but it is a protocol used as transport layer in a lot of telecom protocols for more than a decade now. Virtually all protocols that have traditionally been spoken over time-division multiplex E1/T1 links have been migrated over to SCTP based protocol stackings.

Working on various Open Source telecom related projects, i of course come into contact with SCTP every so often. Particularly some years back when implementing the Erlang SIGTAN code in erlang/osmo_ss7 and most recently now with the introduction of libosmo-sigtran with its OsmoSTP, both part of the libosmo-sccp repository.

I've also hard to work with various proprietary telecom equipment over the years. Whether that's some eNodeB hardware from a large brand telecom supplier, or whether it's a MSC of some other vendor. And they all had one thing in common: Nobody seemed to use the Linux kernel SCTP code. They all used proprietary implementations in userspace, using RAW sockets on the kernel interface.

I always found this quite odd, knowing that this is the route that you have to take on proprietary OSs without native SCTP support, such as Windows. But on Linux? Why? Based on rumors, people find the Linux SCTP implementation not mature enough, but hard evidence is hard to come by.

As much as it pains me to say this, the kind of Linux SCTP bugs I have seen within the scope of our work on Osmocom seem to hint that there is at least some truth to this (see e.g. https://bugzilla.redhat.com/show_bug.cgi?id=1308360 or https://bugzilla.redhat.com/show_bug.cgi?id=1308362).

Sure, software always has bugs and will have bugs. But we at Osmocom are 10-15 years "late" with our implementations of higher-layer protocols compared to what the mainstream telecom industry does. So if we find something, and we find it even already during R&D of some userspace code, not even under load or in production, then that seems a bit unsettling.

One would have expected, with all their market power and plenty of Linux-based devices in the telecom sphere, why did none of those large telecom suppliers invest in improving the mainline Linux SCTP code? I mean, they all use UDP and TCP of the kernel, so it works for most of the other network protocols in the kernel, but why not for SCTP? I guess it comes back to the fundamental lack of understanding how open source development works. That it is something that the given industry/user base must invest in jointly.

The leatest discovered bug

During the last months, I have been implementing SCCP, SUA, M3UA and OsmoSTP (A Signal Transfer Point). They were required for an effort to add 3GPP compliant A-over-IP to OsmoBSC and OsmoMSC.

For quite some time I was seeing some erratic behavior when at some point the STP would not receive/process a given message sent by one of the clients (ASPs) connected. I tried to ignore the problem initially until the code matured more and more, but the problems remained.

It became even more obvious when using Michael Tuexen's m3ua-testtool, where sometimes even the most basic test cases consisting of sending + receiving a single pair of messages like ASPUP -> ASPUP_ACK was failing. And when the test case was re-tried, the problem often disappeared.

Also, whenever I tried to observe what was happening by meas of strace, the problem would disappear completely and never re-appear until strace was detached.

Of course, given that I've written several thousands of lines of new code, it was clear to me that the bug must be in my code. Yesterday I was finally prepare to accept that it might actually be a Linux SCTP bug. Not being able to reproduce that problem on a FreeBSD VM also pointed clearly into this direction.

Now I could simply have collected some information and filed a bug report (which some kernel hackers at RedHat have thankfully invited me to do!), but I thought my use case was too complex. You would have to compile a dozen of different Osmocom libraries, configure the STP, run the scheme-language m3ua-testtool in guile, etc. - I guess nobody would have bothered to go that far.

So today I tried to implement a test case that reproduced the problem in plain C, without any external dependencies. And for many hours, I couldn't make the bug to show up. I tried to be as close as possible to what was happening in OsmoSTP: I used non-blocking mode on client and server, used the SCTP_NODELAY socket option, used the sctp_rcvmsg() library wrapper to receive events, but the bug was not reproducible.

Some hours later, it became clear that there was one setsockopt() in OsmoSTP (actually, libosmo-netif) which enabled all existing SCTP events. I did this at the time to make sure OsmoSTP has the maximum insight possible into what's happening on the SCTP transport layer, such as address fail-overs and the like.

As it turned out, adding that setsockopt for SCTP_FLAGS to my test code made the problem reproducible. After playing around which of the flags, it seems that enabling the SENDER_DRY_EVENT flag makes the bug appear.

You can find my detailed report about this issue in https://bugzilla.redhat.com/show_bug.cgi?id=1442784 and a program to reproduce the issue at http://people.osmocom.org/laforge/sctp-nonblock/sctp-dry-event.c

Inside the Osmocom world, luckily we can live without the SENDER_DRY_EVENT and a corresponding work-around has been submitted and merged as https://gerrit.osmocom.org/#/c/2386/

With that work-around in place, suddenly all the m3ua-testtool and sua-testtool test cases are reliably green (PASSED) and OsmoSTP works more smoothly, too.

What do we learn from this?

Free Software in the Telecom sphere is getting too little attention. This is true even those small portions of telecom relevant protocols that ended up in the kernel like SCTP or more recently the GTP module I co-authored. They are getting too little attention in development, even more lack of attention in maintenance, and people seem to focus more on not using it, rather than fixing and maintaining what is there.

It makes me really sad to see this. Telecoms is such a massive industry, with billions upon billions of revenue for the classic telecom equipment vendors. Surely, they would be able to co-invest in some basic infrastructure like proper and reliable testing / continuous integration for SCTP. More recently, we see millions and more millions of VC cash burned by buzzword-flinging companies doing "NFV" and "SDN". But then rather reimplement network stacks in userspace than to fix, complete and test those little telecom infrastructure components which we have so far, like the SCTP protocol :(

Where are the contributions to open source telecom parts from Ericsson, Nokia (former NSN), Huawei and the like? I'm not even dreaming about the actual applications / network elements, but merely the maintenance of something as basic as SCTP. To be fair, Motorola was involved early on in the Linux SCTP code, and Huawei contributed a long series of fixes in 2013/2014. But that's not the kind of long-term maintenance contribution that one would normally expect from the primary interest group in SCTP.

Finally, let me thank to the Linux SCTP maintainers. I'm not complaining about them! They're doing a great job, given the arcane code base and the fact that they are not working for a company that has SCTP based products as their core business. I'm sure the would love more support and contributions from the Telecom world, too.

SIGTRAN/SS7 stack in libosmo-sigtran merged to master

Harald Welte — Sun, 09 Apr 2017 22:00:00 GMT

As I blogged in my blog post in Fabruary, I was working towards a more fully-featured SIGTRAN stack in the Osmocom (C-language) universe.

The trigger for this is the support of 3GPP compliant AoIP (with a BSSAP/SCCP/M3UA/SCTP protocol stacking), but it is of much more general nature.

The code has finally matured in my development branch(es) and is now ready for mainline inclusion. It's a series of about 77 (!) patches, some of which already are the squashed results of many more incremental development steps.

The result is as follows:

General SS7 core functions maintaining links, linksets and routes
xUA functionality for the various User Adaptations (currently SUA and M3UA supported)
- MTP User SAP according to ITU-T Q.701 (using osmo_prim)
- management of application servers (AS)
- management of application server processes (ASP)
- ASP-SM and ASP-TM state machine for ASP, AS-State Machine (using osmo_fsm)
- server (SG) and client (ASP) side implementation
- validated against ETSI TS 102 381 (by means of Michael Tuexen's m3ua-testtool)
- support for dynamic registration via RKM (routing key management)
- osmo-stp binary that can be used as Signal Transfer Point, with the usual "Cisco-style" command-line interface that all Osmocom telecom software has.
SCCP implementation, with strong focus on Connection Oriented SCCP (as that's what the A interface uses).
- osmo_fsm based state machine for SCCP connection, both incoming and outgoing
- SCCP User SAP according to ITU-T Q.711 (osmo_prim based)
- Interfaces with underlying SS7 stack via MTP User SAP (osmo_prim based)
- Support for SCCP Class 0 (unit data) and Class 2 (connection oriented)
- All SCCP + SUA Address formats (Global Title, SSN, PC, IPv4 Address)
- SCCP and SUA share one implementation, where SCCP messages are transcoded into SUA before processing, and re-encoded into SCCP after processing, as needed.

I have already done experimental OsmoMSC and OsmoHNB-GW over to libosmo-sigtran. They're now all just M3UA clients (ASPs) which connect to osmo-stp to exchange SCCP messages back and for the between them.

What's next on the agenda is to

finish my incomplete hacks to introduce IPA/SCCPlite as an alternative to SUA and M3UA (for backwards compatibility)
port over OsmoBSC to the SCCP User SAP of libosmo-sigtran
- validate with SSCPlite lower layer against existing SCCPlite MSCs
implement BSSAP / A-interface procedures in OsmoMSC, on top of the SCCP-User SAP.

If those steps are complete, we will have a single OsmoMSC that can talk both IuCS to the HNB-GW (or RNCs) for 3G/3.5G as well as AoIP towards OsmoBSC. We will then have fully SIGTRAN-enabled the full Osmocom stack, and are all on track to bury the OsmoNITB that was devoid of such interfaces.

If any reader is interested in interoperability testing with other implementations, either on M3UA or on SCCP or even on A or Iu interface level, please contact me by e-mail.

OsmoCon 2017 Updates: Travel Grants and Schedule

Harald Welte — Sun, 26 Mar 2017 22:00:00 GMT

April 21st is approaching fast, so here some updates. I'm particularly happy that we now have travel grants available. So if the travel expenses were preventing you from attending so far: This excuse is no longer valid!

Get your ticket now, before it is too late. There's a limited number of seats available.

OsmoCon 2017 Schedule

The list of talks for OsmoCon 2017 has been available for quite some weeks, but today we finally published the first actual schedule.

As you can see, the day is fully packed with talks about Osmocom cellular infrastructure projects. We had to cut some talk slots short (30min instead of 45min), but I'm confident that it is good to cover a wider range of topics, while at the same time avoiding fragmenting the audience with multiple tracks.

OsmoCon 2017 Travel Grants

We are happy to announce that we have received donations to permit for providing travel grants!

This means that any attendee who is otherwise not able to cover their travel to OsmoCon 2017 (e.g. because their interest in Osmocom is not related to their work, or because their employer doesn't pay the travel expenses) can now apply for such a travel grant.

For more details see OsmoCon 2017 Travel Grants and/or contact osmocon2017@sysmocom.de.

Upcoming v3 of Open Hardware miniPCIe WWAN modem USB breakout board

Harald Welte — Thu, 23 Mar 2017 23:00:00 GMT

Back in October 2016 I designed a small open hardware breakout board for WWAN modems in mPCIe form-factor. I was thinking some other people might be interested in this, and indeed, the first manufacturing batch is already sold out by now.

Instead of ordering more of the old (v2) design, I decided to do some improvements in the next version:

add mounting holes so the PCB can be mounted via M3 screws
add U.FL and SMA sockets, so the modems are connected via a short U.FL to U.FL cable, and external antennas or other RF components can be attached via SMA. This provides strain relief for the external antenna or cabling and avoids tearing off any of the current loose U.FL to SMA pigtails
flip the SIM slot to the top side of the PCB, so it can be accessed even after mounting the board to some base plate or enclosure via the mounting holes
more meaningful labeling of the silk screen, including the purpose of the jumpers and the input voltage.

A software rendering of the resulting v3 PCB design files that I just sent for production looks like this:

Like before, the design of the board (including schematics and PCB layout design files) is available as open hardware under CC-BY-SA license terms. For more information see http://osmocom.org/projects/mpcie-breakout/wiki

It will take some expected three weeks until I'll see the first assembled boards.

I'm also planning to do a M.2 / NGFF version of it, but haven't found the time to get around doing it so far.

Osmocom - personal thoughts

Harald Welte — Tue, 21 Mar 2017 18:00:00 GMT

As I just wrote in my post about TelcoSecDay, I sometimes worry about the choices I made with Osmocom, particularly when I see all the great stuff people doing in fields that I previously was working in, such as applied IT security as well as Linux Kernel development.

History

When people like Dieter, Holger and I started to play with what later became OpenBSC, it was just for fun. A challenge to master. A closed world to break open and which to attack with the tools, the mindset and the values that we brought with us.

Later, Holger and I started to do freelance development for commercial users of Osmocom (initially basically only OpenBSC, but then OsmoSGSN, OsmoBSC, OsmoBTS, OsmoPCU and all the other bits on the infrastructure side). This lead to the creation of sysmocom in 2011, and ever since we are trying to use revenue from hardware sales as well as development contracts to subsidize and grow the Osmocom projects. We're investing most of our earnings directly into more staff that in turn works on Osmocom related projects.

In order to do this, I basically gave up my previous career[s] in IT security and Linux kernel development (as well as put things like gpl-violations.org on hold). This is a big price to pay for crating more FOSS in the mobile communications world, and sometimes I'm a bit melancholic about the "old days" before.

Financial wealth is clearly not my primary motivation, but let me be honest: I could have easily earned a shitload of money continuing to do freelance Linux kernel development, IT security or related consulting. There's a lot of demand for related skills, particularly with some experience and reputation attached. But I decided against it, and worked several years without a salary (or almost none) on Osmocom related stuff [as did Holger].

But then, even with all the sacrifices made, and the amount of revenue we can direct from sysmocom into Osmocom development: The complexity of cellular infrastructure vs. the amount of funding and resources is always only a fraction of what one would normally want to have to do a proper implementation. So it's constant resource shortage, combined with lots of unpaid work on those areas that are on the immediate short-term feature list of customers, and that nobody else in the community feels like he wants to work on. And that can be a bit frustrating at times.

Is it worth it?

So after 7 years of OpenBSC, OsmocomBB and all the related projects, I'm sometimes asking myself whether it has been worth the effort, and whether it was the right choice.

It was right from the point that cellular technology is still an area that's obscure and unknown to many, and that has very little FOSS (though Improving!). At the same time, cellular networks are becoming more and more essential to many users and applications. So on an abstract level, I think that every step in the direction of FOSS for cellular is as urgently needed as before, and we have had quite some success in implementing many different protocols and network elements. Unfortunately, in most cases incompletely, as the amount of funding and/or resources were always extremely limited.

Satisfaction/Happiness

On the other hand, when it comes to metrics such as personal satisfaction or professional pride, I'm not very happy or satisfied. The community remains small, the commercial interest remains limited, and as opposed to the Linux world, most players have a complete lack of understanding that FOSS is not a one-way road, but that it is important for all stakeholders to contribute to the development in terms of development resources.

Project success?

I think a collaborative development project (which to me is what FOSS is about) is only then truly successful, if its success is not related to a single individual, a single small group of individuals or a single entity (company). And no matter how much I would like the above to be the case, it is not true for the Osmocom cellular infrastructure projects. Take away Holger and me, or take away sysmocom, and I think it would be pretty much dead. And I don't think I'm exaggerating here. This makes me sad, and after all these years, and after knowing quite a number of commercial players using our software, I would have hoped that the project rests on many more shoulders by now.

This is not to belittle the efforts of all the people contributing to it, whether the team of developers at sysmocom, whether those in the community that still work on it 'just for fun', or whether those commercial users that contract sysmocom for some of the work we do. Also, there are known and unknown donors/funders, like the NLnet foundation for some parts of the work. Thanks to all of you, and clearly we wouldn't be where we are now without all of that!

But I feel it's not sufficient for the overall scope, and it's not [yet] sustainable at this point. We need more support from all sides, particularly those not currently contributing. From vendors of BTSs and related equipment that use Osmocom components. From operators that use it. From individuals. From academia.

Yes, we're making progress. I'm happy about new developments like the Iu and Iuh support, the OsmoHLR/VLR split and 2G/3G authentication that Neels just blogged about. And there's progress on the SIMtrace2 firmware with card emulation and MITM, just as well as there's progress on libosmo-sigtran (with a more complete SUA, M3UA and connection-oriented SCCP stack), etc.

But there are too little people working on this, and those people are mostly coming from one particular corner, while most of the [commercial] users do not contribute the way you would expect them to contribute in collaborative FOSS projects. You can argue that most people in the Linux world also don't contribute, but then the large commercial beneficiaries (like the chipset and hardware makers) mostly do, as are the large commercial users.

All in all, I have the feeling that Osmocom is as important as it ever was, but it's not grown up yet to really walk on its own feet. It may be able to crawl, though ;)

So for now, don't panic. I'm not suffering from burn-out, mid-life crisis and I don't plan on any big changes of where I put my energy: It will continue to be Osmocom. But I also think we have to have a more open discussion with everyone on how to move beyond the current situation. There's no point in staying quiet about it, or to claim that everything is fine the way it is. We need more commitment. Not from the people already actively involved, but from those who are not [yet].

If that doesn't happen in the next let's say 1-2 years, I think it's fair that I might seriously re-consider in which field and in which way I'd like to dedicate my [I would think considerable] productive energy and focus.

Returning from TelcoSecDay 2017 / General Musings

Harald Welte — Tue, 21 Mar 2017 17:00:00 GMT

I'm just on my way back from the Telecom Security Day 2017 <https://www.troopers.de/troopers17/telco-sec-day/>, which is an invitation-only event about telecom security issues hosted by ERNW back-to-back with their Troopers 2017 <https://www.troopers.de/troopers17/> conference.

I've been presenting at TelcoSecDay in previous years and hence was again invited to join (as attendee). The event has really gained quite some traction. Where early on you could find lots of IT security / hacker crowds, the number of participants from the operator (and to smaller extent also equipment maker) industry has been growing.

The quality of talks was great, and I enjoyed meeting various familiar faces. It's just a pity that it's only a single day - plus I had to head back to Berlin still today so I had to skip the dinner + social event.

When attending events like this, and seeing the interesting hacks that people are working on, it pains me a bit that I haven't really been doing much security work in recent years. netfilter/iptables was at least somewhat security related. My work on OpenPCD / librfid was clearly RFID security oriented, as was the work on airprobe, OsmocomTETRA, or even the EasyCard payment system hack

I have the same feeling when attending Linux kernel development related events. I have very fond memories of working in both fields, and it was a lot of fun. Also, to be honest, I believe that the work in Linux kernel land and the general IT security research was/is appreciated much more than the endless months and years I'm now spending my time with improving and extending the Osmocom cellular infrastructure stack.

Beyond the appreciation, it's also the fact that both the IT security and the Linux kernel communities are much larger. There are more people to learn from and learn with, to engage in discussions and ping-pong ideas. In Osmocom, the community is too small (and I have the feeling, it's actually shrinking), and in many areas it rather seems like I am the "ultimate resource" to ask, whether about 3GPP specs or about Osmocom code structure. What I'm missing is the feeling of being part of a bigger community. So in essence, my current role in the "Open Source Cellular" corner can be a very lonely one.

But hey, I don't want to sound more depressed than I am, this was supposed to be a post about TelcoSecDay. It just happens that attending IT Security and/or Linux Kernel events makes me somewhat gloomy for the above-mentioned reasons.

Meanwhile, if you have some interesting projcets/ideas at the border between cellular protocols/systems and security, I'd of course love to hear if there's some way to get my hands dirty in that area again :)

Gory details of USIM authentication sequence numbers

Harald Welte — Tue, 07 Mar 2017 23:00:00 GMT

I always though I understood UMTS AKA (authentication and key agreement), including the re-synchronization procedure. It's been years since I wrote tools like osmo-sim-auth which you can use to perform UMTS AKA with a SIM card inserted into a PC reader, i.e. simulate what happens between the AUC (authentication center) in a network and the USIM card.

However, it is only now as the sysmocom team works on 3G support of the dedicated OsmoHLR (outside of OsmoNITB!), that I seem to understand all the nasty little details.

I always thought for re-synchronization it is sufficient to simply increment the SQN (sequence number). It turns out, it isn't as there is a MSB-portion called SEQ and a lower-bit portion called IND, used for some fancy array indexing scheme of buckets of highest-used-SEQ within that IND bucket.

If you're interested in all the dirty details and associated spec references (the always hide the important parts in some Annex) see the discussion between Neels and me in Osmocom redmine issue 1965.

VMware becomes gold member of Linux Foundation: And what about the GPL?

Harald Welte — Tue, 07 Mar 2017 23:00:00 GMT

As we can read in recent news, VMware has become a gold member of the Linux foundation. That causes - to say the least - very mixed feelings to me.

One thing to keep in mind: The Linux Foundation is an industry association, it exists to act in the joint interest of it's paying members. It is not a charity, and it does not act for the public good. I know and respect that, while some people sometimes appear to be confused about its function.

However, allowing an entity like VMware to join, despite their many years long disrespect for the most basic principles of the FOSS Community (such as: Following the GPL and its copyleft principle), really is hard to understand and accept.

I wouldn't have any issue if VMware would (prior to joining LF) have said: Ok, we had some bad policies in the past, but now we fully comply with the license of the Linux kernel, and we release all derivative/collective works in source code. This would be a positive spin: Acknowledge past issues, resolve the issues, become clean and then publicly underlining your support of Linux by (among other things) joining the Linux Foundation. I'm not one to hold grudges against people who accept their past mistakes, fix the presence and then move on. But no, they haven't fixed any issues.

They are having one of the worst track records in terms of intentional GPL compliance issues for many years, showing outright disrespect for Linux, the GPL and ultimately the rights of the Linux developers, not resolving those issues and at the same time joining the Linux Foundation? What kind of message sends that?

It sends the following messages:

you can abuse Linux, the GPL and copyleft while still being accepted amidst the Linux Foundation Members
it means the Linux Foundations has no ethical concerns whatsoever about accepting such entities without previously asking them to become clean
it also means that VMware has still not understood that Linux and FOSS is about your actions, particularly the kind of choices you make how to technically work with the community, and not against it.

So all in all, I think this move has seriously damaged the image of both entities involved. I wouldn't have expected different of VMware, but I would have hoped the Linux Foundation had some form of standards as to which entities they permit amongst their ranks. I guess I was being overly naive :(

It's a slap in the face of every developer who writes code not because he gets paid, but because it is rewarding to know that copyleft will continue to ensure the freedom of related code.

UPDATE (March 8, 2017):: I was mistaken in my original post in that VMware didn't just join, but was a Linux Foundation member already before, it is "just" their upgrade from silver to gold that made the news recently. I stand corrected. Still doesn't make it any better that the are involved inside LF while engaging in stepping over the lines of license compliance.
UPDATE2 (March 8, 2017):: As some people pointed out, there is no verdict against VMware. Yes, that's true. But the mere fact that they rather distribute derivative works of GPL licensed software and take this to court with an armada of lawyers (instead of simply complying with the license like everyone else) is sad enough. By the time there will be a final verdict, the product is EOL. That's probably their strategy to begin with :/

GTA04 project halts GTA04A5 due to OMAP3 PoP soldering issues

Harald Welte — Sun, 05 Mar 2017 23:00:00 GMT

For those of you who don't know what the tinkerphones/OpenPhoenux GTA04 is: It is a 'professional hobbyist' hardware project (with at least public schematics, even if not open hardware in the sense that editable schematics and PCB design files are published) creating updated mainboards that can be used to upgrade Openmoko phones. They fit into the same enclosure and can use the same display/speaker/microphone.

What the GTA04 guys have been doing for many years is close to a miracle anyway: Trying to build a modern-day smartphone in low quantities, using off-the-shelf components available in those low quantities, and without a large company with its associated financial backing.

Smartphones are complex because they are highly integrated devices. A seemingly unlimited amount of components is squeezed in the tiniest form-factors. This leads to complex circuit boards with many layers that take a lot of effort to design, and are expensive to build in low quantities. The fine-pitch components mandated by the integration density is another issue.

Building the original GTA01 (Neo1937) and GTA02 (FreeRunner) devices at Openmoko, Inc. must seem like a piece of cake compared to what the GTA04 guys are up to. We had a team of engineers that were familiar at last with feature phone design before, and we had the backing of a consumer electronics company with all its manufacturing resources and expertise.

Nevertheless, a small group of people around Dr. Nikolaus Schaller has been pushing the limits of what you can do in a small for fun project, and the have my utmost respect. Well done!

Unfortunately, there are bad news. Manufacturing of their latest generation of phones (GTA04A5) has been stopped due to massive soldering problems with the TI OMAP3 package-on-package (PoP). Those PoPs are basically "RAM chip soldered onto the CPU, and the stack of both soldered to the PCB". This is used to save PCB footprint and to avoid having to route tons of extra (sensitive, matched) traces between the SDRAM and the CPU.

According to the mailing list posts, it seems to be incredibly difficult to solder the PoP stack due to the way TI has designed the packaging of the DM3730. If you want more gory details, see this post and yet another post.

It is very sad to see that what appears to be bad design choices at TI are going to bring the GTA04 project to a halt. The financial hit by having only 33% yield is already more than the small community can take, let alone unused parts that are now in stock or even thinking about further experiments related to the manufacturability of those chips.

If there's anyone with hands-on manufacturing experience on the DM3730 (or similar) TI PoP reading this: Please reach out to the GTA04 guys and see if there's anything that can be done to help them.

UPDATE (March 8, 2017):: In an earlier post I was asserting that the GTA04 is open hardware (which I actually believed up to that point) until some readers have pointed out to me that it isn't. It's sad it isn't, but still it has my sympathies.

Manual testing of Linux Kernel GTP module

Harald Welte — Thu, 23 Feb 2017 23:00:00 GMT

In May 2016 we got the GTP-U tunnel encapsulation/decapsulation module developed by Pablo Neira, Andreas Schultz and myself merged into the 4.8.0 mainline kernel.

During the second half of 2016, the code basically stayed untouched. In early 2017, several patch series of (at least) three authors have been published on the netdev mailing list for review and merge.

This poses the very valid question on how do we test those (sometimes quite intrusive) changes. Setting up a complete cellular network with either GPRS/EGPRS or even UMTS/HSPA is possible using OsmoSGSN and related Osmocom components. But it's of course a luxury that not many Linux kernel networking hackers have, as it involves the availability of a supported GSM BTS or UMTS hNodeB. And even if that is available, there's still the issue of having a spectrum license, or a wired setup with coaxial cable.

So as part of the recent discussions on netdev, I tested and described a minimal test setup using libgtpnl, OpenGGSN and sgsnemu.

This setup will start a mobile station + SGSN emulator inside a Linux network namespace, which talks GTP-C to OpenGGSN on the host, as well as GTP-U to the Linux kernel GTP-U implementation.

In case you're interested, feel free to check the following wiki page: https://osmocom.org/projects/linux-kernel-gtp-u/wiki/Basic_Testing

This is of course just for manual testing, and for functional (not performance) testing only. It would be great if somebody would pick up on my recent mail containing some suggestions about an automatic regression testing setup for the kernel GTP-U code. I have way too many spare-time projects in desperate need of some attention to work on this myself. And unfortunately, none of the telecom operators (who are the ones benefiting most from a Free Software accelerated GTP-U implementation) seems to be interested in at least co-funding or otherwise contributing to this effort :/

Cellular re-broadcast over satellite

Harald Welte — Wed, 15 Feb 2017 23:00:00 GMT

I've recently attended a seminar that (among other topics) also covered RF interference hunting. The speaker was talking about various real-world cases of RF interference and illustrating them in detail.

Of course everyone who has any interest in RF or cellular will know about fundamental issues of radio frequency interference. To the biggest part, you have

cells of the same operator interfering with each other due to too frequent frequency re-use, adjacent channel interference, etc.
cells of different operators interfering with each other due to intermodulation products and the like
cells interfering with cable TV, terrestrial TV
DECT interfering with cells
cells or microwave links interfering with SAT-TV reception
all types of general EMC problems

But what the speaker of this seminar covered was actually a cellular base-station being re-broadcast all over Europe via a commercial satellite (!).

It is a well-known fact that most satellites in the sky are basically just "bent pipes", i.e. they consist of a RF receiver on one frequency, a mixer to shift the frequency, and a power amplifier. So basically whatever is sent up on one frequency to the satellite gets re-transmitted back down to earth on another frequency. This is abused by "satellite hijacking" or "transponder hijacking" and has been covered for decades in various publications.

Ok, but how does cellular relate to this? Well, apparently some people are running VSAT terminals (bi-directional satellite terminals) with improperly shielded or broken cables/connectors. In that case, the RF emitted from a nearby cellular base station leaks into that cable, and will get amplified + up-converted by the block up-converter of that VSAT terminal.

The bent-pipe satellite subsequently picks this signal up and re-transmits it all over its coverage area!

I've tried to find some public documents about this, an there's surprisingly little public information about this phenomenon.

However, I could find a slide set from SES, presented at a Satellite Interference Reduction Group: Identifying Rebroadcast (GSM)

It describes a surprisingly manual and low-tech approach at hunting down the source of the interference by using an old nokia net-monitor phone to display the MCC/MNC/LAC/CID of the cell. Even in 2011 there were already open source projects such as airprobe that could have done the job based on sampled IF data. And I'm not even starting to consider proprietary tools.

It should be relatively simple to have a SDR that you can tune to a given satellite transponder, and which then would look for any GSM/UMTS/LTE carrier within its spectrum and dump their identities in a fully automatic way.

But then, maybe it really doesn't happen all that often after all to rectify such a development...

Towards a real SIGTRAN/SS7 stack in libosmo-sigtran

Harald Welte — Sun, 12 Feb 2017 23:00:00 GMT

In the good old days ever since the late 1980ies - and a surprising amount even still today - telecom signaling traffic is still carried over circuit-switched SS7 with its TDM lines as physical layer, and not an IP/Ethernet based transport.

When Holger first created OsmoBSC, the BSC-only version of OpenBSC some 7-8 years ago, he needed to implement a minimal subset of SCCP wrapped in TCP called SCCP Lite. This was due to the simple fact that the MSC to which it should operate implemented this non-standard protocol stacking that was developed + deployed before the IETF SIGTRAN WG specified M3UA or SUA came around. But even after those were specified in 2004, the 3GPP didn't specify how to carry A over IP in a standard way until the end of 2008, when a first A interface over IP study was released.

As time passese, more modern MSCs of course still implement classic circuit-switched SS7, but appear to have dropped SCCPlite in favor of real AoIP as specified by 3GPP meanwhile. So it's time to add this to the osmocom universe and OsmoBSC.

A couple of years ago (2010-2013) implemented both classic SS7 (MTP2/MTP3/SCCP) as well as SIGTRAN stackings (M2PA/M2UA/M3UA/SUA in Erlang. The result has been used in some production deployments, but only with a relatively limited feature set. Unfortunately, this code has nto received any contributions in the time since, and I have to say that as an open source community project, it has failed. Also, while Erlang might be fine for core network equipment, running it on a BSC really is overkill. Keep in miond that we often run OpenBSC on really small ARM926EJS based embedded systems, much more resource constrained than any single smartphone during the late decade.

In the meantime (2015/2016) we also implemented some minimal SUA support for interfacing with UMTS femto/small cells via Iuh (see OsmoHNBGW).

So in order to proceed to implement the required SCCP-over-M3UA-over-SCTP stacking, I originally thought well, take Holgers old SCCP code, remove it from the IPA multiplex below, stack it on top of a new M3UA codebase that is copied partially from SUA.

However, this falls short of the goals in several ways:

The application shouldn't care whether it runs on top of SUA or SCCP, it should use a unified interface towards the SCCP Provider. OsmoHNBGW and the SUA code already introduce such an interface baed on the SCCP-User-SAP implemented using Osmocom primitives (osmo_prim). However, the old OsmoBSC/SCCPlite code doesn't have such abstraction.
The code should be modular and reusable for other SIGTRAN stackings as required in the future

So I found myself sketching out what needs to be done and I ended up pretty much with a re-implementation of large parts. Not quite fun, but definitely worth it.

The strategy is:

Implement the SCCP SCOC state machines for connection-oriented SCCP (of which Iu and A interface are probably the only users) using Osmcoom Finite State Machines (osmo_fsm).
Migrate the existing SUA code on top of that, maintaining the existing osmo_prim based SCCP User SAP
Implement SCCP to SUA and vice-versa message transcoding to makes sure the bulk of the code has to deal only with one message format (parsed SUA).
Introduce a MTP SAP at the lower boundary of the SCCP code
Implement xUA ASP and AS statemachines using osmo_fsm and add ASPTM/ASPSM support to SUA (was missing so far) * Implement
Implement M3UA using the xUA ASP and AS FSMs as well as the general xUA message encoder/decoder, offering the MTP SAP toward SCCP

And then finally stack all those bits on top of each other, rendering a fairly clean and modern implementation that can be used with the IuCS of the virtually unmodified OsmmoHNBGW, OsmoCSCN and OsmoSGSN for testing.

Next steps in the direction of the AoIP are:

Implementation of the MTP-SAP based on the IPA transport
Binding the new SCCP code on top of that
Converting OsmoBSC code base to use the SCCP-User-SAP for its signaling connection

From that point onwards, OsmoBSC doesn't care anymore whether it transports the BSSAP/BSSMAP messages of the A interface over SCCP/IPA/TCP/IP (SCCPlite) SCCP/M3UA/SCTP/IP (3GPP AoIP), or even something like SUA/SCTP/IP.

However, the 3GPP AoIP specs (unlike SCCPlite) actually modify the BSSAP/BSSMAP payload. Rather than using Circuit Identifier Codes and then mapping the CICs to UDP ports based on some secret conventions, they actually encapsulate the IP address and UDP port information for the RTP streams. This is of course the cleaner and more flexible approach, but it means we'll have to do some further changes inside the actual BSC code to accommodate this.

Testing (not only) telecom protocols

Harald Welte — Sat, 11 Feb 2017 23:00:00 GMT

When implementing any kind of communication protocol, one always dreams of some existing test suite that one can simply run against the implementation to check if it performs correct in at least those use cases that matter to the given application.

Of course in the real world, there rarely are protocols where this is true. If test specifications exist at all, they are often just very abstract texts for human consumption that you as the reader should implement yourself.

For some (by far not all) of the protocols found in cellular networks, every so often I have seen some formal/abstract machine-parseable test specifications. Sometimes it was TTCN-2, and sometimes TTCN-3.

If you haven't heard about TTCN-3, it is basically a way to create functional tests in an abstract description (textual + graphical), and then compile that into an actual executable tests suite that you can run against the implementation under test.

However, when I last did some research into this several years ago, I couldn't find any Free / Open Source tools to actually use those formally specified test suites. This is not a big surprise, as even much more fundamental tools for many telecom protocols are missing, such as good/complete ASN.1 compilers, or even CSN.1 compilers.

To my big surprise I now discovered that Ericsson had released their (formerly internal) TITAN TTCN3 Toolset as Free / Open Source Software under EPL 1.0. The project is even part of the Eclipse Foundation. Now I'm certainly not a friend of Java or Eclipse by all means, but well, for running tests I'd certainly not complain.

The project also doesn't seem like it was a one-time code-drop but seems very active with many repositories on gitub. For example for the core module, titan.core shows plenty of activity on an almost daily basis. Also, binary releases for a variety of distributions are made available. They even have a video showing the installation ;)

If you're curious about TTCN-3 and TITAN, Ericsson also have made available a great 200+ pages slide set about TTCN-3 and TITAN.

I haven't yet had time to play with it, but it definitely is rather high on my TODO list to try.

ETSI provides a couple of test suites in TTCN-3 for protocols like DIAMETER, GTP2-C, DMR, IPv6, S1AP, LTE-NAS, 6LoWPAN, SIP, and others at http://forge.etsi.org/websvn/ (It's also the first time I've seen that ETSI has a SVN server. Everyone else is using git these days, but yes, revision control systems rather than periodic ZIP files is definitely a big progress. They should do that for their reference codecs and ASN.1 files, too.

I'm not sure once I'll get around to it. Sadly, there is no TTCN-3 for SCCP, SUA, M3UA or any SIGTRAN related stuff, otherwise I would want to try it right away. But it definitely seems like a very interesting technology (and tool).

FOSDEM 2017

Harald Welte — Fri, 10 Feb 2017 23:00:00 GMT

Last weekend I had the pleasure of attending FOSDEM 2017. For many years, it is probably the most exciting event exclusively on Free Software to attend every year.

My personal highlights (next to meeting plenty of old and new friends) in terms of the talks were:

20 Years of Linux Virtual Memory by MM-Guru Andrea Arcangeli
GPU-Enabled Polyphase Filterbanks by Jan Kraemer
Virtual multi-antenna arrays for estimating the bearing of radio transmitters by Francois Quitin
Secure Microkernel for Deeply Embedded Devices by Jim "jserv" Huang
A discussion of Fedora's Legal state by Tom Callaway
Radio Lockdown Directive by Max Mehl

I was attending but not so excited by Georg Greve's OpenPOWER talk. It was a great talk, and it is an important topic, but the engineer in me would have hoped for some actual beefy technical stuff. But well, I was just not the right audience. I had heard about OpenPOWER quite some time ago and have been following it from a distance.

The LoRaWAN talk couldn't have been any less technical, despite stating technical, political and cultural in the topic. But then, well, just recently 33C3 had the most exciting LoRa PHY Reverse Engineering Talk by Matt Knight.

Other talks whose recordings I still want to watch one of these days:

Osmocom Conference 2017 on April 21st

Harald Welte — Tue, 31 Jan 2017 23:00:00 GMT

I'm very happy that in 2017, we will have the first ever technical conference on the Osmocom cellular infrastructure projects.

For many years, we have had a small, invitation only event by Osmocom developers for Osmocom developers called OsmoDevCon. This was fine for the early years of Osmocom, but during the last few years it became apparent that we also need a public event for our many users. Those range from commercial cellular operators to community based efforts like Rhizomatica, and of course include the many research/lab type users with whom we started.

So now we'll have the public OsmoCon on April 21st, back-to-back with the invitation-only OsmoDevcon from April 22nd through 23rd.

I'm hoping we can bring together a representative sample of our user base at OsmoCon 2017 in April. Looking forward to meet you all. I hope you're also curious to hear more from other users, and of course the development team.

Regards,: Harald

Autodesk: How to lose loyal EAGLE customers

Harald Welte — Sun, 22 Jan 2017 23:00:00 GMT

A few days ago, Autodesk has announecd that the popular EAGLE electronics design automation (EDA) software is moving to a subscription based model.

When previously you paid once for a license and could use that version/license as long as you wanted, there now is a monthly subscription fee. Once you stop paying, you loose the right to use the software. Welcome to the brave new world.

I have remotely observed this subscription model as a general trend in the proprietary software universe. So far it hasn't affected me at all, as the only two proprietary applications I use on a regular basis during the last decade are IDA and EAGLE.

I already have ethical issues with using non-free software, but those two cases have been the exceptions, in order to get to the productivity required by the job. While I can somehow convince my consciousness in those two cases that it's OK - using software under a subscription model is completely out of the question, period. Not only would I end up paying for the rest of my professional career in order to be able to open and maintain old design files, but I would also have to accept software that "calls home" and has "remote kill" features. This is clearly not something I would ever want to use on any of my computers. Also, I don't want software to be associated with any account, and it's not the bloody business of the software maker to know when and where I use my software.

For me - and I hope for many, many other EAGLE users - this move is utterly unacceptable and certainly marks the end of any business between the EAGLE makers and myself and/or my companies. I will happily use my current "old-style" EAGLE 7.x licenses for the near future, and theS see what kind of improvements I would need to contribute to KiCAD or other FOSS EDA software in order to eventually migrate to those.

As expected, this doesn't only upset me, but many other customers, some of whom have been loyal to using EAGLE for many years if not decades, back to the DOS version. This is reflected by some media reports (like this one at hackaday or user posts at element14.com or eaglecentral.ca who are similarly critical of this move.

Rest in Peace, EAGLE. I hope Autodesk gets what they deserve: A new influx of migrations away from EAGLE into the direction of Open Source EDA software like KiCAD.

In fact, the more I think about it, I'm actually very much inclined to work on good FOSS migration tools / converters - not only for my own use, but to help more people move away from EAGLE. It's not that I don't have enough projects at my hand already, but at least I'm motivated to do something about this betrayal by Autodesk. Let's see what (if any) will come out of this.

So let's see it that way: What Autodesk is doing is raising the level off pain of using EAGLE so high that more people will use and contribute FOSS EDA software. And that is actually a good thing!

33C3 talk on dissecting cellular modems

Harald Welte — Fri, 30 Dec 2016 00:00:00 GMT

Yesterday, together with Holger 'zecke' Freyther, I co-presented at 33C3 about Dissectiong modern (3G/4G) cellular modems.

This presentation covers some of our recent explorations into a specific type of 3G/4G cellular modems, which next to the regular modem/baseband processor also contain a Cortex-A5 core that (unexpectedly) runs Linux.

We want to use such modems for building self-contained M2M devices that run the entire application inside the modem itself, without any external needs except electrical power, SIM card and antenna.

Next to that, they also pose an ideal platform for testing the Osmocom network-side projects for running GSM, GPRS, EDGE, UMTS and HSPA cellular networks.

You can find the Slides and the Video recordings in case you're interested in more details about our work.

The results of our reverse engineering can be found in the wiki at http://osmocom.org/projects/quectel-modems/wiki together with links to the various git repositories containing related tools.

As with all the many projects that I happen to end up doing, it would be great to get more people contributing to them. If you're interested in cellular technology and want to help out, feel free to register at the osmocom.org site and start adding/updating/correcting information to the wiki.

You can e.g. help by

playing with the modem and documenting your findings
reviewing the source code released by Qualcomm + Quectel and documenting your findings
help us to create a working OE build with our own kernel and rootfs images as well as opkg package feeds for the modems
help reverse engineering DIAG and QMI protocols as well as the open source programs to interact with them

Some thoughts on 33C3

Harald Welte — Fri, 30 Dec 2016 00:00:00 GMT

I've just had the pleasure of attending all four days of 33C3 and have returned home with somewhat mixed feelings.

I've been a regular visitor and speaker at CCC events since 15C3 in 1998, which among other things means I'm an old man now. But I digress ;)

The event has come extremely far in those years. And to be honest, I struggle with the size. Back then, it was a meeting of like-minded hackers. You had the feeling that you know a significant portion of the attendees, and it was easy to connect to fellow hackers.

These days, both the number of attendees and the size of the event make you feel much rather that you're in general public, rather than at some meeting of fellow hackers. Yes, it is good to see that more people are interested in what the CCC (and the selected speakers) have to say, but somehow it comes at the price that I (and I suspect other old-timers) feel less at home. It feels too much like various other technology related events.

One aspect creating a certain feeling of estrangement is also the venue itself. There are an incredible number of rooms, with a labyrinth of hallways, stairs, lobbies, etc. The size of the venue simply makes it impossible to simply _accidentally_ running into all of your fellow hackers and friends. If I want to meet somebody, I have to make an explicit appointment. That is an option that exits most of the rest of the year, too.

While fefe is happy about the many small children attending the event, to me this seems somewhat alien and possibly inappropriate. I guess from teenage years onward it certainly makes sense, as they can follow the talks and participate in the workshop. But below that age?

The range of topics covered at the event also becomes wider, at least I feel that way. Topics like IT security, data protection, privacy, intelligence/espionage and learning about technology have always been present during all those years. But these days we have bloggers sitting on stage and talking about bottles of wine (seriously?).

Contrary to many, I also really don't get the excitement about shows like 'Methodisch Inkorrekt'. Seems to me like mainstream compatible entertainment in the spirit of the 1990ies Knoff Hoff Show without much potential to make the audience want to dig deeper into (information) technology.

Contribute to Osmocom 3.5G and receive a free femtocell

Harald Welte — Thu, 29 Dec 2016 00:00:00 GMT

In 2016, Osmocom gained initial 3.5G support with osmo-iuh and the Iu interface extensions of our libmsc and OsmoSGSN code. This means you can run your own small open source 3.5G cellular network for SMS, Voice and Data services.

However, the project needs more contributors: Become an active member in the Osmocom development community and get your nano3G femtocell for free.

I'm happy to announce that my company sysmocom hereby issues a call for proposals to the general public. Please describe in a short proposal how you would help us improving the Osmocom project if you were to receive one of those free femtocells.

Details of this proposal can be found at https://sysmocom.de/downloads/accelerate_3g5_cfp.pdf

Please contact mailto:accelerate3g5@sysmocom.de in case of any questions.

Accessing 3GPP specs in PDF format

Harald Welte — Fri, 16 Dec 2016 00:00:00 GMT

When you work with GSM/cellular systems, the definite resource are the specifications. They were originally released by ETSI, later by 3GPP.

The problem start with the fact that there are separate numbering schemes. Everyone in the cellular industry I know always uses the GSM/3GPP TS numbering scheme, i.e. something like 3GPP TS 44.008. However, ETSI assigns its own numbers to the specs, like ETSI TS 144008. Now in most cases, it is as simple s removing the '.' and prefixing the '1' in the beginning. However, that's not always true and there are exceptions such as 3GPP TS 01.01 mapping to ETSI TS 101855. To make things harder, there doesn't seem to be a machine-readable translation table between the spec numbers, but there's a website for spec number conversion at http://webapp.etsi.org/key/queryform.asp

When I started to work on GSM related topics somewhere between my work at Openmoko and the start of the OpenBSC project, I manually downloaded the PDF files of GSM specifications from the ETSI website. This was a cumbersome process, as you had to enter the spec number (e.g. TS 04.08) in a search window, look for the latest version in the search results, click on that and then click again for accessing the PDF file (rather than a proprietary Microsoft Word file).

At some point a poor girlfriend of mine was kind enough to do this manual process for each and every 3GPP spec, and then create a corresponding symbolic link so that you could type something like evince /spae/openmoko/gsm-specs/by_chapter/44.008.pdf into your command line and get instant access to the respective spec.

However, of course, this gets out of date over time, and by now almost a decade has passed without a systematic update of that archive.

To the rescue, 3GPP started at some long time ago to not only provide the obnoxious M$ Word DOC files, but have deep links to ETSI. So you could go to http://www.3gpp.org/DynaReport/44-series.htm and then click on 44.008, and one further click you had the desired PDF, served by ETSI (3GPP apparently never provided PDF files).

However, in their infinite wisdom, at some point in 2016 the 3GPP webmaster decided to remove those deep links. Rather than a nice long list of released versions of a given spec, http://www.3gpp.org/DynaReport/44008.htm now points to some crappy JavaScript tabbed page, where you can click on the version number and then get a ZIP file with a single Word DOC file inside. You can hardly male it any more inconvenient and cumbersome. The PDF links would open immediately in modern browsers built-in JavaScript PDF viewer or your favorite PDF viewer. Single click to the information you want. But no, the PDF links had to go and replaced with ZIP file downloads that you first need to extract, and then open in something like LibreOffice, taking ages to load the document, rendering it improperly in a word processor. I don't want to edit the spec, I want to read it, sigh.

So since the usability of this 3GPP specification resource had been artificially crippled, I was annoyed sufficiently well to come up with a solution:

first create a complete mirror of all ETSI TS (technical specifications) by using a recursive wget on http://www.etsi.org/deliver/etsi_ts/
then use a shell script that utilizes pdfgrep and awk to determine the 3GPP specification number (it is written in the title on the first page of the document) and creating a symlink. Now I have something like 44.008-4.0.0.pdf -> ts_144008v040000p.pdf

It's such a waste of resources to have to download all those files and then write a script using pdfgrep+awk to re-gain the same usability that the 3GPP chose to remove from their website. Now we can wait for ETSI to disable indexing/recursion on their server, and easy and quick spec access would be gone forever :/

Why does nobody care about efficiency these days?

If you're also an avid 3GPP spec reader, I'm publishing the rather trivial scripts used at http://git.osmocom.org/3gpp-etsi-pdf-links

If you have contacts to the 3GPP webmaster, please try to motivate them to reinstate the direct PDF links.

Open Hardware IEEE 802.15.4 adapter "ATUSB" available again

Harald Welte — Wed, 07 Dec 2016 00:00:00 GMT

Many years ago, in the aftermath of Openmoko shutting down, fellow former Linux kernel hacker Werner Almesberger was working on an IEEE 802.15.4 (WPAN) adapter for the Ben Nanonote.

As a spin-off to that, the ATUSB device was designed: A general-purpose open hardware (and FOSS firmware + driver) IEEE 802.15.4 adapter that can be plugged into any USB port.

This adapter has received a mainline Linux kernel driver written by Werner Almesberger and Stefan Schmidt, which was eventually merged into mainline Linux in May 2015 (kernel v4.2 and later).

Earlier in 2016, Stefan Schmidt (the current ATUSB Linux driver maintainer) approached me about the situation that ATUSB hardware was frequently asked for, but currently unavailable in its physical/manufactured form. As we run a shop with smaller electronics items for the wider Osmocom community at sysmocom, and we also frequently deal with contract manufacturers for low-volume electronics like the SIMtrace device anyway, it was easy to say "yes, we'll do it".

As a result, ready-built, programmed and tested ATUSB devices are now finally available from the sysmocom webshop

Note: I was never involved with the development of the ATUSB hardware, firmware or driver software at any point in time. All credits go to Werner, Stefan and other contributors around ATUSB.

The IT security culture, hackers vs. industry consortia

Harald Welte — Tue, 06 Dec 2016 07:00:00 GMT

In a previous life I used to do a lot of IT security work, probably even at a time when most people had no idea what IT security actually is. I grew up with the Chaos Computer Club, as it was a great place to meet people with common interests, skills and ethics. People were hacking (aka 'doing security research') for fun, to grow their skills, to advance society, to point out corporate stupidities and to raise awareness about issues.

I've always shared any results worth noting with the general public. Whether it was in RFID security, on GSM security, TETRA security, etc.

Even more so, I always shared the tools, creating free software implementations of systems that - at that time - were very difficult to impossible to access unless you worked for the vendors of related device, who obviously had a different agenda then to disclose security concerns to the general public.

Publishing security related findings at related conferences can be interpreted in two ways:

On the one hand, presenting at a major event will add to your credibility and reputation. That's a nice byproduct, but that shouldn't be the primarily reason, unless you're some kind of a egocentric stage addict.

On the other hand, presenting findings or giving any kind of presentation or lecture at an event is a statement of support for that event. When I submit a presentation at a given event, I think carefully if that topic actually matches the event.

The reason that I didn't submit any talks in recent years at CCC events is not that I didn't do technically exciting stuff that I could talk about - or that I wouldn't have the reputation that would make people consider my submission in the programme committee. I just thought there was nothing in my work relevant enough to bother the CCC attendees with.

So when Holger 'zecke' Freyther and I chose to present about our recent journeys into exploring modern cellular modems at the annual Chaos Communications Congress, we did so because the CCC Congress is the right audience for this talk. We did so, because we think the people there are the kind of community of like-minded spirits that we would like to contribute to. Whom we would like to give something back, for the many years of excellent presentations and conversations had.

So far so good.

However, in 2016, something happened that I haven't seen yet in my 17 years of speaking at Free Software, Linux, IT Security and other conferences: A select industry group (in this case the GSMA) asking me out of the blue to give them the talk one month in advance at a private industry event.

I could hardly believe it. How could they? Who am I? Am I spending sleepless nights and non-existing spare time into security research of cellular modems to give a free presentation to corporate guys at a closed industry meeting? The same kind of industries that create the problems in the first place, and who don't get their act together in building secure devices that respect people's privacy? Certainly not. I spend sleepless nights of hacking because I want to share the results with my friends. To share it with people who have the same passion, whom I respect and trust. To help my fellow hackers to understand technology one step more.

If that kind of request to undermine the researcher/authors initial publication among friends is happening to me, I'm quite sure it must be happening to other speakers at the 33C3 or other events, too. And that makes me very sad. I think the initial publication is something that connects the speaker/author with his audience.

Let's hope the researchers/hackers/speakers have sufficiently strong ethics to refuse such requests. If certain findings are initially published at a certain conference, then that is the initial publication. Period. Sure, you can ask afterwards if an author wants to repeat the presentation (or a similar one) at other events. But pre-empting the initial publication? Certainly not with me.

I offered the GSMA that I could talk on the importance of having FOSS implementations of cellular protocol stacks as enabler for security research, but apparently this was not to their interest. Seems like all they wanted is an exclusive heads-up on work they neither commissioned or supported in any other way.

And btw, I don't think what Holger and I will present about is all that exciting in the first place. More or less the standard kind of security nightmares. By now we are all so numbed down by nobody considering security and/or privacy in design of IT systems, that is is hardly any news. IoT how it is done so far might very well be the doom of mankind. An unstoppable tsunami of insecure and privacy-invading devices, built on ever more complex technology with way too many security issues. We shall henceforth call IoT the Industry of Thoughtlessness.

DHL zones and the rest of the world

Harald Welte — Tue, 06 Dec 2016 06:50:00 GMT

I typically prefer to blog about technical topics, but the occasional stupidity in every-day (business) life is simply too hard to resist.

Today I updated the shipping pricing / zones in the ERP system of my company to predict shipping rates based on weight and destination of the package.

Deutsche Post, the German Postal system is using their DHL brand for postal packages. They divide the world into four zones:

Zone 1 (EU)
Zone 2 (Europe outside EU)
Zone 3 (World)

You would assume that "World" encompasses everything that's not part of the other zones. So far so good. However, I then stumbled about Zone 4 (rest of world). See for yourself:

So the World according to DHL is a very small group of countries including Libya and Syria, while countries like Mexico are rest of world

Quite charming, I wonder which PR, communicatoins or marketing guru came up with such a disqualifying name. Maybe they should hve called id 3rd world and 4th world instead? Or even discworld?

Ten years anniversary of Openmoko

Harald Welte — Sun, 27 Nov 2016 15:00:00 GMT

In 2006 I first visited Taiwan. The reason back then was Sean Moss-Pultz contacting me about a new Linux and Free Software based Phone that he wanted to do at FIC in Taiwan. This later became the Neo1973 and the Openmoko project and finally became part of both Free Software as well as smartphone history.

Ten years later, it might be worth to share a bit of a retrospective.

It was about building a smartphone before Android or the iPhone existed or even were announced. It was about doing things "right" from a Free Software point of view, with FOSS requirements going all the way down to component selection of each part of the electrical design.

Of course it was quite crazy in many ways. First of all, it was a bunch of white, long-nosed western guys in Taiwan, starting a company around Linux and Free Software, at a time where that was not really well-perceived in the embedded and consumer electronics world yet.

It was also crazy in terms of the many cultural 'impedance mismatches', and I think at some point it might even be worth to write a book about the many stories we experienced. The biggest problem here is of course that I wouldn't want to expose any of the companies or people in the many instances something went wrong. So probably it will remain a secret to those present at the time :/

In any case, it was a great project and definitely one of the most exciting (albeit busy) times in my professional career so far. It was also great that I could involve many friends and FOSS-compatriots from other projects in Openmoko, such as Holger Freyther, Mickey Lauer, Stefan Schmidt, Daniel Willmann, Joachim Steiger, Werner Almesberger, Milosch Meriac and others. I am happy to still work on a daily basis with some of that group, while others have moved on to other areas.

I think we all had a lot of fun, learned a lot (not only about Taiwan), and were working really hard to get the hardware and software into shape. However, the constantly growing scope, the [for western terms] quite unclear and constantly changing funding/budget situation and the many changes in direction have ultimately lead to missing the market opportunity. At the time the iPhone and later Android entered the market, it was too late for a small crazy Taiwanese group of FOSS-enthusiastic hackers to still have a major impact on the landscape of Smartphones. We tried our best, but in the end, after a lot of hype and publicity, it never was a commercial success.

What's more sad to me than the lack of commercial success is also the lack of successful free software that resulted. Sure, there were some u-boot and linux kernel drivers that got merged mainline, but none of the three generations of UI stacks (GTK, Qt or EFL based), nor the GSM Modem abstraction gsmd/libgsmd nor middleware (freesmartphone.org) has manage to survive the end of the Openmoko company, despite having deserved to survive.

Probably the most important part that survived Openmoko was the pioneering spirit of building free software based phones. This spirit has inspired pure volunteer based projects like GTA04/Openphoenux/Tinkerphone, who have achieved extraordinary results - but who are in a very small niche.

What does this mean in practise? We're stuck with a smartphone world in which we can hardly escape any vendor lock-in. It's virtually impossible in the non-free-software iPhone world, and it's difficult in the Android world. In 2016, we have more Linux based smartphones than ever - yet we have less freedom on them than ever before. Why?

the amount of hardware documentation on the processors and chipsets to day is typically less than 10 years ago. Back then, you could still get the full manual for the S3C2410/S3C2440/S3C6410 SoCs. Today, this is not possible for the application processors of any vendor
the tighter integration of application processor and baseband processor means that it is no longer possible on most phone designs to have the 'non-free baseband + free application processor' approach that we had at Openmoko. It might still be possible if you designed your own hardware, but it's impossible with any actually existing hardware in the market.
Google blurring the line between FOSS and proprietary code in the Android OS. Yes, there's AOSP - but how many features are lacking? And on how many real-world phones can you install it? Particularly with the Google Nexus line being EOL'd? One of the popular exceptions is probably Fairphone2 with it's alternative AOSP operating system, even though that's not the default of what they ship.
The many binary-only drivers / blobs, from the graphics stack to wifi to the cellular modem drivers. It's a nightmare and really scary if you look at all of that, e.g. at the binary blob downloads for Fairphone2 to get an idea about all the binary-only blobs on a relatively current Qualcomm SoC based design. That's compressed 70 Megabytes, probably as large as all of the software we had on the Openmoko devices back then...

So yes, the smartphone world is much more restricted, locked-down and proprietary than it was back in the Openmoko days. If we had been more successful then, that world might be quite different today. It was a lost opportunity to make the world embrace more freedom in terms of software and hardware. Without single-vendor lock-in and proprietary obstacles everywhere.

Open Hardware miniPCIe WWAN modem USB breakout board released

Harald Welte — Thu, 24 Nov 2016 23:00:00 GMT

There are plenty of cellular modems on the market in the mPCIe form factor.

Playing with such modems is reasonably easy, you can simply insert them in a mPCIe slot of a laptop or an embedded device (soekris, pc-engines or the like).

However, many of those modems actually export interesting signals like digital PCM audio or UART ports on some of the mPCIe pins, both in standard and in non-standard ways. Those signals are inaccessible in those embedded devices or in your laptop.

So I built a small break-out board which performs the basic function of exposing the mPCIe USB signals on a USB mini-B socket, providing power supply to the mPCIe modem, offering a SIM card slot at the bottom, and exposing all additional pins of the mPCIe header on a standard 2.54mm pitch header for further experimentation.

The design of the board (including schematics and PCB layout design files) is available as open hardware under CC-BY-SA license terms. For more information see http://osmocom.org/projects/mpcie-breakout/wiki

If you don't want to build your own board, fully assembled and tested boards are available from http://shop.sysmocom.de/products/minipcie-wwan-modem-usb-break-out-board

Open Hardware Multi-Voltage USB UART board released

Harald Welte — Thu, 24 Nov 2016 23:00:00 GMT

During the past 16 years I have been playing a lot with a variety of embedded devices.

One of the most important tasks for debugging or analyzing embedded devices is usually to get access to the serial console on the UART of the device. That UART is often exposed at whatever logic level the main CPU/SOC/uC is running on. For 5V and 3.3V that is easy, but for ever more and more unusual voltages I always had to build a custom cable or a custom level shifter.

In 2016, I finally couldn't resist any longer and built a multi-voltage USB UART adapter.

This board exposes two UARTs at a user-selectable voltage of 1.8, 2.3, 2.5, 2.8, 3.0 or 3.3V. It can also use whatever other logic voltage between 1.8 and 3.3V, if it can source a reference of that voltage from the target embedded board.

Rather than just building one for myself, I released the design as open hardware under CC-BY-SA license terms. Full schematics + PCB layout design files are available. For more information see http://osmocom.org/projects/mv-uart/wiki

In case you don't want to build it from scratch, ready-made machine assembled boards are also made available from http://shop.sysmocom.de/products/multi-voltage-usb-dual-uart

Holidays in Taiwan, again

Harald Welte — Wed, 24 Aug 2016 14:00:00 GMT

Ever since I first came to Taiwan in 2006 (which happens to be more or less exactly 10 years ago, watch out for a separate blog post about that), I've been coming back at least once every year until 2014. Sometimes it's business related ,but one trip per year has always been about holidays.

I really like the country for a variety of reasons. One of them is the beautiful landscape from sand beaches to tropical forsts and high mountains (Taiwan has more than 100 peaks higher than 3000m). This is also the reason I keep my Yamaha TW-225 motorbike here, as it's impossible to explore the island without your own transport. And I hate driving bulky, large cars. Plus, some of the narrow roads have ascent/descent levels and road conditions that you actually can only pass them with a motorbike, preferrably using offroad tyres.

But I digress. I like coming to Taiwan, and motorbiking accross the country is one of the main reasons why.

After the various trips before, including the FIXME last trip circling the island in 2014, I wanted to do something special this year. The original plan was to cross the southern cross-island highway (Provincial Highway 20) from Taitung to Tainan. Unfortunately that road has been closed for many years now due to typhoon related damage. Typhonns and Eathquakes (and associated landslides) unfortunately happen quite often around Taiwan.

I've received some news that a group of motorcyclists managed to pass the road very recently (despite officially being closed and having various construction sites enroute). However, the road conditions were very difficult, having to pass narrow gravel sections at the construction sites, etc. So I again postponed this plan until a future year.

Instead, I wanted to travel along provincial highway '7 jia', which passes alongside a river valley into the central mountain ranges towards Lishan, and from there take Nantou Lixing Industry road (aka road 89) further down south towards Ren'ai. This road is of the most narrow roads that you can find on maps of Taiwan, and leads through very remote mountain areas with little population. You can find an interesting report of somebody crazy enough to travel that road on a bicycle in 2015 online. The author describes it as the most epic of all rides in Taiwan and you can see pictures of the road conditions in it.

Unfortuantely, while on our way, a Typhoon struck the southern tip of Taiwan, and also brought loads of torrential rain into the central mountain areas. Given that road 89 is difficult even in dry conditions, I didn't want to take chances in terms of landslides and muddy road conditions, and I had to turn back from Lishan to Taipei.

In order to make the return trip a bit different, I went all the way up to Yilan, and then alongside the north-east coast to Gongliao, before heading back to Tiapei via Shifen and Pinxi.

Soon after returning to Taipei, the second Typhoon affected the weather (passing by off the north-east of Taiwan).

In the following week the weather was excellent and there was time for a second trip. However, due to the rain of the two typhoons I still didn't want to go for road 89, and decided to go towards the north coast and continue to explore some of the waterfalls described at the vonderful http://taiwanswaterfalls.com/ site. Specifically, those alongside the Keelung river valley, close to road 106 form Taipei towards the coast, then heading north along the coast, covering the tip and returning back to Taipei via Dansuei (aka Tamsui).

In summary, the motorbike tours were a lot of fun. What made them more fun than the previous trips was my strategy of using 'white' roads (smaller roads) and avoid the provincial highways whenever three was a 'white' alternative on the map. Also, the newly-discovered map of Taiwans waterfalls was helping a lot to find beautiful sights, and encourage to go on even uncharted roads at times, a real challenge to both bike and biker.

So if you ever consider recreational motorbike riding in Taiwan, my recommendations would be:

get a light-weight motorbike. There's no point for a heavy bike in Taiwan, other than for locals to show it off. Many mountain roads typically have speed limits of 30 or 40 kph, and while it is possible to go 10-20 kph faster, more than that is suicidal. So no need for a fast bike with large engine.
avoid travel on weekends. Everything is super crowded on weekends. I prefer to stay home (and even work) on weekends, while using the quiet weekdays to travel.
always choose mountainous roads over straight roads.
choose smaller 'white' roads over provincial highways. Even though provincial highways on weekdays have less traffic, they still tend to attract a fair number of trucks, and are generally more easy and a less challenging ride.
always carry sufficient water with you.
go in September or October, after the rain season (normally) is over. The weather is still warm to hot (depending on altitude).

(East) European motorbike tour on 20y old BMW F650ST

Harald Welte — Tue, 16 Aug 2016 14:00:00 GMT

For many years I've always been wanting to do some motorbike riding across the Alps, but somehow never managed to do so. It seems when in Germany I've always been too busy - contrary to the many motorbike tours around and across Taiwan which I did during my frequent holidays there.

This year I finally took the opportunity to combine visiting some friends in Hungary and Bavaria with a nice tour starting from Berlin over Prague and Brno (CZ), Bratislava (SK) to Tata and Budapeest (HU), further along lake Balaton (HU) towards Maribor (SI) and finally across the Grossglockner High Alpine Road (AT) to Salzburg and Bavaria before heading back to Berlin.

It was eight fun (but sometimes long) days riding. For some strange turn of luck, not a single drop of rain was encountered during all that time, traveling across six countries.

The most interesting parts of the tour were:

Along the Elbe river from Pirna (DE) to Lovosice (CZ). Beautiful scenery along the river valley, most parts of the road immediately on either side of the river. Quite touristy on the German side, much more pleasant and quiet on the Czech side.
From Mosonmagyarovar via Gyor to Tata (all HU). Very little traffic alongside road '1'. Beautiful scenery with lots of agriculture and forests left and right.
The Northern coast of Lake Balaton, particularly from Tinany to Keszthely (HU). Way too many tourists and traffic for my taste, but still very impressive to realize how large/long that lake really is.
From Maribor to Dravograd (SI) alongside the Drau/Drav river valley.
Finally, of course, the Grossglockner High Alpine Road, which reminded me in many ways of the high mountain tours I did in Taiwan. Not a big surprise, given that both lead you up to about 2500 meters above sea level.

Finally, I have to say I've been very happy with the performance of my 1996 model BMW F 650ST bike, who has coincidentally just celebrated its 20ieth anniversary. I know it's an odd bike design (650cc single-cylinder with two spark plugs, ignition coils and two carburetors) but consider it an acquired taste ;)

I've also published a map with a track log of the trip

In one month from now, I should be reporting from motorbike tours in Taiwan on the equally trusted small Yamaha TW-225 - which of course plays in a totally different league ;)

Going to attend Electromagnetic Field 2016

Harald Welte — Sat, 23 Jul 2016 14:00:00 GMT

Based on some encouragement from friends as well as my desire to find more time again to hang out at community events, I decided to attend Electromagnetic Field 2016 held in Guildford, UK from August 5th through 7th.

As I typically don't like just attending an event without contributing to it in some form, I submitted a couple of talks / workshops, all of which were accepted:

An overview talk about the Osmocom project
A Workshop on running your own cellular network using OpenBSC and related Osmocom software
A Workshop on tracing (U)SIM card communication using Osmocom SIMtrace

I believe the detailed schedule is still in the works, as I haven't yet been able to find any on the event website.

Looking forward to having a great time at EMF 2016. After attending Dutch and German hacker camps for almost 20 years, let's see how the Brits go about it!