WEBVTT 00:00.000 --> 00:11.360 We are going to tell you a bit about the backstory, about our current implementation, 00:11.360 --> 00:17.200 and about future planned improvements for the binary packages. 00:17.200 --> 00:24.320 Okay, so what happens if you emerge a package from the source? 00:24.400 --> 00:29.360 I have to explain, maybe a little bit, our package manager is called Portage. 00:29.360 --> 00:35.680 The main command is emerge, and obviously people are emerging packages when they install them. 00:36.720 --> 00:43.200 And you all know, Gen2 originally, as a source-based distribution, that means what happens when 00:43.200 --> 00:50.480 you emerge a package is you fetch unpack and patch the sources, you configure, compile, 00:50.640 --> 00:55.440 install to a temporary directory, then there are some quality assurance checks. 00:55.440 --> 01:01.360 The binary is stripped, the bug information, for example, is separated out, and as last 01:01.360 --> 01:08.160 step the files emerged into your main system. And that's not so much different from other 01:08.160 --> 01:17.200 package bills. Of course, we are skipping over a lot of details here. The dependencies are provided 01:17.280 --> 01:28.000 by the main system, by the user system. Yeah, we did have binary package support in Gen2 01:28.000 --> 01:32.480 already for a long time. These were the so-called X-pack packages, 01:34.240 --> 01:41.840 basically a combination of a compressed tar archive and a custom trailer for metadata. 01:42.800 --> 01:48.720 Nice thing was, it wasn't single file and tar can unpack it, but there were also a lot of 01:48.720 --> 01:54.480 disadvantages. It's a custom binary format. It relies on undocumented tar behavior. 01:55.920 --> 02:01.680 If you want to access the metadata, you have to use special tools. And there was no specification 02:01.680 --> 02:09.200 versioning, not easily extensible, no clear support for any package signatures. So this had to be improved. 02:12.400 --> 02:18.320 The other thing we should maybe know about Gen2 is, it's a source-based distribution. 02:18.320 --> 02:24.880 Packages usually have been compiled on the user system, and that means there are no global 02:24.880 --> 02:35.040 rebuilds ever. So there is no such thing as a mass rebuild. And we need to make sure that nevertheless, 02:35.120 --> 02:42.560 the packages that are installed on the user system are all consistent. Examples for how problems 02:42.560 --> 02:48.400 creep in are, there's a library upgrade, which increases the library's so-version. 02:49.360 --> 02:56.160 There's a pull update, which means the package path for pull modules, changes, and all this 02:56.240 --> 03:05.280 has to be addressed somehow. Then another problem is dependency automagic. 03:06.480 --> 03:12.240 I mean, you all know what happens if you run a configure phase. I mean, this is like a picture 03:12.240 --> 03:18.320 from image magic. You can see that the configure is checking for all sorts of libraries and 03:18.320 --> 03:25.920 detects whether they are present or not present, and which versions, if you don't 03:26.000 --> 03:33.920 have this under control, then your build result can depend on what packages and what versions 03:33.920 --> 03:42.320 exactly are installed on the builder. And that's dangerous because if you then make a binary package 03:43.120 --> 03:49.360 and install it on a host where maybe different things are installed, it doesn't match together anymore. 03:50.080 --> 03:57.040 Example on the builder, we have a certain library, but not on the target. All we have, for example, 03:57.040 --> 04:04.560 different ICU versions on the builder and the target with different so-versions. So build systems 04:04.560 --> 04:10.560 can create so-called automatic dependencies on libraries and other dependencies that they find on 04:10.560 --> 04:18.480 build time. Autoconf is actually a pretty good example because there you can override this automatic 04:18.560 --> 04:27.760 detection really everywhere. And specify it directly. Yeah, to address all these problems, 04:27.760 --> 04:36.720 there were a few old-style gen to help us. They are now by now all deprecated. So the most famous 04:36.720 --> 04:42.480 one maybe is the rebuild. It just finds a binary that has broken dynamic linking, 04:43.040 --> 04:51.280 identifies the affected packages and then rebuilds them. And similar one, it's per cleaner, 04:51.280 --> 04:58.960 it finds per modules that I installed in the library parts of previous not installed any more 04:58.960 --> 05:08.400 per versions and identifies the packages and then rebuilds them. The problem with these both is 05:08.480 --> 05:15.840 they are essentially trying to fix what is already broken. Because if you have a pearl package 05:15.840 --> 05:21.120 with modules in a directory where pearl is not searching anymore, well then it doesn't work anymore. 05:22.160 --> 05:26.720 And it's surprising how much can break if, for example, some part of get text suddenly fails. 05:31.040 --> 05:36.400 The second column is kind of the next approach. The preserve library is mechanism. 05:37.360 --> 05:43.360 There it was integrated into the package manager. So basically on an installation of libraries, 05:43.360 --> 05:49.920 it checks whether still there's any dynamic linking treat. And if there are still programs that link 05:49.920 --> 05:57.120 to a dynamic library, the library is not removed, but it's re assigned to its reverse dependency 05:57.120 --> 06:04.800 in the system. And at the same time this package is recorded somewhere and with the special 06:04.960 --> 06:11.840 syntax you can then rebuild these packages and this way remove dependencies on obsolete libraries. 06:12.960 --> 06:18.080 That was already an improvement because it doesn't usually leave you with a broken system. 06:19.360 --> 06:24.640 But still it's kind of manual act and have to be replaced eventually. 06:28.080 --> 06:33.120 And this here is what was introduced to fix both of these problems. 06:34.080 --> 06:40.080 So called sub slot rebuild mechanism. Let's not go into detail what a slot or a sub slot 06:40.080 --> 06:47.280 precisely is. It's a property of a package and we declare dependency in a special way. 06:47.280 --> 06:54.080 You can see this here. This is strawberry, the music player. You see the dependency on ICU. 06:55.200 --> 06:59.680 It's using this particular operator here. What that means is 06:59.840 --> 07:09.360 any sub slot value of ICU fulfills the dependency. But if it changes, strawberry has to be rebuilt. 07:11.360 --> 07:18.480 And inside the image of ICU, we have here the declaration of the sub slot, 07:19.840 --> 07:23.680 which is increased when in this case the package version changes. 07:23.840 --> 07:31.600 That is in our specification. It literally no explanation or no standardization what a 07:31.600 --> 07:36.880 sub slot exactly means. It can be many different things. It can be a library as a version. 07:37.440 --> 07:42.880 It can be just a library major version or it can be a package version like here for Pearl 07:42.880 --> 07:54.080 for example. And here we have for example an update run where an ICU version was pulled in. 07:54.800 --> 08:02.560 So in this case 78.2 and you can see that a lot of things have to be rebuilt and this is actually cut 08:02.560 --> 08:09.360 off the list is much longer. It's kind of obvious. It's an essential library. 08:09.680 --> 08:20.320 That's one thing. Then we get to quality control. I talked about the automatic 08:20.320 --> 08:29.280 automatic dependencies. So the rule in general is no build must depend on the presence, 08:29.280 --> 08:34.800 absence or version of packages beyond what you specify in the dependencies. 08:35.280 --> 08:43.440 So basically we have to tell this system exactly do we want to have an image magic with 08:43.440 --> 08:50.400 JPEG or without JPEG support. Having image magic without JPEG support is not particularly useful, 08:51.920 --> 08:58.640 but still. And so basically you tell this the package manager by also called usefacts 08:59.520 --> 09:07.200 and during the configure phase we then have all these function calls here. For example use enable 09:07.200 --> 09:14.880 and then some flag name that expands into enable flag or disable flag. So basically it tells the 09:14.880 --> 09:19.680 configure phase exactly whether a certain thing should be enabled or not. And this way this becomes 09:19.680 --> 09:27.840 predictable. The same you can do with versions for example telling a program it should use 09:27.840 --> 09:39.280 Qt5 or Qt6 and so on. Last important step was a new binary package format. 09:42.000 --> 09:49.760 We went all the way and wrote up a detailed specification. Now we have basically a tar archive. 09:49.760 --> 09:54.240 The auto tar archive is uncompressed. It has a certain list of members. 09:54.240 --> 10:02.080 First, a marker, then metadata tar, it's signature image tar and it's signature and the 10:02.080 --> 10:12.960 manifest which is a list of hashes with inline NupiG signature. This can be easily read. 10:12.960 --> 10:19.680 It can also be easily identified by file and yeah lots of things we can do. 10:20.480 --> 10:30.160 We introduced gen2 certification authority. So basically 111 keys, level 2 keys and then here 10:30.160 --> 10:38.800 the release engineering key all very well defined. And on the client side we now have a small 10:38.800 --> 10:44.320 utility as long as you don't configure it differently. I mean you can configure anything. 10:44.480 --> 10:51.920 But as long as you don't configure it differently, once you first require binary packages 10:51.920 --> 10:57.920 from a server, it checks whether there's already a configuration in it. It doesn't exist. 10:57.920 --> 11:03.440 Then it generates a local trust anchor key and signs the release engineering keys 11:03.440 --> 11:09.840 that came from the installation stage. The key signatures defined the valid package 11:10.000 --> 11:22.400 signatures and many configuration options. Okay, now let's get to how the actual package building 11:22.400 --> 11:32.080 process works. It's actually surprisingly boring. It's a system the inspawn which binds various 11:32.160 --> 11:40.480 directories and runs emerge. There are some bit here, bit here. I have a subvolumes for 11:40.480 --> 11:48.480 snapshotting. We run daily updates. We are both consuming and building our binary packages 11:48.480 --> 12:00.560 and the various inspawns share package directories. We improved this a little bit by introducing 12:00.720 --> 12:08.240 Steve the job server. The idea was basically we want to use our parallelism in an optimized way. 12:09.440 --> 12:17.280 This is a token based job server that is a new development. It's handing out job tokens to the 12:17.280 --> 12:22.800 various make files to the various make processes that run in parallel. This way we can basically 12:22.800 --> 12:30.800 keep our system at a constant load no matter how many emerge processes are started or needed at 12:30.800 --> 12:42.160 a certain moment. The inspawns have different configurations. We have KDE and GNOME based, 12:42.160 --> 12:49.200 Open RC based server based. We build against different Python versions. We, for example, 12:49.280 --> 12:56.480 sometimes pull in one lucky random package from Stan General Stable. We build a ton of GCSE 12:56.480 --> 13:06.560 and Clang versions because we target developers, of course. Any different useful combination 13:06.560 --> 13:12.960 generates a distinct binary package. Any different dependency specification generates a distinct binary 13:12.960 --> 13:20.720 package. Any different dependency sub slot also generates distinct binary packages. 13:21.520 --> 13:24.720 So in a way this is a little bit like Nick sauce without S3. 13:29.360 --> 13:37.120 Right. Instructions sense. AMD 64 2 identical builders with the baseline ABI, 13:38.000 --> 13:47.920 baseline user and the V3 user, AB64, the baseline AB64 user, and the C-flux are pretty 13:48.720 --> 13:58.640 conservative O2, Prib, two-engineeric and a few error flags. The error flags are there because 13:58.640 --> 14:08.960 in addition many of the system packages are built with LTO to improve package speed and optimization 14:08.960 --> 14:17.360 in the end. Yes, we are aware that this is a pretty conservative setting. This is for convenience 14:17.360 --> 14:24.240 and for stability. If you want to, you can still compile stuff here on your own and then you can 14:24.320 --> 14:37.200 put in all the crazy flags that you want. Hardware, we have two servers at Hetzner, 14:40.160 --> 14:48.800 two times at AMD Ryzen 9 with 16 Zen 4 cores and in total 120 GCSE RAM, 14:49.760 --> 14:56.560 there are also used for the stage builds both native and Qemobased for RR64. We have 14:56.560 --> 15:03.840 one server at Oregon State University, and a pair of ultra max, that's a Neoverse and one chip 15:03.840 --> 15:15.520 with 96 cores, 256 g RAM, donated by ARM, and that's also used for the stage builds both for 15:16.240 --> 15:28.000 32bit and 64bit ARM and for the development. Yes, the result across package categories is pretty 15:28.000 --> 15:34.560 interesting because you clearly see here that with the target developers, we can see lots of 15:34.560 --> 15:42.400 clung and LADM versions, and the rest is basically this year's all also development stuff with GCSE and other 15:42.400 --> 15:48.640 languages and so on. Yeah, and then some other things like tish knives, keylips, this is Libro 15:48.640 --> 15:57.280 Office mostly here are the browsers, we end up with KDE somewhere here and here and the frameworks 15:57.280 --> 16:07.360 are not even listed anymore. But as you can see, it's a pretty large offering. Obviously, we can only 16:07.360 --> 16:19.360 do that for AMD 64 and AR64. We can't do that for every single MIPS variant. So if we are going to 16:19.360 --> 16:27.600 the less popular architectures, we had to come up with a compromise, and that compromise is 16:28.560 --> 16:34.720 we limit ourselves to the installation stage content, which is still pretty useful because it's the 16:34.720 --> 16:41.040 compiler, it's basically everything you need to bootstrap a system. And especially GCSE is often 16:41.040 --> 16:48.800 the largest package that you have. It's usually the base ESR, we export the cache of stage builder, 16:49.440 --> 16:55.120 and you end up still with a lot of things because for example MIPS is 20 independent director 16:55.120 --> 17:00.960 is for big and young little and young for three different APIs and so on and so on. 17:01.440 --> 17:08.000 This is how it typically looks like. Here's an exotic example, MIPS, big and young N64, 17:08.880 --> 17:18.400 and still, I mean this is essentially GCSE and so on and so again, focus on compiling. 17:22.720 --> 17:28.960 And the default configuration we still build from source, but we have all the prerequisites set up. 17:31.040 --> 17:38.320 And if you hand over this parameter, get pin package, it changes and you can see here 17:38.320 --> 17:45.920 the output of an emerge run, the green stuff is built from source and the pink stuff is pulled 17:45.920 --> 17:55.520 from the binary server. And as you can see, it's combined freely. You can also do some 17:55.520 --> 18:03.760 more tuning. You can basically tell it whether to respect use flags or also try binary packages 18:03.760 --> 18:11.280 that have slightly different settings. You can tell it whether you should use binary packages 18:11.280 --> 18:17.440 very dependencies change a little bit. In general, there are very safety faults in the package 18:18.240 --> 18:31.200 last remark may be on this chapter. If you don't grade G-lipsy, then it is a safe way to 18:31.200 --> 18:37.920 to the structure. So you can never do that. And also, no package that has been built against 18:37.920 --> 18:45.120 a newer G-lipsy version will ever work against an older G-lipsy version. That's why the built 18:45.200 --> 18:53.200 process automatically inserts a hard larger than version dependency on the C library into the binary 18:53.200 --> 19:01.040 package. So if you end up with an, if you ever try something like that you end up with an error 19:01.120 --> 19:15.760 message like this. So I'm not ending over. Hi, everyone. I'm Sam and I'm going to talk about 19:15.760 --> 19:23.360 the future parts. So this is something I'm quite proud of actually because we started off with 19:23.920 --> 19:28.560 how emerge handles these things is that you enable binary packages and if one is available 19:28.560 --> 19:34.560 and eligible based on your configuration, we always use it. The problem is we don't really know if 19:35.760 --> 19:41.920 your C flags and so on. They're not part of how binary packages keyed. So then they don't 19:41.920 --> 19:46.160 affect dependency resolution. They're still encoding the binary package, but they don't control 19:46.160 --> 19:54.240 what they're selected. So what often happens is you get people who are quite happy to have binary 19:54.320 --> 20:00.240 packages for most things like Qtweb engine or Firefox. They're usually not patching these things. 20:00.240 --> 20:05.680 These are very large, very sophisticated code bases. Most people do not have an interesting 20:05.680 --> 20:10.480 patching these and if they were to, they'd have to rebase all the time anyway. So what they often 20:10.480 --> 20:15.840 want is they want to use binary packages for those, but they have like their own packages of 20:15.840 --> 20:21.120 choice, like maybe get or something, where they have patches to customize, preferences, whatever, 20:21.120 --> 20:25.040 and they always want to use that for the source. At the moment that's not actually possible, 20:25.040 --> 20:31.520 which is strange, but it's a nature of how the metadata words were used like. So I did what any 20:31.520 --> 20:38.000 good false means anything it does and this is quite a frequent request. Someone came in and they said 20:38.000 --> 20:45.520 hey, can portage do that, can I disable selectively just by package name and I had to say no again 20:45.520 --> 20:51.440 to a very disappointed user. But of course I tried my life and I said do you want to give it a go? 20:53.280 --> 20:58.880 And they said yes and I'm very, very pleased with this and they've actually implemented the whole thing. 20:58.880 --> 21:08.160 So they've, yeah, so they've first they implemented it as like a nice little UX thing, 21:08.160 --> 21:12.560 where these are all local packages and then you have a little G when they're being fetched. 21:12.800 --> 21:18.000 Okay fine, but then more importantly, suppose QT web engine, you want to use it as a, 21:19.920 --> 21:23.680 you want to use that as a binary, but like suppose you've patched some of its dependencies. 21:24.400 --> 21:29.600 You can pass a special argument now or you will be soon on the configuration files as well, 21:29.600 --> 21:34.800 where you can just say I'd really really like QT web engine as a binary, but please leave the rest alone 21:34.800 --> 21:39.120 and that does work. Yeah, this is quite well-seaved to draw you to this. 21:39.760 --> 21:44.240 The other thing that's coming soon is Debug InfoD. 21:47.280 --> 21:51.280 A lot of these things, they're things which, in binary distributions, 21:51.840 --> 21:55.920 you're quite used to because they had to have them from the start, whereas for us, you know, 21:55.920 --> 22:02.320 the default has always been source builds. It will continue to be forever, but we encourage people 22:02.320 --> 22:06.800 to use binary if they wish and if they stood there use cases. So we had to deal with some 22:06.800 --> 22:11.520 house serving these Steveug Info on the side, right? Because if we put them in the binary 22:11.520 --> 22:15.760 packages themselves, it would bloat them a lot of the time people aren't interested. Like, 22:15.760 --> 22:20.880 you'll need to be bugging something goes wrong, right? So first it was just about enabling 22:20.880 --> 22:27.520 their lighties, which is easy to tell GGC, it's always passive-grounds of the linker, and the 22:30.240 --> 22:35.920 it was a bit of wiring, essentially. You had to, I had to implement an important 22:35.920 --> 22:40.800 image that it would produce packed debug tables, which is where, essentially, when splitting the 22:41.840 --> 22:46.560 the final debug information out, we produce a little table on the side in a new hierarchy, 22:46.560 --> 22:53.120 which Debug InfoD can serve. I was very straightforward, Arson produced the proof of concept, 22:53.120 --> 22:58.560 and then I sort of polished it and integrated the importage. So that part's done, and hopefully soon 22:58.560 --> 23:04.880 we'll be able to provide Debug InfoD, which for me was actually a kind of a blocker for the next point. 23:06.560 --> 23:13.360 So LTO. A bunch of distributions now, like, these ship LTO, but default, and LTO is great, 23:13.360 --> 23:18.640 it's in time optimization. You give the compiler far more freedom to optimize, and this is useful 23:18.640 --> 23:26.400 both for size and performance. One of the things that we found was, well, even though people had 23:26.400 --> 23:32.240 been shipping this for a while, the problem is, while in Gen2 we default to the upstream configuration 23:32.240 --> 23:39.280 usually we can match the distributions, too. The problem is, with use flags, you have these end 23:39.280 --> 23:46.480 different variants, right? These usually map into pre-procise conditions. For example, you have 23:46.480 --> 23:55.520 some strapped food, and it has some memory that depends on macro that's in a header. If I had 23:55.520 --> 24:01.120 there isn't consistent included, you'll have two different sizes for the food strapped. So we 24:01.120 --> 24:06.960 pass a bunch of error flags, which Andreas mentioned earlier in order to catch these. I've 24:06.960 --> 24:11.920 spent quite a while on this. We've fixed many, many, many packages. We've bought many of them 24:11.920 --> 24:16.560 streamed, but we've played a patch as well. So my hope is soon, we'll be able to enable LTO, 24:16.560 --> 24:21.920 in the big host as well, and get smaller dynamics. You may notice, actually, that the diagnostic here 24:21.920 --> 24:26.080 isn't great, because it just says, food and food, they're the same drink twice, right? But you 24:26.080 --> 24:30.080 can't see all the actual differences. Like, I found a bunch of GCC boats, and I think I may have 24:30.080 --> 24:34.320 got fixing something as well, so that you can actually show what the mismatch is. 24:36.480 --> 24:43.360 One other thing, and again, a lot of these issues have been, they are problems that always 24:43.360 --> 24:47.760 existed, but when it comes to source build, they didn't matter so much, but then we're buying 24:47.760 --> 24:57.200 your eases they do. LTO, when you use normal dynamics libraries, there's no bit code in that's fine. 24:57.280 --> 25:05.440 When it comes to static libraries, it's essentially a dump of IR, and we work with some of the 25:05.440 --> 25:11.040 videos as people are stripping out all that junk, and so we can distribute them freely now as well. 25:11.840 --> 25:17.440 It was an awful lot of code, and most of it was texts, but yeah, quite a wrap in there.