WEBVTT 00:00.000 --> 00:17.920 Okay, so I'm going to start off with your session. 00:17.920 --> 00:27.120 So Stefan and myself, we're going to be talking about the issues of maintaining an Android 00:27.120 --> 00:36.320 release and how you interact your changes with the upstream AUSP. 00:36.320 --> 00:42.720 So it's called forking, Android considered harmful, and as you can guess the basic idea behind 00:42.720 --> 00:50.480 this is that we should try to avoid forking wherever possible. 00:50.480 --> 00:59.840 So the agenda then, Stefan is going to talk a little bit about the different ways that 00:59.840 --> 01:07.240 you can cope with a changing upstream as a changing AUSP and how you can avoid or how you 01:07.240 --> 01:13.440 typically interact with that and different ways of interacting with that. 01:13.440 --> 01:18.720 And then I'm going to come in and talk about why forking is a bad idea and some ideas 01:18.720 --> 01:31.520 I'd like to discuss both in this session and also offline about what we can do about this. 01:31.520 --> 01:36.720 So quick and true to myself, I think I'm reasonably well known. 01:36.720 --> 01:42.040 I've been doing this stuff for quite a long time. 01:42.040 --> 01:43.280 Main topic is of interest. 01:43.280 --> 01:47.240 The point where software meets hardware, I've always been interested in the interface 01:47.240 --> 01:52.480 with devices and I like to see actual hardware if possible. 01:52.480 --> 01:58.760 I've experienced with Linux and its kernel, Yopto and Android. 01:58.760 --> 02:04.360 I'm very interested in open source and I'm very interested in the idea of communities. 02:04.360 --> 02:06.160 So that's me. 02:06.160 --> 02:09.400 Thank you, hand over to Stefan. 02:09.400 --> 02:11.680 Hi, I'm Steve M. Lankfeld. 02:11.680 --> 02:13.400 I'm living in Germany. 02:13.480 --> 02:15.080 I'm working for Innovax. 02:15.080 --> 02:20.600 We are doing projects and consultancy and I'm doing this and that stuff, Linux and that and what 02:20.600 --> 02:25.200 embedded stuff since 10 years now and yeah, main interest and that it's a system, 02:25.200 --> 02:29.960 things can't be developed and built system, perfect stack and performance analysis set 02:29.960 --> 02:35.040 for myself and the short introduction to the problem is again you. 02:35.040 --> 02:36.040 Sorry, good. 02:36.040 --> 02:39.760 Next slide is slide. 02:39.800 --> 02:43.840 Okay, so yeah, there's P developers, Dalema. 02:47.040 --> 02:50.480 So the first point is the basic one. 02:50.480 --> 02:58.480 So in order to make a product out of an AWS P, you basically need to take a copy of 02:58.480 --> 03:02.320 a USB and then modify it in some way. 03:02.320 --> 03:04.640 So what do we need to do this? 03:04.640 --> 03:09.720 So typically you're adding in board support for whichever hardware platform you're using. 03:10.360 --> 03:19.160 Often you need to put in some custom how support we need to add in services, libraries, 03:19.160 --> 03:29.600 apps and we may also want to change the upstream ASP code, we may want to change the 03:29.600 --> 03:34.760 way the framework works in some ways because we need some different launchary behaviors. 03:34.760 --> 03:38.360 We will of course need to add in some SC policy on all kinds of stuff. 03:38.360 --> 03:45.480 So in order to do anything with AUSP apart from just build an emulator, you need to hike 03:45.480 --> 03:47.320 it around a bit. 03:47.320 --> 03:51.320 So how do you handle that? 03:51.320 --> 03:57.280 And that we go into the section and how we are handling with that. 03:57.280 --> 04:01.600 They are typically so we have to look at two different things. 04:01.600 --> 04:08.000 So one thing as at the one point often have to fork something or at least to patch some 04:08.000 --> 04:09.000 existing code. 04:09.000 --> 04:15.680 So patching framework code for example to fix a bug or back for the patch for a fix or adding 04:15.680 --> 04:19.440 new features in framework but of course this should not be done too much because 04:19.440 --> 04:21.880 and plotting is harder to do versions. 04:21.880 --> 04:27.760 So one thing we always do is fork or patch existing code from the ASP source tree or from 04:27.760 --> 04:32.560 the window tree you get from the SOC window and the better thing is the additions. 04:32.560 --> 04:38.240 So a lot of things we add to the ASP tree for example, how implementations, kind of 04:38.240 --> 04:46.040 drivers, additional apps and other things and both forking and additional patching and 04:46.040 --> 04:48.040 adding code. 04:48.040 --> 04:52.680 I do different tasks that we need to do as ASP developers and in the options I will now 04:52.680 --> 04:55.600 present also done differently. 04:55.600 --> 04:59.160 So we will go to the options that I have seen in the white. 04:59.400 --> 05:00.160 One question? 05:00.160 --> 05:10.440 Okay, no speakers, this mic is only for the so I have to speak louder, you are saying 05:10.440 --> 05:17.440 I should have been louder, okay, okay I tried to speak louder, so now we look at the 05:17.440 --> 05:24.640 four different methods I have seen in the ASP in the community or in projects, how 05:24.640 --> 05:29.280 we deal with that, so we look at the good, the bad and the ugly nothings. 05:29.280 --> 05:36.440 We use at the copy script option I have seen which we are using patches, we are looking 05:36.440 --> 05:46.440 at the local manifest feature from the repo tool and we are looking at the way with copying 05:46.440 --> 05:51.080 and extending the upstream repository repo manifest XML manifest. 05:51.080 --> 05:55.240 That does it for options, we are now going through. 05:55.240 --> 06:00.600 So using a copy script, so we have the upstream for example the upstream ASP source 06:00.600 --> 06:05.840 tree from Google or from a BSP vendor and you are the downstream, you are the downstream 06:05.840 --> 06:11.760 project and in with this option what you are doing is you have kind of in the bad case 06:11.760 --> 06:15.800 you have a separate archive with code, a customer has a separate archive with code or you 06:15.800 --> 06:20.880 have a git repository and in this git repository you have files from the 06:20.880 --> 06:27.760 ASP copied and modified, this is maintained by you, this git repository is maintained 06:27.760 --> 06:33.480 by you and then you are white, a weak me file or you have a weak me file where you specify 06:33.480 --> 06:40.960 the upstream manifest version from Google or from your BSP vendor and the version, so someone 06:40.960 --> 06:47.240 who is working on the code then knows on which version you have based on the copied files 06:47.240 --> 06:53.040 and modified, and then to be very convenient you have a copy script, so you place also a 06:53.040 --> 06:58.960 script into the git repository to then copy the patch files over to the actual ASP source 06:58.960 --> 07:04.640 tree where it was actually built, and how does it check out local like, so you first do 07:04.640 --> 07:09.920 your repo in it from the upstream, except all manifest from Google for example, then you 07:09.920 --> 07:15.840 do a repo sync to download on all the code, then you click git clone the downstream repository 07:15.840 --> 07:20.680 so you code or the code of your customer or your downstream project and then you execute 07:20.680 --> 07:26.920 this copy script, and with this way then you have a patched MSP source tree for your device 07:26.920 --> 07:35.640 or for your customer, and just I will not go much into the pull of cons, of all these 07:35.640 --> 07:41.760 options but I personally don't like this option, but I've seen it, for example, I've seen 07:41.760 --> 07:46.800 it in one upstream git repository here and I've seen it by customers, the problem is it's 07:46.800 --> 07:53.520 very hard to sync the files, the modified files in the ASP source tree and the modified files 07:53.520 --> 07:58.720 in the downstream git repository, it's very hard to track these changes, but I've seen it 07:58.720 --> 08:06.640 and in simple case it works, what also I have seen in the wide is patches on top, 08:06.640 --> 08:12.480 I call it patches on top, how does a downstream project does it, it maintains a git repository 08:12.480 --> 08:19.600 of patches, so not modified files, but patches get patches in the other descriptions, 08:19.600 --> 08:25.600 10 patches, 100 patches, then it also writes a written file to specify the upstream, manifest 08:25.600 --> 08:30.600 an upstream version from Google or the BSP vendor, and also it has a convenient script to 08:30.600 --> 08:38.200 apply the patches on the ASP source tree, how does it look like again, we put in it from 08:38.200 --> 08:44.280 the upstream, it's a manifest, we put sync, you get cloned the downstream project, you 08:44.280 --> 08:48.600 change the directory in the downstream project, and then you execute the convenient script 08:48.600 --> 08:53.320 to apply all the patches, and then the patches are applied to ASP source tree and then you 08:53.400 --> 09:03.400 can build and you are your device enabled, and I've also seen some examples that 09:04.360 --> 09:11.160 in the reload project, in the Pinephone project, and also Chris has a small project, this also 09:11.160 --> 09:21.000 uses patches, so let's before we go to the start option, we look at repo, manifest trick 09:21.000 --> 09:27.400 now that you may know or may not know, it's the removed project trick, this is repo, manifest, 09:28.680 --> 09:36.280 manifest for a small project, you see these lines are adding git repository, so here we are 09:36.280 --> 09:42.360 talking about the addition, but below here we see the, we move project line, that is about 09:42.360 --> 09:47.640 forking, so how we can remove every repository from the upstream and replace it with our own fork, 09:47.640 --> 09:54.520 and this works with the remove project, it's an alt tag, you write down the name or the path 09:54.520 --> 10:02.920 of the repository, then it's been moved, and then since always only adds a new fork of it, 10:02.920 --> 10:08.520 you see that it's a fork because the remote is pointing to cool project, it's not pointing 10:08.520 --> 10:17.240 to the upstream anymore, and cool project is our host on GitHub, so with this, you can do the 10:17.240 --> 10:23.720 option 3, it's the local manifest, the local manifest is a feature by the repo 2, how there's 10:23.720 --> 10:29.160 a downstream project use it, it would mean times the git repository will see local manifest xml, 10:30.040 --> 10:38.040 so for example, there's some git repositories added, some removed, this is this xml, 10:38.040 --> 10:44.120 one is in this git repository, again I read me file to specify the versions because you 10:44.120 --> 10:49.400 develop and needs to know that, and then it also adds and forks some repositories of the 10:49.400 --> 10:54.840 AOSP, and plays it on GitHub or somewhere, how does he check out, look like it's again a bit, 10:54.840 --> 11:00.600 it's now a bit shorter, repo in it from the upstream, then we do git clone from the downstream 11:00.600 --> 11:08.040 and project, and the important point is that we clone into the path dot repo slash local manifest, 11:08.120 --> 11:15.400 this is the repo, look in manifest feature, if xml files are in this folder, then it's picked up, 11:15.400 --> 11:21.240 we repo and then we do a repo sync and the whole source tree from the upstream source and now 11:21.240 --> 11:30.120 also from the downstream source, and here also some examples, I've seen from one recipe pie and 11:30.120 --> 11:38.440 Android fork, and also here one from snap mode of, now we go to the fourth option, but before 11:38.440 --> 11:46.200 we do the fourth option, another repo trick you might know of it's include, it's a small trick to 11:46.920 --> 11:58.040 make it more easy to maintain a copy, or maintain your manifest, it's included, you see here, 11:58.040 --> 12:04.600 is that if you have an xml manifest, you can specify the include tag, and if it's included, another 12:04.600 --> 12:13.960 text file is included in this manifest, for example here I copied the upstream manifest from google, 12:13.960 --> 12:24.840 versions 13, and then my own additions, my own forks new projects I added in another xml file, 12:25.080 --> 12:33.960 and with this you can easily maintain a downstream fork and yeah, and with this we come to the 12:33.960 --> 12:41.560 fourth option, if the option copy and x10, the upstream manifest, so now you as a downstream project, 12:41.560 --> 12:48.840 you maintain a own manifest repository, you are not referring to a request, a manifest repository 12:48.840 --> 12:55.320 from google, or for the bsp vendor, in that you copy and include the upstream manifest from google, 12:55.640 --> 13:01.720 and you also again add forks, you add and add and forks some repositories, how to check out 13:01.720 --> 13:07.160 it's now where we easy, we put in it from your downstream manifest repository repo sync, 13:07.160 --> 13:13.400 and that's it, and you have Cloncy source code, and examples, believe it does it, project 13:13.480 --> 13:20.840 celladone does it, and I cannot spell it, collects OS does it too, and this is the fourth 13:20.840 --> 13:26.920 option, how you maintain your downstream fork of android, easy-wired, and with that I hand over to 13:26.920 --> 13:38.200 Chris to talk about why forking is bad, how to improve that. Okay, so all of the examples we've seen 13:38.280 --> 13:48.360 really are in some way rather forking the upstream project, so yeah, why is this a bad thing, 13:48.360 --> 13:53.240 so you may say well it's open source, we can do whatever we like with it, we can hack this around, 13:53.240 --> 14:04.840 whatever, however, forking any project is always about idea, it has a maintenance burden, 14:04.840 --> 14:13.000 it's the main problem, so once you made the fork, you are now responsible for the for that code 14:13.000 --> 14:20.840 for applying security fixes, bug fixes, whatever else, so you have you've taken on the responsibility 14:20.840 --> 14:33.800 for maintaining this code, and all you can do, you can, instead of backwarding stuff at here 14:33.800 --> 14:42.680 into your fork, you can continually rebase your fork on the upstream version, but that also has 14:42.680 --> 14:46.440 some overhead because when you do the rebase, you're bound to have some conflicts and you can 14:46.440 --> 14:53.880 have to resolve them and so on. Next point is something slightly more subtle, I think, 14:54.520 --> 15:01.320 when you make a fork and you start making changes to the code base, those changes tend not to be 15:01.320 --> 15:09.000 well separated, so developers often don't differentiate too much why they're making the changes, 15:09.000 --> 15:13.720 and they'll be changes to get commits, but they don't necessarily relate to each other, so you 15:13.800 --> 15:21.560 end up with a mismatch of stuff, so here's our SP, here's our fork, we change some stuff, 15:22.120 --> 15:26.920 you can look in the Git logs to find out exactly what we change, but it's not really that obvious. 15:29.480 --> 15:35.080 As a result of all this, we typically end up when you're writing, when you're creating your own 15:35.160 --> 15:44.440 Android device, with a barrier which prevents you migrating to a new version, so it's quite 15:44.440 --> 15:49.000 common to find Android devices running old versions of Android, for example Android 9, 15:49.720 --> 15:55.800 and running old versions of kernels, and yeah, it's kind of a nightmare. So to summarize, 15:55.800 --> 16:01.640 forking equals technical debt, we would like not to fork if we can avoid it. 16:05.080 --> 16:14.040 So, how do you do this? Fundamental rule then is don't change a USB code base, 16:14.040 --> 16:22.680 including do not change the manifest. What can you do? Well, you can add stuff, so you can add 16:22.680 --> 16:30.760 stuff into the local manifest, and this is the big thing here really is to enhance patching. 16:31.000 --> 16:38.600 So instead of changing the code in situ, create a series of patches, and apply those patches 16:38.600 --> 16:47.960 at built-in. Additional benefit, if you do this in a clever way, you can organize these patches 16:47.960 --> 16:53.640 into groups, which are going to call layers, which can be applied independently of each other, 16:53.640 --> 16:59.880 and you have a much more modular system. My inspiration for this really came from a number of 16:59.960 --> 17:04.760 places. I'm going to document two here. I'm going to mention first of all, Glodroid. 17:06.280 --> 17:12.840 So Glodroid, as we mentioned, is an ASP distro for various boards, including Raspberry Pi's. 17:14.520 --> 17:21.240 And it's interesting if you look at Glodroid 2.0, they change the way that they are interacting 17:21.240 --> 17:27.400 with the upstream ASP, and they're using what they call a mono repository approach, which is 17:27.480 --> 17:34.040 more or less what I just said. This is a little small, and I don't expect you to read all of it, 17:34.040 --> 17:41.720 only one go. The key thing is, no more forking. Okay, so there are ideas, what I just said 17:41.720 --> 17:48.760 in a couple of slides back, instead we will take ASP, we will create a set of patches, and then we 17:48.760 --> 17:54.680 have some built-in tools, which will apply those patches to the ASP. This makes it much more easy 17:54.680 --> 17:59.960 to maintain those patches. Also, notice this comment here, it says decouple devices or 17:59.960 --> 18:04.680 device groups from each other. In other words, we group those patches together into different 18:04.680 --> 18:14.920 functional areas, one for the changes to the ASP code base, some are device specific. 18:15.000 --> 18:24.440 So, that's my first inspiration. My second source of inspiration is from York to Project. 18:25.800 --> 18:32.840 How many people here are familiar with York 2? Okay, about half. Good, right. 18:33.640 --> 18:43.000 So, York 2 is better than Android, yes? I think that was a yes. I definitely heard a yes there. 18:45.400 --> 18:52.200 Okay, so York 2 essentially does the same thing as the ASP build system, 18:52.200 --> 18:59.480 except it does it better in my opinion. Just maybe in a few concepts then, for those of you 18:59.480 --> 19:06.680 are not familiar with it, maybe in York 2 concepts on to ASP concepts. So, in York 2, we have 19:06.680 --> 19:14.680 everything called BitBake, which does the same job as soon. In York 2, we have recipes which are 19:14.680 --> 19:22.120 Bb files, BitBake files. They are more or less exactly the same in concept as Android Bb files and 19:22.120 --> 19:31.000 make files. I'm not going to go into this in too much more detail, but York 2 also has classes, 19:31.320 --> 19:37.240 BitBake is written in Python, the classes are basically bits of Python code. 19:38.520 --> 19:45.800 That's where the real logic is implemented. In the ASP, the main build logic is implemented in 19:45.800 --> 19:58.280 the build directories in build make and also in build soon. So, we have this metadata, which 19:58.360 --> 20:07.560 are recipes, classes, and config files. And the key concept I want to borrow from York 2 here 20:07.560 --> 20:15.480 is the idea of a layer. So, we have the called meta layers. So, a meta layer is just a collection 20:15.480 --> 20:24.040 of metadata, recipes, classes and so on. And the key thing here is there are several different types 20:24.120 --> 20:32.040 of layer. So, we have BSP layers. For example, Raspberry Pi would be a good example. We have 20:32.040 --> 20:37.800 distros, distros is a slightly complicated thing which I don't want to go into now, but for example, 20:37.800 --> 20:44.760 AGL, automatic grade Linux is A distro. So, if you want to build an AGL version of York 2, 20:44.760 --> 20:52.040 you just include that layer. And it has what they call software. So, these are typically libraries. 20:52.040 --> 21:01.080 So, meta, Qt would be a good example. Qt is the graphics library. And then when you're building 21:01.080 --> 21:05.640 a product, you just patched together the layers you want. You add in the board support, whichever 21:05.640 --> 21:10.760 board you want. You add in the distro for whatever you want it to do basically. And then you add 21:10.760 --> 21:20.520 in software libraries. Kind of neat. There is an official list of layers at the layer index. 21:21.160 --> 21:25.400 And if you go look there, you'll see there's four or five hundred layers you have to choose from. 21:26.280 --> 21:32.600 So, you can build things together quite neatly. So, why can't we do something similar with Android? 21:35.880 --> 21:43.640 Okay, so this is, okay, I'll misspell that as well. This is a very early stage, but I kind of want 21:43.640 --> 21:52.120 to get to feedback either now or later on. So, how do we apply layers to Android in two minutes? 21:54.120 --> 22:00.280 Essentially, the ASP layer basically is a collection of local manifests of patches and also some 22:00.280 --> 22:07.080 meta data explaining what the layer does. And also documenting incompatibilities, which version of 22:07.080 --> 22:17.400 Android does it work with, that kind of thing. Do I think something to layers? We can then 22:17.400 --> 22:22.840 combine these layers in different ways to add in what to make whatever we want. So, we could have 22:22.840 --> 22:28.040 in, we can have a board support layer, we can have some house, we can have build system changes, 22:28.040 --> 22:36.040 and so on. And so, we don't want my wonderful diet. Anybody who has seen my lectures will know 22:36.120 --> 22:41.560 my diagrams or a rubbish, and this is a good example of that. But the basic idea is that essentially 22:41.560 --> 22:48.840 when you import the layers that you want. So, this will be in a directory somewhere on your 22:48.840 --> 22:55.800 system. Here I have three layers. I have my BSP, which is the hardware support. I have a Geno 22:55.800 --> 23:01.640 audio, how, and I've got a launcher. And I've expanded out what you might see under one of these. 23:01.720 --> 23:09.400 So, for the audio how, we have some changes to the hardware directory, a bunch of patches. We 23:09.400 --> 23:14.920 have, of course, some SE policy, because you always need SE policy, a bunch of patches. So, 23:14.920 --> 23:21.000 you download the layers you want. You then apply some magic command to apply the patch that 23:21.000 --> 23:29.960 patches AUSP in a control way, and you build a product. So, this brings me onto another concept 23:30.040 --> 23:36.760 really, really, which is a product. So, product is a collection of layers, a layers of magic 23:36.760 --> 23:45.240 metadata. The product then is a single entity, most likely a file, possibly in JSON format or something. 23:46.120 --> 23:55.320 So, I can then say, go get me the product description from some URL. It will grab the product 23:55.400 --> 24:06.120 description. It will download everything, apply the patches, and away we go. So, to make this work, 24:06.120 --> 24:16.200 we need some tooling, TBD. And just as one final thing then, the idea is that, if I would say, 24:16.200 --> 24:21.560 I want to get a product. I want to build, I don't know, an automotive Android for a Raspberry Pi. 24:22.440 --> 24:27.320 Somebody somewhere has written the JSON file, which explains how to do that. So, I just say, 24:27.320 --> 24:34.520 build Android product, point to that JSON file. It will then read the pilot file, do a repo, 24:34.520 --> 24:40.920 a knit for me to get whichever version of Android it specifies. Download the local manifest into the 24:40.920 --> 24:49.160 directory, do some checks. Do a repo sync in order to bring in all of the local manifest changes, 24:49.240 --> 24:55.080 apply the patches, and then you type M and go away for a long holiday while it builds it for you. 24:58.600 --> 25:04.840 So, that's the basic idea. As I said, this is very much a work in progress. I would very much 25:04.840 --> 25:11.320 value feedback from everybody here. Do you have time for like maybe one question? 25:11.400 --> 25:25.240 Yeah, I'm in charge. So, we have time for one question. So, anybody, I totally overwhelmed. 25:27.800 --> 25:34.840 Okay, this is a good reason then to do the wrap up now. Thank you very much, and let me know what you think.