WEBVTT 00:00.000 --> 00:13.640 Around a glass, please, for Paul Meier, who's going to talk about, go in the next ecosystem 00:13.640 --> 00:20.760 and some interesting work he did in the area. 00:21.760 --> 00:27.760 Welcome to my talk. I'm Paul. My guitar panel is Cut X-O-Kin. 00:27.760 --> 00:31.760 I'm also a master-dom, I'm a security software engineer. 00:31.760 --> 00:35.760 Leak spectators, maintainer, and I'm also part of the Leak Specter good. 00:35.760 --> 00:40.760 Take it to the scope team, so if you're wondering why you didn't receive the latest GoVinability update, 00:40.760 --> 00:46.760 I would tell you that it is currently in staging next and you have to wait while applying overlay. 00:46.760 --> 00:54.760 So I want to quickly give another view over the GoV dependency management and general outside of mix. 00:54.760 --> 01:02.760 GoVin manages dependencies as modules, which is the collection of packages that is released version and distributed together. 01:02.760 --> 01:05.760 So this is usually a repository scope. 01:05.760 --> 01:12.760 We have to go that not file that declares minimum required versions of module dependencies. 01:12.760 --> 01:18.760 Let's also go that some file that contains cryptographic hashes for these dependencies, 01:18.760 --> 01:25.760 but to go that some file is not an exact log file, and also the hashes in there are incompatible with nix, 01:25.760 --> 01:29.760 so we cannot use import from javiation here. 01:29.760 --> 01:40.760 And then you go use this minimum version selection to, which is a deterministic algorithm to select the actual versions that I used in the build list. 01:40.760 --> 01:46.760 And only at the one version of a dependency is selected in the end. 01:46.760 --> 01:50.760 And use a quick look at the GoV Mod file. 01:50.760 --> 01:58.760 Just to get an impression, the required statements declare or direct, and at the bottom indirect. 01:58.760 --> 02:01.760 So transfer to dependencies. 02:02.760 --> 02:09.760 So now come to the nix part, how do we build or package go stuff nix. 02:09.760 --> 02:14.760 In nix packages we have a single build up of this called the GoMod view. 02:14.760 --> 02:17.760 And for most, this might be straightforward. 02:17.760 --> 02:23.760 We have a source that we fetch from GitHub, and we have a package name and version. 02:24.760 --> 02:32.760 And what I really want to point out here is the vendor hashes at the bottom, which pins all dependencies. 02:32.760 --> 02:38.760 So we will take a closer look at this, and also at the delay, the relations structure here. 02:38.760 --> 02:44.760 So we have one derivation of, which is by GoMod view, the thing we want to build. 02:44.760 --> 02:51.760 Then we have the source, which is a fixed output derivation that pulls our source from GitHub. 02:51.760 --> 02:56.760 And then we have a third derivation here called GoMod Views. 02:56.760 --> 03:00.760 And this depends on the source. 03:00.760 --> 03:08.760 And use the GoV Mod file and the Go command to download the dependencies of the module we want to build. 03:08.760 --> 03:10.760 Interfixed output derivation. 03:10.760 --> 03:17.760 And then this is pinned by the vendor hashes. 03:18.760 --> 03:22.760 So if you now look at two different GoPackatures pattern pin act. 03:22.760 --> 03:27.760 Both will have GoMod Views, derivation they depend on. 03:27.760 --> 03:32.760 And yeah both will have some Go dependencies in there. 03:32.760 --> 03:38.760 And as we see here, now these two packages depend on the same GoMod Views. 03:38.760 --> 03:43.760 They have the same dependencies. 03:43.760 --> 03:47.760 But in next, we do not have any introspection in each of this. 03:47.760 --> 03:52.760 So we will download the same dependency again. 03:52.760 --> 04:00.760 And yeah, we do not know that these two fixed output derivations actually have the same content. 04:00.760 --> 04:07.760 So what does this mean for the current status and next packages from GoMod View? 04:07.760 --> 04:10.760 We have to download things multiple times. 04:10.760 --> 04:14.760 We are storing these dependencies over and over again. 04:14.760 --> 04:15.760 Very important. 04:15.760 --> 04:18.760 We don't have any form of caching. 04:18.760 --> 04:23.760 So we have to rebuild the dependencies over and over again. 04:23.760 --> 04:27.760 Both between two different packages. 04:27.760 --> 04:32.760 But also in case we change something about our actual package. 04:32.760 --> 04:33.760 We want to build. 04:33.760 --> 04:36.760 There's no caching for the build of the dependencies. 04:36.760 --> 04:42.760 So we also have to rebuild our dependencies in this case. 04:42.760 --> 04:48.760 Then we do not have any introspection into the dependencies in nix. 04:48.760 --> 04:54.760 It is really hard to update dependencies of the package. 04:54.760 --> 04:59.760 For this we would actually need to create a patch for the GoMod file. 04:59.760 --> 05:04.760 And we would have to do this for each GoMod package individually. 05:04.760 --> 05:12.760 And following we do not have an easy way to patch vulnerabilities. 05:12.760 --> 05:15.760 So first I want to look at the vulnerability aspect. 05:15.760 --> 05:16.760 How bad is this? 05:16.760 --> 05:18.760 I've created a small project. 05:18.760 --> 05:20.760 GoMod check nix packages. 05:20.760 --> 05:26.760 GoMod check is the official vulnerability scanning tool of GoMod. 05:26.760 --> 05:34.760 And it has a separate vulnerability database containing CVE. 05:34.760 --> 05:39.760 But also go vulnerabilities that are not assigned to CVE. 05:39.760 --> 05:45.760 GoMod check is quite cool because it checks if the vulnerable code is actually reachable. 05:45.760 --> 05:50.760 So you get less false positive in case there would just import a vulnerable package. 05:50.760 --> 05:59.760 But the reason isn't a core path that would make your code reach the vulnerability. 05:59.760 --> 06:13.760 And then go Vynscheck nix packages runs GoMod check on the source of all GoPecaches and nix packages that are built with GoMod. 06:13.760 --> 06:15.760 So what are the results here? 06:15.760 --> 06:21.760 And black we see the packages that are discovered by GoMod check nix packages. 06:21.760 --> 06:26.760 This cover is currently based on simple attributes of the output. 06:26.760 --> 06:32.760 Then in blue we have the packages that are actually scanning successfully. 06:32.760 --> 06:35.760 This is just like a really simple project. 06:35.760 --> 06:41.760 For example, for packages that you see go, we do not provide the cd dependencies. 06:41.760 --> 06:46.760 So the vulnerability check I will just fail because the dependencies aren't there. 06:46.760 --> 06:50.760 So we are missing out on some of those. 06:50.760 --> 07:00.760 And then in red we have the number of vulnerable packages, which is quite stable around 30% of the scan packages. 07:00.760 --> 07:05.760 And then in yellow we have the number of vulnerabilities in total. 07:05.760 --> 07:12.760 We have like around three vulnerabilities per vulnerable package. 07:12.760 --> 07:24.760 Just remember we don't have like an easy way to fix vulnerabilities, especially in that at this scale. 07:24.760 --> 07:29.760 So now I want to to make a jump to the caching problem again. 07:29.760 --> 07:38.760 And just yeah, how the cache usually works with Go is we have like an environment variable GoCash, 07:38.760 --> 07:51.760 which points to a five system path and there's our cache located and go will read and write to this path and manage the cache itself. 07:51.760 --> 08:04.760 And now there's a really cool feature in go124 and also in earlier versions as experimental feature that lets us provide binary code GoCash Proc. 08:04.760 --> 08:09.760 And yeah, this gives us a lot more flexibility. 08:09.760 --> 08:16.760 Go will just invoke the binary during the bit presses and talk a JSON based protocol. 08:16.760 --> 08:24.760 He has done that in standard out with two basic operations, get to request a cache entry via an external idea. 08:24.760 --> 08:31.760 And then in case we got the cache miss, go will build the package and then call put. 08:31.760 --> 08:40.760 So the go cache Proc can provide the cache result somewhere. 08:40.760 --> 08:47.760 And at this point I want to introduce goboots.next, which is next generation build up for go. 08:47.760 --> 08:51.760 I use this go cache Proc. 08:51.760 --> 09:00.760 And just so you get an idea, goboots.next tries to provide the package sets of go dependencies. 09:00.760 --> 09:05.760 Where each go module has its own derivation. 09:05.760 --> 09:12.760 You will try to module the go dependency graph index. 09:12.760 --> 09:18.760 And first I want to show how go.next uses go cache Proc. 09:18.760 --> 09:26.760 So we have written go build nix cache binary that implements go cache Proc. 09:26.760 --> 09:35.760 And the way this works is we have an environment variable where we read a number of paths from. 09:35.760 --> 09:44.760 And then go the cache over will save the cache entries found at these paths. 09:44.760 --> 09:49.760 And the paths will point to the next door. 09:49.760 --> 09:59.760 And in case we have a cache miss, the cache just writes the build result into a defined output. 09:59.760 --> 10:06.760 And here we can add the stop paths we set up hook. 10:06.760 --> 10:11.760 So we put the go module dependencies into our build input. 10:11.760 --> 10:21.760 The set up hook will add the stop path to the environment and then the go cache the cache can serve easily. 10:21.760 --> 10:25.760 Now looking at this from a derivation level. 10:25.760 --> 10:32.760 So if we want to build GitHub.com, full slash a, we have the source in green. 10:32.760 --> 10:38.760 Then we have the source of our dependencies, which is required for the build process. 10:38.760 --> 10:49.760 We match that with the source of full slash a by linking the dependencies and so the vendor directory. 10:49.760 --> 10:53.760 And we also provide the cache. 10:53.760 --> 11:00.760 So the build output of when we build the dependencies to the cache. 11:00.760 --> 11:07.760 And now the idea is that we will not have to rebuild the dependencies because they are already cached. 11:07.760 --> 11:17.760 The cache will serve the build cache for the dependencies and we will only need to rebuild the packages that belong to full slash a. 11:17.760 --> 11:23.760 And for these packages, the output of the build process is written to out. 11:23.760 --> 11:32.760 And then the final build result will contain the cache entries for full slash a. 11:32.760 --> 11:37.760 And looking at this at the package set perspective. 11:37.760 --> 11:41.760 And these packages will construct the package set. 11:41.760 --> 11:45.760 And mainly consisting of the build results. 11:45.760 --> 11:51.760 The cache entries that the cache are stored in the output. 11:51.760 --> 12:00.760 Also, quite important is that we have to have the source of our transitive dependencies available for build. 12:00.760 --> 12:10.760 And if we now want to build a go binary, we would do that outside of the package set and just add all the built inputs. 12:10.760 --> 12:14.760 As usual, so the go modules as built and put. 12:14.760 --> 12:23.760 And we would have all the dependencies already available and also built. 12:23.760 --> 12:29.760 And yeah, every dependencies only available in one single version in this package set. 12:29.760 --> 12:35.760 So we are doing like some other writing of the of the version. 12:35.760 --> 12:41.760 Where where package depends on. 12:41.760 --> 12:45.760 So what we get here is we get incremented for fetching. 12:45.760 --> 12:47.760 We can fetch every dependency on it's own. 12:47.760 --> 12:50.760 We get rid of this vendor hash. 12:50.760 --> 12:58.760 Also, we get rid of the of the bad development workflow that comes with that. 12:58.760 --> 13:00.760 We get incremented builds. 13:00.760 --> 13:06.760 Tapping for our for a build process because the dependencies are already cursed. 13:06.760 --> 13:12.760 We get great composability because we're using set up hooks. 13:12.760 --> 13:20.760 We get the possibility to patch vulnerabilities because we only have dependency once in our set and we can then bump that. 13:20.760 --> 13:25.760 And it will it will propagate to to all the dependence. 13:25.760 --> 13:27.760 So that's really cool. 13:27.760 --> 13:31.760 This project targets next packages. 13:31.760 --> 13:33.760 The status is highly experimental. 13:33.760 --> 13:37.760 It's not usable yet. 13:37.760 --> 13:43.760 I did some research and I figured we will have to have like 12,000 go dependencies. 13:43.760 --> 13:51.760 So next packages we've got packages to build all the other packages that are currently. 13:51.760 --> 13:53.760 The next packages. 13:53.760 --> 13:59.760 And yeah, but now we are like about 30 packages that we are building. 13:59.760 --> 14:03.760 So there's let's do a lot of work. 14:03.760 --> 14:09.760 And we still are figuring out like how can this this work at all. 14:09.760 --> 14:11.760 Yeah, and it's a lot of work. 14:11.760 --> 14:15.760 I'm looking for people to collaborate on this. 14:15.760 --> 14:21.760 I hope for the the builder, the new builder, but also for the vulnerability research thingy. 14:21.760 --> 14:23.760 Please talk to me. 14:23.760 --> 14:29.760 You can reach me on metrics or here and person. 14:29.760 --> 14:33.760 And then I want to give shout out to all go related next projects. 14:33.760 --> 14:39.760 And I want to point out especially the one at the bottom which is quite new. 14:39.760 --> 14:43.760 And which uses the same go cash product mechanism 14:43.760 --> 14:49.760 to serve the go cash from outside the sandbox. 14:49.760 --> 14:51.760 So this introduces some impurity. 14:51.760 --> 14:55.760 But it relies on go to ensure the correctness of the build. 14:55.760 --> 14:59.760 So yeah, that's something that might be usable today. 14:59.760 --> 15:01.760 Yeah, thanks a lot. 15:09.760 --> 15:11.760 Thank you very much for the great talk. 15:11.760 --> 15:13.760 I have a lot of time available for questions. 15:13.760 --> 15:15.760 Are there any questions? 15:15.760 --> 15:19.760 Can I see your hand back there? 15:19.760 --> 15:23.760 Are you going to use the go box? 15:23.760 --> 15:27.760 Do you want to use the go box? 15:27.760 --> 15:31.760 So the question is, if this still uses the go.mod file. 15:31.760 --> 15:35.760 And the answer is no, it does not. 15:35.760 --> 15:41.760 You have to add the build dependencies as built inputs in the next. 15:41.760 --> 15:43.760 In your next derivation. 15:43.760 --> 15:47.760 And we will also rewrite the go.mod file. 15:47.760 --> 15:53.760 So because if you as I said, there's only like one version available. 15:53.760 --> 15:57.760 And this means that when there's like a diff in the version. 15:57.760 --> 16:01.760 You have in your go.mod file and the next packages. 16:01.760 --> 16:05.760 There could be like 10 to 50 dependencies that have changed. 16:05.760 --> 16:11.760 So we rewriting the go.mod file completely at this point. 16:11.760 --> 16:13.760 There is another question here. 16:13.760 --> 16:15.760 I think a form of the question. 16:15.760 --> 16:19.760 Is there any kind of equivalent, next equivalent, 16:19.760 --> 16:21.760 and walk file that you are generating? 16:21.760 --> 16:23.760 And next. 16:23.760 --> 16:29.760 Because you have to manually add all the dependencies to your build file. 16:29.760 --> 16:33.760 Or do you create a walk file? 16:33.760 --> 16:37.760 No. So the idea is that you do not decrease. 16:37.760 --> 16:41.760 If you create some kind of lock file for this. 16:41.760 --> 16:43.760 And the answer is no. 16:43.760 --> 16:49.760 That first you have your dependencies available in the next packages. 16:49.760 --> 16:51.760 So you just have to add them as built input. 16:51.760 --> 16:55.760 And then they are pinned as we fetch the source. 16:55.760 --> 16:57.760 And at this point they are pinned. 16:57.760 --> 17:01.760 So that's quite a lot of work initially. 17:01.760 --> 17:05.760 Yes, this is a lot of work. 17:05.760 --> 17:07.760 No. 17:07.760 --> 17:09.760 I don't have better random. 17:09.760 --> 17:11.760 But you can maybe automate this run. 17:11.760 --> 17:13.760 You can kind of think. 17:13.760 --> 17:15.760 Have a tool that takes the walk file. 17:15.760 --> 17:17.760 That gives you the next array of. 17:17.760 --> 17:21.760 That doesn't see that you have been created. 17:21.760 --> 17:23.760 Yes, so the ideas that this could be. 17:23.760 --> 17:25.760 There's like a lot of room for automation. 17:25.760 --> 17:27.760 And sure, there is. 17:27.760 --> 17:31.760 I don't think we will have to. 17:31.760 --> 17:35.760 Yeah, package all these things by hand. 17:35.760 --> 17:39.760 Especially as there are a lot of libraries that are really easy. 17:39.760 --> 17:41.760 And do not have like any, any built input at all. 17:41.760 --> 17:45.760 Or we can use the input stick to go out of Mod file. 17:45.760 --> 17:49.760 Yeah, but we're not there right now. 17:49.760 --> 17:51.760 Quick in between question is Josh. 17:51.760 --> 17:55.760 Can you hear who's going to be the speaker for next talk? 17:55.760 --> 17:57.760 Okay. 17:57.760 --> 17:59.760 Okay. 17:59.760 --> 18:05.760 Question is about the action of the talking. 18:05.760 --> 18:07.760 No. 18:07.760 --> 18:09.760 No more questions. 18:09.760 --> 18:11.760 Well, then thank you very much. 18:11.760 --> 18:13.760 Thank you for finishing on time.