I'm looking to make a custom filesystem for a project I'm working on. Currently I am looking at writing it in Python combined with fusepy, but it got me wondering how a compiled non-userspace filesystem is made in Linux. Are there specific libraries that you need to work with or functions you need to implement for the mount command to work properly. Overall I'm not sure how the entire process works.
In short: Linux is a monolithic kernel with some module-loading capabilities. That means that every kernel feature (filesystems, scheduler, drivers, memory management, etc.) is part of the one big program called Linux. Loadable modules are just a specialized way of run-time linking, which allows the user to pick those features as needed, but they're all still developed mostly as a single program.
So, to create a new filesystem, you just add new C source code files to the kernel code, defining the operations your filesystem has to perform. Then, create an initialization function that allocates a new instance of the VFS structure, fills it with the appropriate function pointers and registers with the VFS.
Note that FUSE is nothing more than a userlevel accessible API to do the same, so the FUSE hooks correspond (roughly) to the VFS operations.