Well, I’ve changed my mind. Some people pointed out that the good thing to do for most GPU is to align on 32-bit. That is, 4 bytes. The alignment should be 4 bytes, then, not 1.

There might be an issue with that. If you store a structure with attributes which sizes are not a multiple of 4 bytes, it’s likely you need to add padding.

However, I just reviewed my code, and found this:

instance (GPU a,KnownNat n,Storable a) => Vertex (V n a) where
instance (Vertex a,Vertex b) => Vertex (a :. b) where

Those are the single instances for Vertex. That means you can only use V and (:.) to build up vertices. Look at the V instance. You’ll find a GPU typeclass constraint. Let’s look at its definition and instances:

class GPU a where
  glType :: Proxy a -> GLenum

instance GPU Float where
  glType _ = GL_FLOAT

instance GPU Int32 where
  glType _ = GL_INT

instance GPU Word32 where
  glType _ = GL_UNSIGNED_INT

Woah. How did I forget that?! Let me translate those information to you. That means we can only have 32-bit vertex component! So the memory inside vertex buffers will always be aligned on 4 bytes! No need to worry about padding then!

The first implication is the fact you won’t be able to use Word16, for instance. You’ll need to stick to the three types that have a GPU instance.

Note: that doesn’t prevent us from adding Double later on, because a Double is a 64-bit type, which is a multiple of 4 bytes!

That’s all I have for today. I’m working on something very exciting linked to render batching. I’ll talk about that when it’s cooked. ;)

Keep the vibe; keep building awesome things, and as always, thank you for reading me!

Actually, it’s from 0.25.2, but I’ll just talk about the latest patched.

That new release received several patches:

• The gl dependencies was updated to gl-0.10.0.
• A new uniform_interface! macro was added.

I’ll introce that uniform_interface! macro in this blog entry.

Auto derive free almost free

That macro was written to provide a smoother and better experience when writing types that will act as uniform interfaces in shader. For the record, when you create a Program<_, _, _> with luminance, the third type variable is called the uniform interface. You’ll be handed back a value of that type when your shader is being used. That will give you the possibility to send data to the sader prior to making any rendering.

The main idea is to have a type like this:

struct Common {
  resolution: Uniform<[f32; 2]>,
  time: Uniform<f32>,
  jitter: Uniform<f32>
}

If you build a program which type is Program<_, _, Common>, whenever you ask a pipeline to use that program, you’ll be handed a Common object so that you can access back the uniforms.

In order for that to happen, though, there’s a constraint: your type must implement a trait. That trait is UniformInterface. There’s no magic behind that – though, it can be a bit harsh to implement that trait. If you’re using luminance or plan to, I strongly advise you to try an impl that trait by hand first.

Anyway, after the fifth or sixth uniform interface you’ll have written, you’ll start to get bored of writing the same code every now and then. That’s the exact same feeling as for serializing and deserializing with serde: it’s interesting to do it by hand the first time, but it gets really, really annoying after a while.

So it’d be great to be able to automatically derive UniformInterface for your own types. There’re two solutions here:

1. Use procedural macros so that we can write something like #[derive(UniformInterface)].
2. Use standard macro_rules!.

(1) is great because it seems seamless when you use it – you already use #[derive(Clone, Debug, Eq, PartialEq, Etc)]. However, even though it’s doable to learn syn in order to parse the Rust token stream, it’ll require some time and patience while I wanted something quick to mockup the idea. For that, (2) is perfect, because regular macros are easy and if correctly written, shouldn’t add too much weirdness over the type.

The idea is to move to (1) if I find (2) to be a success and really useful.

The macro

So… we’re talking about the uniform_interface! macro. This macro has already a pretty decent documentation, so feel free to visit it, but I’ll explain more here.

A macro is generally used to introduce an EDSL. For instance, nom’s do_parse macro uses a funny EDSL (you use operators like >> and return result with (…), which is not normal Rust code).

The uniform_interface! macro uses an EDSL that you know very well: it’s plain Rust code! Hot news: you’ll have nothing more to learn!

Note: it’s not really any Rust code, since you can only define structs. Then, see that as a subset of Rust.

The syntax is very simple. Let’s take back our sample from above with the Common interface but now let’s the macro do all the magic for us!

#[macro_use]
extern crate luminance;

uniform_interface! {
  struct Common {
    resolution: [f32; 2],
    time: f32,
    jitter: f32
  }
}

You’ll notice two interesting properties:

• You define your fields without annotating them with Uniform<_>: the macro does it for you.
• The UniformInterface impl is automatically generated for you!

The second point is the most interesting one, since it’s akin to writing #[derive(UniformInterface)]. As you can see, the syntax overhead is not really a problem.

The cool part of that macro is that it also supports Rust annotations on its fields. You have two possible annotations available:

• as("something") behaves by not using the field’s direct name and instead use the one provided as argument.
• unbound enables not to make the whole uniform interface fail if a field cannot be mapped on the GPU side.

The as(…) annotation is very simple and straight-forward to get, since it lets you change the binding name of the field. The unbound annotation is a bit trickier and you need to understand how UniformInterface expects the value to be constructed.

By default, when you try to map a field to something on the GPU side (shader program), you use the UniformBuilder::ask function. If you have a closer look at function, you can see that it returns Result<Uniform<T>, UniformWarning>. In order to get the Uniform<T> out, you’re left with the choice of how you should handle errors. And that will depend on how you want your value to be built. For instance, some fields might be completely mandatory for your shader to work and others optional. unbound tags a field as optional.

You can mix both annotations if you want to remap an optional field!

#[macro_use]
extern crate luminance;

uniform_interface! {
  // we want to export that, so we also make it pub
  pub struct Common {
    resolution: [f32; 2], // this is really required
    #[as("t")]
    time: f32, // will be referenced with "t" in the shader and is mandatory as well
    #[unbound, as("bias")]
    jitter: f32 // will be referenced with "bias" in the shader and is optional
  }
}

That’s all for today! I hope that feature will be useful for whomever uses luminance – I use it extensively and I’ve been using uniform_interface! for several days now and it’s really appreciated! :D

I plan to add another macro to create buffer types, since they must meet GPU-specific alignment rules, who are boring to maintain without code generation.

Feel free to provide feedback on Reddit or GitHub and have fun!

• My thoughts about editors in 2020.
• My thoughts about editors in 2021.
• Neovim plugins stability.
• Development environments.

My take on all of that has evolved a bit over the last months / weeks. A couple of things happened and I think it’s time for an update.

My views on productivity platforms

If you have read my articles about development environment, you already know what I’m talking about here. If not, let me do a little summary. Basically, as a software engineer, I need to use tools to solve problems. Those problems are daily little issues while working on a project:

• Organizing my work.
• Searching for stuff, like a file name, some documents, discovering a code base, etc..
• Editing code in an efficient way.
• Building and running programs.
• Testing programs.
• Debugging programs.
• Versioning my code.
• Recording notes, journaling, creating meeting summaries, etc..
• Manipulating program output.
• Composing such output.
• Viewing read-only data and grepping into it.
• Testing an API by issuing HTTP / gRPC / whatever calls to target endpoints.
• Using various systems such as bazel, helm, kubernetes, etc. and composing them with the rest.
• Etc. etc.

Whatever the tools you are using, those problems will roughly be the same. Maybe you don’t have them all, but you will come across a couple of them on a daily basis. Let’s start talking about what most people use: IDEs.

IDEs

An IDE (Integrated Development Environment) is a software that provides a solution to a subset of the problems I mentioned above. For instance, IntelliJ IDEA has a solution to edit your code, go to definitions, implementations, lookup files, classes, symbols; version your code and write Git commit messages; it has a debugger; it has a way to build and test your code; it has a terminal so you can run random CLI commands; etc. It doesn’t solve all the problems, but clearly it solves a subset of them. With programs like that, most of the time, you install it, and you’re good to go without configuring anything. JetBrains editors are famous (and loved!) for this reason. You want to work with Java? Install IDEA. Of course, you will still find plugins to customize the experience, but the vanilla editor is most of the time enough.

Then you have “customizable” IDEs, such as VS Code. Such softwares will often require you to download plugins because the default experience is unlikely to have support for your language, even though it should be enough for most people. VS Code is clearly the most famous one and it has a plugin for everything you might need (or not know you need!). I could have merged the two IDE sections into one, but I do make a difference in my mind because of how JetBrains are advanced and ready-to-be-used. VS Code, whether you like it or not, is an important piece of software. Microsoft has made a big change with it, since most developers would agree it’s a good editor and environment to develop in, and it helped introducing things like LSP and DAP. You cannot trash-talk VS Code in a non-joking way, they contributed too much and we all use their work.

The rest

And then, you get into the “do one thing and do it right” way of working, but it’s more complicated than that. Historically, you would find tools such as Vim, Neovim, the git CLI, fzf, terminals, shells, TUI applications, etc. However, as time passes, there is a trend: people tend to transform those “unit tools” into IDEs. The terminology doesn’t really matter (whether you want to call that IDE or your own neologism). What matters is that such tools are not minimal nor “do one thing and do it right” anymore. Neovim, for instance, is now more a Lua interpreter in disguise of an editor, allowing to build via the plugin ecosystem, than a minimal editor. I was told that I was wrong thinking that Lua wasn’t part of the equation since the beginning, so yeah, I was a bit on the wrong track from the start.

It’s the same trend as it has been in the Vim ecosystem for so many years. Just look at plugins like vim-fugitive or nerdtree. In Neovim, you have plugins like mason.nvim (which is basically yet another package manager), lazy.nvim (same thing but different), nvim-tree.lua, a file explorer, neogit, a Git implementation in Lua, and the list goes on.

So, yes, I also contribute to that “plugin-based” ecosystem with hop.nvim and mind.nvim, but I’ve been thinking about all that quite a lot. That adds up to what I discuss in my article about configuration vs. scripting (basically, I think configuration should remain data, not code — which what scripting is about). Quoting something I said on the Neovim Matrix room, “where people see power, I see weakness”. A scripting language brings lots of powerful things, such as extensibility, but it also brings bugs, hence unstability, and a Turing-complete language, preventing the host (i.e. Neovim or even plugins) to easily use the configuration to discover options and data, without having to standardize an API. The most infuriating point here to me is colorscheme. They are just scripts. Some of them are even stateful (they cache things in ~/.cache/nvim). So “applying a colorscheme” is not simply just switching highlights mapping: it requires to execute code, which makes it super hard to reason about the colorscheme. What’s a colorscheme when you know it can run an HTTP command?

I’ve been seeing a lot of new plugins, since I am the author and maintainer of This Week in Neovim. And recently, I saw a video from TJ DeVries: Effective Neovim: Instant IDE. And that video confirms what I said above Neovim becoming an IDE. But it also made me realize something else; TJ uses a plugin as support for explaining how to easily turn your favorite editor into an IDE: kickstart.nvim.

That plugin is basically a single Lua init.lua script which serves as a starting init script for your configuration. You just install it and it downloads a bunch of stuff for you, and calls all the required Lua code to set up correctly the LSP implementation, Tree-sitter, downloading managers (yes, plural, because you need one for plugins and one for LSPs, debuggers, linters, etc.), installing various plugins for completion, Git integration, surrounding pairs, etc. And then, I wondered: “Newcomers are expected to run a script that can download pretty much anything from the Internet… or write pretty much most of what it does — which is a lot — on their own?!” And the thing is that, my own Lua configuration, which is not based on kickstart.nvim since I created years ago, has been completely obsolete and I often need to go back to it to remove or update things, especially regarding LSP, completion, snippets and all.

Why I think it’s bad

Do one thing and do it right?

Most people from the community I talked to disagree with my point of view regarding Neovim. For them, plugins like mason.nvim are amazing, because they close the gap between their editor and the tools required by their editor to work correctly (Mason downloads LSPs / linters / DAP adapters / etc.). I used Mason too, but eventually stopped using it when it started downloading a version of rust-analyzer that was released years ago (that was a bug in Mason, I guess?). I came to the conclusion that I was depending on something doing HTTP calls to download tools that, in theory, could be used by other tools on my machines, and that I could also download myself very easily. In the case of rust-analyzer:

rustup component add rust-analyzer

Worse… some of those tools are actually packaged in my package manager (pacman), so I’m basically using a tool (Mason) that is doing the same thing as a package manager. As if we did not have enough package managers already.

Too high and wrong expectations

I then continued thinking about all those plugins (among some I use or even have created, like Mind!). Why should I use them in Neovim? I’ve never been a file explorer guy that much, but I know about nvim-tree.lua and… and why do we have to have that in our editor? I remember the state of mind I was in when I wrote my article about Doom Emacs, which completely changed the way I think about software development. Emacs doesn’t belong to the “minimal editors”, nor the IDEs directly… it’s a different beast on its own, but if I had to compare it to something else, I wouldn’t say VS Code, or Neovim, or anything else. I would say “My terminal with all of the commands I run, including an editor, a git implementation, etc.” Wait a minute. Why are we trying to push all those features as plugins into Neovim, again?

The reason I didn’t stick around Emacs was basically because of its runtime choice, and ultimately, its Lisp ecosystem. The Emacs community is one of the best I have been talking to, and they have really, really talented people working on insanely complex topics, like turning Elisp code into native code (via C), including Emacs itself. But even with all those optimizations, the editor was still feeling too slow, and has a lot of background that you can feel. All the abstraction layers, all the Lisp macros (oh no), etc.

Eventually, I went back to thinking about that sentence… that haunting sentence… “Do one thing, and do it right.” That sentence has a lot of meaning and I think people have been tearing and bending it to align with their conviction, completely ignoring their biases. I read people from the Emacs community stating that yes, Emacs still applies to that sentence, because “It’s just a Lisp interpreter.” But in the end, the experience users have highly depends on the plugins, which is the same situation as with Neovim. And they have to configure their editor using a Turing-complete language that might introduce bugs, complex statements (who loves to set up a colorscheme by conditionally importing / requiring a Lua file located you-have-no-clue-where on your file system?)

Why are we trying to push all those features as plugins into Neovim, again? Why am I not trying to focus on using tools with a narrower scope, but ensure that the tool remains stable, powerful to use and compose well with the rest?

Exploring new paths

Lately, I have discovered Helix, a “post-modern modal editor”. The first thing I noticed was that it’s different than Neovim in terms of motions. In Neovim, in normal mode, you start with a verb, like deleting is d, and then you type a motion, like a word is w. So typing dw on your keyboard will delete a word. In Helix, it’s reversed. You first select with w, and then you apply the verb you want. So wd. At first, I thought is was a neglible difference. Then I realized how more powerful the [Helix]’ way ways. Since you “see” the selections, you can select first and then decide what to do, or even change your mind and extend a bit the selection. You have this nice visual feedback.

And then comes all the good stuff. Helix comes with those included features, requiring zero configuration:

• An LSP native implementation (the editor is written in Rust, so is the LSP implementation then), with all the features you expect, like preview popups, documentation, signatures, go-to-def, references, implementations, rename, code actions, etc.
• Tree-sitter support, including highlights, indents, text-objects, etc.
• DAP support, both the feature and its UI.
• Multi-cursors.
• Surrounding pairs.
• Git gutters.
• Fuzzy finders for buffers, symbols, jump lists, diagnostics, project-wise / global grepping, etc.
• Etc.

And it has no plugin support for now (they plan on adding it at some point, but not for now). And that made me realize: my editing experience in that editor — even though it took a couple of days to adjust the muscle memory — has been flawless. So yes, I miss my hop.nvim plugin, but I realized I could hack around by using Kitty hints until that kind of motions is built-in.

However, I’m just talking about editing, here. I still have that reflex of pressing SPC g g to bring up a Git prompt in my editor to start committing… but Helix doesn’t have one. So I’m splitting my terminal into another window and I use the git CLI. And it’s fine. Now, the way I think about it is that I could probably invest time into learning lazygit or anything else.

The point is that my editor is now minimal again. My configuration (which is public, you can find it here) is mostly about bépo remappings and some very minimal aesthetics changes. The configuration is data (it’s just a TOML file), and I don’t have to worry about stability anymore since I’m not using any plugin. The fact that I have an amazing editing experience (even better, honestly, due to the selection then verb principle, multi-cursors, out-of-the-box LSP and Treesitter experience, including completion) is just the perfect fit for what I want.

Yes, but

But there’s a catch. See, Helix is about editing. If you like to have a file explorer in your editor, the way I would recommend looking at Helix is that you cannot have it in Helix and you should probably use a proper, standalone file explorer, or consider another alternative like Neovim. If there is something you would like to add to Helix, you have to open a PR and write some Rust code. You cannot extend it on your own, as it doesn’t have plugin.

To me, that’s great, because it means the scope is under the responsibility of the Helix Team. And I love that. I love it because it’s easy to think about the features of my editor. It’s easy because I don’t have to keep fearing something break because of an incompatibility between a plugin and the version of the editor I’m running (or even two plugins between each other).

And honestly, contributing using a statically and strongly typed language (Rust) feels so much sounder to me than using something like Lua. You can benefit from all the tests of the code base, use the API without any ABI conversion in between, and catch bugs at compile time instead of waiting for them to crawl up at runtime.

So what does it mean for “productivity platforms?”

My view on that hasn’t changed since my last articles. I still think that the terminal needs to be revamped and go into a direction similar to Emacs. I have started a project a couple of months ago that tries to explore that. Basically, I’m making something that is not a terminal, a shell nor editor. It’s “a platform”, with primitives like tree views, item lists, read-only / read-write buffers, virtual text, popups, command outputs, cells, etc. And it comes with no way to edit text or file explorers, or anything.

Then, applications targetting that platform can use all those primitives to compose and now the features are emergent. A text editor then uses the buffer, virtual text and popup primitives, for instance. A file explorer would use the tree view primitive. An LSP client could be a daemon that attaches to edit buffers. Etc. etc.

That’s the dream tool I would love to see, and I still think that Emacs is the closest thing to that, but it comes with too much legacy to me. And it’s unlikely that my experiment will ever be mature enough to be usable or even used. But you know, I like experimenting. A cool project to play with is nushell. It’s far from being that dream platform of mine, but it has some very interesting ideas for composing commands that I think are worth mentioning.

And no, I don’t want Helix to become such a platform. Nor Emacs to get rid of its legacy and become it. Nor Neovim. I want to keep playing with Helix and using it for what it is (and shines for!): editing code. If my dream platform ever exists, whether it’s mine or someone else’s, I will probably move away from Helix to whatever that platform provides. But such a change would require a standardization, such as stdout and stdin, but with all those primitives I mentioned. And I’m not sure whether such a thing will or can exist.

And Helix? Is it good?

I’m not going to give my complete opinion on Helix just yet. I have been using it for a couple of days only, and at the time writing, it still has a lot of missing parts / experimental ones. For instance, its DAP support is experimental, so I can’t judge. What I plan to do is to stick to it and move away from “making my own IDE in an editor” and instead enjoying composing tools on the CLI. Then, when I have enough hindsight, I will give a fair review of Helix.

What about mind.nvim, hop.nvim and This Week in Neovim?

So, about mind.nvim, I plan on rewriting the plugin as a standalone tool so that I can use it whatever the editor. It will probably do things like git when you run git commit (opening $EDITOR), but I’m still not sure exactly how I’m going to make it. Maybe I’ll get in touch with people from Charm and rewrite it using some of their work? I still haven’t thought about it, it’s too early.

About hop.nvim, I plan on continuing maintaining it and fixing bugs, even though I haven’t been very active around Hop lately. The reason is mainly a lack of spare-time.

As for This Week in Neovim… I honestly do not know. I discussed the project with some people from the Neovim core team, and I’m a bit stuck. On one side, the community has received it pretty well, given the amount of upvotes I have on each week release on Reddit; the comments; the appreciation issues on GitHub, etc. I know people have been enjoying my work, and I’m happy they do.

On the other side, the core team doesn’t seem to have noticed it that much, and none of their members approached me to talk about it. So I’m not sure what to think. The community enjoys TWiN a lot; the core team doesn’t really care. Then I need to think about exhaustion: I’m really tired of maintaining TWiN.

See, the idea is to communicate, every week, about what has happened in the Neovim world, whether it’s core or plugins. What I had initially in mind was to bootstrap the couple of first releases and let people adopt and contribute to it. On the 2nd of January 2023 is released TWiN #25, which means that I’m currently on a 25 weeks streak. What it basically tells is that, every week (most of the time Sundays), I skim Reddit, GitHub projects, man pages, etc. to get as much information as I can, and create a really big PR containing the weekly update. That PR is merged and available on the very next day (Monday) for every neovimmers to enjoy reading on Monday morning with a nice cup of coffee, tea or whatever you like for breakfast.

So every week, one person (me) spends hours skimming many projects, while what I thought would happen was that many plugin authors would contribute once every two months a very small text to explain their new plugins / change. The difference is massive: on one side, you have a single developer doing a big amount of work… every week. On the other side, you would have many developers doing a very small amount of work every time they release something… which is clearly not every week (and even then?).

I think I have enough distance with the project to admit I failed marketing my idea. Someone once told me that I was basically doing free advertisement for plugin authors, which is actually true. People mention they would like to donate to contribute and ensure that I keep doing what I do, but I don’t want money — hosting costs me 10€/month and the domain name is 10€/year, I can sustain that on my own. What I need is contributions. It wouldn’t cost much for a plugin author to open a PR to twin-contents and add their update to the upcoming week. There’s a few regular contributors, writing good PRs I rarely need to modify. But most of the weeks are contribution-free, and it saddens me even more when I see the reaction of plugin authors on Reddit, like “Oh yeah my plugin made it to TWiN!” Every time I read that, I think “Great, next time maybe they will be pushing the update themselves to help me?” And most of the time, they don’t.

So, people started to mention that I should slow down or I will burn out. And I’m honestly pretty fed up with this read-only relationship: people consume / read; they rarely contribute, even when they could contribute their own update for their own plugins! I don’t filter out anything, as long as it’s about Neovim and doesn’t convey any bad speech (you know the deal). TWiN is about new plugins, updates of existing plugins, tips of the week, blog articles, youtube videos, etc. Anything Neovim related produced by a member of the community. And even with the exposure of TWiN, people still do not contribute. Even after the big refactoring to ease contribution I announced on Reddit. So yes, it’s a personal failure to market my idea regarding TWiN, and I’m not sure what the next steps are.

Nevertheless, we all learn from mistakes and it’s important to understand them. I will collect my thougths and decide what to do next. For the time being, I wish you all a Happy New Year, lots of great things, and a tremedous amount of happy hacking, in your favorite editor, IDE or whether you pipe echo commands directly at the end of files!

Keep the vibes!

Discuss here

In this blog entry, I want to talk about UX more specifically, because I do think it’s more important than UI. I will try to show that the default UX of Kakoune is incredible and that you can very quickly create a super expressive and pragmatic programmer environment. It will cover:

• Surrounding pairs.
• Grepping around
• Pickers.
• System clipboard.
• Bonus: some tools I’ve been making.

Once again, I’d like to greet and thank @Taupiqueur for sharing his thoughts and joy about Kakoune.

What I mean with UI and UX

UI means “anything interfacing the user to the system.” It’s both the visual depiction of the service (the menu, the colors, the fonts, etc.) and the way you interact with the system (with the keyboard, by clicking on buttons, when a system event happens, etc.).

UX means “how the user experiences the system.” For instance, something that is not UI at all but enhances the UX is having a way to filter data in a system with high volume of events with a tag. An even better UX is having fuzzy filtering with any tags. Etc.

There are many possible UI implementations for a given UX item: implementing filtering can be done with a select box in a GUI, but the UX is not great because the user is presented with a set of finite choices, and if there are many, it’s pretty annoying to scroll down the list to find the one we want. Even with this bad UX design, most of select box implementations (e.g. web) allow to press keys to jump to the entry in the select options — most of the time, you don’t see what you type -> bad UX for the user. Another possible UI implementation would be to use a free text box that would filter based on its content — it could even be live for an even better UX. Etc. etc.

Now, would you prefer a nice looking GUI with the select box, or a blander UI but with the fuzzy search box? Clearly, in terms of UX, the second option is much better. But now, imagine a GUI with the fuzzy search box. It would be pretty similar to the blander UI in terms of UX, but it would look (much) better… which is likely to enhance the UX as well!

Small disgression: TUI vs. GUI

TUI: Text User Interface, which is a program mimicking traditional GUIs in the terminal.

GUI: Graphics User Interface, the typical window-based applications you run on your machine.

Before jumping to the Kakoune content, I just want to disgress slightly to talk about something I often see something that itches me a bit: many individuals seem to say that a TUI is often written in a way that optimizes UX and GUI the UI, and hence, oftentimes, using a TUI feels much better. I agree with this (this is the reason why I’m using editors and tools inside my terminal instead of a dedicated GUI, even though I think the UI is worse, for many reasons: no pixel-perfect alignment of things; no direct integration of the application at the OS level, it has to go through the terminal and shell; etc. etc.).

However, is there anything forcing a GUI not to provide the same kind of interactions as a TUI? I don’t think so. If you make your TUI keyboard-oriented, everything you do is just listening to keyboard events provided by whatever terminal protocol / library you use… which could be done exactly the same way in a GUI.

I do think that we should be able to make a GUI as good as a TUI in terms of UX, and some programs did it. For instance, emacs can run both as a TUI and a GUI (and today, people recommend actually using the GUI). I think that could be the topic of another blog article. Let’s go back to the original matter.

Kakoune UX

Kakoune, upon installation, already provides a lot of good things in terms of UX. But as you get more productive with it, you will face some problems. For instance, the first one I came across (pretty quickly being honest, coming from Helix) was surrounding: adding, deleting and replacing surrounding delimiters. Where you need a plugin to do that in the Vim world, in Kakoune, it’s another topic. Some plugins exist to do exactly that, but honestly, read along.

Selections in Kakoune — which are so much more than “iT’s JUsT LikE muLtIcURsOrs oR a nOOb WaY Of DoINg RegEX in vIM” — change the Vim interpretation of appending and preppending. In Vim, i will start inserting on the cursor, whereas a will start inserting after the cursor. In Kakoune, i inserts at the beginning of each selection and a inserts at the end of each selection. That’s already a better UX, and it allows to do many things out of the box.

For instance, since we have the power to insert at the beginning and at the end of each selection… then pressing i' should insert a quote at the beginning of the selection… and a' will do the same at the end! So you can already have a somewhat working surrounding add operation by typing i'<esc>a'!

If you are adventurous, you can read the documentation of <a-;>, which allows to leave insert mode to normal mode for a single command, and come back to insert mode. You can use this to replicate what we did above without the <esc> key: i'<a-;>a'. Magic.

It’s a bit annoying to have to type all that, though, right? So instead, we could make a command! Commands in Kakoune are really simple, but they require reading a bit about them. I recommend the following reads if you want to dig in a bit:

• :doc commands to know about all the commands. Search for ^define-command.
• :doc execeval, which explains what evaluate-commands and execute-keys do. Especially, you will want to read the part of -draft.

So let’s wrap that sequence of keys in a command. Instead of using i and a, we are going to use P and p, which, as the name implies, paste from the default register. p pastes at the end of the each selection and P pastes at the beginning of each selection.

Surrounding pairs

What is great is that the default register, ", is selection-aware: its content will be different from one selection to another. Said otherwise, there is one " register for each selection. Hence, we can write this:

define-command -override my-surround-add -params 2 %{
  evaluate-commands -draft -save-regs '"' %{
    set-register '"' %arg{1}
    execute-keys -draft P
    set-register '"' %arg{2}
    execute-keys -draft p
  }
}

And here you go. You can now invoke :my-surround-add ( )<ret> to add parenthesis around your selections, for instance.

Kakoune has the concept of user modes, which is a nice feature allowing to declare a keyset (that will be displayed by the help in the bottom right of your screen) when entered.

declare-user-mode my-surround-add

We can make one with a bunch of mappings in that user mode:

map global my-surround-add (   ':my-surround-add ( )<ret>'         -docstring 'surround with parenthesis'
map global my-surround-add )   ':my-surround-add ( )<ret>'         -docstring 'surround with parenthesis'
map global my-surround-add [   ':my-surround-add [ ]<ret>'         -docstring 'surround with brackets'
map global my-surround-add ]   ':my-surround-add [ ]<ret>'         -docstring 'surround with brackets'
map global my-surround-add {   ':my-surround-add { }<ret>'         -docstring 'surround with curly brackets'
map global my-surround-add }   ':my-surround-add { }<ret>'         -docstring 'surround with curly brackets'
map global my-surround-add <   ':my-surround-add < ><ret>'         -docstring 'surround with angle brackets'
map global my-surround-add >   ':my-surround-add < ><ret>'         -docstring 'surround with angle brackets'
map global my-surround-add "'" ":my-surround-add ""'"" ""'""<ret>" -docstring 'surround with quotes'
map global my-surround-add '"' ":my-surround-add '""' '""'<ret>"   -docstring 'surround with double quotes'
map global my-surround-add *   ':my-surround-add * *<ret>'         -docstring 'surround with asteriks'
map global my-surround-add _   ':my-surround-add _ _<ret>'         -docstring 'surround with undescores'

Obviously, that’s not all; we would need to delete delimiters and to replace them. Deleting is actually even more straightforward. Kakoune has some native mappings to select everything inside or around a set of delimiters:

• <a-i> to select inside.
• <a-a> to select outside.

So, pressing <a-a>( (or <a-a>), same thing) will select everything around the cursor up to the next pair of parenthesis.

Note: I have personally remapped that to mi and ma, but that collides with the default meaning of the m key in Kakoune, so I will use the native mapping here instead.

We can then simply use the previous i and a command mentioned before to start editing at the beginning and end of the selection. i<del> will start insert mode at the beginning of the selection and will remove a character (left delimiter) and a<backspace> will insert at the end of the selection and remove the previous character (right delimiter). Eh, that looks like so simple it’s almost stupid. But that’s what makes Kakoune so damn good: it’s simple to reason about:

define-command -hidden my-surround-delete-key -params 1 %{
  execute-keys -draft "<a-a>%arg{1}i<del><esc>a<backspace><esc>"
}

define-command my-surround-delete %{
  on-key %{
    my-surround-delete-key %val{key}
  }
}

Because Kakoune composes really well, you can already imagine that you should be able to use the previous commands and mappings. And indeed:

define-command my-surround-replace %{
  on-key %{
    surround-replace-sub %val{key}
  }
}

define-command -hidden my-surround-replace-sub -params 1 %{
  on-key %{
    evaluate-commands -no-hooks -draft %{
      execute-keys "<a-a>%arg{1}"

      # select the surrounding pair and add the new one around it
      enter-user-mode my-surround-add
      execute-keys %val{key}
    }

    # delete the old one
    my-surround-delete-key %arg{1}
  }
}

Revisiting grepping

Ah… who has never had the problem of trying to locate something in a codebase without really knowing where to start. That happens a lot to me when working on a front-end project and taking an issue asking to fix a random page, that is broken. Usually, LSPs don’t help to discover things that are based on the final product. For instance, if you see the checkout page broken — like a <div> is missing an attribute or a tag is misplaced in the DOM, no LSP will help you locate the code that needs to be fixed. Instead, you need other tools.

What I like to do is looking at the page and looking for what I call unique tokens. For instance, a short sentence that might appear only on that page, or modal. Or a header, a title, etc. Then, using that information, I usually grep the code base. The problem is that, doing that using grep or ripgrep as CLI is pretty boring, and not very practical. Indeed, if you get many results, you are likely to try to reduce the result set by constraing more your regex. Once you have some files, you usually look in your terminal scrollback buffer until you find something interesting, then open that file in your editor.

People using something like IntelliJ products, or VS Code, or some plugins with emacs or vim might have a way to perform the search from within their editors, but again, that’s not composability: it’s extensibility, and I explained in a previous article why it’s not a good design to me.

Kakoune, on the other side, went the composition route. grep, ripgrep, etc. are all amazing tools. Kakoune comes with a bunch of what it calls tools, which are basically Kakoune commands shipped with the editor. Among those, there is one that is of interest here: the :grep command. The :grep command forwards its arguments to the underlying %opt{grepcmd} (which defaults to something like grep -RHn). Hence, :grep foo will run grep -RHn foo in a shell behind the scene, then the result will be output in a *grep* buffer. That buffer will get special highlightings, along with some keybindings, all of that provided by the grep.kak tool. If tried the command, you might have noticed that it’s basically a list of lines of the form:

<path>:<line>:<column>:<content-of-the-line>

Then, how do you think grep.kak implements _pressing <ret> jumps to the line, column and file of the line under the cursor? The cursor can be anywhere on the line. Well, it’s simple: execute-keys again! gh will put your cursor at the beginning of the line. Then, you can use f: or t: to jump to the next : — there is no need to parse anything, we can just programmatically interact with the editor!

Here, ghT:"py will go the beginning of the line, select the path and yank it to the p register. We can then do 2l to move the cursor to beginning of the <line> number, and T:"ly will yank the line number to the l register. 2l again to move to the <column> section, ten T:"cy to yank the column number to the c register. And we have everything we need. We can then just simply run the edit -existing %reg{p} %reg{l} %reg{c} command:

aoc-18.md:1:1:lol

define-command -override my-jump-for-current-line %{
  evaluate-commands -save-regs clp %{
    execute-keys -draft 'ghT:"py2lT:"ly2lT:"cy'
    edit -existing %reg{p} %reg{l} %reg{c}
  }
}

A couple of comments here:

• -save-regs clp saves the content of the c, l and p register before evaluating the commands, then restores those registers afterwards. That is required since we copy a bunch of data to those registers, but maybe the user is already using them. -save-regs allows us to specificy a set of registers we locally need, but we do not want those registers to bleed outside.
• -draft makes the command evaluation run in disposable context. That prevents our selections from being changed — execute-keys runs a couple of move and goto commands here, while we don’t really want the cursor to move.
• -existing fails if the file doesn’t exist.

The :grep tool implements something probably very similar to this, but it uses %opt{grepcmd}, which you can change to whatever you like. I personally use:

set-option global grepcmd 'rg --column --smart-case --sort path'

Kakoune lacks pickers… or does it?

Something that is very important having around is being able to locate files very quickly. Most editors today ship with a way to locate files:

• Vim has netrw. It’s old. It’s ugly. But it’s there. It’s basically a file browser with a (very) minimal set of features.
• VS Code has fuzzy pickers, so you can locate a file by fuzzy searching it.
• Etc.

Kakoune has a default powerful completion engine. Pressin :edit (or, for short, :e ) and then typing something will auto-complete the file in the current directory with a somewhat fuzzy algorithm. If you select a directory and type the trailing /, it will list the content of that directory and will auto complete its children. It’s a pretty nice way to start moving around with the vanilla editor.

However, Kakoune doesn’t ship more than this, because, well, it’s composable. You can use any tool you like to locate files, and then compose them with Kakoune. I personally really like fd (a Rust rewrite of find). For instance, fd --type file will locate all the files in the current directory. Piping that to a fuzzy finder would allow to jump to any file in the current directory. And Kakoune supports that. It has several commands for that, but the one you will be interested at first is prompt. It prompts the user for some text and pass the entered text to the provided commands in the %val{text} variable. It supports some switches, and one of interest for us is -shell-script-candidates. That switch accepts a shell command to execute, reading from its standard output asynchronously and displaying into the completion items. For instance, try running the following command:

prompt -shell-script-candidates ls file: %{ info "you selected %val{text}" }

It runs ls, gets the output and allows you to fuzzy complete the result. Now consider this:

prompt -shell-script-candidates 'fd --type file' open: %{ edit -existing %val{text} }

And bam. Here you have it. Fuzzy picker in Kakoune. You can map that to a command, or wrap it in a command:

define-command my-file-picker %{
  prompt -shell-script-candidates 'fd --type file' open: %{ edit -existing %val{text} }
}

If — like many people — you want that to be run when you type SPC f:

map global user f ':my-file-picker<ret>'

You can do the same thing with anything, really. Some plugins like kak-lsp uses that to show document symbols, etc.

System clipboard

By default, Kakoune comes up with registers you can yank to. However, oftentimes, you need to yank and paste from the system cliboard. Kakoune doesn’t have a “system register” as in Vim. Instead, as with the rest, you have to compose some tools with Kakoune to get that feature. And it’s pretty simple. You need to know how to yank some text in CLI. Two situations:

• On macOS, we use the pbcopy and pbpaste to respectively copy and paste from/to the system clipboard.
• On Linux, we use the xclip and xsel -ob commands to do the same.

And then, there is nothing much more to do. We can wrap those in two commands agnostic of the platform by using the uname utility:

declare-option str extra_yank_system_clipboard_cmd %sh{
  test "$(uname)" = "Darwin" && echo 'pbcopy' || echo 'xclip'
}

declare-option str extra_paste_system_clipboard_cmd %sh{
  test "$(uname)" = "Darwin" && echo 'pbpaste' || echo 'xsel -ob'
}

And the actual commands, using the ! key (to run a shell command with the current selection piped as standard input and replace selections with its output) and <a-!> (which runs a shell command and ignore its output):

define-command extra-yank-system -docstring 'yank into the system clipboard' %{
  execute-keys -draft "<a-!>%opt{extra_yank_system_clipboard_cmd}<ret>"
}

define-command extra-paste-system -docstring 'paste from the system clipboard' %{
  execute-keys -draft "!%opt{extra_paste_system_clipboard_cmd}<ret>"
}

I personally map those as in Helix:

map global user y ':extra-yank-system<ret>'  -docstring 'yank to system clipboard'
map global user p ':extra-paste-system<ret>' -docstring 'paste selections from system clipboard'

Bonus: some tools I’ve been making

Because the UX in Kakoune is amazing, and because it composes so well, I have made a couple of binaries and Kakoune commands to enhance the experience. So far (23rd December of 2023), I have wrote those

• kak-tree-sitter, which provides tree-sitter support (so far, only semantic highlighting; semantic selections will come later).
• bookmarks.kak, a super small plugin I use to create persistent bookmarked locations (and list them in a dedicated buffer).
• git.kak, a super set of the (already great) Git integration in Kakoune. I for instance add a cursor-anchored overlay showing git blame information.
• hop.kak, Hop brought to Kakoune. My implementation is by far much simpler than what I did in [hop.nvim] because I read native Kakoune selection descriptions, instead of relying on an internal regex engine run on each line: it composes so much better.
• notes.kak, one of my most useful plugin. It enhances Markdown with some features:
  • Lists marked - TODO, - WIP, - DONE, - WONTDO, - IDEA, etc. are highlighted with specific faces.
  • Todo items from the list above can be gathered in a buffer (similar to a grep buffer) to jump directly to the note file containing them.
  • Open the journal of the day.
  • Open notes.
  • Search journals and notes.
  • Archive and list archives.
  • And much more.
• swiper.kak, my interpretation of the famous swiper emacs package; it allows to reduce a buffer by fuzzy searching something in it, only showing the lines that matches, but showing them all at once (by removing the lines that don’t match), and pressing <ret> jump to the line under the cursor (or <esc> goes back to where you were buffer). I also included a reduce mode that helps with reducing grep buffers or any buffer that can be directly modified.

All in all, I’m really happy with my current Kakoune setup, and as time passes, I realize it’s a nice interactive editing platform so far. For instance, at work, I have started making a k8s.kak to highlight and run commands on Kubernetes cluster from within Kakoune; and it works pretty well.

Have fun and keep hacking around!

The stash is a temporary area working like a stack. You can push changes onto it via git stash or git stash save; you can pop changes from top with git stash pop. You can also apply a very specific part of the stack with git stash apply <stash id>. Finally you can get the list of all the stashes with git stash list.

We often use the git stash command to stash changes in order to make the working directory clear again so that we can apply a patch, pull some changes, change branch, and so on. For those purposes, the stash is pretty great.

However, I often forget about my stashes – I know I’m not the only one. Sometimes, I stash something and go to cook something or just go out, and when I’m back again, I might have forgotten about what I had stashed, especially if it was a very small change.

My current prompt for my shell, zsh, is in two parts. I set the PS1 environnment variable to set the regular prompt, and the RPROMPT environnment variable to set a reversed prompt, starting from the right of the terminal. My reversed prompt just performs a git command to check whether we’re actually in a git project, and get the current branch. Simple, but nice.

I came up to the realization that I could use the exact same idea to know whether I have stashed changes so that I never forget them! Here’s a screenshot to explain that:

As you can see, my prompt now shows me how many stashed changes there are around!

The code

I share the code I wrote with you. Feel free to use it, modify it and share it as well!

# …

function gitPrompt() {
  # git current branch
  currentBranch=`git rev-parse --abbrev-ref HEAD 2> /dev/null`
  if (($? == 0))
  then
    echo -n "%F{green}$currentBranch%f"
  fi

  # git stash
  stashNb=`git stash list 2> /dev/null | wc -l`
  if [ "$stashNb" != "0" ]
  then
    echo -n " %F{blue}($stashNb)%f"
  fi

  echo ''
}

PS1="%F{red}%n%F{cyan}@%F{magenta}%M %F{cyan}%~ %F{yellow}%% %f"
RPROMPT='$(gitPrompt)'

# …

Have fun!

Well, pretty quickly! There’s – yet – no method to make actual renders, because I’m still working on how to implement some stuff (I’ll detail that below), but it’s going toward the right direction!

Framebuffers

Something that is almost done is the framebuffer part. The main idea of framebuffers – in OpenGL – is supporting offscreen renders, so that we can render to several framebuffers and combine them in several fancy ways. Framebuffers are often bound textures, used to pass the rendered information around, especially to shaders, or to get the pixels through texture reads CPU-side.

The thing is… OpenGL’s framebuffers are tedious. You can have incomplete framebuffers if you don’t attach textures with the right format, or to the wrong attachment point. That’s why the framebuffer layer of luminance is there to solve that.

In luminance, a Framebuffer rw c d is a framebuffer with two formats. A color format, c, and a depth format, d. If c = (), then no color will be recorded. If d = (), then no depth will be recorded. That enables the use of color-only or depth-only renders, which are often optimized by GPU. It also includes a rw type variable, which has the same role as for Buffer. That is, you can have read-only, write-only or read-write framebuffers.

And of course, all those features – having a write-only depth-only framebuffer for instance – are set through… types! And that’s what is so cool about how things are handled in luminance. You just tell it what you want, and it’ll create the required state and manage it for you GPU-side.

Textures

The format types are used to know which textures to create and how to attach them internally. The textures are hidden from the interface so that you can’t mess with them. I still need to find a way to provide some kind of access to the information they hold, in order to use them in shaders for instance. I’d love to provide some kind of monoidal properties between framebuffers – to mimick gloss Monoid instance for its Picture type, basically.

You can create textures, of course, by using the createTexture w h mipmaps function. w is the width, h the height of the texture. mipmaps is the number of mipmaps you want for the texture.

You can then upload texels to the texture through several functions. The basic form is uploadWhole tex autolvl texels. It takes a texture tex and the texels to upload to the whole texture region. It’s your responsibility to ensure that you pass the correct number of texels. The texels are represented with a polymorphic type. You’re not bound to any kind of textures. You can pass a list of texels, a Vector of texels, or whatever you want, as long as it’s Foldable.

It’s also possible to fill the whole texture with a single value. In OpenGL slang, such an operation is often called clearing – clearing a buffer, clearing a texture, clearing the back buffer, and so on. You can do that with fillWhole.

There’re two over functions to work with subparts of textures, but it’s not interesting for the purpose of that blog entry.

Pixel format

The cool thing is the fact I’ve unified pixel formats. Textures and framebuffers share the same pixel format type (Format t c). Currently, they’re all phantom types, but I might unify them further and use DataKinds to promote them to the type-level. A format has two type variables, t and c.

t is the underlying type. Currently, it can be either Int32, Word32 or Float. I might add support for Double as well later on.

c is the channel type. There’re basically five channel types:

• CR r, a red channel ;
• CRG r g, red and green channels ;
• CRGB r g b, red, green and blue channels ;
• CRGBA r g b a, red, green, blue and alpha channels ;
• CDepth d, a depth channel (special case of CR; for depths only).

The type variables r, g, b, a and d represent channel sizes. There’re currently three kind of channel sizes:

• C8, for 8-bit ;
• C16, for 16-bit ;
• C32, for 32-bit.

Then, Format Float (CR C32) is a red channel, 32-bit float – the OpenGL equivalent is R32F. Format Word32 (CRGB C8 C8 C16) is a RGB channel with red and green 8-bit unsigned integer channels and the blue one is a 16-bit unsigned integer channel.

Of course, if a pixel format doesn’t exist on the OpenGL part, you won’t be able to use it. Typeclasses are there to enforce the fact pixel format can be represented on the OpenGL side.

Next steps

Currently, I’m working hard on how to represent vertex formats. That’s not a trivial task, because we can send vertices to OpenGL as interleaved – or not – arrays. I’m trying to design something elegant and safe, and I’ll keep you informed when I finally get something. I’ll need to find an interface for the actual render command, and I should be able to release something we can actually use!

By the way, some people already tried it (Git HEAD), and that’s amazing! I’ve created the unstable branch so that I can push unstable things, and keep the master branch as clean as possible.

Keep the vibe, and have fun hacking around!

What people usually do

Uniforms are a way to pass data to shaders. I won’t talk about uniform blocks nor uniform buffers – I’ll make a dedicated post for that purpose. The common OpenGL uniform flow is the following:

1. you ask OpenGL to retrieve the location of a GLSL uniform through the function glGetUniformLocation, or you can use an explicit location if you want to handle the semantics on your own ;
2. you use that location, the identifier of your shader program and send the actual values with the proper glProgramUniform.

You typically don’t retrieve the location each time you need to send values to the GPU – you only retrieve them once, while initializing.

The first thing to make uniforms more elegant and safer is to provide a typeclass to provide a shared interface. Instead of using several functions for each type of uniform – glProgramUniform1i for Int32, glProgramUniform1f for Float and so on – we can just provide a function that will call the right OpenGL function for the type:

class Uniform a where
  sendUniform :: GLuint -> GLint -> a -> IO ()

instance Uniform Int32 where
  sendUniform = glProgramUniform1i

instance Uniform Float where
  sendUniform = glProgramUniform1f

-- and so on…

That’s the first step, and I think everyone should do that. However, that way of doing has several drawbacks:

• it still relies on side-effects; that is, we can call sendUniform pretty much everywhere ;
• imagine we have a shader program that requires several uniforms to be passed each time we draw something; what happens if we forget to call a sendUniform? If we haven’t sent the uniform yet, we might have an undefined behavior. If we already have, we will override all future draws with that value, which is very wrong… ;
• with that way of representing uniforms, we have a very imperative interface; we can have a more composable and pure approach than that, hence enabling us to gain in power and flexibility.

What luminance used to do

In my luminance package, I used to represent uniforms as values.

newtype U a = U { runU :: a -> IO () }

We can then alter the Uniform typeclass to make it simpler:

class Uniform a where
  toU :: GLuint -> GLint -> U a

instance Uniform Int32 where
  toU prog l = U $ glProgramUniform1i prog l

instance Uniform Float where
  toU prog l = U $ glProgramUniform1f prog l

We also have a pure interface now. I used to provide another type, Uniformed, to be able to send uniforms without exposing IO, and an operator to accumulate uniforms settings, (@=):

newtype Uniformed a = Uniformed { runUniformed :: IO a } deriving (Applicative,Functor,Monad)

(@=) :: U a -> a -> Uniformed ()
U f @= a = Uniformed $ f a

Pretty simple.

The new uniform interface

The problem with that is that we still have the completion problem and the side-effects, because we just wrap them without adding anything special – Uniformed is isomorphic to IO. We have no way to create a type and ensure that all uniforms have been sent down to the GPU…

Contravariance to save us!

If you’re an advanced Haskell programmer, you might have noticed something very interesting about our U type. It’s contravariant in its argument. What’s cool about that is that we could then create new uniform types – new U – by contramapping over those types! That means we can enrich the scope of the hardcoded Uniform instances, because the single way we have to get a U is to use Uniform.toU. With contravariance, we can – in theory – extend those types to all types.

Sounds handy eh? First thing first, contravariant functor. A contravariant functor is a functor that flips the direction of the morphism:

class Contravariant f where
  contramap :: (a -> b) -> f b -> f a
  (>$) :: b -> f b -> f a

contramap is the contravariant version of fmap and (>$) is the contravariant version of (<$). If you’re not used to contravariance or if it’s the first time you see such a type signature, it might seem confusing or even magic. Well, that’s the mathematic magic in the place! But you’ll see just below that there’s no magic no trick in the implementation.

Because U is contravariant in its argument, we can define a Contravariant instance:

instance Contravariant U where
  contramap f u = U $ runU u . f

As you can see, nothing tricky here. We just apply the (a -> b) function on the input of the resulting U a so that we can pass it to u, and we just runU the whole thing.

A few friends of mine – not Haskeller though – told me things like “That’s just theory bullshit, no one needs to know what a contravariant thingy stuff is!”. Well, here’s an example:

newtype Color = Color {
    colorName :: String
  , colorValue :: (Float,Float,Float,Float)
  }

Even though we have an instance of Uniform for (Float,Float,Float,Float), there will never be an instance of Uniform for Color, so we can’t have a U Color… Or can we?

uColor = contramap colorValue float4U

The type of uColor is… U Color! That works because contravariance enabled us to adapt the Color structure so that we end up on (Float,Float,Float,Float). The contravariance property is then a very great ally in such situations!

More contravariance

We can even dig in deeper! Something cool would be to do the same thing, but for several fields. Imagine a mouse:

data Mouse = Mouse {
    mouseX :: Float
  , mouseY :: Float
  }

We’d like to find a cool way to have U Mouse, so that we can send the mouse cursor to shaders. We’d like to contramap over mouseX and mouseY. A bit like with Functor + Applicative:

getMouseX :: IO Float
getMouseY :: IO Float

getMouse :: IO Mouse
getMouse = Mouse <$> getMouseX <*> getMouseY

We could have the same thing for contravariance… And guess what. That exists, and that’s called divisible contravariant functors! A Divisible contravariant functor is the exact contravariant version of Applicative!

class (Contravariant f) => Divisible f where
  divide :: (a -> (b,c)) -> f b -> f c -> f a
  conquer :: f a

divide is the contravariant version of (<*>) and conquer is the contravariant version of pure. You know that pure’s type is a -> f a, which is isomorphic to (() -> a) -> f a. Take the contravariant version of (() -> a) -> f a, you end up with (a -> ()) -> f a. (a -> ()) is isomorphic to (), so we can simplify the whole thing to f a. Here you have conquer. Thank you to Edward Kmett for helping me understand that!

Let’s see how we can implement Divisible for U!

instance Divisible U where
  divide f p q = U $ \a -> do
    let (b,c) = f a
    runU p b
    runU q c
  conquer = U . const $ pure ()

And now let’s use it to get a U Mouse!

let uMouse = divide (\(Mouse mx my) -> (mx,my)) mouseXU mouseYU

And here we have uMouse :: U Mouse! As you can see, if you have several uniforms – for each fields of the type, you can divide your type and map all fields to the uniforms by applying several times divide.

The current implementation is almost the one shown here. There’s also a Decidable instance, but I won’t talk about that for now.

The cool thing about that is that I can lose the Uniformed monadic type and rely only on U. Thanks to the Divisible typeclass, we have completion, and we can’t override future uniforms then!

I hope you’ve learnt something cool and useful through this. Keep in mind that category abstractions are powerful and are useful in some contexts.

Keep hacking around, keep being curious. A Haskeller never stops learning! And that’s what so cool about Haskell! Keep the vibe, and see you another luminance post soon!

I came to the realization that I could write a blog entry to discuss designs decisions and, at some extent, what a good design entails. Keep in mind it’s only personal thoughts and that I won’t talk for someone else.

Elegancy

I love mathematics because they’re elegant. Elegancy implies several traits, among simplicity, flexibility and transparency. They solve problems with very nice abstractions. In mathematics, we have a concept that is – astonishingly – not very spread and barely known outside of math geeks circles: free objects.

The concept of free is a bit overwhelming at first, because people are used to put labels and examples on everything. For instance, if I say that an object is free, you might already have associated some kind of lock to that object, so that you can get why it’s free. But we’re mistaken. We don’t need locks to define what free implies. In mathematic, a free object is an object that can’t be defined in terms of others. It’s a bit like a core object. It’s free because it can be there, no matter what other objects are around. It has no dependency, it doesn’t require no other interaction. You can also say that such an object is free of extra features that wouldn’t be linked to its nature.

This free property is a very interesting property in mathematics, because it’s surprisingly simple! We can leverage that mathematic abstraction to software design. I like keeping my softwares as much free as possible. That is – with a more human language to say it – constraining them to keep low responsibilities about what they’re doing.

Responsibility domains

The important thing to keep in mind is that you should, at first, define what the responsibility domain is all about. Let’s say you’d like to create a library to implement audio effects, like the Doppler effect – that effect actually exists for any kind of wave, but it’s interesting to synthetize it for a sound-related application. If you end up writing functions or routines to play sound or to load audio samples, you’re already doing it wrong! You’d have violated your reponsibility domain, which is, “audio effects”. Unfortunately, a lot of libraries do that. Adding extra stuff – and sometimes, worse; relying on them!

A lot of people tend to disagree with that – or they just ignore / don’t know. There’re plenty of examples of libraries and softwares that can do everything and nothing. For instance, take Qt – pronounce cute or cutie. At first, Qt is a library and an API to build up GUIs – Graphical User Interfaces – and handle windows, events and so on. Let’s have a look at the documentation of modules, here.

You can see how the responsibility domain is huge! GUI, radio, audio, video, camera, network, database, printing, concurrency and multithreading… Qt isn’t a library anymore; it’s a whole new language!

People tend to like that. “Yeah, I just have to use Qt, and I can do everything!”. Well, that’s a point. But you can also think it another way. Qt is a very massive “library” you’ll spend hours reading the documentation and will use a lot of different classes / functions from different aspects. That doesn’t compose at all. What happens when you want to – or when you don’t have the choice? – use something else? For instance, if you want to use a smaller–but–dedicated threading library? What happens if you want to use a database service you wrote or that you know it’s great? Do you wipeout your Qt use? Do you… try to make both work in harmony? If so, do you have to write a lot of boilerplate code? Do you forget about those technologies and fallback on Qt? Do the concepts map to each others?

The problem with massive libraries is the tight bound it creates between the libraries and the developers. It’s very hard with such libraries to say that you can use it whenever you want because you perfectly know them. You could even just need a few things from it; like, the SQL part. You’ll then have to install a lot of code you’ll perhaps use 10% of.

KISS

I love how the free objects from mathematics can be leveraged to build simpler libraries here. The good part about free objects is the fact that they don’t have any extra features embedded. That’s very cool, because thanks to that, you can reason in terms of such objects as-is. For instance, OpenAL is a very free audio library. Its responsibility domain is to be able to play sound and apply simple effects on them – raw and primary effects. You won’t find anything to load music from files nor samples. And that’s very nice, because the API is small, simple and straight-forward.

Those adjectives are the base of the KISS principle. The ideas behind KISS are simple: keep it simple and stupid. Keep it simple, because the simpler the better. A too complex architecture is bloated and ends up unmaintainable. Simplicity implies elegancy and then, flexibility and composability.

That’s why I think a good architecture is a small – in terms of responsibility – and simple one. If you need complexity, that’s because your responsibility domain is already a bit more complex than the common ones. And even though the design is complex for someone outside of the domain, for the domain itself, it should stay simple and as most straight-forward as possible.

API

I think a good API design is to pick a domain, and stick to it. Whatever extra features you won’t provide, you’ll be able to create other libraries to add those features. Because those features will also be free, they will be useful in other projects that you don’t even have any idea they exist! That’s a very cool aspect of free objects!

There’s also a case in which you have to make sacrifices – and crucial choices. For instance, event-driven programming can be implemented via several techniques. A popular one in the functional programming world nowadays is FRP. Such a library is an architectural codebase. If you end up adding FRP-related code lines in your networking-oriented library, you might be doing it wrong. Because, eh, what if I just want to use imperative event-driven idioms, like observers? You shouldn’t integrate such architectural design choices in specific libraries. Keep them free, so that everyone can quickly learn them, and enjoy them!

I like to see good-designed libraries as a set of very powerful, tiny tools I can compose and move around freely. If a tool gets broken or if it has wrong performances, I can switch to a new one or write my very own. Achieving such a flexibility without following the KISS principle is harder or may be impossible to reach.

So, in my opinion, we should keep things simple and stupid. They’re simpler to reason about, they compose and scale greatly and they of course are easier to maintain. Compose them with architectural or whatever designs in the actual final executable project. Don’t make premature important choices!

I want to talk about it, because it’s not really the type of keyboard you see quite often, and I guess it comes with many questions.

# The Corne, a 42-key beauty

• The Corne, a 42-key beauty
  • Layers and mod-tap
    • The base layer
    • The symbols and numbers layer
    • The nav/multimedia layer
    • The gaming layer
  • Home-Row-Mods: a false good idea?
    • First issue: shifts are too important
  • Second issue: latency
  • What’s next

The first thing you might have noticed is that it’s a split keyboard. There are a lot of litterature lying around explaining why it’s actually a better setup than a single piece of hardware — better shoulds position, better wrists alignment, etc. Plus, because you have the choice to put both sides close to each other, or far apart, it can easily be adapted to your morphology.

Secondly, it’s easy to see that the Corne has only 42 keys. That comes with a massive advantage to me: when your fingers are resting on the home row, every key on the main grid (i.e. excluding thumb keys) are only one key away. That minimizes a lot the movements you will be doing. For instance, reaching the key on the top-left corner doesn’t require any kind of stretch of my hand, nor rotation. It’s already an improvement compared to the Voyager, which has an additional row above; reaching its top-left key does either make you extend your pinky in a non-natural way, or does make your wrist slightly rotate towards the keyboard, which is not comfortable on the long run.

So every key is reachable by just slightly moving the fingers. It’s a fantastic feeling, especially because of the column staggered keys, which follows the curve of the hands. I’ve been touch typing for a very long time now (> 10 years), so switching to that keyboard wasn’t that hard — especially because I had switched to the Voyager prior to this — but it would a lie to tell it’s natural. It takes some time and practice. If you have never learned to touch type properly, you might suffer a lot. I’ve been touch typing in bépo, so it took me a couple days to be fully comfortable with the Voyager and the Corne.

Finally, I want to talk about the thumb keys, because it’s going to be a central topic of this article. The Corne has three on each side, for a total of six, which is still two more than the Voyager, which has two on each side for a total of four. Thumb keys are a way to leverage the fact that on regular keyboards, we barely use them — on my 60%, I used the left thumb for the space bar, and the right thumb for alt-gr, but that’s pretty much it, besides some occasional modifiers.

So that’s it for the introduction. Let’s dig into my setup.

# Layers and mod-tap

Because of the few amount of keys, the whole idea is that you want to create layers, and dedicate some keys to activate modifiers and layers. Let me explain.

# The base layer

The base layer is usually full of letters, what you will be typing the most. In my case, this is my base layer — · means that nothing is bound there:

T b é p o è      ^ v d l j z
E a u i e ,      c t s r n m
% à y x . k      ' q g h f w
      1 Z ·      B R ·

Here, T stands for tab, E is escape, Z is space, S is left shift, B is a backspace, R is return. Some keys have a hold action, which is what we usually call mod-tap: holding the key down and pressing another key does something different. Even though some keys do not have any tap action (for instance, the left inner thumb key or the right outer thumb key), I can still hold them to have a specific action. Here’s the base layer again, but with the actions when keys are held:

M · · · · ·      A · · · · ·
· · · · · ·      · · · · · ·
A C · · · ·      · · · · · ·
      2 · S      S 1 G

M is meta, A is alt, C is control, S is shift, and G is alt-gr. That latter modifier is especially important in bépo, as it opens up many many characters that the bépo layout has to offer — for instance, … is alt-gr + ., ’ is alt-gr +, œ is alt-gr + o, etc.

The 1 and 2 are not typos, and I’ll talk about it pretty soon.

You might have noticed that I have mirrored the shifts on the inner thumb keys. This is important to me, because shift is actually a very important modifier: it’s by far the modifier I type the most, and I need to be able to interleave normal typing on the main grid with shifting modifiers. Think about the sentences you’re currently reading, and where I might have pressed shift. « I » in the middle of a sentence requires a shifted « i ». Every start of sentence requires a shifted letter.

Now let’s see how I type numbers and symbols.

# The symbols and numbers layer

I used to have this layer mirrored on the outer thumb keys, but such keys are not really practical to hit while fast typing — I usually type around 145 words per minute, so I’m pretty fast. Today, that layer is accessed via the 1 layer key on my base layer — middle right thumb key. If I just tap it — basically press and release without touching any other key — it acts as return. If I hold it, it activates the symbols and numbers layer.

Let’s see that layer:

0 1 2 3 4 5      6 7 8 9 ° ·
$ " < ( ) >      @ + - / * =
# | & [ ] ~      — { } « » ç
      · _ S      · □ ·

The □ is the held key there. So, if I want to type (), I can simply hold the right middle thumb key and press ie to yield (). How to do ->? Easy: press the layer key, and s,. The placement of the keys are easy to remember.

One that is very important to me is _: that one is usually accessed via alt-gr + <space>, but because my alt-gr is on the right outer thumb key, it’s not super practical to type. However, I also realize I don’t type underscore that often, and that I might put backspace there instead. This is still something I’m figuring out.

This is a pretty fantastic layer, because everything is really easy to activate. The weirdest key is definitely —, since the right index and thumb fingers are super close to each other — but it’s a fair tradeoff as I very rarely type this key, but still require it.

The ç is an anomaly due to bépo v1.0, which doesn’t treat alt-gr + c as ç, but as ©, which I do not really care. I need to switch to bépo v1.1 / AFNOR, which comes with issues with ' vs. ’; yes, French is hard!

You can see that I moved 0 on the left side; this is to prevent having to press it on the right side with a weird crab hand; 9 is fine, but 0 would be too weird on the right side.

# The nav/multimedia layer

Next up is my navigation layer, accessible with the 2 key.

That layer is important for many different softwares, including shells, chat applications, etc. It’s the layer that holds arrow keys, page up / down, home / end, etc. Because it’s also a pretty empty layer, I hijack it to also hold useful multimedia keys, such as volume up, down, mute, and the print key:

3 · · · · VU      P H U E · ·
· · · · · VD      l d u r · ·
· · · · · VM      · · D · · ·
      · · ·       S · ·

3 is my gaming layer goto-key (more on that later), VU, VD and VM are volume up, volume down and volume mute; P is the print key, H and E are home and end keys, U and D are page up / down, and l, d, u and r are the left, down, up and right arrow keys.

Yes, the volume keys are not optimally placed, and I might move them on the right side eventually. The reason is that since the nav key is pressed on the left outer thumb key, and is not practical to type keys on the left side of the keyboard.

Alright, now let’s see the last layer I have defined:

# The gaming layer

The gaming layere is a bit weird and heavily optimized towards the games I play — mostly, Quake, or games with a need of special keys all around my movement keys:

T è b é p o      · · · · · 0
E , a u i e      · · · · · ·
%   à y x .      · · · · · ·
      k Z S      · · ·

0 here goes back to the base layer. As you can see, the layout is weird, but the idea is that it will make most games work out of the box with ASDF key, but I shift them by 1 column to have the direction keys right under the fingers without moving from the left home row. I don’t think there’s anything to debate here, it’s just a shifted base layer.

# Home-Row-Mods: a false good idea?

Something that I wondered a lot is whether I want to use HRMs. HRMs assigns hold-actions on the four keys of the resting home row position to control, alt, shift and meta, on each side of the keyboard. That allows to have access to modifiers without moving your fingers. This, basically:

· · · · · ·      · · · · · ·
· M A C S ·      · S C A M ·
· · · · · ·      · · · · · ·
      · · ·      · · ·

I is smart, but not practical, at least not to me. The idea is that it relies heavily on how it’s implemented. I use QMK to build my firmwares, and when you use HRMs, there is a setting you cannot really do without: permissive-hold. In short, permissive-hold changes the behiavor of mod-tap keys in order to active the hold action more often. By default, to get the hold action of a key, you have to press it for more than TAPPING_TERM — usually set to 200ms. With permissive-hold, the hold action can be triggered sooner by pressing the mod-tap key and pressing and releasing an other key, while still holding the mod-tap key. Imagine a mod-tap key that prints t if tapped, but right shift if held. With permissive-hold, you can trigger the hold action without having to wait for TAPPING_TERM:

1. Press down the t key.
2. Press down the a key for instance.
3. Release the a key.
4. You get A printed.
5. Release the t key.

This is required if you want to use HRMs and still trigger hold actions without having to wait for TAPPING_TERM. The other mode — which I use, and I’ll explain why — is named hold-on-other-key-presses, which changes the behavior of mod-tap keys like so:

1. Press down the t key.
2. Press down the a key.
3. A is printed.
4. Release the t key.
5. Release the a key.

This is usually how I type when typing fast, so that mode is really horrible as it generates tons of misfires with HRMs. However, it’s the mode to use for mod-taps set on keys you rarely use doing rollovers, such as my backspace key which is also my right shift: I usually tap it once or twice, without holding on any other key. Whenever I press it and hit another key, 100% of the time, I wanted a shifted key.

So having the shifts on the thumbs and hold-on-other-key-presses allows me to shift keys immediately, which is really important while shifting keys in the middle of sentence or words.

But there’s more. I really tried HRMs — switched to permissive-hold, and removed the shifts from the thumbs. I basically have two big issues with it.

# First issue: shifts are too important

Because shifts are really special and important keys, having them on the home row just gets in the way too much. You don’t realize it until you have to type complex sentences with tons of shifting, but I realized that I often start shifting a character, and my fingers are already moving to the next digram or trigram. Having two fingers forced to ping-pong between shifts, and because you cannot press them quickly (remember: permissive-hold requires you to press a key, and press and release another key before releasing that first key first to get she hold action! this is not how I fast touch type, so I really feel slowed down with this setup).

This issue also outlines a misconception — or a paradox, pick the word that pleases you the most there — about symmetrical shifts. Whenever I need to shift a single character, I will pick the shift key on the opposite hand. For instance, « I » is an i, which I type on the left side on the keyboard, and thus I shift it mith the right middle thumb key held: « I ». However, some words sometimes require to shift a couple keys one after the other. For instance, there’s a city in France called Anet. Let’s yell at that city, for whatever reason: ANET!

The way I type ANET is by following the opposite-hand rule I mentioned above for the first letter, but then I just keep the shift key held and finish the rest of the word. It looks like this:

1. Press the right middle thumb key.
2. Press a.
3. Press n.
4. Press e.
5. Press t.

Now, with HRMs, you cannot do that, because every letter of anet alternates between sides, this is what it would look like:

1. Press t.
2. Press a.
3. Release a.
4. Release t.
5. Press e.
6. Press n.
7. Release n.
8. Release e.
9. Press t.
10. Press e.
11. Release e.
12. Release t.
13. Press e.
14. Press t.
15. Release t.
16. Release e.

Yeah. It’s crazy. And here, I really took the worst example, but in practice, that kind of alternating pattern happens more often than you’d think. For this special reason, I think shifts should be treated as a special key, and that HRMs should not include shifts.

# Second issue: latency

So this one is probably the most infamous one coming with HRMs, and either you just don’t care, or you’re like me and can’t stand it. I’ve heard and read people stating they do not feel it, but on my side, it’s very similar as playing at 30 FPS vs. playing at 244 FPS: I just do not comprehend how someone cannot feel the difference.

What happens with HRMs introducing latency is easy to understand, and it’s about the taps, not the hold action. With permissive-hold, as seen above, if you press and release another key while holding down a mod-tap key, you will get the hold action. If you fast type, and have the interleaving of press / release, you will get taps, which is fine, but. Because they mod-tap key might be a shift key if hold, the firmware must wait until the end of the tapping period, or that the key is released to actually consider it a tap. What it means if that, if you press a key and release it, the associated character will not immediately appear on screen: it will take the time between the moment you press the key and the moment you release it. The character will be printed on screen when you release the key.

You can state that the difference in time between a press and the associated release is negligeable for taps, right? Right? Well, not really. Even when I type really fast, I cannot easily go under 60ms on single taps, and rarely under 40ms for super fast burts.

A simple and KISS way to check this out: open your web browser console, and add this JavaScript to any page:

document.onkeypress = () => { down = window.performance.now(); }
document.onkeyup = () => { console.log(window.performance.now() - down); }

Yeah it’s terrible code but who cares for such a test. Just simply tap a single key the way you normally type. This is just there to show you that with HRMs, you will get most of the time around 60ms of latency if you are a fast typer, probably much more for most of the people.

Just for comparison, for a game to run at 30 FPS, it must compute every frame with a maximum render time boundary of 1/30 ~= 33ms. If a game were taking 70ms to render each frame, it would run at 14 FPS. So this is akin to playing a game at full FPS, like 240 FPS, but having your mouse and keyboard input at 14 Hz. I don’t think you’d like that. So yes, HRMs introduce a very noticeable lag, and I cannot bear it. It feels terrible. Something that people also seem to ignore is that the human brain is pretty good at making correlation between a sensation and what they eyes see. For instance, if you have ever watched a movie with out-of-sync video and audio, you might have realized how quickly you can notice that something is wrong: just having a couple milliseconds of delay between what you see and what you hear already feels bad.

Fun fact: because the speed of light is the top-speed, and the speed of sound is pretty slow compared to light, desynchronizing audio and video will not feed as bad if the audio arrives after the video; think thunderstrucks for instance. But your brain will freak out if audio arrives before.

Contrary to what many people will say, this is an unsolvable problem. There are some options to try to mitigate the problem — for instance, Flow Tap — but those still introduce tradeoffs that are not good ones to me. Flow Tap introduces a timeout after which tapping a key will disable HRMs, so you can speed through without having the latency. However, it doesn’t solve the problem in all situations, and I still think that permissive-hold is not as good as hold-on-other-key-presses on keys you don’t often use.

# What’s next

I’m considering changing a bit my layers, especially regarding backspace, return, etc. All in all, I think that having so few keys does require some tradeoffs, and I think I’ve found the perfect setup for my personal use. Who knows, maybe some day I will find a revolutionary way to implement HRM and will switch to them, but for now, I don’t think it’s actually that good.

It was a lighter topic, but still pretty fun to talk about it. In the end, I just love keyboards.

Keep the vibes!

Today I released a new version of warmy, the warmy-0.6.0 release. That release kicks in with a few additions, among:

• Complete rewrite of its internals, enabling optimizations (mostly about allocations).
• A nasty IO-bound bug was removed.

That bug caused long-lasting reloads on stream-copied resources go into weird behavior.

The bug

In order to get the bug, I must give a bit of context.

Imagine you have a large resource, like a 4K texture or a big mesh. Whenever your favourite software writes it to disk, it’s very likely it’ll stream chunks by chunks of bytes. For instance, it might choose to copy the resource 2 MB by 2 MB on disk. On each copy, your file system will generate WRITE events, that warmy will intercept. Before warmy-0.6.0, the default behavior was to reload the resource on the first WRITE event, which was already wrong, because only a very small part of the resource would have changed – I actually witnessed weird behaviors with textures in a demo of mine I’m working on, not seeing the new textures in my demo.

But there’s worse. There’s a parameter you can set of your Store, called update_await_time_ms. That parameter gives warmy a hint about how much time must have passed since the last update in order to effectively call the Load::reload function. However, this is a bit twisted, because if the resource takes more time to reload than update_await_time_ms, it’ll get repeatedly loaded – for as many as WRITE events were generated. This is a bit sick, yeah.

The fix

The fix was pretty simple: change the semantics of that update_await_time_ms. In 0.5.2, it has the default value of 1s, meaning that a resource wouldn’t reload if it was reloaded less than a second ago. The new semantics works on the future. Whenever a WRITE event is intercepted, warmy will call the Load::reload function only if no WRITE event is intercepted in the next update_await_time_ms. It’s a bit like the implementation of a click and a double click: you must wait a bit after you got a MouseRelease event in order to interpret is as a Click because another MouseRelease could arrive soon (generating a DoubleClick if it’s soon enough).

You’ll also notice that update_await_time_ms name sticks better to the new semantics!

In 0.5.2, the default value for update_await_time_ms was 1s. If we kept that value, it would result in a pretty bad overall latency. The value was lowered to 50ms instead.

You can still tweak that value if it doesn’t suit your needs.

More information can be found in the changelog.

On the future of the crate

I’ve been very happy with what warmy has brought to me so far. Other people also gave it a try and for now seem to enjoy it. I’ve gathered a few ideas for the future, based on IRL talks over a beer several beers, and GitHub issues / pull requests:

• We want to be able to pass around a context when loading and reloading. This would enable tweaking the loading behavior of the objects and pass values from the calling function. However, even though the feature is well-identified, it is not well-defined in warmy and more research must be done.
• Multithreading. There’re a few ideas I’d like to implement, like an IO thread that would do all the file IO-bound computation and dispatch the bytes to the other threads; and using Future.

Feel free to test it, and as always, keep the vibes!

I pushed a pull request to Edward Kmett’s either package to implement two functions some guys was complaining not to find: flipEither :: Either e a -> Either a e and flipEitherT :: EitherT e m a -> EitherT a m e.

When implementing the functions, I wondered: “Hey, flipping stuff is a pretty common operation. Don’t we have an abstraction for that yet?”. I haven’t found any.

Meet Swap

I decided to make a little typeclass to see what it’d be.

class Swap s where
  swap :: s a b -> s b a

instance Swap (,) where
  swap (a,b) = (b,a)

instance Swap Either where
  swap = flipEither

-- let’s go wild and fooled
instance Swap Map where
  swap = fromList . fmap swap . toList

If you think that’s handy, I’ll write a little package with default instances to make it live.

Happy hacking folks!

Kakoune and the shell

Because Kakoune doesn’t have a plugin interface, you are limited to what is called kakspeak, which is basically what you can type in the command line (the : line). Contrary to other editors like Vim, everything you type in that : line can be laid out in a file (a .kak) and be interpreted directly. Another cool aspect about that is that you can just select some part of a .kak file, and simply type :^r.<cr> – in Kakoune, the . is the selection register. That will execute the Kakoune command.

This property is really good, but still, you are limited in what you can do. Among the command you will be running:

• declare-user-mode, to declare your own mode (yes, like insert mode, normal mode, etc.). Yes it’s a default / core feature and yes it’s excellent.
• set-face to declare / update highlighting groups.
• execute-keys and evaluate-commands to execute keys as if the user typed them, in sandboxed / isolated environments. This is incredibly powerful, as you can run some commands that creates, updates registers, move the cursor around, etc. but since it’s sandboxed, as soon as the command is done, you get back to the state you were in before issuing the commands.
• And more.

There is nothing to write plugins… except one thing.

String expansions

Kakoune has this concept of expanding strings. It’s the same feature as in any other editor: some strings can contain special keywords and identifiers to replace their content with what they hold. For instance, in your shell, it’s very likely that this string will contain your username: "$USER", or this will be the current year "$(date +%Y)". However, this will remain verbatim: '$(date +%Y)'. The reason is that, in a shell, " expands while ' doesn’t.

Kakoune has the same mechanism, but the syntax is different, and Kakoune has different source of expansions. For instance, %opt{foo} will be replaced why the foo option, that can be set with set-option. %val{timestamp} contains the current timestamp of the buffer, etc. etc.

There is one interesting expansion that Kakoune has: %sh{}. This is shell expansion. It will execute its content inside a thin shell. For instance, try open Kakoune and enter in the command line, something like :echo %sh{date +%Y}. You can see where we are going here. We can use that to run arbitrary shell commands.

Kakoune is monothreaded, so when you run a command in a shell, it blocks until the shell command finishes. On its own, it’s not that bad. It forces us to call short-living shell commands.

This mechanism allows us to run external programs, but it doesn’t tell us how we can call back Kakoune.

Kakoune and UNIX sockets

Kakoune is monothreaded, but it’s concurrent. It listens on a UNIX socket Kakoune commands that any external programs can send, via the kak -p interface.

I initially tried to send content to the UNIX socket programmatically, but it wasn’t planned for that, and hence hit issues while doing so. It hurts to say that but you have to spawn a programm running kak -p; forget about writing directly into the UNIX socket for now.

So, with this mechanism, we can:

1. Run a short-living program via %sh{} expansion to talk to, for instance, a server, quickly accepting our request and treating asynchronously.
2. Then, once the request is handled, send back the response to Kakoune by running a kak -p program, using the UNIX socket.

The cool thing is that Kakoune follows a server/client architecture and has a session identifier (you must pass it to kak -p). You can retrieve that value with %val{session} — and inside %sh{} expansion, it’s available as an environment variable $kak_session.

And here you have it. The formula I’ve been using successfully to add tree-sitter support to Kakoune. Whenever we need to highlight the buffer again, simply craft a small request to send to the kak-tree-sitter local server and immediately get control back (so that the UI doesn’t freeze). Then, at some point, the highlight request is computed and arrives via kak -p in Kakoune.

But you might have a question… how do I deal with the buffer content? Indeed, the only thing we can do with with %sh{} is starting a program in a shell. We could do something like laying the buffer content on the shell invocation but that’s seriously limiting (especially on giant buffers). We could put the content of the buffer in an environment variable, but it would suffer from the same problem (and damn it’s so dirty). What else?

The final part of the recipe: FIFOs

UNIX systems have this incredible thing called FIFOs. I FIFO — also named pipe — is a special kind of file. It lives on your filesystem (so it’s located at a given path), a bit like UNIX socket. However:

• Its content never exists on the filesystem. It’s all in-memory.
• It’s a streaming primitive, so you cannot just write into it and assume it’s stored somewhere.
• Think of it as a buffer between two entities: a reader and a writer.
• When a reader reads from it, it blocks until data is available to read.
• When a writer writes to it, it blocks until a reader is available to read.

So it implements a rendez-vous buffer between a single reader and a single writer. And yes, the | in your shell is using something that behind the scene. The thing is: you can create your own, and you don’t even need to write any code. Enter the mkfifo program. For instance, you can try it out on your own by creating a FIFO (let’s call it rdv), reading its content with cat first (you’ll see the cat process freeze, so you will need another terminal session!) and then write content to it, for instance with echo:

mkfifo /tmp/rdv

# in shell 1
cat /tmp/rdv

# in shell 2
echo "Hello, world!" > /tmp/rdv

As seen as the echo starts writing, you can see cat return the result.

Now replace cat with tail -f 😏.

Anyway, Kakoune has a mechanism where it scans %sh{} blocks and it sees $kak_command_fifo and/or $kak_response_fifo, it will create those FIFOs for us (and manage their lifetimes). Because we are in the shell, we can write Kakoune commands to execute in $kak_command_fifo, which will be executed as soon as you’re done writing to the FIFO, and you can read $kak_response_fifo from, for instance, an external program, to get more data from Kakoune.

This is the exact mechanism that is used to stream buffer content between Kakoune and kak-tree-sitter. Here’s the Kakoune commands used to highlight a buffer:

# Send a single request to highlight the current buffer.
define-command kak-tree-sitter-highlight-buffer -docstring 'Highlight the current buffer' %{
  nop %sh{
    echo "evaluate-commands -no-hooks -verbatim write $kak_response_fifo" > $kak_command_fifo
    kak-tree-sitter -s $kak_session -c $kak_client -r "{\"type\":\"highlight\",\"buffer\":\"$kak_bufname\",\"lang\":\"$kak_opt_filetype\",\"timestamp\":$kak_timestamp,\"payload\":\"$kak_response_fifo\"}"
  }
}

I won’t explain the protocol into two much details, but the important part is that I start a shell to run kak-tree-sitter … -r, format the request (it’s just plain JSON), and ask kak-tree-sitter -r to read from $kak_response_fifo. You can see in the previous line that we write to it (the write $kak_response_fifo basically means to write the content of the current buffer to $kak_response_fifo, which is a file — everything is a file in UNIX!).

kak-tree-sitter -r is made in a way so that it exits quickly. It will read from the FIFO file and forward the request to the kak-tree-sitter server, which, in turn, will move the request to a different thread so that it can quickly mark the request done. The whole thing is then synchronous, but extremely fast. The only bottleneck we have here (and mind it, it’s important) is that we must read from the FIFO synchronously in the shell kak-tree-sitter -r.

How’s it going?

I’ve been running with this setup for weeks now, since I started kak-tree-sitter around the end of April, 2023. The code you read above runs in a couple of Kakoune hooks (the same kak-lsp is using or very similar), which waits for a idle time after the buffer is edited (in practice, 50ms after the last edit operation). And it’s already fast enough.

However, there is a nice optimization that we could do here. See, the only reason to start a shell is to format the request to send to kak-tree-sitter, the server. We can completely by-pass it. Instead, we could:

• Create the FIFO in kak-tree-sitter, when a new Kakoune session is created.
• Write the requests to the FIFO handled by kak-tree-sitter. No shell is needed to write to files!
• The rest is the same, since the server sends back highlighting Kakoune commands via kak -p.

This on its own should greatly enhance performance, which is already pretty good, even with the shell. I plan on working on this enhancement in the upcoming days.

Alright, but what’s wrong then?

Well, there is one thing. See, in order to support tree-sitter, I had to write this kak-tree-sitter server, and write buffer contents to FIFO files. kak-lsp is doing the same. All of that is synchronous, and even if it’s fast, it’s still two sequential locks of the buffer, copy of the buffer, etc. On my side, I’m currently exploring the %val{history} expanded variable, which contains the textedit operations (so that I don’t have to stream a full buffer and diff in kak-tree-sitter anymore, but instead just small chunks of updates).

However, the problem still holds. The design of Kakoune is nice when you’re working alone with your small integration, but when you have two big ones (LSP and tree-sitter are not small environments), I think Kakoune shows its limit.

We – @krobelus and I – discussed that matter. A middleware program or a change in how Kakoune interfaces with external programs, especially for buffer streaming, would be greatly appreciated. But it’s not currently the case. And even if we decide to come up with a middleware – that could act as a buffer cache / map, for instance – we would still be writing buffer contents to different FIFOs.

Another thing that I’ve been thinking about a lot is… is this the right approach? Something like tree-sitter seems to be ideal as an embedded library, directly inside the code of your editor. Especially since Kakoune is all about selections, being able to have semantic selections by default seems like something pretty good. tree-sitter requires some runtime resources (relocatable objects like .so / .dylib, queries, etc.)… and it’s the same thing with the regular Kakoune highlighters (by default, Kakoune doesn’t have any; they come bundled up with your distribution).

I’m not entirely sure the approach scales very well. Of course, I still think the design is excellent, and it prevents too much maintenance on Kakoune, which is also a good thing. For instance, this runtime resource problem is something I’m solving in kak-tree-sitter (and easing with a controller tool called ktsctl, which can download, compile and install grammars and queries for you), but still. The current version of kak-tree-sitter only supports semantic highlighting, not text-objects just yet (but it shouldn’t be too hard to add). The state of the project is not completly ready for people to jump in (the wiki is not written), but if you tag along on Matrix, I can help you get started.

Please consider kak-tree-sitter as an experimental project, because I’m not sure this is the right approach, even though it’s probably the only one for Kakoune for now (I don’t think @mawww plans on integrating it into the core of Kakoune).

Keep the vibes!

spectra is a crate I’ve been maintaining for a few months / years now. It’s a crate that I mainly use for demoscene productions (I released two with it, Céleri Rémoulade and Outline Invitation) but I also use it to play around and experiment new rendering, animation and video game techniques.

The crate features a few things that I enjoy daily. Among them:

• A rendering layer, backed by another crate of mine, luminance.
• Hot reloading for scarce resources – the feature was extracted into warmy as folks on IRC asked me to!
• Models (.OBJ) loaders.
• Audio simple routines (basically: play a soundtrack and get metadata about it).
• A shading language of mine, built upon GLSL (I’ll make a separate blog entry about this one).
• Splines, animation primitives, etc.
• And more.

The last thing I’ve been working on is being productive. That might seem counter-intuitive, but when you start building “frameworks” and “engines”, you actually end up writing a lot of code for “the beauty and power of it” and lose focus on what’s important: releasing things. I know that and looked for what I could enhance to augment my productivity.

Among interesting topics I came up with, I stated:

• JSON live editing with warmy in spectra is totally awesome (it was an improvement for my productivity when I was, before, editing the curves in the code – you know, the edit-compile-run cycle?). However, even if it’s awesome, it’s quickly limiting, especially when you want to edit quaternions (rotations) or find a specific color mask. So I’d like to implement a way to pick objects around with the mouse and edit them via a nice UI.
• Scripting. Somewhat. I partially done that with hot-reloaded shaders!
• Better tooling, especially for live-coding.

Clearly, there’s something about live coding.

Introducing scripting in Rust

I went through several thought processes. I first had a look at Lua, because lots of people think it’s cool. However, I don’t like the syntax nor the overall safety the language gives doesn’t give you.

I remembered that almost a decade ago, when I was 15 or 16 year old, I implemented some kind of a plugins system in C++ for a side project of mine. The idea was simple:

• Find functions with dlopen, dlsym to open a relocatable object / archive (.so / .dll).
• Use them at runtime.

This is not magic. When you link a crate / library, by default, a lot of compilers will perform static linking. That’s the case of ghc or rustc, in most situations.

If you don’t know yet, a library / crate / dependency statically-linked in a program means that all of its symbols (the ones used in the program at least) are placed in a specific section of the generated executable, so that those symbols have a proper address and “come with the binary”.

Dynamic linking, on the other side, is a way to express a dependency between your binary and some code, which is most of the time living in a relocatable archive (.so on Linux or macOSX, .dll on Windows – macOSX also uses .framework but it’s just for the overall idea). When your binary needs to call a function defined in a dynamically linked library, it’ll open the shared library with the dlopen function, try to locate the function by giving its (unmangled) name to dlsym or dladdr for instance, if the function exists, you’ll be able to run its code.

There exists attacks and interesting things you can do with dynamic libraries. Because the code doesn’t live in the binary, you can for instance replace a legit and safe .dll on Windows with a malicious one. Or you can patch a buggy dynamic library by shipping a new .dll / .so without having its dependent to re-compile their applications (which would be needed in case of static linking). Another funny thing you can do: fraps, a real-time recorder for your video games, performs some kind of .dll injection by pushing an OpenGL .dll into your binary (Windows allows this) and replacing some known functions. For instance, the function responsible in swapping your renderbuffer chains. It can then intercepts pixels regions, adds overlay, etc. Fascinating! ;)

Anyway, the idea is that whenever you link a dynamic library, your compiler / linker will just insert the required code in your binary (think of dlopen) so that your binary can load code at runtime. It’s then pretty easy to implement a plugin system:

• Define your interface. Define a function type you will need to take from the dynamic library and give that interface a name so that you can look it up in the library.
• Open the library, find the symbol, and just use it!

So you could imagine, as a very simple start example, a loop that would simply invokes such a function. Whenever you change the library, the code getting ran will automatically changes as well!

Scripting in spectra

So I decided to reproduce that in Rust using the few crates of mine:

• spectra: obviously, since it’s the crate that will receive the feature first.
• warmy: it’s indirect since spectra re-exports it and adds a few things over it, like a resource system (error messages, timing loading and reloading, etc.).
• libloading: the Rust way to go with dynamic libraries (it has an unsafe interface though).
• That’s pretty much everything of it.

For unix plateforms, you can see that libloading is just a smart wrapper over the functions I mentionned. See for yourself.

Ok, so, let’s try the experiment!

Let’s write plugins in Rust!

Foreword

The first thing we must accomplish is very simple: we want to be able to load some (Rust) code at runtime, inside our application. For achieving that goal, we need to load a dynamic library (.so on Linux) with libloading::Libray::open. Once we have the library, we can just look a symbol up with libloading::Library::get. In case of a successful lookup, that function returns a value of type Symbol, which implements Deref for the symbol you’re asking.

Dynamic library typically gathers functions, so we’ll be looking for fn symbols.

However, we don’t have any dynamic library yet. We only have… a Rust file – .rs. How can we get a dynamic library out of it?

Generating a dynamic library

This is actually pretty simple. We’re gonna start easily by making a .so that will just contain a function called hello_world that will just print "Hello, world!" on stdout. You all know how to implement such a function:

/// hello_world.rs
pub fn hello_world() {
  println!("Hello, world!");
}

We make the function pub so that it gets actually exported when we generate the dynamic library.

Copy that code and put it in, for instance, /tmp.

Then, let’s generate a dynamic library!

cd /tmp
rustc --crate-type dylib hello_world.rs

Once rustc returned, you should see a new file in /tmp: /tmp/libhello_world.so! Here we are! We have a dynamic library! Let’s try to find our function in it with the handy nm program.

I reckon nm is already installed on your machine. If not, it should come with packages like base-devel, build-essentials or that kind of meta packages.

nm /tmp/libhello_world.so | grep hello_world
0000000000000000 N rust_metadata_hello_world_8787f43e282added376259c1adb08b80
0000000000040ba0 T _ZN11hello_world11hello_world17h49fe1e199729658eE

Urgh. We have a problem. Rust has mangling for its symbol names. It means that it will alter the symbols so that it can recognize them in a dynamic library. For instance, the mangle version of a function name foo defined for a type A won’t be the same as the one of foo defined for a type B. However, in our case, we don’t want mangling, because, well, we won’t be able to lookup the name up – no one will even try to guess that _ZN11hello_world11hello_world17h49fe1e199729658eE function name.

rustc has a very simple solution to that: just tell it you don’t want a symbol’s name to be mangled. It’ll be stored in the dynamic library the way you write it. This is done with the #[no_mangle] attribute.

/// hello_world.rs
#[no_mangle]
pub fn hello_world() {
  println!("Hello, world!");
}

Now recompile with the same rustc line, and run the nm + grep oneliner again.

nm /tmp/libhello_world.so | grep hello_world
0000000000040b70 T hello_world
0000000000000000 N rust_metadata_hello_world_8787f43e282added376259c1adb08b80

Now you can see there exists a symbol called hello_world: this is our symbol!

Load the library in Rust and read a symbol

For our little example here, we’re just gonna load and run the code in a new project.

cd /tmp
cargo new --bin dyn-hello-world
     Created binary (application) `dyn-hello-world` project
cd dyn-hello-world

Edit the Cargo.toml file to include libloading = "0.5" in the [dependencies] section. Ok, we’re good to go. Run a second terminal in which you run this command to automatically check whether your code is okay:

cd /tmp/dyn-hello-world
cargo watch

Let’s edit our main.rs file.

extern crate libloading;

use libloading::Library;

fn main() {
  let lib = Library::new("/tmp/libhello_world.so").unwrap();
  let symbol = unsafe { lib.get::<extern fn ()>(b"hello_world").unwrap() };

  symbol();
}

It should check. A bit of explanation:

• The Library::new accepts as first and only argument the path to the dynamic library. In our case, we just use our freshly generated libhello_world.so.
• The unsafe get call takes the symbol we look for as argument and returns it if it’s found it.
• Because Symbol implements Deref, here, we can directly call the function with symbol().

Now compile and run the code:

cargo build
   Compiling cc v1.0.4
   Compiling libloading v0.5.0
   Compiling dyn-hello-world v0.1.0 (file:///tmp/dyn-hello-world)
    Finished dev [unoptimized + debuginfo] target(s) in 2.63 secs
cargo run
    Finished dev [unoptimized + debuginfo] target(s) in 0.0 secs
     Running `target/debug/dyn-hello-world`
Hello, world!

Hurray! We’ve just built our first dynamic library loader!

Let’s compile Rust… from Rust!

Ok, now, let’s iterate: we need to compile the Rust code from our Rust code (haha). For doing that, we’ll need to generate the dynamic library at some place. Because I don’t like to put artifacts in random places, I like to use the tempdir crate, which gives you a TempDir that creates a new temporary directory with a random name when you ask for it, and removes it from your filesystem when the TempDir goes out of scope. We’ll also be using the std::process module to run rustc.

Add the following to your [dependencies] section in your Cargo.toml:

tempdir = "0.3"

Ok, let’s compile a Rust source file into a dynamic library from our Rust source code!

extern crate libloading;
extern crate tempdir;

use libloading::Library;
use std::process::Command;
use std::str::from_utf8_unchecked;
use tempdir::TempDir;

fn main() {
  let dir = TempDir::new("").unwrap(); // we’ll drop the .so here
  let target_path = dir.path().join("libhello_world.so");
  
  let compilation =
    Command::new("rustc")
    .arg("--crate-type")
    .arg("dylib")
    .arg("/tmp/hello_world.rs")
    .arg("-o")
    .arg(&target_path)
    .output()
    .unwrap();

  if compilation.status.success() {
    let lib = Library::new(&target_path).unwrap();
    let symbol = unsafe { lib.get::<extern fn ()>(b"hello_world").unwrap() };

    symbol();
  } else {
    let stderr = unsafe { from_utf8_unchecked(compilation.stderr.as_slice()) };
    eprintln!("cannot compile {}", stderr);
  }
}

Compile, run, and…

cargo run
   Compiling dyn-hello-world v0.1.0 (file:///tmp/dyn-hello-world)
    Finished dev [unoptimized + debuginfo] target(s) in 0.58 secs
     Running `target/debug/dyn-hello-world`
Hello, world!

This piece of code is the first premise of our plugin system. You can see interesting properties:

• You don’t have to re-compile dyn-hello-world to change its behavior, since it comes from a dynamic library.
• There’s no need to depend on a dynamic library directly, since we can just generate it on the fly!
• You can display error messages (the current code is a bit perfunctory but you could enhance to have nice error messages!).

However, there’s a problem. Try adding an extern crate to hello_world.rs, like, for instance:

extern crate spectra;

#[no_mangle]
pub fn hello_world() {
  println!("Hello, world!");
}

Run dyn-hello-world.

cargo run
    Finished dev [unoptimized + debuginfo] target(s) in 0.0 secs
     Running `target/debug/dyn-hello-world`
cannot compile error[E0463]: can't find crate for `spectra`
 --> /tmp/hello_world.rs:1:1
  |
1 | extern crate spectra;
  | ^^^^^^^^^^^^^^^^^^^^^ can't find crate

error: aborting due to previous error

Damn, how are we gonna solve this?

Finding dependencies!

The key is to understand how cargo deals with the dependencies you declare in your Cargo.toml’s [dependencies] section. To do this, clean your project, and recompile in verbose mode – we’re like… reverse engineering cargo build!

cargo clean -p dyn-hello-world
cargo build --verbose
       Fresh libc v0.2.36
       Fresh winapi-build v0.1.1
       Fresh winapi v0.2.8
       Fresh cc v1.0.4
       Fresh rand v0.4.2
       Fresh kernel32-sys v0.2.2
       Fresh libloading v0.5.0
       Fresh remove_dir_all v0.3.0
       Fresh tempdir v0.3.6
   Compiling dyn-hello-world v0.1.0 (file:///tmp/dyn-hello-world)
     Running `rustc --crate-name dyn_hello_world src/main.rs --crate-type bin --emit=dep-info,link -C debuginfo=2 -C metadata=e8b1e5e96f709abe -C extra-filename=-e8b1e5e96f709abe --out-dir /tmp/dyn-hello-world/target/debug/deps -C incremental=/tmp/dyn-hello-world/target/debug/incremental -L dependency=/tmp/dyn-hello-world/target/debug/deps --extern tempdir=/tmp/dyn-hello-world/target/debug/deps/libtempdir-9929bcad6dc8cc47.rlib --extern libloading=/tmp/dyn-hello-world/target/debug/deps/liblibloading-a2854ce154eb4d6f.rlib -L native=/tmp/dyn-hello-world/target/debug/build/libloading-4553b9f132aa813a/out`
    Finished dev [unoptimized + debuginfo] target(s) in 0.50 secs

We can see a few things going on here. First, there are some Fresh lines we’re not interested about. Then cargo tries to compile our application. You can see an invokation to rustc with a long list of arguments. Among them, two interest us:

• The one specifying the list of dependencies your project needs:
  • -L dependency=/tmp/dyn-hello-world/target/debug/deps.
• The ones specifying the list of crates you might use in extern crate statements.
  • --extern tempdir=/tmp/dyn-hello-world/target/debug/deps/libtempdir-9929bcad6dc8cc47.rlib
  • --extern libloading=/tmp/dyn-hello-world/target/debug/deps/liblibloading-a2854ce154eb4d6f.rlib

There’s also a -L native=… one. This is explained in the manual of rustc and corresponds to native library we must link against, like a C library, for instance.

As you can see, the path used in -L dependencie=… is pretty constant. It seems it has the form:

/path/to/project/target/<build type>/deps

However, remember that the initial intent was to write a plugin system for spectra, which is a library. We don’t know the path to the project that will be using spectra, so we must find it in a way.

Two options:

• We could use a macro (macro_rules) to automatically insert it at the calling site. However, I’m not even sure it would work (you need to find a way to have access to cargo’s internal… maybe via the env! macro?).
• We could find the path at runtime.

I chose the second option because it was pretty easy and straight-forward to implement. However, I’m not a huge fan of it – it’s pretty… non-elegant to me. Here’s the code that gives me the root path of the current project a function is defined in:

/// Try to find the project root path so that we can pick dependencies.
fn find_project_root() -> Result<PathBuf, PluginError> {
  let result =
    Command::new("cargo")
    .arg("locate-project")
    .output()
    .unwrap();

  if !result.status.success() {
    Err(PluginError::CannotFindMetaData("cannot locate root project".to_owned()))
  } else {
    let json = unsafe { from_utf8_unchecked(result.stdout.as_slice()) };
    let root =
      json.split(':').nth(1).and_then(|x| {
        if x.len() >= 3 {
          Path::new(&x[1..x.len()-3]).parent().map(|x| x.to_owned())
        } else {
          None
        }
      });

    root.ok_or_else(|| PluginError::CannotFindMetaData("cannot extract root project path from metadata".to_owned()))
  }
}

As you can see, I use the cargo locate-project feature, that gives you the root path of the project the cargo invokation is in. It supports calling it from a subdirectory, which is neat (a bit like git, actually). Most of the code is removing the JSON sugar over it. It’s pretty unsafe because if the output format of cargo locate-project changes, this code will basically break.

Pro reminder for myself: write unit tests for that piece of code. :D

I won’t show the PluginError type, it’s not important and it’s spectra current code for the feature.

Ok, we lack two things:

• The build type.
• The path to our crate – here, spectra.

Finding the build type is pretty easy: you can use the debug_assertions cfg! argument. It’s set to true on debug target and false on release. You can actually witness it works with the following oneliner:

println!("build target is debug: {}", cfg!(debug_assertions));

Ok, now, how do we find our spectra crate’s path? You saw it has a metadata glued to its name in the previous verbose output.

I didn’t find any satisfying solution. I just came up with a solution. I warn you: it’s not elegant, it’s highly unsafe, but for now, it works. I’ll try to find a better way later.

Here’s the code.

/// Find the spectra crate in the given directory.
fn find_spectra_crate(path: &Path) -> Result<PathBuf, PluginError> {
  // we open the directory we pass in as argument to the function
  if let Ok(dir) = path.read_dir() {
    // we iterate over all of its contained files to find if it has our spectra crate
    for entry in dir {
      let path = entry.unwrap().path();

      match path.file_name() {
        // we try to pattern match its name; this is a bit unsafe because it won’t support two
        // versions of libspectra… meh.
        Some(filename) if filename.to_str().unwrap().starts_with("libspectra") => {
          return Ok(path.to_owned());
        }

        _ => ()
      }
    }

    Err(PluginError::CannotFindMetaData("cannot find the spectra crate; try to build your project’s dependencies".to_owned()))
  } else {
    Err(PluginError::CannotFindMetaData("cannot find read dependencies".to_owned()))
  }
}

If you put those three functions altogether, you can now implement the rustc call without hardcoding any paths, since they all will be found at runtime and injected in the call!

A bit of hindsight. I didn’t explain that, but it’s a bit obvious: you won’t be able to use all the crates you want in your plugins, for a very simple reason: you must have them installed somewhere. In order for the find_spectra_crate to find anything, you must have spectra = "…" in the [dependencies] section of your Cargo.toml. If you want to use any crate, you need to add them in the [dependencies] and write a smarter function that parses the Cargo.toml’s [dependencies] section and add them to the rustc invokation… which is basically like re-writting a feature of cargo itself!

How do you make auto-reload again?

I didn’t speak about that, but in spectra, it’s very easy to have a resource to auto-reload if it changes on the disk. This is done via the warmy crate. You just implement the Load trait and ask for your type at a Store by providing a typed key. I won’t talk too much about warmy – if you’re interested, go read the online documentation here!

The idea is that I just created a type that wraps both libloading::Library and tempdir::TempDir. Something like this:

pub struct Plugin {
  lib: Library,
  dir: TempDir
}

I still don’t know whether I need to implement Drop or not – if the TempDir dies first, I don’t know whether the Library object is in an unstable state. If not, that means that the whole library was loaded in RAM at the end of the Library::new call, and that I don’t even need to keep the temporary directory around!

It’s just the beginning

The interface of a Plugin is not yet defined. I’m writing this blog entry on Sunday night / Monday morning, while I had that plugin experiment over the weekend. A few thoughts, though:

• My current code enables me to run my demo (graphics-ish) and edit Rust code to script it live, which is completely amazing to me!
• It should be easy to expose a safe interface by asking the plugin to expose a normal function and inject a header in the Rust code that contains an unsafe fn, calling the safe function or a trait’s method or whatever.
• You have no idea how much I’m excited about all this!

I have a lot of things to say, especially lately. I’ll be posting more thoughts and experiments of mine soon! Thanks for having read through, and as always, keep the vibes!

Note taking, task management, journaling… how?

As an experienced Software Engineer, I know that I need to multi-task and work on many different projects in parallel, with different chronicity. Some new tasks can be done only a couple of minutes after discovering / creating them, and some others will span on days, weeks if not months. That leads to having many parallel things on-going, and since I want to do my job as correctly as possible… I take notes. I take notes about pretty much everything, and I’ve been using note-taking apps more and more.

Years ago, as I discovered Doom Emacs, I also discovered org-mode. I was blown away by the features brought by Doom Emacs, and a bit puzzled with other things. For instance, the agenda (I rarely care about ETA for my personal notes; that’s something that is tracked at the company-wide level, for instance in the infamous Jira or whatever you’re using for project management in your team — and I just don’t care on my spare-time). Also, I was a bit doubtful about having notes laid as plain text (yes, org-mode formatted, but still text). What it means is that, there is no database nor state. Everytime you want to get a list of your TODO, org-mode has to go through all your .org files and generates the list for you. Back then, I found that a bit weird, but today, I have changed my mind.

Zettelkasten and other tools

The way I take notes, whatever I do, is to minimize the note files I create. For instance, at work, I have many notes about commands to run with a specific product. Instead of having a note That product, I tend to create smaller notes, more scoped, like That product - deploy, That product - how to configure, etc. That allows two important things:

• Notes are shorter, easier to read and maintain.
• Looking up for information and fuzzy searching is easier.

Yes, fuzzy searching. Whatever the tool I use, I often need to search for notes, and either I search by note names, or by content.

A couple of months ago, I wrote mind, a rewrite of mind.nvim. I designed that system to be a tree editor, in which nodes are either “document node” containing some metadata and a path to a file, or a “link node”, with metadata and URL. Those tools were very useful to me, and I’ve been using them for a long time.

However, as I moved more and more into the UNIX world with kakoune and kak-tree-sitter, I realized that I should maybe revisit the way I think about notes. For instance, what I often need is to list the on-going tasks I’m doing, and updating them. Oftentimes (if not all the time), tasks emerged from a context, like a discussion with a colleague, some notes / brainstorming with myself I’m doing / a meeting / etc. And mind, Notion — or whatever that is using an external state — are not very suited. The problem relies in interruption. You’re in the editor of your choice, and then you have to switch to another window, look for the task / notes, use some keybindings to change the task to switch it from TODO to WIP or WIP to DONE or WONTDO or whatever.

As I was working more and more with mind, I realized this kind of workflow was starting to annoy me.

Composing

You start to see the pattern with my blog posts: I enjoy the UNIX world a lot. And even though mind is perfectly fine regarding that (I was even able to capture notes by using tmux’’s display-popup with mind), I wanted to switch to something… easier, yet still UNIX.

Then I started to think again about org-mode. See, org-mode is eventually just a format specification. The Emacs mode is the reference implementation, with major modes, keybindings etc. but if Emacs can edit and do stuff to .org files, then the concept is transposable to other tools. The question is: what is the core of org-mode? The idea behind org-mode is that there is no state: the notes themselves are the state. There is no need for an external tool. Any tool can open an .org file and edit it. Yes, convenience is better to navigate .org notes, but that’s just a matter of tooling. If you want to open and edit an .org file using your favorite editor, you should be able to.

Then, for the rest, org-mode has a couple of interesting and useful concepts:

• You write notes in text files, including metadata. So tasks management is also done via .org files.
• Listing tasks, for instance, can be performed pretty easily by grepping headings. For instance, * TODO can be used at the beginning of a line to start a TODO item, that will appear in the listing. That looks arrogantly innocent, but it’s really powerful, and I’ll explain why just below.
• org-mode has the concept of a capture buffer. The idea is simple: you open it to add stuff right off your mind, and you come back to it later to triage its content. This is a fantastic concept, because very often, I need to take notes about things I have to do, but I have no structure yet. For instance, * TODO talk to Bob about service-epsilon. Later on, I might create a note Epsilon and move the TODO item into the note. Since there is no state, the TODO will still appears in my todo-list.
• Another important thing is archiving. You are encouraged to create many small notes, even for task management. At work, I currently have a note called Back from holiday Oct 2023, which contains many TODO items taken from the backlog I read on Slack (basically, stuff my colleagues did and that I want to catch up with). At some point, I will be done with all of that, so instead of keeping DONE notes around (that will show up in my DONE listing), I can just archive that note. It will not appear anymore in my fuzzy search results.

All of that is really exciting, because it makes note taking easy and seamless; it doesn’t require any state, and you can pick the tools you want to implement the workflow.

It’s not a tool; it’s a workflow

Yes, that’s what I think is the most important thing here. I do not want to use org-mode for now. I like Markdown. Even though I agree .org files are more powerful, Markdown, at least for now, is more than enough to me. However, I wanted to augment it to do things slightly similar to org-mode.

And so I did it. I created in my kakoune a couple of commands to:

• Open a capture buffer located at the same place on my disk. It defaults to ~/notes/capture.md.
• Quickly capture a line via kakoune’’s user prompt and append it to the capture buffer (super useful for todo items).
• Create a new note in ~/notes/notes/.
• Archive a note into ~/notes/archives/.
• List / fuzzy search notes.
• List / fuzzy search archives.
• Open a daily journal in ~/notes/journal/<year>/<month>/<day>.md. It’s super easy with kakoune since you have %sh{} expansion. Just use the date command you’re done!
• List daily journals.
• Open task list by:
  • TODO, WIP, DONE or WONTDO items.
  • Those items are put into a special buffers on which I have set buffer keybindings to jump to the place where the item is located. For instance, if I list TODO items, pressing <ret> on a line will open the appropriate file and move my cursor to the right line and column where the TODO item is defined.
  • Labels, like :learn: or :hard: or whatever.
• Local (buffer) keybindings to switch tasks to different status.
• A special convenience I created: Markdown files containing a special <!-- github_project: <user>/<project> -->, usually at the end of the file, allows me to press a keybinding on issue numbers (like #123) to automatically open them in my browser.
• Synchronize notes with a single keybinding, which is running some git commands to rebase / push.
• And many more similar features.

I plan on releasing the .kak file containing all of that if people are interested.

The great thing is that most of the features, like fuzzy searching, are implemented via external tools. I use fd and rg fuzzy search todo items. There is nothing to do in kakoune besides gluing everything together.

This setup has been my daily driver for months now and I really enjoy it. Being able to take notes and just slide in a - TODO I need to check that right in the middle of the file and having it appear in a listing later is a really powerful aspect of this workflow… and org-mode was doing that all along ever since. Eh, it almost makes me nostalgic of my time using the Emacs plugin!

I hope that gave me some (not so fresh) hindsight about note taking and task management (and journaling!). Have fun hacking around!

On the last weekend, I was at the Revision demoparty (Easterparty). It’s a meeting of friends and great people doing graphics, music, code and that kind of fun and pretty stuff. I didn’t release anything, but I’ve been working on a release for a while now, and I was lacking something in my engine: light shafts.

I know a few ways to do that. They almost all fall in one of the two following classes:

1. screen-space-based;
2. raymarching-based.

Both the techniques produce interesting images. However, the screen-space-based method produces bad results when you don’t look directly to the light – it may even produce nothing if the light is out of screen – and the raymarching-based is a step process that might generate artifacts and can be slow.

The idea I had is very simple. I haven’t tested it yet, but it’s planned for very soon. I’ll post images with benchmarks as soon as I have something on screen. I’m not sure it’s an unknown way to do it. I just haven’t found anything describing that yet. If you have, please leave a link in the comments below! :)

Volume extraction

The idea is the following. You need a depthmap. If you’re used to shadow mapping you already have a depthmap for your light. In order to simplify this, we’ll use the depthmap used for shadow mapping, but I guess it’s totally okay to use another depthmap – we’ll see that it could be even handier.

For each point in that depthmap is the distance – in a specific space coordinates system – of the corresponding point in world space to the position of the light. If you have the projection and the view matrices of the light, it’s easy to deproject the depthmap. What would we get if we deproject all the depthmap texels into the world space? We’d get the exact lit surfaces.

For a spot light, you can imagine the deprojected version of the depthmap as a cone of light. The “disk” of the cone will deform and shape as the lit environment. That’s the first part of the algorithm.

We have a points cloud. What happens if, for each deprojected texel – i.e. point – we draw a line to the position of the light? We get an actual lines field representing… photons paths! How amazing is that?! Furthermore, because of the depthmap sampling, if you look at the light and that an object is between you and the light, the photons paths won’t go through the obstructing object! Like the following image:

Of course, because of using raw lines, the render might be a bit blocky at first. If you know the laser trick¹ – i.e. quad-based lasers, you can apply it to our lines as well, in order to get better results. The first improvement is to disable depth test and enable additive blending.

Algorithm

In the first place, you need to generate the depthmap of the light. Then, you need to extract the volumetric shaft. You’ll need a vertex buffer for that. Allocate w*h elements, where w and h are the depthmap’s width and height. Yeah, a point per texel.

Create a shader and make a render with glDrawArrays(GL_POINTS, 0, w*h) with an attributeless vertex array object. Don’t forget to bind the depthmap for use in the shader.

In your vertex shader, use gl_VertexID to sample from the depthmap. Transform the resulting texel in world-space. You can use something like the following deprojection formula:

vec3 deproject() {
  float depth = 2. * texelFetch(depthmap, ivec2(gl_FragCoord.xy), 0).r - 1.;
  vec4 position = vec4(vv, depth, 1.);
  position = iProjView * position;
  position.xyz /= position.w;
  return position.xyz;
}

Pass that to the next stage, the geometry shader. There, build whatever kind of new primitive you want. In the first place, I’ll go for a simple line connected to the light’s position. In further implementation, I’ll go for lasers-like base shapes, like star-crossed quads.

In the fragment shader, put whatever color you want the shaft to have. You could use interpolation to reduce lighting wherever you want to create nice effects.

Don’t forget to use additive blending, as we do for lasers.

Considerations

I see two major problems. The first one is the bright aspect the shaft will have if you don’t blend correctly. Play with alpha to reduce more if the fragment is near the light’s position and make the alpha bigger when you’re far away from the light’s position. Because you’ll blend way more photons paths near the light’s position than far from it.

The second issue is the resolution of the extracted volume. For a 512x512 depthmap, you’ll get around 262k points, then 524k lines. That’s a lot for such an object. And that’s only for a simple spot light. An omnidirectional light would require six times more lines. What happens if we don’t use lines, but star-crossed quads an that we want several shafts? You see the problem.

A solution could be to sample from high mipmap level, so that you don’t use the full resolution of the shadow map. That would result in less visual appealing shafts, but I’m pretty sure it’d be still good. You could also branch a blur shader to smooth out the whole thing.

Conclusion

I’ll try to implement that as soon as possible, because I think my idea is pretty interesting compared to raymarching, which is expensive, and way better than screen-space, because the shaft will still be visible if the light goes out of screen.

Most of the things I’m going to show here in Kakoune are also doable in Helix, but Helix is not as mature as Kakoune, so it’s likely some features won’t be available at the time of writing this blog article, May 2025.

Disclaimer: as always with programming tools, use the tool you are comfortable with. This article only reflects the way I see editing; it doesn’t mean one tool is superior to another in an absolute way. It’s just a way for me to share why I think Kakoune (Helix) is better, and provide some hindsight on both editors.

Thank you: thank you @Screwtape for reviewing this article!

• 1. Assigning numbers to existing enum classes
• 2. Fixing verbose SQL insert scripts
  • Drop all selections but the primary
  • Filter selections
  • Just do it manually
• 3. Extract field names from templates
• 4. Batch process multiple files
• 5. Integrate data from external sources into your project
  • execute-keys
• Kakoune has an incredible design

🔗 1. Assigning numbers to existing enum classes

The example is this:

public enum TranslationCodeId
{
   // buttons
   BUTTON_Accept,
   BUTTON_Cancel,

   // labels
   LABEL_Actions,
   LABEL_Column,

   // messages
   MESSAGE_Confirm,
   MESSAGE_GoBack,

   // more fields...
}

Vim does it with two different actions:

• Assign 0 to all enum fields: :g/,$/v;//;norm $i = 0.
• Increment all numeric values sequentially: vi}o2/0,$<Enter>g<C-a>.

How to do that with Kakoune? The first thing is to be able to get a selection on each enum field. We can use a similar trick as what Vim does by targetting the line ending with a , with s,$. Because we assume — like in Vim — that this enum definition is the whole buffer, we can work on the whole buffer with the % key (similar to the g/ of Vim). So we simply just type %s,$<ret>.

From now, the rest is really trivial, as we have a selection on each ,. We can then simply type i = <c-r>#, which will enter = N before every column, where N will have the right value.

Some explanations: pressing control-r will allow the next key to select a register that will be pasted after all selections. Kakoune has many built-in registers, and # is the selection index register; it gives you the index of each individual selection, starting from 1, ascending.

🔗 2. Fixing verbose SQL insert scripts

This one wants us to fix the ending characters of some lines:

INSERT INTO AdminTranslationCodeText
    (AdminTranslationCodeTextId, AdminTranslationCodeId, LanguageId, Text)
    VALUES
    (NEWID(), 'BUTTON_Accept', 'it', 'Accetta'),
    (NEWID(), 'BUTTON_Accept', 'en', 'Accept'),

    (NEWID(), 'BUTTON_Cancel', 'it', 'Annulla'),
    (NEWID(), 'BUTTON_Cancel', 'en', 'Cancel'),

    (NEWID(), 'LABEL_Actions', 'it', 'Azioni'),
    (NEWID(), 'LABEL_Actions', 'en', 'Actions'),

    (NEWID(), 'LABEL_Column', 'it', 'Colonna');
    (NEWID(), 'LABEL_Column', 'en', 'Column'),

    (NEWID(), 'MESSAGE_Confirm', 'it', 'Conferma'),
    (NEWID(), 'MESSAGE_Confirm', 'en', 'Confirm');

    (NEWID(), 'MESSAGE_GoBack', 'it', 'Torna indietro'),
    (NEWID(), 'MESSAGE_GoBack', 'en', 'Go back'),
GO

The Vim command: :/VALUES$/+,/^GO$/-2s/;$/,/ | /^GO$/-s/,$/;/. No need to comment on it; it’s barely impossible to tell what it does in a quick glance; we have to decode it, and none of it will be executed before you actually run the whole thing. It does a lot of things, and you can see a bunch of text that has to be typed there, like VALUES GO, negative relative lines, etc.

The Kakoune version here is different, and, as expected, more interactive. The first thing we want to do, as always, is to get a cursor on the lines starting with a NEWID, and put each cursor at the end of each line. We still use % to select the whole buffer, sNEW<ret> to select all NEW, and gl to move all selections on the last character, which will be a , or a ;, depending on the line:

Then, simply type r, to replace all ending symbols with ,:

We now need to clean up the last line, since it requires a ;. We have a couple solutions, that I will explain.

🔗 Drop all selections but the primary

I like this one a lot because it’s very interactive and intuitive. Kakoune has the concept of a primary selection and secondary selections. The primary selection is treated specifically from others, as it’s used to perform some specific actions. For instance, imagine you select a function and all of its occurrences. Depending on where your primary selection is, the context action you might get from e.g. LSP will be different.

Kakoune allows you to cycle the primary selection with secondary ones with ( and ). Usually, highlighting will tell which selection is the primary — in my screenshot above, the primary is pink-like; the secondary ones are white-like. If you have typed the keys as mentioned below, you should notice that the primary selection is actually on the last line, which is already perfect! If it’s not, you can use ( and/or ) to adjust it; we want the last line to have the primary selection.

Then, just press ,. That key will drop all selections but the primary one, leaving a single selection on the last ,. You then know what to do: r;. And we’re done!

🔗 Filter selections

Another thing that I use from time to time is to filter (keep) or discard (keep non-matching) selections. This is useful to filter selections based off the already selected, disjoint selections. We use the <a-k> key chord for that. In our case, we can see that the last line has the word back in it, which is unique to our selections. We can select whole lines with x:

And then <a-k>back<ret> to select that line:

Note: <a-k> (keep matching) is different from s (select) in the sense that it will filter selections, while s will select whatever you type, discarding already existing selections. Try them out to experiment!

Then simply do the previous thing: glr;.

🔗 Just do it manually

Something that I think is important to notice is that, in most cases, you will most likely just move your cursor there and modify the character manually. Sometimes, it’s much faster to drop all selections and manually edit a bunch of characters instead of trying to be smart and spending mental effort for no real benefit.

For this reason, I think the first option is clearly the way-to-go solution: just press ,, and the cursor will already be at the right place. Kakouine encourages visual exploration and interactivity; stop thinking too much about how to edit stuff in advance, and do it gradually, incrementally!

🔗 3. Extract field names from templates

This one is absolutely crazy in Vim. Based on this text:

{{#if UserDetails.FirstName}}Nome: {{UserDetails.FirstName}}{{/if}}
{{#if UserDetails.LastName}}Cognome: {{UserDetails.LastName}}{{/if}}
{{#if CallbackDetails.CallbackTimeStamp}}Data e orario indicati dal cliente per la chiamata: {{dateFormat CallbackDetails.CallbackTimeStamp format="d"}}.{{dateFormat CallbackDetails.CallbackTimeStamp format="M"}}.{{dateFormat CallbackDetails.CallbackTimeStamp format="Y"}} {{dateFormat CallbackDetails.CallbackTimeStamp format="H"}}:{{dateFormat CallbackDetails.CallbackTimeStamp format="m"}}{{/if}}
{{#if UserMotivation.ReasonSell}}Motivo della vendita: {{UserMotivation.ReasonSell}}{{/if}}

Informazioni sull'immobile:

{{#if PropertyDetails.ObjectType}}Categoria: {{PropertyDetails.ObjectType}}{{/if}}
{{#if PropertyDetails.ObjectSubType}}Tipo di oggetto: {{PropertyDetails.ObjectSubType}}{{/if}}
{{#if PropertyDetails.Street}}Via: {{PropertyDetails.Street}} {{#if PropertyDetails.Number}}{{PropertyDetails.Number}}{{/if}}{{/if}}
{{#if PropertyDetails.Zipcode}}Luogo: {{PropertyDetails.Zipcode}}{{/if}} {{#if PropertyDetails.City}}{{PropertyDetails.City}}{{/if}}

Valutazione immobiliare: 

{{#if ValuationDetails.EstimatedMarketValue}}Prezzo di mercato stimato: {{ValuationDetails.EstimatedMarketValue}}{{/if}}
{{#if ValuationDetails.MinimumPrice}}Prezzo minimo:  {{ValuationDetails.MinimumPrice}}{{/if}}
{{#if ValuationDetails.MaximumPrice}}Prezzo massimo:  {{ValuationDetails.MaximumPrice}}{{/if}}

We want to extract all field names from that Handlebars template snippet. To do that in Vim, you need to store the names in a register, remove duplicate lines, and format the result. Let’s see how to do that in Kakoune.

We start as always by selecting the whole buffer and follow a similar idea as in Vim by selecting the if, but we will use the # as well — I think it’s more correct than using the trailing whitespace: %s#if<ret>:

That gives us a selection on each #if. We can then press w twice to fully select all field names:

Yank them with y, and drop your selections with ,. As with Vim, move your cursor in an empty location, and paste your selections with <a-p>.

We cannot use p, because p pastes with a 1:1 mapping between selections. What it means is that p is selection-aware, so if you have two selections containing foo and bar, yank them, then move both selections to somewhere else and press p, foo will be pasted to wherever the first selection is, and bar will be pasted on the other place. This is incredibly powerful, but here, since we have dropped all selections, pasting with p will only paste the content of the primary selection. <a-p> will, as the doc says, paste all the selections one after the other, and will put a selection after each of them, which is exactly what we want.

Once pasted, you will notice they are all on the same line. Simply press i<ret><esc>x to put them ont different lines and ensure you have a training newline on each of them:

In order to sort them, we need to end up with a single selection containing them all. We could do this by dropping all selections with ,, then selecting the whole paragraph with <a-i>p, but Kakoune has a key for this exact situation: <a-_>, which merges continuous selections, across lines:

Then simply press |sort -u<ret> to sort them via the sort program:

Now, you can get a selection back on each line by pressing <a-s>, which simply adds a cursor at the end of each line in the current selections — note the plural; if you had several disjoint selections all spanning on several lines, it would apply this logic to all of them, which is really powerful:

Press ; to reduce all selections to a single character (anchor = cursor):

And then simply go with r, to format the whole thing as a single comma-separated line:

You can drop selections to keep the primary (last one) and use the same trick as before to remove the trailing ,: ,d.

It might sound like a lot is going on, but it actually is. The great thing is that everything is interactive, and you see what you are doing. Meanwhile, most of that in Vim occurs without you noticing everything, requiring to type a lot of normal commands and crazy substitutions. A single selection mistake will require you to go back from the beginning.

🔗 4. Batch process multiple files

This one is pretty fun too. Given a git status output, we want to execute a git command on each line.

Note: I would have actually run a single git update-index command with all the paths as arguments instead, but that defeats the exercise, so let’s stick to just calling the git command for each path instead.

The output is:

On branch main
Your branch is up to date with 'origin/main'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
   modified:   ExcelGenerator/App.Debug.config
   modified:   ExcelGenerator/App.config
   modified:   Core/ConfigBase.cs
   modified:   Infrastructure/Security/Password.cs
   modified:   Tools.AgentExport/App.Debug.config
   modified:   Tools.AgentExport/App.config
   modified:   Tools.CheckLeadsForConfirmedEmail/App.Debug.config
   modified:   Tools.CheckLeadsForConfirmedEmail/App.config
   modified:   Tools.EmailSender/App.Debug.config
   modified:   Tools.EmailSender/App.config
   modified:   Tools.LeadChecker/App.Debug.config
   modified:   Tools.LeadChecker/App.config
   modified:   Tools.LeadImport/App.Debug.config
   modified:   Tools.LeadImport/App.config
   modified:   Tools.LeadManager/App.Debug.config
   modified:   Tools.LeadManager/App.config
   modified:   Tools.LeadReminder/App.Debug.config
   modified:   Tools.LeadReminder/App.config
   modified:   Tools.SmsSender/App.Debug.config
   modified:   Tools.SmsSender/App.config
   modified:   Web.Backend/Web.Backend.csproj
   modified:   Web.Backend/Web.Debug.config
   modified:   Web.Backend/Web.config

In Vim, we need to run a bunch of commands: bring the output of git status in the buffer, remove lines not containing a modified path, map all lines to git commands, and execute all lines as bash commands.

Let’s see how I would do that in Kakoune.

First, as a disclaimer, I need to mention that Kakoune has a support for git via its standard distribution and the :git command. It would be cheating, so I will not use it.

From an empty buffer, let’s bring in the output of git status by simply typing !git status — let’s change the output to match whatever the exercise has:

Then, it’s pretty simple: %smod<ret>wwlGl to select all paths:

And it’s almost done. Press <a-|> to pipe each selection as standard input of the program of your choice, and use xargs git update-index --skip-worktree <ret>. This will run that program with each selection as argument.

🔗 5. Integrate data from external sources into your project

Another crazy one from Vim: from a text dump of some random data (IP, masks, subnets), we want to create some C# code that will whitelist those IP addresses. The goal is to format around the data, and it’s really bad for Vim, because its regex thing is absolutely disgusting, while Kakoune was designed specifically with that kind of usecase in mind.

Just so you see it, the Vim way requires to format the data, extract texts in many registers, map the data to actual code, and other things, which results in those lines:

`[v`]:'<,'>s/^.\(\d*\).\{-}\(255.*\)\t.*/'\1': '\2',/ | '<,'>j | s/.*/{ & }/
/new Re<Enter>"ayf";;"byf";;"cy$
`[v`]:'<,'>g/^$/d
gv:'<,'>s/\(.*\)\/\(.*\)$/\='<C-r>a' . submatch(1) . '<C-r>b' . <C-r>m[submatch(2)] .  '<C-r>c'

Now let’s see how to do that in Kakoune. We start with the same buffer as Vim:

I think the best approach here is to use a macro — yes! Kakoune has them! — and a wonderful feature of Kakoune regarding selections: executing arbitrary keys on each of the selection. First, let’s paste one IP address:

We can start the macro by pressing the Q key — the macro will be registered to the @ register by default. Start by moving at the end of the current line with gl:

Then, we are going to do a reverse character search, yank the subnet mask index, and put a mark to save the current selection with <a-t>/yZ:

Z here is a feature that is not available on Helix; you can probably use the jump list’s <c-s> feature in Helix there, but I haven’t tried.

Now, the default " register has the mask index. We can just do a regular search of /N with N the mask index with the //<c-r>"\t<ret> keys:

We can move the cursor to the right to select the actual mask value with wwl?\tH:

? is the extend version of /. We need to use this because the values are separated by \t. If it was a regular space, we could have used t instead.

Now, let’s yank that value in the m register to be sure we won’t lose it with "my:

Press z. It will restore your selection back to the subnet index in the IP address:

From this place, we can start formatting the line. Let’s copy the IP address into the i register, and remove the rest. Go to the beginning of the line with gi, select up to the / with t/, and "iy to yank the IP address into the i register:

Replace the whole line with x_:

Press r and start writing your code. Whenever you need to put the IP address, use <c-r>i. To put the mask, use <c-r>m:

And now that this is done, finish recording with Q. Now, for every more lines we have an IP address on, just press q to replace it with the code:

However, there is a better feature of Kakoune we can use here.

🔗 execute-keys

execute-keys — which we often use abbreviated exec — allows to type keys as if they were typed by the user in a more controlled way. For instance, this command accepts the -itersel flag, which runs the command for every selection. Along with the -with-maps (required for macros, I think?), we can select all IP addresses, spawn a selection on each of them with <a-s> as seen previously:

And then press :exec -itersel -with-maps q:

Press <ret>. Taaadaaa:

🔗 Kakoune has an incredible design

The design of Kakoune shines here. You can see that you can do everything by incrementally and visually progress through your mental model, instead of having to think of everything in advance. The fact that you can use regular commands such as w, y, d, f etc. on many selections at once creates such a friendly environment to implement very complex workflows, without needing to use the crazy regex and substitute commands from Vim.

I hope that blog article provided a good hindsight on how we can do things in Kakoune vs. Vim, and why many people — me included — prefer the modal editing model of Kakoune.

Keep the vibes!

In this blog entry, I want to share something a bit different: instead of presenting my experiences with all those editors, I want to take some hindsight and analyze a bit more the technology and what it seriously means in terms of features and productivity. Obviously, I will reference other editors I have used a lot (I use IntelliJ a lot at work, and I am used to VS Code and Atom as well).

Current status as of March 2021: I am lost

Yeah, you read that title right. I am lost. I have been a Vim / Neovim user for something like 12 years now. I am used to the ecosystem, I have installed hundreds of plugins in the past decade, tried so many things, discovered pearls that I still use today (hello context.vim), I have been using a workflow based on tmux, Neovim and my shell that has proven very powerful and yet to be fully replaced. But — and I think it is important to make that note — I did not start coding with Vim. Today, I am 29. I started coding when I was 11, using a tool called Code::Blocks. Rapidly, I switched to Emacs – by the time, Doom Emacs did not even exist, and spent several years using it. Later, when I was between 16 and 17, I switched to Vim — I guess the modal idea got me intrigued and I felt in love with it. In 2014, I was told that Vim was getting forked for the reasons we all know and I decided to switch to the fork, called Neovim.

Since then, I have been sticking to Neovim but I do not partake in those “editor wars”. “VS Code sucks”, “how do I even exit Vim?”, “Emacs is bloated!”. I have my personal opinions on the technologies (i.e. I dislike very much the front-end ecosystem / electron, even though I think the front-end idea — i.e. running code in a browser — is an interesting idea; I just think the current technology for doing so is really bad), so obviously I will not pick something such as Atom / VS Code, but I cannot say they suck. I have tried them. I see why people like them and it would be very dishonest to say those editors suck. They really do not. Same thing applies to the products by JetBrains. In the past year, I did not understand why some people would pay money for an editor. I have to use IntelliJ at work and even though I would not pay for an editor, I start to grasp why people would actually want to pay for one. Not all people have the patience and time to tweak an editor to their taste, such as with Vim / Neovim / Emacs. For those people, editors such as VS Code, IntelliJ, Atom and others are really just superior.

Just take LSP. Configuring LSP in VS Code is a click on an install button, depending on your language. That is all. You get all the good defaults and behaviors and you can already start enjoying coding with LSP. Doing the same in an editor like Neovim is a very different experience. I am not saying it is bad; it is different. If you are, like me, someone who enjoys spending time tweaking their tools, then you should not care too much. I have a colleague at work who gave me an argument that I find very interesting: he basically wants an editor he does not have to configure, so that we knows how to use it everywhere, on any machine of any colleague. That is not a point I would make, but for someone who cares about that topic, it is a solid argument in favor of tools such as IntelliJ or VS Code.

In the previous company I was working for, I worked with @gagbo, who is an active Doom Emacs contributor, and I was surprised by the editor he was using on his laptop. He mentioned Doom and on the night home, I had a look at it, and I was really blown away. Doom Emacs (Doom for short) was a revelation to me, because it is using emacs as a productivy platform more than a simple editor. @hlissner, Doom’s author, and the Doom community, have been gathering the best of emacs and vim into a single configuration distribution. The result is so interesting and so powerful that I have been doing back and forth between Doom and Neovim. More on that below.

Lately, I have been more present in the Neovim community, contributing mostly via a plugin that I have been making: hop.nvim. That plugin is a complete rewrite from scratch of EasyMotion (I have not even looked at how they do it on their side) I made by using the Neovim Lua API directly. Obviously, it provides a much better experience than EasyMotion in Neovim, simply because it uses features such as virtual text and extended marks and does not touch the buffer, allowing to benefit from Neovim-0.5 crunchy features, such as the native LSP client and treesitter parsers. I also made a bunch of PRs to Neovim core, reading a bit on the C code, etc.

All that to say that… given the experience that I have with Neovim and Doom… I have no idea what to think about “editing” anymore. Neovim is growing and lots of interesting ideas are landing / will land in the next release. I do not really know whether people will stick around Hop, but I think it has found its place — at least, I spotted a missing piece and provided it. I am not a fan of Lua but I have to admit that it attracts more people doing some awesome things. Among the things that I think are going into the right direction:

• The native LSP client.
• The native treesitter implementation.
• Nvim Tree, a file browser written in Lua. Similar to netrw but better and much (much) faster.
• Telescope, a fuzzy finder, a bit similar to Ivy / Helm from the emacs realm. I have issues with it right now because I am used to fzf-vim and all the current sorting / picking algorithms are not up to the user experience of fzf to me. When I compare it to Ivy, it is roughly the same: Ivy’s algorithm is just better to me (it has the same refinement search mechanism as fzf’s, and it’s much faster than Telescope, especially Swiper). I think it will change in the near future as algorithms in Telescope are improved and support is added for (I hope!) fzf, but for now, the bad performances of telescope, the choice of making it pretty before focusing on the algorithms first makes it not really usable on some code bases. However, I do think it is going into the right direction and I will be looking forward it in a few weeks / months, giving it time to mature a bit and get better defaults / algorithms.
• Neogit, an implementation of the best Git client ever, magit, also from the emacs realm. I think it will take them months before it reaches the same maturity and power as magit, but I think it is cruising great.
• Octo, a GitHub-based workflow plugin allowing to list, create, open, add, remove, etc. lots of GitHub objects directly from within a Neovim buffer, streaming changes on buffer exit / save. Similar, again, to magit’s Forge, from the emacs world, even if Forge is a bit more powerful as it abstracts away the concept of “remote”. However, the edit workflow of Octo seems a bit more stable right now, so kudos!
• Probably a ton of other plugins I forgot to mention.

So what is wrong?

My experience with Doom Emacs really changed the way I look at software, and more specifically at “productivity tools.” See, my current workflow is based on tmux, Neovim and my shell, which is zsh (vanilla, I completely ditched Oh My Zsh as I find it useless and I uninstalled starship.rs a few weeks ago because I realized I hated the package manager version annotations on my prompt, that I actually never needed). For people wondering, we have spent so much time trying to install cool things that we forgot how powerful raw and vanilla zsh is. This is my current prompt:

At work, I need a bit more information (i.e. Kubernetes (k8s) namespace and context / cluster I am currently connected to and AWS), but as those information are not path-dependent, I put them in my tmux statusline instead.

The thing is… I just realized by using Doom Emacs that I do not need all of these anymore. Doom Emacs provides an environment that is very similar to my tmux + neovim + shell combo. It provides even more, depending on the packages you install. And the thing is: for each tool, emacs have more opportunities, because it is not limited by a terminal implementation (i.e. cell-grid / line based application). So for instance, you can have a k8s / AWS / docker client inside Emacs that is likely to be much much better than the one you will use in CLI (I honestly dislike kubectl a lot, it has so many switches and grepping through them by piping with rg or using some switches such as -l feels weird). You can easily see a buffer containing the list of pods / containers / whatever, that you should be able to work with the tool you already use (i.e. Ivy for instance), interact dynamically with (instead of having to type one line after another to make effects). It is even more true when you consider the actual shell: you do not need zsh anymore when using Emacs, because it has its eshell implementation that is both super weird and super nice. You want to script something for your shell? Just write ELisp. You do not have to use that horrible bash script language, and you will benefit from the whole ELisp ecosystem. Completion for commands in eshell is basically the same as completing a regular buffer, which is weird at first but actually makes a lot of sense. And you have dozens of other examples. Org-mode is one of the best plugin of Emacs, allowing to take notes, make wikis, handle tasks and calendars, in a way that feels very seamless with the rest of the Emacs ecosystem. I know some people will tell me to use another tool (I do, my current tool for that is toodoux, a tool I made, similar to task-warrior). And of course, magit, which is by far my favorite Emacs plugin. As others say it, it is hard to explain why magit is so much better than the git CLI or VS Code’s integration or basically any other editor. It is just made very well, with pretty much anything actionable from the UI, with amazing workflows and even actions you do not have easily with the CLI version.

What is an editor in 2021, anymore?

Today, when I look at Doom, at Neovim, at what people are trying to do in Lua with Neovim, I feel like I need to stop trying to implement / follow what others are doing and think a bit about whether we are doing the right thing. Neovim, currently, is going towards a very exciting direction with all the effort regarding Lua, LSP, treesitter and community plugins. But I think the problem is bigger than Neovim. And I think I know that because of how I can use emacs with Doom. I think I come back to Doom so often because they understand well what it means to create a productivity platform more than an editor. It does not mean the editor is any less good, au contraire. In Doom, there is a default package that gets installed if you ask for the project module. That package is projectile, a wonderful package that provides with you project management from within emacs. If you use emacs in daemon mode, it means that you can get the same workflow as with tmux, Neovim and a shell, but with a native implementation of this project / workspace workflow, in a much much more powerful way (because it will obviously plug in things like Ivy, allowing you to do fuzzy searching of projects, etc.). The overall experience by having a consistent platform is something I had completely ignored for years and now that Doom hit me, it hit me hard. I do not know any other platform like emacs, besides composable environments such as tmux + Neovim + shell. Yes, you have VS Code / Atom / IntelliJ / etc. that will have tons of plugins to get you terminals and other kinds of integrations, but for having used them, it is clearly not as powerful as what Doom does — and it is most of the time based on electron; big no. To sum up, Doom is a bit my VS Code: it provides this amazing productivity platform, but it is fully native, modal-centric and highly composable / customizable.

Composability is really important to me, but I have to face reality: there is no way to make a workflow such as tmux + Neovim as snappy and fast as projectile. tmux basically reimplements a terminal inside a terminal, which has a lot of implications (copy-pasting with the mouse in tmux is a challenge even Chuck Norris does not want to take). And once you turn the computer off, tmux loses everything. You have plugins, such as tmux-resurrect, that will allow to save and restore the tmux layout (sessions / windows / panes), but you will obviously lose the contents of each panes. With projectile, because it is a native emacs thing, you will not lose anything.

This whole idea has been obsessing me for months. The concept of a “productivity platform.” People laugh at emacs as being bloated but… it basically implements their shell, tmux and editor in a more elegant way — and more. Why would one laugh at that? Especially when it allows more interesting kinds of interaction.

So I have been wondering: what is a productivity platform, today, in 2021? Is it this workflow I have been using since I am 16, with tmux and Vim / Neovim? A friend on IRC told me lately that if I have been using this workflow (as others do) for so long… it has proven being good so why would I want to replace it? Even though this is a valid point, I do think this is missing the point: ask some workers to dig a hole for planting trees, they will start digging the ground with their bare hands. Later, give them a shovel and they will accept it as a better tool and will enjoy it. But does it mean it is the best tool for planting trees? What if a better tool existed? This is basically the reason why I never accept something as an ultimate solution: we can always enhance our tools. And having seen what Doom Emacs do, I do think that I have found new ideas. Not necessarily a better tool, but at least, I think Neovim needs to evolve.

When I think about Neovim, I try to focus on what makes it this tool I love: modal centric, snappy / performance, integrates well with tmux, and that is pretty much all. The integration part with tmux is needed because I have to use the TUI, which is also something that I start not liking anymore. I know since Doom that we can get rid of TUIs and make great GUIs. So what I think I want to go to with Neovim from now on is this:

• Contribute to a GUI or even make my own. I have glanced at all the current solutions and none are really exciting to me. People have basically been making window decorations around TUI buffers, which I truly do not like. Some exceptions that I might invest time in:
  • goneovim, which seems to have the more advanced “GUI” externalized features.
  • uivonim, but I will not contribute nor use it because, heck, electron.
• Contribute to neovim-ui and see where I can help with abstractions.

Something around one year ago, I chatted with core contributors, telling them that something that puzzles me a bit with the current Neovim situation is that there is no unified / consistent UI experience. Most plugins will write their own UI thing, obviously often based on TUI hacks, such as using characters as ─, ╮, ├, etc. The current situation with plenary.nvim is another interesting topic, because I think it is not the answer to my problem. Yes, it will provide an abstraction for people needing to create bordered windows / pop-ups but… the GUI really does not care about that. Technically, Neovim already makes the distinction between “buffers” and “pop-ups” and “floats”, so GUIs are free to render them the way they want, but… what if developers start using APIs, such as plenary, where they pass border arguments? How does that make sense? If you are writing a plugin that needs to list things and requires a fuzzy interaction with the user, you should not depend on something that expects anything from the UI implementators. And for this, I think Neovim is lacking abstractions. Obviously, we can discuss that kind of problems for other topics:

• Displaying messages.
• Displaying notifications (I wish Neovim used the D-Bus protocol or similar on Linux for instance).
• Fully customizable sign columns and “franges.”
• Etc. etc.

I guess some of those items are already addressed, but I have yet to see a Neovim GUI going fully “native without TUI aspects.” Maybe it already exists and I need to search again. I want to invest time in things like goneovim and some other implementations, and I really do want to make my own (maybe with Gtk or even using my own 2D/3D graphics Rust stack).

Next obsession: CLI, TUI and GUI

CLI stands for Command Line Interface; TUI for Terminal User Interface and GUI for Graphics User Interface.

Among all the things that Doom changed in my mind, there is this concept of TUI. I have been a very big user (power user?) of the CLI. Pretty much anything that I do on a computer is via the CLI. For more advanced usage, I tend to use a TUI. I only use GUIs if I have to, such as firefox, blender or a video game. However, guess what: Doom hit hard here too, because I am not using the TUI version: I am using the GUI one. And do not get it twisted, I am not using my mouse, and this is a very important and interesting point about CLI / TUI / GUI.

When talking about this CLI / TUI vs. GUI thing around me, people tend to agree, weirdly, on some misconceptions:

• CLIs and TUIs, because they are mostly implemented as terminal applications, would not rely (for most) on the mouse and then implement super good user experiences (UX): good shortcuts, good navigation through the application, easy completion, great composability, etc. The UI, on the other side, would be rather poorer, because of the limitations of the terminals: no images, only icons, which most of the time require patched fonts to render icons; weird hacks based on unicode characters to mimic UIs like borders, splits, buttons, inputs, etc.
• GUIs would rely more on the mouse and then focus less on the UX, making them harder to use for people on the keyboard, with shortcuts most of the time not that good, preventing fluid and seamless navigation on the keyboard; poor composability with other applications, etc. However, they would have much better UI, because of vector graphics images, proper UI elements such as real inputs, buttons, border, splits, different font families / sizes, etc.

These misconceptions are interesting because, as misconceptions, they are not real. You can find some terrible CLIs (i.e. npm is such a bad CLI — I had a big issue at work a few years ago when running not on purpose a command such as npm i --user, thinking it would install it in user space while it just completely ignored the unknown --user flag and installed the package globally), and you can find terrible GUIs too. However, and this is where I want to focus, GUIs are not necessarily mouse-focused, and since we can implement nice UX in TUI… we should be able to do the same in GUIs, right?

Aaaaaaaand… enter Doom, again. I am using the GUI version of emacs — on Archlinux, I am using this patched version which is a basically gccemacs, an emacs that recompiles all the Lisp as C! It obviously can accept mouse inputs so that you can click on things if you want to but… the most important still applies: you can use it exactly as you would use it in a terminal. However, because it is a GUI, you get access to much more possibilities:

• Proper “pixel scrolling.”
• Use several fonts, of different sizes, for things like Markdowns, note taking, etc.
• Proper pop-up windows.
• Pixel-perfect annotations, marks, hints and signs in your buffer.
• Images, rasterized or vector graphics.
• One of the most important part to me: proper UI elements! No more hacks!
• Even though it is not implemented yet, since we can have images, and especially vector graphics, we should be able to render git branches with pretty links and not the /|\* ASCII art mimics we see in all CLIs / TUIs.
• Etc. etc.

Today, I do think that the right direction is to allow developers to build platforms around the concept of “graphics toolkits”, and not “terminal applications.” Even for CLIs. Think about git branch. My current git branch workflow, in CLI, does not use git branch only: I have an alias for each Git commands that require a branch. For instance, if I want to switch to another branch, this is what my gs alias looks like when invoked:

This is basically a fuzzy-finder version of git switch, using fzf – if you want that script, have a look at these ones. The same feature is doable with Doom / Neovim by using the right plugin, and I have to admit that I prefer doing that in magit as it feels more “well integrated.” Of course, you will always want to have a way to compose applications, and for that, CLIs are great. Something such as kubectl get pods -l service=my-service | rg whatever. However, I am trying to challenge my own conception of “program composability” because I do think we can implement this list k8s pods -> filter them inside an UI such as the ones in Doom Emacs, without having to write a specific tool. And we can actually do it in CLI / TUI as I just showed with my gs alias, so… why not providing that kind of power natively on the platform?

Final obsession: configuration ≠ scripting

And now one of the big issue I have with pretty much any editor today (but more with Neovim as I am actively contributing to it): if you want to configure your editor, you have to code your configuration and I highly dislike that. It is possible, indeed, to reduce the quantity of code to write, but you will eventually have to get your hands dirty and write some Lua (eeew). And I have a big issue with this.

See, configuration should be stupid and limiting. It should be key = value, period. If you need a more structured way to configure things, just add sections / objects like what you can do with TOML or JSON, but that is all. Remember that a configuration file might be read by a human, and that you must do everything possible to make it super clear what the configuration state is. Configuring should be:

• Not Turing-complete. You should not have access to logical constructs, such as conditional statements / jump statements or anything that would turn your configuration language Turing-complete.
• Imports / includes should not be possible. Having to import things from a configuration file makes it harder for the application to read from it and it makes it much harder for people to know what values are currently set.
• If you really want to have “scripting” over your configuration files, use some proper languages made exactly for this purpose, such as Dhall, a template engine or something similar. Leave the choice to the user to decide whether they want to configure, or code.

There is a fundamental difference between having the power to do something and explicitly refusing that power and lacking that power. I think that configuration can be done via code, but it does not mean it should. The reason for that is mainly for cognitive complexity reasons. If you are the author of your application, you will very likely know how to script it. You will know all the quirks and all the little things to know that you will, obviously, have well documented (right?!). Your users will not have this kind of knowledge. I think it is wrong to use Lua to configure a software as it will force people to get into the documentation of the Lua API, while all they should have is a list of configuration options they can set, what setting does what and what are the possible values, and that is all. Providing more power leads to inconsistencies, broken setup and something that is pretty hard to fix without proper semantic versioning (and an actual package manager enforcing it): a slowly rotting configuration.

Rotting configuration happens when you start piling up lots of “configuration code”, using different APIs, that will eventually get deprecated or not even used anymore. If you are lucky, the application will just tell you this and that API calls are deprecated and you should switch to something else. If you are not, you will get some random crashes and failures in your applications and you will have to debug code. For configuration. And this happens a lot with Neovim.

See, blurring the line between configuration and scripting is easy, especially if you use the same language for both. Scripting is enhancing the power of an application, such as an editor or a video game, by providing new behaviors. Those behaviors (should them be native to the applications or new ones provided by plugins) are configured. If you start having to script the configuration side, you lose the distinction between both.

At work, we are using k8s a lot. In order to configure our services, we have to deploy configuration massively everywhere, and we need to generate that configuration. But see, there is a difference between “the configuration” and “generating the configuration.” The “generate part” is basically a template language. This is code, because it relies on the datacenter / cluster / environment / feature flags / etc. we want to deploy in / with. However, we do not configure the software in the template code. We pass fragments of configuration to the template engine, which generates the final configuration. And those fragments are basically exactly what I described above: key-value objects, sometimes nested to implement more complex configuration setup, but nothing else. It is basically YAML. If you do not need the complexity of the datacenter, cluster or environment, you can just run your application with a configuration file that has no code in it, just a bunch of key-value pairs.

The way Neovim treats configuration right now is via a convention, well-accepted among the plugin developers (and since I am now officially a Neovim plugin developer with hop.nvim, I obviously follow it as well). That convention is the setup function plugins expose to pass configuration (via an object often called opts for options). So people have to know whether that convention is used, they have to call that function (i.e. it is a regular Lua function call) and they have to dig in the Lua API. What I think I want to try (I think I am going to work on a plugin for that) is a way to have a central “configuration store” so that users do not have to go through this setup thing anymore — which, for the record, requires you to do something like require'the-plugin'.setup { … }, for every plugin you want to use. The way I see things would be more like this:

-- In your init.lua, for instance

require'config-store'.setup {
  telescope = {
    -- options used by telescope
  },
  hop = {
    -- options used by hop
  },
  neogit = {
    -- options used by neogit
  },
  -- etc. etc.
}

require'telescope'
require'hop'
require'neogit'

Then, each plugin, in their init.lua, could do something like this:

-- Assuming this is the plugin for hop
local config = require'config-store'.hop or require'hop.config'.defaults

This way of doing seems much more natural to me, as it limits the amount of “coding” people have to do to configure their packages, and it is not really different for package maintainers. Instead of exposing a setup function, they can just get the configuration via this config-store thing. I think I will try to experiment with this idea. Obviously, it requires plugins to adopt this config-store thing, and I do think that if people think it is an interesting idea, this should be a vanilla Lua API and not a random plugin, so that we know this config-store object is there, whatever plugins the user have installed. As discussed above with how we do that at work, I do not want to prevent people from coding / scripting their editors. But I think a clear distinction between scripting and configuring is important.

In the end

So those were my (lost) thoughts about the current situation. I might do a follow-up later in the year to see how things have evolved to me. Today, I stick around Neovim in tmux, using various scripts around fzf to make my shell a bit more interesting to use. I have to admit that I often open emacs to do three / four commits. The experience is just consistent. Navigating in projectile, Ivy and magit is unbeaten to me. The simple interactions with dired (to navigate directories a bit like with a file browser) and edit their contents is something I truly seek everywhere else, even in my shell. I recently did a quick benchmarked of Avy, the emacs implementation of EasyMotion, versus hop.nvim, and Hop is really much faster, so I do not really know what to think (I guess I should be proud of what I made 🙂). I think that Doom currently has this level of maturity that Neovim will take years to achieve, not because of talent, but because of the fact Neovim is an editor and has not been designed as being a productivity platform, while emacs was.

Keep the vibes!

• What is a super trait?
• The weird stuff
• A small disgression to Haskell land

What is a super trait?

The weird situation is about super traits. A super trait is a trait that must be implemented for another trait to be usable, because it’s relied on. Traits can be thought of as constraints, so a super trait is a bit like a dependency when implementing a trait, and an implication when using a trait on which a super trait is declared. A trait can have zero to several super traits (added with the + operator). For instance, imagine you have a trait Alive:

trait Alive {
  fn get_health(&self) -> Health;
}

Now imagine you want a trait to move someone or something around:

trait Move: Alive {
  fn go(&mut self, direction: Direction);
}

Here, you can see that:

• Move requires to implement Alive, because it’s a super trait. It’s a dependency.
• Because Move requires Alive, Alive is implied when you use Move. Indeed, that would be redundant to annotate a type T: Move + Alive, because an instance (implementor) for Move cannot exist without Alive to be implemented as well.

So now that we understand what super traits are, let’s get to the weird stuff.

The weird stuff

When you implement a trait which has a super trait, do you think that:

1. Your implementation is valid when the super trait is implemented? After all, you could simply assume it is implemented. If it’s not, instances of your trait won’t be pickable.
2. Or your implementation requires the super trait to be implemented?

The distinction is important. (1.) doesn’t require rustc to prove when implementing a trait that the super trait is implemented. That will be required when using the trait. With (2.), rustc will have to prove that the super trait is, first, implemented, before even considering the implementation of the trait you’re making.

Rust currently uses (2.). If you impl Move for Foo, impl Alive for Foo must be in scope for that implementor to be possible.

But… it would be interesting to actually have (1.). Imagine that you want to implement Move for something a bit complex, like Creature<T>, but not all Creature<T> are alive. Only a subset of them, and you can’t tell exactly when — i.e. you just cannot assume anything about T. So what are you going to write?

impl<T> Move for Creature<T> {
  fn go(&mut self, direction: Direction) {
    // …
  }
}

This code will not compile, because you haven’t implemented Alive for Creature<T>. Remember that the trait solver requires to prove super traits. However, and this is where all the interesting / weird stuff happens:

impl<T> Move for Creature<T> where Self: Alive {
  fn go(&mut self, direction: Direction) {
    // …
  }
}

This compiles. It compiles because the where clause tells rustc that your implementor will be valid if used with Creature<T>: Alive. The distinction is really subtle, but is, to my opinion, very powerful. With the where clause, you state that Move is implemented for any Creature<T> that is also Alive, but you don’t require them all to be Alive! You could implement Alive for a bunch of creatures, like Creature<Vampire> and Creature<CloudDog>.

So, I remember having read somewhere (maybe in some Rust book, but I’m not quite sure) that the where Self: _ clause was not really useful, but in our case, you can see that it allows to express a completely different semantics.

You could also have used Creature<T>: Alive in place of Self: Alive, as here, Self = Creature<T>.

A small disgression to Haskell land

In Haskell, that code requires to use UndecidableInstances. I don’t know exactly why, but GHC states that the constraint (Alive (Creature a)) is no smaller than the instance head (Move (Creature a)), and this is not permitted, as being undecidable. Enabling the UndecidableInstances GHC extension will make it possible to compile:

class Alive a where
  getHealth :: a -> Health

-- The super class is declared on the left side of the => (the parenthesis are
-- optional, but I’m used to put them all the time, as they are required when you
-- have more constraints).
class (Alive a) => Move a where
  go :: Direction -> a -> a

-- This instance requires UndecidableInstances to compile.
instance (Alive (Creature a)) => Move (Creature a) where
  go = -- …

instance Alive (Creature Vampire) where
  getHealth = -- …

instance Alive (Creature CloudDog) where
  getHealth = -- …

I’m not quite sure why this very exact situation requires UndecidableInstances though. In this case, it seems fine.

I hope you’ll have learned something with this small yet fun type theory candy. Keep the vibes!

Disclaimer: this blog is about Rust and some of its intrinsics semantics, along with software design and architecture – especially for public interfaces like APIs. However, you may find it interesting to apply the concepts to any language that would support those ideas directly or indirectly.

I’ve been writing on a few examples code lately to add to documentations of some crates of mine. I write a lot of code that creates new objects that need other objects in order to be built. Most of the APIs you can see around tend to love the borrow principle – and I do. The idea is simple:

• If you want to create a new type that, for instance, must own a String, most APIs tend to agree that the constructor of your type should take a &str or some sort of borrowing – typical way to do that is to use the AsRef trait, that gives you more power (more about that below). You can then just clone the &str in your function.
• Apply that principle to pretty much any kind of object you need but not types that don’t have easy borrowing semantics. For instance, Rc<_> already carries sharing and dynamic borrowing semantics, so I wouldn’t be surprised to find a constructor that takes a Rc<_> instead of a &Rc<_>.

Depending on the crate you look at, the authors and how they see things, you might find a lot of ways to pass that string to your constructor. Let’s get technical. Especially, I want this blog post to give people a hint at how they should shape their APIs by driving the types with semantics.

The base code

struct Person {
  ident: String
}

Very simple to get started. We have a Person type that carries the name of that person. We don’t want to expose the internals of the type because we might add a lot of other things in there and allowing people to pattern-match a Person would break their code when we add more things.

impl Person {
  pub fn new(name: String) -> Person {
    Person { name }
  }
}

That works. This constructor works by move semantics: it expects you to move a String in in order to copy it into the Person value. Move semantics are clear here: the client must allocate a String or move one already allocated by someone else. This can get a bit boring. Imagine:

let someone = Person::new("Patrick Bateman"); // this won’t compile

Person::new("Patrick Bateman") doesn’t typecheck because "Patrick Bateman" has type &'static str here. It’s not a String.

Drive the allocation from within your code

So how can we fix the code above to have it compile?

let someone = Person::new("Patrick Bateman".to_owned()); // okay now

"Patrick Bateman".to_owned() makes a use of ToOwned::to_owned to clone the string. You could also have used Into::into. However, we’re still requiring clients of our API to perform the allocation – or move in, which is not ideal. A typical solution to this is to use a borrow and clone when needing an owned version.

impl Person {
  pub fn new(name: &str) -> Person {
    Person {
      name: name.to_owned()
    }
  }
}

This API enables the current code to compile:

let someone = Person::new("Patrick Bateman"); // okay because 'static is a subtype of 'a

We’re all good now, right? Not exactly: what happens if we try to pass the initial String we had when we moved something in?

let x: String = …;
let someone = Person::new(x); // not okay now since it’s a type mismatch

To fix this, we need to pass a &str out of a String. Since String implements Deref<Target = str>, it’s quite straight-forward:

let x: String = …;
let someone = Person::new(&x); // okay, here &x derefs to &str

You can also use the more explicit String::as_str method.

A nice trick here to be able to pass both &str and String is to use the AsRef trait.

impl Person {
  pub fn new<N>(name: N) -> Person where N: AsRef<str> {
    Person {
      name: name.as_ref().to_owned()
    }
  }
}

In this case, AsRef<str> is implemented for both &str (it just returns itself) and String – it just deref / uses the as_str method).

However, you can still see a problem here. If we keep using x after the call to Person::new, this code is actually okay and we can move on. If we don’t need x afterwards, we’re just wasting an opportunity to move in instead of allocating!

Drive the allocation from your code and allow for moving in

Clearly, to me, the perfect API would enable you to pass borrows and ask the API code to clone for you or just accept the memory region you provide (i.e. you move something in). The idea is then to accept either &str or String in our case, and clone only a &str… There are several traits and types that provide that feature.

ToOwned + Cow

Cow – for Clone on write – is a very interesting concept. It encodes that you’re either borrowing some data or that you’ve already borrowed it. See it as:

enum Cow<'a, T> where T: 'a + ?Sized + ToOwned {
  Borrowed(&'a T),
  Owned(T::ToOwned)
}

impl<'a, T> Cow<'a, T> where T: 'a + ?Sized + ToOwned {
  pub fn into_owned(self) {
    match self {
      Cow::Borrowed(b) => b.to_owned(),
      Owned(o) => o
    }
  }
}

Now, the interesting part: it’s possible to go from &'a str to Cow<'a, str> and from String to Cow<'a, str> by using Into implementors. That enables us to write this:

impl Person {
  pub fn new<'a, N>(name: N) -> Person where N: Into<Cow<'a, str>> {
    Person { name: name.into().into_owned() }
  }
}

This code will move in a String if you passed one and clone if you passed &str. The following lines of code compile and work as a charm:

let _ = Person::new("Patrick Bateman");

let dawg = "Dawg";
let _ = Person::new(format!("Doggo {}", dawg));

What’s interesting here is that all cases are covered:

• The client has the choice to either allocate – or move in – or borrow.
• It’s even possible to implement Into<Cow<str>> on other types to pass other kind of arguments.

There’s just a little nit: Cow::into_owned obviously patterns match on the variant, inducing a small yet present runtime overhead. We tend to prefer using Cow<_> to dynamically dispatch the decision to clone at runtime, while in our case, it’s more about a static choice (which API function version to use).

Into, as simple as it gets

If you look at the Into<String> implementors, you’ll find impl Into<String> for String – actually, this is a blanket implementor – and impl<'a> Into<String> for &'a str. That implements the same semantics as Cow<str>, but at the type level, removing any remaining runtime overhead.

The new code is even simpler:

impl Person {
  pub fn new<N>(name: N) -> Person where N: Into<String> {
    Person { name: name.into() }
  }
}

Obviously, there are drawbacks:

• You cannot express any lifetimes with this Into<String> trait bound. If you need, for any reason, to dynamically check whether to clone or not, the Cow<str> is the right decision. If you know you’ll always need to allocate, go for Into<String>.
• Into<String> is vague and some types might not even implement it.
• Into<String> doesn’t allow for cheap reading while Cow<str> does. Into<String> requires you to move or allocate prior to reading.

Let’s draw a conclusion

What I wanted to highlight in this blog post is that &_, AsRef<_>, Cow<_>, and Into<_> all have different semantics that can be used to encode different contracts in public interfaces.

• &T means that you don’t require your client to clone nor move because you might only perform read-only computations on T. That gives some intuitions to your clients:
  • You don’t require them to own T.
  • If any allocation must be performed, the client doesn’t have to worry about it.
  • You force them to use a reference. It can have interesting consequences. Imagine &[Something]. This carries one information more: contiguity of the input data – slices are contiguous.
• Q: AsRef<T> is a &T on steroids in terms of software design. It gives more power to the client as they’ll be able to pass more types in your functions. However, you now introduce a polymorphic API. The semantics are the same: you plan to perform only read-only computations or accept not to use a client-provided move-in value if you need to own something. Keep in mind an important hidden property here: because you accept values to be moved in while you just have a read-only contract on them, you also accept clients values to be dropped by corollary.
• Cow<T> states that the client can provide a borrow or an owned variable and you will take advantage of that at runtime. This is important as it gives a good idea about how the memory footprint of the function works: it will take decision at runtime whether or not it needs to own memory.
• Q: Into<T> has vague and large semantics but can be used to encode the fact that you want owned data, but you accept your clients to provide a borrow as long as you can clone. If they provide you with an owned data, you’ll just do nothing (move in).
• (bonus) Q: IntoIterator<Item = T> is also something you can use for the slices thing just above. If (and only if) you don’t care about contiguity or will just inspect values one by one, this is the interface to go. This interface gives hints that your function might iterate through the input and perform (perhaps heavy) computation on each items before stepping to the next one. Using IntoIterator instead of Iterator enables you the same kind of trick you have with AsRef<_> vs. &_: you can take a Vec<_> directly instead of the iterator directly. Keep in mind that this might not have sense for some types, especially if they have several iterator interfaces. The str type has for instance the str::bytes method that gives you an iterator over bytes and the str::chars method that yields an iterator over UTF-8 characters.

This list gives you a good idea about what interface you should use in your public interfaces. Read-only? Owned data? Read and maybe write? All those semantics are visible through those types and traits, you should definitely try to wrap your finger around using all of them. Of course, if you’re just writing a small utility function that needs to borrow the .name of a Person, passing in a &Person seems completely sound. Most of the time, if you don’t know where the data comes from, be the more inclusive and generic as possible.

I hope you like that small article, and as always, keep the vibes.

So, as always, once a year, I attend Revision. But this year, it was a bit different for me. Revision is very impressive and most of the “big demogroups” release their productions they’ve been working on for months or even years. I tend to think “If I release something here, I’ll just be kind of muted by all those massive productions.” Well, less than two weeks before Revision 2017, I was contacted by another demogroup. They asked me to write an invitro – a kind of intro or demo acting as a communication production to invite people to go to another party. In my case, I was proposed to make the Outline 2017 invitro. Ouline was the first party I attended years back then, so I immediately accepted and started to work on something. That was something like 12 days before the Revision deadline.

I have to tell. It was a challenge. All productions I wrote before was made in about a month and a half and featured less content than the Outline Invitation. I had to write a lot of code from scratch. A lot. But it was also a very nice project to test my demoscene framework, written in Rust – you can find spectra here for now; it’s unreleased by the time I’m writing this but I plan to push it onto crates.io very soon.

An hour before hitting the deadline, the beam team told me their Ubuntu compo machine died and that it would be neat if I could port the demo to Windows. I rushed like a fool to make a port – I even forked and modified my OpenAL dependency! – and I did it in 35 minutes. I’m still a bit surprised yet proud!

Anyways, this post is not about bragging. It’s about hindsight. I did that for Céleri Rémoulade as I was the only one working on it – music, gfx, direction and obviously the Rust code. I want to draw a list of what went wrong and what went right. In the first time, for me. So that I have enhancement axis for the next set of demos I’ll do. And for sharing those thoughts so that people can have a sneak peek into the internals of what I do mostly – I do a lot of things! :D – as a hobby on my spare time.

What went wrong

Sooooooooo… What went wrong. Well, a lot of things! spectra was designed to build demo productions in the first place, and it’s pretty good at it. But something that I have to enhance is the user interaction. Here’s a list of what went wrong in a concrete way.

Hot-reloading went wiiiiiiiiiiild²

With that version of spectra, I added the possibility to hot-reload almost everything I use as a resource: shaders, textures, meshes, objects, cameras, animation splines, etc. I edit the file, and as soon as I save it, it gets hot-reloaded in realtime, without having to interact with the program (for curious ones, I use the straight-forward notify crate for registering callbacks to handle file system changes). This is very great and I save a lot of time – Rust compilation is slow, and that’s a lesson I had learned from Céleri Rémoulade: keeping closing the program, make a change, compiling, running is a wast of time.

So what’s the issue with that? Well, the main problem is the fact that in order to implement hot-reloading, I wanted performance and something very simple. So I decided to use shared mutable smart states. As a Haskeller, I kind of offended myself there – laughter! Yeah, in the Haskell world, we try hard to avoid using shared states – IORef – because it’s not referentially transparent and reasoning about it is difficult. However, I tend to strongly think that in some very specific cases, you need such side-effects. I’m balanced but I think it’s the way to go.

Anyway, in Rust, shared mutable state is implemented via two types: Rc/Arc and Cell/RefCell.

The former is a runtime implementation of the borrowing rules and enables you to share a pointer. The borrowing rules are not enforced at compile-time anymore but dynamically checked. It’s great because in some time, you can’t know how long your values will be borrowed or live. It’s also dangerous because you have to pay extra attention to how you borrow your data – since it’s checked at runtime, you can crash your program if you’re not extra careful.

Rc means ref counted and Arc means atomic-ref counted. The former is for values that stay on the same and single thread; the latter is for sharing the pointer between threads.

Cell/RefCell are very interesting types that provide internal mutation. By default, Rust gives you external mutation. That is, if you have a value and its address, can mutate what you have at that address. On the other hand, internal mutation is introduced by the Cell and RefCell types. Those types enable you to mutate the content of an object stored at a given address without having the exterior mutatation property. It’s a bit technical and related to Rust, but it’s often used to mutate the content of a value via a function taking an immutable value.

Cell only accepts values that can be copied bit-wise and RefCell works with references.

Now, if you combine both – Rc<RefCell<_>>, you end up with a shareable – Rc<_> – mutable – RefCell<_> – value. If you have a value of type Rc<RefCell<u32>> for instance, that means you can clone that value and store it everywhere in the same thread, and at any time, borrow it and inspect and/or mutate its content. All copies of the values will observe the change. It’s a bit like C++’s shared_ptr, but it’s safer – thank you Rust!

So what went wrong with that? Well, the borrow part. Because Rust is about safety, you still need to tell it how you want to borrow at runtime. This is done with the [RefCell::borrow()](https://doc.rust-lang.org/std/cell/struct.RefCell.html#method.borrow] and RefCell::borrow_mut() functions. Those functions return a special object that borrows the pointed object as long as it lives. Then, when it goes out of scope, it releases the borrow.

So any time you want to use an object that is hot-reloadable with my framework, you have to call one of the borrow functions presented above. You end up with a lot of borrows, and you have to keep in mind that you can litterally crash your program if you violate the borrowing rules. This is a nasty issue. So far, I haven’t really spent time trying to fix that, but that something I have to figure out.

Resources declaration in code

This is a bit tricky. As a programmer, I’m used to write algorithms and smart architectures to transform data and resolve problems. I’m given inputs and I provide the outputs – the solutions. However, a demoscene production is special: you don’t have inputs. You create artsy audiovisual outputs from nothing but time. So you don’t really write code to solve a problem. You write code to create something that will be shown on screen or in a headphone. This aspect of demo coding has an impact on the style and the way you code. Especially in crunchtime. I have to say, I was pretty bad on that part with that demo. To me, code should only be about transformations – that’s why I love Haskell so much. But my code is clearly not.

If you know the let keyword in Rust, well, imagine hundreds and hundreds of lines starting with let in a single function. That’s most of my demo. In rush time, I had to declare a lot of things so that I can use them and transform them. I’m not really happy with that, because those were data only. Something like:

let outline_emblem = cache.get::<Model>("Outline-Logo-final.obj", ()).unwrap();
let hedra_01 = cache.get::<Model>("DSR_OTLINV_Hedra01_Hi.obj", ()).unwrap();
let hedra_02 = cache.get::<Model>("DSR_OTLINV_Hedra02_Hi.obj", ()).unwrap();
let hedra_04 = cache.get::<Model>("DSR_OTLINV_Hedra04_Hi.obj", ()).unwrap();
let hedra_04b = cache.get::<Model>("DSR_OTLINV_Hedra04b_Hi.obj", ()).unwrap();
let twist_01 = cache.get::<Object>("DSR_OTLINV_Twist01_Hi.json", ()).unwrap();

It’s not that bad. As you can see, spectra features a resource cache that provides several candies – hot-reloading, resource dependency resolving and resource caching. However, having to declare those resources directly in the code is a nasty boilerplate to me. If you want to add a new object in the demo, you have to turn it off, add the Rust line, re-compile the whole thing, then run it once again. It breaks the advantage of having hot-reloading and it pollutes the rest of the code, making it harder to spot the actual transformations going on.

This is even worse with the way I handle texts. It’s all &'static str declared in a specific file called script.rs with the same insane load of let. Then I rasterize them in a function and use them in a very specific way regarding the time they appear. Not fun.

Animation edition

Most of the varying things you can see in my demos are driven by animation curves – splines. The bare concept is very interesting: an animation contains control points that you know have a specific value at a given time. Values in between are interpolated using an interpolation mode that can change at each control points if needed. So, I use splines to animate pretty much everything: camera movements, objects rotations, color masking, flickering, fade in / fade out effects, etc.

Because I wanted to be able to edit the animation in a comfy way – lesson learned from Céleri Rémoulade, splines can be edited in realtime because they’re hot-reloadable. They live in JSON files so that you just have to edit the JSON objects in each file and as soon as you save it, the animation changes. I have to say, this was very ace to do. I’m so happy having coded such a feature.

However, it’s JSON. It’s already a thing. Though, I hit a serious problem when editing the orientations data. In spectra, an orientation is encoded with a unit quaternion. This is a 4-floating number – hypercomplex. Editing those numbers in a plain JSON file is… challenging! I think I really need some kind of animation editor to edit the spline.

I’ll be talking about:

My editor

I’ve tried a lot of editors in the last ten years. I spent a year using emacs but eventually discovered – erm, learned! – vim and completely fell in love with the modal editor. It was something like a bit less than ten years ago. I then tried a lot of other editors (because of curiosity) but failed to find something that would help be better than vim to write code. I don’t want to start an editor war; here’s just my very personal point of view on editing. The concept of modes in vim enables me to use a very few keystrokes to perform what I want to do (moving, commands, etc.) and I feel very productive that way.

A year ago, a friend advised me to switch to neovim, which I have. My editor of the time is then neovim, but it’s so close to vim that I tend to use (neo)vim. :)

I don’t use any other editing tool. I even use neovim for taking notes while in a meeting or when I need to format something in Markdown. I just use it everywhere.

My neovim setup

I use several plugins:

Plugin 'VundleVim/Vundle.vim'
Plugin 'ctrlpvim/ctrlp.vim'
Plugin 'gruvbox'
Plugin 'neovimhaskell/haskell-vim'
Plugin 'itchyny/lightline.vim'
Plugin 'rust-lang/rust.vim'
Plugin 'jacoborus/tender.vim'
Plugin 'airblade/vim-gitgutter'
Plugin 'tikhomirov/vim-glsl'
Plugin 'plasticboy/vim-markdown'
Plugin 'cespare/vim-toml'
Plugin 'mzlogin/vim-markdown-toc'
Plugin 'ElmCast/elm-vim'
Plugin 'raichoo/purescript-vim'
Plugin 'easymotion'
Plugin 'scrooloose/nerdtree'
Plugin 'ryanoasis/vim-devicons'
Plugin 'tiagofumo/vim-nerdtree-syntax-highlight'
Plugin 'mhinz/vim-startify'
Plugin 'Xuyuanp/nerdtree-git-plugin'
Plugin 'tpope/vim-fugitive'
Plugin 'MattesGroeger/vim-bookmarks'
Plugin 'luochen1990/rainbow'

Vundle

A package manager. It just takes the list you read above and clones / keeps updated the git repositories of the given vim packages. It’s a must have. There’re alternatives like Pathogen but Vundle is very simple to setup and you don’t have to care about the file system: it cares for you.

ctrlp

This is one is a must-have. It gives you a fuzzy search buffer that traverse files, MRU, tags, bookmarks, etc.

I mapped the file search to the , f keys combination and the tag fuzzy search to , t.

gruvbox

The colorscheme I use. I don’t put an image here since you can find several examples online.

This colorscheme also exists for a lot of other applications, among terminals and window managers.

haskell-vim

Because I write a lot of Haskell, I need that plugin for language hilighting and linters… mostly.

lightline

The Lightline vim statusline. A popular alternative is Powerline for instance. I like Lightline because it’s lightweight and has everything I want.

rust.vim

Same as for Haskell: I wrote a lot of Rust code so I need the language support in vim.

tender

A colorscheme for statusline.

vim-gitgutter

I highly recommend this one. It gives you diff as icons in the symbols list (behind the line numbers).

vim-glsl

GLSL support in vim.

vim-markdown

Markdown support in vim.

vim-toml

TOML support in vim.

TOML is used in Cargo.toml, the configuration file for Rust projects.

vim-markdown-toc

You’re gonna love this one. It enables you to insert a table of contents wherever you want in a Markdown document. As soon as you save the buffer, the plugin will automatically refresh the TOC for you, keeping it up-to-date. A must have if you want table of contents in your specifications or RFC documents.

elm-vim

Elm support in vim.

purescript-vim

Purescript support in vim.

easymotion

A must have. As soon as you hit the corresponding keys, it will replace all words in your visible buffer by a set of letters (typically one or two), enabling you to just press the associated characters to jump to that word. This is the vim motion on steroids.

nerdtree

A file browser that appears at the left part of the current buffer you’re in. I use it to discover files in a file tree I don’t know – I use it very often at work when I work on a legacy project, for instance.

vim-devicons

A neat package that adds icons to vim and several plugins (nerdtree, ctrlp, etc.). It’s not a must-have but I find it gorgeous so… just install it! :)

vim-nerdtree-syntax-highlight

Add more formats and files support to nerdtree.

vim-startify

A cute plugin that will turn the start page of vim into a MRU page, enabling you to jump to the given file by just pressing its associated number.

It also features a cow that gives you fortune catchphrases. Me gusta.

nerdtree-git-plugin

Git support in nerdtree, it adds markers in front of file that have changes, diff, that were added, etc.

vim-fugitive

A good Git integration package to vim. I use it mostly for its Gdiff diff tooling directly in vim, even though I like using the command line directly for that. The best feature of that plugin is the integrated blaming function, giving you the author of each line directly in a read-only vim buffer.

vim-bookmarks

Marks on steroids.

rainbow

This little plugins is very neats as it allows me to add colors to matching symbols so that I can easily see where I am.

Workflow in Rust

I’m a rustacean. I do a lot of Rust on my spare time. My typical workflow is the following:

1. I edit code in neovim
2. Depending on the project (whether I have a robust unit tests base or not or whether I’m writing a library or an app), I use several cargo commands:
   • for a library project, I split my screen in two and have a cargo watch -x test running; this command is a file watcher that will run all the tests suites in the project as soon as a file changes;
   • for an app project, I split my screen in two and have a cargo watch – similar to cargo watch -x check running; this command is a file watcher that will proceed through the compilation of the binary but it will not compile it down to an actual binary; it’s just a check, it doesn’t produce anything you can run. You can manually run that command with cargo check. See more here;
   • for other use cases, I tend to run cargo check by hand and run cargo build to test the actualy binary
3. Because I’m a unix lover, I couldn’t work correctly without a terminal, so I use neovim in a terminal and switch from the console to the editor with the keybinding C-z and the jobs and fg commands. I do that especially to create directories, issue git commands, deploy things, etc. It’s especially ultra cool to run a rg search or even a find / fd. I sometimes do that directly in a neovim buffer – with the :r! command – when I know I need to edit things from the results. Bonus: refactoring with tr, sed and find -exec is ultra neat.

Workflow in Haskell

The workflow is almost exactly the same besides the fact I use the stack build --fast --file-watch command to have a file watcher.

Haskell’s stack doesn’t currently the awesome check command Rust’s cargo has. Duh :(

I also have a similar workflow for other languages I work in, like Elm, even though I use standard unix tools for the file watching process.

Git workflow

Aaaah… git. What could I do without it? Pretty much nothing. git is such an important piece of software and brings such an important project philosophy and behaviors to adopt.

What I find very cool in git – besides the tool itself – is everything around it. For instance, on GitHub, you have the concept of Pull Request (PR) – Merge Request in GitLab (MR). Associated with a good set of options, like disallowing people to push on the master branch, hooks, forcing people to address any comments on their MR, this allows for better code reviews and overall a way better quality assurance of the produced code than you could have without using all this infrastructure. Add a good set of DevOps for deployment and production relatide issues and you have a working team that has no excuse to produce bad code!

Some git candies I love working with

My neovim fugitive plugin allows me to open a special buffer with the :Gblame command that gives me a git blame annotation of the file. This might be trivial but it’s very useful, especially at work when I have my colleagues next me – it’s always better to directly ask them about something than guessing.

Another one that I love is :Gdiff, that gives you a git diff of the modification you’re about to stage. I often directly to a git diff in my terminal, but I also like how this plugin nails it as well. Very pratictal!

General note on workflow

It’s always funny to actually witness difference in workflows, especially at work. People who use mostly IDEs are completely overwhelmed by my workflow. I was completely astonished at work that some people hadn’t even heard of sed before – they even made a Google search! I’m a supporter of the philosophy that one should use the tool they feel comfortable with and that there’s no “ultimate” tool for everyone. However, for my very own person, I really can’t stand IDEs, with all the buttons and required clicks you have to perform all over the screen. I really think it’s a waste of time, while using a modal editor like neovim with a bépo keyboard layout (French dvorak) and going back and forth to the terminal is just incredibly simple, yet powerful.

I had a pretty good experience with Atom, a modern editor. But when I’ve been told it’s written with web technologies, the fact it’s slow as fck as soon as you start having your own tooling (extensions), its pretty bad behavior and incomprensible “Hey, I do whatever the fck I want and I’ll just reset your precious keybindings!” or all the weird bugs – some of my extensions won’t just work if I have an empty pane open, wtf?!… well, I was completely bolstered that GUI interfaces, at least for coding and being productive, are cleary not for me. With my current setup, my hands never move from the keyboard – my thrists are completely static. With all the candies like easymotion, ctrlp, etc. etc. I just can’t find any other setups faster and comfier than this one.

There’s even an extra bonus to my setup: because I use mostly unix tools and neovim, it’s pretty straigth-forward to remote-edit something via ssh, because everything happens in a terminal. That’s not something you can do easily with Atom, Sublime Text or any other editors / IDEs – and you even pay for that shit! No offence!

However, there’s a caveat: because pretty much everything I do is linked to my terminal, the user experience mostly relies on the performance of the terminal. Using a bad terminal will result in an overall pretty bad experience, should it be editing, compiling, git or ssh. That’s why I keep lurking at new terminal emulaters – alacritty seems very promising, yet it’s still too buggy and lacks too many features to be production-ready to me – but it’s written in Rust and is GPU-accelerated, hells yeah!

Conclusion

Whoever you are, whatever you do, whomever you work with, I think the most important thing about workflow is to find the one that fits your needs the most. I have a profound, deep digust for proprietary and closed software like Sublime Text and IDEs that use GUIs while keyboard shortcut are just better. To me, the problem is about the learning curve and actually wanting to pass it – because yes, learning (neo)vim in details and mastering all its nits is not something you’ll do in two weeks; it might take months or years, but it’s worth it. However, as I said, if you just feel good with your IDE, I will not try to convert you to a modal editor or a unix-based workflow… because you wouldn’t be as productive as you already are.

Keep the vibe!

It’s been a while I haven’t written anything on my blog. A bit of refreshment doesn’t hurt much, what do you think?

As a demoscener, I attend demoparties, and there will be a very important and fun one in about a month. I’m rushing on my 3D application so that I can finish something to show up, but I’m not sure I’ll have enough spare time. That being said, I need to be able to represent smooth moves and transitions without any tearing. I had a look into a few Haskell spline libraries, but I haven’t found anything interesting – or not discontinued.

Because I do need splines, I decided to write my very own package. Meet smoothie, my BSD3 Haskell spline library.

Why splines?

A spline is a curve defined by several polynomials. It has several uses, like vectorial graphics, signal interpolation, animation tweening or simply plotting a spline to see how neat and smooth it looks!

Splines are defined using polynomials. Each polynomials is part of the curve and connected one-by-one. Depending on which polynomial(s) you chose, you end up with a different shape.

For instance, 1-degree polynomials are used to implement straight lines.

As you can see, we can define a few points, and interpolate in between. This is great, because we can turn a discrete set of points into lines.

Even better, we could use 3-degree polynomials or cosine functions to make each part of the spline smoother:

We still have discrete points, but in the end, we end up with a smooth set of points. Typically, imagine sampling from the spline with time for a camera movement. It helps us to build smooth moves. This is pretty important when doing animation. If you’re curious about that, I highly recommend having a look into key frames.

Using splines in Haskell

So I’ve been around implementing splines in Haskell the most general way as possible. However, I don’t cover – yet? – all kinds of splines. In order to explain my design choices, I need to explain a few very simple concepts first.

Sampling

A spline is often defined by a set of points and polynomials. The first point has the starting sampling value. For our purpose, we’ll set that to 0:

let startSampler = 0

The sampling value is often Float, but it depends on your spline and the use you want to make of it. It could be Int. The general rule is that it should be orderable. If we take two sampling values s and t, we should be able to compare s and t (that’s done through the typeclass constraint Ord in Haskell).

So, if you have a spline and a sampling value, the idea is that sampling the spline with startSampler gives you the first point, and sampling with t with t > startSampler gives you another point, interpolated using points of the spline. It could use two points, three, four or even more. It actually depends on the polynomials you use, and the interpolating method.

In smoothie, sampling values have types designed by s.

Control points

A spline is made of points. Those points are called control points and smoothie uses CP s a to refer to them, where s is the sampling type and a the carried value.

Although they’re often used to express the fact that the curve should pass through them, they don’t have to lie on the curve itself. A very common and ultra useful kind of spline is the B-spline.

With that kind of spline, the property that the curve passes through the control points doesn’t hold. It passes through the first and last ones, but the ones in between are used to shape it, a bit like magnets attract things around them.

Keep in mind that control points are very important and used to define the main aspect of the curve.

Polynomials

Polynomials are keys to spline interpolation. They’re used to deduce sampled points. Interpolation is a very general term and used in plenty of domains. If you’re not used to that, you should inquiry about linear interpolation and cubic interpolation, which are a very good start.

Polynomials are denoted by Polynomial s a in smoothie, where s and a have the same meaning than in CP s a.

Getting started with smoothie

Types and constraints

smoothie has then three important types:

• CP s a, the control points
• Polynomial, the polynomials used to interpolate between control points
• Spline s a, of course

The whole package is parameterized by s and a. As said earlier, s is very likely to require an Ord constraint. And a… Well, since we want to represent points, let’s wonder: which points? What kind of points? Why even “points”? That’s a good question. And this is why you may find smoothie great: it doesn’t actually know anything about points. It accepts any kind of values. Any? Almost. Any values that are in an additive group.

“What the…”

I won’t go into details, I’ll just vulgarize them so that you get quickly your feet wet. That constraint, when applied to Haskell, makes a to be an endofunctor – i.e. Functor – and additive – i.e. Additive. It also requires it to be a first-class value – i.e. its kind should be * -> *.

With Functor and Additive, we can do two important things:

• First, with Functor. It enables us to lift computation on the inner type. We can for instance apply a single function inside a, like *k or /10.
• Then, with Additive. It enables us to add our types, like a + b.

We can then make linear combinations, like ak + bq. This property is well known for vector spaces.

The fun consequence is that providing correct instances to Functor and Additive will make your type useable with smoothie as carried value in the spline! You might also have to implement Num and Ord as well, though.

Creating a spline

Creating a spline is done with the spline function, which signature is:

spline :: (Ord a, Ord s) => [(CP s a, Polynomial s a)] -> Spline s a

It takes a list of control points associated with polynomials and outputs a spline. That requires some explainations… When you’ll be sampling the spline, smoothie will look for which kind of interpolation method it has to use. This is done by the lower nearest control point to the sampled value. Basically, a pair (cp,polynomial) defines a new point and the interpolation method to use for the curve ahead of the point.

Of course, the latest point’s polynomial won’t be used. You can set whatever you want then – protip: you can even set undefined because of laziness.

Although the list will be sorted by spline, I highly recommend to pass a sorted list, because dealing with unordered points might have no sense.

A control point is created by providing a sample value and the carried value. For instance, using linear’s V2 type:

let cp0 = CP 0 $ V2 1 pi

That’s a control point that represents V2 1 pi when sampling is at 0. Let’s create another:

let cp1 = CP 3.341 $ V2 0.8 10.5

Now, let’t attach a polynomial to them!

Hold that for me please

The simplest polynomial – wich is actually not a polynomial, but heh, don’t look at me that way – is the 0-degree polynomial. Yeah, a constant function. It takes the lower control point, and holds it everwhere on the curve. You could picture that as a staircase function:

You might say that’s useless; it’s actually not; it’s even pretty nice. Imagine you want to attach your camera position onto such a curve. It will make the camera jump in space, which could be desirable for flash looks!

Use the hold Polynomial to use such a behavior.

Oh yeah, it’s straightforward!

1-degree functions often describe lines. That is, linear is the Polynomial to use to connect control points with… straight lines.

Have fun on the unit circle!

One very interesting Polynomial is cosine, that defines a cosine interpolation, used to smooth the spline and make it nicer for moves and transitions.

Highway to the danger zone

If you’re crazy, you can experiment around with linearBy, which, basically, is a 1-degree polynomial if you pass id, but will end up in most complex shapes if you pass another function – (s -> s). Dig in documentation on hackage for further information.

Sampling our spline

Ok, let’s use a linear interpolation to sample our spline:

let spl = spline [(cp0,linear),(cp1,hold)]

Note: I used hold as a final polynomial because I don’t like using undefined.

Ok, let’s see how to sample that. smoothie exports a convenient function for sampling:

smooth :: Ord s => Spline s a -> s -> Maybe a

smooth spl s takes the sampling value s and maybe interpolate it in the spl spline.

“Maybe? Why aren’t you sure?”

Well, that’s pretty simple. In some cases, the curve is not defined at the sampling value you pass. Before the first point and after, basically. In those cases, you get Nothing.

That’s not the end

I wrote smoothie in a few hours, in a single day. You might have ideas. I want it to be spread and widely used by awesome people. Should you do graphics programming, sound programming, animation or whatever implying splines or smoothness, please provide feedback!

For people that would like to get contributing, here’s the github page and the issue tracker.

If no one comes up with, I’ll try to add some cubic interpolation methods, like hermitian splines, and one of my favorite, the famous Catmull Rom spline interpolation method.

As always, have fun hacking around, and keep doing cool stuff and sharing it!

However, I saw interested people, and I think it’s the right time to write a blog post about some design choices. I’ll start off with luminance and I’ll speak about spectra in another blog post because I truly think spectra starts to become very interesting (I have my own shading language bundled up within, which is basically GLSL on steroids).

luminance and how it copes with fast and as-stateless-as-possible graphics

Origins

luminance is a library that I wrote, historically, in Haskell. You can find the package here if you’re interested – careful though, it’s dying and the Rust version has completely drifted path with it. Nevertheless, when I ported the library to Rust, I imported the same “way of programming” that I had·have in Haskell – besides the allocation scheme; remember, I’m a demoscener, I do care a lot about performance, CPU usage, cache friendliness and runtime size. So the Rust luminance crate was made to be hybrid: it has the cool functional side that I imported from my Haskell codebase, and the runtime performance I wanted that I had when I wrote my two first 64k in C++11. I had to remove and work around some features that only Haskell could provide, such as higher-kinded types, type families, functional dependencies, GADTs and a few other things such as existential quantification (trait objects saved me here, even though I don’t use them that much in luminance now).

I have to admit, I dithered a lot about the scope of luminance — both in Haskell and Rust. At first, I thought that it’d be great to have a “base” crate, hosting common and abstracted code, and “backend” crates, implementing the abstract interface. That would enable me to have several backends – OpenGL, Vulkan, Metal, a software implementation, something for Amiigaaaaaaa, etc. Though, time has passed, and now, I think it’s:

• Overkill.
• A waste of framework.

The idea is that if you need to be very low-level on the graphics stack of your application, you’re likely to know what you are doing. And then, your needs will be very precise and well-defined. You might want very specific piece of code to be available, related to a very specific technology. That’s the reason why abstracting over very low-level code is not a good path to me: you need to expose as most as posssible the low-level interface. That’s the goal of luminance: exposing OpenGL’s interface in a stateless, bindless and typesafe way, with no or as minimal as possible runtime overhead.

More reading here.

Today

Today, luminance is almost stable – it still receives massive architecture redesign from time to time, but it’ll hit the 1.0.0 release soon. As discussed with kvark lately, luminance is not about the same scope as gfx’s one. The goals of luminance are:

• To be a typesafe, stateless and bindless OpenGL framework.
• To provide a friendly experience and expose as much as possible all of the OpenGL features.
• To be very lightweight (the target is to be able to use it without std nor core).

To achieve that, luminance is written with several aspects in mind:

• Allocation must be explicitely stated by the user: we must avoid as much as possible to allocate things in luminance since it might become both a bottleneck and an issue to the lightweight aspect.
• Performance is a first priority; safety comes second. If you have a feature that can be either exclusively performant or safe, it must then be performant. Most of the current code is, for our joy, both performant and safe. However, some invariants are left around the place and you might shoot your own feet. This is an issue and some reworking must be done (along with tagging some functions and traits unsafe).
• No concept of backends will ever end up in luminance. If it’s decided to switch to Vulkan, the whole luminance API will and must be impacted, so that people can use Vulkan the best possible way.
• A bit like the first point, the code must be written in a way that the generated binary is as small as possible. Generics are not forbidden – they’re actually recommended – but things like crate dependencies are likely to be forbidden (exception for the gl dependency, of course).
• Windowing must not be addressed by luminance. This is crucial. As a demoscener, if I want to write a 64k with luminance, I must be able to use a library over X11 or the Windows API to setup the OpenGL context myself, set the OpenGL pointers myself, etc. This is not the typical usecase – who cares besides demosceners?! – but it’s still a good advantage since you end up with loose coupling for free.

The new luminance

luminance has received more attention lately, and I think it’s a good thing to talk about how to use it. I’ll add examples on github and its docs.rs online documentation.

I’m going to do that like a tutorial. It’s easier to read and you can test the code in the same time. Let’s render a triangle!

Note: keep in mind that you need a nightly compiler to compile luminance.

Getting your feet wet

I’ll do everything from scratch with you. I’ll work in /tmp:

$ cd /tmp

First thing first, let’s setup a lumitest Rust binary project:

$ cargo init --bin lumitest
  Created binary (application) project
$ cd lumitest

Let’s edit our Cargo.toml to use luminance. We’ll need two crates:

• The luminance crate.
• A way to open the OpenGL context; we’ll use GLFW, so the luminance-glfw crate.

At the time of writing, corresponding versions are luminance-0.23.0 and luminance-glfw-0.3.2.

Have the following [dependencies] section

[dependencies]
luminance = "0.23.0"
luminance-glfw = "0.3.2"

$ cargo check

Everything should be fine at this point. Now, let’s step in in writing some code.

extern crate luminance;
extern crate luminance_glfw;

use luminance_glfw::{Device, WindowDim, WindowOpt};

const SCREEN_WIDTH: u32 = 960;
const SCREEN_HEIGHT: u32 = 540;

fn main() {
  let rdev = Device::new(WindowDim::Windowed(SCREEN_WIDTH, SCREEN_HEIGHT), "lumitest", WindowOpt::default());
}

The main function creates a Device that is responsible in holding the windowing stuff for us.

Let’s go on:

match rdev {
  Err(e) => {
    eprintln!("{:#?}", e);
    ::std::process::exit(1);
  }

  Ok(mut dev) => {
    println!("let’s go!");
  }
}

This block will catch any Device errors and will print them to stderr if there’s any.

Let’s write the main loop:

'app: loop {
  for (_, ev) in dev.events() { // the pair is an interface mistake; it’ll be removed
    match ev {
      WindowEvent::Close | WindowEvent::Key(Key::Escape, _, Action::Release, _) => break 'app,
      _ => ()
    }
  }
}

This loop runs forever and will exit if you hit the escape key or quit the application.

Setting up the resources

Now, the most interesting thing: rendering the actual triangle! You will need a few things:

type Position = [f32; 2];
type RGB = [f32; 3];
type Vertex = (Position, RGB);

const TRIANGLE_VERTS: [Vertex; 3] = [
  ([-0.5, -0.5], [0.8, 0.5, 0.5]), // red bottom leftmost
  ([-0., 0.5], [0.5, 0.8, 0.5]), // green top
  ([0.5, -0.5], [0.5, 0.5, 0.8]) // blue bottom rightmost
];

Position, Color and Vertex define what a vertex is. In our case, we use a 2D position and a RGB color.

You have a lot of choices here to define the type of your vertices. In theory, you can choose any type you want. However, it must implement the Vertex trait. Have a look at the implementors that already exist for a faster start off!

Important: do not confuse between [f32; 2] and (f32, f32). The former is a single 2D vertex component. The latter is two 1D components. It’ll make a huge difference when writing shaders.

The TRIANGLE_VERTS is a constant array with three vertices defined in it: the three vertices of our triangle. Let’s pass those vertices to the GPU with the Tess type:

// at the top location
use luminance::tess::{Mode, Tess, TessVertices};

// just above the main loop
let triangle = Tess::new(Mode::Triangle, TessVertices::Fill(&TRIANGLE_VERTS), None);

This will pass the TRIANGLE_VERTS vertices to the GPU. You’re given back a triangle object. The Mode is a hint object that states how vertices must be connected to each other. TessVertices lets you slice your vertices – this is typically enjoyable when you use a mapped buffer that contains a dynamic number of vertices.

We’ll need a shader to render that triangle. First, we’ll place its source code in data:

$ mkdir data

Paste this in data/vs.glsl:

layout (location = 0) in vec2 co;
layout (location = 1) in vec3 color;

out vec3 v_color;

void main() {
  gl_Position = vec4(co, 0., 1.);
  v_color = color;
}

Paste this in data/fs.glsl:

in vec3 v_color;

out vec4 frag;

void main() {
  frag = vec4(v_color, 1.);
}

And add this to your main.rs:

const SHADER_VS: &str = include_str!("../data/vs.glsl");
const SHADER_FS: &str = include_str!("../data/fs.glsl");

Note: this is not a typical workflow. If you’re interested in shaders, have a look at how I do it in spectra. That is, hot reloading it via SPSL (Spectra Shading Language), which enables to write GLSL modules and compose them in a single file but just writing functions. The functional programming style!

Same thing as for the tessellation, we need to pass the source to the GPU’s compiler to end up with a shader object:

// add this at the top of your main.rs
use luminance::shader::program::Program;

// below declaring triangle
let (shader, warnings) = Program::::from_strings(None, SHADER_VS, None, SHADER_FS).unwrap();

for warning in &warnings {
  eprintln!("{:#?}", warning);
}

Finally, we need to tell luminance in which framebuffer we want to make the render. It’s simple: to the default framebuffer, which ends up to be… your screen’s back buffer! This is done this way with luminance:

use luminance::framebuffer::Framebuffer;

let screen = Framebuffer::default([SCREEN_WIDTH, SCREEN_HEIGHT]);

And we’re done for the resources. Let’s step in the actual render now.

The actual render: the pipeline

luminance’s approach to render is somewhat not very intuitive yet very simple and very efficient: the render pipeline is explicitly defined by the programmer in Rust, on the fly. That means that you must express the actual state the GPU must have for the whole pipeline. Because of the nature of the pipeline, which is an AST (Abstract Syntax Tree), you can batch sub-parts of the pipeline (we call such parts nodes) and you end up with minimal GPU state switches. The theory is as following:

• At top most, you have the pipeline function that introduces the concept of shading things to a framebuffer.
  • Nested, you find the concept of a shader gate. That is, an object linked to its parent (pipeline) and that gives you the concept of shading things with a shader.
    • Nested, you find the concept of rendering things. That is, can set on such nodes GPU states, such as whether you want a depth test, blending, etc.
      • Nested, you find the concept of a tessellation gate, enabling you to render actual Tess objects.

That deep nesting enables you to batch your objects on a very fine granularity. Also, notice that the functions are not about slices of Tess or hashmaps of Program. The allocation scheme is completely ignorant about how the data is traversed, which is good: you decide. If you need to borrow things on the fly in a shading gate, you can.

Let’s get things started:

use luminance::pipeline::{entry, pipeline};

entry(|_| {
  pipeline(&screen, [0., 0., 0., 1.], |shd_gate| {
    shd_gate.shade(&shader, |rdr_gate, _| {
      rdr_gate.render(None, true, |tess_gate| {
        let t = ▵
        tess_gate.render(t.into());
      });
    });
  });
});

We just need a final thing now: since we render to the back buffer of the screen, if we want to see anything appear, we need to swap the buffer chain so that the back buffer become the front buffer and the front buffer become the back buffer. This is done by wrapping our render code in the Device::draw function:

dev.draw(|| {
  entry(|_| {
    pipeline(&screen, [0., 0., 0., 1.], |shd_gate| {
      shd_gate.shade(&shader, |rdr_gate, _| {
        rdr_gate.render(None, true, |tess_gate| {
          let t = ▵
          tess_gate.render(t.into());
        });
      });
    });
  });
});

You should see this:

As you can see, the code is pretty straightforward. Let’s get deeper, and let’s kick some time in!

use std::time::Instant;

// before the main loop
let t_start = Instant::now();
// in your main loop
let t_dur = t_start.elapsed();
let t = (t_dur.as_secs() as f64 + t_dur.subsec_nanos() as f64 * 1e-9) as f32;

We have the time. Now, we need to pass it down to the GPU (i.e. the shader). luminance handles that kind of things with two concepts:

• Uniforms.
• Buffers.

Uniforms are a good match when you want to send data to a specific shader, like a value that customizes the behavior of a shading algorithm.

Because buffers are shared, you can use buffers to share data between shader, leveraging the need to pass the data to all shader by hand – you only pass the index to the buffer that contains the data.

We won’t conver the buffers for this time.

Because of type safety, luminance requires you to state which types the uniforms the shader contains are. We only need the time, so let’s get this done:

// you need to alter this import
use luminance::shader::program::{Program, ProgramError, Uniform, UniformBuilder, UniformInterface, UniformWarning};

struct TimeUniform(Uniform);

impl UniformInterface for TimeUniform {
  fn uniform_interface(builder: UniformBuilder) -> Result<(Self, Vec), ProgramError> {
    // this will fail if the "t" variable is not used in the shader
    //let t = builder.ask("t").map_err(ProgramError::UniformWarning)?;

    // I rather like this one: we just forward up the warning and use the special unbound uniform
    match builder.ask("t") {
      Ok(t) => Ok((TimeUniform(t), Vec::new())),
      Err(e) => Ok((TimeUniform(builder.unbound()), vec![e]))
    }
  }
}

The UniformBuilder::unbound is a simple function that gives you any uniform you want: the resulting uniform object will just do nothing when you pass values in. It’s a way to say “— Okay, I don’t use that in the shader yet, but don’t fail, it’s not really an error”. Handy.

And now, all the magic: how do we access that uniform value? It’s simple: via types! Have you noticed the type of our Program? For the record:

let (shader, warnings) = Program::::from_strings(None, SHADER_VS, None, SHADER_FS).unwrap();

See the type is parametered with three type variables:

• The first one – here Vertex, our own type – is for the input of the shader program.
• The second one is for the output of the shader program. It’s currently not used at all by luminance by is reserved, as it will be used later for enforcing even further type safety.
• The third and latter is for the uniform interface.

You guessed it: we need to change the third parameter from () to TimeUniform:

let (shader, warnings) = Program::::from_strings(None, SHADER_VS, None, SHADER_FS).unwrap();

And that’s all. Whenever you shade with a ShaderGate, the type of the shader object is being inspected, and you’re handed with the uniform interface:

shd_gate.shade(&shader, |rdr_gate, uniforms| {
  uniforms.0.update(t);

  rdr_gate.render(None, true, |tess_gate| {
    let t = ▵
    tess_gate.render(t.into());
  });
});

Now, change your fragment shader to this:

in vec3 v_color;

out vec4 frag;

uniform float t;

void main() {
  frag = vec4(v_color * vec3(cos(t * .25), sin(t + 1.), cos(1.25 * t)), 1.);
}

And enjoy the result! Here’s the gist that contains the whole main.rs.

• The glsl_str! proc-macro would have only survived the 0.1 version. It’s now deprecated and will be removed soon.
• The glsl! proc-macro now supports GLSL pragmas (both #version and #extension).

“Wait. GLSL pragmas? You said it wasn’t possible because of how the Rust tokenizer thinks newlines are meaningless while the GLSL grammar uses them to separate pragmas from other pragmas and code.”

Yes, at the time of writing glsl-quasiquote, I just thought that since TokenStream is about a stream of Rust tokens, GLSL’s pragmas would be impossible to encode with this mechanism. In fact, it’s actually impossible per-se: newlines are not Rust tokens and are just completely ignored by the tokenizer… or are they?

Enter the motivation

At the announcement of the release of glsl-quasiquote on Reddit, someone – actually, somebodddy just pointed out that I could use Rust attributes to encode GLSL pragmas. That would enable me to have them as regular Rust tokens at the cost of modifying very, very briefly the GLSL grammar for the pragmas – nothing bad, really. Their idea, which I liked, was this:

Instead of:

#version 330 core

We could write:

#![version 330 core]

Such a small change yet effective, right? I was appealed by the idea, so I implemented it. The code was a bit messy because I did all the token pattern-matching by hand but I sketched something up that worked in a few minutes. After having a discussion with a friend on IRC (antoyo!), I realized that I was plain wrong from the beginning assertion that newlines are ignored by the Rust tokenizer: they’re not.

Enter the hack

The proc_macro crate, which is used to manipulate Rust tokens, has several items that I used to implement the quasiquoter:

• TokenStream, a stream of tokens. This is a very opaque type, you can just create an empty one, display it as a String and marshall to and from an iterator interface, which element type is TokenTree.
• TokenTree, a single token or a delimited sequence of token trees, like a group of tokens logically united (with brackets, parens, etc.).
• All the types used in TokenTree variants, such as Literal, Punct, Group, etc.

There’s something I just had completely forgotten when claiming that newlines were ignored: pretty much all of the tokens contained in TokenTree have an associated Span. I’m not sure what the initial intent was about with this type, but in order to use it, I had to add the proc_macro_span feature. That type gives positional information along with macro expansion on a token. Specifically, it has a method that gives the line and column at which the token starts and ends.

This information gives us the line on which a token lays. That’s it. We have the newlines! A newline is just whenever a token following another token has a different line in its span. That’s as easy as it gets.

The real fix

So instead of implementing GLSL pragmas via Rust attributes, I decided to implement them the way they are defined in the spec and the glsl crate: plain GLSL. The idea was just to match a # and accumulate tokens in a collection as long as tokens lay on the same line. As soon as the line changes, we have the full pragma and we can call TokenStream::from_iter.

The current implementation expects the pragmas to be at the top part of the quasiquoted GLSL code. You shouldn’t be trying to put the #version at the bottom of your file, but I wanted you to be noticed: don’t do that!

Since glsl-quasiquote-0.2, this Rust code now completely compiles and generates, at compile-time, a GLSL AST:

let ast = glsl!{
  #version 330 core
  #extension GL_ARB_separate_shader_objects : require

  void main() {
  }
};

Future announcements about glsl and glsl-quasiquote

Some future announcements to come (things that I’ve already been working on and that I might release soon):

• The quasiquoter will have interpolation at some time.
• glsl is about to be released with an experimental SPIR-V transpiler. The current implementation uses shaderc as backend in order to test how everything goes.

This article serves as an announcement to a new crate: proc-macro-faithful-display.

Enhancing faith in displaying Rust tokens

The problem

I’ve been around procedural macros in Rust for a few weeks now – in order to write my glsl-quasiquote crate. Quickly, I discovered that most types cannot be inspected. For instance, an Ident has no methods to get the underlying identifier (as a &str for instance). The only way to do so is to allocate a String, for instance, via a Display use, as in:

let ident_str = format!("{}", ident);

Now, imagine you have a fully formatted stream of GLSL tokens flowing in. I’ve already blogged lately about the fact Rust has some specific rules that will prevent you to write preprocessor pragmas. The only way to authorize them was to implement a hack (explained in the blog article).

Lately, I was working on some not-so-trivial glsl transformations in another project when I hit a parse error. I was surprised, because the glsl crate contains more than 130 unit tests about parsing. So I thought “Well, okay, it’s alright, I might have forgotten a very specific case not covered in my nom implementation. I’ll just get a small reproducible sample and use it as a new integration test in glsl”. That kind of reasoning is very important to me: if you find a bug, first thing to do (if you plan to fix it) is to write a unit test that fails first. This is very important because when the test succeeds, you have solved your problem. If someone provides a contribution later and that the test breaks again, you will have just shielded the project against a test regression. Seriously, I know it’s boring but: write tests.

So I wrote a test, a very simple one (this is actually an integration test, since I’ve found the error in a project using glsl). And… the test passed. I was so surprised: “So that code breaks in my crate, but succeeds in the integration test suite?! Well, wait. That means that the parser is actually correct. The problem might come from the way the parser is used… or the input.”

So my second reaction was to have a look at where the parser was called. It was in a glsl! procedural macro invocation – in my project, I hardcoded a vertex shader with glsl-quasiquote. And then, I came to the realization and remembered. glsl-quasiquote, in order to parse the input Rust token, has to use Display on them, in order to yield a String usable by the glsl parser. The parser is okay – i.e. the integration test passes – but the input string… loses information.

If you have taken a look at the integration test just above, you should have noticed an interesting construction:

void main() {
  float x = 1. * .5;
}

This is not a semantically valid vertex shader, but it should parse. And it does parse. But if you write that vertex shader with glsl-quasiquote, what you actually write is this:

let vertex_shader = glsl!{
  void main() {
    float x = 1. * .5;
  }
};

The Rust tokenizer will eat the tokens and the Display implementation will render the stream by using spaces whenever it sees fit. It will not respect the initial whitespaces. It might yield something like this:

void main ( ) { float x = 1. * . 5 ; }

Look closely: our .5 GLSL floating point constant value (it equals 0.5) has become . and 5. rustc has transformed it into two tokens instead of a single one. This is due to the fact that this float format is not allowed in Rust and is interpreted the same way a method call would be.

The solution

So, after realizing that, I just went for a documentation addition stating that numbers cannot be expressed this way, and that you have to use a leading 0. I really disliked writing that documentation exception but I didn’t have any solution to that.

A few days later, I was thinking about glsl variable interpolation and got a mind click. In order to implement preprocessor pragmas, I used a trick with Span, that gave me positional information of tokens. And I came to the realization that I could use the same kind of trick to implement a function that would yield an object implementing Display by respecting the initial layout / formatting!

Basically, a trait – FaithfulDisplay – is implemented for all flavours of Rust tokens:

• Ident.
• Literal.
• Punct.
• Group.
• TokenTree.
• TokenStream.

All the implementations carefuly carry around the previous Span in order to pad enough spaces and newlines before they get formatted into the Formatter. This allows to reconstruct the input layout.

All this code is packaged in the proc-macro-faithful-display crate. You can find the documentation here. I use it – and tested against – the crate with glsl-quasiquote. It especially allowed me to remove all the preprocessor trick because now newlines are taken into account.

Is this viable?

Honestly, I have no idea. Currently, at the time of writing, it works like a charm. If you find any bug or weird behavior, please open an issue here. However, proc-macro hasn’t yet been stabilized and the Span type is accessible through a feature gate. All of this might change. However, I strongly think that:

• Either Rust will keep that Span type because it is really helpful (its first purpose was to provide nice error reporting).
• Either they’ll ditch it, but since quasiquoting is important and wanted (have a look at @bodil’s awesome typed-html crate for instance), something to replace it might come next.

Keep the vibes!

You said Rust?

Yeah, Rust. It’s a strong and static language aiming at system programming. Although it’s an imperative language, it has interesting functional conventions that caught my attention. Because I’m a haskeller and because Rust takes a lot from Haskell, learning it was a piece of cake, even though there are a few concepts I needed a few days to wrap my mind around. Having a strong C++11/14 experience, it wasn’t that hard though.

How does it compare to Haskell?

The first thing that amazed me is the fact that it’s actually not that different from Haskell! Rust has a powerful type system – not as good as Haskell’s but still – and uses immutability as a default semantic for bindings, which is great. For instance, the following is forbidden in Rust and would make rustc – the Rust compiler – freak out:

let a = "foo";
a = "bar"; // wrong; forbidden

Haskell works like that as well. However, you can introduce mutation with the mut keyword:

let mut a = "foo";
a = "bar"; // ok

Mutation should be used only when needed. In Haskell, we have the ST monad, used to introduce local mutation, or more drastically the IO monad. Under the wood, those two monads are actually almost the same type – with different warranties though.

Rust is strict by default while Haskell is lazy. That means that Rust doesn’t know the concept of memory suspensions, or thunks – even though you can create them by hand if you want to. Thus, some algorithms are easier to implement in Haskell thanks to laziness, but some others will destroy your memory if you’re not careful enough – that’s a very common problem in Haskell due to thunks piling up in your stack / heap / whatever as you do extensive lazy computations. While it’s possible to remove those thunks by optimizing a Haskell program – profiling, strictness, etc., Rust doesn’t have that problem because it gives you full access to the memory. And that’s a good thing if you need it. Rust exposes a lot of primitives to work with memory. In contrast with Haskell, it doesn’t have a garbage collector, so you have to handle memory on your own. Well, not really. Rust has several very interesting concepts to handle memory in a very nice way. For instance, objects’ memory is held by scopes – which have lifetimes. RAII is a very well known use of that concept and is important in Rust. You can glue code to your type that will be ran when an instance of that type dies, so that you can clean up memory and scarce resources.

Rust has the concept of lifetimes, used to give names to scopes and specify how long an object reference should live. This is very powerful yet a bit complex to understand in the first place.

I won’t go into comparing the two languages because it would require several articles and a lot of spare time I don’t really have. I’ll stick to what I’d like to tell you: the Rust implementation of luminance.

Porting luminance from Haskell to Rust

The first very interesting aspect of that port is the fact that it originated from a realization while refactoring some of my luminance Haskell code. Although it’s functional, stateless and type-safe, a typical use of luminance doesn’t really require laziness nor a garbage collector. And I don’t like using a tool – read language – like a bazooka. Haskell is the most powerful language ever in terms of abstraction and expressivity over speed ratio, but all of that power comes with an overhead. Even though you’ll find folks around stating that Haskell is pretty okay to code a video game, I think it will never compete with languages that are made to solve real time computations or reactive programming. And don’t get me wrong: I’m sure you can write a decent video game in Haskell – I qualify myself as a Haskeller and I’ve not been writing luminance just for the joy of writing it. However, the way I use Haskell with luminance shouldn’t require all the overhead – and profiling got me right, almost no GC was involved.

So… I looked into Rust and discovered and learned the language in only three days. I think it’s due to the fact that Rust, which is simpler than Haskell in terms of type system features and has almost everything taken from Haskell, is, to me, an imperative Haskell. It’s like having a Haskell minus a few abstraction tools – HKT (but they’ll come soon), GADTs, fundeps, kinds, constraints, etc. – plus a total control of what’s happening. And I like that. A lot. A fucking lot.

Porting luminance to Rust wasn’t hard as a Haskell codebase might map almost directly to Rust. I had to change a few things – for instance, Rust doesn’t have the concept of existential quantification as-is, which is used intensively in the Haskell version of luminance. But most Haskell modules map directly to their respective Rust modules. I changed the architecture of the files to have something clearer. I was working on loose coupling in Haskell for luminance. So I decided to directly introduce loose coupling into the Rust version. And it works like a charm.

So there are, currently, two packages available: luminance, which is the core API, exporting the whole general interface, and luminance-gl, an OpenGL 3.3 backend – though it will contain more backends as the development goes on. The idea is that you need both the dependencies to have access to luminance’s features.

I won’t say much today because I’m working on a demoscene production using luminance. I want it to be a proof that the framework is usable, works and acts as a first true example. Of course, the code will be open-source.

The documentation is not complete yet but I put some effort documenting almost everything. You’ll find both the packages here:

luminance-0.1.0

luminance-gl-0.1.0

I’ll write another article on how to use luminance as soon as possible!

Keep the vibe!

I belong to the category of people who enjoy composing tools. That is, I don’t use an IDE: I make my own IDE by combining several tools I like. In the end, we all use “IDEs” (we all need some kind of development environment). For the sake of generality, I will call those environments “DE” for development environments.

The tools I use are, basically:

• A terminal. Lately, I’ve been enjoying kitty a lot, for it allows me to do almost everything I do with tmux (read below).
• A shell. I’m using zsh mostly but I’ve been playing with others, such as nushell. I’m dissatisfied with zsh because I think it’s way more too complicated (I barely use 10% of its features).
• A text editor. I’ve been using lots lately, but mainly, vim / neovim.
• A git client. I simply use the git CLI (Command Line Interface) program, but also fugitive in neovim.
• A couple of other programs, such as fzf, CLI of various package managers and compilers, etc.
• I have plugins in neovim for note taking and task management, but I’ve also been using Notion at work lately to try it out.
• I used to use tmux a lot but I’m moving away from it, so I’m basically left with kitty’s own way to do the same thing.

The more that I think about all the things I use and all the things people enjoying modern environments use, the more I feel I try to “force” a workflow that used to be the right one a couple of years ago, but not necessarily today. I’m the kind of person who constantly put things into perspective and try to balance my decisions regarding the time I have, the budget (for work) and the knowledge. A couple of months / years ago, I spent a lot of time on emacs and realized that I needed to review my perspective, because even though emacs is super old, it’s fascinating how bleeding edge it still feels (and I can’t even imagine how bleeding edge it was twenty years ago!).

I want to use this blog article to ask questions and not necessarily answer them, but discuss about them. The questions are about “how we should be developing in the modern era” and is not about “is emacs better than neovim” or that kind of useless debates.

• Is tooling composition that any good?
  • The frozen world
  • The nightmare that is composition iteration
  • A small word on TUIs, the land of the unicode hacks
  • Summary of the problems
• What would be the (next) perfect productivity platform in 2022?
  • The terminal needs to change
  • Editing needs to come back to editing
  • Applications need to compose in a new, modern way
  • Applications need to change

Is tooling composition that any good?

The first question I’m wondering is that whether using several tools and composing them is actually good in 2022. For instance, the best example I have in mind is Kubernetes and fzf. At work, I have a workflow where I use Kubernetes and filter/select its output via fzf to run other commands. I do the same with git, btw. I have some scripts where I can do something like:

gr -s # equivalent to git branch | fzf | git rebase origin --auto-stash
gs # equivalent to git branch | fzf | git switch
gm # equivalent to git branch | fzf | git merge
kubectx # switch via FZF k8s clusters (kubectl command)
kubens # same but with k8s namespaces
# etc. etc.

The frozen world

All of that seems to be pretty good. We can leverage the power of small tools such as git, kubectl and fzf and compose them. Shell pipes allow to connect applications together, reading and writing from and to stdin and stdout.

The “main platform”, the “DE” is then the terminal and the shell, and building a DE requires two main ingredients:

• A set of programs, each of them doing one thing right. fzf is excellent at building a list of items and let the user pick one of them, for instance. kubectl allows to operate Kubernetes resources one operation by one operation.
• Some “glue” to combine the results of the programs. This is typically the shell, with its pipes and output pushed to the terminal.

However, I think that even though all of that seems pretty great, it suffers from a major drawback: everything is text and everything is static. See, all of the programs I mentioned here are CLI, not TUI (Text User Interface). That is, the UX is the following:

1. The user enters a command in their terminal, starting with the program name, passing arguments.
2. The program runs, might have side effects on remote resources, etc.
3. The program finishes and displays its results.

That way of working has been a default in the industry for decades. If the program printed more than one terminal page, you will have to scroll to find your information back up. Once you find the information you need, how do you pick that information? If you are using a terminal that doesn’t support hinting, you will have to use your mouse (which completely defeats the purpose of using CLI / TUI applications to me) or the keyboard-base navigation system of your terminal to make selections, which is, let’s be honest, really poor in most terminal. If, like me, you use a terminal supporting hinting (i.e. being able to enter a mode that will allow you to visually select parts of the screen by simply typing a few characters — similar to my hop.nvim plugin, but for the terminal), you will be able to copy parts of the screen, but that’s all. If a program returned a JSON array, it will cost you a lot of keystrokes to reduce the list and, for instance, use an item in the array as input of another curl request, for instance.

The nushell has a very interesting concept to solve that kind of problem. Programs write lines of text as output, but they can also write cells, which is a nushell encoding that allows it to “speak the same language.” It comes with some built-in programs such as open that will be able to recognize a lot of formats (JSON, XML, YAML, TOML, etc.) and convert them into cells, allowing to display them in the terminal in a very convenient and “smart” way. However, it doesn’t change the next problem: composition iteration.

The nightmare that is composition iteration

I think pretty much every programmer out there can relate to the following: you will oftentimes run a command, see its result, then run the command again, piping it to a filter command or sort or field extractor like [jq]. If the output is too long, it’s likely you will output into a file and will work on that file (which is basically a cached version of your command output ;) ).

Iterating on a “session” is a pretty bad experience in a terminal to me. To my knowledge, it would be quite easy to fix by simply allowing to automatically cache the result of all commands, and get access to it via a shell variable (such as what we have with $? for the last command status, for instance). Some hacks exist to recompute the last command (i.e. $(!!)) but it will still re-run the last command.

I thought nushell supported that, because the variable support and Nu language are pretty excellent, but to my surprise, that shell doesn’t have that. Maybe it’s a technical challenge, but I really doubt it. We work with computers with at least 8 GB of RAM these days and most of us have between 16 and 32 GB of RAM. I don’t think it would be that hard to support.

But even if we had that caching… it would still require us to run a command, look at a static text, run a filter command on it, look at the new output that would be appended to the screen… it doesn’t seem super nice in terms of UX to me. And imagine having to change the second or third command in the pipe list while you already have six pipes. Duh. The problem, to me, is that the terminal application scope is being overloaded. The CLI is great when we want to run a command and expect a report. As soon as we need to work with data, I think the CLI is already lacking features.

A small word on TUIs, the land of the unicode hacks

And now, let’s talk about TUIs. As a lot of people, I’m using TUIs, so don’t get what I’m about to say twisted. I use neovim as my daily editor driver, and so far, I haven’t really found a way out and use something else. However, it’s not because nothing better exists that what we use is perfect (far from that).

I have always despised TUIs. I think the idea, which dates from a while, was a fun and interesting idea back then, when graphical elements and graphics kits were not mature and not a thing yet. But today, we know we can have graphical elements that align at the pixel level. We know that we can have real vector images in applications. We can use the GPU to draw arbitrary shapes and some terminals (text-based only!) use the GPU to render glyphs (text), and then people use TUIs that will use the glyphs to mimic graphical elements! That hitches me!

I dislike TUIs because they are sub-optimal versions of GUIs. And here, I think a lot of people make a confusion with UX and UI. A GUI can be beautiful, rich of graphical elements, and have a terrible UX (it’s the case most of the time). People writing GUIs tend to focus on mouse-driven interactions… which doesn’t have to be. So people tend to associate GUIs with mouse-only workflows, while a GUI can completely do everything a TUI does… but it can also do much better.

A couple of arguments against TUIs:

• Graphical elements are faked by using unicode symbols (drawing blocks), which implies having all the elements laid out on a display cell grid. Oftentimes, the unicode symbols will not be correctly aligned, resulting in graphical glitches, or simply ugly pop-ups borders, etc.
• Also, still about unicode glyphs, if the application doesn’t implement it correctly, that text is just “part of the text of the application”, so copy-pasting can be challenging and very boring.
• Since everything has to be put on a display cell grid, you have no way to have pixel-based movements, alignment or anything based on pixel. The smallest unit, the basic primitive, is a unicode character. About that, I think the most interesting “fake” thing is wfxr/minimap.vim, which is a minimap for neovim in a terminal. And you can’t do any better than that, because, well, you can’t go any smaller / different than a glyph.
• Applications run in a terminal, which drives the performance of the TUI application. That results in non-deterministic experiences. Basically, you can write a super great application that will run like an elder snail on some terminal emulators.

Summary of the problems

So let’s do a quick recap of what’s wrong with today terminals:

• TUIs are not rich enough in terms of graphical experience, and everything graphical (splits, lines, pop-ups, etc.) are somehow faked.
• CLI-based composition often ends up being a nightmare of running the same command over and over by adding more piped commands.
• A terminal is a static world, where (besides TUIs) everything is just static text and nothing changes unless you run a command again, wiping the previous result and replacing it with the new one.

Those are the three main pain points I feel about using a terminal. Now, you might already have used something like a monitor application (I work at DataDog, so I’m used to them) or used a web browser or native GUI and see that you can have animations, drop-down lists that actually look good, etc. Putting all that in contrast with terminals makes me a bit sad, especially in 2022.

What would be the (next) perfect productivity platform in 2022?

My latest blog articles are centered around that topic:

• My thoughts about editors in 2020
• My thoughts about editors in (early) 2021

So it’s been an important topic to me lately. I have been thinking about productivity platforms for a while now, and the conclusion I came to is as follows.

The terminal needs to change

Instead of switching to an IDE like IntelliJ or VS Code, I still think we need composition, but not in the form of pipe shells. I think the CLI has to evolve. I have started a project on my spare-time to try and change how programs would work with their environment. See, a shell is basically just a way to call the exec syscall (execve on Linux for instance) and set its arguments, environment and standard input / output (plus all the candies you can get from a shell, such as scripting languages, etc.).

The interpretation of the inputs and outputs of a program is actually completely left to… the running process, which is most of the time the shell for CLI applications. This is why nushell is such an interesting new shell: it can interpret more than just lines of text. And I think we need to move in a direction similar to this.

I think we should see more alternative forms of “terminals”. We could even imagine a program that doesn’t have the concept of “shell” nor “terminal” anymore. It’s easy to imagine having an application that can simply call execve and do whatever it wants the environment, inputs and outputs. It could have some primitives implemented. For instance, imagine that a program could be run in your “new terminal” and output its results in a tree view, while other programs are also running, displaying their own things. You could have a program run without exiting, updating the content of the tree view or even accepting filters, sorts, field extractor or whatever see fit.

Editing needs to come back to editing

Today, text editors are overloaded. Heck, even small editors such as neovim now have a community developing plugins that have nothing to do with editing. For instance, fuzzy searching files or Git management. When you think about it, if the platform (i.e. terminal) had better primitives, it should be possible to compose programs in a way that doesn’t require people to rewrite the same kind of logic (i.e. fuzzy finders) in all the programs they use. fzf is a good example of that, but it’s limited to the CLI (you can still use it in TUI but to me it’s a bit hacky). Terminals should have a primitive to, for instance, run a program in overlay and pass the output to another, running program. That would allow people to write “pure generic fuzzy finders” once and for all, and let people use the editors they like.

Today, all that logic doesn’t really exist outside of individual and scoped efforts. Yes, you can probably find programs supporting that, but they will certainly not run in a terminal and will have their own protocols.

Applications need to compose in a new, modern way

Composing applications with pipes is easy to wrap your fingers around, but it gets limited and limiting as soon as you use something like a GUI. Imagine that programs could speak a common language, based on, for instance, local sockets, to exchange messages in a way that whatever the kind of programs you write, you can still compose them. Composing two GUIs between each other should be something that is widely supported, and standardized.

Something else with piping applications: it’s limited to how we use CLI applications, which is via the command line. Running programs in the command line takes the form of:

program <arg1> <arg2> … | another_program <arg1> <arg2>

This is highly summarized, but you get the idea. CLI programs don’t provide a great discoverability experience, and it’s harder to iterate on them. For instance, in a GUI, if you filter a list with a predicate, it’s super easy to “experiment around” by changing the filter name. In a CLI session with piped programs, you are likely required to use your up arrow on your keyboard (or whatever shortcut) to invoke the last command again, and change the arguments in the middle of the line. While doing this, you will still have to run all intermediary commands, which are going to perform side-effects, etc. etc. And if you want to be smarter and use the > or >> operator to cache the results, then you will still have to remember the name of the file you used, and cat or tee that file again. It gets time to get used to using that, and I truly think that it’s not really worth it, because today I’m 30 and I’ve been developing since I’m 11, and I still don’t really enjoy that kind of process.

Applications need to change

The essence of an application, today, is actually platform-dependent. Heck, it’s even environment dependent. At work, an “application” can be as simple as a small Python script or be a Kubernetes cluster composed of several deployments, stateful sets and DNS rules. However, when we think about applications we use on a daily basis (what people would call “heavy” or “native” applications), they are pretty similar to the following:

• For CLI applications, they are akin to effectful computations. They take inputs (environment, stdin, arguments), they output something (stdout) and they can have side-effects (calling another program, changing shared memory, doing stuff on your file system, various kinds of I/O, etc.).
• For TUI applications, it’s similar but they don’t read from stdin nor output to stdout in a one-shot way. Instead, it’s like an event loop that reads from stdin and a render loop that writes to stdout.
• For GUI, it depends on the platform but most of the time, GUIs will be closer to the hardware, because they are not restricted by the protocol they are written for (i.e. CLIs / TUIs cannot really go any lower stdin and stdout in terms of UX; GUIs can). A GUI application can decide to use OpenGL, Vulkan or something higher such as Gtk, Qt, or a Web kit.

I think that we should change that. Remember that running an application is just a matter of calling execve (or similar on non-Linux), which basically requires a path to the application, the list of arguments (the argv) and the environment (envp). Setting the environment is typically done by the shell (and some variables come from the terminal too), but it could be done completely differently. On the same idea, the argv is filled by the CLI arguments you provide, but they could be computed and filled in a completely different way. And finally, the interpretation of stdout (mostly done by the terminal) could be interpreted in a completely different way as well.

Imagine that we could write to stdout the port / socket on which we would like to open a communication with a new, modern protocol, that would allow us to manipulate a new kind of “terminal.” Instead of using a regular text-based terminal, it would be a piece of software allowing to manipulate pop-ups, text buffers, image / vector image buffers, notification queues, etc. etc.

Obviously, I’m not inventing all that. If you know about emacs, you know that that’s basically what they do, but instead of using stdin, stdout or execve, they just evaluate Elisp. However, what I think could be very interesting would be to allow native applications to do the same thing, with a native platform that would ship without any real applications (so unlike emacs), and let applications implement their own workflow.

That’s basically what I’ve been working on and experimenting lately. A platform that provides buffers, tree views, notifications, pop-pups, overlays, etc. to allow any kind of applications that can read and write to sockets to use those features as if they were native, and platform-independent, in the same way that you use stdout without even thinking how it will be displayed in the terminal.

Leveraging such a platform would solve most of the problems I mentioned above, because:

• We could write a code / text editor that wouldn’t have to be worried about extensions like Git: the Git support would come as a native application (equivalent of the CLI git application) that would open an overlay, for instance. That application wouldn’t even know about the code editor: you would make that connection via the platform.
• Applications could communicate between each other to manipulate buffers, get the contents of the buffer and do various kind of operations on them (LSP servers, for instance).
• Because the protocol would be completely rendering-agnostic, that would allow people to write an application that would work for any kind of “platform implementations.”
• Composition would use the low-level protocol to compose applications. You could imagine supporting an interface between applications allowing to filter the contents of the buffer of one by using another application. Applications such as fzf could branch on the buffers of a text editor, for instance. Or you could even open a buffer in the text editor by running a fuzzy finder as another application. All of that would compose in a seamless way.
• Etc. etc.

Eh, of course, I know that such a platform protocol will probably never appear unless I make it, very opinionated and subject to never be adopted by anyone else but me (or a small group of enthusiasts about that kind of ideas). Maybe this blog article would have given you some ideas or hindsight about your own tools. Or maybe someone knows such a platform (that I don’t) and will make me realize it already exists, or is a work in progress. Maybe, after all, it will just launch a discussion about what developing means in 2022. Because to me, using tools that used to be efficient and proved to be great ones a couple of years ago, doesn’t mean they are still a thing today. And I really mean it, because I’m currently writing this very article in neovim, in a kitty tab among dozens, and I just wish I had the DE I just described above.

Keep the vibe!

Reddit discussion

• The context
• The problem
• Faking higher-kinded types
  • A simple implementor
  • Polymorphic use
• Rationale
• Conclusion

The context

Imagine you want to expose a trait that defines an interface. People know about the interface and can use provided types to switch the implementation. That’s a typical use case of traits.

// This trait is pub so that users can see and use it
pub trait System {
  type Err;

  fn do_something(&mut self) -> Result<(), Self::Err>;
}

You can have a type SimpleSystem that implements System and that will do something special in do_something, as well as a type ComplexSystem doing something completely different.

Now imagine that your System trait needs to expose a function that returns an object that must be typed. What it means is that such an object’s type is tagged with another type. Imagine a type Event that is typed with the type of event it represents:

pub struct Event<T> {
  code: i64,
  msg: String,
  // …
}

That type must be provided and implemented by systems, not by the actual interface code. How would we do this?

Furthermore, we might want another trait to restrict what T can be, but it’s off topic for our current problem here.

The problem

Let’s try a naive implementation first:

pub trait System {
  type Event;

  type Err;

  fn do_something(&mut self) -> Result<(), Self::Err>;

  fn emit_event(&mut self, event: Self::Event);
}

That implementation allows SimpleSystem to implement Event as a single type and then implement emit_event, taking its Event type. However, that event is not typed as we wanted to. We want Event<T>, not Event.

However, Rust doesn’t authorize that per-se. The following is currently illegal in Rust:

pub trait System {
  type Event<T>;

  // …

  fn emit_event<T>(&mut self, event: Self::Event<T>);
}

RFC 1598 is ongoing and will allow that, but until then, we need to come up with a solution.

The problem is that associated types, in Rust, are completely monomorphized when the trait impl is monomorphized, which is, for instance, not the case for trait’s functions. emit_event will not have its F type variable substituted when the implementor is monomorphized — it will be sank to a type when it’s called. So what do we really want to express with our Event<T> type?

pub trait System<EventType> {
  type Event;

  fn emit_event(&mut self, event: Self::Event);
}

That would work but that’s not exactly the same thing as our previous trait. The rest of the trait doesn’t really depend on EventType, so we would duplicate code every time we want to support a new type of event. Meh.

Faking higher-kinded types

Event<T> is a higher-kinded type (HKT). Event, here, is what we call a type constructor — as opposed to data constuctor, like associated new functions. Event takes a type and returns a type, in the type-system world.

As I said earlier, Rust doesn’t have such a construct yet. However, we can emulate it with a nice hack I came across while trying to simulate kinds.

See, when a HKT has its type(s) variable(s) substituted, it becomes a regular, monomorphized type. A Vec<T>, when considered as Vec<u32>, is just a simple type the same way u32 is one. The key is to decompose our trait into two distinct yet entangled interfaces:

• One that will work with the monomorphized version after the trait is monomorphized.
• One that will introduce polymorphism via a method, and not an associated type.

The first trait is implemented on systems and means “[a type] knows how to handle an event of a given type.” The second trait is implemented on systems, too, and means “can handle all events.”

pub trait SystemEvent<T>: System {
  type Event;

  fn emit_system_event(&mut self, event: Self::Event) -> Result<(), Self::Err>;
}

pub trait System {
  type Err;

  fn do_something(&mut self) -> Result<(), Self::Err>;

  fn emit_event<T>(
    &mut self,
    event: <Self as SystemEvent<T>>::Event
  ) -> Result<(), Self::Err>
  where Self: SystemEvent<T> {
    <Self as SystemEvent<T>>::emit_system_event(self, event)
  }
}

The idea is that the SystemEvent<T> trait must be implemented by a system to be able to emit events of type T inside the system itself. Because the Event type is associated in SystemEvent<T>, it is the monomorphized version of the polymorphic type in the actual implementation of the trait, and is provided by the implementation.

Then, System::emit_event can now have a T type variable representing that Event<T> we wanted. We use a special type-system candy of Rust here: Self: Trait, which allows us to state that in order to use that emit_event<T> function, the implementor of System must also satisfy SystemEvent<T>. Even better: because we already have the implementation of emit_system_event, we can blanket-implement emit_event<T>!

Several important things to notice here:

• First, a system can now implement several traits:
  • System, to work as a system, obviously.
  • SystemEvent<T>, to handle events of type T. The actual type is defined by the system itself.
• Second, a system doesn’t have to implement SystemEvent<T> if it doesn’t know how to handle events of type T. That opt-in property is interesting for us as it allows systems to implement only what they support without having to make our design bleed to all systems. Neat.
• The usage of Self: Trait allows to introduce a form of polymorphism that is not available without it.
• Because we can blanket-implement emit_event, the implementor of System doesn’t even have to implement emit_event.
• SystemEvent<T> has System has super-trait to share its Err type.

The last point is the hack key. Without Self: Trait, it would be way harder or more convoluted to achieve the same result.

A simple implementor

Let’s see a simple implementor that will just print out the event it gets and does nothing in do_something.

struct Simple;

#[derive(Debug)]
struct ForwardEvent<T>(pub T);

impl System for Simple {
  type Err = ();

  fn do_something(&mut self) -> Result<(), Self::Err> {
    Ok(())
  }
}

impl<T> SystemEvent<T> for Simple where T: std::fmt::Debug {
  type Event = ForwardEvent<T>;

  fn emit_system_event(&mut self, event: Self::Event) -> Result<(), Self::Err> {
    println!("emit: {:?}", event.0);
    Ok(())
  }
}

As you can see, it’s really simple and straight-forward. We can select which events we want to be able to handle: in our case, anything that implements Debug.

Let’s use it:

fn main() {
  let mut system = Simple;
  system.emit_event(ForwardEvent("Hello, world!"));
  system.emit_event(ForwardEvent(123));
}

Polymorphic use

The idea is that, given a type S: System, we might want to emit some events without knowing the implementation. To make things even more crunchy, let’s say we want the error type to be a ().

Again, it’s quite simple to do:

// we need this to forward the event from the outer world into the system so that we don’t have to
// know the actual type of event used by the implementation
impl<T> From<T> for ForwardEvent<T> {
  fn from(t: T) -> Self {
    ForwardEvent(t)
  }
}

fn emit_specific_event<S, E>(
  system: &mut S,
  event: E
) -> Result<(), S::Err>
where
  S: System<Err = ()> + SystemEvent<E>,
  S::Event: From<E>,
{
  system.emit_event(event.into())
}

fn main() {
  let mut system = Simple;
  system.emit_event(ForwardEvent("Hello, world!"));
  emit_specific_event(&mut system, "Hello, world!");
}

We could event change the signature of System::emit_event to add that From<E> constraint to make the first call easier, but I’ll leave you with that. The important aspect of this code snippet is the fact that the implementor will handle a type of event Event<T> while the interface uses T directly. We have injected a HKT Event.

Rationale

Why do I care about such a design? It might seem complex and hard, but it’s actually a very useful use of a typeclass / trait type system — especially in luminance, where I use type-system concepts a lot; after all, it’s based on its Haskell version! If you compare my solution to the next-to-come HKT proposal from RFC 1598, we have:

• RFC 1598 will bring the syntax type Event<T>; as associated types, which will allow to remove our SystemEvent<T> trait.
• With that RFC, the type will have to be total. What it means is that the T here is the same for all implementors: an implementor cannot restrict T, it must be implemented for all of them. If you want to restrict the type of events your type can receive, if I understand the RFC correctly, you’re handcuffed. That RFC will authorize the author of the trait to restrict your T, not the author of the implementor. That might have some very useful advantages, too.
• With my solution, the author of the trait can still restrict T by applying constraints on emit_event.

Conclusion

In the quest of emulating HKT, I’ve found myself with a type-system toy I like using a lot: equality constraints and self-constraints (I don’t know how those Self: Trait should be named). In Haskell, since we don’t implement a typeclass for a type but we provide an instance for a typeclass, things are slightly different. Haskell doesn’t have a Self type alias, since a typeclasses can be implemented with several types variables (i.e. with the MultiParamTypeClasses and FlexibleInstances GHC extensions), only equality constraints are needed.

In the end, Rust continues to prove that even though it’s a systems programming language, I can express lots of powerful abstractions I miss (a lot) from Haskell with a bit more noise. I think the tradeoff is worth it.

I still haven’t completely made up my mind about GAT / RFC 1598 (for sure I’m among the ones who want it on stable ASAP but I’m yet to figure out exactly how it’s going to change my codebases).

As always, have fun, don’t drink and drive, use condoms, keep the vibes and don’t use impl Trait in argument position. Have fun!

So here’s a quick article I can link in the super fresh TWiN, website I released a couple of days ago and that I hope will help with that kind of issues.

What can you do with Hop

Hop has been around for some time now. I created it back in February 2021, so roughly one year and a half. It has changed and evolved quite a lot ever since. New hint algorithm, new commands, a very big number of new options and features. I really recommand people to read the embedded documentation of Hop. Yes, completely. It took me time to write in a way that it will take you a couple of minutes only to read. So yes, do read it. Get the habit to read the embedded documentation of the plugins you install, especially after new updates. Or read TWiN from now on 😁.

The wiki, the wiki, the wiki!

Yes, Hop has a wiki. It’s available here and you should read a couple of pages from it:

• The “default” commands.
• Mastering advanced Hop. This part contains a couple of examples showing you how to use the various Hop Lua functions and options to create a really advanced motion platform.
• All the configuration options.

Because I know some people will not click on those links and will still open PRs trying to implement features that are already there, here is a non-exhaustive list of advanced things you can already do, out of the box, with Hop.

Recreate f, F, t and T

It’s in the wiki and super simple. For that, you just need three options:

• direction, which must be set with require'hop'.HintDirection.BEFORE_CURSOR or require'hop'.HintDirection.AFTER_CURSOR.
• current_line_only, a boolean.
• hint_offset, a super power allowing you to offset the actual jump off the jump target by a given number of characters.

So f is basically:

require'hop'.hint_char1({
  direction = require'hop.hint'.HintDirection.AFTER_CURSOR,
  current_line_only = true
})

F is then:

require'hop'.hint_char1({
  direction = require'hop.hint'.HintDirection.BEFORE_CURSOR,
  current_line_only = true
})

And in the same idea, t is:

require'hop'.hint_char1({
  direction = require'hop.hint'.HintDirection.AFTER_CURSOR,
  current_line_only = true,
  hint_offset = -1
})

And T:

require'hop'.hint_char1({
  direction = require'hop.hint'.HintDirection.BEFORE_CURSOR,
  current_line_only = true,
  hint_offset = -1
})

Jump at the end of things

Hop already can already understand the concept of beginning and end of a jump target. It does not make sense with all of them, but for most, it can. That is implemented with the hint_position option, which has to be one of:

• require'hop.hint'.HintPosition.BEGIN
• require'hop.hint'.HintPosition.MIDDLE
• require'hop.hint'.HintPosition.END

So if you want to jump to the end of words:

require'hop'.words({
  hint_position = require'hop.hint'.HintPosition.END,
})

You can even jump right after the end of a word!

require'hop'.words({
  hint_position = require'hop.hint'.HintPosition.END,
  hint_offset = 1
})

Callbacks!

You can even use Hop to do something with the jump target that doesn’t imply jumping, like running a Lua function with the actual column and line where the hint is. It’s done with require'hop'.hint_with_callback.

Here, no example but an exercise for you: try to use require'hop'.hint_with_callback to send a notification (via vim.notify) displaying the jump target chosen by the user by hinting words. Tips: you will need to find the jump target generator for words.

Final recommendation

Please, feel free to read :h hop. It contains everything I just explained and so much more, like Hop extensions, multi windows, etc. Have fun hopping around!

1. The first commit of luminance was made in 2016, as I migrated it away from its Haskell version, which first commit was made in 2015. Since then, my knowledge of programming, both Haskell and Rust, but also in general has enhanced quite a lot.
2. The current design has some flaws, both in terms of usability and maintainability.

About the first point, I had a look at code that barely changed since I wrote it in 2016, and I’m not completely satisfied with it anymore. Lots of things happened ever since: I have a sharper vision of how things should be done, Rust itself has changed quite a lot (back in 2016, Rust 1.0 was just released!), we have more type system features that, back then, I was lacking, etc.

About the second point, some issues in the current version of luminance (v0.47 as writing this blog article) are complex to solve, such as the Drop problem. Other parts of the API are insanely complex, like the TessBuilder related code.

I want all that to change, so I started working on a completely new graphics crate, just to see what kind of interface I could come up with. And I came up with something much, much easier. So I faced a dilemma: should I release that other crate and give up on luminance… or should I just port all the new knowledge and ideas from that crate into luminance?

Well, I have decided to keep luminance around and update it. It is going to take time, because luminance is an ecosystem of many crates, it has lots of examples (which is great to ensure that its features are still working correctly once I start redesign it) and because some features are obviously entangled. However, I think it’s worth it, because even though luminance is not as famous or widely used as wgpu, it has a special place in the Rust Graphics community.

So I’m going to blog about the redesign of luminance and it’s going to be a blog article series. I wanted to start with a very new exciting feature that I have been wanting for a very long time: compatible vertex types.

The vertex compatibility problem

When you’re like me and are obsessed with tracking as much information as possible at compile-time, you start imagining lots of features and ways to prevent people from doing things. Preventing and forbidding is actually much more complex than allowing something. Think about it like this: assembly allows for everything and is not a complex language. It has zero constructs, besides opcodes. However, as you add more constructs and abstractions, those abstractions restricts the application domain; they forbid usage, in a constrained way. There’s a reason why, in Haskell, we use the term Contraint — which is called, basically, trait bound in Rust.

The way luminance allows a programmer to render objects, which must have a vertex type V that must implement Vertex. luminance-derive eases the process of implementing lots of traits from luminance. For instance:

#[repr(C)]
#[derive(Clone, Debug, Vertex)] // < notice the Vertex derive here
pub struct MyVertex {
  position: Vector3<f32>,
  normal: Vector3<f32>,
  color: Vector3<u8>,
  weight: Vector3<f32>,
}

This is great, because the user just has to focus on writing types and just using them. For vertex objects, which can be imagined as big arrays / buffers of MyVertex here, there are types and functions allowing to basically render such object, and those render functions expects MyVertex. However, rendering is a multi-stage process. As you may know, the graphics pipeline contains different steps, and among the first ones, you have a vertex shader, which is responsible in mapping vertices (to transform them, for instance). In luminance, vertex shaders are strongly-typed with the kind of vertices they expect. That is to prevent a user from using a vertex shader that expects some vertex attributes that a vertex object doesn’t have. Imagine that we write a vertex shader that expects a position and a normal but our vertex object, which uses OtherVertex vertices, only has position. That would result in probably some visual glitches or just a black screen.

So, we want to have something like this pseudocode:

type VertexShader<V>; // accept only vertex type V
type RenderVertexObjects<W>; // render only vertex type W

If we want to have a fully typed graphics pipeline, we must have V = W. However, that is going to be very annoying. Imagine a vertex shader that only needs position, normal and color. Something like:

#[repr(C)]
#[derive(Clone, Debug, Vertex)]
pub struct ShaderVertexInput {
  position: Vector3<f32>,
  normal: Vector3<f32>,
  color: Vector3<u8>,
}

This is a different type than MyVertex, so if we try to render our MyVertex vertex objects with a vertex shader that works on ShaderVertexInput, it will get a type mismatch at compile-time. The only way to fix that is to create a copy of the shader program that works with MyVertex and ignore the weight field. Meh. Waste of resources and duplication. Everything we don’t want.

When you look at it, the vertex shader is going to use position, normal and color, but it doesn’t have to know there are other vertex attributes. So it should be able, in theory, to work with MyVertex, since MyVertex has position, normal and color…

The solution

One solution to fix the problem is to introduce a trait, CompatibleVertex, that we will use on the vertex shader code. The vertex shader still has its ShaderVertexInput type, but it will not require vertex objects to use that type. Instead, it will require vertex objects to use V and will require ShaderVertexInput: CompatibleVertex<V>. What it means is that ShaderVertexInput must be included in V. Said otherwise, V must have all of ShaderVertexInput’s fields (but it can have more).

pub trait CompatibleVertex<V> {}

Now the big question: how do we implement CompatibleVertex for a given concrete vertex type? Well, in the previous version of luminance, that used to be implemented manually by users of luminance. It was both unsafe and not really practical, so I just removed the concept (and luminance in version v0.47 and less is subject to the problem I described above, where you can render a vertex object which vertex type is not compatible with what the shader expects).

Const generics to the rescue

Since rustc-1.51, const generics can be used at compile-time. The feature allows to manipulate integers at compile-time. For instance:

pub struct Array<const N: usize> {
  // …
}

Here, N is a const generics, and you use it with types like Array<3> or Array<14>. However, that’s currently (rust-1.63) all you can do with a stable compiler… but we can do much more with a nightly compiler.

There is a feature called adt_const_params that allows using more types, and one type that is very important is &'static str. Yes, you heard right. Constant strings. We can write something like this:

#![allow(incomplete_features)] // as of rustc-1.65 nightly, this is still required
#![feature(adt_const_params)]

fn hello<const N: &'static str>() {
  println!("Hello {}!", N);
}

fn main() {
  hello::<"world">();
}

That doesn’t seem like much, but this small change is going to save so many frustration in luminance, and fix our compatible vertex problems.

Applying adt_const_params to CompatibleVertex

What we basically want for one vertex type to compatible with another (i.e. one is included into the other) is to ensure that the all the fields of one are present in the other. Let’s recall our vertex type definitions and trait:

#[repr(C)]
#[derive(Clone, Debug, Vertex)]
pub struct MyVertex {
  position: Vector3<f32>,
  normal: Vector3<f32>,
  color: Vector3<u8>,
  weight: Vector3<f32>,
}

#[repr(C)]
#[derive(Clone, Debug, Vertex)]
pub struct ShaderVertexInput {
  position: Vector3<f32>,
  normal: Vector3<f32>,
  color: Vector3<u8>,
}

pub trait CompatibleVertex<V> {}

We can write a HasField trait like this:

pub trait HasField<const NAME: &'static str> {
  type FieldType;
}

And now, we can implement that trait for all the fields of our vertex types:

// for MyVertex
impl HasField<"position"> for MyVertex {
  type FieldType = Vector3<f32>;
}

impl HasField<"normal"> for MyVertex {
  type FieldType = Vector3<f32>;
}

impl HasField<"color"> for MyVertex {
  type FieldType = Vector3<u8>;
}

impl HasField<"weight"> for MyVertex {
  type FieldType = f32;
}

// for ShaderInputVertex
impl HasField<"position"> for ShaderInputVertex {
  type FieldType = Vector3<f32>;
}

impl HasField<"normal"> for ShaderInputVertex {
  type FieldType = Vector3<f32>;
}

impl HasField<"color"> for ShaderInputVertex {
  type FieldType = Vector3<u8>;
}

Now, we can write a more meaningful implementor for CompatibleVertex and ShaderInputVertex:

impl<V> CompatibleVertex<V> for ShaderInputVertex where
  V: HasField<"position", FieldType = Vector3<f32>>
    + HasField<"normal", FieldType = Vector3<f32>>
    + HasField<"color", FieldType = Vector3<u8>>
{
}

Because MyVertex implements the three HasField trait bounds required, it can be used, as well as ShaderInputVertex.

Obviously, because luminance-derive is a thing, people will never have to write any of those implementors. Actually, deriving Vertex will do two things (besides what it already does) for a given vertex type:

• Provide all the HasField implementors for all of its field.
• Provide the universal CompatibleVertex implementor.

What it means is that, as soon as you start using #[derive(Vertex)], you will automatically have compatible vertex types behind the scenes working for you.

Conclusion

That feature is already, by itself, worth requiring a nightly compiler, as it brings the exact kind of type safety I want for my code. And since I started with const generics strings, the next blog article will be about the redesign of vertex objects (called Tess in the current version of luminance), since, you might have guessed, it will use const generics strings to upload vertex data per-attribute, and map / update them.

Thanks for having read, and keep the vibe!

You can’t even imagine how hard it was to release luminance-0.7. I came accross several difficulties I had to spend a lot of time on but finally, here it is. I made a lot of changes for that very special release, and I have a lot to say about it!

Overview

As for all my projects, I always provide people with a changelog. The 0.7 release is a major release (read as: it was a major increment). I think it’s good to tell people what’s new, but it should be mandatory to warn them about what has changed so that they can directly jump to their code and spot the uses of the deprecated / changed interface.

Anyway, you’ll find patch, minor and major changes in luminance-0.7. I’ll describe them in order.

Patch changes

Internal architecture and debugging

A lot of code was reviewed internally. You don’t have to worry about that. However, there’s a new cool thing that was added internally. It could have been marked as a minor change but it’s not supposed to be used by common people – you can use it via a flag if you use cabal or stack though. It’s about debugging the OpenGL part used in luminance. You shouldn’t have to use it but it could be interesting if you spot a bug someday. Anyway, you can enable it with the flag debug-gl.

Uniform Block / Uniform Buffer Objects

The UBO system was buggy and was fixed. You might experience issue with them though. I spotted a bug and reported it – you can find the bug report here. That bug is not Haskell related and is related to the i915 Intel driver.

Minor changes

The minor changes were the most important part of luminance-0.7. luminance now officially supports OpenGL 3.2! When installing luminance, you default to the gl32 backend. You can select the backend you want with flags – gl45 and gl45-bindless-textures – but keep in mind you need the appropriate hardware to be able to use it. Because you need to use flags, you won’t be able to switch to the backend you want at runtime – that’s not the purpose of such a change though.

The performance gap should be minor between gl32 and gl45 but still. Basically, OpenGL 4.5 adds the support for DSA, which is very handy and less ill-designed that previous iterations of OpenGL. So a lot of code had to be rewritten to implement luminance’s stateless interface without breaking performance nor bring them down.

I might add support for other backends later on – like an OpenGL ES backend and WebGL one – but that won’t ship that soon though because I have a ton of work to do, and yet need to provide you with a concrete, beautiful, fast, appealing and eye-blowing demo with luminance! ;)

Feel free to test the gl32 backend and give me back feedback!

However, if I spent so much time on that 0.7 version, it’s because I had issue whilst writing the gl32 backend. Indeed, I spotted several bugs on my Intel HD card. This is my OpenGL version string for my Intel IGP card:

OpenGL core profile version string: 3.3 (Core Profile) Mesa 11.0.4

The architecture is Haswell. And on such a card (i915 linux driver) I’ve found two bugs while trying the gl32 backend with luminance-samples-0.7.

usampler2D

For unknown reason, the Texture sample failed on my Intel IGP but ran smoothly and flawlessly on my nVidia GPU. I spent a lot of time trying to figure out what I was missing, but eventually changed the sampler type – it’s now a sampler2D – and… it worked. I reported the issue to the intel dev team. So if you hit that error too, please leave a message here so that I can have more hindsight about that error and see what I can do.

Uniform block and vec3

This is a very nasty issue that kept me awoken for days trying to fix my code while it was a driver bug. It’s a big technical, so I’ll just leave a link to the bug tracker so that you can read it if you want to.

Breaking changes

Ok, let’s talk.

When creating a new shader stage, you now have to use the function createStage – instead of several functions like createVertexShader. That change is important so that I can add new shader types without changing the interface, and because some shader can fail to be created. For instance, on the gl32 backend, trying to build a tessellation shader will raise an error.

When a shader stage creation fails, the UnsupportedStage error is raised and holds the type of the stage that failed.

Finally, the interface for the cubemaps changed a bit – you don’t have access to width and height anymore, that was error-prone and useless; you’re stick to a size parameter.

I’d like to thank all people supporting me and luminance. I’ll be watching reactions to that major and important release as it will cover more people with cheaper and well-spread GPUs.

Happy hacking! :)

Although I didn’t add things nor write code at home, I thought a lot about graphics API designs.

Graphics API designs

Pure API

APIs such as lambdacube or GPipe are known to be graphics pure API. That means you don’t have use functions bound to IO. You use some kind of free monads or an AST to represent the code that will run on the target GPU. That pure design brings numerous advantages to us:

• it’s possible to write the GPU code in a declarative, free and elegant way;
• because of not being IO-bound, side-effects are reduced, which is good and improves the code safety;
• one can write GPU code and several interpreters or/and with different technologies, hence a loosely coupled graphics API;
• we could even imagine serializing GPU code to send it through network, store it in a file or whatever.

Those advantages are nice, but there are also drawbacks:

• acquiring resources might not be explicit anymore and none knows exactly when sparce resources will be loaded into memory; even though we could use warmup to solve that, warmup doesn’t help much with dynamic scene where some resources are loaded only if the context enables it (like the player being in a given area of the map, in a game, for instance);
• interfacing! people will have to learn how to use free monads, AST and all that stuff and how to interface it with their own code (FRP, render loops, etc.);
• performance; even though it should be okay, you’ll still hit a performance overhead by using pure graphics frameworks, because of internals mechanisms used to make it work.

So yeah, a pure graphics framework is very neat, but keep in mind there’s – so far – no proof it actually works, scales nor is usable for a decent high-performance for end-users. It’s the same dilemna as with Conal’s FRP: it’s nice, but we don’t really know whether it works “at large scale and in the real world”.

IO-bound API

Most of the API out there are IO-bound. OpenGL is a famous C API known to be one of the worst one on the level of side-effects and global mutations. Trust me, it’s truly wrong. However, the pure API as mentioned above are based on those impure IO-bound APIs. So we couldn’t do much without them.

There are side effects that are not that bad. For instance, in OpenGL, creating a new buffer is a side-effect: it requires that the CPU tell the GPU “Hey buddy, please create a buffer with that data, and please give me back a handle to it!”. Then the GPU would reply ”No problem pal, here’s your handle!”. This side-effect don’t harm anyone, so we shouldn’t worry about it too much.

However, there are nasty side-effects, like binding resources to the OpenGL context.

So what are advantages of IO-bound designs? Well:

• simple: indeed, you have handles and side-effects and you have a (too) fine control of the instruction flow;
• performance, because IO is the naked real-world monad;
• because IO is the high-order kinded type of any application (think of the main function), an IO API is simple to use in any kind of application;
• we can use (MonadIO m) => m to add extra flexibility and create interesting constraints.

And drawbacks:

• IO is very opaque and is not referentially transparent;
• IO is a dangerous type in which no one has no warranty about what’s going on;
• one can fuck up everything if they aren’t careful;
• safety is not enforced as in pure code.

What about luminance’s design?

Since the beginning, luminance has been an API built to be simple, type-safe and stateless.

Type-safe means that all objects you use belong to different type sets and cannot be mixed between each other implicitely – you have to use explicit functions to do so, and it has to be meaningful. For instance, you cannot create a buffer and state that the returned handle is a texture: the type system forbids it, while in OpenGL, almost all objects are in the GLuint set. It’s very confusing and you might end up passing a texture (GLuint) to a function expecting a framebuffer (GLuint). Pretty bad right?

Stateless means that luminance has no state. You don’t have a huge context you have to bind stuff against to make it work. Everything is stored in the objects you use directly and specific context operations are translated into a different workflow so that performance are not destroyed – for instance luminance uses batch rendering so that it performs smart resource bindings.

Lately, I’ve been thinking of all of that. Either turn the API pure or leave it the way it is. I started to implement a pure API using self-recursion. The idea is actually simple. Imagine this GPU type and the once function:

import Control.Arrow ( (***), (&&&) )
import Data.Function ( fix )

newtype GPU f a = GPU { runGPU :: f (a,GPU f a) }

instance (Functor f) => Functor (GPU f) where
  fmap f = GPU . fmap (f *** fmap f) . runGPU

instance (Applicative f) => Applicative (GPU f) where
  pure x = fix $ \g -> GPU $ pure (x,g)
  f <*> a = GPU $ (\(f',fn) (a',an) -> (f' a',fn <*> an)) <$> runGPU f <*> runGPU a

instance (Monad m) => Monad (GPU m) where
  return = pure
  x >>= f = GPU $ runGPU x >>= runGPU . f . fst

once :: (Applicative f) => f a -> GPU f a
once = GPU . fmap (id &&& pure)

We can then build pure values that will have a side-effect for resource acquisition and then hold the same value for ever with the once function:

let buffer = once createBufferIO

-- later, in IO
(_,buffer2) <- runGPU buffer

Above, the type of buffer and buffer2 is GPU IO Buffer. The first call runGPU buffer will execute the once function, calling the createBufferIO IO function and will return buffer2, which just stores a pure Buffer.

Self-recursion is great to implement local states like that and I advise having a look at the Auto type. You can also read my article on netwire, which uses self-recursion a lot.

However, I kinda think that a library should have well defined responsibilities, and building such a pure interface is not the responsibility of luminance because we can have type-safety and a stateless API without wrapping everything in that GPU type. I think that if we want such a pure type, we should add it later on, in a 3D engine or a dedicated framework – and that’s actually what I do for demoscene purposes in another, ultra secret project. ;)

The cool thing with luminance using MonadIO is the fact that it’s very easy to use in any kind of type that developpers want to use in their applications. I really don’t like frameworks which purpose is clearly not flow control that actually enforce flow control and wrapping types! I don’t want to end up with a Luminance type or LuminanceApplication type. It should be simple to use and seamless.

I actually start to think that I did too much about that pure API design idea. The most important part of luminance should be type-safety and statelessness. If one wants a pure API, then they should use FRP frameworks or write their own stuff – with free monads for instance, and it’s actually funny to build!

The next big steps for luminance will be to clean the uniform interfaces which is a bit inconsistent and unfriendly to use with render commands. I’ll let you know.

On the 19th of March 2015, a law was introduced in France. That law was entitled “loi du renseignement” and was presented in regard of what happened in the Charlie Hebdo awful week. The main idea of the law is to put peole on wire so that terrorist activities could be spotted and dismantled.

That law will be voted and apply on the the 5th of May 2015.

Although such a law sounds great for counter-terrorism, it has several major issues people should be aware of.

You’re on wire

Every people – in France and abroad as well of course – could be on wire. The French government might have access to whatever you do with your internet connection. Crypto-analysts working for the goverment will read dozens of thousands of messages from individuals. They’ll know where you go to and where you go from – think of your smartphone’s GPS capabilities. They’ll know which pictures you take, how much time you spend in your bathroom. They’ll even be able to read your e-mails if they want to.

Of course, most people “just don’t care”. “Why should I care that the government knows the litter brand I bought to my cat? I don’t give a damned heck as long as they catch bad people”. That’s a point, but that’s also being blind. Digital communication is not only about yourself. It’s about people. It’s about you and your friends. You, and your contacts. You can’t choose for them whether they’d like people to watch over their shoulders every now and then. If the government has access to your private data, it has access to your friends’ and contacts’ data as well. Not giving a damned heck is being selfish.

You’ll be violated

Then comes the issue with what the law is. It gives the government the right to spy on masses. They could even sell the data they collect. As a counter argument, the “Commission de contrôle des techniques de renseignement” – which French stands for “Control commission of intelligence techniques” – was created. That committe is there to watch the government and ensure it doesn’t go out of control and doesn’t violate people’s rights. The issue with that is that our prime minister has the right to ignore the committee’s decision. If the committee says “Wait a minute. You don’t have the right to gather those information without asking for M. Dupont’s approval”, the prime minister may answer back “Well, fuck off. I will and you can’t stop me”. And the sad truth is that… yeah, with that law, the prime minister and their team has the right to ignore the committee’s decision.

The committee then has no point. It just gives an opinion. Not a veto. What would happen if a terrorist hacked into your computer. Would you go to jail because the prime minister would have stated you were a terrorist? Damn me if I know.

We’re going to lose a right

French people will lose a very important right: the right to privacy. It’ll be sacrificied without your opinion for counter-terrorism, giving the government a power it shouldn’t have. You thought the NSA was/is wrong? Sure it is. But when the NSA watches over American and worldwide people, it is illegal whereas when the French government watches over French and worldwide people, it is legal. That makes the French government way worse than the NSA to me.

I think the first thing I’ll do will be to revoke my French VPS subscription to move it out of France, in a country in which people still have privacy. Keep your communication encrypted as much as possible (ssh and so on). And as a bad joke, don’t leave your camera in your bedroom. You might be spied on by Manuel Valls while having sex with your girlfriend.

In a real time rendering system, it’s not uncommon finding constructs about assets. One famous construct is the resource manager. A resource manager is responsible of several tasks, among:

• providing a simple interface to load objects from disk (1) ;
• ensuring we don’t try to load the same object twice (2) ;
• resolving dependencies between resources (3).

The first point is obvious, but the two others are less intuitive. (2) is important when the user might try to load the same object several times – for instance, a car model, or a character or a weapon. The most known strategy to prevent such a situation from happening is by using a software cache.

A software cache – let’s just say cache – is an opaque object that loads the object on the first request, then just returns that object for future same requests. For instance, consider the following requests and the corresponding cache behavior:

1. load “wood.png” -> not cached ; loading ; return
2. load “grass.png” -> not cached ; loading ; return
3. load “wood.png” -> cached ; return
4. load “metal.png” -> not cached ; loading ; return
5. load “metal.png” -> cached ; return
6. etc.

That behavior is very nice because it will spare a lot of computations and memory space.

(3) is about dependencies. For instance, when you load a car model, you might need to load its textures as well. Well, not really load. Consider the following:

1. load “car.mesh” -> not cached 1. load “metal_car.png” -> not cached ; loading ; return 2. loading ; return
2. load “metal_car.png” -> cached ; return
3. load “other_car.mesh” -> not cached 1. load “metal_car.png” -> cached ; return 2. return
4. load “car.mesh” -> cached ; return

You got the idea. (3) needs (2) to be efficient.

Possible implementations

Singleton

In imperative languages and especially in those that support template and/or generics, people tend to implement the cache system with an ugly design pattern – which is actually an anti design pattern : singleton. Each type of resource is assigned a manager by using a template parameter, and then if a manager needs to load a dependency, it just has to reference the corresponding manager by stating the type in the template parameter :

Model & getResource<Model>(std::string const &name) {
  Texture &dependency = getResource<Texture>(...);
  ...
}

That way of doing might sound great, but eh, singletons are just global variables with a unicity constraint. We don’t want that.

Explicit pure store

We can use an explicit store object. That is, some kind of map. For instance, the store that holds textures would have a type like (in Haskell):

textureStore :: Map String Texture

A model store would have the following type:

modelStore :: Map String Model

And each stores is assigned a function; loadTexture, loadModel, and so on.

There are several drawbacks if we go that way. First, we have to carry all stores when using their functions. Some functions might need other stuff in order to resolve dependencies. Secondly, because of explicit state, we need to manually accumulate state! A loading function would have such a following type:

loadTexture :: Map String Texture -> String -> m (Texture,Map String Texture)

That will expose a lot of boilerplate to the user, and we don’t want that.

Implicit pure store

We can enhance the explicit store by putting it into some kind of context; for instance, in MonadState. We can then write loadTexture to make it nicer to use:

loadTexture :: (MonadState (Map String Texture) m,...)
            => String
            -> m Texture

There is a problem with that. What happens when we add more types? For instance if we want to handle textures and models? MonadState has a type family constraint that forbids two instances for the pair s m. The following is not allowed and will raise a compiler error:

instance MonadState (Map String Texture) MyState where
  ...

instance MonadState (Map String Model) MyState where
  ...

The solution to that problem is to have the carried state a polymorphic type and use typeclass constraint to extract and modify the map we want:

class HasMap a s where
  extractMap :: s -> Map String a
  modifyMap :: (Map String a -> Map String a) -> s -> s

With that, we can do something like this:

loadTexture :: (MonadState s m,HasMap Texture s,...)
            => String
            -> m Texture

loadModel :: (MonadState s m,HasMap Texture s,HasMap Model s,...)
          => String
          -> m Model

However, we hit a new issue here. What are s and m? Well, m doesn’t really matter. For simplicity, let’s state we’re using a monad transformer; that is, we use StateT s m as monad.

We still need s. The problem is that s has to be provided by the user. Worse, they have to implement all instances we need so that the loading functions may work. Less boilerplate than the explicit store solution, but still a lot of boilerplate. Imagine you provide a type for s, like Cache. Expending the cache to support new types – like user-defined ones – will be more extra boilerplate to write.

Closures

The solution I use in my engine might not be the perfect solution. It’s not referentially transparent, an important concept in Haskell. However, Haskell is not designed to be used in convenient situations only. We’re hitting a problematic situation. We need to make a compromise between elegance and simplicity.

The solution required the use of closures. If you don’t know what a closure is, you should check out the wikipedia page for a first shot.

The idea is that our loading functions will perform some IO operations to load objects. Why not putting the cache directly in that function? We’d have a function with an opaque and invisible associated state. Consider the following:

type ResourceMap a = Map String a

getTextureManager :: (MonadIO m,...)
                  => m (String -> m Texture)
getTextureManager = do
  ref <- newIORef empty
  pure $ \name -> do
    -- we can use the ref ResourceMap to insert / lookup value in the map
    -- thanks to the closure!

That solution is great because now, a manager is just a function. How would you implement getModelManager? Well:

getModelManager :: (MonadIO m,...)
                => (String -> m Texture)
                -> m (String -> m Model)
getModelManager loadTexture = ...

We can then get the loader functions with the following:

loadTexture <- getTextureManager
loadModel <- getModelManager loadTexture

And you just have to pass those functions around. The cool thing is that you can wrap them in a type in your library and provide a function that initializes them all at once – I do that in my engine. Later on, the user can extend the available managers by providing new functions for new types. In my engine, I provide a few functions like mkResourceManager that hides the ResourceMap, providing two functions – one for lookup in the map, one for inserting into the map.

Conclusion

I truly believe that my solution is a good compromise between elegance and ease. It has a lot of advantages:

• simple to use ;
• simple to implement; you just have to play around with closures ;
• dependencies resolving is easy to add and hidden once the functions are generated ;
• little runtime overhead (due to closures, might be optimized away by the compiler though) ;
• can be easily extended by the user to support custom/new types ;
• if correctly used, implementations can replace IORef with TVar or similar objects for thread-safe implementations ;
• several replicated functions mean several stores (can be useful in certain cases).

The huge drawback I see in that solution is its opacity. There’s also no way to query the state of each cache. Such a feature could be added by proving a new function, for instance. Same thing for deletion.

I’m one of those Haskellers who love purity. I try to keep my code the purest I can, but there are exceptions, and that cache problem fits that kind of exception.

Feel free to comment, and as always, keep the vibe and happy hacking!

The thing is… Times change. The more it passes, the more I become mature in what I do in the Haskell community. I’m a demoscener, and I need to be productive. Writing a whole 3D engine for such a purpose is a good thing, but I was going round and round in circles, changing the whole architecture every now and then. I couldn’t make my mind and help it. So I decided to stop working on that, and move on.

If you are a Haskell developer, you might already know Edward Kmett. Each talk with him is always interesting and I always end up with new ideas and new knowledge. Sometimes, we talk about graphics, and sometimes, he tells me that writing a 3D engine from scratch and release it to the community is not a very good move.

I’ve been thinking about that, and in the end, I agree with Edward. There’re two reasons making such a project hard and not interesting for a community:

1. a good “3D engine” is a specialized one – for FPS games, for simulations, for sport games, for animation, etc. If we know what the player will do, we can optimize a lot of stuff, and put less details into not-important part of the visuals. For instance, some games don’t really care about skies, so they can use simple skyboxes with nice textures to bring a nice touch of atmosphere, without destroying performance. In a game like a flight simulator, skyboxes have to be avoided to go with other techniques to provide a correct experience to players. Even though an engine could provide both techniques, apply that problem to almost everything – i.e. space partitionning for instance – and you end up with a nightmare to code ;
2. an engine can be a very bloated piece of software – because of point 1. It’s very hard to keep an engine up to date regarding technologies, and make every one happy, especially if the engine targets a large audience of people – i.e. hackage.

Point 2 might be strange to you, but that’s often the case. Building a flexible 3D engine is a very hard and non-trivial task. Because of point 1, you utterly need to restrict things in order to get the required level of performance or design. There are people out there – especially in the demoscene world – who can build up 3D engines quickly. But keep in mind those engines are limited to demoscene applications, and enhancing them to support something else is not a trivial task. In the end, you might end up with a lot of bloated code you’ll eventually zap later on to build something different for another purpose – eh, demoscene is about going dirty, right?! ;)

Basics

So… Let’s go back to the basics. In order to include everyone, we need to provide something that everyone can download, install, learn and use. Something like OpenGL. For Haskell, I highly recommend using gl. It’s built against the gl.xml file – released by Khronos. If you need sound, you can use the complementary library I wrote, using the same name convention, al.

The problem with that is the fact that OpenGL is a low-level API. Especially for new comers or people who need to get things done quickly. The part that bothers – wait, no, annoys – me the most is the fact that OpenGL is a very old library which was designed two decades ago. And we suffer from that. A lot.

OpenGL is a stateful graphics library. That means it maintains a state, a context, in order to work properly. Maintaining a context or state is a legit need, don’t get it twisted. However, if the design of the API doesn’t fit such a way of dealing with the state, we come accross a lot of problems. Is there one programmer who hasn’t experienced black screens yet? I don’t think so.

The OpenGL’s API exposes a lot of functions that perform side-effects. Because OpenGL is weakly typed – almost all objects you can create in OpenGL share the same GL(u)int type, which is very wrong – you might end up doing nasty things. Worse, it uses an internal binding system to select the objects you want to operate on. For instance, if you want to upload data to a texture object, you need to bind the texture before calling the texture upload function. If you don’t, well, that’s bad for you. There’s no way to verify code safety at compile-time.

You’re not convinced yet? OpenGL doesn’t tell you directly how to change things on the GPU side. For instance, do you think you have to bind your vertex buffer before performing a render, or is it sufficient to bind the vertex array object only? All those questions don’t have direct answers, and you’ll need to dig in several wikis and forums to get your answers – the answer to that question is “Just bind the VAO, pal.”

What can we do about it?

Several attempts to enhance that safety have come up. The first thing we have to do is to wrap all OpenGL object types into proper types. For instance, we need several types for Texture and Framebuffer.

Then, we need a way to ensure that we cannot call a function if the context is not setup for. There are a few ways to do that. For instance, indexed monads can be a good start. However, I tried that, and I can tell you it’s way too complicated. You end up with very long types that make things barely unreadable. See this and this for excerpts.

Luminance

In my desperate quest of providing a safer OpenGL’s API, I decided to create a library from scratch called luminance. That library is not really an OpenGL safe wrapper, but it’s very close to being that.

luminance provides the same objects than OpenGL does, but via a safer way to create, access and use them. It’s an effort for providing safe abstractions without destroying performance down and suited for graphics applications. It’s not a 3D engine. It’s a rendering framework. There’s no light, asset managers or that kind of features. It’s just a tiny and simple powerful API.

Example

luminance is still a huge work in progress. However, I can already show an example. The following example opens a window but doesn’t render anything. Instead, it creates a buffer on the GPU and perform several simple operations onto it.

-- Several imports.
import Control.Monad.IO.Class ( MonadIO(..) )
import Control.Monad.Trans.Resource -- from the resourcet package
import Data.Foldable ( traverse_ )
import Graphics.Luminance.Buffer
import Graphics.Luminance.RW
import Graphics.UI.GLFW -- from the GLFW-b package
import Prelude hiding ( init ) -- clash with GLFW-b’s init function

windowW,windowH :: Int
windowW = 800
windowH = 600

windowTitle :: String
windowTitle = "Test"

main :: IO ()
main = do
  init
  -- Initiate the OpenGL context with GLFW.
  windowHint (WindowHint'Resizable False)
  windowHint (WindowHint'ContextVersionMajor 3)
  windowHint (WindowHint'ContextVersionMinor 3)
  windowHint (WindowHint'OpenGLForwardCompat False)
  windowHint (WindowHint'OpenGLProfile OpenGLProfile'Core)
  window <- createWindow windowW windowH windowTitle Nothing Nothing
  makeContextCurrent window
  -- Run our application, which needs a (MonadIO m,MonadResource m) => m
  -- we traverse_ so that we just terminate if we’ve failed to create the
  -- window.
  traverse_ (runResourceT . app) window
  terminate

-- GPU regions. For this example, we’ll just create two regions. One of floats
-- and the other of ints. We’re using read/write (RW) regions so that we can
-- send values to the GPU and read them back.
data MyRegions = MyRegions {
    floats :: Region RW Float
  , ints   :: Region RW Int
  }

-- Our logic.
app :: (MonadIO m,MonadResource m) => Window -> m ()
app window = do
  -- We create a new buffer on the GPU, getting back regions of typed data
  -- inside of it. For that purpose, we provide a monadic type used to build
  -- regions through the 'newRegion' function.
  region <- createBuffer $
    MyRegions
      <$> newRegion 10
      <*> newRegion 5
  clear (floats region) pi -- clear the floats region with pi
  clear (ints region) 10 -- clear the ints region with 10
  readWhole (floats region) >>= liftIO . print -- print the floats as an array
  readWhole (ints region) >>= liftIO . print -- print the ints as an array
  floats region `writeAt` 7 $ 42 -- write 42 at index=7 in the floats region
  floats region @? 7 >>= traverse_ (liftIO . print) -- safe getter (Maybe)
  floats region @! 7 >>= liftIO . print -- unsafe getter
  readWhole (floats region) >>= liftIO . print -- print the floats as an array

Those read/write regions could also have been made read-only or write-only. For such regions, some functions can’t be called, and trying to do so will make your compiler angry and throw errors at you.

Up to now, the buffers are created persistently and coherently. That might cause issues with OpenGL synchronization, but I’ll wait for benchmarks before changing that part. If benchmarking spots performance bottlenecks, I’ll introduce more buffers and regions to deal with special cases.

luminance doesn’t force you to use a specific windowing library. You can then embed it into any kind of host libraries.

What’s to come?

luminance is very young. At the moment of writing this article, it’s only 26 commits old. I just wanted to present it so that people know it exists will be released as soon as possible. The idea is to provide a library that, if you use it, won’t create black screens because of framebuffers incorrectness or buffers issues. It’ll ease debugging OpenGL applications and prevent from making nasty mistakes.

I’ll keep posting about luminance as I get new features implemented.

As always, keep the vibe, and happy hacking!

Easier API

The CHANGELOG is available to read but one of the most important change was the refactoring of the pipeline system and of the overall API to make it less confusing. For instance, the Framebuffer::default is now renamed Framebuffer::back_buffer. Another example is the face culling, that was in pre-0.27 enabled by default (causing people who don’t now about it to be confused because of a black screen). This is now disabled by default; hence less confusion.

The pipeline system types are now easier to use. For instance, redundancy was removed with the various Bound* objects. When you want a bound buffer of f32 for instance, you just have to use the type BoundBuffer<'a, f32> instead of the pre-0.27 BoundBuffer<'a, Buffer<f32>>. That is a small change but it will eventually make working with luminance more comfortable.

Another change that I think is beneficial is the new TessSlice type – that replaces the pre-0.27 TessRender. This type represents a GPU tessellation slice. Such a slice can be used to render part of a tessellation. What’s great is that the legacy methods to build such slices (TessRender::one_whole for instance) have a new, more generalized way to do it: TessSliceIndex<Idx>. This trait gives you a slice method which argument has type Idx. Several implementors exist:

• impl TessSliceIndex<Range<usize>> for Tess.
• impl TessSliceIndex<RangeFrom<usize>> for Tess.
• impl TessSliceIndex<RangeTo<usize>> for Tess.
• impl TessSliceIndex<RangeFull<usize>> for Tess.

They all give you a TessSlice to work with (with the appropriate lifetime). The four implementations listed above can be invoked with:

• tess.slice(0..10).
• tess.slice(0..).
• tess.slice(..10).
• tess.slice(..).

For those who wonder:

Why have you not implemented Index instead?

The current Index trait doesn’t give you enough power to index a value by returning something else than a direct reference. Everything is summed up in the RFC I wrote to fix Index.

Feel free to have a look at the CHANGELOG for further information.

Examples, examples, examples!

One of the most important feature people have asked is adding examples. The online documentation is not enough – even if I spent lots of hours enhancing it. So I added several examples:

• 01-hello-world: learn how to draw two colored triangles by using vertex colors (comes in direct and indexed geometry versions).
• 02-render-state: learn how to change the render state to change the way the triangles are rendered or how fragment blending happens.
• 03-sliced-tess: learn how to slice a single GPU geometry to dynamically select contiguous regions of it to render!
• 04-shader-uniforms: send colors and position information to the GPU to add interaction with a simple yet colorful triangle!
• 05-attributeless: render a triangle without sending any vertex data to the GPU!
• 06-texture: learn how to use a loaded image as a luminance texture on the GPU!
• 07-offscreen: get introduced to offscreen rendering, a powerful technique used to render frames into memory without directly displaying them on your screen. Offscreen framebuffers can be seen as a generalization of your screen.

You can find them in this place.

Please do contribute if you thing something is missing, either by opening an issue or by opening a PR. Any kind of contribution is highly welcomed!

• Context passing: it is now possible to pass a mutable reference (&mut) to a typed context when loading and reloading a resource. This enables per-value loadings, which is neat if you need to add extra data when loading your resources (e.g. increment a counter).
• Methods support: before version 0.7, you had to implement a trait, Load, in order to tell warmy how to load a given object of a given type. This is convenient but carries a bad drawback: if you want to represent your object with both JSON and XML for instance, you need to type wrap your type so that you can impl Load twice. This was annoying and the type system also handed you back an object which type was the wrapper type, not the wrapped type. This annoyance was removed in 0.7 as you now have an extra type variable to Load – it defaults to () though – that you can use to impl Load several times – think of that tag type variable as a type representing the encoding or the method to use to load your value.
• A VFS (Virtual FileSystem): the VFS makes it possible to write resource keys without caring about their real location – e.g. /splines/color_grading.json. Before that, you still had to provide a real filesystem path, which was both confusing and annoying (since you already give one when you create a Store, the object that holds your resource).
• Changelog here.

I posted on reddit in order to make people know of the new version, and interesting talks started to occur on both IRC and GitHub. What people seem to want the most now is asynchronous loading and reloading. I’ve been wanting that feature for a while too so I decided it could be a good idea to start working on it. However, after a day of toying around, I came to the realization that I should write a small blog post about it because I think it’s not trivial and it could help me shape my ideas.

Note: this post is about brainstorming and setting up the frame to why and how async warmy. You will find incomplete code, ideas and questions there. If you want to contribute to the discussion, you’re more than welcome!

Synchronous versus asynchronous

What does it mean to have a synchronous computation? What does it mean to have an asynchronous one? You might find it funny, but a lot of people are still confused with the exact definition, so I’ll try to give you more hindsight.

We talk about a synchronous task when we have to wait for it to finish before moving on to another task. We have to wait until its completion to get the control flow back and call other functions. We often talk about blocking computations, because you cannot do anything else while that computation is running – at least on the thread this computation is running on.

We talk about an asynchronous task when you can get the control flow back without having to wait for the task to finish. However, that doesn’t necessarily mean that the task is being executed in parallel or concurrently. At some extent, you could easily label generators as asynchronous primitives, and this is what actually happens in async / await code: you give back the control flow to the caller and the callee execution gets re-scheduled later. Hence, this is asynchronous programming, yet the scheduling execution could be implemented on a single thread – hence no parallel nor concurrency actually happen. Another example is when you perform a HTTP request: you can send the request and instead of blocking until the response arrive, you can give the control back, do something else, and then, at some time, handle the response. You don’t need parallelism to do this: you need asynchronous programming.

Note: a generalization of a generator is a coroutine, which hands the control flow back to another coroutine instead of the caller when wanted.

What about warmy?

At the time of writing this blog entry, warmy is completely synchronous. Consider the following example:

extern crate serde_json;
extern crate warmy;

use std::error;
use std::fmt;
use std::fs::File;
use std::io;
use std::thread;
use std::time::Duration;
use warmy::{FSKey, Load, Loaded, Res, Store, StoreOpt, Storage};

struct JSON;

#[derive(Clone, Copy, Debug, PartialEq)]
struct Foo {
  array: [f32; 4]
}

#[derive(Debug)]
enum FooError {
  JsonError(serde_json::Error),
  IOError(io::Error)
}

impl fmt::Display for FooError {
  fn fmt(&self, f: &mut fmt::Formatter) -> Result<(), fmt::Error> {
    match *self {
      FooError::JsonError(ref e) => e.fmt(f),
      FooError::IOError(ref e) => e.fmt(f),
    }
  }
}

impl error::Error for FooError {
  fn description(&self) -> &str {
    match *self {
      FooError::JsonError(ref e) => e.description(),
      FooError::IOError(ref e) => e.description(),
    }
  }

  fn cause(&self) -> Option<&error::Error> {
    match *self {
      FooError::JsonError(ref e) => Some(e),
      FooError::IOError(ref e) => Some(e),
    }
  }
}

impl<C> Load<C, JSON> for Foo {
  type Key = FSKey;

  type Error = FooError;
  
  fn load(
    key: Self::Key,
    _: &mut Storage<C>,
    _: &mut C,
  ) -> Result<Loaded<Self>, Self::Error> {
    let file = File::open(key.as_path()).map_err(FooError::IOError)?;
    let array = serde_json::from_reader(file).map_err(FooError::JsonError)?;
    let foo = Foo { array };

    Ok(foo.into())
  }
}

fn main() {
  // read scarce resources in /tmp
  let ctx = &mut ();
  let opt = StoreOpt::default().set_root("/tmp");
  let mut store: Store<()> = Store::new(opt).expect("store creation failed");

  // get our resources via JSON decoding
  let foo: Res<Foo> = store.get_by(&FSKey::new("/foo.json"), ctx, JSON).expect("foo should be there");
  let foo2: Res<Foo> = store.get_by(&FSKey::new("/foo2.json"), ctx, JSON).expect("foo should be there");

  println!("{:#?}", foo);
  println!("{:#?}", foo2);

  loop {
    // sync the resources with the disk
    store.sync(ctx);
    thread::sleep(Duration::from_millis(100));
  }
}

Note: you can test that code by inserting warmy = "0.7" and serde_json = "1" in a Cargo.toml and by creating the JSON files containing 4D arrays of numbers, like [0, 1, 2, 3] in /tmp/foo{,2}.json.

If you look closely at that example, you can spot two important locations when we get resources: when we create the foo and foo2 bindings (search for store.get_by invocations). Here, it’s important to understand what’s really happening:

1. First, /foo.json is loading on the current thread.
2. Then, when it’s loaded and cached, /foo2.json starts loading.

We can already see a pity here: both the files are completely disconnected from each other, yet, the second file must wait for the former to finish before starting loading. We’re not using all the cores / threads of our CPU. So we could perfectly imagine this new scenario:

1. First, ask to load /foo.json and immediately get control flow back.
2. Then, ask to load /foo2.json and immediately get control flow back.
3. Wait for both the resources to be available, then continue.

You could even do whatever you want between (2.) and (3.). The point here is that we can run tasks without having to wait for them to finish before starting others. The explicit waiting part could be a blocking call, such as:

let foo_task = store.get_by(…);
let foo2_task = store.get_by(…);

let foo = foo_task.wait();
let foo2 = foo2_task.wait();

Note: this is not the foreseen or planned interface. It’s just there to illustrate what we want to achieve there.

The goal of this blog entry is to explore possible implementations.

futures

The futures crate is a very promising and interesting crate. What it provides is basically a mean to express data that might get available in the future. For instance:

fn ping(host: &str) -> ???

Here, ping is a function that will perform an ICMP request to a given host, identified by the function argument. Because that function might take a while (at least several milliseconds, which is a lot), we have to make a decision:

1. Either we decide to block the current thread until a response gets available (the host responds to our ping and we get the packet back).
2. Or either we decide to free the control flow from the ping function and do something else until the response arrive.

If you’ve followed all what I said since the beginning of this blog post, you might have noticed that (1.) is synchronous and (2.) is asynchronous (also notice that we haven’t talked about parallelism yet!).

With (1.), we would write ping’s signature like this:

fn ping(host: &str) -> Result<PingResponse, NetworkError>

With (2.), we would write it like this, using the futures crate:

fn ping(host: &str) -> impl Future<Item = PingResponse, Error = NetworkError>

Note: the syntax impl Trait is called conservative impl trait and its description can be found in the corresponding RFC on impl Trait.

So what you basically do here is to send a non-blocking request and get its result in a non-blocking way. Sweet! How does that apply to warmy?

futures applied to warmy

Asynchronous warmy should have the following properties:

• When you ask for a resource to load, you immediately get the control flow back.
• Hot-reloading mustn’t block either, as you will do it in the Store::sync function.
• Because of the asynchronous nature of those computations, the context passing now gets a bit tricky: we cannot handle a mutable reference on our context anymore, because otherwise we would block the main thread. We need to share the context in a smart way.
• Finally, all parallel computations should be ran on some kind of green thread so that we don’t have to worry about destroying the OS threads amount.

Asynchronous loading

The current definition of the Store::get function is:

pub fn get<K, T>(
  &mut self, 
  key: &K, 
  ctx: &mut C
) -> Result<Res<T>, StoreErrorOr<T, C>>
where
  T: Load<C>,
  K: Clone + Into<T::Key>

We want something like this:

pub fn async_get<K, T>(
  &mut self, 
  key: &K, 
  ctx: &mut C
) -> Result<impl Future<Item = AsyncRes<T>, Error = StoreErrorOr<T, C>>, StoreErrorOr<T, C>>
where
  T: Load<C>,
  K: Clone + Into<T::Key>

Woosh. It gets hard to read – eventually, trait aliases will help us there. We can see the use of the futures crate here, in the return type:

impl Future<Item = AsyncRes<T>, Error = StoreErrorOr<T, C>>

Which reads as “An asynchronous resource of type T available in the future or an error due to the loading of T and C or due to the store”.

However, something important to notice is that we miss a crucial point here: we actually want a parallel execution. We could start try by defining a small state machine to step through the process of loading a resource. That could be:

enum LoadState<K, T, C> {
  Pending(K, Shared<C>), // resource loading pending
  Loading(Receiver<AsyncRes<T>>), // resource currently loading
  Done, // resource loaded
}

This small state machine can be stepped through by the following process:

fn start_load<K, T, C>(key: K, ctx: &mut C) -> impl Future<Item = LoadState<K, T, C>> {
  LoadState::Pending(key, ctx.share_somehow())
}

impl<K, T, C> Future for LoadState<K, T, C> {
  type Item = AsyncRes<T>;

  type Error = StoreErrorOr<T, C>;

  fn poll(&mut self, ctx: &mut Context) -> Result<Async<Self::Item>, Self::Error> {
    match *self {
      LoadState::Pending(..) => {
        // start the loading
        let (sx, rx) = oneshot::channel();
        let task = Task::new(move || {
          // load the resource somehow
          let res = …;
          sx.send(res);
        });

        // spawn a new task to load the resource and update the state machine
        ctx.spawn(task);
        *self = LoadState::Loading(rx);
        Ok(Async::NotReady)
      }

      LoadState::Loading(ref mut rx) => {
        match rx.map(|res| {
          *self = LoadState::Done;
          res
        }).poll()?
      }

      LoadState::Done => panic!("poll called after future has arrived")
    }
  }
}

That code might be incomplete or not typecheck completely (not tested), but the idea is there. I still don’t know whether this is the good way to use futures or whether I should run a single, long computation without the small state machine. I just don’t know yet.

Asynchronous reloading

This part is tightly bound to how the Store::sync will behave. Now that I tink about it, maybe it’d be more ergonomic / elegant / simpler to have an event loop living on a separate thread that would benefit from epoll and all those fancy asynchronous primitives to deal with that. That would result in no more by-hand synchronization, since it’d be done by the event loop / reactor / whatever you want to name it.

Currently, I have not settled up on any decision regarding reloading.

Asynchronous context passing

This will be definitely a tricky part as well, because what was before:

store.sync(&mut ctx);

will then become something like:

store.sync(ctx.share());

Also, something that might happen is that because the synchronization will be done in an asynchronous way, you will need to ensure that you do not try to synchronize the same resource several times.

Finally, if you take into account the previous section, the user might not even synchronize by hand anymore, so the context will have to be shared anyway and moved to the event loop. Argh. That doesn’t seem simple at all! :)

Parallel execution

One final aspect of asynchronous warmy is obviously how we’re going to poll the various Futures generated by the get functions. One first, naive idea would be to run those in OS threads. However, imagine that you’ll be loading plenty of resources (on a big and release project, it’s easy to imagine several thousands of resources loading in parallel). You don’t want to allocate that many OS threads but instead rely on a green threading solution… which I haven’t found yet. People on IRC advised to have a look at futures-cpupool, but I’d like to have a complete solution set before deciding anything.

I will post this blog post on reddit with hope that people will come up with ideas and I’m sure enlighting ideas on how to cope with all of this asynchronous properties I want to push to warmy.

Thanks for having read me, and keep the vibes!

OpenAL is an open-source and free audio library developped by Creative Technology. The first version, however, was made by Loki Software. The API targets spacialized audio and sound managing so that it’s simple to simulate sound attenuation over distance and Doppler effect. OpenAL is supported on a wide variety of platforms, from PC (Windows, Linux, FreeBSD, Mac OSX and so on) to consoles (PS3, Xbox…).

The latest version was released in 2005 (9 years old ago!) and is OpenAL 1.1. It’s then pretty logical that we should have OpenAL 1.1 in Haskell.

OpenAL and Haskell

“Wait… we already have OpenAL in Haskell!”

That’s true. However, the OpenAL Haskell package has some serious issues.

The first one is that package is not a direct binding to OpenAL. It’s a high-level wrapper that wraps stuff with obscure types, like StateVar or ObjectName. Even though such abstractions are welcomed, we should have a raw binding so that people who don’t want those abstractions can still use OpenAL. Moreover, such high libraries tend to lift resources allocations / deallocations into the GC¹. That is a pretty bad idea because we, sometimes, want to control how long objects live.

The second one, which is as terrible, is the fact that those abstractions aren’t defined by the Haskell OpenAL package, but by another, which is… OpenGL. The latter is not a raw binding to OpenGL – for that have a look at OpenGLRaw or gl – but the same kind of high-level oriented package. A lot of us have asked the authors of both the package to agree of making those abstractions shared between both the libraries; they never seemed to agree.

And even though, having a dependency between OpenAL and OpenGL is insane!

Sooooo… since Edward Kmett had the idea of gl to fix similar issues, I think I should have the idea of al to go along with gl. The idea is that if you want to use one, you can use the other one without any conflict.

al 0.1

Yesterday night, around 20:00, I decided to make it. It took me several hours to write the raw binding, but I finally made it and released al 0.1 the very next day – uh, today!

Because OpenAL is not shipped with your ² nor anything special, and thus because you have to explicitely install it, installing al requires a bit more than just typing cabal install al.

Installing al

cabal update

First things first:

cabal update

You should have al available if you type cabal info al afterwards.

c2hs

al requires you to have c2hs – version 0.23.* – installed. If not:

cabal install c2hs

Be sure it’s correctly installed:

c2hs --version
C->Haskell Compiler, version 0.23.1 Snowbounder, 31 Oct 2014
  build platform is "i386-mingw32" <1, True, True, 1>

OpenAL SDK

Of course, you should have the OpenAL SDK installed as well. If not, install it. It should be in your package manager if you’re on Linux / FreeBSD, and you can find it here for Windows systems.

Important: write down the installation path, because we’ll need it!

Installing al

Here we are. In order to install, al needs the paths to your OpenAL SDK. I’m not sure whether it could vary a lot, though. The default Windows paths are:

• C:\Program Files (x86)\OpenAL 1.1 SDK\libs for the libraries ;
• C:\Program Files (x86)\OpenAL 1.1 SDK\include for the header files.

In future releases, I’ll default to those so that you don’t have to explicitely pass the paths if you have a default installation. But for the moment, you have to pass those paths when installing:

cabal install al --extra-lib-dirs="C:\Program Files (x86)\OpenAL 1.1 SDK\libs"
--extra-include-dirs="C:\Program Files (x86)\OpenAL 1.1 SDK\include"

Of course, you may adapt paths to your situation. You may even try without that for UNIX systems – it might already be in your PATH, or use slashs for MinGW.

Using al

al lays into two module trees:

• Sound.AL
• Sound.ALC

The former exports anything you might need to use the OpenAL API while the latter exports symbols about the OpenAL Context API. Please feel free to dig in each tree to import specific types or functions.

OpenAL documentation

al doesn’t have any documentation for a very simple reason: since it’s a raw binding to the C OpenAL API, you should read the C documentation. Translating to Haskell is straightforward.

The OpenAL 1.1 Specifications

What’s next?

Up to now, al doesn’t cover extensions. It’s not shipped with ALUT either as I want to keep the package low-level and raw – and don’t even try to make me change my mind about that ;) .

If you have any issues, please let me know on the issues tracker. I’m responsive and I’ll happily try to help you :) .

Once again, have fun, don’t eat too much chocolate nor sweets, and happy Easter Day!

See, the type of application you need to write to make a demo is a bit special. Most programs out there are often either oneshot or reactive.

• A CLI program, an encoder/decoder, a math equation solver or a script are often oneshot programs. You give them a list of inputs and parameters and they do something based on their inputs. They’re are most of the time non-interactive.
• Programs such as GUIs, editors, video games, simulations etc. are often reactive, meaning that they mostly sleep until an event happen (a key press, mouse movement, etc.). For some, sleeping is reduced because they have a lot to do (handling network connections, performing I/O operations, streaming audio, rendering frames to the screen, etc.).

That binary split is not absolute, of course. Some programs don’t belong to any of the two prevous sections (think of a driver, a firmware, etc.). For the purpose of this article, though, it will be enough.

At some extent, we could state that a demoscene production, a video game or a simulation are also reactive programs. They mostly react to time as well as user interaction, network events, etc.. About time, you could imagine dividing time passing with very small quanta and picture time as a virtual clock that ticks such quanta. Every time a new quantum of time is emitted, the whole simulation is notified and reacts. However, representing that problem this way is not necessarily the best idea. See, a function of time is often a continuous function and then, it has an output value for any value of its input that lies in its domain. That property is interesting for us as we can get a value at any time with an arbitrary precision (the boundary being the precision at which we can represent a number on a computer).

The thing is, the kind of program we want generates its own inputs based on, mostly, the speed at which the hardware it’s running on is able to render a complete frame. The faster the more accurate we sample from that continuous function. That is actually quite logical: more FPS means, literally, more images to sample. The difference between two images will get less and less noticeable as the number of FPS rises. That gives you smooth images.

The “challenge” here is to write code to schedule those images. Instead of taking a parameter like the time on the command-line and rendering the corresponding image, we will generate a stream of images and will do different things at different times. Especially in demoscene productions, we want to synchronize what’s on the screen with what’s playing on the audio device.

The overall idea

From my point of view, we need at least two mechanisms of synchronization:

• A high-level synchronization mechanism, to state how globally the application will behave. I also like to see that kind of mechanism as a discrete space problem. You don’t have values to interpolate or sample from but only several pure values to “jump” from one to another every time the simulation time passes a given point in time. This kind of synchronization will state things like:
  • When the application starts, display this scene and let it play for 3 seconds.
  • Then after 3 seconds, switch to this other scene for 5 seconds.
  • After 8 seconds, draw this nice little cube and make it crash into a plane for about 10 seconds. Repeat that scene once more.
  • End the application.
• A low-level synchronization system. That would be used for a movement, a color change, a camera change, etc. This kind of synchronization is, to me, a continuous space problem. It can be solved by sampling from a curve, for instance.

Both those problems are solved by two crates I wrote lately. Respectively, awoo and splines. This blog post is about awoo. splines already has its own dedicated articles here and here. Nevertheless, I will make another blog article about it because I have new ideas I will add to the crate to enrich the splines experience.

awoo and the if / else if problem

Taking on the example of the high-level synchronization described above, one can write quickly the following naive yet working snippet:

let mut time_ms: f32 = 0.;

loop {
  if time_ms <= 5. {
    // render scene 1
  } else if time_ms <= 8. {
    // render scene 2
  } else if time_ms <= 20. {
    // render scene 1 again
  } else if time_ms <= 25. {
    // render scene 3
  } else
    break; // quit
  }

  time_ms += 0.01; // 100 FPS
}

That code is typical in demoscene production when we have to rush or even if we have a few scenes to write. However, it has several problems:

• It’s not really elegant. All those ifs seem like naive and not necessary code.
• The more time passes, the more conditions and branching we will do. What it means is that even if it’s not noticeable (testing floating-point numbers like that is really fast so it won’t change much), it’s easy to get that the second scene requires two tests in order to be rendered while scene 3 requires four and scene 1 requires either one or three!
• We notice that, every time a condition evaluates to true, all the previous branches can be completely discarded — we don’t have to test them anymore! — because time will never go backwards in a simulation (that is a strong assumption and it’s not true if you’re debugging, but for a release application, it is).

So, how can we do better? The idea is actually pretty simple. We want a very simple form a finite-state machine. In our case, the states are just what’s inside our ifs; the initial state is the first scene being rendered and the transitions are a predicate on the current time. Straightforward, right?

The idea of awoo is exactly that: allowing you to write the previous code like this:

let windows = vec![
  Window::new(0., 5.).map(|_| println!("hey, it’s scene 1!")),
  Window::new(5., 8.).map(|_| println!("hey, it’s scene 2!")),
  Window::new(8., 20.).map(|_| println!("hey, it’s scene 1 again!")),
  Window::new(20., 25.).map(|_| println!("hey, it’s scene 3!")),
];
let mut scheduler = SequentialScheduler::new(
  SimpleF32TimeGenerator::new(0., 0.01),
  windows
);

scheduler.schedule();

The code is now declarative and easier to read. Internally, the SequentialScheduler used here will make a single test to know which code it has to run. The implementation is not the typical implementation you would find for a FSM (finite-state machine), which uses a graph, but it’s akin.

You might be wondering why we do that map stuff instead of creating a Window directly with the actions. The answer is simple: a Window doesn’t hold any actions. That allows for creating windows via JSON, for instance, without having to deal with closures (I have no idea how that would even be possible with JSON). The idea is then to zip your windows to your actions by using a hashmap, for instance. This following snippet showcases exactly that (fully available here):

const WINDOWS: &str = r#"
{
  "a": {
    "start": 0,
    "end":   3
  },
  "b": {
    "start": 3,
    "end":  10
  }
}"#;

let windows: HashMap<String, Window<f32>> = from_str(WINDOWS).expect("cannot deserialize windows");
let a = windows.get("a").unwrap().map(|t| println!("in a: {}", t));
let b = windows.get("b").unwrap().map(|t| println!("in b: {}", t));
}

What gets interesting is that you can write your own time generator to manipulate the simulation in other ways — and you can also use different schedulers regarding what you do with time. For instance, you can imagine implementing a time generator that gets time from a HTTP request, a keyboard, a network socket, etc. and then control your simulation with external stimuli.

What happens when the escape key is pressed and that you need to stop the simulation in order to quit? Simple: you need an interruptible scheduler. awoo offers that as well in this form:

use std::sync::mpsc::channel;

let (sx, rx) = channel();
let mut scheduler = create_your_scheduler();

scheduler.interruptible_with(move |_| {
  // here, the closure’s argument is the time at which the scheduler is checking for interruptions
  if let Ok(_) = rx.try_recv() {
    Interrupt::Break
  } else {
    Interrupt::Continue
  }
});

scheduler.schedule();

Here, we use the spinning loop of the scheduler to check for interruptions in a straight-forward way.

So far, I have to admit I haven’t digged the async, await and Future concepts in Rust too much. For a single reason: discussions around those concepts have been heated and I will wait for an official announcement of the feature. Schedulers, especially as simple as the ones in awoo, don’t necessarily requires such IO features but the interruptible feature might. To me, the current implementation of interruptible schedulers in awoo is sufficient, especially for animation purposes — I might even add that feature directly in awoo so that you don’t have to do it by hand.

About the scope of the crate

Currently, the crate’s scope is very narrow — and I actually like that. A tight and small scope implies a better visibility about what the crate must do and how it must do it. The crate is currently simple and it might get more and more complex stuff as needs appear. As I always tell other developers and engineers, I don’t like to overthink too much features I don’t even need. Obviously, it’s important to keep planning possible future additions… But not too much. This is why that crate’s scope, if augmented, will only and always revolve around the concept of scheduling animation code. It’s currently an experimental crate and I’m trying to write demos with it, so we’ll see what time thinks about it.

So that’s all for me for today. I hope you liked it. Keep the vibes!

• neovim, my main, daily editor I use for pretty much almost all projects I work on.
• IntelliJ IDEA, that I currently use at work to hack on Java codebases.
• VS Code, that I have been trying mainly in Rust, TOML and Markdown.
• emacs, that I highly hacked around to play on my Haskell and Rust codebases (and YAML / Markdown / TOML too).
• DOOM Emacs, that I tried when I saw a colleague in a previous company I worked in (I was impressed by the “united” feeling of the UI and by how everything looked so slick). So I gave it a try.
• atom, GitHub’s take on editors. Same thing, mostly Rust, Haskell, etc.
• A bunch of others that I won’t talk about because I quickly stopped using.

The goal of this article is to create a temporal “snapshot” of my views on editors and what I think about the current situation. I have been using vim and especially neovim for more than a decade. I need to explain about my current workflow and what I cherish in editing before talking about each editors. Expect a point of view from a neovimer which is looking around at lots of editors.

This is only a personal workflow / point of view that works well for me right now. It doesn’t mean it will for you and it doesn’t mean a different workflow would be worse.

• What I think powerful editing should be
  • Keyboard layout
  • Modal editors
  • How I like to move around
  • All the other modal candies
• The editors
  • neovim
    • My neovim setup
    • What I like about neovim
    • What I dislike about neovim
  • IntelliJ IDEA
    • What I like about IntelliJ IDEA
    • What I dislike about IntelliJ IDEA
  • VS Code
    • What I like about VS Code
    • What I dislike about VS Code
  • emacs and DOOM emacs
    • What I like about emacs / DOOM emacs
    • What I dislike about emacs / DOOM Emacs
  • atom
    • What I like about atom
    • What I dislike about atom
• Wrap it up

What I think powerful editing should be

Keyboard layout

I am French and I’m using a keyboard layout that is made to type very quickly in French and to code. With hindsight, since I type more often in English than in French, maybe I should have picked another keyboard layout, but the coding experience in my keyboard layout is really great, so I stick around.

The keyboard layout is bépo. I learned bépo the “recommended” way — i.e. you have to practice typing (« dactylographie » in French). It means that I use all my fingers to type on a keyboard, and that each key on the keyboard is assigned a single finger that will press it. That helps a lot with muscle memory and to reduce wrist pain (my wrists barely move when I type on a keyboard), among other things. The typing speed is a side-effect of being accurate and having good comfort (if you are curious, I’m pretty fast but there are faster people — I type at around 120 to 130 words per minute). Because I think the speed doesn’t matter when programming, I think the most important part to remember here is the comfort: the wrists don’t move and my fingers fly around the keyboard, whatever the speed.

Modal editors

I think a modal editor is superior, for various reasons. The first one is that I truly hate having to use a mouse for something that can be done without having to move around hundred of pixels with your cursor and clicking on buttons. For instance, running an application, on my current machines, is simply typing alt d, the name of the program (I typically use completion, so I never type fully the name of the program) and hit enter. All this without moving my hands from the keyboard. And I do that for programs like firefox, kdenlive, etc. but for terminal applications, I simply type and complete them in my terminal, which I open simply with alt return.

So, using a mouse to move around a cursor in an editor feels like completely suboptimal to me, especially because we write code (i.e. we type on a keyboard) most of the time, so moving around with the mouse implies several back and forth movements between the keyboard and the mouse. Maybe you don’t care and it’s cool to you, but to me, this is truly horror land. I feel very uncomfortable when doing this.

Also, modern editors that are not modal typically make people move by using the arrows keys, which are either far on your keyboard, or — like my keyboard, a 60% home made one — not in direct access and then require a function key to enable them.

So that’s the first reason why I like modal editors. They make a smarter use of your keyboard for simple yet recurrent features, like moving around — e.g. h j k l. The second reason why I like them is because of the facts they have a completely new mode for non-modal editor (i.e. the normal mode), you have a whole keyboard / lots of keys to bind actions to and do lots of things people usually do with the mouse. Being able to split an editor into several buffers, move around the buffers, go at the beginning of the paragraph, search and replace, register actions into macros and replay them, etc. All this without even moving the wrists. The learning curve is steep if you’re used to the mouse, but once you’ve passed the mental barrier, really, and this is a personal opinion, but I truly think that using the mouse again after that feels like handicap to me.

How I like to move around

When I look at people coding, I see several kind of programmers:

• The ones who move with the arrows or with h j k l in modal editors. You can spot them very easily at how the cursor moves in a document. It typically implies keeping a key pressed until the cursor reach a given row, then pressing another key until the cursor reach a given column and then adjust if they went passed the location they had in mind.
• People using the mouse, by clicking at the place they want to put the cursor at.
• People using relative numbers. That’s an enhancement of the first group: they typically use relative numbers to know which lines they want to jump to very quickly, so that they don’t have to press the up / down keys for ages until reaching the line they want to. Instead, they look at the number on the lines, press that number, and the direction, and bam, they are on the lines. They then use typical motions from vim like $ (go to end of line), f (go to the first occurrence of the next character typed after f, like f( will make your cursor go to the next (), % (go to the matching delimiter) or w (go to the beginning of the next word) / b (go to the beginning of the previous word), etc. to move faster on the same line (it works across lines too).

I think that represents 99,9% of what I see people do. Obviously, you will get that I don’t belong to the second set of people… but I don’t really belong to any, actually. How I move is, I guess, convoluted for most people and I know some people won’t understand how it can feel. I use h j k l and all the motions from vim described in the third group (and even more; I full lecture of four hours would be required to explain all of them :D), but it all depends on the distance I need to travel. If my cursor is on a word and I want to move to the beginning of a word located very closely to my cursor on the same line, I’ll simply type www if it’s three words apart (or 3w if I’m feeling funny). If the distance is higher, I use a tool called [EasyMotion].

Easymotion really is a wonderful tool. The idea is that it has several modes, depending on the kind of move you want to perform:

• By lines: this mode allows you to jump to any line in the current (or all open) buffers.
• By words: tihs mode allows you to jump to any “word” in the current (or all open) buffers.
• By characters: when the word mode doesn’t help with jumping to a special operator or character (because it’s not recognized as a word), this mode allows you to jump to any character in the current or (or all open) buffers.
• It has other modes but I have never really found useful cases for them.

The way I typically do it is by mapping the three modes to <leader>l, <leader>w> and <leader>c (in my case, <leader> is the space key).

Typing SPC l in my current buffer results in this:

Typing any highlighted character will make my cursor jump to it. The same applies for words, with SPC w:

For the character mode, after pressing SPC c, I have to press another character (the one I want to jump to). Let’s say we want to jump to a # (which is not part of words): SPC c #:

This way of moving is not intuitive at first, but once you get used to it… it’s a must have.

All the other modal candies

Among all the things that I like about modal editing, here is a non-exhaustive list of features I expect to have around my fingers:

• C-i and C-o: those allows me to jump to something / a file / a place in a buffer and then go back to where I was right before with C-o (read it like out) or go back again with C-i (read it like in).
• Macros and registers: those allow me to yank content into different registers (like clipboards) by assigning a single key to paste their content. For instance, I can put a few lines in my t register with "tyi( (“put in the t register the yank inside matching (), and paste that content later with "tp. Macros allow more powerful editing control by assigning a key to set of actions with the q keyword (qa will register all the next keystrokes and actions into the a macro). Then simply replay the macro with @a, for instance.
• Obviously, all the basic vim motions, like d to delete, y to yank, c to change, t to go to the character right before the one you search, % to go to the other delimiter, etc. And more complex text manipulation, such as “Let’s change what’s inside this function parameter list, delimited by (”: ci(.

It would take too much time to list everything, but the main idea is: I need the modal features when editing code.

The editors

So let’s talk about the list of editors I mentioned in the preface. The idea is to give my own opinion on those editors. What I like about them regarding the way I like to edit code and what I think is missing.

neovim

I currently use neovim in its TUI version, because it’s the most stable, fast and easy neovim experience out there so far to me. I have tried several GUI versions but never found what I was looking for — the main reason being that they pretty much all of them use Web™ technologies, and it’s a hard no to me. I think I should elaborate a bit more about that last point.

Why not using Web tech:

• Editing something on my machine has nothing to do with Web technology. This applies to lot of other things, but truly, running a complex JavaScript VM / CSS engine in an editor seems so wrong on so many levels (performance being the first one).
• Most of the time, editors based on Web tech take ages to load. Yes, even VS Code — remember that my daily editor is neovim, which loads in around 23ms with almost 50 packages installed (you can profile with :profile to get that value). Among those 23ms, coc.nvim take around 12ms.
• Scripting. Scripting in JavaScript or CoffeeScript is a hard no to me.
• npm is one of the worst piece of software ever written. Please don’t make me use it again.

My neovim setup

So I use several plugins that I’m going to describe. I think it’s important that people know about all those because, in the collective mind, people still think vim / neovim are editors from the past. Well, not really.

• ryanoasis/vim-devicons
  • Add lots of unicode icons that other packages will use to provide a fancier and sexier interface experience.
• sainnhe/sonokai
  • The current colorscheme I am using. It is very similar to what you get with DOOM Emacs — even though it’s a bit less contrasted.
• neovimhaskell/haskell-vim
  • Haskell syntax support for neovim.
• rust-lang/rust.vim
  • Rust syntax support for neovim.
• plasticboy/vim-markdown
  • Markdown support for neovim. By default, neovim already provides a good support for Markdown, but this package has several cool features, like folds. I might get rid of it as I’m using folds that much.
• mzlogin/vim-markdown-toc
  • A very cool package that gives a way to create table of content in Markdown buffers and let neovim automatically update the sections when you edit headers.
• tikhomirov/vim-glsl
  • GLSL syntax support for neovim.
• cespare/vim-toml
  • TOML syntax support for neovim.
• ElmCast/elm-vim
  • Elm syntax support for neovim.
• idris-hackers/idris-vim
  • Idris syntax support for neovim.
• posva/vim-vue
  • Vue.js syntax support for neovim.
• baskerville/vim-sxhkdrc
  • sxhkd support for neovim.
• norcalli/nvim-colorizer.lua
  • A super cool extension that will automatically change the background color of texts containing an hexadecimal value, such as #f8324F or #42cf69:
  • 
• airblade/vim-gitgutter
  • One of the best plugin I have installed. It provides the gutters you see on the top of your buffer when editing a file versioned in git (added, modified, deleted, etc.):
  • 
  • But it can do more, and among all the things it can do, it can help you preview hunks, stage or discard them inside your editor. That’s just a blast to me:
  • 
• tpope/vim-fugitive
  • All the features you expect to find from git inside neovim. You can see diff, resolve merge conflicts, write commit messages, etc. etc. However, I tend to still keep around the command line as I’m not completely solved on how this plugin deals with fixups and squashing in general.
• rhysd/git-messenger.vim
  • A blame-at-cursor tool. Not really that useful actually, as fugitive already has a git blame window that will annotate each line with the commits. I sometimes use this one, but I might get rid of it.
• tveskag/nvim-blame-line
  • A git blam inlined, at the right part of your lines. Very similar to the default git plugin from VS Code, for instance.
• junegunn/fzf.vim
  • If there is one plugin you should install, it’s this one. It has so many features: opening files, git files, buffers, rip-grepping, searching through history, commands, colorschemes, etc. etc. And as the name implies, it uses fzf as a backend, so you get a very cool fuzzy search experience (with refined search, which I struggle to find in any other editor — i.e. you can type something, then put a space and type again to match things occurring before the actual match).
• machakann/vim-highlightedyank
  • A funny one: it will highlight the lines / objects you yank for a better visual feedback. Surprising neovim doesn’t have that by default.
• liuchengxu/vista.vim
  • I use this one from time to time to get a symbol tree, but the output is not really appealing to me right now. I might ditch it too.
• neoclide/coc.nvim
  • The best completion engine for neovim, by far. I used to use others, such as ale, but this one is ace. It basically provides you with LSP completion for lots of languages. It has an integrated marketplace you can use to install new LSP servers and integrations, and it even has support for things completely unrelated (a bit surprising, I think these should be standalone plugins), such as coc-explorer (which is a replacement of NERDTree to me now), coc-snippets, etc.
• tpope/vim-commentary
  • Easily comment / uncomment lines without having to insert the comment symbols yourself.
• liuchengxu/vim-which-key
  • which-key from emacs, but for neovim. Basically, when setup properly, it will provide you with a visual list of possible keybindings for sequences. I think it’s not that useful (or it is for people trying to use your configuration or if you install a plugin that ships a lot of keybindings — which I truly dislike, btw), but it looks pretty cool.
• itchyny/lightline.vim
  • A status line that looks pretty cool.
• SirVer/ultisnips
  • Snippets support. The snippet engine of ultisnips allows for a lot of room, like interpolation via shell, viml, python, etc.
• honza/vim-snippets
  • A collection of snippets for commonly used languages and file formats.
• junegunn/vim-easy-align
  • A pretty neat plugin to easily align texts / tables with a few keystrokes only.
• liuchengxu/vim-clap
  • A take on uniting all possible source of search / fuzzy finders in a modern and fast UI. Unfortunately, right now, the plugin is pretty unstable to me, so I keep using fzf instead.
• easymotion/vim-easymotion
  • I think I have explained what that is about earlier. ;)

What I like about neovim

• It’s fast. It really is. It opens immediately. Moving around, scrolling, etc. is smooth, whatever the terminal I’m using (even though I’m currently using alacritty).
• The plugins and effort put into neovim are really great. I love vim-gitgutter so much; I love the colorizer plugin a lot too. coc.nvim has been a blast so far (for most part). EasyMotion is typingporn to me. fzf is so fast it should be illegal.
• neovim has a community that is truly passionated about what they are doing, and new versions add lots of very cool features that we quickly adopt into new plugins, such as the popup / floating windows / virtual texts for linter annotations, etc. etc.
• It’s lightweight: your RAM will thank you.
• Something I haven’t talked about but is a killer-feature of vim / neovim: the :help pages. I think no other software has such a cool set of help pages. Really, give it a try. You want to know how to configure coc.nvim? Simply type :help coc-nvim.
• Plugin managers exist (I personally use vim-plug but you will find more), and they make your life so much easier.

What I dislike about neovim

• Even though I love how fast the TUI is (I truly haven’t come across anything faster so far), TUIs feel like hacks to me. For instance, if you split a window into two buffers, the vertical “marks”, “edges”, whatever-you’d-like-to-call-them, are actual unicode characters. The way terminals work make it possible to ignore those characters, but still, it feels hacky. If you want something like a minimap or a simply thin frame around some text, or any kind of visual feedback that is a bit more complicated than simply turning bold mode or underlying, you’re basically not going to get it.
• The GUIs for neovim are… not up to my expectations. Most of them are based on Web tech anyway, so they’re not good candidates to me. For the rest, like clients based on Qt, they feel a bit abandonned :(. There are people trying to make new ones, but they are not ready AFAIK.
• coc.nvim sometimes feel like is doing the wrong thing. For instance, when editing Java, it timeouts very often when trying to jump to the definition of a symbol (or simply looking a symbol up). That’s bad.
• Changing my colorscheme while the editor is running is a waste of time and generates broken syntax highlighting everywhere. A pity to me. :(

IntelliJ IDEA

So I won’t going to be able to talk too much about this one, because I have only started using it at work (community edition). I’m using the vanilla one, with few modifications to none. I edit only Java with it.

What I like about IntelliJ IDEA

The Java support is truly flawless. It does a lot of things for you, and among them, I was really impressed by:

• The refactor mechanism that would allow me to select a block of code inside a function, ask the editor to “move that into a dedicated function”. So far, all editors can do that, but what impressed me is that IntelliJ IDEA was able to find out which variables needed to be captured and put as arguments to the function, then automatically pass them at call site, when replacing the block of moved code. Really neat.
• The speed at which you can look for symbols, look for implemented functions, inherited classes, super classes, etc. It’s all instant and it’s all very well presented to you. Me gusta.
• The syntax highlighting is okay; I especially like the inlined type ascriptions for both var declarations but also when passing arguments to functions.

What I dislike about IntelliJ IDEA

• I’m using the community edition, which only supports Java and a bunch of other configuration languages. You don’t get the profiler, for instance — which would be very well appreciated!
• Even though the asynchronous part of the editor is impressive, it sometimes index projects at random times and if you’re working on a laptop, brace yourself for Mars landing as the machine is lifting off! — i.e. it makes a hella noise and heats up quickly.
• I tried the Vim integration and I wasn’t able to use it correctly with my bépo keymap. Unable to remap some motion and/or mode switches. I disabled it and cried deeply.
• Without Vim support, even though the editor has a lot of shortcuts, it feels like you still have to use the mouse to perform very basic tasks.

VS Code

So this one is an important one, as it’s Microsoft’s take on editing. Everybody seems to love VS Code, and I get why. The UI is slick and fast — for a Web-based editor… ;). LSP support is obviously premium and flawless. It has a lot of community plugins, themes and integrations. For most part, even though it’s Web based, I have been enjoying it.

What I like about VS Code

• The slickness of the editor / UI.
• Language support is premium with LSP and editing with it feels very solid.
• Millions of plugins.
• Big community and most of people use it nowadays, so I guess that if you get any trouble, you can ask around?

What I dislike about VS Code

• The fact it’s written with a Web tech. Among all editors Web-based, it’s the fastest, but still, if you’re used to vim / neovim, you’ll be dissatisfied with it.
• The vim integration is poor / doesn’t work (I tried remapping h to c — remember, bépo keyboard layout). It simply doesn’t work.

emacs and DOOM emacs

I have been using emacs lately (vanilla) as I saw a colleague using DOOM Emacs. I’m putting both in the same section as they are pretty similar. The way I see emacs and DOOM Emacs could be summed up with one word: united. I don’t know how they do that, but all plugins blend so well with each other. I’m using the ivy interface for completion and fuzzy search, and everything is so… well done. The UI is gorgeous, themes are awesome (I like the default dark theme, DOOM One), the editor is pretty fast — still slower than neovim to me, especially when scrolling, but still much faster than a Web-based editor.

What I like about emacs / DOOM emacs

• Once setup correctly (better defaults, etc.), the editor feels very modern (like what you would find in VS Code / atom). It feels slick and well designed.
• Evil-mode, the Vim mode, is to me the best out there (besides vim and neovim themselves, obviously). They cover almost everything — they even have support for EasyMotion!
• You script / configure it with Lisp, which is… super great! Lisp is that kind of old treasure that has been around forever and still feels like something new. I really love it.
• If you are using DOOM Emacs, you are going to get a lot of candies for free. Its modules approach works pretty well and provides a very innovative way to enable / disable features. There are tons of resources out there to get into DOOM Emacs and I highly suggest having a look, even if you don’t plan on using emacs or DOOM Emacs. That’s how I discovered which-key, for instance — that I now use in neovim.
• The LSP integration is pretty well done. It will download for you the servers and if you open a file that you have never gotten the server for before, it will gently ask you whether you’d like to.
• On a more general note, both emacs and DOOM emacs feel more interactive than editors like vim or neovim, which I think is a definitive better approach.
• It uses gtk as main backend on Linux. I think it’s important to note it, because it’s not Web-based! (yay!)
• Magit is a wonderful tool.
• Org-mode is really great too, even though I think it’s a bit too big to me.
• The daemon mode is amazing and I think all editors should have it. It allows you to start an instance of emacs and connect emacsclient to it, reducing the loading time to zero. Very brilliant and very useful!

What I dislike about emacs / DOOM Emacs

• Moving across a huge amount of code leads to stuttering and sometimes feels a bit meh, especially if you’re used to vim / neovim. Most of the time, it should be okay, but just keep in mind that scrolling in emacs has always been a problem.
• Even though it could be seen as a positive point, I think that all the great plugins emacs has make it very bloated, and that’s a problem to me. For instance, Org-mode, which is a wonderful piece of software, should have benefited much more people if it was a standalone application. To me, it feels like starting using emacs will lead to using your computer to run emacs, and all your applications inside emacs. It even has an IRC plugin and an email reader plugin!
• I’m not sure what’s happening with this one, but LSP servers feel… synchronous?! When opening a file for the first time, the LSP server starts and you have to wait a few seconds before being able to jump into the file. I don’t really know whether it’s a configuration thing, but it doesn’t feel very pleasant.
• The defaults of emacs are really, really bad. And making the whole thing like what you get with DOOM Emacs is going to cost you lots of hours reading documentation and experimenting with your configuration. I enjoyed doing it but in the end… why simply not ship emacs with those defaults? Is it due to historical reasons that no one cares about anymore nowadays? /stares at vim

atom

I’ll finish with atom, GitHub’s editor. I remember my first reaction when running atom for the first time was: “Woah, that’s a pretty editor.” The default colorscheme, One, is a universal colorscheme everybody knows. There are have been so many forks of it in so many different editors.

atom looks a lot like VS Code to me, but is a bit prettier in its UI — I do prefer atom’s UI to VS Code’s. The UI is slick and very clean. You can find a lot of extensions / plugins / themes, from LSP integrations to Markdown previews and Vim modes.

What I like about atom

• The killer feature of atom to me is its ability to tell you what commands are associated (or erased) for a keybinding you are pressing, live. It is very practical to debug keybinding issues and I wish more editors had it. Other editors feature similar stuff, but not quite as good as this echo mode for keybinding.
• The themes are pretty cool and the whole typing experience / completion is pretty solid and consistent.
• Lots of plugins to play with.

What I dislike about atom

• Vim mode. Once again, it’s not complete to me as it doesn’t support well my bépo keyboard layout. Worse, they have a weird bug with alt-gr (they call it altgraph in the config), that is not correctly recognized. It sometimes work but I never recall the steps I have to do to fix the problem, and most of the time, I spend a lot of times finding workarounds for something that just works in terminals.
• It’s slow af. You can feel the whole Web with you! :D
• Sometimes, after an update, a plugin will break, and you will basically lose a feature. This is a problem that I have observed with other Web-based softwares (such as the GNOME Desktop Environment), which makes me doubt more and more that kind of tech choice.

Wrap it up

When I started coding, I remember of people talking about IDE / editor wars. Today, as I have tried a lot of editors, I can say that there’s no such thing as an editor war to me. All editors have drawbacks and picking the right editor is often a matter of taste and personal experience. I’m a keyboard lover (I make my own keyboards) and I truly love typing — not necessarily code, hence it was pretty obvious to go to emacs and vim back then (I actually started coding in emacs). Several years later, I joined the vim side and neovim. It’s only one year ago that I started looking at emacs again to see what has changed. And oh my. So many interesting things out there!

What I like about testing editors is that each and every editor has at least one killer feature no other has:

• vim and neovim have modal editing and are super fast. They have been the editors of choice for modal editing for decades and all the motions, macros, commands and mnemonics are implemented the best in those two editors.
• IntelliJ IDEA has, to me, the best Java experience out there, with impressive (and so useful!) refactoring features. It’s not something you’d be absolutely need to be productive, but it will make you feel much comfortable working on your Java codebase, and I truly wish a plugin existed for that in the editors I use!
• VS Code has the best LSP implementation and has — I guess? — on the biggest community. If you like Web-based editors, don’t seek no more: it’s the right editor for you.
• emacs and DOOM emacs have that slick, united interface with lots of amazing plugins and applications. You’ll completely adore Org-mode, Magit and lots of other plugins!
• atom has that echo mode for keybindings, superb defaults for theming and syntax highlighting and is one of — if not the most? — the friendliest editors out there.

Spending some weeks in all those editors made me realize something about vim / neovim: I don’t necessarily think I should be using them, especially since I have used emacs / DOOM Emacs with its Evil-mode. Today, my thoughts revolve around the idea that a good neovim client could be a gtk application like emacs: slick, united, with great defaults and full support of neovim features, with support for gtk floating windows and popups (as it’s native in neovim and feels a bit hacky in TUI). We already have great plugins for things like git (fugitive / vim-gitgutter), completion and syntax highlighting with coc.nvim / vim-lsp / vim-treesitter. The only piece that is missing to me is a great GUI to leverage the “hacks” we have to do in TUI to actually use real popups, “balloons”, etc. Once we get a decent neovim GUI, I think I will have found my favorite editor of all time.

Until then, I’ll stick around neovim TUI as it’s what is as close at what I would like to get. I hope this article will have given a bit of hints about what a vim / neovim enthusiast might expect from a modern editor. And I say a vim enthusiast, not all of them. We all look for different things and I truly thing we live in a wonderful world with all those editors out there. They won’t fit for everybody, but everybody will find one for them, and damn, that’s a pretty good conclusion to this article.

Feel free to tell me about what you think about your editor, whether there’re things you dislike about it or what you would like to see appear in it. Keep the vibes!

Origin

luminance started as a Haskell package, extracted from a “3D engine” I had been working on for a while called quaazar. I came to the realization that I wasn’t using the Haskell garbage collector at all and that I could benefit from using a language without GC. Rust is a very famous language and well appreciated in the Haskell community, so I decided to jump in and learn Rust. I migrated luminance in a month or two. The mapping is described in this blog entry.

What is luminance for?

I’ve been writing 3D applications for a while and I always was frustrated by how OpenGL is badly designed. Let’s sum up the lack of design of OpenGL:

• weakly typed: OpenGL has types, but… it actually does not. GLint, GLuint or GLbitfield are all defined as aliases to primary and integral types (i.e. something like typedef float GLfloat). Try it with grep -Rn "typedef [a-zA-Z]* GLfloat" /usr/include/GL. This leads to the fact that framebuffers, textures, shader stages, shader program or even uniforms, etc. have the same type (GLuint, i.e. unsigned int). Thus, a function like glCompileShader expects a GLuint as argument, though you can pass a framebuffer, because it’s also represented as a GLuint – very bad for us. It’s better to consider that those are just untyped – :( – handles.
• runtime overhead: Because of the point above, functions cannot assume you’re passing a value of a the expected type – e.g. the example just above with glCompileShader and a framebuffer. That means OpenGL implementations have to check against all the values you’re passing as arguments to be sure they match the type. That’s basically several tests for each call of an OpenGL function. If the type doesn’t match, you’re screwed and see the next point.
• error handling: This is catastrophic. Because of the runtime overhead, almost all functions might set the error flag. You have to check the error flag with the glGetError function, adding a side-effect, preventing parallelism, etc.
• global state: OpenGL works on the concept of global mutation. You have a state, wrapped in a context, and each time you want to do something with the GPU, you have to change something in the context. Such a context is important; however, some mutations shouldn’t be required. For instance, when you want to change the value of an object or use a texture, OpenGL requires you to bind the object. If you forget to bind for the next object, the mutation will occurs on the first object. Side effects, side effects…

The goal of luminance is to fix most of those issues by providing a safe, stateless and elegant graphics framework. It should be as low-level as possible, but shouldn’t sacrifice runtime performances – CPU charge as well as memory bandwidth. That is why if you know how to program with OpenGL, you won’t feel lost when getting your feet wet with luminance.

Because of the many OpenGL versions and other technologies (among them, vulkan), luminance has an extra aim: abstract over the trending graphics API.

Types in luminance

In luminance, all graphics resources – and even concepts – have their own respective type. For instance, instead of GLuint for both shader programs and textures, luminance has Program and Texture. That ensures you don’t pass values with the wrong types.

Because of static warranties provided by compile-time, with such a scheme of strong-typing, the runtime shouldn’t have to check for type safety. Unfortunately, because luminance wraps over OpenGL in the luminance-gl backend, we can only add static warranties; we cannot remove the runtime overhead.

Error handling

luminance follows the Rust conventions and uses the famous Option and Result types to specify errors. You will never have to check against a global error flag, because this is just all wrong. Keep in mind, you have the try! macro in your Rust prelude; use it as often as possible!

Even though Rust needs to provide an exception handler – i.e. panics – there’s no such thing as exceptions in Rust. The try! macro is just syntactic sugar to:

match result {
  Ok(x) => x,
  Err(e) => return e
}

Stateless

luminance is stateless. That means you don’t have to bind an object to be able to use it. luminance takes care of that for you in a very simple way. To achieve this and keep performances running, it’s required to add a bit of high-level to the OpenGL API by leveraging how binds should happen.

Whatever the task you’re trying to reach, whatever computation or problem, it’s always better to gather / group the computation by batches. A good example of that is how magnetic hard drive disks work or your RAM. If you spread your data across the disk region (fragmented data) or across several non-contiguous addresses in your RAM, it will end up by unnecessary moves. The hard drive’s head will have to go all over the disk to gather the information, and it’s very likely you’ll destroy the RAM performance (and your CPU caches) if you don’t put the data in a contiguous area.

If you don’t group your OpenGL resources – for instances, you render 400 objects with shader A, 10 objects with shader B, then 20 objects with shader A, 32 objects with shader C, 349 objects with shader A and finally 439 objects with shader B, you’ll add more OpenGL calls to the equation – hence more global state mutations, and those are costly.

Instead of this:

1. 400 objects with shader A
2. 10 objects with shader B
3. 20 objects with shader A
4. 32 objects with shader C
5. 348 objects with shader A
6. 439 objects with shader B

luminance forces you to group your resources like this:

1. 400 + 20 + 348 objects with shader A
2. 10 + 439 objects with shader B
3. 32 objects with shader C

This is done via types called Pipeline, ShadingCommand and RenderCommand.

Pipelines

A Pipeline gathers shading commands under a Framebuffer. That means that all ShadingCommand embedded in the Pipeline will output to the embedded Framebuffer. Simple, yet powerful, because we can bind the framebuffer when executing the pipeline and don’t have to worry about framebuffer until the next execution of another Pipeline.

ShadingCommand

A ShadingCommand gathers render commands under a shader Program along with an update function. The update function is used to customize the Program by providing uniforms – i.e. Uniform. If you want to change a Programs Uniform once a frame – and only if the Program is only called once in the frame – it’s the right place to do it.

All RenderCommand embedded in the ShadingCommand will be rendered using the embedded shader Program. Like with the Pipeline, we don’t have to worry about binding: we just have to use the embedded shader program when executing the ShadingCommand, and we’ll bind another program the next time a ShadingCommand is ran!

RenderCommand

A RenderCommand gathers all the information required to render a Tessellation, that is:

• the blending equation, source and destination blending factors
• whether the depth test should be performed
• an update function to update the Program being in use – so that each object can have different properties used in the shader program
• a reference to the Tessellation to render
• the number of instances of the Tessellation to render
• the size of the rasterized points (if the Tessellation contains any)

What about shaders?

Shaders are written in… the backend’s expected format. For OpenGL, you’ll have to write GLSL. The backends automatically inserts the version pragma (#version 330 core for OpenGL 3.3 for instance). In the first place, I wanted to migrate cheddar, my Haskell shader EDSL. But… the sad part of the story is that Rust is – yet – unable to handle that kind of stuff correctly. I started to implement an EDSL for luminance with macros. Even though it was usable, the error handling is seriously terrible – macros shouldn’t be used for such an important purpose. Then some rustaceans pointed out I could implement a (rustc) compiler plugin. That enables the use of new constructs directly into Rust by extending its syntax. This is great.

However, with the hindsight, I will not do that. For a very simple reason. luminance is, currently, simple, stateless and most of all: it works! I released a PC demo in Köln, Germany using luminance and a demoscene graphics framework I’m working on:

pouët.net link

youtube capture

ion demoscene framework

While developping Céleri Rémoulade, I decided to bake the shaders directly into Rust – to get used to what I had wanted to build, i.e., a shader EDSL. So there’re a bunch of constant &'static str everywhere. Each time I wanted to make a fix to a shader, I had to leave the application, make the change, recompile, rerun… I’m not sure it’s a good thing. Interactive programming is a very good thing we can enjoy – yes, even in strongly typed languages ;).

I saw that gfx doesn’t have its own shader EDSL either and requires you to provide several shader implementations (one per backend). I don’t know; I think it’s not that bad if you only target a single backend (i.e. OpenGL 3.3 or Vulkan). Transpiling shaders is a thing, I’ve been told…

sneaking out…

Feel free to dig in the code of Céleri Rémoulade here. It’s demoscene code, so it had been rushed on before the release – read: it’s not as clean as I wanted it to be.

I’ll provide you with more information in the next weeks, but I prefer spending my spare time writing code than explaining what I’m gonna do – and missing the time to actually do it. ;)

Keep the vibe!

Preliminary work

I’ve been using interpolation (linear, bilinear, trilinear, B-splines, etc.) for a while now because of my demoscene productions. My spectra crate features some interpolation primitives and I’ve been wanting to move them out of spectra for a while.

Basically, my need is simple:

• I use interpolation for a lot of situations: animation, procedural generation, editing, transitions, etc.
• I need several interpolation implicit methods / spaces:
  • Constant/step.
  • Linear / bilinear / trilinear.
  • Cosine.
  • Cubic Hermite spline.
• I also need splines with control points (knots):
  • Catmull-Rom.
  • Bézier.

That’s a lot of things. For what it’s worth, I’ll just use as reference a library I wrote years ago in Haskell, smoothie, doing exactly that.

However, because I didn’t want to bloat the ecosystem, I had a look around to see whether I could simply drop my code and add a simple dependency instead of extracting things from spectra and creating, again, new crates. Here are the reasons why.

bspline

This crate is about B-spline only and the interface is highly unsafe (panics if out of control points for instance). My current spectra code is already safer than that, so no reason to lose features here.

spline

This is… a… placeholder crate – i.e. it doesn’t have anything exposed. People do this to book a crate name for later. I highly dislike that and people doing this should really be warned not to do this, because they prevent others – who actually have some code to put there – from using the name. Plus, for this spline crate, I had already have a look years ago. In years, the so-called “author” hasn’t even uploaded anything and the name is just reserved… for nothing.

I highly suggest people from the Rust team to forbid people from doing this. This is just useless, childish and brings zero value to the community. Just remove that crate as no one has ever had the chance to ever depend on it.

trajectory

This is an interesting crate, but it’s way too much specialized to me. It could be very interesting to use it for camera and objects paths, but I also need splines for a lot of other situations, so this is too much restrictive (you can see functions like acceleration, position, velocity that works only for T: Float).

nbez

A – looks like – very comprehensive Bezier crate, but doesn’t provide anything for all the other interpolation I need.

Let’s build yet another crate!

Looks like the crate I’m looking for doesn’t exist yet – oh yes it does: it’s spectra!, but I want a crate that does interpolation only.

I could be using some of the crates listed above, add them as a bunch of dependencies to a glue crate and just expose the whole thing. However, I’m not sure they will compose quite correctly and I might just end up re-coding the whole thing.

So the crate I’ll be building will be about interpolation. You will find step (constant) interpolation, linear / bilinear / trilinear interpolation, cosine, cubice, Catmull-Rom and Bézier spline interpolations. The crate will be as abstract as possible to enable developers to plug in their own types. Because the spline name is already taken – duh! – I’ll use the splines name…

I’ll take my chance to try to take over spline first, though. I really don’t like that kind of things in an ecosystem, it’s just unaesthetic.

Note²: Ok, I tried, and seems like it’s more complicated than I thought. I’ll just go with the splines name then.

So this blogpost serves as an introduction to splines. I added a few unit tests to start off with and a very few examples in the documentation. Most of the interface is sufficient for now but more complex needs might show up. If you have any, please shoot an issue or even better, open a PR!

Keep the vibes and happy hacking!

I have a MSI GS60 Ghost Pro 2QE I’m very proud of. It’s sexy and powerful. It comes with a fancy and configurable backlit keyboard.

There’s a tool called SteelSeries Engine for Windows we can use to change the colors of the keyboard. It supports several features:

• changing colors of three parts of the keyboard (left, middle and right) ;
• changing modes (normal, breathe, wave, demo, gaming).

Unfortunately, that software doesn’t work on Linux, even with wine. I tried hard to make it work and never actually found a way to run it. Then, I decided to look for alternatives and… found nothing working.

Yesterday, I tried a node.js-powered tool called msi-keyboard. And it worked. However, the interface and user interface was not my cup of tea. I decided to dig in in order to understand how it works, and I decided to write my own tool with a decent interface.

HID access

The key idea is that such keyboards are just HID devices. As a Haskell programmer, I looked for something that would grant me access to such devices, but nothing was working. There’s a hidapi package, but it doesn’t work and has bugs.

I didn’t give up though. I wrote my own Haskell HID API binding, called hid. Several good things about it:

• it does work ;
• it’s simple ;
• I lately wrote a software using it.

Feel free to install and use it!

msi-kb-backlit

Then, I wrote another Haskell package, msi-kb-backlit. It might require super user rights to work. If you’re not a Haskeller, you can find installation details here.

Note: if you use Archlinux, you can directly download msi-kb-backlit through the AUR! Search for msi-kb-backlit with yaourt, or download the tarball.

The software has an embedded documentation to help you tweak with colors and modes. ;)

Feel free to use all those pieces of software. I made them with love for you all!

Enjoy your week end, and keep the vibe!

Just for the record, the initial goal I had in mind was to parse a subset of GLSL for my spectra demoscene framework. I’ve been planning to write my own GLSL-based shading language (with modules, composability, etc. etc.) and hence I need to be able to parse GLSL sources. Because I wanted to share my effort, I decided to create a dedicated project and here we are with a GLSL crate.

Currently, you can successfully parse a GLSL450-formatted source (or part of it, I expose all the intermediary parsers as well as it’s required by my needs to create another shading language over GLSL). See for instance this shading snippet parsed to an AST.

However, because the project is still very young, there a lot of features that are missing:

• I followed the official GLSL450 specifications, which is great, but the types are not very intuitive – some refactoring must be done here;
• I use nom as a parser library. It’s not perfect but it does its job very well. However, error reporting is pretty absent right now (if you have an error in your source, you’re basically left with Error(SomeVariant) flag, which is completely unusable;
• I wrote about 110 unit tests to ensure the parsing process is correct. However, there’s for now zero semantic check. Those are lacking.
• About the semantic checks, the crate is also missing a semantic analysis. I don’t really know what to do here, because it’s a lot of work (like, ensure that the assigned value to a float value is a real float and not a boolean or ensure that a function that must returns a vec3 returns a vec3 and not something else, etc.). This is not a trivial task and because this is already done by the OpenGL drivers, I won’t provide this feature yet. However, to me, it’d be a great value.

If you’re curious, you can start using the crate with glsl = "0.2.1" in your Cargo.toml. You probably want to use the translation_unit parser, which is the most external parser (it parses an actual shader). If you’re looking for something similar to my need and want to parse subparts of GLSL, feel free to dig in the documentation, and especially the parsers module, that exports all the parsers available to parse GLSL’s parts.

Either way, you have to pass a bytes slice. If your source’s type is String or &str, you can use the str::as_bytes() function to feed the input of the parsers.

Not all parsers are exported, only the most important ones. You will not find an octal parser, for instance, while it’s defined and used in the crate internally.

Call to contribution

If you think it’s worth it, I’m looking for people who would like to:

• write a GLSL to SPIR-V writer: it’s an easy task, because you just have to write a function of the form AST -> Result<SPIRV, _>, for being pragmatic;
• test the crate and provide feedback about any error you’d find! Please do not open an issue if you have an error in your source and find “the error from the parsers is not useful”, because it’s already well established this is a problem and I’m doing my best to solve it;
• any kind of contributions you think could be interesting for the crate.

Happy coding, and keep the vibe!

This article serves as an announcement for glsl-0.13.

I’ve been working on glsl-0.13 for a few days now since my previous blog entry about adding pest to glsl – and no, the current code is not pest-based, it’s still good old nom. Lately, I’ve been wanting to enhance my glsl-quasiquote crate to add variable interpolation. I think I will dedicate a whole article to variable interpolation in GLSL because it’s actually tricky to get done right – without duplicating a whole parser or introducing problematic code with pest (see my last article).

Since I’m not a programmer who solves problems that don’t exist, I have problems to get resolved in other crates and binary projects of mine (mostly demoscene and demoscene tooling). However, I’ve been spending days solving those problems because they’re not related to demoscene only: pretty much anyone who would like to cope with shaders might be interested.

The glsl crate prior to 0.13

The glsl crate provides you with:

• A GLSL450 parser. I wrote the parser with nom back then by implementing the strict OpenGL Shading Language Spec (here). That parser works mostly on strings and byte slices (I really doubt people use the byte slice interface and I might deprecate it).
• The parser parses into an Abstract Syntax Tree – AST for short. That AST is, as the name implies, a tree that contains typed nodes. It’s a bit more complicated than having a BTreeMap for instance, because each nodes and levels are completely typed. You can picture the AST as both a type tree and value tree instead of just a value tree.
• Before 0.13, transforming the AST was tedious, because you had to pattern match on the whole structure of the AST, and since it’s a type tree, it’s really, really boring to do so.
• One typical transformation that I always need in several crates of mine is construction. Building ASTs was overwhelming and complicated because you had to create the tree by hand. To solve that issue, I introduced the glsl-quasiquote crate, that enables you to create the AST out of a regular GLSL string. I hit some issues with that, like the impossibility to have typing information exchanged between two compiler phases (the procedural macros are all evaluated before the rest, making it impossible to have type inference like in let x: Expr = glsl!{…};. I will try to fix that later.
• The quasiquote crate works great for static ASTs but as soon as you want to build dynamic ones ( i.e. that depend on variables for instance), since glsl-quasiquote doesn’t have variable interpolation (yet!), you’re back to the boring by-hand construction.

glsl-0.13 comes with several new features and enhancements to partially fix those points that I will explain in this blog entry.

The new features!

First, let’s talk about contribution. I received a contribution (both an issue and a PR, how amazing!) from @JDemler about the #define preprocessor pragma. Those were not taken into account so far and he provided the fix to add them! As for all other pragmas, those have to lay in the global scope – they’re not expected to be found in function bodies, for instance.

Thanks for your contribution! :)

Then, a lot of work has been done around the glsl::syntax module. This module contains all the AST types and some methods were added to, for instance, create functions, variable declarations, if-else blocks, etc. Those functions were made the most general as possible and heavily depend on From / Into, allowing for very short oneliners to create complex AST nodes. The best example for this is the Statement::declare_var that declares a new variable as a Statement. Most of the inner AST nodes won’t ever be needed in the public interface, so those functions hide them from you and give you the easy way to build bigger nodes.

There is an example below that uses Statement::declare_var. Keep on reading! :)

Not all AST nodes have helper methods for construction. The good thing is that adding a new function shouldn’t be a breaking change, so I’ll keep adding them as needed – if you have a specific need for one, feel free to open an issue, I will make the change. That might be surprising but since I haven’t witnessed lots of people using the crate yet, I don’t implement what I don’t need yet – but as soon as someone tells me they need something, either I immediately review the PR, or plan some time to make the patch myself.

Finally, the biggest feature which will be explained in further details in this blog post: AST visitors. AST visitors are the way I imagined would be the best to traverse an AST in order to mutate some nodes, all the nodes, filter, query information, etc. It’s like lenses for glsl! – but trust me: it’s way lighter! :D

Visiting ASTs

AST visitors solve a problem I had when implementing a specific feature in spectra. I needed to be able to change all the references to a given identifier in all the AST at once. Either the identifier is used in an expression assigned to a new variable, returned from a function or passed as argument to a function, I needed to catch that identifier… and change its name.

That was quite of a desperate challenge without a proper solution. Imagine: I would have to write the code to change the identifier myself and… find ALL the places where identifiers might appear in any AST. This is possible, but that would drive many developers mad – especially whevener the glsl crate changes.

So I came up with a better solution. I will not drop you a technical term and have you read Wikipedia, I would dislike that. I will just explain from bottom up why it’s designed this way and why I think it’s the best solution.

Visiting summarized

The idea is that pattern-matching on an identifier might appear in several places in an AST. So whatever solution we choose, we will have to find all those spots and call a function that states:

Hey there. Here is an Identifier. I give you a &mut Identifier so that you can change it.

An Identifier is an AST (a really simple one, but still is). So you might want to implement that function on Identifier… but what happens when your AST is an Expr, and that expr is a variable – that is, Expr::Var(identifier_here)? You will want to implement your function on Expr also, then. And here you see that you need a trait, because, as said earlier, the AST is a type tree.

However, what trait? We could imagine implementing that trait only on AST types that have an Identifier as field or variant. But the most general AST node, TranslationUnit, is a non-empty list of ExternalDeclarations. If we want to visit that, we won’t be able to type match and pass down our function transforming identifiers.

We see that we will need to implement that trait for all AST types so that they can pass the function down the AST to a node that actually has an Identifier.

And since we’re doing it with Identifier, we might want to do it with any AST node. But if we do that, we cannot pass one single function anymore…

The Visitor

So you need an object that will be able to treat an Identifier… or a TranslationUnit, or anything AST. This is a bit boring to implement, but we need a trait that enables a type to visit any AST node:

trait Visitor {
  fn visit_identifier(&mut self, _: &mut Identifier);
  fn visit_translation_unit(&mut self, _: &mut TranslationUnit);
  // …
}

This is great, because if a type implements Visitor, it means that we can call visit_identifier on it if we have an Identifier! When we will be implementing our traversing function, we will just have to carry around a mutable reference to an object implementing Visitor, and call the right function depending on the AST node / type we are at!

We also use mutable references here so that we can also mutate information as we sink into the AST. This might be very useful to know at which block depth we are at, or if we’re in a function’s body, etc.

Something important with that trait though: the current implementation (from this article) would be very boring to implement, because it has a lot of visit_* methods. What if we’re only interested in Identifier? We don’t want to have to implement visit_translation_unit because we don’t care.

A simple fix to that is to give all those visit_* methods a default implementation… that does nothing.

trait Visitor {
  fn visit_identifier(&mut self, _: &mut Identifier) {}
  fn visit_translation_unit(&mut self, _: &mut TranslationUnit) {}
  // …
}

Let’s try it!

struct ReplaceIdentifier<'a> {
  replacement: &'a str
}

impl<'a> Visitor for ReplaceIdentifier<'a> {
  fn visit_identifier(&mut self, ident: &mut Identifier) {
    *ident = self.replacement.clone().into()
  }
}

And that’s all! We now have a visitor that can be used to traverse any AST and change any Identifier to what we have set. For instance:

let mut ast = …;
let mut visitor = ReplaceIdentifier { replacement: "foobar" }

// wait, how do we pass the visitor to the AST here?

Argh, we’re still missing something!

Hosting visitors

We need a visit function, like:

fn visit<V>(ast: &mut TranslationUnit, visitor: &mut V) where V: Visitor

So let’s write it!

fn visit<V>(ast: &mut TranslationUnit, visitor: &mut V) where V: Visitor {
  for external_decl in ast {
    // ah.
  }
}

We cannot call visit again on ExternalDeclaration. Seems like we need another trait! :D

trait Host {
  fn visit<V>(&mut Self, visitor: &mut V) where V: Visitor;
}

Here, a type that implements Host means that it can call a Visitor on itself, and might pass it down to children AST if any. Since we’re going to implement Host for all our AST types, we will be able to do something like this:

impl Host for TranslationUnit {
  fn visit<V>(&mut Self, visitor: &mut V) where V: Visitor {
    visitor.visit_translation_unit(self); // first, we have the visitor visit the AST node

    // then, for all children, we pass down the visitor!
    for external_decl in self {
      external_decl.visit(visitor);
    }
  }
}

And here we go. We are able to pass down the visitor, that will be called for each node. If you have provided an implementation for a given visit_*, it will get invoked, otherwise, the default implementation will fire – and it does nothing.

A simple optimization can be done, here. Since you might know that you’re done at a given level regarding your visitor, we could add a way to make a Host stop visiting and go any deeper. For this, we introduce a simple enum:

enum Visit {
  Children, // keep visiting children
  Parent // stop visiting, go back to parent
}

And we change all the Visitor methods to return a Visit. That will give us the information we need when implementing Host::visit now:

impl Host for TranslationUnit {
  fn visit<V>(&mut Self, visitor: &mut V) where V: Visitor {
    let visit = visitor.visit_translation_unit(self); // first, we have the visitor visit the AST node

    if visit == Visit::Children {
      // then, for all children, we pass down the visitor!
      for external_decl in self {
        external_decl.visit(visitor);
      }
    }
  }
}

We also need to change the default implementation of the visit_* methods. My choice was to have them return Visit::Children by default, because I think it’s a saner default – if people don’t want to go any deeper in the AST, they will just have to dummy-implement the right method.

And we are done! The actual implementation is a bit more complex than that but is really really close to what is described in this article. I’ll give you the example from the official documentation so that can you can see how it’s really used:

use glsl::syntax::{CompoundStatement, Expr, SingleDeclaration, Statement, TypeSpecifierNonArray};
use glsl::visitor::{Host, Visit, Visitor};
use std::iter::FromIterator;

let decl0 = Statement::declare_var(
  TypeSpecifierNonArray::Float,
  "x",
  None,
  Some(Expr::from(3.14).into())
);

let decl1 = Statement::declare_var(
  TypeSpecifierNonArray::Int,
  "y",
  None,
  None
);

let decl2 = Statement::declare_var(
  TypeSpecifierNonArray::Vec4,
  "z",
  None,
  None
);

let mut compound = CompoundStatement::from_iter(vec![decl0, decl1, decl2]);

// our visitor that will count the number of variables it saw
struct Counter {
  var_nb: usize
}

impl Visitor for Counter {
  // we are only interested in single declaration with a name
  fn visit_single_declaration(&mut self, declaration: &mut SingleDeclaration) -> Visit {
    if declaration.name.is_some() {
      self.var_nb += 1;
    }

    // do not go deeper
    Visit::Parent
  }
}

let mut counter = Counter { var_nb: 0 };
compound.visit(&mut counter);
assert_eq!(counter.var_nb, 3);

Future work

The current state of Visitor is great but there is a drawback: your AST has to be mutable. You might want to only traverse it read-only but have your visitor mutated. I might then modify slightly the Visitor trait and Host one with Host::visit_mut if I think it’s needed.

Feel free to experiment around. Next work planned for glsl – besides contributions – quasiquoting variable interpolation.

Keep the vibes!

So, what’s new?

OpenGL allows programmers to send vertices to the GPU through what is called a vertex array. Vertex specification is performed through several functions, operating on several objects. You need, for instance, a vertex buffer object, an index buffer object and a vertex array object. The vertex buffer stores the vertices data.

For instance, you could imagine a teapot as a set of vertices. Those vertices have several attributes. We could use, for instance, a position, a normal and a bone index. The vertex buffer would be responsible of storing those positions, normals and bone indices. There’re two ways to store them:

1. interleaved arrays ;
2. deinterleaved arrays.

I’ll explain those later on. The index buffer stores integral numbers – mainly set to unsigned int – that index the vertices, so that we can connect them and create lines, triangles or more complex shapes.

Finally, the vertex array object is a state object that stores links to the two buffers and makes a connection between pointers in the buffer and attribute indices. Once everything is set up, we might only use the vertex array object. The exception is when we need to change the geometry of an object. We need to access the vertex buffer and the index buffer and upload new data. However, for now, that feature is disabled so that the buffers are not exposed to the programmer. If people think that feature should be implemented, I’ll create specialized code for that very purpose.

Interleaved and deinterleaved arrays

Interleaved arrays might be the most simple to picture, because you use such arrays every day when programming. Let’s imagine you have the following type in Haskell:

data Vertex = Vertex {
    vertPos    :: X
  , vertNor    :: Y
  , vertBoneID :: Z
  } deriving (Eq,Show)

Now, the teapot would have several vertices. Approximately, let’s state the teapot has five vertices – yeah, ugly teapot. We can represent such vertices in an interleaved array by simply recording them in a list or an array:

As you can see, the attributes are interleaved in memory, and the whole pattern is cycling. That’s the common way to represent an array of struct in a lot of languages, and it’s very natural for a machine to do things like that.

The deinterleaved version is:

As you can see, with deinterleaved arrays, all attributes are extracted and grouped. If you want to access the third vertex, you need to read the third X, the third Y and the third Z.

Both the methods have advantages and drawbacks. The cool thing about deinterleaved arrays is that we can copy huge regions of typed memory at once whilst we cannot with interleaved arrays. However, interleaved arrays store continuous structures, so writing and reading a structure back might be faster.

An important point to keep in mind: because we plan to pass those arrays to OpenGL, there’s no alignment restriction on the structure. That is, everything is packed, and we’ll have to pass extra information to OpenGL to tell it how to advance in memory to correctly build vertices back.

Generalized tuple

I think I haven’t told you yet. I have a cool type in luminance: the (:.) type. No, you don’t have to know how to pronounce that. I like to call it the gtuple type, because it’s a generalized tuple. You can encode (a,b), (a,b,c) and all kind of tuples with (:.). You can even encode single-typed infinite tuple! – a very special kind of list, indeed.

data a :. b = a :. b

infixr 6 :.

-- a :. b is isomorphic to (a,b)
-- a :. b :. c is isomorphic to (a,b,c)

newtype Fix f = Fix (f (Fix f)) -- from Control.Monad.Fix
type Inf a = Fix ((:.) a) -- infinite tuple!

Pretty simple, but way more powerful than the regular, monomorphic tuples. As you can see, (:.) is a right-associative. That means that a :. b :. c = a :. (b :. c).

That type will be heavily used in luminance, thus you should get your fet wet with it. There’s actually nothing much to know about it. It’s a Functor. I might add other features to it later on.

The Storable trick

The cool thing about (:.) is that we can provide a Storable instance for packed memory, as OpenGL requires it. Currently, the Storable instance is implemented like this:

instance (Storable a,Storable b) => Storable (a :. b) where
  sizeOf (a :. b) = sizeOf a + sizeOf b
  alignment _ = 1 -- packed data
  peek p = do
    a <- peek $ castPtr p
    b <- peek . castPtr $ p `plusPtr` sizeOf (undefined :: a)
    pure $ a :. b
  poke p (a :. b) = do
    poke (castPtr p) a
    poke (castPtr $ p `plusPtr` sizeOf (undefined :: a)) b

As you can see, the alignment is set to 1 to express the fact the memory is packed. The peek and poke functions use the size of the head of the tuple to advance the pointer so that we effectively write the whole tuple in packed memory.

Then, let’s rewrite our Vertex type in terms of (:.) to see how it’s going on:

type Vertex = X :. Y :. Z

If X, Y and Z are in Storable, we can directly poke one of our Vertex into a luminance buffer! That is, directly into the GPU buffer!

Keep in mind that the Storable instance implements packed-memory uploads and reads, and won’t work with special kinds of buffers, like shader storage ones, which require specific memory alignment. To cover them, I’ll create specific typeclasses instances. No worries.

Creating a vertex array

Creating a vertex array is done through the function createVertexArray. I might change the name of that object – it’s ugly, right? Maybe Shape, or something cooler!

createVertexArray :: (Foldable f,MonadIO m,MonadResource m,Storable v,Traversable t,Vertex v)
                  => t v
                  -> f Word32
                  -> m VertexArray

As you can see, the type signature is highly polymorphic. t and f represent foldable structures storing the vertices and the indices. And that’s all. Nothing else to feed the function with! As you can see, there’s a typeclass constraint on v, the inner vertex type, Vertex. That constraint ensures the vertex type is representable on the OpenGL side and has a known vertex format.

Disclaimer: the Traversable constraint might be relaxed to be Foldable very soon.

Once tested, I’ll move all that code from the unstable branch to the master branch so that you guys can test it. :)

About OpenGL…

I eventually came to the realization that I needed to inform you about the OpenGL prerequisites. Because I want the framework to be as modern and well-designed as possible, you’ll need… OpenGL 4.5. The latest version, indeed. You might also need an extension, ARB_bindless_texture. That would enable the framework to pass textures to shader in a very stateless way, which is our objective!

I’ll let you know what I decide about that. I don’t want to use an extension that is not implemented almost everywhere.

What’s next?

Well, tests! I need to be sure everything is correctly done on the GPU side, especially the vertex format specification. I’m pretty confident though.

Once the vertex arrays are tested, I’ll start defining a render interface as stateless as I can. As always, I’ll keep you informed!

This is a very short article to make you notice that al received two important changes:

• I uploaded documentation (hourra!) ;
• OpenAL paths will default to default installation on Windows systems.

I tested the latter with a Windows 64b:

cabal update
cabal install al

That’s all. You should try it out as well. If you have errors about OpenAL libraries and/or header files, check whether OpenAL is correctly installed. If the error comes up again, proceed as I said here.

Also, I take people on linux feedback about installs ;).

From luminance-0.1 to luminance-0.2 included, it was not possible to use any texture types different than two-dimensional textures. This blog entry tags the new release, luminance-0.3, which adds support for several kinds of texture.

A bit more dimensions

Texture1D, Texture2D and Texture3D are all part of the new release. The interface has changed – hence the breaking changes yield a major version increment – and I’ll explain how it has.

Basically, textures are now fully polymorphic and are constrained by a typeclass: Texture. That typeclass enables ad hoc polymorphism. It is then possible to add more texture types without having to change the interface, which is cool. Like everything else in luminance, you just have to ask the typesystem which kind of texture you want, and everything will be taken care of for you.

Basically, you have three functions to know:

• createTexture, which is used to create a new texture ;
• uploadSub, used to upload texels to a subpart of the texture ;
• fillSub, used to fill – clear – a subpart of the texture with a given value.

All those functions work on (Texture t) => t, so it will work with all kinds of texture.

Cubemaps

Cubemaps are also included. They work like other textures but add the concept of faces. Feel free to dig in the documentation for further details.

What’s next?

I need to find a way to wrap texture arrays, which are very nice and useful for layered rendering. After that, I’ll try to expose the change to the framebuffers so that we can create framebuffers with cubemaps or that kind of cool feature.

In the waiting, have a good a week!

The problem

Just to let you fully understand the problem, let me introduce a few principles from alto. As a wrapper over OpenAL, it exposes quite the same interface, but adds several safe-related types. In order to use the API, you need three objects:

• an Alto object, which represents the API object (it holds dynamic library handles, function pointers, etc. ; we don’t need to know about that)
• a Device object, a regular device (a sound card, for example)
• a Context object, used to create audio resources, handle the audio context, etc.

There are well-defined relationships between those objects that state about their lifetimes. An Alto object must outlive the Device and the Device must outlive the Context. Basically:

let alto = Alto::load_default(None).unwrap(); // bring the default OpenAL implementation in
let dev = alto.open(None).unwrap(); // open the default device
let ctx = dev.new_context(None).unwrap(); // create a default context with no OpenAL extension

As you can see here, the lifetimes are not violated, because alto outlives dev which outlives ctx. Let’s dig in the type and function signatures to get the lifetimes right (documentation here).

fn Alto::open<'s, S: Into<Option<&'s CStr>>>(&self, spec: S) -> AltoResult<Device>

The S type is just a convenient type to select a specific implementation. We need the default one, so just pass None. However, have a look at the result. AltoResult<Device>. I told you about lifetime relationships. This one might be tricky, but you always have to wonder “is there an elided lifetime here?”. Look at the Device type:

pub struct Device<'a> { /* fields omitted */ }

Yep! So, what’s the lifetime of the Device in AltoResult<Device>? Well, that’s simple: the lifetime elision rule in action is one of the simplest:

If there are multiple input lifetime positions, but one of them is &self or &mut self, the lifetime of self is assigned to all elided output lifetimes. (source)

So let’s rewrite the Alto::open function to make it clearer:

fn Alto::open<'a, 's, S: Into<Option<&'s CStr>>>(&'a self, spec: S) -> AltoResult<Device<'a>> // exact same thing as above

So, what you can see here is that the Device must be valid for the same lifetime as the reference we pass in. Which means that Device cannot outlive the reference. Hence, it cannot outlive the Alto object.

impl<'a> Device<'a> {
  // …
  fn new_context<A: Into<Option<ContextAttrs>>>(&self, attrs: A) -> AltoResult<Context>
  // …
}

That looks a bit similar. Let’s have a look at Context:

pub struct Context<'d> { /* fields omitted */ }

Yep, same thing! Let’s rewrite the whole thing:

impl<'a> Device<'a> {
  // …
  fn new_context<'b, A: Into<Option<ContextAttrs>>>(&'b self, attrs: A) -> AltoResult<Context<'b>>
  // …
}

Plus, keep in mind that self is actually Device<'a>. The first argument of this function then awaits a &'b Device<'a> object!

rustc is smart enough to automatically insert the 'a: 'b lifetime bound here – i.e. the ’a lifetime outlives ’b. Which makes sense: the reference will die before the Device<'a> is dropped.

Ok, ok. So, what’s the problem then?!

The (real) problem

The snippet of code above about how to create the three objects is straight-forward (though we don’t take into account errors, but that’s another topic). However, in my demoscene framework, I really don’t want people to use that kind of types. The framework should be completely agnostic about which technology or API is used internally. For my purposes, I just need a single type with a few methods to work with.

Something like that:

struct Audio = {}

impl Audio {
  pub fn new<P>(track_path: P) -> Result<Self> where P: AsRef<Path> {}

  pub fn toggle(&mut self) -> bool {}

  pub fn playback_cursor(&self) -> f32 {}

  pub fn set_playback_cursor(&self, t: f32) {}
}

impl Drop for Audio {
  fn drop(&mut self) {
    // stop the music if playing; do additional audio cleanup
  }
}

This is a very simple interface, yet I don’t need more. Audio::set_playback_cursor is cool when I debug my demos in realtime by clicking a time panel to quickly jump to a part of the music. Audio::toggle() enables me to pause the demo to inspect an effect in the demo. Etc.

However, how can I implement Audio::new?

The (current) limits of borrowing

The problem kicks in as we need to wrap the three types – Alto, Device and Context – as the fields of Audio:

struct Audio<'a> {
  alto: Alto,
  dev: Device<'a>,
  context: Context<'a>
}

We have a problem if we do this. Even though the type is correct, we cannot correctly implement Audio::new. Let’s try:

impl<'a> Audio<'a> {
  pub fn new<P>(_: P) -> Result<Self> where P: AsRef<Path> {
    let alto = Alto::load_default(None).unwrap();
    let dev = alto.open(None).unwrap();
    let ctx = dev.new_context(None).unwrap();

    Ok(Audio {
      alto: alto,
      dev: dev,
      ctx: ctx
    })
  }
}

As you can see, that cannot work:

error: `alto` does not live long enough
  --> /tmp/alto/src/main.rs:14:15
   |
14 |     let dev = alto.open(None).unwrap();
   |               ^^^^ does not live long enough
...
22 |   }
   |   - borrowed value only lives until here
   |
note: borrowed value must be valid for the lifetime 'a as defined on the body at 12:19...
  --> /tmp/alto/src/main.rs:12:20
   |
12 |   fn new() -> Self {
   |                    ^

error: `dev` does not live long enough
  --> /tmp/alto/src/main.rs:15:15
   |
15 |     let ctx = dev.new_context(None).unwrap();
   |               ^^^ does not live long enough
...
22 |   }
   |   - borrowed value only lives until here
   |
note: borrowed value must be valid for the lifetime 'a as defined on the body at 12:19...
  --> /tmp/alto/src/main.rs:12:20
   |
12 |   fn new() -> Self {
   |                    ^

error: aborting due to 2 previous errors

What’s going on here? Well, we’re hitting a problem called the problem of self-borrowing. Look at the first two lines of our implementation of Audio::new:

let alto = Alto::load_default(None).unwrap();
let dev = alto.open(None).unwrap();

As you can see, the call to Alto::open borrows alto – via a &Alto reference. And of course, you cannot move a value that is borrowed – that would invalidate all the references pointing to it. We also have another problem: imagine we could do that. All those types implement Drop. Because they basically all have the same lifetime, there’s no way to know which one borrows information from whom. The dropchecker has no way to know that. It will then refuse to code creating objects of this type, because dropping might be unsafe in that case.

What can we do about it?

Currently, this problem is linked to the fact that the lifetime system is a bit too restrictive and doesn’t allow for self-borrowing. Plus, you also have the dropchecker issue to figure out. Even though we were able to bring in alto and device altogether, how do you handle context? The dropchecker doesn’t know which one must be dropped first – there’s no obvious link at this stage between alto and all the others anymore, because that link was made with a reference to alto that died – we’re moving out of the scope of the Audio::new function.

That’s a bit tough. The current solution I implemented to fix the issue is ok–ish, but I dislike it because it adds a significant performance overhead: I just moved the initialization code in a thread that stays awake until the Audio object dies, and I use a synchronized channel to communicate with the objects in that thread. That works because the thread provides us with a stack, that is the support of lifetimes – think of scopes.

Another solution would be to move that initialization code in a function that would accept a closure – your application. Once everything is initialized, the closure is called with a few callbacks to toggle / set the cursor of the object living “behind” on the stack. I don’t like that solution because it modifies the main design – having an Audio object was the goal.

Other solutions are:

• std::mem::transmute to remove the lifetimes (replace them with 'static). That’s hyper dangerous and we are just breaking Rust’s lifetimes… not okay :(
• change our design to meet the same as alto’s (in a word: use the same three objects)
• cry deeply

I don’t have a satisfying solution yet to that problem. My thread solution works and lets me have a single type abstracting all of that, but having a thread for such a thing is a waste of resources to me. I think I’ll implement the closure solution as, currently, it’s not possible to embed in struct lifetimes’ semantics / logic. I guess it’s okay; I guess the problem is also linked to the fact the concept is pretty young and we’re still kind of experimenting it. But clearly, lifetimes hit a hard problem here that they cannot solve correctly. Keep in mind that even if unsafe solutions exist, we’re talking about a library that’s designed to work with Rust lifetimes as a pretty high level of abstraction. Firing transmute is very symptomatic of something wrong. I’m open to suggestions, because I’ve been thinking the problem all day long without finding a proper solution.

Keep the vibe!

• The lib.rs / main.rs files, that get rendered as front page on docs.rs. This is important because we want people to have nice onboardings when hitting the official documentations of our crates.
• The README file — often README.md, that gets automatically rendered as HTML when you end up on the project’s GitHub page. This page should provide as many as possible hints and information about what the project is and how to use it.

However, sometimes, I just don’t want to bother writing twice the same thing and honestly, I don’t really know whether the README file is a good place to write an onboarding section, since that should also be included in your Rust documentation on docs.rs.

So… I’ve been thinking of a way to fix that, without really investing too much time in it. But lately, I came to the realization that I often have this pattern:

1. Write the onboarding documentation in my lib.rs or main.rs. For instance, I’m pretty proud of the onboarding documentation of warmy. It’s exhaustive, easy for newcomers, it has plenty of links and explanations and it renders pretty nice on docs.rs.
2. Write a header in the README file with badges etc. and then copy the Rust onboarding documentation in the readme.

Here, (2.) is a manual and annoying task: I open my editor, make a block selection of the documentation, store it in a buffer, apply a smart macro on it to remove the Rust annotation and then paste the result in the README after having purged it from the previous documentation… That’s tedious and not very interesting.

cargo sync-readme to the rescue!

So, yesterday, I decided to start working on a small tool that would automate all this for me. The idea is the following:

• cargo sync-readme synchronizes your README (the file specified by the readme key in your Cargo.toml, or just README.md by default) with the entrypoint of your library or binary crate (by default, lib.rs or main.rs, or what is defined at the path key in your manifest).
• In order to perform the synchronization, you need to have put a special marker so that the command is instructed where to put the synchronized documentation.
  • The <!-- cargo-sync-readme --> marker must lie on a single line where you want the documentation to be inserted in your README.
  • Post generation, this marker will disappear and will get replaced by the synchronized documentation, surrounded by two markers: <!-- cargo-sync-readme start> and <!-- cargo-sync-readme end -->.

This is really all you have to do. cargo sync-readme will take care of the rest for you. There’re two hypothesis that the command requires to be true, though:

• Your README file should be a Markdown-formatted file. It doesn’t have to be, but since Rust’s documentation is written in Markdown, it should be.
• You use the //! annotation to write your documentation. This is currently how cargo sync-readme works. It doesn’t have to be solely using this annotation on longer term, but currently, this is the only annotation supported. More can be added if needed.

Basically, insert the marker once, and run cargo sync-readme. Every time you change your Rust documentation, just call cargo sync-readme once again to synchronize the documentation in your README file.

On workspace crates

Currently, cargo sync-readme doesn’t work with workspace crates. You will have to go into each of the workspace’s members to run cargo sync-readme if you want to synchronize their respective READMEs.

Conclusion

cargo sync-readme is already available on crates.io for you to use. You can install it as a development tool with the following command:

cargo install cargo-sync-readme

Disclaimer: after having published cargo-sync-readme, I was told that there is already another cargo plugin, cargo-readme, that already does that. Indeed, that crate does more or less the same job. However, the way it does it is very different. First, it uses a template file while cargo-sync-readme lets you use your README file without having to depend on a template. Also, cargo-readme has special semantics in its template (like {{crate_name}}, etc.) while cargo-sync-readme is simpler and just requires a single marker. To sum up: cargo-readme is heavier and is likely to require you to break your file into two separate files but contains more customization options while cargo-sync-readme only requires a single line in your README and will synchronize from within that same file.

Feel free to comment, open issues, drink beers, share that awesome bear driving a sausage podracer and most of all… keep the vibes!

• cargo-sync-readme on GitHub
• cargo-sync-readme on crates.io

Hello people. It’s been weeks I have started to work on luminance-1.0.0. For a brief recap, luminance is a graphics crate that I originally created in Haskell, when I ripped it off from a demoscene engine called quaazar in order to make and maintain tiner packages. The Rust port was my first Rust project and it became quickly the default language I would develop graphics applications in.

So if you’re interested, yes, luminance in Haskell still exists but I maintain it for strictly minimal support (i.e. support compiler bumps, for instance). The default and official language to use, if you’re interested by luminance, is Rust.

Another important thing I have to explain is the current feature list of luminance and what it’s going to be soon. Currently, luminance’s goals are (according to the documentation):

• Making the unsafe and stateful OpenGL API safe and stateless.
  • Unsafe means that OpenGL performs operations by relying on the fact you know what you’re doing: pointer arithmetic, memory alignment, etc. Lots of situations in which you can literally RIP in pepperonis.
  • Stateful means that lots of OpenGL functions expect a context to be in a special state for a correct behavior. For instance, when uploading texels to the GPU’s VRAM, the texture that must receive the texels must be bound to a texture unit. If you forget to do that, you get a weird behavior that is seen at runtime and not always correctly interpreted. It is sometimes hard to know why and what something has gone wrong. It makes also a pain in the @!#=$ as soon as you want to share work on several threads.
• Providing a simple and easy interface; that is, exposing core concepts without anything extra – just the bare stuff.
  • Most low-level graphics library are hard for newcomers to grasp knowledge about. gfx-hal is an excellent graphics crate (so far, I think it’s the one everyone is recommending everyone in the Rust gamedev community) but it is hard to play with it when you don’t know either OpenGL 4.5 or Vulkan — and all the required concepts might be a bit overwhelming at first.
  • Even though luminance is still a pretty low-level API (you have renderbuffers, framebuffers, shader stages, etc.), the interface is opinionated and — in my opinion — easier to work with. Because it is my idea and my design, the experience might be slightly different from e.g. gfx-hal. It’s neither bad or good, just different. That also responds to the folks who don’t understand why I’m still maintaining luminance: because I want to do graphics code my way, and more choices is always a good sign to me. Yes, we could talk about community efforts, but OH LOOK A BIRD!
• Easy to read with a good documentation and set of tutorials, so that newcomers don’t have to learn a lot of new concepts to get their feet wet.
• Need-driven: every piece of code added in the project must come from a real use case. If you feel something is missing, feel free to open an issue or even contribute!
  • This is an important aspect to me. I don’t write code for the sake of writing code (even libraries). I have a problem and write code to solve it. So, on the surface, luminance might seem a bit less complete that, again, e.g. gfx-hal. And that would be likely right. However, if someone likes the API and wants something in, I accept both issues and PRs. There are several examples of people asking for features and having me implement them. There’s also some folks who actually wrote the code and it’s now in luminance!
  • Head over to https://github.com/phaazon/luminance-rs/issues if you want to start discussing the feature set.

That’s the current situaton. A very important other aspect is that luminance was designed, in the first place, to be backend agnostic. I quickly decided not to go this way and stick to a simpler design with OpenGL only.

A comparison between luminance and other famous crates is available here.

However, I have — again! — changed my mind. Today, I want to have a system of backend in luminance. Maybe not something as complex and complete as the gfx-rs project, but something that will allow those backends:

• Any OpenGL version: missing features from older versions will be implemented (if not too much work is needed) by luminance software patches.
• At least one WebGL implementation and it must be WASM compatible.
• Vulkan.

I really do not target Metal, DirectX12 nor any other graphics backend you might have in mind. However, if you think it’s possible to do so, then we should discuss about it. Nevertheless, as I said above, luminance is need-driven: as I don’t need DirectX12, I will not work on it unless someone REALLY needs it and provides a PR (or at least enough for me to work on it… including beers).

What to expect

luminance-1.0.0 will ship… as soon as possible. The current feature set is huge, though:

• Add the "std" feature-gate. This will help use luminance with specific contexts, especially for demoscene or size-limited devices. Currently, the work on no_std has been saved for later as I had hit a bug last time I checked.
• Change the Tess interface. Now, a TessBuilder must be used. This allows for three major enhancements:
  • Vertex instancing.
  • Deinterleaved memory.
  • Builder pattern.
• Enhance the quality and variety of examples.
• Introduce the luminance-derive crate. This crate provides several proc-macros used to implement various unsafe traits easily, without writing any unsafe code. Among those:
  • The Vertex trait, used to create vertex types.
  • The Semantics trait, used to create vertex semantics types.
  • The UniformInterface trait, used to create shader uniform interfaces.
  • All the previous macro_rules are removed.
• Support a new dynamic way to get uniforms. Sometimes, you don’t know in advance the uniforms your shader is going to use. When such a case arise, you might be tempted to have a minimal uniform interface — if none. Dynamic uniform lookups allow you to query uniforms on the fly. That has obviously a runtime cost but it can be handy for lots of situations (GUI editors, scripting, etc.).
• Update framebuffer code so that color slots and depth slots are easier to use.
• Introduce the concept of drivers. Drivers allow code to be parametered by a type which represents a given implementation to use for the graphics code. You can have a GL33 driver for OpenGL 3.3, a GL40, GL44, GL45 etc., but you can also have WebGL and VK10.
• Add vertex instancing. Vertex instancing allows to instantiate objects by providing instance data directly in a Tess — instead of using the current method with GPU Buffer<_>. This was asked by several people and I just couldn’t ignore such an interesting and useful feature!
• Support for deinterleaved memory. Deinterleaved memory allows for more granularity on how data is fetched GPU-side. With the legacy luminance situation, data is completely interleaved, which means that a vertex’s attributes all follow each other in a GPU buffer. This is both nice and bad: if you need to access all attributes of a vertex, this is pretty good, since cache-locality will be in your advantage. However, if you’re only interested in positions, for instance, you will waste your cache lines with attributes you will never read from! Deinterleaved memory stores each attributes in a different GPU memory region, allowing for a way better memory scheme for such uses. People using the ECS pattern might be used to deinterleaved memory.
• Add vertex primitive restart. This allows to use tessellation indexing modes such as TriangleFan or LineStrip and “cut” a primitive if an index is equal to a given value. This is very useful for implementing terrains, quadrics or a lot of other nice things.
• Introduce vertex semantics. This is one of the sexiest and most exciting feature of this next release. I got the idea by thinking about the fact luminance should be low-level but provide very quick access to building high-level blocks. I got the idea with vertex semantics via my spectra crate. The idea is simple: currently, we must define a Vertex type and the order in which the fields appear in the struct defines the bindings the GPU will present to shaders… but it’s the user responsibility to tell how the shader will fetch those attributes. This can lead to very wrong situations in which, for instance, the GPU present the normal attribute as having the index 4 but the shader tries to fetch them on index 2. Vertex semantics fix this situation by adding an intermediate layer both the user and the GPU (i.e. via luminance) must speak: semantics. Semantics are user-defined and none is hardcoded in luminance. As soon as a type uses vertex semantics, all the type system knows how to forward that information to shaders and what shaders should do to fetch them. The other bonus, ultra cool thing is that shaders and memory buffers now compose way better: since vertex semantics define a sort of namespace, you will never have two shaders using the same index for different semantics (unless you use completely different vertex semantics type). A very cool feature I’m proud of and that I will detail in a future article / luminance tutorial.
• And lots of other doc fixes, enhancements, etc.

One word about the feature set and spare-time

You might be wondering “Woah, this is such a big feature set. A lot of things are coming! Why not having split the feature set into several releases?”, and that’s a good question.

Lately, I’ve been feeling on-and-off about everything I do on my spare-time on the FOSS level. I have several crates I care about and sometimes people show up and start asking for features and/or bug fixes. This takes some time and after a long day of work, I sometimes feel that I just want to hack on things I like, play my electric guitar, go to swim or wakeboard or just spend time with people I care about and love.

Currently, luminance — even though it has an unexpected amount of downloads! — is not a famous crate. People nowadays talk about crates living in a higher-level stack, such as ggez, gfx, amethyst, piston, etc. I don’t communicate a lot about luminance because of, mostly, two things:

1. Imposter syndrom.
2. Avoid success at all cost.

I guess Haskellers will get the second point. ;)

My own philosophy about software and especially my own software is that it should have a top-tier quality, both in terms of API design, elegance, type safety, performance and documentation. The front documentation of warmy is a good example of something I continuously seek, because good documentation helps my future myself to read the code and public interface but also for onboarding people. Lots of projects — especially in professional industries — suffer from knowledge bubbles, in which just a bunch (often only one!) of people know about A and B and when someone joins in, they’re lost and have to ask questions. As an engineer, I always write everything I know in whatever source-of-truth the company I’m in can offer (it can be a Confluence server; a git repository; a wiki; whatever) because sharing knowledge and writing processes is key in engineering — also, pro tip: I highly value that when interviewing candidates!

But the more I advance and grow up, the more things I learn and the more things I master. And as I master new things, I tend to get used to them. And the “mental effort” it requires tends to slowly disappear. I remember when I tried (hard!) to wrap my finger around what a profunctor is, or what free monads or even natural transformations are and how to use them. It was hard. As a friend of mine (@lucasdicciocio) perfectly summarized it some times ago, when we learn something as a beginner, we struggle. But people who master those concepts struggle even harder when trying to learn the next concepts at hand. And I’m not an exception. Today, free monads are something I know (even though I don’t use them; it’s a myth! it’s a trap! :D) and I feel like I’m not legitimate to talk about them because they feel easy — and it’s normal, since I’ve learned them. But I see new things to learn, new hard things that I know I’ll be struggling with for a while. And now it feeds the imposter syndrom. You know, when you’re 20, you feel like you know a lot of things (and you likely do) and that you’re pretty confident with your skill base. Today, I’m 27, and I feel like I will always be a rookie, because there is always something I don’t know to struggle with. And I find that very exciting. Of course, I don’t project that on others, which is soooo paradoxal: I don’t expect a newcomer to tell me what a fundep is or know lifetime elision rules in Rust. But I guess I have a very very strict way to judge my knowledge. I still haven’t figured out if it’s a good thing or if it’s not.

I’ve been told that talking about what I do, what I think, what I know and my engineering philosophy should help with this kind of questioning. So I decided to use that blog article to share some thoughts about the next release of luminance that took me so much of my spare-time (but happily!) and to talk a bit about myself. I know a lot of people in the Rust and Haskell community don’t know me, but I also know lots of other folks do know me, in both communities. I hope my little vent will give people some relief — maybe you too have a similar experience? Please share your comments on the Reddit thread this article is post in.

As always, stick doing awesome things and spend your spare-time wisely. Spend time with the ones you love and do the things you hecking love too!

Keep the vibes!

(Short) story of what happened

A huge amount of features, bug fixes, usability / comfort enhancement, new crates and architecture redesign are gathered in that update. First, let’s talk about the new most important feature: the redesign.

The backend (re)design

Before 0.40, luminance was an OpenGL 3.3 crate. If you’re not into graphics APIs, OpenGL is a graphics normalized specification, built by an open group (Khronos) that has been around for decades. Because it’s an open specification, lots of vendors can implement and provide drivers to have OpenGL support, but also Free and OpenSource projects. The 3.3 version was old enough to support a wide variety of devices, even old ones, while still having interesting features allowing to do lots of useful things.

However, as time passes, new devices and technologies emerge. WebGL 1.0, which appeared in 2011, is to the Web what OpenGL is to desktop and console graphics programming. It is an open API browsers can implement to provide support for accelerated 2D/3D rendering in your browser.

WebGL 2.0, which appeared more recently, in 2017 (partial support in some browsers), provide an API almost identical to what you find in OpenGL 3.3 (with some tricky caveats).

I’ve been wanting to have a way to write “luminance code”, and have it compile for other platforms than desktops, like Web and mobile (Android and iOS). With the previous architecture, since luminance was bound to OpenGL 3.3, it was not possible. Here comes the new archicture.

Backends… backends? backends!

The whole luminance crate was rewritten so that its public-facing interface is completely agnostic of what executing GPU code is about. No OpenGL anymore. Instead, it depends on a type variable, referred as B in luminance’s types, which must satisfy a set of conditions to be considered a backend.

A backend is an implementation of the expectations made on the public-facing interface. For instance, you can set a GPU buffer by providing an index and a value. That’s the public facing part. However, what happens on the GPU is left to the backend implementation.

This new way of doing split the luminance crate into several ones:

• luminance, which is the core, more abstract and common crate to whole. You use types coming from this crate to explain what you want to do. How it’s done is made by other crates.
• luminance-gl, which is the OpenGL backend crate. It roughly contains a type — GL33 — which can be passed around luminance to select the OpenGL 3.3 backend.
• luminance-webgl, which is the WebGL 2.0 crate. I decided to go with WebGL 2.0 instead of 1.0 because of how similar it is to OpenGL 3.3 (which was the existing code base). It’s possible to add support for WebGL 1.0, but it will require a lot of effort to support the feature that are not present in the specification.

Added to this, because of how generic and abstract luminance now is, people might get issues when trying to write code that will work whatever the backend. When using the set functions on a buffer, it is required that the type implements a trait from luminance — luminance::backend::buffer::Buffer. So it’s likely the user will have to constrain types and it might get boring. For this reason, a new crate was created: luminance-front.

luminance-front allows people to write code using types from luminance that got their B type variable replaced by a backend type at compile-time. This is done by inspecting the compilation target, and features you enable. You are then free to write your code without worrying about whether traits are implemented, since luminance-front takes care of that for you. The re-exported types are simple aliases to luminance’s types, so your generic code — if you write any — will be compatible.

Changelog, migration guide

The changelog is quite massive for a single update. I wish I had the opportunity to split it into several updates, but the redesign meant a lot of changes, and I got several very interesting PRs and feature requests. Among very new stuff:

• A new platform crate has appeared: luminance-sdl2, which adds support for the sdl2 crate.
• luminance-webgl and luminance-web-sys, to support the Web!
• luminance-front, which is a front crate to ease working with luminance types.
• The type system experience has been greatly improved. Most of the time, you will not have to annotate types anymore — like Program or Tess.
• About Tess, a BIG update has landed, has it’s now heavily typed (vertex type, index type, vertex instance data type, memory interleaving type).
• More render states features, such as the possibility to enable or disable depth writes, separate RGB/alpha blending, etc. etc.

A huge thanks to all the people who contributed. It means a lot to me! The list of changes is big, so I’ll leave you with the changelog here.

Disclaimer: this is the luminance changelog, but all the crates have their own, too.

Because that update is massive and has some breaking changes, I have also decided to include a migration guide. The migration guide is a section in luminance’s changelog to explain all the things you need to do to migrate from luminance-0.39 to luminance-0.40. Please: if you find something that is missing or you’re struggling with, please open an issue, a PR, or ping me on whatever platform you like.

What’s next to read

The Tess type now supports slicing deinterleaved memory in very, very elegant ways, checked at compile-type and type-driven. The whole new Tess type design is so new that a blog article is on its way about that topic, as I find it very interesting.

While testing the redesign and new features, I have implemented a Conway’s Game of Life for fun — no code available yet. I plan to remake it from scratch and record the whole experience on video. That will be a new format and I want to hear from people and know whether they like the format.

Finally, the book was also updated. The three chapters have been updated and some features have been added — like aspect ratio in chapter 3; I don’t understand why I haven’t talked about that earlier!

The next steps for luminance and myself, besides the blog articles and video, are… getting some rest. I plan to get back to demoscene production, and I’m 100% sure there’s already a lot to do with that current release of luminance. I plan to experiment with new backends as well, such as an OpenGL 2.0 backend (if possible), to support really old hardware; an OpenGL 4.6 backend for modern hardware (easy), as I already know that API pretty well; an OpenGL ES backend for mobile development, and later, a Vulkan and, perhaps, a WebGPU backends.

Stay tuned for the Tess article, the Game of Life video… and keep making super fun video games, animation and toys with luminance!

And keep the vibes!

Before starting up, if you don’t know what hop.nvim is, here’s a small excerpt from the README:

Hop is an EasyMotion-like plugin allowing you to jump anywhere in a document with as few keystrokes as possible. It does so by annotating text in your buffer with hints, short string sequences for which each character represents a key to type to jump to the annotated text. Most of the time, those sequences’ lengths will be between 1 to 3 characters, making every jump target in your document reachable in a few keystrokes.

Today, I want to talk about something at the core of the design of Hop: permutations, and a new algorithm I implemented to (greatly) optimize permutations in Hop.

Disclaimer: it might be possible that the algorithm described here has an official name, as I’m using a well-known data structure, traversing in a way that is also well established. Even though I came up with it, I don’t really know whether it has a name so if you recognize it, or think it is similar to something, feel free to tell me!

Permutations

When you run a Hop command (whether as a Vim command, such as :HopWord, or from Lua, like :lua require'hop'.hint_words()), Hop is going to place labels, called hints, in your window, over your buffer, describing sequences of keys to type to jump to the annotated location in the buffer. Those key sequences are generated as permutations of your input key set, of different lengths. Your input key set is basically the set of keys you expect to type to hop around. For instance, the default key set is made for QWERTY and is:

asdghklqwertyuiopzxcvbnmfj

The way Hop works is taking those keys and generating permutations of increasing length. The goal is to obviously type as less as possible (remember: Neovim motion on speed!), so we want to generate 1-sequence permutations first, such as a, s, d, etc., then switch to 2-sequence permutations, e.g. aj, kn, etc. The most important part of the work done by Hop is to generate those permutations in a smart way.

Until today, I had implemented a single algorithm doing this. Let’s describe it so that I can introduce the actual topic of this article.

First permutation generation algorithm

The first algorithm used to generate permutations in Hop was designed around the idea of being able to generate permutations in a co-recursive way (even though it’s not using co-recursion in Lua): given a permutation, you can generate the next one by running the algorithm on the permutation. Doing that over and over generates more permutations. You can sum it up like this:

let first_perm = next_perm({})
let second_perm = next_perm(first_perm)
let third_perm = next_perm(second_perm)
-- etc. etc. stop at a given condition and return all the permutations

In functional programming languages, we call that kind of co-recursion an unfold. We stop building / generating when meeting a condition. In our case, the condition is when we reach the number of permutations to generate. This first algorithm had several constraints I wanted satisfied:

1. It has to be fast and co-recursive, so that we can pass it the number of permutations to generate and simply generate them until we have reached the number of desired permutations.
2. It has to take into account the user input key set.
3. It has to generate permutations minimizing the number of keys required to jump.
4. Because permutations will be distributed based on the Manhattan distance to the cursor, they must be ordered / sorted in a way that the first permutations in the list are the shorter ones and the last ones are the longer ones.

We already saw 1., so let’s talk about the three remaining points. Taking into account the key set of the user, with this algorithm, is done by splitting it into two sub-sets:

• A set called terminal key set, containing terminal keys. Those keys are used only in terminal position in sequences — i.e. at the very right-most position in the sequence. For instance, in the sequence abcde, e is a terminal key.
• A set called sequence key set, containing sequence keys. Those are keys found as prefixes of terminal keys in sequences. For instance, in the sequence abcde, a, b, c and d all appear before e, so they are sequence keys.

How to know which key is terminal and which is sequence? Well, I use a simple parameter for that, that can be modified in the user configuration of Hop: term_seq_bias (:help hop-config-term_seq_bias if you are interested). It is a floating number parameter specifying the ratio of terminal vs. sequence keys to use. Imagine you have 100 keys in your key set (you typically will have between 20 and 30), setting this parameter to 3 / 4 makes it use 75 terminal keys and 25 sequence keys.

Then, given those two sets, we start from the 0-sequence permutation, a.k.a. {}, and we start using terminal keys. If we have abc as terminal keys and de as sequence keys, we will get the permutations 'a', 'b' and 'c' for n = 3. If we ask permutations for n = 4, we will get 'a', 'b', 'c' but not 'd', as it’s a sequence key. Instead, we will start a new layer by incrementing the dimension of sequences, yielding 2-sequence permutations: we will then generate 'da'. For n = 6, the last permutation in the list is 'dc', which means that for n = 7, we have to do something, as we have run out of terminal keys. Instead of starting a new layer (3-sequence permutations), we traverse the permutation in reverse order, checking that we have exhausted all the sequence keys. Here, we can see that 'd' is not the last sequence key, so we can use the next one, e, and start a new sequence at ea, then eb, ec… and guess what permutation is the next? Since we have run out of both terminal keys and sequence keys, we need to use 3-sequence keys: the next permutation will be dda.

This algorithm is interesting because it allows people to bias (hence the name) the distribution of terminal and sequence keys. However, it is pretty obvious that this algorithm does a pretty poor job at minimizing the overall number of keys to type: it will minimize the number of keys to type for short hops, mostly around the cursor and at mid-distance. I highly advise to use either 3 / 4 or even 0.5 for the bias value. Other values yield… weird results.

The problem and finding a solution

Very quickly, I have noticed something a bit annoying with Hop. Even though it is pretty fast, the distribution of keys for long jumps is often using 3-sequence permutations. And where does it make sense to use Hop the most? For mid to long range jumps. Vim and Neovim are already pretty good at short jumps. For instance, jumping to characters on the same line can be achieved with f and F. If you want to jump to a word on the line above, you can just press k and use horizontal jumps.

I don’t necessarily agree that those are always ideal (and, well, I do use Hop for short hops too), but the situation is not ideal for long jumps in Vim / Neovim. See, I’ve been using Vim and Neovim for more than 12 years now, and I’ve seen lots of people using it. Most of the time, what people use for long jumps is either (or a combination of) one of these:

• Turn on relativenumber and look at relative numbers to jump to the line where the location appears in, press something like 17k or 17j, then use the f / F motions.
• Use the real line number and commands such as :153 or 153gg to jump to line 153, and do the same as the previous point.
• Use { / { / % to quickly skip large portions of text and arrive at destination in less typing effort (I’m actually a big user of %).
• Use marks, LSP, tags etc. to directly jump to semantic locations.

Among all those options, even when you combine them, you will always have to type more keys / take more time to do long jumps. For instance, considering we can jump with f to our location l because it is unique on the line we want to go to, assuming it’s on line 1064, 45 lines above our cursor:

• With relativenumber, we have to press 45kfl: between 5 and 6 keys (depending on whether you have to press shift on your keyboard layout for the digits).
• With real line number, 1064ggfl, between 8 and 9 keys.

So clearly, Hop should help for those jumps as much as possible, and the previous algorithm, even though already an enhancement over typing relative numbers or real line numbers, can still be enhanced. It is already pretty good because it will provide you with, most of the time, between 1-sequence and 3-sequence permutations. For long jumps, assume the longest: 3 keys, plus one key to trigger the Hop mode, you get 4 keys to type at most to jump anywhere.

The goal is to reduce that to 2 keys, i.e. 2-sequence at most, most of the time — 3 keys if you count the key to start Hop. In order to understand the concept of this new algorithm, let’s focus on what was wrong with the former. Consider the following, using abc as terminal keys and de as sequence keys:

-- n = 3
a b c

-- n = 4
--  last 1-sequence
--  v
a b c da

-- n = 6
a b c da db dc

-- n = 9
a b c da db dc ea eb ec

-- n = 10
--                   last 2-sequence
--                   v
a b c da db dc ea eb ec dda

You can see that after only 9 words, we are already using 3-sequence permutations. What about, for instance, the following sequences:

aa ab ac bb dd de …

Obviously, because of the rules explained above about the difference between terminal and sequence keys, those are forbidden: indeed, they would yield ambiguous sequences. For instance, aa and a both start with the same key and one is terminal at a given depth (1-sequence) while the other still has one level (2-sequence): what should we do? Trigger the 1-sequence and never be able to jump to the 2-sequence? Use a timeout so that we have the time to type the second a? But that would make jumping to terminal keys feel delayed / slow, so that’s a hard no. What’s the problem?

The problem is that we are not efficiently using the key space by forbidding those sequences. If you pay attention, you should actually be able to use all possible 2-sequence permutations. But in order to do that… we have to forbid using 1-sequence permutations! Forbidding that will make all 2-sequence permutations unique and unambiguous. The difference is massive: instead of having only 9 permutations shorter than 3-sequence ones, we know have 25, which is the number of keys in the user key set squared: 'abcde' is 5 keys, and 5² is 25. For the default key set for QWERTY, which has 26 keys, it means 26² = 676 maximum 2-sequence permutations, which is a comfortable number for the visible part of your buffer.

The new algorithm: tries and backtrack filling

The new algorithm’s idea relies on using all the keys from your input key set. It doesn’t split it into terminal and sequence keys. It creates tries (a type of search tree, optimized for packed storage and fast traversal) to store the permutations and backtracks to remove ambiguity as we ask for more permutations. For instance, using the user input key set 'abcde':

-- n = 5
a b c d e

-- n = 6
--      e was transformed to ea
--      v
        a b
a b c d e e

-- n = 7
        a b c
a b c d e e e

-- n = 9
        a b c d e
a b c d e e e e e

As you can see, for n = 6, instead of adding a new layer, the algorithm backtracks to fill what is possible to fill: in our case, e, by replacing it with ea and inserting the new permutation at eb. Once a given layer is saturated, the algoritm backtracks again, trying to saturate another layer:

-- n = 10
--    d was transformed to da
--    v
      a b a b c d e
a b c d d e e e e e

-- n = 13
      a b c d e a b c d e
a b c d d d d d e e e e e

-- n = 17
    a b c d e a b c d e a b c d e
a b c c c c c d d d d d e e e e e

-- n = 21
  a b c d e a b c d e a b c d e a b c d e
a b b b b b c c c c c d d d d d e e e e e

-- n = 25
a b c d e a b c d e a b c d e a b c d e a b c d e
a a a a a b b b b b c c c c c d d d d d e e e e e

Here, we have completely saturated the 2-sequence permutations, yielding 25 of them, and cannot backtrack anymore. When backtracking fails, we simply augment the last trie, deepest:

-- n = 26
                                                a b
a b c d e a b c d e a b c d e a b c d e a b c d e e
a a a a a b b b b b c c c c c d d d d d e e e e e e

And we go on. Here, you can see a repetition of the same key, such as e e e e, but because those are tries, they are going to be actually encoded as a single node, having several children. Consider this example:

-- encode these permutations
--
--         a b c d
-- a b c d e e e e

local trie = {
  {
    key = 'a';
    trie = {}
  },
  {
    key = 'b';
    trie = {}
  },
  {
    key = 'c';
    trie = {}
  },
  {
    key = 'd';
    trie = {}
  },
  {
    key = 'e';
    trie = {
      { key = 'a'; trie = {} },
      { key = 'b'; trie = {} },
      { key = 'c'; trie = {} },
      { key = 'd'; trie = {} },
    }
  }
}

Two interesting properties about this algorithm:

• It does distribute permutations more equitably across your buffer as the number of permutations to generate grows, but will do it from the end of tries, making the first permutations the shortest as long as possible.
• When tries are saturated, we can lose 1-sequence permutations, but such a scenario authorizes to jump anywhere with only 2-sequence permutations.

Conclusion

This new algorithm was a lot of fun to come up with, design, test, implement and eventually use, because yes, it is already available (as a default) in hop.nvim. If for any reason you would prefer to use the first one, you can still change that in the user configuration (see :h hop-config-perm_method). Another point, too: since the support for the setup workflow, it’s been possible to locally override Lua function calls opts argument. You can then make a key binding using the first algorithm and another one using the new one – all this is explained in the vim help page 😉. It’s up to you!

Keep the vibes!

• — Day 16: Packet Decoder — // reflections
  • Header parsing
  • The encoding problem
  • The other encoding
  • Bit packing solution
  • Packed next_bits
  • Decoding a literal packet
  • Parsing operators
  • Part 1: summing packet headers’ versions
  • Part 2: evaluating the expression

— Day 16: Packet Decoder — // reflections

This problem doesn’t look too hard at the first glance, but has some hidden complexities than I thought pretty interesting. Just to sum up if you don’t want to open the link, the first part requires you to parse a hexadecimal number and interpret its binary representation as a packet, like what you would find in a network packet format, basically. You need to be able to parse correctly a single packet, which can contain nested packets. In this puzzle, packets represent expressions, that can either be:

• A literal number.
• An operation between packets.

Each packet have a header, comprising a header and its type (literal or operator), and then the actual data of the packet.

From this context, it seems pretty obvious to assume this is going to be a parsing problem. The first part asks you to parse the packet and its nested packets, and add all the version numbers of the headers of all packets. The format goes as this (quoting):

Every packet begins with a standard header: the first three bits encode the packet version, and the next three bits encode the packet type ID. These two values are numbers; all numbers encoded in any packet are represented as binary with the most significant bit first. For example, a version encoded as the binary sequence 100 represents the number 4.

Then, you have the description of a packet representing a literal number:

Packets with type ID 4 represent a literal value. Literal value packets encode a single binary number. To do this, the binary number is padded with leading zeroes until its length is a multiple of four bits, and then it is broken into groups of four bits. Each group is prefixed by a 1 bit except the last group, which is prefixed by a 0 bit. These groups of five bits immediately follow the packet header. For example, the hexadecimal string D2FE28 becomes:

110100101111111000101000
VVVTTTAAAAABBBBBCCCCC

Below each bit is a label indicating its purpose:

• The three bits labeled V (110) are the packet version, 6.
• The three bits labeled T (100) are the packet type ID, 4, which means the packet is a literal value.
• The five bits labeled A (10111) start with a 1 (not the last group, keep reading) and contain the first four bits of the number, 0111.
• The five bits labeled B (11110) start with a 1 (not the last group, keep reading) and contain four more bits of the number, 1110.
• The five bits labeled C (00101) start with a 0 (last group, end of packet) and contain the last four bits of the number, 0101.
• The three unlabeled 0 bits at the end are extra due to the hexadecimal representation and should be ignored.

So, this packet represents a literal value with binary representation 011111100101, which is 2021 in decimal.

I’m just going to focus on this first kind of packet because it’s already interesting.

Header parsing

From this description, if you have ever done bit parsing, you should already know that there is going to be a problem. You can parse a literal packet using this kind of pseudo code (ignoring errors for simplicity, it’s AoC!):

fn parse_packet(&mut self) -> Packet {
  let version = self.next_bits(3);
  let type_id = self.next_bits(3);

  match type_id {
    4 => todo!("literal parser"),
    _ => todo!("operator parser"),
  }
}

The big question is: how do we implement next_bits? Getting 3 bits of information? As you might know, the smallest amount of information a computer can manipulate is a byte, which is 8 bits, so… how do we manipulate less than that?

The encoding problem

I would like to say one thing before going on. The input is a hexadecimal string. The first step of your solution is to extract the binary representation. For instance, A (10 in decimal) is encoded as 1010 and 5A is encoded as 01011010. As you can see (might already know), hexadecimal is easy to parse into bytes: group hexadecimal digits by two, convert each digit to 4 bits, and glue them together. 5A is 5 -> 0101 and A -> 1010, which gives 01011010 as seen above.

But as we have seen with next_bits above, we will want to manipulate bits, not bytes. So you have several choices and design decisions to make here:

You can go the « easy » way and store each bit as a byte. This is very suboptimal, because you are going to waste 8 times the amount of memory you actually need to store all this. For instance, 5A (01011010), which is a single byte on your computer, will require 8 bytes to be stored: 00000000, 00000001, 00000000, 00000001, 00000001, 00000000, 00000001, 00000000. That is very inefficient, but it would work, if you put them in a Vec<u8>. To do that, you would basically need to read one hexadecimal digit at a time (which has 4 bits), and then perform bitwise right shifts with masking (& 0x1) to extract the bit. Something like:

fn inefficient_bit_extract(&mut self, four_bits: u8) {
  for i in 0..4 {
    self.bits.push((four_bits >> i) & 0x1);
  }
}

With this solution, implementing next_bits is easier, because you only need to pop the Vec<u8>. Let’s write next_bits with this kind of implementation. Because we do not need more than 15 bits in this puzzle (explained later in the puzzle text, not really important for now), we will return the read bits on u16:

fn next_bits(&mut self, n: usize) -> u16 {
  assert!(n < 16);

  let mut out = 0;

  for i in 0..n {
    out = (out << i) | self.bits.pop() as u16;
  }

  out
}

Let’s explain quickly.

    out = (out << i) | self.bits.pop() as u16;

This will extract a single bit, like 00000001, will convert it to 16-bit (so 0000000000000001) and will OR it to the number we are building. That number is left shifted at each iteration so that we only OR the least significant bit at each iteration. For instance, if we call next_bits(3) and we get 101, this implementation will basically store this in out at each iteration:

• End of 1st iteration: 0000000000000001.
• End of 2nd iteration: 0000000000000010.
• End of 3nd iteration: 0000000000000101.

Easy, and works.

The other encoding

However, I am a perfectionist. This is not satisfying to me, and it has been years since I have been wanting to implement something like what I am about to describe. I have been thinking about this problem with a different context but it is similar: a bool value, in a computer, is stored on the minimal amount of data a computer can manipulate: a byte. It means than storing a boolean value wastes 7 bits. If you have a single boolean value, there is nothing you can do. But if you need several boolean values, like, six booleans… using 6 bytes still wastes 42 bits for 6 bits of useful data! And 6 bits can hold in a single byte. So we should be able to store booleans in a packed array of bytes. The idea is the following: imagine that you want to store six boolean value in a byte:

00100101
ABCDEFGH

I have labelled each bit with a letter to make it easier to reference them. If you want to, for instance, the boolean value stored at index 2 (F in our representation), all you have to do is to right shift that number using the index of the element and 1-bit mask:

fn read_boolean(byte: u8, index: usize) -> u8 {
  assert!(index < 8);

  (byte >> index) & 0x1
}

If you want to set it, you need to do the opposite operation using a logical OR:

fn set_boolean(byte: &mut u8, index: usize) {
  *byte = *byte | (0x1 << index);
}

I wanted to use this AoC 18 day to encode my solution as a packed array of bits, and still have the next_bits interface that would allow me to get between 0 and 15 bits of data, encoded as u16. The catch is: because I bit pack the hexadecimal number, it is going to lie in a Vec<u8>. And the whole complexity relies in the fact asking for N bits can span across several bytes in the packed array…

Bit packing solution

The first thing we need to do is to think about the interface. I want that mutable next_bits interface. What it means is that I want to be able to read as many bits as I want (given less than 16 bits), I should not have to worry about the byte implications behind the scenes. The puzzle text clearly shows that when we are going to parse a literal value, we will have to read groups of 5 bits. Reading two literal numbers already implies two bytes (10 bits), and even the first group implies reading two bytes, because before that, you have the header on 6 bits (3 bits for the version, 3 bits for the type ID), living 2 bits to read for the beginning of the group, and 3 bits in the next byte to read. That’s a lot of complexity that should be hidden from the interface.

Let’s start with the Decoder type:

#[derive(Clone, Debug)]
struct Decoder {
  bytes: Vec<u8>,
  bitpos: usize, // the position in bit in the byte array
}

This is fairly simple: bytes is the bit-packed array I mentioned, and bitpos is the current position in that array. But it’s not the position in the array elements, it’s the position in bits. If bytes has 3 bytes (24 bits), then bitpos can be between 0 and 23. To know in which actual array element the bit is, we simply need to divide by 8. For instance, if bitpos = 17, then 17 / 8 = 2 tells us that the bit is in bytes[2]. To know the actual bit position in that byte, you will have guessed it, you need to take the modulo: 17 % 8 = 1, which is the second most significant bit. You can convert to least significant bit by subtracting that number to 8: 8 - 1 = 7.

With all that information, let’s fill the bytes:

impl Decoder {
  fn new(input: &str) -> Self {
    let digits: Vec<_> = input.chars().flat_map(|hexa| hexa.to_digit(16)).collect();
    let bytes = digits.chunks(2).map(|d| (d[0] << 4 | d[1]) as u8).collect();

    Self { bytes, bitpos: 0 }
  }

Here, because this is AoC, I don’t care about errors, and hence use flat_map to remove Option or Result in iterators. The important thing here is that I group hexadecimal digits by 2 (the .chunks(2) part), and then simply do this:

|d| (d[0] << 4 | d[1]) as u8

d[0] are the most significant 4 bits, so I left shift them by 4 bits, and d[1] is already correctly positioned, so nothing to do with it. I convert to u8 parse to_digit(16) yields u32.

Then, let’s implement next_bits.

Packed next_bits

  fn next_bits(&mut self, n: usize) -> u16 {
    // how many bits are still available from the current byte
    //
    // 0000000011111111
    //      ^ position is 5, so we have 3 bits still available
    let bits_avail = 8 - self.bitpos % 8;

    if bits_avail >= n {
      // we have enough in the current byte; compute the new position and right shift to align correctly
      let shift = bits_avail - n;
      let byte = (self.bytes[self.bitpos / 8] as u16 >> shift) & ((1 << n) - 1);
      self.bitpos += n;
      byte
    } else {
      // …
    }

The first step to do is to check whether the amount of bits requested would make us overlap with another byte given the current bitpos. If it’s not the case, then the problem is trivial. As shown in the example, we can just simply right shift by the number of available bits minus the requested amount (because we might still have room afterwards), and mask the N least significant bits. By the way, this is a funny one: setting to 1 the N significant bits. If you have N = 3, you want (on 8-bit here to simplify) 00000111. How do you make that quickly? It’s actually quite simple: what is this number? This 111? It’s an odd number (because its least significant bit is 1) and it looks like it’s almost 2³. It’s actually 2³ - 1. And this makes sense: remember in school / when you were learning binary that the biggest number you can represent on N bits is 2^n - 1. Like, on 8-bit, 2⁸ is 256, and the biggest number you can represent is 255, as you might already know, which is 11111111. So, we can simply deduce 00111…1 is the biggest number for 2^N where N is the number of 1. And that number is 2^N - 1. A power of 2, in binary, is a simple shift. So, 2^N is 1 << N. And `2^N

• 1is…(1 << N) - 1`. You have it.

We then increment bitpos by the number of read bits, and return the actual byte, for which the read bits are right shifted.

Now, what happens when bits_avail < n? We have to read bits_avail and shift them in the 16-bit output. Then we will have to switch to the next byte, and read n - bits_avail additional bits, because n is the requested number of bits to read and since we read every bits available from the current bytes, well, n - bits_avail. The « byte switching » logic can be a bit overwhelming at first but is actually not that hard: once we have read bits_avail, we know that the next byte will have 8 bits available. We can then divide the problem in two sub-problems, easier:

1. Read and shift 8 bits of data as long as n >= 8.
2. Read the remaining bits in the last byte.

This logic is implemented in the } else { branch, so I will just copy the else and put the code, and explain it:

    } else {
      // this is the annoying part; first, get all the bits from the current byte
      let byte0 = (self.bytes[self.bitpos / 8] as u16) & ((1 << bits_avail) - 1);
      self.bitpos += bits_avail;

      // then, left shift those bits as most significant bits for the final number
      let mut final_byte = byte0;

      // then, we need to get the remaining bits; compute the number of bytes we need
      let n = n - bits_avail;
      let bytes_needed = n / 8;

      // while we require more than 1 byte, the problem is trivial; just copy and left shift the bytes
      for _ in 1..=bytes_needed {
        let byte = self.bytes[self.bitpos / 8] as u16;
        self.bitpos += 8;
        final_byte = (final_byte << 8) | byte;
      }

      // the remaining bits can be extracted from the last byte
      let n = n % 8;
      let shift = 8 - n;
      let byte = (self.bytes[self.bitpos / 8] as u16 >> shift) & ((1 << n) - 1);
      self.bitpos += n;

      final_byte = (final_byte << n) | byte;

      final_byte
    }
  }

Let’s dive in.

      let byte0 = (self.bytes[self.bitpos / 8] as u16) & ((1 << bits_avail) - 1);
      self.bitpos += bits_avail;

This is the current byte, from which, as said earlier, we need to read all available bits. Because we are going to read everything from the current position to the least significant bit, we can just read the whole byte and apply a mask for the bits_avail least significant bits (remember the explanation earlier). So & ((1 << bits_avail) - 1) does the job. Because we have read bits_avail bits, we increment bitpos as well.

      // then, left shift those bits as most significant bits for the final number
      let mut final_byte = byte0;

This part is not strictly needed as I could have called byte0 final_byte in the first place, but I wanted a clear mind about what was doing what. The idea for the following steps is that we are going to accumulate bits in final_byte by left shifting it before applying the logical OR. The amount of shift to apply depends on the amount of bits read. For the case where we read a whole byte, we will just right shift by 8 bits.

      // then, we need to get the remaining bits; compute the number of bytes we need
      let n = n - bits_avail;
      let bytes_needed = n / 8;

This is a pretty straight-forward code given what I said above. n - bits_avail is the remaining amount of bits to read. bytes_needed is the number of full bytes we will have to go through (in self.bytes), and from which we will extract 8 bits each. The rest then makes more sense:

      // while we require more than 1 byte, the problem is trivial; just copy and left shift the bytes
      for _ in 1..=bytes_needed {
        let byte = self.bytes[self.bitpos / 8] as u16;
        self.bitpos += 8;
        final_byte = (final_byte << 8) | byte;
      }

Then, the remaining bits are the last part of the parsing problem. The idea is that we know we will have to read between 1 and 7 bits from that byte. That information is stored as n and shift:

      let n = n % 8;
      let shift = 8 - n;

We then extract the information, and return the N-bit number:

      let byte = (self.bytes[self.bitpos / 8] as u16 >> shift) & ((1 << n) - 1);
      self.bitpos += n;

      final_byte = (final_byte << n) | byte;

      final_byte

Decoding a literal packet

Armed with next_bits, the problem really becomes trivial. Let’s decode a literal packet and introduce the Packet type:

#[derive(Clone, Debug, Eq, PartialEq)]
struct Header {
  version: u8,
  type_id: u8,
}

#[derive(Clone, Debug, Eq, PartialEq)]
enum Packet {
  Literal {
    header: Header,
    value: u128,
  },

  Operator {
    header: Header,
    length_type_id: u8,
    packets: Vec<Packet>,
  },
}

And Decoder::decode:

impl Decoder {
  // …

  fn decode(&mut self) -> Packet {
    let version = self.next_bits(3) as u8;
    let type_id = self.next_bits(3) as u8;
    let header = Header { version, type_id };

    match type_id {
      // literal value
      4 => {
        let mut value = 0u128;
        loop {
          let bits = self.next_bits(5) as u128;
          value = (value << 4) | (bits & 0xF);

          if (bits >> 4) & 0x1 == 0 {
            break Packet::Literal { header, value };
          }
        }
      }

      // operator
      _ => {
        todo!()
      }
    }
  }
}

Some explanation. Given the format, we know that a literal number is made of packets of 5 bits, where the most significant bit does not belong to the number, but instead is a flag. If set to 0, it means that the literal is over.

          value = (value << 4) | (bits & 0xF);

This accumulates the 4 least significant bits of the 5 bits into the actual literal number.

          if (bits >> 4) & 0x1 == 0 {

And this performs the test on the most significant bit to check whether the literal number is fully built.

Parsing operators

Operators represent the other kind of packets, and are introduced, quoting:

Every other type of packet (any packet with a type ID other than 4) represent an operator that performs some calculation on one or more sub-packets contained within. Right now, the specific operations aren’t important; focus on parsing the hierarchy of sub-packets.

An operator packet contains one or more packets. To indicate which subsequent binary data represents its sub-packets, an operator packet can use one of two modes indicated by the bit immediately after the packet header; this is called the length type ID:

• If the length type ID is 0, then the next 15 bits are a number that represents the total length in bits of the sub-packets contained by this packet.
• If the length type ID is 1, then the next 11 bits are a number that represents the number of sub-packets immediately contained by this packet.

Finally, after the length type ID bit and the 15-bit or 11-bit field, the sub-packets appear.

There is nothing else to do than just applying those rules:

      // in Decoder::decode’s match on type ID

      // operator
      _ => {
        let length_type_id = self.next_bits(1) as u8;
        let mut packets = Vec::new();

        if length_type_id == 0 {
          let total_length_bits = self.next_bits(15) as usize;
          let mut bits_read = 0;

          while bits_read < total_length_bits {
            let bitpos = self.bitpos;
            packets.push(self.decode());
            bits_read += self.bitpos - bitpos;
          }
        } else {
          let sub_packets_nb = self.next_bits(11);

          for _ in 0..sub_packets_nb {
            packets.push(self.decode());
          }
        }

        Packet::Operator {
          header,
          length_type_id,
          packets,
        }
      }

Because of the next_bits interface, for when length_type_id == 0, we have no way to know deterministically when to stop, because we don’t really know how long each packet is. Because of this, the bits_read variable is introduced so that we know how many bits we read. That inforation is easy to get by comparing the value of bitpos before and after reading a packet. For the other branch, there is nothing interesting to comment here.

Part 1: summing packet headers’ versions

The first question was:

Decode the structure of your hexadecimal-encoded BITS transmission; what do you get if you add up the version numbers in all packets?

And my input was (it was a single string but I break it on several lines, because it’s pretty long):

60552F100693298A9EF0039D24B129BA56D67282E600A4B5857002439CE580E5E5AEF67803600D2E294B2FCE8AC489BAEF37FEACB31A678548034EA0
086253B183F4F6BDDE864B13CBCFBC4C10066508E3F4B4B9965300470026E92DC2960691F7F3AB32CBE834C01A9B7A933E9D241003A520DF31664700
2E57C1331DFCE16A249802DA009CAD2117993CD2A253B33C8BA00277180390F60E45D30062354598AA4008641A8710FCC01492FB75004850EE5210AC
EF68DE2A327B12500327D848028ED0046661A209986896041802DA0098002131621842300043E3C4168B12BCB6835C00B6033F480C493003C4008002
9F1400B70039808AC30024C009500208064C601674804E870025003AA400BED8024900066272D7A7F56A8FB0044B272B7C0E6F2392E3460094FAA500
2512957B98717004A4779DAECC7E9188AB008B93B7B86CB5E47B2B48D7CAD3328FB76B40465243C8018F49CA561C979C182723D76964220041275627
1FC80460A00CC0401D8211A2270803D10A1645B947B3004A4BA55801494BC330A5BB6E28CCE60BE6012CB2A4A854A13CD34880572523898C7EDE1A9F
A7EED53F1F38CD418080461B00440010A845152360803F0FA38C7798413005E4FB102D004E6492649CC017F004A448A44826AB9BFAB5E0AA8053306B
0CE4D324BB2149ADDA2904028600021909E0AC7F0004221FC36826200FC3C8EB10940109DED1960CCE9A1008C731CB4FD0B8BD004872BC8C3A432BC8
C3A4240231CF1C78028200F41485F100001098EB1F234900505224328612AF33A97367EA00CC4585F315073004E4C2B003530004363847889E200C45
985F140C010A005565FD3F06C249F9E3BC8280804B234CA3C962E1F1C64ADED77D10C3002669A0C0109FB47D9EC58BC01391873141197DCBCEA401E2
CE80D0052331E95F373798F4AF9B998802D3B64C9AB6617080

So:

1. Parse the input and decode it into a Packet.
2. Recurse on the version field of all sub-packets, summing them.

fn solve1(input: &str) -> u32 {
  let mut decoder = Decoder::new(input);
  checksum(&decoder.decode())
}

fn checksum(packet: &Packet) -> u32 {
  match packet {
    Packet::Literal { header, .. } => header.version as _,
    Packet::Operator {
      header, packets, ..
    } => header.version as u32 + packets.iter().map(checksum).sum::<u32>(),
  }
}

Yes, it’s that simple once you have made the right framework. :) Note that this function is not tail-recursive. A tail-recursive version would probably perform faster, but it was so convenient to write and took me 15s.

Part 2: evaluating the expression

Part 2 explains that the operator packets are actually expression operators, operating on literal values. Basically, it’s a whole AST. The rules are:

• Packets with type ID 0 are sum packets - their value is the sum of the values of their sub-packets. If they only have a single sub-packet, their value is the value of the sub-packet.
• Packets with type ID 1 are product packets - their value is the result of multiplying together the values of their sub-packets. If they only have a single sub-packet, their value is the value of the sub-packet.
• Packets with type ID 2 are minimum packets - their value is the minimum of the values of their sub-packets.
• Packets with type ID 3 are maximum packets - their value is the maximum of the values of their sub-packets.
• Packets with type ID 5 are greater than packets - their value is 1 if the value of the first sub-packet is greater than the value of the second sub-packet; otherwise, their value is 0. These packets always have exactly two sub-packets.
• Packets with type ID 6 are less than packets - their value is 1 if the value of the first sub-packet is less than the value of the second sub-packet; otherwise, their value is 0. These packets always have exactly two sub-packets.
• Packets with type ID 7 are equal to packets - their value is 1 if the value of the first sub-packet is equal to the value of the second sub-packet; otherwise, their value is 0. These packets always have exactly two sub-packets.

This is my solution:

fn solve2(input: &str) -> u64 {
  let mut decoder = Decoder::new(input);
  eval(&decoder.decode())
}

fn eval(packet: &Packet) -> u64 {
  match packet {
    Packet::Literal { value, .. } => *value as _,
    Packet::Operator {
      header, packets, ..
    } => match header.type_id {
      0 => packets.iter().map(eval).sum::<u64>(),
      1 => packets.iter().map(eval).product::<u64>(),
      2 => packets.iter().map(eval).min().unwrap(),
      3 => packets.iter().map(eval).max().unwrap(),
      5 => {
        if eval(&packets[0]) > eval(&packets[1]) {
          1
        } else {
          0
        }
      }
      6 => {
        if eval(&packets[0]) < eval(&packets[1]) {
          1
        } else {
          0
        }
      }
      7 => {
        if eval(&packets[0]) == eval(&packets[1]) {
          1
        } else {
          0
        }
      }

      id => panic!("unknown type id: {}", id),
    },
  }
}

And here you have it all. The complete solution is available on my GitHub repository for AoC 21. I don’t think it was the hardest problem, nor the more interesting, but the decision (and challenge) I decided to go with (bit-packing with a bit-driven interface) made the whole thing much more fun and interesting to me.

I hoped you liked it and that you learned something, especially regarding bitwise operations. That was pretty heavy! I think that you should also now have an idea about how you could write a specialized version of Vec<bool> that bit-packs booleans value. Obviously, we do not want Vec<bool> to actually do that kind of specialization behind the scenes (think FFI: when getting a *const bool or *mut bool from the Vec<bool>, if those pointers point to bit-packed data… it’s going to be a nightmare if the FFI function you use expect an actual array of unpacked boolean values).

For the rest of AoC, I’m probably going to continue working on it but I’m currently pausing at day 18 (Christmas, fatigue, a bit of OSS burnout too). I will probably write some more about AoC once I feel rested.

Have fun and keep the vibes!

I’m very happy about people getting interested about my luminance graphics framework. I haven’t received use case feedback yet, but I’m pretty confident I will sooner or later.

In the waiting, I decided to write an embedded tutorial. It can be found here.

That tutorial explains all the basic types of luminance – not all though, you’ll have to dig in the documentation ;) – and describes how you should use it. I will try to add more documentation for each modules in order to end up with a very well documented piece of software!

Let’s sum up what you need

People on reddit complain – they are right to – about the fact the samples just “didn’t work. They actually did, but the errors were muted. I released luminance-0.1.1 to fix that issue. Now you’ll get the proper error messages.

The most common issue is when you try to run a sample without having the required hardware implementation. luminance requires OpenGL 4.5. On Linux, you might need to use primusrun or optirun if you have the Optimus technology. On Windows, I guess you have to allow the samples to run on the dedicated GPU. And on Mac OSX… I have no idea; primusrun / optirun, I’d go.

Anyways, I’d like to thank all people who have/will tried/try the package. As always, I’ll keep you informed about all the big steps I take about luminance. Keep the vibe!

I’ve made a listing with pros and cons. in the README file of luminance. I’ll just copy that section and report it below so that you’re not even a click away from the listing!

How does it compare to…

A lot of folks have asked me on IRC / Reddit / gitter / etc. how luminance compares to other famous frameworks. Here are a few comparisons:

luminance vs. glium

glium has been around for a while – way longer than luminance.

According to crates.io, glium is no longer actively developed by its original author.

• It has more macros to generate impl, which is neat – however, this tends to be less and less true as luminance has been getting new macros lately.
• It addresses the OpenGL concepts more directly. For instance, you find a glium::VertexBuffer type – a buffer to hold vertices – while luminance gives you a Tess type that hides that kind of complexity away for you.
• glium doesn’t prevent you to pick a specific version of OpenGL / GLSL if you want to. For instance, the tuto-02-triangle.md sample uses GLSL140. luminance is different on that side as it must be used with an OpenGL 3.3 Core Context – hence a GLSL330. This can be seen as a luminance drawback as it will only work with a specific version of OpenGL (GL 3.3 and GLSL 330).
• Drawing is done by requesting a Frame object in glium and passing the whole pipeline at once. The pipeline then must be completely resolved prior to render. See this. luminance is more flexible on that side since it enables you to provide the rendering pipeline in a dynamic way, allowing for tricky structure traversal without cloning nor borrowing, for instance. glium requires a tuple of (vertices, indices, program, uniforms, render_state) while luminance works by building the AST node by node, level by level.
• The program customization (uniforms) feature of glium is per-value while luminance is per-type. In glium, you can change the uniforms you pass to a program by using the uniform! macro when invoking the draw command on your Frame. This has the effect to lookup the uniform in the shader for every call – modulo caching – and requires you to explicitly pass the uniforms to use. luminance uses a contravariant approach here: you give the type of the uniform interface you want to use and you will be, later, handed back a reference to an object of that type so that you can alter it. All of this is done at the type level; you have nothing to do when passing values to the shader in the pipeline. Opposite approaches, then. Also, notice that luminance looks up the uniform names exactly once – at Program creation – and provides several hints about uniform utilization (inactive, type mismatch, etc.). glium seems to have runtime type checking as well, though.
• Linear algebra seems to be handled the same way (e.g. 4×4 matrices are encoded as [[f32; 4] 4] in both crates).
• Textures are handled a bit similarily, even though, because glium uses a fixed pipeline, the user doesn’t have to do anything special but passing references around. luminance exposes a scoped API to use opaque texture values, which adds more noise. A good point for glium there since the texture passing is just about references while luminance has a specific API for that. This is the same situation for GPU buffers.
• About vertices, glium is more permissive about how data can be stored on the GPU. For instance, it supports deinterleaved vertices – e.g. when you have your positions in a GPU buffer and the normals or colors in another. luminance requires you to interleave your vertices. This is due to the fact that if you use all your vertex attributes in a shader – which was an assumed situation when designing the vertex API – this will provide better performances than using deinterleaved data (because of cache friendliness). However, for cases when you don’t want to use other attributes – depth peeling, shadow mapping, z-pre-pass, etc. – that might be a huge drawback as you’ll peek / load data from the buffer you won’t even use, killing the point of having a cache-friendly peek.
• Draw state (i.e. depth test, depth buffer comparison equations, culling, etc.) is done pretty much the same way in both crates: by creating an object representing the state when calling the rendering pipeline.
• glium supports compute shaders while luminance doesn’t (so far! ;) ).
• Additionnally, because of the dynamic nature of its pipelines, luminance allows for better sharing of resources, especially shaders and render states. This is due to the fact that luminance represents rendering pipelines as dynamic ASTs in which you can change all the branches and leaves on the fly. See the point about the Frame of glium above.

Sources and tutorials for glium here.