glsl-quasiquote-0.2 was released early this morning. This new version provides a more stable public API. Two major changes:
glsl_str! proc-macro would have only survived
the 0.1 version. It’s now deprecated and will be
removed soon.glsl! proc-macro now supports GLSL pragmas
(both #version and #extension).“Wait. GLSL pragmas? You said it wasn’t possible because of how the Rust tokenizer thinks newlines are meaningless while the GLSL grammar uses them to separate pragmas from other pragmas and code.”
Yes, at the time of writing glsl-quasiquote, I
just thought that since TokenStream
is about a stream of Rust tokens, GLSL’s pragmas would be
impossible to encode with this mechanism. In fact, it’s actually
impossible per-se: newlines are not Rust tokens and are just
completely ignored by the tokenizer… or are they?
At the announcement of the release of
glsl-quasiquote on Reddit, someone – actually, somebodddy
just pointed out that I could use Rust attributes to encode GLSL
pragmas. That would enable me to have them as regular Rust tokens
at the cost of modifying very, very briefly the GLSL grammar for
the pragmas – nothing bad, really. Their idea, which I liked, was
this:
Instead of:
We could write:
Such a small change yet effective, right? I was appealed by the idea, so I implemented it. The code was a bit messy because I did all the token pattern-matching by hand but I sketched something up that worked in a few minutes. After having a discussion with a friend on IRC (antoyo!), I realized that I was plain wrong from the beginning assertion that newlines are ignored by the Rust tokenizer: they’re not.
The proc_macro
crate, which is used to manipulate Rust tokens, has several items
that I used to implement the quasiquoter:
TokenStream, a stream of tokens. This is a very
opaque type, you can just create an empty one, display it as a
String and marshall to and from an iterator
interface, which element type is TokenTree.TokenTree, a single token or a delimited sequence
of token trees, like a group of tokens logically united (with
brackets, parens, etc.).TokenTree variants, such as
Literal, Punct, Group,
etc.There’s something I just had completely forgotten when claiming
that newlines were ignored: pretty much all of the tokens
contained in TokenTree have an associated Span.
I’m not sure what the initial intent was about with this type, but
in order to use it, I had to add the proc_macro_span
feature. That type gives positional information along with macro
expansion on a token. Specifically, it has a method that gives the
line and column at which the token starts and ends.
This information gives us the line on which a token lays. That’s it. We have the newlines! A newline is just whenever a token following another token has a different line in its span. That’s as easy as it gets.
So instead of implementing GLSL pragmas via Rust attributes, I
decided to implement them the way they are defined in the spec and
the glsl crate: plain
GLSL. The idea was just to match a # and accumulate
tokens in a collection as long as tokens lay on the same line. As
soon as the line changes, we have the full pragma and we can call
TokenStream::from_iter.
The current implementation expects the pragmas to be at the top
part of the quasiquoted GLSL code. You shouldn’t be trying to put
the #version at the bottom of your file, but I wanted
you to be noticed: don’t do that!
Since glsl-quasiquote-0.2, this Rust code now completely compiles and generates, at compile-time, a GLSL AST:
let ast = glsl!{
#version 330 core
#extension GL_ARB_separate_shader_objects : require
void main() {
}
};Some future announcements to come (things that I’ve already been working on and that I might release soon):