I started reading Sutton & Barto’s classic Reinforcement Learning book a few years ago. Last year, I took a deep dive into learning RL, building a game env to truly practice RL and ended up making optimizations to an RL library (PufferLib). I am now concluding this journey by publishing my work as OSS.

Here is the main repo: https://github.com/rlplays/rlplays

Editor view

This RL env includes an editor to build individual blocks in C++, have them show up in an editor to build levels/maps, and be instantly RL trainable with no specialization.

The code for a single block looks like this in C++. The block then shows up in the editor; you can move/size it around, twiddle properties, build levels with it. The RL pipeline does not need additional code/obs to know about this “brick” block (it’s automatic). Similar ethos to Spelunky 2, where blocks are self-contained God blocks.

Env block code

Fun thing is you can use an LLM to create blocks. Prompt: “Create an enemy that moves left and right and if it sees the player within a radius, it jumps towards and attacks them. The player can also jump and squash this character.”

Jumping enemy

I did not use much LLMs for the actual code though, (mostly for the blocks/characters/enemies).

Here is another fun one that attacks the player when they are under this triggering block:

Trigger block

Or “create a hovering enemy that bobs around and attacks the player when they are nearby”. I think LLMs can even reason about the level and create them but haven’t tried them yet.

Hover enemy block

The RL env supports some ‘advanced’ features such as curriculum learning (ramp up difficulty with many levels in a syllabus starting from simple to more complex) so you would see a hill climbing like this:

Syllabus/curriculum training

This is what it looks like during RL training:

Training ghost view

All of this is mostly experimental/prototype RL env (read: crappy) code that I wrote.

The goals that I ended up accomplishing were:

  • to make pufferlib faster with lots of low-level optimizations
  • to build a realistic env to understand and play with RL as it’s an awesome branch of science (and something that is not already LLM’ified)
  • to try out advanced concepts like self-play/curriculum learning etc (caveat: not an actual researcher, so I don’t fully understanding them).

Here is my code: https://github.com/rlplays/rlplays (MIT licensed).

If you find this useful, drop me a note (caveat: I am no longer working on this, so can’t support/help!) + support/donate to the awesome OSS projects above.