Presenting tomelr!

In my previous post Defining tomelr, I started toying with the idea of creating a library that would help convert any Lisp expression to a TOML config, and I share my vision (specification) of what this library would look like.

I wasn’t even sure if I would be able to make the tomelr library feature-complete at least to the extent of what ox-hugo was already doing! But to my surprise, the library development snowballed to a completion much earlier than I thought, and additionally it helped fix some inconsistencies that the older TOML generation code had in ox-hugo.

In this post, I start by (i) giving a broad overview of how the development of tomelr happened, then (ii) briefly describe how it got integrated into ox-hugo, and finally (iii) how the use of this library will unblock the path to addition of some cool features to ox-hugo.

My flow for creating this library #

1. Write the spec for the library.
   • List all the formats of Lisp data I would expect it to process.
   • List the corresponding TOML data I would expect it to generate.
   • Ensure that I am not inventing my own lisp syntax by confirming that the expected TOML output matches the JSON generated from that same lisp form (using the Emacs built-in json.el library).
2. That helped me write the tests first! – Test Driven Development (TDD).
3. I started with writing tests for TOML booleans and then implementing that (because that was the simplest and easiest). Of course, I used ert for this! ert helped me quickly create small modular tests  Here’s the ert test for booleans as an example. and efficiently iterate through modifications in the library code until I got the tests to pass.
4. Once that got working, I set up a continuous integration system using GitHub Actions (GHA). I used GHA because I host my library on GitHub. Also I already have a tried and tested setup that I could get up and going in a matter of few seconds. In general, this concept would apply to any Continuous Integration system. The CI setup step should come early in the development of any project so that incremental feature additions don’t start breaking previously added features 😃.
5. The library development just snowballed after this point .. added support for integers, floats, regular strings, multi-line strings, arrays, TOML tables, arrays of TOML tables. By this time, the library was about 80% finished.
6. Then came the difficult part .. stabilizing the library to support all the varieties of Lisp data I can think of. I must have put in double the time spent so far to finish the remaining 20% of the planned features for this library 👉 tomelr test suite
7. Once I had the test suite complete and passing, it was time to do some code cleanup:
   • Remove duplicate code and break them off into smaller helper functions.
   • See if the function defined in this library is already defined somewhere else (in this case, I was able to use json-plist-p directly from json.el).
   • Proof read the code.
   • Proof read the docstrings and run M-x checkdoc to fix their formatting.
   • Ensure that the code compiles without any warnings.
   • Remove unnecessary customization options and case statements from the library (while continuously ensuring that the ert tests still pass).

Adapting the library to fit ox-hugo #

After polishing the library by its stand-alone testing, I decided to use it with ox-hugo and see how the test suite in that repo fared.

Of course I saw that a lot of tests failed now 😁.

The main issue was that tomelr was constructing multi-line strings such that the spaces translated exactly from Lisp data to TOML. So (tomelr-encode '((foo . "line1\nline2"))) would generate:

foo = """
line1
line2"""

whereas ox-hugo expected the same TOML to look like:

foo = """
  line1
  line2
  """

I had intentionally decided for ox-hugo to have this latter format for multi-line strings because (i) it made it more readable with the triple-quotes out of the way on their own lines, (ii) the indented lines prevented the multi-line string from getting mixed with surrounding TOML parameters, and most importantly (iii) these strings were processed by the Hugo Markdown parser, and so it wasn’t sensitive to horizontal spaces.

And so the tomelr-indent-multi-line-strings feature was born (commit) which optionally made tomelr export multi-line strings as expected by ox-hugo 😎.

Changes in ox-hugo tests #

Once I had finalized the integration of tomelr into ox-hugo, I had only about 30 tests change out of roughly 400 tests. These changes were welcome as they fixed all the inconsistencies in the older TOML generation code in ox-hugo. If interested, you can see this commit for the diff and details, but here’s the gist:

1. Now nil value of a key in Lisp consistently implies that the key should not be exported to TOML. So '((foo . nil)) will result in foo not getting exported to TOML, whether that’s a top-level key or a key in a nested TOML map or array. If you need to set a key to a boolean false, use "false" or any value from tomelr-false.
2. Earlier empty string value as in '((foo . "")) behaved like the current nil implementation. That’s not the case any more. Now that empty string will export as foo = "" in TOML.
3. Now if a string has a quote character (") in it, that value will auto-export as TOML multi-line string. I like the readability of this more than that of backslash-escaped double-quotes.
4. Now the nested tables like [menu."nested menu"] export with their parent table keys like [menu]. As per the TOML spec, this is not required. But now that tomelr has added a generic support for any TOML table, this change happens as a result of consistency 💯.

In summary, the changes in ox-hugo TOML front-matter exports were mostly cosmetic, and if they were not cosmetic, they were consistency fixes.

What’s next? #

tomelr

The library is pretty much feature complete ✨ as many of the examples from TOML v1.0.0 spec have been added to its test suite, and .. it is supporting all the ox-hugo use cases.

The library though has one limitation that I’d like to resolve at some point — Right now, we require the Lisp data to first list all the scalar keys and then list the TOML tables and arrays of tables. But at the moment, I don’t know how to fix that, and also ox-hugo is not affected by that (because it already populates the front-matter alist in the correct order). So fixing this is not urgent, but of course, if someone can help me out with that, I’d welcome that! 🙏.

ox-hugo

Given that tomelr allows robustly exporting any Lisp data expression to TOML, I do not see any value in continuing with YAML generation support using the old custom code.

📢 In near future, I plan to get rid of the org-hugo-front-matter-format customization variable from ox-hugo — thus deprecating YAML export support  This change should not functionally affect the YAML front-matter fans out there because the front-matter that ox-hugo is exporting is mainly for Hugo’s consumption. The only scenario where I see that this change can be breaking is if the user is using YAML format extra front-matter blocks. If so, unfortunately, they will need to convert those to TOML manually. and sticking with using just TOML for the front-matter.

Unblocking some future ox-hugo improvements #

This decision will open up the doors to add more features to ox-hugo like:

1. Exporting Org :LOGBOOK: drawers to TOML front-matter (ox-hugo # 504)

2. Exporting Org Special Blocks to user-configurable front-matter (ox-hugo # 627)

3. Supporting more complex data in Lisp form using :EXPORT_HUGO_CUSTOM_FRONT_MATTER: which could translate to nested TOML tables or arrays of TOML tables.

   Finally, there won’t be a need to use the “Extra front-matter” workaround. For example, it would be possible to represent the data in that first example on that page as :foo ((:bar 1 :zoo "abc") (:bar 2 :zoo "def")) in the :EXPORT_HUGO_CUSTOM_FRONT_MATTER: property.