More and more people are looking into adopting DSLs. While they see the general benefits, and enthusiastically learn the technicalities there is one recurring issue many are facing: figure out on which problem to apply DSLs.
For this reason, I think that sharing experiences and reflections is useful. Today I am very happy to share a conversation I recently had with Glen Braun, a new member of the Strumenta Community. Here Glen explains his own journey into using DSLs and he shares his thoughts of how past projects he’s worked on could have benefitted from DSL’s.
Glen is now retired from a career as a software engineer. The first half of his career was spent in small businesses developing PC database applications, and the second half at Microsoft as a developer and then manager of the Bing Maps data team, and then finally as a program manager at Amazon AWS.
Q: How did you become interested in DSL’s?
I think every programmer is interested in the tools they use for their job, but I got into actual metaprogramming by chance. About 20 years ago, while working at Microsoft, my group changed office buildings, and my new office had a left-behind copy of a book on Lex/Yacc. I was working as a software developer on a documentation tools team, and was lucky to have a boss who gave me a lot of room to learn new things. I ended up writing a parser for the Windows source code to extract the error and event messages for documentation purposes.
Q: What are you working on these days?
Today, I’m retired so I have the time to study DSL technology again. For example, I attempted to work through some textbooks on the pi calculus, and the “Types and Programming Languages (TAPL)” book. I say attempted because these are pretty dense subjects. I study them for a while, then set them aside and go back to them again later.
There are two projects that are always on my mind. One is a language I call JsonPi, based on the pi calculus which is a model of computation based on communication between processes. I want to be able to express agreements around how people use computing together. For example, you say the weekly Strumenta Community Meetups are not recorded – and I believe and trust you when you say that. But anyone could simply use a screen capture tool, or just their phone to record the meeting. I think there’s no way around that, but that doesn’t mean it isn’t important to be able to express agreements around computing, whether enforceable or not.
Q: Can you give an example of that in JsonPi?
To give a trivial example, imagine a simple addition expression like: “1 + 2”. We could represent that in a JSON AST as: { “left”: 1, “op”: “+”, “right”: 2 }, and then an interpreter could reduce this to the value: 3. This is straightforward, but what I was experimenting with was using JSON Schema to express constraints on the AST, and make these constraints expressible within the JsonPi language. I was calling that a “behavioral type system”.
Because the pi calculus is about computing through communication, one could imagine sending and receiving not just data but code, like the “1 + 2” expression. What if, as a silly example, the receiver of a computable expression didn’t want to allow evaluation of expressions which add negative numbers? JSON Schema can express that easily by constraining the valid values of the “left” and “right” members to just positive integers.
A more practical example would be, say, for two people communicating over email to agree that their messages can only be sent between each other. It would be a breach of their agreement to forward the message to someone else. I don’t mean it would be impossible for them to do so, only that there would be an agreement, stated formally, between each other.
As I say, this was pretty ambitious, so I’ve set it aside for a while. I was a music minor back in college so I’ve recently switched to the second project I’m thinking about, which is a functional language for music composition.
Q: That sounds interesting, what do you have in mind?
As I mentioned, I’m retired, so I have time to explore music again too. A lot has changed with respect to computers and music in the last 30 years. There are some very robust API’s, like Music21, and some really good software tools for working with music, either as digital audio or music notation. Some of these are text based like LilyPond, and I was just reading about using TeX for music notation.
I haven’t found too many DSL’s for computing music expressions though. ThoseI have found seem more focused on algorithmic music and languages which express the waveforms of the instruments themselves. Pretty fascinating stuff, but I think it would be useful to have a language for composing musical notation expressions. Kind of like LilyPond and ABC Notation but with actual computable expressions. Simple things like expressions for adding the “intro” and “A” sections of a song together. I think it would also be fun to see how a musical type system might come together. For example, adding some measures in 3/4 time to measures in 4/4 time could cause an error, unless using a cast to a more general type.
Right now I’m working on an ABC Notation parser as a learning example for the DSL technologies I’m finding out about through the Strumenta Community. The problem is I’m learning about so many interesting DSL tools, I’m not making much progress on my parser.
Q: Yes, I know the feeling. You mentioned that for some of the projects you worked on throughout your career, you’d thought a DSL solution might have been a better choice. What makes you think that?
There are a number of reasons. One is that on the application development projects I was a part of, we spent a lot of time working on user interfaces. These user interfaces were then tied to an API which was tied to a backend database. Any change required changes to the database schema, then the API, and finally the UI. Not to mention the time spent in UI design and usability testing. It often felt too much, and too expensive.
I would find myself thinking: “If I owned the company, and I had to do the work of the domain expert myself, would I spend my time building a UI, an API, and so on?” I often thought it was at least worth examining a solution which didn’t do those things, but instead used structured text files, parsers, and some set of programs which worked over these files. Essentially, to use the programmer’s workflow but for the domain our applications were built for.
Q: Do you think the programmer’s workflow would be effective for the user’s of the applications you built?
In some cases I know it was effective because I worked in documentation teams which used text based solutions and programmer workflow tools such as Git for versioning, ticketing systems, and automated build and publishing pipelines. Granted this was a highly technical group of users but even in the less technically oriented domains, the domain experts were very willing to learn new tools, and wanted a part in building a solution for themselves. And, the tools for managing text get better and better.
I think tool building is a natural human thing to do, and manipulating text, the idea of a language both the human and the computer understand, is very appealing. I think the success of very large data spaces like Wikipedia and OpenStreetMap serve as good examples of how powerful it is to give users the option to express themselves to the system in text, even if there might also be tools which provide more structured usage patterns.
Q: You mentioned an application you worked on before which you might use a DSL approach for today. Can you describe that?
Really there were many application engineering projects over my career which I think a DSL approach would be worth considering. I’d use the term “forms-over-data” to describe the design these all had in common. There would be some UI layer, either app based or browser based, connected to one or more services through some API, which ultimately stored the data in, usually, a SQL database. Towards the end of my career, I felt an approach using text files and a programmer’s workflow might work better.
Q: Why is that?
Well, I think, in the past we thought that users needed to be hand guided through the use of an application, but, more and more, I think that they could do just as well, or even better, using a text-based application. Primarily, I felt we could iterate much faster on a text based approach. Each of the applications I’m thinking of required a lot of work to write a forms-based user interface. Also, the projects would have benefited from all the metadata and workflow we get as programmers using tools like Git and the related features of ticketing directly linked to the data of the application, change management directly linked to the data, etc.
Thinking about it again now, we were using transaction processing technology for systems which had very few transactions. Often, a large percentage, 80% or more, of the application had transaction rates of just a few per day, or even less. Text would have worked great for that, and it seems reasonable to build a hybrid of text plus transaction processing where it was required.
Federico: On this topic, I am lately reflecting on how we are seeing intermediate solutions emerge so that in the end we can get a continuum between textual solutions and non-textual ones (either forms-over-data or other graphical or GUI based solution). I think that the way we edit text in typical IDEs is changing: it is becoming more and more like editing “augmented text”, with IDEs showing information that is not present in the text. For example, some IDEs show the name of the parameters being passed in a function call. Those parameter names are read by the IDE on the function declaration and displayed on the function call. Those same IDEs also offer refactoring options which are calculated on the AST and then result in code changes (and therefore text changes). On the other extreme we can get GUI-based editors which permit to define some scripting using text, or projectional editors, which store the AST but in the editor they represent the information in a text-like format, as it is familiar to certain users. So I think that we will get more possibilities to combine textual and forms or other GUI elements and not necessarily have to choose one or the other
Q: Can you explain a specific application which you’d strongly consider a DSL approach today?
On one project I was working on an application for managing training – essentially a small school. The primary entities were courses, instructors, classes, and students. Most of the courses for a school are pretty static, they don’t change thousands of times a second, at least. Even the volume of change related to relationships such as between courses and classes, classes and teachers, classes and students was small. I don’t remember exactly, but we could imagine not even thousands of times every day, though perhaps peak times would be more than this, at least for classes to students. Other data wouldn’t change for months or even years. This is what I mean that a hybrid between portions of the application benefiting from a DSL but others using transaction processing. I haven’t done a detailed analysis, probably you have good guidance on this.
Q: You mentioned you thought faster iteration would be possible. How is that?
In many cases I was called in to convert an application developed by the domain experts themselves who had used some technology they knew such as spreadsheets, into a “real” application. Thinking back, I’m not so sure the forms-over-data solutions we provided then would be the best choice today.
In the case of the training application, the administrators had built a very capable application but using a technology with very real limitations. It was based on a spreadsheet-like technology which had a limit of how many rows it could handle, I think only 100,000 rows, if my memory is correct. They would surely need to exceed this limit, or be forced to partition data in an awkward way. So, there were reasons to use the expertise of engineers. However, the spreadsheet application was proof that the domain experts had the modeling skills to think abstractly about their domain and wanted to take an active role in building their own tools. Using a DSL approach, our team of engineers could have been partners in their design, doing the work of building tools which support their text-based models rather than all the inflexible UI work.
Summary
We can all feel enthusiastic about some approach that looks great on paper, but the problems invariably emerge when we try to apply the ideas to the imperfect reality and work context we are immersed in. For this reason, I think it is very important to discuss real experiences, describe the difficulties encountered and how we cope with them. I am grateful to Glen who shared his experience and I hope more of you will reach out and do the same, so that we can all learn from each other.
Published on Java Code Geeks with permission by Federico Tomassetti, partner at our JCG program. See the original article here: Experiences of Practical DSLs usages: a talk with Glen Braun Opinions expressed by Java Code Geeks contributors are their own. |
