I am very interested in tools which support the software development process, making automatic the boring bits. One system which permits to go in that direction is Walkmod: a smart tool which can refactor Java projects and enforce code conventions. It is a quite powerful tool and it should ges more and more attention.
TL; DR: Tools like Sonar find issues in your code, Walkmod fix them for you
I had the chance to get in touch with the lead developer, Raquel Pau, because we are both contributors to JavaParser. I could not resist so I started asking her lot of questions on the project and I thought it would be nice to share her answers. So, here they are.
At the end of this post there is also an example of applying walkmod on JavaParser to remove unused imports.
Hi Raquel, can you tell us something about yourself and your interests?
I am a software engineer from Barcelona, and my interests are related to software modeling with specification languages, metamodeling transformations, language interpreters and compilers. A part from that, I also like to understand how people explains their thoughts and passions through the art.
Can you tell us which problem Walkmod is trying to solve?
Correct the code automatically according to a set of code conventions.
In which scenarios do you think it makes sense to use Walkmod? Is that suitable only for large projects or also small projects can benefit of it?
All projects with more than one developer involved can take benefit from Walkmod. However, it is easy to see that the magnitude of the project has a direct repercussion on the number of developers.
Here there are a few interesting adoption stories:
- MetricStream, a worlwide spread company, has been using Walkmod for the last 10 months and it has created its own plugins for WalkMod to automatically refactor code. They were mainly interested in correcting a set of Sonar issues automatically and using Walkmod they have been able to correct thousands of lines of code. Right now,they are working to publish their plugins on GitHub. A part from MetricStream, ThoughWorks is also using some Walkmod libraries in one of its open source projects, which also contributes to improve the quality of our tool.
- An another side, there are consultancy companies. They are in an interesting position because they work on the same codebase with their customers. I am aware of a few of them that have also used Walkmod (I am not sure I can mention their names). In one case they also developed plugins to integrate Walkmod with other tools (i.e., Gradle).
- Open source Java projects like JUnit, Guava, Arquilian and Apache projects could take special benefit from Walkmod because there are a lot of developers involved and they change all the time. In fact, even if they have some code conventions specified in some documents, these are difficult to manage and review. However, I have learned from them that applying code conventions into the whole project (e.g., applying the Eclipse formatter according their own rules) in a single commit implies too much code to review and consequently, risk. Therefore, they reject these kind of commits. In fact, this is the main reason why the current version of walkmod allows:
- To apply code conventions for a set of files .
- To apply a set of code conventions without rewriting the whole source file according a set of code conventions.
Federico: I see. Yes, I have been doing a lot of code reviews recently and it is tiring and distracting when the code to review is polluted by a lot of minor changes related to style. I can see the benefits of having these changes applied automatically and just real changes being examined during code reviews. It will make life much easier and bring more attention to the real changes, which are often overlooked because of all the attention dedicated to the number of spaces between operators or finding the occasional tab character that slip in a commit.
How Walkmod compare to existing solutions?
Currently, companies control people are applying their own conventions using PMD or Sonar. However, this kind of software just checks if a rule is not satisfied, but never corrects the code even it was very easy to solve.
For another hand, IDEs have set of generic and automatic quick fixes (e.g., remove unusued imports). However, there are a lot of editors, and everybody has its prefered editor; and consequently, specially in open source projects, you can’t prevent that people don’t execute the quick fixes of an specific editor before pushing the code.
Federico: Cool, Walkmod is not only more powerful than more automatic quick fixed available in editors but it can be triggered by the command line and permit to have consistency in a project also if everyone is using a different IDE.
How easy is to customize and develop new transformations? I have used M2M languages like QVT before and they were painful
I have had experience designing model-to-model and model-to-text transformations (e.g., using ATL). Considering this experience I decided to not couple Walkmod to any specific transformation language. I think that if we expect people to contribute to create code conventions through code transformations, Walkmod should not require to learn any additional language. Therefore, I have designed 3 ways to design code transformations:
- Using the Visitor pattern: where people just need to add a function called visit for the type element that they want to modify. Afterwards, developers should upload their visitor as a Java library into the Maven repository.
- Using templates: many Java developers have experience working with template technologies such as JSPs, Velocity, Freemaker or Groovy. We selected the Grovy template system (GStringTemplateEngine) as the default template technology for Walkmod.
- Using scripting: Groovy is the scripting language preferred by Java developers and it easy to integrate with Maven.
Who should develop the transformations? Should they be software architects or single developers should do that?
Conventions should be managed by the project leader, but transformations can be created by any Java developer.
Do you think that transformations tend to be very general and can be shared across projects or are they more typically project specific?
I think that in general, software architectures are the composition of several generic solutions configured or parameterized for a specific project. So, I think that if transformations are well coded, they tend to be general and the code conventions of a project consist of configuring these generic transformations.
How Walkmod integrate with other tools? Is it typically used with CI tools like Jenkins or Travis?
Walkmod can be integrated in Forge, Eclipse, Maven or Gradle and it has been designed to be executed locally. If developers add Walkmod as a Maven or Gradle plugin and run it from a Continuous Integration tool, they just can be notified if some source files do not follow their conventions.
Are transformations whitespace preserving? Do they preserve comments?
Currently Walkmod allows multiple ways to apply the changes into the code: using the Eclipse formatter or applying just the minimum changes made by the transformation. In the second scenario, the transformations are whitespace preventing.
All comments are always preserved as any part of the source code during the parsing process.
Are transformations based only on the AST or are they aware of symbol resolution? Can I define rules like “all the classes extend this base class should have a default constructor?”
Transformations are aware of symbol resolution if the visitor class have the annotation @RequiresSemanticAnalysis. Therefore, when your code transformation is executed, all declaration types and expressions have the reflection element that the node is referencing (java.reflect.Class, java.reflect.Method, java.reflect.Field, etc.).
How difficult would be to use Walkmod for other languages, like Python or Ruby?
The Walkmod architecture is completely independent of the programming language because it is completely extensible through plugins. If people would like to work with another language, they should create plugins with the implementation for some parts of the process (e.g., the parsing process).
Walkmod is an open-source tool, can you tell us something about the community? What kind of users do you have? What feedback are you getting?
Walkmod was announced in 2014 and from that moment our community involves MetricStream, ThoughtWorks and developers that work in consultancy services.
How the community could help Walkmod? Do you need more plugins, help with the development, more documentation or more success stories being spread about WalkMod?
Mainly, I would like the community to contribute by writing plugins for Walkmod and sharing their feedback.
What are the plans for the future of Walkmod?
Our plans is to improve the configuration style making it more easy to write and create new plugins. Moreover, we are working on creating a service around the tool.
How is your experience with open-source? There were negative or positive surprises?
I have had the positive surprise on people who give you thanks to create the product and any help that you give to them to solve their problems. The best experience I have had was presenting Walkmod in the Devoxx UK, when one guy comes personally to me to say thank you for my support to create the Maven plugin for Walkmod, which it was an issue he reported some months ago.
So, hopefully that should be enough to convince you to start using Walkmod: the next step is visit www.walkmod.com and give it a try!
Example: using walkmod on the JavaParser source code to remove unused imports
Ok, we described how cool walkmod is, listed a bunch of features, etc. etc. Let’s how it works in practice.
I first of all downloaded walkmod from their website (current version is 1.2.0, available here). I unzipped it and set a few environment variables:
export JAVA_HOME=/usr/lib/jvm/java-8-oracle/jre export WALKMOD_HOME=/home/federico/tmp/walkmod-1.2.0 export PATH=$PATH:$WALKMOD_HOME/bin
Then I configured walkmod to perform one single operation: remove the useless imports. I just created a file name walkmod.xml in the root of the project:
<!DOCTYPE walkmod PUBLIC "-//WALKMOD//DTD" "http://www.walkmod.com/dtd/walkmod-1.0.dtd" > <walkmod> <plugins> <plugin groupId="org.walkmod" artifactId="walkmod-imports-cleaner-plugin" version="2.0" /> </plugins> <chain name="example-chain" > <reader path="/home/federico/repos/javaparser/javaparser-core/src/main/java" /> <transformation type="walkmod:commons:import-cleaner" /> <writer path="/home/federico/repos/javaparser/javaparser-core/src/main/java"/> </chain> </walkmod>
At this point all what I had to do was to run walkmod apply from the root of the project:
It took a little while to retrieve some dependencies but then it went though the source code of the project quite fast and found three files to correct. As a natural reflex I ran the tests and they all passed: the corrections done by walkmod are correct. I then checked the changed files and I noticed that walkmod reformatted them preserving all comments (good) but using an indentation of three spaces which struck me.
Luckily it is super easy to customize the behavior of the formatter: open walkmod-1.2.0/config/formatter.xml and go through the properties to understand what you need to change. For example I was not happy the tabulation size and the indentation size (those two properties having value 3 in the screenshots below):
Now, you could just change the values in place if you are going to use walkmod for one simple project. If you plan to use it for several projects, which have different formatting guidelines that does not work. You can instead specifying a forrmatter configuration in you walkmod file (read here for details).
Then I remembered that JavaParser has its own formatting configuration for Eclipse and Walkmod uses the same format so I just specified the path to this file in the configuration. I reverted my changes on the source code (so that the duplicate imports were back) and re-ran Walkmod: I obtained correct files and nicely formatting.
And the story ended with a new Pull-Request being sent to JavaParser:
I have to say that my impression of Walkmod was quite positive and I will look into using it regularly to ensure the code is formatted correctly and no cruft (like unused imports) creep into the projects that I collaborate with.
In the last few days they have also launched walkmodhub: it is a service that you can use as a webhook for GitHub. Each time you push some code to your repository walkmod will run and send you a Pull-Request if there are violations of your coding guidelines. I think I am going to love this.
| Reference: | Walkmod: automatically refactor code to apply code conventions from our JCG partner Federico Tomassetti at the Federico Tomassetti blog. |
