Command-line JSON processing with jq

In this post we will learn how to parse, pretty-print and process JSON from the command-line with jq. At the end we will even use jq to do a simple JSON to CSV conversion. jq describes itself as a lightweight and flexible command-line JSON processor. You can think of unix shell tools like sed, grep and awk but for JSON.

jq works on various platforms. Prebuild binaries are available for Linux, Windows and Mac OS. See the jq download site for instructions.

For many of the following examples we will use a file named artist.json with the following JSON content:

Pretty-printing JSON and basic jq usage

jq is typically invoked by piping a piece of JSON to its standard input. For example:

Without any arguments jq simply outputs the JSON input data. Note that the output data has been reformatted. jq outputs pretty-printed JSON by default. This lets us pipe minimized JSON to jq and get a nicely formatted output.

jq accepts one or more filter(s) as parameter. The simplest filter is . which returns the whole JSON document. So this example produces the same output as the previous example:

We can now add a simple object identifier to the filter. For this we will use the previously mentioned artist.json file. With .name we select the value of the name element:

Arrays can be navigated using the [] syntax:

To get the name of the first painting we use:

If we want to get the names of all artworks we simply skip the array index parameter:

Processing curl and wget responses

Of course we can also pipe responses from remote systems to jq. This is not a specific feature of jq, but because this is a common use-case we look into two short examples. For these examples we will use the public GitHub API to get information about my blog-examples repository.

With curl this is very simple. This extracts the name and full_name properties from the GitHub API response:

Note we used a comma here to separate different two different filters.

With wget we need to add a few parameters to get the output in the right format:

Pipes, functions and operators

In this section we will into a more ways of filtering JSON data.

With the | operator we can combine two filters. It works similar as the standard unix shell pipe. The output of the filter on the left is passed to the one on the right.

Note that .foo.bar is the same as .foo | .bar (the JSON element .foo is passed to the second filter which then selects .bar).

Pipes can be combined with functions. For example we can use the keys function to get the keys of an JSON object:

With the length function we can get the number of elements in an array:

The output of the length function depends on the input element:

• If a string is passed, then it returns the number of characters
• For arrays the number of elements is returned
• For objects the number of key-value pairs is returned

We can combine the length function with comparison operators:

Assume we want only the artworks whose type is Painting. We can accomplish this using the select function:

select accepts an expression and returns only those inputs that match the expression.

Transforming JSON documents

In this section we will transform the input JSON document into a completely different format.

We start with this:

Here we create a new JSON object which uses the .name element as key. To use an expression as an object key we need to add parentheses around the key (this does not apply to values as we will see with the next example)

Now let’s add the list of artworks as value:   

Next we apply the map function to the artworks array:

map allows us to modify each array element with an expression. Here, we simply select the name value of each array element.

Using the join function we can join the array elements into a single string:

The resulting JSON document now contains only the artist and a comma-separated list of his artworks.

Converting JSON to CSV

We can also use jq to perform simple JSON to CSV transformation. As example we will transform the artworks array of our artist.json file to CSV.

We start with adding the .artworks[] filter:

This deconstructs the artworks array into separate JSON objects.

Note: If we would use .artworks (without []) we would get an array containing both elements. By adding [] we get two separate JSON objects we can now process individually.

Next we convert these JSON objects to arrays. For this we pipe the JSON objects into a new filter:

The new filter returns an JSON array containing two elements (selected by .name and .type)

Now we can apply the @csv operator which formats a JSON array as CSV row:

jq applies JSON encoding to its output by default. Therefore, we now see two CSV rows with JSON escaping, which is not that useful.

To get the raw CSV output we need to add the -r parameter:

Summary

jq is a powerful tool for command-line JSON processing. Simple tasks like pretty-printing or extracting a specific value from a JSON document are quickly done in a shell with jq. Furthermore the powerful filter syntax combined with pipes, functions and operators allows us to do more complex operations. We can transform input documents to completely different output documents and even convert JSON to CSV.

If you want to learn more about jq you should look at its excellent documentation.

Do you want to know how to develop your skillset to become a Java Rockstar?

Subscribe to our newsletter to start Rocking right now!

To get you started we give you our best selling eBooks for FREE!

1. JPA Mini Book

2. JVM Troubleshooting Guide

3. JUnit Tutorial for Unit Testing

4. Java Annotations Tutorial

5. Java Interview Questions

6. Spring Interview Questions

7. Android UI Design

and many more ....

I agree to the Terms and Privacy Policy

Sign up

Thank you!

We will contact you soon.