This article is part of our Academy Course titled Developing Modern Applications with Scala.
In this course, we provide a framework and toolset so that you can develop modern Scala applications. We cover a wide range of topics, from SBT build and reactive applications, to testing and database acceess. With our straightforward tutorials, you will be able to get your own projects up and running in minimum time. Check it out here!
1. Introduction
Needless to say that Web and mobile have penetrated very deeply into our lives, affecting a lot our day to day habits and expectations about things. As such, overwhelming majority of the applications being developed these days are either mobile apps, or web APIs or full-fledged web sites and portals.
Classic, old style, console-based applications have largely faded away. They are living their lives primarily on Linux / Unix operating systems, being at the core of their philosophies. However, console-based applications are extremely useful in solving a wide range of problems and by no means should be forgotten.
In this section of the tutorial we are going to talk about developing console (or to say it a bit differently, command line) applications using Scala programming language and ecosystem. The sample application we are about to start building will do only one simple thing: fetch the data from provided URL address. To make it a little bit more interesting, the application will require to provide timeout and, optionally, output file to store the content of the response.
Table Of Contents
2. UI-less and Command Line Oriented
Although there are some exceptions, console applications usually do not have any kind of graphical UI or pseudo-graphical interface.
Their input is either command line arguments passed to execute the application, or just a piped stream from another command or source. Their output is typically printed out in the console (that is why those applications are often called console apps) or, to be more precise, there could be multiple output streams like for example standard output (console) and error output.
One of the most useful capabilities of the console applications is pipelining: the output of one application could be piped as an input to another application. Such extremely power composition allows to express very complex processing pipelines with ease, for example:
ps -ef | grep bash | awk '{ print "PID="$2; }'3. Driven by Arguments
Along this section we are going to get introduced into two powerful Scala frameworks and develop two versions of the sample console application we have outlined before. The first library, scopt, aims to help us a lot by taking care of parsing and interpreting command line arguments (and their combinations) so let us take a closer look at it.
As we already know all the requirements, let us start from restating what we would like to achieve in terms of command line input. Our application would require the first argument to be URL to fetch, the timeout is also required and should be specified using -t (or alternatively --timeout) argument, while output file is optional and could be provided using -o (or alternatively --out) argument. So the complete command line looks like that:
java –jar console-cli.jar <url> -t <timeout> [-o <file>]
Or like that, while using the verbose argument names (please notice that a combination of both is totally legitimate):
java –jar console-cli.jar <url> --timeout <timeout> [--out <file>]
Using terrific scopt library this task becomes rather trivial. First, let us introduce a simple configuration class which reflects command line arguments (and their semantics):
case class Configuration(url: String = "", output: Option[File] = None, timeout: Int = 0)
Now, the problem we are facing is how to get from the command line arguments to the instance of the configuration we need? With scopt, we start from creating the instance of OptionParser where all our command line options are described:
val parser = new OptionParser[Configuration]("java -jar console-cli.jar") {
override def showUsageOnError = true
arg[String]("")
.required()
.validate { url => Right(new URL(url)) }
.action { (url, config) => config.copy(url = url) }
.text("URL to fetch")
opt[File]('o', "out")
.optional()
.valueName("")
.action((file, config) => config.copy(output = Some(file)))
.text("optionally, the file to store the output (printed in console by default)")
opt[Int]('t', "timeout")
.required()
.valueName("")
.validate { _ match {
case t if t > 0 => Right(Unit)
case _ => Left("timeout should be positive")
}
}
.action((timeout, config) => config.copy(timeout = timeout))
.text("timeout (in seconds) for waiting HTTP response")
help("help").text("prints the usage")
}
Let us walk over this parser definition and match each code snippet to respective command line argument. The first entry, arg[String]("<url>"), describes <url> option, it has no name and comes as-is right after the application name. Please notice that it is required and should represent a valid URL address as per validation logic.
The second entry, opt[File]('o', "out"), is used for specifying the file to store the response to. It has short (o) and long (out) variations and is marked as optional (so it could be omitted). In similar fashion, opt[Int]('t', "timeout"), allows to specify the timeout and is required argument, moreover it must be greater than zero. And last but not least, special help("help") entry prints out the details about command line options and arguments.
Once we have parser definition, we can apply it to command line arguments using parse method of the OptionParser class.
parser.parse(args, Configuration()) match {
case Some(config) => {
val result = Await.result(Http(url(config.url) OK as.String),
config.timeout seconds)
config.output match {
case Some(f) => new PrintWriter(f) {
write(result)
close
}
case None => println(result)
}
Http.shutdown()
}
case _ => /* Do nothing, just terminate the application */
}
The result of the parsing is either a valid instance of the Configuration class or application is going to terminate, outputting encountered errors in the console. For example, if we do not specify any arguments, here is what is going to be printed:
$ java -jar console-cli-assembly-0.0.1-SNAPSHOT.jar Error: Missing option --timeout Error: Missing argument Usage: console-cli [options] <url> URL to fetch -o, --out <file> optionally, the file to store the output (printed on the console by default) -t, --timeout <seconds> timeout (in seconds) for waiting HTTP response --help prints the usage
Out of curiosity you can try to run this command providing only some command line arguments, or passing the invalid values, scopt will figure this one out and complain. However, it is going to keep silence if everything is fine and let the application to fetch the URL and print out the response in the console, for example:
$ java -jar console-cli-assembly-0.0.1-SNAPSHOT.jar http://freegeoip.net/json/www.google.com -t 1
{
"ip":"216.58.219.196",
"country_code":"US",
"country_name":"United States",
"region_code":"CA",
"region_name":"California",
"city":"Mountain View",
"zip_code":"94043",
"time_zone":"America/Los_Angeles",
"latitude":37.4192,"longitude":-122.0574,
"metro_code":807
}Awesome, isn’t it? Although we have played with only basic use cases, it is worth noting that scopt is capable of supporting quite sophisticated combinations of command line options and arguments while keeping the parser definitions readable and maintainable.
4. The Power of Interactivity
Another class of the console applications is the ones which offer an interactive, command-driven shell, which could range from somewhat trivial (like ftp) to quite sophisticated (like sbt or Scala REPL).
Surprisingly, all the necessary building blocks are already available as a part of the sbt tooling (which by itself offers very powerful interactive shell). sbt distribution provides the foundation as well as a dedicated launcher to run your applications from anywhere. Following the requirements we have set for ourselves, let us embody them as interactive console applications using sbt scaffolding.
In the core of sbt-based applications lays xsbti.AppMain interface which others (in our example, ConsoleApp class) should implement. Let us take a look at the typical implementation.
class ConsoleApp extends AppMain {
def run(configuration: AppConfiguration) =
MainLoop.runLogged(initialState(configuration))
val logFile = File.createTempFile("console-interactive", "log")
val console = ConsoleOut.systemOut
def initialState(configuration: AppConfiguration): State = {
...
}
def globalLogging: GlobalLogging =
GlobalLogging.initial(MainLogging.globalDefault(console), logFile, console)
class Exit(val code: Int) extends xsbti.Exit
}The most important function in the code snipped above is initialState. We left it blank for now, but don’t worry, once we understand the basics, it will be full of code pretty quickly.
State is the container of all available information in sbt. Performing some action may require to introduce the modifications of the current State, producing a new State thereafter. One class of such actions in sbt is the command (although there are more).
It sounds like a good idea to have a dedicated command which fetches the URL and prints out the response so let us introduce it:
val FetchCommand = "fetch"
val FetchCommandHelp = s"""$FetchCommand
Fetches the and prints out the response
"""
val fetch = Command(FetchCommand, Help.more(FetchCommand, FetchCommandHelp)) {
...
}
Looks simple but we need to somehow supply URL to fetch. Luckily, commands in sbt may have own arguments but it is a responsibility of the command to tell what the shape of its arguments is by defining the instance of Parser class. From that, sbt takes care of feeding the input to the parser and either extracting the valid arguments or failing with an error. In case of our fetch command, we need to provide a parser for URL (however, sbt significantly simplifies our task by defining basicUri parser in the sbt.complete.DefaultParsers object which we can reuse).
lazy val url = (token(Space) ~> token(basicUri, "")) <~ SpaceClass.*
Great, now we have to modify our command instantiation a little bit to hint sbt that we expect some arguments to be passed and essentially provide the command implementation as well.
val fetch = Command(FetchCommand, Help.more(FetchCommand, FetchCommandHelp))
(_ => mapOrFail(url)(_.toURL()) !!! "URL is not valid") { (state, url) =>
val result = Await.result(Http(dispatch.url(url.toString()) OK as.String),
state get timeout getOrElse 5 seconds)
state.log.info(s"${result}")
state
}Excellent, we have just defined our own command! However, an attentive reader may notice the presence of timeout variable in the code snipped above. The State in sbt may contain additional attributes which could be shared. Here is how the timeout is being defined:
val timeout = AttributeKey[Int]("timeout",
"The timeout (in seconds) to wait for HTTP response")With that, we have covered the last piece of the puzzle and are ready to provide the implementation of the initialState function.
def initialState(configuration: AppConfiguration): State = {
val commandDefinitions = fetch +: BasicCommands.allBasicCommands
val commandsToRun = "iflast shell" +: configuration.arguments.map(_.trim)
State(
configuration,
commandDefinitions,
Set.empty,
None,
commandsToRun,
State.newHistory,
AttributeMap(AttributeEntry(timeout, 1)),
globalLogging,
State.Continue
)
}Please notice how we included our fetch command into the initial state (fetch +: BasicCommands.allBasicCommands) and specified the default timeout value of 1 second (AttributeEntry(timeout, 1)).
The last topic which needs some clarification is how to launch our interactive console application? For this purposes sbt provides a launcher. In its minimal form, it is just a single file
sbt-launch.jar that should be downloaded and used to launch the applications by resolving them through Apache Ivy dependency management. Each application is responsible for supplying its launcher configuration which for our simple example may look like this one (stored in console.boot.properties file):[app]
org: com.javacodegeeks
name: console-interactive
version: 0.0.1-SNAPSHOT
class: com.javacodegeeks.console.ConsoleApp
components: xsbti
cross-versioned: binary
[scala]
version: 2.11.8
[boot]
directory: ${sbt.boot.directory-${sbt.global.base-${user.home}/.sbt}/boot/}
[log]
level: info
[repositories]
local
maven-central
typesafe-ivy-releases: http://repo.typesafe.com/typesafe/ivy-releases
typesafe-releases: http://repo.typesafe.com/typesafe/releases
Nothing prevents us from running the sample application anymore so let us do that by publishing it to the local Apache Ivy repository first:
$ sbt publishLocal
And running via sbt launcher right after:
$ java -Dsbt.boot.properties=console.boot.properties -jar sbt-launch.jar
Getting com.javacodegeeks console-interactive_2.11 0.0.1-SNAPSHOT ...
:: retrieving :: org.scala-sbt#boot-app
confs: [default]
22 artifacts copied, 0 already retrieved (8605kB/333ms)
>Awesome, we are now in the interactive shell of our application! Let us type help fetch to make sure our own command is there.
> help fetch
fetch <url>
Fetches the <url> and prints out the response
>How about fetching the data from some real URL addresses?
> fetch http://freegeoip.net/json/www.google.com
[info] {"ip":"216.58.219.196","country_code":"US","country_name":"United States","region_code":"CA","region_name":"California","city":"Mountain View","zip_code":"94043","time_zone":"America/Los_Angeles","latitude":37.4192,"longitude":-122.0574,"metro_code":807}
>
It works perfectly fine! But what if we made some typo and our URL address is not valid? Would our fetch command figure this one out? Let us see …
> fetch htp://freegeoip.net/json/www.google.com [error] URL is not valid [error] fetch htp://freegeoip.net/json/www.google.com [error] ^ >
As expected, it did and promptly reported about the error. Nice but could we execute the fetch command without the need to run the interactive shell? The answer is “Yes, sure!”, we just need to pass the command to be executed as an argument to sbt launcher, wrapped in double quotes, for example:
$ java -Dsbt.boot.properties=console.boot.properties -jar sbt-launch.jar "fetch http://freegeoip.net/json/www.google.com"
[info] {"ip":"172.217.4.68","country_code":"US","country_name":"United States","region_code":"CA","region_name":"California","city":"Mountain View","zip_code":"94043","time_zone":"America/Los_Angeles","latitude":37.4192,"longitude":-122.0574,"metro_code":807}The results printed out in the console are exactly the same. You may notice that we did not implement optional support for writing the output into the file however for console-based applications we can use a simple trick with stream redirection:
$ java -Dsbt.boot.properties=console.boot.properties -jar sbt-launch.jar "fetch http://freegeoip.net/json/www.google.com" > response.json
Although fully functional, our simple interactive application uses just a tiny piece of sbt power. Please feel free to explore the documentation section of this great tool related to creation of the command line applications.
5. Conclusions
In this section of the tutorial we have talked about building console applications using terrific Scala programming language and libraries. The value and usefulness of the plain old command line applications is certainly much underestimated and hopefully this section proves the point. Either you are developing a simple command-line driven tool or interactive one, Scala ecosystem is here to offer the full support.
6. What’s next
In the next section we are going to talk a lot about concurrency and parallelism, more precisely discussing the ideas and concepts behind Actor model and continuing our acquaintance with other parts of the awesome Akka toolkit.
The complete projects are available for download.
