Xuggler Tutorial: Transcoding and Media Modification

In my previous tutorial, I performed a short Introduction to Xuggler for Video Manipulation. In this part, we are going to see some more exciting capabilities provided by Xuggler and FFmpeg, like video transcoding and media modification. Don’t forget that Xuggler is a Java library that can be used to uncompress, manipulate, and compress recorded or live video in real time.

Xuggler offers two distinct programming APIs which can be used for the same purpose. First, we have the MediaTool API:

MediaTool is a simple Application Programming Interface (API) for decoding, encoding and modifying video in Java. MediaTool hides many of the nitty-gritty details of containers, codecs, and others so you can focus on the media, and not the tools. That said, MediaTool still provides access to the underlying Xuggler objects, so you have fine grain control if you need it.

And then there is the Xuggler Advanced API, which allows you to delve into the details of video manipulation, but adds a layer of complexity.

To begin, we will use the MediaTool API and in subsequent tutorials we will also deal with the Advanced API too.

Let’s start by transcoding media from one format to another. Transcoding is the direct digital-to-digital conversion of one encoding to another. This is usually done in cases where a target device does not support the format or has limited storage capacity that mandates a reduced file size, or to convert incompatible or obsolete data to a better-supported or modern format. Transcoding is commonly a lossy process, where “lossy” compression is a data encoding method which discards (loses) some of the data, in order to achieve its goal, with the result that decompressing the data yields content that is different from the original, though similar enough to be useful in some way.

Let’s see some high level code for transcoding and I will explain the details later.

package com.javacodegeeks.xuggler;

import com.xuggle.mediatool.IMediaReader;
import com.xuggle.mediatool.IMediaViewer;
import com.xuggle.mediatool.IMediaWriter;
import com.xuggle.mediatool.ToolFactory;

public class TranscodingExample {

    private static final String inputFilename = "c:/myvideo.mp4";
    private static final String outputFilename = "c:/myvideo.flv";

    public static void main(String[] args) {

        // create a media reader
        IMediaReader mediaReader = 
               ToolFactory.makeReader(inputFilename);
        
        // create a media writer
        IMediaWriter mediaWriter = 
               ToolFactory.makeWriter(outputFilename, mediaReader);

        // add a writer to the reader, to create the output file
        mediaReader.addListener(mediaWriter);
        
        // create a media viewer with stats enabled
        IMediaViewer mediaViewer = ToolFactory.makeViewer(true);
        
        // add a viewer to the reader, to see the decoded media
        mediaReader.addListener(mediaViewer);

        // read and decode packets from the source file and
        // and dispatch decoded audio and video to the writer
        while (mediaReader.readPacket() == null) ;

    }

}

With a few lines of code, we are able to convert an MPEG-4 input file to a FLV one. We start by creating an IMediaReader which is used to read and decode media. It opens up a media container, reads packets from it, decodes the data, and then dispatches information about the data to any registered IMediaListener objects. This is where the IMediaWriter class comes into play. It encodes and decodes media, handling both audio and video streams. To make things more interesting, we also attach an IMediaViewer to our reader. This is used as a debuging tool, allowing us to view the video while it is being decoded. Additionally, we are presented with various stats during the process. Note that this class is in experimental mode, meaning it has some bugs where it can hang, so handle with care.

Essentially, with the above code, we attach our two listeners, the IMediaWriter and the IMediaViewer, to our IMediaReader object and we handle the callbacks while the reader reads and decodes packets from the source file. This is performed in the “while” loop. If we run the application with a sample input file, we will be presented with a screen similar to this:

After the process is completed (it will last as long as the source video file since we are simultaneously viewing it in real time), a new output file will have been created in FLV format.

Let’s use Ffmpeg from command line in order to compare the input and output files:

ffmpeg.exe -i c:/myvideo.mp4
Seems stream 1 codec frame rate differs from container frame rate: 59.92 (14981/250) -> 29.96 (14981/500)
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from ‘c:/myvideo.mp4’:
Metadata:
major_brand : mp42
minor_version : 0
compatible_brands: isomavc1mp42
Duration: 00:04:20.96, start: 0.000000, bitrate: 582 kb/s
Stream #0.0(und): Audio: aac, 44100 Hz, stereo, s16, 115 kb/s
Stream #0.1(und): Video: h264, yuv420p, 480×270 [PAR 1:1 DAR 16:9], 464 kb/s, 29.96 fps, 29.96 tbr, 29962 tbn, 59.92 tbc

In the original video file, the container is MPEG-4 and there are two streams: an audio stream using AAC at 44100Hz and a video stream using H.264.

ffmpeg.exe -i c:/myvideo.flv
Seems stream 0 codec frame rate differs from container frame rate: 1000.00 (1000/1) -> 29.97 (30000/1001)
Input #0, flv, from ‘c:/myvideo.flv’:
Metadata:
duration : 261
width : 480
height : 270
videodatarate : 62
framerate : 30
videocodecid : 2
audiodatarate : 62
audiosamplerate : 44100
audiosamplesize : 16
stereo : true
audiocodecid : 2
filesize : 43117478
Duration: 00:04:20.98, start: 0.000000, bitrate: 128 kb/s
Stream #0.0: Video: flv, yuv420p, 480×270, 64 kb/s, 29.97 tbr, 1k tbn, 1k tbc
Stream #0.1: Audio: mp3, 44100 Hz, 2 channels, s16, 64 kb/s

After the transcoding, the generated Flash video file uses an FLV video stream and an MP3 audio stream.

We are now ready to modify a media file using Xuggler. But before we write code, we need to understand how MediaTool works:

MediaTool uses an event listener paradigm. The writer is automatically added as a “listener” to the reader and receives all decoded media. The IMediaViewer and IMediaWriter interfaces (what the viewer and writer actually are) implement the IMediaListener interface, and may be added as listeners to an IMediaReader.

We confirmed that with our previous example. The thing is that in order to perform various modifications to an input file, we have to set up a “media pipeline”. We create custom implementations of IMediaTool and then set up listeners on each tool in a chain way so they pass data from one to the other.

Let’s suppose we wish to add a static image to our video and at the same time we wish to reduce the audio volume. In that case, we create two custom IMediaTool objects:

• StaticImageMediaTool: Takes a video picture and stamps a static image file on a specific location on the screen.
• VolumeAdjustMediaTool: Adjusts volume by a constant factor.

Additionally, we create an IMediaWriter object which will be used for the output file creation. With all those, we create a chain that goes as follows:

reader -> addStaticImage -> reduceVolume -> writer

Let’s see the code that implements all the above:

package com.javacodegeeks.xuggler;

import java.awt.Color;
import java.awt.Graphics2D;
import java.awt.geom.Rectangle2D;
import java.awt.image.BufferedImage;
import java.io.File;
import java.io.IOException;
import java.nio.ShortBuffer;

import javax.imageio.ImageIO;

import com.xuggle.mediatool.IMediaReader;
import com.xuggle.mediatool.IMediaTool;
import com.xuggle.mediatool.IMediaWriter;
import com.xuggle.mediatool.MediaToolAdapter;
import com.xuggle.mediatool.ToolFactory;
import com.xuggle.mediatool.event.IAudioSamplesEvent;
import com.xuggle.mediatool.event.IVideoPictureEvent;

public class ModifyMediaExample {

    private static final String inputFilename = "c:/myvideo.mp4";
    private static final String outputFilename = "c:/myvideo.flv";
    private static final String imageFilename = "c:/jcg_logo_small.png";

    public static void main(String[] args) {

        // create a media reader
        IMediaReader mediaReader = ToolFactory.makeReader(inputFilename);
        
        // configure it to generate BufferImages
        mediaReader.setBufferedImageTypeToGenerate(BufferedImage.TYPE_3BYTE_BGR);

        IMediaWriter mediaWriter = 
               ToolFactory.makeWriter(outputFilename, mediaReader);
        
        IMediaTool imageMediaTool = new StaticImageMediaTool(imageFilename);
        IMediaTool audioVolumeMediaTool = new VolumeAdjustMediaTool(0.1);
        
        // create a tool chain:
        // reader -> addStaticImage -> reduceVolume -> writer
        mediaReader.addListener(imageMediaTool);
        imageMediaTool.addListener(audioVolumeMediaTool);
        audioVolumeMediaTool.addListener(mediaWriter);
        
        while (mediaReader.readPacket() == null) ;

    }
    
    private static class StaticImageMediaTool extends MediaToolAdapter {
        
        private BufferedImage logoImage;
        
        public StaticImageMediaTool(String imageFile) {
            
            try {
                logoImage = ImageIO.read(new File(imageFile));
            } 
            catch (IOException e) {
                e.printStackTrace();
                throw new RuntimeException("Could not open file");
            }
            
        }

        @Override
        public void onVideoPicture(IVideoPictureEvent event) {
            
            BufferedImage image = event.getImage();
            
            // get the graphics for the image
            Graphics2D g = image.createGraphics();
            
            Rectangle2D bounds = new 
              Rectangle2D.Float(0, 0, logoImage.getWidth(), logoImage.getHeight());

            // compute the amount to inset the time stamp and 
            // translate the image to that position
            double inset = bounds.getHeight();
            g.translate(inset, event.getImage().getHeight() - inset);

            g.setColor(Color.WHITE);
            g.fill(bounds);
            g.setColor(Color.BLACK);
            g.drawImage(logoImage, 0, 0, null);
            
            // call parent which will pass the video onto next tool in chain
            super.onVideoPicture(event);
            
        }
        
    }

    private static class VolumeAdjustMediaTool extends MediaToolAdapter {
        
        // the amount to adjust the volume by
        private double mVolume;
        
        public VolumeAdjustMediaTool(double volume) {
            mVolume = volume;
        }

        @Override
        public void onAudioSamples(IAudioSamplesEvent event) {
            
            // get the raw audio bytes and adjust it's value
            ShortBuffer buffer = 
               event.getAudioSamples().getByteBuffer().asShortBuffer();
            
            for (int i = 0; i < buffer.limit(); ++i) {
                buffer.put(i, (short) (buffer.get(i) * mVolume));
            }

            // call parent which will pass the audio onto next tool in chain
            super.onAudioSamples(event);
            
        }
        
    }

}

As always, we first create an IMediaReader and use the setBufferedImageTypeToGenerate method to to generate BufferedImage images when calling IMediaListener.onVideoPicture. This is necessary in order to overlay our custom image on top of the actual video pictures. We then create the IMediaWriter and the media tool objects and configure the tool chain as described above. Let’s take a closer look to the custom media tools.

First, we have the StaticImageMediaTool class. It extends the MediaToolAdapter and overrides the onVideoPicture method since we wish to manipulate the video stream with this one. In the constructor, we have loaded an image file using the ImageIO.read method. The JavaCodeGeeks logo is used for that purpose (actually, a scaled down version of it). Then, in the implemented onVideoPicture method, we get the underlying BufferedImage by calling IVideoPictureEvent.getImage and create a Graphics2D object. We then use the Graphics.drawImage method to overlay the static image. Finally, we call the parent onVideoPicture method which will pass the video onto next tool in chain.

Then, we have the VolumeAdjustMediaTool. It also extends the MediaToolAdapter, but overrides the onAudioSamples method, which is called after audio samples have been decoded or encoded. There, we get the raw audio bytes by calling IAudioSamplesEvent.getAudioSamples and adjust it’s value by using the corresponding ShortBuffer class. Again, after our custom processing, we call the parent onAudioSamples method which will pass the audio onto next tool in chain. If we now run this application, we will see the added image on top of the original video and the audio volume will have been considerably reduced.
That’s it. Transcoding and media manipulation powered by Xuggler. As always, you can download the Eclipse project created for this tutorial. Stay tuned for more Xuggler tutorials here at JavaCodeGeeks! And don’t forget to share!

Related Articles:

• Introduction to Xuggler for Video Manipulation
• Xuggler Tutorial: Frames Capture and Video Creation
• RTMP To RTSP re-stream using wowza and xuggler