How to copy a file in small portions into a target file?
Let’s talk about the why?
When you copy large files, you want to ensure that you are copying it piece by piece to avoid out-of-memory errors. As a good practice, it’s best to consider optimizations during development to avoid edge cases that might creep up in production.
Imagine a scenario where your copy file has been working flawlessly until one fine day when a 5GB file is thrown at it to be copied from one location to another. If memory is a problem, then it will throw a out-of-memory and crash.
Now the complete example
package main
import (
"bufio"
"io"
"os"
)
func main() {
// Objective: Copy contents of FileA to FileB
// Perform this step in chuncks instead of using the
// built-in copy functionality.
// Why?
// Illustrates how to use bufio library with a reader and writer
// Source
inFile, err := os.Open("./source-file.txt")
if err != nil {
panic(err)
}
defer inFile.Close()
// Target
outFile, err := os.Create("./target-file.txt")
if err != nil {
panic(err)
}
defer outFile.Close()
// Read a bit then write a bit -- repeat until EOF
reader := bufio.NewReader(inFile)
writer := bufio.NewWriter(outFile)
for {
if slice, err := reader.ReadSlice('\n'); err == nil || err == io.EOF {
writer.Write(slice)
// IMPORTANT: Invoke flush()
writer.Flush()
if err == io.EOF {
break
}
}
}
}Deeper analysis
- In lines 17-20, we Open() the source and panic if we have a file error.
- In line 21, we defer closing the file when we exit that function. You got to love Go for this one capability alone! :-)
- In lines 24-28, we repeat the same process for the outgoing file.
- In lines 31,32, we create the NewReader and NewWriter.
- In lines 33-42, we create loop until we reach the End-Of-File marker and in line 35 we write the output out to the file.
- In lines 35, It is important to always call writer.Flush() to commit the writes to file. Failure to do this will end up with an empty file.
Ready to try this on your local system?
Feel free to check out the project located here:
All source code and steps to compile have been included at the repository above. Ping me if you run into any issues.
Cheers!
Published on Java Code Geeks with permission by Venkatt Guhesan, partner at our JCG program. See the original article here: GoLang: Reader-Writer implemented via bufio library to copy file Opinions expressed by Java Code Geeks contributors are their own. |
