Best way to read Spring Multipartfile line by line in Java 8

What is the best handle a csv Spring Multipartfile?
I have used something like this before:

public void handleFile(MultipartFile multipartFile){
    try{
        InputStream inputStream = multipartFile.getInputStream();
        IOUtils.readLines(inputStream, StandardCharsets.UTF_8)
                .stream()
                .forEach(this::handleLine);
    } catch (IOException e) {
        // handle exception
    }
}

private void handleLine(String s) {
    // do stuff per line
}

As far as I know, this first loads the whole file into a list in memory before processing it, which will probably take quite some time for files with tens of thousends of lines.

Is there a way to handle it line by line without the overhead of implementing the iteration by hand (i.e. using stuff like read(), hasNext(), ...)?
I am looking for something concise similar to this example for files from the file system:

try (Stream<String> stream = Files.lines(Paths.get("file.csv"))) {
        stream.forEach(this::handleLine);
} catch (IOException e) {
    // handle exception
}

In cases when you have InputStream you can use this one:

InputStream inputStream = multipartFile.getInputStream();
new BufferedReader(new InputStreamReader(inputStream, StandardCharsets.UTF_8))
                    .lines()
                    .forEach(this::handleLine);

In other cases:

No matter whether it is multipart file or you have multiple independent files, there are many approaches to do it in Java 8 using Stream API:

Solution 1:

If your files are in different directories you can do it this way:

Imagine you have a List of String which contains paths of your files like below:

List<String> files = Arrays.asList(
                "/test/test.txt",
                "/test2/test2.txt");

Then you can read all lines of above files as below:

files.stream().map(Paths::get)
        .flatMap(path -> {
            try {
                return Files.lines(path);
            } catch (IOException e) {
                e.printStackTrace();
            }
            return Stream.empty();
        }).forEach(System.out::println);

Solution 2:

You can also read all lines of files that exist in /test/ehsan directory using Files.walk in the following way:

try (Stream<Path> stream = Files.walk(Paths.get("/test/ehsan"), 1)) {
    stream.filter(Files::isRegularFile)
            .flatMap(path -> {
                try {
                    return Files.lines(path);
                } catch (IOException e) {
                    e.printStackTrace();
                }
                return Stream.empty();
            })
            .forEach(System.out::println);
} catch (IOException e) {
    e.printStackTrace();
}

And if you want to read all lines of files in /test/ehsan directory recursively you can do it this way:

try (Stream<Path> stream = Files.walk(Paths.get("/test/ehsan"))) {
    stream.filter(Files::isRegularFile)
            .flatMap(path -> {
                try {
                    return Files.lines(path);
                } catch (IOException e) {
                    e.printStackTrace();
                }
                return Stream.empty();
            })
            .forEach(System.out::println);
} catch (IOException e) {
    e.printStackTrace();
}

As you can see the second parameter to Files.walk specifies the maximum number of directory levels to visit and if you don't pass it the default will be used which is Integer.MAX_VALUE.

Solution 3:

Lets not stop here, we can go further. what if we wanted to read all lines of files exist in two completely different directories for example /test/ehsan and /test2/ehsan1?

We can do it but we should be cautious, Stream should not be so long( because it reduces readability of our program) it will be better to break them in separate methods, However because it is not possible to write multiple methods here I will write in one place how to do that:

Imagine you have a List of String which contains paths of your directories like below

list<String> dirs = Arrays.asList(
                "/test/ehsan",
                "/test2/ehsan1");

Then we can do that this way:

dirs.stream()
        .map(Paths::get)
        .flatMap(path -> {
            try {
                return Files.walk(path);
            } catch (IOException e) {
                e.printStackTrace();
            }
            return Stream.empty();
        })
        .filter(Files::isRegularFile)
        .flatMap(path -> { 
            try {
                return Files.lines(path);
            } catch (IOException e) {
                e.printStackTrace();
            }
            return Stream.empty();
        })
        .forEach(System.out::println);

public static List<String> readCSV(String fileName) throws IOException {
        List<String> records = new ArrayList<>();
        try (BufferedReader br = new BufferedReader(
                new FileReader(fileName))) {
            String line;
            while ((line = br.readLine()) != null) {
                records.add(line);
            }
        }
        return records;
}