Java 8 Stream to determine a maximum count in a text file

Question

For my assignment I have to replace for loops with streams that count the frequency of words in a text document, and I am having trouble figuring the TODO part out.

String filename = "SophieSallyJack.txt";
if (args.length == 1) {
    filename = args[0];
}
Map<String, Integer> wordFrequency = new TreeMap<>();

List<String> incoming = Utilities.readAFile(filename);

wordFrequency = incoming.stream()
    .map(String::toLowerCase)
    .filter(word -> !word.trim().isEmpty())
    .collect(Collectors.toMap(word -> word, word -> 1, (a, b) -> a + b, TreeMap::new));                

int maxCnt = 0;

// TODO add a single statement that uses streams to determine maxCnt
for (String word : incoming) {
    Integer cnt = wordFrequency.get(word);
    if (cnt != null) {
        if (cnt > maxCnt) {
            maxCnt = cnt;
        }
    }
}
System.out.print("Words that appear " + maxCnt + " times:");

I have tried this:

wordFrequency = incoming.parallelStream().
    collect(Collectors.toConcurrentMap(w -> w, w -> 1, Integer::sum));

But that is not right and I'm not sure how to incorporate maxCnt into the stream.

Answer 1

Assuming you have all the words extracted from a file in a List<String> this word count for each word can be computed using this approach,

Map<String, Long> wordToCountMap = words.stream()
                .collect(Collectors.groupingBy(Function.identity(), Collectors.counting()));

The most freequent word can then be computed using the above map like so,

Entry<String, Long> mostFreequentWord = wordToCountMap.entrySet().stream()
    .max(Map.Entry.comparingByValue())
    .orElse(new AbstractMap.SimpleEntry<>("Invalid", 0l));

You may change the above two pipelines together if you wish like this,

Entry<String, Long> mostFreequentWord = words.stream()
    .collect(Collectors.groupingBy(Function.identity(), Collectors.counting()))
    .entrySet().stream()
    .max(Map.Entry.comparingByValue())
    .orElse(new AbstractMap.SimpleEntry<>("Invalid", 0l));

Update

As per the following discussion it is always good to return an Optional from your computation like so,

Optional<Entry<String, Long>> mostFreequentWord = words.stream()
    .collect(Collectors.groupingBy(Function.identity(), Collectors.counting()))
    .entrySet().stream()
    .max(Map.Entry.comparingByValue());

Answer 2

Ok, first of all, your wordFrequency line can make use of Collectors#groupingBy and Collectors#counting instead of writing your own accumulator:

    List<String> incoming = Arrays.asList("monkey", "dog", "MONKEY", "DOG", "giraffe", "giraffe", "giraffe", "Monkey");
    wordFrequency = incoming.stream()
            .filter(word -> !word.trim().isEmpty()) // filter first, so we don't lowercase empty strings
            .map(String::toLowerCase)
            .collect(Collectors.groupingBy(s -> s, Collectors.counting()));

Now that we got that out of the way... Your TODO line says use streams to determine maxCnt. You can do that easily by using max with naturalOrder:

    int maxCnt = wordFrequency.values()
            .stream()
            .max(Comparator.naturalOrder())
            .orElse(0L)
            .intValue();

However, your comments make me think that what you actually want is a one-liner to print the most frequent words (all of them), i.e. the words that have maxCnt as value in wordFrequency. So what we need is to "reverse" the map, grouping the words by count, and then pick the entry with highest count:

    wordFrequency.entrySet().stream() // {monkey=3, dog=2, giraffe=3}
            .collect(groupingBy(Map.Entry::getValue, mapping(Map.Entry::getKey, toList()))).entrySet().stream() // reverse map: {3=[monkey, giraffe], 2=[dog]}
            .max(Comparator.comparingLong(Map.Entry::getKey)) // maxCnt and all words with it: 3=[monkey, giraffe]
            .ifPresent(e -> {
                System.out.println("Words that appear " + e.getKey() + " times: " + e.getValue());
            });

This solution prints all the words with maxCnt, instead of just one:

Words that appear 3 times: [monkey, giraffe].

Of course, you can concatenate the statements to get one big do-it-all statement, like this:

    incoming.stream() // [monkey, dog, MONKEY, DOG, giraffe, giraffe, giraffe, Monkey]
            .filter(word -> !word.trim().isEmpty()) // filter first, so we don't lowercase empty strings
            .map(String::toLowerCase)
            .collect(groupingBy(s -> s, counting())).entrySet().stream() // {monkey=3, dog=2, giraffe=3}
            .collect(groupingBy(Map.Entry::getValue, mapping(Map.Entry::getKey, toList()))).entrySet().stream() // reverse map: {3=[monkey, giraffe], 2=[dog]}
            .max(Comparator.comparingLong(Map.Entry::getKey)) // maxCnt and all words with it: 3=[monkey, giraffe]
            .ifPresent(e -> {
                System.out.println("Words that appear " + e.getKey() + " times: " + e.getValue());
            });

But now we're stretching the meaning of "one statement" :)

Answer 3

Well, you have done almost everything you needed with that TreeMap, but it seems you don't know that it has a method called lastEntry and that is the only one you need to call after you computed wordFrequency to get the word with the highest frequency.

The only problem is that this is not very optimal, since TreeMap sorts the data on each insert and you don't need sorted data, you need the max. Sorting in case of TreeMap is O(nlogn), while inserting into a HashMap is O(n).

So instead of using that TreeMap, all you need to change is to a HashMap:

wordFrequency = incoming.stream()
    .map(String::toLowerCase)
    .filter(word -> !word.trim().isEmpty())
    .collect(Collectors.toMap(
             Function.identity(), 
             word -> 1, 
             (a, b) -> a + b, 
             HashMap::new)); 

Once you have this Map, you need to find max - this operation is O(n) in general and could be achieved with a stream or without one:

 Collections.max(wordFrequency.entrySet(), Map.Entry.comparingByValue())

This approach with give you O(n) for HashMap insert, and O(n) for finding the max - thus O(n) in general, so it's faster than TreeMap

Answer 4

By piecing together information I was able to successfully replace the for loop with

    int maxCnt = wordFrequency.values().stream().max(Comparator.naturalOrder()).get();
    System.out.print("Words that appear " + maxCnt + " times:");

I appreciate all the help.