MergeContent with nifi - inconsistent length

The most important thing here is actually the minimum number of entries. What is happening is that the binning algorithm takes a lenient approach in terms of the number of items.

For your specific logic, you would want to let things as they stand and:

  • Set Minimum Number of Entries to 5000
  • Optionally, increase the maximum number of entries. Leaving it as configured will generate bins that are exactly 5000 entries except for those periods where the age interval has been eclipsed

Below is an image of the configuration above where min and max bin size are both 5000 and only 1 bin is handled at a time. In this case you'll see that exactly 20000 files have been merged into 4.

Sample execution for a min and max bin size of 5000

Tags:

Apache Nifi