Improving performance when using jq to process large files

I think the answer by thanasisp is great and covers a lot of the problems. The use of cut is very clever to address the sed memory problems.

the sed commands you have tried are fast, but need memory, because sed, the stream editor, is reading line by line, and if the file has no newlines at all, it will load all of it into memory, sed needs 2-3 times the size of the maximum line of the stream

However, as noted in the answer, that sed script only works for very simple JSON objects (no nested objects and no } anywhere except to denote end of first level objects.

A more advanced sed script

This can improved with a more sophisticated sed script which in addition to the patterns posted handles.

[{"id": 1, "name": "foo"}
,{"id": 2, "name": "bar"}
,{"id": 3, "name": "baz"}
,{"id":4, "name": 10}
,{"id":5, "name":"\\\" },{"}
,{"id": {"a":6}, "name": 10}]

We achieve this mainly by taking advantage of the fact that {/} and " come in pairs.

json-newline-json.sed

#!/bin/sed -nf

# Skip empty lines
/^$/d

# From first line to first line starting with [
0 , /\[/ {
# Replace opening [ if exists
# + stripping leading whitespace
    /^[[:space:]]*\[/  s@^[[:space:]]*\[@,@
}

# Line starts with comma
/^\,/ {
# Strip it  
}

# Start of loop
: x

# Save to hold
h

# delete all chars except " and \
s@[^"\\]@@g
# Delete all reverse solidi and non-escaped " chars
s@\(\\"\|\\\)@@g

# Even match
/^\(""\)\+$/ {
    # Fetch hold
    g
    # Delete everything between ".."
    s@"[^"]*"@@g
    # Delete all chars except {}
    s@[^{}]@@g
    # Match even {} pairs
    /^\([{}][{}]\)\+$/ {
        # The hold space contains our assembled ,{..} object
        g
        # Strip the leading comma
        s@^\,@@
        # Print
        p
        # Skip to next line
        d
    }
    # The hold space contains our partial ,{.. object
    g
    N
    s@\n@@
    t x
}

# Odd match
/^\(""\)*"$/ {
    # The hold space contains our partial ,{.." object
    g
    # Fetch the next line to append
    N
    # Delete the newline added by N (append without newline)
    s@\n@@
    # Restart loop x
    t x
}

Combining with cut we do

< huge.json cut -d '}' -f1- --output-delimeter="}"$'\n' |\
 json-newline-json.sed |\
 split ...

NB

If you have relatively well defined (and non weird) JSON input, solving your problem with sed is powerful, portable and fast. However, it's precarious. The downside is that neither the input or output is validated. Parsing serialized formats is best done with more care. So for most intents and purposes I'd probably stick with jq.


Restrictions

In the general case, JSON needs parsing with a tool that can understand JSON. You could make an exception and follow these suggestions, only if you are sure that:

  • You have an array with flat JSON objects (like in the use case) without nested objects.

  • Curly braces do not exist anywhere inside the objects, that means you don't have any content like this: {id:1, name:"foo{bar}"}.

Use the shell

If the above conditions are met, you can use the shell to convert to JSONL and split to smaller files, and it would be many times faster than JSON parsing or full text-processing. Additonally it can be almost memoryless, especially if you use core-utils with or without some sed or awk.

Even the simpler approach:

grep -o '{[^}]*}' file.json

will be faster, but will need some memory (less than jq).

And the sed commands you have tried are fast, but need memory, because sed, the stream editor, is reading line by line, and if the file has no newlines at all, it will load all of it into memory, sed needs 2-3 times the size of the maximum line of the stream. But if you first split the stream with newlines, using core-utils like tr, cut etc, then memory usage is extremely low, with great perfomance.

Solution

After some testing, I found this one to be faster and memoryless. Besides that, it doesn't depend on the extra characters outside the objects, like comma and a few spaces, or comma alone etc. It will only match the objects {...} and print each of them to a new line.

#!/bin/sh -
LC_ALL=C < "$1" cut -d '}' -f1- --output-delimiter="}"$'\n' |\
    cut -sd '{' -f2 | sed 's/^/{/' > "$2"

to split the JSONL, use -l rather than -c, to ensure you don't split any object, use something like this:

split -l 1000 -d --additional-suffix='.json' - path/to/file/prefix

or all together

#!/bin/sh -
n=1000
LC_ALL=C < "$1" cut -d '}' -f1- --output-delimiter="}"$'\n' |\
    cut -sd '{' -f2 | sed 's/^/{/' |\
    split -l "$n" -d --additional-suffix='.json' - "$2"

Usage:

sh script.sh input.json path/to/new/files/output

will create files output1.json, output2.json etc in the selected path.

Note: If your stream contains non UTF-8 multi-bute characters, remove LC_ALL=C, it is just a small speed optimization which is not necessary.

Note: I have assumed input with no newlines at all, or with newlines like in your first use case. To generalize and include any newlines anywhere in the file, I add a small modification. In this version tr will truncate all newlines initially, with almost no impact to perfomance:

#!/bin/sh -
n=1000
LC_ALL=C < "$1" tr -d $'\n' |\
    cut -d '}' -f1- --output-delimiter="}"$'\n' |\
    cut -sd '{' -f2 | sed 's/^/{/' > "$2"

Testing

Here are some testing results. They are representative, times were similar for all executions.

Here is the script I used, with input for various values of n:

#!/bin/bash

make_json() {
    awk -v n=2000000 'BEGIN{
        x = "{\"id\": 1, \"name\": \"foo\"}"
        printf "["
        for (i=1;i<n;i++) { printf x ", " }
        printf x"]"
    }' > big.json
    return 0
}

tf="Real: %E  System: %S  User: %U  CPU%%: %P  Maximum Memory: %M KB\n"
make_json

for i in {1..7}; do
    printf "\n==> "
    cat "${i}.sh"
    command time -f "$tf" sh "${i}.sh" big.json "output${i}.json"
done

I used small files when testing together with jq because it gets early into swap. Then with larger files using only the efficient solutions.

==> LC_ALL=C jq -c '.[]' "$1" > "$2"
Real: 0:16.26  System: 1.46  User: 14.74  CPU%: 99%  Maximum Memory: 1004200 KB


==> LC_ALL=C jq length "$1" > /dev/null
Real: 0:09.19  System: 1.30  User: 7.85  CPU%: 99%  Maximum Memory: 1002912 KB


==> LC_ALL=C < "$1" sed 's/^\[//; s/}[^}]*{/}\n{/g; s/]$//' > "$2"
Real: 0:02.21  System: 0.33  User: 1.86  CPU%: 99%  Maximum Memory: 153180 KB


==> LC_ALL=C < "$1" grep -o '{[^}]*}' > "$2"
Real: 0:02.08  System: 0.34  User: 1.71  CPU%: 99%  Maximum Memory: 103064 KB


==> LC_ALL=C < "$1" awk -v RS="}, {" -v ORS="}\n{" '1' |\
    head -n -1 | sed '1 s/^\[//; $ s/]}$//' > "$2"
Real: 0:01.38  System: 0.32  User: 1.52  CPU%: 134%  Maximum Memory: 3468 KB


==> LC_ALL=C < "$1" cut -d "}" -f1- --output-delimiter="}"$'\n' |\
    sed '1 s/\[//; s/^, //; $d;' > "$2"
Real: 0:00.94  System: 0.24  User: 0.99  CPU%: 131%  Maximum Memory: 3488 KB


==> LC_ALL=C < "$1" cut -d '}' -f1- --output-delimiter="}"$'\n' |\
    cut -sd '{' -f2 | sed 's/^/{/' > "$2"
Real: 0:00.63  System: 0.28  User: 0.86  CPU%: 181%  Maximum Memory: 3448 KB

# Larger files testing

==> LC_ALL=C < "$1" grep -o '{[^}]*}' > "$2"
Real: 0:20.99  System: 2.98  User: 17.80  CPU%: 99%  Maximum Memory: 1017304 KB


==> LC_ALL=C < "$1" awk -v RS="}, {" -v ORS="}\n{" '1' |\
    head -n -1 | sed '1 s/^\[//; $ s/]}$//' > "$2"
Real: 0:16.44  System: 2.96  User: 15.88  CPU%: 114%  Maximum Memory: 3496 KB


==> LC_ALL=C < "$1" cut -d "}" -f1- --output-delimiter="}"$'\n' |\
    sed '1 s/\[//; s/^, //; $d;' > "$2"
Real: 0:09.34  System: 1.93  User: 10.27  CPU%: 130%  Maximum Memory: 3416 KB


==> LC_ALL=C < "$1" cut -d '}' -f1- --output-delimiter="}"$'\n' |\
    cut -sd '{' -f2 | sed 's/^/{/' > "$2"
Real: 0:07.22  System: 2.79  User: 8.74  CPU%: 159%  Maximum Memory: 3380 KB

jq's streaming parser (the one invoked with the --stream command-line option) intentionally sacrifices speed for the sake of reduced memory requirements, as illustrated below in the metrics section. A tool which strikes a different balance (one which seems to be closer to what you're looking for) is jstream, the homepage of which is https://github.com/bcicen/jstream

Running the sequence of commands in a bash or bash-like shell:

cd
go get github.com/bcicen/jstream
cd go/src/github.com/bcicen/jstream/cmd/jstream/
go build

will result in an executable, which you can invoke like so:

jstream -d 1 < INPUTFILE > STREAM

Assuming INPUTFILE contains a (possibly ginormous) JSON array, the above will behave like jq's .[], with jq's -c (compact) command-line option. In fact, this is also the case if INPUTFILE contains a stream of JSON arrays, or a stream of JSON non-scalars ...

Illustrative space-time metrics

Summary

For the task at hand (streaming the top-level items of an array):

                  mrss   u+s
jq --stream:      2 MB   447
jstream    :      8 MB   114
jq         :  5,582 MB    39

In words:

  1. space: jstream is economical with memory, but not as much as jq's streaming parser.

  2. time: jstream runs slightly slower than jq's regular parser but about 4 times faster than jq's streaming parser.

Interestingly, space*time is about the same for the two streaming parsers.

Characterization of the test file

The test file consists of an array of 10,000,000 simple objects:

[
{"key_one": 0.13888342355537053, "key_two": 0.4258700286271502, "key_three": 0.8010012924267487}
,{"key_one": 0.13888342355537053, "key_two": 0.4258700286271502, "key_three": 0.8010012924267487}
...
]

$ ls -l input.json
-rw-r--r--  1 xyzzy  staff  980000002 May  2  2019 input.json

$ wc -l input.json
 10000001 input.json

jq times and mrss

$ /usr/bin/time -l jq empty input.json
       43.91 real        37.36 user         4.74 sys
4981452800  maximum resident set size

$ /usr/bin/time -l jq length input.json
10000000
       48.78 real        41.78 user         4.41 sys
4730941440  maximum resident set size

/usr/bin/time -l jq type input.json
"array"
       37.69 real        34.26 user         3.05 sys
5582196736  maximum resident set size

/usr/bin/time -l jq 'def count(s): reduce s as $i (0;.+1); count(.[])' input.json
10000000
       39.40 real        35.95 user         3.01 sys
5582176256  maximum resident set size

/usr/bin/time -l jq -cn --stream 'fromstream(1|truncate_stream(inputs))' input.json | wc -l
      449.88 real       444.43 user         2.12 sys
   2023424  maximum resident set size
 10000000

jstream times and mrss

$ /usr/bin/time -l jstream -d 1 < input.json > /dev/null
       61.63 real        79.52 user        16.43 sys
   7999488  maximum resident set size

$ /usr/bin/time -l jstream -d 1 < input.json | wc -l
       77.65 real        93.69 user        20.85 sys
   7847936  maximum resident set size
 10000000

Tags:

Split

Json

Sed

Jq

Related

Javascript .replaceAll() is not a function type error Is Function Cloud in Firebase Free or Not (Cloud Functions deployment requires the pay-as-you-go (Blaze) billing plan) Can I access the static variables of the 'Class' Object? Error: Could not find browser revision 756035. Run "npm install" What is the most efficient way to fill missing values in this data frame? ssh_init: Network error: Cannot assign requested address ( Flutter and Mac ) Possible to create a new project to upgrade an existing app? IDFA iOS14 returns denied without displaying auth popup Are the "C mock tests" at tutorialspoint correct? Is it best practice to test my Web API controllers directly or through an HTTP client? Preventing rapid clicks on a button and making a request using rxjava Hilt dependency injection for UI testing says "HiltAndroidRule" wasn't added but it was

Recent Posts