How can I use Python to transform MongoDB's bsondump into JSON?

What you have is a dump in Mongo Extended JSON in TenGen mode (see here). Some possible ways to go:

1. If you can dump again, use Strict output mode through the MongoDB REST API. That should give you real JSON instead of what you have now.

2. Use bson from http://pypi.python.org/pypi/bson/ to read the BSON you already have into Python data structures and then do whatever processing you need on those (possibly outputting JSON).

3. Use the MongoDB Python bindings to connect to the database to get the data into Python, and then do whatever processing you need. (If needed, you could set up a local MongoDB instance and import your dumped files into that.)

4. Convert the Mongo Extended JSON from TenGen mode to Strict mode. You could develop a separate filter to do it (read from stdin, replace TenGen structures with Strict structures, and output the result on stdout) or you could do it as you process the input.

Here's an example using Python and regular expressions:

import json, re
from bson import json_util

with open("data.tengenjson", "rb") as f:
    # read the entire input; in a real application,
    # you would want to read a chunk at a time
    bsondata = f.read()

    # convert the TenGen JSON to Strict JSON
    # here, I just convert the ObjectId and Date structures,
    # but it's easy to extend to cover all structures listed at
    # http://www.mongodb.org/display/DOCS/Mongo+Extended+JSON
    jsondata = re.sub(r'ObjectId\s*\(\s*\"(\S+)\"\s*\)',
                      r'{"$oid": "\1"}',
                      bsondata)
    jsondata = re.sub(r'Date\s*\(\s*(\S+)\s*\)',
                      r'{"$date": \1}',
                      jsondata)

    # now we can parse this as JSON, and use MongoDB's object_hook
    # function to get rich Python data structures inside a dictionary
    data = json.loads(jsondata, object_hook=json_util.object_hook)

    # just print the output for demonstration, along with the type
    print(data)
    print(type(data))

    # serialise to JSON and print
    print(json_util.dumps(data))

Depending on your goal, one of these should be a reasonable starting point.

loading an entire bson document into python memory is expensive.

If you want to stream it in rather than loading the whole file and doing a load all, you can try this library.

https://github.com/bauman/python-bson-streaming

from bsonstream import KeyValueBSONInput
from sys import argv
for file in argv[1:]:
    f = open(file, 'rb')
    stream = KeyValueBSONInput(fh=f,  fast_string_prematch="somthing") #remove fast string match if not needed
    for id, dict_data in stream:
        if id:
         ...process dict_data...

You can convert lines of the bson file like this:

>>> import bson
>>> bs = open('file.bson', 'rb').read()
>>> for valid_dict in bson.decode_all( bs ):
....

Each valid_dict element will be a valid python dict that you can convert to json.