MongoDB pipeline for Scrapy. This library supports both MongoDB in standalone setups and replica sets. It will insert items to MongoDB as soon as your spider finds data to extract. scrapy-mongodb can also buffer objects if you prefer to write chunks of data to MongoDB rather than one write per document (see MONGODB_BUFFER_DATA option for details).

Read more here.

Install via pip:

pip install -r requirements.txt
pip install scrapy-mongodb

Add these options to settings.py file:

ITEM_PIPELINES = {
    ...
    'scrapy_mongodb.MongoDBPipeline': 300,
    ...
}

MONGODB_URI = 'mongodb://localhost:27017'
MONGODB_DATABASE = 'scrapy'
MONGODB_COLLECTION = 'my_items'

If you want a unique key in your database, name the key with this option:

MONGODB_UNIQUE_KEY = 'url'

You can configure scrapy-mongodb to support MongoDB replica sets by adding MONGODB_REPLICA_SET option and specify additional replica set hosts in MONGODB_URI:

MONGODB_REPLICA_SET = 'myReplicaSetName'
MONGODB_URI = 'mongodb://host1.example.com:27017,host2.example.com:27017,host3.example.com:27017'

If you need to ensure that your data has been replicated, use the MONGODB_REPLICA_SET_W option. It is an implementation of the w parameter in pymongo. Details from the pymongo documentation:

Write operations will block until they have been replicated to the specified number or tagged set of servers. w=<int> always includes the replica set primary (e.g. w=3 means write to the primary and wait until replicated to two secondaries). Passing w=0 disables write acknowledgement and all other write concern options.

To ease the load on MongoDB, scrapy-mongodb has a buffering feature. You can enable it by setting MONGODB_BUFFER_DATA to the buffer size you want. E.g: scrapy-mongodb will write 10 documents at a time to the database if you set:

MONGODB_BUFFER_DATA = 10

It is not possible to combine this feature with MONGODB_UNIQUE_KEY. Technically due to that the update method in pymongo doesn't support multi-doc updates.

scrapy-mongodb can append a timestamp to your item when inserting it to the database. Enable this feature with:

MONGODB_ADD_TIMESTAMP = True

This will modify the document to something like this:

{
    ...
    'scrapy-mongodb': {
        'ts': ISODate("2013-01-10T07:43:56.797Z")
    }
    ...
}

The timestamp is in UTC.

It's possible to write data to 1 collection per spider. To enable that feature, set this environment variable:

MONGODB_SEPARATE_COLLECTIONS = True

Since scrapy-mongodb 0.5.0

make release

Read more here.

This project is maintained by: Sebastian Dahlgren (GitHub | Twitter | LinkedIn).

Read more here.