Debugging Scrapy Project in Visual Studio Code

In order to execute the typical scrapy runspider <PYTHON_FILE> command you must to set the following config into your launch.json:

{
    "version": "0.1.0",
    "configurations": [
        {
            "name": "Python: Launch Scrapy Spider",
            "type": "python",
            "request": "launch",
            "module": "scrapy",
            "args": [
                "runspider",
                "${file}"
            ],
            "console": "integratedTerminal"
        }
    ]
}

Set the breakpoints wherever you want and then debug.


  1. Inside your scrapy project folder create a runner.py module with the following:

    import os
    from scrapy.cmdline import execute
    
    os.chdir(os.path.dirname(os.path.realpath(__file__)))
    
    try:
        execute(
            [
                'scrapy',
                'crawl',
                'SPIDER NAME',
                '-o',
                'out.json',
            ]
        )
    except SystemExit:
        pass
    
  2. Place a breakpoint in the line you wish to debug

  3. Run runner.py with vscode debugger