Write to a csv file scrapy

This is what worked for me using Python3:

scrapy runspider spidername.py -o file.csv -t csv

simply crawl with -o csv, like:

scrapy crawl <spider name> -o file.csv -t csv

You need to

1. Write your header row; then
2. Write the entry rows for each object.

You could approach it like:

fields = ["reference_link", "rss_link"] # define fields to use
with open(filename,'a+') as f: # handle the source file
    f.write("{}\n".format('\t'.join(str(field) 
                              for field in fields))) # write header 
    for item in items:
        f.write("{}\n".format('\t'.join(str(item[field]) 
                              for field in fields))) # write items

Note that "{}\n".format(s) gives the same result as "%s\n" % s.

Best approach to solve this problem is to use python in-build csv package.

import csv

file_name = open('Output_file.csv', 'w') #Output_file.csv is name of output file

fieldnames = ['reference_link', 'rss_link'] #adding header to file
writer = csv.DictWriter(file_name, fieldnames=fieldnames)
writer.writeheader()
for rss in rsslinks:
    base_url = get_base_url(response)
    writer.writerow({'reference_link': response.url, 'rss_link': urljoin_rfc(base_url, rss)}) #writing data into file.