Inspecting your spider's runtime environment with the Job Console

With the job console you can open a Unix shell directly into the container where your job is running. Once in the console, you can perform tasks such as:

• tailing the Scrapy log (tail -f scrapy.log)
• accessing the Scrapy Telnet Console with telnet localhost 6023
• inspecting the files generated by your Scrapy process
• use top to inspect CPU and memory consumption
• ... and anything else you can do in a Unix console

To open the job console, click the Console tab:

Finally, a couple of things to keep in mind:

• the job console is only available for running jobs
• access is restricted to the directory of your scrapy project (in other words, you don't have root access)