Tag: web crawler

Error run scrapy after install docker

I want to crawl from dynamic content using scrapy. I get from the internet that I must install docker. But after installing it, I always get error when I run: scrapy runspider example.py or other scrapy command. Then I uninstall docker. But the error is still showed. This is the error: then I try to […]

docker of solr and nutch working together?

I’m trying to activate nutch and solr as dockers out of the box to consume that in REST. Tried same docker – started it with docker run –name nutch_solr -d -p 8899:8899 -p 8983:8983 -t momer/docker-solr-nutch:4.6.1 can’t surf to ..:8983, or …:8899 Tried different dockers – nutch and solr started it with: docker run –name […]

Finding the layers and layer sizes for each Docker image

For research purposes I’m trying to crawl the public Docker registry ( https://registry.hub.docker.com/ ) and find out 1) how many layers an average image has and 2) the sizes of these layers to get an idea of the distribution. However I studied the API and public libraries as well as the details on the github […]

Docker will be the best open platform for developers and sysadmins to build, ship, and run distributed applications.