Dev

See merge request !3

The new offer_percent field should help with identifying the best offers available.

In order to maximize Jumbo's scraped items without ending on a loop, a page limit has been placed by category.
Also, the api key and the page number are passed via headers intead of URL parameters.

This change should facilitate the linking between products and categories tables

Now using string-based date stamp. Also fixed Tottus' string categories.

Scrape date and simultaneous spider scraping

See merge request !2

With the run.py script, one can execute all crawlers in a single process

Now that the items are dated, it's posible to build a 'price history' to check changes in prices over time.

Reached stable and operating state.

See merge request !1

Now, to get a spider running, one must use scrapy crawl <Spider Name>

The logger replaces the 'debug' print statements lying around. The 'store' item parameter is for telling from what store the item comes from.

with this, one can check from which store the scraped item comes from

Now the scraper can throw the crawled items to a MongoDB database, with the connection credentials stored in scraper_tiendas/settings.py. The Database must have the scraper's user created beforehand.

To check product availability an API call is needed. Because scrapy doesn't allow to make sub-requests in the parse stage I was forced to use requests.

Also the next page section is now fixed. It'll no longer go to the marketplace main page after reaching the last page.

Merge branch 'dev' of gitlab.labcomp.cl:speedwagon-iwg101-2019-2/gran-super/scraper-tiendas into dev
Recovering changes made with GitLab's Web IDE

Spider básica para Lider, completando las 3 tiendas principales.
Cambiado parser html.parser por lxml

Añadidas las spiders de las tiendas Jumbo y Tottus (WIP), además de un placeholder para la spider de la tienda Lider