API documentation

One the main features of Open Scraper is to allow the scraped data and the spiders' information to exit the application thanks to an API (Application Programming Interface).

Here are listed the url routes and arguments you can use to request the data to the Open Scraper app. You can test these routes by clicking on the example corresponding to the argument.

route : api/stats

Argument Default Value Type Description Example Returns
only_counts_simple False unique retrieve only simple counts http://cis-openscraper.com/api/stats?only_counts_simple=true
only_tags_stats False unique retrieve only stats counts http://cis-openscraper.com/api/stats?only_tags_stats=true
only_spiders_stats False unique retrieve only stats counts http://cis-openscraper.com/api/stats?only_spiders_stats=true

route : api/infos

Argument Default Value Type Description Example Returns
only_dm_list False unique retrieve only the datamodel http://cis-openscraper.com/api/infos?only_dm_list=true
only_spiders_list False unique retrieve only the list of spiders http://cis-openscraper.com/api/infos?only_spiders_list=true
get_all_spiders False unique retrieve all spiders infos, even not tested http://cis-openscraper.com/api/infos?get_all_spiders=true

route : api/data

Argument Default Value Type Description Example Returns
page_n 1 integer page number of the results http://cis-openscraper.com/api/data?page_n=3
results_per_page 25 integer number of items by page http://cis-openscraper.com/api/data?results_per_page=14
token None unique client's JWT (token) to check user's permissions http://cis-openscraper.com/api/data?token=my_token
spider_id all id_spider search items from a spider or a list of spiders by its/their id http://cis-openscraper.com/api/data?spider_id=5a95754ee175ef55698a025b
item_id None id_item search for an item by its id http://cis-openscraper.com/api/data?item_id=5bce2c673ba14c6fa2516659
is_complete False unique only complete records... a bit optimistic isn't it ? http://cis-openscraper.com/api/data?is_complete=true
search_for [] list list of words to search in data collection http://cis-openscraper.com/api/data?search_for=my_query_string
open_level opendata unique fields of data to be shown ( 'all' == 'opendata' + 'commons' + 'private' ) http://cis-openscraper.com/api/data?open_level=commons
all_results False unique to override results_per_page http://cis-openscraper.com/api/data?all_results=true
added_by None id_user list of user having added the data http://cis-openscraper.com/api/data?added_by=5ac55aa63ba14c4b1672a29d
sort_by None unique sort the results by a custom field id or a default field http://cis-openscraper.com/api/data?sort_by=spider_name
shuffle_seed None unique seed to randomize list order http://cis-openscraper.com/api/data?shuffle_seed=123456