Provides functions to download and parse 'robots.txt' files. Ultimately the package makes it easy to check if bots (spiders, crawler, scrapers, ...) are allowed to access specific resources on a domain.
| Version: | 0.7.15 |
| Depends: | R (≥ 3.0.0) |
| Imports: | stringr (≥ 1.0.0), httr (≥ 1.0.0), spiderbar (≥ 0.2.0), future.apply (≥ 1.0.0), magrittr, utils |
| Suggests: | knitr, rmarkdown, dplyr, testthat, covr, curl |
| Published: | 2024-08-29 |
| DOI: | 10.32614/CRAN.package.robotstxt |
| Author: | Pedro Baltazar [aut, cre], Peter Meissner [aut], Kun Ren [aut, cph] (Author and copyright holder of list_merge.R.), Oliver Keys [ctb] (original release code review), Rich Fitz John [ctb] (original release code review) |
| Maintainer: | Pedro Baltazar <pedrobtz at gmail.com> |
| BugReports: | https://github.com/ropensci/robotstxt/issues |
| License: | MIT + file LICENSE |
| URL: | https://docs.ropensci.org/robotstxt/, https://github.com/ropensci/robotstxt |
| NeedsCompilation: | no |
| Materials: | NEWS |
| In views: | WebTechnologies |
| CRAN checks: | robotstxt results |
| Reference manual: | robotstxt.pdf |
| Vignettes: |
using_robotstxt (source) |
| Package source: | robotstxt_0.7.15.tar.gz |
| Windows binaries: | r-devel: robotstxt_0.7.15.zip, r-release: robotstxt_0.7.15.zip, r-oldrel: robotstxt_0.7.15.zip |
| macOS binaries: | r-devel (arm64): robotstxt_0.7.15.tgz, r-release (arm64): robotstxt_0.7.15.tgz, r-oldrel (arm64): robotstxt_0.7.15.tgz, r-devel (x86_64): robotstxt_0.7.15.tgz, r-release (x86_64): robotstxt_0.7.15.tgz, r-oldrel (x86_64): robotstxt_0.7.15.tgz |
| Old sources: | robotstxt archive |
| Reverse imports: | polite, ralger, readapra |
| Reverse suggests: | newsanchor, spiderbar, vosonSML, webchem |
Please use the canonical form https://CRAN.R-project.org/package=robotstxt to link to this page.