我把Google的robotstxt解析库port到rust了,并且用ffi的方式跑通了C++的测试用例。
地址:https://github.com/Folyd/robotstxt
robotstxt
A native Rust port of Google's robots.txt parser and matcher C++ library.
- Native Rust port, no third-part crate dependency
- Preserves all behavior of original library
- Consistent API with original library
- 100% google original test passed
Installation
[dependencies]
robotstxt = "0.2.0"
Quick start
use robotstxt::DefaultMatcher;
let mut matcher = DefaultMatcher::default();
let robots_body = "user-agent: FooBot\n\
disallow: /\n";
assert_eq!(false, matcher.one_agent_allowed_by_robots(robots_body, "FooBot", "https://foo.com/"));
1
共 1 条评论, 1 页
评论区
写评论666