This content has been marked as final. Show 2 replies
It looks like there's several others referenced from there as well. Good luck!
Furthermore, I would say that the Internet Archive's open source crawler is pretty well "tested" all thing's considered.