An Improved T-Spider Distributed Crawler

To increase the speed of the crawler,this paper proposes a model that is based on the T-Spider.During the time of extracting links from the page content,the crawler takes use of the page cutting algorithm,and then uses a new algorithm of link priority computing to enhance the stability and increase the speed of the crawler.The experiment shows that it is availability.