通過robots.txt屏蔽國際公認垃圾蜘蛛的訪問

網棍

通過robots.txt屏蔽國際公認垃圾蜘蛛的訪問


全球互聯網上有多少搜索引擎機器人（爬行蜘蛛）在工作？這個問題很難回答，你要知道有很多人開發了自己的機器人來竊取別人的信息，也有很多為了其他的利益而開發了機器人。這些機器人，都是垃圾機器人，不但占用了網站的帶寬，而且網站的用戶信息都可能已經被偷走了。這裏，我根據國外同行的一些小技巧介紹下如何給自己的添加robots.txt並進行設置，來徹底與垃圾搜索引擎機器人說Goodbye。

首先，打開記事本，復制下面這些代碼。代碼解釋：下面是126個國際上公認的垃圾搜索引擎機器人、蜘蛛、搜索代理等，通過disallow:/來完全禁止他們爬行網站。

1. 
   
2. User-agent: larbin
   
3. Disallow: /
   
4. User-agent: b2w/0.1
   
5. Disallow: /
   
6. User-agent: Copernic
   
7. Disallow: /
   
8. User-agent: psbot
   
9. Disallow: /
   
10. User-agent: Python-urllib
    
11. Disallow: /
    
12. User-agent: URL_Spider_Pro
    
13. Disallow: /
    
14. User-agent: CherryPicker
    
15. Disallow: /
    
16. User-agent: EmailCollector
    
17. Disallow: /
    
18. User-agent: EmailSiphon
    
19. Disallow: /
    
20. User-agent: WebBandit
    
21. Disallow: /
    
22. User-agent: EmailWolf
    
23. Disallow: /
    
24. User-agent: ExtractorPro
    
25. Disallow: /
    
26. User-agent: CopyRightCheck
    
27. Disallow: /
    
28. User-agent: Crescent
    
29. Disallow: /
    
30. User-agent: SiteSnagger
    
31. Disallow: /
    
32. User-agent: ProWebWalker
    
33. Disallow: /
    
34. User-agent: CheeseBot
    
35. Disallow: /
    
36. User-agent: LNSpiderguy
    
37. Disallow: /
    
38. User-agent: Alexibot
    
39. Disallow: /
    
40. User-agent: Teleport
    
41. Disallow: /
    
42. User-agent: TeleportPro
    
43. Disallow: /
    
44. User-agent: MIIxpc
    
45. Disallow: /
    
46. User-agent: Telesoft
    
47. Disallow: /
    
48. User-agent: Website Quester
    
49. Disallow: /
    
50. User-agent: WebZip
    
51. Disallow: /
    
52. User-agent: moget/2.1
    
53. Disallow: /
    
54. User-agent: WebZip/4.0
    
55. Disallow: /
    
56. User-agent: WebStripper
    
57. Disallow: /
    
58. User-agent: WebSauger
    
59. Disallow: /
    
60. User-agent: WebCopier
    
61. Disallow: /
    
62. User-agent: NetAnts
    
63. Disallow: /
    
64. User-agent: Mister PiX
    
65. Disallow: /
    
66. User-agent: WebAuto
    
67. Disallow: /
    
68. User-agent: TheNomad
    
69. Disallow: /
    
70. User-agent: WWW-Collector-E
    
71. Disallow: /
    
72. User-agent: RMA
    
73. Disallow: /
    
74. User-agent: libWeb/clsHTTP
    
75. Disallow: /
    
76. User-agent: asterias
    
77. Disallow: /
    
78. User-agent: httplib
    
79. Disallow: /
    
80. User-agent: turingos
    
81. Disallow: /
    
82. User-agent: spanner
    
83. Disallow: /
    
84. User-agent: InfoNaviRobot
    
85. Disallow: /
    
86. User-agent: Harvest/1.5
    
87. Disallow: /
    
88. User-agent: Bullseye/1.0
    
89. Disallow: /
    
90. User-agent: Mozilla/4.0 (compatible; BullsEye; Windows 95)
    
91. Disallow: /
    
92. User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0
    
93. Disallow: /
    
94. User-agent: CherryPickerSE/1.0
    
95. Disallow: /
    
96. User-agent: CherryPickerElite/1.0
    
97. Disallow: /
    
98. User-agent: WebBandit/3.50
    
99. Disallow: /
    
100. User-agent: NICErsPRO
     
101. Disallow: /
     
102. User-agent: Microsoft URL Control – 5.01.4511
     
103. Disallow: /
     
104. User-agent: Not Your Business!
     
105. Disallow: /
     
106. User-agent: Hidden-Referrer
     
107. Disallow: /
     
108. User-agent: DittoSpyder
     
109. Disallow: /
     
110. User-agent: Foobot
     
111. Disallow: /
     
112. User-agent: WebmasterWorldForumBot
     
113. Disallow: /
     
114. User-agent: SpankBot
     
115. Disallow: /
     
116. User-agent: BotALot
     
117. Disallow: /
     
118. User-agent: lwp-trivial/1.34
     
119. Disallow: /
     
120. User-agent: lwp-trivial
     
121. Disallow: /
     
122. User-agent: BunnySlippers
     
123. Disallow: /
     
124. User-agent: Microsoft URL Control – 6.00.8169
     
125. Disallow: /
     
126. User-agent: URLy Warning
     
127. Disallow: /
     
128. User-agent: Wget/1.6
     
129. Disallow: /
     
130. User-agent: Wget/1.5.3
     
131. Disallow: /
     
132. User-agent: Wget
     
133. Disallow: /
     
134. User-agent: LinkWalker
     
135. Disallow: /
     
136. User-agent: cosmos
     
137. Disallow: /
     
138. User-agent: moget
     
139. Disallow: /
     
140. User-agent: hloader
     
141. Disallow: /
     
142. User-agent: humanlinks
     
143. Disallow: /
     
144. User-agent: LinkextractorPro
     
145. Disallow: /
     
146. User-agent: Offline Explorer
     
147. Disallow: /
     
148. User-agent: Mata Hari
     
149. Disallow: /
     
150. User-agent: LexiBot
     
151. Disallow: /
     
152. User-agent: Web Image Collector
     
153. Disallow: /
     
154. User-agent: The Intraformant
     
155. Disallow: /
     
156. User-agent: True_Robot/1.0
     
157. Disallow: /
     
158. User-agent: True_Robot
     
159. Disallow: /
     
160. User-agent: BlowFish/1.0
     
161. Disallow: /
     
162. User-agent: JennyBot
     
163. Disallow: /
     
164. User-agent: MIIxpc/4.2
     
165. Disallow: /
     
166. User-agent: BuiltBotTough
     
167. Disallow: /
     
168. User-agent: ProPowerBot/2.14
     
169. Disallow: /
     
170. User-agent: BackDoorBot/1.0
     
171. Disallow: /
     
172. User-agent: toCrawl/UrlDispatcher
     
173. Disallow: /
     
174. User-agent: WebEnhancer
     
175. Disallow: /
     
176. User-agent: suzuran
     
177. Disallow: /
     
178. User-agent: TightTwatBot
     
179. Disallow: /
     
180. User-agent: VCI WebViewer VCI WebViewer Win32
     
181. Disallow: /
     
182. User-agent: VCI
     
183. Disallow: /
     
184. User-agent: Szukacz/1.4
     
185. Disallow: /
     
186. User-agent: QueryN Metasearch
     
187. Disallow: /
     
188. User-agent: Openfind data gathere
     
189. Disallow: /
     
190. User-agent: Openfind
     
191. Disallow: /
     
192. User-agent: Xenu's Link Sleuth 1.1c
     
193. Disallow: /
     
194. User-agent: Xenu's
     
195. Disallow: /
     
196. User-agent: Zeus
     
197. Disallow: /
     
198. User-agent: RepoMonkey Bait & Tackle/v1.01
     
199. Disallow: /
     
200. User-agent: RepoMonkey
     
201. Disallow: /
     
202. User-agent: Microsoft URL Control
     
203. Disallow: /
     
204. User-agent: Openbot
     
205. Disallow: /
     
206. User-agent: URL Control
     
207. Disallow: /
     
208. User-agent: Zeus Link Scout
     
209. Disallow: /
     
210. User-agent: Zeus 32297 Webster Pro V2.9 Win32
     
211. Disallow: /
     
212. User-agent: Webster Pro
     
213. Disallow: /
     
214. User-agent: EroCrawler
     
215. Disallow: /
     
216. User-agent: LinkScan/8.1a Unix
     
217. Disallow: /
     
218. User-agent: Keyword Density/0.9
     
219. Disallow: /
     
220. User-agent: Kenjin Spider
     
221. Disallow: /
     
222. User-agent: Iron33/1.0.2
     
223. Disallow: /
     
224. User-agent: Bookmark search tool
     
225. Disallow: /
     
226. User-agent: GetRight/4.2
     
227. Disallow: /
     
228. User-agent: FairAd Client
     
229. Disallow: /
     
230. User-agent: Gaisbot
     
231. Disallow: /
     
232. User-agent: Aqua_Products
     
233. Disallow: /
     
234. User-agent: Radiation Retriever 1.1
     
235. Disallow: /
     
236. User-agent: WebmasterWorld Extractor
     
237. Disallow: /
     
238. User-agent: Flaming AttackBot
     
239. Disallow: /
     
240. User-agent: Oracle Ultra Search
     
241. Disallow: /
     
242. User-agent: MSIECrawler
     
243. Disallow: /
     
244. User-agent: PerMan
     
245. Disallow: /
     
246. User-agent: searchpreview
     
247. Disallow: /

複製代碼

然後保存這個記事本，命名為robots.txt。

把這個保存好的文件上傳到你的網站根目錄下，也就是你的網站index.htm的相同目錄下。

大功告成。

就這麽簡單，利用robots.txt設置，可以完全把垃圾機器人和蜘蛛屏蔽掉，他們以後再也不會來到你的網站為非作歹了。


相關閱讀

巧妙使用robots.txt提升網站的權重


作者：網棍@南天 - SEO與網絡營銷
南天SEO - 全面知識，免費分享
文章名稱：通過robots.txt屏蔽國際公認垃圾蜘蛛的訪問，版權所有