Previous Thread
Next Thread
Print Thread
Rate Thread
[7.x] Robots (Crawlers) List for UBB.threads
#320566 06/25/2014 4:45 AM
Joined: Jul 2001
Posts: 1,072
Likes: 39
isaac Online Happy OP
coffee and code
OP Online Happy
coffee and code
Joined: Jul 2001
Posts: 1,072
Likes: 39
Robots (Crawlers) List for UBB.threads - UPDATED 2020-01-14
formerly named "Search Engine Spiders List for UBB.threads"

About:
The advantage to using this list is that Search Engine Spiders get put into the correct "Search Spiders" group when viewing the forums/ubbthreads.php/online page.

This translation list is not used anywhere else in UBB.threads except on the "ubbthreads.php/online page". Having a long list of Search Engine Spiders will not slow down your forums -- the only exception is that depending on the server in which your site is hosted on, there may be a minimal delay in generating the "ubbthreads.php/online " page.

Details:
Total Agent Strings: 925
Sources:
https://udger.com/resources/ua-list/crawlers (2020-01-13) and
https://github.com/monperrus/crawler-user-agents (2019-12-11)

I've simply converted, combined, and cleaned the data for use with UBB.threads.

How-To:
1) COPY/PASTE this list to the "Search Engine Agents" box at: Control Panel > Display Options > Who's Online
2) Click "Update General Display Options"
done.

Having problems? REMOVE any blank lines from the top/bottom of your copied list.


Fresh Installs of UBB.threads 7.7.4+ are prepopulated with this list.
The 2020-01-14 List :
Code
007ac9=007ac9
13TABS=13TABS
1stwarning bot=1stwarning bot
28logsSpider=28logsSpider
2bone=2bone
2GDPR bot=2GDPR bot
2gisbot=2gisbot
2ip.ru=2ip.ru
360Spider=360Spider
3w24bot=3w24bot
A6-Indexer=A6-Indexer
Aboundex=Aboundex
acapbot=acapbot
acoonbot=acoonbot
AdAuth=AdAuth
Adbeat bot=Adbeat bot
adbeat_bot=adbeat_bot
AddSearchBot=AddSearchBot
AddThis.com=AddThis.com
AddThis=AddThis
Adform crawler=Adform crawler
Adidxbot=Adidxbot
ADmantX crawler=ADmantX crawler
ADmantX=ADmantX
AdminLabs=AdminLabs
Adolab bot=Adolab bot
AdsBot-Google=AdsBot-Google
AdsBot-Google-Mobile=AdsBot-Google-Mobile
adscanner=adscanner
AdsrvrBot=AdsrvrBot
adstxt.com crawler=adstxt.com crawler
AdsTxtCrawler=AdsTxtCrawler
adstxtlab.com Crawler=adstxtlab.com Crawler
AdvBot=AdvBot
agentslug=agentslug
Ahrefs=Ahrefs
aiHitBot=aiHitBot
AISearchBot=AISearchBot
Akamai crawler=Akamai crawler
Alertra=Alertra
Alexabot=Alexabot
Alignabot=Alignabot
AlphaBot=AlphaBot
AlphaSeoBot=AlphaSeoBot
alyze.info=alyze.info
Amazon bot=Amazon bot
Amazon CloudFront=Amazon CloudFront
AmazonAdBot=AmazonAdBot
AndersPinkBot=AndersPinkBot
AntBot=AntBot
antibot=antibot
Anturis Agent=Anturis Agent
AnyEvent=AnyEvent
Apercite bot=Apercite bot
Apercite=Apercite
APIs-Google=APIs-Google
AppBeat=AppBeat
AppEngine-Google=AppEngine-Google
AppInsights=AppInsights
Applebot=Applebot
AppNexus crawler=AppNexus crawler
arabot=arabot
arachnode.net=arachnode.net
Aranea=Aranea
archive.org_bot=archive.org_bot
Archive.St crawler=Archive.St crawler
ArchiveBot=ArchiveBot
ArchiveTeam crawler=ArchiveTeam crawler
Awario crawler=Awario crawler
awesomecrawler=awesomecrawler
axios=axios
aylienbot=aylienbot
B2B Bot=B2B Bot
backlink-check.de=backlink-check.de
BacklinkCrawler=BacklinkCrawler
BackupLand crawler=BackupLand crawler
Bad-Neighborhood=Bad-Neighborhood
Baiduspider=Baiduspider
Baidu-YunGuanCe=Baidu-YunGuanCe
Barkrowler=Barkrowler
BazQux=BazQux
BDCbot=BDCbot
BehloolBot=BehloolBot
betaBot=betaBot
bidswitchbot=bidswitchbot
BigInfoLabs crawler=BigInfoLabs crawler
BIGLOTRON=BIGLOTRON
bingbot=bingbot
BingPreview=BingPreview
binlar=binlar
BirilBot=BirilBot
bitlybot=bitlybot
bl.uk_lddc_bot=bl.uk_lddc_bot
Blackboard crawler=Blackboard crawler
Blackboard=Blackboard
BLEXBot=BLEXBot
blogmuraBot=blogmuraBot
Blogtrottr=Blogtrottr
BLP_bbot=BLP_bbot
bne.es crawler=bne.es crawler
bnf.fr_bot=bnf.fr_bot
BoardReader crawler=BoardReader crawler
BomboraBot=BomboraBot
Bot.AraTurka.com=Bot.AraTurka.com
botify=botify
bot-pge.chlooe.com=bot-pge.chlooe.com
BoxcarBot=BoxcarBot
brainobot=brainobot
BrandProtect bot=BrandProtect bot
BrandVerity=BrandVerity
Brandwatch crawler=Brandwatch crawler
BrokenLinkCheck.com=BrokenLinkCheck.com
Browsershots=Browsershots
BrowserSpyBot=BrowserSpyBot
BTWebClient=BTWebClient
BUbiNG=BUbiNG
BublupBot=BublupBot
Buck=Buck
BuiltWith=BuiltWith
bumb.ly crawler=bumb.ly crawler
Burf.co=Burf.co
buzzbot=buzzbot
ByteDance crawler=ByteDance crawler
Bytespider=Bytespider
Caliperbot=Caliperbot
CapsuleChecker=CapsuleChecker
careerbot=careerbot
CastFeedValidator=CastFeedValidator
CATExplorador=CATExplorador
CC Metadata Scaper=CC Metadata Scaper
CCBot=CCBot
centurybot9=centurybot9
changedetection=changedetection
check_http=check_http
Check-Host=Check-Host
CheckMarkNetwork=CheckMarkNetwork
Chirp=Chirp
Chrome-Lighthouse=Chrome-Lighthouse
citeseerxbot=citeseerxbot
CityGrid crawler=CityGrid crawler
Clarabot=Clarabot
CLcrawler=CLcrawler
Clickagy=Clickagy
Cliqzbot=Cliqzbot
Closure Compiler Service=Closure Compiler Service
Cloud mapping=Cloud mapping
CloudFlare crawler=CloudFlare crawler
CloudFlare-AlwaysOnline=CloudFlare-AlwaysOnline
Cloudinary=Cloudinary
CMS Crawler=CMS Crawler
coccoc=coccoc
Cocolyze bot=Cocolyze bot
[email protected][email protected]
CommaFeed=CommaFeed
Companybook-Crawler=Companybook-Crawler
COMSYS crawler=COMSYS crawler
content crawler spider=content crawler spider
ContentKing=ContentKing
ContextAd Bot=ContextAd Bot
contxbot=contxbot
convera=convera
CopperEgg=CopperEgg
Corax=Corax
CouponWCode Bot=CouponWCode Bot
Coveobot=Coveobot
crawler4j=crawler4j
CrawlForMe=CrawlForMe
CronDroid=CronDroid
Cronjob.de bot=Cronjob.de bot
cron-job.org=cron-job.org
Cronless=Cronless
CrowdTanglebot=CrowdTanglebot
CrunchBot=CrunchBot
CrystalSemanticsBot=CrystalSemanticsBot
Cula=Cula
cXensebot=cXensebot
CyberPatrol=CyberPatrol
DaaS.sh crawler=DaaS.sh crawler
DareBoost=DareBoost
Datafeedwatch=Datafeedwatch
datagnionbot=datagnionbot
Datanyze=Datanyze
Dataprovider Spider=Dataprovider Spider
Dataprovider.com=Dataprovider.com
DataXu=DataXu
Daum=Daum
Daumoa=Daumoa
DaveCrawler=DaveCrawler
DAWINCI ANTIPLAG=DAWINCI ANTIPLAG
dcrawl=dcrawl
ddline.cn crawler=ddline.cn crawler
Dead Link Checker=Dead Link Checker
deadlinkchecker=deadlinkchecker
deepcrawl=deepcrawl
demandbase-bot=demandbase-bot
Deskyobot=Deskyobot
DeuSu=DeuSu
DF Bot=DF Bot
Diffbot=Diffbot
Digg Deeper=Digg Deeper
Digincore bot=Digincore bot
DIGMATO.com crawler=DIGMATO.com crawler
discobot=discobot
Discordbot=Discordbot
Disqus=Disqus
dlvr.it=dlvr.it
DNS-Tools Header-Analyzer=DNS-Tools Header-Analyzer
DnyzBot=DnyzBot
Docoloc=Docoloc
Domain Re-Animator Bot=Domain Re-Animator Bot
domaincrawler=domaincrawler
domainsbot=domainsbot
DomainsDB.net MetaCrawler=DomainsDB.net MetaCrawler
DomainSigmaCrawler=DomainSigmaCrawler
DomainStatsBot=DomainStatsBot
dotbot=dotbot
Dotcom-Monitor bot=Dotcom-Monitor bot
DownloaderChrome=DownloaderChrome
downnotifier.com monitoring=downnotifier.com monitoring
DowntimeDetector=DowntimeDetector
draw.io=draw.io
drupact=drupact
DuckDuckBot=DuckDuckBot
DuckDuckGo-Favicons-Bot=DuckDuckGo-Favicons-Bot
Dun&BradstreetBot=Dun&BradstreetBot
e.ventures crawler=e.ventures crawler
EasyBib crawler=EasyBib crawler
EasyCron=EasyCron
Easy-Thumb=Easy-Thumb
ec2linkfinder=ec2linkfinder
EchoboxBot=EchoboxBot
edisterbot=edisterbot
electricmonk=electricmonk
Elefent=Elefent
elisabot=elisabot
Embedly=Embedly
EnliteAI crawler=EnliteAI crawler
EPFL cravler=EPFL cravler
epicbot=epicbot
eright=eright
europarchive.org=europarchive.org
evc-batch=evc-batch
Everyonedomainsbot=Everyonedomainsbot
EveryoneSocialBot=EveryoneSocialBot
exabot=exabot
Experian crawler=Experian crawler
Experibot=Experibot
ExtLinksBot=ExtLinksBot
EZID=EZID
ezooms=ezooms
FacebookBot=FacebookBot
facebookexternalhit=facebookexternalhit
Facebot=Facebot
Fake bot=Fake bot
FAST Enterprise Crawler=FAST Enterprise Crawler
FastCrawler=FastCrawler
FAST-WebCrawler=FAST-WebCrawler
Favicon=Favicon
faviconkit=faviconkit
FCCN crawler=FCCN crawler
fedoraplanet=fedoraplanet
Feedbin=Feedbin
FeedBunch=FeedBunch
FeedBurner=FeedBurner
feeder.co=feeder.co
Feedfetcher-Google=Feedfetcher-Google
Feedly=Feedly
Feedpresso crawler=Feedpresso crawler
Feedspot=Feedspot
Feedspotbot=Feedspotbot
FeedValidator=FeedValidator
FeedViewer=FeedViewer
Feedwind=Feedwind
FemtosearchBot=FemtosearchBot
Fetch=Fetch
Fever=Fever
Filestack=Filestack
filterdb.iss.netcrawler=filterdb.iss.netcrawler
FindITAnswersbot=FindITAnswersbot
findl.sk bot=findl.sk bot
findlink=findlink
findthatfile=findthatfile
findxbot=findxbot
FirmoGraph=FirmoGraph
Flamingo_SearchEngine=Flamingo_SearchEngine
FlipboardBot=FlipboardBot
FlipboardProxy=FlipboardProxy
fluffy=fluffy
fr-crawler=fr-crawler
freefind=freefind
FreeWebMonitoring SiteChecker=FreeWebMonitoring SiteChecker
Freshworks bot=Freshworks bot
Friendica=Friendica
Frontera=Frontera
fuelbot=fuelbot
Fuze_Bot=Fuze_Bot
Fyrebot=Fyrebot
g00g1e.net=g00g1e.net
G2 Web Services=G2 Web Services
g2reader-bot=g2reader-bot
GAChecker=GAChecker
GarlikCrawler=GarlikCrawler
GDNP crawler=GDNP crawler
Geek-Tools=Geek-Tools
Genieo Web filter=Genieo Web filter
Genieo=Genieo
GetLinkInfo.com=GetLinkInfo.com
Ghost Inspector=Ghost Inspector
Gigablast=Gigablast
Gigabot=Gigabot
GingerCrawler=GingerCrawler
github-camo=github-camo
Gluten Free Crawler=Gluten Free Crawler
gnam gnam spider=gnam gnam spider
GnowitNewsbot=GnowitNewsbot
gocrawl=gocrawl
Go-http-client=Go-http-client
Google AMP crawler=Google AMP crawler
Google AppsViewer=Google AppsViewer
Google Cloud AI=Google Cloud AI
Google Favicon=Google Favicon
Google Page Speed Insights=Google Page Speed Insights
Google Read Aloud=Google Read Aloud
Google Web Preview=Google Web Preview
Google-Ads=Google-Ads
Google-AdWords=Google-AdWords
Google-Adwords-Instant=Google-Adwords-Instant
Googlebot AdsBot Mobile=AdsBot-Google-Mobile
Googlebot AdSense=Mediapartners-Google
Googlebot Image=Googlebot-Image
Googlebot Mobile=Googlebot-Mobile
Googlebot Snippet=Googlebot snippet
Googlebot Video=Googlebot-Video
Googlebot Web Preview=Google Web Preview
Googlebot=Googlebot
Googlebot-Image=Googlebot-Image
Googlebot-Mobile=Googlebot-Mobile
Googlebot-News=Googlebot-News
Googlebot-Video=Googlebot-Video
GoogleDocs=GoogleDocs
Google-PhysicalWeb=Google-PhysicalWeb
GoogleSites=GoogleSites
Google-Site-Verification=Google-Site-Verification
Google-Structured-Data-Testing-Tool=Google-Structured-Data-Testing-Tool
Google-Test=Google-Test
google-xrawler=google-xrawler
Google-Youtube-Links=Google-Youtube-Links
Gosign crawler=Gosign crawler
GoSquared=GoSquared
GotSiteMonitor=GotSiteMonitor
Gowikibot=Gowikibot
Grammarly=Grammarly
GrapeshotCrawler=GrapeshotCrawler
Graydon crawler=Graydon crawler
Grobbot=Grobbot
GroupHigh=GroupHigh
grub.org=grub.org
gslfbot=gslfbot
GTmetrix=GTmetrix
GumGum Bot=GumGum Bot
HappyApps crawler=HappyApps crawler
Hatena=Hatena
hCardValidator=hCardValidator
HeadlessChrome=HeadlessChrome
HeartRailsBot=HeartRailsBot
heritrix=heritrix
HetrixTools crawler=HetrixTools crawler
Heurekabot=Heurekabot
Hexometer.com crawler=Hexometer.com crawler
historious=historious
hledejLevne.cz=hledejLevne.cz
Hlidam.to robot=Hlidam.to robot
hosterstats com=hosterstats com
HostTracker=HostTracker
htmlyse crawler=htmlyse crawler
http_get=http_get
httpunit=httpunit
HttpUrlConnection=HttpUrlConnection
HTTrack=HTTrack
HubSpot Crawler=HubSpot Crawler
HubSpot=HubSpot
Hypefactors crawler=Hypefactors crawler
HyperZbozi.cz Feeder=HyperZbozi.cz Feeder
HypeStat=HypeStat
ia_archiver=ia_archiver
IAB crawler=IAB crawler
IAS crawler=IAS crawler
ICBot=ICBot
ICC-Crawler=ICC-Crawler
ichiro=ichiro
iCjobs=iCjobs
Iframely=Iframely
iGooglePortal=iGooglePortal
ImageFetcher=ImageFetcher
ImplisenseBot=ImplisenseBot
imrbot=imrbot
IndeedBot=IndeedBot
Infegy=Infegy
InoopaBot=InoopaBot
integromedb=integromedb
intelium_bot=intelium_bot
InterfaxScanBot=InterfaxScanBot
InternetSeer=InternetSeer
internetVista monitor=internetVista monitor
internetwache.org crawler=internetwache.org crawler
iodc crawler=iodc crawler
IPIP.NET crawler=IPIP.NET crawler
ips-agent=ips-agent
ipv6-test.com validator=ipv6-test.com validator
ip-web-crawler.com=ip-web-crawler.com
iqdb=iqdb
Irokez.cz monitoring=Irokez.cz monitoring
Is it up=Is it up
isa=isa
iSec_Bot=iSec_Bot
iskanie=iskanie
ISOWQ=ISOWQ
IstellaBot=IstellaBot
isUp.li Website Monitoring=isUp.li Website Monitoring
it2media-domain-crawler=it2media-domain-crawler
IzABee Spider=IzABee Spider
JAHHO crawler=JAHHO crawler
James BOT=James BOT
JamesBOT=JamesBOT
Jamie's Spider=Jamie's Spider
Jeffrey Exif Viewer=Jeffrey Exif Viewer
Jetslide=Jetslide
Jetty=Jetty
JobboerseBot=JobboerseBot
JobKereso=JobKereso
Jooblebot=Jooblebot
jpg-newsbot=jpg-newsbot
Jugendschutzprogramm-Crawler=Jugendschutzprogramm-Crawler
jyxobot=jyxobot
K7MLWCBot=K7MLWCBot
kakao crawler=kakao crawler
kansalliskirjasto.fi crawler=kansalliskirjasto.fi crawler
Kaspersky Lab bot=Kaspersky Lab bot
kazbtbot=kazbtbot
KD Bot=KD Bot
KeepRight OpenStreetMap Checker=KeepRight OpenStreetMap Checker
Kemvibot=Kemvibot
Keybot Translation=Keybot Translation
KeyCDN bot=KeyCDN bot
keyword-hero bot=keyword-hero bot
KOCMOHABT bot=KOCMOHABT bot
KomodiaBot=KomodiaBot
KosmioBot=KosmioBot
Landau-Media-Spider=Landau-Media-Spider
Laserlikebot=Laserlikebot
lb-spider=lb-spider
LCC=LCC
Lead Inspector Crawler=Lead Inspector Crawler
leady.com crawler=leady.com crawler
Leikibot=Leikibot
LetsearchBot=LetsearchBot
letsencrypt bot=letsencrypt bot
libwww-perl=libwww-perl
Licorne crawler=Licorne crawler
LightspeedSystems Crawler=LightspeedSystems Crawler
Linespider=Linespider
Linguee Bot=Linguee Bot
Link Valet Online=Link Valet Online
link_thumbnailer=link_thumbnailer
LinkAlarm=LinkAlarm
linkapediabot=linkapediabot
LinkArchiver=LinkArchiver
linkdex=linkdex
linkdexbot=linkdexbot
LinkedInBot=LinkedInBot
linkfluence=linkfluence
LinkpadBot=LinkpadBot
linkpeek=linkpeek
LinkTiger=LinkTiger
LinqiaBot=LinqiaBot
lipperhey=lipperhey
LivelapBot=LivelapBot
looid.com crawler=looid.com crawler
lssbot=lssbot
lssrocketcrawler=lssrocketcrawler
ltx71=ltx71
Luminator-robots=Luminator-robots
LumtelBot=LumtelBot
LXRbot=LXRbot
MagiBot=MagiBot
magpie-crawler=magpie-crawler
Mail.Ru bot=Mail.Ru bot
Mail.RU_Bot=Mail.RU_Bot
Mailkit crawler=Mailkit crawler
Mappy=Mappy
mappydata=mappydata
MarketingMiner Bot=MarketingMiner Bot
MarketMuseBot=MarketMuseBot
masscan=masscan
Mastodon=Mastodon
MauiBot=MauiBot
MBCrawler=MBCrawler
MB-SiteCrawler=MB-SiteCrawler
Mediapartners=Mediapartners
Mediapartners-Google=Mediapartners-Google
Mediatoolkitbot=Mediatoolkitbot
Mediumbot=Mediumbot
MegaIndex=MegaIndex
MeltwaterNews=MeltwaterNews
memorybot=memorybot
Mergadobot=Mergadobot
MetaInspector=MetaInspector
MetaJobBot=MetaJobBot
MetaURI=MetaURI
Minds crawler=Minds crawler
mindUpBot=mindUpBot
Miniature.io=Miniature.io
Miniflux=Miniflux
Miralinks Robot=Miralinks Robot
Mixnode=Mixnode
MixnodeCache=MixnodeCache
MixrankBot=MixrankBot
MJ12bot=MJ12bot
mlbot=mlbot
moatbot=moatbot
MojeekBot=MojeekBot
Monibot=Monibot
monitis=monitis
Monitive Uptime Monitoring Bot=Monitive Uptime Monitoring Bot
monitorabot=monitorabot
monitorbacklinks.com bot=monitorbacklinks.com bot
montastic-monitor=montastic-monitor
MonTools.Com=MonTools.Com
MoodleBot=MoodleBot
Moreover=Moreover
msnbot=msnbot
msrbot=msrbot
MuckRack=MuckRack
Multiviewbot=Multiviewbot
Naver scrapbook=Naver scrapbook
NaverBot=NaverBot
nbertaupete95=nbertaupete95
NerdByNature.Bot=NerdByNature.Bot
nerdybot=nerdybot
netarkivet.dk crawler=netarkivet.dk crawler
Netcraft crawler=Netcraft crawler
NetcraftSurveyAgent=NetcraftSurveyAgent
netEstate Crawler=netEstate Crawler
netEstate NE Crawler=netEstate NE Crawler
Netpeak bot=Netpeak bot
netresearchserver=netresearchserver
NetSystemsResearch crawler=NetSystemsResearch crawler
Netvibes crawler=Netvibes crawler
Netvibes=Netvibes
netzzappen crawler=netzzappen crawler
newsharecounts=newsharecounts
newspaper=newspaper
NextCloud=NextCloud
niki-bot=niki-bot
Nimbostratus-Bot=Nimbostratus-Bot
NING=NING
NinjaBot=NinjaBot
NIXStatsbot=NIXStatsbot
Nmap Scripting Engine=Nmap Scripting Engine
Nmap=Nmap
Nodemeter=Nodemeter
NodePing=NodePing
northcutt.com crawler=northcutt.com crawler
NortheasternSysNetBot=NortheasternSysNetBot
NTENTbot=NTENTbot
nutch=nutch
Nuzzel=Nuzzel
oasis.go.kr crawler=oasis.go.kr crawler
oBot=oBot
Ocarinabot=Ocarinabot
OdklBot=OdklBot
OhDear crawler=OhDear crawler
okhttp=okhttp
omgili=omgili
omgilibot=omgilibot
OnCrawl=OnCrawl
Onespot crawler=Onespot crawler
Online Domain Tools=Online Domain Tools
online-webceo-bot=online-webceo-bot
OpenGraphCheck=OpenGraphCheck
OpenHoseBot=OpenHoseBot
openindexspider=openindexspider
Openstat=Openstat
OrangeBot=OrangeBot
OrgProbe=OrgProbe
OSZKbot=OSZKbot
outbrain=outbrain
OutclicksBot=OutclicksBot
Owlin=Owlin
page2rss=page2rss
PageModified Crawler=PageModified Crawler
PagePeeker=PagePeeker
Pagespeed=Pagespeed
Panopta=Panopta
Panscient crawler=Panscient crawler
panscient=panscient
PaperLiBot=PaperLiBot
ParaCrawl=ParaCrawl
PartyflockLinkChecker=PartyflockLinkChecker
Pcore-HTTP=Pcore-HTTP
PDFDriveCrawler=PDFDriveCrawler
Pearltrees=Pearltrees
PhantomJS=PhantomJS
PhantomJsCloud.com=PhantomJsCloud.com
Photon=Photon
phpcrawl=phpcrawl
phpservermon=phpservermon
pilicanbot=pilicanbot
Pingability.com bot=Pingability.com bot
PingAdmin.Ru=PingAdmin.Ru
Pingdom.com bot=Pingdom.com bot
pingdom=pingdom
Pinterest bot=Pinterest bot
pinterest.com.bot=pinterest.com.bot
PiplBot=PiplBot
PLAZOO bot=PLAZOO bot
PleskBot=PleskBot
Plukkie=Plukkie
PlurkBot=PlurkBot
Pocket bot=Pocket bot
PocketParser=PocketParser
Port Monitor check service=Port Monitor check service
postrank=postrank
PowerMapper Crawler=PowerMapper Crawler
pr-cy.ru Bot=pr-cy.ru Bot
PR-CY.RU=PR-CY.RU
Premsgo.fr bot=Premsgo.fr bot
Prerender.cloud=Prerender.cloud
Prerender=Prerender
presearch crawler=presearch crawler
Primalbot=Primalbot
PrintFriendly.com=PrintFriendly.com
PritTorrent=PritTorrent
PrivacyAwareBot=PrivacyAwareBot
PrivacyScore.org crawler=PrivacyScore.org crawler
Prlog=Prlog
probethenet.com scanner=probethenet.com scanner
Project 25499=Project 25499
Promotion_Tools=Promotion_Tools
Protopage=Protopage
proximic=proximic
psbot=psbot
PubMatic Crawler=PubMatic Crawler
Pulsepoint crawler=Pulsepoint crawler
Pulsepoint=Pulsepoint
purebot=purebot
python-requests=python-requests
Python-urllib=Python-urllib
Qwantify=Qwantify
R6 bot=R6 bot
Rambler bot=Rambler bot
RankActiveLinkBot=RankActiveLinkBot
RankingBot=RankingBot
RankurBot=RankurBot
RavenCrawler=RavenCrawler
REDbot=REDbot
redditbot=redditbot
RegionStuttgartBot=RegionStuttgartBot
Re-re crawler=Re-re crawler
RetrevoPageAnalyzer=RetrevoPageAnalyzer
Riddlerbot=Riddlerbot
Rigor crawler=Rigor crawler
Rivva=Rivva
Robots_Tester=Robots_Tester
RobotsChecker=RobotsChecker
rogerbot=rogerbot
Roistat Bot=Roistat Bot
ROR Sitemap Generator=ROR Sitemap Generator
RSiteAuditor=RSiteAuditor
RSSingBot=RSSingBot
RSSMicro.com RSS/Atom Feed Robot=RSSMicro.com RSS/Atom Feed Robot
RukiCrawler=RukiCrawler
Runet-Research-Crawler=Runet-Research-Crawler
rushBot=rushBot
RyowlEngine=RyowlEngine
Ryte Bot=Ryte Bot
s4a=s4a
SabsimBot=SabsimBot
SafeDNSBot=SafeDNSBot
SafeSearch microdata crawler=SafeSearch microdata crawler
SBL-BOT=SBL-BOT
SBSearch=SBSearch
SCFCrawler=SCFCrawler
ScoutJet=ScoutJet
Scrapy=Scrapy
Screaming Frog SEO Spider=Screaming Frog SEO Spider
screeenly-bot=screeenly-bot
scribdbot=scribdbot
seekbot=seekbot
Seeker=Seeker
Seekport Crawler=Seekport Crawler
SeeWithKids bot=SeeWithKids bot
semanticbot=semanticbot
SemanticJuice crawler=SemanticJuice crawler
SemanticScholarBot=SemanticScholarBot
semantic-visions.com crawler=semantic-visions.com crawler
SEMrushBot=SEMrushBot
Semtix.cz bot=Semtix.cz bot
SentiBot=SentiBot
sentry=sentry
Seo Servis=Seo Servis
Seobility=Seobility
SEOdiver=SEOdiver
SEOkicks=SEOkicks
SEOkicks-Robot=SEOkicks-Robot
SeopultContentAnalyzer=SeopultContentAnalyzer
seoscanners.net bot=seoscanners.net bot
seoscanners=seoscanners
SeoSiteCheckup=SeoSiteCheckup
SerendeputyBot=SerendeputyBot
serpstatbot=serpstatbot
SERPtimizerBot=SERPtimizerBot
Server Density=Server Density
ServiceUptime.robot=ServiceUptime.robot
SetCronJob=SetCronJob
Seznam Zbozi.cz=Seznam Zbozi.cz
seznambot=seznambot
Sideqik crawler=Sideqik crawler
SimpleCrawler=SimpleCrawler
SimpleScraper=SimpleScraper
SimplyFast.info=SimplyFast.info
sistrix crawler=sistrix crawler
SISTRIX Optimizer=SISTRIX Optimizer
Site24x7=Site24x7
sitebot=sitebot
SiteCheckerBot=SiteCheckerBot
Sitedomain-Bot=Sitedomain-Bot
siteexplorer.info=siteexplorer.info
SiteExplorer=SiteExplorer
Siteimprove.com bot=Siteimprove.com bot
Siteimprove.com=Siteimprove.com
SiteProbe=SiteProbe
Site-Shot=Site-Shot
SiteTruth.com=SiteTruth.com
SiteUptime.com=SiteUptime.com
SJUUPBot=SJUUPBot
SKB Kontur bot=SKB Kontur bot
SkypeUriPreview=SkypeUriPreview
Slackbot=Slackbot
Slack-ImgProxy=Slack-ImgProxy
Slurp=Slurp
smtbot=smtbot
Snacktory=Snacktory
snapchat-proxy=snapchat-proxy
SnapSearch=SnapSearch
sniptracker=sniptracker
SocialRankIOBot=SocialRankIOBot
sogou spider=sogou spider
Sogou=Sogou
SOLOFIELD bot=SOLOFIELD bot
Sonic=Sonic
SpazioDati bot=SpazioDati bot
spbot=spbot
Specificfeeds=Specificfeeds
speedy=speedy
SpiderLing=SpiderLing
Splash=Splash
SpuhexBot=SpuhexBot
SpyOnWeb=SpyOnWeb
SSL Labs=SSL Labs
ssl-tools.net=ssl-tools.net
sstbot=sstbot
startmebot=startmebot
statdom.ru/Bot=statdom.ru/Bot
StatOnlineRuBot=StatOnlineRuBot
StatusCake=StatusCake
Statusdroid=Statusdroid
Steeler=Steeler
Stephan Kopp bot=Stephan Kopp bot
StorygizeBot=StorygizeBot
Streamline3Bot=Streamline3Bot
Structured Data Markup Helper=Structured Data Markup Helper
Sucuri bot=Sucuri bot
summify=summify
Super Monitoring=Super Monitoring
SuperPagesUrlVerifyBot=SuperPagesUrlVerifyBot
SurdotlyBot=SurdotlyBot
SurveyBot=SurveyBot
SvetaBot=SvetaBot
SWIMGBot=SWIMGBot
Synthesio Crawler=Synthesio Crawler
Sysomos bot=Sysomos bot
Sysomos=Sysomos
TA SEO Crawler=TA SEO Crawler
tagoobot=tagoobot
TangibleeBot=TangibleeBot
Taringa=Taringa
TelegramBot=TelegramBot
Teoma=Teoma
Testomatobot=Testomatobot
TestURI crawler=TestURI crawler
TextRazor crawler=TextRazor crawler
The Knowledge AI=The Knowledge AI
theoldreader.com=theoldreader.com
Thinklab=Thinklab
Thither.Direct Bot=Thither.Direct Bot
thumbshots-de-Bot=thumbshots-de-Bot
TigerBot=TigerBot
TinEye=TinEye
Tiny Tiny RSS=Tiny Tiny RSS
TMM Crawler=TMM Crawler
TomBot=TomBot
Tools4noobs.com Spider=Tools4noobs.com Spider
toplistbot=toplistbot
ToutiaoSpider=ToutiaoSpider
Toweyabot=Toweyabot
Traackr.com=Traackr.com
tracemyfile=tracemyfile
trendictionbot=trendictionbot
Trove=Trove
TurnitinBot=TurnitinBot
TweetmemeBot=TweetmemeBot
Tweezler=Tweezler
twengabot=twengabot
Twingly Recon=Twingly Recon
Twingly=Twingly
Twitterbot=Twitterbot
Twurly=Twurly
UASlinkChecker=UASlinkChecker
uclassify.com=uclassify.com
uMBot=uMBot
umich.edu crawler=umich.edu crawler
um-LN=um-LN
Unshorten.It=Unshorten.It
updown.io bot=updown.io bot
Upflow=Upflow
uptime.com=uptime.com
Uptimebot.org=Uptimebot.org
Uptimebot=Uptimebot
UptimeRobot=UptimeRobot
Uptimia=Uptimia
uptm.io=uptm.io
urlappendbot=urlappendbot
Urlcheckr=Urlcheckr
URLitor.com=URLitor.com
useeBookChecker=useeBookChecker
UsineNouvelleCrawler=UsineNouvelleCrawler
Vagabondo=Vagabondo
Validator.nu=Validator.nu
vebidoobot=vebidoobot
VelenPublicWebCrawler=VelenPublicWebCrawler
Veoozbot=Veoozbot
VerticalLeap bot=VerticalLeap bot
veu=veu
Virusdie crawler=Virusdie crawler
virustotal=virustotal
vkShare=vkShare
voilabot=voilabot
Vsevjednom.cz crawler=Vsevjednom.cz crawler
vuhuvBot=vuhuvBot
W3C_CSS_Validator=W3C_CSS_Validator
W3C_I18n-Checker=W3C_I18n-Checker
W3C_Unicorn=W3C_Unicorn
W3C_Validator=W3C_Validator
W3C-checklink=W3C-checklink
W3C-mobileOK=W3C-mobileOK
w3dt.net crawler=w3dt.net crawler
Wappalyzer=Wappalyzer
WatchMouse=WatchMouse
wbsearchbot=wbsearchbot
Web Manifest Validator=Web Manifest Validator
webarchiv.cz=webarchiv.cz
web-archive-net.com.bot=web-archive-net.com.bot
WebBot=WebBot
webcompanycrawler=webcompanycrawler
WebCookies=WebCookies
WebCrawler=WebCrawler
Webcron.org crawler=Webcron.org crawler
WebDataStats=WebDataStats
webdepozit.sk crawler=webdepozit.sk crawler
WebGazer=WebGazer
webmon=webmon
weborama crawler=weborama crawler
Webshot=Webshot
websitepulse bot=websitepulse bot
WebSniffer=WebSniffer
WebThumb=WebThumb
WEDOS OnLine monitoring=WEDOS OnLine monitoring
WeSEE=WeSEE
WGETbot=WGETbot
WhatCMS.org bot=WhatCMS.org bot
WhatsApp=WhatsApp
WhoAPI=WhoAPI
WikiDo=WikiDo
Wikipedia crawler=Wikipedia crawler
wmtips.com=wmtips.com
wocbot=wocbot
woobot=woobot
WordPress.com bots=WordPress.com bots
WordupInfoSearch=WordupInfoSearch
woriobot=woriobot
WormlyBot=WormlyBot
wotbox=wotbox
WPSec=WPSec
WURFL ImageEngine=WURFL ImageEngine
WWRanks.com crawler=WWRanks.com crawler
Xenu Link Sleuth=Xenu Link Sleuth
XML Schema Validator=XML Schema Validator
XML Sitemaps Generator=XML Sitemaps Generator
XmlSitemapGenerator=XmlSitemapGenerator
xovibot=xovibot
yacybot=yacybot
Yahoo=Yahoo
Yandex.Metrica=Yandex.Metrica
Yandex.Webmaster=Yandex.Webmaster
YandexAccessibilityBot=YandexAccessibilityBot
YandexBot=YandexBot
YandexImages=YandexImages
YandexMobileBot=YandexMobileBot
yanga=yanga
YellowLabTools=YellowLabTools
Yeti=Yeti
YFF35=YFF35
YioopBot=YioopBot
YisouSpider=YisouSpider
YoozBot=yoozBot
YP crawler=YP crawler
Zabbix=Zabbix
zgrab=zgrab
Zombiebot=Zombiebot
ZoomBot=ZoomBot
Zoominfo=Zoominfo
Zoxh.Com Bot=Zoxh.Com Bot
ZumBot=ZumBot
ZuperlistBot=ZuperlistBot

Fresh Installs of UBB.threads 7.6.2 to 7.7.3 are prepopulated with this list.
The 2014-10-14 List:
Code
008=008
192.com=192.com
200PleaseBot=200PleaseBot
360Spider=360Spider
4seohuntBot=4seohuntBot
50.nu=50.nu
^Nail=^Nail
A6-Indexer=A6-Indexer
abby=abby
Aboundexbot=Aboundexbot
AboutUsBot=AboutUsBot
Abrave Spider=Abrave Spider
Accelobot=Accelobot
AcoonBot=Acoon
AddThis.com=AddThis.com robot
ADmantX=ADmantX
adressendeutschland.de=adressendeutschland.de
AdsBot-Google=AdsBot-Google
AhrefsBot=AhrefsBot
aiHitBot=aiHitBot
akula=akula
alexa site audit=alexa site audit
Alexabot=Alexabot
Amagit.COM=Amagit.COM
amibot=amibot
AMZNKAssocBot=AMZNKAssocBot
AntBot=AntBot
Apercite=Apercite
AportWorm=AportWorm
AraBot=AraBot
arachnode.net=arachnode.net
Arachnophilia=Arachnophilia
archive.org arc=special_archiver
archive.org bot=archive.org_bot
Ask Jeeves=Ask Jeeves
AskQuickly=AskQuickly
Automattic Analytics Crawler=Automattic Analytics Crawler
BabalooSpider=BabalooSpider
backlink-check.de=backlink-check.de
BacklinkCrawler=BacklinkCrawler
Bad-Neighborhood=Bad Neighborhood Header Detector
Bad-Neighborhood=Bad-Neighborhood Link Analyzer
Baiduspider JP=Baiduspider japan
Baiduspider-Image=Baiduspider-image
Baiduspider=Baiduspider
baypup=baypup
BDCbot=BDCbot
BDFetch=BDFetch
BegunAdvertising=BegunAdvertising
bingbot=bingbot
bingbot=bingbot SitemapProbe
BingPreview=BingPreview
bitlybot=bitlybot
biwec=biwec
bixocrawler=adbeat-publisher-description-fetcher
bixocrawler=bixo
bixocrawler=bixocrawler
bixocrawler=bixolabs
bixocrawler=ptd-crawler
bl.uk_lddc_bot=bl.uk_lddc_bot
Blekkobot=Blekkobot
BLEXBot=BLEXBot
BlinkaCrawler=BlinkaCrawler
BlogPulse=BlogPulse
BlogPulse=BlogPulseLive
bnf.fr_bot=bnf.fr_bot
bot-pge.chlooe.com=bot-pge.chlooe.com
bot.wsowner.com=bot.wsowner.com
botmobi=botmobi
BotOnParade=BotOnParade
BrainbruBot=BrainbruBot
Browsershots=Browsershots
BUbiNG=BUbiNG
Butterfly=Butterfly
CamontSpider=CamontSpider
CareerBot=CareerBot
Castabot=Castabot
CatchBot=CatchBot
CCBot=CCBot
CCResearchBot=CCResearchBot
ChangeDetection=ChangeDetection
Charlotte=Charlotte
CirrusExplorer=CirrusExplorer
cityreview=cityreview
classbot=classbot
CligooRobot=CligooRobot
CliqzBot=Cliqz
CloudFlare-AlwaysOnline=CloudFlare-AlwaysOnline
CloudServerMarketSpider=CloudServerMarketSpider
CMS Crawler=CMS Crawler
coccoc=coccoc
COMODOSpider=Comodo
CompSpyBot=CompSpyBot
contentDetection=contentDetection
copyright sheriff=copyright sheriff
CorpusCrawler=CorpusCrawler
Covario-IDS=Covario
crawler for netopian=crawler for netopian
Crawler4j=Crawler4j
CrazyWebCrawler=crazywebcrawler
Crowsnest=Crowsnest
Curious George=Curious George
datagnionbot=datagnionbot
Daumoa=Daumoa
DBLBot=DBLBot
DCPbot=DCPbot
DealGates Bot=DealGates Bot
discoverybot=discobot
discoverybot=discoverybot
DKIMRepBot=DKIMRepBot
dlcbot=dlcbot
dlvr.it=dlvr.it
DNS-Digger-Explorer=DNS-Digger-Explorer
DomainAppender=DomainAppender
DomainDB=DomainDB
DomainTools-HeaderCheck=HTTP Headers Online
DomainTools-LinksCheck=Online Website Link Checker
DomainTools-SitemapGen=Online Sitemap Generator
DotBot=DotBot
dotSemantic=dotSemantic
DripfeedBot=DripfeedBot
drupact=drupact
DuckDuckPreview=DuckDuckPreview
EasouSpider=EasouSpider
EasyBib AutoCite=EasyBib AutoCite
eCairn-Grabber=eCairn-Grabber
eCommerceBot=eCommerceBot
EdisterBot=EdisterBot
Embedly=Embedly
emefgebot=emefgebot
envolk=envolk
Esribot=Esribot
EuripBot=EuripBot
Eurobot=Eurobot
EventGuruBot=EventGuruBot
EveryoneSocialBot=EveryoneSocialBot
EvriNid=EvriNid
Exabot Images=Exabot-Images
Exabot Thumbnails=Exabot-Thumbnails
Exabot=Exabot
Exabot=ExaleadCloudview
Experibot=Experibot
Ezooms=Ezooms
FacebookExternalHit=FacebookExternalHit
facebookPlatform=facebookplatform
factbot=Factbot 1.09
FairShare=FairShare
Falconsbot=Falconsbot
fastbot crawler=fastbot crawler
FauBot=FauBot
FeedCatBot=FeedCatBot
FeedFinder=FeedFinder
Feedly=Feedly
Feedly=FeedlyBot
Fetch-Guess=Fetch
findlinks=findlinks
firmilybot=firmilybot
Flatland Industries Web Spider=flatlandbot
FlightDeckReportsBot=FlightDeckReportsBot
FlipboardProxy=FlipboardProxy
Flocke bot=Flocke bot
FollowSite Bot=FollowSite Bot
Fooooo_Web_Video_Crawl=Fooooo_Web_Video_Crawl
FreeWebMonitoring=FreeWebMonitoring
FyberSpider=FyberSpider
Gaisbot=Gaisbot
GarlikCrawler=GarlikCrawler
GeliyooBot=GeliyooBot
Genieo Web filter=Genieo
GigablastOpenSource=GigablastOpenSource
Gigabot=Gigabot
GingerCrawler=GingerCrawler
Girafabot=Girafabot
gocrawl=gocrawl
GOFORITBOT=GOFORITBOT
gonzo=gonzo
Googlebot AdsBot Mobile=AdsBot-Google-Mobile
Googlebot AdSense=Mediapartners-Google
Googlebot Image=Googlebot-Image
Googlebot Mobile=Googlebot-Mobile
Googlebot Snippet=Googlebot snippet
Googlebot Video=Googlebot-Video
Googlebot Web Preview=Google Web Preview
Googlebot=Googlebot
Grahambot=Grahambot
GrapeshotCrawler=grapeFX
GrapeshotCrawler=GrapeshotCrawler
GurujiBot=GurujiBot
Hailoobot=Hailoobot
HatenaScreenshot=HatenaScreenshot
hawkReader=hawkReader
HeartRails_Capture=HeartRails_Capture
heritrix=heritrix
Holmes=holmes
HolmesBot=HolmesBot
HomeTags=HomeTags
HostTracker.com=HostTracker.com
HostTracker=HostTracker
HuaweiSymantecSpider=HuaweiSymantecSpider
HubSpot Connect=HubSpot Connect
HubSpot Crawler=HubSpot Crawler
HypeStat=HypeStat
ia_archiver=ia_archiver
ICC-Crawler=ICC-Crawler
ichiro=ichiro
iCjobs=iCjobs
IdeelaborPlagiaat=IdeelaborPlagiaat
idmarch=idmarch Automatic.beta
Iframely=Iframely
imbot=imbot
immediatenet thumbnails=immediatenet thumbnails
ImplisenseBot=ImplisenseBot
Impressumscrawler=Impressumscrawler
Influencebot=Influencebot
Infohelfer=Infohelfer
IntegromeDB=IntegromeDB
IstellaBot=IstellaBot
IXEbot=IXEbot
Jabse.com Crawler=Jabse.com Crawler
JadynAveBot=JadynAveBot
JamesBOT=JamesBOT
JikeSpider=JikeSpider
Job Roboter Spider=Job Roboter Spider
JUST-CRAWLER=JUST-CRAWLER
Jyxobot=Jyxobot
Jyxobot=JyxobotRSS
Kalooga=Kalooga
Karneval-Bot=Karneval-Bot
kinshoo=KiNShooboT
KomodiaBot=KomodiaBot
Kraken=Kraken
KrOWLer=KrOWLer
kulturarw=kulturarw3
L.webis=L.webis
Leikibot=Leikibot
LemurWebCrawler=LemurWebCrawler
LexxeBot=LexxeBot
Lijit=Lijit
LinguaBot=LinguaBot
linguatools=linguatools
Linguee Bot=Linguee Bot
Link Valet Online=Link Valet Online
LinkAider=LinkAider
linkdex.com=linkdex.com
linkdexbot=linkdexbot
LinkedInBot=LinkedInBot
LinkWalker=LinkWalker
Lipperhey Spider=Lipperhey Spider
livedoor ScreenShot=livedoor ScreenShot
LivelapBot=LivelapBot
LoadImpactPageAnalyzer=LoadImpactPageAnalyzer
LoadTimeBot=LoadTimeBot
LuminateBot=LuminateBot
magpie-crawler=magpie-crawler
Mail.Ru bot=Mail.Ru
meanpathbot=meanpathbot
MeMoNewsBot=MeMoNewsBot
memoryBot=memoryBot
MetaGeneratorCrawler=MetaGeneratorCrawler
MetaHeadersBot=MetaHeadersBot
MetaJobBot=MetaJobBot
MetamojiCrawler=MetamojiCrawler
Metaspinner=Metaspinner
MetaURI API=MetaURI API
MetaURI=MetaURI
MIA Bot=MIA Bot
MiaDev=MiaDev
MixBot=MixBot
MJ12bot=MJ12bot
MLBot=MLBot
MnoGoSearch=MnoGoSearch
Moatbot=Moatbot
moba-crawler=moba-crawler
MojeekBot=MojeekBot
Motoricerca-Robots.txt-Checker=Motoricerca-Robots.txt-Checker
Mp3Bot=Mp3Bot
MSNBot Media=msnbot-media
MSNBot News=msnbot-NewsBlogs
MSNBot UDiscovery=msnbot-UDiscovery
MSNBot=adidxbot
MSNBot=MSNBot
MSRBOT=MSRBOT
musobot=musobot
Najdi.si=Najdi.si
NalezenCzBot=NalezenCzBot
NaverBot=NaverBot
NaverBot=Yepi
NaverBot=Yeti
NaverBot=Yeti-FeedItemCrawler
NaverBot=Yeti-Mobile
nekstbot=nekstbot
NerdByNature.Bot=NerdByNature.Bot
NerdyBot=NerdyBot
NetcraftSurveyAgent=NetcraftSurveyAgent
netEstate Crawler=netEstate NE Crawler
netEstate Crawler=netEstate RSS crawler
NetResearchServer=nrsbot
Netseer=Netseer crawler
NextGenSearchBot=NextGenSearchBot
Nigma.ru=Nigma.ru
NLNZ_IAHarvester2013=NLNZ_IAHarvester2013
nodestackbot=nodestackbot
Nuhk=Nuhk
Nutch=Nutch
nworm=nwormFeedFinder
Nymesis=Nymesis
oBot=oBot
Ocelli=Ocelli
omgilibot=omgilibot
OoyyoBot=OoyyoBot
Open Web Analytics Bot=Open Web Analytics Bot
OpenCalaisSemanticProxy=OpenCalaisSemanticProxy
OpenindexSpider=OpenindexDeepSpider
OpenindexSpider=OpenindexShalooowSpider
OpenindexSpider=OpenindexSpider
OpenWebSpider=OpenWebSpider
OrgbyBot=OrgbyBot
OsObot=OsObot
ownCloud Server Crawler=ownCloud Server Crawler
Page2RSS=Page2RSS
page_verifier=page_verifier
PagePeeker=PagePeeker
Panscient web crawler=Panscient web crawler
PaperLiBot=PaperLiBot
ParchBot=ParchBot
parsijoo=parsijoo
PayPal IPN=PayPal IPN
Peeplo Screenshot Bot=Peeplo Screenshot Bot
Peepowbot=Peepowbot
peerindex=peerindex
Peew=Peew
PercolateCrawler=percbotspider
PercolateCrawler=PercolateCrawler
pingdom.com_bot=pingdom.com_bot
Pinterest=Pinterest
PiplBot=PiplBot
Pixray-Seeker=Pixray-Seeker
Plukkie=Plukkie
pmoz.info ODP link checker=pmoz.info ODP link checker
Pompos=Pompos
PostPost=PostPost
pr-cy.ru Screenshot Bot=pr-cy.ru
ProCogBot=ProCogBot
ProCogSEOBot=ProCogSEOBot
proximic=proximic
psbot=psbot
psbot=psbot-page
Qirina Hurdler=Qirina Hurdler
Qseero=Qseero
Qualidator.com Bot=Qualidator.com Bot
Qualidator.com SiteAnalyzer=Qualidator.com SiteAnalyzer
QuerySeekerSpider=QuerySeekerSpider
quickobot=quickobot
R6 bot=R6_CommentReader
R6 bot=R6_FeedFetcher
RADaR-Bot=RADaR-Bot
RankurBot=RankurBot
Readability=Readability
Robots_Tester=Robots_Tester
Robozilla=Robozilla
rogerbot=rogerbot
Ronzoobot=Ronzoobot
RSSMicro.com RSS Atom Feed Robot=RSSMicro.com RSS Atom Feed Robot
Ruky-Roboter=Ruky-Roboter
RyzeCrawler=RyzeCrawler
SAI Crawler=SAI Crawler
SanszBot=SanszBot
SBIder=SBIder
SBSearch=SBSearch
Scarlett=Scarlett
SCFCrawler=SCFCrawler
Scooter=Scooter
ScoutJet=ScoutJet
ScoutJet=ScoutJet old
Scrapy=Scrapy
ScreenerBot Crawler=ScreenerBot Crawler
Scrubby=Scrubby
search.KumKie.com=search.KumKie.com
Search17Bot=Search17Bot
SearchmetricsBot=SearchmetricsBot
SecurityResearchBot=SecurityResearchBot
seegnifybot=seebot
seegnifybot=seegnifybot
Semager=Semager
Semantifire=Semantifire1
SemrushBot=SemrushBot
Seobility=Seobility SEO-Check
Seobility=Seobility Urlstat
SEOCentro Keywords=KeywordDensityRobot
SEOCentro MetaTags=MetaTagRobot
SEOCentro SEO Keywords=SEOCentro Page Keyword Analyzer v1.2
SeoCheckBot=SeoCheck
SEODat=SEODat
SEOdiver=SEOdiver
SEOENGBot=SEOENGBot
SEOkicks-Robot=SEOkicks-Robot
Setoozbot=OOZBOT
Setoozbot=SETOOZBOT
SeznamBot=Seznam screenshot-generator
SeznamBot=SeznamBot
SeznamBot=SklikBot
Shareaholicbot=Shareaholicbot
Shelob=Shelob
ShopWiki=ShopWiki
ShowyouBot=ShowyouBot
sistrix=sistrix
SiteCondor=SiteCondor
Sitedomain-Bot=Sitedomain-Bot
Slackbot=Slackbot-LinkExpanding
smart.apnoti.com Robot=smart.apnoti.com Robot
SMTBot=SMTBot
Snapbot=Snapbot
SniffRSS=SniffRSS
socialbm_bot=socialbm_bot
sogou spider=Sogou
SolomonoBot=SolomonoBot
Sosospider=Sosoimagespider
Sosospider=Sosospider
spbot=spbot
Speedy=Speedy Spider
Speedy=Speedy Spider Beta
SpiderLing=SpiderLing
Spinn3r=Spinn3r
SputnikBot=SputnikBot
SSL-Crawler=SSL-Crawler
SSLBot=SSLBot
StackRambler=StackRambler
StatoolsBot=StatoolsBot
Steeler=Steeler
STINGbot=STINGbot
stq_bot=stq_bot
Strokebot=Strokebot
suggybot=suggybot
SurcentroBot=SurcentroBot
Surphace Scout=Surphace Scout
SurveyBot=SurveyBot
SWEBot=SWEBot
SygolBot=SygolBot
Symfony Spider=Symfony Spider
Szukacz=Szukacz
Tagoobot=Tagoobot
taptubot=taptubot
Technoratibot=Technoratibot
Thumbnail.CZ robot=Thumbnail.CZ robot
ThumbShots-Bot=ThumbShots-Bot
thumbshots-de-Bot=thumbshots-de-bot
Thumbshots.ru=Thumbshots.ru
ThumbSniper=ThumbSniper
TinEye=TinEye
TomTom places company search=TomTom places company search
Topicbot=Topicbot
Toread-Crawler=Toread-Crawler
Touche=Touche
trendictionbot=trendictionbot
TurnitinBot=TurnitinBot
TwengaBot=TwengaBot
TwengaBot=TwengaBot-Discover
Twiceler=Twiceler
Twikle=Twikle
Twingly Recon=Twingly Recon
UASlinkChecker=UASlinkChecker
uMBot FC=uMBot-FC
uMBot LN=uMBot-LN
UnisterBot=UnisterBot
UnwindFetchor=UnwindFetchor
Updownerbot=Updownerbot
UptimeDog=UptimeDog
UptimeRobot=UptimeRobot
URLAppendBot=URLAppendBot
urlfan-bot=urlfan-bot
Urlfilebot (Urlbot)=Urlfilebot
Vagabondo=Vagabondo
Vedma=Vedma
VideoSurf_bot=VideoSurf_bot
Visbot=Visbot
VoilaBot=VoilaBot
voltron=voltron
voyager=voyager
WASALive-Bot= WASALive-Bot
WatchMouse=WatchMouse
WBSearchBot=WBSearchBot
Web-Monitoring=Web-Monitoring
Web-sniffer=Web-sniffer
WebCookies=WebCookies
WebCorp=WebCorp
WebImages=WebImages
webinatorbot=webinatorbot
webmastercoffee=webmastercoffee
WebNL=WebNL
WebRankSpider=WebRankSpider
WebTarantula.com Crawler=WebTarantula.com Crawler
WebThumbnail=WebThumbnail
WebWatch Robot_txtChecker=WebWatch
WeSEE:Ads=WeSEE:Ads/PageBot
WeSEE:Ads=WeSEE:Ads/PictureBot
WeSEE:Search=WeSEE
WeSEE:Search=WeSEE:Search
WeViKaBot=WeViKaBot
Whoismindbot=Whoismindbot
WikioFeedBot=WikioFeedBot
wikiwix-bot=wikiwix-bot
Willow Internet Crawler=Willow Internet Crawler
WillyBot=WillyBot
WinWebBot=WinWebBot
WMCAI_robot=WMCAI_robot
Woko=Woko
WordPress.com mShots=WordPress.com mShots
woriobot=woriobot
Wotbox=Wotbox
wsAnalyzer=wsAnalyzer
wscheck.com=wscheck.com
x28-job-bot=x28-job-bot
XmarksFetch=XmarksFetch
XML Sitemaps Generator=XML Sitemaps Generator
XoviBot=XoviBot
XRL=XRL
Yaanb=Yaanb
yacybot=yacybot
Yahoo!=Y!J-BRI
Yahoo!=Y!J-BRJ
Yahoo!=Y!J-BRO
Yahoo!=Y!J-BRW
Yahoo!=Y!J-BSC
Yahoo!=Yahoo! Slurp
Yahoo!=Yahoo-MMCrawler
Yahoo!=YahooCacheSystem
YamanaLab-bot=Sonic
YandexBot AntiVirus=YandexAntivirus
YandexBot Blogs=YandexBlogs
YandexBot Catalog=YandexCatalog
YandexBot Direct=YandexDirect
YandexBot Favicons=YandexFavicons
YandexBot ImageResizer=YandexImageResizer
YandexBot Images=YandexImages
YandexBot Media=YandexMedia
YandexBot Metrix=YandexMetrika
YandexBot News=YandexNews
YandexBot Server=Yandex.Server
YandexBot Something=YandexSomething
YandexBot Video=YandexVideo
YandexBot Webmaster=YandexWebmaster
YandexBot Zakladki=YandexZakladki
YandexBot=Yandex
Yanga=Yanga
YioopBot=gofind
YioopBot=YioopBot
YodaoBot Image=YodaoBot-Image
YodaoBot=YodaoBot
YoudaoBot=YoudaoBot
YowedoBot=YowedoBot
YRSpider=YRSpider
YYSpider=YYSpider
ZeerchBot LA1=ZeerchBot LA1
ZeerchBot LA2=ZeerchBot LA2
ZeerchBot=ZeerchBot
Zookabot=Zookabot
ZumBot=ZumBot

Fresh Installs prior to UBB.threads 7.6.2 do not come prepopulated with a Search Engine Agents list.

Last edited by isaac; 01/14/2020 12:03 AM. Reason: Updated the list for 2020-01-14

Current developer of UBB.threads PHP Forum Software // 7.7.4 released!
isaac @ id242.com // my forum @ CelicaHobby.com
2 members like this: isaac, Gizmo
Sponsored Links
Re: [7.x] Robots (Crawlers) List for UBB.threads
isaac #320567 06/25/2014 5:12 AM
Joined: Jul 2001
Posts: 1,072
Likes: 39
isaac Online Happy OP
coffee and code
OP Online Happy
coffee and code
Joined: Jul 2001
Posts: 1,072
Likes: 39
Although the source information for you to create your own updates and conversions is in the OP, I plan on updating this post every couple of months, making your job as an UBBT forum admin much easier.

---
EDIT: user-agent-string.info no longer provides a list of user agent strings without a subscription. this means that until another resource for this data is found, this list will remain as it currently is.

Last edited by id242; 11/02/2015 10:39 PM.

Current developer of UBB.threads PHP Forum Software // 7.7.4 released!
isaac @ id242.com // my forum @ CelicaHobby.com
1 member likes this: Gizmo
Re: [7.x] Robots (Crawlers) List for UBB.threads
isaac #320568 06/25/2014 5:47 AM
Joined: Dec 2001
Posts: 87
Power User
Offline
Power User
Joined: Dec 2001
Posts: 87
This is absolutely awesome!!! Many, many thanks from all of us.


Bill Barker
Issaquah, Wa
Re: [7.x] Robots (Crawlers) List for UBB.threads
isaac #320584 08/03/2014 11:20 PM
Joined: Aug 2014
Posts: 5
Lurker
Offline
Lurker
Joined: Aug 2014
Posts: 5
This is superb. I only ever had about 6 lines in there. That's so comprehensive as to be unbelievable.

Thank you.

Re: [7.x] Robots (Crawlers) List for UBB.threads
isaac #320618 10/14/2014 11:57 AM
Joined: Jul 2001
Posts: 1,072
Likes: 39
isaac Online Happy OP
coffee and code
OP Online Happy
coffee and code
Joined: Jul 2001
Posts: 1,072
Likes: 39
This list has been updated 2014-10-14.


Current developer of UBB.threads PHP Forum Software // 7.7.4 released!
isaac @ id242.com // my forum @ CelicaHobby.com
1 member likes this: Gizmo
Sponsored Links
Re: [7.x] Robots (Crawlers) List for UBB.threads
isaac #320629 10/30/2014 3:10 PM
Joined: Oct 2010
Posts: 6
Lurker
Offline
Lurker
Joined: Oct 2010
Posts: 6
Thanks again.. This really makes the bot display more accurate. I just saw some bots that we from a company that I trusted... ouch.

Re: [7.x] Robots (Crawlers) List for UBB.threads
isaac #320971 06/13/2015 8:15 AM
Joined: Feb 2007
Posts: 10
Newbie
Offline
Newbie
Joined: Feb 2007
Posts: 10
In my "anonymous" list, I see a number of IPs like this: 157.55.39.xxx. Hovering over the "i" icon it shows:
Agent: Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)

The list above has bingbot entries, and I added one from an older list, so I have this in my search engine agents list:
Code
bingbot=bingbot
bingbot=bingbot/2.0
bingbot=bingbot SitemapProbe
BingPreview=BingPreview
In spite of that, bingbot continues to stay in the anonymous group. Is there a bug, or something I can do to fix it?

Re: [7.x] Robots (Crawlers) List for UBB.threads
isaac #320972 06/13/2015 9:03 AM
Joined: Jul 2001
Posts: 1,072
Likes: 39
isaac Online Happy OP
coffee and code
OP Online Happy
coffee and code
Joined: Jul 2001
Posts: 1,072
Likes: 39
bingbot=bingbot
bingbot=bingbot/2.0
bingbot=bingbot SitemapProbe

Are all the same thing.

USAGE: SearchEngineName=AgentBotString

The "AgentBotString" on the right side of the equation will search the whole "Agent" and return the matching "SearchEngineName"

So basically, if only list "bingbot=bingbot", you will cover all the other variations of "AgentBotString" for Microsoft's Bing spider/bot.


Current developer of UBB.threads PHP Forum Software // 7.7.4 released!
isaac @ id242.com // my forum @ CelicaHobby.com
Re: [7.x] Robots (Crawlers) List for UBB.threads
isaac #320973 06/13/2015 10:11 AM
Joined: Feb 2007
Posts: 10
Newbie
Offline
Newbie
Joined: Feb 2007
Posts: 10
Ok, thanks. "Bing" shows up on Who's online on this forum, so why do they show as anonymous users on mine? (I'm at v 7.5.8, will move to 7.5.9 soon.)

Re: [7.x] Robots (Crawlers) List for UBB.threads
isaac #320974 06/13/2015 10:17 AM
Joined: Jul 2001
Posts: 1,072
Likes: 39
isaac Online Happy OP
coffee and code
OP Online Happy
coffee and code
Joined: Jul 2001
Posts: 1,072
Likes: 39
If an unregistered user arrives at your site through a search using bing, they will be clasified as anonymous. if your site is being crawled by bingbot (not a live person), it should be classified and shown within the spider section.

Sounds like you have visitors finding your site through Bing. This is good!


Current developer of UBB.threads PHP Forum Software // 7.7.4 released!
isaac @ id242.com // my forum @ CelicaHobby.com
Sponsored Links
Re: [7.x] Robots (Crawlers) List for UBB.threads
isaac #320975 06/13/2015 10:50 AM
Joined: Feb 2007
Posts: 10
Newbie
Offline
Newbie
Joined: Feb 2007
Posts: 10
Originally Posted by id242
Sounds like you have visitors finding your site through Bing. This is good!
I am pretty sure that is not the case.
My Who's Online shows these anonymous guests:
157.55.39.120
157.55.39.9
157.55.39.232
157.55.39.218
157.55.39.231
157.55.39.224
Each one shows bingbot when I hover over the "i" icon, and the Referrer: part is blank.


Re: [7.x] Robots (Crawlers) List for UBB.threads
isaac #320976 06/13/2015 11:46 AM
Joined: Jan 2000
Posts: 5,805
Likes: 10
UBB.Dev / UBB.Wiki Owner
Time Lord
Online Tapedshut
UBB.Dev / UBB.Wiki Owner
Time Lord
Joined: Jan 2000
Posts: 5,805
Likes: 10
FWIW, 157.55.39 is owned by Microsoft, just run the IP's through Domain Tools


UBB.Dev - Putting Dev into UBB.threads
Company: VNC Web Services - UBB.threads Scripts and Scripting, Install and Upgrade Services, Site and Server Maintenance.
Forums: A Gardeners Forum, I Find Rocks, Scouters World, and UGN Security
UBB.Threads: My UBB Themes, My UBB Scripts
Re: [7.x] Robots (Crawlers) List for UBB.threads
Gizmo #320979 06/14/2015 3:33 PM
Joined: Feb 2007
Posts: 10
Newbie
Offline
Newbie
Joined: Feb 2007
Posts: 10
Originally Posted by Gizmo
FWIW, 157.55.39 is owned by Microsoft, just run the IP's through Domain Tools
Yes, Microsoft bingbot. Which brings me back to the original question. Why are six of those showing up as anonymous guests in my Who's Online list.

I would like to move them down into the Search Spiders section, but that is not happening. Any recommendations?

Re: [7.x] Robots (Crawlers) List for UBB.threads
isaac #320980 06/14/2015 4:24 PM
Joined: Jan 2000
Posts: 5,805
Likes: 10
UBB.Dev / UBB.Wiki Owner
Time Lord
Online Tapedshut
UBB.Dev / UBB.Wiki Owner
Time Lord
Joined: Jan 2000
Posts: 5,805
Likes: 10
Well, if the referrer listing is blank, there isn't much you can do; as the WoL system just parses the spider data based on what is being supplied by bots IN the referrer variable.


UBB.Dev - Putting Dev into UBB.threads
Company: VNC Web Services - UBB.threads Scripts and Scripting, Install and Upgrade Services, Site and Server Maintenance.
Forums: A Gardeners Forum, I Find Rocks, Scouters World, and UGN Security
UBB.Threads: My UBB Themes, My UBB Scripts
Re: [7.x] Robots (Crawlers) List for UBB.threads
Gizmo #320981 06/14/2015 4:57 PM
Joined: Feb 2007
Posts: 10
Newbie
Offline
Newbie
Joined: Feb 2007
Posts: 10
Originally Posted by Gizmo
Well, if the referrer listing is blank, there isn't much you can do; as the WoL system just parses the spider data based on what is being supplied by bots IN the referrer variable.
The referrer listing is blank on all the Search Spider entries, and is blank on all the bingbot "guest" entries. So I don't understand ho the referrer being blank causes it to show up in the guest area.

I do appreciate your taking the time to post in this dialog. I am mostly a "grasshopper" here.

Re: [7.x] Robots (Crawlers) List for UBB.threads
isaac #320982 06/15/2015 4:40 AM
Joined: Jan 2000
Posts: 5,805
Likes: 10
UBB.Dev / UBB.Wiki Owner
Time Lord
Online Tapedshut
UBB.Dev / UBB.Wiki Owner
Time Lord
Joined: Jan 2000
Posts: 5,805
Likes: 10
I talked with Isaac last night and evidently when the user is viewing a "cached result" from Bing there is no referrer variable passed as it's not "BingBot", but a user that's requesting data through a Bing server.


UBB.Dev - Putting Dev into UBB.threads
Company: VNC Web Services - UBB.threads Scripts and Scripting, Install and Upgrade Services, Site and Server Maintenance.
Forums: A Gardeners Forum, I Find Rocks, Scouters World, and UGN Security
UBB.Threads: My UBB Themes, My UBB Scripts
Re: [7.x] Robots (Crawlers) List for UBB.threads
isaac #320983 06/15/2015 4:18 PM
Joined: Feb 2007
Posts: 10
Newbie
Offline
Newbie
Joined: Feb 2007
Posts: 10
You're giving Bing waaay too much credit. Right now, I have 11 guests, 5 search spiders, and one user. Normal for this small forum with very little activity at night.

Of the 11 guests, 5 are bingbot. And 3 of them are walking this silly thread that has 99 pages. Long continuing threads confound the search spiders. Every time there is a new post, they spend a long time walking through every page in the sequence of pages.

Re: [7.x] Robots (Crawlers) List for UBB.threads
isaac #320984 06/16/2015 7:49 AM
Joined: Jan 2000
Posts: 5,805
Likes: 10
UBB.Dev / UBB.Wiki Owner
Time Lord
Online Tapedshut
UBB.Dev / UBB.Wiki Owner
Time Lord
Joined: Jan 2000
Posts: 5,805
Likes: 10
Well, not really giving them too much credit; they force SSL for all queries now (source), so it could really be either incoming users from bing are coming in on their SSL (which is the default) or they're coming in from the cache.


UBB.Dev - Putting Dev into UBB.threads
Company: VNC Web Services - UBB.threads Scripts and Scripting, Install and Upgrade Services, Site and Server Maintenance.
Forums: A Gardeners Forum, I Find Rocks, Scouters World, and UGN Security
UBB.Threads: My UBB Themes, My UBB Scripts
Re: [7.x] Robots (Crawlers) List for UBB.threads
Gizmo #321283 11/02/2015 4:57 PM
Joined: Jul 2001
Posts: 1,072
Likes: 39
isaac Online Happy OP
coffee and code
OP Online Happy
coffee and code
Joined: Jul 2001
Posts: 1,072
Likes: 39
Some further reading regarding http/https referer data:
https://yoast.com/web-https/


Current developer of UBB.threads PHP Forum Software // 7.7.4 released!
isaac @ id242.com // my forum @ CelicaHobby.com
1 member likes this: Gizmo
Re: [7.x] Robots (Crawlers) List for UBB.threads
isaac #321284 11/02/2015 10:21 PM
Joined: Mar 2007
Posts: 15
Newbie
Offline
Newbie
Joined: Mar 2007
Posts: 15
At the moment, I have FOUR of the anonymous Bingbots, and EIGHTEEN from 72.21.217.XXX (Amazon).

Of course, that pales in comparison to the THIRTY FOUR properly identified Baidu spiders.

Last edited by SteveS; 11/02/2015 10:24 PM.
Re: [7.x] Robots (Crawlers) List for UBB.threads
isaac #321285 11/02/2015 10:31 PM
Joined: Jul 2001
Posts: 1,072
Likes: 39
isaac Online Happy OP
coffee and code
OP Online Happy
coffee and code
Joined: Jul 2001
Posts: 1,072
Likes: 39
SteveS, are you confident your Amazon stuff is not related caching of content within your site by AWS Cloud Computing /Route 53? https://aws.amazon.com/

Additional reading at:
https://en.wikipedia.org/wiki/Amazon_Route_53

The IP 72.21.217.n has been used as a proxy for a User Agent of MSIE-6, which is in itself highly deprecated. Headers can also be consistent with either a battened-down proxy or a bot.

Additional reading at:
"amazonaws.com plays host to wide variety of bad bots"
https://www.webmasterworld.com/search_engine_spiders/3828718.htm


Current developer of UBB.threads PHP Forum Software // 7.7.4 released!
isaac @ id242.com // my forum @ CelicaHobby.com
1 member likes this: Gizmo
Re: [7.x] Robots (Crawlers) List for UBB.threads
isaac #321286 11/02/2015 10:39 PM
Joined: Mar 2007
Posts: 15
Newbie
Offline
Newbie
Joined: Mar 2007
Posts: 15
I am not confident in anything, but I do a lot with Amazon, so I figured they were "botting" for that.

Re: [7.x] Robots (Crawlers) List for UBB.threads
isaac #323385 11/16/2019 3:58 PM
Joined: Jul 2001
Posts: 1,072
Likes: 39
isaac Online Happy OP
coffee and code
OP Online Happy
coffee and code
Joined: Jul 2001
Posts: 1,072
Likes: 39
Changelog 2019-11-16
The Robots (Crawlers) List for UBB.threads has been updated in OP


Current developer of UBB.threads PHP Forum Software // 7.7.4 released!
isaac @ id242.com // my forum @ CelicaHobby.com
Re: [7.x] Robots (Crawlers) List for UBB.threads
isaac #323441 01/13/2020 10:20 PM
Joined: Jul 2001
Posts: 1,072
Likes: 39
isaac Online Happy OP
coffee and code
OP Online Happy
coffee and code
Joined: Jul 2001
Posts: 1,072
Likes: 39
Changelog 2020-01-14
The Robots (Crawlers) List for UBB.threads has been updated in OP


Current developer of UBB.threads PHP Forum Software // 7.7.4 released!
isaac @ id242.com // my forum @ CelicaHobby.com
1 member likes this: Morgan

Link Copied to Clipboard
Donate Today!
Donate via PayPal

Donate to UBBDev today to help aid in Operational, Server and Script Maintenance, and Development costs.

Please also see our parent organization VNC Web Services if you're in the need of a new UBB.threads Install or Upgrade, Site/Server Migrations, or Security and Coding Services.
Recommended Hosts
We have personally worked with and recommend the following Web Hosts:
Stable Host
bluehost
InterServer
Visit us on Facebook
Member Spotlight
Bill B
Bill B
Issaquah, WA
Posts: 87
Joined: December 2001
Forum Statistics
Forums63
Topics37,543
Posts293,776
Members13,837
Most Online5,166
Sep 15th, 2019
Today's Statistics
Currently Online 328
Topics Created 0
Posts Made 0
Users Online 2
Birthdays 5
Top Posters
AllenAyres 21,079
JoshPet 10,369
LK 7,394
Lord Dexter 6,708
Gizmo 5,805
Greg Hard 4,625
Top Posters(30 Days)
isaac 2
Top Likes Received
isaac 39
Gizmo 10
Brett 7
Morgan 2
Top Likes Received (30 Days)
None yet
The UBB.Developers Network (UBB.Dev/Threads.Dev) is ©2000-2020 VNC Web Services

 
Powered by UBB.threads™ PHP Forum Software 7.7.5
(Snapshot build 20200802.dev)
Responsive Width:

PHP: 5.4.45 Page Time: 0.074s Queries: 69 (0.026s) Memory: 3.5147 MB (Peak: 3.9024 MB) Data Comp: Zlib Server Time: 2020-08-06 00:51:31 UTC
Valid HTML 5 and Valid CSS