Previous Thread
Next Thread
Print Thread
Rate Thread
[7.x] List of Search Engine Spiders for UBBThreads #320566
06/25/2014 4:45 AM
06/25/2014 4:45 AM
Joined: Jul 2001
Posts: 1,170
California
isaac Offline OP
$coffee=code(true);
isaac  Offline OP
$coffee=code(true);
Joined: Jul 2001
Posts: 1,170
California
List of Search Engine Spiders for UBB.Threads - UPDATED 2014-10-14

About:
The advantage to using this list is that Search Engine Spiders get put into the correct "Search Spiders" group when viewing the forums/ubbthreads.php/online page.

This translation list is not used anywhere else in UBBT except in the forums/ubbthreads.php/online page. Having a long list of Search Engine Spiders will not slow down your board -- the only exception is that depending on the server in which your site is hosted on, there may be a minimal delay in generating the "online" page.

Details:
Total Agent Strings: 557
Original Sourced From: http://user-agent-string.info/list-of-ua/bots
Source Version: uas_20141013-01

I've cleaned and converted it for use with UBB.Threads. I've also removed the version numvers from each bot string. This should catch current and future updates which the bot owner may make to their individual bot.

How-To:
1) COPY/PASTE this list to the "Search Engine Agents" box at: Control Panel > Display Options > Who's Online
2) Click "Update General Display Options"
done.

Having problems? REMOVE any blank lines from the top/bottom of your copied list.


The List:
Code
008=008
192.com=192.com
200PleaseBot=200PleaseBot
360Spider=360Spider
4seohuntBot=4seohuntBot
50.nu=50.nu
^Nail=^Nail
A6-Indexer=A6-Indexer
abby=abby
Aboundexbot=Aboundexbot
AboutUsBot=AboutUsBot
Abrave Spider=Abrave Spider
Accelobot=Accelobot
AcoonBot=Acoon
AddThis.com=AddThis.com robot
ADmantX=ADmantX
adressendeutschland.de=adressendeutschland.de
AdsBot-Google=AdsBot-Google
AhrefsBot=AhrefsBot
aiHitBot=aiHitBot
akula=akula
alexa site audit=alexa site audit
Alexabot=Alexabot
Amagit.COM=Amagit.COM
amibot=amibot
AMZNKAssocBot=AMZNKAssocBot
AntBot=AntBot
Apercite=Apercite
AportWorm=AportWorm
AraBot=AraBot
arachnode.net=arachnode.net
Arachnophilia=Arachnophilia
archive.org arc=special_archiver
archive.org bot=archive.org_bot
Ask Jeeves=Ask Jeeves
AskQuickly=AskQuickly
Automattic Analytics Crawler=Automattic Analytics Crawler
BabalooSpider=BabalooSpider
backlink-check.de=backlink-check.de
BacklinkCrawler=BacklinkCrawler
Bad-Neighborhood=Bad Neighborhood Header Detector
Bad-Neighborhood=Bad-Neighborhood Link Analyzer
Baiduspider JP=Baiduspider japan
Baiduspider-Image=Baiduspider-image
Baiduspider=Baiduspider
baypup=baypup
BDCbot=BDCbot
BDFetch=BDFetch
BegunAdvertising=BegunAdvertising
bingbot=bingbot
bingbot=bingbot SitemapProbe
BingPreview=BingPreview
bitlybot=bitlybot
biwec=biwec
bixocrawler=adbeat-publisher-description-fetcher
bixocrawler=bixo
bixocrawler=bixocrawler
bixocrawler=bixolabs
bixocrawler=ptd-crawler
bl.uk_lddc_bot=bl.uk_lddc_bot
Blekkobot=Blekkobot
BLEXBot=BLEXBot
BlinkaCrawler=BlinkaCrawler
BlogPulse=BlogPulse
BlogPulse=BlogPulseLive
bnf.fr_bot=bnf.fr_bot
bot-pge.chlooe.com=bot-pge.chlooe.com
bot.wsowner.com=bot.wsowner.com
botmobi=botmobi
BotOnParade=BotOnParade
BrainbruBot=BrainbruBot
Browsershots=Browsershots
BUbiNG=BUbiNG
Butterfly=Butterfly
CamontSpider=CamontSpider
CareerBot=CareerBot
Castabot=Castabot
CatchBot=CatchBot
CCBot=CCBot
CCResearchBot=CCResearchBot
ChangeDetection=ChangeDetection
Charlotte=Charlotte
CirrusExplorer=CirrusExplorer
cityreview=cityreview
classbot=classbot
CligooRobot=CligooRobot
CliqzBot=Cliqz
CloudFlare-AlwaysOnline=CloudFlare-AlwaysOnline
CloudServerMarketSpider=CloudServerMarketSpider
CMS Crawler=CMS Crawler
coccoc=coccoc
COMODOSpider=Comodo
CompSpyBot=CompSpyBot
contentDetection=contentDetection
copyright sheriff=copyright sheriff
CorpusCrawler=CorpusCrawler
Covario-IDS=Covario
crawler for netopian=crawler for netopian
Crawler4j=Crawler4j
CrazyWebCrawler=crazywebcrawler
Crowsnest=Crowsnest
Curious George=Curious George
datagnionbot=datagnionbot
Daumoa=Daumoa
DBLBot=DBLBot
DCPbot=DCPbot
DealGates Bot=DealGates Bot
discoverybot=discobot
discoverybot=discoverybot
DKIMRepBot=DKIMRepBot
dlcbot=dlcbot
dlvr.it=dlvr.it
DNS-Digger-Explorer=DNS-Digger-Explorer
DomainAppender=DomainAppender
DomainDB=DomainDB
DomainTools-HeaderCheck=HTTP Headers Online
DomainTools-LinksCheck=Online Website Link Checker
DomainTools-SitemapGen=Online Sitemap Generator
DotBot=DotBot
dotSemantic=dotSemantic
DripfeedBot=DripfeedBot
drupact=drupact
DuckDuckPreview=DuckDuckPreview
EasouSpider=EasouSpider
EasyBib AutoCite=EasyBib AutoCite
eCairn-Grabber=eCairn-Grabber
eCommerceBot=eCommerceBot
EdisterBot=EdisterBot
Embedly=Embedly
emefgebot=emefgebot
envolk=envolk
Esribot=Esribot
EuripBot=EuripBot
Eurobot=Eurobot
EventGuruBot=EventGuruBot
EveryoneSocialBot=EveryoneSocialBot
EvriNid=EvriNid
Exabot Images=Exabot-Images
Exabot Thumbnails=Exabot-Thumbnails
Exabot=Exabot
Exabot=ExaleadCloudview
Experibot=Experibot
Ezooms=Ezooms
FacebookExternalHit=FacebookExternalHit
facebookPlatform=facebookplatform
factbot=Factbot 1.09
FairShare=FairShare
Falconsbot=Falconsbot
fastbot crawler=fastbot crawler
FauBot=FauBot
FeedCatBot=FeedCatBot
FeedFinder=FeedFinder
Feedly=Feedly
Feedly=FeedlyBot
Fetch-Guess=Fetch
findlinks=findlinks
firmilybot=firmilybot
Flatland Industries Web Spider=flatlandbot
FlightDeckReportsBot=FlightDeckReportsBot
FlipboardProxy=FlipboardProxy
Flocke bot=Flocke bot
FollowSite Bot=FollowSite Bot
Fooooo_Web_Video_Crawl=Fooooo_Web_Video_Crawl
FreeWebMonitoring=FreeWebMonitoring
FyberSpider=FyberSpider
Gaisbot=Gaisbot
GarlikCrawler=GarlikCrawler
GeliyooBot=GeliyooBot
Genieo Web filter=Genieo
GigablastOpenSource=GigablastOpenSource
Gigabot=Gigabot
GingerCrawler=GingerCrawler
Girafabot=Girafabot
gocrawl=gocrawl
GOFORITBOT=GOFORITBOT
gonzo=gonzo
Googlebot AdsBot Mobile=AdsBot-Google-Mobile
Googlebot AdSense=Mediapartners-Google
Googlebot Image=Googlebot-Image
Googlebot Mobile=Googlebot-Mobile
Googlebot Snippet=Googlebot snippet
Googlebot Video=Googlebot-Video
Googlebot Web Preview=Google Web Preview
Googlebot=Googlebot
Grahambot=Grahambot
GrapeshotCrawler=grapeFX
GrapeshotCrawler=GrapeshotCrawler
GurujiBot=GurujiBot
Hailoobot=Hailoobot
HatenaScreenshot=HatenaScreenshot
hawkReader=hawkReader
HeartRails_Capture=HeartRails_Capture
heritrix=heritrix
Holmes=holmes
HolmesBot=HolmesBot
HomeTags=HomeTags
HostTracker.com=HostTracker.com
HostTracker=HostTracker
HuaweiSymantecSpider=HuaweiSymantecSpider
HubSpot Connect=HubSpot Connect
HubSpot Crawler=HubSpot Crawler
HypeStat=HypeStat
ia_archiver=ia_archiver
ICC-Crawler=ICC-Crawler
ichiro=ichiro
iCjobs=iCjobs
IdeelaborPlagiaat=IdeelaborPlagiaat
idmarch=idmarch Automatic.beta
Iframely=Iframely
imbot=imbot
immediatenet thumbnails=immediatenet thumbnails
ImplisenseBot=ImplisenseBot
Impressumscrawler=Impressumscrawler
Influencebot=Influencebot
Infohelfer=Infohelfer
IntegromeDB=IntegromeDB
IstellaBot=IstellaBot
IXEbot=IXEbot
Jabse.com Crawler=Jabse.com Crawler
JadynAveBot=JadynAveBot
JamesBOT=JamesBOT
JikeSpider=JikeSpider
Job Roboter Spider=Job Roboter Spider
JUST-CRAWLER=JUST-CRAWLER
Jyxobot=Jyxobot
Jyxobot=JyxobotRSS
Kalooga=Kalooga
Karneval-Bot=Karneval-Bot
kinshoo=KiNShooboT
KomodiaBot=KomodiaBot
Kraken=Kraken
KrOWLer=KrOWLer
kulturarw=kulturarw3
L.webis=L.webis
Leikibot=Leikibot
LemurWebCrawler=LemurWebCrawler
LexxeBot=LexxeBot
Lijit=Lijit
LinguaBot=LinguaBot
linguatools=linguatools
Linguee Bot=Linguee Bot
Link Valet Online=Link Valet Online
LinkAider=LinkAider
linkdex.com=linkdex.com
linkdexbot=linkdexbot
LinkedInBot=LinkedInBot
LinkWalker=LinkWalker
Lipperhey Spider=Lipperhey Spider
livedoor ScreenShot=livedoor ScreenShot
LivelapBot=LivelapBot
LoadImpactPageAnalyzer=LoadImpactPageAnalyzer
LoadTimeBot=LoadTimeBot
LuminateBot=LuminateBot
magpie-crawler=magpie-crawler
Mail.Ru bot=Mail.Ru
meanpathbot=meanpathbot
MeMoNewsBot=MeMoNewsBot
memoryBot=memoryBot
MetaGeneratorCrawler=MetaGeneratorCrawler
MetaHeadersBot=MetaHeadersBot
MetaJobBot=MetaJobBot
MetamojiCrawler=MetamojiCrawler
Metaspinner=Metaspinner
MetaURI API=MetaURI API
MetaURI=MetaURI
MIA Bot=MIA Bot
MiaDev=MiaDev
MixBot=MixBot
MJ12bot=MJ12bot
MLBot=MLBot
MnoGoSearch=MnoGoSearch
Moatbot=Moatbot
moba-crawler=moba-crawler
MojeekBot=MojeekBot
Motoricerca-Robots.txt-Checker=Motoricerca-Robots.txt-Checker
Mp3Bot=Mp3Bot
MSNBot Media=msnbot-media
MSNBot News=msnbot-NewsBlogs
MSNBot UDiscovery=msnbot-UDiscovery
MSNBot=adidxbot
MSNBot=MSNBot
MSRBOT=MSRBOT
musobot=musobot
Najdi.si=Najdi.si
NalezenCzBot=NalezenCzBot
NaverBot=NaverBot
NaverBot=Yepi
NaverBot=Yeti
NaverBot=Yeti-FeedItemCrawler
NaverBot=Yeti-Mobile
nekstbot=nekstbot
NerdByNature.Bot=NerdByNature.Bot
NerdyBot=NerdyBot
NetcraftSurveyAgent=NetcraftSurveyAgent
netEstate Crawler=netEstate NE Crawler
netEstate Crawler=netEstate RSS crawler
NetResearchServer=nrsbot
Netseer=Netseer crawler
NextGenSearchBot=NextGenSearchBot
Nigma.ru=Nigma.ru
NLNZ_IAHarvester2013=NLNZ_IAHarvester2013
nodestackbot=nodestackbot
Nuhk=Nuhk
Nutch=Nutch
nworm=nwormFeedFinder
Nymesis=Nymesis
oBot=oBot
Ocelli=Ocelli
omgilibot=omgilibot
OoyyoBot=OoyyoBot
Open Web Analytics Bot=Open Web Analytics Bot
OpenCalaisSemanticProxy=OpenCalaisSemanticProxy
OpenindexSpider=OpenindexDeepSpider
OpenindexSpider=OpenindexShalooowSpider
OpenindexSpider=OpenindexSpider
OpenWebSpider=OpenWebSpider
OrgbyBot=OrgbyBot
OsObot=OsObot
ownCloud Server Crawler=ownCloud Server Crawler
Page2RSS=Page2RSS
page_verifier=page_verifier
PagePeeker=PagePeeker
Panscient web crawler=Panscient web crawler
PaperLiBot=PaperLiBot
ParchBot=ParchBot
parsijoo=parsijoo
PayPal IPN=PayPal IPN
Peeplo Screenshot Bot=Peeplo Screenshot Bot
Peepowbot=Peepowbot
peerindex=peerindex
Peew=Peew
PercolateCrawler=percbotspider
PercolateCrawler=PercolateCrawler
pingdom.com_bot=pingdom.com_bot
Pinterest=Pinterest
PiplBot=PiplBot
Pixray-Seeker=Pixray-Seeker
Plukkie=Plukkie
pmoz.info ODP link checker=pmoz.info ODP link checker
Pompos=Pompos
PostPost=PostPost
pr-cy.ru Screenshot Bot=pr-cy.ru
ProCogBot=ProCogBot
ProCogSEOBot=ProCogSEOBot
proximic=proximic
psbot=psbot
psbot=psbot-page
Qirina Hurdler=Qirina Hurdler
Qseero=Qseero
Qualidator.com Bot=Qualidator.com Bot
Qualidator.com SiteAnalyzer=Qualidator.com SiteAnalyzer
QuerySeekerSpider=QuerySeekerSpider
quickobot=quickobot
R6 bot=R6_CommentReader
R6 bot=R6_FeedFetcher
RADaR-Bot=RADaR-Bot
RankurBot=RankurBot
Readability=Readability
Robots_Tester=Robots_Tester
Robozilla=Robozilla
rogerbot=rogerbot
Ronzoobot=Ronzoobot
RSSMicro.com RSS Atom Feed Robot=RSSMicro.com RSS Atom Feed Robot
Ruky-Roboter=Ruky-Roboter
RyzeCrawler=RyzeCrawler
SAI Crawler=SAI Crawler
SanszBot=SanszBot
SBIder=SBIder
SBSearch=SBSearch
Scarlett=Scarlett
SCFCrawler=SCFCrawler
Scooter=Scooter
ScoutJet=ScoutJet
ScoutJet=ScoutJet old
Scrapy=Scrapy
ScreenerBot Crawler=ScreenerBot Crawler
Scrubby=Scrubby
search.KumKie.com=search.KumKie.com
Search17Bot=Search17Bot
SearchmetricsBot=SearchmetricsBot
SecurityResearchBot=SecurityResearchBot
seegnifybot=seebot
seegnifybot=seegnifybot
Semager=Semager
Semantifire=Semantifire1
SemrushBot=SemrushBot
Seobility=Seobility SEO-Check
Seobility=Seobility Urlstat
SEOCentro Keywords=KeywordDensityRobot
SEOCentro MetaTags=MetaTagRobot
SEOCentro SEO Keywords=SEOCentro Page Keyword Analyzer v1.2
SeoCheckBot=SeoCheck
SEODat=SEODat
SEOdiver=SEOdiver
SEOENGBot=SEOENGBot
SEOkicks-Robot=SEOkicks-Robot
Setoozbot=OOZBOT
Setoozbot=SETOOZBOT
SeznamBot=Seznam screenshot-generator
SeznamBot=SeznamBot
SeznamBot=SklikBot
Shareaholicbot=Shareaholicbot
Shelob=Shelob
ShopWiki=ShopWiki
ShowyouBot=ShowyouBot
sistrix=sistrix
SiteCondor=SiteCondor
Sitedomain-Bot=Sitedomain-Bot
Slackbot=Slackbot-LinkExpanding
smart.apnoti.com Robot=smart.apnoti.com Robot
SMTBot=SMTBot
Snapbot=Snapbot
SniffRSS=SniffRSS
socialbm_bot=socialbm_bot
sogou spider=Sogou
SolomonoBot=SolomonoBot
Sosospider=Sosoimagespider
Sosospider=Sosospider
spbot=spbot
Speedy=Speedy Spider
Speedy=Speedy Spider Beta
SpiderLing=SpiderLing
Spinn3r=Spinn3r
SputnikBot=SputnikBot
SSL-Crawler=SSL-Crawler
SSLBot=SSLBot
StackRambler=StackRambler
StatoolsBot=StatoolsBot
Steeler=Steeler
STINGbot=STINGbot
stq_bot=stq_bot
Strokebot=Strokebot
suggybot=suggybot
SurcentroBot=SurcentroBot
Surphace Scout=Surphace Scout
SurveyBot=SurveyBot
SWEBot=SWEBot
SygolBot=SygolBot
Symfony Spider=Symfony Spider
Szukacz=Szukacz
Tagoobot=Tagoobot
taptubot=taptubot
Technoratibot=Technoratibot
Thumbnail.CZ robot=Thumbnail.CZ robot
ThumbShots-Bot=ThumbShots-Bot
thumbshots-de-Bot=thumbshots-de-bot
Thumbshots.ru=Thumbshots.ru
ThumbSniper=ThumbSniper
TinEye=TinEye
TomTom places company search=TomTom places company search
Topicbot=Topicbot
Toread-Crawler=Toread-Crawler
Touche=Touche
trendictionbot=trendictionbot
TurnitinBot=TurnitinBot
TwengaBot=TwengaBot
TwengaBot=TwengaBot-Discover
Twiceler=Twiceler
Twikle=Twikle
Twingly Recon=Twingly Recon
UASlinkChecker=UASlinkChecker
uMBot FC=uMBot-FC
uMBot LN=uMBot-LN
UnisterBot=UnisterBot
UnwindFetchor=UnwindFetchor
Updownerbot=Updownerbot
UptimeDog=UptimeDog
UptimeRobot=UptimeRobot
URLAppendBot=URLAppendBot
urlfan-bot=urlfan-bot
Urlfilebot (Urlbot)=Urlfilebot
Vagabondo=Vagabondo
Vedma=Vedma
VideoSurf_bot=VideoSurf_bot
Visbot=Visbot
VoilaBot=VoilaBot
voltron=voltron
voyager=voyager
WASALive-Bot= WASALive-Bot
WatchMouse=WatchMouse
WBSearchBot=WBSearchBot
Web-Monitoring=Web-Monitoring
Web-sniffer=Web-sniffer
WebCookies=WebCookies
WebCorp=WebCorp
WebImages=WebImages
webinatorbot=webinatorbot
webmastercoffee=webmastercoffee
WebNL=WebNL
WebRankSpider=WebRankSpider
WebTarantula.com Crawler=WebTarantula.com Crawler
WebThumbnail=WebThumbnail
WebWatch Robot_txtChecker=WebWatch
WeSEE:Ads=WeSEE:Ads/PageBot
WeSEE:Ads=WeSEE:Ads/PictureBot
WeSEE:Search=WeSEE
WeSEE:Search=WeSEE:Search
WeViKaBot=WeViKaBot
Whoismindbot=Whoismindbot
WikioFeedBot=WikioFeedBot
wikiwix-bot=wikiwix-bot
Willow Internet Crawler=Willow Internet Crawler
WillyBot=WillyBot
WinWebBot=WinWebBot
WMCAI_robot=WMCAI_robot
Woko=Woko
WordPress.com mShots=WordPress.com mShots
woriobot=woriobot
Wotbox=Wotbox
wsAnalyzer=wsAnalyzer
wscheck.com=wscheck.com
x28-job-bot=x28-job-bot
XmarksFetch=XmarksFetch
XML Sitemaps Generator=XML Sitemaps Generator
XoviBot=XoviBot
XRL=XRL
Yaanb=Yaanb
yacybot=yacybot
Yahoo!=Y!J-BRI
Yahoo!=Y!J-BRJ
Yahoo!=Y!J-BRO
Yahoo!=Y!J-BRW
Yahoo!=Y!J-BSC
Yahoo!=Yahoo! Slurp
Yahoo!=Yahoo-MMCrawler
Yahoo!=YahooCacheSystem
YamanaLab-bot=Sonic
YandexBot AntiVirus=YandexAntivirus
YandexBot Blogs=YandexBlogs
YandexBot Catalog=YandexCatalog
YandexBot Direct=YandexDirect
YandexBot Favicons=YandexFavicons
YandexBot ImageResizer=YandexImageResizer
YandexBot Images=YandexImages
YandexBot Media=YandexMedia
YandexBot Metrix=YandexMetrika
YandexBot News=YandexNews
YandexBot Server=Yandex.Server
YandexBot Something=YandexSomething
YandexBot Video=YandexVideo
YandexBot Webmaster=YandexWebmaster
YandexBot Zakladki=YandexZakladki
YandexBot=Yandex
Yanga=Yanga
YioopBot=gofind
YioopBot=YioopBot
YodaoBot Image=YodaoBot-Image
YodaoBot=YodaoBot
YoudaoBot=YoudaoBot
YowedoBot=YowedoBot
YRSpider=YRSpider
YYSpider=YYSpider
ZeerchBot LA1=ZeerchBot LA1
ZeerchBot LA2=ZeerchBot LA2
ZeerchBot=ZeerchBot
Zookabot=Zookabot
ZumBot=ZumBot

Sponsored Links
Re: [7.x] List of Search Engine Spiders for UBBThreads [Re: isaac] #320567
06/25/2014 5:12 AM
06/25/2014 5:12 AM
Joined: Jul 2001
Posts: 1,170
California
isaac Offline OP
$coffee=code(true);
isaac  Offline OP
$coffee=code(true);
Joined: Jul 2001
Posts: 1,170
California
Although the source information for you to create your own updates and conversions is in the OP, I plan on updating this post every couple of months, making your job as an UBBT forum admin much easier.

---
EDIT: user-agent-string.info no longer provides a list of user agent strings without a subscription. this means that until another resource for this data is found, this list will remain as it currently is.

Last edited by id242; 11/02/2015 10:39 PM.
Re: [7.x] List of Search Engine Spiders for UBBThreads [Re: isaac] #320568
06/25/2014 5:47 AM
06/25/2014 5:47 AM
Joined: Dec 2001
Posts: 84
Issaquah, WA
Bill B Offline
Power User
Bill B  Offline
Power User
Joined: Dec 2001
Posts: 84
Issaquah, WA
This is absolutely awesome!!! Many, many thanks from all of us.
[Linked Image]


Bill Barker
Issaquah, Wa
[7.x] List of Search Engine Spiders for UBBThreads [Re: isaac] #320584
08/03/2014 11:20 PM
08/03/2014 11:20 PM
Joined: Aug 2014
Posts: 5
Northern Ireland, UK
Mark J.Cairns Offline
Lurker
Mark J.Cairns  Offline
Lurker
Joined: Aug 2014
Posts: 5
Northern Ireland, UK
This is superb. I only ever had about 6 lines in there. That's so comprehensive as to be unbelievable.

Thank you.


Mark J.Cairns
Producer, Airwolf Themes CD soundtracks

AIRWOLF WEBSITE http://airwolfthemes.com/
OFFICIAL AIRWOLF THEMES VIDEOS http://youtube.com/markjcairns
Airwolf on FACEBOOK http://facebook.com/airwolf.themes
Re: [7.x] List of Search Engine Spiders for UBBThreads [Re: isaac] #320618
10/14/2014 11:57 AM
10/14/2014 11:57 AM
Joined: Jul 2001
Posts: 1,170
California
isaac Offline OP
$coffee=code(true);
isaac  Offline OP
$coffee=code(true);
Joined: Jul 2001
Posts: 1,170
California
This list has been updated 2014-10-14.

Sponsored Links
Re: [7.x] List of Search Engine Spiders for UBBThreads [Re: isaac] #320629
10/30/2014 3:10 PM
10/30/2014 3:10 PM
Joined: Oct 2010
Posts: 6
wa
B
Bill BB Offline
Lurker
Bill BB  Offline
Lurker
B
Joined: Oct 2010
Posts: 6
wa
Thanks again.. This really makes the bot display more accurate. I just saw some bots that we from a company that I trusted... ouch.

Re: [7.x] List of Search Engine Spiders for UBBThreads [Re: isaac] #320971
06/13/2015 8:15 AM
06/13/2015 8:15 AM
Joined: Feb 2007
Posts: 10
California
S
Steve C Offline
Newbie
Steve C  Offline
Newbie
S
Joined: Feb 2007
Posts: 10
California
In my "anonymous" list, I see a number of IPs like this: 157.55.39.xxx. Hovering over the "i" icon it shows:
Agent: Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)

The list above has bingbot entries, and I added one from an older list, so I have this in my search engine agents list:
Code
bingbot=bingbot
bingbot=bingbot/2.0
bingbot=bingbot SitemapProbe
BingPreview=BingPreview

In spite of that, bingbot continues to stay in the anonymous group. Is there a bug, or something I can do to fix it?

Re: [7.x] List of Search Engine Spiders for UBBThreads [Re: isaac] #320972
06/13/2015 9:03 AM
06/13/2015 9:03 AM
Joined: Jul 2001
Posts: 1,170
California
isaac Offline OP
$coffee=code(true);
isaac  Offline OP
$coffee=code(true);
Joined: Jul 2001
Posts: 1,170
California
bingbot=bingbot
bingbot=bingbot/2.0
bingbot=bingbot SitemapProbe

Are all the same thing.

USAGE: SearchEngineName=AgentBotString

The "AgentBotString" on the right side of the equation will search the whole "Agent" and return the matching "SearchEngineName"

So basically, if only list "bingbot=bingbot", you will cover all the other variations of "AgentBotString" for Microsoft's Bing spider/bot.

Re: [7.x] List of Search Engine Spiders for UBBThreads [Re: isaac] #320973
06/13/2015 10:11 AM
06/13/2015 10:11 AM
Joined: Feb 2007
Posts: 10
California
S
Steve C Offline
Newbie
Steve C  Offline
Newbie
S
Joined: Feb 2007
Posts: 10
California
Ok, thanks. "Bing" shows up on Who's online on this forum, so why do they show as anonymous users on mine? (I'm at v 7.5.8, will move to 7.5.9 soon.)

Re: [7.x] List of Search Engine Spiders for UBBThreads [Re: isaac] #320974
06/13/2015 10:17 AM
06/13/2015 10:17 AM
Joined: Jul 2001
Posts: 1,170
California
isaac Offline OP
$coffee=code(true);
isaac  Offline OP
$coffee=code(true);
Joined: Jul 2001
Posts: 1,170
California
If an unregistered user arrives at your site through a search using bing, they will be clasified as anonymous. if your site is being crawled by bingbot (not a live person), it should be classified and shown within the spider section.

Sounds like you have visitors finding your site through Bing. This is good!

Sponsored Links
Re: [7.x] List of Search Engine Spiders for UBBThreads [Re: isaac] #320975
06/13/2015 10:50 AM
06/13/2015 10:50 AM
Joined: Feb 2007
Posts: 10
California
S
Steve C Offline
Newbie
Steve C  Offline
Newbie
S
Joined: Feb 2007
Posts: 10
California
Originally Posted by id242
Sounds like you have visitors finding your site through Bing. This is good!
I am pretty sure that is not the case.
My Who's Online shows these anonymous guests:
157.55.39.120
157.55.39.9
157.55.39.232
157.55.39.218
157.55.39.231
157.55.39.224
Each one shows bingbot when I hover over the "i" icon, and the Referrer: part is blank.


Re: [7.x] List of Search Engine Spiders for UBBThreads [Re: isaac] #320976
06/13/2015 11:46 AM
06/13/2015 11:46 AM
Joined: Jan 2000
Posts: 5,938
Portland, OR, USA
Gizmo Offline

UBB.Dev / UBB.Wiki Owner
Time Lord
Gizmo  Offline

UBB.Dev / UBB.Wiki Owner
Time Lord
Joined: Jan 2000
Posts: 5,938
Portland, OR, USA
FWIW, 157.55.39 is owned by Microsoft, just run the IP's through Domain Tools


UBB.Dev - Putting Dev into UBB.threads
Company: VNC Web Services - UBB.threads Scripts and Scripting, Install and Upgrade Services, Site and Server Maintenance.
Forums: A Gardeners Forum, Scouters World, and UGN Security
UBB.Threads: My UBB Themes, UBB.Sitemaps
Re: [7.x] List of Search Engine Spiders for UBBThreads [Re: Gizmo] #320979
06/14/2015 3:33 PM
06/14/2015 3:33 PM
Joined: Feb 2007
Posts: 10
California
S
Steve C Offline
Newbie
Steve C  Offline
Newbie
S
Joined: Feb 2007
Posts: 10
California
Originally Posted by Gizmo
FWIW, 157.55.39 is owned by Microsoft, just run the IP's through Domain Tools

Yes, Microsoft bingbot. Which brings me back to the original question. Why are six of those showing up as anonymous guests in my Who's Online list.

I would like to move them down into the Search Spiders section, but that is not happening. Any recommendations?

Re: [7.x] List of Search Engine Spiders for UBBThreads [Re: isaac] #320980
06/14/2015 4:24 PM
06/14/2015 4:24 PM
Joined: Jan 2000
Posts: 5,938
Portland, OR, USA
Gizmo Offline

UBB.Dev / UBB.Wiki Owner
Time Lord
Gizmo  Offline

UBB.Dev / UBB.Wiki Owner
Time Lord
Joined: Jan 2000
Posts: 5,938
Portland, OR, USA
Well, if the referrer listing is blank, there isn't much you can do; as the WoL system just parses the spider data based on what is being supplied by bots IN the referrer variable.


UBB.Dev - Putting Dev into UBB.threads
Company: VNC Web Services - UBB.threads Scripts and Scripting, Install and Upgrade Services, Site and Server Maintenance.
Forums: A Gardeners Forum, Scouters World, and UGN Security
UBB.Threads: My UBB Themes, UBB.Sitemaps
Re: [7.x] List of Search Engine Spiders for UBBThreads [Re: Gizmo] #320981
06/14/2015 4:57 PM
06/14/2015 4:57 PM
Joined: Feb 2007
Posts: 10
California
S
Steve C Offline
Newbie
Steve C  Offline
Newbie
S
Joined: Feb 2007
Posts: 10
California
Originally Posted by Gizmo
Well, if the referrer listing is blank, there isn't much you can do; as the WoL system just parses the spider data based on what is being supplied by bots IN the referrer variable.

The referrer listing is blank on all the Search Spider entries, and is blank on all the bingbot "guest" entries. So I don't understand ho the referrer being blank causes it to show up in the guest area.

I do appreciate your taking the time to post in this dialog. I am mostly a "grasshopper" here.

Re: [7.x] List of Search Engine Spiders for UBBThreads [Re: isaac] #320982
06/15/2015 4:40 AM
06/15/2015 4:40 AM
Joined: Jan 2000
Posts: 5,938
Portland, OR, USA
Gizmo Offline

UBB.Dev / UBB.Wiki Owner
Time Lord
Gizmo  Offline

UBB.Dev / UBB.Wiki Owner
Time Lord
Joined: Jan 2000
Posts: 5,938
Portland, OR, USA
I talked with Isaac last night and evidently when the user is viewing a "cached result" from Bing there is no referrer variable passed as it's not "BingBot", but a user that's requesting data through a Bing server.


UBB.Dev - Putting Dev into UBB.threads
Company: VNC Web Services - UBB.threads Scripts and Scripting, Install and Upgrade Services, Site and Server Maintenance.
Forums: A Gardeners Forum, Scouters World, and UGN Security
UBB.Threads: My UBB Themes, UBB.Sitemaps
Re: [7.x] List of Search Engine Spiders for UBBThreads [Re: isaac] #320983
06/15/2015 4:18 PM
06/15/2015 4:18 PM
Joined: Feb 2007
Posts: 10
California
S
Steve C Offline
Newbie
Steve C  Offline
Newbie
S
Joined: Feb 2007
Posts: 10
California
You're giving Bing waaay too much credit. Right now, I have 11 guests, 5 search spiders, and one user. Normal for this small forum with very little activity at night.

Of the 11 guests, 5 are bingbot. And 3 of them are walking this silly thread that has 99 pages. Long continuing threads confound the search spiders. Every time there is a new post, they spend a long time walking through every page in the sequence of pages.

Re: [7.x] List of Search Engine Spiders for UBBThreads [Re: isaac] #320984
06/16/2015 7:49 AM
06/16/2015 7:49 AM
Joined: Jan 2000
Posts: 5,938
Portland, OR, USA
Gizmo Offline

UBB.Dev / UBB.Wiki Owner
Time Lord
Gizmo  Offline

UBB.Dev / UBB.Wiki Owner
Time Lord
Joined: Jan 2000
Posts: 5,938
Portland, OR, USA
Well, not really giving them too much credit; they force SSL for all queries now (source), so it could really be either incoming users from bing are coming in on their SSL (which is the default) or they're coming in from the cache.


UBB.Dev - Putting Dev into UBB.threads
Company: VNC Web Services - UBB.threads Scripts and Scripting, Install and Upgrade Services, Site and Server Maintenance.
Forums: A Gardeners Forum, Scouters World, and UGN Security
UBB.Threads: My UBB Themes, UBB.Sitemaps
Re: [7.x] List of Search Engine Spiders for UBBThreads [Re: Gizmo] #321283
11/02/2015 4:57 PM
11/02/2015 4:57 PM
Joined: Jul 2001
Posts: 1,170
California
isaac Offline OP
$coffee=code(true);
isaac  Offline OP
$coffee=code(true);
Joined: Jul 2001
Posts: 1,170
California
Some further reading regarding http/https referer data:
https://yoast.com/web-https/

Re: [7.x] List of Search Engine Spiders for UBBThreads [Re: isaac] #321284
11/02/2015 10:21 PM
11/02/2015 10:21 PM
Joined: Mar 2007
Posts: 16
S
SteveS Offline
Newbie
SteveS  Offline
Newbie
S
Joined: Mar 2007
Posts: 16
At the moment, I have FOUR of the anonymous Bingbots, and EIGHTEEN from 72.21.217.XXX (Amazon).

Of course, that pales in comparison to the THIRTY FOUR properly identified Baidu spiders.

Last edited by SteveS; 11/02/2015 10:24 PM.
Re: [7.x] List of Search Engine Spiders for UBBThreads [Re: isaac] #321285
11/02/2015 10:31 PM
11/02/2015 10:31 PM
Joined: Jul 2001
Posts: 1,170
California
isaac Offline OP
$coffee=code(true);
isaac  Offline OP
$coffee=code(true);
Joined: Jul 2001
Posts: 1,170
California
SteveS, are you confident your Amazon stuff is not related caching of content within your site by AWS Cloud Computing /Route 53? https://aws.amazon.com/

Additional reading at:
https://en.wikipedia.org/wiki/Amazon_Route_53

The IP 72.21.217.n has been used as a proxy for a User Agent of MSIE-6, which is in itself highly deprecated. Headers can also be consistent with either a battened-down proxy or a bot.

Additional reading at:
"amazonaws.com plays host to wide variety of bad bots"
https://www.webmasterworld.com/search_engine_spiders/3828718.htm

Re: [7.x] List of Search Engine Spiders for UBBThreads [Re: isaac] #321286
11/02/2015 10:39 PM
11/02/2015 10:39 PM
Joined: Mar 2007
Posts: 16
S
SteveS Offline
Newbie
SteveS  Offline
Newbie
S
Joined: Mar 2007
Posts: 16
I am not confident in anything, but I do a lot with Amazon, so I figured they were "botting" for that.


Donate Today!
Donate via PayPal

Donate to UBBDev today to help aid in Operational, Server and Script Maintenance, and Development costs.

Please also see our parent organization VNC Web Services if you're in the need of a new UBB.threads Install or Upgrade, Site/Server Migrations, or Security and Coding Services.
Recommended Hosts
We have personally worked with, and recommend, the following Web Hosts:
· Stable Host
· Blue Host
· Interserver.net
Visit Us on Facebook
Member Spotlight
isaac
isaac
California
Posts: 1,170
Joined: July 2001
Show All Member Profiles 
Forum Statistics
Forums64
Topics37,448
Posts293,484
Members13,793
Most Online1,498
Mar 17th, 2017
Top Posters(All Time)
AllenAyres 25,587
JoshPet 11,330
Rick 8,373
LK 7,396
Lord Dexter 6,503
Gizmo 5,938
Greg Hard 5,533
Top Posters(30 Days)
isaac 4
Today's Statistics
Currently Online 724
Topics Created 0
Posts Made 0
Users Online 0
Birthdays 19
The UBB.Developers Network (UBB.Dev/Threads.Dev) is ©2000-2018 VNC Web Services

 
Powered by UBB.threads™ PHP Forum Software 7.6.2
(Preview build 20180611.dev)
Page Time: 0.146s Queries: 15 (0.038s) Memory: 3.4238 MB (Peak: 3.7352 MB) Zlib enabled. Server Time: 2018-06-19 20:26:04 UTC