Previous Thread
Next Thread
Print Thread
Rate Thread
Joined: Jul 2001
Posts: 1,157
Likes: 82
coffee and code
coffee and code
Joined: Jul 2001
Posts: 1,157
Likes: 82
Robots (Crawlers) List for UBB.threads - UPDATED 2022-07-10
formerly named "Search Engine Spiders List for UBB.threads"

About:
The advantage to using this list is that Robots (Crawlers and Search Engine Spiders) get put into the correct "Robots (Crawlers)" group when viewing your forum's Who's Online page.

This translation list is not used anywhere else in UBB.threads except on the Who's Online page at /ubbthreads.php/online. Having a long list of Robots will not slow down your forums.

Details:
Robots List: 2022-07-10
Total Agent Strings: 912
Sources:
https://udger.com/resources/ua-list/crawlers (2022-07-10) and
https://github.com/monperrus/crawler-user-agents (2022-05-03)
I've simply converted, combined, and cleaned the data for use with UBB.threads.

How to install:
1) Go to Control Panel > Display Options > Who's Online Settings.
2) Copy/paste the text from the newest list in to the "Robots (Crawlers)" box at the bottom of the page.
3) Click the "Update General Display Options" button.
done.

Notes:
Always use the newest list. Older lists contain old robots and Inactive crawlers.
Installs prior to 7.6.2 did not include a list of robot agents.
robots_20141014-UBBT762.txt Fresh Installs of UBB.threads 7.6.2 to 7.7.3 are pre-populated with this list.
robots_20200114-UBBT774.txt Fresh Installs of UBB.threads 7.7.4 are pre-populated with this list.
robots_20200924-UBBT775.txt Fresh Installs of UBB.threads 7.7.5 are pre-populated with this list.
robots_20220710-UBBT800.txt Fresh Installs of UBB.threads 8.0.0+ are pre-populated with this list.
robots_20220710-CATEGORIES.txt is the same robots list as the stock list, plus it includes a category name for many of the robots, such as Search, Marketing, Monitoring, Link Checker, Tool, etc.

Having problems using this list on an older version of UBB.threads? Remove any blank lines from the top/bottom of your copied list.
Attachments
robots_20141014-UBBT762.txt (13.87 KB, 0 downloads)
Fresh Installs of UBB.threads 7.6.2 to 7.7.3 are pre-populated with this
SHA1: dfe7ed85b91ad5b197e5926d845eb12bf71f0f74
robots_20200114-UBBT774.txt (24.31 KB, 0 downloads)
Fresh Installs of UBB.threads 7.7.4 are pre-populated with this
SHA1: 864216eace8eff40a05d56e2f2f21cde2714b602
robots_20200924-UBBT775.txt (18.71 KB, 3 downloads)
Fresh Installs of UBB.threads 7.7.5 are pre-populated with this
SHA1: 4c6dca844f125e1c4865f1c37b33e1add09e0b45
robots_20220710-UBBT800.txt (23.44 KB, 3 downloads)
Fresh Installs of UBB.threads 8.0.0+ are pre-populated with this
SHA1: e540578c7309298f824cc3e1b407918e153526ff
robots_20220710-CATEGORIES.txt (30.02 KB, 6 downloads)
SHA1: be349d4983a1358114a2f2de5340e9adcd6d510e

Last edited by isaac; 07/10/2022 10:41 AM. Reason: updated lists for 20220710

Current developer of UBB.threads PHP Forum Software
Current Release: UBBT 7.7.5 // Preview: UBBT 8.0.0
isaac @ id242.com // my forum @ CelicaHobby.com
2 members like this: isaac, Gizmo
Sponsored Links
Joined: Jul 2001
Posts: 1,157
Likes: 82
coffee and code
coffee and code
Joined: Jul 2001
Posts: 1,157
Likes: 82
Although the source information for you to create your own updates and conversions is in the OP, I plan on updating this post every couple of months, making your job as an UBBT forum admin much easier.

---
EDIT: user-agent-string.info no longer provides a list of user agent strings without a subscription. this means that until another resource for this data is found, this list will remain as it currently is.

Last edited by id242; 11/02/2015 10:39 PM.

Current developer of UBB.threads PHP Forum Software
Current Release: UBBT 7.7.5 // Preview: UBBT 8.0.0
isaac @ id242.com // my forum @ CelicaHobby.com
1 member likes this: Gizmo
Joined: Dec 2001
Posts: 87
Power User
Power User
Joined: Dec 2001
Posts: 87
This is absolutely awesome!!! Many, many thanks from all of us.


Bill Barker
Issaquah, Wa
Joined: Aug 2014
Posts: 6
Lurker
Lurker
Joined: Aug 2014
Posts: 6
This is superb. I only ever had about 6 lines in there. That's so comprehensive as to be unbelievable.

Thank you.


Mark J.Cairns
Producer, Airwolf Themes CD soundtracks

AIRWOLF WEBSITE https://airwolfthemes.com/
OFFICIAL AIRWOLF THEMES VIDEOS https://www.youtube.com/airwolfthemes
Airwolf on FACEBOOK https://facebook.com/airwolf.themes
Joined: Jul 2001
Posts: 1,157
Likes: 82
coffee and code
coffee and code
Joined: Jul 2001
Posts: 1,157
Likes: 82
This list has been updated 2014-10-14.


Current developer of UBB.threads PHP Forum Software
Current Release: UBBT 7.7.5 // Preview: UBBT 8.0.0
isaac @ id242.com // my forum @ CelicaHobby.com
1 member likes this: Gizmo
Sponsored Links
Joined: Oct 2010
Posts: 6
Lurker
Lurker
Offline
Joined: Oct 2010
Posts: 6
Thanks again.. This really makes the bot display more accurate. I just saw some bots that we from a company that I trusted... ouch.

Joined: Feb 2007
Posts: 10
Newbie
Newbie
Offline
Joined: Feb 2007
Posts: 10
In my "anonymous" list, I see a number of IPs like this: 157.55.39.xxx. Hovering over the "i" icon it shows:
Agent: Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)

The list above has bingbot entries, and I added one from an older list, so I have this in my search engine agents list:
Code
bingbot=bingbot
bingbot=bingbot/2.0
bingbot=bingbot SitemapProbe
BingPreview=BingPreview
In spite of that, bingbot continues to stay in the anonymous group. Is there a bug, or something I can do to fix it?

Joined: Jul 2001
Posts: 1,157
Likes: 82
coffee and code
coffee and code
Joined: Jul 2001
Posts: 1,157
Likes: 82
bingbot=bingbot
bingbot=bingbot/2.0
bingbot=bingbot SitemapProbe

Are all the same thing.

USAGE: SearchEngineName=AgentBotString

The "AgentBotString" on the right side of the equation will search the whole "Agent" and return the matching "SearchEngineName"

So basically, if only list "bingbot=bingbot", you will cover all the other variations of "AgentBotString" for Microsoft's Bing spider/bot.


Current developer of UBB.threads PHP Forum Software
Current Release: UBBT 7.7.5 // Preview: UBBT 8.0.0
isaac @ id242.com // my forum @ CelicaHobby.com
Joined: Feb 2007
Posts: 10
Newbie
Newbie
Offline
Joined: Feb 2007
Posts: 10
Ok, thanks. "Bing" shows up on Who's online on this forum, so why do they show as anonymous users on mine? (I'm at v 7.5.8, will move to 7.5.9 soon.)

Joined: Jul 2001
Posts: 1,157
Likes: 82
coffee and code
coffee and code
Joined: Jul 2001
Posts: 1,157
Likes: 82
If an unregistered user arrives at your site through a search using bing, they will be clasified as anonymous. if your site is being crawled by bingbot (not a live person), it should be classified and shown within the spider section.

Sounds like you have visitors finding your site through Bing. This is good!


Current developer of UBB.threads PHP Forum Software
Current Release: UBBT 7.7.5 // Preview: UBBT 8.0.0
isaac @ id242.com // my forum @ CelicaHobby.com
Sponsored Links
Joined: Feb 2007
Posts: 10
Newbie
Newbie
Offline
Joined: Feb 2007
Posts: 10
Originally Posted by id242
Sounds like you have visitors finding your site through Bing. This is good!
I am pretty sure that is not the case.
My Who's Online shows these anonymous guests:
157.55.39.120
157.55.39.9
157.55.39.232
157.55.39.218
157.55.39.231
157.55.39.224
Each one shows bingbot when I hover over the "i" icon, and the Referrer: part is blank.


Joined: Jan 2000
Posts: 5,833
Likes: 20
UBBDev / UBBWiki Owner
Time Lord
UBBDev / UBBWiki Owner
Time Lord
Joined: Jan 2000
Posts: 5,833
Likes: 20
FWIW, 157.55.39 is owned by Microsoft, just run the IP's through Domain Tools


UBB.Dev - Putting Dev into UBB.threads
Company: VNC Web Services - UBB.threads Scripts and Scripting, Install and Upgrade Services, Site and Server Maintenance.
Forums: A Gardeners Forum, Scouters World, and UGN Security
UBB.Threads: My UBB Themes, My UBB Scripts
Joined: Feb 2007
Posts: 10
Newbie
Newbie
Offline
Joined: Feb 2007
Posts: 10
Originally Posted by Gizmo
FWIW, 157.55.39 is owned by Microsoft, just run the IP's through Domain Tools
Yes, Microsoft bingbot. Which brings me back to the original question. Why are six of those showing up as anonymous guests in my Who's Online list.

I would like to move them down into the Search Spiders section, but that is not happening. Any recommendations?

Joined: Jan 2000
Posts: 5,833
Likes: 20
UBBDev / UBBWiki Owner
Time Lord
UBBDev / UBBWiki Owner
Time Lord
Joined: Jan 2000
Posts: 5,833
Likes: 20
Well, if the referrer listing is blank, there isn't much you can do; as the WoL system just parses the spider data based on what is being supplied by bots IN the referrer variable.


UBB.Dev - Putting Dev into UBB.threads
Company: VNC Web Services - UBB.threads Scripts and Scripting, Install and Upgrade Services, Site and Server Maintenance.
Forums: A Gardeners Forum, Scouters World, and UGN Security
UBB.Threads: My UBB Themes, My UBB Scripts
Joined: Feb 2007
Posts: 10
Newbie
Newbie
Offline
Joined: Feb 2007
Posts: 10
Originally Posted by Gizmo
Well, if the referrer listing is blank, there isn't much you can do; as the WoL system just parses the spider data based on what is being supplied by bots IN the referrer variable.
The referrer listing is blank on all the Search Spider entries, and is blank on all the bingbot "guest" entries. So I don't understand ho the referrer being blank causes it to show up in the guest area.

I do appreciate your taking the time to post in this dialog. I am mostly a "grasshopper" here.

Joined: Jan 2000
Posts: 5,833
Likes: 20
UBBDev / UBBWiki Owner
Time Lord
UBBDev / UBBWiki Owner
Time Lord
Joined: Jan 2000
Posts: 5,833
Likes: 20
I talked with Isaac last night and evidently when the user is viewing a "cached result" from Bing there is no referrer variable passed as it's not "BingBot", but a user that's requesting data through a Bing server.


UBB.Dev - Putting Dev into UBB.threads
Company: VNC Web Services - UBB.threads Scripts and Scripting, Install and Upgrade Services, Site and Server Maintenance.
Forums: A Gardeners Forum, Scouters World, and UGN Security
UBB.Threads: My UBB Themes, My UBB Scripts
Joined: Feb 2007
Posts: 10
Newbie
Newbie
Offline
Joined: Feb 2007
Posts: 10
You're giving Bing waaay too much credit. Right now, I have 11 guests, 5 search spiders, and one user. Normal for this small forum with very little activity at night.

Of the 11 guests, 5 are bingbot. And 3 of them are walking this silly thread that has 99 pages. Long continuing threads confound the search spiders. Every time there is a new post, they spend a long time walking through every page in the sequence of pages.

Joined: Jan 2000
Posts: 5,833
Likes: 20
UBBDev / UBBWiki Owner
Time Lord
UBBDev / UBBWiki Owner
Time Lord
Joined: Jan 2000
Posts: 5,833
Likes: 20
Well, not really giving them too much credit; they force SSL for all queries now (source), so it could really be either incoming users from bing are coming in on their SSL (which is the default) or they're coming in from the cache.


UBB.Dev - Putting Dev into UBB.threads
Company: VNC Web Services - UBB.threads Scripts and Scripting, Install and Upgrade Services, Site and Server Maintenance.
Forums: A Gardeners Forum, Scouters World, and UGN Security
UBB.Threads: My UBB Themes, My UBB Scripts
Joined: Jul 2001
Posts: 1,157
Likes: 82
coffee and code
coffee and code
Joined: Jul 2001
Posts: 1,157
Likes: 82
Some further reading regarding http/https referer data:
https://yoast.com/web-https/


Current developer of UBB.threads PHP Forum Software
Current Release: UBBT 7.7.5 // Preview: UBBT 8.0.0
isaac @ id242.com // my forum @ CelicaHobby.com
1 member likes this: Gizmo
Joined: Mar 2007
Posts: 16
Newbie
Newbie
Offline
Joined: Mar 2007
Posts: 16
At the moment, I have FOUR of the anonymous Bingbots, and EIGHTEEN from 72.21.217.XXX (Amazon).

Of course, that pales in comparison to the THIRTY FOUR properly identified Baidu spiders.

Last edited by SteveS; 11/02/2015 10:24 PM.
Joined: Jul 2001
Posts: 1,157
Likes: 82
coffee and code
coffee and code
Joined: Jul 2001
Posts: 1,157
Likes: 82
SteveS, are you confident your Amazon stuff is not related caching of content within your site by AWS Cloud Computing /Route 53? https://aws.amazon.com/

Additional reading at:
https://en.wikipedia.org/wiki/Amazon_Route_53

The IP 72.21.217.n has been used as a proxy for a User Agent of MSIE-6, which is in itself highly deprecated. Headers can also be consistent with either a battened-down proxy or a bot.

Additional reading at:
"amazonaws.com plays host to wide variety of bad bots"
https://www.webmasterworld.com/search_engine_spiders/3828718.htm


Current developer of UBB.threads PHP Forum Software
Current Release: UBBT 7.7.5 // Preview: UBBT 8.0.0
isaac @ id242.com // my forum @ CelicaHobby.com
1 member likes this: Gizmo
Joined: Mar 2007
Posts: 16
Newbie
Newbie
Offline
Joined: Mar 2007
Posts: 16
I am not confident in anything, but I do a lot with Amazon, so I figured they were "botting" for that.

Joined: Jul 2001
Posts: 1,157
Likes: 82
coffee and code
coffee and code
Joined: Jul 2001
Posts: 1,157
Likes: 82
Changelog 2019-11-16
The Robots (Crawlers) List for UBB.threads has been updated in OP


Current developer of UBB.threads PHP Forum Software
Current Release: UBBT 7.7.5 // Preview: UBBT 8.0.0
isaac @ id242.com // my forum @ CelicaHobby.com
Joined: Jul 2001
Posts: 1,157
Likes: 82
coffee and code
coffee and code
Joined: Jul 2001
Posts: 1,157
Likes: 82
Changelog 2020-01-14
The Robots (Crawlers) List for UBB.threads has been updated in OP


Current developer of UBB.threads PHP Forum Software
Current Release: UBBT 7.7.5 // Preview: UBBT 8.0.0
isaac @ id242.com // my forum @ CelicaHobby.com
1 member likes this: Morgan
Joined: Jul 2001
Posts: 1,157
Likes: 82
coffee and code
coffee and code
Joined: Jul 2001
Posts: 1,157
Likes: 82
Changelog 2020-09-24
The Robots (Crawlers) List for UBB.threads has been updated in OP


Current developer of UBB.threads PHP Forum Software
Current Release: UBBT 7.7.5 // Preview: UBBT 8.0.0
isaac @ id242.com // my forum @ CelicaHobby.com
1 member likes this: Gizmo
Joined: Apr 2022
Posts: 25
User
User
Offline
Joined: Apr 2022
Posts: 25
When I try to add the list, I paste them, click save, and I get a 403 forbidden page.

But when I do something else on the page, click save, I don't.

Not sure why. My forum is https://ChristianDiscussion.org I have the latest version of UBB.

Last edited by Aaron101; 04/05/2022 6:43 PM.
Joined: Jul 2001
Posts: 1,157
Likes: 82
coffee and code
coffee and code
Joined: Jul 2001
Posts: 1,157
Likes: 82
Originally Posted by Aaron101
When I try to add the list, I paste them, click save, and I get a 403 forbidden page.

But when I do something else on the page, click save, I don't.

Its your host. they are censoring the content that you send. theyre basically blocking you from typing words/phrases they do not agree with... for "security" lol

refer back to your other post when you had the same problem
https://www.ubbcentral.com/forums/ubbthreads.php/topics/265214/403-forbidden


Current developer of UBB.threads PHP Forum Software
Current Release: UBBT 7.7.5 // Preview: UBBT 8.0.0
isaac @ id242.com // my forum @ CelicaHobby.com
Joined: Apr 2022
Posts: 25
User
User
Offline
Joined: Apr 2022
Posts: 25
Do I even need to add this list if I have the latest version of UBB?

Joined: Jul 2001
Posts: 1,157
Likes: 82
coffee and code
coffee and code
Joined: Jul 2001
Posts: 1,157
Likes: 82
Originally Posted by Aaron101
Do I even need to add this list if I have the latest version of UBB?

7.7.5 changelog says:
Quote
Updated Robots (Crawlers) list from 20200114 to build 20200924

Source:
https://www.ubbcentral.com/changelog.php#775


edit -
If it was an initial install of ubb.threads that you did, its included as the changelog says.
If this was just an upgrade you did, the robots.txt will be what came with your original version PLUS any customizations that you had done to it. ie;the content in this post is for you.


Current developer of UBB.threads PHP Forum Software
Current Release: UBBT 7.7.5 // Preview: UBBT 8.0.0
isaac @ id242.com // my forum @ CelicaHobby.com
Joined: Apr 2022
Posts: 25
User
User
Offline
Joined: Apr 2022
Posts: 25
Okay, I am good then. I just purchased UBB this week. smile

Joined: Jul 2001
Posts: 1,157
Likes: 82
coffee and code
coffee and code
Joined: Jul 2001
Posts: 1,157
Likes: 82
This list has been updated 2022-07-09.


Current developer of UBB.threads PHP Forum Software
Current Release: UBBT 7.7.5 // Preview: UBBT 8.0.0
isaac @ id242.com // my forum @ CelicaHobby.com
1 member likes this: Gizmo

Link Copied to Clipboard
Donate Today!
Donate via PayPal

Donate to UBBDev today to help aid in Operational, Server and Script Maintenance, and Development costs.

Please also see our parent organization VNC Web Services if you're in the need of a new UBB.threads Install or Upgrade, Site/Server Migrations, or Security and Coding Services.
Recommended Hosts
We have personally worked with and recommend the following Web Hosts:
Stable Host
bluehost
InterServer
Visit us on Facebook
Member Spotlight
JAISP
JAISP
PA
Posts: 449
Joined: February 2008
Forum Statistics
Forums63
Topics37,573
Posts293,925
Members13,849
Most Online5,166
Sep 15th, 2019
Today's Statistics
Currently Online
Topics Created
Posts Made
Users Online
Birthdays
Top Posters
AllenAyres 21,079
JoshPet 10,369
LK 7,394
Lord Dexter 6,708
Gizmo 5,833
Greg Hard 4,625
Top Posters(30 Days)
Top Likes Received
isaac 82
Gizmo 20
Brett 7
WebGuy 2
Morgan 2
Top Likes Received (30 Days)
None yet
The UBB.Developers Network (UBB.Dev/Threads.Dev) is ©2000-2024 VNC Web Services

 
Powered by UBB.threads™ PHP Forum Software 8.0.0
(Preview build 20221218)