Previous Thread
Next Thread
Print Thread
Rate Thread
#204501 03/28/2000 5:07 PM
Joined: Aug 2000
Posts: 3,590
Moderator
Moderator
Offline
Joined: Aug 2000
Posts: 3,590
Anyone have the solution to getting your posts indexed by spiders. I know the problem is with th "?" and the solution probably quite involved.

Any takers? :)

Jay
http://wwlive.net
Online Magazine ala Controversial

Sponsored Links
Joined: May 1999
Posts: 241
Coder
Coder
Offline
Joined: May 1999
Posts: 241
I'm not sure that w3t posts can be indexed like that, due to their dynamic nature... that is, they are stored in the database until the perl script to display them on the users browsers... it's not like other systems that store the posts in html or flat-file format....

Sorry!

Matt



--------------------
Matt Reinfeldt
http://www.mattreinfeldt.com/
Joined: Aug 1999
Posts: 25
Member
Member
Offline
Joined: Aug 1999
Posts: 25
I would suggest creating a robots.txt file in your server root dirctory with a "disallow" line pointing to your wwwthreads path.

[:red]
# go away
User-agent: *
Disallow: /cgi-bin/wwwthreads # point to your wwwthreads directory.


This should work for most robots. I have testing it with our robot and it works,but sadly a few robots only work with META tags.

The problem is that META tags need to be in the page header. I do not think you can do this with wwwthreads with out hacking the perl code. The header in the includes directory, comes after the html pages header so META tags will not work in this file. Maybe this is a feature that can be added.

[:red]
<META NAME="ROBOTS"
CONTENT="ALL | NONE | NOINDEX | NOFOLLOW">

default = empty = "ALL"
"NONE" = "NOINDEX, NOFOLLOW"

The filler is a comma separated list of terms:
ALL, NONE, INDEX, NOINDEX, FOLLOW, NOFOLLOW.

Discussion: This tag is meant to provide users who cannot control
the robots.txt file at their sites. It provides a last chance to
keep their content out of search services. It was decided not to
add syntax to allow robot specific permissions within the meta-tag.

INDEX means that robots are welcome to include this page in
search services.

FOLLOW means that robots are welcome to follow links from this
page to find other pages.

So a value of "NOINDEX" allows the subsidiary links to be explored,
even though the page is not indexed. A value of "NOFOLLOW" allows the
page to be indexed, but no links from the page are explored (this may
be useful if the page is a free entry point into pay-per-view content,
for example. A value of "NONE" tells the robot to ignore the page.

-Ken Torbeck [:blue]WWW.INFOSITE.[:red]ORG Special Needs & disAbilities Info. Center


-Ken Torbeck [blue]www.INFOSITE.[/blue][red]ORG[/red] Special Needs & disAbilities Info. Center

Link Copied to Clipboard
Donate Today!
Donate via PayPal

Donate to UBBDev today to help aid in Operational, Server and Script Maintenance, and Development costs.

Please also see our parent organization VNC Web Services if you're in the need of a new UBB.threads Install or Upgrade, Site/Server Migrations, or Security and Coding Services.
Recommended Hosts
We have personally worked with and recommend the following Web Hosts:
Stable Host
bluehost
InterServer
Visit us on Facebook
Member Spotlight
isaac
isaac
California
Posts: 1,157
Joined: July 2001
Forum Statistics
Forums63
Topics37,573
Posts293,925
Members13,849
Most Online5,166
Sep 15th, 2019
Today's Statistics
Currently Online
Topics Created
Posts Made
Users Online
Birthdays
Top Posters
AllenAyres 21,079
JoshPet 10,369
LK 7,394
Lord Dexter 6,708
Gizmo 5,833
Greg Hard 4,625
Top Posters(30 Days)
Top Likes Received
isaac 82
Gizmo 20
Brett 7
WebGuy 2
Top Likes Received (30 Days)
None yet
The UBB.Developers Network (UBB.Dev/Threads.Dev) is ©2000-2024 VNC Web Services

 
Powered by UBB.threads™ PHP Forum Software 8.0.0
(Preview build 20221218)