Author Topic: Search engine repellants (Read 6853 times)

jcarter · « **on:** February 06, 2009, 09:37:13 AM »

Im interested in keeping the search engines away from my family sites,
I use these 3 meta things at the moment,

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
"http://www.w3.org/TR/html4/loose.dtd">
<html>
<head>
<title>Whatever</title>
<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1">
<meta name="robots" content="noindex,nofollow,noarchive">
<meta http-equiv="Pragma" content="no-cache">
<meta http-equiv="Cache-Control" content="no-cache, must-revalidate">
<style type="text/css">

Seems to work fairly well, but people here suggest to use the robots.txt to keep them away from the entire sites.

So if I wanted to keep them away from jcarter.net and jcdouglass.net, which are both mine,
Where in the code would I put this,

User-agent: *
Disallow: /

Thanks in advance,
Jane

Mrious_be · « **Reply #1 on:** February 06, 2009, 03:45:20 PM »

Jane,

Why would you want to keep search engines away?
They are the ones that will list you when people are searching for stuff.

jcarter · « **Reply #2 on:** February 06, 2009, 04:24:28 PM »

I dont want my grandkids(and pets) coming up on a Google search. If there was a need of it, then OK, but I just don't think its a good idea for personal, well not really personal, but family stuff to be easily searched on the web. And most of my classmates really dont want our lessons to be up there too. Though I think our classwork is password protected, but once we stick it on our sites, its there. I dont really have any reason to not have my stuff Googled, but I would rather not have my family pictures easily found on the search engines.
I do want all our relatives and family and classmates to find our stuff, but the rest of the world, nahhhh. Anybody who needs it, I have sent links to.

And if that is easily done with what Ive put on there so far, like the noindex, nofollow stuff, then that would have to be added to all the pages. But the robots.txt one looks like if we put that into our main index page, it should take care of the whole site?
Anybody have info on that and where to place it in the code?
Jane

Xairbusdriver · « **Reply #3 on:** February 06, 2009, 04:35:35 PM »

QUOTE

I do want all our relatives and family and classmates to find our stuff, but the rest of the world

I'm not sure we can have it both ways, Jane. If your friends can already find your stuff then it's already indexed.

I would think a simple password system would suffice to keep folks away from your entire domain, including all the sites there.

I think that is can be done with info in a ".htaccess" file. Can't remember how I did that once...but that ".htaccess" file is a great place for telling the browser where to go for different errors encountered, like 404 Page not found.

jcarter · « **Reply #4 on:** February 06, 2009, 05:14:05 PM »

I think this .htaccess is coming up in a future lesson, I will have to see.
I am learning about all this stuff so fast that its all got my head spinning.
There is so much to learn!
I am not really concerned about search engines, but dont want to be on the top few pages. But with all the stuff out there,
my stuff probably wont be way up there anyway. So I am most likely not having to worry.
It would be interesting to find out about this stuff and how it works, where to place the code, and more.
Its fun.
Jane

Xairbusdriver · « **Reply #5 on:** February 06, 2009, 05:33:00 PM »

It is very hard to get onto the first page. It relies more on 'hits' your pages get than what your site might be about. Using general terms will probably not show any of your sites until several pages back. Specific terms (exact first, last names and other info on your pages would be more likely to be nearer the first page. And if the search terms are specific enough and Google has indexed your page with those terms, you could actually end up as the only link on that first page.

In fact, instead of using <meta> space for 'keywords,' you might make it more readily found by your family by using a large number of terms on your Home page that only your family would think of. Of course, it would probably be a lot easier to just send them the link, as you've done!

But it might be an interesting way to see how fast Google's bots/spiders work.

tacit · « **Reply #6 on:** February 06, 2009, 06:08:50 PM »

QUOTE(jcarter @ Feb 6 2009, 03:37 PM) <{POST_SNAPBACK}>

Im interested in keeping the search engines away from my family sites,
I use these 3 meta things at the moment,

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
"http://www.w3.org/TR/html4/loose.dtd">
<html>
<head>
<title>Whatever</title>
<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1">
<meta name="robots" content="noindex,nofollow,noarchive">
<meta http-equiv="Pragma" content="no-cache">
<meta http-equiv="Cache-Control" content="no-cache, must-revalidate">
<style type="text/css">

Seems to work fairly well, but people here suggest to use the robots.txt to keep them away from the entire sites.

So if I wanted to keep them away from jcarter.net and jcdouglass.net, which are both mine,
Where in the code would I put this,

User-agent: *
Disallow: /

Thanks in advance,
Jane

You do not put those lines in your meta tags or anywhere else in your HTML. You put those lines in a robots.txt file.

A robots.txt file is just a text file. You make it with any text editor, like TextEdit,

Create a brand-new text file. In the text file, put the lines

User-agent: *
Disallow: /

Save the file as "robots.txt". Its name must be exactly that; all lower case. It is very important that the file be saved as plain text not rich text. A rich text file WILL NOT work. (In TextEdit, you use the "Make PLain Text" command from the Format menu to make a file be a plain text file.)

Upload it to the top level of your Web site and you're done. There is nothing else you need to do. No changes to any other files and no modification to any of your HTML.

Using keywords in your meta tags won't make any difference one way or the other; search engines no longer look at keywords in meta tags.

jcarter · « **Reply #7 on:** February 06, 2009, 07:42:44 PM »

I wonder if my pages have ever been hit? I doubt it. And our names are all pretty ordinary so might have thousands of hits to other things and names.
Though I have found some of my stuff on Pbase.com, with ordinary dog names.
I put in 'no index' so hopefully they didnt index anything of mine.
I should test it.
Jane Carter comes up as a hair cosmetic.

Someone here might come up with how to use this
User-agent: *
Disallow: /
Though I probably dont need it at all.
I sure learn a lot here!
Jane

jcarter · « **Reply #8 on:** February 06, 2009, 08:05:23 PM »

Hi Tacit,
I posted that former one just seconds before I got yours,,,,,,,,
So the meta tags are useless now, interesting.

I always have TextEdit on plain text, so I just make a new one with only
User-agent: *
Disallow: /
in it.
Nothing else.
Named it

robots.txt

And it goes right in with my index page at the top level.
I'm not good at this, but your instructions are wonderful, here goes.
Jane

jcarter · « **Reply #9 on:** February 06, 2009, 08:38:42 PM »

Here it is,
http://www.jcarter.net/robots.txt
DDT, flit, or bug bombs, whatever
Thanks,
Jane

Xairbusdriver · « **Reply #10 on:** February 06, 2009, 10:39:58 PM »

QUOTE

I wonder if my pages have ever been hit? I doubt it.

Doesn't your host give you some kind of tracking report each month? Even a very basic stats report should show you that kind of information, usually by each calendar month for a year...

sandbox · « **Reply #11 on:** February 07, 2009, 04:38:58 AM »

Yes jane, use a plain text file for robot.txt
When you observe your server logs you will see SOME bots hitting the txt file and going no further. Not all pay attention.
read= http://en.wikipedia.org/wiki/Robot.txt

.hdaccess files are explained here. Keep in mind once you put a dot (period) in front of the file it will become invisible.
info= http://en.wikipedia.org/wiki/.htaccess
practice= http://www.besthostratings.com/articles/ht...-redirects.html
You can write them without the dot, upload it and then add the dot. Some server provide txt files, in cPanel for example.

If you loose an invisible file search for it using the visible- invisible option. It could be on your desktop and you wouldn't know it.

jcarter · « **Reply #12 on:** February 07, 2009, 06:53:45 AM »

A tracking report, I was unaware that this is possible. Wonder how to find it, perhaps on my account page?
I will take a look there anyway.

.htaccess is beyond me, tho I did see it mentioned in a future lesson.

I really dont need to keep the search engines away, as there is no harm, but it just seemed that it would be better not to have all my pictures of family accessable. I tried to Google some of my stuff, and it must be way way down the list, as I didnt see anything.

I love to learn about this web stuff, seems there is always another question Ive got when I learn something new, one thing leads to another.
And while taking all these classes, more and more makes sense to me as I learn.
Thanks to you all here, Ive learned more than I could ever imagine about computers and the web(and more, like airplanes and such.
Jane

Xairbusdriver · « **Reply #13 on:** February 07, 2009, 10:01:53 AM »

Your accounts page should tell you what 'benefits' you should be getting. Of course, checking the hosts various offerings should also mention that info. And it may not even be included in a minimum service plan.

Passwords on a single page or an entire site are not difficult, even I could do it.

But they may not be as secure as you want. OTOH, it doesn't sound like you are really terribly worried about security, at least when it comes to people finding your sites.

Paddy · « **Reply #14 on:** February 07, 2009, 11:09:12 AM »

Jane, htaccess is just another text file you place at the root level of your site. If you follow the directions - or just use someone else's text on any of the various pages that describe how to do it, you should be fine.

News:

Author Topic: Search engine repellants (Read 6853 times)