SourceWatch:Google-index
Contents
robots.txt check
Click here to check SourceWatch's robots.txt file: http://www.SourceWatch.org/robots.txt
Its contents should be exactly as follows:
User-agent: * Disallow:
The December robots.txt problem
The robots.txt file was identified as missing on Dec 13, and recreated almost immediately by Sheldon Rampton.
As at Dec 19, 2004, the total number of pages indexed by Google is still falling. However the number of pages indexed since Dec 13 is rising, as measured by the " Number of pages indexed by Google since introduction of Events and News Feeds sections" search.
This appears to indicate that Google is shedding old pages from its index faster than it indexes new pages.
Total number of SourceWatch pages indexed by Google
Number of results in Google when searching for site:www.SourceWatch.org "log in"
- Oct 28: 650 (203)
- Oct 29: 205 (201)
- Oct 30: 347 (334)
- Oct 31: 512 (498)
- Nov 1: 512 (498)
- Nov 3: 1,040 (692)
- Nov 4: 2,000 (713)
- Nov 5: 2,450 (912)
- Nov 6: 2,990
- Nov 7: 5,820
- Nov 9: 6,520
- Nov 10: 12,400
- Nov 11: 15,300
- Nov 12: 17,400
- Nov 13: 18,700
- Nov 14: 19,400
- Nov 16: 19,700
- Nov 17: 20,400
- Nov 19: 21,500
- Nov 20: 21,800
- Nov 22: 22,100
- Nov 23: 22,100
- Nov 25: 22,200
- Nov 26: 22,300
- Nov 27: 22,200
- Dec 9: 9,860
- Dec 13: robots.txt identified as missing, and recreated.
- Dec 15: 4,550
- Dec 16: 3,750
- Dec 17: 9,360
- Dec 19: 2,970
- Dec 20: 5,920
- Dec 21: 7,550
- Dec 23: 9,470
- Dec 24: 10,600
- Dec 25: 12,400
- Dec 26: 17,000
- Dec 27: 19,600
- Dec 29: 20,000
- Jan 1: 23,700
- Jan 4: 24,500
- Jan 5: 25,300
Number of pages indexed by Google since introduction of Events and News Feeds sections
Relevant Google search: forum events "news feeds" donate site:SourceWatch.org
- Dec 13: robots.txt identified as missing, and recreated.
- Dec 17: 192
- Dec 19: 600
- Dec 20: 823
- Dec 21: 4,430
- Dec 23: 6,370
- Dec 24: 7,450
- Dec 25: 9,240
- Dec 26: 13,900
- Dec 27: 16,500
- Dec 29: 17,200
- Jan 1: 22,200
- Jan 4: 23,400
- Jan 5: 24,200
Number of pages indexed by Google since Oct 27 2004
Relevant Google search: "log in" site:SourceWatch.org within last 3 months
All the pages returned but one have been reindexed since Oct 27th. The exception is this page: http://www.SourceWatch.org/wiki.phtml?title=Rhodes
So to get the number of pages reindexed since Oct 27th, subtract one from the number of search results.
- Oct 29: 22-1 = 21
- Oct 30: 124-1 = 123
- Oct 31: 222-1 = 221
- Nov 1: 222-1 = 221
- Nov 3: 451-1 = 450
- Nov 4: 686-1 = 685
- Nov 5: 819-1 = 818
- Nov 6: 1,000 (Google only shows 3 significant digits)
- Nov 7: 2,700
- Nov 9: 3,000
- Nov 10: 5,570
- Nov 11: 6,820
- Nov 12: 7,750
- Nov 13: 8,210
- Nov 14: 8,400
- Nov 16: 8,410
- Nov 17: 8,640
- Nov 19: 9,020
- Nov 20: 9,120
- Nov 22: 9,190
- Nov 23: 9,210
- Nov 25: 9,240
- Nov 26: 9,100
- Nov 27: 9,130
- Dec 9; 11,800
- Dec 13: robots.txt identified as missing, and recreated.
- Dec 15: 1,840
- Dec 16: 3,640
- Dec 17: 4,030
- Dec 19: 981
- Dec 20: 1,450
- Dec 21: 3,890
- Dec 23: 4,860
- Dec 24: 5,200
- Dec 25: 6,060
- Dec 26: 7,950
- Dec 27: 9,220
- Dec 29: 9,300
- Jan 1: 11,100
- Jan 4: 11,500
- Jan 5: 11,800
