Update robots.txt #441
Update robots.txt #441
Conversation
| Disallow: /_image? | ||
| Disallow: /Benutzer/ | ||
| Disallow: /users/ | ||
| Disallow: /*?flavour=mobile |
There was a problem hiding this comment.
Although the URLs do no switch to a mobile version any more, they are still valid (f.e. https://ubuntuusers.de/?flavour=mobile displays the same content like https://ubuntuusers.de/ with HTTP 200). So a possible drawback could be duplicate URLs in the search index, because links still use this variant or are still in the google index etc. Thus, i would avoid this removal. Any other opinions?
There was a problem hiding this comment.
Yes, that’s probably right. However fixed now.
There was a problem hiding this comment.
If the flavour parameter is deprecated, could we instead use redirects? Then search engines and alike would take notice and not crawl them.
There was a problem hiding this comment.
IMO that's hard to do within Inyoka, as this potentially affects every view (and also 'only' old intances). I dont know whether the used proxy has the ability to 'remove' all flavour HTTP-Parameter with a 301.
| @@ -1,16 +1,6 @@ | |||
| User-agent: * | |||
| Crawl-delay: 10 | |||
| Disallow: /search | |||
| Disallow: /_image? | |||
| Disallow: /Benutzer/ | |||
There was a problem hiding this comment.
If /Benutzer/ is blocked for bots, shouldn't /user/ be blocked too?
There was a problem hiding this comment.
Anonymous users see at /user/-URLs a login page, so i dont see really a benefit.
There was a problem hiding this comment.
But they see a login page at /Benutzer/ too. So wouldn’t it be consequent also to remove that line and the /users/ one?
There was a problem hiding this comment.
Valid Point. However, to resolve this i would suggest another thing: Return not HTTP 200 on login pages, instead 403 (or 404, if we want to hide the page's existence).
There was a problem hiding this comment.
As the user names and links to it's profiles are known, we don't need to and can hide the pages' existence.
This pull request removes no more needed entrys in robots.txt file.