Implementing Robots.txt Editor in Sitefinity for SEO

Robots.txt file is usually placed in the root folder of the website, and for security purposes root folders are more secure than the specific subfolder. Because changes to Robots.txt are managed by SEO personnel, they were often unable to upload the updated Robots file through File manager in Sitefinity, and were dependent on the programmers to make edits to that file.

At Americaneagle.com, we thought SEO personnel should be able to manage robots.txt more easily through the editor in the CMS. Adding a custom configuration was the obvious choice. But the custom configurations that appear under Advanced Settings are limited to simple textboxes. We wanted text area on the custom configuration so it looks more like a text editor and more readable. What we found was the template for advanced setting only supports textbox input and changing that would change it for all the advanced settings.

As an alternative, there is a way to create text area under the Basic setting screen as described here: http://www.sitefinity.com/developer-network/forums/general-discussions-/multisite-custom-field#Zo7eg2ycdkSGElmwBsX5rA

Slightly tweaking this functionality to our needs, we were able create a whole new screen as a Basic setting entry and have a template with the long text field as needed.

With this change we removed the physical robots file, because it is highly possible that if someone is looking at that, they may tend to change that. We used a redirect rule to grab that configuration content and display it whenever the user types in /Robots.txt

Here is the screenshot of how it looks:

/Sitefinity/Administration/Settings/Basic/Robots/

Sitefinity Robots.TXT

It should be noted that this editor does not validate the contents of the file, it is simply for editing the contents. There are other tools available online for testing Robots file.

Full code is available for download from Gist:

https://gist.github.com/mani-ae/d1fdab9297264d0522063e131487a52c

Here are code snippets:

Robots.txt code snippets

Americaneagle.com can help you in implementing this Robots file editor in Sitefinity, which will help SEO personnel to manage using the visual interface. Contact us for more details.

Contact Us Today!

About Author

ManiBalasubramaniyan
Mani Bala is a Technical lead at Americaneagle.com, with over dozen years of experience in tech. He has Masters Degree in Information Systems, with an Engineering background. He is currently working on Enterprise websites using Sitefinity. He loves problem solving and learning new technologies.


Featured Posts