Overview

UT_SEO: The Ultimate Upgrade to Mendix SEO_Sitemap! 🚀

Boost your SEO game with UT_SEO - our module for sitemap.xml and robots.txt management. This lightweight module not only streamlines sitemap creation and indexing but also redirects bots to static HTML pages, leaving your Mendix SPA untouched and get indexed. Enjoy seamless indexing, optimized share links, and advanced sitemap capabilities!

Documentation

UT_SEO; the Mendix SEO_Sitemap module on steroids ;-)

Overview

A lightweight module that provides request handlers for sitemap.xml and robots.txt. Additionally, it offers helper functions, a template implementation for setting up multiple sitemaps, and the ability to redirect bots to static HTML pages without entering the Mendix SPA. This ensures efficient indexing and optimized share link functionality.

Documentation

UT_SEO Is built on top of the Mendix SEO_Sitemap module, this module offers developers a lightweight solution that provides request handlers for sitemap.xml and robots.txt. It also includes functionality to redirect bots to static HTML pages without entering the Mendix SPA. This allows the app to build an HTML file on the fly or retrieve it from the database for bots, while users who open the link are redirected to the appropriate Mendix page.

Process 

When creating share links or a sitemap, a custom suffix should used, such as 'link'. In the sitemap or share links, URLs must be modified accordingly. For example, a standard Mendix link like https://www.url.com/p/123456 will use the suffix and become https://www.url.com/link/123456. The Java action, placed in the after-startup phase, registers a request handler that captures all traffic to this specific suffix. If the requesting party is identified as a bot (based on its user agent, a complete list is provided below), a microflow will be triggered, which should return a string of a complete HTML page (example provided in GetStaticPage). If the requester is identified as a normal user, the Java action replaces the suffix with the redirect suffix (by default, it replaces 'link' with 'p'). By using the bot URL for both the sitemap and share links, data will be indexed correctly by Google and others, and links sent via platforms like Teams or WhatsApp will display the provided texts and images.

Setup

1. UseMe This folder contains the steps required to set up the module.

A. Copy Implementationo Copy or move the reference implementation in the _UseMe/1. Setup/A. Copy Implementation folder to a new module and include (right-click, -> include in project) the copied or moved documents in the project.- GetSiteMapXML is an example microflow that returns sitemap.xml contents.- GetRobotsTXT is an example that returns sample robots.txt contents. (By default, it allows all user agents with 'User-agent: * Allow /', which can be changed in the @UT_SEO.AdditionalRobotsTxtContent constant.)- SiteMapService is a published REST service template that specifies endpoints for multiple sitemaps.

B. Constantso AdditionalRobotsTxtContent: If you want to block certain bots, add them here. Default value: 'User-agent: * Allow /'.o BlockAllContent: Changes the robots.txt to disallow all for all user agents.o CanonicalDomain: Optional, for when the GetApplicationRootURL returns the wrong URL.o UrlSuffix: Default URL extension for bots and share links. Default value: 'link'.o UrlRedirectSuffix: The suffix that will replace UrlSuffix for normal users. Default value: 'p'.

C. Configure startup Add the ASU_StartSEO Java action to the startup microflow of the project and configure it.o URL suffix for the bots: Set the UrlSuffix constant.o URL redirect suffix for the users: Set the UrlRedirectSuffix constant.o Static HTML microflow: The flow that provides well-formed HTML for the bots, e.g., GetStaticPage.o The robots.txt microflow, e.g., GetRobotsTXT.o The sitemap.xml microflow, e.g., GetSiteMapXML.Both GetRobotsTXT and GetSiteMapXML microflows should either return a String type or a System.HttpResponse entity object. Using the HttpResponse object gives you more flexibility, such as adding additional HTTP Headers to the response. By following these steps, your app will be ready to be indexed by search engines like Google. To do this, create a profile on Google Search Console and add the provided Google site verification HTML to your theme/web folder.

Extra

o Enable multiple sitemaps To enable multiple sitemaps, utilize the Published REST Service SiteMapService, which needs to be copied to a newly created module. In the ASU_StartSEO Java action within the startup microflow, configure the GetSiteIndexXML microflow. This microflow will return a site index representing the endpoints available in the published sitemap REST service.

o Prevent weird back behavior for users Since users open links with a different URL suffix ('link' by default) and get redirected to the correct one (Mendix default is 'p'), an extra history step appears in the browser. Clicking back will result in no action. To fix this, add the snippet (SNP_AddExtraBackClick) from the D. Add javascript to page folder to the relevant pages/layouts. This ensures that if a user clicks back and the previous page is a redirecting page, a double back click will be executed to prevent this issue.

List of bot user agents

- AdsBot-Google

- AhrefsBot

- APIs-Google

- Baiduspider

- BLEXBot

- Bingbot

- Bot

- bot

- Crowsnest

- DotBot

- DuckDuckBot

- Exabot

- facebot

- facebookexternalhit

- FeedFetcher

-Google

- FlipboardProxy

- Gigabot

- getprismatic.com

- Google-Read-Aloud

- Google-StructuredDataTestingTool

- Googlebot

- Googlebot-Image

- Googlebot-News

- Googlebot-Video

- Google.HTTP.Java.Client

- HTTP_Request2

- ia_archiver

- iaskspider

- JS.Kit URL Resolver

- LinkedInBot

- Mediapatners-Google

- MetaURI

- MJ12bot

- NING.com

- PinterestBot

- SemrushBot

- Slackbot

-LinkExpanding

- Slurp

- Sogou

- Twitterbot

- web.snippet

- Yahoo

- YandexBot