AEM fails to redirect /robots.txt via Apache rewrite rule
When Apache is configured to rewrite /robots.txt to a DAM path in AEM but the request still returns a 404, it’s likely due to how Dispatcher handles URLs. This article explains how the DispatcherUseProcessedURL setting affects rewrite behavior and how updating it resolves the redirect failure.
Description description
Environment
ÃÛ¶¹ÊÓƵ Experience Manager (AEM) On-Premise, v6.5
Issue/Symptoms
- A rewrite rule is set to redirect
/robots.txtto/content/dam/xxxx/robots.txt. The asset exists and is accessible directly. - Dispatcher allow rules permit access to bothÂ
/robots.txtÌý²¹²Ô»å/content/dam/xxxx/robots.txt. Despite this, accessing/robots.txtreturns a 404 error.
Root cause
The Apache configuration includes the following rewrite rule:
RewriteRule /robots.txt$ /content/dam/xxxx/robots.txt [ PT,L]
This rule correctly rewrites requests for /robots.txt to the DAM path. However, the Dispatcher is set with:
DispatcherUseProcessedURL 0
With this setting, Dispatcher ignores the rewritten URL and instead uses the original request URI. As a result, it doesn’t recognize the rewritten path and returns a 404 error.
According to the Installing Dispatcher document, setting DispatcherUseProcessedURL to 1 ensures Dispatcher uses the processed URL after mod_rewrite, allowing the redirect to work as expected.
Resolution resolution
To resolve the issue:
- Open your Apache Dispatcher configuration file, typically named
httpd-dispatcher.conf. - Locate the
DispatcherUseProcessedURLdirective. - Change its value from 0 to 1:
DispatcherUseProcessedURL 1
- Save the file and restart Apache for the changes to take effect.
This change ensures the Dispatcher uses the rewritten URL (/content/dam/xxxx/robots.txt) instead of the original request (/robots.txt), allowing the redirect to work as expected.
Related reading
The Dispatcher Security Checklist in the AEM Dispatcher Guide