Small PoC will test out this Cloudflare worker to deal with parameters and bots.
When you are a user, the srsltid parameter will be appended to the URL. If the user-agent is a bot (like Googlebot). It will remove the parameter from the URL before it is served to the bot.
Go to this URL with the srsltid parameter appended. Now, change your user-agent to Googlebot in your browser and refresh the page. Make sure you disable caching in Google Chrome. You should now load to the same page but without the parameter appended to the URL. In this PoC, other parameters will be ignored.
Your User-Agent:
This code is not production ready! For example, it does not include all the different Googlebots and it does not verify based on IP ranges.
export default {
async fetch(request) {
try {
const userAgent = (request.headers.get("User-Agent") || "").toLowerCase();
const url = new URL(request.url);
const parameter = "srsltid";
const botAgents = new Set(["googlebot", "bingbot"]);
if ([...botAgents].some(bot => userAgent.includes(bot)) && url.searchParams.has(parameter)) {
url.searchParams.delete(parameter);
const newUrl = url.toString();
return new Response(null, {
status: 301,
headers: {
'Location': newUrl,
'Cache-Control': 'no-cache'
}
});
}
// For non-bot traffic or no srsltid param, just fetch the original request
const response = await fetch(request);
return response;
} catch (err) {
// Log the error for debugging purposes (can integrate with a monitoring service)
console.error("Error in worker:", err);
return new Response(`Error in worker: ${err.message}`, {
status: 500,
});
}
}
};
User Agent: mozilla/5.0 applewebkit/537.36 (khtml, like gecko; compatible; claudebot/1.0; +claudebot@anthropic.com)
Is Bot: false
Has Parameter Initially: false
Original URL: https://measure.marketing/parameters-bots
URL Modified: No
Parameter Check After Modification: