Crawling and Search Console

WP-ImmoMakler’s property overview is reachable under more addresses than belong in a search engine’s index: sort links, the infinite scrolling back-links, the map frame. These addresses are meant for your visitors’ browsers. To search engines they are copies of a page they already know — and in the Google Search Console they show up as entries that say nothing about how easy your properties are to find.

This article describes which addresses these are, how WP-ImmoMakler handles them, and what prerequisite your server must meet for this to work.

Which addresses are affected

Address	Purpose	Example
`?im_order=…`	Sort links in the property overview	`/immobilien/?im_order=priceasc`
`/page/12/loadall/`	“Back to overview” link under infinite scrolling: loads pages 1 to 12 at once, so visitors can keep reading where they opened a property	`/immobilien/page/12/loadall/`
`?callback=map-iframe`	The frame in which a property’s or the overview’s map loads	`/immobilien/haus-am-see/?callback=map-iframe&immomakler_map_frame=map-1`

Each of these addresses already carries a signal that points search engines the right way: the sort links are marked rel="nofollow"; with an SEO plugin such as Yoast SEO or Rank Math, they and the /loadall/ pages also point to the regular page via a canonical tag. The map frame redirects a request with no referring page — which is how a crawler arrives — to the property overview via an HTTP 301. The frame itself additionally sends X-Robots-Tag: noindex, nofollow.

A search engine only reads all of these signals after it has fetched the address. So the fetch itself still happens — and that is exactly what shows up in the Search Console.

How WP-ImmoMakler keeps search engines away

WP-ImmoMakler adds its own rule group to your website’s robots.txt that blocks these addresses from being fetched in the first place:

User-agent: *
Disallow: /*callback=map-iframe
Disallow: /*im_order=
Disallow: /*/loadall/

The rules apply to every crawler that understands the * wildcard — Google, Bing, and the other major search engines. Nothing changes for your visitors: the browser still loads the map frame, sorting, and back-link as before.

Prerequisite: WordPress must serve the robots.txt

WordPress generates the robots.txt dynamically — there is no file on the server that answers requests to /robots.txt; WordPress itself does. Only then can plugins such as WP-ImmoMakler or your SEO plugin add their rules. Two situations override this:

The web server answers the request itself. Some nginx or Apache configurations treat /robots.txt as a static file and return a 404 error if none exists. The request never reaches WordPress — in this case your website has no robots.txt at all, not even the one from WordPress and your SEO plugin.
An actual robots.txt file exists in the root directory. The web server serves it directly; WordPress is never asked. WP-ImmoMakler’s rules are then missing, just like your SEO plugin’s.

Here is how to check your website’s state:

curl -I https://your-website.com/robots.txt

If the server responds with HTTP/2 200 and content-type: text/plain, everything is fine. Also open the address in your browser: if the output contains the line Disallow: /*im_order=, it comes from WordPress and WP-ImmoMakler’s rules are active.

If the server responds with 404, situation 1 applies. Ask your hosting provider to pass requests for /robots.txt through to WordPress — with nginx, typically via try_files $uri /index.php?$args; for this path. If the line is missing from the output even though the server responds with 200, situation 2 applies: add the three Disallow lines from the section above to your file by hand, or delete the file so WordPress generates the robots.txt again.

What to expect in the Search Console

After the switch, the affected addresses move into the Blocked by robots.txt file area of the Search Console. This is the desired state: Google only lists the addresses there because it still knows them from earlier crawls, and no longer fetches them. The area empties itself over a few weeks.

Addresses from earlier versions of WP-ImmoMakler where page numbers stacked up — for example /immobilien/page/7/page/5/ — WP-ImmoMakler redirects to the intended page via an HTTP 301, here /immobilien/page/5/. In the Search Console they then show up as Page with redirect instead of a 404 error, and likewise disappear on their own.

Three reports, on the other hand, are no cause for concern even if they show many entries:

Alternate page with proper canonical tag — Google fetched an address whose canonical tag points to another page, and followed the recommendation.
Page with redirect — Google fetched an address that redirects via 301, and adopted the target address.
Blocked by robots.txt file — see above.