FAQ section

Adding Content

How Magnity works and what to expect.
Back to overview

Questions and answers

Q1How does Magnity get content into the platform?

Magnity uses a combination of automatic crawling and manual additions to bring content into the library.

Q2How does the automatic crawler work?

The crawler runs every 6 hours

It is configured via backend rules per market

Crawling is controlled by folder rulesets where paths can be included or excluded.

Example: Include /products/, Exclude /products/producta/.

Each crawl also checks the sitemap for:

  1. Lastmod values → if a page is updated, it is re-crawled.
  2. New pages → newly published pages are added.
  3. Language variants → new /xx/ folders or .xx domains mapped and added.

Crawl jobs and results are visible in the Content Engine → Crawler, where you can see:

  1. Pages identified and crawled
  2. Success, errors, not found
  3. Whether content is new or updated
  4. Which market it belongs to
Q3Why use the crawler?

It ensures content is onboarded automatically, fast, and structured, so teams always work with the latest sources.

Q4How are automatic crawl rules controlled?

Automatic crawl rules are managed in the backend by Magnity Support to avoid extreme jobs (e.g., millions of pages).

You can request adjustments if you want us to: Contact your admin (or us, if you are an admin). Based on the rules we can:

  • Include/exclude specific folders
  • Add a new folder for automatic crawling
  • Look for page variants in a specific subdomain
  • Adjust for folder names that differ across markets
  • Decide what types of images should be fetched with content

Contact your admin or Magnity Support if you need changes to the crawl configuration.