Website import

Takes over an existing website (Webflow, WordPress, Squarespace, static pages …) with page tree, HTML, CSS, JS and media. The import runs in two steps as background jobs: first analyze, then import with the desired selection.

All import endpoints require write permission ✏️.

Process and behavior

1. Analysis (POST /import/scan): reads robots.txt and the sitemap(s) (including sitemap index and .xml.gz), optionally follows all internal links and evaluates which hosts the website loads files from – including hosts that only appear in CSS files (fonts).
2. Import (POST /import/run) with the selected pages and hosts:
   • Page tree from the URL paths. The home page of the source is mapped to target_page_id. Missing intermediate levels (e.g. /leistungen for /leistungen/webdesign) are created as drafts.
   • Paths are normalized: lowercase, umlauts (ä → ae), special characters → -, collisions → -2. Reserved names at the top level (api, media, cms-admin) get a -seite suffix. Internal links are rewritten accordingly; query and fragment are kept.
   • <head>: <title> → title, meta description → description; charset, viewport, generator and <base> are dropped (IllusionCMS sets them itself), everything else goes into head.
   • <html>/<body> attributes → html_attrs/body_attrs (important for Webflow interactions).
   • CSS/JS become assets. Stylesheets at the beginning of <head> and scripts at the end of <body> that are identical on all imported pages are attached to the target page and inherited. All others stay exactly in their position and reference the library via {{ asset_url(id) }} – this preserves the loading order. If the home page is not imported as well, each page gets its own files (not inherited).
   • integrity/crossorigin are removed from taken-over files (otherwise an edited file would be blocked). preload hints for taken-over CSS/JS files are dropped.
   • Media (images, fonts, videos, PDFs, favicons, social media images) go into file management, folder Import <domain>. Captured are, among others, src, srcset, poster, style="…url()…", <style>, url()/@import in CSS, data-src/data-srcset/data-bg (lazy loading), data-video-urls, og:image.
   • Images with srcset: only the largest version is downloaded; the srcset is replaced with the CMS’s own variants. og:image & co. get absolute URLs on the original domain.
   • Template syntax in the original ({{, {%, {#) is protected via {% verbatim %}.
   • Files from the site’s own domain are always taken over, external ones only from the hosts in download_hosts; everything else stays linked externally.

Limits: pages that are only built in the browser via JavaScript (React/Vue SPAs) deliver hardly any HTML – the analysis reports this as js_rendered. Forms, search, logins and other server-side functions of the source are only taken over visually. Scripts that load further files at runtime relative to their own URL may need to be adjusted.

Job object

{
  "id": "9f2c41d0a7b3e815",
  "kind": "import",
  "status": "running",
  "phase": "Loading pages and files",
  "phase_code": "import.phaseLoad",
  "done": 12,
  "total": 48,
  "files": 87,
  "assets": 5,
  "log": [
    { "time": 1791197160, "level": "warn", "message": "file not imported: HTTP 404", "url": "https://cdn.example.com/alt.png",
      "code": "import.fileNotImported", "params": { "error": "HTTP 404" } }
  ],
  "result": null,
  "started_at": 1791197150,
  "finished_at": 0
}

Field	Description
kind	scan or import
status	running, done, failed, cancelled
done / total	Progress in pages
files / assets	Files or CSS/JS files taken over so far (import only)
phase / phase_code	Current step (English text) and its code for translation
log	Log (info, warn, error), max. 3000 entries; entries with code/params (see Errors)
error / error_code / error_params	For failed: message, code and parameters
result	Result, once status = done (see below)

Jobs are kept in memory; the last 20 remain retrievable, a restart deletes them.

POST /import/scan ✏️

{ "url": "https://www.firma.de", "crawl": false, "respect_robots": true, "max_pages": 500 }

Field	Description
url	Start address; without a scheme, https:// is assumed. Redirects are followed, the final domain counts as the “own domain” (www. is ignored)
crawl	Additionally follow all internal links. Enabled automatically if no sitemap is found
respect_robots	Pages blocked by robots.txt are still listed, but marked with "excluded": "robots" (the UI deselects them in advance, the crawler doesn’t follow their links). robots.txt is aimed at search engines – when moving your own website, blocked pages such as the legal notice or privacy policy usually belong to it
max_pages	Upper limit (default 500, max. 5000)

Response 202 with the job object. Result (result) on completion:

{
  "base": "https://www.firma.de",
  "sitemaps": ["https://www.firma.de/sitemap.xml"],
  "pages": [
    { "url": "https://www.firma.de/", "path": "/", "title": "Firma GmbH", "source": "start" },
    { "url": "https://www.firma.de/ueber-uns", "path": "/ueber-uns", "title": "Über uns", "source": "sitemap" },
    { "url": "https://www.firma.de/rechtliches/impressum", "path": "/rechtliches/impressum", "source": "sitemap", "excluded": "robots" }
  ],
  "hosts": [
    { "host": "www.firma.de", "category": "site", "label": "Own domain", "count": 14, "kinds": { "img": 10, "file": 4 }, "suggested": true },
    { "host": "cdn.prod.website-files.com", "category": "builder", "label": "Site builder CDN", "count": 96, "kinds": { "css": 3, "js": 6, "img": 80, "font": 7 }, "suggested": true },
    { "host": "fonts.gstatic.com", "category": "fonts", "label": "Web fonts", "count": 4, "kinds": { "font": 4 }, "suggested": true },
    { "host": "www.googletagmanager.com", "category": "service", "label": "Service / embed", "count": 1, "kinds": { "js": 1 }, "suggested": false }
  ],
  "js_rendered": false
}

Host categories and preselection (suggested). label is an English name of the category; clients translate via category:

category	Examples	Preselected
site	own domain	always taken over
builder	Webflow, Squarespace, Wix, Shopify, WordPress.com, Contentful, Cloudinary CDNs	yes
fonts	Google Fonts, Bunny Fonts	yes (self-hosted fonts are more GDPR-friendly)
cdn	cdnjs, jsDelivr, unpkg, code.jquery.com	yes
service	analytics, Tag Manager, YouTube, Maps, cookie banners, chat, Adobe Fonts	no – must stay live or must not be copied
other	all others	yes, unless the host only serves scripts

Besides the home page, the analysis loads up to 5 further pages and up to 10 CSS files as a sample (when crawling, all pages found).

POST /import/run ✏️

{
  "base": "https://www.firma.de",
  "urls": ["https://www.firma.de/", "https://www.firma.de/ueber-uns"],
  "target_page_id": 1,
  "overwrite": true,
  "publish": true,
  "download_hosts": ["cdn.prod.website-files.com", "fonts.googleapis.com", "fonts.gstatic.com"]
}

Field	Description
base	base from the analysis result
urls	Pages to import (typically a selection from pages[].url)
target_page_id	Page onto which the home page of the source is mapped; all others go below it
new_site	Instead of target_page_id: { "title": "…", "domains": "a.de\nwww.a.de" } creates a new website and imports into it. 409 if a domain is already taken (nothing is created in that case). result.target_page_id contains the ID of the new website
overwrite	Update existing, unedited pages (content, head, attributes, CSS/JS assignments). Otherwise they are skipped
overwrite_modified	Also overwrite pages and CSS/JS files edited in the CMS (default: false – they stay as they are)
force	Also rewrite pages whose source hasn’t changed since the last import
publish	Publish newly created pages (intermediate levels remain drafts). The status of existing pages is kept
download_hosts	External hosts whose files are taken over

Response 202 with the job object. Result on completion:

{
  "target_page_id": 1,
  "created": 46, "updated": 1, "unchanged": 0, "modified": 0, "foreign": 0, "skipped": 0, "failed": 1, "placeholders": 2,
  "files": 214, "files_updated": 0, "files_reused": 0,
  "assets": 7, "assets_updated": 0, "assets_reused": 0, "assets_kept": 0,
  "shared_css": 1, "shared_js": 3,
  "pages": [
    { "id": 1, "path": "/", "source": "https://www.firma.de/", "action": "updated" },
    { "id": 12, "path": "/ueber-uns", "source": "https://www.firma.de/ueber-uns", "action": "created" },
    { "id": 0, "path": "/kaputt", "source": "https://www.firma.de/kaputt", "action": "failed" }
  ]
}

action per page: created, updated, unchanged (source unchanged), modified (edited in the CMS, not overwritten), foreign (path belongs to a page from the import of another website – never overwritten), skipped, failed. files/assets count newly created files, *_updated those updated in place, *_reused reused ones, assets_kept CSS/JS files changed in the source but edited in the CMS (kept). shared_css/shared_js: number of files attached to the target page as inherited.

Repeated imports

An import can be repeated any number of times without creating duplicates:

• Pages are found first via their origin (source_url + target page) – even if they have been moved or renamed in the meantime – otherwise via the path. If the source HTML is unchanged since the last import, the page is skipped (unchanged) without reloading its files.
• Pages edited in the CMS (local_modified) are only overwritten with overwrite_modified.
• Files are recognized via their source URL and requested with If-None-Match (ETag) – unchanged ones are not downloaded again. Changed content is replaced in place (same ID, same URL, variants recomputed). Media files are therefore only cached in the browser for one day.
• CSS/JS are recognized via their source URL; unchanged ones are reused, changed ones updated – unless they were edited in the CMS (then the CMS version is kept, with a warning in the log).
• The download hosts are remembered per website and source and used by POST /pages/{id}/reimport – an import of another source into the same website doesn’t overwrite them.
• If none of the given URLs belongs to the source (base), the job fails with an error message instead of silently doing nothing.

Orphaned files from earlier imports are not deleted automatically. Files and assets from imports before this version have no stored origin and are created anew once on the first re-import.

GET /import/jobs ✏️

The most recent jobs (newest first), without log and result.

GET /import/jobs/{id} ✏️

State of a job. With ?log_since=n, only log entries from index n onward are returned – handy for polling:

curl -H "Authorization: Bearer $TOKEN" "$API/import/jobs/9f2c41d0a7b3e815?log_since=120"

DELETE /import/jobs/{id} ✏️

Cancels a running job (204). Files and assets already created are kept; pages are only written at the end of an import.

Example: import via script

JOB=$(curl -s -X POST "$API/import/scan" -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d '{"url": "https://www.firma.de", "respect_robots": true}' | jq -r .id)
until [ "$(curl -s -H "Authorization: Bearer $TOKEN" "$API/import/jobs/$JOB" | jq -r .status)" != running ]; do sleep 1; done
SCAN=$(curl -s -H "Authorization: Bearer $TOKEN" "$API/import/jobs/$JOB" | jq .result)

curl -s -X POST "$API/import/run" -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d "$(jq '{base, urls: [.pages[].url], target_page_id: 1, overwrite: true, publish: true,
            download_hosts: [.hosts[] | select(.suggested and .category != "site") | .host]}' <<< "$SCAN")"
# Note: urls here also includes pages blocked by robots.txt – to exclude them,
# use [.pages[] | select(.excluded == null) | .url]

---