The explicit set of `rurl` canonicalization arguments that determine a pagerankr node identity (scheme + host + path). Every knob that *shapes that key* is pinned here with an explicit value, so node keys never depend on `rurl`'s own defaults – which have changed across `rurl` versions (e.g. `case_handling` flipped from `"keep"` to `"lower_host"`) and previously desynced the pagerankr <-> semantic join. Knobs that only govern components the key drops (port, query, fragment, userinfo) are deliberately left unpinned – see the "Knobs deliberately left unpinned" note in @details.
Details
The canonical node key is **scheme + host + non-default port + path + contentful query**, with the path normalized under the WHATWG URL standard and percent-encoding preserved byte-for-byte. Fragment and userinfo are dropped by `get_clean_url` and identify no resource – userinfo under `credential_handling = "strip"`, which is pinned because the alternative (`"reject"`) yields `NA` rather than a key.
**The governing principle: parse, do not fold.** pagerankr has redirects and canonical tags as first-class inputs, and those are the site's own statement about which URLs are the same page. Canonicalization therefore normalizes only what the *standard* says is the same resource, and asserts nothing about site configuration. If `www.example.com/a` and `example.com/a` have no redirect and no canonical between them, they are two nodes – and that is a finding, not a defect to be papered over. The knobs that would fold them (`www_handling`, `trailing_slash_handling`, `index_page_handling`, `protocol_handling`) are all pinned to their non-folding values for exactly this reason.
Two corollaries that are easy to get backwards: * A **non-default port is a different origin**, so it stays in the key. `rurl`'s default `port_handling = "exclude"` drops every port and would merge `host:8080` with `host`. `"strip_default"` removes only `:80` on http and `:443` on https, which the standard makes redundant. * An **IDN host and its punycode form are the same request on the wire**, so no redirect or canonical can ever fold them – the graph has to. `host_encoding = "idna"` normalizes both to the punycode form.
**Why `whatwg`, and why `path_encoding = "keep"`.** These two are the only knobs here that do not simply mirror a `rurl` default, and they are the load-bearing pair. `path_encoding` is a *presentation* dial – `rurl`'s own documentation says only `"keep"` preserves a profile's canonical identity path verbatim, and that `"encode"`/`"decode"` "may re-encode or decode reserved octets (so ` pinned `"decode"` for several releases and thereby merged `/a `/a/b`: two different resources, one node. Identity semantics live on `url_standard` instead, which reaches a profile-internal path-identity axis no presentation dial can touch.
`"whatwg"` rather than `"rfc3986"` because pagerankr models what a search engine sees. Measured over the node-key fixture, `whatwg` additionally resolves percent-encoded dot segments (` tab/newline from paths instead of failing the parse, percent-encodes literal spaces instead of failing the parse, and keeps `http://host` and `http://host/` as one node. `rfc3986` splits that last pair and returns `NA` for the whitespace classes, both of which are common in crawl exports. The cost is that `whatwg` preserves percent spellings, so `/a
The remaining knobs equal `rurl`'s current defaults and are pinned only to freeze them. (The anti-drift guarantee ultimately lives in the golden-key fixtures in `test-canonicalization.R`: `rurl` 3.0.0 re-keyed six of them by reordering decode after dot-segment removal *without touching any argument*, which the surface guard cannot see.)
All twenty of these arguments are accepted by both `rurl::get_clean_url()` (the cleaning path) and `rurl::safe_parse_url()` (the domain-filtering path), so one profile drives both and the two paths stay symmetrical.
**Knobs deliberately left unpinned.** `rurl::get_clean_url()` has grown options that govern components the node key does not carry, or that select a parsing route rather than a key component: * `source` (default `"all"`) – the public-suffix source, reachable only through `www_handling`/`subdomain_levels_to_keep`, both pinned to values that do not consult it. * Added in `rurl` 2.7.0: `scheme_policy` (default `"infer"`), `scheme_acceptance` (default `"web"`), `engine` (default `NULL`) and `profile` (default `NULL`, an unrelated `rurl` concept that merely shares a name with this function). In particular `profile = "seo"` is **not** used and must not be: it bundles `protocol_handling = "https"`, `www_handling = "strip"`, `trailing_slash_handling = "strip"` and `index_page_handling = "strip"`, every one of which is a redirect class pagerankr resolves through its own auditable fold map ([build_fold_map()], applied by [pagerank()]). Folding those at canonicalization time makes the redirect row self-referential, which the fold map drops as a no-op – the hop is erased before anything can audit or report it.
Under the scheme+host+path key these have no visible effect at their defaults, so pinning them would add noise without changing identity. They are intentionally **not** part of this profile; instead `test-canonicalization.R` guards them from two sides, so a `rurl` change is caught on the pagerankr side rather than silently changing node identity: a behavioral guard asserts that a canonical key really does drop the port, query and fragment (catching a default *flip*), and a surface guard reads `rurl`'s own formals and fails on any argument this profile has neither pinned nor listed above (catching an *addition*). A committed node-key probe covering the cross-platform parse-determinism risk surface pins the keys themselves.
The cross-repo contract requires **semantic** to pin the identical profile; change both repos together.
**Accepted divergence on un-canonicalizable input.** For a value `rurl` cannot parse (an unsupported scheme like `mailto:`/`tel:`, whitespace, a dotless bare token), `rurl` returns `NA`. pagerankr's [clean_url_columns()] keeps such a value as its raw self so it survives as an opaque graph node (see that function; PR #50), whereas semantic's `canonical_url()` returns `None` and drops it (FR-05 rurl byte-parity). This is intentional and does **not** break the `node_score` <-> `page` join: valid URLs still produce byte-identical keys on both sides (the actual contract), and in the semantic -> pagerankr bridge semantic canonicalizes and drops un-canonicalizable inputs *before* pagerankr sees the edges, so the raw fallback never fires on that path. It only affects pagerankr run standalone on raw crawl data, where such tokens become opaque nodes instead of being dropped.
Examples
# The pinned profile that determines pagerankr node identity.
profile <- canonical_profile()
str(profile)
#> List of 20
#> $ protocol_handling : chr "keep"
#> $ case_handling : chr "lower_host"
#> $ www_handling : chr "none"
#> $ trailing_slash_handling : chr "none"
#> $ index_page_handling : chr "keep"
#> $ path_normalization : chr "dot_segments"
#> $ scheme_relative_handling: chr "keep"
#> $ subdomain_levels_to_keep: NULL
#> $ host_encoding : chr "idna"
#> $ path_encoding : chr "keep"
#> $ url_standard : chr "whatwg"
#> $ port_handling : chr "strip_default"
#> $ query_handling : chr "filter"
#> $ params_keep : NULL
#> $ params_drop : NULL
#> $ params_case_sensitive : logi FALSE
#> $ sort_params : logi FALSE
#> $ empty_param_handling : chr "keep"
#> $ decode_plus : logi FALSE
#> $ credential_handling : chr "strip"
# Key knobs that shape the scheme + host + path node key.
profile$case_handling # "lower_host"
#> [1] "lower_host"
profile$path_normalization # "dot_segments"
#> [1] "dot_segments"
profile$path_encoding # "keep" (presentation dial, held at identity)
#> [1] "keep"
profile$url_standard # "whatwg" (where identity semantics live)
#> [1] "whatwg"