Changelog • pagerankr

pagerankr 0.1.0

The GitHub Actions workflows are gone, and codemeta.json names GitLab. .gitlab-ci.yml has replaced all eight workflows since the move to GitLab, so .github/workflows/ and .github/dependabot.yml were dead weight that the new read-only GitHub mirror would have republished. codemeta.json now points at gitlab.com for every URL it carries – repository, issue tracker, release notes and CI – matching what DESCRIPTION has said since PAGE-mehcgwzv (PAGE-yfmrrrhp).
BugReports: points at /-/issues, the form the CRAN incoming check requires. The incoming check on R-devel notes any gitlab.com BugReports: whose path is not /-/issues; a browser is redirected to /-/work_items. README and the generated pagerankr-package help topic follow (PAGE-mehcgwzv).
rurl now installs from CRAN; the Remotes: field is gone. rurl reached CRAN as 3.0.1 on 2026-09-09, so DESCRIPTION no longer needs a Remotes: entry – a field CRAN does not honor and which drew a NOTE on every --as-cran run. Every hard dependency now resolves from CRAN (PAGE-agtgxupt).
The declared rurl floor is >= 3.0.1, up from >= 3.0.0. rurl went 1.2.0 straight to 3.0.1 on CRAN and never published a 3.0.0, so the old floor named a version nobody could install – the second time this package declared a minimum that existed nowhere. 3.0.1 is the lowest rurl that exists as a release and that the suite has been run against (PAGE-tsbkxhoz).
canonical_profile() now pins credential_handling = "strip". rurl 3.0.1 added the argument. Userinfo is not part of a node key, but this knob decides whether a credentialed URL yields a key at all: the alternative, "reject", returns NA, which clean_url_columns() keeps as its raw self and so turns every credentialed URL into an opaque node. "strip" is rurl’s current default, so no node key changes – the pin freezes a default that could otherwise flip underneath the graph.
The package is now hosted on GitLab, and the install command has changed to devtools::install_gitlab("bart-turczynski/pagerankr"). The previous install_github() form no longer resolves. Bug reports move to the GitLab tracker, and security reports are now made by email to the maintainer rather than through GitHub private vulnerability reporting (see SECURITY.md).
Convergence controls and damping now fail as documented instead of silently changing the request or raising an internal R error. Three fixes, all at the public boundary of both pagerank() and compute_pagerank():
- A fractional niter is rejected. It was passed through as.integer(), so niter = 2.7 became an iteration cap of 2 with nothing said. Values above .Machine$integer.max are rejected for the same reason: the coercion produced NA and reached the solver as a missing cap.
- damping = NA_real_ (and NA_integer_, NaN, Inf, -Inf) reports `damping` must be a single numeric value between 0 and 1. compute_pagerank() tested damping < 0 on an NA, so the if received NA and R raised “missing value where TRUE/FALSE needed”. Bare NA is logical and already produced the named error; only the typed missings and the infinities slipped through. Both entry points now share one validator.
- A non-finite eps is rejected. Inf passed the eps <= 0 test, and an infinite tolerance is not a tolerance.
Added deterministic fixtures for graph shapes whose convergence and mass diagnostics were unpinned: disconnected components (a closed component holds exactly its share of the teleport vector; a dangling node moves mass across the component boundary), zero-weight edges (equivalent to no edge, with the target left on teleport mass alone), and the all-zero-weight graph (a named weight error).
Node identity moves off a presentation dial: canonical_profile() now pins url_standard = "whatwg" and path_encoding = "keep". This changes node keys. path_encoding is documented by rurl as a presentation knob whose "decode" value “may re-encode or decode reserved octets (so %2F may fold to a path-separating /)”; pagerankr had been using it as its identity function. Two consequences, both defects, are fixed:
- An encoded slash is no longer folded into a path separator, so http://example.com/a%2Fb and http://example.com/a/b are two nodes rather than one. Same for http://example.com/..%2fa, which previously popped a segment it should not have.
- Percent-encoded dot segments resolve again. rurl 3.0.0 stopped resolving them under the old pin by reordering decode after dot-segment removal, so /%2e%2e/a keyed to /../a instead of /a. Under "whatwg" the standard recognizes the encoded form directly and no presentation dial is involved.
Three further classes improve for crawl data: paths containing a tab or newline, and URLs with surrounding whitespace, now parse (they previously returned NA and fell through to the raw-token fallback), and a literal space is percent-encoded rather than failing the parse. The cost is that "whatwg" preserves percent spellings, so /a%7Eb and /a~b are now two nodes, as they are to a crawler.

rurl_params is narrowed. Because the profile now pins a standard selector, rurl rejects a conflicting override of the axes that selector governs — case_handling and path_normalization. Those two are node identity, so a caller who could move them could silently re-key the graph. Every other knob still overrides as before.

Note rurl’s profile = "seo" is deliberately not used and is not a substitute: it bundles protocol_handling = "https", www_handling = "strip", trailing_slash_handling = "strip" and index_page_handling = "strip", each of which is a redirect class pagerankr models as an edge. Folding those at canonicalization time turns the redirect into a self-loop.
canonical_profile() now follows “parse, do not fold”. pagerankr has redirects and canonical tags as first-class inputs, and those are the site’s own statement about which URLs are the same page. Canonicalization therefore normalizes only what the standard makes the same resource and asserts nothing about site configuration. Three knobs move to match:
- port_handling = "strip_default" (was the rurl default "exclude"). A non-default port is a different origin, so http://host:8080/a and http://host/a are now two nodes. :80 on http and :443 on https are redundant per spec and still fold.
- host_encoding = "idna" (was "keep"). An IDN host and its punycode form are the same request on the wire, so no redirect or canonical can ever fold them – the graph has to. http://münchen.de/a and http://xn--mnchen-3ya.de/a are now one node, keyed on the punycode form. Pass rurl_params = list(host_encoding = "keep") to restore the split.
- query_handling = "filter" (was the rurl default "drop"). A contentful parameter is part of the resource: ?color=red and ?color=blue are two pages, and no redirect or canonical need exist between them. Tracking parameters are not, and are still dropped via the built-in rurl denylist (utm_*, fbclid, gclid). This is the largest behavior change here – on a faceted site, node counts will rise. The six query sub-options (params_keep, params_drop, params_case_sensitive, sort_params, empty_param_handling, decode_plus) are now key-shaping and so are pinned explicitly too, all at the rurl defaults. sort_params = FALSE matches the rurl key contract, under which query order and duplicates are significant.
The knobs that would fold a redirect class – www_handling, trailing_slash_handling, index_page_handling, protocol_handling – were already pinned to their non-folding values and are unchanged.
The declared rurl minimum is >= 3.0.0. Two corrections landed in this cycle. Imports: previously required rurl (>= 2.1.0), a version that was never released — rurl has no 2.1.0 tag, and its 2.x line begins at 2.2.0 — so the stated minimum named an artifact no user could install; that was corrected to >= 2.2.0, verified against rurl 2.2.0, 2.2.1 and 2.8.0 in isolated libraries with byte-identical canonical keys across all three. The floor then moved to >= 3.0.0, which is where the path-identity axis the profile now depends on lives. Note the 2.2.0 floor had also become impossible to test in practice: rurl inserted scheme_policy and scheme_acceptance before url_standard rather than appending them, so a positional url_standard argument silently binds to the wrong argument on eight functions across that range.
New vignette, vignette("case-study"), works the two-crawl fixture end to end. A worked example that scores the same site before and after an internal-linking change on two lenses — the editorial (content-only) graph and the full graph — and shows that a change which moves the top page from rank #1 to #38 on the first is invisible on the second (Pearson 0.9999). Every figure is computed from inst/extdata/ at build time rather than quoted, so the vignette doubles as a regression check on the fixture. It also documents the three traps the comparison sets: scores do not sum to 1, so shares must be taken against sum(pagerank); the two crawls have different node sets, so absolute PageRank is not comparable across them; and pages must be matched on the full URL, since external hosts share path strings with internal ones. _pkgdown.yml gains an articles: index grouping all six vignettes.
A two-crawl case-study fixture ships in inst/extdata/. Two Screaming Frog exports of the same small site, taken either side of a deliberate internal-linking intervention, are now available as package data: system.file("extdata", "reviews-microsite-before", package = "pagerankr") and its -after counterpart, each holding all_inlinks.csv and internal_all.csv. Because the same site is measured twice, a change in PageRank is attributable to the intervention rather than to a difference between sites — which is what makes the pair usable for validating the placement and boilerplate models. Both exports are trimmed to the columns sf_contract() declares (18 and 24, down from Screaming Frog’s 103), and Link Path is retained deliberately as the boilerplate detector’s signal. The whole fixture is roughly 70 KB gzipped. The underlying site is pseudonymous: hosts, paths, anchors, alt text and page free-text are replaced rather than redacted, and inst/extdata/README.md documents exactly what is synthetic and what is preserved verbatim.
pagerank() surfaces a per-URL wasted_mass figure alongside page_state. When indexability_df or status_df is supplied, the result now carries a wasted_mass column next to page_state: the authority each page collected and black-holed, i.e. its share of the shared waste sink’s stationary mass. page_state labels the “collects but cannot pass” class per row; wasted_mass quantifies the amount, answering the other half of “a 404 page amassed X PageRank and it ends up in a black hole.” Because a waste-class page routes its whole throughput through one edge to the absorbing sink, the value is damping / (1 - damping) times the page’s own reported score — larger than, and deliberately distinct from, that score. Summed over the class it equals the evaporated mass in the transition audit (mass$sink); a "live" page routes nothing to the sink, so its wasted_mass is 0. Like page_state, the column appears only when one of those inputs is present, mirroring how prior_weight appears only with prior_df.
Documented that page_state and simulate_changes()’s node_status are distinct axes, not two names for one thing. page_state (live / noindex / robots_blocked / response_dead) is a page’s health/index state; node_status (normal / new-target / removed-dead) is a node’s role in a before/after comparison. They are kept separate on purpose: new-target has no health analogue, and removed-dead is the single value bridging both axes (a node removed because its proposed health state is a forced 404). The two @return sections now cross-reference each other.
simulate_changes() gains a remove_urls verb for URL-level what-ifs. Pass a character vector of URLs to model them returning 404: each keeps its inbound links (other pages still point at it) but their authority now flows into a dead page and evaporates to the shared waste sink, rather than redistributing across the site via teleport (dangle) or self-amplifying via a self-loop. The page’s outbound links are dropped and the node stays in the output flagged "removed-dead" in node_status, so its residual absorbed mass is never misread as earned authority. This is the first consumer of the status_df / waste-sink dead-class mechanism: remove_urls forces a 404 into the proposed model only, leaving the baseline live. A URL appearing in both remove_urls and redirect_urls_df is an error (a page cannot be both a 301 and a 404); the manifest gains urls_removed. Available on the Screaming Frog path via simulate_changes_screaming_frog(remove_urls = ...), where the forced 404 composes on top of the bundle’s real crawled status.
Page-state modeling — one faithful policy for pages that collect PageRank but cannot pass it (epic PAGE-qzskzcfd). The three changes immediately below — status_df, the unified waste sink, and prior_exclude_waste — together replace three accidental, inconsistent behaviors for the noindex / robots-blocked / 4xx-5xx class with a single model: authority flows in, is booked, and stops. Before, the same class behaved three ways: a robots-blocked page self-looped and self-amplified up to 8.3× (scoring 0.8875 trapped vs 0.1065 evaporating — holding 89% of a graph), 4xx/5xx pages were scored as ordinary live vertices, and uniform teleport let 1,000 fake dead URLs capture 95.2% of a site’s PageRank for merely existing. The class now routes to a shared waste sink, is excluded from the teleport vector, and is labeled per-row in page_state so absorbed mass is never read as earned authority. robots_blocked_action survives only as a presentation toggle ("show" / "vanish"); the self-looping "trap" is gone. Any graph containing a class member changes score — accepted pre-CRAN (experimental, single consumer). Reproducible experiment and full rationale: notes/pagerank-behavior-field-notes.md §11.
pagerank() gains prior_exclude_waste to keep the “collects but cannot pass” class out of the teleport vector. With the default TRUE, the noindex, robots-blocked, and 4xx/5xx pages identified by indexability_df / status_df receive zero teleport mass: they still collect the authority their inlinks send, but are no longer paid the uniform teleport share for merely existing, so a page can no longer manufacture authority by pointing at many dead ends (Page & Brin 1998 criticize uniform teleport for “valuing pages simply because they exist”). This is a first-class replacement for the error-prone prior_df + prior_alpha workaround previously needed to zero teleport entries. Set FALSE to restore uniform teleport over every page, matching igraph::page_rank() for canonical comparisons. Scores change whenever a non-empty class is present; graphs with no such class — and any run without indexability_df/status_df — are numerically unchanged. The synthetic evaporation and leak sinks are excluded from teleport regardless of this flag, which is recorded in the transition audit as config$prior_exclude_waste. No effect under reverse = TRUE, where the class inputs are already rejected.
pagerank() gains a status_df input for HTTP response status. Supply a url + status_code frame (column names configurable via status_url_col and status_col) and pages that returned a 4xx or 5xx code are identified as response-dead and counted in the transition audit (config$has_status and dropped$n_status_dead). 4xx and 5xx are treated identically — a crawl is a snapshot, and a transient 503 is indistinguishable from a permanent 404 at crawl time (the same reason a 302 is folded exactly like a 301); 3xx stays with redirects_df, and unparseable or sub-400 codes are treated as live. pagerank_screaming_frog() now feeds status from the bundle’s node table automatically instead of discarding it, and the raw preset leaves it off. Not supported with reverse = TRUE.
Unified waste-sink mechanism for the “collects PageRank but cannot pass it” class. noindex, robots-blocked, and response-dead (4xx/5xx) pages now share one mechanism: each loses its outgoing edges and gains exactly one edge to a shared internal waste sink, so it absorbs the authority its inlinks send but passes none back into the graph. This removes the old robots-blocked self-loop, which was an absorbing rank sink that compounded inbound authority every iteration (a measured 8.3× inflation); robots-blocked scores change accordingly. A no-outlink 404 now gets the one sink edge that stops it from dangling and recycling its authority to every page via teleport.
- robots_blocked_action values are renamed c("trap", "vanish") → c("show", "vanish"), default "show" (the self-loop “trap” is gone). Both values route the page’s throughput to the sink; they differ only in whether the page is shown ("show") or removed with its own mass booked as hidden ("vanish").
- noindex is decoupled from nofollow_action: a noindex page always routes to the sink, and nofollow_action now governs only real rel=nofollow edges. This changes noindex behavior only under nofollow_action "drop" / "keep"; the default "evaporate" is numerically unchanged.
- When indexability_df or status_df is supplied, the result gains a page_state column tagging each visible page "live", "noindex", "robots_blocked", or "response_dead", so wasted mass is attributable per URL (robots-blocked > response-dead > noindex when a page carries more than one signal). It appears only with those inputs, mirroring how prior_weight appears only with prior_df.
simulate_changes() can now model URL-level what-ifs, not just edge-level ones. The new redirect_urls_df argument (a two-column from/to frame) models retiring a page behind a redirect: it strips the live source’s own outbound links before folding, so the target inherits the source’s inbound authority only — never its outlinks — and the source drops out of the proposed set. A row for a source overrides any prior redirect for it (an earlier row, or the baseline crawl’s real 3xx), so “repoint A→B into A→C” is a single override. Redirects pass authority through at 100% (the only per-hop loss is the global damping every edge already incurs). This replaces the removed add_redirects_df, which folded a live source without stripping its outedges and so modeled a content move rather than a retire.
New on_unknown_target = c("warn", "error", "allow") controls what happens when a redirect or link target is not yet a node in the graph (it may be a legitimate new page); the default "warn" proceeds and models it as a new node carrying inbound authority.
The comparison output gains a node_status column ("normal" or "new-target") plus proposed (the full proposed pagerank() result, with its transition audit) and manifest (redirects applied/overridden, link add/remove counts, unknown targets) attributes.
New simulate_changes_screaming_frog() applies the same what-if verbs to a screaming_frog_bundle, reusing the same bundle → pagerank() adapter as pagerank_screaming_frog() so placement, boilerplate, nofollow, and status are preserved automatically. Both entry points share one changeset engine.
Positional decay: pagerank() can now weight links by their reading order within the page. Point the new position_col at a numeric per-source position index — 1 for the first link, 2 for the second — and each edge’s weight is scaled by a reading-order decay. This is the orthogonal axis of the edge-weighting model: where placement and boilerplate describe templatedness and feed one graded axis combined by minimum, position describes reading order and composes by multiplication, so an above-the-fold boilerplate CTA (0.5 × 1.0) outranks a trailing organic link (1.0 × 0.2) with no special-casing. The decay reuses the existing transform_weights() shapes: position_transform is "zipf" (default, 1 / rank^position_alpha) or "rank_linear", applied within each source page’s choice set, and position_floor (default 0.01) clamps the result strictly above zero so compounding two axes can never reach an effective drop. Off unless position_col is supplied, and crawler-neutral: any crawler that can report a link’s order can drive it. Recorded separately in the transition audit’s config$position, so an edge’s weight can be explained as region times reading order.
screaming_frog_links() now materializes that index at ingest. The edge table gains a position_index column carrying each link’s reading-order rank among its source page’s content links — computed from document order for an All Outlinks export, and left NA for All Inlinks, whose row order is destination-alphabetical rather than document order. pagerank_screaming_frog() therefore drives the axis with position_col = "position_index"; it stays opt-in, so pass that argument to switch it on.
pagerank_screaming_frog() gains a preset argument, and preset = "raw" now actually produces the as-crawled graph. Previously the wrapper fed pagerank() the bundle’s declared canonical, redirect and indexability tables unconditionally, so a preset — which sets policy, never data — could not unset them, and preset = "raw" still folded canonicals/redirects and trapped robots-blocked pages. The "raw" view now switches all three declared tables off at the wrapper level. An explicit apply_canonicals or apply_redirects still overrides the "raw" default for that table; the import audit records apply_indexability alongside the existing two flags.
Boilerplate detection: pagerank() can now discount repetitive template links that sit in the main content. Point the new container_col at a column identifying each link’s source-side component and every edge is scored by container-conditioned recurrence: of the pages that component appears on, the share on which it points at this same target. A recycled call-to-action that always links the same place scores near 1 and is discounted; a related-posts module that links somewhere different on every page scores near 0 and keeps full weight — even though both recur identically. This catches what placement never can, because bylines, compliance links and recycled CTAs are structurally content. Tuned by boilerplate_threshold (default 0.5, the ratio at which an edge is classified), min_container_pages (default 10, an evidence floor) and boilerplate_weight (default 0.5, the discount applied once classified). The last two constants are unrelated quantities that happen to share a value. Off unless container_col is supplied, and crawler-neutral: any crawler that can identify a link’s component can drive it.
New vignette("boilerplate") documents the detector: the container-conditioned metric and the direction of its ratio, the four arguments and why boilerplate_threshold and boilerplate_weight are unrelated despite sharing a default, how region weighting and recurrence compose as one graded axis rather than two multiplied ones, and which kinds of page the detector characteristically discounts.
New sf_container_from_path() derives that component identity from a Screaming Frog Link Path, and screaming_frog_links() now carries a container column on both the observation and edge tables (counted in the link diagnostics as container_rows). The rule is: strip numeric [n] predicates, keep [@class='…'], then drop the trailing <a> step. This cuts the opposite way from sf_region_from_path(), which strips class predicates so a div[@class='site-footer'] is not read as a <footer>; the two answer different questions (“which region is this” versus “is this the same component”) and the inconsistency is deliberate. Rows with no Link Path get NA and stay unscored — unlike placement there is no Link Position fallback, because a region label cannot manufacture a component identity.
pagerank_screaming_frog() does not enable the detector on its own. Pass container_col = "container" to opt in, exactly as a non-Screaming-Frog caller would; supplying the column is what switches detection on, so passing it automatically would change the default view.
Placement and recurrence are two detectors feeding one graded axis, not two independent axes. A nav link is boilerplate by construction, so the two factors are not multiplied — that would discount the same link twice for the same fact. The strongest applicable discount wins: chrome 0.1, repetitive in-content 0.5, unique in-content 1. The transition audit still records both factor sets separately in config$placement and config$boilerplate, since the resulting weight alone cannot say which detector produced it.
The internal synthetic weight column is renamed .__pr_placement_weight__ to .__pr_edge_weight__, since both weighting axes now write it. It surfaces in attr(result, "transition_audit")$config$weight_col.
Two new presets complete the set: "reversed" and "content". pr_preset() now registers raw · declared · reversed · content. "reversed" is the feeder view (reverse = TRUE); it is a no-op rather than an error under topic_feeder_pagerank(), which reverses the graph itself. "content" carries the placement recipe c(content = 1, nav = 0.1, header = 0.1, footer = 0.1, aside = 0.1), so links found in site chrome are discounted to a tenth of an in-content link — downweighted, never dropped. All five regions are named on purpose: unnamed placements keep weight 1, so a partial recipe would leave footer and aside outweighing nav tenfold. Because a preset sets policy and never data, "content" still needs you to supply placement_col; pagerank_screaming_frog() supplies it from the bundle.
New vignette("presets") documenting every preset’s full expansion, the precedence rule, and how provenance is recorded in the transition audit.
Asking for placement_weights or accepted_placements without placement_col now names the preset responsible when a preset is what set them, instead of reporting an argument the caller never typed.
Placement-aware scoring is now crawler-neutral. pagerank() gains placement_col, accepted_placements, and placement_weights: point placement_col at a column holding the page region each link sits in and the region can filter or weight edges directly, e.g. pagerank(edges, placement_col = "region", placement_weights = c(content = 1, nav = 0.1, header = 0.1, footer = 0.1, aside = 0.1)). Placement is not a Screaming Frog concept — any crawler that reports link regions can drive it, with a per-crawler adapter (sf_normalize_position() for Screaming Frog) mapping vendor labels onto the shared vocabulary. All three arguments default to NULL, so nothing changes for callers that do not use them, and because they are pagerank() formals they compose with preset and are inherited by every wrapper that forwards ....
A Screaming Frog bundle’s placement is now derived from the DOM path rather than from Link Position, via a new exported sf_region_from_path(). Link Position loses the enclosing region whenever a <nav> is nested inside one: on a site whose footer is marked up as footer > nav > a, Screaming Frog reports every footer link as Navigation and emits no Footer bucket at all, so footer is not merely mislabeled but unreachable — a user wanting footer at 0.05 and nav at 0.2 had no way to express it. Other sites do emit Footer, so the vocabulary silently varied with the site’s markup. The region is the outermost layout container on the path (header, footer, aside, else content), and nav applies only to a <nav> that is not inside one of those; Link Position remains the fallback for rows with no path, counted in the link diagnostics as placement_from_position_rows. Expect footer and header navigation to reclassify out of nav — this changes which edges accepted_placements selects, though not the usual recipe, which weights all three alike.
The placement vocabulary term "sidebar" is renamed "aside", matching both Screaming Frog’s own label and the HTML element; "sidebar" is a layout word rather than a semantic one. sf_normalize_position() now returns "aside" where it previously returned "sidebar", and "sidebar" is no longer accepted in accepted_placements / placement_weights.
pagerank_screaming_frog() keeps its signature but is now a thin adapter: accepted_placements and placement_weights are forwarded to pagerank(), which owns the filtering and weighting. Two consequences for the "screaming_frog_import" attribute: scoring$scored_edge_rows is renamed scoring$edge_rows_to_pagerank (it counts rows after the wrapper-owned link origin filter, since placement filtering has moved down a layer), and scoring$effective_weight_col is dropped — the transition audit already carries it, alongside a new config$placement entry recording the placement column, filter, weights, and the number of rows the filter dropped.
New preset argument on pagerank() and a new exported pr_preset() constructor: named argument bundles for recurring views of a link graph, so a view is a one-liner instead of a hand-assembled argument list. Two presets ship so far: "raw" (the graph exactly as crawled – self loops and isolates kept, rel=nofollow ignored, no relabeling onto uncrawled fold targets) and "declared" (honor the signals the site declares – nofollow evaporates, declared canonical and redirect targets are followed, robots-blocked pages keep the authority they collect, self loops and isolates dropped). "declared" is a pure pin of the package defaults: it changes nothing about how pagerank() behaves, it states the default view so a run can record which view was intended and stay pinned to that bundle if a default later moves. preset accepts a preset name, a pr_preset() result, or any hand-rolled named list of pagerank() arguments; bundles are plain named lists, so they are inspectable and spliceable via do.call(). Precedence is explicit argument > preset > base default – a preset value is applied only to arguments the caller did not name – and this holds through the wrappers that forward ... to pagerank() (trustrank(), topic_sensitive_pagerank(), topic_feeder_pagerank(), pagerank_screaming_frog()), with the boundary that arguments a wrapper sets itself stay wrapper-owned. Presets are strictly opt-in; no default behavior changes.
The transition_audit attached to a pagerank() result now records which preset produced it, in config$preset: the preset name for a registered preset (passed by name or as a pr_preset() result), "custom" for a hand-rolled bundle, and NULL when no preset was used. Two runs that expand to the same configuration are no longer indistinguishable in the audit – which named view was asked for is itself a provenance fact, and it is the only durable record of intent for a pin like "declared". print() shows a Preset: line only when a preset was used, so default output is unchanged.
Breaking: the six dot-prefixed Screaming Frog helpers are renamed without their leading dot and are now documented public API: .sf_contract(), .sf_read_input(), .sf_parse_follow(), .sf_rel_nofollow(), .sf_normalize_position(), and .sf_graph_eligible() become sf_contract(), sf_read_input(), sf_parse_follow(), sf_rel_nofollow(), sf_normalize_position(), and sf_graph_eligible(). A leading dot conventionally signals “internal”, but these were already exported – and exported with @noRd, so they had no help pages at all. They are intended for downstream Screaming Frog workflows, so they now carry full documentation (parameters, return values, examples) and are grouped as @family Screaming Frog toolkit with their own pkgdown reference section. Behavior and signatures are unchanged. The many genuinely internal .sf_* helpers keep their leading dot and remain unexported.
Breaking: resolve_urls() is renamed to resolve_redirect_urls(). The function originally resolved redirects and canonicals together; once that was split apart (mixing the two is lossy) the old name no longer described what it does. The new name states the operation and restores symmetry with the rest of the resolve_* family, which pairs an edge-list form with a URL-vector form: resolve_redirects() / resolve_redirect_urls() alongside the existing resolve_canonicals() / resolve_canonical_urls(). Behavior, arguments, and return value are unchanged.
The resolve_* naming convention is now documented and enforced by grouping: functions taking a character vector of URLs are tagged @family URL-vector resolvers (resolve_redirect_urls(), resolve_canonical_urls(), resolve_folded_urls()) and functions taking an edge_list_df are tagged @family edge-list resolvers (resolve_redirects(), resolve_canonicals(), resolve_links()), so each help page cross-links its input-type peers.
The weight/teleport transform vocabulary now uses one canonical option order ("none", "log", "percentile", "minmax", "zipf", "rank_linear") at every call site – transform_weights(), transform_edge_weights(), pagerank(prior_transform=), align_prior_to_vertices(), and ga4_entrance_teleport(). No defaults change: transform_edge_weights() still defaults to "zipf" (now declared explicitly rather than by option order) and every other site still defaults to "none".
Documented that canonical_profile() deliberately leaves rurl’s component-dropping knobs unpinned (query_handling, port_handling, and the url_standard selector added in rurl 2.2.0) – they have no effect on the scheme+host+path node key at their defaults. Added a behavioral guard in test-canonicalization.R asserting a canonical key drops the port, query, and fragment, so a future rurl default flip on an unpinned knob is caught here rather than silently changing node identity. No node keys change; verified against rurl 2.2.0.
canonical_profile() now pins path_normalization = "dot_segments" and path_encoding = "decode" (previously "none" / "keep"). rurl 2.1.0 silently redefined those two default values to keep the path verbatim, which changed node keys for any URL with dot-segments (/a/../b) or percent-encoding (/%41, %20) and desynced the pagerankr <-> semantic node join. Pinning the explicit values restores the original committed key (path percent-decoded, dot-segments removed) and keeps node identities stable across the rurl upgrade. The semantic sibling pins the identical profile (changed together).
clean_url_columns() now preserves tokens that rurl cannot parse as a URL (e.g. a dotless bare label such as "A") as their raw value instead of turning them into NA. Newer rurl (>= 2.1.0) normalizes such dotless tokens to NA; combined with the rurl floor bump in the follow-up, this had silently collapsed non-URL node identities to NA — they were then dropped by get_unique_edges(), so [pagerank()] returned an empty result for any graph built from bare labels. Unparseable-but-present tokens are now kept as opaque nodes (only genuinely missing NA inputs stay NA), mirroring .apply_fold_map()’s leave-unmapped-values-untouched contract.
filter_links_by_domain()’s encoding-independent registrable-domain matching now reads rurl’s new domain_ascii column (rurl >= 2.1.0) instead of a separate IDNA-forced parse. .build_url_maps() parses each unique URL once (host + domain_ascii) rather than twice, and the .domain_profile() forced-idna helper is gone. Behavior is unchanged — münchen.de and xn--mnchen-3ya.de still fold to one key under every host_encoding.
pagerank_screaming_frog() gains apply_canonicals and apply_redirects toggles (both TRUE by default, preserving current behavior). Setting either to FALSE skips folding the bundle’s canonical / redirect signals into [pagerank()] (passes canonicals_df / redirects_df as NULL), giving a supported as-crawled run that keeps the crawled node identities — the escape hatch for crawls whose canonicals point off the crawled domain (mirror / staging hosts) and would otherwise relabel crawled pages onto uncrawled targets. The reserved-argument guard still blocks the raw canonicals_df / redirects_df pagerank arguments. screaming_frog_bundle() now reports an off-domain canonical count (counts$canonicals_off_domain, surfaced in summary()/print()), reusing the existing absent-target classification, and the wrapper exposes it on the screaming_frog_import audit so the mirror-staging scenario is visible at import and scoring time.
pagerank() now detects fold-target collisions: when a canonical/redirect relabels a crawled page’s node onto an uncrawled URL that is ALSO independently referenced as a genuine link endpoint, the two silently merge into one vertex and the crawled page absorbs the inbound link equity of that uncrawled URL. pagerank() emits a warning() naming the merged URL(s) and records them in the fold section of the transition_audit object under a new collisions field (a data frame of target / n_independent_refs / source, or NULL when none). The crawl’s known-URL set (indexability_df) is used to tell an uncrawled fold target from a genuinely crawled leaf page, so the diagnostic is only computed when an indexability_df is supplied.
pagerank() now warns when a keep_domains / exclude_domains / keep_hosts / exclude_hosts value matched the crawled input but no node after folding — i.e. an out-of-scope canonical/redirect rewrote the crawled domain/host away before filtering (which runs after folding). The warning names the folded-away value(s) and points at the fold as the cause. The fold-then-filter ordering is now documented explicitly in the pagerank() and filter_links_by_domain() docs; to scope the crawled input, filter with filter_links_by_domain() before calling pagerank().
out_of_scope_fold gains a third policy, "leak": a crawled page whose canonical/redirect folds out of scope is treated like an external redirect — its inbound equity is routed onto a dedicated leak sink and evaporates out of the measured graph (its outbound edges are dropped), so it does not rank and its equity is not credited to any surviving page. The evaporated equity is reported as a new leaked term in the transition_audit mass accounting, which now decomposes as reported + sink + leaked + hidden = total (= 1); leaked is 0 for "relabel"/"keep" runs so their totals are unchanged. The fold audit section reports policy == "leak".
pagerank() gains an out_of_scope_fold argument ("relabel" default, or "keep") governing composed fold-map entries whose target is not itself a crawled node. "relabel" preserves current behavior (fold crawled sources onto uncrawled canonical/redirect targets); "keep" drops those out-of-scope entries before folding so crawled pages retain their as-crawled identity rather than being relabeled to phantom vertices (the same filtered map folds the TIPR prior). Regardless of policy, the count and list of out-of-scope folds (source, target, signal) are recorded in a new fold section of the transition_audit object.
Code-quality pass: anyNA() replaces any(is.na()), !all(x) replaces any(!x), nested ifelse() replaced with vectorized assignment, expect_gt()/expect_lt()/expect_length()/expect_null() adopted where applicable, and redundant c() wrappers around single-string aliases removed.
pagerank() and compute_pagerank() gain convergence controls and reporting. The new algo argument selects the igraph::page_rank() back-end ("prpack", the fast exact default, or "arpack", the iterative solver), and the friendly eps / niter aliases re-introduce the L1 tolerance and maximum iteration count that modern igraph dropped, mapping onto the ARPACK options$tol / options$maxiter; supplying either transparently switches to ARPACK. Every non-empty result now carries a "convergence" attribute (a pagerank_convergence object) reporting the solver, iteration count (when the solver exposes it), and a solver-independent post-hoc L1 residual ||Gx - x||_1 of the returned vector — a genuine quality check comparable across both back-ends. Docs cover the damping/iteration-count rule of thumb log10(eps) / log10(damping).
New topic_feeder_pagerank() answers the inverse of topic_sensitive_pagerank(): not “which page is most authoritative for this cluster” but “which pages feed / power this cluster” — the internal hubs whose outlinks point into the target pages. It seeds the teleport prior on the cluster and runs pagerank() on the transposed graph (reverse = TRUE), so mass walks backward along links and accumulates on the feeders, attenuating with link distance. This is the reverse-graph sibling of Topic-Sensitive PageRank and the cluster-biased counterpart to the global inverse PageRank (pagerank(reverse = TRUE)); the feeders are the high-pagerank rows with prior_weight == 0 (cluster pages carry teleport mass directly). Pure orchestration over the existing TIPR personalization path on the reversed graph — no new solver. New topic_feeder_pagerank vignette walks through the AI-Agent-cluster use case and contrasts it with the forward authority view and with HITS hubs.
New salsa() and compute_salsa() add Lempel & Moran’s (2001) SALSA hub and authority scores: a stochastic variant of HITS that runs the mutual-reinforcement step as PageRank-style random walks on the bipartite hub/authority graph, so the scores are stationary distributions rather than dominant eigenvectors. Computed over the same cleaned, redirect/canonical-folded, domain-filtered, deduplicated link graph as pagerank(), so hub, authority, and PageRank share node identities. Uses the degree-based closed form (Proposition 6) — no eigenvector iteration — with the required weakly-connected-component mass-weighting correction so cross-component scores stay comparable on crawls with orphan clusters. Each side sums to 1; coverage differs from PageRank by design (hub is NA for pure sinks, authority is NA for pure sources). v1 is unweighted; a weighted extension is deferred. Documented as a site-graph adaptation of the original focused-subgraph algorithm.
New trustrank() and the shared seed_prior() builder add TrustRank-style seed-biased PageRank (Gyöngyi, Garcia-Molina & Pedersen, 2004): personalized PageRank whose teleport vector is concentrated on a set of trusted seed pages, so trust flows outward from the seeds and attenuates with distance (the damping factor is the attenuation). seed_prior() builds a prior_df from a seed set (character vector or weighted data.frame; equal weights reproduce the original uniform seed distribution) and is orientation-agnostic — the same builder feeds topic_feeder_pagerank() on the reversed graph. trustrank() is the one-call wrapper that builds the seed prior and runs pagerank() with it. Pure orchestration over the existing TIPR personalization path — no new solver; seed selection is the caller’s (it is seed-biased PageRank, not a spam classifier). New trustrank vignette walks through a worked example.
New topic_sensitive_pagerank() computes per-topic PageRank by running the existing pagerank() engine once per topic with a teleport prior biased toward each topic’s seed cluster, then blends the per-topic scores into a single ranking. This is Haveliwala’s (2002) Topic-Sensitive PageRank adapted to a single site: each “topic” is a content cluster (e.g. the pricing or AI-Agent area) given as a character vector of seed URLs or a weighted data.frame. Returns one score column per topic plus a weight-normalized blended column, with the per-topic transition_audit objects attached. Pure orchestration over the TIPR personalization path — no new solver, and topic membership is supplied by the caller, not inferred.
New smooth_transitions() shrinks sparse empirical page-transition shares (e.g. from ga4_page_transitions()) toward the crawl-graph link structure, so no valid crawled link is ever assigned exactly zero probability. Uses a per-source Dirichlet/pseudocount shrinkage weight lambda_i = n_i / (n_i + k) that increases with the source page’s sample size, with min_support fallback to the prior, optional weighted priors, and an origin diagnostic (both / empirical_only / structural_only). Time decay and device/template/channel segmentation are handled upstream by shaping the count input.
New hits() and compute_hits() add Kleinberg’s HITS hub and authority scores, computed with igraph::hits_scores() over the same cleaned, redirect/canonical-folded, domain-filtered, deduplicated link graph as pagerank(), so hub, authority, and PageRank share node identities. Docs cover the matrix formulation (authority = dominant eigenvector of A^T A, hub = dominant eigenvector of A A^T) and the whole-graph caveat: unlike Kleinberg’s query-focused base set, these are site-wide structural centralities.
pagerank() now has an explicit duplicate_edge_policy for repeated from -> to rows after URL folding and filtering. The default "collapse" preserves the standard binary/destination-level PageRank convention and previous results; opt-in "aggregate" sums duplicate numeric weights with aggregate_edges() semantics; opt-in "count_instances" models a link-slot surfer where repeated links increase transition probability and records instance-count details in the transition audit.
Recorded representative Screaming Frog crawl acceptance results and added package-level operational documentation for required exports, default graph policy, optional origin/placement policies, and contract pinning.
New pagerank_screaming_frog() scores a screaming_frog_bundle() through the existing pagerank() pipeline, feeding only graph-eligible hyperlink edges while attaching Screaming Frog import diagnostics beside the transition audit. Placement/origin filtering and placement-derived weighting are explicit opt-ins.
New screaming_frog_bundle() composes Internal: All and All Inlinks/Outlinks adapters into the stable crawl handoff object with raw observations, graph edges, node signals, cross-table reconciliation diagnostics, provenance, and concise print/summary methods.
New canonical and composed URL-resolution helpers: resolve_canonicals(), resolve_canonical_urls(), and resolve_folded_urls() expose the existing fold-map engine for rel=canonical and redirect+canonical URL folding without duplicating resolver logic.
Documented the indexed-corpus assumption used by pagerank(): noindex pages may receive authority but their outlinks are treated as nofollow for propagation within the indexed graph. The docs now distinguish slot-consuming "evaporate", slot-removing "drop", and normally followed "keep" without attributing this package model to Google.
New screaming_frog_links() imports All Inlinks and All Outlinks with identical Source-to-Destination orientation, preserving raw duplicate observations while deriving explicit Hyperlink-only graph edges with nofollow, placement, origin, endpoint, and exclusion diagnostics.
New screaming_frog_internal() imports UTF-8/BOM Internal: All exports with alias-insensitive schema detection and selective file reads. It returns deterministic node, redirect, canonical, and indexability tables while preserving raw URLs and reporting missing, duplicate, invalid, and ignored input facts.
pagerank() now attaches a transition_audit provenance object to its result as attr(result, "transition_audit") (backward-compatible): row/edge counts, behavioral-weight coverage, normalization total, dropped data (NA / dedup / self-loop rows, unmatched prior URLs), robots-blocked count, and the model configuration used. Has a print method.
The transition_audit object’s mass field now decomposes the page-mass deficit precisely into reported (visible page mass), sink (evaporated nofollow-sink mass), hidden (robots-blocked mass), and total (= 1 by construction) — replacing undifferentiated “leakage” language with precise evaporated/hidden accounting.
New aggregate_edges(): loss-aware post-fold edge aggregation with explicit per-column semantics (sum counts, boolean conflict policy any/all/ majority/error, preserve_cols list-columns for placement features).
New transform_edge_weights(): per-source grouped weight transforms reusing transform_weights() methods, emitting a per-source transition_probability that sums to 1 within each by group.
New ga4_page_transitions(): consecutive-page-view transition counts from a GA4 BigQuery export with a deterministic session/event ordering contract (timestamp + batch tie-breaks). A transition signal, not a link-click signal.
New ga4_entrance_teleport(): entrance/landing-page counts as a teleport (reset) vector reusing the prior_df / align_prior_to_vertices() machinery; documented as a proxy, distinct from the backlink-authority prior.
Initial CI and lint baseline.

Internal

The OSS Index dependency audit in tests/testthat/test-security.R scopes to hard dependencies (Depends + Imports) instead of the Suggests tree. oysteR::expect_secure() audits Suggests too, which pulled in oysteR’s own recursive dependencies – curl among them – and failed the pre-push gate on a vulnerability in the auditor rather than in anything pagerankr ships. Scoped to hard dependencies the audit covers 23 packages and is clean; the old scope covered 126 (PAGE-aqnbdkov).
CI adds CRAN behind the image’s Posit Package Manager repo. p3m lags CRAN by up to a day for a new release – on the day rurl 3.0.1 was published, the 2026-09-09 p3m snapshot and p3m latest both still served 1.2.0 – so Imports: rurl (>= 3.0.1) was unresolvable at any pin and every dependency-resolving job failed. p3m stays first, so packages still arrive as prebuilt binaries; CRAN only supplies what p3m has not synced yet.
The rurl-floor CI job resolves the floor from CRAN (current or archived) instead of reading the source forge out of Remotes: and installing a git tag. It now runs on every merge request rather than manually, because it can pass. Its GitHub-hosted twin, which had no live runner and hard-failed on the removed Remotes: field, was deleted; the GitLab job supersedes it.
The pre-push hook’s non-gating rurl skew notice compares the installed rurl against CRAN rather than against the Remotes: target. The skew it warns about is not hypothetical: a local 3.0.0 build sat in front of CRAN 3.0.1 here and hid a new get_clean_url argument from the suite.