🐿 linkinator
A super simple site crawler and broken link checker.
Behold my latest inator! The linkinator
provides an API and CLI for crawling websites and validating links. It's got a ton of sweet features:
- 🔥 Easily perform scans on remote sites or local files
- 🔥 Scan any element that includes links, not just
<a href>
- 🔥 Supports redirects, absolute links, relative links, all the things
- 🔥 Configure specific regex patterns to skip
- 🔥 Scan markdown files without transpilation
Installation
Node.js / npm
npm install linkinator
Standalone Binaries (no Node.js required)
Don't have Node.js installed? No problem! Browse all releases at github.com/JustinBeckwith/linkinator/releases.
These binaries are completely standalone - no runtime dependencies needed. Just download, make executable (Linux/macOS), and run!
Command Usage
You can use this as a library, or as a CLI. Let's see the CLI!
$ linkinator LOCATIONS [ --arguments ]
Positional arguments
LOCATIONS
Required. Either the URLs or the paths on disk to check for broken links.
Supports multiple paths, and globs.
Flags
--concurrency
The number of connections to make simultaneously. Defaults to 100.
--config
Path to the config file to use. Looks for `linkinator.config.json` by default.
--directory-listing
Include an automatic directory index file when linking to a directory.
Defaults to 'false'.
--format, -f
Return the data in CSV or JSON format.
--help
Show this command.
--markdown
Automatically parse and scan markdown if scanning from a location on disk.
--recurse, -r
Recursively follow links on the same root domain.
--retry,
Automatically retry requests that return HTTP 429 responses and include
a 'retry-after' header. Defaults to false.
--retry-errors,
Automatically retry requests that return 5xx or unknown response.
--retry-errors-count,
How many times should an error be retried?
--retry-errors-jitter,
Random jitter applied to error retry.
--server-root
When scanning a locally directory, customize the location on disk
where the server is started. Defaults to the path passed in [LOCATION].
--skip, -s
List of urls in regexy form to not include in the check. Can be repeated multiple times.
--timeout
Request timeout in ms. Defaults to 0 (no timeout).
--url-rewrite-search
Pattern to search for in urls. Must be used with --url-rewrite-replace.
--url-rewrite-replace
Expression used to replace search content. Must be used with --url-rewrite-search.
Example: --url-rewrite-search "https://example\.com" --url-rewrite-replace "http://localhost:3000"
--user-agent
The user agent passed in all HTTP requests. Defaults to 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.117 Safari/537.36'
--verbosity
Override the default verbosity for this command. Available options are
'debug', 'info', 'warning', 'error', and 'none'. Defaults to 'warning'.
Command Examples
You can run a shallow scan of a website for busted links:
npx linkinator https://jbeckwith.com
That was fun. What about local files? The linkinator will stand up a static web server for yinz:
npx linkinator ./docs
But that only gets the top level of links. Lets go deeper and do a full recursive scan!
npx linkinator ./docs --recurse
Aw, snap. I didn't want that to check those links. Let's skip em:
npx linkinator ./docs --skip www.googleapis.com
Need to skip multiple patterns? Just use --skip
multiple times:
npx linkinator ./docs --skip www.googleapis.com --skip example.com --skip github.com
The --skip
parameter will accept any regex! You can do more complex matching, or even tell it to only scan links with a given domain:
npx linkinator http://jbeckwith.com --skip '^(?!http://jbeckwith.com)'
Maybe you're going to pipe the output to another program. Use the --format
option to get JSON or CSV!
npx linkinator ./docs --format CSV
Let's make sure the README.md
in our repo doesn't have any busted links:
npx linkinator ./README.md --markdown
You know what, we better check all of the markdown files!
npx linkinator "**/*.md" --markdown
Configuration file
You can pass options directly to the linkinator
CLI, or you can define a config file. By default, linkinator
will look for a linkinator.config.json
file in the current working directory.
All options are optional. It should look like this:
{
"concurrency": 100,
"config": "string",
"recurse": true,
"skip": "www.googleapis.com",
"format": "json",
"silent": true,
"verbosity": "error",
"timeout": 0,
"markdown": true,
"serverRoot": "./",
"directoryListing": true,
"retry": true,
"retryErrors": true,
"retryErrorsCount": 3,
"retryErrorsJitter": 5,
"urlRewriteSearch": "https://example\\.com",
"urlRewriteReplace": "http://localhost:3000",
"userAgent": "Mozilla/4.0 (compatible; MSIE 6.0; MSIE 5.5; Windows NT 5.1)"
}
For skipping multiple URL patterns, use an array:
{
"skip": ["www.googleapis.com", "example.com", "github.com"]
}
To load config settings outside the CWD, you can pass the --config
flag to the linkinator
CLI:
npx linkinator --config /some/path/your-config.json
GitHub Actions
You can use linkinator
as a GitHub Action as well, using JustinBeckwith/linkinator-action:
on:
push:
branches:
- main
pull_request:
name: ci
jobs:
linkinator:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- uses: JustinBeckwith/linkinator-action@v1
with:
paths: README.md
To see all options or to learn more, visit JustinBeckwith/linkinator-action.
API Usage
linkinator.check(options)
Asynchronous method that runs a site wide scan. Options come in the form of an object that includes:
path
(string|string[]) - A fully qualified path to the url to be scanned, or the path(s) to the directory on disk that contains files to be scanned. required.concurrency
(number) - The number of connections to make simultaneously. Defaults to 100.port
(number) - When thepath
is provided as a local path on disk, theport
on which to start the temporary web server. Defaults to a random high range order port.recurse
(boolean) - By default, all scans are shallow. Only the top level links on the requested page will be scanned. By settingrecurse
totrue
, the crawler will follow all links on the page, and continue scanning links on the same domain for as long as it can go. Results are cached, so no worries about loops.retry
(boolean|RetryConfig) - Automatically retry requests that respond with an HTTP 429, and include aretry-after
header. TheRetryConfig
option is a placeholder for fine-grained controls to be implemented at a later time, and is only included here to signal forward-compatibility.serverRoot
(string) - When scanning a locally directory, customize the location on disk where the server is started. Defaults to the path passed inpath
.timeout
(number) - By default, requests made by linkinator do not time out (or follow the settings of the OS). This option (in milliseconds) will fail requests after the configured amount of time.markdown
(boolean) - Automatically parse and scan markdown if scanning from a location on disk.linksToSkip
(array | function) - An array of regular expression strings that should be skipped (e.g.,['example.com', 'github.com', '^http://']
), OR an async function that's called for each link with the link URL as its only argument. Return a Promise that resolves totrue
to skip the link orfalse
to check it.directoryListing
(boolean) - Automatically serve a static file listing page when serving a directory. Defaults tofalse
.urlRewriteExpressions
(array) - Collection of objects that contain a search pattern, and replacement. Use this to rewrite URLs before they are checked. For example, to rewrite a production URL to a local development URL:urlRewriteExpressions: [ { pattern: /https:\/\/example\.com/, replacement: 'http://localhost:3000' } ]
userAgent
(string) - The user agent that should be passed with each request. This uses a reasonable default.
linkinator.LinkChecker()
Constructor method that can be used to create a new LinkChecker
instance. This is particularly useful if you want to receive events as the crawler crawls. Exposes the following events:
pagestart
(string) - Provides the url that the crawler has just started to scan.link
(object) - Provides an object withurl
(string) - The url that was scannedstate
(string) - The result of the scan. Potential values includeBROKEN
,OK
, orSKIPPED
.status
(number) - The HTTP status code of the request.
Examples
Simple example
import { LinkChecker } from 'linkinator';
async function simple() {
const checker = new LinkChecker();
const results = await checker.check({
path: 'http://example.com'
});
// To see if all the links passed, you can check `passed`
console.log(`Passed: ${results.passed}`);
// Show the list of scanned links and their results
console.log(results);
// Example output:
// {
// passed: true,
// links: [
// {
// url: 'http://example.com',
// status: 200,
// state: 'OK'
// },
// {
// url: 'http://www.iana.org/domains/example',
// status: 200,
// state: 'OK'
// }
// ]
// }
}
simple();
Complete example
In most cases you're going to want to respond to events, as running the check command can kinda take a long time.
import { LinkChecker } from 'linkinator';
async function complex() {
// create a new `LinkChecker` that we'll use to run the scan.
const checker = new LinkChecker();
// Respond to the beginning of a new page being scanned
checker.on('pagestart', url => {
console.log(`Scanning ${url}`);
});
// After a page is scanned, check out the results!
checker.on('link', result => {
// check the specific url that was scanned
console.log(` ${result.url}`);
// How did the scan go? Potential states are `BROKEN`, `OK`, and `SKIPPED`
console.log(` ${result.state}`);
// What was the status code of the response?
console.log(` ${result.status}`);
// What page linked here?
console.log(` ${result.parent}`);
});
// Go ahead and start the scan! As events occur, we will see them above.
const result = await checker.check({
path: 'http://example.com',
// port: 8673,
// recurse: true,
// Skip multiple URL patterns using an array of regex strings
// linksToSkip: [
// 'example.com/skip-this',
// 'github.com',
// '^https://restricted'
// ]
});
// Check to see if the scan passed!
console.log(result.passed ? 'PASSED :D' : 'FAILED :(');
// How many links did we scan?
console.log(`Scanned total of ${result.links.length} links!`);
// The final result will contain the list of checked links, and the pass/fail
const brokeLinksCount = result.links.filter(x => x.state === 'BROKEN');
console.log(`Detected ${brokeLinksCount.length} broken links.`);
}
complex();
Skipping links example
import { LinkChecker } from 'linkinator';
async function skipExample() {
const checker = new LinkChecker();
// Skip multiple URL patterns using an array
const result = await checker.check({
path: 'https://example.com',
recurse: true,
linksToSkip: [
'www.google.com', // Skip all Google links
'example.com/skip-me', // Skip specific paths
'^https://internal.corp' // Skip all internal corp links
]
});
console.log(`Scanned ${result.links.length} links`);
}
skipExample();
Tips & Tricks
Using a proxy
This library supports proxies via the HTTP_PROXY
and HTTPS_PROXY
environment variables. This guide provides a nice overview of how to format and set these variables.
Globbing
You may have noticed in the example, when using a glob the pattern is encapsulated in quotes:
npx linkinator "**/*.md" --markdown
Without the quotes, some shells will attempt to expand the glob paths on their own. Various shells (bash, zsh) have different, somewhat unpredictable behaviors when left to their own devices. Using the quotes ensures consistent, predictable behavior by letting the library expand the pattern.
Debugging
Oftentimes when a link fails, it's an easy to spot typo, or a clear 404. Other times ... you may need more details on exactly what went wrong. To see a full call stack for the HTTP request failure, use --verbosity DEBUG
:
npx linkinator https://jbeckwith.com --verbosity DEBUG
Controlling Output
The --verbosity
flag offers preset options for controlling the output, but you may want more control. Using jq
and --format JSON
- you can do just that!
npx linkinator https://jbeckwith.com --verbosity DEBUG --format JSON | jq '.links | .[] | select(.state | contains("BROKEN"))'
Cloudflare Bot Protection
Some websites use Cloudflare bot protection, which may return a 403
status code with a JavaScript challenge page when detecting automated tools. Linkinator automatically detects this scenario by checking for the cf-mitigated
response header.
When Cloudflare bot protection is detected, linkinator treats the link as valid (status 200
, state OK
) rather than broken. This is because:
- The link itself is valid and works for human users
- The
403
is specifically for bot traffic, not a broken or missing page - Marking it as broken would create false positives in your link checks
If you need to verify these links work for actual users, test them manually in a browser.