What you’ll learn in this article…
- World Digital Preservation Day falls on the first Thursday of each November
- Roughly 38 percent of 2013 webpages were gone by 2023, Pew found
- AIP's library uses Archive-It; a UMD MLIS intern built the collection
Inside a research library's web-archiving program — and the MLIS skills behind it

How do research libraries keep a scientific society's website from vanishing when its server goes dark?
World Digital Preservation Day, the first Thursday of November, is organized by the Digital Preservation Coalition and runs less as a celebration than an audit deadline. AIP's Niels Bohr Library & Archives treats it that way: its Archive-It program captured FYI: Science Policy News on October 6, 2025, and Northwestern's physics department site on August 20, 2026.
Scale is the obstacle. Small societies and individual scientists run sites with no backup infrastructure, and a crawl schedule is all that stands between their work and a dead link.
Awareness days are easy to ignore, which is why the fixed date matters more than the banner. World Digital Preservation Day is held on the first Thursday of every November, organized by the Digital Preservation Coalition with support from digital preservation networks worldwide, and in 2026 it falls on Thursday, November 5.
Each edition is built around a theme, and the 2026 framing is "Save our Stories (SOS): Our digital heritage in danger," which DPC describes as a call to protect records, research, memories, and cultural assets. The coalition does not publish a single headline tally of events or countries; its 2026 global events page instead lists individual activities, among them work by Cambridge University Libraries and Archives.
Strip away the branding and the day's ask is operational: know which formats you hold, know who is responsible for each one, and write down what is at risk before a server is retired or a domain lapses. That is an audit with a calendar date attached, not a celebration.
Web archiving matters here because so much of the scientific record no longer lives in a vault. It lives on a lab site, a small society's news page, or a retired program's homepage, where a lapsed hosting bill can erase years of work. The day names the risk. Crawling is how institutions answer it.
Pew Research Center found that roughly 38 percent of webpages that existed in 2013 were no longer accessible by 2023, and broken links hit 23 percent of news pages and 21 percent of government pages.1 A 2014 study of more than a million URL-bearing references put link rot at 66 percent for Elsevier STM articles and 80 percent for PubMed Commons items.
This is routine, not crisis. AIP retired its Inside Science news service, and its site was captured on June 12, 2024. The IRIS consortium's retired site was captured on August 6, 2026, alongside NBLA's acquisition of its archives.
Research datasets, scientific software, and instrument data each carry separate preservation needs and repositories, from domain data repositories following FAIR practice to Software Heritage for source code and development history.
The stakes are professional: a citation to a dead URL is an unverifiable claim, and libraries are the institutions expected to have prevented it.
The dates below come from the web archiving program at the Niels Bohr Library & Archives at AIP, which runs on Archive-It, the Internet Archive service. Read the mix of organizations rather than any single row: a national society, one academic department, a small society homepage, a multi-university consortium, and a single professor's teaching site. That spread is the program's collection policy in practice.
| Website | Organization Type | Capture Date | Why It Matters |
|---|---|---|---|
| AIP FYI: Science Policy News | Professional society news publication | October 6, 2025 | In-house pilot capture: policy reporting published online with no print counterpart |
| Society of Physics Students (retired site) | Student program of a professional society | March 23, 2024 | Retired student-program site: content was taken down rather than migrated |
| AIP Inside Science (retired program site) | Society-run science news service | June 12, 2024 | A news service the publisher itself wound down, a common trigger for loss |
| Society of Rheology (retired homepage) | Small professional society | August 18, 2025 | Small society without separate archiving staff; a retired homepage captured before it vanished |
| American Association of Physicists in Medicine | Professional membership society | June 2, 2026 | Active society site captured on a regular schedule, not only after a shutdown |
| Northwestern University Department of Physics and Astronomy | University academic department | August 20, 2026 | A single department site: smaller scale than a society, identical risk of link rot |
| IRIS (retired site) | Nonprofit consortium of over 100 U.S. universities | August 6, 2026 | Captured alongside NBLA's acquisition of a larger IRIS archival collection, tying web capture to collection acquisition |
| Physics Demos by John Johnston, The Faraday Center | Individual scientist's teaching site | June 3, 2026 | Donated by the creator: a one-person teaching resource with no institutional backup |
The tooling question now is less about whether to crawl than how much of a page you can honestly claim to have saved. Archive-It, the Internet Archive's fee-based subscription service, is where most research libraries start: curators build separate collections, set crawl frequency across a daily-to-annual range, and store captures in Internet Archive data centers with backup copies, publishing them publicly or restricting access.
WARC, standardized as ISO 28500, keeps the HTTP responses, headers, timestamps, URLs, redirects, and MIME types that let a page render again years later. A PDF flattens a document; a screenshot freezes one viewport at one moment. Both are useful supplements, not substitutes.
Librarians set seeds, scope rules, exclusions, frequency, and duration, then run test crawls (up to three days, 1,000,000 documents) before spending budget.1 Post-capture QA means reading the downloadable crawl reports: seed-versus-captured counts, out-of-scope pulls, failed resources, and embedded-asset replay.2 Expect blind spots in JavaScript-heavy pages, forms, maps, streaming media, and login-gated content, and document them.
Robots.txt is a publisher instruction, not an access control, and Archive-It can be configured to override it.3 That option adds risk without resolving copyright or privacy questions, so scope, fair use for access copies, and takedown or donor restrictions remain curator decisions.
Northwestern University's Department of Physics and Astronomy website entered the Niels Bohr Library & Archives crawl on August 20, 2026. The service that captured it, Archive-It, is the easy part. The judgment is a written collection-development policy that states what is in scope, what is out, and why.
Run four questions against any candidate site:
AIP's program sorts targets into three tiers: institutional sites (AIP's own properties, the American Association of Physicists in Medicine, Northwestern's department), small-society sites such as the Society of Rheology's retired homepage, and individual scientist pages. The smallest are usually the most endangered, because no IT department is coming to save them.
Two other routes bring material in: capturing a retired consortium site in step with an acquired archival collection, or accepting a personal site donated by its creator. Just as important is saying no. Scope creep is what kills pilots, and a written policy is what lets a program defend its boundaries later.
The digital preservation job market follows a pattern that catches new graduates off guard: the skills are in demand, but the stable line item to fund them is often missing.
Ask a room of practitioners what they do and you will hear a dozen titles: digital archivist, digital preservation librarian, web archiving archivist, digital collections specialist, repository manager. The Bureau of Labor Statistics folds most of them into one category, archivists, which showed a median annual wage of $64,550 in May 2025 and a mean of $70,640 across roughly 7,970 jobs.1 The wider archivists, curators, and museum workers group spans $37,030 to over $101,400 and is projected to grow 4%, about 1,600 jobs, from 2025 to 2035.2 Pay shifts with how an employer classifies the post: archivist, librarian, or IT.
Metadata standards and PREMIS, comfort at the command line, crawl configuration, file-format identification, project management, and the diplomacy of asking a donor for written permission. Employers span university libraries and archives, museums, national and state libraries, research consortia, and preservation vendors.
Many first roles are project-funded and fixed-term, tied to a grant cycle. That is the argument for treating an internship as a portfolio, not a line on a resume.
The University of Maryland's MLIS field study is the structure worth copying: 3 credits, 120 supervised hours (LBSC707), run in 14-week spring and fall terms plus a shorter summer session.1 At NBLA, MLIS student Margaret Stine was handed a real scoping problem rather than a shadowing assignment: survey AIP's Oral History Interview collection, identify the gaps, and propose a new scientist-websites collection. Her scoping work became program infrastructure.
Ask for a defined output, access to the crawl account itself, scoping or quality-review responsibility, and a written record of what you built. When you talk to employers, name the tools, the standards, the size of the collection, and the selection decisions you made. Hours logged describe attendance; decisions describe competence.
You do not need a vendor contract or a dedicated preservation department to start. The Digital Preservation Coalition's first-step guidance is to assess what you hold, what it depends on, and what resources you can realistically commit, then work up from there. The items below are ordered that way, and the NDSA Levels of Digital Preservation give you a four-step maturity ladder to climb at your own pace: protect the data, then gain knowledge of it, then monitor it, then repair it through migration when formats fail.