ABCDEF - The 6 key features behind scalable, multi-tenant web archive processing with ARCH: Archive, Big Data, Concurrent, Distributed, Efficient, Flexible

dc.contributor.authorHolzmann, Helge
dc.contributor.authorRuest, Nick
dc.contributor.authorBailey, Jefferson
dc.contributor.authorDempsey, Alex
dc.contributor.authorFritz, Samantha
dc.contributor.authorLee, Peggy
dc.contributor.authorMilligan, Ian
dc.date.accessioned2022-04-28T19:29:58Z
dc.date.available2022-04-28T19:29:58Z
dc.date.issued2022-06-20
dc.description.abstractOver the past quarter-century, web archive collection has emerged as a user-friendly process thanks to cloud-hosted solutions such as the Internet Archive’s Archive-It subscription service. Despite advancements in collecting web archive content, no equivalent has been found by way of a user-friendly cloud-hosted analysis system. Web archive processing and research require significant hardware resources and cumbersome tools that interdisciplinary researchers find difficult to work with. In this paper, we identify six principles - the ABCDEFs (Archive, Big data, Concurrent, Distributed, Efficient, and Flexible) - used to guide the development and design of a system. These make the transformation of, and working with, web archive data as enjoyable as the collection process. We make these objectives – largely common sense – explicit and transparent in this paper. They can be employed by every computing platform in the area of digital libraries and archives and adapted by teams seeking to implement similar infrastructures. Furthermore, we present ARCH (Archives Research Compute Hub), the first cloud-based system designed from scratch to meet all of these six key principles. ARCH is an interactive interface, closely connected with Archive-It, engineered to provide analytical actions, specifically generating datasets and in-browser visualizations. It efficiently streamlines research workflows while eliminating the burden of computing requirements. Building off past work by both the Internet Archive (Archive-It Research Services) and the Archives Unleashed Project (the Archives Unleashed Cloud), this merged platform achieves a scalable processing pipeline for web archive research. It will be made open-source shortly and can be considered a reference implementation of the ABCDEF, which we have evaluated and discussed in terms of feasibility and compliance as a benchmark for similar platforms.en_US
dc.description.sponsorshipThis research was supported by the Andrew W. Mellon Foundation's Public Knowledge program, the Social Sciences and Humanities Research Council of Canada, as well as Start Smart Labs, the University of Waterloo, and York University.en_US
dc.identifier.isbn978-1-4503-9345-4/22/06
dc.identifier.urihttps://doi.org/10.1145/3529372.3530916en_US
dc.identifier.urihttp://hdl.handle.net/10315/39458
dc.language.isoenen_US
dc.publisherACMen_US
dc.rightsAttribution 4.0 International*
dc.rights.articlehttps://doi.org/10.1145/3529372.3530916en_US
dc.rights.journalhttps://2022.jcdl.org/en_US
dc.rights.urihttp://creativecommons.org/licenses/by/4.0/*
dc.subjectbig dataen_US
dc.subjectweb archivesen_US
dc.subjectinternet archiveen_US
dc.subjectdata processingen_US
dc.subjectapache sparken_US
dc.subjectscalaen_US
dc.subjectscalatraen_US
dc.subjectcomputing infrastructureen_US
dc.subjecthadoopen_US
dc.subjecthdfsen_US
dc.titleABCDEF - The 6 key features behind scalable, multi-tenant web archive processing with ARCH: Archive, Big Data, Concurrent, Distributed, Efficient, Flexibleen_US
dc.typeArticleen_US

Files

Original bundle
Now showing 1 - 1 of 1
Loading...
Thumbnail Image
Name:
JCDL_2022___ABCDEF_Paper.pdf
Size:
3.07 MB
Format:
Adobe Portable Document Format
Description:
License bundle
Now showing 1 - 1 of 1
No Thumbnail Available
Name:
license.txt
Size:
1.83 KB
Format:
Item-specific license agreed upon to submission
Description: