Save websites before they disappear with the Browsertrix Crawler tool

print-browsertrix-tutorial

Last updated on September 14th, 2024 at 10:07 pm

The month of September marks the beginning of a year’s work and also the end of many websites that are hopelessly lost. Remodelled or shut down without making a good copy of their content, this is how historic websites are lost unnecessarily.

There are tools that allow websites to be saved immediately by the organisations that manage them. In addition, there is the on-demand archiving service for high-quality websites that Arquivo.pt provides to partner organisations or in occasional collaborations.

This article aims to highlight the Browsertrix Crawler used by Arquivo.pt, without excluding other tools, which can be useful to information managers and IT departments.

Use of Browsertrix-crawler by Arquivo.pt for high-quality collections

Browsertrix Crawler is a tool that lets you record entire websites and lists of web pages automatically and in a format compatible with web archives.

Arquivo.pt uses the Browsertrix Crawler to make high-quality site collections (RAQs) on-demand of the community. For example, when a site is about to be shut down, when it’s going to undergo remodelling or, periodically, to maintain a good history of a particular site.

An illustrative case is the Almada City Council website, recorded in April 2021 at the request of the Municipal Archive. Another case is the website of the newspaper Notícias de Leiria, which was recorded before its closure in December 2023.

Requests for high-quality collections (RAQs) to Arquivo.pt are increasingly frequent: 77 requests from January to September 2024. This is a sign that there is greater concern about the preservation of web content.

What you need to use Browsertrix-crawler locally

The group that developed the Browsertrix Crawler, Webrecorder.net, led by Ilya Kreymer, has the motto ‘web archiving for all’. Its tools make it possible to record the Internet in a decentralised way and on a small scale.

The Browsertrix Crawler is available and can be installed on your computer for small collections.

The basic version of Browsertrix that Arquivo.pt is using requires basic command line knowledge, which is the only barrier for non-experts.

From Arquivo.pt’s own experience, using the Browsertrix Crawler is easy in multidisciplinary teams, where there is always someone with minimal knowledge to use Linux commands and provide occasional support.

Demonstration of recording entire websites on your own computer

To promote the preservation of sites in Web archive format, Arquivo.pt presents a use case for the Browsertrix Crawler. It’s useful for anyone who wants to deepen their knowledge and practice of saving sites in a local environment.

Other tools used by Arquivo.pt to record content

Brozzler: a tool for improving the history of daily and monthly collection sites

Brozzler is a similar tool to Browsertrix Crawler in that it also bases its recording on a browser. It is used and maintained by the Internet Archive.

Arquivo.pt has been using Brozzler since at least 2018 to record web pages with interactive content present on the web pages and for high-quality collections (RAQs).

Lists of up to 200 sites are successfully recorded by Brozzler. For example, the 125 daily collection sites (FAWP) are recorded with Brozzler at the beginning of each month.During the month, another list of 75 monthly collection sites (MAWP) is recorded using Brozzler.

At the end of 2023, Arquivo.pt compared Brozzler and Browsertrix Crawler and chose to keep these two tools.

Heritrix, pywb and ArchiveWeb.page: tools for thousands of sites or one page

The Heritrix crawler is Arquivo.pt’s main recording tool. It is used on huge lists of websites, such as the .PT domain sites, to which other Portuguese sites are added, totalling  more than half a million.

On the opposite side is the ArchiveWeb.page extension, which Arquivo.pt uses for short page-by-page recordings and also for the Web archiving: do-it-yourself! training course.

To complete the list of recording tools used by Arquivo.pt, mention should be made of pywb, which comes into play, for example, when an Arquivo.pt user uses the ‘Complete the page’ functionality or the SavePageNow service.

2024 European and Portuguese elections in special Arquivo.pt collections

European Elections

Last updated on October 9th, 2024 at 05:48 pm

Arquivo.pt made special collections on the three elections that took place this year: the Parlamentary elections on 10 March, the elections in Madeira island on 26 May and the European elections on 9 June.

More than 70,000 pages with content related to the elections and political life in Portugal and Europe were identified and around 4 terabytes of information collected.

We would like to thank the people who contributed to the selection of pages. Teachers and students are encouraged to do work using the special collections on elections that Arquivo.pt has produced over the years.

Find out more about the collection procedure and the results obtained.

Portuguese Parlamentary Elections (Legislativas 2024)

The Portuguese Parlamentary Elections  took place on 10 March 2024 to elect the members of the Assembly of the Republic for the 16th Legislature of the Third Portuguese Republic.

We would like to highlight the community’s contribution to this collection with a manual selection of 827 pages, which helped to improve the quality of the collection.

Around 500 compound terms or keywords were used to search for content published on the web about the elections. The service used for the automatic search was the Bing Search API. The results were limited to the top 20.

For example, the compound term ‘head-to-head legislative 2024’ found pages relating to debates between candidates. The term ‘legislative housing 2024’ found pages relating to party proposals for housing. The term ‘legislativas 2024 site:expresso.pt’ identified Expresso pages about the elections. The names of the candidates were also used.

After the elections, search terms specific to that period were used, such as ‘legislative victory 2024’, ‘legislative defeat 2024’ or ‘legislative results 2024’, among others.

The automatic search in the Bing Search API resulted in 34,120 addresses obtained before the elections and 5,803 after the elections.

The websites of political parties, including parties without parliamentary seats, were also collected during the election period.

Not all the content identified could actually be recorded, due to the limitations of the recording tools or the restrictions of the websites themselves.

The tools Heritrix, Brozzler and Browsertrix-cloud (beta version), courtesy of Webrecorder.net, were used for the recording.

The recording took place between 6 and 20 March and resulted in 3.2 Terabytes of information. The contents have been included in the EAWP45 special collection and will be available after one year.

To find out more, consult the open dataset:

Madeira Legislative Assembly elections 2024

The elections for the Legislative Assembly of Madeira took place on 26 May. Arquivo.pt carried out a special collection of content published on the web.

We began by automatically searching for news, election pages and websites related to the elections in Madeira. We used a list of search terms to put into the Bing Search API.

The aim was to obtain as many URLs as possible related to the event or topic in question, i.e. the Madeiran elections. To do this, several limits were set for the results: top 10, top20, top50 and top100. This process was documented, which shows that the more we expand the number of results, the greater the number of pages that are not very relevant and sometimes outside the intended target.

All the addresses (12,656) were recorded on 7 June in the Heritrix crawler.

Find out more by consulting the open dataset:

European elections 2024 in multilingual collection

The European elections took place on 9 June in Portugal. In some countries, such as Estonia, Czechia and Italy, the elections were held on a different date.

Arquivo.pt collected pages relating to the European Elections in the 27 countries of the European Union and in the 24 official languages.

The same methodology was used for the 2019 European Elections collection, i.e. a multilingual and semi-automatic search.

A list of 40 compound terms or keywords was used and translated into the 24 official EU languages. The terms were translated into the various languages in 2019 by the EU Publications Office. This resulted in a multilingual list of 960 terms to put into the Bing Search API.

Before the elections, on 3 June, the first search was carried out, resulting in 8,986 unique addresses, limiting the number of results to the top 20.

After the elections, new search terms were added with the names of the main candidates for the European Parliament in each country of the European Union. This second post-election search yielded 15,371 unique addresses.

The tool used for this collection was Heritrix. The collection was limited to three ‘hops’. In this case, the crawler follows links up to three times. This means that we opted for a certain restraint in the depth of the recording. Three ‘hops’ in the Heritrix crawler is enough to record one page (in other applications also called ‘page’ or ‘single page’ recording).

The content was recorded between 7 and 20 June and included in the EAWP46 special collection. It will be available after 1 year.

Find out more by consulting the open dataset:

Know more about past collections about elections

Portuguese at the 2024 Olympics and Paralympics in IIPC’s international collection of websites

print-replay-comiteolimpicoportugal-

Last updated on September 11th, 2024 at 04:23 pm

print-noticia-rtp-rececao-atletas-paralimpicos-paris-2021
Paralympic Games. Miguel Monteiro, gold medallist, returns to Lisbon (News on the RTP website, 2 September, selected for international collection)

Arquivo.pt has contributed to the international collection of web pages on the Summer Olympics Games taking place in Paris from 26 July to 11 August 2024 and is doing the same for the Summer Paralympics taking place from 28 August to 8 September.

The initiative to create the “2024 Summer Olympics/Paralympics IIPC CDG” collection is the responsibility of the International Internet Preservation Consortium (IIPC), the world’s leading organisation in the field of Internet preservation, through its Content Development Working Group.

The IIPC’s collaborative collections aim to promote the creation of thematic collections and collections based on international events. The web pages are recorded and then made available on the Archive-it service.

The pages of this collection will also be available on Arquivo.pt for those who want to carry out studies on sport and Olympism.

How the pages about Portuguese athletes were selected

At the Olympic Games 73 athletes represented Portugal in 15 sports, and at the Paralympic Games 27 athletes in 10 sports.

The criterion for selecting pages for the international collection was news about the athletes. For each athlete, pages were selected about their expectations before the games, their performance in the competition and their comments during and after the competition.

Some athletes have more news selected than others, and the same goes for the sites from which the news comes. The selection of pages was not limited to the first results presented by the search engine. We looked for a variety of channels and news from regional and local sites, some from the region or city where the athletes came from.

More than 500 pages to remember the Portuguese presence in Paris

The contribution of Arquivo.pt, as you can see in the table, already has more than 500 web pages.

print-tabela-seeds-ilustrativa-jogosolimpicos
Portuguese Seeds – 2024 Summer Olympics and Paralímpics, International Internet Preservation Consortium – Content Development Working Group (IIPC CDG)

Collaborate in the collection via the IIPC form

Helena Byrne, curator of web archives at the British Library and main curator of this collection, invites everyone to send in interesting pages to record: And we’re off – Get Involved in Web Archiving the Summer Games – Paris 2024.

The following public form is available to contribute:

2024 Summer Olympics & Paralympics

IPL – Politécnico de Lisboa organised a series of webinars with Arquivo.pt

thumbnail-ciclo-de-webinars-ipl

IPL – Politécnico de Lisboa, through its Distance Learning Group (EaD@IPL), organised a series of webinars for its community dedicated to Arquivo.pt and the preservation of content published on the Internet.

This initiative was attended by IPL – Politécnico de Lisboa lecturers and researchers, as well as people linked to the institution’s communications department.

The cycle of webinars took place in three sessions, between May and July 2024, and followed the training programme that Arquivo.pt has been offering for several years.

Presentation materials

Why training on web preservation is important

Archiving content published on the web and using a web archive on a day-to-day basis is an unusual practice, largely due to the community’s lack of knowledge about the existence and operation of Arquivo.pt.

For example, in this cycle of webinars with the IPL – Politécnico de Lisboa, participants were given tools that allow them to use the web archive immediately and creatively, such as the SavePageNow service, the historical content search service and, for use in interdisciplinary teams, Application Programming Interfaces (APIs).

As a result of this series of webinars, the collaboration between the IPL – Politécnico de Lisboa and Arquivo.pt was strengthened, with a view to preserving its institutional websites and other interesting content that is available on various online media (news, events, references to teachers, researchers and students).

Meet the winners of the Arquivo.pt Award 2024!

thumbnail-video-venced

Last updated on September 26th, 2024 at 06:13 pm

The winners of the Arquivo.pt 2024 Award were announced by the Público newspaper, the official media partner for this edition.

27 applications were received.

The awards ceremony took place during the closing session of the Ciência 2024 meeting, on 5 July, at Centro de Congressos da Alfândega do Porto.

1st place – “Noticioso – Desafiar percepções”

The winner of the 10,000 euro prize was the work “Noticioso – Challenging perceptions” developed by Carmen Fonseca and António Ramiro (Cubbo Team).

“Noticioso” is a platform where users can compare media coverage of various topics through a game (Quiz). It also allows users to explore trends over time using an analytical tool. How well do you know Portuguese news? Come and find out.

For example, which topic made the most news between 2000 and 2020: global warming or Sporting Club of Portugal? Arquivo.pt data says it was Sporting football club.

2nd place – “Habitação.PT: Uma visão do Mercado de Habitação em Portugal”

The 2nd prize of 3,000 euros was awarded to the work “Habitação.PT: Uma visão do Mercado de Habitação em Portugal” (“Habitação: An overview of the housing market in Portugal”) by Diogo Gonçalves.

“Habitação” is a tool that allows the user to interactively explore the evolution of the average value of the Portuguese housing and rental market, contextualised with news published on the subject and housing policies.

For example, in Lisbon in 2009 the price was around 1600 €/m2, rising to 4800 €/m2 in 2023. The rise in housing prices is contextualised by news stories over time.

3rd place – “Pegada Lusa”

The 3rd place prize of 2,000 euros was awarded to the work “Pegada Lusa” (Portuguese green footprint), developed by Sérgio Teixeira and Diana Teixeira.

“Pegada Lusa” is a work that shows the evolution of sustainable policies and initiatives in the various regions of the country, based on an analysis of projects and good practice from the United Nations Sustainable Development Goals (SDGs).

For example, the Porto region has a sustainability index (“Green Score”) of 57%, based on the content of the news analysed.

Honorable Mention granted by Público newspaper: “Uma viagem no tempo com o Público e o Expresso”

The newspaper Público, official partner of the 7th edition of the Arquivo.pt prize, awarded its Honourable Mention to the work “Uma viagem no tempo com o Público e o Expresso” (“A time travel with Público and Expresso newspappers”), by Rita Marques Costa and Beatriz Malveiro.

“A journey through time with Público and Expresso” analyses and compares the web pages of Público and Expresso since 1998, showing the website user how the digital versions of these media have evolved.

For example, in 2014, both Público and Expresso began to emphasise the headlines on their homepages and Expresso began to have a daily digital edition.

Honorable Mention granted by Aveiro Media Competence Center (AMCC): “discordAR: a Proximidade dos Partidos na Assembleia da República”

The Aveiro Media Competence Centre (AMCC) has awarded its Honourable Mention to the work “discordAR: a Proximidade dos Partidos na Assembleia da República”, by Miguel Salema and Sebastião Fonte.

“discordAR: The Proximity of the Parties in the Assembly of the Republic” is an app that shows the proximity between political parties, using votes in the Portuguese Parliament.

For example, we can see the percentage of votes in the same direction between the Parties in the period relating to the XII Legislature (2012 to 2015).

Honorable Mention granted by .PT: “ArquivoNC – o arquivo web do Jornal de Notícias da Covilhã”

The DNS.PT Association awarded an Honourable Mention to the Professor who encouraged the submission of the work “ArquivoNC – o arquivo web do Jornal de Notícias da Covilhã” (“ArquivoNC – the web archive of the Jornal de Notícias da Covilhã”), thus promoting the use of Arquivo.pt as a training and learning tool in the classroom. The work was created by student Rodrigo Dias da Silva, supervised by Professor Ricardo Campos, from the University of Beira Interior (UBI).

“ArquivoNC – the web archive of the Jornal de Notícias da Covilhã” is a work within the scope of the final project of the Engineering course at the University of Beira Interior (UBI) that provides access to ten years of web pages of the newspaper Notícias da Covilhã from the news preserved by Arquivo.pt between 2009 and 2019.

Awards Ceremony

Image gallery

premio-arquivo--10
premio-arquivo--8
premio-arquivo--7
premio-arquivo--6
premio-arquivo--1
premio-arquivo--3
premio-arquivo--4
premio-arquivo--2
premio-arquivo--9
premio-arquivo--10 premio-arquivo--8 premio-arquivo--7 premio-arquivo--6 premio-arquivo--1 premio-arquivo--3 premio-arquivo--4 premio-arquivo--2 premio-arquivo--9

Interviews

Dissemination materials

Press

Know more

Higher education library mobility program brings professionals to Arquivo.pt

FCCN_A Minha Biblioteca_24 maio 2024_2

Arquivo.pt operado pela FCCN FCT e localizado no Campus do LNEC

Arquivo.pt headquarters, operated by FCCN FCT, in Lisbon.

On May 24, the FCCN welcomed professionals from Higher Education Libraries (HEL) for the first time as part of the program promoted by the Higher Education Libraries Working Group (GT-BES) of the Portuguese Association of Librarians, Archivists, Documentalists and Information Professionals (BAD), My library is your library.

This is a mobility program that aims to carry out short-term visits with a view to exchanging experiences and hands-on contact with good practices, fostering collaboration and knowledge of Portuguese HEIs among professionals in the field.

Advanced services for knowledge

In this first edition of the program at FCCN, the participating colleagues (3 professionals from the University of Lisbon and 1 from the Catholic University of Porto) were offered a tour of the digital support services for higher education institutions operated by FCCN-FCT

Some services are familiar to information professionals, such as B-On and RCAAP. Others are back-office services and therefore less visible, but they are essential for higher education institutions. For example, Eduroam, which guarantees access to the Internet, RCTSaai for authentication or RCTS CERT for responding to security incidents.

Highlights include the Arquivo.pt and NAU services

The day highlighted Arquivo.pt and the NAU Platform, two services in the field of knowledge that are available to higher education institutions and also to society.

The Arquivo.pt team showed the backoffice of this Internet preservation service in Portugal and carried out a practical exercise in recording and integrating content into the web archive.

The NAU Platform is a platform for MOOCs (Massive Open Online Courses) created with the aim of democratizing knowledge, promoting digital literacy, enabling education and training for broad communities of users, particularly the Portuguese and Lusophone population.

More recently, with its integration into the North American platform edx.org, it has also been made available to all potential Portuguese-speaking trainees around the world. Participants in the program were shown how to build a MOOC course on the edx platform.

The program also included a visit to the Data Center and the professional television studio at the FCCN.

Visit by participants in the Higher Education Libraries mobility program to the FCCN Tv Studio
Visit by participants in the Higher Education Libraries mobility program to the FCCN Tv Studio

To know more

Week job shadowing at the Arquivo.pt from Prague to Lisbon

FCCN TV studio

By: Marie Haškovcová and Luboš Svoboda, Webarchiv, National Library of the Czech Republic, May 13th to 17th, 2024.

A visit within the EU Erasmus+ programme

Thanks to the EU Erasmus+ programme, focused on adult education – staff mobility, we were able to spend a week job shadowing at the Portuguese web archive Arquivo.pt and compare the strategies of the Czech web archive – Webarchiv with the approaches of our Portuguese colleagues.

In both cases, these are archives focused on national (Czech and Portuguese) content on the Internet.

The Arquivo.pt

While the Czech web archive is part of the National Library of the Czech Republic, the Portuguese archive (Arquivo.pt) is part of the FCCN, under the FCT – Foundation for Science and Technology, which aims to contribute to the development of science, technology and knowledge.

FCT provides IT services to the Portuguese higher education and research system, as well as high-speed internet connectivity. The institutional background of both archives is also reflected in the specifics of their concepts.

The visit included a presentation of the team and the campus and departmental spaces, a presentation of the activities of both archives and a discussion of the different aspects of our work – technical and curatorial tools, technologies and processes, the legislative environment and ethical issues, data storage, some services, research activities, perspectives and future plans.

The Czech web archive

The Czech web archive was founded in 2000, the oldest archival copies date back to 2001 and currently has more than 580 TB of data. Like Arquivo.pt, it harvests content on a national domain based on a list of url addresses obtained from its provider. It supplements these so-called comperhensive harvests with thematic and selective harvests in its acquisition strategy.

Topic collections relate to a specific topic or event, can be one-off or continuously built, and combine manually selected and automated scraped resources. Selective ones are intended for long-term harvesting, have detailed cataloging records that are part of the Czech national bibliography and are licensed – archival copies are therefore freely available through the catalogue.

From the Webarchive’s research activities, we presented our project aimed at detecting so-called dead webs through the Extinct Websites application and creating a database to serve as a basis for monitoring broader changes in the Czech web, and the WACloud project aimed at extracting big data from the web archive.

Exchanging knowledge and experience

Among the Portuguese projects we were interested in, for example, CitationSaver, and we also discussed the Memorial project, the harvesting of the Portuguese Wikipedia, and the activities of the Portuguese archive related to education in web archiving (training courses).

The meeting was enriched by the discussion of specific topic collections.

  • The Czech net art collection documents digital art and its transformation in the online space, providing a unique art historical perspective.
  • Another important collection is the Social networks of Members of Parliament of the Czech Republic 2021-2025 collection, which preserves the online communications and interactions of Czech MPs, invaluable for the study of political marketing and public political life.
  • The GitHub collection archives important repositories from this popular developer platform, preserving key domestic software projects and their code for future generations.
  • Finally, the Crypto, NFT, Blockchain, Web3, Metaverse collection charts the rise and impact of technology in the digital asset space. These collections are key resources for research and analysis of digital culture, policy, and technology, and the discussion of these collections at web archivist meetings contributes to the further development of archival methods and technological innovation.

We focused on exchanging knowledge and experience in seeds acquisition, workflow optimization and sharing technical tips and tricks.

Sharing best practices

We discussed best practices for identifying and collecting key web resources, a critical step in ensuring a comprehensive and representative archive. We shared various strategies for automating and streamlining workflows, including the use of web scraping tools and advanced content filtering.

Technical discussions included solutions to common problems such as harvesting dynamic web pages and overcoming access restrictions. The meeting provided a valuable platform for sharing innovative methods and fostering collaboration among experts, furthering the development of effective and sustainable digital archiving.

Erasmus+ visti to FCCN TV studio
Luboš Svoboda, web curator, Marie Haškovcová, chief of the Webarchiv e Ricardo Basílio, Arquivo.pt web curator visiting the FCCN-FCT TV Studio.

Exhibition of old websites to mark International Museum Day

Heritales Crowd-Recycling e Arquivo.pt no Dia Internacional dos Museus

May 18, International Museum Day, was celebrated all over the country with free admission, guided tours, entertainment and exhibitions related to memory and heritage.

Arquivo.pt contributed with an exhibition of old web pages, entitled “Digital Memory through the Internet of the Past”, which was on display at one of the stands at the National Coach Museum in Lisbon.

The pages were selected to show different aspects of the Alentejo over time. From 2016, pages relating to the Heritales project were selected.

Heritales and Crowd-Recycling drew attention to the preservation of the Internet’s memory

Heritales is a project based in Évora that aims to study and disseminate heritage in all its manifestations. It is known for its main event created in 2016, HERITALES – International Heritage Film Festival.

Crowd-Recycling is a project focused on good practices for sustainability.

Heritales, Crowd-Recycling and Arquivo.pt carried out this action in collaboration with the aim of giving visibility to content published on the web over time. Preserving and giving access to digital content is fundamental to enhancing heritage.

Why an exhibition of old websites is a good idea

Making an exhibition of websites over time is relatively easy, all you have to do is come up with a theme, which can also be the history of an institution, and choose pages preserved on Arquivo.pt.

An exhibition of old websites is an original idea for the target audience. It often features texts and images that only existed on the web.

By drawing attention to the websites, we realize that many things were left unrecorded and this changes our view of the content we publish today. We start taking more care to save important pages, for example by taking action or saving them on the spot with SavePageNow.

Heritales Crowd-Recycling e Arquivo.pt no Dia Internacional dos Museus
Heritales, Crowd-Recycling and Arquivo.pt on International Museum Day at the National Coach Museum

World Internet Day was on May 17th

The day before International Museum Day was World Internet Day (May 17). The proximity of the two commemorations ties in with the theme of preserving memory.

Portugal connected to the Internet for the first time in 1991, with the FCCN project “RCCN IP Service”.

To remember how it all happened, here are the three suggestions that FCCN published on social media for this day:

Arquivo.pt is finalist for the DPC Awards 2024

dpc-award-thumb

Last updated on August 12th, 2024 at 11:50 am

The Digital Preservation Coalition Awards

The Digital Preservation Coalition (DPC) is dedicated to promoting digital preservation and associated best practices.

The DPC Awards promote exemplary and innovative digital preservation use cases from all over the world.

The Arquivo.pt team submitted two applications to the DPC Awards 2024 in the categories of “Safeguarding the Digital Legacy” and “Research and Innovation”.

The Award for Safeguarding the Digital Legacy celebrates the practical application of preservation tools to protect at-risk digital objects.

The Award for Research and Innovation recognizes excellence in practical research and innovation activities.

Arquivo.pt applications to the DPC Awards

#1 Arquivo.pt catalog of tools for digital preservation

Information that rules modern-day lives is born-digital and disseminated online. However, invaluable digital objects published online have been continuously lost.

Arquivo.pt is a public infrastructure which supports the preservation of digital objects published online to safeguard this digital legacy for future generations.

Thus, in October 2023 after 15 years of research and development, Arquivo.pt released a Catalog of 13 innovative tools to support the preservation of at-risk online content, from acquisition to dissemination (e.g. search and access, APIs, training, open data sets, exhibitions).

Arquivo.pt safeguards online digital objects of worldwide interest for research and education.

The Arquivo.pt Catalog was selected as finalist to the Safeguarding the Digital Legacy Award.

#2 Searching preserved web-images

Images published online are precious digital assets that document contemporary times for future generations.

This initiative describes the research and development of an innovative image search system that enables the discovery and access to billions of preserved images acquired from the web since the 1990s.

This research was applied to enhance the Arquivo.pt web archive with an image search service publicly available to any Internet user, officially launched in August 2022.

The resulting scientific and technical publications are available in open-access and the developed software is available as free open-source to be reused and enhanced by the community.

This work on searching images preserved in web archives applied for the Research and Innovation Award.

Know more

Analysis of the Arquivo.pt query dataset

demo-wordcloud-arquivopt3

Last updated on October 1st, 2024 at 10:34 am

Arquivo.pt query logs are unique resources for research

Arquivo.pt provides a “Google-like” service that enables searching pages and images collected from the web since the 1990s. Notice that Arquivo.pt search complements live-web search engines because it enables temporal search over information that is no longer available online on its original websites.

Analyzing user behavior is an important research topic to understand users’ information needs and enhance the quality of search results. Thus, when a user interacts with a search engine, the system records the user’s actions in a file called the query log. Query logs from web archives are unique resources for research because they describe the real needs of web-archive users about the historical information published online over time.

Research case study

Flavie Gallois and Adam Jatowt from the University of Innsbruck, and Ricardo Campos from the University of Beira Interior and INESC TEC analyzed user search behavior based on the Arquivo.pt search query log dataset collected over a period of 3 months from June to September 2021 (Analyzing User Search Behaviour in Temporal Web Repositories through Search Query Log Analysis).

This study analyzed query features such as length, type or frequency and compared the obtained results with previous work about user search behavior over web-archives and live-web search engines.

This study revealed interesting trends and patterns about how users search for information within web archives, with strong potential for future research work.

How do web-archive users search?

Figure 1 : Distribution of country origin of users
Figure 1 : Distribution of country origin of users
Figure 2: Distribution of languages used in queries
Figure 2: Distribution of languages used in queries

The users came from Portugal in 85.7% of the queries. However, the Portuguese language was identified through automatic language identification of queries as being used in only 37% of the queries. This suggests that users apply other languages than their own to search in web archives.

Users of Arquivo.pt tend to use longer queries with more words and characters in comparison to previous studies, both over web archives and live-web search engines. About 92% of the queries had 5 or fewer terms (average of 25 characters), with 3 being the most common number of submitted terms. In previous work about search behavior in web archives, it was observed that users tended to submit from 1 to 3 terms per query, with 1 term as the most common submission.

Users tend to issue multiple queries within a session instead of a single query, possibly indicating a need for refining their search queries or exploring multiple options for inquiry.

87,7% of the queries submitted to Arquivo.pt used Desktop Browsers, despite Arquivo.pt providing mobile-friendly user interfaces. Old web-archived pages are not responsive and render poorly on mobile devices. Thus, it is expectable that users mostly use web archives through their desktops.

Figure 3: Arquivo.pt users can refine the time span of their queries by using the From and To datepickers.
Figure 3: Arquivo.pt users can refine the time span of their queries by using the From and To datepickers.

Users refined the time span of the search (using the datepickers) in about 50% of queries which indicates awareness of temporal needs peculiar to web-archive usage. Interestingly, users modified the From datepicker more frequently than the To datepicker. Notice that keeping the default time span may fit the user information needs and does not necessarily indicate the lack of awareness about the existence of the function to define time span (peculiar to web-archive search).

Only a small percentage of users included specific years in their query terms (4%), potentially suggesting that in these cases the time span function was insufficient, or unnoticed by some users.

The obtained results suggest that users are more conscious of their information needs and have improved their search techniques to be more effective over web-archives instead of just using them out of curiosity as first-comers.

What is searched in a web-archive?

The authors of the study applied automatic named entity recognition over the user queries and derived a set of word clouds that graphically provide a glimpse of the most common information needs of Arquivo.pt users:

Figure 4: Word cloud of the most frequent query terms submitted to Arquivo.pt.
Figure 4: Word cloud of the most frequent query terms submitted to Arquivo.pt.
Figure 6: The most frequent Geographical Locations in query terms submitted to Arquivo.pt.
Figure 6: The most frequent Geographical Locations in query terms submitted to Arquivo.pt.
Figure 6: The most frequent Organizations in query terms submitted to Arquivo.pt
Figure 6: The most frequent Organizations in query terms submitted to Arquivo.pt
Figure 7: The most frequent Persons in query terms submitted to Arquivo.pt.
Figure 7: The most frequent Persons in query terms submitted to Arquivo.pt.

Access to research Arquivo.pt query dataset

Arquivo.pt released a set of resources to support research studies over its

Query log dataset

Original log files (samples)

Documentation

Evaluation Metrics for web-archive search

The first step to understand user behavior is to define evaluation metrics. Defining metrics is a powerful tool to set long and short-term goals to decide which new products and features should be released to the users.

We share a work-in-progress report which aggregates information about Web Archive Search Evaluation Metrics. This contributes to comparing users’ search behavior between live-web and web-archive search engines. Feel free to comment directly on the collaborative document or to contact us.

This report also provides a summary of references about previous work, query workflows and structure of the corresponding query logs produced by Arquivo.pt, to facilitate the work from the researchers to study these data sets.

Know more