HikarIA at the UNESCO symposium on the role of artificial intelligence in museums

Video transcription

We will now give the floor to Édouard de Saint-Ours.

Édouard, you are a curator of photography. You are a specialist in nineteenth-century Asian photography, and you have been working at the Guimet Museum as a curator since October 2023. You direct projects of conservation, acquisition, research, and exhibition related to the museum’s extensive photographic collections, which comprise around 600,000 items.

These include the HikarIA project, led in partnership with the French company Teklia, which aims to develop new digital and artificial intelligence tools to improve access to the museum’s collections.

Could you present this project to us? Thank you.

Édouard de Saint-Ours

Thank you very much, Benjamin, for that introduction, and thank you to all of you who are here with us today. I would also like to thank the organizers for the wonderful welcome extended to the Guimet Museum here in Uzbekistan.

In partnership with a French company specializing in computer vision technologies, I have been working on a project called HikarIA.

To explain the origin of the name: HikarIA is a portmanteau combining hikari, the Japanese word for “light,” and IA, the French acronym for artificial intelligence. The project aims to improve the management of photographic collections, ensure their accessibility, and facilitate deeper historical study—particularly given our exceptional collection of early Japanese photographs.

We have approximately 20,000 photographs, dating from the nineteenth century to the 1920s. The goal of the HikarIA project is to conserve, study, and disseminate knowledge about the Guimet Museum’s photographic collections. We aim to develop new digital tools that can support research into the history of photography.

The project makes use of artificial intelligence and computer vision. Another equally important objective is to research the beginnings of photography in Japan during the Bakumatsu, Meiji, and Taishō periods, which span roughly from the mid-nineteenth century to the early twentieth century. The HikarIA project has been partially funded by the French government through the France 2030 investment plan.

The project originated from a collection acquired by the museum between 2007 and 2009, originally assembled by Joseph Dubois, who was fascinated by Japan and spent more than thirty years collecting Japanese photographs from this period. Dubois was able to access truly exceptional material, and I have shown you a few examples so that you can better visualize what I am referring to.

We have, for instance, photographs by Apollinaire Le Bas, who took part in the Shimonoseki campaign in the 1860s and documented the surrounding environment at that time. We also have photographs by Frederick Sutton, who was on a diplomatic mission and had the rare opportunity to produce a portrait of the last shogun in 1867.

The collection also includes early photographic studios in Japan, notably those founded in Yokohama by Felice Beato, a former war photographer who established his own studio there. We also have works by Raimund von Stillfried-Ratenicz, often described as Beato’s Austro-Hungarian alter ego.

In addition, the collection features major Japanese photographers such as Ichida Sōta, who was based in Kobe. You can see here an extraordinary panorama of Kobe harbor. Another example is Shin’ichi Suzuki, who is well known for his luxurious albums, hand-colored by artists.

The collection is extremely diverse. It includes more than 200 photographic albums, of which only 40 have been restored so far. Restoring 40 albums required a tremendous amount of work, but we still have much to do, as the total collection includes around 200 albums. The restoration work was financed by the Caisse des Dépôts in France.

We worked with a specialist in the restoration of albums and Japanese photography. The restoration phase concluded in the summer, and we subsequently continued the project with a full digitization campaign. Another foundation has supported this digitization effort, which is essential, because without digitized images, it would be impossible to deploy artificial intelligence tools.

From the very beginning of the project, we aimed to include a technological development component. We did not want to create or train a new AI model. Instead, we sought to identify existing open-source models that could be useful for our work. This was a pragmatic approach, as we do not have the human or financial resources to start from scratch.

AI development is currently advancing at an extraordinary pace, and we were able to test a wide range of recently created models. The strong open-source culture in the field of AI gave us access to tools that we could fully take advantage of. We identified the models that were most relevant to our objectives, which I will describe in more detail shortly.

The third phase of the project, which is still ongoing, is the research phase. We have been developing tools to make the collection accessible. We have created an online platform and database that are freely accessible to everyone, allowing users to consult the entire corpus and all available data.

This is also a research project. We have been working with Saki Toriumi, a specialist in Japanese photography from a university in Japan, who has begun collaborating with us. Her work helps us empirically identify which datasets are most useful, as well as the advantages and limitations of these technologies. Above all, we must remain pragmatic.

I do not have much time, so that is a brief overview of the HikarIA project. I would now like to address the technical challenges we face. There are also methodological challenges, which we can discuss later if you wish.

Regarding technical challenges, there are several ways in which AI can be applied to historical photographs. We identified seven main challenges. Some of these were addressed in other phases of the project, particularly the restoration phase and the accessibility phase.

One basic but very useful tool automatically corrects images that have been digitized in the wrong orientation.

Another key aspect concerns image content and iconography. We use a tool that recognizes people and objects within a photograph and segments them so they can be visually identified. We also use geolocation. A large portion of the corpus includes location information, which has been identified by manuscript specialists based on handwritten captions.

This allows us to visualize where images originate across the Japanese archipelago. As you can see, the Kantō and Kansai regions contain the highest concentration of photographs.

Moving on to cataloguing: this may seem basic, but we have implemented a set of tools that allow us to aggregate and restitute all available information. We experimented with Getty terminology for indexing, although we are not yet certain how we will ultimately use it. This process has nevertheless allowed us to test the relevance of the models.

We also have an image similarity search tool that provides a probability score of similarity between images. In some cases, these are identical images that differ only in coloration. For each photograph, the system generates a list of visually similar images, which is particularly valuable for research into iconography and stylistic evolution.

We have also created a thematic index. We deliberately chose not to use a traditional thesaurus, as it is nearly impossible to find an exhaustive dataset applicable to such a diverse corpus. Instead, we used a hierarchical structure developed by Corinne Jorgensen in the 2000s, which provides a highly resilient system for term classification.

You will get a clearer sense of this by exploring the platform itself.

With regard to historical research, this is just one example of how quantitative data can be visualized. We cross-referenced the number of people depicted in images with different image typologies, such as religious architecture, civil architecture, natural landscapes, and urban landscapes.

These categories were defined by humans, but the data concerning the number of people per image was generated by an AI model.

Finally, accessibility is a core concern. We aim to ensure that all data is freely accessible and reusable. We use the IIIF protocol to facilitate interoperability with other institutions and research projects.

We also experimented with ChatGPT to generate automatic image descriptions. We have not yet decided exactly how this tool will be used, but it could be particularly useful for visually impaired audiences, allowing them to access photographic content through textual description.

Linguistic accessibility is also essential. The platform is available in French, English, and Japanese. To translate both the website and the metadata tags, we used the SIGLIP model, which performs semantic search as well as translation. All translations were reviewed and corrected with our partners.

The platform is currently in beta. If you would like to test it and provide feedback, you are very welcome to do so.

Thank you very much.

Closing

Thank you very much, Édouard. When people say that AI is magical and that everything can be solved by clicking a button, this presentation offers a concrete example of the complexity and sophistication of the tools involved, as well as the expertise required to manage such projects.

Thank you very much.