Fonduer: Knowledge Base Construction from Richly Formatted Data

Knowledge bases are incredible enablers of valuable downstream applications such as information retrieval, question answering, medical diagnosis, and data visualization. However, building high quality knowledge bases can be incredibly difficult. While extensive efforts have been focused on unstructured text, troves of information remains untapped in richly formatted data, where relations are conveyed using textual, structural, tabular, and visual cues.

We recently built Fonduer, a knowledge base construction framework for richly formatted information extraction. Fonduer is the first knowledge base construction system for richly formatted data, and uses a new unified data model, which preserves structural and semantic information across different data modalities, and a human-in-the-loop paradigm called data programming to train machine learning systems.

Why is it called Fonduer?

We think our system for extracting information from richly formatted data resembles some of the characteristics of rich and savory fondue. Specifically, there are some analogies with the challenges of richly formatted data that Fonduer seeks to address.

Prevalent Document-level Relations

document level relations

Fig. 1: With richly formatted data, we need to look at the whole picture.

The first challenge is the prevalence of document-level relations. In order to extract information from a PDF document, for example, we typically cannot look at the context of just a single sentence. If we limit the context to a single sentence or table, we can miss up to 97% of the relations in the document! Instead, we need to step back and consider the document as a whole to in order to appreciate and capture all of the rich information contained within.

Multimodality

multimodality

Fig. 2: We need to consider signals from multiple data modalities together, not in isolation.

The second challenge is multimodality. Just as fondue is made up of a variety of ingredients, each with their own flavor and textures that come together to make a meal, richly formatted documents rely on a variety of data modalities to convey information. For example, bold text, placement on a page, and visual alignment in a table column all convey meaning. Fonduer captures textual, structural, tabular, and visual information in a unified data model.

Data Variety

data variety

Fig. 3: There is a huge variety in the types of richly formatted data.

The third challenge is data variety. Fondue isn’t just bread and cheese; it could be meat and oil, or even chocolate and fruit! Similarly, there is a huge amount of variety in richly formatted documents. This can come from format variety (e.g., different file formats) and stylistic variety (e.g., linguistic variation or differences in table formatting). Fonduer adopts a data model that is generalizable and robust against heterogeneous input data.

Learn More

Read about it in the HazyResarch blog post, or view the full paper.

Posts from blogs I follow

Resilient LL Parsing Tutorial

In this tutorial, I will explain a particular approach to parsing, which gracefully handles syntax errors and is thus suitable for language servers, which, by their nature, have to handle incomplete and invalid code. Explaining the problem and the solution…

via matklad May 21, 2023

🤠 "Where have all the hackers gone?" + a way to discuss programming languages 👨‍🎤

🎵 The song for this post is CHEMICAL LOVE, by "Kaleb James and Chey" for the game Bust-a-Groove. 🎵 I'm talking with friends and coworkers about programming languages (surprise), and I'm landing on a rough shape that these conversations take. I'll share it …

via More Pablo: all May 15, 2023

The Perfect Weekend-Project, or You Should Build a Blog

The latest from atthis.link

via atthis.link May 12, 2023

Generated by openring-rs