Fonduer: Knowledge Base Construction from Richly Formatted Data

Knowledge bases are incredible enablers of valuable downstream applications such as information retrieval, question answering, medical diagnosis, and data visualization. However, building high quality knowledge bases can be incredibly difficult. While extensive efforts have been focused on unstructured text, troves of information remains untapped in richly formatted data, where relations are conveyed using textual, structural, tabular, and visual cues.

We recently built Fonduer, a knowledge base construction framework for richly formatted information extraction. Fonduer is the first knowledge base construction system for richly formatted data, and uses a new unified data model, which preserves structural and semantic information across different data modalities, and a human-in-the-loop paradigm called data programming to train machine learning systems.

Why is it called Fonduer?

We think our system for extracting information from richly formatted data resembles some of the characteristics of rich and savory fondue. Specifically, there are some analogies with the challenges of richly formatted data that Fonduer seeks to address.

Prevalent Document-level Relations

document level relations
Fig. 1: With richly formatted data, we need to look at the whole picture.

The first challenge is the prevalence of document-level relations. In order to extract information from a PDF document, for example, we typically cannot look at the context of just a single sentence. If we limit the context to a single sentence or table, we can miss up to 97% of the relations in the document! Instead, we need to step back and consider the document as a whole to in order to appreciate and capture all of the rich information contained within.

Multimodality

multimodality
Fig. 2: We need to consider signals from multiple data modalities together, not in isolation.

The second challenge is multimodality. Just as fondue is made up of a variety of ingredients, each with their own flavor and textures that come together to make a meal, richly formatted documents rely on a variety of data modalities to convey information. For example, bold text, placement on a page, and visual alignment in a table column all convey meaning. Fonduer captures textual, structural, tabular, and visual information in a unified data model.

Data Variety

data variety
Fig. 3: There is a huge variety in the types of richly formatted data.

The third challenge is data variety. Fondue isn’t just bread and cheese; it could be meat and oil, or even chocolate and fruit! Similarly, there is a huge amount of variety in richly formatted documents. This can come from format variety (e.g., different file formats) and stylistic variety (e.g., linguistic variation or differences in table formatting). Fonduer adopts a data model that is generalizable and robust against heterogeneous input data.

Learn More

Read about it in the HazyResarch blog post, or view the full paper.

Posts from blogs I follow

A Baseball Startup with Paul Freedman and Bryan Carmel

Bryan, Adam, Steve, and the Oxide Friends are joined by the founders of the Oakland Ballers, the continuation of a long history of baseball in Oakland. There turns out to be a plenty in common between founding a computer company and founding a baseball tea…

via Oxide and Friends April 17, 2024

Platform Support

Supporting multiple platforms can be a repetitive and boring problem, or a fun challenge.

via Stay SaaSy April 17, 2024

Day 2: Cold email results

After sending a batch of cold emails to WooCommerce agencies who might be interested in Dashify, there’s been two visible results! One is that visitors to the website spiked to 9 people, myself excluded, whereas previously it might have received 1 or 2 in …

via John Jago April 16, 2024

Generated by openring-rs from my blogroll.