Photo of Taesoo Kim
Taesoo Kim

Papeos

Research consumption has been traditionally limited to the reading of academic papers-a static, dense, and formally written format. Alternatively, pre-recorded conference presentation videos, which are more dynamic, concise, and colloquial, have recently become more widely available but potentially under-utilized. In this work, we explore the design space and benefits for combining academic papers and talk videos to leverage their complementary nature to provide a rich and fluid research consumption experience. Based on formative and co-design studies, we present Papeos, a novel reading and authoring interface that allow authors to augment their papers by segmenting and localizing talk videos alongside relevant paper passages with automatically generated suggestions. With Papeos, readers can visually skim a paper through clip thumbnails, and fluidly switch between consuming dense text in the paper or visual summaries in the video. In a comparative lab study (n=16), Papeos reduced mental load, scaffolded navigation, and facilitated more comprehensive reading of papers.

Conference Presentation

Interface Walkthrough

This tutorial explains how to use the Papeo Reader interface and its various features! This interface lets you read Papeos: Schoarly Papers with clips from their talk videos on the margins.screenshot of papeoVideo clips are attached to parts of the paper with corresponding content. These parts of the paper are indicated by a colored sidebars.

(1) Sidebars

To watch the video clips, you can click on the sidebar. This will focus the video clip (i.e., enlarge it and show the full transcript) and will start playing the video clip.screenshot of papeoClicking on the sidebar will also pull the video clip closer to the sidebar. You can click on the sidebar multiple times to bring it even closer.screenshot of papeoInstead of watching a clip or to get a quick glimpse of what is shown in the clip, you can also hover over the sidebar and move up/down to quickly scrub through the clip.screenshot of papeo

(2) Video Clips

To watch clips, you can also click on the clips themselves. To increase the speed of the video, you can hover over the video and then click on the controls on the top right to set the playback speed (e.g., 1x, 1.25x, 1.5x, 1.75x, or 2.00x). This setting will apply to all video clips.screenshot of papeoAfter watching a clip, you will be asked if you want to replay the clip or go to next video clip.

  • Clicking on replay will replay the clip.
  • Clicking on next will change to the next clip in the order of the video. The video clip will stay in place and the paper will scroll to the location of that clip

screenshot of papeoIf you want the interface to automatically go to the next clip, you can also switch the autoplay on. For this, hover over a clip and click on the toggle on the top-left. With autoplay, you can watch the whole video while seeing where the information is located in the paper.screenshot of papeoOn top of the video clip, you can also see the video's timeline.

  • Each block represents a clip. Their width represents the clip's duration.
  • Lighter blocks are clips you have watched. Colored block is the current clip.
  • Hovering over a block will show the title of the paper section where that clip is found.

screenshot of papeoYou can also click on the blocks in the timeline to navigate to those clips.screenshot of papeoPaper author(s) can decide to not map certain clips to their paper. Unmapped clips will not be shown in the margin, but can be accessed with the timeline and are presented with a gray border.

(3) Highlights

At the bottom of the clips you can see the video transcript with lines colored in as the video plays. For some of the clips, you will also see highlights in the transcript shown as bolded words. These highlighted words or phrases are connected to similar or relevant words or phrases in the paper. Hover over the highlights in the transcript (or the paper) and the corresponding words or phrases in the paper (or transcript) will be highlighted in sync.screenshot of papeoThis can help you more easily find keywords or information mentioned in the video in the paper.

Publication

Papeos: Augmenting Research Papers with Talk Videos
Tae Soo Kim, Matt Latzke, Jonathan Bragg, Amy X. Zhang, Joseph Chee Chang. The ACM Symposium on User Interface Software and Technology. 2023.