Speed versus specificity: Designing text annotation tasks for the people and algorithms that drive human-in-the-loop (HIL) products

Description

Clara is a scheduling service that coordinates when, where, and how you meet with prospects, candidates, and collaborators. Clara delivers unprecedented efficiency and accuracy by combining the precision and consistency of intelligent software with the judgement of an expert team. This human-in-the-loop approach ensures scheduling communication is always clear, swift, and professional.

Clara’s platform leverages human skill by breaking up work into small tasks of two types: stateless annotation tasks and stateful automation-review workflow tasks. Annotations are generated by “mapping” email message text to distinct scheduling-parameter and machine learning (ML) labeling tasks. The task results are aggregated and reduced with state to automate the final scheduling outputs. These outputs are then sometimes reviewed for quality assurance in workflow tasks.

In contrast to conventional offline annotation systems, Clara’s annotation tasks may be used to drive automation in real time and are therefore time constrained. This means tasks must yield machine learning training data and be completed quickly and accurately. The tasks also need to be easily understandable by nonexperts. These properties are sometimes in conflict and may not all be equally achievable via a single interface or label definition.

Jason Laska explores the trade-offs between text annotations defined for fast data entry and those defined with high specificity and granularity meant for training ML models, using the application of DateTime text—modeled as recurrence-rule and coreference annotations—as it pertains to meeting-attendee availability to guide the discussion.

This is joint work with Joey Carmello, a software engineer at Clara Labs.