Indexer Efficiency

HIVE Overview | Demo | HIVE Community | Publications | Archives

Indexer Efficiency (proposal)

Evaluation of the impact of HIVE on indexer productivity/efficiency.

Indexer efficiency:

• How long does it take an indexer (or user) to manually assign terms from multiple vocabularies?
• Does HIVE usability compare with native vocabulary system for novice, intermediate, and expert indexers?
• Does HIVE (without automatic term suggestion) improve term assignment efficiency?
• Does HIVE with automatic term suggestion improve term assignment efficiency?

Phases

Phase	Title	Summary	Notes	Required?
A	HIVE Usability	The HIVE interface should be at least as usable as native vocabulary interface for intermediate indexers.	Index n documents using native interfaces and HIVE. Resulting documents and assigned terms used in (D).	This phase is only necessary if attempting to measure indexer efficiency using HIVE versus native vocabularies.’
B	Indexing documents (no HIVE)	Measure performance of indexers assigning terms from multiple vocabularies to a document collection using the native vocabulary interface.	Index n documents using native interface. Resulting documents and assigned terms used in (D). Capture timing, terms searched, terms selected, terms not selected.	This phase is only required if we are attempting to measure indexer efficiency using HIVE versus native vocabularies. It may be worth considering measuring indexer efficiency when creating the gold set, regardless of whether measuring native v. HIVE.
C	Indexing documents (HIVE)	Measure performance of indexers assigning terms from multiple vocabularies to a document collection using HIVE.	Index n documents using HIVE. Resulting documents and assigned terms used in (D). Capture timing, terms searched, terms selected, terms not selected.	This phase is only necessary if attempting to measure indexer efficiency using HIVE versus native vocabularies.
D	Index documents (HIVE + automatic)	Measure performance of indexers assigning terms from multiple vocabularies to a document collection using HIVE with automatic term recommendation using algorithm(s) selected in D.	Index n documents using HIVE with automatic term recommendation. Capture timing, terms searched, terms selected, terms not selected.	This phase is only necessary if attempting to measure indexer efficiency with term suggestion.

A. HIVE Usability

Question: Does HIVE usability compare with native vocabulary system for novice, intermediate, and expert indexers?

Context: If HIVE is intended to replace native vocabulary systems (e.g. LC Class Web, TGN, NBII, shouldn’t the usability of HIVE be evaluated against these systems?

Steps

• Train indexers to index using selected vocabularies directly
• Train indexers to index using HIVE (no keyphrase recommendation)
• Train indexers to index using HIVE + keyphrase recommentation
• Indexer evaluates HIVE usability against native vocabulary system.
  • Ideally, indexer indexes real documents during evaluation.

B. Indexing Documents (no HIVE)

Question: How long does it take to index documents without HIVE?

Context: Indexer assigns terms to documents using multiple vocabularies. Vocabularies services are used directly (no HIVE). No keyphrases are automatically recommended. (A proxy server could be used to capture all HTTP traffic. A parser would need to be written to extract usage information).

Steps:

• Train indexer to index using selected vocabularies directly
• Indexer enters information about what is being indexed
  • Citation, URL, or upload full-text
• Indexer starts indexing document (unique ID generated)
  • Timing starts
• For each vocabulary, index document
  • Indexer searches for terms
    • What terms were displayed?
    • What terms were searched?
  • Indexer selects terms
    • What terms where selected?
    • What terms were displayed by not selected?
• Indexer stops indexing document
  • Timing stops

C. Indexing documents (HIVE)

Question: How long does it take to index documents with HIVE (without term suggestion)?

Context: Indexer assigns terms to documents using multiple vocabularies. Vocabularies are through HIVE only. No keyphrases are automatically recommended.

Steps:

• Train indexer to index using selected vocabularies through HIVE
• Indexer enters information about what is being indexed
  • Citation, URL, or upload full-text
• Indexer starts indexing document (unique ID generated)
  • Timing starts
• For each vocabulary, index document
  • Indexer searches for terms
    • What terms were displayed?
    • What terms were searched?
  • Indexer selects terms
    • What terms where selected?
    • What terms were displayed by not selected?
• Indexer stops indexing document
  • Timing stops

D. Index documents (HIVE + automatic)

Question: How long does it take to index documents with HIVE with automatic term recommendation?

Context: Indexer assigns terms to documents using multiple vocabularies. Vocabularies are through HIVE only. Terms are automatically recommended.

Steps:

• Train indexer to index using selected vocabularies through HIVE
• Train indexer to use term recommender
• Indexer enters information about what is being indexed
  • Citation, URL, or upload full-text
• Indexer starts indexing document (unique ID generated)
  • Timing starts
• For each vocabulary, index document
  • Indexer searches for terms
    • What terms were displayed?
    • What terms were searched?
  • Terms are automatically recommended
    • What terms were recommended?
    • What terms were selected?
    • What terms were not selected?
  • Indexer selects terms
    • What terms where selected?
    • What terms were displayed by not selected?
• Indexer stops indexing document
  • Timing stops