Understanding Metadata Latency with MDWorkbench

Julian Martin Kunkel, Georgios Markomanolis

    Research output: Chapter in Book/Report/Conference proceedingConference contribution

    1 Scopus citations

    Abstract

    While parallel file systems often satisfy the need of applications with bulk synchronous I/O, they lack capabilities of dealing with metadata intense workloads. Typically, in procurements, the focus lies on the aggregated metadata throughput using the MDTest benchmark (https://www.vi4io.org/tools/benchmarks/mdtest ). However, metadata performance is crucial for interactive use. Metadata benchmarks involve even more parameters compared to I/O benchmarks. There are several aspects that are currently uncovered and, therefore, not in the focus of vendors to investigate. Particularly, response latency and interactive workloads operating on a working set of data. The lack of capabilities from file systems can be observed when looking at the IO-500 list, where metadata performance between best and worst system does not differ significantly. In this paper, we introduce a new benchmark called MDWorkbench which generates a reproducible workload emulating many concurrent users or – in an alternative view – queuing systems. This benchmark provides a detailed latency profile, overcomes caching issues, and provides a method to assess the quality of the observed throughput. We evaluate the benchmark on state-of-the-art parallel file systems with GPFS (IBM Spectrum Scale), Lustre, Cray’s Datawarp, and DDN IME, and conclude that we can reveal characteristics that could not be identified before.
    Original languageEnglish (US)
    Title of host publicationLecture Notes in Computer Science
    PublisherSpringer International Publishing
    Pages75-88
    Number of pages14
    ISBN (Print)9783030024642
    DOIs
    StatePublished - Jan 25 2019

    Fingerprint Dive into the research topics of 'Understanding Metadata Latency with MDWorkbench'. Together they form a unique fingerprint.

    Cite this