An Empirical Guide to the Behavior and Use of Scalable Persistent Memory

https://www.usenix.org/conference/fast20/presentation/yang

Presentation

• Optane DIMMs

  • Not just slow dense DRAM

  • Slower media

    • More complex architecture

      • Second-order performance anomalies

        • Fundamentally different

• Outline

  • Background

  • Basics: Optane DIMM performance

  • Lessons: Optane DIMM best practices

  • Conclusion

Background: Optane in the machine

• iMC: integrated memory controller

• use optane

  • Memory mode : use Optane to expand main memory capacity without persistence

    • Combine Optane DIMM with a conventional DRAM DIMM on the same memory channel that serves as a direct-mapped cache for the NVDIMM

  • App Direct mode : provides persistence and does not use a DRAM cache

Focus on the right

• Optane controller: merging 4 cache lines into 256B block and issuing that block to media

  • Buffer: merging

• AIT: address interaction table

  • Resides in optane media, but also cache on device dram

• ADR:

  • If power failure, then we have enough stand-up power to flush the write pending queue to media

    • The ADR domain does not include the processor caches, so stores are only persistent once they reach WPQs

  • WPQ: write pending queue

  • The iMC communicates wih the Optane DIMM using the DDR-T interface in cache-line (64-byte) granularity

    • Has to do with the physical features of how to data is laid out

ADR

• First 4K: optane DIMM 1

• Next 4K: optane DIMM 4

• Touch all six optane DIMM

Basics: our approach

Microbenchmark

• Sequentially: no access amplification

  • In contrast, random, 3x as small

• Write

  • Is acknowledged as soon as it hits the on socket integrated memory controller

    • Not measure all the way to media

      • Not possible to measure this?

    • Writing to the cache + flushing it out

      • Write to cache is the same for both cases

      • Flush

• DRAM

  • More cores, performance goes up

• NI: non-interleave

  • Read from single DIMM

  • Bandwidth low

  • Speculate: slow media, contentions, extra delay because of this contention

• Interleave

  • Write: saturate bandwidth and contention is growing worse

    • 3 cores and slow down

• Use all optane dimms, interleave reads (scale well), efficiently

• Write (max out) constant with respect to thread counts

• Using the block size efficiently

• Read: Peak bandwidth at 512B, but valley at 4K (weird contention)

  • File system use this 4KB size

Lessons: what are Optane best practices?

• Avoid small random accesses

• Use ntstores for large writes

• Limit threads accessing one NVDIMM

• Avoid mixed and multi-threaded NUMA accesses

Lesson 1: avoid small random accesses

• Small accesses: smaller than 256B, lose bandwidth

• Vary the working set size

  • How large this on-device buffer is? About 16K

Lesson 2: Use ntstores for large writes

• ntstore (non-temporal store): bypass the cache hierarchy and issue directly to the backing DIMM

• store + clwb: doing Aa store and then evicting that cache line or cleaning the cache line using a cache line write back (CLWB)

  • Con: Lost bandwidth

  • Preserve the sequential access if all possible

  • Doing a read plus a write, using double the bandwidth

• store: trickle out from the cache into media

  • Con: Lost locality

  • Cache is going to evict the cache line when it decides to, not optimize for evicting things in a sequential pattern (i.e. introduce randomness) --> terrible device utilization

Lesson #3: Limit threads accessing one NVDIMM

• Contention at Optane Buffer: merging adjacent accesses

• Contention at iMC

Contention at Optane Buffer

• Multiple threads: thrash on this on-device cache

  • What would otherwise can be sequential access that can be merged is no longer the case

  • All threads are putting their own writes, lose the opportunity to merge

  • Read

    • Spike: max bandwidth

    • Add more: thrashing the cache

• Multiple threads - multiple DIMM - clogging the single DIMM (bandwidth falls down)

• Ends up with stall

• Exactly interleave size: burst of the accesses (hit the memory controller and the same DIMM)

Lesson #4: avoid mixed and multi-threaded NUMA accesses