drm/i915: Workaround incoherence between fences and LLC across multiple CPUs
authorChris Wilson <chris@chris-wilson.co.uk>
Thu, 4 Apr 2013 20:31:03 +0000 (21:31 +0100)
committerDaniel Vetter <daniel.vetter@ffwll.ch>
Thu, 18 Apr 2013 07:43:10 +0000 (09:43 +0200)
In order to fully serialize access to the fenced region and the update
to the fence register we need to take extreme measures on SNB+, and
manually flush writes to memory prior to writing the fence register in
conjunction with the memory barriers placed around the register write.

Fixes i-g-t/gem_fence_thrash

v2: Bring a bigger gun
v3: Switch the bigger gun for heavier bullets (Arjan van de Ven)
v4: Remove changes for working generations.
v5: Reduce to a per-cpu wbinvd() call prior to updating the fences.
v6: Rewrite comments to ellide forgotten history.

Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=62191
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Jon Bloomfield <jon.bloomfield@intel.com>
Tested-by: Jon Bloomfield <jon.bloomfield@intel.com> (v2)
Cc: stable@vger.kernel.org
Reviewed-by: Jesse Barnes <jbarnes@virtuousgeek.org>
Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>
drivers/gpu/drm/i915/i915_gem.c

index 74c5601502939a754d95e1f9711f98596bc655a0..63c05ddea6120bab244080a8978911a627841062 100644 (file)
@@ -2683,17 +2683,35 @@ static inline int fence_number(struct drm_i915_private *dev_priv,
        return fence - dev_priv->fence_regs;
 }
 
+static void i915_gem_write_fence__ipi(void *data)
+{
+       wbinvd();
+}
+
 static void i915_gem_object_update_fence(struct drm_i915_gem_object *obj,
                                         struct drm_i915_fence_reg *fence,
                                         bool enable)
 {
-       struct drm_i915_private *dev_priv = obj->base.dev->dev_private;
-       int reg = fence_number(dev_priv, fence);
-
-       i915_gem_write_fence(obj->base.dev, reg, enable ? obj : NULL);
+       struct drm_device *dev = obj->base.dev;
+       struct drm_i915_private *dev_priv = dev->dev_private;
+       int fence_reg = fence_number(dev_priv, fence);
+
+       /* In order to fully serialize access to the fenced region and
+        * the update to the fence register we need to take extreme
+        * measures on SNB+. In theory, the write to the fence register
+        * flushes all memory transactions before, and coupled with the
+        * mb() placed around the register write we serialise all memory
+        * operations with respect to the changes in the tiler. Yet, on
+        * SNB+ we need to take a step further and emit an explicit wbinvd()
+        * on each processor in order to manually flush all memory
+        * transactions before updating the fence register.
+        */
+       if (HAS_LLC(obj->base.dev))
+               on_each_cpu(i915_gem_write_fence__ipi, NULL, 1);
+       i915_gem_write_fence(dev, fence_reg, enable ? obj : NULL);
 
        if (enable) {
-               obj->fence_reg = reg;
+               obj->fence_reg = fence_reg;
                fence->obj = obj;
                list_move_tail(&fence->lru_list, &dev_priv->mm.fence_list);
        } else {