lib: cpu_rmap: avoid flushing all workqueues