dm btree: fix a recursion depth bug in btree walking code