hashtable: some refactoring, signed-ness

Since 'capacity' is unsigned, so should the index that compares with it.

The (duplicated) compare/resize code can be a bit shorter and (IMO) easier to
read.