We were not harmed

This can be translated in active form. AT: "Nobody harmed us" or "We were safe" (See: rc://en/ta/man/translate/figs-activepassive)

did not miss anything as long as we went with them

"we did not lose anything when we were with them." David's men kept wild animals and other people from stealing Nabal's flocks.