the rest who did the work

This refers to the men who would later rebuild the walls. Alternate translation: "the others who would later do the work of rebuilding the walls" (See: rc://en/ta/man/translate/figs-explicit)