id_tn_l3/Cleanup process.md

Indonesian tN issues

The correct .md file format is documented at https://www.techadvancement.com/community/train/how-to-format-translation-notes-for-publishing-v-mast

Cleanup steps:

1. Initial inspection of the id_tn_l3 data. Preliminary report to team on 11/12/19.
2. Conference call on 11/20/19. (Chuck, John, Craig, MAx, Tabitha, Christine)
3. Documented correct file format at https://www.techadvancement.com/community/train/how-to-format-translation-notes-for-publishing-v-mast.
4. More extensive analysis and documentation of deviations.
5. Forked the Indonesian repository to https://wacs.bibletranslationtools.org/lversaw/id_tn_l3. This is the WACS workspace for data cleanup for this project.

Automated cleanup steps:

1. Removed the first line of every note file other than the intro.md files. Also removed the second line if blank.
   • Consistently, notes files other than intro.md had two extraneous lines at the top.
   • RISK: might possibly have deleted some valid data.
   • 23,055 files affected
2. Removed the first two lines from intro.md files that start with some variation of "# Pendahuluan"
   • 446 files affected
3. Removed lines containing empty HTML comments, and the lines following, if blank.
   • All HTML comment tags found were empty comments.
   • 469 files affected
4. Converted instances of & nbsp; to a single space.
   • 11694 files affected
5. Removed top line of file if blank. Consolidated consecutive blank lines elsewhere in file.
   • 842 files affected
6. Removed instances of <o:p></o:p> and <o:p> </o:p>
   • They had no apparent purpose or meaning.
   • 620 files affected
7. Removed blank headers and the blank line following (if any).
   • 1858 files affected
8. Fixed language code in tA links. Replaced rc://en/ with rc://id/
   • 20,065 files affected
9. Removed blank lines between list items.
   • 427 files affected
10. Removed high level hash tags in files showing the first classic pattern of corrupted heading levels.
    • Classic pattern means: First heading at level 1. Subsequent headings alternate higher level to level 1.
    • Ends with higher level heading. No untagged text lines anywhere.
    • 681 files affected
11. Promoted headings to level 1 in files showing the second classic pattern of corrupted heading levels.
    • Classic pattern means: First heading at level 2 or higher. Subsequent headings always the same level.
    • Plain text lines alternate with headings. Ends with plain text line.
    • 1665 files affected
12. Removed top two lines of files meeting these criteria:
    • At least 5 lines long
    • First line contains a verse reference (space followed by digits, colon, and digits)
    • Second line is blank, and third line starts with hash mark
    • 3394 files affected
    • RISK: might possibly have deleted some valid data.
13. Removed "Kata-kata Terjemahan" section from files that had it.
    • Those are from an older tN version.
    • 4817 files affected
14. Reapplied #11.
    • 475 files affected
15. Reapplied #10
    • 241 files affected
16. Volunteers manually edited the files identified in issues.txt.
    • about 4300 files affected
17. Renamed all folders and files in Psalms from 2-digit to 3-digit names.
    • 2385 files/folders affected

Remaining issues (will be) documented in issues.txt.

Remaining steps:

TBD