I've made more progress:
The problem I ran into today is that, since the pattern on the paper behind the weaving is not held close to the weaving, I must lift it near the weaving to decide where to fill with each colour. I discovered today that I must have been holding it weird for much of the left half as now some of the lines don't line up to what I've woven. This isn't really an issue with what I've done so far as I think it looks ok (I'm trying to ignore the not-straight lines), but I'm not sure how to continue. I can't just shift the lines over to the pattern as that would make the lines be crooked. I will have to try to either adjust the lines on the paper pattern or just try to keep the weaving a certain distance from the existing lines.
I think the difference was caused by lifting the paper from the side as I did on the left. I think on the right that I put my hand in under the front of the loom which made the paper lay flatter against the weaving. Since I had a bit of a rough day today, I felt it best to not mess with it tonight.