LilyPond Regression Tests

As promised on the lilypond.bugs newsgroup, I've written a C# program that does a pixel-by-pixel check on the "old" and "new" regtest images. It only took 2 or 3 hours to write - it's currently single threaded, so the status display doesn't work, but anyway, it works and shows what's possible. A pixel was counted as "different" if the brightness of the pixels was different by greater than 1, where the brightness scale is 0 to 255. I ran it against the 2.13.39 regtests using the 2.13.39 .exe, comparing the results with the 2.13.40 regtests compiled using the 2.13.40 .exe. It took 30 minutes to run the comparisons (plus about 45 to compile the images with LilyPond) and it threw up 21 image differences. This page shows them all, no matter how trivial. The images have been cropped by hand.

custos.ly. A very slight difference shown enlarged below. It's only a single pixel, but quite different in intensity beween the 2 versions - brightness is 46 in one version and 68 in the other.

figured-bass-continuation-forbid.ly. Well - this is really the one I wrote the program for. As you can see, it clearly picks up the bug fix in the figured bass notation. The red signifies pixels present in .40 and absent in .39.

 

 

les-nereides.ly. Changes in the vertical spacing. To my mind, a difference we should be aware of and see, but not a cause for concern.

markup-brace-warning.ly. Dunno. Scaled up image below:

ottava-broken.ly. Again, vertical spacing changes. To my eye the ottava bracket on .39 is too close to the clef and should have been picked up in previous regtests.

page-breaking-min-systems-per-page2.ly. Vertical spacing changes.

 

 

page-breaking-outside-staff-estimation2.ly. Ditto

 

page-breaking-page-count3.ly. Ditto

page-layout.ly. Presumably changes to the staff/page spacing names.

 

page-spacing.ly. Ditto

 

page-spacing-nonstaff-lines-between.ly. Ditto

 

page-spacing-nonstaff-lines-header-padding.ly. Ditto

 

 

paper-nested-override2.ly. Ditto

 

rest-collision.ly. Vertical spacing changes. Rests look too close to me in .39.

spacing-horizontal-skyline-grace.ly. Not too sure what's happening here.

stem-tremolo.ly. Looking at the originals in photoshop, this looks like the staves have been moved about 1/2 pixel, the note heads not at all, but the tremolo marks a pixel or so. Slightly odd.

tablature-grace-notes.ly. An amazingly minor brightness change at the intersection of a bar line and staff line. It's a red dot at the first bar line, lowest line of the top staff.

test-output-distance.ly. I believe this deliberately changes with every release.

text-spanner-override-order.ly. Vertical spacing changes.

typography-demo.ly. Another pretty minor change - 2 characters in the non-Roman script on the top line. Dunno if this is significant.

 

utf-8.ly Vertical spacing changes. It looks like .39 was too close.