Bug 120755

Summary: Lots of pixel tests are failing on first run, so tests cannot finish
Product: WebKit Reporter: WebKit Commit Bot <commit-queue>
Component: Tools / TestsAssignee: Alexey Proskuryakov <ap>
Status: REOPENED    
Severity: Blocker CC: 06_genius.quibble, ap, arianabeballin, asnasachu00, b7r.mmxxx, borisrangelov71, cabatimi, cameronbryan12, cyberflixtvappco, dino, dpranke, emad.zh3, fa-680, glenn, hd86782, izma143, krit, lodi, mhahnenberg, mjs, moonarya123, ngockhanhlam87, odai640680odai, ramtinbeheshti, rexlsf99, rniwa, simon.fraser, sj12152251, skyevo938, thorton, todobancosvenezuela, vodrickcarter987, windymcentire55
Priority: P1 Keywords: MakingBotsRed
Version: 528+ (Nightly build)   
Hardware: Unspecified   
OS: Unspecified   
Bug Depends on: 120919    
Bug Blocks: 50856    
Attachments:
Description Flags
Archive of layout-test-results from webkit-cq-02
none
Archive of layout-test-results from webkit-cq-02
none
proposed fix
rniwa: review+
Archive of layout-test-results from webkit-cq-01
none
Archive of layout-test-results from webkit-cq-01
none
Archive of layout-test-results from webkit-cq-03
none
Archive of layout-test-results from webkit-cq-03 none

WebKit Commit Bot
Reported 2013-09-04 23:48:00 PDT
This is an automatically generated bug from the commit-queue. animations/cross-fade-background-image.html has been flaky on the commit-queue. animations/cross-fade-background-image.html was authored by krit@webkit.org and thorton@apple.com. http://trac.webkit.org/browser/trunk/LayoutTests/animations/cross-fade-background-image.html The commit-queue just saw animations/cross-fade-background-image.html flake (image diff) while processing attachment 210550 [details] on bug 119955. Bot: webkit-cq-02 Port: <class 'webkitpy.common.config.ports.MacPort'> Platform: Mac OS X 10.8.4 The bots will update this with information from each new failure. If you believe this bug to be fixed or invalid, feel free to close. The bots will re-open if the flake re-occurs. If you would like to track this test fix with another bug, please close this bug as a duplicate. The bots will follow the duplicate chain when making future comments.
Attachments
Archive of layout-test-results from webkit-cq-02 (2.18 MB, application/zip)
2013-09-04 23:48 PDT, WebKit Commit Bot 		no flags
Details
Archive of layout-test-results from webkit-cq-02 (2.33 MB, application/zip)
2013-09-06 06:34 PDT, WebKit Commit Bot 		no flags
Details
proposed fix (2.16 KB, patch)
2013-09-06 09:39 PDT, Alexey Proskuryakov 		rniwa: review+
DetailsFormatted DiffDiff
Archive of layout-test-results from webkit-cq-01 (1.44 MB, application/zip)
2013-09-06 22:32 PDT, WebKit Commit Bot 		no flags
Details
Archive of layout-test-results from webkit-cq-01 (1.44 MB, application/zip)
2013-09-06 22:32 PDT, WebKit Commit Bot 		no flags
Details
Archive of layout-test-results from webkit-cq-03 (1.41 MB, application/zip)
2013-09-06 22:34 PDT, WebKit Commit Bot 		no flags
Details
Archive of layout-test-results from webkit-cq-03 (1.41 MB, application/zip)
2013-09-06 22:34 PDT, WebKit Commit Bot 		no flags
Details
View All Add attachment proposed patch, testcase, etc.
WebKit Commit Bot
Comment 1 2013-09-04 23:48:03 PDT
Created attachment 210581 [details] Archive of layout-test-results from webkit-cq-02
Dirk Schulze
Comment 2 2013-09-04 23:59:04 PDT
It might be that there are platform differences when compositing two images. I have the same Mac version and it passes for me. This is a bit strange, since the PNG that I had before passed on the server as well. Not sure why the ref test doesn't. The ref tests worked on the EWS bots too. Needs investigation.
Alexey Proskuryakov
Comment 3 2013-09-05 14:44:40 PDT
I can reproduce this locally. The issue is that this test is color profile sensitive, and run-webkit-tests only forces a color profile for pixel tests, not for reftests. Pixel tests are also used when retrying tests, so these kinds of failures are essentially hidden - the tests are "flaky", but that's not reported loudly enough. Also, regular color profile on some machines likely just happens to match what tests want. Looking at attached test results, we are now getting too many of these failures, so run-webkit-tests doesn't retry, so it finally started reporting those failures. It may be that our tools are stuck until someone who understands color WebCore management can have a look.
WebKit Commit Bot
Comment 4 2013-09-06 06:34:46 PDT
The commit-queue just saw animations/cross-fade-background-image.html flake (image diff) while processing attachment 210723 [details] on bug 120847. Bot: webkit-cq-02 Port: <class 'webkitpy.common.config.ports.MacPort'> Platform: Mac OS X 10.8.4
WebKit Commit Bot
Comment 5 2013-09-06 06:34:50 PDT
Created attachment 210755 [details] Archive of layout-test-results from webkit-cq-02
Dirk Schulze
Comment 6 2013-09-06 06:49:32 PDT
For now we could either turn it back to a pixel test or just check DRT. IMO there would not be much of a difference since we sadly don't run pixel tests on the server.
Alexey Proskuryakov
Comment 7 2013-09-06 09:32:45 PDT
This does kill EWS and other tools. I can't figure out why this became more serious only now, but the only fix I can think of anyway it to roll out <http://trac.webkit.org/changeset/115601>. We need to set the profile not just when running pixel tests, but for reftests too, and that means that we can just as well set it unconditionally.
Alexey Proskuryakov
Comment 8 2013-09-06 09:39:41 PDT
Created attachment 210762 [details] proposed fix
Alexey Proskuryakov
Comment 9 2013-09-06 09:42:42 PDT
*** Bug 120727 has been marked as a duplicate of this bug. ***
Alexey Proskuryakov
Comment 10 2013-09-06 09:42:51 PDT
*** Bug 120729 has been marked as a duplicate of this bug. ***
Alexey Proskuryakov
Comment 11 2013-09-06 09:42:55 PDT
*** Bug 120730 has been marked as a duplicate of this bug. ***
Alexey Proskuryakov
Comment 12 2013-09-06 09:43:03 PDT
*** Bug 120728 has been marked as a duplicate of this bug. ***
Alexey Proskuryakov
Comment 13 2013-09-06 09:43:12 PDT
*** Bug 120731 has been marked as a duplicate of this bug. ***
Alexey Proskuryakov
Comment 14 2013-09-06 09:43:21 PDT
*** Bug 120732 has been marked as a duplicate of this bug. ***
Alexey Proskuryakov
Comment 15 2013-09-06 09:43:30 PDT
*** Bug 120734 has been marked as a duplicate of this bug. ***
Alexey Proskuryakov
Comment 16 2013-09-06 09:43:38 PDT
*** Bug 120733 has been marked as a duplicate of this bug. ***
Alexey Proskuryakov
Comment 17 2013-09-06 09:43:45 PDT
*** Bug 120735 has been marked as a duplicate of this bug. ***
Alexey Proskuryakov
Comment 18 2013-09-06 09:43:49 PDT
*** Bug 120736 has been marked as a duplicate of this bug. ***
Alexey Proskuryakov
Comment 19 2013-09-06 09:43:53 PDT
*** Bug 120737 has been marked as a duplicate of this bug. ***
Alexey Proskuryakov
Comment 20 2013-09-06 09:43:57 PDT
*** Bug 120738 has been marked as a duplicate of this bug. ***
Alexey Proskuryakov
Comment 21 2013-09-06 09:44:01 PDT
*** Bug 120739 has been marked as a duplicate of this bug. ***
Alexey Proskuryakov
Comment 22 2013-09-06 09:44:05 PDT
*** Bug 120740 has been marked as a duplicate of this bug. ***
Alexey Proskuryakov
Comment 23 2013-09-06 09:44:09 PDT
*** Bug 120741 has been marked as a duplicate of this bug. ***
Alexey Proskuryakov
Comment 24 2013-09-06 09:44:13 PDT
*** Bug 120742 has been marked as a duplicate of this bug. ***
Alexey Proskuryakov
Comment 25 2013-09-06 09:44:23 PDT
*** Bug 120743 has been marked as a duplicate of this bug. ***
Alexey Proskuryakov
Comment 26 2013-09-06 09:44:26 PDT
*** Bug 120744 has been marked as a duplicate of this bug. ***
Alexey Proskuryakov
Comment 27 2013-09-06 09:44:30 PDT
*** Bug 120745 has been marked as a duplicate of this bug. ***
Alexey Proskuryakov
Comment 28 2013-09-06 09:44:34 PDT
*** Bug 120746 has been marked as a duplicate of this bug. ***
Alexey Proskuryakov
Comment 29 2013-09-06 09:44:38 PDT
*** Bug 120747 has been marked as a duplicate of this bug. ***
Alexey Proskuryakov
Comment 30 2013-09-06 09:44:44 PDT
*** Bug 120748 has been marked as a duplicate of this bug. ***
Alexey Proskuryakov
Comment 31 2013-09-06 09:44:49 PDT
*** Bug 120749 has been marked as a duplicate of this bug. ***
Alexey Proskuryakov
Comment 32 2013-09-06 09:44:53 PDT
*** Bug 120750 has been marked as a duplicate of this bug. ***
Alexey Proskuryakov
Comment 33 2013-09-06 09:44:57 PDT
*** Bug 120751 has been marked as a duplicate of this bug. ***
Alexey Proskuryakov
Comment 34 2013-09-06 09:45:07 PDT
*** Bug 120752 has been marked as a duplicate of this bug. ***
Alexey Proskuryakov
Comment 35 2013-09-06 09:45:11 PDT
*** Bug 120753 has been marked as a duplicate of this bug. ***
Alexey Proskuryakov
Comment 36 2013-09-06 09:45:14 PDT
*** Bug 120754 has been marked as a duplicate of this bug. ***
Ryosuke Niwa
Comment 37 2013-09-06 10:30:06 PDT
Comment on attachment 210762 [details] proposed fix View in context: https://bugs.webkit.org/attachment.cgi?id=210762&action=review > Tools/Scripts/webkitpy/layout_tests/controllers/manager.py:153 > - if self._options.pixel_tests: > - self._printer.write_update("Starting pixel test helper ...") > - self._port.start_helper() > + self._printer.write_update("Starting pixel test helper ...") > + self._port.start_helper() We should be able to detect whether we're running a ref test or not but I suppose we can live with it for now. Perhaps we should do this only for Mac port?
Alexey Proskuryakov
Comment 38 2013-09-06 11:14:30 PDT
Committed <http://trac.webkit.org/r155196>. Looks like this may not help EWS though! I know that this fixes it for me locally, but why is EWS still yellow? I'll have to keep looking. > We should be able to detect whether we're running a ref test or not but I suppose we can live with it for now. Yeah, I guess that would be helpful for people who only ever run fast/js tests. Otherwise, we have reftests everywhere, even in http directory. > Perhaps we should do this only for Mac port? Currently, only the Mac port has a helper, it's a no-op for others.
Dirk Pranke
Comment 39 2013-09-06 11:41:10 PDT
(In reply to comment #37) > We should be able to detect whether we're running a ref test or not but I suppose we can live with it for now. Given that we don't have a global list of reftests (we check each test on demand), you'd have to check that up front, and that could be fairly expensive.
Dirk Schulze
Comment 40 2013-09-06 11:44:09 PDT
(In reply to comment #38) > Committed <http://trac.webkit.org/r155196>. > > Looks like this may not help EWS though! I know that this fixes it for me locally, but why is EWS still yellow? I'll have to keep looking. Currently the expected file on animations/cross-fade-background-image.html has a result that worked for me on my machine. If this test is failing now, I can submit another result.
Alexey Proskuryakov
Comment 41 2013-09-06 12:28:12 PDT
Looks like animations/cross-fade-background-image.html now passes on WK1, but it's still failing (with color management badness) on WK2. We might need a separate new bug for this test specifically.
WebKit Commit Bot
Comment 42 2013-09-06 21:28:22 PDT
Re-opened since this is blocked by bug 120919
Ryosuke Niwa
Comment 43 2013-09-06 21:36:57 PDT
The EWS bots were simply encountering the display initialization problem. The right fix is to log into each one of them via VNC/screen share.
WebKit Commit Bot
Comment 44 2013-09-06 22:32:10 PDT
The commit-queue just saw fast/inline/layout-after-inserting-nested-br.html flake (image diff) while processing attachment 210780 [details] on bug 120884. Bot: webkit-cq-01 Port: <class 'webkitpy.common.config.ports.MacPort'> Platform: Mac OS X 10.8.4
WebKit Commit Bot
Comment 45 2013-09-06 22:32:16 PDT
Created attachment 210870 [details] Archive of layout-test-results from webkit-cq-01
WebKit Commit Bot
Comment 46 2013-09-06 22:32:27 PDT
The commit-queue just saw fast/shapes/shape-inside/shape-inside-overflow-fixed-dimensions-block-content.html flake (image diff) while processing attachment 210780 [details] on bug 120884. Bot: webkit-cq-01 Port: <class 'webkitpy.common.config.ports.MacPort'> Platform: Mac OS X 10.8.4
WebKit Commit Bot
Comment 47 2013-09-06 22:32:32 PDT
Created attachment 210874 [details] Archive of layout-test-results from webkit-cq-01
WebKit Commit Bot
Comment 48 2013-09-06 22:34:11 PDT
The commit-queue just saw fast/shapes/shape-outside-floats/shape-outside-floats-image-002.html flake (image diff) while processing attachment 210820 [details] on bug 120911. Bot: webkit-cq-03 Port: <class 'webkitpy.common.config.ports.MacPort'> Platform: Mac OS X 10.8.4
WebKit Commit Bot
Comment 49 2013-09-06 22:34:17 PDT
Created attachment 210896 [details] Archive of layout-test-results from webkit-cq-03
WebKit Commit Bot
Comment 50 2013-09-06 22:34:26 PDT
The commit-queue just saw fast/shapes/shape-outside-floats/shape-outside-floats-image-001.html flake (image diff) while processing attachment 210820 [details] on bug 120911. Bot: webkit-cq-03 Port: <class 'webkitpy.common.config.ports.MacPort'> Platform: Mac OS X 10.8.4
WebKit Commit Bot
Comment 51 2013-09-06 22:34:32 PDT
Created attachment 210897 [details] Archive of layout-test-results from webkit-cq-03
Ryosuke Niwa
Comment 52 2013-09-06 23:25:40 PDT
I've rebooted all EWS bots, manually logged into them via screen sharing, and manually set sRGB as the color profile. We'll see if test failures will go away now. If not, then something must have regressed in DRT/WTR or WebKit itself.
Ryosuke Niwa
Comment 53 2013-09-06 23:51:13 PDT
Some bots are still seeing image only failures. However, it appears that tests do pass on those bots immediately after I've manually logged into them via screen sharing. I've realized that screen saver was enabled on some of them so I've disabled them manually. Is it possible for screen saver somehow affect the color profile or accelerated compositing?
Ryosuke Niwa
Comment 54 2013-09-06 23:58:57 PDT
Not fixed. Somehow a different color profile is used when accelerated compositing is enabled.
Mark Hahnenberg
Comment 55 2014-05-23 19:26:25 PDT
Is this bug still a thing? css2.1/20110323/border-conflict-element-001d.htm is flaking a lot on the Mavericks Debug WK2 bots, and the bug I found for that test was duped to this one.
Alexey Proskuryakov
Comment 56 2014-05-23 23:44:36 PDT
Color management troubles are still a thing indeed, although I'm not sure if this bug is the best way to track fixing them. For now, I've switched the bot that had trouble with css2.1/20110323/border-conflict-element-001d.htm to sRGB manually, which should resolve this (and I believe someone else - maybe Beth - has switched one or more bots to sRGB this week already).
izma143
Comment 68 2024-11-26 18:39:58 PST Comment hidden (spam)
Boris Rangelov
Comment 69 2024-12-19 19:33:46 PST Comment hidden (spam)
Note You need to log in before you can comment on or make changes to this bug.