Bug 230617

Summary:

gl.texImage2D upload of getUserMedia streams via <video> element fails

Product:

WebKit

Reporter:

connell

Component:

WebGL

Assignee:

Kimmo Kinnunen <kkinnunen>

Status:

RESOLVED FIXED

Severity:

Critical

CC:

bfulgham, brianpeiris, calvaris, cdumez, cgarcia, changseok, chris, dino, eric.carlson, esprehn+autocc, ews-watchlist, glenn, gustavo, gyuyoung.kim, hta, jer.noble, joel, kbr, kkinnunen, kondapallykalyan, menard, michael, philipj, pnormand, rigel, sergio, simontaylor1, tommyw, tony, vjaquez, webkit-bug-importer, youennf

Priority:

Keywords:

InRadar

Version:

Other

Hardware:

iPhone / iPad

OS:

Other

See Also:

https://bugs.webkit.org/show_bug.cgi?id=230589
https://bugs.webkit.org/show_bug.cgi?id=228821

Bug Depends on:

Bug Blocks:

232727

Attachments:

Description	Flags
Patch	ews-feeder: commit-queue-
Patch	none

connell

Reported 2021-09-22 05:59:04 PDT

Uploading a camera stream to a WebGL texture using getUserMedia(...), a <video> element, and gl.texImage2D(...) is failing in Safari on the iOS 15.1 beta 1 release. It works correctly on the release version of iOS 15.0 and earlier. I've prepared a minimal test case here: https://clv.zappar.io/3571850569055456732/1.0.0/ The page should show a live camera feed in the canvas after allowing camera permissions. On iOS 15.1 beta 1 no camera feed appears. The source for this test case is entirely within the page itself should you wish to take a look at how it's implemented. This is a common flow for web-based augmented reality and image processing applications. We have many sites and customers who rely on this behaviour to work correctly, so this is a major regression and concern for us. The same issue is also reported in the following bug but I've filed this one with a minimal test case and a more specific title in the hopes that it helps surface this significant regression to the right team :-) https://bugs.webkit.org/show_bug.cgi?id=230589 Let me know if I can help with any questions!

Attachments
Patch (52.27 KB, patch) 2021-09-24 08:39 PDT, Kimmo Kinnunen	ews-feeder: commit-queue-	Details Formatted Diff Diff
Patch (14.27 KB, patch) 2022-01-14 04:22 PST, Kimmo Kinnunen	no flags	Details Formatted Diff Diff
View All Add attachment proposed patch, testcase, etc.

Radar WebKit Bug Importer

Comment 1 2021-09-22 10:45:26 PDT

<rdar://problem/83407577>

Dean Jackson

Comment 2 2021-09-22 11:37:39 PDT

*** Bug 230589 has been marked as a duplicate of this bug. ***

Brent Fulgham

Comment 3 2021-09-22 11:43:48 PDT

It looks like this regressed in Bug 228821.

Simon Taylor

Comment 4 2021-09-23 02:51:38 PDT

Thanks Brett. I see the commit in that bug references it affecting the CPU codepath only - @kkinnunen is this perhaps another case like in 216259 and 215908 where some change has forced these uploads down the CPU path rather than the GPU one?

Kimmo Kinnunen

Comment 5 2021-09-24 08:39:52 PDT

Created attachment 439150 [details] Patch

Eric Carlson

Comment 6 2021-09-24 09:03:37 PDT

Comment on attachment 439150 [details] Patch View in context: https://bugs.webkit.org/attachment.cgi?id=439150&action=review > Source/WebCore/html/HTMLVideoElement.cpp:-318 > -RefPtr<NativeImage> HTMLVideoElement::nativeImageForCurrentTime() > -{ > - if (!player()) > - return nullptr; > - > - return player()->nativeImageForCurrentTime(); > -} > - I would rather not remove this as I think MediaPlayer should be an implementation detail, and I'd like to make it private. > Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateAVFoundationObjC.mm:2559 > + if (m_lastImage) > + return std::nullopt; You return null if there *is* an image?

Kenneth Russell

Comment 7 2021-09-24 11:00:19 PDT

Comment on attachment 439150 [details] Patch View in context: https://bugs.webkit.org/attachment.cgi?id=439150&action=review >> Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateAVFoundationObjC.mm:2559 >> + return std::nullopt; > > You return null if there *is* an image? Looks like a typo compared to MediaPlayerPrivateMediaStreamAVFObjC - maybe the cause of the widespread test failures.

Kenneth Russell

Comment 8 2021-09-24 11:00:28 PDT

Kenneth Russell

Comment 9 2021-09-24 11:01:32 PDT

The new accelerated code paths added in this patch look very cool though!

connell

Comment 10 2021-09-29 07:39:51 PDT

Just a note to say that we still see this issue on iOS 15.1 Beta 2.

Brent Fulgham

Comment 11 2021-09-29 08:55:59 PDT

(In reply to connell from comment #10) > Just a note to say that we still see this issue on iOS 15.1 Beta 2. I believe this will be part of Beta 3.

Kimmo Kinnunen

Comment 12 2021-09-29 22:36:25 PDT

*** Bug 230879 has been marked as a duplicate of this bug. ***

Kimmo Kinnunen

Comment 13 2022-01-13 05:11:59 PST

Regression was fixed by reverting r280963 (Bug 228821) for iOS 15.2 (rdar://83587220) This bug still persists in trunk. Repurposing this bug to fix in trunk.

Kimmo Kinnunen

Comment 14 2022-01-14 04:22:31 PST

Created attachment 449160 [details] Patch

youenn fablet

Comment 15 2022-01-14 06:15:19 PST

Comment on attachment 449160 [details] Patch View in context: https://bugs.webkit.org/attachment.cgi?id=449160&action=review > Source/WebCore/ChangeLog:9 > + Fix MSE camera to WebGL texture uploads. s/MSE/MediaStreamTrack/ > Source/WebCore/ChangeLog:12 > + Turns out MSE does not have these implemented, so currently fall back It should be easy to add such methods. We have m_currentVideoSample we can grab for that using a lock.

EWS

Comment 16 2022-01-14 12:42:29 PST

Committed r288025 (246051@main): <https://commits.webkit.org/246051@main> All reviewed patches have been landed. Closing bug and clearing flags on attachment 449160 [details].

rigel

Comment 17 2022-01-29 21:10:11 PST

I'm currently seeing this behavior (black screen, no camera feed) on iOS 15.4b1 (iPhone 11) on all 8th Wall and Zappar websites, including the test case listed at the top: https://clv.zappar.io/3571850569055456732/1.0.0/

Note You need to log in before you can comment on or make changes to this bug.