Embedded panels and its impact on application frame rate and UI fluidity

DISPLAY BASICS [1]

In general, the display subsystem of an embedded system is designed to transfer the data from a block of internal processor memory called a Frame Buffer (FB), which software running on the main Processor updates to change the image being displayed. The display data consists of some number of bits for each pixel of the displayed area. In most cases there are specific bits which describe the intensity of the red, green and blue (RGB) components of each pixel, although other formats such as YUV/YCrCb (luminance, red chrominance, blue chrominance) are occasionally used.

A logic block generally referred to as a Display Controller fetches data from the Frame Buffer, formats it according to the desired display interface, and transmits it to the Display. Figure 1 shows a basic system, where the Display Controller is internal to the Embedded Processor and communicates directly to the Display

 

In many systems the Embedded Processor includes a Display Interface Controller, which creates an intermediate communication structure which is separate from the Display Interface. In this case the Display Controller is external to the Embedded processor, and is often included within the Display itself. This external Display Controller often includes its own Frame Buffer. Systems of this type are shown in Figure 2. Note that it is also possible to connect to an external Display Controller via a standard bus such as PCI.

 

The intensity of each pixel on a display, whether it is a CRT, LCD or other type, generally must be periodically refreshed. The architecture of this function was developed in the days of CRTs, but has remained quite consistent as shown in Figure 3. The refresh is usually done by "raster scanning", which starts at the first pixel (typically the upper left hand corner), generates an intensity for that pixel, and then moves horizontally through all pixels of the first scan line. At that point a "horizontal synchronization" or HSYNC signal occurs, which causes the refresh to move to the beginning of the second line, and so on. This process continues until all lines have been refreshed, at which point a "vertical synchronization" or VSYNC signal occurs. This causes the refresh to return to the first pixel, and the process repeats.

The physical nature of the display generally dictates that the response to HSYNC or VSYNC is not instantaneous, and thus nothing is refreshed for a time after each signal is received. These times are referred to as "blanking periods". They are shown as the dashed lines In Figure 3.

 

Command Mode Panels (Generally used in high end smartphones)

•        DSI (Display Serial Interface) in command mode expects the panel to have an internal frame buffer. The job of DSS (Display Sub System) is just to push out a frame to the panel. The panel has a controller inside it which refreshes the screen using the content of the internal buffer; the panel's controller refreshes the screen using its own timings.

•        In DSI command mode, you control the rate at which data is pushed out to the panel. So if application generates 50 fps of data. It pushes out data at that rate. However, since the panel's controller is refreshing at 60 Hz by itself, you will see tearing since the internal buffer is filled slower than 60 Hz. 

Video Mode Panels (Generally used in Tablets)

•        DSI in video mode requires us to push out data continuously to the panel. The DSS completely controls the panels timings and hence the rate of refresh of the panel. This is needed as the panel has no internal buffer and needs the host to pump out data to it continuously. 

•        In the video mode, the DSS driver provides some API which tells an upper layer the right time to present a new frame to DSS, this is the time when a vsync occurs. If you use this API to wait for a vsync, because of its slow rate of pushing out data, the application is never ready with a frame for every alternate vsync from the panel, hence leading to half the frame rate. 

Conclusion

In general command mode panels have better frame rates than video mode panels for browser use cases because of additional internal frame buffer.

References

(1) http://www.kozio.com/view_files/Embedded_Display_Interfaces_WP_final.pdf

How LinkedIn WebView app mitigate the infinite scrolling: HTML5 developer perspective

This is a 2 part article. Please refer to the following blogs(see below) on issues faced by WebView apps in handling page scrolling scenario (infinite page scroll).

• http://karthkl.blogspot.com/2013/04/another-article-on-why-not-webview-app.html
• http://karthkl.blogspot.com/2013/04/whats-slowing-down-mobile-facebook.html

One advantage gained by the WebView application is the content is infinite and can be fetched from the web servers forever. This results in the DOM tree in WebKit growing infinitely. This advantage of WebView application is used by book store, mp3 store, video store vendors to provide an infinite list of books, songs, videos they prefer to sell. The disadvantage of this model is how effectively the local device memory is managed and how the user interactions are made smooth enough to hit the 60 fps scrolling rate (for the video mode LCD panels).
The first part of the article is looking at the infinite scrolling problem from the HTML5 perspective. Second part of the article will cover from WebKit perspective.

Please refer to this article [http://engineering.linkedin.com/linkedin-ipad-5-techniques-smooth-infinite-scrolling-html5] for what changes as an HTML5 app developer you can adopt to achieve smooth, crash free infinite page scrolling.

In short the article recommends:

• unloading offscreen images
• hiding offscreen pages
• removing offscreen pages
• avoid scaling for images and CSS box-shadow effects
• minimizing DOM nodes

All the above techniques are aimed at  keeping DOM tree to a reasonable size (removing DOM tree nodes which are few pages away from the current view port) thereby ensuring WebKit Threads and UI threads have reasonable data to work with.

How LinkedIn WebView app effectively uses the local storage in mobile devices.

Excerpts from the article: http://engineering.linkedin.com/mobile/linkedin-ipad-using-local-storage-snappy-mobile-apps

In short LinkedIn WebView application is designed bottom up to effectively use local storage.

• they used the local storage as persistent storage
• they reduced the number of network fetches by effectively using the local storage and using a stale time for a resource
• asynchronous network requests similar to AJAX running in background

Another article on why not WebView App - By Zynga

Expressing similar sentiments by Zynga CTO. Excerpts from the article: http://lists.w3.org/Archives/Public/public-web-perf/2012Sep/0016.html

Memory

While the navigation timing APIs help us understand bottlenecks in our loading time performance,

we’d like you to help us analyze what’s happening during runtime.

On mobile devices, the biggest performance bottleneck for large scale, high fidelity apps and games turns out to be memory consumption. As soon as you run out of physical memory, constant swapping occurs and runtime performance of your web project dramatically suffers.

As web developers, we currently have no way of understanding how memory is consumed.

1. Javascript

  *   We don’t know how much memory a newly created object or function allocates

  *   We can’t call the GC manually, and there is no guarantee that calling “delete” will trigger the GC right away

2. Images

  *   We do not have access to the resource of an image, just the DOM representation

  *   We can’t “unload” images reliably

  *   We don’t know when assets are uncompressed

  *   We don’t know when an uncompressed, compressed or both representations of an image is kept in memory and for how long

3. Static assets

  *   Static assets include:

     *   Non-interpreted JavaScript

     *   Stylesheets

     *   Uncompressed images

  *   We want to know how much total memory is consumed by these static assets, as memory is usually shared across all of the above

4. HW Acceleration

This is tricky as it is a platform specific problem, but it doesn’t make it less important. A large number of browsers is using a different render path for content they can hardware accelerate. Said content is usually composited on the CPU first and then uploaded as a texture to the GPU. At this point, it

 becomes a “layer” and easy modifications like opacity and moving the layer around to not trigger recreating the layer. We need information on when layers are created, destroyed and recomposited. We need this to understand performance characteristics, but we also need it to control our total memory heap - an image wrapped in a div with a red border that gets uploaded as a GPU layer consumes twice the amount of uncompressed pixel memory.

We realize and appreciate that browser vendors work hard on abstracting this issue and ideally agree that

 acceleration should be automatic and no concern to the web developer – but reality shows that we do not have that luxury yet, as false layer management is hazardous.

Canvas

Almost all modern browsers hardware accelerate Canvas nowadays and do so by trying to understand how long a certain texture is needed on the GPU through a set of deterministic algorithms. Unfortunately as developer, this often does more harm than good in an environment when you have to be in full control of the situation. We want to be able to understand when textures are kept in a buffer and when they’re released again.

As a practical example: We have implemented a scrolling worldmap as a proof-of-concept in canvas that 

you could pan with the mouse/touch, and one viewport was represented by one image drawn to the canvas. At first, we’d always draw two adjacent pictures (so the user never see’s white background), but if you let go of the mouse, we’d do a “bounce” deceleration of the movement that shows the next logical image in the row. As soon as the logic detects that it needs to draw a new image, texture swapping happens in the background (especially visible on iOS) and you will notice a visible hitch (50+ms).  The unfortunate solution was to always draw all three images at any given time, even when one of them was off -screen. 

5. Total available memory

We want to be able to query the total available physical memory on the device running the browser 

(or if it is capped for my web app, the capped size).

Even though it doesn’t guarantee anything, it allows us to better judge what we can do, and what we shouldn’t do. 

6. Display refresh rates

It needs to be understood how much of a real world problem this is, but it is relevant with any screen that isn’t set to 60 Hz or a multiple of it.

We’d like to query the display refresh rate and the rAF interval in order to minimize jutter.

7. Garbage Collection

We want to be able to:

  *   Trigger garbage collection manually

  *   Understand the execution interval of GC (i.e an event triggered on the window)

  *   Understand the time a GC took (in order to optimize our framerate against it, could be reported through the same event)

  *   Disable GC and only trigger it manually

What's slowing down Mobile Facebook?

Reasons cited by Facebook for switching from WebView to native app in iOS platform. 
In short the issues to some extent can be resolved by exposing new JS APIs:

• API to enable JS to query/manage available physical/image memory, total texture memory, # of tiles, etc.
• API to query HW FPS (PVRTune like)
•  API to query JS heap size, object count
• API to invoke JS GC manually
• API to query display refresh rate and requestAnimationFrame trigger frequency
• API to query browser composition info (# layers, # SW composited, # HW composited, etc.)

Excerpts from http://lists.w3.org/Archives/Public/public-coremob/2012Sep/0021.html.

Following the recent announcements[1] we (Facebook) made about rebuilding
our iOS app using more native technology, we have had a lot of requests to
provide detailed feedback on the performance issues we encountered while
building for the mobile Web. Here it is. Comments welcomed.

1. Tooling / Developer APIs
---------------------------

The lack of tooling in mobile browsers makes it very difficult to dig down
and find out what the real issues are. Hence tooling, or rather,
lack-thereof is a key issue.

The biggest issues we've been facing here are memory related. Given the
size of our content, it's not uncommon for our application to exhaust the
hardware capabilities of the device, causing crashes. Unfortunately, it's
difficult for us to understand exactly what's causing these issues. GPU
buffer exhaustion? Reaching resource limits? Something else? hard to say.

### What's missing? ###

Mainly, dev tools on the device and/or easily accessible remotely.

Things we'd want to know more about as we develop:

#### Down memory lane ####

- Heap size,
- Object count,
- GC cycles,
- GPU buffer size,
- resource limits.

Some of those are very much useful outside of the development phase,
however. E.g.: Linkedin uses UA string sniffing[2] to determine how many
pictures can be kept in memory before hitting the device's limit, an API
to the device's memory resource would be a much more appropriate (albeit
finger-printable) way of doing that.

#### FPS ####

- The ability to measure fps at the hardware level. This is essential for
testing[3], but would also be very useful to determine whether to use
infinite scrolling or pagination, for example. Same fingerprinting caveat
as above.


2. Scrolling performance
------------------------
I've already started sharing some of it with the W3C WebPerf WG[4]. Will
continue bringing it to other relevant WG in the upcoming weeks.

This is one of our most important issues. It's typically a problem on the
newsfeed and on Timeline which use infinite scrolling (content is
prefetched as the user scrolls down the app and appended) and end up
containing large amounts of content (both text AND images). Currently, we
do all of the scrolling using JS, as other options were not fast enough
(because of implementation issues).

### QoI Issues

- Inconsistent framerates, UI thread lag (stuttering).
- GPU buffer exhaustion due to size of content and number of images.
- Native momentum scrolling has a different feel across operating systems.
JS implementation end up being tailored for one OS and feels wrong on
other ones (uncanny valley).
- Perf issue with touch events on Android devices (latency, not enough
events) which makes JS implementations of scrolling more brittle there.

### Requirements:

- Scrolling must be fast and smooth (this is really important for
engagement).
- It must trigger a given set of events so content can be prefetched,
computed and appended as the user scrolls towards the bottom of the loaded
content.
- It must allow i/o and computation in the background (without affecting
the smoothness).
- It must allow appending fresh content to the main content while
scrolling (again, without affecting the smoothness).
- It must reliably handle scrolling though a lot of content, including
lots of images.
- It must be possible to capture touch events during scrolling.

### new API suggestions:

- A standardized way to enable momentum scrolling across browsers.
- `onscroll` events triggered *during* scrolling.
- Structured cloning of rootless document fragments (for building doc
fragments in workers and moving them back to the main thread to be
appended).
- Simple way to implement pull to refresh (via dedicated off-bound-scroll
events?). 


3. GPU
------

Currently, the GPU is a black-box (which from what I understand is what
vendors would like to keep it as). In truth however, developers rely on
tricks[5] to force given content to be hardware accelerated. So it
basically a black-box with a clunky API to add things to it. Given the
size of GPU buffers relative to the size of content consumed on devices
nowadays, I doubt well get to a place where managing GPU can be left
strictly to the browser in a reasonable amount of time.

Think there's value in at least discussing the pros and cons of providing
some form of API to the GPU, if that's possible at all.


4. Other
--------

- Better touch tracking support, especially on Android.
- Smoother animations are always an asset.
- Better caching.
- AppCache is soooooo busted we stopped using it[6].

Best,

--tobie

---
[1]: 
https://www.facebook.com/notes/facebook-engineering/under-the-hood-rebuildi
ng-facebook-for-ios/10151036091753920
[2]: 
http://engineering.linkedin.com/linkedin-ipad-5-techniques-smooth-infinite-
scrolling-html5
[3]: http://lists.w3.org/Archives/Public/public-coremob/2012Aug/0014.html
[4]: http://www.w3.org/2012/09/12-webperf-minutes.html
[5]: http://davidwalsh.name/translate3d
[6]: https://etherpad.mozilla.org/appcache-london

Good resources for Web App developers

(1) To get involved with the World Wide Web Consortium (W3C)

Now anyone can form a W3C community group that could lead to standardization.

W3C Launches Agile Track to Speed Web Innovation

It would be good to monitor these W3C sponsored community groups and any new ones that are formed. I expect some of these to evolve into full W3C level (or even at ISO level) standards. Declarative-3D and Web Payments groups looks interesting from our perspective.

• Colloquial Web
• Declarative 3D for the Web Architecture
• ODRL Initiative
• Ontology-Lexica
• Semantic News
• Web Education
• Web Payments
• XML Performance
• Native Web Apps Community Group
• Augmented Reality Community Group

 (2)  Graphics and OGL programming

•         http://visual.ly/idc-next-level-mobile-web (W3C Core Mob group)
•         http://www.arcsynthesis.org/gltut/index.html

·         

(3) Benchmarks

•        http://www.yafla.com/dforbes/resources/moonbat/moonbat-driver.html (JS Moonbat benchmark)
•        http://www.kaizou.org/code/kaizoumark/ (CSS3 benchmark)
•        http://www.craftymind.com/guimark2/ (Canvas 2D)
•        http://webglsamples.googlecode.com/hg/aquarium/aquarium.html (WebGL)
•       https://github.com/sibblingz/HTML5-Game-Benchmarks (How fast CSS Transforms can be done by a browser – does not test Transitions or Animations though. How Fast drawImage can be done on a 2D Canvas)
•        http://www.robohornet.org/

·         

(4) Testing

·        A neat tool from Google GWT team that will give you lot of insight into a Chrome desktop browser – unlike Firebug/FF or WebInspector/Chrome, Speed Tracer tool will break down all Chrome web processing – including paint, layout, style calculations.But it only works on a desktop Chrome browser and on a dev channel Chrome build.So next time you want to compare a scenario on desktop Chrome, I suggest try it with Speed Tracer.

http://code.google.com/webtoolkit/speedtracer/ (To monitor time spent on different WebKit activities likes loading resources, parsing (html, css, & js),  layout, painting, and rendering)

(5) Good resources on HTML5 know-how

• http://www.webplatform.org/