Browser performance is mainly measured across 2 metrics: how quickly it loads the pages and how much resources, mainly CPU, it uses. Both of them directly depend on the content loaded as well as network conditions. All tests were performed on a Macbook Pro laptop with Intel Core i5, dual-core CPU and compared Brave 0.24.0 version against Brave Core 0.55.12 Beta release.
Update: Note that we are comparing released versions; Brave 0.24.0 is based on Chromium build 69.0.3497.100, while Brave Core 0.55.12 uses a newer Chromium build 70.0.3538.45. We do not expect the move to a later underlying Chromium version to be responsible for significant performance variations. We don’t observe a statistically significant difference for unthrottled measurements—in fact, Chrome 70 is sightly slower for the sites we considered.
An inherent property of computer networks is their varying performance – latency and bandwidth can both change quickly and there are no minimum quality of service guarantees. This combined with varying server response times and even more – different content delivered in response to subsequent requests create some challenges in detecting performance differences due to changes in the application itself.
We look at Alexa News Top 10 for this post, as they are frequently visited by a lot of people, but are still run by reputable companies that pay attention to their readers. The top list at the time of testing was (descriptions provided by Alexa):
- Reddit.com – User-generated news links. Votes promote stories to the front page.
- Cnn.com – News, weather, sports, and services including e-mail news alerts and downloadable audio/video reports.
- News.google.com. Aggregated headlines and a search engine of many of the world’s news sources.
- Nytimes.com – Online edition of the newspaper’s news and commentary.
- Theguardian.com – Home of the Guardian, Observer and Guardian Weekly newspapers plus special-interest web sites.
- Indiatimes.com – Covers home and international updates, sports, entertainment, health and lifestyle. With sections for weird and trending news.
- Weather.com – Offers forecasts for cities worldwide as well as radar and satellite maps. Also includes news stories and allergy information.
- News.yahoo.com – Daily news and full coverage of current issues.
- Washingtonpost.com – Daily. Offers news, opinion, sports, arts and living and entertainment. Includes archives since 1977 and subscription information.
- Foxnews.com – Breaking News, Latest News and Current News from FoxNews.com. Breaking news and video. Latest Current News: U.S., World, Entertainment, Health, Business, Technology, Politics, Sports.
The sites that are crossed out have paywalls or consent / GDPR compliance walls when visited from Europe and are excluded from further evaluation.
We measured page load times across both browsers using the built-in developer tools, with load times exposed with standard Performance.timing APIs supported by all browsers. While Brave Core makes a lot more sophisticated instrumentation possible, not everything is available on the older version, and we used the manual approach to maintain identical measurement methodology.
Realistic conditions: using mobile settings to reproduce desktop throttling
Developers often forget that while themselves and many power users may be tech-savvy and own the latest hardware – the reality is that the majority of people might not have the latest computers on the fastest networks. It’s important to think about these devices when building for the web, and especially so for Brave, with the browser very popular among mobile users. Although a proper deep-dive into mobile web performance is due in a separate article, we also look at the effects of Brave Core performance changes in realistic environments.
Ben Schwarz in his 2017 article on real world performance compiled some information on real network access out there, with a few pieces of shocking news: 60% of the world’s average mobile connections are 2G and the cost just 500mb of data can be a few days worth of work for some people in India, Brazil or Indonesia. Suddenly a fast 3G connection sounds pretty good!
What does a “Fast 3G connection” mean exactly? The actual settings used by the WPO Foundation (who run the webpagetest.org) are:
- Latency: 150ms
- Throughput: 1.6Mbps down / 750 Kbps up
- Packet loss: none
The same settings are used in Google Lighthouse throttling. Among other limitations, the browser does throttling at request level, rather than packet, which would leave the throttling presets too optimistic, and the Chrome team have worked out that they should use latency and throughput multiplier factors of 3.5x and 0.9x in Lighthouse as well as Chrome to simulate realistic network conditions.
We separately tested our chosen websites with webpagetest.org to confirm the choice of throttling settings with two devices: iPhone SE and Motorola Moto G4. Over 21% of iOS users are still using the iPhone 6s / 6s Plus / SE (all with the same chipset) and the Moto G4 is often the choice for a mid-range, budget Android device. We compared the times of loading live versions of the chosen websites on a Fast 3G connection using standard web browsers (Safari for the iPhone and Google Chrome – for Moto G4) when comparing the upcoming Brave 1.0 browser with 4x and 6x CPU throttling. As expected, there is a lot of variance when comparing real load times against emulated ones, however the Moto G4 is slower than the iPhone, iPhone is closer to 4x CPU throttling and Moto G4 – to 6x.
Overall throttling does give us an idea of how the browser performs on a “much slower device”, but we shouldn’t expect direct per-site comparison to be accurate as the results are noisy when compared to real mobile devices. We hypothesise that some of the bigger differences are due to natural performance variations across the browsers, e.g. on foxnews.com, Safari appears a lot faster than Google Chrome. Chrome’s approach of emulating mobile CPU by simply slowing down the main thread running on a multi-core x86 CPU is only meant as crude approximation of the actual performance a generic resource-poor device that might be running on a lower-frequency ARM CPU with a completely different architecture.
Comparing load times
The metric that is most visible to users is overall page load time as measured by wall clock time. Different metrics can be used to measure perceived loading speed, including ones that look at when most of the visible page has finished painting or when the page is sufficiently interactive to be usable. For comparison between our two browsers it is most appropriate to focus on the overall loading time however, i.e. when the entire DOM was complete, from Navigation Timing Recommendation:
Load Time Gains
We measured page load times on both Brave Core and Muon-based Brave to see if there are any differences, unthrottled (running on a laptop, with all requests served locally from a replay cache) as well as with the different permutations of network and CPU throttling. As mentioned before, some content cannot be replayed completely deterministically and we see some missed content in page traces, e.g. in the case of theguardian.com 4 requests out of more than 260 fail to be replayed.
Slightly surprisingly, the unthrottled case also demonstrates the biggest relative gains, with the tested pages loading 22% on average (with 18% median), which saves between just 100ms for weather.com and up to 1.3s for Fox News, half a second in the median case. When only CPU is slowed down by a factor of 6, the saving is still around 20% on average (24% median), but the absolute saving goes up to over 4 seconds in the best cases, however for Google News we observe a performance degradation. Noticeably, Google News adds last bits of styling by lazy-loading some images and stylesheets after a delay, annotating stylesheets with “data-late-css”, and CPU throttling appears to be interfering with the specific way of late-loading the less critical resources.
While load times due to network throttling can be affected by the changes in how ad-blocking is done within the new browser core, in some cases with both network and CPU throttling we see very small gains or losses that can be attributed to measurement noise. In some cases, the total difference adds up to 9.3s for weather.com and 12.3s for Reddit. This is caused by a combination of how the different bottlenecks in loading a page actually change depending on processing and network speed, as well the limitations of the throttling approach. In fact, the differences with real mobile devices on webpagetest.org highlight the necessity to test on specific hardware, as the very crude approximations do not necessarily paint a realistic picture.
- Calculate style – taking a CSS stylesheet and matching its selector rules with elements in the DOM. This is also known as “formatting.”
- Calculate layout – taking those CSS styles we calculated in step #2 and figuring out where the boxes should be laid out on the screen. This is also known as “reflow.”
- Render – the process of actually putting pixels on the screen. This often involves painting, compositing, GPU acceleration, and a separate rendering thread.
Using browser performance tools, we looked at the details of two pages: 1) theguardian.com, because the savings are very different in throttled and unthrottled cases, and 2) reddit.com, because the savings are big in both cases. It is worth noting that performance tracing adds overheads of its own, so the numbers are not directly compared to the tests above with tracing disabled.
When loading The Guardian with throttling, Rendering and to some extent Scripting and Loading consumed more time, however with network as the bottleneck their impact on load time was hidden. This difference in CPU use, although not necessarily noticeable for the user, would add up to significant differences in energy use, or cause more prominent effects when browsing on a faster network. This is why with no network limitations the difference in load time becomes large, adding up across Loading, Scripting, Rendering and Other tasks with virtually the same idle time.
In the case of Reddit, the most notable difference is in time spent on Scripting as well as Rendering – with CPU throttling those differ by over 6 seconds and almost 3 seconds respectively, with a few more spent on Other tasks. Without CPU throttling, the differences in Scripting and Other tasks cover most of the difference in overall loading time. With the computationally heavy site Brave Core improvements in rendering tasks deliver as much as 8 seconds faster page loads with no additional ads or trackers blocked.
We found a number of interesting results when comparing the upcoming Brave 1.0 browser with the current version of Muon-based Brave on a desktop (laptop) computer. We also looked at what performance might look like on a much slower device by emulating CPU and network slowdown.
- Savings on the desktop: For today’s common laptops with fast connections, we observe load time savings of 10% – 34% on the tested popular media websites with same page content and blocking. Average and median load time savings are 22% and 18% respectively.
- Slower Environments: For slower processors similar to what is found on today’s average Android device on a fast 3G connection, our savings range up to 44%, with the mean and median of 14% and 4% respectively.
- CPU utilization: CPU utilization is better in Brave Core, with all computationally intensive tasks running faster across all tested websites and configurations. We plan to explore the importance of multi-core capacity on the browser speed in an upcoming post.
- Emulation: Emulating hardware and network conditions real users are likely to use helps us see a realistic picture of browser performance improvements. We plan to provide a much more detailed look at mobile performance in the future.
In future posts, we plan to explore performance measurements on mobile devices, for a much more detailed look at what real users gain from using Brave on their mobile devices. We will also be writing about improvements in memory consumption and the importance of multi-core capability on browser speed. Finally, with the recent improvements in browser performance with Brave Core, we have a great platform for further privacy and performance features, some of which we have already written about and some in the works. We will be writing about them too, so stay tuned!