blogccasion

Remember the Cross-Origin-Embedder-Policy (COEP) and the Cross-Origin-Opener-Policy (COOP) headers for making your site cross-origin isolated? If not, here's my colleague Eiji Kitamura's article Making your website "cross-origin isolated" using COOP and COEP . To be effective, they need to be sent as in the example below.

cross-origin-embedder-policy: credentialless
cross-origin-opener-policy: same-origin

Cross-origin isolated documents operate with fewer restrictions when using the following APIs:

SharedArrayBuffer can be created and sent via a Window.postMessage() or a MessagePort.postMessage() call. Performance.now() offers better precision. Performance.measureUserAgentSpecificMemory() can be called.

Typically, sending non-default HTTP headers like COOP and COEP means controlling the server so you can configure it to send them. I recently learned that they are also honored if set through a service worker 🤯! This means you can make apps on static hosting like on GitHub Pages cross-origin isolated!

• Blog post by stefnotch: Enabling COOP/COEP without touching the server
• Library by Guido Zuidhof: coi-serviceworker
• Demo: Are we cross-origin isolated?

One example where cross-origin isolating your site is needed is with SQLite Wasm when you want to use persistent storage with the origin private file system virtual file system called OPFS sqlite3_vfs. I'm glad to have this coi-serviceworker trick up my sleeve now, and you do, too!

This post appeared first on https://blog.tomayac.com/2025/03/08/setting-coop-coep-headers-on-static-hosting-like-github-pages/.

Recently, in a blog post titled Running inference in web extensions, Mozilla announced a pretty interesting experiment on their blog:

We've recently shipped a new component inside of Firefox that leverages Transformers.js […] and the underlying ONNX runtime engine. This component lets you run any machine learning model that is compatible with Transformers.js in the browser, with no server-side calls beyond the initial download of the models. This means Firefox can run everything on your device and avoid sending your data to third parties.

They expose this component to Web extensions under the browser.trial.ml namespace. Where it gets really juicy is at the detail how models are stored (emphasis mine):

Model files are stored using IndexedDB and shared across origins

Typically when you develop an app with Transformers.js, the model needs to be cached for each origin separately, so if two apps on different origins end up using the same model, the model needs to be downloaded and stored redundantly. (Together with Chris and François, I have thought about this problem, too, but that's not the topic of this blog post.)

To get a feeling for the platform, I extracted their example extension from the Firefox source tree and put it separately in a GitHub repository, so you can more easily test it on your own.

1. Make sure that the following flags are toggled to true on the special about:config page:

   browser.ml.enable
   extensions.ml.enabled

2. Check out the source code.

   git clone git@github.com:tomayac/firefox-ml-extension.git

3. Load the extension as a temporary extension on the This Nightly tab of the special about:debugging page. It's important to actually use Firefox Nightly.

   

4. After loading the extension, you're brought to the welcome page, where you need to grant the ML permission. The permission reads "Example extension requests additional permissions. It wants to: Download and run AI models on your device". In the manifest.json, it looks like this:

   {
     "optional_permissions": ["trialML"]
   }

   

5. After granting permission, right-click any image on a page, for example, Unsplash. In the context menu, select ✨ Generate Alt Text.

   

6. If this was the first time, this triggers the download of the model. On the JavaScript code side, this is the relevant part:

   // Initialize the event listener
   browser.trial.ml.onProgress.addListener((progressData) => {
     console.log(progressData);
   });
   
   // Create the inference engine. This may trigger model downloads.
   await browser.trial.ml.createEngine({
     modelHub: 'mozilla',
     taskName: 'image-to-text',
   });

   You can see the extension display download progress in the lower left corner.

   

7. Once the model download is complete, the inference engine is ready to run.

   // Call the engine.
   const res = await browser.trial.ml.runEngine({
     args: [imageUrl],
   });
   console.log(res[0].generated_text);

   It's not the most detailed description, but "A computer desk with a monitor, keyboard, and a plant" definitely isn't wrong.

   

   If you click Inspect on the extension debugging page, you can play with the WebExtensions AI APIs directly.

   

8. The browser.trial.ml namespace exposes the following functions:

   • createEngine(): creates an inference engine.
   • runEngine(): runs an inference engine.
   • onProgress(): listener for engine events
   • deleteCachedModels(): delete model(s) files

   

   I played with various tasks, and initially, I had some trouble getting translation to run, so I hopped on the firefox-ai channel on the Mozilla AI Discord, where Tarek Ziade from the Firefox team helped me out and also pointed me at about:inference, another cool special page in Firefox Nightly where you can manage the installed AI models. If you want to delete models from JavaScript, it seems like it's all or nothing, as the deleteCachedModels() function doesn't seem to take an argument. (It also threw a DOMException when I tried to run it on Firefox Nightly 137.0a1.)

   // Delete all AI models.
   await browser.trial.ml.deleteCachedModels();

   

9. The about:inference page also lets you play directly with many AI tasks supported by Transformers.js and hence Firefox WebExtensions AI APIs.

   

Concluding, I think this is a very interesting way of working with AI inference in the browser. The obvious downside is that you need to convince your users to download an extension, but the obvious upside is that you possibly can save them from having to download a model they may already have downloaded and stored on their disk. When you experiment with AI models a bit, disk space can definitely become a problem, especially on smaller SSDs, which led me to a fun random discovery the other day, when I was trying to free up some disk space for Gemini Nano…

As teased before, Chris, François, and I have some ideas around cross-origin storage in general, but the Firefox WebExtensions AI APIs definitely solve the problem for AI models. Be sure to read their documentation and play with their demo extension! On the Chrome team, we're experimenting with built-in AI APIs in Chrome. It's a very exciting space for sure! Special thanks again to Tarek Ziade on the Mozilla AI Discord for his help in getting me started.

This post appeared first on https://blog.tomayac.com/2025/02/07/playing-with-ai-inference-in-firefox-web-extensions/.

I'm not a big LinkedIn user, but the other day, my Google colleague Franziska Hinkelmann posted something about a project called browser-use that caught my eye:

Got low stakes repetitive tasks in the browser? Playwright + LLMs (Gemini 2.0) to the rescue! Super easy to make somebody else cough agents cough do the work for you, especially if you have to repeat a task for many rows in a Google Sheet.

After seeing her demo, I went and tried it out myself. Here are the steps that worked for me on macOS:

1. Install uv following their installation instructions. (The usual caveat of first checking the source code before pasting anything in the Terminal applies.)

   curl -LsSf https://astral.sh/uv/install.sh | less

2. Create a new Python environment and activate it. This is from browser-use's quickstart instructions.

   uv venv --python 3.11
   source .venv/bin/activate

3. Install the dependencies and Playwright.

   uv pip install browser-use
   playwright install

4. Create a .env file and add your OpenAI API key in the form OPENAI_API_KEY=abc123.

5. Create an agent.py file with the source code of your agent. Here's the one I tried. As you can see, I'm tasking the agent with the following job: "Go to developer.chrome.com and find out what built-in AI APIs Chrome supports".

   from langchain_openai import ChatOpenAI
   from browser_use import Agent
   import asyncio
   from dotenv import load_dotenv
   load_dotenv()
   
   async def main():
       agent = Agent(
           task="Go to developer.chrome.com and find out what built-in AI APIs Chrome supports.",
           llm=ChatOpenAI(model="gpt-4o"),
       )
       result = await agent.run()
       print(result)
   
   asyncio.run(main())

The coolest thing is that you can watch the agent do its job in real time and see the reasoning steps in the command line.

Here's an excerpt of the logs.

INFO     [agent] 🚀 Starting task: Go to developer.chrome.com and find out what built-in AI APIs Chrome supports.
INFO     [agent] 📍 Step 1
INFO     [agent] 🤷 Eval: Unknown - It's the starting state.
INFO     [agent] 🧠 Memory: Start searching for built-in AI APIs supported by Chrome on developer.chrome.com.
INFO     [agent] 🎯 Next goal: Navigate to developer.chrome.com
INFO     [agent] 🛠️  Action 1/1: {"go_to_url":{"url":"https://developer.chrome.com"}}
INFO     [controller] 🔗  Navigated to https://developer.chrome.com
INFO     [agent] 📍 Step 2
INFO     [agent] 👍 Eval: Success - Navigated to the correct URL.
INFO     [agent] 🧠 Memory: Visited developer.chrome.com. Need to search for information on built-in AI APIs supported by Chrome.
INFO     [agent] 🎯 Next goal: Use search input to find information on AI APIs.
INFO     [agent] 🛠️  Action 1/1: {"click_element":{"index":9}}
INFO     [controller] 🖱️  Clicked button with index 9:
INFO     [agent] 📍 Step 3
INFO     [agent] 👍 Eval: Success - The search bar was clicked but no input was entered yet.
INFO     [agent] 🧠 Memory: Need to input search query about AI APIs in Chrome using the search bar.
INFO     [agent] 🎯 Next goal: Enter 'AI APIs' in the search input to find relevant information.
INFO     [agent] 🛠️  Action 1/1: {"input_text":{"index":4,"text":"AI APIs"}}
INFO     [controller] ⌨️  Input AI APIs into index 4
[…]

At the end of the log messages is the agent's full response to your task, formatted in Markdown. FWIW, the answer is exactly correct:

Chrome supports several built-in AI APIs, including:

1. Translator API - Available in origin trial for early preview participants. Use cases include on-demand translation and supporting multi-language communication.
2. Language Detector API - Available in Chrome origin trial. It helps detect languages without user input and can label texts for better screen reader pronunciation.
3. Summarizer API - Allows for local experimentation to preview program participants. It can summarize meeting transcripts, articles, and forum questions.
4. Writer and Rewriter APIs - Experimental status in early preview program, used for creating and refining text content.
5. Prompt API - Allows natural language requests to Gemini Nano in Chrome, in an experimental early stage.

Visit developer.chrome.com for complete details and participation in early trials.

It's pretty wild what this scriptable agent is capable of doing today. Be sure to check out some of the other demos and also try the browser-use web-ui, which adds a nice UI on top.

This post appeared first on https://blog.tomayac.com/2025/02/05/testing-browser-use-a-scriptable-ai-browser-agent/.

I love hackable URLs. A hackable URL is a URL that makes sense to a human reader, and where the human reader can guess what to change to get to another page. For example, if you look at the URL of this very blog post, https://blog.tomayac.com/2024/11/02/eleventy-11ty-year-year-month-and-year-month-day-indexes/, what would you expect happens if you hack the URL to any of the following values?

• https://blog.tomayac.com/2024/11/02/ lists all posts that were published on November 2, 2024.
• https://blog.tomayac.com/2024/11/ lists all posts that were published in November, 2024.
• https://blog.tomayac.com/2024/ lists all posts that were published in 2024.
• https://blog.tomayac.com/ leads to this blog's home.

.eleventy.js 🔗

If you have a URL structure that's similar to mine, feel free to copy the relevant excerpts quoted in the following snippet from my .eleventy.js and add them to your .eleventy.js:

// Year collection
eleventyConfig.addCollection('postsByYear', (collection) => {
  return _.chain(collection.getAllSorted())
    .filter((item) => 'tags' in item.data && item.data.tags.includes('posts'))
    .groupBy((post) => post.date.getFullYear())
    .toPairs()
    .reverse()
    .value();
});

// Year / Month collection
eleventyConfig.addCollection('postsByYearMonth', (collection) => {
  return _.chain(collection.getAllSorted())
    .filter((item) => 'tags' in item.data && item.data.tags.includes('posts'))
    .groupBy((post) => {
      const year = post.date.getFullYear();
      const month = String(post.date.getMonth() + 1).padStart(2, '0');
      return `${year}/${month}`;
    })
    .toPairs()
    .reverse()
    .value();
});

// Year / Month / Day collection
eleventyConfig.addCollection('postsByYearMonthDay', (collection) => {
  return _.chain(collection.getAllSorted())
    .filter((item) => 'tags' in item.data && item.data.tags.includes('posts'))
    .groupBy((post) => {
      const year = post.date.getFullYear();
      const month = String(post.date.getMonth() + 1).padStart(2, '0');
      const day = String(post.date.getDate()).padStart(2, '0');
      return `${year}/${month}/${day}`;
    })
    .toPairs()
    .reverse()
    .value();
});

// Helper filter to format month names
eleventyConfig.addFilter('monthName', (monthNum) => {
  const date = new Date(2000, parseInt(monthNum) - 1, 1);
  return date.toLocaleString('en-US', { month: 'long' });
});

// Helper filters for parsing date parts
eleventyConfig.addFilter('getYear', (dateStr) => dateStr.split('/')[0]);
eleventyConfig.addFilter('getMonth', (dateStr) => dateStr.split('/')[1]);
eleventyConfig.addFilter('getDay', (dateStr) => dateStr.split('/')[2]);

Nunjucks templates 🔗

Then, in your blog's root, add three files:

• year-index.njk
• year-month-index.njk
• year-month-day-index.njk

They're all three pretty similar, but for the sake of completeness, here are all three.

year-index.njk: 🔗

---
pagination:
  data: collections.postsByYear
  size: 1
  alias: year
layout: layouts/home.njk
permalink: /{{ year[0] }}/
---

<h2>{{ year[0] }} Archive</h2>

{% for postedYear, yearPosts in collections.postsByYear %}
  {% if postedYear === year[0] %}
  <ul class="postlist">
    {% for post in yearPosts | reverse %}
      <li class="postlist-item{% if post.url == url %} postlist-item-active{% endif %}">
        <a href="{{ post.url | url }}" class="postlist-link">{% if post.data.title %}{{ post.data.title }}{% else %}<code>{{ post.url }}</code>{% endif %}</a>
        <time class="postlist-date" datetime="{{ post.date | htmlDateString }}">{{ post.date | readableDate }}</time>
      </li>
    {% endfor %}
  </ul>
  {% endif %}
{% endfor %}

year-month-index.njk: 🔗

---
pagination:
  data: collections.postsByYearMonth
  size: 1
  alias: yearMonth
layout: layouts/home.njk
permalink: /{{ yearMonth[0] }}/
---

<h2>{{ yearMonth[0] | getMonth | monthName }} {{ yearMonth[0] | getYear }} Archive</h2>

{% for postedYearMonth, monthPosts in collections.postsByYearMonth %}
  {% if postedYearMonth === yearMonth[0] %}
  <ul class="postlist">
    {% for post in monthPosts | reverse %}
      <li class="postlist-item{% if post.url == url %} postlist-item-active{% endif %}">
        <a href="{{ post.url | url }}" class="postlist-link">{% if post.data.title %}{{ post.data.title }}{% else %}<code>{{ post.url }}</code>{% endif %}</a>
        <time class="postlist-date" datetime="{{ post.date | htmlDateString }}">{{ post.date | readableDate }}</time>
      </li>
    {% endfor %}
  </ul>
  {% endif %}
{% endfor %}

year-month-day-index.njk: 🔗

---
pagination:
  data: collections.postsByYearMonthDay
  size: 1
  alias: yearMonthDay
layout: layouts/home.njk
permalink: /{{ yearMonthDay[0] }}/
---

<h2>{{ yearMonthDay[0] | getMonth | monthName }} {{ yearMonthDay[0] | getDay }}, {{ yearMonthDay[0] | getYear }} Archive</h2>

{% for postedYearMonthDay, dayPosts in collections.postsByYearMonthDay %}
  {% if postedYearMonthDay === yearMonthDay[0] %}
  <ul class="postlist">
    {% for post in dayPosts | reverse %}
      <li class="postlist-item{% if post.url == url %} postlist-item-active{% endif %}">
        <a href="{{ post.url | url }}" class="postlist-link">{% if post.data.title %}{{ post.data.title }}{% else %}<code>{{ post.url }}</code>{% endif %}</a>
        <time class="postlist-date" datetime="{{ post.date | htmlDateString }}">{{ post.date | readableDate }}</time>
      </li>
    {% endfor %}
  </ul>
  {% endif %}
{% endfor %}

Helped by AI 🔗

And here's my dirty, little secret 🤫: I only actually coded year-index.njk myself, and then asked Claude to code the rest for me.

Initial prompt 🔗

I have a blog built with Eleventy. It uses a URL structure
that is https://blog.tomayac.com/$year/$month/$day/$title/.
For example, https://blog.tomayac.com/2024/08/26/my-response-to-the-cma/.

I already have a way to list all posts published in a year by
navigating to https://blog.tomayac.com/$year/. Now I want two
levels deeper and get first https://blog.tomayac.com/$year/$month/,
that is, all posts published in a given month, and
https://blog.tomayac.com/$year/$month/$day/, that is, all posts
published on a given year.

For the year index, this is how I got it to work:

In .eleventy.js:
eleventyConfig.addCollection('postsByYear', (collection) => {
    return _.chain(collection.getAllSorted())
      .filter((item) => 'tags' in item.data && item.data.tags.includes('posts'))
      .groupBy((post) => post.date.getFullYear())
      .toPairs()
      .reverse()
      .value();
  });

And then a Nunjucks file year-index.njk:

---
pagination:
  data: collections.postsByYear
  size: 1
  alias: year
layout: layouts/home.njk
permalink: /{{ year[0] }}/
---
<h2>{{ year[0] }} Archive</h2>
{% for postedYear, yearPosts in collections.postsByYear %}
  {% if postedYear === year[0] %}
  <ul class="postlist">
    {% for post in yearPosts | reverse %}
      <li class="postlist-item{% if post.url == url %} postlist-item-active{% endif %}">
        <a href="{{ post.url | url }}" class="postlist-link">{% if post.data.title %}{{ post.data.title }}{% else %}<code>{{ post.url }}</code>{% endif %}</a>
        <time class="postlist-date" datetime="{{ post.date | htmlDateString }}">{{ post.date | readableDate }}</time>
      </li>
    {% endfor %}
  </ul>
  {% endif %}
{% endfor %}

Can you create the rest?

Correcting prompt 🔗

It worked on the second attempt. In the first attempt, it invented a split Nunjucks filter, so I just told it about the error, and after that it just worked.

This fails now:
Error: filter not found: split (via Template render error)

Conclusion 🔗

There may be more elegant ways to achieve this, but this approach is what worked for me, and, hey, it all happens on the server at build time, so you, dear reader, get just the optimized HTML. Happy URL hacking! Oh, and whatever happened on March 3, 2009?

This post appeared first on https://blog.tomayac.com/2024/11/02/eleventy-11ty-year-year-month-and-year-month-day-indexes/.

Background 🔗

The Web Conference (formerly known as WWW) is an international conference focused on exploring the current state and the evolution of the Web through the lens of different scientific disciplines, including computing science, social science, economics, and political sciences. It's organized by the Association for Computing Machinery (ACM) Special Interest Group on the Web (SIGWEB) and is held annually in a different location around the world. The 2024 conference took place in Singapore from May 13 to 17. It's attended by 70% academia and 30% industry. Google was a Gold sponsor, together with TikTok.

Conference 🔗

Day 1 🔗

Online trust day 🔗

Keynote I: Factuality Challenges in the Era of Large Language Models

Speaker: Dr. Preslav Nakov, Professor and Department Chair of NLP, MBZUAI

Fact-checking the output of LLMs: Decompose the output of an LLM into its individual claims, decide which are check-worthy, check one-by-one:

• Factcheck-Bench: Fine-Grained Evaluation Benchmark for Automatic Fact-checkers
• yuxiaw/Factcheck-GPT

Detecting LLM-generated texts:

• Release Strategies and the Social Impacts of Language Models

• RoBERTa (for GTP-2)

Arabic LLM:

• Jais Arabic LLM, see the prompt instructions.

• Demo

Audience question: Why don't we use LLMs for what they are good for: working with language like reformulating or summarizing, but not asking them to come up with facts. — We probably could, but hallucination problems there as well.

Keynote II: Building Trust and Safety on Facebook

Speaker: Lluís Garcia Pueyo, Director of Engineering, Meta

For many languages there isn't enough actually harmful labeled content, so models are trained on artificially oversampled labeled examples.

• Semi Supervised Monotonic Regression For Calibrating Social Media Classifiers

Facebook posts ranking formula: probability you like something, probability you share something, probability you hide something. Like, comment, and send are not good signals for bad experiences. Hiding, reporting from the three dot overflow menu are.

• Detecting and Limiting Negative User Experiences in Social Media Platforms

Future challenges with LLMs:

• Actions Speak Louder than Words: Trillion-Parameter Sequential Transducers for Generative Recommendations

Meta folks are organizing the Integrity Workshop series.

The Dynamics of (Not) Unfollowing Misinformation Spreaders

Collected health misinformation URLs and tweets tagged by PolitiFact. Found users who share this content on Twitter. Denoted these users misinformation spreaders. Also pulled the followers of spreaders. They found that misinformation ties are rarely severed, with unfollowing rates of 0.52% per month. Users are 31% more likely to unfollow non-misinformation spreaders than they are to unfollow misinformation spreaders. Reciprocity, initial exposure, and ideology are the most important factors for predicting unfollowing.

• The Dynamics of (Not) Unfollowing Misinformation Spreaders

Web4All (Sponsored by Google) 🔗

Touchpad Mapper: Exploring Non-Visual Touchpad Interactions for Screen-Reader Users

Touchpad Mapper: maps the position of objects in images to the touchpad area, so when the screen reader user moves their finger over the touchpad, the position of the finger is taken into account for announcing the image contents.

• Touchpad Mapper, requires a backend app to extract the exact physical coordinates of the finger on the touchpad.

Beyond Facts: 4th International Workshop on Computational Methods for Online Discourse Analysis 🔗

Leveraging Large Language Models to Detect Influence Campaigns on Social Media

They used an LLM to determine if, based on user metadata and network structures, a user is part of an organized information campaign. Their model was trained with Russian troll tweets. These Moderation Research datasets are available freely from the X Transparency Center.

Escaping the Echo Chamber: The Quest for Normative News Recommender Systems by Abraham Bernstein

Looked at news recommendations. Interesting datasets: MovieLens, Book-Crossing.

Towards Fact-check Summarization Leveraging on Argumentation Elements tied to Entity Graphs

Used PolitiFact as ground truth and compared GTP4 vs. Custom GPT to see if the models could come up with similar results.

Detection Distortions in Science Reporting by Isabelle Augenstein

Looked at how journalists cover scientific research. Scientific findings frequently undergo subtle distortions when reported, e.g., with regard to certainty, generality, and causality.

• Sentence BERT: framework to compute sentence / text embeddings for more than 100 languages. These embeddings can then be compared, e.g. with cosine-similarity, to find sentences with a similar meaning.

Day 2 🔗

Web4All (https://www.w4a.info/2024/) 🔗

Keynote Speech: Liddy Nevile "Accessibility?"

(Liddy Nevile is the mother of one of the organizers, Charles McCathieNevile, aka. Chaals.) Her son went to university at 10 to learn Logo, so she would learn it, too. Got to know folks at MIT. One of them was Tim Berners-Lee. Worked with Mosaic folks and how blind kids would use it. Concerned about inappropriate content. Founded Platform for Internet Content Selection (PICS) W3C group, which created a numbering system to classify content. Opened the way for what people at the time thought of as curation of content. Was well received by the adult industry. Eric Miller wondered why, if PICS worked, couldn't embedded descriptions, ideally structured, also work? Created "metadata" catalog, which ended up becoming Dublin Core. Published An Introduction to the Resource Description Framework.

Platform for Internet Content Selection (PICS, source) W3C used a grant to set up the Web Accessibility Initiative (WAI). Worked on how to encode resources so they would be accessible to users. How could a blind person find out where the cursor is? How could flickering content be avoided? WAI brought people together to learn about making accessible websites. Some countries converted WAI into national laws. Could compliance be put into the resource, just like PICS? Different countries do things differently. Australia doesn't prosecute people for not complying with Web Content Accessibility Guidelines (WCAG). Worked on making math books accessible with MathML. Inclusion (make something accessible just in case) or accessibility (make something accessible just in time). Worked on structured accessibility data. Jutta Treviranus worked on Access4Al: "Whether using a public workstation, or engaging in an on-line learning environment, computer systems should fit the individual preferences and requirements of the user, especially if the user requires an alternative access system. An international effort is underway to create a common specification for expressing personal preferences for all systems affecting the user interface and content retrieval". Took accessibility description work further with schema.org in the form of accessibilitySummary. Now there's ISO/IEC 4932 (Core Accessibility Metadata). Looking forward to seeing accessibility services working with AI at last doing what we dreamed of so many years ago. If accessibility is an opt-in, it needs to be made sure that the data isn't abused. Good privacy fields help, it's about the people, everyone occasionally needs accessibility features. (The speaker said they were very thankful to Google for the schema.org work.)

Decoding the Privacy Policies of Assistive Technologies

They looked at the privacy policies of various assistive technologies companies. It's not great; some of them collect data about sexual orientation.

QualState: Finding Website States for Accessibility Evaluation

Web accessibility evaluation engine called QualState for automatically testing the accessibility of web apps. QualState loads a page, performs actions. Identifies events on page. Clicks links, buttons, and submits forms based on the DOM tree. Ignores some nodes, but needs a full DOM tree to see which states the page was already in.

A Universal Web Accessibility Feedback Form: A Participatory Design Study

Hypothesis: companies don't get accessibility complaints because the feedback forms are inaccessible. Placement of the feedback link reduces or creates entry barriers. Add an introduction paragraph. Describe each step clearly. Make sure constraints (multiple choice checkboxes, radio buttons, etc.) are clearly explained, and don't rely purely on technical error message handling. Allow contact information to be added optionally. The form should provide details on where exactly on the website a problem occurred. Allow for system settings to be shared and the used assistive technology. Make their tool available on GitHub: human-centered-systems-lab/a11y-feedback.

Accessibility and AI

Can AI coding assistants produce accessible UI code? Yes, when explicitly instructed to do so. They are not reliable and subject to hallucinations. Dark mode button worked fine. Image had mixed Japanese/English alt text. Can AI coding assistants eliminate the need for developer accessibility awareness? Accessibility features are not applied consistently. There might be states that get missed. Empty alt texts are hard to catch, since it looks intended. More benchmarks are needed. Fine-tuning models for accessible UI. AI powered DevTools can help, too. Copilot doesn't make any claims about the accessibility of its created code.

Evaluating the Effectiveness of STEM Images Captioning

Teaches university students Web development. Asks them to take the #NoMouse Challenge. Split their group in two groups: one was trained on image accessibility, one wasn't. Correctness (does the description accurately describe what the image depicts) vs. usefulness (does the description accurately describe the conveyed meaning) of describing images. Created AI image descriptions with IDEFICS (demo). Students were asked to evaluate human-generated vs. AI-generated descriptions. Describing STEM images (like diagrams describing photosynthesis) generally is hard. In all cases, humans performed better. The AI had more problems with STEM images. Planning to compare other AI engines. Also thinking about ways to improve the prompt engineering (e.g., "Describe this detailed scientific diagram in a way that the description is useful to a blind user").

Making Accessible Movies Easily: An Intelligent Tool for Authoring and Integrating Audio Descriptions to Movies

Steps for creating movie audio descriptions (AD): read existing subtitles with OCR, identify speech gaps based on missing subtitles, create scene description texts with VideoChat or VideoLLaMa, then use ChatGPT to merge the subtitles and the scene description. The final step is to run text to audio and audio mixing. Created an app called EasyAD that incorporates all these steps. Quality was evaluated as being good, but speed was still slow, feedback also suggested more languages than Chinese should be supported.

Towards Effective Communication of AI-Based Decisions in Assistive Tools: Conveying Confidence and Doubt to People with Visual Impairments at Accelerated Speech

Screen reader users typically comprehend speech 3 times faster than sighted users. Speeded up voices lose some of the emotions of speech like confidence or doubt. Up to a factor of 1.5 to 2 this effect isn't noticeable, at faster speeds it's noticeable and research is required to reintroduce these emotions.

Welcome reception 🔗

The welcome reception took place in the Tipsy Unicorn.

Day 3 🔗

Keynote#1: Challenges toward AGI and its impact to [sic] the Web 🔗

Speaker: Jie Tang This keynote was about how they created the Chinese ChatGPT called ChatGLM (智谱清言), which means "clear words of wisdom".

Web4Good 🔗

CapAlign: Improving Cross Modal Alignment via Informative Captioning for Harmful Meme Detection

The authors prompt a large language model (ChatGPT) to ask informative questions to a pre-trained vision-language model (BLIP-2) and use the dialogs to generate a high-quality image caption. To align the generated caption with the textual content of a meme, they use an LLM with instructions to generate informative captions of the meme and then prepend it with the attributes of the visual content of a meme to a prompt-based LLM for prediction. (I would love to see this run on top of Memegen.) [Paper]

Panel Discussion on LLM Impact on the Web 🔗

Panelists:

• Panel Chair:
  • Andrew Tomkins, Research Scientist, Google
• Panelists:
  • Jon Kleinberg, Cornell University
  • Yoelle Maarek, Chief Researcher, Technology Innovation Institute
  • Jie Tang, Tsinghua University

Questions:

• Do we expect websites to have LLM-based front-ends?

  • Jie thinks it's quite possible. Each website may have an agent that could also interact with other websites' agents.
  • Yoelle thinks that if the content of websites is generated by LLMs and LLMs train themselves on the content they created, this may lead to a rich get richer symptom and all LLMs learn the same. Maybe more diverse LLMs can help rather than one dominating one. Hallucinations are a big problem, and they will continue to be. People need the feeling that the information comes from somewhere, to have sources. We need to be careful not to take this feeling away.
  • The most popular app on the Web is search. Search puts itself between the page and the searcher. If search doesn't lead to traffic to pages, there's no incentive to create content.

• Websites don't need to expose APIs anymore, agents can just talk to websites using natural language. Will this cause specialized search engines to arise? Do we expect one central agent to rule them all?

  • Yoelle says before common Web search engines, there was a federated search engine, but it died. Strongly believing in RAG, you need special agents to surface hidden content.
  • Jon states it's an old question, special agents like for flight search. You could take special agents and hide them under one common interface. It's mostly a UI question.
  • Jie says we have a network of webpages, and later Linked Data. Now AI to answer questions. We could have linked AI to answer special questions.

• The Web is special. Someone has gone through the work of, for example, collecting great spots to visit at a place. We would love for this person to keep the benefits. We can do so through advertising. Now the model is changing. Why would people keep creating content under these circumstances? What are possible models for this to work in the future?

  • Jon says the Web has always been powered by altruism. Search isn't always about finding the answer, but also about exploring the landscape. People want to hear different takes on a question. LLMs will not just be used to find one answer. If there's economic value created. Mashups is a 2005 concept, we mashed up Google Maps with stuff.
  • Yoelle states most websites are automatically created. It's like AirBnB, it's business, not regular people renting out a spare room. If you have specialized RAG-supported LLMs, you have a transaction when hidden content is being found. This isn't the Open Web, maybe it has already disappeared. Economic value could be protected like this.
  • Jie isn't sure about this. People only want to consume, they don't necessarily search for something concrete, like TikTok or Douyin.
  • Yoelle really wants to disagree. We want serendipity, I love this journey of searching. It's something I must have to earn.
  • Jon talks about the economic model. If you're a standup comedian, you start imitating others, and eventually you develop your own style. That's LLMs. You don't owe the comedians you took inspiration from early on.

• Audience question by Natasha Noy: I want to broaden this. If we have a highly curated extra layer, it hides this personal layer. We need to discuss this layer.

  • Jon says this could be something like an LLM giving out "Likes", or Google Scholar citation counts. Could this be self-prophesying, so people create content only to be cited by LLMs.

• Want to talk about privacy and safety. But before that, I want to talk about crawling. It's a massive business. Crawlers asking for access to content could look different than regular users, they already do. Could there be a third class of LLM traffic to websites?

  • Yoelle sees the point. People don't protect their websites too much, like with robots.txt, because they know they get traffic back. Now with LLMs it's different. They don't get the value back. It's costly for providers to crawl, Azure, Google, AWS, they make a lot of money off LLM crawlers. We need to think about protocols to support this.
  • Jie thinks in the future this depends, if in the future the Web will interact with human beings, if the Web itself is a personal assistant, the Web will change. Not sure how.
  • Jon thinks this question is orthogonal to the question of LLMs. We already have alerts and notifications like has the flight price changed, has an article been published on a topic. This is a pilot case for what LLMs could do in a general manner.

• Audience question: One of the issues now is who owns the content creator? Google etc. make profit from profiling users. Would we all come to a conference to listen to bot-written papers? When we lose trust in the reflective power of an LLM, who's going to pay for this technology? Is it either you, or my personal information? Will this model work in the future?

  • Jie isn't sure about the economic aspect. AI is still occupied with improving its performance. The trust still needs to be earned.
  • Yoelle means people won't use AI for trivial things they can do themselves. Reasoning examples we see today are trivial, because the AI is still learning. In the early days of the Web it was authority through PageRank and clicks that brought you value as a creator. Research may be needed to explore if we can somehow give back to creators, I hope it will come naturally. We can also think as creators, what information do we want to make accessible to LLMs.
  • Jon asks what's the value we're adding. If we're angry at an LLM, what can I add? Wolfram Alpha now just solves math problems which a hundred years ago you could publish a math paper about.

• Audience question: The age of information abundance. In the past, there was information scarcity. For example, you needed to travel to different countries to get information. Now it's the opposite, you have way too much information. Generative AI makes this a lot worse, it creates so much information.

  • Yoelle thinks the question is funny. LLMs are good at finding hidden information. But you don't know if it's a hallucination. You don't have the context, so you can't easily verify. The world now isn't deterministic. We're all computer scientists, we're used to determinism. Now it's not the one truth. You live in a fuzzy world now. Maybe we all become computer artists. Scared by the uncertainty. I want proof, I lost this.
  • Jon disagrees a bit. You visit a doctor, and each doctor you visit tells you something different. Indeterminism existed before. It's a fascinating tension. Huge fan of the abundance question, wrote papers about this. In the early 1970ies, there was a book about information abundance. Abundance is consuming human attention.

• Let's touch on trust and safety. There's privacy questions, there's government standpoint questions, there's questions on where LLMs should be hosted. Finance has concerns about data safety. Can you share thoughts about what's the biggest risk right now?

  • Jie thinks all this is very important. Technology is super important. AI will self-improve and self-reflect. We could have built a common model to check the quality of models. If AIs in the future will be smarter about this, this would be great.
  • Yoelle says it's super important to have many open source LLMs. We need diversity. You could come back to different models, even if they are biased. For many sensitive topics, you can bring models in-house trained on your data, even with lower general quality.
  • Jon means powerful tools will reflect society, including its biases. We rely on LLMs as to make decisions.

History of the Web #1 🔗

Viola, Pei Wei, and the Fights for Interactive Media

Interesting historical reference: MediaView: a general multimedia digital publication system.

Digital Democracy at Crossroads: A Meta-Analysis of Web and AI Influence on Global Elections

They looked at papers from the past until today that looked at elections. Suggestions for generative AI companies to tackle AI disinformation: Implement watermarking and strict verification, regulate AI chatbots, mark AI-generated content as such. Government should require politics-related material to be marked specially if AI was involved. Educate users to identify AI-generated content. Fake news isn't new, but the scale is way different now. [Paper]

History in Making: Political Campaigns in the Era of Artificial Intelligence-Generated Content

Historically, we had user-generated content. Now we have AI-generated content. It's the year of the elections, in almost 60 countries, covering half of the Earth's population. Political campaigns make use of AI. Huma/real life person impersonating makes caller bots possible. Ashley caller bot in the US. AI-generated deepfake makes campaigning from prison possible. Also malicious deepfakes. ChatGPT is known to be left-leaning in the US. Governments crack down on services and tools to create and spread AI-generated content and limit access to user data. [Paper]

Me, the Web and Digital Accessibility

Fun anecdote: IE showed the alt attribute like a tooltip. The author is the official translator of the WCAG standard for Portuguese and got into accessibility when he was made aware that Brazil's government required websites to be accessible. [Paper]

From Files to Streams: Revisiting Web History and Exploring Potentials for Future Prospects

Users love fast web content and there's an economic value in performance. 1991: all content text-based and delivered from one server. Now all types of content are delivered via CDN. Cites HTTP Archive stats on website size and First Contentful Paint. FCP didn't improve. HTTP was FTP inspired. TCP handshake cost needed to be paid. Less files meant faster loading time. Keep-alive allows reusing TCP connections. HTTP/2 and HTTP/3 reduced the overhead, no more line-blocking, multiplexing and streams, 0-RTT. Server delivery was improved. Client side lacked. JS (1995), CSS (1996), DOM (1998). Bundling as a solution to make less requests. Browserify (2013), then Webpack. Webpack's popularity peaked just when HTTP/2 was introduced. A solution (bundling) for a problem that doesn't exist anymore. Erwin Hofman: "Bundling is an antipattern in HTTP/2". Render-blocking as a major annoyance, can use dead code elimination and critical CSS identification. Can stream content over HTTP/2 or /3. Sees research challenge in automatic content usage detection and ordering of JavaScript. They stream Web content via WebSocket (demo). [Paper]

Posters 🔗

Automating Website Registration for Studying GDPR Compliance

They used a headless browser to sign up to websites and then see if they had GDPR violations. [Paper]

Breaking the Trilemma of Privacy, Utility, Efficiency via Controllable Machine Unlearning

TIL about the concept of Machine Unlearning. The work explains how parts of a model's training data can be removed without having to retrain the entire model. [Paper]

A Worldwide View on the Reachability of Encrypted DNS Services

They compare different privacy-preserving ways of how DNS can work over encrypted data and how they affect global reachability. [Paper]

Uncovering the Hidden Data Costs of Mobile YouTube Video Ads

They look at wasted bandwidth from YouTube ads. Apparently we preload quite eagerly, even if most ads are skipped as soon as possible. [Paper]

Day 4 🔗

Keynote#2: Revisiting the Behavioral Foundations of User Modeling Algorithms 🔗

Speaker: Jon Kleinberg Algorithms as partners, GPT should stand for General Purpose Technology. In contrast to algorithms as creators of environments, like when they curate social media experiences. When consuming a linear feed, after each item the user has the chance to quit or continue scrolling. cThe algorithms are tuned to bring you chips because you know you crave them, while at the same time you also know that you should be having a salad. Sees AI as a semi-autonomous vehicle.

Systems #2 🔗

A Multifaceted Look at Starlink Performance

M-Lab Tests makes open-source data about Internet speed tests available. Measurement Lab is led by teams based at Code for Science & Society; Google, Inc; and supported by partners around the world. Internally, the Google team responsible is called Open Internet Measurement. Median latencies for Starlink is ~40–50ms, while mobile networks are ~30ms. NA and EU (regions with dense ground stations concentration) enjoy the best coverage. Very few locations where Starlink outperforms cellular. Closeness to the ground station determines latency a lot. [Paper]

PASS: Predictive Auto-Scaling System for Large-scale Enterprise Web Applications

Paper that looks at the Web app scaling of Meituan, a Chinese shopping platform for locally found consumer products and retail services including entertainment, dining, delivery, travel, and other services. Uses app's recent performance data to predict upcoming load. Offline model (looks at historical data) doesn't have information about spontaneous spikes, online model (looks at just passed data) has a slight lag. Uses hybrid auto-scaling by combining predictive scaling with reactive scaling. [Paper]

FusionRender: Harnessing WebGPU's Power for Enhanced Graphics Performance on Web Browsers

Smaller language translation overhead. Omits runtime error checks. Recucs data communication between GPU and CPU by using pre-packed configuration. They compared WebGL vs. WebGPU. Three.js, Babylon.js, PlayCanvas, and Orillusion. WebGPU is slower(!) on all frameworks. Frameworks render objects separately , leading to redundant transmission. Merged rendering leads to reduced transmission. How can it be determined which objects can be merged? Graphics rendering configurations. They introduce FusionRender. Input: user-defined configurations, output: WebGPU. Joins objects with identical signatures. Uses a hash map, objects are grouped based on their hash. Implemented a prototype for Three.js, tested on MacBook Pro, ThinkPad X1, and Pixel 6 with Chrome and Firefox. FusionRender shows improvements between ~29% and ~120% with synthetic data, about ~30% with real world data. (Code: qqyzk/FusionRender) [Paper]

QUIC is not Quick Enough over Fast Internet

The paper examines QUIC's performance over high-speed networks. They find that over fast Internet, the UDP+QUIC+HTTP/3 stack suffers a data rate reduction of up to 45.2% compared to the TCP+TLS+HTTP/2 counterpart. This performance gap between QUIC and HTTP/2 grows as the underlying bandwidth increases. The root cause is high receiver-side processing overhead, in particular, excessive data packets and QUIC's user-space ACKs. QUIC perceives much more packets than HTTP/2. In Chromium, much more netif_receive_skb calls are invoked for QUIC. The issue is observed on CLI data transfer clients and browsers (Chrome, Edge, Firefox, Opera), on different hosts (desktop, mobile), and over diverse networks (wired broadband, cellular). Ruled out server software, UDP/TCP protocols, HTTP syntax, TLS encryption, client OS, etc. as reasons. [Paper]

History of the Web #2 🔗

Toward Making Opaque Web Content More Accessible: Accessibility From Adobe Flash to Canvas-Rendered Apps

[Slides] [Paper]

Revisiting 30 years of the Network Time Protocol

Network Time Protocol (NTP) has a hierarchical structure that delivers the time, the stratum 0 server has the most accurate time and passes it on to lower levels. NASA has proposed the Interplanetary Internet. Korea Pathfinder Lunar Orbiter played K-pop from the Web. SpaceX and Blue Origin look at Mars Internet. The Proximity-1 Interleaved Time Synchronization (PITS) protocol looks at how time synchronization could work in space. [Paper]

History of the Semantic Web

A walk down memory lane of the Semantic Web with Jim Hendler. Started with the Scientific American article in roughly 2000. In 2005, started moving from reasoning to linking data. 2010 was the year of Web 3.0, the dawn of semantic search. 2014 Google Sem Webbers: R.V. Guha, Dan Brickley, Denny Vrandecic, Natasha Noy, Chris Welty. Guha in 2014: > 20% of pages included structured data. In 2016 Peter Norvig mentioned >60%. Facebook created Open Graph in 2011. IBM Watson in 2017. Facebook's Graph API made the Knowledge Graph concept more well-known. 44% of pages now use schema.org markup. Wikidata as a free editable knowledge base. The semantic web sort of won, but where are the intelligent agents? AI is getting there, but they are not directly using semantic web technologies.

Verso: A web browser that plays old world blues to build new world hope 🔗

A browser called Verso by Daniel Thompson-Yvetot, the creator of the Tauri apps, a framework that uses the system's WebView to ship desktop apps. Tauri uses WebView2 on Windows, WKWewbView on macOS, and webkitgtk on Linux. WebView2 is based on Chromium, which has a good update frequency. WKWebView means some people are stuck on old macOS. WebView W3C effort is slow and won't change things meaningfully. Thought about using the Servo engine. Collaborated with Igalia. Was at Mozilla, now hosted by the Linux Foundation. Tauri folks maintain HTML5ever, used by Servo, Tauri, and Vercel. Engine is based on Servo, and a CLI for headless integration. Deep local language integration for local translation and reader mode transformation. Default incognito profile management mode. Provides a WebView, too. Shards identities, storage, sign-in. Next steps: close early funding round, set up non-profit organization at Commons Conservancy, don't sell search, convince Next Generation Internet (NGI) EU framework to accept the project. They want to launch in summer, coming to this conference was the first step. (Nightly builds)

Day 5 🔗

Keynote#4: AI deepfakes on the Web: the 'wicked' challenges for AI ethics, law and technology 🔗

Speaker: Jeannie Marie Paterson The word deepfake is a combination of deep learning and fake. Can be used for fun or in movies (e.g., Princess Leia in Star Wars). Can be used for malicious purposes, like Elon Musk deepfake-generated get-rich-quick scams or scammers to fake family members' voices. Romance fakes with face swapping. Deepfaked synthetic porn affecting many K-pop stars. Political deepfakes can affect elections. The liars dividend: people start questioning actual images and calling them deepfake. Responses to deepfakes:

• Law: Can get active via transparency requirements, consumer regulators, criminal offenses, online safety. OECD principles of ethical AI.

• Education: Look for errors in images and videos, but early signals like blinking patterns have been improved by technology.

• Tech: Enforce voluntary guardrails to not create terrorist material, pornography,… Release detection tools, but they often only work on the companies' own AI products. Industry has introduced an official Content Credentials Icon (C2PA).

  Participants in the initiative:

  

  Digital watermarks, but it's a cat and mouse game. Watermarks might not survive screenshots or photos of AI-generated photos.

Resource 🔗

Ducho 2.0: Towards a More Up-to-Date Unified Framework for the Extraction of Multimodal Features in Recommendation

Multimodal recommender system that integrates with deep learning frameworks like TensorFlow, HuggingFace, or PyTorch. (Demo) [Paper]

The Web Data Commons Schema.org Table Corpora

There are many table corpora, typically used to evaluate ML systems. The corpora use different schemata and formats. Introducing schema.org table corpora to bridge the gap. Use the Common Crawl corpus. Group by host (for example imdb.com) and class. Remove sparse entities and poor annotations. The resulting tables can contain nested entities, for example, the actor type. Needs flattening. Tables and meta statistics files are available for download. Used for table annotation benchmarks, for Q&A datasets, entity matching (based on unique identifiers, like telephone numbers), or as a source for training data. [Paper]

Tel2Veh: Fusion of Telecom Data and Vehicle Flow to Predict Camera-Free Traffic via a Spatio-Temporal Framework

Objective is to apply telecom data to improve traffic flow. Crossing telecom data with vision-based camera data. Make a dataset available with the crossed data. Based on this data, train a predictor using the vision-based data as the groundtruth. System can be used for traffic flow monitoring and traffic optimization. [Paper]

An Open Platform for Quality Measures in a Linked Data Index

Finding a good dataset is a challenge. How can the quality be measured? Need defined quality measures, Propose IndeGx, builds an index of public SPARQL endpoints. Used to compare FAIRness (Findable, Accessible, Interoperable, Reusable) and accountability (traceability, transparency, trust). For dataset creators, quality matters. [Paper]

CompMix: A Benchmark for Heterogeneous Question Answering

Heterogenous question answering systems where the answers come from different sources, like text or tables. CompMix is a dataset of questions and answers. Covers comparatives, superlatives, ad-hoc, count, ordinal questions, etc. Used generative LLMs and other methods to see if they could answer the questions. None of the systems were able to answer 50% or more of the questions, which means the questions dataset is really challenging. Ideally a system should ground the data in a source and make the answer traceable. [Paper]

SE-PQA: Personalized Community Question Answering

Working on personalization in information retrieval. Used StackExchange community questions and answers, tags, and user profile metadata. Trained different models on the dataset. Personalization based on tags improved the quality. [Paper]

Can LLM Substitute Human Labeling? A Case Study of Fine-grained Chinese Address Entity Recognition Dataset for UAV Delivery

Drone delivery systems in China use named entity recognition to convert raw addresses into precise locations using large language models. Needs specifically trained Chinese dataset. Released CNER-UAV dataset. Contains lots of "in the building" or "in the unit complex" or even "in the room" address refinements. Tested with different models and evaluated precision and recall. ChatGLM (the Chinese ChatGPT) performs poorly, GPT compares fine, but struggles with room and other address annotations. [Paper]

Graphameleon: Relational Learning and Anomaly Detection on Web Navigation Traces Captured as Knowledge Graphs

Graphameleon is a browser extension to capture web navigation. Motivations like tracking one's carbon footprint of a browsing session. Has a macro mode that captures the request/response traffic and micro mode that also captures mouse clicks. Use the UCO ontology. Creates a knowledge graph and a 3D graph visualization. Tracks website complexity with and without Firefox strict or standard tracking prevention on. Another use case is to detect attacks like XSS. [Paper]

This post appeared first on https://blog.tomayac.com/2024/05/22/the-web-conf-2024-singapore-trip-report/.

Disclaimer, just in case… 🔗

I work for Google on the Chrome Developer Relations team. But for this post, I want to make it super duper clear that I'm speaking not on behalf of my employer and that all views and opinions expressed in this blog post are purely my own: Thomas Steiner's, the guy commonly known for his avatar with a green hat, but today in my pajamas having my second morning coffee. Oh, thanks for asking, the two cats are Lluna (yes, with double 'l', it means moon in Catalan) Norris, looking at the camera, and Skinny Norris, looking out of the window.

How I noticed 🔗

With this out of the way, it's time to dive in and answer the question of what exactly did Apple break in the EU? I'm physically located in the European Union and my iPhone has a German SIM card. On January 30, 2024, I sent the following toot with attached screenshot (cropped here):

Hope this is a bug in the beta, but opening previously installed Home Screen Web apps on iOS 17.4 (21E5184i) results in a prompt:

Open "Example app" in Safari. "Example app" will open in your default browser from now on.

Newly installed apps always open in the browser. There doesn't appear to be a standalone mode anymore.

Reported as FB13567834.

The toot that all the news outlets cited was the one from Mysk from February 1, 2024, that said:

🎬 Finally, iOS treats all browsers equally when it comes to PWAs. Previously, only Safari was able to install and run PWA apps. With iOS 17.4 beta in the EU, no browser can install PWA apps, even Safari. It seems PWAs have been disabled entirely.

Oh yes, when you set a third-party browser as the default browser and then you delete it, iOS sets Safari as the default browser. Watch this:

#iOS #Apple #DMA #EU #maliciouscompliance

youtu.be/AST12aDGf0Q

Then, on February 2, 2024, Tixie opened a WebKit bug titled "Bug 268643 - [iOS 17.4 Beta (21E5184k)] REGRESSION: PWA added to Home Screen are forced to open in Safari."

🆕 Update: The message in the release candidate of iOS 17.4 (21E217) is now: "Open 'Example app' in 'Default browser'? In your region, web apps now open in your default browser".

What does Apple say? 🔗

By now, you have probably heard that users in the EU don't have access to Home Screen web apps anymore. Here is Apple's statement in its full glorious detail:

To comply with the Digital Markets Act, Apple has done an enormous amount of engineering work to add new functionality and capabilities for developers and users in the European Union — including more than 600 new APIs and a wide range of developer tools.

The iOS system has traditionally provided support for Home Screen web apps by building directly on WebKit and its security architecture. That integration means Home Screen web apps are managed to align with the security and privacy model for native apps on iOS, including isolation of storage and enforcement of system prompts to access privacy impacting capabilities on a per-site basis.

Without this type of isolation and enforcement, malicious web apps could read data from other web apps and recapture their permissions to gain access to a user's camera, microphone or location without a user's consent. Browsers also could install web apps on the system without a user's awareness and consent. Addressing the complex security and privacy concerns associated with web apps using alternative browser engines would require building an entirely new integration architecture that does not currently exist in iOS and was not practical to undertake given the other demands of the DMA and the very low user adoption of Home Screen web apps. And so, to comply with the DMA's requirements, we had to remove the Home Screen web apps feature in the EU.

EU users will be able to continue accessing websites directly from their Home Screen through a bookmark with minimal impact to their functionality. We expect this change to affect a small number of users. Still, we regret any impact this change — that was made as part of the work to comply with the DMA — may have on developers of Home Screen web apps and our users.

These changes are iOS only! 🔗

The first important thing to note here is that this statement exclusively applies to iOS, but not iPadOS and not macOS. This works because Apple managed to convince the EU Commission that iPadOS and macOS are distinct core platform services. The relevant section of the DMA says:

Moreover, the Commission considers, in line with Apple's view, that iOS, iPadOS, macOS, watchOS, and tvOS constitute distinct CPSs [core platform > services] within the meaning of Article 2, point (2), sub (f), of Regulation (EU) 2022/1925.

This means on iPadOS and macOS, everything will stay the same. You can still add Web apps to the Home Screen on iPadOS or the Dock on macOS, and they will open in standalone mode as they always did.

💡 Note: This article exclusively talks about Home Screen Web Apps, not bookmarks. According to Apple's documentation, "Web developers have the option to create a manifest file (with its display member set to standalone or fullscreen) and serve it along with their website. If they do, that site becomes a Home Screen web app. Then, when you tap on its icon, the web app opens like any other app on iOS or iPadOS instead of opening in a browser. You can see its app preview in the App Switcher, separate from Safari or any other browser."

What happens on iOS? 🔗

Looking now at iOS. If…

1. you have an iPhone that runs (betas of) iOS 17.4 or later, and iff (if and only if)…
2. you are detected as being in the European Union (EU), you can still add apps to the Home Screen, but they will open in a regular new browser tab in your default browser.

How exactly Apple detects if you're in the EU isn't clear yet. It seems not to be based on the SIM operator, as some users claim they are affected even on SIM-less iPhones. Possibly IP geolocation as it doesn't require location access? Or maybe GPS for improved accuracy based on a system-level access grant? What about travelers in the EU from non-EU countries? I hope we will find out eventually. People started noticing an IDENTIFIABLE_REGION string in iOS 17.4 beta 1 (21E5184i) as early as January 25, 2024, but it was removed in the next beta.

💡 Note: Since iOS 16.4, apart from Safari, alternative browsers, too, have the ability to add apps to the Home Screen. Based on Apple's instructions, "if your app has the com.apple.developer.web-browser entitlement, the iOS share sheet can offer Add to Home Screen for an http or https webpage, creating a convenient link to a web app or bookmark. To allow someone to add the current webpage to the Home Screen, include the WKWebView instance in the activityItems array when you call init(activityItems:applicationActivities:) to create the UIActivityViewController."

There are different scenarios listed in the following.

You previously added an app to the Home Screen with Safari 🔗

In this case, the app will open in a regular new browser tab in your current default browser. It doesn't matter if Safari was your default browser when you added the app to the Home Screen, it will still open in your current default browser.

You previously added an app to the Home Screen with an alternative browser that has the com.apple.developer.web-browser entitlement 🔗

In this case, the app will open in a regular new browser tab in your current default browser. It doesn't matter if the alternative browser was your default browser when you added the app to the Home Screen, it will still open in your current default browser.

You newly add an app to the Home Screen with Safari 🔗

In this case, the app will open in a regular new browser tab in your current default browser. It doesn't matter if Safari was your default browser when you added the app to the Home Screen, it will still open in your current default browser.

You newly add an app to the Home Screen with an alternative browser that has the com.apple.developer.web-browser entitlement 🔗

In this case, the app will open in a regular new browser tab in your current default browser. It doesn't matter if the alternative browser was your default browser when you added the app to the Home Screen, it will still open in your current default browser.

What breaks? 🔗

As you see, all the cases mentioned above lead to the same result, a new tab in your current default browser. While simple to understand, there are a number of things that now break.

Push API 🔗

The Push API was described in the article Web Push for Web Apps on iOS and iPadOS published on February 16, 2023. It's important to note the caveat: "A web app that has been added to the Home Screen can request permission to receive push notifications as long as that request is in response to direct user interaction — such as tapping on a 'subscribe' button provided by the web app." Since Home Screen web apps are no longer available in the EU, the Push API is effectively broken for EU users.

Badging API 🔗

The Badging API was described in the article Badging for Home Screen Web Apps published on April 25, 2023. The important caveat here is: "The user must grant the app permission to display notifications before the badge will appear." Since the Push API is no longer exposed, the Badging API breaks as collateral damage.

Standalone mode 🔗

Running in standalone mode allows Web apps to look and feel like native apps without any browser UI. This was particularly useful for game streaming services like NVIDIA GeForce Now or XBox Cloud Gaming, but also just any other app that wants to make best use of the limited screen real estate. Even manually entering fullscreen mode isn't possible anymore, as Safari 17.4 "[f]ixed multiple issues by disabling support for the Fullscreen API on iOS."

Stored data 🔗

Home Screen Web apps ran in a different isolated context than regular in-tab Web apps. This means that if you were logged in to a Web app from the Home Screen, you need to log in again in the browser tab, and all previously stored data is gone. This includes any data stored in:

• IndexedDB
• LocalStorage
• Media keys
• SessionStorage
• Service Worker registrations and cache
• Origin private file system

Exclusion from storage eviction 🔗

Home Screen Web apps were exempt from Safari's 7-Day Cap on All Script-Writeable Storage, but now they aren't anymore. Unless you use a Web app regularly enough, its data will be evicted from storage. This also applies to WKWebView-based browsers that have the com.apple.developer.web-browser entitlement:

Additionally in iOS 14.0 and macOS Big Sur, Intelligent Tracking Prevention (ITP), is enabled by default in all WKWebView applications.

[…]

Note that applications taking the new Default Web Browser entitlement always have a user control in Settings to disable ITP[.]

Multiple installs of the same Web app 🔗

iOS has supported multiple installs of the same Web app since the very beginning. Apple highlighted the ability for people to install any Web app more than once on their device, which can indeed be useful:

When adding a web app to the Home Screen, users are given the opportunity to change the app's name. iOS and iPadOS 16.4 combine this name with the Manifest ID to uniquely identify the web app. That way, a user can install multiple copies of the web app on one device and give them different identities. For example, notifications from "Shiny (personal)" can be silenced by Focus while notifications from "Shiny (work)" can be allowed. If the user gives their favorite website the same name on multiple devices, Focus settings on one device will sync and apply to the others as well.

Technically, this still works and people can add apps more than once, but because the apps now open in the same browser context, the multiple installs people used, for example, to sign in to different accounts, are now effectively useless.

What now? 🔗

The DMA opened the door for browser vendors to ship their own engines on iOS. This would mean that push notifications, app icon badges, storage management, storage eviction, and fullscreen/standalone mode could be decoupled from the previous model of creating a browser shell that until now needed to embed a WKWebView and at best could inject JavaScript to expose APIs that WKWebView didn't support natively to Web apps. The process of Using alternative browser engines in the European Union is going to be maximally painful, as Alex Russell points out and as Mozilla has gone on the record to say.

According to the Financial Times and The Verge, the European Commission is on the case. This is what spokesperson Lea Zuber shared with both publications:

We are indeed looking at the compliance packages of all gatekeepers, including Apple.

In that context, we're in particular looking into the issue of progressive web apps, and can confirm sending the requests for information to Apple and to app developers, who can provide useful information for our assessment.

An open letter to Tim Cook 🔗

The good folks from Open Web Advocacy have written an open letter addressed at Tim Cook in which they outline why Sabotaging Web Apps Is Indefensible. As an immediate action, I would very much encourage you to go 🖋️ sign it. I did. And now back to my third morning coffee and my cats.

This post appeared first on https://blog.tomayac.com/2024/02/28/so-what-exactly-did-apple-break-in-the-eu/.

Disclaimer, just in case… 🔗

I work for Google on the Chrome Developer Relations team. But for this post, I want to make it super duper clear that I'm speaking not on behalf of my employer and that all views and opinions expressed in this blog post are purely my own: Thomas Steiner's, the guy commonly known for his avatar with a green hat, but today in my pajamas having my second morning coffee. Oh, thanks for asking, the two cats are Lluna (yes, with double 'l', it means moon in Catalan) Norris, looking at the camera, and Skinny Norris, looking out of the window.

How I noticed 🔗

With this out of the way, it's time to dive in and answer the question of what exactly did Apple break in the EU? I'm physically located in the European Union and my iPhone has a German SIM card. On January 30, 2024, I sent the following toot with attached screenshot (cropped here):

Hope this is a bug in the beta, but opening previously installed Home Screen Web apps on iOS 17.4 (21E5184i) results in a prompt:

Open "Example app" in Safari. "Example app" will open in your default browser from now on.

Newly installed apps always open in the browser. There doesn't appear to be a standalone mode anymore.

Reported as FB13567834.

The toot that all the news outlets cited was the one from Mysk from February 1, 2024, that said:

🎬 Finally, iOS treats all browsers equally when it comes to PWAs. Previously, only Safari was able to install and run PWA apps. With iOS 17.4 beta in the EU, no browser can install PWA apps, even Safari. It seems PWAs have been disabled entirely.

Oh yes, when you set a third-party browser as the default browser and then you delete it, iOS sets Safari as the default browser. Watch this:

#iOS #Apple #DMA #EU #maliciouscompliance

youtu.be/AST12aDGf0Q

Then, on February 2, 2024, Tixie opened a WebKit bug titled "Bug 268643 - [iOS 17.4 Beta (21E5184k)] REGRESSION: PWA added to Home Screen are forced to open in Safari."

🆕 Update: The message in the release candidate of iOS 17.4 (21E217) is now: "Open 'Example app' in 'Default browser'? In your region, web apps now open in your default browser".

What does Apple say? 🔗

By now, you have probably heard that users in the EU don't have access to Home Screen web apps anymore. Here is Apple's statement in its full glorious detail:

To comply with the Digital Markets Act, Apple has done an enormous amount of engineering work to add new functionality and capabilities for developers and users in the European Union — including more than 600 new APIs and a wide range of developer tools.

The iOS system has traditionally provided support for Home Screen web apps by building directly on WebKit and its security architecture. That integration means Home Screen web apps are managed to align with the security and privacy model for native apps on iOS, including isolation of storage and enforcement of system prompts to access privacy impacting capabilities on a per-site basis.

Without this type of isolation and enforcement, malicious web apps could read data from other web apps and recapture their permissions to gain access to a user's camera, microphone or location without a user's consent. Browsers also could install web apps on the system without a user's awareness and consent. Addressing the complex security and privacy concerns associated with web apps using alternative browser engines would require building an entirely new integration architecture that does not currently exist in iOS and was not practical to undertake given the other demands of the DMA and the very low user adoption of Home Screen web apps. And so, to comply with the DMA's requirements, we had to remove the Home Screen web apps feature in the EU.

EU users will be able to continue accessing websites directly from their Home Screen through a bookmark with minimal impact to their functionality. We expect this change to affect a small number of users. Still, we regret any impact this change — that was made as part of the work to comply with the DMA — may have on developers of Home Screen web apps and our users.

These changes are iOS only! 🔗

The first important thing to note here is that this statement exclusively applies to iOS, but not iPadOS and not macOS. This works because Apple managed to convince the EU Commission that iPadOS and macOS are distinct core platform services. The relevant section of the DMA says:

Moreover, the Commission considers, in line with Apple's view, that iOS, iPadOS, macOS, watchOS, and tvOS constitute distinct CPSs [core platform > services] within the meaning of Article 2, point (2), sub (f), of Regulation (EU) 2022/1925.

This means on iPadOS and macOS, everything will stay the same. You can still add Web apps to the Home Screen on iPadOS or the Dock on macOS, and they will open in standalone mode as they always did.

💡 Note: This article exclusively talks about Home Screen Web Apps, not bookmarks. According to Apple's documentation, "Web developers have the option to create a manifest file (with its display member set to standalone or fullscreen) and serve it along with their website. If they do, that site becomes a Home Screen web app. Then, when you tap on its icon, the web app opens like any other app on iOS or iPadOS instead of opening in a browser. You can see its app preview in the App Switcher, separate from Safari or any other browser."

What happens on iOS? 🔗

Looking now at iOS. If…

1. you have an iPhone that runs (betas of) iOS 17.4 or later, and iff (if and only if)…
2. you are detected as being in the European Union (EU), you can still add apps to the Home Screen, but they will open in a regular new browser tab in your default browser.

How exactly Apple detects if you're in the EU isn't clear yet. It seems not to be based on the SIM operator, as some users claim they are affected even on SIM-less iPhones. Possibly IP geolocation as it doesn't require location access? Or maybe GPS for improved accuracy based on a system-level access grant? What about travelers in the EU from non-EU countries? I hope we will find out eventually. People started noticing an IDENTIFIABLE_REGION string in iOS 17.4 beta 1 (21E5184i) as early as January 25, 2024, but it was removed in the next beta.

💡 Note: Since iOS 16.4, apart from Safari, alternative browsers, too, have the ability to add apps to the Home Screen. Based on Apple's instructions, "if your app has the com.apple.developer.web-browser entitlement, the iOS share sheet can offer Add to Home Screen for an http or https webpage, creating a convenient link to a web app or bookmark. To allow someone to add the current webpage to the Home Screen, include the WKWebView instance in the activityItems array when you call init(activityItems:applicationActivities:) to create the UIActivityViewController."

There are different scenarios listed in the following.

You previously added an app to the Home Screen with Safari 🔗

In this case, the app will open in a regular new browser tab in your current default browser. It doesn't matter if Safari was your default browser when you added the app to the Home Screen, it will still open in your current default browser.

You previously added an app to the Home Screen with an alternative browser that has the com.apple.developer.web-browser entitlement 🔗

In this case, the app will open in a regular new browser tab in your current default browser. It doesn't matter if the alternative browser was your default browser when you added the app to the Home Screen, it will still open in your current default browser.

You newly add an app to the Home Screen with Safari 🔗

In this case, the app will open in a regular new browser tab in your current default browser. It doesn't matter if Safari was your default browser when you added the app to the Home Screen, it will still open in your current default browser.

You newly add an app to the Home Screen with an alternative browser that has the com.apple.developer.web-browser entitlement 🔗

In this case, the app will open in a regular new browser tab in your current default browser. It doesn't matter if the alternative browser was your default browser when you added the app to the Home Screen, it will still open in your current default browser.

What breaks? 🔗

As you see, all the cases mentioned above lead to the same result, a new tab in your current default browser. While simple to understand, there are a number of things that now break.

Push API 🔗

The Push API was described in the article Web Push for Web Apps on iOS and iPadOS published on February 16, 2023. It's important to note the caveat: "A web app that has been added to the Home Screen can request permission to receive push notifications as long as that request is in response to direct user interaction — such as tapping on a 'subscribe' button provided by the web app." Since Home Screen web apps are no longer available in the EU, the Push API is effectively broken for EU users.

Badging API 🔗

The Badging API was described in the article Badging for Home Screen Web Apps published on April 25, 2023. The important caveat here is: "The user must grant the app permission to display notifications before the badge will appear." Since the Push API is no longer exposed, the Badging API breaks as collateral damage.

Standalone mode 🔗

Running in standalone mode allows Web apps to look and feel like native apps without any browser UI. This was particularly useful for game streaming services like NVIDIA GeForce Now or XBox Cloud Gaming, but also just any other app that wants to make best use of the limited screen real estate. Even manually entering fullscreen mode isn't possible anymore, as Safari 17.4 "[f]ixed multiple issues by disabling support for the Fullscreen API on iOS."

Stored data 🔗

Home Screen Web apps ran in a different isolated context than regular in-tab Web apps. This means that if you were logged in to a Web app from the Home Screen, you need to log in again in the browser tab, and all previously stored data is gone. This includes any data stored in:

• IndexedDB
• LocalStorage
• Media keys
• SessionStorage
• Service Worker registrations and cache
• Origin private file system

Exclusion from storage eviction 🔗

Home Screen Web apps were exempt from Safari's 7-Day Cap on All Script-Writeable Storage, but now they aren't anymore. Unless you use a Web app regularly enough, its data will be evicted from storage. This also applies to WKWebView-based browsers that have the com.apple.developer.web-browser entitlement:

Additionally in iOS 14.0 and macOS Big Sur, Intelligent Tracking Prevention (ITP), is enabled by default in all WKWebView applications.

[…]

Note that applications taking the new Default Web Browser entitlement always have a user control in Settings to disable ITP[.]

Multiple installs of the same Web app 🔗

iOS has supported multiple installs of the same Web app since the very beginning. Apple highlighted the ability for people to install any Web app more than once on their device, which can indeed be useful:

When adding a web app to the Home Screen, users are given the opportunity to change the app's name. iOS and iPadOS 16.4 combine this name with the Manifest ID to uniquely identify the web app. That way, a user can install multiple copies of the web app on one device and give them different identities. For example, notifications from "Shiny (personal)" can be silenced by Focus while notifications from "Shiny (work)" can be allowed. If the user gives their favorite website the same name on multiple devices, Focus settings on one device will sync and apply to the others as well.

Technically, this still works and people can add apps more than once, but because the apps now open in the same browser context, the multiple installs people used, for example, to sign in to different accounts, are now effectively useless.

What now? 🔗

The DMA opened the door for browser vendors to ship their own engines on iOS. This would mean that push notifications, app icon badges, storage management, storage eviction, and fullscreen/standalone mode could be decoupled from the previous model of creating a browser shell that until now needed to embed a WKWebView and at best could inject JavaScript to expose APIs that WKWebView didn't support natively to Web apps. The process of Using alternative browser engines in the European Union is going to be maximally painful, as Alex Russell points out and as Mozilla has gone on the record to say.

According to the Financial Times and The Verge, the European Commission is on the case. This is what spokesperson Lea Zuber shared with both publications:

We are indeed looking at the compliance packages of all gatekeepers, including Apple.

In that context, we're in particular looking into the issue of progressive web apps, and can confirm sending the requests for information to Apple and to app developers, who can provide useful information for our assessment.

An open letter to Tim Cook 🔗

The good folks from Open Web Advocacy have written an open letter addressed at Tim Cook in which they outline why Sabotaging Web Apps Is Indefensible. As an immediate action, I would very much encourage you to go 🖋️ sign it. I did. And now back to my third morning coffee and my cats.

This post appeared first on https://blog.tomayac.com/2024/02/28/so-what-exactly-did-apple-break-in-the-eu/.

Disclaimer, just in case… 🔗

I work for Google on the Chrome Developer Relations team. But for this post, I want to make it super duper clear that I'm speaking not on behalf of my employer and that all views and opinions expressed in this blog post are purely my own: Thomas Steiner's, the guy commonly known for his avatar with a green hat, but today in my pajamas having my second morning coffee. Oh, thanks for asking, the two cats are Lluna (yes, with double 'l', it means moon in Catalan) Norris, looking at the camera, and Skinny Norris, looking out of the window.

How I noticed 🔗

With this out of the way, it's time to dive in and answer the question of what exactly did Apple break in the EU? I'm physically located in the European Union and my iPhone has a German SIM card. On January 30, 2024, I sent the following toot with attached screenshot (cropped here):

Hope this is a bug in the beta, but opening previously installed Home Screen Web apps on iOS 17.4 (21E5184i) results in a prompt:

Open "Example app" in Safari. "Example app" will open in your default browser from now on.

Newly installed apps always open in the browser. There doesn't appear to be a standalone mode anymore.

Reported as FB13567834.

The toot that all the news outlets cited was the one from Mysk from February 1, 2024, that said:

🎬 Finally, iOS treats all browsers equally when it comes to PWAs. Previously, only Safari was able to install and run PWA apps. With iOS 17.4 beta in the EU, no browser can install PWA apps, even Safari. It seems PWAs have been disabled entirely.

Oh yes, when you set a third-party browser as the default browser and then you delete it, iOS sets Safari as the default browser. Watch this:

#iOS #Apple #DMA #EU #maliciouscompliance

youtu.be/AST12aDGf0Q

Then, on February 2, 2024, Tixie opened a WebKit bug titled "Bug 268643 - [iOS 17.4 Beta (21E5184k)] REGRESSION: PWA added to Home Screen are forced to open in Safari."

What does Apple say? 🔗

By now, you have probably heard that users in the EU don't have access to Home Screen web apps anymore. Here is Apple's statement in its full glorious detail:

To comply with the Digital Markets Act, Apple has done an enormous amount of engineering work to add new functionality and capabilities for developers and users in the European Union — including more than 600 new APIs and a wide range of developer tools.

The iOS system has traditionally provided support for Home Screen web apps by building directly on WebKit and its security architecture. That integration means Home Screen web apps are managed to align with the security and privacy model for native apps on iOS, including isolation of storage and enforcement of system prompts to access privacy impacting capabilities on a per-site basis.

Without this type of isolation and enforcement, malicious web apps could read data from other web apps and recapture their permissions to gain access to a user's camera, microphone or location without a user's consent. Browsers also could install web apps on the system without a user's awareness and consent. Addressing the complex security and privacy concerns associated with web apps using alternative browser engines would require building an entirely new integration architecture that does not currently exist in iOS and was not practical to undertake given the other demands of the DMA and the very low user adoption of Home Screen web apps. And so, to comply with the DMA's requirements, we had to remove the Home Screen web apps feature in the EU.

EU users will be able to continue accessing websites directly from their Home Screen through a bookmark with minimal impact to their functionality. We expect this change to affect a small number of users. Still, we regret any impact this change — that was made as part of the work to comply with the DMA — may have on developers of Home Screen web apps and our users.

These changes are iOS only! 🔗

The first important thing to note here is that this statement exclusively applies to iOS, but not iPadOS and not macOS. This works because Apple managed to convince the EU Commission that iPadOS and macOS are distinct core platform services. The relevant section of the DMA says:

Moreover, the Commission considers, in line with Apple's view, that iOS, iPadOS, macOS, watchOS, and tvOS constitute distinct CPSs [core platform services] within the meaning of Article 2, point (2), sub (f), of Regulation (EU) 2022/1925.

This means on iPadOS and macOS, everything will stay the same. You can still add Web apps to the Home Screen on iPadOS or the Dock on macOS, and they will open in standalone mode as they always did.

💡 Note: This article exclusively talks about Home Screen Web Apps, not bookmarks. According to Apple's documentation "Web developers have the option to create a manifest file (with its display member set to standalone or fullscreen) and serve it along with their website. If they do, that site becomes a Home Screen web app. Then, when you tap on its icon, the web app opens like any other app on iOS or iPadOS instead of opening in a browser. You can see its app preview in the App Switcher, separate from Safari or any other browser."

What happens on iOS? 🔗

Looking now at iOS. If…

1. you have an iPhone that runs (betas of) iOS 17.4 or later, and iff (if and only if)…
2. you are detected as being in the European Union (EU), you can still add apps to the Home Screen, but they will open in a regular new browser tab in your default browser.

How exactly Apple detects if you're in the EU isn't clear yet. It seems not to be based on the SIM operator, as some users claim they are affected even on SIM-less iPhones. Possibly IP geolocation as it doesn't require location access? Or maybe GPS for improved accuracy based on a system-level access grant? What about travelers in the EU from non-EU countries? I hope we will find out eventually. People started noticing an IDENTIFIABLE_REGION string in iOS 17.4 beta 1 (21E5184i) as early as January 25, 2024, but it was removed in the next beta.

💡 Note: Since iOS 16.4, apart from Safari, alternative browsers, too, have the ability to add apps to the Home Screen. Based on Apple's instructions, "if your app has the com.apple.developer.web-browser entitlement, the iOS share sheet can offer Add to Home Screen for an http or https webpage, creating a convenient link to a web app or bookmark. To allow someone to add the current webpage to the Home Screen, include the WKWebView instance in the activityItems array when you call init(activityItems:applicationActivities:) to create the UIActivityViewController."

There are different scenarios listed in the following.

You previously added an app to the Home Screen with Safari 🔗

In this case, the app will open in a regular new browser tab in your current default browser. It doesn't matter if Safari was your default browser when you added the app to the Home Screen, it will still open in your current default browser.

You previously added an app to the Home Screen with an alternative browser that has the com.apple.developer.web-browser entitlement 🔗

In this case, the app will open in a regular new browser tab in your current default browser. It doesn't matter if the alternative browser was your default browser when you added the app to the Home Screen, it will still open in your current default browser.

You newly add an app to the Home Screen with Safari 🔗

In this case, the app will open in a regular new browser tab in your current default browser. It doesn't matter if Safari was your default browser when you added the app to the Home Screen, it will still open in your current default browser.

You newly add an app to the Home Screen with an alternative browser that has the com.apple.developer.web-browser entitlement 🔗

In this case, the app will open in a regular new browser tab in your current default browser. It doesn't matter if the alternative browser was your default browser when you added the app to the Home Screen, it will still open in your current default browser.

What breaks? 🔗

As you see, all the cases mentioned above lead to the same result, a new tab in your current default browser. While simple to understand, there are a number of things that now break.

Push API 🔗

The Push API was described in the article Web Push for Web Apps on iOS and iPadOS published on February 16, 2023. It's important to note the caveat: "A web app that has been added to the Home Screen can request permission to receive push notifications as long as that request is in response to direct user interaction — such as tapping on a 'subscribe' button provided by the web app." Since Home Screen web apps are no longer available in the EU, the Push API is effectively broken for EU users.

Badging API 🔗

The Badging API was described in the article Badging for Home Screen Web Apps published on April 25, 2023. The important caveat here is: "The user must grant the app permission to display notifications before the badge will appear." Since the Push API is no longer exposed, the Badging API breaks as collateral damage.

Standalone mode 🔗

Running in standalone mode allows Web apps to look and feel like native apps without any browser UI. This was particularly useful for game streaming services like NVIDIA GeForce Now or XBox Cloud Gaming, but also just any other app that wants to make best use of the limited screen real estate. Even manually entering fullscreen mode isn't possible anymore, as Safari 17.4 "[f]ixed multiple issues by disabling support for the Fullscreen API on iOS."

Stored data 🔗

Home Screen Web apps ran in a different isolated context than regular in-tab Web apps. This means that if you were logged in to a Web app from the Home Screen, you need to log in again in the browser tab, and all previously stored data is gone. This includes any data stored in:

• IndexedDB
• LocalStorage
• Media keys
• SessionStorage
• Service Worker registrations and cache
• Origin private file system

Exclusion from storage eviction 🔗

Home Screen Web apps were exempt from Safari's 7-Day Cap on All Script-Writeable Storage, but now they aren't anymore. Unless you use a Web app regularly enough, its data will be evicted from storage. This also applies to WKWebView-based browsers that have the com.apple.developer.web-browser entitlement:

Additionally in iOS 14.0 and macOS Big Sur, Intelligent Tracking Prevention (ITP), is enabled by default in all WKWebView applications.

[…]

Note that applications taking the new Default Web Browser entitlement always have a user control in Settings to disable ITP[.]

Multiple installs of the same Web app 🔗

iOS has supported multiple installs of the same Web app since the very beginning. Apple highlighted the ability for people to install any Web app more than once on their device, which can indeed be useful:

When adding a web app to the Home Screen, users are given the opportunity to change the app's name. iOS and iPadOS 16.4 combine this name with the Manifest ID to uniquely identify the web app. That way, a user can install multiple copies of the web app on one device and give them different identities. For example, notifications from "Shiny (personal)" can be silenced by Focus while notifications from "Shiny (work)" can be allowed. If the user gives their favorite website the same name on multiple devices, Focus settings on one device will sync and apply to the others as well.

Technically, this still works and people can add apps more than once, but because the apps now open in the same browser context, the multiple installs people used, for example, to sign in to different accounts, are now effectively useless.

What now? 🔗

The DMA opened the door for browser vendors to ship their own engines on iOS. This would mean that push notifications, app icon badges, storage management, storage eviction, and fullscreen/standalone mode could be decoupled from the previous model of creating a browser shell that until now needed to embed a WKWebView and at best could inject JavaScript to expose APIs that WKWebView didn't support natively to Web apps. The process of Using alternative browser engines in the European Union is going to be maximally painful, as Alex Russell points out and as Mozilla has gone on the record to say.

According to the Financial Times and The Verge, the European Commission is on the case. This is what spokesperson Lea Zuber shared with both publications:

We are indeed looking at the compliance packages of all gatekeepers, including Apple.

In that context, we're in particular looking into the issue of progressive web apps, and can confirm sending the requests for information to Apple and to app developers, who can provide useful information for our assessment.

An open letter to Tim Cook 🔗

The good folks from Open Web Advocacy have written an open letter addressed at Tim Cook in which they outline why Sabotaging Web Apps Is Indefensible. As an immediate action, I would very much encourage you to go 🖋️ sign it. I did. And now back to my third morning coffee and my cats.

This post appeared first on https://blog.tomayac.com/2024/02/so-what-exactly-did-apple-break-in-the-eu.md/.

The Lenovo ThinkVision P27h-20 screen I get from work is a decent 27 inch screen. Coming from the Retina screen of my laptop that I worked on for a long time, I was initially (and still am) not impressed by the resolution of 2560×1440. It took some time to get used to the low resolution on such a big screen, but it gets the job done…

My biggest gripe with the screen was that it just randomly turned off when connected to my MacBook Pro in clamshell mode. I finally found the culprit after combing through the Console system logs for any trace for the longest time. I found out that the MacBook Pro thought the power was changing from grid to battery and vice versa (all while being constantly on-power), and whenever it did that, the screen would turn off.

The solution was to disable the "Smart Power" option in the screen's settings. According to the manual, the "Smart Power" option does the following:

Smart Power intelligently distributes power to connected USB and USB Type-C devices, maximizing power supply efficiency while also reducing overall consumption.

Turns out, it wasn't so smart after all. I saw it range between 65W and 90W, but after turning the option off, the laptop gets a constant 65W, all my USB-C devices still work, and I'm happy to report that the screen no longer randomly turns off. This is the blog post I wish I had found when I was looking for a solution, so I hope it helps someone else.

This post appeared first on https://blog.tomayac.com/2024/02/lenovo-p27h-20-randomly-turns-off/.