{"id":3555,"date":"2025-02-07T18:15:43","date_gmt":"2025-02-07T17:15:43","guid":{"rendered":"https:\/\/blog.tomayac.com\/2025\/02\/07\/playing-with-ai-inference-in-firefox-web-extensions\/"},"modified":"2025-02-07T18:15:43","modified_gmt":"2025-02-07T17:15:43","slug":"playing-with-ai-inference-in-firefox-web-extensions","status":"publish","type":"post","link":"https:\/\/tomayac.com\/wordpress\/2025\/02\/07\/playing-with-ai-inference-in-firefox-web-extensions\/","title":{"rendered":"Playing with AI inference in Firefox Web extensions"},"content":{"rendered":"\n\t\t\t

Recently, in a blog post titled\nRunning inference in web extensions<\/a>,\nMozilla announced a pretty interesting experiment on their blog:<\/p>\n

\n

We've recently shipped a new\ncomponent<\/a>\ninside of Firefox that leverages\nTransformers.js<\/a> [\u2026] and\nthe underlying ONNX runtime engine<\/a>. This component\nlets you run any machine learning model that is compatible with\nTransformers.js in the browser, with no server-side calls beyond the initial\ndownload of the models. This means Firefox can run everything on your device\nand avoid sending your data to third parties.<\/p>\n<\/blockquote>\n

They expose this component to Web extensions under the browser.trial.ml<\/code>\nnamespace. Where it gets really juicy is at the detail how models are stored\n(emphasis mine):<\/p>\n

\n

Model files are stored using IndexedDB and shared across origins<\/strong><\/p>\n<\/blockquote>\n

Typically when you develop an app with Transformers.js, the model needs to be\ncached for each\norigin<\/a> separately,\nso if two apps on different origins end up using the same model, the model needs\nto be downloaded and stored redundantly. (Together with\nChris<\/a> and\nFran\u00e7ois<\/a>, I have\nthought about this problem<\/a>,\ntoo, but that's not the topic of this blog post.)<\/p>\n

To get a feeling for the platform, I extracted their\nexample extension<\/a>\nfrom the Firefox source tree and put it separately\nin a GitHub repository<\/a>, so\nyou can more easily test it on your own.<\/p>\n

    \n
  1. \n

    Make sure that the following flags are toggled to true<\/code> on the special\nabout:config<\/code> page:<\/p>\n

    browser.ml.enable\nextensions.ml.enabled<\/code><\/pre>\n<\/li>\n
  2. \n
    Check out the source code.<\/p>\n
    git<\/span> clone git@github.com:tomayac\/firefox-ml-extension.git<\/code><\/pre>\n<\/li>\n
  3. \n
    Load the extension as a temporary extension on the This Nightly<\/strong> tab of\nthe special about:debugging<\/code> page. It's important to actually use\nFirefox Nightly<\/a>.<\/p>\n
    \"Special<\/p>\n<\/li>\n
  4. \n
    After loading the extension, you're brought to the welcome page, where you\nneed to grant the ML permission. The permission reads "Example extension\nrequests additional permissions. It wants to: Download and run AI models on\nyour device"<\/em>. In the manifest.json<\/code>, it looks like this:<\/p>\n
    {<\/span>\n  \"optional_permissions\"<\/span>:<\/span> [<\/span>\"trialML\"<\/span>]<\/span>\n}<\/span><\/code><\/pre>\n
    \"Permission<\/p>\n<\/li>\n
  5. \n
    If this was the first time, this triggers the download of the model. On the\nJavaScript code side, this is the relevant part:<\/p>\n
    \/\/ Initialize the event listener<\/span>\nbrowser.<\/span>trial.<\/span>ml.<\/span>onProgress.<\/span>addListener<\/span>(<\/span>(<\/span>progressData<\/span>)<\/span> =><\/span> {<\/span>\n  console.<\/span>log<\/span>(<\/span>progressData)<\/span>;<\/span>\n}<\/span>)<\/span>;<\/span>\n\n\/\/ Create the inference engine. This may trigger model downloads.<\/span>\nawait<\/span> browser.<\/span>trial.<\/span>ml.<\/span>createEngine<\/span>(<\/span>{<\/span>\n  modelHub<\/span>:<\/span> 'mozilla'<\/span>,<\/span>\n  taskName<\/span>:<\/span> 'image-to-text'<\/span>,<\/span>\n}<\/span>)<\/span>;<\/span><\/code><\/pre>\n
    You can see the extension display download progress in the lower left corner.<\/p>\n
    \"Model<\/p>\n<\/li>\n
  6. \n
    Once the model download is complete, the inference engine is ready to run.<\/p>\n
    \/\/ Call the engine.<\/span>\nconst<\/span> res =<\/span> await<\/span> browser.<\/span>trial.<\/span>ml.<\/span>runEngine<\/span>(<\/span>{<\/span>\n  args<\/span>:<\/span> [<\/span>imageUrl]<\/span>,<\/span>\n}<\/span>)<\/span>;<\/span>\nconsole.<\/span>log<\/span>(<\/span>res[<\/span>0<\/span>]<\/span>.<\/span>generated_text)<\/span>;<\/span><\/code><\/pre>\n
    It's not the most detailed description, but "A computer desk with a monitor,\nkeyboard, and a plant" definitely isn't wrong.<\/p>\n
    \"Injected<\/p>\n
    If you click Inspect<\/strong> on the extension debugging page, you can play with\nthe\nWebExtensions AI APIs<\/a>\ndirectly.<\/p>\n
    \"Special<\/p>\n<\/li>\n
  7. \n
    The browser.trial.ml<\/code> namespace exposes the following functions:<\/p>\n
  8. \n
    After granting permission, right-click any image on a page, for example,\nUnsplash<\/a>. In the context menu, select \u2728 Generate\nAlt Text<\/strong>.<\/p>\n
    \"Context<\/p>\n<\/li>\n