Microsoft Unveils Copilot Vision: A Revolutionary AI-Powered Web Browsing Assistant

Microsoft Copilot Vision AI Agent : The Future of Internet Browsing

Microsoft has unveiled "Copilot Vision," an innovative AI-powered feature integrated into its Edge browser. Currently available in limited preview for Copilot Pro subscribers in the U.S., this tool is designed to redefine how you interact with the web. By offering contextual assistance, visual analysis, and personalized recommendations, Copilot Vision simplifies a wide range of tasks, from gaming to travel planning. This advanced feature represents a significant step forward in AI tools, promising to enhance productivity and streamline everyday browsing experiences. Whether you're shopping, planning, gaming, or just trying to stay productive, this tool promises to simplify your online experience in ways that feel almost magical. At its core, Copilot Vision isn't just about convenience -- it's about making your time online more meaningful and less overwhelming. Imagine having real-time suggestions tailored to your preferences, insights that save you from endless scrolling, and even visual analysis to help you tackle challenges, whether in a game or a research project. Currently available to select users in the U.S., this feature is already showing how AI can bridge the gap between information overload and actionable clarity. In the following sections, we'll explore how Copilot Vision works and why it might just become your favorite digital companion. At its core, Copilot Vision delivers real-time contextual assistance, allowing you to navigate the web with greater efficiency. Whether you're conducting in-depth research or exploring unfamiliar websites, the AI analyzes the content on your screen and provides actionable insights. For example, if you're reading about a product, it might highlight its key features, suggest related items, or even compare alternatives. This seamless integration eliminates the need to switch between tabs or perform additional searches, allowing you to stay focused on your tasks. By reducing distractions and offering relevant, on-the-spot information, Copilot Vision enhances your ability to make informed decisions quickly. This feature is particularly useful for professionals, students, and anyone who relies on the internet for detailed research or multitasking. One of the standout features of Copilot Vision is its ability to analyze visual content in real time. This capability is especially beneficial for gaming enthusiasts. For instance, in games like GeoGuessr, the AI can interpret visual clues from images and provide hints to improve your gameplay. By combining image recognition with contextual understanding, the tool enhances your gaming experience, helping you solve challenges more effectively. Beyond gaming, this feature extends to practical applications in everyday life. Copilot Vision can identify landmarks, interpret complex diagrams, or analyze images for research purposes. Whether you're studying architectural designs, decoding scientific visuals, or planning a trip, this functionality enables you to make quicker, more informed decisions. Its ability to bridge the gap between visual data and actionable insights makes it a versatile tool for both leisure and professional use. Check out more relevant guides from our extensive collection on AI-powered browser assistant that you might find useful. Planning activities, whether for work or leisure, becomes significantly more efficient with Copilot Vision. The AI tailors its recommendations to your preferences, making sure that the suggestions align with your specific needs. For example: These personalized suggestions save time and reduce the stress of decision-making, allowing you to focus on enjoying the experience. By understanding your preferences and goals, Copilot Vision ensures that your plans are not only efficient but also tailored to your unique requirements. Online shopping can often feel overwhelming due to the sheer volume of options available. Copilot Vision simplifies this process by offering detailed product comparisons. It evaluates features, prices, and user reviews, helping you make well-informed purchasing decisions. For instance, if you're considering a new laptop, the AI can compare specifications, highlight user feedback, and even suggest accessories to complement your purchase. Additionally, Copilot Vision provides advice on product maintenance and long-term suitability, making sure that the items you choose meet your specific needs. This functionality is particularly valuable for high-investment purchases, such as electronics or home appliances, where thorough research is essential. By streamlining the shopping process, Copilot Vision not only saves time but also enhances your confidence in your choices. Travel planning is another area where Copilot Vision excels. The AI offers customized recommendations based on your location, budget, and preferences. For example, it can suggest accommodations with family-friendly amenities, highlight attractions near your destination, or recommend dining options that match your tastes. It also provides insights into: Whether you're organizing a weekend getaway or an international trip, Copilot Vision helps you create a comprehensive itinerary. By addressing every detail, from logistics to leisure, it ensures that your travel plans are both efficient and enjoyable. Copilot Vision is designed to enhance your overall productivity by delivering relevant data and insights in real time. This capability supports a variety of tasks, from brainstorming and research to decision-making and content creation. For instance, if you're drafting a report, the AI can suggest additional resources, highlight key points from your research, or even recommend ways to structure your content. This feature is particularly valuable for professionals and students who need to manage complex projects or meet tight deadlines. By providing timely, actionable information, Copilot Vision not only saves time but also improves the quality of your work. Its ability to adapt to different tasks makes it an indispensable tool for optimizing your workflow. Microsoft's Copilot Vision represents a fantastic advancement in AI-powered browser tools. By integrating contextual assistance, visual analysis, and personalized recommendations into the Edge browser, it redefines how you interact with the internet. Whether you're gaming, shopping, planning, or conducting research, this feature streamlines tasks and enhances your overall experience. As Microsoft's Copilot Vision continues to evolve, it has the potential to become an essential part of your digital toolkit. By addressing a wide range of needs with precision and efficiency, it sets a new standard for AI-driven productivity and convenience, paving the way for a more intuitive and effective browsing experience.

Microsoft's Copilot Vision is a new AI assistant that will change the way we surf the web

Microsoft just announced the launch of Copilot Vision, an innovative feature for its Edge browser. Currently available in limited preview for select Copilot Pro subscribers in the United States, Copilot Vision represents a major leap forward in browser-based AI assistance. This feature allows Microsoft's Copilot to not only browse the web with you but also interpret and interact with the content on your screen. This is sure to enhance the web browsing experience, making it more personal and dynamic. What Is Copilot Vision? Copilot Vision brings context-aware AI capabilities to your web browsing. Unlike previous iterations of AI tools that rely on user queries and preloaded data, this new feature lets Microsoft's AI assistant analyze live web pages in real time. Imagine shopping for the best deals, researching complex topics, or simply catching up on the news, while Copilot Vision offers you relevant insights personalized to the content on your screen. For example, if you're reading a product review, Copilot Vision can compare similar items, highlight key specifications, or even find better prices across the web. If you're navigating an article, the AI can summarize important points or clarify technical jargon that may otherwise be difficult to understand. This seamless interaction turns browsing into a collaborative effort, streamlining workflows and saving time. How does Copilot Vision work? Currently, Copilot Vision is still in preview but will be part of the Copilot Pro suite of tools priced at $20 per month. The subscription offers priority access to premium AI features and the latest models, enhancing creativity and productivity across Microsoft applications. To activate Copilot Vision, users must grant the AI permission to access and analyze the content on their web pages. This feature is embedded directly into the Edge browser and integrates with Microsoft's broader Copilot ecosystem. Once enabled, Copilot Vision can view text, images, and multimedia, offering suggestions, answering questions, and even generating actionable insights based on your browsing activity. Another highlight is the ability to provide contextually aware recommendations. For instance, while browsing a travel website, Copilot Vision can suggest alternative destinations, calculate travel costs, or even compare hotels based on your preferences -- all within the same browsing session. These capabilities make Copilot Vision a versatile tool for both casual users and experts. What about privacy? Microsoft has taken significant steps to ensure that Copilot Vision adheres to stringent privacy and security standards. Because the feature operates on an opt-in basis, users have complete control over when and how Copilot interacts with their browsing data. During the preview phase, Copilot Vision is restricted to a select group of websites, minimizing risks and providing a controlled environment for testing. Additionally, Copilot Vision does not store or reuse data from visited web pages. The AI's interactions are confined to the immediate session and do not contribute to broader model training. The future of browsing is here Microsoft's Copilot Vision marks a turning point in AI-assisted browsing. By blending advanced reasoning models with real-time web interaction, the feature has the potential to transform the way users navigate the internet. Whether you're a casual user looking for smarter recommendations or a professional seeking to streamline your workflow, Copilot Vision promises to make browsing more intuitive, efficient and engaging. For now, Copilot Vision remains in its early stages, so we will have to see how it compares to Claude and other rivals. As the feature rolls out to more users, it could very well become a cornerstone of Microsoft's AI strategy, setting a new standard for how artificial intelligence interacts with the web. As Microsoft continues to refine Copilot Vision, its competitors are also racing to enhance their own AI offerings. Google, for example, is developing similar features for Chrome, aiming to bring its Gemini AI directly into the browser experience. However, Microsoft's early push to implement AI web browsing gives it a distinct advantage for now. More from Tom's Guide

Microsoft Copilot Vision is here, letting AI see what you do online

Join our daily and weekly newsletters for the latest updates and exclusive content on industry-leading AI coverage. Learn More Microsoft Copilot is getting smarter by the day. The Satya Nadella-led company has just announced that its AI assistant now has 'vision' capabilities that enable it to browse the internet with users. While the feature was first announced in October this year, the company is now previewing it with a select set of Pro subscribers. According to Microsoft, these users will be able to trigger Copilot Vision on webpages opened on their Edge browser and interact with it regarding the contents visible on the screen. The feature is still in the early stages of development and pretty restricted, but once fully evolved, it could prove to be a game-changer for Microsoft's enterprise customers -- helping them with analysis and decision-making as they interact with products the company has in its ecosystem (OneDrive, Excel, SharePoint, etc.) In the long run, it will also be interesting to see how Copilot Vision fares against more open and capable agentic offerings, such as those from Anthropic and Emergence AI, that allow developers to integrate agents to see, reason and take actions across applications from different vendors. What to expect with Copilot Vision? When a user opens a website, they may or may not have an intended goal. But, when they do, like researching for an academic paper, the process of executing the desired task revolves around going through the website, reading all its content and then taking a call on it (like whether the website's content should be used as a reference for the paper or not). The same applies to other day-to-day web tasks like shopping. With the new Copilot Vision experience, Microsoft aims to make this entire process simpler. Essentially, the user now has an assistant that sits at the bottom of their browser and can be called upon whenever needed to read the contents of the website, covering all the texts and images, and help with decision-making. It can immediately scan, analyze and provide all the required information, considering the intended goal of the user -- just like a second set of eyes. The capability has far-reaching benefits -- it can accelerate your workflows in not time -- as well as major implications, given the agent is reading and assessing whatever you're browsing. However, Microsoft has assured that all the context and information shared by the users is deleted as soon as the Vision session is closed. It also noted that websites' data is not captured/stored for training the underlying models. "In short, we're prioritizing copyright, creators, and our user's privacy and safety - and are putting them all first," the Copilot team wrote in a blog post announcing the preview of the capability. Expansion based on feedback Currently, a select set of Copilot Pro subscribers in the US, who have signed up for the early-access Copilot Labs program, will be able to use vision capabilities in their Edge browser. The capability will be opt-in, which means they don't have to worry about AI reading their screens all the time. Further, at this stage, it will only work with select websites. Microsoft says it will take feedback from the early users and gradually improve the capability while expanding support to more Pro users and other websites. In the long run, the company may even expand these capabilities to other products in its ecosystem, such as OneDrive and Excel, allowing enterprise users to work and make decisions more easily. However, there's no official confirmation yet. Not to mention, given the cautious approach signaled here, it may take some time to become a reality. Microsoft's move to launch Copilot Vision's preview comes at a time when competitors are pushing the bar in the agentic AI space. Salesforce has already rolled out AgentForce across its Customer 360 offerings to automate workflows across domains like sales, marketing and service. Meanwhile, Anthropic has launched 'Computer Use,' which allows developers to integrate Claude to interact with a computer desktop environment, performing tasks that were previously handled only by human workers, such as opening applications, interacting with interfaces and filling out forms.

Microsoft announces that Copilot Vision is rolling out, bringing AI to Edge browsers

It seems December is the month for exciting AI announcements. Microsoft announced on Thursday that its Copilot Vision tool is now in preview for select ysers. The company declared in a blog post that "Vision is the first AI experience of its kind, available only on Microsoft Edge." The tool is meant to see what you're seeing on your screen, making it easier for the AI to assist you in whatever task you're working on. And if you've tried any Copilot-related service in the past and had a bad experience, don't write Copilot Vision off just yet. Furthermore, the company has acknowledged the criticism from users: "In the last few weeks, we have taken our users' feedback and worked to improve further the speed, simplicity, and personality of Copilot and its Voice to make it a more helpful AI companion." Copilot Vision is described as "having a second set of eyes as you browse," and that sounds like the ideal AI companion to me. This is an opt-in tool, so you don't have to enable it if you don't want to, but the way Microsoft is painting it, it looks like a useful feature in your AI toolbox. Microsoft showed off Copilot Vision in action in a few different ways. In one example, Copilot Vision was used to plan a museum visit. The AI could answer conversational questions, navigate on-screen, and show where to click on the website for more related information. Copilot Voice can also assist with holiday shopping, although Microsoft clarified that it can only interact with "a select set of websites to start." In the video, Copilot Voice heads to Target's website and can find ugly Christmas sweaters and filter options for men specifically, as well as read on-screen details about how to clean the sweater. Another supported website is GeoGuessr.com, which hosts a browser-based game that Copilot Voice can teach you how to play in real time. Copilot Voice could prove to be quite useful in the first scenario, which involves a lot of planning, especially for those who love to research before eating at a new restaurant, moving to a new city, or going on vacation. An AI companion -- perhaps Copilot Voice -- can help someone moving cross country how much they need to budget for moving costs, when they should start packing, and the best cities to stop in along the drive, based on total trip time. One perk to Copilot Voice versus other options would be the ability to speak their questions and requests rather than type them. For those worried about privacy or security, Microsoft makes it clear that it is "prioritizing copyright, creators, and [its] users' privacy and safety." Copilot Vision is entirely opt-in, and your data and the context Vision gathers during a session isn't saved. The company announced in the Thursday blog post that "Only Copilot's responses are logged to improve our safety systems. " The company plans to roll out Copilot Vision slowly. At first, only "a limited number of Copilot Pro subscribers in the United States, through Copilot Labs," will be able to test Vision as a preview, and it'll only work with a select set of websites. Then, the company "will expand access to more Pro subscribers and more websites over time."

Microsoft rolls out Copilot Vision that reads the web with you

Microsoft has released a preview of Copilot Vision, an AI tool capable of reading and interacting with content on websites. Available for U.S. subscribers of Microsoft's $20 monthly Copilot Pro plan, the feature allows users to engage with web pages more dynamically. Copilot Vision can respond to queries about text and images found on the sites visited, summarize information, and provide translations. Users can ask for help with tasks, such as finding recipes or tracking down discounts, making browsing more interactive. The initial rollout is limited to select popular sites, with restrictions on paywalled and sensitive content. "Since we first announced the new consumer Copilot in October, we have loved seeing the ways people have made Copilot their own. It has helped people think through their ideas and get a fresh perspective, has made it less overwhelming to find the right information, and has even helped people feel supported when navigating difficult conversations." -Microsoft Copilot Vision integrates into Microsoft Edge, functioning solely in the U.S. at present. It enables the AI to "see" the page the user is visiting, read alongside them, and assist in navigating challenges encountered online. Users who enable this functionality can converse with the AI about the web content, promoting a collaborative browsing experience. Importantly, the feature is opt-in and designed with privacy in mind; all data viewed during a session is deleted afterward. Microsoft states that no processed audio, images, or text will be stored or used for AI training during this preview phase. Despite its capabilities, Copilot Vision currently restricts access to a predefined selection of websites, which Microsoft has labeled as "popular" and "approved." Content deemed sensitive or paywalled remains inaccessible to the tool. A Microsoft spokesperson indicated that the list of authorized sites is determined on a case-by-case basis, suggesting that the categorization process may include varied content types. Many publishers are wary of AI tools using their data without permission, a concern stemming from ongoing legal disputes over AI training and paywalls. Microsoft has faced pushback from major publishers regarding how AI processes web content. The company has confirmed that they will respect sites' machine-readable controls to protect against unauthorized scraping, though specifics on which controls will be acknowledged remain unclear. Furthermore, Microsoft has committed to receiving feedback from those it collaborates with to enhance user engagement while respecting publisher interests. The concern for user data privacy is echoed in Microsoft's marketing of Copilot Vision. The company emphasizes that the tool facilitates a more seamless exploration of the internet while maintaining strict privacy protocols. Optional features lend themselves to user autonomy; individuals can choose to enable or disable AI browsing assistance as they see fit. The lack of information about which types of content will be blocked or allowed signals ongoing developments and potential adjustments as user feedback is gathered.

Microsoft's Copilot Vision arrives to surf the web with select users

Microsoft's new Copilot Vision feature that can "see what you see, and hear what you hear" while you navigate the internet is finally being made available, though only to a limited number of Copilot Pro subscribers in the U.S. "Starting today, we are introducing an experience where - with your permission - Copilot can now understand the full context of what you're doing online," according to a Microsoft blog post. "When you choose to enable Copilot Vision, it sees the page you're on, it reads along with you, and you can talk through the problem you're facing together." Recommended Videos The feature was first teased in October alongside the debut of Copilot Voice, Microsoft's answer to ChatGPT's Advanced Voice Mode and Google's Gemini Live. It is currently available as a preview to Pro subscribers through Copilot Labs and available exclusively on Microsoft's Edge browser. To help alleviate user concerns (and distance the new feature from the company's troubled Recall launch), Vision will have to be specifically activated whenever the user wants to employ it and will display a persistent icon (akin to your webcam's On light) until the user turns the feature back off. While active, the AI assistant will "scan, analyze, and offer insights based on what it sees." The system can suggest next steps to take, answer questions about the displayed content, navigate to other parts of the site, and assist with various online tasks. Having Copilot help you surf the web is only the start to Microsoft's AI assistant plans. In January, the company is expected to release the first of its next-generation AI agents, which will autonomously analyze available data to perform tasks on the user's behalf. "Copilot will ultimately be able to act on your behalf, smoothing life's complexities and giving you more time to focus on what matters to you," Mustafa Suleyman, executive vice president and CEO of Microsoft AI, wrote in October. "It'll be an advocate for you in many of life's most important moments. It'll accompany you to that doctor's appointment, take notes and follow up at the right time. It'll share the load of planning and preparing for your child's birthday party. And it'll be there at the end of the day to help you think through a tricky life decision."

Microsoft's Copilot Vision Can See and Understand Your Browsing Activity

Microsoft introduced a new artificial intelligence (AI) capability for Copilot on Thursday. Dubbed Copilot Vision, it now enables the AI chatbot to see and understand the context of what a user does online. The Redmond-based tech giant said that Copilot can access the Microsoft Edge browser. The vision-enabled chatbot can now monitor which websites and web pages the user browses, and then can perform certain commands when prompted. This capability is currently available in preview to a select number of Copilot Pro subscribers in the US. The Windows maker has been trying to bring its AI-powered chatbot Copilot to the forefront of the PC experience. Earlier, it unveiled purpose-built AI agents to complete tasks on the PC, and now, it has added another capability to Copilot that will let it answer user queries based on their browser-related activity, as per the blog post. Notably, Copilot Vision is not an agentic function, so the AI cannot take actions on behalf of the user. Copilot Vision can understand the full context of what a user sees on the Edge browser. It can see the images and elements, read the text, and even comprehend the navigation buttons. The vision-enabled chatbot will be present at the bottom of the screen and users can either type or speak the prompt. Coming to what the AI can do, it can answer questions, summarise and translate text, and spotlight certain parts of the page when prompted to do so. Microsoft states that Copilot Vision can also act as a game assistant and help a user understand how to play games like GeoGuessr, and offer tips while playing a game of Chess online. Highlighting examples, the tech giant said Copilot Vision can help users plan an outing at the museum by pointing out the important information a user should know. It can also help users with shopping by recommending the right products based on the user's needs and preferences. On privacy and security, Microsoft stated that Copilot Vision will be entirely opt-in and users can decide when to turn it on and when to keep it turned off. The AI will not be able to see something that the user does not want to. Additionally, any conversation data with the AI will be deleted at the end of each session. However, the Copilot's responses are logged and they will be used to improve the company's safety systems. Microsoft further added that Copilot Vision does not capture, store, or use any data from publishers to train its AI models. It is also working with a group of external testers (red teamers) to improve the safety of the new feature. Copilot Vision is currently rolling out to a limited number of Copilot Pro subscribers in the US and it can be accessed via Copilot Labs. Notably, Vision will currently work only on a select set of websites, and the functionality will be expanded gradually.

Microsoft Copilot Vision preview watches your screen to offer AI help

Microsoft is slowly launching Copilot Vision in preview, a version of Copilot that "reads over your shoulder" as you browse the web, to Copilot Pro subscribers today. Microsoft is pitching Copilot Vision as a collaborative assistant that works along with you as you browse the Internet. Vision was part of a suite of new Copilot demonstrations that Microsoft showed off in October, and was easily the most underwhelming example of new Copilot technologies that I'd seen. Copilot Vision isn't a multimodal AI capability that can interpret the world around you. Instead, it's a cross between Windows Recall and the current version of Copilot. Vision is basically a real-time version of Copilot that's only available in Microsoft's Edge browser. Instead of asking Copilot for insights, Vision apparently can "scan, analyze and offer insights" based on what your eyes, and its AI sensors, see on the page. Where might Copilot Vision be useful? Dealing with an overwhelming array of information, possibly. Think of a densely packed shopping page, all with similar products; making sense of the mess is what Copilot Vision promises. In this scenario, asking Copilot Vision what items meet your preferences, or offer the most value based on the available features, might be of interest. In the October demonstration Copilot didn't offer anything insightful, often basically reading aloud what it -- and presumably you -- saw on the page. Furthermore, Microsoft is taking a very hesitant approach to Vision, possibly as a reaction to the negative response to its Recall tool. Copilot Vision is only available via Edge, and it's only available on a limited group of websites, too. Microsoft isn't saying how many sites Vision will work on, or what those sites they will be. "The list of allowed websites is determined by category and on a case-by-case basis," a Microsoft representative wrote in response to my question, asking for more clarification on what sites would be included. Copilot Vision is opt-in, however. For one thing, you'll need a $20 monthly Copilot Pro subscription to activate it, and participate on Copilot Labs. You'll need to manually turn it on, too. When you're done using Copilot Vision, all of the data that you've shared with Copilot Vision and how you've interacted with it will be deleted, Microsoft says. After Microsoft's Recall fiasco, a substantial number of Windows users probably want nothing to do with yet another Microsoft tool that records what you see, even temporarily. Others won't care quite as much. It's those adventurers that Microsoft is hoping to entice with the Copilot Vision preview.

Microsoft Copilot Vision will browse the internet with you and chat about it

Microsoft Copilot's vision capabilities are available to test out in preview. Credit: Gabby Jones / Bloomberg / Getty Images Internet browsing is a "lonely experience," says Microsoft. Copilot Vision is your AI companion that can visually process what you're browsing and help you out along the way. On Thursday, Microsoft announced a preview version of Copilot Vision for Copilot Pro subscribers. Microsoft introduced Vision in October as a solution to the problem that users have to explain what they're seeing to Copilot. Vision sits on the bottom of your Microsoft Edge browser and, when enabled, "it sees the page you're on, it reads along with you, and you can talk through the problem you're facing together." Vision is one of the first features to debut in Copilot Labs, its testing ground for AI projects. Labs, which has a noticeably softer and warmer look is Microsoft's way of sharing "works-in-progress" geared towards helping users with everyday tasks. One demo shows Copilot Vision helping a user find the right place to stay, another helps a user make sense of photos of their grandmother's handwritten recipes. It can also help you out in a Minecraft game or with holiday shopping by pointing out the right products on the page. If Copilot Vision reminds you of Windows Recall, Microsoft's "privacy nightmare" tool that consistently takes screenshots of your computer screen, there are several key differences. First, Vision is opt-in, so you can choose if you want to use it. Second, as soon as you end the session, you Copilot conversation is deleted. That said, "only Copilot's responses are logged to improve our safety systems," read the announcement. Microsoft has also thought through some of the sticky issues of web browsing when it comes to paywalled and sensitive information. Vision only works with a select set of websites for now, and it is programmed to "to not provide responses based on paywalled content," according to its FAQs page. Crucially, Microsoft says Copilot Vision does not store or use any publisher content to train its models. Microsoft also says Vision "honors website controls for usage of their content by Copilot," which presumably means sites with robots.txt code that block web crawlers. Mashable has reached out to Microsoft for further clarification of Vision's data controls and will update this story with a response. Starting today, Copilot Vision is rolling out to a limited set of Copilot Pro subscribers. A Copilot Pro subscription costs $20 a month.

Microsoft's Copilot Vision Learns to See, Shop, and Scroll Like You

Similar to Copilot Vision, OpenAI is set to launch its agent, Operator, in January. Microsoft has launched Copilot Vision, an experimental feature integrated into the Edge browser, enabling AI-assisted browsing by analysing web pages in real time. The feature is available in preview for select Copilot Pro subscribers in the United States through Copilot Labs, the company announced on its blog. The opt-in service allows Copilot Vision to read and analyse web pages with user permission, providing insights, simplifying information, and assisting with tasks such as holiday shopping or planning outings. "Browsing no longer needs to be a lonely experience with just you and all your tabs," the Copilot Team stated in the announcement. Vision provides support by scanning and interpreting the page's content, helping users make decisions or learn from the information presented. For instance, Vision can guide users through learning new games or finding specific products that match their preferences during online shopping. Microsoft emphasised privacy and security in the development of Vision, ensuring that user data is not stored after a session ends and is handled in line with the company's privacy policies. The company said, "Only Copilot's responses are logged to improve our safety systems." Currently, Vision is limited to interacting with a select number of websites. Microsoft plans to expand its availability gradually, gathering user feedback to refine the experience. The company is also collaborating with third-party publishers to enhance how Vision interacts with web pages. "Vision does not capture, store, or use any data from publishers to train our models," the blog post added. Similar to Copilot Vision, OpenAI is set to launch its agent, Operator, in January. Meanwhile, Google is working on an experimental AI assistant called "Jarvis," likely to be powered by the Gemini 2.0. Jarvis operates within Chrome and interacts with on-screen elements like fields and buttons. It handles complex tasks, such as booking flights and assisting with online shopping, simplifying digital activities for users. Similarly, Anthropic has launched the 'computer use' feature for the Claude 3.5 Sonnet version, which allows the AI to autonomously perform tasks like moving the mouse, clicking, and typing. Targeted at software developers, it can handle complex activities such as coding a basic website or planning outings across different applications.

Microsoft releases preview of voice-enabled 'Copilot Vision' AI browsing companion for Edge

A new opt-in Microsoft Edge feature, released in limited preview today, augments the browsing experience with AI that "sees" the same page the user is viewing, interacting via voice to offer thoughts and recommendations. Microsoft says the feature, dubbed Copilot Vision, will be available starting today to a limited number of paid Copilot Pro subscribers via Copilot Labs. "It is a new way to invite AI along with you as you navigate the web, tucked neatly into the bottom of your Edge browser whenever you want to ask for help," a Microsoft blog post says. "It's almost like having a second set of eyes as you browse, just turn on Copilot Vision to instantly scan, analyze, and offer insights based on what it sees." The feature was originally announced in October. It's part of a larger effort by Microsoft and others to expand the capabilities of AI as agents and companions. Some users have struggled with Microsoft's simultaneous shift away from an earlier, utilitarian version of its Copilot sidebar for Edge to a more chatty and less functional consumer adaptation.

Microsoft previews Copilot Vision for Pro subscribers

Edge-exclusive tool promises 'second set of eyes' for browsing Microsoft has rolled out a Copilot Vision preview that says more about how poorly websites are designed rather than justifying the need to bring more AI into users' lives. The preview is only available to Copilot Pro subscribers, and only in the US for the time being. The service "looks" at the website a user is viewing to provide contextual suggestions and insights. Doing some holiday shopping? Copilot Vision can suggest products that match a user's needs and preferences. Thinking about taking a trip to a museum? Copilot Vision will highlight what a user might need to know. And so on. Microsoft said: "Browsing no longer needs to be a lonely experience with just you and all your tabs," which makes The Reg wonder exactly what the company thinks its users are like. The tool will only work with Microsoft's browser and is accessed from the base of Edge's window. "It's almost like having a second set of eyes as you browse," said Microsoft. "Just turn on Copilot Vision to instantly scan, analyze, and offer insights based on what it sees." It is debatable if Copilot Vision would even be required if every website was designed with the user in mind. It also sounds like a potential privacy disaster, which is why Microsoft is keen to insist the service is "entirely opt-in." It is up to the user to decide when to turn it on. In addition, any shared data is deleted. However, remember that the service is governed by Microsoft's Privacy Statement, meaning the company could theoretically collect data for its own purposes. That said, in stark contrast to the original unveiling of Recall, Microsoft is taking a tentative approach with this feature. The company said: "It's important to stress that Vision does not capture, store, or use any data from publishers to train our models. In short, we're prioritizing copyright, creators, and our users' privacy and safety - and are putting them all first." Copilot Vision was first announced in October, and this week's broader release reflects early feedback. It will also only work with a select set of websites. Microsoft said: "We're taking our time to gather feedback, iterate on the technology, and will expand access to more Pro subscribers and more websites over time." ®

Microsoft starts testing Copilot Vision in Edge

The opt-in AI feature sees the websites you browse and can chat about their content. Microsoft is rolling out a preview for one of the experimental AI features it previewed in October. Copilot Vision, which scans your web browsing and has a chat with you about it, is available for testing in Copilot Labs. Copilot Vision has strict limits, at least at first. It only works in Microsoft Edge and only on "a select set of websites." (The company says it will carefully expand that list over time.) Even the trial has restrictions: It's only available to a limited number of Copilot Pro ($20 monthly) subscribers in the US. Like the available websites, Microsoft will expand access to more users over time. Microsoft says Copilot Visions sessions are "opt-in and ephemeral," so you'll have to activate it manually, and each session's data is wiped as soon as you're done. The company says the feature only interprets text and images from browsed websites and doesn't process content beyond that or train on web publishers' data. The cautious approach seems to be a course correction from the company's sloppy Recall launch. That "photographic memory" feature, which snaps screenshots of everything you do on your PC, drew the ire of security and privacy experts, who called our Microsoft for -- in a baffling decision -- leaving the feature's data unencrypted. It's finally available for Windows Insiders to try after three delays and security adjustments. If you're a Copilot Pro subscriber, you can sign up to request access to Copilot Vision through Copilot Labs today. You can check out Microsoft's demo video below.

Microsoft's Copilot can now browse the web with you, if you're in the early preview

It's entirely opt-in, but you'll be able to converse with Copilot about what you're seeing Similar to how Windows Recall will give AI access to what's on your screen, Microsoft's rolling out a similar experience for Copilot in a limited preview. Originally teased alongside the redesign of Microsoft's AI assistant, Copilot Vision is now rolling out as an early preview for a select few in the United States. As the name suggests, it will let Copilot see what's on your screen, specifically within Microsoft's Edge browser, so it can essentially browse the web with you. This way, you can hit it with topical questions about what you're seeing - be it an illustration, a video, type-based text, or even handwriting. And no, you won't need to type these out, but you can converse with Copilot with just your voice. In an announcement post published by Microsoft, Copilot Vision is described as: "When you choose to enable Copilot Vision, it sees the page you're on, it reads along with you, and you can talk through the problem you're facing together. Browsing no longer needs to be a lonely experience with just you and all your tabs." It seems like it's almost being pitched as a companion, so if you're browsing a shopping website like Amazon or Zappos, you could potentially ask for suggestions about what you're looking at. Moreover, though, if you're browsing restaurant reviews, you could ask if any mention a specific event - like good for anniversaries - or ask for other recommendations. Seeing how it's built directly into Edge - if you opt-in - Microsoft seemingly hopes that it'll be called upon quite a bit and prove to be useful. Microsoft is also sharing three of Copilot Vision being used for shopping, planning a trip to a museum, and gaming. The latter is almost similar to Game Assist in Edge, which reduces the need to alt-tab. Suppose you're concerned about privacy with Copilot Vision, which may be similar to Recall, which finally launched in November in beta. In that case, you'll be happy that Microsoft is taking a better approach with this new AI feature. It's entirely opt-in and is only currently available via Copilot Labs to a limited number of Copilot Pro subscribers in the United States. Furthermore, in its current build, Copilot Vision is only available for a select set of websites, and it seems Microsoft will only expand on that when it deems it ready to do so. Slow and steady will be the approach for adding more subscribers and expanding the sites it works with; Microsoft explains: "We're taking our time to gather feedback, iterate on the technology, and will expand access to more Pro subscribers and more websites over time." If you're in the early beta of Copilot Vision, another win is that after the session, the associated data and the context it's used within will be deleted. So that information does not stick around Copilot, only the response to the tools "are logged to improve our safety systems," says Microsoft. If you're keen to give it a go and are a Copilot Pro subscriber in the United States, you can check via Copilot Labs to see if Copilot Vision for Edge is available as a preview for you.

Copilot Vision, Microsoft's AI tool that can read your screen, launches in preview | TechCrunch

Microsoft's AI can now read your screen -- or rather, the websites you're browsing. On Thursday, the company began rolling out a limited, U.S.-only preview of Copilot Vision, a tool that can understand and respond to questions about sites you're visiting using Microsoft Edge. Gated behind Copilot Labs, an opt-in program for experimental AI capabilities, Copilot Vision can analyze text and images on web pages to answer queries like "What's the recipe for this lasagna?" Note that Copilot Labs requires a subscription to Microsoft's Copilot Pro plan, which costs $20 per month. Beyond answering questions, Copilot Vision can summarize and translate text, and handle tasks like spotlighting discounted products in a store catalog. It can also serve as a game assistant, for example offering pointers during matches on Chess.com. "When you choose to enable Copilot Vision, it sees the page you're on, it reads along with you, and you can talk through the problem you're facing together," Microsoft wrote in a blog post shared with TechCrunch. "It's a new way to invite AI along with you as you navigate the web, tucked neatly into the bottom of your Edge browser whenever you want to ask for help." No doubt eager to avoid more bad press from AI privacy fumbles, Microsoft is stressing that Copilot Vision deletes data following every session. Processed audio, images or text aren't stored or used to train models, the company says -- at least not in this preview release. Copilot Vision is also limited in the types of websites it can view. For the time being, Microsoft's blocking the feature from working on paywalled and "sensitive" content, limiting Vision to a pre-approved list of "popular" sites. What's "sensitive" entail, exactly? Porn? Graphic violence? Microsoft wouldn't say. Microsoft's cautious approach is partly the product of legal disputes with news outlets. In one ongoing suit, The New York Times alleges that Microsoft let users get around its paywall by serving NY Times articles through the Copilot chatbot on Bing. Many major publishers have opted to block AI tools from trawling their websites not only out of fear their data will be used without permission, but also to prevent these tools from sending their server costs soaring. Microsoft said that Copilot Vision will respect sites' "machine-readable controls on AI," like rules that disallow bots from scraping data for AI training. However, the company hasn't said precisely which controls Vision will respect; there are several in use. If the current trend holds, Copilot Vision may not work on some of the web's top news sites. But Microsoft said it's committed to "taking feedback" to allay publishers' concerns. "Some of those we've collaborated with are third-party publishers, who help us understand how Vision could be used to help people better engage and make decisions on their pages," the company wrote in the blog. "These inputs will help us design Vision to be more helpful in how it interacts with web pages."

Microsoft's Copilot can now see what you see in real time - and speak to you about it

Need a second opinion on something you are looking at online? Now, you can have an AI-powered assistant at your beck and call. Microsoft Copliot has proven a worthy competitor to ChatGPT. As ChatGPT continues to evolve, so has Copilot, but recently, it has been trailing behind. However, this latest update might be enough to return its competitive edge. On Thursday, Microsoft made Copilot Vision, an experience in which Copilot can view and understand the context of what you're doing online to provide verbal real-time assistance, available in preview for Pro subscribers through Copilot Labs. Also: Google Photos unveils its 'Spotify Wrapped' style end-of-year recap. How to access it When the user enables Copilot Vision, it can read along with you, discuss issues you are having while browsing, analyze your site, and provide insights based on what it sees. It's essentially an assistant for all your browsing needs -- on-call whenever you need it. I had the opportunity to demo the feature at a NYC Microsoft Copilot and Windows Event in October where I witnessed real-world applications that showcased some of Copilot Visions' assistance value. Also: OpenAI's 12 days of product drops and demos advances - what's new on day 2 In my demo, the user asked Copilot Vision for assistance with picking outfit inspiration from Pinterest. Copilot Vision suggested an outfit from all the options on the page and offered encouragement when the user said he didn't think he had what it took to pull the outfit off. This interaction took place verbally, making it a natural experience. Other real-life use cases Microsoft provided for Copilot Vision included helping users with holiday shopping by pointing out products on the page that match what they are looking for or helping them plan a day at the museum by highlighting important information to know before leaving for the trip. The new experience lives on Microsoft Edge, tucked at the bottom of the browser. Copilot Vision is rolling out first to a limited number of Copilot Pro subscribers in the United States through Copilot Labs and will initially only work on a select number of websites. As it collects feedback, Microsoft says it will expand access. Also: The best AI chatbots of 2024: ChatGPT, Copilot, and worthy alternatives For users concerned about privacy, Microsoft says it will be entirely opt-in, giving the user control over whether they want Copilot to see their browsing activity. Even for those who do choose to opt-in, all the data shared with Copilot during that session, including what you say and the context you share, is deleted, according to the blog post. The Copilot Pro subscription is $20 per month. It includes other perks, such as priority access to the latest models, more access to Copilot Voice, and Copilot in select Microsoft 365 apps, including Word, Excel, PowerPoint, OneNote, and Outlook.

Microsoft rolls out new AI feature that views everything you do on the internet

TL;DR: Microsoft announced Copilot Vision, a new feature for Copilot Pro, offering an AI companion for browsing. Available to select US users, it requires user permission and deletes data post-session. It's optional, prioritizes privacy, and initially supports limited websites. Copilot Pro costs $20 monthly, enhancing Office apps and browsing. Microsoft has taken to its blog to announce the availability of Copilot Vision, a new Copilot Pro feature that is designed to provide a helpful AI companion for internet browsing. Microsoft has detailed the new feature in its blog post where it states Copilot Vision will require permission of the user and, if enabled, will be able to see what webpage a user is on, digest the information on the webpage, and provide helpful answers and information about what the user is searching. The new feature is currently only available in the US to a small group of Copilot Pro subscribers, which is a $20 per month subscription that unlocks Copilot across Office applications, priority access to Copilot during peak times, and now Copilot Vision. The Windows maker explains that Copilot will be able to "understand the full context of what you're doing online," and when Copilot Vision is enabled, it "sees the page you're on, it reads along with you, and you can talk through the problem you're facing together. Browsing no longer needs to be a lonely experience with just you and all your tabs." For those worried about security, Microsoft has emphasized that Copilot VIsion is an entirely optional feature that needs to be manually switched on by a user for it to be enabled. As for all of the visual data of users' web browsers, Copilot Vision will be gathering; Microsoft says all Copilot VIsion data is immediately deleted once the session is terminated. Furthermore, Copilot Vision is exclusively available on Microsoft's Edge browser. "As we've built Copilot Vision, we have made your security, control and privacy the top priority. To start, Vision is entirely opt-in, so you decide when to turn it on as your second set of eyes on the web. In addition, once you end a session with Vision, all data about what you say and the context you share with Copilot is deleted," writes Microsoft

Microsoft's new Copilot Vision feature one-ups ChatGPT. Here's how to access it

Have you ever needed a second opinion on something you are looking at online, but didn't have anyone readily available to chat with? Now, you can have an AI-powered assistant at your beck and call. Microsoft Copliot has proven a worthy competitor to ChatGPT. As ChatGPT continues to evolve, so has Copilot, but recently, it has been trailing behind. However, this latest update might be enough to return its competitive edge. On Thursday, Microsoft made Copilot Vision, an experience in which Copilot can view and understand the context of what you're doing online to provide verbal real-time assistance, available in preview for Pro subscribers through Copilot Labs. Also: Don't trust ChatGPT Search and definitely verify anything it tells you When the user enables Copilot Vision, it can read along with you, discuss issues you are having while browsing, analyze your site, and provide insights based on what it sees. It's essentially an assistant for all your browsing needs -- on-call whenever you need it. I had the opportunity to demo the feature at a NYC Microsoft Copilot and Windows Event in October where I witnessed real-world applications that showcased some of Copilot Visions' assistance value. Also: '12 Days of OpenAI' product launches and demos start today - here's how to watch In my demo, the user asked Copilot Vision for assistance with picking outfit inspiration from Pinterest. Copilot Vision suggested an outfit from all the options on the page and offered encouragement when the user said he didn't think he had what it took to pull the outfit off. This interaction took place verbally, making it a natural experience. Other real-life use cases Microsoft provided for Copilot Vision included helping users with holiday shopping by pointing out products on the page that match what they are looking for or helping them plan a day at the museum by highlighting important information to know before leaving for the trip. The new experience lives on Microsoft Edge, tucked at the bottom of the browser. Copilot Vision is rolling out first to a limited number of Copilot Pro subscribers in the United States through Copilot Labs and will initially only work on a select number of websites. As it collects feedback, Microsoft says it will expand access. Also: The best AI chatbots of 2024: ChatGPT, Copilot, and worthy alternatives For users concerned about privacy, Microsoft says it will be entirely opt-in, giving the user control over whether they want Copilot to see their browsing activity. Even for those who do choose to opt-in, all the data shared with Copilot during that session, including what you say and the context you share, is deleted, according to the blog post. The Copilot Pro subscription is $20 per month. It includes other perks, such as priority access to the latest models, more access to Copilot Voice, and Copilot in select Microsoft 365 apps, including Word, Excel, PowerPoint, OneNote, and Outlook.

I'm curious to see if Microsoft's new Copilot Vision can actually help me browse better

Key Takeaways Copilot Vision is now available for select Copilot Pro users in the US. Microsoft assures users that Copilot Vision prioritizes security and privacy. Users must wait for the rollout to try Copilot Vision, which is only compatible with Microsoft Edge. ✕ Remove Ads Hey, remember when Microsoft revealed that huge batch of Copilot features in early October? If you don't, one of the nuggets within the big list of planned tools was something called "Copilot Vision," which could scan a webpage and allow people to ask questions about it. At the time, Microsoft hadn't given an exact release date for Copilot Vision; however, the Redmond giant has now announced that a "limited number of Copilot Pro subscribers in the US" can now give it a shot. Related Microsoft Copilot: What is it, and how does it work? Is Microsoft Copilot the best AI chatbot available right now? Posts Copilot Vision is now available for select users As announced on the Microsoft Copilot blog, Copilot Vision is now beginning its big rollout. This AI tool sits alongside your browser and can be invoked at any time to read the webpage you're currently on. Once Copilot has scanned the page, you can ask questions about the content or what to do next. It sounds especially useful for breaking down long or complicated articles into a more digestible manner. ✕ Remove Ads Fortunately, it seems that Microsoft has learned from its recent Recall fiasco, as it's taking extra measures to reassure people that it won't harvest your data: As we've built Copilot Vision, we have made your security, control and privacy the top priority. To start, Vision is entirely opt-in, so you decide when to turn it on as your second set of eyes on the web. In addition, once you end a session with Vision, all data about what you say and the context you share with Copilot is deleted. If you want to try Copilot Vision, you'll need to be in the US and subscribed to Copilot Pro. If you tick both boxes, you'll need to wait for the rollout to arrive in your account. Once that happens, you should see Copilot Vision as part of the new Copilot Labs program. Also, note that Copilot Vision will only work on Microsoft Edge. ✕ Remove Ads

Copilot Vision Is Coming To Microsoft Edge

If Your PC Won't Run Windows 11, Here Are 5 Great Linux Operating Systems to Try Instead Microsoft has introduced a preview version of Copilot Vision. It's a new AI tool for Microsoft Edge that can look at your current tab and provide answers. Right now, it's available to a small group of Copilot Pro subscribers in the United States through Copilot Labs. This feature lets Copilot look at web pages as you browse. With your permission, it can read the content on a webpage and offer helpful information and insights based on what you're viewing. Users can ask the AI for help with specific tasks or questions about the information on a page. For example, it can make planning a trip to a museum easier by finding important details, help with online shopping by pointing out products that fit what you like, and can make tough information easier to grasp. Arc has a similar feature with its 'find' feature, where it will read the page only if asked. It can answer questions about things on the page, give direct quotes, and explain its answers. Microsoft clarifies that using this feature is completely optional, so users can decide when to use it. Microsoft has also set up ways to delete data, making sure that any information from user interactions is erased automatically after each session. Microsoft's privacy policy covers all data handling, and only the responses from Copilot are saved to help improve safety. Microsoft is rolling out Copilot Vision carefully by giving access to a small group of users and certain websites. Over time, Microsoft plans to expand access to more Pro subscribers and websites, depending on the feedback. The company also made sure to state that it does not collect, store, or use any publisher data to train the model, prioritizing copyright protection and user privacy. The company is likely making sure to be clear about this due to a similar feature, Microsoft Recall, coming under fire for its privacy concerns. Source: Microsoft, The Verge

Microsoft Copilot will now be able to see what you're doing online

Microsoft has begun testing a new feature for its artificial intelligence (AI) assistant Copilot called Copilot Vision which allows the AI to see what you are looking at online. When enabled, Copilot vision allows the AI to read information along with the user, allowing the user to talk through problems. The feature is currently only available as a preview for a limited number of Copilot Pro subscribers. The feature is only available on Microsoft Edge and will initially only work with select websites. This isn't the first time Microsoft has tried to launch an AI-based feature that can see what users are doing on their devices. In May this year, the company announced Copilot+PCs that come with a feature called Recall. Recall allows users to scroll through a timeline of their work across every application, website, and document. To do so, the feature would take frequent screenshots of users' active screens every few seconds. Right from the announcement, people compared the feature to 'spyware' and flagged potential privacy risks that it posed. The feature also came under the regulatory radar of the UK's Information Commissioner's Office which led the company to make it "opt-in" meaning that it would only save screenshots with user consent. After delaying the launch of Copilot+AI, the feature recently became available in a preview version. Given that just like Recall, Copilot Vision can also access what you are doing on your web browser, there could be privacy concerns about the feature's access to sensitive or confidential information. To address these concerns, the company has made Vision opt-in, which means that users decide whether or not to activate the feature. "Once you end a session with Vision, all data about what you say and the context you share with Copilot is deleted," the company mentions, adding that the company only records Copilot's responses to improve the safety of Microsoft's systems.

Microsoft's Copilot can browse the web with you using AI "Vision"

It's an entirely optional experience, and you have to explicitly grant permission for Copilot Vision to be able to read webpages in Microsoft's Edge browser. Microsoft originally demonstrated the feature in October by showing how its AI assistant could read images from a collection on OneDrive on the web and even decipher hand-written recipes and offer up cooking tips. You could also use this feature while you're shopping on the web to find product recommendations.