Google's Bard AI can tap the company's apps — and your personal data — for better responses

We’ve already seen OpenAI and Salesforce incorporate their standalone chatbots into larger, more comprehensive machine learning platforms that span the breadth and depth of their businesses. On Tuesday, Google announced that its Bard AI is receiving the same treatment and has been empowered to pull real-time data from other Google applications including Docs, Maps, Lens, Flights, Hotels and YouTube, as well as the users’ own silo of stored personal data, to provide more relevant and actionable chatbot responses.

“I've had the great fortune of being a part of the team from the inception,” Jack Krawczyk,bproduct lead for Bard, told Engadget. “This Thursday marks six months since Bard entered into the world.”

But despite of the technology’s rapid spread, Krawczyk concedes that many users remain wary of it, either because they don’t see an immediate use-case for it in their personal lives or “some others are saying, ‘I've also heard that it makes things up a lot.’” Bard’s new capabilities are meant to help assuage those concerns and build public trust with the technology through increased transparency and more fully explained reasoning by the AI.

“We started off talking about Bard as a creative collaborator because that we saw in our initial testing, that's how people use it,” he continued.”Six months into the experiment, that hypothesis is truly validating.”

The new iteration of Bard, “is the first time a language model will not only talk about how confident it is in its answer by finding content from across the web and linking to it,” Krawczyk said. “It's also the first time the language model is willing to admit that it made a mistake or got something wrong, and we think that's a critical step.” Krawczyk notes that feedback provided by the experimental tool’s users over the past half year has enabled the company to rapidly iterate increasingly robust, “more intuitive and imaginative” language models.

To that end, the chatbot can now parse and respond to more extensive and complicated prompts, such as “It’s my first semester in college and I want to get involved, but also would like to get strong grades. Help me formulate a point of view on why it’s important to balance my involvement in school clubs and extracurriculars, while also focusing on my studies.”

In order to provide these more expansive responses, Google is following OpenAI and Salesforce’s lead in enabling its AI to access the real-time capabilities of the company’s other apps — including Maps, YouTube, Hotels and Flights, among others. What’s more, users will be able to mix and match those API requests using natural language requests.

That is, if you want to take your partner to Puerto Rico on February 14, 2024 and go sightseeing, you’ll be able to ask Bard, “can you show me flights to Puerto Rico and available hotels on Valentines Day next year?” and then follow up with, “show me a map of interesting sites near our hotel” and Bard should be able to provide a list of potential flights, available hotel rooms and a list of stuff to do outside of said hotel room once you book it.

“We believe there's already a high bar for the transparency choice and control that you have with your data,” Krawczyk said. “It needs to be even higher as it relates to bringing in your private data.”

In an effort to improve the transparency of its AI’s reasoning, Google is both explicitly linking to the sites that it is summarizing, and introducing a Double Check feature that will highlight potentially unfounded responses. When users click on Bard’s G button, the AI will independently audit its latest response and search the web for supporting information. If Search turns up contradictory evidence, the statement is highlighted orange. Conversely, heavily referenced and supported statements will be highlighted green.

Users will also be able to opt-in to a feature, dubbed Bard Extensions, that will allow the AI access to their personal Google data (emails, photos, calendar entries, et cetera) so that it can provide specific answers about their daily lives. Instead of digging through email chains looking for a specific important date, for example, users will be able to ask Bard to scour their Gmail account for the information, as well as summarize the most important points of the overall discussion. Or, the user could work with the chatbot to draft a cover letter based specifically on the work experience listed in their resume.

And to allay concerns over Google potentially having even more access to your personal data than it already does, the company has pledged that “your content from Gmail, Docs and Drive is not seen by human reviewers, used by Bard to show you ads or used to train the Bard model.” What’s more, users will be able to opt in and out of the system at will and can allow or deny access to specific files. The service is initially only available to non-enterprise users in English, though the company is working to expand those offerings in the future.

“We think that this is a really critical step, but so much context is required in communication,” Krawczyk said. “We think really harnessing the healthy and open web is key because what we found in the first six months of Bard is, people will see a response and then follow up with trusted content to actually understand and go deeper. We're excited to provide that for people with this new experience.”