Forget about Google Assistant, Samsung's new Android flagships, the Galaxy S8 and the Galaxy S8+ launched their own voice assistant called Bixby.

It's Samsung's attempt to take on the likes of Siri, Google Assistant, Cortana and Alexa, with a new AI agent that's going to span Samsung devices. First and foremost it's appearing on Samsung's phones, but what is it and what will it do?

Pocket-lintWhat Is Bixby Samsung S Smart Ai Explained image 16

Fully announced via a Samsung blog post, Bixby is an artificial intelligence system that's designed to make device interaction easier, specifically designed to avoid the complexity of increasingly fully-featured devices. It made its debut on the Samsung Galaxy S8 devices, but is designed to work across a range of Samsung products.

It encompasses a range of different features, Bixby Voice, Bixby Vision, Bixby Home and Bixby Reminder, wiping out a set of previous Samsung technologies like S Voice, while also looking to be more omnipresent across your whole device.

In the future we expect to be using Bixby to control the TV, air conditioning as well as phones and Samsung uses this new service to provide one method of interaction with a full range of technologies, although currently it's limited to smartphones.

In October 2017, Samsung announced the second-generation version of Bixby. According to the company, Bixby 2.0 is a "fundamental leap forward for digital assistants" and a "bold reinvention of the platform". In a nutshell: Bixby 2.0 is designed to make Bixby available on "any and all devices". It will be available on smartphones, TVs, refrigerators, washers, speakers, and other connected devices.

Bixby 2.0 will be "open", allowing developers to integrate the voice assistant into their products and choose how you will be able to interact with their services. So, we suspect several companies will announce Bixby-powered smart speakers in the coming months. Samsung also said Bixby 2.0 features enhanced natural language capabilities, which will result in more natural commands and complex processing.

Samsung ran into performance issues with Bixby early on and wasn't able to include the English version on the Galaxy S8 and S8 Plus at launch. Its English voice recognition capabilities weren't up to par, and we're guessing Samsung has been working hard to improve it. Going forward, the company said Bixby 2.0 will "really get to know and understand" who you are as well as the members of your family.

Samsung is launching a private beta program with the Bixby SDK. It's now available for select developers.

Samsung has confirmed some of the underlying principles of Bixby, detailing that there are three pillars that help drive it:

  1. Bixby is a complete solution: it is designed to let you carry out a full range of interactions, rather than launching an app for example, or carrying out a single task. Samsung says that Bixby will be able to do just about everything you can do with an app using touch.
  2. Bixby is contextually aware: this is one of the buzzwords of AI, demonstrated by Google Assistant, for example. This will mean that Bixby can recognise the state that the app is in and take the right actions based on your requests, also letting you mix voice or touch.
  3. Bixby understands natural language: this means that you don't need to use set phrases, but you can give incomplete information and Bixby can interpret and take action. Natural language recognition has been key to the rise of Alexa, for example, and is now a key element of modern AI.

The service essentially works in the same way as other AI solutions like Google Assistant or Amazon Alexa in that it listens to your voice, interprets the information and returns the resulting action. The contextual awareness means you can get it to take actions without laboriously detailing exactly what to do with what - it already understands where you are so can take the logical next step, so there's no need to use the precise language that you did on older voice-control systems.

On the Samsung Galaxy S8, S8+ and Note 8 there is a Bixby button on the left-hand side of the phone. This is used to open the Bixby agent so that you can speak a command, or you can use the hotword "Hi Bixby", once you've set it up. To access Bixby Voice you'll have to press and hold the button and talk, once you release the button, Bixby interprets what it has heard. 

For those using the "Hi Bixby" wake word, you'll be able to talk to your device using natural language as you might do with Google Assistant. However, Bixby seems prone to launching accidentally, so using the button press method prevents false recognition.

This only applies to the voice control side of Bixby, otherwise you'll find things like Bixby Vision integrated into the camera and ready to use at a tap.

Here's what Bixby will do:

One of the main aims of Bixby according to Samsung, is to deal with increasingly complicated devices. Not only did Samsung say that in its blog post, but every time Samsung demos Bixby or talks about it, it's focusing on device control first and foremost - especially around complex tasks.

Here are some examples of what Bixby will do:

  • On saying "show this on my TV", the Galaxy S8 screen can mirrored on a Samsung TV.
  • On saying "use this as my wallpaper" the S8 saves the onscreen image as the wallpaper for the phone.
  • On saying "take a selfie and share it will Facebook, the phone tries to do exactly that.

These commands all make logical sense, and fitting with the outlined aims that Samsung originally set out for Bixby, it's easy to ask it to do things like changing the volume or increasing the brightness of the phone. Generally, when it comes to device control, Bixby is very good, as it is with composing messages for you, or reading incoming messages from your contacts. 

Care needs to be taken, however: the Hi Bixby hotword does trigger easily and we once had the phone erroneously fire up Bixby, where it listened to some of the conversation in the room (about assistants) and then launched the phone's visual disability mode, which took some time to then disable again so we could use the phone as normal.

Pocket-lintwhat is bixby samsung s smart ai explained image 5

Bixby Vision offers a range of functions that take advantage of the camera on the phone. Working in a similar way to Google Goggles or the Amazon shopping app you can either ask Bixby what something is, or open the camera app and hit the Bixby Vision button. This is a great feature, but one we've used plenty of times before in other apps.

Bixby Vision will essentially identify whatever the camera is pointed at, with options presented depending on what it sees, offering to identify an image, place, text or go to shopping options. When the Bixby Vision thinks it can see wine, it offers a wine search function. These then lead into more information, once the device has been identified.

We've seen it:

  • Identify The Shard in London and suggest local restaurants.
  • Recognise a box of Lindt Lindor (strawberries and cream flavour) and suggest shopping options.
  • Spot a bottle of Casillero di Diablo and offer details on the wine.
  • Recognise text, making it easy to copy text from a poster.

The best bit about Bixby Vision is that it's not dependent on the full Bixby Voice service, with direct access through the camera, so even those who don't yet have language support for the full Bixby Voice service will get Bixby Vision.

Pocket-lintwhat is bixby samsung s smart ai explained image 9

There's another side to Bixby. For those not diving into the voice service - or those who can't get it because it's not supported in their country - then there's Bixby Home. This is part of your smartphone home screen, living off to the left of your home page.

In the UK this integrates the Upday news service (accessed via a left-swipe on the screen or a short press on the button) or Flipboard for those elsewhere and pulls in cards from supported apps. This is very much like the Google Now page, filled with information cards, so it's like a personal feed.

Many apps and services can be integrated into Bixby Home, like Twitter, Facebook, Spotify and your Gallery, so you can access highlights from all these things there. That means you'll have a card showing pictures you've recently taken, a card showing current Twitter trends or controls for your Spotify music.

In reality there's very little that Bixby Home does that's unique or useful: it's just another feeds app like HTC's BlinkFeed and we question whether it will be useful to anyone. Smartphone users are likely to head to the information they want, rather than scrolling through snippets of information scraped from elsewhere on the phone.

You can, however, turn it off easily, so it doesn't appear on your home screen, however it's always there and launches whenever you press and release the Bixby button.

Pocket-lintwhat is bixby samsung s smart ai explained image 14

Samsung has confirmed that Bixby will be integrated with a number of apps, detailing that it would support 10 native Samsung apps at launch. This is where Bixby will hit rougher ground, because of the variety of apps that you'll have on your phone. Samsung hasn't detailed what those apps will be, but it's confirmed to work with:

  • Phone
  • Messages
  • Internet
  • Gallery
  • Camera
  • Contacts
  • Settings
  • Bixby
  • Reminder
  • Weather

Exactly what Bixby does with Bixby, we don't know.

Samsung has also said that it plans to support third-party apps through an SDK, but the big question is surrounding Google apps. Samsung relies on some core Google apps for the experience it offers - like Play Music and Google Maps, apps that definitely need integration.

This does throw up some stumbling point for Bixby. Although you can ask it to open Google Maps, you can't get any sort of navigation from it and it won't be able to locate you or tell you where you are. You're better with Google Assistant if that's what you want to do.

Injong Rhee, executive VP and head of R&D software and services at Samsung has said "Bixby will be gradually applied to all our appliances. In the future you would be able to control your air conditioner or TV through Bixby." Rhee goes on to say that because Bixby is a cloud service, all you need is an internet connection and a mic and Bixby could be set to task.

We've also seen some very clever demos with Samsung's 2017 televisions, which also offer full voice control. In a similar demo, we saw a voice command to change the contrast demonstrating the same sort of functionality. There was no mention of Bixby at this demo, we asked Samsung at the time if it was part of the same system and we were told that Bixby hadn't been mentioned - but we suspect it is.

Samsung is planning to expand the reach of Bixby with the launch of a Bixby speaker to compete with Google Home and Amazon Alexa.  

Bixby launched with support for Korean and US English, although Bixby is in all S8 devices - it just might not work if you're device is set to a different language. Samsung told us that there would be support for British English for the UK and other languages for Europe, but as the service rolls-out to more territories, it still only supports US English and Korean.

On 22 August Samsung confirmed that Bixby is being enabled for 200 countries, although the language support remains the same - Korean and US English. Samsung specifically mentions UK, Canada, South Africa and Australia, which all happen to be English speaking countries.

Bixby launched on the Samsung Galaxy S8 and Galaxy S8+ and as we said, should now be available to a wide range of people, if you speak English or Korean. It is also available on the Galaxy Note 8.

There's no word on whether Samsung will enable Bixby on Galaxy S7 or the Gear S3 for example, but as Samsung has confirmed that all you need is an internet connection and a microphone, it seems likely that Samsung will enable it for other devices. 

Samsung's ambition is clear to have Bixby on all its devices, so we suspect there will be plenty on the update list.

Currently supported devices:

  • Samsung Galaxy S8
  • Samsung Galaxy S8+
  • Samsung Galaxy Note 8