Wal-E: Retail Assistant with AR and NLP:

Let’s create a generic template how AR(Augmented Reality) and NLP(Natural Language Processing) can be applied to Retail market to leverage the maximum benefits to customer like Offers, finding items in your shopping list, creating the shopping list using NLP based system, getting offers from assistant, showing the navigation path in AR etc.

Main Components used here:

(1). Google dialogflow: Fulfilled by our own back-end hosted in heroku with clearDB mySql database.

(2). iOS mobile app having AR.

(3). Estimote beacons: To give the indoor positioning of the user inside the store.

(4). Google map: it helps to identify the user location whether he/she is inside the store or not.

(5). Dialogflow interacts with our own back-end which is hosted in https://explore-world.herokuapp.com/. Here is the project description how to create the server locally and hosted to heroku (https://github.com/ashislaha/Let-s-learn-server-side-coding).

Video:

https://www.youtube.com/watch?v=C-ln0f2mMxc&feature=youtu.be

Architecture of the entire project:

Flow:

Part 1: Store identification:

App will decide whether user is inside the store or not using google maps and user core location.

(1). Based on user location, app send the user lat and lng to back-end

(2). Back-end will do a reverse geo code to convert (lat, lng) to formatted address where you will get the zipCode.

(3). Get the stores information from ZipCode with their physical position (lat, lng)

(4). Send the information to the mobile app, app will decide whether user location belongs to any of the stores or not.

(5). If user is inside the store, it will open the store map else it will open the Walmart assistant

Part 2: Store Map:

If user is inside the store, user will get the user location (blue dot) on store 2-D map.

We are using Estimote beacons to identify the user location inside store.

User will get 2 options: (1). Assistant (2). AR view

Part 3: NLP (Natural language processing) based assistant:

(1). We are using DialogFlow for assistant where user can get offers of the day at the very begining.

(2). User will add product to the shopping list with the help of assistant. Dialogflow send the information to our back-end which is hosted to heroku, it called the walmartlabs developer open api to fetch the products list.

If user choose any of the product, it will be added to the shopping list.

(3). User can ask to the assistant to show the shopping list:

(3). User can remove the shopping list any time with the help of assistant.

(4). you can start shopping, the Augmented Reality view will help the user to navigate throught the products

(5). user can also get the offers based on his past history purchase list. We are using the mock data in server and recommending the user about product busket level offers.

Part 4: Open AR (Augmented Reality) view:

(1). User will get a path from his/her location to the shopping list products.

(2). The path will be created based on single-source shorest path to all the items.

(3). User can see the offers of other department also in AR while navigating through the path.

(3). User will get a collection view of items in AR view. Once user will get the item, he/she can update the shopping list by tapping on that.

(4). User will get notified through notification when he/she is close to a particular products department.

General Information about beacons:

Using iBeacons:

Beacon:
a low energy bluetooth device which transmit signal.

iBeacon:
iBeacon is the Apple provided framework to capture the beacon signal in iOS platform.

Define a Beacon:

import CoreLocation

class Item: NSObject, NSCoding {

let name: String
let icon: Int
let uuid: UUID // 128 bit Int
let majorValue: CLBeaconMajorValue
let minorValue: CLBeaconMinorValue
var beacon: CLBeacon?

init(name: String, icon: Int, uuid: UUID, majorValue: Int, minorValue: Int) {
    self.name = name
    self.icon = icon
    self.uuid = uuid
    self.majorValue = CLBeaconMajorValue(majorValue) // UInt32

self.minorValue = CLBeaconMinorValue(minorValue) // UInt32
   }

func asBeaconRegion() -> CLBeaconRegion {
    return CLBeaconRegion(proximityUUID: uuid, major: majorValue, minor: minorValue, identifier: name)
}

func nameForProximity(_ proximity: CLProximity) -> String {
    switch proximity {
    case .far: return "far"
    case .immediate: return "immediate"
    case .near: return "near"
    case .unknown: return "unknown"
    }
}

func locationString() -> String {
    guard let beacon = beacon else { return "Location Unknown" }
    let proximity = nameForProximity(beacon.proximity)
    let accurancy = String(format: "%0.2f", beacon.accuracy)
    var location = "location \(proximity)"
    if beacon.proximity != .unknown {
        location += "approx. \(accurancy) m "
    }
    return location
}
}

A beacon has identifier, UUID, majorValue and minorValue. It emits signal. App will receive those signal through CLLocationManagerDelegate.

Start Monitoring Beacons signal:

func startMonitoringBeaconItems(_ item: Item) {
    let beaconRegion = item.asBeaconRegion()
    locationManager.startMonitoring(for: beaconRegion)
    locationManager.startRangingBeacons(in: beaconRegion)
}

func stopMonitoringBeaconItem(_ item: Item) {
    let beaconRegion = item.asBeaconRegion()
    locationManager.stopMonitoring(for: beaconRegion)
    locationManager.stopRangingBeacons(in: beaconRegion)
}

Listening Beacons signal:

// MARK: CLLocationDelegate
extension ItemsViewController: CLLocationManagerDelegate {

// error handling
  func locationManager(_ manager: CLLocationManager, didFailWithError error: Error) {
     print(error.localizedDescription)
  }
  func locationManager(_ manager: CLLocationManager, monitoringDidFailFor region: CLRegion?, withError error: Error) {
     print("Error while detectiong Region:\(region?.identifier ?? ""), with error: \(error.localizedDescription)")
   }
   func locationManager(_ manager: CLLocationManager, rangingBeaconsDidFailFor region: CLBeaconRegion, withError error: Error) {
      print("Error while finding beacon region: \(region.identifier) with error:\(error.localizedDescription)")
   }

   // listening beacons
   func locationManager(_ manager: CLLocationManager, didRangeBeacons beacons: [CLBeacon], in region: CLBeaconRegion) {
      print(beacons)
    }
}