07 February 2018

Building a floating audio player in Mixed Reality

Intro

imageAs I promised in my previous blog post, I would write about how I created the floating audio player designed to easily demonstrate how to download and play audio files in Mixed Reality (or actually, just Unity, because the code is not MR specific). I kind of skipped over the UI side. In this post I am going to talk a little more about the floating audio player itself. This code is using the Mixed Reality Toolkit and so actually is Mixed Reality specific.

Dissecting the AudioPlayer prefab

The main game object

imageThe AudioPlayer consists out of two other prefabs, a SquareButton and a Slider. I have talked about this button before, so I won’t go over that one in detail again. The main game object of the AudioPlayer has an AudioSource and two extra scripts. The simple version of the Sound Playback Controller was already described in the previous blog post, and will be handled in great detail here. The other script is a standard Billboard script from the Mixed Reality toolkit. It essentially keeps the object rotated towards the camera, so you will never see it from the side of the backside where it’s hard to read and operate. Note I have restricted pivot axis to Y, so it only rotates over a vertical axis.

The button

imageIt’s a fairly standard SquareButton, and I have set the text and icon as I described here. Now that button only shows in the editor, the runtime text and the icon are set by a simple script that toggles icon and text, so that the button cycles between being a “Play” and a “Pause” button. That script is pretty easy:

using HoloToolkit.Unity.InputModule;
using UnityEngine;

public class IconToggler : MonoBehaviour, IInputClickHandler
{
    public Texture2D Icon1;

    public Texture2D Icon2;

    public string Text1;

    public string Text2;

    private TextMesh _textMesh;

    private GameObject _buttonFace;

    void Awake ()
    {
        _buttonFace = gameObject.transform.
           Find("UIButtonSquare/UIButtonSquareIcon").gameObject;
        var text = gameObject.transform.Find("UIButtonSquare/Text").gameObject;
        _textMesh = text.GetComponent<TextMesh>();
        SetBaseState();
    }

    public void SetBaseState()
    {
       _textMesh.text = Text1;
       _buttonFace.GetComponent<Renderer>().sharedMaterial.mainTexture = Icon1;
    }

    private float _lastClick;

    public void OnInputClicked(InputClickedEventData eventData)
    {
        if (Time.time - _lastClick > 0.1)
        {
            _lastClick = Time.time;
            Toggle();
        }
    }

    public void Toggle()
    {
        var material = _buttonFace.GetComponent<Renderer>().sharedMaterial;
        material.mainTexture = material.mainTexture == Icon1 ? Icon2 : Icon1;
       _textMesh.text = _textMesh.text == Text1 ? Text2 : Text1;
    }
}

It has four public properties, as already is visible in the image: Image1 and Text1 for the default image and text (“Play”), Image 2 and Text 2 for the alternate image and text (“Pause”). The Awake method grabs some objects within the button itself, then sets the base state – which is, the default icon and text.

It also implements IInputClickHandler, so the user can tap it. In OnInputClicked it calls the Toggle method. That then toggles both text and image. Notice there’s simple time based guard OnInputClicked. This is to prevent the button from sending a burst of click events. In the Unity editor, I mostly get two clicks every time I press the XBox controller A button, and then nothing happens. Annoying, but easily mitigated this way.

The Slider

I can be short about that one. I did not create that, but simply nicked it from the Mixed Reality Toolkit Examples. It sits in HoloToolkit-Examples\UX\Prefabs. I like making stuff, but I like stealing reusing stuff even better.

The extended Sound Playback Controller

Let’s start at Start ;). Note: the BaseMediaLoader was handled in the previous blog post,

public class SoundPlaybackController : BaseMediaLoader
{
    public AudioSource Audio;

    public GameObject Slider;

    public GameObject Button;

    private SliderGestureControl _sliderControl;

    private IconToggler _iconToggler;

    public AudioType TypeAudio = AudioType.OGGVORBIS;

    void Start()
    {
        _sliderControl = Slider.GetComponent<SliderGestureControl>();
        _sliderControl.OnUpdateEvent.AddListener(ValueUpdated);
        Slider.SetActive(false);
        Button.SetActive(false);
        _iconToggler = Button.GetComponent<IconToggler>();
    }
}

In the Start method, we first grab a bunch of stuff. Note the fact that we not only turn off the slider control but also actually attach an event handler to that.

We continue with StartLoadMedia and LoadMediaFromUrl

protected override IEnumerator StartLoadMedia()
{
    Slider.SetActive(false);
    Button.SetActive(false);
    yield return LoadMediaFromUrl(MediaUrl);
}
private IEnumerator LoadMediaFromUrl(string url) { var handler = new DownloadHandlerAudioClip(url, TypeAudio); yield return ExecuteRequest(url, handler); if (handler.audioClip.length > 0) { Audio.clip = handler.audioClip; _sliderControl.SetSpan(0, Audio.clip.length); Slider.SetActive(true); Button.SetActive(true); _iconToggler.SetBaseState(); } }

The override from StartLoadMedia in this version turns off the whole UI while we are actually loading data, and turns it on when we are done loading. Since that fails when we load MP3, the MP3 player in the demo project disappears and on startup. The others one disappear too, in fact, but immediately appear again since we are loading small clips. This goes so fast you can’t even see it.

LoadMediaFromUrl not only executes the request and sets the downloaded clip to the Audio Souce, as we saw before, but we also set the span of the Slider Control between 0 and the length of the AudioClip in seconds. Easy, right?

Now the Update method, which as you know is called 60 times per second, is the trick to keeping the slider equal to the the current time of the clips that’s now playing:

protected override void Update()
{
    base.Update();
    if (Audio.isPlaying)
    {
        _sliderControl.SetSliderValue(Audio.time);
    }
    if (Mathf.Abs(Audio.time - _sliderControl.MaxSliderValue) < 0.1f)
    {
        Audio.Stop();
        Audio.time = 0;
        _iconToggler.SetBaseState();
        _sliderControl.SetSliderValue(0);
    }
}

Thus if the audio clip plays, the slider moves along. It’s not quite rocket science. If the clip has nearly finished playing, it is stopped and everything is set to the base state: the icon, the time of the audio clip, and the slider is set to 0 again.

And finally – remember that event handler we added to the OnValueUpdated event of the slider? Guess what:

private void ValueUpdated()
{
    Audio.time = _sliderControl.SliderValue;
}

It’s the opposite of the third line of Update – now we set the Audio time to the Slider value.

Conclusion

And that’s it. You can simply use some out-of-the-box components in the Mixed Reality Toolkit and/or it’s examples to build a simple but effective control to play audio. You can grab the demo project (it’s still the same) from here.

04 February 2018

Downloading audio files from Azure blob storage and playing them in Mixed Reality apps

Intro

All but the most trivial apps have some kind of back end. In the past, I have written about accessing all kinds of services from HoloLens and Mixed Reality apps, But apart from mere data, you can download all kind of media from external sources and use those in you Mixed Reality apps. This is highly useful if you want to change used media, use different files for like instructions depending on some factor. Or heck, some random background music. In fact, what I will be describing is not even Mixed Reality specific – the principle can be used in any Unity app, although the code that sits around to demonstrate the workings definitely only works for Mixed Reality.

Oh, and by the way – for us Microsoft geeks “back end” equals to “Microsoft Azure” – if it’s not for the competitive pricing, then for the way Microsoft makes it easy for developers to get going (heaven knows this was quite different in the early days). But to be clear: this will work with any backend that hosts files.

This post will be 2-part: the first part concentrates on the actual technique of downloading and playing audio files, the second part will explain the ‘floating audio player’ I built to make this easily demonstrable. The floating part in ‘floating player’ should be taken in the most literal way possible:

image

imageYou can find it in the demo project. You can just jump to there if you want to skip all my mumbling.

Global overview

If you open the project in Unity, you will not see one but three floating players, next to each other. They will all attempt to play on of my ringtones – an excerpt from the Doctor Who theme song (one is a nerd or one is not). But it will attempt to use one of three different audio formats – the well-known MP3, Ogg Vorbis, and WAVE audio format (aka ye good ole’ WAV).

image

To hear them, either build the app or just hit the play button. If you choose the second option, you will need to attach an XBox One controller to you PC, to steer the cursor and click on the buttons. You will notice the Ogg and the WAV file playing nicely. The mp3 one won’t. In fact, the player just disappears before you even can get to it. We will get to that later.

Using UnityWebRequest

In previous posts I have shown you either how to use the Unity WWW class, or how to resort to pure UWP code and use HttpClient/HttpRequestMessage. It seems like the WWW class is being deprecated, although I have no official information on that. However,the new (third) kid on the block seems to be UnityWebRequest.

This is a bit on an oddball. It uses a handler. Generally, a UnityWebRequest looks like this:

var request = UnityWebRequest.Get(url);
request.downloadHandler = handler;
yield return request.SendWebRequest();

The second line is optional – if the you don’t set the handler, it’s the default DownloadHandler class, which sports a “text” property you can query. This is very useful for accessing data services like an API app on Azure App Service. If you want to download audio files, you will need a DownloadHandlerAudioClip handler.

Putting it in code

To make this all work easily, I have created the following base class for download loading media:

using System.Collections;
using UnityEngine;
using UnityEngine.Networking;

public abstract class BaseMediaLoader : MonoBehaviour
{
    public string MediaUrl;

    private string _currentMediaUrl;

    protected virtual void Update()
    {
        if (_currentMediaUrl != MediaUrl)
        {
            _currentMediaUrl = MediaUrl;
            StartCoroutine(StartLoadMedia());
        }
    }

    protected abstract IEnumerator StartLoadMedia();

    protected IEnumerator ExecuteRequest(string url, DownloadHandler handler)
    {
        var request = UnityWebRequest.Get(url);
        request.downloadHandler = handler;
        yield return request.SendWebRequest();
    }
}

MediaUrl is an url that points to a place where the actual audio file can be downloaded – in this case, as SAS link to a file in my own Azure blob storage. Every time Update is called, it checks if the MediaUrl has been changed, and if so, it starts the StartLoadMedia in the background. ExecuteRequest is a helper method that can be used from a StartLoadMedia. Like this:

using System.Collections;
using UnityEngine;
using UnityEngine.Networking;

public class SoundPlaybackController : BaseMediaLoader
{
    public AudioSource Audio;

    public AudioType TypeAudio = AudioType.OGGVORBIS;

    protected override IEnumerator StartLoadMedia()
    {
        yield return LoadMediaFromUrl(MediaUrl);
    }

    private IEnumerator LoadMediaFromUrl(string url)
    {
        var handler = new DownloadHandlerAudioClip(url, TypeAudio);
        yield return ExecuteRequest(url, handler);
        if (handler.audioClip.length > 0)
        {
            Audio.clip = handler.audioClip;
        }
    }
}

This is a very reduced version of the SoundPlaybackController that is in the demo project, concentrating only on the actual downloading and playing of the data.

The override of StartLoadMedia simply calls StartLoadMedia, which proceeds to create a DownloadHandlerAudioClip handler. Now the odd thing is, this handler wants and url as well as the request. Why this is so, I have no idea. Also notice the fact you need to supply the handler with the type of audio you are going to download – there’s a AudioType enumeration for that.

If the audio has been downloaded successfully, you only have to set the “clip” property of an AudioSource to the handler’s “audioClip” property, and call the Audiosource’s “Play” method. And you are good to go.

Audio types are important

Since the TypeAudio property is public, the Unity editor makes a nice dropdown for us to select the type of audio:

image

I’ll be the first one to admit I haven’t heard of most of these file types, let alone know them. Actually, before I started, I only knew MP3 and WAV. I learned to know Ogg Vorbis. And for a good reason too. I already mentioned the fact the left (mp3) player disappears when you start the code. The SoundPlaybackController that I actually created (not the simple version above) hides the UI of the whole player while actually downloading the audio file. You might have noticed an error in the Unity status bar when you run the code. That actually says:

Streaming of 'mp3?st=2018-02-02t15%3a51%3a00z&se=2020-02-03t17%3a51%3a00z&sp=rl&sv=2017-04-17&sr=b&sig=kymvql5q1yyr%2bqilcxhfc3popwo56vd0ejibennldzw%3d' on this platform is not supported
UnityEngine.Networking.DownloadHandlerAudioClip:get_audioClip()
<LoadMediaFromUrl>c__Iterator0:MoveNext() (at Assets/App/Scripts/SoundPlaybackController.cs:34)
UnityEngine.SetupCoroutine:InvokeMoveNext(IEnumerator, IntPtr)

imageThe code in that player crashes, and never gets to showing the UI again. Well, that’s a bummer. Unity simply does not seem to support MP3 on ‘this platform’, which is apparently UWP. You can convert it to WAV, but unless you like to burn a lot of battery downloading stuff, you must be pretty much out of your mind doing so – Ogg Voribs is a much better option, as this simple list of files shows.

And Audacity, that good old workhorse of audio-artists and podcasters all around the globe makes conversion easy, so why not use it, right.

image

Conclusion

The very short version of this blog post: use UnityWebRequest and DownloadHandlerAudioClip to download an audio clip from Azure blob storage, and if you value your users’ bandwidth, their devices’ battery life, and your own sanity – use Ogg Vorbis audio files.

I have not tried all the other audio files types, simply because I did not have to do so – Ogg Vorbis works fine. WAV too, but is absurdly big compared to MP3 and Ogg Vorbis. In addition, I have no idea what those audio types are, how I should create/convert them, and why I should use them. To paraphrase Star Trek’s TOS medical officer Dr. Leonard “Bones” McCoy – I am a developer, not an audio engineer. 

Next time, I will explain the workings of the floating audio player around this code.

29 January 2018

Fantastic buttons and where to find them–in the Mixed Reality Toolkit

imageIntro

When I started making Mixed Reality apps (or better, HoloLens apps, as Mixed Reality as we learned to know only became available with the Fall Creators Update) I had the nerve to create my own ‘buttons’, as I assumed simple 3D ‘switches’ were too skeuomorphic. I assumed a kind of 3D variant of “that-what-not-should-be-called-Metro” would appear, and I took a shot at it myself with a red rotating spherical OK button. Let’s say the responses have been less then enthusiastic, especially from real designers.

Fast forward half a year, the Mixed Reality Toolkit now contains a number of standard UI components, and also have started to incorporate some elements of then Mixed Reality Design Labs project (that apparently has been abandoned, as its last commits are now 7 months old, and that’s only a README update – the code has not been touched for longer).

What’s in there?

If you copy the Mixed Reality Toolkit Samples UI elements into your project next to the Mixed Reality Toolkit itself (as I described here) you will find no less than 9 different push buttons that I think are usable

image

For your and my convenience, I labeled them with numbers.

  1. Prefab “Button”,  from HoloToolkit-Examples/Ux/Prefabs
  2. Prefab “RectangleButton” fromHoloToolkit/Ux/Prefabs/Buttons
  3. Prefab “MiniButton”, from HoloToolkit-Examples/Prototyping/Prefabs
  4. Prefab “ButtonHolographic”, from HoloToolkit-Examples/Ux/Prefabs
  5. Prefab “SquareButton”, fromHoloToolkit/Ux/Prefabs/Buttons
  6. Prefab “ButtonPush”, from HoloToolkit-Examples/Ux/Prefabs
  7. Prefab “CircleButton”, from HoloToolkit/Ux/Prefabs/Buttons
  8. Prefab “BasicButton”, from HoloToolkit-Examples/Prototyping/Prefabs
  9. Prefab “ButtonTraditional”, from HoloToolkit-Examples/Ux/Prefabs

There is some more stuff, but I don’t think they are very usable compared to these.

Interactive or not interactive, that’s the question

Some buttons (but not all) have an “Interactive” script component attached. Only the Button (1), the MiniButton(3) and the BasicButton (8) have this script attached. For the other 6 buttons, it’s simply a matter of dragging it onto the button from HoloToolkit-Examples/Ux/Scripts.

To see how that works, I have made a little script, that shows game object for about 1/10th of a second when the method “Click”  is called. It’s not quite rocket science

using System.Collections;
using UnityEngine;

public class Clicker : MonoBehaviour
{

    public GameObject ObjectToShow;

    private void Awake()
    {
        ObjectToShow.SetActive(false);
    }

    public void Click()
    {
        ObjectToShow.SetActive(true);
        StartCoroutine(HideAfterTimeout());
    }

    IEnumerator HideAfterTimeout()
    {
        yield return new WaitForSeconds(0.1f);
        ObjectToShow.SetActive(false);
    }
}

Then I add a 3DTextPrefab to the scene (let’s call that ClickText), attach the script about it, and then drag the ClickText object on top of the “ObjectToShow” field

image

Now if we want the MiniButton (3) to actually respond to these events…

image

…we have to click the + button under “On Select Events()”, select “Editor and Runtime”, drag the ClickText on the object box, and then select “Clicker.Click” from the dropdown (first select “Clicker”, then “Click” – it’s a menu with a submenu). Now if you have done everything right, the text “Click!” will briefly flash when you tap the MiniButton.

You can use this trick for all the buttons – if they don’t have the Interactive script by default, just drag it on them, and go from there. In the sample project, all buttons make the text appear.

Feedback sounds

In Mixed Reality environments, most users are new, so giving feedback that the actually selected something is key. This may happen trough sound. You are lazy (like me):  RectangleButton (2), MiniButton (3), ButtonHolographic (4) and CircleButton actually have these built-in. When you press those, they will make a click sound.

Visual feedback

Basically, there are two types of feedback involved with buttons. First, you want to make sure the user understand something is actually selectable. A nice way is to have the button light up in some way when the gaze cursor hits it. And guess what, apart from the BasicButton, they all have that built-in for free.

  • RectangleButton (2), MiniButton(3), SquareButton (5), ButtonPush (6) and CircleButton(7) become brighter if the gaze cursor strikes them.
  • ButtonHolographic (4) does not only become brighter, but text and icon on the button move a bit ‘forward’ when the gaze cursor strikes it. A pretty cool sight to see.
  • Button (1) and ButtonTraditional (9) not only become brighter, but also animate part of themselves outward in an animation that makes it very clear this can be clicked upon. They very much resemble real physical buttons, in that way.

When it comes to actually clicking the button:

  • RectangleButton (2), MiniButton(3),SquareButton (5), and CircleButton(7) flicker briefly when they are actually clicked
  • ButtonHolographic (4) does not flicker, but the text and it’s icon move slightly backward into the button. Note: default the ButtonHolographic only shows text, more on that later.
  • Button (1), ButtonPush (6) and ButtonTraditional (9) animate backwards and change color slightly when pushed, really mimicking the behavior of a physical button.

Setting Icons – the tricky way

SquareButton, RectangularButton and CircleButton

These at least show a default icon, called ObjectCollectionSphere

image

The UI suggests you can use select from a number of predefined icons using a dropdown. Now that is technically correct, but unfortunately that does not have any effect – the dropdown always jumps back to “ObjectCollectionSphere” and the icon never changes. Fortunately there’s another way to change the Icon.

imageFirst of all, we need an icon to use. I took this one as displayed to the right: I want to make this into a kind of “Play” button, as if to start music. First copy the image (play.png) into your project (be aware – the background is not black, but transparent). I typically make a folder “Textures” for these kind of images. Then simply select “Override Icon” and drag the Icon in “Icon Override”

image

And lo and behold:

image

One icon with button. The same procedure can be followed for SquareButton and CircleButton

HolographicButton

This is a bit harder, but not much. In theory, you should be able to using the same UI as with the previous buttons, but unless I miss something, I can’t get it to work. So I dived a little deeper, and found the sub prefab UIButtonSquareIcon. That has a Mesh renderer, but that’s turned off, and has no materials defined either

image

imageIf you just fix enable the Mesh Render, you just get an ugly magenta square – the color Unity typically uses when something is wrong.

So how do we this thing to display an icon on this button? suppose we want to make this a “play” button, with an icon like before. Make a new Material (I always use a folder “Materials” for that). After that,you only need to select the Fast Configurable shaded, set Rendering mode to transparent, and drag the Play image on top of the square I indicated.

image

And then it’s simply a matter of dragging the material in the first entry of the Materials list

image

imageAnd boom – a button with an Icon.Maybe I am doing it wrong, and am I not following the ideas of the developer of this button. However, this works, and that’s what counts.



Audio feedback

I have found out people also very much like audio feedback – since all computers have sounds and HoloLens even has the awesome Spatial Sound, so why not use it? RectangleButton (2), ButtonHolographic (4), SquareButton (5) and CircleButton (6) have a built-in click sound. It’s quite subtle, so maybe you want to replace it with a louder or different sound. You can change the click sound, and assign a lot more sounds to other events in these buttons, by finding the “Compound Button Sounds” script and setting a whole lot of other sounds. I would like to advise modesty: I don’t think Mixed Reality sounding like a pinball machine will be particularly convincing to business stakeholders – but there’s at least the possibility to tinker with sound on events:

image

Some bits, pieces and observations

  • Although all buttons work, I see no reason to use BasicButton (8) because it has no built-in feedback at all. I just mentioned it for the sake of completeness.
  • In some cases, I have noticed the click sounds not always works, or stops working halfway in the app. I have not yet been able to track down when and why this is happening.
  • Using older versions of Unity and/or of the Mixed Reality Toolkit I have observed the icons or texts disappear under some viewing angles. In the demo app, I have not been able to observe that. Maybe the newer version of Unity (I used 2017.3.0f3 this time) fixed it
  • The TraditionalButton has a bit of weird origin. If you select the button, you can see it’s origin is a bit above, a bit before, and quite a bit to the right of the actual graphics. This makes aligning it a bit of a challenge, but once you know that, it’s not a real problem

image

Conclusion

There’s quite a bit more of fun UI elements in the Mixed Reality Toolkit and it’s examples, and I intend to explore these a little bit more soon. Finally, I have to thank my colleague Edwin van Manen for coming up with the blog post title when I was describing it’s contents this morning. For those who don’t recognize it, it’s a reference to the movie “Fantastic Beast and Where To Find Them” that’s set in the Harry Potter ‘Universe’. Given the fact Mixed Reality in general and HoloLens in particular still feels like magic, it’s an awesome title indeed :D.

Code (although there is hardly any) can be found here.

27 December 2017

Centralized reusable audio feedback mechanisms for Mixed Reality apps

Intro – feedback is key

Although speech recognition in Mixed Reality apps is very good, sometimes the best recognition fails, or you slightly mispronounced something. Or the command you just said is recognized, but not applicable in the current state. Silence,  nothing happens and you wonder - did the app just not understand me, is the response slow or what? The result is always undesirable – users wait for something to happen and nothing does. They start to repeat the command and halfway the app executes the first command after all, or even worse – they start start shouting, which makes for a quite embarrassing situation (both for user and bystanders). Believe me, I’ve been there. So – it’s super important to inform your Mixed Reality app’s user  that a voice command has been understood and is being processed right away. And if you can’t process it, inform the user of that as well.

What kind of feedback?

Well, that’s basically up to you. I usually choose a simple audio feedback sound – if you have been following my blog or downloading my apps you are by now very familiar with the ‘pringggg’ sound I use in every app, be it an app in the Windows Store or one of my many sample apps on GitHub. If someone uses a voice command that’s not appropriate in the current context or state of the app, I tend to give some spoken feedback, telling the user that although the app has understood the command, can’t be executed now and if possible for what reason. Or prompt for some additional action. For both mechanisms I use a kind of centralized mechanism that uses my Messenger behaviour, that already has played a role in multiple samples.

Project setup overview

The hierarchy of the project is as displayed below, and all is does is showing the user interface on the right:

imageimage

If you say “Test command”, you will hear the “pringggg” sound I already described, and if you push the button the spoken feedback “Thank you for pressing this button”. Now this is rather trivial, but it only serves to show the principle. Notice, by the way, the button comes from the Mixed Reality Toolkit examples – I described before how to extract those samples and use them in your app. 

The Audio Feedback Manager and Spoken Feedback Manager look like this:

imageimage

The Audio Feedback Manager contains an Audio Source that just contains the confirmation sound, and a little script “Confirm Sound Ringer” by yours truly, which will be explained below. This sound is intentionally not spatialized, as it’s a global confirmation sound. If it was spatialized, it would also be localized, and the user would be able to walk away from confirmation sounds or spoken feedback, which is not what we want.

The Spoken Feedback Manager contains an empty Audio Source (also not spatialized), a Text To Speech Script from the Mixed Reality Toolkit, and a the “Spoken Feedback Manager’ script also by me.

ConfirmSoundRinger

using HoloToolkitExtensions.Messaging;
using UnityEngine;

namespace HoloToolkitExtensions.Audio
{
    public class ConfirmSoundRinger : MonoBehaviour
    {
        void Start()
        {
            Messenger.Instance.AddListener<ConfirmSoundMessage>(ProcessMessage);
        }

        private void ProcessMessage(ConfirmSoundMessage arg1)
        {
            PlayConfirmationSound();
        }

        private AudioSource _audioSource;
        private void PlayConfirmationSound()
        {
            if (_audioSource == null)
            {
                _audioSource = GetComponent<AudioSource>();
            }
            if (_audioSource != null)
            {
                _audioSource.Play();
            }
        }
    }
}

Not quite rocket science. If a message of type ConfirmSoundMessage arrives, try to find an Audio Source. If found, play the sound. ConfirmSoundMessage  is just an empty class with not properties or methods whatsoever – it’s a bare signal class.

SpokenFeedbackManager

Marginally more complex, but not a lot:

using HoloToolkit.Unity;
using HoloToolkitExtensions.Messaging;
using System.Collections.Generic;
using UnityEngine;

namespace HoloToolkitExtensions.Audio
{
    public class SpokenFeedbackManager : MonoBehaviour
    {
        private Queue<string> _messages = new Queue<string>();
        private void Start()
        {
            Messenger.Instance.AddListener<SpokenFeedbackMessage>(AddTextToQueue);
            _ttsManager = GetComponent<TextToSpeech>();
        }

        private void AddTextToQueue(SpokenFeedbackMessage msg)
        {
            _messages.Enqueue(msg.Message);
        }

        private TextToSpeech _ttsManager;

        private void Update()
        {
            SpeakText();
        }

        private void SpeakText()
        {
            if (_ttsManager != null && _messages.Count > 0)
            {
                if(!(_ttsManager.SpeechTextInQueue() || _ttsManager.IsSpeaking()))
                {
                    _ttsManager.StartSpeaking(_messages.Dequeue());
                }
            }
        }
    }
}

If a SpokenFeedbackMessage comes in, it’s added to the queue. In the Update method, SpeakText is called, which first checks if there are any messages to process, then checks if the TextToSpeech is available – and if so, it pops the message out of the queue and actually speaks it. The queue has two functions. First, the message may come from a background thread, and by having SpeakText called from Update, it’s automatically transferred to the main loop. Second, it prevents messages being ‘overwritten’ before they are even spoken.

The trade-off of course is that you might stack up messages if the user quickly repeats an action, resulting in the user getting a lot of talk while the action is already over.

On the Count > 0 in stead of any – apparently you are to refrain from using LINQ extensively in Unity apps, as this is deemed inefficient. It hurts my eyes to see it used this way, but when in Rome…

Wiring it up

There is a script SpeechCommandExecuter sitting in Managers, next to a Speech Input Source and a Speech Input Handler, that is being called by the Speech Input Handler when you say “Test Command”. This is not quite rocket science, to put it mildly:

public class SpeechCommandExecuter : MonoBehaviour
{
    public void ExecuteTestCommand()
    {
        Messenger.Instance.Broadcast(new ConfirmSoundMessage());
    }
}

As is the ButtonClick script, that’s attached to the ButtonPush:

using HoloToolkit.Unity.InputModule;
using HoloToolkitExtensions.Audio;
using HoloToolkitExtensions.Messaging;
using UnityEngine;

public class ButtonClick : MonoBehaviour, IInputClickHandler
{
    public void OnInputClicked(InputClickedEventData eventData)
    {
        Messenger.Instance.Broadcast(
            new SpokenFeedbackMessage { Message = "Thank you for pressing this button"});
    }
}

The point of doing it like this

Anywhere you now have to give confirmation or feedback, you now just need to send a message – and you don’t have to worry about setting up an Audio Source, a Text To Speech and wiring that up correctly. Two reusable components take care of that. Typically, you would not send the conformation directly from the pushed button or the speech command – you would first validate if the command can be processed in the component that holds the logic, and then give confirmation or feedback from there.

Conclusion

I hope to have convinced you of the importance of feedback, and I showed you a simple and reusable way of implementing that. You can find the sample code, as always, on GitHub.

26 December 2017

Short tip: using the UI components from the Mixed Reality Toolkit examples

The Mixed Reality Toolkit is the foundation for nearly all apps built for HoloLens and Windows Mixed Reality immersive head sets. It has a lot of indispensable components that make building these apps not so much a breeze, but al least a lot less tedious. It’s a bit sparse as far as actual UI components is concerned. They are now in the process of merging the awesome UI stuff from the Mixed Reality Design Labs project in it. This process is not ready by far, so far there’s only some simple buttons in it.

But if you look into this part of the code in the Mixed Reality Toolkit on GitHub , you will notice there being three folders:

image

The last one contains the actual Mixed Reality Toolkit, the 2nd the Unit Test which are not very interesting for us right now, but the first contains a whole lot of examples. And in those examples, a whole lot of nice UI controls like a host of buttons, a slider, a check box, toggles, toolbars, a very cool loader animation, helpers to draw lines and curves and probably a lot I have not noticed yet. There’s also a lot of demo scenes to show them off. But the Examples folder contains 114 MB of assets, almost doubling the size of already hefty MRKT itself.

Extracting only the UI elements is pretty simple:

  • Clone the project from Github
  • Copy the whole HoloToolkit-Examples into your projects Assets folder, next to the HoloToolkit
  • Delete everything but the these two folders:
    • HoloToolkit-Examples\UX
    • HoloToolkit-Examples\Prototyping
  • [optional] remove the “Scenes” subfolder from both the UX and the Prototyping folder. But maybe you should have look around first.

So you will end up with the following asset folder structure:

image

And then you can use cool stuff like this, as shown in the InteractiveButtonComponents scene:

image

Or these holographic buttons as you can find in ObjectCollectionExample scene (amongst a lot of other stuff)

image

And these samples of how to draw lines in the ‘air’ in the LineExamples scene

image

So far I have only used the icon buttons, but those are pretty cool in itself because they quite resemble the buttons in the Holographic shell, so your app’s user should immediately recognize what they are for.

No separate code this time, as all the code is in the MRTK repo itself. I do encourage your to root around through it, there’s quite some cool surprises in there.

06 December 2017

Getting your logos and splash screens right for Mixed Reality apps

Intro

Way too often I still see Mixed Reality apps with default (white) splash screens, app windows, or even worse – the default Unity Icon. When Mixed Reality was HoloLens-only and brand new, you kinda got away with that, as the WOW factor of the actual app eclipsed almost everything else, and the iconography did not matter much as there was only the Holographic Start menu. Also, although judging from my download numbers a fair number of HoloLenses are in use worldwide, the number is pretty limited compared to, say, PCs.

These days of leeway are over. With the advent of the Fall Creators Update, Mixed Reality apps can run on PCs with the right hardware – and in stead of a $3000+ device, a $400 headset will do. And as soon as I made an app available for immersive headsets, my download numbers exploded. Much more eyeballs so much more visibility, and in Windows, the flexible start menu and it’s different tile sizes makes omissions in the iconography glaringly visible. I also notice ‘ordinary’ users are a lot less forgiving. So, time to step up your game.

Note: I am currently using Unity 2017.2.0p2 MRTP5. Things may change. They tend to do that very much in this very new and exiting field :)

So what do you need?

At the very minimum, and icon of size 620x620 and a 1240x600 splash screen. 620x620 is the 310x310 icon at 200% scale. I tend to used 200% icons as they are the default sizes that are created by the Unity editor. I give the start 620x620 file the default name “Square310x310Logo.scale-200.png” and go from there.

In my Unity Assets folder I create a folder “UWPassets” and I drag the “Square310x310Logo.scale-200.png” in imagethere. I then proceed to make the following file from the first one

  • Wide310x150Logo.scale-200.png (620x310 pixels)
  • Square150x150Logo.scale-200.png (300x300 pixels)
  • Square71x71Logo.scale-200.png (142x142 pixels)
  • Square44x44Logo.scale-200.png (88x88 pixels)
  • StoreLogo.scale-100.png (50x50 pixels)

Do pay attention to the last one – that‘s 100% scale, so it’s actual size. To the right you see my actual icon.

These are just the standard sizes for UWP apps, if you are coming in from a Unity perspective you may not be familiar with them.

And then, as I said, you need the splash screen. This needs to be 1240x600. I have created this ‘awesome’ splash screen:

image

I suggest you take something that says something more about your app, but this will to for an example.

The result in the Unity editor:

image

Putting icons in the right place in Unity

This is not rocket science: Hit File/Build Settings and then the button “Player Settings”. This will open the “PlayerSettings” pane on the right side of your editor. First, you open the pane “Icon” and then the section “Store logo”. From the UWPAssets folder, drag the “StoreLogo.scale-100” into the top box.


image

You can leave the rest empty. Then skip all the other sections, scroll all the way down to the section “Universal 10 tiles and logos”. Open “Square 44x44 Logo”

image

Drag the “Square44x44Logo.scale-200.png” into the 4th box from the top, marked 200%. Then open section “Square 71x71 Logo” and drag  “Square71x71Logo.scale-200.png” in. Proceed similarly with the remaining three sizes till all the 200% boxes are filled.

Splash screen, splash screen and splash screen

Now dig this – there are actually three places to put splash screens in. Potentially you can have different splash images - I tend to use the same one everywhere. So scroll further down to the Splash Screen section. First up, on top, is the “Virtual Reality Splash Image”. You can drag your splash screen in there if you want.

image

Next up, a bit further down, is the Windows splash screen:

image

This is the most important one. This appears on your desktop when the app starts, it also is shown in a rectangle in Immersive Headsets during the app ‘transition scene’ (when your Cliff House is replaced by the app), and it’s displayed on your default app screen:

imageimage


And finally, there’s Windows Holographic Splash screen:

image

This is actually important in a HoloLens. As this does not have a ‘transition scene’, it shows a floating splash screen for a few seconds. You can see it in an immersive headset too, but usually for a very short time – most of the time you see it flash by in a split second before the actual app appears.

So where is the very first splash screen for? To be honest, I have no idea. Possibly it’s for other, non Windows Mixed Reality based apps.

Just tiny thing left

I tend to generate the app in a subfolder “App” in the root of the project. Typically, I place a .gitignore in there that, look like this:

*.*
*

true to it’s name, it ignores – everything. Because, well, it’s a generated app, right? But if you go into the generated app’s assets (App\SplashScreen\Assets) you will notice something odd:image

Although we nicely created an icon for every size, there’s still this pesky “Square44x44Logo.targetsize-24_altform-unplated.png” icon with the default logo that – interestingly, is 24x24 pixels big. So where is this used? Apparently in the task bar and – as I have noticed – in the developer portal. So although your customers won’t see it in the Store (I think), you will, it will clutter op your app list in the developer portal, and that’s annoying. I have found no way to actually create this from Unity, so I take the easy approach: I create yet another icon, now 24x24, overwrite that “Square44x44Logo.targetsize-24_altform-unplated.png” in  App\SplashScreen\Assets and add it to Git manually there. Unity won’t overwrite it, and if it get’s deleted or overwritten, I can always revert. You just need to remember to check the icon before actually submitting.

Conclusion

So now you know how to properly assign the minimum iconography and splash screens, and I let’s agree on no longer publishing stuff with default icons or empty splash screens to the store, right? ;)

Project code – although it does not actually do anything – can be found here. But it’s useful reference material.

12 November 2017

Finding the floor - and displaying holograms at floor level in HoloLens apps

Intro

Both of my HoloLens apps in the store ask the user to identify some kind of horizontal space to put holograms on, in one case Schiphol Airport, in the other case a map of some part of the world. Now you can of course use Spatial Understanding, which is awesome and offers very advanced capabilities, but requires some initial activity by the user. Sometimes, you just want to identify the floor or some other horizontal place - if only to find out how long the user is. This the actual code I wrote for Walk the World and extracted it for you. In the demo project, it displays a white plane at floor level.

Setting the stage.

We start with creating and empty project. The we proceed with importing the newest Mixed Reality Toolkit. If you have done that, you will notice the extra menu option "HoloToolkit" no longer appears, but it now says "Mixed Reality Toolkit". It still has three settings, but one of them has profoundly changed: the setting "Apply Mixed Reality Scene Settings" basically adds everything you need to get started:

imageimage

  • A "MixedRealityCameraParent" - the replacement for the HoloLensCamera that encapsulates a camera that will work both on HoloLens and immersive headsets.
  • A default cursor
  • An input manager that processes input from gestures, controllers or other devices depending on whether your app runs on a HoloLens or an immersive headset

Now I tend to organize stuff a little bit different, so after I have set up the scene I have like this:

image

I make an empty game object "HologramCollection" that will hold my holograms, and all the standard or none-graphic I tend to chuck in a game object "Managers". Notice I also added SpatialMapping. If you click the other two options in the Mixed Reality Toolkit/Configure menu our basic app setup is now ready to go.

Some external stuff to get

Then we proceed to import LeanTween and Mobile Power Ups Vol Free 1 from the Unity Store. Both are free. Note – the latter one is deprecated, but the one arrow asset we need from it still is usable. If you can't get it from the store anymore, just nick it from my code ;)

image

Recurring guest appearances

We need some more stuff I wrote about before:

  • The Messenger, although it’s internals have changed a little since my original article
  • The KeepInViewController class to keep an object in view – a improved version of the MoveByGaze class about my post about a floating info screen
  • The LookAtCamera class to keep an object oriented to the camera

Setting up the initial game objects

The end result should be this:

image

HologramCollection has three objects in it:image

  • A 3DTextPrefab "LookAtFloorText"
  • An empty game object "ArrowHolder"
  • A simple plane. This is the object we are going to project on the floor.

Inside the ArrowHolder we place two more objects:

  • A 3DTextPrefab "ConfirmText"
  • The "Arrows Green" prefab from the Mobile Power Ups Vol Free 1. That initially looks like on the right

So let's start at the top setting up our objects:

LookAtFloorText

imageThis is fairly easy. We scale it to 0.005 in all directions, enter the text "Please look towards the floor" in the text mesh, and set a couple of other parameters:

  • Z position is 1. So it will spawn 1 meter before you.
  • Character size to 0.1
  • Anchor to middle center
  • Font size to 480
  • Color to #00FF41FF (or any other color you like)

After that, we drag two components on it

  • KeepInViewController
  • LookAtCamera

imageWith the following settings:

  • Max Distance 2.0
  • Move time 0.8
  • Rotate Angle 180

This will keep the text at a maximum distance of 2 meters, or closer if you are looking at an object that is nearer, it will move in 0.8 seconds to a new position, and with an angle of 180 degrees it will always be readable to you.

image

ConfirmText

This is hardly worth it's own sub paragraph, but for the sake of completeness I show it's configuration.

Notice:

  • Y = 0.27, Z is 0 here.
  • This has only the LookAtCamera script attached to it, with the same settings. There is no LookAtCamera here.




imageArrows Green

This is the object nicked from Mobile Power Ups Vol Free 1.

  • Y = 0.1, I moved it a little upwards so it will always appear just above the floor
  • I rotated it over X so it will point to the floor.
  • I added a HorizontalAnimator to it with a spin time of 2.5 seconds over an absolute (vertical) axis so it will slowly spin around.

Plane

The actual object we are going to show on the floor. It's a bit big (default size = 10x10 meters), so we are going to scale it down a little:

image

And now for some code.

The general idea

Basically, there are two classes doing the work, the rest only is 'support crew.

  • FloorFinder actually looks for the floor, and controls a prompt object (LookAtFloorText) that prompts you - well, to look at the floor
  • FloorConfirmer displays an object that shows you where the floor is find, and then waits for a method to be called that either accepts or reject the floor. In sample the app, this is done by speech commands.
  • They both communicate by a simple messaging system

The message class is of course very simple:

using UnityEngine;

public class PositionFoundMessage
{
    public Vector3 Location { get; }

    public PositionFoundMessage(Vector3 location)
    {
        Location = location;
    }

    public PositionFoundStatus Status { get; set; }
}

With an enum to go with it indicating where in the process we are

public enum PositionFoundStatus
{
    Unprocessed,
    Accepted,
    Rejected
}

On startup, the FloorConfirmer hides it's confirmation object (the arrow with text). FloorFinder shows it's prompt object. When it detects a floor, it sends the position in a PositionFoundMessage message with status "Unprocessed". It also listens to PositionFoundMessages. If it receives one that has status "Unprocessed" (which is, in this sample, only sent by itself), it will disable itself and hide the prompt object (the text saying "Please look at the floor").

If the FloorConfirmer receives a PositionFoundMessage of status unprocessed "Unprocessed" it will show it's confirmation object on the location where the floor is detected. And then, as I wrote, it waits for it's Accept or Reject method being called. If Accept is called, it resends the PositionFoundMessage with status "Accepted" to anyone who might be interested - in this app, that's a simple class "ObjectDisplayer" that shows a game object that has been assigned to it on the correct height below the user's head. If the Reject method is called, FloorConfirmer resend the message as well - but with status Rejected. Which will wake up the FloorFinder again.

Finding the actual floor

using HoloToolkit.Unity.InputModule;
using HoloToolkitExtensions.Messaging;
using HoloToolkitExtensions.Utilities;
using UnityEngine;

public class FloorFinder : MonoBehaviour
{
    public float MaxDistance = 3.0f;

    public float MinHeight = 1.0f;

    private Vector3? _foundPosition = null;

    public GameObject LabelText;

    private float _delayMoment;

    void Start()
    {
        _delayMoment = Time.time + 2;
        Messenger.Instance.AddListener<PositionFoundMessage>(ProcessMessage);
#if !UNITY_EDITOR
        Reset();
#else
        LabelText.SetActive(false);
#endif
    }

    void Update()
    {
        if (_foundPosition == null && Time.time > _delayMoment)
        {
            _foundPosition = LookingDirectionHelpers.GetPositionOnSpatialMap(MaxDistance, 
GazeManager.Instance.Stabilizer); if (_foundPosition != null) { if (GazeManager.Instance.Stabilizer.StablePosition.y - _foundPosition.Value.y > MinHeight) { Messenger.Instance.Broadcast( new PositionFoundMessage(_foundPosition.Value)); PlayConfirmationSound(); } else { _foundPosition = null; } } } } public void Reset() { _delayMoment = Time.time + 2; _foundPosition = null; if(LabelText!= null) LabelText.SetActive(true); } private void ProcessMessage(PositionFoundMessage message) { if (message.Status == PositionFoundStatus.Rejected) { Reset(); } else { LabelText.SetActive(false); } } private void PlayConfirmationSound() { Messenger.Instance.Broadcast(new ConfirmSoundMessage()); } }

imageThis has three public properties - a prompt object (this becomes the text "please look towards the floor", a maximum distance to try to find the floor, and the minimum height the floor should be below the user's head. These properties are set as displayed on the right:

The Update method does all the work - if a position on the spatial map is found that's at least MinHeight below the user's head, then we might have found the floor, and we send out a message (with default status Unprocessed). The method below Update, ProcessMessage, actually gets that message too and hides the prompt text.

The helper method "GetPositionOnSpatialMap" in LookingDirectionHelpers simply tries to project a point on the spatial map at maximum distance along the viewing direction of the user. It's like drawing a line projecting from the users head ;)

public static Vector3? GetPositionOnSpatialMap(float maxDistance = 2,
                                               BaseRayStabilizer stabilizer = null)
{
    RaycastHit hitInfo;

    var headReady = stabilizer != null
        ? stabilizer.StableRay
        : new Ray(Camera.main.transform.position, Camera.main.transform.forward);

    if (SpatialMappingManager.Instance != null &&
        Physics.Raycast(headReady, out hitInfo, maxDistance, 
        SpatialMappingManager.Instance.LayerMask))
    {
        return hitInfo.point;
    }

    return null;
}

Is this the floor we want?

using HoloToolkitExtensions.Messaging;
using UnityEngine;

public class FloorConfirmer : MonoBehaviour
{
    private PositionFoundMessage _lastReceivedMessage;

    public GameObject ConfirmObject;

    // Use this for initialization
    void Start()
    {
        Messenger.Instance.AddListener<PositionFoundMessage>(ProcessMessage);
        Reset();
#if UNITY_EDITOR
        _lastReceivedMessage =  new PositionFoundMessage(new Vector3(0, -1.6f, 0));
        ResendMessage(true);
#endif
    }

    public void Reset()
    {
        if(ConfirmObject != null) ConfirmObject.SetActive(false);
        _lastReceivedMessage = null;
    }

    public void Accept()
    {
        ResendMessage(true);
    }

    public void Reject()
    {
        ResendMessage(false);
    }

    private void ResendMessage(bool accepted)
    {
        if (_lastReceivedMessage != null)
        {
            _lastReceivedMessage.Status = accepted ? 
                 PositionFoundStatus.Accepted : PositionFoundStatus.Rejected;
            Messenger.Instance.Broadcast(_lastReceivedMessage);
            Reset();
            if( !accepted) PlayConfirmationSound();
        }
    }

    private void ProcessMessage(PositionFoundMessage message)
    {
        _lastReceivedMessage = message;
        if (message.Status != PositionFoundStatus.Unprocessed)
        {
            Reset();
        }
        else
        {
            ConfirmObject.SetActive(true);
            ConfirmObject.transform.position = 
                message.Location + Vector3.up * 0.05f;
        }
    }


    private void PlayConfirmationSound()
    {
        Messenger.Instance.Broadcast(new ConfirmSoundMessage());
    }
}

A rather simple class - it disables it's confirm object at startup. If it gets a PositionFoundMessage, two things might happen:

  • If it's an Unprocessed message, it will activate it's confirm object (the arrow) and place it on the location provided inside the message (well, 5 cm above that).
  • For any other PositionFoundMessage, it will deactivate itself and hide the confirm object

If the Accept method is called from outside, it will resend the message with status Accepted for any interested listener and deactivate itself. If the reject method is called, it will send the message with status Rejected - effectively deactivating itself too, but waking up the floor finder again.

And thus these two objects, the FloorFinder and the FloorConfirmer can work seamlessly together while having no knowlegde of each other whatsover.

The final basket

For anything to happen after a PositionFoundMessage with status Accepted is sent, there need to also be something that actually receives it, and acts upon it. I places the game object it's attached to the same vertical position as the point it received - that is, 5 cm above it. It's not advisable to do it at the exact vertical floor position, as stuff might disappear under the floor.I have found horizontal planes are never smooth or, indeed - actually horizontal.

using HoloToolkitExtensions.Messaging;
using UnityEngine;

public class ObjectDisplayer : MonoBehaviour
{
    void Start()
    {
        Messenger.Instance.AddListener<PositionFoundMessage>(ShowObject);

#if !UNITY_EDITOR
        gameObject.SetActive(false);
#endif
    }

    private void ShowObject(PositionFoundMessage m)
    {
        if (m.Status == PositionFoundStatus.Accepted)
        {
            transform.position = new Vector3(transform.position.x, m.Location.y,
                transform.parent.transform.position.z) + Vector3.up * 0.05f;
            if (!gameObject.activeSelf)
            {
                gameObject.SetActive(true);
            }
        }
    }
}

imageThis script is dragged upon the plane that will appear on the floor

Wiring it all together

FloorFinder and FloorConfirmer sit together in the Managers object, but there's more stuff in the to tie all the knots:

  • The Messenger, for if there are messages to be sent, there should also be something to send it around
  • A Speech Input Source and a Speech Input Handler. Notice the last one calls FloorConfirmer's Accept method on "Yes", and the Reject method upon no.







Adding some sound


If yoimageu download the source code and run it, you might notice my trademark "pringggg" sound when the app does something. You might also have noticed various scripts sending ConfirmSoundMessage messages. In the Managers object there's another game object called "ConfirmSoundManager". It has an audio source and a ConfirmSoundRinger, that as you might expect is not too complicated:











using HoloToolkitExtensions.Messaging;
using UnityEngine;

public class ConfirmSoundRinger : MonoBehaviour
{
    void Start()
    {
        Messenger.Instance.AddListener<ConfirmSoundMessage>(ProcessMessage);
    }

    private void ProcessMessage(ConfirmSoundMessage arg1)
    {
        PlayConfirmationSound();
    }

    private AudioSource _audioSource;
private void PlayConfirmationSound() { if (_audioSource == null) { _audioSource = GetComponent<AudioSource>(); } if (_audioSource != null) { _audioSource.Play(); } } }

Conclusion

And that's it. Simply stare at a place below your head (default at least 1 meter), say "Yes" and the white plane will appear exactly on the ground. Or, as I explained, a little above it. Replace the plane by your object of choice and you are good to go, without using Spatial Understanding over complex code.

As usual, demo code can be found on GitHub.