LEARNing, Scheduling and speaking in other languages

LEARN support (#1039)

A small step but major change is in this release. For the first time we are enabling Mycroft users to provide data to our machine learning algorithms so we can improve the behavior of the system. For Mark 1 users, a LEARN menu item is now available, for Picroft and Github installs use a manual mycroft.conf entry.

When enabled, the 10 seconds of audio containing and preceding a “Hey Mycroft” wake-up are shared with us. We gather the previous 10 seconds to capture many of the false negatives, that is the times when you had to say ‘Hey Mycroft’ multiple times to wake it up.

We respect your privacy, so this is disabled by default. (Despite the developer accidentally checking it in his own ‘wake_word_upload.enabled’ setting originally! Fixed in #1088.) But we encourage you to help us make Mycroft better for all by enabling this simple option.

Translations

Our stated goal with the Mycroft Project is to create an AI for Everyone. This includes far more than just English speakers. We have built-in support for internationalization and several community members have already started work on support for:

• French. Thanks @lezohtml! (#1011)
• Portuguese. Thanks @00tiagopolicarpo00! (#1033)

Translation efforts have also begun in German and Dutch, look for those soon!

FEATURE: Event scheduling. (#1032)

Previously, Mycroft supported scheduling by basing a skill on the ScheduledSkill() class. However the need for repeating event is so common that this mechanism was limiting. Additionally, the need for a time-based “callback” is common (for example retrying after several seconds or periodically polling), and the common Thread-based method for implementing this was inefficient.

So now the MycroftSkill contains several event scheduling methods:

 def schedule_event(self, handler, when, data=None, name=None):
 def schedule_repeating_event(self, handler, when, frequency, data=None, name=None):
 def update_event(self, name, data=None):
 def cancel_event(self, name):

These capabilities replace the need for the ScheduledSkill(), which is now deprecated.

New/Updated APIs

• Allow update of local configuration (e.g. TTS engine). Thanks @JarbasAI! (#980)

• Add commonly used classes and methods to the mycroft and mycroft.util modules, simplifying import’ing them into a skill. (#1023)

  The mycroft module now includes:
  Classes: Api, MycroftSkill, FallbackSkill, Message
  intent_handler(), intent_file_handler(), adds_context(), removes_context()

  The mycroft.util module now includes:
  getLogger(), extract_datetime(), extractnumber(), normalize(), nice_number(), convert_number()

• Add concept of Priority Skills. Include ‘skill-pairing’ in priority list, delay network check until all priority skills are loaded (#1084)

FIXES

• ISSUE #847: Static method get() in ConfigurationManager was being called with an actual instance in several locations
• ISSUE #1001: Prevent multiple reloads of skills on startup
• ISSUE #1007: Add SkillContainer support after converse() changes
• ISSUE #1014: Skill unload/reload had memory leaks
• ISSUE #1019: The expect_response flag for speak_dialog() wasn’t being passed along correctly
• ISSUE #1022: Workaround an issue with Adapt and context management while determining proper behavior.
• ISSUE #1027: Change to io.open for better handling of character sets in dialog
• ISSUE #1036: Correct main() signature to restore support for the mycroft-cli-client shortcut in package builds
• ISSUE #1047: The start.sh help did not show the support for the ‘audio’ service
• The websocket client reconnection interval doubled with each failure to avoid bogging down a disconnected system, but wasn’t being reset upon successfull connection (#1080)
• The “I’m awake” message wasn’t being played, and remove ‘I didn’t catch that’ (#1087)

Misc

• Testing extensions:
  • Move test for intent_service to correct folder (#1006, #1063)
  • Extend core testing, e.g. skill loading, (#1003)
• Changed converse() mechanism to use skill_id instead of name (#1005)
• Skill system cleanup (#1012)
  • Documenting
  • Moving testing support methods out of main code
  • Import cleanup
• CLI improvements (#1038, #1081)
  • Add :find (or Ctrl+F) for searching logs
  • Add :history size to change the chat history window size
  • Invert PgUp/PgDn (it felt backwards)
  • Add ‘oldest’ and ‘newest’ indicators at the top and bottom of the log
  • Stop message scrolling when paging back through logs
  • Add heading to mic input indicator and limit size
  • Fixed default viseme filter and saving of filters.
• Clean-up shutdown of skills to allow skill shutdown() methods (#1042)
• Update Padatious to 0.2.2 (#1058)
• Refine dev_setup.sh (removing separate build_host_setup_debian.sh, build_host_setup_arch.sh, and build_host_setup_fedora.sh scripts); plus add notification to mycroft.sh when a rerun of dev_setup.sh is needed (#1058)
• Filtered ‘enclosure.mouth.display’ from the log by default (#1060)
• Removed wolframalpha as a default skill (#1096, #1097)
• MSM overhaul. (#1048, #1093)
  • Verifies skill is on ‘master’ branch before updating
  • Handle quoted search string properly
  • Improve error handling and use return codes to reflect results
  • Make install/remove output parser-friendly
  • Add support for ‘msm remove ’
• Volume change now happens via messagebus messages ‘mycroft.volume.increase’ and ‘mycroft.volume.decrease’ (#1089)
• Added a hotword factory, allowing integration of alternative wake-word tools such as Snowboy. Thanks @JarbasAI! (#1062)
• Improved handling of microphone muting (#1078)
• Add messagebus messages ‘mycroft.skill.handler.start’ and ‘mycroft.skill.handler.complete’ as skills are being processed (#1079)
• Added mycroft_use.sh script, allowing easy switching between the stable, unstable and github installs on Raspbian (including Mark 1, Picroft) platforms (#1073)
• Add support for bitrates other than 16000 Hertz in Mycroft STT (#1054)
• Add Context manager sanity checks (#1071)
• Rework configuration loading, reducing unneeded remote configuration loads during testing (#1064)

Time for us to talk: Padatious, Conversation, Context and Fallbacks

This release provides many new tools and techniques for Skills to implement more conversational interaction with speakers.

FEATURE: Conversational support for skills. (#925)

The most recently used skills now have an opportunity to preview all utterances before they hit the intent system. Skills remain "active" for 5 minutes after last use. Active skills get a preview in the order of activation -- most recent first -- and if they can consume the utterance or ignore it. If consumed, processing stops. If ignored, the next most recent skill gets a shot at it. Finally, if no skill consumes it the intent system takes over running as usual.

A skill achieves this by implementing the converse() method, e.g.

    def def converse(self, utterances, lang="en-us"):
        if .... :
            return True  # handled, consume utterance
        else:
            return False  # not for this skill, pass it along

Thanks JarbasAI!

FEATURE: Padatious, an examples based intent system (#939)

Padatious is an alternative mechanism to Adapt for defining intent handlers. Rather than defining explicit vocabulary and structures, you provide example sentences.

__init__.py:

from mycroft.skills.core import MycroftSkill
class TomatoSkill(MycroftSkill):
    def __init__(self):
        MycroftSkill.__init__(self)

    @intent_file_handler('what.is.intent')
    def handle_what_is(self, message):
        self.speak('A tomato is a big red thing')

    @intent_file_handler('do.you.like.intent')
    def handle_do_you_like(self, message):
        tomato_type = message.get('type')
        if tomato_type is not None:
            self.speak("Well, I'm not sure if I like " + tomato_type + " tomatoes.")
        else:
            self.speak('Of course I like tomatoes!')

vocab/en-us/what.is.intent:

What would you say a tomato is?
What's a tomato?
Describe a tomato.
What defines a tomato?

vocab/en-us/do.you.like.intent:

Are you fond of tomatoes?
Do you like tomatoes?
What are your thoughts on tomatoes?

Are you fond of {type} tomatoes?
Do you like {type} tomatoes?
What are your thoughts on {type} tomatoes?

FEATURE: Skill Context manager (#934)

Adding a basic context system. This allows conversations where a skill can get information from previous interactions. This includes:

• Creating an intent handler which includes a requirement for a context when registered. This context can then be provided by other intent handlers using the self.add_context() method. (See examples below)
• Injecting context such as location or time for other skills to use. This provides the possibility to have conversations like this

Hey Mycroft, what's the time in London
10:42
Right, what's the weather over there?
Rain with a temperature of 14 degrees C'''

The Context manager will by default remember 3 entries back and with a time limit of 2 minutes.

See PR #934 for more details.

FEATURE: Intent multi-fallback system (#938)

This new system now allows multiple skills to register as fallbacks for utterances that weren't caught by conversation or explicit intent handlers. This includes general knowledge skills -- such as Wikipedia or IMDB -- as well as more creative things like AIML. Fallbacks are executed in self-defined priority order until one claims to have handled the utterance.

By default Mycroft now falls back to Wolfram for general knowledge queries and AIML to provide personality. Such as:

Hey Mycroft, I'm tired
Better get some rest

Skills implement this by registering and defining a handle_fallback() method.

    def initialize(self):
        self.register_fallback(self.handle_fallback, 80)

    def handle_fallback(self, message):
        if 'what is' in message.data['utterance']:
            self.speak_dialog('the answer is always 42')
            return True
        return False

See PR #938 for more details.

FIXES

• ISSUE #958 - Make visime stream end together with audio
• ISSUE #969 - Audio service doesn't log audio service messages
• Fixed naming of file for wake-word recording/debugging mechanism Thanks @reginaneon! (#971)
• ISSUE #978 - PulseAudio integration could have muting issue in rare circumstances
• ISSUE #982 - Fix # core calculation issue for Mimic CPU. Thanks @el-tocino! (#983)

New/Updated APIs:

• Add mycroft.util.download.download(url, dest, complete_action=None). If specified, complete_action function will be called once the download is complete. (#990)
• The .build() is no longer necessary when defining intents (#996)

Misc

• Extensions to developer setup tools:
  • Prevent unnecessary mimic compile in dev_setup.sh (#957)
  • Mimic clone from dev_setup.sh uses depth of 1 to reduce size (#988)
  • Add mycroft-core to virtualenv path (#992)
• MSM cleanup: (#973, #972)
  • Improve help message, documenting all options. Thanks @el-tocino!
  • Make the DEFAULT_SKILLS list readable (wrapped lines)
  • Prefix messages with ERROR or WARNING
  • Cache list of skills (instead of hitting web repeatedly)
  • Improve format of output messages
  • Sort skills from 'list' and 'search' commands
  • Return result codes (was always 0)
  • Modified return codes, they can't be > 255. 301 and 303 are now 201 and 203
• Add DeviceAPI.is_subscriber() to query for user account status. (#991)
• Testing enhancements (#840):
  • Add messagebus testing
  • Add test/skills/integrationtest framework to support self-testing of skills. More to come soon!

AudioService, DisplayManager, Skill Settings

It's Christmas in July! Lots of goodies to unwrap in this one.

FEATURE: Audio Service (#433)

Major unification of audio handling in Mycroft. The AudioService now manages all sound output, including text to speech, music files and audio streams. This enables several new capabilities and behaviors:

• Audio will lower/pause and resume when Mycroft speaks
• Skills can queue-up multiple audio files/streams for playback
• Unified playback skip/pause/resume/stop
• Audio can be redirected to external players besides the local speakers. Currently supports VLC and Mopidy. (Kodi and Chromecast on the way!) "Hey Mycroft, play the news in the kitchen"

See the NPR News (https://github.com/MycroftAI/skill-npr-news) and Playback Control (https://github.com/MycroftAI/skill-playback-control) skills for examples of interacting with the service.

FEATURE: Skill Graphic Output / DisplayManager (#838)

Skills can draw to the Mark 1 face (or future enclosures) by sending encoded or PNG graphics.

Graphics are sent via EnclosureAPI.mouth_display_png(). Graphics are automatically converted to two colors if necessary.

Also created the concept of an active skill which owns the display. Say "Hey Mycroft, what time is it?" and see https://github.com/MycroftAI/skill-date-time for an example of a Skill drawing to the display when it is 'inactive'.

FEATURE: Skill Settings Web UI (#933)

Skill developers can now add a settingsmeta.json file into their skill. Then mycroft-core will automatically upload the file to home.mycroft.ai, allowing users to input the settings via a web UI. Entered values are sent back as a settings.json file that can be used by the skill.

Example settingsmeta.json:

{
   "identifier": "PandoraSkill",
   "name": "Pandora",
   "skillMetadata": {
      "sections": [
         {
            "name": "Login",
            "fields": [
               {
                  "name": "email",
                  "type": "email",
                  "label": "Email",
                  "value": ""
               },
               {
                  "name": "password",
                  "type": "password",
                  "label": "Password",
                  "value": ""
               }
            ]
         }
      ]
   }
}

FIXES

• ISSUE #960: Stop command would stop stopping if any skill handler for mycroft.stop has an error (and stops). (#961)
• ISSUE #950: Upgrade netifaces in requirements.txt to netifaces==0.10.6 (#951)
• FIX: Using an undeclared "logger" could crash during exception handling (#963,#964)

MSM Enhancements

• Clarified MSM return codes, distinguishing 'duplicate' and 'missing' skill requests (thanks el-tocino!) (#945)
• FIX: Correct user ownership on /opt/mycroft subdirectories when needed (#947)

Misc

• Add a standard commit message template (#952)
• Restrict requirement to mock==2.0.0 (#949)
• Upgrade requirement to netifaces==0.1.6 (#951)

The big five-Oh!

This is the 50th release for mycroft-core!

Fixes

• ISSUE #887: Mycroft Skills Manager (MSM) was failing on a Mark 1 with skills that had missing library requirements. (PR #907)
• Fixed missing period in cached WAV files from the Text to Speech system. They were missing a period before the file extension, e.g. "cachedfilewav" instead of "cachedfile.wav". (PR #921)
• Fixed bug that prevented the "expect_response" mechanism from triggering. (PR #920)

Developer and API enhancements

• Added official support for running Kaldi locally as a STT server. (PR #878)
• Added script for updating dev environment. (PR #914)
• Added new intent fallback system for dealing with unhandled intents. Multiple Skills can now register as fall-back handlers with a self-assigned priority. Handlers are invoked by priority until one handles the intent. (PR #899)
• Skills are no longer need to provide a name parameter when calling their base initializer. It now defaults to the classname of the skill. (PR #918)

Developer and API Enhancements

Fixes

• ISSUE #874: Handling $WORKON_HOME in start.sh. Thanks nealmcb! (PR #875)
  Mycroft wasn’t recognizing the “wake up” phrase to come out of deep sleep mode entered by saying “Hey Mycroft, go to sleep”. (PR #888)
• ISSUE #871: Automatic skill update was not occurring. Files automatically generated by Python were making the system think the skill was being worked on by a developer. Now the Mycroft Skill Manager (MSM) behavior properly updates skills that have not been modified, but stops updates if a developer intentionally makes changes to the code for a skill. (PR #873, #890, #894)
• Hack to deal with duplicate mycroft-skill process on a Mark 1 unit after upgrade. A reboot is forced to resolve.

Developer and API enhancements

• Blacklisted skills are now read from a list in the config. The value is under skills.blacklisted_skills. (PR #857)
• Added support for “mycroft.sh restart” to mycroft.sh utility. (PR #868)
• Documented the Message class. (PR #869 and #748)
• Intents now can be defined via a decorator. This replaces the need to create an initialize() method purely for connecting the skill handlers. Here is an example:Decorator-style intent definition

   @intent_handler(IntentBuilder('HelloWorldIntent').require('hello').build())
   def handle_hello_world(self, message):
       self.speak('hello!')

Old-style intent definition

   def initialize(self):
       hello_intent = IntentBuilder("HelloWorldIntent").require("hello").build()
       self.register_intent(hello_world_intent, self.handle_hello_world)

   def handle_hello_world(self, message):
       self.speak('hello!')

The old style is still supported and will remain available. But both methods are functionally identical and we believe the new method is simpler and clearer. (PR #865)

• Modularization of Wake Word (similar to TTS and STT). This will allow alternatives to PocketSphinx for wake word spotting. (PR #876)
• Text to Speech is now unified in an independent thread. This groups multiple-sentence replies and improves handling of for the “Stop” action and button on a Mark 1. This architectural change will also allow things like applying the audio caching mechanism for TTS engines besides the default Mimic, and viseme (mouth shape) support for other TTS providers. (PR #804, #892)
• Removed unused / unneeded libraries from the requirements.txt (PR #806)
• Added dev_setup.sh support for using packaged mimic (eliminating the lengthy compile time). (PR #882)
  Errors encountered during skill update were failing with no indication of why in the logs. (PR #879)
• Deprecated version mechanism was still being used in setup tools. (PR #829)

Developer support tools, etc.

Fixes

• PID and locking mechanism was causing issues on auto-reloads. (PR #837)

Mark 1

• Added support for menu item SSH > DISABLE and switch to using 'systemctl'. (PR #744, #745)
• Tweaks to wifi-setup disconnect detection (PR #861)

General

• Added a docker setup script (PR #775)
• Improved multi-core support in dev_setup.sh script (PR #823)
• Refinement of mycroft.sh operation (PR #835)
• CLI client refinements to support backspace on some terminals and suppress TTS output that occasionally corrupted the screen. (PR #833)
• Some JSON loads didn't use load_commented_json(), causing issues if devs commented some config files. (PR #845)
• Added local caching for remote Mycroft config (PR #828)
• SkillSettings could miss updates to lists or dicts (PR #844)
• Added remove_all_listeners() to EventEmitter on the websocket (PR #854)
• MSM enhancements. Added support for Picroft installs, installing list of skills, etc. (PR #813)

Minor bug fixes

Fixes

• Issue #807: MSM could report successful install even after a failure. (PR #813)
• Minor onboarding refinements: no timeout for wifi setup, preventing from talking over itself in certain situations. (PR #817)
• Rework of has_been_paired() to deal with a changing identity file. (PR #825)

Added 'out of box' experience

Mark 1

• Created an 'out of box' experience. This walks a new Mark 1 user through the process of setting up seamlessly and provides a simple introduction to using the device. (PR #808, #811)

Enhancements

• Added mycroft.util.parse.extract_number(). This will support parsing a number out of phrases like "a cup and a half of milk" or "one and three fourths cup", extracting 1.5 and 1.75 respectively. (PR #793)
• Added mycroft.util.format.nice_number(). This converts numbers into natural spoken phrases, e.g. 3.5 into "three and a half." (PR #786)

Fixes

• Catching of KeyboardInterrupt by a signal handler was causing issues when shutting down services as a developer. Ctrl+C didn't work properly anymore. (PR #798)
• Scaled the duration of wake-up-word listening based on number of phonemes. This allows longer wake-up phrases to work properly. (PR #794)
• Fixed broken link in CONTRIBUTING.md (PR #784)
• Added check of exit status after calling the Mycroft Skills Manager (MSM) to verify skill install success. (PR #782)

Changing wifi-setup password

• Changed the wifi-setup client password to '12345678' to make easier to type, based on user testing feedback. (PR #780)

SSL fix and rework of startup

Fixes

• Issue #746 : Google STT was broken by a typo (PR #750)
• Issue #705: Unable to talk to the API over SSL on some Linux distros. Switching to pyOpenSSL version 16.2.0, resolved it. (PR #751)

Interaction refinements

• Prevented double-announcements when connecting to (PR #766)
• Corrected announcement when SSH is enabled/disabled -- reboot is not required anymore. (PR #765)
• Reworked startup sequence of events and documented behavior. This corrected occasionally not prompting user to connect to the internet and unified the startup prompt and prompts needed if internet is lost later. (PR #764, 760)
• Simplified the package manifest files by adding wildcards to handle resource files under 'mycroft/res/' (PR #763)