Update on Tesla .ssq files

Sometime back, I noticed the car downloaded a large file (5.1 GB) which was a .ssq file. I hadn’t heard of a ssq file, and was curious on what this was.

I researched a little and as it turns out, a .ssq file is a compressed file system which is often used in an embedded Linux system, where storage size might be a area of concern. This file-system is called SquashFS, and is usually used on a read-only mode.

SquashFS is interesting, as it lets one mount the file-system directly and is distributed as a kernel source patch – which makes it easy to daisy chain and use it other regular Linux tools.

SquashFS tools are useful to mount and create a SquashFS file-system. As shown below, I can mount the downloaded file, using unsquashfs.

unsquashfs to mount a SquashFS file-system

I think it is known that Tesla uses Valhalla for their maps and this file is the updated maps data. Valhalla, is a open source routing library which is using OpenStreetMap. Valhalla, also incorporates the traditional travelling salesman problem which is a non-deterministic polynomial problem.

When extracted and mounted, we see the following directory structure; each of these folders (and files therein) are in fact the tiles that make up the maps (next time in the car, when you zoom in or out or search of a non-cached location, notice carefully on how it is loading and you can just about make out the tiles – it is quick and easy to miss). And it is these tiles that is used for routing as part of the navigation. 

Tiled based routing is supposed to be beneficial – it uses less memory (the graph can be decomposed much easier, with a smaller set of it loaded in memory), cahce-able, easier to manage (update-able), etc. We can see a glimpse on how the routing and calculation happen on a tile basis below.

tiles based routing

When, extracted we see there are three levels of hierarchy (0, 1, and, 2). In the file-system these are shown as directories, but there is a method to the madness.

  • Level 0 – these contain edges pertaining to roads that are considered highway / freeway / motorway roads. These are stored as 4 degree tiles.
  • Level 1 – contains roads that are at a arterial level and are saved in 1 degree tiles.
  • Level 2 – these are local roads and are saved as 0.25 degree tiles.

For example, the world at Level 0 would look like what we are seeing in the image below. And Pennsylvania can be seen below that; Level 0 colored in light blue, Level 1 in light green, and finally Level 2 in light red (which might not be obvious with the translucency).

World Level 0 tiles
Pennsylvania Level 0, 1, and 2 tiles

So, to use this, one can use a few helper functions to get the exact tile to load and vice-versa. For example using the GPS coordinate of 41.413203, -73.623787 (which is just outside of Brewster, NY), loading Level 2 (via the get_title_2 function) would give us the structure of /2/000/756/425.gph using which we know which tile to load.

Helper function (in python) that help obtain levels, tile ids, tile lists, lat/long coordinates, etc. from an intersecting box.

valhalla_tiles = [{'level': 2, 'size': 0.25}, {'level': 1, 'size': 1.0}, {'level': 0, 'size': 4.0}]
LEVEL_BITS = 3
TILE_INDEX_BITS = 22
ID_INDEX_BITS = 21
LEVEL_MASK = (2**LEVEL_BITS) - 1
TILE_INDEX_MASK = (2**TILE_INDEX_BITS) - 1
ID_INDEX_MASK = (2**ID_INDEX_BITS) - 1
INVALID_ID = (ID_INDEX_MASK << (TILE_INDEX_BITS + LEVEL_BITS)) | (TILE_INDEX_MASK << LEVEL_BITS) | LEVEL_MASK

def get_tile_level(id):
  return id & LEVEL_MASK

def get_tile_index(id):
  return (id >> LEVEL_BITS) & TILE_INDEX_MASK

def get_index(id):
  return (id >> (LEVEL_BITS + TILE_INDEX_BITS)) & ID_INDEX_MASK

def tiles_for_bounding_box(left, bottom, right, top):
  #if this is crossing the anti meridian split it up and combine
  if left > right:
    east = tiles_for_bounding_box(left, bottom, 180.0, top)
    west = tiles_for_bounding_box(-180.0, bottom, right, top)
    return east + west
  #move these so we can compute percentages
  left += 180
  right += 180
  bottom += 90
  top += 90
  tiles = []
  #for each size of tile
  for tile_set in valhalla_tiles:
    #for each column
    for x in range(int(left/tile_set['size']), int(right/tile_set['size']) + 1):
      #for each row
      for y in range(int(bottom/tile_set['size']), int(top/tile_set['size']) + 1):
        #give back the level and the tile index
        tiles.append((tile_set['level'], int(y * (360.0/tile_set['size']) + x)))
  return tiles

def get_tile_id(tile_level, lat, lon):
  level = filter(lambda x: x['level'] == tile_level, valhalla_tiles)[0]
  width = int(360 / level['size'])
  return int((lat + 90) / level['size']) * width + int((lon + 180 ) / level['size'])

def get_ll(id):
  tile_level = get_tile_level(id)
  tile_index = get_tile_index(id)
  level = filter(lambda x: x['level'] == tile_level, valhalla_tiles)[0]
  width = int(360 / level['size'])
  height = int(180 / level['size'])
  return int(tile_index / width) * level['size'] - 90, (tile_index % width) * level['size'] - 180

Tesla has actually open-sourced their implementation of Valhalla, which is based on C++. This still seems like an active project, but parts of the code haven’t been updated for a while.

Whilst I haven’t tried to set this up myself, it seems quite simple. Below are the instructions to get this going on Ubuntu or Debian (I think Mac is also supported, but needs a little different dependency set).

#below are the dependencies needed
sudo add-apt-repository -y ppa:valhalla-core/valhalla
sudo apt-get update
sudo apt-get install -y autoconf automake make libtool pkg-config g++ gcc jq lcov protobuf-compiler vim-common libboost-all-dev libboost-all-dev libcurl4-openssl-dev libprime-server0.6.3-dev libprotobuf-dev prime-server0.6.3-bin
#if you plan to compile with data building support, see below for more info
sudo apt-get install -y libgeos-dev libgeos++-dev liblua5.2-dev libspatialite-dev libsqlite3-dev lua5.2
if [[ $(grep -cF xenial /etc/lsb-release) > 0 ]]; then sudo apt-get install -y libsqlite3-mod-spatialite; fi
#if you plan to compile with python bindings, see below for more info
sudo apt-get install -y python-all-dev

#install with the following
git submodule update --init --recursive
./autogen.sh
./configure
make test -j$(nproc)
sudo make install

There you have it – we know now what the .ssq files are and how they are used. Just need more time to get it going and play with it – perhaps another project for another time. 🙂

Tesla v9 API endpoints

In case you haven’t been following the news, Tesla is in the process of releasing the new firmware beta. I think many folks online are super interested in new autopilot upgrades.

I reverse engineered the associated app and there are certainly a few new end points exposed, as outlined below. Need time to now figure out more details on this and what they entail. Also need time to see what changes in the existing code and json (data structure). 

Is it interesting to go noodle on this, and see the associated calls. This outlines all the products as of today

{
  "AUTHENTICATE": {
    "TYPE": "POST",
    "URI": "oauth/token",
    "AUTH": false
  },
  "REVOKE_AUTH_TOKEN": {
    "TYPE": "POST",
    "URI": "oauth/revoke",
    "AUTH": true
  },
  "PRODUCT_LIST": {
    "TYPE": "GET",
    "URI": "api/1/products",
    "AUTH": true
  },
  "VEHICLE_LIST": {
    "TYPE": "GET",
    "URI": "api/1/vehicles",
    "AUTH": true
  },
  "VEHICLE_SUMMARY": {
    "TYPE": "GET",
    "URI": "api/1/vehicles/{vehicle_id}",
    "AUTH": true
  },
  "VEHICLE_DATA": {
    "TYPE": "GET",
    "URI": "api/1/vehicles/{vehicle_id}/data",
    "AUTH": true
  },
  "WAKE_UP": {
    "TYPE": "POST",     
    "URI": "api/1/vehicles/{vehicle_id}/wake_up",
    "AUTH": true
  },
  "UNLOCK": {
    "TYPE": "POST",
    "URI": "api/1/vehicles/{vehicle_id}/command/door_unlock",
    "AUTH": true
  },
  "LOCK": {
    "TYPE": "POST",
    "URI": "api/1/vehicles/{vehicle_id}/command/door_lock",
    "AUTH": true
  },
  "HONK_HORN": {
    "TYPE": "POST",
    "URI": "api/1/vehicles/{vehicle_id}/command/honk_horn",
    "AUTH": true
  },
  "FLASH_LIGHTS": {
    "TYPE": "POST",
    "URI": "api/1/vehicles/{vehicle_id}/command/flash_lights",
    "AUTH": true
  },
  "CLIMATE_ON": {
    "TYPE": "POST",
    "URI": "api/1/vehicles/{vehicle_id}/command/auto_conditioning_start",
    "AUTH": true
  },
  "CLIMATE_OFF": {
    "TYPE": "POST",
    "URI": "api/1/vehicles/{vehicle_id}/command/auto_conditioning_stop",
    "AUTH": true
  },
  "CHANGE_CLIMATE_TEMPERATURE_SETTING": {
    "TYPE": "POST",
    "URI": "api/1/vehicles/{vehicle_id}/command/set_temps",
    "AUTH": true
  },
  "CHANGE_CHARGE_LIMIT": {
    "TYPE": "POST",
    "URI": "api/1/vehicles/{vehicle_id}/command/set_charge_limit",
    "AUTH": true
  },
  "CHANGE_SUNROOF_STATE": {
    "TYPE": "POST",
    "URI": "api/1/vehicles/{vehicle_id}/command/sun_roof_control",
    "AUTH": true
  },
  "ACTUATE_TRUNK": {
    "TYPE": "POST",
    "URI": "api/1/vehicles/{vehicle_id}/command/actuate_trunk",
    "AUTH": true
  },
  "REMOTE_START": {
    "TYPE": "POST",
    "URI": "api/1/vehicles/{vehicle_id}/command/remote_start_drive",
    "AUTH": true
  },
  "CHARGE_PORT_DOOR_OPEN": {
    "TYPE": "POST",
    "URI": "api/1/vehicles/{vehicle_id}/command/charge_port_door_open",
    "AUTH": true
  },
  "CHARGE_PORT_DOOR_CLOSE": {
    "TYPE": "POST",
    "URI": "api/1/vehicles/{vehicle_id}/command/charge_port_door_close",
    "AUTH": true
  },
  "START_CHARGE": {
    "TYPE": "POST",
    "URI": "api/1/vehicles/{vehicle_id}/command/charge_start",
    "AUTH": true
  },
  "STOP_CHARGE": {
    "TYPE": "POST",
    "URI": "api/1/vehicles/{vehicle_id}/command/charge_stop",
    "AUTH": true
  },
  "MEDIA_TOGGLE_PLAYBACK": {
    "TYPE": "POST",
    "URI": "api/1/vehicles/{vehicle_id}/command/media_toggle_playback",
    "AUTH": true
  },
  "MEDIA_NEXT_TRACK": {
    "TYPE": "POST",
    "URI": "api/1/vehicles/{vehicle_id}/command/media_next_track",
    "AUTH": true
  },
  "MEDIA_PREVIOUS_TRACK": {
    "TYPE": "POST",
    "URI": "api/1/vehicles/{vehicle_id}/command/media_prev_track",
    "AUTH": true
  },
  "MEDIA_NEXT_FAVORITE": {
    "TYPE": "POST",
    "URI": "api/1/vehicles/{vehicle_id}/command/media_next_fav",
    "AUTH": true
  },
  "MEDIA_PREVIOUS_FAVORITE": {
    "TYPE": "POST",
    "URI": "api/1/vehicles/{vehicle_id}/command/media_prev_fav",
    "AUTH": true
  },
  "MEDIA_VOLUME_UP": {
    "TYPE": "POST",
    "URI": "api/1/vehicles/{vehicle_id}/command/media_volume_up",
    "AUTH": true
  },
  "MEDIA_VOLUME_DOWN": {
    "TYPE": "POST",
    "URI": "api/1/vehicles/{vehicle_id}/command/media_volume_down",
    "AUTH": true
  },
  "SEND_LOG": {
    "TYPE": "POST",
    "URI": "api/1/logs",
    "AUTH": true
  },
  "RETRIEVE_NOTIFICATION_PREFERENCES": {
    "TYPE": "GET",
    "URI": "api/1/notification_preferences",
    "AUTH": true
  },
  "SEND_NOTIFICATION_PREFERENCES": {
    "TYPE": "POST",
    "URI": "api/1/notification_preferences",
    "AUTH": true
  },
  "RETRIEVE_NOTIFICATION_SUBSCRIPTION_PREFERENCES": {
    "TYPE": "GET",
    "URI": "api/1/vehicle_subscriptions",
    "AUTH": true
  },
  "SEND_NOTIFICATION_SUBSCRIPTION_PREFERENCES": {
    "TYPE": "POST",
    "URI": "api/1/vehicle_subscriptions",
    "AUTH": true
  },
  "DEACTIVATE_DEVICE_TOKEN": {
    "TYPE": "POST",
    "URI": "api/1/device/{device_token}/deactivate",
    "AUTH": true
  },
  "CALENDAR_SYNC": {
    "TYPE": "POST",
    "URI": "api/1/vehicles/{vehicle_id}/command/upcoming_calendar_entries",
    "AUTH": true
  },
  "SET_VALET_MODE": {
    "TYPE": "POST",
    "URI": "api/1/vehicles/{vehicle_id}/command/set_valet_mode",
    "AUTH": true
  },
  "RESET_VALET_PIN": {
    "TYPE": "POST",
    "URI": "api/1/vehicles/{vehicle_id}/command/reset_valet_pin",
    "AUTH": true
  },
  "SPEED_LIMIT_ACTIVATE": {
    "TYPE": "POST",
    "URI": "api/1/vehicles/{vehicle_id}/command/speed_limit_activate",
    "AUTH": true
  },
  "SPEED_LIMIT_DEACTIVATE": {
    "TYPE": "POST",
    "URI": "api/1/vehicles/{vehicle_id}/command/speed_limit_deactivate",
    "AUTH": true
  },
  "SPEED_LIMIT_SET_LIMIT": {
    "TYPE": "POST",
    "URI": "api/1/vehicles/{vehicle_id}/command/speed_limit_set_limit",
    "AUTH": true
  },
  "SPEED_LIMIT_CLEAR_PIN": {
    "TYPE": "POST",
    "URI": "api/1/vehicles/{vehicle_id}/command/speed_limit_clear_pin",
    "AUTH": true
  },
  "SCHEDULE_SOFTWARE_UPDATE": {
    "TYPE": "POST",
    "URI": "api/1/vehicles/{vehicle_id}/command/schedule_software_update",
    "AUTH": true
  },
  "CANCEL_SOFTWARE_UPDATE": {
    "TYPE": "POST",
    "URI": "api/1/vehicles/{vehicle_id}/command/cancel_software_update",
    "AUTH": true
  },
  "POWERWALL_ORDER_SESSION_DATA": {
    "TYPE": "GET",
    "URI": "api/1/users/powerwall_order_entry_data",
    "AUTH": true
  },
  "POWERWALL_ORDER_PAGE": {
    "TYPE": "GET",
    "URI": "powerwall_order_page",
    "AUTH": true,
    "CONTENT": "HTML"
  },
  "ONBOARDING_EXPERIENCE": {
    "TYPE": "GET",
    "URI": "api/1/users/onboarding_data",
    "AUTH": true
  },
  "ONBOARDING_EXPERIENCE_PAGE": {
    "TYPE": "GET",
    "URI": "onboarding_page",
    "AUTH": true,
    "CONTENT": "HTML"
  },
  "REFERRAL_DATA": {
    "TYPE": "GET",
    "URI": "api/1/users/referral_data",
    "AUTH": true
  },
  "REFERRAL_PAGE": {
    "TYPE": "GET",
    "URI": "referral_page",
    "AUTH": true,
    "CONTENT": "HTML"
  },
  "MESSAGE_CENTER_MESSAGE_LIST": {
    "TYPE": "GET",
    "URI": "api/1/messages",
    "AUTH": true
  },
  "MESSAGE_CENTER_MESSAGE": {
    "TYPE": "GET",
    "URI": "api/1/messages/{message_id}",
    "AUTH": true
  },
  "MESSAGE_CENTER_COUNTS": {
    "TYPE": "GET",
    "URI": "api/1/messages/count",
    "AUTH": true
  },
  "MESSAGE_CENTER_MESSAGE_ACTION_UPDATE": {
    "TYPE": "POST",
    "URI": "api/1/messages/{message_id}/actions",
    "AUTH": true
  },
  "MESSAGE_CENTER_CTA_PAGE": {
    "TYPE": "GET",
    "URI": "messages_cta_page",
    "AUTH": true,
    "CONTENT": "HTML"
  },
  "AUTH_COMMAND_TOKEN": {
    "TYPE": "POST",
    "URI": "api/1/users/command_token",
    "AUTH": true
  },
  "SEND_DEVICE_KEY": {
    "TYPE": "POST",
    "URI": "api/1/users/keys",
    "AUTH": true
  },
  "DIAGNOSTICS_ENTITLEMENTS": {
    "TYPE": "GET",
    "URI": "api/1/diagnostics",
    "AUTH": true
  },
  "SEND_DIAGNOSTICS": {
    "TYPE": "POST",
    "URI": "api/1/diagnostics",
    "AUTH": true
  },
  "BATTERY_SUMMARY": {
    "TYPE": "GET",
    "URI": "api/1/powerwalls/{battery_id}/status",
    "AUTH": true
  },
  "BATTERY_DATA": {
    "TYPE": "GET",
    "URI": "api/1/powerwalls/{battery_id}",
    "AUTH": true
  },
  "BATTERY_POWER_TIMESERIES_DATA": {
    "TYPE": "GET",
    "URI": "api/1/powerwalls/{battery_id}/powerhistory",
    "AUTH": true
  },
  "BATTERY_ENERGY_TIMESERIES_DATA": {
    "TYPE": "GET",
    "URI": "api/1/powerwalls/{battery_id}/energyhistory",
    "AUTH": true
  },
  "BATTERY_BACKUP_RESERVE": {
    "TYPE": "POST",
    "URI": "api/1/powerwalls/{battery_id}/backup",
    "AUTH": true
  },
  "BATTERY_SITE_NAME": {
    "TYPE": "POST",
    "URI": "api/1/powerwalls/{battery_id}/site_name",
    "AUTH": true
  },
  "BATTERY_OPERATION_MODE": {
    "TYPE": "POST",
    "URI": "api/1/powerwalls/{battery_id}/operation",
    "AUTH": true
  },
  "SITE_SUMMARY": {
    "TYPE": "GET",
    "URI": "api/1/energy_sites/{site_id}/status",
    "AUTH": true
  },
  "SITE_DATA": {
    "TYPE": "GET",
    "URI": "api/1/energy_sites/{site_id}/live_status",
    "AUTH": true
  },
  "SITE_CONFIG": {
    "TYPE": "GET",
    "URI": "api/1/energy_sites/{site_id}/site_info",
    "AUTH": true
  },
  "HISTORY_DATA": {
    "TYPE": "GET",
    "URI": "api/1/energy_sites/{site_id}/history",
    "AUTH": true
  },
  "BACKUP_RESERVE": {
    "TYPE": "POST",
    "URI": "api/1/energy_sites/{site_id}/backup",
    "AUTH": true
  },
  "SITE_NAME": {
    "TYPE": "POST",
    "URI": "api/1/energy_sites/{site_id}/site_name",
    "AUTH": true
  },
  "OPERATION_MODE": {
    "TYPE": "POST",
    "URI": "api/1/energy_sites/{site_id}/operation",
    "AUTH": true
  },
  "TIME_OF_USE_SETTINGS": {
    "TYPE": "POST",
    "URI": "api/1/energy_sites/{site_id}/time_of_use_settings",
    "AUTH": true
  },
  "STORM_MODE_SETTINGS": {
    "TYPE": "POST",
    "URI": "api/1/energy_sites/{site_id}/storm_mode",
    "AUTH": true
  },
  "SEND_NOTIFICATION_CONFIRMATION": {
    "TYPE": "POST",
    "URI": "api/1/notification_confirmations",
    "AUTH": true
  },
  "NAVIGATION_REQUEST": {
    "TYPE": "POST",
    "URI": "api/1/vehicles/{vehicle_id}/command/navigation_request",
    "AUTH": true
  }
}

Setting up your own Model 3 “keyfob” – using a IoT Button

Some time ago, I talked about my Tesla Model 3 “keyfob” which essentially uses a Amazon IoT button to call some of Tesla API’s and “talk” to the car. This for me, is cool as it allows my daughter to unlock, and lock the car at home. And of course it is a bit geeky, and allowing one to play with more things. 🙂

Since publishing this, I was surprised how many of you ping me asking on details on how they can did this for themselves. Given the level of interest, I thought I will document this and outline the steps here. I do have to warn you, that this would be a little long – it entails getting a IoT Button configured, and then the code deployed. Before you get started, and if you aren’t techy, I would recommend to go through the post completely, so you get a sense of what is needed.

At a high level, below are the steps that you need to go through to get this working. And this might seem cumbersome and a lot but it is not that difficult. Also if you prefer you can follow the official AWS documentation online here.

  1. Create a AWS Login (if you have a existing Amazon.com login, you can use the same one if you prefer)
  2. Order a IoT Button
  3. Register the IoT Button in the AWS Registry (this is done via the AWS console)
  4. Create (and activate) a device certificate
  5. Create a IoT security policy
  6. Attach the IoT security policy (from the previous step) to the device certificate created earlier
  7. Attach the IoT security policy (now with the associated certificate) to the IoT button
  8. Configure the IoT button
  9. Deploy some code – this is done via a server-less function (also called a Lambda function) – this is the code that gets executed
  10. Test and Deploy
  11. Enjoy the Fob! 🙂

Step 1 – Get the IoT Button

Of course you need to get a IoT Button; I got the AWS IoT Button (2nd Generation) which is what I would recommend.

Step 2 – Login to AWS IoT Console

Open AWS home page and login with your amazon.com credentials. Of course if you don’t have a Amazon.com account, then you want to click in sign up on the top right corner, to get this started.

AWS Login

After I login, I see something similar to the screenshot below. Your exact view might differ a little.

AWS Console

I recommend to change the region to one closer to you. To do this, click on the region on the top right corner and choose a region that is physically closest to you. In the longer run this would help with latency issues between you clicking the button and the car responding. For example in my case, Oregon makes most sense.

AWS Region Selection

Once you have a AWS account setup, login to the AWS IoT console or on the AWS page in the previous step, scroll down to IoT Core as shown in the screenshot below.

AWS Console

Step 3 – Register IoT Button

Next step would be to register your IoT button – which of course means you physically have the button with you. The best way to register is to follow the instructions here. I don’t see much sense in trying to replicate that here.

Note: If you are not very technical, or comfortable, it might be best to use either the “AWS IoT Button Dev” app which is available both on the Apple Store (for iOS) and Google play (for Android).

Once you have registered a button (it doesn’t matter what you call it) – it will show up similar to the screenshot below. I only have one device listed.

List of IoT things

Step 4 – Create a Device Certificate

Next, we need to create and activate a certificate for the device. Without this, the button won’t work. The certificate (which is a X.509 certificate) protects the communication between the button and AWS.

For most people, the one-click certification creation that AWS has, is probably the way to go. To get to this, on the AWS IoT console, click on Secure and then choose Certificates on the left if not already selected as shown below. I already have a certificate that you can see in the screenshot below.

Certificates

If you need to create a certificate, click on the Create button on the top right corner, and choose one of the options shown in the image below. In most cases you would want to use the One-click certificate option.

Certificate creation options

NOTE: Once you create a Certificate, you get three files (these are the keys) that you need to download and keep safe. The certificate itself can be downloaded anytime, but the private and the public keys CANNOT be retrieved again after you close this page. It is IMPORTANT that you download these and save them in a safe place.

Certificate Keys

Once you have these downloaded then click on Activate on the bottom. And you should see a different certificate number than what you are seeing here. And don’t worry I have long deleted what you are seeing on this screen. 🙂

You can also see these in the developer guide on AWS documentation.

Step 5 – Create a IoT Security Policy

Next step is go back to the AWS IoT Console page and click on Policies under Security. This is used to create a IoT policy that you will need to attach to the certificate. Once you have a policy created, then it will look something like the screenshot below.

IoT Policies

To create a policy, click on Create (or you might be prompted automatically if you don’t have one). On the create screen, in the Name you can enter anything that you prefer. I would suggest naming this something that you can remember and differentiate if you will have more than one button. In my case I named it as the same thing as my device.

  • In the policy statements for Action enter “iot:Connect” – without the quotes, but this is case sensitive so make sure you match is exactly.
  • For the Resource ARN enter “*” (again without the quotes) as shown below.
  • And finally for the effect, make sure “Allow” is checked.
  • And click on Create at the bottom.
IoT Policy Creation

After this is created this you will see the policies listed as shown below. You can see the new one we just created with “WhateverNameYouWillRecognize“. You can also see these and more details on the developer documentation – Create a AWS IoT Policy.

IoT Policies

Step 6 – Attach a IoT Policy

Next step is to attach the policy that is just created to the certificate created earlier. To do that, click on Secure and Certificates on the left, and then click on the three dots (called ellipses) on the top right of the Certificate you created earlier. From the new menu that you get, choose “Attach Policy” as shown below.

Attach Policy to Certificate

From the resulting menu, select the policy that you had created earlier and select Attach. Using a sensible name that you would recognize would be helpful. You can also see these details on the developer documentation.

Attach Policy to Certificate

Step 7 – Attach Certificate to IoT Device

Next step is to attach the certificate to the IoT device (or thing). A device must have a certificate, a private key and a root CA certificate to authenticate with AWS. Amazon also recommends to attach a device certificate to the device – this probably isn’t helpful right now, but might be in the future if you start playing with this more.

To do this, select the certificate under Security on the left, and same as the previous step, by click on the three dots on the top right corner, select “Attach thing”.

Attach Certificate

And from the next screen select the IoT button that you registered earlier, and select “Attach”.

Attach Certificate

Step 8 – Configure IoT Button

To validate that everything is setup correctly – the certificate needs to be associated with a policy, and a thing (the IoT button in our case). So on the Certificates menu on the left, select your certificate by clicking on it (not the three dots this time – but rather the name). You will see a new screen that shows the details of the certificate as shown below.

Certificate Details

And on the new menu on the left, if you click on Policies you should see the policy you created, and the Things should have the IoT button you created earlier.

Once all of this is done the next step is to configure the device. You can see more detailed steps on this on the developer guide here.

  • KEY TIP: The documentation doesn’t make it too obvious, but as part of configuring – the device (IoT Button) will become an access point that you will need to connect to and upload the certificates and key you created earlier. You cannot do this from a phone and it is best done from a desktop/laptop that has wifi network. Whilst these days all laptops will have a wifi network card, that isn’t necessarily true for desktops. So use a machine which has a wifi that you can temporarily connect to the access point that the IoT device creates.
  • Note this is only needed for getting the device configured to authenticate for AWS, and get on your Wifi network; once that is done you don’t need to do this.
  • Once you have configured the device as outlined (https://docs.aws.amazon.com/iot/latest/developerguide/configure-iot.html) then continue to the next step.

Step 9 – Deploy some code

At last we are starting to get the interesting part – a lot of what we were doing until now, was getting the button configured and ready.

Now that you have a IoT button configured and registered, the next step is to deploy some code. For this you need to setup a Lambda function using the AWS Lambda Console.

When you login, click on Create Function. On the Create function screen, choose the Blueprints option as shown below. You can see some of these in the developer documentation here.

Create Function screen

Step 10 – Blueprint Search

On the Blueprints search box (which says Filters by tags), type in “button” (without quotes) and press enter. You should see an option called “iot-button-email” as shown below, select that and click configure on the bottom right corner.

IoT Button filter

Step 11 – Basic Information

On the next screen that says “Basic information”, enter the details as shown below. The names should be meaningful for you to remember. Roles can be reused across other areas, for now you can use a simple name something like “unlockCar” or “unlockCarSomeName” if you have more than one vehicle. The policy template should already be populated and you shouldn’t need to do anything else.

Function basic information

For the 2nd half – AWS IoT Trigger, select the IoT type as “IoT Button” and enter your device serial number as outlined in the screenshot below.

IoT Trigger

It won’t hurt to download these certificate and keys in addition to the ones created separately and save them in different folders. And for the Lambda function code, it doesn’t matter on the template code as we will be deleting it all. At this point that will be read-only and you won’t be able to modify anything – as shown in the screen shot below.

Lambda function

And finally scrolling down more, you will see the environment variables. Here is where you need to specify your Tesla credentials to it to be able to use create the token and call the Tesla API. For that you need the following two variables: TESLA_EMAIL and TESLA_PASS. These case sensitive so you need to enter them as is. And then finally click on Create function.

Environment Variables

Step 12 – Code upload

Once you create a function, you will see something like the screen below. In my case the function is called “unlockSquirty” which is what you are seeing. This is divided in to two parts – when on the Configuration page. The top part is the designer that visually shows you what inputs are the triggers that execute the function, and then what it outputs to on the right hand side.  And below the designer is the editor where one can edit the code inline or upload a zip file with the code.

In the function code section, on the first drop down in the left (Code entry type) select upload a .zip file.

And on the next screen upload the function package that you can download from here.

  • Make sure the Runtime is Node.js 8.10
  • Keep the Handler as the default.
  • Double check your Environment variable contain TESLA_EMAIL, and TESLA_PASS.

And scroll down and in the Basic settings, change the timeout to 1 minute. We run thus asynchronously and adding a little buffer would be better. You can leave all the other settings at their default. If your network might be iffy you can make this 2 mins.

Environment Settings

Step 13 – Code Publish

Once you have entered all of this, click on Save on the top right corner and then publish new version. Finally once it is published you will be able to see the code show up as shown in the screenshot below.

Again, a single click will unlock the car, a double-click would lock it, and a long press (holding it for 2-3 seconds) would open the charge port door.

And here is the code:

 var tjs = require('teslajs');

 var username = process.env.TESLA_EMAIL;
 var password = process.env.TESLA_PASS;

 exports.handler = (event, context, callback) => 
 {
  tjs.loginAsync(username, password).done(function(result) 
  {
   var token = JSON.stringify(result.authToken);
   if (token)
    console.log("Login Succesful!");

   var options = 
   {
    authToken: result.authToken
   };

   tjs.vehicleAsync(options).done(function(vehicle) 
   {
    console.log("Vehicle " + vehicle.vin + " is: " + vehicle.state);
    var options = 
    {
     authToken: result.authToken,
     vehicleID: vehicle.id_s
    };

    if(event.clickType == "SINGLE")
    {
     console.log("Single click, attempting to UNLOCK");
     tjs.doorUnlockAsync(options).done(function(unlockResult) 
     {
      console.log("Doors are now UNLOCKED");
     });
    }
    else if(event.clickType == "DOUBLE")
    {
     console.log("Double click, attempting to LOCK");
     tjs.doorLockAsync(options).done(function(lockResults) {
      console.log("Doors are now LOCKED");
     });              
    }
    else if(event.clickType == "LONG")
    {
     console.log("Long click, attempting to CHARGE PORT");
     tjs.openChargePortAsync(options).done(function(openResult) {
      console.log("Charge port is now OPEN");
     });              
    }    
   });
  });
 };

Generating Tesla authentication token – cURL script

I did write a simple Windows (desktop) app called TeslaTokenGenerator, for those who wanted to create authentication tokens for their Tesla, and use with 3rd party apps/data loggers. 

TeslaTokenGenerator can also create a cURL script for you to use, if you prefer not wanting to install anything. It is easy to find this online, but some of you have pinged me to get more details on this. So, I have the script below that you can use. Once you copy this, you will need to update your Tesla account login details (email and password) and run it in a console (command line) and it will all the same API’s to create the token, which then you can save.

curl -X POST -H "Cache-Control: no-cache" -H "Content-Type: multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW" -F "grant_type=password" -F "client_id=81527cff06843c8634fdc09e8ac0abefb46ac849f38fe1e431c2ef2106796384" -F "client_secret=c7257eb71a564034f9419ee651c7d0e5f7aa6bfbd18bafb5c5c033b093bb2fa3" -F "email=YOUR-TESLA-LOGIN-EMAIL@SOMEWHERE.COM" -F "password=YOUR-TESLA-ACCOUNT-PASSWORD" "https://owner-api.teslamotors.com/oauth/token"

You can see the screenshots of this below too – one in Windows, and another in Linux (well Bash on Windows, but it is real Linux).

Windows Tesla Auth Token Generator

If you have a Tesla, and are using (or wanting to use) 3rd party tools or data loggers, the one think they of course need is to authenticate your details with Tesla. A simple, but insecure way is to use your Tesla credentials – and surprisingly many people just happily share and use this.

I wasn’t comfortable doing this – after-all, they have access to your account where you can control a lot of things. Also, there are a few online tools that can generate the auth token, but again I wasn’t comfortable, as I did not know what they saved, or what they did not. 🙂

So, I wrote a simple Windows app that can allow you to generate a auth token that you can save. The application itself is simple. You enter your Tesla credentials, click on Generate Token and can save the generated token. 

Tesla Token Generator Application

To test, if the generated token is working – click on the Test Token button. If everything is working as expected, you will see a list of vehicles that is associated with your account.

If you prefer to use the cURL script, click on the Generate cURL, will generate this and copy it to your clipboard. And it works across operating systems as you can see below (Windows, and Linux), but should also work on Mac.

I do intent to open source this, so folks can have a look at the code, and the Tesla REST APIs. Until then you can download the setup from here.

Leave a comment if you have any issues or any requests.

Update: v1.0.1 Published with minor updates. You can download from the same link above. This adds the revoke screen and some house keeping.

Download Build deck and video (2018)

Just as last year, I wrote a PowerShell script using which you can download the PowerPoint decks, and, videos from Microsoft Build’s conference, instead of streaming it (or manually download it one by one). You can choose if you want the decks, or the videos, or both. For the videos you can choose the desired resolution (Low, Medium, High) – of course the higher the resolution, the more space is needed. The script also downloads the description and if there is a session image (if there is one).

A few points to note:

  • The slides only once downloaded is ~10GB and with videos (high-resolution), the size goes up to 90.5 GB. So make sure you have enough space.
  • By default the download location is “C:\build-2018\”; you can change this to whatever you want, but make sure there is a trailing backslash. Think of this as the ‘base’ folder.
  • For each session a sub-folder with the session name will be created in the ‘base’ folder setup in the previous step.
  • If a file already exists, it will be skipped.
  • As each file is downloaded, it save it in the root folder and once the download is complete, only then moves it in the relevant subfolder.
  • If a download fails for some reason to retry it, delete the ‘left over’ file(s) in the base folder and then run the script again. The script itself will ‘eat’ the exception and move on to the next file.
  • The video quality parameter is 1 for Low, 2 for Medium, and 3 for High (default).

And if you read through, the script is quite self-explanatory.

# Comments that you should read, before you kick this off. Yes, seriously. 🙂
# 1. Setup the folder where to download using the parameters outlined below
# 2. Loop through and get the slides first
# 3. Finally, loop through and get the videos last
 
 param (
    [string]$path = "C:\build-2018\",
    [switch]$sessionvideo = $true,
    [int][ValidateRange(1,3)]$videoquality = 3,
    [switch]$sessiondeck = $true
 )

[Environment]::CurrentDirectory=(Get-Location -PSProvider FileSystem).ProviderPath 
$rss = (new-object net.webclient)

#Filenames might get long, so keep this short!
#$downloadlocation = "D:\build-2018"

if (-not (Test-Path $path)) {
	Write-Host "Folder $fpath dosen't exist. Creating it..." 
    New-Item $path -type directory 
}

set-location $path

if($sessiondeck) {  
# Grab the Slides RSS feed - Build 2018
$slides = ($rss.downloadstring("http://s.ch9.ms/events/build/2018/rss/slides")) 

    # ********** download the decks **********
    try { 
        foreach($item in $slides.rss.channel.item) {  
            $code = $item.comments.split("/") | select -last 1     
    
            # Get the url for the pptx file
            $urlpptx = New-Object System.Uri($item.enclosure.url)
        
            # make the filename readable
            $filepptx = $code + " - " + $item.title.Replace(":", "-").Replace("?", "").Replace("/", "-").Replace("&lt;", "").Replace("|", "").Replace('"',"").Replace("*","").Replace("’","'").Replace("â€","").Replace("'NEW SESSION'","").Replace("™","").Replace("œ","")
            $filepptx = $filepptx.substring(0, [System.Math]::Min(120, $filepptx.Length))
            $filepptx = $filepptx.trim()
            $filejpg = $filepptx
 
            $filepptx = $filepptx + ".pptx" 
            $filejpg = $filejpg + "_960.png"
 
     
            $folder = $item.title.Replace(":", "-").Replace("?", "").Replace("/", "-").Replace("&lt;", "").Replace("|", "").Replace('"',"").Replace("*","").Replace("’","'").Replace("â€","").Replace("'NEW SESSION'","").Replace("™","").Replace("œ","")
            $folder = $folder.substring(0, [System.Math]::Min(100, $folder.Length))
            $folder = $folder.trim()
             
            if (-not (Test-Path $folder)) { 
                Write-Host "Folder $folder doesnt exist. Creating it..."  
                New-Item $folder -type directory 
            }
     
            # Make sure the PowerPoint file doesn't already exist
            if (!(test-path "$path\$folder\$filepptx")) {   
                # Echo out the file that's being downloaded
                $filepptx
                $wc = (New-Object System.Net.WebClient)  
 
                # Download the pptx file
                Invoke-WebRequest $urlpptx -OutFile $path\$filepptx
 
                # download the jpg but don't want to break if this doesn't exist; hence the nested try blocks
                try {
                    if($item.thumbnail -ne $null) {
                        $urljpg = New-Object System.Uri($item.thumbnail.url)

                        if (!(test-path "$path\$filejpg")) {    
                            $wc.DownloadFile($urljpg, "$path\$folder\$filejpg")
                        }
                    }
                }
                catch {
                    Write-Host "Image (jpeg) $filejpg doesn't exist ... eating the exception and moving on ..."
                }
         
                mv $filepptx $folder 
                #mv $filejpg $folder 
            } #endif
            else {
                Write-Host "PPTX: $filepptx exist; skipping download."  
            }

            #try and get the sessions details
            try {
                $descriptionFileName = "$($path)\$($folder)\$($Code.trim()).txt"

                if (!(test-path "$descriptionFileName")) {
                    $OutFile = New-Item -type file $descriptionFileName -Force  
                    $Content = "Title: " + $item.title.ToString().trim() + "`r`n" + "`r`n" + "Presenter: " + $item.creator + "`r`n" + "`r`n" + "Summary: " + $item.summary.ToString().trim() + "`r`n" + "`r`n" + "Link: " + $item.comments.ToString().trim() 

                    #some categories are missing; so need to eat the exception
                    #this is a hack and not very elegant
                    try {
                        if($item.category -ne $null) {
                            $Content = $Content + "`r`n" + "`r`n" + "Category: " + $item.category.ToString().trim().Replace("+"," ")
                        }
                    }
                    catch {
                        #do nothing; eat the exception
                    }

                    add-content $OutFile $Content
                }
            }
            catch {
                $ErrorMessage = $_.Exception.Message
                $FailedItem = $_.Exception.ItemName
                Write-host "\t" $ErrorMessage + "\n" + $FailedItem
            }
        } #end-loop foreach
 
        Write-host "*************** Downloading all the decks complete ***************"
    }
    catch
    {
        Write-host "Oops, could not find any slides."
        $ErrorMessage = $_.Exception.Message
        $FailedItem = $_.Exception.ItemName
        Write-host "\t" $ErrorMessage + "\n" + $FailedItem
    }
} #download session-deck 

# ********** download the videos **********
# if you don't want the video but only the slides just comment all the code below in the foreach loop
try { 
	# check for video download
	if($sessionvideo) {
        switch ($videoquality) {
            1 {$video = ($rss.downloadstring("http://s.ch9.ms/events/build/2018/rss/mp3")); break}
            2 {$video = ($rss.downloadstring("http://s.ch9.ms/events/build/2018/rss/mp4")); break}
            default {$video = ($rss.downloadstring("http://s.ch9.ms/events/build/2018/rss/mp4high")); break}
        }
 
        foreach($item in $video.rss.channel.item) {    
            # Grab the URL for the MP4 file
            $url = New-Object System.Uri($item.enclosure.url)  
     
            # Create the local file name for the video download
            $file = $item.title.Replace(":", "-").Replace("?", "").Replace("/", "-").Replace("&lt;", "").Replace("|", "").Replace('"',"").Replace("*","").Replace("’","'").Replace("â€","").Replace("'NEW SESSION'","").Replace("™","").Replace("œ","")
            $file = $file.substring(0, [System.Math]::Min(120, $file.Length))
            $file = $file.trim()
            $file = $file + ".mp4"  
     
            $folder = $item.title.Replace(":", "-").Replace("?", "").Replace("/", "-").Replace("&lt;", "").Replace("|", "").Replace('"',"").Replace("*","").Replace("’","'").Replace("â€","").Replace("'NEW SESSION'","").Replace("™","").Replace("œ","")
            $folder = $folder.substring(0, [System.Math]::Min(100, $folder.Length))
            $folder = $folder.trim()
     
            if (-not (Test-Path $folder)) { 
                Write-Host "Folder $folder doesn't exist. Creating it..."  
                New-Item $folder -type directory 
            }
         
     
            # Make sure the video file doesn't already exist
            if (!(test-path "$folder\$file"))     
            {   
                # Echo out the  file that's being downloaded
                $file
             
                # Download the video file
                try{
                    if (!(test-path "$path\$file"))
                    {
                        Invoke-WebRequest $url -OutFile $path\$file
             
                        #move it from the current working folder to the target
                        mv $file $folder
                    }
                    else {
                        Write-Host "Video: $file - anoter process possibly working on this; skipping download."
                    }
                }
                catch {
                    $ErrorMessage = $_.Exception.Message
                    $FailedItem = $_.Exception.ItemName
                    Write-host "\t" $ErrorMessage + "\n" + $FailedItem
                }
            }
            else {
                Write-Host "Video: $file exist; skipping download."  
            }
         
        } #end-loop foreach
    } #end - video check
}
catch
{
    Write-host "Oops, could not find any videos or some other error happened."
    $ErrorMessage = $_.Exception.Message
    $FailedItem = $_.Exception.ItemName
    Write-host "\t" $ErrorMessage + "\n" + $FailedItem
} # ********** End - download the videos section **********
 
Write-host "*************** All Done! Woot! ***************"

Download Build (2017) decks and video

Update: Modified the script to handle multiple instances but pay heed to the warning here.

Similar to last year, I have a PowerShell script that will allow you to download the various PowerPoint decks and videos to watch locally rather than stream. This makes some improvements from the earlier scripts (e.g. if a file is already downloaded it will skip downloading it again) and does the following:

  • Creates the relevant folder which includes the Session details (including the Title, and the Presenters)
  • For each session, saves the description in a text file in the created folder.
  • Downloads the relevant presentation (if any)
  • Downloads a jpg which shows the image session – sometimes it is easier just to see the title slide. I thought better to have it and not use it, than the other way.
  • And finally downloads the high-quality video of that session.

In the script, you can change the following (and if you understand Build then this should be easy):

  • Change the path where to download this to (default is d:\build)
  • Choose a lower quality video if you prefer (which of course takes less space and might not be bad depending on which device you are seeing). Of course this also uses less bandwidth.
  • Of course. And if you want only the decks, then you can comment out parts of the script where it doesn’t download the video.

The script will spit out some basic errors and will ‘eat’ some of the exceptions that are expected (e.g. every session doesn’t have a pptx or a video). That won’t break the script, it will just move to the next session.

And finally here is the script:

# First setup the folder where to download using the parameters outlined below.
# Second, loop through and get the decks first
# Third. loop through and get the videos last
# Note: IF you don't want to download the videos, and want only the pptx then comment the section later in the script

# parameters
[Environment]::CurrentDirectory=(Get-Location -PSProvider FileSystem).ProviderPath 
$rss = (new-object net.webclient)

#Filenames might get long, so keep this short!
$downloadlocation = "D:\build"

	if (-not (Test-Path $downloadlocation)) { 
		Write-Host "Folder $fpath dosen't exist. Creating it..."  
		New-Item $downloadlocation -type directory 
	}
set-location $downloadlocation

# Grab the RSS feed - Build 2017
$a = ($rss.downloadstring("http://s.ch9.ms/events/build/2017/rss/mp4high")) 
$b = ($rss.downloadstring("http://s.ch9.ms/events/build/2017/rss/slides")) 

# Video quality default is high; you can select regular (mp4) or lower quality (mp3)
#$a = ($rss.downloadstring("http://s.ch9.ms/events/build/2017/rss/mp4")) 
#$a = ($rss.downloadstring("http://s.ch9.ms/events/build/2017/rss/mp3")) 


# ********** download the decks **********
try { 

    foreach($item in $b.rss.channel.item) {   
	    $code = $item.comments.split("/") | select -last 1	   
	
	    # Get the url for the pptx file
	    $urlpptx = New-Object System.Uri($item.enclosure.url)  
        $urljpg = New-Object System.Uri($item.thumbnail.url)

        # make the filename readable
        $filepptx = $code + "-" + $item.creator + " - " + $item.title.Replace(":", "-").Replace("?", "").Replace("/", "-").Replace("<", "").Replace("|", "").Replace('"',"").Replace("*","").Replace("’","'").Replace("â€","")
	    $filepptx = $filepptx.substring(0, [System.Math]::Min(120, $filepptx.Length))
	    $filepptx = $filepptx.trim()
        $filejpg = $filepptx

	    $filepptx = $filepptx + ".pptx" 
        $filejpg = $filejpg + "_960.jpg"

    
	    if ($code -ne "") {
		     $folder = $code + " - " + $item.title.Replace(":", "-").Replace("?", "").Replace("/", "-").Replace("<", "").Replace("|", "").Replace('"',"").Replace("*","").Replace("’","'").Replace("â€","")
		     $folder = $folder.substring(0, [System.Math]::Min(100, $folder.Length))
		     $folder = $folder.trim()
	    }
	    else {
		    $folder = "NoCodeSessions"
	    }
	
	    if (-not (Test-Path $folder)) { 
		    Write-Host "Folder $folder dosen't exist. Creating it..."  
		    New-Item $folder -type directory 
	    }
	
	    # Make sure the PowerPoint file doesn't already exist
	    if (!(test-path "$downloadlocation\$folder\$filepptx")) { 	
		    # Echo out the file that's being downloaded
		    $filepptx
		    $wc = (New-Object System.Net.WebClient)  

		    # Download the pptx file
		    Invoke-WebRequest $urlpptx -OutFile $downloadlocation\$filepptx

            # download the jpg but don't want to break if this doesn't exist; hence the nested try blocks
            try {
                if (!(test-path "$downloadlocation\$filejpg")) { 	
                    $wc.DownloadFile($urljpg, "$downloadlocation\$filejpg")
                }
            }
            catch {
                Write-Host "Jpeg $filejpg doesn't exist ... eating the exception and moving on ..."
            }
        
		    mv $filepptx $folder 
            mv $filejpg $folder 
	    } #endif
        else {
            Write-Host "PPTX: $filepptx exist; skipping download."  
        }
	} #end-loop foreach

    Write-host "*************** Downloading all the decks complete ***************"
}
catch
{
    Write-host "Oops, could not find any slides."
    $ErrorMessage = $_.Exception.Message
    $FailedItem = $_.Exception.ItemName
    Write-host "\t" $ErrorMessage + "\n" + $FailedItem
}

# ********** download the videos **********
# if you don't want the video but only the slides just comment all the code below in the foreach loop
try { 
    foreach($item in $a.rss.channel.item) {
	    $code = $item.comments.split("/") | select -last 1	   
	
	    # Grab the URL for the MP4 file
	    $url = New-Object System.Uri($item.enclosure.url)  
	
	    # Create the local file name for the MP4 download
	    $file = $code + "-" + $item.creator + "-" + $item.title.Replace(":", "-").Replace("?", "").Replace("/", "-").Replace("<", "").Replace("|", "").Replace('"',"").Replace("*","").Replace("’","'").Replace("â€","")
	    $file = $file.substring(0, [System.Math]::Min(120, $file.Length))
	    $file = $file.trim()
	    $file = $file + ".mp4"  
	
	    if ($code -ne "")
	    {
		     $folder = $code + " - " + $item.title.Replace(":", "-").Replace("?", "").Replace("/", "-").Replace("<", "").Replace("|", "").Replace('"',"").Replace("*","").Replace("’","'").Replace("â€","")
		     $folder = $folder.substring(0, [System.Math]::Min(100, $folder.Length))
		     $folder = $folder.trim()
	    }
	    else
	    {
		    $folder = "NoCodeSessions"
	    }
	
	    if (-not (Test-Path $folder)) { 
		    Write-Host "Folder $folder doesn't exist. Creating it..."  
		    New-Item $folder -type directory 
	    }
		
	
	    # Make sure the MP4 file doesn't already exist
	    if (!(test-path "$folder\$file"))     
	    { 	
		    # Echo out the  file that's being downloaded
		    $file
		    
		    # Download the MP4 file
		    try{
				if (!(test-path "$downloadlocation\$file"))
				{
					Invoke-WebRequest $url -OutFile $downloadlocation\$file
		    
					#move it from the current working folder to the target
					mv $file $folder
				}
				else {
					Write-Host "Video: $file - anoter process possibly working on this; skipping download."
				}
			}
			catch {
				$ErrorMessage = $_.Exception.Message
				$FailedItem = $_.Exception.ItemName
				Write-host "\t" $ErrorMessage + "\n" + $FailedItem
			}
	    }
        else {
            Write-Host "Video: $file exist; skipping download."  
        }

        #Try and get the Sessions text description
        try {
	        $OutFile = New-Item -type file "$($downloadlocation)\$($Folder)\$($Code.trim()).txt" -Force  
            $Content = ""
            $Content = $item.title.ToString().trim() + "`r`n" + $item.creator + "`r`n" + $item.summary.ToString().trim() + "`r`n" + "`r`n" + $item.description.ToString().trim() + "`r`n" + "`r`n" + $item.comments.ToString().trim() 
            add-content $OutFile $Content
        }
        catch {
            $ErrorMessage = $_.Exception.Message
            $FailedItem = $_.Exception.ItemName
            Write-host "\t" $ErrorMessage + "\n" + $FailedItem
        }
		
    } #end-loop foreach
}
catch
{
    Write-host "Oops, could not find any videos or some other error happened."
    $ErrorMessage = $_.Exception.Message
    $FailedItem = $_.Exception.ItemName
    Write-host "\t" $ErrorMessage + "\n" + $FailedItem
} # ********** End - download the videos section **********

Write-host "*************** All Done! ***************"

If you read through the script it is pretty self explanatory.

bash on Windows is real–not a VM

I have talked to a few folks recently, and they still don’t believe bash on Windows (RS1) is ‘real’ and think it some kind of a VM. No it is not. It is the ‘real’ user mode running on Windows. It is not Cygwin, and it is not a VM. It is essentially all of the user mode (I.e. Linux without the kernel).

The kernel in this case is a wrapper around the NT kernel that translates the Linux commands to Windows and then things run. As far as Linux is concerned, its the same code and doesn’t have any changes). Technically this is called Windows Subsystem for Linux (WSL).

On windows, this is installed in the user space; so each user get their own instance effectively which is isolated from the other users. Once you install it (and if you are still reading this, then you probably know how to install it), then this shows up under C:\Users\your-user-ID\AppData\Local\lxss. If you can’t find that folder, you can still type it and navigate to it. Below is  a screen shot on what this looks like:

image

It is a little interesting and been mucking around this. Here is you can see the installation of gcc:

image

And here is the output of the CPU details:

image

root@localhost:/proc# cat cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 78
model name      : Intel(R) Core(TM) i7-6600U CPU @ 2.60GHz
stepping        : 3
microcode       : 0xffffffff
cpu MHz         : 2808.000
cache size      : 256 KB
physical id     : 0
siblings        : 4
core id         : 0
cpu cores       : 2
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 6
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm pni pclmulqdq est tm2 ssse3 fma cx16 xtpr sse4_1 sse4_2 movbe popcnt aes xsave osxsave avx f16c rdrand hypervisor
bogomips        : 5616.00
clflush size    : 64
cache_alignment : 64
address sizes   : 36 bits physical, 48 bits virtual
power management:

processor       : 1
vendor_id       : GenuineIntel
cpu family      : 6
model           : 78
model name      : Intel(R) Core(TM) i7-6600U CPU @ 2.60GHz
stepping        : 3
microcode       : 0xffffffff
cpu MHz         : 2808.000
cache size      : 256 KB
physical id     : 0
siblings        : 4
core id         : 0
cpu cores       : 2
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 6
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm pni pclmulqdq est tm2 ssse3 fma cx16 xtpr sse4_1 sse4_2 movbe popcnt aes xsave osxsave avx f16c rdrand hypervisor
bogomips        : 5616.00
clflush size    : 64
cache_alignment : 64
address sizes   : 36 bits physical, 48 bits virtual
power management:

processor       : 2
vendor_id       : GenuineIntel
cpu family      : 6
model           : 78
model name      : Intel(R) Core(TM) i7-6600U CPU @ 2.60GHz
stepping        : 3
microcode       : 0xffffffff
cpu MHz         : 2808.000
cache size      : 256 KB
physical id     : 0
siblings        : 4
core id         : 1
cpu cores       : 2
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 6
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm pni pclmulqdq est tm2 ssse3 fma cx16 xtpr sse4_1 sse4_2 movbe popcnt aes xsave osxsave avx f16c rdrand hypervisor
bogomips        : 5616.00
clflush size    : 64
cache_alignment : 64
address sizes   : 36 bits physical, 48 bits virtual
power management:

processor       : 3
vendor_id       : GenuineIntel
cpu family      : 6
model           : 78
model name      : Intel(R) Core(TM) i7-6600U CPU @ 2.60GHz
stepping        : 3
microcode       : 0xffffffff
cpu MHz         : 2808.000
cache size      : 256 KB
physical id     : 0
siblings        : 4
core id         : 1
cpu cores       : 2
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 6
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm pni pclmulqdq est tm2 ssse3 fma cx16 xtpr sse4_1 sse4_2 movbe popcnt aes xsave osxsave avx f16c rdrand hypervisor
bogomips        : 5616.00
clflush size    : 64
cache_alignment : 64
address sizes   : 36 bits physical, 48 bits virtual
power management:

root@localhost:/proc#

All, in all a very interesting world. A few things to note:

  • This is still in beta, so there will be issues.
  • It is user mode and not server mode. Live with it.
  • There would be path issues if you stray into the 256 character limit of Windows and then try and manipulate it in bash.

Happy hacking!

Download Build (2016) decks and videos

Update – Was a typo in the script, which I fixed. If someone had an issue, copy it again and give it a go.

I prefer to download and see the decks and videos ‘offline’ instead of streaming, as I can easily pause them and then pick off where I left – especially handy when I need to go to a meeting, or take care of some work, or just on a crappy network.

To that end, I downloaded all the sessions and the videos from Build (2016); I think one of the sessions might have gotten corrupted, but ignoring that I have about 140GB of data downloaded and 219 Sessions.

To do this, I used a Power Shell script that helped me download. The script (below) is simple enough – it loops through and for each session does the following:

  • Creates the relevant folder – this includes the Session code, Title, and the Presenters.
  • For each session, extracts the abstract / description of that session and saves it in a text file in that folder.
  • Downloads the presentation for that session.
  • Downloads a jpg which shows the image session – sometimes it is easier just to see the title slide. I thought better to have it and not use it, than the other way.
  • And finally downloads the high-quality video of that session. In the script, you can choose a lower quality video if you prefer that. And if you want only the decks, then you can comment out parts of the script where it doesn’t download the video.

Here is an example on what the output for one of the sessions (P517 – Debugging Events in Edge) looks like:

Build-2016-download-example

A few points to note:

  • In the script, the root folder it tries to create and use is “d:\build”; you might want to change this to something else you prefer or what works for you. I would recommend to not making the root too long as you would get into long file name issues.
  • The script first loops through and downloads everything else except the videos first. And then 2nd time goes and grabs the videos.
  • If there is an exception with the images, then it just ‘eats’ it and moves on. It is not a breaking condition to stop the script.
# First setup the folder where to download using the parameters outlined below.
# Second, loop through and get the decks first
# Third. loop through and get the videos last
# Note: IF you don't want to download the videos, and want only the pptx then comment the section later in the script

# parameters
[Environment]::CurrentDirectory=(Get-Location -PSProvider FileSystem).ProviderPath 
$rss = (new-object net.webclient)

#Filenames might get long, so keep this short!
$downloadlocation = "D:\build"

	if (-not (Test-Path $downloadlocation)) { 
		Write-Host "Folder $fpath dosen't exist. Creating it..."  
		New-Item $downloadlocation -type directory 
	}
set-location $downloadlocation

# Grab the RSS feed - Build 2016
$a = ($rss.downloadstring("http://s.ch9.ms/events/build/2016/rss/mp4high")) 
$b = ($rss.downloadstring("http://s.ch9.ms/events/build/2016/rss/slides")) 

# Video quality default is high; you can select regular (mp4) or lower quality (mp3)
#$a = ($rss.downloadstring("http://s.ch9.ms/events/build/2015/rss/mp4")) 
#$a = ($rss.downloadstring("http://s.ch9.ms/events/build/2015/rss/mp3")) 


# ********** download the decks **********
try { 
    $b.rss.channel.item | foreach {   
	    $code = $_.comments.split("/") | select -last 1	   
	
	    # Get the url for the pptx file
	    $urlpptx = New-Object System.Uri($_.enclosure.url)  
        $urljpg = New-Object System.Uri($_.thumbnail.url)

        # make the filename readable
        $filepptx = $code + "-" + $_.creator + " - " + $_.title.Replace(":", "-").Replace("?", "").Replace("/", "-").Replace("<", "").Replace("|", "").Replace('"',"").Replace("*","")
	    $filepptx = $filepptx.substring(0, [System.Math]::Min(120, $filepptx.Length))
	    $filepptx = $filepptx.trim()
        $filejpg = $filepptx

	    $filepptx = $filepptx + ".pptx" 
        $filejpg = $filejpg + "_960.jpg"

	    if ($code -ne "") {
		     $folder = $code + " - " + $_.title.Replace(":", "-").Replace("?", "").Replace("/", "-").Replace("<", "").Replace("|", "").Replace('"',"").Replace("*","")
		     $folder = $folder.substring(0, [System.Math]::Min(100, $folder.Length))
		     $folder = $folder.trim()
	    }
	    else {
		    $folder = "NoCodeSessions"
	    }
	
	    if (-not (Test-Path $folder)) { 
		    Write-Host "Folder $folder dosen't exist. Creating it..."  
		    New-Item $folder -type directory 
	    }
	
	
	    # Make sure the PowerPoint file doesn't already exist
	    if (!(test-path "$downloadlocation\$folder\$filepptx")) { 	
		    # Echo out the file that's being downloaded
		    $filepptx
		    $wc = (New-Object System.Net.WebClient)  

		    # Download the pptx file
		    $wc.DownloadFile($urlpptx, "$downloadlocation\$filepptx")

            # download the jpg but don't want to break if this doesn't exist; hence the nested try blocks
            try {
                $wc.DownloadFile($urljpg, "$downloadlocation\$filejpg")
            }
            catch {
                Write-Host "Jpeg $filejpg doesn't exist ... eating the exception and moving on ..."
            }
        
		    mv $filepptx $folder 
            mv $filejpg $folder 
	    } #endif
	} #end-loop foreach

    Write-host "*************** Downloading all the decks complete ***************"
}
catch
{
    Write-host "Oops, could not find any slides."
}

# ********** download the videos **********
$a.rss.channel.item | foreach {   
	$code = $_.comments.split("/") | select -last 1	   
	
	# Grab the URL for the MP4 file
	$url = New-Object System.Uri($_.enclosure.url)  
	
	# Create the local file name for the MP4 download
	$file = $code + "-" + $_.creator + "-" + $_.title.Replace(":", "-").Replace("?", "").Replace("/", "-").Replace("<", "").Replace("|", "").Replace('"',"").Replace("*","")
	$file = $file.substring(0, [System.Math]::Min(120, $file.Length))
	$file = $file.trim()
	$file = $file + ".mp4"  
	
	if ($code -ne "") {
		 $folder = $code + " - " + $_.title.Replace(":", "-").Replace("?", "").Replace("/", "-").Replace("<", "").Replace("|", "").Replace('"',"").Replace("*","")
		 $folder = $folder.substring(0, [System.Math]::Min(100, $folder.Length))
		 $folder = $folder.trim()
	}
	else {
		$folder = "NoCodeSessions"
	}
	
	if (-not (Test-Path $folder)) { 
		Write-Host "Folder $folder) dosen't exist. Creating it..."  
		New-Item $folder -type directory 
	}
	
	# Make sure the MP4 file doesn't already exist
	if (!(test-path "$folder\$file")) { 	
		# Echo out the  file that's being downloaded
		$file
		$wc = (New-Object System.Net.WebClient)  

		# Download the MP4 file
		$wc.DownloadFile($url, "$downloadlocation\$file")
		mv $file $folder
	}

    #Try and get the Sessions text description
	$OutFile = New-Item -type file "$($downloadlocation)\$($Folder)\$($Code.trim()).txt" -Force  
    $Category = "" ; $Content = ""
    $_.category | foreach {$Category += $_ + ","}
    $Content = $_.title.trim() + "`r`n" + $_.creator + "`r`n" + $_.summary.trim() + "`r`n" + "`r`n" + $Category.Substring(0,$Category.Length -1)
    add-content $OutFile $Content		
} #end-loop foreach

Write-host "*************** All Done! ***************"

Unity and Visual Studio

Have been playing, a little with the new Visual Studio “preview” version (the installation of which is super smooth and takes waaaaay less time!). And as part of that saw the Unity Debugger option. Was this always there or is that new? Or did the Unity Tools beta add it? Interesting times to start playing with this, primarily for AR/VR and #HoloLens.

image

Humans and threading

We,  humans, are multi-threaded by design and can do many things in parallel –  with two exceptions I think. The only two blocking function we have to deal with are sneezing and farting. During these times, all current activity must be suspended for the duration. And of course it can be pretty annoying (or depending on the function, embarrassing).

So next time you check in some code, think about it –  is this smelly and sneezy (yep, that’s a word, now) or have I done the right thing?

Open Live Writer

Just downloaded and installed and posting this via Open Live Writer. Remember the lovely (but dead) Windows Live Writer from Microsoft? Well this is a forked version of that which is open source, based on MIT license.

The editor is offline and is very similar to word and can support a number of blogging platforms – including the common ones as you would expect. You can muck around the code (zip) or check it out on GIT. It is still work in progress of course (e.g. Plugin’s are not implemented yet). Irrespective, hugely grateful to Scott (Hanselman) to get this going.

Is rand() harmful?

​I saw this awesome presentation on why rand() is considered harmful. When you need a random number, don’t call rand() and especially don’t say rand() % 100! This presentation will explain why that’s so terrible, and how C++11’s header can make your life so much easier.

If you need uniqueness and non-deterministic, especially on the context of security or crypto then you need to think about a few things. For example the frequency, non-uniform distribution, and not using a pseudo random number generator (such as Mersenne twister) and not a linear congruential generator.

C++ Comment

int MyFunction()
{
    // There once was a man named Dave
    int Result = 0;

    // Whose code just wouldn't behave
    MyObject *Ptr = new MyObject();

    // He left to go to a meetin'
    Result = Ptr->DoSomething();

    // And left his memory a leakin'
    return Result;
}

How to insult a developer?

How to insult a developer
How to insult a developer

How not to handle exceptions!

Was trying to pay my Electricity bill online via a site called Bangalore One, which is the Governments, premier one-stop shop for Electronic Delivery of Citizen Services.

I could not pay because it seems like some backend services they need for credit card payment is down. How do I know this? Because the site is revealing too much detail! See the exception details pasted below.

This is a great example of what not to do! I have seen this often, and it is lazy developers and even lazier testers who approved this and get this into production. One would have thought that government managing the “Silicon Valley of India” would know better!

It is also interesting to see that they are on a very old version of .NET – running on v1.1.

Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding.

Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.

Exception Details: System.Exception: Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding.

Source Error:

An unhandled exception was generated during the execution of the current web request. Information regarding the origin and location of the exception can be identified using the exception stack trace below.

Stack Trace:

[Exception: Timeout expired.  The timeout period elapsed prior to completion of the operation or the server is not responding.]
   BangaloreOne.clsBESCOM.fnCheckTransCnt(String LocationRRNo, String StaffCode, Int32 intDeptCode) +381
   bOneWebPortal.BESCOMConfirm.Page_Load(Object sender, EventArgs e) +721
   System.Web.UI.Control.OnLoad(EventArgs e) +67
   System.Web.UI.Control.LoadRecursive() +35
   System.Web.UI.Page.ProcessRequestMain() +750

Version Information: Microsoft .NET Framework Version:1.1.4322.2300; ASP.NET Version:1.1.4322.2300

Writing a compiler using C#

I was cleaning out my old papers (finally!) and came across an old paper I had titled “Compiler Writing Tools using C#” which essentially shows how you can write a number of tools like lex and yacc but instead of C/C++ on Unix, you use C# and .NET.

This paper covers the tokenizer, grammar, DFA, NFA, etc. I think conflicts and precedence is one area it would need a little more work. But overall its a very interesting piece of work – especially for one to learn the ropes if a lot of this is new.

Of course, writing compilers with C, overall is not the most productive experience (it sure is fun though!); if you have never done it before Standard ML or Haskell would be a better place to start. If you want to stick to .NET and roll out your own language (or create extensions) then check out this paper on MSDN.

All of this brought back memories when I wrote my own C++ compilers mainly with those tools on Unix System V and also on Xenix. Most people did not know (or perhaps remember) that Microsoft had a Unix version called Xenix which was later bought over by SCO and eventually become part of SCO Unix. Back in the days I use to run a dual-boot machine with Xenix and DOS. 🙂

Metro Apps in C++ anyone?

In Visual Studio “11” when I try and create a new C++ Metro app using the built-in template, I get the following error: “Can’t find localized resources”.

I wonder if anyone else has managed to get around this? I am running the Consumer Preview Build of Win 8 (Build 8250).

image

image