Monday, November 2, 2015

How to authenticate your Android App to SharePoint 2013

This blog post shows how to authenticate against an on premise SharePoint 2013 environment and do a simple REST call to retrieve the title of the root web. The SharePoint environment is configured to authenticate with Forms Based Authentication and runs on https.

Various other posts about an Android app authenticating to SharePoint are using a WebView solution. But I didn't want the user to use a integrated web view, but just do the authentication in code. The main point is to get hold of the FedAuth token from a cookie and pass the token back into all calls to SharePoint.

The main steps to authenticate and do a REST call to SharePoint are as follows:
1- Configure the CookieManager
2- Authenticate user: Start a AsyncTask to do a SOAP request to https://sharepoint.dev/_vti_bin/authentication.asmx
3- The cookie returned in step 2 will contain the FedAuth token.
4- the REST call: Start a HttpURLConnection to do a GET request to https://sharepoint.dev/_api/web/title
     Make sure this HttpURLConnection contains the FedAuth token in Cookie.
5- Parse JSON result

Now some more details please…

1- Configure the CookieManager
In Frament: onCreateView. Create a CookieManager that handles cookies within the VM. Make sure to set it to default. The network calls will use this CookieManager. That includes the CookieStore which will contain the cookies received by network calls.

   1: CookieHandler cookieHandler = CookieHandler.getDefault();
   2: if (cookieHandler==null) {
   3:     CookieManager cookieManager = new CookieManager();
   4:     CookieHandler.setDefault(cookieManager);
   5: }


2- Authenticate user
Doing a SOAP call to  https://sharepoint.dev/_vti_bin/authentication.asmx in AsyncTask, doInBackground.

Line 1-4: various SOAP settings
Line 6/7, 12/13: username and password
Line 24: doing the SOAP call

   1: String SOAP_ACTION1 = "http://schemas.microsoft.com/sharepoint/soap/Login";
   2: String NAMESPACE = "http://schemas.microsoft.com/sharepoint/soap/";
   3: String METHOD_NAME1 = "Login";
   4: String AuthURL = "https://sharepoint.dev/_vti_bin/authentication.asmx";
   5:  
   6: String username = "koenvanderlinden";
   7: String password = "P@ssw0rd";
   8:  
   9: SoapObject request = new SoapObject(NAMESPACE, METHOD_NAME1);
  10:  
  11: //insert username and password to SoapEnvelope
  12: request.addProperty("username", username);
  13: request.addProperty("password", password);
  14:  
  15: //Declare the version of the SOAP request
  16: SoapSerializationEnvelope envelope = new SoapSerializationEnvelope(SoapEnvelope.VER11);
  17:  
  18: envelope.setOutputSoapObject(request);
  19: envelope.dotNet = true;
  20:  
  21: try {
  22:     HttpTransportSE androidHttpTransport = new HttpTransportSE(AuthURL);
  23:  
  24:     // do the SoapCall
  25:     androidHttpTransport.call(SOAP_ACTION1, envelope, null);
  26:     // At this point the CookieManager will containt a cookie that has the FedAuth token.
  27:     
  28: } catch (Exception e) {
  29:     Log.v(LOG_TAG, e.toString());
  30:     e.printStackTrace();
  31: }

3- Inspect cookies
After the call androidHttpTransport.call() (line 22) the CookieManager will contain the cookies, including the FedAuth token.
To inspect the cookies add the following lines:

   1: // check cookies
   2: CookieManager cookieManager = (CookieManager)CookieHandler.getDefault();
   3: List<HttpCookie> cookies = cookieManager.getCookieStore().getCookies();

 
4- the REST call:

Line 4: Set REST url to get web title
Line 7: Create HttpUrlConnection
Line 10: Add application/json;odata=verbose to “Accept” header to get response in JSON format.
Line 11: Start the HttpURLConnection to do a GET request

The HttpURLConnection will use the CookieManager that is set as default. The CookieManager contains the CookieStore, which contains all cookies. Because the HttpURLConnection is doing a request to domain sharepoint.dev, the cookies will be added to the request that match on that domain. In this case the FedAuth cookie matches and is added to the request.

   1: HttpURLConnection urlConnection;
   2: BufferedReader reader;
   3:  
   4: Uri uri = Uri.parse("https://sharepoint.dev/_api/web/title");
   5: URL url = new URL(uri.toString());
   6:  
   7: urlConnection = (HttpURLConnection) url.openConnection();
   8: urlConnection.setRequestMethod("GET");
   9: // we need JSON formatted result
  10: urlConnection.setRequestProperty("Accept", "application/json;odata=verbose");
  11: urlConnection.connect();
  12:  
  13: // Read the input stream into a String
  14: InputStream inputStream = urlConnection.getInputStream();
  15: StringBuffer buffer = new StringBuffer();
  16: if (inputStream == null) {
  17:     // Nothing to do.
  18:     return null;
  19: }
  20: reader = new BufferedReader(new InputStreamReader(inputStream));
  21:  
  22: String line;
  23: while ((line = reader.readLine()) != null) {
  24:     // Since it's JSON, adding a newline isn't necessary (it won't affect parsing)
  25:     // But it does make debugging a *lot* easier if you print out the completed
  26:     // buffer for debugging.
  27:     buffer.append(line + "\n");
  28: }
  29:  
  30: if (buffer.length() == 0) {
  31:     // Stream was empty.  No point in parsing.
  32:     return null;
  33: }
  34:  
  35: // JSON result
  36: String restResult = buffer.toString();

5- Parse JSON result

   1: // parse JSON result
   2: JSONObject jsonResult = new JSONObject(restResultString).getJSONObject("d");
   3: title = jsonResult.getString("Title");

Additional Information
This code makes use of the ksoap2-android library. The Gradle configuratoin has been changes to include ksoap2 library by adding the following parts:

Add: repositories to android.buildTypes
        repositories {
            maven { url 'https://oss.sonatype.org/content/repositories/ksoap2-android-releases/' }
        }
Add: compile 'com.google.code.ksoap2-android:ksoap2-android:3.1.1' to dependencies.

Sample gradle:

   1: apply plugin: 'com.android.application'
   2:  
   3: android {
   4:     compileSdkVersion 23
   5:     buildToolsVersion "23.0.1"
   6:  
   7:     defaultConfig {
   8:         applicationId "nl.idoconsultancy.androidauthsp"
   9:         minSdkVersion 21
  10:         targetSdkVersion 22
  11:         versionCode 1
  12:         versionName "1.0"
  13:     }
  14:     buildTypes {
  15:         release {
  16:             minifyEnabled false
  17:             proguardFiles getDefaultProguardFile('proguard-android.txt'), 'proguard-rules.pro'
  18:         }
  19:         repositories {
  20:             maven { url 'https://oss.sonatype.org/content/repositories/ksoap2-android-releases/' }
  21:         }
  22:     }
  23: }
  24:  
  25: dependencies {
  26:     compile fileTree(dir: 'libs', include: ['*.jar'])
  27:     compile 'com.android.support:appcompat-v7:23.0.1'
  28:     compile 'com.android.support:design:23.0.1'
  29:     compile 'com.google.code.ksoap2-android:ksoap2-android:3.1.1'
  30: }

Friday, October 17, 2014

Operating your Hue remotely using Hue Remote API – a working Proof of Concept

Introduction

To operate your Hue system on local network you can send REST based messages like described in the Philips hue API to your Hue bridge. However if you want to do this in your custom application over the internet you have a problem. No API, no official documentation about how to sent messages through the Hue Portal that forwards it to your Hue bridge.

This blog post will show how you can send messages to your Hue bridge over the internet (without using port forwards). In this proof of concept I reuse the access token of IFTTT. The code samples are in C#, however you could make the same REST calls using the language of your choice.

BTW: I assume you have an account on the Hue portal and that you registered your Hue bridge.

What others write about this topic

Just a few people have some information about this like:
http://blog.paulshi.me/technical/2013/11/27/Philips-Hue-Remote-API-Explained.html
Hacking Lightbulbs: Security Evaluation of the Philips Hue Personal Wireless Lighting System by Nitesh Dhanjani
http://stackoverflow.com/questions/19900657/how-to-connect-backend-service-with-philips-hue-bridge-remotely

I investigated some of the findings from above resources. The parts about the getting the access token is not working, it’s probably out of date because of changes to the Hue Portal. So if you don’t have an access token you cannot send commands to the Hue Portal.

Local REST Messages

Sending a REST based message to your local bridge is done by sending a http request in the following format.

URL http://<your bridge ip>/api/<username>/<some resource>
Method GET/PUT/POST/DELETE
Body JSON formatted data

To turn on light 1 you will send a message like:

URL /api/<username>/lights/1/state
Method PUT
Body {
“on”: true
}

Remote REST Message

Now it gets interesting. Sending a message to your bridge through the Hue Portal is a bit different. You should always use POST and send the request to the Hue portal. It should also include a valid access token. The body contains a clipmessage which contains a clipCommand that encapsulates the JSON command you normally sent to your bridge (in local network mode).

The format is as follows:

URL https://www.meethue.com/api/sendmessage?token=<AccessToken>
Method always POST
Body clipmessage=<JSON formatted clipCommand>

clipCommand==>
{
   “clipCommand” :
         {
            “url”: “api/0/<some resource>”,
            “method”: “GET/PUT/POST/DELETE”,
            “body”: JSON formatted data
         }
}
Header encoding: UFT-8
application/x-www-form-urlencoded

Note: the url in clipCommand uses ‘0’ what in a local scenario would be the username.

Below an example of a command that will turn on light 1.

URL https://www.meethue.com/api/sendmessage?token=<AccessToken>
Method POST
Body clipmessage=
{
    “clipCommand” :
         {
            “url”: “/api/0/lights/1/state”,
            “method”: “PUT”,
            “body”:
               { “on”: true }
         }
}

To retrieve the status of the Hue system you need to sent a message as follows:

URL https://www.meethue.com/api/getbridge?token=<AccessToken>
Method GET

The result will be a JSON string representing the current state of your Hue system.

AccessToken

The biggest challenge in all this is to obtain a valid access token. In the next step I’m going to get the access token that IFTTT is using. But you could also use the access token of the official Philips Hue app.

Steps:

1- Go to IFTTT

2- Search for Channel ‘Philips Hue’

3- Activate Channel, this will bring you to the Hue Portal and log you in, and ask you to trust IFTTT. Click ‘Yes’. The Hue Portal will generate an access token and IFTTT will use that access token to send command to your Hue bridge.

4- When channel is activated logon to Hue Portal

5- Go to ‘Settings’

6- Click on ‘My Apps’ and you will see a list of apps you trusted for accessing your Hue bridge.

image

7- Copy the link of the ‘(De-activate)’ link. It will contain the access token. Here’s an example of my token: https://www.meethue.com/en-us/user/preferencesappsrevoke?tokenId=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaTT0%3D
The access token you should send in the remote message should be: “aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaTT0=” (without the double quotes and make sure to URL encode the whole thing).

BTW: If you revoke the access token by clicking ‘De-activate’ the access token cannot be used anymore. You will get a JSON response containing the message “I don't know that token”.

Code samples

We now have the message structure and the access token. We can now put this together in a C# sample.

   1: dynamic command = new ExpandoObject();
   2: command.clipCommand = new
   3: {
   4:     //change state of light 1
   5:     url = "/api/0/lights/1/state",
   6:     method = "PUT",
   7:     body = new
   8:     {
   9:       //set the light state changes
  10:       on = true
  11:     }
  12: };
  13:  
  14: SendMessage(command).Wait();

Lines 1-12 builds the clipCommand.
Line 14: sends the command

   1: private static async Task SendMessage(dynamic command)
   2: {
   3:   string jsonMessage = JsonConvert.SerializeObject(command);
   4:  
   5:   HttpClient client = new HttpClient();      
   6:   HttpContent content = new StringContent("clipmessage=" + jsonMessage, Encoding.UTF8, "application/x-www-form-urlencoded");
   7:  
   8:   Uri remoteMessageUrl = new Uri("https://www.meethue.com/api/sendmessage?token=<YOUR ACCESS TOKEN>");
   9:   var result = await client.PostAsync(remoteMessageUrl, content).ConfigureAwait(false);
  10:   var jsonResult = await result.Content.ReadAsStringAsync().ConfigureAwait(false);  
  11: }

Line 3: creates JSON formatted string of the command.
line 6: creates the body of the message, including the encoding and mediatype
line 8: creates URI including the access token
Line 9: sends request to Hue Portal
line 10: will contain results of the request.
Possible results could be:
success: "{\"code\":200,\"message\":\"ok\",\"result\":\"ok\"}"
failure: "{\"code\":109,\"message\":\"I don\\u0027t know that token.\",\"result\":\"error\"}"
failure: "{\"code\":113,\"message\":\"Invalid JSON.\",\"result\":\"error\"}"

 

Show me the Code

Get the C# sample from https://github.com/koenvanderlinden/RemoteHueDemo that contains sample for sending a message to change lights and how to retrieve bridge status.
Let me know if you think this blog post helped you in building a Hue enable application ;-).

TODO: getting the access token is still done by hand and it reuses another popular Hue application. Next step will be how to make the Hue Portal trust your own third party Hue app so you can use your own access token.

Wednesday, February 5, 2014

SharePoint issue “The context has expired and can no longer be used. Exception from HRESULT 0x80090317”

Problem
A client had problems with some workflow instances. Most instances did complete without a problem others failed.  Especially workflow that continued after a few days reported the following error in the workflow history log:
“{Microsoft.SharePoint.SPException: The context has expired and can no longer be used. (Exception from HRESULT: 0x80090317) ---> System.Runtime.InteropServices.COMException (0x80090317): The context has expired and can no longer be used. (Exception from HRESULT: 0x80090317)
   at Microsoft.SharePoint.Library.SPRequestInternalClass.SetHttpParameters(String bstrHttpMethod, String “

I did some googling and found some hits that found a solution in synchronizing the date/time of the WFE’s and admin servers. In our case that wasn’t the the case.

Cause
After digging into the problem I noticed particular workflow activity opened a site collection using an SPUserToken.
using (SPSite site = new SPSite(WorkflowProperties.SiteId, InitializationData.UserToken))
{
//do some work
}
The SPUserToken was part of the workflow context and was populated when the workflow instance was first created. Default the SPUserToken is valid for 1440 minutes (1 day). In case the workflow instance is de-hydrated after more than 1 day the SPUserToken is expired and isn’t valid anymore. You will get the message “The context has expired and can no longer be used. (Exception from HRESULT: 0x80090317” and are not allowed to open the sitecollection.

Conclusion
So when you get this specific error make sure to check if you use an old SPUserToken! And be aware the the SPUserToken can expire.

Detailed exception information
Full error message: "The context has expired and can no longer be used. (Exception from HRESULT: 0x80090317)"
Sample Stacktrace:
{Microsoft.SharePoint.SPException: The context has expired and can no longer be used. (Exception from HRESULT: 0x80090317) ---> System.Runtime.InteropServices.COMException (0x80090317): The context has expired and can no longer be used. (Exception from HRESULT: 0x80090317)
   at Microsoft.SharePoint.Library.SPRequestInternalClass.SetHttpParameters(String bstrHttpMethod, String bstrRequestDigest, UInt32 flags, Guid gTranLockerId, Byte[]& ppsaImpersonateUserToken, Boolean bIgnoreTimeout, String bstrUserLogin, String bstrUserKey, UInt32 ulRoleCount, String bstrRoles, Boolean bWindowsMode, ApplicationPrincipalInfo& pAppUserInfo, Boolean bInvalidateCachedConfigurationProperties, Int32 lAppDomainId, ISPManagedObjectFactory pFactory, Boolean bCallstack)
   at Microsoft.SharePoint.Library.SPRequest.SetHttpParameters(String bstrHttpMethod, String bstrRequestDigest, UInt32 flags, Guid gTranLockerId, Byte[]& ppsaImpersonateUserToken, Boolean bIgnoreTimeout, String bstrUserLogin, String bstrUserKey, UInt32 ulRoleCount, String bstrRoles, Boolean bWindowsMode, ApplicationPrincipalInfo& pAppUserInfo, Boolean bInvalidateCachedConfigurationProperties, Int32 lAppDomainId, ISPManagedObjectFactory pFactory, Boolean bCallstack)
   --- End of inner exception stack trace ---
   at Microsoft.SharePoint.SPGlobal.HandleComException(COMException comEx)
   at Microsoft.SharePoint.Library.SPRequest.SetHttpParameters(String bstrHttpMethod, String bstrRequestDigest, UInt32 flags, Guid gTranLockerId, Byte[]& ppsaImpersonateUserToken, Boolean bIgnoreTimeout, String bstrUserLogin, String bstrUserKey, UInt32 ulRoleCount, String bstrRoles, Boolean bWindowsMode, ApplicationPrincipalInfo& pAppUserInfo, Boolean bInvalidateCachedConfigurationProperties, Int32 lAppDomainId, ISPManagedObjectFactory pFactory, Boolean bCallstack)
   at Microsoft.SharePoint.SPGlobal.CreateSPRequestAndSetIdentity(SPSite site, String name, Boolean bNotGlobalAdminCode, String strUrl, Boolean bNotAddToContext, Byte[] UserToken, String userName, Boolean bIgnoreTokenTimeout, Boolean bAsAnonymous)
   at Microsoft.SharePoint.SPWeb.InitializeSPRequest()
   at Microsoft.SharePoint.SPWeb.InitWebPublic()
   at Microsoft.SharePoint.SPWeb.get_CurrentUser()

Thursday, January 30, 2014

How to handle ‘multicultural’ issues with the formula of a SharePoint SPFieldCalculated

Setting a formula in code for a SharePoint SPFieldCalculated is a real nasty task. I spent a lot of time figuring out why some formula’s did work and others did not in various cases.

When designing a formula you may think you need to take a few things into account like language and separator. During testing I came to the conclusion that the following has impact:
- thread culture
- separator character used for separating parameters in the formula
- SPContext.Current == null or not
- SharePoint default language (what language was the original SharePoint setup)
- and that the language of the formula method (like CONCATENATE) is not important at all.

Simple Test Case
First let me tell you about my test case I used. Two text fields, one is a company tag, the other is the first name of a contact person. A third will be the calculated field. It will need to show the following: ‘<Company Tag> - <First Name>’, without the ‘.
SPFieldText internalName: companyTag, DisplayName (English) = “Company Tag”, DisplayName (Dutch) = “Tag bedrijf”
SPFieldText internalName: contactFirstName, DisplayName (English) = “First Name”, DisplayName (Dutch) = “Voornaam”,
The creation of the calculated field should be in code (in this case a feature receiver). The deployment should work in various cases like:
Case 1: Activating feature by hand in UI.
Case 2: Feature is activated by onet.xml of a web template.

The problem:
The problem arises when you need it to work for case 1 and case 2 in different cultures. In my case in English and Dutch. By testing the creation using the following scenario’s you will see where it fails.

Scenario 1: Case 1 in English thread culture
Formula: “=CONCATENATE([Company Tag],\” – \”,[First Name])”
Separator: , (English)
Display names:  all English
Result: It works…

Scenario 2: Case 1 in Dutch thread culture
Formula: “=CONCATENATE([Tag bedrijf];\” – \”;[Voornaam])”
Separator: ; (Dutch)
Display names:  all Dutch
Result: It works…

Scenario 3: Case 2 in English thread culture
Same as Scenario 1:
Result: It works…

Scenario 4: Case 2 in Dutch thread culture
Same as Scenario 2:
Result: It does NOT work. 

Why doesn’t scenario 4 work? Threads are all Dutch, passing all the same parameters for the formula as scenario 2. So why…?
Also tried various combinations on the formula like:
- English display names, English and Dutch separator,
- Using Internal names, English and Dutch separator,
- Using Dutch formula method names like “=TEKST.SAMENVOEGEN([Tag bedrijf];\” – \”;[Voornaam])”

Only thing that seems to work was setting the separator to the default English (the default SharePoint installation culture) variant:
Formula: “=CONCATENATE([Tag bedrijf];\” – \”;[Voornaam])”
Separator: , (English)
Display names:  all Dutch

The main difference I detected between case 1 and case 2 was the SPContext.Current was null in case 2.
In case you design a formula and need to determine the separator, do not only check the Thread culture, but also the existence of SPContext.Current.

Guidelines regarding formula
Adding all variations of testing and failures together I came up with the following guideline when designing the formula for the SPFieldCalculated.
- Formula string should start with a '=' (not shown in above cases)
- always encapsulate the display name with brackets [], better safe than sorry (not shown in above cases)
- in case SPContext.Current is NOT null
* use field display names for the Thread.CurrentThread.CurrentUICulture
* use separator for the Thread.CurrentThread.CurrentUICulture  (Dutch= ‘;’ English=”,”)
- in case SPContext.Current==null (e.g.: when feature code via activated by an onet.xml of a webtemplate)
* use field display names for the Thread.CurrentThread.CurrentUICulture
* use separator that matched the SharePoint Initial Installation Language (English on my development machine. Contact me if you know how to determine this in code!)
- Methods names in the formula are culture independent, you can use English/Dutch/Whatever without a problem. (e.g. CONCATENATE and TEKST.SAMENVOEGEN work in English and Dutch thread culture)

I hope I saved you some valuable time in creating formula’s for SPFieldCalculated fields in code.
New insights on the formula are welcome.

Wednesday, September 25, 2013

SharePoint Synchronous Event Receiver with message to UI and saving form data

The goal is to save data from the new/edit form to list item and in certain situations show a message to UI by using event receivers.

As we all know showing a message to UI is not going to work when using async event receivers. So we’re stuck to use sync event receivers. At first you will see examples that shows a message by using code like this:

try
{
    //some code
}
catch (Exception ex)
{
    properties.Status = SPEventReceiverStatus.CancelWithError;
    properties.ErrorMessage = ex.Message;
    properties.Cancel = true;
}

The options for properties.Status are:
Continue, there is no error.
CancelNoError, silently cancel the request as if the request has succeeded.
CancelWithError, cancel the request with the error message specified in ErrorMessage.
CancelWithRedirectUrl cancel the request and redirect to the URL specified in RedirectUrl.

The only usable is CancelWithRedirectUrl. However this will not save your form data to the listitem. So all changes are lost.
Where is the option to ‘ContinueWithRedirectUrl’? It doesn’t exist.
Luckily we have the SPUtility class with the Redirect method. When using the SPRedirectFlag.DoNotEndResponse flag it will not abort the event receiver thread (Flag Default will abort event receiver thread and therefor will not save the afterproperties on the list item.). In this case the event receiver thread will be completed and afterproperties are save to the list item. And the redirect will be processed. You could redirect a user to a custom page, provide some parameters and show a message as you like.

Code sample of the redirect:

SPUtility.Redirect("notificationpage.aspx?status=somestatusvalue", SPRedirectFlags.DoNotEndResponse, HttpContext.Current);