Youtube and Android

Youtube is great and has a wealth of API's. On mobile, though, I just want simple playback. Up until around July 22, 2010, you could pass a url like this:

http://www.youtube.com/get_video?video_id=6Yd5DQykZow&t=some_token&fmt=17

into an Android VideoView and it would play the video. This used progressive download meaning the video would play once it had buffered enough, then keep downloading as play progressed. It also provided very good quality, since fmt (format) controlled whether it was a low-bandwidth or high-bandwidth data connection. In case you are wondering, if on WiFi or Wimax, use high bandwidth. Then something changed and this type of url does not work anymore. I would love to know what happened. I suspect it was just too easy to download content from Youtube this way and save it to a file and go build your own site.

You still play videos a variety of ways on Android. Here are some options found by using the gdata API's for Youtube (see http://code.google.com/apis/youtube/2.0/developers_guide_protocol.html)

Using this url

http://gdata.youtube.com/feeds/api/videos/6Yd5DQykZow

you get an XML feed about a given video. Here's the part relevant to playback:

<media:content url='http://www.youtube.com/v/6Yd5DQykZow?f=videos&app=youtube_gdata' type='application/x-shockwave-flash' medium='video' isDefault='true' expression='full' duration='322' yt:format='5'/>

<media:content url='rtsp://v1.cache8.c.youtube.com/CiILENy73wIaGQmMZqQMDXmH6RMYDSANFEgGUgZ2aWRlb3MM/0/0/0/video.3gp' type='video/3gpp' medium='video' expression='full' duration='322' yt:format='1'/>

<media:content url='rtsp://v3.cache4.c.youtube.com/CiILENy73wIaGQmMZqQMDXmH6RMYESARFEgGUgZ2aWRlb3MM/0/0/0/video.3gp' type='video/3gpp' medium='video' expression='full' duration='322' yt:format='6'/>

Format 1 = h.263, 5= flash video, and 6 = mp4. Formats 1 and 6 are low quality and meant for old cell phones on slow networks (176 x 144 resolution), but they work fine when passed to VideoView.setVideoURI(uri)

What's more interesting is the flash url (don't ask where html5 video comes in- I don't know). You can play this in a browser or in the Youtube application on HTC Hero and EVO and many more phones. Clicking the link will bring up the chooser for you to pick how to handle the video link. You don't need a VideoView; in fact, it won't work if you try one- go figure. Google is definitely having some struggle with all of this. Froyo (Android 2.2) will support flash, also, so lots of phones will work with the flash option.

There is still a question of quality of the playback- I want more control (i.e., flash video parameters for what I want) or the ability to give hints via SDP to the RTSP server, but no dice.

There is more mystery

This link

http://www.youtube.com/get_video_info?&video_id=6Yd5DQykZow

retrieves a different info feed entirely. It is all url-encoded name/value pairs, easy enough to decode and see that there are urls in it. In fact, you can download videos using them. The content type is flash, so if you download the video and save it as an *.flv file you could play the video with VLC (http://www.videolan.org)

Secure file distribution with apache, openssl, and curl

This is a long post. Be warned.

I have a problem to solve. I need to share a file on a regular basis with a small group of users. It has to be done securely and upload and download must be automated. It also has to be cheap and simple.

The plan is to layer HTTP Basic and SSL Client Auth for directory access to the BSA files. This provides encryption, strong authentication, and with HTTP Basic, the ability to disable an account easily (even though their client certificate may still be good. I didn't want to deal with certificate revocation lists). For retrieval, curl is a great option and meets all requirements.

Tools and assumptions

I used Ubuntu (8.04 or later), apache2 (2.2), and the latest curl and openssl packages. You know some *nix.

Apache2 configuration

The BSA file will be placed in a directory called 'bsa', so create this in /var/www (the default)

Add the following to the /etc/apache2/httpd.conf file to protect your new directory.

<Directory /var/www/bsa>

AuthType Basic

AuthName "Approved BSA users"

AuthUserFile /usr/local/apache/passwd/passwords

AuthGroupFile /usr/local/apache/passwd/groups

Require group BsaUsers

</Directory>

Note that this will be changed later for the final solution. I am just showing how to do this in general.

You will need to create the directories apache and passwd so that the path /usr/local/apache/passwd exists. Here is how to create the files 'passwords' and 'groups'.

group file creation

You simply make a text file (called 'groups') with an entry like this:

BsaUsers: user1 user2 user3

Everything after "BsaUsers:" are the user names for who has access. Each will need a password.

passwords file creation

For this you must use an apache tool for creating password files, htpasswd. The command below creates the passwords file (-c option) and adds one user, user1. You will be prompted to add their password.

>htpasswd -c /usr/local/apache/passwd/passwords user1

Adding more users to the password file

>htpasswd /usr/local/apache/passwd/passwords user2

>htpasswd /usr/local/apache/passwd/passwords user3

(each time you are prompted for the password for the user)

Note: htpasswd is in /usr/local/apache2/bin in case that location is not on your path.

Adding a new user

1. add the user name to the 'group's file.
2. add a password for the new user as above

Now test that http basic auth is working for the direcory 'bsa' in the root of the web server. After writing to the httpd.conf file you may need to restart the server.

TIP: apache2 service name is not httpd, it is apache2. So restart is this:

>service apache2 restart

Now on to how to access an http basic protected directory using curl. First, when you try curl against the protected directory it will appear to work. If you look at what you downloaded, though, it will be a 401 Authentication Required file.

Here is how to use curl on a resource (URL) requiring http basic authentication:

>curl -u name:password www.example.com

So for my example, the command is this:

>curl -u user1:password1 -O localhost/bsa/testfile.zip

This will send the credentials to the server and download and save the requested file. It's that simple.

Now for the main event: SSL CLient authentication set up

(see http://httpd.apache.org/docs/2.2/ssl/ssl_howto.html and http://httpd.apache.org/docs/2.2/ssl/ssl_faq.html for complete details)

Enabling SSL

>a2enmod ssl

>service apache2 restart

You can verify that /etc/apache2/mods-enabled now has the ssl.conf and ssl.load symbolic links in it.

Creating Certs and Keys

I tried CA.pl, a script for working with openssl for creating server certificates and it caused no end of problems arround the private key not being found. I believe it is all due to a format issue but not worth hassling with. By using openssl directly, as follows, I got it working easily.

Create a RSA private key for your server (will be Triple-DES encrypted and PEM formatted):

>openssl genrsa -des3 -out server.key 1024

Please backup this host.key file and the pass-phrase you entered in a secure location. You can create a stronger key by changing 1024 to something like 4096.

You can see the details of this RSA private key by using the command:

>openssl rsa -noout -text -in server.key

Create a self-signed Certificate (X509 structure) with the RSA key you just created (output will be PEM formatted):

>openssl req -new -x509 -nodes -sha1 -days 365 -key server.key -out server.crt

This signs the server CSR and results in a server.crt file. Set 'days' to whatever you like.

Copy the server.crt file and the server.key file to a directory under /etc/apache2. You can make your own- just don't use conf.d since that will cause other errors.

You must set up apache2 so your server uses SSL and your new certs. I did this using a virtual host for SSL. You will find these under /etc/apache2/sites-enabled/000-default (I tried using the main config file, but ran into problems. Using a virtual host works.)

Add the following to 000-default:

<VirtualHost *:443>

DocumentRoot /var/www

SSLEngine on

SSLCertificateFile /etc/apache2/conf/ssl/ca.crt

SSLCertificateKeyFile /etc/apache2/conf/ssl/server.key

<Directory /var/www/bsa>

#ssl client auth set up

SSLVerifyClient require

SSLVerifyDepth 1

SSLCACertificateFile conf/ssl/ca.crt

# http basic auth set up

AuthType Basic

AuthName "Approved BSA users"

AuthUserFile /usr/local/apache/passwd/passwords

AuthGroupFile /usr/local/apache/passwd/groups Require group BsaUsers

</Directory>

</VirtualHost>

Second, I ensure you can only access the private area using https with this:

<Directory /var/www/bsa>

Deny from all

</Directory>

This goes in the virtual host for port 80 (in 000-default) and blocks all access except through port 443. At this point, you have a web server that supports both http and https traffic with a protected directory (bsa) that can only be accessed as follows:

• https
• with username and password
• and a client certificate

Now when I try to run the server, here's what I get. Note that the cert private key is found, as evidenced by being challenged for its passphrase.

>/etc/apache2# service apache2 restart

* Restarting web server apache2 Apache/2.2.12 mod_ssl/2.2.12 (Pass Phrase Dialog)

Some of your private key files are encrypted for security reasons.

In order to read them you have to provide the pass phrases.

Server localhost:443 (RSA)

Enter pass phrase:

OK: Pass Phrase Dialog successful.

When accessing using a browser I am getting the message I want- the server certificate is untrusted. This confirms apache is using my self-signed cert.

I accept the 'untrusted' cert, and get the following error in Firefox:

An error occurred during a connection to localhost.

SSL peer was unable to negotiate an acceptable set of security parameters.

(Error code: ssl_error_handshake_failure_alert)

This is also expected, since I am requiring client authentication, but my client has no certificate. Just to confirm, I removed the requirement for client auth from httpd.conf, and restarted the server. Now when I use https, I can get to the web page after accepting the cert. This is all good.

Next step- use a client cert for authentication

First, create a sample client certificate using own CA created earlier (and stored in 'selfCA').

Create a private key

>openssl genrsa -des3 -out server.key 1024

Create a CSR using your new key

>openssl req -new -key server.key -out server.csr

Sign CSR using your own CA

>openssl x509 -req -days 1100 -in server.csr -CA ../selfCA/server.crt -CAkey ../selfCA/server.key -set_serial 01 -out test.crt

You will be asked for the passphrase for server.key.

Now to use test.crt and the key. To work with curl, the cert and key need to be in the same file.

>cat test.crt server.key > client.crt

This just combines the files into one text file. Take a look.

Here's the finished curl command to download the file using client authentication and basic authentication.

>curl -u user1:password1 -k https://localhost/bsa/testfile.zip -E client.crt:passphrase1 -O https://localhost/bsa/testfile.zip

The explanation of everything

• -u username and password set up on the server for http basic authentication
• -k ignore certificate warnings from my server (since using self-signed CA with mismatched CN and hostname)
• -E specifies file with the client cert and key to use plus the key passphrase
• -O download and save the file from give URL and using the filename in the URL

Pretty cool, huh?

Location-aware Twitter search app for Android

I just wrote and published a new free app for android. It is called Locobird and a chunk of the code is below. It uses your location to search twitter around you. What's new about this app is that it uses the Clearwire Location Platform to get your rough location when you are connected to the 4G network. Now you are wondering, how do I connect to that? Simple, you get an EVO.

Information on the Clearwire Location Platform

android locobird market link

This app uses the new static map API from Google and a java Twitter API from http://www.winterwell.com/software/jtwitter.php

Here's the code for using the Clearwire Location Platform (CLP)

package com.thehopemachine.locobird;

import java.io.IOException;

import java.io.InputStream;

import java.net.HttpURLConnection;

import java.util.Date;

import org.apache.http.HttpEntity;

import org.apache.http.HttpResponse;

import org.apache.http.client.methods.HttpPost;

import org.apache.http.entity.StringEntity;

import org.apache.http.impl.client.DefaultHttpClient;

import org.json.JSONObject;

import org.json.JSONStringer;

import android.location.Location;

import android.util.Log;

public class WimaxLocationProvider {

private static String CLP_ADDRESS = "http://testlocation.clearwire-wmx.net:8000/json/";

private static Long STALE = 60000l; //one minutes worth of milliseconds

private static String HOST = "host";

private static String VERSION = "version";

private static String VERSION_VAL = "1.1.0";

private static String LATITUDE = "latitude";

private static String LONGITUDE = "longitude";

private static String ACCURACY = "accuracy";

private static String LOCATION = "location";

private static String ERROR = "error";

private static String QUERYARG = "/?app_id=";

private Location mLastFix = null;

public static String WIMAX = "the Clear 4G Network";

private String mHost = null;

private String mAppID = null;

private Date mLastCall = null;

public WimaxLocationProvider (String host, String appID)

{

mHost = host;

mAppID = appID;

}

public Location getLocation() throws Exception

{

//www.example.com/?app_id=freiu4898fhjrh4yshefheoiuhr

//do we have a last fix? is it still good?

String hostValue = mHost + QUERYARG + mAppID;

if (mLastFix != null && !isStale())

{

Log.d("locobird", "returning last fix");

return mLastFix;

}

//now test if called too quick

if (mLastCall != null && (new Date().getTime() - mLastCall.getTime() <= Locobird.SPURIOUS_FILTER))

{

//this implies we've been getting errors since we have no last fix but do have a last call

throw new CLPException("unable to provide location from CLP");

}

//else get a fix

Location l = null;

DefaultHttpClient httpclient = new DefaultHttpClient();

HttpPost post = new HttpPost(CLP_ADDRESS);

HttpResponse response = null;

HttpEntity entity = null;

InputStream input = null;

//create a json request (basic)

String request = null;

try

{

request = new JSONStringer().object().key(HOST).value(hostValue).key(VERSION).value(VERSION_VAL).endObject().toString();

post.setEntity(new StringEntity(request));

JSONObject locationData = null;

StringBuffer data = new StringBuffer(512);

//send the request (POST it)

response = httpclient.execute(post);

entity = response.getEntity();

//get IO stream from entity and turn into JSON object

input = entity.getContent();

//look at response code

if (response.getStatusLine().getStatusCode() >= HttpURLConnection.HTTP_BAD_REQUEST)

{

//400 bad request, 404 not found, 500 server error ,etc

//do not try to parse json- will cause error

l = null;

}

else if (response.getStatusLine().getStatusCode() == HttpURLConnection.HTTP_OK)

{

byte[] buffer = new byte[512];

int x = 0;

while ((x = input.read()) != -1)

{

data.append((char)x);

}

//create location from the response

Log.d("locobird", "CLP response= " + data.toString());

locationData = new JSONObject(data.toString());

//test for location object in the json (if you just query you will get an exception)

if (!locationData.has(LOCATION))

{

//get the error object and examine it for cause

//{"error":"server error"}

if (locationData.has(ERROR))

{

String errorCause = locationData.getString(ERROR);

//could return error message in a bundle or just throw an error

throw new CLPException(errorCause);

}

else

{

//neither location nor error so a clp problem

throw new CLPException("Missing location and error objects from CLP response");

}

}

else

{

JSONObject jsonLocation = locationData.getJSONObject(LOCATION);

//we have a good location object from the CLP

//load the location object with the locationData

l = new Location(WIMAX);

l.setLatitude(jsonLocation.getDouble(LATITUDE));

l.setLongitude(jsonLocation.getDouble(LONGITUDE));

l.setAccuracy((float) jsonLocation.getDouble(ACCURACY));

}

}

}

finally

{

//ensure call time is recorded

mLastCall = new Date();

try

{

if (entity != null)

{

entity.consumeContent();

input.close(); //may not be needed

}

} catch (IOException e) {

// TODO Auto-generated catch block

e.printStackTrace();

}

}

//can return null in case of error. if this happens, do not update the fix- keep old one.

if (l == null)

{

throw new CLPException("unable to provide location from CLP");

}

else

{

mLastFix = l;

mLastFix.setTime(new Date().getTime());

Log.d("locobird", "returning new fix");

return l;

}

}

//if time since last fix exceed STALE, the fix is stale so get a new one.

private boolean isStale()

{

return ((new Date().getTime() - mLastFix.getTime()) > STALE);

}

}

For checking if you are on 4G, here's the code.

public static boolean onWimax(ConnectivityManager cm)

{

NetworkInfo info = cm.getActiveNetworkInfo();

if (info == null)

{

//no active network=

return false;

}

int networkType = cm.getActiveNetworkInfo().getType();

//type = 6 means wimax

if((networkType == 6) && (cm.getActiveNetworkInfo().getDetailedState().compareTo(DetailedState.CONNECTED)) == 0)

{

return true;

}

else

{

return false;

}

}

You will need these imports:

import android.net.ConnectivityManager;

import android.net.NetworkInfo;

import android.net.NetworkInfo.DetailedState;