Guest Post - Carlos Abalde (CTO at http://dot2code.com)
Nowadays every serious web project, sooner rather than later, needs to face the fact that a website is going to be visited by a wide range of different devices. Ensuring the correct rendering and operation in a few different desktop browsers is no longer enough. The number of possible web clients grows larger day after day, and specialized tools are required to deal with the problem in an effective way.
Among these tools stands out the server-side device detectors. Use cases of this type of tool range from adaptation of content based on the simple categorization of devices accessing a website into mobile, tablet & desktop categories and even identification of detailed characteristics of devices (such as brand, model or screen size) for further analysis and strategic decision making.
Despite the importance of the problem, there is a lack of high quality device detection products. This is specially truth if you are looking for an affordable real-time 100% server side product (i.e. not a slower cloud-based solution), periodically updated to ensure the accuracy of all detections and open source to safely include it in your back-end. If, on top of that, Python is the target platform, you'll shortly notice that a service with those requirements simply does not exist.
Luckily, the 51Degrees.mobi device detection solution, available in C, PHP, .NET and Java platforms, has been recently ported to Python.
This article presents the solution, introduces the different Python packages composing the product and shows the implementation of a couple of simple use cases.
51Degrees.mobi meets Python
51Degrees.mobi mobile detector for Python (2.6) allows fetching properties of devices based on http headers and user agent strings. The solution is composed by several packages available in the Python Package Index (PyPI) repository:the core package (51degreesmobiledetector) and two packages containing specific detection methods: pattern (51degreesmobiledetectorlitepatternwrapper) and trie (51degreesmobiledetectortriewrapper).
Both detection methods are Python wrappers of very efficient and extensively tested C implementations. On one hand, the pattern method is the same matching method used by the .NET, Java and PHP solutions. All the necessary data and regular expressions required when fetching properties of mobile devices is embedded in the source code of the method. This results in a self contained and fast implementation with modest working memory requirements.
On the other hand, the trie method does not embed the required data in the source code. This method depends on an external database file. A completely free and periodically updated database can be download from the 51Degrees.mobi website, however if you want weekly updates and extra device properties you can upgrade to the premium version.
The trie method is slightly less memory efficient than the pattern method, however, the method is exceptionally fast, achieving millions of detections per second.
All three packages are completely free and open source projects. Depending on their requirements, developers can upgrade to premium versions of the pattern and trie detection methods. Premium pattern detection is available by downloading and installing an additional Python package (51degreesmobiledetectorpremiumpatternwrapper). Premium trie detection is available by upgrading and using the premium device detection data. Both files are available for download at the 51Degrees.mobi website for developers owning a valid license key.
Quickstart
Let's start using lite versions of both detection methods. Check you have a C compiler and the Python development headers installed in your system, create a test virtualenv and install the detection packages using pip:
# virtualenv /tmp/51degrees
# source /tmp/51degrees/bin/activate
# pip install 51degrees-mobile-detector-trie-wrapper 51degrees-mobile-detector-lite-pattern-wrapper
Note that the core package will be automatically installed as a dependency. Note also that this may take some time. The pattern package embeds a complex and large C implementation that will be compiled on installation time.
Before starting to detect mobile devices, you need to download the trie data file (required by the trie method) and to configure the solution. You can easily generate a sample settings file running the following command:
# 51degrees-mobile-detector settings > ~/51degrees.settings.py
Edit the generated settings and set your preferences. Ensure you set at least set DETECTION_METHOD to 'litepatternwrapper' or 'triewrapper', and set TRIE_WRAPPER_DATABASE to point to the path of the unzipped lite trie database file.
Finally, link your settings file from the 51Degrees.mobi environment variable. Otherwise the detection solution will not be able to find your preferences:
# export FIFTYONE_DEGREES_MOBILE_DETECTOR_SETTINGS=~/51degrees.settings.py
Now you are ready to start matching user agent strings. Open a Python console, load the core module and start matching:
# python
>>> from fiftyone_degrees import mobile_detector
>>> device = mobile_detector.match('Mozilla/5.0 (iPad; CPU OS 5_1 like Mac OS X) AppleWebKit/534.46 (KHTML, like Gecko) Mobile/9B176')
>>> device.IsMobile
'True'
>>> device.ScreenPixelsWidth
'768'
>>> device.method
'litepatternwrapper'
>>> device.properties
{'PostMessage': 'True', 'IndexedDB': 'False', 'Canvas': 'True', 'GeoLocation':
'True', 'FileSaver': 'False', 'CssUI': 'True', 'CssTransforms': 'True',
'DataSet': 'False', 'WebWorkers': 'False', 'Json': 'True',
'ScreenPixelsHeight': '1024', 'CssImages': 'False' ...
Note that you can override the detection method set in the settings file using the method argument. For example:
>>> device = mobile_detector.match('Mozilla/5.0 (iPad; CPU OS 5_1 like Mac OS
X) AppleWebKit/534.46 (KHTML, like Gecko) Mobile/9B176', method='triewrapper')
>>> device.method
'triewrapper'
>>> device.IsMobile
'True'
Example #1: Log analyzer
Imagine you have been collecting user agent strings from log files over a one month period after you launched your mobile-friendly website. You have now decided to include some advertisements on your site. You know your users are particularly interested in mobile accessories, but you have no idea about the brands of mobile devices they have been using to visit your site.
Let's create a toy Python project to go through a list of user agent strings and build a ranking of mobile brands that are visiting your site. This way you'll have all the information you need to make an informed decision about the most suitable advertisements.
The project hierarchy is simple:
acme/
__init__.py
fiftyone_degrees_settings.py
This example will use an alternative method to configure the 51Degrees.mobi mobile detector. This time the settings file has been embedded into the project as a normal Python module.
Next the contents of acme/fiftyone_degrees_settings.py are shown. PROPERTIES has been used to restrict the set of fetched device properties to only two: IsMobile and HardwareVendor. This is not mandatory, but this way, matching of user agent strings will be even faster. Note also that a premium detection method is used in this example. This is required because the HardwareVendor property is not available in lite versions of the detectors:
DETECTION_METHOD = 'premiumpatternwrapper'
PROPERTIES = ('IsMobile', 'HardwareVendor',)
The rest of the implementation can be found in acme/__init__.py:
import os
import sys
import collections
def main():
# Load 51Degrees.mobi mobile detector module.
from fiftyone_degrees import mobile_detector
# Aggregate number of visits by hardware vendor.
ranking = collections.defaultdict(int)
with open('/tmp/user_agents.log') as f:
for user_agent in f:
device = mobile_detector.match(user_agent)
if device.IsMobile == 'True':
ranking[device.HardwareVendor] += 1
# Short & print the ranking of hardware vendors.
ranking = sorted(ranking.items(), key=lambda item: item[1], reverse=True)
for vendor, visits in ranking:
print '%s: %d' % (vendor, visits,)
if __name__ == '__main__':
# Adjust Python path & set 51Degrees.mobi settings file location. Beware of
# doing this *before* loading the mobile detector module.
sys.path.append(
os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
os.environ.setdefault(
'FIFTYONE_DEGREES_MOBILE_DETECTOR_SETTINGS',
'acme.fiftyone_degrees_settings')
# Run the main program.
main()
Finally, run the script and get the ranking of most popular mobile brands visiting your site. Now you can choose the most adequate advertisements for your site:
# python acme/__init__.py
Apple: 76544
HTC: 32343
Samsung: 30456
Nokia: 12303
...
Example #2: Mobile site redirection
This second example shows how to use 51Degrees.mobi to detect mobile devices to redirect visitors to a lightweight mobile version of your website. Imagine you have built your site using the most popular Python web framework: Django. You need to offer an optimised mobile experience by creating a optimised mobile version of your website. Mobile users should be automatically redirected to m.example.com when visiting example.com. Once redirected to the mobile site users may choose to go back to the normal site. This decision should be remembered to avoid annoying further automatic redirections to the simplified site.
The core package of 51Degrees.mobi Python mobile detector includes a middleware and context processor designed to simplify the integration as much as possible into the detection solution in Django platforms. First of all, you need to add the 51Degrees.mobi middleware to your Django settings. Inserting it just after the SessionMiddleware is a safe place:
MIDDLEWARE_CLASSES = (
...
'django.contrib.sessions.middleware.SessionMiddleware',
'fiftyone_degrees.mobile_detector.contrib.django.middleware.DetectorMiddleware',
...
)
Next, you need to configure the device detector. When integrating the detector in a Django project, there is a more convenient option to insert your preferences directly into the Django settings file:
FIFTYONE_DEGREES_MOBILE_DETECTOR_SETTINGS = {
'DETECTION_METHOD': 'lite-pattern-wrapper',
'PROPERTIES': ('IsMobile',),
}
Apart from the standard settings, there are some Django specific settings available to fine tune the behavior of the 51Degrees.mobi middleware. By default, detected device properties will be cached in the user's session to execute the detection logic only once per user session. In this case, this is not useful because a specific caching logic we'll be later implemented, so let's disable it adding the following line to the Django settings file:
FIFTYONE_DEGREES_MOBILE_DETECTOR_SESSION_CACHE = False
Finally, the example site will be running in two different domains (example.com and m.example.com). Therefore, the Django settings needs to be adjusted to use crossdomain session cookies:
SESSION_COOKIE_DOMAIN = '.example.com'
Once you have updated the Django settings file and restarted the HTTP server, every Request instance will contain a generated device attribute with the properties of the device visiting the site. Now it is time to use that information to automatically redirect mobile users to m.example.com.
Let's implement a new middleware (MobileRedirector) and insert it just after the 51Degrees.mobi middleware in the Django settings file:
MIDDLEWARE_CLASSES = (
...
'django.contrib.sessions.middleware.SessionMiddleware',
'fiftyone_degrees.mobile_detector.contrib.django.middleware.DetectorMiddleware',
'acme.middleware.MobileRedirector',
...
)
The implementation of the new redirection middleware is straightforward:
from django.http import HttpResponseRedirect
class MobileRedirector(object):
MOBILE_DOMAIN = 'm.example.com'
def process_request(self, request):
# Ensure 'skip-redirector' attribute exists in user's session.
if 'skip-redirector' not in request.session:
request.session['skip-redirector'] = False
# If skip of automatic redirections has been requested,
# persistently store the preference in the user's session.
if 'skip-redirector' in request.GET:
request.session['skip-redirector'] = True
# If the redirector has not been disabled, execute the
# redirection logic.
if not request.session['skip-redirector']:
# Ensure 'is_mobile' attribute exists in user's session.
if 'is-mobile' not in request.session:
request.session['is-mobile'] =
request.device.IsMobile == 'True'
# If required redirect to the mobile site. Note that in a
# real case the requesting HTTP method should be checked
# (GET, POST, etc.), the requesting protocol should be
# taken into account (HTTP vs HTTPS), the requesting HTTP
# GET & POST query string may have to be preserved, etc.
if request.session['is-mobile'] and \
request.META['HTTP_HOST'] != self.MOBILE_DOMAIN:
return HttpResponseRedirect(
'http://' + self.MOBILE_DOMAIN + request.path)
A special key in the HTTP GET query string (skipredirector) is used to skip the mobile redirector execution. Typically you'll insert a 'view normal site' link including that parameter (e.g. http://example.com/somecontent?skipredirector) in the footer of all mobile templates. Once clicked, the visitor will never be automatically redirected to the mobile site again, and that preference will be remembered in further visits.
Note this is just sample middleware. For example, it assumes all possible URLs in the normal site are also available in the mobile site. This is a recommended practice but may not be the truth for your site. Additionally, correct handling of different HTTP methods, protocols, etc. should be included in a production implementation.
Conclusions
This article has summarized the main features of the new 51Degrees.mobi mobile detector for Python. The product offers a high performance, accurate and periodically updated 100% server side solution to detect mobile devices. Both the lite and premium versions are available depending on your requirements. And, in all cases, the source code of all these modules is publicly available, so you can safely integrate the solution in your backend.
In summary, a excellent product that enables the Python community the power of delivering customized and optimised mobile web browsing experiences.