\r\n

51Degrees Device Detection Python  4.4

Device Detection services for 51Degrees Pipeline

onpremise/datafileupdate_console.py

This example illustrates various parameters that can be adjusted when using the on-premise device detection engine, and controls when a new data file is sought and when it is loaded by the device detection software. Three main aspects are demonstrated:

  • Update on Start-Up
  • Filesystem Watcher
  • Daily auto-update

License Key

In order to test this example you will need a 51Degrees Enterprise license which can be purchased from our pricing page. Look for our "Bigger" or "Biggest" options.

Data Files

You can find out more about data files, licenses etc. at our (FAQ page)[//51degrees.com/resources/faqs]

Enterprise Data File

Enterprise (fully-featured) data files are typically released by 51Degrees four days a week (Mon-Thu) and on-premise deployments can fetch and download those files automatically. Equally, customers may choose to download the files themselves and move them into place to be detected by the 51Degrees filesystem watcher.

Manual Download

If you prefer to download files yourself, you may do so here:

1 https://distributor.51degrees.com/api/v2/download?LicenseKeys=<your_license_key>&Type=27&Download=True&Product=22

Lite Data File

Lite data files (free-to-use, limited capabilities, no license key required) are created roughly once a month and cannot be updated using auto-update, they may be downloaded from (Github)[href=https://github.com/51Degrees/device-detection-data] and are included with source distributions of this software.

Update on Start-Up

You can configure the pipeline builder to download an Enterprise data file on start-up.

Pre-Requisites

  • a license key
  • a file location for the download
    • this may be an existing file - which will be overwritten
    • or if it does not exist must end in ".hash" and must be in an existing directory

      Configuration

  • the pipeline must be configured to use a temp file
    1 create_temp_copy = True,
  • a DataFileUpdateService must be supplied
    1  update_event = UpdateEvent()
    2  update_service = DataFileUpdateService()
    3  update_service.on_complete(lambda status, file: update_event.set(status))
    4 ...
    5  data_file_update_service = update_service,
  • update on start-up must be specified, which will cause pipeline creation to block until a file is downloaded
    1 update_on_start = True,

File System Watcher

You can configure the pipeline builder to watch for changes to the currently loaded device detection data file, and to replace the file currently in use with the new one. This is useful, for example, if you wish to download and update the device detection file "manually" - i.e. you would download it then drop it into place with the same path as the currently loaded file. That location is checked periodically (by default every 30 mins) and this frequency can be configured.

Pre-Requisites

  • a license key
  • the file location of the existing file

    Configuration

  • the pipeline must be configured to use a temp file
    1 create_temp_copy = True,
  • a DataFileUpdateService must be supplied
    1  update_event = UpdateEvent()
    2  update_service = DataFileUpdateService()
    3  update_service.on_complete(lambda status, file: update_event.set(status))
    4 ...
    5  data_file_update_service = update_service,
  • configure the frequency with which the location is checked, in seconds (10 mins as shown)
    1 polling_interval = (10*60),

Daily auto-update

Enterprise data files are usually created four times a week. Each data file contains a date for when the next data file is expected. You can configure the pipeline so that it starts looking for a newer data file after that time, by connecting to the 51Degrees distributor to see if an update is available. If one is, then it is downloaded and will replace the existing device detection file, which is currently in use.

Pre-Requisites

  • a license key
  • the file location of the existing file

    Configuration

  • the pipeline must be configured to use a temp file
    1 create_temp_copy = True,
  • a DataFileUpdateService must be supplied
    1  update_event = UpdateEvent()
    2  update_service = DataFileUpdateService()
    3  update_service.on_complete(lambda status, file: update_event.set(status))
    4 ...
    5  data_file_update_service = update_service,
  • Set the frequency in seconds that the pipeline should check for updates to data files. A recommended polling interval in a production environment is around 30 minutes.
    1 polling_interval = (10*60),
  • Set the max amount of time in seconds that should be added to the polling interval. This is useful in datacenter applications where multiple instances may be polling for updates at the same time. A recommended ammount in production environments is 600 seconds.
    1 update_time_maximum_randomisation = (10*60),

Location

This example is available in full on GitHub.

This example requires a subscription to 51Degrees Device Data, a subscription can be acquired from the 51Degrees pricing page.

Required PyPi Dependencies:

  • fiftyone_devicedetection
1 # *********************************************************************
2 # This Original Work is copyright of 51 Degrees Mobile Experts Limited.
3 # Copyright 2022 51 Degrees Mobile Experts Limited, Davidson House,
4 # Forbury Square, Reading, Berkshire, United Kingdom RG1 3EU.
5 #
6 # This Original Work is licensed under the European Union Public Licence
7 # (EUPL) v.1.2 and is subject to its terms as set out below.
8 #
9 # If a copy of the EUPL was not distributed with this file, You can obtain
10 # one at https://opensource.org/licenses/EUPL-1.2.
11 #
12 # The 'Compatible Licences' set out in the Appendix to the EUPL (as may be
13 # amended by the European Commission) shall be deemed incompatible for
14 # the purposes of the Work and the provisions of the compatibility
15 # clause in Article 5 of the EUPL shall not apply.
16 #
17 # If using the Work as, or as part of, a network application, by
18 # including the attribution notice(s) required under Article 5 of the EUPL
19 # in the end user terms of the application under an appropriate heading,
20 # such notice(s) shall fulfill the requirements of that article.
21 # *********************************************************************
22 
23 
158 
159 from datetime import datetime
160 import os
161 import shutil
162 import sys
163 import threading
164 from fiftyone_devicedetection.devicedetection_pipelinebuilder import DeviceDetectionPipelineBuilder
165 from fiftyone_devicedetection_examples.example_utils import ExampleUtils
166 from fiftyone_devicedetection_shared.key_utils import KeyUtils
167 from fiftyone_pipeline_core.logger import Logger
168 from fiftyone_devicedetection_shared.example_constants import ENTERPRISE_DATAFILE_NAME
169 from fiftyone_pipeline_engines.datafile_update_service import DataFileUpdateService
170 from fiftyone_pipeline_engines.datafile_update_service import UpdateStatus
171 
172 UPDATE_EXAMPLE_LICENSE_KEY_NAME = "license_key"
173 DEFAULT_DATA_FILENAME = os.path.expanduser("~") + os.path.sep + ENTERPRISE_DATAFILE_NAME
174 
175 class UpdateEvent(threading.Event):
176  def __init__(self):
177  self.status = None
178  super().__init__()
179 
180  def set(self, status):
181  super().set()
182  self.status = status
183 
184  def clear(self):
185  self.status = None
186  super().clear()
187 
188 class DataFileUpdateConsole():
189  def run(self, data_file, license_key, interactive, logger, output):
190  logger.log("info", "Starting example")
191 
192  # try to find a license key
193  if (license_key == None):
194  license_key = KeyUtils.get_named_key(UPDATE_EXAMPLE_LICENSE_KEY_NAME)
195 
196  if (license_key == None or KeyUtils.is_invalid_key(license_key)):
197  logger.log("error",
198  "In order to test this example you will need a 51Degrees Enterprise "
199  "license which can be obtained on a trial basis or purchased from our\n"
200  "pricing page http://51degrees.com/pricing. You must supply the license "
201  "key as an argument to this program, or as an environment or system variable "
202  f"named '{UPDATE_EXAMPLE_LICENSE_KEY_NAME}'")
203  raise Exception("No license key available")
204 
205  # work out where the downloaded file will be put, directory must exist
206  if (data_file != None):
207  try:
208  data_file = ExampleUtils.find_file(data_file)
209  except:
210  if (os.path.exists(os.path.dirname(data_file)) == False):
211  logger.log("error",
212  "The directory must exist when specifying a location for a new "
213  f"file to be downloaded. Path specified was '{data_file}'")
214  raise Exception("Directory for new file must exist")
215  logger.log("warning",
216  f"File {data_file} not found, a file will be downloaded to that location on "
217  "start-up")
218 
219  # no filename specified use the default
220  if (data_file == None):
221  data_file = os.path.realpath(DEFAULT_DATA_FILENAME)
222  logger.log("warning",
223  f"No filename specified. Using default '{data_file}' which will be downloaded to "
224  "that location on start-up, if it does not exist already")
225 
226  copy_data_file_name = data_file + ".bak"
227  if (os.path.exists(data_file)):
228  # let's check this file out
229  pipeline = DeviceDetectionPipelineBuilder(
230  data_file_path = data_file,
231  performance_profile = "LowMemory",
232  usage_sharing = False,
233  auto_update = False,
234  licence_keys = "").add_logger(logger).build()
235 
236  # and output the results
237  ExampleUtils.check_data_file(pipeline, logger)
238  if (ExampleUtils.get_data_file_tier(pipeline.get_element("device")) == "Lite"):
239  logger.log("error",
240  "Will not download an 'Enterprise' data file over the top of "
241  "a 'Lite' data file, please supply another location.")
242  raise Exception("File supplied has wrong data tier")
243  logger.log("info", "Existing data file will be replaced with downloaded data file")
244  logger.log("info", f"Existing data file will be copied to {copy_data_file_name}")
245 
246  # do we really want to do this
247  if (interactive):
248  output("Please note - this example will use available downloads "
249  "in your licensed allocation.")
250  user_input = input("Do you wish to continue with this example (y)? ")
251  if (user_input == None or user_input == "" or user_input.startswith("y") == False):
252  logger.log("info", "Stopping example without download")
253  return
254 
255  logger.log("info", "Checking file exists")
256  if os.path.exists(data_file):
257  logger.log("info", f"Existing data file copied to {copy_data_file_name}")
258  shutil.move(data_file, copy_data_file_name)
259 
260  logger.log("info",
261  "Creating pipeline and initiating update on start-up - please wait for that "
262  "to complete");
263 
264  update_event = UpdateEvent()
265  update_service = DataFileUpdateService()
266  update_service.on_complete(lambda status, file: update_event.set(status))
267 
268  # Build the device detection pipeline and pass in the desired settings to configure
269  # automatic updates.
270  pipeline = DeviceDetectionPipelineBuilder(
271  # specify the filename for the data file. When using update on start-up
272  # the file need not exist, but the directory it is in must exist.
273  # Any file that is present is overwritten. Because the file will be
274  # overwritten the pipeline must be configured to copy the supplied
275  # file to a temporary file (create_temp_copy parameter == True).
276  data_file_path = data_file,
277  create_temp_copy = True,
278  # pass in the update listener which has been configured
279  # to notify when update complete
280  data_file_update_service = update_service,
281  # For automatic updates to work you will need to provide a license key.
282  # A license key can be obtained with a subscription from https://51degrees.com/pricing
283  licence_keys = license_key,
284  # Enable update on startup, the auto update system
285  # will be used to check for an update before the
286  # device detection engine is created. This will block
287  # creation of the pipeline.
288  update_on_start = True,
289  # Enable automatic updates once the pipeline has started
290  auto_update = True,
291  # Watch the data file on disk and refresh the engine
292  # as soon as that file is updated.
293  file_system_watcher = True
294  ).add_logger(logger).build()
295 
296 
297  # thread blocks till update checking is complete - or if there is an
298  # exception we don't get this far
299  update_event.wait()
300  output(f"Update on start-up complete - status - {update_event.status}")
301 
302  if update_event.status == UpdateStatus.AUTO_UPDATE_SUCCESS:
303 
304  output("Modifying downloaded file to trigger reload - please wait for that"
305  " to complete");
306 
307  # wait for the dataUpdateService to notify us that it has updated
308  update_event.clear()
309 
310  # it's the same file but changing the file metadata will trigger reload,
311  # demonstrating that if you download a new file and replace the
312  # existing one, then it will be loaded
313  now = datetime.now().timestamp()
314  try:
315  os.utime(data_file, (now, now))
316  except:
317  raise Exception("Could not modify file time, abandoning "
318  "example");
319 
320  if update_event.wait(120):
321  output(f"Update on file modification complete, status: {update_event.status}")
322  else:
323  output("Update on file modification timed out")
324  else:
325  logger.log("error", "Auto update was not successful, abandoning example")
326  raise Exception(f"Auto update failed: {update_event.status}")
327 
328  output("Finished Example")
329 
330 
331 def main(argv):
332  # Use the supplied path for the data file or find the lite
333  # file that is included in the repository.
334  license_key = argv[0] if len(argv) > 0 else None
335  data_file = argv[1] if len(argv) > 1 else None
336 
337  # Configure a logger to output to the console.
338  logger = Logger(min_level="info")
339 
340  DataFileUpdateConsole().run(data_file, license_key, True, logger, print)
341 
342 if __name__ == "__main__":
343  main(sys.argv[1:])