51Degrees Pipeline for Python

fiftyone_pipeline_engines_fiftyone.share_usage.ShareUsage Class Reference

Inheritance diagram for fiftyone_pipeline_engines_fiftyone.share_usage.ShareUsage:

[legend]

Collaboration diagram for fiftyone_pipeline_engines_fiftyone.share_usage.ShareUsage:

[legend]

Public Member Functions

 __init__ (self, interval=1200, requested_package_size=REQUESTED_PACKAGE_SIZE_DEFAULT, cookie=None, query_whitelist=[], header_blacklist=[], share_percentage=1, endpoint="https://devices-v4.51degrees.com/new.ashx", request_timeout=REQUEST_TIMEOUT, share_all=False, track_session=False, max_queue_size=None, failures_to_enter_recovery=FAILURES_TO_ENTER_RECOVERY_DEFAULT, failures_window_seconds=FAILURES_WINDOW_SECONDS_DEFAULT, initial_recovery_delay_seconds=INITIAL_RECOVERY_DELAY_SECONDS_DEFAULT, max_recovery_delay_seconds=MAX_RECOVERY_DELAY_SECONDS_DEFAULT, recovery_multiplier=RECOVERY_MULTIPLIER_DEFAULT)
 Constructor for ShareUsage element.
 __del__ (self)
 close (self)
 Wait for any usage data already being sent, then send what is still queued and stop the threads that send it.
 get_constant_xml (self)
 get_flow_elements (self)
 get_evidence_key_filter (self)
 The usage sharing element accepts no evidence.
 found_evidence (self, flow_data)
 The evidence the element finds to share.
 share_send_usage (self)
 Internal method to start sending the queued usage data to the 51Degrees servers.
 send_thread (self, endpoint, data)
 Send one message of usage data.
 add_to_share_usage (self, data)
 Internal method which adds to the share usage bundle (generating XML) @type key: dict.
 process_internal (self, flow_data)
 Internal process method which uses the ShareUsageTracker to determine whether to add usage data to a batch and adds it if necessary.
 get_data_from_evidence (self, flow_data)
 Creates a ShareUsageData instance populated from the evidence within the flow data provided.

Public Member Functions inherited from fiftyone_pipeline_engines.engine.Engine

 __init__ (self, data_file=None)
 Constructor for an engine.
 set_cache (self, cache)
 Add a cache to an engine @type cashe: Cache.
 set_restricted_properties (self, properties_list)
 Add a subset of properties.
 is_cacheable (self, data)
 Whether a result may be stored in the cache and served from it.
 in_cache (self, flowdata)
 A method to check if a flowdata's evidence is in the cache.
 refresh (self, identifier)
 process (self, flowdata)
 Engine's core process function.
 on_registration (self, pipeline)
 Called when an engine is registered with a pipeline and if there is a data file, a data file update service is attached to the parent pipeline.
 register_data_file (self, data_file)
 Register a data_file of the DataFile class with the engine @type data_file: DataFile.

Public Member Functions inherited from fiftyone_pipeline_core.flowelement.FlowElement

 __init__ (self)
 List of Pipelines the FlowElement has been added to.
 filter_evidence (self, flowdata)
 Filter FlowData evidence using the FlowElement's EvidenceKeyFilter.
 filter_evidence_key (self, key)
 Filter FlowData.evidence using the flowElement's EvidenceKeyFilter with the property key of evidence of interest.
 get_properties (self)
 Get the FlowElement.properties of a FlowElement.

Public Attributes

 query_whitelist = query_whitelist
 header_blacklist = header_blacklist
 interval = interval
 requested_package_size = requested_package_size
 cookie = cookie
 share_percentage = share_percentage
 endpoint = endpoint
 request_timeout = request_timeout
 share_all = share_all
 track_session = track_session
 evidence_key_filter = BasicListEvidenceKeyFilter([])
 data_evidence_key_filter
 tracker_evidence_key_filter
 tracker = ShareUsageTracker(interval = interval)
list share_data = []
 max_queue_size
 backoff
 cleanup_timeout_seconds = CLEANUP_TIMEOUT_SECONDS
 constant_xml = None
 thread_pool

Public Attributes inherited from fiftyone_pipeline_engines.engine.Engine

 data_file = data_file
 cache = cache
 restricted_properties = properties_list
dict restricted_property_keys

Public Attributes inherited from fiftyone_pipeline_core.flowelement.FlowElement

list pipelines = []
dict properties = {}
str datakey = ""
bool exclude_from_messages = False

Protected Member Functions

 _send_remaining (self, deadline, done)
 Send the entries still queued when closing, even if there are fewer than the requested package size.
 _schedule_resume (self, pause)
 Start sending when the pause ends.
 _resume (self)
 _send_while_queued (self)
 Send messages of up to twice requested_package_size entries until fewer than requested_package_size are queued, or a failed send pauses sending.
 _send_messages (self)

Protected Member Functions inherited from fiftyone_pipeline_core.flowelement.FlowElement

 _log (self, level, message)

Protected Attributes

int _discarded = 0
int _message_size = 2 * requested_package_size
 _lock = threading.Lock()
bool _sending = False
 _resume_timer = None
 _closing = threading.Event()
 _send_while_queued

Constructor & Destructor Documentation

◆ __init__()

fiftyone_pipeline_engines_fiftyone.share_usage.ShareUsage.__init__ ( self,
interval = 1200,
requested_package_size = REQUESTED_PACKAGE_SIZE_DEFAULT,
cookie = None,
query_whitelist = [],
header_blacklist = [],
share_percentage = 1,
endpoint = "https://devices-v4.51degrees.com/new.ashx",
request_timeout = REQUEST_TIMEOUT,
share_all = False,
track_session = False,
max_queue_size = None,
failures_to_enter_recovery = FAILURES_TO_ENTER_RECOVERY_DEFAULT,
failures_window_seconds = FAILURES_WINDOW_SECONDS_DEFAULT,
initial_recovery_delay_seconds = INITIAL_RECOVERY_DELAY_SECONDS_DEFAULT,
max_recovery_delay_seconds = MAX_RECOVERY_DELAY_SECONDS_DEFAULT,
recovery_multiplier = RECOVERY_MULTIPLIER_DEFAULT )

Constructor for ShareUsage element.

@type interval: int

Parameters
intervalIf exactly the same evidence values are seen multiple times within this time limit (in seconds) then they will only be shared once. @type requested_package_size: int
requested_package_sizeThe usage element will group data into single requests before sending it. This setting controls the minimum number of entries before data is sent, and a message holds from this to twice this number. 50 by default. If you are sharing large amounts of data, increasing this value is recommended in order to reduce the overhead of sending HTTP messages. For example, the 51Degrees cloud service uses a value of 2500. @type cookie: string
cookieThe name of the cookie that holds the session id. It has no effect unless track_session is True. The cookie itself is not shared unless share_all is True. @type track_session: bool
track_sessionIf True, the cookie named by cookie is used to tell user sessions apart, so evidence already shared is shared again when it comes from a new session. False by default. It needs cookie to be set, and has no effect when share_all is True, because every cookie is then shared and used to find repeats. @type query_whitelist: list
query_whitelistBy default query string and HTTP form parameters are not shared unless prefixed with '51D_'. If you need to share query string parameters, a list can be specified here. None shares every parameter. @type header_blacklist: list
header_blacklistBy default, all HTTP headers except Cookie are shared. Individual headers can be excluded from sharing by adding them to this list. @type share_percentage : float
share_percentageapproximate proportion of requests to be shared. 1 = 100%, 0.5 = 50%, etc.. @type share_all: bool
share_allIf True, all evidence is shared apart from the email address, the salt used for hashed email identifiers, and the request's query string and path. False by default. @type max_queue_size: int
max_queue_sizeThe most entries held while waiting to be sent. When the queue is full, new entries are dropped. The default is 1000, or ten times requested_package_size if that is larger. @type failures_to_enter_recovery: int
failures_to_enter_recoveryThe number of failed sends within failures_window_seconds that pauses sending, from 1 to 100. Usage data is still queued while sending is paused. 10 by default. @type failures_window_seconds: int
failures_window_secondsThe time in seconds over which failed sends are counted. 100 by default. @type initial_recovery_delay_seconds: float
initial_recovery_delay_secondsThe length in seconds of the first pause after failed sends. 2.0 by default. @type max_recovery_delay_seconds: float
max_recovery_delay_secondsThe longest a pause can be, in seconds. 300.0 by default. @type recovery_multiplier: float
recovery_multiplierWhat the pause is multiplied by each time the send tried when a pause ends fails. 2.0 by default.

◆ __del__()

fiftyone_pipeline_engines_fiftyone.share_usage.ShareUsage.__del__ ( self)

Member Function Documentation

◆ _resume()

fiftyone_pipeline_engines_fiftyone.share_usage.ShareUsage._resume ( self)
protected

◆ _schedule_resume()

fiftyone_pipeline_engines_fiftyone.share_usage.ShareUsage._schedule_resume ( self,
pause )
protected

Start sending when the pause ends.

Called with the lock held.

◆ _send_messages()

fiftyone_pipeline_engines_fiftyone.share_usage.ShareUsage._send_messages ( self)
protected

◆ _send_remaining()

fiftyone_pipeline_engines_fiftyone.share_usage.ShareUsage._send_remaining ( self,
deadline,
done )
protected

Send the entries still queued when closing, even if there are fewer than the requested package size.

A send already running finishes first, so only one message goes out at a time. Nothing is sent while paused, after a message fails, or once deadline has passed, and the entries left are discarded. Closing logs the ones left when the time ran out, and this logs the ones left for the other two reasons.

@type deadline: float

Parameters
deadlineThe monotonic time sending has to stop at. @type done: threading.Event
doneSet when sending stops for a reason other than the deadline.

◆ _send_while_queued()

fiftyone_pipeline_engines_fiftyone.share_usage.ShareUsage._send_while_queued ( self)
protected

Send messages of up to twice requested_package_size entries until fewer than requested_package_size are queued, or a failed send pauses sending.

◆ add_to_share_usage()

fiftyone_pipeline_engines_fiftyone.share_usage.ShareUsage.add_to_share_usage ( self,
data )

Internal method which adds to the share usage bundle (generating XML) @type key: dict.

Parameters
keydata value store of current evidence in FlowData

◆ close()

fiftyone_pipeline_engines_fiftyone.share_usage.ShareUsage.close ( self)

Wait for any usage data already being sent, then send what is still queued and stop the threads that send it.

The pipeline calls this when it is closed. Closing does not wait for a pause after failed sends, and sends nothing while paused.

Closing gives up after cleanup_timeout_seconds, discards the entries still queued and logs a warning, so that it does not hold up the process that closes it. A message already going out when that time passes is left to the request timeout on its own thread.

Reimplemented from fiftyone_pipeline_engines.engine.Engine.

◆ found_evidence()

fiftyone_pipeline_engines_fiftyone.share_usage.ShareUsage.found_evidence ( self,
flow_data )

The evidence the element finds to share.

These are the values of the web request, when the web integration added it as server.http-request, and every evidence value in the flow data, whether an element accepted it or not. A key found in both places takes the evidence value. The web request itself is never one of them.

@type flow_data: FlowData

Parameters
flow_dataThe flow data being processed @rtype: dict
Returns
: The evidence found, before the sharing rules apply

◆ get_constant_xml()

fiftyone_pipeline_engines_fiftyone.share_usage.ShareUsage.get_constant_xml ( self)

◆ get_data_from_evidence()

fiftyone_pipeline_engines_fiftyone.share_usage.ShareUsage.get_data_from_evidence ( self,
flow_data )

Creates a ShareUsageData instance populated from the evidence within the flow data provided.

Parameters
{FlowData}flowData the flow data containing the evidence to use
Returns
a new ShareUsageData instance, populated from the evidence provided

◆ get_evidence_key_filter()

fiftyone_pipeline_engines_fiftyone.share_usage.ShareUsage.get_evidence_key_filter ( self)

The usage sharing element accepts no evidence.

See found_evidence for what it shares.

Reimplemented from fiftyone_pipeline_core.flowelement.FlowElement.

◆ get_flow_elements()

fiftyone_pipeline_engines_fiftyone.share_usage.ShareUsage.get_flow_elements ( self)

◆ process_internal()

fiftyone_pipeline_engines_fiftyone.share_usage.ShareUsage.process_internal ( self,
flow_data )

Internal process method which uses the ShareUsageTracker to determine whether to add usage data to a batch and adds it if necessary.

@type flow_data: FlowData

Parameters
flow_dataFlowData to process

Reimplemented from fiftyone_pipeline_core.flowelement.FlowElement.

◆ send_thread()

fiftyone_pipeline_engines_fiftyone.share_usage.ShareUsage.send_thread ( self,
endpoint,
data )

Send one message of usage data.

Returns
: True if the service accepted it

◆ share_send_usage()

fiftyone_pipeline_engines_fiftyone.share_usage.ShareUsage.share_send_usage ( self)

Internal method to start sending the queued usage data to the 51Degrees servers.

Nothing starts if a thread is already sending, if fewer than requested_package_size entries are queued, or while sending is paused after failed sends, in which case a timer starts sending when the pause ends.

Member Data Documentation

◆ _closing

fiftyone_pipeline_engines_fiftyone.share_usage.ShareUsage._closing = threading.Event()
protected

◆ _discarded

int fiftyone_pipeline_engines_fiftyone.share_usage.ShareUsage._discarded = 0
protected

◆ _lock

fiftyone_pipeline_engines_fiftyone.share_usage.ShareUsage._lock = threading.Lock()
protected

◆ _message_size

int fiftyone_pipeline_engines_fiftyone.share_usage.ShareUsage._message_size = 2 * requested_package_size
protected

◆ _resume_timer

fiftyone_pipeline_engines_fiftyone.share_usage.ShareUsage._resume_timer = None
protected

◆ _send_while_queued

fiftyone_pipeline_engines_fiftyone.share_usage.ShareUsage._send_while_queued
protected

◆ _sending

bool fiftyone_pipeline_engines_fiftyone.share_usage.ShareUsage._sending = False
protected

◆ backoff

fiftyone_pipeline_engines_fiftyone.share_usage.ShareUsage.backoff
Initial value:
= ShareUsageBackOff(
failures_to_enter_recovery,
failures_window_seconds,
initial_recovery_delay_seconds,
max_recovery_delay_seconds,
recovery_multiplier)

◆ cleanup_timeout_seconds

fiftyone_pipeline_engines_fiftyone.share_usage.ShareUsage.cleanup_timeout_seconds = CLEANUP_TIMEOUT_SECONDS

◆ constant_xml

fiftyone_pipeline_engines_fiftyone.share_usage.ShareUsage.constant_xml = None

◆ cookie

fiftyone_pipeline_engines_fiftyone.share_usage.ShareUsage.cookie = cookie

◆ data_evidence_key_filter

fiftyone_pipeline_engines_fiftyone.share_usage.ShareUsage.data_evidence_key_filter
Initial value:
= ShareUsageEvidenceKeyFilter(
cookie, included, header_blacklist, False, share_all)

◆ endpoint

fiftyone_pipeline_engines_fiftyone.share_usage.ShareUsage.endpoint = endpoint

◆ evidence_key_filter

fiftyone_pipeline_engines_fiftyone.share_usage.ShareUsage.evidence_key_filter = BasicListEvidenceKeyFilter([])

◆ header_blacklist

fiftyone_pipeline_engines_fiftyone.share_usage.ShareUsage.header_blacklist = header_blacklist

◆ interval

fiftyone_pipeline_engines_fiftyone.share_usage.ShareUsage.interval = interval

◆ max_queue_size

fiftyone_pipeline_engines_fiftyone.share_usage.ShareUsage.max_queue_size
Initial value:
= max_queue_size if max_queue_size is not None \
else max(MAX_QUEUE_SIZE_DEFAULT, 10 * requested_package_size)

◆ query_whitelist

fiftyone_pipeline_engines_fiftyone.share_usage.ShareUsage.query_whitelist = query_whitelist

◆ request_timeout

fiftyone_pipeline_engines_fiftyone.share_usage.ShareUsage.request_timeout = request_timeout

◆ requested_package_size

fiftyone_pipeline_engines_fiftyone.share_usage.ShareUsage.requested_package_size = requested_package_size

◆ share_all

fiftyone_pipeline_engines_fiftyone.share_usage.ShareUsage.share_all = share_all

◆ share_data

fiftyone_pipeline_engines_fiftyone.share_usage.ShareUsage.share_data = []

◆ share_percentage

fiftyone_pipeline_engines_fiftyone.share_usage.ShareUsage.share_percentage = share_percentage

◆ thread_pool

fiftyone_pipeline_engines_fiftyone.share_usage.ShareUsage.thread_pool
Initial value:
= ThreadPoolExecutor(
max_workers=1, thread_name_prefix='ShareUsage')

◆ track_session

fiftyone_pipeline_engines_fiftyone.share_usage.ShareUsage.track_session = track_session

◆ tracker

fiftyone_pipeline_engines_fiftyone.share_usage.ShareUsage.tracker = ShareUsageTracker(interval = interval)

◆ tracker_evidence_key_filter

fiftyone_pipeline_engines_fiftyone.share_usage.ShareUsage.tracker_evidence_key_filter
Initial value:
= \
ShareUsageTrackerEvidenceKeyFilter(
cookie, included, header_blacklist, track_session,
share_all)