|
| | __init__ (self, interval=1200, requested_package_size=REQUESTED_PACKAGE_SIZE_DEFAULT, cookie=None, query_whitelist=[], header_blacklist=[], share_percentage=1, endpoint="https://devices-v4.51degrees.com/new.ashx", request_timeout=REQUEST_TIMEOUT, share_all=False, track_session=False, max_queue_size=None, failures_to_enter_recovery=FAILURES_TO_ENTER_RECOVERY_DEFAULT, failures_window_seconds=FAILURES_WINDOW_SECONDS_DEFAULT, initial_recovery_delay_seconds=INITIAL_RECOVERY_DELAY_SECONDS_DEFAULT, max_recovery_delay_seconds=MAX_RECOVERY_DELAY_SECONDS_DEFAULT, recovery_multiplier=RECOVERY_MULTIPLIER_DEFAULT) |
| | Constructor for ShareUsage element.
|
| | __del__ (self) |
| | close (self) |
| | Wait for any usage data already being sent, then send what is still queued and stop the threads that send it.
|
| | get_constant_xml (self) |
| | get_flow_elements (self) |
| | get_evidence_key_filter (self) |
| | The usage sharing element accepts no evidence.
|
| | found_evidence (self, flow_data) |
| | The evidence the element finds to share.
|
| | share_send_usage (self) |
| | Internal method to start sending the queued usage data to the 51Degrees servers.
|
| | send_thread (self, endpoint, data) |
| | Send one message of usage data.
|
| | add_to_share_usage (self, data) |
| | Internal method which adds to the share usage bundle (generating XML) @type key: dict.
|
| | process_internal (self, flow_data) |
| | Internal process method which uses the ShareUsageTracker to determine whether to add usage data to a batch and adds it if necessary.
|
| | get_data_from_evidence (self, flow_data) |
| | Creates a ShareUsageData instance populated from the evidence within the flow data provided.
|
| | __init__ (self, data_file=None) |
| | Constructor for an engine.
|
| | set_cache (self, cache) |
| | Add a cache to an engine @type cashe: Cache.
|
| | set_restricted_properties (self, properties_list) |
| | Add a subset of properties.
|
| | is_cacheable (self, data) |
| | Whether a result may be stored in the cache and served from it.
|
| | in_cache (self, flowdata) |
| | A method to check if a flowdata's evidence is in the cache.
|
| | refresh (self, identifier) |
| | process (self, flowdata) |
| | Engine's core process function.
|
| | on_registration (self, pipeline) |
| | Called when an engine is registered with a pipeline and if there is a data file, a data file update service is attached to the parent pipeline.
|
| | register_data_file (self, data_file) |
| | Register a data_file of the DataFile class with the engine @type data_file: DataFile.
|
| | __init__ (self) |
| | List of Pipelines the FlowElement has been added to.
|
| | filter_evidence (self, flowdata) |
| | Filter FlowData evidence using the FlowElement's EvidenceKeyFilter.
|
| | filter_evidence_key (self, key) |
| | Filter FlowData.evidence using the flowElement's EvidenceKeyFilter with the property key of evidence of interest.
|
| | get_properties (self) |
| | Get the FlowElement.properties of a FlowElement.
|
| fiftyone_pipeline_engines_fiftyone.share_usage.ShareUsage.__init__ |
( |
| self, |
|
|
| interval = 1200, |
|
|
| requested_package_size = REQUESTED_PACKAGE_SIZE_DEFAULT, |
|
|
| cookie = None, |
|
|
| query_whitelist = [], |
|
|
| header_blacklist = [], |
|
|
| share_percentage = 1, |
|
|
| endpoint = "https://devices-v4.51degrees.com/new.ashx", |
|
|
| request_timeout = REQUEST_TIMEOUT, |
|
|
| share_all = False, |
|
|
| track_session = False, |
|
|
| max_queue_size = None, |
|
|
| failures_to_enter_recovery = FAILURES_TO_ENTER_RECOVERY_DEFAULT, |
|
|
| failures_window_seconds = FAILURES_WINDOW_SECONDS_DEFAULT, |
|
|
| initial_recovery_delay_seconds = INITIAL_RECOVERY_DELAY_SECONDS_DEFAULT, |
|
|
| max_recovery_delay_seconds = MAX_RECOVERY_DELAY_SECONDS_DEFAULT, |
|
|
| recovery_multiplier = RECOVERY_MULTIPLIER_DEFAULT ) |
Constructor for ShareUsage element.
@type interval: int
- Parameters
-
| interval | If exactly the same evidence values are seen multiple times within this time limit (in seconds) then they will only be shared once. @type requested_package_size: int |
| requested_package_size | The usage element will group data into single requests before sending it. This setting controls the minimum number of entries before data is sent, and a message holds from this to twice this number. 50 by default. If you are sharing large amounts of data, increasing this value is recommended in order to reduce the overhead of sending HTTP messages. For example, the 51Degrees cloud service uses a value of 2500. @type cookie: string |
| cookie | The name of the cookie that holds the session id. It has no effect unless track_session is True. The cookie itself is not shared unless share_all is True. @type track_session: bool |
| track_session | If True, the cookie named by cookie is used to tell user sessions apart, so evidence already shared is shared again when it comes from a new session. False by default. It needs cookie to be set, and has no effect when share_all is True, because every cookie is then shared and used to find repeats. @type query_whitelist: list |
| query_whitelist | By default query string and HTTP form parameters are not shared unless prefixed with '51D_'. If you need to share query string parameters, a list can be specified here. None shares every parameter. @type header_blacklist: list |
| header_blacklist | By default, all HTTP headers except Cookie are shared. Individual headers can be excluded from sharing by adding them to this list. @type share_percentage : float |
| share_percentage | approximate proportion of requests to be shared. 1 = 100%, 0.5 = 50%, etc.. @type share_all: bool |
| share_all | If True, all evidence is shared apart from the email address, the salt used for hashed email identifiers, and the request's query string and path. False by default. @type max_queue_size: int |
| max_queue_size | The most entries held while waiting to be sent. When the queue is full, new entries are dropped. The default is 1000, or ten times requested_package_size if that is larger. @type failures_to_enter_recovery: int |
| failures_to_enter_recovery | The number of failed sends within failures_window_seconds that pauses sending, from 1 to 100. Usage data is still queued while sending is paused. 10 by default. @type failures_window_seconds: int |
| failures_window_seconds | The time in seconds over which failed sends are counted. 100 by default. @type initial_recovery_delay_seconds: float |
| initial_recovery_delay_seconds | The length in seconds of the first pause after failed sends. 2.0 by default. @type max_recovery_delay_seconds: float |
| max_recovery_delay_seconds | The longest a pause can be, in seconds. 300.0 by default. @type recovery_multiplier: float |
| recovery_multiplier | What the pause is multiplied by each time the send tried when a pause ends fails. 2.0 by default. |
| fiftyone_pipeline_engines_fiftyone.share_usage.ShareUsage._schedule_resume |
( |
| self, |
|
|
| pause ) |
|
protected |
Start sending when the pause ends.
Called with the lock held.
| fiftyone_pipeline_engines_fiftyone.share_usage.ShareUsage._send_remaining |
( |
| self, |
|
|
| deadline, |
|
|
| done ) |
|
protected |
Send the entries still queued when closing, even if there are fewer than the requested package size.
A send already running finishes first, so only one message goes out at a time. Nothing is sent while paused, after a message fails, or once deadline has passed, and the entries left are discarded. Closing logs the ones left when the time ran out, and this logs the ones left for the other two reasons.
@type deadline: float
- Parameters
-
| deadline | The monotonic time sending has to stop at. @type done: threading.Event |
| done | Set when sending stops for a reason other than the deadline. |
| fiftyone_pipeline_engines_fiftyone.share_usage.ShareUsage._send_while_queued |
( |
| self | ) |
|
|
protected |
Send messages of up to twice requested_package_size entries until fewer than requested_package_size are queued, or a failed send pauses sending.
| fiftyone_pipeline_engines_fiftyone.share_usage.ShareUsage.add_to_share_usage |
( |
| self, |
|
|
| data ) |
Internal method which adds to the share usage bundle (generating XML) @type key: dict.
- Parameters
-
| key | data value store of current evidence in FlowData |
| fiftyone_pipeline_engines_fiftyone.share_usage.ShareUsage.close |
( |
| self | ) |
|
Wait for any usage data already being sent, then send what is still queued and stop the threads that send it.
The pipeline calls this when it is closed. Closing does not wait for a pause after failed sends, and sends nothing while paused.
Closing gives up after cleanup_timeout_seconds, discards the entries still queued and logs a warning, so that it does not hold up the process that closes it. A message already going out when that time passes is left to the request timeout on its own thread.
Reimplemented from fiftyone_pipeline_engines.engine.Engine.
| fiftyone_pipeline_engines_fiftyone.share_usage.ShareUsage.found_evidence |
( |
| self, |
|
|
| flow_data ) |
The evidence the element finds to share.
These are the values of the web request, when the web integration added it as server.http-request, and every evidence value in the flow data, whether an element accepted it or not. A key found in both places takes the evidence value. The web request itself is never one of them.
@type flow_data: FlowData
- Parameters
-
| flow_data | The flow data being processed @rtype: dict |
- Returns
- : The evidence found, before the sharing rules apply
| fiftyone_pipeline_engines_fiftyone.share_usage.ShareUsage.get_data_from_evidence |
( |
| self, |
|
|
| flow_data ) |
Creates a ShareUsageData instance populated from the evidence within the flow data provided.
- Parameters
-
| {FlowData} | flowData the flow data containing the evidence to use |
- Returns
- a new ShareUsageData instance, populated from the evidence provided
| fiftyone_pipeline_engines_fiftyone.share_usage.ShareUsage.get_evidence_key_filter |
( |
| self | ) |
|
The usage sharing element accepts no evidence.
See found_evidence for what it shares.
Reimplemented from fiftyone_pipeline_core.flowelement.FlowElement.
| fiftyone_pipeline_engines_fiftyone.share_usage.ShareUsage.process_internal |
( |
| self, |
|
|
| flow_data ) |
Internal process method which uses the ShareUsageTracker to determine whether to add usage data to a batch and adds it if necessary.
@type flow_data: FlowData
- Parameters
-
| flow_data | FlowData to process |
Reimplemented from fiftyone_pipeline_core.flowelement.FlowElement.
| fiftyone_pipeline_engines_fiftyone.share_usage.ShareUsage.send_thread |
( |
| self, |
|
|
| endpoint, |
|
|
| data ) |
Send one message of usage data.
- Returns
- : True if the service accepted it
| fiftyone_pipeline_engines_fiftyone.share_usage.ShareUsage.share_send_usage |
( |
| self | ) |
|
Internal method to start sending the queued usage data to the 51Degrees servers.
Nothing starts if a thread is already sending, if fewer than requested_package_size entries are queued, or while sending is paused after failed sends, in which case a timer starts sending when the pause ends.