NSS

HTTP monitoring rules

DnC — 2020-11-26T08:30:00Z

NSS allows to regularly send an HTTP request, waits for a response and interprets it by applying rules. If an error situation is detected, NSS generates an alert.

The HTTP monitoring rules offered by NSS Lite allow you to not only verify that a website is functioning and has not been hijacked, but also to monitor the response of any HTTP device: DNS, Proxy, Web service etc.

NSS Lite allows in particular to test a response in JSON format or the data-xxx values of HTML5.

HTTP request

NSS Lite accepts an HTTP request limited to the scheme, host, path, query and fragment components.

An HTTP request is entered in the "Request" field of the task in the format:

/// [] [? ] [# ]

Examples:

https://buy.dnc.global/-Nos-produits-.html
 https://www.ossec.dnc.global/web/spip.php?article1

"Rules" field empty

The response expected by NSS following an HTTP request appears in the "Rules" field of the task. This field is optional.

When the "Rules" field is not filled in, NSS takes into account the HTTP code returned in the header of the response, regardless of the payload of the response:

– When the HTTP response code is 500 or more (server error), an event is created with a level equal to 6 (Alert).

– When the HTTP code of the response is 400 or more and less than 500 (web client error), an event is created with a level equal to 5 (Critical).

– When the HTTP code of the response is 300 or more and less than 400 (redirect), NSS tries to follow the redirect. If successful, the HTTP code is 200.

– When the process has ended without error (HTTP code <400), NSS applies a set of standard rules.

"Rules" field not empty

Analysis of the response

An HTTP response includes a "response header" and a "response payload" also referred to as the "body" of the response.

NSS first processes the HTTP response as described above. When the process has finished without error (HTTP code <400), NSS processes the response by applying the rules indicated in the "Rules" field of the task.

If the test fails, an event is created with a level equal to 4, or the level specified in the task definition.

There are two types of rules: "general rules" which do not take into account the payload of the response, and those which apply to that payload.

General rules

– TIME< number of ms: Checks that the server returns the response within a time less than the value indicated in ms.

– MEAN< number of ms: Checks that the server returns the response within an average delay less than the value indicated in ms.

– SAME LOCATION: Check that the URL of the response is identical to the request (there was no redirection).

– SRCIP = nnn.nnn.nnn.nnn: Check that the host of the real response has the indicated address for IPv4 (source IP). In particular, it is possible to check that a redirection does not leave the machine targeted by the request.

– SAME HOST: checks that the domain of the response is identical to that of the request. In particular, it is possible to check that a redirection does not leave the initial domain.

– HTTP = nnn: the HTTP code of the response must have the value indicated.

Response payload analysis rules

– NULL: the body of the response must have zero length.

Note: NSS Lite does not load more than 20,000 bytes from the response body. This must be taken into account for the following rules.

– LENGTH> integer: the body of the response must have a length greater than or equal to the specified value. Note that this is the HTML response, not the section of the HTML page.

– STRING = character string: the body of the response must be equal to the specified string.

– CONTAINS character string: the body of the response must contain the specified string.

– MATCH character string: this rule allows you to check the presence of any string in the response. The string can be a regular expression, so it is possible to perform any check in the response.

– JSON array ['a'] ['b'] ... = value, JSON array ['a'] ['b'] ...> value: The response must be a associative array in JSON format. The rule tests the value of the array element ['a'] ['b'] ...
The number of index dimensions (recursion depth) is limited to 10.
Unlike other rules, the names of the array and indexes are case sensitive.

– DATA-XXX = value, DATA-XXX> value: The response must contain the HTML5 data-xxx attribute. NSS compares the value of the attribute to the value specified in the rule. Note that if the response has multiple data-xxxs, only the first occurrence will be tested.

Rule inversion

We can precede the payload analysis rules with NOT to test the reverse condition (it is often more interesting), the rules are thus valid: NOT NULL, NOT HTTP =, NOT LENGTH =, NOT STRING =, NOT CONTAINS etc ..
However, MATCH cannot be reversed.

Examples of rules

SRCIP = 52.128.11.2 Checks that the response comes from the indicated IP.

CONTAINS checks that the returned document contains a body (without prejudging its content).

LENGTH> 10000 checks that the response has a length> 10000. Used to detect a page generation error if its content (including header) is normally greater than the value indicated.

NOT CONTAINS tep stop The response must not contain the string 'tep stop' (which is a fatal error returned by osCommerce).

NOT JSON Values ['temperature'] ['output']> 55 The response must be a JSON array 'Values' whose element ['temperature'] ['output' ] must be less than or equal to 55.

DATA-price> 50 The price indicated in the data-price field must be greater than 50.

Combination of several rules for the same task

Several rules can be indicated in the "Rules" field, one per line.
They are combined with the logical operator AND, ie the task is in error if one of the rules is verified.

The number of rules is limited only by the total length of the "Rules" field, limited to 1024 characters.

Here is an example of DNS monitoring for a domain [1], In which the rules are applied to the response from a DNS test site such as testdns.fr:

CONTAINS ns102.ovh.net
 CONTAINS dns102.ovh.net
 CONTAINS 51.178.18.44

This other example shows how to verify that a page is non-empty, doesn't show a MySQL error, hasn't been hijacked, and how to track response time:

SRCIP = NNN.NNN; NNN; NNN
 LENGTH> 10,000: 6
 CONTAINS TheTitleOfThePageByExample: 6
 NOT CONTAINS MySQL error
 NOT CONTAINS forbidden
 TIME <1000: 2
 TIME <3000: 3
 TIME <10000: 4

Please note that rules such as "NOT CONTAINS MySQL error" or "NOT CONTAINS forbidden" are not universal: they depend on the application or the firewall.

Standard rule set applied by default

When the rules field is empty, NSS applies the following standard rule set (not requiring the loading of the response body):

TIME <2000: 2
 TIME <5000: 3 "
 TIME <10000: 4

Response time

If no rule has generated an error (including the TIME and MEAN rules described above), NSS compares the response time against the average response time observed for the task. An alert is generated with a level equal to 2 (Notice), or the level indicated in the definition of the task, if the response time exceeds 4 times the average time.

For more details on this topic, see: Surveillance du temps de réponse.

Alert level

Definition

NSS classifies events according to their severity into 8 levels, described in the following table:

Level	Severity	Elevation
7	Emergency: the system cannot be used	NA
6	Alert: action must be taken immediately	7 if number exceeds 5
5	Critical condition	6 if number exceeds 5
4	Error	5 if number exceeds 60
3	Attention	NA
2	Significant information	3 if number exceeds 60
1	Information	NA
0	Debug Info	NA

Alert level assignment

In the event of an error, NSS assigns an alert level of 4 by default.

This alert level can be modified for a given task by entering a different value in the "Error level" field of the task creation / edition form.

Thus defined, the level of error applies regardless of the rule in error. It is possible to define a particular level of error for the rule by following it with ': N' where N is the desired level, as in the following example:
LENGTH> 10000: 6

Of course, the defined error levels must be consistent with the "Minimum alert level for sending" defined for notifications (see below).

Compaction of events

When identical events follow one another, NSS does not generate an alert each time, but counts the number of repetitions. Only one line appears in the Alerts table.
As indicated in the "Elevation" column, the alert level is raised by one when the same error is repeated.
Thus, the Alerts table remains synthetic, and the rate of notifications reasonable.

Notification

The Manage notifications form allows you to define the destination addresses and the minimum level triggering a notification.

By default, the alert level is 5 (send "critical conditions" and above).
This level can be modified in the "Minimum alert level for sending" field, indicating a number from 0 (send all alerts) to 7 (send only "unusable system").

NSS issues a notification message at the start of an alert series and when the level rises.

[1] It is much more efficient to monitor DNS using a dns: // query. See: Règles de surveillance DNS.

DNS monitoring rules

DnC — 2020-10-27T11:12:25Z

NSS allows you to regularly send a DNS query, waits for a response and interprets it by applying rules. If an error situation is detected, NSS generates an alert.

The DNS watch rules offered by NSS Lite allow you to verify all aspects of the DNS configuration of the domain in question, in addition to checking response time.

Requête DNS

A DNS request is entered in the "Request" field of the task in the format:

dns: [//[:] /] [? ]

where :

– server: (optional) URL of a particular DNS server.
– domain (required): the domain of the record in the DNS zone.
– type: (optional, A) the type of record in the DNS zone of the domain: A, AAAA, CNAME, MX, NS, SOA, SPF, TXT. If this parameter is not understood, an error will be generated.

examples:

dns:buy.dnc.global
 dns:dnc.global?NS
 dns://dns102.ovh.net/buy.dnc.global
 dns:degoy.com?A
 dns:degoy.com?AAAA
 dns://dns200.anycast.me/degoy.com?AAAA
 dns:degoy.com?MX
 dns://ns-219-a.gandi.net/www.spip.net
 dns://1.1.1.1/dnc.global /* Cloudflare resolver */
 dns://8.8.8.8/dnc.global /* Google resolver */
 dns://213.186.33.99/dnc.global?AAAA /* OVH resolver France*/
 dns://5.196.123.133/dnc.global?AAAA /* OVH resolver, Spain */
 dns://193.252.10.2/dnc.global /* Orange, Aubervilliers, France */

Reply

NSS issues the DNS query as a Unix / Linux DIG of the following form:
dig [] [@ server [: port]]

If the request is successful, the response parsed by NSS is the literal form returned by DIG.
Here is for example the response to the request dns: degoy.com? MX (translated into dig MX degoy.com):

; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.68.rc1.el6_10.7 <<>> MX degoy.com
 ;; global options: +cmd
 ;; Got answer:
 ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 36151
 ;; flags: qr rd ra; QUERY: 1, ANSWER: 6, AUTHORITY: 0, ADDITIONAL: 0
 
 ;; QUESTION SECTION:
 ;degoy.com. IN MX
 
 ;; ANSWER SECTION:
 degoy.com. 600 IN MX 5 mx1.mail.ovh.net.
 degoy.com. 600 IN MX 100 mx3.mail.ovh.net.
 degoy.com. 600 IN MX 1 mx0.mail.ovh.net.
 degoy.com. 600 IN MX 200 mail.rollernet.us.
 degoy.com. 600 IN MX 200 mail2.rollernet.us.
 degoy.com. 600 IN MX 50 mx2.mail.ovh.net.
 
 ;; Query time: 11 msec
 ;; SERVER: 213.186.33.99#53(213.186.33.99)
 ;; WHEN: Tue Sep 1 17:22:07 2020
 ;; MSG SIZE rcvd: 174

If this fails, NSS will log an error.

Implicit and Default Rules

The response expected by NSS following a DNS request appears in the "Rules" field of the task. This field is optional. Whether or not it is entered, implicit rules apply:

status: NOERROR : NSS vérifie que la réponse du DNS comporte "status: NOERROR". Dans le cas contraire, une alerte de niveau 5 au moins est générée.

Response Time: NSS always monitors the response time against the average time observed for the task.
An alert is generated with a level equal to 2, or the level indicated in the task definition, if the response time exceeds 4 times the average time.

For more details on this topic, see: Surveillance du temps de réponse.

Default rules: moreover, if the rule field is not filled in, NSS applies the following rules, assuming an expected response within a period of less than 50 ms:

TIME<50:3
 TIME<100:4
 TIME<500:5

These rules already allow effective monitoring without having to assign rules to the task. So the simple query:

dns://ns-219-a.gandi.net/www.spip.net

allows you to monitor the resolution of the domain name www.spip.net by one of its authoritative DNS servers.

Rules applicable to the DNS request

– TIME< number of ms: Checks that the server returns the response within a time period less than the value indicated in ms.

– MEAN< number of ms: Checks that the server returns the response within an average delay less than the value indicated in ms.

– IPV4= NNN.NNN.NNN.NNN : the most essential rule, allowing to verify that the domain name indicated in the DNS request is resolved and equal to the indicated IPv4 address.

– QUERY TIME< number of ms: DNS server response time.

Note: The TIME and MEAN rules take into account the total response time of the request, including transmission times, whether the DNS is responding or not), while QUERY TIME covers the time shown in the DNS response.

– CONTAINS character string: this rule allows you to check for the presence of a string in the raw response.

– MATCH character string: this rule allows you to check for the presence of any string in the raw response. The string can be a standard regular expression, so it is possible to perform any check in the response. However, the use of the ARRAY rule will most often be preferable.

– ARRAY [index] = character string: Checks that the character string is found in the value of the array at index A. Note that this is an inclusion and not an equality , the rule behaving like CONTAINS. The index can be multidimensional.

– ARRAY [index]> number: Checks that the value of the array at index A is greater than the number indicated. The index can be multidimensional.

Learn more about the ARRAY rule ...

The ARRAY ... rule completes the rule set to test any information returned by the DNS query.

How NSS translates the responses

To make a request, NSS calls on DIG and then breaks down the response into an associative array. The rules relate to the elements of this array.
Usually, you won't have to worry about this table. However, the formulation of an ARRAY rule supposes to know the structure of this table well.

Here's how NSS internally translates the previous query into an array:

: array =
RAW: string = " ; <<>> DiG 9.11.5-P4-5.1+deb10u2-Debian <<>> www.spip.net\n;; global options: +cmd\n;; Got answer:\n;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 594\n;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 3, ADDITIONAL: 3\n\n;; OPT PSEUDOSECTION:\n; EDNS: version: 0, flags:; udp: 4096\n; COOKIE: f182cd5130c105c8056d4eed5f50bf97a6b668e584c7b2e7 (good)\n;; QUESTION SECTION:\n;www.spip.net.\t\t\tIN\tA\n\n;; ANSWER SECTION:\nwww.spip.net.\t\t1979\tIN\tA\t151.80.20.125\n\n;; AUTHORITY SECTION:\nspip.net.\t\t34795\tIN\tNS\tns-145-c.gandi.net.\nspip.net.\t\t34795\tIN\tNS\tns-77-b.gandi.net.\nspip.net.\t\t34795\tIN\tNS\tns-219-a.gandi.net.\n\n;; ADDITIONAL SECTION:\nns-219-a.gandi.net.\t34795\tIN\tA\t173.246.100.220\nns-219-a.gandi.net.\t34795\tIN\tAAAA\t2001:4b98:aaaa::dc\n\n;; Query time: 0 msec\n;; SERVER: 127.0.0.1#53(127.0.0.1)\n;; WHEN: Thu Sep 03 10:04:07 UTC 2020\n;; MSG SIZE rcvd: 203\n"
; <<>> DIG 9.11.5-P4-5.1+DEB10U2-DEBIAN <<>> WWW.SPIP.NET: string = ""
GLOBAL OPTIONS: string = " +cmd "
GOT ANSWER: string = " "
HEADER: array =
opcode: string = " QUERY"
status: string = " NOERROR"
id: string = " 594 "
flags: string = " qr rd ra"
QUERY: string = " 1"
ANSWER: string = " 1"
AUTHORITY: string = " 3"
ADDITIONAL: string = " 3 "
OPT PSEUDOSECTION: string = " ; EDNS: version: 0, flags:; udp: 4096 ; COOKIE: f182cd5130c105c8056d4eed5f50bf97a6b668e584c7b2e7 (good) "
QUESTION SECTION: string = "www.spip.net. IN A "
QUERY TIME: string = " 0 msec "
SERVER: string = " 127.0.0.1#53(127.0.0.1) "
WHEN: string= "Thu Sep 03 10:04:07 UTC 2020"
MSG SIZE RCVD: string = " 203 "
ANSWER SECTION: array =
0: array =
host: string = "www.spip.net"
class: string = "IN"
ttl: long = 1974
type: string = "A"
ip: string = "151.80.20.125"
AUTHORITY SECTION: array =
0: array =
host: string = "spip.net"
class: string = "IN"
ttl: long = 34790
type: string = "NS"
target: string = "ns-145-c.gandi.net"
1: array =
host: string = "spip.net"
class: string = "IN"
ttl: long = 34790
type: string = "NS"
target: string = "ns-77-b.gandi.net"
2: array =
host: string = "spip.net"
class: string = "IN"
ttl: long = 34790
type: string = "NS"
target: string = "ns-219-a.gandi.net"
3: array =
host: string = "spip.net"
class: string = "IN"
ttl: long = 34790
type: string = "NS"
target: string = "ns-77-b.gandi.net"
4: array =
host: string = "spip.net"
class: string = "IN"
ttl: long = 34790
type: string = "NS"
target: string = "ns-145-c.gandi.net"
5: array =
host: string = "spip.net"
class: string = "IN"
ttl: long = 34790
type: string = "NS"
target: string = "ns-219-a.gandi.net"
ADDITIONAL SECTION: array =
0: array =
host: string = "ns-219-a.gandi.net"
class: string = "IN"
ttl: long = 34790
type: string = "A"
ip: string = "173.246.100.220"
1: array =
host: string = "ns-219-a.gandi.net"
class: string = "IN"
ttl: long = 34790
type: string = "AAAA"
ipv6: string = "2001:4b98:aaaa::dc"
2: array =
host: string = "ns-219-a.gandi.net"
class: string = "IN"
ttl: long = 34790
type: string = "A"
ip: string = "173.246.100.220"
3: array =
host: string = "ns-219-a.gandi.net"
class: string = "IN"
ttl: long = 34790
type: string = "AAAA"
ipv6: string = "2001:4b98:aaaa::dc"
IPV4: string = "151.80.20.125"

Download

Examples of ARRAY rules

ARRAY[HEADER][AUTHORITY]= 2 /*verify that there are two authoritative servers */
 ARRAY[HEADER][ADDITIONAL]> 0 /* check that there is at least one additional server */
 ARRAY[RAW]=EDNS /* check EDNS support*/
 ARRAY[AUTHORITY SECTION][0][target]=ns-145-c.gandi.net

The last example shows a difficulty: if there are several authoritative servers, what is the rank of the one whose presence we want to check? The '*' sign in the ARRAY [] = rule is used to test all indexes:

ARRAY[AUTHORITY SECTION][*][target]=ns-145-c.gandi.net

Inversion, combination, alert level

The inversion of the rules by NOT is possible for the TIME, MEAN and CONTAINS rules.

The combination of the rules and the definition of the alert level are done in the same way as in the case ofHTTP monitoring rules.

Which DNS server is responding?

A DNS query specifies the server to query by providing the @ parameter as in the following examples:

dns: //ns-219-a.géat/www.spip.net
 dns: // 2001: 4b98: aaaa :: dc / www.spip.net

If the request does not specify the server, the local NSS resolver at address 127.0.0.1 will respond. While this is appropriate in some cases, it is not the best way to ensure that a domain's DNS is working properly, as the resolver will not recursion until the SOA has expired.

Furthermore, when the local resolver responds, it does so in Query time: 0 msec. It is not very interesting to know!

Which DNS server should I query?

To ensure the proper functioning of the DNS, you must contact a domain authoritative nameserver. In the previous example, we queried one of the three authority servers for the domain www.spip.net.

You can find out the list of authority servers for the domain by issuing the following command in command mode:

dig www.spip.net
 
 that returns:
 
 ; <<>> DiG 9.11.5-P4-5.1+deb10u2-Debian <<>> www.spip.net
 ;; global options: +cmd
 ;; Got answer:
 ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 3800
 ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 3, ADDITIONAL: 7
 
 ;; OPT PSEUDOSECTION:
 ; EDNS: version: 0, flags:; udp: 4096
 ; COOKIE: 798ddf1f8a880829b2ae2fae5f4f590bd9ae0131be4067c2 (good)
 ;; QUESTION SECTION:
 ;www.spip.net. IN A
 
 ;; ANSWER SECTION:
 www.spip.net. 1427 IN A 151.80.20.125
 
 ;; AUTHORITY SECTION:
 spip.net. 126583 IN NS ns-219-a.gandi.net.
 spip.net. 126583 IN NS ns-77-b.gandi.net.
 spip.net. 126583 IN NS ns-145-c.gandi.net.
 
 ;; ADDITIONAL SECTION:
 ns-77-b.gandi.net. 590 IN A 213.167.230.78
 ns-145-c.gandi.net. 545 IN A 217.70.187.146
 ns-219-a.gandi.net. 126583 IN A 173.246.100.220
 ns-77-b.gandi.net. 590 IN AAAA 2001:4b98:aaab::4e
 ns-145-c.gandi.net. 545 IN AAAA 2604:3400:aaac::92
 ns-219-a.gandi.net. 126583 IN AAAA 2001:4b98:aaaa::dc
 
 ;; Query time: 0 msec
 ;; SERVER: 127.0.0.1#53(127.0.0.1)
 ;; WHEN: Wed Sep 02 08:34:19 UTC 2020
 ;; MSG SIZE rcvd: 291


A good query would therefore be:
dns://ns-219-a.gandi.net/www.spip.net
Public resolvers are also interesting, for example:
dns://1.1.1.1/www.spip.net /* Cloudflare resolver */
 dns://8.8.8.8/www.spip.net /* Google resolver */
Note about HTTP codes
...