Internet filtering settings: What it means to Ignore a category

When onboarding new internet content filtering customers, one of the recurring questions that inevitably comes up sooner or later relates to the “ignore” setting. Allow and Block are black-and-white, and are easy to understand; but what Ignore does, and what it’s for, obviously is not readily apparent. This post is an attempt to explain it in terms simple enough that anyone can understand.

First of all, we need a little primer on how the content scoring and decision-making in the Redwood content filtering software works.

What’s the filter thinking?

Each time a page is loaded, Redwood analyzes the page URL and content and compares what it finds with its database of domain lists, regular expressions, and weighted phraselists. Using this information, Redwood calculates a numerical score for each category that contained matching phrases, regular expressions, or URLs. For example, if I load today’s current Canadian Tire home page with the rules Redwood is presently using on my content filter, the following category scores are tallied:

1. Automotive: 3693
2. Tools: 1060
3. Woodworking: 700
4. Lawn and Garden: 590
5. Janitorial / office supplies: 475
6. … and a bunch more, with lesser scores.

It matters not that there are more matching categories with lower point scores. The main thing to focus on is the highest-scoring category, which in this case is Automotive.

When reaching an Allow / Block decision, Redwood reaches its decision based upon the highest-scoring category. That’s it. In the example above, the page is allowed because the Automotive category is allowed. It would not matter what Tools, Woodworking, Lawn, Janitorial, and all the rest were set to. They could all be set to Block, if you please, but the page would load because Automotive is allowed and Redwood has decided the page is Automotive.

Of course, the converse is also true. If for any reason we decided to Block the Automotive category, then no matter what Tools, Woodworking, and so on, are set to, the page request would be denied and you’d see the block page instead of the Canadian Tire home page.

The numerical scores might be a very close tie, but the same rule applies. Suppose Automotive scored 500 points and Racing scored 499 points. If you have Automotive allowed, it matters not if Racing is set to Allow or Block; the page will load. Of course, if Racing was blocked and scored 501 points, the page request would be denied even if Automotive scored 500. Further, if two scores tie (Auto and Racing both scored 500 for instance), Block takes precedence over Allow.

So none of that is too hard to figure out. But what does Ignore do, anyhow? Actually, the concept is very simple and adds incredible power and flexibility to the content filter administrator’s toolset. The documentation, accessible from your web filter console, has this to say about Ignore:

Ignored categories are ignored in the sense Action – such categories never specifically Allow or Block a page. They’re useful to de-activate a category and also provide useful Subject information for Usage Reports.

That says it about right, and in few words. Yet, in spite of its simplicity, it’s been a frequent recurring question. I find it most effective to teach about Ignore, and how it’s different from Block and Allow, using graphical teaching aids. So what follows will be a very wordy version of the concise sentence above, with graphics thrown in to help drive the point home.

Charts and more charts

This chart shows the first 4 category matches of a Softpedia article about running Ubuntu on a Raspberry Pi 3. The numbers on each bar show how many points Redwood scored for each category shown. Blue bars represent allowed categories; red categories are blocked. This was an accurate categorization and it’s not hard to see that the page request will be allowed because the highest-scoring category is an allowed one.

 

This chart is just as simple as the first, but shows a scenario where the page request was blocked.

 

 

So basically, Redwood looks at the top-scoring category and allows or blocks the page based upon that category’s setting. But what if you ignore a category? Take a look at the following chart. This is the same chart as Scenario 1, but with the Computer category now set to Ignore rather than Allow.

 

When a category is ignored, its score does not factor into the Allow / Block decision process at all. The highest-scoring category that is set to Allow or Block is used to make the decision. If the top-scoring category is ignored, well then Redwood ignores that score. It’s that simple.

In the following chart, ignoring the top-scoring category allows the page to load. Note that I have a custom “CBC News” category, which I’ve made to allow filtered access to CBC while keeping many other (more celeb-focused) news outlets blocked.

 

Here is how an Amazon search for “dress pants” fared, score-wise. If you ignore the Clothing category, the page request will be allowed in this case.

 

And lastly, here’s the same chart as Scenario 5 above but with Footwear set to Block, just to demonstrate the point again. If the categories were configured this way, the page request would be blocked.

 

So why is Ignore powerful? And what good does it do to have entire categories of phrase lists, and then just go and ignore them?

There are several answers to these questions. First of all, it’s helpful to have categories encompassing all manner of content because it provides a granular level of control.  Different ones of us draw the lines at different places, and it’s useful to have enough categories that you can draw the line fairly fine instead of catching large sections of the internet with a few wide-ranging categories.

The Ignore directive is very useful for situations where you don’t really mind if content in that category loads, but you don’t want that category over-riding others. To understand this, look again at the last two charts above. The Clothing category is one that can come with a lot of pictures of… well, clothes. And how are clothes advertised? You may wish to buy dress pants on Amazon but not have too much undergarment content load because of associated graphics. If clothing is set to Allow, and Intimate Apparel is set to Block, there may be times when pages that you wish would be blocked won’t be because Clothing scored higher than Intimate Apparel. But if Clothing is ignored, then it won’t be used to reach the decision to allow or block the page. This is just one example, and pardon me if it’s a poor one; make your own applications.

Lastly, ignored categories are useful for accurate usage / accountability reporting. To use the example of clothing again: with Clothing ignored, the usage report will still show how much clothing-related content was accessed.

A word of caution

A word of caution is in order here: Use Ignore sparingly. The filter has many categories for a reason: it needs to correctly categorize a page. Setting a category to Ignore means that the filter no longer uses the data in that category to score content. Ignoring every category on the system would result in no filter at all; and ignoring many categories makes the page score meaningless. By far, the majority of filtering decisions should be black and white, Allow or Block.

 

Let’s wrap this up

Well, that was a long post. What do the official docs say?

Ignored categories are ignored in the sense Action – such categories never specifically Allow or Block a page. They’re useful to de-activate a category and also provide useful Subject information for Usage Reports.

I hope the long explanation clarified things enough that the short version now makes sense to you. Enjoy, provide feedback for improvement if you feel so inclined, and happy filter administration.

 

(post updated Nov 16, 2018 to include a cautionary note about ignoring categories)