[TriLUG] Cautionary Tale: Intel/McAfee VSEL & Python Web Apps

[Scott Chilcote via TriLUG]

Mr. Rogers raises an interesting point. 

McAfee VSEL, for all of its faults, also has a web console that can be
quite useful in debugging issues like the one we had.  For example, one
of its displays shows what the active scanner in doing, in something
that approximates real-time.

However, if your $CUSTOMER is obligated under contractual agreements to
follow government cyber-security guidelines for the configuration of
their servers and VSEL in particular, then $CUSTOMER's VSEL team may not
realize that it exists.  That's because the Security Technical
Implementation Guide recommends that the web UI in VSEL must be disabled. 

This is a routine policy intended to prevent it from being accessed by
bad people via the internet.
[Details here, if you're interested:
https://www.stigviewer.com/stig/mcafee_vsel_1.92.0_managed_client/2016-04-06/finding/V-63051]

I see this as a new-ish problem that the next generation of computer
security people are going to need to address. 

Implementation guidelines are *not* commandments handed down from on
high!  Just because it's a best practice to keep the UI disabled on a
production server does NOT mean 1) that you can't ever turn it on, like
when it might be useful to fix a problem, and 2) that you don't need to
learn about it, experiment with it, and gain some experience with it so
that you don't have to start from zero when that problem arrives.

If certain persons responsible for VSEL in this situation had not taken
the guidelines for stone carvings, we would have gotten through this
investigation significantly faster.

   Scott C.


On 12/16/2016 11:00 AM, Rogers, Matthew wrote:
> If you want to talk IDS/Antimalware, checkout Cisco AMP.  Has Linux/Mac/Win support and runs off Talos.  Huge fan of Cisco lately in this area.  You manage it all via web console.  
>
> http://www.cisco.com/c/en/us/products/security/fireamp-endpoints/index.html
>
> Matthew Rogers
> Senior Security Manager
> O: +1.919.329.1130    M: +1.919.691.4636
> allscripts.com | matthew.rogers2 at allscripts 
>
> Allscripts: Building open, connected communities of health
> _______________________________________________
>
> -----Original Message-----
> From: TriLUG [mailto:trilug-bounces+matthew.rogers2=allscripts.com at trilug.org] On Behalf Of Scott Chilcote via TriLUG
> Sent: Friday, December 16, 2016 10:55 AM
> To: Triangle Linux Users Group discussion list <trilug at trilug.org>
> Subject: [TriLUG] Cautionary Tale: Intel/McAfee VSEL & Python Web Apps
>
> Hi LUGgers,
>
> VSEL is VirusScan Enterprise for Linux.  My employer provides some web applications based on a JavaScript/Python framework for our $CUSTOMER. 
> $CUSTOMER mandates the use of Intel/McAfee VSEL for its RHEL webservers, which is not a bad idea in itself.  However, read on.
>
> VSEL does active signature-based scanning of all scripts launched via Apache, among much else.  We discovered in early testing that the VSEL scanner processes had a noticeable impact on our script performance. 
> This led to substantial troubleshooting.
>
> We ran "top" and found that the VSEL scanner tasks were all at the upper end, pushing python way down.  Scanner tasks were getting around 60% of core utilization, python around 3%. 
>
> Some of our scripts are CPU intensive for brief periods.  They read zip files in a repository and build hashes based on their contents.  So we spoke with $CUSTOMER's people who handle VSEL.  They decided to try adding "exclusions" for our web app's python scripts, which is how you tell VSEL to trust installed scripts. 
>
> The performance immediately improved, so we moved on.
>
> Some weeks further into the code cleanup and system tests, we find that VSEL is clobbering our scripts again.  It seems to have stopped honoring its exclusions.  The VSEL team runs a test using a "fake virus"
> sentinel, and proves it.  They uninstall/reinstall VSEL, and we're good again.  Back to work.
>
> A few weeks later with final tests approaching, VSEL interference is worse than ever.  Even low demand script operations, like reading directory contents are staggering. 
>
> We go back to work with the VSEL people.  They uninstall/reinstall VSEL again.  Nope.  They add all of the data directories as exclusions ("just in case").  No help.  They add wildcard exclusions, removing swaths of directories and file types we barely touch.  No help.  They call Intel Security (who bought McAfee recently).  After they finally get through, Intel's support people start working down a "we think your dumb"
> checklist of steps that we ruled out weeks ago. S-l-o-w-l-y.  Frustration!
>
> We go through more weeks of trying stuff and head scratching.  Deadlines impend. 
>
> Then we have that zen moment.  "Do we have *all* of the python scripts that the web applications use excluded?", they ask.  "Well yeah, all of the ones we wrote.  The entire web application directories are wild-carded."
>
> "Are there others?"
>
> "Well, sure.  The python API libraries.  Those are part of the system. 
> Redhat installs those...!   Aren't they excluded by default??"
>
> We aren't VSEL guys.  We aren't HBSS or active IDS people.  Why would VSEL waste time scanning the system libraries?  Sure, doing a signature check on them following a system update, that makes sense.  But would VSEL be designed to signature scan the system libs *every* *time* they were accessed by a running application?  Seriously?
>
> As it turns out, McAfee thinks so. 
>
> We added "/usr/lib64/python[2,3].[0-9]/.*/.*.py" to the exclusion list, which covers all of the system Python libraries.  The slowdown vanished.
>
> We bought our own license for VSEL to help this problem (which costs beer money).  I'm not saying everyone should, working with Intel Security brings its own challenges.  But for us it was time to learn up.  My idea of a decent IDS was tripwire, until recently.  With the Linux webserver hacks that are widespread in the last 4-5 years, active intrusion detection on internet facing services will become much more common.  If it isn't already. 
>
> I haven't needed to understand how it works until this huge time-waster fell into our laps.  I suspect we'll have even more learning to do as this project proceeds.
>
> Scott C.
>
> --
> Scott Chilcote
> scottchilcote at ncrrbiz.com
> Cary, NC USA
>
> --
> This message was sent to: Matthew <matthew.rogers2 at allscripts.com> To unsubscribe, send a blank message to trilug-leave at trilug.org from that address.
> TriLUG mailing list : http://www.trilug.org/mailman/listinfo/trilug
> Unsubscribe or edit options on the web	: http://www.trilug.org/mailman/options/trilug/matthew.rogers2%40allscripts.com
> Welcome to TriLUG: http://trilug.org/welcome


-- 
Scott Chilcote
scottchilcote at ncrrbiz.com
Cary, NC USA