Thursday, July 8, 2021

Creating regular expression for Qradar

 To deal with these, we first need to properly extract the second double-quoted field. Note that Apache log files use backslashes to escape extra quotes or other special characters. This means naive regular expressions such as "[^"]*" aren't good enough.

Using grep to extract the referrer field (second double-quoted field):

grep -oP '^[^"]+"[^"\\]*(?:\\.[^"\\]*)*"[^"]+"\K[^"\\]*(?:\\.[^"\\]*)*(?=")' logfile.txt

Looks crazy! Let's break it down:

  • The o argument to grep means we just get the matching part of the line, not the rest of it
  • The P argument to grep tells it to use Perl-compatible regular expressions
  • The overall structure of the regular expression in use here, ...\K...(?=...), means we are checking the whole pattern but only the things between the \K and the (?=...) will be output

Breaking the regular expression down further:

  1. ^[^"]+ – Get everything between the start of the line and the first "
  2. "[^"\\]*(?:\\.[^"\\]*)*" – Get the entire first double-quoted string. 
  3. [^"]+ – Get everything between the two strings
  4. "\K[^"\\]*(?:\\.[^"\\]*)*(?=") The same as above, but we have the \K after the first " to start matching data after that and the (?=") to stop matching data before the last ".


After this point the data will be much easier to process because you no longer have to worry about the quotes and extracting the field properly from the log file.

For example, you could pipe the output into another grep:

grep -oP ... logfile.txt | grep -oPi '^https?://www\.google\.com/search\?\K.*'

Here the i option to the second grep makes it case-insensitive.

Alternatively, you could add the check for the the start of the google.com referrer directly into the first regular expression and move the \K as appropriate, but I would recommend against this since it's better to run two regular expressions which both do one job and do it well than to combine them into one where its job is not clear.

Note that if you want to collect referrers from other Google domains you will need to modify your regular expression a fair bit. Google owns a lot of search domains.

If you didn't mind potentially catching a few non-Google sites, you could do:

... | grep -oPi '^https?://(www\.)?google\.[a-z]{2,3}(\.[a-z]{2})?/search\?\K.*'

Otherwise you would need to attempt to match only Google-owned search domains, which is a constantly moving target:

... | grep -oPi '^https?://(www\.)?google\.(a[cdelmstz]|b[aefgijsty]|cat|c[acdfghilmnvz]|co\.(ao|bw|c[kr]|i[dln]|jp|k[er]|ls|m[az]|nz|t[hz]|u[gkz]|v[ei]|z[amw])|com(\.(a[fgiru]|b[dhnorz]|c[ouy]|do|e[cgt]|fj|g[hit]|hk|jm|k[hw]|l[bcy]|m[mtxy]|n[afgip]|om|p[aeghkry]|qa|s[abglv]|t[jrw]|u[ay]|v[cn]))?|d[ejkmz]|e[es]|f[imr]|g[aefglmpry]|h[nrtu]|i[emoqst]|j[eo]|k[giz]|l[aiktuv]|m[degklnsuvw]|n[eloru]|p[lnst]|r[osuw]|s[cehikmnort]|t[dgklmnot]|us|v[gu]|ws)/search\?\K.*'

Also note if you want to include Google's image search and other search subdomains, you will need to change the (www\.)? in one of the above grep commands to something like ((www|images|other|sub|domains)\.)?.

Tuesday, April 10, 2018

Onion Elastic Stack General Availability Release and Security Onion 14.04.5.11 ISO Image!

Elastic Stack integration has now reached General Availability (GA)!  This includes a new 14.04.5.11 ISO image that contains these GA components and all the latest Ubuntu and Security Onion updates as of March 28, 2018!

GA Highlights

Monday, October 23, 2017

Pen Testing Checklist for Cloud

Start with the Contract. Your Cloud services are provided under contract between you and your CSP. This forms the base of the relationship, and defines what activities each party is responsible to perform. Not all CSP’s are the same, nor are all contracts identical. Some will have various tiers of service, others may provide a base offering with additional “add-on” options. Whatever your situation, it is vital to have a clear understanding of R&R, policies, service commitments, and restrictions.
  1. Check the Service Level Agreement (SLA) to ensure the appropriate Pen Test policy has been identified, and R&R clearly defined. In many cases, elements of Pen Testing are spread across multiple players such as the CSP and the client, so it is necessary to clearly document who does what, and when it is to be done.
  2. Governance & Compliance requirements need to be understood. Factors need to include which party will be responsible to define, configure and validate security settings required to meet applicable regulatory controls for your business. This includes providing appropriate evidence for audits and inspections.
  3. Security and Vulnerability Patching and general maintenance responsibilities and timeframes need to be documented. You as the client may have responsibility for maintaining your virtual images and resources, but the CSP will likely be accountable for the underlying physical hardware systems. Both need to be actively managed, along with all network and SAN equipment.
  4. Computer access and Internet usage policies need to be clearly defined and properly implemented to ensure appropriate traffic is permitted while inappropriate traffic is denied at the perimeter.
  5. Ensure all unused ports are disabled and unused protocols are either not installed or disabled and locked down to prevent unauthorized activation.
  6. Data encryption while both in transit and at rest is becoming more common, but never assume. Ensure that encryption is either set as the default or that appropriate steps are implemented to ensure it is activated.
  7. Verify that your requirements for Two Factor Authentication and One Time Passwords are implemented and actively securing network access. Check if your CSP permits any bypass scenarios.
  8. SSL is only as good as the Certificate Authority (CA) that issued the certificates. Ensure SSL is active, and that a reputable CA stands behind the certificates.
  9. Hold your CSP accountable and validate they are using appropriate security controls for physical and logical access to the data center and the infrastructure hardware with which they provide your services.
  10. Know your CSP’s policy and procedures relative to data disclosure to third parties, both for unauthorized access and providing data when requested or subpoenaed by law enforcement.
More than just running a scan, Pen Testing requires an understanding your environment and the associated roles and responsibilities and even liabilities between you and your CSP.

PCI Pen Testing Requirements

Scanning Isn’t Testing

Requirement 11 is the juggernaut of PCI v3.2. It’s packed full of objectives involving network scanning (11.2) and penetration testing (11.3). Just because scanning and testing are lumped into the same PCI requirement, doesn’t mean they accomplish the same goals.

The PCI Standards Security Council is helping to clear the air with a couple of simple descriptions.
Vulnerability Scanning: Identifies, ranks, and reports on vulnerabilities that, if
exploited, may result in an intentional or unintentional compromise of a system. Translation? It’s an automated process taking a short amount of time, with no verification and very high chance of false-positives.
Penetration Testing: Identifies ways to exploit vulnerabilities to circumvent or defeat the security features of system components. Translation? It’s a manual or automated testing process that uses vulnerability scanning results as a baseline, lasting days or weeks depending on the scope.

Testing Ain’t Easy

Whether you use a tool or a service, complying with PCI Requirement 11.3 is a formidable task taking a combination of resources, time and a little bit of planning. In its simplest form, Requirement 11.3 mandates internal and external penetration testing at least annually or after “any significant infrastructure or application upgrade or modification”. Considering the volume of iterations of common operating systems and applications, I’d say we can throw out the idea of only testing annually. So, how do you test regularly and stay agile?
  1. The right person or service: Take a look to see if any of your team members have pen testing experience. The great thing is that some penetration testing products, like Core Impact, have the option of using wizards to run various tests. This helps automate some of the more laborious tasks, while accomplishing the same goals in less time. As a supplement to this, try bringing on a more seasoned penetration tester or dedicated third-party security service.
  2. The right tool: Depending on your needs and overall requirements scope, the tool you require may differ. Whatever it is, you need to be able to test regularly as things change in your environment. So, if you have one person on your team who only uses an open-source command-line based pen testing tool, with no other options, this could leave you in a bit of a pickle if he or she leaves. Be sure to have a backup plan.
  3. The right methodology: Penetration testing doesn’t just happen and disappear. According to the new PCI Security Standards Council penetration testing guidelines, it requires a phased-approach involving pre-engagement, engagement, and post-engagement steps. These steps cover important topics such as timing, frequency, reporting, systems to target, success criteria etc. Preparing properly will help you stay on target, meet the PCI requirements, and not become overwhelmed. Well, not too overwhelmed at least.

Friday, February 17, 2017

Security and Sandbox

It’s time to go beyond using sandboxing as a standalone capability in order to get the most out of it. You need a more robust malware analysis tool that fits seamlessly into your infrastructure and can continuously detect even the most advanced threats that are environmentally aware and can evade detection.
There are three typical ways that organizations purchase and deploy sandbox technology.
  1. A stand-alone solution designed to feed itself samples for analysis without dependency on other security products. This has the most flexibility in deployment but adds significant hardware costs and complexity to management and analysis, especially for distributed enterprises.
  2. A distributed feeding sensor approach, such as firewalls, IPS, or UTMs with built-in sandboxing capabilities. These solutions are usually cost effective and easy to deploy but are less effective in detecting a broad range of suspicious files including web files. They can also introduce bandwidth limitations that can hamper network performance and privacy concerns when a cloud-based solution is the only option.
  3. Built into secure content gateways, such as web or email gateways. This approach is also cost effective but focuses on web and email channels only and also introduces performance limitations and privacy concerns.


Arcsight Threat Tracking Method

This Activate Method is used to track attackers and target system state and their progression through the attack life cycle. It consists of a set of rules that update the attacker and target threat scores as well as their progression and indicator and warning frequency within the attack life cycle. All this information is tracked in a set of lists with varying TTL for the entries.
For this to work efficiently, indicators and warnings have predetermined categorization requirements. For example, an IDS reporting shellcode over the wire is tagged with Category Custom Format Field = "/Attack Life Cycle/Delivery". The rules look for these conditions populate attacker and target information in the appropriate lists as well as the threat score tracking information.
ThreatTrackingEventFlow.jpg

Attack Life Cycle

The Attack Life Cycle lists are very straight forward. There is one list for each phase of the attack life cycle and the rules for getting attackers and targets into the lists follow two simple laws:
  • Attackers and Targets can live in multiple list at the same time
  • Only Activate rules will populate the lists
NOTE:
  • Custom Category Format Field will be used to move data into the appropriate list
  • All content is stored under:
    /All /ArcSight Activate/Use Cases/Threat Tracking/Attacke Life Cycle/ System Perspective/
ListsDescription
Phase 1 ReconnaissanceWill track attackers and targets that are conducting research and identification of targets. If the attacker's target can be derived from the event/analysis, then the target will also be tracked. Typically, these indicators and warnings and found by monitoring NIDS, HIPS, firewall ACLs, and web analytics.
Phase 2 WeaponizationWeaponization is not normally 'viewable' as the attacker normally creates weaponized packaged on systems that we are not in control of. This category will be used for file analysis tools that detect known IOC. (Mandiant, STIX, Tripwire)
Phase 3 DeliveryIndicators and warnings that intercept the transmission of executable code to a target. NIDS/HIPS/Proxies/in-line AV events are sources capable of detecting these events.
Phase 4 ExploitationExecution of the attackers code. This list will track when code is executed either by the user or by exploiting a particular vulnerability. Indicators are warnings are detected at the OS/HIDS level. (SRP/ASLR/DEP)
Phase 5 InstallationInstallation of remote access code. This can be detected at the OS/HIDS/AV
Phase 6 C2Track attackers and hosts that are displaying beaconing characteristics. These are often detected by NIDS/FW acl/Honeypots/DNS, and detection can be enriched with external intel data
Phase 7 ObjectivesPost "pawnage" activities (data exfil, corruption, destruction, pivots).

Arcsight Alert Suppression System

Suppression System

Analysts frequently need a method to suppress rules from firing until misconfigurations are resolved or point solutions are tuned. To support this activity a set of active lists will be maintained so the analysts can suppress correlation rules and data monitor alerts.
There are two sets of active lists. The first set is writable by all analysts and will have a TTL of 24 hours. These lists will be stored in "/All Active Lists/ArcSight Activate/Core/Suppression Lists". The second set of lists is only writable by L2 analysts and the TTL will be '0'. These lists will be stored in "/All Active Lists/ArcSight Activate/Core/Suppression Lists/Static".
NOTE: lists should have no more than 100 or so entries.
All logic used to compare new events to the suppression lists are stored in a filter under "/All Filters/ArcSight Activate/Core/Suppression List Filters". Should the need arise, simply create more lists and/or filters and document the hooks below.
To create more suppression lists, simply
  1. Add an L1 list with 24hr TTL, "/All Active Lists/ArcSight Solutions/Administration/Suppression Lists"
  2. Copy the list to the static directory and change the TTL to 0, "/All Active Lists/ArcSight Solutions/Administration/Suppression Lists/Static"
  3. Change the appropriate filter, "All Filters/ArcSight Solutions/Administration/Suppression List Filters"
  4. Update the documentation below
Hooking into the methodology
  • Include the appropriate filter from "/All Filters/ArcSight Activate/Core/Suppression List Filters"
    • For network based events where all network based lists should take affect use: "All Network Based Suppression Lists"

Thursday, December 29, 2016

Arcsight Health monitoring

Arcsight Monitoring tools

Since there are many elements in an ArcSight environment, ensuring continuous operation of all might require:
·         Ensuring that source devices keep sending events.
·         Ensuring that connectors operate properly.
·         Ensuring that the ArcSight systems (Logger, ESM, Express) operate properly.
·         Ensuring that the infrastructure used by those elements, including operating system and network.

Some of the solutions you might want to check to monitor an ArcSight environment are:

ArcSight Management Center


ArcSight Management Center (ArcMC) is an ArcSight product that can be used to manage and monitor ArcSight systems. It currently can have managed and monitors Loggers, Connectors, Connector Appliances and other ArcMC systems.

ESM health monitoring


ESM has system resources that are useful for monitoring:

Resource
URI
What can you find there?
Database performance statistics
/All Dashboards/ArcSight Administration/ESM/System Health/Storage/CORR Engine/Database Performance Statistics
free space available for the tables such as Arc_Event_Data and Arc_System_Data
ESM system Information
/All Dashboards/ArcSight Administration/ESM/System Health/ESM System Information
Memory use
Event throughput
/Dashboards/System Health/Events/Event Throughput
Peak and average EPS
Connector status
Right click "connectors-->All Connectors" and select "Grid View" to quickly see if any connector is down.

Other tools available are:
·         You can also try the ESM Health Monitoring package developed at HP CDC, is a combination of scripts and ESM content to monitor the ESM manager itself. It monitors the like of as IO, heap usage, garbage collections, cpu, eps and connector caching. The package is robust, but complex and not fully documented.
·         Monitor ESM/CORR with JMX, ELK and TICK - Use Java JMX monitoring API to monitor the manager and connectors operating environment (community contribution)

Logger content packs

·         Logger Operations Health Dashboards is a presentation which contains queries for creating Logger dashboards for monitoring event ingress rate and quality.

Other options

·         We! Analyze (open source: discussion, download) - a stand alone tool for monitoring connectors and devices to ensure they send events. The tool analyzes connector logs files rather than events and therefore has pros and cons when compared to the "ArcSight System Monitoring" package above.
·         ArcSight Connector watchdog FlexConnector - a Flex connector for analyzing agent.log and providing the results as events to ESM/Logger. As a FlexConnector, it is a great starting point for others to add their own parsing code.





Wednesday, December 21, 2016

TOP 6 SIEM Use Cases

Use Case 1

Detection of Possible Brute Force Attack

With the evolution of faster and more efficient password cracking tools, brute force attacks are on a high against the services of an organization. As a best practice, every organization should configure logging practices for security events such as invalid number of login attempts, any modification to system files, etc., so that any possible attack underway will get noticed and treated before the attack succeeds. Organizations generally apply these security policies via a Group Policy Object (GPO) to all the hosts in their network.
To check for brute force pattern, I have enabled auditing on logon events in the Local Security Policy and I will be feeding my System Win:Security logs to Splunk to check for a brute force pattern against local login attempts.


Note: EventCode: 4625 is used in new versions of the Windows family like Win 7. In older versions, the event code for invalid login attempts is 675, 529.
After this, I log off my machine, and entered the password incorrectly three times in attempt to impersonate a brute force attack.

Since these activities gets logged in Win:Security, which in turn is feeding Splunk in real time, an alert will be created in Splunk, giving analysts an incident to investigate and take responsive actions, like changing the firewall policy to blacklist that IP.

Use Case 2

Detection of Insider Threat

Reportedly, more than 30 percent of attacks are from malicious insiders in any organization. Therefore, every organization must keep the same level of security policies for insiders also.

Acceptable Use Monitoring (AUP)

Acceptable Use Monitoring covers a basic questions, i.e. what resource is being accessed by whom and when. Organizations generally publish policies for users to understand how they can use the organization’s resources in the best way. Organizations should develop a baseline document to set up threshold limits, critical resources information, user roles, and policies, and use that baseline document to monitor user activity, even after business hours, with the help of the SIEM solution.
For example, the below illustration is of logging a user activity on an object. For demonstrative purposes, I have created a file named “Test_Access” on my system. Auditing on object access is enabled in my system, like below in the Local Security Policy.

Enabling auditing on security policies is not enough, and now I have to enable the auditing on the respective file, also named “Test_Access” in this case. I have enabled auditing for Group Name –”Everyone” on this file. Organizations should fingerprint all the sensitive files and corresponding privileges and user group access on them.

For demonstrative purposes, I have selected all the object properties to be audited.

After this, I accessed the “Test_Access” file, which generates an event in Security logs with Event ID 4663, giving user name, action performed, time it was accessed, etc. This useful information can be fed into the SIEM solution through security logs to detect any unauthorized or suspicious object access.

Organizations should develop fingerprints on all the sensitive documents, files and folders, and feed all this information to respective security solutions such as data leakage prevention solutions, application logs, WAF, etc. into the SIEM solution to detect a potential insider threat. Organizations can develop the below use cases in the SIEM solution under AUP.
·         Top malicious DNS requests from user.
·         Incidents from users reported at DLP, spam filtering, web proxy, etc.
·         Transmission of sensitive data in plain text.
·         3rd party users network resource access.
·         Resource access outside business hours.
·         Sensitive resource access failure by user.
·         Privileged user access by resource criticality, access failure, etc.
Use Case 3

Application Defense Check

Besides network, perimeter, and end point security, organizations must develop security measures to protect applications. With attacks like SQL injection, Cross site scripting (XSS), Buffer overflow, and insecure direct object references, organizations have adopted security measures like secure coding practices, use of Web Application Firewall (WAF) which can inspect traffic at layer 7 (Application layer) against a signature, pattern based rules, etc. Along with the log of applications, organizations must also feed SIEM with logs of technologies such as WAF, which can correlate among various security incidents to detect a potential web application attack. One of the very important points to check for in a sensitive application is that the application should encrypt the sensitive information like PII in the logs as well, as these logs will be fed into SIEM, and if unencrypted, sensitive information could be exposed in SIEM.
Organizations must also develop a strategy to secure the operating system (OS) platform onto which the application is hosted. OS as well as application performance logging features must also be enabled. Below are some of the use cases that can be implemented in SIEM to check Application defense.
·         Top Web application Attacks per server.
·         Malicious SQL commands issued by administrator.
·         Applications suspicious performance indicator, resource utilization vector.
·         Application Platform (OS) patch-related status.
·         Web attacks post configuration changed on applications.


Use Case 4

Suspicious Behavior of Log Source

Expected Host/Log Source Not Reporting

Log sources are the feeds for any SIEM solution. Most of the SIEM solution these days comes with an agent-manager deployment model, which means that on all the log sources, light weight SIEM agent software is installed to collect logs and pass them to a manager for analysis. An attacker, after gaining control over a compromised machine/account, tends to stop all such agent services, so that their unauthorized and illegitimate behavior goes unnoticed.
To counter such malformed actions, SIEM should be configured to raise an alert if a host stops forwarding logs after a threshold limit. For example, the below search query (SPL) in Splunk will raise an alert if a host has not forwarded the logs for more than one hour.
As soon as an alert is received with the IP address of the machin under attack, the Incident Response Team (IRT) can start mitigating this issue.

Unexpected Events Per Second (EPS) from Log Sources

Another common pattern found among compromised log sources is that attackers tends to change the configuration files of endpoint agents installed and forward a lot of irrelevant files to the SIEM manager, causing a bandwidth choke between the endpoint agent and manager. This affects the performance of real time searches configured, storage capacity of underlying index for storing logs, etc. Organizations must develop a use case to handle this suspicious behavior of log sources. For example, below is the search (SPL) created in Splunk which can detect unusual forwarding of events from log sources in one day.
An alert will be configured with it to get triggered whenever the amount of EPS from a log source exceeds a threshold value for the IRT team to investigate.
Use Case 5
Malware Check
These days, organizations believe in protecting their network end to end, i.e. right from their network perimeter with devices like firewall, Network Intrusion Prevention System (NIPS), till the endpoints hosts with security features like antivirus and Host Intrusion Prevention System (HIPS), but most organizations collect reports of security incidents from these security products in a standalone mode, which brings problem like false positives, etc.
Correlation logic is the backbone of every SIEM solution, and correlation is more effective when it is built over the output from disparate log sources. For example, an organization can correlate various security events like unusual port activities in firewall, suspicious DNS requests, warnings from Web Application firewall and IDS/IPS, threats recognized from antivirus, HIPS, etc. to detect a potential threat. Organizations can make following sub-use case under this category.
·         Unusual network traffic spikes to and from sources.
·         Endpoints with maximum number of malware threats.
·         Top trends of malware observed; detected, prevented, mitigated.
·         Brute force pattern check on Bastion host.

Use Case 6

Detection of Anomalous Ports, Services and Unpatched Hosts/Network Devices

Hosts or network devices usually get exploited because they often left unhardened, unpatched. Organizations first must develop a baseline hardening guideline that includes rules for all required ports and services rules as per business needs, in addition to best practices like “default deny-all”.
For example, to check for the services being started, systems logs from event-viewer must be fed into the SIEM solution, and a corresponding correlation search must be created against the source name of “Service Control Manager” to detect what anomalous services got started or stopped.

Organizations can also check out for vulnerable ports. Services can be exposed by deploying a vulnerability manager and running a regular scan on the network. The report can be fed into the SIEM solution to get a more comprehensive report encompassing risk rate of the machines in the network. Some use cases that an organization can build from reports are:
·         Top vulnerabilities detected in network.
·         Most vulnerable hosts in the network with highest vulnerabilities.
Another important aspect that an organization should constantly monitor as part of the SIEM process is that all clients or endpoints are properly patched with software updates and feed the client patch status information into the SIEM solution. There are various ways an organization can plan out for this check.
·         Organizations can plan out to check the patch–related status by deploying a Vulnerability Manager and running a regular scan to check out for unpatched endpoints.
·         Organizations can deploy a “centralized update manager” like WSUS and feed the results of the updated status of endpoints into the SIEM solution or can feed the logs of the manager endpoint deployed on endpoints directly into SIEM to detect all unpatched endpoints in the network.

CONCLUSION

Above use-cases are not a comprehensive SIEM security check list, but in order to have success with SIEM, the above listed use cases must be implemented at the minimum on every organization’s check list.