Linux - System Performance tools

sysstat is a great set of tools that you can use to collect statistics for I/O, CPU, memory, network and processes for troubleshooting and monitoring purposes.

Here's summary of the tools.

• iostat reports CPU statistics and input/output statistics for block devices and partitions.
• mpstat reports individual or combined processor related statistics.
• pidstat reports statistics for Linux tasks (processes) : I/O, CPU, memory, etc.
• tapestat reports statistics for tape drives connected to the system.
• cifsiostat reports CIFS statistics.

Sysstat also contains tools you can schedule via cron or systemd to collect and historize performance and activity data:

• sar collects, reports and saves system activity information (see below a list of metrics collected by sar).
• sadc is the system activity data collector, used as a backend for sar.
• sa1 collects and stores binary data in the system activity daily data file. It is a front end to sadc designed to be run from cron or systemd.
• sa2 writes a summarized daily activity report. It is a front end to sar designed to be run from cron or systemd.
• sadf displays data collected by sar in multiple formats (CSV, XML, JSON, etc.) and can be used for data exchange with other programs. This command can also be used to draw graphs for the various activities collected by sar using SVG (Scalable Vector Graphics) format.

 

IPsec VPN Lab, with stateful failover (SSO, Dual Interface Model)

Lots of stuff going on here.
We have two sites for our company named Central-Office and Branch-33.

The CO is connected to our ISP w/ two edge routers, CE_1 w/ an E3 line and CE_2 w/ a backup E1 line. Our branch33 site is using R4, which will establish the IPSec tunnels, allowing the private networks 10.1.33.0/21 talk to the central office 10.1.8.0/21 networks.

Behing them are the two IPSec Concentrators (R1 and R2, 7200 Cisco routers) which have public IP addresses on their out facing interfaces. Here all 4 routers connected to a switch, but of course in a production network you would have redundant switches there also.

Two first hop redundancy plans are integrated as an HSRP group facing to our internal network and the HSRP group facing to our edge routers, which also provides the IPsec redundandcy.

Crypto maps are setup as SSO stateful failover, R1 acting as the primary IPsec point, with R2 being the standby.

There is an extra flavor as IP SLA and reachability tracking is done where R1 and R2 are checking their next-hop routers to the outside world, CE_1 & CE_2. No routing protocol is run inside our company, for claritys sake. Thus we have two static default routes w/ different AD and the mentioned icmp tracking.

One Note, considering the Stateful Failover: "Each time an active device relinquishes control to become the standby device, the active device will reload. This functionality ensures that the state of the new standby device synchronizes correctly with the new active device." (Ref) So since Dynamips doesnt support reloading, the Router dynamips process will crash and you must shut down and turn on the router again to get it going.

Download Project files for GNS3

Links:
Cisco High Availability Solution: Stateful Failover for IPsec
Fun with IPsec stateful failover @ packetlife

Clear VTY Lines

Sometimes a switch/router won't let you connect to it. Coming back as telnet/shh connection refused. If the hardware is otherwise functioning correctly, its likely that your VTY lines are full.
This is how you can:
see all sessions / see active sessions / kill sessions

# sh users all
# sh users
# clear line

Better Wireshark Font/Colors setup

 One of the first things I configure in a newly installed Wireshark is the color for the active/inactive selected item. Normally this item will be displayed in a color which is very close to the default colors for normal packets and its hard to see which line/packet you are on and displaying the details for. So this configuration will make it very easy to see quickly where you are in the file.

Removing Fortigate Reminders on Login to Dashboard

When logging into Fortigate, certain operations are checked and a reminder window has to be passed before you get to the dashboard. For example you will definetly have this window show up after a firmware upgrade. If you have the actions planned for a later date do not want to get the reminder window here's how.

config system global

set gui-forticare-registration-setup-warning disable

set gui-firmware-upgrade-warning disable 

end 

 

 

          

Cisco ASA IPsec setup

Cisco ASA has a different way setting up Phase1/Phase2 parameters for an IPsec tunnel. Where other vendors have specific menu/commands to enter the specific Phase1/Phase2 parameters, here we have groups of commands to accomplish basically the same thing.

PHASE 1

Enter your acceptable Phase1 parameters which will be offered for negotiation with the other ipsec peer.

For IKEv1:

#crypto ikev1 policy 10
   encryption 3des
   hash sha
   group 2
   lifetime 28800

#crypto ikev1 policy 20
   authentication rsa-sig
   encryption aes-256
   hash sha
   group 5
   lifetime 86400

   authentication pre-share

or for IKEv2:

#crypto ikev2 policy 2

   encryption aes-256

   integrity sha

   group 14

   prf sha

   lifetime seconds 86400

Then you setup the "tunnel-group" for attributes and most importantly pre-shared key (or certificate)

#tunnel-group 182.11.104.167 type ipsec-l2l

#tunnel-group 182.11.104.167 general-attributes

   default-group-policy Turkcell_GroupPolicy2

#tunnel-group 182.11.104.167 ipsec-attributes

   ikev1 pre-shared-key *****

PHASE 2

#crypto ipsec ikev1 transform-set ESP-AES-128-SHA esp-aes esp-sha-hmac

#crypto ipsec ikev1 transform-set ESP-AES-128-MD5 esp-aes esp-md5-hmac

#crypto ipsec ikev1 transform-set ESP-AES-192-SHA esp-aes-192 esp-sha-hmac

#crypto ipsec ikev2 ipsec-proposal STRONGPROPOSAL

   protocol esp encryption aes-256

   protocol esp integrity sha-1

#crypto ipsec ikev2 ipsec-proposal WEAKPROPOSAL 

   protocol esp encryption 3des

   protocol esp integrity sha-1

Here's where we define the interesting traffic and 

IKEv1 example cryptomap entry:

crypto map outside_map1 1 match address outside_cryptomap

crypto map outside_map1 1 set pfs group5

crypto map outside_map1 1 set peer 182.11.104.167

crypto map outside_map1 1 set ikev1 transform-set ESP-AES-256-SHA

crypto map outside_map1 1 set security-association lifetime seconds 28800

an IKEv2 example cryptomap entry:

crypto map internet_map0 1 match address internet_cryptomap

crypto map internet_map0 1 set peer 182.11.104.167

crypto map internet_map0 1 set ikev2 ipsec-proposal AES256-SHA1

crypto map internet_map0 interface internet

Loading a new site certificate to FortiAuthenticator

You can receive the following error message when trying to import a PKCS12 (.p12 or .pfx) file into the FortiAuthenticator to use as a certificate/private key pair for your Portal or anyother Local Service.

Unsupported cipher algorithm. This can happen if the PKCS12 file uses unsupported weak ciphers, e.g. RC2. Use the OpenSSL command 'openssl pkcs12 -info -in <file>' to view the file's ciphers on a computer.

One of the reasons this can happen is FortiAuthenticator rejects any PFX files which contain certificate or keys that are encrypted with weak ciphers. So to check what your PFX file contains use openssl.

$ openssl pkcs12 -info -in myexample.pfx

Enter Import Password:

This will give you output listing, Your Private Key encryption cipher:

Shrouded Keybag: pbeWithSHA1And3-KeyTripleDES-CBC, Iteration 2000

And the ciphers used to encrypt the certificates:

PKCS7 Encrypted data: pbeWithSHA1And40BitRC2-CBC, Iteration 2000

We can see here that FortiAuthenticator was right to determine that your PFX file has a3DES encrpyted private key and 40BitRC2 encrypted certificates inside.

Let's see how you fix this and allow FortiAuthenticator to install your PFX file.

One way is to export the private key and certificates to seperate files and use the FortiAuthenticator GUI option to load "Certificate and Private Key"

To get seperate files use the following openssl commands:

openssl pkcs12 -in myexample.pfx -out myexample_certs.pem -nokeys -clcerts

openssl pkcs12 -in myexample.pfx -out myexample_privkey.pem -nocerts -nodes 

 

The alternative to this is to create a new PFX file with stronger ciphers, one that FortiAuthenticator will not reject. This method will have the benefit of teaching you how to create PFX files with openssl.

 

IOS Packaging

Starting with version 12.3, Cisco introduced a new IOS packaging system for routers and switches.

Here are some links from cisco for detailed explanation:
Cisco IOS Packaging Product Bulletin
IOS Packaging, main page
Cisco IOS Packaging Customer Q&A

Building a Security Operations Center (SOC), notes on a blog post

Building a Security Operations Center (SOC): Sharing Experiences from the Front Lines Link

The link above will take you to a very informative blog article on the process of building a Security Operation Center (SOC) and things you must consider, information you should gather while working on this project. The blog is from a website that has since disappeared, but luckily we can read it from the Wayback Machine. Anyway, here are my notes from following this informative peace:

SOC's normally have teams for:

• Incident Response
• Vulnerability Management
• Engineering
• Threat Intelligence
• Threat Hunting

The start-up process

• Find what's plugged into your network (asset management)
• Reduce Noise (find out the "Normal Behavior", which will prevent false positives). Thus anything outside Policy is now an incident
• Develop "right" policies, make all employees aware

SOC Infrastructure

The SOC has it's own applications and infrastructure.

• Security Information and Event Management (SIEM) platform. Needs to meet your needs, good integration with your IT Service Management platform, liked and experienced by your Incident Response team, be scalable, costs inside your budget.
• Incident Management platform. Needs to have good integration with your SIEM, EDR.
• Sysmon or Endpoint Detection and Response (EDR)
• Vulnerability scanner

Communication with other departments

Start with IT Services teams such as System/Service admins, Network team, IAM/Active Directory team, Antivirus team if there is one. Then as the SOC matures, Legal and Privacy departments will need communication and collaboration policies with the SOC.

Collaboration within the SOC

"Threat Intelligence (TI) feeds information to Vulnerability Management (VM), Threat Hunting (TH) and Incident Response (IR). VM takes this information to find out how many systems are vulnerable to the potential threat. IR adds the shared Indicators of Compromise (IOCs) to its detections rules. Threat Hunting (TH) proactively searches in the network for Techniques, Tactics and Procedures (TTPs) based on those IOCs. If Threat Hunting finds activity following those TTPs, if the SIEM triggers an alert based on a IOC hit, or if there is evidence supporting that a vulnerability has been exploited, an incident is raised for the IR team to handle"

Read more details in this fantastic blog post:

Building a Security Operations Center (SOC): Sharing Experiences from the Front Lines

Reference Books for this topic:

• The Tao of Network Security Monitoring
• Incident Response & Computer Forensics