Showing posts with label Juniper. Show all posts
Showing posts with label Juniper. Show all posts

Thursday, March 30, 2017

Juniper Space License Issue on Citrix Xen Environment

Based on Juniper "Junos Space Virtual Appliance Installation and Configuration Guide" , JunOS Space " must deploy the virtual appliance on a VMware ESX, VMWare ESXi or KVM server, which provides a CPU, hard disk, RAM, and a network controller, but requires installation of an operating system and applications to become fully functional."

In my test environment, one JunOS Space has been installed on Citrix Xen environment and it is working fine until we tried to import a license.

The license was generated from Juniper License site and emailed to us in a txt file. It used to work on another machine hosted in Vmware ESX environment. Unfortunately, this time, JunOS Space said no.

The License Information windows says:
License upload failed. Please check the following:
1) License data format
2) License Keys
Juniper Space VE at Citrix Xen Server - License Error



Thursday, March 23, 2017

Add Juniper SRX Cluster into JunOS Space 16.1 Security Director

My old post "Import Existing Juniper SRX Cluster into JunOS Space Security Director" was created based on Space 14.1 and SRX11.x version. Now both have been upgraded. Space NMP and Security Director have been upgrade to 16.1 (Post is here). SRX240H has been upgrade to 12.1D46.55.

Basically, all steps are similar except the web interface is different. What you need to do is to configure your SRX cluster with a master-only ip on both nodes. The configuration should looks like this:

Wednesday, March 22, 2017

Juniper JUNOS Commands (Tips and Tricks)

Juniper Networks has a Day one book for 'JunOS Tips, Techniques, and Templates 2011' in Junos Fundamentals Series. To record some my own tips, I put them together in this post. Let me know if you have some more to share.

1.  Find big size files 

find . -type f -size +10000 -exec ls -lh {} \; 


root@FW% find . -type f -size +10000 -exec ls -lh {} \;
-rw-r--r--  1 930  929   134M Jan  5 17:34 ./cf/packages/junos-11.4R6.6-domestic
-rw-r--r--  1 root  wheel   139M Sep  8  2011 ./cf/var/log/junos-srxsme-11.2R2.4-domestic.tgz
-rw-r-----  1 root  wheel   4.9M Feb 11 17:12 ./cf/var/db/idpd/db/secdb_02.db
-rw-r-----  1 root  wheel   6.7M Feb 11 17:13 ./cf/var/db/idpd/db/secdb_03.db
-rw-r-----  1 root  wheel    64M Feb 11 17:13 ./cf/var/db/idpd/db/secdb_06.db
-rwxr-xr-x  1 admin  20    24M May 23 08:38 ./cf/var/db/idpd/nsm-download/SignatureUpdate.xml
-r-xr-xr-x  1 root  wheel   5.2M Jan  5 17:33 ./jail/html/dynamic-vpn/client/jam/InstallerComponentSRX.exe
-rw-r--r--  1 root  wheel   139M Sep  8  2011 ./jail/var/log/junos-srxsme-11.2R2.4-domestic.tgz
-rw-r-----  1 root  config    14M Feb  8 22:16 ./mfs/var/run/db/schema.db
-rw-r-----  1 root  wheel    10M Feb  8 22:19 ./mfs/var/sdb/log.0000000001
-r--r--r--  1 root  wheel   6.5M Jan  5 13:59 ./usr/lib/dd/libjkernel-dd.so
-r-xr-xr-x  1 root  wheel    13M Jan  5 15:39 ./usr/sbin/authd
-r-xr-xr-x  1 root  wheel   6.0M Jan  5 16:51 ./usr/sbin/chassisd
-r-xr-xr-x  1 root  wheel    27M Jan  5 13:05 ./usr/sbin/flowd_octeon
-r-xr-xr-x  1 root  wheel    34M Jan  5 13:05 ./usr/sbin/flowd_octeon_hm
-r-xr-xr-x  1 root  wheel   5.5M Jan  5 16:51 ./usr/sbin/kmd
-r-xr-xr-x  1 root  wheel    13M Jan  5 16:24 ./usr/sbin/rpd

% find / -size +100000 | xargs ls -lhS
find: /mfs/var/spool/opielocks: Permission denied
-rw-r--r--  1 930   929     142M Aug 28  2014 /cf/packages/junos-12.1X44-D40.2-domestic
-rw-r-----  1 root  wheel    84M Feb 23 21:31 /cf/var/db/idpd/db/secdb_06.db


Tuesday, March 21, 2017

JunOS Space Network Management Platform Basic Configuration including Log Collector

JunOS Space is in my environment and starting to replace NSM. I have played with in testing lab which recorded in my previous posts:
In this post, I will focus on more regarding JunOS Space itself, some basic configuration to get JunOS space into your production environment.

Notes: Recently Space has been upgraded from 14.1 to 16.1 with my post: Juniper JunOS Space Upgrade Procedures from 14.1 to 16.1. The installation and configuration steps for 16.1 is similar as 14.1. This post is updated during configuring JunOS Space 16.1.

Juniper JunOS Space Upgrade Procedures from 14.1 to 16.1

Usually you can easily upgrade an application from the Junos Space user interface. You must download the image file for the new version of the application, navigate to the Applications page (Network Management Platform > Administration > Applications) and select the application that you want to upgrade. From the right-click menu, choose Upgrade Application to upload the image file into Junos Space via HTTP or SCP.

But upgrade JunOS Space to latest version 16.1 is different and it is not a easy task. There are many steps to follow especially the last step to upgrade to 16.1 from 15.2R2. Here is my recent upgrade procedures.

Steps to upgrade JunOS Space 14.1  to the latest version 16.1:

Monday, November 28, 2016

Procedures to Deploy RMA device into Juniper SRX Chassis Cluster

Juniper KB mentioned some RMA steps for failed Juniper device replacement. There are some steps not clear enough. I put some more configuration steps in this post for future reference:

There are many preparation works before you can add RMA device into your chassis group.


Saturday, November 26, 2016

Juniper Firewall SRX240H Crashed with Error 'nearing maxproc limit by uid 0,please see tuning(7) and login.conf(5)'

One of Juniper Firewall SRX240H had a serious crash. Manual reboot/shutdown did not work. To reset it, I would have to do a hard reset / power cycle device.

It would allow to log in from console, but you wont be able to see any configuration.

Here is outputs from this crashed Juniper SRX240H console:

Wednesday, October 26, 2016

Juniper SRX340 HA Configuraiton

The SRX340 Services Gateway has a capacity of 3 gigabits per second (Gbps) and is 1 rack unit (U) tall. This services gateway has eight 1 G Ethernet ports, eight 1 G SFP ports, one management port, 4 GB of DRAM memory, 8 GB of flash memory, and four Mini-Physical Interface Module (Mini-PIM) slots.

SRX 340 Front Panel

SRX 340 Back Panel














The connection is a little different from SRX 240 and 1400. Here are some related posts:

Wednesday, August 10, 2016

JunOS SRX Cluster Upgrade Failed


For SRX1400, SRX3400, SRX3600, SRX5600, and SRX5800 devices, command introduced in Junos OS Release 9.6 and support for reboot as a required parameter added in Junos OS Release 11.2R2. For SRX100, SRX210, SRX220, SRX240, and SRX650 devices, command introduced in Junos OS Release 11.2R2. For SRX5400 devices, the command is introduced in Junos OS Release 12.1X46-D20.

Symptoms: 

Symptom 1: "tar: Archive contains obsolescent base-64 headers"

root@fw-1> request system software add no-copy /var/tmp/junos-srxsme-12.1X44-D40.2-domestic.tgz no-validate 
Formatting alternate root (/dev/da0s2a)...
/dev/da0s2a: 627.4MB (1284940 sectors) block size 16384, fragment size 2048
        using 4 cylinder groups of 156.86MB, 10039 blks, 20096 inodes.
super-block backups (for fsck -b #) at:
 32, 321280, 642528, 963776
Extracting /var/tmp/junos-srxsme-12.1X44-D40.2-domestic.tgz ...
tar: Skipping to next header
tar: Archive contains obsolescent base-64 headers

gzip: stdin: invalid compressed data--format violated
tar: Child returned status 1
tar: Error exit delayed from previous errors
ERROR: Failed to extract /var/tmp/junos-srxsme-12.1X44-D40.2-domestic.tgz to /altroot/cf/packages/install-tmp/junos-12.1X44-D40.2-domestic

Sunday, January 17, 2016

Juniper SRX Minor Alarm Messages - Autorecovery and Rescue Information

During working on Juniper products, such as SRX, NSM and Space, sometimes, there are minor alarms in the system and it requires some actions to clear those alarms based on different situation.

Symptoms:

Here are some minor alarms I met during working on Juniper products.

1.  Autorecovery has recovered corrupted information

root@fw1-1> show system alarms    
node0:
--------------------------------------------------------------------------
No alarms currently active

node1:
--------------------------------------------------------------------------
1 alarms currently active
Alarm time               Class  Description
2015-10-18 12:36:25 UTC  Minor  Autorecovery has recovered corrupted information


2. Autorecovery information needs to be saved


{primary:node1}
root@fw2-1> show system alarms                        
node0:
--------------------------------------------------------------------------
2 alarms currently active
Alarm time               Class  Description
2015-11-22 16:49:31 UTC  Minor  Autorecovery has recovered corrupted information
2015-11-22 16:49:29 UTC  Minor  Autorecovery information needs to be saved

node1:
--------------------------------------------------------------------------
No alarms currently active


3. Rescue configuration is not set

Noticed there is a minor alarm message at SRX status in NSM, showing in the following screenshot:



When the mouse hanging over the Monor, it shows no value specified.

Lets log into device from CLI:


root@M-1> show system alarms
node0:
--------------------------------------------------------------------------
2 alarms currently active
Alarm time               Class  Description
2014-03-16 12:55:30 UTC  Minor  Autorecovery information needs to be saved
2014-03-16 12:55:29 UTC  Minor  Rescue configuration is not set

node1:
--------------------------------------------------------------------------
No alarms currently active


Solution:

1.  for Minor Autorecovery has recovered corrupted information

Autorecovery has recovered corrupted information
Minor
This alarm indicates:
  • Boot time integrity check failed for certain items; however, the items have been recovered successfully.
  • No action is required.
  • Alarm will be cleared on next bootup.

2. for Autorecovery information needs to be saved

Save the Autorecovery state information

3. for Rescue configuration is not set



root@M-1> request system configuration rescue save

{primary:node0}
john@M-1> show system alarms
node0:
--------------------------------------------------------------------------
1 alarms currently active
Alarm time               Class  Description
2014-03-16 12:55:30 UTC  Minor  Autorecovery information needs to be saved

node1:
--------------------------------------------------------------------------
No alarms currently active


root@M-1> request system autorecovery state ?
Possible completions:
  clear                Delete previously saved autorecovery state
  recover              Check for problems and recover state if needed
  save                 Save autorecovery state
{primary:node0}
root@M-1> request system autorecovery state save
Saving config recovery information
Saving license recovery information
Saving BSD label recovery information

{primary:node0}
root@M-1> show system alarms
node0:
--------------------------------------------------------------------------
No alarms currently active

node1:
--------------------------------------------------------------------------
No alarms currently active




After a couple of minutes, take a look at NSM Device Status again, it shows green without any alarms.

Notes: you may need to reboot both cluster members one by one to clear those alarm messages from system.

Reference:

SRX alarm: Autorecovery information needs to be saved
Understanding Integrity Check and Autorecovery of Configuration, Licenses, and Disk Information


Tuesday, January 12, 2016

JunOS Space - Warning Message for Consolidated Configuration

Junos Space is a comprehensive network management solution that simplifies and automates management of Juniper’s switching, routing, and security devices. To those security administrator who like command line, they will still prefer to use command line to change certain things such as host name, routes, system configuration etc. Juniper call it out-of-band commit since it happens at device itself, not from JunOS Space. Usually the JunOS space will auto-synchronize the JunOS Space configuration database with device configuration.

In a network managed by Network Director, three separate repositories about device configuration are maintained:

  • The configuration information on the devices themselves. Each switch and wireless LAN controller maintains its own configuration record.
  • The configuration information maintained by the Junos Space Network Management Platform. When a device is discovered, either by Junos Space or Network Director, Junos Space stores a record of the configuration on that device.
  • The configuration information maintained by Network Director in Build mode. This information takes the form of the profiles assigned to the device, plus the additional configuration, such as LAG and access point configuration, that you can do under device management.

In Network Director, the configuration state of a device is shown as In Sync when the configuration information in all three repositories match. If there is a conflict between the configuration information in one or more of the repositories, Network Director shows the device configuration state as Out of Sync. An Out of Sync state is usually the result of out-of-band configuration changes—that is, configuration changes made to a device using a management tool other than Network Director.

When configuration changes are made on a physical device that Junos Space Network Management Platform manages, Junos Space Network Management Platform reacts differently depending on whether the network itself is the system of record (NSOR) or Junos Space Network Management Platform is the system of record (SSOR).



In the NSOR case which is default, Junos Space Network Management Platform receives a system log message and automatically resynchronizes with the device. This ensures that the device inventory information in the Junos Space Network Management Platform database matches the current configuration information on the device. Please check Juniper Doc "Understanding How Junos Space Automatically Resynchronizes Managed Devices" to get more information about it.

I met one warning message during working on CLI and Space. Sometimes, after I made some changes on firewall itself , usually it relates to system settings or routes. I thought it would be  auto-synchronized with space in 25 seconds. But in fact, it did not. Then when you worked on Space again to make firewall policy changes, during push and update process, it will warn you with following information:
"Some of the selected devices have their status as 'Device changed' and some may have consolidated configurations in 'Created' or 'Approved' states. Device update may cause conflict with consolidated configuration changes. Do you want to continue with Publish and update?
"





To resolve this, only need two steps:
1. Resynchronize with network
Select Devices -> Device Management. Right click the device, and select Resynchronize with network. Please select all your cluster members to do it when enabled clustering on your devices.


2. Resynchronize with Platform:
Select Security Director > Security Director Devices. Right-click the device, and select Resynchronize with Platform. The Sync Device Status page appears to confirm the sync action. To sync the changes, click Sync.

You will see synchronizing from device's Managed Status


After re-synchronization completed, the warning message will go away.

3. Other Steps
If the warning message is still there, try following steps:

3.1.Import device configuration from Security Director devices.
3.2.Assign the newly imported policy to the device.
3.3.Publish the policy once you have assign it.
3.4.Update the device from Security Director devices.


4. Last Resort


The last resource would be to delete the device and re-add, then re-import all FW, IPS, NAT, VPN Policies and assign it to the device, then publish and update.






Tuesday, December 15, 2015

Understanding Juniper SRX TCP Security Check

Juniper SRX is a stateful firewall and allows traffic which matches an existing session. Sessions are created when a TCP SYN packet is received and it is permitted by the security policy. This of course means that the firewall needs to see both directions of a flow (client-server and server-client), otherwise these checks will block legitimate packets.

Following flow chart illustrates packet flow sequences both when SYN flag checking is enabled and when it is disabled.
SYN Flag Checking

By default, security TCP check is enabled on all TCP flow sessions. The Junos operating system (Junos OS) performs the following operations during TCP sessions:

  • Checks for SYN flags in the first packet of a session and rejects any TCP segments with non- SYN flags that attempt to initiate a session.
  • Validates the TCP sequence numbers during stateful inspection.

Reset packet is turned off for non-SYN session TCP packets:
{primary:node0}
root@fw-mgmt-trn1-1> show security zones
node0:
--------------------------------------------------------------------------

Security zone: MGMT1
  Send reset for non-SYN session TCP packets: Off
  Policy configurable: Yes
  Interfaces bound: 1
  Interfaces:
    reth4.201

Security zone: TSMGMT
  Send reset for non-SYN session TCP packets: Off
  Policy configurable: Yes
  Interfaces bound: 1
  Interfaces:
    reth4.198

Security zone: MGMT2
  Send reset for non-SYN session TCP packets: Off
  Policy configurable: Yes
  Interfaces bound: 1
  Interfaces:
    reth3.0
We can enable reset packets when received non-syn tcp session packets.
{primary:node0}[edit]
root@fw-mgmt-1# set security zones security-zone MGMT1 tcp-rst

{primary:node0}[edit]
root@fw-mgmt-1# set security zones security-zone TSMGMT tcp-rst

{primary:node0}[edit]
root@fw-mgmt-1# set security zones security-zone MGMT tcp-rst        

Check the settings again:
root@fw-mgmt-1> show security zones 
node0:
--------------------------------------------------------------------------

Security zone: MGMT1
  Send reset for non-SYN session TCP packets: On
  Policy configurable: Yes  
  Interfaces bound: 1
  Interfaces:
    reth4.201

Security zone: TSMGMT
  Send reset for non-SYN session TCP packets: On
  Policy configurable: Yes  
  Interfaces bound: 1
  Interfaces:
    reth4.198

Security zone: MGMT2
  Send reset for non-SYN session TCP packets: On
  Policy configurable: Yes  
  Interfaces bound: 1
  Interfaces:reth3.0


Junos OS provides a mechanism for disabling security checks on TCP packets to ensure interoperability with hosts and devices with faulty TCP implementations. During no-SYN-check the Junos OS does not look for the TCP SYN packet for session creation. No-sequence check disables TCP sequence checking validation. Also, increases throughput. SYN check and sequence check are enabled by default. The set security flow command disables TCP SYN checks and TCP sequence checks on all TCP sessions thus reduces security. This may be required in scenarios with customers like big transfer files, or with applications that do not correctly work with standards.

Another reason to disable syn-check and sequence-check is the asymmetric flows in your environment. It is best, whenever possible, to ensure that asymmetric flows do not occur; but this is not always possible. So, you can disable these checks globally on the SRX device.

To disable TCP packet security checks:
set security flow tcp-session no-syn-check
set security flow tcp-session no-sequence-check

After you disabled the tcp options, tcp-syn-check, and tcp-sequence-check that are configured at global level, you might want to configure TCP packet security checks at the policy level.

Note: Disabling the global SYN check and enforcing the SYN check after policy search will greatly impact the number of packets that the router can process. This in turn will result in intense CPU operations.

Configure the checking for the TCP SYN bit before creating a session:
[edit]
user@host# set security policies from-zone Zone-A to-zone Zone-B policy pol1 then permit tcp-options syn-check-required

Configure the checking for sequence numbers in TCP segments during stateful inspection:
[edit]
user@host# set security policies from-zone Zone-A to-zone Zone-B policy pol1 then permit tcp-options sequence-check-required

It is also possible to disable TCP SYN or sequence checking on one policy and enable them on all other policies, an apply-group can be used to complete this configuration based on KB24566.


Reference:



Thursday, December 10, 2015

Juniper SRX Logging Methods and Configuration: Stream Mode vs Event Mode

JunOS has strong flexibility on many features. One of them is logging. It support flexible logging options. This post summarizes some concepts I learned from my work and studying. 

1.Understand Juniper SRX logging Type:

1.1 System Logging

Junos OS supports configuring and monitoring of system log messages (also called syslog messages). You can configure files to log system messages and also assign attributes, such as severity levels, to messages. Reboot requests are recorded to the system log files, which you can view with the show log command. SRX Series devices can send system log messages from the control plane (Routing Engine) to one or more destinations. Destinations can include local files on the SRX Series device (because the SRX Series device is a syslog server), remote syslog servers, user terminals, and the system console.


admin@fw-1> show configuration system syslog
archive size 750k files 2;
user * {
    any emergency;
}
host 10.9.0.33 {
    any any;
    change-log none;
    interactive-commands none;
    explicit-priority;
}
host 10.9.8.52 {
    any any;
    source-address 10.9.8.20;
}
file messages {
    any critical;
    authorization info;
    explicit-priority;
}
file interactive-commands {
    interactive-commands error;
}
}


1.2 Traffic Logging (Event Mode)

You can use traffic logs to track usage patterns or troubleshoot issues for a specific policy. You can configure a policy so that traffic information is logged when a session begins (session-init) and/or closes (session-close). To generate traffic logs for multiple policies, you must configure each policy to log traffic information. You also must configure syslog messages with a severity level of info or any. In the default configuration, these messages and all other logging messages are sent to a local log file named messages.



admin@fw-1> show configuration system syslog
archive size 750k files 2;
user * {
    any emergency;
}
host 10.9.0.33 {
    any any;
    change-log none;
    interactive-commands none;
    explicit-priority;
}
host 10.9.8.52 {
    any any;
    source-address 10.9.8.20;
}
file messages {
    any critical;
    authorization info;
    explicit-priority;
}
file interactive-commands {
    interactive-commands error;
}
file traffic-create {
    any any;
    match RT_FLOW_SESSION_CREATE;
    structured-data;
}
file traffic-deny {
    any any;
    match RT_FLOW_SESSION_DENY;
}
file traffic-flow {
    user info;
    match RT_FLOW;
    archive size 1000k files 5 world-readable;
    structured-data;
}



admin@fw-1> show log ?
Possible completions:
  <[Enter]>            Execute this command
  <filename>           Name of log file
  IKELOG               Size: 270913, Last changed: Feb 15 2015
  PKITRACE             Size: 138153, Last changed: Oct 02 02:22:41
  PKITRACE.0.gz        Size: 98723, Last changed: Sep 27 05:12:14
  __jsrpd_commit_check__  Size: 6456, Last changed: Dec 21 2014
  appidd               Size: 0, Last changed: May 13 2014
  authd_libstats       Size: 0, Last changed: May 13 2014
  authd_profilelib     Size: 0, Last changed: May 13 2014
  authd_sdb.log        Size: 0, Last changed: May 13 2014
  authlib_jdhcpd_trace.log  Size: 0, Last changed: Jan 18 2015
  bin_messages         Size: 7, Last changed: May 13 2014
  chassisd             Size: 1173869, Last changed: Oct 01 22:54:45
  cosd                 Size: 98079, Last changed: Sep 20 11:36:47
  dcd                  Size: 251523, Last changed: Sep 21 14:25:57
  default-log-messages  Size: 612840, Last changed: Oct 02 02:19:39
  default-log-messages.0.gz  Size: 1027366, Last changed: Sep 20 18:45:01
  default-log-messages.1.gz  Size: 1323072, Last changed: Sep 20 18:30:00
  dfwc                 Size: 0, Last changed: May 13 2014
  e2e_events           Size: 239, Last changed: Sep 20 11:45:31
  eccd                 Size: 0, Last changed: May 13 2014
  ext/                 Last changed: May 13 2014
  flowc/               Last changed: May 13 2014
  fwauthd_chk_only     Size: 298, Last changed: Dec 21 2014
  ggsn/                Last changed: May 13 2014
  gprsd_chk_only       Size: 1335, Last changed: Dec 21 2014
  gres-tp              Size: 23569, Last changed: Sep 20 11:36:47
  group_db.log         Size: 0, Last changed: May 13 2014
  helplog              Size: 64, Last changed: Nov 17 2014
  hostname-cached      Size: 408, Last changed: Dec 21 2014
  httpd.log            Size: 1533, Last changed: Sep 20 18:36:17
  idpd                 Size: 0, Last changed: May 13 2014
  idpd.addver          Size: 185, Last changed: Sep 20 19:15:01
  idpd_err             Size: 208962, Last changed: Sep 20 19:15:11
  idpd_err.1           Size: 1048851, Last changed: Sep 20 18:55:14
  ifstraced            Size: 120, Last changed: Dec 21 2014
  indb                 Size: 967833, Last changed: Dec 21 2014
  install              Size: 3927, Last changed: Dec 21 2014
  interactive-commands  Size: 82, Last changed: Sep 21 14:25:52
  inventory            Size: 17170, Last changed: Sep 20 11:45:34
  ipfd                 Size: 97046, Last changed: Sep 28 10:01:08
  ipfd_chk_only        Size: 32, Last changed: Dec 21 2014
  jdhcpd_era_discover.log  Size: 8892, Last changed: Oct 01 20:07:42
  jdhcpd_era_discover.log.0  Size: 43387, Last changed: Aug 13 23:11:34
  jdhcpd_era_discover.log.1  Size: 25529, Last changed: Jun 19 14:49:47
  jdhcpd_era_discover.log.2  Size: 422808, Last changed: Apr 17 16:00:00
  jdhcpd_era_discover.log.3  Size: 0, Last changed: Jan 18 2015
  jdhcpd_era_solicit.log  Size: 595, Last changed: Sep 20 11:36:47
  jdhcpd_era_solicit.log.0  Size: 595, Last changed: Jul 19 13:01:48
  jdhcpd_era_solicit.log.1  Size: 595, Last changed: May 17 12:26:15
  jdhcpd_era_solicit.log.2  Size: 595, Last changed: Jan 18 2015
  jdhcpd_era_solicit.log.3  Size: 0, Last changed: Jan 18 2015
  jdhcpd_sdb.log       Size: 0, Last changed: Jan 18 2015
  jsrpd                Size: 841811, Last changed: Sep 28 10:01:17
  kmd                  Size: 369441, Last changed: Sep 20 18:36:26
  license              Size: 0, Last changed: May 13 2014
  license_subs_trace.log  Size: 16976, Last changed: Sep 20 11:36:47
  lsys-cpu-utilization-log  Size: 0, Last changed: May 13 2014
  mastership           Size: 13036, Last changed: Sep 20 11:36:47
  messages             Size: 687915, Last changed: Oct 02 02:26:11
  messages.0.gz        Size: 38283, Last changed: Sep 26 05:45:00
  messages.1.gz        Size: 38105, Last changed: Sep 24 22:15:00
  nsd_chk_only         Size: 1021282, Last changed: Sep 29 18:26:35
  nstraced             Size: 58027, Last changed: Sep 20 11:43:30
  nstraced_chk_only    Size: 370, Last changed: Mar 18 2015
  pcre_db.log          Size: 0, Last changed: May 13 2014
  pf                   Size: 1152, Last changed: Dec 21 2014
  pfed                 Size: 0, Last changed: May 13 2014
  pfed_jdhcpd_trace.log  Size: 0, Last changed: Jan 18 2015
  pgmd                 Size: 385, Last changed: Dec 21 2014
  pkid                 Size: 828994, Last changed: Dec 21 2014
  rexp_db.log          Size: 0, Last changed: May 13 2014
  rsi.1400.0118        Size: 4620227, Last changed: Jan 18 2015
  rsi_2015_02_04       Size: 4762354, Last changed: Feb 04 2015
  rtlogd               Size: 3952, Last changed: Sep 29 18:26:56
  smartd.trace         Size: 133439, Last changed: Oct 01 23:51:05
  traffic-create       Size: 9307887, Last changed: Oct 02 02:26:11
  traffic-create.0.gz  Size: 593738, Last changed: Oct 02 02:15:00
  traffic-create.1.gz  Size: 679624, Last changed: Oct 02 02:00:00
  traffic-deny         Size: 733722, Last changed: Oct 02 02:26:11
  traffic-deny.0.gz    Size: 30893, Last changed: Oct 02 02:00:00
  traffic-deny.1.gz    Size: 29997, Last changed: Oct 02 01:30:00
  traffic-flow         Size: 14043535, Last changed: Oct 02 02:26:11
  traffic-flow.0.gz    Size: 1110300, Last changed: Oct 02 02:15:00
  traffic-flow.1.gz    Size: 1194867, Last changed: Oct 02 02:00:00
  traffic-flow.2.gz    Size: 1223703, Last changed: Oct 02 01:45:00
  traffic-flow.3.gz    Size: 1205868, Last changed: Oct 02 01:30:00
  traffic-flow.4.gz    Size: 1196097, Last changed: Oct 02 01:15:01
  user                 Show recent user logins
  utmd-av              Size: 960, Last changed: Sep 20 11:36:47
  utmp                 Size: 0, Last changed: May 13 2014
  |                    Pipe through a command

1.3 Notes from Juniper KB:

System LoggingTraffic Logging
SRX Branch Devices
SRX100
SRX110
SRX210
SRX220
SRX240
SRX550
SRX650
 KB16502 KB16509
SRX High-End Devices
SRX1400
SRX3400
SRX3600
SRX5600
SRX5800
 KB16502 KB16506

2. Understand Juniper SRX Logging Methods:

Control Plane and Data Plane

2.1 Control Plane Logging

The control plane logs have to do with events triggered by daemons on the control plane. This includes messages about the underlying hardware (chassisd), general-purpose messages (messages), and various protocol daemons like IDPD, appidd, and so on. Control plane logging is on by default to log locally, but you can override this with your own logfiles, syslog hosts, and criteria for different log messages. All logs are stored in the /var/log directory on the control plane. The configuration has been described at section 1.1

Services on the control plane:
  • Management Daemon (MGD):  Provides the interface between the UI components and the backend configuration and is responsible for acting on the Junos configuration to the system itself.
  • Routing Protocol Daemon (RPD) : All routing protocols including RIP, OSPF, IS-IS, BGP, PIM, IPv6 counterparts, and so on.
  • User interfaces: Console, Telnet, SSH, J-Web, NetConf.
  • Filesystem interfaces: FTP/SCP.
  • Syslogd: Logging subsystem on the control plane, different than what is on the data plane. This generates the OS and application logs on the control plane.
  • Networking services: DNS, DHCP, NTP, ICMP, ARP/ND, SNMP.
  • Chassisd: Controls the hardware operations of the data plane and interfaces with the components to ensure they are active and operating properly.
  • JSRPD: This is the high availability daemon that runs the HA functionality between two SRX chassis in an HA cluster.

2.2 Data Plane Logging

Data plane logs are primarily those generated by components that process traffic on the data plane. These include the firewall logs (RT_LOG, which stands for Real-Time Log because it is not stored on the data plane) from the flowd process, IPS logs, UTM logs, and logs from other security components like Screens. Data plane logging is off by default and must be configured. Typically, it is recommended that you send logs off the SRX to a syslog host due to the large volume of logs that can be generated from the data plane, particularly on high-end SRX platforms like the 5800. In fact, it can take an entire infrastructure of syslog servers to handle the large volume of syslog messages that the high-end SRX can generate per second. For this reason, there are two different mechanisms that we can use to log messages to the control plane, as discussed in the next section.

Services on the data plane:
  • Intrusion Detection and Prevention Daemon (IDPD)
  • IKED
  • PKID

2.2.1 Event Mode 

Event mode  - control plane log processing - used on low end devices. Optionally even rate can be specified. Once event mode is enabled under "security" then the logging to local file can configure under "system syslog" as above at section 1.2.  You also can configure that security traffic logs are handled through the eventd process and sent with system logs though control panel Routing Engine.


admin@fw-1> show configuration security
log {
    mode event;
    event-rate 1000;
    format sd-syslog;
    source-address 10.9.8.20;
    stream securitylog {
        format sd-syslog;
        category all;
        host {
            10.9.8.52;
        }
    }
    stream LogCollector {
        host {
            10.9.20.17;
        }
    }
    stream TO-10.9.20.33 {
        format sd-syslog;
        category all;
        host {
            10.9.20.33;
        }
    }
}


2.2.2 Stream Mode

Stream mode - data plane logging - Normally used on high end SRX devcies but can be configured on any SRX devices. Under security the syslog parameters can be specified, e.g. syslog server, syslog format, facility.

Note: SRX can only log to the control plane (Event mode) or log out the data plane (Stream mode) at one time

Security logs such as traffic and IDP logs are able to be streamed through the traffic interface ports to a remote syslog server. SRX devices do not send streamed session logs to the Routing Engine (RE). Because system logging is performed on the RE, session or traffic logs cannot be written to the RE file system. Therefore, all traffic logging must be sent to a remote syslog server. Because fxp0 belongs to the RE, the remote syslog server must be reachable by an interface on an IOC. Traffic logging cannot be sent out through fxp0.

When the logging mode is set to stream, security traffic logs generated in the data plane are streamed out a revenue traffic port directly to a remote server. That also means your local log file will stop logging. Match condition configuration in System -> Syslog part does not work in Stream mode.  Its as per design, the Routing engine is the one which puts the match condition and filters the log,
since when we use stream mode the traffic is streamed out of the data plane itself and doesn't reach the RE the match condition dose not work when using stream mode and only works in event mode.

Basically, only thing works at System - Syslog section are those generated from control plane.


admin@fw-twn1-1> show configuration security
log {
    cache;
    mode stream;
    format sd-syslog;
    source-address 10.2.2.13;
    stream TO-10-0-0-4 {
        format sd-syslog;
        category all;
        host {
            10.0.00.4;
        }
    }
    stream TO-10.4.20.33 {
        format sd-syslog;
        category all;
        host {
            10.4.20.33;
        }
    }
    inactive: traceoptions {
        file jtac;
        flag all;
    }
}

Please keep one thing in mind. Maximum stream destination is three. If you are configuration more than three destination, you will get following error messages from CLI. If you are using JunOS Space, it wont allow you add more than three destination either.


admin@fw-1> commit and-quit
[edit security log]
  'stream'
    number of elements exceeds limit of 3
error: commit failed: (number of elements exceeds limit)




Control plane pushing configuration to data plane



admin@fw-srx1> show security log
Security logging is disabled

“show security log” will show you something about audit log but not policy logging after enabled cache in the security log section, else SRX will show you Security Log disabled.

After you enabled cache under security -> log configuration, as shown at the configuration of section 2.2.2, you will get output like below once you use command show security log:

admin@fw-1> show security log
Event time               Message
2015-10-02 09:15:04 UTC  UI_CMDLINE_READ_LINE: User 'root', command 'xml-mode netconf need-trailer '
2015-10-02 09:15:04 UTC  UI_LOGOUT_EVENT: User 'root' logout
2015-10-02 09:15:04 UTC  UI_LOGIN_EVENT: User 'root' login, class 'super-user' [55330], ssh-connection '10.4.20.21 7804 10.2.1.14 59097', client-mode 'cli'
2015-10-02 09:15:04 UTC  UI_CMDLINE_READ_LINE: User 'root', command 'xml-mode netconf need-trailer '


Reference:







Sunday, September 20, 2015

SRX Load Rescue Configuration After Reboot

It did not happen often, but when it happened, you will need to know how to fix it.


The rescue configuration is a previously committed, valid configuration. You must have previously set the rescue configuration through the J-Web interface or the CLI.

test@fw-srx-2> request system configuration rescue save 

test@fw-srx-2> show system configuration rescue 



During today's work, one of SRX firewalls got a problem to load regular configuration file, and it loaded rescue configuration. Here is what the console output told me:



*** FINAL System shutdown message from admin@fw-srx-2 ***           

System going down IMMEDIATELY                                                  

                                                                          


{secondary:node1}

john@fw-srx-2> Waiting (max 60 seconds) for system process `vnlru' to stop...done
Waiting (max 60 seconds) for system process `vnlru_mem' to stop...done
Waiting (max 60 seconds) for system process `bufdaemon' to stop...done
Waiting (max 60 seconds) for system process `syncer' to stop...
Syncing disks, vnodes remaining...0 0 0 done

syncing disks... All buffers synced.

Uptime: 32m35s
Rebooting...
cpu_reset: Stopping other CPUs


U-Boot 1.1.6-JNPR-1.7 (Build time: May  4 2010 - 06:59:58)


SRX_240_HIGHMEM board revision major:1, minor:50, serial #: AAEK3334

OCTEON CN5230R-SCP pass 2.0, Core clock: 600 MHz, DDR clock: 333 MHz (666 Mhz data rate)
DRAM:  1024 MB
Starting Memory POST... 
Checking datalines... OK
Checking address lines... OK
Checking 512K memory for U-Boot... OK.
Running U-Boot CRC Test... OK.
Flash:  4 MB
USB:   scanning bus for devices... 
Root Hub 0: 3 USB Device(s) found
Root Hub 1: 1 USB Device(s) found
       scanning bus for storage devices... 1 Storage Device(s) found
Clearing DRAM........ done
BIST check passed.
1:00:00.0 Vendor/Device ID = 0x811210b5
1:01:07.0 Vendor/Device ID = 0xc72414e4
Boot Media: nand-flash usb 
Net:   octeth0
POST Passed
Press SPACE to abort autoboot in 1 seconds
ELF file is 32 bit
Loading .text @ 0x8f000078 (246092 bytes)
Loading .rodata @ 0x8f03c1c4 (13940 bytes)
Loading .rodata.str1.4 @ 0x8f03f838 (16580 bytes)
Loading set_Xcommand_set @ 0x8f0438fc (104 bytes)
Loading .rodata.cst4 @ 0x8f043964 (20 bytes)
Loading .data @ 0x8f044000 (5620 bytes)
Loading .data.rel.ro @ 0x8f0455f4 (120 bytes)
Loading .data.rel @ 0x8f04566c (136 bytes)
Clearing .bss @ 0x8f0456f8 (11912 bytes)
## Starting application at 0x8f000078 ...
Consoles: U-Boot console  
Found compatible API, ver. 1.7

FreeBSD/MIPS U-Boot bootstrap loader, Revision 1.7

(builder@shoth.juniper.net, Tue May  4 07:15:51 UTC 2010)
Memory: 1024MB
[0]Booting from nand-flash slice 1
Un-Protected 1 sectors
writing to flash...
Protected 1 sectors
Loading /boot/defaults/loader.conf 
/kernel data=0xb0567c+0x134494 syms=[0x4+0x8aa50+0x4+0xc8fc6]


Hit [Enter] to boot immediately, or space bar for command prompt.

Booting [/kernel]...               
Kernel entry at 0x801000e0 ...
init regular console
Primary ICache: Sets 64 Size 128 Asso 4
Primary DCache: Sets 1 Size 128 Asso 64
Secondary DCache: Sets 512 Size 128 Asso 8
GDB: debug ports: uart
GDB: current port: uart
KDB: debugger backends: ddb gdb
KDB: current backend: ddb
kld_map_v: 0x8ff80000, kld_map_p: 0x0
Copyright (c) 1996-2014, Juniper Networks, Inc.
All rights reserved.
Copyright (c) 1992-2006 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
JUNOS 12.1X44-D40.2 #0: 2014-08-28 12:20:14 UTC
    builder@alaranth.juniper.net:/volume/build/junos/12.1/service/12.1X44-D40.2/obj-octeon/junos/bsd/kernels/JSRXNLE/kernel
JUNOS 12.1X44-D40.2 #0: 2014-08-28 12:20:14 UTC
    builder@alaranth.juniper.net:/volume/build/junos/12.1/service/12.1X44-D40.2/obj-octeon/junos/bsd/kernels/JSRXNLE/kernel
real memory  = 1073741824 (1024MB)
avail memory = 526438400 (502MB)
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
Security policy loaded: JUNOS MAC/pcap (mac_pcap)
Security policy loaded: JUNOS MAC/runasnonroot (mac_runasnonroot)
netisr_init: !debug_mpsafenet, forcing maxthreads from 4 to 1
cpu0 on motherboard
: CAVIUM's OCTEON 52XX CPU Rev. 0.8 with no FPU implemented
        L1 Cache: I size 32kb(128 line), D size 8kb(128 line), sixty four way.
        L2 Cache: Size 512kb, 8 way
obio0 on motherboard
uart0: <Octeon-16550 channel 0> on obio0
uart0: console (9600,n,8,1)
twsi0 on obio0
dwc0: <Synopsis DWC OTG Controller Driver> on obio0
usb0: <USB Bus for DWC OTG Controller> on dwc0
usb0: USB revision 2.0
uhub0: vendor 0x0000 DWC OTG root hub, class 9/0, rev 2.00/1.00, addr 1
uhub0: 1 port with 1 removable, self powered
uhub1: vendor 0x0409 product 0x005a, class 9/0, rev 2.00/1.00, addr 2
uhub1: single transaction translator
uhub1: 3 ports with 2 removable, self powered
umass0: STMicroelectronics ST72682  High Speed Mode, rev 2.00/2.10, addr 3
dwc1: <Synopsis DWC OTG Controller Driver> on obio0
usb1: <USB Bus for DWC OTG Controller> on dwc1
usb1: USB revision 2.0
uhub2: vendor 0x0000 DWC OTG root hub, class 9/0, rev 2.00/1.00, addr 1
uhub2: 1 port with 1 removable, self powered
cpld0 on obio0
pcib1: <Cavium on-chip PCIe HOST bridge> on obio0
Disabling Octeon big bar support
PCIe: Waiting for port 0 to finish reset
PCIe: Port 0 link active, 2 lanes
PCIe: Waiting for port 1 to finish reset
PCIe: Port 1 link active, 1 lanes
pcib1: Initialized controller
pci0: <PCI bus> on pcib1
pcib2: <PCI-PCI bridge> irq 0 at device 0.0 on pci0
pci1: <PCI bus> on pcib2
pci1: <serial bus, USB> at device 2.0 (no driver attached)
pci1: <network> at device 7.0 (no driver attached)
pcib0: <Cavium on-chip PCIe HOST bridge> on obio0
pci2: <PCI bus> on pcib0
pci2: <processor> at device 0.0 (no driver attached)
gblmem0 on obio0
octpkt0: <Octeon RGMII> on obio0
cfi0: <AMD/Fujitsu - 4MB> on obio0
Timecounter "mips" frequency 600000000 Hz quality 0
###PCB Group initialized for udppcbgroup
###PCB Group initialized for tcppcbgroup
da0 at umass-sim0 bus 0 target 0 lun 0
da0: <ST ST72682 2.10> Removable Direct Access SCSI-2 device 
da0: 40.000MB/s transfers
da0: 1000MB (2048000 512 byte sectors: 64H 32S/T 1000C)
Trying to mount root from ufs:/dev/da0s1a
Attaching /cf/packages/junos via /dev/mdctl...
Mounted junos package on /dev/md0...

Media check on da0

Zone 04 Block 0499 Addr 11f300 : Bad read
Recovering Block
Automatic reboot in progress...
** /dev/da0s1a
** Last Mounted on /
** Root file system
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
** Phase 5 - Check Cyl groups
250 files, 75946 used, 73580 free (28 frags, 9194 blocks, 0.0% fragmentation)
Verified junos signed by PackageProduction_12_1_0
Verified jboot signed by PackageProduction_12_1_0
Verified junos-12.1X44-D40.2-domestic signed by PackageProduction_12_1_0
Checking integrity of BSD labels:
  s1: Passed
  s2: Passed
  s3: Passed
  s4: Passed
** /dev/bo0s3e
** Last Mounted on /config
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
** Phase 5 - Check Cyl groups
28 files, 52 used, 12386 free (10 frags, 1547 blocks, 0.1% fragmentation)
** /dev/bo0s3f
** Last Mounted on /cf/var
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
** Phase 5 - Check Cyl groups
616 files, 98342 used, 76976 free (544 frags, 9554 blocks, 0.3% fragmentation)
Checking integrity of licenses:
  JUNOS137657.lic: Passed
  JUNOS187398.lic: Passed
  JUNOS187665.lic: Passed
  JUNOS628672.lic: Passed
Checking integrity of configuration:
  rescue.conf.gz: Passed
Loading configuration ...
mgd: error: Cannot open configuration file: /config/juniper.conf
mgd: warning: loading configuration from /config/rescue.conf.gz
Time and ticks drifted too much,             resetting synchronization...
mgd: commit complete
Setting initial options: .
Starting optional daemons:  usbd.
Doing initial network setup:.
Initial interface configuration:
additional daemons: eventd.
Additional routing options:kern.module_path: /boot//kernel;/boot/modules -> /boot/modules;/modules/ifpfe_drv;/modules;
kld netpfe drv: ifpfed_dialer.
Doing additional network setup:.
Starting final network daemons:.
setting ldconfig path: /usr/lib /opt/lib
starting standard daemons: cron.
Initial rc.mips initialization:.
Local package initialization:.
starting local daemons:set cores for group access
.
kern.securelevel: -1 -> 1
Creating JAIL MFS partition...
JAIL MFS partition created
boot.upgrade.uboot="0xBFC00000"
boot.upgrade.loader="0xBFE00000"
Boot media /dev/da0 has dual root support
WARNING: JUNOS versions running on dual partitions are not same
** /dev/da0s2a
** Last Mounted on /mfs/tmp/snap-tmp.1334/mnt.1334
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
** Phase 5 - Check Cyl groups
250 files, 75914 used, 74124 free (28 frags, 9262 blocks, 0.0% fragmentation)
Sun Sep 20 17:48:34 UTC 2015

fw-srx-2 (ttyu0)


login: 

For somehow, the regular configuration could not be loaded, and system used rescue configuration instead.

Fix is quite simple. Made a little change to configuration and commit it to generate a new configuration. Reboot and this time console showed the regular configuration loaded successfully:



Checking integrity of licenses:

  JUNOS137657.lic: Passed
  JUNOS187398.lic: Passed
  JUNOS187665.lic: Passed
  JUNOS628672.lic: Passed
Checking integrity of configuration:
  rescue.conf.gz: Passed
Loading configuration ...
mgd: commit complete
Setting initial options: .