Memorise

JunOS: Cleanup Storage Space

Sometimes you will want to install a switch or router update, and you will find that there is not enough space:

root@Switch01> request system software add /var/tmp/ex-2300-18.3R1.9.tgz reboot
ERROR: estimate of space required: 115 Mbytes, available: 89 Mbytes

One option is to request a ‘cleanup’. The dry-run option below lists the files that are candidates to be removed. If you’re happy with the list, run the command again without ‘dry-run’ to do the actual cleanup.

root@Switch01> request system storage cleanup dry-run
fpc0:
--------------------------------------------------------------------------

List of files to delete:

         Size Date         Name
     6B Jan  1 13:07 /var/jail/tmp/alarmd.ts
  7416B Jan  1 14:01 /var/log/interactive-commands.0.gz
  25.1K Jan  1 14:01 /var/log/messages.0.gz
    27B Jan  1 10:03 /var/log/wtmp.0.gz
    27B Jan  1 10:06 /var/log/wtmp.1.gz
    45B Jan  1 10:05 /var/preserve/jdhcp_client_data
    45B Jan  1 10:05 /var/preserve/jdhcp_client_data_bkp
    50B Jan  1 10:36 /var/tmp/bcast.bdisp.log
    73B Jan  1 10:36 /var/tmp/bcast.disp.log
    57B Jan  1 10:36 /var/tmp/bcast.rstdisp.log
    64B Jan  1 10:36 /var/tmp/bcast.undisp.log
 321.4M Jan  1 13:44 /var/tmp/ex-2300-18.3R1.9.tgz
  4740B Jan  1 10:04 /var/tmp/ex_autod_config
  3701B Jan  1 10:03 /var/tmp/ex_autod_rollback_cfg
6298.8K Jan  1 13:44 /var/tmp/jweb-ex-app-x86-32-18.3A1.tgz
    57B Jan  1 10:03 /var/tmp/krt_rpf_filter.txt
    72B Jan  1 13:53 /var/tmp/package.log
    42B Jan  1 10:05 /var/tmp/pfe_debug_commands
     0B Jan  1 10:06 /var/tmp/pkg_cleanup.log.err
     0B Jan  1 10:03 /var/tmp/rtsdb/if-rtsdb
     0B Jan  1 10:04 /var/tmp/stable

WARNING: This cleanup cleans out the /var/tmp directory, which may contain the image that you’re trying to install.

Cleaning up Packages

Sometimes a regular cleanup will not free up enough space, especially after the system has been updated.

In this case, we can look at cleaning up unused packages:

User@Switch01> start shell user root
root@Switch01:RE:0% pkg setop rm previous
root@Switch01:RE:0% pkg delete old

If you run df -h before and after these commands, you can see how much was cleaned up.

Further Cleanup

There may be packages installed that you don’t need. For example, you may not need jweb and phone-home. If you don’t need these, you can uninstall them:

request system software delete jweb-ex
request system software delete jweb-ex-app 
request system software delete jphone-home

If you still don’t have enough space, it’s time to look for bigger files:

User@Switch01> start shell user root
root@Switch01:RE:0% find / -size +100000
/var/rundb/render.db
/packages/db/junos-runtime-arm-32-20180920.185504_builder_junos_183_r1/contents/contents.izo
/packages/mnt/jpfe-EX34XX32-cc3f6403/usr/sbin/fxpc

In the case above, we found three large files. If you know what you’re doing, you can delete some of these files.

If you’re not sure, contact J-TAC for assistance.

root@AWABA-NET-SW-AM01:RE:0% cli
User@Switch01> file delete /packages/db/junos-runtime-arm-32-20180920.185504_builder_junos_183_r1/contents/contents.izo

How to delete Service in Windows Server 2012

Syntax

Copy

sc [<ServerName>] delete [<ServiceName>]

Parameters

ParameterDescription
<ServerName>Specifies the name of the remote server on which the service is located. The name must use the Universal Naming Convention (UNC) format (for example, \\myserver). To run SC.exe locally, omit this parameter.
<ServiceName>Specifies the service name returned by the getkeyname operation.
?Displays help at the command prompt.

Remarks

Use Add or Remove Programs on Control Panel to delete DHCP, DNS, or any other built-in operating system services. Note that Add or Remove Programs will not only remove the registry subkey for the service, but it will also uninstall the service and delete any shortcuts to it.

Examples

To delete the service subkey NewServ from the registry on the local computer, type:Copy

sc delete newserv

Source: https://docs.microsoft.com/en-us/previous-versions/windows/it-pro/windows-server-2012-r2-and-2012/cc742045(v=ws.11)


Troubleshooting a Site to Site VPN on a SRX

1. Confirm Configuration

First of all check the VPN configuration. This is also useful if and when you need to confirm the Phase 1 and Phase 2 parameter’s with the remote end.

admin@srx> show configuration security ike
admin@srx> show configuration security ipsec

{loadposition content_lock}

2. Confirm Phase 1

To confirm the successful completion of Phase 1 run the following command. If Phase 1 fails to complete revisit your Phase 1 parameters using the commands shown in Section 1.

admin@srx> show security ike security-associations
node1:
————————————————————————–
Index   Remote Address  State  Initiator cookie  Responder cookie  Mode
6950    [LOCAL PEER IP]  UP     33204fba87663d94  70acacd5f938f89b  Main

3. Confirm Phase 2

To confirm the successful completion of Phase 2 run the following command. If Phase 2 fails to complete revisist your Phase 2 parameters using the commands shown in Section 1.

admin@srx> show security ipsec security-associations
node1:
————————————————————————–
Total active tunnels: 2
ID    Gateway          Port  Algorithm       SPI      Life:sec/kb  Mon vsys
<131073 [LOCAL PEER IP] 500   ESP:aes-128/sha1 4fb2c1cc 2041/ unlim  –   root
>131073 [LOCAL PEER IP] 500   ESP:aes-128/sha1 3e576ead 2041/ unlim  –   root

If Phase 2 has completed you can confirm further details on each of the SA`s (Security Associations) by using the SA index.

admin@srx> show security ipsec security-associations index 131073
node1:
————————————————————————–
Virtual-system: root
Local Gateway: [REMOTE PEER IP], Remote Gateway: [LOCAL PEER IP]
Local Identity: ipv4_subnet(any:0,[0..7]=0.0.0.0/0)
Remote Identity: ipv4_subnet(any:0,[0..7]=0.0.0.0/0)
DF-bit: clear
Direction: inbound, SPI: 4fb2c1cc, AUX-SPI: 0
, VPN Monitoring: –
Hard lifetime: Expires in 2028 seconds
Lifesize Remaining:  Unlimited
Soft lifetime: Expires in 1448 seconds
Mode: tunnel, Type: dynamic, State: installed
Protocol: ESP, Authentication: hmac-sha1-96, Encryption: aes-cbc (128 bits)
Anti-replay service: counter-based enabled, Replay window size: 64

    Direction: outbound, SPI: 3e576ead, AUX-SPI: 0
, VPN Monitoring: –
Hard lifetime: Expires in 2028 seconds
Lifesize Remaining:  Unlimited
Soft lifetime: Expires in 1448 seconds
Mode: tunnel, Type: dynamic, State: installed
Protocol: ESP, Authentication: hmac-sha1-96, Encryption: aes-cbc (128 bits)
Anti-replay service: counter-based enabled, Replay window size: 64

4. IPSEC Statistics

To confirm statistics based on the Phase 2 SA run the following command. The output will contain a number of counters. The most interesting of these (for troubleshooting purposes) are the Encrypted and Decrypted counters.

admin@srx> show security ipsec statistics index 131073
node1:
————————————————————————–
ESP Statistics:
Encrypted bytes:        133593600
Decrypted bytes:       1128704777
Encrypted packets:         923864
Decrypted packets:        1438716
AH Statistics:
Input bytes:                    0
Output bytes:                   0
Input packets:                  0
Output packets:                 0
Errors:
AH authentication failures: 0, Replay errors: 1021
ESP authentication failures: 0, ESP decryption failures: 0
Bad headers: 0, Bad trailers: 0

5. Perform Debug (Traffic)

If Phase 1 and Phase 2 are both establishing but traffic is still not passing the VPN tunnel, a packet-filter traffic debug of the tunnel will provide further granularity into each of the steps the packet takes.

admin@srx> configuration
admin@srx# edit security flow traceoptions

[edit security flow traceoptions]
admin@srx# set file vpn-debug
admin@srx# set flag basic-datapath
admin@srx# set flag packet-drops
admin@srx# set level 15

admin@srx# set packet-filter filter1 source-prefix [LOCAL PEER IP]
admin@srx# set packet-filter filter1 destination-prefix [REMOTE PEER IP]
admin@srx# set packet-filter filter1 protocol esp
admin@srx# set packet-filter filter2 destination-prefix [LOCAL PEER IP]
admin@srx# set packet-filter filter2 source-prefix [REMOTE PEER IP]
admin@srx# set packet-filter filter2 protocol esp

admin@srx# set packet-filter filter3 destination-prefix [INTERNAL SERVER IP]
admin@srx# set packet-filter filter3 destination-port ssh
admin@srx# set packet-filter filter3 protocol tcp
admin@srx# set packet-filter filter4 source-prefix [INTERNAL SERVER IP]
admin@srx# set packet-filter filter4 destination-port ssh
admin@srx# set packet-filter filter4 protocol tcp

admin@srx# run show log vpn-debug

6. Perform Debug (Crypto)

To debug the crypto engine the following commands are run.

admin@srx> configuration
admin@srx# edit security ike traceoptions

[edit security ike traceoptions]
admin@srx# set file vpn-debug-ike
admin@srx# set flag all
admin@srx# set level 15
admin@srx# top

[edit]
admin@srx# edit security ipsec traceoptions

[edit security ipsec traceoptions]
admin@srx# set file vpn-debug-ipsec
admin@srx# set flag all
admin@srx# set level 15

admin@srx# run show log vpn-debug-ike
admin@srx# run show log vpn-debug-ipsec

7. Additional

A useful tip when viewing the debug logs is to tail the file via the shell whilst also removing the empty lines. This a) makes it easier to view and 2) also (as long as your ssh client buffer is configured correctly) allows you to go back over previous output should the debug log reach its maximum size.

root@srx100> start shell
root@srx100% tail -f /var/log/[logfile] | grep -Evi ^$


Force Active Directory replication on a domain controller

To force Active Directory replication run the command ‘repadmin /syncall /AeD’ on the domain controller.  Run this command on the domain controller in which you wish to update the Active Directory database.  For example, if DC2 is out of Sync, run the command on DC2.

A = All Partitions
e = Enterprise (Cross Site)
D = Identify servers by distinguished name in messages.

By default, this does a pull replication – which is how AD works by default.  If you want to do a push replication use the following command:

repadmin /syncall /APeD

P = Push

You want to do a push replication if you make changes on a DC and you want to replicate those changes to all other DC’s.  For example, you make a change on DC1 and you want all other changes to get that change instantly, run repadmin /syncall /APeD on DC1.

For all repadmin syntax please see:

http://technet.microsoft.com/en-us/library/cc736571(v=ws.10).aspx


Juniper SRX – Configuring BT FTTP PPPoE

This configuration is set up on Juniper SRX 340 running JUNOS 20.2R1.10

Note: The username is the same for everyone
btbusinesshub@business.btclick.com
password is anything
chap authentication method
outside/untrust interface being ge-0/0/7.0


set interfaces ge-0/0/7 unit 0 encapsulation ppp-over-ether

–Optional —
set security zones security-zone Internet interfaces pp0.0 host-inbound-traffic system-services ping
set security zones security-zone Internet interfaces pp0.0 host-inbound-traffic system-services ssh
set interfaces pp0 traceoptions flag all
set interfaces pp0 unit 0 bandwidth 900m
–Optional —
set interfaces pp0 unit 0 ppp-options chap default-chap-secret “$9$kmPTn/A”
set interfaces pp0 unit 0 ppp-options chap local-name “btbusinesshub@business.btclick.com”
set interfaces pp0 unit 0 ppp-options chap passive
set interfaces pp0 unit 0 pppoe-options underlying-interface ge-0/0/7.0
set interfaces pp0 unit 0 pppoe-options idle-timeout 0
set interfaces pp0 unit 0 pppoe-options auto-reconnect 1
set interfaces pp0 unit 0 pppoe-options client
set interfaces pp0 unit 0 no-keepalives
set interfaces pp0 unit 0 family inet mtu 1492
set interfaces pp0 unit 0 family inet negotiate-address

Troubleshooting

show ppp statistics
show pppoe statistics

show interfaces pp0
Check for
LCP state: Opened
NCP state: inet: Opened, inet6: Not-configured, iso: Not-configured, mpls: Not-configured
CHAP state: Success

Cabling guide:

Plug the RJ45 cable direct from the Openreach socket to ge-0/0/7


Juniper SRX certificate ‘aamw-srx-cert’: certificate does not exist

error: certificate ‘aamw-srx-cert’: certificate does not exist .
error: trusted-ca ‘aamw-cloud-ca’ does not exist!
error: trusted-ca ‘aamw-secintel-ca’ does not exist!

Error:

{primary:node0}[edit]
root# commit and-quit
[edit security pki]
‘ca-profile aamw-secintel-ca’
Missing mandatory statement: ‘ca-identity’
[edit security pki]
‘ca-profile aamw-cloud-ca’
Missing mandatory statement: ‘ca-identity’
error: commit failed: (missing mandatory statements)

FIX:

{primary:node0}[edit]
root# delete security pki

{primary:node0}[edit]
root# commit and-quit
warning: You have changed enhanced services mode.
You must reboot the system for your change to take effect.
If you have deployed a cluster, be sure to reboot all nodes.
node0:
commit complete
Exiting configuration mode

Once joined to the cluster sync this with working SRX that will update all the cert.

Location of Certificates

The certificates/key-pairs used for IKE negotiations are stored in following locations,

/var/db/certs/common/key-pair
/var/db/certs/common/local
/var/db/certs/common/certification-authority

If the cert is missing, use WinSCP to copy the /var/db/certs folder.


Aruba Resetting Admin Password

Resetting Admin Password

This section describes how to reset the password for the default administrator user account (admin) on the managed device. Use this procedure if the administrator user account password is lost or forgotten.

1. Connect a local console to the serial port on the managed device.

2. From the console, login into the managed device as a password recovery user. For information, read Password Recovery user.

3. Enter configuration mode by typing in configure terminal.

4. To reset the administrator user account password, use the mgmt-user admin root command.

5. Enter a new password for this account and retype the same to confirm.

6. Exit from the configuration mode and the user mode.

If you have defined a management user password policy, make sure that the new password conforms to this policy. For details, see Implementing Specific Management Password Policy.

The following is an example of how to reset the admin password as a default password recovery user. If you have configured an alternate password recovery user, use its credentials to login to the controller. The commands in bold type are what you enter:

User: password

Password: forgetme!

(host) #configure terminal

Enter Configuration commands, one per line. End with CNTL/Z

(host) (config) #mgmt-user admin root

Password:********

Re-Type password:********

(host) (config) #exit

(host) #exit

Password Recovery user

A password recovery user is a management user with root rights that is used to reset the admin password in the event of a lost or forgotten password. Starting with ArubaOS 8.4.0.0, a configurable alternate password recovery user can be created in addition to the default password recovery feature.

 Password recovery access using either the default password recovery user or the alternate password recovery user is allowed only through the serial console of a controller.
 Password recovery users can be configured only through SSH sessions and serial console sessions with a controller and not through WebUI.
 Aruba recommends to enable the default password recovery user before generating and sharing the tech-support logs or configuration files with customer support.
 It is recommended that either the default password recovery user is disabled or the alternate password recovery user is configured when setting up the network to ensure. This is to ensure that there are no vulnerabilities.

Default password recovery user

In the event of a lost/forgotten password, the administrator can login to the controller and reset the admin password as the default password recovery user using the username password and the password forgetme!. The default password recovery user is defined and is enabled by default . Disabling the Default password recovery user is recommended if the network uses a TACACS server to authenticate its management users.

To disable the default password recovery user, execute the following command in the configuration mode:

(host) (config) #password-recovery-disable

To enable the default password recovery user, execute the following command in the configuration mode:

(host) (config) #no password-recovery-disable

Alternate password recovery user

Starting with ArubaOS 8.4.0.0, an alternate password recovery user with a username and password can be created to reset the admin password. The alternate user’s username can be 16 characters long and the password can be 32 characters long. Configuring the alternate password recovery user automatically disables the default password recovery user. Configuring the alternate password recovery user is highly recommended if the network is managed locally.

 The alternate password recovery user will not be shown in the management user section of the WebUI. This user role cannot be configured through the WebUI.

To configure the alternate password recovery user, execute the following command in the configuration mode:

(host) (config) #password-recovery-user <username>

Password:******

Re-Type password:******

To disable the alternate password recovery user, execute the following command in the configuration mode:

(host) (config) #no password-recovery-user

The following is an example to configure the alternate password recovery user:

(host) #configure terminal

Enter Configuration commands, one per line. End with CNTL/Z

(host) (config) #password-recovery-user recadmin

Password:******

Re-Type password:******

(host) (config) #exit

Use the show mgmt-user command to view the configured management users and the status of the default password recovery user.

The following is an example of the show mgmt-user command with the default password recovery user enabled.

(host) #show mgmt-user

Default password recovery user: Enabled

Management User Table

———————

USER PASSWD ROLE STATUS

—- —— —- ——

admin ***** root ACTIVE

The following is an example of the show mgmt-user command when the alternate password recovery user is configured.

(host) #show mgmt-user

Default password recovery user: Disabled

Management User Table

———————

USER PASSWD ROLE STATUS

—- —— —- ——

admin ***** root ACTIVE

recadmin ***** passR ACTIVE

source: https://www.arubanetworks.com/techdocs/ArubaOS_83_Web_Help/Content/ArubaFrameStyles/Management_Utilities/enab_radsec_reset_admin_enabl_pwd.htm


Juniper SRX Stuck in loader prompt

Insert a USB on a working SRX, then copy the partition

Insert the USB on the broken SRX and boot from USB

loader> nextboot
Platform: srx-trident
eUSB
usb
loader> nextboot usb
Setting next boot dev usb
loader> reboot
Resetting…

To do so, use the command below: request system snapshot media internal slice alternate
The slice seems to be a hidden command; therefore, you would have to type it in manually.

Recovering the Junos image in primary partition

When you spot that a primary partition has failed you should try to recover it as soon as possible as you are left with only one root partition. The recovery of the primary partition can be done easily by taking a snapshot of the root file system in the secondary partition and copying it to the primary partition.

The following command takes a snapshot of the currently active partition (secondary partition) and copies it to the alternate partition (primary partition).

root@SRX345>request system snapshot slice alternate
Formatting alternate root (//dev//da0s1a)...
Copying '//dev//da0s2a' to '//dev//da0s1a' .. (this may take a few minutes)
The following filesystems were archived: //

Run the following command to verify that you have a valid backup image.

root@SRX> show system software backup
Backup JUNOS package information:
File name: //cf//packages//junos-15.1X49-D150.2-domestic
File size: 254838138

You can also use the show system storage partitions command to check both partitions.

https://kb.juniper.net/InfoCenter/index?page=content&id=KB20554

Working output – shorten version

loader> ?
Available commands:
watchdog enable or disable kernel watchdog
bcachestat get disk block cache stats
autoboot boot automatically after a delay
boot boot a file or loaded kernel
lsdev list all devices
nextboot set next boot device
more show contents of a file
read read input from the terminal
echo echo arguments
unset unset a variable
set set a variable
show show variable(s)
? list commands
help detailed help
install install JunOS
include read commands from a file
ls list files
lsmod list loaded modules
unload unload all modules
load load a kernel or module
reboot reboot the system
heap show heap usage
save save U-Boot environment
export export variables to U-Boot environment
loader> boot
can’t load ‘/kernel’
can’t load ‘/kernel.old’
no bootable kernel
loader> nextboot
Platform: srx-trident
eUSB
usb
loader> nextboot usb
Setting next boot dev usb
loader> reboot
Resetting…

SPI stage 1 bootloader (Build time: Apr 26 2020 – 21:42:44)

U-Boot 2013.07-JNPR-3.9 (Build time: Apr 26 2020 – 21:42:45)

Octeon unique ID: 040000708015f31e0245
…..

PCIe: Port 2 not in PCIe mode, skipping
Net: octrgmii0
Node 0 Interface 4 has 1 ports (AGL)
Boot Media: eUSB usb
Found TPM SLB9660 TT 1.2 by Infineon
TPM initialized
USB1: Starting the controller
USB XHCI 1.00
scanning bus 1 for devices… 2 USB Device(s) found
USB0: Starting the controller
USB XHCI 1.00
scanning bus 0 for devices… 2 USB Device(s) found
scanning usb for storage devices… 2 Storage Device(s) found
Type the command ‘usb start’ to scan for USB storage devices.

Press SPACE to stop autoboot: 0
SF: Detected SF with page size 256 Bytes, erase size 64 KiB, total 8 MiB
SF: 1048512 bytes Read: OK
SF: 1048576 bytes Read: OK

Starting application …

SF: Detected SF with page size 256 Bytes, erase size 64 KiB, total 8 MiB
[0]Booting from usb slice 1
Consoles: U-Boot console
Found compatible API, ver. 3.9

FreeBSD/MIPS U-Boot bootstrap loader, Revision 2.10
(slt-builder@svl-junos-pool87.juniper.net, Sun Mar 4 10:30:52 PST 2018)
Memory: 4096MB
[0]Booting from usb slice 1
/boot/init.4th loaded.
Loading /boot/defaults/loader.conf
/kernel data=0x126bb74+0x1c04e4 syms=[0x4+0xba2c0+0x4+0x11d559]

Hit [Enter] to boot immediately, or space bar for command prompt.
Booting [/kernel]…
Kernel entry at 0x801000c0 …
init regular console
Primary ICache: Sets 16 Size 128 Asso 39
Primary DCache: Sets 8 Size 128 Asso 32
Secondary DCache: Sets 1024 Size 128 Asso 4

Timecounter “mips” frequency 1200000000 Hz quality 0
da1 at umass-sim1 bus 1 target 0 lun 0
da1: Removable Direct Access SCSI-4 device
da1: 40.000MB/s transfers
da1: 7680MB (15728640 512 byte sectors: 255H 63S/T 979C)
da0 at umass-sim0 bus 0 target 0 lun 0
da0: Fixed Direct Access SCSI-4 device
da0: 40.000MB/s transfers
da0: 7672MB (15712256 512 byte sectors: 255H 63S/T 978C)
random: unblocking device.
hwpmc: OCTEON/4/64/0x1ff
Trying to mount root from ufs:/dev/da1s1a
MFSINIT: Initialising MFSROOT
Process-1 beginning MFSROOT initialization…
Creating MFSROOT…
/dev/md0: 20.0MB (40956 sectors) block size 16384, fragment size 2048
using 4 cylinder groups of 5.00MB, 320 blks, 640 inodes.
super-block backups (for fsck -b #) at:
32, 10272, 20512, 30752
Populating MFSROOT…
Creating symlinks…
Setting up mounts…
Continuing boot from MFSROOT…
Attaching /cf/packages/junos via /dev/mdctl…
Mounted junos package on /dev/md1…
J
Automatic reboot in progress…
Verified jboot signed by PackageProductionECP256_2020 method ECDSA256+SHA256
Verified junos signed by PackageProductionECP256_2020 method ECDSA256+SHA256
Verified junos-20.2R1.10 signed by PackageProductionECP256_2020 method ECDSA256+ SHA256
Checking integrity of BSD labels:
s1: Passed
s2: Passed
s3: Passed
s4: Passed
** /dev/bo0s3e
FILE SYSTEM CLEAN; SKIPPING CHECKS
clean, 94741 free (21 frags, 11840 blocks, 0.0% fragmentation)
** /dev/bo0s3f
FILE SYSTEM CLEAN; SKIPPING CHECKS
clean, 1746346 free (386 frags, 218245 blocks, 0.0% fragmentation)
Checking integrity of licenses:
Checking integrity of configuration:
rescue.conf.gz: Passed

LPC bus driver
lpcbus0 on cpld0
tpm0: on lpcbus0
tpm: IFX SLB 9660 TT 1.2 rev 0x10
Loading configuration …
.
..
Additional routing options:kern.module_path: /boot//kernel;/boot/modules -> /boo t/modules;/modules/ifpfe_drv;/modules;
kld netpfe drv: ifpfed_dialer pvid_cryptosoft0: on motherboard
IPsec: Initialized Security Association Processing.
db kld ipsecs
.
Doing additional network setup:.
Starting final network daemons:.
setting ldconfig path: /usr/lib /opt/lib
starting standard daemons: cron.
root@SEDG-ABC-SRX01% alization:.
root@SEDG-ABC-SRX01% ization:.
root@SEDG-ABC-SRX01% s:set cores for group access
root@SEDG-ABC-SRX01%

root@SEDG-ABC-SRX01% clear
FILE SYSTEM CLEAN; SKIPPING CHECKS
clean, 98068 free (28 frags, 12255 blocks, 0.0% fragmentation)
chassis.ko loaded Loading JUNOS chassis module
chassis_init_hw_chassis_startup_time: chassis startup time 0.000000
Thu Jan 13 17:16:25 GMT 2022

SEDG-ABC-SRX01 (ttyu0)

login: root
Password:

— JUNOS 20.2R1.10 built 2020-06-25 13:55:10 UTC

— NOTICE: System is running on alternate media device (/dev/da1s1a).

root@SEDG-ABC-SRX01%
root@SEDG-ABC-SRX01%
root@SEDG-ABC-SRX01% cli
{secondary:node0}
root@SEDG-ABC-SRX01> show system snapshot media internal

node0:

Information for snapshot on internal (/dev/da0s1a) (backup)
Creation date: Aug 27 17:03:31 2020
JUNOS version on snapshot:
junos : 20.2R1.10

node1:

Information for snapshot on internal (/dev/da0s1a) (primary)
Creation date: Nov 27 18:07:49 2021
JUNOS version on snapshot:
junos : 20.2R1.10
Information for snapshot on internal (/dev/da0s2a) (backup)
Creation date: Aug 27 17:10:28 2020
JUNOS version on snapshot:
junos : 20.2R1.10

root@SEDG-ABC-SRX01> … media internal slice alternate

node0:

error: Snapshot to alternate slice cannot be performed as internal is not the boot media

node1:

Formatting alternate root (/dev/da0s2a)…
Copying ‘/dev/da0s1a’ to ‘/dev/da0s2a’ .. (this may take a few minutes)
The following filesystems were ABChived: /

{secondary:node0}
root@SEDG-ABC-SRX01>

{secondary:node0}
root@SEDG-ABC-SRX01> show system snapshot media internal

node0:

Information for snapshot on internal (/dev/da0s1a) (backup)
Creation date: Aug 27 17:03:31 2020
JUNOS version on snapshot:
junos : 20.2R1.10

node1:

Information for snapshot on internal (/dev/da0s1a) (primary)
Creation date: Nov 27 18:07:49 2021
JUNOS version on snapshot:
junos : 20.2R1.10
Information for snapshot on internal (/dev/da0s2a) (backup)
Creation date: Jan 13 17:27:02 2022
JUNOS version on snapshot:
junos : 20.2R1.10

{secondary:node0}
root@SEDG-ABC-SRX01>

{secondary:node0}
root@SEDG-ABC-SRX01> show chassis cluster status
Monitor Failure codes:
CS Cold Sync monitoring FL Fabric Connection monitoring
GR GRES monitoring HW Hardware monitoring
IF Interface monitoring IP IP monitoring
LB Loopback monitoring MB Mbuf monitoring
NH Nexthop monitoring NP NPC monitoring
SP SPU monitoring SM Schedule monitoring
CF Config Sync monitoring RE Relinquish monitoring
IS IRQ storm

Cluster ID: 1
Node Priority Status Preempt Manual Monitor-failures

Redundancy group: 0 , Failover count: 0
node0 100 secondary no no None
node1 50 primary no no None

Redundancy group: 1 , Failover count: 0
node0 100 secondary no no None
node1 50 primary no no None

{secondary:node0}
root@SEDG-ABC-SRX01> show system snapshot media al?
No valid completions
{secondary:node0}
root@SEDG-ABC-SRX01> show system snapshot media ?
Possible completions:
internal Show snapshot information from internal flash
usb Show snapshot information from device connected to USB port

{secondary:node0}
root@SEDG-ABC-SRX01> show system snapshot media internal

node0:

Information for snapshot on internal (/dev/da0s1a) (backup)
Creation date: Aug 27 17:03:31 2020
JUNOS version on snapshot:
junos : 20.2R1.10

node1:

Information for snapshot on internal (/dev/da0s1a) (primary)
Creation date: Nov 27 18:07:49 2021
JUNOS version on snapshot:
junos : 20.2R1.10
Information for snapshot on internal (/dev/da0s2a) (backup)
Creation date: Jan 13 17:27:02 2022
JUNOS version on snapshot:
junos : 20.2R1.10
root@SEDG-ABC-SRX01% unmount /altroot
unmount: Command not found.
root@SEDG-ABC-SRX01% unmount /altroot
unmount: Command not found.
root@SEDG-ABC-SRX01% exitcal, noatime, read-only)
logouton /dev (devfs, local, multilabel)
root@SEDG-ABC-SRX01% exitlocal, noatime)
exitpackages on /junos/cf/packages (nullfs, local, noatime)
devfs on /junos/cf/dev (devfs, local, noatime, multilabel)
{secondary:node0}s (cd9660, local, noatime, read-only, verified)
root@SEDG-ABC-SRX01> lfs, local, noatime)
devfs on /junos/dev/ (devfs, local, noatime, noexec, read-only, multilabel)
{secondary:node0}junos/cf/packages1 (nullfs, local, noatime)
root@SEDG-ABC-SRX01> fs, local, noatime)
/dev/bo0s3e on /config (ufs, local, noatime)
{secondary:node0}f/var (ufs, local, noatime)
root@SEDG-ABC-SRX01> , asynchronous, local, noatime)
/cf/var/jail on /jail/var (nullfs, local, noatime)
{secondary:node0}t-api on /web-api/var (nullfs, local, noatime)
root@SEDG-ABC-SRX01> var/log (nullfs, local, noatime)
devfs on /jail/dev (devfs, local, noatime, noexec, read-only, multilabel)
{secondary:node0}/mfs (ufs, asynchronous, local, noatime)
root@SEDG-ABC-SRX01>
root@SEDG-ABC-SRX01% su –
{secondary:node0}
root@SEDG-ABC-SRX01>

{secondary:node0}
root@SEDG-ABC-SRX01>

{secondary:node0}
root@SEDG-ABC-SRX01>

{secondary:node0}
root@SEDG-ABC-SRX01> request system snapshot slice alternate

node0:

Formatting alternate root (/dev/da1s2a)…
Copying ‘/dev/da1s1a’ to ‘/dev/da1s2a’ .. (this may take a few minutes)
The following filesystems were ABChived: /

node1:

Formatting alternate root (/dev/da0s2a)…
Copying ‘/dev/da0s1a’ to ‘/dev/da0s2a’ .. (this may take a few minutes)
The following filesystems were ABChived: /

{secondary:node0}
root@SEDG-ABC-SRX01> show system storage partitions

node0:

Boot Media: usb (da1)
Active Partition: da1s1a
Backup Partition: da1s2a
Currently booted from: active (da1s1a)

Partitions information:
Partition Size Mountpoint
s1a 579M /
s2a 587M altroot
s3e 185M /config
s3f 5.0G /var
s4a 324M recovery
s4b
s4e 15M

node1:

Boot Media: internal (da0)
Active Partition: da0s1a
Backup Partition: da0s2a
Currently booted from: active (da0s1a)

Partitions information:
Partition Size Mountpoint
s1a 2.4G /
s2a 2.4G altroot
s3e 185M /config
s3f 2.1G /var
s4a 224M recovery
s4e 15M

{secondary:node0}
root@SEDG-ABC-SRX01> request system reboot media internal
Reboot the system ? yes,no yes

Shutdown NOW!

[pid 5208]

{secondary:node0}
root@SEDG-ABC-SRX01>
*** FINAL System shutdown message from root@SEDG-ABC-SRX01 ***

System going down IMMEDIATELY

Jan 13 17:59:32 init: interface-control (PID 2371) terminate signal 15 sent
JWaiting (max 60 seconds) for system process vnlru_mem' to stop...done Waiting (max 60 seconds) for system processvnlru’ to stop…done
Waiting (max 60 seconds) for system process bufdaemon' to stop...done Waiting (max 60 seconds) for system processsyncer’ to stop…
Syncing disks, vnodes remaining…0 0 0 done

syncing disks… Syncing disks, buffers remaining… 2 2
Final sync complete
Uptime: 48m19s
Rebooting…

Starting application …

SF: Detected SF with page size 256 Bytes, erase size 64 KiB, total 8 MiB
[0]Booting from usb slice 1
Consoles: U-Boot console
Found compatible API, ver. 3.9

FreeBSD/MIPS U-Boot bootstrap loader, Revision 2.10
(slt-builder@svl-junos-pool87.juniper.net, Sun Mar 4 10:30:52 PST 2018)
Memory: 4096MB
[0]Booting from usb slice 1
/boot/init.4th loaded.
Loading /boot/defaults/loader.conf
/kernel data=0x126bb74+0x1c04e4 syms=[0x4+0xba2c0+0x4+0x11d559]

Hit [Enter] to boot immediately, or space bar for command prompt.
Booting [/kernel]…
Kernel entry at 0x801000c0 …
init regular console
Primary ICache: Sets 16 Size 128 Asso 39
Primary DCache: Sets 8 Size 128 Asso 32
Secondary DCache: Sets 1024 Size 128 Asso 4

Continuing boot from MFSROOT…
Attaching /cf/packages/junos via /dev/mdctl…
Mounted junos package on /dev/md1…
J
Automatic reboot in progress…
Verified jboot signed by PackageProductionECP256_2020 method ECDSA256+SHA256
Verified junos signed by PackageProductionECP256_2020 method ECDSA256+SHA256
Verified junos-20.2R1.10 signed by PackageProductionECP256_2020 method ECDSA256+SHA256
Checking integrity of BSD labels:
s1: Passed
s2: Passed
s3: Passed
s4: Passed
** /dev/bo0s3e
FILE SYSTEM CLEAN; SKIPPING CHECKS
clean, 94741 free (21 frags, 11840 blocks, 0.0% fragmentation)
** /dev/bo0s3f
FILE SYSTEM CLEAN; SKIPPING CHECKS
clean, 1746310 free (462 frags, 218231 blocks, 0.0% fragmentation)
Checking integrity of licenses:
Checking integrity of configuration:
rescue.conf.gz: Passed

Creating JAIL MFS partition…
JAIL MFS partition created
Boot media /dev/da1 has dual root support
** /dev/da1s2a
FILE SYSTEM CLEAN; SKIPPING CHECKS
clean, 98068 free (76 frags, 12249 blocks, 0.0% fragmentation)
chassis.ko loaded Loading JUNOS chassis module
chassis_init_hw_chassis_startup_time: chassis startup time 0.000000
Thu Jan 13 18:04:35 GMT 2022

SEDG-ABC-SRX01 (ttyu0)

login:
SEDG-ABC-SRX01 (ttyu0)


Last Error: Message deferred by categorizer agent.

Exchange 2013 mail flow issue. It turned out to be the Malware agent causing the issue

Once it is enabled and the Exchange transport service is restarted, all emails are stuck in the submission queue with “Last Error: Message deferred by categorizer agent.”

Check the mail queue

Get-Queue -Identity submission

Get-Transport Agent will list all the transport agents, you can disable one at a time to isolate problematic agents. in my case it was the Malware Agent.

Use the Shell to disable malware filtering on a specific Exchange server

To disable malware filtering, run the following command: PowerShellCopy

& $env:ExchangeInstallPath\Scripts\Disable-Antimalwarescanning.ps1

Note

To re-enable malware filtering, use Enable-Antimalwarescanning.ps1 instead of Disable-Antimalwarescanning.ps1.

How do you know this step worked?

To verify that malware filtering is disabled, run the following command and confirm that it returns a value of False: PowerShellCopy

Get-TransportAgent "Malware Agent"

https://docs.microsoft.com/en-us/exchange/disable-or-bypass-anti-malware-scanning-exchange-2013-help