step 1: service snmpd restart
step 2: edit /etc/snmp/snmpd.users.conf and replace public with your actual
snmp community string
step 3: service snmpd restart
step 4: netstat -an | grep 161
for checkpoint snmpd port 260:
step 1: modify the $FWDIR/conf/snmp.C file and place the actual snmp
community inside the read and write (). If you leave the write empty,
it will use "private" as the community string. This is a security risk.
step 2: run sysconfig and start the checkpoint snmpd extension
step 3: perform cpstop;cpstart
step 4: netstat -an | grep 260
Solution The internal_sendmail indicator is not a script. It is an internal Security Gateway indicator that directs the Check Point alerts daemon to send email, using the specified arguments. It does not require a mail server or mail client to be installed on the SmartCenter Server.
If the subject is longer than one word, it must be written within quotation marks.
The internal_sendmail indicator can be used in any of the script options on the 'Alert Commands' configuration page.
When choosing logging actions in rules or other Gateway logging properties, set the action to correspond to the alert you define on the 'Global Properties > Log and Alert > Alert Commands' property screen.
In SmartDashboard:
Select 'Global Properties > Log and Alert > Alert Commands'.
Check 'Send mail alert to SmartView Status'.
Check 'Run mail alert script'.
Enter script:
internal_sendmail [-s ] [-t ] [-f ] ...
Example (default):
internal_sendmail -s alert -t mailserver_ip mail_account
In the Rule Base, define a rule that will generate an alert. In the Track field, enter 'Mail'.
Install the Security Policy.
Read More ...
If the subject is longer than one word, it must be written within quotation marks.
The internal_sendmail indicator can be used in any of the script options on the 'Alert Commands' configuration page.
When choosing logging actions in rules or other Gateway logging properties, set the action to correspond to the alert you define on the 'Global Properties > Log and Alert > Alert Commands' property screen.
In SmartDashboard:
Select 'Global Properties > Log and Alert > Alert Commands'.
Check 'Send mail alert to SmartView Status'.
Check 'Run mail alert script'.
Enter script:
internal_sendmail [-s ] [-t ] [-f ] ...
Example (default):
internal_sendmail -s alert -t mailserver_ip mail_account
In the Rule Base, define a rule that will generate an alert. In the Track field, enter 'Mail'.
Install the Security Policy.
Below are some of the various files and commands which you may find useful on a Checkpoint.
Smart Centre Server
$CPDIR/conf - Contains parts of the CPShared system
* cp.license - license of machine
* sic_cert.p12 - SIC certificate
$FWDIR/lib - .def files which are used when the rulebase is complied into inspection code for Enforcement points.
$FWDIR/conf - the rule base and the rest of the security policy can be found here.
* rulebases_5_0.fws - Contains rulebases and duplicate in *.w files
* objects_5.0.C - Contains all the objects. objects.C is created when sent to the Enforcement Points
$FWDIR/conf/fwauth.* - User Database, main file being fwauth.NDB
$FWDIR/conf/masters - Defines the local log definition in Dashboard
$FWDIR/database/fwauth.* - User Datbase, main file being fwauth.NDB
$FWDIR/log - Logs
Enforcement Point
$CPDIR/conf - Contains parts of the CPShared system
* cp.license - license of machine
* sic_cert.p12 - SIC certificate
$FWDIR/conf/discntd.if - Add interfaces you want to show as disconnected for ClusterXL.
Misc
/etc/sysconfig/netconf.C - Used to configure interface as down, this is useful for ClusterXL when interfaces have no link.
I once SSH login alert presented the way to send mail alert after successful login by ssh to any Linux-based machine , including Checkpoint firewalls. Now, thanks to folks at cpug.orgthat draw my attention to it, I will show how to get mail Alert on ANY rule in the security rulebase of the firewall, and also simplified script using Checkpoint version Of the sendmail.
First , rules alerts – on any rule in the Security Rulebase you can set in its Track column toMail . Now all hits
On such rule will be sending mail alerts to specified recipient(s) through the specified mail server (Checkpoint doesn't have a mail server of its own) . So, if you create rule that allows access by SSH you can set in Track Mail and each time this rule is used to access the firewall mail will be sent. Now how to configure mail server settings, you do it in
Policy -> Global Properties -> Log and Alert -> Alert Commands , check " Send mail alert to SmartviewView Monitor" and "Run mail alert script" . In the "Run mail alert script" field set to the string of form:
internal_sendmail -s [subject of the mail] -t [ip of mail server to receive mail goes here] -f [from_who_field_in_mail] [to_whom_send_this_mail]
e.g. internal_sendmail -s SSH_login_alert -t 63.161.169.140 -f yurisk@yurisk.info president@whitehouse.gov
The mail you get on such alert looks like:
6Jan2010 7:29:55 accept fw-tokyo >External mail rule: 2; rule_uid: {85A905A7-951E-4100-A23A-E280FAAA1D29}; SmartDefense profile: Default_Protection; service_id: ssh; src: my-management-host; dst: fw-tokyo ; proto: tcp; product: VPN-1 & FireWall-1; service: ssh; s_port: 47145; NOTE. Some don'ts
- You can't send to multiple recepients;
- You can't send using IP of the firewall for the mail server
- The mail server you specify should be the one accepting mails for the recepient's address or be doing
mail relay without authentication. And no, Checkpoint sendmail doesn't support authentication.
Deleting IKE/IPsec security associations of established VPNs is inevitable part of any VPN related debug. The standard tool promoted by Checkpoint (take CCSA,CCSE etc.,) is vpn tuthat neveretheless has always had a very annoying bug (feature?) – you can delete ALL VPN tunnels at a time and none individually !! It indeed presents option to delete
" Delete all IPsec SAs for a given peer (GW)" – but it just plain doesn't work. And once confronted with this problem that could make debug more devastating than the problem itself I started looking for alternatives. To much of my surprise CP has a perfect alternative for this
- vpn shell, that provides acceptable means of managing tunnels. Here are details:
vpn shell can : delete IKE/IPsec SAs selectively, add/delete VTI interfaces,show information about all that.
To enter this shell :
[Expert@gw1]# vpn shell
? – This help
.. – Go up one level
quit – Quit
[interface ] – Manipulate tunnel interfaces
[show ] – Show internal data
[tunnels ] – Manipulate tunnel data After hitting enter you are put into subshell that has hierarchy way of moving around, so to continue to show subtree you type show and hit Enter:
VPN shell:[/] > show
? – This help
.. – Go up one level
[interface ] – Show interface(s) and their status
[tunnels ] – Show SA(s)
VPN shell:[/show] > Your prompt changes to the path inside vpn shell, to go 1 level up (return) type .. and Enter:
VPN shell:[/show] > ..
? – This help
.. – Go up one level
quit – Quit
[interface ] – Manipulate tunnel interfaces
[show ] – Show internal data
[tunnels ] – Manipulate tunnel data
VPN shell:[/] > In addition if you know the full path inside vpn shell to the command you wish to run you can type it too: e.g. To see all IKE tunnels:
[Expert@gw1]# vpn shell
? – This help
.. – Go up one level
quit – Quit
[interface ] – Manipulate tunnel interfaces
[show ] – Show internal data
[tunnels ] – Manipulate tunnel data
VPN shell:[/] > tunnels show IKE all Peer 193.x.x.x: 1. IKE SA <8755c7fb24a52e9b,5d46b29d0f0bb5b7>:
VPN shell:[/] >
e.g. 2 To delete IKE SAs for specific peer:
VPN shell:[/] > tunnels delete IKE peer 193.3.3.3 NOTE: interface subtree is for dealing with VTI interfaces. And finally to leave the vpn shell to SSH shell:
Get to the root by typing .. as many times as needed and then quit: VPN shell:[/show/tunnels/IKE] > ../../..
? – This help
.. – Go up one level
quit – Quit
[interface ] – Manipulate tunnel interfaces
[show ] – Show internal data
[tunnels ] – Manipulate tunnel data
VPN shell:[/] > quit
[Expert@gw1]#
Read More ...
" Delete all IPsec SAs for a given peer (GW)" – but it just plain doesn't work. And once confronted with this problem that could make debug more devastating than the problem itself I started looking for alternatives. To much of my surprise CP has a perfect alternative for this
- vpn shell, that provides acceptable means of managing tunnels. Here are details:
vpn shell can : delete IKE/IPsec SAs selectively, add/delete VTI interfaces,show information about all that.
To enter this shell :
[Expert@gw1]# vpn shell
? – This help
.. – Go up one level
quit – Quit
[interface ] – Manipulate tunnel interfaces
[show ] – Show internal data
[tunnels ] – Manipulate tunnel data After hitting enter you are put into subshell that has hierarchy way of moving around, so to continue to show subtree you type show and hit Enter:
VPN shell:[/] > show
? – This help
.. – Go up one level
[interface ] – Show interface(s) and their status
[tunnels ] – Show SA(s)
VPN shell:[/show] > Your prompt changes to the path inside vpn shell, to go 1 level up (return) type .. and Enter:
VPN shell:[/show] > ..
? – This help
.. – Go up one level
quit – Quit
[interface ] – Manipulate tunnel interfaces
[show ] – Show internal data
[tunnels ] – Manipulate tunnel data
VPN shell:[/] > In addition if you know the full path inside vpn shell to the command you wish to run you can type it too: e.g. To see all IKE tunnels:
[Expert@gw1]# vpn shell
? – This help
.. – Go up one level
quit – Quit
[interface ] – Manipulate tunnel interfaces
[show ] – Show internal data
[tunnels ] – Manipulate tunnel data
VPN shell:[/] > tunnels show IKE all Peer 193.x.x.x: 1. IKE SA <8755c7fb24a52e9b,5d46b29d0f0bb5b7>:
VPN shell:[/] >
e.g. 2 To delete IKE SAs for specific peer:
VPN shell:[/] > tunnels delete IKE peer 193.3.3.3 NOTE: interface subtree is for dealing with VTI interfaces. And finally to leave the vpn shell to SSH shell:
Get to the root by typing .. as many times as needed and then quit: VPN shell:[/show/tunnels/IKE] > ../../..
? – This help
.. – Go up one level
quit – Quit
[interface ] – Manipulate tunnel interfaces
[show ] – Show internal data
[tunnels ] – Manipulate tunnel data
VPN shell:[/] > quit
[Expert@gw1]#
Checkpoint Common debugging
Kernel debugging
Usage
% fw ctl debug -buf [buffer size]
% fw ctl debug [-x] [-m ] [+|-]
% fw ctl kdebug –f >
To disable the Kernel debugging, execute:
% fw ctl debug –buf 0
% fw ctl debug x
Common Syntax
% fw ctl debug –buf 12288
% fw ctl debug –m fw conn drop ld packet if
% fw ctl kdebug –f >
The ld option may cause high CPU usage. It is advised to use it for short session debugging
only.
To execute the kernel you can also use fw ctl zdebug to allocate the buffer (where the
buffer can only be 1024).
% fw ctl zdebug
% fw ctl kdebug -f > TDERROR_ALL_ALL=
CPD is treated differently from the other User Mode processes and will be executed
differently
Debugging CPD
CPD is a high in the hierarchichal chain and helps to execute many services, such as Secure
Internal Communcation (SIC), Licensing and status report.
For CPD debug, execute: % cpd_admin debug on TDERROR_ALL_ALL=5
The debug file is located under $CPDIR/log/cpd.elg
To stop the CPD debug, execute: % cpd_admin debug off TDERROR_ALL_ALL=1
Debugging FWM
The FWM process is responsible for the execution of the database activities of the
SmartCenter server. It is; therefore, responsible for Policy installation, Management High
Availability (HA) Synchronization, saving the Policy, Database Read/Write action, Log
Display, etc.
For FWM debug, execute:
% fw debug fwm on TDERROR_ALL_ALL=5
% fw debug fwm on OPSEC_DEBUG_LEVEL=9
The debug file is located under $FWDIR/log/fwm.elg
To stop the FWM debug, execute:
% fw debug fwm off TDERROR_ALL_ALL=1
% fw debug fwm off OPSEC_DEBUG_LEVEL=1
Debugging FWD
The FWD process is responsible for logging. It is executed in relation to logging, Security
Servers and communication with OPSEC applications.
For FWD debug, execute: % fw debug fwd debug on TDERROR_ALL_ALL=5
The debug file is located under $FWDIR/log/fwd.elg
To stop the FWD debug, execute: % fw debug fwd off TDERROR_ALL_ALL=1
FireWall Monitor Network Capturing
The FireWall Monitor is responsible for packet flow analysis.
To execute: % fw monitor –e "accept;" –o
Security Server debugging
Debugging User Authentication
Usage
Debugging is done on the service itself (in.ahttpd, in.atelnetd, in.aftpd etc.)
% fw debug on TDERROR_ALL_ALL=5
The debug file is located under: $FWDIR/log/ahttpd.elg* or $FWDIR/log/aftpd.elg* or
$FWDIR/log/atelnetd.elg* depending on the service that you are debugging.
HTTP Security Server
For HTTP Security Server debug, execute:
% fw debug in.ahttpd on TDERROR_ALL_ALL=5
% fw debug in.ahttpd on OPSEC_DEBUG_LEVEL=3
The debug file is located under: $FWDIR/log/ahttpd.elg*
If more than one HTTP Security Server process is running, execute:
% fw kill fwd
% setenv TDERROR_ALL_ALL=5
% setenv OPSEC_DEBUG_LEVEL=3
% fwd –d >& &
Note - The setenv commands used above correlate with Unix environment. For other platforms, execute
the relevant command.
SMTP Security Server
To debug the SMTP Security Server, execute:
% fw debug in.asmtpd on TDERROR_ALL_ALL=5.
The debug file is located under $FWDIR/log/asmtpd.elg*
To debug the mdq, execute the following commands:
% fw debug mdq on TDERROR_ALL_ALL=5.
The debug file is located under $FWDIR/log/mdq.elg*
Debugging Session Authentication
To debug Session Authentication, execute:
% fw debug in.asessiond on TDERROR_ALL_ALL=5
The debug file is located under: $FWDIR/log/asessiond.elg*
Debugging Client Authentication
For HTTP to port 900, execute:
Check Point Troubleshooting and Debugging Tools for Faster Resolution. Last Update — January 24, 2006 5
% fw debug in.ahclientd on TDERROR_ALL_ALL=5
For Telnet to port 259, execute:
% fw debug in.aclientd on TDERROR_ALL_ALL=5
The debug file is located under: $FWDIR/log/ahclientd.elg*
VPN debugging
On the Module
To start, execute:
% vpn debug trunc.
This command is equivalent to these two commands: vpn debug on, vpn debug ikeon.
To stop, execute:
% vpn debug off; vpn debug ikeoff.
The debug file is located under $FWDIR/log/ike.elg and $FWDIR/log/vpnd.elg
FireWall Monitor for packet flow analysis
% fw monitor –e "accept;" –o
Client Side
The Client side can only run under the root directory (C :/…)
To start, execute:
% sc debug on
To stop, execute:
% sc debug off
The debug file is located under sr_service_tde.log, under the SecuRemote installation
folder, for example: C:\Program files\CheckPoint\SecuRemote.
For packet capture from the Client side, execute:
% srfw monitor -e "accept;" -o
Provider-1 debugging
MDS Level
Most of the MDS actions are performed by the MDS's fwm process, execute:
% mdsenv
% fw debug mds on TDERROR_ALL_ALL=5
% fw debug mds on OPSEC_DEBUG_LEVEL=9
The debug file is located under /opt/CPsuit-R60/fw1/log/mds.elg
Check Point Troubleshooting and Debugging Tools for Faster Resolution. Last Update — January 24, 2006 6
ClusterXL debugging
For ClusterXL debugging for Clustering, Synchronization, High Availability, Fail-over,
execute:
% cphaprob state
% cphaprob -ia list
% cphaprob -a if
% fw ctl pstat
Kernel debug for packet filter analysis
% fw ctl debug –buf 12288
% fw ctl debug –m fw conn drop packet if sync
% fw ctl debug –m cluster all
% fw ctl kdebug –f >
Connectra debugging
For Connectra debugging issues relating to Web, files, Webmail, OWA, iNotes, Citrix, the
httpd process should be debugged:
To turn the debug on, under: $CVPNDIR/conf/httpd.conf change LogLevel to debug.
You should execute the process: cvpnrestart
The output is located at: $CVPNDIR/log/httpd.log
For debugging authentication issues, execute: Debug cvpnd
Run: cvpnd_admin debugset TDERROR_ALL_ALL=5
To start, execute: % cvpnrestart
The debug file is located under $CVPNDIR/log/cvpnd.elg
To stop debug, run:
% cvpnd_admin debug off
FireWall-1 GX debugging
Kernel debug for packet filter analysis
Check Point Troubleshooting and Debugging Tools for Faster Resolution.
% fw ctl debug –buf 12288
% fw ctl debug –m fw conn drop ld packet filter
% fw ctl kdebug –T –f >
InterSpect debugging
Kernel debug for packet filter analysis
% fw ctl debug –buf 12288
% fw ctl debug –m fw conn drop packet if
% fw ctl kdebug –f >
Additional kernel debug options for InterSpect:
• portscan, for port scanning issues
• dynlog, for dynamic logging
• mail, for mail security in the kernel
• sam, for SAM IP address blocking
Kernel debug for Packet Drop, execute:
% fw ctl zdebug + drop
Kernel debug for SmartDefense TCP Streaming, execute:
% fw ctl zdebug + tcpstr + cifs
Kernel debug for Dynamic list (SAM), execute:
% fw tab -t sam_requests_v2 -u -f
% fw samp
SNX – SSL Network Extender debugging
Server Side
% vpn debug trunc
% vpn debug on slim=5
Debug can be found at $FWDIR/log/vpnd.elg.
You should execute vpn debug on [DEBUG_TOPIC=5]. The relevant debug topics are:
proxy, rasta, rasta_protocol and slim.)
Client Side
For the service:
Type regedit at the command prompt and set:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\cpextender\parameters\d
bg_level to 5
Open the Command Line interface window and execute:
Check Point Troubleshooting and Debugging Tools for Faster Resolution. Last Update — January 24, 2006 8
% net stop cpextender
% net start cpextender (or kill slimsvc.exe)
The debug file is located under:
%Program Files%\CheckPoint\SSL Network Extender\slimsvc.log
For the ActiveX: (only when using ActiveX with Internet Explorer), type regedit at the
command prompt and set the following:
% set HKEY_CURRENT_USER\Software\CheckPoint\SSL Network
Extender\parameters\dbg_level to 5
The debug file is located under %APPDATA%\Check Point\extender\activex.log.
For the Applet: (when using the Applet version) SNX can be used by Microsoft JVM or by
other vendors (SUN, IBM…). To view the Java console when using Microsoft JVM you
need to check Java console enabled (requires restart) in the Internet Options Advanced tab
and restart Internet Explorer. You can also switch between the different JVMs (in case you
have two or more) in the same tab.
Further Debugging – Memory Diagnostics
The following utilities applies to all non-Windows systems supported by Check Point:
% free
% vmstat 2 10
% sar –k 2 10
% top
% ps -auxw
% cat /proc/meminfo
% cat /proc/slabinfo
Routing information
% arp –a
% netstat –ie
Kernel debugging
Usage
% fw ctl debug -buf [buffer size]
% fw ctl debug [-x] [-m ] [+|-]
% fw ctl kdebug –f >
To disable the Kernel debugging, execute:
% fw ctl debug –buf 0
% fw ctl debug x
Common Syntax
% fw ctl debug –buf 12288
% fw ctl debug –m fw conn drop ld packet if
% fw ctl kdebug –f >
The ld option may cause high CPU usage. It is advised to use it for short session debugging
only.
To execute the kernel you can also use fw ctl zdebug to allocate the buffer (where the
buffer can only be 1024).
% fw ctl zdebug
% fw ctl kdebug -f > TDERROR_ALL_ALL=
CPD is treated differently from the other User Mode processes and will be executed
differently
Debugging CPD
CPD is a high in the hierarchichal chain and helps to execute many services, such as Secure
Internal Communcation (SIC), Licensing and status report.
For CPD debug, execute: % cpd_admin debug on TDERROR_ALL_ALL=5
The debug file is located under $CPDIR/log/cpd.elg
To stop the CPD debug, execute: % cpd_admin debug off TDERROR_ALL_ALL=1
Debugging FWM
The FWM process is responsible for the execution of the database activities of the
SmartCenter server. It is; therefore, responsible for Policy installation, Management High
Availability (HA) Synchronization, saving the Policy, Database Read/Write action, Log
Display, etc.
For FWM debug, execute:
% fw debug fwm on TDERROR_ALL_ALL=5
% fw debug fwm on OPSEC_DEBUG_LEVEL=9
The debug file is located under $FWDIR/log/fwm.elg
To stop the FWM debug, execute:
% fw debug fwm off TDERROR_ALL_ALL=1
% fw debug fwm off OPSEC_DEBUG_LEVEL=1
Debugging FWD
The FWD process is responsible for logging. It is executed in relation to logging, Security
Servers and communication with OPSEC applications.
For FWD debug, execute: % fw debug fwd debug on TDERROR_ALL_ALL=5
The debug file is located under $FWDIR/log/fwd.elg
To stop the FWD debug, execute: % fw debug fwd off TDERROR_ALL_ALL=1
FireWall Monitor Network Capturing
The FireWall Monitor is responsible for packet flow analysis.
To execute: % fw monitor –e "accept;" –o
Security Server debugging
Debugging User Authentication
Usage
Debugging is done on the service itself (in.ahttpd, in.atelnetd, in.aftpd etc.)
% fw debug on TDERROR_ALL_ALL=5
The debug file is located under: $FWDIR/log/ahttpd.elg* or $FWDIR/log/aftpd.elg* or
$FWDIR/log/atelnetd.elg* depending on the service that you are debugging.
HTTP Security Server
For HTTP Security Server debug, execute:
% fw debug in.ahttpd on TDERROR_ALL_ALL=5
% fw debug in.ahttpd on OPSEC_DEBUG_LEVEL=3
The debug file is located under: $FWDIR/log/ahttpd.elg*
If more than one HTTP Security Server process is running, execute:
% fw kill fwd
% setenv TDERROR_ALL_ALL=5
% setenv OPSEC_DEBUG_LEVEL=3
% fwd –d >& &
Note - The setenv commands used above correlate with Unix environment. For other platforms, execute
the relevant command.
SMTP Security Server
To debug the SMTP Security Server, execute:
% fw debug in.asmtpd on TDERROR_ALL_ALL=5.
The debug file is located under $FWDIR/log/asmtpd.elg*
To debug the mdq, execute the following commands:
% fw debug mdq on TDERROR_ALL_ALL=5.
The debug file is located under $FWDIR/log/mdq.elg*
Debugging Session Authentication
To debug Session Authentication, execute:
% fw debug in.asessiond on TDERROR_ALL_ALL=5
The debug file is located under: $FWDIR/log/asessiond.elg*
Debugging Client Authentication
For HTTP to port 900, execute:
Check Point Troubleshooting and Debugging Tools for Faster Resolution. Last Update — January 24, 2006 5
% fw debug in.ahclientd on TDERROR_ALL_ALL=5
For Telnet to port 259, execute:
% fw debug in.aclientd on TDERROR_ALL_ALL=5
The debug file is located under: $FWDIR/log/ahclientd.elg*
VPN debugging
On the Module
To start, execute:
% vpn debug trunc.
This command is equivalent to these two commands: vpn debug on, vpn debug ikeon.
To stop, execute:
% vpn debug off; vpn debug ikeoff.
The debug file is located under $FWDIR/log/ike.elg and $FWDIR/log/vpnd.elg
FireWall Monitor for packet flow analysis
% fw monitor –e "accept;" –o
Client Side
The Client side can only run under the root directory (C :/…)
To start, execute:
% sc debug on
To stop, execute:
% sc debug off
The debug file is located under sr_service_tde.log, under the SecuRemote installation
folder, for example: C:\Program files\CheckPoint\SecuRemote.
For packet capture from the Client side, execute:
% srfw monitor -e "accept;" -o
Provider-1 debugging
MDS Level
Most of the MDS actions are performed by the MDS's fwm process, execute:
% mdsenv
% fw debug mds on TDERROR_ALL_ALL=5
% fw debug mds on OPSEC_DEBUG_LEVEL=9
The debug file is located under /opt/CPsuit-R60/fw1/log/mds.elg
Check Point Troubleshooting and Debugging Tools for Faster Resolution. Last Update — January 24, 2006 6
ClusterXL debugging
For ClusterXL debugging for Clustering, Synchronization, High Availability, Fail-over,
execute:
% cphaprob state
% cphaprob -ia list
% cphaprob -a if
% fw ctl pstat
Kernel debug for packet filter analysis
% fw ctl debug –buf 12288
% fw ctl debug –m fw conn drop packet if sync
% fw ctl debug –m cluster all
% fw ctl kdebug –f >
Connectra debugging
For Connectra debugging issues relating to Web, files, Webmail, OWA, iNotes, Citrix, the
httpd process should be debugged:
To turn the debug on, under: $CVPNDIR/conf/httpd.conf change LogLevel to debug.
You should execute the process: cvpnrestart
The output is located at: $CVPNDIR/log/httpd.log
For debugging authentication issues, execute: Debug cvpnd
Run: cvpnd_admin debugset TDERROR_ALL_ALL=5
To start, execute: % cvpnrestart
The debug file is located under $CVPNDIR/log/cvpnd.elg
To stop debug, run:
% cvpnd_admin debug off
FireWall-1 GX debugging
Kernel debug for packet filter analysis
Check Point Troubleshooting and Debugging Tools for Faster Resolution.
% fw ctl debug –buf 12288
% fw ctl debug –m fw conn drop ld packet filter
% fw ctl kdebug –T –f >
InterSpect debugging
Kernel debug for packet filter analysis
% fw ctl debug –buf 12288
% fw ctl debug –m fw conn drop packet if
% fw ctl kdebug –f >
Additional kernel debug options for InterSpect:
• portscan, for port scanning issues
• dynlog, for dynamic logging
• mail, for mail security in the kernel
• sam, for SAM IP address blocking
Kernel debug for Packet Drop, execute:
% fw ctl zdebug + drop
Kernel debug for SmartDefense TCP Streaming, execute:
% fw ctl zdebug + tcpstr + cifs
Kernel debug for Dynamic list (SAM), execute:
% fw tab -t sam_requests_v2 -u -f
% fw samp
SNX – SSL Network Extender debugging
Server Side
% vpn debug trunc
% vpn debug on slim=5
Debug can be found at $FWDIR/log/vpnd.elg.
You should execute vpn debug on [DEBUG_TOPIC=5]. The relevant debug topics are:
proxy, rasta, rasta_protocol and slim.)
Client Side
For the service:
Type regedit at the command prompt and set:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\cpextender\parameters\d
bg_level to 5
Open the Command Line interface window and execute:
Check Point Troubleshooting and Debugging Tools for Faster Resolution. Last Update — January 24, 2006 8
% net stop cpextender
% net start cpextender (or kill slimsvc.exe)
The debug file is located under:
%Program Files%\CheckPoint\SSL Network Extender\slimsvc.log
For the ActiveX: (only when using ActiveX with Internet Explorer), type regedit at the
command prompt and set the following:
% set HKEY_CURRENT_USER\Software\CheckPoint\SSL Network
Extender\parameters\dbg_level to 5
The debug file is located under %APPDATA%\Check Point\extender\activex.log.
For the Applet: (when using the Applet version) SNX can be used by Microsoft JVM or by
other vendors (SUN, IBM…). To view the Java console when using Microsoft JVM you
need to check Java console enabled (requires restart) in the Internet Options Advanced tab
and restart Internet Explorer. You can also switch between the different JVMs (in case you
have two or more) in the same tab.
Further Debugging – Memory Diagnostics
The following utilities applies to all non-Windows systems supported by Check Point:
% free
% vmstat 2 10
% sar –k 2 10
% top
% ps -auxw
% cat /proc/meminfo
% cat /proc/slabinfo
Routing information
% arp –a
% netstat –ie
Posted
what is fw ctl pstat??
According to me, its nothing but a fw command with which we can monitor the heath of your CP box., especially Syc Status.. Am sure that you will love this command and say thanks for CP for this……
According to me, its nothing but a fw command with which we can monitor the heath of your CP box., especially Syc Status.. Am sure that you will love this command and say thanks for CP for this……
Am taking an example to explain the same.. here we go…….
Sync:
Version: new
Status: Able to Send/Receive sync packets
Sync packets sent:
total : 466729198, retransmitted : 241305, retrans reqs : 6089, acks : 809
Sync packets received:
total : 77283541, were queued : 6715, dropped by net : 6079
retrans reqs : 37462, received 175 acks
retrans reqs for illegal seq : 0
dropped updates as a result of sync overload: 0
Delta Sync memory usage: currently using XX KB mem
Callback statistics: handled 138 cb, average delay : 2, max delay : 34
Number of Pending packets currently held: 1
Packets released due to timeout: 18
Explanation:
Version: new
This line must appear if synchronization is configured (versions above 4.1)
Status: Able to Send/Receive sync packets
If sync is unable to either send or receive packets, there is a problem
Sync packets sent:
total : 466729198, retransmitted : 241305, retrans reqs : 6089, acks : 809
TOTAL number of sync packets is non-zero and increasing
RETRANS REQS may increase under load
Sync packets received:
total : 77283541, were queued : 6715, dropped by net : 6079
QUEUED value never decreases - A non-zero value does not indicate a problem
DROPPED BY NET number may indicate network congestion
The "dropped by net" counter is incremented when the cluster member receives a sync packet with a sequence number which is higher than the expected seq num. This means packets with lower seq where lost somewhere along the way, and we need to find out where.
retrans reqs : 37462, received 175 acks
RETRANS REQS growing very fast may indicate that the load is becoming too high
retrans reqs for illegal seq : 0
May indicate a sync problem
dropped updates as a result of sync overload: 0
In a heavily loaded system, the cluster member may drop synchronization updates sent from another cluster member
Delta Sync memory usage: currently using XX KB mem
This statistic only appears for a non-zero value.
It requires memory only while full sync is occurring at other times, Delta sync requires no memory
Callback statistics: handled 138 cb, average delay : 2, max delay : 34
This statistic only appears for a non-zero value.
AVERAGE DELAY should be 1-5 packets, otherwise indicates an overload of sync traffic
Number of Pending packets currently held: 1
This statistic only appears for a non-zero value.
Packets released due to timeout: 18
This statistic only appears for a non-zero value.
If the it is large (more than 100 pending packets), and the "Number of Pending packets currently held" is small, you should take action to reduce the number of pending packets.
To tackle this problem, try google "Reducing the Number of Pending Packets". Read More ...
Login to Linux box as root and enter root's password:
[me@mybox me]$ su
password:
Check the current date and time of the Linux box by entering:
[root@mybox me]# date
Linux yields the current settings:
[root@mybox me]# Wed Apr 7 12:03:45 EDT 2004
Change the current time and date of the Linux box by entering:
[root@mybox me]# date 040713032004
would change the time and yield:
[root@mybox me]$ Wed Apr 7 13:03:00 EDT 2004
[me@mybox me]$ su
password:
Check the current date and time of the Linux box by entering:
[root@mybox me]# date
Linux yields the current settings:
[root@mybox me]# Wed Apr 7 12:03:45 EDT 2004
Change the current time and date of the Linux box by entering:
[root@mybox me]# date 040713032004
would change the time and yield:
[root@mybox me]$ Wed Apr 7 13:03:00 EDT 2004
====================================================
One more option:
Linux Set Date
Use the following syntax to set new data and time:date set="STRING"
For example, set new data to 2 Oct 2006 18:00:00, type the following command as
root user:# date -s "2 OCT 2006 18:00:00" Read More ...
The following article is a list of steps one should go through when troubleshooting logging related issues in a distributed setup.
1. Ensure that you have not run out of disk space on the hard disk that the logs are being sent to. If this is the case, delete or move the logs to an external storage device.2. Is there communication between the MS and the Module? Test using ping to the MS from the module and then from the Module to the MS (your rules must allow for this). If this fails, and your rules allow for this, then it is most likely a routing issue.
3. Check to see if the fw.log file is growing on the module. It should be if the logs are not going to the MS. From the console run these commands:
cd $FWDIR/log
ls -la
ls -la
Verify that the fw.log file is increasing. If it is increasing then the modules are logging locally instead of forwarding the traffic to the MS. This could be a connectivity issue, or it could be the way the logging is setup. Check the FW object to ensure it is setup to send logs to the MS.
4. Can you fetch a policy? Verify that you can fetch using the hostname and IP address. If this fails then you probably have a SIC issue. To test this run the following commands:
fw fetch hostname_of_MS
fw fetch IP_Addr_of_MS (fetch by IP address also to ensure it is not a DNS issue)
5. Check the masters file. The hostname or IP address of the management station should be listed in there. To check this run the following commands:
cd $FWDIR/conf
cat masters
It should be look like this:[Policy]
hostname_of_MS
[Log]
hostname_of_MS
[Alert]
hostname_of_MS
6. Run tcpdumps on the module, listening for port 257 on the interface facing the MS, to see if it is attempting to send logs. To check this run the following command:
tcpdump -i eth-facing-MS port 257 (use the Ctrl+C to break out of the dump)
You should see traffic leaving the FW and heading to the IP address of the MS.
You should also see traffic coming back from the MS.
7. The log file may have gotten corrupt. Run a log switch on the MS and reboot the MS to create a new log file. If logswitch does not work, move all contents of the log directory (do not move the directory itself) to a temp folder outside of the log directory. Reboot and see if the logs start again.
8. Delete the $FWDIR/log files and $FWDIR/state directory files on the module; reboot the module.
Reboot and see if the logs start again.
9. Look to see if there is a listening port for logging. Run the following command on the MS and the module:
netstat -na
You should see the *.257 LISTEN for logging connections. You should also see the IP address of the MS :257 associated with the IP address of each module, and showing an ESTABLISHED connection.
10. Check the log settings for the FW object and make sure the 'Log Server' is set to the MS that should be receiving the logs. This is usually done by default, but may have been changed by a user.
If after going through these steps you are still experiencing logging issues, please open a ticket with Corresponding TAC for further troubleshooting and ofcoz try your way with help of our all time Gaint Mr. Google.. :-)
Date, System Uptime and Clock:
Confirm the correct date is set on the system using the 'date' command.
The system uptime can be examined using the command:
uptime
Example output:
Zulu# uptime
09:46:34 up 124 days, 9:40, 1 user, load average: 0.36, 0.19, 0.14
If a low uptime is shown it normally indicates that the firewall has been administratively rebooted but it may also have been due to a self-reboot, for example due to a panic.
Low uptime - if you suspect the uptime is less than it should be check the
/var/log/messages file for the reason of the last reboot.
Disk Space
The disk space usage can be examined using the command:
df –k
Example output:
[Expert@Zulu]# df –k
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda5 600832 187800 382512 33% /
none 600832 187800 382512 33% /dev/pts
/dev/sda1 147766 10124 130013 8% /boot
/dev/sda7 1541680 930324 533044 64% /opt
none 2045688 0 2045688 0% /dev/shm
/dev/sda6 1541680 593844 869524 41% /sysimg
/dev/sda8 27024000 5472984 20178264 22% /var
[Expert@Zulu]#
In the above example, all partitions are under 70% usage.
If a partition has a „use%' that is more than 70% but less than 90%
If the „use%' is 90% or more
See if the partition can be cleaned up to free up disk space.
/var/opt/CPsuite-RXX/fw1/log may be filled with old log files if the firewall has been logging locally.
/var/log may have old messages files
Physical RAM and Swap Space:
Examine the RAM and swap space usage (kilobytes) with:
free –k –t
Example output:
[Expert@Zulu]# free –k -t
total used free shared buffers
cached
Mem: 2058236 971332 1086904 0 95104
268984
-/+ buffers/cache: 607244 1450992
Swap: 4192944 0 4192944
Total: 6251180 971332 5279848 [Expert@Zulu]#
The „total‟ column shows the amount of RAM installed in the system (2GB in the above example)
and the amount of disk space allocated for swap space (4GB).
The amount of swap space is normally automatically set to twice the size of the physical memory, with 4 GB being the maximum.
The „used‟ column indicates how much RAM and swap space are being used.
The „free‟ column indicates how much RAM and swap space are available.
In the above example output the „used‟ column indicates <1 GB of RAM is being used and no
swap space is being used.
If for some reason the amount of free RAM becomes low, the appliance will start to preserve free RAM by swapping out the contents of the memory to the hard disk (swap space). The performance will be sub-optimal if swap space is being used due to time and resources spent writing and reading to the hard-disk.
Example Output:
[Expert@Zulu]# free –k -t
total used free shared buffers
cached
Mem: 2055120 1897424 157696 0 98732
697688
-/+ buffers/cache: 1101004 954116
Swap: 4192912 735980 3456932
Total: 6248032 2633404 3614628 [Expert@Zulu]#
Swap space usage may indicate not enough memory is installed in the appliance. The kernel is
32 bit and can use up to 4GB. It is recommended to upgrade the memory if less than 4GB of RAM
are installed.
For further information about the amount of RAM that is supported by SecurePlatform refer to:
sk22343: What is the maximum memory supported by SecurePlatform?
Memory Usage
The firewall‟s memory usage can be examined by using the command:
fw ctl pstat
The output of this command is vast and can be difficult to understand as not all the output is intuitive. The statistics that need to be checked to ensure memory is healthy are:
· hash kernel memory „hmem‟
· system kernel memory „smem‟
· kernel memory „kmem‟.
Example output:
[Expert@Zulu]# fw ctl pstat | more
Machine Capacity Summary:
Memory used: 7% (128MB out of 1638MB) - below low watermark
Concurrent Connections: 21% (43253 out of 199900) - below low watermark
Aggressive Aging is not active
Hash kernel memory (hmem) statistics:
Total memory allocated: 142606336 bytes in 34782 4KB blocks using 34 pools
Initial memory allocated: 20971520 bytes (Hash memory extended by
121634816 bytes)
Memory allocation limit: 335544320 bytes using 512 pools
Total memory bytes used: 39254196 unused: 103352140 (72.47%) peak:
133739228
Total memory blocks used: 10335 unused: 24447 (70%) peak:
32795
Allocations: 3375437074 alloc, 0 failed alloc, 3375001310 free
System kernel memory (smem) statistics:
Total memory bytes used: 188577580 peak: 227270504
Blocking memory bytes used: 1958392 peak: 2205256
Non-Blocking memory bytes used: 186619188 peak: 225065248
Allocations: 979925174 alloc, 0 failed alloc, 979924513 free, 0 failed
free
Kernel memory (kmem) statistics:
Total memory bytes used: 84876956 peak: 177110948
Allocations: 3375820431 alloc, 0 failed alloc, 3375384380 free, 0 failed
free
External Allocations: 0 for packets, 31589936 for SXL
In the above example there are no hmem, smem, kmem failed allocations.
Presence of „hmem‟ failed allocations indicates that the hash kernel memory was full. This is not a serious memory problem but indicates there is a configuration problem. The value assigned to the hash memory pool, (either manually or automatically by changing the number concurrent
connections in the capacity optimization section of a firewall) determines the size of the hash kernel memory. If a low hmem limit was configured it leads to improper usage of the OS memory. See
„Capacity Optimization‟ in the „Firewall Health Checks‟ section for further information.
Presence of „smem‟ failed allocations indicates that the OS memory was exhausted or there are large non-sleep allocations. This is symptomatic of a memory shortage. If there are failed smem allocations and the memory is less than 2 GB, upgrading to 2GB may fix the problem. Decreasing
the TCP end timeout and decreasing the number of concurrent connections can also help reduce memory consumption.
Presence of „kmem‟ failed allocations means that some applications did not get memory. This is usually an indication of a memory problem; most commonly a memory shortage. The natural limit is
2GB, since the Kernel is 32bit.)
Memory shortage sometimes indicates a memory leak. In order to troubleshoot memory shortage, stop the load you need to stop the load and let connections close. If the memory consumption returns back to normal, you are not dealing with a memory leak. Such shortage might happen when traffic volumes are too high for the device capacity. If the memory shortage happens after a change in the system or the environment, undo the change, and check whether kmem memory consumption goes down.
Read More ...