Infrastructure Software

Announcement

For appeals, questions and feedback about Oracle Forums, please email oracle-forums-moderators_us@oracle.com. Technical questions should be asked in the appropriate category. Thank you!

Interested in getting your voice heard by members of the Developer Marketing team at Oracle? Check out this post for AppDev or this post for AI focus group information.

Using the Linux netem Module to Test Your Network Without Causing Adverse Effects

Johan LouwersOct 18 2016 — edited Oct 21 2016

by Johan Louwers

Learn how to test your application's behavior when your network is not providing optimal service, how to test whether your network monitoring scripts work, and how to simulate strange network behavior in a contained, controlled manner and without impacting the network adversely.

Networking has and will always be an important part of IT solutions. You can build the most beautiful solutions, but if you are unable to expose them to the enterprise, they will not be of any use to users. With the trend of enterprises moving to the cloud and the adoption of microservices-based solutions, the network becomes even more important.

The area in which administrators, in general, lack available tools is monitoring the network in depth. Most enterprises implement monitoring to check the status of network equipment, for example, routers and switches. Also it is quite common to use monitoring scripts to check if a ping command sent to a server returns. However, checking whether a ping command comes back within a given amount of time is beyond the capability of many IT departments, unfortunately.

Doing deeper netflow analysis, routing analysis, and end-user-perspective analysis is something that should be common practice; however, that is not commonly seen within most enterprises. In the cases where it is done, it is commonly done by the network department in a "silo" manner where the data is not shared (by default) with other departments who could receive a huge benefit from it.

Using Oracle Enterprise Manager

In general, solutions such as Oracle Enterprise Manager (and other solutions) are moving away from monitoring an instance or single component as a target and, instead, the entire end-to-end landscape is becoming the monitoring target. This means that monitoring from an end-user perspective is becoming more adopted, for example, by using Oracle Real User Experience Insight and monitoring network connections between zones.

For example, monitoring the network speed between two components—the application server and the database server—is becoming (almost) a standard in enterprises that realize network monitoring deserves more attention than it is getting.

For monitoring the network between an Oracle WebLogic application server (or any other application server) and an instance of Oracle Database you can use, as an example, Oracle Enterprise Manager and remote beacons. Remote beacons can be deployed across the network and execute a number of custom checks. Remote beacons are a standard component of Oracle Enterprise Manager; however, they are commonly overlooked even though they are a powerful tool especially for monitoring network connectivity and speed between all kinds of components (targets).

How to Develop, Deploy, and Test Remote Beacons and Scripts

The question of how to develop and deploy remote beacons and possibly the custom-coded checks you want them to execute is "simply" answered by reading the Oracle Enterprise Manager documentation. The trickier part is how you test them.

Without too much effort, you can develop checks that monitor the results of TNSPING, ping, or whatever network-related command you would like to monitor. However, testing the checks becomes a bit more difficult. Slowing down your network to test whether your script is doing the right thing and the alert is triggered in the right manner under the right condition can be a daunting task; you might not want to play with switches or flood the network with bogus traffic.

This is where the netem Linux kernel module comes into play. The module needs to be compiled with your kernel, and then you can use it to emulate conditions on the network on a Linux machine. The Linux operating system will interpret the emulated conditions as a true network condition while the conditions are, in fact, emulated behavior controlled by the netem module.

Using the `netem` Kernel Module

When you need to test certain conditions on your network, for example, slow network response, you do not want to do this by playing with switches and routers in your network. You most likely want to do this in a contained and controlled manner. Using the netem kernel module will help enormously with this.

If you use lsmod to check for netem, you most likely will find that is it not loaded directly.

[root@testbox09 ~]# lsmod | grep netem[root@testbox09 ~]#

However, you can load it using the command below as an example; this will add a delay 97 ms on all traffic on eth0 by using netem.

[root@testbox09 ~]# tc qdisc add dev eth0 root netem delay 97ms

[root@testbox09 ~]#

The command above means that if you used to have a ping time between about 13 ms and 16 ms when pinging google.com, it will now take between 110 ms and 113 ms.

[root@testbox09 ~]# ping google.com

PING google.com (74.125.136.139) 56(84) bytes of data.

64 bytes from ea-in-f139.1e100.net (74.125.136.139): icmp_seq=1 ttl=49 time=112 ms

64 bytes from ea-in-f139.1e100.net (74.125.136.139): icmp_seq=2 ttl=49 time=112 ms

64 bytes from ea-in-f139.1e100.net (74.125.136.139): icmp_seq=3 ttl=49 time=110 ms

64 bytes from ea-in-f139.1e100.net (74.125.136.139): icmp_seq=4 ttl=49 time=111 ms

64 bytes from ea-in-f139.1e100.net (74.125.136.139): icmp_seq=5 ttl=49 time=113 ms

--- google.com ping statistics ---

5 packets transmitted, 5 received, 0% packet loss, time 9006ms

rtt min/avg/max/mdev = 110.690/112.165/113.122/0.885 ms

[root@testbox09 ~]#

Now, in the example above, we pinged google.com, but you can see that you can easily use this command to add a delay on eth0 to test an Oracle Enterprise Manager remote beacon (or any other alerting script that is used). When you need to check which rules you have applied, you can use the command below to see a list of all rules that are active:

[root@testbox09 ~]# tc -s qdisc

qdisc netem 8001: dev eth0 root refcnt 2 limit 1000 delay 97.0ms

Sent 7982 bytes 62 pkt (dropped 0, overlimits 0 requeues 0)

backlog 118b 1p requeues 0

[root@testbox09 ~]#

To ensure your machine is acting normally and things are working as expected, you need to remove the rule. As we just have seen, we have one rule active.

Here are the commands for removing the rule and checking if it has been removed:

[root@testbox09 ~]# tc qdisc del dev eth0 root netem

[root@testbox09 ~]#

[root@testbox09 ~]# tc -s qdisc

qdisc pfifo_fast 0: dev eth0 root refcnt 2 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1

Sent 734 bytes 5 pkt (dropped 0, overlimits 0 requeues 0)

backlog 0b 0p requeues 0

[root@testbox09 ~]#

And here is the command to check whether things are back to normal again:

[root@testbox09 ~]# ping google.com

PING google.com (74.125.136.102) 56(84) bytes of data.

64 bytes from ea-in-f102.1e100.net (74.125.136.102): icmp_seq=1 ttl=49 time=13.7 ms

64 bytes from ea-in-f102.1e100.net (74.125.136.102): icmp_seq=2 ttl=49 time=13.6 ms

64 bytes from ea-in-f102.1e100.net (74.125.136.102): icmp_seq=3 ttl=49 time=14.2 ms

--- google.com ping statistics ---

3 packets transmitted, 3 received, 0% packet loss, time 2339ms

rtt min/avg/max/mdev = 13.692/13.905/14.284/0.285 ms

[root@testbox09 ~]#

How Does `netem` Work?

As you can see, we used something named netem, which is shown by lsmod as sch_netem. However, we also used tc qdisc, so you might be wondering how this all ties together.

If you read the netem man page, you'll see the following: "NetEm is an enhancement of the Linux traffic control facilities that allow to add delay, packet loss, duplication and more other characteristics to packets outgoing from a selected network interface. NetEm is built using the existing Quality Of Service (QOS) and Differentiated Services (diffserv) facilities in the Linux kernel."

This means that netem is an enhancement component of Linux traffic control to simulate specific behavior in IP traffic.

Figure 1 shows how the entire "stack" is being used to allow a process to communicate to an address somewhere on the network:

Figure 1. The stack that allows a process to communicate to a network address

The command we used, tc, configures traffic control in the Linux kernel. As you have noticed, the command was tc qdisc add dev eth0 root netem delay 97ms, which roughly translates to the following:

traffic control queuing disciple, add on device eth0, as the user root, a network emulator rule for a delay of 97 ms

By using this command, you have a netem delay rule that influences the queuing disciple that will hold traffic for 97 ms before releasing it to the driver queue of the associated NIC. This also means that the rest of your system is completely unaware of any changes to the system, because the only part that has knowledge of this netem rule is the queuing disciple. No applications will ever know this is simulated behavior.

Doing More with `netem`

As you look at the netem (tc-netem) man page, you will notice that the following is stated: "add delay, packet loss, duplication and more." In the example above, we saw how to add delay.

If you read the netem man page in more detail, you will find that next to the delay option we used in the example above, many other options are available to simulate the behavior of your network. Other available options include the following:

limit packets
distribution
loss random
loss state
loss gemodel
ecn
corrupt
duplicate
reorder
rate

All of the above will provide you with more options to simulate strange network behavior and test how your application, or testing scripts, will cope without the need to interfere with the actual network itself.

As an example, you can state a loss of 50 percent of the packages going out by using the following command. One small piece of advice, though, is to remember that if you use a network connection to connect to the machine and you use the same NIC for this, your own connection will also suffer from this behavior, meaning you will also be hit by the 50 percent package loss in your SSH session.

[root@testbox09 ~]# tc qdisc add dev eth0 root netem loss 50%

[root@testbox09 ~]#

[root@testbox09 ~]# ping google.com

PING google.com (216.58.217.46) 56(84) bytes of data.

64 bytes from 216.58.217.46: icmp_seq=6 ttl=48 time=116 ms

64 bytes from 216.58.217.46: icmp_seq=7 ttl=48 time=116 ms

64 bytes from 216.58.217.46: icmp_seq=18 ttl=48 time=116 ms

64 bytes from 216.58.217.46: icmp_seq=20 ttl=48 time=116 ms

64 bytes from 216.58.217.46: icmp_seq=21 ttl=48 time=116 ms

64 bytes from 216.58.217.46: icmp_seq=24 ttl=48 time=116 ms

64 bytes from 216.58.217.46: icmp_seq=25 ttl=48 time=116 ms

64 bytes from 216.58.217.46: icmp_seq=27 ttl=48 time=116 ms

--- google.com ping statistics ---

28 packets transmitted, 8 received, 71% packet loss, time 36813ms

rtt min/avg/max/mdev = 116.300/116.444/116.530/0.251 ms

[root@testbox09 ~]#

Conclusion

Testing how your application behaves when your network is not providing optimal service, testing how your network monitoring scripts work, and simulating strange network behavior all become much easier when you use netem, and using netem lowers the risk of conducting such tests.

Testing application behavior under strange network situations, ensuring you have the right monitoring scripts in place, and ensuring the scripts are tested in the right way can be vital for delivering a service to end users. Investing time in using netem and including it in your toolset will be a huge advantage.

About the Author

Johan Louwers is an Oracle ACE Director and leads the Capgemini Global Oracle Architect Office in his role as Global Chief Architect for Oracle Technology within the Capgemini infrastructure division.

794117

Java != javascript.
They are two very different things.

Think of the JSP lifecycle.
JSP is invoked. It runs Java/JSTL, produces an HTML page.
Java stops running.
HTML page arrives at your client. Javascript runs.
The only way to run java/jstl again is to make a request: click a link, submit a form... and get a new response.

I can't exactly figure out what you're doing there with your mix of javascript and JSTL.

You can use JSTL to produce javascript code.
For instance

function myFunction(){
  var stateArray = new Array();
  <c:forEach var="state" items="${stateList}" varStatus="status">
    stateArray[${status.index}] = "${state}";
  </c:forEach>
}

When this run it executes the JSTL and would produce something like

function myFunction(){
  var stateArray = new Array();  
  stateArray[0] = "North Carolina";
  stateArray[1] = "SouthCarolina";
  stateArray[2] = "North Dakota";
  stateArray[3] = "South Dakota";
  ...
}

Does that make things clearer?

843840

Evnafets,

Thanks for the good explanation. I understand that javascript runs on the client side. As you said, the JSTL is converted into HTML on the server side and sent to the client. If you look at my function, I have some JSTL code inside the javascript string. I understand from your code that the JSTL functions you use inside the javascript will be converted into actual javascript code when the HTML page is rendered. I have one question though. Will this conversion take place when the HTML is initially created (OR) will it take place only when the function is invoked by the client. I am almost sure the first option is right, but I want to double check with you.

Thanks again!!

794117

The first option is right. It happens at the server end when the JSP is requested.
Basically if you "view source" on the generated page, thats what your javascript has to work with.

I have some JSTL code inside the javascript string

And thats where things where going wrong. You're mixing it badly.

var str="";
str +='<c:forEach var="statesDetail" items="${statesList}">';
str +='<c:out value="${statesDetail}"></c:out>';
str +='</c:forEach>';

The "escape" characters that you put in there stopped the tags being evaluated.
Nesting the <c:forEach> loop in a String is incorrect.
I can understand the <c:out> tag being in the string - that bits fine.
But not the <c:forEach>

Anything that is not JSP/JSTL tags is just template text and gets output as is.

843840

Great!...i get it now...thanks for the prompt response.

843840

Hi,

I tried pasting a similar code into a javascript function like this:

var stateArray = new Array();
<c:forEach var="state" items="${stateList}" varStatus="status">
stateArray[${status.index}] = "${state}";
</c:forEach>

But I get an error message saying that:

"Character data not allowed in element head". What cud be the reason for this?

843840

That's an XML error. You apparently declared your JSP as XHTML instead of HTML.

You need to wrap special chatacters in a CDATA block. If you know XML, you should know how.

1 - 6

Added on Oct 18 2016

#legacy-documents, #linux, #linux-discussion

0 comments

5,552 views

Infrastructure Software

Using the Linux netem Module to Test Your Network Without Causing Adverse Effects

Learn how to test your application's behavior when your network is not providing optimal service, how to test whether your network monitoring scripts work, and how to simulate strange network behavior in a contained, controlled manner and without impacting the network adversely.

Using Oracle Enterprise Manager

How to Develop, Deploy, and Test Remote Beacons and Scripts

Using the `netem` Kernel Module

How Does `netem` Work?

Doing More with `netem`

Conclusion

See Also

About the Author

Comments

Post Details

Infrastructure Software

Using the Linux netem Module to Test Your Network Without Causing Adverse Effects

Learn how to test your application's behavior when your network is not providing optimal service, how to test whether your network monitoring scripts work, and how to simulate strange network behavior in a contained, controlled manner and without impacting the network adversely.

Using Oracle Enterprise Manager

How to Develop, Deploy, and Test Remote Beacons and Scripts

Using the netem Kernel Module

How Does netem Work?

Doing More with netem

Conclusion

See Also

About the Author

Comments

Post Details

Using the `netem` Kernel Module

How Does `netem` Work?

Doing More with `netem`