Skip navigation
1 2 3 4 Previous Next

caroljmcdonald

58 posts
This post will help you get started using Apache Spark DataFrames with Scala on the MapR Sandbox. The new Spark DataFrames API is designed to make big data processing on tabular data easier. A Spark DataFrame is a distributed collection of data organized into named columns that provides operations to filter, group, or compute aggregates, and can be used with Spark SQL. https://www.mapr.com/blog/using-apache-spark-dataframes-processing-tabular-data  
This tutorial will help you get started with Standalone Spark applications on the MapR Sandbox. https://www.mapr.com/products/mapr-sandbox-hadoop/tutorials/spark-tutorial  
Recommendation engines help narrow your choices to those that best meet your particular needs. In this post, we  
SQL will become one of the most prolific use cases in the Hadoop ecosystem, according to Forrester Research. Apache Drill is an open source SQL query engine for big data exploration. REST services and clients have emerged as popular technologies on the Internet. Apache HBase is a hugely popular Hadoop NoSQL database. This blog post discusses combining all of these technologies: SQL, Hadoop, Drill, REST with JSON, NoSQL, and HBase, by showing how to use the Drill REST API to query HBase and Hive. It also shares a simple jQuery client that uses the Drill REST API, with JSON as the data exchange, to provide a basic user interface https://www.mapr.com/blog/how-use-sql-hadoop-drill-rest-json-nosql-and-hbase-simple-rest-client#.VKyvdCvF90Y  

Backbone.js gives structure to web applications by providing models with key-value binding and custom events, collections with a rich API of enumerable functions, views with declarative event handling, and connects it all to your existing API over a RESTful JSON interface. JAX-RS provides a standardized API for building RESTful web services in Java. This example will show how to build a simple pet catalog application using backbone.js and JAX-RS.

http://java.dzone.com/sites/all/files/image002_5.jpg
http://java.dzone.com/sites/all/files/image004_0.jpg
The Backbone.js Client architecture
http://java.dzone.com/sites/all/files/image005_2.png
http://java.dzone.com/sites/all/files/image006_0.png
 

Models contain the  data and  defines the methods related to the data. You extend Backbone.Model with your domain-specific methods, and Model provides a basic set of functionality for managing changes.

Here is  the source code for Item.js:

    
var ItemModel = Backbone.Model.extend({
  urlRoot: "resources/items",

  defaults: {
  id: null,
  name: "",
description:"",
  }

});

Collections are ordered sets of models which synchronize with the server. Collections can trigger  events to be notified when any model in the collection has been changed, add, remove , to fetch  from the server.

http://java.dzone.com/sites/all/files/image007_0.png

Here is  the source code for  Items Collection Items.js:

    
var ItemList = Backbone.Collection.extend({
  model: ItemModel,
  url: "resources/items",
}
});

var Items = new ItemList();

The model property specifies the model class that the collection contains.

The url property (or function) on a collection references its location on the server.

Backbone.sync is the function that Backbone calls every time it attempts to read or save a model to the server. By default, it uses (jQuery/Zepto).ajax to make a RESTful JSON request.  The default sync handler maps Fetch,Save,Destroy CRUD requests  to REST using the collection url :

  • Save (new) ? create ? HTTP POST  /url
  • Fetch ? read ? GET  /url/id
  • Save ? update ? PUT  /url/id
  • Destroy ? delete ? DELETE  /url/id

The View

  • a view is bound to a model/collection
  • manipulates the model data in the DOM
  • delegates DOM events
  • Render function uses the model data to render the template

With the backbone view you can bind your view's render function to the model's "change" event , then everywhere that model data is displayed in the UI, it will be updated.

Here is  the source code for the  Item View ItemView.js:

window.ItemView = Backbone.View.extend({
    initialize: function () {
        this.render();
    },
    render: function () {
        $(this.el).html(this.template(this.model.toJSON()));
        return this;
    },
    events: {
        "change"        : "change",
        "click .save"   : "beforeSave",
        "click .delete" : "deleteItem"
    },
});

In the view you  override the render function, specify events, and in this example specify a template. In the code above , the initialize function  binds  the  View to the change events of its model, which call the render function when the event is fired. The render function  renders the view template from model data, and updates this.el with the new HTML.

this View  renders the  model data into HTML using a template engine.

    
  <script type="text/template" id="item-template">
  <h3>Name: {{name}}</h3>
  <p>{{description}}</p>
  </script>
  • model attributes: {"name":"Sneaky Cat","Description":"my cat  is so sneaky"}
  • render method: this.$el.html(this.template(this.model.toJSON()));
  • template output:   <h3>Name: Sneaky Cat</h3> <p>my cat is so sneaky</p>

The Catalog Items database table

The Catalog items are contained in a  database table:

  • Item. Contains the title and text for each item in the catalog.
CREATE TABLE item (
 id INT NOT NULL,
 productid VARCHAR(10) ,
 name VARCHAR(30) ,
 description VARCHAR(500) ,
 imageurl VARCHAR(155),
 imagethumburl VARCHAR(155),
 price DECIMAL(14,2) ,
 primary key (id)
);

The Model

What is FindBugs?

    FindBugsis an open source static analysis tool that analyzes Java class files

    http://findbugs.sourceforge.net/

    Developed from Research at University of Maryland, led by Bill Pugh

    Looks for  for programming defects based on  ~ 300 different bug patterns from real bugs

    bug patterns are grouped into categories: correctness, bad practice, performance…

    each pattern assigned a priority: high, medium or low.

    High-Medium priority have low false positive rates: effort was put into trying to ensure that issues reported as high or medium priority correctness issues were issues that had a low false positive rate, and that developers would be interested inexamining all issues reported as high or medium priority correctness issues, even in large code bases.

           

FindBugslooks for  Bugs  based on real bug Patterns:

     broad and common patterns:

     a read or write on a null pointer

     typos

     Methods whose return value should not be ignored

     Also specific bug patterns:

     Every Programming Puzzler

     Eclipse documented bug fixes

     Every chapter inEffective Java

     lots from Worse than failure: http://thedailywtf.com/

 

 

Misconceptions about Bugs:

    Programmers are smart

    Smart people don’t make dumb mistakes

    WRONG!

    Smart people make dumb mistakes

    Common errors:

    wrong booleanoperator, forgetting parentheses, etc. Misunderstood class or method !

 

Who uses Findbugs?

    Developed from Research at University of Maryland

    Google, Ebay, Sun, Wells Fargo…

    Bill Pugh , Professor from University of Maryland, spent a year sabbatical at Google  working Findbugs  into their development process:

    Google runs FindBugs over all Java code

    1800s issues identified, > 600 fixed.

    Ebay found 2 developers reviewing  Findbugs  was 10 times more effective than 2 testers

 

Some Bug Categories

• Correctness - the code is doing something wrong, you should look at it

• Bad practice - the code violates good practice

     Dodgy Code

     Concurrency

     Performance

• Security defect

 

Can you find the Bug below ?

publicStringsendMessage (User user, String body, Date time) {

    return sendMessage(user, body, null);

  }

publicStringsendMessage (User user, String body, Date time, List attachments) {

   String xml = buildXML (body, attachments);

   String response = sendMessage(user, xml);

   return response;

  }

This line causes an Infinite recursive loop. This bug is  high priority,  in the correctness category.

returnsendMessage(user, body, null);

 

Can you find the Bug below ?

publicStringfoundType() {

   return this.foundType();

}

Should be

publicStringfoundType() {

   return this.foundType;

}

 

This is another example of an infinite recursive loop. This bug is found often  in a decorator pattern when you forget to delegate to another method

 Findbugs found 5 infinite recursive loopsin JDK1.6.0-b13. Including this one written by Joshua Bloch

• Smart people make dumb mistakes

• 27 across all versions of JDK, 31 in Google’s Java code

• Embrace and fix your dumb mistakes!

 

Can you find the Bug?

if(name != null || name.length > 0)

 

Should be

if(name != null && name.length > 0)

this null pointer bug was found in com.sun.corba

 

Can you find the Bug?
privatefinal String _lock = "LOCK";
...
synchronized(_lock)
{
...
}

Constant Strings are shared (even private ones) across all other classes loaded by the JVM. This could lead to unexpected deadlocks in conjunction with other code. This bug was found in Jetty.

 

This null pointer bug was in Eclipse since 3.2:

if(adapters == null && adapters.length == 0)

     return;

•example of  statement orbranch that if executed guarantee that a null pointer exception will occur

but in this case adapters is probably never null, that’s why it didn’t get noticed.

     somemistakes don’t matter much, because  the impact of the mistake is minimal.

null pointer exceptions usually get noticed. A harder error to find is that it won’t return if the length is 0.

 

Can you find the bug?

try{ ... }

catch(IOException e) {

  newSAXException("Server side Exception:" + e);

}

The Exception is created and dropped rather than thrown, Should be :

  thrownew SAXException("Server side Exception:" + e);

 

Can you find the Bug?

publicstatic String getNameById(String userId) {

    String str = userId;

    str.replace(' ', '_');

    return str;

  }

 

Ignores the return value of the replace() method, Should be:

     str= str.replace(' ', '_');

 

A very common mistake is ignoring the return value on methods whose return value shouldn't be ignored.

• Strings are immutable, so functions like trim() and replace() return a new String.

 

What does this Print?

Integer one = 1;

Long addressTypeCode = 1L;

if(addressTypeCode.equals(one)) {

     System.out.println("equals");

} else {

     System.out.println("not equals");

}

 

It prints "not equals". According to the contract of equals(), objects of different classes should always compare as unequal

Incomparable equality:

     Using .equals to compare incompatible types

     Using .equals to compare arrays

     only checks if the same array

     Checking to see if a Set<Long> contains anInteger

     never found, even if the same integral value is contained in the map

     Calling get(String)on a Map<Integer,String>

     Returns false , not an error

     These bugs may be hard to find on your own

     Types not always explicit

     May be introduced by refactoring

           For example a Google refactoring changed a method to return byte[ ] rather than String

Bugs that Matter…

You can have mistakes which don't actually cause problems. Bugs that matter depend on the context.  Not every bug found matters, but the effective use of Findbugs can remove bugs cheaper than with other methods

What is the best way to use Findbugs?

You want to find an effective way to use static analysis to improve software quality. 5-10% of mistakes can be found with Findbugs, for some types of bugs it can find all of them.  Testing and running code in production will find the bugs that matter, the bugs that cause problems .  If you run Findbugs on code in production , then a lot of bugs have been fixed, if you run on new code your are more likely to find bugs. The reason to use findbugs is not to fix all your quality problems, but rather for the issues it does find,  finding and fixing with Findbugs is cheaper than finding with testing or causing a problem in production.

     Use Findbugs on new code to remove bugs cheaplybefore bugs are detected using more expensive techniques

     The best time to have a developer  look at a warning is when it is introduced, while the code is fresh in a developer’s head.

Expensive Mistakes are …

     Mistakes that fail silently

     Bugs that silently cause the wrong answer to be computed are the scariest bugs, because they will not be noticed easily .

     Mistakes that cause loss of money when they occur

     Mistakes that are hard to fix

Mistakes that matter ($$)  can be found by testing or eventually in production,  but it is cost effective to find bugs  before testing and production with findbugs.

 

Runtime exceptions can be your friend… runtime exceptions can point you right where the error is

If a mistake causes a runtime exception, then the mistake will be found and corrected .

Throwing a runtime exception is often a reasonable way to fail safely and report a failure.

Runtime exceptions represent conditions that reflect errors in your program's logic and cannot be reasonably recovered from

 

Can you find the Bug?

// calculate DR amount by aggregating CR amounts

BigDecimaldrAmount = new BigDecimal(0);

for(JournalEntry je: journalEntries)

     drAmount.add(je.getCrAmount());

     // persist to db

     getTrxnService().saveJournalEntry(id,

        drAmount, // aggregated amount

        true, // Debit

        "USD",

        "Revenue");

 

Ignores return value of BigDecimal.add(), should be

 drAmount=drAmount.add(je.getCrAmount());

This bug would have cost $ in production, but fixed at Google, within 30 minutes of being reported.

 

Can you find the Bug?

intvalue2;

Public boolean equals(Integer value1){

  returnvalue1== intValue() ;

}

publicInteger intValue() {

  returnvalue2;

}

 

Uses reference equality rather than .equals, should be:

Public boolean equals(Integer value1){

  returnvalue1.equals(intValue() );

}

For boxed primitives, == and != are computed using pointer equality, but <, <=, >, >= are computed by comparing unboxed primitive values. This can bite you on other classes (e.g., String), but boxed primitives is where people get bit

 

Concurrency Bugs

     Inconsistent synchronization –

     a lock is held   sometimes when field accessed

     Problems with wait/notify –

     e.g., call to wait() not in loop

     unsafe lazy initialization of static field

 

Can you find the bug?

synchronized(object) {

  if (<condition does not hold>) {

    object.wait();

  }

  //Proceed when condition holds

}

 

Should be:

synchronized(object) {

  while (<condition does not hold>) {

    object.wait();

  }

  //Proceed when condition holds

}

 

Use Java 5!  simplified concurrency , less likely to make these mistakes.

 

Running FindBugs

     Eclipse plugin

           http://findbugs.sourceforge.net/manual/eclipse.html

     Run with Hudson build

References:

     Findbugs home page

           http://findbugs.sourceforge.net/

     Bill Pugh Findbugs Devoxxtalk

           http://www.parleys.com/#id=2106&st=5

     Bill Pugh Oredev talk:

           http://oredev.org/2010/sessions/defective-java-mistakes-that-matter

            

This Pet Catalog app explains a web application that uses Wicket, JPA, GlassFish and MySQL. I took this example JSF 2.0, JPA, GlassFish and MySQL  http://java.sun.com/developer/technicalArticles/glassfish/GFandMySQL_Part2.html and modified it to use Wicket instead of JSF. 
 

Explanation of the usage of Wicket, Java Persistence APIs, Glassfish and MySQL in a sample Store Catalog Application

The image below shows the Catalog Listing page, which allows a user to page through a list of items in a store.

 
http://blogs.sun.com/carolmcdonald/resource/wicket_catalog.jpg




The Wicket Framework



The Wicket Web Framework provides a component-oriented programmatic manipulation of HTML markup. The Diagram below shows how the Wicket Framework fits into the MVC design pattern.


http://blogs.sun.com/carolmcdonald/resource/wickframeworkd.jpg

A Wicket component controller receives user input. The component uses the user input to interact with the model, to handle page navigation and events. The Model provide components with an interface to domain data. The Component view renders the User Interface HTML elements. The controller and view parts are combined into the Component  in Wicket.

http://blogs.sun.com/carolmcdonald/resource/wicketcomponent.jpg

Wicket Triad of HTML Templates, Components, and Model


The Wicket component, the model, and HTML template work together. The HTML template defines the static parts of the pages, components fill in the dynamic parts, and models are used by components to get the domain data for the dynamic parts.  Wicket matches wicket:id tags with components, Models are a reference to the data for the Java components.
http://blogs.sun.com/carolmcdonald/resource/wicketbinding.jpg

Wicket  HTML Templates


Wicket templates are written in plain HTML, consumable by standard HTML editors. Wicket uses HTML namespace standards to extend HTML with placeholders,  where Wicket components are hooked in, so that there is no custom syntax at all. Wicket matches wicket: id attributes and attaches Java components to the tags in which these attributes are defined.

Wicket Web pages are all Java classes that map to an HTML template, with the same name as the Java class.
http://blogs.sun.com/carolmcdonald/resource/wicketproject.jpg
 


The ListPage.html file holds the markup for the ListPage.java.  Wicket matches the Java page instance and the HTML template file with the same name in the same Java package.

           
Code Sample from:  ListPage.html

<table>
  <tr>
   <th> Name </th>
   <th> Photo</th>
   <th> Price </th>
  </tr>
  <tr wicket:id="rows">
     <td>
        <a wicket:id="details"href="#">
         <span wicket:id="name">name</span>
        </a>
     </td>
     <td> <img wicket:id="photo"/></td>
     <td><span wicket:id="price"></span></td>

   </tr>
</table>
<span wicket:id="pager">navigation controls here</span>



Pages are special top-level components that hold the root for component trees. Below is the component tree for the ListPage above:

http://blogs.sun.com/carolmcdonald/resource/componenttree.jpg
 
 

Wicket Components


Wicket matches the wicket:id="rows"attribute in the ListPage.html with the corresponing Java component in ListPage.java,  which is a DataViewComponent which has a DataProvider model as shown in the diagram below.
http://blogs.sun.com/carolmcdonald/resource/listpage.jpg

The corresponding ListPage.java code is shown below:

           
Code Sample from: ListPage.java


public class ListPage extends WebPage {

        // create the Model DataProvider
        IDataProvider itemDataProvider = new IDataProvider<Item>() {

            public Iterator iterator(int first, int count) {
                return itemController.findItemEntities(count, first).iterator();
            }
            public int size() {
                return itemController.getItemCount();
            }
            public IModel model(final Item object) {
                return new LoadableDetachableModel() {
                    @Override
                    protected Item load() {
                        return (Item) object;
                    }
                };
            }
            public void detach() {
            }
        };

       
// create the DataView component corresponding to the wicketid "rows" attribute in ListPage.html
        DataView dataView = new DataView<Item>("rows", itemDataProvider, ROWS_PER_PAGE) {

            @Override
            protected void populateItem(org.apache.wicket.markup.repeater.Item<Item> repItem) {
                Item item = (Item) repItem.getModelObject();
                repItem.setModel(new CompoundPropertyModel<Item>(item));
                repItem.add(ItemDetails.link("details", item));
                repItem.add(new Image("photo", new ResourceReference(this.getClass(), item.getImagethumburl()))); 
                repItem.add(new Label("price"));

            }
        };
       
// add the DataView component to the page
        add(dataView);
       
// create the PagingNavigator component corresponding to the "pager" attribute in ListPage.html
        PagingNavigator pager = new PagingNavigator("pager", dataView);
        add(pager);



DataView is a component which makes it simple to populate a Wicket RepeatingView from a database by utilizing IDataProvider to act as an interface between the database and the dataview.  A Wicket RepeatingView renders the components added to it from data from a collection of objects.

The  DataView instantiation passes to the constructor  the wicket id "rows" , the itemDataProvider, and the number of rows for paging, and
anonymously overrides the populateItem() method, which will be called for each Item object provided by itemDataProvider.  The populateItemmethod adds the child components ( Link, Image, Label)  with the Item  Model to the Dataview.
http://blogs.sun.com/carolmcdonald/resource/componenttree.jpg
The itemDataProviderIDataProvider provides the Pet Catalog  Item data to the DataView.  The IDataProvider iterator(int first, int count) method gets an iterator for the subset of total data.  The itemDataProvider iterator method calls the itemController.findItemEntities , which uses JPA to query the database and return a list of Itementities.  The itemDataProvider model method  is a Callback used by the consumer of this data provider to wrap objects retrieved from iterator(int, int) with a model (usually a detachable one).
http://blogs.sun.com/carolmcdonald/resource/wicketbinding.jpg





 

Using the Java Persistence API (JPA) with Wicket


The ItemJpaControllerfindItemEntitiesmethod is defined as shown below:

           
Code Sample from: ItemJpaController.java

public class ItemJpaController {

    public ItemJpaController() {
        emf = Persistence.createEntityManagerFactory("wicketCatalogPU");
    }
    private EntityManagerFactory emf = null;

    public EntityManager getEntityManager() {
        return emf.createEntityManager();
    }
    private List<Item> findItemEntities(boolean all, int maxResults, int firstResult) {
        EntityManager em = getEntityManager();
        try {
            Query q = em.createNamedQuery("Item.findAll");
            if (!all) {
                q.setMaxResults(maxResults);
                q.setFirstResult(firstResult);
            }
            return q.getResultList();
        } finally {
            em.close();
        }
    }



 

TheItemJpaController uses the Java Persistence API EntityManager Query object to return a list of items. The ItemJpaController calls Persistence.createEntityManagerFactorywhich gets an EntityManagerFactory  when it is instatiated. 
 

The Java Persistence Query APIs are used to create and execute queries that can return a list of results.  The JPA Query interface provides support for pagination via the setFirstResult() and setMaxResults() methods: q.setMaxResults(int maxResult) sets the maximum number of results to retrieve.q.setFirstResult(int startPosition) sets the position of the first result to retrieve.

In the code below, we show the Itementity class which maps to the  ITEM table that stores the item instances. This is a typical Java Persistence entity object. There are two requirements for an entity:
 
  1. annotating the class with an @Entityannotation.
  2. annotating   the primary key identifier with @Id
Because the fields name, description.... are basic mappings from the object fields to columns of the same name in the database table, they don't have to be annotated. 

           
Code Sample from: Item.java

@Entity
public class Item implements java.io.Serializable {

   
@Id
    private Integer id;

    private String name;   
    private String description;   
    private String imageurl;   
    private String imagethumburl; 
    private BigDecimal price;


    public Item() { }

    public String getName() {
        return name;
    }

    public void setName(String name) {
        this.name = name;
    }


    ...
}   


A Wicket PagingNavigatorcomponent  is used to provide navigation links to the next, previous, first and last page of catalog Items.  The PagingNavigator component maintains a complete page navigator, meant to be easily added to any PageableListView.

           
Code Sample from: ListPage.html

<span wicket:id="pager">navigation controls here</span>


Code below for adding creating a PagingNavigator for thedataView and adding it to the ListPage:

           
Code Sample from: ListPage.java


public class ListPageextends WebPage {


       
PagingNavigator pager = new PagingNavigator("pager", dataView);
        add(pager);
 



A Wicket Link component  is used to provide a link to click on to navigate to a page with the selected item details.

           
Code Sample from: ListPage.html

  <a wicket:id="details"href="#">
        <span wicket:id="name">name</span>
  </a>
 


Code below show the call to  the ItemDetails.link method to create and add the link:

           
Code Sample from: ListPage.java


public class ListPage extends WebPage {

       // create the DataView component corresponding to the wicketid "rows" attribute in ListPage.html
        DataView dataView = new DataView<Item>("rows", itemDataProvider, ROWS_PER_PAGE) {
            @Override
            protected void populateItem(org.apache.wicket.markup.repeater.Item<Item> repItem) {
           '    ...
               
// call ItemDetails to create the link component
                repItem.add(ItemDetails.link("details", item));
               ...

            }
        };
       
// add the DataView component to the page
        add(dataView);
 


Code below for creating a BookmarkablePageLinkfor adding to the dataView, clicking on this link will Navigate to the ItemDetails page, passing the selected Itemid as a parameter.

           
Code Sample from: ItemDetails.java

public class ItemDetails extends BasePage {

        public static BookmarkablePageLink<Void> link(final String name, final Item item) {
       
        final BookmarkablePageLink<Void> link = new BookmarkablePageLink<Void>(name,ItemDetails.class);

        if (item != null) {
            link.setParameter("itemid", item.getItemid());
            link.add(new Label("name", new Model<Item>(item)));
        }

        return link;
    }


The ItemDetails page, shown below, displays details about the selected Catalog Item:

           
Code Sample from: ItemDetails.html

<html xmlns:wicket="http://wicket.apache.org/">
    <head>
        <title></title>
        <link wicket:id='stylesheet'/>
    </head>
    <body>
        <span wicket:id='mainNavigation'/>
        <table>
            <tr>
                <td align="right">Name:</td>
                <td>
                    <span wicket:id="name">name </span>
                </td>
            </tr>
            <tr>
                <td align="right">Description:</td>
                <td> <span wicket:id = "description">
                  description
                    </span>
                </td>
            </tr>
            <tr>
                <td align="right">Photo:</td>
                <td> <img wicket:id="imagethumburl"/>   </td>
            </tr>
        </table>
    </body>
</html>
 



The ItemDetails constructor gets theitemdata, and adds Labels and a image, for the name, description and photo to the ItemDetails page.

           
Code Sample from: ItemDetails.java

public class ItemDetails extends BasePage {

    public ItemDetails(PageParameters params) {
        Item item = itemController.findItem(params.getString("itemid"));
        add(new Label("name", item.getName()));
        add(new Label("description", item.getDescription()));
        add(new Image("imagethumburl", new ResourceReference(this.getClass(),item.getImageurl())));
    }



 
  http://blogs.sun.com/carolmcdonald/resource/itemDetails.png


 

Hot Deployment with Wicket and Glassfish

  • Incremental compile of all Wicket  artifacts when you save.
  • Auto-deploy of all web artifacts

Conclusion
This concludes the sample application which demonstrates a pet catalog web application which uses  Wicket, JPA, GlassFish and MySQL.

 

Running the Sample Application

  1. If you haven't already done so, download and install NetBeans IDE , GlassFish , and MySQL Community Server . You can download and install GlassFish with NetBeans as a single bundle.
  2. Follow these instructions to setup Netbeans with the Wicket plugin.
  3. Download the sample code.

Create the Pet Catalog database

In order to run the sample code you first have to create the Pet Catalog database and fill in  the Item table.

  1. Start NetBeans IDE
  2. Ensure that GlassFish is registered in the NetBeans IDE, as follows: 
    • Click the Services tab in the NetBeans IDE.
    • Expand the Servers node. You should see GlassFish v2 in the list of servers. If not, register GlassFish v2 as follows: 
      • Right-click the Servers node and select Add Server. This opens an Add Server Instance wizard.
      • Select GlassFish v2 in the server list of the wizard and click the Next button.
      • Enter the location information for the server and click the Next button.
      • Enter the admin name and password and click the Finish button.
  3. Start the MySQL or Java DB database as follows: 
    • Click the Services tab in the NetBeans IDE.
    • Expand the databases node. You should see the Java DB database in the list of databases. If you have installed the MySQL server database, you should also see the MySQL database in the list of databases.. Note:  Java DB  comes bundled with Netbeans, you can  download MySQL separately.        
      http://weblogs.java.net/blog/caroljmcdonald/archive/databases.jpg
    • Right-mouse click on the Java DB or MySQL server database and select Start.
  4. If you installed MySQL, set the properties of the MySQL server database as follows: 
    • Right-click on the MySQL server database and select Properties. This opens the MySQL Server Properties dialog box, as shown in Figure 8.

               
      MySQL Server Basic Properties 
      Figure 8. MySQL Server Basic Properties
       
    • In the Basic Properties tab, enter the server host name and port number. The IDE specifies localhost as the default server host name and 3306 as the default server port number.
    • Enter the administrator user name, if not displayed, and the administrator password -- the default administrator password is blank.
    • Click the Admin Properties tab.
    • Enter an appropriate path in the Path/URL to admin tool field. You can find the path by browsing to the location of a MySQL Administration application such as the MySQL Admin Tool.
    • Enter an appropriate path in the Path to start command. You can find the path by browsing to the location of the MySQL start command. To find the start command, look for mysqld in the bin folder of the MySQL installation directory.
    • Enter an appropriate path in the Path to stop command field. You can find the path by browsing to the location of the MySQL stop command. This is usually the path to mysqladmin in thebin folder of the MySQL installation directory. If the command is mysqladmin, in the Arguments field, type-u root stop to grant root permissions for stopping the server. The Admin Properties tab should look similar to Figure 9.

               
      MySQL Server Administration Properties 
      Figure 9. MySQL Server Administration Properties
       
    • Click the OK button.
     
  5. Right-click on the MySQL server or Java DB database and select Start.
  6. Create the petcatalog database as follows: 
    • Right-mouse click on the Java DB or MySQL server database and select Create Database. This will open a create Database window.
    • Enter the database name catalog for Java DB or petcatalog for MySQL.
      http://weblogs.java.net/blog/caroljmcdonald/archive/createdbmysql.jpg

      For Java DB enter userid password app app as shown below:

      http://weblogs.java.net/blog/caroljmcdonald/archive/createdb.jpg
       Click O.K. to accept the displayed settings.
  7. Create the tables in the catalog database as follows: 
    • Underneath Databases you should see a database connection for the petcatalog database. For example MySQL:       
      http://weblogs.java.net/blog/caroljmcdonald/archive/mysqlconnection.jpg
      or Java DB:       
      http://weblogs.java.net/blog/caroljmcdonald/archive/derbyconnection.jpg
    • Right-mouse click on the petcatalog connection and select Connect.
    • Right-mouse click on the petcatalog connection and select Execute Command. This will open up a SQL command window.
    • Copy the contents of the catalog.sql file in theriapetcatalog\exercises\exercise0directory and paste the contents into the SQL command window, as shown in below:

        Creating Tables in the Database
    • Click the Run SQL icon Run SQL icon (Ctrl+Shift+E) above the SQL command window.
    • Note: It is ok to see this: "Error code -1, SQL state 42Y55: 'DROP TABLE' cannot be performed on 'ITEM' because it does not exist. Line 2, column 1" . This just means you are deleting a table that does not exist.  If you need to delete and recreate the tables you will not  see this message the second time.
  8. View the data in the Pet Catalog database Item table as follows:
     
    • Underneath Databases you should see a database connection for the petcatalog database. For example MySQL:       
      http://weblogs.java.net/blog/caroljmcdonald/archive/mysqlconnection.jpg
      or Java DB:       
      http://weblogs.java.net/blog/caroljmcdonald/archive/derbyconnection.jpg
    • If the database connection is broken like in the following diagram:
      http://weblogs.java.net/blog/caroljmcdonald/archive/brokenconnection.JPG
       
      • Right-mouse click on the petcatalog connection and select Connect. as shown below:
        http://weblogs.java.net/blog/caroljmcdonald/archive/mysqlconnect.JPG
      • if prompted for a password, for MySQL leave it blank, for JavaDB enter user app password app.
    • Expand the Tables node below the petcatalogdatabase in the Services window. You should see theitem table under the Tables node. You can expand theitem table node to see the table columns, indexes, and any foreign keys, as shown in below :        
      An Expanded Table Node 
      Figure 12. An Expanded Table Node
       
        You can view the contents of a table or column by right-clicking the table or column and selecting View Data as shown  below:

               
      Viewing the Contents of a Table 
      Figure 13. Viewing the Contents of a Table


  9. Follow these instructions to Create a JDBC Connection pool and JDBC resource.Name the  pool mysql_petcatalog_rootPool and the jndi resource jdbc/petcatalog. Note: you do not have to create a JDBC connection pool and resource if you use the Netbeans wizard to generate JPA entities from database tables as described in this article GlassFish and MySQL, Part 2: Building a CRUD Web Application With Data Persistence.

Running the Sample solution:

If you want to run the sample solution, you have to create the catalog database tables first as described above.

 
  1. If you haven't already download the sample code and start the NetBeans IDE. Unzip the catalog.zip file which you downloaded, this will create a catalog directory with the project code.
  2. Open thecatalog/setup/sun-resources.xml file and verify that the property values it specifies match those of the petcatalog database and jdbc resources you created. Edit the property values as necessary.

  3. Open the catalog project as follows: 
    • In NetBeans IDE, click Open Project in the File menu. This opens the Open Project dialog.
    • Navigate in the Open Project dialog to thecatalog  directory and click the Open Project button.
     
    In response, the IDE opens the catalog project.  You can view the logical structure of the project in the Projects window (Ctrl-1).
  4. Run the catalog by right-clicking on thecatalog project in the Projects window and selecting Run Project. The NetBeans IDE compiles the application, deploys it on Glassfish, and brings up the default page in your browser.  (at http://localhost:8080/catalog/).

 

For more information see the following resources:


 

Number 3 in the Top 10 most critical web application security vulnerabilitiesidentified by the Open Web Application Security Project (OWASP) is Malicious File Execution, which occurs when attacker's files are executed or processed by the web server. This can happen when an input filename is compromised or an uploaded file is improperly trusted.

Examples

  • file is accepted from the user without validating content
  • filename is accepted from the user
In the example below a file name is accepted from the user and appended to the server's filesystem path.
    
// get the absolute file path on the server's filesystem 
String dir = servlet.getServletContext().getRealPath("/ebanking")
// get input file name
String file = request.getParameter(“file”); 
// Create a new File instance from pathname string   
File f = new File((dir + "\\" + file).replaceAll("\\\\", "/")); 



If the filename was compromised to  ../../web.xml , it might allow access to web server properties

Malicious File Execution can result in:

  • files loaded from another server and executed within the context of the web server
  • modifying paths to gain access to directories on the web server
  • malicious scripts put into a directory with inadequate access controls

Protecting against Malicious File Execution

  • the Java EE Security Manager should be properly configured to not allow access to files outside the web root.
  • do not allow user input to influence the path name for server resources 
    • Inspect code containing a file open, include, create, delete...
  • firewall rules should prevent new outbound connections to external web sites or internally back to any other server. Or isolate the web server in a private subnet
  • Upload files to a destination outside of the web application directory. 
    • Enable virus scan on the destination directory.

Java specific Protecting against Malicious File Exection

Use the OWASP ESAPI  HTTPUtilities interface:

  • The ESAPI HTTPUtilities interface is a collection of methods that provide additional security related to HTTP requests, responses, sessions, cookies, headers, and logging.

    The HTTPUtilities getSafeFileUploads method uses the Apache Commons FileUploader to parse the multipart HTTP request and extract any files therein
     
        
    public class HTTPUtilities 
    
        public void getSafeFileUploads(java.io.File tempDir,
                                   java.io.File finalDir)
                            throws ValidationException
    
    http://blogs.sun.com/carolmcdonald/resource/esapihttputil.jpg

References and More Information:
 




OWASP Top 10 number 2: Injection Flaws

Number 2 in the Top 10 most critical web application security vulnerabilitiesidentified by the Open Web Application Security Project (OWASP) is Injection Flaws. Injection happens whenever an attacker's data is able to modify a query or command sent to a database, LDAP server, operating system or other Interpreter. Types of injections are SQL, LDAP, XPath, XSLT, HTML, XML, OS command... SQL injection and Cross-Site Scripting account for more than 80% of the vulnerabilities being discovered against Web applications (SANS Top Cyber Security Risks).
 

SQL Injection Example

use of string concantenation to build query: SQL Injection can happen with dynamic database queries concatenated with user supplied input, for example with the following query:
   
    
 "select * from MYTABLE where name=" + parameter
if the user supplies "name' OR 'a'='a' " as the parameter it results in the following:
   
    
"select * from MYTABLE where name= 'name' OR 'a'='a';
the OR 'a'='a' causes the where clause to always be true which is the equivalent of the following:
   
    
"select * from MYTABLE;
if the user supplies "name' OR 'a'='a' ; delete from MYTABLE" as the parameter it results in the following:
   
    
"select * from MYTABLE where name= 'name' OR 'a'='a'; delete from MYTABLE;
the OR 'a'='a' causes the where clause to always be true which is the equivalent of the following:
   
    
"select * from MYTABLE; delete from MYTABLE;
some database servers, allow multiple SQL statements separated by semicolons to be executed at once.

SQL Injection can be used to:
 
  • create , read , update, or delete database data

Protecting against SQL Injection

  • Don't concatenate user input data to a query or command! 
    • Use Query Parameter binding with typed parameters, this ensures the input data can only be interpreted as the value for the intended parameter so the attacker can not change the intent of a query.
  • Validate all input data to the application using white list (what is allowed) for type, format, length, range, reject if invalid. (see previous blog entry)
  • don't provide too much information in error messages (like SQL Exception Information, table names..) to the user.

Java specific Protecting against SQL Injection

Don't concatenate user input data to a query or command:

  • Don't do this with JDBC:   
        
    String empId= req.getParameter("empId") // input parameter
    String query = "SELECT * FROM Employee WHERE
                         id = '" + empId +"'"?

  • Don't do this with JPA:  
        

    q = entityManager.createQuery(
This and the next series of blog entries will highlight the Top 10 most critical web application security vulnerabilitiesidentified by the Open Web Application Security Project (OWASP).

You can use OWASP's WebGoatto learn more about the OWASP Top Ten security vulnerabilties. WebGoat is an example web application, which has lessons showing "what not to do code", how to exploit the code, and corrected code for each vulnerability.

http://blogs.sun.com/carolmcdonald/resource/300px-WebGoat-Phishing-XSS-Lesson.JPG


You can use the OWASP Enterprise Security API Toolkit to protect against the OWASP Top Ten security vulnerabilties.

http://blogs.sun.com/carolmcdonald/resource/550px-Esapi-before-after.JPG

The ESAPI Swingset is a web application which demonstrates the many uses of the Enterprise Security API.

http://blogs.sun.com/carolmcdonald/resource/swingset.jpg

OWASP Top 10 number 1: XSS = Cross Site Scripting

Cross Site Scripting (XSS) is one of the most common security problems in today's web applications. According to the SANS Top Cyber Security Risks, 60% of the total attack attempts observed on the Internet are against Web applications and SQL injection and Cross-Site Scripting account for more than 80% of the vulnerabilities being discovered. You are at risk of an XSS attack any time you put content that could contain scripts from someone un-trusted into your web pages.
There are 3 types of cross site scripting:
  • Reflected XSS: is when an html page reflects user input data, e.g. from HTTP query parameters or a HTML form, back to the browser, without properly sanitizing the response. Below is an example of this in a servlet:
        
     out.writeln(
    
Here is a review of some concurrency tips from Joshua Bloch, Brian Goetz and others.
 

Prefer immutable objects/data

Immutable objects do not change after construction. Immutable objects are simpler, safer, require no locks, and are thread safe. To make an object immutable don't provide setters/mutator methods, make fields private final, and prevent subclassing. If immutability is not an option, limit mutable state, less mutable state means less coordination.  Declare fields final wherever practical, final fields are simpler than mutable fields.
When threads share mutable data, each thread that reads or writes must coordinate access to the data. Failing to synchronize shared mutable data can lead to atomicity failures, race conditions, inconsistent state, and other forms of non-determinism. These erratic problems are among the most difficult to debug.

Limit concurrent interactions to well defined points, limit shared data, consider copying instead of sharing.
 

Threading risks for Web applications

A Servlet get, post, service method can be called for multiple clients at the same time. Multi-threaded Servlet Instance and Static variables are shared and therefore if mutable, access must be coordinated. Servlets are typically long-lived objects with a high thread load, if you over-synchronize performance suffers, try to either share immutable (final) data, or don  

JPA Performance, Don't Ignore the Database

Database Schema

Good Database schema design is important for performance. One of the most basic optimizations is to design your tables to take as little space on the disk as possible , this makes disk reads faster and uses less memory for query processing.

Data Types

You should use the smallest data types possible, especially for indexed fields. The smaller your data types, the more indexes (and data) can fit into a block of memory, the faster your queries will be.

Normalization

Database Normalization eliminates redundant data, which usually makes updates faster since there is less data to change. However a Normalized schema causes joins for queries, which makes queries slower, denormalization speeds retrieval. More normalized schemas are better for applications involving many transactions, less normalized are better for reporting types of applications.  You should normalize your schema first, then de-normalize later.  Applications often need to mix the approaches, for example use a partially normalized schema, and duplicate, or cache, selected columns from one table in another table. With JPA O/R mapping you can use the @Embedded annotation for denormalized columns to specify a persistent field whose @Embeddable type can be stored as an intrinsic part of the owning entity and share the identity of the entity.

http://blogs.sun.com/carolmcdonald/resource/normalization.jpg

 

Database Normalization and Mapping Inheritance Hiearchies

The Class Inheritance hierarchy shown below will be used as an example of JPA O/R mapping.
http://blogs.sun.com/carolmcdonald/resource/mappinginheritance.jpg

In the Single table per class mapping shown below, all classes in the hierarchy are mapped to a single table in the database. This table has a discriminator column (mapped by@DiscriminatorColumn), which identifies the subclass.  Advantages: This is fast for querying, no joins are required. Disadvantages:  wastage of space since all inherited fields are in every row, a deep inheritance hierarchy will result in wide tables with many, some empty columns.

http://blogs.sun.com/carolmcdonald/resource/singletableperclass.jpg

In the Joined Subclass mapping shown below, the root of the class hierarchy is represented by a single table, and each subclass has a separate table that only contains those fields specific to that subclass. This is normalized (eliminates redundant data) which is better for storage and updates. However queries cause joins which makes queries slower especially for deep hierachies, polymorphic queries and relationships.
http://blogs.sun.com/carolmcdonald/resource/joinedsubclass.jpg

In the Table per Class mapping (in JPA 2.0, optional in JPA 1.0),  every concreteclass is mapped to a table in the database and all the inherited state is repeated in that table. This is not normlalized, inherited data is repeated which wastes space.  Queries for Entities of the same type are fast, however  polymorphic queries cause unions which are slower.

http://blogs.sun.com/carolmcdonald/resource/tableperclass.jpg

Know what SQL is executed

You need to understand the SQL queries your application makes and evaluate their performance. Its a good idea to enable SQL logging, then go through a use case scenario to check the executed SQL.  Logging is not part of the JPA specification, With EclipseLink you can enable logging of SQL by setting the following property in the persistence.xml file:

    
<properties>
    <property name="eclipselink.logging.level" value="FINE"/>
</properties>


With Hibernate you set the following property in the persistence.xml file:

    
<properties>
    <property name="hibernate.show_sql" value="true" />
</properties>


Basically you want to make your queries access less data, is your application retrieving more data than it needs, are queries accessing too many rows or columns? Is the database query analyzing more rows than it needs? Watch out for the following:
  • queries which execute too often to retrieve needed data
  • retrieving more data than needed
  • queries which are too slow 
    • you can use EXPLAIN to see where you should add indexes

With MySQL you can use the slow query log to see which queries are executing slowly, or you can use the MySQL query analyzer to see slow queries, query execution counts, and results of EXPLAIN statements.

Understanding EXPLAIN

For slow queries, you can precede a SELECT statement with the keyword EXPLAIN  to get information about the query execution plan, which explains how it would process the SELECT,  including information about how tables are joined and in which order. This helps find missing indexes early in the development process.



http://blogs.sun.com/carolmcdonald/resource/fulltablescan.jpg
You should index columns that are frequently used in Query WHERE, GROUP BY clauses, and columns frequently used in joins, but be aware that indexes can slow down inserts and updates.

Lazy Loading and JPA

With JPA many-to-one and many-to-many relationships lazy load by default, meaning they will be loaded when the entity in the relationship is accessed. Lazy loading is usually good, but if you need to access all of the "many" objects in a relationship, it will cause n+1 selects where n is the number of "many" objects.


http://blogs.sun.com/carolmcdonald/resource/lazyloading.jpg

You can change the relationship to be loaded eagerly as follows :

http://blogs.sun.com/carolmcdonald/resource/eagerloading.jpg


However you should be careful with eager loading which could cause SELECT statements that fetch too much data. It can cause a Cartesian product if you eagerly load entities with several related collections.


If you want to override the LAZY fetch type for specific use cases, you can use Fetch Join. For example this query would eagerly load the employee addresses:
http://blogs.sun.com/carolmcdonald/resource/joinfetch.jpg
In General you should lazily load relationships, test your use case scenarios, check the SQL log, and use @NameQueries with JOIN FETCH to eagerly load when needed.

Partitioning

the main goal of partitioning is to reduce the amount of data read for particular SQL operations so that the overall response time is reduced

Vertical Partitioning  splits tables with many columns into multiple tables with fewer columns, so that only certain columns are included in a particular dataset, with each partition including all rows.

Horizontal Partitioning segments table rows so that distinct groups of physical row-based datasets are formed. All columns defined to a table are found in each set of partitions. An example of horizontal partitioning might be a table that contains historical data being partitioned by date.
http://blogs.sun.com/carolmcdonald/resource/partitioning.jpg

Vertical Partitioning


In the example of vertical partitioning below a table that contains a number of very wide text or BLOB columns that aren't referenced often is split into two tables with the most referenced columns in one table and the seldom-referenced text or BLOB columns in another.

By removing the large data columns from the table, you get a faster query response time for the more frequently accessed Customer data. Wide tables can slow down queries, so you should always ensure that all columns defined to a table are actually needed.
http://blogs.sun.com/carolmcdonald/resource/verticalpartitioning.jpg
The example below shows the JPA mapping for the tables above. The Customer data table with the more frequently accessed and smaller data types  is mapped to the Customer Entity, the CustomerInfo table with the less frequently accessed and larger data types is mapped to the CustomerInfo Entity with a lazily loaded one to one relationship to the Customer.

http://blogs.sun.com/carolmcdonald/resource/verticalpart2.jpg


Horizontal Partitioning

The major forms of horizontal partitioning are by Range, Hash, Hash Key, List, and Composite.

Horizontal partitioning can make queries faster because the query optimizer knows what partitions contain the data that will satisfy a particular query and will access only those necessary partitions during query execution. Horizontal Partitioning works best for large database Applications that contain a lot of query activity that targets specific ranges of database tables.
http://blogs.sun.com/carolmcdonald/resource/horizontalpartitioning.jpg

Hibernate Shards

Partitioning data horizontally into "Shards" is used by google,linkedin, and others to give extreme scalability for very large amounts of data. eBay "shards" data horizontally along its primary access path.

Hibernate Shardsis a framework that is designed to encapsulate support for horizontal partitioning into the Hibernate Core.
http://blogs.sun.com/carolmcdonald/resource/sharding.jpg

Caching

JPA Level 2 caching avoids database access for already loaded entities, this make reading reading frequently accessed unmodified entities faster, however it can give bad scalability for frequent or concurrently updated entities.

You should configure L2 caching for entities that are:
  • read often
  • modified infrequently
  • Not critical if stale
You should also configure L2 (vendor specific) caching for maxElements, time to expire, refresh... 

References and More Information:

JPA Best Practices presentation
MySQL for Developers Article
MySQL for developers presentation
MySQL for developers screencast
Keeping a Relational Perspective for Optimizing Java Persistence
Java Persistence with Hibernate
Pro EJB 3: Java Persistence API
Java Persistence API 2.0: What's New ?
High Performance MySQL book
Pro MySQL, Chapter 6: Benchmarking and Profiling
EJB 3 in Action
sharding the hibernate way
JPA Caching
Best Practices for Large-Scale Web Sites: Lessons from eBay





JPA Caching

JPA Level 1 caching

JPA has 2 levels of caching. The first level of caching is the persistence context.
http://blogs.sun.com/carolmcdonald/resource/lev1lev2.jpg
The JPA Entity Manager maintains a set of Managed Entities in the Persistence Context.
http://blogs.sun.com/carolmcdonald/resource/img26.jpg
The Entity Manager guarantees that within a single Persistence Context, for any particular database row, there will be only one object instance. However the same entity could be managed in another User's transaction, so you should use either optimistic or pessimistic locking  as explained in JPA 2.0 Concurrency and locking
http://blogs.sun.com/carolmcdonald/resource/img57.jpg
The code below shows that a find on a managed entity with the same id and class as another in the same persistence context , will return the same instance.

       
@Stateless public ShoppingCartBean implements ShoppingCart {

 @PersistenceContext EntityManager entityManager;

 public OrderLine createOrderLine(Product product,Order order) {
        OrderLine orderLine = new OrderLine(order, product);
        entityManager.persist(orderLine);  //Managed
        OrderLine orderLine2=entityManager.find(OrderLine,
                    orderLine.getId()));
        (orderLine == orderLine2  // TRUE
        return (orderLine);
    }

}

The diagram below shows the life cycle of an Entity in relation to the Persistent Context.
http://blogs.sun.com/carolmcdonald/resource/img28.jpg
The code below illustrates the life cycle of an Entity. A reference to a container managed EntityManager is injected using the persistence context annotation. A new order entity is created and the entity has the state of new. Persist is called, making this a managed entity. because it is a stateless session bean it is by default using container managed transactions , when this transaction commits , the order is made persistent in the database. When the orderline entity is returned at the end of the transaction it is a detached entity.
http://blogs.sun.com/carolmcdonald/resource/img29.jpg
The Persistence Context can be either Transaction Scoped-- the Persistence Context 'lives' for the length of the transaction, or Extended-- the Persistence Context spans multiple transactions. With a Transaction scoped Persistence Context, Entities are "Detached" at the end of a transaction.
http://blogs.sun.com/carolmcdonald/resource/img48.jpg
As shown below, to persist the changes on a detached entity, you call the EntityManager's merge() operation, which returns an updated managed entity, the entity updates will be persisted to the database at the end of the transaction.
http://blogs.sun.com/carolmcdonald/resource/img49.jpg
An Extended Persistence Context spans multiple transactions, and the set of Entities in the Persistence Context stay Managed. This can be useful in a work flow scenario where a "conversation" with a user spans multiple requests.
http://blogs.sun.com/carolmcdonald/resource/img51.jpg
The code below shows an example of a Stateful Session EJB with an Extended Persistence Context in a use case scenario to add line Items to an Order. After the Order is persisted in the createOrder method, it remains managed until the EJB remove method is called. In the addLineItem method , the Order Entity can be updated because it is managed, and the updates will be persisted at the end of the transaction.

http://blogs.sun.com/carolmcdonald/resource/img52.jpg
The example below contrasts updating the Order using a transaction scoped Persistence Context verses an extended Persistence context. With the transaction scoped persistence context, an Entity Manager find must be done to look up the Order, this returns a Managed Entity which can be updated. With the Extended Persistence Context the find is not necessary. The performance advantage of not doing a database read to look up the Entity, must be weighed against the disadvantages of memory consumption for caching, and the risk of cached entities being updated by another transaction.  Depending on the application and the risk of contention among concurrent transactions this may or may not give better performance / scalability.
http://blogs.sun.com/carolmcdonald/resource/img54.jpg

JPA second level (L2) caching

JPA second level (L2) caching shares entity state across various persistence contexts.
http://blogs.sun.com/carolmcdonald/resource/lev1lev2.jpg

JPA 1.0 did not specify support of a second level cache, however, most of the persistence providers provided support for second level cache(s). JPA 2.0 specifies support for basic cache operations with the new Cache API, which is accessible from the EntityManagerFactory, shown below:

http://blogs.sun.com/carolmcdonald/resource/img73.jpg
If L2 caching is enabled, entities not found in persistence context, will be loaded from L2 cache, if found.
http://blogs.sun.com/carolmcdonald/resource/img69.jpg
The advantages of L2 caching are:
  • avoids database access for already loaded entities
  • faster for reading frequently accessed  unmodified entities
The disadvantages of L2 caching are:
  • memory consumption for large amount of objects
  • Stale data for updated objects
  • Concurrency for write (optimistic lock exception, or pessimistic lock)
  • Bad scalability for frequent or concurrently updated entities

You should configure L2 caching for entities that are:
  • read often
  • modified infrequently
  • Not critical if stale
You should protect any data that can be concurrently modified with a locking strategy:
  • Must handle optimistic lock failures on flush/commit
  • configure expiration, refresh policy to minimize lock failures
The Query cache is useful for queries that are run frequently with the same parameters, for not modified tables.

 

The EclipseLink JPA persistence provider caching Architecture

The  EclipseLink caching Architecture is shown below.

http://blogs.sun.com/carolmcdonald/resource/img74.jpg
Support for second level cache in EclipseLink is turned on by default, entities read are L2 cached. You can disable the L2 cache. EclipseLink caches entities in L2, Hibernate caches entity id and state in L2. You can configure caching by Entity type or Persistence Unit with the following configuration parameters: 
  • Cache isolation, type, size, expiration, coordination, invalidation,refreshing
  • Coordination (cluster-messaging)
  • Messaging: JMS, RMI, RMI-IIOP,
This Pet Catalog app explains a web application that uses JSF 2.0, Java EE 6, GlassFish and MySQL. I took this example  GlassFish and MySQL, Part 2: Building a CRUD Web Application With Data Persistence and modified it to use some of the new features of JSF 2.0 and Java EE 6.

Explanation of the usage of JSF 2.0 and Java EE 6 in a sample Store Catalog Application

The image below shows the Catalog Listing page, which allows a user to page through a list of items in a store.

http://weblogs.java.net/blog/caroljmcdonald/archive/petcatjsf2.jpg

JSF 2.0 Facelets XHTML instead of JSP

For JSF 2.0, Facelets XHTML is the preferred way to declare JSF Web Pages. JSP is supported for backwards compatibility, but not all JSF 2.0 features will be available for views using JSP as their page declaration language.  JSF 2.0 Facelets has some nice features like templating (similar in functionality to Tiles) and composite components, which I'm not going to discuss here but you can read about that in this article: http://www.ibm.com/developerworks/java/library/j-jsf2fu2/index.htmland in this Tech Tip Composite UI Components in JSF 2.0.


The Catalog application's resources


JSF 2.0 standardizes how to define web resources. Resources are any artifacts that a component may need in order to be rendered properly -- images, CSS, or JavaScript files.  With JSF 2.0 you put resources in a resources directory or a subdirectory.
http://weblogs.java.net/blog/caroljmcdonald/archive/resources.jpg

In your Facelets pages, you can access css files with the  <h:outputStylesheet>,  javascript files with the <h:outputScript> , and images with the <h:graphicImage> JSF tags. The list.xhtml uses the <h:outputStylesheettag to load the styles.css stylesheet , and the<h:graphicImagetag to display images from the resources as shown below:

        
Code Sample from:  list.xhtml

<h:outputStylesheetname="css/styles.css" target="body"/>

<h:graphicImagelibrary="images" name="banner_logo.gif"  />   




The Catalog application uses a resource bundle to contain the static text and error messages used by the Facelets pages. Putting messages in a resource bundle makes it easier to modify and internationalize your Application text.  The messages are in a properties file in a java package directory.

        
Code Sample from:  messages.properties 

Title=Pet Catalog
Next=Next
Previous=Prev
Name=Name
 


The resource bundle is configured in the faces-config.xml File (you don't need any other configuration in the faces-config.xml for JSF 2.0, as explained later you no longer have to configure managed beans and navigation with XML).


        
Code Sample from:  faces-config.xml

<application>
    <resource-bundle>
        <base-name>web.WebMessages</base-name>
        <var>msgs</var>
      </resource-bundle>
</application>
 



The List.xhtml facelets page uses a JSF dataTable component to display a list of catalog items in an html table.  The dataTable component is useful when you want to show a set of results in a table. In a JavaServer Faces application, the UIData component (the superclass of dataTable)  supports binding to a collection of data objects. It does the work of iterating over each record in the data source. The HTML dataTable rendererdisplays the data as an HTML table.

In the list.xhtml web page the dataTable is defined as shown below:  (Note: Red colors are for Java EE tags, annotations code,  and Green is formy code or variables)

        
Code Sample from:  list.xhtml

<h:dataTablevalue='#{catalog.items}' var='row' border="1"
      cellpadding="2" cellspacing="0">



The value attribute of a dataTable tag references the data to be included in the table. The var attribute specifies a name that is used by the components within the dataTable tag as an alias to the data referenced in the value attribute of dataTable.  In the dataTable tag from the List.jsppage, the value attribute points to a list of catalog items. The var attribute points to a single item in that list. As the dataTable component iterates through the list, each reference to dataTableItempoints to the current item in the list.

JSF 2.0 Annotations instead of XML configuration

The dataTable's value is bound to the itemsproperty of the catalog managed bean. With JSF 2.0 managed beans do not have to be configured in the faces-config.xml file, you annotate the managed beans instead as shown below:

        
Code Sample from: Catalog.java


@ManagedBean
@SessionScoped
  public class Catalogimplements Serializable {




By convention, the name of a managed bean is the same as the class name, with the first letter of the class name in lowercase. To specify a managed bean name you can use the name attribute of the ManagedBean annotation, like this: @ManagedBean(name = "Catalog").


This CatalogManagedBean itemsproperty is defined as shown below:

        
Code Sample from: Catalog.java

    private List<Item> items = null;

    public List<Item> getItems() {
        if (items == null) {
            getPagingInfo();
            items = getNextItems(pagingInfo.getBatchSize(), pagingInfo.getFirstItem());
        }
        return items;
    }
   




The getItems() method returns a List of item objects. The JSF dataTable, supports data binding to a collection of data objects.  The dataTable object is modeled as a collection of row objects that can be accessed by a row index.  The APIs provide mechanisms to position to a specified row index, and to retrieve an object that represents the data that corresponds to the current row index.   

The Itemproperties name, imagethumburl, andprice  are displayed with the columncomponent:

        
Code Sample from: list.xhtml

<h:dataTablevar="row" value="#{catalog.items}">
  <h:column>
      <f:facet name="header">
          <h:outputText value="#{msgs.Name}"/>
      </f:facet>
      <h:commandLink action="#{catalog.showDetail(row)}" value="#{row.name}" />
  </h:column>
 
  <h:column>
      <f:facet name="header">
          <h:outputText value="#{msgs.Photo}"/>
      </f:facet>
     
<h:graphicImage library="images" name="#{row.imagethumburl}"/>
  </h:column>

  <h:column>
      <f:facet name="header">
          <h:outputTextvalue="#{msgs.Price}"/>
      </f:facet>
      <h:outputText value="#{row.price}"/>
  </h:column>
</h:dataTable>


The columntags represent columns of data in a dataTable component. While the dataTable component is iterating over the rows of data, it processes the UIColumn component associated with eachcolumntag for each row in the table.

The dataTable component  iterates through the list of items (catalog.items)  and displays the item (var="row") attribute value.Each time UIData iterates through the list of items, it renders one cell in each column.

The dataTable and column tags use facet to represent parts of the table that are not repeated or updated. These include headers, footers, and captions.

Java EE 6: JSF 2.0,  EJB 3.1, and Java Persistence API (JPA)  2.0

The CatalogManagedBean annotates the field private ItemFacade itemFacade;  with @EJB , which causes an itemFacadeEJB to be injected when the managed bean is instatiated. TheCataloggetNextItemsmethod calls the ItemFacadeStateless EJB which uses the Java Persistence API EntityManager Query object to return a list of items.
 

        
Code Sample from: Catalog.java

@ManagedBean
@SessionScoped
  public class Catalogimplements Serializable {

   
    @EJB
    private ItemFacade itemFacade;

    public List<Item>  getNextItems(int maxResults, int firstResult) { 
       return itemFacade.findRange(maxResults, firstResult);    
   }
   

EJB 3.1 No-interface local client View

With EJB 3.1, local EJBs  do not  have to a implement separate interface, that is, all public methods of the bean class are automatically exposed to the caller.

Simplified Packaging

With Java EE 6, EJBs can be directly packaged in a WAR file just like web components.
 
The ItemFacadeEJB uses the Java Persistence API EntityManager Query object to return a list of items. The ItemFacadeEJB annotates the field private EntityManagerem;  with @PersistenceContext , which causes an entity manager to be injected when it is instatiated.

        
Code Sample from: ItemFacade.java

@Stateless
public class ItemFacade{

   
    @PersistenceContext(unitName = "catalogPU")
    private EntityManagerem;

    public List<Item> findRange(int maxResults, int firstResult) {
        Query q = em.createQuery("select object(o) from Item as o");
        q.setMaxResults(maxResults);
        q.setFirstResult(firstResult);
        return q.getResultList();
    }
   

The Java Persistence Query APIs are used to create and execute queries that can return a list of results.  The JPA Query interface provides support for pagination via the setFirstResult() and setMaxResults() methods: q.setMaxResults(int maxResult) sets the maximum number of results to retrieve.q.setFirstResult(int startPosition) sets the position of the first result to retrieve.

In the code below, we show the Itementity class which maps to the  ITEM table that stores the item instances. This is a typical Java Persistence entity object. There are two requirements for an entity:
  1. annotating the class with an @Entityannotation.
  2. annotating   the primary key identifier with @Id
Because the fields name, description.... are basic mappings from the object fields to columns of the same name in the database table, they don't have to be annotated.  The O/R  relationships with Addressand Productare also annotated. For more information on defining JPA entities see Pro EJB 3: Java Persistence API book.

           
Code Sample from: Item.java

@Entity
public class Item implements java.io.Serializable {

   
@Id 
    private Integer id;

    private String name;   
    private String description;   
    private String imageurl;   
    private String imagethumburl; 
    private BigDecimal price;
    @ManyToOne
    private Addressaddress;
    @ManyToOne
    private Productproduct;

    
    public Item() { }
    

    public String getName() {
        return name;
    }

    public void setName(String name) {
        this.name = name;
    }
   

    ...
}   




The CatalogManagedBean pages through the list of Items by maintaining the PagingInfo.firstItem and PagingInfo.batchSize attributes and passing these as parameters to the  getNextItems(firstItem, batchSize) method. The catalog's scope  is defined with the annotation @SessionScoped, a JSF Managedbean with session scope will be stored in the session meaning that the bean's properties will stay alive for the life of the Http Session.


A JSF commandButtonis  used to provide a button to click on to display the next page of items.  The commandButtontag is used to submit an action event to the application. 

        
Code Sample from: list.xhtml

 <h:commandButtonaction="#{catalog.next}" value="#{msgs.Next}" />   

This commandButton action attribute references the catalogManaged bean next() method which calculates the next page's first row number  and returns a logical outcome String, which causes the list.xhtml page to display the next page's list . The catalognextmethod is defined as shown below:

        
Code Sample from: catalog.java

   public String next() {
       if (firstItem + batchSize < itemCount()) {
           firstItem += batchSize;
       }
       return "list";
   }

JSF 2.0 Simplified Navigation


The JavaServer Faces 2.0  NavigationHandlerconvention adds .xhtml to the logical outcome of the action method (in this example list) and loads that file, in this case, it loads the list.xhtml page after this method returns. If the action doesn't begin with a forward slash (/), JSF assumes that it's a relative path.  You can specify an absolute path by adding the slash like this "/items/list".


A JSF commandLinkis  used to provide a link to click on to display a page with the item details. This commandLink actionattribute  references ThecatalogshowDetail()method:

        
Code Sample from: list.xhtml

   <h:column>
       <f:facet name="header">
          <h:outputText value="Name"/>
       </f:facet>
       <h:commandLink action="#{catalog.showDetail(row)}" value="#{row.name}"/>   
   </h:column>

With JSF 2.0 you can now specify parameters in method expressions. The dataTable row object associated with the selected link is passed as a parameter in the  "#{catalog.showDetail(row)}"method expression.
The Catalog showDetail()method  gets the item data from the input parameter, and returns a string which causes thedetail.xhtml page to display the item details :

        
Code Sample from: Catalog.java

    public String showDetail(Item item) {
        this.item = item;
        return "detail";
    }

The JavaServer Faces NavigationHandleradds .xhtml to the logical outcome of the action, detailand loads that file. In this case, the JavaServer Faces implementation loads the detail.xhtml page after this method returns.

The detail.xhtml uses the outputTextcomponent to display the catalogManagedBean's itemproperties:

        
Code Sample from: detail.xhtml

    <h:outputTextvalue="#{catalog.item.name}" title="Name" />
    <h:outputText value="#{catalog.item.description}" title="Description"/>
    <h:graphicImage library="images" name="#{catalog.item.imageurl}" title="Imageurl" />

    <h:outputText value="#{
catalog.item.price}" title="Price" />
    <h:outputText value="#{
catalog.item.address.city}" title="Address" />
    <h:outputText value="#{
catalog.item.contactinfo.email}" title="Address"/>  

     http://weblogs.java.net/blog/caroljmcdonald/archive/showdetail.jpg


GlassFish v3 is a lightweight server

  • OSGi-based; Embedded API; RESTful admin API; Lightweight and fast startup;
  • iterative development cycle "edit-save-refresh browser": 
    • Incremental compile of all JSF 2.0 artifacts when you save.
    • Auto-deploy of all web or Java EE 6 artifacts
  • Session retention: maintain sessions across re-deployments


Conclusion
This concludes the sample application which demonstrates a pet catalog web application which uses Java EE 6, GlassFish v3 and MySQL.

Running the Sample Application

  1. Download and install NetBeans IDE 6.8 M1 with GlassFish v3 b57 (Glassfish v3 preview is Java EE 6 Preview) , and MySQL Community Server .
  2. Follow these instructions to set up a  jdbc-driver for MySQL. (Normally this is already setup with Glassfish, but I got an errror message with Glassfish v3 b57 that it was missing)
  3. Download the sample code. Unzip the catalog.zip file which you downloaded, this will create a catalog directory with the project code.

Create the Pet Catalog database

In order to run the sample code you first have to create the Pet Catalog database and fill in  the Item table.

  1. Start NetBeans IDE
  2. Ensure that GlassFish is registered in the NetBeans IDE, as follows: 
    • Click the Services tab in the NetBeans IDE.
    • Expand the Servers node. You should see GlassFish v2 in the list of servers. If not, register GlassFish v2 as follows: 
      • Right-click the Servers node and select Add Server. This opens an Add Server Instance wizard.
      • Select GlassFish v2 in the server list of the wizard and click the Next button.
      • Enter the location information for the server and click the Next button.
      • Enter the admin name and password and click the Finish button.
  3. Start the MySQL or Java DB database as follows: 
    • Click the Services tab in the NetBeans IDE.
    • Expand the databases node. You should see the Java DB database in the list of databases. If you have installed the MySQL server database, you should also see the MySQL database in the list of databases.. Note:  Java DB  comes bundled with Netbeans, you can  download MySQL separately.         
      http://weblogs.java.net/blog/caroljmcdonald/archive/databases.jpg
    • Right-mouse click on the Java DB or MySQL server database and select Start.
  4. If you installed MySQL, set the properties of the MySQL server database as follows: 
    • Right-click on the MySQL server database and select Properties. This opens the MySQL Server Properties dialog box, as shown in Figure 8.
       
                
      MySQL Server Basic Properties 
      Figure 8.MySQL Server Basic Properties
       
    • In the Basic Properties tab, enter the server host name and port number. The IDE specifies localhost as the default server host name and 3306 as the default server port number.
    • Enter the administrator user name, if not displayed, and the administrator password -- the default administrator password is blank.
    • Click the Admin Properties tab.
    • Enter an appropriate path in the Path/URL to admin tool field. You can find the path by browsing to the location of a MySQL Administration application such as the MySQL Admin Tool.
    • Enter an appropriate path in the Path to start command. You can find the path by browsing to the location of the MySQL start command. To find the start command, look for mysqld in the bin folder of the MySQL installation directory.
    • Enter an appropriate path in the Path to stop command field. You can find the path by browsing to the location of the MySQL stop command. This is usually the path to mysqladmin in thebin folder of the MySQL installation directory. If the command is mysqladmin, in the Arguments field, type-u root stop to grant root permissions for stopping the server. The Admin Properties tab should look similar to Figure 9.
       
                
      MySQL Server Administration Properties 
      Figure 9.MySQL Server Administration Properties
       
    • Click the OK button.
     
  5. Right-click on the MySQL server or Java DB database and select Start.
  6. Create the petcatalog database as follows: 
    • Right-mouse click on the Java DB or MySQL server database and select Create Database. This will open a create Database window.
    • Enter the database name catalog for Java DB or petcatalog for MySQL.
      http://weblogs.java.net/blog/caroljmcdonald/archive/createdbmysql.jpg

      For Java DB enter userid password app app as shown below:

      http://weblogs.java.net/blog/caroljmcdonald/archive/createdb.jpg
       Click O.K. to accept the displayed settings.
  7. Create the tables in the catalog database as follows: 
    • Underneath Databases you should see a database connection for the petcatalog database. For example MySQL:       
      http://weblogs.java.net/blog/caroljmcdonald/archive/mysqlconnection.jpg
      or Java DB:       
      http://weblogs.java.net/blog/caroljmcdonald/archive/derbyconnection.jpg
    • Right-mouse click on the petcatalog connection and select Connect.
    • Right-mouse click on the petcatalog connection and select Execute Command. This will open up a SQL command window.
    • Copy the contents of the catalog.sql file in thecatalog directory and paste the contents into the SQL command window, as shown in below:
       
         Creating Tables in the Database
    • Click the Run SQL icon Run SQL icon (Ctrl+Shift+E) above the SQL command window.
    • Note: It is ok to see this: "Error code -1, SQL state 42Y55: 'DROP TABLE' cannot be performed on 'ITEM' because it does not exist. Line 2, column 1" . This just means you are deleting a table that does not exist.  If you need to delete and recreate the tables you will not  see this message the second time.
  8. View the data in the Pet Catalog database Item table as follows:
     
    • Underneath Databases you should see a database connection for the petcatalog database. For example MySQL:       
      http://weblogs.java.net/blog/caroljmcdonald/archive/mysqlconnection.jpg
      or Java DB:       
      http://weblogs.java.net/blog/caroljmcdonald/archive/derbyconnection.jpg
    • If the database connection is broken like in the following diagram:
      http://weblogs.java.net/blog/caroljmcdonald/archive/brokenconnection.JPG
       
      • Right-mouse click on the petcatalog connection and select Connect. as shown below:
        http://weblogs.java.net/blog/caroljmcdonald/archive/mysqlconnect.JPG
      • if prompted for a password, for MySQL leave it blank, for JavaDB enter user app password app.
    • Expand the Tables node below the petcatalogdatabase in the Services window. You should see theitem table under the Tables node. You can expand theitem table node to see the table columns, indexes, and any foreign keys, as shown in below :        
      An Expanded Table Node 
      Figure 12. An Expanded Table Node
       
         You can view the contents of a table or column by right-clicking the table or column and selecting View Data as shown  below:
       
                
      Viewing the Contents of a Table 
      Figure 13.Viewing the Contents of a Table


  9. Follow these instructions to Create a JDBC Connection pool and JDBC resource. Name the  pool mysql_petcatalog_rootPool and the jndi resource jdbc/petcatalog. Note: you do not have to create a JDBC connection pool and resource if you use the Netbeans wizard to generate JPA entities from database tables as described in this article GlassFish and MySQL, Part 2: Building a CRUD Web Application With Data Persistence.
  10. Open the catalog/setup/sun-resources.xml file and verify that the property values it specifies match those of the petcatalog database and jdbc resources you created. Edit the property values as necessary.

Running the Sample solution:

If you want to run the sample solution, you have to create the catalog database tables first as described above.
 
  1. Open the catalog project as follows: 
    • In NetBeans IDE, click Open Project in the File menu. This opens the Open Project dialog.
    • Navigate in the Open Project dialog to thecatalog  directory and click the Open Project button.
     
    In response, the IDE opens the catalog project.  You can view the logical structure of the project in the Projects window (Ctrl-1).
  2. Run the catalog by right-clicking on thecatalog project in the Projects window and selecting Run Project. The NetBeans IDE compiles the application, deploys it on Glassfish, and brings up the default page in your browser.  (at http://localhost:8080/catalog/).

For more information see the following resources:

Optimistic Concurrency


Optimistic locking lets concurrent transactions process simultaneously, but detects and prevent collisions, this works best for applications where most concurrent transactions do not conflict. JPA Optimistic locking allows anyone to read and update an entity, however a version check is made upon commit and an exception is thrown if the version was updated in the database since the entity was read.  In JPA for Optimistic locking you annotate an attribute with @Version as shown below:

    
public class Employee {
    @ID int id;
    @Version int version;

The Version attribute will be incremented with a successful commit. The Version attribute can be an int, short, long, or timestamp.  This results in SQL like the following:

    

Filter Blog

By date: