This content has been marked as final. Show 5 replies
user13407224 wrote:Hi Robert,
We have a product which has long use Berkeley DB JE for its storage layer. We are now experimenting with JE HA, with nodes in a replication group connecting over the public internet.
Note that for us, writes are rare, and we believe write performance is unlikely to be a bottleneck.
One particular issue is controlling access to the ports used for replication. Is there any sensible way to add authentication to the replication protocol? Or do we just need to configure firewalls to restrict access only to the other nodes by their IP addresses? (That would be somewhat unfortunate as we hope to make the replication group easily extensible.) What do other groups using JE HA over the public internet do?
You're correct that we don't have authentication in the HA protocol. It's assumed that all the nodes reside in a "safe" environment (i.e. behind a firewall in a data center, or at least with secure WAN links between nodes). In your case, would it be possible to use ssh tunnels for the specific HA ports? Or perhaps to set up a VPN for the protocol? Your approach of using firewalls and only allowing specific ipaddrs seems reasonable too.
Regarding authentication: we've thought about it, but we have no immediate plans for it.
Thanks. Tunneling or VPN sound like the easiest solutions, though we were hoping to avoid significant configuration outside the server itself.
We've had a cursory look through the JE HA code to see if we could easily add authentication ourselves. We suspect that it may be enough to introduce modifications to two classes: com.sleepycat.je.rep.impl.TextProtocol.MessageExchange for the client side and com.sleepycat.je.rep.utilint.ServiceDispatcher for the server side. Can you tell us if that is clearly right, or if there is something obvious we are missing?
I would say yes, you're on the right track having found those two classes. Nothing obvious that I can think of that you're missing. Of course without going through a full analysis and design I can't be sure there aren't some other related pieces that would have to come into the picture. But it seems like the right place to start looking.
I was able to modify the JE source to make replication use SSL if a system property is set (-Dje.rep.useSSL=true). The approach was to conditionally replace uses of Socket with SSLSocket (from the JDK), and SocketChannel/ServerSocketChannel with SSLSocketChannel/SSLServerSocketChannel. The latter classes are not in the JDK but can be implemented.
Here are more specifics:
1) Implement an SSLSocketChannel and an SSLServerSocketChannel that wrap SocketChannel and ServerSocketChannel, respectively. (I started with the SSLEngineManager in http://www.amazon.com/Fundamental-Networking-Java-Esmond-Pitt/dp/1849965455 and wrapped them with my own implementations of the channels).
2) In ServiceDispatcher.java, in the ctor conditionally create an SSLServerSocketChannel that wraps the original ServerSocketChannel if the system flag is set. Registration with any selectors must occur on the original channel and not the SSL version. The attachment to the selector key is modified to be a composite of the SSLSocketChannel and a buffer, rather than just a buffer. This is so when we call processRead() we can get back to the SSLSocketChannel (since registration with the selector is on the original SocketChannel).
3) In TextProtocol.java, in messageExchange() conditionally create an SSLSocket instead of a Socket if the system flag is set. Also call startHandshake() on the SSLSocket.
4) In Replica.java, in createReplicaFeederChannel(), conditionally create a SSLSocketChannel that wraps the original SocketChannel if the system flag is set.
5) In Acceptor.java, Learner.java, GroupService.java, TextProtocol.java and ServiceDispatcher.java, replace calls to socket.getInputStream()/socket.getOutputStream() with Channels.newInputStream(channel)/Channels.newOutputStream(channel).