1 2 3 Previous Next

evanx

129 posts
evanx

What is Redis? Blog

Posted by evanx Dec 31, 2014
Redis is good for prototyping, shared memory, messaging, caching and maximum performance. It might be used orthogonally and/or complementary to your SQL relational store, and/or NoSQL document store. For example, Redis might be used to cache dimensional aggregates of relational data for analytical purposes. It's use-case as "persistent shared memory" is important especially for microservices and distributed systems in general. I spent the last few weeks of this holiday season applying Redis to some use-cases. Herewith my write up for my colleagues on Redis. You can find the original (and likely updated) article on my github wiki: https://github.com/evanx/vellum/wiki/redis_intro. So let's get into it... Redis is a persistent in-memory key-value store. It was developed by Salvatore Sanfilippo. (Thank you so much!) http://upload.wikimedia.org/wikipedia/en/thumb/6/6b/Redis_Logo.svg/500px-Redis_Logo.svg.png The values of keys can be strings, lists, sets, sorted sets or hash tables. - Hash tables can store structured data, i.e. different fields of a record (or columns of a row). - Lists are queues with left and right sides (or head and tail), from which members can be pushed and popped - Sets have unique members, i.e. no duplicates. - Sorted sets' members have a "score" which is a floating point number. Members are ranked by this score, and/or the member (string) in the case of equal scores. They can be fetched by rank, or a range of their scores. All scores might be set to 0 to rank via the member string lexographically. Keys, and "members" of the above data types, are strings. However strings are "binary-safe" and so can be any type of data. Often strings (as keys, values and members) are stringified integer ids, or serialized objects e.g. JSON. Incidently, there is an operation to treat a string as an integer to be incremented - e.g. for generating sequential ids. The whole "database" is stored in memory. This deliberate "feature" gives it fantastic performance compared to on-disk databases. But it's also its "bug" that limits the database size. Considering this limitation, trimming the data set is critical - i.e. removing older data we don't actually need in memory for our use-case. (This might be archived into a disk-based database.) Besides snapshots, Redis has an "append-only log" which is configurable to sync to disk every second, or after every write (slow). This enables one to trade off the level of performance vs durability, e.g. potentially losing the last second of writes in the event of a power failure or crash. Actually Redis has various use-cases, besides a database per se: - memory cache in place of memcached: Redis supports expiry times on keys for this purpose. - message queue: Redis' list data type can be used as a queue, since it offers blocking pop operations. Redis also supports pubsub i.e. for asynchronous decoupled message passing. An important use-case especially for microservices and distributed systems in general, is using Redis for "persistent shared memory." - "persistent" because the data is still available when our application is restarted, or indeed Redis is restarted e.g. after reboot or power failure. - "shared" because it's a server which can be accessed remotely by multiple application instances. Redis is typically used to share data required by microservices running in different application containers, which is like different hosts, with different filesystems. For web applications, user session state might be stored in Redis, so that it is accessible by various microservices, e.g. by a session ID which contained in the HTTP request. (Authentication and authorisation could happen before the request reaches this service.) If not using Redis, one might use a RDBMS for persistent session state and queues. However extra management and integration effort is required when using RDBMS and SQL, compared to Redis, for such use-cases. The killer feature is that Redis might be 100x or 500x faster than an RDBMS, considering the latency of hard disk access compared to RAM. In general, web developers are increasingly finding NoSQL technologies attractive, in terms of ease of integration, performance and/or scalability. Nevertheless if one's data is relational, with foreign key relationships that must exist, and strict ACID compliance is required, then clearly an RDBMS is an appriopriate technology. However, a mix of SQL/NoSQL technologies is potentially advantageous. Redis is good for prototyping, shared memory, messaging, caching and maximum performance. It might be used orthogonally and/or complementary to your SQL relational store, and/or NoSQL document store. For example, Redis might be used to cache dimensional aggregates of relational data for analytical purposes. For comparative information on various NoSQL databases, see http://kkovacs.eu/cassandra-vs-mongodb-vs-couchdb-vs-redis. If you want to play with Redis, it's easy to install and start without any configuration required. Then just use the `redis-cli` command-line. See http://redis.io.
evanx

Password Rehash Blog

Posted by evanx Apr 30, 2013
evanx

Password Salt Blog

Posted by evanx Jan 24, 2013
evanx

Timestamped Millis Blog

Posted by evanx Nov 21, 2012
evanx

Deque shredder Blog

Posted by evanx Aug 1, 2012
evanx

Timestamped Deque Blog

Posted by evanx Jun 27, 2012
evanx

Counter map Blog

Posted by evanx Jun 7, 2012
evanx

The Prodigal Android Blog

Posted by evanx Aug 18, 2010

I enjoyed reading the Android=Java blog just now, and agree with Osvaldo's sentiments.

Android is clearly a Java knock-off - using javac and Apache class libraries, but implementing a new VM  designed for better performance in the resource-constrained environments of smartphones - hence the dex class file format. Otherwise they could have adopted JamVM. What approach would the reader take in their situation?

I thought that the Java language was an open standard, allowing clean-room implementations of the compiler, runtime libraries and VM? And if your implementation doesn't pass the TCK, then sorry you can't call it Java(tm), but good luck to you and yours.

Compared to what Microsoft did with .NET, Google took a very sensible route by not developing a new java'esque language and toolchain - but rather leveraging existing Java skills and familiar tools to hasten adoption. This is better for us Java developers than an unfamiliar language, IDE etc?

Isn't the problem that Sun hasn't really innovated  in the smartphone space, as Google has done so impressively with Android? While Google was leveraging opensource Linux and Java goodies,  and building a better VM for smartphones,  ironically Sun was off building a new language and toolset (JavaFX).

The Java trademark and promise of compatibility is great, but  Android is great too apparently - according to developers and the market.

In fairness, I don't see why Oracle should win this lawsuit, because there's no copyright infringment (in Apache Harmony), no infringment of the Java trademark (it's Android not JavaTM), and software patents are (as always) ridiculous.  So maybe Google is taking this opportunity to stand up against software patents? Well in my book, fighting software patents is always to be lauded...

 

evanx

Betwixt the Brackets Blog

Posted by evanx Jul 15, 2010

So we need to convert objects into XML and back again, eg. to store some data in the database in XML format,  because otherwise maybe  we just gonna have too many tables and joins and what-not.

Actually this is for a PNR, which is "Person Name Record" or something of that nature, as used in the travel industry for any booking eg. flights but also car hire, hotels, cruises etc. The other columns in the PNR table will be search terms for an agent to find a PNR, but otherwise all the gazillion details  that passengers seem to require are stuffed into the XML, from meal preferences, seat preferences, who's got whose baby on their lap, and when was this baby born, not to mention various comments and remarks communicated betwixt the airline and  the agents via this PNR.

Having used XStream before - which i must say is really great - i thought, let's try getting in and amongst Betwixt,  not least because most of our third-party jars are from the Apache behemoth.

For a start, lets say we have an entity as follows.

public class Traveler extends IdEntity {    
    @Id
    @GeneratedValue(strategy = GenerationType.AUTO)
    Long id;

    String firstNames;

    String surname;

    String email;

    String telephone;
    
    @Enumerated(EnumType.STRING)
    Salutation salutation;

    @Temporal(TemporalType.DATE)
    Date dateOfBirth;
    ...
}

So we write and read as follows.

    @Test
    public void testWrite() throws Exception {
        StringWriter outputWriter = new StringWriter();
        BeanWriter beanWriter = new BeanWriter(outputWriter);
        beanWriter.getXMLIntrospector().getConfiguration().setAttributesForPrimitives(false);
        beanWriter.getXMLIntrospector().getConfiguration().setTypeBindingStrategy(
            new CustomTypeBindingStrategy());
        beanWriter.getBindingConfiguration().setMapIDs(false);
        beanWriter.getBindingConfiguration().setObjectStringConverter(
            new CustomObjectStringConverter());
        beanWriter.enablePrettyPrint();
        Traveler traveler = new TravelerBean();
        traveler.setDateOfBirth(new GregorianCalendar(1969, 12, 31).getTime());
        traveler.setEmail("test@test.com");
        traveler.setSalutation(Salutation.MR);
        beanWriter.write("traveler", traveler);
        String xml = outputWriter.toString();
        outputWriter.close();
        System.out.println(xml);
        read(xml);
    }

    protected void read(String xml) throws Exception {
        StringReader xmlReader = new StringReader(xml);
        BeanReader beanReader = new BeanReader();
        beanReader.getXMLIntrospector().getConfiguration().setAttributesForPrimitives(false);
        beanReader.getXMLIntrospector().getConfiguration().setTypeBindingStrategy(
            new CustomTypeBindingStrategy());
        beanReader.getBindingConfiguration().setMapIDs(false);
        beanReader.getBindingConfiguration().setObjectStringConverter(
            new CustomObjectStringConverter());
        beanReader.registerBeanClass("traveler", Traveler.class);
        System.out.println(beanReader.parse(xmlReader));
    }

Googling the interwebs, we got the hint to use the following custom convertors and strategies eg. to handle enum's and what-not.

public class CustomObjectStringConverter extends ObjectStringConverter {

    final DateFormat dateFormat = new SimpleDateFormat("dd/MM/yyyy");

    @Override
    public String objectToString(Object object, Class type, Context context) {
        if (object == null) {
            return null;
        }
        if (type == Date.class) {
            return dateFormat.format((Date) object);
        }
        if (type.getSuperclass() == Enum.class) {
            return object.toString();
        }
        return super.objectToString(object, type, context);
    }

    @Override
    public Object stringToObject(String string, Class type, Context context) {
        if (string == null || string.length() == 0) {
            return null;
        }
        if (type == Date.class) {
            return formatDate(string);
        }
        if (type.getSuperclass() == Enum.class) {
            return Enum.valueOf(type, string);
        }
        return super.stringToObject(string, type, context);
    }

    protected Date formatDate(String string) {
        try {
            return dateFormat.parse(string);
        } catch (ParseException e) {
            throw new IllegalArgumentException(string, e);
        }
    }
}

What we probably should do above is check for annotations like @Temporal to see if our Date property is a date only, or a timestamp.

By the way, we can set an extended PropertySuppressionStrategy to suppress writing properties, and for reading XML, ElementSuppressionStrategy etc.

Finally, in order to make enum's work, we specify a BindingType.PRIMITIVE as follows.

public class CustomTypeBindingStrategy extends TypeBindingStrategy {

    @Override
    public BindingType bindingType(Class type) {
        if (isPrimitive(type)) {
            return BindingType.PRIMITIVE;
        }
        return BindingType.COMPLEX;
    }

    protected boolean isPrimitive(Class type) {
        if (type.isPrimitive()) {
            return true;
        } else if (type.equals(Object.class)) {
            return false;
        } else if (type.getName().startsWith("java.lang.")) {
            return true;
        } else if (type == Date.class) {
            return true;
        } else if (Enum.class.isAssignableFrom(type)) {
            return true;
        }
        return false;
    }
}

What else? Ja, you need quite a few commons jars like beanutils and digester 'cos they all build on each other apparently.

evanx

Trivial Templating Blog

Posted by evanx Jul 9, 2010

Filter Blog

By date: