This discussion is archived
14 Replies Latest reply: Sep 27, 2013 4:35 AM by Bruno.Borges RSS

[Idea] Bring JavaFX to all browsers with VNC

walec52 Newbie
Currently Being Moderated

Hi,

 

An idea that has been going around my mind for quite some time and I'd like to bring it up the community.

 

JavaFX is a great technology for building UI's but it requires a full blown JVM on the client side. This is ok for many projects but more and more clients like the idea of not having to install anything and just use the browser. Plus as we all now the "normal" Java has been pushed out of the main stream mobile and tablet market AND browser plugins like Java and Flash are also becoming a thing of the past.

 

HTML5 and tons of JavaScript libraries are the UI mainstream now for enterprise systems but is this the way to go ? The slim client which the browser has represented before is now really FAT and an irritating environment for enterprise development. HTML5/JS is great to content driven sites that must have SEO but its not the right tool for complex UIs for the enterprise. So I started thinking can we have a powerful RIA framework that would bring UI computing back to the Java server, make the client slim again and not require us to know HTML/JS even when building custom UI components.

 

The ideal solution that came to my head was:

 

     noVNC - the JS VNC client

                     ^                                   browser

--------------websocket---------------------------------------

                     v                                   server

JavaFX application instances which

receives input events from the client

and render a stage to the VNC client

 

JavaFX is always present in the standard JVM on the server. Why don't we put it to good use.

The GTK people are already experimenting with the same aproach in the Broadway project.

GTK+3 With Broadway (HTML5) Backend Enabled In Ubuntu - YouTube

 

Question to the community: how do you like this idea ?

 

Question to the JavaFX team: would you support adding features to the JVM that would support the creation of an jfx2vnc library ?

Things that would be needed:

  • run multiple JavaFX Applications on a servlet environment without any graphical environment like X-Windows - maybe its already doable ?
  • receive events from JavaFXs rendering engine about rerendered areas,
  • something like the java.awt.Robot class - or do have we already a mechanism to grammatically push events to JavaFX ?
  • 1. Re: [Idea] Bring JavaFX to all browsers with VNC
    Jeff Martin Newbie
    Currently Being Moderated

    Apparently Oracle has some idea up their sleeve for running JavaFX in a standard browser:

     

         WebFX: Running JavaFX Like HTML5 Apps

     

    At JavaOne. I don't know about VNC - I think it is only useful when you have limited needs and/or are very desperate. :-)

     

    jeff

  • 2. Re: [Idea] Bring JavaFX to all browsers with VNC
    jsmith Guru
    Currently Being Moderated

    I agree with Jeff on this, circle back on this idea after it has been seen what is presented at JavaOne.

     

    Also Jeff, note that the WebFX session is completely different than what the originator of the thread proposes.

     

    WebFX is this:

    https://blogs.oracle.com/brunoborges/entry/webfx_running_javafx_as_web

    https://github.com/brunoborges/webfx

    A JavaFX standalone application which can load and execute both html and fxml over a network, not an html client for rendering JavaFX over the network.

     

    There are some sessions on html clients for rendering JavaFX over the network:

    The Chuck Norris Experiment: Running Java in Any Browser Without a Plug-in [CON4044]

    Spice Up Your Browser with JavaFX [BOF7830]

     

     

    Chuck Norris's session is based on =>

    Bck2Brwsr - APIDesign

  • 3. Re: [Idea] Bring JavaFX to all browsers with VNC
    walec52 Newbie
    Currently Being Moderated

    This has NOTHING to do with the standard browsers (IE, FF, Chrome):

    WebFX - JavaFX as web pages - YouTube

    its just an JavaFX app that looks like a browser and embeds other JavaFX apps in it (+WebView to open a web page).

     

    I don't see any real world scenario in which JavaFX would get deeply integrated in IE, FF, Chrome or where a new JavaFX based browser would dominate the market.

     

    Also I do not see any point in a project like this and I don't see what problem does it solves ?

    Should we require the users to install Java(FX) AND THEN WebFX AND THEN type the URL of our appp AND THEN it will deliver the app to them ? why ? WHY ?   its worse then sending jars ...

  • 4. Re: [Idea] Bring JavaFX to all browsers with VNC
    jsmith Guru
    Currently Being Moderated

    One brain-dead solution for this, which would be easy to prototype is:

     

    1. Run a JavaFX app on a server.

    2. The JavaFX app renders UI and takes snapshots of the UI each pulse (using an AnimationTimer, the Snapshot API and ImageIO to output images).

    3. An html page is served up by a webserver.

    4. The html page contains a timer and an image link.

    5. On each tick of the html timer, an ajax or websocket call requests the latest app image from the server.

    6. Input events (mouse clicks) with the image on the html page are captured by JavaScript.

    7. The captured input events are sent over the wire (using ajax or websockets) to an application server.

    8. The application server generates JavaFX events from the event wire format and triggers the events on the JavaFX app running on the server.

     

    Presto, JavaFX app running in web browser with no Java runtime distribution.

     

    Over a local area network, it might perform OK, over a wide area network, there would likely be significant issues.

    But then again, I can stream HD video from the internet over my Comcast internet connection, why not stream a JavaFX application display.

    As Jeff says, perhaps "only useful when you have limited needs and/or are very desperate", but I'd be curious how even a simplistic solution actually performed.

  • 5. Re: [Idea] Bring JavaFX to all browsers with VNC
    jsmith Guru
    Currently Being Moderated

    > Should we require the users to install Java(FX) AND THEN WebFX AND THEN type the URL of our appp AND THEN it will deliver the app to them ? why ? WHY ?

     

    I agree, it's probably of limited use, but I can imagine use cases for this.

     

    For example, let's say you write an application to be delivered via an application store (e.g. Windows Store, Mac App Store, Android Play, etc) and you wanted to deliver dynamic content to that application through server updates, then you could use a component like WebFX.  The actual installation of your application would be the same as any other application to be deployed to that application store (because the application is built using ahead of time compilation with statically linked libraries and no dynamic libraries, includes all the runtime libraries it needs, so does not require a separate JVM installation and is packaged just like any other application for those store deployments).  If the application is implemented in JavaFX it may be significantly simpler to use a WebFX style component than providing separate implementations for Windows, OS X, iOS and Android.  As a Java developer (or any developer really), you might prefer to deliver the app using JavaFX/FXML than HTML alone.  User experience-wise, such an app might be well received by your users.  Of course some of the above is hypothetical, as not all pieces of infrastructure are currently in place to deliver that complete experience.

     

    Anyway, yes WebFX is a different thing entirely than what you were proposing in your post.

     

  • 6. Re: [Idea] Bring JavaFX to all browsers with VNC
    walec52 Newbie
    Currently Being Moderated

    As for the JVM implementation in JavaScript to run bytecode:

    • I've seen an experiment like this a few year ago - interesting but
    • slow as mud
    • ether way you look at it - it IS a FAT client concept
    • in my opinion unrealistic to get the most important parts of Java SE libraries there
    • even if achieved - the load time would be a killer

     

    As for the minimal JVM that "will" be embedded in standard browsers (I understand thats how Bck2Brwsr wants to achieve its goal)

    • totally not compatible by design with the Java SE library
    • didn't Android get sued for that ?
    • ah yes.... Chrome will accept that minimal JVM for sure...
    • we already have minimal JVMs like Avian - no traction there from the browser people
    • Google has a more broader VM/Sandbox concept called Native Client - Google Code - no traction there besides Chrome
  • 7. Re: [Idea] Bring JavaFX to all browsers with VNC
    walec52 Newbie
    Currently Being Moderated

    I think I'll try to devise a prototype in some weekend.

     

    I know we have the snapshot capability added to JavaFX 8. My take on the solution with the current JavaFX possibilities would be to:

    - Periodically take screenshots.

    - Compare the current one with the last one and only send rectangles that include changes - its more the VNC way

     

    > only useful when you have limited needs and/or are very desperate

     

    well... VNC and RDP are in wide use in the enterprise and manufacturing to for example:

    - remote classical fat client systems

    - on data capture devices with scanners that mostly run on Windows CE - no one wants to write native software for that ...

    - baaaa we even still use telnet on those devices and text based terminal aps with success in logistics

  • 8. Re: [Idea] Bring JavaFX to all browsers with VNC
    jsmith Guru
    Currently Being Moderated

    > run multiple JavaFX Applications on a servlet environment without any graphical environment like X-Windows - maybe its already doable ?

     

    I am curious if this is already possible and if anybody has tried it.

    Seems an obvious solution for rendering graphs and other things server side.

     

    > receive events from JavaFXs rendering engine about rerendered areas

     

    JavaFX has some concept known as dirty regions.

    There is almost certainly existing hooks in the source to get feedback on changes to these dirty regions so that they can be rendered.

     

    > something like the java.awt.Robot class - or do have we already a mechanism to grammatically push events to JavaFX ?


    Yes, awt.Robot can still be used to generate events that JavaFX will accept.

    I believe JavaFX also has internal Robot implementations in it's private API (though I don't know the state of those).

    Java 8 adds event constructors to the public API so that you can generate events like mouse clicks, etc.


    > JavaFX is always present in the standard JVM on the server.

     

    Depends on the server.  JavaFX is not in any packaged OpenJDK distribution yet (as far as I know).  Also, Oracle recently released a server only version of Java which does not include client code.  But yes, if you install a recent full Oracle JDK on the server, JavaFX will be there.

  • 9. Re: [Idea] Bring JavaFX to all browsers with VNC
    jsmith Guru
    Currently Being Moderated

    > I think I'll try to devise a prototype in some weekend.

     

    That would be very cool.

     

    > Periodically take screenshots.- Compare the current one with the last one and only send rectangles that include changes

     

    I'd advise to start with taking whole app snapshots (on an app with a small stage size), then look at the rectangle comparison stuff later (on another weekend ;-)

     

    There may be something you can leverage in the JavaFX dirty region work to help work out what areas of the scene to take snapshots of to minimize the size and number of snapshots taken - or you could just go with rolling your own implementation for this.

     

    Another possible optimization is to use an HTML canvas and pass vector drawing instructions from the server to the client for rendering in the canvas.  But that approach seems significantly more difficult to me than a straight bitmapped or image based approach.  (I do think Oracle have internally prototyped this approach at some time in the past).

  • 10. Re: [Idea] Bring JavaFX to all browsers with VNC
    walec52 Newbie
    Currently Being Moderated

    > Another possible optimization is to use an HTML canvas and pass vector drawing instructions

     

    You mean vector drawing instructions like in the X-Server protocol: draw rect there, draw circle there ?

    If so I think this optimization is a lot of pain with little gain because:

    - with all the gradients, overlays, bells and whistles in a modern nice looking app theme there is not much simple shapes to draw,

    - binary formats like PNG can compress vector graphics very well - an PNG render can take less then a source SVG file if the resolution isn't too high

     

    so I'll stay with my dead simple rectangle approach

  • 11. Re: [Idea] Bring JavaFX to all browsers with VNC
    jsmith Guru
    Currently Being Moderated

    > so I'll stay with my dead simple rectangle approach

     

    Good call :-)

  • 12. Re: [Idea] Bring JavaFX to all browsers with VNC
    ryan29 Explorer
    Currently Being Moderated

    You might find this article interesting.  It's fairly short.  The part about scaling is something that surprised me.

     

    Chasing the pixel streaming dream | VentureBeat

     

    Something else to keep in mind is that you can't beat latency.  Anything that acts like a remote frame buffer is going to suffer from latency which is as bad as it can possibly be on wireless and, in turn, most mobile devices.  Microsoft claimed users could notice 20ms of input latency when they were testing for their tablets.  In terms of perceived performance, network latency can be higher, but the client (application) needs to be fat enough to react to user input and give feedback in <20ms or things start to feel sluggish.  You can hide the network latency in a transition, but that transition needs to start almost instantly for an application to feel snappy.  That's why true thin clients where actions are processed on the server kind of suck.  Browsers are going to keep getting fatter until they can do what JavaFX already does.  At least that's my opinion.

     

    I don't know if it's in that article or if I read it on their site, but they basically revert to pixel streaming from an on-LAN companion computer when dealing with mobile.  I'm not sure why, but there's a good chance it's not a technical barrier IMO.  Apple (iTunes Store), MS (Windows Store), and Google (Play Store) are pretty protectionist when it comes to their platforms, so if this is anything more than a though experiment, you should consider how quickly you'd get punted it ever evolved out of the browser and turned into something compelling.

  • 13. Re: [Idea] Bring JavaFX to all browsers with VNC
    walec52 Newbie
    Currently Being Moderated

    Yes, I agree that the remote frame buffer approach I'm aiming at will have at least a 30ms lag on WAN which will be noticeable.

    (at least to the time we start using quantum entanglement in network communications joke... but interesting topic)

     

    But I still see a wide range of usage in the fields I work in. Example scenarios/arguments:

    - LAN based usage or WAN with a direct connection via ATM (primarily set up for video conferencing in bigger companies) don't suffer such a lag,

    - JavaFX desktop client is the work horse used inside the company but we still need a non-installable client for our partners for less intense tasks,

    - Eclipse RAP which is something between a fat and slim client (GUI uses HTML/JS but relies intensively on the server) has a 30ms lag on even the simplest mouse click events,

    - non mouse/touch base interactions like via barcode scanned don't notice a 30-100 ms lag in my opinion - real thin clients are very popular on data acquisition devices.

     

    As for the article: very interesting approach - I'm watching the videos on their page - a first glace it look like an - execute client code before all exes/dll are downloaded - approach.

    Nice for CAD apps but for client-server systems ? who knows... great technology or not client machines should not execute business logic - because we cannot trust all of them

  • 14. Re: [Idea] Bring JavaFX to all browsers with VNC
    Bruno.Borges Newbie
    Currently Being Moderated

    Should we require the users to install Java(FX) AND THEN WebFX AND THEN type the URL of our appp AND THEN it will deliver the app to them ? why ? WHY ?

    Hi,

     

    I'm Bruno, the creator of the WebFX project.

     

    About the questions above, I would say first that this is just a concept, but instead of thinking about the "browser", it could actually be used by real projects today that want to provide a JavaFX experience, without having to push big JARs for clients that might be in a very distant location with really poor connectivity. Also, it would make it even easier for the developer to update the application "on the fly", and his users would have access to the new version instantaneously.

     

    Consider wrapping the "WebFX" concept in your "base" desktop application, and instead of using WebView to load parts of your app using HTML+JS, why not using a FXView (based on the WebFX concept) to render remote code?

     

    Again, it's a concept, but I'm sure it can be really useful :-)

     

    Hope I could clear all the questions!

     

    Thanks,

    Bruno Borges

    @brunoborges