Xojo 2019 R2, I mean API 2.0, Review

Lot of chatter in the blogosphere and forums about API 2.0. What do I think? 

I think renaming the events was a mistake. Not because the philosophy behind the changes is wrong but because the IDE was not designed for it. In fact, I cannot think of any language that renamed events on their standard library classes. It just does not happen. Events are part of the core contract of the whole object hierarchy and all code developed in an OOP fashion relies on the stability of those events. In other languages they would release a new class. 

The crux of the issue is the language and standard library are presented as a single entity in the IDE. It would be ridiculously awkward and painful if new projects started with AppV2 or WindowV2. Platform makers often just create entire new frameworks like WPF, UWP or WinForms. It is ridiculous, but it gives the developer choice. If I want to stick with my trustworthy but not entirely clear when it fires .Open() event I can just stick with Xojo Desktop v1. 

When you create a project, it should ask you "API v1 or API v2" and that's it. The constructs of classes having events is the same regardless of their names so why does the IDE care which version you use? All it needs to know is which library to use for code completion. The fact that the choice is not possible or was never considered is the part that makes event renaming as delivered a rookie platform mistake.

I am not as upset as I could be as there was an even more asinine change that we luckily got reverted. The <beta> class that we all use every day had been changed in ways that are completely and utterly mind boggling. Who is driving the technical direction here?

If I sold you a plugin and it did not work and I told you that I implemented .Open() and your use of .Opening() broke my code you would understand. It makes sense. It is annoying and one questions how or why this is possible, but it makes sense.

What if I had said the reason your code does not work was a subtle change to the default constructor that made the entire object immutable? All I would get back from you is a puzzled look. Likely, you would ask for clarification about why I mentioned the default constructor. 

That's right. Depending on the constructor you use, all properties silently changed their behavior and there is no way to know this in advance. If you have any existing code, you would reject using R2 on the premise that <beta> is completely and utterly borked. Funny, when they realized this, they changed the name of the class which is actually what should have been done in all cases of event, property, and method renaming.

Some of you will not use R2 because of event renaming but I promise you it could have been much worse. Worst part about all of this is R2 is actually a good release. Lots of things are fixed and improved and the database improvements without needing horrible prepared statements are much improved. However, there is also nothing shiny or stand out about the release. Subtle changes here and there give the appearance of a release with lots of effort, but it is mostly semantical nuances.

Array's moving from Append() to AddRow() is untenable for anyone with a computer science background. My thinking is in the future arrays can likely be used as a bindable data source and AddRow() represents the action occurring in the user interface. That's my imagination and hope for the product as I seek creative justification for these mostly pointless and subjective changes. All for the sake of a hypothetical future citizen developer who didn't buy and selected "did not understand the exact timing of Window open" on a marketing survey.  

The struggle is I don't know why API 2.0 exists at all as a version milestone. It does not move the product forward in any meaningful way. If Apple had not released macOS Catalina, then this release would serve no function. I am concerned these changes were built in a vacuum not mapped to any real-world deliverable. The time to optimize is when you release new things like DateTime. Release Web 2.0 with API 2.0. Release Android with API 2.0. Release an enhanced desktop framework with new controls like a date picker and database binding with API 2.0. Allow API 2.0 to grow out of necessity and not as a milestone in and of itself. 

The fact is if I use API 2.0 it won't improve my software and at the end of the day that is what creates Xojo renewals. Did my software just get a little bit better because I upgraded Xojo? Sadly, when it comes to R2, it does not. 

For the future I recommend being less ambitious and evolve more areas of the product simultaneously so timespans between meaningful new additions is not so great. Produce more meaningful new additions. We are coming up on a second XDC with nothing new for the platform to encourage new developers to use it. 

My XDC 2019 Feelings

I had a good time. It was as good as you could possibly expect from a programming conference in Miami. I met new friends, old friends, customers, and team Xojo. I walked away optimistic about Web 2.0, excited about upcoming opportunities, and confused about why Xojo is so secretive online.

The truth is time and time again you hear people admire the conversations with engineers and general comradery at XDC. I had good discussions with a few on the team and generally felt like I was a valuable member of the community.

I do struggle in some areas though.

Throughout the conference many raised concerns about the lack of engineers on the team. Geoff was always prepared for the question and was insistent that they are staffed appropriately for their current needs and roadmap. I believe them. Frankly, the roadmap has not expanded, and some items like plugins (built in Xojo) have been completely dropped. I see some efforts to align efforts with reality.

While discussing estimates and why Xojo does not commit to timelines Geoff said they try to “manage our expectations”. I believe Xojo should be releasing numerous builds even in early stages. Monthly alpha builds at a minimum. Soliciting our feedback on the design goals and direction earlier on would be very valuable for us. I don’t pay Xojo to manage my expectations of their product. I pay them to develop it and I would be happier if I knew the direction it was going in.

I wish Xojo was prouder of their third-party ecosystem. The design award to GraffitiSuite was great and deserved and I’m sure MBS has earned them in the past. At the same time outside of a single session I didn’t see a general awareness for the third-party ecosystem. More plugin and providers should be showcased. While self-serving, I wish there was a certified hosting program. ServerWarp would love to be a certified Xojo web host. Someone at XDC mentioned why there was no section for vendors. I whole heartedly agree.

Everyone believes they attempt to do too much.

I think everyone in the conference was in awe at the tremendous effort and progress of the overall platform. However, it is painfully obvious to us that while engineer staffing may be optimal the overall ecosystem is not healthy. Statistics and marketing data presented to us suggests that the number of women and young people has risen. Yet forum participation and general Xojo developer penetration seems flat or worse. It does not seem entirely impossible that perhaps the number of adult males has simply declined. 

During the final feedback session Geoff was great. He talks about vision and opportunity. Yet, I find myself disgruntled because I am convinced the company lacks technical vision. Nowhere in the conference did we see anything awe inspiring or cutting edge from Xojo. Just further iteration on the same old ideas and concepts with the same limitations and caveats. 

Upon later reflection I realized that Xojo has quietly made some very strong and strategic technical choices. For a long time, they suffered from tremendous “not invented here syndrome” and would recreate everything. Custom database server? Check. Custom HTTP socket? Check. Custom web framework? Check.

Yet more and more these items are being deprecated. Engineering efforts now are on pulling in the best and most accessible libraries available and simplifying them for us. The potential output of any given engineer is likely doubled or tripled due to the reliance on more third-party code. The new web framework for example is relying heavily on bootstrap and jQuery and various other user interface libraries. Interops is designed specifically to make utilizing operating system libraries easier so Xojo itself can iterate faster.

I think the keynote was lackluster because they were unable to communicate how much impact the new technical investments are going to have. As soon as web framework 2.0 is done they will be able to create considerably more controls very quickly. Plus, I hope they are dog-fooding their own Web SDK as much as possible. Android even if terrible will open several new doors and channels to insert Xojo into the conversation. 

So now it is more confusion for me versus disappointment. They apparently do have technical vision but are completely incapable or unwilling to share it. No blogs or forum posts about new abilities or plans. No videos highlighting features being copied from other languages and frameworks. You don’t see the passion and drive that fuels this effort, so it feels hollow. Like a grind. And it becomes one for us constantly following along hoping our feedback request gets fixed and our subscription renewal becomes worthwhile once again.

What do I want?

Monthly alpha builds
Certified third party ecosystem for plugins AND services
Less management of my expectations

I think a little more effort into battling the not invented here syndrome when it comes to community and the third-party market would change the game. If you aspire to be more than just a secret weapon you need to shepherd its growth versus dictating it.

XDC is about being a part of the conversation. Being heard. It should not take a physical gathering to keep that spirit alive.

Freezing Xojo Web Apps

As reported by Ralph Alvy and several others on the forums (https://forum.xojo.com/51170-2018r3-web-app-appears-stuck-when-it-quits/0#p414878) and in Feedback (53291) Xojo web applications built in Xojo 2018r1+ can lock up with no visual indication or recovery.

This is most prevalent on mobile devices when the user is interacting with your Xojo Web app and then moves on to something else. Behind the scenes the iOS or Android operating system suspends the javascript execution of the browser tab. This severs the link between the user application and the Xojo Web server.

In Xojo 2017 and before this was not a problem because the entire communication stack between the browser and server was built on XMLHttpRequest. This is the browser provided class that enables all AJAX interactivity on the web. Since the inception of AJAX however many new technologies like WebSocket’s and Server-sent Events (SSE) have been released.

They provide a persistent connection between the browser and server in either bidirectional or unidirectional capacities. This helps web applications respond faster to changes as building up a request is very expensive in regards to latency and when generating tons of events can really impair web application performance.

In Xojo 2018r1+ the web framework now defaults to SSE as opposed to XMLHttpRequest for all devices (except Firefox). This is great but unfortunately there is a bug. When a users session times out the users browser needs to be reloaded so a new session can be created and use continued. In the old XMLHttpRequest handlers this was recognized and if a session is no longer available on the server then the browser reloads itself.

Sadly, with Xojo’s SSE implementation there is no recognition that the connection to the server has been closed. A generic error has fired but because Xojo Web was working fine prior to the users tab being suspended it is not aware of connectivity issues. The app essentially locks up from the users perspective as the web framework is no longer listening for server side events and client generated events are no longer capable of firing as the session is invalid. Technically the client generated events do fire but the server rejects them.

Fortunately, there is an easy fix! When the SSE connection is closed the ‘readyState’ property of EventSource (https://developer.mozilla.org/en-US/docs/Web/API/EventSource) changes to the integer 2. I have verified a fix for this behavior by dynamically changing the error handler of Xojo’s SSE implementation to recognize a ‘readyState’ of 2. When it sees that it recognizes there is no recovery possible and it then starts pinging the server to see if it is still available. If it is then it reloads the browser and starts a new session exactly as the XMLHttpRequest implementation does.

To apply the fix is very simple. You want to execute some Javascript after the framework has fully loaded. Preferably you do this when your first window ‘Shown’ event fires. So do the following:

1. Add an event handler for the ‘Shown’ event for the first WebPage in your Xojo Web project if it does not already exist.

2. Add the following code:
   
   me.ExecuteJavaScript("if (Xojo.eventsource !== undefined){Xojo.eventsource.onerror = function (event) {Xojo.eventsource.msgcount--;if (Xojo.eventsource.msgcount < 1 || Xojo.eventsource.readyState === 2) {Xojo.view.preventInteraction();}};};")
   

3. That’s it! You can now deploy your app and verify the fix works on ALL devices.

Feedback for Xojo

1. There is a lot of parsing in the framework going on for WebSockets, Server-sent events, and conventional AJAX to identify when connections have broken, server unavailable, etc. In most cases you capture these events just fine and do the appropriate thing. However, at times when you miss one the web applications appears “locked up” with no obvious way to resume. You don’t know if it is lagging out, frozen, or what.
   
   I recommend when the server sees data being sent to a session that is no longer valid that instead of returning a generic 404 you send back a “session invalid error” that can be handled. You can then force the browser to reload regardless of the failures that led to the error in the first place. If the session is invalid there is no point in letting the client linger any longer than necessary as no recovery is possible.
   

2. There is a lot of EventSource handling code in Xojo.comm.ajax that relies on message counting to determine if there was connectivity failure. However, if connections break down or fail there is no graceful cutover back to XMLHttpRequest as a fail safe. The current implementation just reloads the browser and then attempts to use SSE again despite a failure in the previous attempt. Instead gracefully switch to XMLHttpRequest and only then if failures persist continue with a reload.
   

3. You should revisit the absence of using SSE in Firefox. Mozilla is very standards based and I’m sure appropriate workarounds could be implemented to support this methodology.
   

4. This is the second Javascript framework issue I have fixed (45691). My previous fix I submitted to Feedback on June 3, 2017 and all you have to do is implement it! Why the delay? I will submit this resolution to Feedback as well but I hope it can be rolled into R4.
   

5. One of the great things about Xojo Web is I can investigate and fix the client-side issues. I hope this provides some evidence that an open source standard library would be a good thing! The bits that make Xojo the most valuable (The IDE, the compiler, etc.) can and should remain closed source. The non-proprietary implementations of standard protocols (SSE) could benefit from more eyes on them in my opinion. See part 2 of my 2017 review series on where I talked about opening up more of Xojo’s standard libraries for cases just like these: http://www.dev.1701software.com/blog/xojo-in-2017-part2