Sparks 3 is live!
I will do a full post with all the nitty gritty details for those interested in how this version (which is just getting started) came to pass. For now, just the facts:
- Universal binary. Buy once for iOS, use on all your iOS devices. Forever. Period.
- FREE for a limited time. This is something we wanted to do anyway but the gotcha is that we are going to be killing the existing iPhone SKU very soon. This is an opportunity for existing phone customers to get the new version for free. IF there is any issue with an existing customer not getting the universal build for free, please contact me at email@example.com immediately.
- Built for speed, built for performance, built for durability. Sparks 3.0 is tested against hundreds of rooms, tens of thousands of messages, crazy amounts of data and handles it like a boss.
Never have we been more excited to share software with you. We hope you’ll enjoy Sparks 3.0 and let us know what you think!
This is not an easy app to develop…
Not that any app is particularly easy to develop but with Sparks there were so many things that we encountered during the first version that we set out to solve in the second that I felt that it would be worth sharing the experience here for those curious and those looking to venture into building an app themselves that deals with 3rd party services.
Threads, Queues, Blocking and Blocks
One of the most difficult things to manage in Sparks is the crazy amount of network activity that is required to do even the most simple of tasks. 37Signal’s API is certainly well documented but, IMHO, a bit too restful for its own good. Architecturally, it makes it quite simple to know where to get data but unfortunately for us, most refreshes to rooms require several network requests and the initial loading of messages can take some time if you’ve got multiple sites and rooms. I tried several approaches to the chattiness of the networking code each with it’s advantages and disadvantages and in the end settled on combinations of a few of them.
Danger! It’s about to get geeky in here…
If anyone’s interested I’m more than happy to delve deep into the details on our final choices but for now, I’ll leave you with an indication of what’s happening now using my own account as a basis for which you can make some of your own comparisons.
When you first load the application, there is nothing stored about you or your habits as a user. So, we send you to 37Signals to authorize Sparks which allows us access to all your campsites. This information is very basic and doesn’t give us quite enough to show you anything yet, so we then cache some of the small stuff (API, OAuth tokens, site urls, etc) and then start the network requests.
I have 4 Campsites that I’m a member of so for me, that’s 4 requests that I need to make just to get a list of rooms. Depending on network traffic, latency, and the device I’m using this is a small start. From these 4 requests I have enough information to see the names of the rooms that belong to each site and that’s about it. So, I hang on to as many connections to the site subdomains as possible on background threads (via GCD) and start asking each room for it’s metadata, today’s transcripts and recent uploads. I have between 10-20 rooms for each site, for an average of 15 rooms and 3 requests per room for the first load. If you’re not keeping score…
4 sites, 60 rooms, and 3 requests per room for initial load for a total of 184 URL Requests in the first few moments of the app’s launch. Once we have the messages, most of them are either white noise (I don’t really care if someone entered the room 12 hours ago…) or missing information.
Missing you say? Well, not missing, per se, but RESTful so each message has a reference to a user-id but no information that may be useful for displaying the message like the user name. Since some users are not actual members but guests I can’t guarantee that a user is going to have certain data points so I have to make a request for every new user-id I find. On average I have 50 users across my sites so I’ve got to get them as soon as possible or risk delaying what I can show in the room if you happen to hop in before I’ve had a chance to cache the users. So, while I’m parsing the messages (did I mention that the Objective-C regex engine leave a bit to be desired…) I’m firing off between 30-50 new requests for user information. Other messages, like document upload messages, don’t have enough information for me to share, preview, load the resources, or even link to it reliably so I have to call another end point to get that information. This has been an average of one initial request per room’s initial load with a separate request for any new messages that happen to arrive during the use of the app.
If you’re losing count we’re near 250-300 network requests and you haven’t even joined a room yet. So you do. You “Enter” a room.
Well, this isn’t a typical UDP socket connection to stream a chat room or IM session. No, instead, to get more information and to enable https streaming over SSL in your room you MUST already be ‘in’ the room. So, one request to a DIFFERENT subdomain than the site subdomain to enter the room. Which makes reusing the network connection that is already in use on one domain difficult, if not outright impossible. So we don’t. This means, that if you’re wanting to get into a room and start streaming the existing connections cannot be reused so we have to fire up a new one and wait until we’ve got confirmation that you’re inside a room before we can stream the messages. This is not a Socket connection to a remote server. It’s a TCP connection to a standard HTTP server with the keep-alive headers on so that you can hold the connection. On a desktop app, this is usually fine, I’ve done it a million times. But on a resource constrained device with questionable network reliability you’re going to be counting bytes in packet bursts and performing health checks as often as possible without revealing the inner workings of all this data coordination to the end user. If I showed a message every time a connection was dropped or forced to reload or that your credentials needed a re-challenge you’d never stop seeing notifications.
That’s not to even mention the polling of rooms that you’re actively in and the possibility that a user may decided to tap the refresh button at anytime… which is actually valid given that if you have 60 requests to make to refresh your rooms it’s possible that 10 could be 2 minutes behind while 30 are up to date and 20 are in queue to be refreshed, including the only one you even care about at that moment in time.
In 10 minutes, I can reach hundreds of requests, in an hour thousands and it goes up from there. This is a problem that we solve constantly to ensure the bare minimum amount of data is being used while keeping the inner workings of what’s going on as transparent as possible. I don’t want to go into too many details but, in case you’re wondering, we’re constantly tweaking the loading/refreshing algorithms to ensure the most optimal way to refresh your data based on your usage patterns, network reliability, 3G vs WiFi connection, and so on.
ALL of this is performed away from the main thread (The UI thread). All of it. In fact, it’s on multiple threads depending on your needs and what the iPad will give me (for me that’s between 10-20 on a given full day of use), mostly moved over to GCD from NSThreads and RunLoops (which v1 relied on) which means that most of the codebase had to be re-built to take advantage of the block based async patterns that are favored by GCD (and myself!). Even though the heavy lifting is done off the main thread, the sheer volume of possible callbacks that can be invoked can flood the main thread with updates that then must be throttled and queued again to be performed when it won’t destroy the user experience. Can you imagine if I refreshed the lobby for every network request callback? When we first entered beta one of the feedback points was the I should never load data on the “main thread”. Yes, that’s true and I don’t. I never did. But I did have to figure out just how long to pause between actions before reloading/refreshing so that not only would the refresh happen quickly but so that indexes of arrays weren’t resorted, lost or augmented so that by the time a callback did return, the target objects still existed and existed where the callback’s lexical context expected them to be.
iPad 2 is your friend… sorry.
All this really comes down to the fact that iPad’s dual core setup is the best way to get the most out of Sparks. It can handle the load, concurrently distribute tasks and hang on to network connections more reliably than it’s predecessor ever did. Testing the app on my iPad 2 vs Jerry’s iPad 1 reminds me every day that I seriously need to buy him a new one. (I’m waiting for iPad 3. Jerry you didn’t read that!) Most of the reports of hangs or long load times are due to the fact that the system is optimized for multi-core architectures and when one is available is the performance and latency is exponentially better. There’s not much I can do or say about it. I made a choice to support the latest hardware and given Apple’s direction (or what I see as its direction) it’s a forward thinking choice that positions the application for long term flexibility which is good for us and for Sparks and ultimately good for you.
All of this for… (TL;DR;)
Chat. On the surface, it’s simple. In fact, most of our users imply that they think of chat in one of 3 ways: ( at least the ones that email us… )
- IRC style - one room, one context, dedicated socket connection UDP data that is consumed and lost after you leave.
- Instant Message style - UDP/RDP/TCP messages between one user or a few users. Simple. Even Text messages follow a simplistic User B pings User A repeat.
- Web chat (campfire, hip chat, convoke) which is a new standard but most of the low-level networking is taking place on your behalf by the browser. A unit of code that has been tested and optimized OVER AND OVER AND OVER again and it still gets it wrong. (WebSockets are nice though!)
Our view of chat is considerably more interesting, much more complex and on a device that has limited resources. We don’t own the backing service which means that we have to play by someone else’s view of the way chat should work for them. Which it does, Campfire is a great service and it’s popularity is a testament to it’s usefulness. Unfortunately that’s sometimes at odds with what would make a smooth device chat experience.
I have spent many late nights working hard to solve these problems in a way that this complexity is hidden from you so that you can work, chat, communicate on the go in a simple and reliable fashion. Sparks for iPad 2.0 is a reflection of all the thought that Jerry and I have poured into this particular domain and we believe that we’re really starting to uncover the best parts of what Campfire has to offer and are looking forward to how we can contribute to the experience and provide something that’s not just “the website on the iPad” but a valuable extension to an already fantastic collaboration system. These problems (if you can call them that) are problems that make me excited to sit down at my computer and write some code. They are challenging yet rewarding and I can only hope that the end result and the path that we’ve been on is something that others will also enjoy sharing with us.
It’s been really quiet on the blog and that’s for good reason, I’ve been working hard to complete the latest version of Sparks for the iPad! In fact, even the Apple review process turned around our first review so fast that we were somewhat surprised to learn we were in the store as I was about to push a minor point release to deal with some issues with the new message viewer! That is currently awaiting review and will be released soon (fingers crossed!)
I’d like a moment to take you through some of the new features, what they mean and how we decided to include them into the new version. These features are based entirely on our use, the feedback from our beta team and emails from users of Sparks who took the time to reach out and let us know what they like, hated and desperately wanted.
The User Interface
Eric, one of our beta testers, often remarked that he was frustrated with the layout of the original app and how the ‘real-estate’ provided by the iPad was being under-utilized. We agreed. In the original version, there was great care to create an experience that mirrored the “campfire” theme with a rustic look and views of loose paper. The problem was that while this design was visually pleasing, once you’re using the app, there’s more that can be taken advantage of.
Where are all the buttons, labels and standard interface elements?!?
They’re gone. We took a long hard look at what we could show or hide, what was primary focus and what you needed to do in order to be effective in the app. The new “Dashboard” is an attempt to show details at a high level about the current state of affairs in your campsites while improving on 37Signal’s concept of a lobby. For instance, the list of users shows everyone that is currently available in a room, no matter where that room happens to be. If I’m looking for a user in the web app, I have to scan all the rooms in one site, then go to another page to see if the user is in a different room that may be more appropriate for the topic I have for them. Now, I find the user, tap their name and see all the rooms they are in as well as a link to their email address to send them a note when I need to.
The “active campers” number will not equal the number of folks in the list. This is on purpose. The count is an indication of how many of those people are actual chatting, engaging with other campers and not just “signed in” or “lurking” in rooms. It’s no use to try and get someone’s attention if they’re not listening.
The charts currently contain a small bug which causes some of their numbers to conflict (Timestamps are being included in room counts…). This bug is fixed in the build that is under review.
The pie chart shows message distribution for each room that has messages loaded. Since Messages are loaded on a daily basis, this typically refers to the distribution for the day. If everyone is chatting in one room, head on in and be heard!
The bar chart, on the other hand, is to provide information about the global rate of messages hour over hour. A feature that has been requested is the ability to tap the bars and pie slices and they are under active development and will be released soon. (Pie selection is in the point release, bars will follow soon after.)
The Message (aka “In Room”) View
Another choice was to remove many of the buttons and room titles. Instead, if you’re towards the bottom of the room’s messages, simple scroll past them to reveal the room you’re in, the current topic and a few stats on the room at that moment as seen through Spark’s eyes.
Transcripts for the room are available via the calendar icon on the menu and provide a simple and easy way to load the message from any given day in the past. Do you think that Sparks missed a few messages for the current day? Load today’s transcript again and we’ll try to fill in the gaps. Need to return to streaming after viewing the transcripts? Just select today again and you’ll be back to where you left off.
A little known fact, is that if you’re chatting and want to “reply” to a user, type an @ with the keyboard and then continue typing any part of a user’s name to auto select that user and press return (enter) to complete the name. No need to move your fingers from the keyboard to reply.
Something that we started over on the iPhone version that made it into the iPad is the idea of “peeking”. Even though I hate to admit it, there are times when I’m curious about what’s going on in a room but I don’t really want to alter the other users that I’m visiting in. Say I can’t chat at the moment, or am on vacation, but I still am curious what’s happening on a project. I can double tap on any of the rooms in the lobby list and a popover will display the recent messages on disk so that I can quickly see what’s up and move on. The peek does not immediately force a reload of the messages in the room so if you’re not actually in the room, you may want to press the refresh button that is located on the right menu bar in the dashboard which will refresh all your rooms. Once the spinner is gone you’re ready to peek away.
That’s a quick overview for now, I’ll be following up with reports of the new stuff that’s in the works along the way and a few clarifications based on user questions and feedback.
As always, I want to hear from you. This app is a labor of love for us, we’re not a VC backed 100% of the time development effort on this application. We’re 2 guys trying to make the best campfire app on the iPad, something we want to use and share with the world. If we miss something or can help you use the app more effectively, don’t hesitate to ping us: firstname.lastname@example.org.
Thanks for your continued support and stay tuned for more Sparks announcements that are coming your way!
Hey all, I’ll write a bit more in detail here in the next few weeks but I wanted to let you all know that Sparks for iPhone has reached a minor milestone and hit 1.1.0 today. Far from a bug fix release there are a few new ideas and concepts that have been added. Here’s a few highlights:
First and foremost, Jerry has delivered a phenomenal visual update. There are visual enhancements from the minute the app loads to when you lock it. There are more of these in the works and the aesthetic is a precursor to all the great things to come in Sparks iPhone and iPad. The messages, tables, backgrounds, splash page, icons, toolbars, everything has been redesigned or re-articulated with the user at the core of every choice.
Least you think I’ve been resting on my laurels, I’ve added a few bells and whistles of my own on the code side of things! Here’s a few highlights I’d like to quickly mention:
"Peek Mode" a.k.a. incognito mode. I hate to say it, but there are times when I want to see what’s been going on in a room but I don’t want to alert other users that I’m up working at 3:00AM EST and available for questions. This is what Peek mode is all about. From the Lobby tab, double click (or 2 finger single tap) on any room and the messages from the day are shown under a page curl. You’re now peeking in the room without joining. If the messages seem a bit out of date, run a refresh from the lobby. If you see the last updated date change (you know, within normal variance. Seconds are going to change even without a new message. ah relativity!) then there’s a good bet you might want to peek again.
Tab reordering now successfully remembers the tabs that you moved and will be saved the minute you make a change. The bug that removed tabs from the “edit” list is squashed so re-order away. My tab setup is: “Lobby - Pulse - Important - Search”.
Async and Parallel take center stage. For my tech/code savy geek friends out there, while looking at our overall plan for Sparks one thing was made clear: we can count on more than one core. The A5 chip has to be heading to the iPhone soon and the iPad 2 has it. We’re already highly async (as much as we can be) but this version is optimized for multi-core setups. If you’ve got an iPad 2 and you run Sparks for iPhone on it you’re going to simply be astounded.
It’s an exciting time for us; iPhone 1.1.0 is here and continues to evolve, alpha for iPad is just around the corner and we’ve a few more surprises up our sleeve that we’ll be sharing soon. We aren’t stopping any time soon and can’t wait to show you want we’re doing next! :) As always if you need anything holler to me: leon [at] nospoonsoftware.com
We’ve had a few emails and tweets asking about the iPad version of Sparks and whether features that are in the new iPhone version will be making their way over to the iPad. The short answer is an emphatic YES! I suppose that shouldn’t surprise anyone but I’d like to give a little info on where we are right now, where we’re going and some background on our process for the 2 apps. So, in no particular order:
Why iPhone? Well, we never set out to do it. Jerry and I wanted an experience on the iPad and that’s what we set out do. While we were at this year’s CodeMash I had no less than 10 people demand an iPhone version. By the end of the conference I had a working prototype and the rest is history.
Cool new features but where are they on the iPad? Patience, my friends! :) One of the big wins of the phone’s UX constraints mean that there is a lot less work to be done to bring ideas to life. We work hard to provide a compelling experience not just a list of features. If a feature seems cool but doesn’t quite fit then we take it back to the whiteboard and iterate on it. The iPad version has a lot of real estate to account for and it’s a delicate balance to find just where some functionality should go, if it’s simple to interact with and truly provides value. We could easily ship a “big iPhone” version to the iPad but it just doesn’t feel right (we tried it and hated it thus no universal binary). So, instead we shipped some features we’ve been discussing for a long time to the Phone to see if they were worth bringing over to the Pad. Many of them are but it means that the iPad is going to need a fresh coat of paint.
The Icon When we were creating the iPad version the only client representations of Campfire used the exact same HTML presentation that exists on the web. Jerry and I don’t actively dislike them (far from it!) but wanted to present an experience that utilized the best parts of the target platform. For example, if you look at how text is rendered in Safari vs native Text controls on either the iPhone or the iPad you’ll notice a clear difference. To us, this is important. We wanted to love using campfire away from our desktops. Jerry conceived a fantastic aesthetic for the iPad that would evoke the “camping/campfire” aspect of the namesake while not infringing on 37Signals’ brand. While this was a great introduction to the platform we eventually felt boxed in by the choice. Suddenly, new features had some kitschy outdoorsy pun in the name and when we started rolling our own eyes, we knew it was time for a change. We want to focus on chat and less on being a “campfire” client.
Long story short, we’re currently developing a major update to the iPad version. The iPhone version allowed us to quickly prototype some new ideas (not all of them… there have to be some surprises, right?) and provide value to a new group of users while taking our time in taking the iPad to the next level. The Icon hints at the new visual direction Sparks is taking that will eventually make its way to the iPad.
WHEN, WHEN, WHEN?! Jerry and I believe strongly in the notion that you iterate quickly and ship as often as possible. With this in mind we prioritize on a daily basis and push when we’ve had enough time with updates that we feel they are stable enough to ship. We could have held onto the iPhone until the iPad was ready but that would have meant time would go by with a pretty solid version sitting on the shelf while we worked on the iPad version. That’s not who we are and while it may feel odd for iPad owners to see new features hit the iPhone first, they’ll be reaping the benefits of real world testing and use from iPhone users when they finally come to the iPad. It all leads to a more stable, usable and enjoyable experience in the long run and that’s really what’s most important.
Hopefully that clarifies our position on the 2 platforms a bit. As always, if you have any questions, concerns or comments please shoot us an email at email@example.com. We’d love to hear from you and can’t wait to show you what we’ve got in store this year!
For the first few weeks, we’re going to be running both the iPhone version and the existing iPad version at a discount to celebrate! So tell your friends (perhaps in Campfire… using Sparks!) to get the most simple and elegant way to use Campfire on your iOS devices with Sparks for iPhone and iPad