Our Good, Bad, and Ugly Experiences with Services

Developing modern information technology solutions of any reasonable complexity will require that you integrate existing technologies at some point during the system creation process. Whether you’re just starting out or you’re looking to scale, you will face the decision of using an existing piece of software or building your own. The level of sophistication needed today can only be attained by building on top of other components, such as libraries, platforms or services. Just like in the old Western, experiences with these services can be good, bad, or ugly, depending on the providers you choose.

Even as InOrbit is building a platform to make it easier for robotics companies to focus on their own secret sauce, our software engineers have faced these decisions. In the course of our development, we have chosen services and components along the way. By choosing this path, we’ve also been able to add our own expertise, allowing us to augment those services to create a better platform for our customers.

Along the way, we’ve also had some mixed results that we’d like to share with you. We hope this lets you avoid some of the mistakes we made, as a way to help you speed up your own development process. 

And should you choose to use the InOrbit platform yourself, we expect to be held to the highest standard.

The good

The most positive decision we made in our development was to use mLab as our database provider. Given the chance to do it over again, we would choose them each and every time.

We began using MongoDB for some of our microservices and applications early on, as we were doing rapid experimentation on new features, defining and refining them with our early adopters.

The initial decision to use mLab was based on our need to move quickly, so we picked the easiest to find, recommended alternative. In addition, mLab had a free plan, making it very easy to get started. However, the real value and the reason we would choose it again would only become evident over time — which we believe always happens with all great service providers.

Once we began scaling with our customers and the component became more mission critical for our operation, it was comforting to discover that the service was very reliable, had clear pricing with no surprises, and most importantly had top-of-the-line customer service. Whenever there was a required update or scheduled maintenance, we would be informed well in advance, with clear suggestions on how to proceed.

As the scale of our adoption increased to the point where we needed to look in detail at production topics, such as high-availability, performance and optimization, it was nice to see that every time we reached out to mLab (timidly at first, but excitedly later on), their support staff would always be ready to respond quickly. They were able to attend to our needs with knowledgeable, deep answers that always let us improve faster without needing to invest additional time.

Sadly for us, and happily for them, mLab was acquired by Mongo and its service was sunsetted earlier this year. The process for migrating to their acquiring company was smooth as silk, and was the final confirmation of what a great service they offered, through the elegance and care shown in their helping us onboard with the new provider, well-prepared migration tools, and even an offer to refund credit for many months forward if the new provider's pricing model didn't match our expectations.

mLab is dead, long live great service providers like them.

So far, we're also satisfied with the service provided by MongoDB Atlas. This is a case where delegating this concern to another service provider continues to be a good choice. Even though MongoDB is well-known, and there are plenty of tools to implement it ourselves, this is one area where we leave the details of running and maintaining an efficient database service to someone else who is well versed in it and instead focus on the pieces where we can add more differential value.

The bad

Unfortunately, not all of our experiences were like mLab. Another service provider we used early on was responsible for our multi-cloud hosted message brokers.

This is a pretty simple and well-known service based on industry-standard technology. Our main goal at the time was to get started quickly and remain cloud-agnostic — as many cloud providers also include their own proprietary solution for real-time messaging.

We chose a provider and stuck with it for quite some time. In general, it provided a reasonable and easy-to-use solution with clear pricing. As we started scaling, however, we would bump into the occasional problem of a message broker glitch or unexpected downtime. Most of the time, we would get a timely answer and solution after reaching out to their support team.

But as time went on, problems continued to pile up. With more usage and frequency, the glitches happened at unfortunate times for our internal tech ops staff. It got to a point where answers provided by the vendor’s support staff just weren’t satisfactory. Many times we were left with unanswered questions as to the root cause of our problems.

At this point, we realized that despite their good intentions, the service provider’s ability to deliver high-performance and high-availability service did not match our needs. We had outgrown them, so we started to migrate this component to our own self-hosted infrastructure.

With the help of the provider, we transitioned this function in a planned fashion without any noticeable disruption to our adopters, or affecting our road map notably. As a side effect of this experience, we hardened our internal monitoring, incident management and on-call support.

The migration process, which we estimated to be one to two weeks, ended up taking a few months, and it still takes time today to maintain and operate. This gave us a better understanding of how services that sometimes seem simple on the surface can often take more time than expected, and can become a distraction from the company’s goals.

We believe we chose a provider that was right for us at the time, but one that we eventually outgrew. We’re grateful for the work we did together, as it served us well and helped us focus on other areas that required our attention. But we also learned about the importance of choosing a provider that is prepared to scale as things change.

The ugly

This final example shows how ugly things can get. It stems from when we decided to use an embedded analytics service provider to experiment with time series visualization and analysis on our system.

Integrating this component let us add robot and fleet-wide, time-based visualizations to our system for rapid experimentation with our early adopters, without having to worry about the necessary infrastructure, such as storage, APIs, and visualization libraries.

Again, it was easy to get started from a technical standpoint, but it required us to add credit-card information early on. In addition, since the service used a non-standard API, it led to a bit more coupling than we would have liked. We were early in our company’s life, so we watched every penny (actually, we still do.)

At first, we thought this wouldn’t be a problem, given our low transaction volume during this early exploration. Imagine our surprise and dismay when we received a bill that was 300 times higher than any previous month, caused by a poorly announced unilateral change on the way they metered and billed for usage.

Not only did we not receive any timely or reasonable response from this provider, we couldn’t even get our credit-card information removed from their system. For many months, we kept receiving invoices for unused services.

This poor service forced us to temporarily disable the feature from our system, and prompted us to implement our own time-series online analytics service, based on open source components. While we might have ended up doing so regardless, this came at a time when we hadn’t planned on dedicating time to it.

We still think it was valuable to get started with this service provider early on, as it let us perform some customer validation in the early stages. Had we better understood the poor service provided by the vendor, we would have selected another vendor, or at least planned migrating to our own platform at a less disruptive time.

What we learned

Through these and many other experiences, we learned some valuable lessons that have helped shape how we approach providing world-class service ourselves. It has also increased our appreciation for comments from our customers, some of whom have scaled their robot fleets 10x in less than a year, who have told us that “InOrbit is mission critical to their operations” or that they “couldn’t have done it without us”.

As you consider what to build and what to use or integrate, perhaps the most important lesson we can share is that what truly matters is to find companies and people that you can grow to trust and become a partner to your own endeavors, helping you grow faster and better than you could on your own. It’s not just about choosing the right technical features; choosing the right company is just as important, if not more so.

At InOrbit, we take pride in our service quality, and make every customer engagement an opportunity to help another company in their journey of robotics automation. This is why we aim to make it as easy as possible for companies just starting to build their product, or investigating cloud-based robot management technology. To that end, we make it free at the start (no credit-card is required to get started with us) and our team of customer success engineers are ready to engage to help you with your first steps.

Choose the right pardner to ride with you, and leave “the bad” and “the ugly” in the dust.