We’re working on some great new features here at Boundary, one of which is an Active Flows metric for tracking the number of in-use connections or interactions between hosts.  Here’s a teaser of an updated Streams view with its supporting traffic table:

What’s this for?

The Active Flows metric counts the number of active (sending packets/data) flows per time period. If the protocol being counted is connection oriented (like TCP) then this metric will tell you the total number of active connections.  Likewise, if the protocol is message oriented (like UDP) it will tell you how many clients are interacting with a datagram service.

Why would this number be higher than I think it should be?

You may have some misbehaved applications and/or clients that are flapping and/or setting up new connections in a tight loop.  We see the normal connection setup and teardown as the beginning of a flow, and do not wait for the connection to be established in order to report on them.

Can I set alerts / reports / dashboards on active flows?

Getting active flows in the streaming graph is the first step.  We’ll be adding them to all the other pieces shortly.

We’re currently beta testing this feature with select customers.  If you’re interested in checking it out, please let me know! michael@boundary.com

DNSimple shares how Boundary “saved our bacon” during a sudden disruption in service.

We love it when our customers tell us how our solution helped them solve a problem –and  this week we got a note from Darrin Eden, senior software engineer and operations specialist with our long-time customer, DNSimple, a hosted DNS provider with around 8,000 customers:

“Boundary saved our bacon yesterday. DNSimple is under a DDoS style attack and without Boundary I’m pretty sure we would be flying blind.”

We called Darrin to get a little more insight into what happened during the attack and how Boundary helped. Apparently, distributed denial of service (DDoS) attacks are nothing new for DNSimple and Boundary has helped in the past identify when these attacks are occurring. “In DNS land if you are a managed provider, you are often the target of malicious attacks,” Eden explains.  The attack typically has a distinct pattern in the Boundary dashboard, which Eden can quickly identify.

“We knew how to react to this style of attack because we had seen it before, but this time was different,” he recalls. “Boundary showed a different pattern and in fact, our software actually failed, and all of our customers went offline.” This was the first time the company had suffered such an outage that affected customers.

Saving customers hours of downtime

Eden and his teammates quickly began researching the cause of the issue and were able to start working on a fix almost immediately. Without Boundary, it could have taken 10 times as long to discover the source of the issue and where to focus resolution efforts, he adds. “The time from the start of the event to the initial resolution was an hour, and then it took a few more hours to come to complete resolution.”

Naturally, the three-person company was quickly flooded with customer emails, Twitter messages and calls as the disruption began. Yet because of the granular information that Eden was getting from Boundary, he could provide Twitter updates every 10 or 20 minutes so customers knew what was going on and how the restore was progressing. That level of detail insight was critical, he says.

“Without Boundary, there’s a chance we could have been dead as a company.  If a web hosting company is down for hours and is not providing regular updates to their customers, that usually results in a mass exodus of customers.”

Twitter updates proved invaluable

Since Boundary allowed the company to restore normal operations to its customers in around an hour, and during that time provided useful updates, customers didn’t panic and jump ship. In fact, he says, since the attack occurred a little over a week ago, DNSimple has actually seen an uptick in new sign-ups.

“I am sure we lost some customers, but in the end our communications on Twitter was actually really good publicity for us,” he says.

While there’s no foolproof way to prevent denial of service attacks, this type of attack won’t happen again since DNSimple now knows the pattern and its software has been modified accordingly to respond without failing. More importantly, Eden and team gained confidence that in the event of any future significant outage, they’ll be able to find the root cause quickly and keep customers apprised of the situation.

“When something is looking wonky in our metrics, we always look at Boundary first,” Eden says.

Joe Panettieri

Joe Panettieri is executive VP and editorial director of Nine Lives Media, a division of Penton Media. He oversees  The VAR Guy, Talkin’ Cloud, and MSPmentor.net, the leading online communities for VARs, CSPs and MSPs. Follow Joe on Twitter: @joepanettieri.

Boundary: How has the cloud IaaS market changed in the last 12 months in terms of both demand and vendor positioning?

Panettieri: We have reached a tipping point in the channel for cloud. Even though we are a decade into this journey, thanks to SaaS and public cloud companies, integration with the channel is really just beginning. The large cloud providers are now acknowledging that their partner ecosystems need to push beyond ISVs to VARs, integrators and MSPs. Smaller startups and nimble IaaS providers are starting up cloud partner programs from the very beginning. It’s not just an agent model with a one-time fee but a true model for generating recurring revenues, and to become the partner of record and really own the customer relationship.

B: How does this shift help end customers?

Panettieri: It’s a great thing. CIOs have discovered that they had multiple cloud services running without their approval. CMOs and many others are activating services and billing the employer every month, and if you roll it all up it’s a big expense for the organization. So how do you regain control? CIOs and other corporate leaders are reaching out to their channel partners and asking them to bring some order to their cloud strategy. They’re looking for a cloud broker who recommends and gives seal of approval to cloud providers and whom can be the one throat to choke. This is giving customers and CIOs clarity in terms of consolidating cloud providers and getting expenses under management.

B: Let’s talk about the big players. It seems like Microsoft Azure doesn’t get much respect. How do you rate the big ones?

Panettieri: To start with, Amazon (AWS) is an interesting beast because so many channel-focused software companies are now going to the cloud and are launching their services in the Amazon cloud. Microsoft has been very early on focused on Azure for ISVs and corporations, versus channel partners and VARs. The channel play was initially focused on Hosted Exchange and Sharepoint. In the last six months, however, we are seeing Microsoft connect the dots between Azure and Office 365. Channel partners are launching Office 365 extensions in the cloud.  So beyond reselling Office 365 they layer on additional management and maintenance services, and they can make last-minute changes for customers.  A company called 365Command makes tools for managing Office365 on Azure. Quosal is another company and they focus on customer approval software to help VARs close business more quickly, in Azure. Google Compute Engine (GCE) is still early days. From my last check, which was about a year ago, there are about 6,000 Google Apps resellers. I think we’ll see the trend when they begin to offer GCE services to customers. Finally, the Rackspace channel is committed as well. Rackspace began a partner advisory council in 2012 that has partners with a range of expertise, and the company has an added interest in open stack, an open source cloud platform. If I were a cloud reseller I would be watching Rackspace closing with regard to open stack.

B: What are the gaps right now?

Panettieri: The biggest gap is localization.  If I am working with big cloud provider can they work well with smaller VARs and their smaller customers. We are seeing upstart providers hire seasoned channel execs who know how to work with local partners. ViaWest hired Rackspace execs to get their channel program going. Tier 3 has a channel program where they are trying to work on a local level with VARs and MSPs and gaining some momentum there.

B: How do you see the segmentation of the IaaS market in the coming 12 months, and what will this bring to customers?

Panettieri: The cloud market will begin to resemble the retail market here in the United States. We will see big horizontal, massive players like Target that compete on price and offer great inventory. The large cloud providers are pretty similar and they will compete on price, but back to the retail model, there will be boutiques in various verticals such as healthcare. More and more cloud providers will focus on vertical expertise.

B: Due to the cloud, are we now experiencing a renaissance for the sector of IT management and monitoring technologies?

Panettieri: A lot of the management and monitoring tools grew up on the corporate IT side to manage servers and apps inside the network. Then MSPs began to offer that service. But now we are shifting again, to cloud-based management tools. There are three areas of applications: first, for managing SaaS apps on third-party cloud services; second, for managing internal systems hosted in the cloud; and third, for monitoring third-party public cloud services. These tools will alert you when AWS begins to slow down, for instance. There’s a lot of funding right now in that last area. As well, there will be some consolidation and acquisition of all of these tools. Avanade, a large MSP and Microsoft consulting firm, just acquired a cloud monitoring and management company.

The official start to summer is almost here, which can only mean one thing: it’s time to beat the heat and hit the movies. After reviewing the lineup of new releases, we got to thinking that a few of the leading storylines seemed akin to the drama playing out in big tech today.

Epic: Microsoft’s Ultimate Battle… For survival

This lush animated movie with adorable talking slugs masquerades as the ultimate fight between good and evil. Will the smart and chatty heroine Mary Katherine (Microsoft) prevail against the evil spider Queen Tara (Google, Apple, the entire US press force…) to save her world?

After Earth: will Google be there?

Will and Jaden Smith star in this film about a futuristic Earth full of darkness and danger: Google Glasses to the rescue? Rest assured, Google will come up with something cool or weird to battle the demons still alive on this barren planet, in about 1,000 years.

Owen Wilson and Vince Vaughn, using Web video chat for the first time as the Google interns.

The Internship: Hiring Them (Marissa Mayer) was A Brilliant Mistake

Yahoo’s still fresh CEO Marissa Mayer is somewhat like Vince Vaughn and Owen Wilson in this spoof-like film about a Google internship unwittingly thrust upon two “old” and out of touch sales reps. People want to hate them/her, but they can’t. Will they/she have the most profound impact on the company since its founding?

Now You See Me

This film portrays a band of big-stage entertainers practicing the fine art of illusion while stealing millions of dollars, kind of like Apple’s latest tax evasion strategies overseas. Will the deceivers own up to their acts, or will they continue to be the heroes of the young, beautiful and gullible?

Fast & Furious 6

Vin Diesel’s latest action-packed thriller is like Twitter: the company’s moving so quickly, teenagers are running screaming from Facebook to jump into this once geeky, social media race car.

Star Trek (Tumblr): Into Darkness

Star Trek’s latest film finds the crew of the Enterprise facing “an unstoppable force of terror from within their own organization, that has detonated the fleet and everything it stands for, leaving our world in a state of crisis.” Oh Tumblr, as you peer into the dwindling remnants of your bank account, Yahoo has stepped into save you, but which way is salvation? Or do both paths lead to unfathomable darkness in the end?

Facebook: Hangover 3?

Facebook, we still love you, but really, it’s time to stop these antics. First, your IPO: enough said. Then, the teenagers who made you famous are now leaving you, for Twitter! And now, more disappointing news on the earnings front? Like moviegoers everywhere, investors can only pray there’s not a Hangover 4.

Leo Capri as “Gatsby”

The Great Gatsby: Dream on, Larry

Much like the charming, enigmatic Jay Gatsby, Larry Ellison is on top of his game. He still seems to think that he can win the world by buying it. So far, this self-serving, hedonistic strategy has worked reasonably well, but let’s get real. The world is changing Larry, and you are not changing with it. People think you’re old school, inefficient, overpriced and inherently selfish. Eventually, you and your lavish, overwrought software company are going to crash right into the San Francisco Bay. But for now, live hard and enjoy the ride. Why not?

Iron Man 3:  Free Michael Dell!

Dear Michael: Hang tight man, you’re nearly free of those irritating, overbearing, power-hungry investors. Once your chains to the public markets have been broken (it’s only a matter of time), you can at last do whatever you wish and believe is best to win again in this crappy PC market! Who cares that your Q1 really sucked? You’ve got gumption, and that is what matters for the Iron Man: “With his back against the wall, Stark (Michael) is left to survive by his own devices, relying on his ingenuity and instincts to protect those closest to him. As he fights his way back, Stark discovers the answer to the question that has secretly haunted him: does the man make the suit or does the suit make the man?”

Stephane Bailliez

Guest post by Stephane Bailliez, Principal Software Engineer, Gilt Groupe (originally published on Gilt’s tech blog)

We recently published a case study with Boundary regarding how we at Gilt Groupe are using their product. I wanted to give some additional details concerning our decision process, what we were looking for, what we looked at and why we decided that going to Boundary was the best choice for us moving forward.

Back story: managing service complexity

Gilt Groupe’s architecture is now very much a case of micro-service architecture. We have hundreds of JVM-based HTTP services interacting with each others or with backend systems such as PostgreSQL, MongoDB, RabbitMQ, Kafka, Zookeeper, and many more third-party solutions over various data interchange formats and protocols.

A few months ago, we felt we needed to get more insight into the detailed amount of traffic that was going in and out of every service or backend system. When various teams are working on new features that require more communication patterns and data exchange, it starts to be difficult to do capacity planning when you don’t know where you are.

Moreover, in our experience, we have seen that most features generally go from a normal usage pattern for months to a sudden very large adoption by our business operations. The amount of data can suddenly grow 1 to 2 orders of magnitude, which does not generally go without its own set of challenges.

To get better insight into the amount of data exchanged, we started the effort to monitor the data transferred out of our HTTP services (we use Jetty) using the excellent Metrics library from Coda Hale. This can be trivially done extending the existing Metrics InstrumentedHandler for Jetty (go to the original post for the code display).

The issue: instrumenting various HTTP clients

The client’s effort would be a bit more challenging however. In our JVM-based services we end using a menagerie of HTTP clients: AsyncHTTPClient with Netty 3.x provider, Apache HttpComponents 4.x, Apache Commons HTTPClient3.x, and the venerable JDK HttpURLConnection.

This is the reality of having to deal with various third-party integrations, it makes things more complicated than we would like to, but it can be a bit annoying to rewrite or extend some existing SDKs to try to use one and only one HTTP client across the platform (especially when non-extensible, or worse, closed-source).

The immediate problem faced is effectively how to instrument *all* those clients.

AsyncHTTPClient can be done easily using a RequestFilter and an AsyncHandler. The code would be something similar to the snippet below. There is not much overhead of doing it as you just need to count chunk size as they the HttpResponseBodyPart objects are received.

Note that we tend to give a name to each service client which would map to a Metrics scope, which makes it useful to distinguish metrics between each client (some services use a dozen of clients).

For all the others clients, it is a bit more intrusive to be practical. And it doesn’t address how to monitor the traffic in/out going directly through the Socket api like for Zookeeper, Play Framework (Netty server), MongoDB and JDBC drivers, etc…

Another solution would be to write a JVM Java Agent via thejava.lang.Instrument API to instrument some well known libraries (NewRelic uses a similar technique, but doesn’t track traffic). While it may looked like the less intrusive solution, it is also a fairly significant undertaking to develop instrumentations for several third-party libraries which you have to maintain over time.

Also, knowing we were looking with an interested eye to add systems such as Riak, Redis and possibly some various asynchronous drivers and having to deal with multiple versions of Scala… this was a cool project to work on technically, but maybe not excessively practical.

What we needed was something similar to nethogs minus the text interface. A tool capable of grouping the bandwidth by process, but ideally it would have some features also found in Wireshark.

We did not find anything matching those requirements, until a week or two later.

Boundary’s Introduction

We had Cliff Moon, Co-Founder and CTO ofBoundary, visiting our New-York office to present Boundary and do a Tech Talk on Distributed Systems (which we blogged about).

We installed Boundary on some our servers to get a better idea. This was truly a revelation. The installation was painless with just a single command and as soon as the agent was up, it started to report data to the dashboard within the next second.

[dashboard graphic on the original post]

Each of the lines represent the traffic volume happening on a given port/protocol across all nodes at a 1 second resolution. Traffic can easily be broken down. For example you have the ability to group servers, either manually or dynamically using pattern matching which makes it easy to segment your front-end from your backend machines and see traffic flowing between those groups (this is where a descriptive naming policy for your machines comes handy).

Digging in deeper

You can further segment your traffic by port / protocol. For example TCP 5432 would be the traffic to/from PostgreSQL. You can then easily analyze the traffic that is going from your backend machines (or a subset of those) to your PostgreSQL. Same thing could be done to know the chatter around our messaging infrastructure on RabbitMQ.

A lot more details on how all of this can be done is visible on a Youtube video ‘Isolate your traffic with filters and conversations’.

There is a shortcoming currently for us where we are effectively losing a bit of visibility in our conversations. For instance, traffic to our services is always going through a set of dedicated service load balancers. For example we reach to it via a canonical url such as http://svc-product and the load balancer will balance between node1:7501, node2:7501, node3:7501. It means traffic from the caller to/from the load balancer happens on port 80 while the traffic to/from the load balancer to the callee is on port 7501.

caller ← port 80 → svc-lb ← port 7501 → callee

This  means that the traffic flowing on port 80 is basically the aggregate of all service traffic and that we cannot see the traffic directly from caller to callee, but only the aggregate from caller to svc-lb and from svc-lb to callee.

This is something that may be alleviated a bit in the future as we are thinking about removing the load balancer and having applications doing the load balancing themselves using information from Zookeeper.

Boundary settings on the dashboard can be driven entirely from their REST API, which provides the added convenience of being able to integrate with your own configuration management system such as Puppet or Chef and a set of backend applications which may contain metadata about your environment.

The REST API is useful to define application aliases which gives names to a protocol:port (eg: ‘svc-product’ for TCP 7501), send deployment events or integrate with other systems (it can subscribe to NewRelic events via RSS).

We have only scratched the surface of Boundary so far and we are very excited about the direction it is taking and what is being developed. It has already proved extremely useful in identifying traffic volume and patterns occurring between services and databases. Something that would have required a lot more tedious investigative work can basically be done now in a few minutes and with much more flexibility than we could initially imagine and with no direct investment.

I hope that this (long) blog post will be helpful to some people who are facing the same challenges of not having enough visibility in their network traffic. If however you know of any interesting tool in that space, feel free to drop a note.

On a slightly unrelated note, we are also users of a nice library from Boundary called Ordasity. It is a great way to distribute workload across nodes via Zookeeper. It was brought to our attention during Scott Andreas’s tech talk at Gilt Groupe (another one !), and it might be the topic of another blog post.