DevOps Chat: Crowdsourced Continuous Testing with Rainforest's Derek Choy

Scaling your testing is the scourge of many software teams. While test automation and continuous testing have made great strides, it still is difficult to automate every QA test. Also, engineering resources who can script these tests are difficult to find. Enter Rainforest.

Transcript

Alan Shimel: Hey, everyone, it’s Alan Shimel, DevOps.com, and you’re listening to another DevOps Chat. Today’s DevOps Chat features a first-time guest on DevOps Chat. He’s Derek Choy, CEO at Rainforest, or Rainforest QA, as some may know them. Derek, welcome.

Derek Choy: Thank you. Just a little correction: I’m CIO at Rainforest.

Shimel: What did I call you?

Choy: Responsible for primarily the technical side of the functions here.

Shimel: Did I call you “CEO”?

Choy: I think you did, but I just want to make a little correction there.

Shimel: You know, it’s 2019; that’s my prediction.

[Laughter]

Choy: I don’t know that I would support that prediction, but thank you. [Laughs]

Shimel: Well, I _____ – CIO’s enough. But, anyway, Derek, thanks. Yes, it’s Derek Choy, CIO at Rainforest. So, Derek, you know what? I didn’t make the mistake on purpose; it was an honest mistake. Good. That leads us – why don’t we talk a little bit about Rainforest for our audience who may not be familiar?

Choy: Absolutely. So Rainforest is a testing platform. What we do is we help our customers with their software testing. You know, as you know, we’re moving – a lot of companies are moving towards a rapid development cycle, CI/CD, frequent releases and stuff. Testing becomes much of a challenge for a lot of our customers because now they not only have to do testing, but they have to do it fast. They have to have accurate results and stuff.

So Rainforest is actually a crowd-based testing platform that allows our customers to simply input test cases in plain English, so no Selenium or any sort of programming language, and then we can actually execute them as if they are automated test. We’re fully integrated into any CI/CD tools, like Jenkins or CircleCI and so on, so you can actually trigger a Rainforest test. Typically, we can return results in 30 minutes or so, and you can actually test across many different platforms, like different browsers, different OS’s, and the same for mobile as well. And so that allows our customers a wide variety of options, in terms of where they want to run their plain-English test cases and when they want to run it – it’s on demand. And we typically return results in 30 minutes, fully integrated into an automated release cycle.

Shimel: Great. Excellent. And, Derek, how did it come to be that you became the CIO of Rainforest? Let’s hear a little bit your own personal journey.

Choy: Cool. Oh, thank you. So I have been always involved in the product engineering side of things. I started off my career earlier with Accenture. Spent a number of years at eBay. And then my last company at Aria Systems, building SAS-based billing system, also running their engineering and product function. So, coming to Rainforest, I brought my experience and knowledge in this space, to help build the engineering and the product organizations. Now here, besides product engineering, I’m also responsible for our professional services team and our client success team. So essentially kind of helping build out product and making sure that our customers are successful in using that.

Shimel: Absolutely. So, Derek, you know what? Here we are in December. We’re looking at 2019.

Choy: Mm-hmm.

Shimel: I probably have spent the last three weeks – if I tell you I’ve spoken to dozens of vendors and C-level execs at large organizations, and, to me, the real story of 2019, as well as what’s emerged as the story of 2018, is around – you know, you wanna call it “digital transformation” and “DevOps,” _____ say great, but, really, down to these two things: companies are – they need scale in what they’re doing today, right?

Choy: Yeah, absolutely.

Shimel: When we’re talking about data and computation and transactions or whatever the metric is that’s driving your company, everybody wants to do it at huge scale, like unprecedented scale that, 20 years ago, you would’ve went – you know, I’m just reminded of Carl Sagan saying, “Billions and billions of stars,” _____.

Choy: [Laughs] Yes, that’s right.

Shimel: That’s the scale. And the only way to achieve that kind of scale is automation.

Choy: Yes. Absolutely.

Shimel: So these are two kind of twins, co-joined at the hip – scale and automation.You can’t have scale, I think – well, you can’t handle scale without automation.

Choy: Absolutely. Yeah. And – sorry.

Shimel: Go ahead. No, no, go ahead.

Choy: I –

Shimel: You know, to me, that’s it right there. That’s the crux of it.

Choy: Exactly. And, Alan, I would say that I’ve never been more excited, right, than where we are right now. I think we are definitely at a very exciting time and, just to kinda set the context and maybe kinda recapping a little bit of what you just said, right, the world is very different now, even compared to five years ago. Right? A while ago, you would hear companies having a quarterly release. Some would even have an annual release, if you remember, especially the bigger companies, right? You know, release cycle was a lot longer, takes a lot more time to get things out to the market. Today, release cycles are typically maybe in a month, right? Maybe two weeks and, in some cases, days or even multiple times a day. Right?

And exactly to you – and the interesting thing is that the world has got accustomed to that and has come to expect that, frankly. Right? Think about the apps that you’re using. If they come back to you and say, “Well, you know what? We update our apps once every six months,” or something, you’re like, “Wait, six months? Wow, that’s unheard of.” Right? “I was expecting a weekly update or at least monthly or something.” So the world has actually comes to expect that.

Now the challenge, exactly as you pointed out, is how do companies deal with that? Right? How can we scale our organization through automation, among other things, right, to make sure that we can keep up with that pace? I think that that is the challenge with a lot of technical leaders, is we understand what the companies are looking for, understand what the consumers and our customers are looking for, right? But how do we do that? Right?

And there’re different aspects of it, Alan. I think that there is definitely a development aspect of it, right? How do we scale development? How do we do rapid development, Agile, sprint? You know, all these things. Things have evolved a lot over the last, I would say, ten years, in terms of faster development. There’s a lot _____ –

Shimel: On the development side?

Choy: Exactly. And there continue to be more evolution on that, which is super exciting. Now one thing that hasn’t really, at least in my opinion, gone as far in this particular context is testing. Right? If you think about testing, we’ve had automated testing for a long time – 10, 20 years, maybe even longer, right? We’ve always had manual testing, right?

Now, when you think about testing, that’s – which is a critical part of the release cycle. Of course, right? You cannot push things out without some sort of testing at least, right? There hasn’t really been a lot of innovations and evolutions in testing. We’re still trying to do automated testing; we’re still doing some manual testing; we’re still doing it kinda the same way, so to speak. There’re new automated testing language, but they all kinda do similar things, right? And they all have similar –

Shimel: Absolutely. And, Derek, let me stop you there for a second.

Choy: Yeah.

Shimel: To me, here’s the real crux in the issue.

Choy: Yeah.

Shimel: We talked about that we’ve certainly seen, let’s call it, a “revolution” in development acceleration, rapid development over, let’s say, 10 years. I’ll say even 20 years, right? _____ _____ _____ around for about 20 years.

Choy: Agreed.

Shimel: Do you consider testing part of development? Or did DevOps come along and say, “Wait a second, it’s all one continuity here, with development, testing, operations, security – you wanna put security next to testing, whatever – but it’s one continuum”?

Choy: Yeah. It’s definitely – I completely agree with that. It is definitely one continuum, meaning that they are all part of the overall workflow, right, to make automation possible, to make a fast release process possible. You cannot do it without any one of those components, right?

Shimel: No. But they haven’t kept pace with the changes that we’re in – what we consider when we talk development.

Choy: Yes.

Shimel: Where many of us think, you know, of coding.

Choy: Exactly.

Shimel: That’s the issue right there.

Choy: That’s the issue, exactly as you pointed out, right? All the components of that whole continuum hasn’t really kept pace, right, with the developments that we’ve made on coding, right? On the actual coding of the product. And I think that that’s starting to show or actually has been starting to show, in terms of how other components of that, testing in particular, has been frankly dragging down the overall speed of being able to push things out faster and so on, so forth. Right?

If you talk to – and you do that all the time. If you talk to a lot of CIOs and CTOs and so on, frequently, you’ll hear challenges with testing. Right? And most people will tell you, “Yeah, I’m okay. I get it going,” but very few people –

Shimel: “It’s good enough.”

Choy: Yeah, it’s just good enough, but it’s not great, right?

Shimel: No, it’s not.

Choy: Yeah.

Shimel: And, Derek, I’ll go one further with you. What I see is this, is that, when we talk about testing – and _____ _____ the same thing with security, right? Is we’ve got twin challenges.

Choy: Yes.

Shimel: I can’t do it fast enough, so I’m constantly behind, running as fast as I can, but I can’t – just not fast enough.

Choy: Exactly.

Shimel: And then, number two, the train won’t wait for me.

Choy: Yeah. Exactly.

Shimel: If I can’t finish my testing, “Well, you did it good enough.”

Choy: Yeah. “Let’s go.”

Shimel: “We’ve gotta release. We’ve gotta release this hour,” or, “We’ve gotta release this week.” You didn’t do – dirty testing? That’s okay; it’s probably secure anyway. And, boom, out the door it goes, right? And that’s the dilemma, the enigma of testing, is “I can’t do it fast enough, but, if I don’t do it fast enough, they don’t wait for me.”

Choy: Exactly. Right. And there is a reason for that. Exactly, right? Like we talked about earlier, because customers actually expect releases, right, whether you’re ready or not, right? They are expecting releases in a very frequent cycle. Right? And so companies therefore cannot wait. Right? It is not acceptable to go back to the customer and say, “You know what? My release is delayed and therefore you have to wait another week or two.” In the past, that used to be the common thing to do, right? Nowadays, that’s not acceptable.

And so then what happen, exactly like you said, is that people compromise, right? They’re like, “Oh, you know what? Testing is not done. Okay, well, you know what? It should be fine. Let’s just go.” Right? “Security testing is not complete, but we’ve done a portion,” right? “Okay, well, let’s just go.” Right?

The downside, however, is that – and you’re seeing it frequently; some people are just not paying attention to it – is that there is a high price to pay for that, right? You’ve seen a lot of security loopholes out there, right? And, yes, the release is out, right, but then at what cost, right? If people found a loophole and there is a data breach and costs companies lots of moneys and reputations and things like that.

And equally with software quality as well, right? Things don’t work. Even bigger companies, you’ve heard companies like Apple, for example, struggle with that. Google, right, struggle with that. And they release something and people like, “Wow, this is junk,” right? “And I don’t want to upgrade,” and you hear that all the time, right? And it tarnish your reputation.

Now bigger companies have, obviously, a lot more buffer to kind of weather these kind of quality problems, right? You know, whether you like it or not, you’re gonna continue to use Apple, right? Or Android or whatever. But then smaller businesses may have less tolerance for that, right? And customers of those businesses, if they’re not happy with your quality, they’ll move somewhere else.

So I think it’s very important for technical leaders to figure that out and not just kinda brush it aside and say, “You know what? It’s not important. If quality’s not good enough, let’s just go. It’s fine. We’ll find bugs in production; we’ll fix it.” Personally, I don’t think that that’s the right approach and I think that it’s gonna hurt businesses down the road.

Shimel: So you know what I call that? So that’s the double-edged sword of MVP.

Choy: Yeah.

Shimel: Right? ‘Cause you talk to these people and they say, “Don’t worry. We’re just doing an MVP. It’s a minimally viable product. We know it’s not 100-percent tested. We know it’s not a 100-percent baked. We know it doesn’t have everything, but we’ve gotta get an MVP out there, god darn it.” Right? [Chuckles] And so –

Choy: Yeah.

Shimel: And we appreciate – look, I’ve been – I founded, cofounded several venture-backed companies. I’ve been on pushing the VP of engineering, saying, “Hey, we promised our customer we had a new release coming out before Thanksgiving, man, and it has to get out. It has to get out. I don’t care if it’s fully tested. Get it out.”

Choy: Yeah.

Shimel: It’s not – you know, MVPs were being chewed to death.

Choy: Yeah. I – [laughs]

Shimel: So that “minimum” – that “M” in MVP keeps getting bigger and bigger and bigger. Right?

Choy: Exactly. Right, exactly.

Shimel: And more and more critical.

Choy: Exactly. I think we continue to push the boundary of MVP. You know, I think that – and that’s really the challenge, right? So, you being kind of the leader of the business, right, you’re not wrong because customers really need it, right? “In order to move my business forward, I really need the release. I really need to get that feature out. I really need to get ahead of the market.” That’s not wrong. Right? And I think that that’s the needs of the business; that’s the market; that’s what people expect today. The challenge that we technical leaders needs to solve – CTOs, CIO, VP of engineering, VP of product – what we need to solve is “How can we support the business in doing that but also maintain the quality?” Right?

And so, as a leader, in my opinion, going back to the business and say, “You know what? I can’t actually make your delivery date. You will have to wait another two weeks, four weeks, six weeks,” or something – that’s gonna be a tough sell, right? But, at the same time, releasing it without quality assurance, without security and all these stuff, is also not the right approach.

So I think what we leaders needs to figure out is “How can we do better, right, in QA? How can we do better in security?” All these other things that are probably less mature than development itself, coding itself, right, how can we do better in these areas so that we can also support the business in shipping this MVP? Whatever the definition of this MVP is, right? We can support a business in shipping the MVP because that’s very critical to the business but, at the same time, maintain the level of quality and the level of security and, among other things, right, that we are proud of, right? That we want to have, right?

And I think that that’s the biggest thing that we need to figure out in 2018 and definitely going into 2019, right? How fo businesses figure that out? How do we implement processes and tools and technologies that will allow us to do that?

Shimel: Yep. So, Warren – Warren? I’m sorry. Derek, fantastic discussion. Let’s bring it home a little bit to Rainforest.

Choy: Sure.

Shimel: What can you do to help?

Choy: Yeah, so Rainforest has a very – takes a very different approach to QA, right? You know, so the traditional way is, obviously, automated test with Selenium and all these different Appium, all these different technologies, or we can go manual testing and perhaps building an AMI offshore – that’s very, very typical as well. Rainforest looks at it and say, “Well, neither one are the best approach,” right? And therefore we come up with a testing platform, you know, that is powered by the crowd. And we allow our customers to actually put plain-English test cases on our platform. We can execute these tests without actually building an AMI offshore, but, at the same time, you don’t have to worry about the downside of automation, having to hire QEs and spend a lot of time writing Selenium scripts and things like that.

And so we offer a solution that, at least we think, solves problems on both sides, right, and also allows you to hook it up into your CI/CD process so that you can actually push this MVP out in an automated fashion, fully kinda connected with all the rest of the continuum, right, and be able to actually ship code faster but with higher quality.

Shimel: So let me ask, so, at some level, though, you are reintroducing or relying on human element?

Choy: Right. Correct.

Shimel: So why don’t you have the same scale problems that others do?

Choy: That is an excellent question. So I would say that there are two elements, right? So, when we look at the human elements, we look at it from a crowdsourcing basis, right? So think about Uber, for example, right? If I were to hire my own team of drivers, that would be a little bit challenging, but then, leveraging the crowd and scaling it based on demand, we’re able to actually solve a lot of the human aspect of the problem. We can have testers that come in whenever they like to come in and we have enough testers to support the test that our customers would like to do. Right?

And now the challenge there and probably a followup question then is “Well, but what about the quality of these people? Do they know what they’re doing?” Just like a Uber driver, right? How do you know that that guy actually drives well, right? So we actually developed a bunch of AI algorithms that actually analyze the work that they do and also take in feedback from customers, based on their former – the previous tests that they have done. And a combination of that with a lot of data mining and AI analysis helps our customers determine whether that test result is correct or not. If it’s not correct, we will engage additional testers to make sure that the tests are consistent and results are correct.

So we have refined that algorithm over the years and now we have achieved a very high level of accuracy that, based on our data, is showing us that the accuracy’s actually higher than having your own manual QA team. And so you get better results, without actually hiring an AMI, either onshore or offshore. You can actually run your test on demand – typically, you’ll get results in 30 minutes or so. Right? And it’s fully integrated into your CI/CD cycle. And the best part is you can do all these things and achieve all these benefits without having to hire QEs to write automated scripts and having to maintain it, and so on, so forth.

Shimel: Excellent. So, Derek, I get the outsource. I get the army of people doing it. But, on the back end, are you doing some automated testing as well?

Choy: Yes, we are. And so what we’re also doing is we’re developing additional AI algorithms to actually learn from these human executions. So what we do is we have lots of human executions that are happening as we speak. In our environment, we have 80 million, 90 millions of tests that have already conducted on our platform, so what we do is we actually do machine learning through these data and learn from previous executions and develop algorithms so that, instead of actually having to send it to a human tester, we can actually achieve it with a robot.

Very similar to the approach that Uber is taking. Today, human drivers are driving people around, right? In the back end, they’re developing these self-driving cars so that, in the future, we don’t really need a human drivers anymore. Very, very similar approach, from a QA standpoint. Today, we’re sending testers, but then, in the future, we’ll have these robots that will execute these tests based on what they have learned from these human testers.

And tomorrow is already here for us today because we’re already doing that and a number of our customers are already using these automated AI algorithms. So they just need to write their test cases in plain English and they’ll see the tests executed by robots automatically, without having to worry about building automations or anything like that. So that’s kind of our approach to solve the QA problem.

Shimel: So Rainforest, the Uber of QA.

Choy: I don’t know that marketing will approve that line, but –

Shimel: No, I don’t think _____ _____ –

Choy: – personally, but, yes. Yes.

Shimel: It does remind me – you know, when – you’re out in San Francisco, we’ve all been in venture-backed world.

Choy: That’s right.

Shimel: There was a period there where every pitch was “We wanna be the Uber of,” right?

Choy: Yeah. Exactly.

Shimel: Uber of this or the Uber of that and _____ _____ to be disruption, or crowdsource in this case, but fascinating take on solving an age-old problem.

Choy: Mm-hmm. Exactly.

Shimel: Interesting to see. Just – Rainforest is, what, about two-and-a-half years, three years old now?

Choy: Yeah, it’s just a little bit longer. We’ve been – our total existence, probably about six, seven years. We’ve kinda pivoted a little bit at the beginning, so our current product is about three to four years old.

Shimel: That’s when I think I first became aware of it.

Choy: That’s right.

Shimel: And, look, testing and continuous testing is such a – I mean, it’s a hot area. There’s a lot of different solutions vying in the market, but this one is really kinda unique, so it’ll be interesting to watch how it happens.

Choy: Yeah.

Shimel: Derek, as I promised you before we started, the time goes really fast here. We’re way out of time, but, hey, man, I wanna thank you for being a guest on DevOps Chat. And maybe we’ll catch up with you again in the new year and find out how things are going.

Choy: That will be awesome. Well, thank you, Alan, and have a wonderful holiday to you and everybody listening. And, hopefully, I’ll talk to you again soon.

Shimel: Absolutely. Derek Choy, CIO still, not CEO, of Rainforest. Thanks for being our guest on DevOps Chat today. Have a great holiday and thanks very much.

Choy: Thank you. Buh-bye.

Shimel: Buh-bye. This is Alan Shimel for DevOps.com and you’ve just listened to another DevOps Chat.

Jun	JUL	Aug
	18
2018	2019	2020

Transcript

Like this: