Formed in 2009, the Archive Team (not to be confused with the archive.org Archive-It Team) is a rogue archivist collective dedicated to saving copies of rapidly dying or deleted websites for the sake of history and digital heritage. The group is 100% composed of volunteers and interested parties, and has expanded into a large amount of related projects for saving online and digital history.
The adoption of IT technologies has increased exponentially over the last few decades. Nowadays, the need to create quality web applications while simultaneously managing their extensive data effectively is a major concern for any organization. As such, there have been continuous efforts to come up with better approaches to software development, data analysis and data management.
In order to meet the needs of today’s internet users, technology companies have intensified their efforts and are increasingly making use of all their available resources to find progressively efficient ways to create web applications within a shorter time-space and with superior quality, and to find more effective ways to integrate data-driven decision-making.
In order to accomplish these feats when it comes to application development as well as data pipelining management, most of the larger organizations are now following a set of practices, DevOps and DataOps, respectively. Within this environment, developers must create applications that can function across various hosted providers, each with its own specific administrators and requirements.
This can only be accomplished by looking past the traditional storage capabilities. Instead, developers and organizations they work for are creating an underlying data fabric that facilitates what they’re trying to do, and any cloud provider they want to do it with.
An Introduction to DevOps and DataOps
DevOps and DataOps are a set of practices that relates to software development and data management, respectively.
These practices can be applied to a single project or can be followed at an organizational level. DevOps emphasizes the automation of the processes of the different teams (software development team, quality assurance team and IT operational team) so that they can easily complete their relative tasks (i.e., design, code, create, check, deliver, deploy, operate and monitor).
DataOps, on the other hand, can be described as DevOps that includes data analysis, which empowers businesses to create processes to meet the needs of users throughout any data usage life cycle.
DevOps emphasizes the cooperation of the software development team (including testers) with system administrators (operational teams). In a similar way, DataOps relies on the cooperation of teams of data engineers, data scientists and other data practitioners who manage data-related processes (i.e., data storage, data processing, data decision-making, data archiving or data delivery).
The concept underscores both communication and collaboration between the different teams within the same organization, ultimately removing all obstacles for seamless integration between the practices and the end product, including cross-device compatibility.
The combined team can make the project more visible and deliver faster results. It also increases the morale of the company by showing that, when it comes to project management, people will still be above processes and resources.
Benefits of DevOps
The benefits of DevOps include:
- Transparency — By regular communication and coordination with the development team, the QA team and the IT operations team will be updated with the latest project details and will be more aware of the processes of other departments.
- Quicker and high-quality deployment cycles — On any great DevOps platform, the pace of deployment will be improved by applying DevOps practices. The scope of the teams will be limited (i.e., for a small task/module) for a deployment period, as such their efficiencies will increase.
- Faster bug fixes — Because tasks are smaller and continually distributed on the appropriate servers, the development team can receive input very easily from the QA team, allowing the development team to provide much faster resolution of any non-working components. Application stability would be a by-product of this method, because the cause behind any bug can be established very easily.
- Flexibility — Because agile methodology insists on managing the raised change request, teams aligned with DevOps will be able to manage any unplanned work more effectively.
Benefits of DataOps
Now, let’s take a look at the benefits of DataOps:
- Faster data processing — When different teams cooperate and interact on a daily basis and work on micro databases rather than on all at once, they not only avoid data overwhelm, they also accomplish much faster processing of data at all the different stages.
- Quicker problem solution — Similarly to DevOps, businesses can benefit from DataOps in multiple ways. DataOps data is processed in batches as this helps teams to recognize issues related to a batch of data faster as the emphasis is limited.
- Better data security — We can protect our data in the cloud by using a range of safety features such as two-factor authentication and virtual private networks, the best of which provide a greater level of security than is available merely through your ISP. As data security professionals are in constant communication with data integrators and data analysts in a DataOps scenario, data security at each process level is made easier.
- Improved resource output — When the team conforms to the basic principles of DataOps, seamless communication and transparency between the different teams will help the performance of each team member and lead to better resource output.
The Lessons We Can Learn from eharmony
The eharmony membership pool consists of singles in the U.S. as well as more than 200 countries around the world, making up about 5 million visitors per month.
The company is continuously bringing out new features such as their most recent one-on-one video communication feature aimed at helping their members to maintain meaningful connections in the current era of social distancing. But the company has come a long way since its debut in 2000, learning very valuable lessons along the way.
Overcoming Several Challenges
eharmony is leading the online dating industry through innovation by continuously improving its two key applications — Singles and Matching. But they’ve had to make significant changes on their journey to remain at the top. Their challenges included:
- The IT team needed a considerable amount of time to refresh the database, requiring the involvement of DBAs, storage specialists and network administrators.
- The labor-intensive process of creating, scanning and storing database snapshots made it difficult to create test datasets for new releases. As a result, QA often carried out tests using stale data.
- Database improvements triggered development freezes, which hindered the team’s ability to deliver new features and functionality.
- Their data volumes were increasing constantly, with several Oracle databases requiring between 20 to 70 TB of storage.
Combining DevOps and DataOps
eharmony has also implemented a self-service data management solution called Delphix, which enables the individual teams to develop and update environments as needed.
Today, their refreshes can be completed within an hour, providing accurate and relevant information to development teams on demand.
Other solutions included:
- They have saved over five thousand person-hours in just three years by automating database provisioning, refresh and rollback tasks. In the process, this has also saved the company over $3 million in operational costs.
- Through compression and virtualization, eharmony has reduced their database storage footprint to a fraction of the original size, reducing their storage requirements by almost 80%.
- As development teams adopt agile methods and more frequent releases, demand for non-production environments is expected to increase, further increasing storage cost savings. eharmony has also eliminated data related defects by providing accurate, up-to-date test data. Unrestricted access to test environments significantly improves the ability of QA teams to identify issues before they reach production.
Why DataOps and DevOps Are a Match Made in Heaven
It is easy to see why DataOps and DevOps go together so well. DevOps calls for continuous software monitoring, testing and deployment. DataOps are inherently modular because they are designed to execute a single function. Software that is scalable and easily fits into the DevOps framework.
Using DataOps in the DevOps process will help to fully recognize the benefits of continuous integration by validating the implementation promised by DevOps. These tools can be combined to optimize computing in a way that should be used during development whenever possible.
