Corporations are increasingly including cloud services in their data architecture, says Gartner. As well they should. Compared to proprietary on-premises systems, cloud services are more easily scalable, reliably available and reasonably priced.
The concern that the IT community has felt over cloud data security is fading, too. In the care of full-time security professionals, the data security of cloud environments is often better than their in-house counterparts.
But there is one set of companies whose cloud adoption lags behind. Companies in regulated industries face an interesting conundrum. The companies themselves may be well aware that a cloud service is secure…
If we go back in history a few years, the typical data pipeline was an app creating events and pushing them to a backend database. Data was then propagated to downstream applications via dedicated ETL flows at regular intervals, usually daily.
In these modern times, Apache Kafka has become the default data platform. Apps write events to Kafka, which then distributes them in near-real-time to downstream sinks like databases or cloud storages. …
Software and systems, especially open source ones, should be accessible and easy to get started with. That is the bottom line.
If you’ve signed up for a trial of Aiven services, you probably want to be more productive quickly. Because that’s what we’re here for: letting developers do their job, while we take care of the less dazzling aspects. You want to quickly evaluate whether Aiven is a good fit for you (it probably is), and start planning how to set up a full implementation if it’s needed.
In other words, you’re not here so you can explore every aspect…
Disaster recovery is a strange topic in the world of security and compliance. There is a lot of conversation about it, but often little gets done in practical terms. This may be because, for most organisations, it’s perceived as too expensive or too remote. In normal circumstances, a true disaster is a High Impact — Low Frequency (HILF) event and something in the human psyche prefers to focus on Low Impact — High Frequency (LIHF) events. This is why you tend to worry more about burning dinner than getting hit by a meteorite.
When you design your data storage and…
Due to scalability and duplication issues, Kafka is engaged in the long process of breaking its dependency on ZooKeeper. In this article, we’ll look at how this will change the way that brokers communicate with each other and with Kafka as a whole.
Kafka and ZooKeeper today
When setting up a Kafka broker, there are three essential broker configurations that you need to define. The broker needs to be able to know and tell other system elements who it is (broker.id). It needs to know where to write topic data (log.dirs). And it needs to connect to ZooKeeper (zookeeper.connect) in order to join its…
Recently Aiven conducted a market study, and one question asked about the respondent’s company’s current cloud arrangements. Given that we’re quite familiar with the terrain, we were a bit startled that 61% of respondents stated that their data was stored in a private cloud. The same held true even in industries where we were pretty sure that couldn’t be the case.
SELECT 1; isn’t all it’s cracked up to be. As a test of verifying database availability, connecting to it and running one simple query is hardly a realistic workload. This simple operation can succeed while the server’s underlying disk system is having serious issues, or if backups have not been generated on schedule. In short, the database can respond to simple queries just fine but your service could still be down.
Aiven commits to 99.99% uptime, but we don’t calculate that figure based on simple availability checks. We believe our customers deserve better guarantees.
What Aiven does is use the…
Have you been to Venice? It’s soooo beautiful. There are hundreds of graceful bridges, countless churches and palaces from the awesome to the charming. The city’s canals, filled with iconic gondolas, stir the heart. If you pick your time right, you can even do your sightseeing without being mobbed by thousands of other tourists. In fact, outside of the tourist season, it’s a very quiet city.
Like… really quiet.
That’s because not a lot of people actually choose to live in Venice. The roads are uneven, the buildings aren’t readily wheelchair and pushchair accessible, the plumbing sucks and heating is…
Implementing your online, globally accessible service can seem daunting. Especially if your background is in the traditional industries, your first instinct may be to go DIY. You buy the hardware and software, set up the connections, install everything, and hope that everything comes together on launch day.
This is certainly a valid approach in certain circumstances. If you’re doing gene sequencing or storing sensitive surveillance data for the defence forces, your particular situation is rare or unique. Then it might make sense to build your own environment and store your own data at a location that no one can get…
When you’re setting up a microservice architecture, one of your first major decisions is whether to have the services communicate directly with each other, or whether to use a broker system. Since the broker model is more flexible and failure-resistant, it’s likely you’ve more or less decided to choose that. However, you may be concerned that the broker will become a bottleneck in the system if traffic is heavy.
Allow me to set your mind at ease by introducing you to Apache Kafka, a distributed, partitioned commit log service that acts as a messaging system but with a flair all…
