What is DevOps?
There is no definitive answer, only lots of opinions about what is covered under DevOps and what’s not. It Born of the need to improve IT service delivery agility, the DevOps movement emphasizes communication, collaboration and integration between software developers and IT operations. Rather than seeing these two groups as silos who pass things along but don’t really work together, DevOps recognizes the interdependence of software development and IT operations and helps an organization produce software and IT services more rapidly, with frequent iterations.
First, whole point of devops is that change the culture of dev and ops. Yeah but the reality is that many companies want to designate someone as the DevOps engineer. Usually a more accurate title for that person would be Automation Engineer or something along those lines but we work within the constraints we are given.
Common DevOps mistakes when scaling
When it come to scaling, most companies are pretty decent at scaling up infrastructure, and pretty awful at scaling up code (at least on the infrastructure side of things i.e Ops.) Lot of people taking a 2000 line mentality into a 20000 line project. Some companies have engineers that can code a large architecture and keep scalability in mind but they get hamstrung by their ops people.
The main issue when scaling up is cultural change. When the organization is small, everything relies on one go-to guy under a lot of pressure to work quickly. Choices are made which aren’t always the best ones or the most scalable and every fix is a quick and dirty job. Naturally, documentation is an afterthought.
When the same organization grows, the same person is likely very insecure about the choices he or she has made. Collaboration with the new guys is often an issue.
Management also often makes the common mistake of pushing to retrofit automation tools before standardization and process which creates as many problems as it solves and introduces new risks. Design your infrastructure in modules as soon as possible. Make it so each module can easily be replaced added or removed as needed. Also document stuff and don’t feed the IT hero culture.
Other common pitfalls would be,
- Premature optimization.
- Reinventing the wheel… in a completely non-scalable way.
- Not using indexes/foreign keys.
- Fashion driven development (mongo, hey!)
- Not creating a stateless infrastructure where possible
- Not throwing hardware at a problem first (is your database server a bit slow? have you tried just installing an SSD/upgrading the RAM?)
- Not doing performance testing before optimizing
- Not admitting that used schema sucks
- Not admitting that the method use for the database sucks, not the database itself. Yes, the problem is us, it is not the database.
- Thinking that things are easy.
- Wanting to solve problems with new technologies with fancy names.
- Jumping into those technologies without researching them.
- Not hiring a devops person, but instead getting me a developer to act like a sysadmi
- Taking prototypes turning them into production, then complaining when they crash. Prototypes which were explicitly stated as things that would-not-scale.
Fashion driven development really hits home. Its understandable how it happens. I mean, who doesn’t want to be an early adopter of the latest framework, DB etc…. But in the real world, you really need to have an actual data driven argument for why the new system is actually measurably better than the tried and true. Most engineers learn this eventually.
Exposing too much information about your environment is another problem. People start to hardcode around that information and it makes scaling hard. For example work environment where have three data centers: west, central and east. Servers in those data centers have a number associated with its location in the hostname. I have seen far too much code wired to this information. If we added/removed a location there would be a ton of refactoring. Staying ambiguous makes changing architecture and technology easier.
Every year DevOps deal with at least a few clients where the founder knows that their infrastructure is buggy and insecure, but there’s often a main “First Hire” developer that is very reluctant to let you look under the hood, ostensibly because they are embarrassed by what’s there. Dont judge, because those decisions get made, often under extreme pressure from people who may or may not understand the technical risks and tradeoffs. Hopefully DevOPs will be able to close the gap somewhat for founders so they can make better informed decisions regarding this stuff and maybe even work more harmoniously with their technical people.