As a former DBA, I find a disturbing trend toward a value proposition that is almost nonexistent among a recent crop of database administrators. Maybe having some background and/or working with other stellar DBAs in the past has spoiled me, but here’s the workflow I’ve find more and more common.
scenario – production application has slowed down for a few transaction types, dynatrace shows a critical sql statement has slowed down. None of the development team has access to run explains, we can’t “afford” hardware to load the production dataset into another environment (because we’re using a DBMS that costs 13.6 bajillion dollars per CPU nanosecond with an additional upcharge of 1 million pounds sterling every time we execute a query that uses DML… Explains indicate everything is optimal in the lower environments. The decision to use this particular platform after the salesman for the product took the DBA team to Vegas for a “conference” and after tough negotiations (forever documented by the excellent ‘based on a real life story’ movie: “The Hangover”).
Step 1 (optional) DBA team notices the query is slow also…email from DBA team to application team
“Dear application team, while eating donuts and sipping single malt whiskey this morning, I accidentally hit a button on this funny looking device sitting on my desk and the monitor in front of me popped up a report indicating this sql is pretty slow, thought I’d let you know. Please fix ASAP as I believe we’re wearing our our disk spindles and they’re VERY expensive. Also, do you know how to close the window with this report…it overlaid the Rugby World Cup streaming live in HD and I really want to see the ‘All Blacks’ win!”
Step 2 (optional step one) email from application team to DBA team
“Dear DBA team, we’ve (also) noticed this query is very slow. In every other environment it runs fine (less than 5 milliseconds), and it uses the primary key on every table, we’re unclear why it takes 10 minutes to complete production. Can you investigate?”
Step 3 email from DBA team to application team
“Dear Application Team – as I mentioned before, this query is slow and my children’s college fund is contingent on our database using less than 20 IOPS under peak load, fix this IMMEDIATELY! upon investigation I think if you removed that query the system run much better…I will note that your application is causing our database to use a lot more cpu and IO than when we initially powered the system up. Obviously you don’t know how to write software as prior to your application coming online the 3 ‘hello world’ applications we used to test the database platform didn’t cause any problems like this. Just because you collect data from sensors around the globe and provide real time data to thousands of users simultaneously doesn’t mean you can just slap crappy SQL in and expect it to run well. Please let us know if you need any assistance writing application software as we’re clearly much more intelligent than you.”
Step 4 followup email from DBA team to application team
“Dear Application Team – we further noticed you’re heavily using the database during our backup window from 1am to 8am eastern. It’s critical the system not be used during this time window as we have a team of interns backing up the entire database on 3.25″ floppies. Please tell your users not to use the system during this window or remove these SQL statements ASAP. Also, do you know a good torrent client? I really need to catch up on the last season of ‘Game of Thrones’ PS did you see the latest news on CNN, evidently a lot of users of your system are complaining about about performance during peak online shopping hours in Europe…Hope you figure out what you did wrong when writing your application software.”
Step 5 application team changes system to use BDB running on an old android device connected to the internet via a 2g wifi hotspot.
…sets up elaborate VPN solution to enable application servers to use this database from production data center…problem goes away
Step 6 entire DBA team is promoted for “saving the company millions of dollars by optimizing key SQL queries
Evidently, this is the new normal. Apologies to any DBAs who might, in fact, have been more helpful or proactive in helping to solve the problem.
| Reference: | The (slightly tongue in cheek) role of the database administrator from our JCG partner Mike Mainguy at the mike.mainguy blog. |
I enjoyed this cheeky story. I think this sort of relationship between application and DBA teams is symptomatic of the “over the wall” approach to software development. The application team writes the software, tosses it over the wall the DBA team, and then they both bicker and finger point when problems crop up.