Potential F5 Vulnerability

It all started with an email about a WebInspect report. It listed a buffer overflow, which we had marked as a false positive. I read the WebInspect report carefully, and found a note at the bottom that said you could test manually to confirm whether it was a false positive or not. Unfortunately, the manual test listed had a few problems. First, it jammed the lines together, without the proper line-breaks. Second, it assumed the site was using HTTP, not HTTPS, so used telnet. Third, it was testing against a page that didn’t exist, giving a 404. Keeping those in mind, I tried the manual test using the openssl s_client command:

openssl s_client -quiet -connect mysite.example.com:443
POST /login HTTP/1.1
Host: mysite.example.com
Transfer-Encoding: chunked

c00000000

The test terminated immediately. The report said that this meant that the server was vulnerable to a buffer overflow and arbitrary code execution.

At first, we thought this was caused by Apache or Tomcat, or possibly the application code. But the reported vulnerability was an Apache CVE from 2002 (CVE-2002-0392), fixed by vendors long ago. After a while, we realized that if we hit the servers directly, we did not get the indication of a vulnerability. If we hit the site through the F5 VIP, we saw the immediate termination of the connection. The issue is with handling of HTTP chunked encoding. Nginx had a similar issue in 2013 (CVE-2013-2028).

So we turned our attention to the F5 load balancers. We were able to confirm that other sites using F5 load balancers were exhibiting the same behavior. We also confirmed that WebInspect run against the servers directly did not show the issue (even as a false positive). We reported the issue to F5, and they are looking into it.

I’m disclosing this publicly now for a few reasons. First, I’m not a security researcher, and almost nobody follows my blog — especially people looking for security issues that could be exploited. Second, I’ve not developed a payload, so I have no idea whether this is exploitable. But at this point, it’s merely a potential vulnerability. I’m not sure I’ll even spend the time to research and create a payload to prove the vulnerability. If I do, I’ll be more careful with the disclosure.

 

Resolutions

January kept me pretty busy, so I’m a little late to this. But better late than never. And as an Agile practitioner, I don’t think personal retrospectives should be limited to one time of year.

Review of 2014

Last year I wrote a blog entry listing my goals for 2014. As far as New Year’s resolutions go, I was relatively successful — about 50% of my goals accomplished. Unfortunately, my Open Source contributions weren’t as strong as I had hoped; while I released some of my own work, I didn’t do much else. I did increase my blogging; getting in on a weekly blogging pact helped immensely. I also increased my participation on the This Agile Life podcast to a level that I’m happy with. But the accomplishment I’m most proud of was giving a presentation at RubyConf.

Plans for 2015

I’d like to keep things rolling from last year, but crank up a few things. My plans are quite ambitious, so I don’t expect to get everything done by any means. But I think by setting the bar high, I’ll end up with a lot I can be proud of.

Job Hunting

Late last year, I took the jump into independent consulting. So far, I’ve really enjoyed it, and I’m booked up through April. My wife graduates in May, so we’ve got the possibility of moving if that makes sense. So I’ll be looking for consulting projects in town, but I’ll also be looking at jobs in San Francisco and Chicago. The possibilities are exciting, and I’ll be taking my time to find something just right.

Conferences

I was incredibly nervous leading up to my RubyConf presentation. Part of that was just the common fear of public speaking. For me, that only kicks in at around 100 people, and this audience was around 250. I think another reason was that I chose a really ambitious topic, and I kept finding more that I wanted to talk about, but wasn’t prepared for. But I think I did a pretty good job presenting an advanced topic. And I was so pumped by the sense of accomplishment as soon as I finished. So I’m hoping to do it more. I’ve already submitted a couple proposals, and plan to submit several more.

Blogging

I believe that blogging is important for me to get my thoughts down — for myself and to share with others. I was really successful last year when I had a partner to keep me honest, via a pact. So I’ve started up another pact this year, which will hopefully ensure I’ll keep things going. I’ve got a really long backlog of topics, so as long as I keep at it, I’ll have plenty to write about.

I also want to move away from WordPress to a static system — probably Middleman. I’ve got 2 major problems with WordPress. First, I no longer trust its security, nor the security of any application written in PHP. Second, it generates HTML every time someone requests a page, instead of when the content is updated. I find that to be a waste of resources, and problematic from a security standpoint. The main problem with moving to a static blogging system is that I really want to allow comments, pingbacks, and tweetbacks. So I’ll have to find a way to integrate those.

Programming Language Design

Last year I started thinking about programming language design, and started implementing a language tentatively called Brilliant. I’ve done a lot more thinking on the topic, and have a lot of notes. But I haven’t implemented much more yet. This year, I’d like to get my thoughts more organized, and write a series of blog posts on various aspects of language design. The most interesting part seems to be the trade-offs involved in the ways that various language features interact. So I’d like to make some progress on the language implementation, but more importantly, I’d like to get a lot of my design ideas written down.

I’m also going to spend a lot of time learning a bunch more programming languages, so I have a better understanding of possible features, combinations of features, and their interactions. I’ve already start with Elixir, Clojure, and Racket. I’m hoping to also look at OCaml, Factor, and Haskell. I’ll probably also take a look at the 2 “Seven Languages in Seven Weeks” books.

Agile Book

I think people often have trouble getting started with Agile. I started on a book last year, and got down quite a lot of good ideas. But I realized that I’m going to have a hard time organizing all those ideas into something coherent. Still, I’d like to try to get something out there that lets people get started with Agile. My idea is to present a toolbox of practices to get started with and build on that foundation over time with additional practices. Sort of a playbook on how to get started over the first 6 to 12 months and be successful. I want to make some progress on the book, at least enough to decide whether it’s worth the effort to finish it and self-publish it.

 

Empathy

I facilitated our team retrospective this morning. I felt like we made a little forward progress, but not as much as I would have liked. But it really brought one thing to the forefront of my thoughts today — empathy gained through communication.

We have a pretty large team by Agile standards — we had 20 people in our retro: 16 developers, 3 QA folks, and 1 manager. Out of those, only about 5 or 6 speak up regularly. I recently sent out a survey to the team, trying to get feedback on how we could improve our retros. A couple of the questions tried to get a feel for why people aren’t speaking up more. Only about half the people responded, and the answers didn’t really answer my question as well as I had hoped.

So on Amos‘s suggestion, we did the Safety Check exercise. We got a good set of answers to why people don’t feel safe. About half of the answers were about the fear of looking stupid in front of other people. About half of those mentioned the manager — they’re worried they might get in trouble for what they say. We talked some about fear and how it’s more often than not misplaced. And that the worst consequences are usually not as bad as you might think. But then we got to the crux of the problem — there’s not enough trust amongst the team, and especially between the team members and the manager.

About half of our team is new (within the past 6 months) — including the manager. While the developers have made some good progress building trust amongst each other, we haven’t had as much time with the manager to build trust between him and the rest of the team. So the lack of trust isn’t at all surprising.

Honestly, I already knew we had trust issues, and wanted to address them, but needed a way to lead the team to that same realization. With this exercise driving out the issue, we were then able to have a conversation about trust. The conversation was pretty good. We got more voices to contribute than probably any other topic we’d previously covered. (I was disappointed that the manager kept quiet though. I later found that he was trying to mitigate people’s fears by keeping quiet, but I urged him to contribute more in the future.)

But one point really stood out in my mind — a point of view that I hadn’t previously thought much about. Lauren, one of our QA analysts, pointed out that most human communication is non-verbal. We give tons of cues via body language, facial expressions, eye contact, tone of voice, even posture. I don’t recall if Lauren said it explicitly, but she pointed out that these cues build empathy between the speakers. She encouraged us to use more voice chat and video chat, as opposed to IRC text chat, because it would create more empathy between the people communicating, which would lead to more trust.

I spent most of the rest of the day talking to people on the phone or via Google Hangouts voice. And every single time, I consciously noticed that I was gaining empathy for the person I was speaking with. I assume (and hope) that that’s working both ways. I suppose that it’s always been happening, but I never really noticed it.

I’ve heard a lot of talk about empathy among Agile practitioners lately. It’s been mentioned on the Ruby Rogues podcast, and Angela Harms has been preaching it for years. I already understood how important it is. But until today, I didn’t really feel it happening.

So if you’re looking to build trust with someone, spend some time talking with them. Preferably in person, but if that’s not possible, seriously consider video or voice modes of communication, instead of sending an email or an instant message.

 

What I Want in a Blog Engine

I’m considering moving away from WordPress, for a couple reasons:

  • Security. There have been several vulnerabilities over the past few years, and I’ve never really had a high level of confidence that it’s secure. In addition, I now find the whole PHP model — every web app running as a single user, instead of leveraging UNIX permissions — to be broken.
  • Speed. I started to realize a couple years ago that a blog engine should generate static pages. The static pages should be updated whenever new content is added. There’s really no reason to re-generate the page every time someone wants to read it. At the very least, Server-Side Includes (SSI) or Edge-Side Includes (ESI) should be used.

Now that I’m starting to actually blog consistently, it makes sense to change. But before I move to something different, I want to be sure that I find all the features that I need. This post is my thought process about what I want.

Requirements

  • Self hosted. I like to run my own servers. Partly so I can keep full control, and partly because I enjoy doing some system admin stuff (as long as it doesn’t take too much time).
  • Generation of static pages. Almost everything should be a static page by the time a user requests a page. Obviously, submitting comments would hit a dynamic page, and comment approval would likely be done through a dynamic part of the site. But publishing a blog post or comments should generate static files to be served by Apache or nginx.
  • Categories (or maybe tags). I want readers to be able to find other things that I’ve written that might interest them. A list of categories in a sidebar would help them find what interests them.
  • Site map. Especially an XML sitemap, so Google can index better. I also like the idea of readers being able to see everything in one place.
  • Archives. This is another way of letting readers see everything I’ve written, organized by publication date.
  • RSS and/or Atom. I want readers to be able to subscribe to my blog.
  • Media. I don’t think I necessarily need a media manager, just a simple way to add pictures to blog posts.
  • Comments. I want to allow comments, but I don’t want to use anything like Disqus. I want the comments to be in my own database, so they won’t disappear. I also dislike requiring JavaScript just to see comments. And I’ve worked at places where Disqus wouldn’t work, due to web proxy servers. Comments should be moderated by the site admin, editor, or author. Readers should not have to log in to post a comment, but it would be nice to allow users to log into Twitter or some other social network to provide their info.
  • Pingbacks and Tweetbacks. I’d like to be able to see who’s referring to my articles.

Nice to Have

  • Free. I almost consider this to be a requirement, but I’d be willing to entertain a 1-time payment for something that meets my needs really well. I need to try to counteract my DIY-at-all-non-monetary-costs mentality.
  • Open Source. I think Open Source is likely to provide a more useful product, especially in this space. An Open Source option is more likely to improve over time.
  • Not PHP. I’ve got several problems with PHP. First, the whole PHP model of running in the same context as Apache (or nginx) breaks UNIX conventions. A security issue in a PHP program exposes more than just the program itself — it exposes Apache and every other PHP program. Under the UNIX model, each program runs in a different user context, isolating bugs to that program only. PHP is also a “hacky” language. PHP programmers tend to not be as rigorous (in many ways), leading to more vulnerabilities. Finally, the PHP language itself has had several misfeatures that have lead to security issues — much more than Ruby or Java.
  • Not MySQL. I’d like to be able to remove MySQL from my servers. PostgreSQL would be much preferred. Storing everything in git would be fine too — maybe even preferred. I’d even be fine with MongoDB or some other simple NoSQL database.
  • Admin console. I can’t imagine approving comments without this, but I’m not closed to that possibility.
  • Edit history. I want to be able to update posts, but see what the previous versions looked like. (Not sure if I care whether the public can or cannot see the various revisions.)
  • WYSIWYG. I’d be willing to settle for Markdown. In fact, I’d prefer (extended) Markdown as the canonical form behind the WYSIWYG.
  • Code highlighting. A lot of my blog posts (will) include code in various languages. I’d like the blogging software to be able to handle that well. Syntax highlighting would be great. Any other related features would also be nice.
  • Through-the-web editing. I’d like to be able to edit a page from any device from anywhere that has Internet access.
  • Reader-submitted edits. This is probably going to be the hardest thing to find, but I’d like a really easy way for readers to let me know of typos and such by simply submitting a fix for them. The author or editor of the post would have to approve the edits, obviously.
  • Import from WordPress. I don’t have a ton of content in WordPress, or else this would probably be a more firm requirement.
  • Community. Community helps software thrive, adding improvements and ensuring security.
  • Plugins. It would be nice for the blog engine to be extensible.
  • Unpublished drafts and previews. I’d really like to be able to see by blog posts before I release them to the world.
  • CMS abilities. I’d really like the ability to edit non-blog web pages the same way as blog entries.
  • Themes. It would be nice to have decently looking themes, so I could have other people help me make the site look nice.
  • Posting my tweets in a sidebar area. I’d be OK with this being done in client-side JavaScript. I’d also be OK with periodic polling of my Twitter feed, and updating the static content when appropriate.

Candidates

I’ve been playing with Jekyll lately, using it to generate a simple static site that can be updated easily. It’s pretty nice, and pretty easy to use. Something based on Jekyll would work well. The trickiest part will be figuring out how to manage comments. Or maybe I’ll just have to learn to live more minimally.

Octopress is the obvious candidate. It fits most of my needs and wants. Most likely I’ll write my own commenting engine and add it to my Octopress template.

The only other candidate that I’ve found is Ghost. It’s written in JavaScript and runs on Node. I don’t find Node to be that compelling, but the templates look great, and it’s designed for coding blogs.

I’ll probably take a quick look at Droplets, too. It’s written in PHP, but seems to fit my other requirements.

Unfortunately, that seems to be all there is in this space. Maybe I need to investigate more. Or maybe Octopress just has the static blog generation marked cornered. Either way, I’ll probably change my blogging software in the next month or two. Wish me luck!