about

rss

Quick multipart post for nodejs

Mostly because I wanted to experiment, but also because I add hard time finding one that just work. My “reinvent the wheel” contribution to post a form/multipart via nodejs:


events = require 'events'

class Multipart extends events.EventEmitter
  constructor: ()->
    @boundary_length = 40
    @random_boundary_part_length = 12
    @boundary = @get_boundary()
    @crlf = '\r\n'
  get_boundary: ()->
    # algorithm borrowed from curl Curl_FormBoundary
    allowed_characters = "0123456789abcdef" 
    boundary = ''
    for i in [1..@boundary_length-@random_boundary_part_length]
      do (i)->
        boundary += '-'
    for i in [1..@random_boundary_part_length]
      do (i)->
        boundary += allowed_characters[Math.floor(Math.random() * 16)]
    boundary
  write: (string)->
    @emit 'data', string
  add_parameter: (name, value)->
    @write "--#{@boundary}" 
    @write @crlf
    @write "Content-Disposition: form-data; name=\"#{name}\"" 
    @write @crlf
    @write @crlf
    @write value
    @write @crlf
  end: ()->
    @write "--#{@boundary}" 
    @write "--" 
    @emit 'end'

module.exports = Multipart

Intended to be use like that:


https = require 'https'
Multipart = require './multipart'

m = new Multipart()
query = ''

m.on 'data', (data)->
  query += data

m.on 'end', ()->
  console.log 'sending request'
  console.log query

  req = https.request {port: 443, host: 'graph.facebook.com', method: 'POST', path: '/', headers: {'Content-Length': query.length, 'Content-Type': "multipart/form-data; boundary=#{m.boundary}"}}, (res)->
    res.setEncoding('utf8')
    console.log res.statusCode
    console.log res.headers
    res.on 'data', (chunk)->
        console.log('BODY: ' + chunk)
  req.write query
  req.end()

m.add_parameter 'access_token', 'bla'
m.add_parameter 'batch', '[{"method":"GET","relative_url":"/me"},{"method":"GET","relative_url":"/me"}]'

m.end()

Using a scriptable proxy to test remote API without hurting them

Sometimes when you’re using an API provided by a partner/client, and even if your code is well tested, you want to be sure that the first runs won’t hurt the other side datas. In a REST context, it means that you can let GET requests query the API while keeping an eye on PUT, POST, DELETE requests. Still, it would be nice to have a response for those requests too.

Basically you need a scriptable proxy which will let go GET through and handle the other verbs without querying the API.

For this purpose I have used em-proxy, which provides a scritable proxy based on eventmachine.

For the impatients, here the code:


#! /usr/bin/env ruby
require 'set'
require 'rubygems'
require 'em-proxy'
require 'unicorn'
require "addressable/uri" 

class MyFakeWeb
  @@pairs = Set.new
  @@counters = Hash.new(0)

  def self.register_uri(verb, url, options = {})
    @@pairs << MyFakeWebPair.new(verb, url, options[:body])
  end

  def self.answer_to(verb, url)
    hash = @@pairs.classify{|p| [p.verb, p.url.path]}
    parsed_url=Addressable::URI.parse(url)
    key = [verb.downcase, parsed_url.path]
    response = hash[key] ? hash[key].to_a[@@counters[key] % hash[key].size] : nil
    @@counters[key] += 1
    response ? response.body : ''
  end
end

class MyFakeWebPair
  attr_accessor(:verb,:body)
  attr_reader :url

  def initialize(string_verb, string_url, string_body)
    self.verb = string_verb.downcase
    self.url = string_url
    self.body = string_body
  end

  def url=(string_url)
    @url = Addressable::URI.parse string_url
  end
end

MyFakeWeb.register_uri('post', 'http://production.com/bla', :body => 'hey1')
MyFakeWeb.register_uri('post', 'http://production.com/bla', :body => 'hey2')
MyFakeWeb.register_uri('post', 'http://production.com/bla', :body => 'hey3')
MyFakeWeb.register_uri('post', 'http://production.com/bla', :body => 'hey4')

Proxy.start(:host => "0.0.0.0", :port => 8080, :debug => true) do |conn|
  conn.server :production, :host => 'production.com', :port => 80

  conn.on_data do |data|
    parser = Unicorn::HttpParser.new
    headers = parser.headers({}, data.clone)
    if %w(POST PUT DELETE).include? headers['REQUEST_METHOD']
      http_string = StringIO.new
      body = MyFakeWeb.answer_to(headers['REQUEST_METHOD'], headers['REQUEST_URI'])
      Unicorn::HttpResponse.write(http_string, [200, {'Content-Length' => body.length}, body])
      conn.send_data http_string.string
      data = nil
    else
      data.gsub!(/Host: .*?\r\n/, "Host: production.com\r\n")
    end
    data
  end

  conn.on_response do |server, resp|
    resp
  end
end


MyFakeWeb and MyFakeWebPair classes are juste meant to encapsulate fake responses.

The all filtering process happens in the Proxy block.
  • It parses the request with the HttpParser provided by Unicorn
  • determine the HTTP verb and queried path
  • either let it go through to the remote API (just changing the Host header to be sure that the right server will handle it) or answer with a fake response.

That’s it!

A quick and dirty collectd plugin for the Varnish cache


#!/usr/bin/env ruby

require 'rubygems'
require 'eventmachine'
require 'getoptlong'

opts = GetoptLong.new(
  [ '--hostid', '-h', GetoptLong::OPTIONAL_ARGUMENT ],
  [ '--sampling-interval', '-i',  GetoptLong::OPTIONAL_ARGUMENT ]
)

def usage
  puts("#{$0} -h <host_id> [-i <sampling_interval>]")
  exit
end

sampling_interval = nil
hostname = `hostname`.chomp

opts.each do |opt, arg|
  case opt
    when '--hostid'
      hostname = arg
    when '--sampling-interval'
      sampling_interval = arg.to_i
  end
end

def underscore(camel_cased_word)
  camel_cased_word.to_s.gsub(/::/, '/').
  gsub(/([A-Z]+)([A-Z][a-z])/,'\1_\2').
  gsub(/([a-z\d])([A-Z])/,'\1_\2').
  tr("-", "_").
  tr(" ", "_").
  downcase
end

EM.run do
    EM.add_periodic_timer(sampling_interval||2) do
    now = Time.now.to_i
    stats = `varnishstat -1` 
    stats.each_line do |line|
      #client_conn            211980         0.30 Client connections accepted
      next unless /^(\w+)\s+(\d+)\s+(\d+\.\d+)\s(.+)$/.match(line)
      puts("PUTVAL #{hostname}/varnish/counter-#{underscore $4} #{now}:#{$2}")
    end
    end
end

Copy this code in /usr/lib/collectd/varnish.rb

Then edit your collectd configuration to add (or uncomment) (don’t forget to change the user)


LoadPlugin exec

<Plugin exec>
  Exec "your_user" "/usr/lib/collectd/varnish.rb" 
  #Exec "user:group" "/path/to/exec" 
  #NotificationExec user "/path/to/exec" 
</Plugin>

A simple backupninja script for mongodb

After reading this stuff about corruption of mongodb http://ivoras.sharanet.org/blog/tree/2009-11-05.a-short-time-with-mongodb.html, I thought it was time to write a small script to backup our brand new mongodb databases at YeastyMobs

Here is the handler (in /usr/share/backupninja/mongodb):

getconf host localhost
getconf backupdir
getconf mongodump_path /opt/mongodb/bin/mongodump

info "Going to backup mongodb of $host in $backupdir" 

if [ ! -f $mongodump_path ]; then
        fatal "$mongodump_path does not exists" 
fi

if [ ! -d $backupdir ]; then
        fatal "$backupdir does not exists" 
fi

if [ ! -w $backupdir ]; then
        fatal "$backupdir is not writable by you" 
fi

if [ ! $test ]; then
        output=`$mongodump_path -h $host -o $backupdir`
        code=$?

        if [ "$code" == "0" ]; then
                debug $output
                info "Successfully finished dump of all mongodb databases" 
        else
                warning $output
                warning "Failed to dump all mongodb databases" 
        fi
else
        debug "running $mongodump_path -h $host -o $backupdir" 
fi


and the backup script (in /etc/backup.d/10-action.mongodb):


backupdir = /var/backups/mongodb
mongodump_path = /opt/mongodb/bin/mongodump

DB

10+ Deploys Per Day: Dev and Ops Cooperation at Flickr

ruby + arduino

Fuzzycom le tumblelog de Vincent Hellot