Qiang Blog

Just another zhangjingqiang's blog.

How to build a BaaS platform?

It was my second time to take part in the DeNA study last night. It was wonderful because I started to pay attention to the BaaS(Backend as a Service).

Because of it was a topic from a game company, the BaaS is specially means Game BaaS.
There are many games will be released in a big game company every year. The general features are the same. So why don't make a common platform for all the games? It will reduce cost, save time, and so on.

For example, the platform image will be:

One Proxy Server

This layer is responsible for multi process, authentication, allocate for every API server, manage common API, etc.

By EventMachine

Common APIs

This is the most important layer. Be sure to do the restful architecture. Make the common features to gems.

  • Session API
  • Player API
  • Payment API
  • Asset API
  • Ranking API
  • Information API
  • MasterData API
  • etc

By Sinatra, Sequel and Unicorn.

Databases

  • Master-Slavel (for Game)
  • Master-Slavel Sharding (for Player)
  • Master-Slavel Sharding (for Ranking)
  • Master-Slavel Sharding (for Log)
  • etc

By MySQL and Q4M, also Redis.

One Management Tool

By Rails, Unicorn and SwitchPoint.


All the above are the topic about BaaS yesterday. All rights reserved by DeNA

References

api baas database game platform proxy ruby-on-rails sinatra

The Next Generation Website Workflow

Phase0 | The Given Condition 与件整理
Phase1 | Current Status Analysis 現状把握
Phase2 | Target Setting 目標の明確化
Phase3 | The User Experience ユーザ体験シナリオ
Phase4 | Strategic Planning 戦略策定
Phase5 | Tactics Planning 戦術策定
Phase6 | Contents Design コンテンツ制作
Phase7 | Prototyping Website Design サイト設計
Phase8 | Prototyping System Design システム設計
Phase9 | Development 開発・制作・テスト
Phase10 | Data Injection データ投入
Phase11 | Operation & Management PDCA サイクル
Plan, Do, Check, Action

workflow

Install Ruby on Rails 3.2 on Ubuntu 12.04

Install Git

> sudo apt-get install git

Install packages

> sudo apt-get install curl g++
> sudo apt-get install zlib1g-dev libssl-dev libreadline-dev libyaml-dev libxml2-dev libxslt-dev libsqlite3-dev nodejs

Remove RVM

> rvm implode

Install rbenv

> sudo apt-get install rbenv
> echo 'eval "$(rbenv init -)"' >> ~/.bashrc

If you are using ZSH,change .bashrc to .zshrc,please.

Install ruby-build

> git clone git://github.com/sstephenson/ruby-build.git
> cd ruby-build
> sudo ./install.sh

Install ruby

(Close your terminal and start it again)

> rbenv install 1.9.3-p194
> rbenv rehash
> rbenv global 1.9.3-p194

Confirm setting

> rbenv version

Confirm ruby's path

> which ruby

Confirm ruby's version

> ruby -v

Create .gemrc file

vim ~/.gemrc

install: --no-ri --no-rdoc
update: --no-ri --no-rdoc

Install Ruby on Rails 3.2

> gem install rails -v 3.2.0
> rbenv rehash

Create work folder

> cd
> mkdir work

Create confirm application

> cd work
> rails new foo --slip-bundle
> cd foo

Install depend versions

> bundle install

Generate simple user management function

> rails g scaffold user name:string email:string
> rake db:migrate

Start application

> rails s

Confirm by browser

http://0.0.0.0:3000/users

Reference

ruby-on-rails ubuntu

在GAE上开发Python程序

创建APP

https://appengine.google.com/

安装SDK

$ wget http://googleappengine.googlecode.com/files/google_appengine_1.7.1.zip
$ unzip aoogle_appengine_1.7.1.zip

官方下载主页

https://developers.google.com/appengine/downloads

创建简单的应用

$ mkdir -p ~/workspace/gae_helloworld
$ mv google_appengine ~/workspace/gae_helloword/
$ cd ~/workspace/gae_helloworld

创建 app.yaml 文件(gae_helloworld的根目录下)

    application: helloworld
    version: 1
    runtime: python27
    api_version: 1
    threadsafe: true

    handlers:
    - url: /.*
      script: helloworld.application

创建 helloworld.py 文件(同样在根目录下)

def application(environ, start_response):
    """包含 environ, start_response 的函数
    """
    start_response('200 OK',[('Content-Type', 'text/plain')])
    return 'Hello, world!'

确认运行效果

$ google_appengine/dev_appserver.py .

# 指定端口
$ google_appengine/dev_appserver.py . --port=5000

发布

$ google_appengine/appcfg.py update .

访问 http://helloworld.appspot.com/ 确认运行效果。

参考资料

Pythonプロフェッショナルプログラミング第12章

google-app-engine python

在全新Ubuntu系统搭建Python开发环境

假如你在VirtualBox上安装了Ubuntu Server,用SSH连接的话,查看进程:

$ ps aux | grep sshd

验证SSH是否启动。

没有安装的话,执行下面命令:

$ sudo apt-get install ssh

设置语言

$ sudo locale-gen ja_JP.UTF-8
$ sudo dpkg-reconfigure locales

设置.bashrc中语言设定用的环境变量

export LANGUAGE=ja_JP:ja
export LANG=ja_JP.UTF-8

安装Python

安装apt包

$ sudo apt-get -y update
$ sudo apt-get -y upgrade
$ sudo apt-get -y install build-essential
$ sudo apt-get -y install libsqlite3-dev
$ sudo apt-get -y install libreadline6-dev
$ sudo apt-get -y install libgdbm-dev
$ sudo apt-get -y install zlib1g-dev
$ sudo apt-get -y install libbz2-dev
$ sudo apt-get -y install sqlite3
$ sudo apt-get -y install tk-dev
$ sudo apt-get -y install zip

安装Python相关的包

# 安装python-dev
$ sudo apt-get -y install python-dev

# 安装Distribute
$ sudo chmod -R 0775 /usr/local
$ sudo chgrp -R bpbook /usr/local
$ wget http://python-distribute.org/distribute_setup.py
$ sudo python distribute_setup.py

确认Python版本

$ python -V

安装easy_install和pip

$ wget https://raw.github.com/pypa/pip/master/contrib/get-pip.py
$ sudo python get-pip.py

确认pip的版本

$ pip --version

安装virtualenv

$ pip install virtualenv

确认virtualenv的help

$ virtualenv --help

virtualenv的用法

确认安装的包

$ pip freeze

用virtualenv创建虚拟环境

$ export VIRTUALENV_USE_DISTRIBUTE=true
$ mkdir ~/work
$ cd ~/work
$ virtualenv env

执行虚拟环境

$ source env/bin/activate

确认虚拟环境的版本

(env) $ pip freeze
distribute==0.6.24
wsgiref==0.1.2

退出虚拟环境

(env) $ deactivate

virtualenvwrapper

安装virtualenvwrapper

$ pip install virtualenvwrapper

设置.bashrc

if [ -f /usr/local/bin/virtualenvwrapper.sh ]; then
  export WORKON_HOME=$HOME/.virtualenvs
  source /usr/local/bin/virtualenvwrapper.sh
fi

重新加载.bashrc

$ source ~/.bashrc

确认mkvirtualenv的帮助

$ mkvirtualenv --help

创建虚拟环境

$ mkvirtualenv newenv

确认虚拟环境

$ ls -la $HOME/.virtualenvs

执行虚拟环境

$ workon newenv

虚拟环境一览

$ workon

删除虚拟环境

$ rmvirtualenv newenv

适用指定版本的Python

$ mkvirtualenv --python=/usr/bin/python2.7 newenv2

确认虚拟环境的Python

(newenv2) $ python -v

设置Mercurial

安装Mercurial

$ sudo pip install mercurial

确认Mercurial

$ hg --version

创建库

设置.hgrc的用户名和邮件地址

[ui]
username=bpbook <bpbook@beproud.jp>
editor = vim

[extensions]
graphlog=
color=
pager=

[pager]
pager=LESS='FSRX' less

hg init(初始化库)

$ mkdir ~/hgtest
cd ~/hgtest
hg init

操作文件

hg status(确认状态)

$ touch test.txt
$ hg status

? test.txt

hg add(添加文件)

$ hg add test.txt
$ hg status

A test.txt

hg commit(提交)

$ hg commit 

test commit

...

确认编辑器

$ echo $EDITOR
vim

hg diff(确认区分)

$ hg status
M test.txt

$ hg diff

diff -r 23742103740 test.txt
...

hg revert(取消编辑)

$ hg revert test.txt

hg rollback(取消提交)

$ hg rollback
$ hg status

A test.txt

hg log(确认提交日志)

$ hg log

编辑器

Vim

编辑.vimrc

Python用的设定

syntax on
filetype plugin indent on

准备配置Python的文件

$ mkdir ~/.vim
$ mkdir ~/.vim/ftplugin
$ touch ~/.vim/ftplugin/python.vim

把下面的内容添加到python.vim文件中

setl expandtab
setl tabstop=4
setl shiftwidth=4
setl softtabstop=4

行尾空格删除的设置

autocmd BufWritePre * :%s/\s\+$//ge

一行80文字换行的设置

setlocal textwidth=80

Emacs

创建.emacs文件

$ touch ~/.emacs

缩进空格

(setq indent-tabs-mode nil)
(global-font-lock-mode 1)

Good Wiki

开发Python的Tips

感应模式

$ python
>>> import sys
>>> sys.path
...

安装IPython

$ pip install IPython

安装pep8(编码样式检查)

$ pip install pep8

安装pyflakes(语法检查/模型)

$ pip install pyflakes

pdb(调试器)

添加pdb的代码

def add(x,y)
  return x + y

x = 0
import pdb; pdb.set_trace()
x = add(1, 2)

执行pdb的例子

$ python pdbtest.py
> pdbtest.py(7)<module>()
-> x = add(1, 2)
(Pdb)

The Zen of Python

>>> import this

参考资料

Pythonプロフェッショナルプログラミング第1章

emacs mercurial python ubuntu vim

开发WEB应用程序的流程

Chinese

  • 1.从需求明确必要的功能。
  • 2.从功能明确必要的页面。
  • 3.制作页面。
  • 4.实现功能。
  • 5.连接页面和功能。
  • 6.确认运行效果。
  • 7.完成。

Japanese

  • 1.要件から必要な機能を明らかにする。
  • 2.機能から必要な画面を明らかにする。
  • 3.画面を作る。
  • 4.機能を作る。
  • 5.画面に機能を組み込む。
  • 6.動作を確認する。
  • 7.完成。

workflow

项目开发中必要的文档

中文

  • 模型的安排:关于系统技术方面的接口
  • 设计上的决定事项:设计或者架构的重要的决定记录
  • 面向重要的人的概述说明:系统的构想、需求,现在的金额估算、风险、人员规划、日程安排
  • 运行文档:运行时必要的信息、操作顺序、环境的规定
  • 项目概述:开发时必要的信息、操作顺序、环境的规定
  • 要求文件:定义系统应该是什么
  • 培训材料:如人员的支持、故障排除、维修队伍的联系人名单等
  • 系统文档:系统概述、架构和要求事项等
  • 用户文档:用户手册、培训教材等

日本語

  • 取り決めモデル:システムの技術的インターフェースについて
  • 設計上の決定事項:設計やアーキテクチャの重要な決定の記録
  • 偉い人向け概要説明:システムの構想・要件、現在の見積り、リスク、人員計画、スケジュール
  • 運用ドキュメント:運用に必要な情報、手順、環境の要約
  • プロジェクト概要:開発に必要な情報、手順、環境の要約
  • 要求文書:システムが何をすべきかを定義する
  • サポートドキュメント:サポート担当者向けのトレーニング資料、トラブルシュート、保守チームの連絡先一覧など
  • システムドキュメント:システム概要、アーキテクチャ、要求事項などの概要
  • ユーザードキュメント:取り扱い説明書、トレーニング資料など

document

Setting Up Ruby on Rails Production Server With Nginx and Unicorn

  • OS: Ubuntu 12.04
  • Nginx: 1.1.19
  • Unicorn: 4.8.3
  • Rails: 4.2

Step 1: Set up unicorn

# Gemfile
gem 'unicorn'

# From the command line, in your application's root directory.
bundle

# Add to ~/.bashrc or ~/.bash_profile
PATH=./bin:$PATH

# Run under Rails application folder
bundle --binstubs

Configuring config/unicorn.rb

vim config/unicorn.rb
# Set the current app's path for later reference. Rails.root isn't available at
# this point, so we have to point up a directory.
app_path = File.expand_path(File.dirname(__FILE__) + '/..')

# The number of worker processes you have here should equal the number of CPU
# cores your server has.
worker_processes (ENV['RAILS_ENV'] == 'production' ? 4 : 1)

# You can listen on a port or a socket. Listening on a socket is good in a
# production environment, but listening on a port can be useful for local
# debugging purposes.
listen app_path + '/tmp/unicorn.sock', backlog: 64

# For development, you may want to listen on port 3000 so that you can make sure
# your unicorn.rb file is soundly configured.
listen(3000, backlog: 64) if ENV['RAILS_ENV'] == 'development'

# After the timeout is exhausted, the unicorn worker will be killed and a new
# one brought up in its place. Adjust this to your application's needs. The
# default timeout is 60. Anything under 3 seconds won't work properly.
timeout 300

# Set the working directory of this unicorn instance.
working_directory app_path

# Set the location of the unicorn pid file. This should match what we put in the
# unicorn init script later.
pid app_path + '/tmp/unicorn.pid'

# You should define your stderr and stdout here. If you don't, stderr defaults
# to /dev/null and you'll lose any error logging when in daemon mode.
stderr_path app_path + '/log/unicorn.log'
stdout_path app_path + '/log/unicorn.log'

# Load the app up before forking.
preload_app true

# Garbage collection settings.
GC.respond_to?(:copy_on_write_friendly=) &&
  GC.copy_on_write_friendly = true

# If using ActiveRecord, disconnect (from the database) before forking.
before_fork do |server, worker|
  defined?(ActiveRecord::Base) &&
    ActiveRecord::Base.connection.disconnect!
end

# After forking, restore your ActiveRecord connection.
after_fork do |server, worker|
  defined?(ActiveRecord::Base) &&
    ActiveRecord::Base.establish_connection
end

Test config

bin/unicorn -c config/unicorn.rb

Step 2: Set up MySQL Database

sudo apt-get install mysql-server mysql-client libmysqlclient-dev

Step 3: Set up Unicorn Init Script

sudo vim /etc/init.d/unicorn
#!/bin/sh

# File: /etc/init.d/unicorn

### BEGIN INIT INFO
# Provides:          unicorn
# Required-Start:    $local_fs $remote_fs $network $syslog
# Required-Stop:     $local_fs $remote_fs $network $syslog
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: starts the unicorn web server
# Description:       starts unicorn
### END INIT INFO

# Feel free to change any of the following variables for your app:

# ubuntu is the default user on Amazon's EC2 Ubuntu instances.
USER=vagrant
# Replace [PATH_TO_RAILS_ROOT_FOLDER] with your application's path. I prefer
# /srv/app-name to /var/www. The /srv folder is specified as the server's
# "service data" folder, where services are located. The /var directory,
# however, is dedicated to variable data that changes rapidly, such as logs.
# Reference https://help.ubuntu.com/community/LinuxFilesystemTreeOverview for
# more information.
APP_ROOT=/home/vagrant/testapp
# Set the environment. This can be changed to staging or development for staging
# servers.
RAILS_ENV=production
# This should match the pid setting in $APP_ROOT/config/unicorn.rb.
PID=$APP_ROOT/tmp/unicorn.pid
# A simple description for service output.
DESC="Unicorn app - $RAILS_ENV"
# Run unicorn command
CMD="unicorn -D -c $APP_ROOT/config/unicorn.rb -E $RAILS_ENV"
# Give your upgrade action a timeout of 60 seconds.
TIMEOUT=60

# Store the action that we should take from the service command's first
# argument (e.g. start, stop, upgrade).
action="$1"

# Make sure the script exits if any variables are unset. This is short for
# set -o nounset.
set -u

# Set the location of the old pid. The old pid is the process that is getting
# replaced.
old_pid="$PID.oldbin"

# Make sure the APP_ROOT is actually a folder that exists. An error message from
# the cd command will be displayed if it fails.
cd $APP_ROOT || exit 1

# A function to send a signal to the current unicorn master process.
sig () {
  test -s "$PID" && kill -$1 `cat $PID`
}

# Send a signal to the old process.
oldsig () {
  test -s $old_pid && kill -$1 `cat $old_pid`
}

# A switch for handling the possible actions to take on the unicorn process.
case $action in
  # Start the process by testing if it's there (sig 0), failing if it is,
  # otherwise running the command as specified above.
  start)
    sig 0 && echo >&2 "$DESC is already running" && exit 0
    su - $USER -c "$CMD"
    ;;

  # Graceful shutdown. Send QUIT signal to the process. Requests will be
  # completed before the processes are terminated.
  stop)
    sig QUIT && echo "Stopping $DESC" exit 0
    echo >&2 "Not running"
    ;;

  # Quick shutdown - kills all workers immediately.
  force-stop)
    sig TERM && echo "Force-stopping $DESC" && exit 0
    echo >&2 "Not running"
    ;;

  # Graceful shutdown and then start.
  restart)
    sig QUIT && echo "Restarting $DESC" && sleep 2 \
      && su - $USER -c "$CMD" && exit 0
    echo >&2 "Couldn't restart."
    ;;

  # Reloads config file (unicorn.rb) and gracefully restarts all workers. This
  # command won't pick up application code changes if you have `preload_app
  # true` in your unicorn.rb config file.
  reload)
    sig HUP && echo "Reloading configuration for $DESC" && exit 0
    echo >&2 "Couldn't reload configuration."
    ;;

  # Re-execute the running binary, then gracefully shutdown old process. This
  # command allows you to have zero-downtime deployments. The application may
  # spin for a minute, but at least the user doesn't get a 500 error page or
  # the like. Unicorn interprets the USR2 signal as a request to start a new
  # master process and phase out the old worker processes. If the upgrade fails
  # for some reason, a new process is started.
  upgrade)
    if sig USR2 && echo "Upgrading $DESC" && sleep 10 \
      && sig 0 && oldsig QUIT
    then
      n=$TIMEOUT
      while test -s $old_pid && test $n -ge 0
      do
        printf '.' && sleep 1 && n=$(( $n - 1 ))
      done
      echo

      if test $n -lt 0 && test -s $old_pid
      then
        echo >&2 "$old_pid still exists after $TIMEOUT seconds"
        exit 1
      fi
      exit 0
    fi
    echo >&2 "Couldn't upgrade, starting 'su - $USER -c \"$CMD\"' instead"
    su - $USER -c "$CMD"
    ;;

  # A basic status checker. Just checks if the master process is responding to
  # the `kill` command.
  status)
    sig 0 && echo >&2 "$DESC is running." && exit 0
    echo >&2 "$DESC is not running."
    ;;

  # Reopen all logs owned by the master and all workers.
  reopen-logs)
    sig USR1
    ;;

  # Any other action gets the usage message.
  *)
    # Usage
    echo >&2 "Usage: $0 <start|stop|restart|reload|upgrade|force-stop|reopen-logs>"
    exit 1
    ;;
esac

Change permission:

sudo chmod +x /etc/init.d/unicorn

Let unicorn starts on reboot:

sudo update-rc.d unicorn defaults

Step 4: Nginx configuration

Installing nginx

sudo apt-get install nginx

It's unnecessary to change /etc/nginx/nginx.conf

Configuring /etc/nginx/sites-available/sitename

sudo vim /etc/nginx/sites-available/testapp
upstream unicorn {
  server unix:/home/vagrant/testapp/tmp/unicorn.sock;
}

server {
  listen 80;
  server_name localhost; # Replace this with your site's domain.

  keepalive_timeout 300;

  client_max_body_size 4G;

  root /home/vagrant/testapp/public; # Set this to the public folder location of your Rails application.

  try_files $uri/index.html $uri.html $uri @unicorn;

  location @unicorn {
          proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
          proxy_set_header Host $http_host;
          proxy_set_header X-Forwarded_Proto $scheme;
          proxy_redirect off;
          # This passes requests to unicorn, as defined in /etc/nginx/nginx.conf
          proxy_pass http://unicorn;
          proxy_read_timeout 300s;
          proxy_send_timeout 300s;
  }

  # You can override error pages by redirecting the requests to a file in your
  # application's public folder, if you so desire:
  error_page 500 502 503 504 /500.html;
  location = /500.html {
          root /home/vagrant/testapp/public;
  }
}

Add a symlink to enable site:

# Enter the sites-enabled folder
cd /etc/nginx/sites-enabled

# Add a symlink to your configuration file
sudo ln -s ../sites-available/testapp

# Reload nginx. Make sure to use the `reload` action so that nginx can check
# your configuration before reloading, thereby saving you from causing downtime.
sudo service nginx reload

Check the processes of unicorn and nginx:

sudo service unicorn status
sudo service nginx status

Step 5: Set rails production secret key

Generate a key with rake:

rake secret

Copy the key to config/secrets.yml:

production:
  secret_key_base: <%= ENV['SECRET_KEY_BASE'] %>

Step 6: Open your site

In the Vagrantfile:

vim Vagrantfile
VAGRANTFILE_API_VERSION = "2"
Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
  config.vm.box = "precise32"
  config.vm.box_url = "http://files.vagrantup.com/precise32.box"
  # config.vm.network :forwarded_port, guest: 80, host: 8080
  config.vm.network :private_network, ip: "192.168.33.10"
  # config.vm.network :public_network
end

Reload vagrant and login again:

vagrant reload
vagrant ssh

Access your site from browser:

http://192.168.33.10/

If there are something wrong, delete /etc/nginx/sites-available/default and /etc/nginx/sites-enabled/default then try again.

Reference

http://www.gotealeaf.com/blog/setting-up-your-production-server-with-nginx-and-unicorn

nginx ruby-on-rails ubuntu unicorn vagrant