Qiang Blog

Just another zhangjingqiang's blog.

How to setup Local Infrastructure Development with Ansible and Vagrant by Ad-Hoc Commands

Make work directory

$ mkdir work && cd work

Make Vagrantfile

# -*- mode: ruby -*-
# vi: set ft=ruby :

VAGRANTFILE_API_VERSION = "2"

Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
  config.ssh.insert_key = false
    config.vm.provider :virtualbox do |vb|
    vb.customize ["modifyvm", :id, "--memory", "256"]
  end

  # Application server 1.
  config.vm.define "app1" do |app|
    app.vm.hostname = "orc-app1.dev"
    app.vm.box = "geerlingguy/centos7"
    app.vm.network :private_network, ip: "192.168.60.4"
  end

  # Application server 2.
  config.vm.define "app2" do |app|
    app.vm.hostname = "orc-app2.dev"
    app.vm.box = "geerlingguy/centos7"
    app.vm.network :private_network, ip: "192.168.60.5"
  end

  # Database server.
  config.vm.define "db" do |db|
    db.vm.hostname = "orc-db.dev"
    db.vm.box = "geerlingguy/centos7"
    db.vm.network :private_network, ip: "192.168.60.6"
  end
end

Run vagrant

$ vagrant up

Edit /etc/ansible/hosts

# Lines beginning with a # are comments, and are only included for
# illustration. These comments are overkill for most inventory files.

# Application servers
[app]
192.168.60.4
192.168.60.5

# Database server
[db]
192.168.60.6

# Group 'multi' with all servers
[multi:children]
app
db

# Variables that will be applied to all servers
[multi:vars]
ansible_ssh_user=vagrant
ansible_ssh_private_key_file=~/.vagrant.d/insecure_private_key

Setup

$ ansible multi -a "hostname"
$ ansible multi -a "hostname" -f 1
$ ansible multi -a "df -h"
$ ansible multi -a "free -m"
$ ansible multi -a "date"
$ ansible multi -s -m yum -a "name=ntp state=present"
$ ansible multi -s -m service -a "name=ntpd state=started enabled=yes"
$ ansible multi -s -a "service ntpd stop"
$ ansible multi -s -a "ntpdate -q 0.rhel.pool.ntp.org"
$ ansible multi -s -a "service ntpd start"
$ ansible app -s -m yum -a "name=MySQL-python state=present"
$ ansible app -s -m yum -a "name=python-setuptools state=present"
$ ansible app -s -m easy_install -a "name=django"
$ ansible app -a "python -c 'import django; print django.get_version()'"
$ ansible db -s -m yum -a "name=mariadb-server state=present"
$ ansible db -s -m service -a "name=mariadb state=started enabled=yes"
$ ansible db -s -a "iptables -F"
$ ansible db -s -a "iptables -A INPUT -s 192.168.60.0/24 -p tcp -m tcp --dport 3306 -j ACCEPT"
$ ansible db -s -m yum -a "name=MySQL-python state=present"
$ ansible db -s -m mysql_user -a "name=django host=% password=12345 priv=*.*:ALL state=present"
$ ansible app -s -a "service ntpd status"
$ ansible app -s -a "service ntpd restart" --limit "192.168.60.4"
$ ansible app -s -a "service ntpd restart" --limit "*.4"
$ ansible app -s -a "service ntpd restart" --limit ~".*\.4"
$ ansible app -s -m group -a "name=admin state=present"
$ ansible app -s -m user -a "name=johndoe group=admin createhome=yes"
$ ansible app -s -m user -a "name=johndoe state=absent remove=yes"
$ ansible multi -m stat -a "path=/etc/environment"
$ ansible multi -m copy -a "src=/etc/hosts dest=/tmp/hosts"
$ ansible multi -s -m fetch -a "src=/etc/hosts dest=/tmp"
$ ansible multi -m file -a "dest=/tmp/test mode=644 state=directory"
$ ansible multi -m file -a "src=/src/symlink dest=/dest/symlink owner=root group=root state=link"
$ ansible multi -m file -a "dest=/tmp/test state=absent"
$ ansible multi -s -B 3600 -a "yum -y update"
$ ansible multi -s -m async_status -a "jid=763350539037"
$ ansible multi -B 3600 -P 0 -a "/path/to/fire-and-forget-script.sh"
$ ansible multi -s -a "tail /var/log/messages"
$ ansible multi -s -m shell -a "tail /var/log/messages | grep ansible-command | wc -l"
$ ansible multi -s -m cron -a "name='daily-cron-all-servers' hour=4 job='/path/to/daily-script.sh'"
$ ansible multi -s -m cron -a "name='daily-cron-all-servers' state=absent"
$ ansible app -s -m git -a "repo=git://example.com/path/to/repo.git dest=/opt/myapp update=yes version=1.2.4"
$ ansible app -s -a "/opt/myapp/update.sh"

Reference

Ansible for DevOps(Chapter 3)

ansible vagrant

Kubernetes deploy Rails app to Google Container Engine

Create a Rails App

rails new todo
rails g scaffold task title:string notes:string due:datetime completion:integer

Config Gemfile

gem 'rails', '4.2.0'
gem 'sass-rails', '~> 5.0'
gem 'uglifier', '>= 1.3.0'
gem 'coffee-rails', '~> 4.1.0'

gem 'jquery-rails'
gem 'turbolinks'
gem 'jbuilder', '~> 2.0'
gem 'sdoc', '~> 0.4.0', group: :doc

group :production do
  gem 'pg'
end

gem 'capistrano-rails', group: :development

group :development, :test do
  gem 'byebug'
  gem 'web-console', '~> 2.0'
  gem 'spring'
  gem 'sqlite3'
end

Config config/database.yml

production:
  <<: *default
  adapter: postgresql
  encoding: unicode
  database: todo_production
  username: <%= ENV['PG_ENV_POSTGRES_USER'] %>
  password: <%= ENV['PG_ENV_POSTGRES_PASSWORD'] %>
  host:     <%= ENV['PG_PORT_5432_TCP_ADDR'] %>

Add Dockerfile at Rails Root

FROM rails:onbuild

ENV RAILS_ENV=production

CMD ["sh", "/usr/src/app/init.sh"]

Use official rails on docker hub: https://hub.docker.com/_/rails/

Make init.sh at Rails Root

export SECRET_KEY_BASE=$(bundle exec rake secret)

bundle exec rake db:create db:migrate
bundle exec rails server -b 0.0.0.0

Deploying Locally

docker build -t your_user_name/todo .

Run PostgreSQL Database

docker run --name db -e POSTGRES_PASSWORD=password -e POSTGRES_USER=rails -d postgres

Check Process

$ docker ps

CONTAINER ID        IMAGE               COMMAND                CREATED             STATUS              PORTS               NAMES
30e1420e8a57        postgres:9          "/docker-entrypoint.   7 seconds ago       Up 6 seconds        5432/tcp            db

Run Web App

docker run --name web -d -p 3000:3000 --link db:pg your_user_name/todo

Deploying with Kubernetes on Google Container Engine

Create a project on http://console.developers.google.com. Enable Google Container Engine API, Compute Engine API, and Cloud Storage API.

Use Google Cloud SDK do local config

gcloud config set project my-project-id
gcloud config set compute/zone asia-east1-a

Create Google Container Cluster

gcloud container clusters create todo
gcloud container clusters get-credentials todo

Building and Sharing the Container Image

docker build -t todo .
docker tag -f todo gcr.io/my_project_id/todo:v1
gcloud docker push gcr.io/my_project_id/todo:v1

Deploying the Database

Make Database Pod File

# db-pod.yml
apiVersion: v1
kind: Pod
metadata:
  labels:
    name: db
  name: db
spec:
  containers:
  - image: postgres
    name: db
    env:
    - name: POSTGRES_PASSWORD
      value: password
    - name: POSTGRES_USER
      value: rails
    ports:
    - name: pg
      containerPort: 5432
      hostPort: 5432

Create Database Pod

kubectl create -f db-pod.yml

Make Database Service File

#db-service.yml
apiVersion: v1
kind: Service
metadata:
  labels:
    name: db
  name: db
spec:
  ports:
    - port: 5432
      targetPort: 5432
  selector:
    name: db

Create Database Service

kubectl create -f db-service.yml

Deploying Rails

Make Web Controller File

# web-controller.yml
apiVersion: v1
kind: ReplicationController
metadata:
  labels:
    name: web
  name: web-controller
spec:
  replicas: 2
  selector:
    name: web
  template:
    metadata:
      labels:
        name: web
    spec:
      containers:
      - image: gcr.io/YOUR_PROJECT_ID_HERE/todo:v1
        env:
          - name: POSTGRES_PASSWORD
            value: password
          - name: POSTGRES_USER
            value: rails
        name: web
        ports:
        - containerPort: 3000
          name: http-server

Create Web Controller

kubectl create -f web-controller.yml

Make Web Service File

# web-service.yml
apiVersion: v1
kind: Service
metadata:
  name: web
  labels:
    name: web
spec:
  type: LoadBalancer
  ports:
    - port: 3000
      targetPort: 3000
      protocol: TCP
  selector:
    name: web

Create Web Service

kubectl create -f web-service.yml

Check pods, controllers, services on kubernetes

kubectl get pods
kubectl get rc
kubectl get services

Accessing the Running Site

Check Nodes

kubectl get nodes
NAME                          LABELS                                               STATUS
gke-todo-5ccd2616-node-3bj5   kubernetes.io/hostname=gke-todo-5ccd2616-node-3bj5   Ready
gke-todo-5ccd2616-node-7afx   kubernetes.io/hostname=gke-todo-5ccd2616-node-7afx   Ready
gke-todo-5ccd2616-node-fd7e   kubernetes.io/hostname=gke-todo-5ccd2616-node-fd7e   Ready

Create Firewall

gcloud compute firewall-rules create --allow=tcp:3000 --target-tags=gke-todo-5ccd2616-node todo

Get Address

gcloud compute forwarding-rules list
NAME                             REGION      IP_ADDRESS    IP_PROTOCOL TARGET
9ea34fd96220f1e15959942001a0f5ae asia-east1-a 104.197.XXX.XXX TCP         asia-east1-a/targetPools/9ea34fd96220f1e15959942001a0f5ae

Access IP_ADDRESS

http://104.197.XXX.XXX:3000/tasks


Thanks To

docker google-cloud-platform kubernetes

Deploy Rails with Docker

A sample using Rails + MySQL

Install Docker on Mac OS X

http://docs.docker.com/engine/installation/mac/

Create docker machine

$ docker-machine create --driver virtualbox testapp

Import ENV

eval "$(docker-machine env testapp)"

Create Dockerfile

FROM ruby:2.2.0
RUN apt-get update -qq && apt-get install -y build-essential mysql-client libmysqlclient-dev nodejs
RUN mkdir /testapp
WORKDIR /testapp
ADD Gemfile /testapp/Gemfile
ADD Gemfile.lock /testapp/Gemfile.lock
RUN bundle install
ADD . /testapp

Create docker-compose.yml

db:
  image: mysql
  environment:
    MYSQL_ROOT_PASSWORD: 1234
  ports:
    - "3306:3306"
web:
  build: .
  command: bundle exec rails s -p 3000 -b '0.0.0.0'
  volumes:
    - .:/testapp
  ports:
    - "3000:3000"
  links:
    - db

Modify config/database.yml

default: &default
  adapter: mysql2
  encoding: utf8
  pool: 5
  username: root
  password: 1234
  host: db

development:
  <<: *default
  database: testapp_development

test:
  <<: *default
  database: testapp_test

production:
  <<: *default
  database: testapp_production

Build image

$ docker-compose build

Boot app

$ docker-compose up

Create database

$ docker-compose run web rake db:create

Get docker machine's ip

$ docker-machine ip testapp

Access app

http://[IP_ADDRESS]:3000


Reference

https://docs.docker.com/compose/rails/

docker mysql ruby-on-rails

How to use git subtree

Example

repository 1

app1

app1
|-app
|-submodule/app2

repository 2

app2
|-something

What to do

After change something in submodule/app2 in repository 1, push the change to repository 2, too.

Command

# Under repository 1
git subtree push --prefix submodule/app2 --squash https://github.com/your_account/your_repository.git your_branch

When to use

Such as rails engine

http://guides.rubyonrails.org/engines.html

git

How to use vim 7.4 after install it by brew on MacOS

Problem

After install vim 7.4 by brew:

brew install vim

Type brew info vim, will show:

vim: stable 7.4.891, HEAD
Vi "workalike" with many additional features
http://www.vim.org/
Conflicts with: ex-vi
/usr/local/Cellar/vim/7.4.891 (1612 files, 28M) *
  Built from source
From: https://github.com/Homebrew/homebrew/blob/master/Library/Formula/vim.rb
...

But if type vim, will show:

VIM - Vi IMproved
  version 7.3
...

The reason is the PATH.

Solution

Install macvim:

brew install macvim

Add these settings to ~/.bashrc or ~/.zshrc:

alias vi='/Applications/MacVim.app/Contents/MacOS/Vim'
alias vim='/Applications/MacVim.app/Contents/MacOS/Vim'
alias gvi='/Applications/MacVim.app/Contents/MacOS/Vim -g'

Then type vim will use 7.4 version.

Reference

https://www.reddit.com/r/vim/comments/2ukp5j/starting_homebrew_version_of_vim/

mac vim

How to start Google Cloud Platform

Install SDK

https://cloud.google.com/sdk/

$ curl https://sdk.cloud.google.com | bash
$ exec -l $SHELL
$ gcloud init

Config SSH

$ gcloud compute config-ssh

Will generate these files under ~/.ssh:

➜  .ssh  tree
.
├── config
├── google_compute_engine
└── google_compute_engine.pub

Now ssh the Host name in the config file can login into the google compute engine instance. For example:

$ ssh your-app.asia-east1-c.your-environment

References

google-cloud-platform

Useful Phrases for Socializing

sitting next to someone

  • May I sit here?
  • Do you mind if I sit here?
  • Is this seat taken?

joining a conversation

  • Sorry to bother you...
  • I couldn't help but hear you mention...

active listening

  • I hear you. (agree)
  • Tell me about it! (agree)
  • Me too! (agree)
  • Seriously? (surprise)
  • No way! (surprise)

exchanging contact information

  • Let me give you my number/email...
  • Are you on Facebook/Twitter/etc.?
  • Email/call me sometime.

ending a conversation

  • I should get going. It was great talking to you. (polite)
  • Look at the time! I need to go. Nice chatting with you. (polite)
  • I'm going to call it a night. Talk to you later. (casual)

English socializing

How to build a BaaS platform?

It was my second time to take part in the DeNA study last night. It was wonderful because I started to pay attention to the BaaS(Backend as a Service).

Because of it was a topic from a game company, the BaaS is specially means Game BaaS.
There are many games will be released in a big game company every year. The general features are the same. So why don't make a common platform for all the games? It will reduce cost, save time, and so on.

For example, the platform image will be:

One Proxy Server

This layer is responsible for multi process, authentication, allocate for every API server, manage common API, etc.

By EventMachine

Common APIs

This is the most important layer. Be sure to do the restful architecture. Make the common features to gems.

  • Session API
  • Player API
  • Payment API
  • Asset API
  • Ranking API
  • Information API
  • MasterData API
  • etc

By Sinatra, Sequel and Unicorn.

Databases

  • Master-Slavel (for Game)
  • Master-Slavel Sharding (for Player)
  • Master-Slavel Sharding (for Ranking)
  • Master-Slavel Sharding (for Log)
  • etc

By MySQL and Q4M, also Redis.

One Management Tool

By Rails, Unicorn and SwitchPoint.


All the above are the topic about BaaS yesterday. All rights reserved by DeNA

References

api baas database game platform proxy ruby-on-rails sinatra

The Next Generation Website Workflow

Phase0 | The Given Condition 与件整理
Phase1 | Current Status Analysis 現状把握
Phase2 | Target Setting 目標の明確化
Phase3 | The User Experience ユーザ体験シナリオ
Phase4 | Strategic Planning 戦略策定
Phase5 | Tactics Planning 戦術策定
Phase6 | Contents Design コンテンツ制作
Phase7 | Prototyping Website Design サイト設計
Phase8 | Prototyping System Design システム設計
Phase9 | Development 開発・制作・テスト
Phase10 | Data Injection データ投入
Phase11 | Operation & Management PDCA サイクル
Plan, Do, Check, Action

workflow

Install Ruby on Rails 3.2 on Ubuntu 12.04

Install Git

> sudo apt-get install git

Install packages

> sudo apt-get install curl g++
> sudo apt-get install zlib1g-dev libssl-dev libreadline-dev libyaml-dev libxml2-dev libxslt-dev libsqlite3-dev nodejs

Remove RVM

> rvm implode

Install rbenv

> sudo apt-get install rbenv
> echo 'eval "$(rbenv init -)"' >> ~/.bashrc

If you are using ZSH,change .bashrc to .zshrc,please.

Install ruby-build

> git clone git://github.com/sstephenson/ruby-build.git
> cd ruby-build
> sudo ./install.sh

Install ruby

(Close your terminal and start it again)

> rbenv install 1.9.3-p194
> rbenv rehash
> rbenv global 1.9.3-p194

Confirm setting

> rbenv version

Confirm ruby's path

> which ruby

Confirm ruby's version

> ruby -v

Create .gemrc file

vim ~/.gemrc

install: --no-ri --no-rdoc
update: --no-ri --no-rdoc

Install Ruby on Rails 3.2

> gem install rails -v 3.2.0
> rbenv rehash

Create work folder

> cd
> mkdir work

Create confirm application

> cd work
> rails new foo --slip-bundle
> cd foo

Install depend versions

> bundle install

Generate simple user management function

> rails g scaffold user name:string email:string
> rake db:migrate

Start application

> rails s

Confirm by browser

http://0.0.0.0:3000/users

Reference

ruby-on-rails ubuntu