Qiang Blog

Just another zhangjingqiang's blog.

MongoDB Deployment Checklist

Hardware

  • RAM
  • Disk space
  • Disk speed
  • CPU
  • Network

Security

  • Protection of network traffic
  • Access control

Monitoring

  • Hardware usage
  • Health checks
  • MMS Monitoring
  • Client performance monitoring

Disaster recovery

  • Evaluate risk
  • Have a plan
  • Test your plan
  • Have a backup plan

Performance

  • Load testing

MongoDB in Action, 2nd Edition

deployment mongodb

DevOps Deployment Pipeline Steps

  1. Checkout the code
  2. Run pre-deployment tests
  3. Compile and/or package the code
  4. Build the container
  5. Push the container to the registry
  6. Deploy the container to the production server
  7. Integrate the container
  8. Run post-deployment tests
  9. Push the tests container to the registry

The DevOps 2.0 Toolkit

deployment devops

Django Workflow

Setup

  • Within a new directory, create and activate a new virtualenv.
  • Install Django.
  • Create your project: django-admin.py startproject
  • Create a new app: python manage.py startapp
  • Add your app to the INSTALLED_APPS tuple.

Add Basic URLs and Views

  • Map your Project’s urls.py file to the new app.
  • In your App directory, create a urls.py file to define your App’s URLs.
  • Add views, associated with the URLs, in your App’s views.py; make sure they return a HttpResponse object. Depending on the situation, you may also need to query the model (database) to get the required data back requested by the end user.

Templates and Static Files

  • Create a templates and static directory within your project root.
  • Update settings.py to include the paths to your templates.
  • Add a template (HTML file) to the templates directory. Within that file, you can include the static file with - {% load static %} and {% static "filename" %}. Also, you may need to pass in data requested by the user.
  • Update the views.py file as necessary.

Models and Databases

  • Update the database engine to settings.py (if necessary, as it defaults to SQLite).
  • Create and apply a new migration.
  • Create a super user.
  • Add an admin.py file in each App that you want access to in the Admin.
  • Create your models for each App.
  • Create and apply a new migration. (Do this whenever you make any change to a model).

Forms

  • Create a forms.py file at the App to define form-related classes; define your ModelForm classes here.
  • Add or update a view for handling the form logic - e.g., displaying the form, saving the form data, alerting the user about validation errors, etc.
  • Add or update a template to display the form.
  • Add a urlpattern in the App’s urls.py file for the new view.

User Registration

  • Create a UserForm
  • Add a view for creating a new user.
  • Add a template to display the form.
  • Add a urlpattern for the new view.

User Login

  • Add a view for handling user credentials.
  • Create a template to display a login form.
  • Add a urlpattern for the new view.

Setup the template structure

  • Find the common parts of each page (i.e., header, sidebar, footer).
  • Add these parts to a base template
  • Create specific. templates that inherent from the base template.

Reference

Real Python Part 2

django workflow

How to setup Local Infrastructure Development with Ansible and Vagrant by Ad-Hoc Commands

Make work directory

$ mkdir work && cd work

Make Vagrantfile

# -*- mode: ruby -*-
# vi: set ft=ruby :

VAGRANTFILE_API_VERSION = "2"

Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
  config.ssh.insert_key = false
    config.vm.provider :virtualbox do |vb|
    vb.customize ["modifyvm", :id, "--memory", "256"]
  end

  # Application server 1.
  config.vm.define "app1" do |app|
    app.vm.hostname = "orc-app1.dev"
    app.vm.box = "geerlingguy/centos7"
    app.vm.network :private_network, ip: "192.168.60.4"
  end

  # Application server 2.
  config.vm.define "app2" do |app|
    app.vm.hostname = "orc-app2.dev"
    app.vm.box = "geerlingguy/centos7"
    app.vm.network :private_network, ip: "192.168.60.5"
  end

  # Database server.
  config.vm.define "db" do |db|
    db.vm.hostname = "orc-db.dev"
    db.vm.box = "geerlingguy/centos7"
    db.vm.network :private_network, ip: "192.168.60.6"
  end
end

Run vagrant

$ vagrant up

Edit /etc/ansible/hosts

# Lines beginning with a # are comments, and are only included for
# illustration. These comments are overkill for most inventory files.

# Application servers
[app]
192.168.60.4
192.168.60.5

# Database server
[db]
192.168.60.6

# Group 'multi' with all servers
[multi:children]
app
db

# Variables that will be applied to all servers
[multi:vars]
ansible_ssh_user=vagrant
ansible_ssh_private_key_file=~/.vagrant.d/insecure_private_key

Setup

$ ansible multi -a "hostname"
$ ansible multi -a "hostname" -f 1
$ ansible multi -a "df -h"
$ ansible multi -a "free -m"
$ ansible multi -a "date"
$ ansible multi -s -m yum -a "name=ntp state=present"
$ ansible multi -s -m service -a "name=ntpd state=started enabled=yes"
$ ansible multi -s -a "service ntpd stop"
$ ansible multi -s -a "ntpdate -q 0.rhel.pool.ntp.org"
$ ansible multi -s -a "service ntpd start"
$ ansible app -s -m yum -a "name=MySQL-python state=present"
$ ansible app -s -m yum -a "name=python-setuptools state=present"
$ ansible app -s -m easy_install -a "name=django"
$ ansible app -a "python -c 'import django; print django.get_version()'"
$ ansible db -s -m yum -a "name=mariadb-server state=present"
$ ansible db -s -m service -a "name=mariadb state=started enabled=yes"
$ ansible db -s -a "iptables -F"
$ ansible db -s -a "iptables -A INPUT -s 192.168.60.0/24 -p tcp -m tcp --dport 3306 -j ACCEPT"
$ ansible db -s -m yum -a "name=MySQL-python state=present"
$ ansible db -s -m mysql_user -a "name=django host=% password=12345 priv=*.*:ALL state=present"
$ ansible app -s -a "service ntpd status"
$ ansible app -s -a "service ntpd restart" --limit "192.168.60.4"
$ ansible app -s -a "service ntpd restart" --limit "*.4"
$ ansible app -s -a "service ntpd restart" --limit ~".*\.4"
$ ansible app -s -m group -a "name=admin state=present"
$ ansible app -s -m user -a "name=johndoe group=admin createhome=yes"
$ ansible app -s -m user -a "name=johndoe state=absent remove=yes"
$ ansible multi -m stat -a "path=/etc/environment"
$ ansible multi -m copy -a "src=/etc/hosts dest=/tmp/hosts"
$ ansible multi -s -m fetch -a "src=/etc/hosts dest=/tmp"
$ ansible multi -m file -a "dest=/tmp/test mode=644 state=directory"
$ ansible multi -m file -a "src=/src/symlink dest=/dest/symlink owner=root group=root state=link"
$ ansible multi -m file -a "dest=/tmp/test state=absent"
$ ansible multi -s -B 3600 -a "yum -y update"
$ ansible multi -s -m async_status -a "jid=763350539037"
$ ansible multi -B 3600 -P 0 -a "/path/to/fire-and-forget-script.sh"
$ ansible multi -s -a "tail /var/log/messages"
$ ansible multi -s -m shell -a "tail /var/log/messages | grep ansible-command | wc -l"
$ ansible multi -s -m cron -a "name='daily-cron-all-servers' hour=4 job='/path/to/daily-script.sh'"
$ ansible multi -s -m cron -a "name='daily-cron-all-servers' state=absent"
$ ansible app -s -m git -a "repo=git://example.com/path/to/repo.git dest=/opt/myapp update=yes version=1.2.4"
$ ansible app -s -a "/opt/myapp/update.sh"

Reference

Ansible for DevOps(Chapter 3)

ansible vagrant

Kubernetes deploy Rails app to Google Container Engine

Create a Rails App

rails new todo
rails g scaffold task title:string notes:string due:datetime completion:integer

Config Gemfile

gem 'rails', '4.2.0'
gem 'sass-rails', '~> 5.0'
gem 'uglifier', '>= 1.3.0'
gem 'coffee-rails', '~> 4.1.0'

gem 'jquery-rails'
gem 'turbolinks'
gem 'jbuilder', '~> 2.0'
gem 'sdoc', '~> 0.4.0', group: :doc

group :production do
  gem 'pg'
end

gem 'capistrano-rails', group: :development

group :development, :test do
  gem 'byebug'
  gem 'web-console', '~> 2.0'
  gem 'spring'
  gem 'sqlite3'
end

Config config/database.yml

production:
  <<: *default
  adapter: postgresql
  encoding: unicode
  database: todo_production
  username: <%= ENV['PG_ENV_POSTGRES_USER'] %>
  password: <%= ENV['PG_ENV_POSTGRES_PASSWORD'] %>
  host:     <%= ENV['PG_PORT_5432_TCP_ADDR'] %>

Add Dockerfile at Rails Root

FROM rails:onbuild

ENV RAILS_ENV=production

CMD ["sh", "/usr/src/app/init.sh"]

Use official rails on docker hub: https://hub.docker.com/_/rails/

Make init.sh at Rails Root

export SECRET_KEY_BASE=$(bundle exec rake secret)

bundle exec rake db:create db:migrate
bundle exec rails server -b 0.0.0.0

Deploying Locally

docker build -t your_user_name/todo .

Run PostgreSQL Database

docker run --name db -e POSTGRES_PASSWORD=password -e POSTGRES_USER=rails -d postgres

Check Process

$ docker ps

CONTAINER ID        IMAGE               COMMAND                CREATED             STATUS              PORTS               NAMES
30e1420e8a57        postgres:9          "/docker-entrypoint.   7 seconds ago       Up 6 seconds        5432/tcp            db

Run Web App

docker run --name web -d -p 3000:3000 --link db:pg your_user_name/todo

Deploying with Kubernetes on Google Container Engine

Create a project on http://console.developers.google.com. Enable Google Container Engine API, Compute Engine API, and Cloud Storage API.

Use Google Cloud SDK do local config

gcloud config set project my-project-id
gcloud config set compute/zone asia-east1-a

Create Google Container Cluster

gcloud container clusters create todo
gcloud container clusters get-credentials todo

Building and Sharing the Container Image

docker build -t todo .
docker tag -f todo gcr.io/my_project_id/todo:v1
gcloud docker push gcr.io/my_project_id/todo:v1

Deploying the Database

Make Database Pod File

# db-pod.yml
apiVersion: v1
kind: Pod
metadata:
  labels:
    name: db
  name: db
spec:
  containers:
  - image: postgres
    name: db
    env:
    - name: POSTGRES_PASSWORD
      value: password
    - name: POSTGRES_USER
      value: rails
    ports:
    - name: pg
      containerPort: 5432
      hostPort: 5432

Create Database Pod

kubectl create -f db-pod.yml

Make Database Service File

#db-service.yml
apiVersion: v1
kind: Service
metadata:
  labels:
    name: db
  name: db
spec:
  ports:
    - port: 5432
      targetPort: 5432
  selector:
    name: db

Create Database Service

kubectl create -f db-service.yml

Deploying Rails

Make Web Controller File

# web-controller.yml
apiVersion: v1
kind: ReplicationController
metadata:
  labels:
    name: web
  name: web-controller
spec:
  replicas: 2
  selector:
    name: web
  template:
    metadata:
      labels:
        name: web
    spec:
      containers:
      - image: gcr.io/YOUR_PROJECT_ID_HERE/todo:v1
        env:
          - name: POSTGRES_PASSWORD
            value: password
          - name: POSTGRES_USER
            value: rails
        name: web
        ports:
        - containerPort: 3000
          name: http-server

Create Web Controller

kubectl create -f web-controller.yml

Make Web Service File

# web-service.yml
apiVersion: v1
kind: Service
metadata:
  name: web
  labels:
    name: web
spec:
  type: LoadBalancer
  ports:
    - port: 3000
      targetPort: 3000
      protocol: TCP
  selector:
    name: web

Create Web Service

kubectl create -f web-service.yml

Check pods, controllers, services on kubernetes

kubectl get pods
kubectl get rc
kubectl get services

Accessing the Running Site

Check Nodes

kubectl get nodes
NAME                          LABELS                                               STATUS
gke-todo-5ccd2616-node-3bj5   kubernetes.io/hostname=gke-todo-5ccd2616-node-3bj5   Ready
gke-todo-5ccd2616-node-7afx   kubernetes.io/hostname=gke-todo-5ccd2616-node-7afx   Ready
gke-todo-5ccd2616-node-fd7e   kubernetes.io/hostname=gke-todo-5ccd2616-node-fd7e   Ready

Create Firewall

gcloud compute firewall-rules create --allow=tcp:3000 --target-tags=gke-todo-5ccd2616-node todo

Get Address

gcloud compute forwarding-rules list
NAME                             REGION      IP_ADDRESS    IP_PROTOCOL TARGET
9ea34fd96220f1e15959942001a0f5ae asia-east1-a 104.197.XXX.XXX TCP         asia-east1-a/targetPools/9ea34fd96220f1e15959942001a0f5ae

Access IP_ADDRESS

http://104.197.XXX.XXX:3000/tasks


Thanks To

docker google-cloud-platform kubernetes

Deploy Rails with Docker

A sample using Rails + MySQL

Install Docker on Mac OS X

http://docs.docker.com/engine/installation/mac/

Create docker machine

$ docker-machine create --driver virtualbox testapp

Import ENV

eval "$(docker-machine env testapp)"

Create Dockerfile

FROM ruby:2.2.0
RUN apt-get update -qq && apt-get install -y build-essential mysql-client libmysqlclient-dev nodejs
RUN mkdir /testapp
WORKDIR /testapp
ADD Gemfile /testapp/Gemfile
ADD Gemfile.lock /testapp/Gemfile.lock
RUN bundle install
ADD . /testapp

Create docker-compose.yml

db:
  image: mysql
  environment:
    MYSQL_ROOT_PASSWORD: 1234
  ports:
    - "3306:3306"
web:
  build: .
  command: bundle exec rails s -p 3000 -b '0.0.0.0'
  volumes:
    - .:/testapp
  ports:
    - "3000:3000"
  links:
    - db

Modify config/database.yml

default: &default
  adapter: mysql2
  encoding: utf8
  pool: 5
  username: root
  password: 1234
  host: db

development:
  <<: *default
  database: testapp_development

test:
  <<: *default
  database: testapp_test

production:
  <<: *default
  database: testapp_production

Build image

$ docker-compose build

Boot app

$ docker-compose up

Create database

$ docker-compose run web rake db:create

Get docker machine's ip

$ docker-machine ip testapp

Access app

http://[IP_ADDRESS]:3000


Reference

https://docs.docker.com/compose/rails/

docker mysql ruby-on-rails

How to use git subtree

Example

repository 1

app1

app1
|-app
|-submodule/app2

repository 2

app2
|-something

What to do

After change something in submodule/app2 in repository 1, push the change to repository 2, too.

Command

# Under repository 1
git subtree push --prefix submodule/app2 --squash https://github.com/your_account/your_repository.git your_branch

When to use

Such as rails engine

http://guides.rubyonrails.org/engines.html

git

How to use vim 7.4 after install it by brew on MacOS

Problem

After install vim 7.4 by brew:

brew install vim

Type brew info vim, will show:

vim: stable 7.4.891, HEAD
Vi "workalike" with many additional features
http://www.vim.org/
Conflicts with: ex-vi
/usr/local/Cellar/vim/7.4.891 (1612 files, 28M) *
  Built from source
From: https://github.com/Homebrew/homebrew/blob/master/Library/Formula/vim.rb
...

But if type vim, will show:

VIM - Vi IMproved
  version 7.3
...

The reason is the PATH.

Solution

Install macvim:

brew install macvim

Add these settings to ~/.bashrc or ~/.zshrc:

alias vi='/Applications/MacVim.app/Contents/MacOS/Vim'
alias vim='/Applications/MacVim.app/Contents/MacOS/Vim'
alias gvi='/Applications/MacVim.app/Contents/MacOS/Vim -g'

Then type vim will use 7.4 version.

Reference

https://www.reddit.com/r/vim/comments/2ukp5j/starting_homebrew_version_of_vim/

mac vim

How to start Google Cloud Platform

Install SDK

https://cloud.google.com/sdk/

$ curl https://sdk.cloud.google.com | bash
$ exec -l $SHELL
$ gcloud init

Config SSH

$ gcloud compute config-ssh

Will generate these files under ~/.ssh:

➜  .ssh  tree
.
├── config
├── google_compute_engine
└── google_compute_engine.pub

Now ssh the Host name in the config file can login into the google compute engine instance. For example:

$ ssh your-app.asia-east1-c.your-environment

References

google-cloud-platform

Useful Phrases for Socializing

sitting next to someone

  • May I sit here?
  • Do you mind if I sit here?
  • Is this seat taken?

joining a conversation

  • Sorry to bother you...
  • I couldn't help but hear you mention...

active listening

  • I hear you. (agree)
  • Tell me about it! (agree)
  • Me too! (agree)
  • Seriously? (surprise)
  • No way! (surprise)

exchanging contact information

  • Let me give you my number/email...
  • Are you on Facebook/Twitter/etc.?
  • Email/call me sometime.

ending a conversation

  • I should get going. It was great talking to you. (polite)
  • Look at the time! I need to go. Nice chatting with you. (polite)
  • I'm going to call it a night. Talk to you later. (casual)

English socializing