使用 fabric

fabric是一个用python实现的远程部署工具,它可以批量操作远程服务器。实际工作中,很多时候需要在多台线上机器上执行同样的操作,这时fabric会很有用。

安装

pip install fabric

使用(fabric.py)

# from fabric.api import cd,run,env,hosts,execute,roles
from fabric.api import *

# 远程机器,多个机器fabric会依次启动
env.hosts = [
    'user@10.16.29.176:22',
    'user@10.16.29.44:22'
]
env.password = '*******'

# 进入/home/user目录,执行命令
def test():
    with cd('/home/user'):
        run('du -sh')

# ----在不同的机器上执行不同的任务
env.roledefs = {
    'server1': [
        'user@10.16.29.176:22',
    ],
    'server2': [
        'user@10.16.29.44:22',
    ]
}
env.password = '*******'

# server1执行task1
@roles('server1')
def task1():
    run('ls /home/ -l | wc -l')

# server2执行task2
@roles('server2')
def task2():
    run('du -sh /home/user')

def test1():
    # 遇到报错警告,而不是退出
    with settings(warn_only=True):
    execute(task1)
    execute(task2)
    exit()

# ----接收参数
def hello(foo):
    print 'Hello, %s'%foo;

执行

fab test

用fabric来部署代码也是一件很爽的事情(upload.py)

# coding=utf-8
from datetime import datetime
from fabric.api import *

env.hosts = ['user@10.16.29.176:22']
env.password = '********'

# 本地打包
def pack():
    tar_files = ['*.py', 'testdir/*']
    local('rm -f deploy.tar.gz')
    local('tar -zcvf deploy.tar.gz --exclude=\'*.tar.gz\' --exclude=\'upload.py\' %s' % ' '.join(tar_files))

# 部署
def deploy():
    tmp_dir = '/tmp/deploy.tar.gz'
    put('deploy.tar.gz', tmp_dir)

    tag = datetime.now().strftime('%y.%m.%d_%H.%M.%S')
    deploy_dir = '/home/user/%s' % tag

    run('mkdir %s' % deploy_dir)
    with cd(deploy_dir):
        run('tar -zxvf %s' % tmp_dir)

    run('rm %s' % tmp_dir)

如果文件名不是fabric.py,需要指定文件名

fab -f upload.py pack
fab -f upload.py deploy

参考

关于 .user.ini 文件

昨天转移一个wp站点的时候遇到的一个小问题,绕了不少弯路,写出来分享下。

由于域名不变,按照官网的流程,只需要复制代码,转移数据库文件,在新的服务器配置一个虚拟主机,最后修改IP指向,就OK了。

但是事情总不会那么顺利,访问网站,提示No input file specified,这个错误引发的原因可能性很多,于是看了下nginx的error.log,发现如下错误:

2015/10/14 21:14:13 [error] 1832#0: *130022 FastCGI sent in stderr: "PHP message: PHP Warning:  Unknown: open_basedir restriction in effect. File(/home/****/***/index.php) is not within the allowed path(s): (/tmp/:/proc/) in Unknown on line 0
PHP message: PHP Warning:  Unknown: failed to open stream: Operation not permitted in Unknown on line 0
Unable to open primary script: /home/****/***/index.php (Operation not permitted)" while reading response header from upstream, client: 218.30.116.8, server: zhangrenqin.com, request: "GET /favicon.ico HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "***", referrer: "***"

答案已经很明显了,由于wp使用了open_basedir这个函数,这个函数可以访问系统中的任何文件,安全隐患比较大,要对其进行限制,一般在php.ini里禁用或者限制在哪些目录使用。所以只要在php.ini文件中将当前网站的root目录加入到open_basedir的配置中即可,用:分割

open_basedir=/home/****/***/:/tmp/:/proc/

好了,现在访问就正常了。

这样添加的一个问题是全局生效,如果配置了多个虚拟主机,其中一个站点的权限被拿掉,其他站点也就跟着倒霉。.user.ini文件可以完美的解决这个问题,它能动态扩展php.ini。将上面的配置写入.user.ini,然后放到网站根目录即可。

php.ini的配置清单可参考官方文档:http://www.php.net/manual/en/ini.list.php
其中Changeable为PHP_INI_PERDIR、PHP_INI_USER、PHP_INI_ALL可以配置到.user.ini中,而PHP_INI_SYSTEM只能配置到php.ini。

可以从配置中看到,open_basedir选项在PHP > 5.2.3版本中已经被加入。

Name    Default Changeable  Changelog
open_basedir    NULL    PHP_INI_ALL PHP_INI_SYSTEM in PHP < 5.2.3.

PHP 通过 Thrift 操作 Hbase

Hbase1.0之后的版本中没有提供Thrift源码,需要到官网下载。地址

生成PHP文件:

thrift -gen php <PATH>/Hbase.thrift

启动Hbase和Hbase的Thrift守护程序

<PATH-TO-HBASE>/bin/start-hbase.sh
<PATH-TO-HBASE>/bin/hbase-daemon.sh start thrift

Thrift默认会监听9090端口,下面是一个常用操作的例子。命名为index.php,目录结构为:

gen-php/
lib/
index.php

Hbase里的数据结构如下,表名是blog,包含了articleauthor两个列族。

hbase(main):015:0> scan 'blog'
ROW          COLUMN+CELL
 1           column=article:content, timestamp=1434687037710, value=Hello,World
 1           column=article:tags, timestamp=1434533143824, value=Hadoop,HBase,NoSQL
 1           column=article:title, timestamp=1434533099302, value=Head First HBase
 1           column=author:name, timestamp=1434533489863, value=itsmikej1
 1           column=author:nickname, timestamp=1434687081713, value=\xE5\xB0\x8F\xE9\xBB\x91
1 row(s) in 0.0410 seconds

index.php

<?php
/**
 * Hbase操作
 *
 * @author itsmikej
 * @link https://mikej.me
 */

require_once __DIR__.'/lib/Thrift/ClassLoader/ThriftClassLoader.php';

use Thrift\ClassLoader\ThriftClassLoader;

$loader = new ThriftClassLoader();
$loader->registerNamespace('Thrift', __DIR__ . '/lib');
# $loader->registerDefinition("tService", realpath(__DIR__ . "/gen-php"));
$loader->register();

use Thrift\Protocol\TBinaryProtocol;
use Thrift\Transport\TSocket;
use Thrift\Transport\TBufferedTransport;

require_once __DIR__ . "/gen-php/Hbase/Hbase.php";
require_once __DIR__ . "/gen-php/Hbase/Types.php";


try {
    $socket = new TSocket('127.0.0.1', '9090');
    $socket->setSendTimeout(10000);
    $socket->setRecvTimeout(20000);
    $transport = new TBufferedTransport($socket);
    $protocol = new TBinaryProtocol($transport);
    $client = new Hbase\HbaseClient($protocol);
    $transport->open();

    $tables = $client->getTableNames();
    $table = current($tables);

    $mutations = array();
    $rowKey = '2';
    $row = array(
        'article' => array(
            'content' => 'Hello,Thrift',
            'tags' => 'Hbase',
            'title' => 'Thrift Test'
        ),
        'author' => array(
            'name' => 'tom',
            'nickname' => '小白',
            'sex' => 'man'  # 列可以动态扩展
        )
    );

    foreach ($row as $column_family => $data) {
        foreach ($data as $qualifier => $content) {
            $info = array(
                'column' => $column_family. ':' .$qualifier,
                'value' => $content
            );
            $mutations[] = new \Hbase\Mutation($info);
        }
    }

    # 添加数据
    $client->mutateRow($table, $rowKey, $mutations, array());


    $rows = array(
        '3' => $row,
        '4' => $row,
    );
    $row['author']['sex'] = 'woman';
    $rows['5'] = $row;

    # 批量添加
    $batchRecord = array();
    foreach ($rows as $row_key => $row) {
        $mutations = array();
        foreach ($row as $column_family => $data) {

            foreach ($data as $qualifier => $content) {
                $info = array(
                    'column' => $column_family . ':' . $qualifier,
                    'value' => $content
                );
                $mutations[] = new \Hbase\Mutation($info);
            }
        }
        $batchRecord[] = new \Hbase\BatchMutation(array(
            'row' => $row_key,
            'mutations' => $mutations
        ));

    }
    $client->mutateRows($table, $batchRecord, array());


    # 删除数据
    $client->deleteAllRow($table, $rowKey, array());


    # 查询数据
    $ret = $client->getRow($table, $rowKey, array());
    var_dump($ret, $ret[0]->columns['author:name']->value);


    # 扫描数据
    $scan = $client->scannerOpenWithStop($table, 1, 3, array(), array());
    $ret = $client->scannerGetList($scan, 2);
    var_dump($ret);


    # 过滤器
    $filter = array();
    $filter[] = "SingleColumnValueFilter('author', 'name', =, 'binary:tom')";
    $filter[] = "SingleColumnValueFilter('author', 'sex', =, 'binary:woman')";
    $filterString = implode(" AND ", $filter);
    $scanFilter = new \Hbase\TScan();
    $scanFilter->filterString = $filterString;
    $scanFilter->startRow = 1;
    $scanFilter->stopRow = 6;
    # $scanFilter->columns = array('column' => 'author'); # 指定返回列族

    $scan = $client->scannerOpenWithScan($table, $scanFilter, array());
    $ret = $client->scannerGetList($scan, 2);
    var_dump($ret);

    $transport->close();

} catch (Exception $e) {
    var_dump($e->getMessage());
}

参考

SingleColumnValueFilter过滤器在手册里的说明参数位置是错误的,正确的应该为:

Syntax: SingleColumnValueFilter(‘<family>’, ‘<qualifier>’, <compare operator>, ‘<comparator>’, ‘<filterIfColumnMissing_boolean>, <latest_version_boolean>’)
Syntax: SingleColumnValueFilter(‘<family>’, ‘<qualifier>, <compare operator>, ‘<comparator>’)
Example: “SingleColumnValueFilter (‘FamilyA’, ‘Column1’, <=, ‘abc’,‘true, false’)”
Example: “SingleColumnValueFilter (‘FamilyA’, ‘Column1’, <=, ‘abc’)”

使用 Thrift

Thrift是一个用于不同语言间通讯的开发框架。

首先用Thrift的语法定义好交互细节,命名为hello.thrift

namespace php hello

enum SexType {
  MALE = 1,
  FEMALE = 2
}

struct User {
  1: string firstname,
  2: string lastname,
  3: i32 user_id = 0,
  4: SexType sex,
  5: bool active = false,
  6: optional string description
}

exception InvalidValueException {
  1: i32 error_code,
  2: string error_msg
}

service UserExchange {
  void ping(),
  i32 add_user(1:User u) throws (1: InvalidValueException e),
  User get_user(1:i32 uid) throws (1: InvalidValueException e),
  oneway void clear_list()
}

生成php文件

thrift -r --gen php hello.thrift

生成python文件

thrift -r --gen py hello.thrift

python服务端,命名为python_server.py

#!/usr/bin/env python

import sys
sys.path.append('./gen-py')

from hello import UserExchange
from hello.ttypes import *

from thrift.transport import TSocket
from thrift.transport import TTransport
from thrift.protocol import TBinaryProtocol
from thrift.server import TServer

users = []

class UserManagerHandler:
    def __init__(self):
        pass
        #self.log = {}

    def ping(self):
        print 'ping()'

    def add_user(self, user):
        if user.firstname == None:
            raise InvalidValueException(1,'no firstname exception')
        if user.lastname == None:
            raise InvalidValueException(2, 'no lastname exception')
        if user.user_id <= 0:
            raise InvalidValueException(3, 'wrong user_id')
        if user.sex != SexType.MALE and user.sex != SexType.FEMALE:
            raise InvalidValueException(4, 'wrong sex id')
        print 'Processing user '+user.firstname+' '+user.lastname
        users.append(user)
        print users
        return True

    def get_user(self, user_id):
        if user_id < 0:
            raise InvalidValueException(5, 'wrong id')
        return users[user_id]
        
    def clear_list(self):
        print 'Clearing list'
        print users
        del users [:]
        print users



handler = UserManagerHandler()
processor = UserExchange.Processor(handler)
transport = TSocket.TServerSocket(port=9090)
tfactory = TTransport.TBufferedTransportFactory()
pfactory = TBinaryProtocol.TBinaryProtocolFactory()

server = TServer.TSimpleServer(processor, transport, tfactory, pfactory)

# You could do one of these for a multithreaded server
#server = TServer.TThreadedServer(processor, transport, tfactory, pfactory)
#server = TServer.TThreadPoolServer(processor, transport, tfactory, pfactory)

print 'Starting the server...'
server.serve()
print 'done.'

php客户端,命名为php_client.php

<?php

require_once __DIR__.'/lib/Thrift/ClassLoader/ThriftClassLoader.php';

use Thrift\ClassLoader\ThriftClassLoader;

$loader = new ThriftClassLoader();
$loader->registerNamespace('Thrift', __DIR__ . '/lib');
$loader->registerDefinition("tService", realpath(__DIR__ . "/gen-php"));
$loader->register();

use Thrift\Protocol\TBinaryProtocol;
use Thrift\Transport\TSocket;
use Thrift\Transport\THttpClient;
use Thrift\Transport\TBufferedTransport;
use Thrift\Exception\TException;

require_once __DIR__ . "/gen-php/hello/userExchange.php";
require_once __DIR__ . "/gen-php/hello/Types.php";

try {

    $socket = new TSocket('localhost', 9090);
    $transport = new TBufferedTransport($socket, 1024, 1024);
    $protocol = new TBinaryProtocol($transport);
    $client = new \hello\userExchangeClient($protocol);

    $transport->open();
    $client->ping();
    $u = new \hello\User();
    $u->user_id = 1;
    $u->firstname = 'John';
    $u->lastname = 'Smith';
    $u->sex = \hello\SexType::MALE;
    if ($client->add_user($u))
    {
        echo 'user added succesfully</br>'."\n";
    }
    
    var_dump($client->get_user(0));
    
    $client->clear_list();
    
    
    
    $u2 = new \hello\User();
    $client->add_user($u2);
    
} catch (\hello\InvalidValueException $e) {
    echo $e->error_msg."<br/>\n";
}

最后先运行python_server.py启动服务,然后执行php_client.php,一切正常的话会server端会看到如下数据

Starting the server...
ping()
Processing user John Smith
[User(user_id=1, description=None, firstname='John', lastname='Smith', sex=1, active=False)]
Clearing list
[User(user_id=1, description=None, firstname='John', lastname='Smith', sex=1, active=False)]
[]

客户端会看到

user added succesfully</br>
class hello\User#9 (6) {
  public $firstname =>
  string(4) "John"
  public $lastname =>
  string(5) "Smith"
  public $user_id =>
  int(1)
  public $sex =>
  int(1)
  public $active =>
  bool(false)
  public $description =>
  NULL
}
no firstname exception<br/>

这就表明数据已经正确的请求和响应了。

参考

Moonrise 开发框架

简介

Moonrise是我用业余时间写的一个PHP框架,借鉴了其他很多框架的思想,比如CILaravel,当然使用了一些优秀第三方Composer包。为了能让框架尽快跑起来,很多细节处理得不够好,目前还在完善中。

Github地址:https://github.com/itsmikej/Moonrise。欢迎Fork和Pull Request

核心功能

  • MVC
  • 路由自动映射(支持web和cli)
  • 数据过滤
  • 异常和错误处理
  • 数据驱动支持Mysqli和PDO, 以及Memcache, MongoDB, Redis
  • ORM使用Active RecordEloquent也是可以的)
  • ......

环境要求

  • PHP >= 5.4.0
  • Composer(以及库中所需扩展)
  • Mysqli, PDO扩展

后记

  • Moonrise是因为小时候在夏天喜欢洗完澡躺在院子里的凉铺上边扇扇子边看星空和月亮直到睡着。。。